2023-01-15T14:15:42.034 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2023-01-15T14:15:42.035 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} --name rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi --archive /home/teuthworker/archive/rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi/7137228 --verbose --owner scheduled_rfriedma@teuthology -- /home/teuthworker/archive/rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi/7137228/orig.config.yaml 2023-01-15T14:15:42.074 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-15T14:15:42.150 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi/7137228 branch: wip-rf-newsched-v3 description: rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects} email: null first_in_suite: false job_id: '7137228' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: ubuntu os_version: '20.04' override: conf: osd: osd deep scrub update digest min age: 0 overrides: admin_socket: branch: wip-rf-newsched-v3 ceph: conf: global: mon election default strategy: 3 ms type: async osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 filestore odsync write: true osd backoff on degraded: true osd backoff on peering: true osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd max backfills: 2 osd max markdown count: 1000 osd max scrubs: 3 osd mclock override recovery settings: true osd objectstore: filestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd sloppy crc: true osd snap trim sleep: 0.5 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: a21b2f4fd40d883dac62c8cd315cbb54848ee067 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: flavor: default sha1: a21b2f4fd40d883dac62c8cd315cbb54848ee067 thrashosds: min_in: 4 workunit: branch: wip-rf-newsched-v3 sha1: a21b2f4fd40d883dac62c8cd315cbb54848ee067 owner: scheduled_rfriedma@teuthology priority: 95 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: a21b2f4fd40d883dac62c8cd315cbb54848ee067 sleep_before_teardown: 0 suite: rados:thrash suite_branch: wip-rf-newsched-v3 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a21b2f4fd40d883dac62c8cd315cbb54848ee067/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: a21b2f4fd40d883dac62c8cd315cbb54848ee067 targets: smithi090.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi141.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0= tasks: - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: chance_pgnum_grow: 2 chance_pgpnum_fix: 1 timeout: 1200 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 pool_snaps: true teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 19d18db866afdc31fd6586f558fc29b95a87ccfb timestamp: 2023-01-12_16:28:35 tube: smithi user: rfriedma verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1670 2023-01-15T14:15:42.151 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a21b2f4fd40d883dac62c8cd315cbb54848ee067/qa; will attempt to use it 2023-01-15T14:15:42.151 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a21b2f4fd40d883dac62c8cd315cbb54848ee067/qa/tasks 2023-01-15T14:15:42.152 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-15T14:15:42.153 INFO:teuthology.task.internal:Checking packages... 2023-01-15T14:15:42.182 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'a21b2f4fd40d883dac62c8cd315cbb54848ee067' 2023-01-15T14:15:42.182 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-15T14:15:42.182 INFO:teuthology.packaging:ref: None 2023-01-15T14:15:42.183 INFO:teuthology.packaging:tag: None 2023-01-15T14:15:42.183 INFO:teuthology.packaging:branch: wip-rf-newsched-v3 2023-01-15T14:15:42.183 INFO:teuthology.packaging:sha1: a21b2f4fd40d883dac62c8cd315cbb54848ee067 2023-01-15T14:15:42.183 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-rf-newsched-v3 2023-01-15T14:15:42.343 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-1736-ga21b2f4f-1focal 2023-01-15T14:15:42.343 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-15T14:15:42.352 INFO:teuthology.task.internal:no buildpackages task found 2023-01-15T14:15:42.352 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-15T14:15:42.375 INFO:teuthology.task.internal:Saving configuration 2023-01-15T14:15:42.391 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-15T14:15:42.402 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-15T14:15:42.425 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi090.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi/7137228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-15 14:08:37.258801', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-01-15T14:15:42.447 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi141.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi/7137228', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-15 14:08:37.257491', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2023-01-15T14:15:42.447 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-15T14:15:42.459 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-01-15T14:15:42.459 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-01-15T14:15:42.460 INFO:teuthology.run_tasks:Running task console_log... 2023-01-15T14:15:42.537 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe22390ec10>, signals=[15]) 2023-01-15T14:15:42.537 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-15T14:15:42.546 INFO:teuthology.task.internal:Opening connections... 2023-01-15T14:15:42.546 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi090.front.sepia.ceph.com 2023-01-15T14:15:42.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-15T14:15:42.595 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi141.front.sepia.ceph.com 2023-01-15T14:15:42.595 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-15T14:15:42.647 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-15T14:15:42.656 DEBUG:teuthology.orchestra.run.smithi090:> uname -m 2023-01-15T14:15:42.676 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2023-01-15T14:15:42.676 DEBUG:teuthology.orchestra.run.smithi090:> cat /etc/os-release 2023-01-15T14:15:42.724 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="Ubuntu" 2023-01-15T14:15:42.724 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-15T14:15:42.724 INFO:teuthology.orchestra.run.smithi090.stdout:ID=ubuntu 2023-01-15T14:15:42.724 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE=debian 2023-01-15T14:15:42.724 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-15T14:15:42.725 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="20.04" 2023-01-15T14:15:42.725 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-15T14:15:42.725 INFO:teuthology.orchestra.run.smithi090.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-15T14:15:42.725 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-15T14:15:42.725 INFO:teuthology.orchestra.run.smithi090.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-15T14:15:42.725 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_CODENAME=focal 2023-01-15T14:15:42.725 INFO:teuthology.orchestra.run.smithi090.stdout:UBUNTU_CODENAME=focal 2023-01-15T14:15:42.726 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2023-01-15T14:15:42.741 DEBUG:teuthology.orchestra.run.smithi141:> uname -m 2023-01-15T14:15:42.747 INFO:teuthology.orchestra.run.smithi141.stdout:x86_64 2023-01-15T14:15:42.748 DEBUG:teuthology.orchestra.run.smithi141:> cat /etc/os-release 2023-01-15T14:15:42.799 INFO:teuthology.orchestra.run.smithi141.stdout:NAME="Ubuntu" 2023-01-15T14:15:42.799 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2023-01-15T14:15:42.799 INFO:teuthology.orchestra.run.smithi141.stdout:ID=ubuntu 2023-01-15T14:15:42.799 INFO:teuthology.orchestra.run.smithi141.stdout:ID_LIKE=debian 2023-01-15T14:15:42.800 INFO:teuthology.orchestra.run.smithi141.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2023-01-15T14:15:42.800 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_ID="20.04" 2023-01-15T14:15:42.800 INFO:teuthology.orchestra.run.smithi141.stdout:HOME_URL="https://www.ubuntu.com/" 2023-01-15T14:15:42.800 INFO:teuthology.orchestra.run.smithi141.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-01-15T14:15:42.800 INFO:teuthology.orchestra.run.smithi141.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-01-15T14:15:42.800 INFO:teuthology.orchestra.run.smithi141.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-01-15T14:15:42.800 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_CODENAME=focal 2023-01-15T14:15:42.801 INFO:teuthology.orchestra.run.smithi141.stdout:UBUNTU_CODENAME=focal 2023-01-15T14:15:42.801 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server 2023-01-15T14:15:42.815 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-15T14:15:42.837 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-15T14:15:42.843 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-15T14:15:42.843 DEBUG:teuthology.orchestra.run.smithi090:> test '!' -e /home/ubuntu/cephtest 2023-01-15T14:15:42.845 DEBUG:teuthology.orchestra.run.smithi141:> test '!' -e /home/ubuntu/cephtest 2023-01-15T14:15:42.851 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-15T14:15:42.856 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-15T14:15:42.857 DEBUG:teuthology.orchestra.run.smithi090:> test -z $(ls -A /var/lib/ceph) 2023-01-15T14:15:42.892 DEBUG:teuthology.orchestra.run.smithi141:> test -z $(ls -A /var/lib/ceph) 2023-01-15T14:15:42.906 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-15T14:15:42.957 INFO:teuthology.run_tasks:Running task kernel... 2023-01-15T14:15:42.971 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-15T14:15:42.971 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-15T14:15:42.972 DEBUG:teuthology.orchestra.run.smithi090:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-15T14:15:42.972 DEBUG:teuthology.orchestra.run.smithi141:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-15T14:15:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-15T14:15:42.978 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2023-01-15T14:15:42.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-15T14:15:42.979 DEBUG:teuthology.orchestra.run.smithi141:> uname -r 2023-01-15T14:15:43.024 INFO:teuthology.orchestra.run.smithi090.stdout:5.4.0-124-generic 2023-01-15T14:15:43.025 INFO:teuthology.task.kernel:Running kernel on smithi090: 5.4.0-124-generic 2023-01-15T14:15:43.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get clean 2023-01-15T14:15:43.029 INFO:teuthology.orchestra.run.smithi141.stdout:5.4.0-124-generic 2023-01-15T14:15:43.029 INFO:teuthology.task.kernel:Running kernel on smithi141: 5.4.0-124-generic 2023-01-15T14:15:43.029 DEBUG:teuthology.orchestra.run.smithi141:> sudo apt-get clean 2023-01-15T14:15:43.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get update 2023-01-15T14:15:43.119 DEBUG:teuthology.orchestra.run.smithi141:> sudo apt-get update 2023-01-15T14:15:43.435 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-15T14:15:43.435 INFO:teuthology.orchestra.run.smithi090.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-01-15T14:15:43.489 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-01-15T14:15:43.580 INFO:teuthology.orchestra.run.smithi141.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-15T14:15:43.580 INFO:teuthology.orchestra.run.smithi141.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2023-01-15T14:15:43.593 INFO:teuthology.orchestra.run.smithi141.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2023-01-15T14:15:43.825 INFO:teuthology.orchestra.run.smithi090.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-01-15T14:15:43.899 INFO:teuthology.orchestra.run.smithi090.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,948 kB] 2023-01-15T14:15:43.934 INFO:teuthology.orchestra.run.smithi141.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2023-01-15T14:15:44.015 INFO:teuthology.orchestra.run.smithi141.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,948 kB] 2023-01-15T14:15:44.047 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [772 kB] 2023-01-15T14:15:44.155 INFO:teuthology.orchestra.run.smithi141.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,324 kB] 2023-01-15T14:15:44.356 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [540 kB] 2023-01-15T14:15:44.356 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [2,324 kB] 2023-01-15T14:15:44.370 INFO:teuthology.orchestra.run.smithi090.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [318 kB] 2023-01-15T14:15:44.379 INFO:teuthology.orchestra.run.smithi090.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.7 kB] 2023-01-15T14:15:44.379 INFO:teuthology.orchestra.run.smithi090.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [29.1 kB] 2023-01-15T14:15:44.379 INFO:teuthology.orchestra.run.smithi090.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,449 kB] 2023-01-15T14:15:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [205 kB] 2023-01-15T14:15:44.449 INFO:teuthology.orchestra.run.smithi090.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [616 B] 2023-01-15T14:15:44.449 INFO:teuthology.orchestra.run.smithi090.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [574 kB] 2023-01-15T14:15:44.465 INFO:teuthology.orchestra.run.smithi090.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [783 kB] 2023-01-15T14:15:44.467 INFO:teuthology.orchestra.run.smithi141.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [540 kB] 2023-01-15T14:15:44.481 INFO:teuthology.orchestra.run.smithi141.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [318 kB] 2023-01-15T14:15:44.486 INFO:teuthology.orchestra.run.smithi090.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [152 kB] 2023-01-15T14:15:44.489 INFO:teuthology.orchestra.run.smithi090.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.9 kB] 2023-01-15T14:15:44.490 INFO:teuthology.orchestra.run.smithi141.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [11.7 kB] 2023-01-15T14:15:44.490 INFO:teuthology.orchestra.run.smithi141.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [1,449 kB] 2023-01-15T14:15:44.521 INFO:teuthology.orchestra.run.smithi090.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [402 kB] 2023-01-15T14:15:44.532 INFO:teuthology.orchestra.run.smithi090.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.2 kB] 2023-01-15T14:15:44.533 INFO:teuthology.orchestra.run.smithi090.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,547 kB] 2023-01-15T14:15:44.554 INFO:teuthology.orchestra.run.smithi141.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [29.1 kB] 2023-01-15T14:15:44.555 INFO:teuthology.orchestra.run.smithi141.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [205 kB] 2023-01-15T14:15:44.560 INFO:teuthology.orchestra.run.smithi141.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [616 B] 2023-01-15T14:15:44.561 INFO:teuthology.orchestra.run.smithi141.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [574 kB] 2023-01-15T14:15:44.576 INFO:teuthology.orchestra.run.smithi141.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [783 kB] 2023-01-15T14:15:44.582 INFO:teuthology.orchestra.run.smithi090.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [30.4 kB] 2023-01-15T14:15:44.582 INFO:teuthology.orchestra.run.smithi090.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [218 kB] 2023-01-15T14:15:44.584 INFO:teuthology.orchestra.run.smithi141.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [772 kB] 2023-01-15T14:15:44.589 INFO:teuthology.orchestra.run.smithi090.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [616 B] 2023-01-15T14:15:44.590 INFO:teuthology.orchestra.run.smithi090.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,014 kB] 2023-01-15T14:15:44.598 INFO:teuthology.orchestra.run.smithi141.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [152 kB] 2023-01-15T14:15:44.601 INFO:teuthology.orchestra.run.smithi141.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [16.9 kB] 2023-01-15T14:15:44.602 INFO:teuthology.orchestra.run.smithi141.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [402 kB] 2023-01-15T14:15:44.611 INFO:teuthology.orchestra.run.smithi141.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [16.2 kB] 2023-01-15T14:15:44.612 INFO:teuthology.orchestra.run.smithi090.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [706 kB] 2023-01-15T14:15:44.612 INFO:teuthology.orchestra.run.smithi141.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [30.4 kB] 2023-01-15T14:15:44.612 INFO:teuthology.orchestra.run.smithi141.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,547 kB] 2023-01-15T14:15:44.629 INFO:teuthology.orchestra.run.smithi090.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [235 kB] 2023-01-15T14:15:45.197 INFO:teuthology.orchestra.run.smithi090.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.2 kB] 2023-01-15T14:15:45.197 INFO:teuthology.orchestra.run.smithi090.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.2 kB] 2023-01-15T14:15:45.197 INFO:teuthology.orchestra.run.smithi090.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,408 B] 2023-01-15T14:15:45.197 INFO:teuthology.orchestra.run.smithi090.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [604 B] 2023-01-15T14:15:45.197 INFO:teuthology.orchestra.run.smithi090.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-01-15T14:15:45.197 INFO:teuthology.orchestra.run.smithi090.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.9 kB] 2023-01-15T14:15:45.197 INFO:teuthology.orchestra.run.smithi090.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-01-15T14:15:45.198 INFO:teuthology.orchestra.run.smithi141.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [218 kB] 2023-01-15T14:15:45.198 INFO:teuthology.orchestra.run.smithi141.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [616 B] 2023-01-15T14:15:45.198 INFO:teuthology.orchestra.run.smithi141.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [706 kB] 2023-01-15T14:15:45.198 INFO:teuthology.orchestra.run.smithi141.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,014 kB] 2023-01-15T14:15:45.198 INFO:teuthology.orchestra.run.smithi141.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [235 kB] 2023-01-15T14:15:45.198 INFO:teuthology.orchestra.run.smithi141.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [23.2 kB] 2023-01-15T14:15:45.199 INFO:teuthology.orchestra.run.smithi141.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [25.2 kB] 2023-01-15T14:15:45.199 INFO:teuthology.orchestra.run.smithi141.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,408 B] 2023-01-15T14:15:45.199 INFO:teuthology.orchestra.run.smithi141.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [604 B] 2023-01-15T14:15:45.199 INFO:teuthology.orchestra.run.smithi141.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2023-01-15T14:15:45.199 INFO:teuthology.orchestra.run.smithi141.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [24.9 kB] 2023-01-15T14:15:45.199 INFO:teuthology.orchestra.run.smithi141.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2023-01-15T14:15:47.813 INFO:teuthology.orchestra.run.smithi141.stdout:Fetched 13.7 MB in 3s (5,115 kB/s) 2023-01-15T14:15:48.050 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 13.7 MB in 3s (5,312 kB/s) 2023-01-15T14:15:49.185 INFO:teuthology.orchestra.run.smithi141.stdout:Reading package lists... 2023-01-15T14:15:49.210 DEBUG:teuthology.orchestra.run.smithi141:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-15T14:15:49.281 INFO:teuthology.orchestra.run.smithi141.stdout:Reading package lists... 2023-01-15T14:15:49.466 INFO:teuthology.orchestra.run.smithi141.stdout:Building dependency tree... 2023-01-15T14:15:49.466 INFO:teuthology.orchestra.run.smithi141.stdout:Reading state information... 2023-01-15T14:15:49.476 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-01-15T14:15:49.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-15T14:15:49.576 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-01-15T14:15:49.631 INFO:teuthology.orchestra.run.smithi141.stdout:The following additional packages will be installed: 2023-01-15T14:15:49.631 INFO:teuthology.orchestra.run.smithi141.stdout: linux-generic linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-15T14:15:49.632 INFO:teuthology.orchestra.run.smithi141.stdout: linux-headers-generic linux-image-5.4.0-137-generic 2023-01-15T14:15:49.632 INFO:teuthology.orchestra.run.smithi141.stdout: linux-modules-5.4.0-137-generic linux-modules-extra-5.4.0-137-generic 2023-01-15T14:15:49.632 INFO:teuthology.orchestra.run.smithi141.stdout:Suggested packages: 2023-01-15T14:15:49.633 INFO:teuthology.orchestra.run.smithi141.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-01-15T14:15:49.633 INFO:teuthology.orchestra.run.smithi141.stdout:Recommended packages: 2023-01-15T14:15:49.633 INFO:teuthology.orchestra.run.smithi141.stdout: thermald 2023-01-15T14:15:49.704 INFO:teuthology.orchestra.run.smithi141.stdout:The following NEW packages will be installed: 2023-01-15T14:15:49.705 INFO:teuthology.orchestra.run.smithi141.stdout: linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-15T14:15:49.705 INFO:teuthology.orchestra.run.smithi141.stdout: linux-image-5.4.0-137-generic linux-modules-5.4.0-137-generic 2023-01-15T14:15:49.705 INFO:teuthology.orchestra.run.smithi141.stdout: linux-modules-extra-5.4.0-137-generic 2023-01-15T14:15:49.706 INFO:teuthology.orchestra.run.smithi141.stdout:The following packages will be upgraded: 2023-01-15T14:15:49.706 INFO:teuthology.orchestra.run.smithi141.stdout: linux-generic linux-headers-generic linux-image-generic 2023-01-15T14:15:49.761 INFO:teuthology.orchestra.run.smithi090.stdout:Building dependency tree... 2023-01-15T14:15:49.762 INFO:teuthology.orchestra.run.smithi090.stdout:Reading state information... 2023-01-15T14:15:49.884 INFO:teuthology.orchestra.run.smithi141.stdout:3 upgraded, 5 newly installed, 0 to remove and 116 not upgraded. 2023-01-15T14:15:49.884 INFO:teuthology.orchestra.run.smithi141.stdout:Need to get 77.1 MB of archives. 2023-01-15T14:15:49.885 INFO:teuthology.orchestra.run.smithi141.stdout:After this operation, 380 MB of additional disk space will be used. 2023-01-15T14:15:49.885 INFO:teuthology.orchestra.run.smithi141.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-137-generic amd64 5.4.0-137.154 [15.0 MB] 2023-01-15T14:15:49.921 INFO:teuthology.orchestra.run.smithi090.stdout:The following additional packages will be installed: 2023-01-15T14:15:49.922 INFO:teuthology.orchestra.run.smithi090.stdout: linux-generic linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-15T14:15:49.922 INFO:teuthology.orchestra.run.smithi090.stdout: linux-headers-generic linux-image-5.4.0-137-generic 2023-01-15T14:15:49.922 INFO:teuthology.orchestra.run.smithi090.stdout: linux-modules-5.4.0-137-generic linux-modules-extra-5.4.0-137-generic 2023-01-15T14:15:49.922 INFO:teuthology.orchestra.run.smithi090.stdout:Suggested packages: 2023-01-15T14:15:49.923 INFO:teuthology.orchestra.run.smithi090.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2023-01-15T14:15:49.923 INFO:teuthology.orchestra.run.smithi090.stdout:Recommended packages: 2023-01-15T14:15:49.923 INFO:teuthology.orchestra.run.smithi090.stdout: thermald 2023-01-15T14:15:49.980 INFO:teuthology.orchestra.run.smithi090.stdout:The following NEW packages will be installed: 2023-01-15T14:15:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: linux-headers-5.4.0-137 linux-headers-5.4.0-137-generic 2023-01-15T14:15:49.980 INFO:teuthology.orchestra.run.smithi090.stdout: linux-image-5.4.0-137-generic linux-modules-5.4.0-137-generic 2023-01-15T14:15:49.981 INFO:teuthology.orchestra.run.smithi090.stdout: linux-modules-extra-5.4.0-137-generic 2023-01-15T14:15:49.981 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages will be upgraded: 2023-01-15T14:15:49.982 INFO:teuthology.orchestra.run.smithi090.stdout: linux-generic linux-headers-generic linux-image-generic 2023-01-15T14:15:50.159 INFO:teuthology.orchestra.run.smithi090.stdout:3 upgraded, 5 newly installed, 0 to remove and 116 not upgraded. 2023-01-15T14:15:50.159 INFO:teuthology.orchestra.run.smithi090.stdout:Need to get 77.1 MB of archives. 2023-01-15T14:15:50.159 INFO:teuthology.orchestra.run.smithi090.stdout:After this operation, 380 MB of additional disk space will be used. 2023-01-15T14:15:50.159 INFO:teuthology.orchestra.run.smithi090.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-137-generic amd64 5.4.0-137.154 [15.0 MB] 2023-01-15T14:15:51.042 INFO:teuthology.orchestra.run.smithi141.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-137-generic amd64 5.4.0-137.154 [10.5 MB] 2023-01-15T14:15:51.261 INFO:teuthology.orchestra.run.smithi090.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-137-generic amd64 5.4.0-137.154 [10.5 MB] 2023-01-15T14:15:51.306 INFO:teuthology.orchestra.run.smithi141.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-137-generic amd64 5.4.0-137.154 [39.2 MB] 2023-01-15T14:15:51.525 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-137-generic amd64 5.4.0-137.154 [39.2 MB] 2023-01-15T14:15:52.549 INFO:teuthology.orchestra.run.smithi141.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.137.135 [1,896 B] 2023-01-15T14:15:52.549 INFO:teuthology.orchestra.run.smithi141.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.137.135 [2,528 B] 2023-01-15T14:15:52.550 INFO:teuthology.orchestra.run.smithi141.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137 all 5.4.0-137.154 [11.0 MB] 2023-01-15T14:15:52.748 INFO:teuthology.orchestra.run.smithi090.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.137.135 [1,896 B] 2023-01-15T14:15:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.137.135 [2,528 B] 2023-01-15T14:15:52.749 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137 all 5.4.0-137.154 [11.0 MB] 2023-01-15T14:15:53.023 INFO:teuthology.orchestra.run.smithi141.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137-generic amd64 5.4.0-137.154 [1,400 kB] 2023-01-15T14:15:53.085 INFO:teuthology.orchestra.run.smithi141.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.137.135 [2,392 B] 2023-01-15T14:15:53.193 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-137-generic amd64 5.4.0-137.154 [1,400 kB] 2023-01-15T14:15:53.258 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.137.135 [2,392 B] 2023-01-15T14:15:53.809 INFO:teuthology.orchestra.run.smithi141.stdout:Fetched 77.1 MB in 3s (22.9 MB/s) 2023-01-15T14:15:53.998 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 77.1 MB in 3s (23.6 MB/s) 2023-01-15T14:15:54.012 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package linux-modules-5.4.0-137-generic. 2023-01-15T14:15:54.215 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-modules-5.4.0-137-generic. 2023-01-15T14:15:55.754 INFO:teuthology.orchestra.run.smithi141.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-01-15T14:15:55.759 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../0-linux-modules-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-15T14:15:55.859 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:15:56.590 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2023-01-15T14:15:56.595 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../0-linux-modules-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-15T14:15:56.650 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:15:57.762 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package linux-image-5.4.0-137-generic. 2023-01-15T14:15:57.776 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../1-linux-image-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-15T14:15:57.905 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:15:58.697 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-image-5.4.0-137-generic. 2023-01-15T14:15:58.713 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../1-linux-image-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-15T14:15:58.822 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:15:59.007 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-137-generic. 2023-01-15T14:15:59.020 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-15T14:15:59.065 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:15:59.942 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-137-generic. 2023-01-15T14:15:59.956 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-15T14:16:00.000 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:03.794 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../3-linux-generic_5.4.0.137.135_amd64.deb ... 2023-01-15T14:16:03.912 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking linux-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-15T14:16:04.247 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.137.135_amd64.deb ... 2023-01-15T14:16:04.374 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking linux-image-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-15T14:16:04.727 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package linux-headers-5.4.0-137. 2023-01-15T14:16:04.741 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../5-linux-headers-5.4.0-137_5.4.0-137.154_all.deb ... 2023-01-15T14:16:04.777 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-15T14:16:05.449 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../3-linux-generic_5.4.0.137.135_amd64.deb ... 2023-01-15T14:16:05.575 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-15T14:16:05.921 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.137.135_amd64.deb ... 2023-01-15T14:16:06.047 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-image-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-15T14:16:06.384 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-headers-5.4.0-137. 2023-01-15T14:16:06.401 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../5-linux-headers-5.4.0-137_5.4.0-137.154_all.deb ... 2023-01-15T14:16:06.442 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-15T14:16:10.728 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package linux-headers-5.4.0-137-generic. 2023-01-15T14:16:10.748 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../6-linux-headers-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-15T14:16:10.827 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:13.153 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package linux-headers-5.4.0-137-generic. 2023-01-15T14:16:13.174 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../6-linux-headers-5.4.0-137-generic_5.4.0-137.154_amd64.deb ... 2023-01-15T14:16:13.211 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:13.753 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.137.135_amd64.deb ... 2023-01-15T14:16:13.875 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking linux-headers-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-15T14:16:14.210 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-15T14:16:14.346 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:15.565 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:15.683 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:15.760 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.137.135_amd64.deb ... 2023-01-15T14:16:15.887 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking linux-headers-generic (5.4.0.137.135) over (5.4.0.124.125) ... 2023-01-15T14:16:16.265 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-headers-5.4.0-137 (5.4.0-137.154) ... 2023-01-15T14:16:16.369 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-modules-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:16.829 INFO:teuthology.orchestra.run.smithi141.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-137-generic 2023-01-15T14:16:16.830 INFO:teuthology.orchestra.run.smithi141.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-137-generic 2023-01-15T14:16:16.985 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:17.565 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-headers-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:17.717 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:18.170 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up linux-image-generic (5.4.0.137.135) ... 2023-01-15T14:16:18.296 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up linux-headers-generic (5.4.0.137.135) ... 2023-01-15T14:16:19.159 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up linux-generic (5.4.0.137.135) ... 2023-01-15T14:16:19.307 INFO:teuthology.orchestra.run.smithi141.stdout:Processing triggers for linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:19.370 INFO:teuthology.orchestra.run.smithi141.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-01-15T14:16:19.371 INFO:teuthology.orchestra.run.smithi141.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-137-generic 2023-01-15T14:16:19.916 INFO:teuthology.orchestra.run.smithi090.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-137-generic 2023-01-15T14:16:19.916 INFO:teuthology.orchestra.run.smithi090.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-137-generic 2023-01-15T14:16:21.062 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-modules-extra-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:22.230 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-image-generic (5.4.0.137.135) ... 2023-01-15T14:16:22.373 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-headers-generic (5.4.0.137.135) ... 2023-01-15T14:16:22.580 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up linux-generic (5.4.0.137.135) ... 2023-01-15T14:16:22.619 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for linux-image-5.4.0-137-generic (5.4.0-137.154) ... 2023-01-15T14:16:23.610 INFO:teuthology.orchestra.run.smithi090.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-01-15T14:16:23.610 INFO:teuthology.orchestra.run.smithi090.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-137-generic 2023-01-15T14:16:41.784 INFO:teuthology.orchestra.run.smithi141.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-01-15T14:16:41.784 INFO:teuthology.orchestra.run.smithi141.stdout:Sourcing file `/etc/default/grub' 2023-01-15T14:16:41.798 INFO:teuthology.orchestra.run.smithi141.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-15T14:16:41.812 INFO:teuthology.orchestra.run.smithi141.stdout:Generating grub configuration file ... 2023-01-15T14:16:42.114 INFO:teuthology.orchestra.run.smithi141.stdout:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-15T14:16:42.140 INFO:teuthology.orchestra.run.smithi141.stdout:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-15T14:16:42.523 INFO:teuthology.orchestra.run.smithi141.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-15T14:16:42.535 INFO:teuthology.orchestra.run.smithi141.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-15T14:16:44.858 INFO:teuthology.orchestra.run.smithi090.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-01-15T14:16:44.858 INFO:teuthology.orchestra.run.smithi090.stdout:Sourcing file `/etc/default/grub' 2023-01-15T14:16:44.876 INFO:teuthology.orchestra.run.smithi090.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-15T14:16:44.905 INFO:teuthology.orchestra.run.smithi090.stdout:Generating grub configuration file ... 2023-01-15T14:16:45.054 INFO:teuthology.orchestra.run.smithi141.stdout:done 2023-01-15T14:16:45.347 INFO:teuthology.orchestra.run.smithi090.stdout:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-15T14:16:45.387 INFO:teuthology.orchestra.run.smithi090.stdout:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-15T14:16:45.453 DEBUG:teuthology.orchestra.run.smithi141:> dpkg -s linux-image-generic 2023-01-15T14:16:45.496 INFO:teuthology.orchestra.run.smithi141.stdout:Package: linux-image-generic 2023-01-15T14:16:45.496 INFO:teuthology.orchestra.run.smithi141.stdout:Status: install ok installed 2023-01-15T14:16:45.496 INFO:teuthology.orchestra.run.smithi141.stdout:Priority: optional 2023-01-15T14:16:45.497 INFO:teuthology.orchestra.run.smithi141.stdout:Section: kernel 2023-01-15T14:16:45.497 INFO:teuthology.orchestra.run.smithi141.stdout:Installed-Size: 19 2023-01-15T14:16:45.497 INFO:teuthology.orchestra.run.smithi141.stdout:Maintainer: Ubuntu Kernel Team 2023-01-15T14:16:45.497 INFO:teuthology.orchestra.run.smithi141.stdout:Architecture: amd64 2023-01-15T14:16:45.497 INFO:teuthology.orchestra.run.smithi141.stdout:Source: linux-meta 2023-01-15T14:16:45.497 INFO:teuthology.orchestra.run.smithi141.stdout:Version: 5.4.0.137.135 2023-01-15T14:16:45.498 INFO:teuthology.orchestra.run.smithi141.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-15T14:16:45.498 INFO:teuthology.orchestra.run.smithi141.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-15T14:16:45.498 INFO:teuthology.orchestra.run.smithi141.stdout:Recommends: thermald 2023-01-15T14:16:45.498 INFO:teuthology.orchestra.run.smithi141.stdout:Description: Generic Linux kernel image 2023-01-15T14:16:45.498 INFO:teuthology.orchestra.run.smithi141.stdout: This package will always depend on the latest generic kernel image 2023-01-15T14:16:45.498 INFO:teuthology.orchestra.run.smithi141.stdout: available. 2023-01-15T14:16:45.499 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-137-generic 2023-01-15T14:16:45.499 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-15T14:16:45.499 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-15T14:16:45.499 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-01-15T14:16:45.500 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi141.front.sepia.ceph.com, path=None, version=distro) 2023-01-15T14:16:45.500 DEBUG:teuthology.orchestra.run.smithi141:> sudo apt-get clean 2023-01-15T14:16:45.612 DEBUG:teuthology.orchestra.run.smithi141:> sudo apt-get update 2023-01-15T14:16:45.891 INFO:teuthology.orchestra.run.smithi141.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-15T14:16:45.891 INFO:teuthology.orchestra.run.smithi141.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-15T14:16:45.932 INFO:teuthology.orchestra.run.smithi090.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-15T14:16:45.944 INFO:teuthology.orchestra.run.smithi090.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-15T14:16:45.959 INFO:teuthology.orchestra.run.smithi141.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-15T14:16:46.048 INFO:teuthology.orchestra.run.smithi141.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-15T14:16:47.536 INFO:teuthology.orchestra.run.smithi141.stdout:Reading package lists... 2023-01-15T14:16:47.560 DEBUG:teuthology.orchestra.run.smithi141:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-15T14:16:47.628 INFO:teuthology.orchestra.run.smithi141.stdout:Reading package lists... 2023-01-15T14:16:47.792 INFO:teuthology.orchestra.run.smithi141.stdout:Building dependency tree... 2023-01-15T14:16:47.793 INFO:teuthology.orchestra.run.smithi141.stdout:Reading state information... 2023-01-15T14:16:47.980 INFO:teuthology.orchestra.run.smithi141.stdout:linux-image-generic is already the newest version (5.4.0.137.135). 2023-01-15T14:16:47.980 INFO:teuthology.orchestra.run.smithi141.stdout:0 upgraded, 0 newly installed, 0 to remove and 116 not upgraded. 2023-01-15T14:16:47.982 DEBUG:teuthology.orchestra.run.smithi141:> dpkg -s linux-image-generic 2023-01-15T14:16:48.003 INFO:teuthology.orchestra.run.smithi141.stdout:Package: linux-image-generic 2023-01-15T14:16:48.003 INFO:teuthology.orchestra.run.smithi141.stdout:Status: install ok installed 2023-01-15T14:16:48.003 INFO:teuthology.orchestra.run.smithi141.stdout:Priority: optional 2023-01-15T14:16:48.003 INFO:teuthology.orchestra.run.smithi141.stdout:Section: kernel 2023-01-15T14:16:48.003 INFO:teuthology.orchestra.run.smithi141.stdout:Installed-Size: 19 2023-01-15T14:16:48.003 INFO:teuthology.orchestra.run.smithi141.stdout:Maintainer: Ubuntu Kernel Team 2023-01-15T14:16:48.004 INFO:teuthology.orchestra.run.smithi141.stdout:Architecture: amd64 2023-01-15T14:16:48.004 INFO:teuthology.orchestra.run.smithi141.stdout:Source: linux-meta 2023-01-15T14:16:48.004 INFO:teuthology.orchestra.run.smithi141.stdout:Version: 5.4.0.137.135 2023-01-15T14:16:48.004 INFO:teuthology.orchestra.run.smithi141.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-15T14:16:48.004 INFO:teuthology.orchestra.run.smithi141.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-15T14:16:48.004 INFO:teuthology.orchestra.run.smithi141.stdout:Recommends: thermald 2023-01-15T14:16:48.004 INFO:teuthology.orchestra.run.smithi141.stdout:Description: Generic Linux kernel image 2023-01-15T14:16:48.004 INFO:teuthology.orchestra.run.smithi141.stdout: This package will always depend on the latest generic kernel image 2023-01-15T14:16:48.004 INFO:teuthology.orchestra.run.smithi141.stdout: available. 2023-01-15T14:16:48.005 DEBUG:teuthology.orchestra.run.smithi141:> mktemp 2023-01-15T14:16:48.053 INFO:teuthology.orchestra.run.smithi141.stdout:/tmp/tmp.hk3lRsPSju 2023-01-15T14:16:48.054 DEBUG:teuthology.orchestra.run.smithi141:> sudo cp /boot/grub/grub.cfg /tmp/tmp.hk3lRsPSju 2023-01-15T14:16:48.076 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 0666 /tmp/tmp.hk3lRsPSju 2023-01-15T14:16:48.121 DEBUG:teuthology.orchestra.remote:smithi141:/tmp/tmp.hk3lRsPSju is 9KB 2023-01-15T14:16:48.135 DEBUG:teuthology.orchestra.run.smithi141:> rm -fr /tmp/tmp.hk3lRsPSju 2023-01-15T14:16:48.141 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-01-15T14:16:48.202 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:16:48.202 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-01-15T14:16:48.202 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-01-15T14:16:48.233 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-137-generic 2023-01-15T14:16:48.233 DEBUG:teuthology.orchestra.run.smithi141:> sudo update-grub 2023-01-15T14:16:48.681 INFO:teuthology.orchestra.run.smithi090.stdout:done 2023-01-15T14:16:49.172 DEBUG:teuthology.orchestra.run.smithi090:> dpkg -s linux-image-generic 2023-01-15T14:16:49.229 INFO:teuthology.orchestra.run.smithi090.stdout:Package: linux-image-generic 2023-01-15T14:16:49.229 INFO:teuthology.orchestra.run.smithi090.stdout:Status: install ok installed 2023-01-15T14:16:49.229 INFO:teuthology.orchestra.run.smithi090.stdout:Priority: optional 2023-01-15T14:16:49.230 INFO:teuthology.orchestra.run.smithi090.stdout:Section: kernel 2023-01-15T14:16:49.230 INFO:teuthology.orchestra.run.smithi090.stdout:Installed-Size: 19 2023-01-15T14:16:49.230 INFO:teuthology.orchestra.run.smithi090.stdout:Maintainer: Ubuntu Kernel Team 2023-01-15T14:16:49.230 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: amd64 2023-01-15T14:16:49.230 INFO:teuthology.orchestra.run.smithi090.stdout:Source: linux-meta 2023-01-15T14:16:49.230 INFO:teuthology.orchestra.run.smithi090.stdout:Version: 5.4.0.137.135 2023-01-15T14:16:49.230 INFO:teuthology.orchestra.run.smithi090.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-15T14:16:49.231 INFO:teuthology.orchestra.run.smithi090.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-15T14:16:49.231 INFO:teuthology.orchestra.run.smithi090.stdout:Recommends: thermald 2023-01-15T14:16:49.231 INFO:teuthology.orchestra.run.smithi090.stdout:Description: Generic Linux kernel image 2023-01-15T14:16:49.231 INFO:teuthology.orchestra.run.smithi090.stdout: This package will always depend on the latest generic kernel image 2023-01-15T14:16:49.231 INFO:teuthology.orchestra.run.smithi090.stdout: available. 2023-01-15T14:16:49.231 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-137-generic 2023-01-15T14:16:49.231 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-01-15T14:16:49.232 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-01-15T14:16:49.232 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-01-15T14:16:49.232 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi090.front.sepia.ceph.com, path=None, version=distro) 2023-01-15T14:16:49.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get clean 2023-01-15T14:16:49.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get update 2023-01-15T14:16:49.481 INFO:teuthology.orchestra.run.smithi141.stderr:Sourcing file `/etc/default/grub' 2023-01-15T14:16:49.488 INFO:teuthology.orchestra.run.smithi141.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-15T14:16:49.492 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-15T14:16:49.501 INFO:teuthology.orchestra.run.smithi141.stderr:Generating grub configuration file ... 2023-01-15T14:16:49.741 INFO:teuthology.orchestra.run.smithi141.stderr:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-15T14:16:49.753 INFO:teuthology.orchestra.run.smithi141.stderr:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-15T14:16:49.782 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-15T14:16:49.862 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-15T14:16:49.942 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-15T14:16:50.132 INFO:teuthology.orchestra.run.smithi141.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-15T14:16:50.144 INFO:teuthology.orchestra.run.smithi141.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-15T14:16:51.289 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-01-15T14:16:51.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-01-15T14:16:51.380 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-01-15T14:16:51.561 INFO:teuthology.orchestra.run.smithi141.stderr:done 2023-01-15T14:16:51.563 DEBUG:teuthology.orchestra.run.smithi141:> sudo shutdown -r now 2023-01-15T14:16:51.579 INFO:teuthology.orchestra.run.smithi090.stdout:Building dependency tree... 2023-01-15T14:16:51.580 INFO:teuthology.orchestra.run.smithi090.stdout:Reading state information... 2023-01-15T14:16:51.777 INFO:teuthology.orchestra.run.smithi090.stdout:linux-image-generic is already the newest version (5.4.0.137.135). 2023-01-15T14:16:51.777 INFO:teuthology.orchestra.run.smithi090.stdout:0 upgraded, 0 newly installed, 0 to remove and 116 not upgraded. 2023-01-15T14:16:51.779 DEBUG:teuthology.orchestra.run.smithi090:> dpkg -s linux-image-generic 2023-01-15T14:16:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:Package: linux-image-generic 2023-01-15T14:16:51.803 INFO:teuthology.orchestra.run.smithi090.stdout:Status: install ok installed 2023-01-15T14:16:51.803 INFO:teuthology.orchestra.run.smithi090.stdout:Priority: optional 2023-01-15T14:16:51.803 INFO:teuthology.orchestra.run.smithi090.stdout:Section: kernel 2023-01-15T14:16:51.803 INFO:teuthology.orchestra.run.smithi090.stdout:Installed-Size: 19 2023-01-15T14:16:51.804 INFO:teuthology.orchestra.run.smithi090.stdout:Maintainer: Ubuntu Kernel Team 2023-01-15T14:16:51.804 INFO:teuthology.orchestra.run.smithi090.stdout:Architecture: amd64 2023-01-15T14:16:51.804 INFO:teuthology.orchestra.run.smithi090.stdout:Source: linux-meta 2023-01-15T14:16:51.804 INFO:teuthology.orchestra.run.smithi090.stdout:Version: 5.4.0.137.135 2023-01-15T14:16:51.804 INFO:teuthology.orchestra.run.smithi090.stdout:Provides: virtualbox-guest-modules (= 6.1.38-dfsg-3~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2023-01-15T14:16:51.804 INFO:teuthology.orchestra.run.smithi090.stdout:Depends: linux-image-5.4.0-137-generic, linux-modules-extra-5.4.0-137-generic, linux-firmware, intel-microcode, amd64-microcode 2023-01-15T14:16:51.804 INFO:teuthology.orchestra.run.smithi090.stdout:Recommends: thermald 2023-01-15T14:16:51.804 INFO:teuthology.orchestra.run.smithi090.stdout:Description: Generic Linux kernel image 2023-01-15T14:16:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: This package will always depend on the latest generic kernel image 2023-01-15T14:16:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: available. 2023-01-15T14:16:51.805 DEBUG:teuthology.orchestra.run.smithi090:> mktemp 2023-01-15T14:16:51.809 INFO:teuthology.orchestra.run.smithi090.stdout:/tmp/tmp.TcwY3z36Hy 2023-01-15T14:16:51.810 DEBUG:teuthology.orchestra.run.smithi090:> sudo cp /boot/grub/grub.cfg /tmp/tmp.TcwY3z36Hy 2023-01-15T14:16:51.824 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0666 /tmp/tmp.TcwY3z36Hy 2023-01-15T14:16:51.908 DEBUG:teuthology.orchestra.remote:smithi090:/tmp/tmp.TcwY3z36Hy is 9KB 2023-01-15T14:16:51.923 DEBUG:teuthology.orchestra.run.smithi090:> rm -fr /tmp/tmp.TcwY3z36Hy 2023-01-15T14:16:51.929 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-01-15T14:16:51.985 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:16:51.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-01-15T14:16:51.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-01-15T14:16:52.017 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-137-generic 2023-01-15T14:16:52.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo update-grub 2023-01-15T14:16:53.542 INFO:teuthology.orchestra.run.smithi090.stderr:Sourcing file `/etc/default/grub' 2023-01-15T14:16:53.548 INFO:teuthology.orchestra.run.smithi090.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-01-15T14:16:53.563 INFO:teuthology.orchestra.run.smithi090.stderr:Generating grub configuration file ... 2023-01-15T14:16:54.021 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-5.4.0-137-generic 2023-01-15T14:16:54.033 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-5.4.0-137-generic 2023-01-15T14:16:54.617 INFO:teuthology.orchestra.run.smithi090.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2023-01-15T14:16:54.628 INFO:teuthology.orchestra.run.smithi090.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2023-01-15T14:16:56.191 INFO:teuthology.orchestra.run.smithi090.stderr:done 2023-01-15T14:16:56.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo shutdown -r now 2023-01-15T14:17:21.588 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-01-15T14:17:21.643 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-15T14:17:21.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-15T14:17:26.200 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-01-15T14:17:26.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-15T14:17:26.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-15T14:17:40.005 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.141 2023-01-15T14:17:44.612 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.90 2023-01-15T14:18:11.004 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-15T14:18:11.005 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-15T14:18:14.616 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-15T14:18:14.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-15T14:18:26.693 DEBUG:teuthology.orchestra.run.smithi141:> true 2023-01-15T14:18:27.908 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-15T14:18:27.908 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-137-generic"... 2023-01-15T14:18:27.908 DEBUG:teuthology.orchestra.run.smithi141:> uname -r 2023-01-15T14:18:27.915 INFO:teuthology.orchestra.run.smithi141.stdout:5.4.0-137-generic 2023-01-15T14:18:27.915 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-137-generic vs 5.4.0-137-generic 2023-01-15T14:18:27.916 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-15T14:18:27.916 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-01-15T14:18:28.917 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-01-15T14:18:28.966 DEBUG:teuthology.orchestra.run.smithi141:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-15T14:18:29.110 INFO:teuthology.orchestra.run.smithi141.stdout:ttyS1 2023-01-15T14:18:29.139 DEBUG:teuthology.parallel:result is None 2023-01-15T14:19:14.619 DEBUG:teuthology.orchestra.remote:timed out 2023-01-15T14:19:44.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-15T14:19:44.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-15T14:19:44.816 DEBUG:teuthology.orchestra.run.smithi090:> true 2023-01-15T14:19:45.577 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-15T14:19:45.577 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-137-generic"... 2023-01-15T14:19:45.578 DEBUG:teuthology.orchestra.run.smithi090:> uname -r 2023-01-15T14:19:45.584 INFO:teuthology.orchestra.run.smithi090.stdout:5.4.0-137-generic 2023-01-15T14:19:45.584 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-137-generic vs 5.4.0-137-generic 2023-01-15T14:19:45.584 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-01-15T14:19:45.584 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-01-15T14:19:46.585 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-01-15T14:19:46.586 DEBUG:teuthology.orchestra.run.smithi090:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-15T14:19:46.686 INFO:teuthology.orchestra.run.smithi090.stdout:ttyS1 2023-01-15T14:19:46.715 DEBUG:teuthology.parallel:result is None 2023-01-15T14:19:46.715 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-15T14:19:46.727 INFO:teuthology.task.internal:Creating test directory... 2023-01-15T14:19:46.727 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-15T14:19:46.729 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-15T14:19:46.737 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-15T14:19:46.759 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-15T14:19:46.766 INFO:teuthology.task.internal:Creating archive directory... 2023-01-15T14:19:46.766 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-15T14:19:46.780 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-15T14:19:46.809 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-15T14:19:46.817 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-15T14:19:46.817 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-15T14:19:46.820 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-15T14:19:46.839 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-15T14:19:46.840 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-15T14:19:46.849 INFO:teuthology.orchestra.run.smithi090.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-15T14:19:46.851 INFO:teuthology.orchestra.run.smithi141.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-15T14:19:46.853 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-15T14:19:46.869 INFO:teuthology.task.internal:Configuring sudo... 2023-01-15T14:19:46.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-15T14:19:46.895 DEBUG:teuthology.orchestra.run.smithi141:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-15T14:19:46.914 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-15T14:19:46.926 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-15T14:19:46.926 DEBUG:teuthology.orchestra.run.smithi090:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-15T14:19:46.929 DEBUG:teuthology.orchestra.run.smithi141:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-15T14:19:46.936 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-15T14:19:46.942 DEBUG:teuthology.orchestra.run.smithi090:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-15T14:19:46.991 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:19:46.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-15T14:19:47.016 DEBUG:teuthology.orchestra.run.smithi141:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-15T14:19:47.028 DEBUG:teuthology.orchestra.run.smithi141:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-15T14:19:47.075 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:19:47.076 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-15T14:19:47.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo service rsyslog restart 2023-01-15T14:19:47.104 DEBUG:teuthology.orchestra.run.smithi141:> sudo service rsyslog restart 2023-01-15T14:19:47.155 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-15T14:19:47.178 INFO:teuthology.task.internal:Starting timer... 2023-01-15T14:19:47.179 INFO:teuthology.run_tasks:Running task pcp... 2023-01-15T14:19:47.191 INFO:teuthology.run_tasks:Running task selinux... 2023-01-15T14:19:47.206 DEBUG:teuthology.task.selinux:Excluding smithi090: OS 'ubuntu' does not support SELinux 2023-01-15T14:19:47.206 DEBUG:teuthology.task.selinux:Excluding smithi141: OS 'ubuntu' does not support SELinux 2023-01-15T14:19:47.207 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-15T14:19:47.207 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-01-15T14:19:47.207 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-15T14:19:47.207 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-15T14:19:47.219 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-01-15T14:19:47.219 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-15T14:19:47.240 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-15T14:19:47.242 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi090.front.sepia.ceph.com,smithi141.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-15T14:24:43.515 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi090.front.sepia.ceph.com'), Remote(name='ubuntu@smithi141.front.sepia.ceph.com')] 2023-01-15T14:24:43.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2023-01-15T14:24:43.518 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi090.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-15T14:24:43.568 DEBUG:teuthology.orchestra.run.smithi090:> true 2023-01-15T14:24:43.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2023-01-15T14:24:43.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi141.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-15T14:24:43.739 DEBUG:teuthology.orchestra.run.smithi141:> true 2023-01-15T14:24:43.859 INFO:teuthology.run_tasks:Running task clock... 2023-01-15T14:24:43.869 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-01-15T14:24:43.869 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-15T14:24:43.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-15T14:24:43.872 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-01-15T14:24:43.872 DEBUG:teuthology.orchestra.run.smithi141:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-01-15T14:24:43.906 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-01-15T14:24:43.906 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: Command line: ntpd -gq 2023-01-15T14:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: ntpd 4.2.8p12@1.3728-o (1): Starting 2023-01-15T14:24:43.907 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: Command line: ntpd -gq 2023-01-15T14:24:43.907 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: proto: precision = 0.116 usec (-23) 2023-01-15T14:24:43.908 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: proto: precision = 0.093 usec (-23) 2023-01-15T14:24:43.908 INFO:teuthology.orchestra.run.smithi141.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-15T14:24:43.909 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-15T14:24:43.909 INFO:teuthology.orchestra.run.smithi141.stderr:restrict ::: KOD does nothing without LIMITED. 2023-01-15T14:24:43.910 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: restrict ::: KOD does nothing without LIMITED. 2023-01-15T14:24:43.910 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: Listen and drop on 0 v6wildcard [::]:123 2023-01-15T14:24:43.910 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-01-15T14:24:43.910 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: Listen normally on 2 lo 127.0.0.1:123 2023-01-15T14:24:43.910 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: Listen normally on 3 enp3s0f1 172.21.15.141:123 2023-01-15T14:24:43.910 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: Listen normally on 4 lo [::1]:123 2023-01-15T14:24:43.910 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:726d%5]:123 2023-01-15T14:24:43.911 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:43 ntpd[19208]: Listening on routing socket on fd #22 for interface updates 2023-01-15T14:24:43.911 INFO:teuthology.orchestra.run.smithi090.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-15T14:24:43.911 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-01-15T14:24:43.912 INFO:teuthology.orchestra.run.smithi090.stderr:restrict ::: KOD does nothing without LIMITED. 2023-01-15T14:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: restrict ::: KOD does nothing without LIMITED. 2023-01-15T14:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: Listen and drop on 0 v6wildcard [::]:123 2023-01-15T14:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-01-15T14:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: Listen normally on 2 lo 127.0.0.1:123 2023-01-15T14:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: Listen normally on 3 enp3s0f1 172.21.15.90:123 2023-01-15T14:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: Listen normally on 4 lo [::1]:123 2023-01-15T14:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d767%5]:123 2023-01-15T14:24:43.913 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:43 ntpd[19219]: Listening on routing socket on fd #22 for interface updates 2023-01-15T14:24:44.907 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:44 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:44.908 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:44 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:44.908 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:44 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:44.908 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:44 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:44.908 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:44 ntpd[19208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:44.908 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:44 ntpd[19208]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:44.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:44 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:44.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:44 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:45.909 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:45 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:45.909 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:45 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:45.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:45 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:45.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:45 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:45.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:45 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:45.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:45 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:45.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:45 ntpd[19219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:45.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:45 ntpd[19219]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:46.907 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:46 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:46.908 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:46 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:46.908 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:46 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:46.908 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:46 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:46.908 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:46 ntpd[19208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:46.908 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:46 ntpd[19208]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:46.909 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:46.910 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:46 ntpd[19219]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:47 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:47 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:47 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:47 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:47 ntpd[19219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:47.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:47 ntpd[19219]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:48.909 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:48.909 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:48.909 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:48.909 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:48.911 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:48.912 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:48.912 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:48.912 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:48.912 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:48.912 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:48.912 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:48.912 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:48.913 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:48.913 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:48.913 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:48.913 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:48.913 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:48.913 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:48 ntpd[19208]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:48.916 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:48.916 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:48.916 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:48.916 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:48.918 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:48.918 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:48 ntpd[19219]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:49.907 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:49 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:49 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:49 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:49 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:49 ntpd[19219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:49.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:49 ntpd[19219]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:50.910 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:50.910 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:50.910 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:50.910 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:50.910 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:50.910 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:50.910 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:50.911 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:50.911 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:50.911 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:50.911 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:50.911 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:50.911 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:50.911 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:50.911 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:50.911 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:50.911 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:50.912 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:50.912 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:50.912 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:50.912 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:50.912 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:50 ntpd[19219]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:50.912 INFO:teuthology.orchestra.run.smithi141.stdout:15 Jan 14:24:50 ntpd[19208]: ntpd: time slew -0.000019 s 2023-01-15T14:24:50.912 INFO:teuthology.orchestra.run.smithi141.stdout:ntpd: time slew -0.000019s 2023-01-15T14:24:50.913 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-01-15T14:24:50.913 INFO:teuthology.orchestra.run.smithi141.stderr:15 Jan 14:24:50 ntpd[19208]: can't open /var/log/ntpstats/loopstats.20230115: Permission denied 2023-01-15T14:24:50.979 INFO:teuthology.orchestra.run.smithi141.stdout: remote refid st t when poll reach delay offset jitter 2023-01-15T14:24:50.980 INFO:teuthology.orchestra.run.smithi141.stdout:============================================================================== 2023-01-15T14:24:50.980 INFO:teuthology.orchestra.run.smithi141.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-15T14:24:50.980 INFO:teuthology.orchestra.run.smithi141.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-15T14:24:50.980 INFO:teuthology.orchestra.run.smithi141.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-15T14:24:50.980 INFO:teuthology.orchestra.run.smithi141.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-15T14:24:51.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:51 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:51.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:51 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:51.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:51 ntpd[19219]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-01-15T14:24:51.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:51 ntpd[19219]: can't open /var/log/ntpstats/rawstats.20230115: Permission denied 2023-01-15T14:24:51.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:51 ntpd[19219]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-01-15T14:24:51.908 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:51 ntpd[19219]: can't open /var/log/ntpstats/peerstats.20230115: Permission denied 2023-01-15T14:24:51.909 INFO:teuthology.orchestra.run.smithi090.stdout:15 Jan 14:24:51 ntpd[19219]: ntpd: time slew +0.001535 s 2023-01-15T14:24:51.909 INFO:teuthology.orchestra.run.smithi090.stdout:ntpd: time slew +0.001535s 2023-01-15T14:24:51.909 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:51 ntpd[19219]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-01-15T14:24:51.909 INFO:teuthology.orchestra.run.smithi090.stderr:15 Jan 14:24:51 ntpd[19219]: can't open /var/log/ntpstats/loopstats.20230115: Permission denied 2023-01-15T14:24:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: remote refid st t when poll reach delay offset jitter 2023-01-15T14:24:51.977 INFO:teuthology.orchestra.run.smithi090.stdout:============================================================================== 2023-01-15T14:24:51.977 INFO:teuthology.orchestra.run.smithi090.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-15T14:24:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-15T14:24:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-15T14:24:51.978 INFO:teuthology.orchestra.run.smithi090.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2023-01-15T14:24:51.979 INFO:teuthology.run_tasks:Running task install... 2023-01-15T14:24:51.993 DEBUG:teuthology.task.install:project ceph 2023-01-15T14:24:51.993 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a21b2f4fd40d883dac62c8cd315cbb54848ee067'}} 2023-01-15T14:24:51.994 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a21b2f4fd40d883dac62c8cd315cbb54848ee067'} 2023-01-15T14:24:51.994 INFO:teuthology.task.install:Using flavor: default 2023-01-15T14:24:52.003 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-01-15T14:24:52.003 INFO:teuthology.task.install:extra packages: [] 2023-01-15T14:24:52.004 DEBUG:teuthology.orchestra.run.smithi090:> sudo lsb_release -is 2023-01-15T14:24:52.071 INFO:teuthology.orchestra.run.smithi090.stdout:Ubuntu 2023-01-15T14:24:52.076 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-15T14:24:52.076 DEBUG:teuthology.orchestra.run.smithi141:> sudo lsb_release -is 2023-01-15T14:24:52.076 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-key list | grep Ceph 2023-01-15T14:24:52.128 INFO:teuthology.orchestra.run.smithi141.stdout:Ubuntu 2023-01-15T14:24:52.133 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-15T14:24:52.134 DEBUG:teuthology.orchestra.run.smithi141:> sudo apt-key list | grep Ceph 2023-01-15T14:24:52.173 INFO:teuthology.orchestra.run.smithi090.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-01-15T14:24:52.234 INFO:teuthology.orchestra.run.smithi090.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-01-15T14:24:52.234 INFO:teuthology.orchestra.run.smithi090.stdout:uid [ unknown] Ceph.com (release key) 2023-01-15T14:24:52.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo lsb_release -is 2023-01-15T14:24:52.236 INFO:teuthology.orchestra.run.smithi141.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2023-01-15T14:24:52.284 INFO:teuthology.orchestra.run.smithi090.stdout:Ubuntu 2023-01-15T14:24:52.290 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-15T14:24:52.290 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-01-15T14:24:52.291 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=a21b2f4fd40d883dac62c8cd315cbb54848ee067 2023-01-15T14:24:52.298 INFO:teuthology.orchestra.run.smithi141.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-01-15T14:24:52.298 INFO:teuthology.orchestra.run.smithi141.stdout:uid [ unknown] Ceph.com (release key) 2023-01-15T14:24:52.299 DEBUG:teuthology.orchestra.run.smithi141:> sudo lsb_release -is 2023-01-15T14:24:52.349 INFO:teuthology.orchestra.run.smithi141.stdout:Ubuntu 2023-01-15T14:24:52.355 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-15T14:24:52.355 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2023-01-15T14:24:52.355 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=a21b2f4fd40d883dac62c8cd315cbb54848ee067 2023-01-15T14:24:52.444 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default/ 2023-01-15T14:24:52.444 INFO:teuthology.task.install.deb:Package version is 18.0.0-1736-ga21b2f4f-1focal 2023-01-15T14:24:52.505 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default/ 2023-01-15T14:24:52.505 INFO:teuthology.task.install.deb:Package version is 18.0.0-1736-ga21b2f4f-1focal 2023-01-15T14:24:52.745 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:24:52.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-01-15T14:24:52.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo apt-get update 2023-01-15T14:24:52.801 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:24:52.801 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-01-15T14:24:52.818 DEBUG:teuthology.orchestra.run.smithi141:> sudo apt-get update 2023-01-15T14:24:53.077 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-15T14:24:53.077 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-15T14:24:53.129 INFO:teuthology.orchestra.run.smithi141.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2023-01-15T14:24:53.130 INFO:teuthology.orchestra.run.smithi141.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2023-01-15T14:24:53.145 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-15T14:24:53.200 INFO:teuthology.orchestra.run.smithi141.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2023-01-15T14:24:53.206 INFO:teuthology.orchestra.run.smithi090.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal InRelease 2023-01-15T14:24:53.235 INFO:teuthology.orchestra.run.smithi090.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-15T14:24:53.263 INFO:teuthology.orchestra.run.smithi141.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal InRelease 2023-01-15T14:24:53.279 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-01-15T14:24:53.282 INFO:teuthology.orchestra.run.smithi141.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2023-01-15T14:24:53.336 INFO:teuthology.orchestra.run.smithi141.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal Release [7,662 B] 2023-01-15T14:24:53.355 INFO:teuthology.orchestra.run.smithi090.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal Release.gpg 2023-01-15T14:24:53.412 INFO:teuthology.orchestra.run.smithi141.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal Release.gpg 2023-01-15T14:24:53.429 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-01-15T14:24:53.487 INFO:teuthology.orchestra.run.smithi141.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main i386 Packages [3,418 B] 2023-01-15T14:24:53.592 INFO:teuthology.orchestra.run.smithi141.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2023-01-15T14:24:53.607 INFO:teuthology.orchestra.run.smithi090.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main i386 Packages [3,418 B] 2023-01-15T14:24:53.733 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 28.5 kB in 1s (37.4 kB/s) 2023-01-15T14:24:53.780 INFO:teuthology.orchestra.run.smithi141.stdout:Fetched 28.5 kB in 1s (37.8 kB/s) 2023-01-15T14:24:54.667 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-01-15T14:24:54.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-1736-ga21b2f4f-1focal cephadm=18.0.0-1736-ga21b2f4f-1focal ceph-mds=18.0.0-1736-ga21b2f4f-1focal ceph-mgr=18.0.0-1736-ga21b2f4f-1focal ceph-common=18.0.0-1736-ga21b2f4f-1focal ceph-fuse=18.0.0-1736-ga21b2f4f-1focal ceph-test=18.0.0-1736-ga21b2f4f-1focal ceph-volume=18.0.0-1736-ga21b2f4f-1focal radosgw=18.0.0-1736-ga21b2f4f-1focal python3-rados=18.0.0-1736-ga21b2f4f-1focal python3-rgw=18.0.0-1736-ga21b2f4f-1focal python3-cephfs=18.0.0-1736-ga21b2f4f-1focal python3-rbd=18.0.0-1736-ga21b2f4f-1focal libcephfs2=18.0.0-1736-ga21b2f4f-1focal libcephfs-dev=18.0.0-1736-ga21b2f4f-1focal librados2=18.0.0-1736-ga21b2f4f-1focal librbd1=18.0.0-1736-ga21b2f4f-1focal rbd-fuse=18.0.0-1736-ga21b2f4f-1focal 2023-01-15T14:24:54.726 INFO:teuthology.orchestra.run.smithi141.stdout:Reading package lists... 2023-01-15T14:24:54.749 DEBUG:teuthology.orchestra.run.smithi141:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-1736-ga21b2f4f-1focal cephadm=18.0.0-1736-ga21b2f4f-1focal ceph-mds=18.0.0-1736-ga21b2f4f-1focal ceph-mgr=18.0.0-1736-ga21b2f4f-1focal ceph-common=18.0.0-1736-ga21b2f4f-1focal ceph-fuse=18.0.0-1736-ga21b2f4f-1focal ceph-test=18.0.0-1736-ga21b2f4f-1focal ceph-volume=18.0.0-1736-ga21b2f4f-1focal radosgw=18.0.0-1736-ga21b2f4f-1focal python3-rados=18.0.0-1736-ga21b2f4f-1focal python3-rgw=18.0.0-1736-ga21b2f4f-1focal python3-cephfs=18.0.0-1736-ga21b2f4f-1focal python3-rbd=18.0.0-1736-ga21b2f4f-1focal libcephfs2=18.0.0-1736-ga21b2f4f-1focal libcephfs-dev=18.0.0-1736-ga21b2f4f-1focal librados2=18.0.0-1736-ga21b2f4f-1focal librbd1=18.0.0-1736-ga21b2f4f-1focal rbd-fuse=18.0.0-1736-ga21b2f4f-1focal 2023-01-15T14:24:54.761 INFO:teuthology.orchestra.run.smithi090.stdout:Reading package lists... 2023-01-15T14:24:54.824 INFO:teuthology.orchestra.run.smithi141.stdout:Reading package lists... 2023-01-15T14:24:54.947 INFO:teuthology.orchestra.run.smithi090.stdout:Building dependency tree... 2023-01-15T14:24:54.948 INFO:teuthology.orchestra.run.smithi090.stdout:Reading state information... 2023-01-15T14:24:55.010 INFO:teuthology.orchestra.run.smithi141.stdout:Building dependency tree... 2023-01-15T14:24:55.010 INFO:teuthology.orchestra.run.smithi141.stdout:Reading state information... 2023-01-15T14:24:55.112 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages were automatically installed and are no longer required: 2023-01-15T14:24:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-15T14:24:55.113 INFO:teuthology.orchestra.run.smithi090.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-15T14:24:55.113 INFO:teuthology.orchestra.run.smithi090.stdout:The following additional packages will be installed: 2023-01-15T14:24:55.113 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-01-15T14:24:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-01-15T14:24:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-01-15T14:24:55.114 INFO:teuthology.orchestra.run.smithi090.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-01-15T14:24:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-01-15T14:24:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-01-15T14:24:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2023-01-15T14:24:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2023-01-15T14:24:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-01-15T14:24:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2023-01-15T14:24:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-01-15T14:24:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-01-15T14:24:55.115 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-01-15T14:24:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-01-15T14:24:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-01-15T14:24:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-01-15T14:24:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2023-01-15T14:24:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-01-15T14:24:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-01-15T14:24:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-01-15T14:24:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-01-15T14:24:55.116 INFO:teuthology.orchestra.run.smithi090.stdout: qttranslations5-l10n socat xmlstarlet 2023-01-15T14:24:55.117 INFO:teuthology.orchestra.run.smithi090.stdout:Suggested packages: 2023-01-15T14:24:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-01-15T14:24:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-01-15T14:24:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-01-15T14:24:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-01-15T14:24:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-01-15T14:24:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-01-15T14:24:55.117 INFO:teuthology.orchestra.run.smithi090.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-01-15T14:24:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-01-15T14:24:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-01-15T14:24:55.118 INFO:teuthology.orchestra.run.smithi090.stdout:Recommended packages: 2023-01-15T14:24:55.118 INFO:teuthology.orchestra.run.smithi090.stdout: btrfs-tools 2023-01-15T14:24:55.174 INFO:teuthology.orchestra.run.smithi141.stdout:The following packages were automatically installed and are no longer required: 2023-01-15T14:24:55.177 INFO:teuthology.orchestra.run.smithi141.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2023-01-15T14:24:55.177 INFO:teuthology.orchestra.run.smithi141.stdout:Use 'sudo apt autoremove' to remove them. 2023-01-15T14:24:55.177 INFO:teuthology.orchestra.run.smithi141.stdout:The following additional packages will be installed: 2023-01-15T14:24:55.177 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-01-15T14:24:55.178 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2023-01-15T14:24:55.178 INFO:teuthology.orchestra.run.smithi141.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2023-01-15T14:24:55.178 INFO:teuthology.orchestra.run.smithi141.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2023-01-15T14:24:55.178 INFO:teuthology.orchestra.run.smithi141.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2023-01-15T14:24:55.178 INFO:teuthology.orchestra.run.smithi141.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2023-01-15T14:24:55.178 INFO:teuthology.orchestra.run.smithi141.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2023-01-15T14:24:55.178 INFO:teuthology.orchestra.run.smithi141.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2023-01-15T14:24:55.178 INFO:teuthology.orchestra.run.smithi141.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2023-01-15T14:24:55.178 INFO:teuthology.orchestra.run.smithi141.stdout: python3-ceph-common python3-cherrypy3 python3-cryptography python3-decorator 2023-01-15T14:24:55.178 INFO:teuthology.orchestra.run.smithi141.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2023-01-15T14:24:55.178 INFO:teuthology.orchestra.run.smithi141.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout: python3-scipy python3-simplegeneric python3-simplejson 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout: qttranslations5-l10n socat xmlstarlet 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout:Suggested packages: 2023-01-15T14:24:55.179 INFO:teuthology.orchestra.run.smithi141.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2023-01-15T14:24:55.180 INFO:teuthology.orchestra.run.smithi141.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2023-01-15T14:24:55.180 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2023-01-15T14:24:55.180 INFO:teuthology.orchestra.run.smithi141.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2023-01-15T14:24:55.180 INFO:teuthology.orchestra.run.smithi141.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2023-01-15T14:24:55.180 INFO:teuthology.orchestra.run.smithi141.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2023-01-15T14:24:55.180 INFO:teuthology.orchestra.run.smithi141.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2023-01-15T14:24:55.180 INFO:teuthology.orchestra.run.smithi141.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-01-15T14:24:55.180 INFO:teuthology.orchestra.run.smithi141.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2023-01-15T14:24:55.180 INFO:teuthology.orchestra.run.smithi141.stdout:Recommended packages: 2023-01-15T14:24:55.181 INFO:teuthology.orchestra.run.smithi141.stdout: btrfs-tools 2023-01-15T14:24:55.232 INFO:teuthology.orchestra.run.smithi090.stdout:The following NEW packages will be installed: 2023-01-15T14:24:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-01-15T14:24:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-01-15T14:24:55.232 INFO:teuthology.orchestra.run.smithi090.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-01-15T14:24:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-01-15T14:24:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-01-15T14:24:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-01-15T14:24:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-01-15T14:24:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2023-01-15T14:24:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2023-01-15T14:24:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-01-15T14:24:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-01-15T14:24:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-01-15T14:24:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-01-15T14:24:55.233 INFO:teuthology.orchestra.run.smithi090.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout: xmlstarlet 2023-01-15T14:24:55.234 INFO:teuthology.orchestra.run.smithi090.stdout:The following packages will be upgraded: 2023-01-15T14:24:55.235 INFO:teuthology.orchestra.run.smithi090.stdout: librados2 librbd1 2023-01-15T14:24:55.287 INFO:teuthology.orchestra.run.smithi141.stdout:The following NEW packages will be installed: 2023-01-15T14:24:55.287 INFO:teuthology.orchestra.run.smithi141.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-01-15T14:24:55.287 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-01-15T14:24:55.287 INFO:teuthology.orchestra.run.smithi141.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-01-15T14:24:55.287 INFO:teuthology.orchestra.run.smithi141.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2023-01-15T14:24:55.287 INFO:teuthology.orchestra.run.smithi141.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2023-01-15T14:24:55.287 INFO:teuthology.orchestra.run.smithi141.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-01-15T14:24:55.288 INFO:teuthology.orchestra.run.smithi141.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2023-01-15T14:24:55.290 INFO:teuthology.orchestra.run.smithi141.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-asyncssh-doc 2023-01-15T14:24:55.290 INFO:teuthology.orchestra.run.smithi141.stdout: python-natsort-doc python-pastedeploy-tpl python3-asyncssh 2023-01-15T14:24:55.290 INFO:teuthology.orchestra.run.smithi141.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2023-01-15T14:24:55.290 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-01-15T14:24:55.290 INFO:teuthology.orchestra.run.smithi141.stdout: python3-cherrypy3 python3-cryptography python3-decorator python3-google-auth 2023-01-15T14:24:55.290 INFO:teuthology.orchestra.run.smithi141.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2023-01-15T14:24:55.290 INFO:teuthology.orchestra.run.smithi141.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2023-01-15T14:24:55.290 INFO:teuthology.orchestra.run.smithi141.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2023-01-15T14:24:55.290 INFO:teuthology.orchestra.run.smithi141.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2023-01-15T14:24:55.291 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2023-01-15T14:24:55.291 INFO:teuthology.orchestra.run.smithi141.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pytest 2023-01-15T14:24:55.291 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2023-01-15T14:24:55.291 INFO:teuthology.orchestra.run.smithi141.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2023-01-15T14:24:55.291 INFO:teuthology.orchestra.run.smithi141.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-01-15T14:24:55.291 INFO:teuthology.orchestra.run.smithi141.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2023-01-15T14:24:55.291 INFO:teuthology.orchestra.run.smithi141.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2023-01-15T14:24:55.292 INFO:teuthology.orchestra.run.smithi141.stdout: python3-webtest python3-werkzeug qttranslations5-l10n radosgw rbd-fuse socat 2023-01-15T14:24:55.292 INFO:teuthology.orchestra.run.smithi141.stdout: xmlstarlet 2023-01-15T14:24:55.292 INFO:teuthology.orchestra.run.smithi141.stdout:The following packages will be upgraded: 2023-01-15T14:24:55.292 INFO:teuthology.orchestra.run.smithi141.stdout: librados2 librbd1 2023-01-15T14:24:55.420 INFO:teuthology.orchestra.run.smithi090.stdout:2 upgraded, 110 newly installed, 0 to remove and 116 not upgraded. 2023-01-15T14:24:55.420 INFO:teuthology.orchestra.run.smithi090.stdout:Need to get 211 MB of archives. 2023-01-15T14:24:55.420 INFO:teuthology.orchestra.run.smithi090.stdout:After this operation, 1,031 MB of additional disk space will be used. 2023-01-15T14:24:55.420 INFO:teuthology.orchestra.run.smithi090.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-01-15T14:24:55.477 INFO:teuthology.orchestra.run.smithi141.stdout:2 upgraded, 110 newly installed, 0 to remove and 116 not upgraded. 2023-01-15T14:24:55.477 INFO:teuthology.orchestra.run.smithi141.stdout:Need to get 211 MB of archives. 2023-01-15T14:24:55.478 INFO:teuthology.orchestra.run.smithi141.stdout:After this operation, 1,031 MB of additional disk space will be used. 2023-01-15T14:24:55.478 INFO:teuthology.orchestra.run.smithi141.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2023-01-15T14:24:55.631 INFO:teuthology.orchestra.run.smithi090.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.0.0-1736-ga21b2f4f-1focal [3,429 kB] 2023-01-15T14:24:55.689 INFO:teuthology.orchestra.run.smithi141.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.0.0-1736-ga21b2f4f-1focal [3,429 kB] 2023-01-15T14:24:55.809 INFO:teuthology.orchestra.run.smithi090.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-01-15T14:24:55.825 INFO:teuthology.orchestra.run.smithi090.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-01-15T14:24:55.866 INFO:teuthology.orchestra.run.smithi141.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2023-01-15T14:24:55.879 INFO:teuthology.orchestra.run.smithi141.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2023-01-15T14:24:55.905 INFO:teuthology.orchestra.run.smithi090.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-01-15T14:24:55.962 INFO:teuthology.orchestra.run.smithi141.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2023-01-15T14:24:56.144 INFO:teuthology.orchestra.run.smithi090.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-01-15T14:24:56.154 INFO:teuthology.orchestra.run.smithi090.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-01-15T14:24:56.182 INFO:teuthology.orchestra.run.smithi090.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-01-15T14:24:56.190 INFO:teuthology.orchestra.run.smithi090.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-01-15T14:24:56.191 INFO:teuthology.orchestra.run.smithi090.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-01-15T14:24:56.195 INFO:teuthology.orchestra.run.smithi090.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-01-15T14:24:56.196 INFO:teuthology.orchestra.run.smithi090.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-01-15T14:24:56.201 INFO:teuthology.orchestra.run.smithi141.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2023-01-15T14:24:56.208 INFO:teuthology.orchestra.run.smithi090.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-01-15T14:24:56.209 INFO:teuthology.orchestra.run.smithi090.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-01-15T14:24:56.213 INFO:teuthology.orchestra.run.smithi141.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2023-01-15T14:24:56.225 INFO:teuthology.orchestra.run.smithi090.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-01-15T14:24:56.241 INFO:teuthology.orchestra.run.smithi141.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2023-01-15T14:24:56.249 INFO:teuthology.orchestra.run.smithi141.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2023-01-15T14:24:56.249 INFO:teuthology.orchestra.run.smithi141.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2023-01-15T14:24:56.253 INFO:teuthology.orchestra.run.smithi141.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2023-01-15T14:24:56.254 INFO:teuthology.orchestra.run.smithi141.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2023-01-15T14:24:56.267 INFO:teuthology.orchestra.run.smithi141.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2023-01-15T14:24:56.267 INFO:teuthology.orchestra.run.smithi141.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2023-01-15T14:24:56.282 INFO:teuthology.orchestra.run.smithi141.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2023-01-15T14:24:56.294 INFO:teuthology.orchestra.run.smithi090.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-01-15T14:24:56.294 INFO:teuthology.orchestra.run.smithi090.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-01-15T14:24:56.295 INFO:teuthology.orchestra.run.smithi090.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-01-15T14:24:56.297 INFO:teuthology.orchestra.run.smithi090.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-01-15T14:24:56.297 INFO:teuthology.orchestra.run.smithi090.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-01-15T14:24:56.298 INFO:teuthology.orchestra.run.smithi090.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-01-15T14:24:56.299 INFO:teuthology.orchestra.run.smithi090.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-01-15T14:24:56.300 INFO:teuthology.orchestra.run.smithi090.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-01-15T14:24:56.305 INFO:teuthology.orchestra.run.smithi090.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-01-15T14:24:56.352 INFO:teuthology.orchestra.run.smithi141.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2023-01-15T14:24:56.353 INFO:teuthology.orchestra.run.smithi141.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2023-01-15T14:24:56.354 INFO:teuthology.orchestra.run.smithi141.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2023-01-15T14:24:56.356 INFO:teuthology.orchestra.run.smithi141.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-01-15T14:24:56.357 INFO:teuthology.orchestra.run.smithi141.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2023-01-15T14:24:56.357 INFO:teuthology.orchestra.run.smithi141.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2023-01-15T14:24:56.359 INFO:teuthology.orchestra.run.smithi141.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2023-01-15T14:24:56.359 INFO:teuthology.orchestra.run.smithi141.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2023-01-15T14:24:56.361 INFO:teuthology.orchestra.run.smithi141.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2023-01-15T14:24:56.382 INFO:teuthology.orchestra.run.smithi090.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-01-15T14:24:56.383 INFO:teuthology.orchestra.run.smithi090.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-01-15T14:24:56.392 INFO:teuthology.orchestra.run.smithi090.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-01-15T14:24:56.392 INFO:teuthology.orchestra.run.smithi090.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-01-15T14:24:56.393 INFO:teuthology.orchestra.run.smithi090.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-01-15T14:24:56.394 INFO:teuthology.orchestra.run.smithi090.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-01-15T14:24:56.396 INFO:teuthology.orchestra.run.smithi090.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2023-01-15T14:24:56.399 INFO:teuthology.orchestra.run.smithi090.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-01-15T14:24:56.401 INFO:teuthology.orchestra.run.smithi090.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-01-15T14:24:56.441 INFO:teuthology.orchestra.run.smithi141.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2023-01-15T14:24:56.442 INFO:teuthology.orchestra.run.smithi141.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2023-01-15T14:24:56.450 INFO:teuthology.orchestra.run.smithi141.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2023-01-15T14:24:56.451 INFO:teuthology.orchestra.run.smithi141.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2023-01-15T14:24:56.452 INFO:teuthology.orchestra.run.smithi141.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2023-01-15T14:24:56.453 INFO:teuthology.orchestra.run.smithi141.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2023-01-15T14:24:56.454 INFO:teuthology.orchestra.run.smithi141.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2023-01-15T14:24:56.456 INFO:teuthology.orchestra.run.smithi141.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2023-01-15T14:24:56.458 INFO:teuthology.orchestra.run.smithi141.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2023-01-15T14:24:56.469 INFO:teuthology.orchestra.run.smithi090.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2023-01-15T14:24:56.469 INFO:teuthology.orchestra.run.smithi090.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-01-15T14:24:56.524 INFO:teuthology.orchestra.run.smithi141.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2023-01-15T14:24:56.527 INFO:teuthology.orchestra.run.smithi141.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2023-01-15T14:24:56.546 INFO:teuthology.orchestra.run.smithi090.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-01-15T14:24:56.547 INFO:teuthology.orchestra.run.smithi090.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-01-15T14:24:56.548 INFO:teuthology.orchestra.run.smithi090.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-01-15T14:24:56.549 INFO:teuthology.orchestra.run.smithi090.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-01-15T14:24:56.550 INFO:teuthology.orchestra.run.smithi090.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-01-15T14:24:56.552 INFO:teuthology.orchestra.run.smithi090.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-01-15T14:24:56.553 INFO:teuthology.orchestra.run.smithi090.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-01-15T14:24:56.553 INFO:teuthology.orchestra.run.smithi090.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-01-15T14:24:56.604 INFO:teuthology.orchestra.run.smithi141.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2023-01-15T14:24:56.605 INFO:teuthology.orchestra.run.smithi141.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2023-01-15T14:24:56.606 INFO:teuthology.orchestra.run.smithi141.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2023-01-15T14:24:56.609 INFO:teuthology.orchestra.run.smithi141.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2023-01-15T14:24:56.609 INFO:teuthology.orchestra.run.smithi141.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2023-01-15T14:24:56.610 INFO:teuthology.orchestra.run.smithi141.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2023-01-15T14:24:56.611 INFO:teuthology.orchestra.run.smithi141.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2023-01-15T14:24:56.612 INFO:teuthology.orchestra.run.smithi141.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2023-01-15T14:24:56.629 INFO:teuthology.orchestra.run.smithi090.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-01-15T14:24:56.632 INFO:teuthology.orchestra.run.smithi090.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-01-15T14:24:56.684 INFO:teuthology.orchestra.run.smithi141.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2023-01-15T14:24:56.687 INFO:teuthology.orchestra.run.smithi141.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-01-15T14:24:56.710 INFO:teuthology.orchestra.run.smithi090.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-01-15T14:24:56.713 INFO:teuthology.orchestra.run.smithi090.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-01-15T14:24:56.715 INFO:teuthology.orchestra.run.smithi090.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-01-15T14:24:56.717 INFO:teuthology.orchestra.run.smithi090.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-01-15T14:24:56.718 INFO:teuthology.orchestra.run.smithi090.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-01-15T14:24:56.719 INFO:teuthology.orchestra.run.smithi090.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-01-15T14:24:56.739 INFO:teuthology.orchestra.run.smithi090.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.0.0-1736-ga21b2f4f-1focal [3,965 kB] 2023-01-15T14:24:56.764 INFO:teuthology.orchestra.run.smithi141.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2023-01-15T14:24:56.767 INFO:teuthology.orchestra.run.smithi141.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2023-01-15T14:24:56.769 INFO:teuthology.orchestra.run.smithi141.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2023-01-15T14:24:56.772 INFO:teuthology.orchestra.run.smithi141.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2023-01-15T14:24:56.773 INFO:teuthology.orchestra.run.smithi141.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2023-01-15T14:24:56.774 INFO:teuthology.orchestra.run.smithi141.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2023-01-15T14:24:56.986 INFO:teuthology.orchestra.run.smithi090.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-01-15T14:24:57.043 INFO:teuthology.orchestra.run.smithi141.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2023-01-15T14:24:57.062 INFO:teuthology.orchestra.run.smithi090.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-01-15T14:24:57.065 INFO:teuthology.orchestra.run.smithi090.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-01-15T14:24:57.116 INFO:teuthology.orchestra.run.smithi090.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-01-15T14:24:57.116 INFO:teuthology.orchestra.run.smithi090.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-01-15T14:24:57.117 INFO:teuthology.orchestra.run.smithi090.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-01-15T14:24:57.119 INFO:teuthology.orchestra.run.smithi141.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2023-01-15T14:24:57.119 INFO:teuthology.orchestra.run.smithi090.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-01-15T14:24:57.120 INFO:teuthology.orchestra.run.smithi090.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-01-15T14:24:57.120 INFO:teuthology.orchestra.run.smithi090.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-01-15T14:24:57.120 INFO:teuthology.orchestra.run.smithi141.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2023-01-15T14:24:57.120 INFO:teuthology.orchestra.run.smithi090.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-01-15T14:24:57.133 INFO:teuthology.orchestra.run.smithi141.stdout:Get:55 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.0.0-1736-ga21b2f4f-1focal [3,965 kB] 2023-01-15T14:24:57.148 INFO:teuthology.orchestra.run.smithi090.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-01-15T14:24:57.148 INFO:teuthology.orchestra.run.smithi090.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-01-15T14:24:57.170 INFO:teuthology.orchestra.run.smithi141.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2023-01-15T14:24:57.170 INFO:teuthology.orchestra.run.smithi141.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2023-01-15T14:24:57.171 INFO:teuthology.orchestra.run.smithi141.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2023-01-15T14:24:57.172 INFO:teuthology.orchestra.run.smithi141.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2023-01-15T14:24:57.173 INFO:teuthology.orchestra.run.smithi141.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2023-01-15T14:24:57.174 INFO:teuthology.orchestra.run.smithi141.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2023-01-15T14:24:57.174 INFO:teuthology.orchestra.run.smithi141.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2023-01-15T14:24:57.204 INFO:teuthology.orchestra.run.smithi141.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2023-01-15T14:24:57.204 INFO:teuthology.orchestra.run.smithi141.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2023-01-15T14:24:57.228 INFO:teuthology.orchestra.run.smithi090.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-01-15T14:24:57.233 INFO:teuthology.orchestra.run.smithi090.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-01-15T14:24:57.236 INFO:teuthology.orchestra.run.smithi090.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-01-15T14:24:57.239 INFO:teuthology.orchestra.run.smithi090.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-01-15T14:24:57.241 INFO:teuthology.orchestra.run.smithi090.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-01-15T14:24:57.244 INFO:teuthology.orchestra.run.smithi090.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-01-15T14:24:57.246 INFO:teuthology.orchestra.run.smithi090.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-01-15T14:24:57.284 INFO:teuthology.orchestra.run.smithi141.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2023-01-15T14:24:57.288 INFO:teuthology.orchestra.run.smithi141.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2023-01-15T14:24:57.291 INFO:teuthology.orchestra.run.smithi141.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2023-01-15T14:24:57.293 INFO:teuthology.orchestra.run.smithi141.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2023-01-15T14:24:57.295 INFO:teuthology.orchestra.run.smithi141.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2023-01-15T14:24:57.299 INFO:teuthology.orchestra.run.smithi141.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2023-01-15T14:24:57.301 INFO:teuthology.orchestra.run.smithi141.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2023-01-15T14:24:57.312 INFO:teuthology.orchestra.run.smithi090.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-01-15T14:24:57.318 INFO:teuthology.orchestra.run.smithi090.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2023-01-15T14:24:57.322 INFO:teuthology.orchestra.run.smithi090.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-01-15T14:24:57.368 INFO:teuthology.orchestra.run.smithi141.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2023-01-15T14:24:57.374 INFO:teuthology.orchestra.run.smithi141.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2023-01-15T14:24:57.378 INFO:teuthology.orchestra.run.smithi141.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2023-01-15T14:24:57.478 INFO:teuthology.orchestra.run.smithi090.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-01-15T14:24:57.479 INFO:teuthology.orchestra.run.smithi090.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-01-15T14:24:57.479 INFO:teuthology.orchestra.run.smithi090.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-01-15T14:24:57.480 INFO:teuthology.orchestra.run.smithi090.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-01-15T14:24:57.496 INFO:teuthology.orchestra.run.smithi090.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-01-15T14:24:57.497 INFO:teuthology.orchestra.run.smithi090.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-01-15T14:24:57.498 INFO:teuthology.orchestra.run.smithi090.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-01-15T14:24:57.498 INFO:teuthology.orchestra.run.smithi090.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-01-15T14:24:57.498 INFO:teuthology.orchestra.run.smithi090.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-01-15T14:24:57.537 INFO:teuthology.orchestra.run.smithi141.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2023-01-15T14:24:57.538 INFO:teuthology.orchestra.run.smithi141.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2023-01-15T14:24:57.539 INFO:teuthology.orchestra.run.smithi141.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2023-01-15T14:24:57.539 INFO:teuthology.orchestra.run.smithi141.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2023-01-15T14:24:57.554 INFO:teuthology.orchestra.run.smithi141.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-01-15T14:24:57.554 INFO:teuthology.orchestra.run.smithi141.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2023-01-15T14:24:57.555 INFO:teuthology.orchestra.run.smithi141.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2023-01-15T14:24:57.555 INFO:teuthology.orchestra.run.smithi141.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2023-01-15T14:24:57.556 INFO:teuthology.orchestra.run.smithi141.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2023-01-15T14:24:57.673 INFO:teuthology.orchestra.run.smithi090.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.0.0-1736-ga21b2f4f-1focal [721 kB] 2023-01-15T14:24:57.815 INFO:teuthology.orchestra.run.smithi090.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.0.0-1736-ga21b2f4f-1focal [376 kB] 2023-01-15T14:24:57.873 INFO:teuthology.orchestra.run.smithi090.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.0.0-1736-ga21b2f4f-1focal [22.4 kB] 2023-01-15T14:24:57.874 INFO:teuthology.orchestra.run.smithi090.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.0.0-1736-ga21b2f4f-1focal [185 kB] 2023-01-15T14:24:57.919 INFO:teuthology.orchestra.run.smithi090.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.0.0-1736-ga21b2f4f-1focal [46.2 kB] 2023-01-15T14:24:57.920 INFO:teuthology.orchestra.run.smithi090.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.0.0-1736-ga21b2f4f-1focal [354 kB] 2023-01-15T14:24:57.971 INFO:teuthology.orchestra.run.smithi090.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.0.0-1736-ga21b2f4f-1focal [6,499 kB] 2023-01-15T14:24:58.164 INFO:teuthology.orchestra.run.smithi141.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.0.0-1736-ga21b2f4f-1focal [721 kB] 2023-01-15T14:24:58.314 INFO:teuthology.orchestra.run.smithi141.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.0.0-1736-ga21b2f4f-1focal [376 kB] 2023-01-15T14:24:58.401 INFO:teuthology.orchestra.run.smithi141.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.0.0-1736-ga21b2f4f-1focal [22.4 kB] 2023-01-15T14:24:58.427 INFO:teuthology.orchestra.run.smithi141.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.0.0-1736-ga21b2f4f-1focal [185 kB] 2023-01-15T14:24:58.458 INFO:teuthology.orchestra.run.smithi141.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.0.0-1736-ga21b2f4f-1focal [46.2 kB] 2023-01-15T14:24:58.460 INFO:teuthology.orchestra.run.smithi141.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.0.0-1736-ga21b2f4f-1focal [354 kB] 2023-01-15T14:24:58.569 INFO:teuthology.orchestra.run.smithi141.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.0.0-1736-ga21b2f4f-1focal [6,499 kB] 2023-01-15T14:24:58.986 INFO:teuthology.orchestra.run.smithi090.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.0.0-1736-ga21b2f4f-1focal [104 kB] 2023-01-15T14:24:59.015 INFO:teuthology.orchestra.run.smithi090.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.0.0-1736-ga21b2f4f-1focal [466 kB] 2023-01-15T14:24:59.087 INFO:teuthology.orchestra.run.smithi090.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.0.0-1736-ga21b2f4f-1focal [29.5 MB] 2023-01-15T14:24:59.612 INFO:teuthology.orchestra.run.smithi141.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.0.0-1736-ga21b2f4f-1focal [104 kB] 2023-01-15T14:24:59.615 INFO:teuthology.orchestra.run.smithi141.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.0.0-1736-ga21b2f4f-1focal [466 kB] 2023-01-15T14:24:59.659 INFO:teuthology.orchestra.run.smithi141.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.0.0-1736-ga21b2f4f-1focal [29.5 MB] 2023-01-15T14:25:01.332 INFO:teuthology.orchestra.run.smithi090.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.0.0-1736-ga21b2f4f-1focal [10.7 MB] 2023-01-15T14:25:01.983 INFO:teuthology.orchestra.run.smithi090.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.0.0-1736-ga21b2f4f-1focal [219 kB] 2023-01-15T14:25:02.790 INFO:teuthology.orchestra.run.smithi141.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.0.0-1736-ga21b2f4f-1focal [10.7 MB] 2023-01-15T14:25:02.791 INFO:teuthology.orchestra.run.smithi090.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.0.0-1736-ga21b2f4f-1focal [133 kB] 2023-01-15T14:25:02.791 INFO:teuthology.orchestra.run.smithi090.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.0.0-1736-ga21b2f4f-1focal [1,534 kB] 2023-01-15T14:25:02.792 INFO:teuthology.orchestra.run.smithi090.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.0.0-1736-ga21b2f4f-1focal [7,855 kB] 2023-01-15T14:25:02.792 INFO:teuthology.orchestra.run.smithi090.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.0.0-1736-ga21b2f4f-1focal [28.9 MB] 2023-01-15T14:25:03.072 INFO:teuthology.orchestra.run.smithi141.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.0.0-1736-ga21b2f4f-1focal [219 kB] 2023-01-15T14:25:03.075 INFO:teuthology.orchestra.run.smithi141.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.0.0-1736-ga21b2f4f-1focal [133 kB] 2023-01-15T14:25:03.080 INFO:teuthology.orchestra.run.smithi141.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.0.0-1736-ga21b2f4f-1focal [1,534 kB] 2023-01-15T14:25:03.174 INFO:teuthology.orchestra.run.smithi141.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.0.0-1736-ga21b2f4f-1focal [7,855 kB] 2023-01-15T14:25:03.749 INFO:teuthology.orchestra.run.smithi141.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.0.0-1736-ga21b2f4f-1focal [28.9 MB] 2023-01-15T14:25:04.577 INFO:teuthology.orchestra.run.smithi090.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.0.0-1736-ga21b2f4f-1focal [3,812 B] 2023-01-15T14:25:04.577 INFO:teuthology.orchestra.run.smithi090.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.0.0-1736-ga21b2f4f-1focal [829 kB] 2023-01-15T14:25:04.591 INFO:teuthology.orchestra.run.smithi090.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.0.0-1736-ga21b2f4f-1focal [2,280 kB] 2023-01-15T14:25:04.764 INFO:teuthology.orchestra.run.smithi090.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.0.0-1736-ga21b2f4f-1focal [95.2 kB] 2023-01-15T14:25:04.766 INFO:teuthology.orchestra.run.smithi090.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.0.0-1736-ga21b2f4f-1focal [114 kB] 2023-01-15T14:25:04.768 INFO:teuthology.orchestra.run.smithi090.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.0.0-1736-ga21b2f4f-1focal [1,722 kB] 2023-01-15T14:25:04.991 INFO:teuthology.orchestra.run.smithi090.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.0.0-1736-ga21b2f4f-1focal [7,876 kB] 2023-01-15T14:25:06.199 INFO:teuthology.orchestra.run.smithi090.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.0.0-1736-ga21b2f4f-1focal [3,932 B] 2023-01-15T14:25:06.200 INFO:teuthology.orchestra.run.smithi090.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.0.0-1736-ga21b2f4f-1focal [53.7 MB] 2023-01-15T14:25:10.075 INFO:teuthology.orchestra.run.smithi141.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.0.0-1736-ga21b2f4f-1focal [3,812 B] 2023-01-15T14:25:10.076 INFO:teuthology.orchestra.run.smithi141.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.0.0-1736-ga21b2f4f-1focal [829 kB] 2023-01-15T14:25:10.256 INFO:teuthology.orchestra.run.smithi141.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.0.0-1736-ga21b2f4f-1focal [2,280 kB] 2023-01-15T14:25:10.853 INFO:teuthology.orchestra.run.smithi141.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.0.0-1736-ga21b2f4f-1focal [95.2 kB] 2023-01-15T14:25:10.881 INFO:teuthology.orchestra.run.smithi141.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.0.0-1736-ga21b2f4f-1focal [114 kB] 2023-01-15T14:25:10.911 INFO:teuthology.orchestra.run.smithi141.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.0.0-1736-ga21b2f4f-1focal [1,722 kB] 2023-01-15T14:25:11.929 INFO:teuthology.orchestra.run.smithi141.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.0.0-1736-ga21b2f4f-1focal [7,876 kB] 2023-01-15T14:25:14.332 INFO:teuthology.orchestra.run.smithi141.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.0.0-1736-ga21b2f4f-1focal [3,932 B] 2023-01-15T14:25:14.453 INFO:teuthology.orchestra.run.smithi141.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.0.0-1736-ga21b2f4f-1focal [53.7 MB] 2023-01-15T14:25:17.156 INFO:teuthology.orchestra.run.smithi090.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.0.0-1736-ga21b2f4f-1focal [225 kB] 2023-01-15T14:25:17.160 INFO:teuthology.orchestra.run.smithi090.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.0.0-1736-ga21b2f4f-1focal [27.4 kB] 2023-01-15T14:25:17.160 INFO:teuthology.orchestra.run.smithi090.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.0.0-1736-ga21b2f4f-1focal [13.3 MB] 2023-01-15T14:25:20.409 INFO:teuthology.orchestra.run.smithi090.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.0.0-1736-ga21b2f4f-1focal [97.2 kB] 2023-01-15T14:25:21.031 INFO:teuthology.orchestra.run.smithi090.stdout:Fetched 211 MB in 25s (8,370 kB/s) 2023-01-15T14:25:22.214 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-01-15T14:25:22.216 INFO:teuthology.orchestra.run.smithi090.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141754 files and directories currently installed.) 2023-01-15T14:25:22.216 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-01-15T14:25:22.217 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-15T14:25:22.217 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-01-15T14:25:22.217 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-01-15T14:25:22.217 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-15T14:25:22.217 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-01-15T14:25:22.217 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-01-15T14:25:22.217 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-15T14:25:22.610 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-01-15T14:25:23.687 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-15T14:25:23.687 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:25:23.688 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-01-15T14:25:23.688 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-15T14:25:23.688 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:25:23.688 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-01-15T14:25:23.688 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-15T14:25:23.688 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:25:24.040 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-01-15T14:25:24.057 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-01-15T14:25:24.099 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-15T14:25:24.625 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../007-librbd1_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:24.625 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librbd1 (18.0.0-1736-ga21b2f4f-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-15T14:25:25.314 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../008-librados2_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:25.440 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librados2 (18.0.0-1736-ga21b2f4f-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-15T14:25:26.194 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libcephfs2. 2023-01-15T14:25:26.212 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../009-libcephfs2_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:26.253 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libcephfs2 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:26.582 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rados. 2023-01-15T14:25:26.599 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../010-python3-rados_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:26.640 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rados (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:27.026 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-01-15T14:25:27.043 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:25:27.076 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-ceph-argparse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:27.321 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cephfs. 2023-01-15T14:25:27.340 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../012-python3-cephfs_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:27.379 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cephfs (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:28.517 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-ceph-common. 2023-01-15T14:25:28.532 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../013-python3-ceph-common_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:25:28.575 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-ceph-common (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:28.928 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-prettytable. 2023-01-15T14:25:29.174 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2023-01-15T14:25:29.174 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-01-15T14:25:29.248 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rbd. 2023-01-15T14:25:29.814 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../015-python3-rbd_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:29.814 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rbd (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:29.815 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-01-15T14:25:29.815 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-01-15T14:25:29.815 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-15T14:25:30.063 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-01-15T14:25:30.134 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2023-01-15T14:25:30.135 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-01-15T14:25:30.500 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-01-15T14:25:30.516 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-01-15T14:25:32.071 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-15T14:25:32.071 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package librgw2. 2023-01-15T14:25:32.072 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../019-librgw2_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:32.072 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking librgw2 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:32.072 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rgw. 2023-01-15T14:25:32.072 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:32.072 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rgw (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:32.244 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liboath0. 2023-01-15T14:25:32.390 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-01-15T14:25:32.391 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-01-15T14:25:32.589 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libradosstriper1. 2023-01-15T14:25:32.712 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:32.712 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libradosstriper1 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:33.110 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-common. 2023-01-15T14:25:33.125 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../023-ceph-common_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:33.202 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-common (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:35.133 INFO:teuthology.orchestra.run.smithi141.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.0.0-1736-ga21b2f4f-1focal [225 kB] 2023-01-15T14:25:35.145 INFO:teuthology.orchestra.run.smithi141.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.0.0-1736-ga21b2f4f-1focal [27.4 kB] 2023-01-15T14:25:35.145 INFO:teuthology.orchestra.run.smithi141.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.0.0-1736-ga21b2f4f-1focal [13.3 MB] 2023-01-15T14:25:36.042 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-base. 2023-01-15T14:25:36.058 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../024-ceph-base_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:36.127 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-base (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:37.581 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cherrypy3. 2023-01-15T14:25:37.598 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2023-01-15T14:25:37.640 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-01-15T14:25:37.994 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-natsort. 2023-01-15T14:25:38.013 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2023-01-15T14:25:38.051 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-01-15T14:25:38.355 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-logutils. 2023-01-15T14:25:38.415 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2023-01-15T14:25:38.415 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-01-15T14:25:38.905 INFO:teuthology.orchestra.run.smithi141.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-rf-newsched-v3/a21b2f4fd40d883dac62c8cd315cbb54848ee067/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.0.0-1736-ga21b2f4f-1focal [97.2 kB] 2023-01-15T14:25:39.059 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-markupsafe. 2023-01-15T14:25:39.079 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-01-15T14:25:39.235 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-01-15T14:25:39.412 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-mako. 2023-01-15T14:25:39.438 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-01-15T14:25:39.470 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-15T14:25:39.525 INFO:teuthology.orchestra.run.smithi141.stdout:Fetched 211 MB in 44s (4,831 kB/s) 2023-01-15T14:25:39.707 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2023-01-15T14:25:39.763 INFO:teuthology.orchestra.run.smithi141.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141754 files and directories currently installed.) 2023-01-15T14:25:39.852 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2023-01-15T14:25:39.852 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-simplegeneric. 2023-01-15T14:25:39.852 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2023-01-15T14:25:39.856 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-15T14:25:39.892 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-01-15T14:25:40.195 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-singledispatch. 2023-01-15T14:25:40.262 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-01-15T14:25:40.262 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-01-15T14:25:40.262 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-01-15T14:25:40.263 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2023-01-15T14:25:40.263 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-15T14:25:40.564 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-webob. 2023-01-15T14:25:40.581 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2023-01-15T14:25:40.606 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-01-15T14:25:40.622 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-01-15T14:25:40.626 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../002-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2023-01-15T14:25:40.664 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-15T14:25:40.952 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-soupsieve. 2023-01-15T14:25:40.969 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-01-15T14:25:41.010 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-15T14:25:41.127 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-01-15T14:25:41.144 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../003-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-15T14:25:41.188 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:25:41.296 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-bs4. 2023-01-15T14:25:41.316 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2023-01-15T14:25:41.354 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-01-15T14:25:41.710 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-01-15T14:25:41.728 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../004-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-15T14:25:41.733 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-waitress. 2023-01-15T14:25:41.750 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-01-15T14:25:41.761 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:25:41.795 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-15T14:25:42.110 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-01-15T14:25:42.113 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-tempita. 2023-01-15T14:25:42.126 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../005-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2023-01-15T14:25:42.132 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2023-01-15T14:25:42.171 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-01-15T14:25:42.185 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:25:42.466 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-paste. 2023-01-15T14:25:42.482 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-01-15T14:25:42.524 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-01-15T14:25:42.557 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libthrift-0.13.0. 2023-01-15T14:25:42.573 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../006-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2023-01-15T14:25:42.615 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-15T14:25:42.854 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-01-15T14:25:42.870 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-01-15T14:25:42.912 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-01-15T14:25:43.125 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../007-librbd1_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:43.221 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pastedeploy. 2023-01-15T14:25:43.237 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2023-01-15T14:25:43.281 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-01-15T14:25:43.285 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking librbd1 (18.0.0-1736-ga21b2f4f-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-15T14:25:43.610 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-webtest. 2023-01-15T14:25:43.626 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2023-01-15T14:25:43.741 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-01-15T14:25:43.964 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pecan. 2023-01-15T14:25:44.050 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../008-librados2_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:44.050 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-01-15T14:25:44.051 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-01-15T14:25:44.098 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking librados2 (18.0.0-1736-ga21b2f4f-1focal) over (15.2.17-0ubuntu0.20.04.1) ... 2023-01-15T14:25:44.386 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-werkzeug. 2023-01-15T14:25:44.402 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2023-01-15T14:25:44.444 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-15T14:25:44.732 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-01-15T14:25:44.752 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:25:44.784 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libcephfs2. 2023-01-15T14:25:44.798 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-modules-core (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:44.804 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../009-libcephfs2_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:44.843 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libcephfs2 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:45.178 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-rados. 2023-01-15T14:25:46.729 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-01-15T14:25:46.886 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:46.886 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:46.886 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cryptography. 2023-01-15T14:25:46.887 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../045-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-01-15T14:25:46.888 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-15T14:25:46.888 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-openssl. 2023-01-15T14:25:46.888 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../046-python3-openssl_19.0.0-1build1_all.deb ... 2023-01-15T14:25:46.888 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-01-15T14:25:46.888 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr. 2023-01-15T14:25:46.888 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../047-ceph-mgr_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:46.889 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:46.889 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../010-python3-rados_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:46.890 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-rados (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:46.890 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-01-15T14:25:46.890 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../011-python3-ceph-argparse_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:25:46.890 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-ceph-argparse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:46.893 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-cephfs. 2023-01-15T14:25:46.893 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../012-python3-cephfs_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:46.893 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-cephfs (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:46.893 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-ceph-common. 2023-01-15T14:25:46.893 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../013-python3-ceph-common_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:25:46.893 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-ceph-common (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:46.893 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-prettytable. 2023-01-15T14:25:46.894 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../014-python3-prettytable_0.7.2-5_all.deb ... 2023-01-15T14:25:46.894 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2023-01-15T14:25:46.979 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mon. 2023-01-15T14:25:46.995 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../048-ceph-mon_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:47.037 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mon (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:47.049 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-rbd. 2023-01-15T14:25:47.067 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../015-python3-rbd_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:47.107 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-rbd (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:47.478 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2023-01-15T14:25:47.523 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../016-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2023-01-15T14:25:47.536 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-15T14:25:47.923 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package librabbitmq4:amd64. 2023-01-15T14:25:47.943 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../017-librabbitmq4_0.10.0-1_amd64.deb ... 2023-01-15T14:25:47.981 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2023-01-15T14:25:48.124 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-osd. 2023-01-15T14:25:48.143 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../049-ceph-osd_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:48.182 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-osd (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:48.309 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-01-15T14:25:48.325 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../018-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2023-01-15T14:25:48.367 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-15T14:25:48.686 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package librgw2. 2023-01-15T14:25:48.706 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../019-librgw2_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:48.744 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking librgw2 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:49.597 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-rgw. 2023-01-15T14:25:49.613 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../020-python3-rgw_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:49.655 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-rgw (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:50.033 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package liboath0. 2023-01-15T14:25:50.046 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2023-01-15T14:25:50.091 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2023-01-15T14:25:50.402 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libradosstriper1. 2023-01-15T14:25:50.418 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../022-libradosstriper1_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:50.460 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libradosstriper1 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:50.930 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-common. 2023-01-15T14:25:50.946 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph. 2023-01-15T14:25:50.950 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../023-ceph-common_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:50.964 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../050-ceph_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:50.988 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-common (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:51.004 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:51.324 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-fuse. 2023-01-15T14:25:51.339 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../051-ceph-fuse_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:51.382 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-fuse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:51.795 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mds. 2023-01-15T14:25:51.815 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../052-ceph-mds_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:51.861 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mds (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:52.398 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package cephadm. 2023-01-15T14:25:52.414 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../053-cephadm_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:52.456 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking cephadm (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:52.818 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-asyncssh. 2023-01-15T14:25:53.241 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../054-python3-asyncssh_1.12.2-1_all.deb ... 2023-01-15T14:25:53.241 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2023-01-15T14:25:53.241 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jinja2. 2023-01-15T14:25:53.246 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../055-python3-jinja2_2.10.1-2_all.deb ... 2023-01-15T14:25:53.289 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-01-15T14:25:53.584 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-01-15T14:25:53.603 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:25:53.642 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-cephadm (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:53.844 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-base. 2023-01-15T14:25:53.861 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../024-ceph-base_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:25:53.912 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-base (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:53.972 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-bcrypt. 2023-01-15T14:25:53.989 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../057-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-01-15T14:25:54.033 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-15T14:25:54.412 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-jwt. 2023-01-15T14:25:54.431 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-01-15T14:25:54.470 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-15T14:25:54.758 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-attr. 2023-01-15T14:25:54.774 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2023-01-15T14:25:54.808 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-01-15T14:25:55.103 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pluggy. 2023-01-15T14:25:55.123 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2023-01-15T14:25:55.161 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-01-15T14:25:55.282 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-cherrypy3. 2023-01-15T14:25:55.297 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../025-python3-cherrypy3_8.9.1-6_all.deb ... 2023-01-15T14:25:55.340 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2023-01-15T14:25:55.464 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-py. 2023-01-15T14:25:55.481 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-01-15T14:25:55.524 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-01-15T14:25:55.660 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-natsort. 2023-01-15T14:25:55.679 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../026-python3-natsort_7.0.1-1_all.deb ... 2023-01-15T14:25:55.718 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-natsort (7.0.1-1) ... 2023-01-15T14:25:55.820 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-atomicwrites. 2023-01-15T14:25:55.837 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-01-15T14:25:55.869 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-01-15T14:25:56.063 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-logutils. 2023-01-15T14:25:56.081 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../027-python3-logutils_0.3.3-7_all.deb ... 2023-01-15T14:25:56.121 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-logutils (0.3.3-7) ... 2023-01-15T14:25:56.173 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-packaging. 2023-01-15T14:25:56.192 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2023-01-15T14:25:56.231 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-packaging (20.3-1) ... 2023-01-15T14:25:56.424 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-markupsafe. 2023-01-15T14:25:56.443 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../028-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2023-01-15T14:25:56.482 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2023-01-15T14:25:56.560 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-wcwidth. 2023-01-15T14:25:56.577 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-01-15T14:25:56.618 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-15T14:25:56.827 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-mako. 2023-01-15T14:25:56.846 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../029-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2023-01-15T14:25:56.885 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-15T14:25:56.930 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pytest. 2023-01-15T14:25:56.946 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2023-01-15T14:25:56.988 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-01-15T14:25:57.230 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-simplegeneric. 2023-01-15T14:25:57.246 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../030-python3-simplegeneric_0.8.1-3_all.deb ... 2023-01-15T14:25:57.288 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-01-15T14:25:57.302 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-repoze.lru. 2023-01-15T14:25:57.319 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2023-01-15T14:25:57.360 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-01-15T14:25:57.607 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-singledispatch. 2023-01-15T14:25:57.621 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libjs-underscore. 2023-01-15T14:25:57.627 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../031-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2023-01-15T14:25:57.640 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-01-15T14:25:57.665 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2023-01-15T14:25:57.679 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-15T14:25:57.958 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-01-15T14:25:57.968 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-webob. 2023-01-15T14:25:57.974 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-01-15T14:25:57.987 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../032-python3-webob_1%3a1.8.5-2_all.deb ... 2023-01-15T14:25:58.015 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-15T14:25:58.026 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2023-01-15T14:25:58.320 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-routes. 2023-01-15T14:25:58.337 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2023-01-15T14:25:58.363 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-soupsieve. 2023-01-15T14:25:58.378 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../033-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2023-01-15T14:25:58.386 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-01-15T14:25:58.454 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-15T14:25:58.700 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-01-15T14:25:58.720 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:25:58.757 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-bs4. 2023-01-15T14:25:58.766 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-dashboard (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:25:58.773 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../034-python3-bs4_4.8.2-1_all.deb ... 2023-01-15T14:25:58.815 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2023-01-15T14:25:59.177 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-waitress. 2023-01-15T14:25:59.196 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../035-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2023-01-15T14:25:59.238 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-15T14:25:59.289 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-decorator. 2023-01-15T14:25:59.306 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-01-15T14:25:59.347 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-01-15T14:25:59.597 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-tempita. 2023-01-15T14:25:59.613 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../036-python3-tempita_0.5.2-6_all.deb ... 2023-01-15T14:25:59.642 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-01-15T14:25:59.655 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-tempita (0.5.2-6) ... 2023-01-15T14:25:59.658 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-01-15T14:25:59.700 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-15T14:25:59.958 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-paste. 2023-01-15T14:25:59.976 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../037-python3-paste_3.4.0+dfsg1-1_all.deb ... 2023-01-15T14:26:00.020 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-scipy. 2023-01-15T14:26:00.032 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-01-15T14:26:00.041 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2023-01-15T14:26:00.078 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-01-15T14:26:00.365 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-01-15T14:26:00.382 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../038-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2023-01-15T14:26:00.416 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2023-01-15T14:26:00.736 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-pastedeploy. 2023-01-15T14:26:00.755 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../039-python3-pastedeploy_2.1.0-1_all.deb ... 2023-01-15T14:26:00.794 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2023-01-15T14:26:01.131 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-webtest. 2023-01-15T14:26:01.146 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../040-python3-webtest_2.0.32-2_all.deb ... 2023-01-15T14:26:01.189 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-webtest (2.0.32-2) ... 2023-01-15T14:26:01.484 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-pecan. 2023-01-15T14:26:01.503 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../041-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2023-01-15T14:26:01.514 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-01-15T14:26:01.533 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-01-15T14:26:01.541 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2023-01-15T14:26:01.572 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-15T14:26:01.930 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-werkzeug. 2023-01-15T14:26:01.945 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../042-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2023-01-15T14:26:01.988 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-15T14:26:02.103 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-joblib. 2023-01-15T14:26:02.119 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2023-01-15T14:26:02.153 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-01-15T14:26:02.325 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-01-15T14:26:02.341 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../043-ceph-mgr-modules-core_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:26:02.383 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-mgr-modules-core (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:02.507 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-sklearn. 2023-01-15T14:26:02.523 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-01-15T14:26:02.565 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-15T14:26:02.771 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-01-15T14:26:02.788 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../044-libsqlite3-mod-ceph_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:02.829 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:03.158 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-cryptography. 2023-01-15T14:26:03.173 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../045-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2023-01-15T14:26:03.240 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-15T14:26:03.352 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-01-15T14:26:03.368 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:26:03.410 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:03.619 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-openssl. 2023-01-15T14:26:03.635 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../046-python3-openssl_19.0.0-1build1_all.deb ... 2023-01-15T14:26:03.678 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2023-01-15T14:26:04.047 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-mgr. 2023-01-15T14:26:04.064 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../047-ceph-mgr_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:04.105 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-mgr (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:04.584 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-mon. 2023-01-15T14:26:04.600 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../048-ceph-mon_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:04.642 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-mon (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:04.697 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-cachetools. 2023-01-15T14:26:04.716 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2023-01-15T14:26:04.763 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-01-15T14:26:05.074 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyasn1. 2023-01-15T14:26:05.093 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-01-15T14:26:05.141 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-01-15T14:26:05.471 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-01-15T14:26:05.487 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-01-15T14:26:05.529 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-15T14:26:05.804 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-osd. 2023-01-15T14:26:05.821 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../049-ceph-osd_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:05.862 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-osd (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:05.960 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-rsa. 2023-01-15T14:26:05.980 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2023-01-15T14:26:06.026 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-rsa (4.0-3) ... 2023-01-15T14:26:06.331 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-google-auth. 2023-01-15T14:26:06.349 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2023-01-15T14:26:06.389 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-01-15T14:26:06.684 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-blinker. 2023-01-15T14:26:06.704 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-01-15T14:26:06.742 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-15T14:26:07.029 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-oauthlib. 2023-01-15T14:26:07.048 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-01-15T14:26:07.088 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-15T14:26:07.392 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-01-15T14:26:08.189 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-01-15T14:26:08.189 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-15T14:26:08.189 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-websocket. 2023-01-15T14:26:08.189 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-01-15T14:26:08.189 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-01-15T14:26:08.189 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-kubernetes. 2023-01-15T14:26:08.193 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-01-15T14:26:08.233 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-01-15T14:26:08.648 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph. 2023-01-15T14:26:08.667 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../050-ceph_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:08.706 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:08.871 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-01-15T14:26:08.890 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:26:08.937 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:09.009 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-fuse. 2023-01-15T14:26:09.027 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../051-ceph-fuse_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:09.091 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-fuse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:09.267 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libonig5:amd64. 2023-01-15T14:26:09.283 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2023-01-15T14:26:09.325 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-01-15T14:26:09.521 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-mds. 2023-01-15T14:26:09.538 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../052-ceph-mds_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:09.579 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-mds (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:09.962 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libjq1:amd64. 2023-01-15T14:26:09.981 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-15T14:26:10.020 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-15T14:26:10.074 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package cephadm. 2023-01-15T14:26:10.089 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../053-cephadm_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:10.157 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking cephadm (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:10.313 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package jq. 2023-01-15T14:26:10.330 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-15T14:26:10.371 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-01-15T14:26:10.468 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-asyncssh. 2023-01-15T14:26:10.484 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../054-python3-asyncssh_1.12.2-1_all.deb ... 2023-01-15T14:26:10.526 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2023-01-15T14:26:10.633 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package socat. 2023-01-15T14:26:10.652 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2023-01-15T14:26:10.691 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking socat (1.7.3.3-2) ... 2023-01-15T14:26:10.938 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-jinja2. 2023-01-15T14:26:10.956 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../055-python3-jinja2_2.10.1-2_all.deb ... 2023-01-15T14:26:10.996 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2023-01-15T14:26:10.996 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package xmlstarlet. 2023-01-15T14:26:11.014 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-01-15T14:26:11.054 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-01-15T14:26:11.332 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-01-15T14:26:11.388 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../056-ceph-mgr-cephadm_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:26:11.390 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-mgr-cephadm (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:11.451 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-test. 2023-01-15T14:26:11.471 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../094-ceph-test_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:11.593 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-test (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:11.763 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-bcrypt. 2023-01-15T14:26:11.781 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../057-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2023-01-15T14:26:11.822 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-15T14:26:12.099 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-jwt. 2023-01-15T14:26:12.117 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../058-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2023-01-15T14:26:12.158 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-15T14:26:12.502 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-attr. 2023-01-15T14:26:12.518 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../059-python3-attr_19.3.0-2_all.deb ... 2023-01-15T14:26:12.560 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-attr (19.3.0-2) ... 2023-01-15T14:26:12.897 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-pluggy. 2023-01-15T14:26:12.913 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../060-python3-pluggy_0.13.0-2_all.deb ... 2023-01-15T14:26:12.955 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2023-01-15T14:26:13.266 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-py. 2023-01-15T14:26:13.283 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../061-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2023-01-15T14:26:13.345 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2023-01-15T14:26:14.061 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-atomicwrites. 2023-01-15T14:26:14.508 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../062-python3-atomicwrites_1.1.5-2build1_all.deb ... 2023-01-15T14:26:14.508 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2023-01-15T14:26:14.508 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-packaging. 2023-01-15T14:26:14.508 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../063-python3-packaging_20.3-1_all.deb ... 2023-01-15T14:26:14.508 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-packaging (20.3-1) ... 2023-01-15T14:26:14.792 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-wcwidth. 2023-01-15T14:26:14.825 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../064-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2023-01-15T14:26:14.850 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-15T14:26:15.196 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-pytest. 2023-01-15T14:26:15.215 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../065-python3-pytest_4.6.9-1_all.deb ... 2023-01-15T14:26:15.253 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-pytest (4.6.9-1) ... 2023-01-15T14:26:15.617 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-repoze.lru. 2023-01-15T14:26:15.635 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../066-python3-repoze.lru_0.7-2_all.deb ... 2023-01-15T14:26:15.673 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-01-15T14:26:15.909 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libjs-underscore. 2023-01-15T14:26:15.928 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../067-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2023-01-15T14:26:15.984 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-15T14:26:16.132 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package ceph-volume. 2023-01-15T14:26:16.150 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../095-ceph-volume_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:26:16.190 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking ceph-volume (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:16.270 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libjs-sphinxdoc. 2023-01-15T14:26:16.288 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../068-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2023-01-15T14:26:16.328 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-15T14:26:16.499 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package fonts-mathjax. 2023-01-15T14:26:16.518 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-15T14:26:16.557 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-15T14:26:16.664 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-routes. 2023-01-15T14:26:16.684 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../069-python3-routes_2.4.1-2_all.deb ... 2023-01-15T14:26:16.722 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-routes (2.4.1-2) ... 2023-01-15T14:26:17.042 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-01-15T14:26:17.060 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../070-ceph-mgr-dashboard_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:26:17.078 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libcephfs-dev. 2023-01-15T14:26:17.095 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../097-libcephfs-dev_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:17.100 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-mgr-dashboard (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:17.138 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libcephfs-dev (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:17.542 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package nvme-cli. 2023-01-15T14:26:17.558 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-01-15T14:26:17.571 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-decorator. 2023-01-15T14:26:17.591 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../071-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2023-01-15T14:26:17.617 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-01-15T14:26:17.629 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2023-01-15T14:26:17.940 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2023-01-15T14:26:17.957 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../072-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2023-01-15T14:26:17.982 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-01-15T14:26:17.998 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-15T14:26:17.999 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2023-01-15T14:26:18.040 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2023-01-15T14:26:18.326 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-scipy. 2023-01-15T14:26:18.344 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../073-python3-scipy_1.3.3-3build1_amd64.deb ... 2023-01-15T14:26:18.384 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2023-01-15T14:26:18.452 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package libjs-mathjax. 2023-01-15T14:26:18.483 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-15T14:26:18.510 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-15T14:26:19.752 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-sklearn-lib. 2023-01-15T14:26:19.769 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2023-01-15T14:26:19.810 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-15T14:26:20.364 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-joblib. 2023-01-15T14:26:20.383 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../075-python3-joblib_0.14.0-3_all.deb ... 2023-01-15T14:26:20.447 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-joblib (0.14.0-3) ... 2023-01-15T14:26:20.477 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python-natsort-doc. 2023-01-15T14:26:20.498 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2023-01-15T14:26:20.544 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-01-15T14:26:20.817 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-sklearn. 2023-01-15T14:26:20.833 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../076-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2023-01-15T14:26:20.875 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-15T14:26:20.915 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-webencodings. 2023-01-15T14:26:20.931 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-01-15T14:26:20.973 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-15T14:26:21.268 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-html5lib. 2023-01-15T14:26:21.286 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-01-15T14:26:21.326 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-01-15T14:26:21.643 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-01-15T14:26:21.659 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../077-ceph-mgr-diskprediction-local_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:26:21.664 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-01-15T14:26:21.682 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-01-15T14:26:21.700 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:21.723 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-15T14:26:22.184 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pastescript. 2023-01-15T14:26:22.202 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2023-01-15T14:26:22.251 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-01-15T14:26:22.596 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-psutil. 2023-01-15T14:26:22.614 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-01-15T14:26:22.655 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-01-15T14:26:22.820 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-cachetools. 2023-01-15T14:26:22.839 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../078-python3-cachetools_4.0.0-1_all.deb ... 2023-01-15T14:26:22.878 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2023-01-15T14:26:22.984 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-pyinotify. 2023-01-15T14:26:23.002 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-01-15T14:26:23.042 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-15T14:26:23.206 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-pyasn1. 2023-01-15T14:26:23.222 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../079-python3-pyasn1_0.4.2-3build1_all.deb ... 2023-01-15T14:26:23.264 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2023-01-15T14:26:23.353 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package python3-simplejson. 2023-01-15T14:26:23.371 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-01-15T14:26:23.412 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-15T14:26:23.609 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-01-15T14:26:23.628 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../080-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2023-01-15T14:26:23.667 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-15T14:26:23.675 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-01-15T14:26:23.691 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-01-15T14:26:23.733 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-15T14:26:24.055 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-rsa. 2023-01-15T14:26:24.074 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../081-python3-rsa_4.0-3_all.deb ... 2023-01-15T14:26:24.138 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-rsa (4.0-3) ... 2023-01-15T14:26:24.437 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package radosgw. 2023-01-15T14:26:24.450 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-google-auth. 2023-01-15T14:26:24.455 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../110-radosgw_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:24.466 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../082-python3-google-auth_1.5.1-2_all.deb ... 2023-01-15T14:26:24.495 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking radosgw (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:24.499 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2023-01-15T14:26:24.870 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-blinker. 2023-01-15T14:26:24.886 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../083-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2023-01-15T14:26:24.928 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-15T14:26:25.281 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-oauthlib. 2023-01-15T14:26:25.298 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../084-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2023-01-15T14:26:25.339 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-15T14:26:25.659 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-01-15T14:26:25.677 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../085-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2023-01-15T14:26:25.737 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-15T14:26:25.883 INFO:teuthology.orchestra.run.smithi090.stdout:Selecting previously unselected package rbd-fuse. 2023-01-15T14:26:25.901 INFO:teuthology.orchestra.run.smithi090.stdout:Preparing to unpack .../111-rbd-fuse_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:25.950 INFO:teuthology.orchestra.run.smithi090.stdout:Unpacking rbd-fuse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:26.045 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-websocket. 2023-01-15T14:26:26.061 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../086-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2023-01-15T14:26:26.103 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2023-01-15T14:26:26.297 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-attr (19.3.0-2) ... 2023-01-15T14:26:26.448 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-kubernetes. 2023-01-15T14:26:26.465 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../087-python3-kubernetes_7.0.0~a1-2_all.deb ... 2023-01-15T14:26:26.506 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2023-01-15T14:26:26.623 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-01-15T14:26:26.749 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-15T14:26:26.877 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-01-15T14:26:27.062 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up cephadm (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:27.103 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-01-15T14:26:27.121 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../088-ceph-mgr-k8sevents_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:26:27.161 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:27.497 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libonig5:amd64. 2023-01-15T14:26:27.515 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../089-libonig5_6.9.4-1_amd64.deb ... 2023-01-15T14:26:27.555 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2023-01-15T14:26:27.673 INFO:teuthology.orchestra.run.smithi090.stdout:Adding system user cephadm....done 2023-01-15T14:26:27.836 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-15T14:26:27.957 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libjq1:amd64. 2023-01-15T14:26:27.962 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-15T14:26:27.973 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../090-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-15T14:26:28.015 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-15T14:26:28.293 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package jq. 2023-01-15T14:26:28.310 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../091-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2023-01-15T14:26:28.323 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-01-15T14:26:28.351 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2023-01-15T14:26:28.616 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-15T14:26:28.637 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package socat. 2023-01-15T14:26:28.655 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../092-socat_1.7.3.3-2_amd64.deb ... 2023-01-15T14:26:28.695 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking socat (1.7.3.3-2) ... 2023-01-15T14:26:28.742 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-01-15T14:26:29.574 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-01-15T14:26:29.574 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-01-15T14:26:29.574 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package xmlstarlet. 2023-01-15T14:26:29.575 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../093-xmlstarlet_1.6.1-2build1_amd64.deb ... 2023-01-15T14:26:29.575 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2023-01-15T14:26:29.575 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-test. 2023-01-15T14:26:29.575 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../094-ceph-test_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:29.575 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-test (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:29.631 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-15T14:26:29.957 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-01-15T14:26:30.259 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-15T14:26:30.551 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-15T14:26:30.853 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-ceph-argparse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:31.155 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-15T14:26:31.281 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-01-15T14:26:31.574 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-15T14:26:31.858 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-15T14:26:31.984 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-01-15T14:26:32.361 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-01-15T14:26:32.654 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-01-15T14:26:32.940 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-15T14:26:33.266 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-01-15T14:26:33.542 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-01-15T14:26:33.910 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-01-15T14:26:34.194 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-packaging (20.3-1) ... 2023-01-15T14:26:34.275 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package ceph-volume. 2023-01-15T14:26:34.295 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../095-ceph-volume_18.0.0-1736-ga21b2f4f-1focal_all.deb ... 2023-01-15T14:26:34.333 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking ceph-volume (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:34.504 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-01-15T14:26:34.630 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-15T14:26:34.648 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package fonts-mathjax. 2023-01-15T14:26:34.667 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../096-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-15T14:26:34.706 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-15T14:26:34.757 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-15T14:26:35.042 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-01-15T14:26:35.219 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libcephfs-dev. 2023-01-15T14:26:35.967 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-15T14:26:35.967 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-15T14:26:35.967 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../097-libcephfs-dev_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:35.967 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libcephfs-dev (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:35.967 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package nvme-cli. 2023-01-15T14:26:35.967 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../098-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2023-01-15T14:26:35.968 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2023-01-15T14:26:36.093 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-01-15T14:26:36.172 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../099-python-asyncssh-doc_1.12.2-1_all.deb ... 2023-01-15T14:26:36.173 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2023-01-15T14:26:36.206 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up socat (1.7.3.3-2) ... 2023-01-15T14:26:36.578 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-ceph-common (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:36.578 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package libjs-mathjax. 2023-01-15T14:26:36.578 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../100-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2023-01-15T14:26:36.596 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-15T14:26:36.651 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:26:36.786 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-01-15T14:26:37.129 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-15T14:26:37.255 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-01-15T14:26:37.415 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-01-15T14:26:37.691 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-15T14:26:38.010 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:26:38.136 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-15T14:26:38.262 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-01-15T14:26:38.385 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python-natsort-doc. 2023-01-15T14:26:38.403 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../101-python-natsort-doc_7.0.1-1_all.deb ... 2023-01-15T14:26:38.443 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2023-01-15T14:26:38.589 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-01-15T14:26:38.779 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-webencodings. 2023-01-15T14:26:38.797 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../102-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2023-01-15T14:26:38.837 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-15T14:26:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-01-15T14:26:39.165 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-html5lib. 2023-01-15T14:26:39.174 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-15T14:26:39.182 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../103-python3-html5lib_1.0.1-2_all.deb ... 2023-01-15T14:26:39.223 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2023-01-15T14:26:39.326 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-01-15T14:26:39.610 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-lxml:amd64. 2023-01-15T14:26:39.611 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-01-15T14:26:39.627 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../104-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2023-01-15T14:26:39.668 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-15T14:26:39.905 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-15T14:26:40.171 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-pastescript. 2023-01-15T14:26:40.189 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-01-15T14:26:40.189 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../105-python3-pastescript_2.0.2-4_all.deb ... 2023-01-15T14:26:40.229 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-01-15T14:26:40.583 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-psutil. 2023-01-15T14:26:40.600 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../106-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2023-01-15T14:26:40.642 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2023-01-15T14:26:40.994 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-pyinotify. 2023-01-15T14:26:41.003 INFO:teuthology.orchestra.run.smithi090.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-01-15T14:26:41.011 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2023-01-15T14:26:41.052 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-15T14:26:41.258 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-01-15T14:26:41.348 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package python3-simplejson. 2023-01-15T14:26:41.369 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../108-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2023-01-15T14:26:41.385 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-15T14:26:41.491 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-15T14:26:41.708 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-01-15T14:26:41.712 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-01-15T14:26:41.726 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../109-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2023-01-15T14:26:41.860 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-15T14:26:42.081 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-15T14:26:42.206 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-15T14:26:42.333 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-01-15T14:26:42.479 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package radosgw. 2023-01-15T14:26:42.506 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../110-radosgw_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:42.537 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking radosgw (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:42.667 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-15T14:26:42.985 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-01-15T14:26:43.948 INFO:teuthology.orchestra.run.smithi141.stdout:Selecting previously unselected package rbd-fuse. 2023-01-15T14:26:43.968 INFO:teuthology.orchestra.run.smithi141.stdout:Preparing to unpack .../111-rbd-fuse_18.0.0-1736-ga21b2f4f-1focal_amd64.deb ... 2023-01-15T14:26:44.008 INFO:teuthology.orchestra.run.smithi141.stdout:Unpacking rbd-fuse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:44.324 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-attr (19.3.0-2) ... 2023-01-15T14:26:44.476 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-15T14:26:44.602 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-01-15T14:26:44.637 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2023-01-15T14:26:44.772 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2023-01-15T14:26:44.979 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-01-15T14:26:45.239 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2023-01-15T14:26:45.298 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-01-15T14:26:45.466 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up cephadm (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:45.599 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:26:45.717 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-01-15T14:26:46.043 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-15T14:26:46.127 INFO:teuthology.orchestra.run.smithi141.stdout:Adding system user cephadm....done 2023-01-15T14:26:46.289 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2023-01-15T14:26:46.370 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2023-01-15T14:26:46.431 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2023-01-15T14:26:46.755 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-15T14:26:46.783 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-01-15T14:26:47.076 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2023-01-15T14:26:47.099 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-15T14:26:47.202 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up liboath0 (2.6.1-1.3) ... 2023-01-15T14:26:47.345 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2023-01-15T14:26:47.672 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-joblib (0.14.0-3) ... 2023-01-15T14:26:48.081 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2023-01-15T14:26:48.172 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-15T14:26:48.298 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-01-15T14:26:48.391 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-cachetools (4.0.0-1) ... 2023-01-15T14:26:48.425 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-01-15T14:26:48.568 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rsa (4.0-3) ... 2023-01-15T14:26:48.700 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2023-01-15T14:26:48.861 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-15T14:26:48.987 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-01-15T14:26:48.993 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2023-01-15T14:26:49.280 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-15T14:26:49.319 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-ceph-argparse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:49.581 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-01-15T14:26:49.612 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2023-01-15T14:26:49.738 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2023-01-15T14:26:49.908 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-01-15T14:26:50.014 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2023-01-15T14:26:50.710 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2023-01-15T14:26:50.710 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2023-01-15T14:26:50.711 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librados2 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:50.711 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2023-01-15T14:26:50.711 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librgw2 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:50.711 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libsqlite3-mod-ceph (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:50.825 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-natsort (7.0.1-1) ... 2023-01-15T14:26:50.918 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libcephfs2 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:51.012 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-routes (2.4.1-2) ... 2023-01-15T14:26:51.110 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2023-01-15T14:26:51.314 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libradosstriper1 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:51.428 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2023-01-15T14:26:51.533 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-01-15T14:26:51.733 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up librbd1 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:51.787 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2023-01-15T14:26:51.936 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-modules-core (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:51.985 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-fuse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:52.117 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2023-01-15T14:26:52.208 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-15T14:26:52.208 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-15T14:26:52.456 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-tempita (0.5.2-6) ... 2023-01-15T14:26:52.647 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up libcephfs-dev (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:52.773 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-packaging (20.3-1) ... 2023-01-15T14:26:52.839 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rados (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:52.991 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-01-15T14:26:53.083 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2023-01-15T14:26:53.209 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2023-01-15T14:26:53.360 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2023-01-15T14:26:53.645 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2023-01-15T14:26:53.939 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rbd (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:54.038 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2023-01-15T14:26:54.065 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up rbd-fuse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:54.225 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-rgw (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:54.350 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up python3-cephfs (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:54.398 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2023-01-15T14:26:54.477 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-common (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:54.766 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up socat (1.7.3.3-2) ... 2023-01-15T14:26:54.917 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-ceph-common (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:55.039 INFO:teuthology.orchestra.run.smithi090.stdout:Adding group ceph....done 2023-01-15T14:26:55.244 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:26:55.388 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-html5lib (1.0.1-2) ... 2023-01-15T14:26:55.463 INFO:teuthology.orchestra.run.smithi090.stdout:Adding system user ceph....done 2023-01-15T14:26:55.601 INFO:teuthology.orchestra.run.smithi090.stdout:Setting system user ceph properties....done 2023-01-15T14:26:55.627 INFO:teuthology.orchestra.run.smithi090.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-01-15T14:26:55.729 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2023-01-15T14:26:55.776 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-01-15T14:26:55.855 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2023-01-15T14:26:56.014 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-pluggy (0.13.0-2) ... 2023-01-15T14:26:56.107 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-01-15T14:26:56.308 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2023-01-15T14:26:56.599 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-test (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:56.651 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:26:56.725 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up radosgw (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:58.285 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-15T14:26:58.286 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-15T14:26:58.286 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-base (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:58.286 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-01-15T14:26:58.286 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2023-01-15T14:26:58.286 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2023-01-15T14:26:58.286 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2023-01-15T14:26:58.287 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-logutils (0.3.3-7) ... 2023-01-15T14:26:58.287 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2023-01-15T14:26:58.287 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-01-15T14:26:58.287 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-prettytable (0.7.2-5) ... 2023-01-15T14:26:58.518 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2023-01-15T14:26:58.604 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mds (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-15T14:26:58.763 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-15T14:26:58.843 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2023-01-15T14:26:59.197 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:59.253 INFO:teuthology.orchestra.run.smithi141.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2023-01-15T14:26:59.420 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-15T14:26:59.420 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-15T14:26:59.488 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2023-01-15T14:26:59.647 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2023-01-15T14:26:59.858 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-osd (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:26:59.965 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2023-01-15T14:27:00.325 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2023-01-15T14:27:00.384 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-15T14:27:00.385 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-15T14:27:00.459 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2023-01-15T14:27:00.585 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-webob (1:1.8.5-2) ... 2023-01-15T14:27:00.827 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-k8sevents (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:00.936 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2023-01-15T14:27:00.955 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:01.088 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mon (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:01.246 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2023-01-15T14:27:01.306 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-15T14:27:01.306 INFO:teuthology.orchestra.run.smithi090.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-15T14:27:01.748 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-cephadm (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:01.892 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:02.019 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-mgr-dashboard (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:02.161 INFO:teuthology.orchestra.run.smithi090.stdout:Setting up ceph-volume (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:02.280 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-01-15T14:27:02.441 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-01-15T14:27:02.710 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2023-01-15T14:27:02.733 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-01-15T14:27:02.886 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-pytest (4.6.9-1) ... 2023-01-15T14:27:03.254 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2023-01-15T14:27:03.563 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2023-01-15T14:27:03.848 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2023-01-15T14:27:03.999 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-bs4 (4.8.2-1) ... 2023-01-15T14:27:04.350 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2023-01-15T14:27:04.702 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2023-01-15T14:27:05.087 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2023-01-15T14:27:05.421 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2023-01-15T14:27:05.621 INFO:teuthology.orchestra.run.smithi090.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-01-15T14:27:06.198 INFO:teuthology.orchestra.run.smithi090.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-01-15T14:27:06.485 DEBUG:teuthology.parallel:result is None 2023-01-15T14:27:06.487 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2023-01-15T14:27:06.621 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2023-01-15T14:27:06.747 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2023-01-15T14:27:06.873 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-rsa (4.0-3) ... 2023-01-15T14:27:07.182 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2023-01-15T14:27:07.308 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-webtest (2.0.32-2) ... 2023-01-15T14:27:07.618 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2023-01-15T14:27:07.911 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-01-15T14:27:08.212 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2023-01-15T14:27:08.598 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up librados2 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:10.371 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2023-01-15T14:27:10.371 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up librgw2 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:10.371 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libsqlite3-mod-ceph (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:10.371 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libcephfs2 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:10.371 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-routes (2.4.1-2) ... 2023-01-15T14:27:10.371 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libradosstriper1 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:10.372 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-google-auth (1.5.1-2) ... 2023-01-15T14:27:10.372 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up librbd1 (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:10.372 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-mgr-modules-core (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:10.394 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-fuse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:10.615 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-15T14:27:10.785 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-01-15T14:27:11.054 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up libcephfs-dev (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:12.238 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-rados (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:12.238 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2023-01-15T14:27:12.260 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-rbd (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:12.411 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up rbd-fuse (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:12.537 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-rgw (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:12.662 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up python3-cephfs (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:12.788 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-common (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:13.853 INFO:teuthology.orchestra.run.smithi141.stdout:Adding group ceph....done 2023-01-15T14:27:13.853 INFO:teuthology.orchestra.run.smithi141.stdout:Adding system user ceph....done 2023-01-15T14:27:13.885 INFO:teuthology.orchestra.run.smithi141.stdout:Setting system user ceph properties....done 2023-01-15T14:27:13.987 INFO:teuthology.orchestra.run.smithi141.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-01-15T14:27:14.054 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-01-15T14:27:14.370 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-01-15T14:27:14.823 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-test (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:14.974 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up radosgw (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:15.258 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-15T14:27:15.270 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-01-15T14:27:16.223 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-base (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:16.596 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-01-15T14:27:17.045 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-mds (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:17.276 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-15T14:27:17.354 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-01-15T14:27:17.755 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-mgr (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:18.005 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-15T14:27:18.015 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-01-15T14:27:18.440 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-osd (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:18.983 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-15T14:27:20.126 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-01-15T14:27:20.127 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-mgr-k8sevents (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:20.128 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:20.128 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-mon (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:20.128 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-15T14:27:20.128 INFO:teuthology.orchestra.run.smithi141.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-01-15T14:27:20.370 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-mgr-cephadm (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:20.513 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:20.640 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-mgr-dashboard (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:20.782 INFO:teuthology.orchestra.run.smithi141.stdout:Setting up ceph-volume (18.0.0-1736-ga21b2f4f-1focal) ... 2023-01-15T14:27:20.942 INFO:teuthology.orchestra.run.smithi141.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2023-01-15T14:27:21.094 INFO:teuthology.orchestra.run.smithi141.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2023-01-15T14:27:21.395 INFO:teuthology.orchestra.run.smithi141.stdout:Processing triggers for man-db (2.9.1-1) ... 2023-01-15T14:27:24.331 INFO:teuthology.orchestra.run.smithi141.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2023-01-15T14:27:24.922 INFO:teuthology.orchestra.run.smithi141.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-01-15T14:27:25.118 DEBUG:teuthology.parallel:result is None 2023-01-15T14:27:25.118 DEBUG:teuthology.orchestra.run.smithi090:> sudo lsb_release -is 2023-01-15T14:27:25.169 INFO:teuthology.orchestra.run.smithi090.stdout:Ubuntu 2023-01-15T14:27:25.326 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-15T14:27:25.328 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=a21b2f4fd40d883dac62c8cd315cbb54848ee067 2023-01-15T14:27:25.476 DEBUG:teuthology.orchestra.run.smithi090:> dpkg-query -W -f '${Version}' ceph 2023-01-15T14:27:25.508 INFO:teuthology.orchestra.run.smithi090.stdout:18.0.0-1736-ga21b2f4f-1focal 2023-01-15T14:27:25.636 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1736-ga21b2f4f-1focal 2023-01-15T14:27:25.636 INFO:teuthology.task.install:The correct ceph version 18.0.0-1736-ga21b2f4f-1focal is installed. 2023-01-15T14:27:25.637 DEBUG:teuthology.orchestra.run.smithi141:> sudo lsb_release -is 2023-01-15T14:27:25.690 INFO:teuthology.orchestra.run.smithi141.stdout:Ubuntu 2023-01-15T14:27:25.807 DEBUG:teuthology.misc:System to be installed: Ubuntu 2023-01-15T14:27:25.808 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=a21b2f4fd40d883dac62c8cd315cbb54848ee067 2023-01-15T14:27:25.955 DEBUG:teuthology.orchestra.run.smithi141:> dpkg-query -W -f '${Version}' ceph 2023-01-15T14:27:25.982 INFO:teuthology.orchestra.run.smithi141.stdout:18.0.0-1736-ga21b2f4f-1focal 2023-01-15T14:27:25.982 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-1736-ga21b2f4f-1focal 2023-01-15T14:27:25.982 INFO:teuthology.task.install:The correct ceph version 18.0.0-1736-ga21b2f4f-1focal is installed. 2023-01-15T14:27:25.983 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-01-15T14:27:25.983 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:25.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-15T14:27:26.000 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:26.127 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-01-15T14:27:26.144 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-01-15T14:27:26.144 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:26.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/daemon-helper 2023-01-15T14:27:26.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-15T14:27:26.302 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:26.460 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/usr/bin/daemon-helper 2023-01-15T14:27:26.476 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-01-15T14:27:26.622 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-01-15T14:27:26.622 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:26.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-15T14:27:26.638 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-15T14:27:26.779 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:26.939 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/usr/bin/adjust-ulimits 2023-01-15T14:27:26.955 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-01-15T14:27:27.114 INFO:teuthology.run_tasks:Running task ceph... 2023-01-15T14:27:27.182 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-01-15T14:27:27.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 777 /var/log/ceph 2023-01-15T14:27:27.265 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 777 /var/log/ceph 2023-01-15T14:27:27.409 INFO:tasks.ceph:Disabling ceph logrotate... 2023-01-15T14:27:27.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-15T14:27:27.437 DEBUG:teuthology.orchestra.run.smithi141:> sudo rm -f -- /etc/logrotate.d/ceph 2023-01-15T14:27:27.469 INFO:tasks.ceph:Creating extra log directories... 2023-01-15T14:27:27.470 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-15T14:27:27.472 DEBUG:teuthology.orchestra.run.smithi141:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-01-15T14:27:27.502 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-01-15T14:27:27.572 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore odsync write': True, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd sloppy crc': True, 'osd snap trim sleep': 0.5}, 'global': {'mon election default strategy': 3, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-01-15T14:27:27.572 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi/7137228', 'branch': 'wip-rf-newsched-v3', 'description': 'rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/pool-snaps-few-objects}', 'email': None, 'first_in_suite': False, 'job_id': '7137228', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rfriedma-2023-01-12_16:28:35-rados:thrash-wip-rf-newsched-v3-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'override': {'conf': {'osd': {'osd deep scrub update digest min age': 0}}}, 'overrides': {'admin_socket': {'branch': 'wip-rf-newsched-v3'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore odsync write': True, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd sloppy crc': True, 'osd snap trim sleep': 0.5}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': 'a21b2f4fd40d883dac62c8cd315cbb54848ee067'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a21b2f4fd40d883dac62c8cd315cbb54848ee067'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-rf-newsched-v3', 'sha1': 'a21b2f4fd40d883dac62c8cd315cbb54848ee067'}}, 'owner': 'scheduled_rfriedma@teuthology', 'priority': 95, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': 'a21b2f4fd40d883dac62c8cd315cbb54848ee067', 'sleep_before_teardown': 0, 'suite': 'rados:thrash', 'suite_branch': 'wip-rf-newsched-v3', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_a21b2f4fd40d883dac62c8cd315cbb54848ee067/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'a21b2f4fd40d883dac62c8cd315cbb54848ee067', 'targets': {'smithi090.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi141.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwINnmi+N3LgR9ET61cABYBnDJDAAQE1klaPINljTOvitiIM3/5ICam+qurn41RkfyLPz4uwE1yDyHeI7QBKChMImagaJPfltOSsFsTn67ifp6uunrBJblDzycEd7/lJfdMwBn8dApfOw5SSnVdpzpaKWGNpPqshrinViULNhTz2PMAkza/R7qM6v4qqKKxQx/yVi8/Lt03nxO672k5WH/Wkz7KHX4nwyeTelksSojH9SBt4yR4WpLbMjuFX33Ag14KBWr9YQ30L5/h7VtBLQL+2c7sq3Bnll5zj8H+sIa0hOHmwR5sxs4cVs6Eb4Cblr6xTs0Q9vQefWDDYNL4qRpuj1NO+bW1xGcALK9TujCJ0/fdcMLQ0/r4QukVxSE1j6frwWwo/u8lxTTp/wbNVU14w+37ppSvzcPbOD5FNnCu4PMHgrtqp02ggncjqMYpHvxXoQ53JPn84XfXX04vWf5emRMQ76T0NvWW0jYtj7x7ex7l/iEGUGHVnaOmjLqij0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore odsync write': True, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd sloppy crc': True, 'osd snap trim sleep': 0.5}, 'global': {'mon election default strategy': 3, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': 'a21b2f4fd40d883dac62c8cd315cbb54848ee067', 'cluster': 'ceph'}}, {'thrashosds': {'chance_pgnum_grow': 2, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000, 'pool_snaps': True}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '19d18db866afdc31fd6586f558fc29b95a87ccfb', 'timestamp': '2023-01-12_16:28:35', 'tube': 'smithi', 'user': 'rfriedma', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1670'} 2023-01-15T14:27:27.573 DEBUG:teuthology.orchestra.run.smithi090:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-15T14:27:27.578 DEBUG:teuthology.orchestra.run.smithi141:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-01-15T14:27:27.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo install -d -m0777 -- /var/run/ceph 2023-01-15T14:27:27.625 DEBUG:teuthology.orchestra.run.smithi141:> sudo install -d -m0777 -- /var/run/ceph 2023-01-15T14:27:27.753 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:27.866 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/scratch_devs of=/dev/stdout 2023-01-15T14:27:27.873 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-15T14:27:27.874 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_1 2023-01-15T14:27:27.881 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-15T14:27:27.882 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-15T14:27:27.882 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 359 Links: 1 2023-01-15T14:27:27.882 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-15T14:27:27.882 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-01-15 14:23:33.674410332 +0000 2023-01-15T14:27:27.882 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-01-15 14:23:33.354417713 +0000 2023-01-15T14:27:27.883 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-01-15 14:23:33.354417713 +0000 2023-01-15T14:27:27.883 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-01-15T14:27:27.883 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-15T14:27:27.934 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-01-15T14:27:27.935 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-01-15T14:27:27.935 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000310732 s, 1.6 MB/s 2023-01-15T14:27:27.936 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-15T14:27:27.945 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_2 2023-01-15T14:27:27.952 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-15T14:27:27.952 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-15T14:27:27.952 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 364 Links: 1 2023-01-15T14:27:27.952 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-15T14:27:27.952 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-01-15 14:23:34.222397691 +0000 2023-01-15T14:27:27.952 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-01-15 14:23:33.878405625 +0000 2023-01-15T14:27:27.952 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-01-15 14:23:33.878405625 +0000 2023-01-15T14:27:27.953 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-01-15T14:27:27.953 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-15T14:27:27.965 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-01-15T14:27:27.965 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-01-15T14:27:27.966 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000342985 s, 1.5 MB/s 2023-01-15T14:27:27.967 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-15T14:27:27.974 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_3 2023-01-15T14:27:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-15T14:27:28.060 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-15T14:27:28.060 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 370 Links: 1 2023-01-15T14:27:28.061 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-15T14:27:28.061 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-01-15 14:23:34.742385696 +0000 2023-01-15T14:27:28.061 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-01-15 14:23:34.446392523 +0000 2023-01-15T14:27:28.061 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-01-15 14:23:34.446392523 +0000 2023-01-15T14:27:28.061 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-01-15T14:27:28.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-15T14:27:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-01-15T14:27:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-01-15T14:27:28.117 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000398278 s, 1.3 MB/s 2023-01-15T14:27:28.118 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-15T14:27:28.126 DEBUG:teuthology.orchestra.run.smithi090:> stat /dev/vg_nvme/lv_4 2023-01-15T14:27:28.132 INFO:teuthology.orchestra.run.smithi090.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-15T14:27:28.132 INFO:teuthology.orchestra.run.smithi090.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-15T14:27:28.132 INFO:teuthology.orchestra.run.smithi090.stdout:Device: 6h/6d Inode: 379 Links: 1 2023-01-15T14:27:28.132 INFO:teuthology.orchestra.run.smithi090.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-15T14:27:28.132 INFO:teuthology.orchestra.run.smithi090.stdout:Access: 2023-01-15 14:23:35.254373886 +0000 2023-01-15T14:27:28.132 INFO:teuthology.orchestra.run.smithi090.stdout:Modify: 2023-01-15 14:23:34.946380991 +0000 2023-01-15T14:27:28.132 INFO:teuthology.orchestra.run.smithi090.stdout:Change: 2023-01-15 14:23:34.946380991 +0000 2023-01-15T14:27:28.132 INFO:teuthology.orchestra.run.smithi090.stdout: Birth: - 2023-01-15T14:27:28.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-15T14:27:28.189 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records in 2023-01-15T14:27:28.189 INFO:teuthology.orchestra.run.smithi090.stderr:1+0 records out 2023-01-15T14:27:28.190 INFO:teuthology.orchestra.run.smithi090.stderr:512 bytes copied, 0.000412868 s, 1.2 MB/s 2023-01-15T14:27:28.191 DEBUG:teuthology.orchestra.run.smithi090:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-15T14:27:28.199 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-15T14:27:28.199 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:28.199 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/scratch_devs of=/dev/stdout 2023-01-15T14:27:28.206 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-01-15T14:27:28.206 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_1 2023-01-15T14:27:28.213 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-01-15T14:27:28.213 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-15T14:27:28.214 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 360 Links: 1 2023-01-15T14:27:28.214 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-15T14:27:28.214 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2023-01-15 14:23:25.652226101 +0000 2023-01-15T14:27:28.214 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2023-01-15 14:23:25.360232821 +0000 2023-01-15T14:27:28.214 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2023-01-15 14:23:25.360232821 +0000 2023-01-15T14:27:28.214 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2023-01-15T14:27:28.214 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-01-15T14:27:28.272 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2023-01-15T14:27:28.272 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2023-01-15T14:27:28.272 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000390408 s, 1.3 MB/s 2023-01-15T14:27:28.273 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-01-15T14:27:28.281 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_2 2023-01-15T14:27:28.287 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-01-15T14:27:28.287 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-15T14:27:28.287 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 365 Links: 1 2023-01-15T14:27:28.287 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-15T14:27:28.288 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2023-01-15 14:23:26.184213859 +0000 2023-01-15T14:27:28.288 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2023-01-15 14:23:25.888220670 +0000 2023-01-15T14:27:28.288 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2023-01-15 14:23:25.888220670 +0000 2023-01-15T14:27:28.288 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2023-01-15T14:27:28.288 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-01-15T14:27:28.344 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2023-01-15T14:27:28.344 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2023-01-15T14:27:28.344 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000404461 s, 1.3 MB/s 2023-01-15T14:27:28.345 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-01-15T14:27:28.352 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_3 2023-01-15T14:27:28.359 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-01-15T14:27:28.359 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-15T14:27:28.359 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 371 Links: 1 2023-01-15T14:27:28.360 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-15T14:27:28.360 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2023-01-15 14:23:26.716201616 +0000 2023-01-15T14:27:28.360 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2023-01-15 14:23:26.420208427 +0000 2023-01-15T14:27:28.360 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2023-01-15 14:23:26.420208427 +0000 2023-01-15T14:27:28.360 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2023-01-15T14:27:28.360 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-01-15T14:27:28.415 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2023-01-15T14:27:28.416 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2023-01-15T14:27:28.416 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000396541 s, 1.3 MB/s 2023-01-15T14:27:28.417 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-01-15T14:27:28.426 DEBUG:teuthology.orchestra.run.smithi141:> stat /dev/vg_nvme/lv_4 2023-01-15T14:27:28.432 INFO:teuthology.orchestra.run.smithi141.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-01-15T14:27:28.433 INFO:teuthology.orchestra.run.smithi141.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-01-15T14:27:28.433 INFO:teuthology.orchestra.run.smithi141.stdout:Device: 6h/6d Inode: 377 Links: 1 2023-01-15T14:27:28.433 INFO:teuthology.orchestra.run.smithi141.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-01-15T14:27:28.433 INFO:teuthology.orchestra.run.smithi141.stdout:Access: 2023-01-15 14:23:27.256189189 +0000 2023-01-15T14:27:28.433 INFO:teuthology.orchestra.run.smithi141.stdout:Modify: 2023-01-15 14:23:26.964195909 +0000 2023-01-15T14:27:28.433 INFO:teuthology.orchestra.run.smithi141.stdout:Change: 2023-01-15 14:23:26.964195909 +0000 2023-01-15T14:27:28.433 INFO:teuthology.orchestra.run.smithi141.stdout: Birth: - 2023-01-15T14:27:28.433 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-01-15T14:27:28.488 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records in 2023-01-15T14:27:28.488 INFO:teuthology.orchestra.run.smithi141.stderr:1+0 records out 2023-01-15T14:27:28.488 INFO:teuthology.orchestra.run.smithi141.stderr:512 bytes copied, 0.000415208 s, 1.2 MB/s 2023-01-15T14:27:28.489 DEBUG:teuthology.orchestra.run.smithi141:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-01-15T14:27:28.497 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-15T14:27:28.497 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi141.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-01-15T14:27:28.497 INFO:tasks.ceph:Generating config... 2023-01-15T14:27:28.499 INFO:tasks.ceph:[osd] debug monc = 20 2023-01-15T14:27:28.499 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-01-15T14:27:28.499 INFO:tasks.ceph:[osd] debug ms = 1 2023-01-15T14:27:28.499 INFO:tasks.ceph:[osd] debug osd = 20 2023-01-15T14:27:28.499 INFO:tasks.ceph:[osd] filestore odsync write = True 2023-01-15T14:27:28.499 INFO:tasks.ceph:[osd] osd backoff on degraded = True 2023-01-15T14:27:28.499 INFO:tasks.ceph:[osd] osd backoff on peering = True 2023-01-15T14:27:28.499 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-01-15T14:27:28.499 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-01-15T14:27:28.500 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-01-15T14:27:28.500 INFO:tasks.ceph:[osd] osd max backfills = 2 2023-01-15T14:27:28.500 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-01-15T14:27:28.500 INFO:tasks.ceph:[osd] osd max scrubs = 3 2023-01-15T14:27:28.500 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-01-15T14:27:28.500 INFO:tasks.ceph:[osd] osd objectstore = filestore 2023-01-15T14:27:28.500 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-01-15T14:27:28.500 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-01-15T14:27:28.500 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2023-01-15T14:27:28.500 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2023-01-15T14:27:28.500 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-01-15T14:27:28.501 INFO:tasks.ceph:[osd] osd sloppy crc = True 2023-01-15T14:27:28.501 INFO:tasks.ceph:[osd] osd snap trim sleep = 0.5 2023-01-15T14:27:28.501 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-01-15T14:27:28.501 INFO:tasks.ceph:[global] ms type = async 2023-01-15T14:27:28.501 INFO:tasks.ceph:[global] osd debug inject dispatch delay duration = 0.1 2023-01-15T14:27:28.501 INFO:tasks.ceph:[global] osd debug inject dispatch delay probability = 0.1 2023-01-15T14:27:28.501 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2023-01-15T14:27:28.501 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2023-01-15T14:27:28.501 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-01-15T14:27:28.501 INFO:tasks.ceph:[mgr] debug ms = 1 2023-01-15T14:27:28.501 INFO:tasks.ceph:[mon] debug mon = 20 2023-01-15T14:27:28.502 INFO:tasks.ceph:[mon] debug ms = 1 2023-01-15T14:27:28.502 INFO:tasks.ceph:[mon] debug paxos = 20 2023-01-15T14:27:28.502 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-01-15T14:27:28.502 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-01-15T14:27:28.502 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-01-15T14:27:28.502 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-01-15T14:27:28.502 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-01-15T14:27:28.502 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-01-15T14:27:28.502 INFO:tasks.ceph:Setting up mon.a... 2023-01-15T14:27:28.503 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-01-15T14:27:28.542 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.keyring 2023-01-15T14:27:28.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-01-15T14:27:28.635 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-15T14:27:28.651 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.90'), ('mon.c', '[v2:172.21.15.90:3301,v1:172.21.15.90:6790]'), ('mon.b', '172.21.15.141')] 2023-01-15T14:27:28.651 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.90,[v2:172.21.15.90:3301,v1:172.21.15.90:6790],172.21.15.141', 'mon election default strategy': 3, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore odsync write': True, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd sloppy crc': True, 'osd snap trim sleep': 0.5}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-01-15T14:27:28.702 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:28.703 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-01-15T14:27:28.710 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.90 --addv c '[v2:172.21.15.90:3301,v1:172.21.15.90:6790]' --add b 172.21.15.141 --print /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:28.797 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:28.825 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: generated fsid 1c77fb2b-d1a7-4884-a24e-3935e16d0b6b 2023-01-15T14:27:28.825 INFO:teuthology.orchestra.run.smithi090.stdout:setting min_mon_release = pacific 2023-01-15T14:27:28.825 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 0 2023-01-15T14:27:28.825 INFO:teuthology.orchestra.run.smithi090.stdout:fsid 1c77fb2b-d1a7-4884-a24e-3935e16d0b6b 2023-01-15T14:27:28.825 INFO:teuthology.orchestra.run.smithi090.stdout:last_changed 2023-01-15T14:27:28.795717+0000 2023-01-15T14:27:28.825 INFO:teuthology.orchestra.run.smithi090.stdout:created 2023-01-15T14:27:28.795717+0000 2023-01-15T14:27:28.826 INFO:teuthology.orchestra.run.smithi090.stdout:min_mon_release 16 (pacific) 2023-01-15T14:27:28.826 INFO:teuthology.orchestra.run.smithi090.stdout:election_strategy: 3 2023-01-15T14:27:28.826 INFO:teuthology.orchestra.run.smithi090.stdout:0: [v2:172.21.15.90:3300/0,v1:172.21.15.90:6789/0] mon.a 2023-01-15T14:27:28.826 INFO:teuthology.orchestra.run.smithi090.stdout:1: [v2:172.21.15.141:3300/0,v1:172.21.15.141:6789/0] mon.b 2023-01-15T14:27:28.827 INFO:teuthology.orchestra.run.smithi090.stdout:2: [v2:172.21.15.90:3301/0,v1:172.21.15.90:6790/0] mon.c 2023-01-15T14:27:28.827 INFO:teuthology.orchestra.run.smithi090.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-01-15T14:27:28.829 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-01-15T14:27:28.836 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 1c77fb2b-d1a7-4884-a24e-3935e16d0b6b... 2023-01-15T14:27:28.867 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-15T14:27:28.870 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-01-15T14:27:30.315 INFO:teuthology.orchestra.run.smithi090.stdout:[global] 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: chdir = "" 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: auth supported = cephx 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: filestore xattr use omap = true 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: mon clock drift allowed = 1.000 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: osd crush chooseleaf type = 0 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: auth debug = true 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.316 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on old message = true 2023-01-15T14:27:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: ms die on bug = true 2023-01-15T14:27:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-15T14:27:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: mon pg warn max object skew = 0 2023-01-15T14:27:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: # disable pg_autoscaler by default for new pools 2023-01-15T14:27:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-15T14:27:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default size = 2 2023-01-15T14:27:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.317 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow primary affinity = true 2023-01-15T14:27:30.318 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd allow pg remap = true 2023-01-15T14:27:30.318 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on legacy crush tunables = false 2023-01-15T14:27:30.318 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on crush straw calc version zero = false 2023-01-15T14:27:30.318 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on no sortbitwise = false 2023-01-15T14:27:30.318 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on osd down out interval zero = false 2023-01-15T14:27:30.318 INFO:teuthology.orchestra.run.smithi090.stdout: mon warn on too few osds = false 2023-01-15T14:27:30.318 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-15T14:27:30.318 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-15T14:27:30.318 INFO:teuthology.orchestra.run.smithi090.stdout: mon_allow_pool_size_one = true 2023-01-15T14:27:30.318 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.318 INFO:teuthology.orchestra.run.smithi090.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: osd default data pool replay window = 5 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: mon allow pool delete = true 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: mon cluster log file level = debug 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: debug asserts on shutdown = true 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: mon health detail to clog = false 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: mon host = "172.21.15.90,[v2:172.21.15.90:3301,v1:172.21.15.90:6790],172.21.15.141" 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: mon election default strategy = 3 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: ms type = async 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug inject dispatch delay duration = 0.1 2023-01-15T14:27:30.319 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug inject dispatch delay probability = 0.1 2023-01-15T14:27:30.320 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_min_size = 2 2023-01-15T14:27:30.320 INFO:teuthology.orchestra.run.smithi090.stdout: osd_pool_default_size = 3 2023-01-15T14:27:30.320 INFO:teuthology.orchestra.run.smithi090.stdout: fsid = 1c77fb2b-d1a7-4884-a24e-3935e16d0b6b 2023-01-15T14:27:30.320 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.320 INFO:teuthology.orchestra.run.smithi090.stdout:[osd] 2023-01-15T14:27:30.320 INFO:teuthology.orchestra.run.smithi090.stdout: osd journal size = 100 2023-01-15T14:27:30.320 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.320 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub load threshold = 5.0 2023-01-15T14:27:30.320 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub max interval = 120 2023-01-15T14:27:30.320 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.321 INFO:teuthology.orchestra.run.smithi090.stdout: osd recover clone overlap = true 2023-01-15T14:27:30.321 INFO:teuthology.orchestra.run.smithi090.stdout: osd recovery max chunk = 1048576 2023-01-15T14:27:30.321 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.321 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug shutdown = true 2023-01-15T14:27:30.321 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug op order = true 2023-01-15T14:27:30.321 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify stray on activate = true 2023-01-15T14:27:30.321 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.321 INFO:teuthology.orchestra.run.smithi090.stdout: osd open classes on start = true 2023-01-15T14:27:30.321 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug pg log writeout = true 2023-01-15T14:27:30.322 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.322 INFO:teuthology.orchestra.run.smithi090.stdout: osd deep scrub update digest min age = 30 2023-01-15T14:27:30.322 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.322 INFO:teuthology.orchestra.run.smithi090.stdout: osd map max advance = 10 2023-01-15T14:27:30.322 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.322 INFO:teuthology.orchestra.run.smithi090.stdout: journal zero on create = true 2023-01-15T14:27:30.322 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.322 INFO:teuthology.orchestra.run.smithi090.stdout: filestore ondisk finisher threads = 3 2023-01-15T14:27:30.322 INFO:teuthology.orchestra.run.smithi090.stdout: filestore apply finisher threads = 3 2023-01-15T14:27:30.322 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.323 INFO:teuthology.orchestra.run.smithi090.stdout: bdev debug aio = true 2023-01-15T14:27:30.323 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug misdirected ops = true 2023-01-15T14:27:30.323 INFO:teuthology.orchestra.run.smithi090.stdout: debug monc = 20 2023-01-15T14:27:30.323 INFO:teuthology.orchestra.run.smithi090.stdout: bluestore zero block detection = True 2023-01-15T14:27:30.323 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-01-15T14:27:30.323 INFO:teuthology.orchestra.run.smithi090.stdout: debug osd = 20 2023-01-15T14:27:30.323 INFO:teuthology.orchestra.run.smithi090.stdout: filestore odsync write = True 2023-01-15T14:27:30.323 INFO:teuthology.orchestra.run.smithi090.stdout: osd backoff on degraded = True 2023-01-15T14:27:30.323 INFO:teuthology.orchestra.run.smithi090.stdout: osd backoff on peering = True 2023-01-15T14:27:30.323 INFO:teuthology.orchestra.run.smithi090.stdout: osd blocked scrub grace period = 3600 2023-01-15T14:27:30.324 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify cached snaps = True 2023-01-15T14:27:30.324 INFO:teuthology.orchestra.run.smithi090.stdout: osd debug verify missing on start = True 2023-01-15T14:27:30.324 INFO:teuthology.orchestra.run.smithi090.stdout: osd max backfills = 2 2023-01-15T14:27:30.324 INFO:teuthology.orchestra.run.smithi090.stdout: osd max markdown count = 1000 2023-01-15T14:27:30.324 INFO:teuthology.orchestra.run.smithi090.stdout: osd max scrubs = 3 2023-01-15T14:27:30.324 INFO:teuthology.orchestra.run.smithi090.stdout: osd mclock override recovery settings = True 2023-01-15T14:27:30.324 INFO:teuthology.orchestra.run.smithi090.stdout: osd objectstore = filestore 2023-01-15T14:27:30.324 INFO:teuthology.orchestra.run.smithi090.stdout: osd op queue = debug_random 2023-01-15T14:27:30.325 INFO:teuthology.orchestra.run.smithi090.stdout: osd op queue cut off = debug_random 2023-01-15T14:27:30.325 INFO:teuthology.orchestra.run.smithi090.stdout: osd scrub min interval = 60 2023-01-15T14:27:30.325 INFO:teuthology.orchestra.run.smithi090.stdout: osd shutdown pgref assert = True 2023-01-15T14:27:30.325 INFO:teuthology.orchestra.run.smithi090.stdout: osd sloppy crc = True 2023-01-15T14:27:30.325 INFO:teuthology.orchestra.run.smithi090.stdout: osd snap trim sleep = 0.5 2023-01-15T14:27:30.325 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.325 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr] 2023-01-15T14:27:30.325 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-01-15T14:27:30.325 INFO:teuthology.orchestra.run.smithi090.stdout: debug mgr = 20 2023-01-15T14:27:30.325 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2023-01-15T14:27:30.325 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout: mgr/telemetry/nag = false 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout:[mon] 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout: debug ms = 1 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout: debug mon = 20 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout: debug paxos = 20 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout: debug auth = 20 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout: mon data avail warn = 5 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout: mon mgr mkfs grace = 240 2023-01-15T14:27:30.326 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min pgs per osd = 4 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd reporter subtree level = osd 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: mon osd prime pg temp = true 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: mon reweight min bytes per osd = 10 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: auth mon ticket ttl = 660 # 11m 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: auth service ticket ttl = 240 # 4m 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: # don't complain about insecure global_id in the test suite 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-15T14:27:30.327 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.328 INFO:teuthology.orchestra.run.smithi090.stdout: # 1m isn't quite enough 2023-01-15T14:27:30.328 INFO:teuthology.orchestra.run.smithi090.stdout: mon_down_mkfs_grace = 2m 2023-01-15T14:27:30.328 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.328 INFO:teuthology.orchestra.run.smithi090.stdout: mon_warn_on_filestore_osds = false 2023-01-15T14:27:30.328 INFO:teuthology.orchestra.run.smithi090.stdout: mon min osdmap epochs = 50 2023-01-15T14:27:30.328 INFO:teuthology.orchestra.run.smithi090.stdout: mon osdmap full prune interval = 2 2023-01-15T14:27:30.328 INFO:teuthology.orchestra.run.smithi090.stdout: mon osdmap full prune min = 15 2023-01-15T14:27:30.329 INFO:teuthology.orchestra.run.smithi090.stdout: mon osdmap full prune txsize = 2 2023-01-15T14:27:30.329 INFO:teuthology.orchestra.run.smithi090.stdout: mon scrub interval = 300 2023-01-15T14:27:30.329 INFO:teuthology.orchestra.run.smithi090.stdout: paxos service trim min = 10 2023-01-15T14:27:30.329 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:27:30.329 INFO:teuthology.orchestra.run.smithi090.stdout:[client] 2023-01-15T14:27:30.330 INFO:teuthology.orchestra.run.smithi090.stdout: rgw cache enabled = true 2023-01-15T14:27:30.330 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable ops log = true 2023-01-15T14:27:30.330 INFO:teuthology.orchestra.run.smithi090.stdout: rgw enable usage log = true 2023-01-15T14:27:30.330 INFO:teuthology.orchestra.run.smithi090.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-15T14:27:30.330 INFO:teuthology.orchestra.run.smithi090.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-15T14:27:30.330 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.a] 2023-01-15T14:27:30.330 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.c] 2023-01-15T14:27:30.330 INFO:teuthology.orchestra.run.smithi090.stdout:[mon.b] 2023-01-15T14:27:30.345 INFO:teuthology.orchestra.run.smithi141.stdout:[global] 2023-01-15T14:27:30.461 INFO:teuthology.orchestra.run.smithi141.stdout: chdir = "" 2023-01-15T14:27:30.461 INFO:teuthology.orchestra.run.smithi141.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-01-15T14:27:30.461 INFO:teuthology.orchestra.run.smithi141.stdout: auth supported = cephx 2023-01-15T14:27:30.461 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.461 INFO:teuthology.orchestra.run.smithi141.stdout: filestore xattr use omap = true 2023-01-15T14:27:30.461 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: mon clock drift allowed = 1.000 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: osd crush chooseleaf type = 0 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: auth debug = true 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: ms die on old message = true 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: ms die on bug = true 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: mon max pg per osd = 10000 # >= luminous 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: mon pg warn max object skew = 0 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.462 INFO:teuthology.orchestra.run.smithi141.stdout: # disable pg_autoscaler by default for new pools 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: osd_pool_default_pg_autoscale_mode = off 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: osd pool default size = 2 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd allow primary affinity = true 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd allow pg remap = true 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on legacy crush tunables = false 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on crush straw calc version zero = false 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on no sortbitwise = false 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on osd down out interval zero = false 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: mon warn on too few osds = false 2023-01-15T14:27:30.463 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-01-15T14:27:30.464 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_pool_no_redundancy = false 2023-01-15T14:27:30.464 INFO:teuthology.orchestra.run.smithi141.stdout: mon_allow_pool_size_one = true 2023-01-15T14:27:30.464 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.464 INFO:teuthology.orchestra.run.smithi141.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-01-15T14:27:30.464 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.464 INFO:teuthology.orchestra.run.smithi141.stdout: osd default data pool replay window = 5 2023-01-15T14:27:30.464 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.464 INFO:teuthology.orchestra.run.smithi141.stdout: mon allow pool delete = true 2023-01-15T14:27:30.464 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.464 INFO:teuthology.orchestra.run.smithi141.stdout: mon cluster log file level = debug 2023-01-15T14:27:30.464 INFO:teuthology.orchestra.run.smithi141.stdout: debug asserts on shutdown = true 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout: mon health detail to clog = false 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout: mon host = "172.21.15.90,[v2:172.21.15.90:3301,v1:172.21.15.90:6790],172.21.15.141" 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout: mon election default strategy = 3 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout: ms type = async 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug inject dispatch delay duration = 0.1 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug inject dispatch delay probability = 0.1 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout: osd_pool_default_min_size = 2 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout: osd_pool_default_size = 3 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout: fsid = 1c77fb2b-d1a7-4884-a24e-3935e16d0b6b 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout:[osd] 2023-01-15T14:27:30.465 INFO:teuthology.orchestra.run.smithi141.stdout: osd journal size = 100 2023-01-15T14:27:30.466 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.466 INFO:teuthology.orchestra.run.smithi141.stdout: osd scrub load threshold = 5.0 2023-01-15T14:27:30.466 INFO:teuthology.orchestra.run.smithi141.stdout: osd scrub max interval = 120 2023-01-15T14:27:30.466 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.466 INFO:teuthology.orchestra.run.smithi141.stdout: osd recover clone overlap = true 2023-01-15T14:27:30.466 INFO:teuthology.orchestra.run.smithi141.stdout: osd recovery max chunk = 1048576 2023-01-15T14:27:30.466 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.466 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug shutdown = true 2023-01-15T14:27:30.466 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug op order = true 2023-01-15T14:27:30.466 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug verify stray on activate = true 2023-01-15T14:27:30.466 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.467 INFO:teuthology.orchestra.run.smithi141.stdout: osd open classes on start = true 2023-01-15T14:27:30.467 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug pg log writeout = true 2023-01-15T14:27:30.467 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.467 INFO:teuthology.orchestra.run.smithi141.stdout: osd deep scrub update digest min age = 30 2023-01-15T14:27:30.467 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.467 INFO:teuthology.orchestra.run.smithi141.stdout: osd map max advance = 10 2023-01-15T14:27:30.467 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.467 INFO:teuthology.orchestra.run.smithi141.stdout: journal zero on create = true 2023-01-15T14:27:30.467 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.467 INFO:teuthology.orchestra.run.smithi141.stdout: filestore ondisk finisher threads = 3 2023-01-15T14:27:30.467 INFO:teuthology.orchestra.run.smithi141.stdout: filestore apply finisher threads = 3 2023-01-15T14:27:30.468 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.468 INFO:teuthology.orchestra.run.smithi141.stdout: bdev debug aio = true 2023-01-15T14:27:30.468 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug misdirected ops = true 2023-01-15T14:27:30.468 INFO:teuthology.orchestra.run.smithi141.stdout: debug monc = 20 2023-01-15T14:27:30.468 INFO:teuthology.orchestra.run.smithi141.stdout: bluestore zero block detection = True 2023-01-15T14:27:30.468 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2023-01-15T14:27:30.468 INFO:teuthology.orchestra.run.smithi141.stdout: debug osd = 20 2023-01-15T14:27:30.468 INFO:teuthology.orchestra.run.smithi141.stdout: filestore odsync write = True 2023-01-15T14:27:30.468 INFO:teuthology.orchestra.run.smithi141.stdout: osd backoff on degraded = True 2023-01-15T14:27:30.468 INFO:teuthology.orchestra.run.smithi141.stdout: osd backoff on peering = True 2023-01-15T14:27:30.468 INFO:teuthology.orchestra.run.smithi141.stdout: osd blocked scrub grace period = 3600 2023-01-15T14:27:30.469 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug verify cached snaps = True 2023-01-15T14:27:30.469 INFO:teuthology.orchestra.run.smithi141.stdout: osd debug verify missing on start = True 2023-01-15T14:27:30.469 INFO:teuthology.orchestra.run.smithi141.stdout: osd max backfills = 2 2023-01-15T14:27:30.469 INFO:teuthology.orchestra.run.smithi141.stdout: osd max markdown count = 1000 2023-01-15T14:27:30.469 INFO:teuthology.orchestra.run.smithi141.stdout: osd max scrubs = 3 2023-01-15T14:27:30.469 INFO:teuthology.orchestra.run.smithi141.stdout: osd mclock override recovery settings = True 2023-01-15T14:27:30.469 INFO:teuthology.orchestra.run.smithi141.stdout: osd objectstore = filestore 2023-01-15T14:27:30.469 INFO:teuthology.orchestra.run.smithi141.stdout: osd op queue = debug_random 2023-01-15T14:27:30.469 INFO:teuthology.orchestra.run.smithi141.stdout: osd op queue cut off = debug_random 2023-01-15T14:27:30.469 INFO:teuthology.orchestra.run.smithi141.stdout: osd scrub min interval = 60 2023-01-15T14:27:30.469 INFO:teuthology.orchestra.run.smithi141.stdout: osd shutdown pgref assert = True 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout: osd sloppy crc = True 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout: osd snap trim sleep = 0.5 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout:[mgr] 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout: debug mgr = 20 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout: debug mon = 20 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout: debug auth = 20 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min pgs per osd = 4 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min bytes per osd = 10 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout: mgr/telemetry/nag = false 2023-01-15T14:27:30.470 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.471 INFO:teuthology.orchestra.run.smithi141.stdout:[mon] 2023-01-15T14:27:30.471 INFO:teuthology.orchestra.run.smithi141.stdout: debug ms = 1 2023-01-15T14:27:30.471 INFO:teuthology.orchestra.run.smithi141.stdout: debug mon = 20 2023-01-15T14:27:30.471 INFO:teuthology.orchestra.run.smithi141.stdout: debug paxos = 20 2023-01-15T14:27:30.471 INFO:teuthology.orchestra.run.smithi141.stdout: debug auth = 20 2023-01-15T14:27:30.471 INFO:teuthology.orchestra.run.smithi141.stdout: mon data avail warn = 5 2023-01-15T14:27:30.471 INFO:teuthology.orchestra.run.smithi141.stdout: mon mgr mkfs grace = 240 2023-01-15T14:27:30.471 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min pgs per osd = 4 2023-01-15T14:27:30.471 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd reporter subtree level = osd 2023-01-15T14:27:30.471 INFO:teuthology.orchestra.run.smithi141.stdout: mon osd prime pg temp = true 2023-01-15T14:27:30.471 INFO:teuthology.orchestra.run.smithi141.stdout: mon reweight min bytes per osd = 10 2023-01-15T14:27:30.472 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.472 INFO:teuthology.orchestra.run.smithi141.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-01-15T14:27:30.472 INFO:teuthology.orchestra.run.smithi141.stdout: auth mon ticket ttl = 660 # 11m 2023-01-15T14:27:30.472 INFO:teuthology.orchestra.run.smithi141.stdout: auth service ticket ttl = 240 # 4m 2023-01-15T14:27:30.472 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.472 INFO:teuthology.orchestra.run.smithi141.stdout: # don't complain about insecure global_id in the test suite 2023-01-15T14:27:30.472 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-01-15T14:27:30.472 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-01-15T14:27:30.472 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.472 INFO:teuthology.orchestra.run.smithi141.stdout: # 1m isn't quite enough 2023-01-15T14:27:30.472 INFO:teuthology.orchestra.run.smithi141.stdout: mon_down_mkfs_grace = 2m 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout: mon_warn_on_filestore_osds = false 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout: mon min osdmap epochs = 50 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout: mon osdmap full prune interval = 2 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout: mon osdmap full prune min = 15 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout: mon osdmap full prune txsize = 2 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout: mon scrub interval = 300 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout: paxos service trim min = 10 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout: 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout:[client] 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout: rgw cache enabled = true 2023-01-15T14:27:30.473 INFO:teuthology.orchestra.run.smithi141.stdout: rgw enable ops log = true 2023-01-15T14:27:30.474 INFO:teuthology.orchestra.run.smithi141.stdout: rgw enable usage log = true 2023-01-15T14:27:30.474 INFO:teuthology.orchestra.run.smithi141.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-01-15T14:27:30.474 INFO:teuthology.orchestra.run.smithi141.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-01-15T14:27:30.474 INFO:teuthology.orchestra.run.smithi141.stdout:[mon.a] 2023-01-15T14:27:30.474 INFO:teuthology.orchestra.run.smithi141.stdout:[mon.c] 2023-01-15T14:27:30.474 INFO:teuthology.orchestra.run.smithi141.stdout:[mon.b] 2023-01-15T14:27:30.475 INFO:tasks.ceph:Creating admin key on mon.a... 2023-01-15T14:27:30.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-01-15T14:27:30.533 INFO:tasks.ceph:Copying monmap to all nodes... 2023-01-15T14:27:30.533 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:30.533 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-01-15T14:27:30.540 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:30.540 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-01-15T14:27:30.544 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi090.front.sepia.ceph.com 2023-01-15T14:27:30.545 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:30.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-15T14:27:30.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-15T14:27:30.574 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:30.574 DEBUG:teuthology.orchestra.run.smithi090:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:30.581 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi141.front.sepia.ceph.com 2023-01-15T14:27:30.581 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:30.581 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd of=/etc/ceph/ceph.keyring 2023-01-15T14:27:30.581 DEBUG:teuthology.orchestra.run.smithi141:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-01-15T14:27:30.609 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:30.609 DEBUG:teuthology.orchestra.run.smithi141:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:30.615 INFO:tasks.ceph:Setting up mon nodes... 2023-01-15T14:27:30.615 INFO:tasks.ceph:Setting up mgr nodes... 2023-01-15T14:27:30.615 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-01-15T14:27:30.673 INFO:teuthology.orchestra.run.smithi090.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-01-15T14:27:30.676 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-01-15T14:27:30.723 INFO:teuthology.orchestra.run.smithi141.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-01-15T14:27:31.852 INFO:tasks.ceph:Setting up mds nodes... 2023-01-15T14:27:31.852 INFO:tasks.ceph_client:Setting up client nodes... 2023-01-15T14:27:31.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-01-15T14:27:31.897 INFO:teuthology.orchestra.run.smithi090.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-01-15T14:27:32.005 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-01-15T14:27:32.050 INFO:teuthology.orchestra.run.smithi141.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-01-15T14:27:32.173 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-01-15T14:27:32.173 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi090.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi141.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-01-15T14:27:32.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-01-15T14:27:32.190 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-15T14:27:32.333 INFO:tasks.ceph:role: osd.0 2023-01-15T14:27:32.333 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com 2023-01-15T14:27:32.333 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-15T14:27:32.881 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-15T14:27:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-15T14:27:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-15T14:27:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 2023-01-15T14:27:33.720 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-15T14:27:33.720 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-01-15T14:27:33.720 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-15T14:27:33.724 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-15T14:27:33.724 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-15T14:27:33.725 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-15T14:27:33.726 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-01-15T14:27:33.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-01-15T14:27:33.750 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-01-15T14:27:33.934 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2023-01-15T14:27:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-15T14:27:33.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-01-15T14:27:33.995 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-15T14:27:34.129 INFO:tasks.ceph:role: osd.1 2023-01-15T14:27:34.129 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com 2023-01-15T14:27:34.129 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-15T14:27:34.633 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-15T14:27:34.746 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-15T14:27:34.746 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-15T14:27:34.747 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 2023-01-15T14:27:34.747 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-15T14:27:34.747 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-01-15T14:27:34.747 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-15T14:27:34.747 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-15T14:27:34.747 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-15T14:27:34.747 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-15T14:27:34.748 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-01-15T14:27:34.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-01-15T14:27:34.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-01-15T14:27:35.163 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2023-01-15T14:27:35.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-15T14:27:35.165 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-01-15T14:27:35.181 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-15T14:27:35.181 INFO:tasks.ceph:role: osd.2 2023-01-15T14:27:35.181 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com 2023-01-15T14:27:35.181 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-15T14:27:35.740 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-15T14:27:35.769 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-15T14:27:35.769 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-15T14:27:35.769 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 2023-01-15T14:27:35.769 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-15T14:27:35.769 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-01-15T14:27:35.769 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-15T14:27:35.769 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-15T14:27:35.769 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-15T14:27:35.769 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-15T14:27:35.770 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-01-15T14:27:35.771 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-01-15T14:27:35.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-01-15T14:27:35.991 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2023-01-15T14:27:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-15T14:27:35.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-01-15T14:27:36.051 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-01-15T14:27:36.176 INFO:tasks.ceph:role: osd.3 2023-01-15T14:27:36.177 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com 2023-01-15T14:27:36.177 DEBUG:teuthology.orchestra.run.smithi090:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-01-15T14:27:36.666 INFO:teuthology.orchestra.run.smithi090.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-01-15T14:27:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-15T14:27:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-15T14:27:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: = reflink=1 2023-01-15T14:27:36.792 INFO:teuthology.orchestra.run.smithi090.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-15T14:27:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: = sunit=0 swidth=0 blks 2023-01-15T14:27:36.792 INFO:teuthology.orchestra.run.smithi090.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-15T14:27:36.792 INFO:teuthology.orchestra.run.smithi090.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-15T14:27:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-15T14:27:36.793 INFO:teuthology.orchestra.run.smithi090.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-15T14:27:36.794 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi090.front.sepia.ceph.com -o noatime 2023-01-15T14:27:36.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-01-15T14:27:36.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-01-15T14:27:37.018 INFO:teuthology.orchestra.run.smithi090.stderr:sudo: /sbin/restorecon: command not found 2023-01-15T14:27:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-15T14:27:37.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:37.284 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.278+0000 7f23c1708680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-01-15T14:27:37.566 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.282+0000 7f23c1708680 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-01-15T14:27:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.322+0000 7f23c1708680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:37.567 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.322+0000 7f23c1708680 -1 journal FileJournal::_open_file : zeroing journal 2023-01-15T14:27:37.612 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.606+0000 7f23c1708680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:37.726 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.606+0000 7f23c1708680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:37.726 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.606+0000 7f23c1708680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:37.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-01-15T14:27:37.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:37.786 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.778+0000 7f6943573680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-01-15T14:27:37.886 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.782+0000 7f6943573680 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-01-15T14:27:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.822+0000 7f6943573680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:37.887 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.822+0000 7f6943573680 -1 journal FileJournal::_open_file : zeroing journal 2023-01-15T14:27:37.980 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.974+0000 7f6943573680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.974+0000 7f6943573680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:38.053 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:37.974+0000 7f6943573680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:38.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-01-15T14:27:38.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:38.259 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.250+0000 7f42be860680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-01-15T14:27:38.369 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.254+0000 7f42be860680 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-01-15T14:27:38.370 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.266+0000 7f42be860680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:38.370 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.266+0000 7f42be860680 -1 journal FileJournal::_open_file : zeroing journal 2023-01-15T14:27:38.404 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.398+0000 7f42be860680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.398+0000 7f42be860680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:38.531 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.398+0000 7f42be860680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:38.533 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-01-15T14:27:38.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:38.740 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.734+0000 7fee14017680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-01-15T14:27:38.863 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.734+0000 7fee14017680 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-01-15T14:27:38.865 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.746+0000 7fee14017680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:38.865 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.746+0000 7fee14017680 -1 journal FileJournal::_open_file : zeroing journal 2023-01-15T14:27:38.888 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.882+0000 7fee14017680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:39.012 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.882+0000 7fee14017680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:39.012 INFO:teuthology.orchestra.run.smithi090.stderr:2023-01-15T14:27:38.882+0000 7fee14017680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:39.014 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-01-15T14:27:39.031 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-01-15T14:27:39.048 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-15T14:27:39.172 INFO:tasks.ceph:role: osd.4 2023-01-15T14:27:39.172 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi141.front.sepia.ceph.com 2023-01-15T14:27:39.172 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-01-15T14:27:39.690 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-01-15T14:27:39.863 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-15T14:27:39.863 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-15T14:27:39.864 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 2023-01-15T14:27:39.864 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-15T14:27:39.864 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2023-01-15T14:27:39.864 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-15T14:27:39.864 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-15T14:27:39.864 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-15T14:27:39.864 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-15T14:27:39.865 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2023-01-15T14:27:39.866 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-01-15T14:27:39.958 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-01-15T14:27:40.037 INFO:teuthology.orchestra.run.smithi141.stderr:sudo: /sbin/restorecon: command not found 2023-01-15T14:27:40.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-15T14:27:40.038 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-01-15T14:27:40.048 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-15T14:27:40.184 INFO:tasks.ceph:role: osd.5 2023-01-15T14:27:40.185 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi141.front.sepia.ceph.com 2023-01-15T14:27:40.185 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-01-15T14:27:40.674 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-01-15T14:27:42.361 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-15T14:27:42.361 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-15T14:27:42.361 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 2023-01-15T14:27:42.361 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-15T14:27:42.361 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2023-01-15T14:27:42.361 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-15T14:27:42.361 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-15T14:27:42.361 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-15T14:27:42.362 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-15T14:27:42.363 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2023-01-15T14:27:42.363 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-01-15T14:27:42.390 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-01-15T14:27:42.530 INFO:teuthology.orchestra.run.smithi141.stderr:sudo: /sbin/restorecon: command not found 2023-01-15T14:27:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-15T14:27:43.824 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-01-15T14:27:43.844 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-15T14:27:43.949 INFO:tasks.ceph:role: osd.6 2023-01-15T14:27:43.949 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi141.front.sepia.ceph.com 2023-01-15T14:27:43.950 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-01-15T14:27:44.439 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-01-15T14:27:44.458 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-15T14:27:44.458 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-15T14:27:44.459 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 2023-01-15T14:27:44.459 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-15T14:27:44.459 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2023-01-15T14:27:44.459 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-15T14:27:44.459 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-15T14:27:44.459 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-15T14:27:44.459 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-15T14:27:44.460 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2023-01-15T14:27:44.461 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-01-15T14:27:44.518 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-01-15T14:27:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-15T14:27:44.599 INFO:teuthology.orchestra.run.smithi141.stderr:sudo: /sbin/restorecon: command not found 2023-01-15T14:27:44.599 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-01-15T14:27:44.658 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-01-15T14:27:44.717 INFO:tasks.ceph:role: osd.7 2023-01-15T14:27:44.717 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi141.front.sepia.ceph.com 2023-01-15T14:27:44.718 DEBUG:teuthology.orchestra.run.smithi141:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-01-15T14:27:45.247 INFO:teuthology.orchestra.run.smithi141.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-01-15T14:27:46.209 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 attr=2, projid32bit=1 2023-01-15T14:27:46.210 INFO:teuthology.orchestra.run.smithi141.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-01-15T14:27:46.210 INFO:teuthology.orchestra.run.smithi141.stdout: = reflink=1 2023-01-15T14:27:46.210 INFO:teuthology.orchestra.run.smithi141.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-01-15T14:27:46.210 INFO:teuthology.orchestra.run.smithi141.stdout: = sunit=0 swidth=0 blks 2023-01-15T14:27:46.210 INFO:teuthology.orchestra.run.smithi141.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-01-15T14:27:46.210 INFO:teuthology.orchestra.run.smithi141.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-01-15T14:27:46.210 INFO:teuthology.orchestra.run.smithi141.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-01-15T14:27:46.210 INFO:teuthology.orchestra.run.smithi141.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-01-15T14:27:46.211 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi141.front.sepia.ceph.com -o noatime 2023-01-15T14:27:46.211 DEBUG:teuthology.orchestra.run.smithi141:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-01-15T14:27:46.239 DEBUG:teuthology.orchestra.run.smithi141:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-01-15T14:27:46.297 INFO:teuthology.orchestra.run.smithi141.stderr:sudo: /sbin/restorecon: command not found 2023-01-15T14:27:46.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-15T14:27:46.353 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:46.444 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.438+0000 7f86bdb10680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-01-15T14:27:46.619 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.438+0000 7f86bdb10680 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-01-15T14:27:46.621 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.450+0000 7f86bdb10680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:46.621 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.450+0000 7f86bdb10680 -1 journal FileJournal::_open_file : zeroing journal 2023-01-15T14:27:46.626 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.598+0000 7f86bdb10680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:46.627 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.598+0000 7f86bdb10680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:46.627 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.598+0000 7f86bdb10680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:46.629 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-01-15T14:27:46.645 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:46.736 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.730+0000 7fb236eda680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-01-15T14:27:46.829 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.730+0000 7fb236eda680 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-01-15T14:27:46.829 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.742+0000 7fb236eda680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:46.829 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.742+0000 7fb236eda680 -1 journal FileJournal::_open_file : zeroing journal 2023-01-15T14:27:46.892 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.886+0000 7fb236eda680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:46.892 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.886+0000 7fb236eda680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:46.892 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.886+0000 7fb236eda680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:46.911 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-01-15T14:27:46.927 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:46.972 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.966+0000 7f2e4d4cc680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-01-15T14:27:47.097 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.966+0000 7f2e4d4cc680 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-01-15T14:27:47.098 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.978+0000 7f2e4d4cc680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:47.098 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:46.978+0000 7f2e4d4cc680 -1 journal FileJournal::_open_file : zeroing journal 2023-01-15T14:27:47.127 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:47.122+0000 7f2e4d4cc680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:47.127 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:47.122+0000 7f2e4d4cc680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:47.128 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:47.122+0000 7f2e4d4cc680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:47.146 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-01-15T14:27:47.276 DEBUG:teuthology.orchestra.run.smithi141:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:47.364 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:47.358+0000 7f14fa3e6680 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-01-15T14:27:47.444 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:47.358+0000 7f14fa3e6680 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-01-15T14:27:47.446 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:47.370+0000 7f14fa3e6680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:47.446 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:47.370+0000 7f14fa3e6680 -1 journal FileJournal::_open_file : zeroing journal 2023-01-15T14:27:47.532 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:47.526+0000 7f14fa3e6680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:27:47.646 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:47.526+0000 7f14fa3e6680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:47.646 INFO:teuthology.orchestra.run.smithi141.stderr:2023-01-15T14:27:47.526+0000 7f14fa3e6680 -1 journal do_read_entry(4096): bad header magic 2023-01-15T14:27:47.647 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-01-15T14:27:47.665 INFO:tasks.ceph:Reading keys from all nodes... 2023-01-15T14:27:47.667 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:47.667 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-01-15T14:27:47.685 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:47.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-01-15T14:27:47.700 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:48.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-01-15T14:27:48.075 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:48.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-01-15T14:27:48.091 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:48.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-01-15T14:27:48.150 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:48.150 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-01-15T14:27:48.167 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:48.167 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-01-15T14:27:48.226 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:48.262 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-01-15T14:27:48.278 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:48.278 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-01-15T14:27:48.337 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:48.588 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-01-15T14:27:48.614 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:48.614 DEBUG:teuthology.orchestra.run.smithi090:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-01-15T14:27:48.622 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:48.622 DEBUG:teuthology.orchestra.run.smithi141:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-01-15T14:27:48.628 INFO:tasks.ceph:Adding keys to all mons... 2023-01-15T14:27:48.628 DEBUG:teuthology.orchestra.run.smithi090:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-15T14:27:48.631 DEBUG:teuthology.orchestra.run.smithi141:> sudo tee -a /etc/ceph/ceph.keyring 2023-01-15T14:27:48.672 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.y] 2023-01-15T14:27:48.752 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBSDcRjAuECKBAAp5J1KguT+lz9w5F1PA3UbQ== 2023-01-15T14:27:48.753 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.0] 2023-01-15T14:27:48.753 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBZDcRjPNvOEBAAELdBj4HfqZHX0sKOcM6LLQ== 2023-01-15T14:27:48.753 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.1] 2023-01-15T14:27:48.753 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBZDcRjnTPDLhAABSIFP52LMxre1WFWsVjW0Q== 2023-01-15T14:27:48.753 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.2] 2023-01-15T14:27:48.753 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBaDcRjmE9VDxAAnHJcfxD9V2/pbewFDXpHoA== 2023-01-15T14:27:48.753 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.3] 2023-01-15T14:27:48.753 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBaDcRjzLT9KxAAJxQGahU3xiTUtwKZeRxYuA== 2023-01-15T14:27:48.753 INFO:teuthology.orchestra.run.smithi090.stdout:[mgr.x] 2023-01-15T14:27:48.753 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBSDcRjwmoLKxAAlFdEIJixHdYzs8+3/G1f5w== 2023-01-15T14:27:48.754 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.4] 2023-01-15T14:27:48.754 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBiDcRjHZphGhAA+nbRysU/WY5orCcv/KPvKQ== 2023-01-15T14:27:48.754 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.5] 2023-01-15T14:27:48.754 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBiDcRj0mrEKxAAAf2bQV5Rn6uND0kAEzGafQ== 2023-01-15T14:27:48.754 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.6] 2023-01-15T14:27:48.754 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBiDcRjJB3gORAAyScYPWNDYxfTAM94oeVwuQ== 2023-01-15T14:27:48.754 INFO:teuthology.orchestra.run.smithi090.stdout:[osd.7] 2023-01-15T14:27:48.754 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBjDcRjwpaeFRAApwHiXdcs40GPLT1jNjbU/A== 2023-01-15T14:27:48.754 INFO:teuthology.orchestra.run.smithi090.stdout:[client.0] 2023-01-15T14:27:48.754 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBTDcRj/wVfNRAAuexgIHVMm52bg0AIDg3C7A== 2023-01-15T14:27:48.754 INFO:teuthology.orchestra.run.smithi090.stdout:[client.1] 2023-01-15T14:27:48.755 INFO:teuthology.orchestra.run.smithi090.stdout: key = AQBUDcRjWMDsAhAAkcFrLqEgMwgUgeXYP3eaGQ== 2023-01-15T14:27:48.756 INFO:teuthology.orchestra.run.smithi141.stdout:[mgr.y] 2023-01-15T14:27:48.913 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBSDcRjAuECKBAAp5J1KguT+lz9w5F1PA3UbQ== 2023-01-15T14:27:48.914 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.0] 2023-01-15T14:27:48.914 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBZDcRjPNvOEBAAELdBj4HfqZHX0sKOcM6LLQ== 2023-01-15T14:27:48.914 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.1] 2023-01-15T14:27:48.914 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBZDcRjnTPDLhAABSIFP52LMxre1WFWsVjW0Q== 2023-01-15T14:27:48.914 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.2] 2023-01-15T14:27:48.916 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBaDcRjmE9VDxAAnHJcfxD9V2/pbewFDXpHoA== 2023-01-15T14:27:48.916 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.3] 2023-01-15T14:27:48.916 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBaDcRjzLT9KxAAJxQGahU3xiTUtwKZeRxYuA== 2023-01-15T14:27:48.964 INFO:teuthology.orchestra.run.smithi141.stdout:[mgr.x] 2023-01-15T14:27:48.964 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBSDcRjwmoLKxAAlFdEIJixHdYzs8+3/G1f5w== 2023-01-15T14:27:48.964 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.4] 2023-01-15T14:27:48.964 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBiDcRjHZphGhAA+nbRysU/WY5orCcv/KPvKQ== 2023-01-15T14:27:48.964 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.5] 2023-01-15T14:27:48.964 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBiDcRj0mrEKxAAAf2bQV5Rn6uND0kAEzGafQ== 2023-01-15T14:27:48.964 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.6] 2023-01-15T14:27:48.964 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBiDcRjJB3gORAAyScYPWNDYxfTAM94oeVwuQ== 2023-01-15T14:27:48.965 INFO:teuthology.orchestra.run.smithi141.stdout:[osd.7] 2023-01-15T14:27:48.965 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBjDcRjwpaeFRAApwHiXdcs40GPLT1jNjbU/A== 2023-01-15T14:27:48.965 INFO:teuthology.orchestra.run.smithi141.stdout:[client.0] 2023-01-15T14:27:48.965 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBTDcRj/wVfNRAAuexgIHVMm52bg0AIDg3C7A== 2023-01-15T14:27:48.965 INFO:teuthology.orchestra.run.smithi141.stdout:[client.1] 2023-01-15T14:27:48.965 INFO:teuthology.orchestra.run.smithi141.stdout: key = AQBUDcRjWMDsAhAAkcFrLqEgMwgUgeXYP3eaGQ== 2023-01-15T14:27:48.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-15T14:27:48.968 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-15T14:27:49.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:49.237 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:49.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:49.400 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:49.453 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:49.555 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:49.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:49.618 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:49.764 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-15T14:27:49.774 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-01-15T14:27:49.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:49.882 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:49.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:49.946 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:50.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:50.093 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:50.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:50.195 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-01-15T14:27:50.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-15T14:27:50.247 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-15T14:27:50.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-15T14:27:50.359 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-01-15T14:27:50.419 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-01-15T14:27:50.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-01-15T14:27:50.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-15T14:27:50.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-01-15T14:27:50.735 DEBUG:teuthology.orchestra.run.smithi090:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-01-15T14:27:50.746 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-15T14:27:50.947 DEBUG:teuthology.orchestra.run.smithi090:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-01-15T14:27:51.003 DEBUG:teuthology.orchestra.run.smithi141:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-01-15T14:27:51.055 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-01-15T14:27:51.223 DEBUG:teuthology.orchestra.run.smithi141:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-01-15T14:27:51.345 DEBUG:teuthology.orchestra.run.smithi090:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:51.372 DEBUG:teuthology.orchestra.run.smithi141:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-01-15T14:27:51.391 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-01-15T14:27:51.558 INFO:tasks.ceph.mon.a:Restarting daemon 2023-01-15T14:27:51.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-01-15T14:27:51.561 INFO:tasks.ceph.mon.a:Started 2023-01-15T14:27:51.561 INFO:tasks.ceph.mon.c:Restarting daemon 2023-01-15T14:27:51.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-01-15T14:27:51.563 INFO:tasks.ceph.mon.c:Started 2023-01-15T14:27:51.564 INFO:tasks.ceph.mon.b:Restarting daemon 2023-01-15T14:27:51.564 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-01-15T14:27:51.566 INFO:tasks.ceph.mon.b:Started 2023-01-15T14:27:51.566 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-01-15T14:27:51.566 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-01-15T14:27:51.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-01-15T14:27:51.569 INFO:tasks.ceph.mgr.y:Started 2023-01-15T14:27:51.569 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-01-15T14:27:51.569 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-01-15T14:27:51.571 INFO:tasks.ceph.mgr.x:Started 2023-01-15T14:27:51.571 INFO:tasks.ceph:Setting crush tunables to default 2023-01-15T14:27:51.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd crush tunables default 2023-01-15T14:27:57.864 INFO:teuthology.orchestra.run.smithi090.stderr:adjusted tunables profile to default 2023-01-15T14:27:57.888 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-01-15T14:27:57.888 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:57.889 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-01-15T14:27:57.901 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:57.901 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-01-15T14:27:58.074 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:27:57.949+0000 7fe2658292c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-15T14:27:58.089 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:58.089 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-01-15T14:27:58.145 DEBUG:teuthology.orchestra.run.smithi090:> set -ex 2023-01-15T14:27:58.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-01-15T14:27:58.293 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:27:58.281+0000 7f1a1aee22c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-01-15T14:27:58.310 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:58.499 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-01-15T14:27:58.515 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:58.515 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-01-15T14:27:58.541 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:27:58.533+0000 7fe2658292c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-15T14:27:58.745 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:27:58.589+0000 7fe2658292c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-15T14:27:58.745 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:27:58.701+0000 7fe2658292c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-15T14:27:58.760 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:58.760 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-01-15T14:27:58.769 DEBUG:teuthology.orchestra.run.smithi141:> set -ex 2023-01-15T14:27:59.092 DEBUG:teuthology.orchestra.run.smithi141:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-01-15T14:27:59.095 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:27:58.925+0000 7fe2658292c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-15T14:27:59.103 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:27:58.973+0000 7fe2658292c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-15T14:27:59.104 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:27:58.909+0000 7f1a1aee22c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-01-15T14:27:59.104 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:27:58.965+0000 7f1a1aee22c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-01-15T14:27:59.104 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:27:59.081+0000 7f1a1aee22c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-01-15T14:27:59.120 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new e55c5ac4-3cff-4ee5-8719-373648f83d8d 0 2023-01-15T14:27:59.148 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:27:59.141+0000 7fe2658292c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-15T14:27:59.295 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:27:59.201+0000 7fe2658292c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-15T14:27:59.295 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:27:59.261+0000 7fe2658292c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-15T14:27:59.325 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:27:59.317+0000 7f1a1aee22c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-01-15T14:27:59.544 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:27:59.373+0000 7f1a1aee22c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-01-15T14:27:59.545 INFO:teuthology.orchestra.run.smithi141.stdout:0 2023-01-15T14:27:59.546 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 2c1b33c9-0a1e-4734-936d-d51d64fe997a 1 2023-01-15T14:27:59.568 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:27:59.561+0000 7f1a1aee22c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-01-15T14:27:59.715 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:27:59.665+0000 7fe2658292c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-15T14:27:59.715 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:27:59.621+0000 7f1a1aee22c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-01-15T14:27:59.715 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:27:59.685+0000 7f1a1aee22c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-01-15T14:27:59.751 INFO:teuthology.orchestra.run.smithi141.stdout:1 2023-01-15T14:27:59.985 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:27:59.781+0000 7fe2658292c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-15T14:27:59.986 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new b314174a-eca9-4baa-a60c-2589f1fc9a8e 2 2023-01-15T14:28:00.072 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:00.065+0000 7fe2658292c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-01-15T14:28:00.144 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:00.089+0000 7f1a1aee22c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-01-15T14:28:00.205 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:00.197+0000 7f1a1aee22c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-01-15T14:28:00.344 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:00.229+0000 7fe2658292c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-15T14:28:00.344 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:00.325+0000 7fe2658292c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-15T14:28:00.345 INFO:teuthology.orchestra.run.smithi141.stdout:2 2023-01-15T14:28:00.345 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 29eb0b17-ed5d-4665-a347-2fb8406d9c62 3 2023-01-15T14:28:00.388 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:00.381+0000 7fe2658292c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-15T14:28:00.554 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:00.505+0000 7f1a1aee22c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-01-15T14:28:00.558 INFO:teuthology.orchestra.run.smithi141.stdout:3 2023-01-15T14:28:00.571 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 59362feb-caa9-44e3-b6d0-b79397dacbfc 4 2023-01-15T14:28:00.681 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:00.673+0000 7f1a1aee22c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-01-15T14:28:00.756 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:00.745+0000 7f1a1aee22c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-01-15T14:28:00.777 INFO:teuthology.orchestra.run.smithi141.stdout:4 2023-01-15T14:28:00.961 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new fbc8821c-8d78-40f8-a571-b18a70d42000 5 2023-01-15T14:28:00.966 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:00.801+0000 7f1a1aee22c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-01-15T14:28:01.178 INFO:teuthology.orchestra.run.smithi141.stdout:5 2023-01-15T14:28:01.371 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:01.341+0000 7fe2658292c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-15T14:28:01.372 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 34550470-3561-492f-b605-b5f0e8052c1e 6 2023-01-15T14:28:01.408 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:01.401+0000 7fe2658292c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-15T14:28:01.580 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:01.573+0000 7fe2658292c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-15T14:28:01.593 INFO:teuthology.orchestra.run.smithi141.stdout:6 2023-01-15T14:28:01.605 DEBUG:teuthology.orchestra.run.smithi141:> sudo ceph --cluster ceph osd new 1f123d0d-d617-4771-b516-21aa5a29b734 7 2023-01-15T14:28:01.626 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:01.621+0000 7f1a1aee22c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-01-15T14:28:01.779 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:01.685+0000 7f1a1aee22c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-01-15T14:28:01.779 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:01.637+0000 7fe2658292c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-15T14:28:01.779 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:01.705+0000 7fe2658292c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-15T14:28:01.806 INFO:teuthology.orchestra.run.smithi141.stdout:7 2023-01-15T14:28:01.981 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:01.881+0000 7fe2658292c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-15T14:28:01.981 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:01.953+0000 7fe2658292c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-15T14:28:01.982 INFO:tasks.ceph.osd.0:Restarting daemon 2023-01-15T14:28:01.982 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-01-15T14:28:01.982 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:01.845+0000 7f1a1aee22c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-01-15T14:28:01.982 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:01.909+0000 7f1a1aee22c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-01-15T14:28:01.984 INFO:tasks.ceph.osd.0:Started 2023-01-15T14:28:01.985 INFO:tasks.ceph.osd.1:Restarting daemon 2023-01-15T14:28:01.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-01-15T14:28:01.986 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:01.981+0000 7f1a1aee22c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-01-15T14:28:01.987 INFO:tasks.ceph.osd.1:Started 2023-01-15T14:28:01.987 INFO:tasks.ceph.osd.2:Restarting daemon 2023-01-15T14:28:01.987 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-01-15T14:28:01.990 INFO:tasks.ceph.osd.2:Started 2023-01-15T14:28:02.187 INFO:tasks.ceph.osd.3:Restarting daemon 2023-01-15T14:28:02.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-01-15T14:28:02.188 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:02.153+0000 7f1a1aee22c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-01-15T14:28:02.188 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:02.049+0000 7ffaf15d2680 -1 Falling back to public interface 2023-01-15T14:28:02.190 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:02.053+0000 7f4f3cb39680 -1 Falling back to public interface 2023-01-15T14:28:02.190 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:02.053+0000 7fe35bb77680 -1 Falling back to public interface 2023-01-15T14:28:02.191 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:02.069+0000 7ffaf15d2680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:28:02.191 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:02.069+0000 7ffaf15d2680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.191 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:02.069+0000 7ffaf15d2680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.191 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:02.069+0000 7fe35bb77680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:28:02.192 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:02.069+0000 7fe35bb77680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.192 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:02.069+0000 7fe35bb77680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.192 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:02.073+0000 7f4f3cb39680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:28:02.192 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:02.073+0000 7f4f3cb39680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.192 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:02.073+0000 7f4f3cb39680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.192 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:02.093+0000 7f4f3cb39680 -1 osd.0 0 log_to_monitors true 2023-01-15T14:28:02.193 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:02.093+0000 7ffaf15d2680 -1 osd.1 0 log_to_monitors true 2023-01-15T14:28:02.193 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:02.093+0000 7fe35bb77680 -1 osd.2 0 log_to_monitors true 2023-01-15T14:28:02.195 INFO:tasks.ceph.osd.3:Started 2023-01-15T14:28:02.195 INFO:tasks.ceph.osd.4:Restarting daemon 2023-01-15T14:28:02.195 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-01-15T14:28:02.197 INFO:tasks.ceph.osd.4:Started 2023-01-15T14:28:02.198 INFO:tasks.ceph.osd.5:Restarting daemon 2023-01-15T14:28:02.198 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-01-15T14:28:02.200 INFO:tasks.ceph.osd.5:Started 2023-01-15T14:28:02.200 INFO:tasks.ceph.osd.6:Restarting daemon 2023-01-15T14:28:02.201 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-01-15T14:28:02.203 INFO:tasks.ceph.osd.6:Started 2023-01-15T14:28:02.204 INFO:tasks.ceph.osd.7:Restarting daemon 2023-01-15T14:28:02.204 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-01-15T14:28:02.206 INFO:tasks.ceph.osd.7:Started 2023-01-15T14:28:02.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-15T14:28:02.234 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:02.229+0000 7f1a1aee22c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-01-15T14:28:02.403 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:02.261+0000 7f51609e5680 -1 Falling back to public interface 2023-01-15T14:28:02.404 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:02.261+0000 7f16b58b9680 -1 Falling back to public interface 2023-01-15T14:28:02.404 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:02.261+0000 7f9109512680 -1 Falling back to public interface 2023-01-15T14:28:02.404 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:02.289+0000 7f9109512680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:28:02.404 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:02.289+0000 7f9109512680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.405 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:02.289+0000 7f9109512680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.406 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:02.317+0000 7f9109512680 -1 osd.3 0 log_to_monitors true 2023-01-15T14:28:02.409 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:02.281+0000 7f51609e5680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:28:02.409 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:02.281+0000 7f51609e5680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.409 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:02.281+0000 7f51609e5680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.410 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:02.281+0000 7f9570c95680 -1 Falling back to public interface 2023-01-15T14:28:02.410 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:28:02.411 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":10,"fsid":"1c77fb2b-d1a7-4884-a24e-3935e16d0b6b","created":"2023-01-15T14:27:57.735717+0000","modified":"2023-01-15T14:28:01.797608+0000","last_up_change":"0.000000","last_in_change":"2023-01-15T14:28:01.797608+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"e55c5ac4-3cff-4ee5-8719-373648f83d8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2c1b33c9-0a1e-4734-936d-d51d64fe997a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b314174a-eca9-4baa-a60c-2589f1fc9a8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"29eb0b17-ed5d-4665-a347-2fb8406d9c62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"59362feb-caa9-44e3-b6d0-b79397dacbfc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fbc8821c-8d78-40f8-a571-b18a70d42000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"34550470-3561-492f-b605-b5f0e8052c1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1f123d0d-d617-4771-b516-21aa5a29b734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-15T14:28:02.411 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:02.289+0000 7f16b58b9680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:28:02.411 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:02.289+0000 7f16b58b9680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.411 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:02.289+0000 7f16b58b9680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.412 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-01-15T14:28:02.412 INFO:tasks.ceph:Waiting for OSDs to come up 2023-01-15T14:28:02.412 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:02.289+0000 7fea4df8d680 -1 Falling back to public interface 2023-01-15T14:28:02.412 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:02.301+0000 7f51609e5680 -1 osd.4 0 log_to_monitors true 2023-01-15T14:28:02.413 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:02.317+0000 7f9570c95680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:28:02.413 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:02.317+0000 7f9570c95680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.413 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:02.317+0000 7f9570c95680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.413 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:02.317+0000 7f16b58b9680 -1 osd.6 0 log_to_monitors true 2023-01-15T14:28:02.414 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:02.321+0000 7fea4df8d680 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2023-01-15T14:28:02.414 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:02.321+0000 7fea4df8d680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.414 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:02.321+0000 7fea4df8d680 -1 journal do_read_entry(8192): bad header magic 2023-01-15T14:28:02.414 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:02.341+0000 7f9570c95680 -1 osd.7 0 log_to_monitors true 2023-01-15T14:28:02.414 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:02.341+0000 7fea4df8d680 -1 osd.5 0 log_to_monitors true 2023-01-15T14:28:02.809 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:02.797+0000 7fe2658292c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-15T14:28:03.038 INFO:tasks.ceph.mgr.x.smithi141.stderr:2023-01-15T14:28:03.033+0000 7f1a1aee22c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-01-15T14:28:03.312 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-15T14:28:03.606 INFO:teuthology.misc.health.smithi090.stdout: 2023-01-15T14:28:03.619 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":11,"fsid":"1c77fb2b-d1a7-4884-a24e-3935e16d0b6b","created":"2023-01-15T14:27:57.735717+0000","modified":"2023-01-15T14:28:02.799633+0000","last_up_change":"0.000000","last_in_change":"2023-01-15T14:28:01.797608+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[],"osds":[{"osd":0,"uuid":"e55c5ac4-3cff-4ee5-8719-373648f83d8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2c1b33c9-0a1e-4734-936d-d51d64fe997a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b314174a-eca9-4baa-a60c-2589f1fc9a8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"29eb0b17-ed5d-4665-a347-2fb8406d9c62","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"59362feb-caa9-44e3-b6d0-b79397dacbfc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fbc8821c-8d78-40f8-a571-b18a70d42000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"34550470-3561-492f-b605-b5f0e8052c1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1f123d0d-d617-4771-b516-21aa5a29b734","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-15T14:28:03.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-01-15T14:28:03.828 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:03.821+0000 7f515c822700 -1 osd.4 0 waiting for initial osdmap 2023-01-15T14:28:03.828 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:03.821+0000 7f956cad2700 -1 osd.7 0 waiting for initial osdmap 2023-01-15T14:28:03.829 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:03.821+0000 7fea49dca700 -1 osd.5 0 waiting for initial osdmap 2023-01-15T14:28:03.829 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:03.821+0000 7fe35833b700 -1 osd.2 0 waiting for initial osdmap 2023-01-15T14:28:03.829 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:03.821+0000 7f9105cd6700 -1 osd.3 0 waiting for initial osdmap 2023-01-15T14:28:03.829 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:03.821+0000 7f4f38976700 -1 osd.0 0 waiting for initial osdmap 2023-01-15T14:28:03.831 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:03.821+0000 7ffaedd96700 -1 osd.1 0 waiting for initial osdmap 2023-01-15T14:28:03.833 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:03.825+0000 7f16b16f6700 -1 osd.6 0 waiting for initial osdmap 2023-01-15T14:28:03.851 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:03.841+0000 7fe24cdf6700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-01-15T14:28:04.032 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:03.929+0000 7f51582ea700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-15T14:28:04.033 INFO:tasks.ceph.mgr.y.smithi090.stderr:2023-01-15T14:28:03.845+0000 7fe24cdf6700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-01-15T14:28:04.033 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:03.929+0000 7f956859a700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-15T14:28:04.033 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:03.925+0000 7fe35347c700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-15T14:28:04.034 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:03.929+0000 7fea45892700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-15T14:28:04.034 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:03.925+0000 7f9100e17700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-15T14:28:04.034 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:03.929+0000 7f16ad1be700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-15T14:28:04.034 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:03.925+0000 7f4f3443e700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-15T14:28:04.035 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:03.925+0000 7ffae8ed7700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-01-15T14:28:10.427 DEBUG:teuthology.orchestra.run.smithi090:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-01-15T14:28:10.720 INFO:teuthology.misc.health.smithi090.stdout: 2023-01-15T14:28:10.778 INFO:teuthology.misc.health.smithi090.stdout:{"epoch":16,"fsid":"1c77fb2b-d1a7-4884-a24e-3935e16d0b6b","created":"2023-01-15T14:27:57.735717+0000","modified":"2023-01-15T14:28:07.861798+0000","last_up_change":"2023-01-15T14:28:04.837522+0000","last_in_change":"2023-01-15T14:28:01.797608+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-15T14:28:04.899282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e55c5ac4-3cff-4ee5-8719-373648f83d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":23473},{"type":"v1","addr":"172.21.15.90:6811","nonce":23473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":23473},{"type":"v1","addr":"172.21.15.90:6815","nonce":23473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":23473},{"type":"v1","addr":"172.21.15.90:6823","nonce":23473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":23473},{"type":"v1","addr":"172.21.15.90:6820","nonce":23473}]},"public_addr":"172.21.15.90:6811/23473","cluster_addr":"172.21.15.90:6815/23473","heartbeat_back_addr":"172.21.15.90:6823/23473","heartbeat_front_addr":"172.21.15.90:6820/23473","state":["exists","up"]},{"osd":1,"uuid":"2c1b33c9-0a1e-4734-936d-d51d64fe997a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":23474},{"type":"v1","addr":"172.21.15.90:6801","nonce":23474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":23474},{"type":"v1","addr":"172.21.15.90:6803","nonce":23474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":23474},{"type":"v1","addr":"172.21.15.90:6807","nonce":23474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":23474},{"type":"v1","addr":"172.21.15.90:6805","nonce":23474}]},"public_addr":"172.21.15.90:6801/23474","cluster_addr":"172.21.15.90:6803/23474","heartbeat_back_addr":"172.21.15.90:6807/23474","heartbeat_front_addr":"172.21.15.90:6805/23474","state":["exists","up"]},{"osd":2,"uuid":"b314174a-eca9-4baa-a60c-2589f1fc9a8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":23475},{"type":"v1","addr":"172.21.15.90:6810","nonce":23475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":23475},{"type":"v1","addr":"172.21.15.90:6814","nonce":23475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":23475},{"type":"v1","addr":"172.21.15.90:6821","nonce":23475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":23475},{"type":"v1","addr":"172.21.15.90:6817","nonce":23475}]},"public_addr":"172.21.15.90:6810/23475","cluster_addr":"172.21.15.90:6814/23475","heartbeat_back_addr":"172.21.15.90:6821/23475","heartbeat_front_addr":"172.21.15.90:6817/23475","state":["exists","up"]},{"osd":3,"uuid":"29eb0b17-ed5d-4665-a347-2fb8406d9c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":23778},{"type":"v1","addr":"172.21.15.90:6825","nonce":23778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":23778},{"type":"v1","addr":"172.21.15.90:6827","nonce":23778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":23778},{"type":"v1","addr":"172.21.15.90:6831","nonce":23778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":23778},{"type":"v1","addr":"172.21.15.90:6829","nonce":23778}]},"public_addr":"172.21.15.90:6825/23778","cluster_addr":"172.21.15.90:6827/23778","heartbeat_back_addr":"172.21.15.90:6831/23778","heartbeat_front_addr":"172.21.15.90:6829/23778","state":["exists","up"]},{"osd":4,"uuid":"59362feb-caa9-44e3-b6d0-b79397dacbfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":23523},{"type":"v1","addr":"172.21.15.141:6801","nonce":23523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":23523},{"type":"v1","addr":"172.21.15.141:6803","nonce":23523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":23523},{"type":"v1","addr":"172.21.15.141:6807","nonce":23523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":23523},{"type":"v1","addr":"172.21.15.141:6805","nonce":23523}]},"public_addr":"172.21.15.141:6801/23523","cluster_addr":"172.21.15.141:6803/23523","heartbeat_back_addr":"172.21.15.141:6807/23523","heartbeat_front_addr":"172.21.15.141:6805/23523","state":["exists","up"]},{"osd":5,"uuid":"fbc8821c-8d78-40f8-a571-b18a70d42000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":23525},{"type":"v1","addr":"172.21.15.141:6825","nonce":23525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":23525},{"type":"v1","addr":"172.21.15.141:6827","nonce":23525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":23525},{"type":"v1","addr":"172.21.15.141:6831","nonce":23525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":23525},{"type":"v1","addr":"172.21.15.141:6829","nonce":23525}]},"public_addr":"172.21.15.141:6825/23525","cluster_addr":"172.21.15.141:6827/23525","heartbeat_back_addr":"172.21.15.141:6831/23525","heartbeat_front_addr":"172.21.15.141:6829/23525","state":["exists","up"]},{"osd":6,"uuid":"34550470-3561-492f-b605-b5f0e8052c1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":23524},{"type":"v1","addr":"172.21.15.141:6809","nonce":23524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":23524},{"type":"v1","addr":"172.21.15.141:6811","nonce":23524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":23524},{"type":"v1","addr":"172.21.15.141:6815","nonce":23524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":23524},{"type":"v1","addr":"172.21.15.141:6813","nonce":23524}]},"public_addr":"172.21.15.141:6809/23524","cluster_addr":"172.21.15.141:6811/23524","heartbeat_back_addr":"172.21.15.141:6815/23524","heartbeat_front_addr":"172.21.15.141:6813/23524","state":["exists","up"]},{"osd":7,"uuid":"1f123d0d-d617-4771-b516-21aa5a29b734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":23526},{"type":"v1","addr":"172.21.15.141:6817","nonce":23526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":23526},{"type":"v1","addr":"172.21.15.141:6819","nonce":23526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":23526},{"type":"v1","addr":"172.21.15.141:6823","nonce":23526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":23526},{"type":"v1","addr":"172.21.15.141:6821","nonce":23526}]},"public_addr":"172.21.15.141:6817/23526","cluster_addr":"172.21.15.141:6819/23526","heartbeat_back_addr":"172.21.15.141:6823/23526","heartbeat_front_addr":"172.21.15.141:6821/23526","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.117195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.065464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.138817+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.284776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.271024+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.318787+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.333018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.375590+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-15T14:28:10.780 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-01-15T14:28:10.780 INFO:tasks.ceph:Creating RBD pool 2023-01-15T14:28:10.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd pool create rbd 8 2023-01-15T14:28:11.795 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'rbd' created 2023-01-15T14:28:11.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-01-15T14:28:12.799 INFO:teuthology.orchestra.run.smithi090.stderr:enabled application 'rbd' on pool 'rbd' 2023-01-15T14:28:12.835 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-01-15T14:28:12.836 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-01-15T14:28:12.836 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-01-15T14:28:12.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-01-15T14:28:12.837 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-01-15T14:28:13.191 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:28:13.319 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":4,"active_gid":4107,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6832","nonce":23303},{"type":"v1","addr":"172.21.15.90:6833","nonce":23303}]},"active_addr":"172.21.15.90:6833/23303","active_change":"2023-01-15T14:28:02.826615+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4110,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":3762191053}]},{"addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":3368220733}]},{"addrvec":[{"type":"v2","addr":"172.21.15.90:0","nonce":1359712955}]}]}} 2023-01-15T14:28:13.485 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-01-15T14:28:13.485 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-01-15T14:28:13.485 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-15T14:28:13.788 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:28:13.808 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":18,"fsid":"1c77fb2b-d1a7-4884-a24e-3935e16d0b6b","created":"2023-01-15T14:27:57.735717+0000","modified":"2023-01-15T14:28:12.780305+0000","last_up_change":"2023-01-15T14:28:04.837522+0000","last_in_change":"2023-01-15T14:28:01.797608+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-15T14:28:04.899282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-15T14:28:11.115002+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e55c5ac4-3cff-4ee5-8719-373648f83d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":23473},{"type":"v1","addr":"172.21.15.90:6811","nonce":23473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":23473},{"type":"v1","addr":"172.21.15.90:6815","nonce":23473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":23473},{"type":"v1","addr":"172.21.15.90:6823","nonce":23473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":23473},{"type":"v1","addr":"172.21.15.90:6820","nonce":23473}]},"public_addr":"172.21.15.90:6811/23473","cluster_addr":"172.21.15.90:6815/23473","heartbeat_back_addr":"172.21.15.90:6823/23473","heartbeat_front_addr":"172.21.15.90:6820/23473","state":["exists","up"]},{"osd":1,"uuid":"2c1b33c9-0a1e-4734-936d-d51d64fe997a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":23474},{"type":"v1","addr":"172.21.15.90:6801","nonce":23474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":23474},{"type":"v1","addr":"172.21.15.90:6803","nonce":23474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":23474},{"type":"v1","addr":"172.21.15.90:6807","nonce":23474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":23474},{"type":"v1","addr":"172.21.15.90:6805","nonce":23474}]},"public_addr":"172.21.15.90:6801/23474","cluster_addr":"172.21.15.90:6803/23474","heartbeat_back_addr":"172.21.15.90:6807/23474","heartbeat_front_addr":"172.21.15.90:6805/23474","state":["exists","up"]},{"osd":2,"uuid":"b314174a-eca9-4baa-a60c-2589f1fc9a8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":23475},{"type":"v1","addr":"172.21.15.90:6810","nonce":23475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":23475},{"type":"v1","addr":"172.21.15.90:6814","nonce":23475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":23475},{"type":"v1","addr":"172.21.15.90:6821","nonce":23475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":23475},{"type":"v1","addr":"172.21.15.90:6817","nonce":23475}]},"public_addr":"172.21.15.90:6810/23475","cluster_addr":"172.21.15.90:6814/23475","heartbeat_back_addr":"172.21.15.90:6821/23475","heartbeat_front_addr":"172.21.15.90:6817/23475","state":["exists","up"]},{"osd":3,"uuid":"29eb0b17-ed5d-4665-a347-2fb8406d9c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":23778},{"type":"v1","addr":"172.21.15.90:6825","nonce":23778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":23778},{"type":"v1","addr":"172.21.15.90:6827","nonce":23778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":23778},{"type":"v1","addr":"172.21.15.90:6831","nonce":23778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":23778},{"type":"v1","addr":"172.21.15.90:6829","nonce":23778}]},"public_addr":"172.21.15.90:6825/23778","cluster_addr":"172.21.15.90:6827/23778","heartbeat_back_addr":"172.21.15.90:6831/23778","heartbeat_front_addr":"172.21.15.90:6829/23778","state":["exists","up"]},{"osd":4,"uuid":"59362feb-caa9-44e3-b6d0-b79397dacbfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":23523},{"type":"v1","addr":"172.21.15.141:6801","nonce":23523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":23523},{"type":"v1","addr":"172.21.15.141:6803","nonce":23523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":23523},{"type":"v1","addr":"172.21.15.141:6807","nonce":23523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":23523},{"type":"v1","addr":"172.21.15.141:6805","nonce":23523}]},"public_addr":"172.21.15.141:6801/23523","cluster_addr":"172.21.15.141:6803/23523","heartbeat_back_addr":"172.21.15.141:6807/23523","heartbeat_front_addr":"172.21.15.141:6805/23523","state":["exists","up"]},{"osd":5,"uuid":"fbc8821c-8d78-40f8-a571-b18a70d42000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":23525},{"type":"v1","addr":"172.21.15.141:6825","nonce":23525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":23525},{"type":"v1","addr":"172.21.15.141:6827","nonce":23525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":23525},{"type":"v1","addr":"172.21.15.141:6831","nonce":23525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":23525},{"type":"v1","addr":"172.21.15.141:6829","nonce":23525}]},"public_addr":"172.21.15.141:6825/23525","cluster_addr":"172.21.15.141:6827/23525","heartbeat_back_addr":"172.21.15.141:6831/23525","heartbeat_front_addr":"172.21.15.141:6829/23525","state":["exists","up"]},{"osd":6,"uuid":"34550470-3561-492f-b605-b5f0e8052c1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":23524},{"type":"v1","addr":"172.21.15.141:6809","nonce":23524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":23524},{"type":"v1","addr":"172.21.15.141:6811","nonce":23524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":23524},{"type":"v1","addr":"172.21.15.141:6815","nonce":23524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":23524},{"type":"v1","addr":"172.21.15.141:6813","nonce":23524}]},"public_addr":"172.21.15.141:6809/23524","cluster_addr":"172.21.15.141:6811/23524","heartbeat_back_addr":"172.21.15.141:6815/23524","heartbeat_front_addr":"172.21.15.141:6813/23524","state":["exists","up"]},{"osd":7,"uuid":"1f123d0d-d617-4771-b516-21aa5a29b734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":23526},{"type":"v1","addr":"172.21.15.141:6817","nonce":23526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":23526},{"type":"v1","addr":"172.21.15.141:6819","nonce":23526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":23526},{"type":"v1","addr":"172.21.15.141:6823","nonce":23526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":23526},{"type":"v1","addr":"172.21.15.141:6821","nonce":23526}]},"public_addr":"172.21.15.141:6817/23526","cluster_addr":"172.21.15.141:6819/23526","heartbeat_back_addr":"172.21.15.141:6823/23526","heartbeat_front_addr":"172.21.15.141:6821/23526","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.117195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.065464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.138817+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.284776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.271024+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.318787+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.333018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.375590+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-15T14:28:13.810 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-01-15T14:28:13.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-01-15T14:28:14.096 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:28:14.244 INFO:teuthology.orchestra.run.smithi090.stdout:{"epoch":19,"fsid":"1c77fb2b-d1a7-4884-a24e-3935e16d0b6b","created":"2023-01-15T14:27:57.735717+0000","modified":"2023-01-15T14:28:13.795231+0000","last_up_change":"2023-01-15T14:28:04.837522+0000","last_in_change":"2023-01-15T14:28:01.797608+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-01-15T14:28:04.899282+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-01-15T14:28:11.115002+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"e55c5ac4-3cff-4ee5-8719-373648f83d8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6808","nonce":23473},{"type":"v1","addr":"172.21.15.90:6811","nonce":23473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6813","nonce":23473},{"type":"v1","addr":"172.21.15.90:6815","nonce":23473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6822","nonce":23473},{"type":"v1","addr":"172.21.15.90:6823","nonce":23473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6818","nonce":23473},{"type":"v1","addr":"172.21.15.90:6820","nonce":23473}]},"public_addr":"172.21.15.90:6811/23473","cluster_addr":"172.21.15.90:6815/23473","heartbeat_back_addr":"172.21.15.90:6823/23473","heartbeat_front_addr":"172.21.15.90:6820/23473","state":["exists","up"]},{"osd":1,"uuid":"2c1b33c9-0a1e-4734-936d-d51d64fe997a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6800","nonce":23474},{"type":"v1","addr":"172.21.15.90:6801","nonce":23474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6802","nonce":23474},{"type":"v1","addr":"172.21.15.90:6803","nonce":23474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6806","nonce":23474},{"type":"v1","addr":"172.21.15.90:6807","nonce":23474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6804","nonce":23474},{"type":"v1","addr":"172.21.15.90:6805","nonce":23474}]},"public_addr":"172.21.15.90:6801/23474","cluster_addr":"172.21.15.90:6803/23474","heartbeat_back_addr":"172.21.15.90:6807/23474","heartbeat_front_addr":"172.21.15.90:6805/23474","state":["exists","up"]},{"osd":2,"uuid":"b314174a-eca9-4baa-a60c-2589f1fc9a8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6809","nonce":23475},{"type":"v1","addr":"172.21.15.90:6810","nonce":23475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6812","nonce":23475},{"type":"v1","addr":"172.21.15.90:6814","nonce":23475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6819","nonce":23475},{"type":"v1","addr":"172.21.15.90:6821","nonce":23475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6816","nonce":23475},{"type":"v1","addr":"172.21.15.90:6817","nonce":23475}]},"public_addr":"172.21.15.90:6810/23475","cluster_addr":"172.21.15.90:6814/23475","heartbeat_back_addr":"172.21.15.90:6821/23475","heartbeat_front_addr":"172.21.15.90:6817/23475","state":["exists","up"]},{"osd":3,"uuid":"29eb0b17-ed5d-4665-a347-2fb8406d9c62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6824","nonce":23778},{"type":"v1","addr":"172.21.15.90:6825","nonce":23778}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6826","nonce":23778},{"type":"v1","addr":"172.21.15.90:6827","nonce":23778}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6830","nonce":23778},{"type":"v1","addr":"172.21.15.90:6831","nonce":23778}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.90:6828","nonce":23778},{"type":"v1","addr":"172.21.15.90:6829","nonce":23778}]},"public_addr":"172.21.15.90:6825/23778","cluster_addr":"172.21.15.90:6827/23778","heartbeat_back_addr":"172.21.15.90:6831/23778","heartbeat_front_addr":"172.21.15.90:6829/23778","state":["exists","up"]},{"osd":4,"uuid":"59362feb-caa9-44e3-b6d0-b79397dacbfc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6800","nonce":23523},{"type":"v1","addr":"172.21.15.141:6801","nonce":23523}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6802","nonce":23523},{"type":"v1","addr":"172.21.15.141:6803","nonce":23523}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6806","nonce":23523},{"type":"v1","addr":"172.21.15.141:6807","nonce":23523}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6804","nonce":23523},{"type":"v1","addr":"172.21.15.141:6805","nonce":23523}]},"public_addr":"172.21.15.141:6801/23523","cluster_addr":"172.21.15.141:6803/23523","heartbeat_back_addr":"172.21.15.141:6807/23523","heartbeat_front_addr":"172.21.15.141:6805/23523","state":["exists","up"]},{"osd":5,"uuid":"fbc8821c-8d78-40f8-a571-b18a70d42000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6824","nonce":23525},{"type":"v1","addr":"172.21.15.141:6825","nonce":23525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6826","nonce":23525},{"type":"v1","addr":"172.21.15.141:6827","nonce":23525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6830","nonce":23525},{"type":"v1","addr":"172.21.15.141:6831","nonce":23525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6828","nonce":23525},{"type":"v1","addr":"172.21.15.141:6829","nonce":23525}]},"public_addr":"172.21.15.141:6825/23525","cluster_addr":"172.21.15.141:6827/23525","heartbeat_back_addr":"172.21.15.141:6831/23525","heartbeat_front_addr":"172.21.15.141:6829/23525","state":["exists","up"]},{"osd":6,"uuid":"34550470-3561-492f-b605-b5f0e8052c1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6808","nonce":23524},{"type":"v1","addr":"172.21.15.141:6809","nonce":23524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6810","nonce":23524},{"type":"v1","addr":"172.21.15.141:6811","nonce":23524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6814","nonce":23524},{"type":"v1","addr":"172.21.15.141:6815","nonce":23524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6812","nonce":23524},{"type":"v1","addr":"172.21.15.141:6813","nonce":23524}]},"public_addr":"172.21.15.141:6809/23524","cluster_addr":"172.21.15.141:6811/23524","heartbeat_back_addr":"172.21.15.141:6815/23524","heartbeat_front_addr":"172.21.15.141:6813/23524","state":["exists","up"]},{"osd":7,"uuid":"1f123d0d-d617-4771-b516-21aa5a29b734","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6816","nonce":23526},{"type":"v1","addr":"172.21.15.141:6817","nonce":23526}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6818","nonce":23526},{"type":"v1","addr":"172.21.15.141:6819","nonce":23526}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6822","nonce":23526},{"type":"v1","addr":"172.21.15.141:6823","nonce":23526}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.141:6820","nonce":23526},{"type":"v1","addr":"172.21.15.141:6821","nonce":23526}]},"public_addr":"172.21.15.141:6817/23526","cluster_addr":"172.21.15.141:6819/23526","heartbeat_back_addr":"172.21.15.141:6823/23526","heartbeat_front_addr":"172.21.15.141:6821/23526","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.117195+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.065464+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.138817+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.284776+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.271024+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.318787+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.333018+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-01-15T14:28:03.375590+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-01-15T14:28:14.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-01-15T14:28:14.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-01-15T14:28:14.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-01-15T14:28:14.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-01-15T14:28:14.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-01-15T14:28:14.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-01-15T14:28:14.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-01-15T14:28:14.252 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-01-15T14:28:14.454 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:14.688 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-01-15T14:28:14.689 INFO:teuthology.orchestra.run.smithi090.stdout:55834574852 2023-01-15T14:28:14.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-15T14:28:14.690 INFO:teuthology.orchestra.run.smithi090.stdout:55834574852 2023-01-15T14:28:14.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-15T14:28:14.692 INFO:teuthology.orchestra.run.smithi090.stdout:55834574852 2023-01-15T14:28:14.692 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-01-15T14:28:14.693 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:14.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-15T14:28:14.694 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:14.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-01-15T14:28:14.695 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:14.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-01-15T14:28:14.696 INFO:teuthology.orchestra.run.smithi090.stdout:55834574852 2023-01-15T14:28:14.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-01-15T14:28:15.215 INFO:teuthology.orchestra.run.smithi090.stdout:55834574850 2023-01-15T14:28:15.307 INFO:teuthology.orchestra.run.smithi090.stdout:55834574850 2023-01-15T14:28:15.308 INFO:teuthology.orchestra.run.smithi090.stdout:55834574850 2023-01-15T14:28:15.309 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:15.309 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:15.310 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:15.310 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:15.311 INFO:teuthology.orchestra.run.smithi090.stdout:55834574850 2023-01-15T14:28:15.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.4 2023-01-15T14:28:15.319 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.3 2023-01-15T14:28:15.320 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2023-01-15T14:28:15.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.2 2023-01-15T14:28:15.322 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2023-01-15T14:28:15.323 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2023-01-15T14:28:15.323 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.7 2023-01-15T14:28:15.324 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.1 2023-01-15T14:28:16.318 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-01-15T14:28:16.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-01-15T14:28:16.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-01-15T14:28:16.321 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-01-15T14:28:16.322 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-01-15T14:28:16.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-01-15T14:28:16.324 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-01-15T14:28:16.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-01-15T14:28:16.776 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:16.894 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:16.895 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.4 2023-01-15T14:28:16.901 DEBUG:teuthology.parallel:result is None 2023-01-15T14:28:16.902 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.3 2023-01-15T14:28:16.902 INFO:teuthology.orchestra.run.smithi090.stdout:55834574852 2023-01-15T14:28:16.903 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:16.904 DEBUG:teuthology.parallel:result is None 2023-01-15T14:28:16.907 INFO:teuthology.orchestra.run.smithi090.stdout:55834574852 2023-01-15T14:28:16.908 INFO:teuthology.orchestra.run.smithi090.stdout:55834574852 2023-01-15T14:28:16.909 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.2 2023-01-15T14:28:16.911 DEBUG:teuthology.parallel:result is None 2023-01-15T14:28:16.911 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.1 2023-01-15T14:28:16.911 INFO:teuthology.orchestra.run.smithi090.stdout:55834574851 2023-01-15T14:28:16.912 DEBUG:teuthology.parallel:result is None 2023-01-15T14:28:16.912 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.5 2023-01-15T14:28:16.913 INFO:teuthology.orchestra.run.smithi090.stdout:55834574852 2023-01-15T14:28:16.913 DEBUG:teuthology.parallel:result is None 2023-01-15T14:28:16.915 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.0 2023-01-15T14:28:17.051 DEBUG:teuthology.parallel:result is None 2023-01-15T14:28:17.051 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.7 2023-01-15T14:28:17.052 DEBUG:teuthology.parallel:result is None 2023-01-15T14:28:17.052 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2023-01-15T14:28:17.052 DEBUG:teuthology.parallel:result is None 2023-01-15T14:28:17.052 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-15T14:28:17.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:28:17.334 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:28:17.375 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:28:17.376 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-01-15T14:28:16.842816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6370468,"kb_used_data":743226480,"kb_used_omap":510,"kb_used_meta":0,"kb_avail":743224572,"statfs":{"total":767585320960,"available":761061961728,"internally_reserved":1953792,"allocated":761063915520,"data_stored":761063915520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":522946,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":327696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":37,"num_write":57,"num_write_kb":980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001544"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.912524+0000","last_change":"2023-01-15T14:28:12.896363+0000","last_active":"2023-01-15T14:28:13.912524+0000","last_peered":"2023-01-15T14:28:13.912524+0000","last_clean":"2023-01-15T14:28:13.912524+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:28:13.912524+0000","last_undegraded":"2023-01-15T14:28:13.912524+0000","last_fullsized":"2023-01-15T14:28:13.912524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.332","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431977+0000","last_change":"2023-01-15T14:28:13.217588+0000","last_active":"2023-01-15T14:28:14.431977+0000","last_peered":"2023-01-15T14:28:14.431977+0000","last_clean":"2023-01-15T14:28:14.431977+0000","last_became_active":"2023-01-15T14:28:13.216947+0000","last_became_peered":"2023-01-15T14:28:13.216947+0000","last_unstale":"2023-01-15T14:28:14.431977+0000","last_undegraded":"2023-01-15T14:28:14.431977+0000","last_fullsized":"2023-01-15T14:28:14.431977+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.810","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.920710+0000","last_change":"2023-01-15T14:28:13.318466+0000","last_active":"2023-01-15T14:28:13.920710+0000","last_peered":"2023-01-15T14:28:13.920710+0000","last_clean":"2023-01-15T14:28:13.920710+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:28:13.920710+0000","last_undegraded":"2023-01-15T14:28:13.920710+0000","last_fullsized":"2023-01-15T14:28:13.920710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.665","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431923+0000","last_change":"2023-01-15T14:28:12.897110+0000","last_active":"2023-01-15T14:28:14.431923+0000","last_peered":"2023-01-15T14:28:14.431923+0000","last_clean":"2023-01-15T14:28:14.431923+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:28:14.431923+0000","last_undegraded":"2023-01-15T14:28:14.431923+0000","last_fullsized":"2023-01-15T14:28:14.431923+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:27.569","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.438810+0000","last_change":"2023-01-15T14:28:12.993663+0000","last_active":"2023-01-15T14:28:14.438810+0000","last_peered":"2023-01-15T14:28:14.438810+0000","last_clean":"2023-01-15T14:28:14.438810+0000","last_became_active":"2023-01-15T14:28:12.993023+0000","last_became_peered":"2023-01-15T14:28:12.993023+0000","last_unstale":"2023-01-15T14:28:14.438810+0000","last_undegraded":"2023-01-15T14:28:14.438810+0000","last_fullsized":"2023-01-15T14:28:14.438810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:31.082","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431994+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:28:14.431994+0000","last_peered":"2023-01-15T14:28:14.431994+0000","last_clean":"2023-01-15T14:28:14.431994+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:28:14.431994+0000","last_undegraded":"2023-01-15T14:28:14.431994+0000","last_fullsized":"2023-01-15T14:28:14.431994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.689","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.820597+0000","last_change":"2023-01-15T14:28:13.013616+0000","last_active":"2023-01-15T14:28:13.820597+0000","last_peered":"2023-01-15T14:28:13.820597+0000","last_clean":"2023-01-15T14:28:13.820597+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:28:13.820597+0000","last_undegraded":"2023-01-15T14:28:13.820597+0000","last_fullsized":"2023-01-15T14:28:13.820597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:29.249","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.533586+0000","last_change":"2023-01-15T14:28:12.893975+0000","last_active":"2023-01-15T14:28:14.533586+0000","last_peered":"2023-01-15T14:28:14.533586+0000","last_clean":"2023-01-15T14:28:14.533586+0000","last_became_active":"2023-01-15T14:28:12.893100+0000","last_became_peered":"2023-01-15T14:28:12.893100+0000","last_unstale":"2023-01-15T14:28:14.533586+0000","last_undegraded":"2023-01-15T14:28:14.533586+0000","last_fullsized":"2023-01-15T14:28:14.533586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.496","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'61","reported_seq":96,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.739397+0000","last_change":"2023-01-15T14:28:07.312081+0000","last_active":"2023-01-15T14:28:14.739397+0000","last_peered":"2023-01-15T14:28:14.739397+0000","last_clean":"2023-01-15T14:28:14.739397+0000","last_became_active":"2023-01-15T14:28:07.309738+0000","last_became_peered":"2023-01-15T14:28:07.309738+0000","last_unstale":"2023-01-15T14:28:14.739397+0000","last_undegraded":"2023-01-15T14:28:14.739397+0000","last_fullsized":"2023-01-15T14:28:14.739397+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:25.933","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902868,"kb_used_omap":97,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":495616,"allocated":95132536832,"data_stored":95132536832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":100042,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.457999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.370999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.516999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797204,"kb_used_data":92902868,"kb_used_omap":90,"kb_used_meta":0,"kb_avail":92902176,"statfs":{"total":95948165120,"available":95131828224,"internally_reserved":708608,"allocated":95132536832,"data_stored":95132536832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":93103,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795848,"kb_used_data":92903572,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903532,"statfs":{"total":95948165120,"available":95133216768,"internally_reserved":40960,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43962,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795856,"kb_used_data":92903572,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903524,"statfs":{"total":95948165120,"available":95133208576,"internally_reserved":49152,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43830,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795820,"kb_used_data":92903584,"kb_used_omap":37,"kb_used_meta":0,"kb_avail":92903560,"statfs":{"total":95948165120,"available":95133245440,"internally_reserved":24576,"allocated":95133270016,"data_stored":95133270016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38504,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795868,"kb_used_data":92903572,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":92903512,"statfs":{"total":95948165120,"available":95133196288,"internally_reserved":61440,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53724,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.124000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795880,"kb_used_data":92903572,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92903500,"statfs":{"total":95948165120,"available":95133184000,"internally_reserved":73728,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54652,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902872,"kb_used_omap":92,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":499712,"allocated":95132540928,"data_stored":95132540928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95129,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.633}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]}]}],"pool_statfs":[]}} 2023-01-15T14:28:17.378 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:28:17.655 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:28:17.706 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:28:17.706 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-01-15T14:28:16.842816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6370468,"kb_used_data":743226480,"kb_used_omap":510,"kb_used_meta":0,"kb_avail":743224572,"statfs":{"total":767585320960,"available":761061961728,"internally_reserved":1953792,"allocated":761063915520,"data_stored":761063915520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":522946,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":327696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":37,"num_write":57,"num_write_kb":980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001544"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.912524+0000","last_change":"2023-01-15T14:28:12.896363+0000","last_active":"2023-01-15T14:28:13.912524+0000","last_peered":"2023-01-15T14:28:13.912524+0000","last_clean":"2023-01-15T14:28:13.912524+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:28:13.912524+0000","last_undegraded":"2023-01-15T14:28:13.912524+0000","last_fullsized":"2023-01-15T14:28:13.912524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.332","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431977+0000","last_change":"2023-01-15T14:28:13.217588+0000","last_active":"2023-01-15T14:28:14.431977+0000","last_peered":"2023-01-15T14:28:14.431977+0000","last_clean":"2023-01-15T14:28:14.431977+0000","last_became_active":"2023-01-15T14:28:13.216947+0000","last_became_peered":"2023-01-15T14:28:13.216947+0000","last_unstale":"2023-01-15T14:28:14.431977+0000","last_undegraded":"2023-01-15T14:28:14.431977+0000","last_fullsized":"2023-01-15T14:28:14.431977+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.810","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.920710+0000","last_change":"2023-01-15T14:28:13.318466+0000","last_active":"2023-01-15T14:28:13.920710+0000","last_peered":"2023-01-15T14:28:13.920710+0000","last_clean":"2023-01-15T14:28:13.920710+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:28:13.920710+0000","last_undegraded":"2023-01-15T14:28:13.920710+0000","last_fullsized":"2023-01-15T14:28:13.920710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.665","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431923+0000","last_change":"2023-01-15T14:28:12.897110+0000","last_active":"2023-01-15T14:28:14.431923+0000","last_peered":"2023-01-15T14:28:14.431923+0000","last_clean":"2023-01-15T14:28:14.431923+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:28:14.431923+0000","last_undegraded":"2023-01-15T14:28:14.431923+0000","last_fullsized":"2023-01-15T14:28:14.431923+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:27.569","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.438810+0000","last_change":"2023-01-15T14:28:12.993663+0000","last_active":"2023-01-15T14:28:14.438810+0000","last_peered":"2023-01-15T14:28:14.438810+0000","last_clean":"2023-01-15T14:28:14.438810+0000","last_became_active":"2023-01-15T14:28:12.993023+0000","last_became_peered":"2023-01-15T14:28:12.993023+0000","last_unstale":"2023-01-15T14:28:14.438810+0000","last_undegraded":"2023-01-15T14:28:14.438810+0000","last_fullsized":"2023-01-15T14:28:14.438810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:31.082","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431994+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:28:14.431994+0000","last_peered":"2023-01-15T14:28:14.431994+0000","last_clean":"2023-01-15T14:28:14.431994+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:28:14.431994+0000","last_undegraded":"2023-01-15T14:28:14.431994+0000","last_fullsized":"2023-01-15T14:28:14.431994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.689","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.820597+0000","last_change":"2023-01-15T14:28:13.013616+0000","last_active":"2023-01-15T14:28:13.820597+0000","last_peered":"2023-01-15T14:28:13.820597+0000","last_clean":"2023-01-15T14:28:13.820597+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:28:13.820597+0000","last_undegraded":"2023-01-15T14:28:13.820597+0000","last_fullsized":"2023-01-15T14:28:13.820597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:29.249","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.533586+0000","last_change":"2023-01-15T14:28:12.893975+0000","last_active":"2023-01-15T14:28:14.533586+0000","last_peered":"2023-01-15T14:28:14.533586+0000","last_clean":"2023-01-15T14:28:14.533586+0000","last_became_active":"2023-01-15T14:28:12.893100+0000","last_became_peered":"2023-01-15T14:28:12.893100+0000","last_unstale":"2023-01-15T14:28:14.533586+0000","last_undegraded":"2023-01-15T14:28:14.533586+0000","last_fullsized":"2023-01-15T14:28:14.533586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.496","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'61","reported_seq":96,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.739397+0000","last_change":"2023-01-15T14:28:07.312081+0000","last_active":"2023-01-15T14:28:14.739397+0000","last_peered":"2023-01-15T14:28:14.739397+0000","last_clean":"2023-01-15T14:28:14.739397+0000","last_became_active":"2023-01-15T14:28:07.309738+0000","last_became_peered":"2023-01-15T14:28:07.309738+0000","last_unstale":"2023-01-15T14:28:14.739397+0000","last_undegraded":"2023-01-15T14:28:14.739397+0000","last_fullsized":"2023-01-15T14:28:14.739397+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:25.933","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902868,"kb_used_omap":97,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":495616,"allocated":95132536832,"data_stored":95132536832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":100042,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.457999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.370999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.516999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797204,"kb_used_data":92902868,"kb_used_omap":90,"kb_used_meta":0,"kb_avail":92902176,"statfs":{"total":95948165120,"available":95131828224,"internally_reserved":708608,"allocated":95132536832,"data_stored":95132536832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":93103,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795848,"kb_used_data":92903572,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903532,"statfs":{"total":95948165120,"available":95133216768,"internally_reserved":40960,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43962,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795856,"kb_used_data":92903572,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903524,"statfs":{"total":95948165120,"available":95133208576,"internally_reserved":49152,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43830,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795820,"kb_used_data":92903584,"kb_used_omap":37,"kb_used_meta":0,"kb_avail":92903560,"statfs":{"total":95948165120,"available":95133245440,"internally_reserved":24576,"allocated":95133270016,"data_stored":95133270016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38504,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795868,"kb_used_data":92903572,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":92903512,"statfs":{"total":95948165120,"available":95133196288,"internally_reserved":61440,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53724,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.124000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795880,"kb_used_data":92903572,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92903500,"statfs":{"total":95948165120,"available":95133184000,"internally_reserved":73728,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54652,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902872,"kb_used_omap":92,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":499712,"allocated":95132540928,"data_stored":95132540928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95129,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.633}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]}]}],"pool_statfs":[]}} 2023-01-15T14:28:17.847 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-15T14:28:17.848 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-01-15T14:28:17.848 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-01-15T14:28:17.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-01-15T14:28:18.167 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:28:18.167 INFO:teuthology.orchestra.run.smithi090.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-01-15T14:28:18.180 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-01-15T14:28:18.180 INFO:teuthology.run_tasks:Running task thrashosds... 2023-01-15T14:28:18.197 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 2, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2023-01-15T14:28:18.332 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-rf-newsched-v3'}, 'ceph': {'conf': {'global': {'mon election default strategy': 3, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'filestore odsync write': True, 'osd backoff on degraded': True, 'osd backoff on peering': True, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd max backfills': 2, 'osd max markdown count': 1000, 'osd max scrubs': 3, 'osd mclock override recovery settings': True, 'osd objectstore': 'filestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd sloppy crc': True, 'osd snap trim sleep': 0.5}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': 'a21b2f4fd40d883dac62c8cd315cbb54848ee067'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a21b2f4fd40d883dac62c8cd315cbb54848ee067'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'wip-rf-newsched-v3', 'sha1': 'a21b2f4fd40d883dac62c8cd315cbb54848ee067'}} 2023-01-15T14:28:18.332 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 2, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'min_in': 4} 2023-01-15T14:28:18.332 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2023-01-15T14:28:18.642 INFO:tasks.thrashosds:Beginning thrashosds... 2023-01-15T14:28:18.650 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-01-15T14:28:18.650 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:28:18.933 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:28:18.965 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:28:18.966 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-01-15T14:28:18.843241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6370468,"kb_used_data":743226480,"kb_used_omap":510,"kb_used_meta":0,"kb_avail":743224572,"statfs":{"total":767585320960,"available":761061961728,"internally_reserved":1953792,"allocated":761063915520,"data_stored":761063915520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":522946,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":327696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":37,"num_write":57,"num_write_kb":980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.059022"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.912524+0000","last_change":"2023-01-15T14:28:12.896363+0000","last_active":"2023-01-15T14:28:13.912524+0000","last_peered":"2023-01-15T14:28:13.912524+0000","last_clean":"2023-01-15T14:28:13.912524+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:28:13.912524+0000","last_undegraded":"2023-01-15T14:28:13.912524+0000","last_fullsized":"2023-01-15T14:28:13.912524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.332","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431977+0000","last_change":"2023-01-15T14:28:13.217588+0000","last_active":"2023-01-15T14:28:14.431977+0000","last_peered":"2023-01-15T14:28:14.431977+0000","last_clean":"2023-01-15T14:28:14.431977+0000","last_became_active":"2023-01-15T14:28:13.216947+0000","last_became_peered":"2023-01-15T14:28:13.216947+0000","last_unstale":"2023-01-15T14:28:14.431977+0000","last_undegraded":"2023-01-15T14:28:14.431977+0000","last_fullsized":"2023-01-15T14:28:14.431977+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.810","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.920710+0000","last_change":"2023-01-15T14:28:13.318466+0000","last_active":"2023-01-15T14:28:13.920710+0000","last_peered":"2023-01-15T14:28:13.920710+0000","last_clean":"2023-01-15T14:28:13.920710+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:28:13.920710+0000","last_undegraded":"2023-01-15T14:28:13.920710+0000","last_fullsized":"2023-01-15T14:28:13.920710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.665","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431923+0000","last_change":"2023-01-15T14:28:12.897110+0000","last_active":"2023-01-15T14:28:14.431923+0000","last_peered":"2023-01-15T14:28:14.431923+0000","last_clean":"2023-01-15T14:28:14.431923+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:28:14.431923+0000","last_undegraded":"2023-01-15T14:28:14.431923+0000","last_fullsized":"2023-01-15T14:28:14.431923+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:27.569","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.438810+0000","last_change":"2023-01-15T14:28:12.993663+0000","last_active":"2023-01-15T14:28:14.438810+0000","last_peered":"2023-01-15T14:28:14.438810+0000","last_clean":"2023-01-15T14:28:14.438810+0000","last_became_active":"2023-01-15T14:28:12.993023+0000","last_became_peered":"2023-01-15T14:28:12.993023+0000","last_unstale":"2023-01-15T14:28:14.438810+0000","last_undegraded":"2023-01-15T14:28:14.438810+0000","last_fullsized":"2023-01-15T14:28:14.438810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:31.082","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431994+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:28:14.431994+0000","last_peered":"2023-01-15T14:28:14.431994+0000","last_clean":"2023-01-15T14:28:14.431994+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:28:14.431994+0000","last_undegraded":"2023-01-15T14:28:14.431994+0000","last_fullsized":"2023-01-15T14:28:14.431994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.689","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.820597+0000","last_change":"2023-01-15T14:28:13.013616+0000","last_active":"2023-01-15T14:28:13.820597+0000","last_peered":"2023-01-15T14:28:13.820597+0000","last_clean":"2023-01-15T14:28:13.820597+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:28:13.820597+0000","last_undegraded":"2023-01-15T14:28:13.820597+0000","last_fullsized":"2023-01-15T14:28:13.820597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:29.249","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.533586+0000","last_change":"2023-01-15T14:28:12.893975+0000","last_active":"2023-01-15T14:28:14.533586+0000","last_peered":"2023-01-15T14:28:14.533586+0000","last_clean":"2023-01-15T14:28:14.533586+0000","last_became_active":"2023-01-15T14:28:12.893100+0000","last_became_peered":"2023-01-15T14:28:12.893100+0000","last_unstale":"2023-01-15T14:28:14.533586+0000","last_undegraded":"2023-01-15T14:28:14.533586+0000","last_fullsized":"2023-01-15T14:28:14.533586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.496","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'61","reported_seq":96,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.739397+0000","last_change":"2023-01-15T14:28:07.312081+0000","last_active":"2023-01-15T14:28:14.739397+0000","last_peered":"2023-01-15T14:28:14.739397+0000","last_clean":"2023-01-15T14:28:14.739397+0000","last_became_active":"2023-01-15T14:28:07.309738+0000","last_became_peered":"2023-01-15T14:28:07.309738+0000","last_unstale":"2023-01-15T14:28:14.739397+0000","last_undegraded":"2023-01-15T14:28:14.739397+0000","last_fullsized":"2023-01-15T14:28:14.739397+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:25.933","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902868,"kb_used_omap":97,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":495616,"allocated":95132536832,"data_stored":95132536832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":100042,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.457999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.370999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.516999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797204,"kb_used_data":92902868,"kb_used_omap":90,"kb_used_meta":0,"kb_avail":92902176,"statfs":{"total":95948165120,"available":95131828224,"internally_reserved":708608,"allocated":95132536832,"data_stored":95132536832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":93103,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795848,"kb_used_data":92903572,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903532,"statfs":{"total":95948165120,"available":95133216768,"internally_reserved":40960,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43962,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795856,"kb_used_data":92903572,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903524,"statfs":{"total":95948165120,"available":95133208576,"internally_reserved":49152,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43830,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795820,"kb_used_data":92903584,"kb_used_omap":37,"kb_used_meta":0,"kb_avail":92903560,"statfs":{"total":95948165120,"available":95133245440,"internally_reserved":24576,"allocated":95133270016,"data_stored":95133270016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38504,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795868,"kb_used_data":92903572,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":92903512,"statfs":{"total":95948165120,"available":95133196288,"internally_reserved":61440,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53724,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.124000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795880,"kb_used_data":92903572,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92903500,"statfs":{"total":95948165120,"available":95133184000,"internally_reserved":73728,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54652,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902872,"kb_used_omap":92,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":499712,"allocated":95132540928,"data_stored":95132540928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95129,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.633}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]}]}],"pool_statfs":[]}} 2023-01-15T14:28:18.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:28:19.275 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:28:19.435 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:28:19.436 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-01-15T14:28:18.843241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6370468,"kb_used_data":743226480,"kb_used_omap":510,"kb_used_meta":0,"kb_avail":743224572,"statfs":{"total":767585320960,"available":761061961728,"internally_reserved":1953792,"allocated":761063915520,"data_stored":761063915520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":522946,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":327696,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":37,"num_write":57,"num_write_kb":980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.059022"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.912524+0000","last_change":"2023-01-15T14:28:12.896363+0000","last_active":"2023-01-15T14:28:13.912524+0000","last_peered":"2023-01-15T14:28:13.912524+0000","last_clean":"2023-01-15T14:28:13.912524+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:28:13.912524+0000","last_undegraded":"2023-01-15T14:28:13.912524+0000","last_fullsized":"2023-01-15T14:28:13.912524+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.332","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431977+0000","last_change":"2023-01-15T14:28:13.217588+0000","last_active":"2023-01-15T14:28:14.431977+0000","last_peered":"2023-01-15T14:28:14.431977+0000","last_clean":"2023-01-15T14:28:14.431977+0000","last_became_active":"2023-01-15T14:28:13.216947+0000","last_became_peered":"2023-01-15T14:28:13.216947+0000","last_unstale":"2023-01-15T14:28:14.431977+0000","last_undegraded":"2023-01-15T14:28:14.431977+0000","last_fullsized":"2023-01-15T14:28:14.431977+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.810","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.920710+0000","last_change":"2023-01-15T14:28:13.318466+0000","last_active":"2023-01-15T14:28:13.920710+0000","last_peered":"2023-01-15T14:28:13.920710+0000","last_clean":"2023-01-15T14:28:13.920710+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:28:13.920710+0000","last_undegraded":"2023-01-15T14:28:13.920710+0000","last_fullsized":"2023-01-15T14:28:13.920710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.665","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431923+0000","last_change":"2023-01-15T14:28:12.897110+0000","last_active":"2023-01-15T14:28:14.431923+0000","last_peered":"2023-01-15T14:28:14.431923+0000","last_clean":"2023-01-15T14:28:14.431923+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:28:14.431923+0000","last_undegraded":"2023-01-15T14:28:14.431923+0000","last_fullsized":"2023-01-15T14:28:14.431923+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:27.569","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.438810+0000","last_change":"2023-01-15T14:28:12.993663+0000","last_active":"2023-01-15T14:28:14.438810+0000","last_peered":"2023-01-15T14:28:14.438810+0000","last_clean":"2023-01-15T14:28:14.438810+0000","last_became_active":"2023-01-15T14:28:12.993023+0000","last_became_peered":"2023-01-15T14:28:12.993023+0000","last_unstale":"2023-01-15T14:28:14.438810+0000","last_undegraded":"2023-01-15T14:28:14.438810+0000","last_fullsized":"2023-01-15T14:28:14.438810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:31.082","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.431994+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:28:14.431994+0000","last_peered":"2023-01-15T14:28:14.431994+0000","last_clean":"2023-01-15T14:28:14.431994+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:28:14.431994+0000","last_undegraded":"2023-01-15T14:28:14.431994+0000","last_fullsized":"2023-01-15T14:28:14.431994+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.689","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:13.820597+0000","last_change":"2023-01-15T14:28:13.013616+0000","last_active":"2023-01-15T14:28:13.820597+0000","last_peered":"2023-01-15T14:28:13.820597+0000","last_clean":"2023-01-15T14:28:13.820597+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:28:13.820597+0000","last_undegraded":"2023-01-15T14:28:13.820597+0000","last_fullsized":"2023-01-15T14:28:13.820597+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:29.249","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.533586+0000","last_change":"2023-01-15T14:28:12.893975+0000","last_active":"2023-01-15T14:28:14.533586+0000","last_peered":"2023-01-15T14:28:14.533586+0000","last_clean":"2023-01-15T14:28:14.533586+0000","last_became_active":"2023-01-15T14:28:12.893100+0000","last_became_peered":"2023-01-15T14:28:12.893100+0000","last_unstale":"2023-01-15T14:28:14.533586+0000","last_undegraded":"2023-01-15T14:28:14.533586+0000","last_fullsized":"2023-01-15T14:28:14.533586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.496","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'61","reported_seq":96,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-01-15T14:28:14.739397+0000","last_change":"2023-01-15T14:28:07.312081+0000","last_active":"2023-01-15T14:28:14.739397+0000","last_peered":"2023-01-15T14:28:14.739397+0000","last_clean":"2023-01-15T14:28:14.739397+0000","last_became_active":"2023-01-15T14:28:07.309738+0000","last_became_peered":"2023-01-15T14:28:07.309738+0000","last_unstale":"2023-01-15T14:28:14.739397+0000","last_undegraded":"2023-01-15T14:28:14.739397+0000","last_fullsized":"2023-01-15T14:28:14.739397+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:25.933","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":62,"num_write":93,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61,"ondisk_log_size":61,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902868,"kb_used_omap":97,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":495616,"allocated":95132536832,"data_stored":95132536832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":100042,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.457999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.370999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":64.516999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":797204,"kb_used_data":92902868,"kb_used_omap":90,"kb_used_meta":0,"kb_avail":92902176,"statfs":{"total":95948165120,"available":95131828224,"internally_reserved":708608,"allocated":95132536832,"data_stored":95132536832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":93103,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795848,"kb_used_data":92903572,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903532,"statfs":{"total":95948165120,"available":95133216768,"internally_reserved":40960,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43962,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795856,"kb_used_data":92903572,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903524,"statfs":{"total":95948165120,"available":95133208576,"internally_reserved":49152,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43830,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.496}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795820,"kb_used_data":92903584,"kb_used_omap":37,"kb_used_meta":0,"kb_avail":92903560,"statfs":{"total":95948165120,"available":95133245440,"internally_reserved":24576,"allocated":95133270016,"data_stored":95133270016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38504,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.337}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3660000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.25}]}]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795868,"kb_used_data":92903572,"kb_used_omap":52,"kb_used_meta":0,"kb_avail":92903512,"statfs":{"total":95948165120,"available":95133196288,"internally_reserved":61440,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":53724,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.124000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795880,"kb_used_data":92903572,"kb_used_omap":53,"kb_used_meta":0,"kb_avail":92903500,"statfs":{"total":95948165120,"available":95133184000,"internally_reserved":73728,"allocated":95133257728,"data_stored":95133257728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":54652,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85099999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5429999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":796996,"kb_used_data":92902872,"kb_used_omap":92,"kb_used_meta":0,"kb_avail":92902384,"statfs":{"total":95948165120,"available":95132041216,"internally_reserved":499712,"allocated":95132540928,"data_stored":95132540928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95129,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.633}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.587}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.464}]}]}],"pool_statfs":[]}} 2023-01-15T14:28:19.451 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-01-15T14:28:19.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2023-01-15T14:28:19.759 INFO:teuthology.orchestra.run.smithi090.stdout:epoch 19 2023-01-15T14:28:19.766 INFO:teuthology.orchestra.run.smithi090.stdout:fsid 1c77fb2b-d1a7-4884-a24e-3935e16d0b6b 2023-01-15T14:28:19.767 INFO:teuthology.orchestra.run.smithi090.stdout:created 2023-01-15T14:27:57.735717+0000 2023-01-15T14:28:19.767 INFO:teuthology.orchestra.run.smithi090.stdout:modified 2023-01-15T14:28:13.795231+0000 2023-01-15T14:28:19.769 INFO:teuthology.orchestra.run.smithi090.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-01-15T14:28:19.769 INFO:teuthology.orchestra.run.smithi090.stdout:crush_version 4 2023-01-15T14:28:19.770 INFO:teuthology.orchestra.run.smithi090.stdout:full_ratio 0.95 2023-01-15T14:28:19.770 INFO:teuthology.orchestra.run.smithi090.stdout:backfillfull_ratio 0.9 2023-01-15T14:28:19.770 INFO:teuthology.orchestra.run.smithi090.stdout:nearfull_ratio 0.85 2023-01-15T14:28:19.795 INFO:teuthology.orchestra.run.smithi090.stdout:require_min_compat_client luminous 2023-01-15T14:28:19.795 INFO:teuthology.orchestra.run.smithi090.stdout:min_compat_client jewel 2023-01-15T14:28:19.796 INFO:teuthology.orchestra.run.smithi090.stdout:require_osd_release reef 2023-01-15T14:28:19.796 INFO:teuthology.orchestra.run.smithi090.stdout:stretch_mode_enabled false 2023-01-15T14:28:19.796 INFO:teuthology.orchestra.run.smithi090.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2023-01-15T14:28:19.796 INFO:teuthology.orchestra.run.smithi090.stdout:pool 2 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 19 flags hashpspool stripe_width 0 application rbd 2023-01-15T14:28:19.796 INFO:teuthology.orchestra.run.smithi090.stdout:max_osd 8 2023-01-15T14:28:19.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6808/23473,v1:172.21.15.90:6811/23473] [v2:172.21.15.90:6813/23473,v1:172.21.15.90:6815/23473] exists,up e55c5ac4-3cff-4ee5-8719-373648f83d8d 2023-01-15T14:28:19.802 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6800/23474,v1:172.21.15.90:6801/23474] [v2:172.21.15.90:6802/23474,v1:172.21.15.90:6803/23474] exists,up 2c1b33c9-0a1e-4734-936d-d51d64fe997a 2023-01-15T14:28:19.803 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6809/23475,v1:172.21.15.90:6810/23475] [v2:172.21.15.90:6812/23475,v1:172.21.15.90:6814/23475] exists,up b314174a-eca9-4baa-a60c-2589f1fc9a8e 2023-01-15T14:28:19.803 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6824/23778,v1:172.21.15.90:6825/23778] [v2:172.21.15.90:6826/23778,v1:172.21.15.90:6827/23778] exists,up 29eb0b17-ed5d-4665-a347-2fb8406d9c62 2023-01-15T14:28:19.803 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6800/23523,v1:172.21.15.141:6801/23523] [v2:172.21.15.141:6802/23523,v1:172.21.15.141:6803/23523] exists,up 59362feb-caa9-44e3-b6d0-b79397dacbfc 2023-01-15T14:28:19.803 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6824/23525,v1:172.21.15.141:6825/23525] [v2:172.21.15.141:6826/23525,v1:172.21.15.141:6827/23525] exists,up fbc8821c-8d78-40f8-a571-b18a70d42000 2023-01-15T14:28:19.803 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6808/23524,v1:172.21.15.141:6809/23524] [v2:172.21.15.141:6810/23524,v1:172.21.15.141:6811/23524] exists,up 34550470-3561-492f-b605-b5f0e8052c1e 2023-01-15T14:28:19.803 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6816/23526,v1:172.21.15.141:6817/23526] [v2:172.21.15.141:6818/23526,v1:172.21.15.141:6819/23526] exists,up 1f123d0d-d617-4771-b516-21aa5a29b734 2023-01-15T14:28:19.804 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6808/23473,v1:172.21.15.90:6811/23473] [v2:172.21.15.90:6813/23473,v1:172.21.15.90:6815/23473] exists,up e55c5ac4-3cff-4ee5-8719-373648f83d8d', 'osd.1 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6800/23474,v1:172.21.15.90:6801/23474] [v2:172.21.15.90:6802/23474,v1:172.21.15.90:6803/23474] exists,up 2c1b33c9-0a1e-4734-936d-d51d64fe997a', 'osd.2 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6809/23475,v1:172.21.15.90:6810/23475] [v2:172.21.15.90:6812/23475,v1:172.21.15.90:6814/23475] exists,up b314174a-eca9-4baa-a60c-2589f1fc9a8e', 'osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.90:6824/23778,v1:172.21.15.90:6825/23778] [v2:172.21.15.90:6826/23778,v1:172.21.15.90:6827/23778] exists,up 29eb0b17-ed5d-4665-a347-2fb8406d9c62', 'osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6800/23523,v1:172.21.15.141:6801/23523] [v2:172.21.15.141:6802/23523,v1:172.21.15.141:6803/23523] exists,up 59362feb-caa9-44e3-b6d0-b79397dacbfc', 'osd.5 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6824/23525,v1:172.21.15.141:6825/23525] [v2:172.21.15.141:6826/23525,v1:172.21.15.141:6827/23525] exists,up fbc8821c-8d78-40f8-a571-b18a70d42000', 'osd.6 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6808/23524,v1:172.21.15.141:6809/23524] [v2:172.21.15.141:6810/23524,v1:172.21.15.141:6811/23524] exists,up 34550470-3561-492f-b605-b5f0e8052c1e', 'osd.7 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.141:6816/23526,v1:172.21.15.141:6817/23526] [v2:172.21.15.141:6818/23526,v1:172.21.15.141:6819/23526] exists,up 1f123d0d-d617-4771-b516-21aa5a29b734'] 2023-01-15T14:28:19.805 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2023-01-15T14:28:19.920 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:28:19.932 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "mon.a", 2023-01-15T14:28:19.932 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster": "ceph", 2023-01-15T14:28:19.933 INFO:teuthology.orchestra.run.smithi090.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2023-01-15T14:28:19.933 INFO:teuthology.orchestra.run.smithi090.stdout: "admin_socket_mode": "", 2023-01-15T14:28:19.933 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2023-01-15T14:28:19.933 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_client_required": "cephx, none", 2023-01-15T14:28:19.933 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_cluster_required": "cephx", 2023-01-15T14:28:19.933 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_debug": "true", 2023-01-15T14:28:19.933 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2023-01-15T14:28:19.933 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_mon_ticket_ttl": "660.000000", 2023-01-15T14:28:19.934 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_service_required": "cephx", 2023-01-15T14:28:19.938 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_service_ticket_ttl": "240.000000", 2023-01-15T14:28:19.938 INFO:teuthology.orchestra.run.smithi090.stdout: "auth_supported": "cephx", 2023-01-15T14:28:19.940 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio": "true", 2023-01-15T14:28:19.940 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio_max_queue_depth": "1024", 2023-01-15T14:28:19.941 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio_poll_ms": "250", 2023-01-15T14:28:19.941 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_aio_reap_max": "16", 2023-01-15T14:28:19.941 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_async_discard": "false", 2023-01-15T14:28:19.941 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_block_size": "4096", 2023-01-15T14:28:19.941 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_aio": "false", 2023-01-15T14:28:19.941 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_aio_log_age": "5.000000", 2023-01-15T14:28:19.941 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2023-01-15T14:28:19.942 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_debug_inflight_ios": "false", 2023-01-15T14:28:19.942 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_enable_discard": "false", 2023-01-15T14:28:19.942 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_flock_retry": "3", 2023-01-15T14:28:19.942 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_flock_retry_interval": "0.100000", 2023-01-15T14:28:19.942 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_inject_crash": "0", 2023-01-15T14:28:19.942 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_inject_crash_flush_delay": "2", 2023-01-15T14:28:19.942 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_ioring": "false", 2023-01-15T14:28:19.943 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_ioring_hipri": "false", 2023-01-15T14:28:19.943 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_ioring_sqthread_poll": "false", 2023-01-15T14:28:19.957 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_nvme_unbind_from_kernel": "false", 2023-01-15T14:28:19.958 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_read_buffer_alignment": "4096", 2023-01-15T14:28:19.958 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_read_preallocated_huge_buffers": "", 2023-01-15T14:28:19.960 INFO:teuthology.orchestra.run.smithi090.stdout: "bdev_type": "", 2023-01-15T14:28:19.963 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_alloc_size": "1048576", 2023-01-15T14:28:19.963 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_allocator": "hybrid", 2023-01-15T14:28:19.963 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_buffered_io": "true", 2023-01-15T14:28:19.964 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_check_for_zeros": "false", 2023-01-15T14:28:19.965 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_check_volume_selector_often": "false", 2023-01-15T14:28:19.965 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_check_volume_selector_on_umount": "false", 2023-01-15T14:28:19.965 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_compact_log_sync": "false", 2023-01-15T14:28:19.965 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2023-01-15T14:28:19.965 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_log_compact_min_size": "16777216", 2023-01-15T14:28:19.965 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_log_replay_check_allocations": "true", 2023-01-15T14:28:19.965 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_max_log_runway": "4194304", 2023-01-15T14:28:19.965 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_max_prefetch": "1048576", 2023-01-15T14:28:19.965 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_min_flush_size": "524288", 2023-01-15T14:28:19.965 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_min_log_runway": "1048576", 2023-01-15T14:28:19.965 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_replay_recovery": "false", 2023-01-15T14:28:19.966 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_replay_recovery_disable_compact": "false", 2023-01-15T14:28:19.966 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_shared_alloc_size": "65536", 2023-01-15T14:28:19.966 INFO:teuthology.orchestra.run.smithi090.stdout: "bluefs_sync_write": "false", 2023-01-15T14:28:19.966 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2023-01-15T14:28:19.966 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2023-01-15T14:28:19.966 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2023-01-15T14:28:19.966 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_allocation_from_file": "true", 2023-01-15T14:28:19.966 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_allocator": "hybrid", 2023-01-15T14:28:19.966 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2023-01-15T14:28:19.966 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2023-01-15T14:28:19.966 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2023-01-15T14:28:19.967 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2023-01-15T14:28:19.967 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2023-01-15T14:28:19.967 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bitmapallocator_span_size": "1024", 2023-01-15T14:28:19.967 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_blobid_prealloc": "10240", 2023-01-15T14:28:19.967 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_create": "true", 2023-01-15T14:28:19.967 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_db_create": "false", 2023-01-15T14:28:19.967 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_db_path": "", 2023-01-15T14:28:19.967 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_db_size": "0", 2023-01-15T14:28:19.967 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_path": "", 2023-01-15T14:28:19.967 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_preallocate_file": "false", 2023-01-15T14:28:19.968 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_size": "107374182400", 2023-01-15T14:28:19.968 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_wal_create": "false", 2023-01-15T14:28:19.968 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_wal_path": "", 2023-01-15T14:28:19.968 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_block_wal_size": "100663296", 2023-01-15T14:28:19.968 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs": "true", 2023-01-15T14:28:19.968 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2023-01-15T14:28:19.968 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs_env_mirror": "false", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_bluefs_max_free": "10737418240", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_autotune": "true", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_autotune_interval": "5.000000", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_kv_ratio": "0.450000", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_meta_ratio": "0.450000", 2023-01-15T14:28:19.969 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_size": "0", 2023-01-15T14:28:19.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_size_hdd": "1073741824", 2023-01-15T14:28:19.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_size_ssd": "3221225472", 2023-01-15T14:28:19.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_trim_interval": "0.050000", 2023-01-15T14:28:19.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2023-01-15T14:28:19.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cache_type": "2q", 2023-01-15T14:28:19.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2023-01-15T14:28:19.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_clone_cow": "true", 2023-01-15T14:28:19.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_algorithm": "snappy", 2023-01-15T14:28:19.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_max_blob_size": "0", 2023-01-15T14:28:19.970 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2023-01-15T14:28:19.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2023-01-15T14:28:19.971 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_min_blob_size": "0", 2023-01-15T14:28:20.087 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2023-01-15T14:28:20.087 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2023-01-15T14:28:20.087 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_mode": "none", 2023-01-15T14:28:20.087 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_compression_required_ratio": "0.875000", 2023-01-15T14:28:20.087 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_csum_type": "crc32c", 2023-01-15T14:28:20.087 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_enforce_settings": "default", 2023-01-15T14:28:20.087 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_freelist": "false", 2023-01-15T14:28:20.088 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_fsck_abort": "false", 2023-01-15T14:28:20.088 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2023-01-15T14:28:20.088 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_inject_bug21040": "false", 2023-01-15T14:28:20.088 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2023-01-15T14:28:20.088 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_inject_read_err": "false", 2023-01-15T14:28:20.088 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_legacy_omap": "false", 2023-01-15T14:28:20.088 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_no_reuse_blocks": "false", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_omit_block_device_write": "false", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_omit_kv_commit": "false", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_prefill": "0.000000", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_prefragment_max": "1048576", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_random_read_err": "0.000000", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_small_allocations": "0", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_default_buffered_read": "true", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_default_buffered_write": "false", 2023-01-15T14:28:20.089 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_deferred_batch_ops": "0", 2023-01-15T14:28:20.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2023-01-15T14:28:20.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2023-01-15T14:28:20.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2023-01-15T14:28:20.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_max_size": "1200", 2023-01-15T14:28:20.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_min_size": "150", 2023-01-15T14:28:20.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_target_size": "500", 2023-01-15T14:28:20.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2023-01-15T14:28:20.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fail_eio": "false", 2023-01-15T14:28:20.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_freelist_blocks_per_key": "128", 2023-01-15T14:28:20.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2023-01-15T14:28:20.090 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2023-01-15T14:28:20.091 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2023-01-15T14:28:20.091 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mkfs": "true", 2023-01-15T14:28:20.091 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2023-01-15T14:28:20.108 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mount": "false", 2023-01-15T14:28:20.108 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_mount_deep": "false", 2023-01-15T14:28:20.109 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_umount": "false", 2023-01-15T14:28:20.115 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_on_umount_deep": "false", 2023-01-15T14:28:20.115 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2023-01-15T14:28:20.115 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_quick_fix_threads": "2", 2023-01-15T14:28:20.115 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2023-01-15T14:28:20.115 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2023-01-15T14:28:20.115 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_gc_enable_blob_threshold": "0", 2023-01-15T14:28:20.115 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_gc_enable_total_threshold": "0", 2023-01-15T14:28:20.115 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2023-01-15T14:28:20.115 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_ignore_data_csum": "false", 2023-01-15T14:28:20.116 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2023-01-15T14:28:20.116 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_kvbackend": "rocksdb", 2023-01-15T14:28:20.116 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_log_collection_list_age": "60.000000", 2023-01-15T14:28:20.116 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2023-01-15T14:28:20.116 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_log_op_age": "5.000000", 2023-01-15T14:28:20.116 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_alloc_size": "0", 2023-01-15T14:28:20.116 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_blob_size": "0", 2023-01-15T14:28:20.116 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_blob_size_hdd": "65536", 2023-01-15T14:28:20.116 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_blob_size_ssd": "65536", 2023-01-15T14:28:20.116 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_defer_interval": "3.000000", 2023-01-15T14:28:20.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_max_deferred_txc": "32", 2023-01-15T14:28:20.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_min_alloc_size": "0", 2023-01-15T14:28:20.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_min_alloc_size_hdd": "4096", 2023-01-15T14:28:20.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_min_alloc_size_ssd": "4096", 2023-01-15T14:28:20.117 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_nid_prealloc": "1024", 2023-01-15T14:28:20.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_prefer_deferred_size": "0", 2023-01-15T14:28:20.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2023-01-15T14:28:20.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2023-01-15T14:28:20.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_qfsck_on_mount": "true", 2023-01-15T14:28:20.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_retry_disk_reads": "3", 2023-01-15T14:28:20.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_cf": "true", 2023-01-15T14:28:20.119 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2023-01-15T14:28:20.122 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0,ttl=21600", 2023-01-15T14:28:20.122 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_rocksdb_options_annex": "", 2023-01-15T14:28:20.122 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_coremask": "0x1", 2023-01-15T14:28:20.122 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_io_sleep": "5", 2023-01-15T14:28:20.122 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_max_io_completion": "0", 2023-01-15T14:28:20.122 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_spdk_mem": "512", 2023-01-15T14:28:20.122 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_sync_submit_transaction": "false", 2023-01-15T14:28:20.122 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_bytes": "67108864", 2023-01-15T14:28:20.123 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_cost_per_io": "0", 2023-01-15T14:28:20.123 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2023-01-15T14:28:20.123 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2023-01-15T14:28:20.123 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2023-01-15T14:28:20.123 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_throttle_trace_rate": "0.000000", 2023-01-15T14:28:20.123 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_tracing": "false", 2023-01-15T14:28:20.123 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2023-01-15T14:28:20.123 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2023-01-15T14:28:20.123 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_volume_selection_reserved": "0", 2023-01-15T14:28:20.124 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2023-01-15T14:28:20.124 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2023-01-15T14:28:20.124 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_legacy_statfs": "true", 2023-01-15T14:28:20.124 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2023-01-15T14:28:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2023-01-15T14:28:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2023-01-15T14:28:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "bluestore_zero_block_detection": "false", 2023-01-15T14:28:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "cephadm_path": "/usr/sbin/cephadm", 2023-01-15T14:28:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_action_update_interval": "2", 2023-01-15T14:28:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_directory_scan_interval": "10", 2023-01-15T14:28:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2023-01-15T14:28:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2023-01-15T14:28:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2023-01-15T14:28:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_mount_timeout": "10", 2023-01-15T14:28:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2023-01-15T14:28:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2023-01-15T14:28:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2023-01-15T14:28:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "cephsqlite_blocklist_dead_locker": "true", 2023-01-15T14:28:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "cephsqlite_lock_renewal_interval": "2000", 2023-01-15T14:28:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2023-01-15T14:28:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_cluster_require_signatures": "false", 2023-01-15T14:28:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_cluster_require_version": "2", 2023-01-15T14:28:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_require_signatures": "false", 2023-01-15T14:28:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_require_version": "2", 2023-01-15T14:28:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_service_require_signatures": "false", 2023-01-15T14:28:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_service_require_version": "2", 2023-01-15T14:28:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "cephx_sign_messages": "true", 2023-01-15T14:28:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "chdir": "", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_acl_type": "", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_asio_thread_count": "2", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_cache_mid": "0.750000", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_cache_size": "16384", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_caps_release_delay": "5", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_check_pool_perm": "true", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_collect_and_send_global_metrics": "false", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_debug_force_sync_read": "false", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_debug_getattr_caps": "false", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_debug_inject_tick_delay": "0", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_die_on_failed_dentry_invalidate": "true", 2023-01-15T14:28:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_die_on_failed_remount": "false", 2023-01-15T14:28:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_dirsize_rbytes": "true", 2023-01-15T14:28:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_force_lazyio": "false", 2023-01-15T14:28:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_fs": "", 2023-01-15T14:28:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_inject_fixed_oldest_tid": "false", 2023-01-15T14:28:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_inject_release_failure": "false", 2023-01-15T14:28:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_max_inline_size": "4096", 2023-01-15T14:28:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mds_namespace": "", 2023-01-15T14:28:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client_metadata": "", 2023-01-15T14:28:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mount_gid": "-1", 2023-01-15T14:28:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mount_timeout": "300", 2023-01-15T14:28:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mount_uid": "-1", 2023-01-15T14:28:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "client_mountpoint": "/", 2023-01-15T14:28:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "client_notify_timeout": "10", 2023-01-15T14:28:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc": "true", 2023-01-15T14:28:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_max_dirty": "104857600", 2023-01-15T14:28:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_max_dirty_age": "5.000000", 2023-01-15T14:28:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_max_objects": "1000", 2023-01-15T14:28:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_size": "209715200", 2023-01-15T14:28:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: "client_oc_target_dirty": "8388608", 2023-01-15T14:28:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: "client_permissions": "true", 2023-01-15T14:28:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: "client_quota": "true", 2023-01-15T14:28:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: "client_quota_df": "true", 2023-01-15T14:28:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_readahead_max_bytes": "0", 2023-01-15T14:28:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_readahead_max_periods": "4", 2023-01-15T14:28:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_readahead_min": "131072", 2023-01-15T14:28:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_reconnect_stale": "false", 2023-01-15T14:28:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_shutdown_timeout": "30", 2023-01-15T14:28:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_snapdir": ".snap", 2023-01-15T14:28:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_tick_interval": "1", 2023-01-15T14:28:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_trace": "", 2023-01-15T14:28:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: "client_try_dentry_invalidate": "false", 2023-01-15T14:28:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_use_faked_inos": "false", 2023-01-15T14:28:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_use_random_mds": "false", 2023-01-15T14:28:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_graylog": "false", 2023-01-15T14:28:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_graylog_host": "127.0.0.1", 2023-01-15T14:28:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_graylog_port": "12201", 2023-01-15T14:28:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_monitors": "default=true", 2023-01-15T14:28:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_syslog": "false", 2023-01-15T14:28:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2023-01-15T14:28:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: "clog_to_syslog_level": "info", 2023-01-15T14:28:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "-", 2023-01-15T14:28:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_network": "", 2023-01-15T14:28:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_network_interface": "", 2023-01-15T14:28:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zlib_isal": "false", 2023-01-15T14:28:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zlib_level": "5", 2023-01-15T14:28:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zlib_winsize": "-15", 2023-01-15T14:28:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: "compressor_zstd_level": "1", 2023-01-15T14:28:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2023-01-15T14:28:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "crash_dir": "/var/lib/ceph/crash", 2023-01-15T14:28:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_alien_op_num_threads": "6", 2023-01-15T14:28:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_alien_thread_cpu_cores": "", 2023-01-15T14:28:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_osd_obc_lru_size": "10", 2023-01-15T14:28:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "crimson_osd_scheduler_concurrency": "0", 2023-01-15T14:28:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_location": "", 2023-01-15T14:28:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_location_hook": "", 2023-01-15T14:28:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_location_hook_timeout": "10", 2023-01-15T14:28:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: "daemonize": "false", 2023-01-15T14:28:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: "daos_pool": "tank", 2023-01-15T14:28:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2023-01-15T14:28:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2023-01-15T14:28:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "dbstore_db_name_prefix": "dbstore", 2023-01-15T14:28:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_alienstore": "0/5", 2023-01-15T14:28:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_allow_any_pool_priority": "false", 2023-01-15T14:28:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_asok": "1/5", 2023-01-15T14:28:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_asok_assert_abort": "false", 2023-01-15T14:28:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_asserts_on_shutdown": "true", 2023-01-15T14:28:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_auth": "20/20", 2023-01-15T14:28:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_bdev": "1/3", 2023-01-15T14:28:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_bluefs": "1/5", 2023-01-15T14:28:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_bluestore": "1/5", 2023-01-15T14:28:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_buffer": "0/1", 2023-01-15T14:28:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_ceph_exporter": "1/5", 2023-01-15T14:28:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_cephfs_mirror": "0/5", 2023-01-15T14:28:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_cephsqlite": "0/5", 2023-01-15T14:28:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_client": "0/5", 2023-01-15T14:28:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_compressor": "1/5", 2023-01-15T14:28:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_context": "0/1", 2023-01-15T14:28:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_crush": "1/1", 2023-01-15T14:28:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_crypto": "1/5", 2023-01-15T14:28:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_cyanstore": "0/5", 2023-01-15T14:28:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_deliberately_leak_memory": "false", 2023-01-15T14:28:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_disable_randomized_ping": "false", 2023-01-15T14:28:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_dpdk": "1/5", 2023-01-15T14:28:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_eventtrace": "1/5", 2023-01-15T14:28:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_filer": "0/1", 2023-01-15T14:28:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_filestore": "1/3", 2023-01-15T14:28:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_finisher": "1/1", 2023-01-15T14:28:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_fuse": "1/5", 2023-01-15T14:28:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_heartbeat_testing_span": "0", 2023-01-15T14:28:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_heartbeatmap": "1/5", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_immutable_obj_cache": "0/5", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_javaclient": "1/5", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_journal": "1/3", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_journaler": "0/5", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_kstore": "1/5", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_leveldb": "4/5", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_lockdep": "0/1", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mclock": "1/5", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds": "1/5", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_balancer": "1/5", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_locker": "1/5", 2023-01-15T14:28:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_log": "1/5", 2023-01-15T14:28:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_log_expire": "1/5", 2023-01-15T14:28:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mds_migrator": "1/5", 2023-01-15T14:28:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_memstore": "1/5", 2023-01-15T14:28:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mgr": "2/5", 2023-01-15T14:28:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mgrc": "1/5", 2023-01-15T14:28:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_mon": "20/20", 2023-01-15T14:28:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_monc": "0/10", 2023-01-15T14:28:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_ms": "1/1", 2023-01-15T14:28:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_none": "0/5", 2023-01-15T14:28:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_objclass": "0/5", 2023-01-15T14:28:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_objectcacher": "0/5", 2023-01-15T14:28:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_objecter": "0/1", 2023-01-15T14:28:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_optracker": "0/5", 2023-01-15T14:28:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_osd": "1/5", 2023-01-15T14:28:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_paxos": "20/20", 2023-01-15T14:28:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_perfcounter": "1/5", 2023-01-15T14:28:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_prioritycache": "1/5", 2023-01-15T14:28:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rados": "0/5", 2023-01-15T14:28:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd": "0/5", 2023-01-15T14:28:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd_mirror": "0/5", 2023-01-15T14:28:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd_pwl": "0/5", 2023-01-15T14:28:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rbd_replay": "0/5", 2023-01-15T14:28:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_refs": "0/0", 2023-01-15T14:28:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_reserver": "1/1", 2023-01-15T14:28:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw": "1/5", 2023-01-15T14:28:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_access": "1/5", 2023-01-15T14:28:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_datacache": "1/5", 2023-01-15T14:28:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_dbstore": "1/5", 2023-01-15T14:28:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rgw_sync": "1/5", 2023-01-15T14:28:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_rocksdb": "4/5", 2023-01-15T14:28:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore": "0/5", 2023-01-15T14:28:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_backref": "0/5", 2023-01-15T14:28:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_cache": "0/5", 2023-01-15T14:28:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_cleaner": "0/5", 2023-01-15T14:28:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_device": "0/5", 2023-01-15T14:28:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_epm": "0/5", 2023-01-15T14:28:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_fixedkv_tree": "0/5", 2023-01-15T14:28:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_journal": "0/5", 2023-01-15T14:28:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_lba": "0/5", 2023-01-15T14:28:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_odata": "0/5", 2023-01-15T14:28:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_omap": "0/5", 2023-01-15T14:28:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_onode": "0/5", 2023-01-15T14:28:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_t": "0/5", 2023-01-15T14:28:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_seastore_tm": "0/5", 2023-01-15T14:28:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_striper": "0/1", 2023-01-15T14:28:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_test": "0/5", 2023-01-15T14:28:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_throttle": "1/1", 2023-01-15T14:28:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_timer": "0/1", 2023-01-15T14:28:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "debug_tp": "0/5", 2023-01-15T14:28:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "device_failure_prediction_mode": "none", 2023-01-15T14:28:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2023-01-15T14:28:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2023-01-15T14:28:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_graylog": "false", 2023-01-15T14:28:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_journald": "false", 2023-01-15T14:28:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_stderr": "true", 2023-01-15T14:28:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "err_to_syslog": "false", 2023-01-15T14:28:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event_tracing": "false", 2023-01-15T14:28:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_addr": "0.0.0.0", 2023-01-15T14:28:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_http_port": "9926", 2023-01-15T14:28:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_prio_limit": "5", 2023-01-15T14:28:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_sock_dir": "/var/run/ceph/", 2023-01-15T14:28:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_sort_metrics": "true", 2023-01-15T14:28:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "exporter_stats_period": "5", 2023-01-15T14:28:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "fake_statfs_for_testing": "0", 2023-01-15T14:28:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "fatal_signal_handlers": "true", 2023-01-15T14:28:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "filer_max_purge_ops": "10", 2023-01-15T14:28:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "filer_max_truncate_ops": "128", 2023-01-15T14:28:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_apply_finisher_threads": "1", 2023-01-15T14:28:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_blackhole": "false", 2023-01-15T14:28:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_btrfs_clone_range": "true", 2023-01-15T14:28:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_btrfs_snap": "true", 2023-01-15T14:28:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_caller_concurrency": "10", 2023-01-15T14:28:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_collect_device_partition_information": "true", 2023-01-15T14:28:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_commit_timeout": "600.000000", 2023-01-15T14:28:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_debug_inject_read_err": "false", 2023-01-15T14:28:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_debug_omap_check": "false", 2023-01-15T14:28:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_debug_random_read_err": "0.000000", 2023-01-15T14:28:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_debug_verify_split": "false", 2023-01-15T14:28:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_dump_file": "", 2023-01-15T14:28:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2023-01-15T14:28:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_expected_throughput_ops": "200.000000", 2023-01-15T14:28:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fadvise": "true", 2023-01-15T14:28:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fail_eio": "true", 2023-01-15T14:28:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fd_cache_shards": "16", 2023-01-15T14:28:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fd_cache_size": "128", 2023-01-15T14:28:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fiemap": "false", 2023-01-15T14:28:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fiemap_threshold": "4096", 2023-01-15T14:28:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_fsync_flushes_journal_data": "false", 2023-01-15T14:28:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_index_retry_probability": "0.000000", 2023-01-15T14:28:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_inject_stall": "0", 2023-01-15T14:28:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_journal_parallel": "false", 2023-01-15T14:28:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_journal_trailing": "false", 2023-01-15T14:28:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_journal_writeahead": "false", 2023-01-15T14:28:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_kill_at": "0", 2023-01-15T14:28:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_alloc_hint_size": "1048576", 2023-01-15T14:28:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size": "0", 2023-01-15T14:28:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2023-01-15T14:28:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size_other": "512", 2023-01-15T14:28:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2023-01-15T14:28:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs": "0", 2023-01-15T14:28:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2023-01-15T14:28:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs_other": "2", 2023-01-15T14:28:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_inline_xattrs_xfs": "10", 2023-01-15T14:28:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_sync_interval": "5.000000", 2023-01-15T14:28:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size": "0", 2023-01-15T14:28:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2023-01-15T14:28:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size_other": "1024", 2023-01-15T14:28:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2023-01-15T14:28:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_merge_threshold": "-10", 2023-01-15T14:28:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_min_sync_interval": "0.010000", 2023-01-15T14:28:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_odsync_write": "false", 2023-01-15T14:28:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_omap_backend": "rocksdb", 2023-01-15T14:28:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_omap_backend_path": "", 2023-01-15T14:28:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_omap_header_cache_size": "1024", 2023-01-15T14:28:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_ondisk_finisher_threads": "1", 2023-01-15T14:28:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_op_thread_suicide_timeout": "180", 2023-01-15T14:28:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_op_thread_timeout": "60", 2023-01-15T14:28:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_op_threads": "2", 2023-01-15T14:28:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_punch_hole": "false", 2023-01-15T14:28:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2023-01-15T14:28:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2023-01-15T14:28:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2023-01-15T14:28:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_high_threshhold": "0.900000", 2023-01-15T14:28:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_low_threshhold": "0.300000", 2023-01-15T14:28:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_bytes": "104857600", 2023-01-15T14:28:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2023-01-15T14:28:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2023-01-15T14:28:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2023-01-15T14:28:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_queue_max_ops": "50", 2023-01-15T14:28:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2023-01-15T14:28:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_seek_data_hole": "false", 2023-01-15T14:28:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_sloppy_crc": "false", 2023-01-15T14:28:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_sloppy_crc_block_size": "65536", 2023-01-15T14:28:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_splice": "false", 2023-01-15T14:28:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_split_multiple": "2", 2023-01-15T14:28:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_split_rand_factor": "20", 2023-01-15T14:28:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_update_to": "1000", 2023-01-15T14:28:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2023-01-15T14:28:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2023-01-15T14:28:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2023-01-15T14:28:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2023-01-15T14:28:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2023-01-15T14:28:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2023-01-15T14:28:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_enable": "true", 2023-01-15T14:28:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2023-01-15T14:28:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2023-01-15T14:28:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2023-01-15T14:28:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2023-01-15T14:28:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2023-01-15T14:28:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2023-01-15T14:28:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_xfs_extsize": "false", 2023-01-15T14:28:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "filestore_zfs_snap": "false", 2023-01-15T14:28:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "fio_dir": "/tmp/fio", 2023-01-15T14:28:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "1c77fb2b-d1a7-4884-a24e-3935e16d0b6b", 2023-01-15T14:28:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_allow_other": "true", 2023-01-15T14:28:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_atomic_o_trunc": "true", 2023-01-15T14:28:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_big_writes": "true", 2023-01-15T14:28:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_debug": "false", 2023-01-15T14:28:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_default_permissions": "false", 2023-01-15T14:28:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_disable_pagecache": "false", 2023-01-15T14:28:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_max_write": "0", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_multithreaded": "true", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_require_active_mds": "true", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_set_user_groups": "true", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_splice_move": "true", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_splice_read": "true", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_splice_write": "true", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_syncfs_on_mksnap": "true", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "fuse_use_invalidate_cb": "true", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "gss_target_name": "ceph", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_file": "", 2023-01-15T14:28:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_inject_failure": "0", 2023-01-15T14:28:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_interval": "5", 2023-01-15T14:28:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "host": "smithi090", 2023-01-15T14:28:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2023-01-15T14:28:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_max_inflight_ops": "128", 2023-01-15T14:28:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_max_size": "1073741824", 2023-01-15T14:28:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2023-01-15T14:28:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_bps_burst": "0", 2023-01-15T14:28:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2023-01-15T14:28:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_bps_limit": "0", 2023-01-15T14:28:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_iops_burst": "0", 2023-01-15T14:28:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2023-01-15T14:28:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_iops_limit": "0", 2023-01-15T14:28:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2023-01-15T14:28:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2023-01-15T14:28:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "immutable_object_cache_watermark": "0.900000", 2023-01-15T14:28:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "inject_early_sigterm": "false", 2023-01-15T14:28:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "jaeger_tracing_enable": "false", 2023-01-15T14:28:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_aio": "true", 2023-01-15T14:28:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_align_min_size": "65536", 2023-01-15T14:28:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_block_align": "true", 2023-01-15T14:28:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_block_size": "4096", 2023-01-15T14:28:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_dio": "true", 2023-01-15T14:28:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_discard": "false", 2023-01-15T14:28:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_force_aio": "false", 2023-01-15T14:28:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_ignore_corruption": "false", 2023-01-15T14:28:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_max_write_bytes": "10485760", 2023-01-15T14:28:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_max_write_entries": "100", 2023-01-15T14:28:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_replay_from": "0", 2023-01-15T14:28:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_high_multiple": "0.000000", 2023-01-15T14:28:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_high_threshhold": "0.900000", 2023-01-15T14:28:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_low_threshhold": "0.600000", 2023-01-15T14:28:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_throttle_max_multiple": "0.000000", 2023-01-15T14:28:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_write_header_frequency": "0", 2023-01-15T14:28:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "journal_zero_on_create": "false", 2023-01-15T14:28:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "journaler_prefetch_periods": "10", 2023-01-15T14:28:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "journaler_prezero_periods": "5", 2023-01-15T14:28:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "journaler_write_head_interval": "15", 2023-01-15T14:28:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "key": "", 2023-01-15T14:28:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "keyfile": "", 2023-01-15T14:28:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2023-01-15T14:28:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_backend": "rocksdb", 2023-01-15T14:28:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_default_stripe_size": "65536", 2023-01-15T14:28:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_fsck_on_mount": "false", 2023-01-15T14:28:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_fsck_on_mount_deep": "true", 2023-01-15T14:28:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_max_bytes": "67108864", 2023-01-15T14:28:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_max_ops": "512", 2023-01-15T14:28:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_nid_prealloc": "1024", 2023-01-15T14:28:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_onode_map_size": "1024", 2023-01-15T14:28:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2023-01-15T14:28:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_sync_submit_transaction": "false", 2023-01-15T14:28:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: "kstore_sync_transaction": "false", 2023-01-15T14:28:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_block_size": "65536", 2023-01-15T14:28:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_bloom_size": "0", 2023-01-15T14:28:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_cache_size": "536870912", 2023-01-15T14:28:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_compact_on_mount": "false", 2023-01-15T14:28:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_compression": "false", 2023-01-15T14:28:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_log": "", 2023-01-15T14:28:20.451 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_log_to_ceph_log": "true", 2023-01-15T14:28:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_max_open_files": "0", 2023-01-15T14:28:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_paranoid": "false", 2023-01-15T14:28:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "leveldb_write_buffer_size": "33554432", 2023-01-15T14:28:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "librados_thread_count": "2", 2023-01-15T14:28:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "lockdep": "false", 2023-01-15T14:28:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "lockdep_force_backtrace": "false", 2023-01-15T14:28:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "log_coarse_timestamps": "true", 2023-01-15T14:28:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2023-01-15T14:28:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "log_flush_on_exit": "false", 2023-01-15T14:28:20.452 INFO:teuthology.orchestra.run.smithi090.stdout: "log_graylog_host": "127.0.0.1", 2023-01-15T14:28:20.453 INFO:teuthology.orchestra.run.smithi090.stdout: "log_graylog_port": "12201", 2023-01-15T14:28:20.453 INFO:teuthology.orchestra.run.smithi090.stdout: "log_max_new": "1000", 2023-01-15T14:28:20.453 INFO:teuthology.orchestra.run.smithi090.stdout: "log_max_recent": "10000", 2023-01-15T14:28:20.569 INFO:teuthology.orchestra.run.smithi090.stdout: "log_stderr_prefix": "", 2023-01-15T14:28:20.569 INFO:teuthology.orchestra.run.smithi090.stdout: "log_stop_at_utilization": "0.970000", 2023-01-15T14:28:20.569 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_file": "true", 2023-01-15T14:28:20.569 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_graylog": "false", 2023-01-15T14:28:20.569 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_journald": "false", 2023-01-15T14:28:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_stderr": "false", 2023-01-15T14:28:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "log_to_syslog": "false", 2023-01-15T14:28:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "max_rotating_auth_attempts": "10", 2023-01-15T14:28:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_action_on_write_error": "1", 2023-01-15T14:28:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_alternate_name_max": "8192", 2023-01-15T14:28:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_asio_thread_count": "2", 2023-01-15T14:28:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_export_pin": "true", 2023-01-15T14:28:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_dirs": "true", 2023-01-15T14:28:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2023-01-15T14:28:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_interval": "5", 2023-01-15T14:28:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_fragment_size_max": "100000", 2023-01-15T14:28:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_idle_threshold": "0.000000", 2023-01-15T14:28:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_interval": "10", 2023-01-15T14:28:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_max": "-1", 2023-01-15T14:28:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_max_until": "-1", 2023-01-15T14:28:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_merge_size": "50", 2023-01-15T14:28:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_midchunk": "0.300000", 2023-01-15T14:28:20.571 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_min_rebalance": "0.100000", 2023-01-15T14:28:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_min_start": "0.200000", 2023-01-15T14:28:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_minchunk": "0.001000", 2023-01-15T14:28:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_mode": "0", 2023-01-15T14:28:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_need_max": "1.200000", 2023-01-15T14:28:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_need_min": "0.800000", 2023-01-15T14:28:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_replicate_threshold": "8000.000000", 2023-01-15T14:28:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_sample_interval": "3.000000", 2023-01-15T14:28:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_bits": "3", 2023-01-15T14:28:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_rd": "25000.000000", 2023-01-15T14:28:20.572 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_size": "10000", 2023-01-15T14:28:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_split_wr": "10000.000000", 2023-01-15T14:28:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_target_decay": "10.000000", 2023-01-15T14:28:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2023-01-15T14:28:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_beacon_grace": "15.000000", 2023-01-15T14:28:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_beacon_interval": "4.000000", 2023-01-15T14:28:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_beacon_mon_down_grace": "60", 2023-01-15T14:28:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_memory_limit": "4294967296", 2023-01-15T14:28:20.573 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_mid": "0.700000", 2023-01-15T14:28:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_release_free_interval": "10", 2023-01-15T14:28:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_reservation": "0.050000", 2023-01-15T14:28:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_trim_decay_rate": "1.000000", 2023-01-15T14:28:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_trim_interval": "1", 2023-01-15T14:28:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cache_trim_threshold": "262144", 2023-01-15T14:28:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2023-01-15T14:28:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2023-01-15T14:28:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_client_delegate_inos_pct": "50", 2023-01-15T14:28:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_client_prealloc_inos": "1000", 2023-01-15T14:28:20.574 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2023-01-15T14:28:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_connect_bootstrapping": "false", 2023-01-15T14:28:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_damage_table_max_entries": "10000", 2023-01-15T14:28:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2023-01-15T14:28:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_auth_pins": "false", 2023-01-15T14:28:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_frag": "false", 2023-01-15T14:28:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_scatterstat": "false", 2023-01-15T14:28:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_debug_subtrees": "false", 2023-01-15T14:28:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_decay_halflife": "5.000000", 2023-01-15T14:28:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_default_dir_hash": "2", 2023-01-15T14:28:20.575 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_defer_session_stale": "true", 2023-01-15T14:28:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_deny_all_reconnect": "false", 2023-01-15T14:28:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dir_keys_per_op": "16384", 2023-01-15T14:28:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dir_max_commit_size": "10", 2023-01-15T14:28:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dir_max_entries": "0", 2023-01-15T14:28:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dir_prefetch": "true", 2023-01-15T14:28:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dirstat_min_interval": "1.000000", 2023-01-15T14:28:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_after_rejoin": "false", 2023-01-15T14:28:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_on_map": "false", 2023-01-15T14:28:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_threshold_file": "0", 2023-01-15T14:28:20.576 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2023-01-15T14:28:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_early_reply": "true", 2023-01-15T14:28:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_enable_op_tracker": "true", 2023-01-15T14:28:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_enforce_unique_name": "true", 2023-01-15T14:28:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_distributed": "true", 2023-01-15T14:28:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2023-01-15T14:28:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_random": "true", 2023-01-15T14:28:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_export_ephemeral_random_max": "0.010000", 2023-01-15T14:28:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_extraordinary_events_dump_interval": "0", 2023-01-15T14:28:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_forward_all_requests_to_auth": "false", 2023-01-15T14:28:20.577 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_freeze_tree_timeout": "30.000000", 2023-01-15T14:28:20.597 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_fscrypt_last_block_max_size": "4096", 2023-01-15T14:28:20.597 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2023-01-15T14:28:20.597 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_health_cache_threshold": "1.500000", 2023-01-15T14:28:20.598 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_health_summarize_threshold": "10", 2023-01-15T14:28:20.598 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_heartbeat_grace": "15.000000", 2023-01-15T14:28:20.598 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_heartbeat_reset_grace": "1000", 2023-01-15T14:28:20.598 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_health_dummy": "false", 2023-01-15T14:28:20.598 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_migrator_session_race": "false", 2023-01-15T14:28:20.598 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2023-01-15T14:28:20.598 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_join_fs": "", 2023-01-15T14:28:20.598 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_journal_format": "1", 2023-01-15T14:28:20.598 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_create_at": "0", 2023-01-15T14:28:20.600 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_export_at": "0", 2023-01-15T14:28:20.600 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_import_at": "0", 2023-01-15T14:28:20.600 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_journal_at": "0", 2023-01-15T14:28:20.600 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_journal_expire_at": "0", 2023-01-15T14:28:20.600 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_journal_replay_at": "0", 2023-01-15T14:28:20.600 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_link_at": "0", 2023-01-15T14:28:20.601 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_mdstable_at": "0", 2023-01-15T14:28:20.601 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_openc_at": "0", 2023-01-15T14:28:20.601 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_kill_rename_at": "0", 2023-01-15T14:28:20.602 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_events_per_segment": "1024", 2023-01-15T14:28:20.602 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_max_events": "-1", 2023-01-15T14:28:20.604 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_max_segments": "128", 2023-01-15T14:28:20.604 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_pause": "false", 2023-01-15T14:28:20.604 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_segment_size": "0", 2023-01-15T14:28:20.604 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_skip_corrupt_events": "false", 2023-01-15T14:28:20.605 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_log_warn_factor": "2.000000", 2023-01-15T14:28:20.605 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_caps_per_client": "1048576", 2023-01-15T14:28:20.605 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_completed_flushes": "100000", 2023-01-15T14:28:20.605 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_completed_requests": "100000", 2023-01-15T14:28:20.605 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_export_size": "20971520", 2023-01-15T14:28:20.605 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_file_recover": "32", 2023-01-15T14:28:20.605 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_purge_files": "64", 2023-01-15T14:28:20.606 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_purge_ops": "8192", 2023-01-15T14:28:20.606 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2023-01-15T14:28:20.606 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_retries_on_remount_failure": "5", 2023-01-15T14:28:20.606 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_scrub_ops_in_progress": "5", 2023-01-15T14:28:20.606 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_snaps_per_dir": "100", 2023-01-15T14:28:20.606 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_max_xattr_pairs_size": "65536", 2023-01-15T14:28:20.606 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_metrics_update_interval": "2", 2023-01-15T14:28:20.606 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_min_caps_per_client": "100", 2023-01-15T14:28:20.607 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_min_caps_working_set": "10000", 2023-01-15T14:28:20.607 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_mon_shutdown_timeout": "5.000000", 2023-01-15T14:28:20.607 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_numa_node": "-1", 2023-01-15T14:28:20.607 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_oft_prefetch_dirfrags": "false", 2023-01-15T14:28:20.607 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_complaint_time": "30.000000", 2023-01-15T14:28:20.607 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_history_duration": "600", 2023-01-15T14:28:20.734 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_history_size": "20", 2023-01-15T14:28:20.734 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_op_log_threshold": "5", 2023-01-15T14:28:20.734 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_ping_grace": "15", 2023-01-15T14:28:20.734 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_ping_interval": "5", 2023-01-15T14:28:20.734 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2023-01-15T14:28:20.734 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_global_max_decay_threshold": "131072", 2023-01-15T14:28:20.735 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_max_caps": "30000", 2023-01-15T14:28:20.735 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_max_decay_rate": "1.500000", 2023-01-15T14:28:20.735 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_max_decay_threshold": "131072", 2023-01-15T14:28:20.735 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_warning_decay_rate": "60.000000", 2023-01-15T14:28:20.735 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_recall_warning_threshold": "262144", 2023-01-15T14:28:20.735 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_reconnect_timeout": "45.000000", 2023-01-15T14:28:20.735 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_replay_interval": "1.000000", 2023-01-15T14:28:20.735 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_replay_unsafe_with_closed_session": "false", 2023-01-15T14:28:20.735 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_request_load_average_decay_rate": "60.000000", 2023-01-15T14:28:20.735 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_root_ino_gid": "0", 2023-01-15T14:28:20.736 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_root_ino_uid": "0", 2023-01-15T14:28:20.736 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_scatter_nudge_interval": "5.000000", 2023-01-15T14:28:20.736 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_blocklist_on_evict": "true", 2023-01-15T14:28:20.736 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_blocklist_on_timeout": "true", 2023-01-15T14:28:20.736 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2023-01-15T14:28:20.736 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cache_liveness_magnitude": "10", 2023-01-15T14:28:20.736 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cap_acquisition_decay_rate": "10.000000", 2023-01-15T14:28:20.736 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_cap_acquisition_throttle": "500000", 2023-01-15T14:28:20.738 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2023-01-15T14:28:20.738 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_sessionmap_keys_per_op": "1024", 2023-01-15T14:28:20.738 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_shutdown_check": "0", 2023-01-15T14:28:20.738 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_skip_ino": "0", 2023-01-15T14:28:20.746 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_sleep_rank_change": "0.000000", 2023-01-15T14:28:20.746 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_snap_max_uid": "4294967294", 2023-01-15T14:28:20.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_snap_min_uid": "0", 2023-01-15T14:28:20.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_snap_rstat": "false", 2023-01-15T14:28:20.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_standby_replay_damaged": "false", 2023-01-15T14:28:20.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_symlink_recovery": "true", 2023-01-15T14:28:20.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_task_status_update_interval": "2.000000", 2023-01-15T14:28:20.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_thrash_exports": "0", 2023-01-15T14:28:20.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_thrash_fragments": "0", 2023-01-15T14:28:20.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_tick_interval": "5.000000", 2023-01-15T14:28:20.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_valgrind_exit": "false", 2023-01-15T14:28:20.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_verify_backtrace": "1", 2023-01-15T14:28:20.748 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_verify_scatter": "false", 2023-01-15T14:28:20.748 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_wipe_ino_prealloc": "false", 2023-01-15T14:28:20.748 INFO:teuthology.orchestra.run.smithi090.stdout: "mds_wipe_sessions": "false", 2023-01-15T14:28:20.748 INFO:teuthology.orchestra.run.smithi090.stdout: "mempool_debug": "false", 2023-01-15T14:28:20.748 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_debug_omit_block_device_write": "false", 2023-01-15T14:28:20.748 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_device_bytes": "1073741824", 2023-01-15T14:28:20.748 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_page_set": "false", 2023-01-15T14:28:20.749 INFO:teuthology.orchestra.run.smithi090.stdout: "memstore_page_size": "65536", 2023-01-15T14:28:20.749 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_client_bytes": "134217728", 2023-01-15T14:28:20.749 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_client_messages": "512", 2023-01-15T14:28:20.749 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2023-01-15T14:28:20.749 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_connect_retry_interval": "1.000000", 2023-01-15T14:28:20.749 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2023-01-15T14:28:20.749 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2023-01-15T14:28:20.750 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_disabled_modules": "", 2023-01-15T14:28:20.750 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_initial_modules": "restful iostat nfs", 2023-01-15T14:28:20.750 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_max_pg_num_change": "128", 2023-01-15T14:28:20.750 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mds_bytes": "134217728", 2023-01-15T14:28:20.752 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mds_messages": "128", 2023-01-15T14:28:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2023-01-15T14:28:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mon_bytes": "134217728", 2023-01-15T14:28:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_mon_messages": "128", 2023-01-15T14:28:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_osd_bytes": "536870912", 2023-01-15T14:28:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_osd_messages": "8192", 2023-01-15T14:28:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_pool": "true", 2023-01-15T14:28:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_service_beacon_grace": "60.000000", 2023-01-15T14:28:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_standby_modules": "true", 2023-01-15T14:28:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_stats_period": "5", 2023-01-15T14:28:20.757 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_stats_threshold": "5", 2023-01-15T14:28:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_tick_period": "2", 2023-01-15T14:28:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr_ttl_cache_expire_seconds": "0", 2023-01-15T14:28:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_accept_timeout_factor": "2.000000", 2023-01-15T14:28:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_allow_pool_delete": "true", 2023-01-15T14:28:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_allow_pool_size_one": "true", 2023-01-15T14:28:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_auth_validate_all_caps": "true", 2023-01-15T14:28:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2023-01-15T14:28:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2023-01-15T14:28:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_bytes": "104857600", 2023-01-15T14:28:20.758 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_directed_command_retry": "2", 2023-01-15T14:28:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval": "3.000000", 2023-01-15T14:28:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2023-01-15T14:28:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2023-01-15T14:28:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2023-01-15T14:28:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_hunt_parallel": "3", 2023-01-15T14:28:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_log_interval": "1.000000", 2023-01-15T14:28:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_max_log_entries_per_message": "1000", 2023-01-15T14:28:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_ping_interval": "10.000000", 2023-01-15T14:28:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_client_ping_timeout": "30.000000", 2023-01-15T14:28:20.759 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_clock_drift_allowed": "1.000000", 2023-01-15T14:28:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2023-01-15T14:28:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2023-01-15T14:28:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_file_level": "debug", 2023-01-15T14:28:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_file": "true", 2023-01-15T14:28:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_graylog": "false", 2023-01-15T14:28:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2023-01-15T14:28:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_graylog_port": "12201", 2023-01-15T14:28:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_journald": "false", 2023-01-15T14:28:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_stderr": "false", 2023-01-15T14:28:20.760 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_syslog": "default=false", 2023-01-15T14:28:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2023-01-15T14:28:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cluster_log_to_syslog_level": "info", 2023-01-15T14:28:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_compact_on_bootstrap": "false", 2023-01-15T14:28:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_compact_on_start": "false", 2023-01-15T14:28:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_compact_on_trim": "true", 2023-01-15T14:28:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_con_tracker_persist_interval": "10", 2023-01-15T14:28:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_con_tracker_score_halflife": "43200", 2023-01-15T14:28:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_config_key_max_entry_size": "65536", 2023-01-15T14:28:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_cpu_threads": "4", 2023-01-15T14:28:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_crush_min_required_version": "hammer", 2023-01-15T14:28:20.761 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_daemon_bytes": "419430400", 2023-01-15T14:28:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2023-01-15T14:28:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data_avail_crit": "5", 2023-01-15T14:28:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data_avail_warn": "5", 2023-01-15T14:28:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_data_size_warn": "16106127360", 2023-01-15T14:28:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_block_osdmap_trim": "false", 2023-01-15T14:28:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_deprecated_as_obsolete": "false", 2023-01-15T14:28:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_dump_json": "false", 2023-01-15T14:28:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2023-01-15T14:28:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_dump_transactions": "false", 2023-01-15T14:28:20.762 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_extra_checks": "false", 2023-01-15T14:28:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_initial_persistent_features": "false", 2023-01-15T14:28:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2023-01-15T14:28:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_require_quincy": "false", 2023-01-15T14:28:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_no_require_reef": "false", 2023-01-15T14:28:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2023-01-15T14:28:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_delta_reset_interval": "10.000000", 2023-01-15T14:28:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_dns_srv_name": "ceph-mon", 2023-01-15T14:28:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_down_mkfs_grace": "120", 2023-01-15T14:28:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_election_default_strategy": "3", 2023-01-15T14:28:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_election_timeout": "5.000000", 2023-01-15T14:28:20.763 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2023-01-15T14:28:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_elector_ping_divisor": "2", 2023-01-15T14:28:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_elector_ping_timeout": "2.000000", 2023-01-15T14:28:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_enable_op_tracker": "true", 2023-01-15T14:28:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_fake_pool_delete": "false", 2023-01-15T14:28:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_force_quorum_join": "false", 2023-01-15T14:28:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_globalid_prealloc": "10000", 2023-01-15T14:28:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_detail_to_clog": "false", 2023-01-15T14:28:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_log_update_period": "5", 2023-01-15T14:28:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_max_detail": "50", 2023-01-15T14:28:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_to_clog": "true", 2023-01-15T14:28:20.764 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_to_clog_interval": "600", 2023-01-15T14:28:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2023-01-15T14:28:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_host": "172.21.15.90,[v2:172.21.15.90:3301,v1:172.21.15.90:6790],172.21.15.141", 2023-01-15T14:28:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_host_override": "", 2023-01-15T14:28:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_initial_members": "", 2023-01-15T14:28:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2023-01-15T14:28:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2023-01-15T14:28:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_transaction_delay_max": "10.000000", 2023-01-15T14:28:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2023-01-15T14:28:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_keyvaluedb": "rocksdb", 2023-01-15T14:28:20.765 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_lease": "5.000000", 2023-01-15T14:28:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2023-01-15T14:28:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_lease_renew_interval_factor": "0.600000", 2023-01-15T14:28:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_log_full_interval": "50", 2023-01-15T14:28:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_log_max": "10000", 2023-01-15T14:28:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_log_max_summary": "50", 2023-01-15T14:28:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_log_entries_per_event": "4096", 2023-01-15T14:28:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_log_epochs": "500", 2023-01-15T14:28:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_mdsmap_epochs": "500", 2023-01-15T14:28:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_mgrmap_epochs": "500", 2023-01-15T14:28:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_osd": "10000", 2023-01-15T14:28:20.766 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_pg_per_osd": "10000", 2023-01-15T14:28:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_pool_pg_num": "65536", 2023-01-15T14:28:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_max_snap_prune_per_epoch": "100", 2023-01-15T14:28:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mds_blocklist_interval": "86400.000000", 2023-01-15T14:28:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mds_force_trim_to": "0", 2023-01-15T14:28:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mds_skip_sanity": "false", 2023-01-15T14:28:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_memory_autotune": "true", 2023-01-15T14:28:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_memory_target": "2147483648", 2023-01-15T14:28:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_beacon_grace": "30", 2023-01-15T14:28:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2023-01-15T14:28:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_digest_period": "5", 2023-01-15T14:28:20.767 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_inactive_grace": "60", 2023-01-15T14:28:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_mkfs_grace": "240", 2023-01-15T14:28:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2023-01-15T14:28:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_min_osdmap_epochs": "50", 2023-01-15T14:28:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_complaint_time": "30", 2023-01-15T14:28:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_duration": "600", 2023-01-15T14:28:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_size": "20", 2023-01-15T14:28:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_slow_op_size": "20", 2023-01-15T14:28:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_history_slow_op_threshold": "10", 2023-01-15T14:28:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_op_log_threshold": "5", 2023-01-15T14:28:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_adjust_down_out_interval": "true", 2023-01-15T14:28:20.768 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2023-01-15T14:28:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2023-01-15T14:28:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_auto_mark_in": "false", 2023-01-15T14:28:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_auto_mark_new_in": "true", 2023-01-15T14:28:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2023-01-15T14:28:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2023-01-15T14:28:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_cache_size": "500", 2023-01-15T14:28:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_cache_size_min": "134217728", 2023-01-15T14:28:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_crush_smoke_test": "true", 2023-01-15T14:28:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_destroyed_out_interval": "600", 2023-01-15T14:28:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_down_out_interval": "600", 2023-01-15T14:28:20.769 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_down_out_subtree_limit": "rack", 2023-01-15T14:28:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2023-01-15T14:28:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_force_trim_to": "0", 2023-01-15T14:28:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_full_ratio": "0.950000", 2023-01-15T14:28:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2023-01-15T14:28:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_laggy_halflife": "3600", 2023-01-15T14:28:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_laggy_max_interval": "300", 2023-01-15T14:28:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_laggy_weight": "0.300000", 2023-01-15T14:28:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2023-01-15T14:28:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_max_creating_pgs": "1024", 2023-01-15T14:28:20.770 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_max_initial_pgs": "1024", 2023-01-15T14:28:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_min_down_reporters": "2", 2023-01-15T14:28:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_min_in_ratio": "0.750000", 2023-01-15T14:28:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_min_up_ratio": "0.300000", 2023-01-15T14:28:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_nearfull_ratio": "0.850000", 2023-01-15T14:28:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_prime_pg_temp": "true", 2023-01-15T14:28:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2023-01-15T14:28:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2023-01-15T14:28:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_report_timeout": "900", 2023-01-15T14:28:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_reporter_subtree_level": "osd", 2023-01-15T14:28:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2023-01-15T14:28:20.771 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_warn_num_repaired": "10", 2023-01-15T14:28:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osd_warn_op_age": "32.000000", 2023-01-15T14:28:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_enabled": "true", 2023-01-15T14:28:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_interval": "2", 2023-01-15T14:28:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_min": "15", 2023-01-15T14:28:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_osdmap_full_prune_txsize": "2", 2023-01-15T14:28:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2023-01-15T14:28:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_stuck_threshold": "60", 2023-01-15T14:28:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2023-01-15T14:28:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_min_objects": "10000", 2023-01-15T14:28:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_min_per_osd": "0", 2023-01-15T14:28:20.772 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pg_warn_min_pool_objects": "1000", 2023-01-15T14:28:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pool_quota_crit_threshold": "0", 2023-01-15T14:28:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_pool_quota_warn_threshold": "0", 2023-01-15T14:28:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_probe_timeout": "2.000000", 2023-01-15T14:28:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_max_change": "0.050000", 2023-01-15T14:28:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_max_osds": "4", 2023-01-15T14:28:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_min_bytes_per_osd": "10", 2023-01-15T14:28:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_reweight_min_pgs_per_osd": "4", 2023-01-15T14:28:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2023-01-15T14:28:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2023-01-15T14:28:20.773 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2023-01-15T14:28:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_interval": "300", 2023-01-15T14:28:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_max_keys": "100", 2023-01-15T14:28:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_scrub_timeout": "300", 2023-01-15T14:28:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_session_timeout": "300", 2023-01-15T14:28:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_smart_report_timeout": "5", 2023-01-15T14:28:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stat_smooth_intervals": "6", 2023-01-15T14:28:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2023-01-15T14:28:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_pool_min_size": "2", 2023-01-15T14:28:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_pool_size": "4", 2023-01-15T14:28:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2023-01-15T14:28:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_subscribe_interval": "86400.000000", 2023-01-15T14:28:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_debug": "false", 2023-01-15T14:28:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_max_payload_keys": "2000", 2023-01-15T14:28:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_max_payload_size": "1048576", 2023-01-15T14:28:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_provider_kill_at": "0", 2023-01-15T14:28:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_requester_kill_at": "0", 2023-01-15T14:28:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_sync_timeout": "60.000000", 2023-01-15T14:28:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_target_pg_per_osd": "100", 2023-01-15T14:28:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_tick_interval": "5", 2023-01-15T14:28:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_timecheck_interval": "300.000000", 2023-01-15T14:28:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_timecheck_skew_interval": "30.000000", 2023-01-15T14:28:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_use_min_delay_socket": "false", 2023-01-15T14:28:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_older_version_delay": "604800", 2023-01-15T14:28:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2023-01-15T14:28:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2023-01-15T14:28:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2023-01-15T14:28:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_filestore_osds": "false", 2023-01-15T14:28:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2023-01-15T14:28:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2023-01-15T14:28:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2023-01-15T14:28:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_misplaced": "false", 2023-01-15T14:28:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2023-01-15T14:28:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_older_version": "true", 2023-01-15T14:28:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2023-01-15T14:28:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_pool_no_app": "true", 2023-01-15T14:28:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_pool_no_redundancy": "false", 2023-01-15T14:28:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2023-01-15T14:28:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2023-01-15T14:28:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2023-01-15T14:28:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_on_too_few_osds": "false", 2023-01-15T14:28:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2023-01-15T14:28:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2023-01-15T14:28:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "monmap": "", 2023-01-15T14:28:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2023-01-15T14:28:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2023-01-15T14:28:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2023-01-15T14:28:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2023-01-15T14:28:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "motr_tracing_enabled": "false", 2023-01-15T14:28:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_op_threads": "3", 2023-01-15T14:28:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_buffer_size": "131072", 2023-01-15T14:28:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_cm": "false", 2023-01-15T14:28:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_device_name": "", 2023-01-15T14:28:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_dscp": "96", 2023-01-15T14:28:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_enable_hugepage": "false", 2023-01-15T14:28:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_gid_idx": "0", 2023-01-15T14:28:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_local_gid": "", 2023-01-15T14:28:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_polling_us": "1000", 2023-01-15T14:28:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_port_num": "1", 2023-01-15T14:28:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_receive_buffers": "32768", 2023-01-15T14:28:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_receive_queue_len": "4096", 2023-01-15T14:28:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_roce_ver": "1", 2023-01-15T14:28:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_send_buffers": "1024", 2023-01-15T14:28:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_sl": "3", 2023-01-15T14:28:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_support_srq": "true", 2023-01-15T14:28:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_rdma_type": "ib", 2023-01-15T14:28:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_async_reap_threshold": "5", 2023-01-15T14:28:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_before_connect": "false", 2023-01-15T14:28:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_ipv4": "true", 2023-01-15T14:28:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_ipv6": "false", 2023-01-15T14:28:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_msgr1": "true", 2023-01-15T14:28:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_msgr2": "true", 2023-01-15T14:28:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_port_max": "7568", 2023-01-15T14:28:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_port_min": "6800", 2023-01-15T14:28:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_prefer_ipv4": "false", 2023-01-15T14:28:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_retry_count": "3", 2023-01-15T14:28:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_bind_retry_delay": "5", 2023-01-15T14:28:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_client": "false", 2023-01-15T14:28:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_mds": "false", 2023-01-15T14:28:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_mgr": "false", 2023-01-15T14:28:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_mon": "false", 2023-01-15T14:28:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_blackhole_osd": "false", 2023-01-15T14:28:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_client_mode": "crc secure", 2023-01-15T14:28:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_cluster_mode": "crc secure", 2023-01-15T14:28:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_cluster_type": "", 2023-01-15T14:28:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_compress_secure": "false", 2023-01-15T14:28:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_connection_idle_timeout": "900", 2023-01-15T14:28:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_connection_ready_timeout": "10", 2023-01-15T14:28:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_crc_data": "true", 2023-01-15T14:28:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_crc_header": "true", 2023-01-15T14:28:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_bad_msg": "false", 2023-01-15T14:28:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_bug": "true", 2023-01-15T14:28:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_old_message": "true", 2023-01-15T14:28:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_skipped_message": "false", 2023-01-15T14:28:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_die_on_unhandled_msg": "false", 2023-01-15T14:28:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dispatch_throttle_bytes": "104857600", 2023-01-15T14:28:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_coremask": "0xF", 2023-01-15T14:28:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_debug_allow_loopback": "false", 2023-01-15T14:28:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_devs_allowlist": "", 2023-01-15T14:28:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_enable_tso": "true", 2023-01-15T14:28:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2023-01-15T14:28:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_host_ipv4_addr": "", 2023-01-15T14:28:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_hugepages": "", 2023-01-15T14:28:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_hw_flow_control": "true", 2023-01-15T14:28:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2023-01-15T14:28:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_lro": "true", 2023-01-15T14:28:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_memory_channel": "4", 2023-01-15T14:28:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2023-01-15T14:28:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_pmd": "", 2023-01-15T14:28:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_port_id": "0", 2023-01-15T14:28:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2023-01-15T14:28:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dump_corrupt_message_level": "1", 2023-01-15T14:28:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_dump_on_send": "false", 2023-01-15T14:28:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_initial_backoff": "0.200000", 2023-01-15T14:28:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_delay_max": "1.000000", 2023-01-15T14:28:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_delay_probability": "0.000000", 2023-01-15T14:28:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_delay_type": "", 2023-01-15T14:28:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_internal_delays": "0.000000", 2023-01-15T14:28:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_network_congestion": "0", 2023-01-15T14:28:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_inject_socket_failures": "0", 2023-01-15T14:28:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_learn_addr_from_peer": "true", 2023-01-15T14:28:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_max_accept_failures": "4", 2023-01-15T14:28:20.928 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_max_backoff": "15.000000", 2023-01-15T14:28:20.928 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_mon_client_mode": "secure crc", 2023-01-15T14:28:20.928 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_mon_cluster_mode": "secure crc", 2023-01-15T14:28:20.929 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_mon_service_mode": "secure crc", 2023-01-15T14:28:20.929 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_osd_compress_min_size": "1024", 2023-01-15T14:28:20.929 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_osd_compress_mode": "none", 2023-01-15T14:28:20.929 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2023-01-15T14:28:20.929 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2023-01-15T14:28:20.929 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_pq_min_cost": "65536", 2023-01-15T14:28:20.929 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_public_type": "", 2023-01-15T14:28:20.929 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_service_mode": "crc secure", 2023-01-15T14:28:20.929 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_listen_backlog": "512", 2023-01-15T14:28:20.930 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_nodelay": "true", 2023-01-15T14:28:20.930 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_prefetch_max_size": "4096", 2023-01-15T14:28:20.930 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_tcp_rcvbuf": "0", 2023-01-15T14:28:20.930 INFO:teuthology.orchestra.run.smithi090.stdout: "ms_type": "async", 2023-01-15T14:28:20.930 INFO:teuthology.orchestra.run.smithi090.stdout: "no_config_file": "false", 2023-01-15T14:28:20.930 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_completion_locks_per_session": "32", 2023-01-15T14:28:20.930 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_debug_inject_relock_delay": "false", 2023-01-15T14:28:20.930 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_inflight_op_bytes": "104857600", 2023-01-15T14:28:20.930 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_inflight_ops": "1024", 2023-01-15T14:28:20.931 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_inject_no_watch_ping": "false", 2023-01-15T14:28:20.931 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_retry_writes_after_first_reply": "false", 2023-01-15T14:28:20.931 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_tick_interval": "5.000000", 2023-01-15T14:28:20.931 INFO:teuthology.orchestra.run.smithi090.stdout: "objecter_timeout": "10.000000", 2023-01-15T14:28:20.931 INFO:teuthology.orchestra.run.smithi090.stdout: "objectstore_blackhole": "false", 2023-01-15T14:28:20.931 INFO:teuthology.orchestra.run.smithi090.stdout: "openssl_engine_opts": "", 2023-01-15T14:28:20.931 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_delay_time": "5.000000", 2023-01-15T14:28:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_hist_halflife": "1000", 2023-01-15T14:28:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_max_low_ops": "2", 2023-01-15T14:28:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_max_ops": "4", 2023-01-15T14:28:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_min_evict_effort": "0.100000", 2023-01-15T14:28:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_quantize_effort": "0.100000", 2023-01-15T14:28:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_agent_slop": "0.020000", 2023-01-15T14:28:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_aggregated_slow_ops_logging": "true", 2023-01-15T14:28:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_allow_recovery_below_min_size": "true", 2023-01-15T14:28:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_asio_thread_count": "2", 2023-01-15T14:28:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_async_recovery_min_cost": "100", 2023-01-15T14:28:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_auto_mark_unfound_lost": "false", 2023-01-15T14:28:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backfill_retry_interval": "30.000000", 2023-01-15T14:28:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backfill_scan_max": "512", 2023-01-15T14:28:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backfill_scan_min": "64", 2023-01-15T14:28:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backoff_on_degraded": "false", 2023-01-15T14:28:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backoff_on_peering": "false", 2023-01-15T14:28:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_backoff_on_unfound": "true", 2023-01-15T14:28:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_beacon_report_interval": "300", 2023-01-15T14:28:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_duration": "30", 2023-01-15T14:28:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_large_size_max_throughput": "104857600", 2023-01-15T14:28:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_max_block_size": "67108864", 2023-01-15T14:28:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_bench_small_size_max_iops": "100", 2023-01-15T14:28:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_blkin_trace_all": "false", 2023-01-15T14:28:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_blocked_scrub_grace_period": "120", 2023-01-15T14:28:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2023-01-15T14:28:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2023-01-15T14:28:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2023-01-15T14:28:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_check_for_log_corruption": "false", 2023-01-15T14:28:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_check_max_object_name_len_on_startup": "true", 2023-01-15T14:28:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-01-15T14:28:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2023-01-15T14:28:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2023-01-15T14:28:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_class_update_on_start": "true", 2023-01-15T14:28:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_message_cap": "256", 2023-01-15T14:28:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_message_size_cap": "524288000", 2023-01-15T14:28:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_op_priority": "63", 2023-01-15T14:28:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_client_watch_timeout": "30", 2023-01-15T14:28:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_command_max_records": "256", 2023-01-15T14:28:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_compact_on_start": "false", 2023-01-15T14:28:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_copyfrom_max_chunk": "8388608", 2023-01-15T14:28:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_chooseleaf_type": "0", 2023-01-15T14:28:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_initial_weight": "-1.000000", 2023-01-15T14:28:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_update_on_start": "true", 2023-01-15T14:28:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_crush_update_weight_set": "true", 2023-01-15T14:28:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2023-01-15T14:28:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2023-01-15T14:28:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2023-01-15T14:28:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_drop_ping_duration": "0", 2023-01-15T14:28:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_drop_ping_probability": "0.000000", 2023-01-15T14:28:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_feed_pullee": "-1", 2023-01-15T14:28:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_inject_copyfrom_error": "false", 2023-01-15T14:28:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2023-01-15T14:28:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_inject_dispatch_delay_probability": "0.100000", 2023-01-15T14:28:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_misdirected_ops": "false", 2023-01-15T14:28:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_no_acting_change": "false", 2023-01-15T14:28:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_no_purge_strays": "false", 2023-01-15T14:28:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_op_order": "false", 2023-01-15T14:28:21.086 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_pg_log_writeout": "false", 2023-01-15T14:28:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_pretend_recovery_active": "false", 2023-01-15T14:28:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_random_push_read_error": "0.000000", 2023-01-15T14:28:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2023-01-15T14:28:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_shutdown": "false", 2023-01-15T14:28:21.087 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2023-01-15T14:28:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2023-01-15T14:28:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_cached_snaps": "false", 2023-01-15T14:28:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_missing_on_start": "false", 2023-01-15T14:28:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_snaps": "false", 2023-01-15T14:28:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_debug_verify_stray_on_activate": "false", 2023-01-15T14:28:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_interval": "604800.000000", 2023-01-15T14:28:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_keys": "1024", 2023-01-15T14:28:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2023-01-15T14:28:21.088 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2023-01-15T14:28:21.089 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2023-01-15T14:28:21.089 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_stride": "524288", 2023-01-15T14:28:21.089 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2023-01-15T14:28:21.089 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_default_data_pool_replay_window": "5", 2023-01-15T14:28:21.089 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_default_notify_timeout": "30", 2023-01-15T14:28:21.089 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep": "0.000000", 2023-01-15T14:28:21.090 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep_hdd": "5.000000", 2023-01-15T14:28:21.090 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep_hybrid": "1.000000", 2023-01-15T14:28:21.090 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_delete_sleep_ssd": "1.000000", 2023-01-15T14:28:21.090 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_discard_disconnected_ops": "true", 2023-01-15T14:28:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_enable_op_tracker": "true", 2023-01-15T14:28:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2023-01-15T14:28:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_extblkdev_plugins": "vdo", 2023-01-15T14:28:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_failsafe_full_ratio": "0.970000", 2023-01-15T14:28:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_fail_on_connection_refused": "true", 2023-01-15T14:28:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_info": "true", 2023-01-15T14:28:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_shutdown": "true", 2023-01-15T14:28:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_shutdown_notify_mon": "true", 2023-01-15T14:28:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_fast_shutdown_timeout": "15", 2023-01-15T14:28:21.091 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_find_best_info_ignore_history_les": "false", 2023-01-15T14:28:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_force_auth_primary_missing_objects": "100", 2023-01-15T14:28:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2023-01-15T14:28:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_function_tracing": "false", 2023-01-15T14:28:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_grace": "20", 2023-01-15T14:28:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_interval": "6", 2023-01-15T14:28:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2023-01-15T14:28:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_min_peers": "10", 2023-01-15T14:28:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_min_size": "2000", 2023-01-15T14:28:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_stale": "600", 2023-01-15T14:28:21.092 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2023-01-15T14:28:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_hit_set_max_size": "100000", 2023-01-15T14:28:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_hit_set_min_size": "1000", 2023-01-15T14:28:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_hit_set_namespace": ".ceph-internal", 2023-01-15T14:28:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_ignore_stale_divergent_priors": "false", 2023-01-15T14:28:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2023-01-15T14:28:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_inject_failure_on_pg_removal": "false", 2023-01-15T14:28:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2023-01-15T14:28:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_journal_flush_on_shutdown": "true", 2023-01-15T14:28:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_journal_size": "5120", 2023-01-15T14:28:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_kill_backfill_at": "0", 2023-01-15T14:28:21.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_loop_before_reset_tphandle": "64", 2023-01-15T14:28:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_cache_size": "50", 2023-01-15T14:28:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_dedup": "true", 2023-01-15T14:28:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_message_max": "40", 2023-01-15T14:28:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_message_max_bytes": "10485760", 2023-01-15T14:28:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_map_share_max_epochs": "40", 2023-01-15T14:28:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_attr_name_len": "100", 2023-01-15T14:28:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_attr_size": "0", 2023-01-15T14:28:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_backfills": "1", 2023-01-15T14:28:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_markdown_count": "5", 2023-01-15T14:28:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_markdown_period": "600", 2023-01-15T14:28:21.094 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_object_name_len": "2048", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_object_namespace_len": "256", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_object_size": "134217728", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_omap_entries_per_request": "1024", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pg_blocked_by": "16", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pg_log_entries": "10000", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_pgls": "1024", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_push_cost": "8388608", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_push_objects": "10", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_scrubs": "1", 2023-01-15T14:28:21.095 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2023-01-15T14:28:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_trimming_pgs": "2", 2023-01-15T14:28:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_write_op_reply_len": "64", 2023-01-15T14:28:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_max_write_size": "90", 2023-01-15T14:28:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_cost_per_byte_usec": "0.000000", 2023-01-15T14:28:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_cost_per_byte_usec_hdd": "2.600000", 2023-01-15T14:28:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_cost_per_byte_usec_ssd": "0.011000", 2023-01-15T14:28:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_cost_per_io_usec": "0.000000", 2023-01-15T14:28:21.096 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_cost_per_io_usec_hdd": "11400.000000", 2023-01-15T14:28:21.098 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_cost_per_io_usec_ssd": "50.000000", 2023-01-15T14:28:21.098 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2023-01-15T14:28:21.098 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2023-01-15T14:28:21.098 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2023-01-15T14:28:21.098 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2023-01-15T14:28:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2023-01-15T14:28:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_override_recovery_settings": "false", 2023-01-15T14:28:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_profile": "high_client_ops", 2023-01-15T14:28:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2023-01-15T14:28:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2023-01-15T14:28:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2023-01-15T14:28:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2023-01-15T14:28:21.099 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2023-01-15T14:28:21.100 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2023-01-15T14:28:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2023-01-15T14:28:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_client_lim": "999999", 2023-01-15T14:28:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_client_res": "1", 2023-01-15T14:28:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_scheduler_client_wgt": "1", 2023-01-15T14:28:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mclock_skip_benchmark": "false", 2023-01-15T14:28:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_base": "805306368", 2023-01-15T14:28:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_cache_min": "134217728", 2023-01-15T14:28:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_cache_resize_interval": "1.000000", 2023-01-15T14:28:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_expected_fragmentation": "0.150000", 2023-01-15T14:28:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_target": "4294967296", 2023-01-15T14:28:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_target_autotune": "false", 2023-01-15T14:28:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2023-01-15T14:28:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_min_pg_log_entries": "250", 2023-01-15T14:28:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_min_recovery_priority": "0", 2023-01-15T14:28:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_heartbeat_interval": "30", 2023-01-15T14:28:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2023-01-15T14:28:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_report_interval": "5", 2023-01-15T14:28:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_report_max_in_flight": "2", 2023-01-15T14:28:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_mon_shutdown_timeout": "5.000000", 2023-01-15T14:28:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_num_cache_shards": "32", 2023-01-15T14:28:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_num_op_tracker_shard": "32", 2023-01-15T14:28:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_numa_auto_affinity": "true", 2023-01-15T14:28:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_numa_node": "-1", 2023-01-15T14:28:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_numa_prefer_iface": "true", 2023-01-15T14:28:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_object_clean_region_max_num_intervals": "10", 2023-01-15T14:28:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objecter_finishers": "1", 2023-01-15T14:28:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objectstore": "bluestore", 2023-01-15T14:28:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objectstore_fuse": "false", 2023-01-15T14:28:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_objectstore_tracing": "false", 2023-01-15T14:28:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_complaint_time": "30.000000", 2023-01-15T14:28:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_duration": "600", 2023-01-15T14:28:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_size": "20", 2023-01-15T14:28:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_slow_op_size": "20", 2023-01-15T14:28:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2023-01-15T14:28:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_log_threshold": "5", 2023-01-15T14:28:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_shards": "0", 2023-01-15T14:28:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_shards_hdd": "5", 2023-01-15T14:28:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_shards_ssd": "8", 2023-01-15T14:28:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_threads_per_shard": "0", 2023-01-15T14:28:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2023-01-15T14:28:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2023-01-15T14:28:21.218 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2023-01-15T14:28:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_pq_min_cost": "65536", 2023-01-15T14:28:21.220 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_queue": "mclock_scheduler", 2023-01-15T14:28:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_queue_cut_off": "high", 2023-01-15T14:28:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_thread_suicide_timeout": "150", 2023-01-15T14:28:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_op_thread_timeout": "15", 2023-01-15T14:28:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_open_classes_on_start": "true", 2023-01-15T14:28:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_os_flags": "0", 2023-01-15T14:28:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_peering_op_priority": "255", 2023-01-15T14:28:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_delete_cost": "1048576", 2023-01-15T14:28:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_delete_priority": "5", 2023-01-15T14:28:21.221 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2023-01-15T14:28:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2023-01-15T14:28:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_log_dups_tracked": "3000", 2023-01-15T14:28:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_log_trim_max": "10000", 2023-01-15T14:28:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_log_trim_min": "100", 2023-01-15T14:28:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2023-01-15T14:28:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_object_context_cache_count": "64", 2023-01-15T14:28:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pg_stat_report_interval_max": "500", 2023-01-15T14:28:21.222 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_min_evict_age": "0", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_min_flush_age": "0", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_crush_rule": "-1", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_ec_fast_read": "false", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_bulk": "false", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_hashpspool": "true", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_nodelete": "false", 2023-01-15T14:28:21.223 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_nopgchange": "false", 2023-01-15T14:28:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flag_nosizechange": "false", 2023-01-15T14:28:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_flags": "0", 2023-01-15T14:28:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2023-01-15T14:28:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_min_size": "2", 2023-01-15T14:28:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2023-01-15T14:28:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_pg_num": "32", 2023-01-15T14:28:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_pgp_num": "0", 2023-01-15T14:28:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2023-01-15T14:28:21.224 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_size": "3", 2023-01-15T14:28:21.225 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_default_type": "replicated", 2023-01-15T14:28:21.225 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2023-01-15T14:28:21.225 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_pool_use_gmt_hitset": "true", 2023-01-15T14:28:21.225 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_push_per_object_cost": "1000", 2023-01-15T14:28:21.225 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_read_ec_check_for_errors": "false", 2023-01-15T14:28:21.249 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recover_clone_overlap": "true", 2023-01-15T14:28:21.249 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recover_clone_overlap_limit": "10", 2023-01-15T14:28:21.250 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_cost": "20971520", 2023-01-15T14:28:21.250 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_delay_start": "0.000000", 2023-01-15T14:28:21.250 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_active": "0", 2023-01-15T14:28:21.250 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_active_hdd": "3", 2023-01-15T14:28:21.250 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_active_ssd": "10", 2023-01-15T14:28:21.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_chunk": "8388608", 2023-01-15T14:28:21.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2023-01-15T14:28:21.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_max_single_start": "1", 2023-01-15T14:28:21.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_op_priority": "3", 2023-01-15T14:28:21.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_op_warn_multiple": "16", 2023-01-15T14:28:21.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_priority": "5", 2023-01-15T14:28:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_retry_interval": "30.000000", 2023-01-15T14:28:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep": "0.000000", 2023-01-15T14:28:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep_hdd": "0.100000", 2023-01-15T14:28:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2023-01-15T14:28:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_recovery_sleep_ssd": "0.000000", 2023-01-15T14:28:21.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_repair_during_recovery": "false", 2023-01-15T14:28:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_requested_scrub_priority": "120", 2023-01-15T14:28:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2023-01-15T14:28:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_rollback_to_cluster_snap": "", 2023-01-15T14:28:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_auto_repair": "false", 2023-01-15T14:28:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_auto_repair_num_errors": "5", 2023-01-15T14:28:21.253 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_backoff_ratio": "0.660000", 2023-01-15T14:28:21.254 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_begin_hour": "0", 2023-01-15T14:28:21.254 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_begin_week_day": "0", 2023-01-15T14:28:21.254 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_busy_replicas_penalty": "300", 2023-01-15T14:28:21.254 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_chunk_max": "25", 2023-01-15T14:28:21.254 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_chunk_min": "5", 2023-01-15T14:28:21.254 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_cost": "52428800", 2023-01-15T14:28:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_during_recovery": "false", 2023-01-15T14:28:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_end_hour": "0", 2023-01-15T14:28:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_end_week_day": "0", 2023-01-15T14:28:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_extended_sleep": "0.000000", 2023-01-15T14:28:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2023-01-15T14:28:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_invalid_stats": "true", 2023-01-15T14:28:21.255 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_load_threshold": "0.500000", 2023-01-15T14:28:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_max_interval": "604800.000000", 2023-01-15T14:28:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_max_preemptions": "5", 2023-01-15T14:28:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_min_interval": "86400.000000", 2023-01-15T14:28:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_priority": "5", 2023-01-15T14:28:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_reservation_timeout": "5000", 2023-01-15T14:28:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_retry_busy_replicas": "5", 2023-01-15T14:28:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_retry_delay": "3", 2023-01-15T14:28:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_retry_pg_state": "60", 2023-01-15T14:28:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_retry_wrong_time": "300", 2023-01-15T14:28:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_sleep": "0.000000", 2023-01-15T14:28:21.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_scrub_slow_reservation_response": "2200", 2023-01-15T14:28:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_shutdown_pgref_assert": "false", 2023-01-15T14:28:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_skip_data_digest": "false", 2023-01-15T14:28:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_smart_report_timeout": "5", 2023-01-15T14:28:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_cost": "1048576", 2023-01-15T14:28:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_priority": "5", 2023-01-15T14:28:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep": "0.000000", 2023-01-15T14:28:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2023-01-15T14:28:21.257 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2023-01-15T14:28:21.258 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2023-01-15T14:28:21.258 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_stats_update_period_not_scrubbing": "120", 2023-01-15T14:28:21.258 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_stats_update_period_scrubbing": "15", 2023-01-15T14:28:21.424 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2023-01-15T14:28:21.424 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_target_transaction_size": "30", 2023-01-15T14:28:21.425 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_count": "4", 2023-01-15T14:28:21.425 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2023-01-15T14:28:21.425 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2023-01-15T14:28:21.425 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2023-01-15T14:28:21.425 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2023-01-15T14:28:21.426 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2023-01-15T14:28:21.426 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2023-01-15T14:28:21.426 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_default_cache_mode": "writeback", 2023-01-15T14:28:21.426 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2023-01-15T14:28:21.426 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tier_promote_max_objects_sec": "25", 2023-01-15T14:28:21.426 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_tracing": "false", 2023-01-15T14:28:21.426 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_use_stale_snap": "false", 2023-01-15T14:28:21.427 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2023-01-15T14:28:21.427 INFO:teuthology.orchestra.run.smithi090.stdout: "osdc_blkin_trace_all": "false", 2023-01-15T14:28:21.427 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_kill_at": "0", 2023-01-15T14:28:21.427 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_max_join_drift": "10", 2023-01-15T14:28:21.427 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_min": "500", 2023-01-15T14:28:21.427 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_min_wait": "0.050000", 2023-01-15T14:28:21.427 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_propose_interval": "1.000000", 2023-01-15T14:28:21.427 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_service_trim_max": "500", 2023-01-15T14:28:21.427 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_service_trim_max_multiplier": "20", 2023-01-15T14:28:21.427 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_service_trim_min": "10", 2023-01-15T14:28:21.427 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_stash_full_interval": "25", 2023-01-15T14:28:21.428 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_trim_max": "500", 2023-01-15T14:28:21.428 INFO:teuthology.orchestra.run.smithi090.stdout: "paxos_trim_min": "250", 2023-01-15T14:28:21.428 INFO:teuthology.orchestra.run.smithi090.stdout: "perf": "true", 2023-01-15T14:28:21.428 INFO:teuthology.orchestra.run.smithi090.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2023-01-15T14:28:21.428 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin_crypto_accelerator": "crypto_isal", 2023-01-15T14:28:21.428 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin_dir": "/usr/lib/ceph", 2023-01-15T14:28:21.428 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "-", 2023-01-15T14:28:21.428 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrv": "", 2023-01-15T14:28:21.537 INFO:teuthology.orchestra.run.smithi090.stdout: "public_bind_addr": "-", 2023-01-15T14:28:21.538 INFO:teuthology.orchestra.run.smithi090.stdout: "public_network": "", 2023-01-15T14:28:21.538 INFO:teuthology.orchestra.run.smithi090.stdout: "public_network_interface": "", 2023-01-15T14:28:21.538 INFO:teuthology.orchestra.run.smithi090.stdout: "qat_compressor_enabled": "false", 2023-01-15T14:28:21.538 INFO:teuthology.orchestra.run.smithi090.stdout: "qat_compressor_session_max_number": "256", 2023-01-15T14:28:21.538 INFO:teuthology.orchestra.run.smithi090.stdout: "rados_mon_op_timeout": "0", 2023-01-15T14:28:21.538 INFO:teuthology.orchestra.run.smithi090.stdout: "rados_osd_op_timeout": "0", 2023-01-15T14:28:21.538 INFO:teuthology.orchestra.run.smithi090.stdout: "rados_tracing": "false", 2023-01-15T14:28:21.538 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_atime_update_interval": "60", 2023-01-15T14:28:21.538 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2023-01-15T14:28:21.539 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_balance_parent_reads": "false", 2023-01-15T14:28:21.539 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_balance_snap_reads": "false", 2023-01-15T14:28:21.542 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_blkin_trace_all": "false", 2023-01-15T14:28:21.542 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_blocklist_expire_seconds": "0", 2023-01-15T14:28:21.542 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_blocklist_on_break_lock": "true", 2023-01-15T14:28:21.542 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache": "true", 2023-01-15T14:28:21.542 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_block_writes_upfront": "false", 2023-01-15T14:28:21.542 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_max_dirty": "25165824", 2023-01-15T14:28:21.542 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_max_dirty_age": "1.000000", 2023-01-15T14:28:21.542 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_max_dirty_object": "0", 2023-01-15T14:28:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_policy": "writearound", 2023-01-15T14:28:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_size": "33554432", 2023-01-15T14:28:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_target_dirty": "16777216", 2023-01-15T14:28:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_cache_writethrough_until_flush": "true", 2023-01-15T14:28:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_clone_copy_on_read": "false", 2023-01-15T14:28:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_compression_hint": "none", 2023-01-15T14:28:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_concurrent_management_ops": "10", 2023-01-15T14:28:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_config_pool_override_update_timestamp": "0", 2023-01-15T14:28:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_clone_format": "auto", 2023-01-15T14:28:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_data_pool": "", 2023-01-15T14:28:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_features": "61", 2023-01-15T14:28:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_format": "2", 2023-01-15T14:28:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_map_options": "", 2023-01-15T14:28:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_order": "22", 2023-01-15T14:28:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_pool": "rbd", 2023-01-15T14:28:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2023-01-15T14:28:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_stripe_count": "0", 2023-01-15T14:28:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_default_stripe_unit": "0", 2023-01-15T14:28:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_disable_zero_copy_writes": "true", 2023-01-15T14:28:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_discard_granularity_bytes": "65536", 2023-01-15T14:28:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_discard_on_zeroed_write_same": "true", 2023-01-15T14:28:21.545 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_enable_alloc_hint": "true", 2023-01-15T14:28:21.545 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2023-01-15T14:28:21.545 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_io_scheduler": "simple", 2023-01-15T14:28:21.545 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2023-01-15T14:28:21.545 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_commit_age": "5.000000", 2023-01-15T14:28:21.545 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2023-01-15T14:28:21.545 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_max_payload_bytes": "16384", 2023-01-15T14:28:21.545 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_flush_age": "0.000000", 2023-01-15T14:28:21.545 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_flush_bytes": "1048576", 2023-01-15T14:28:21.545 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_flush_interval": "0", 2023-01-15T14:28:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2023-01-15T14:28:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2023-01-15T14:28:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_order": "24", 2023-01-15T14:28:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_pool": "", 2023-01-15T14:28:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_journal_splay_width": "4", 2023-01-15T14:28:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_localize_parent_reads": "false", 2023-01-15T14:28:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_localize_snap_reads": "false", 2023-01-15T14:28:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2023-01-15T14:28:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2023-01-15T14:28:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2023-01-15T14:28:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_type": "simple", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_image_state_check_interval": "30", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_autotune": "true", 2023-01-15T14:28:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_base": "805306368", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_cache_min": "134217728", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_memory_target": "4294967296", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_perf_stats_prio": "5", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_delete_delay": "0", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_replay_delay": "0", 2023-01-15T14:28:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_move_to_trash_on_remove": "false", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_mtime_update_interval": "60", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_non_blocking_aio": "true", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_op_thread_timeout": "60", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_op_threads": "1", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_parent_cache_enabled": "false", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_persistent_cache_mode": "disabled", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_persistent_cache_path": "/tmp", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_persistent_cache_size": "1073741824", 2023-01-15T14:28:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_plugins": "", 2023-01-15T14:28:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_bps_burst": "0", 2023-01-15T14:28:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_bps_burst_seconds": "1", 2023-01-15T14:28:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_bps_limit": "0", 2023-01-15T14:28:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_exclude_ops": "0", 2023-01-15T14:28:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_iops_burst": "0", 2023-01-15T14:28:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_iops_burst_seconds": "1", 2023-01-15T14:28:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_iops_limit": "0", 2023-01-15T14:28:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_bps_burst": "0", 2023-01-15T14:28:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2023-01-15T14:28:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_bps_limit": "0", 2023-01-15T14:28:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_iops_burst": "0", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_read_iops_limit": "0", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_schedule_tick_min": "50", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_bps_burst": "0", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_bps_limit": "0", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_iops_burst": "0", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_qos_write_iops_limit": "0", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_quiesce_notification_attempts": "10", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_read_from_replica_policy": "default", 2023-01-15T14:28:21.551 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2023-01-15T14:28:21.552 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_readahead_max_bytes": "524288", 2023-01-15T14:28:21.552 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_readahead_trigger_requests": "10", 2023-01-15T14:28:21.552 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_request_timed_out_seconds": "30", 2023-01-15T14:28:21.552 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_skip_partial_discard": "true", 2023-01-15T14:28:21.552 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2023-01-15T14:28:21.552 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_tracing": "false", 2023-01-15T14:28:21.552 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_validate_names": "true", 2023-01-15T14:28:21.552 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd_validate_pool": "true", 2023-01-15T14:28:21.552 INFO:teuthology.orchestra.run.smithi090.stdout: "restapi_base_url": "", 2023-01-15T14:28:21.552 INFO:teuthology.orchestra.run.smithi090.stdout: "restapi_log_level": "", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_acl_grants_max_num": "100", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_admin_entry": "admin", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_backend_store": "rados", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_barbican_url": "", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_beast_enable_async": "true", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_default_quota_max_size": "-1", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_index_max_aio": "128", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_quota_cache_size": "10000", 2023-01-15T14:28:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_quota_ttl": "600", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_bucket_sync_spawn_window": "20", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cache_enabled": "true", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cache_expiry_interval": "900", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cache_lru_size": "10000", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_config_store": "rados", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_content_length_compat": "false", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_copy_obj_progress": "true", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cors_rules_max_num": "100", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_cross_domain_policy": "", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_default_encryption_key": "", 2023-01-15T14:28:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_addr": "", 2023-01-15T14:28:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_ca_path": "", 2023-01-15T14:28:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_client_cert": "", 2023-01-15T14:28:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_client_key": "", 2023-01-15T14:28:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_kms_key_template": "", 2023-01-15T14:28:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_password": "", 2023-01-15T14:28:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2023-01-15T14:28:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_kmip_username": "", 2023-01-15T14:28:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_require_ssl": "true", 2023-01-15T14:28:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2023-01-15T14:28:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2023-01-15T14:28:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_backend": "vault", 2023-01-15T14:28:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2023-01-15T14:28:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2023-01-15T14:28:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2023-01-15T14:28:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2023-01-15T14:28:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2023-01-15T14:28:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2023-01-15T14:28:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2023-01-15T14:28:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2023-01-15T14:28:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2023-01-15T14:28:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2023-01-15T14:28:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2023-01-15T14:28:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_suppress_logs": "true", 2023-01-15T14:28:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_addr": "", 2023-01-15T14:28:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_auth": "token", 2023-01-15T14:28:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_namespace": "", 2023-01-15T14:28:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_prefix": "", 2023-01-15T14:28:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_secret_engine": "transit", 2023-01-15T14:28:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_ssl_cacert": "", 2023-01-15T14:28:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2023-01-15T14:28:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2023-01-15T14:28:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_token_file": "", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_crypt_vault_verify_ssl": "true", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_buffersize": "524288", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_low_speed_limit": "1024", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_low_speed_time": "300", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_curl_wait_timeout_ms": "1000", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_fadvise": "4", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2023-01-15T14:28:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_libaio_aio_num": "64", 2023-01-15T14:28:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_d3n_libaio_aio_threads": "20", 2023-01-15T14:28:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2023-01-15T14:28:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_changes_size": "1000", 2023-01-15T14:28:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_num_shards": "128", 2023-01-15T14:28:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_obj_prefix": "data_log", 2023-01-15T14:28:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_log_window": "30", 2023-01-15T14:28:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_notify_interval_msec": "0", 2023-01-15T14:28:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_sync_poll_interval": "20", 2023-01-15T14:28:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_data_sync_spawn_window": "20", 2023-01-15T14:28:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_data_log_backing": "fifo", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_realm_info_oid": "default.realm", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_region_info_oid": "default.region", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_zone_info_oid": "default.zone", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_defer_to_bucket_acls": "", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_delete_multi_obj_max_num": "1000", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_admin_lim": "0.000000", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_admin_res": "100.000000", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_auth_lim": "0.000000", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_auth_res": "200.000000", 2023-01-15T14:28:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2023-01-15T14:28:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_data_lim": "0.000000", 2023-01-15T14:28:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_data_res": "500.000000", 2023-01-15T14:28:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_data_wgt": "500.000000", 2023-01-15T14:28:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2023-01-15T14:28:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_metadata_res": "500.000000", 2023-01-15T14:28:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2023-01-15T14:28:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dns_name": "", 2023-01-15T14:28:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dns_s3website_name": "", 2023-01-15T14:28:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_dynamic_resharding": "true", 2023-01-15T14:28:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2023-01-15T14:28:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_gc_threads": "true", 2023-01-15T14:28:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_lc_threads": "true", 2023-01-15T14:28:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_ops_log": "false", 2023-01-15T14:28:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_quota_threads": "true", 2023-01-15T14:28:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_static_website": "false", 2023-01-15T14:28:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enable_usage_log": "false", 2023-01-15T14:28:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_enforce_swift_acls": "true", 2023-01-15T14:28:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_exit_timeout_secs": "120", 2023-01-15T14:28:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_expose_bucket": "false", 2023-01-15T14:28:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_extended_http_attrs": "", 2023-01-15T14:28:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_filter": "none", 2023-01-15T14:28:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2023-01-15T14:28:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_frontends": "beast port=7480", 2023-01-15T14:28:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_concurrent_io": "10", 2023-01-15T14:28:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_deferred": "50", 2023-01-15T14:28:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2023-01-15T14:28:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_objs": "32", 2023-01-15T14:28:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_queue_size": "134213632", 2023-01-15T14:28:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_max_trim_chunk": "16", 2023-01-15T14:28:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_obj_min_wait": "7200", 2023-01-15T14:28:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_processor_max_time": "3600", 2023-01-15T14:28:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_gc_processor_period": "3600", 2023-01-15T14:28:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_get_obj_max_req_size": "4194304", 2023-01-15T14:28:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_get_obj_window_size": "16777216", 2023-01-15T14:28:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_healthcheck_disabling_path": "", 2023-01-15T14:28:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ignore_get_invalid_range": "false", 2023-01-15T14:28:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_init_timeout": "300", 2023-01-15T14:28:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2023-01-15T14:28:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2023-01-15T14:28:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_accepted_admin_roles": "", 2023-01-15T14:28:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2023-01-15T14:28:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_domain": "", 2023-01-15T14:28:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_password": "", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_password_path": "", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_project": "", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_tenant": "", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_token": "", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_token_path": "", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_admin_user": "", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_api_version": "2", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_domain": "", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_password": "", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_project": "", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_tenant": "", 2023-01-15T14:28:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_barbican_user": "", 2023-01-15T14:28:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2023-01-15T14:28:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_implicit_tenants": "false", 2023-01-15T14:28:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2023-01-15T14:28:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_service_token_enabled": "false", 2023-01-15T14:28:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_token_cache_size": "10000", 2023-01-15T14:28:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_url": "", 2023-01-15T14:28:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_keystone_verify_ssl": "true", 2023-01-15T14:28:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_debug_interval": "-1", 2023-01-15T14:28:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_lock_max_time": "90", 2023-01-15T14:28:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_objs": "32", 2023-01-15T14:28:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_rules": "1000", 2023-01-15T14:28:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_worker": "3", 2023-01-15T14:28:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_max_wp_worker": "3", 2023-01-15T14:28:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lc_thread_delay": "0", 2023-01-15T14:28:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2023-01-15T14:28:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_dnattr": "uid", 2023-01-15T14:28:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2023-01-15T14:28:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_searchfilter": "", 2023-01-15T14:28:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2023-01-15T14:28:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ldap_uri": "ldaps://", 2023-01-15T14:28:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2023-01-15T14:28:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_list_bucket_min_readahead": "1000", 2023-01-15T14:28:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_list_buckets_max_chunk": "1000", 2023-01-15T14:28:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_http_headers": "", 2023-01-15T14:28:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_nonexistent_bucket": "false", 2023-01-15T14:28:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2023-01-15T14:28:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_log_object_name_utc": "false", 2023-01-15T14:28:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2023-01-15T14:28:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_attr_name_len": "0", 2023-01-15T14:28:21.698 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_attr_size": "0", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_attrs_num_in_req": "0", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_chunk_size": "4194304", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_concurrent_requests": "1024", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_dynamic_shards": "1999", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_listing_results": "1000", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_notify_retries": "3", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_objs_per_shard": "100000", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_put_param_size": "1048576", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_put_size": "5368709120", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_max_slo_entries": "1000", 2023-01-15T14:28:21.699 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_md_log_max_shards": "64", 2023-01-15T14:28:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_md_notify_interval_msec": "200", 2023-01-15T14:28:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_meta_sync_poll_interval": "20", 2023-01-15T14:28:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_meta_sync_spawn_window": "20", 2023-01-15T14:28:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_mime_types_file": "/etc/mime.types", 2023-01-15T14:28:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_mp_lock_max_time": "600", 2023-01-15T14:28:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_multi_obj_del_max_aio": "16", 2023-01-15T14:28:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_multipart_min_part_size": "5242880", 2023-01-15T14:28:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_multipart_part_upload_limit": "10000", 2023-01-15T14:28:21.700 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_fhcache_partitions": "3", 2023-01-15T14:28:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_fhcache_size": "2017", 2023-01-15T14:28:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_frontends": "rgw-nfs", 2023-01-15T14:28:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2023-01-15T14:28:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_lru_lanes": "5", 2023-01-15T14:28:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_max_gc": "300", 2023-01-15T14:28:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_namespace_expire_secs": "300", 2023-01-15T14:28:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_gc_threads": "false", 2023-01-15T14:28:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_lc_threads": "false", 2023-01-15T14:28:21.702 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_quota_threads": "false", 2023-01-15T14:28:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_run_sync_thread": "false", 2023-01-15T14:28:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_s3_fast_attrs": "false", 2023-01-15T14:28:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_nfs_write_completion_interval_s": "10", 2023-01-15T14:28:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_num_async_rados_threads": "32", 2023-01-15T14:28:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_num_control_oids": "8", 2023-01-15T14:28:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_numa_node": "-1", 2023-01-15T14:28:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_obj_stripe_size": "4194304", 2023-01-15T14:28:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_obj_tombstone_cache_size": "1000", 2023-01-15T14:28:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_objexp_chunk_size": "100", 2023-01-15T14:28:21.703 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_objexp_gc_interval": "600", 2023-01-15T14:28:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_objexp_hints_num_shards": "127", 2023-01-15T14:28:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_olh_pending_timeout_sec": "3600", 2023-01-15T14:28:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_op_thread_suicide_timeout": "0", 2023-01-15T14:28:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_op_thread_timeout": "600", 2023-01-15T14:28:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_op_tracing": "false", 2023-01-15T14:28:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_opa_token": "", 2023-01-15T14:28:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_opa_url": "", 2023-01-15T14:28:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_opa_verify_ssl": "true", 2023-01-15T14:28:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_data_backlog": "5242880", 2023-01-15T14:28:21.704 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2023-01-15T14:28:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_rados": "false", 2023-01-15T14:28:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_ops_log_socket_path": "", 2023-01-15T14:28:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_override_bucket_index_max_shards": "0", 2023-01-15T14:28:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2023-01-15T14:28:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2023-01-15T14:28:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_push_interval": "2.000000", 2023-01-15T14:28:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_push_interval_max": "30.000000", 2023-01-15T14:28:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_period_root_pool": ".rgw.root", 2023-01-15T14:28:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_policy_reject_invalid_principals": "true", 2023-01-15T14:28:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_print_continue": "true", 2023-01-15T14:28:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_print_prohibited_content_length": "false", 2023-01-15T14:28:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_put_obj_max_window_size": "67108864", 2023-01-15T14:28:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_put_obj_min_window_size": "16777216", 2023-01-15T14:28:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2023-01-15T14:28:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rados_pool_recovery_priority": "5", 2023-01-15T14:28:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rados_tracing": "false", 2023-01-15T14:28:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_realm": "", 2023-01-15T14:28:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_realm_id": "", 2023-01-15T14:28:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_realm_root_pool": ".rgw.root", 2023-01-15T14:28:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_region": "", 2023-01-15T14:28:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_region_root_pool": ".rgw.root", 2023-01-15T14:28:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_relaxed_region_enforcement": "false", 2023-01-15T14:28:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_relaxed_s3_bucket_names": "false", 2023-01-15T14:28:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2023-01-15T14:28:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_request_uri": "", 2023-01-15T14:28:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_batch_size": "64", 2023-01-15T14:28:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_bucket_lock_duration": "360", 2023-01-15T14:28:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_max_aio": "128", 2023-01-15T14:28:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_num_logs": "16", 2023-01-15T14:28:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_reshard_thread_interval": "600", 2023-01-15T14:28:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_resolve_cname": "false", 2023-01-15T14:28:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_rest_getusage_op_compat": "false", 2023-01-15T14:28:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_run_sync_thread": "true", 2023-01-15T14:28:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_order": "sts, external, local", 2023-01-15T14:28:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_keystone": "false", 2023-01-15T14:28:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_ldap": "false", 2023-01-15T14:28:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_rados": "true", 2023-01-15T14:28:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_auth_use_sts": "false", 2023-01-15T14:28:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_client_max_sig_ver": "-1", 2023-01-15T14:28:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_s3_success_create_obj_status": "0", 2023-01-15T14:28:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_safe_max_objects_per_shard": "102400", 2023-01-15T14:28:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_scheduler_type": "throttler", 2023-01-15T14:28:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_script_uri": "", 2023-01-15T14:28:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_service_provider_name": "", 2023-01-15T14:28:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_shard_warning_threshold": "90.000000", 2023-01-15T14:28:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_client_id": "", 2023-01-15T14:28:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_client_secret": "", 2023-01-15T14:28:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_entry": "sts", 2023-01-15T14:28:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_key": "sts", 2023-01-15T14:28:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_max_session_duration": "43200", 2023-01-15T14:28:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_min_session_duration": "900", 2023-01-15T14:28:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sts_token_introspection_url": "", 2023-01-15T14:28:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_account_in_url": "false", 2023-01-15T14:28:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_auth_entry": "auth", 2023-01-15T14:28:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_auth_url": "", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_custom_header": "", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_enforce_content_length": "false", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_need_stats": "true", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_tenant_name": "", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_token_expiration": "86400", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_url": "", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_url_prefix": "swift", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_swift_versioning_enabled": "false", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_lease_period": "120", 2023-01-15T14:28:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2023-01-15T14:28:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_interval": "1200", 2023-01-15T14:28:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_max_buckets": "16", 2023-01-15T14:28:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2023-01-15T14:28:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2023-01-15T14:28:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_obj_etag_verify": "false", 2023-01-15T14:28:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_trace_history_size": "4096", 2023-01-15T14:28:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_trace_per_node_log_size": "32", 2023-01-15T14:28:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2023-01-15T14:28:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_thread_pool_size": "512", 2023-01-15T14:28:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_comment": "", 2023-01-15T14:28:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_createby": "", 2023-01-15T14:28:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_encoding": "", 2023-01-15T14:28:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_flag": "false", 2023-01-15T14:28:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_origin": "", 2023-01-15T14:28:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_sha_unit": "524288", 2023-01-15T14:28:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_torrent_tracker": "", 2023-01-15T14:28:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_trust_forwarded_https": "false", 2023-01-15T14:28:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_log_flush_threshold": "1024", 2023-01-15T14:28:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_log_tick_interval": "30", 2023-01-15T14:28:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_max_shards": "32", 2023-01-15T14:28:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_usage_max_user_shards": "1", 2023-01-15T14:28:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_use_opa_authz": "false", 2023-01-15T14:28:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_default_quota_max_objects": "-1", 2023-01-15T14:28:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_default_quota_max_size": "-1", 2023-01-15T14:28:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_max_buckets": "1000", 2023-01-15T14:28:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_policies_max_num": "100", 2023-01-15T14:28:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2023-01-15T14:28:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_sync_idle_users": "false", 2023-01-15T14:28:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_sync_interval": "86400", 2023-01-15T14:28:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_quota_sync_wait_time": "86400", 2023-01-15T14:28:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_user_unique_email": "true", 2023-01-15T14:28:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_verify_ssl": "true", 2023-01-15T14:28:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_website_routing_rules_max_num": "50", 2023-01-15T14:28:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zone": "", 2023-01-15T14:28:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zone_id": "", 2023-01-15T14:28:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zone_root_pool": ".rgw.root", 2023-01-15T14:28:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zonegroup": "", 2023-01-15T14:28:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zonegroup_id": "", 2023-01-15T14:28:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2023-01-15T14:28:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "rgwlc_auto_session_clear": "true", 2023-01-15T14:28:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "rgwlc_skip_bucket_step": "false", 2023-01-15T14:28:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_block_size": "4096", 2023-01-15T14:28:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_bloom_bits_per_key": "20", 2023-01-15T14:28:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2023-01-15T14:28:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2023-01-15T14:28:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_row_ratio": "0.000000", 2023-01-15T14:28:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_shard_bits": "4", 2023-01-15T14:28:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_size": "536870912", 2023-01-15T14:28:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cache_type": "binned_lru", 2023-01-15T14:28:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cf_compact_on_deletion": "true", 2023-01-15T14:28:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_collect_compaction_stats": "false", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_collect_extended_stats": "false", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_collect_memory_stats": "false", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_delete_range_threshold": "1048576", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_index_type": "binary_search", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_log_to_ceph_log": "true", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_metadata_block_size": "4096", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_partition_filters": "false", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_perf": "false", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2023-01-15T14:28:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "rotating_keys_bootstrap_timeout": "30", 2023-01-15T14:28:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "rotating_keys_renewal_timeout": "10", 2023-01-15T14:28:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "run_dir": "/var/run/ceph", 2023-01-15T14:28:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_block_create": "true", 2023-01-15T14:28:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_cache_lru_size": "67108864", 2023-01-15T14:28:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_default_max_object_size": "16777216", 2023-01-15T14:28:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_default_object_metadata_reservation": "16777216", 2023-01-15T14:28:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_device_size": "53687091200", 2023-01-15T14:28:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_batch_capacity": "16", 2023-01-15T14:28:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_batch_flush_size": "16777216", 2023-01-15T14:28:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2023-01-15T14:28:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_journal_iodepth_limit": "5", 2023-01-15T14:28:21.730 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_obj_data_write_amplification": "1.250000", 2023-01-15T14:28:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "seastore_segment_size": "67108864", 2023-01-15T14:28:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "set_keepcaps": "false", 2023-01-15T14:28:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "setgroup": "", 2023-01-15T14:28:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "setuser": "", 2023-01-15T14:28:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "setuser_match_path": "", 2023-01-15T14:28:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_misplaced_ratio": "0.050000", 2023-01-15T14:28:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "thp": "false", 2023-01-15T14:28:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "threadpool_default_timeout": "60", 2023-01-15T14:28:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "threadpool_empty_queue_max_wait": "2", 2023-01-15T14:28:21.731 INFO:teuthology.orchestra.run.smithi090.stdout: "throttler_perf_counter": "true" 2023-01-15T14:28:21.731 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:28:21.734 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2023-01-15T14:28:21.878 INFO:teuthology.orchestra.run.smithi090.stdout:mon.a: {} 2023-01-15T14:28:21.892 INFO:teuthology.orchestra.run.smithi090.stdout:mon.b: {} 2023-01-15T14:28:21.904 INFO:teuthology.orchestra.run.smithi090.stdout:mon.c: {} 2023-01-15T14:28:21.999 INFO:teuthology.orchestra.run.smithi090.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-01-15T14:28:21.999 INFO:teuthology.orchestra.run.smithi090.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-01-15T14:28:22.000 INFO:teuthology.orchestra.run.smithi090.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2023-01-15T14:28:22.000 DEBUG:teuthology.orchestra.run.smithi090:> type ceph-objectstore-tool 2023-01-15T14:28:22.009 INFO:teuthology.orchestra.run.smithi090.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-01-15T14:28:22.025 DEBUG:teuthology.orchestra.run.smithi141:> type ceph-objectstore-tool 2023-01-15T14:28:22.032 INFO:teuthology.orchestra.run.smithi141.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2023-01-15T14:28:22.035 INFO:teuthology.run_tasks:Running task rados... 2023-01-15T14:28:22.046 INFO:tasks.rados:Beginning rados... 2023-01-15T14:28:22.051 DEBUG:teuthology.run_tasks:Unwinding manager rados 2023-01-15T14:28:22.059 INFO:tasks.rados:joining rados 2023-01-15T14:28:22.059 INFO:tasks.thrashosds.thrasher:starting do_thrash 2023-01-15T14:28:22.059 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:28:22.059 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:28:22.060 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2023-01-15T14:28:22.060 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2023-01-15T14:28:22.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:28:22.061 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2023-01-15T14:28:22.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:28:22.061 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2023-01-15T14:28:22.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:28:22.062 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2023-01-15T14:28:22.062 INFO:tasks.rados:starting run 0 out of 1 2023-01-15T14:28:22.062 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2023-01-15T14:28:22.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2023-01-15T14:28:22.073 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:22.070+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:22.171 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:22.166+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:28:22.197 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:28:22.217 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:22.217 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:22.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:28:22.227 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:28:22.350 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:28:22.351 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:28:22.351 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:28:22.351 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:28:22.351 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:22.351 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:22.352 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:28:22.353 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:28:22.353 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:28:22.353 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:28:22.353 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:28:22.354 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:28:22.354 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:28:22.354 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:28:22.354 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:28:22.354 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:28:22.354 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:28:22.354 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:22.341+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:22.451 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:22.509 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:22.510 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:22.445+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:28:22.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:28:22.552 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:22.545+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:22.672 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:22.672 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:22.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:28:22.673 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:22.665+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:22.738 INFO:teuthology.orchestra.run.smithi090.stderr:pool 'unique_pool_0' created 2023-01-15T14:28:22.811 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:28:22.812 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2023-01-15T14:28:22.814 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:22.805+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:22.818 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:22.819 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:22.833 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:28:22.912 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:22.905+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:28:22.982 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:22.989 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:22.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:28:23.013 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:23.010+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:28:23.094 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:23.123 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:23.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:28:23.125 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:23.117+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:23.226 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:23.217+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:28:23.289 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:23.289 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:23.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:28:23.327 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:23.322+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:23.395 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:23.395 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:23.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:28:23.549 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:23.542+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:28:23.650 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:23.641+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:28:23.756 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:23.756 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:23.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:28:23.758 INFO:teuthology.orchestra.run.smithi090.stderr:enabled application 'rados' on pool 'unique_pool_0' 2023-01-15T14:28:23.758 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:23.750+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:28:23.856 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:23.850+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:28:24.130 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:24.130 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:24.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:28:24.136 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:24.121+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:24.231 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:24.225+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:24.307 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:24.307 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:24.308 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:28:24.351 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:24.345+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:24.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:28:24.726 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:24.717+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:28:24.727 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:24.728 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:24.728 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:28:24.759 DEBUG:teuthology.orchestra.run.smithi090:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2023-01-15T14:28:24.795 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight read -> 100 2023-01-15T14:28:24.917 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight write -> 50 2023-01-15T14:28:24.917 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight delete -> 50 2023-01-15T14:28:24.917 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight snap_create -> 50 2023-01-15T14:28:24.917 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight snap_remove -> 50 2023-01-15T14:28:24.917 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight rollback -> 50 2023-01-15T14:28:24.917 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight copy_from -> 50 2023-01-15T14:28:24.918 INFO:tasks.rados.rados.0.smithi090.stdout:adding op weight write_excl -> 50 2023-01-15T14:28:24.918 INFO:tasks.rados.rados.0.smithi090.stdout:ceph version 18.0.0-1736-ga21b2f4f (a21b2f4fd40d883dac62c8cd315cbb54848ee067) reef (dev) 2023-01-15T14:28:24.918 INFO:tasks.rados.rados.0.smithi090.stdout:Configuration: 2023-01-15T14:28:24.918 INFO:tasks.rados.rados.0.smithi090.stdout: Number of operations: 4000 2023-01-15T14:28:24.918 INFO:tasks.rados.rados.0.smithi090.stdout: Number of objects: 50 2023-01-15T14:28:24.918 INFO:tasks.rados.rados.0.smithi090.stdout: Max in flight operations: 16 2023-01-15T14:28:24.918 INFO:tasks.rados.rados.0.smithi090.stdout: Object size (in bytes): 4000000 2023-01-15T14:28:24.918 INFO:tasks.rados.rados.0.smithi090.stdout: Write stride min: 400000 2023-01-15T14:28:24.919 INFO:tasks.rados.rados.0.smithi090.stdout: Write stride max: 800000 2023-01-15T14:28:24.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:28:24.921 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:28:24.921 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:28:24.921 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:28:24.922 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:24.922 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:24.922 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:28:24.922 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:24.913+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:24.930 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:28:24.951 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:28:25.133 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:28:25.133 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:28:25.133 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:28:25.134 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:25.134 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:28:25.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:28:25.135 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [] 2023-01-15T14:28:25.135 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:25.135 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:28:25.136 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:28:25.136 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:28:25.136 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:28:25.136 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:28:25.136 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:28:25.538 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:28:25.538 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:28:25.538 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:28:25.538 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:25.125+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:28:25.540 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:25.533+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:28:25.641 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:25.633+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:25.737 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:25.738 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:28:25.738 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:28:25.738 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:25.738 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:28:25.757 INFO:tasks.rados.rados.0.smithi090.stdout:1: write initial oid 1 2023-01-15T14:28:25.924 INFO:tasks.rados.rados.0.smithi090.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2023-01-15T14:28:25.925 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:25.781+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:28:25.926 INFO:tasks.rados.rados.0.smithi090.stdout:1: writing smithi09025264-1 from 416807 to 892056 tid 1 2023-01-15T14:28:25.926 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:28:25.927 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:25.927 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:28:25.927 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:28:25.928 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:28:25.928 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:25.928 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:28:25.929 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:25.921+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:25.929 INFO:tasks.rados.rados.0.smithi090.stdout:1: writing smithi09025264-1 from 1542129 to 2016807 tid 2 2023-01-15T14:28:25.929 INFO:tasks.rados.rados.0.smithi090.stdout:2: write initial oid 2 2023-01-15T14:28:25.929 INFO:tasks.rados.rados.0.smithi090.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2023-01-15T14:28:25.929 INFO:tasks.rados.rados.0.smithi090.stdout:2: writing smithi09025264-2 from 416807 to 892056 tid 1 2023-01-15T14:28:25.934 INFO:tasks.rados.rados.0.smithi090.stdout:2: writing smithi09025264-2 from 1542129 to 2016807 tid 2 2023-01-15T14:28:25.934 INFO:tasks.rados.rados.0.smithi090.stdout:3: write initial oid 3 2023-01-15T14:28:25.934 INFO:tasks.rados.rados.0.smithi090.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2023-01-15T14:28:25.934 INFO:tasks.rados.rados.0.smithi090.stdout:3: writing smithi09025264-3 from 433614 to 984112 tid 1 2023-01-15T14:28:25.934 INFO:tasks.rados.rados.0.smithi090.stdout:3: writing smithi09025264-3 from 1600611 to 2033614 tid 2 2023-01-15T14:28:25.934 INFO:tasks.rados.rados.0.smithi090.stdout:4: write initial oid 4 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:4: writing smithi09025264-4 from 450421 to 1076168 tid 1 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:4: writing smithi09025264-4 from 1659093 to 2050421 tid 2 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:5: write initial oid 5 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:5: writing smithi09025264-5 from 467228 to 1168224 tid 1 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:5: writing smithi09025264-5 from 1717575 to 2067228 tid 2 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:6: write initial oid 6 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:6: writing smithi09025264-6 from 484035 to 1260280 tid 1 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:6: writing smithi09025264-6 from 2059704 to 2084035 tid 2 2023-01-15T14:28:25.935 INFO:tasks.rados.rados.0.smithi090.stdout:7: write initial oid 7 2023-01-15T14:28:25.936 INFO:tasks.rados.rados.0.smithi090.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2023-01-15T14:28:25.936 INFO:tasks.rados.rados.0.smithi090.stdout:7: writing smithi09025264-7 from 500842 to 952336 tid 1 2023-01-15T14:28:25.936 INFO:tasks.rados.rados.0.smithi090.stdout:7: writing smithi09025264-7 from 1718186 to 2100842 tid 2 2023-01-15T14:28:25.936 INFO:tasks.rados.rados.0.smithi090.stdout:8: write initial oid 8 2023-01-15T14:28:25.936 INFO:tasks.rados.rados.0.smithi090.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2023-01-15T14:28:25.936 INFO:tasks.rados.rados.0.smithi090.stdout:8: writing smithi09025264-8 from 517649 to 1044392 tid 1 2023-01-15T14:28:25.938 INFO:tasks.rados.rados.0.smithi090.stdout:8: writing smithi09025264-8 from 1776668 to 2117649 tid 2 2023-01-15T14:28:25.938 INFO:tasks.rados.rados.0.smithi090.stdout:9: write initial oid 9 2023-01-15T14:28:25.938 INFO:tasks.rados.rados.0.smithi090.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2023-01-15T14:28:25.938 INFO:tasks.rados.rados.0.smithi090.stdout:9: writing smithi09025264-9 from 534456 to 1252801 tid 1 2023-01-15T14:28:25.939 INFO:tasks.rados.rados.0.smithi090.stdout:9: writing smithi09025264-9 from 1951503 to 2134456 tid 2 2023-01-15T14:28:25.939 INFO:tasks.rados.rados.0.smithi090.stdout:10: write initial oid 10 2023-01-15T14:28:25.939 INFO:tasks.rados.rados.0.smithi090.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2023-01-15T14:28:25.939 INFO:tasks.rados.rados.0.smithi090.stdout:10: writing smithi09025264-10 from 551263 to 1344857 tid 1 2023-01-15T14:28:25.939 INFO:tasks.rados.rados.0.smithi090.stdout:10: writing smithi09025264-10 from 1893632 to 2151263 tid 2 2023-01-15T14:28:25.939 INFO:tasks.rados.rados.0.smithi090.stdout:11: write initial oid 11 2023-01-15T14:28:25.939 INFO:tasks.rados.rados.0.smithi090.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2023-01-15T14:28:25.939 INFO:tasks.rados.rados.0.smithi090.stdout:11: writing smithi09025264-11 from 568070 to 1036913 tid 1 2023-01-15T14:28:25.939 INFO:tasks.rados.rados.0.smithi090.stdout:11: writing smithi09025264-11 from 1552114 to 2168070 tid 2 2023-01-15T14:28:25.939 INFO:tasks.rados.rados.0.smithi090.stdout:12: write initial oid 12 2023-01-15T14:28:25.939 INFO:tasks.rados.rados.0.smithi090.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2023-01-15T14:28:25.940 INFO:tasks.rados.rados.0.smithi090.stdout:12: writing smithi09025264-12 from 584877 to 1128969 tid 1 2023-01-15T14:28:25.940 INFO:tasks.rados.rados.0.smithi090.stdout:12: writing smithi09025264-12 from 1610596 to 2172599 tid 2 2023-01-15T14:28:25.940 INFO:tasks.rados.rados.0.smithi090.stdout:13: write initial oid 13 2023-01-15T14:28:25.940 INFO:tasks.rados.rados.0.smithi090.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2023-01-15T14:28:25.940 INFO:tasks.rados.rados.0.smithi090.stdout:13: writing smithi09025264-13 from 601684 to 1221025 tid 1 2023-01-15T14:28:25.940 INFO:tasks.rados.rados.0.smithi090.stdout:13: writing smithi09025264-13 from 1669078 to 2201684 tid 2 2023-01-15T14:28:25.940 INFO:tasks.rados.rados.0.smithi090.stdout:14: write initial oid 14 2023-01-15T14:28:25.940 INFO:tasks.rados.rados.0.smithi090.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2023-01-15T14:28:25.940 INFO:tasks.rados.rados.0.smithi090.stdout:14: writing smithi09025264-14 from 618491 to 1313081 tid 1 2023-01-15T14:28:25.942 INFO:tasks.rados.rados.0.smithi090.stdout:14: writing smithi09025264-14 from 2011207 to 2218491 tid 2 2023-01-15T14:28:25.942 INFO:tasks.rados.rados.0.smithi090.stdout:15: write initial oid 15 2023-01-15T14:28:25.942 INFO:tasks.rados.rados.0.smithi090.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2023-01-15T14:28:25.945 INFO:tasks.rados.rados.0.smithi090.stdout:15: writing smithi09025264-15 from 635298 to 1405137 tid 1 2023-01-15T14:28:26.085 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:26.086 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:28:26.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:28:26.086 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:28:26.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.088 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:13 3.2 e22/20 3:4d8f159b:::smithi09025264-4:head v 22'1, mlcod=0'0)", 2023-01-15T14:28:26.088 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.936674+0000", 2023-01-15T14:28:26.088 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.10341817, 2023-01-15T14:28:26.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0045248969999999999, 2023-01-15T14:28:26.089 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:26.089 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:26.089 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:26.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:26.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936674+0000", 2023-01-15T14:28:26.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:26.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:26.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936674+0000", 2023-01-15T14:28:26.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:26.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:26.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936674+0000", 2023-01-15T14:28:26.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:26.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:26.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.937836+0000", 2023-01-15T14:28:26.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001162228 2023-01-15T14:28:26.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:26.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.937860+0000", 2023-01-15T14:28:26.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3606000000000001e-05 2023-01-15T14:28:26.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:26.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.937870+0000", 2023-01-15T14:28:26.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0550000000000001e-05 2023-01-15T14:28:26.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:26.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.938120+0000", 2023-01-15T14:28:26.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024984300000000003 2023-01-15T14:28:26.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:26.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.938182+0000", 2023-01-15T14:28:26.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1755999999999996e-05 2023-01-15T14:28:26.097 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.097 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:26.097 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.939174+0000", 2023-01-15T14:28:26.097 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000991818 2023-01-15T14:28:26.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.098 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.098 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:26.098 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.939224+0000", 2023-01-15T14:28:26.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0430000000000003e-05 2023-01-15T14:28:26.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.100 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:26.100 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.939522+0000", 2023-01-15T14:28:26.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029811300000000002 2023-01-15T14:28:26.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:26.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.939556+0000", 2023-01-15T14:28:26.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3905999999999997e-05 2023-01-15T14:28:26.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:26.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.941199+0000", 2023-01-15T14:28:26.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001643147 2023-01-15T14:28:26.333 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:26.333 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:26.333 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:26.334 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:26.334 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:26.334 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:26.334 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:28:26.335 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:26.077+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:26.339 INFO:tasks.rados.rados.0.smithi090.stdout:15: writing smithi09025264-15 from 2069689 to 2235298 tid 2 2023-01-15T14:28:26.339 INFO:tasks.rados.rados.0.smithi090.stdout:16: write initial oid 16 2023-01-15T14:28:26.341 INFO:tasks.rados.rados.0.smithi090.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2023-01-15T14:28:26.341 INFO:tasks.rados.rados.0.smithi090.stdout:16: writing smithi09025264-16 from 652105 to 1097193 tid 1 2023-01-15T14:28:26.341 INFO:tasks.rados.rados.0.smithi090.stdout:16: writing smithi09025264-16 from 1728171 to 2181159 tid 2 2023-01-15T14:28:26.342 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.342 INFO:tasks.rados.rados.0.smithi090.stdout:3: finishing write tid 1 to smithi09025264-3 2023-01-15T14:28:26.342 INFO:tasks.rados.rados.0.smithi090.stdout:7: finishing write tid 1 to smithi09025264-7 2023-01-15T14:28:26.342 INFO:tasks.rados.rados.0.smithi090.stdout:3: finishing write tid 2 to smithi09025264-3 2023-01-15T14:28:26.342 INFO:tasks.rados.rados.0.smithi090.stdout:1: finishing write tid 1 to smithi09025264-1 2023-01-15T14:28:26.502 INFO:tasks.rados.rados.0.smithi090.stdout:11: finishing write tid 1 to smithi09025264-11 2023-01-15T14:28:26.502 INFO:tasks.rados.rados.0.smithi090.stdout:10: finishing write tid 1 to smithi09025264-10 2023-01-15T14:28:26.502 INFO:tasks.rados.rados.0.smithi090.stdout:4: finishing write tid 1 to smithi09025264-4 2023-01-15T14:28:26.505 INFO:tasks.rados.rados.0.smithi090.stdout:9: finishing write tid 1 to smithi09025264-9 2023-01-15T14:28:26.505 INFO:tasks.rados.rados.0.smithi090.stdout:8: finishing write tid 1 to smithi09025264-8 2023-01-15T14:28:26.505 INFO:tasks.rados.rados.0.smithi090.stdout:2: finishing write tid 1 to smithi09025264-2 2023-01-15T14:28:26.505 INFO:tasks.rados.rados.0.smithi090.stdout:3: finishing write tid 3 to smithi09025264-3 2023-01-15T14:28:26.505 INFO:tasks.rados.rados.0.smithi090.stdout:3: finishing write tid 4 to smithi09025264-3 2023-01-15T14:28:26.506 INFO:tasks.rados.rados.0.smithi090.stdout:3: oid 3 updating version 0 to 1 2023-01-15T14:28:26.506 INFO:tasks.rados.rados.0.smithi090.stdout:3: oid 3 updating version 1 to 2 2023-01-15T14:28:26.506 INFO:tasks.rados.rados.0.smithi090.stdout:3: oid 3 updating version 2 to 3 2023-01-15T14:28:26.506 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2023-01-15T14:28:26.506 INFO:tasks.rados.rados.0.smithi090.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2023-01-15T14:28:26.506 INFO:tasks.rados.rados.0.smithi090.stdout:3: done (15 left) 2023-01-15T14:28:26.507 INFO:tasks.rados.rados.0.smithi090.stdout:17: write initial oid 17 2023-01-15T14:28:26.507 INFO:tasks.rados.rados.0.smithi090.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2023-01-15T14:28:26.507 INFO:tasks.rados.rados.0.smithi090.stdout:17: writing smithi09025264-17 from 668912 to 1305602 tid 1 2023-01-15T14:28:26.507 INFO:tasks.rados.rados.0.smithi090.stdout:17: writing smithi09025264-17 from 1903006 to 2268912 tid 2 2023-01-15T14:28:26.507 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.508 INFO:tasks.rados.rados.0.smithi090.stdout:12: finishing write tid 1 to smithi09025264-12 2023-01-15T14:28:26.508 INFO:tasks.rados.rados.0.smithi090.stdout:4: finishing write tid 2 to smithi09025264-4 2023-01-15T14:28:26.508 INFO:tasks.rados.rados.0.smithi090.stdout:4: finishing write tid 3 to smithi09025264-4 2023-01-15T14:28:26.509 INFO:tasks.rados.rados.0.smithi090.stdout:4: finishing write tid 4 to smithi09025264-4 2023-01-15T14:28:26.509 INFO:tasks.rados.rados.0.smithi090.stdout:4: oid 4 updating version 0 to 2 2023-01-15T14:28:26.509 INFO:tasks.rados.rados.0.smithi090.stdout:4: oid 4 version 2 is already newer than 1 2023-01-15T14:28:26.509 INFO:tasks.rados.rados.0.smithi090.stdout:4: oid 4 updating version 2 to 3 2023-01-15T14:28:26.509 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2023-01-15T14:28:26.509 INFO:tasks.rados.rados.0.smithi090.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2023-01-15T14:28:26.510 INFO:tasks.rados.rados.0.smithi090.stdout:4: done (15 left) 2023-01-15T14:28:26.510 INFO:tasks.rados.rados.0.smithi090.stdout:18: write initial oid 18 2023-01-15T14:28:26.510 INFO:tasks.rados.rados.0.smithi090.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2023-01-15T14:28:26.510 INFO:tasks.rados.rados.0.smithi090.stdout:18: writing smithi09025264-18 from 685719 to 1397658 tid 1 2023-01-15T14:28:26.510 INFO:tasks.rados.rados.0.smithi090.stdout:18: writing smithi09025264-18 from 1845135 to 2285719 tid 2 2023-01-15T14:28:26.510 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.510 INFO:tasks.rados.rados.0.smithi090.stdout:5: finishing write tid 1 to smithi09025264-5 2023-01-15T14:28:26.510 INFO:tasks.rados.rados.0.smithi090.stdout:8: finishing write tid 2 to smithi09025264-8 2023-01-15T14:28:26.510 INFO:tasks.rados.rados.0.smithi090.stdout:6: finishing write tid 1 to smithi09025264-6 2023-01-15T14:28:26.510 INFO:tasks.rados.rados.0.smithi090.stdout:2: finishing write tid 2 to smithi09025264-2 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:2: finishing write tid 3 to smithi09025264-2 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:2: finishing write tid 4 to smithi09025264-2 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:2: oid 2 updating version 0 to 3 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:2: oid 2 updating version 3 to 4 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:2: oid 2 version 4 is already newer than 1 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 4 (ObjNum 1 snap 0 seq_num 1) dirty exists 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:2: done (15 left) 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:19: write initial oid 19 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:19: writing smithi09025264-19 from 702526 to 1489714 tid 1 2023-01-15T14:28:26.511 INFO:tasks.rados.rados.0.smithi090.stdout:19: writing smithi09025264-19 from 1903617 to 2302526 tid 2 2023-01-15T14:28:26.512 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.512 INFO:tasks.rados.rados.0.smithi090.stdout:1: finishing write tid 2 to smithi09025264-1 2023-01-15T14:28:26.512 INFO:tasks.rados.rados.0.smithi090.stdout:13: finishing write tid 1 to smithi09025264-13 2023-01-15T14:28:26.512 INFO:tasks.rados.rados.0.smithi090.stdout:13: finishing write tid 2 to smithi09025264-13 2023-01-15T14:28:26.512 INFO:tasks.rados.rados.0.smithi090.stdout:13: finishing write tid 3 to smithi09025264-13 2023-01-15T14:28:26.512 INFO:tasks.rados.rados.0.smithi090.stdout:13: finishing write tid 4 to smithi09025264-13 2023-01-15T14:28:26.512 INFO:tasks.rados.rados.0.smithi090.stdout:13: oid 13 updating version 0 to 4 2023-01-15T14:28:26.512 INFO:tasks.rados.rados.0.smithi090.stdout:13: oid 13 updating version 4 to 5 2023-01-15T14:28:26.513 INFO:tasks.rados.rados.0.smithi090.stdout:13: oid 13 updating version 5 to 6 2023-01-15T14:28:26.513 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2023-01-15T14:28:26.513 INFO:tasks.rados.rados.0.smithi090.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2023-01-15T14:28:26.663 INFO:tasks.rados.rados.0.smithi090.stdout:13: done (15 left) 2023-01-15T14:28:26.672 INFO:tasks.rados.rados.0.smithi090.stdout:20: write initial oid 20 2023-01-15T14:28:26.673 INFO:tasks.rados.rados.0.smithi090.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2023-01-15T14:28:26.673 INFO:tasks.rados.rados.0.smithi090.stdout:20: writing smithi09025264-20 from 719333 to 1181770 tid 1 2023-01-15T14:28:26.673 INFO:tasks.rados.rados.0.smithi090.stdout:20: writing smithi09025264-20 from 1962099 to 2319333 tid 2 2023-01-15T14:28:26.673 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.673 INFO:tasks.rados.rados.0.smithi090.stdout:16: finishing write tid 1 to smithi09025264-16 2023-01-15T14:28:26.673 INFO:tasks.rados.rados.0.smithi090.stdout:20: finishing write tid 1 to smithi09025264-20 2023-01-15T14:28:26.674 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:26.329+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:28:26.682 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:26.682 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:28:26.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.682 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:89 3.d e23/20)", 2023-01-15T14:28:26.682 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.400388+0000", 2023-01-15T14:28:26.682 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.050985764000000003, 2023-01-15T14:28:26.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.051003671, 2023-01-15T14:28:26.823 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "no flag points reached", 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.400388+0000", 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.400388+0000", 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.400387+0000", 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.400844+0000", 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045635499999999998 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.400876+0000", 2023-01-15T14:28:26.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2852999999999999e-05 2023-01-15T14:28:26.829 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:26.829 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:26.829 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:26.829 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:26.829 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2023-01-15T14:28:26.829 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 1 2023-01-15T14:28:26.829 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:26.830 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:28:26.831 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:26.673+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:28:26.839 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:26.825+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:26.839 INFO:tasks.rados.rados.0.smithi090.stdout:9: finishing write tid 2 to smithi09025264-9 2023-01-15T14:28:26.839 INFO:tasks.rados.rados.0.smithi090.stdout:7: finishing write tid 2 to smithi09025264-7 2023-01-15T14:28:26.839 INFO:tasks.rados.rados.0.smithi090.stdout:7: finishing write tid 3 to smithi09025264-7 2023-01-15T14:28:26.839 INFO:tasks.rados.rados.0.smithi090.stdout:7: finishing write tid 4 to smithi09025264-7 2023-01-15T14:28:26.839 INFO:tasks.rados.rados.0.smithi090.stdout:7: oid 7 updating version 0 to 1 2023-01-15T14:28:26.840 INFO:tasks.rados.rados.0.smithi090.stdout:7: oid 7 updating version 1 to 2 2023-01-15T14:28:26.840 INFO:tasks.rados.rados.0.smithi090.stdout:7: oid 7 updating version 2 to 3 2023-01-15T14:28:26.840 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2023-01-15T14:28:26.840 INFO:tasks.rados.rados.0.smithi090.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2023-01-15T14:28:26.840 INFO:tasks.rados.rados.0.smithi090.stdout:7: done (15 left) 2023-01-15T14:28:26.840 INFO:tasks.rados.rados.0.smithi090.stdout:21: write initial oid 21 2023-01-15T14:28:26.840 INFO:tasks.rados.rados.0.smithi090.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2023-01-15T14:28:26.840 INFO:tasks.rados.rados.0.smithi090.stdout:21: writing smithi09025264-21 from 736140 to 1273826 tid 1 2023-01-15T14:28:26.840 INFO:tasks.rados.rados.0.smithi090.stdout:21: writing smithi09025264-21 from 2020581 to 2336140 tid 2 2023-01-15T14:28:26.840 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.840 INFO:tasks.rados.rados.0.smithi090.stdout:9: finishing write tid 3 to smithi09025264-9 2023-01-15T14:28:26.841 INFO:tasks.rados.rados.0.smithi090.stdout:9: finishing write tid 4 to smithi09025264-9 2023-01-15T14:28:26.841 INFO:tasks.rados.rados.0.smithi090.stdout:9: oid 9 updating version 0 to 1 2023-01-15T14:28:26.841 INFO:tasks.rados.rados.0.smithi090.stdout:9: oid 9 updating version 1 to 2 2023-01-15T14:28:26.841 INFO:tasks.rados.rados.0.smithi090.stdout:9: oid 9 updating version 2 to 3 2023-01-15T14:28:26.841 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2023-01-15T14:28:26.841 INFO:tasks.rados.rados.0.smithi090.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2023-01-15T14:28:26.841 INFO:tasks.rados.rados.0.smithi090.stdout:8: finishing write tid 3 to smithi09025264-8 2023-01-15T14:28:26.841 INFO:tasks.rados.rados.0.smithi090.stdout:8: finishing write tid 4 to smithi09025264-8 2023-01-15T14:28:26.841 INFO:tasks.rados.rados.0.smithi090.stdout:8: oid 8 updating version 0 to 2 2023-01-15T14:28:26.841 INFO:tasks.rados.rados.0.smithi090.stdout:8: oid 8 version 2 is already newer than 1 2023-01-15T14:28:26.841 INFO:tasks.rados.rados.0.smithi090.stdout:8: oid 8 updating version 2 to 3 2023-01-15T14:28:26.842 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2023-01-15T14:28:26.842 INFO:tasks.rados.rados.0.smithi090.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2023-01-15T14:28:26.842 INFO:tasks.rados.rados.0.smithi090.stdout:8: done (15 left) 2023-01-15T14:28:26.842 INFO:tasks.rados.rados.0.smithi090.stdout:9: done (14 left) 2023-01-15T14:28:26.842 INFO:tasks.rados.rados.0.smithi090.stdout:22: write initial oid 22 2023-01-15T14:28:26.842 INFO:tasks.rados.rados.0.smithi090.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2023-01-15T14:28:26.842 INFO:tasks.rados.rados.0.smithi090.stdout:22: writing smithi09025264-22 from 752947 to 1365882 tid 1 2023-01-15T14:28:26.842 INFO:tasks.rados.rados.0.smithi090.stdout:22: writing smithi09025264-22 from 1962710 to 2352947 tid 2 2023-01-15T14:28:26.842 INFO:tasks.rados.rados.0.smithi090.stdout:23: write initial oid 23 2023-01-15T14:28:26.842 INFO:tasks.rados.rados.0.smithi090.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2023-01-15T14:28:26.842 INFO:tasks.rados.rados.0.smithi090.stdout:23: writing smithi09025264-23 from 769754 to 1457938 tid 1 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout:23: writing smithi09025264-23 from 2021192 to 2369754 tid 2 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout:10: finishing write tid 2 to smithi09025264-10 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout:10: finishing write tid 3 to smithi09025264-10 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout:10: finishing write tid 4 to smithi09025264-10 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout:10: oid 10 updating version 0 to 3 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout:10: oid 10 version 3 is already newer than 1 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout:10: oid 10 updating version 3 to 4 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 4 (ObjNum 9 snap 0 seq_num 9) dirty exists 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout:10: done (15 left) 2023-01-15T14:28:26.843 INFO:tasks.rados.rados.0.smithi090.stdout:24: write initial oid 24 2023-01-15T14:28:26.844 INFO:tasks.rados.rados.0.smithi090.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2023-01-15T14:28:26.844 INFO:tasks.rados.rados.0.smithi090.stdout:24: writing smithi09025264-24 from 786561 to 1266347 tid 1 2023-01-15T14:28:26.844 INFO:tasks.rados.rados.0.smithi090.stdout:24: writing smithi09025264-24 from 1796027 to 2263691 tid 2 2023-01-15T14:28:26.844 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.844 INFO:tasks.rados.rados.0.smithi090.stdout:12: finishing write tid 2 to smithi09025264-12 2023-01-15T14:28:26.844 INFO:tasks.rados.rados.0.smithi090.stdout:12: finishing write tid 3 to smithi09025264-12 2023-01-15T14:28:26.844 INFO:tasks.rados.rados.0.smithi090.stdout:12: finishing write tid 4 to smithi09025264-12 2023-01-15T14:28:26.844 INFO:tasks.rados.rados.0.smithi090.stdout:12: oid 12 updating version 0 to 2 2023-01-15T14:28:26.844 INFO:tasks.rados.rados.0.smithi090.stdout:12: oid 12 updating version 2 to 6 2023-01-15T14:28:26.844 INFO:tasks.rados.rados.0.smithi090.stdout:12: oid 12 version 6 is already newer than 5 2023-01-15T14:28:26.846 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2023-01-15T14:28:26.846 INFO:tasks.rados.rados.0.smithi090.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2023-01-15T14:28:26.847 INFO:tasks.rados.rados.0.smithi090.stdout:16: finishing write tid 2 to smithi09025264-16 2023-01-15T14:28:26.847 INFO:tasks.rados.rados.0.smithi090.stdout:12: done (15 left) 2023-01-15T14:28:26.847 INFO:tasks.rados.rados.0.smithi090.stdout:25: write initial oid 25 2023-01-15T14:28:26.847 INFO:tasks.rados.rados.0.smithi090.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2023-01-15T14:28:26.847 INFO:tasks.rados.rados.0.smithi090.stdout:25: writing smithi09025264-25 from 403368 to 958403 tid 1 2023-01-15T14:28:26.847 INFO:tasks.rados.rados.0.smithi090.stdout:25: writing smithi09025264-25 from 1454509 to 2182184 tid 2 2023-01-15T14:28:26.847 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.847 INFO:tasks.rados.rados.0.smithi090.stdout:5: finishing write tid 2 to smithi09025264-5 2023-01-15T14:28:26.848 INFO:tasks.rados.rados.0.smithi090.stdout:14: finishing write tid 1 to smithi09025264-14 2023-01-15T14:28:26.983 INFO:tasks.rados.rados.0.smithi090.stdout:15: finishing write tid 1 to smithi09025264-15 2023-01-15T14:28:26.983 INFO:tasks.rados.rados.0.smithi090.stdout:11: finishing write tid 2 to smithi09025264-11 2023-01-15T14:28:26.983 INFO:tasks.rados.rados.0.smithi090.stdout:11: finishing write tid 3 to smithi09025264-11 2023-01-15T14:28:26.983 INFO:tasks.rados.rados.0.smithi090.stdout:11: finishing write tid 4 to smithi09025264-11 2023-01-15T14:28:26.983 INFO:tasks.rados.rados.0.smithi090.stdout:11: oid 11 updating version 0 to 2 2023-01-15T14:28:26.985 INFO:tasks.rados.rados.0.smithi090.stdout:11: oid 11 updating version 2 to 5 2023-01-15T14:28:26.985 INFO:tasks.rados.rados.0.smithi090.stdout:11: oid 11 version 5 is already newer than 4 2023-01-15T14:28:26.985 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 5 (ObjNum 10 snap 0 seq_num 10) dirty exists 2023-01-15T14:28:26.985 INFO:tasks.rados.rados.0.smithi090.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2023-01-15T14:28:26.985 INFO:tasks.rados.rados.0.smithi090.stdout:1: finishing write tid 3 to smithi09025264-1 2023-01-15T14:28:26.985 INFO:tasks.rados.rados.0.smithi090.stdout:1: finishing write tid 4 to smithi09025264-1 2023-01-15T14:28:26.985 INFO:tasks.rados.rados.0.smithi090.stdout:1: oid 1 updating version 0 to 1 2023-01-15T14:28:26.986 INFO:tasks.rados.rados.0.smithi090.stdout:1: oid 1 updating version 1 to 3 2023-01-15T14:28:26.986 INFO:tasks.rados.rados.0.smithi090.stdout:1: oid 1 updating version 3 to 6 2023-01-15T14:28:26.986 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 6 (ObjNum 0 snap 0 seq_num 0) dirty exists 2023-01-15T14:28:26.986 INFO:tasks.rados.rados.0.smithi090.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2023-01-15T14:28:26.986 INFO:tasks.rados.rados.0.smithi090.stdout:1: done (15 left) 2023-01-15T14:28:26.986 INFO:tasks.rados.rados.0.smithi090.stdout:11: done (14 left) 2023-01-15T14:28:26.986 INFO:tasks.rados.rados.0.smithi090.stdout:26: write initial oid 26 2023-01-15T14:28:26.986 INFO:tasks.rados.rados.0.smithi090.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2023-01-15T14:28:26.987 INFO:tasks.rados.rados.0.smithi090.stdout:26: writing smithi09025264-26 from 420175 to 1050459 tid 1 2023-01-15T14:28:26.987 INFO:tasks.rados.rados.0.smithi090.stdout:26: writing smithi09025264-26 from 1796638 to 2267971 tid 2 2023-01-15T14:28:26.987 INFO:tasks.rados.rados.0.smithi090.stdout:27: write initial oid 27 2023-01-15T14:28:26.987 INFO:tasks.rados.rados.0.smithi090.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2023-01-15T14:28:26.987 INFO:tasks.rados.rados.0.smithi090.stdout:27: writing smithi09025264-27 from 436982 to 1142515 tid 1 2023-01-15T14:28:26.987 INFO:tasks.rados.rados.0.smithi090.stdout:27: writing smithi09025264-27 from 1855120 to 2436982 tid 2 2023-01-15T14:28:26.987 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.987 INFO:tasks.rados.rados.0.smithi090.stdout:6: finishing write tid 2 to smithi09025264-6 2023-01-15T14:28:26.987 INFO:tasks.rados.rados.0.smithi090.stdout:6: finishing write tid 3 to smithi09025264-6 2023-01-15T14:28:26.988 INFO:tasks.rados.rados.0.smithi090.stdout:6: finishing write tid 4 to smithi09025264-6 2023-01-15T14:28:26.988 INFO:tasks.rados.rados.0.smithi090.stdout:6: oid 6 updating version 0 to 5 2023-01-15T14:28:26.988 INFO:tasks.rados.rados.0.smithi090.stdout:6: oid 6 version 5 is already newer than 2 2023-01-15T14:28:26.988 INFO:tasks.rados.rados.0.smithi090.stdout:6: oid 6 updating version 5 to 6 2023-01-15T14:28:26.988 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2023-01-15T14:28:26.988 INFO:tasks.rados.rados.0.smithi090.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2023-01-15T14:28:26.988 INFO:tasks.rados.rados.0.smithi090.stdout:6: done (15 left) 2023-01-15T14:28:26.988 INFO:tasks.rados.rados.0.smithi090.stdout:28: write initial oid 28 2023-01-15T14:28:26.989 INFO:tasks.rados.rados.0.smithi090.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2023-01-15T14:28:26.989 INFO:tasks.rados.rados.0.smithi090.stdout:28: writing smithi09025264-28 from 453789 to 1234571 tid 1 2023-01-15T14:28:26.989 INFO:tasks.rados.rados.0.smithi090.stdout:28: writing smithi09025264-28 from 1913602 to 2388604 tid 2 2023-01-15T14:28:26.989 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.989 INFO:tasks.rados.rados.0.smithi090.stdout:17: finishing write tid 1 to smithi09025264-17 2023-01-15T14:28:26.989 INFO:tasks.rados.rados.0.smithi090.stdout:16: finishing write tid 3 to smithi09025264-16 2023-01-15T14:28:26.989 INFO:tasks.rados.rados.0.smithi090.stdout:16: finishing write tid 4 to smithi09025264-16 2023-01-15T14:28:26.989 INFO:tasks.rados.rados.0.smithi090.stdout:16: oid 16 updating version 0 to 4 2023-01-15T14:28:26.990 INFO:tasks.rados.rados.0.smithi090.stdout:16: oid 16 updating version 4 to 6 2023-01-15T14:28:26.990 INFO:tasks.rados.rados.0.smithi090.stdout:16: oid 16 version 6 is already newer than 5 2023-01-15T14:28:26.990 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2023-01-15T14:28:26.990 INFO:tasks.rados.rados.0.smithi090.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2023-01-15T14:28:26.990 INFO:tasks.rados.rados.0.smithi090.stdout:16: done (15 left) 2023-01-15T14:28:26.990 INFO:tasks.rados.rados.0.smithi090.stdout:29: write initial oid 29 2023-01-15T14:28:26.990 INFO:tasks.rados.rados.0.smithi090.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2023-01-15T14:28:26.990 INFO:tasks.rados.rados.0.smithi090.stdout:29: writing smithi09025264-29 from 470596 to 926627 tid 1 2023-01-15T14:28:26.990 INFO:tasks.rados.rados.0.smithi090.stdout:29: writing smithi09025264-29 from 1572084 to 2190744 tid 2 2023-01-15T14:28:26.990 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.991 INFO:tasks.rados.rados.0.smithi090.stdout:20: finishing write tid 2 to smithi09025264-20 2023-01-15T14:28:26.991 INFO:tasks.rados.rados.0.smithi090.stdout:20: finishing write tid 3 to smithi09025264-20 2023-01-15T14:28:26.991 INFO:tasks.rados.rados.0.smithi090.stdout:20: finishing write tid 4 to smithi09025264-20 2023-01-15T14:28:26.991 INFO:tasks.rados.rados.0.smithi090.stdout:20: oid 20 updating version 0 to 2 2023-01-15T14:28:26.991 INFO:tasks.rados.rados.0.smithi090.stdout:20: oid 20 version 2 is already newer than 1 2023-01-15T14:28:26.991 INFO:tasks.rados.rados.0.smithi090.stdout:20: oid 20 updating version 2 to 3 2023-01-15T14:28:26.991 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2023-01-15T14:28:26.991 INFO:tasks.rados.rados.0.smithi090.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2023-01-15T14:28:26.991 INFO:tasks.rados.rados.0.smithi090.stdout:20: done (15 left) 2023-01-15T14:28:26.991 INFO:tasks.rados.rados.0.smithi090.stdout:30: write initial oid 30 2023-01-15T14:28:26.992 INFO:tasks.rados.rados.0.smithi090.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2023-01-15T14:28:26.992 INFO:tasks.rados.rados.0.smithi090.stdout:30: writing smithi09025264-30 from 487403 to 1018683 tid 1 2023-01-15T14:28:26.992 INFO:tasks.rados.rados.0.smithi090.stdout:30: writing smithi09025264-30 from 1514213 to 1992884 tid 2 2023-01-15T14:28:26.992 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.992 INFO:tasks.rados.rados.0.smithi090.stdout:14: finishing write tid 2 to smithi09025264-14 2023-01-15T14:28:26.992 INFO:tasks.rados.rados.0.smithi090.stdout:15: finishing write tid 2 to smithi09025264-15 2023-01-15T14:28:26.992 INFO:tasks.rados.rados.0.smithi090.stdout:21: finishing write tid 1 to smithi09025264-21 2023-01-15T14:28:26.992 INFO:tasks.rados.rados.0.smithi090.stdout:23: finishing write tid 1 to smithi09025264-23 2023-01-15T14:28:26.992 INFO:tasks.rados.rados.0.smithi090.stdout:17: finishing write tid 2 to smithi09025264-17 2023-01-15T14:28:26.993 INFO:tasks.rados.rados.0.smithi090.stdout:17: finishing write tid 3 to smithi09025264-17 2023-01-15T14:28:26.993 INFO:tasks.rados.rados.0.smithi090.stdout:5: finishing write tid 3 to smithi09025264-5 2023-01-15T14:28:26.993 INFO:tasks.rados.rados.0.smithi090.stdout:5: finishing write tid 4 to smithi09025264-5 2023-01-15T14:28:26.993 INFO:tasks.rados.rados.0.smithi090.stdout:5: oid 5 updating version 0 to 1 2023-01-15T14:28:26.993 INFO:tasks.rados.rados.0.smithi090.stdout:5: oid 5 updating version 1 to 2 2023-01-15T14:28:26.993 INFO:tasks.rados.rados.0.smithi090.stdout:5: oid 5 updating version 2 to 3 2023-01-15T14:28:26.993 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2023-01-15T14:28:26.993 INFO:tasks.rados.rados.0.smithi090.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2023-01-15T14:28:26.993 INFO:tasks.rados.rados.0.smithi090.stdout:29: finishing write tid 1 to smithi09025264-29 2023-01-15T14:28:26.993 INFO:tasks.rados.rados.0.smithi090.stdout:5: done (15 left) 2023-01-15T14:28:26.994 INFO:tasks.rados.rados.0.smithi090.stdout:31: write initial oid 31 2023-01-15T14:28:26.994 INFO:tasks.rados.rados.0.smithi090.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2023-01-15T14:28:26.994 INFO:tasks.rados.rados.0.smithi090.stdout:31: writing smithi09025264-31 from 504210 to 1110739 tid 1 2023-01-15T14:28:26.994 INFO:tasks.rados.rados.0.smithi090.stdout:31: writing smithi09025264-31 from 1572695 to 2195024 tid 2 2023-01-15T14:28:26.994 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.994 INFO:tasks.rados.rados.0.smithi090.stdout:18: finishing write tid 1 to smithi09025264-18 2023-01-15T14:28:26.994 INFO:tasks.rados.rados.0.smithi090.stdout:15: finishing write tid 3 to smithi09025264-15 2023-01-15T14:28:26.994 INFO:tasks.rados.rados.0.smithi090.stdout:15: finishing write tid 4 to smithi09025264-15 2023-01-15T14:28:26.995 INFO:tasks.rados.rados.0.smithi090.stdout:15: oid 15 updating version 0 to 5 2023-01-15T14:28:26.995 INFO:tasks.rados.rados.0.smithi090.stdout:15: oid 15 updating version 5 to 8 2023-01-15T14:28:26.995 INFO:tasks.rados.rados.0.smithi090.stdout:15: oid 15 version 8 is already newer than 7 2023-01-15T14:28:26.995 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 8 (ObjNum 14 snap 0 seq_num 14) dirty exists 2023-01-15T14:28:26.995 INFO:tasks.rados.rados.0.smithi090.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2023-01-15T14:28:26.995 INFO:tasks.rados.rados.0.smithi090.stdout:14: finishing write tid 3 to smithi09025264-14 2023-01-15T14:28:26.995 INFO:tasks.rados.rados.0.smithi090.stdout:14: finishing write tid 4 to smithi09025264-14 2023-01-15T14:28:26.995 INFO:tasks.rados.rados.0.smithi090.stdout:14: oid 14 updating version 0 to 4 2023-01-15T14:28:26.996 INFO:tasks.rados.rados.0.smithi090.stdout:14: oid 14 updating version 4 to 6 2023-01-15T14:28:26.996 INFO:tasks.rados.rados.0.smithi090.stdout:14: oid 14 updating version 6 to 9 2023-01-15T14:28:26.996 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 9 (ObjNum 13 snap 0 seq_num 13) dirty exists 2023-01-15T14:28:26.996 INFO:tasks.rados.rados.0.smithi090.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2023-01-15T14:28:26.996 INFO:tasks.rados.rados.0.smithi090.stdout:14: done (15 left) 2023-01-15T14:28:26.996 INFO:tasks.rados.rados.0.smithi090.stdout:15: done (14 left) 2023-01-15T14:28:26.996 INFO:tasks.rados.rados.0.smithi090.stdout:32: write initial oid 32 2023-01-15T14:28:26.996 INFO:tasks.rados.rados.0.smithi090.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2023-01-15T14:28:26.996 INFO:tasks.rados.rados.0.smithi090.stdout:32: writing smithi09025264-32 from 521017 to 1319148 tid 1 2023-01-15T14:28:26.997 INFO:tasks.rados.rados.0.smithi090.stdout:32: writing smithi09025264-32 from 1747530 to 2229870 tid 2 2023-01-15T14:28:26.997 INFO:tasks.rados.rados.0.smithi090.stdout:33: write initial oid 33 2023-01-15T14:28:26.997 INFO:tasks.rados.rados.0.smithi090.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2023-01-15T14:28:26.997 INFO:tasks.rados.rados.0.smithi090.stdout:33: writing smithi09025264-33 from 537824 to 1011204 tid 1 2023-01-15T14:28:26.997 INFO:tasks.rados.rados.0.smithi090.stdout:33: writing smithi09025264-33 from 1806012 to 2432010 tid 2 2023-01-15T14:28:26.997 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:26.997 INFO:tasks.rados.rados.0.smithi090.stdout:17: finishing write tid 4 to smithi09025264-17 2023-01-15T14:28:26.997 INFO:tasks.rados.rados.0.smithi090.stdout:17: oid 17 updating version 0 to 8 2023-01-15T14:28:26.997 INFO:tasks.rados.rados.0.smithi090.stdout:17: oid 17 updating version 8 to 9 2023-01-15T14:28:26.997 INFO:tasks.rados.rados.0.smithi090.stdout:17: oid 17 version 9 is already newer than 7 2023-01-15T14:28:26.998 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 9 (ObjNum 16 snap 0 seq_num 16) dirty exists 2023-01-15T14:28:26.998 INFO:tasks.rados.rados.0.smithi090.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2023-01-15T14:28:26.998 INFO:tasks.rados.rados.0.smithi090.stdout:17: done (15 left) 2023-01-15T14:28:26.998 INFO:tasks.rados.rados.0.smithi090.stdout:34: write initial oid 34 2023-01-15T14:28:26.998 INFO:tasks.rados.rados.0.smithi090.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2023-01-15T14:28:27.002 INFO:tasks.rados.rados.0.smithi090.stdout:34: writing smithi09025264-34 from 554631 to 1103260 tid 1 2023-01-15T14:28:27.002 INFO:tasks.rados.rados.0.smithi090.stdout:34: writing smithi09025264-34 from 1748141 to 2234150 tid 2 2023-01-15T14:28:27.002 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:27.002 INFO:tasks.rados.rados.0.smithi090.stdout:34: finishing write tid 1 to smithi09025264-34 2023-01-15T14:28:27.002 INFO:tasks.rados.rados.0.smithi090.stdout:19: finishing write tid 1 to smithi09025264-19 2023-01-15T14:28:27.003 INFO:tasks.rados.rados.0.smithi090.stdout:21: finishing write tid 2 to smithi09025264-21 2023-01-15T14:28:27.003 INFO:tasks.rados.rados.0.smithi090.stdout:21: finishing write tid 3 to smithi09025264-21 2023-01-15T14:28:27.003 INFO:tasks.rados.rados.0.smithi090.stdout:21: finishing write tid 4 to smithi09025264-21 2023-01-15T14:28:27.003 INFO:tasks.rados.rados.0.smithi090.stdout:21: oid 21 updating version 0 to 2 2023-01-15T14:28:27.003 INFO:tasks.rados.rados.0.smithi090.stdout:21: oid 21 version 2 is already newer than 1 2023-01-15T14:28:27.003 INFO:tasks.rados.rados.0.smithi090.stdout:21: oid 21 updating version 2 to 3 2023-01-15T14:28:27.003 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2023-01-15T14:28:27.003 INFO:tasks.rados.rados.0.smithi090.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2023-01-15T14:28:27.003 INFO:tasks.rados.rados.0.smithi090.stdout:22: finishing write tid 1 to smithi09025264-22 2023-01-15T14:28:27.003 INFO:tasks.rados.rados.0.smithi090.stdout:21: done (15 left) 2023-01-15T14:28:27.004 INFO:tasks.rados.rados.0.smithi090.stdout:35: write initial oid 35 2023-01-15T14:28:27.004 INFO:tasks.rados.rados.0.smithi090.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2023-01-15T14:28:27.004 INFO:tasks.rados.rados.0.smithi090.stdout:35: writing smithi09025264-35 from 571438 to 1195316 tid 1 2023-01-15T14:28:27.004 INFO:tasks.rados.rados.0.smithi090.stdout:35: writing smithi09025264-35 from 1806623 to 2436290 tid 2 2023-01-15T14:28:27.004 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:27.004 INFO:tasks.rados.rados.0.smithi090.stdout:26: finishing write tid 1 to smithi09025264-26 2023-01-15T14:28:27.004 INFO:tasks.rados.rados.0.smithi090.stdout:28: finishing write tid 1 to smithi09025264-28 2023-01-15T14:28:27.004 INFO:tasks.rados.rados.0.smithi090.stdout:27: finishing write tid 1 to smithi09025264-27 2023-01-15T14:28:27.004 INFO:tasks.rados.rados.0.smithi090.stdout:24: finishing write tid 1 to smithi09025264-24 2023-01-15T14:28:27.007 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:27.146 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:28:27.146 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:28:27.146 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:28:27.146 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:27.147 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:28:27.147 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:27.001+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:27.148 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:28:27.149 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:28:27.149 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-01-15T14:28:27.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-01-15T14:28:27.158 INFO:tasks.rados.rados.0.smithi090.stdout:25: finishing write tid 1 to smithi09025264-25 2023-01-15T14:28:27.158 INFO:tasks.rados.rados.0.smithi090.stdout:34: finishing write tid 2 to smithi09025264-34 2023-01-15T14:28:27.158 INFO:tasks.rados.rados.0.smithi090.stdout:34: finishing write tid 3 to smithi09025264-34 2023-01-15T14:28:27.158 INFO:tasks.rados.rados.0.smithi090.stdout:34: finishing write tid 4 to smithi09025264-34 2023-01-15T14:28:27.158 INFO:tasks.rados.rados.0.smithi090.stdout:34: oid 34 updating version 0 to 1 2023-01-15T14:28:27.158 INFO:tasks.rados.rados.0.smithi090.stdout:34: oid 34 updating version 1 to 3 2023-01-15T14:28:27.159 INFO:tasks.rados.rados.0.smithi090.stdout:34: oid 34 version 3 is already newer than 2 2023-01-15T14:28:27.159 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2023-01-15T14:28:27.159 INFO:tasks.rados.rados.0.smithi090.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2023-01-15T14:28:27.159 INFO:tasks.rados.rados.0.smithi090.stdout:34: done (15 left) 2023-01-15T14:28:27.159 INFO:tasks.rados.rados.0.smithi090.stdout:36: write initial oid 36 2023-01-15T14:28:27.159 INFO:tasks.rados.rados.0.smithi090.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2023-01-15T14:28:27.159 INFO:tasks.rados.rados.0.smithi090.stdout:36: writing smithi09025264-36 from 588245 to 1287372 tid 1 2023-01-15T14:28:27.159 INFO:tasks.rados.rados.0.smithi090.stdout:36: writing smithi09025264-36 from 1865105 to 2354783 tid 2 2023-01-15T14:28:27.159 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:27.159 INFO:tasks.rados.rados.0.smithi090.stdout:29: finishing write tid 2 to smithi09025264-29 2023-01-15T14:28:27.160 INFO:tasks.rados.rados.0.smithi090.stdout:29: finishing write tid 3 to smithi09025264-29 2023-01-15T14:28:27.160 INFO:tasks.rados.rados.0.smithi090.stdout:29: finishing write tid 4 to smithi09025264-29 2023-01-15T14:28:27.160 INFO:tasks.rados.rados.0.smithi090.stdout:29: oid 29 updating version 0 to 7 2023-01-15T14:28:27.160 INFO:tasks.rados.rados.0.smithi090.stdout:29: oid 29 updating version 7 to 9 2023-01-15T14:28:27.160 INFO:tasks.rados.rados.0.smithi090.stdout:29: oid 29 version 9 is already newer than 8 2023-01-15T14:28:27.160 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2023-01-15T14:28:27.160 INFO:tasks.rados.rados.0.smithi090.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2023-01-15T14:28:27.160 INFO:tasks.rados.rados.0.smithi090.stdout:29: done (15 left) 2023-01-15T14:28:27.160 INFO:tasks.rados.rados.0.smithi090.stdout:37: write initial oid 37 2023-01-15T14:28:27.160 INFO:tasks.rados.rados.0.smithi090.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2023-01-15T14:28:27.160 INFO:tasks.rados.rados.0.smithi090.stdout:37: writing smithi09025264-37 from 605052 to 1379428 tid 1 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:37: writing smithi09025264-37 from 1923587 to 2556923 tid 2 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:30: finishing write tid 1 to smithi09025264-30 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:35: finishing write tid 1 to smithi09025264-35 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:23: finishing write tid 2 to smithi09025264-23 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:27: finishing write tid 2 to smithi09025264-27 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:25: finishing write tid 2 to smithi09025264-25 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:24: finishing write tid 2 to smithi09025264-24 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:24: finishing write tid 3 to smithi09025264-24 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:24: finishing write tid 4 to smithi09025264-24 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:24: oid 24 updating version 0 to 12 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:24: oid 24 version 12 is already newer than 10 2023-01-15T14:28:27.161 INFO:tasks.rados.rados.0.smithi090.stdout:24: oid 24 version 12 is already newer than 11 2023-01-15T14:28:27.162 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 12 (ObjNum 23 snap 0 seq_num 23) dirty exists 2023-01-15T14:28:27.162 INFO:tasks.rados.rados.0.smithi090.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2023-01-15T14:28:27.302 INFO:tasks.rados.rados.0.smithi090.stdout:24: done (15 left) 2023-01-15T14:28:27.303 INFO:tasks.rados.rados.0.smithi090.stdout:38: write initial oid 38 2023-01-15T14:28:27.303 INFO:tasks.rados.rados.0.smithi090.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2023-01-15T14:28:27.304 INFO:tasks.rados.rados.0.smithi090.stdout:38: writing smithi09025264-38 from 621859 to 1071484 tid 1 2023-01-15T14:28:27.305 INFO:tasks.rados.rados.0.smithi090.stdout:38: writing smithi09025264-38 from 1865716 to 2621859 tid 2 2023-01-15T14:28:27.305 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:27.305 INFO:tasks.rados.rados.0.smithi090.stdout:26: finishing write tid 2 to smithi09025264-26 2023-01-15T14:28:27.305 INFO:tasks.rados.rados.0.smithi090.stdout:26: finishing write tid 3 to smithi09025264-26 2023-01-15T14:28:27.306 INFO:tasks.rados.rados.0.smithi090.stdout:26: finishing write tid 4 to smithi09025264-26 2023-01-15T14:28:27.306 INFO:tasks.rados.rados.0.smithi090.stdout:26: oid 26 updating version 0 to 10 2023-01-15T14:28:27.306 INFO:tasks.rados.rados.0.smithi090.stdout:26: oid 26 version 10 is already newer than 7 2023-01-15T14:28:27.307 INFO:tasks.rados.rados.0.smithi090.stdout:26: oid 26 version 10 is already newer than 9 2023-01-15T14:28:27.307 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 10 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-01-15T14:28:27.307 INFO:tasks.rados.rados.0.smithi090.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2023-01-15T14:28:27.311 INFO:tasks.rados.rados.0.smithi090.stdout:26: done (15 left) 2023-01-15T14:28:27.311 INFO:tasks.rados.rados.0.smithi090.stdout:39: write initial oid 39 2023-01-15T14:28:27.311 INFO:tasks.rados.rados.0.smithi090.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2023-01-15T14:28:27.311 INFO:tasks.rados.rados.0.smithi090.stdout:39: writing smithi09025264-39 from 638666 to 1163540 tid 1 2023-01-15T14:28:27.311 INFO:tasks.rados.rados.0.smithi090.stdout:39: writing smithi09025264-39 from 1924198 to 2561203 tid 2 2023-01-15T14:28:27.311 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:27.323 INFO:tasks.rados.rados.0.smithi090.stdout:28: finishing write tid 2 to smithi09025264-28 2023-01-15T14:28:27.323 INFO:tasks.rados.rados.0.smithi090.stdout:30: finishing write tid 2 to smithi09025264-30 2023-01-15T14:28:27.323 INFO:tasks.rados.rados.0.smithi090.stdout:30: finishing write tid 3 to smithi09025264-30 2023-01-15T14:28:27.323 INFO:tasks.rados.rados.0.smithi090.stdout:30: finishing write tid 4 to smithi09025264-30 2023-01-15T14:28:27.323 INFO:tasks.rados.rados.0.smithi090.stdout:30: oid 30 updating version 0 to 2 2023-01-15T14:28:27.323 INFO:tasks.rados.rados.0.smithi090.stdout:30: oid 30 version 2 is already newer than 1 2023-01-15T14:28:27.323 INFO:tasks.rados.rados.0.smithi090.stdout:30: oid 30 updating version 2 to 3 2023-01-15T14:28:27.324 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 3 (ObjNum 29 snap 0 seq_num 29) dirty exists 2023-01-15T14:28:27.324 INFO:tasks.rados.rados.0.smithi090.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2023-01-15T14:28:27.324 INFO:tasks.rados.rados.0.smithi090.stdout:22: finishing write tid 2 to smithi09025264-22 2023-01-15T14:28:27.324 INFO:tasks.rados.rados.0.smithi090.stdout:33: finishing write tid 1 to smithi09025264-33 2023-01-15T14:28:27.324 INFO:tasks.rados.rados.0.smithi090.stdout:30: done (15 left) 2023-01-15T14:28:27.324 INFO:tasks.rados.rados.0.smithi090.stdout:40: write initial oid 40 2023-01-15T14:28:27.324 INFO:tasks.rados.rados.0.smithi090.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2023-01-15T14:28:27.324 INFO:tasks.rados.rados.0.smithi090.stdout:40: writing smithi09025264-40 from 655473 to 1371949 tid 1 2023-01-15T14:28:27.324 INFO:tasks.rados.rados.0.smithi090.stdout:40: writing smithi09025264-40 from 2099033 to 2655473 tid 2 2023-01-15T14:28:27.324 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:27.325 INFO:tasks.rados.rados.0.smithi090.stdout:25: finishing write tid 3 to smithi09025264-25 2023-01-15T14:28:27.326 INFO:tasks.rados.rados.0.smithi090.stdout:28: finishing write tid 3 to smithi09025264-28 2023-01-15T14:28:27.326 INFO:tasks.rados.rados.0.smithi090.stdout:32: finishing write tid 1 to smithi09025264-32 2023-01-15T14:28:27.326 INFO:tasks.rados.rados.0.smithi090.stdout:28: finishing write tid 4 to smithi09025264-28 2023-01-15T14:28:27.327 INFO:tasks.rados.rados.0.smithi090.stdout:28: oid 28 updating version 0 to 11 2023-01-15T14:28:27.327 INFO:tasks.rados.rados.0.smithi090.stdout:28: oid 28 version 11 is already newer than 8 2023-01-15T14:28:27.327 INFO:tasks.rados.rados.0.smithi090.stdout:28: oid 28 updating version 11 to 12 2023-01-15T14:28:27.327 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 12 (ObjNum 27 snap 0 seq_num 27) dirty exists 2023-01-15T14:28:27.327 INFO:tasks.rados.rados.0.smithi090.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2023-01-15T14:28:27.327 INFO:tasks.rados.rados.0.smithi090.stdout:28: done (15 left) 2023-01-15T14:28:27.327 INFO:tasks.rados.rados.0.smithi090.stdout:41: write initial oid 41 2023-01-15T14:28:27.327 INFO:tasks.rados.rados.0.smithi090.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2023-01-15T14:28:27.327 INFO:tasks.rados.rados.0.smithi090.stdout:41: writing smithi09025264-41 from 672280 to 1464005 tid 1 2023-01-15T14:28:27.327 INFO:tasks.rados.rados.0.smithi090.stdout:41: writing smithi09025264-41 from 2157515 to 2672280 tid 2 2023-01-15T14:28:27.327 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:27.328 INFO:tasks.rados.rados.0.smithi090.stdout:37: finishing write tid 1 to smithi09025264-37 2023-01-15T14:28:27.328 INFO:tasks.rados.rados.0.smithi090.stdout:18: finishing write tid 2 to smithi09025264-18 2023-01-15T14:28:27.328 INFO:tasks.rados.rados.0.smithi090.stdout:22: finishing write tid 3 to smithi09025264-22 2023-01-15T14:28:27.328 INFO:tasks.rados.rados.0.smithi090.stdout:18: finishing write tid 3 to smithi09025264-18 2023-01-15T14:28:27.461 INFO:tasks.rados.rados.0.smithi090.stdout:35: finishing write tid 2 to smithi09025264-35 2023-01-15T14:28:27.461 INFO:tasks.rados.rados.0.smithi090.stdout:18: finishing write tid 4 to smithi09025264-18 2023-01-15T14:28:27.461 INFO:tasks.rados.rados.0.smithi090.stdout:18: oid 18 updating version 0 to 7 2023-01-15T14:28:27.461 INFO:tasks.rados.rados.0.smithi090.stdout:18: oid 18 version 7 is already newer than 6 2023-01-15T14:28:27.461 INFO:tasks.rados.rados.0.smithi090.stdout:18: oid 18 version 7 is already newer than 4 2023-01-15T14:28:27.461 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 7 (ObjNum 17 snap 0 seq_num 17) dirty exists 2023-01-15T14:28:27.461 INFO:tasks.rados.rados.0.smithi090.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:22: finishing write tid 4 to smithi09025264-22 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:22: oid 22 updating version 0 to 4 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:22: oid 22 updating version 4 to 5 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:22: oid 22 updating version 5 to 6 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:19: finishing write tid 2 to smithi09025264-19 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:31: finishing write tid 1 to smithi09025264-31 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:33: finishing write tid 2 to smithi09025264-33 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:36: finishing write tid 1 to smithi09025264-36 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:18: done (15 left) 2023-01-15T14:28:27.462 INFO:tasks.rados.rados.0.smithi090.stdout:22: done (14 left) 2023-01-15T14:28:27.463 INFO:tasks.rados.rados.0.smithi090.stdout:42: write initial oid 42 2023-01-15T14:28:27.463 INFO:tasks.rados.rados.0.smithi090.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2023-01-15T14:28:27.463 INFO:tasks.rados.rados.0.smithi090.stdout:42: writing smithi09025264-42 from 689087 to 1156061 tid 1 2023-01-15T14:28:27.463 INFO:tasks.rados.rados.0.smithi090.stdout:42: writing smithi09025264-42 from 1699644 to 2483976 tid 2 2023-01-15T14:28:27.463 INFO:tasks.rados.rados.0.smithi090.stdout:43: write initial oid 43 2023-01-15T14:28:27.463 INFO:tasks.rados.rados.0.smithi090.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2023-01-15T14:28:27.463 INFO:tasks.rados.rados.0.smithi090.stdout:43: writing smithi09025264-43 from 705894 to 1248117 tid 1 2023-01-15T14:28:27.463 INFO:tasks.rados.rados.0.smithi090.stdout:43: writing smithi09025264-43 from 1758126 to 2402469 tid 2 2023-01-15T14:28:27.463 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:27.463 INFO:tasks.rados.rados.0.smithi090.stdout:25: finishing write tid 4 to smithi09025264-25 2023-01-15T14:28:27.464 INFO:tasks.rados.rados.0.smithi090.stdout:25: oid 25 updating version 0 to 9 2023-01-15T14:28:27.464 INFO:tasks.rados.rados.0.smithi090.stdout:25: oid 25 version 9 is already newer than 7 2023-01-15T14:28:27.464 INFO:tasks.rados.rados.0.smithi090.stdout:25: oid 25 version 9 is already newer than 8 2023-01-15T14:28:27.464 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2023-01-15T14:28:27.464 INFO:tasks.rados.rados.0.smithi090.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2023-01-15T14:28:27.464 INFO:tasks.rados.rados.0.smithi090.stdout:25: done (15 left) 2023-01-15T14:28:27.464 INFO:tasks.rados.rados.0.smithi090.stdout:44: write initial oid 44 2023-01-15T14:28:27.464 INFO:tasks.rados.rados.0.smithi090.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2023-01-15T14:28:27.465 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:27.145+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:28:27.470 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:27.470 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:28:27.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:28:27.470 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:28:27.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.470 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4202.0:67 1.0 e17/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'41, mlcod=17'39)", 2023-01-15T14:28:27.471 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:12.203292+0000", 2023-01-15T14:28:27.476 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 15.064604115, 2023-01-15T14:28:27.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20109781299999999, 2023-01-15T14:28:27.476 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:27.476 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:27.476 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:27.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.203292+0000", 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.203292+0000", 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:27.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.203289+0000", 2023-01-15T14:28:27.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:28:27.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:27.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.203307+0000", 2023-01-15T14:28:27.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8355999999999998e-05 2023-01-15T14:28:27.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:27.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.203342+0000", 2023-01-15T14:28:27.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4913999999999998e-05 2023-01-15T14:28:27.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:27.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303496+0000", 2023-01-15T14:28:27.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100154057 2023-01-15T14:28:27.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:27.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303646+0000", 2023-01-15T14:28:27.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014934099999999999 2023-01-15T14:28:27.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:27.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303692+0000", 2023-01-15T14:28:27.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.5924000000000003e-05 2023-01-15T14:28:27.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.659 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.660 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:27.660 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303849+0000", 2023-01-15T14:28:27.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015698099999999999 2023-01-15T14:28:27.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:27.661 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303875+0000", 2023-01-15T14:28:27.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.62e-05 2023-01-15T14:28:27.661 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:27.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.304001+0000", 2023-01-15T14:28:27.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012617800000000001 2023-01-15T14:28:27.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:27.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.404288+0000", 2023-01-15T14:28:27.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10028693700000001 2023-01-15T14:28:27.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:27.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.404389+0000", 2023-01-15T14:28:27.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101378 2023-01-15T14:28:27.663 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:27.664 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:27.664 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:27.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.664 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4202.0:68 1.0 e17/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'42, mlcod=17'39)", 2023-01-15T14:28:27.664 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:12.303542+0000", 2023-01-15T14:28:27.665 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.964353491000001, 2023-01-15T14:28:27.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202074581, 2023-01-15T14:28:27.665 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:27.665 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:27.665 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:27.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:27.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303542+0000", 2023-01-15T14:28:27.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:27.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:27.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303542+0000", 2023-01-15T14:28:27.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:27.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:27.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303541+0000", 2023-01-15T14:28:27.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:27.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:27.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303553+0000", 2023-01-15T14:28:27.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1387e-05 2023-01-15T14:28:27.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:27.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303568+0000", 2023-01-15T14:28:27.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5255999999999999e-05 2023-01-15T14:28:27.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:27.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303577+0000", 2023-01-15T14:28:27.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0969999999999995e-06 2023-01-15T14:28:27.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:27.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.404042+0000", 2023-01-15T14:28:27.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10046513899999999 2023-01-15T14:28:27.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:27.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.404073+0000", 2023-01-15T14:28:27.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.0917000000000002e-05 2023-01-15T14:28:27.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:27.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.404251+0000", 2023-01-15T14:28:27.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017836499999999999 2023-01-15T14:28:27.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.864 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:27.866 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.404306+0000", 2023-01-15T14:28:27.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4620000000000002e-05 2023-01-15T14:28:27.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.866 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:27.867 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.404383+0000", 2023-01-15T14:28:27.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6944000000000001e-05 2023-01-15T14:28:27.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:27.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505519+0000", 2023-01-15T14:28:27.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101135931 2023-01-15T14:28:27.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:27.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505617+0000", 2023-01-15T14:28:27.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7948000000000005e-05 2023-01-15T14:28:27.869 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:27.869 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:27.869 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:27.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.869 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.869 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4202.0:69 1.0 e17/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'43, mlcod=17'39)", 2023-01-15T14:28:27.869 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:12.303627+0000", 2023-01-15T14:28:27.869 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.96426898, 2023-01-15T14:28:27.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20362253, 2023-01-15T14:28:27.869 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:27.869 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:27.870 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:27.870 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.870 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:27.870 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303627+0000", 2023-01-15T14:28:27.870 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:27.870 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.870 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.870 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:27.870 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303627+0000", 2023-01-15T14:28:27.870 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:27.870 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:27.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303626+0000", 2023-01-15T14:28:27.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:27.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:27.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303672+0000", 2023-01-15T14:28:27.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.5287000000000002e-05 2023-01-15T14:28:27.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:27.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303697+0000", 2023-01-15T14:28:27.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5677000000000002e-05 2023-01-15T14:28:27.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:27.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.303712+0000", 2023-01-15T14:28:27.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4543e-05 2023-01-15T14:28:27.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:27.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.504566+0000", 2023-01-15T14:28:27.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200853907 2023-01-15T14:28:27.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:27.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.504612+0000", 2023-01-15T14:28:27.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.6183999999999997e-05 2023-01-15T14:28:27.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:27.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505240+0000", 2023-01-15T14:28:27.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062824500000000004 2023-01-15T14:28:27.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:27.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505301+0000", 2023-01-15T14:28:27.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1030999999999999e-05 2023-01-15T14:28:27.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:27.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505411+0000", 2023-01-15T14:28:27.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110277 2023-01-15T14:28:27.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:27.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505627+0000", 2023-01-15T14:28:27.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021529200000000001 2023-01-15T14:28:27.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:27.877 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.507249+0000", 2023-01-15T14:28:27.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0016225370000000001 2023-01-15T14:28:27.877 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:27.877 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:27.877 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:27.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.877 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.877 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4202.0:70 1.0 e17/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'44, mlcod=17'39)", 2023-01-15T14:28:27.877 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:12.305756+0000", 2023-01-15T14:28:27.877 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.962139743, 2023-01-15T14:28:27.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20290486399999999, 2023-01-15T14:28:27.878 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:27.878 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:27.878 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:27.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:27.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.305756+0000", 2023-01-15T14:28:27.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:27.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:27.879 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.305756+0000", 2023-01-15T14:28:27.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:27.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.879 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.879 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:27.879 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.305754+0000", 2023-01-15T14:28:27.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:28:27.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.879 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.879 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:27.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.305773+0000", 2023-01-15T14:28:27.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9181000000000002e-05 2023-01-15T14:28:27.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:27.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.305811+0000", 2023-01-15T14:28:27.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7743000000000001e-05 2023-01-15T14:28:27.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:27.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.305829+0000", 2023-01-15T14:28:27.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8009999999999999e-05 2023-01-15T14:28:27.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:27.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505308+0000", 2023-01-15T14:28:27.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19947904 2023-01-15T14:28:27.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:27.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505335+0000", 2023-01-15T14:28:27.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7146999999999999e-05 2023-01-15T14:28:27.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:27.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505483+0000", 2023-01-15T14:28:27.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014761400000000001 2023-01-15T14:28:27.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:27.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505499+0000", 2023-01-15T14:28:27.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6217000000000001e-05 2023-01-15T14:28:27.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:27.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505544+0000", 2023-01-15T14:28:27.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4647000000000001e-05 2023-01-15T14:28:27.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:27.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.505667+0000", 2023-01-15T14:28:27.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012354099999999999 2023-01-15T14:28:27.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:27.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:12.508661+0000", 2023-01-15T14:28:27.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.002993571 2023-01-15T14:28:27.884 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:27.885 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:27.885 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:27.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:27.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.885 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4202.0:158 1.0 e19/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 19'112, mlcod=19'107)", 2023-01-15T14:28:27.885 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:20.967147+0000", 2023-01-15T14:28:27.885 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 6.3007486789999998, 2023-01-15T14:28:27.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20433929100000001, 2023-01-15T14:28:27.885 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:27.885 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:27.885 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:27.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:27.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:27.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.967147+0000", 2023-01-15T14:28:27.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.074 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.074 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:28.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.967147+0000", 2023-01-15T14:28:28.075 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.075 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:28.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.967146+0000", 2023-01-15T14:28:28.075 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:28.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.075 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:28.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.967244+0000", 2023-01-15T14:28:28.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8126000000000002e-05 2023-01-15T14:28:28.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:28.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.967260+0000", 2023-01-15T14:28:28.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5693999999999999e-05 2023-01-15T14:28:28.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:28.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.067432+0000", 2023-01-15T14:28:28.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100172448 2023-01-15T14:28:28.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:28.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.167710+0000", 2023-01-15T14:28:28.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10027805200000001 2023-01-15T14:28:28.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:28.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.167763+0000", 2023-01-15T14:28:28.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2787000000000001e-05 2023-01-15T14:28:28.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:28.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.168179+0000", 2023-01-15T14:28:28.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000415733 2023-01-15T14:28:28.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:28.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.168201+0000", 2023-01-15T14:28:28.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2073999999999999e-05 2023-01-15T14:28:28.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:28.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.168313+0000", 2023-01-15T14:28:28.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001125 2023-01-15T14:28:28.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:28.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.168708+0000", 2023-01-15T14:28:28.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039465300000000002 2023-01-15T14:28:28.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:28.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.171486+0000", 2023-01-15T14:28:28.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0027782100000000001 2023-01-15T14:28:28.085 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.085 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:28.085 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.085 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:10 3.a 3:52c3e133:::smithi09025264-3:head [write 1600611~433003] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:28:28.085 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.799275+0000", 2023-01-15T14:28:28.085 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 1.468620705, 2023-01-15T14:28:28.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30320612600000002, 2023-01-15T14:28:28.273 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:28.273 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:28.273 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:28.273 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:28.273 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:28.273 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 10 2023-01-15T14:28:28.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.274 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:28.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:28.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799275+0000", 2023-01-15T14:28:28.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:28.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799275+0000", 2023-01-15T14:28:28.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:28.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799272+0000", 2023-01-15T14:28:28.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:28:28.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799654+0000", 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038120299999999999 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799677+0000", 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3010000000000002e-05 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:28.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799688+0000", 2023-01-15T14:28:28.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0967e-05 2023-01-15T14:28:28.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:28.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799737+0000", 2023-01-15T14:28:28.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9209000000000003e-05 2023-01-15T14:28:28.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:28.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799818+0000", 2023-01-15T14:28:28.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1050000000000005e-05 2023-01-15T14:28:28.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:28:28.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799942+0000", 2023-01-15T14:28:28.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012419800000000001 2023-01-15T14:28:28.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:28.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.800157+0000", 2023-01-15T14:28:28.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021442400000000001 2023-01-15T14:28:28.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:28.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.800173+0000", 2023-01-15T14:28:28.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6211000000000001e-05 2023-01-15T14:28:28.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:28.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.800372+0000", 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000199163 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.800913+0000", 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054149000000000005 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.898605+0000", 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097691365000000002 2023-01-15T14:28:28.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.282 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:28.282 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102351+0000", 2023-01-15T14:28:28.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.2037458 2023-01-15T14:28:28.282 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.282 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:28.282 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102424+0000", 2023-01-15T14:28:28.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3282999999999996e-05 2023-01-15T14:28:28.282 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:28.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102481+0000", 2023-01-15T14:28:28.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7259999999999997e-05 2023-01-15T14:28:28.283 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.283 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:28.283 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.284 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:11 3.a 3:52c3e133:::smithi09025264-3:head [setxattr _header (55) in=7b,truncate 2033614] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:28:28.284 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.799707+0000", 2023-01-15T14:28:28.284 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 1.468188286, 2023-01-15T14:28:28.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.303249402, 2023-01-15T14:28:28.284 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:28.284 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:28.284 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:28.284 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:28.284 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:28.284 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 11 2023-01-15T14:28:28.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.285 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:28.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:28.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799707+0000", 2023-01-15T14:28:28.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:28.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799707+0000", 2023-01-15T14:28:28.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799707+0000", 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799712+0000", 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.383e-06 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:28.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799721+0000", 2023-01-15T14:28:28.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.884e-06 2023-01-15T14:28:28.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:28.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799729+0000", 2023-01-15T14:28:28.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2700000000000004e-06 2023-01-15T14:28:28.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:28.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.800221+0000", 2023-01-15T14:28:28.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049171299999999996 2023-01-15T14:28:28.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:28.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.800383+0000", 2023-01-15T14:28:28.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016218600000000001 2023-01-15T14:28:28.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:28:28.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.800629+0000", 2023-01-15T14:28:28.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024592199999999998 2023-01-15T14:28:28.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:28.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.800852+0000", 2023-01-15T14:28:28.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000222711 2023-01-15T14:28:28.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:28.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.800867+0000", 2023-01-15T14:28:28.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4918e-05 2023-01-15T14:28:28.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.800940+0000", 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3484000000000003e-05 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.800981+0000", 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0799e-05 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:28.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.899983+0000", 2023-01-15T14:28:28.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099002268000000004 2023-01-15T14:28:28.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:28.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102625+0000", 2023-01-15T14:28:28.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20264195400000001 2023-01-15T14:28:28.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102779+0000", 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015422400000000001 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102957+0000", 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017745900000000001 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.294 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:12 3.a 3:52c3e133:::smithi09025264-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:28:28.294 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.799749+0000", 2023-01-15T14:28:28.294 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 1.468146212, 2023-01-15T14:28:28.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30375770000000002, 2023-01-15T14:28:28.294 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:28.294 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:28:28.294 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:28.294 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:28.294 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:28.294 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 12 2023-01-15T14:28:28.294 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799749+0000", 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799749+0000", 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:28.296 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799748+0000", 2023-01-15T14:28:28.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:28.296 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:28.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799755+0000", 2023-01-15T14:28:28.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1389999999999998e-06 2023-01-15T14:28:28.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:28.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799772+0000", 2023-01-15T14:28:28.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6827e-05 2023-01-15T14:28:28.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:28.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.899902+0000", 2023-01-15T14:28:28.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100129729 2023-01-15T14:28:28.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:28.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.998797+0000", 2023-01-15T14:28:28.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098894686999999995 2023-01-15T14:28:28.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.300 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:28.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.998954+0000", 2023-01-15T14:28:28.300 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015711699999999999 2023-01-15T14:28:28.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:28.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103084+0000", 2023-01-15T14:28:28.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104129765 2023-01-15T14:28:28.435 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.435 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.435 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:28.435 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103276+0000", 2023-01-15T14:28:28.435 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019277899999999999 2023-01-15T14:28:28.436 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.436 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.436 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:28.436 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103507+0000", 2023-01-15T14:28:28.436 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023087699999999999 2023-01-15T14:28:28.436 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.436 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:28.436 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.436 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.436 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.437 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:10 3.a e23/20 ondisk, result = 0)", 2023-01-15T14:28:28.437 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.901942+0000", 2023-01-15T14:28:28.437 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 1.3659536509999999, 2023-01-15T14:28:28.437 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20055753800000001, 2023-01-15T14:28:28.437 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:28.437 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:28:28.437 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:28.437 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.437 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:28.438 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.901942+0000", 2023-01-15T14:28:28.438 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.438 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.438 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:28.438 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.901942+0000", 2023-01-15T14:28:28.438 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.438 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.438 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:28.438 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.901940+0000", 2023-01-15T14:28:28.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:28.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:28.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.901951+0000", 2023-01-15T14:28:28.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1003e-05 2023-01-15T14:28:28.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:28.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.901978+0000", 2023-01-15T14:28:28.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6380000000000002e-05 2023-01-15T14:28:28.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.440 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:28.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.002106+0000", 2023-01-15T14:28:28.440 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10012784499999999 2023-01-15T14:28:28.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.440 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:28.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102330+0000", 2023-01-15T14:28:28.440 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100223997 2023-01-15T14:28:28.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:28.441 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102342+0000", 2023-01-15T14:28:28.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2099999999999999e-05 2023-01-15T14:28:28.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:28.441 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102500+0000", 2023-01-15T14:28:28.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157813 2023-01-15T14:28:28.441 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.441 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:28.442 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.442 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.442 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:45 3.0 e23/20 3:03b54904:::smithi09025264-12:head v 23'2, mlcod=0'0)", 2023-01-15T14:28:28.442 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.012130+0000", 2023-01-15T14:28:28.442 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 1.255765851, 2023-01-15T14:28:28.442 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202632118, 2023-01-15T14:28:28.442 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:28.442 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:28.443 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:28.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:28.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.012130+0000", 2023-01-15T14:28:28.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.443 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:28.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.012130+0000", 2023-01-15T14:28:28.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.444 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:28.444 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.012129+0000", 2023-01-15T14:28:28.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:28.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.444 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:28.444 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.012459+0000", 2023-01-15T14:28:28.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033069600000000001 2023-01-15T14:28:28.445 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.445 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:28.445 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.012479+0000", 2023-01-15T14:28:28.445 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9400000000000001e-05 2023-01-15T14:28:28.445 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.445 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:28.445 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.112568+0000", 2023-01-15T14:28:28.445 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088909 2023-01-15T14:28:28.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:28.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.212744+0000", 2023-01-15T14:28:28.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100176721 2023-01-15T14:28:28.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:28.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.212782+0000", 2023-01-15T14:28:28.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7673000000000003e-05 2023-01-15T14:28:28.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:28.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.213145+0000", 2023-01-15T14:28:28.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036323900000000001 2023-01-15T14:28:28.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:28.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.213192+0000", 2023-01-15T14:28:28.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7311999999999997e-05 2023-01-15T14:28:28.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:28.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.213443+0000", 2023-01-15T14:28:28.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025037099999999999 2023-01-15T14:28:28.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:28.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.213478+0000", 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.561e-05 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.214762+0000", 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012835139999999999 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.449 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:49 3.a 3:5919d83e:::smithi09025264-13:head [write 601684~619341,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:28.450 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.102530+0000", 2023-01-15T14:28:28.450 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 1.165366066, 2023-01-15T14:28:28.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.205031149, 2023-01-15T14:28:28.450 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:28.450 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:28.450 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:28.450 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:28.450 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:28.450 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 49 2023-01-15T14:28:28.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.450 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102530+0000", 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102530+0000", 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:28.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102527+0000", 2023-01-15T14:28:28.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:28:28.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:28.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.102984+0000", 2023-01-15T14:28:28.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045688899999999998 2023-01-15T14:28:28.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:28.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103019+0000", 2023-01-15T14:28:28.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.472e-05 2023-01-15T14:28:28.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:28.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203160+0000", 2023-01-15T14:28:28.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100141388 2023-01-15T14:28:28.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:28.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:28.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:28.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203331+0000", 2023-01-15T14:28:29.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017130300000000001 2023-01-15T14:28:29.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:29.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:29.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203594+0000", 2023-01-15T14:28:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000262229 2023-01-15T14:28:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:28:29.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203855+0000", 2023-01-15T14:28:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026152599999999999 2023-01-15T14:28:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:29.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.204601+0000", 2023-01-15T14:28:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00074620800000000005 2023-01-15T14:28:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.204630+0000", 2023-01-15T14:28:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8430000000000001e-05 2023-01-15T14:28:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:29.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.204979+0000", 2023-01-15T14:28:29.994 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000348795 2023-01-15T14:28:29.994 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.205201+0000", 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000222439 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.305123+0000", 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099922212999999996 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:29.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:29.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.307142+0000", 2023-01-15T14:28:29.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020185369999999999 2023-01-15T14:28:29.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.307279+0000", 2023-01-15T14:28:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013758600000000001 2023-01-15T14:28:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:29.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.307561+0000", 2023-01-15T14:28:29.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028138300000000001 2023-01-15T14:28:29.999 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:29.999 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:29.999 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.000 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.000 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.000 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:39 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'4, mlcod=0'0)", 2023-01-15T14:28:30.000 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.116096+0000", 2023-01-15T14:28:30.000 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 1.1517997710000001, 2023-01-15T14:28:30.000 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200333285, 2023-01-15T14:28:30.001 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:30.001 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:30.001 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:30.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:30.001 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.116096+0000", 2023-01-15T14:28:30.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:30.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.116096+0000", 2023-01-15T14:28:30.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:30.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.116095+0000", 2023-01-15T14:28:30.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:30.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:30.003 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.116105+0000", 2023-01-15T14:28:30.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0349999999999999e-05 2023-01-15T14:28:30.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:30.003 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.116137+0000", 2023-01-15T14:28:30.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1699999999999998e-05 2023-01-15T14:28:30.004 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.004 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:30.004 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.116147+0000", 2023-01-15T14:28:30.004 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4900000000000006e-06 2023-01-15T14:28:30.004 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.004 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.004 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.315599+0000", 2023-01-15T14:28:30.004 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19945253900000001 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.315637+0000", 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7663000000000002e-05 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.315779+0000", 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014272300000000001 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:30.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.315796+0000", 2023-01-15T14:28:30.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6756999999999999e-05 2023-01-15T14:28:30.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:30.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.315865+0000", 2023-01-15T14:28:30.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8514000000000002e-05 2023-01-15T14:28:30.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:30.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.316329+0000", 2023-01-15T14:28:30.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000464244 2023-01-15T14:28:30.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:30.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.316429+0000", 2023-01-15T14:28:30.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010020499999999999 2023-01-15T14:28:30.007 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.007 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:30.008 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.012 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:50 3.a 3:5919d83e:::smithi09025264-13:head [write 1669078~532606] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:30.012 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.203200+0000", 2023-01-15T14:28:30.012 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 1.0646959030000001, 2023-01-15T14:28:30.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.107626886, 2023-01-15T14:28:30.013 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:30.013 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:30.014 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:30.014 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:30.014 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:30.014 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 50 2023-01-15T14:28:30.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.014 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:30.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:30.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203200+0000", 2023-01-15T14:28:30.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:30.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203200+0000", 2023-01-15T14:28:30.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:30.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203198+0000", 2023-01-15T14:28:30.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:28:30.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:30.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203693+0000", 2023-01-15T14:28:30.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049565500000000001 2023-01-15T14:28:30.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:30.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203713+0000", 2023-01-15T14:28:30.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9313999999999998e-05 2023-01-15T14:28:30.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203724+0000", 2023-01-15T14:28:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1620000000000001e-05 2023-01-15T14:28:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.204770+0000", 2023-01-15T14:28:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010460059999999999 2023-01-15T14:28:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.204949+0000", 2023-01-15T14:28:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000178467 2023-01-15T14:28:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.307690+0000", 2023-01-15T14:28:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10274158999999999 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.307831+0000", 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014093600000000001 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.308027+0000", 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000195556 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.308681+0000", 2023-01-15T14:28:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000654085 2023-01-15T14:28:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.308711+0000", 2023-01-15T14:28:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.9692999999999998e-05 2023-01-15T14:28:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.308988+0000", 2023-01-15T14:28:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027727300000000003 2023-01-15T14:28:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.310446+0000", 2023-01-15T14:28:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0014585920000000001 2023-01-15T14:28:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:30.162 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.310492+0000", 2023-01-15T14:28:30.162 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.5513000000000003e-05 2023-01-15T14:28:30.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.162 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.163 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:30.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.310570+0000", 2023-01-15T14:28:30.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8499999999999997e-05 2023-01-15T14:28:30.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:30.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.310689+0000", 2023-01-15T14:28:30.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118363 2023-01-15T14:28:30.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:30.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.310827+0000", 2023-01-15T14:28:30.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013789299999999999 2023-01-15T14:28:30.166 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.166 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:30.166 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.167 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:52 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:30.167 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.203808+0000", 2023-01-15T14:28:30.167 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 1.064087995, 2023-01-15T14:28:30.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.107861925, 2023-01-15T14:28:30.167 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:30.167 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:28:30.167 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:30.167 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:30.168 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:30.168 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 52 2023-01-15T14:28:30.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.168 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:30.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:30.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203808+0000", 2023-01-15T14:28:30.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.169 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:30.169 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203808+0000", 2023-01-15T14:28:30.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.169 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.169 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.169 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:30.169 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203807+0000", 2023-01-15T14:28:30.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:30.169 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.169 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.169 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:30.170 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203812+0000", 2023-01-15T14:28:30.170 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7439999999999998e-06 2023-01-15T14:28:30.170 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.170 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.170 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:30.170 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203822+0000", 2023-01-15T14:28:30.170 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0492999999999999e-05 2023-01-15T14:28:30.170 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.170 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.170 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:30.170 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.203829+0000", 2023-01-15T14:28:30.171 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8530000000000003e-06 2023-01-15T14:28:30.171 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.172 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.172 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.172 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.207238+0000", 2023-01-15T14:28:30.173 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0034092269999999999 2023-01-15T14:28:30.173 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.173 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.173 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:30.173 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.207377+0000", 2023-01-15T14:28:30.173 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013856599999999999 2023-01-15T14:28:30.173 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.173 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.173 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.173 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.310139+0000", 2023-01-15T14:28:30.173 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102762521 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.310359+0000", 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000219316 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.311368+0000", 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010092670000000001 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.174 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.175 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:30.175 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.311499+0000", 2023-01-15T14:28:30.175 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013080899999999999 2023-01-15T14:28:30.175 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.175 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.175 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.311670+0000", 2023-01-15T14:28:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000170886 2023-01-15T14:28:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:89 3.d e23/20)", 2023-01-15T14:28:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.400388+0000", 2023-01-15T14:28:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.86750768700000003, 2023-01-15T14:28:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20357592599999999, 2023-01-15T14:28:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.400388+0000", 2023-01-15T14:28:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.400388+0000", 2023-01-15T14:28:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.400387+0000", 2023-01-15T14:28:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.400844+0000", 2023-01-15T14:28:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045635499999999998 2023-01-15T14:28:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.400876+0000", 2023-01-15T14:28:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2852999999999999e-05 2023-01-15T14:28:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.500992+0000", 2023-01-15T14:28:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100115448 2023-01-15T14:28:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601254+0000", 2023-01-15T14:28:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100261741 2023-01-15T14:28:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601287+0000", 2023-01-15T14:28:30.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3529999999999999e-05 2023-01-15T14:28:30.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:30.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601700+0000", 2023-01-15T14:28:30.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000412452 2023-01-15T14:28:30.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:30.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601765+0000", 2023-01-15T14:28:30.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5179999999999996e-05 2023-01-15T14:28:30.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:30.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602078+0000", 2023-01-15T14:28:30.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00031326299999999999 2023-01-15T14:28:30.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:30.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602403+0000", 2023-01-15T14:28:30.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032492900000000001 2023-01-15T14:28:30.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:30.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603964+0000", 2023-01-15T14:28:30.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0015609479999999999 2023-01-15T14:28:30.330 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.330 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:30.330 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.330 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:97 3.a 3:52df5138:::smithi09025264-25:head [write 403368~555035,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:30.330 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:28:30.330 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.76675283100000002, 2023-01-15T14:28:30.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.340758171, 2023-01-15T14:28:30.330 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:30.331 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:30.331 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:30.331 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:30.331 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:30.331 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 97 2023-01-15T14:28:30.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.331 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:30.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:30.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:28:30.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:30.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:28:30.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:30.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501140+0000", 2023-01-15T14:28:30.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:28:30.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:30.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501684+0000", 2023-01-15T14:28:30.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054441799999999996 2023-01-15T14:28:30.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:30.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501713+0000", 2023-01-15T14:28:30.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8833e-05 2023-01-15T14:28:30.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.334 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:30.334 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501726+0000", 2023-01-15T14:28:30.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3137e-05 2023-01-15T14:28:30.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.334 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601982+0000", 2023-01-15T14:28:30.335 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100256518 2023-01-15T14:28:30.336 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.336 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.336 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:30.336 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602426+0000", 2023-01-15T14:28:30.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044401500000000002 2023-01-15T14:28:30.336 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.336 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.336 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:28:30.336 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602703+0000", 2023-01-15T14:28:30.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000276889 2023-01-15T14:28:30.337 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.337 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.337 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:30.337 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603066+0000", 2023-01-15T14:28:30.337 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036278599999999998 2023-01-15T14:28:30.337 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.337 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.337 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:30.337 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603098+0000", 2023-01-15T14:28:30.337 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1433000000000002e-05 2023-01-15T14:28:30.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.338 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:30.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603382+0000", 2023-01-15T14:28:30.338 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028402899999999999 2023-01-15T14:28:30.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.338 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:30.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603632+0000", 2023-01-15T14:28:30.338 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025033 2023-01-15T14:28:30.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:30.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805354+0000", 2023-01-15T14:28:30.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20172168700000001 2023-01-15T14:28:30.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:30.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841465+0000", 2023-01-15T14:28:30.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.036111852999999999 2023-01-15T14:28:30.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:30.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841682+0000", 2023-01-15T14:28:30.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000216949 2023-01-15T14:28:30.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:30.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841901+0000", 2023-01-15T14:28:30.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021858700000000001 2023-01-15T14:28:30.546 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.546 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:30.546 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.547 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:98 3.a 3:52df5138:::smithi09025264-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:30.547 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:28:30.547 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.76614316000000005, 2023-01-15T14:28:30.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.44410839299999999, 2023-01-15T14:28:30.547 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:30.547 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:30.547 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:30.547 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:30.547 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:30.547 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 98 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:30.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501752+0000", 2023-01-15T14:28:30.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:30.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:30.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502223+0000", 2023-01-15T14:28:30.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000471843 2023-01-15T14:28:30.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502240+0000", 2023-01-15T14:28:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6663e-05 2023-01-15T14:28:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502248+0000", 2023-01-15T14:28:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5669999999999997e-06 2023-01-15T14:28:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603236+0000", 2023-01-15T14:28:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100988439 2023-01-15T14:28:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603384+0000", 2023-01-15T14:28:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014844200000000001 2023-01-15T14:28:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842016+0000", 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23863163800000001 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842267+0000", 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025101300000000002 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842561+0000", 2023-01-15T14:28:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029432900000000002 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843120+0000", 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055900500000000001 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843146+0000", 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5273e-05 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843512+0000", 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036621599999999999 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943324+0000", 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099812008999999993 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945437+0000", 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021126069999999999 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945619+0000", 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018280600000000001 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945744+0000", 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012511299999999999 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945861+0000", 2023-01-15T14:28:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011646000000000001 2023-01-15T14:28:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:99 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:28:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.76563290399999995, 2023-01-15T14:28:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.54355587900000002, 2023-01-15T14:28:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 99 2023-01-15T14:28:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502262+0000", 2023-01-15T14:28:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502270+0000", 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1999999999999994e-06 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502286+0000", 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5423000000000001e-05 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502295+0000", 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4339999999999998e-06 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603467+0000", 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101172225 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603585+0000", 2023-01-15T14:28:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117149 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843335+0000", 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23975083799999999 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843573+0000", 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023753000000000001 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:28:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843978+0000", 2023-01-15T14:28:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040533799999999999 2023-01-15T14:28:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844309+0000", 2023-01-15T14:28:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000330713 2023-01-15T14:28:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844423+0000", 2023-01-15T14:28:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114176 2023-01-15T14:28:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844510+0000", 2023-01-15T14:28:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7026000000000003e-05 2023-01-15T14:28:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943338+0000", 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098827398999999996 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945520+0000", 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021824169999999999 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045652+0000", 2023-01-15T14:28:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013193200000001 2023-01-15T14:28:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045733+0000", 2023-01-15T14:28:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1435999999999998e-05 2023-01-15T14:28:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045819+0000", 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5292999999999998e-05 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:110 3.3 e23/20 3:c834566e:::smithi09025264-28:head v 23'11, mlcod=23'7)", 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.805535+0000", 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.46236061000000001, 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.108544238, 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805535+0000", 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805535+0000", 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805535+0000", 2023-01-15T14:28:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805887+0000", 2023-01-15T14:28:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035198400000000001 2023-01-15T14:28:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805903+0000", 2023-01-15T14:28:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6015000000000002e-05 2023-01-15T14:28:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805910+0000", 2023-01-15T14:28:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8399999999999997e-06 2023-01-15T14:28:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906437+0000", 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100526904 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906482+0000", 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.5612999999999998e-05 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906876+0000", 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039385499999999998 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906900+0000", 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4130000000000001e-05 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907152+0000", 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025230899999999999 2023-01-15T14:28:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907437+0000", 2023-01-15T14:28:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028408900000000001 2023-01-15T14:28:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.914079+0000", 2023-01-15T14:28:30.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0066427960000000003 2023-01-15T14:28:30.706 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:111 3.3 e23/20 3:c834566e:::smithi09025264-28:head v 23'12, mlcod=23'7)", 2023-01-15T14:28:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.805920+0000", 2023-01-15T14:28:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.461975368, 2023-01-15T14:28:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.111413541, 2023-01-15T14:28:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:30.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:30.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805920+0000", 2023-01-15T14:28:30.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:30.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805920+0000", 2023-01-15T14:28:30.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:30.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805920+0000", 2023-01-15T14:28:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.877 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:30.877 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805926+0000", 2023-01-15T14:28:30.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9000000000000003e-06 2023-01-15T14:28:30.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.877 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:30.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805933+0000", 2023-01-15T14:28:30.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9600000000000003e-06 2023-01-15T14:28:30.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:30.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:30.879 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:30.879 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805941+0000", 2023-01-15T14:28:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5739999999999996e-06 2023-01-15T14:28:34.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:34.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:34.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:34.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906974+0000", 2023-01-15T14:28:34.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101032648 2023-01-15T14:28:34.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:34.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:34.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:34.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907027+0000", 2023-01-15T14:28:34.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2916e-05 2023-01-15T14:28:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907268+0000", 2023-01-15T14:28:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024122600000000001 2023-01-15T14:28:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907286+0000", 2023-01-15T14:28:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7900000000000001e-05 2023-01-15T14:28:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907368+0000", 2023-01-15T14:28:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2349999999999996e-05 2023-01-15T14:28:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907468+0000", 2023-01-15T14:28:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9980000000000002e-05 2023-01-15T14:28:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:34.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:34.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.917334+0000", 2023-01-15T14:28:34.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0098654869999999992 2023-01-15T14:28:34.352 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:34.352 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:34.353 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:34.353 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:34.353 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:34.354 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:34.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:28:34.355 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:28:34.355 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:28:34.558 INFO:tasks.rados.rados.0.smithi090.stdout:44: writing smithi09025264-44 from 722701 to 1340173 tid 1 2023-01-15T14:28:34.558 INFO:tasks.rados.rados.0.smithi090.stdout:44: writing smithi09025264-44 from 1816608 to 2604609 tid 2 2023-01-15T14:28:34.558 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:34.558 INFO:tasks.rados.rados.0.smithi090.stdout:27: finishing write tid 3 to smithi09025264-27 2023-01-15T14:28:34.559 INFO:tasks.rados.rados.0.smithi090.stdout:37: finishing write tid 2 to smithi09025264-37 2023-01-15T14:28:34.559 INFO:tasks.rados.rados.0.smithi090.stdout:31: finishing write tid 2 to smithi09025264-31 2023-01-15T14:28:34.559 INFO:tasks.rados.rados.0.smithi090.stdout:31: finishing write tid 3 to smithi09025264-31 2023-01-15T14:28:34.559 INFO:tasks.rados.rados.0.smithi090.stdout:31: finishing write tid 4 to smithi09025264-31 2023-01-15T14:28:34.559 INFO:tasks.rados.rados.0.smithi090.stdout:31: oid 31 updating version 0 to 15 2023-01-15T14:28:34.559 INFO:tasks.rados.rados.0.smithi090.stdout:31: oid 31 version 15 is already newer than 14 2023-01-15T14:28:34.560 INFO:tasks.rados.rados.0.smithi090.stdout:31: oid 31 version 15 is already newer than 13 2023-01-15T14:28:34.560 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 15 (ObjNum 30 snap 0 seq_num 30) dirty exists 2023-01-15T14:28:34.560 INFO:tasks.rados.rados.0.smithi090.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2023-01-15T14:28:34.560 INFO:tasks.rados.rados.0.smithi090.stdout:44: finishing write tid 1 to smithi09025264-44 2023-01-15T14:28:34.560 INFO:tasks.rados.rados.0.smithi090.stdout:31: done (15 left) 2023-01-15T14:28:34.560 INFO:tasks.rados.rados.0.smithi090.stdout:45: write initial oid 45 2023-01-15T14:28:34.561 INFO:tasks.rados.rados.0.smithi090.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2023-01-15T14:28:34.561 INFO:tasks.rados.rados.0.smithi090.stdout:45: writing smithi09025264-45 from 739508 to 1432229 tid 1 2023-01-15T14:28:34.561 INFO:tasks.rados.rados.0.smithi090.stdout:45: writing smithi09025264-45 from 1875090 to 2523102 tid 2 2023-01-15T14:28:34.561 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:34.561 INFO:tasks.rados.rados.0.smithi090.stdout:39: finishing write tid 1 to smithi09025264-39 2023-01-15T14:28:34.561 INFO:tasks.rados.rados.0.smithi090.stdout:33: finishing write tid 3 to smithi09025264-33 2023-01-15T14:28:34.757 INFO:tasks.rados.rados.0.smithi090.stdout:33: finishing write tid 4 to smithi09025264-33 2023-01-15T14:28:34.757 INFO:tasks.rados.rados.0.smithi090.stdout:33: oid 33 updating version 0 to 3 2023-01-15T14:28:34.757 INFO:tasks.rados.rados.0.smithi090.stdout:33: oid 33 version 3 is already newer than 1 2023-01-15T14:28:34.757 INFO:tasks.rados.rados.0.smithi090.stdout:33: oid 33 version 3 is already newer than 2 2023-01-15T14:28:34.757 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 3 (ObjNum 32 snap 0 seq_num 32) dirty exists 2023-01-15T14:28:34.757 INFO:tasks.rados.rados.0.smithi090.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2023-01-15T14:28:34.757 INFO:tasks.rados.rados.0.smithi090.stdout:33: done (15 left) 2023-01-15T14:28:34.757 INFO:tasks.rados.rados.0.smithi090.stdout:46: write initial oid 46 2023-01-15T14:28:34.757 INFO:tasks.rados.rados.0.smithi090.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2023-01-15T14:28:34.757 INFO:tasks.rados.rados.0.smithi090.stdout:46: writing smithi09025264-46 from 756315 to 1524285 tid 1 2023-01-15T14:28:34.758 INFO:tasks.rados.rados.0.smithi090.stdout:46: writing smithi09025264-46 from 1933572 to 2725242 tid 2 2023-01-15T14:28:34.758 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:34.758 INFO:tasks.rados.rados.0.smithi090.stdout:35: finishing write tid 3 to smithi09025264-35 2023-01-15T14:28:34.758 INFO:tasks.rados.rados.0.smithi090.stdout:27: finishing write tid 4 to smithi09025264-27 2023-01-15T14:28:34.758 INFO:tasks.rados.rados.0.smithi090.stdout:27: oid 27 updating version 0 to 5 2023-01-15T14:28:34.758 INFO:tasks.rados.rados.0.smithi090.stdout:27: oid 27 version 5 is already newer than 4 2023-01-15T14:28:34.759 INFO:tasks.rados.rados.0.smithi090.stdout:27: oid 27 version 5 is already newer than 2 2023-01-15T14:28:34.759 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 5 (ObjNum 26 snap 0 seq_num 26) dirty exists 2023-01-15T14:28:34.759 INFO:tasks.rados.rados.0.smithi090.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2023-01-15T14:28:34.759 INFO:tasks.rados.rados.0.smithi090.stdout:23: finishing write tid 3 to smithi09025264-23 2023-01-15T14:28:34.759 INFO:tasks.rados.rados.0.smithi090.stdout:27: done (15 left) 2023-01-15T14:28:34.759 INFO:tasks.rados.rados.0.smithi090.stdout:47: write initial oid 47 2023-01-15T14:28:34.759 INFO:tasks.rados.rados.0.smithi090.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2023-01-15T14:28:34.759 INFO:tasks.rados.rados.0.smithi090.stdout:47: writing smithi09025264-47 from 773122 to 1332694 tid 1 2023-01-15T14:28:34.759 INFO:tasks.rados.rados.0.smithi090.stdout:47: writing smithi09025264-47 from 1992054 to 2643735 tid 2 2023-01-15T14:28:34.760 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:34.760 INFO:tasks.rados.rados.0.smithi090.stdout:23: finishing write tid 4 to smithi09025264-23 2023-01-15T14:28:34.760 INFO:tasks.rados.rados.0.smithi090.stdout:23: oid 23 updating version 0 to 6 2023-01-15T14:28:34.761 INFO:tasks.rados.rados.0.smithi090.stdout:23: oid 23 version 6 is already newer than 1 2023-01-15T14:28:34.761 INFO:tasks.rados.rados.0.smithi090.stdout:23: oid 23 version 6 is already newer than 3 2023-01-15T14:28:34.761 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2023-01-15T14:28:34.761 INFO:tasks.rados.rados.0.smithi090.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2023-01-15T14:28:34.761 INFO:tasks.rados.rados.0.smithi090.stdout:23: done (15 left) 2023-01-15T14:28:34.761 INFO:tasks.rados.rados.0.smithi090.stdout:48: write initial oid 48 2023-01-15T14:28:34.761 INFO:tasks.rados.rados.0.smithi090.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2023-01-15T14:28:34.762 INFO:tasks.rados.rados.0.smithi090.stdout:48: writing smithi09025264-48 from 789929 to 1424750 tid 1 2023-01-15T14:28:34.762 INFO:tasks.rados.rados.0.smithi090.stdout:48: writing smithi09025264-48 from 2050536 to 2789929 tid 2 2023-01-15T14:28:34.762 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:34.762 INFO:tasks.rados.rados.0.smithi090.stdout:19: finishing write tid 3 to smithi09025264-19 2023-01-15T14:28:34.762 INFO:tasks.rados.rados.0.smithi090.stdout:19: finishing write tid 4 to smithi09025264-19 2023-01-15T14:28:34.762 INFO:tasks.rados.rados.0.smithi090.stdout:19: oid 19 updating version 0 to 8 2023-01-15T14:28:34.762 INFO:tasks.rados.rados.0.smithi090.stdout:19: oid 19 updating version 8 to 9 2023-01-15T14:28:34.762 INFO:tasks.rados.rados.0.smithi090.stdout:19: oid 19 version 9 is already newer than 5 2023-01-15T14:28:34.762 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 9 (ObjNum 18 snap 0 seq_num 18) dirty exists 2023-01-15T14:28:34.762 INFO:tasks.rados.rados.0.smithi090.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout:19: done (15 left) 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout:49: write initial oid 49 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout:49: writing smithi09025264-49 from 406736 to 1116806 tid 1 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout:49: writing smithi09025264-49 from 1709018 to 2364368 tid 2 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout:32: finishing write tid 2 to smithi09025264-32 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout:37: finishing write tid 3 to smithi09025264-37 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout:37: finishing write tid 4 to smithi09025264-37 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout:37: oid 37 updating version 0 to 13 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout:37: oid 37 updating version 13 to 14 2023-01-15T14:28:34.763 INFO:tasks.rados.rados.0.smithi090.stdout:37: oid 37 updating version 14 to 15 2023-01-15T14:28:34.764 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 15 (ObjNum 36 snap 0 seq_num 36) dirty exists 2023-01-15T14:28:34.764 INFO:tasks.rados.rados.0.smithi090.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2023-01-15T14:28:34.764 INFO:tasks.rados.rados.0.smithi090.stdout:37: done (15 left) 2023-01-15T14:28:34.764 INFO:tasks.rados.rados.0.smithi090.stdout:50: write initial oid 50 2023-01-15T14:28:34.764 INFO:tasks.rados.rados.0.smithi090.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2023-01-15T14:28:34.764 INFO:tasks.rados.rados.0.smithi090.stdout:50: writing smithi09025264-50 from 423543 to 1208862 tid 1 2023-01-15T14:28:34.764 INFO:tasks.rados.rados.0.smithi090.stdout:50: writing smithi09025264-50 from 1767500 to 2566508 tid 2 2023-01-15T14:28:34.764 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:34.764 INFO:tasks.rados.rados.0.smithi090.stdout:39: finishing write tid 2 to smithi09025264-39 2023-01-15T14:28:34.764 INFO:tasks.rados.rados.0.smithi090.stdout:39: finishing write tid 3 to smithi09025264-39 2023-01-15T14:28:34.765 INFO:tasks.rados.rados.0.smithi090.stdout:36: finishing write tid 2 to smithi09025264-36 2023-01-15T14:28:34.765 INFO:tasks.rados.rados.0.smithi090.stdout:36: finishing write tid 3 to smithi09025264-36 2023-01-15T14:28:34.765 INFO:tasks.rados.rados.0.smithi090.stdout:43: finishing write tid 1 to smithi09025264-43 2023-01-15T14:28:34.765 INFO:tasks.rados.rados.0.smithi090.stdout:35: finishing write tid 4 to smithi09025264-35 2023-01-15T14:28:34.765 INFO:tasks.rados.rados.0.smithi090.stdout:35: oid 35 updating version 0 to 6 2023-01-15T14:28:34.954 INFO:tasks.rados.rados.0.smithi090.stdout:35: oid 35 version 6 is already newer than 4 2023-01-15T14:28:34.955 INFO:tasks.rados.rados.0.smithi090.stdout:35: oid 35 version 6 is already newer than 5 2023-01-15T14:28:34.955 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-01-15T14:28:34.955 INFO:tasks.rados.rados.0.smithi090.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2023-01-15T14:28:34.955 INFO:tasks.rados.rados.0.smithi090.stdout:42: finishing write tid 1 to smithi09025264-42 2023-01-15T14:28:34.955 INFO:tasks.rados.rados.0.smithi090.stdout:45: finishing write tid 1 to smithi09025264-45 2023-01-15T14:28:34.955 INFO:tasks.rados.rados.0.smithi090.stdout:35: done (15 left) 2023-01-15T14:28:34.955 INFO:tasks.rados.rados.0.smithi090.stdout:51: copy_from oid 2 from oid 11 current snap is 0 2023-01-15T14:28:34.955 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:34.955 INFO:tasks.rados.rados.0.smithi090.stdout:47: finishing write tid 1 to smithi09025264-47 2023-01-15T14:28:34.955 INFO:tasks.rados.rados.0.smithi090.stdout:44: finishing write tid 2 to smithi09025264-44 2023-01-15T14:28:34.955 INFO:tasks.rados.rados.0.smithi090.stdout:44: finishing write tid 3 to smithi09025264-44 2023-01-15T14:28:34.956 INFO:tasks.rados.rados.0.smithi090.stdout:44: finishing write tid 4 to smithi09025264-44 2023-01-15T14:28:34.956 INFO:tasks.rados.rados.0.smithi090.stdout:44: oid 44 updating version 0 to 12 2023-01-15T14:28:34.956 INFO:tasks.rados.rados.0.smithi090.stdout:44: oid 44 version 12 is already newer than 11 2023-01-15T14:28:34.956 INFO:tasks.rados.rados.0.smithi090.stdout:44: oid 44 version 12 is already newer than 10 2023-01-15T14:28:34.956 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 12 (ObjNum 43 snap 0 seq_num 43) dirty exists 2023-01-15T14:28:34.956 INFO:tasks.rados.rados.0.smithi090.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2023-01-15T14:28:34.956 INFO:tasks.rados.rados.0.smithi090.stdout:44: done (15 left) 2023-01-15T14:28:34.956 INFO:tasks.rados.rados.0.smithi090.stdout:52: write oid 25 current snap is 0 2023-01-15T14:28:34.956 INFO:tasks.rados.rados.0.smithi090.stdout:52: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2023-01-15T14:28:34.957 INFO:tasks.rados.rados.0.smithi090.stdout:52: writing smithi09025264-25 from 440350 to 900918 tid 1 2023-01-15T14:28:34.958 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:34.349+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:28:34.958 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:27.461+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:34.962 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:34.957+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:28:34.967 INFO:tasks.rados.rados.0.smithi090.stdout:52: writing smithi09025264-25 from 1309629 to 1852295 tid 2 2023-01-15T14:28:34.967 INFO:tasks.rados.rados.0.smithi090.stdout:52: writing smithi09025264-25 from 2323973 to 2840350 tid 3 2023-01-15T14:28:34.967 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:34.968 INFO:tasks.rados.rados.0.smithi090.stdout:38: finishing write tid 1 to smithi09025264-38 2023-01-15T14:28:34.968 INFO:tasks.rados.rados.0.smithi090.stdout:39: finishing write tid 4 to smithi09025264-39 2023-01-15T14:28:34.968 INFO:tasks.rados.rados.0.smithi090.stdout:39: oid 39 updating version 0 to 6 2023-01-15T14:28:34.968 INFO:tasks.rados.rados.0.smithi090.stdout:39: oid 39 version 6 is already newer than 5 2023-01-15T14:28:34.968 INFO:tasks.rados.rados.0.smithi090.stdout:39: oid 39 version 6 is already newer than 4 2023-01-15T14:28:34.969 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 6 (ObjNum 38 snap 0 seq_num 38) dirty exists 2023-01-15T14:28:34.969 INFO:tasks.rados.rados.0.smithi090.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2023-01-15T14:28:34.969 INFO:tasks.rados.rados.0.smithi090.stdout:39: done (15 left) 2023-01-15T14:28:34.969 INFO:tasks.rados.rados.0.smithi090.stdout:53: rollback oid 30 current snap is 0 2023-01-15T14:28:34.969 INFO:tasks.rados.rados.0.smithi090.stdout:53: done (15 left) 2023-01-15T14:28:34.969 INFO:tasks.rados.rados.0.smithi090.stdout:54: write (excl) oid 21 current snap is 0 2023-01-15T14:28:35.161 INFO:tasks.rados.rados.0.smithi090.stdout:54: seq_num 51 ranges {457157=535817,1768111=402677} 2023-01-15T14:28:35.161 INFO:tasks.rados.rados.0.smithi090.stdout:54: writing smithi09025264-21 from 457157 to 992974 tid 1 2023-01-15T14:28:35.161 INFO:tasks.rados.rados.0.smithi090.stdout:54: writing smithi09025264-21 from 1768111 to 2170788 tid 2 2023-01-15T14:28:35.161 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:35.161 INFO:tasks.rados.rados.0.smithi090.stdout:40: finishing write tid 1 to smithi09025264-40 2023-01-15T14:28:35.161 INFO:tasks.rados.rados.0.smithi090.stdout:36: finishing write tid 4 to smithi09025264-36 2023-01-15T14:28:35.161 INFO:tasks.rados.rados.0.smithi090.stdout:36: oid 36 updating version 0 to 8 2023-01-15T14:28:35.162 INFO:tasks.rados.rados.0.smithi090.stdout:36: oid 36 updating version 8 to 9 2023-01-15T14:28:35.162 INFO:tasks.rados.rados.0.smithi090.stdout:36: oid 36 version 9 is already newer than 7 2023-01-15T14:28:35.162 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-01-15T14:28:35.162 INFO:tasks.rados.rados.0.smithi090.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2023-01-15T14:28:35.162 INFO:tasks.rados.rados.0.smithi090.stdout:41: finishing write tid 1 to smithi09025264-41 2023-01-15T14:28:35.162 INFO:tasks.rados.rados.0.smithi090.stdout:32: finishing write tid 3 to smithi09025264-32 2023-01-15T14:28:35.162 INFO:tasks.rados.rados.0.smithi090.stdout:32: finishing write tid 4 to smithi09025264-32 2023-01-15T14:28:35.162 INFO:tasks.rados.rados.0.smithi090.stdout:32: oid 32 updating version 0 to 10 2023-01-15T14:28:35.162 INFO:tasks.rados.rados.0.smithi090.stdout:32: oid 32 updating version 10 to 12 2023-01-15T14:28:35.162 INFO:tasks.rados.rados.0.smithi090.stdout:32: oid 32 version 12 is already newer than 11 2023-01-15T14:28:35.163 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 12 (ObjNum 31 snap 0 seq_num 31) dirty exists 2023-01-15T14:28:35.163 INFO:tasks.rados.rados.0.smithi090.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2023-01-15T14:28:35.165 INFO:tasks.rados.rados.0.smithi090.stdout:32: done (15 left) 2023-01-15T14:28:35.168 INFO:tasks.rados.rados.0.smithi090.stdout:36: done (14 left) 2023-01-15T14:28:35.168 INFO:tasks.rados.rados.0.smithi090.stdout:55: snap_create 2023-01-15T14:28:35.168 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 14 2023-01-15T14:28:35.169 INFO:tasks.rados.rados.0.smithi090.stdout:45: finishing write tid 2 to smithi09025264-45 2023-01-15T14:28:35.169 INFO:tasks.rados.rados.0.smithi090.stdout:38: finishing write tid 2 to smithi09025264-38 2023-01-15T14:28:35.169 INFO:tasks.rados.rados.0.smithi090.stdout:38: finishing write tid 3 to smithi09025264-38 2023-01-15T14:28:35.169 INFO:tasks.rados.rados.0.smithi090.stdout:38: finishing write tid 4 to smithi09025264-38 2023-01-15T14:28:35.169 INFO:tasks.rados.rados.0.smithi090.stdout:38: oid 38 updating version 0 to 7 2023-01-15T14:28:35.169 INFO:tasks.rados.rados.0.smithi090.stdout:38: oid 38 updating version 7 to 8 2023-01-15T14:28:35.169 INFO:tasks.rados.rados.0.smithi090.stdout:38: oid 38 updating version 8 to 9 2023-01-15T14:28:35.169 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout:38: done (13 left) 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 13 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout:41: finishing write tid 2 to smithi09025264-41 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout:41: finishing write tid 3 to smithi09025264-41 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout:41: finishing write tid 4 to smithi09025264-41 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout:41: oid 41 updating version 0 to 10 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout:41: oid 41 updating version 10 to 12 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout:41: oid 41 version 12 is already newer than 11 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 12 (ObjNum 40 snap 0 seq_num 40) dirty exists 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2023-01-15T14:28:35.170 INFO:tasks.rados.rados.0.smithi090.stdout:41: done (12 left) 2023-01-15T14:28:35.171 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 12 2023-01-15T14:28:35.171 INFO:tasks.rados.rados.0.smithi090.stdout:45: finishing write tid 3 to smithi09025264-45 2023-01-15T14:28:35.171 INFO:tasks.rados.rados.0.smithi090.stdout:45: finishing write tid 4 to smithi09025264-45 2023-01-15T14:28:35.171 INFO:tasks.rados.rados.0.smithi090.stdout:45: oid 45 updating version 0 to 2 2023-01-15T14:28:35.171 INFO:tasks.rados.rados.0.smithi090.stdout:45: oid 45 updating version 2 to 3 2023-01-15T14:28:35.171 INFO:tasks.rados.rados.0.smithi090.stdout:45: oid 45 version 3 is already newer than 1 2023-01-15T14:28:35.172 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 3 (ObjNum 44 snap 0 seq_num 44) dirty exists 2023-01-15T14:28:35.172 INFO:tasks.rados.rados.0.smithi090.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2023-01-15T14:28:35.172 INFO:tasks.rados.rados.0.smithi090.stdout:45: done (11 left) 2023-01-15T14:28:35.172 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 11 2023-01-15T14:28:35.172 INFO:tasks.rados.rados.0.smithi090.stdout:47: finishing write tid 2 to smithi09025264-47 2023-01-15T14:28:35.172 INFO:tasks.rados.rados.0.smithi090.stdout:47: finishing write tid 3 to smithi09025264-47 2023-01-15T14:28:35.172 INFO:tasks.rados.rados.0.smithi090.stdout:48: finishing write tid 1 to smithi09025264-48 2023-01-15T14:28:35.172 INFO:tasks.rados.rados.0.smithi090.stdout:46: finishing write tid 1 to smithi09025264-46 2023-01-15T14:28:35.180 INFO:tasks.rados.rados.0.smithi090.stdout:54: finishing write tid 1 to smithi09025264-21 2023-01-15T14:28:35.180 INFO:tasks.rados.rados.0.smithi090.stdout:50: finishing write tid 1 to smithi09025264-50 2023-01-15T14:28:35.181 INFO:tasks.rados.rados.0.smithi090.stdout:51: finishing copy_from to smithi09025264-2 2023-01-15T14:28:35.181 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 5 (ObjNum 10 snap 0 seq_num 10) dirty exists 2023-01-15T14:28:35.181 INFO:tasks.rados.rados.0.smithi090.stdout:51: finishing copy_from racing read to smithi09025264-2 2023-01-15T14:28:35.181 INFO:tasks.rados.rados.0.smithi090.stdout:51: done (10 left) 2023-01-15T14:28:35.181 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 10 2023-01-15T14:28:35.181 INFO:tasks.rados.rados.0.smithi090.stdout:42: finishing write tid 2 to smithi09025264-42 2023-01-15T14:28:35.181 INFO:tasks.rados.rados.0.smithi090.stdout:42: finishing write tid 3 to smithi09025264-42 2023-01-15T14:28:35.181 INFO:tasks.rados.rados.0.smithi090.stdout:47: finishing write tid 4 to smithi09025264-47 2023-01-15T14:28:35.181 INFO:tasks.rados.rados.0.smithi090.stdout:47: oid 47 updating version 0 to 17 2023-01-15T14:28:35.370 INFO:tasks.rados.rados.0.smithi090.stdout:47: oid 47 updating version 17 to 18 2023-01-15T14:28:35.370 INFO:tasks.rados.rados.0.smithi090.stdout:47: oid 47 version 18 is already newer than 16 2023-01-15T14:28:35.370 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 18 (ObjNum 46 snap 0 seq_num 46) dirty exists 2023-01-15T14:28:35.370 INFO:tasks.rados.rados.0.smithi090.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2023-01-15T14:28:35.370 INFO:tasks.rados.rados.0.smithi090.stdout:47: done (9 left) 2023-01-15T14:28:35.370 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 9 2023-01-15T14:28:35.371 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:28:35.371 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:28:35.371 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 23, 2023-01-15T14:28:35.373 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "1c77fb2b-d1a7-4884-a24e-3935e16d0b6b", 2023-01-15T14:28:35.374 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2023-01-15T14:27:57.735717+0000", 2023-01-15T14:28:35.374 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2023-01-15T14:28:25.741673+0000", 2023-01-15T14:28:35.374 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2023-01-15T14:28:04.837522+0000", 2023-01-15T14:28:35.374 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2023-01-15T14:28:01.797608+0000", 2023-01-15T14:28:35.374 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-01-15T14:28:35.374 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5806080, 2023-01-15T14:28:35.375 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2023-01-15T14:28:35.375 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2023-01-15T14:28:35.375 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2023-01-15T14:28:35.375 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2023-01-15T14:28:35.375 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2023-01-15T14:28:35.375 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2023-01-15T14:28:35.375 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2023-01-15T14:28:35.375 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:28:35.375 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2023-01-15T14:28:35.375 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2023-01-15T14:28:35.375 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-01-15T14:28:35.376 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2023-01-15T14:28:35.376 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2023-01-15T14:28:35.376 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 3, 2023-01-15T14:28:35.376 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2023-01-15T14:28:35.376 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2023-01-15T14:28:35.376 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "jewel", 2023-01-15T14:28:35.376 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "reef", 2023-01-15T14:28:35.376 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2023-01-15T14:28:35.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:35.376 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2023-01-15T14:28:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2023-01-15T14:28:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:04.899282+0000", 2023-01-15T14:28:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:28:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:28:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:28:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:28:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:28:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:28:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:28:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:28:35.377 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:28:35.378 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:28:35.378 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:28:35.378 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:28:35.378 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2023-01-15T14:28:35.378 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2023-01-15T14:28:35.378 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2023-01-15T14:28:35.378 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2023-01-15T14:28:35.378 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2023-01-15T14:28:35.379 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:28:35.379 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:28:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:28:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:28:35.571 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:28:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:28:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:28:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "16", 2023-01-15T14:28:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:28:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:28:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:28:35.572 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:28:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:28:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:28:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:28:35.573 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:28:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:28:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:28:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:28:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:28:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:28:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:28:35.575 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:28:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:28:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:28:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:28:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:28:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:28:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:28:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:28:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:28:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:28:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:28:35.576 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:28:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:28:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:28:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:28:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:28:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:28:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:28:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:28:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:28:35.577 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:28:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:28:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:28:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2023-01-15T14:28:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2023-01-15T14:28:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2023-01-15T14:28:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:28:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2023-01-15T14:28:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:35.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2023-01-15T14:28:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2023-01-15T14:28:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:11.115002+0000", 2023-01-15T14:28:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:28:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:28:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:28:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:28:35.579 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:28:35.738 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:28:35.738 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:28:35.738 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:28:35.738 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:28:35.738 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:28:35.739 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:28:35.740 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:28:35.740 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:28:35.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:35.740 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "19", 2023-01-15T14:28:35.740 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:28:35.740 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:28:35.740 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:28:35.740 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:28:35.741 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:28:35.741 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:28:35.741 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:28:35.741 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:28:35.741 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:28:35.745 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:28:35.746 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:28:35.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:28:35.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:28:35.747 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:28:35.747 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:28:35.748 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:28:35.749 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:28:35.749 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:28:35.749 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:28:35.749 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:28:35.749 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:28:35.749 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:28:35.749 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:28:35.749 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:28:35.750 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:28:35.750 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:28:35.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:35.750 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:28:35.750 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:28:35.750 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:28:35.750 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:28:35.750 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:28:35.751 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:28:35.751 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:28:35.751 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:28:35.751 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:28:35.751 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:28:35.751 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:28:35.751 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:28:35.752 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:28:35.752 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2023-01-15T14:28:35.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:35.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:35.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:35.752 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:28:35.752 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "unique_pool_0", 2023-01-15T14:28:35.753 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:22.434903+0000", 2023-01-15T14:28:35.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:28:35.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:28:35.753 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:28:35.753 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:28:35.753 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:28:35.753 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:28:36.898 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:28:36.899 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:28:36.899 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:28:36.899 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:28:36.899 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:28:36.899 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:28:36.899 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 16, 2023-01-15T14:28:36.899 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 16, 2023-01-15T14:28:36.899 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 16, 2023-01-15T14:28:36.900 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 16, 2023-01-15T14:28:36.900 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 16, 2023-01-15T14:28:36.900 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:28:36.900 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:28:36.900 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:28:36.900 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:28:36.900 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:28:36.900 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:28:36.900 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:28:36.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:36.901 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "23", 2023-01-15T14:28:36.901 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:28:36.901 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:28:36.901 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:28:36.901 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:28:36.901 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:28:36.901 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:28:36.901 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:28:36.902 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:28:36.902 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:28:36.902 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:28:36.902 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:28:36.902 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:28:36.902 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:28:36.902 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:28:36.902 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:28:36.902 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:28:36.903 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:28:36.903 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:28:36.903 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:28:36.903 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:28:36.903 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:28:36.903 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:28:36.903 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:28:36.903 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:28:36.903 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:28:36.904 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:28:36.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:36.904 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:28:36.904 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:28:36.904 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:28:36.905 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:28:36.906 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:28:36.906 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:28:36.906 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:28:36.906 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:28:36.906 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:28:36.906 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:28:36.906 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:28:36.906 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:28:36.906 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:28:36.906 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2023-01-15T14:28:36.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:36.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:36.907 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:28:36.907 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:28:36.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:36.907 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:28:36.907 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e55c5ac4-3cff-4ee5-8719-373648f83d8d", 2023-01-15T14:28:36.907 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:28:36.907 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:28:36.908 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:28:36.908 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:28:36.908 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:28:36.908 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:28:36.908 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:28:36.908 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:28:36.908 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:28:36.908 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:28:36.909 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:28:36.909 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:36.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:36.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:36.909 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2023-01-15T14:28:36.909 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:28:36.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:36.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:36.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:36.909 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2023-01-15T14:28:36.910 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:28:36.910 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:36.910 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:36.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:36.910 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:28:36.910 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:36.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:36.910 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:36.910 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2023-01-15T14:28:36.911 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:28:36.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:36.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:36.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:37.413 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2023-01-15T14:28:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:28:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:28:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2023-01-15T14:28:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:28:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2023-01-15T14:28:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:28:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:28:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2023-01-15T14:28:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:28:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2023-01-15T14:28:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:28:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6811/23473", 2023-01-15T14:28:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6815/23473", 2023-01-15T14:28:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/23473", 2023-01-15T14:28:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6820/23473", 2023-01-15T14:28:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:28:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:28:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:28:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:28:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "2c1b33c9-0a1e-4734-936d-d51d64fe997a", 2023-01-15T14:28:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:28:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:28:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:28:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:28:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:28:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:28:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:28:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:28:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:28:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:28:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:28:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2023-01-15T14:28:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:28:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2023-01-15T14:28:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:28:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:28:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2023-01-15T14:28:37.615 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:28:37.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.616 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:37.616 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2023-01-15T14:28:37.616 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:28:37.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:37.616 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:37.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.616 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:28:37.616 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:37.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.616 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:37.617 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2023-01-15T14:28:37.617 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:28:37.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.617 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:37.617 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2023-01-15T14:28:37.617 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:28:37.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:37.617 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:37.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.617 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:28:37.618 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:37.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:37.618 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2023-01-15T14:28:37.618 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:28:37.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:37.620 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2023-01-15T14:28:37.620 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:28:37.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:37.620 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:37.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.620 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/23474", 2023-01-15T14:28:37.620 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/23474", 2023-01-15T14:28:37.620 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/23474", 2023-01-15T14:28:37.620 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/23474", 2023-01-15T14:28:37.621 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:28:37.621 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:28:37.621 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:28:37.621 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:37.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:37.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.621 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:28:37.621 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "b314174a-eca9-4baa-a60c-2589f1fc9a8e", 2023-01-15T14:28:37.621 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:28:37.621 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:28:37.622 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:28:37.622 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:28:37.622 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:28:37.622 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:28:37.622 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:28:37.622 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 17, 2023-01-15T14:28:37.622 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:28:37.622 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:28:37.622 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:28:37.623 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:37.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:37.623 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:37.820 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2023-01-15T14:28:38.042 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:28:38.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.045 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2023-01-15T14:28:38.045 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:28:38.045 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.045 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.045 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:28:38.045 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.045 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:28:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:28:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2023-01-15T14:28:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:28:38.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.229 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.229 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2023-01-15T14:28:38.229 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:28:38.229 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.229 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.229 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6810/23475", 2023-01-15T14:28:38.229 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/23475", 2023-01-15T14:28:38.232 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6821/23475", 2023-01-15T14:28:38.232 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6817/23475", 2023-01-15T14:28:38.233 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:28:38.233 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:28:38.233 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:28:38.233 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.233 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:28:38.233 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "29eb0b17-ed5d-4665-a347-2fb8406d9c62", 2023-01-15T14:28:38.233 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:28:38.233 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:28:38.233 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:28:38.234 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:28:38.234 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:28:38.234 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:28:38.234 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:28:38.234 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:28:38.234 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:28:38.234 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:28:38.234 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:28:38.234 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.235 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.235 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6824", 2023-01-15T14:28:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:28:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6825", 2023-01-15T14:28:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:28:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.237 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.238 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:28:38.238 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.238 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.238 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6826", 2023-01-15T14:28:38.238 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:28:38.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.238 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.239 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6827", 2023-01-15T14:28:38.239 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:28:38.239 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.239 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.239 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:28:38.437 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.437 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.437 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6830", 2023-01-15T14:28:38.437 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:28:38.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.440 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.441 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6831", 2023-01-15T14:28:38.441 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:28:38.441 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.441 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.442 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:28:38.442 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.442 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.443 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6828", 2023-01-15T14:28:38.443 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:28:38.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.443 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.443 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6829", 2023-01-15T14:28:38.444 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:28:38.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.445 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.445 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6825/23778", 2023-01-15T14:28:38.445 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6827/23778", 2023-01-15T14:28:38.445 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6831/23778", 2023-01-15T14:28:38.445 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6829/23778", 2023-01-15T14:28:38.446 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:28:38.446 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:28:38.446 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:28:38.446 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.446 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:28:38.603 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "59362feb-caa9-44e3-b6d0-b79397dacbfc", 2023-01-15T14:28:38.603 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:28:38.603 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:28:38.603 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:28:38.603 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:28:38.604 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:28:38.604 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:28:38.604 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:28:38.604 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:28:38.604 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:28:38.604 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:28:38.605 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:28:38.608 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.759 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.759 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6800", 2023-01-15T14:28:38.760 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:28:38.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.760 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.760 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6801", 2023-01-15T14:28:38.760 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:28:38.760 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.761 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.761 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:28:38.761 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.761 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.761 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6802", 2023-01-15T14:28:38.761 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:28:38.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.761 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.762 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6803", 2023-01-15T14:28:38.762 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:28:38.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.762 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.762 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:28:38.762 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.762 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.762 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6806", 2023-01-15T14:28:38.763 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:28:38.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.763 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.763 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6807", 2023-01-15T14:28:38.763 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:28:38.763 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.763 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.764 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:28:38.764 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6804", 2023-01-15T14:28:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:28:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6805", 2023-01-15T14:28:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:28:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6801/23523", 2023-01-15T14:28:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6803/23523", 2023-01-15T14:28:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6807/23523", 2023-01-15T14:28:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6805/23523", 2023-01-15T14:28:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:28:38.923 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:28:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:28:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:28:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "fbc8821c-8d78-40f8-a571-b18a70d42000", 2023-01-15T14:28:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:28:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:28:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:28:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:28:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:28:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:28:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:28:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:28:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:28:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:28:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:28:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.928 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6824", 2023-01-15T14:28:38.928 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:28:38.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.929 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.929 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6825", 2023-01-15T14:28:38.929 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:28:38.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.929 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.929 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:28:38.929 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.930 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.930 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6826", 2023-01-15T14:28:38.930 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:28:38.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.930 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.930 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6827", 2023-01-15T14:28:38.930 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:28:38.930 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.930 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.931 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:28:38.931 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.931 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.931 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6830", 2023-01-15T14:28:38.931 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:28:38.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.931 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.931 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6831", 2023-01-15T14:28:38.932 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:28:38.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.932 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.932 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:28:38.932 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.932 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.932 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6828", 2023-01-15T14:28:38.932 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:28:38.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.933 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:38.933 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6829", 2023-01-15T14:28:38.933 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:28:38.933 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:38.933 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.933 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6825/23525", 2023-01-15T14:28:38.933 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6827/23525", 2023-01-15T14:28:38.933 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6831/23525", 2023-01-15T14:28:38.934 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6829/23525", 2023-01-15T14:28:38.934 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:28:38.934 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:28:38.934 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:28:38.934 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:38.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:38.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.934 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:28:38.934 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "34550470-3561-492f-b605-b5f0e8052c1e", 2023-01-15T14:28:38.934 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:28:38.934 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:28:38.935 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:28:38.935 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:28:38.935 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:28:38.935 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:28:38.935 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:28:38.935 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:28:38.935 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:28:38.935 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:28:38.935 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:28:38.935 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:38.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:38.936 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:38.936 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6808", 2023-01-15T14:28:38.936 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:28:38.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.084 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:39.084 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6809", 2023-01-15T14:28:39.084 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:28:39.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:39.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.084 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:28:39.084 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:39.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.084 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:39.085 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6810", 2023-01-15T14:28:39.085 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:28:39.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:39.087 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6811", 2023-01-15T14:28:39.087 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:28:39.087 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.088 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:39.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.088 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:28:39.088 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:39.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.088 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:39.088 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6814", 2023-01-15T14:28:39.088 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:28:39.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:39.089 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6815", 2023-01-15T14:28:39.089 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:28:39.089 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.089 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:39.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.089 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:28:39.089 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:39.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:39.090 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6812", 2023-01-15T14:28:39.090 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:28:39.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.090 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:39.090 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6813", 2023-01-15T14:28:39.090 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:28:39.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.090 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:39.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.090 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6809/23524", 2023-01-15T14:28:39.091 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6811/23524", 2023-01-15T14:28:39.091 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6815/23524", 2023-01-15T14:28:39.091 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6813/23524", 2023-01-15T14:28:39.091 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:28:39.091 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:28:39.091 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:28:39.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:39.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.091 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:28:39.092 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1f123d0d-d617-4771-b516-21aa5a29b734", 2023-01-15T14:28:39.092 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:28:39.092 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:28:39.092 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:28:39.092 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:28:39.092 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:28:39.092 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:28:39.092 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:28:39.092 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:28:39.092 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:28:39.093 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:28:39.093 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:28:39.093 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:39.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.093 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:39.093 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6816", 2023-01-15T14:28:39.093 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:28:39.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.243 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:39.243 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6817", 2023-01-15T14:28:39.243 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:28:39.243 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.243 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:39.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.244 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:28:39.244 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:39.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.244 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:39.244 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6818", 2023-01-15T14:28:39.244 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:28:39.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.244 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:39.244 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6819", 2023-01-15T14:28:39.244 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:28:39.245 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.245 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:39.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.245 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6822", 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6823", 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.247 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:39.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.248 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:28:39.248 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:28:39.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.249 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:28:39.249 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6820", 2023-01-15T14:28:39.249 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:28:39.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.250 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:28:39.250 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6821", 2023-01-15T14:28:39.250 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:28:39.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.250 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:39.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.250 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6817/23526", 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6819/23526", 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6823/23526", 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6821/23526", 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.251 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:28:39.252 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:28:39.252 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:28:39.252 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:28:39.252 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:28:39.252 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:28:39.252 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.117195+0000", 2023-01-15T14:28:39.252 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:28:39.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.252 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:28:39.252 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:28:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:28:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:28:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:28:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:28:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.065464+0000", 2023-01-15T14:28:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:28:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:28:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:28:39.253 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.138817+0000", 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:28:39.254 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:28:39.255 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:28:39.255 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.284776+0000", 2023-01-15T14:28:39.255 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:28:39.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.255 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:28:39.255 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:28:39.255 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:28:39.255 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:28:39.255 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:28:39.255 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:28:39.256 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.271024+0000", 2023-01-15T14:28:39.256 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:28:39.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.256 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:28:39.256 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:28:39.256 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:28:39.256 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:28:39.256 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:28:39.256 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:28:39.402 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.318787+0000", 2023-01-15T14:28:39.402 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:28:39.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.402 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:28:39.402 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:28:39.402 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:28:39.402 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:28:39.402 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:28:39.403 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:28:39.403 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.333018+0000", 2023-01-15T14:28:39.403 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:28:39.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:39.403 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:28:39.403 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:28:39.403 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:28:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:28:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:28:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:28:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.375590+0000", 2023-01-15T14:28:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:28:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:28:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2023-01-15T14:28:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [], 2023-01-15T14:28:39.404 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:39.406 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [], 2023-01-15T14:28:39.406 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2023-01-15T14:28:39.407 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2023-01-15T14:28:39.407 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2023-01-15T14:28:39.407 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2023-01-15T14:28:39.408 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2023-01-15T14:28:39.408 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2023-01-15T14:28:39.408 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2023-01-15T14:28:39.408 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2023-01-15T14:28:39.408 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2023-01-15T14:28:39.408 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2023-01-15T14:28:39.408 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:39.408 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:28:39.409 INFO:tasks.rados.rados.0.smithi090.stdout:40: finishing write tid 2 to smithi09025264-40 2023-01-15T14:28:39.409 INFO:tasks.rados.rados.0.smithi090.stdout:40: finishing write tid 3 to smithi09025264-40 2023-01-15T14:28:39.410 INFO:tasks.rados.rados.0.smithi090.stdout:40: finishing write tid 4 to smithi09025264-40 2023-01-15T14:28:39.410 INFO:tasks.rados.rados.0.smithi090.stdout:40: oid 40 updating version 0 to 10 2023-01-15T14:28:39.410 INFO:tasks.rados.rados.0.smithi090.stdout:40: oid 40 version 10 is already newer than 7 2023-01-15T14:28:39.410 INFO:tasks.rados.rados.0.smithi090.stdout:40: oid 40 version 10 is already newer than 9 2023-01-15T14:28:39.410 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 10 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-01-15T14:28:39.410 INFO:tasks.rados.rados.0.smithi090.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2023-01-15T14:28:39.410 INFO:tasks.rados.rados.0.smithi090.stdout:46: finishing write tid 2 to smithi09025264-46 2023-01-15T14:28:39.410 INFO:tasks.rados.rados.0.smithi090.stdout:40: done (8 left) 2023-01-15T14:28:39.410 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:28:39.412 INFO:tasks.thrashosds.thrasher:j is {'epoch': 23, 'fsid': '1c77fb2b-d1a7-4884-a24e-3935e16d0b6b', 'created': '2023-01-15T14:27:57.735717+0000', 'modified': '2023-01-15T14:28:25.741673+0000', 'last_up_change': '2023-01-15T14:28:04.837522+0000', 'last_in_change': '2023-01-15T14:28:01.797608+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-01-15T14:28:04.899282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-01-15T14:28:11.115002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-01-15T14:28:22.434903+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e55c5ac4-3cff-4ee5-8719-373648f83d8d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 23473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 23473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 23473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6820', 'nonce': 23473}]}, 'public_addr': '172.21.15.90:6811/23473', 'cluster_addr': '172.21.15.90:6815/23473', 'heartbeat_back_addr': '172.21.15.90:6823/23473', 'heartbeat_front_addr': '172.21.15.90:6820/23473', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2c1b33c9-0a1e-4734-936d-d51d64fe997a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 23474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 23474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 23474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 23474}]}, 'public_addr': '172.21.15.90:6801/23474', 'cluster_addr': '172.21.15.90:6803/23474', 'heartbeat_back_addr': '172.21.15.90:6807/23474', 'heartbeat_front_addr': '172.21.15.90:6805/23474', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b314174a-eca9-4baa-a60c-2589f1fc9a8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 23475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 23475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6819', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 23475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 23475}]}, 'public_addr': '172.21.15.90:6810/23475', 'cluster_addr': '172.21.15.90:6814/23475', 'heartbeat_back_addr': '172.21.15.90:6821/23475', 'heartbeat_front_addr': '172.21.15.90:6817/23475', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '29eb0b17-ed5d-4665-a347-2fb8406d9c62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6824', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6825', 'nonce': 23778}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6826', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6827', 'nonce': 23778}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6830', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6831', 'nonce': 23778}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6828', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6829', 'nonce': 23778}]}, 'public_addr': '172.21.15.90:6825/23778', 'cluster_addr': '172.21.15.90:6827/23778', 'heartbeat_back_addr': '172.21.15.90:6831/23778', 'heartbeat_front_addr': '172.21.15.90:6829/23778', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '59362feb-caa9-44e3-b6d0-b79397dacbfc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6800', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6801', 'nonce': 23523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6802', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6803', 'nonce': 23523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6806', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6807', 'nonce': 23523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6804', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6805', 'nonce': 23523}]}, 'public_addr': '172.21.15.141:6801/23523', 'cluster_addr': '172.21.15.141:6803/23523', 'heartbeat_back_addr': '172.21.15.141:6807/23523', 'heartbeat_front_addr': '172.21.15.141:6805/23523', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fbc8821c-8d78-40f8-a571-b18a70d42000', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6824', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6825', 'nonce': 23525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6826', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6827', 'nonce': 23525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6830', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6831', 'nonce': 23525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6828', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6829', 'nonce': 23525}]}, 'public_addr': '172.21.15.141:6825/23525', 'cluster_addr': '172.21.15.141:6827/23525', 'heartbeat_back_addr': '172.21.15.141:6831/23525', 'heartbeat_front_addr': '172.21.15.141:6829/23525', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '34550470-3561-492f-b605-b5f0e8052c1e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6808', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6809', 'nonce': 23524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6810', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6811', 'nonce': 23524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6814', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6815', 'nonce': 23524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6812', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6813', 'nonce': 23524}]}, 'public_addr': '172.21.15.141:6809/23524', 'cluster_addr': '172.21.15.141:6811/23524', 'heartbeat_back_addr': '172.21.15.141:6815/23524', 'heartbeat_front_addr': '172.21.15.141:6813/23524', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f123d0d-d617-4771-b516-21aa5a29b734', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6816', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6817', 'nonce': 23526}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6818', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6819', 'nonce': 23526}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6822', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6823', 'nonce': 23526}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6820', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6821', 'nonce': 23526}]}, 'public_addr': '172.21.15.141:6817/23526', 'cluster_addr': '172.21.15.141:6819/23526', 'heartbeat_back_addr': '172.21.15.141:6823/23526', 'heartbeat_front_addr': '172.21.15.141:6821/23526', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.117195+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.065464+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.138817+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.284776+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.271024+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.318787+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.333018+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.375590+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-01-15T14:28:39.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:28:39.415 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:39.563 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:28:39.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.563 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:285 3.f 3.c1803e1f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:39.564 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.887431+0000", 2023-01-15T14:28:39.564 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.18820943800000001, 2023-01-15T14:28:39.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.188250748, 2023-01-15T14:28:39.564 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:39.564 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:28:39.565 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:39.565 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:39.565 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:39.566 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 285 2023-01-15T14:28:39.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.572 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:39.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:39.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887431+0000", 2023-01-15T14:28:39.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:39.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887431+0000", 2023-01-15T14:28:39.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:39.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887429+0000", 2023-01-15T14:28:39.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:28:39.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:39.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887444+0000", 2023-01-15T14:28:39.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5113000000000001e-05 2023-01-15T14:28:39.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:39.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887520+0000", 2023-01-15T14:28:39.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6290000000000003e-05 2023-01-15T14:28:39.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:39.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887543+0000", 2023-01-15T14:28:39.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2963e-05 2023-01-15T14:28:39.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:39.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887682+0000", 2023-01-15T14:28:39.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013889300000000001 2023-01-15T14:28:39.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:28:39.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887701+0000", 2023-01-15T14:28:39.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9049e-05 2023-01-15T14:28:39.577 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.577 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:39.577 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:286 3.f 3.c1803e1f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.887589+0000", 2023-01-15T14:28:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.18805163899999999, 2023-01-15T14:28:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18820803799999999, 2023-01-15T14:28:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:28:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 286 2023-01-15T14:28:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887589+0000", 2023-01-15T14:28:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887589+0000", 2023-01-15T14:28:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887588+0000", 2023-01-15T14:28:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887599+0000", 2023-01-15T14:28:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1313e-05 2023-01-15T14:28:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887617+0000", 2023-01-15T14:28:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8597000000000002e-05 2023-01-15T14:28:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887631+0000", 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3322999999999999e-05 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887827+0000", 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019593200000000001 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:28:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887836+0000", 2023-01-15T14:28:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4339999999999998e-06 2023-01-15T14:28:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:287 3.f 3.c1803e1f (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.887648+0000", 2023-01-15T14:28:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.18799237599999999, 2023-01-15T14:28:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18824750100000001, 2023-01-15T14:28:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:28:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 287 2023-01-15T14:28:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887648+0000", 2023-01-15T14:28:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887648+0000", 2023-01-15T14:28:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887647+0000", 2023-01-15T14:28:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887656+0000", 2023-01-15T14:28:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.1269999999999992e-06 2023-01-15T14:28:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.887674+0000", 2023-01-15T14:28:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7863000000000002e-05 2023-01-15T14:28:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.987823+0000", 2023-01-15T14:28:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100148681 2023-01-15T14:28:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988617+0000", 2023-01-15T14:28:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00079369499999999997 2023-01-15T14:28:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:28:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988634+0000", 2023-01-15T14:28:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7082999999999999e-05 2023-01-15T14:28:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:299 3.d 3.ad888fcd (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.987912+0000", 2023-01-15T14:28:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.087728720999999996, 2023-01-15T14:28:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.088079685000000005, 2023-01-15T14:28:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:28:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 299 2023-01-15T14:28:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.987912+0000", 2023-01-15T14:28:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.987912+0000", 2023-01-15T14:28:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.987909+0000", 2023-01-15T14:28:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:28:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.987925+0000", 2023-01-15T14:28:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6120000000000002e-05 2023-01-15T14:28:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.987949+0000", 2023-01-15T14:28:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4709e-05 2023-01-15T14:28:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.987961+0000", 2023-01-15T14:28:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1274000000000001e-05 2023-01-15T14:28:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988087+0000", 2023-01-15T14:28:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000126315 2023-01-15T14:28:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:28:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988104+0000", 2023-01-15T14:28:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6759999999999999e-05 2023-01-15T14:28:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:300 3.d 3.ad888fcd (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.987991+0000", 2023-01-15T14:28:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.087649615, 2023-01-15T14:28:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.088104034999999997, 2023-01-15T14:28:39.885 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:39.886 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:28:39.889 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:39.889 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:39.889 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:39.889 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 300 2023-01-15T14:28:39.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.889 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:39.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:39.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.987991+0000", 2023-01-15T14:28:39.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:39.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.987991+0000", 2023-01-15T14:28:39.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:39.891 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.987990+0000", 2023-01-15T14:28:39.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:39.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:39.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.987999+0000", 2023-01-15T14:28:39.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5459999999999992e-06 2023-01-15T14:28:39.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:39.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988015+0000", 2023-01-15T14:28:39.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6379999999999999e-05 2023-01-15T14:28:39.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988026+0000", 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.154e-05 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:301 3.d 3.ad888fcd (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.988049+0000", 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.087591698999999995, 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.088123335999999997, 2023-01-15T14:28:39.894 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:39.895 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:28:39.895 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:39.895 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:39.895 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:39.895 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 301 2023-01-15T14:28:39.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.895 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:39.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:39.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988049+0000", 2023-01-15T14:28:39.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:39.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988049+0000", 2023-01-15T14:28:39.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:39.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988048+0000", 2023-01-15T14:28:39.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:39.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:39.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988056+0000", 2023-01-15T14:28:39.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8900000000000007e-06 2023-01-15T14:28:39.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:39.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988072+0000", 2023-01-15T14:28:39.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5879000000000002e-05 2023-01-15T14:28:39.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:39.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988081+0000", 2023-01-15T14:28:39.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2699999999999993e-06 2023-01-15T14:28:39.898 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.898 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:39.899 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:39.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.899 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:302 3.f 3.72cbb71f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:39.899 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.988096+0000", 2023-01-15T14:28:39.899 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.087544226000000003, 2023-01-15T14:28:39.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.088155894999999998, 2023-01-15T14:28:39.899 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:39.899 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:28:39.899 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:39.900 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:39.900 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:39.900 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 302 2023-01-15T14:28:39.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.900 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:39.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:39.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988096+0000", 2023-01-15T14:28:39.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:39.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:39.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988096+0000", 2023-01-15T14:28:39.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:39.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:39.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:40.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988096+0000", 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988466+0000", 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037093199999999998 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988486+0000", 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9969000000000001e-05 2023-01-15T14:28:40.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:40.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988495+0000", 2023-01-15T14:28:40.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5569999999999996e-06 2023-01-15T14:28:40.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:40.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989024+0000", 2023-01-15T14:28:40.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052927200000000001 2023-01-15T14:28:40.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:28:40.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989034+0000", 2023-01-15T14:28:40.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5040000000000008e-06 2023-01-15T14:28:40.057 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.057 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:40.057 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.059 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.059 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:303 3.f 3.72cbb71f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:40.061 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.988513+0000", 2023-01-15T14:28:40.061 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.087127799000000006, 2023-01-15T14:28:40.064 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087831424000000005, 2023-01-15T14:28:40.064 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:40.065 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:28:40.065 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:40.065 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:40.065 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:40.065 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 303 2023-01-15T14:28:40.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.065 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:40.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:40.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988513+0000", 2023-01-15T14:28:40.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:40.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:40.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988513+0000", 2023-01-15T14:28:40.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:40.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:40.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988512+0000", 2023-01-15T14:28:40.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:40.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:40.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988936+0000", 2023-01-15T14:28:40.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042455100000000001 2023-01-15T14:28:40.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:40.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988951+0000", 2023-01-15T14:28:40.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4260999999999999e-05 2023-01-15T14:28:40.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.068 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.068 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:40.068 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988957+0000", 2023-01-15T14:28:40.068 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2700000000000001e-06 2023-01-15T14:28:40.068 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.068 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.068 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:40.068 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989170+0000", 2023-01-15T14:28:40.068 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021282500000000001 2023-01-15T14:28:40.068 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.068 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.069 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:28:40.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989178+0000", 2023-01-15T14:28:40.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3280000000000006e-06 2023-01-15T14:28:40.069 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.069 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:40.069 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.069 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.069 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:304 3.f 3.72cbb71f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:40.069 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.988970+0000", 2023-01-15T14:28:40.069 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.086670284, 2023-01-15T14:28:40.070 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087468245, 2023-01-15T14:28:40.070 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:40.070 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:28:40.070 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:40.070 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:40.070 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:40.070 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 304 2023-01-15T14:28:40.070 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.070 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:40.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.070 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:40.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988970+0000", 2023-01-15T14:28:40.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:40.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.071 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:40.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988970+0000", 2023-01-15T14:28:40.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:40.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.071 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:40.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.988969+0000", 2023-01-15T14:28:40.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:40.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:40.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989084+0000", 2023-01-15T14:28:40.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011511200000000001 2023-01-15T14:28:40.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:40.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989097+0000", 2023-01-15T14:28:40.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2217e-05 2023-01-15T14:28:40.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.073 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:40.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989102+0000", 2023-01-15T14:28:40.075 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.3020000000000002e-06 2023-01-15T14:28:40.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:40.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989223+0000", 2023-01-15T14:28:40.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120914 2023-01-15T14:28:40.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:28:40.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989230+0000", 2023-01-15T14:28:40.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.464e-06 2023-01-15T14:28:40.076 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.077 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:40.078 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.078 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.078 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.078 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:305 3.f 3.72cbb71f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:40.078 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.989114+0000", 2023-01-15T14:28:40.079 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.086526949000000006, 2023-01-15T14:28:40.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087416578999999994, 2023-01-15T14:28:40.079 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:40.079 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:28:40.079 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:40.079 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:40.079 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:40.079 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 305 2023-01-15T14:28:40.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.079 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:40.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:40.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989114+0000", 2023-01-15T14:28:40.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:40.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:40.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989114+0000", 2023-01-15T14:28:40.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:40.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:40.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989113+0000", 2023-01-15T14:28:40.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:40.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:40.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989120+0000", 2023-01-15T14:28:40.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3679999999999996e-06 2023-01-15T14:28:40.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:40.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989130+0000", 2023-01-15T14:28:40.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0169999999999999e-05 2023-01-15T14:28:40.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:40.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989137+0000", 2023-01-15T14:28:40.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0480000000000001e-06 2023-01-15T14:28:40.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989277+0000", 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014001700000000001 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989285+0000", 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5299999999999999e-06 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.084 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:306 3.f 3.72cbb71f (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:40.084 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.989160+0000", 2023-01-15T14:28:40.084 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.086480562999999996, 2023-01-15T14:28:40.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087480932999999997, 2023-01-15T14:28:40.084 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:40.084 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:28:40.084 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:40.084 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:40.084 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:40.084 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 306 2023-01-15T14:28:40.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.085 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:40.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:40.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989160+0000", 2023-01-15T14:28:40.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:40.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989160+0000", 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989159+0000", 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989164+0000", 2023-01-15T14:28:40.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3420000000000001e-06 2023-01-15T14:28:40.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.215 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.215 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:40.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989175+0000", 2023-01-15T14:28:40.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1702000000000001e-05 2023-01-15T14:28:40.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:40.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989182+0000", 2023-01-15T14:28:40.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7939999999999997e-06 2023-01-15T14:28:40.218 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.218 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:40.218 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.218 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:265 3.7 e30/20)", 2023-01-15T14:28:40.218 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.989373+0000", 2023-01-15T14:28:40.218 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.086267436000000003, 2023-01-15T14:28:40.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087342008999999998, 2023-01-15T14:28:40.219 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:40.219 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "no flag points reached", 2023-01-15T14:28:40.219 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:40.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:40.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989373+0000", 2023-01-15T14:28:40.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:40.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989373+0000", 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989373+0000", 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.990499+0000", 2023-01-15T14:28:40.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001126606 2023-01-15T14:28:40.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:40.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:40.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:40.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.990529+0000", 2023-01-15T14:28:40.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.934e-05 2023-01-15T14:28:40.221 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.221 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:40.221 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.221 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:40.222 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2023-01-15T14:28:40.222 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 12 2023-01-15T14:28:40.222 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:40.222 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:28:40.223 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:39.405+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:28:40.230 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:40.221+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:40.540 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:40.540 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:28:40.540 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:28:40.540 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:28:40.540 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:40.541 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:28:40.543 INFO:tasks.rados.rados.0.smithi090.stdout:42: finishing write tid 4 to smithi09025264-42 2023-01-15T14:28:40.543 INFO:tasks.rados.rados.0.smithi090.stdout:42: oid 42 updating version 0 to 9 2023-01-15T14:28:40.543 INFO:tasks.rados.rados.0.smithi090.stdout:42: oid 42 version 9 is already newer than 7 2023-01-15T14:28:40.543 INFO:tasks.rados.rados.0.smithi090.stdout:42: oid 42 version 9 is already newer than 8 2023-01-15T14:28:40.543 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2023-01-15T14:28:40.544 INFO:tasks.rados.rados.0.smithi090.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2023-01-15T14:28:40.544 INFO:tasks.rados.rados.0.smithi090.stdout:42: done (7 left) 2023-01-15T14:28:40.544 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:28:40.544 INFO:tasks.rados.rados.0.smithi090.stdout:49: finishing write tid 1 to smithi09025264-49 2023-01-15T14:28:40.544 INFO:tasks.rados.rados.0.smithi090.stdout:48: finishing write tid 2 to smithi09025264-48 2023-01-15T14:28:40.544 INFO:tasks.rados.rados.0.smithi090.stdout:46: finishing write tid 3 to smithi09025264-46 2023-01-15T14:28:40.544 INFO:tasks.rados.rados.0.smithi090.stdout:46: finishing write tid 4 to smithi09025264-46 2023-01-15T14:28:40.544 INFO:tasks.rados.rados.0.smithi090.stdout:46: oid 46 updating version 0 to 11 2023-01-15T14:28:40.544 INFO:tasks.rados.rados.0.smithi090.stdout:46: oid 46 updating version 11 to 12 2023-01-15T14:28:40.544 INFO:tasks.rados.rados.0.smithi090.stdout:46: oid 46 version 12 is already newer than 8 2023-01-15T14:28:40.544 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 12 (ObjNum 45 snap 0 seq_num 45) dirty exists 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:46: done (6 left) 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:54: finishing write tid 2 to smithi09025264-21 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:50: finishing write tid 2 to smithi09025264-50 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:43: finishing write tid 2 to smithi09025264-43 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:43: finishing write tid 3 to smithi09025264-43 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:49: finishing write tid 2 to smithi09025264-49 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:52: finishing write tid 1 to smithi09025264-25 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:52: finishing write tid 2 to smithi09025264-25 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:52: finishing write tid 3 to smithi09025264-25 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:52: finishing write tid 4 to smithi09025264-25 2023-01-15T14:28:40.545 INFO:tasks.rados.rados.0.smithi090.stdout:52: finishing write tid 5 to smithi09025264-25 2023-01-15T14:28:40.546 INFO:tasks.rados.rados.0.smithi090.stdout:52: oid 25 updating version 0 to 16 2023-01-15T14:28:40.546 INFO:tasks.rados.rados.0.smithi090.stdout:52: oid 25 version 16 is already newer than 15 2023-01-15T14:28:40.546 INFO:tasks.rados.rados.0.smithi090.stdout:52: oid 25 version 16 is already newer than 14 2023-01-15T14:28:40.546 INFO:tasks.rados.rados.0.smithi090.stdout:52: oid 25 version 16 is already newer than 13 2023-01-15T14:28:40.546 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 16 (ObjNum 50 snap 0 seq_num 50) dirty exists 2023-01-15T14:28:40.546 INFO:tasks.rados.rados.0.smithi090.stdout:52: left oid 25 (ObjNum 50 snap 0 seq_num 50) 2023-01-15T14:28:40.546 INFO:tasks.rados.rados.0.smithi090.stdout:52: done (5 left) 2023-01-15T14:28:40.546 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:28:40.546 INFO:tasks.rados.rados.0.smithi090.stdout:48: finishing write tid 3 to smithi09025264-48 2023-01-15T14:28:40.546 INFO:tasks.rados.rados.0.smithi090.stdout:48: finishing write tid 4 to smithi09025264-48 2023-01-15T14:28:40.546 INFO:tasks.rados.rados.0.smithi090.stdout:48: oid 48 updating version 0 to 11 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout:48: oid 48 version 11 is already newer than 10 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout:48: oid 48 updating version 11 to 12 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 12 (ObjNum 47 snap 0 seq_num 47) dirty exists 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout:48: done (4 left) 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout:43: finishing write tid 4 to smithi09025264-43 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout:43: oid 43 updating version 0 to 5 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout:43: oid 43 updating version 5 to 6 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout:43: oid 43 version 6 is already newer than 4 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2023-01-15T14:28:40.547 INFO:tasks.rados.rados.0.smithi090.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2023-01-15T14:28:40.548 INFO:tasks.rados.rados.0.smithi090.stdout:43: done (3 left) 2023-01-15T14:28:40.548 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:28:40.548 INFO:tasks.rados.rados.0.smithi090.stdout:49: finishing write tid 3 to smithi09025264-49 2023-01-15T14:28:40.548 INFO:tasks.rados.rados.0.smithi090.stdout:49: finishing write tid 4 to smithi09025264-49 2023-01-15T14:28:40.548 INFO:tasks.rados.rados.0.smithi090.stdout:49: oid 49 updating version 0 to 11 2023-01-15T14:28:40.548 INFO:tasks.rados.rados.0.smithi090.stdout:49: oid 49 updating version 11 to 12 2023-01-15T14:28:40.548 INFO:tasks.rados.rados.0.smithi090.stdout:49: oid 49 version 12 is already newer than 10 2023-01-15T14:28:40.548 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2023-01-15T14:28:40.548 INFO:tasks.rados.rados.0.smithi090.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2023-01-15T14:28:40.548 INFO:tasks.rados.rados.0.smithi090.stdout:49: done (2 left) 2023-01-15T14:28:40.548 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout:54: finishing write tid 3 to smithi09025264-21 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout:50: finishing write tid 3 to smithi09025264-50 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout:54: finishing write tid 4 to smithi09025264-21 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout:54: oid 21 updating version 0 to 13 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout:54: oid 21 updating version 13 to 15 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout:54: oid 21 version 15 is already newer than 14 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 15 (ObjNum 51 snap 0 seq_num 51) dirty exists 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout:54: left oid 21 (ObjNum 51 snap 0 seq_num 51) 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout:54: done (1 left) 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout:50: finishing write tid 4 to smithi09025264-50 2023-01-15T14:28:40.549 INFO:tasks.rados.rados.0.smithi090.stdout:50: oid 50 updating version 0 to 11 2023-01-15T14:28:40.550 INFO:tasks.rados.rados.0.smithi090.stdout:50: oid 50 version 11 is already newer than 10 2023-01-15T14:28:40.550 INFO:tasks.rados.rados.0.smithi090.stdout:50: oid 50 updating version 11 to 12 2023-01-15T14:28:40.550 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2023-01-15T14:28:40.550 INFO:tasks.rados.rados.0.smithi090.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2023-01-15T14:28:40.550 INFO:tasks.rados.rados.0.smithi090.stdout:50: done (0 left) 2023-01-15T14:28:40.550 INFO:tasks.rados.rados.0.smithi090.stdout:55: done (0 left) 2023-01-15T14:28:40.550 INFO:tasks.rados.rados.0.smithi090.stdout:56: delete oid 34 current snap is 1 2023-01-15T14:28:40.550 INFO:tasks.rados.rados.0.smithi090.stdout:56: done (0 left) 2023-01-15T14:28:40.550 INFO:tasks.rados.rados.0.smithi090.stdout:57: snap_create 2023-01-15T14:28:40.550 INFO:tasks.rados.rados.0.smithi090.stdout:57: done (0 left) 2023-01-15T14:28:40.550 INFO:tasks.rados.rados.0.smithi090.stdout:58: write oid 33 current snap is 2 2023-01-15T14:28:40.551 INFO:tasks.rados.rados.0.smithi090.stdout:58: seq_num 53 ranges {490771=686315,1885075=406346} 2023-01-15T14:28:40.551 INFO:tasks.rados.rados.0.smithi090.stdout:58: writing smithi09025264-33 from 490771 to 1177086 tid 1 2023-01-15T14:28:40.551 INFO:tasks.rados.rados.0.smithi090.stdout:58: writing smithi09025264-33 from 1885075 to 2291421 tid 2 2023-01-15T14:28:40.551 INFO:tasks.rados.rados.0.smithi090.stdout:59: delete oid 3 current snap is 2 2023-01-15T14:28:40.551 INFO:tasks.rados.rados.0.smithi090.stdout:59: done (1 left) 2023-01-15T14:28:40.551 INFO:tasks.rados.rados.0.smithi090.stdout:60: write oid 12 current snap is 2 2023-01-15T14:28:40.551 INFO:tasks.rados.rados.0.smithi090.stdout:60: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2023-01-15T14:28:40.551 INFO:tasks.rados.rados.0.smithi090.stdout:60: writing smithi09025264-12 from 524385 to 1077551 tid 1 2023-01-15T14:28:40.551 INFO:tasks.rados.rados.0.smithi090.stdout:60: writing smithi09025264-12 from 1602039 to 2012054 tid 2 2023-01-15T14:28:40.551 INFO:tasks.rados.rados.0.smithi090.stdout:60: writing smithi09025264-12 from 2577441 to 2924385 tid 3 2023-01-15T14:28:40.551 INFO:tasks.rados.rados.0.smithi090.stdout:61: rollback oid 47 current snap is 2 2023-01-15T14:28:40.552 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 47 to 1 2023-01-15T14:28:40.552 INFO:tasks.rados.rados.0.smithi090.stdout:62: snap_remove snap 0 2023-01-15T14:28:40.552 INFO:tasks.rados.rados.0.smithi090.stdout:62: done (3 left) 2023-01-15T14:28:40.552 INFO:tasks.rados.rados.0.smithi090.stdout:63: read oid 26 snap -1 2023-01-15T14:28:40.552 INFO:tasks.rados.rados.0.smithi090.stdout:63: expect (ObjNum 25 snap 0 seq_num 25) 2023-01-15T14:28:40.552 INFO:tasks.rados.rados.0.smithi090.stdout:64: read oid 31 snap -1 2023-01-15T14:28:40.552 INFO:tasks.rados.rados.0.smithi090.stdout:64: expect (ObjNum 30 snap 0 seq_num 30) 2023-01-15T14:28:40.552 INFO:tasks.rados.rados.0.smithi090.stdout:65: delete oid 17 current snap is 2 2023-01-15T14:28:40.552 INFO:tasks.rados.rados.0.smithi090.stdout:58: finishing write tid 1 to smithi09025264-33 2023-01-15T14:28:40.552 INFO:tasks.rados.rados.0.smithi090.stdout:58: finishing write tid 2 to smithi09025264-33 2023-01-15T14:28:40.552 INFO:tasks.rados.rados.0.smithi090.stdout:58: finishing write tid 3 to smithi09025264-33 2023-01-15T14:28:40.553 INFO:tasks.rados.rados.0.smithi090.stdout:58: finishing write tid 4 to smithi09025264-33 2023-01-15T14:28:40.553 INFO:tasks.rados.rados.0.smithi090.stdout:58: oid 33 updating version 0 to 5 2023-01-15T14:28:40.553 INFO:tasks.rados.rados.0.smithi090.stdout:58: oid 33 updating version 5 to 6 2023-01-15T14:28:40.553 INFO:tasks.rados.rados.0.smithi090.stdout:58: oid 33 updating version 6 to 7 2023-01-15T14:28:40.553 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 7 (ObjNum 53 snap 2 seq_num 53) dirty exists 2023-01-15T14:28:40.553 INFO:tasks.rados.rados.0.smithi090.stdout:58: left oid 33 (ObjNum 53 snap 2 seq_num 53) 2023-01-15T14:28:40.553 INFO:tasks.rados.rados.0.smithi090.stdout:60: finishing write tid 1 to smithi09025264-12 2023-01-15T14:28:40.553 INFO:tasks.rados.rados.0.smithi090.stdout:61: finishing rollback tid 0 to smithi09025264-47 2023-01-15T14:28:40.553 INFO:tasks.rados.rados.0.smithi090.stdout:61: finishing rollback tid 1 to smithi09025264-47 2023-01-15T14:28:40.553 INFO:tasks.rados.rados.0.smithi090.stdout:60: finishing write tid 2 to smithi09025264-12 2023-01-15T14:28:40.553 INFO:tasks.rados.rados.0.smithi090.stdout:60: finishing write tid 3 to smithi09025264-12 2023-01-15T14:28:40.694 INFO:tasks.rados.rados.0.smithi090.stdout:61: finishing rollback tid 2 to smithi09025264-47 2023-01-15T14:28:40.694 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 22 (ObjNum 46 snap 0 seq_num 46) dirty exists 2023-01-15T14:28:40.694 INFO:tasks.rados.rados.0.smithi090.stdout:60: finishing write tid 4 to smithi09025264-12 2023-01-15T14:28:40.694 INFO:tasks.rados.rados.0.smithi090.stdout:60: finishing write tid 5 to smithi09025264-12 2023-01-15T14:28:40.694 INFO:tasks.rados.rados.0.smithi090.stdout:60: oid 12 updating version 0 to 17 2023-01-15T14:28:40.694 INFO:tasks.rados.rados.0.smithi090.stdout:60: oid 12 updating version 17 to 20 2023-01-15T14:28:40.694 INFO:tasks.rados.rados.0.smithi090.stdout:60: oid 12 version 20 is already newer than 18 2023-01-15T14:28:40.694 INFO:tasks.rados.rados.0.smithi090.stdout:60: oid 12 version 20 is already newer than 19 2023-01-15T14:28:40.694 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 20 (ObjNum 55 snap 2 seq_num 55) dirty exists 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout:60: left oid 12 (ObjNum 55 snap 2 seq_num 55) 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout:64: expect (ObjNum 30 snap 0 seq_num 30) 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout:58: done (5 left) 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout:60: done (4 left) 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout:61: done (3 left) 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout:64: done (2 left) 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout:65: done (1 left) 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout:66: copy_from oid 28 from oid 22 current snap is 2 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout:67: snap_create 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout:63: expect (ObjNum 25 snap 0 seq_num 25) 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout:63: done (1 left) 2023-01-15T14:28:40.695 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:40.696 INFO:tasks.rados.rados.0.smithi090.stdout:66: finishing copy_from to smithi09025264-28 2023-01-15T14:28:40.696 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-01-15T14:28:40.696 INFO:tasks.rados.rados.0.smithi090.stdout:66: finishing copy_from racing read to smithi09025264-28 2023-01-15T14:28:40.697 INFO:tasks.rados.rados.0.smithi090.stdout:66: done (0 left) 2023-01-15T14:28:40.697 INFO:tasks.rados.rados.0.smithi090.stdout:67: done (0 left) 2023-01-15T14:28:40.697 INFO:tasks.rados.rados.0.smithi090.stdout:68: snap_remove snap 1 2023-01-15T14:28:40.697 INFO:tasks.rados.rados.0.smithi090.stdout:68: done (0 left) 2023-01-15T14:28:40.697 INFO:tasks.rados.rados.0.smithi090.stdout:69: read oid 23 snap -1 2023-01-15T14:28:40.697 INFO:tasks.rados.rados.0.smithi090.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2023-01-15T14:28:40.697 INFO:tasks.rados.rados.0.smithi090.stdout:70: read oid 24 snap -1 2023-01-15T14:28:40.698 INFO:tasks.rados.rados.0.smithi090.stdout:70: expect (ObjNum 23 snap 0 seq_num 23) 2023-01-15T14:28:40.698 INFO:tasks.rados.rados.0.smithi090.stdout:71: read oid 41 snap -1 2023-01-15T14:28:40.698 INFO:tasks.rados.rados.0.smithi090.stdout:71: expect (ObjNum 40 snap 0 seq_num 40) 2023-01-15T14:28:40.698 INFO:tasks.rados.rados.0.smithi090.stdout:72: snap_remove snap 2 2023-01-15T14:28:40.698 INFO:tasks.rados.rados.0.smithi090.stdout:72: done (3 left) 2023-01-15T14:28:40.698 INFO:tasks.rados.rados.0.smithi090.stdout:73: write oid 47 current snap is 3 2023-01-15T14:28:40.698 INFO:tasks.rados.rados.0.smithi090.stdout:73: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2023-01-15T14:28:40.699 INFO:tasks.rados.rados.0.smithi090.stdout:73: writing smithi09025264-47 from 557999 to 1261663 tid 1 2023-01-15T14:28:40.699 INFO:tasks.rados.rados.0.smithi090.stdout:73: writing smithi09025264-47 from 1719003 to 2132687 tid 2 2023-01-15T14:28:40.699 INFO:tasks.rados.rados.0.smithi090.stdout:73: writing smithi09025264-47 from 2632287 to 2957999 tid 3 2023-01-15T14:28:40.699 INFO:tasks.rados.rados.0.smithi090.stdout:74: copy_from oid 48 from oid 25 current snap is 3 2023-01-15T14:28:40.699 INFO:tasks.rados.rados.0.smithi090.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2023-01-15T14:28:40.699 INFO:tasks.rados.rados.0.smithi090.stdout:71: expect (ObjNum 40 snap 0 seq_num 40) 2023-01-15T14:28:40.699 INFO:tasks.rados.rados.0.smithi090.stdout:70: expect (ObjNum 23 snap 0 seq_num 23) 2023-01-15T14:28:40.699 INFO:tasks.rados.rados.0.smithi090.stdout:73: finishing write tid 1 to smithi09025264-47 2023-01-15T14:28:40.700 INFO:tasks.rados.rados.0.smithi090.stdout:73: finishing write tid 2 to smithi09025264-47 2023-01-15T14:28:40.700 INFO:tasks.rados.rados.0.smithi090.stdout:73: finishing write tid 3 to smithi09025264-47 2023-01-15T14:28:40.700 INFO:tasks.rados.rados.0.smithi090.stdout:69: done (4 left) 2023-01-15T14:28:40.700 INFO:tasks.rados.rados.0.smithi090.stdout:70: done (3 left) 2023-01-15T14:28:40.700 INFO:tasks.rados.rados.0.smithi090.stdout:71: done (2 left) 2023-01-15T14:28:40.700 INFO:tasks.rados.rados.0.smithi090.stdout:75: copy_from oid 24 from oid 1 current snap is 3 2023-01-15T14:28:40.700 INFO:tasks.rados.rados.0.smithi090.stdout:76: snap_create 2023-01-15T14:28:40.701 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:28:40.701 INFO:tasks.rados.rados.0.smithi090.stdout:73: finishing write tid 4 to smithi09025264-47 2023-01-15T14:28:40.701 INFO:tasks.rados.rados.0.smithi090.stdout:73: finishing write tid 5 to smithi09025264-47 2023-01-15T14:28:40.701 INFO:tasks.rados.rados.0.smithi090.stdout:73: oid 47 updating version 0 to 34 2023-01-15T14:28:40.701 INFO:tasks.rados.rados.0.smithi090.stdout:73: oid 47 version 34 is already newer than 33 2023-01-15T14:28:40.701 INFO:tasks.rados.rados.0.smithi090.stdout:73: oid 47 version 34 is already newer than 31 2023-01-15T14:28:40.701 INFO:tasks.rados.rados.0.smithi090.stdout:73: oid 47 version 34 is already newer than 32 2023-01-15T14:28:40.702 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 34 (ObjNum 57 snap 3 seq_num 57) dirty exists 2023-01-15T14:28:40.702 INFO:tasks.rados.rados.0.smithi090.stdout:73: left oid 47 (ObjNum 57 snap 3 seq_num 57) 2023-01-15T14:28:40.702 INFO:tasks.rados.rados.0.smithi090.stdout:73: done (2 left) 2023-01-15T14:28:40.702 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:28:40.702 INFO:tasks.rados.rados.0.smithi090.stdout:74: finishing copy_from to smithi09025264-48 2023-01-15T14:28:40.702 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 16 (ObjNum 50 snap 0 seq_num 50) dirty exists 2023-01-15T14:28:40.702 INFO:tasks.rados.rados.0.smithi090.stdout:74: finishing copy_from racing read to smithi09025264-48 2023-01-15T14:28:40.703 INFO:tasks.rados.rados.0.smithi090.stdout:74: done (1 left) 2023-01-15T14:28:40.703 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:40.703 INFO:tasks.rados.rados.0.smithi090.stdout:75: finishing copy_from to smithi09025264-24 2023-01-15T14:28:40.703 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 6 (ObjNum 0 snap 0 seq_num 0) dirty exists 2023-01-15T14:28:40.703 INFO:tasks.rados.rados.0.smithi090.stdout:75: finishing copy_from racing read to smithi09025264-24 2023-01-15T14:28:40.703 INFO:tasks.rados.rados.0.smithi090.stdout:75: done (0 left) 2023-01-15T14:28:40.704 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:34.553+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:40.716 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:40.705+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:28:40.721 INFO:tasks.rados.rados.0.smithi090.stdout:76: done (0 left) 2023-01-15T14:28:40.854 INFO:tasks.rados.rados.0.smithi090.stdout:77: copy_from oid 22 from oid 49 current snap is 4 2023-01-15T14:28:40.855 INFO:tasks.rados.rados.0.smithi090.stdout:78: read oid 39 snap 3 2023-01-15T14:28:40.855 INFO:tasks.rados.rados.0.smithi090.stdout:78: expect (ObjNum 38 snap 0 seq_num 38) 2023-01-15T14:28:40.855 INFO:tasks.rados.rados.0.smithi090.stdout:79: write oid 30 current snap is 4 2023-01-15T14:28:40.855 INFO:tasks.rados.rados.0.smithi090.stdout:79: seq_num 58 ranges {574806=778913,2061132=557342} 2023-01-15T14:28:40.855 INFO:tasks.rados.rados.0.smithi090.stdout:79: writing smithi09025264-30 from 574806 to 1353719 tid 1 2023-01-15T14:28:40.856 INFO:tasks.rados.rados.0.smithi090.stdout:79: writing smithi09025264-30 from 2061132 to 2618474 tid 2 2023-01-15T14:28:40.856 INFO:tasks.rados.rados.0.smithi090.stdout:80: write (excl) oid 29 current snap is 4 2023-01-15T14:28:40.856 INFO:tasks.rados.rados.0.smithi090.stdout:80: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2023-01-15T14:28:40.856 INFO:tasks.rados.rados.0.smithi090.stdout:80: writing smithi09025264-29 from 591613 to 1045775 tid 1 2023-01-15T14:28:40.856 INFO:tasks.rados.rados.0.smithi090.stdout:80: writing smithi09025264-29 from 1719614 to 2136967 tid 2 2023-01-15T14:28:40.857 INFO:tasks.rados.rados.0.smithi090.stdout:80: writing smithi09025264-29 from 2570780 to 2991613 tid 3 2023-01-15T14:28:40.857 INFO:tasks.rados.rados.0.smithi090.stdout:81: read oid 36 snap 3 2023-01-15T14:28:40.857 INFO:tasks.rados.rados.0.smithi090.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2023-01-15T14:28:40.857 INFO:tasks.rados.rados.0.smithi090.stdout:82: copy_from oid 42 from oid 28 current snap is 4 2023-01-15T14:28:40.857 INFO:tasks.rados.rados.0.smithi090.stdout:83: rollback oid 6 current snap is 4 2023-01-15T14:28:40.857 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 6 to 3 2023-01-15T14:28:40.858 INFO:tasks.rados.rados.0.smithi090.stdout:84: read oid 5 snap -1 2023-01-15T14:28:40.858 INFO:tasks.rados.rados.0.smithi090.stdout:84: expect (ObjNum 4 snap 0 seq_num 4) 2023-01-15T14:28:40.858 INFO:tasks.rados.rados.0.smithi090.stdout:85: read oid 50 snap -1 2023-01-15T14:28:40.858 INFO:tasks.rados.rados.0.smithi090.stdout:85: expect (ObjNum 49 snap 0 seq_num 49) 2023-01-15T14:28:40.858 INFO:tasks.rados.rados.0.smithi090.stdout:86: read oid 34 snap 3 2023-01-15T14:28:40.858 INFO:tasks.rados.rados.0.smithi090.stdout:86: expect deleted 2023-01-15T14:28:40.859 INFO:tasks.rados.rados.0.smithi090.stdout:87: read oid 32 snap -1 2023-01-15T14:28:40.859 INFO:tasks.rados.rados.0.smithi090.stdout:87: expect (ObjNum 31 snap 0 seq_num 31) 2023-01-15T14:28:40.859 INFO:tasks.rados.rados.0.smithi090.stdout:88: copy_from oid 24 from oid 35 current snap is 4 2023-01-15T14:28:40.859 INFO:tasks.rados.rados.0.smithi090.stdout:89: read oid 10 snap -1 2023-01-15T14:28:40.859 INFO:tasks.rados.rados.0.smithi090.stdout:89: expect (ObjNum 9 snap 0 seq_num 9) 2023-01-15T14:28:40.859 INFO:tasks.rados.rados.0.smithi090.stdout:90: read oid 48 snap -1 2023-01-15T14:28:40.860 INFO:tasks.rados.rados.0.smithi090.stdout:90: expect (ObjNum 50 snap 0 seq_num 50) 2023-01-15T14:28:40.860 INFO:tasks.rados.rados.0.smithi090.stdout:91: write oid 19 current snap is 4 2023-01-15T14:28:40.860 INFO:tasks.rados.rados.0.smithi090.stdout:91: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2023-01-15T14:28:40.860 INFO:tasks.rados.rados.0.smithi090.stdout:91: writing smithi09025264-19 from 608420 to 1137831 tid 1 2023-01-15T14:28:40.860 INFO:tasks.rados.rados.0.smithi090.stdout:91: writing smithi09025264-19 from 1778096 to 2339107 tid 2 2023-01-15T14:28:40.861 INFO:tasks.rados.rados.0.smithi090.stdout:91: writing smithi09025264-19 from 2881850 to 3008420 tid 3 2023-01-15T14:28:40.861 INFO:tasks.rados.rados.0.smithi090.stdout:92: read oid 47 snap -1 2023-01-15T14:28:40.861 INFO:tasks.rados.rados.0.smithi090.stdout:92: expect (ObjNum 57 snap 3 seq_num 57) 2023-01-15T14:28:40.861 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:40.861 INFO:tasks.rados.rados.0.smithi090.stdout:78: expect (ObjNum 38 snap 0 seq_num 38) 2023-01-15T14:28:40.862 INFO:tasks.rados.rados.0.smithi090.stdout:89: expect (ObjNum 9 snap 0 seq_num 9) 2023-01-15T14:28:40.862 INFO:tasks.rados.rados.0.smithi090.stdout:80: finishing write tid 1 to smithi09025264-29 2023-01-15T14:28:40.862 INFO:tasks.rados.rados.0.smithi090.stdout:78: done (15 left) 2023-01-15T14:28:40.862 INFO:tasks.rados.rados.0.smithi090.stdout:86: done (14 left) 2023-01-15T14:28:40.862 INFO:tasks.rados.rados.0.smithi090.stdout:89: done (13 left) 2023-01-15T14:28:40.863 INFO:tasks.rados.rados.0.smithi090.stdout:93: write (excl) oid 44 current snap is 4 2023-01-15T14:28:40.863 INFO:tasks.rados.rados.0.smithi090.stdout:93: seq_num 61 ranges {625227=721013,1952931=704669} 2023-01-15T14:28:40.863 INFO:tasks.rados.rados.0.smithi090.stdout:93: writing smithi09025264-44 from 625227 to 1346240 tid 1 2023-01-15T14:28:40.863 INFO:tasks.rados.rados.0.smithi090.stdout:93: writing smithi09025264-44 from 1952931 to 2657600 tid 2 2023-01-15T14:28:40.863 INFO:tasks.rados.rados.0.smithi090.stdout:94: write oid 14 current snap is 4 2023-01-15T14:28:40.864 INFO:tasks.rados.rados.0.smithi090.stdout:94: seq_num 62 ranges {642034=796262,1895060=564680} 2023-01-15T14:28:40.864 INFO:tasks.rados.rados.0.smithi090.stdout:94: writing smithi09025264-14 from 642034 to 1438296 tid 1 2023-01-15T14:28:40.864 INFO:tasks.rados.rados.0.smithi090.stdout:94: writing smithi09025264-14 from 1895060 to 2459740 tid 2 2023-01-15T14:28:40.864 INFO:tasks.rados.rados.0.smithi090.stdout:95: read oid 41 snap -1 2023-01-15T14:28:40.864 INFO:tasks.rados.rados.0.smithi090.stdout:95: expect (ObjNum 40 snap 0 seq_num 40) 2023-01-15T14:28:40.865 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:28:40.865 INFO:tasks.rados.rados.0.smithi090.stdout:84: expect (ObjNum 4 snap 0 seq_num 4) 2023-01-15T14:28:40.865 INFO:tasks.rados.rados.0.smithi090.stdout:84: done (15 left) 2023-01-15T14:28:40.865 INFO:tasks.rados.rados.0.smithi090.stdout:96: copy_from oid 31 from oid 8 current snap is 4 2023-01-15T14:28:40.865 INFO:tasks.rados.rados.0.smithi090.stdout:87: expect (ObjNum 31 snap 0 seq_num 31) 2023-01-15T14:28:40.866 INFO:tasks.rados.rados.0.smithi090.stdout:88: finishing copy_from to smithi09025264-24 2023-01-15T14:28:40.866 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-01-15T14:28:40.866 INFO:tasks.rados.rados.0.smithi090.stdout:88: finishing copy_from racing read to smithi09025264-24 2023-01-15T14:28:40.866 INFO:tasks.rados.rados.0.smithi090.stdout:87: done (15 left) 2023-01-15T14:28:40.866 INFO:tasks.rados.rados.0.smithi090.stdout:88: done (14 left) 2023-01-15T14:28:40.866 INFO:tasks.rados.rados.0.smithi090.stdout:97: copy_from oid 21 from oid 10 current snap is 4 2023-01-15T14:28:40.867 INFO:tasks.rados.rados.0.smithi090.stdout:93: finishing write tid 1 to smithi09025264-44 2023-01-15T14:28:40.868 INFO:tasks.rados.rados.0.smithi090.stdout:83: finishing rollback tid 0 to smithi09025264-6 2023-01-15T14:28:40.868 INFO:tasks.rados.rados.0.smithi090.stdout:83: finishing rollback tid 1 to smithi09025264-6 2023-01-15T14:28:40.868 INFO:tasks.rados.rados.0.smithi090.stdout:83: finishing rollback tid 2 to smithi09025264-6 2023-01-15T14:28:40.868 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 11 (ObjNum 5 snap 0 seq_num 5) dirty exists 2023-01-15T14:28:40.868 INFO:tasks.rados.rados.0.smithi090.stdout:93: finishing write tid 2 to smithi09025264-44 2023-01-15T14:28:40.868 INFO:tasks.rados.rados.0.smithi090.stdout:79: finishing write tid 1 to smithi09025264-30 2023-01-15T14:28:40.868 INFO:tasks.rados.rados.0.smithi090.stdout:90: expect (ObjNum 50 snap 0 seq_num 50) 2023-01-15T14:28:40.868 INFO:tasks.rados.rados.0.smithi090.stdout:83: done (14 left) 2023-01-15T14:28:40.868 INFO:tasks.rados.rados.0.smithi090.stdout:90: done (13 left) 2023-01-15T14:28:40.868 INFO:tasks.rados.rados.0.smithi090.stdout:98: read oid 35 snap -1 2023-01-15T14:28:40.869 INFO:tasks.rados.rados.0.smithi090.stdout:98: expect (ObjNum 34 snap 0 seq_num 34) 2023-01-15T14:28:40.869 INFO:tasks.rados.rados.0.smithi090.stdout:99: rollback oid 26 current snap is 4 2023-01-15T14:28:40.869 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 26 to 3 2023-01-15T14:28:40.869 INFO:tasks.rados.rados.0.smithi090.stdout:79: finishing write tid 2 to smithi09025264-30 2023-01-15T14:28:40.869 INFO:tasks.rados.rados.0.smithi090.stdout:85: expect (ObjNum 49 snap 0 seq_num 49) 2023-01-15T14:28:40.869 INFO:tasks.rados.rados.0.smithi090.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2023-01-15T14:28:40.869 INFO:tasks.rados.rados.0.smithi090.stdout:81: done (14 left) 2023-01-15T14:28:40.869 INFO:tasks.rados.rados.0.smithi090.stdout:85: done (13 left) 2023-01-15T14:28:40.869 INFO:tasks.rados.rados.0.smithi090.stdout:100: snap_create 2023-01-15T14:28:40.869 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 13 2023-01-15T14:28:40.870 INFO:tasks.rados.rados.0.smithi090.stdout:77: finishing copy_from to smithi09025264-22 2023-01-15T14:28:40.870 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2023-01-15T14:28:40.870 INFO:tasks.rados.rados.0.smithi090.stdout:77: finishing copy_from racing read to smithi09025264-22 2023-01-15T14:28:40.870 INFO:tasks.rados.rados.0.smithi090.stdout:92: expect (ObjNum 57 snap 3 seq_num 57) 2023-01-15T14:28:40.870 INFO:tasks.rados.rados.0.smithi090.stdout:77: done (12 left) 2023-01-15T14:28:40.870 INFO:tasks.rados.rados.0.smithi090.stdout:92: done (11 left) 2023-01-15T14:28:40.870 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 11 2023-01-15T14:28:40.870 INFO:tasks.rados.rados.0.smithi090.stdout:93: finishing write tid 3 to smithi09025264-44 2023-01-15T14:28:40.870 INFO:tasks.rados.rados.0.smithi090.stdout:93: finishing write tid 4 to smithi09025264-44 2023-01-15T14:28:40.870 INFO:tasks.rados.rados.0.smithi090.stdout:93: oid 44 updating version 0 to 25 2023-01-15T14:28:40.871 INFO:tasks.rados.rados.0.smithi090.stdout:93: oid 44 updating version 25 to 26 2023-01-15T14:28:40.871 INFO:tasks.rados.rados.0.smithi090.stdout:93: oid 44 version 26 is already newer than 24 2023-01-15T14:28:40.871 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 26 (ObjNum 61 snap 4 seq_num 61) dirty exists 2023-01-15T14:28:40.871 INFO:tasks.rados.rados.0.smithi090.stdout:93: left oid 44 (ObjNum 61 snap 4 seq_num 61) 2023-01-15T14:28:40.871 INFO:tasks.rados.rados.0.smithi090.stdout:80: finishing write tid 2 to smithi09025264-29 2023-01-15T14:28:40.871 INFO:tasks.rados.rados.0.smithi090.stdout:80: finishing write tid 3 to smithi09025264-29 2023-01-15T14:28:40.871 INFO:tasks.rados.rados.0.smithi090.stdout:80: finishing write tid 4 to smithi09025264-29 2023-01-15T14:28:40.871 INFO:tasks.rados.rados.0.smithi090.stdout:93: done (10 left) 2023-01-15T14:28:40.871 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 10 2023-01-15T14:28:40.871 INFO:tasks.rados.rados.0.smithi090.stdout:80: finishing write tid 5 to smithi09025264-29 2023-01-15T14:28:40.871 INFO:tasks.rados.rados.0.smithi090.stdout:80: oid 29 updating version 0 to 13 2023-01-15T14:28:40.872 INFO:tasks.rados.rados.0.smithi090.stdout:80: oid 29 version 13 is already newer than 12 2023-01-15T14:28:40.872 INFO:tasks.rados.rados.0.smithi090.stdout:80: oid 29 updating version 13 to 14 2023-01-15T14:28:40.872 INFO:tasks.rados.rados.0.smithi090.stdout:80: oid 29 version 14 is already newer than 11 2023-01-15T14:28:40.872 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 14 (ObjNum 59 snap 4 seq_num 59) dirty exists 2023-01-15T14:28:40.872 INFO:tasks.rados.rados.0.smithi090.stdout:80: left oid 29 (ObjNum 59 snap 4 seq_num 59) 2023-01-15T14:28:40.872 INFO:tasks.rados.rados.0.smithi090.stdout:79: finishing write tid 3 to smithi09025264-30 2023-01-15T14:28:40.872 INFO:tasks.rados.rados.0.smithi090.stdout:79: finishing write tid 4 to smithi09025264-30 2023-01-15T14:28:40.872 INFO:tasks.rados.rados.0.smithi090.stdout:79: oid 30 updating version 0 to 10 2023-01-15T14:28:40.883 INFO:tasks.rados.rados.0.smithi090.stdout:79: oid 30 version 10 is already newer than 9 2023-01-15T14:28:40.883 INFO:tasks.rados.rados.0.smithi090.stdout:79: oid 30 version 10 is already newer than 8 2023-01-15T14:28:40.883 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 10 (ObjNum 58 snap 4 seq_num 58) dirty exists 2023-01-15T14:28:40.883 INFO:tasks.rados.rados.0.smithi090.stdout:79: left oid 30 (ObjNum 58 snap 4 seq_num 58) 2023-01-15T14:28:40.883 INFO:tasks.rados.rados.0.smithi090.stdout:94: finishing write tid 1 to smithi09025264-14 2023-01-15T14:28:40.883 INFO:tasks.rados.rados.0.smithi090.stdout:79: done (9 left) 2023-01-15T14:28:40.883 INFO:tasks.rados.rados.0.smithi090.stdout:80: done (8 left) 2023-01-15T14:28:40.883 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:28:40.883 INFO:tasks.rados.rados.0.smithi090.stdout:82: finishing copy_from to smithi09025264-42 2023-01-15T14:28:40.884 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 42 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-01-15T14:28:40.884 INFO:tasks.rados.rados.0.smithi090.stdout:82: finishing copy_from racing read to smithi09025264-42 2023-01-15T14:28:40.884 INFO:tasks.rados.rados.0.smithi090.stdout:82: done (7 left) 2023-01-15T14:28:40.884 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:28:40.884 INFO:tasks.rados.rados.0.smithi090.stdout:95: expect (ObjNum 40 snap 0 seq_num 40) 2023-01-15T14:28:40.884 INFO:tasks.rados.rados.0.smithi090.stdout:95: done (6 left) 2023-01-15T14:28:40.884 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:28:40.884 INFO:tasks.rados.rados.0.smithi090.stdout:98: expect (ObjNum 34 snap 0 seq_num 34) 2023-01-15T14:28:40.884 INFO:tasks.rados.rados.0.smithi090.stdout:94: finishing write tid 2 to smithi09025264-14 2023-01-15T14:28:40.884 INFO:tasks.rados.rados.0.smithi090.stdout:94: finishing write tid 3 to smithi09025264-14 2023-01-15T14:28:40.884 INFO:tasks.rados.rados.0.smithi090.stdout:91: finishing write tid 1 to smithi09025264-19 2023-01-15T14:28:40.885 INFO:tasks.rados.rados.0.smithi090.stdout:91: finishing write tid 2 to smithi09025264-19 2023-01-15T14:28:40.885 INFO:tasks.rados.rados.0.smithi090.stdout:91: finishing write tid 3 to smithi09025264-19 2023-01-15T14:28:40.885 INFO:tasks.rados.rados.0.smithi090.stdout:97: finishing copy_from to smithi09025264-21 2023-01-15T14:28:40.885 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 4 (ObjNum 9 snap 0 seq_num 9) dirty exists 2023-01-15T14:28:40.885 INFO:tasks.rados.rados.0.smithi090.stdout:97: finishing copy_from racing read to smithi09025264-21 2023-01-15T14:28:40.885 INFO:tasks.rados.rados.0.smithi090.stdout:99: finishing rollback tid 0 to smithi09025264-26 2023-01-15T14:28:40.885 INFO:tasks.rados.rados.0.smithi090.stdout:99: finishing rollback tid 1 to smithi09025264-26 2023-01-15T14:28:40.885 INFO:tasks.rados.rados.0.smithi090.stdout:94: finishing write tid 4 to smithi09025264-14 2023-01-15T14:28:40.885 INFO:tasks.rados.rados.0.smithi090.stdout:94: oid 14 updating version 0 to 15 2023-01-15T14:28:40.885 INFO:tasks.rados.rados.0.smithi090.stdout:94: oid 14 version 15 is already newer than 14 2023-01-15T14:28:40.886 INFO:tasks.rados.rados.0.smithi090.stdout:94: oid 14 updating version 15 to 16 2023-01-15T14:28:40.886 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 16 (ObjNum 62 snap 4 seq_num 62) dirty exists 2023-01-15T14:28:40.886 INFO:tasks.rados.rados.0.smithi090.stdout:94: left oid 14 (ObjNum 62 snap 4 seq_num 62) 2023-01-15T14:28:40.886 INFO:tasks.rados.rados.0.smithi090.stdout:91: finishing write tid 4 to smithi09025264-19 2023-01-15T14:28:40.886 INFO:tasks.rados.rados.0.smithi090.stdout:91: finishing write tid 5 to smithi09025264-19 2023-01-15T14:28:40.886 INFO:tasks.rados.rados.0.smithi090.stdout:91: oid 19 updating version 0 to 15 2023-01-15T14:28:40.886 INFO:tasks.rados.rados.0.smithi090.stdout:91: oid 19 version 15 is already newer than 14 2023-01-15T14:28:40.886 INFO:tasks.rados.rados.0.smithi090.stdout:91: oid 19 updating version 15 to 16 2023-01-15T14:28:40.886 INFO:tasks.rados.rados.0.smithi090.stdout:91: oid 19 updating version 16 to 17 2023-01-15T14:28:40.886 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 17 (ObjNum 60 snap 4 seq_num 60) dirty exists 2023-01-15T14:28:40.886 INFO:tasks.rados.rados.0.smithi090.stdout:91: left oid 19 (ObjNum 60 snap 4 seq_num 60) 2023-01-15T14:28:40.887 INFO:tasks.rados.rados.0.smithi090.stdout:91: done (5 left) 2023-01-15T14:28:40.887 INFO:tasks.rados.rados.0.smithi090.stdout:94: done (4 left) 2023-01-15T14:28:40.887 INFO:tasks.rados.rados.0.smithi090.stdout:97: done (3 left) 2023-01-15T14:28:40.887 INFO:tasks.rados.rados.0.smithi090.stdout:98: done (2 left) 2023-01-15T14:28:40.887 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:28:40.887 INFO:tasks.rados.rados.0.smithi090.stdout:96: finishing copy_from to smithi09025264-31 2023-01-15T14:28:40.887 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2023-01-15T14:28:40.887 INFO:tasks.rados.rados.0.smithi090.stdout:96: finishing copy_from racing read to smithi09025264-31 2023-01-15T14:28:40.887 INFO:tasks.rados.rados.0.smithi090.stdout:96: done (1 left) 2023-01-15T14:28:40.888 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:40.888 INFO:tasks.rados.rados.0.smithi090.stdout:99: finishing rollback tid 2 to smithi09025264-26 2023-01-15T14:28:40.888 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 38 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-01-15T14:28:40.888 INFO:tasks.rados.rados.0.smithi090.stdout:99: done (0 left) 2023-01-15T14:28:40.889 INFO:tasks.rados.rados.0.smithi090.stdout:100: done (0 left) 2023-01-15T14:28:40.889 INFO:tasks.rados.rados.0.smithi090.stdout:101: rollback oid 10 current snap is 5 2023-01-15T14:28:40.889 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 10 to 4 2023-01-15T14:28:40.889 INFO:tasks.rados.rados.0.smithi090.stdout:102: read oid 12 snap -1 2023-01-15T14:28:40.889 INFO:tasks.rados.rados.0.smithi090.stdout:102: expect (ObjNum 55 snap 2 seq_num 55) 2023-01-15T14:28:40.889 INFO:tasks.rados.rados.0.smithi090.stdout:103: snap_create 2023-01-15T14:28:40.889 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:28:40.889 INFO:tasks.rados.rados.0.smithi090.stdout:101: finishing rollback tid 0 to smithi09025264-10 2023-01-15T14:28:40.889 INFO:tasks.rados.rados.0.smithi090.stdout:102: expect (ObjNum 55 snap 2 seq_num 55) 2023-01-15T14:28:40.889 INFO:tasks.rados.rados.0.smithi090.stdout:101: finishing rollback tid 1 to smithi09025264-10 2023-01-15T14:28:40.889 INFO:tasks.rados.rados.0.smithi090.stdout:102: done (1 left) 2023-01-15T14:28:40.890 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:40.890 INFO:tasks.rados.rados.0.smithi090.stdout:101: finishing rollback tid 2 to smithi09025264-10 2023-01-15T14:28:40.890 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 37 (ObjNum 9 snap 0 seq_num 9) dirty exists 2023-01-15T14:28:40.890 INFO:tasks.rados.rados.0.smithi090.stdout:101: done (0 left) 2023-01-15T14:28:40.890 INFO:tasks.rados.rados.0.smithi090.stdout:103: done (0 left) 2023-01-15T14:28:40.890 INFO:tasks.rados.rados.0.smithi090.stdout:104: write (excl) oid 43 current snap is 6 2023-01-15T14:28:40.890 INFO:tasks.rados.rados.0.smithi090.stdout:104: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2023-01-15T14:28:40.890 INFO:tasks.rados.rados.0.smithi090.stdout:104: writing smithi09025264-43 from 658841 to 1130352 tid 1 2023-01-15T14:28:40.890 INFO:tasks.rados.rados.0.smithi090.stdout:104: writing smithi09025264-43 from 1553542 to 2261880 tid 2 2023-01-15T14:28:40.890 INFO:tasks.rados.rados.0.smithi090.stdout:104: writing smithi09025264-43 from 2964119 to 3058841 tid 3 2023-01-15T14:28:40.891 INFO:tasks.rados.rados.0.smithi090.stdout:105: copy_from oid 30 from oid 38 current snap is 6 2023-01-15T14:28:40.891 INFO:tasks.rados.rados.0.smithi090.stdout:106: write oid 12 current snap is 6 2023-01-15T14:28:40.891 INFO:tasks.rados.rados.0.smithi090.stdout:106: seq_num 64 ranges {675648=546760,2012024=568349} 2023-01-15T14:28:40.891 INFO:tasks.rados.rados.0.smithi090.stdout:106: writing smithi09025264-12 from 675648 to 1222408 tid 1 2023-01-15T14:28:41.015 INFO:tasks.rados.rados.0.smithi090.stdout:106: writing smithi09025264-12 from 2012024 to 2580373 tid 2 2023-01-15T14:28:41.015 INFO:tasks.rados.rados.0.smithi090.stdout:107: copy_from oid 25 from oid 32 current snap is 6 2023-01-15T14:28:41.015 INFO:tasks.rados.rados.0.smithi090.stdout:108: snap_remove snap 4 2023-01-15T14:28:41.015 INFO:tasks.rados.rados.0.smithi090.stdout:108: done (4 left) 2023-01-15T14:28:41.015 INFO:tasks.rados.rados.0.smithi090.stdout:109: read oid 47 snap -1 2023-01-15T14:28:41.015 INFO:tasks.rados.rados.0.smithi090.stdout:109: expect (ObjNum 57 snap 3 seq_num 57) 2023-01-15T14:28:41.015 INFO:tasks.rados.rados.0.smithi090.stdout:110: snap_remove snap 3 2023-01-15T14:28:41.015 INFO:tasks.rados.rados.0.smithi090.stdout:110: done (5 left) 2023-01-15T14:28:41.016 INFO:tasks.rados.rados.0.smithi090.stdout:111: delete oid 28 current snap is 6 2023-01-15T14:28:41.016 INFO:tasks.rados.rados.0.smithi090.stdout:104: finishing write tid 1 to smithi09025264-43 2023-01-15T14:28:41.016 INFO:tasks.rados.rados.0.smithi090.stdout:106: finishing write tid 1 to smithi09025264-12 2023-01-15T14:28:41.016 INFO:tasks.rados.rados.0.smithi090.stdout:107: finishing copy_from to smithi09025264-25 2023-01-15T14:28:41.016 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 12 (ObjNum 31 snap 0 seq_num 31) dirty exists 2023-01-15T14:28:41.016 INFO:tasks.rados.rados.0.smithi090.stdout:107: finishing copy_from racing read to smithi09025264-25 2023-01-15T14:28:41.016 INFO:tasks.rados.rados.0.smithi090.stdout:104: finishing write tid 2 to smithi09025264-43 2023-01-15T14:28:41.019 INFO:tasks.rados.rados.0.smithi090.stdout:104: finishing write tid 3 to smithi09025264-43 2023-01-15T14:28:41.019 INFO:tasks.rados.rados.0.smithi090.stdout:104: finishing write tid 4 to smithi09025264-43 2023-01-15T14:28:41.019 INFO:tasks.rados.rados.0.smithi090.stdout:104: finishing write tid 5 to smithi09025264-43 2023-01-15T14:28:41.019 INFO:tasks.rados.rados.0.smithi090.stdout:104: oid 43 updating version 0 to 13 2023-01-15T14:28:41.019 INFO:tasks.rados.rados.0.smithi090.stdout:104: oid 43 updating version 13 to 14 2023-01-15T14:28:41.019 INFO:tasks.rados.rados.0.smithi090.stdout:104: oid 43 version 14 is already newer than 12 2023-01-15T14:28:41.020 INFO:tasks.rados.rados.0.smithi090.stdout:104: oid 43 updating version 14 to 15 2023-01-15T14:28:41.020 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 15 (ObjNum 63 snap 6 seq_num 63) dirty exists 2023-01-15T14:28:41.020 INFO:tasks.rados.rados.0.smithi090.stdout:104: left oid 43 (ObjNum 63 snap 6 seq_num 63) 2023-01-15T14:28:41.020 INFO:tasks.rados.rados.0.smithi090.stdout:105: finishing copy_from to smithi09025264-30 2023-01-15T14:28:41.020 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-01-15T14:28:41.021 INFO:tasks.rados.rados.0.smithi090.stdout:105: finishing copy_from racing read to smithi09025264-30 2023-01-15T14:28:41.021 INFO:tasks.rados.rados.0.smithi090.stdout:106: finishing write tid 2 to smithi09025264-12 2023-01-15T14:28:41.021 INFO:tasks.rados.rados.0.smithi090.stdout:106: finishing write tid 3 to smithi09025264-12 2023-01-15T14:28:41.021 INFO:tasks.rados.rados.0.smithi090.stdout:106: finishing write tid 4 to smithi09025264-12 2023-01-15T14:28:41.022 INFO:tasks.rados.rados.0.smithi090.stdout:106: oid 12 updating version 0 to 40 2023-01-15T14:28:41.022 INFO:tasks.rados.rados.0.smithi090.stdout:106: oid 12 version 40 is already newer than 39 2023-01-15T14:28:41.022 INFO:tasks.rados.rados.0.smithi090.stdout:106: oid 12 updating version 40 to 41 2023-01-15T14:28:41.022 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 41 (ObjNum 64 snap 6 seq_num 64) dirty exists 2023-01-15T14:28:41.023 INFO:tasks.rados.rados.0.smithi090.stdout:106: left oid 12 (ObjNum 64 snap 6 seq_num 64) 2023-01-15T14:28:41.023 INFO:tasks.rados.rados.0.smithi090.stdout:109: expect (ObjNum 57 snap 3 seq_num 57) 2023-01-15T14:28:41.023 INFO:tasks.rados.rados.0.smithi090.stdout:104: done (5 left) 2023-01-15T14:28:41.023 INFO:tasks.rados.rados.0.smithi090.stdout:105: done (4 left) 2023-01-15T14:28:41.023 INFO:tasks.rados.rados.0.smithi090.stdout:106: done (3 left) 2023-01-15T14:28:41.024 INFO:tasks.rados.rados.0.smithi090.stdout:107: done (2 left) 2023-01-15T14:28:41.024 INFO:tasks.rados.rados.0.smithi090.stdout:109: done (1 left) 2023-01-15T14:28:41.024 INFO:tasks.rados.rados.0.smithi090.stdout:111: done (0 left) 2023-01-15T14:28:41.024 INFO:tasks.rados.rados.0.smithi090.stdout:112: write oid 25 current snap is 6 2023-01-15T14:28:41.024 INFO:tasks.rados.rados.0.smithi090.stdout:112: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2023-01-15T14:28:41.024 INFO:tasks.rados.rados.0.smithi090.stdout:112: writing smithi09025264-25 from 709262 to 1406520 tid 1 2023-01-15T14:28:41.024 INFO:tasks.rados.rados.0.smithi090.stdout:112: writing smithi09025264-25 from 2012635 to 2584653 tid 2 2023-01-15T14:28:41.024 INFO:tasks.rados.rados.0.smithi090.stdout:112: writing smithi09025264-25 from 3046388 to 3109262 tid 3 2023-01-15T14:28:41.025 INFO:tasks.rados.rados.0.smithi090.stdout:113: rollback oid 3 current snap is 6 2023-01-15T14:28:41.025 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 3 to 5 2023-01-15T14:28:41.025 INFO:tasks.rados.rados.0.smithi090.stdout:114: copy_from oid 32 from oid 35 current snap is 6 2023-01-15T14:28:41.026 INFO:tasks.rados.rados.0.smithi090.stdout:115: write oid 1 current snap is 6 2023-01-15T14:28:41.026 INFO:tasks.rados.rados.0.smithi090.stdout:115: seq_num 67 ranges {726069=772507,2071117=715676} 2023-01-15T14:28:41.026 INFO:tasks.rados.rados.0.smithi090.stdout:115: writing smithi09025264-1 from 726069 to 1498576 tid 1 2023-01-15T14:28:41.026 INFO:tasks.rados.rados.0.smithi090.stdout:115: writing smithi09025264-1 from 2071117 to 2786793 tid 2 2023-01-15T14:28:41.026 INFO:tasks.rados.rados.0.smithi090.stdout:116: copy_from oid 29 from oid 24 current snap is 6 2023-01-15T14:28:41.027 INFO:tasks.rados.rados.0.smithi090.stdout:117: write (excl) oid 39 current snap is 6 2023-01-15T14:28:41.027 INFO:tasks.rados.rados.0.smithi090.stdout:117: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2023-01-15T14:28:41.027 INFO:tasks.rados.rados.0.smithi090.stdout:117: writing smithi09025264-39 from 742876 to 1190632 tid 1 2023-01-15T14:28:41.027 INFO:tasks.rados.rados.0.smithi090.stdout:117: writing smithi09025264-39 from 1729599 to 2305286 tid 2 2023-01-15T14:28:41.027 INFO:tasks.rados.rados.0.smithi090.stdout:117: writing smithi09025264-39 from 3101234 to 3142876 tid 3 2023-01-15T14:28:41.027 INFO:tasks.rados.rados.0.smithi090.stdout:118: snap_create 2023-01-15T14:28:41.027 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:28:41.028 INFO:tasks.rados.rados.0.smithi090.stdout:113: finishing rollback tid 1 to smithi09025264-3 2023-01-15T14:28:41.028 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2023-01-15T14:28:41.028 INFO:tasks.rados.rados.0.smithi090.stdout:113: done (5 left) 2023-01-15T14:28:41.028 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:28:41.028 INFO:tasks.rados.rados.0.smithi090.stdout:115: finishing write tid 1 to smithi09025264-1 2023-01-15T14:28:41.028 INFO:tasks.rados.rados.0.smithi090.stdout:115: finishing write tid 2 to smithi09025264-1 2023-01-15T14:28:41.028 INFO:tasks.rados.rados.0.smithi090.stdout:117: finishing write tid 1 to smithi09025264-39 2023-01-15T14:28:41.028 INFO:tasks.rados.rados.0.smithi090.stdout:115: finishing write tid 3 to smithi09025264-1 2023-01-15T14:28:41.028 INFO:tasks.rados.rados.0.smithi090.stdout:114: finishing copy_from to smithi09025264-32 2023-01-15T14:28:41.029 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-01-15T14:28:41.029 INFO:tasks.rados.rados.0.smithi090.stdout:114: finishing copy_from racing read to smithi09025264-32 2023-01-15T14:28:41.029 INFO:tasks.rados.rados.0.smithi090.stdout:114: done (4 left) 2023-01-15T14:28:41.029 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:28:41.029 INFO:tasks.rados.rados.0.smithi090.stdout:115: finishing write tid 4 to smithi09025264-1 2023-01-15T14:28:41.029 INFO:tasks.rados.rados.0.smithi090.stdout:115: oid 1 updating version 0 to 45 2023-01-15T14:28:41.029 INFO:tasks.rados.rados.0.smithi090.stdout:115: oid 1 version 45 is already newer than 44 2023-01-15T14:28:41.029 INFO:tasks.rados.rados.0.smithi090.stdout:115: oid 1 updating version 45 to 46 2023-01-15T14:28:41.029 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 46 (ObjNum 67 snap 6 seq_num 67) dirty exists 2023-01-15T14:28:41.029 INFO:tasks.rados.rados.0.smithi090.stdout:115: left oid 1 (ObjNum 67 snap 6 seq_num 67) 2023-01-15T14:28:41.030 INFO:tasks.rados.rados.0.smithi090.stdout:115: done (3 left) 2023-01-15T14:28:41.030 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:28:41.031 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:40.537+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:41.034 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:41.174 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:28:41.174 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:28:41.174 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:28:41.174 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.174 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4202.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-01-15T14:28:41.174 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:15.887514+0000", 2023-01-15T14:28:41.174 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 24.931974671999999, 2023-01-15T14:28:41.174 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61511010300000002, 2023-01-15T14:28:41.174 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:41.174 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:41.175 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:41.175 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4202", 2023-01-15T14:28:41.175 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/3762191053", 2023-01-15T14:28:41.175 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 115 2023-01-15T14:28:41.176 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.177 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:41.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:41.178 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887514+0000", 2023-01-15T14:28:41.178 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.180 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:41.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887514+0000", 2023-01-15T14:28:41.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:41.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887512+0000", 2023-01-15T14:28:41.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:28:41.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.182 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:41.182 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887682+0000", 2023-01-15T14:28:41.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016936800000000001 2023-01-15T14:28:41.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.182 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:41.182 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887709+0000", 2023-01-15T14:28:41.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7416999999999999e-05 2023-01-15T14:28:41.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:41.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887722+0000", 2023-01-15T14:28:41.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2952999999999999e-05 2023-01-15T14:28:41.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.184 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.398568+0000", 2023-01-15T14:28:41.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51084611000000002 2023-01-15T14:28:41.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:41.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.398781+0000", 2023-01-15T14:28:41.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021276499999999999 2023-01-15T14:28:41.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.398967+0000", 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186541 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.399257+0000", 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028912399999999998 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.399279+0000", 2023-01-15T14:28:41.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2473e-05 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.399471+0000", 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000192201 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401531+0000", 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020596049999999999 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.187 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.402273+0000", 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00074173000000000002 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502430+0000", 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100157149 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502511+0000", 2023-01-15T14:28:41.188 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0821000000000004e-05 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502624+0000", 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011370999999999999 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4202.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-01-15T14:28:41.189 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:15.887742+0000", 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 24.931746934, 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61526019300000001, 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4202", 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/3762191053", 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 116 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:41.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887742+0000", 2023-01-15T14:28:41.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.200 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.200 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.200 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:41.200 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887742+0000", 2023-01-15T14:28:41.200 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887741+0000", 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887749+0000", 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6109999999999996e-06 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887758+0000", 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5459999999999997e-06 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887764+0000", 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2539999999999998e-06 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.399900+0000", 2023-01-15T14:28:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51213585500000003 2023-01-15T14:28:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400017+0000", 2023-01-15T14:28:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000116817 2023-01-15T14:28:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:28:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400204+0000", 2023-01-15T14:28:41.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186684 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400505+0000", 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030162500000000003 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400527+0000", 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1350000000000001e-05 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400612+0000", 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5550000000000006e-05 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401555+0000", 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00094224700000000001 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.205 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502743+0000", 2023-01-15T14:28:41.206 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101188207 2023-01-15T14:28:41.206 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.206 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.206 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.206 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502836+0000", 2023-01-15T14:28:41.206 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.302e-05 2023-01-15T14:28:41.206 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.206 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:41.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502903+0000", 2023-01-15T14:28:41.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7033999999999995e-05 2023-01-15T14:28:41.207 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.207 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:41.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.503002+0000", 2023-01-15T14:28:41.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9287e-05 2023-01-15T14:28:41.207 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.207 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:41.207 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.208 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.208 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4202.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-01-15T14:28:41.336 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:15.887774+0000", 2023-01-15T14:28:41.336 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 24.931714800999998, 2023-01-15T14:28:41.337 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61553897800000001, 2023-01-15T14:28:41.337 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:41.337 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:41.337 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:41.337 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4202", 2023-01-15T14:28:41.337 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/3762191053", 2023-01-15T14:28:41.338 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 117 2023-01-15T14:28:41.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.338 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:41.338 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:41.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887774+0000", 2023-01-15T14:28:41.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:41.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887774+0000", 2023-01-15T14:28:41.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:41.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887773+0000", 2023-01-15T14:28:41.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:41.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.341 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:41.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887780+0000", 2023-01-15T14:28:41.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.08e-06 2023-01-15T14:28:41.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.342 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:41.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887789+0000", 2023-01-15T14:28:41.343 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3869999999999995e-06 2023-01-15T14:28:41.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:41.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887795+0000", 2023-01-15T14:28:41.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0530000000000003e-06 2023-01-15T14:28:41.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400641+0000", 2023-01-15T14:28:41.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51284605900000002 2023-01-15T14:28:41.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:41.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400815+0000", 2023-01-15T14:28:41.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017370299999999999 2023-01-15T14:28:41.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:28:41.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401003+0000", 2023-01-15T14:28:41.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000187875 2023-01-15T14:28:41.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:41.349 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401433+0000", 2023-01-15T14:28:41.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043028800000000002 2023-01-15T14:28:41.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.349 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:41.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401453+0000", 2023-01-15T14:28:41.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.033e-05 2023-01-15T14:28:41.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:41.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401528+0000", 2023-01-15T14:28:41.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5277000000000004e-05 2023-01-15T14:28:41.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:41.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401567+0000", 2023-01-15T14:28:41.494 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8127000000000003e-05 2023-01-15T14:28:41.494 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.494 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.503083+0000", 2023-01-15T14:28:41.494 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10151600099999999 2023-01-15T14:28:41.494 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.494 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.503170+0000", 2023-01-15T14:28:41.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7324000000000006e-05 2023-01-15T14:28:41.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.495 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.495 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:41.495 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.503256+0000", 2023-01-15T14:28:41.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6213999999999998e-05 2023-01-15T14:28:41.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.495 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.495 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:41.495 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.503313+0000", 2023-01-15T14:28:41.496 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6889999999999999e-05 2023-01-15T14:28:41.496 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.496 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:41.496 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.496 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.496 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.496 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4202.0:156 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-01-15T14:28:41.497 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:20.562557+0000", 2023-01-15T14:28:41.497 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 20.256932209999999, 2023-01-15T14:28:41.497 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60806470999999995, 2023-01-15T14:28:41.497 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:41.497 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:41.498 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:41.498 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4202", 2023-01-15T14:28:41.498 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/3762191053", 2023-01-15T14:28:41.498 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 156 2023-01-15T14:28:41.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.498 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:41.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.499 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:41.499 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562557+0000", 2023-01-15T14:28:41.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.499 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.499 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.499 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:41.499 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562557+0000", 2023-01-15T14:28:41.500 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.500 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.500 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.500 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:41.500 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562556+0000", 2023-01-15T14:28:41.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:41.501 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.501 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.501 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:41.501 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562565+0000", 2023-01-15T14:28:41.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0429999999999996e-06 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562574+0000", 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9299999999999992e-06 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562583+0000", 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3340000000000008e-06 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.503 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.503 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.766728+0000", 2023-01-15T14:28:41.503 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20414501700000001 2023-01-15T14:28:41.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.503 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.503 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:41.503 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.766877+0000", 2023-01-15T14:28:41.503 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014928399999999999 2023-01-15T14:28:41.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.503 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.503 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:28:41.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767078+0000", 2023-01-15T14:28:41.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000200984 2023-01-15T14:28:41.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:41.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767363+0000", 2023-01-15T14:28:41.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028497 2023-01-15T14:28:41.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:41.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767383+0000", 2023-01-15T14:28:41.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9947000000000001e-05 2023-01-15T14:28:41.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:41.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767458+0000", 2023-01-15T14:28:41.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4809999999999997e-05 2023-01-15T14:28:41.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:41.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.768111+0000", 2023-01-15T14:28:41.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065325699999999997 2023-01-15T14:28:41.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.506 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.506 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.506 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.868290+0000", 2023-01-15T14:28:41.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100179142 2023-01-15T14:28:41.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.506 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.506 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.506 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170426+0000", 2023-01-15T14:28:41.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30213604500000002 2023-01-15T14:28:41.507 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.507 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.507 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:41.507 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170510+0000", 2023-01-15T14:28:41.507 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3876999999999997e-05 2023-01-15T14:28:41.507 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.507 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.507 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:41.507 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170621+0000", 2023-01-15T14:28:41.507 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011102 2023-01-15T14:28:41.507 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.508 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:41.508 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.637 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4202.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-01-15T14:28:41.637 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:20.562592+0000", 2023-01-15T14:28:41.637 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 20.256896622999999, 2023-01-15T14:28:41.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60826002300000004, 2023-01-15T14:28:41.638 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:41.638 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:41.638 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:41.638 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4202", 2023-01-15T14:28:41.638 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/3762191053", 2023-01-15T14:28:41.639 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 157 2023-01-15T14:28:41.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.639 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:41.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:41.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562592+0000", 2023-01-15T14:28:41.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562592+0000", 2023-01-15T14:28:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:41.661 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562592+0000", 2023-01-15T14:28:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562600+0000", 2023-01-15T14:28:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2670000000000006e-06 2023-01-15T14:28:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562608+0000", 2023-01-15T14:28:41.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3930000000000008e-06 2023-01-15T14:28:41.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:41.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.662745+0000", 2023-01-15T14:28:41.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100136908 2023-01-15T14:28:41.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767494+0000", 2023-01-15T14:28:41.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104748904 2023-01-15T14:28:41.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:41.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767616+0000", 2023-01-15T14:28:41.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121567 2023-01-15T14:28:41.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:28:41.677 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767801+0000", 2023-01-15T14:28:41.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018572700000000001 2023-01-15T14:28:41.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:41.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.768039+0000", 2023-01-15T14:28:41.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023774000000000001 2023-01-15T14:28:41.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.682 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:41.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.768053+0000", 2023-01-15T14:28:41.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4157e-05 2023-01-15T14:28:41.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:41.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.769013+0000", 2023-01-15T14:28:41.684 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000959888 2023-01-15T14:28:41.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:41.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.769080+0000", 2023-01-15T14:28:41.684 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7323000000000006e-05 2023-01-15T14:28:41.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.868342+0000", 2023-01-15T14:28:41.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099261861000000007 2023-01-15T14:28:41.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170715+0000", 2023-01-15T14:28:41.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30237277200000001 2023-01-15T14:28:41.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:41.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170795+0000", 2023-01-15T14:28:41.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0246000000000004e-05 2023-01-15T14:28:41.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:41.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170852+0000", 2023-01-15T14:28:41.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6894000000000001e-05 2023-01-15T14:28:41.689 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.689 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:41.689 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.689 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:70 3.f 3:fe04f54c:::smithi09025264-18:head [write 1845135~440584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:41.689 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252308+0000", 2023-01-15T14:28:41.689 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.567180609999999, 2023-01-15T14:28:41.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.72689689099999999, 2023-01-15T14:28:41.689 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:41.689 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:41.690 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:41.690 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:41.690 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:41.690 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 70 2023-01-15T14:28:41.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.690 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:41.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:41.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252308+0000", 2023-01-15T14:28:41.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:41.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252308+0000", 2023-01-15T14:28:41.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:41.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252307+0000", 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252613+0000", 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000306929 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252634+0000", 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0276e-05 2023-01-15T14:28:41.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:41.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252647+0000", 2023-01-15T14:28:41.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3114e-05 2023-01-15T14:28:41.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355188+0000", 2023-01-15T14:28:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102540914 2023-01-15T14:28:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355319+0000", 2023-01-15T14:28:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013152699999999999 2023-01-15T14:28:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.657662+0000", 2023-01-15T14:28:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30234277199999998 2023-01-15T14:28:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.657813+0000", 2023-01-15T14:28:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015124600000000001 2023-01-15T14:28:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:28:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658070+0000", 2023-01-15T14:28:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025707600000000002 2023-01-15T14:28:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658459+0000", 2023-01-15T14:28:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038831500000000003 2023-01-15T14:28:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658480+0000", 2023-01-15T14:28:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1610000000000001e-05 2023-01-15T14:28:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658736+0000", 2023-01-15T14:28:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025607899999999999 2023-01-15T14:28:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760108+0000", 2023-01-15T14:28:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10137170199999999 2023-01-15T14:28:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760226+0000", 2023-01-15T14:28:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011748600000000001 2023-01-15T14:28:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978988+0000", 2023-01-15T14:28:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.218762238 2023-01-15T14:28:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.979116+0000", 2023-01-15T14:28:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000128021 2023-01-15T14:28:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.979205+0000", 2023-01-15T14:28:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9248999999999999e-05 2023-01-15T14:28:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:71 3.f 3:fe04f54c:::smithi09025264-18:head [setxattr _header (56) in=7b,truncate 2285719] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:28:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.566823951, 2023-01-15T14:28:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.82719834800000003, 2023-01-15T14:28:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 71 2023-01-15T14:28:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:28:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:28:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252664+0000", 2023-01-15T14:28:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252675+0000", 2023-01-15T14:28:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1296e-05 2023-01-15T14:28:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252697+0000", 2023-01-15T14:28:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1874000000000001e-05 2023-01-15T14:28:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:41.840 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252708+0000", 2023-01-15T14:28:41.840 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.15e-05 2023-01-15T14:28:41.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.840 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.840 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355413+0000", 2023-01-15T14:28:41.840 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10270497100000001 2023-01-15T14:28:41.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.840 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:41.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355622+0000", 2023-01-15T14:28:41.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000208982 2023-01-15T14:28:41.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.758709+0000", 2023-01-15T14:28:41.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40308671800000001 2023-01-15T14:28:41.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:41.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.758958+0000", 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024949 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759375+0000", 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000416208 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759708+0000", 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033326900000000002 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:41.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759744+0000", 2023-01-15T14:28:41.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6477000000000003e-05 2023-01-15T14:28:41.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:41.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759822+0000", 2023-01-15T14:28:41.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7440000000000004e-05 2023-01-15T14:28:41.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760139+0000", 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316996 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760440+0000", 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000300945 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079578+0000", 2023-01-15T14:28:41.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31913862599999998 2023-01-15T14:28:41.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:41.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079709+0000", 2023-01-15T14:28:41.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013108 2023-01-15T14:28:41.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:41.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079863+0000", 2023-01-15T14:28:41.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000153819 2023-01-15T14:28:41.845 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.846 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:41.846 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.846 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:72 3.f 3:fe04f54c:::smithi09025264-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:41.975 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:28:41.975 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.566764614, 2023-01-15T14:28:41.975 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.82782374599999997, 2023-01-15T14:28:41.975 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:41.975 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:28:41.975 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:41.975 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:41.976 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:41.976 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 72 2023-01-15T14:28:41.976 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.976 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:41.976 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.976 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:41.976 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:28:41.976 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.976 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.976 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.977 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:41.977 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:28:41.977 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:41.977 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.977 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.977 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:41.977 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252723+0000", 2023-01-15T14:28:41.977 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:41.978 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.978 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.978 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:41.978 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252730+0000", 2023-01-15T14:28:41.978 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.703e-06 2023-01-15T14:28:41.978 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.978 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.978 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:41.978 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252747+0000", 2023-01-15T14:28:41.978 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6739999999999999e-05 2023-01-15T14:28:41.978 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.979 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.979 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:41.979 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252758+0000", 2023-01-15T14:28:41.979 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1389999999999999e-05 2023-01-15T14:28:41.979 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.979 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.979 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.980 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.455798+0000", 2023-01-15T14:28:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20303954199999999 2023-01-15T14:28:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.456092+0000", 2023-01-15T14:28:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029407199999999999 2023-01-15T14:28:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759852+0000", 2023-01-15T14:28:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.303760525 2023-01-15T14:28:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760038+0000", 2023-01-15T14:28:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018584600000000001 2023-01-15T14:28:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080005+0000", 2023-01-15T14:28:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31996668299999997 2023-01-15T14:28:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080286+0000", 2023-01-15T14:28:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028087899999999999 2023-01-15T14:28:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080548+0000", 2023-01-15T14:28:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026241300000000002 2023-01-15T14:28:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.984 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:41.984 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:41.984 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:41.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:41.998 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:74 3.f 3:f8edd34e:::smithi09025264-19:head [write 1903617~398909] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:41.999 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356537+0000", 2023-01-15T14:28:41.999 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.462952067, 2023-01-15T14:28:41.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.72430009100000003, 2023-01-15T14:28:41.999 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:41.999 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:42.000 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:42.000 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:42.000 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:42.000 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 74 2023-01-15T14:28:42.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.001 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:42.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:42.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356537+0000", 2023-01-15T14:28:42.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:42.003 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356537+0000", 2023-01-15T14:28:42.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:42.004 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356535+0000", 2023-01-15T14:28:42.004 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:42.004 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:42.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356854+0000", 2023-01-15T14:28:42.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000318979 2023-01-15T14:28:42.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:42.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356871+0000", 2023-01-15T14:28:42.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6509999999999999e-05 2023-01-15T14:28:42.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:42.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356881+0000", 2023-01-15T14:28:42.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0159999999999999e-05 2023-01-15T14:28:42.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.142 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.142 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457503+0000", 2023-01-15T14:28:42.142 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100622054 2023-01-15T14:28:42.142 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.142 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457677+0000", 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017399200000000001 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.777230+0000", 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.319552742 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:42.143 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.777471+0000", 2023-01-15T14:28:42.144 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024115200000000001 2023-01-15T14:28:42.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.144 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.144 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:28:42.144 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.777842+0000", 2023-01-15T14:28:42.144 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037174199999999999 2023-01-15T14:28:42.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.144 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.144 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:42.144 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778275+0000", 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043252900000000002 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778295+0000", 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9572000000000002e-05 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778511+0000", 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021633700000000001 2023-01-15T14:28:42.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.146 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:42.146 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:28:42.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060300800000000004 2023-01-15T14:28:42.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.146 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:42.146 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079412+0000", 2023-01-15T14:28:42.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30029809899999999 2023-01-15T14:28:42.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:42.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080661+0000", 2023-01-15T14:28:42.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001249497 2023-01-15T14:28:42.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:42.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080737+0000", 2023-01-15T14:28:42.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:28:42.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:42.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080837+0000", 2023-01-15T14:28:42.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9653000000000003e-05 2023-01-15T14:28:42.148 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:42.148 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:42.148 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:42.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.148 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:75 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:42.148 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:28:42.148 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.462592264, 2023-01-15T14:28:42.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92335155400000002, 2023-01-15T14:28:42.149 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:42.149 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:42.149 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:42.149 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:42.149 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:42.149 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 75 2023-01-15T14:28:42.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.149 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:42.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:42.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:28:42.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:42.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:28:42.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:42.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356896+0000", 2023-01-15T14:28:42.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:42.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:42.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356903+0000", 2023-01-15T14:28:42.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4619999999999999e-06 2023-01-15T14:28:42.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:42.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356917+0000", 2023-01-15T14:28:42.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4195999999999999e-05 2023-01-15T14:28:42.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:42.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356926+0000", 2023-01-15T14:28:42.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:28:42.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457743+0000", 2023-01-15T14:28:42.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100817668 2023-01-15T14:28:42.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:42.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457841+0000", 2023-01-15T14:28:42.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7079999999999999e-05 2023-01-15T14:28:42.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778474+0000", 2023-01-15T14:28:42.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32063301799999999 2023-01-15T14:28:42.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:42.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778627+0000", 2023-01-15T14:28:42.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015389599999999999 2023-01-15T14:28:42.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:28:42.163 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778815+0000", 2023-01-15T14:28:42.163 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018726600000000001 2023-01-15T14:28:42.163 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.163 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:42.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779033+0000", 2023-01-15T14:28:42.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021819999999999999 2023-01-15T14:28:42.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:42.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779051+0000", 2023-01-15T14:28:42.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7640000000000001e-05 2023-01-15T14:28:42.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:28:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3590999999999995e-05 2023-01-15T14:28:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779163+0000", 2023-01-15T14:28:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:28:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.081002+0000", 2023-01-15T14:28:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30183937799999999 2023-01-15T14:28:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280043+0000", 2023-01-15T14:28:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19904084699999999 2023-01-15T14:28:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280138+0000", 2023-01-15T14:28:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4915999999999997e-05 2023-01-15T14:28:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280248+0000", 2023-01-15T14:28:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110233 2023-01-15T14:28:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:76 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:28:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.462550655999999, 2023-01-15T14:28:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92420393899999997, 2023-01-15T14:28:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:28:42.169 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:42.169 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:42.169 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:42.169 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 76 2023-01-15T14:28:42.169 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.169 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:42.169 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.170 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:42.170 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:28:42.170 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.170 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.170 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.170 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:42.170 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:28:42.170 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.170 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.170 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.170 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:42.308 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356937+0000", 2023-01-15T14:28:42.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:42.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.308 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:42.308 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356948+0000", 2023-01-15T14:28:42.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.094e-05 2023-01-15T14:28:42.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:42.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356959+0000", 2023-01-15T14:28:42.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1058e-05 2023-01-15T14:28:42.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.310 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.310 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:42.311 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356968+0000", 2023-01-15T14:28:42.311 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5339999999999999e-06 2023-01-15T14:28:42.311 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.311 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.311 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459524+0000", 2023-01-15T14:28:42.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102556147 2023-01-15T14:28:42.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:42.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459662+0000", 2023-01-15T14:28:42.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000137756 2023-01-15T14:28:42.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978621+0000", 2023-01-15T14:28:42.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51895887600000001 2023-01-15T14:28:42.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:42.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978870+0000", 2023-01-15T14:28:42.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024928999999999999 2023-01-15T14:28:42.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280761+0000", 2023-01-15T14:28:42.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30189113400000001 2023-01-15T14:28:42.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:42.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281023+0000", 2023-01-15T14:28:42.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026172800000000003 2023-01-15T14:28:42.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:42.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281142+0000", 2023-01-15T14:28:42.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000119296 2023-01-15T14:28:42.316 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:42.316 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:42.316 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:42.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.316 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:91 3.d 3:bc1c6b44:::smithi09025264-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:42.316 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.416062275, 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.844937151, 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 91 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:42.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:28:42.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:42.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:28:42.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:42.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403425+0000", 2023-01-15T14:28:42.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:42.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:42.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403434+0000", 2023-01-15T14:28:42.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3299999999999999e-06 2023-01-15T14:28:42.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:42.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403446+0000", 2023-01-15T14:28:42.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2176e-05 2023-01-15T14:28:42.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:42.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403453+0000", 2023-01-15T14:28:42.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5830000000000003e-06 2023-01-15T14:28:42.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403654+0000", 2023-01-15T14:28:42.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020136000000000001 2023-01-15T14:28:42.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:42.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403807+0000", 2023-01-15T14:28:42.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015342599999999999 2023-01-15T14:28:42.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907836+0000", 2023-01-15T14:28:42.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50402854699999999 2023-01-15T14:28:42.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:42.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907934+0000", 2023-01-15T14:28:42.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8472000000000002e-05 2023-01-15T14:28:42.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:28:42.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908096+0000", 2023-01-15T14:28:42.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000162139 2023-01-15T14:28:42.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:42.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908335+0000", 2023-01-15T14:28:42.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000238891 2023-01-15T14:28:42.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:42.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908350+0000", 2023-01-15T14:28:42.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4868000000000001e-05 2023-01-15T14:28:42.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:42.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908416+0000", 2023-01-15T14:28:42.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5502999999999999e-05 2023-01-15T14:28:42.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:42.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.952370+0000", 2023-01-15T14:28:42.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.043954663999999997 2023-01-15T14:28:42.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:42.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.008749+0000", 2023-01-15T14:28:42.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.056378696999999998 2023-01-15T14:28:42.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:42.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248164+0000", 2023-01-15T14:28:42.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23941479500000001 2023-01-15T14:28:42.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:42.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248244+0000", 2023-01-15T14:28:42.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9616000000000005e-05 2023-01-15T14:28:42.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:42.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248364+0000", 2023-01-15T14:28:42.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120167 2023-01-15T14:28:42.554 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:42.554 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:42.554 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:42.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.554 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:92 3.d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:42.555 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:28:42.555 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.416026582000001, 2023-01-15T14:28:42.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.84526834699999998, 2023-01-15T14:28:42.555 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:42.555 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:28:42.562 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:42.562 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:42.563 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:42.563 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 92 2023-01-15T14:28:42.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.563 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:42.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:42.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:28:42.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:42.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:28:42.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:42.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403461+0000", 2023-01-15T14:28:42.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:42.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:42.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403466+0000", 2023-01-15T14:28:42.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.34e-06 2023-01-15T14:28:42.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:42.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403476+0000", 2023-01-15T14:28:42.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8530000000000004e-06 2023-01-15T14:28:42.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:42.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403492+0000", 2023-01-15T14:28:42.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6646999999999998e-05 2023-01-15T14:28:42.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501333+0000", 2023-01-15T14:28:42.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097840420999999997 2023-01-15T14:28:42.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:42.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501483+0000", 2023-01-15T14:28:42.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150123 2023-01-15T14:28:42.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.007528+0000", 2023-01-15T14:28:42.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50604486299999996 2023-01-15T14:28:42.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:42.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.007649+0000", 2023-01-15T14:28:42.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001217 2023-01-15T14:28:42.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248441+0000", 2023-01-15T14:28:42.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.240791267 2023-01-15T14:28:42.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:42.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248561+0000", 2023-01-15T14:28:42.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120516 2023-01-15T14:28:42.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:42.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248731+0000", 2023-01-15T14:28:42.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016937000000000001 2023-01-15T14:28:42.570 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:42.570 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:42.570 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:42.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.571 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:107 3.d 3:b2a32959:::smithi09025264-27:head [setxattr _header (56) in=7b,truncate 2436982] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:42.571 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.508410+0000", 2023-01-15T14:28:42.571 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.311078408, 2023-01-15T14:28:42.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63911129799999999, 2023-01-15T14:28:42.571 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:42.572 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:42.572 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:42.572 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:42.572 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:42.572 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 107 2023-01-15T14:28:42.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.572 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:42.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:42.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508410+0000", 2023-01-15T14:28:42.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:42.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508410+0000", 2023-01-15T14:28:42.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:42.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:42.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508409+0000", 2023-01-15T14:28:42.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:42.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:42.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508418+0000", 2023-01-15T14:28:42.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4830000000000005e-06 2023-01-15T14:28:42.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:42.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508436+0000", 2023-01-15T14:28:42.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7419999999999999e-05 2023-01-15T14:28:42.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:42.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508448+0000", 2023-01-15T14:28:42.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2274e-05 2023-01-15T14:28:42.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508766+0000", 2023-01-15T14:28:42.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00031751199999999999 2023-01-15T14:28:42.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:42.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508906+0000", 2023-01-15T14:28:42.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014061999999999999 2023-01-15T14:28:42.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:42.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906456+0000", 2023-01-15T14:28:42.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39754930500000002 2023-01-15T14:28:42.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:42.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906561+0000", 2023-01-15T14:28:42.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000105159 2023-01-15T14:28:42.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:42.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:42.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:28:42.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906729+0000", 2023-01-15T14:28:42.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016813900000000001 2023-01-15T14:28:42.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.008 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:43.008 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906908+0000", 2023-01-15T14:28:43.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000179049 2023-01-15T14:28:43.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:43.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906974+0000", 2023-01-15T14:28:43.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6360999999999997e-05 2023-01-15T14:28:43.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:43.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907147+0000", 2023-01-15T14:28:43.010 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000172991 2023-01-15T14:28:43.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:43.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.952355+0000", 2023-01-15T14:28:43.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.045207642999999999 2023-01-15T14:28:43.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:43.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.008697+0000", 2023-01-15T14:28:43.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.056342148000000002 2023-01-15T14:28:43.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:43.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.147257+0000", 2023-01-15T14:28:43.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.13856006800000001 2023-01-15T14:28:43.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:43.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.147356+0000", 2023-01-15T14:28:43.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9083e-05 2023-01-15T14:28:43.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:43.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.147522+0000", 2023-01-15T14:28:43.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016548999999999999 2023-01-15T14:28:43.019 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:43.019 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:43.019 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:43.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.019 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:108 3.d 3:b2a32959:::smithi09025264-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:43.019 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.508466+0000", 2023-01-15T14:28:43.019 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.311022364999999, 2023-01-15T14:28:43.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.73959223500000004, 2023-01-15T14:28:43.019 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:43.020 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:28:43.020 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:43.020 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:43.020 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:43.020 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 108 2023-01-15T14:28:43.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.020 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:43.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:43.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508466+0000", 2023-01-15T14:28:43.020 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:43.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:43.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508466+0000", 2023-01-15T14:28:43.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:43.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:43.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508465+0000", 2023-01-15T14:28:43.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:43.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:43.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508473+0000", 2023-01-15T14:28:43.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.374e-06 2023-01-15T14:28:43.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:43.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508502+0000", 2023-01-15T14:28:43.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.9709e-05 2023-01-15T14:28:43.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:43.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508513+0000", 2023-01-15T14:28:43.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0200000000000001e-05 2023-01-15T14:28:43.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:43.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508947+0000", 2023-01-15T14:28:43.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043390699999999998 2023-01-15T14:28:43.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:43.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.509045+0000", 2023-01-15T14:28:43.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8400000000000007e-05 2023-01-15T14:28:43.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:43.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907035+0000", 2023-01-15T14:28:43.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39798985799999997 2023-01-15T14:28:43.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:43.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907172+0000", 2023-01-15T14:28:43.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013714599999999999 2023-01-15T14:28:43.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:43.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.247729+0000", 2023-01-15T14:28:43.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.34055681599999998 2023-01-15T14:28:43.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:43.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.247863+0000", 2023-01-15T14:28:43.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013370999999999999 2023-01-15T14:28:43.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:43.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248059+0000", 2023-01-15T14:28:43.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019614900000000001 2023-01-15T14:28:43.034 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:43.034 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:43.034 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:43.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.035 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:191 3.d 3:b3f111b5:::smithi09025264-48:head [setxattr _header (56) in=7b,truncate 2789929] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:43.035 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.289388+0000", 2023-01-15T14:28:43.035 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 13.530100343999999, 2023-01-15T14:28:43.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67091844199999995, 2023-01-15T14:28:43.035 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:43.035 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:43.035 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:43.035 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:43.035 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:43.036 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 191 2023-01-15T14:28:43.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.036 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:43.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.036 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:43.036 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289388+0000", 2023-01-15T14:28:43.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:43.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.037 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:43.037 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289388+0000", 2023-01-15T14:28:43.037 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:43.037 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.037 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:43.037 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289387+0000", 2023-01-15T14:28:43.037 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:43.037 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.172 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:43.173 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289396+0000", 2023-01-15T14:28:43.173 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6000000000000007e-06 2023-01-15T14:28:43.173 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.174 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.174 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:43.174 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289410+0000", 2023-01-15T14:28:43.174 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3933e-05 2023-01-15T14:28:43.174 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.174 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.174 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:43.174 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.389520+0000", 2023-01-15T14:28:43.175 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100110488 2023-01-15T14:28:43.175 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.175 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.175 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:43.175 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.552712+0000", 2023-01-15T14:28:43.175 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.16319123799999999 2023-01-15T14:28:43.176 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.176 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:43.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.552851+0000", 2023-01-15T14:28:43.176 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000139128 2023-01-15T14:28:43.176 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.176 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:43.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.658429+0000", 2023-01-15T14:28:43.176 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10557828399999999 2023-01-15T14:28:43.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:43.177 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.658606+0000", 2023-01-15T14:28:43.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017746 2023-01-15T14:28:43.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.178 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.178 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:28:43.178 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.658814+0000", 2023-01-15T14:28:43.178 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020789199999999999 2023-01-15T14:28:43.178 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.178 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.178 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:43.178 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659030+0000", 2023-01-15T14:28:43.180 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021556 2023-01-15T14:28:43.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:43.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659054+0000", 2023-01-15T14:28:43.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4182999999999999e-05 2023-01-15T14:28:43.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:43.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659131+0000", 2023-01-15T14:28:43.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.729e-05 2023-01-15T14:28:43.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.182 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:43.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659442+0000", 2023-01-15T14:28:43.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310575 2023-01-15T14:28:43.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.349 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:43.349 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.817131+0000", 2023-01-15T14:28:43.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.15768874399999999 2023-01-15T14:28:43.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:43.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.960024+0000", 2023-01-15T14:28:43.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.14289289399999999 2023-01-15T14:28:43.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:43.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.960168+0000", 2023-01-15T14:28:43.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144186 2023-01-15T14:28:43.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:43.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.960307+0000", 2023-01-15T14:28:43.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013917000000000001 2023-01-15T14:28:43.356 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:43.356 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:43.356 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:43.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.356 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:199 3.f 3:f87c0183:::smithi09025264-50:head [setxattr _header (56) in=7b,truncate 2823543] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:43.553 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.491047+0000", 2023-01-15T14:28:43.553 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 13.328441675000001, 2023-01-15T14:28:43.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.619490506, 2023-01-15T14:28:43.553 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:43.553 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:43.553 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:43.553 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:43.553 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 199 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491047+0000", 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491047+0000", 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:43.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:43.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491046+0000", 2023-01-15T14:28:43.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:43.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:43.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491052+0000", 2023-01-15T14:28:43.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0569999999999998e-06 2023-01-15T14:28:43.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:43.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491061+0000", 2023-01-15T14:28:43.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8400000000000001e-06 2023-01-15T14:28:43.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:43.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491068+0000", 2023-01-15T14:28:43.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3929999999999996e-06 2023-01-15T14:28:43.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:43.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.692271+0000", 2023-01-15T14:28:43.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20120313400000001 2023-01-15T14:28:43.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:43.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.692455+0000", 2023-01-15T14:28:43.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018380600000000001 2023-01-15T14:28:43.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:43.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.808238+0000", 2023-01-15T14:28:43.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.115783805 2023-01-15T14:28:43.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:43.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.808361+0000", 2023-01-15T14:28:43.964 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000122476 2023-01-15T14:28:43.964 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.964 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.964 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:28:43.964 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.808731+0000", 2023-01-15T14:28:43.964 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000370596 2023-01-15T14:28:43.964 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.964 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.964 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:43.964 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.809089+0000", 2023-01-15T14:28:43.965 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035795199999999998 2023-01-15T14:28:43.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.965 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:43.965 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.809135+0000", 2023-01-15T14:28:43.965 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.5243e-05 2023-01-15T14:28:43.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.965 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:43.965 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.809271+0000", 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000136494 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.909550+0000", 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100278703 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.910579+0000", 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010287969999999999 2023-01-15T14:28:43.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.967 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:43.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110209+0000", 2023-01-15T14:28:43.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19963028599999999 2023-01-15T14:28:43.967 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.967 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:43.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110352+0000", 2023-01-15T14:28:43.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014284599999999999 2023-01-15T14:28:43.967 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:43.968 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110538+0000", 2023-01-15T14:28:43.968 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185842 2023-01-15T14:28:43.968 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:43.968 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:43.968 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:43.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:43.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:43.968 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:200 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:43.968 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.491107+0000", 2023-01-15T14:28:43.968 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 13.328381609000001, 2023-01-15T14:28:43.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81997309299999999, 2023-01-15T14:28:43.969 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:43.969 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:28:44.179 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:44.179 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:44.179 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:44.179 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 200 2023-01-15T14:28:44.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.179 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:44.179 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.179 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:44.179 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491107+0000", 2023-01-15T14:28:44.179 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:44.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.180 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.180 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:44.180 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491107+0000", 2023-01-15T14:28:44.180 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:44.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.180 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.180 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:44.180 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491106+0000", 2023-01-15T14:28:44.180 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:44.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:44.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491113+0000", 2023-01-15T14:28:44.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9129999999999997e-06 2023-01-15T14:28:44.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:44.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491132+0000", 2023-01-15T14:28:44.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8593999999999998e-05 2023-01-15T14:28:44.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.182 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:44.182 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491146+0000", 2023-01-15T14:28:44.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4036000000000001e-05 2023-01-15T14:28:44.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.182 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:44.182 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.692622+0000", 2023-01-15T14:28:44.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20147609 2023-01-15T14:28:44.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.692862+0000", 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024064600000000001 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.909332+0000", 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.216469364 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:44.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.909470+0000", 2023-01-15T14:28:44.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013843300000000001 2023-01-15T14:28:44.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.184 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.184 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:44.184 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.310780+0000", 2023-01-15T14:28:44.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.401309939 2023-01-15T14:28:44.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.184 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.184 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:44.184 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.310914+0000", 2023-01-15T14:28:44.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013376600000000001 2023-01-15T14:28:44.372 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.372 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:44.373 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.311080+0000", 2023-01-15T14:28:44.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000166405 2023-01-15T14:28:44.373 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:44.373 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:44.373 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:44.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.373 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:305 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 3008420] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:44.373 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.989114+0000", 2023-01-15T14:28:44.373 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 5.830375063, 2023-01-15T14:28:44.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61679676699999997, 2023-01-15T14:28:44.374 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:44.374 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:44.374 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:44.374 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:44.374 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:44.374 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 305 2023-01-15T14:28:44.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.374 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:44.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:44.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989114+0000", 2023-01-15T14:28:44.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:44.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:44.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989114+0000", 2023-01-15T14:28:44.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:44.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:44.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989113+0000", 2023-01-15T14:28:44.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:28:44.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:44.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989120+0000", 2023-01-15T14:28:44.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3679999999999996e-06 2023-01-15T14:28:44.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:44.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989130+0000", 2023-01-15T14:28:44.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0169999999999999e-05 2023-01-15T14:28:44.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:44.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989137+0000", 2023-01-15T14:28:44.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0480000000000001e-06 2023-01-15T14:28:44.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:44.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989277+0000", 2023-01-15T14:28:44.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014001700000000001 2023-01-15T14:28:44.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.378 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.378 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:28:44.378 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989285+0000", 2023-01-15T14:28:44.378 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5299999999999999e-06 2023-01-15T14:28:44.378 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.378 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.378 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:44.378 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.402952+0000", 2023-01-15T14:28:44.378 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.41366716199999998 2023-01-15T14:28:44.378 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.378 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.379 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:44.379 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.403119+0000", 2023-01-15T14:28:44.379 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016668600000000001 2023-01-15T14:28:44.379 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.379 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:28:44.379 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.403502+0000", 2023-01-15T14:28:44.379 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038385699999999998 2023-01-15T14:28:44.379 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.379 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:44.380 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.403876+0000", 2023-01-15T14:28:44.380 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037308799999999999 2023-01-15T14:28:44.380 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.380 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:44.380 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.403918+0000", 2023-01-15T14:28:44.380 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2363000000000001e-05 2023-01-15T14:28:44.380 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.380 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:44.380 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.404030+0000", 2023-01-15T14:28:44.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111826 2023-01-15T14:28:44.381 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.381 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.381 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:28:44.381 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.504533+0000", 2023-01-15T14:28:44.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10050276800000001 2023-01-15T14:28:44.381 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.381 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.381 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:44.381 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.505244+0000", 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071137100000000003 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605718+0000", 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100473689 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605817+0000", 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9666000000000004e-05 2023-01-15T14:28:44.382 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.383 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:44.383 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605911+0000", 2023-01-15T14:28:44.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3256000000000003e-05 2023-01-15T14:28:44.383 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:44.383 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:44.383 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:44.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.383 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:306 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:44.383 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.989160+0000", 2023-01-15T14:28:44.384 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 5.8303286769999998, 2023-01-15T14:28:44.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61707688900000002, 2023-01-15T14:28:44.384 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:28:44.384 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:28:44.384 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:28:44.384 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:28:44.384 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:44.384 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 306 2023-01-15T14:28:44.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.384 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989160+0000", 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989160+0000", 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:28:44.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989159+0000", 2023-01-15T14:28:44.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:44.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:28:44.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989164+0000", 2023-01-15T14:28:44.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3420000000000001e-06 2023-01-15T14:28:44.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:28:44.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989175+0000", 2023-01-15T14:28:44.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1702000000000001e-05 2023-01-15T14:28:44.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:28:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989182+0000", 2023-01-15T14:28:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7939999999999997e-06 2023-01-15T14:28:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.088032+0000", 2023-01-15T14:28:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098849918999999994 2023-01-15T14:28:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:28:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.088042+0000", 2023-01-15T14:28:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.876e-06 2023-01-15T14:28:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605424+0000", 2023-01-15T14:28:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51738228100000005 2023-01-15T14:28:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:44.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605579+0000", 2023-01-15T14:28:44.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015515900000000001 2023-01-15T14:28:44.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:28:44.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605966+0000", 2023-01-15T14:28:44.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038621100000000002 2023-01-15T14:28:44.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:28:44.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.606091+0000", 2023-01-15T14:28:44.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000125499 2023-01-15T14:28:44.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:28:44.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:28:44.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:28:44.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.606237+0000", 2023-01-15T14:28:44.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000145816 2023-01-15T14:28:44.593 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:44.593 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:44.593 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:44.593 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:28:44.593 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:28:44.593 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:44.595 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:28:44.596 INFO:tasks.rados.rados.0.smithi090.stdout:112: finishing write tid 1 to smithi09025264-25 2023-01-15T14:28:44.788 INFO:tasks.rados.rados.0.smithi090.stdout:112: finishing write tid 2 to smithi09025264-25 2023-01-15T14:28:44.788 INFO:tasks.rados.rados.0.smithi090.stdout:112: finishing write tid 3 to smithi09025264-25 2023-01-15T14:28:44.788 INFO:tasks.rados.rados.0.smithi090.stdout:116: finishing copy_from to smithi09025264-29 2023-01-15T14:28:44.788 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-01-15T14:28:44.788 INFO:tasks.rados.rados.0.smithi090.stdout:116: finishing copy_from racing read to smithi09025264-29 2023-01-15T14:28:44.788 INFO:tasks.rados.rados.0.smithi090.stdout:116: done (2 left) 2023-01-15T14:28:44.788 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout:112: finishing write tid 4 to smithi09025264-25 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout:112: finishing write tid 5 to smithi09025264-25 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout:112: oid 25 updating version 0 to 36 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout:112: oid 25 version 36 is already newer than 35 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout:112: oid 25 updating version 36 to 37 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout:112: oid 25 updating version 37 to 38 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 38 (ObjNum 66 snap 6 seq_num 66) dirty exists 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout:112: left oid 25 (ObjNum 66 snap 6 seq_num 66) 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout:112: done (1 left) 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout:117: finishing write tid 2 to smithi09025264-39 2023-01-15T14:28:44.789 INFO:tasks.rados.rados.0.smithi090.stdout:117: finishing write tid 3 to smithi09025264-39 2023-01-15T14:28:44.790 INFO:tasks.rados.rados.0.smithi090.stdout:117: finishing write tid 4 to smithi09025264-39 2023-01-15T14:28:44.790 INFO:tasks.rados.rados.0.smithi090.stdout:117: finishing write tid 5 to smithi09025264-39 2023-01-15T14:28:44.790 INFO:tasks.rados.rados.0.smithi090.stdout:117: oid 39 updating version 0 to 19 2023-01-15T14:28:44.790 INFO:tasks.rados.rados.0.smithi090.stdout:117: oid 39 updating version 19 to 20 2023-01-15T14:28:44.790 INFO:tasks.rados.rados.0.smithi090.stdout:117: oid 39 version 20 is already newer than 18 2023-01-15T14:28:44.790 INFO:tasks.rados.rados.0.smithi090.stdout:117: oid 39 updating version 20 to 21 2023-01-15T14:28:44.790 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 21 (ObjNum 68 snap 6 seq_num 68) dirty exists 2023-01-15T14:28:44.790 INFO:tasks.rados.rados.0.smithi090.stdout:117: left oid 39 (ObjNum 68 snap 6 seq_num 68) 2023-01-15T14:28:44.790 INFO:tasks.rados.rados.0.smithi090.stdout:117: done (0 left) 2023-01-15T14:28:44.791 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:28:44.791 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:28:44.793 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:28:44.793 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:28:44.793 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:28:44.793 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:28:44.793 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:28:44.793 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:28:44.794 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:28:44.794 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:28:44.794 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:28:44.794 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:28:44.794 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:28:44.794 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:28:44.794 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:28:44.795 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:28:44.795 INFO:tasks.rados.rados.0.smithi090.stdout:118: done (0 left) 2023-01-15T14:28:44.795 INFO:tasks.rados.rados.0.smithi090.stdout:119: snap_create 2023-01-15T14:28:44.796 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:28:44.796 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:41.025+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:28:44.799 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:28:44.799 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2023-01-15T14:28:40.907300+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":138211550,"num_objects":60,"num_object_clones":11,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":1,"num_read":378,"num_read_kb":170078,"num_write":439,"num_write_kb":99918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6777352,"kb_used_data":742856096,"kb_used_omap":1685,"kb_used_meta":0,"kb_avail":742817688,"statfs":{"total":767585320960,"available":760645312512,"internally_reserved":39329792,"allocated":760684642304,"data_stored":760684642304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1725868,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":78,"commit_latency_ns":0,"apply_latency_ns":78000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10531244,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":56,"num_read_kb":59156,"num_write":36,"num_write_kb":16412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.052694"},"pg_stats":[{"pgid":"3.f","version":"30'17","reported_seq":61,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-01-15T14:28:38.895853+0000","last_change":"2023-01-15T14:28:38.996744+0000","last_active":"2023-01-15T14:28:38.895853+0000","last_peered":"2023-01-15T14:28:38.895853+0000","last_clean":"2023-01-15T14:28:38.895853+0000","last_became_active":"2023-01-15T14:28:23.846063+0000","last_became_peered":"2023-01-15T14:28:23.846063+0000","last_unstale":"2023-01-15T14:28:38.895853+0000","last_undegraded":"2023-01-15T14:28:38.895853+0000","last_fullsized":"2023-01-15T14:28:38.895853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:52.368","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101152061,"stat_sum":{"num_bytes":11272365,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":11041,"num_write":21,"num_write_kb":5839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"8","length":"1"}]},{"pgid":"3.e","version":"23'3","reported_seq":41,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-01-15T14:28:38.895371+0000","last_change":"2023-01-15T14:28:38.895804+0000","last_active":"2023-01-15T14:28:38.895371+0000","last_peered":"2023-01-15T14:28:38.895371+0000","last_clean":"2023-01-15T14:28:38.895371+0000","last_became_active":"2023-01-15T14:28:23.846858+0000","last_became_peered":"2023-01-15T14:28:23.846858+0000","last_unstale":"2023-01-15T14:28:38.895371+0000","last_undegraded":"2023-01-15T14:28:38.895371+0000","last_fullsized":"2023-01-15T14:28:38.895371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:44.514","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000740042,"stat_sum":{"num_bytes":2134456,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"8","length":"1"}]},{"pgid":"3.d","version":"29'13","reported_seq":62,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-01-15T14:28:38.995512+0000","last_change":"2023-01-15T14:28:38.996184+0000","last_active":"2023-01-15T14:28:38.995512+0000","last_peered":"2023-01-15T14:28:38.995512+0000","last_clean":"2023-01-15T14:28:38.995512+0000","last_became_active":"2023-01-15T14:28:23.743427+0000","last_became_peered":"2023-01-15T14:28:23.743427+0000","last_unstale":"2023-01-15T14:28:38.995512+0000","last_undegraded":"2023-01-15T14:28:38.995512+0000","last_fullsized":"2023-01-15T14:28:38.995512+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:27.774","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088358999999999996,"stat_sum":{"num_bytes":10268945,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":25457,"num_write":17,"num_write_kb":7568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"8","length":"1"}]},{"pgid":"3.c","version":"23'3","reported_seq":46,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-01-15T14:28:40.076792+0000","last_change":"2023-01-15T14:28:40.077290+0000","last_active":"2023-01-15T14:28:40.076792+0000","last_peered":"2023-01-15T14:28:40.076792+0000","last_clean":"2023-01-15T14:28:40.076792+0000","last_became_active":"2023-01-15T14:28:23.842530+0000","last_became_peered":"2023-01-15T14:28:23.842530+0000","last_unstale":"2023-01-15T14:28:40.076792+0000","last_undegraded":"2023-01-15T14:28:40.076792+0000","last_fullsized":"2023-01-15T14:28:40.076792+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:36.640","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086397700000000004,"stat_sum":{"num_bytes":2319333,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"7","length":"2"}]},{"pgid":"3.b","version":"23'3","reported_seq":44,"reported_epoch":34,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:28:39.890730+0000","last_change":"2023-01-15T14:28:39.890730+0000","last_active":"2023-01-15T14:28:39.890730+0000","last_peered":"2023-01-15T14:28:39.890730+0000","last_clean":"2023-01-15T14:28:39.890730+0000","last_became_active":"2023-01-15T14:28:23.934183+0000","last_became_peered":"2023-01-15T14:28:23.934183+0000","last_unstale":"2023-01-15T14:28:39.890730+0000","last_undegraded":"2023-01-15T14:28:39.890730+0000","last_fullsized":"2023-01-15T14:28:39.890730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:45.586","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001706817,"stat_sum":{"num_bytes":2739508,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"8","length":"1"}]},{"pgid":"3.a","version":"34'38","reported_seq":85,"reported_epoch":34,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:28:40.656517+0000","last_change":"2023-01-15T14:28:39.946739+0000","last_active":"2023-01-15T14:28:40.656517+0000","last_peered":"2023-01-15T14:28:40.656517+0000","last_clean":"2023-01-15T14:28:40.656517+0000","last_became_active":"2023-01-15T14:28:23.845124+0000","last_became_peered":"2023-01-15T14:28:23.845124+0000","last_unstale":"2023-01-15T14:28:40.656517+0000","last_undegraded":"2023-01-15T14:28:40.656517+0000","last_fullsized":"2023-01-15T14:28:40.656517+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:38.384","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.80403145600000003,"stat_sum":{"num_bytes":11176523,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":13,"num_read_kb":7,"num_write":32,"num_write_kb":11357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"8","length":"1"}]},{"pgid":"3.9","version":"34'23","reported_seq":79,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-01-15T14:28:40.811322+0000","last_change":"2023-01-15T14:28:40.811898+0000","last_active":"2023-01-15T14:28:40.811322+0000","last_peered":"2023-01-15T14:28:40.811322+0000","last_clean":"2023-01-15T14:28:40.811322+0000","last_became_active":"2023-01-15T14:28:24.109843+0000","last_became_peered":"2023-01-15T14:28:24.109843+0000","last_unstale":"2023-01-15T14:28:40.811322+0000","last_undegraded":"2023-01-15T14:28:40.811322+0000","last_fullsized":"2023-01-15T14:28:40.811322+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.81620507099999995,"stat_sum":{"num_bytes":10218524,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":28481,"num_write":22,"num_write_kb":10015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"7","length":"2"}]},{"pgid":"3.8","version":"30'14","reported_seq":68,"reported_epoch":34,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:28:39.890685+0000","last_change":"2023-01-15T14:28:39.890777+0000","last_active":"2023-01-15T14:28:39.890685+0000","last_peered":"2023-01-15T14:28:39.890685+0000","last_clean":"2023-01-15T14:28:39.890685+0000","last_became_active":"2023-01-15T14:28:23.935904+0000","last_became_peered":"2023-01-15T14:28:23.935904+0000","last_unstale":"2023-01-15T14:28:39.890685+0000","last_undegraded":"2023-01-15T14:28:39.890685+0000","last_fullsized":"2023-01-15T14:28:39.890685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:33.903","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082552999999999999,"stat_sum":{"num_bytes":12487436,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":20648,"num_write":17,"num_write_kb":7107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"8","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":177,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-01-15T14:28:37.894242+0000","last_change":"2023-01-15T14:28:07.312081+0000","last_active":"2023-01-15T14:28:37.894242+0000","last_peered":"2023-01-15T14:28:37.894242+0000","last_clean":"2023-01-15T14:28:37.894242+0000","last_became_active":"2023-01-15T14:28:07.309738+0000","last_became_peered":"2023-01-15T14:28:07.309738+0000","last_unstale":"2023-01-15T14:28:37.894242+0000","last_undegraded":"2023-01-15T14:28:37.894242+0000","last_fullsized":"2023-01-15T14:28:37.894242+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:25.933","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":25,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-01-15T14:28:39.087365+0000","last_change":"2023-01-15T14:28:12.893975+0000","last_active":"2023-01-15T14:28:39.087365+0000","last_peered":"2023-01-15T14:28:39.087365+0000","last_clean":"2023-01-15T14:28:39.087365+0000","last_became_active":"2023-01-15T14:28:12.893100+0000","last_became_peered":"2023-01-15T14:28:12.893100+0000","last_unstale":"2023-01-15T14:28:39.087365+0000","last_undegraded":"2023-01-15T14:28:39.087365+0000","last_fullsized":"2023-01-15T14:28:39.087365+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.496","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"30'14","reported_seq":57,"reported_epoch":34,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:28:39.891199+0000","last_change":"2023-01-15T14:28:39.891199+0000","last_active":"2023-01-15T14:28:39.891199+0000","last_peered":"2023-01-15T14:28:39.891199+0000","last_clean":"2023-01-15T14:28:39.891199+0000","last_became_active":"2023-01-15T14:28:23.742582+0000","last_became_peered":"2023-01-15T14:28:23.742582+0000","last_unstale":"2023-01-15T14:28:39.891199+0000","last_undegraded":"2023-01-15T14:28:39.891199+0000","last_fullsized":"2023-01-15T14:28:39.891199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:30.630","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017473199999999999,"stat_sum":{"num_bytes":8165654,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":17,"num_write_kb":4187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"8","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-01-15T14:28:37.893958+0000","last_change":"2023-01-15T14:28:13.013616+0000","last_active":"2023-01-15T14:28:37.893958+0000","last_peered":"2023-01-15T14:28:37.893958+0000","last_clean":"2023-01-15T14:28:37.893958+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:28:37.893958+0000","last_undegraded":"2023-01-15T14:28:37.893958+0000","last_fullsized":"2023-01-15T14:28:37.893958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:29.249","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"34'18","reported_seq":67,"reported_epoch":34,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:28:39.948560+0000","last_change":"2023-01-15T14:28:39.948560+0000","last_active":"2023-01-15T14:28:39.948560+0000","last_peered":"2023-01-15T14:28:39.948560+0000","last_clean":"2023-01-15T14:28:39.948560+0000","last_became_active":"2023-01-15T14:28:23.842545+0000","last_became_peered":"2023-01-15T14:28:23.842545+0000","last_unstale":"2023-01-15T14:28:39.948560+0000","last_undegraded":"2023-01-15T14:28:39.948560+0000","last_fullsized":"2023-01-15T14:28:39.948560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:30.758","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.101100332,"stat_sum":{"num_bytes":9915998,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":14781,"num_write":20,"num_write_kb":5226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"8","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-01-15T14:28:33.815624+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:28:33.815624+0000","last_peered":"2023-01-15T14:28:33.815624+0000","last_clean":"2023-01-15T14:28:33.815624+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:28:33.815624+0000","last_undegraded":"2023-01-15T14:28:33.815624+0000","last_fullsized":"2023-01-15T14:28:33.815624+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.689","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"30'33","reported_seq":77,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-01-15T14:28:35.615815+0000","last_change":"2023-01-15T14:28:33.817824+0000","last_active":"2023-01-15T14:28:35.615815+0000","last_peered":"2023-01-15T14:28:35.615815+0000","last_clean":"2023-01-15T14:28:35.615815+0000","last_became_active":"2023-01-15T14:28:24.308255+0000","last_became_peered":"2023-01-15T14:28:24.308255+0000","last_unstale":"2023-01-15T14:28:35.615815+0000","last_undegraded":"2023-01-15T14:28:35.615815+0000","last_fullsized":"2023-01-15T14:28:35.615815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.769","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001071893,"stat_sum":{"num_bytes":14285752,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":29746,"num_write":29,"num_write_kb":13047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":25,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-01-15T14:28:39.087525+0000","last_change":"2023-01-15T14:28:12.993663+0000","last_active":"2023-01-15T14:28:39.087525+0000","last_peered":"2023-01-15T14:28:39.087525+0000","last_clean":"2023-01-15T14:28:39.087525+0000","last_became_active":"2023-01-15T14:28:12.993023+0000","last_became_peered":"2023-01-15T14:28:12.993023+0000","last_unstale":"2023-01-15T14:28:39.087525+0000","last_undegraded":"2023-01-15T14:28:39.087525+0000","last_fullsized":"2023-01-15T14:28:39.087525+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:31.082","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"34'46","reported_seq":103,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-01-15T14:28:40.654644+0000","last_change":"2023-01-15T14:28:40.655214+0000","last_active":"2023-01-15T14:28:40.654644+0000","last_peered":"2023-01-15T14:28:40.654644+0000","last_clean":"2023-01-15T14:28:40.654644+0000","last_became_active":"2023-01-15T14:28:23.946074+0000","last_became_peered":"2023-01-15T14:28:23.946074+0000","last_unstale":"2023-01-15T14:28:40.654644+0000","last_undegraded":"2023-01-15T14:28:40.654644+0000","last_fullsized":"2023-01-15T14:28:40.654644+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:40.271","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.70859643000000005,"stat_sum":{"num_bytes":16402819,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":40,"num_read_kb":32063,"num_write":39,"num_write_kb":12016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"7","length":"2"}]},{"pgid":"3.5","version":"34'13","reported_seq":57,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-01-15T14:28:40.711591+0000","last_change":"2023-01-15T14:28:40.712210+0000","last_active":"2023-01-15T14:28:40.711591+0000","last_peered":"2023-01-15T14:28:40.711591+0000","last_clean":"2023-01-15T14:28:40.711591+0000","last_became_active":"2023-01-15T14:28:24.110109+0000","last_became_peered":"2023-01-15T14:28:24.110109+0000","last_unstale":"2023-01-15T14:28:40.711591+0000","last_undegraded":"2023-01-15T14:28:40.711591+0000","last_fullsized":"2023-01-15T14:28:40.711591+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:23.917","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.620724783,"stat_sum":{"num_bytes":4252105,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":11,"num_write_kb":3830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"7","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-01-15T14:28:33.815015+0000","last_change":"2023-01-15T14:28:12.897110+0000","last_active":"2023-01-15T14:28:33.815015+0000","last_peered":"2023-01-15T14:28:33.815015+0000","last_clean":"2023-01-15T14:28:33.815015+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:28:33.815015+0000","last_undegraded":"2023-01-15T14:28:33.815015+0000","last_fullsized":"2023-01-15T14:28:33.815015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:27.569","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"26'16","reported_seq":52,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-01-15T14:28:34.951823+0000","last_change":"2023-01-15T14:28:33.815799+0000","last_active":"2023-01-15T14:28:34.951823+0000","last_peered":"2023-01-15T14:28:34.951823+0000","last_clean":"2023-01-15T14:28:34.951823+0000","last_became_active":"2023-01-15T14:28:24.108804+0000","last_became_peered":"2023-01-15T14:28:24.108804+0000","last_unstale":"2023-01-15T14:28:34.951823+0000","last_undegraded":"2023-01-15T14:28:34.951823+0000","last_fullsized":"2023-01-15T14:28:34.951823+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:45.648","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095506599999999997,"stat_sum":{"num_bytes":8050454,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":7739,"num_write":17,"num_write_kb":4915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":24,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-01-15T14:28:37.894309+0000","last_change":"2023-01-15T14:28:13.318466+0000","last_active":"2023-01-15T14:28:37.894309+0000","last_peered":"2023-01-15T14:28:37.894309+0000","last_clean":"2023-01-15T14:28:37.894309+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:28:37.894309+0000","last_undegraded":"2023-01-15T14:28:37.894309+0000","last_fullsized":"2023-01-15T14:28:37.894309+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.665","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"33'21","reported_seq":64,"reported_epoch":34,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:28:39.890903+0000","last_change":"2023-01-15T14:28:39.890981+0000","last_active":"2023-01-15T14:28:39.890903+0000","last_peered":"2023-01-15T14:28:39.890903+0000","last_clean":"2023-01-15T14:28:39.890903+0000","last_became_active":"2023-01-15T14:28:23.741266+0000","last_became_peered":"2023-01-15T14:28:23.741266+0000","last_unstale":"2023-01-15T14:28:39.890903+0000","last_undegraded":"2023-01-15T14:28:39.890903+0000","last_fullsized":"2023-01-15T14:28:39.890903+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.60579131799999997,"stat_sum":{"num_bytes":11040539,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":18,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"8","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-01-15T14:28:33.915599+0000","last_change":"2023-01-15T14:28:13.217588+0000","last_active":"2023-01-15T14:28:33.915599+0000","last_peered":"2023-01-15T14:28:33.915599+0000","last_clean":"2023-01-15T14:28:33.915599+0000","last_became_active":"2023-01-15T14:28:13.216947+0000","last_became_peered":"2023-01-15T14:28:13.216947+0000","last_unstale":"2023-01-15T14:28:33.915599+0000","last_undegraded":"2023-01-15T14:28:33.915599+0000","last_fullsized":"2023-01-15T14:28:33.915599+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.810","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"28'11","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-01-15T14:28:39.948892+0000","last_change":"2023-01-15T14:28:39.949183+0000","last_active":"2023-01-15T14:28:39.948892+0000","last_peered":"2023-01-15T14:28:39.948892+0000","last_clean":"2023-01-15T14:28:39.948892+0000","last_became_active":"2023-01-15T14:28:23.841189+0000","last_became_peered":"2023-01-15T14:28:23.841189+0000","last_unstale":"2023-01-15T14:28:39.948892+0000","last_undegraded":"2023-01-15T14:28:39.948892+0000","last_fullsized":"2023-01-15T14:28:39.948892+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:34.816","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046995799999999998,"stat_sum":{"num_bytes":2890771,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":8,"num_write_kb":2143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1","length":"2"},{"start":"4","length":"1"},{"start":"7","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":26,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-01-15T14:28:38.889683+0000","last_change":"2023-01-15T14:28:12.896363+0000","last_active":"2023-01-15T14:28:38.889683+0000","last_peered":"2023-01-15T14:28:38.889683+0000","last_clean":"2023-01-15T14:28:38.889683+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:28:38.889683+0000","last_undegraded":"2023-01-15T14:28:38.889683+0000","last_fullsized":"2023-01-15T14:28:38.889683+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.332","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":137621182,"num_objects":58,"num_object_clones":11,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":1,"num_read":262,"num_read_kb":169978,"num_write":280,"num_write_kb":97502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":286,"ondisk_log_size":286,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":859160,"kb_used_data":92844104,"kb_used_omap":249,"kb_used_meta":0,"kb_avail":92840220,"statfs":{"total":95948165120,"available":95068385280,"internally_reserved":3977216,"allocated":95072362496,"data_stored":95072362496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":255444,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":13,"commit_latency_ns":0,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0289999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":869536,"kb_used_data":92838820,"kb_used_omap":311,"kb_used_meta":0,"kb_avail":92829844,"statfs":{"total":95948165120,"available":95057760256,"internally_reserved":9191424,"allocated":95066951680,"data_stored":95066951680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":318495,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":10,"commit_latency_ns":0,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.02}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":824980,"kb_used_data":92874436,"kb_used_omap":133,"kb_used_meta":0,"kb_avail":92874400,"statfs":{"total":95948165120,"available":95103385600,"internally_reserved":36864,"allocated":95103422464,"data_stored":95103422464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":137143,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":40.482999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":96.370999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.181}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":843384,"kb_used_data":92858564,"kb_used_omap":240,"kb_used_meta":0,"kb_avail":92855996,"statfs":{"total":95948165120,"available":95084539904,"internally_reserved":2629632,"allocated":95087169536,"data_stored":95087169536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":246479,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":8,"commit_latency_ns":0,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.03}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2150000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":854864,"kb_used_data":92848404,"kb_used_omap":166,"kb_used_meta":0,"kb_avail":92844516,"statfs":{"total":95948165120,"available":95072784384,"internally_reserved":3981312,"allocated":95076765696,"data_stored":95076765696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":170437,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":12,"commit_latency_ns":0,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.129}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0680000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":826152,"kb_used_data":92878024,"kb_used_omap":137,"kb_used_meta":0,"kb_avail":92873228,"statfs":{"total":95948165120,"available":95102185472,"internally_reserved":4911104,"allocated":95107096576,"data_stored":95107096576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":141249,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.171}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1080000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":844564,"kb_used_data":92865192,"kb_used_omap":183,"kb_used_meta":0,"kb_avail":92854816,"statfs":{"total":95948165120,"available":95083331584,"internally_reserved":10625024,"allocated":95093956608,"data_stored":95093956608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":188086,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1140000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":854712,"kb_used_data":92848552,"kb_used_omap":262,"kb_used_meta":0,"kb_avail":92844668,"statfs":{"total":95948165120,"available":95072940032,"internally_reserved":3977216,"allocated":95076917248,"data_stored":95076917248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":268535,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":14,"commit_latency_ns":0,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.873}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90900000000000003}]}]}],"pool_statfs":[]}} 2023-01-15T14:28:44.994 INFO:tasks.thrashosds.thrasher:Setting 3.8 to [4, 5, 7, 2, 3, 1] 2023-01-15T14:28:44.994 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.8', '4', '5', '7', '2', '3', '1'] 2023-01-15T14:28:44.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.8 4 5 7 2 3 1 2023-01-15T14:28:45.001 INFO:tasks.rados.rados.0.smithi090.stdout:119: done (0 left) 2023-01-15T14:28:45.399 INFO:tasks.rados.rados.0.smithi090.stdout:120: delete oid 39 current snap is 8 2023-01-15T14:28:45.400 INFO:tasks.rados.rados.0.smithi090.stdout:120: done (0 left) 2023-01-15T14:28:45.400 INFO:tasks.rados.rados.0.smithi090.stdout:121: snap_remove snap 6 2023-01-15T14:28:45.400 INFO:tasks.rados.rados.0.smithi090.stdout:121: done (0 left) 2023-01-15T14:28:45.400 INFO:tasks.rados.rados.0.smithi090.stdout:122: read oid 14 snap 5 2023-01-15T14:28:45.400 INFO:tasks.rados.rados.0.smithi090.stdout:122: expect (ObjNum 62 snap 4 seq_num 62) 2023-01-15T14:28:45.400 INFO:tasks.rados.rados.0.smithi090.stdout:123: write oid 47 current snap is 8 2023-01-15T14:28:45.400 INFO:tasks.rados.rados.0.smithi090.stdout:123: seq_num 70 ranges {776490=714607,2246563=579356} 2023-01-15T14:28:45.400 INFO:tasks.rados.rados.0.smithi090.stdout:123: writing smithi09025264-47 from 776490 to 1491097 tid 1 2023-01-15T14:28:45.400 INFO:tasks.rados.rados.0.smithi090.stdout:123: writing smithi09025264-47 from 2246563 to 2825919 tid 2 2023-01-15T14:28:45.401 INFO:tasks.rados.rados.0.smithi090.stdout:124: snap_remove snap 7 2023-01-15T14:28:45.401 INFO:tasks.rados.rados.0.smithi090.stdout:124: done (2 left) 2023-01-15T14:28:45.401 INFO:tasks.rados.rados.0.smithi090.stdout:125: read oid 3 snap -1 2023-01-15T14:28:45.401 INFO:tasks.rados.rados.0.smithi090.stdout:125: expect deleted 2023-01-15T14:28:45.401 INFO:tasks.rados.rados.0.smithi090.stdout:126: read oid 5 snap 5 2023-01-15T14:28:45.401 INFO:tasks.rados.rados.0.smithi090.stdout:126: expect (ObjNum 4 snap 0 seq_num 4) 2023-01-15T14:28:45.401 INFO:tasks.rados.rados.0.smithi090.stdout:127: write (excl) oid 39 current snap is 8 2023-01-15T14:28:45.401 INFO:tasks.rados.rados.0.smithi090.stdout:127: seq_num 71 ranges {793297=789856,2305045=723014} 2023-01-15T14:28:45.401 INFO:tasks.rados.rados.0.smithi090.stdout:127: writing smithi09025264-39 from 793297 to 1583153 tid 1 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:127: writing smithi09025264-39 from 2305045 to 3028059 tid 2 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:128: write oid 48 current snap is 8 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:128: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:128: writing smithi09025264-48 from 410104 to 875209 tid 1 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:128: writing smithi09025264-48 from 1563527 to 2146552 tid 2 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:128: writing smithi09025264-48 from 2810926 to 3210104 tid 3 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:129: rollback oid 42 current snap is 8 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 42 to 5 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:122: expect (ObjNum 62 snap 4 seq_num 62) 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:123: finishing write tid 1 to smithi09025264-47 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:123: finishing write tid 2 to smithi09025264-47 2023-01-15T14:28:45.402 INFO:tasks.rados.rados.0.smithi090.stdout:123: finishing write tid 3 to smithi09025264-47 2023-01-15T14:28:45.407 INFO:tasks.rados.rados.0.smithi090.stdout:123: finishing write tid 4 to smithi09025264-47 2023-01-15T14:28:45.407 INFO:tasks.rados.rados.0.smithi090.stdout:123: oid 47 updating version 0 to 50 2023-01-15T14:28:45.407 INFO:tasks.rados.rados.0.smithi090.stdout:123: oid 47 version 50 is already newer than 49 2023-01-15T14:28:45.408 INFO:tasks.rados.rados.0.smithi090.stdout:123: oid 47 version 50 is already newer than 48 2023-01-15T14:28:45.408 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 50 (ObjNum 70 snap 8 seq_num 70) dirty exists 2023-01-15T14:28:45.408 INFO:tasks.rados.rados.0.smithi090.stdout:123: left oid 47 (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:28:45.408 INFO:tasks.rados.rados.0.smithi090.stdout:122: done (6 left) 2023-01-15T14:28:45.408 INFO:tasks.rados.rados.0.smithi090.stdout:123: done (5 left) 2023-01-15T14:28:45.408 INFO:tasks.rados.rados.0.smithi090.stdout:130: rollback oid 1 current snap is 8 2023-01-15T14:28:45.409 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 1 to 5 2023-01-15T14:28:45.409 INFO:tasks.rados.rados.0.smithi090.stdout:131: write (excl) oid 17 current snap is 8 2023-01-15T14:28:45.409 INFO:tasks.rados.rados.0.smithi090.stdout:131: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2023-01-15T14:28:45.410 INFO:tasks.rados.rados.0.smithi090.stdout:131: writing smithi09025264-17 from 426911 to 967265 tid 1 2023-01-15T14:28:45.410 INFO:tasks.rados.rados.0.smithi090.stdout:131: writing smithi09025264-17 from 1622009 to 2348692 tid 2 2023-01-15T14:28:45.410 INFO:tasks.rados.rados.0.smithi090.stdout:131: writing smithi09025264-17 from 3121996 to 3226911 tid 3 2023-01-15T14:28:45.410 INFO:tasks.rados.rados.0.smithi090.stdout:132: copy_from oid 33 from oid 26 current snap is 8 2023-01-15T14:28:45.410 INFO:tasks.rados.rados.0.smithi090.stdout:133: read oid 27 snap -1 2023-01-15T14:28:45.410 INFO:tasks.rados.rados.0.smithi090.stdout:133: expect (ObjNum 26 snap 0 seq_num 26) 2023-01-15T14:28:45.410 INFO:tasks.rados.rados.0.smithi090.stdout:134: write (excl) oid 13 current snap is 8 2023-01-15T14:28:45.411 INFO:tasks.rados.rados.0.smithi090.stdout:134: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2023-01-15T14:28:45.411 INFO:tasks.rados.rados.0.smithi090.stdout:134: writing smithi09025264-13 from 443718 to 1059321 tid 1 2023-01-15T14:28:45.411 INFO:tasks.rados.rados.0.smithi090.stdout:134: writing smithi09025264-13 from 1564138 to 2034479 tid 2 2023-01-15T14:28:45.411 INFO:tasks.rados.rados.0.smithi090.stdout:134: writing smithi09025264-13 from 2633066 to 3243718 tid 3 2023-01-15T14:28:45.411 INFO:tasks.rados.rados.0.smithi090.stdout:135: snap_create 2023-01-15T14:28:45.411 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 10 2023-01-15T14:28:45.412 INFO:tasks.rados.rados.0.smithi090.stdout:126: expect (ObjNum 4 snap 0 seq_num 4) 2023-01-15T14:28:45.412 INFO:tasks.rados.rados.0.smithi090.stdout:125: done (9 left) 2023-01-15T14:28:45.412 INFO:tasks.rados.rados.0.smithi090.stdout:126: done (8 left) 2023-01-15T14:28:45.412 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:28:45.412 INFO:tasks.rados.rados.0.smithi090.stdout:131: finishing write tid 1 to smithi09025264-17 2023-01-15T14:28:45.412 INFO:tasks.rados.rados.0.smithi090.stdout:129: finishing rollback tid 0 to smithi09025264-42 2023-01-15T14:28:45.412 INFO:tasks.rados.rados.0.smithi090.stdout:131: finishing write tid 2 to smithi09025264-17 2023-01-15T14:28:45.413 INFO:tasks.rados.rados.0.smithi090.stdout:129: finishing rollback tid 1 to smithi09025264-42 2023-01-15T14:28:45.413 INFO:tasks.rados.rados.0.smithi090.stdout:133: expect (ObjNum 26 snap 0 seq_num 26) 2023-01-15T14:28:45.413 INFO:tasks.rados.rados.0.smithi090.stdout:128: finishing write tid 1 to smithi09025264-48 2023-01-15T14:28:45.413 INFO:tasks.rados.rados.0.smithi090.stdout:128: finishing write tid 2 to smithi09025264-48 2023-01-15T14:28:45.413 INFO:tasks.rados.rados.0.smithi090.stdout:128: finishing write tid 3 to smithi09025264-48 2023-01-15T14:28:45.413 INFO:tasks.rados.rados.0.smithi090.stdout:128: finishing write tid 4 to smithi09025264-48 2023-01-15T14:28:45.413 INFO:tasks.rados.rados.0.smithi090.stdout:128: finishing write tid 5 to smithi09025264-48 2023-01-15T14:28:45.413 INFO:tasks.rados.rados.0.smithi090.stdout:128: oid 48 updating version 0 to 17 2023-01-15T14:28:45.413 INFO:tasks.rados.rados.0.smithi090.stdout:128: oid 48 updating version 17 to 20 2023-01-15T14:28:45.414 INFO:tasks.rados.rados.0.smithi090.stdout:128: oid 48 version 20 is already newer than 18 2023-01-15T14:28:45.414 INFO:tasks.rados.rados.0.smithi090.stdout:128: oid 48 version 20 is already newer than 19 2023-01-15T14:28:45.414 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 20 (ObjNum 72 snap 8 seq_num 72) dirty exists 2023-01-15T14:28:45.414 INFO:tasks.rados.rados.0.smithi090.stdout:128: left oid 48 (ObjNum 72 snap 8 seq_num 72) 2023-01-15T14:28:45.414 INFO:tasks.rados.rados.0.smithi090.stdout:131: finishing write tid 3 to smithi09025264-17 2023-01-15T14:28:45.414 INFO:tasks.rados.rados.0.smithi090.stdout:128: done (7 left) 2023-01-15T14:28:45.414 INFO:tasks.rados.rados.0.smithi090.stdout:133: done (6 left) 2023-01-15T14:28:45.414 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:28:45.415 INFO:tasks.rados.rados.0.smithi090.stdout:127: finishing write tid 1 to smithi09025264-39 2023-01-15T14:28:45.415 INFO:tasks.rados.rados.0.smithi090.stdout:130: finishing rollback tid 0 to smithi09025264-1 2023-01-15T14:28:45.415 INFO:tasks.rados.rados.0.smithi090.stdout:129: finishing rollback tid 2 to smithi09025264-42 2023-01-15T14:28:45.415 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 42 v 20 (ObjNum 21 snap 0 seq_num 21) dirty exists 2023-01-15T14:28:45.415 INFO:tasks.rados.rados.0.smithi090.stdout:129: done (5 left) 2023-01-15T14:28:45.415 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:28:45.415 INFO:tasks.rados.rados.0.smithi090.stdout:127: finishing write tid 2 to smithi09025264-39 2023-01-15T14:28:45.416 INFO:tasks.rados.rados.0.smithi090.stdout:131: finishing write tid 4 to smithi09025264-17 2023-01-15T14:28:45.416 INFO:tasks.rados.rados.0.smithi090.stdout:134: finishing write tid 1 to smithi09025264-13 2023-01-15T14:28:45.416 INFO:tasks.rados.rados.0.smithi090.stdout:132: finishing copy_from to smithi09025264-33 2023-01-15T14:28:45.416 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 38 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-01-15T14:28:45.416 INFO:tasks.rados.rados.0.smithi090.stdout:132: finishing copy_from racing read to smithi09025264-33 2023-01-15T14:28:45.416 INFO:tasks.rados.rados.0.smithi090.stdout:132: done (4 left) 2023-01-15T14:28:45.416 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:28:45.416 INFO:tasks.rados.rados.0.smithi090.stdout:127: finishing write tid 3 to smithi09025264-39 2023-01-15T14:28:45.417 INFO:tasks.rados.rados.0.smithi090.stdout:127: finishing write tid 4 to smithi09025264-39 2023-01-15T14:28:45.417 INFO:tasks.rados.rados.0.smithi090.stdout:127: oid 39 updating version 0 to 27 2023-01-15T14:28:45.417 INFO:tasks.rados.rados.0.smithi090.stdout:127: oid 39 updating version 27 to 28 2023-01-15T14:28:45.417 INFO:tasks.rados.rados.0.smithi090.stdout:127: oid 39 version 28 is already newer than 26 2023-01-15T14:28:45.417 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 28 (ObjNum 71 snap 8 seq_num 71) dirty exists 2023-01-15T14:28:45.417 INFO:tasks.rados.rados.0.smithi090.stdout:127: left oid 39 (ObjNum 71 snap 8 seq_num 71) 2023-01-15T14:28:45.417 INFO:tasks.rados.rados.0.smithi090.stdout:127: done (3 left) 2023-01-15T14:28:45.418 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:28:45.418 INFO:tasks.rados.rados.0.smithi090.stdout:131: finishing write tid 5 to smithi09025264-17 2023-01-15T14:28:45.418 INFO:tasks.rados.rados.0.smithi090.stdout:131: oid 17 updating version 0 to 56 2023-01-15T14:28:45.419 INFO:tasks.rados.rados.0.smithi090.stdout:131: oid 17 updating version 56 to 57 2023-01-15T14:28:45.419 INFO:tasks.rados.rados.0.smithi090.stdout:131: oid 17 version 57 is already newer than 55 2023-01-15T14:28:45.419 INFO:tasks.rados.rados.0.smithi090.stdout:131: oid 17 version 57 is already newer than 54 2023-01-15T14:28:45.419 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 57 (ObjNum 73 snap 8 seq_num 73) dirty exists 2023-01-15T14:28:45.419 INFO:tasks.rados.rados.0.smithi090.stdout:131: left oid 17 (ObjNum 73 snap 8 seq_num 73) 2023-01-15T14:28:45.419 INFO:tasks.rados.rados.0.smithi090.stdout:131: done (2 left) 2023-01-15T14:28:45.419 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:28:45.419 INFO:tasks.rados.rados.0.smithi090.stdout:134: finishing write tid 2 to smithi09025264-13 2023-01-15T14:28:45.419 INFO:tasks.rados.rados.0.smithi090.stdout:130: finishing rollback tid 1 to smithi09025264-1 2023-01-15T14:28:45.419 INFO:tasks.rados.rados.0.smithi090.stdout:134: finishing write tid 3 to smithi09025264-13 2023-01-15T14:28:45.419 INFO:tasks.rados.rados.0.smithi090.stdout:134: finishing write tid 4 to smithi09025264-13 2023-01-15T14:28:45.420 INFO:tasks.rados.rados.0.smithi090.stdout:134: finishing write tid 5 to smithi09025264-13 2023-01-15T14:28:45.420 INFO:tasks.rados.rados.0.smithi090.stdout:134: oid 13 updating version 0 to 42 2023-01-15T14:28:45.420 INFO:tasks.rados.rados.0.smithi090.stdout:134: oid 13 updating version 42 to 43 2023-01-15T14:28:45.420 INFO:tasks.rados.rados.0.smithi090.stdout:134: oid 13 version 43 is already newer than 40 2023-01-15T14:28:45.420 INFO:tasks.rados.rados.0.smithi090.stdout:134: oid 13 version 43 is already newer than 41 2023-01-15T14:28:45.420 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 43 (ObjNum 74 snap 8 seq_num 74) dirty exists 2023-01-15T14:28:45.420 INFO:tasks.rados.rados.0.smithi090.stdout:134: left oid 13 (ObjNum 74 snap 8 seq_num 74) 2023-01-15T14:28:45.420 INFO:tasks.rados.rados.0.smithi090.stdout:134: done (1 left) 2023-01-15T14:28:45.420 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:45.421 INFO:tasks.rados.rados.0.smithi090.stdout:130: finishing rollback tid 2 to smithi09025264-1 2023-01-15T14:28:45.421 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 53 (ObjNum 0 snap 0 seq_num 0) dirty exists 2023-01-15T14:28:45.421 INFO:tasks.rados.rados.0.smithi090.stdout:130: done (0 left) 2023-01-15T14:28:45.421 INFO:tasks.rados.rados.0.smithi090.stdout:135: done (0 left) 2023-01-15T14:28:45.421 INFO:tasks.rados.rados.0.smithi090.stdout:136: snap_create 2023-01-15T14:28:45.421 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:44.829+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:28:45.423 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:44.993+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:28:45.423 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:45.565 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:45.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:28:45.567 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:45.417+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:45.568 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:45.561+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:28:45.666 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:45.732 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:45.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:28:45.734 INFO:tasks.rados.rados.0.smithi090.stdout:136: done (0 left) 2023-01-15T14:28:45.734 INFO:tasks.rados.rados.0.smithi090.stdout:137: snap_remove snap 5 2023-01-15T14:28:45.734 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:45.725+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:45.832 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:45.898 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:45.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:28:45.901 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:45.897+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:28:46.001 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:45.993+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:46.060 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:46.060 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:46.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:28:46.107 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:46.101+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:28:46.233 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:46.234 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:46.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:28:46.235 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:46.229+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:46.333 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:46.329+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:28:46.399 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:46.400 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:46.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:28:46.455 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:46.449+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:28:46.629 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:46.630 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:46.630 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:28:46.631 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:46.621+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:46.730 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:46.725+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:46.830 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:28:46.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:28:46.996 INFO:tasks.rados.rados.0.smithi090.stdout:137: done (0 left) 2023-01-15T14:28:46.996 INFO:tasks.rados.rados.0.smithi090.stdout:138: write oid 21 current snap is 10 2023-01-15T14:28:46.996 INFO:tasks.rados.rados.0.smithi090.stdout:138: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2023-01-15T14:28:46.996 INFO:teuthology.orchestra.run.smithi090.stderr:set 3.8 pg_upmap_items mapping to [4->5,7->2,3->1] 2023-01-15T14:28:46.997 INFO:tasks.rados.rados.0.smithi090.stdout:138: writing smithi09025264-21 from 460525 to 1151377 tid 1 2023-01-15T14:28:46.997 INFO:tasks.rados.rados.0.smithi090.stdout:138: writing smithi09025264-21 from 1622620 to 2352972 tid 2 2023-01-15T14:28:46.998 INFO:tasks.rados.rados.0.smithi090.stdout:138: writing smithi09025264-21 from 3060489 to 3260525 tid 3 2023-01-15T14:28:46.999 INFO:tasks.rados.rados.0.smithi090.stdout:139: read oid 47 snap -1 2023-01-15T14:28:46.999 INFO:tasks.rados.rados.0.smithi090.stdout:139: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:28:46.999 INFO:tasks.rados.rados.0.smithi090.stdout:140: delete oid 19 current snap is 10 2023-01-15T14:28:47.000 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:47.000 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:47.001 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:28:47.001 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:46.993+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:47.040 INFO:tasks.rados.rados.0.smithi090.stdout:139: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:28:47.162 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:28:47.162 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:47.162 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:47.162 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:47.153+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:28:47.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:28:47.175 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:28:47.323 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:28:47.323 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:28:47.323 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:28:47.324 INFO:tasks.rados.rados.0.smithi090.stdout:138: finishing write tid 1 to smithi09025264-21 2023-01-15T14:28:47.327 INFO:tasks.rados.rados.0.smithi090.stdout:139: done (2 left) 2023-01-15T14:28:47.327 INFO:tasks.rados.rados.0.smithi090.stdout:140: done (1 left) 2023-01-15T14:28:47.328 INFO:tasks.rados.rados.0.smithi090.stdout:141: write oid 25 current snap is 10 2023-01-15T14:28:47.328 INFO:tasks.rados.rados.0.smithi090.stdout:141: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2023-01-15T14:28:47.328 INFO:tasks.rados.rados.0.smithi090.stdout:141: writing smithi09025264-25 from 494139 to 1051842 tid 1 2023-01-15T14:28:47.328 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:28:47.330 INFO:tasks.rados.rados.0.smithi090.stdout:141: writing smithi09025264-25 from 1455937 to 2189958 tid 2 2023-01-15T14:28:47.330 INFO:tasks.rados.rados.0.smithi090.stdout:141: writing smithi09025264-25 from 2948041 to 3294139 tid 3 2023-01-15T14:28:47.330 INFO:tasks.rados.rados.0.smithi090.stdout:142: write (excl) oid 29 current snap is 10 2023-01-15T14:28:47.330 INFO:tasks.rados.rados.0.smithi090.stdout:142: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2023-01-15T14:28:47.330 INFO:tasks.rados.rados.0.smithi090.stdout:142: writing smithi09025264-29 from 510946 to 1143898 tid 1 2023-01-15T14:28:47.330 INFO:tasks.rados.rados.0.smithi090.stdout:142: writing smithi09025264-29 from 1798066 to 2275745 tid 2 2023-01-15T14:28:47.330 INFO:tasks.rados.rados.0.smithi090.stdout:142: writing smithi09025264-29 from 2742758 to 3199975 tid 3 2023-01-15T14:28:47.330 INFO:tasks.rados.rados.0.smithi090.stdout:143: write (excl) oid 8 current snap is 10 2023-01-15T14:28:47.330 INFO:tasks.rados.rados.0.smithi090.stdout:143: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2023-01-15T14:28:47.330 INFO:tasks.rados.rados.0.smithi090.stdout:143: writing smithi09025264-8 from 527753 to 1235954 tid 1 2023-01-15T14:28:47.330 INFO:tasks.rados.rados.0.smithi090.stdout:143: writing smithi09025264-8 from 1856548 to 2594238 tid 2 2023-01-15T14:28:47.331 INFO:tasks.rados.rados.0.smithi090.stdout:143: writing smithi09025264-8 from 3286534 to 3327753 tid 3 2023-01-15T14:28:47.331 INFO:tasks.rados.rados.0.smithi090.stdout:144: delete oid 50 current snap is 10 2023-01-15T14:28:47.331 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:47.317+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:28:47.341 INFO:tasks.rados.rados.0.smithi090.stdout:138: finishing write tid 2 to smithi09025264-21 2023-01-15T14:28:47.341 INFO:tasks.rados.rados.0.smithi090.stdout:138: finishing write tid 3 to smithi09025264-21 2023-01-15T14:28:47.361 INFO:tasks.rados.rados.0.smithi090.stdout:141: finishing write tid 1 to smithi09025264-25 2023-01-15T14:28:47.489 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:28:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:28:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:442 3.2 3:47dfe935:::smithi09025264-29:head [write 510946~632952 in=632952b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:28:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.270522+0000", 2023-01-15T14:28:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.16312254600000001, 2023-01-15T14:28:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16316993900000001, 2023-01-15T14:28:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:28:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 442 2023-01-15T14:28:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.270522+0000", 2023-01-15T14:28:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.270522+0000", 2023-01-15T14:28:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.270520+0000", 2023-01-15T14:28:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:28:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.270996+0000", 2023-01-15T14:28:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047654799999999998 2023-01-15T14:28:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.271024+0000", 2023-01-15T14:28:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8127999999999999e-05 2023-01-15T14:28:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.271037+0000", 2023-01-15T14:28:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2374999999999999e-05 2023-01-15T14:28:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.271123+0000", 2023-01-15T14:28:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5955e-05 2023-01-15T14:28:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.271267+0000", 2023-01-15T14:28:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014434100000000001 2023-01-15T14:28:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:28:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.271517+0000", 2023-01-15T14:28:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025031199999999998 2023-01-15T14:28:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.272285+0000", 2023-01-15T14:28:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076716700000000002 2023-01-15T14:28:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.272297+0000", 2023-01-15T14:28:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.279e-05 2023-01-15T14:28:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.273120+0000", 2023-01-15T14:28:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082306999999999996 2023-01-15T14:28:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.273210+0000", 2023-01-15T14:28:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9327999999999996e-05 2023-01-15T14:28:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.377441+0000", 2023-01-15T14:28:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10423091499999999 2023-01-15T14:28:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:443 3.2 3:47dfe935:::smithi09025264-29:head [write 1798066~477679 in=477679b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:28:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.276973+0000", 2023-01-15T14:28:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.15667142000000001, 2023-01-15T14:28:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.156880938, 2023-01-15T14:28:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "delayed", 2023-01-15T14:28:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 443 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.276973+0000", 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.276973+0000", 2023-01-15T14:28:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.276971+0000", 2023-01-15T14:28:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:28:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.277270+0000", 2023-01-15T14:28:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000299093 2023-01-15T14:28:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.277305+0000", 2023-01-15T14:28:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4037e-05 2023-01-15T14:28:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:47.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.277315+0000", 2023-01-15T14:28:47.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0767e-05 2023-01-15T14:28:47.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:47.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.277391+0000", 2023-01-15T14:28:47.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6026999999999996e-05 2023-01-15T14:28:47.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:47.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.277512+0000", 2023-01-15T14:28:47.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120586 2023-01-15T14:28:47.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:47.651 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:47.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:47.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.651 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:444 3.2 3:47dfe935:::smithi09025264-29:head [write 2742758~457217 in=457217b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:28:47.651 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.282268+0000", 2023-01-15T14:28:47.651 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.15137693399999999, 2023-01-15T14:28:47.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15168525499999999, 2023-01-15T14:28:47.651 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:47.652 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "delayed", 2023-01-15T14:28:47.652 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:47.653 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:47.653 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:47.653 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 444 2023-01-15T14:28:47.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.653 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:47.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:47.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282268+0000", 2023-01-15T14:28:47.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:47.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:47.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282268+0000", 2023-01-15T14:28:47.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:47.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:47.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282266+0000", 2023-01-15T14:28:47.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:47.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:47.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282528+0000", 2023-01-15T14:28:47.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026183800000000002 2023-01-15T14:28:47.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:47.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282584+0000", 2023-01-15T14:28:47.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6403999999999997e-05 2023-01-15T14:28:47.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:47.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282599+0000", 2023-01-15T14:28:47.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4968e-05 2023-01-15T14:28:47.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:47.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282664+0000", 2023-01-15T14:28:47.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5043999999999996e-05 2023-01-15T14:28:47.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:47.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282755+0000", 2023-01-15T14:28:47.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0593999999999994e-05 2023-01-15T14:28:47.661 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:47.661 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:47.661 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:47.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.662 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:445 3.2 3:47dfe935:::smithi09025264-29:head [setxattr _header (57) in=64b,truncate 3310946] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:28:47.662 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.282651+0000", 2023-01-15T14:28:47.662 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.15099341199999999, 2023-01-15T14:28:47.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15140055199999999, 2023-01-15T14:28:47.662 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:47.663 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "delayed", 2023-01-15T14:28:47.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:47.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:47.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:47.663 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 445 2023-01-15T14:28:47.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.664 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:47.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:47.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282651+0000", 2023-01-15T14:28:47.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:47.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:47.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282651+0000", 2023-01-15T14:28:47.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:47.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:47.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282650+0000", 2023-01-15T14:28:47.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:47.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:47.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282659+0000", 2023-01-15T14:28:47.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4269999999999997e-06 2023-01-15T14:28:47.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:47.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282675+0000", 2023-01-15T14:28:47.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5267000000000001e-05 2023-01-15T14:28:47.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:47.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282684+0000", 2023-01-15T14:28:47.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4900000000000006e-06 2023-01-15T14:28:47.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:47.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282787+0000", 2023-01-15T14:28:47.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010244899999999999 2023-01-15T14:28:47.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:47.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282860+0000", 2023-01-15T14:28:47.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3514000000000001e-05 2023-01-15T14:28:47.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:47.865 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:47.865 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:47.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.865 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:446 3.2 3:47dfe935:::smithi09025264-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:28:47.865 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.282704+0000", 2023-01-15T14:28:47.865 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.15094042799999999, 2023-01-15T14:28:47.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.151442514, 2023-01-15T14:28:47.866 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:47.867 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "delayed", 2023-01-15T14:28:47.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:47.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:47.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:47.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 446 2023-01-15T14:28:47.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.867 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:47.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:47.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282704+0000", 2023-01-15T14:28:47.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:47.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:47.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282704+0000", 2023-01-15T14:28:47.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:47.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:47.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282703+0000", 2023-01-15T14:28:47.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:47.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:47.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282708+0000", 2023-01-15T14:28:47.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1019999999999996e-06 2023-01-15T14:28:47.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:47.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282721+0000", 2023-01-15T14:28:47.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.256e-05 2023-01-15T14:28:47.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:47.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282730+0000", 2023-01-15T14:28:47.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3870000000000001e-06 2023-01-15T14:28:47.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:47.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:47.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:47.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282910+0000", 2023-01-15T14:28:48.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018028299999999999 2023-01-15T14:28:48.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:48.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.282995+0000", 2023-01-15T14:28:48.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4955000000000002e-05 2023-01-15T14:28:48.060 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.060 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:48.060 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.061 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.4530.0:442 3.2 e41/20)", 2023-01-15T14:28:48.061 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.377261+0000", 2023-01-15T14:28:48.061 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.056383181999999997, 2023-01-15T14:28:48.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.056972647000000001, 2023-01-15T14:28:48.061 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:48.061 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:28:48.061 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:48.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:48.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.377261+0000", 2023-01-15T14:28:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.377261+0000", 2023-01-15T14:28:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:48.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.377261+0000", 2023-01-15T14:28:48.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:48.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:48.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.377267+0000", 2023-01-15T14:28:48.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4779999999999996e-06 2023-01-15T14:28:48.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.377313+0000", 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6248999999999997e-05 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.377322+0000", 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2479999999999996e-06 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:48.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.066 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.066 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:28:48.066 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 6 2023-01-15T14:28:48.066 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:28:48.066 INFO:tasks.rados.rados.0.smithi090.stdout:138: finishing write tid 4 to smithi09025264-21 2023-01-15T14:28:48.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:28:48.068 INFO:tasks.rados.rados.0.smithi090.stdout:141: finishing write tid 2 to smithi09025264-25 2023-01-15T14:28:48.068 INFO:tasks.rados.rados.0.smithi090.stdout:142: finishing write tid 1 to smithi09025264-29 2023-01-15T14:28:48.068 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:28:48.069 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:47.481+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:28:48.069 INFO:tasks.rados.rados.0.smithi090.stdout:143: finishing write tid 1 to smithi09025264-8 2023-01-15T14:28:48.069 INFO:tasks.rados.rados.0.smithi090.stdout:144: done (4 left) 2023-01-15T14:28:48.069 INFO:tasks.rados.rados.0.smithi090.stdout:145: rollback oid 38 current snap is 10 2023-01-15T14:28:48.069 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 38 to 8 2023-01-15T14:28:48.069 INFO:tasks.rados.rados.0.smithi090.stdout:146: snap_remove snap 9 2023-01-15T14:28:48.070 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:28:48.071 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:28:48.071 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:28:48.071 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:28:48.071 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:28:48.071 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:28:48.071 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:28:48.071 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:28:48.071 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:28:48.072 INFO:tasks.rados.rados.0.smithi090.stdout:146: done (5 left) 2023-01-15T14:28:48.072 INFO:tasks.rados.rados.0.smithi090.stdout:147: read oid 45 snap -1 2023-01-15T14:28:48.072 INFO:tasks.rados.rados.0.smithi090.stdout:147: expect (ObjNum 44 snap 0 seq_num 44) 2023-01-15T14:28:48.072 INFO:tasks.rados.rados.0.smithi090.stdout:148: write (excl) oid 26 current snap is 10 2023-01-15T14:28:48.072 INFO:tasks.rados.rados.0.smithi090.stdout:145: finishing rollback tid 0 to smithi09025264-38 2023-01-15T14:28:48.072 INFO:tasks.rados.rados.0.smithi090.stdout:148: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2023-01-15T14:28:48.073 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:28:48.073 INFO:tasks.rados.rados.0.smithi090.stdout:148: writing smithi09025264-26 from 561367 to 1020066 tid 1 2023-01-15T14:28:48.076 INFO:tasks.rados.rados.0.smithi090.stdout:148: writing smithi09025264-26 from 1573512 to 2314871 tid 2 2023-01-15T14:28:48.078 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:48.065+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:28:48.256 INFO:tasks.rados.rados.0.smithi090.stdout:148: writing smithi09025264-26 from 2941380 to 3361367 tid 3 2023-01-15T14:28:48.256 INFO:tasks.rados.rados.0.smithi090.stdout:149: write oid 15 current snap is 10 2023-01-15T14:28:48.256 INFO:tasks.rados.rados.0.smithi090.stdout:145: finishing rollback tid 1 to smithi09025264-38 2023-01-15T14:28:48.256 INFO:tasks.rados.rados.0.smithi090.stdout:149: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2023-01-15T14:28:48.256 INFO:tasks.rados.rados.0.smithi090.stdout:149: writing smithi09025264-15 from 578174 to 1112122 tid 1 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout:149: writing smithi09025264-15 from 1515641 to 2000658 tid 2 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout:149: writing smithi09025264-15 from 2736097 to 3238402 tid 3 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout:150: snap_create 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout:138: finishing write tid 5 to smithi09025264-21 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout:138: oid 21 updating version 0 to 28 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout:138: oid 21 version 28 is already newer than 25 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout:138: oid 21 version 28 is already newer than 27 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout:138: oid 21 version 28 is already newer than 26 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 28 (ObjNum 75 snap 10 seq_num 75) dirty exists 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout:138: left oid 21 (ObjNum 75 snap 10 seq_num 75) 2023-01-15T14:28:48.257 INFO:tasks.rados.rados.0.smithi090.stdout:141: finishing write tid 3 to smithi09025264-25 2023-01-15T14:28:48.258 INFO:tasks.rados.rados.0.smithi090.stdout:141: finishing write tid 4 to smithi09025264-25 2023-01-15T14:28:48.259 INFO:tasks.rados.rados.0.smithi090.stdout:141: finishing write tid 5 to smithi09025264-25 2023-01-15T14:28:48.259 INFO:tasks.rados.rados.0.smithi090.stdout:141: oid 25 updating version 0 to 50 2023-01-15T14:28:48.259 INFO:tasks.rados.rados.0.smithi090.stdout:141: oid 25 updating version 50 to 52 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout:141: oid 25 version 52 is already newer than 51 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout:141: oid 25 version 52 is already newer than 49 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 52 (ObjNum 77 snap 10 seq_num 77) dirty exists 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout:141: left oid 25 (ObjNum 77 snap 10 seq_num 77) 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout:142: finishing write tid 2 to smithi09025264-29 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout:138: done (7 left) 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout:141: done (6 left) 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout:142: finishing write tid 3 to smithi09025264-29 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout:143: finishing write tid 2 to smithi09025264-8 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout:143: finishing write tid 3 to smithi09025264-8 2023-01-15T14:28:48.260 INFO:tasks.rados.rados.0.smithi090.stdout:143: finishing write tid 4 to smithi09025264-8 2023-01-15T14:28:48.261 INFO:tasks.rados.rados.0.smithi090.stdout:143: finishing write tid 5 to smithi09025264-8 2023-01-15T14:28:48.261 INFO:tasks.rados.rados.0.smithi090.stdout:143: oid 8 updating version 0 to 25 2023-01-15T14:28:48.261 INFO:tasks.rados.rados.0.smithi090.stdout:143: oid 8 updating version 25 to 26 2023-01-15T14:28:48.261 INFO:tasks.rados.rados.0.smithi090.stdout:143: oid 8 version 26 is already newer than 24 2023-01-15T14:28:48.261 INFO:tasks.rados.rados.0.smithi090.stdout:143: oid 8 updating version 26 to 27 2023-01-15T14:28:48.261 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 27 (ObjNum 79 snap 10 seq_num 79) dirty exists 2023-01-15T14:28:48.261 INFO:tasks.rados.rados.0.smithi090.stdout:143: left oid 8 (ObjNum 79 snap 10 seq_num 79) 2023-01-15T14:28:48.261 INFO:tasks.rados.rados.0.smithi090.stdout:142: finishing write tid 4 to smithi09025264-29 2023-01-15T14:28:48.261 INFO:tasks.rados.rados.0.smithi090.stdout:143: done (5 left) 2023-01-15T14:28:48.261 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:28:48.261 INFO:tasks.rados.rados.0.smithi090.stdout:145: finishing rollback tid 2 to smithi09025264-38 2023-01-15T14:28:48.262 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 24 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-01-15T14:28:48.262 INFO:tasks.rados.rados.0.smithi090.stdout:142: finishing write tid 5 to smithi09025264-29 2023-01-15T14:28:48.262 INFO:tasks.rados.rados.0.smithi090.stdout:142: oid 29 updating version 0 to 24 2023-01-15T14:28:48.262 INFO:tasks.rados.rados.0.smithi090.stdout:142: oid 29 version 24 is already newer than 22 2023-01-15T14:28:48.262 INFO:tasks.rados.rados.0.smithi090.stdout:142: oid 29 updating version 24 to 25 2023-01-15T14:28:48.262 INFO:tasks.rados.rados.0.smithi090.stdout:142: oid 29 version 25 is already newer than 23 2023-01-15T14:28:48.262 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 25 (ObjNum 78 snap 10 seq_num 78) dirty exists 2023-01-15T14:28:48.262 INFO:tasks.rados.rados.0.smithi090.stdout:142: left oid 29 (ObjNum 78 snap 10 seq_num 78) 2023-01-15T14:28:48.262 INFO:tasks.rados.rados.0.smithi090.stdout:142: done (4 left) 2023-01-15T14:28:48.263 INFO:tasks.rados.rados.0.smithi090.stdout:145: done (3 left) 2023-01-15T14:28:48.263 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:28:48.263 INFO:tasks.rados.rados.0.smithi090.stdout:147: expect (ObjNum 44 snap 0 seq_num 44) 2023-01-15T14:28:48.263 INFO:tasks.rados.rados.0.smithi090.stdout:147: done (2 left) 2023-01-15T14:28:48.263 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:28:48.263 INFO:tasks.rados.rados.0.smithi090.stdout:148: finishing write tid 1 to smithi09025264-26 2023-01-15T14:28:48.263 INFO:tasks.rados.rados.0.smithi090.stdout:148: finishing write tid 2 to smithi09025264-26 2023-01-15T14:28:48.263 INFO:tasks.rados.rados.0.smithi090.stdout:148: finishing write tid 3 to smithi09025264-26 2023-01-15T14:28:48.264 INFO:tasks.rados.rados.0.smithi090.stdout:148: finishing write tid 4 to smithi09025264-26 2023-01-15T14:28:48.264 INFO:tasks.rados.rados.0.smithi090.stdout:148: finishing write tid 5 to smithi09025264-26 2023-01-15T14:28:48.264 INFO:tasks.rados.rados.0.smithi090.stdout:148: oid 26 updating version 0 to 65 2023-01-15T14:28:48.264 INFO:tasks.rados.rados.0.smithi090.stdout:148: oid 26 updating version 65 to 66 2023-01-15T14:28:48.264 INFO:tasks.rados.rados.0.smithi090.stdout:148: oid 26 version 66 is already newer than 63 2023-01-15T14:28:48.264 INFO:tasks.rados.rados.0.smithi090.stdout:148: oid 26 version 66 is already newer than 64 2023-01-15T14:28:48.264 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 66 (ObjNum 81 snap 10 seq_num 81) dirty exists 2023-01-15T14:28:48.264 INFO:tasks.rados.rados.0.smithi090.stdout:148: left oid 26 (ObjNum 81 snap 10 seq_num 81) 2023-01-15T14:28:48.264 INFO:tasks.rados.rados.0.smithi090.stdout:148: done (1 left) 2023-01-15T14:28:48.265 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:48.265 INFO:tasks.rados.rados.0.smithi090.stdout:149: finishing write tid 1 to smithi09025264-15 2023-01-15T14:28:48.265 INFO:tasks.rados.rados.0.smithi090.stdout:149: finishing write tid 2 to smithi09025264-15 2023-01-15T14:28:48.265 INFO:tasks.rados.rados.0.smithi090.stdout:149: finishing write tid 3 to smithi09025264-15 2023-01-15T14:28:48.265 INFO:tasks.rados.rados.0.smithi090.stdout:149: finishing write tid 4 to smithi09025264-15 2023-01-15T14:28:48.266 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:48.261+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:28:48.272 INFO:tasks.rados.rados.0.smithi090.stdout:149: finishing write tid 5 to smithi09025264-15 2023-01-15T14:28:48.472 INFO:tasks.rados.rados.0.smithi090.stdout:149: oid 15 updating version 0 to 33 2023-01-15T14:28:48.472 INFO:tasks.rados.rados.0.smithi090.stdout:149: oid 15 updating version 33 to 34 2023-01-15T14:28:48.472 INFO:tasks.rados.rados.0.smithi090.stdout:149: oid 15 version 34 is already newer than 32 2023-01-15T14:28:48.472 INFO:tasks.rados.rados.0.smithi090.stdout:149: oid 15 version 34 is already newer than 31 2023-01-15T14:28:48.472 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 34 (ObjNum 82 snap 10 seq_num 82) dirty exists 2023-01-15T14:28:48.472 INFO:tasks.rados.rados.0.smithi090.stdout:149: left oid 15 (ObjNum 82 snap 10 seq_num 82) 2023-01-15T14:28:48.473 INFO:tasks.rados.rados.0.smithi090.stdout:149: done (0 left) 2023-01-15T14:28:48.474 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:28:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:28:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:28:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:28:48.474 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:28:48.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:28:48.475 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:48.469+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:28:48.575 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:48.569+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:48.642 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:28:48.642 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:28:48.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:28:48.642 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:28:48.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.642 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:18 3.8 3:17d5d741:::smithi09025264-5:head [write 1717575~349653] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:28:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:25.911853+0000", 2023-01-15T14:28:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.717069057, 2023-01-15T14:28:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.54398078999999999, 2023-01-15T14:28:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 18 2023-01-15T14:28:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:48.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:48.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.911853+0000", 2023-01-15T14:28:48.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.911853+0000", 2023-01-15T14:28:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.911852+0000", 2023-01-15T14:28:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.912100+0000", 2023-01-15T14:28:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024813899999999998 2023-01-15T14:28:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.912112+0000", 2023-01-15T14:28:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1586000000000001e-05 2023-01-15T14:28:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.012214+0000", 2023-01-15T14:28:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102278 2023-01-15T14:28:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.012307+0000", 2023-01-15T14:28:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2896e-05 2023-01-15T14:28:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.012457+0000", 2023-01-15T14:28:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149958 2023-01-15T14:28:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.251306+0000", 2023-01-15T14:28:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23884853 2023-01-15T14:28:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.251433+0000", 2023-01-15T14:28:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012694400000000001 2023-01-15T14:28:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:28:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.251635+0000", 2023-01-15T14:28:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020235500000000001 2023-01-15T14:28:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.251979+0000", 2023-01-15T14:28:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034400799999999998 2023-01-15T14:28:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.252006+0000", 2023-01-15T14:28:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6789999999999999e-05 2023-01-15T14:28:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.252386+0000", 2023-01-15T14:28:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037994899999999998 2023-01-15T14:28:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.253008+0000", 2023-01-15T14:28:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062204000000000003 2023-01-15T14:28:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.315519+0000", 2023-01-15T14:28:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062511019000000001 2023-01-15T14:28:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.455655+0000", 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14013627200000001 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.455729+0000", 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4264000000000006e-05 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.455834+0000", 2023-01-15T14:28:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104429 2023-01-15T14:28:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:19 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (55) in=7b,truncate 2067228] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:28:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.012252+0000", 2023-01-15T14:28:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.616670469999999, 2023-01-15T14:28:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.54358349299999997, 2023-01-15T14:28:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 19 2023-01-15T14:28:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.012252+0000", 2023-01-15T14:28:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.012252+0000", 2023-01-15T14:28:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.012249+0000", 2023-01-15T14:28:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:28:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.012262+0000", 2023-01-15T14:28:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2086000000000001e-05 2023-01-15T14:28:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.012281+0000", 2023-01-15T14:28:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9539e-05 2023-01-15T14:28:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112374+0000", 2023-01-15T14:28:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100093083 2023-01-15T14:28:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112486+0000", 2023-01-15T14:28:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112197 2023-01-15T14:28:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112696+0000", 2023-01-15T14:28:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020946199999999999 2023-01-15T14:28:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.252138+0000", 2023-01-15T14:28:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.139442548 2023-01-15T14:28:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.252250+0000", 2023-01-15T14:28:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011189299999999999 2023-01-15T14:28:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:28:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.252436+0000", 2023-01-15T14:28:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185768 2023-01-15T14:28:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.252697+0000", 2023-01-15T14:28:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026125900000000001 2023-01-15T14:28:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.252742+0000", 2023-01-15T14:28:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4354999999999998e-05 2023-01-15T14:28:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.252817+0000", 2023-01-15T14:28:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5576999999999998e-05 2023-01-15T14:28:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.253024+0000", 2023-01-15T14:28:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020725799999999999 2023-01-15T14:28:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414834+0000", 2023-01-15T14:28:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.161809339 2023-01-15T14:28:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.555756+0000", 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.140922245 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.555795+0000", 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9316000000000001e-05 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.555835+0000", 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9663999999999998e-05 2023-01-15T14:28:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:140 3.8 3:18f3951e:::smithi09025264-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.798272+0000", 2023-01-15T14:28:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.830650227, 2023-01-15T14:28:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.53498803399999995, 2023-01-15T14:28:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 140 2023-01-15T14:28:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.798272+0000", 2023-01-15T14:28:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.798272+0000", 2023-01-15T14:28:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.798271+0000", 2023-01-15T14:28:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.798275+0000", 2023-01-15T14:28:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3329999999999998e-06 2023-01-15T14:28:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.798285+0000", 2023-01-15T14:28:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4739999999999994e-06 2023-01-15T14:28:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:48.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.798292+0000", 2023-01-15T14:28:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.956e-06 2023-01-15T14:28:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.896393+0000", 2023-01-15T14:28:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098100755999999997 2023-01-15T14:28:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.896522+0000", 2023-01-15T14:28:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012919099999999999 2023-01-15T14:28:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332982+0000", 2023-01-15T14:28:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.43646017999999998 2023-01-15T14:28:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333099+0000", 2023-01-15T14:28:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116857 2023-01-15T14:28:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333260+0000", 2023-01-15T14:28:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161034 2023-01-15T14:28:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:166 3.8 3:1e54a7f5:::smithi09025264-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:28:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.514207026000001, 2023-01-15T14:28:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62664632300000001, 2023-01-15T14:28:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 166 2023-01-15T14:28:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:28:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:48.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:28:48.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114713+0000", 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115178+0000", 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046493199999999999 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115208+0000", 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0689999999999999e-05 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115221+0000", 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3013000000000001e-05 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115334+0000", 2023-01-15T14:28:48.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112403 2023-01-15T14:28:48.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:48.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115505+0000", 2023-01-15T14:28:48.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170915 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333496+0000", 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217991667 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333606+0000", 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109967 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:28:48.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333796+0000", 2023-01-15T14:28:48.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018928099999999999 2023-01-15T14:28:48.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:48.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334289+0000", 2023-01-15T14:28:48.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049322600000000004 2023-01-15T14:28:48.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:48.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334339+0000", 2023-01-15T14:28:48.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0439999999999998e-05 2023-01-15T14:28:48.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:48.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334990+0000", 2023-01-15T14:28:48.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065082300000000003 2023-01-15T14:28:48.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:48.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335100+0000", 2023-01-15T14:28:48.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109357 2023-01-15T14:28:48.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:48.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.640854+0000", 2023-01-15T14:28:48.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30575451300000001 2023-01-15T14:28:48.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741190+0000", 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100335568 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741263+0000", 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3275999999999994e-05 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741361+0000", 2023-01-15T14:28:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8435000000000002e-05 2023-01-15T14:28:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:167 3.8 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:28:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.513642337, 2023-01-15T14:28:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62638390099999997, 2023-01-15T14:28:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 167 2023-01-15T14:28:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:28:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115279+0000", 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115289+0000", 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0509999999999999e-05 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115306+0000", 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7260000000000001e-05 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115337+0000", 2023-01-15T14:28:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0664000000000003e-05 2023-01-15T14:28:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115579+0000", 2023-01-15T14:28:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024194400000000001 2023-01-15T14:28:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115744+0000", 2023-01-15T14:28:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016493700000000001 2023-01-15T14:28:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334392+0000", 2023-01-15T14:28:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21864830299999999 2023-01-15T14:28:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334453+0000", 2023-01-15T14:28:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1047e-05 2023-01-15T14:28:49.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:28:49.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334601+0000", 2023-01-15T14:28:49.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014824700000000001 2023-01-15T14:28:49.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:49.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334812+0000", 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210208 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334996+0000", 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183878 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335126+0000", 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013001399999999999 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:49.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335160+0000", 2023-01-15T14:28:49.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4876999999999998e-05 2023-01-15T14:28:49.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:49.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741044+0000", 2023-01-15T14:28:49.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40588326000000002 2023-01-15T14:28:49.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:49.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741453+0000", 2023-01-15T14:28:49.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040963099999999998 2023-01-15T14:28:49.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:49.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741548+0000", 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4961000000000001e-05 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741664+0000", 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115333 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.118 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:49.119 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:28:49.119 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.513550222999999, 2023-01-15T14:28:49.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:28:49.119 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:49.119 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:49.119 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:49.119 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:49.119 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:49.119 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:28:49.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.119 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:49.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:49.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:28:49.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:49.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:28:49.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:28:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:28:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:28:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:28:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:28:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:28:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:28:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:28:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:28:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:28:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:28:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:28:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:28:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:28:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:28:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:28:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:28:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:28:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:28:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:28:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:28:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:28:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.500558143999999, 2023-01-15T14:28:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:28:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:28:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:28:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:28:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:28:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:28:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:49.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:28:49.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:28:49.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:49.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:28:49.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:28:49.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:49.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:28:49.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:28:49.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:28:49.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:28:49.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:49.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:28:49.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:28:49.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:28:49.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:28:49.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:49.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:28:49.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:28:49.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:28:49.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:28:49.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:28:49.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:49.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:28:49.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:28:49.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:49.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:28:49.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:28:49.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:49.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:28:49.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:28:49.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:49.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:28:49.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:28:49.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:49.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:28:49.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:28:49.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:49.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:28:49.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:28:49.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:49.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:28:49.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:28:49.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:49.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:28:49.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:28:49.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.286 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:49.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.287 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:49.287 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:28:49.287 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.500154808000001, 2023-01-15T14:28:49.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:28:49.288 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:49.288 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:49.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:49.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:49.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:49.288 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:28:49.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.288 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:49.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:49.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:28:49.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:49.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:28:49.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:49.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:28:49.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:49.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:49.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:28:49.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:28:49.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:49.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:28:49.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:28:49.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:49.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:28:49.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:28:49.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:28:49.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:28:49.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:49.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:28:49.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:28:49.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:28:49.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:28:49.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:49.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:28:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:28:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:28:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:28:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:28:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:28:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:28:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:28:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:28:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:28:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:28:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:28:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:28:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:28:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:28:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:28:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:28:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:28:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:28:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:28:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:28:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:28:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.399931188, 2023-01-15T14:28:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:28:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:28:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:28:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:28:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:28:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:28:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:28:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:28:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:28:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:49.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:28:49.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:28:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:193 3.8 3:1eaad226:::smithi09025264-49:head [write 406736~710070,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.294040+0000", 2023-01-15T14:28:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.334882294, 2023-01-15T14:28:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.54804442399999997, 2023-01-15T14:28:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2023-01-15T14:28:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294040+0000", 2023-01-15T14:28:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294040+0000", 2023-01-15T14:28:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294037+0000", 2023-01-15T14:28:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:28:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294382+0000", 2023-01-15T14:28:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034469500000000001 2023-01-15T14:28:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294421+0000", 2023-01-15T14:28:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9597000000000001e-05 2023-01-15T14:28:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294441+0000", 2023-01-15T14:28:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0000000000000002e-05 2023-01-15T14:28:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335292+0000", 2023-01-15T14:28:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040850274999999998 2023-01-15T14:28:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335605+0000", 2023-01-15T14:28:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031334499999999998 2023-01-15T14:28:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:28:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335831+0000", 2023-01-15T14:28:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000225961 2023-01-15T14:28:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.336189+0000", 2023-01-15T14:28:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035799099999999999 2023-01-15T14:28:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.336207+0000", 2023-01-15T14:28:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7808e-05 2023-01-15T14:28:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.336471+0000", 2023-01-15T14:28:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026395500000000002 2023-01-15T14:28:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.336516+0000", 2023-01-15T14:28:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5318999999999998e-05 2023-01-15T14:28:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:49.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741140+0000", 2023-01-15T14:28:49.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40462405200000001 2023-01-15T14:28:49.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:49.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841833+0000", 2023-01-15T14:28:49.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100693281 2023-01-15T14:28:49.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:49.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841932+0000", 2023-01-15T14:28:49.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8387999999999994e-05 2023-01-15T14:28:49.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:49.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842084+0000", 2023-01-15T14:28:49.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015241699999999999 2023-01-15T14:28:49.861 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.861 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:49.861 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:49.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.864 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:194 3.8 3:1eaad226:::smithi09025264-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:49.865 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:28:49.865 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.327529944999998, 2023-01-15T14:28:49.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64372577200000003, 2023-01-15T14:28:49.866 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:49.868 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:49.868 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:49.868 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:49.868 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:49.868 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 194 2023-01-15T14:28:49.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.869 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:49.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:49.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:28:49.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:49.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:28:49.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:49.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:49.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301390+0000", 2023-01-15T14:28:49.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:28:49.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:49.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301991+0000", 2023-01-15T14:28:49.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060122599999999995 2023-01-15T14:28:49.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:49.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.302026+0000", 2023-01-15T14:28:49.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5040000000000003e-05 2023-01-15T14:28:49.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:49.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402147+0000", 2023-01-15T14:28:49.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100121084 2023-01-15T14:28:49.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402219+0000", 2023-01-15T14:28:49.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2045999999999994e-05 2023-01-15T14:28:49.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:49.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402315+0000", 2023-01-15T14:28:49.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5561000000000002e-05 2023-01-15T14:28:49.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:49.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842189+0000", 2023-01-15T14:28:49.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.439874129 2023-01-15T14:28:49.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:49.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842296+0000", 2023-01-15T14:28:49.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107021 2023-01-15T14:28:49.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:28:49.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842486+0000", 2023-01-15T14:28:49.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018965799999999999 2023-01-15T14:28:49.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:49.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842839+0000", 2023-01-15T14:28:49.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035330700000000002 2023-01-15T14:28:49.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:49.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842859+0000", 2023-01-15T14:28:49.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0347e-05 2023-01-15T14:28:49.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:49.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843212+0000", 2023-01-15T14:28:49.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352534 2023-01-15T14:28:49.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:49.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843835+0000", 2023-01-15T14:28:49.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062338599999999997 2023-01-15T14:28:49.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:49.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944861+0000", 2023-01-15T14:28:49.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10102596899999999 2023-01-15T14:28:49.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:49.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944908+0000", 2023-01-15T14:28:49.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6773000000000001e-05 2023-01-15T14:28:49.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:49.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:49.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945039+0000", 2023-01-15T14:28:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130684 2023-01-15T14:28:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945118+0000", 2023-01-15T14:28:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9194000000000001e-05 2023-01-15T14:28:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:195 3.8 3:1eaad226:::smithi09025264-49:head [setxattr _header (56) in=7b,truncate 2806736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.402188+0000", 2023-01-15T14:28:50.018 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.226734075, 2023-01-15T14:28:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60706172400000002, 2023-01-15T14:28:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 195 2023-01-15T14:28:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.019 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402188+0000", 2023-01-15T14:28:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402188+0000", 2023-01-15T14:28:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402186+0000", 2023-01-15T14:28:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:28:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402199+0000", 2023-01-15T14:28:50.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.24e-05 2023-01-15T14:28:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402222+0000", 2023-01-15T14:28:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-05 2023-01-15T14:28:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402233+0000", 2023-01-15T14:28:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2023-01-15T14:28:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402389+0000", 2023-01-15T14:28:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015541799999999999 2023-01-15T14:28:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402518+0000", 2023-01-15T14:28:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129534 2023-01-15T14:28:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842949+0000", 2023-01-15T14:28:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.440430722 2023-01-15T14:28:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843094+0000", 2023-01-15T14:28:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144955 2023-01-15T14:28:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:28:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843391+0000", 2023-01-15T14:28:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000297041 2023-01-15T14:28:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843731+0000", 2023-01-15T14:28:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033986200000000002 2023-01-15T14:28:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843747+0000", 2023-01-15T14:28:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6243999999999999e-05 2023-01-15T14:28:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843804+0000", 2023-01-15T14:28:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7240000000000001e-05 2023-01-15T14:28:50.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843883+0000", 2023-01-15T14:28:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9040000000000002e-05 2023-01-15T14:28:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945184+0000", 2023-01-15T14:28:50.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101300903 2023-01-15T14:28:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009101+0000", 2023-01-15T14:28:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063916599000000004 2023-01-15T14:28:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009176+0000", 2023-01-15T14:28:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5716999999999994e-05 2023-01-15T14:28:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009250+0000", 2023-01-15T14:28:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3213000000000005e-05 2023-01-15T14:28:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:50.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:196 3.8 3:1eaad226:::smithi09025264-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.402249+0000", 2023-01-15T14:28:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.226673228999999, 2023-01-15T14:28:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60743240399999998, 2023-01-15T14:28:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:50.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:50.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 196 2023-01-15T14:28:50.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402249+0000", 2023-01-15T14:28:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402249+0000", 2023-01-15T14:28:50.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402248+0000", 2023-01-15T14:28:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402253+0000", 2023-01-15T14:28:50.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8829999999999997e-06 2023-01-15T14:28:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402263+0000", 2023-01-15T14:28:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0013999999999999e-05 2023-01-15T14:28:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402269+0000", 2023-01-15T14:28:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1360000000000002e-06 2023-01-15T14:28:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402551+0000", 2023-01-15T14:28:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000281799 2023-01-15T14:28:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402620+0000", 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9289999999999996e-05 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.942361+0000", 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.53974109800000003 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.942474+0000", 2023-01-15T14:28:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112713 2023-01-15T14:28:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009383+0000", 2023-01-15T14:28:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.066909386000000001 2023-01-15T14:28:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009511+0000", 2023-01-15T14:28:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012762 2023-01-15T14:28:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009681+0000", 2023-01-15T14:28:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017044099999999999 2023-01-15T14:28:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:267 3.7 3:e089c230:::smithi09025264-30:head [setxattr _header (56) in=7b,truncate 2974806] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:34.869815+0000", 2023-01-15T14:28:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.759106616, 2023-01-15T14:28:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52709425300000001, 2023-01-15T14:28:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 267 2023-01-15T14:28:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.185 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:50.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869815+0000", 2023-01-15T14:28:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869815+0000", 2023-01-15T14:28:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869814+0000", 2023-01-15T14:28:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869821+0000", 2023-01-15T14:28:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6830000000000002e-06 2023-01-15T14:28:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869836+0000", 2023-01-15T14:28:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.488e-05 2023-01-15T14:28:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869844+0000", 2023-01-15T14:28:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.61e-06 2023-01-15T14:28:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.955927+0000", 2023-01-15T14:28:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.086083326000000002 2023-01-15T14:28:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.956069+0000", 2023-01-15T14:28:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000142569 2023-01-15T14:28:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:28:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.956278+0000", 2023-01-15T14:28:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0002086 2023-01-15T14:28:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.956450+0000", 2023-01-15T14:28:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172139 2023-01-15T14:28:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.956470+0000", 2023-01-15T14:28:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9423e-05 2023-01-15T14:28:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.956533+0000", 2023-01-15T14:28:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3503000000000005e-05 2023-01-15T14:28:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.956673+0000", 2023-01-15T14:28:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013997099999999999 2023-01-15T14:28:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.057614+0000", 2023-01-15T14:28:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10094116 2023-01-15T14:28:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.396810+0000", 2023-01-15T14:28:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.33919538399999999 2023-01-15T14:28:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.396854+0000", 2023-01-15T14:28:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4184999999999998e-05 2023-01-15T14:28:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.396910+0000", 2023-01-15T14:28:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5800000000000001e-05 2023-01-15T14:28:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:268 3.7 3:e089c230:::smithi09025264-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:34.869853+0000", 2023-01-15T14:28:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.759068873, 2023-01-15T14:28:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52726519599999999, 2023-01-15T14:28:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 268 2023-01-15T14:28:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869853+0000", 2023-01-15T14:28:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869853+0000", 2023-01-15T14:28:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869852+0000", 2023-01-15T14:28:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869857+0000", 2023-01-15T14:28:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2570000000000001e-06 2023-01-15T14:28:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869870+0000", 2023-01-15T14:28:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3716999999999999e-05 2023-01-15T14:28:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.869886+0000", 2023-01-15T14:28:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.558e-05 2023-01-15T14:28:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.956565+0000", 2023-01-15T14:28:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.086678701999999996 2023-01-15T14:28:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.956654+0000", 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9392000000000001e-05 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.396969+0000", 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.44031461999999999 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.397037+0000", 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8119000000000003e-05 2023-01-15T14:28:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.397118+0000", 2023-01-15T14:28:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1606000000000004e-05 2023-01-15T14:28:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:270 3.2 3:47dfe935:::smithi09025264-29:head [write 1719614~417353] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:34.881457+0000", 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.747465511, 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50216781799999999, 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 270 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.881457+0000", 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.881457+0000", 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.881454+0000", 2023-01-15T14:28:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:28:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.881666+0000", 2023-01-15T14:28:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151000000000001 2023-01-15T14:28:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.881700+0000", 2023-01-15T14:28:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4446999999999997e-05 2023-01-15T14:28:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.881715+0000", 2023-01-15T14:28:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4250000000000001e-05 2023-01-15T14:28:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.881826+0000", 2023-01-15T14:28:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011184300000000001 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.881992+0000", 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000166007 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.081082+0000", 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199089552 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.081179+0000", 2023-01-15T14:28:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7183999999999997e-05 2023-01-15T14:28:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:28:50.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.081336+0000", 2023-01-15T14:28:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015696899999999999 2023-01-15T14:28:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.081536+0000", 2023-01-15T14:28:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020015699999999999 2023-01-15T14:28:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:50.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.081554+0000", 2023-01-15T14:28:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7499999999999998e-05 2023-01-15T14:28:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.081793+0000", 2023-01-15T14:28:50.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023954600000000001 2023-01-15T14:28:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.082989+0000", 2023-01-15T14:28:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011956270000000001 2023-01-15T14:28:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.084665+0000", 2023-01-15T14:28:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001675756 2023-01-15T14:28:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.383512+0000", 2023-01-15T14:28:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29884680000000002 2023-01-15T14:28:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.383570+0000", 2023-01-15T14:28:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8798000000000002e-05 2023-01-15T14:28:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:50.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.383624+0000", 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3974999999999997e-05 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:271 3.2 3:47dfe935:::smithi09025264-29:head [write 2570780~420833] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:34.886236+0000", 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.742686424, 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49771737999999999, 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:50.524 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 271 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886236+0000", 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886236+0000", 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886233+0000", 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:50.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886581+0000", 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034827000000000002 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886621+0000", 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9509999999999999e-05 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886635+0000", 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4250000000000001e-05 2023-01-15T14:28:50.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.986903+0000", 2023-01-15T14:28:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100267908 2023-01-15T14:28:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.987058+0000", 2023-01-15T14:28:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015458 2023-01-15T14:28:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.081670+0000", 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094611838000000004 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.081858+0000", 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018873100000000001 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:28:50.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.082103+0000", 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000244811 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.082470+0000", 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036684 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.082487+0000", 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7167e-05 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.082740+0000", 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025322300000000001 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.083019+0000", 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027857000000000001 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.084886+0000", 2023-01-15T14:28:50.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001867049 2023-01-15T14:28:50.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:50.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.383743+0000", 2023-01-15T14:28:50.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29885742300000001 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.383836+0000", 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2236000000000003e-05 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.383953+0000", 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117407 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:50.733 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:273 3.2 3:47dfe935:::smithi09025264-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:34.886733+0000", 2023-01-15T14:28:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.742189379999999, 2023-01-15T14:28:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49764941400000001, 2023-01-15T14:28:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 273 2023-01-15T14:28:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886733+0000", 2023-01-15T14:28:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886733+0000", 2023-01-15T14:28:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886732+0000", 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886737+0000", 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7759999999999997e-06 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886757+0000", 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0157e-05 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886763+0000", 2023-01-15T14:28:52.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3230000000000003e-06 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.987668+0000", 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10090472 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.987925+0000", 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025695999999999998 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.083076+0000", 2023-01-15T14:28:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095151461000000007 2023-01-15T14:28:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.083238+0000", 2023-01-15T14:28:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016132200000000001 2023-01-15T14:28:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.384191+0000", 2023-01-15T14:28:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30095322499999999 2023-01-15T14:28:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.384284+0000", 2023-01-15T14:28:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3387000000000006e-05 2023-01-15T14:28:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.384382+0000", 2023-01-15T14:28:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7853e-05 2023-01-15T14:28:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:277 3.8 3:1e54a7f5:::smithi09025264-42:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:52.298 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:34.886790+0000", 2023-01-15T14:28:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.74213241, 2023-01-15T14:28:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.54036727399999995, 2023-01-15T14:28:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 277 2023-01-15T14:28:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:52.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886790+0000", 2023-01-15T14:28:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886790+0000", 2023-01-15T14:28:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886789+0000", 2023-01-15T14:28:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886794+0000", 2023-01-15T14:28:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.13e-06 2023-01-15T14:28:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886808+0000", 2023-01-15T14:28:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4277000000000001e-05 2023-01-15T14:28:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886817+0000", 2023-01-15T14:28:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7530000000000007e-06 2023-01-15T14:28:52.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.951421+0000", 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.064603586000000005 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.951584+0000", 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163416 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:28:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.317704+0000", 2023-01-15T14:28:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.366119694 2023-01-15T14:28:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.319016+0000", 2023-01-15T14:28:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013117319999999999 2023-01-15T14:28:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:52.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.319034+0000", 2023-01-15T14:28:52.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8300000000000001e-05 2023-01-15T14:28:52.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.321344+0000", 2023-01-15T14:28:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023099850000000001 2023-01-15T14:28:52.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.321387+0000", 2023-01-15T14:28:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3034999999999997e-05 2023-01-15T14:28:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.426907+0000", 2023-01-15T14:28:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105519687 2023-01-15T14:28:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.426974+0000", 2023-01-15T14:28:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7288999999999999e-05 2023-01-15T14:28:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.427052+0000", 2023-01-15T14:28:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8640000000000006e-05 2023-01-15T14:28:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.427157+0000", 2023-01-15T14:28:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104454 2023-01-15T14:28:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:278 3.8 3:1e54a7f5:::smithi09025264-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:28:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:34.886841+0000", 2023-01-15T14:28:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.742080664, 2023-01-15T14:28:52.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.54068689199999997, 2023-01-15T14:28:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 278 2023-01-15T14:28:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:52.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886841+0000", 2023-01-15T14:28:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886841+0000", 2023-01-15T14:28:52.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886840+0000", 2023-01-15T14:28:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:52.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886846+0000", 2023-01-15T14:28:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8270000000000003e-06 2023-01-15T14:28:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.886877+0000", 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0827000000000001e-05 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.987012+0000", 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100135272 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.087218+0000", 2023-01-15T14:28:52.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100205815 2023-01-15T14:28:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:28:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.087329+0000", 2023-01-15T14:28:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011048300000000001 2023-01-15T14:28:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.319223+0000", 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23189444100000001 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.319366+0000", 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014332900000000001 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.427230+0000", 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10786367500000001 2023-01-15T14:28:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.427359+0000", 2023-01-15T14:28:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012859300000000001 2023-01-15T14:28:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.427528+0000", 2023-01-15T14:28:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016966 2023-01-15T14:28:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:52.609 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:28:52.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:28:52.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:28:52.610 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:28:52.610 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:28:52.611 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:28:52.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2023-01-15T14:28:52.613 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:52.605+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:28:52.614 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:28:52.626 INFO:tasks.rados.rados.0.smithi090.stdout:150: done (0 left) 2023-01-15T14:28:52.759 INFO:tasks.rados.rados.0.smithi090.stdout:151: read oid 29 snap 10 2023-01-15T14:28:52.759 INFO:tasks.rados.rados.0.smithi090.stdout:151: expect (ObjNum 78 snap 10 seq_num 78) 2023-01-15T14:28:52.759 INFO:tasks.rados.rados.0.smithi090.stdout:152: write (excl) oid 21 current snap is 11 2023-01-15T14:28:52.759 INFO:tasks.rados.rados.0.smithi090.stdout:152: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2023-01-15T14:28:52.759 INFO:tasks.rados.rados.0.smithi090.stdout:152: writing smithi09025264-21 from 594981 to 1204178 tid 1 2023-01-15T14:28:52.759 INFO:tasks.rados.rados.0.smithi090.stdout:152: writing smithi09025264-21 from 1974123 to 2719151 tid 2 2023-01-15T14:28:52.759 INFO:tasks.rados.rados.0.smithi090.stdout:152: writing smithi09025264-21 from 3279873 to 3394981 tid 3 2023-01-15T14:28:52.760 INFO:tasks.rados.rados.0.smithi090.stdout:153: delete oid 23 current snap is 11 2023-01-15T14:28:52.760 INFO:tasks.rados.rados.0.smithi090.stdout:152: finishing write tid 1 to smithi09025264-21 2023-01-15T14:28:52.760 INFO:tasks.rados.rados.0.smithi090.stdout:152: finishing write tid 2 to smithi09025264-21 2023-01-15T14:28:52.760 INFO:tasks.rados.rados.0.smithi090.stdout:153: done (2 left) 2023-01-15T14:28:52.760 INFO:tasks.rados.rados.0.smithi090.stdout:154: read oid 1 snap 8 2023-01-15T14:28:52.760 INFO:tasks.rados.rados.0.smithi090.stdout:154: expect (ObjNum 0 snap 0 seq_num 0) 2023-01-15T14:28:52.760 INFO:tasks.rados.rados.0.smithi090.stdout:155: copy_from oid 25 from oid 13 current snap is 11 2023-01-15T14:28:52.760 INFO:tasks.rados.rados.0.smithi090.stdout:156: copy_from oid 46 from oid 50 current snap is 11 2023-01-15T14:28:52.760 INFO:tasks.rados.rados.0.smithi090.stdout:157: write oid 16 current snap is 11 2023-01-15T14:28:52.760 INFO:tasks.rados.rados.0.smithi090.stdout:157: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2023-01-15T14:28:52.760 INFO:tasks.rados.rados.0.smithi090.stdout:157: writing smithi09025264-16 from 628595 to 1104643 tid 1 2023-01-15T14:28:52.761 INFO:tasks.rados.rados.0.smithi090.stdout:157: writing smithi09025264-16 from 1807440 to 2439784 tid 2 2023-01-15T14:28:52.761 INFO:tasks.rados.rados.0.smithi090.stdout:157: writing smithi09025264-16 from 2934719 to 3387193 tid 3 2023-01-15T14:28:52.761 INFO:tasks.rados.rados.0.smithi090.stdout:158: read oid 2 snap -1 2023-01-15T14:28:52.761 INFO:tasks.rados.rados.0.smithi090.stdout:158: expect (ObjNum 10 snap 0 seq_num 10) 2023-01-15T14:28:52.761 INFO:tasks.rados.rados.0.smithi090.stdout:159: snap_create 2023-01-15T14:28:52.761 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:28:52.761 INFO:tasks.rados.rados.0.smithi090.stdout:152: finishing write tid 3 to smithi09025264-21 2023-01-15T14:28:52.761 INFO:tasks.rados.rados.0.smithi090.stdout:152: finishing write tid 4 to smithi09025264-21 2023-01-15T14:28:52.761 INFO:tasks.rados.rados.0.smithi090.stdout:156: finishing copy_from to smithi09025264-46 2023-01-15T14:28:52.761 INFO:tasks.rados.rados.0.smithi090.stdout:156: got expected ENOENT (src dne) 2023-01-15T14:28:52.761 INFO:tasks.rados.rados.0.smithi090.stdout:156: finishing copy_from racing read to smithi09025264-46 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout:156: done (6 left) 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout:154: expect (ObjNum 0 snap 0 seq_num 0) 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout:152: finishing write tid 5 to smithi09025264-21 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout:152: oid 21 updating version 0 to 34 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout:152: oid 21 version 34 is already newer than 32 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout:152: oid 21 version 34 is already newer than 33 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout:152: oid 21 updating version 34 to 35 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 35 (ObjNum 83 snap 11 seq_num 83) dirty exists 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout:152: left oid 21 (ObjNum 83 snap 11 seq_num 83) 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout:152: done (5 left) 2023-01-15T14:28:52.762 INFO:tasks.rados.rados.0.smithi090.stdout:154: done (4 left) 2023-01-15T14:28:52.763 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:28:52.763 INFO:tasks.rados.rados.0.smithi090.stdout:158: expect (ObjNum 10 snap 0 seq_num 10) 2023-01-15T14:28:52.763 INFO:tasks.rados.rados.0.smithi090.stdout:158: done (3 left) 2023-01-15T14:28:52.763 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:28:52.763 INFO:tasks.rados.rados.0.smithi090.stdout:151: expect (ObjNum 78 snap 10 seq_num 78) 2023-01-15T14:28:52.763 INFO:tasks.rados.rados.0.smithi090.stdout:155: finishing copy_from to smithi09025264-25 2023-01-15T14:28:52.763 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 43 (ObjNum 74 snap 8 seq_num 74) dirty exists 2023-01-15T14:28:52.764 INFO:tasks.rados.rados.0.smithi090.stdout:155: finishing copy_from racing read to smithi09025264-25 2023-01-15T14:28:52.764 INFO:tasks.rados.rados.0.smithi090.stdout:151: done (2 left) 2023-01-15T14:28:52.764 INFO:tasks.rados.rados.0.smithi090.stdout:155: done (1 left) 2023-01-15T14:28:52.764 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:52.764 INFO:tasks.rados.rados.0.smithi090.stdout:157: finishing write tid 1 to smithi09025264-16 2023-01-15T14:28:52.764 INFO:tasks.rados.rados.0.smithi090.stdout:157: finishing write tid 2 to smithi09025264-16 2023-01-15T14:28:52.764 INFO:tasks.rados.rados.0.smithi090.stdout:157: finishing write tid 3 to smithi09025264-16 2023-01-15T14:28:52.764 INFO:tasks.rados.rados.0.smithi090.stdout:157: finishing write tid 4 to smithi09025264-16 2023-01-15T14:28:52.764 INFO:tasks.rados.rados.0.smithi090.stdout:157: finishing write tid 5 to smithi09025264-16 2023-01-15T14:28:52.765 INFO:tasks.rados.rados.0.smithi090.stdout:157: oid 16 updating version 0 to 29 2023-01-15T14:28:52.765 INFO:tasks.rados.rados.0.smithi090.stdout:157: oid 16 updating version 29 to 30 2023-01-15T14:28:52.765 INFO:tasks.rados.rados.0.smithi090.stdout:157: oid 16 updating version 30 to 31 2023-01-15T14:28:52.765 INFO:tasks.rados.rados.0.smithi090.stdout:157: oid 16 updating version 31 to 32 2023-01-15T14:28:52.765 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 32 (ObjNum 85 snap 11 seq_num 85) dirty exists 2023-01-15T14:28:52.765 INFO:tasks.rados.rados.0.smithi090.stdout:157: left oid 16 (ObjNum 85 snap 11 seq_num 85) 2023-01-15T14:28:52.765 INFO:tasks.rados.rados.0.smithi090.stdout:157: done (0 left) 2023-01-15T14:28:52.765 INFO:tasks.rados.rados.0.smithi090.stdout:159: done (0 left) 2023-01-15T14:28:52.765 INFO:tasks.rados.rados.0.smithi090.stdout:160: rollback oid 38 current snap is 12 2023-01-15T14:28:52.765 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 38 to 8 2023-01-15T14:28:52.765 INFO:tasks.rados.rados.0.smithi090.stdout:161: write oid 33 current snap is 12 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:161: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:161: writing smithi09025264-33 from 645402 to 1196699 tid 1 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:161: writing smithi09025264-33 from 1749569 to 2241924 tid 2 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:161: writing smithi09025264-33 from 2845789 to 3445402 tid 3 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:162: read oid 34 snap 11 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:162: expect deleted 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:163: write (excl) oid 22 current snap is 12 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:163: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:163: writing smithi09025264-22 from 662209 to 1288755 tid 1 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:163: writing smithi09025264-22 from 1808051 to 2444064 tid 2 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:163: writing smithi09025264-22 from 2873212 to 3462209 tid 3 2023-01-15T14:28:52.766 INFO:tasks.rados.rados.0.smithi090.stdout:164: read oid 19 snap -1 2023-01-15T14:28:52.767 INFO:tasks.rados.rados.0.smithi090.stdout:164: expect deleted 2023-01-15T14:28:52.767 INFO:tasks.rados.rados.0.smithi090.stdout:165: delete oid 31 current snap is 12 2023-01-15T14:28:52.767 INFO:tasks.rados.rados.0.smithi090.stdout:160: finishing rollback tid 0 to smithi09025264-38 2023-01-15T14:28:52.767 INFO:tasks.rados.rados.0.smithi090.stdout:160: finishing rollback tid 1 to smithi09025264-38 2023-01-15T14:28:52.767 INFO:tasks.rados.rados.0.smithi090.stdout:160: finishing rollback tid 2 to smithi09025264-38 2023-01-15T14:28:52.767 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 32 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-01-15T14:28:52.767 INFO:tasks.rados.rados.0.smithi090.stdout:161: finishing write tid 1 to smithi09025264-33 2023-01-15T14:28:52.767 INFO:tasks.rados.rados.0.smithi090.stdout:161: finishing write tid 2 to smithi09025264-33 2023-01-15T14:28:52.767 INFO:tasks.rados.rados.0.smithi090.stdout:161: finishing write tid 3 to smithi09025264-33 2023-01-15T14:28:52.767 INFO:tasks.rados.rados.0.smithi090.stdout:160: done (5 left) 2023-01-15T14:28:52.769 INFO:tasks.rados.rados.0.smithi090.stdout:162: done (4 left) 2023-01-15T14:28:52.769 INFO:tasks.rados.rados.0.smithi090.stdout:164: done (3 left) 2023-01-15T14:28:52.769 INFO:tasks.rados.rados.0.smithi090.stdout:165: done (2 left) 2023-01-15T14:28:52.769 INFO:tasks.rados.rados.0.smithi090.stdout:166: write (excl) oid 34 current snap is 12 2023-01-15T14:28:52.769 INFO:tasks.rados.rados.0.smithi090.stdout:166: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2023-01-15T14:28:52.769 INFO:tasks.rados.rados.0.smithi090.stdout:166: writing smithi09025264-34 from 695823 to 1472867 tid 1 2023-01-15T14:28:52.769 INFO:tasks.rados.rados.0.smithi090.stdout:166: writing smithi09025264-34 from 1925015 to 2564697 tid 2 2023-01-15T14:28:52.769 INFO:tasks.rados.rados.0.smithi090.stdout:166: writing smithi09025264-34 from 3328058 to 3495823 tid 3 2023-01-15T14:28:52.769 INFO:tasks.rados.rados.0.smithi090.stdout:167: snap_create 2023-01-15T14:28:52.769 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout:166: finishing write tid 1 to smithi09025264-34 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout:161: finishing write tid 4 to smithi09025264-33 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout:161: finishing write tid 5 to smithi09025264-33 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout:161: oid 33 updating version 0 to 42 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout:161: oid 33 version 42 is already newer than 40 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout:161: oid 33 version 42 is already newer than 41 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout:161: oid 33 version 42 is already newer than 39 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 42 (ObjNum 86 snap 12 seq_num 86) dirty exists 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout:161: left oid 33 (ObjNum 86 snap 12 seq_num 86) 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout:161: done (2 left) 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:28:52.770 INFO:tasks.rados.rados.0.smithi090.stdout:166: finishing write tid 2 to smithi09025264-34 2023-01-15T14:28:52.771 INFO:tasks.rados.rados.0.smithi090.stdout:166: finishing write tid 3 to smithi09025264-34 2023-01-15T14:28:52.771 INFO:tasks.rados.rados.0.smithi090.stdout:163: finishing write tid 1 to smithi09025264-22 2023-01-15T14:28:52.771 INFO:tasks.rados.rados.0.smithi090.stdout:166: finishing write tid 4 to smithi09025264-34 2023-01-15T14:28:52.771 INFO:tasks.rados.rados.0.smithi090.stdout:166: finishing write tid 5 to smithi09025264-34 2023-01-15T14:28:52.771 INFO:tasks.rados.rados.0.smithi090.stdout:166: oid 34 updating version 0 to 34 2023-01-15T14:28:52.771 INFO:tasks.rados.rados.0.smithi090.stdout:166: oid 34 version 34 is already newer than 33 2023-01-15T14:28:52.771 INFO:tasks.rados.rados.0.smithi090.stdout:166: oid 34 updating version 34 to 35 2023-01-15T14:28:52.771 INFO:tasks.rados.rados.0.smithi090.stdout:166: oid 34 updating version 35 to 36 2023-01-15T14:28:52.771 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 36 (ObjNum 89 snap 12 seq_num 89) dirty exists 2023-01-15T14:28:52.771 INFO:tasks.rados.rados.0.smithi090.stdout:166: left oid 34 (ObjNum 89 snap 12 seq_num 89) 2023-01-15T14:28:52.772 INFO:tasks.rados.rados.0.smithi090.stdout:166: done (1 left) 2023-01-15T14:28:52.772 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:52.772 INFO:tasks.rados.rados.0.smithi090.stdout:163: finishing write tid 2 to smithi09025264-22 2023-01-15T14:28:52.772 INFO:tasks.rados.rados.0.smithi090.stdout:163: finishing write tid 3 to smithi09025264-22 2023-01-15T14:28:52.772 INFO:tasks.rados.rados.0.smithi090.stdout:163: finishing write tid 4 to smithi09025264-22 2023-01-15T14:28:52.772 INFO:tasks.rados.rados.0.smithi090.stdout:163: finishing write tid 5 to smithi09025264-22 2023-01-15T14:28:52.772 INFO:tasks.rados.rados.0.smithi090.stdout:163: oid 22 updating version 0 to 37 2023-01-15T14:28:52.772 INFO:tasks.rados.rados.0.smithi090.stdout:163: oid 22 updating version 37 to 38 2023-01-15T14:28:52.772 INFO:tasks.rados.rados.0.smithi090.stdout:163: oid 22 updating version 38 to 39 2023-01-15T14:28:52.772 INFO:tasks.rados.rados.0.smithi090.stdout:163: oid 22 updating version 39 to 40 2023-01-15T14:28:52.773 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 40 (ObjNum 87 snap 12 seq_num 87) dirty exists 2023-01-15T14:28:52.773 INFO:tasks.rados.rados.0.smithi090.stdout:163: left oid 22 (ObjNum 87 snap 12 seq_num 87) 2023-01-15T14:28:52.773 INFO:tasks.rados.rados.0.smithi090.stdout:163: done (0 left) 2023-01-15T14:28:52.773 INFO:tasks.rados.rados.0.smithi090.stdout:167: done (0 left) 2023-01-15T14:28:52.773 INFO:tasks.rados.rados.0.smithi090.stdout:168: snap_create 2023-01-15T14:28:52.773 INFO:tasks.rados.rados.0.smithi090.stdout:168: done (0 left) 2023-01-15T14:28:52.773 INFO:tasks.rados.rados.0.smithi090.stdout:169: write (excl) oid 5 current snap is 14 2023-01-15T14:28:52.773 INFO:tasks.rados.rados.0.smithi090.stdout:169: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2023-01-15T14:28:52.921 INFO:tasks.rados.rados.0.smithi090.stdout:169: writing smithi09025264-5 from 712630 to 1164923 tid 1 2023-01-15T14:28:52.921 INFO:tasks.rados.rados.0.smithi090.stdout:169: writing smithi09025264-5 from 1583497 to 2083190 tid 2 2023-01-15T14:28:52.921 INFO:tasks.rados.rados.0.smithi090.stdout:169: writing smithi09025264-5 from 2555481 to 2980668 tid 3 2023-01-15T14:28:52.921 INFO:tasks.rados.rados.0.smithi090.stdout:170: read oid 22 snap -1 2023-01-15T14:28:52.922 INFO:tasks.rados.rados.0.smithi090.stdout:170: expect (ObjNum 87 snap 12 seq_num 87) 2023-01-15T14:28:52.922 INFO:tasks.rados.rados.0.smithi090.stdout:171: snap_remove snap 12 2023-01-15T14:28:52.924 INFO:tasks.rados.rados.0.smithi090.stdout:170: expect (ObjNum 87 snap 12 seq_num 87) 2023-01-15T14:28:52.924 INFO:tasks.rados.rados.0.smithi090.stdout:169: finishing write tid 1 to smithi09025264-5 2023-01-15T14:28:52.924 INFO:tasks.rados.rados.0.smithi090.stdout:169: finishing write tid 2 to smithi09025264-5 2023-01-15T14:28:52.924 INFO:tasks.rados.rados.0.smithi090.stdout:169: finishing write tid 3 to smithi09025264-5 2023-01-15T14:28:52.924 INFO:tasks.rados.rados.0.smithi090.stdout:169: finishing write tid 4 to smithi09025264-5 2023-01-15T14:28:52.924 INFO:tasks.rados.rados.0.smithi090.stdout:169: finishing write tid 5 to smithi09025264-5 2023-01-15T14:28:52.925 INFO:tasks.rados.rados.0.smithi090.stdout:169: oid 5 updating version 0 to 25 2023-01-15T14:28:52.925 INFO:tasks.rados.rados.0.smithi090.stdout:169: oid 5 updating version 25 to 26 2023-01-15T14:28:52.925 INFO:tasks.rados.rados.0.smithi090.stdout:169: oid 5 updating version 26 to 27 2023-01-15T14:28:52.925 INFO:tasks.rados.rados.0.smithi090.stdout:169: oid 5 version 27 is already newer than 24 2023-01-15T14:28:52.925 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 27 (ObjNum 90 snap 14 seq_num 90) dirty exists 2023-01-15T14:28:52.926 INFO:tasks.rados.rados.0.smithi090.stdout:169: left oid 5 (ObjNum 90 snap 14 seq_num 90) 2023-01-15T14:28:52.926 INFO:tasks.rados.rados.0.smithi090.stdout:169: done (2 left) 2023-01-15T14:28:52.927 INFO:tasks.rados.rados.0.smithi090.stdout:170: done (1 left) 2023-01-15T14:28:52.927 INFO:tasks.rados.rados.0.smithi090.stdout:171: done (0 left) 2023-01-15T14:28:52.927 INFO:tasks.rados.rados.0.smithi090.stdout:172: copy_from oid 3 from oid 17 current snap is 14 2023-01-15T14:28:52.927 INFO:tasks.rados.rados.0.smithi090.stdout:173: snap_create 2023-01-15T14:28:52.927 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:52.929 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:52.929 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:28:52.929 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:28:52.929 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:52.930 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:28:52.930 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:52.917+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:52.994 INFO:tasks.rados.rados.0.smithi090.stdout:172: finishing copy_from to smithi09025264-3 2023-01-15T14:28:53.098 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 57 (ObjNum 73 snap 8 seq_num 73) dirty exists 2023-01-15T14:28:53.100 INFO:tasks.rados.rados.0.smithi090.stdout:172: finishing copy_from racing read to smithi09025264-3 2023-01-15T14:28:53.100 INFO:tasks.rados.rados.0.smithi090.stdout:172: done (0 left) 2023-01-15T14:28:53.100 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:28:53.101 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:53.093+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:53.101 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:28:53.101 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:28:53.101 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:28:53.101 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:28:53.102 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:28:53.102 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:28:53.102 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:28:53.132 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:28:53.269 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:28:53.269 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:28:53.270 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-01-15T14:28:53.213+0000 7fb90e2b4700 -1 mon.a@0(leader).osd e47 definitely_dead 0 2023-01-15T14:28:53.270 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:53.270 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:53.271 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:28:53.271 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:28:53.271 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:28:53.271 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:53.261+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:28:53.278 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:28:53.280 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:28:53.280 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:28:53.280 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:28:53.281 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:28:53.281 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:28:53.281 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:28:53.281 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:28:53.281 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:28:53.370 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:53.365+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:53.428 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:53.429 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:53.429 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:28:53.471 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:53.465+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:53.569 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:53.569 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:53.570 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:28:53.571 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:53.569+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:53.672 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:53.665+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:28:53.756 INFO:tasks.rados.rados.0.smithi090.stdout:173: done (0 left) 2023-01-15T14:28:53.756 INFO:tasks.rados.rados.0.smithi090.stdout:174: write (excl) oid 42 current snap is 15 2023-01-15T14:28:53.757 INFO:tasks.rados.rados.0.smithi090.stdout:174: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2023-01-15T14:28:53.759 INFO:teuthology.orchestra.run.smithi090.stderr:marked out osd.5. 2023-01-15T14:28:53.760 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:28:53.762 INFO:tasks.rados.rados.0.smithi090.stdout:174: writing smithi09025264-42 from 729437 to 1256979 tid 1 2023-01-15T14:28:53.762 INFO:tasks.rados.rados.0.smithi090.stdout:174: writing smithi09025264-42 from 1925626 to 2568977 tid 2 2023-01-15T14:28:53.763 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:53.763 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:53.764 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:28:53.766 INFO:tasks.rados.rados.0.smithi090.stdout:174: writing smithi09025264-42 from 3266551 to 3529437 tid 3 2023-01-15T14:28:53.766 INFO:tasks.rados.rados.0.smithi090.stdout:175: copy_from oid 6 from oid 31 current snap is 15 2023-01-15T14:28:53.766 INFO:tasks.rados.rados.0.smithi090.stdout:176: snap_remove snap 13 2023-01-15T14:28:53.811 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:53.805+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:53.934 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:53.935 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:28:53.935 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:53.925+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:54.035 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:54.029+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:28:54.132 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:54.132 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:54.132 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:28:54.136 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:54.129+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:54.318 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:54.318 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:54.318 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:28:54.319 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:54.313+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:28:54.419 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:54.413+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:54.513 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:54.513 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:54.514 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:28:54.519 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:28:54.517+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:28:54.620 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:54.613+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:28:54.700 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:54.701 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:54.701 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:28:54.734 INFO:tasks.rados.rados.0.smithi090.stdout:174: finishing write tid 1 to smithi09025264-42 2023-01-15T14:28:54.888 INFO:tasks.rados.rados.0.smithi090.stdout:174: finishing write tid 2 to smithi09025264-42 2023-01-15T14:28:54.888 INFO:tasks.rados.rados.0.smithi090.stdout:174: finishing write tid 3 to smithi09025264-42 2023-01-15T14:28:54.888 INFO:tasks.rados.rados.0.smithi090.stdout:174: finishing write tid 4 to smithi09025264-42 2023-01-15T14:28:54.888 INFO:tasks.rados.rados.0.smithi090.stdout:174: finishing write tid 5 to smithi09025264-42 2023-01-15T14:28:54.888 INFO:tasks.rados.rados.0.smithi090.stdout:174: oid 42 updating version 0 to 32 2023-01-15T14:28:54.888 INFO:tasks.rados.rados.0.smithi090.stdout:174: oid 42 version 32 is already newer than 31 2023-01-15T14:28:54.888 INFO:tasks.rados.rados.0.smithi090.stdout:174: oid 42 updating version 32 to 34 2023-01-15T14:28:54.889 INFO:tasks.rados.rados.0.smithi090.stdout:174: oid 42 version 34 is already newer than 33 2023-01-15T14:28:54.889 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 42 v 34 (ObjNum 91 snap 15 seq_num 91) dirty exists 2023-01-15T14:28:54.889 INFO:tasks.rados.rados.0.smithi090.stdout:174: left oid 42 (ObjNum 91 snap 15 seq_num 91) 2023-01-15T14:28:54.889 INFO:tasks.rados.rados.0.smithi090.stdout:174: done (2 left) 2023-01-15T14:28:54.889 INFO:tasks.rados.rados.0.smithi090.stdout:176: done (1 left) 2023-01-15T14:28:54.889 INFO:tasks.rados.rados.0.smithi090.stdout:177: copy_from oid 4 from oid 49 current snap is 15 2023-01-15T14:28:54.889 INFO:tasks.rados.rados.0.smithi090.stdout:178: delete oid 10 current snap is 15 2023-01-15T14:28:54.890 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:54.741+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:28:54.890 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:54.885+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:54.934 INFO:tasks.rados.rados.0.smithi090.stdout:178: done (2 left) 2023-01-15T14:28:55.046 INFO:tasks.rados.rados.0.smithi090.stdout:179: rollback oid 43 current snap is 15 2023-01-15T14:28:55.046 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 43 to 8 2023-01-15T14:28:55.046 INFO:tasks.rados.rados.0.smithi090.stdout:180: delete oid 12 current snap is 15 2023-01-15T14:28:55.050 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:55.050 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:55.051 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:55.045+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:55.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:28:55.138 INFO:tasks.rados.rados.0.smithi090.stdout:179: finishing rollback tid 0 to smithi09025264-43 2023-01-15T14:28:55.150 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:55.145+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:28:55.218 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:55.218 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:55.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:28:55.237 INFO:tasks.rados.rados.0.smithi090.stdout:180: done (3 left) 2023-01-15T14:28:55.237 INFO:tasks.rados.rados.0.smithi090.stdout:181: snap_create 2023-01-15T14:28:55.238 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:28:55.238 INFO:tasks.rados.rados.0.smithi090.stdout:179: finishing rollback tid 1 to smithi09025264-43 2023-01-15T14:28:55.238 INFO:tasks.rados.rados.0.smithi090.stdout:179: finishing rollback tid 2 to smithi09025264-43 2023-01-15T14:28:55.238 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 33 (ObjNum 63 snap 6 seq_num 63) dirty exists 2023-01-15T14:28:55.238 INFO:tasks.rados.rados.0.smithi090.stdout:179: done (2 left) 2023-01-15T14:28:55.239 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:28:55.251 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:55.245+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:55.291 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:28:55.300 INFO:tasks.rados.rados.0.smithi090.stdout:175: finishing copy_from to smithi09025264-6 2023-01-15T14:28:55.300 INFO:tasks.rados.rados.0.smithi090.stdout:175: got expected ENOENT (src dne) 2023-01-15T14:28:55.301 INFO:tasks.rados.rados.0.smithi090.stdout:175: finishing copy_from racing read to smithi09025264-6 2023-01-15T14:28:55.301 INFO:tasks.rados.rados.0.smithi090.stdout:175: done (1 left) 2023-01-15T14:28:55.301 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:55.337 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:28:55.337 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:28:55.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:28:55.352 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:28:55.345+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:28:55.436 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:28:55.454 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:28:55.449+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:28:55.455 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:28:55.473 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:28:55.562 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:28:55.563 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:28:55.565 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:28:55.565 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:28:55.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:28:55.565 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:28:55.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.566 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:30 3.1 3:8b0a2647:::smithi09025264-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:55.566 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:28:55.566 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.642715377000002, 2023-01-15T14:28:55.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40319718700000001, 2023-01-15T14:28:55.567 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:55.567 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:55.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:55.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:55.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:55.570 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 30 2023-01-15T14:28:55.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.570 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:55.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:28:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:28:55.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859909+0000", 2023-01-15T14:28:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:28:55.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.860361+0000", 2023-01-15T14:28:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045164599999999999 2023-01-15T14:28:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.860392+0000", 2023-01-15T14:28:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1331000000000002e-05 2023-01-15T14:28:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960486+0000", 2023-01-15T14:28:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100093583 2023-01-15T14:28:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.160839+0000", 2023-01-15T14:28:55.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20035291199999999 2023-01-15T14:28:55.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:55.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.160982+0000", 2023-01-15T14:28:55.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014350800000000001 2023-01-15T14:28:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161183+0000", 2023-01-15T14:28:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000200411 2023-01-15T14:28:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161525+0000", 2023-01-15T14:28:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034216500000000001 2023-01-15T14:28:55.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161587+0000", 2023-01-15T14:28:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2032999999999994e-05 2023-01-15T14:28:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161842+0000", 2023-01-15T14:28:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000255305 2023-01-15T14:28:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262815+0000", 2023-01-15T14:28:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100972992 2023-01-15T14:28:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262914+0000", 2023-01-15T14:28:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9166999999999994e-05 2023-01-15T14:28:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262967+0000", 2023-01-15T14:28:55.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2244000000000003e-05 2023-01-15T14:28:55.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:55.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263050+0000", 2023-01-15T14:28:55.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2999999999999998e-05 2023-01-15T14:28:55.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:55.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263110+0000", 2023-01-15T14:28:55.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0297000000000002e-05 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:31 3.1 3:8b0a2647:::smithi09025264-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.542113011000001, 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405055944, 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:55.712 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:55.713 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:55.713 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:55.713 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 31 2023-01-15T14:28:55.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.713 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:55.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:55.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:28:55.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:28:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960513+0000", 2023-01-15T14:28:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:28:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960524+0000", 2023-01-15T14:28:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0614e-05 2023-01-15T14:28:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960538+0000", 2023-01-15T14:28:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4226999999999999e-05 2023-01-15T14:28:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:55.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060704+0000", 2023-01-15T14:28:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166252 2023-01-15T14:28:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.261723+0000", 2023-01-15T14:28:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20101896699999999 2023-01-15T14:28:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.261869+0000", 2023-01-15T14:28:55.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014578099999999999 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262092+0000", 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223198 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262397+0000", 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030490500000000001 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:55.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262446+0000", 2023-01-15T14:28:55.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8492999999999998e-05 2023-01-15T14:28:55.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:55.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262537+0000", 2023-01-15T14:28:55.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1248000000000005e-05 2023-01-15T14:28:55.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:55.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262847+0000", 2023-01-15T14:28:55.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031007799999999998 2023-01-15T14:28:55.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365234+0000", 2023-01-15T14:28:55.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102386856 2023-01-15T14:28:55.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365326+0000", 2023-01-15T14:28:55.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2226999999999996e-05 2023-01-15T14:28:55.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:55.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365413+0000", 2023-01-15T14:28:55.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6996999999999994e-05 2023-01-15T14:28:55.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:55.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365571+0000", 2023-01-15T14:28:55.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015794100000000001 2023-01-15T14:28:55.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.721 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:55.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.721 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:53 3.1 3:883e9a29:::smithi09025264-14:head [write 618491~694590,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:55.721 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:28:55.721 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.441804735000002, 2023-01-15T14:28:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659587899999998, 2023-01-15T14:28:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 53 2023-01-15T14:28:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:55.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:28:55.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:55.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:28:55.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:55.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060822+0000", 2023-01-15T14:28:55.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:55.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:55.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061225+0000", 2023-01-15T14:28:55.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040297199999999998 2023-01-15T14:28:55.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:55.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061244+0000", 2023-01-15T14:28:55.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:28:55.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:55.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061253+0000", 2023-01-15T14:28:55.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8200000000000003e-06 2023-01-15T14:28:55.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263154+0000", 2023-01-15T14:28:55.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20190054900000001 2023-01-15T14:28:55.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:55.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263511+0000", 2023-01-15T14:28:55.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035690599999999999 2023-01-15T14:28:55.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:55.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263928+0000", 2023-01-15T14:28:55.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041701200000000003 2023-01-15T14:28:55.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:55.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264611+0000", 2023-01-15T14:28:55.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068358700000000004 2023-01-15T14:28:55.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:55.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264661+0000", 2023-01-15T14:28:55.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.952e-05 2023-01-15T14:28:55.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:55.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264962+0000", 2023-01-15T14:28:55.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030116799999999999 2023-01-15T14:28:55.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:55.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365102+0000", 2023-01-15T14:28:55.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100139698 2023-01-15T14:28:55.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366354+0000", 2023-01-15T14:28:55.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001252059 2023-01-15T14:28:55.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457209+0000", 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090855667000000001 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457307+0000", 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7621000000000006e-05 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:55.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457419+0000", 2023-01-15T14:28:55.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112246 2023-01-15T14:28:55.731 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.731 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:55.731 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.731 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:55.731 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:28:55.731 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.441363282000001, 2023-01-15T14:28:55.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:28:55.731 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:55.732 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:55.732 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:55.732 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:55.732 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:55.732 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:28:55.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.732 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:55.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:55.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:55.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:55.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:28:55.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:55.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:28:55.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:28:55.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:55.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:28:55.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:28:55.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:55.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:28:55.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:28:55.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:28:55.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:28:55.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:28:55.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:55.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:28:55.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:28:55.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:55.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:28:55.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:28:55.740 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.740 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:55.740 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.741 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:55.741 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:28:55.741 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.441083033999998, 2023-01-15T14:28:55.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:28:55.741 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:55.829 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:55.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:55.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:55.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:55.829 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:28:55.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.830 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:55.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:55.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:28:55.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:55.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:28:55.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:55.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:28:55.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:55.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:55.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:28:55.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:28:55.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:55.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:28:55.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:28:55.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:55.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:28:55.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:28:55.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:28:55.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:28:55.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:55.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:28:55.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:28:55.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:55.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:28:55.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:28:55.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:55.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:28:55.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:28:55.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:55.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:28:55.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:28:55.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:28:55.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:28:55.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:28:55.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:28:55.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:55.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:28:55.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:28:55.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:55.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:28:55.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:28:55.838 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.838 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:55.838 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.838 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:55.838 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:28:55.839 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.441050417, 2023-01-15T14:28:55.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:28:55.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:55.839 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:55.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:55.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:55.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:55.839 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:28:55.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.839 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:55.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:55.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:28:55.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:55.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:28:55.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:55.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:28:55.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:28:55.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:55.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:28:55.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:28:55.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:28:55.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:55.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:28:55.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:28:55.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:55.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:28:55.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:28:55.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.845 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:55.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:58 3.1 3:86bba67f:::smithi09025264-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.339879851999999, 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39908735699999998, 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 58 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:55.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:55.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162747+0000", 2023-01-15T14:28:55.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:55.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:55.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162859+0000", 2023-01-15T14:28:55.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112137 2023-01-15T14:28:55.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:55.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162874+0000", 2023-01-15T14:28:55.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4793999999999999e-05 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162883+0000", 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5559999999999996e-06 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390699+0000", 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.227815712 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390900+0000", 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000201518 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458887+0000", 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067986183000000006 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:55.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459000+0000", 2023-01-15T14:28:55.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011291800000000001 2023-01-15T14:28:55.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:55.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459244+0000", 2023-01-15T14:28:55.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024485000000000002 2023-01-15T14:28:55.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:55.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459480+0000", 2023-01-15T14:28:55.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023518599999999999 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459498+0000", 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8892000000000001e-05 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459822+0000", 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032378700000000002 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:55.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460258+0000", 2023-01-15T14:28:55.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043563500000000003 2023-01-15T14:28:55.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465816+0000", 2023-01-15T14:28:55.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055581099999999998 2023-01-15T14:28:55.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561720+0000", 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095903587999999998 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561771+0000", 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1433e-05 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561836+0000", 2023-01-15T14:28:55.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4591000000000006e-05 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.339732431000002, 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:55.855 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:28:55.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:28:55.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:55.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:28:55.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:28:55.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:55.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:28:55.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:28:55.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:28:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:28:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:28:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:28:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:55.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:28:55.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:28:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:28:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:28:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:28:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:55.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.864 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.339702427999999, 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:55.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:55.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:28:55.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:28:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:28:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:28:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:28:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:28:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:28:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:28:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:28:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:28:55.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:55.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:28:55.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:28:55.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.871 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:55.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.871 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:125 3.1 3:8889e450:::smithi09025264-32:head [write 521017~798131,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:55.871 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:28:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.829813609999999, 2023-01-15T14:28:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40348810600000001, 2023-01-15T14:28:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 125 2023-01-15T14:28:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672812+0000", 2023-01-15T14:28:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:28:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673338+0000", 2023-01-15T14:28:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052553599999999995 2023-01-15T14:28:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673372+0000", 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3652999999999998e-05 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673386+0000", 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4171e-05 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773535+0000", 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148822 2023-01-15T14:28:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773727+0000", 2023-01-15T14:28:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192189 2023-01-15T14:28:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773918+0000", 2023-01-15T14:28:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019143099999999999 2023-01-15T14:28:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774291+0000", 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037288399999999998 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774341+0000", 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0117999999999997e-05 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774696+0000", 2023-01-15T14:28:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000355268 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874892+0000", 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100195981 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.977868+0000", 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102975649 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.075846+0000", 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097977838999999997 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.076004+0000", 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015859700000000001 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.076303+0000", 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029816400000000001 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.824693863, 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:28:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:28:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:28:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:28:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:28:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:28:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:28:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:28:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:56.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:28:56.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:28:56.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:56.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:28:56.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:28:56.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:56.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:28:56.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:28:56.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:56.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:28:56.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:28:56.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:56.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:28:56.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:28:56.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:56.064 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:56.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:56.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.064 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:56.065 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:28:56.065 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.824211509000001, 2023-01-15T14:28:56.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:28:56.065 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:56.065 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:56.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:56.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:56.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:56.066 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:28:56.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.067 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:56.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:56.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:28:56.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:56.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:56.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:28:56.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:56.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:56.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:28:56.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:28:56.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:56.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:28:56.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:28:56.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:56.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:28:56.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:28:56.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:56.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:28:56.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:28:56.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:56.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:28:56.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:28:56.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:56.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:28:56.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:28:56.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:56.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:28:56.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:28:56.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:56.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:28:56.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:28:56.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:56.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:28:56.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:56.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:28:56.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:28:56.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:56.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:56.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:28:56.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:28:56.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:56.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.824158946000001, 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:28:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:57.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:28:57.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:57.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:57.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:28:57.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:57.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:57.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:28:57.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:28:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:28:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:28:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:28:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:28:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:28:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:28:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:28:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:28:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:28:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:28:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:28:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:57.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:58.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:28:58.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:28:58.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.301 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:58.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.301 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:130 3.6 3:68fb3263:::smithi09025264-33:head [write 1806012~625998] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:28:58.301 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.784600+0000", 2023-01-15T14:28:58.301 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.718027647, 2023-01-15T14:28:58.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30627860400000001, 2023-01-15T14:28:58.301 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:58.316 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:58.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:58.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:58.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:58.316 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2023-01-15T14:28:58.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.316 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:58.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:58.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.784600+0000", 2023-01-15T14:28:58.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:58.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.784600+0000", 2023-01-15T14:28:58.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:58.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.784598+0000", 2023-01-15T14:28:58.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:28:58.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:58.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.784936+0000", 2023-01-15T14:28:58.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033822000000000002 2023-01-15T14:28:58.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.784969+0000", 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3083999999999997e-05 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.885103+0000", 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100133947 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.885212+0000", 2023-01-15T14:28:58.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108324 2023-01-15T14:28:58.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:58.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.885359+0000", 2023-01-15T14:28:58.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014727100000000001 2023-01-15T14:28:58.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:58.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.987685+0000", 2023-01-15T14:28:58.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102326639 2023-01-15T14:28:58.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:58.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.987826+0000", 2023-01-15T14:28:58.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014092999999999999 2023-01-15T14:28:58.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:28:58.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.988045+0000", 2023-01-15T14:28:58.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021909499999999999 2023-01-15T14:28:58.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:58.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.988546+0000", 2023-01-15T14:28:58.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050102199999999995 2023-01-15T14:28:58.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.988570+0000", 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3190999999999999e-05 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.988875+0000", 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030542399999999999 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.090269+0000", 2023-01-15T14:28:58.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101393448 2023-01-15T14:28:58.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:58.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.090411+0000", 2023-01-15T14:28:58.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014229600000000001 2023-01-15T14:28:58.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:58.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.090674+0000", 2023-01-15T14:28:58.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026322299999999998 2023-01-15T14:28:58.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:58.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.090785+0000", 2023-01-15T14:28:58.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011103300000000001 2023-01-15T14:28:58.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:58.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.090879+0000", 2023-01-15T14:28:58.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4005999999999994e-05 2023-01-15T14:28:58.519 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.519 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:58.519 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.523 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:28:58.523 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:28:58.523 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.452957242, 2023-01-15T14:28:58.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:28:58.524 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:58.524 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:58.524 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:58.524 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:58.525 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:58.525 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:28:58.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.526 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:58.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:58.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:28:58.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:58.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:28:58.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:58.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:28:58.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:28:58.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:58.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:28:58.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:28:58.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:58.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:28:58.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:28:58.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:58.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:28:58.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:28:58.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:58.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:28:58.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:28:58.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:28:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:28:58.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:28:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:28:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:28:58.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:28:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:28:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:28:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:28:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:28:58.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:58.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:28:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:28:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:28:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:28:58.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:28:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:28:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:28:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:28:58.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:28:58.720 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.452840567999999, 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:28:58.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:28:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:28:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:28:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:28:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:58.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:28:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:28:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:28:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:28:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:58.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:28:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:28:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:28:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:28:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:28:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:28:58.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:28:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:28:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:28:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:28:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:28:58.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:28:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:28:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:28:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:28:58.731 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:448 3.1 3:8b0a2647:::smithi09025264-8:head [write 1856548~737690] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:28:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:28:58.732 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.2055826599999993, 2023-01-15T14:28:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40560468799999999, 2023-01-15T14:28:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 448 2023-01-15T14:28:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:28:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:28:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:58.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297043+0000", 2023-01-15T14:28:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:28:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297477+0000", 2023-01-15T14:28:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043336500000000001 2023-01-15T14:28:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297505+0000", 2023-01-15T14:28:58.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8583000000000001e-05 2023-01-15T14:28:58.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397598+0000", 2023-01-15T14:28:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092164 2023-01-15T14:28:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397683+0000", 2023-01-15T14:28:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5986e-05 2023-01-15T14:28:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397798+0000", 2023-01-15T14:28:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114916 2023-01-15T14:28:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498684+0000", 2023-01-15T14:28:58.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10088510000000001 2023-01-15T14:28:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498848+0000", 2023-01-15T14:28:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164575 2023-01-15T14:28:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499083+0000", 2023-01-15T14:28:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000235171 2023-01-15T14:28:58.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499670+0000", 2023-01-15T14:28:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058650200000000003 2023-01-15T14:28:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499688+0000", 2023-01-15T14:28:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8547e-05 2023-01-15T14:28:58.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.500037+0000", 2023-01-15T14:28:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034891399999999997 2023-01-15T14:28:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.501925+0000", 2023-01-15T14:28:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00188766 2023-01-15T14:28:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.601919+0000", 2023-01-15T14:28:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099994371999999998 2023-01-15T14:28:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702377+0000", 2023-01-15T14:28:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045810500000001 2023-01-15T14:28:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702537+0000", 2023-01-15T14:28:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015941600000000001 2023-01-15T14:28:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702650+0000", 2023-01-15T14:28:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011338899999999999 2023-01-15T14:28:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.910 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:451 3.1 3:8b0a2647:::smithi09025264-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:28:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.397793+0000", 2023-01-15T14:28:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.1048347199999995, 2023-01-15T14:28:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30620719800000001, 2023-01-15T14:28:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:58.911 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 451 2023-01-15T14:28:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:58.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397793+0000", 2023-01-15T14:28:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397793+0000", 2023-01-15T14:28:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:58.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397793+0000", 2023-01-15T14:28:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397797+0000", 2023-01-15T14:28:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.143e-06 2023-01-15T14:28:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397808+0000", 2023-01-15T14:28:58.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0893999999999999e-05 2023-01-15T14:28:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397818+0000", 2023-01-15T14:28:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0506000000000001e-05 2023-01-15T14:28:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.493632+0000", 2023-01-15T14:28:58.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095813551999999996 2023-01-15T14:28:58.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:58.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.493911+0000", 2023-01-15T14:28:58.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027954800000000001 2023-01-15T14:28:58.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:58.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.502017+0000", 2023-01-15T14:28:58.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081052439999999993 2023-01-15T14:28:58.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:58.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.502157+0000", 2023-01-15T14:28:58.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000140696 2023-01-15T14:28:58.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:58.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.703469+0000", 2023-01-15T14:28:58.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201311512 2023-01-15T14:28:58.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:58.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.703724+0000", 2023-01-15T14:28:58.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025488099999999999 2023-01-15T14:28:58.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:58.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.704001+0000", 2023-01-15T14:28:58.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027700199999999999 2023-01-15T14:28:58.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.919 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:58.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:58.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.920 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:468 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (57) in=7b,truncate 3378174] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:28:58.920 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775893+0000", 2023-01-15T14:28:58.920 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.7267355740000001, 2023-01-15T14:28:58.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39428173999999999, 2023-01-15T14:28:58.920 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:58.920 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:28:58.920 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:58.920 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:58.920 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:58.920 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2023-01-15T14:28:58.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.921 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:58.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:58.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775893+0000", 2023-01-15T14:28:58.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:58.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775893+0000", 2023-01-15T14:28:58.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:58.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:58.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:58.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775892+0000", 2023-01-15T14:28:58.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:28:58.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:58.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:59.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775897+0000", 2023-01-15T14:28:59.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4820000000000002e-06 2023-01-15T14:28:59.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:59.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775906+0000", 2023-01-15T14:28:59.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9199999999999993e-06 2023-01-15T14:28:59.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:59.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775911+0000", 2023-01-15T14:28:59.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5179999999999998e-06 2023-01-15T14:28:59.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:59.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.967706+0000", 2023-01-15T14:28:59.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.191795189 2023-01-15T14:28:59.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:59.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.967988+0000", 2023-01-15T14:28:59.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028248100000000001 2023-01-15T14:28:59.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:28:59.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968459+0000", 2023-01-15T14:28:59.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000470917 2023-01-15T14:28:59.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:28:59.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968829+0000", 2023-01-15T14:28:59.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036951399999999999 2023-01-15T14:28:59.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:28:59.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968842+0000", 2023-01-15T14:28:59.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2801e-05 2023-01-15T14:28:59.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:28:59.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968890+0000", 2023-01-15T14:28:59.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8775e-05 2023-01-15T14:28:59.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:28:59.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969229+0000", 2023-01-15T14:28:59.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033879800000000002 2023-01-15T14:28:59.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:59.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.169674+0000", 2023-01-15T14:28:59.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20044432000000001 2023-01-15T14:28:59.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:28:59.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.169983+0000", 2023-01-15T14:28:59.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030946800000000003 2023-01-15T14:28:59.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:28:59.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.170076+0000", 2023-01-15T14:28:59.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2872000000000001e-05 2023-01-15T14:28:59.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:59.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.170174+0000", 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.7267082919999996, 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:28:59.139 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:28:59.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:28:59.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:28:59.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:28:59.140 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:28:59.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.140 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:28:59.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:28:59.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:28:59.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:59.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:28:59.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:28:59.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:28:59.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:28:59.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:28:59.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:28:59.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:28:59.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:28:59.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:28:59.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:28:59.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:28:59.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:28:59.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:59.145 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:59.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:59.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:28:59.145 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:28:59.145 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:28:59.146 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:28:59.146 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:28:59.147 INFO:tasks.rados.rados.0.smithi090.stdout:177: finishing copy_from to smithi09025264-4 2023-01-15T14:28:59.147 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 4 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2023-01-15T14:28:59.147 INFO:tasks.rados.rados.0.smithi090.stdout:177: finishing copy_from racing read to smithi09025264-4 2023-01-15T14:28:59.527 INFO:tasks.rados.rados.0.smithi090.stdout:177: done (0 left) 2023-01-15T14:28:59.528 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:28:59.528 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:28:59.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:28:59.530 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:28:59.530 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:28:59.531 INFO:tasks.thrashosds.thrasher:Reweighting osd 6 to 0.3273561279476326 2023-01-15T14:28:59.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 0.3273561279476326 2023-01-15T14:28:59.532 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:28:55.557+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:28:59.533 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:28:59.535 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:28:59.535 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:28:59.535 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:28:59.536 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:28:59.536 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:28:59.536 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:28:59.536 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:28:59.540 INFO:tasks.rados.rados.0.smithi090.stdout:181: done (0 left) 2023-01-15T14:28:59.540 INFO:tasks.rados.rados.0.smithi090.stdout:182: write (excl) oid 12 current snap is 16 2023-01-15T14:28:59.540 INFO:tasks.rados.rados.0.smithi090.stdout:182: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2023-01-15T14:28:59.540 INFO:tasks.rados.rados.0.smithi090.stdout:182: writing smithi09025264-12 from 779858 to 1249500 tid 1 2023-01-15T14:28:59.540 INFO:tasks.rados.rados.0.smithi090.stdout:182: writing smithi09025264-12 from 1817425 to 2324456 tid 2 2023-01-15T14:28:59.541 INFO:tasks.rados.rados.0.smithi090.stdout:182: writing smithi09025264-12 from 2781526 to 3368154 tid 3 2023-01-15T14:28:59.541 INFO:tasks.rados.rados.0.smithi090.stdout:183: snap_create 2023-01-15T14:28:59.541 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:59.541 INFO:tasks.rados.rados.0.smithi090.stdout:182: finishing write tid 1 to smithi09025264-12 2023-01-15T14:28:59.541 INFO:tasks.rados.rados.0.smithi090.stdout:182: finishing write tid 2 to smithi09025264-12 2023-01-15T14:28:59.541 INFO:tasks.rados.rados.0.smithi090.stdout:182: finishing write tid 3 to smithi09025264-12 2023-01-15T14:28:59.541 INFO:tasks.rados.rados.0.smithi090.stdout:182: finishing write tid 4 to smithi09025264-12 2023-01-15T14:28:59.541 INFO:tasks.rados.rados.0.smithi090.stdout:182: finishing write tid 5 to smithi09025264-12 2023-01-15T14:28:59.541 INFO:tasks.rados.rados.0.smithi090.stdout:182: oid 12 updating version 0 to 67 2023-01-15T14:28:59.541 INFO:tasks.rados.rados.0.smithi090.stdout:182: oid 12 version 67 is already newer than 66 2023-01-15T14:28:59.541 INFO:tasks.rados.rados.0.smithi090.stdout:182: oid 12 updating version 67 to 68 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:182: oid 12 updating version 68 to 69 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 69 (ObjNum 94 snap 16 seq_num 94) dirty exists 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:182: left oid 12 (ObjNum 94 snap 16 seq_num 94) 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:182: done (0 left) 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:183: done (0 left) 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:184: delete oid 7 current snap is 17 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:184: done (0 left) 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:185: snap_create 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:185: done (0 left) 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:186: read oid 32 snap 14 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:186: expect (ObjNum 34 snap 0 seq_num 34) 2023-01-15T14:28:59.542 INFO:tasks.rados.rados.0.smithi090.stdout:187: snap_create 2023-01-15T14:28:59.543 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:59.543 INFO:tasks.rados.rados.0.smithi090.stdout:186: expect (ObjNum 34 snap 0 seq_num 34) 2023-01-15T14:28:59.543 INFO:tasks.rados.rados.0.smithi090.stdout:186: done (0 left) 2023-01-15T14:28:59.543 INFO:tasks.rados.rados.0.smithi090.stdout:187: done (0 left) 2023-01-15T14:28:59.543 INFO:tasks.rados.rados.0.smithi090.stdout:188: copy_from oid 19 from oid 22 current snap is 19 2023-01-15T14:28:59.544 INFO:tasks.rados.rados.0.smithi090.stdout:189: read oid 21 snap 15 2023-01-15T14:28:59.544 INFO:tasks.rados.rados.0.smithi090.stdout:189: expect (ObjNum 83 snap 11 seq_num 83) 2023-01-15T14:28:59.544 INFO:tasks.rados.rados.0.smithi090.stdout:190: write oid 49 current snap is 19 2023-01-15T14:28:59.544 INFO:tasks.rados.rados.0.smithi090.stdout:190: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2023-01-15T14:28:59.544 INFO:tasks.rados.rados.0.smithi090.stdout:190: writing smithi09025264-49 from 413472 to 1033612 tid 1 2023-01-15T14:28:59.544 INFO:tasks.rados.rados.0.smithi090.stdout:190: writing smithi09025264-49 from 1818036 to 2328736 tid 2 2023-01-15T14:28:59.545 INFO:tasks.rados.rados.0.smithi090.stdout:190: writing smithi09025264-49 from 3120019 to 3613472 tid 3 2023-01-15T14:28:59.545 INFO:tasks.rados.rados.0.smithi090.stdout:191: read oid 24 snap -1 2023-01-15T14:28:59.545 INFO:tasks.rados.rados.0.smithi090.stdout:191: expect (ObjNum 34 snap 0 seq_num 34) 2023-01-15T14:28:59.545 INFO:tasks.rados.rados.0.smithi090.stdout:192: delete oid 43 current snap is 19 2023-01-15T14:28:59.545 INFO:tasks.rados.rados.0.smithi090.stdout:188: finishing copy_from to smithi09025264-19 2023-01-15T14:28:59.545 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 40 (ObjNum 87 snap 12 seq_num 87) dirty exists 2023-01-15T14:28:59.545 INFO:tasks.rados.rados.0.smithi090.stdout:188: finishing copy_from racing read to smithi09025264-19 2023-01-15T14:28:59.545 INFO:tasks.rados.rados.0.smithi090.stdout:190: finishing write tid 1 to smithi09025264-49 2023-01-15T14:28:59.545 INFO:tasks.rados.rados.0.smithi090.stdout:190: finishing write tid 2 to smithi09025264-49 2023-01-15T14:28:59.545 INFO:tasks.rados.rados.0.smithi090.stdout:190: finishing write tid 3 to smithi09025264-49 2023-01-15T14:28:59.546 INFO:tasks.rados.rados.0.smithi090.stdout:191: expect (ObjNum 34 snap 0 seq_num 34) 2023-01-15T14:28:59.546 INFO:tasks.rados.rados.0.smithi090.stdout:188: done (4 left) 2023-01-15T14:28:59.546 INFO:tasks.rados.rados.0.smithi090.stdout:191: done (3 left) 2023-01-15T14:28:59.546 INFO:tasks.rados.rados.0.smithi090.stdout:192: done (2 left) 2023-01-15T14:28:59.546 INFO:tasks.rados.rados.0.smithi090.stdout:193: copy_from oid 9 from oid 5 current snap is 19 2023-01-15T14:28:59.546 INFO:tasks.rados.rados.0.smithi090.stdout:194: snap_create 2023-01-15T14:28:59.546 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:28:59.546 INFO:tasks.rados.rados.0.smithi090.stdout:190: finishing write tid 4 to smithi09025264-49 2023-01-15T14:28:59.546 INFO:tasks.rados.rados.0.smithi090.stdout:190: finishing write tid 5 to smithi09025264-49 2023-01-15T14:28:59.546 INFO:tasks.rados.rados.0.smithi090.stdout:190: oid 49 updating version 0 to 41 2023-01-15T14:28:59.546 INFO:tasks.rados.rados.0.smithi090.stdout:190: oid 49 version 41 is already newer than 40 2023-01-15T14:28:59.547 INFO:tasks.rados.rados.0.smithi090.stdout:190: oid 49 updating version 41 to 43 2023-01-15T14:28:59.547 INFO:tasks.rados.rados.0.smithi090.stdout:190: oid 49 version 43 is already newer than 42 2023-01-15T14:28:59.547 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 43 (ObjNum 96 snap 19 seq_num 96) dirty exists 2023-01-15T14:28:59.547 INFO:tasks.rados.rados.0.smithi090.stdout:190: left oid 49 (ObjNum 96 snap 19 seq_num 96) 2023-01-15T14:28:59.547 INFO:tasks.rados.rados.0.smithi090.stdout:190: done (2 left) 2023-01-15T14:28:59.547 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:28:59.547 INFO:tasks.rados.rados.0.smithi090.stdout:189: expect (ObjNum 83 snap 11 seq_num 83) 2023-01-15T14:28:59.547 INFO:tasks.rados.rados.0.smithi090.stdout:189: done (1 left) 2023-01-15T14:28:59.547 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:28:59.548 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:28:59.525+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:28:59.630 INFO:tasks.rados.rados.0.smithi090.stdout:193: finishing copy_from to smithi09025264-9 2023-01-15T14:28:59.737 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 27 (ObjNum 90 snap 14 seq_num 90) dirty exists 2023-01-15T14:28:59.738 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:28:59.625+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:28:59.739 INFO:tasks.rados.rados.0.smithi090.stdout:193: finishing copy_from racing read to smithi09025264-9 2023-01-15T14:28:59.739 INFO:tasks.rados.rados.0.smithi090.stdout:193: done (0 left) 2023-01-15T14:28:59.740 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:28:59.740 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:28:59.740 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:28:59.740 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:28:59.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:28:59.741 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:28:59.733+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:28:59.795 INFO:tasks.rados.rados.0.smithi090.stdout:194: done (0 left) 2023-01-15T14:28:59.938 INFO:tasks.rados.rados.0.smithi090.stdout:195: copy_from oid 41 from oid 40 current snap is 20 2023-01-15T14:28:59.938 INFO:tasks.rados.rados.0.smithi090.stdout:196: copy_from oid 6 from oid 47 current snap is 20 2023-01-15T14:28:59.938 INFO:tasks.rados.rados.0.smithi090.stdout:197: read oid 3 snap -1 2023-01-15T14:28:59.938 INFO:tasks.rados.rados.0.smithi090.stdout:197: expect (ObjNum 73 snap 8 seq_num 73) 2023-01-15T14:28:59.938 INFO:tasks.rados.rados.0.smithi090.stdout:198: rollback oid 48 current snap is 20 2023-01-15T14:28:59.938 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 48 to 18 2023-01-15T14:28:59.938 INFO:tasks.rados.rados.0.smithi090.stdout:199: write (excl) oid 45 current snap is 20 2023-01-15T14:28:59.939 INFO:tasks.rados.rados.0.smithi090.stdout:199: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2023-01-15T14:28:59.940 INFO:tasks.rados.rados.0.smithi090.stdout:199: writing smithi09025264-45 from 447086 to 1217724 tid 1 2023-01-15T14:28:59.940 INFO:tasks.rados.rados.0.smithi090.stdout:199: writing smithi09025264-45 from 1935000 to 2733016 tid 2 2023-01-15T14:28:59.941 INFO:tasks.rados.rados.0.smithi090.stdout:199: writing smithi09025264-45 from 3458512 to 3647086 tid 3 2023-01-15T14:28:59.941 INFO:tasks.rados.rados.0.smithi090.stdout:200: read oid 50 snap -1 2023-01-15T14:28:59.941 INFO:tasks.rados.rados.0.smithi090.stdout:200: expect deleted 2023-01-15T14:28:59.941 INFO:tasks.rados.rados.0.smithi090.stdout:201: write (excl) oid 44 current snap is 20 2023-01-15T14:28:59.941 INFO:tasks.rados.rados.0.smithi090.stdout:201: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2023-01-15T14:28:59.941 INFO:tasks.rados.rados.0.smithi090.stdout:201: writing smithi09025264-44 from 463893 to 1026133 tid 1 2023-01-15T14:28:59.941 INFO:tasks.rados.rados.0.smithi090.stdout:201: writing smithi09025264-44 from 1593482 to 2251509 tid 2 2023-01-15T14:28:59.941 INFO:tasks.rados.rados.0.smithi090.stdout:201: writing smithi09025264-44 from 2685935 to 3245276 tid 3 2023-01-15T14:28:59.941 INFO:tasks.rados.rados.0.smithi090.stdout:202: copy_from oid 9 from oid 30 current snap is 20 2023-01-15T14:28:59.941 INFO:tasks.rados.rados.0.smithi090.stdout:197: expect (ObjNum 73 snap 8 seq_num 73) 2023-01-15T14:28:59.942 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:28:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:28:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:28:59.942 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:28:59.943 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:28:59.943 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:28:59.943 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:28:59.933+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:28:59.959 INFO:tasks.rados.rados.0.smithi090.stdout:197: done (7 left) 2023-01-15T14:28:59.959 INFO:tasks.rados.rados.0.smithi090.stdout:203: write (excl) oid 11 current snap is 20 2023-01-15T14:28:59.959 INFO:tasks.rados.rados.0.smithi090.stdout:203: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2023-01-15T14:28:59.973 INFO:tasks.rados.rados.0.smithi090.stdout:203: writing smithi09025264-11 from 480700 to 1118189 tid 1 2023-01-15T14:29:00.146 INFO:tasks.rados.rados.0.smithi090.stdout:203: writing smithi09025264-11 from 1651964 to 2053649 tid 2 2023-01-15T14:29:00.146 INFO:tasks.rados.rados.0.smithi090.stdout:203: writing smithi09025264-11 from 2713358 to 3483971 tid 3 2023-01-15T14:29:00.147 INFO:tasks.rados.rados.0.smithi090.stdout:204: snap_create 2023-01-15T14:29:00.147 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:29:00.147 INFO:tasks.rados.rados.0.smithi090.stdout:199: finishing write tid 1 to smithi09025264-45 2023-01-15T14:29:00.147 INFO:tasks.rados.rados.0.smithi090.stdout:196: finishing copy_from to smithi09025264-6 2023-01-15T14:29:00.147 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 50 (ObjNum 70 snap 8 seq_num 70) dirty exists 2023-01-15T14:29:00.147 INFO:tasks.rados.rados.0.smithi090.stdout:201: finishing write tid 1 to smithi09025264-44 2023-01-15T14:29:00.149 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:00.149 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:29:00.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:29:00.149 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:68 3.0 3:0b54190a:::smithi09025264-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.223534+0000", 2023-01-15T14:29:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.822333684999997, 2023-01-15T14:29:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49993064500000001, 2023-01-15T14:29:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 68 2023-01-15T14:29:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223534+0000", 2023-01-15T14:29:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223534+0000", 2023-01-15T14:29:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223533+0000", 2023-01-15T14:29:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223539+0000", 2023-01-15T14:29:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9970000000000004e-06 2023-01-15T14:29:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223556+0000", 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6543000000000001e-05 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223567+0000", 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0893999999999999e-05 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.520962+0000", 2023-01-15T14:29:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29739558300000002 2023-01-15T14:29:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.521142+0000", 2023-01-15T14:29:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017972100000000001 2023-01-15T14:29:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.723135+0000", 2023-01-15T14:29:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20199320400000001 2023-01-15T14:29:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.723274+0000", 2023-01-15T14:29:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013887700000000001 2023-01-15T14:29:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.723465+0000", 2023-01-15T14:29:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019079399999999999 2023-01-15T14:29:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:94 3.0 3:07a51d33:::smithi09025264-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.606328443000002, 2023-01-15T14:29:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50760159199999999, 2023-01-15T14:29:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 94 2023-01-15T14:29:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439538+0000", 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439857+0000", 2023-01-15T14:29:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000319086 2023-01-15T14:29:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439891+0000", 2023-01-15T14:29:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4449999999999997e-05 2023-01-15T14:29:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439906+0000", 2023-01-15T14:29:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5082999999999999e-05 2023-01-15T14:29:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824189+0000", 2023-01-15T14:29:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38428238100000001 2023-01-15T14:29:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824440+0000", 2023-01-15T14:29:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025139800000000001 2023-01-15T14:29:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824834+0000", 2023-01-15T14:29:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039422200000000001 2023-01-15T14:29:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825347+0000", 2023-01-15T14:29:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051323600000000003 2023-01-15T14:29:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825364+0000", 2023-01-15T14:29:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6478e-05 2023-01-15T14:29:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825589+0000", 2023-01-15T14:29:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022497500000000001 2023-01-15T14:29:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:00.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826457+0000", 2023-01-15T14:29:00.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00086797600000000001 2023-01-15T14:29:00.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:00.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.846539+0000", 2023-01-15T14:29:00.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020081742 2023-01-15T14:29:00.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:00.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.946989+0000", 2023-01-15T14:29:00.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045032199999999 2023-01-15T14:29:00.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:00.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947076+0000", 2023-01-15T14:29:00.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6922999999999994e-05 2023-01-15T14:29:00.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:00.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947141+0000", 2023-01-15T14:29:00.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5357000000000005e-05 2023-01-15T14:29:00.352 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.352 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:00.352 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.353 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:95 3.0 3:07a51d33:::smithi09025264-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:00.353 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439927+0000", 2023-01-15T14:29:00.353 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.605940838000002, 2023-01-15T14:29:00.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50742387499999997, 2023-01-15T14:29:00.353 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:00.353 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:00.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:00.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:00.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:00.353 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 95 2023-01-15T14:29:00.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.354 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:00.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:00.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439927+0000", 2023-01-15T14:29:00.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:00.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439927+0000", 2023-01-15T14:29:00.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:00.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439926+0000", 2023-01-15T14:29:00.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:00.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:00.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439933+0000", 2023-01-15T14:29:00.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2999999999999998e-06 2023-01-15T14:29:00.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:00.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439947+0000", 2023-01-15T14:29:00.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4071e-05 2023-01-15T14:29:00.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:00.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439953+0000", 2023-01-15T14:29:00.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4160000000000001e-06 2023-01-15T14:29:00.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825479+0000", 2023-01-15T14:29:00.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.385525544 2023-01-15T14:29:00.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:00.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825576+0000", 2023-01-15T14:29:00.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7342000000000004e-05 2023-01-15T14:29:00.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:00.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825814+0000", 2023-01-15T14:29:00.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023816700000000001 2023-01-15T14:29:00.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:00.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826048+0000", 2023-01-15T14:29:00.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000234019 2023-01-15T14:29:00.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:00.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826067+0000", 2023-01-15T14:29:00.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9252000000000001e-05 2023-01-15T14:29:00.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:00.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826137+0000", 2023-01-15T14:29:00.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0004000000000003e-05 2023-01-15T14:29:00.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:00.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.828120+0000", 2023-01-15T14:29:00.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019824619999999999 2023-01-15T14:29:00.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:00.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.846644+0000", 2023-01-15T14:29:00.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018524134000000001 2023-01-15T14:29:00.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:00.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947206+0000", 2023-01-15T14:29:00.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100562254 2023-01-15T14:29:00.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:00.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947279+0000", 2023-01-15T14:29:00.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2992999999999997e-05 2023-01-15T14:29:00.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:00.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947351+0000", 2023-01-15T14:29:00.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1847000000000005e-05 2023-01-15T14:29:00.361 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.361 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:00.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:96 3.0 3:07a51d33:::smithi09025264-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:00.362 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:00.362 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.605906330000003, 2023-01-15T14:29:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50769745899999996, 2023-01-15T14:29:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:00.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 96 2023-01-15T14:29:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439961+0000", 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:00.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439966+0000", 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0769999999999998e-06 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439979+0000", 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3343000000000001e-05 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439987+0000", 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2400000000000001e-06 2023-01-15T14:29:00.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826191+0000", 2023-01-15T14:29:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38620452799999999 2023-01-15T14:29:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826424+0000", 2023-01-15T14:29:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023264800000000001 2023-01-15T14:29:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947394+0000", 2023-01-15T14:29:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120970102 2023-01-15T14:29:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947505+0000", 2023-01-15T14:29:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001105 2023-01-15T14:29:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947659+0000", 2023-01-15T14:29:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015468799999999999 2023-01-15T14:29:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:122 3.0 3:094fbf13:::smithi09025264-31:head [write 1572695~622329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.664895+0000", 2023-01-15T14:29:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.380973062000002, 2023-01-15T14:29:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.48643288000000001, 2023-01-15T14:29:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 122 2023-01-15T14:29:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.664895+0000", 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.664895+0000", 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.664892+0000", 2023-01-15T14:29:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.665508+0000", 2023-01-15T14:29:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061576299999999997 2023-01-15T14:29:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.665545+0000", 2023-01-15T14:29:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6677000000000001e-05 2023-01-15T14:29:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.665565+0000", 2023-01-15T14:29:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0129e-05 2023-01-15T14:29:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.827872+0000", 2023-01-15T14:29:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16230762300000001 2023-01-15T14:29:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.828086+0000", 2023-01-15T14:29:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395799999999999 2023-01-15T14:29:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.047375+0000", 2023-01-15T14:29:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21928845399999999 2023-01-15T14:29:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.047525+0000", 2023-01-15T14:29:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015062100000000001 2023-01-15T14:29:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.047738+0000", 2023-01-15T14:29:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021218100000000001 2023-01-15T14:29:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.048121+0000", 2023-01-15T14:29:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038368099999999997 2023-01-15T14:29:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.048144+0000", 2023-01-15T14:29:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2344000000000002e-05 2023-01-15T14:29:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.048477+0000", 2023-01-15T14:29:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033343799999999997 2023-01-15T14:29:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.049152+0000", 2023-01-15T14:29:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067540600000000003 2023-01-15T14:29:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.050324+0000", 2023-01-15T14:29:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001171353 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.151195+0000", 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100871158 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.151266+0000", 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1230999999999996e-05 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.151328+0000", 2023-01-15T14:29:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1662999999999996e-05 2023-01-15T14:29:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.961353666000001, 2023-01-15T14:29:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:29:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:29:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:29:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:29:00.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:00.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:29:00.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:29:00.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:00.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:29:00.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:29:00.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:29:00.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:29:00.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:00.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:29:00.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:29:00.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:29:00.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:29:00.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:00.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:29:00.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:29:00.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:00.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:29:00.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:29:00.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:00.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:29:00.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:29:00.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:00.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:29:00.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:29:00.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:00.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:29:00.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:29:00.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:00.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:29:00.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:29:00.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:00.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:29:00.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:29:00.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:00.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:29:00.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:29:00.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.880 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:00.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.880 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:00.880 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:00.880 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.961053868999997, 2023-01-15T14:29:00.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:29:00.880 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:00.881 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:00.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:00.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:00.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:00.881 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:29:00.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.881 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:00.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:00.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:00.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:00.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:00.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:00.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:29:00.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:00.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:00.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:29:00.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:29:00.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:00.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:29:00.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:29:00.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:00.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:29:00.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:29:00.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:29:00.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:29:00.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:00.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:29:00.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:29:00.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:29:00.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:29:00.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:00.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:29:00.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:29:00.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:00.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:29:00.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:29:00.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:00.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:29:00.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:29:00.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:00.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:29:00.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:29:00.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:00.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:29:00.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:29:00.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:00.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:29:00.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:29:00.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:00.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:29:00.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:29:00.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:00.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:29:00.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:29:00.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:00.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:29:00.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:29:00.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:00.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:29:00.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:29:00.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.895 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:00.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:00.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.895 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:00.895 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:00.895 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.961024084999998, 2023-01-15T14:29:00.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:29:00.895 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:00.895 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:00.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:00.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:00.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:00.896 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:29:00.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.896 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:00.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:00.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:00.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:00.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:00.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:00.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:00.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:29:00.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:00.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:00.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:29:00.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:29:00.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:00.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:29:00.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:29:00.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:00.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:29:00.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:29:00.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:29:00.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:29:00.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:00.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:29:00.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:29:00.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:29:00.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:29:00.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:00.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:29:00.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:29:00.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:00.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:00.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:00.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:29:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:29:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:29:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:29:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:29:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:29:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:01.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:181 3.4 3:2022fdfa:::smithi09025264-46:head [write 756315~767970,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.206931+0000", 2023-01-15T14:29:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.838937096000002, 2023-01-15T14:29:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.48671660999999999, 2023-01-15T14:29:01.038 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 181 2023-01-15T14:29:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.042 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.206931+0000", 2023-01-15T14:29:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.206931+0000", 2023-01-15T14:29:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.206929+0000", 2023-01-15T14:29:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.207300+0000", 2023-01-15T14:29:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000371398 2023-01-15T14:29:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.207336+0000", 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6494e-05 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.207357+0000", 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0213000000000001e-05 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389075+0000", 2023-01-15T14:29:01.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.181718506 2023-01-15T14:29:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389352+0000", 2023-01-15T14:29:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027699199999999999 2023-01-15T14:29:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389589+0000", 2023-01-15T14:29:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023647699999999999 2023-01-15T14:29:01.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.390051+0000", 2023-01-15T14:29:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046192200000000003 2023-01-15T14:29:01.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.390079+0000", 2023-01-15T14:29:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8824e-05 2023-01-15T14:29:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.390455+0000", 2023-01-15T14:29:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000375461 2023-01-15T14:29:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491484+0000", 2023-01-15T14:29:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10102902599999999 2023-01-15T14:29:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:01.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593226+0000", 2023-01-15T14:29:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101742349 2023-01-15T14:29:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693306+0000", 2023-01-15T14:29:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10007964 2023-01-15T14:29:01.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693403+0000", 2023-01-15T14:29:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7533000000000003e-05 2023-01-15T14:29:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693648+0000", 2023-01-15T14:29:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024423799999999997 2023-01-15T14:29:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:01.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:182 3.4 3:2022fdfa:::smithi09025264-46:head [write 1933572~791670] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.832162015999998, 2023-01-15T14:29:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.582304865, 2023-01-15T14:29:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:01.064 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 182 2023-01-15T14:29:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:01.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213704+0000", 2023-01-15T14:29:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214574+0000", 2023-01-15T14:29:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087037799999999999 2023-01-15T14:29:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214608+0000", 2023-01-15T14:29:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:29:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214627+0000", 2023-01-15T14:29:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:29:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491537+0000", 2023-01-15T14:29:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27691020300000002 2023-01-15T14:29:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491721+0000", 2023-01-15T14:29:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345500000000001 2023-01-15T14:29:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693760+0000", 2023-01-15T14:29:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202039252 2023-01-15T14:29:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693898+0000", 2023-01-15T14:29:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013789 2023-01-15T14:29:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694230+0000", 2023-01-15T14:29:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220899999999998 2023-01-15T14:29:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694866+0000", 2023-01-15T14:29:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063640599999999995 2023-01-15T14:29:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694887+0000", 2023-01-15T14:29:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0622999999999999e-05 2023-01-15T14:29:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695950+0000", 2023-01-15T14:29:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001062688 2023-01-15T14:29:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696014+0000", 2023-01-15T14:29:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4697999999999996e-05 2023-01-15T14:29:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795898+0000", 2023-01-15T14:29:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099883338000000002 2023-01-15T14:29:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795922+0000", 2023-01-15T14:29:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4326000000000001e-05 2023-01-15T14:29:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795955+0000", 2023-01-15T14:29:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3191000000000001e-05 2023-01-15T14:29:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796011+0000", 2023-01-15T14:29:01.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5631999999999999e-05 2023-01-15T14:29:01.567 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:01.568 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:01.568 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:01.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.568 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:01.569 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:01.569 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.831215284999999, 2023-01-15T14:29:01.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:29:01.569 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:01.569 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:01.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:01.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:01.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:29:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:01.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:01.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:01.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:29:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:01.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:29:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:29:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:29:01.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:29:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:29:01.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:29:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:29:01.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:29:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:29:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:29:01.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:29:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:29:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:01.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:29:01.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:29:01.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:01.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:29:01.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:29:01.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:01.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:29:01.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:29:01.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:01.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:29:01.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:29:01.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:01.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:29:01.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:29:01.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:01.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:29:01.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:29:01.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:01.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:29:01.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:29:01.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:01.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:29:01.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:29:01.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:01.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:29:01.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:29:01.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:01.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:29:01.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:29:01.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:01.804 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:01.804 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:01.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.804 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:01.804 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:01.805 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 32.831158588000001, 2023-01-15T14:29:01.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:29:01.959 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:01.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:01.960 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:01.960 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:01.960 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:01.960 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:29:01.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.960 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:01.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:01.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:01.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:01.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:01.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:01.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:01.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:01.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:29:01.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:01.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:01.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:29:01.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:29:01.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:01.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:29:01.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:29:01.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:01.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:29:01.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:29:01.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:01.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:29:01.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:29:01.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:29:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:29:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:29:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:29:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:29:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:29:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:29:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:29:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:29:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:29:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:01.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:29:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:29:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:223 3.0 3:03b54904:::smithi09025264-12:head [setxattr _header (56) in=7b,truncate 2924385] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:29:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831168+0000", 2023-01-15T14:29:02.233 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.214699970000002, 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.495961084, 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 223 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:02.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831168+0000", 2023-01-15T14:29:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831168+0000", 2023-01-15T14:29:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:02.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831167+0000", 2023-01-15T14:29:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831174+0000", 2023-01-15T14:29:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8669999999999996e-06 2023-01-15T14:29:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831184+0000", 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5899999999999997e-06 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831192+0000", 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1429999999999998e-06 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:02.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.906004+0000", 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.074812054000000003 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.906020+0000", 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6327000000000001e-05 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.024319+0000", 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.118299324 2023-01-15T14:29:02.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.024579+0000", 2023-01-15T14:29:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025920400000000002 2023-01-15T14:29:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.024885+0000", 2023-01-15T14:29:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030684700000000001 2023-01-15T14:29:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025233+0000", 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034726499999999998 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025249+0000", 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6773999999999999e-05 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025322+0000", 2023-01-15T14:29:02.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2497999999999996e-05 2023-01-15T14:29:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025523+0000", 2023-01-15T14:29:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000200979 2023-01-15T14:29:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.226458+0000", 2023-01-15T14:29:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20093532 2023-01-15T14:29:02.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.326966+0000", 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10050793199999999 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.327045+0000", 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8713000000000003e-05 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.327129+0000", 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4303999999999998e-05 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.214662414999999, 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:29:02.243 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:29:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.244 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:02.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:02.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:29:02.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:02.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:02.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:29:02.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:29:02.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:02.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:29:02.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:29:02.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:02.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:29:02.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:29:02.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:02.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:29:02.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:29:02.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:02.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:29:02.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:29:02.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:02.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:29:02.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:29:02.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:02.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:29:02.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:29:02.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:02.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:29:02.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:29:02.406 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:02.406 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:02.407 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:02.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:333 3.0 3:0a998d04:::smithi09025264-10:head [rollback 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:29:02.407 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860002+0000", 2023-01-15T14:29:02.407 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.185865654000001, 2023-01-15T14:29:02.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50655092599999996, 2023-01-15T14:29:02.407 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:02.407 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:02.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:02.408 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:02.408 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:02.408 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 333 2023-01-15T14:29:02.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.408 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:02.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:02.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860002+0000", 2023-01-15T14:29:02.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:02.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:02.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860002+0000", 2023-01-15T14:29:02.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:02.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:02.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860001+0000", 2023-01-15T14:29:02.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:02.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:02.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860010+0000", 2023-01-15T14:29:02.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6570000000000003e-06 2023-01-15T14:29:02.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:02.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860027+0000", 2023-01-15T14:29:02.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7180000000000002e-05 2023-01-15T14:29:02.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:02.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860036+0000", 2023-01-15T14:29:02.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1929999999999999e-06 2023-01-15T14:29:02.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:02.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860266+0000", 2023-01-15T14:29:02.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000229769 2023-01-15T14:29:02.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860276+0000", 2023-01-15T14:29:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8800000000000003e-06 2023-01-15T14:29:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.163322+0000", 2023-01-15T14:29:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30304593099999999 2023-01-15T14:29:02.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.163486+0000", 2023-01-15T14:29:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016389900000000001 2023-01-15T14:29:02.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:02.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.163798+0000", 2023-01-15T14:29:02.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031278300000000002 2023-01-15T14:29:02.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:02.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164058+0000", 2023-01-15T14:29:02.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259609 2023-01-15T14:29:02.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:02.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164086+0000", 2023-01-15T14:29:02.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8113999999999999e-05 2023-01-15T14:29:02.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164236+0000", 2023-01-15T14:29:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014949900000000001 2023-01-15T14:29:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293740+0000", 2023-01-15T14:29:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12950431100000001 2023-01-15T14:29:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.294109+0000", 2023-01-15T14:29:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036941200000000001 2023-01-15T14:29:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366415+0000", 2023-01-15T14:29:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072305164000000005 2023-01-15T14:29:02.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366503+0000", 2023-01-15T14:29:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8101000000000002e-05 2023-01-15T14:29:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366553+0000", 2023-01-15T14:29:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0726999999999997e-05 2023-01-15T14:29:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.810 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:29:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.185813944, 2023-01-15T14:29:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:29:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:02.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:02.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:29:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.814 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:02.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:29:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:29:02.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:29:02.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:29:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:29:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:29:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:29:02.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:29:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:29:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:02.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:29:02.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:29:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:29:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:29:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:29:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:29:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:29:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:29:02.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:02.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:29:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:29:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:373 3.4 3:2029fa71:::smithi09025264-39:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:02.828 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.967566736999999, 2023-01-15T14:29:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58298356399999995, 2023-01-15T14:29:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-01-15T14:29:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078299+0000", 2023-01-15T14:29:03.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078644+0000", 2023-01-15T14:29:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034494099999999998 2023-01-15T14:29:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078676+0000", 2023-01-15T14:29:03.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1183000000000003e-05 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078694+0000", 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-05 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078748+0000", 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3634999999999997e-05 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078753+0000", 2023-01-15T14:29:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6370000000000004e-06 2023-01-15T14:29:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110841+0000", 2023-01-15T14:29:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032087439000000002 2023-01-15T14:29:03.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110967+0000", 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126029 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.457874+0000", 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.346907725 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458066+0000", 2023-01-15T14:29:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000191592 2023-01-15T14:29:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458321+0000", 2023-01-15T14:29:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025454200000000002 2023-01-15T14:29:03.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458918+0000", 2023-01-15T14:29:03.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059740800000000001 2023-01-15T14:29:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458945+0000", 2023-01-15T14:29:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-05 2023-01-15T14:29:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459253+0000", 2023-01-15T14:29:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030836200000000001 2023-01-15T14:29:03.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460943+0000", 2023-01-15T14:29:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016903580000000001 2023-01-15T14:29:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461340+0000", 2023-01-15T14:29:03.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039665799999999998 2023-01-15T14:29:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661129+0000", 2023-01-15T14:29:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19978933500000001 2023-01-15T14:29:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661188+0000", 2023-01-15T14:29:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8687000000000002e-05 2023-01-15T14:29:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661285+0000", 2023-01-15T14:29:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779000000000003e-05 2023-01-15T14:29:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:374 3.4 3:2029fa71:::smithi09025264-39:head [write 3101234~41642] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.962904824999999, 2023-01-15T14:29:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57848911800000002, 2023-01-15T14:29:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:03.220 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 374 2023-01-15T14:29:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082961+0000", 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083019+0000", 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999997e-05 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083051+0000", 2023-01-15T14:29:03.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1813999999999997e-05 2023-01-15T14:29:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:03.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083069+0000", 2023-01-15T14:29:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:29:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083143+0000", 2023-01-15T14:29:03.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4363000000000006e-05 2023-01-15T14:29:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083147+0000", 2023-01-15T14:29:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4529999999999998e-06 2023-01-15T14:29:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111017+0000", 2023-01-15T14:29:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027870564 2023-01-15T14:29:03.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111099+0000", 2023-01-15T14:29:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1277000000000001e-05 2023-01-15T14:29:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459156+0000", 2023-01-15T14:29:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34805700499999997 2023-01-15T14:29:03.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:03.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459393+0000", 2023-01-15T14:29:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023757499999999999 2023-01-15T14:29:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459778+0000", 2023-01-15T14:29:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000385335 2023-01-15T14:29:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460020+0000", 2023-01-15T14:29:03.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024141899999999999 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460037+0000", 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6665999999999999e-05 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460220+0000", 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018367599999999999 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461197+0000", 2023-01-15T14:29:03.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00097708600000000006 2023-01-15T14:29:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:03.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461720+0000", 2023-01-15T14:29:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052235400000000005 2023-01-15T14:29:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661345+0000", 2023-01-15T14:29:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199624884 2023-01-15T14:29:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661405+0000", 2023-01-15T14:29:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0965999999999999e-05 2023-01-15T14:29:03.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661452+0000", 2023-01-15T14:29:03.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6864999999999999e-05 2023-01-15T14:29:03.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:375 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (56) in=7b,truncate 3142876] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.962763059, 2023-01-15T14:29:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57851114599999998, 2023-01-15T14:29:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:03.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 375 2023-01-15T14:29:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:03.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083104+0000", 2023-01-15T14:29:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:03.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083113+0000", 2023-01-15T14:29:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6500000000000008e-06 2023-01-15T14:29:03.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083130+0000", 2023-01-15T14:29:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2023-01-15T14:29:03.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083142+0000", 2023-01-15T14:29:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2252999999999999e-05 2023-01-15T14:29:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083173+0000", 2023-01-15T14:29:03.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0474999999999999e-05 2023-01-15T14:29:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083175+0000", 2023-01-15T14:29:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2023-01-15T14:29:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183469+0000", 2023-01-15T14:29:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10029313099999999 2023-01-15T14:29:03.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183747+0000", 2023-01-15T14:29:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000277998 2023-01-15T14:29:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460165+0000", 2023-01-15T14:29:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27641872899999997 2023-01-15T14:29:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460342+0000", 2023-01-15T14:29:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176792 2023-01-15T14:29:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460579+0000", 2023-01-15T14:29:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000237266 2023-01-15T14:29:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460833+0000", 2023-01-15T14:29:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025390900000000002 2023-01-15T14:29:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460853+0000", 2023-01-15T14:29:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9273e-05 2023-01-15T14:29:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:03.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461024+0000", 2023-01-15T14:29:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017111900000000001 2023-01-15T14:29:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461230+0000", 2023-01-15T14:29:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020672600000000001 2023-01-15T14:29:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:03.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461879+0000", 2023-01-15T14:29:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064846999999999995 2023-01-15T14:29:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661524+0000", 2023-01-15T14:29:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19964509599999999 2023-01-15T14:29:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661560+0000", 2023-01-15T14:29:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6264999999999999e-05 2023-01-15T14:29:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661616+0000", 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5952000000000002e-05 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:376 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.962680803000001, 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57864278800000002, 2023-01-15T14:29:03.641 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 376 2023-01-15T14:29:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:29:03.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:29:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083186+0000", 2023-01-15T14:29:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083193+0000", 2023-01-15T14:29:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2023-01-15T14:29:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083211+0000", 2023-01-15T14:29:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7609999999999999e-05 2023-01-15T14:29:03.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083223+0000", 2023-01-15T14:29:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140000000000001e-05 2023-01-15T14:29:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093534+0000", 2023-01-15T14:29:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010311529 2023-01-15T14:29:03.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093551+0000", 2023-01-15T14:29:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.666e-05 2023-01-15T14:29:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183834+0000", 2023-01-15T14:29:03.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090283063999999996 2023-01-15T14:29:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.184076+0000", 2023-01-15T14:29:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024241900000000001 2023-01-15T14:29:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461001+0000", 2023-01-15T14:29:03.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27692504299999998 2023-01-15T14:29:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461152+0000", 2023-01-15T14:29:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150345 2023-01-15T14:29:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661642+0000", 2023-01-15T14:29:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200489742 2023-01-15T14:29:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:03.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661720+0000", 2023-01-15T14:29:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8604999999999997e-05 2023-01-15T14:29:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:03.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661830+0000", 2023-01-15T14:29:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010992099999999999 2023-01-15T14:29:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:03.837 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:03.838 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:03.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:29:03.841 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:29:03.842 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:00.141+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:03.846 INFO:tasks.rados.rados.0.smithi090.stdout:200: done (7 left) 2023-01-15T14:29:03.846 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:29:03.847 INFO:tasks.rados.rados.0.smithi090.stdout:196: finishing copy_from racing read to smithi09025264-6 2023-01-15T14:29:03.847 INFO:tasks.rados.rados.0.smithi090.stdout:196: done (6 left) 2023-01-15T14:29:03.847 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:29:03.847 INFO:tasks.rados.rados.0.smithi090.stdout:201: finishing write tid 2 to smithi09025264-44 2023-01-15T14:29:03.847 INFO:tasks.rados.rados.0.smithi090.stdout:201: finishing write tid 3 to smithi09025264-44 2023-01-15T14:29:03.848 INFO:tasks.rados.rados.0.smithi090.stdout:195: finishing copy_from to smithi09025264-41 2023-01-15T14:29:03.848 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 10 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-01-15T14:29:03.848 INFO:tasks.rados.rados.0.smithi090.stdout:195: finishing copy_from racing read to smithi09025264-41 2023-01-15T14:29:03.848 INFO:tasks.rados.rados.0.smithi090.stdout:195: done (5 left) 2023-01-15T14:29:03.848 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:29:03.848 INFO:tasks.rados.rados.0.smithi090.stdout:203: finishing write tid 1 to smithi09025264-11 2023-01-15T14:29:04.041 INFO:tasks.rados.rados.0.smithi090.stdout:203: finishing write tid 2 to smithi09025264-11 2023-01-15T14:29:04.042 INFO:tasks.rados.rados.0.smithi090.stdout:203: finishing write tid 3 to smithi09025264-11 2023-01-15T14:29:04.042 INFO:tasks.rados.rados.0.smithi090.stdout:203: finishing write tid 4 to smithi09025264-11 2023-01-15T14:29:04.042 INFO:tasks.rados.rados.0.smithi090.stdout:203: finishing write tid 5 to smithi09025264-11 2023-01-15T14:29:04.042 INFO:tasks.rados.rados.0.smithi090.stdout:203: oid 11 updating version 0 to 70 2023-01-15T14:29:04.042 INFO:tasks.rados.rados.0.smithi090.stdout:203: oid 11 updating version 70 to 71 2023-01-15T14:29:04.042 INFO:tasks.rados.rados.0.smithi090.stdout:203: oid 11 version 71 is already newer than 68 2023-01-15T14:29:04.042 INFO:tasks.rados.rados.0.smithi090.stdout:203: oid 11 version 71 is already newer than 69 2023-01-15T14:29:04.042 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 71 (ObjNum 100 snap 20 seq_num 100) dirty exists 2023-01-15T14:29:04.042 INFO:tasks.rados.rados.0.smithi090.stdout:203: left oid 11 (ObjNum 100 snap 20 seq_num 100) 2023-01-15T14:29:04.042 INFO:tasks.rados.rados.0.smithi090.stdout:203: done (4 left) 2023-01-15T14:29:04.043 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:29:04.043 INFO:tasks.rados.rados.0.smithi090.stdout:201: finishing write tid 4 to smithi09025264-44 2023-01-15T14:29:04.043 INFO:tasks.rados.rados.0.smithi090.stdout:201: finishing write tid 5 to smithi09025264-44 2023-01-15T14:29:04.043 INFO:tasks.rados.rados.0.smithi090.stdout:201: oid 44 updating version 0 to 59 2023-01-15T14:29:04.043 INFO:tasks.rados.rados.0.smithi090.stdout:201: oid 44 updating version 59 to 60 2023-01-15T14:29:04.043 INFO:tasks.rados.rados.0.smithi090.stdout:201: oid 44 updating version 60 to 61 2023-01-15T14:29:04.043 INFO:tasks.rados.rados.0.smithi090.stdout:201: oid 44 updating version 61 to 62 2023-01-15T14:29:04.043 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 62 (ObjNum 99 snap 20 seq_num 99) dirty exists 2023-01-15T14:29:04.043 INFO:tasks.rados.rados.0.smithi090.stdout:201: left oid 44 (ObjNum 99 snap 20 seq_num 99) 2023-01-15T14:29:04.043 INFO:tasks.rados.rados.0.smithi090.stdout:201: done (3 left) 2023-01-15T14:29:04.043 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:29:04.044 INFO:tasks.rados.rados.0.smithi090.stdout:199: finishing write tid 2 to smithi09025264-45 2023-01-15T14:29:04.044 INFO:tasks.rados.rados.0.smithi090.stdout:198: finishing rollback tid 0 to smithi09025264-48 2023-01-15T14:29:04.044 INFO:tasks.rados.rados.0.smithi090.stdout:202: finishing copy_from to smithi09025264-9 2023-01-15T14:29:04.044 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-01-15T14:29:04.048 INFO:tasks.rados.rados.0.smithi090.stdout:202: finishing copy_from racing read to smithi09025264-9 2023-01-15T14:29:04.048 INFO:tasks.rados.rados.0.smithi090.stdout:202: done (2 left) 2023-01-15T14:29:04.048 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:04.048 INFO:tasks.rados.rados.0.smithi090.stdout:199: finishing write tid 3 to smithi09025264-45 2023-01-15T14:29:04.049 INFO:tasks.rados.rados.0.smithi090.stdout:199: finishing write tid 4 to smithi09025264-45 2023-01-15T14:29:04.049 INFO:tasks.rados.rados.0.smithi090.stdout:199: finishing write tid 5 to smithi09025264-45 2023-01-15T14:29:04.049 INFO:tasks.rados.rados.0.smithi090.stdout:199: oid 45 updating version 0 to 5 2023-01-15T14:29:04.049 INFO:tasks.rados.rados.0.smithi090.stdout:199: oid 45 updating version 5 to 7 2023-01-15T14:29:04.049 INFO:tasks.rados.rados.0.smithi090.stdout:199: oid 45 version 7 is already newer than 6 2023-01-15T14:29:04.049 INFO:tasks.rados.rados.0.smithi090.stdout:199: oid 45 updating version 7 to 8 2023-01-15T14:29:04.049 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 8 (ObjNum 98 snap 20 seq_num 98) dirty exists 2023-01-15T14:29:04.049 INFO:tasks.rados.rados.0.smithi090.stdout:199: left oid 45 (ObjNum 98 snap 20 seq_num 98) 2023-01-15T14:29:04.050 INFO:tasks.rados.rados.0.smithi090.stdout:199: done (1 left) 2023-01-15T14:29:04.050 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:04.050 INFO:tasks.rados.rados.0.smithi090.stdout:198: finishing rollback tid 1 to smithi09025264-48 2023-01-15T14:29:04.050 INFO:tasks.rados.rados.0.smithi090.stdout:198: finishing rollback tid 2 to smithi09025264-48 2023-01-15T14:29:04.050 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 36 (ObjNum 72 snap 8 seq_num 72) dirty exists 2023-01-15T14:29:04.050 INFO:tasks.rados.rados.0.smithi090.stdout:198: done (0 left) 2023-01-15T14:29:04.051 INFO:tasks.rados.rados.0.smithi090.stdout:204: done (0 left) 2023-01-15T14:29:04.051 INFO:tasks.rados.rados.0.smithi090.stdout:205: write oid 27 current snap is 21 2023-01-15T14:29:04.051 INFO:tasks.rados.rados.0.smithi090.stdout:205: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2023-01-15T14:29:04.051 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:29:04.052 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.6 to 0.327356 (53cd) 2023-01-15T14:29:04.052 INFO:tasks.rados.rados.0.smithi090.stdout:205: writing smithi09025264-27 from 497507 to 1210245 tid 1 2023-01-15T14:29:04.053 INFO:tasks.rados.rados.0.smithi090.stdout:205: writing smithi09025264-27 from 1710446 to 2372142 tid 2 2023-01-15T14:29:04.059 INFO:tasks.rados.rados.0.smithi090.stdout:205: writing smithi09025264-27 from 3140781 to 3697507 tid 3 2023-01-15T14:29:04.244 INFO:tasks.rados.rados.0.smithi090.stdout:206: write (excl) oid 20 current snap is 21 2023-01-15T14:29:04.244 INFO:tasks.rados.rados.0.smithi090.stdout:206: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2023-01-15T14:29:04.244 INFO:tasks.rados.rados.0.smithi090.stdout:206: writing smithi09025264-20 from 514314 to 1302301 tid 1 2023-01-15T14:29:04.244 INFO:tasks.rados.rados.0.smithi090.stdout:206: writing smithi09025264-20 from 1768928 to 2174282 tid 2 2023-01-15T14:29:04.244 INFO:tasks.rados.rados.0.smithi090.stdout:206: writing smithi09025264-20 from 2768204 to 3277714 tid 3 2023-01-15T14:29:04.244 INFO:tasks.rados.rados.0.smithi090.stdout:207: write (excl) oid 5 current snap is 21 2023-01-15T14:29:04.244 INFO:tasks.rados.rados.0.smithi090.stdout:207: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2023-01-15T14:29:04.244 INFO:tasks.rados.rados.0.smithi090.stdout:207: writing smithi09025264-5 from 531121 to 994357 tid 1 2023-01-15T14:29:04.244 INFO:tasks.rados.rados.0.smithi090.stdout:207: writing smithi09025264-5 from 1711057 to 2376422 tid 2 2023-01-15T14:29:04.244 INFO:tasks.rados.rados.0.smithi090.stdout:207: writing smithi09025264-5 from 3079274 to 3731121 tid 3 2023-01-15T14:29:04.245 INFO:tasks.rados.rados.0.smithi090.stdout:208: delete oid 49 current snap is 21 2023-01-15T14:29:04.245 INFO:tasks.rados.rados.0.smithi090.stdout:207: finishing write tid 1 to smithi09025264-5 2023-01-15T14:29:04.245 INFO:tasks.rados.rados.0.smithi090.stdout:207: finishing write tid 2 to smithi09025264-5 2023-01-15T14:29:04.245 INFO:tasks.rados.rados.0.smithi090.stdout:207: finishing write tid 3 to smithi09025264-5 2023-01-15T14:29:04.245 INFO:tasks.rados.rados.0.smithi090.stdout:207: finishing write tid 4 to smithi09025264-5 2023-01-15T14:29:04.245 INFO:tasks.rados.rados.0.smithi090.stdout:205: finishing write tid 1 to smithi09025264-27 2023-01-15T14:29:04.245 INFO:tasks.rados.rados.0.smithi090.stdout:206: finishing write tid 1 to smithi09025264-20 2023-01-15T14:29:04.245 INFO:tasks.rados.rados.0.smithi090.stdout:207: finishing write tid 5 to smithi09025264-5 2023-01-15T14:29:04.245 INFO:tasks.rados.rados.0.smithi090.stdout:207: oid 5 updating version 0 to 47 2023-01-15T14:29:04.245 INFO:tasks.rados.rados.0.smithi090.stdout:207: oid 5 version 47 is already newer than 46 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:207: oid 5 updating version 47 to 48 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:207: oid 5 version 48 is already newer than 45 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 48 (ObjNum 103 snap 21 seq_num 103) dirty exists 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:207: left oid 5 (ObjNum 103 snap 21 seq_num 103) 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:207: done (3 left) 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:208: done (2 left) 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:209: snap_remove snap 10 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:209: done (2 left) 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:210: snap_remove snap 18 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:210: done (2 left) 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:211: rollback oid 3 current snap is 21 2023-01-15T14:29:04.246 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 3 to 20 2023-01-15T14:29:04.247 INFO:tasks.rados.rados.0.smithi090.stdout:205: finishing write tid 2 to smithi09025264-27 2023-01-15T14:29:04.247 INFO:tasks.rados.rados.0.smithi090.stdout:206: finishing write tid 2 to smithi09025264-20 2023-01-15T14:29:04.247 INFO:tasks.rados.rados.0.smithi090.stdout:206: finishing write tid 3 to smithi09025264-20 2023-01-15T14:29:04.247 INFO:tasks.rados.rados.0.smithi090.stdout:206: finishing write tid 4 to smithi09025264-20 2023-01-15T14:29:04.250 INFO:tasks.rados.rados.0.smithi090.stdout:206: finishing write tid 5 to smithi09025264-20 2023-01-15T14:29:04.250 INFO:tasks.rados.rados.0.smithi090.stdout:206: oid 20 updating version 0 to 6 2023-01-15T14:29:04.250 INFO:tasks.rados.rados.0.smithi090.stdout:206: oid 20 updating version 6 to 8 2023-01-15T14:29:04.250 INFO:tasks.rados.rados.0.smithi090.stdout:206: oid 20 version 8 is already newer than 5 2023-01-15T14:29:04.250 INFO:tasks.rados.rados.0.smithi090.stdout:206: oid 20 version 8 is already newer than 7 2023-01-15T14:29:04.250 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 8 (ObjNum 102 snap 21 seq_num 102) dirty exists 2023-01-15T14:29:04.250 INFO:tasks.rados.rados.0.smithi090.stdout:206: left oid 20 (ObjNum 102 snap 21 seq_num 102) 2023-01-15T14:29:04.250 INFO:tasks.rados.rados.0.smithi090.stdout:205: finishing write tid 3 to smithi09025264-27 2023-01-15T14:29:04.250 INFO:tasks.rados.rados.0.smithi090.stdout:205: finishing write tid 4 to smithi09025264-27 2023-01-15T14:29:04.251 INFO:tasks.rados.rados.0.smithi090.stdout:206: done (2 left) 2023-01-15T14:29:04.251 INFO:tasks.rados.rados.0.smithi090.stdout:212: read oid 5 snap -1 2023-01-15T14:29:04.251 INFO:tasks.rados.rados.0.smithi090.stdout:212: expect (ObjNum 103 snap 21 seq_num 103) 2023-01-15T14:29:04.251 INFO:tasks.rados.rados.0.smithi090.stdout:213: write (excl) oid 50 current snap is 21 2023-01-15T14:29:04.251 INFO:tasks.rados.rados.0.smithi090.stdout:213: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2023-01-15T14:29:04.251 INFO:tasks.rados.rados.0.smithi090.stdout:213: writing smithi09025264-50 from 564735 to 1178469 tid 1 2023-01-15T14:29:04.251 INFO:tasks.rados.rados.0.smithi090.stdout:213: writing smithi09025264-50 from 1828021 to 2497055 tid 2 2023-01-15T14:29:04.251 INFO:tasks.rados.rados.0.smithi090.stdout:213: writing smithi09025264-50 from 3134120 to 3764735 tid 3 2023-01-15T14:29:04.252 INFO:tasks.rados.rados.0.smithi090.stdout:214: copy_from oid 44 from oid 29 current snap is 21 2023-01-15T14:29:04.252 INFO:tasks.rados.rados.0.smithi090.stdout:205: finishing write tid 5 to smithi09025264-27 2023-01-15T14:29:04.252 INFO:tasks.rados.rados.0.smithi090.stdout:205: oid 27 updating version 0 to 39 2023-01-15T14:29:04.252 INFO:tasks.rados.rados.0.smithi090.stdout:205: oid 27 updating version 39 to 41 2023-01-15T14:29:04.253 INFO:tasks.rados.rados.0.smithi090.stdout:205: oid 27 version 41 is already newer than 38 2023-01-15T14:29:04.253 INFO:tasks.rados.rados.0.smithi090.stdout:205: oid 27 version 41 is already newer than 40 2023-01-15T14:29:04.253 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 41 (ObjNum 101 snap 21 seq_num 101) dirty exists 2023-01-15T14:29:04.253 INFO:tasks.rados.rados.0.smithi090.stdout:205: left oid 27 (ObjNum 101 snap 21 seq_num 101) 2023-01-15T14:29:04.253 INFO:tasks.rados.rados.0.smithi090.stdout:205: done (4 left) 2023-01-15T14:29:04.253 INFO:tasks.rados.rados.0.smithi090.stdout:215: copy_from oid 49 from oid 15 current snap is 21 2023-01-15T14:29:04.253 INFO:tasks.rados.rados.0.smithi090.stdout:216: snap_create 2023-01-15T14:29:04.253 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:29:04.253 INFO:tasks.rados.rados.0.smithi090.stdout:213: finishing write tid 1 to smithi09025264-50 2023-01-15T14:29:04.253 INFO:tasks.rados.rados.0.smithi090.stdout:213: finishing write tid 2 to smithi09025264-50 2023-01-15T14:29:04.253 INFO:tasks.rados.rados.0.smithi090.stdout:213: finishing write tid 3 to smithi09025264-50 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:213: finishing write tid 4 to smithi09025264-50 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:213: finishing write tid 5 to smithi09025264-50 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:213: oid 50 updating version 0 to 44 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:213: oid 50 version 44 is already newer than 41 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:213: oid 50 version 44 is already newer than 42 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:213: oid 50 version 44 is already newer than 43 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 50 v 44 (ObjNum 105 snap 21 seq_num 105) dirty exists 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:213: left oid 50 (ObjNum 105 snap 21 seq_num 105) 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:213: done (4 left) 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:212: expect (ObjNum 103 snap 21 seq_num 103) 2023-01-15T14:29:04.254 INFO:tasks.rados.rados.0.smithi090.stdout:212: done (3 left) 2023-01-15T14:29:04.255 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:29:04.255 INFO:tasks.rados.rados.0.smithi090.stdout:215: finishing copy_from to smithi09025264-49 2023-01-15T14:29:04.255 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 34 (ObjNum 82 snap 10 seq_num 82) dirty exists 2023-01-15T14:29:04.255 INFO:tasks.rados.rados.0.smithi090.stdout:215: finishing copy_from racing read to smithi09025264-49 2023-01-15T14:29:04.255 INFO:tasks.rados.rados.0.smithi090.stdout:215: done (2 left) 2023-01-15T14:29:04.255 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:04.255 INFO:tasks.rados.rados.0.smithi090.stdout:211: finishing rollback tid 0 to smithi09025264-3 2023-01-15T14:29:04.255 INFO:tasks.rados.rados.0.smithi090.stdout:211: finishing rollback tid 1 to smithi09025264-3 2023-01-15T14:29:04.255 INFO:tasks.rados.rados.0.smithi090.stdout:211: finishing rollback tid 2 to smithi09025264-3 2023-01-15T14:29:04.256 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 66 (ObjNum 73 snap 8 seq_num 73) dirty exists 2023-01-15T14:29:04.256 INFO:tasks.rados.rados.0.smithi090.stdout:211: done (1 left) 2023-01-15T14:29:04.256 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:04.256 INFO:tasks.rados.rados.0.smithi090.stdout:214: finishing copy_from to smithi09025264-44 2023-01-15T14:29:04.256 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 25 (ObjNum 78 snap 10 seq_num 78) dirty exists 2023-01-15T14:29:04.256 INFO:tasks.rados.rados.0.smithi090.stdout:214: finishing copy_from racing read to smithi09025264-44 2023-01-15T14:29:04.256 INFO:tasks.rados.rados.0.smithi090.stdout:214: done (0 left) 2023-01-15T14:29:04.256 INFO:tasks.rados.rados.0.smithi090.stdout:216: done (0 left) 2023-01-15T14:29:04.256 INFO:tasks.rados.rados.0.smithi090.stdout:217: write oid 24 current snap is 22 2023-01-15T14:29:04.256 INFO:tasks.rados.rados.0.smithi090.stdout:217: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2023-01-15T14:29:04.257 INFO:tasks.rados.rados.0.smithi090.stdout:217: writing smithi09025264-24 from 581542 to 1270525 tid 1 2023-01-15T14:29:04.257 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:03.833+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:04.259 INFO:tasks.rados.rados.0.smithi090.stdout:217: writing smithi09025264-24 from 1886503 to 2299195 tid 2 2023-01-15T14:29:04.259 INFO:tasks.rados.rados.0.smithi090.stdout:217: writing smithi09025264-24 from 2761543 to 3432494 tid 3 2023-01-15T14:29:04.260 INFO:tasks.rados.rados.0.smithi090.stdout:218: read oid 34 snap -1 2023-01-15T14:29:04.260 INFO:tasks.rados.rados.0.smithi090.stdout:218: expect (ObjNum 89 snap 12 seq_num 89) 2023-01-15T14:29:04.260 INFO:tasks.rados.rados.0.smithi090.stdout:219: rollback oid 36 current snap is 22 2023-01-15T14:29:04.260 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 36 to 15 2023-01-15T14:29:04.260 INFO:tasks.rados.rados.0.smithi090.stdout:220: snap_create 2023-01-15T14:29:04.260 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:29:04.260 INFO:tasks.rados.rados.0.smithi090.stdout:217: finishing write tid 1 to smithi09025264-24 2023-01-15T14:29:04.260 INFO:tasks.rados.rados.0.smithi090.stdout:219: finishing rollback tid 0 to smithi09025264-36 2023-01-15T14:29:04.260 INFO:tasks.rados.rados.0.smithi090.stdout:219: finishing rollback tid 1 to smithi09025264-36 2023-01-15T14:29:04.260 INFO:tasks.rados.rados.0.smithi090.stdout:219: finishing rollback tid 2 to smithi09025264-36 2023-01-15T14:29:04.260 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 54 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-01-15T14:29:04.261 INFO:tasks.rados.rados.0.smithi090.stdout:219: done (2 left) 2023-01-15T14:29:04.261 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:04.261 INFO:tasks.rados.rados.0.smithi090.stdout:218: expect (ObjNum 89 snap 12 seq_num 89) 2023-01-15T14:29:04.261 INFO:tasks.rados.rados.0.smithi090.stdout:218: done (1 left) 2023-01-15T14:29:04.261 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:04.262 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:04.077+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:04.427 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:04.428 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:29:04.428 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:04.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:29:04.437 INFO:tasks.rados.rados.0.smithi090.stdout:217: finishing write tid 2 to smithi09025264-24 2023-01-15T14:29:04.437 INFO:tasks.rados.rados.0.smithi090.stdout:217: finishing write tid 3 to smithi09025264-24 2023-01-15T14:29:04.438 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:04.438 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:04.439 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:04.421+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:04.439 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:04.454 INFO:tasks.rados.rados.0.smithi090.stdout:217: finishing write tid 4 to smithi09025264-24 2023-01-15T14:29:04.587 INFO:tasks.rados.rados.0.smithi090.stdout:217: finishing write tid 5 to smithi09025264-24 2023-01-15T14:29:04.587 INFO:tasks.rados.rados.0.smithi090.stdout:217: oid 24 updating version 0 to 71 2023-01-15T14:29:04.587 INFO:tasks.rados.rados.0.smithi090.stdout:217: oid 24 updating version 71 to 72 2023-01-15T14:29:04.587 INFO:tasks.rados.rados.0.smithi090.stdout:217: oid 24 updating version 72 to 73 2023-01-15T14:29:04.587 INFO:tasks.rados.rados.0.smithi090.stdout:217: oid 24 updating version 73 to 74 2023-01-15T14:29:04.587 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 74 (ObjNum 106 snap 22 seq_num 106) dirty exists 2023-01-15T14:29:04.587 INFO:tasks.rados.rados.0.smithi090.stdout:217: left oid 24 (ObjNum 106 snap 22 seq_num 106) 2023-01-15T14:29:04.588 INFO:tasks.rados.rados.0.smithi090.stdout:217: done (0 left) 2023-01-15T14:29:04.588 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:04.588 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:04.588 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:04.589 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:04.589 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:04.589 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:04.590 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:04.590 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:29:04.590 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:29:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:29:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:29:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:29:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:29:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:29:04.591 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:29:04.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:29:04.594 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:04.581+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:04.688 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:04.681+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:04.750 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:04.754 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:04.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:29:04.789 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:04.785+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:04.903 INFO:tasks.rados.rados.0.smithi090.stdout:220: done (0 left) 2023-01-15T14:29:04.903 INFO:tasks.rados.rados.0.smithi090.stdout:221: snap_remove snap 20 2023-01-15T14:29:04.904 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:04.904 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:04.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:29:04.905 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:04.897+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:05.004 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:04.997+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:05.064 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:05.065 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:05.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:29:05.107 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:05.101+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:05.227 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:05.227 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:05.228 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:05.221+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:05.230 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:29:05.328 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:05.321+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:05.384 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:05.385 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:05.385 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:29:05.429 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:05.425+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:05.539 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:05.540 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:05.540 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:29:05.540 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:05.533+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:05.641 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:05.633+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:05.710 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:05.711 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:05.711 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:29:05.742 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:05.737+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:05.860 INFO:tasks.rados.rados.0.smithi090.stdout:221: done (0 left) 2023-01-15T14:29:05.862 INFO:tasks.rados.rados.0.smithi090.stdout:222: write oid 20 current snap is 23 2023-01-15T14:29:05.863 INFO:tasks.rados.rados.0.smithi090.stdout:222: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2023-01-15T14:29:05.863 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:05.853+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:05.863 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:05.863 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:05.864 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:29:05.864 INFO:tasks.rados.rados.0.smithi090.stdout:222: writing smithi09025264-20 from 598349 to 1078934 tid 1 2023-01-15T14:29:05.870 INFO:tasks.rados.rados.0.smithi090.stdout:222: writing smithi09025264-20 from 1544985 to 2217688 tid 2 2023-01-15T14:29:05.877 INFO:tasks.rados.rados.0.smithi090.stdout:222: writing smithi09025264-20 from 2905319 to 3387542 tid 3 2023-01-15T14:29:05.882 INFO:tasks.rados.rados.0.smithi090.stdout:223: write oid 32 current snap is 23 2023-01-15T14:29:05.882 INFO:tasks.rados.rados.0.smithi090.stdout:223: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2023-01-15T14:29:05.886 INFO:tasks.rados.rados.0.smithi090.stdout:223: writing smithi09025264-32 from 615156 to 1170990 tid 1 2023-01-15T14:29:05.889 INFO:tasks.rados.rados.0.smithi090.stdout:223: writing smithi09025264-32 from 1603467 to 2019828 tid 2 2023-01-15T14:29:05.897 INFO:tasks.rados.rados.0.smithi090.stdout:223: writing smithi09025264-32 from 2816389 to 3509884 tid 3 2023-01-15T14:29:06.025 INFO:tasks.rados.rados.0.smithi090.stdout:224: delete oid 17 current snap is 23 2023-01-15T14:29:06.032 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:06.021+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:06.054 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:29:06.180 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:06.182 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:06.183 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:29:06.183 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:06.173+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:06.187 INFO:tasks.rados.rados.0.smithi090.stdout:222: finishing write tid 1 to smithi09025264-20 2023-01-15T14:29:06.187 INFO:tasks.rados.rados.0.smithi090.stdout:222: finishing write tid 2 to smithi09025264-20 2023-01-15T14:29:06.188 INFO:tasks.rados.rados.0.smithi090.stdout:222: finishing write tid 3 to smithi09025264-20 2023-01-15T14:29:06.190 INFO:tasks.rados.rados.0.smithi090.stdout:224: done (2 left) 2023-01-15T14:29:06.191 INFO:tasks.rados.rados.0.smithi090.stdout:225: write (excl) oid 47 current snap is 23 2023-01-15T14:29:06.191 INFO:tasks.rados.rados.0.smithi090.stdout:225: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2023-01-15T14:29:06.208 INFO:tasks.rados.rados.0.smithi090.stdout:225: writing smithi09025264-47 from 648770 to 1355102 tid 1 2023-01-15T14:29:06.213 INFO:tasks.rados.rados.0.smithi090.stdout:225: writing smithi09025264-47 from 2120431 to 2540461 tid 2 2023-01-15T14:29:06.220 INFO:tasks.rados.rados.0.smithi090.stdout:225: writing smithi09025264-47 from 3271235 to 3703627 tid 3 2023-01-15T14:29:06.343 INFO:tasks.rados.rados.0.smithi090.stdout:226: copy_from oid 12 from oid 35 current snap is 23 2023-01-15T14:29:06.344 INFO:tasks.rados.rados.0.smithi090.stdout:223: finishing write tid 1 to smithi09025264-32 2023-01-15T14:29:06.344 INFO:tasks.rados.rados.0.smithi090.stdout:223: finishing write tid 2 to smithi09025264-32 2023-01-15T14:29:06.344 INFO:tasks.rados.rados.0.smithi090.stdout:223: finishing write tid 3 to smithi09025264-32 2023-01-15T14:29:06.344 INFO:tasks.rados.rados.0.smithi090.stdout:227: delete oid 3 current snap is 23 2023-01-15T14:29:06.346 INFO:tasks.rados.rados.0.smithi090.stdout:222: finishing write tid 4 to smithi09025264-20 2023-01-15T14:29:06.346 INFO:tasks.rados.rados.0.smithi090.stdout:222: finishing write tid 5 to smithi09025264-20 2023-01-15T14:29:06.351 INFO:tasks.rados.rados.0.smithi090.stdout:222: oid 20 updating version 0 to 17 2023-01-15T14:29:06.351 INFO:tasks.rados.rados.0.smithi090.stdout:222: oid 20 version 17 is already newer than 16 2023-01-15T14:29:06.352 INFO:tasks.rados.rados.0.smithi090.stdout:222: oid 20 updating version 17 to 19 2023-01-15T14:29:06.352 INFO:tasks.rados.rados.0.smithi090.stdout:222: oid 20 version 19 is already newer than 18 2023-01-15T14:29:06.352 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 19 (ObjNum 107 snap 23 seq_num 107) dirty exists 2023-01-15T14:29:06.352 INFO:tasks.rados.rados.0.smithi090.stdout:222: left oid 20 (ObjNum 107 snap 23 seq_num 107) 2023-01-15T14:29:06.352 INFO:tasks.rados.rados.0.smithi090.stdout:225: finishing write tid 1 to smithi09025264-47 2023-01-15T14:29:06.352 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:06.353 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:06.353 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:06.337+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:06.353 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:29:06.394 INFO:tasks.rados.rados.0.smithi090.stdout:223: finishing write tid 4 to smithi09025264-32 2023-01-15T14:29:06.503 INFO:tasks.rados.rados.0.smithi090.stdout:226: finishing copy_from to smithi09025264-12 2023-01-15T14:29:06.503 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 6 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-01-15T14:29:06.503 INFO:tasks.rados.rados.0.smithi090.stdout:226: finishing copy_from racing read to smithi09025264-12 2023-01-15T14:29:06.504 INFO:tasks.rados.rados.0.smithi090.stdout:225: finishing write tid 2 to smithi09025264-47 2023-01-15T14:29:06.504 INFO:tasks.rados.rados.0.smithi090.stdout:222: done (4 left) 2023-01-15T14:29:06.504 INFO:tasks.rados.rados.0.smithi090.stdout:226: done (3 left) 2023-01-15T14:29:06.504 INFO:tasks.rados.rados.0.smithi090.stdout:227: done (2 left) 2023-01-15T14:29:06.504 INFO:tasks.rados.rados.0.smithi090.stdout:228: snap_remove snap 19 2023-01-15T14:29:06.504 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:06.497+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:06.504 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:06.505 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:06.505 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:29:06.595 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:29:06.678 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:06.678 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:06.678 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:29:06.679 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:06.673+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:06.779 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:06.773+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:06.833 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:06.837 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:06.837 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:06.842 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:06.847 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:29:06.862 INFO:tasks.rados.rados.0.smithi090.stdout:228: done (2 left) 2023-01-15T14:29:06.992 INFO:tasks.rados.rados.0.smithi090.stdout:229: rollback oid 24 current snap is 23 2023-01-15T14:29:06.992 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 24 to 14 2023-01-15T14:29:06.992 INFO:tasks.rados.rados.0.smithi090.stdout:223: finishing write tid 5 to smithi09025264-32 2023-01-15T14:29:06.992 INFO:tasks.rados.rados.0.smithi090.stdout:223: oid 32 updating version 0 to 38 2023-01-15T14:29:06.992 INFO:tasks.rados.rados.0.smithi090.stdout:223: oid 32 version 38 is already newer than 37 2023-01-15T14:29:06.992 INFO:tasks.rados.rados.0.smithi090.stdout:223: oid 32 version 38 is already newer than 36 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:223: oid 32 updating version 38 to 39 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 39 (ObjNum 108 snap 23 seq_num 108) dirty exists 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:223: left oid 32 (ObjNum 108 snap 23 seq_num 108) 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:225: finishing write tid 3 to smithi09025264-47 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:225: finishing write tid 4 to smithi09025264-47 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:225: finishing write tid 5 to smithi09025264-47 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:225: oid 47 updating version 0 to 80 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:225: oid 47 version 80 is already newer than 77 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:225: oid 47 version 80 is already newer than 78 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:225: oid 47 version 80 is already newer than 79 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 80 (ObjNum 110 snap 23 seq_num 110) dirty exists 2023-01-15T14:29:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:225: left oid 47 (ObjNum 110 snap 23 seq_num 110) 2023-01-15T14:29:06.994 INFO:tasks.rados.rados.0.smithi090.stdout:223: done (2 left) 2023-01-15T14:29:06.994 INFO:tasks.rados.rados.0.smithi090.stdout:225: done (1 left) 2023-01-15T14:29:06.994 INFO:tasks.rados.rados.0.smithi090.stdout:230: snap_remove snap 16 2023-01-15T14:29:06.998 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:06.999 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:06.999 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:29:07.000 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:29:07.000 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:07.001 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:07.001 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:06.989+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:07.090 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:07.154 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:07.155 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:07.155 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:07.155 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:07.149+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:07.155 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:07.155 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:07.156 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:29:07.156 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:29:07.156 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:29:07.157 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:29:07.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:29:07.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:29:07.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:29:07.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:29:07.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:29:07.255 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:07.249+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:07.306 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:07.306 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:29:07.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:29:07.307 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:29:07.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.307 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4202.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-01-15T14:29:07.307 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:15.887514+0000", 2023-01-15T14:29:07.307 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 51.381888527000001, 2023-01-15T14:29:07.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61511010300000002, 2023-01-15T14:29:07.307 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.307 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.308 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.308 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4202", 2023-01-15T14:29:07.308 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/3762191053", 2023-01-15T14:29:07.309 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 115 2023-01-15T14:29:07.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.309 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887514+0000", 2023-01-15T14:29:07.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.310 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.310 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887514+0000", 2023-01-15T14:29:07.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887512+0000", 2023-01-15T14:29:07.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:07.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887682+0000", 2023-01-15T14:29:07.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016936800000000001 2023-01-15T14:29:07.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887709+0000", 2023-01-15T14:29:07.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7416999999999999e-05 2023-01-15T14:29:07.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887722+0000", 2023-01-15T14:29:07.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2952999999999999e-05 2023-01-15T14:29:07.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.398568+0000", 2023-01-15T14:29:07.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51084611000000002 2023-01-15T14:29:07.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.398781+0000", 2023-01-15T14:29:07.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021276499999999999 2023-01-15T14:29:07.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:29:07.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.398967+0000", 2023-01-15T14:29:07.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186541 2023-01-15T14:29:07.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.399257+0000", 2023-01-15T14:29:07.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028912399999999998 2023-01-15T14:29:07.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.399279+0000", 2023-01-15T14:29:07.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2473e-05 2023-01-15T14:29:07.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.399471+0000", 2023-01-15T14:29:07.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000192201 2023-01-15T14:29:07.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401531+0000", 2023-01-15T14:29:07.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020596049999999999 2023-01-15T14:29:07.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.402273+0000", 2023-01-15T14:29:07.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00074173000000000002 2023-01-15T14:29:07.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502430+0000", 2023-01-15T14:29:07.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100157149 2023-01-15T14:29:07.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502511+0000", 2023-01-15T14:29:07.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0821000000000004e-05 2023-01-15T14:29:07.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502624+0000", 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011370999999999999 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4202.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:15.887742+0000", 2023-01-15T14:29:07.327 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 51.381660789000001, 2023-01-15T14:29:07.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61526019300000001, 2023-01-15T14:29:07.328 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.328 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.328 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.328 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4202", 2023-01-15T14:29:07.328 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/3762191053", 2023-01-15T14:29:07.328 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 116 2023-01-15T14:29:07.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.328 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887742+0000", 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887742+0000", 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887741+0000", 2023-01-15T14:29:07.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887749+0000", 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6109999999999996e-06 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887758+0000", 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5459999999999997e-06 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887764+0000", 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2539999999999998e-06 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.399900+0000", 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51213585500000003 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400017+0000", 2023-01-15T14:29:07.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000116817 2023-01-15T14:29:07.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:29:07.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400204+0000", 2023-01-15T14:29:07.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186684 2023-01-15T14:29:07.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400505+0000", 2023-01-15T14:29:07.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030162500000000003 2023-01-15T14:29:07.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400527+0000", 2023-01-15T14:29:07.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1350000000000001e-05 2023-01-15T14:29:07.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400612+0000", 2023-01-15T14:29:07.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5550000000000006e-05 2023-01-15T14:29:07.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.334 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.334 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401555+0000", 2023-01-15T14:29:07.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00094224700000000001 2023-01-15T14:29:07.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.334 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.334 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502743+0000", 2023-01-15T14:29:07.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101188207 2023-01-15T14:29:07.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502836+0000", 2023-01-15T14:29:07.335 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.302e-05 2023-01-15T14:29:07.335 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.335 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.502903+0000", 2023-01-15T14:29:07.335 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7033999999999995e-05 2023-01-15T14:29:07.335 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.335 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.503002+0000", 2023-01-15T14:29:07.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9287e-05 2023-01-15T14:29:07.545 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.545 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.545 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.546 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4202.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-01-15T14:29:07.546 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:15.887774+0000", 2023-01-15T14:29:07.546 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 51.381628655999997, 2023-01-15T14:29:07.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61553897800000001, 2023-01-15T14:29:07.546 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.547 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.547 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.547 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4202", 2023-01-15T14:29:07.547 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/3762191053", 2023-01-15T14:29:07.547 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 117 2023-01-15T14:29:07.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.547 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887774+0000", 2023-01-15T14:29:07.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887774+0000", 2023-01-15T14:29:07.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887773+0000", 2023-01-15T14:29:07.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:07.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887780+0000", 2023-01-15T14:29:07.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.08e-06 2023-01-15T14:29:07.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887789+0000", 2023-01-15T14:29:07.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3869999999999995e-06 2023-01-15T14:29:07.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:15.887795+0000", 2023-01-15T14:29:07.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0530000000000003e-06 2023-01-15T14:29:07.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400641+0000", 2023-01-15T14:29:07.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51284605900000002 2023-01-15T14:29:07.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.400815+0000", 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017370299999999999 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401003+0000", 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000187875 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401433+0000", 2023-01-15T14:29:07.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043028800000000002 2023-01-15T14:29:07.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401453+0000", 2023-01-15T14:29:07.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.033e-05 2023-01-15T14:29:07.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401528+0000", 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5277000000000004e-05 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.401567+0000", 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8127000000000003e-05 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.503083+0000", 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10151600099999999 2023-01-15T14:29:07.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.503170+0000", 2023-01-15T14:29:07.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7324000000000006e-05 2023-01-15T14:29:07.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.503256+0000", 2023-01-15T14:29:07.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6213999999999998e-05 2023-01-15T14:29:07.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:16.503313+0000", 2023-01-15T14:29:07.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6889999999999999e-05 2023-01-15T14:29:07.559 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.560 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.560 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.560 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4202.0:156 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-01-15T14:29:07.560 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:20.562557+0000", 2023-01-15T14:29:07.560 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 46.706846065000001, 2023-01-15T14:29:07.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60806470999999995, 2023-01-15T14:29:07.561 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.561 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.561 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.561 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4202", 2023-01-15T14:29:07.561 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/3762191053", 2023-01-15T14:29:07.561 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 156 2023-01-15T14:29:07.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.561 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562557+0000", 2023-01-15T14:29:07.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562557+0000", 2023-01-15T14:29:07.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562556+0000", 2023-01-15T14:29:07.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:07.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562565+0000", 2023-01-15T14:29:07.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0429999999999996e-06 2023-01-15T14:29:07.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562574+0000", 2023-01-15T14:29:07.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9299999999999992e-06 2023-01-15T14:29:07.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562583+0000", 2023-01-15T14:29:07.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3340000000000008e-06 2023-01-15T14:29:07.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.766728+0000", 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20414501700000001 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.766877+0000", 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014928399999999999 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767078+0000", 2023-01-15T14:29:07.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000200984 2023-01-15T14:29:07.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767363+0000", 2023-01-15T14:29:07.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028497 2023-01-15T14:29:07.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767383+0000", 2023-01-15T14:29:07.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9947000000000001e-05 2023-01-15T14:29:07.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767458+0000", 2023-01-15T14:29:07.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4809999999999997e-05 2023-01-15T14:29:07.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.768111+0000", 2023-01-15T14:29:07.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065325699999999997 2023-01-15T14:29:07.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.868290+0000", 2023-01-15T14:29:07.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100179142 2023-01-15T14:29:07.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170426+0000", 2023-01-15T14:29:07.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30213604500000002 2023-01-15T14:29:07.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170510+0000", 2023-01-15T14:29:07.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3876999999999997e-05 2023-01-15T14:29:07.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170621+0000", 2023-01-15T14:29:07.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011102 2023-01-15T14:29:07.571 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.571 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.571 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.571 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4202.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2023-01-15T14:29:07.571 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:20.562592+0000", 2023-01-15T14:29:07.571 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 46.706810478000001, 2023-01-15T14:29:07.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60826002300000004, 2023-01-15T14:29:07.572 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.572 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.572 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.572 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4202", 2023-01-15T14:29:07.572 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/3762191053", 2023-01-15T14:29:07.572 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 157 2023-01-15T14:29:07.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.572 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562592+0000", 2023-01-15T14:29:07.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562592+0000", 2023-01-15T14:29:07.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562592+0000", 2023-01-15T14:29:07.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:07.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562600+0000", 2023-01-15T14:29:07.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2670000000000006e-06 2023-01-15T14:29:07.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.562608+0000", 2023-01-15T14:29:07.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3930000000000008e-06 2023-01-15T14:29:07.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.662745+0000", 2023-01-15T14:29:07.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100136908 2023-01-15T14:29:07.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767494+0000", 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104748904 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767616+0000", 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121567 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.767801+0000", 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018572700000000001 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.768039+0000", 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023774000000000001 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.768053+0000", 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4157e-05 2023-01-15T14:29:07.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.769013+0000", 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000959888 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.769080+0000", 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7323000000000006e-05 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:20.868342+0000", 2023-01-15T14:29:07.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099261861000000007 2023-01-15T14:29:07.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170715+0000", 2023-01-15T14:29:07.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30237277200000001 2023-01-15T14:29:07.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170795+0000", 2023-01-15T14:29:07.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0246000000000004e-05 2023-01-15T14:29:07.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:21.170852+0000", 2023-01-15T14:29:07.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6894000000000001e-05 2023-01-15T14:29:07.580 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.581 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.581 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.581 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:70 3.f 3:fe04f54c:::smithi09025264-18:head [write 1845135~440584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.581 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252308+0000", 2023-01-15T14:29:07.581 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 41.017094465, 2023-01-15T14:29:07.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.72689689099999999, 2023-01-15T14:29:07.581 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.582 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.582 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.582 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.582 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.582 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 70 2023-01-15T14:29:07.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.582 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252308+0000", 2023-01-15T14:29:07.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252308+0000", 2023-01-15T14:29:07.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252307+0000", 2023-01-15T14:29:07.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:07.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252613+0000", 2023-01-15T14:29:07.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000306929 2023-01-15T14:29:07.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252634+0000", 2023-01-15T14:29:07.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0276e-05 2023-01-15T14:29:07.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252647+0000", 2023-01-15T14:29:07.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3114e-05 2023-01-15T14:29:07.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355188+0000", 2023-01-15T14:29:07.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102540914 2023-01-15T14:29:07.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355319+0000", 2023-01-15T14:29:07.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013152699999999999 2023-01-15T14:29:07.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.657662+0000", 2023-01-15T14:29:07.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30234277199999998 2023-01-15T14:29:07.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.657813+0000", 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015124600000000001 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658070+0000", 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025707600000000002 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658459+0000", 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038831500000000003 2023-01-15T14:29:07.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658480+0000", 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1610000000000001e-05 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658736+0000", 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025607899999999999 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760108+0000", 2023-01-15T14:29:07.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10137170199999999 2023-01-15T14:29:07.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760226+0000", 2023-01-15T14:29:07.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011748600000000001 2023-01-15T14:29:07.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.732 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978988+0000", 2023-01-15T14:29:07.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.218762238 2023-01-15T14:29:07.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.734 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.734 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.979116+0000", 2023-01-15T14:29:07.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000128021 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.979205+0000", 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9248999999999999e-05 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:71 3.f 3:fe04f54c:::smithi09025264-18:head [setxattr _header (56) in=7b,truncate 2285719] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.735 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:29:07.736 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 41.016737806000002, 2023-01-15T14:29:07.736 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.82719834800000003, 2023-01-15T14:29:07.736 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.736 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.736 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.736 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.736 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.736 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 71 2023-01-15T14:29:07.736 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.736 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.736 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:29:07.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:29:07.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252664+0000", 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252675+0000", 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1296e-05 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252697+0000", 2023-01-15T14:29:07.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1874000000000001e-05 2023-01-15T14:29:07.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.739 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.739 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.739 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252708+0000", 2023-01-15T14:29:07.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.15e-05 2023-01-15T14:29:07.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.739 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.739 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.739 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355413+0000", 2023-01-15T14:29:07.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10270497100000001 2023-01-15T14:29:07.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.740 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.740 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355622+0000", 2023-01-15T14:29:07.740 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000208982 2023-01-15T14:29:07.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.740 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.740 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.758709+0000", 2023-01-15T14:29:07.740 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40308671800000001 2023-01-15T14:29:07.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.758958+0000", 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024949 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759375+0000", 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000416208 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759708+0000", 2023-01-15T14:29:07.742 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033326900000000002 2023-01-15T14:29:07.742 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759744+0000", 2023-01-15T14:29:07.742 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6477000000000003e-05 2023-01-15T14:29:07.742 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.743 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759822+0000", 2023-01-15T14:29:07.743 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7440000000000004e-05 2023-01-15T14:29:07.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.743 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.743 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760139+0000", 2023-01-15T14:29:07.743 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316996 2023-01-15T14:29:07.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.743 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.743 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760440+0000", 2023-01-15T14:29:07.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000300945 2023-01-15T14:29:07.744 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.744 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.744 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.744 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079578+0000", 2023-01-15T14:29:07.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31913862599999998 2023-01-15T14:29:07.744 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.744 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.744 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.744 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079709+0000", 2023-01-15T14:29:07.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013108 2023-01-15T14:29:07.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.745 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.745 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079863+0000", 2023-01-15T14:29:07.745 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000153819 2023-01-15T14:29:07.745 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.745 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.745 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.745 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:72 3.f 3:fe04f54c:::smithi09025264-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 41.016678468999999, 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.82782374599999997, 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 72 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:29:07.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:29:07.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252723+0000", 2023-01-15T14:29:07.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:07.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252730+0000", 2023-01-15T14:29:07.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.703e-06 2023-01-15T14:29:07.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252747+0000", 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6739999999999999e-05 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252758+0000", 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1389999999999999e-05 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.455798+0000", 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20303954199999999 2023-01-15T14:29:07.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.456092+0000", 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029407199999999999 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759852+0000", 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.303760525 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760038+0000", 2023-01-15T14:29:07.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018584600000000001 2023-01-15T14:29:07.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080005+0000", 2023-01-15T14:29:07.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31996668299999997 2023-01-15T14:29:07.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080286+0000", 2023-01-15T14:29:07.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028087899999999999 2023-01-15T14:29:07.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080548+0000", 2023-01-15T14:29:07.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026241300000000002 2023-01-15T14:29:07.752 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.752 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.752 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.753 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:74 3.f 3:f8edd34e:::smithi09025264-19:head [write 1903617~398909] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.753 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356537+0000", 2023-01-15T14:29:07.753 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 40.912865922000002, 2023-01-15T14:29:07.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.72430009100000003, 2023-01-15T14:29:07.753 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.753 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.753 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.753 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.753 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.753 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 74 2023-01-15T14:29:07.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.754 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356537+0000", 2023-01-15T14:29:07.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356537+0000", 2023-01-15T14:29:07.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356535+0000", 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356854+0000", 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000318979 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.755 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356871+0000", 2023-01-15T14:29:07.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6509999999999999e-05 2023-01-15T14:29:07.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356881+0000", 2023-01-15T14:29:07.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0159999999999999e-05 2023-01-15T14:29:07.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457503+0000", 2023-01-15T14:29:07.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100622054 2023-01-15T14:29:07.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457677+0000", 2023-01-15T14:29:07.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017399200000000001 2023-01-15T14:29:07.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.777230+0000", 2023-01-15T14:29:07.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.319552742 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.777471+0000", 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024115200000000001 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.777842+0000", 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037174199999999999 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778275+0000", 2023-01-15T14:29:07.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043252900000000002 2023-01-15T14:29:07.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778295+0000", 2023-01-15T14:29:07.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9572000000000002e-05 2023-01-15T14:29:07.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778511+0000", 2023-01-15T14:29:07.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021633700000000001 2023-01-15T14:29:07.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:29:07.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060300800000000004 2023-01-15T14:29:07.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079412+0000", 2023-01-15T14:29:07.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30029809899999999 2023-01-15T14:29:07.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080661+0000", 2023-01-15T14:29:07.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001249497 2023-01-15T14:29:07.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080737+0000", 2023-01-15T14:29:07.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:29:07.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080837+0000", 2023-01-15T14:29:07.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9653000000000003e-05 2023-01-15T14:29:07.886 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.886 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.886 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.886 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:75 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.886 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:29:07.887 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 40.912506119, 2023-01-15T14:29:07.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92335155400000002, 2023-01-15T14:29:07.887 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.887 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.887 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.887 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.887 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.887 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 75 2023-01-15T14:29:07.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.887 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:29:07.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:29:07.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356896+0000", 2023-01-15T14:29:07.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:07.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356903+0000", 2023-01-15T14:29:07.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4619999999999999e-06 2023-01-15T14:29:07.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356917+0000", 2023-01-15T14:29:07.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4195999999999999e-05 2023-01-15T14:29:07.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356926+0000", 2023-01-15T14:29:07.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:29:07.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457743+0000", 2023-01-15T14:29:07.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100817668 2023-01-15T14:29:07.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457841+0000", 2023-01-15T14:29:07.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7079999999999999e-05 2023-01-15T14:29:07.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778474+0000", 2023-01-15T14:29:07.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32063301799999999 2023-01-15T14:29:07.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778627+0000", 2023-01-15T14:29:07.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015389599999999999 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778815+0000", 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018726600000000001 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779033+0000", 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021819999999999999 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779051+0000", 2023-01-15T14:29:07.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7640000000000001e-05 2023-01-15T14:29:07.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:29:07.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3590999999999995e-05 2023-01-15T14:29:07.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779163+0000", 2023-01-15T14:29:07.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:29:07.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.081002+0000", 2023-01-15T14:29:07.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30183937799999999 2023-01-15T14:29:07.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280043+0000", 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19904084699999999 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280138+0000", 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4915999999999997e-05 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280248+0000", 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110233 2023-01-15T14:29:07.897 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:76 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 40.912464511000003, 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92420393899999997, 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.898 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.899 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.899 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 76 2023-01-15T14:29:07.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.899 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:29:07.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:29:07.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356937+0000", 2023-01-15T14:29:07.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:07.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356948+0000", 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.094e-05 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356959+0000", 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1058e-05 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356968+0000", 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5339999999999999e-06 2023-01-15T14:29:07.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459524+0000", 2023-01-15T14:29:07.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102556147 2023-01-15T14:29:07.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459662+0000", 2023-01-15T14:29:07.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000137756 2023-01-15T14:29:07.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978621+0000", 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51895887600000001 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978870+0000", 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024928999999999999 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280761+0000", 2023-01-15T14:29:07.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30189113400000001 2023-01-15T14:29:07.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281023+0000", 2023-01-15T14:29:07.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026172800000000003 2023-01-15T14:29:07.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281142+0000", 2023-01-15T14:29:07.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000119296 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:91 3.d 3:bc1c6b44:::smithi09025264-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 40.86597613, 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.844937151, 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.905 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.906 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 91 2023-01-15T14:29:07.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.906 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:29:07.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403425+0000", 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403434+0000", 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3299999999999999e-06 2023-01-15T14:29:07.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403446+0000", 2023-01-15T14:29:07.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2176e-05 2023-01-15T14:29:07.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403453+0000", 2023-01-15T14:29:07.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5830000000000003e-06 2023-01-15T14:29:07.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403654+0000", 2023-01-15T14:29:07.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020136000000000001 2023-01-15T14:29:07.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403807+0000", 2023-01-15T14:29:07.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015342599999999999 2023-01-15T14:29:07.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907836+0000", 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50402854699999999 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907934+0000", 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8472000000000002e-05 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908096+0000", 2023-01-15T14:29:07.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000162139 2023-01-15T14:29:07.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908335+0000", 2023-01-15T14:29:07.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000238891 2023-01-15T14:29:07.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908350+0000", 2023-01-15T14:29:07.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4868000000000001e-05 2023-01-15T14:29:07.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908416+0000", 2023-01-15T14:29:07.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5502999999999999e-05 2023-01-15T14:29:07.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.952370+0000", 2023-01-15T14:29:07.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.043954663999999997 2023-01-15T14:29:07.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.008749+0000", 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.056378696999999998 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248164+0000", 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23941479500000001 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248244+0000", 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9616000000000005e-05 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248364+0000", 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120167 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.914 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:92 3.d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 40.865940436999999, 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.84526834699999998, 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 92 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403461+0000", 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403466+0000", 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.34e-06 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403476+0000", 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8530000000000004e-06 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403492+0000", 2023-01-15T14:29:07.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6646999999999998e-05 2023-01-15T14:29:07.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501333+0000", 2023-01-15T14:29:07.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097840420999999997 2023-01-15T14:29:07.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501483+0000", 2023-01-15T14:29:07.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150123 2023-01-15T14:29:07.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.007528+0000", 2023-01-15T14:29:07.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50604486299999996 2023-01-15T14:29:07.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.007649+0000", 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001217 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248441+0000", 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.240791267 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.921 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248561+0000", 2023-01-15T14:29:07.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120516 2023-01-15T14:29:07.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.921 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.921 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248731+0000", 2023-01-15T14:29:07.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016937000000000001 2023-01-15T14:29:07.921 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.921 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.921 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:107 3.d 3:b2a32959:::smithi09025264-27:head [setxattr _header (56) in=7b,truncate 2436982] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.508410+0000", 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 40.760992262999999, 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63911129799999999, 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 107 2023-01-15T14:29:07.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.923 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.923 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.923 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.923 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508410+0000", 2023-01-15T14:29:07.923 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.923 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.923 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.923 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.923 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508410+0000", 2023-01-15T14:29:07.923 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.923 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508409+0000", 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508418+0000", 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4830000000000005e-06 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.924 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.925 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508436+0000", 2023-01-15T14:29:07.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7419999999999999e-05 2023-01-15T14:29:07.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.925 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.925 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508448+0000", 2023-01-15T14:29:07.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2274e-05 2023-01-15T14:29:07.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508766+0000", 2023-01-15T14:29:07.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00031751199999999999 2023-01-15T14:29:07.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508906+0000", 2023-01-15T14:29:07.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014061999999999999 2023-01-15T14:29:07.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906456+0000", 2023-01-15T14:29:07.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39754930500000002 2023-01-15T14:29:07.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906561+0000", 2023-01-15T14:29:07.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000105159 2023-01-15T14:29:07.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:29:07.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906729+0000", 2023-01-15T14:29:07.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016813900000000001 2023-01-15T14:29:07.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906908+0000", 2023-01-15T14:29:07.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000179049 2023-01-15T14:29:07.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906974+0000", 2023-01-15T14:29:07.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6360999999999997e-05 2023-01-15T14:29:07.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907147+0000", 2023-01-15T14:29:07.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000172991 2023-01-15T14:29:07.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.952355+0000", 2023-01-15T14:29:07.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.045207642999999999 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.008697+0000", 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.056342148000000002 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.147257+0000", 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.13856006800000001 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.931 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.931 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.147356+0000", 2023-01-15T14:29:07.931 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9083e-05 2023-01-15T14:29:07.931 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.931 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.931 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.931 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.147522+0000", 2023-01-15T14:29:07.931 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016548999999999999 2023-01-15T14:29:07.931 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.931 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.931 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:108 3.d 3:b2a32959:::smithi09025264-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.508466+0000", 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 40.760936219999998, 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.73959223500000004, 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.932 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 108 2023-01-15T14:29:07.933 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.933 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.933 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.933 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.933 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508466+0000", 2023-01-15T14:29:07.933 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.933 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.933 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.933 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.933 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508466+0000", 2023-01-15T14:29:07.933 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508465+0000", 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508473+0000", 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.374e-06 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.934 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508502+0000", 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.9709e-05 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508513+0000", 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0200000000000001e-05 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.935 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508947+0000", 2023-01-15T14:29:07.936 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043390699999999998 2023-01-15T14:29:07.936 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.936 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.936 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.936 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.509045+0000", 2023-01-15T14:29:07.936 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8400000000000007e-05 2023-01-15T14:29:07.936 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.936 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.936 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.936 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907035+0000", 2023-01-15T14:29:07.936 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39798985799999997 2023-01-15T14:29:07.937 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.937 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.937 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.937 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907172+0000", 2023-01-15T14:29:07.937 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013714599999999999 2023-01-15T14:29:07.937 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.937 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.937 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.937 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.247729+0000", 2023-01-15T14:29:07.937 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.34055681599999998 2023-01-15T14:29:07.937 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.247863+0000", 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013370999999999999 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248059+0000", 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019614900000000001 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.938 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:191 3.d 3:b3f111b5:::smithi09025264-48:head [setxattr _header (56) in=7b,truncate 2789929] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.289388+0000", 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 39.980014199000003, 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67091844199999995, 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.939 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 191 2023-01-15T14:29:07.940 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.940 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.940 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.940 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.940 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289388+0000", 2023-01-15T14:29:07.940 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.940 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.940 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.940 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.940 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289388+0000", 2023-01-15T14:29:07.940 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289387+0000", 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289396+0000", 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6000000000000007e-06 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.941 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.942 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.942 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289410+0000", 2023-01-15T14:29:07.942 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3933e-05 2023-01-15T14:29:07.942 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.942 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.942 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.942 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.389520+0000", 2023-01-15T14:29:07.942 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100110488 2023-01-15T14:29:07.942 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.942 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.942 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.552712+0000", 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.16319123799999999 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.552851+0000", 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000139128 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.658429+0000", 2023-01-15T14:29:07.943 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10557828399999999 2023-01-15T14:29:07.944 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.944 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.944 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.944 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.658606+0000", 2023-01-15T14:29:07.944 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017746 2023-01-15T14:29:07.944 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.944 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.944 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:29:07.944 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.658814+0000", 2023-01-15T14:29:07.944 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020789199999999999 2023-01-15T14:29:07.944 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659030+0000", 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021556 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659054+0000", 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4182999999999999e-05 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.945 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659131+0000", 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.729e-05 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659442+0000", 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310575 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.817131+0000", 2023-01-15T14:29:07.946 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.15768874399999999 2023-01-15T14:29:07.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.947 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.947 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.947 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.960024+0000", 2023-01-15T14:29:07.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.14289289399999999 2023-01-15T14:29:07.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.947 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.947 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:07.947 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.960168+0000", 2023-01-15T14:29:07.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144186 2023-01-15T14:29:07.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.948 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.948 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:07.948 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.960307+0000", 2023-01-15T14:29:07.948 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013917000000000001 2023-01-15T14:29:07.948 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.948 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:07.948 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:07.948 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.948 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.948 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:199 3.f 3:f87c0183:::smithi09025264-50:head [setxattr _header (56) in=7b,truncate 2823543] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:07.948 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.491047+0000", 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 39.778355529999999, 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.619490506, 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 199 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.949 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:07.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491047+0000", 2023-01-15T14:29:07.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:07.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491047+0000", 2023-01-15T14:29:07.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:07.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:07.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491046+0000", 2023-01-15T14:29:07.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:07.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.951 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:07.951 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491052+0000", 2023-01-15T14:29:07.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0569999999999998e-06 2023-01-15T14:29:07.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.951 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:07.951 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491061+0000", 2023-01-15T14:29:07.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8400000000000001e-06 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491068+0000", 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3929999999999996e-06 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.692271+0000", 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20120313400000001 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.952 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.692455+0000", 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018380600000000001 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.808238+0000", 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.115783805 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:07.953 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.808361+0000", 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000122476 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.808731+0000", 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000370596 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.809089+0000", 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035795199999999998 2023-01-15T14:29:07.954 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.809135+0000", 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.5243e-05 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.809271+0000", 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000136494 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.955 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:07.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.909550+0000", 2023-01-15T14:29:07.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100278703 2023-01-15T14:29:07.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.956 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.910579+0000", 2023-01-15T14:29:07.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010287969999999999 2023-01-15T14:29:07.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.956 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:07.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110209+0000", 2023-01-15T14:29:07.957 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19963028599999999 2023-01-15T14:29:07.957 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:07.957 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:07.957 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:08.046 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110352+0000", 2023-01-15T14:29:08.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014284599999999999 2023-01-15T14:29:08.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:08.047 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110538+0000", 2023-01-15T14:29:08.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185842 2023-01-15T14:29:08.047 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:08.047 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:08.047 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:08.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:200 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.491107+0000", 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 39.778295464000003, 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81997309299999999, 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 200 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.048 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:08.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.049 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:08.049 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491107+0000", 2023-01-15T14:29:08.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:08.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.049 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:08.049 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491107+0000", 2023-01-15T14:29:08.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:08.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:08.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491106+0000", 2023-01-15T14:29:08.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:08.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.050 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:08.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491113+0000", 2023-01-15T14:29:08.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9129999999999997e-06 2023-01-15T14:29:08.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:08.051 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491132+0000", 2023-01-15T14:29:08.051 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8593999999999998e-05 2023-01-15T14:29:08.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:08.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491146+0000", 2023-01-15T14:29:08.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4036000000000001e-05 2023-01-15T14:29:08.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:08.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.692622+0000", 2023-01-15T14:29:08.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20147609 2023-01-15T14:29:08.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:08.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.692862+0000", 2023-01-15T14:29:08.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024064600000000001 2023-01-15T14:29:08.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:08.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.909332+0000", 2023-01-15T14:29:08.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.216469364 2023-01-15T14:29:08.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:08.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.909470+0000", 2023-01-15T14:29:08.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013843300000000001 2023-01-15T14:29:08.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:08.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.310780+0000", 2023-01-15T14:29:08.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.401309939 2023-01-15T14:29:08.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:08.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.310914+0000", 2023-01-15T14:29:08.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013376600000000001 2023-01-15T14:29:08.057 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.057 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:08.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.311080+0000", 2023-01-15T14:29:08.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000166405 2023-01-15T14:29:08.057 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:08.057 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:08.057 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:08.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.058 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.058 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:305 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 3008420] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:29:08.058 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.989114+0000", 2023-01-15T14:29:08.058 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 32.280288917999997, 2023-01-15T14:29:08.058 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61679676699999997, 2023-01-15T14:29:08.058 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:08.058 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:08.058 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:08.058 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:08.058 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:08.059 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 305 2023-01-15T14:29:08.059 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.059 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:08.059 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.059 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:08.059 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989114+0000", 2023-01-15T14:29:08.059 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:08.059 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.059 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.059 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:08.059 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989114+0000", 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989113+0000", 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989120+0000", 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3679999999999996e-06 2023-01-15T14:29:08.060 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.061 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.061 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:08.061 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989130+0000", 2023-01-15T14:29:08.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0169999999999999e-05 2023-01-15T14:29:08.061 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.061 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.061 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:08.061 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989137+0000", 2023-01-15T14:29:08.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0480000000000001e-06 2023-01-15T14:29:08.061 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989277+0000", 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014001700000000001 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989285+0000", 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5299999999999999e-06 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:08.063 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.402952+0000", 2023-01-15T14:29:08.063 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.41366716199999998 2023-01-15T14:29:08.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.063 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:08.063 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.403119+0000", 2023-01-15T14:29:08.063 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016668600000000001 2023-01-15T14:29:08.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.063 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:29:08.063 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.403502+0000", 2023-01-15T14:29:08.064 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038385699999999998 2023-01-15T14:29:08.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.064 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.064 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:08.064 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.403876+0000", 2023-01-15T14:29:08.064 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037308799999999999 2023-01-15T14:29:08.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:08.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.403918+0000", 2023-01-15T14:29:08.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2363000000000001e-05 2023-01-15T14:29:08.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:08.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.404030+0000", 2023-01-15T14:29:08.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111826 2023-01-15T14:29:08.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:08.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.504533+0000", 2023-01-15T14:29:08.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10050276800000001 2023-01-15T14:29:08.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.505244+0000", 2023-01-15T14:29:08.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071137100000000003 2023-01-15T14:29:08.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605718+0000", 2023-01-15T14:29:08.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100473689 2023-01-15T14:29:08.068 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.069 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.069 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:08.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605817+0000", 2023-01-15T14:29:08.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9666000000000004e-05 2023-01-15T14:29:08.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.069 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.069 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:08.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605911+0000", 2023-01-15T14:29:08.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3256000000000003e-05 2023-01-15T14:29:08.069 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:08.069 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:306 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.989160+0000", 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 32.280242532000003, 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61707688900000002, 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:08.070 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 306 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989160+0000", 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989160+0000", 2023-01-15T14:29:08.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:08.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:08.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989159+0000", 2023-01-15T14:29:08.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:08.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:08.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989164+0000", 2023-01-15T14:29:08.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3420000000000001e-06 2023-01-15T14:29:08.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.073 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.073 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:08.073 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989175+0000", 2023-01-15T14:29:08.073 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1702000000000001e-05 2023-01-15T14:29:08.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.073 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.073 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:08.073 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.989182+0000", 2023-01-15T14:29:08.073 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7939999999999997e-06 2023-01-15T14:29:08.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.074 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.074 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:08.074 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.088032+0000", 2023-01-15T14:29:08.074 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098849918999999994 2023-01-15T14:29:08.074 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.074 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.074 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:08.074 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.088042+0000", 2023-01-15T14:29:08.074 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.876e-06 2023-01-15T14:29:08.074 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.074 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605424+0000", 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51738228100000005 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605579+0000", 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015515900000000001 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:08.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.605966+0000", 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038621100000000002 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.606091+0000", 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000125499 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.606237+0000", 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000145816 2023-01-15T14:29:08.076 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:08.077 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:08.077 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:08.077 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:08.077 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:08.077 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:08.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:29:08.079 INFO:tasks.rados.rados.0.smithi090.stdout:230: done (1 left) 2023-01-15T14:29:08.080 INFO:tasks.rados.rados.0.smithi090.stdout:231: read oid 49 snap -1 2023-01-15T14:29:08.080 INFO:tasks.rados.rados.0.smithi090.stdout:231: expect (ObjNum 82 snap 10 seq_num 82) 2023-01-15T14:29:08.080 INFO:tasks.rados.rados.0.smithi090.stdout:232: write oid 28 current snap is 23 2023-01-15T14:29:08.080 INFO:tasks.rados.rados.0.smithi090.stdout:232: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2023-01-15T14:29:08.080 INFO:tasks.rados.rados.0.smithi090.stdout:232: writing smithi09025264-28 from 682384 to 1139214 tid 1 2023-01-15T14:29:08.080 INFO:tasks.rados.rados.0.smithi090.stdout:232: writing smithi09025264-28 from 1721042 to 2144741 tid 2 2023-01-15T14:29:08.080 INFO:tasks.rados.rados.0.smithi090.stdout:232: writing smithi09025264-28 from 2809728 to 3264664 tid 3 2023-01-15T14:29:08.080 INFO:tasks.rados.rados.0.smithi090.stdout:232: writing smithi09025264-28 from 3731357 to 3882384 tid 4 2023-01-15T14:29:08.080 INFO:tasks.rados.rados.0.smithi090.stdout:233: read oid 1 snap -1 2023-01-15T14:29:08.080 INFO:tasks.rados.rados.0.smithi090.stdout:233: expect (ObjNum 0 snap 0 seq_num 0) 2023-01-15T14:29:08.080 INFO:tasks.rados.rados.0.smithi090.stdout:234: snap_remove snap 21 2023-01-15T14:29:08.081 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:08.073+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:08.180 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:08.173+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:08.211 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:08.211 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:08.211 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:29:08.211 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:08.212 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:29:08.281 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:08.277+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:08.383 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:08.383 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:08.383 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:29:08.383 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:29:08.383 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:08.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:29:08.384 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:08.377+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:08.483 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:08.477+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:08.547 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:08.547 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:29:08.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:29:08.547 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:08.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:30 3.1 3:8b0a2647:::smithi09025264-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:29:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.634513679000001, 2023-01-15T14:29:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40319718700000001, 2023-01-15T14:29:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 30 2023-01-15T14:29:08.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.549 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:08.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:08.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:29:08.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:08.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:29:08.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:08.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859909+0000", 2023-01-15T14:29:08.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:29:08.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:08.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.860361+0000", 2023-01-15T14:29:08.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045164599999999999 2023-01-15T14:29:08.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:08.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.860392+0000", 2023-01-15T14:29:08.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1331000000000002e-05 2023-01-15T14:29:08.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:08.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960486+0000", 2023-01-15T14:29:08.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100093583 2023-01-15T14:29:08.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:08.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.160839+0000", 2023-01-15T14:29:08.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20035291199999999 2023-01-15T14:29:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.160982+0000", 2023-01-15T14:29:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014350800000000001 2023-01-15T14:29:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161183+0000", 2023-01-15T14:29:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000200411 2023-01-15T14:29:08.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:08.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161525+0000", 2023-01-15T14:29:08.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034216500000000001 2023-01-15T14:29:08.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:08.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161587+0000", 2023-01-15T14:29:08.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2032999999999994e-05 2023-01-15T14:29:08.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:08.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161842+0000", 2023-01-15T14:29:08.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000255305 2023-01-15T14:29:08.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:08.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262815+0000", 2023-01-15T14:29:08.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100972992 2023-01-15T14:29:08.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262914+0000", 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9166999999999994e-05 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262967+0000", 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2244000000000003e-05 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:08.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263050+0000", 2023-01-15T14:29:08.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2999999999999998e-05 2023-01-15T14:29:08.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:08.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263110+0000", 2023-01-15T14:29:08.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0297000000000002e-05 2023-01-15T14:29:08.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:08.560 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:08.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:08.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:31 3.1 3:8b0a2647:::smithi09025264-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.533911312999997, 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405055944, 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 31 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.561 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:08.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:08.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:29:08.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:08.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:29:08.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:08.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960513+0000", 2023-01-15T14:29:08.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:08.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:08.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960524+0000", 2023-01-15T14:29:08.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0614e-05 2023-01-15T14:29:08.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:08.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960538+0000", 2023-01-15T14:29:08.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4226999999999999e-05 2023-01-15T14:29:08.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:08.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060704+0000", 2023-01-15T14:29:08.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166252 2023-01-15T14:29:08.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:08.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.261723+0000", 2023-01-15T14:29:08.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20101896699999999 2023-01-15T14:29:08.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:08.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.261869+0000", 2023-01-15T14:29:08.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014578099999999999 2023-01-15T14:29:08.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:08.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262092+0000", 2023-01-15T14:29:08.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223198 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262397+0000", 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030490500000000001 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262446+0000", 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8492999999999998e-05 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262537+0000", 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1248000000000005e-05 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262847+0000", 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031007799999999998 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365234+0000", 2023-01-15T14:29:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102386856 2023-01-15T14:29:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365326+0000", 2023-01-15T14:29:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2226999999999996e-05 2023-01-15T14:29:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365413+0000", 2023-01-15T14:29:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6996999999999994e-05 2023-01-15T14:29:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365571+0000", 2023-01-15T14:29:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015794100000000001 2023-01-15T14:29:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:53 3.1 3:883e9a29:::smithi09025264-14:head [write 618491~694590,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:29:08.747 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.433603036999997, 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659587899999998, 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 53 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:08.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:29:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:29:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060822+0000", 2023-01-15T14:29:08.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061225+0000", 2023-01-15T14:29:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040297199999999998 2023-01-15T14:29:08.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061244+0000", 2023-01-15T14:29:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:29:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061253+0000", 2023-01-15T14:29:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8200000000000003e-06 2023-01-15T14:29:08.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263154+0000", 2023-01-15T14:29:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20190054900000001 2023-01-15T14:29:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263511+0000", 2023-01-15T14:29:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035690599999999999 2023-01-15T14:29:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263928+0000", 2023-01-15T14:29:08.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041701200000000003 2023-01-15T14:29:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264611+0000", 2023-01-15T14:29:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068358700000000004 2023-01-15T14:29:08.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264661+0000", 2023-01-15T14:29:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.952e-05 2023-01-15T14:29:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264962+0000", 2023-01-15T14:29:08.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030116799999999999 2023-01-15T14:29:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365102+0000", 2023-01-15T14:29:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100139698 2023-01-15T14:29:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366354+0000", 2023-01-15T14:29:08.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001252059 2023-01-15T14:29:08.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457209+0000", 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090855667000000001 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457307+0000", 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7621000000000006e-05 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457419+0000", 2023-01-15T14:29:08.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112246 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.433161583999997, 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:08.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:08.953 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:08.953 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:08.954 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:29:08.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.954 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:08.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:29:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:29:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:08.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:29:08.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:08.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:29:08.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:29:08.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:29:08.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:08.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:29:08.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:29:08.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:29:08.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:29:08.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:29:08.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:29:08.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:08.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:29:08.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:29:08.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.432881336000001, 2023-01-15T14:29:08.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:29:08.964 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:08.964 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:08.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:08.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:08.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:08.964 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:29:08.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.964 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:08.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:08.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:29:08.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:08.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:29:08.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:08.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:29:08.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:08.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:08.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:29:08.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:29:08.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:08.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:29:08.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:29:08.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:08.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:29:08.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:08.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:29:08.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:29:08.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:08.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:29:08.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:29:08.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:08.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:29:08.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:29:08.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:08.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:29:08.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:29:08.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:08.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:29:08.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:29:08.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:08.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:29:08.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:29:08.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:08.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:29:08.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:29:08.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:29:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:29:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:29:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:29:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:29:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.432848718999999, 2023-01-15T14:29:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:29:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:29:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:29:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:29:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:29:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:29:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:29:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:29:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:29:09.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:29:09.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:29:09.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:29:09.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:29:09.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:29:09.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:29:09.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:29:09.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:29:09.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:29:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:29:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:29:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:29:09.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:29:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:29:09.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:58 3.1 3:86bba67f:::smithi09025264-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:09.167 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:29:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.331678154000002, 2023-01-15T14:29:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39908735699999998, 2023-01-15T14:29:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:09.168 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 58 2023-01-15T14:29:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:09.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:29:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:29:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162747+0000", 2023-01-15T14:29:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162859+0000", 2023-01-15T14:29:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112137 2023-01-15T14:29:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162874+0000", 2023-01-15T14:29:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4793999999999999e-05 2023-01-15T14:29:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162883+0000", 2023-01-15T14:29:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5559999999999996e-06 2023-01-15T14:29:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390699+0000", 2023-01-15T14:29:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.227815712 2023-01-15T14:29:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390900+0000", 2023-01-15T14:29:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000201518 2023-01-15T14:29:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458887+0000", 2023-01-15T14:29:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067986183000000006 2023-01-15T14:29:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459000+0000", 2023-01-15T14:29:09.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011291800000000001 2023-01-15T14:29:09.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459244+0000", 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024485000000000002 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459480+0000", 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023518599999999999 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:09.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459498+0000", 2023-01-15T14:29:09.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8892000000000001e-05 2023-01-15T14:29:09.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459822+0000", 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032378700000000002 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460258+0000", 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043563500000000003 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465816+0000", 2023-01-15T14:29:09.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055581099999999998 2023-01-15T14:29:09.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561720+0000", 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095903587999999998 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561771+0000", 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1433e-05 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561836+0000", 2023-01-15T14:29:09.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4591000000000006e-05 2023-01-15T14:29:09.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.380 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.380 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:09.380 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:29:09.380 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.331530733000001, 2023-01-15T14:29:09.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:29:09.380 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:09.381 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:09.381 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:09.381 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:09.381 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:09.381 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:29:09.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.381 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:09.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:09.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:29:09.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:09.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:29:09.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:09.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:29:09.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:09.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:09.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:29:09.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:29:09.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:29:09.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:29:09.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:29:09.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:09.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:29:09.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:29:09.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:09.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:29:09.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:29:09.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:09.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:29:09.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:29:09.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:29:09.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:29:09.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:29:09.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:29:09.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:09.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:29:09.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:29:09.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:29:09.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.331500730000002, 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:09.391 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:29:09.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.391 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:09.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:09.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:29:09.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:09.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:29:09.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:09.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:29:09.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:09.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:09.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:29:09.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:29:09.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:09.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:29:09.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:29:09.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:09.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:29:09.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:29:09.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:29:09.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:29:09.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:29:09.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:29:09.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:29:09.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:29:09.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:29:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:29:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:29:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:29:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:29:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:29:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:29:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:29:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:125 3.1 3:8889e450:::smithi09025264-32:head [write 521017~798131,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:29:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.821611912000002, 2023-01-15T14:29:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40348810600000001, 2023-01-15T14:29:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:09.580 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:09.580 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:09.580 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 125 2023-01-15T14:29:09.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.580 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:09.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:09.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:29:09.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:09.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:29:09.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:09.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672812+0000", 2023-01-15T14:29:09.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:09.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:09.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673338+0000", 2023-01-15T14:29:09.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052553599999999995 2023-01-15T14:29:09.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:09.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673372+0000", 2023-01-15T14:29:09.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3652999999999998e-05 2023-01-15T14:29:09.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:09.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673386+0000", 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4171e-05 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773535+0000", 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148822 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773727+0000", 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192189 2023-01-15T14:29:09.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:09.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773918+0000", 2023-01-15T14:29:09.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019143099999999999 2023-01-15T14:29:09.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:09.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774291+0000", 2023-01-15T14:29:09.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037288399999999998 2023-01-15T14:29:09.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774341+0000", 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0117999999999997e-05 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774696+0000", 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000355268 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:09.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874892+0000", 2023-01-15T14:29:09.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100195981 2023-01-15T14:29:09.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.977868+0000", 2023-01-15T14:29:09.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102975649 2023-01-15T14:29:09.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.075846+0000", 2023-01-15T14:29:09.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097977838999999997 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.076004+0000", 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015859700000000001 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.076303+0000", 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029816400000000001 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.587 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.588 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:09.588 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:29:09.588 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.816492165, 2023-01-15T14:29:09.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:29:09.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:09.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:09.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:29:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:29:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:09.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:29:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:29:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:29:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:29:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:29:09.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:29:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:29:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:29:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:29:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:29:09.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:29:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:29:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:29:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:29:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:09.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:29:09.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:29:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:29:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:29:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:29:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:29:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.816009811000001, 2023-01-15T14:29:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:29:09.598 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:09.779 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:09.779 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:09.779 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:09.779 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:09.780 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:29:09.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.780 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:09.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:09.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:29:09.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:29:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:29:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:29:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:29:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:29:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:29:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:29:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:29:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:29:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:29:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:29:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:29:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:29:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:29:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:09.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:29:09.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:29:09.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:09.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:29:09.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:29:09.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:09.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:29:09.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:29:09.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:09.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:29:09.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:29:09.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:09.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:29:09.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:29:09.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:29:09.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:29:09.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:29:09.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:29:09.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:09.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:29:09.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:29:09.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:29:09.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:29:09.792 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.793 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.793 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:09.793 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:29:09.793 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.815957247999997, 2023-01-15T14:29:09.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:29:09.793 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:09.793 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:09.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:09.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:09.794 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:09.794 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:29:09.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.794 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:09.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:09.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:29:09.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:09.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:29:09.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:09.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:29:09.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:09.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:09.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:29:09.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:29:09.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:09.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:29:09.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:29:09.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:09.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:29:09.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:29:09.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:29:09.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:29:09.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:29:09.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:29:09.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:29:09.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:29:09.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:29:09.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:29:09.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:29:09.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:29:09.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:29:09.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:29:09.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:29:09.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:29:09.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.801 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.801 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:130 3.6 3:68fb3263:::smithi09025264-33:head [write 1806012~625998] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:09.801 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.784600+0000", 2023-01-15T14:29:09.801 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.709825948999999, 2023-01-15T14:29:09.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30627860400000001, 2023-01-15T14:29:09.802 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:09.802 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:09.802 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:09.802 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:09.802 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:09.802 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 130 2023-01-15T14:29:09.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.802 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:09.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:09.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.784600+0000", 2023-01-15T14:29:09.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:09.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.784600+0000", 2023-01-15T14:29:09.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:09.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.784598+0000", 2023-01-15T14:29:09.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:09.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:09.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.784936+0000", 2023-01-15T14:29:09.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033822000000000002 2023-01-15T14:29:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.784969+0000", 2023-01-15T14:29:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3083999999999997e-05 2023-01-15T14:29:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:09.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.885103+0000", 2023-01-15T14:29:09.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100133947 2023-01-15T14:29:09.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.885212+0000", 2023-01-15T14:29:09.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108324 2023-01-15T14:29:09.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.885359+0000", 2023-01-15T14:29:09.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014727100000000001 2023-01-15T14:29:09.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.987685+0000", 2023-01-15T14:29:09.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102326639 2023-01-15T14:29:09.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.987826+0000", 2023-01-15T14:29:09.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014092999999999999 2023-01-15T14:29:09.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:09.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.988045+0000", 2023-01-15T14:29:09.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021909499999999999 2023-01-15T14:29:09.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:09.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.988546+0000", 2023-01-15T14:29:09.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050102199999999995 2023-01-15T14:29:09.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:09.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.988570+0000", 2023-01-15T14:29:09.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3190999999999999e-05 2023-01-15T14:29:09.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:09.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.988875+0000", 2023-01-15T14:29:09.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030542399999999999 2023-01-15T14:29:09.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:09.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.090269+0000", 2023-01-15T14:29:09.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101393448 2023-01-15T14:29:09.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.090411+0000", 2023-01-15T14:29:09.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014229600000000001 2023-01-15T14:29:09.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.090674+0000", 2023-01-15T14:29:09.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026322299999999998 2023-01-15T14:29:09.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:09.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.090785+0000", 2023-01-15T14:29:09.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011103300000000001 2023-01-15T14:29:09.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.090879+0000", 2023-01-15T14:29:09.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4005999999999994e-05 2023-01-15T14:29:09.812 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.813 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.813 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:09.813 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:29:09.813 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.444755544, 2023-01-15T14:29:09.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:29:09.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:09.814 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:09.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:09.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:09.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:09.814 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:29:09.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.814 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:09.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:09.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:29:09.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:09.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:29:09.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:09.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:29:09.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:09.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:09.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:29:09.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:29:09.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:09.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:29:09.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:29:09.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:09.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:29:09.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:29:09.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:29:09.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:29:09.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:29:09.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:29:09.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:09.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:29:09.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:29:09.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:09.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:29:09.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:29:09.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:09.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:29:09.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:29:09.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:09.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:29:09.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:29:09.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:09.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:29:09.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:29:09.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:29:09.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:29:09.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:29:09.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:29:09.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:29:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:29:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:29:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:29:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:29:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.444638869999999, 2023-01-15T14:29:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:29:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:09.969 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:09.969 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:09.973 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:29:09.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.974 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:09.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:09.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:29:09.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:09.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:29:09.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:29:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:29:09.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:29:09.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:09.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:29:09.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:29:09.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:09.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:29:09.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:29:09.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:29:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:29:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:29:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:29:09.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:29:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:29:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:29:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:29:09.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:29:09.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:29:09.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:29:09.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:29:09.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:29:09.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:29:09.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:29:09.982 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.982 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.983 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:448 3.1 3:8b0a2647:::smithi09025264-8:head [write 1856548~737690] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:29:09.983 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:29:09.983 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.197380962, 2023-01-15T14:29:09.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40560468799999999, 2023-01-15T14:29:09.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:09.984 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:09.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:09.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:09.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:09.984 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 448 2023-01-15T14:29:09.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.985 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:09.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:09.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:29:09.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:09.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:29:09.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:09.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:09.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297043+0000", 2023-01-15T14:29:09.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:09.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297477+0000", 2023-01-15T14:29:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043336500000000001 2023-01-15T14:29:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297505+0000", 2023-01-15T14:29:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8583000000000001e-05 2023-01-15T14:29:09.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397598+0000", 2023-01-15T14:29:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092164 2023-01-15T14:29:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397683+0000", 2023-01-15T14:29:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5986e-05 2023-01-15T14:29:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397798+0000", 2023-01-15T14:29:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114916 2023-01-15T14:29:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:09.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498684+0000", 2023-01-15T14:29:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10088510000000001 2023-01-15T14:29:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498848+0000", 2023-01-15T14:29:09.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164575 2023-01-15T14:29:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499083+0000", 2023-01-15T14:29:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000235171 2023-01-15T14:29:09.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499670+0000", 2023-01-15T14:29:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058650200000000003 2023-01-15T14:29:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499688+0000", 2023-01-15T14:29:09.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8547e-05 2023-01-15T14:29:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.500037+0000", 2023-01-15T14:29:09.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034891399999999997 2023-01-15T14:29:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.501925+0000", 2023-01-15T14:29:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00188766 2023-01-15T14:29:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.601919+0000", 2023-01-15T14:29:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099994371999999998 2023-01-15T14:29:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702377+0000", 2023-01-15T14:29:09.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045810500000001 2023-01-15T14:29:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702537+0000", 2023-01-15T14:29:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015941600000000001 2023-01-15T14:29:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702650+0000", 2023-01-15T14:29:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011338899999999999 2023-01-15T14:29:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:451 3.1 3:8b0a2647:::smithi09025264-8:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:29:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.397793+0000", 2023-01-15T14:29:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.096633021999999, 2023-01-15T14:29:09.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30620719800000001, 2023-01-15T14:29:10.000 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:10.000 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:10.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:10.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:10.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:10.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 451 2023-01-15T14:29:10.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.000 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:10.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:10.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397793+0000", 2023-01-15T14:29:10.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:10.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:10.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397793+0000", 2023-01-15T14:29:10.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:10.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:10.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397793+0000", 2023-01-15T14:29:10.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:10.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:10.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397797+0000", 2023-01-15T14:29:10.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.143e-06 2023-01-15T14:29:10.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:10.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397808+0000", 2023-01-15T14:29:10.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0893999999999999e-05 2023-01-15T14:29:10.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:10.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397818+0000", 2023-01-15T14:29:10.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0506000000000001e-05 2023-01-15T14:29:10.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:10.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.493632+0000", 2023-01-15T14:29:10.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095813551999999996 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.493911+0000", 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027954800000000001 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.502017+0000", 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0081052439999999993 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.502157+0000", 2023-01-15T14:29:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000140696 2023-01-15T14:29:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.703469+0000", 2023-01-15T14:29:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201311512 2023-01-15T14:29:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.703724+0000", 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025488099999999999 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.704001+0000", 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027700199999999999 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:468 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (57) in=7b,truncate 3378174] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775893+0000", 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 20.718533875999999, 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39428173999999999, 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.007 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775893+0000", 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775893+0000", 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775892+0000", 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775897+0000", 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4820000000000002e-06 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775906+0000", 2023-01-15T14:29:10.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9199999999999993e-06 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775911+0000", 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5179999999999998e-06 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.967706+0000", 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.191795189 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:10.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.967988+0000", 2023-01-15T14:29:10.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028248100000000001 2023-01-15T14:29:10.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:10.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968459+0000", 2023-01-15T14:29:10.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000470917 2023-01-15T14:29:10.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:10.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968829+0000", 2023-01-15T14:29:10.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036951399999999999 2023-01-15T14:29:10.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:10.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968842+0000", 2023-01-15T14:29:10.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2801e-05 2023-01-15T14:29:10.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:10.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968890+0000", 2023-01-15T14:29:10.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8775e-05 2023-01-15T14:29:10.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:10.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969229+0000", 2023-01-15T14:29:10.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033879800000000002 2023-01-15T14:29:10.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:10.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.169674+0000", 2023-01-15T14:29:10.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20044432000000001 2023-01-15T14:29:10.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:10.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.169983+0000", 2023-01-15T14:29:10.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030946800000000003 2023-01-15T14:29:10.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:10.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.170076+0000", 2023-01-15T14:29:10.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2872000000000001e-05 2023-01-15T14:29:10.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:10.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.170174+0000", 2023-01-15T14:29:10.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:29:10.150 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.151 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.151 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.151 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:29:10.151 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:29:10.151 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 20.718506594000001, 2023-01-15T14:29:10.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:29:10.151 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:10.151 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:10.152 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:10.152 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:10.152 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:10.152 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:29:10.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.152 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:10.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:10.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:29:10.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:10.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:10.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:29:10.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:29:10.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:10.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:29:10.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:29:10.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:10.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:29:10.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:29:10.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:10.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:29:10.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:29:10.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:10.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:29:10.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:29:10.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:10.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:29:10.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:29:10.158 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.158 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.158 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.158 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.158 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.158 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:10.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:29:10.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:29:10.160 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:29:10.160 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:29:10.160 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-01-15T14:29:10.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-01-15T14:29:10.160 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:29:10.165 INFO:tasks.rados.rados.0.smithi090.stdout:234: done (4 left) 2023-01-15T14:29:10.165 INFO:tasks.rados.rados.0.smithi090.stdout:235: write (excl) oid 7 current snap is 23 2023-01-15T14:29:10.166 INFO:tasks.rados.rados.0.smithi090.stdout:235: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2023-01-15T14:29:10.166 INFO:tasks.rados.rados.0.smithi090.stdout:235: writing smithi09025264-7 from 699191 to 1231270 tid 1 2023-01-15T14:29:10.166 INFO:tasks.rados.rados.0.smithi090.stdout:235: writing smithi09025264-7 from 1779524 to 2346881 tid 2 2023-01-15T14:29:10.166 INFO:tasks.rados.rados.0.smithi090.stdout:235: writing smithi09025264-7 from 2837151 to 3503359 tid 3 2023-01-15T14:29:10.166 INFO:tasks.rados.rados.0.smithi090.stdout:236: write (excl) oid 29 current snap is 23 2023-01-15T14:29:10.166 INFO:tasks.rados.rados.0.smithi090.stdout:229: finishing rollback tid 0 to smithi09025264-24 2023-01-15T14:29:10.167 INFO:tasks.rados.rados.0.smithi090.stdout:236: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2023-01-15T14:29:10.167 INFO:tasks.rados.rados.0.smithi090.stdout:236: writing smithi09025264-29 from 715998 to 1323326 tid 1 2023-01-15T14:29:10.167 INFO:tasks.rados.rados.0.smithi090.stdout:236: writing smithi09025264-29 from 1838006 to 2265374 tid 2 2023-01-15T14:29:10.167 INFO:tasks.rados.rados.0.smithi090.stdout:236: writing smithi09025264-29 from 2864574 to 3342054 tid 3 2023-01-15T14:29:10.167 INFO:tasks.rados.rados.0.smithi090.stdout:236: writing smithi09025264-29 from 3863835 to 3915998 tid 4 2023-01-15T14:29:10.167 INFO:tasks.rados.rados.0.smithi090.stdout:237: read oid 40 snap -1 2023-01-15T14:29:10.168 INFO:tasks.rados.rados.0.smithi090.stdout:237: expect (ObjNum 39 snap 0 seq_num 39) 2023-01-15T14:29:10.168 INFO:tasks.rados.rados.0.smithi090.stdout:238: read oid 42 snap -1 2023-01-15T14:29:10.168 INFO:tasks.rados.rados.0.smithi090.stdout:238: expect (ObjNum 91 snap 15 seq_num 91) 2023-01-15T14:29:10.168 INFO:tasks.rados.rados.0.smithi090.stdout:239: snap_remove snap 11 2023-01-15T14:29:10.168 INFO:tasks.rados.rados.0.smithi090.stdout:239: done (8 left) 2023-01-15T14:29:10.169 INFO:tasks.rados.rados.0.smithi090.stdout:240: delete oid 41 current snap is 23 2023-01-15T14:29:10.169 INFO:tasks.rados.rados.0.smithi090.stdout:229: finishing rollback tid 1 to smithi09025264-24 2023-01-15T14:29:10.169 INFO:tasks.rados.rados.0.smithi090.stdout:229: finishing rollback tid 2 to smithi09025264-24 2023-01-15T14:29:10.169 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 90 (ObjNum 34 snap 0 seq_num 34) dirty exists 2023-01-15T14:29:10.169 INFO:tasks.rados.rados.0.smithi090.stdout:232: finishing write tid 1 to smithi09025264-28 2023-01-15T14:29:10.169 INFO:tasks.rados.rados.0.smithi090.stdout:231: expect (ObjNum 82 snap 10 seq_num 82) 2023-01-15T14:29:10.170 INFO:tasks.rados.rados.0.smithi090.stdout:232: finishing write tid 2 to smithi09025264-28 2023-01-15T14:29:10.170 INFO:tasks.rados.rados.0.smithi090.stdout:232: finishing write tid 3 to smithi09025264-28 2023-01-15T14:29:10.170 INFO:tasks.rados.rados.0.smithi090.stdout:232: finishing write tid 4 to smithi09025264-28 2023-01-15T14:29:10.170 INFO:tasks.rados.rados.0.smithi090.stdout:232: finishing write tid 5 to smithi09025264-28 2023-01-15T14:29:10.171 INFO:tasks.rados.rados.0.smithi090.stdout:232: finishing write tid 6 to smithi09025264-28 2023-01-15T14:29:10.171 INFO:tasks.rados.rados.0.smithi090.stdout:232: oid 28 updating version 0 to 87 2023-01-15T14:29:10.171 INFO:tasks.rados.rados.0.smithi090.stdout:232: oid 28 version 87 is already newer than 85 2023-01-15T14:29:10.171 INFO:tasks.rados.rados.0.smithi090.stdout:232: oid 28 updating version 87 to 88 2023-01-15T14:29:10.171 INFO:tasks.rados.rados.0.smithi090.stdout:232: oid 28 version 88 is already newer than 86 2023-01-15T14:29:10.171 INFO:tasks.rados.rados.0.smithi090.stdout:232: oid 28 updating version 88 to 89 2023-01-15T14:29:10.172 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 89 (ObjNum 112 snap 23 seq_num 112) dirty exists 2023-01-15T14:29:10.172 INFO:tasks.rados.rados.0.smithi090.stdout:232: left oid 28 (ObjNum 112 snap 23 seq_num 112) 2023-01-15T14:29:10.172 INFO:tasks.rados.rados.0.smithi090.stdout:233: expect (ObjNum 0 snap 0 seq_num 0) 2023-01-15T14:29:10.172 INFO:tasks.rados.rados.0.smithi090.stdout:235: finishing write tid 1 to smithi09025264-7 2023-01-15T14:29:10.172 INFO:tasks.rados.rados.0.smithi090.stdout:235: finishing write tid 2 to smithi09025264-7 2023-01-15T14:29:10.172 INFO:tasks.rados.rados.0.smithi090.stdout:238: expect (ObjNum 91 snap 15 seq_num 91) 2023-01-15T14:29:10.173 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:10.153+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:10.262 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:10.257+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:10.328 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.1.0:100 3.0 e63/55)", 2023-01-15T14:29:10.328 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:10.185307+0000", 2023-01-15T14:29:10.329 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.086839886000000005, 2023-01-15T14:29:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.086863599999999999, 2023-01-15T14:29:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:29:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:10.185307+0000", 2023-01-15T14:29:10.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:10.185307+0000", 2023-01-15T14:29:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:10.185306+0000", 2023-01-15T14:29:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:10.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:10.185312+0000", 2023-01-15T14:29:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3119999999999999e-06 2023-01-15T14:29:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:10.185322+0000", 2023-01-15T14:29:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0145e-05 2023-01-15T14:29:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:10.185329+0000", 2023-01-15T14:29:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8329999999999996e-06 2023-01-15T14:29:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:29:10.333 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:29:10.333 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:10.334 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:29:10.334 INFO:tasks.rados.rados.0.smithi090.stdout:236: finishing write tid 1 to smithi09025264-29 2023-01-15T14:29:10.335 INFO:tasks.rados.rados.0.smithi090.stdout:237: expect (ObjNum 39 snap 0 seq_num 39) 2023-01-15T14:29:10.377 INFO:tasks.rados.rados.0.smithi090.stdout:236: finishing write tid 2 to smithi09025264-29 2023-01-15T14:29:10.491 INFO:tasks.rados.rados.0.smithi090.stdout:235: finishing write tid 3 to smithi09025264-7 2023-01-15T14:29:10.491 INFO:tasks.rados.rados.0.smithi090.stdout:235: finishing write tid 4 to smithi09025264-7 2023-01-15T14:29:10.491 INFO:tasks.rados.rados.0.smithi090.stdout:235: finishing write tid 5 to smithi09025264-7 2023-01-15T14:29:10.491 INFO:tasks.rados.rados.0.smithi090.stdout:235: oid 7 updating version 0 to 47 2023-01-15T14:29:10.491 INFO:tasks.rados.rados.0.smithi090.stdout:235: oid 7 version 47 is already newer than 45 2023-01-15T14:29:10.491 INFO:tasks.rados.rados.0.smithi090.stdout:235: oid 7 updating version 47 to 48 2023-01-15T14:29:10.491 INFO:tasks.rados.rados.0.smithi090.stdout:235: oid 7 version 48 is already newer than 46 2023-01-15T14:29:10.491 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 48 (ObjNum 113 snap 23 seq_num 113) dirty exists 2023-01-15T14:29:10.491 INFO:tasks.rados.rados.0.smithi090.stdout:235: left oid 7 (ObjNum 113 snap 23 seq_num 113) 2023-01-15T14:29:10.491 INFO:tasks.rados.rados.0.smithi090.stdout:236: finishing write tid 3 to smithi09025264-29 2023-01-15T14:29:10.492 INFO:tasks.rados.rados.0.smithi090.stdout:236: finishing write tid 4 to smithi09025264-29 2023-01-15T14:29:10.492 INFO:tasks.rados.rados.0.smithi090.stdout:229: done (8 left) 2023-01-15T14:29:10.492 INFO:tasks.rados.rados.0.smithi090.stdout:231: done (7 left) 2023-01-15T14:29:10.492 INFO:tasks.rados.rados.0.smithi090.stdout:232: done (6 left) 2023-01-15T14:29:10.492 INFO:tasks.rados.rados.0.smithi090.stdout:233: done (5 left) 2023-01-15T14:29:10.492 INFO:tasks.rados.rados.0.smithi090.stdout:235: done (4 left) 2023-01-15T14:29:10.492 INFO:tasks.rados.rados.0.smithi090.stdout:237: done (3 left) 2023-01-15T14:29:10.492 INFO:tasks.rados.rados.0.smithi090.stdout:238: done (2 left) 2023-01-15T14:29:10.492 INFO:tasks.rados.rados.0.smithi090.stdout:240: done (1 left) 2023-01-15T14:29:10.492 INFO:tasks.rados.rados.0.smithi090.stdout:241: snap_remove snap 15 2023-01-15T14:29:10.493 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:10.369+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:10.493 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:10.494 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:10.494 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:10.494 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:10.495 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:10.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:29:10.496 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:10.496 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:10.497 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:10.497 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:10.497 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 64, 2023-01-15T14:29:10.497 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "1c77fb2b-d1a7-4884-a24e-3935e16d0b6b", 2023-01-15T14:29:10.497 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2023-01-15T14:27:57.735717+0000", 2023-01-15T14:29:10.497 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2023-01-15T14:29:09.877713+0000", 2023-01-15T14:29:10.497 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2023-01-15T14:28:04.837522+0000", 2023-01-15T14:29:10.497 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2023-01-15T14:28:53.706589+0000", 2023-01-15T14:29:10.497 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-01-15T14:29:10.497 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5804032, 2023-01-15T14:29:10.497 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2023-01-15T14:29:10.498 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2023-01-15T14:29:10.498 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2023-01-15T14:29:10.498 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2023-01-15T14:29:10.498 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2023-01-15T14:29:10.498 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2023-01-15T14:29:10.498 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:29:10.498 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2023-01-15T14:29:10.498 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2023-01-15T14:29:10.498 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-01-15T14:29:10.498 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2023-01-15T14:29:10.498 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2023-01-15T14:29:10.499 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 3, 2023-01-15T14:29:10.499 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2023-01-15T14:29:10.499 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2023-01-15T14:29:10.499 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2023-01-15T14:29:10.499 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "reef", 2023-01-15T14:29:10.499 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2023-01-15T14:29:10.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.499 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2023-01-15T14:29:10.499 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2023-01-15T14:29:10.499 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:04.899282+0000", 2023-01-15T14:29:10.499 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:29:10.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:29:10.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:29:10.500 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:29:10.500 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:29:10.500 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:29:10.500 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:29:10.500 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:29:10.500 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:29:10.500 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:29:10.500 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:29:10.500 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:29:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2023-01-15T14:29:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2023-01-15T14:29:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2023-01-15T14:29:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2023-01-15T14:29:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2023-01-15T14:29:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:29:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:29:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:29:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:29:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:29:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "16", 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:29:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:29:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:29:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:29:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:29:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:29:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:29:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:29:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:29:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:29:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:29:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:29:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:29:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:29:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:29:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:29:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:29:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:29:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:29:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:29:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:29:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:29:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:29:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:29:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:29:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:29:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:29:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:29:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:29:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:29:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2023-01-15T14:29:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2023-01-15T14:29:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2023-01-15T14:29:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:29:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2023-01-15T14:29:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2023-01-15T14:29:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2023-01-15T14:29:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:11.115002+0000", 2023-01-15T14:29:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:29:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:29:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:29:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:29:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:29:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:29:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:29:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:29:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:29:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:29:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:29:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:29:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2023-01-15T14:29:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2023-01-15T14:29:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2023-01-15T14:29:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2023-01-15T14:29:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2023-01-15T14:29:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:29:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:29:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:29:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:29:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:29:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:29:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "19", 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:29:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:29:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:29:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:29:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:29:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:29:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:29:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:29:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:29:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:29:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:29:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:29:10.512 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:29:10.512 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:29:10.512 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:29:10.512 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:29:10.512 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:29:10.512 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:29:10.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.512 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:29:10.512 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:29:10.512 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:29:10.513 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:29:10.513 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:29:10.513 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:29:10.513 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:29:10.513 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "unique_pool_0", 2023-01-15T14:29:10.514 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:22.434903+0000", 2023-01-15T14:29:10.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 16385, 2023-01-15T14:29:10.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,pool_snaps", 2023-01-15T14:29:10.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:29:10.515 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:29:10.515 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:29:10.515 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:29:10.515 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:29:10.515 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:29:10.515 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:29:10.515 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:29:10.515 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:29:10.516 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:29:10.516 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 16, 2023-01-15T14:29:10.516 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 16, 2023-01-15T14:29:10.516 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 16, 2023-01-15T14:29:10.516 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 16, 2023-01-15T14:29:10.516 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 16, 2023-01-15T14:29:10.516 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:29:10.516 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:29:10.516 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:29:10.516 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:29:10.516 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:29:10.517 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:29:10.517 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:29:10.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.517 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "64", 2023-01-15T14:29:10.517 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:29:10.517 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:29:10.517 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:29:10.517 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:29:10.517 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "pool", 2023-01-15T14:29:10.517 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 41, 2023-01-15T14:29:10.517 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 64, 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [ 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 16, 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:28:44.706724+0000", 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap9" 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 25, 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:28:52.993772+0000", 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap15" 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.519 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 27, 2023-01-15T14:29:10.519 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:28:55.550715+0000", 2023-01-15T14:29:10.519 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap16" 2023-01-15T14:29:10.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.519 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 29, 2023-01-15T14:29:10.519 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:28:56.974245+0000", 2023-01-15T14:29:10.519 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap18" 2023-01-15T14:29:10.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.519 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 36, 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:29:04.453734+0000", 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap23" 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:29:10.520 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.521 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:29:10.522 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:29:10.522 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:29:10.652 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:29:10.652 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:29:10.652 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:29:10.652 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:29:10.652 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:29:10.652 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:29:10.653 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:29:10.653 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:29:10.653 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:29:10.653 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:29:10.653 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2023-01-15T14:29:10.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.653 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:29:10.653 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:29:10.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.653 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e55c5ac4-3cff-4ee5-8719-373648f83d8d", 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:29:10.654 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:29:10.655 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.655 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2023-01-15T14:29:10.655 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:29:10.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.656 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2023-01-15T14:29:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:29:10.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:29:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2023-01-15T14:29:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:29:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.667 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.667 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2023-01-15T14:29:10.667 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:29:10.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.667 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.667 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:29:10.668 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.668 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.668 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2023-01-15T14:29:10.668 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:29:10.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.668 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.668 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2023-01-15T14:29:10.668 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:29:10.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.669 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.669 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:29:10.669 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.669 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.669 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2023-01-15T14:29:10.669 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:29:10.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.669 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6811/23473", 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6815/23473", 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/23473", 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6820/23473", 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:29:10.670 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:29:10.671 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.671 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:29:10.671 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "2c1b33c9-0a1e-4734-936d-d51d64fe997a", 2023-01-15T14:29:10.671 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:29:10.671 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:29:10.671 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:29:10.671 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:29:10.671 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:29:10.671 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 55, 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:29:10.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.674 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2023-01-15T14:29:10.675 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:29:10.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.675 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.675 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2023-01-15T14:29:10.675 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:29:10.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.675 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.676 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:29:10.676 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.676 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2023-01-15T14:29:10.676 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:29:10.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.677 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2023-01-15T14:29:10.677 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:29:10.677 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.677 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.677 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/23474", 2023-01-15T14:29:10.677 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/23474", 2023-01-15T14:29:10.677 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/23474", 2023-01-15T14:29:10.677 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/23474", 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "b314174a-eca9-4baa-a60c-2589f1fc9a8e", 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:29:10.678 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:29:10.679 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:29:10.679 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:29:10.679 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:29:10.679 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 48, 2023-01-15T14:29:10.679 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:29:10.679 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:29:10.679 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:29:10.679 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.679 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.679 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.681 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.681 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2023-01-15T14:29:10.681 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:29:10.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.681 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.681 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2023-01-15T14:29:10.681 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:29:10.681 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.681 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.823 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:29:10.823 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:29:10.824 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.825 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.825 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2023-01-15T14:29:10.825 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:29:10.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.827 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2023-01-15T14:29:10.827 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:29:10.827 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.827 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.827 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6810/23475", 2023-01-15T14:29:10.827 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/23475", 2023-01-15T14:29:10.827 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6821/23475", 2023-01-15T14:29:10.828 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6817/23475", 2023-01-15T14:29:10.828 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:29:10.828 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:29:10.828 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:29:10.828 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.828 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:29:10.828 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "29eb0b17-ed5d-4665-a347-2fb8406d9c62", 2023-01-15T14:29:10.828 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:29:10.828 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:29:10.829 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:29:10.829 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:29:10.829 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:29:10.829 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:29:10.829 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:29:10.829 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 48, 2023-01-15T14:29:10.829 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:29:10.829 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:29:10.829 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:29:10.829 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.830 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.830 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6824", 2023-01-15T14:29:10.830 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:29:10.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.830 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.830 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6825", 2023-01-15T14:29:10.830 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:29:10.830 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.830 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.831 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:29:10.831 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.831 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.831 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6826", 2023-01-15T14:29:10.831 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:29:10.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.831 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.831 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6827", 2023-01-15T14:29:10.833 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:29:10.833 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.833 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.834 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:29:10.834 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.834 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.834 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6830", 2023-01-15T14:29:10.834 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:29:10.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.834 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.834 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6831", 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6828", 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.836 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6829", 2023-01-15T14:29:10.836 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:29:10.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.836 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.836 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6825/23778", 2023-01-15T14:29:10.836 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6827/23778", 2023-01-15T14:29:10.836 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6831/23778", 2023-01-15T14:29:10.836 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6829/23778", 2023-01-15T14:29:10.836 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:29:10.837 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:29:10.837 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:29:10.837 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.837 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:29:10.837 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "59362feb-caa9-44e3-b6d0-b79397dacbfc", 2023-01-15T14:29:10.837 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:29:10.837 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:29:10.837 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:29:10.837 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:29:10.838 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:29:10.838 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:29:10.838 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:29:10.838 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 55, 2023-01-15T14:29:10.838 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:29:10.838 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:29:10.838 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:29:10.838 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6800", 2023-01-15T14:29:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:29:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6801", 2023-01-15T14:29:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:29:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.839 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.840 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:29:10.840 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.840 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.840 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6802", 2023-01-15T14:29:10.840 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:29:10.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6803", 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6806", 2023-01-15T14:29:10.841 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:29:10.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.842 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.842 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6807", 2023-01-15T14:29:10.842 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:29:10.842 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.842 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.842 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:29:10.842 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6804", 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6805", 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.843 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6801/23523", 2023-01-15T14:29:10.844 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6803/23523", 2023-01-15T14:29:10.844 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6807/23523", 2023-01-15T14:29:10.844 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6805/23523", 2023-01-15T14:29:10.844 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:29:10.844 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:29:10.844 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:29:10.844 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.844 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:29:10.844 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "fbc8821c-8d78-40f8-a571-b18a70d42000", 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:29:10.845 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6824", 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6825", 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6826", 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6827", 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:29:10.847 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.848 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.848 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:29:10.848 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.848 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.848 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6830", 2023-01-15T14:29:10.848 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6831", 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.849 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.850 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6828", 2023-01-15T14:29:10.850 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:29:10.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.850 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.850 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6829", 2023-01-15T14:29:10.850 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:29:10.850 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.850 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.850 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6825/23525", 2023-01-15T14:29:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6827/23525", 2023-01-15T14:29:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6831/23525", 2023-01-15T14:29:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6829/23525", 2023-01-15T14:29:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:29:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:29:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:29:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:29:10.851 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "34550470-3561-492f-b605-b5f0e8052c1e", 2023-01-15T14:29:10.852 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:29:10.852 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:29:10.852 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.3273468017578125, 2023-01-15T14:29:10.852 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:29:10.852 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:29:10.852 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:29:10.852 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:29:10.852 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:29:10.852 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:29:10.852 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:29:10.852 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:29:10.853 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.853 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.853 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6808", 2023-01-15T14:29:10.853 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:29:10.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.853 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.853 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6809", 2023-01-15T14:29:10.853 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:29:10.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6810", 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.854 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6811", 2023-01-15T14:29:10.855 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:29:10.855 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.855 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.855 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:29:10.855 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.855 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6814", 2023-01-15T14:29:10.855 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:29:10.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6815", 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.856 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6812", 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6813", 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6809/23524", 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6811/23524", 2023-01-15T14:29:10.857 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6815/23524", 2023-01-15T14:29:10.858 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6813/23524", 2023-01-15T14:29:10.858 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:29:10.858 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:29:10.858 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:29:10.858 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.858 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:29:10.858 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1f123d0d-d617-4771-b516-21aa5a29b734", 2023-01-15T14:29:10.858 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:29:10.858 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 58, 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.860 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6816", 2023-01-15T14:29:10.860 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:29:10.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.860 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.860 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6817", 2023-01-15T14:29:10.860 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:29:10.979 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.981 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.981 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:29:10.981 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:10.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.981 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:10.981 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6818", 2023-01-15T14:29:10.981 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:29:10.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:10.994 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:10.994 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6819", 2023-01-15T14:29:10.994 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:29:10.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:10.994 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:10.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:10.998 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:29:10.998 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6822", 2023-01-15T14:29:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:29:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6823", 2023-01-15T14:29:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:29:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:29:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:29:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:29:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6820", 2023-01-15T14:29:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:29:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:29:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6821", 2023-01-15T14:29:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:29:11.015 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.015 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:11.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.015 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6817/23526", 2023-01-15T14:29:11.015 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6819/23526", 2023-01-15T14:29:11.015 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6823/23526", 2023-01-15T14:29:11.015 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6821/23526", 2023-01-15T14:29:11.015 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:29:11.016 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:29:11.016 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:29:11.016 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:11.016 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.016 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:29:11.016 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2023-01-15T14:29:11.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.016 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:29:11.017 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:29:11.017 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:29:11.017 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:29:11.017 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:29:11.017 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:29:11.017 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.117195+0000", 2023-01-15T14:29:11.017 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:29:11.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:29:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:29:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:29:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:29:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:29:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:29:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.065464+0000", 2023-01-15T14:29:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:29:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.138817+0000", 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:29:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:29:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:29:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:29:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:29:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:29:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.284776+0000", 2023-01-15T14:29:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:29:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:29:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:29:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:29:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:29:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:29:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:29:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.271024+0000", 2023-01-15T14:29:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:29:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:29:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:29:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:29:11.022 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:29:11.022 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:29:11.022 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:29:11.022 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:29:05.367738+0000", 2023-01-15T14:29:11.022 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:29:11.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.022 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:29:11.022 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:29:11.022 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:29:11.022 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:29:11.023 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:29:11.023 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:29:11.023 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:28:03.333018+0000", 2023-01-15T14:29:11.023 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:29:11.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.023 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:29:11.023 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:29:11.023 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:29:11.023 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:29:11.023 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:29:11.024 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:29:11.024 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:29:06.105073+0000", 2023-01-15T14:29:11.024 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:29:11.024 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.024 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:29:11.024 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2023-01-15T14:29:11.025 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2023-01-15T14:29:11.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.026 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.8", 2023-01-15T14:29:11.026 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:29:11.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.026 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 7, 2023-01-15T14:29:11.026 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2023-01-15T14:29:11.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.026 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 3, 2023-01-15T14:29:11.026 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 1 2023-01-15T14:29:11.026 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:29:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2023-01-15T14:29:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2023-01-15T14:29:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2023-01-15T14:29:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2023-01-15T14:29:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2023-01-15T14:29:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2023-01-15T14:29:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2023-01-15T14:29:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2023-01-15T14:29:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2023-01-15T14:29:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2023-01-15T14:29:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2023-01-15T14:29:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2023-01-15T14:29:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2023-01-15T14:29:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.160 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:29:11.161 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2023-01-15T14:29:11.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.161 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 20, 2023-01-15T14:29:11.161 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2023-01-15T14:29:11.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.162 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 23, 2023-01-15T14:29:11.162 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:29:11.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.163 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 28, 2023-01-15T14:29:11.164 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:29:11.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.164 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 30, 2023-01-15T14:29:11.164 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 3 2023-01-15T14:29:11.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:11.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.165 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 35, 2023-01-15T14:29:11.169 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:29:11.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.170 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:11.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.170 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:29:11.170 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [ 2023-01-15T14:29:11.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.171 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:29:11.171 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2023-01-15T14:29:11.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:11.171 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 21, 2023-01-15T14:29:11.171 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:29:11.171 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.171 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:11.172 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.172 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:29:11.172 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2023-01-15T14:29:11.172 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2023-01-15T14:29:11.172 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2023-01-15T14:29:11.173 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2023-01-15T14:29:11.173 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2023-01-15T14:29:11.173 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2023-01-15T14:29:11.174 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2023-01-15T14:29:11.174 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2023-01-15T14:29:11.174 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2023-01-15T14:29:11.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:11.174 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:11.176 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:11.182 INFO:tasks.thrashosds.thrasher:j is {'epoch': 64, 'fsid': '1c77fb2b-d1a7-4884-a24e-3935e16d0b6b', 'created': '2023-01-15T14:27:57.735717+0000', 'modified': '2023-01-15T14:29:09.877713+0000', 'last_up_change': '2023-01-15T14:28:04.837522+0000', 'last_in_change': '2023-01-15T14:28:53.706589+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-01-15T14:28:04.899282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-01-15T14:28:11.115002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-01-15T14:28:22.434903+0000', 'flags': 16385, 'flags_names': 'hashpspool,pool_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'pool', 'snap_seq': 41, 'snap_epoch': 64, 'pool_snaps': [{'snapid': 16, 'stamp': '2023-01-15T14:28:44.706724+0000', 'name': 'smithi09025264-snap9'}, {'snapid': 25, 'stamp': '2023-01-15T14:28:52.993772+0000', 'name': 'smithi09025264-snap15'}, {'snapid': 27, 'stamp': '2023-01-15T14:28:55.550715+0000', 'name': 'smithi09025264-snap16'}, {'snapid': 29, 'stamp': '2023-01-15T14:28:56.974245+0000', 'name': 'smithi09025264-snap18'}, {'snapid': 36, 'stamp': '2023-01-15T14:29:04.453734+0000', 'name': 'smithi09025264-snap23'}], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e55c5ac4-3cff-4ee5-8719-373648f83d8d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 23473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 23473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 23473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6820', 'nonce': 23473}]}, 'public_addr': '172.21.15.90:6811/23473', 'cluster_addr': '172.21.15.90:6815/23473', 'heartbeat_back_addr': '172.21.15.90:6823/23473', 'heartbeat_front_addr': '172.21.15.90:6820/23473', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2c1b33c9-0a1e-4734-936d-d51d64fe997a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 55, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 23474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 23474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 23474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 23474}]}, 'public_addr': '172.21.15.90:6801/23474', 'cluster_addr': '172.21.15.90:6803/23474', 'heartbeat_back_addr': '172.21.15.90:6807/23474', 'heartbeat_front_addr': '172.21.15.90:6805/23474', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b314174a-eca9-4baa-a60c-2589f1fc9a8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 23475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 23475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6819', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 23475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 23475}]}, 'public_addr': '172.21.15.90:6810/23475', 'cluster_addr': '172.21.15.90:6814/23475', 'heartbeat_back_addr': '172.21.15.90:6821/23475', 'heartbeat_front_addr': '172.21.15.90:6817/23475', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '29eb0b17-ed5d-4665-a347-2fb8406d9c62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 48, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6824', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6825', 'nonce': 23778}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6826', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6827', 'nonce': 23778}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6830', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6831', 'nonce': 23778}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6828', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6829', 'nonce': 23778}]}, 'public_addr': '172.21.15.90:6825/23778', 'cluster_addr': '172.21.15.90:6827/23778', 'heartbeat_back_addr': '172.21.15.90:6831/23778', 'heartbeat_front_addr': '172.21.15.90:6829/23778', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '59362feb-caa9-44e3-b6d0-b79397dacbfc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 55, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6800', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6801', 'nonce': 23523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6802', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6803', 'nonce': 23523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6806', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6807', 'nonce': 23523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6804', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6805', 'nonce': 23523}]}, 'public_addr': '172.21.15.141:6801/23523', 'cluster_addr': '172.21.15.141:6803/23523', 'heartbeat_back_addr': '172.21.15.141:6807/23523', 'heartbeat_front_addr': '172.21.15.141:6805/23523', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fbc8821c-8d78-40f8-a571-b18a70d42000', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6824', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6825', 'nonce': 23525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6826', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6827', 'nonce': 23525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6830', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6831', 'nonce': 23525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6828', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6829', 'nonce': 23525}]}, 'public_addr': '172.21.15.141:6825/23525', 'cluster_addr': '172.21.15.141:6827/23525', 'heartbeat_back_addr': '172.21.15.141:6831/23525', 'heartbeat_front_addr': '172.21.15.141:6829/23525', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '34550470-3561-492f-b605-b5f0e8052c1e', 'up': 1, 'in': 1, 'weight': 0.3273468017578125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6808', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6809', 'nonce': 23524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6810', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6811', 'nonce': 23524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6814', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6815', 'nonce': 23524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6812', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6813', 'nonce': 23524}]}, 'public_addr': '172.21.15.141:6809/23524', 'cluster_addr': '172.21.15.141:6811/23524', 'heartbeat_back_addr': '172.21.15.141:6815/23524', 'heartbeat_front_addr': '172.21.15.141:6813/23524', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f123d0d-d617-4771-b516-21aa5a29b734', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6816', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6817', 'nonce': 23526}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6818', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6819', 'nonce': 23526}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6822', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6823', 'nonce': 23526}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6820', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6821', 'nonce': 23526}]}, 'public_addr': '172.21.15.141:6817/23526', 'cluster_addr': '172.21.15.141:6819/23526', 'heartbeat_back_addr': '172.21.15.141:6823/23526', 'heartbeat_front_addr': '172.21.15.141:6821/23526', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.117195+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.065464+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.138817+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.284776+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.271024+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:29:05.367738+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:28:03.333018+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:29:06.105073+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.8', 'mappings': [{'from': 7, 'to': 2}, {'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 20, 'length': 2}, {'begin': 23, 'length': 1}, {'begin': 28, 'length': 1}, {'begin': 30, 'length': 3}, {'begin': 35, 'length': 1}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 21, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-01-15T14:29:11.183 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:29:11.184 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:11.184 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:11.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:29:11.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:29:11.186 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:29:11.186 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:29:11.186 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:29:11.186 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:29:11.186 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:29:11.187 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:29:11.187 INFO:tasks.rados.rados.0.smithi090.stdout:241: done (1 left) 2023-01-15T14:29:11.188 INFO:tasks.rados.rados.0.smithi090.stdout:242: write oid 3 current snap is 23 2023-01-15T14:29:11.188 INFO:tasks.rados.rados.0.smithi090.stdout:236: finishing write tid 5 to smithi09025264-29 2023-01-15T14:29:11.188 INFO:tasks.rados.rados.0.smithi090.stdout:242: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2023-01-15T14:29:11.188 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:29:11.189 INFO:tasks.rados.rados.0.smithi090.stdout:242: writing smithi09025264-3 from 749612 to 1223791 tid 1 2023-01-15T14:29:11.189 INFO:tasks.rados.rados.0.smithi090.stdout:242: writing smithi09025264-3 from 1954970 to 2386007 tid 2 2023-01-15T14:29:11.191 INFO:tasks.rados.rados.0.smithi090.stdout:242: writing smithi09025264-3 from 2919420 to 3535797 tid 3 2023-01-15T14:29:11.191 INFO:tasks.rados.rados.0.smithi090.stdout:243: snap_remove snap 17 2023-01-15T14:29:11.192 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:11.173+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:11.277 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:11.273+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:11.319 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:11.319 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:11.319 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:29:11.377 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:11.369+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:11.416 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:11.496 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:11.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:29:11.498 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:11.489+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:11.537 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:11.539 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:29:11.551 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2023-01-15T14:29:10.901151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":207994933,"num_objects":93,"num_object_clones":42,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":5,"num_read":593,"num_read_kb":388358,"num_write":649,"num_write_kb":190895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":90937982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":849,"ondisk_log_size":849,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":78,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":6414804,"kb_used_data":649507528,"kb_used_omap":3898,"kb_used_meta":0,"kb_avail":649480856,"statfs":{"total":671637155840,"available":665068396544,"internally_reserved":27312128,"allocated":665095708672,"data_stored":665095708672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3991810,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":52,"commit_latency_ns":0,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4470564,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":-1,"num_read":23,"num_read_kb":30295,"num_write":21,"num_write_kb":6482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043931"},"pg_stats":[{"pgid":"3.f","version":"63'42","reported_seq":172,"reported_epoch":64,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:09.898576+0000","last_change":"2023-01-15T14:29:09.898576+0000","last_active":"2023-01-15T14:29:09.898576+0000","last_peered":"2023-01-15T14:29:09.898576+0000","last_clean":"2023-01-15T14:29:09.898576+0000","last_became_active":"2023-01-15T14:28:23.846063+0000","last_became_peered":"2023-01-15T14:28:23.846063+0000","last_unstale":"2023-01-15T14:29:09.898576+0000","last_undegraded":"2023-01-15T14:29:09.898576+0000","last_fullsized":"2023-01-15T14:29:09.898576+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:52.368","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088394999999999999,"stat_sum":{"num_bytes":21344659,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":24,"num_read_kb":11043,"num_write":35,"num_write_kb":12817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"23","length":"1"}]},{"pgid":"3.e","version":"62'15","reported_seq":142,"reported_epoch":64,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:09.898677+0000","last_change":"2023-01-15T14:29:09.898774+0000","last_active":"2023-01-15T14:29:09.898677+0000","last_peered":"2023-01-15T14:29:09.898677+0000","last_clean":"2023-01-15T14:29:09.898677+0000","last_became_active":"2023-01-15T14:28:23.846858+0000","last_became_peered":"2023-01-15T14:28:23.846858+0000","last_unstale":"2023-01-15T14:29:09.898677+0000","last_undegraded":"2023-01-15T14:29:09.898677+0000","last_fullsized":"2023-01-15T14:29:09.898677+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:44.514","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078494899999999995,"stat_sum":{"num_bytes":4756315,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":6,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"23","length":"1"}]},{"pgid":"3.d","version":"62'63","reported_seq":223,"reported_epoch":64,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:09.898332+0000","last_change":"2023-01-15T14:29:09.898577+0000","last_active":"2023-01-15T14:29:09.898332+0000","last_peered":"2023-01-15T14:29:09.898332+0000","last_clean":"2023-01-15T14:29:09.898332+0000","last_became_active":"2023-01-15T14:28:54.932913+0000","last_became_peered":"2023-01-15T14:28:54.932913+0000","last_unstale":"2023-01-15T14:29:09.898332+0000","last_undegraded":"2023-01-15T14:29:09.898332+0000","last_fullsized":"2023-01-15T14:29:09.898332+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:33.416","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10628691,"stat_sum":{"num_bytes":13273658,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":38,"num_read_kb":37363,"num_write":34,"num_write_kb":10871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"23","length":"1"}]},{"pgid":"3.c","version":"62'23","reported_seq":89,"reported_epoch":63,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:09.102999+0000","last_change":"2023-01-15T14:29:09.102999+0000","last_active":"2023-01-15T14:29:09.102999+0000","last_peered":"2023-01-15T14:29:09.102999+0000","last_clean":"2023-01-15T14:29:09.102999+0000","last_became_active":"2023-01-15T14:28:54.833568+0000","last_became_peered":"2023-01-15T14:28:54.833568+0000","last_unstale":"2023-01-15T14:29:09.102999+0000","last_undegraded":"2023-01-15T14:29:09.102999+0000","last_fullsized":"2023-01-15T14:29:09.102999+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.421","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.22583681799999999,"stat_sum":{"num_bytes":6627201,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":14,"num_write_kb":4063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2319333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"3"}]},{"pgid":"3.b","version":"63'16","reported_seq":140,"reported_epoch":63,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:09.806846+0000","last_change":"2023-01-15T14:29:09.806846+0000","last_active":"2023-01-15T14:29:09.806846+0000","last_peered":"2023-01-15T14:29:09.806846+0000","last_clean":"2023-01-15T14:29:09.806846+0000","last_became_active":"2023-01-15T14:28:23.934183+0000","last_became_peered":"2023-01-15T14:28:23.934183+0000","last_unstale":"2023-01-15T14:29:09.806846+0000","last_undegraded":"2023-01-15T14:29:09.806846+0000","last_fullsized":"2023-01-15T14:29:09.806846+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:45.586","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.82981534599999995,"stat_sum":{"num_bytes":5215740,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":13386,"num_write":9,"num_write_kb":3028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"3"}]},{"pgid":"3.a","version":"58'72","reported_seq":217,"reported_epoch":60,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:05.946734+0000","last_change":"2023-01-15T14:29:05.946795+0000","last_active":"2023-01-15T14:29:05.946734+0000","last_peered":"2023-01-15T14:29:05.946734+0000","last_clean":"2023-01-15T14:29:05.946734+0000","last_became_active":"2023-01-15T14:29:02.112091+0000","last_became_peered":"2023-01-15T14:29:02.112091+0000","last_unstale":"2023-01-15T14:29:05.946734+0000","last_undegraded":"2023-01-15T14:29:05.946734+0000","last_fullsized":"2023-01-15T14:29:05.946734+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:28.279","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.22806705799999999,"stat_sum":{"num_bytes":21358007,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15773,"num_write":52,"num_write_kb":25910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":21341200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"3.9","version":"60'56","reported_seq":180,"reported_epoch":60,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:05.846743+0000","last_change":"2023-01-15T14:29:05.846743+0000","last_active":"2023-01-15T14:29:05.846743+0000","last_peered":"2023-01-15T14:29:05.846743+0000","last_clean":"2023-01-15T14:29:05.846743+0000","last_became_active":"2023-01-15T14:28:24.109843+0000","last_became_peered":"2023-01-15T14:28:24.109843+0000","last_unstale":"2023-01-15T14:29:05.846743+0000","last_undegraded":"2023-01-15T14:29:05.846743+0000","last_fullsized":"2023-01-15T14:29:05.846743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89790155699999996,"stat_sum":{"num_bytes":17308043,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":57,"num_read_kb":55284,"num_write":40,"num_write_kb":17440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"3.8","version":"62'71","reported_seq":241,"reported_epoch":64,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:09.990450+0000","last_change":"2023-01-15T14:29:09.990450+0000","last_active":"2023-01-15T14:29:09.990450+0000","last_peered":"2023-01-15T14:29:09.990450+0000","last_clean":"2023-01-15T14:29:09.990450+0000","last_became_active":"2023-01-15T14:28:47.937231+0000","last_became_peered":"2023-01-15T14:28:47.937231+0000","last_unstale":"2023-01-15T14:29:09.990450+0000","last_undegraded":"2023-01-15T14:29:09.990450+0000","last_fullsized":"2023-01-15T14:29:09.990450+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:29.801","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.030868455,"stat_sum":{"num_bytes":19469750,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":73603,"num_write":41,"num_write_kb":16481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"23","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":233,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-01-15T14:29:07.868368+0000","last_change":"2023-01-15T14:29:05.329442+0000","last_active":"2023-01-15T14:29:07.868368+0000","last_peered":"2023-01-15T14:29:07.868368+0000","last_clean":"2023-01-15T14:29:07.868368+0000","last_became_active":"2023-01-15T14:29:05.228004+0000","last_became_peered":"2023-01-15T14:29:05.228004+0000","last_unstale":"2023-01-15T14:29:07.868368+0000","last_undegraded":"2023-01-15T14:29:07.868368+0000","last_fullsized":"2023-01-15T14:29:07.868368+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:34.812","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":24,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-01-15T14:29:03.946136+0000","last_change":"2023-01-15T14:28:54.829192+0000","last_active":"2023-01-15T14:29:03.946136+0000","last_peered":"2023-01-15T14:29:03.946136+0000","last_clean":"2023-01-15T14:29:03.946136+0000","last_became_active":"2023-01-15T14:28:54.828857+0000","last_became_peered":"2023-01-15T14:28:54.828857+0000","last_unstale":"2023-01-15T14:29:03.946136+0000","last_undegraded":"2023-01-15T14:29:03.946136+0000","last_fullsized":"2023-01-15T14:29:03.946136+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.638","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"64'48","reported_seq":200,"reported_epoch":64,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:09.992888+0000","last_change":"2023-01-15T14:29:09.907724+0000","last_active":"2023-01-15T14:29:09.992888+0000","last_peered":"2023-01-15T14:29:09.992888+0000","last_clean":"2023-01-15T14:29:09.992888+0000","last_became_active":"2023-01-15T14:28:55.217912+0000","last_became_peered":"2023-01-15T14:28:55.217912+0000","last_unstale":"2023-01-15T14:29:09.992888+0000","last_undegraded":"2023-01-15T14:29:09.992888+0000","last_fullsized":"2023-01-15T14:29:09.992888+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.256","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.52428845499999999,"stat_sum":{"num_bytes":15714162,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":24,"num_read_kb":16181,"num_write":35,"num_write_kb":14028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10821306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"3"}]},{"pgid":"2.0","version":"0'0","reported_seq":55,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-01-15T14:29:08.872774+0000","last_change":"2023-01-15T14:28:13.013616+0000","last_active":"2023-01-15T14:29:08.872774+0000","last_peered":"2023-01-15T14:29:08.872774+0000","last_clean":"2023-01-15T14:29:08.872774+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:29:08.872774+0000","last_undegraded":"2023-01-15T14:29:08.872774+0000","last_fullsized":"2023-01-15T14:29:08.872774+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:29.249","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"64'47","reported_seq":188,"reported_epoch":64,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:10.284719+0000","last_change":"2023-01-15T14:29:10.284719+0000","last_active":"2023-01-15T14:29:10.284719+0000","last_peered":"2023-01-15T14:29:10.284719+0000","last_clean":"2023-01-15T14:29:10.284719+0000","last_became_active":"2023-01-15T14:28:23.842545+0000","last_became_peered":"2023-01-15T14:28:23.842545+0000","last_unstale":"2023-01-15T14:29:10.284719+0000","last_undegraded":"2023-01-15T14:29:10.284719+0000","last_fullsized":"2023-01-15T14:29:10.284719+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:30.758","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.80250399299999997,"stat_sum":{"num_bytes":16523579,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":39236,"num_write":36,"num_write_kb":12307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"23","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":58,"state":"active+clean","last_fresh":"2023-01-15T14:29:04.046205+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:29:04.046205+0000","last_peered":"2023-01-15T14:29:04.046205+0000","last_clean":"2023-01-15T14:29:04.046205+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:29:04.046205+0000","last_undegraded":"2023-01-15T14:29:04.046205+0000","last_fullsized":"2023-01-15T14:29:04.046205+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.689","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"60'82","reported_seq":241,"reported_epoch":60,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:06.067812+0000","last_change":"2023-01-15T14:29:06.067874+0000","last_active":"2023-01-15T14:29:06.067812+0000","last_peered":"2023-01-15T14:29:06.067812+0000","last_clean":"2023-01-15T14:29:06.067812+0000","last_became_active":"2023-01-15T14:29:02.007426+0000","last_became_peered":"2023-01-15T14:29:02.007426+0000","last_unstale":"2023-01-15T14:29:06.067812+0000","last_undegraded":"2023-01-15T14:29:06.067812+0000","last_fullsized":"2023-01-15T14:29:06.067812+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:48.446","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.185385772,"stat_sum":{"num_bytes":19034546,"num_objects":10,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":64,"num_read_kb":53742,"num_write":54,"num_write_kb":18739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16722767,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1e","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":35,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-15T14:29:04.943981+0000","last_change":"2023-01-15T14:29:02.110582+0000","last_active":"2023-01-15T14:29:04.943981+0000","last_peered":"2023-01-15T14:29:04.943981+0000","last_clean":"2023-01-15T14:29:04.943981+0000","last_became_active":"2023-01-15T14:29:02.009443+0000","last_became_peered":"2023-01-15T14:29:02.009443+0000","last_unstale":"2023-01-15T14:29:04.943981+0000","last_undegraded":"2023-01-15T14:29:04.943981+0000","last_fullsized":"2023-01-15T14:29:04.943981+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:27.063","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"57'75","reported_seq":204,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-01-15T14:29:05.846906+0000","last_change":"2023-01-15T14:29:05.847496+0000","last_active":"2023-01-15T14:29:05.846906+0000","last_peered":"2023-01-15T14:29:05.846906+0000","last_clean":"2023-01-15T14:29:05.846906+0000","last_became_active":"2023-01-15T14:28:23.946074+0000","last_became_peered":"2023-01-15T14:28:23.946074+0000","last_unstale":"2023-01-15T14:29:05.846906+0000","last_undegraded":"2023-01-15T14:29:05.846906+0000","last_fullsized":"2023-01-15T14:29:05.846906+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:40.271","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082756699999999997,"stat_sum":{"num_bytes":17079648,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":51233,"num_write":55,"num_write_kb":16632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"1"}]},{"pgid":"3.5","version":"63'21","reported_seq":121,"reported_epoch":63,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:09.680787+0000","last_change":"2023-01-15T14:29:09.478104+0000","last_active":"2023-01-15T14:29:09.680787+0000","last_peered":"2023-01-15T14:29:09.680787+0000","last_clean":"2023-01-15T14:29:09.680787+0000","last_became_active":"2023-01-15T14:28:54.831872+0000","last_became_peered":"2023-01-15T14:28:54.831872+0000","last_unstale":"2023-01-15T14:29:09.680787+0000","last_undegraded":"2023-01-15T14:29:09.680787+0000","last_fullsized":"2023-01-15T14:29:09.680787+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:46.735","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50066762399999998,"stat_sum":{"num_bytes":7428595,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":12,"num_write_kb":6933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4252105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1e","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-15T14:29:05.043432+0000","last_change":"2023-01-15T14:28:12.897110+0000","last_active":"2023-01-15T14:29:05.043432+0000","last_peered":"2023-01-15T14:29:05.043432+0000","last_clean":"2023-01-15T14:29:05.043432+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:29:05.043432+0000","last_undegraded":"2023-01-15T14:29:05.043432+0000","last_fullsized":"2023-01-15T14:29:05.043432+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:27.569","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"44'36","reported_seq":182,"reported_epoch":60,"state":"active+clean","last_fresh":"2023-01-15T14:29:05.882689+0000","last_change":"2023-01-15T14:29:05.883173+0000","last_active":"2023-01-15T14:29:05.882689+0000","last_peered":"2023-01-15T14:29:05.882689+0000","last_clean":"2023-01-15T14:29:05.882689+0000","last_became_active":"2023-01-15T14:28:54.825996+0000","last_became_peered":"2023-01-15T14:28:54.825996+0000","last_unstale":"2023-01-15T14:29:05.882689+0000","last_undegraded":"2023-01-15T14:29:05.882689+0000","last_fullsized":"2023-01-15T14:29:05.882689+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.440","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073272099999999998,"stat_sum":{"num_bytes":12100908,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":30,"num_read_kb":21402,"num_write":32,"num_write_kb":8984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12100908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1e","length":"1"},{"start":"20","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":55,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-01-15T14:29:08.873110+0000","last_change":"2023-01-15T14:28:13.318466+0000","last_active":"2023-01-15T14:29:08.873110+0000","last_peered":"2023-01-15T14:29:08.873110+0000","last_clean":"2023-01-15T14:29:08.873110+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:29:08.873110+0000","last_undegraded":"2023-01-15T14:29:08.873110+0000","last_fullsized":"2023-01-15T14:29:08.873110+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.665","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"62'41","reported_seq":159,"reported_epoch":63,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:09.477992+0000","last_change":"2023-01-15T14:29:09.477992+0000","last_active":"2023-01-15T14:29:09.477992+0000","last_peered":"2023-01-15T14:29:09.477992+0000","last_clean":"2023-01-15T14:29:09.477992+0000","last_became_active":"2023-01-15T14:28:23.741266+0000","last_became_peered":"2023-01-15T14:28:23.741266+0000","last_unstale":"2023-01-15T14:29:09.477992+0000","last_undegraded":"2023-01-15T14:29:09.477992+0000","last_fullsized":"2023-01-15T14:29:09.477992+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.60109975100000002,"stat_sum":{"num_bytes":5680700,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":8,"num_read_kb":4,"num_write":21,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"3"}]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-15T14:29:04.944014+0000","last_change":"2023-01-15T14:29:01.810622+0000","last_active":"2023-01-15T14:29:04.944014+0000","last_peered":"2023-01-15T14:29:04.944014+0000","last_clean":"2023-01-15T14:29:04.944014+0000","last_became_active":"2023-01-15T14:29:01.809869+0000","last_became_peered":"2023-01-15T14:29:01.809869+0000","last_unstale":"2023-01-15T14:29:04.944014+0000","last_undegraded":"2023-01-15T14:29:04.944014+0000","last_fullsized":"2023-01-15T14:29:04.944014+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:34.307","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"64'24","reported_seq":150,"reported_epoch":64,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:10.185911+0000","last_change":"2023-01-15T14:29:10.185911+0000","last_active":"2023-01-15T14:29:10.185911+0000","last_peered":"2023-01-15T14:29:10.185911+0000","last_clean":"2023-01-15T14:29:10.185911+0000","last_became_active":"2023-01-15T14:28:23.841189+0000","last_became_peered":"2023-01-15T14:28:23.841189+0000","last_unstale":"2023-01-15T14:29:10.185911+0000","last_undegraded":"2023-01-15T14:29:10.185911+0000","last_fullsized":"2023-01-15T14:29:10.185911+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:34.816","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100754633,"stat_sum":{"num_bytes":4489054,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":14,"num_write_kb":6113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"14","length":"1"},{"start":"17","length":"1"},{"start":"1c","length":"1"},{"start":"1e","length":"3"},{"start":"23","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":52,"reported_epoch":59,"state":"active+clean","last_fresh":"2023-01-15T14:29:04.935555+0000","last_change":"2023-01-15T14:28:12.896363+0000","last_active":"2023-01-15T14:29:04.935555+0000","last_peered":"2023-01-15T14:29:04.935555+0000","last_clean":"2023-01-15T14:29:04.935555+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:29:04.935555+0000","last_undegraded":"2023-01-15T14:29:04.935555+0000","last_fullsized":"2023-01-15T14:29:04.935555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.332","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":207404565,"num_objects":91,"num_object_clones":42,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":5,"num_read":477,"num_read_kb":388258,"num_write":490,"num_write_kb":188479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":90347614,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":732,"ondisk_log_size":732,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574863,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":895000,"kb_used_data":92806228,"kb_used_omap":528,"kb_used_meta":0,"kb_avail":92804380,"statfs":{"total":95948165120,"available":95031685120,"internally_reserved":1892352,"allocated":95033577472,"data_stored":95033577472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":541100,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.0700000000000003,"5min":4.0700000000000003,"15min":4.0700000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":75.893000000000001,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":9.2089999999999996,"5min":9.2089999999999996,"15min":9.2089999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":71.308999999999997,"5min":71.308999999999997,"15min":71.308999999999997},"last":0.65300000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":75.959999999999994,"5min":75.959999999999994,"15min":75.959999999999994},"last":0.79000000000000004},{"interface":"front","average":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":6.2800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":68.605000000000004,"5min":68.605000000000004,"15min":68.605000000000004},"last":0.83399999999999996}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.7910000000000004,"5min":4.7910000000000004,"15min":4.7910000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":94.733999999999995,"5min":94.733999999999995,"15min":94.733999999999995},"last":94.733999999999995},{"interface":"front","average":{"1min":4.1779999999999999,"5min":4.1779999999999999,"15min":4.1779999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":75.998999999999995},"last":0.90800000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.7770000000000001,"5min":3.7770000000000001,"15min":3.7770000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":76.034999999999997,"5min":76.034999999999997,"15min":76.034999999999997},"last":0.95999999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":9.4160000000000004,"5min":9.4160000000000004,"15min":9.4160000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":76.658000000000001,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.754},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":3.5129999999999999,"5min":3.5129999999999999,"15min":3.5129999999999999},"last":0.498}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.327,"5min":6.327,"15min":6.327},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":68.679000000000002,"5min":68.679000000000002,"15min":68.679000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":4.9450000000000003,"5min":4.9450000000000003,"15min":4.9450000000000003},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":95.430000000000007,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.70999999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":8.5239999999999991,"5min":8.5239999999999991,"15min":8.5239999999999991},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":68.635999999999996,"5min":68.635999999999996,"15min":68.635999999999996},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.6520000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.88300000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574863,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":875720,"kb_used_data":92823804,"kb_used_omap":594,"kb_used_meta":0,"kb_avail":92823660,"statfs":{"total":95948165120,"available":95051427840,"internally_reserved":147456,"allocated":95051575296,"data_stored":95051575296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":609202,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.052}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":81.686999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":78.888000000000005}]}]},{"osd":5,"up_from":0,"seq":55834574863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574862,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":951660,"kb_used_data":92753024,"kb_used_omap":552,"kb_used_meta":0,"kb_avail":92747720,"statfs":{"total":95948165120,"available":94973665280,"internally_reserved":5431296,"allocated":94979096576,"data_stored":94979096576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":565265,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":8,"commit_latency_ns":0,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.1}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90700000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574863,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":952804,"kb_used_data":92748280,"kb_used_omap":627,"kb_used_meta":0,"kb_avail":92746576,"statfs":{"total":95948165120,"available":94972493824,"internally_reserved":1744896,"allocated":94974238720,"data_stored":94974238720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":642540,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.8479999999999999,"5min":4.8479999999999999,"15min":4.8479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":93.632999999999996,"5min":93.632999999999996,"15min":93.632999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":7.6420000000000003,"5min":7.6420000000000003,"15min":7.6420000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":93.674000000000007,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.68700000000000006}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.532,"5min":13.532,"15min":13.532},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":159.256,"5min":159.256,"15min":159.256},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.878}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.3490000000000002,"5min":3.3490000000000002,"15min":3.3490000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":59.896999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.85299999999999998},{"interface":"front","average":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":31.233000000000001,"5min":31.233000000000001,"15min":31.233000000000001},"last":0.746}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":6.5620000000000003,"5min":6.5620000000000003,"15min":6.5620000000000003},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":66.311999999999998,"5min":66.311999999999998,"15min":66.311999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":13.874000000000001,"5min":13.874000000000001,"15min":13.874000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":159.47200000000001,"5min":159.47200000000001,"15min":159.47200000000001},"last":0.76600000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":13.903,"15min":13.903},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":159.43000000000001,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":1.105}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":1.1479999999999999},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":60.098999999999997,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.96199999999999997}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.0359999999999996,"5min":4.0359999999999996,"15min":4.0359999999999996},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":60.165999999999997,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.0049999999999999},{"interface":"front","average":{"1min":18.09,"5min":18.09,"15min":18.09},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":159.512,"5min":159.512,"15min":159.512},"last":0.97699999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":901704,"kb_used_data":92802860,"kb_used_omap":391,"kb_used_meta":0,"kb_avail":92797676,"statfs":{"total":95948165120,"available":95024820224,"internally_reserved":5308416,"allocated":95030128640,"data_stored":95030128640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":400447,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":18,"commit_latency_ns":0,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.73399999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.5979999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":3.3559999999999999,"5min":3.3559999999999999,"15min":3.3559999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":47.518000000000001,"5min":47.518000000000001,"15min":47.518000000000001},"last":0.85299999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":10.401999999999999,"5min":10.401999999999999,"15min":10.401999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":183.53,"5min":183.53,"15min":183.53},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.995}]},{"osd":4,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.6499999999999999,"15min":3.6499999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":53.139000000000003,"5min":53.139000000000003,"15min":53.139000000000003},"last":1.3149999999999999},{"interface":"front","average":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":3.8090000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":53.186999999999998,"5min":53.186999999999998,"15min":53.186999999999998},"last":1.0289999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":1.198}]},{"osd":6,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":1.135}]},{"osd":7,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":37.124000000000002,"5min":37.124000000000002,"15min":37.124000000000002},"last":1.2649999999999999},{"interface":"front","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":30.445,"5min":30.445,"15min":30.445},"last":1.0509999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574863,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":910500,"kb_used_data":92798000,"kb_used_omap":543,"kb_used_meta":0,"kb_avail":92788880,"statfs":{"total":95948165120,"available":95015813120,"internally_reserved":9338880,"allocated":95025152000,"data_stored":95025152000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":556478,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":7,"commit_latency_ns":0,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":3.8929999999999998,"15min":3.8929999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":46.591000000000001,"5min":46.591000000000001,"15min":46.591000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":50.616,"5min":50.616,"15min":50.616},"last":0.58199999999999996}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":46.747999999999998,"5min":46.747999999999998,"15min":46.747999999999998},"last":0.66200000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":19.835999999999999,"5min":19.835999999999999,"15min":19.835999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":282.01400000000001,"5min":282.01400000000001,"15min":282.01400000000001},"last":34.75},{"interface":"front","average":{"1min":7.2450000000000001,"5min":7.2450000000000001,"15min":7.2450000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":53.145000000000003,"5min":53.145000000000003,"15min":53.145000000000003},"last":53.145000000000003}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.7279999999999998,"5min":4.7279999999999998,"15min":4.7279999999999998},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":50.887999999999998,"5min":50.887999999999998,"15min":50.887999999999998},"last":0.72299999999999998},{"interface":"front","average":{"1min":3.605,"5min":3.605,"15min":3.605},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":52.512,"5min":52.512,"15min":52.512},"last":0.83899999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":4.4379999999999997},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":46.905999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.0609999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.0019999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":50.844999999999999,"5min":50.844999999999999,"15min":50.844999999999999},"last":1.125},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.095}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":52.595999999999997,"5min":52.595999999999997,"15min":52.595999999999997},"last":1.0169999999999999},{"interface":"front","average":{"1min":4.75,"5min":4.75,"15min":4.75},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":50.804000000000002,"5min":50.804000000000002,"15min":50.804000000000002},"last":1.032}]}]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":927416,"kb_used_data":92775332,"kb_used_omap":660,"kb_used_meta":0,"kb_avail":92771964,"statfs":{"total":95948165120,"available":94998491136,"internally_reserved":3448832,"allocated":95001939968,"data_stored":95001939968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":676778,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.55800000000000005}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.755},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.66600000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":16.779,"5min":16.779,"15min":16.779},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":37.426000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.69199999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.79500000000000004}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":9.798,"5min":9.798,"15min":9.798},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":176.34399999999999,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":1.268}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":1.163}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":15.670999999999999,"5min":15.670999999999999,"15min":15.670999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":292.61099999999999,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.218},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.0049999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:29:11.689 INFO:tasks.thrashosds.thrasher:Setting 2.2 to [2, 1, 6] 2023-01-15T14:29:11.689 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.2', '2', '1', '6'] 2023-01-15T14:29:11.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.2 2 1 6 2023-01-15T14:29:11.691 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:11.691 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:11.691 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:29:11.692 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:11.685+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:11.712 INFO:tasks.rados.rados.0.smithi090.stdout:243: done (2 left) 2023-01-15T14:29:11.712 INFO:tasks.rados.rados.0.smithi090.stdout:244: delete oid 45 current snap is 23 2023-01-15T14:29:11.712 INFO:tasks.rados.rados.0.smithi090.stdout:236: finishing write tid 6 to smithi09025264-29 2023-01-15T14:29:11.712 INFO:tasks.rados.rados.0.smithi090.stdout:236: oid 29 updating version 0 to 41 2023-01-15T14:29:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:236: oid 29 updating version 41 to 42 2023-01-15T14:29:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:236: oid 29 updating version 42 to 43 2023-01-15T14:29:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:236: oid 29 version 43 is already newer than 40 2023-01-15T14:29:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:236: oid 29 updating version 43 to 44 2023-01-15T14:29:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 44 (ObjNum 114 snap 23 seq_num 114) dirty exists 2023-01-15T14:29:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:236: left oid 29 (ObjNum 114 snap 23 seq_num 114) 2023-01-15T14:29:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:242: finishing write tid 1 to smithi09025264-3 2023-01-15T14:29:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:242: finishing write tid 2 to smithi09025264-3 2023-01-15T14:29:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:242: finishing write tid 3 to smithi09025264-3 2023-01-15T14:29:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:242: finishing write tid 4 to smithi09025264-3 2023-01-15T14:29:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:242: finishing write tid 5 to smithi09025264-3 2023-01-15T14:29:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:242: oid 3 updating version 0 to 91 2023-01-15T14:29:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:242: oid 3 version 91 is already newer than 89 2023-01-15T14:29:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:242: oid 3 version 91 is already newer than 90 2023-01-15T14:29:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:242: oid 3 updating version 91 to 92 2023-01-15T14:29:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 92 (ObjNum 116 snap 23 seq_num 116) dirty exists 2023-01-15T14:29:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:242: left oid 3 (ObjNum 116 snap 23 seq_num 116) 2023-01-15T14:29:11.792 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:11.785+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:11.848 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:11.858 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:11.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:29:11.892 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:11.885+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:11.994 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:11.989+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:12.100 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:12.100 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:12.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:29:12.101 INFO:tasks.rados.rados.0.smithi090.stdout:236: done (2 left) 2023-01-15T14:29:12.101 INFO:tasks.rados.rados.0.smithi090.stdout:242: done (1 left) 2023-01-15T14:29:12.101 INFO:tasks.rados.rados.0.smithi090.stdout:244: done (0 left) 2023-01-15T14:29:12.101 INFO:tasks.rados.rados.0.smithi090.stdout:245: delete oid 20 current snap is 23 2023-01-15T14:29:12.102 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:12.097+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:12.115 INFO:tasks.rados.rados.0.smithi090.stdout:245: done (0 left) 2023-01-15T14:29:12.115 INFO:tasks.rados.rados.0.smithi090.stdout:246: copy_from oid 48 from oid 47 current snap is 23 2023-01-15T14:29:12.115 INFO:tasks.rados.rados.0.smithi090.stdout:247: copy_from oid 34 from oid 25 current snap is 23 2023-01-15T14:29:12.115 INFO:tasks.rados.rados.0.smithi090.stdout:248: copy_from oid 19 from oid 10 current snap is 23 2023-01-15T14:29:12.116 INFO:tasks.rados.rados.0.smithi090.stdout:249: read oid 9 snap -1 2023-01-15T14:29:12.116 INFO:tasks.rados.rados.0.smithi090.stdout:249: expect (ObjNum 37 snap 0 seq_num 37) 2023-01-15T14:29:12.116 INFO:tasks.rados.rados.0.smithi090.stdout:250: read oid 46 snap -1 2023-01-15T14:29:12.116 INFO:tasks.rados.rados.0.smithi090.stdout:250: expect (ObjNum 45 snap 0 seq_num 45) 2023-01-15T14:29:12.116 INFO:tasks.rados.rados.0.smithi090.stdout:251: snap_remove snap 14 2023-01-15T14:29:12.202 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:12.197+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:12.225 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:12.225 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:12.235 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:29:12.303 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:12.297+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:12.400 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:12.401 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:12.404 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:12.401+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:12.411 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:29:12.504 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:12.501+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:12.527 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:12.527 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:12.536 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:29:12.605 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:12.601+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:12.681 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:12.681 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:12.681 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:29:12.706 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:12.697+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:12.844 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:12.844 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:12.845 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:29:12.845 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:12.837+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:12.914 INFO:tasks.rados.rados.0.smithi090.stdout:251: done (5 left) 2023-01-15T14:29:14.071 INFO:tasks.rados.rados.0.smithi090.stdout:252: rollback oid 26 current snap is 23 2023-01-15T14:29:14.071 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 26 to 8 2023-01-15T14:29:14.072 INFO:tasks.rados.rados.0.smithi090.stdout:248: finishing copy_from to smithi09025264-19 2023-01-15T14:29:14.072 INFO:tasks.rados.rados.0.smithi090.stdout:248: got expected ENOENT (src dne) 2023-01-15T14:29:14.072 INFO:tasks.rados.rados.0.smithi090.stdout:248: finishing copy_from racing read to smithi09025264-19 2023-01-15T14:29:14.072 INFO:tasks.rados.rados.0.smithi090.stdout:249: expect (ObjNum 37 snap 0 seq_num 37) 2023-01-15T14:29:14.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:29:14.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:29:14.075 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:14.075 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:14.075 INFO:teuthology.orchestra.run.smithi090.stderr:set 2.2 pg_upmap mapping to [2,1,6] 2023-01-15T14:29:14.076 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:29:14.080 INFO:tasks.rados.rados.0.smithi090.stdout:250: expect (ObjNum 45 snap 0 seq_num 45) 2023-01-15T14:29:14.080 INFO:tasks.rados.rados.0.smithi090.stdout:247: finishing copy_from to smithi09025264-34 2023-01-15T14:29:14.080 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 43 (ObjNum 74 snap 8 seq_num 74) dirty exists 2023-01-15T14:29:14.080 INFO:tasks.rados.rados.0.smithi090.stdout:247: finishing copy_from racing read to smithi09025264-34 2023-01-15T14:29:14.080 INFO:tasks.rados.rados.0.smithi090.stdout:246: finishing copy_from to smithi09025264-48 2023-01-15T14:29:14.081 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 80 (ObjNum 110 snap 23 seq_num 110) dirty exists 2023-01-15T14:29:14.081 INFO:tasks.rados.rados.0.smithi090.stdout:246: finishing copy_from racing read to smithi09025264-48 2023-01-15T14:29:14.081 INFO:tasks.rados.rados.0.smithi090.stdout:246: done (5 left) 2023-01-15T14:29:14.081 INFO:tasks.rados.rados.0.smithi090.stdout:247: done (4 left) 2023-01-15T14:29:14.081 INFO:tasks.rados.rados.0.smithi090.stdout:248: done (3 left) 2023-01-15T14:29:14.081 INFO:tasks.rados.rados.0.smithi090.stdout:249: done (2 left) 2023-01-15T14:29:14.081 INFO:tasks.rados.rados.0.smithi090.stdout:250: done (1 left) 2023-01-15T14:29:14.081 INFO:tasks.rados.rados.0.smithi090.stdout:253: read oid 47 snap 8 2023-01-15T14:29:14.081 INFO:tasks.rados.rados.0.smithi090.stdout:253: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:29:14.081 INFO:tasks.rados.rados.0.smithi090.stdout:254: copy_from oid 12 from oid 40 current snap is 23 2023-01-15T14:29:14.081 INFO:tasks.rados.rados.0.smithi090.stdout:255: delete oid 11 current snap is 23 2023-01-15T14:29:14.082 INFO:tasks.rados.rados.0.smithi090.stdout:253: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:29:14.082 INFO:tasks.rados.rados.0.smithi090.stdout:252: finishing rollback tid 0 to smithi09025264-26 2023-01-15T14:29:14.082 INFO:tasks.rados.rados.0.smithi090.stdout:252: finishing rollback tid 1 to smithi09025264-26 2023-01-15T14:29:14.082 INFO:tasks.rados.rados.0.smithi090.stdout:252: finishing rollback tid 2 to smithi09025264-26 2023-01-15T14:29:14.082 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 107 (ObjNum 25 snap 0 seq_num 25) dirty exists 2023-01-15T14:29:14.082 INFO:tasks.rados.rados.0.smithi090.stdout:254: finishing copy_from to smithi09025264-12 2023-01-15T14:29:14.082 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 10 (ObjNum 39 snap 0 seq_num 39) dirty exists 2023-01-15T14:29:14.082 INFO:tasks.rados.rados.0.smithi090.stdout:254: finishing copy_from racing read to smithi09025264-12 2023-01-15T14:29:14.082 INFO:tasks.rados.rados.0.smithi090.stdout:252: done (3 left) 2023-01-15T14:29:14.082 INFO:tasks.rados.rados.0.smithi090.stdout:253: done (2 left) 2023-01-15T14:29:14.082 INFO:tasks.rados.rados.0.smithi090.stdout:254: done (1 left) 2023-01-15T14:29:14.083 INFO:tasks.rados.rados.0.smithi090.stdout:255: done (0 left) 2023-01-15T14:29:14.083 INFO:tasks.rados.rados.0.smithi090.stdout:256: write oid 4 current snap is 23 2023-01-15T14:29:14.083 INFO:tasks.rados.rados.0.smithi090.stdout:256: seq_num 120 ranges {416840=775175,1672545=344295} 2023-01-15T14:29:14.083 INFO:tasks.rados.rados.0.smithi090.stdout:256: writing smithi09025264-4 from 416840 to 1192015 tid 1 2023-01-15T14:29:14.083 INFO:tasks.rados.rados.0.smithi090.stdout:256: writing smithi09025264-4 from 1672545 to 2016840 tid 2 2023-01-15T14:29:14.083 INFO:tasks.rados.rados.0.smithi090.stdout:257: write oid 39 current snap is 23 2023-01-15T14:29:14.083 INFO:tasks.rados.rados.0.smithi090.stdout:257: seq_num 121 ranges {433647=450424,1331027=582033} 2023-01-15T14:29:14.083 INFO:tasks.rados.rados.0.smithi090.stdout:257: writing smithi09025264-39 from 433647 to 884071 tid 1 2023-01-15T14:29:14.083 INFO:tasks.rados.rados.0.smithi090.stdout:257: writing smithi09025264-39 from 1331027 to 1913060 tid 2 2023-01-15T14:29:14.083 INFO:tasks.rados.rados.0.smithi090.stdout:258: read oid 24 snap -1 2023-01-15T14:29:14.083 INFO:tasks.rados.rados.0.smithi090.stdout:258: expect (ObjNum 34 snap 0 seq_num 34) 2023-01-15T14:29:14.084 INFO:tasks.rados.rados.0.smithi090.stdout:259: delete oid 14 current snap is 23 2023-01-15T14:29:14.084 INFO:tasks.rados.rados.0.smithi090.stdout:256: finishing write tid 1 to smithi09025264-4 2023-01-15T14:29:14.084 INFO:tasks.rados.rados.0.smithi090.stdout:259: done (3 left) 2023-01-15T14:29:14.084 INFO:tasks.rados.rados.0.smithi090.stdout:260: write (excl) oid 28 current snap is 23 2023-01-15T14:29:14.084 INFO:tasks.rados.rados.0.smithi090.stdout:260: seq_num 123 ranges {467261=717275,1847991=219270} 2023-01-15T14:29:14.084 INFO:tasks.rados.rados.0.smithi090.stdout:260: writing smithi09025264-28 from 467261 to 1184536 tid 1 2023-01-15T14:29:14.084 INFO:tasks.rados.rados.0.smithi090.stdout:260: writing smithi09025264-28 from 1847991 to 2067261 tid 2 2023-01-15T14:29:14.084 INFO:tasks.rados.rados.0.smithi090.stdout:261: rollback oid 46 current snap is 23 2023-01-15T14:29:14.084 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 46 to 22 2023-01-15T14:29:14.084 INFO:tasks.rados.rados.0.smithi090.stdout:262: read oid 38 snap -1 2023-01-15T14:29:14.085 INFO:tasks.rados.rados.0.smithi090.stdout:262: expect (ObjNum 37 snap 0 seq_num 37) 2023-01-15T14:29:14.085 INFO:tasks.rados.rados.0.smithi090.stdout:263: write oid 17 current snap is 23 2023-01-15T14:29:14.085 INFO:tasks.rados.rados.0.smithi090.stdout:263: seq_num 124 ranges {484068=792524,1906473=177595} 2023-01-15T14:29:14.085 INFO:tasks.rados.rados.0.smithi090.stdout:263: writing smithi09025264-17 from 484068 to 1276592 tid 1 2023-01-15T14:29:14.085 INFO:tasks.rados.rados.0.smithi090.stdout:263: writing smithi09025264-17 from 1906473 to 2084068 tid 2 2023-01-15T14:29:14.085 INFO:tasks.rados.rados.0.smithi090.stdout:264: read oid 29 snap -1 2023-01-15T14:29:14.287 INFO:tasks.rados.rados.0.smithi090.stdout:264: expect (ObjNum 114 snap 23 seq_num 114) 2023-01-15T14:29:14.287 INFO:tasks.rados.rados.0.smithi090.stdout:265: read oid 37 snap -1 2023-01-15T14:29:14.287 INFO:tasks.rados.rados.0.smithi090.stdout:265: expect (ObjNum 36 snap 0 seq_num 36) 2023-01-15T14:29:14.287 INFO:tasks.rados.rados.0.smithi090.stdout:266: write (excl) oid 20 current snap is 23 2023-01-15T14:29:14.287 INFO:tasks.rados.rados.0.smithi090.stdout:266: seq_num 125 ranges {500875=467773,1564955=535920} 2023-01-15T14:29:14.288 INFO:tasks.rados.rados.0.smithi090.stdout:266: writing smithi09025264-20 from 500875 to 968648 tid 1 2023-01-15T14:29:14.288 INFO:tasks.rados.rados.0.smithi090.stdout:266: writing smithi09025264-20 from 1564955 to 2100875 tid 2 2023-01-15T14:29:14.288 INFO:tasks.rados.rados.0.smithi090.stdout:267: write oid 5 current snap is 23 2023-01-15T14:29:14.288 INFO:tasks.rados.rados.0.smithi090.stdout:267: seq_num 126 ranges {517682=543022,1623437=494245} 2023-01-15T14:29:14.288 INFO:tasks.rados.rados.0.smithi090.stdout:267: writing smithi09025264-5 from 517682 to 1060704 tid 1 2023-01-15T14:29:14.288 INFO:tasks.rados.rados.0.smithi090.stdout:267: writing smithi09025264-5 from 1623437 to 2117682 tid 2 2023-01-15T14:29:14.288 INFO:tasks.rados.rados.0.smithi090.stdout:268: write (excl) oid 33 current snap is 23 2023-01-15T14:29:14.288 INFO:tasks.rados.rados.0.smithi090.stdout:262: expect (ObjNum 37 snap 0 seq_num 37) 2023-01-15T14:29:14.288 INFO:tasks.rados.rados.0.smithi090.stdout:257: finishing write tid 1 to smithi09025264-39 2023-01-15T14:29:14.288 INFO:tasks.rados.rados.0.smithi090.stdout:257: finishing write tid 2 to smithi09025264-39 2023-01-15T14:29:14.288 INFO:tasks.rados.rados.0.smithi090.stdout:257: finishing write tid 3 to smithi09025264-39 2023-01-15T14:29:14.289 INFO:tasks.rados.rados.0.smithi090.stdout:257: finishing write tid 4 to smithi09025264-39 2023-01-15T14:29:14.289 INFO:tasks.rados.rados.0.smithi090.stdout:257: oid 39 updating version 0 to 46 2023-01-15T14:29:14.289 INFO:tasks.rados.rados.0.smithi090.stdout:257: oid 39 version 46 is already newer than 45 2023-01-15T14:29:14.289 INFO:tasks.rados.rados.0.smithi090.stdout:257: oid 39 version 46 is already newer than 44 2023-01-15T14:29:14.289 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 46 (ObjNum 121 snap 23 seq_num 121) dirty exists 2023-01-15T14:29:14.289 INFO:tasks.rados.rados.0.smithi090.stdout:257: left oid 39 (ObjNum 121 snap 23 seq_num 121) 2023-01-15T14:29:14.289 INFO:tasks.rados.rados.0.smithi090.stdout:258: expect (ObjNum 34 snap 0 seq_num 34) 2023-01-15T14:29:14.290 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:14.065+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:14.293 INFO:tasks.rados.rados.0.smithi090.stdout:261: finishing rollback tid 0 to smithi09025264-46 2023-01-15T14:29:14.293 INFO:tasks.rados.rados.0.smithi090.stdout:263: finishing write tid 1 to smithi09025264-17 2023-01-15T14:29:14.293 INFO:tasks.rados.rados.0.smithi090.stdout:261: finishing rollback tid 1 to smithi09025264-46 2023-01-15T14:29:14.293 INFO:tasks.rados.rados.0.smithi090.stdout:261: finishing rollback tid 2 to smithi09025264-46 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 49 (ObjNum 45 snap 0 seq_num 45) dirty exists 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:265: expect (ObjNum 36 snap 0 seq_num 36) 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:260: finishing write tid 1 to smithi09025264-28 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:263: finishing write tid 2 to smithi09025264-17 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:268: seq_num 127 ranges {534489=618271,1565566=568923} 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:268: writing smithi09025264-33 from 534489 to 1152760 tid 1 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:268: writing smithi09025264-33 from 1565566 to 2134489 tid 2 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:257: done (11 left) 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:258: done (10 left) 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:261: done (9 left) 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:262: done (8 left) 2023-01-15T14:29:14.294 INFO:tasks.rados.rados.0.smithi090.stdout:265: done (7 left) 2023-01-15T14:29:14.295 INFO:tasks.rados.rados.0.smithi090.stdout:269: write (excl) oid 15 current snap is 23 2023-01-15T14:29:14.295 INFO:tasks.rados.rados.0.smithi090.stdout:263: finishing write tid 3 to smithi09025264-17 2023-01-15T14:29:14.295 INFO:tasks.rados.rados.0.smithi090.stdout:269: seq_num 128 ranges {551296=693520,2024048=127248} 2023-01-15T14:29:14.295 INFO:tasks.rados.rados.0.smithi090.stdout:269: writing smithi09025264-15 from 551296 to 1244816 tid 1 2023-01-15T14:29:14.295 INFO:tasks.rados.rados.0.smithi090.stdout:269: writing smithi09025264-15 from 2024048 to 2151296 tid 2 2023-01-15T14:29:14.295 INFO:tasks.rados.rados.0.smithi090.stdout:270: write (excl) oid 22 current snap is 23 2023-01-15T14:29:14.295 INFO:tasks.rados.rados.0.smithi090.stdout:270: seq_num 129 ranges {568103=768769,2082530=85573} 2023-01-15T14:29:14.295 INFO:tasks.rados.rados.0.smithi090.stdout:270: writing smithi09025264-22 from 568103 to 1336872 tid 1 2023-01-15T14:29:14.295 INFO:tasks.rados.rados.0.smithi090.stdout:270: writing smithi09025264-22 from 2082530 to 2168103 tid 2 2023-01-15T14:29:14.295 INFO:tasks.rados.rados.0.smithi090.stdout:271: read oid 47 snap -1 2023-01-15T14:29:14.295 INFO:tasks.rados.rados.0.smithi090.stdout:271: expect (ObjNum 110 snap 23 seq_num 110) 2023-01-15T14:29:14.296 INFO:tasks.rados.rados.0.smithi090.stdout:272: snap_create 2023-01-15T14:29:14.296 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 10 2023-01-15T14:29:14.296 INFO:tasks.rados.rados.0.smithi090.stdout:260: finishing write tid 2 to smithi09025264-28 2023-01-15T14:29:14.296 INFO:tasks.rados.rados.0.smithi090.stdout:260: finishing write tid 3 to smithi09025264-28 2023-01-15T14:29:14.296 INFO:tasks.rados.rados.0.smithi090.stdout:260: finishing write tid 4 to smithi09025264-28 2023-01-15T14:29:14.296 INFO:tasks.rados.rados.0.smithi090.stdout:260: oid 28 updating version 0 to 111 2023-01-15T14:29:14.296 INFO:tasks.rados.rados.0.smithi090.stdout:260: oid 28 version 111 is already newer than 110 2023-01-15T14:29:14.296 INFO:tasks.rados.rados.0.smithi090.stdout:260: oid 28 updating version 111 to 112 2023-01-15T14:29:14.296 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 112 (ObjNum 123 snap 23 seq_num 123) dirty exists 2023-01-15T14:29:14.296 INFO:tasks.rados.rados.0.smithi090.stdout:260: left oid 28 (ObjNum 123 snap 23 seq_num 123) 2023-01-15T14:29:14.296 INFO:tasks.rados.rados.0.smithi090.stdout:260: done (9 left) 2023-01-15T14:29:14.297 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 9 2023-01-15T14:29:14.297 INFO:tasks.rados.rados.0.smithi090.stdout:271: expect (ObjNum 110 snap 23 seq_num 110) 2023-01-15T14:29:14.298 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:14.298 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:14.299 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:29:14.299 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:14.289+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:14.394 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:14.385+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:14.489 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:14.490 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:14.490 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:14.490 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:29:14.494 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:14.489+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:14.526 INFO:tasks.rados.rados.0.smithi090.stdout:270: finishing write tid 1 to smithi09025264-22 2023-01-15T14:29:14.671 INFO:tasks.rados.rados.0.smithi090.stdout:263: finishing write tid 4 to smithi09025264-17 2023-01-15T14:29:14.672 INFO:tasks.rados.rados.0.smithi090.stdout:263: oid 17 updating version 0 to 126 2023-01-15T14:29:14.672 INFO:tasks.rados.rados.0.smithi090.stdout:263: oid 17 updating version 126 to 128 2023-01-15T14:29:14.672 INFO:tasks.rados.rados.0.smithi090.stdout:263: oid 17 version 128 is already newer than 127 2023-01-15T14:29:14.672 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 128 (ObjNum 124 snap 23 seq_num 124) dirty exists 2023-01-15T14:29:14.672 INFO:tasks.rados.rados.0.smithi090.stdout:263: left oid 17 (ObjNum 124 snap 23 seq_num 124) 2023-01-15T14:29:14.672 INFO:tasks.rados.rados.0.smithi090.stdout:256: finishing write tid 2 to smithi09025264-4 2023-01-15T14:29:14.673 INFO:tasks.rados.rados.0.smithi090.stdout:256: finishing write tid 3 to smithi09025264-4 2023-01-15T14:29:14.673 INFO:tasks.rados.rados.0.smithi090.stdout:256: finishing write tid 4 to smithi09025264-4 2023-01-15T14:29:14.673 INFO:tasks.rados.rados.0.smithi090.stdout:256: oid 4 updating version 0 to 52 2023-01-15T14:29:14.673 INFO:tasks.rados.rados.0.smithi090.stdout:256: oid 4 updating version 52 to 54 2023-01-15T14:29:14.673 INFO:tasks.rados.rados.0.smithi090.stdout:256: oid 4 version 54 is already newer than 53 2023-01-15T14:29:14.674 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 4 v 54 (ObjNum 120 snap 23 seq_num 120) dirty exists 2023-01-15T14:29:14.674 INFO:tasks.rados.rados.0.smithi090.stdout:256: left oid 4 (ObjNum 120 snap 23 seq_num 120) 2023-01-15T14:29:14.674 INFO:tasks.rados.rados.0.smithi090.stdout:269: finishing write tid 1 to smithi09025264-15 2023-01-15T14:29:14.674 INFO:tasks.rados.rados.0.smithi090.stdout:264: expect (ObjNum 114 snap 23 seq_num 114) 2023-01-15T14:29:14.677 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:14.677 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:14.678 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:29:14.679 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:14.679 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:14.679 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:14.679 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:14.679 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:14.679 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:14.679 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:29:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:29:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:29:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:29:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:29:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:29:14.681 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:29:14.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:29:14.682 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:14.669+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:14.741 INFO:tasks.rados.rados.0.smithi090.stdout:270: finishing write tid 2 to smithi09025264-22 2023-01-15T14:29:14.878 INFO:tasks.rados.rados.0.smithi090.stdout:270: finishing write tid 3 to smithi09025264-22 2023-01-15T14:29:14.878 INFO:tasks.rados.rados.0.smithi090.stdout:270: finishing write tid 4 to smithi09025264-22 2023-01-15T14:29:14.878 INFO:tasks.rados.rados.0.smithi090.stdout:270: oid 22 updating version 0 to 85 2023-01-15T14:29:14.880 INFO:tasks.rados.rados.0.smithi090.stdout:270: oid 22 updating version 85 to 86 2023-01-15T14:29:14.880 INFO:tasks.rados.rados.0.smithi090.stdout:270: oid 22 version 86 is already newer than 84 2023-01-15T14:29:14.881 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 86 (ObjNum 129 snap 23 seq_num 129) dirty exists 2023-01-15T14:29:14.881 INFO:tasks.rados.rados.0.smithi090.stdout:270: left oid 22 (ObjNum 129 snap 23 seq_num 129) 2023-01-15T14:29:14.881 INFO:tasks.rados.rados.0.smithi090.stdout:268: finishing write tid 1 to smithi09025264-33 2023-01-15T14:29:14.883 INFO:tasks.rados.rados.0.smithi090.stdout:267: finishing write tid 1 to smithi09025264-5 2023-01-15T14:29:14.883 INFO:tasks.rados.rados.0.smithi090.stdout:267: finishing write tid 2 to smithi09025264-5 2023-01-15T14:29:14.883 INFO:tasks.rados.rados.0.smithi090.stdout:267: finishing write tid 3 to smithi09025264-5 2023-01-15T14:29:14.884 INFO:tasks.rados.rados.0.smithi090.stdout:267: finishing write tid 4 to smithi09025264-5 2023-01-15T14:29:14.884 INFO:tasks.rados.rados.0.smithi090.stdout:267: oid 5 updating version 0 to 83 2023-01-15T14:29:14.884 INFO:tasks.rados.rados.0.smithi090.stdout:267: oid 5 updating version 83 to 85 2023-01-15T14:29:14.884 INFO:tasks.rados.rados.0.smithi090.stdout:267: oid 5 version 85 is already newer than 84 2023-01-15T14:29:14.884 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 85 (ObjNum 126 snap 23 seq_num 126) dirty exists 2023-01-15T14:29:14.885 INFO:tasks.rados.rados.0.smithi090.stdout:267: left oid 5 (ObjNum 126 snap 23 seq_num 126) 2023-01-15T14:29:14.885 INFO:tasks.rados.rados.0.smithi090.stdout:269: finishing write tid 2 to smithi09025264-15 2023-01-15T14:29:14.885 INFO:tasks.rados.rados.0.smithi090.stdout:268: finishing write tid 2 to smithi09025264-33 2023-01-15T14:29:14.885 INFO:tasks.rados.rados.0.smithi090.stdout:269: finishing write tid 3 to smithi09025264-15 2023-01-15T14:29:14.885 INFO:tasks.rados.rados.0.smithi090.stdout:266: finishing write tid 1 to smithi09025264-20 2023-01-15T14:29:14.886 INFO:tasks.rados.rados.0.smithi090.stdout:266: finishing write tid 2 to smithi09025264-20 2023-01-15T14:29:14.886 INFO:tasks.rados.rados.0.smithi090.stdout:266: finishing write tid 3 to smithi09025264-20 2023-01-15T14:29:14.886 INFO:tasks.rados.rados.0.smithi090.stdout:268: finishing write tid 3 to smithi09025264-33 2023-01-15T14:29:14.886 INFO:tasks.rados.rados.0.smithi090.stdout:266: finishing write tid 4 to smithi09025264-20 2023-01-15T14:29:14.886 INFO:tasks.rados.rados.0.smithi090.stdout:266: oid 20 updating version 0 to 29 2023-01-15T14:29:14.887 INFO:tasks.rados.rados.0.smithi090.stdout:266: oid 20 updating version 29 to 30 2023-01-15T14:29:14.887 INFO:tasks.rados.rados.0.smithi090.stdout:266: oid 20 updating version 30 to 31 2023-01-15T14:29:14.887 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 31 (ObjNum 125 snap 23 seq_num 125) dirty exists 2023-01-15T14:29:14.887 INFO:tasks.rados.rados.0.smithi090.stdout:266: left oid 20 (ObjNum 125 snap 23 seq_num 125) 2023-01-15T14:29:14.888 INFO:tasks.rados.rados.0.smithi090.stdout:269: finishing write tid 4 to smithi09025264-15 2023-01-15T14:29:14.889 INFO:tasks.rados.rados.0.smithi090.stdout:269: oid 15 updating version 0 to 57 2023-01-15T14:29:14.889 INFO:tasks.rados.rados.0.smithi090.stdout:269: oid 15 updating version 57 to 58 2023-01-15T14:29:14.889 INFO:tasks.rados.rados.0.smithi090.stdout:269: oid 15 updating version 58 to 59 2023-01-15T14:29:14.890 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 59 (ObjNum 128 snap 23 seq_num 128) dirty exists 2023-01-15T14:29:14.890 INFO:tasks.rados.rados.0.smithi090.stdout:269: left oid 15 (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:29:14.890 INFO:tasks.rados.rados.0.smithi090.stdout:256: done (8 left) 2023-01-15T14:29:14.891 INFO:tasks.rados.rados.0.smithi090.stdout:263: done (7 left) 2023-01-15T14:29:14.891 INFO:tasks.rados.rados.0.smithi090.stdout:264: done (6 left) 2023-01-15T14:29:15.053 INFO:tasks.rados.rados.0.smithi090.stdout:266: done (5 left) 2023-01-15T14:29:15.054 INFO:tasks.rados.rados.0.smithi090.stdout:267: done (4 left) 2023-01-15T14:29:15.054 INFO:tasks.rados.rados.0.smithi090.stdout:269: done (3 left) 2023-01-15T14:29:15.054 INFO:tasks.rados.rados.0.smithi090.stdout:270: done (2 left) 2023-01-15T14:29:15.054 INFO:tasks.rados.rados.0.smithi090.stdout:271: done (1 left) 2023-01-15T14:29:15.054 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:15.057 INFO:tasks.rados.rados.0.smithi090.stdout:268: finishing write tid 4 to smithi09025264-33 2023-01-15T14:29:15.058 INFO:tasks.rados.rados.0.smithi090.stdout:268: oid 33 updating version 0 to 43 2023-01-15T14:29:15.058 INFO:tasks.rados.rados.0.smithi090.stdout:268: oid 33 updating version 43 to 45 2023-01-15T14:29:15.058 INFO:tasks.rados.rados.0.smithi090.stdout:268: oid 33 version 45 is already newer than 44 2023-01-15T14:29:15.058 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 45 (ObjNum 127 snap 23 seq_num 127) dirty exists 2023-01-15T14:29:15.058 INFO:tasks.rados.rados.0.smithi090.stdout:268: left oid 33 (ObjNum 127 snap 23 seq_num 127) 2023-01-15T14:29:15.058 INFO:tasks.rados.rados.0.smithi090.stdout:268: done (0 left) 2023-01-15T14:29:15.058 INFO:tasks.rados.rados.0.smithi090.stdout:272: done (0 left) 2023-01-15T14:29:15.058 INFO:tasks.rados.rados.0.smithi090.stdout:273: snap_remove snap 22 2023-01-15T14:29:15.059 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:15.059 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:29:15.059 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:29:15.059 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:29:15.059 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:15.060 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:29:15.061 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:15.049+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:15.061 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:29:15.157 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:15.153+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:15.244 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:29:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:29:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:29:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:12 3.a 3:52c3e133:::smithi09025264-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.799749+0000", 2023-01-15T14:29:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 49.373573622000002, 2023-01-15T14:29:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30375770000000002, 2023-01-15T14:29:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 12 2023-01-15T14:29:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799749+0000", 2023-01-15T14:29:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799749+0000", 2023-01-15T14:29:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799748+0000", 2023-01-15T14:29:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799755+0000", 2023-01-15T14:29:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1389999999999998e-06 2023-01-15T14:29:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.250 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:15.250 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.799772+0000", 2023-01-15T14:29:15.250 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6827e-05 2023-01-15T14:29:15.250 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.250 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.250 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:15.250 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.899902+0000", 2023-01-15T14:29:15.250 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100129729 2023-01-15T14:29:15.250 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.250 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.251 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:15.251 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.998797+0000", 2023-01-15T14:29:15.251 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098894686999999995 2023-01-15T14:29:15.251 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.251 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.251 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:15.251 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.998954+0000", 2023-01-15T14:29:15.251 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015711699999999999 2023-01-15T14:29:15.251 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.251 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.251 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:15.252 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103084+0000", 2023-01-15T14:29:15.252 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104129765 2023-01-15T14:29:15.252 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.252 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.252 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:15.252 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103276+0000", 2023-01-15T14:29:15.252 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019277899999999999 2023-01-15T14:29:15.252 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.252 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.252 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:15.252 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103507+0000", 2023-01-15T14:29:15.253 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023087699999999999 2023-01-15T14:29:15.253 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:15.253 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:15.253 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:15.253 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.253 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.253 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:97 3.a 3:52df5138:::smithi09025264-25:head [write 403368~555035,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:15.257 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:29:15.258 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 48.672180241, 2023-01-15T14:29:15.258 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.340758171, 2023-01-15T14:29:15.258 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:15.258 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:15.258 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:15.258 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:15.258 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:15.258 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 97 2023-01-15T14:29:15.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.258 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:15.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:15.259 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:29:15.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:15.259 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:29:15.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:15.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501140+0000", 2023-01-15T14:29:15.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:29:15.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:15.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501684+0000", 2023-01-15T14:29:15.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054441799999999996 2023-01-15T14:29:15.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:15.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501713+0000", 2023-01-15T14:29:15.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8833e-05 2023-01-15T14:29:15.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:15.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501726+0000", 2023-01-15T14:29:15.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3137e-05 2023-01-15T14:29:15.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:15.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601982+0000", 2023-01-15T14:29:15.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100256518 2023-01-15T14:29:15.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:15.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602426+0000", 2023-01-15T14:29:15.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044401500000000002 2023-01-15T14:29:15.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:15.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602703+0000", 2023-01-15T14:29:15.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000276889 2023-01-15T14:29:15.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:15.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603066+0000", 2023-01-15T14:29:15.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036278599999999998 2023-01-15T14:29:15.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:15.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603098+0000", 2023-01-15T14:29:15.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1433000000000002e-05 2023-01-15T14:29:15.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:15.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603382+0000", 2023-01-15T14:29:15.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028402899999999999 2023-01-15T14:29:15.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603632+0000", 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025033 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805354+0000", 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20172168700000001 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841465+0000", 2023-01-15T14:29:15.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.036111852999999999 2023-01-15T14:29:15.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:15.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841682+0000", 2023-01-15T14:29:15.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000216949 2023-01-15T14:29:15.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:15.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841901+0000", 2023-01-15T14:29:15.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021858700000000001 2023-01-15T14:29:15.529 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:15.529 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:15.529 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:15.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.529 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:98 3.a 3:52df5138:::smithi09025264-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:15.530 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:29:15.530 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 48.67157057, 2023-01-15T14:29:15.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.44410839299999999, 2023-01-15T14:29:15.530 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:15.530 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:15.530 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:15.530 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:15.530 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:15.530 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 98 2023-01-15T14:29:15.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.530 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:15.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501752+0000", 2023-01-15T14:29:15.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:15.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:15.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502223+0000", 2023-01-15T14:29:15.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000471843 2023-01-15T14:29:15.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502240+0000", 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6663e-05 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502248+0000", 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5669999999999997e-06 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603236+0000", 2023-01-15T14:29:15.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100988439 2023-01-15T14:29:15.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:15.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603384+0000", 2023-01-15T14:29:15.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014844200000000001 2023-01-15T14:29:15.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:15.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842016+0000", 2023-01-15T14:29:15.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23863163800000001 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842267+0000", 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025101300000000002 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842561+0000", 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029432900000000002 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:15.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843120+0000", 2023-01-15T14:29:15.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055900500000000001 2023-01-15T14:29:15.536 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.536 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:15.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843146+0000", 2023-01-15T14:29:15.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5273e-05 2023-01-15T14:29:15.536 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.536 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:15.537 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843512+0000", 2023-01-15T14:29:15.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036621599999999999 2023-01-15T14:29:15.537 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.537 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.537 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:15.537 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943324+0000", 2023-01-15T14:29:15.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099812008999999993 2023-01-15T14:29:15.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945437+0000", 2023-01-15T14:29:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021126069999999999 2023-01-15T14:29:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945619+0000", 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018280600000000001 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945744+0000", 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012511299999999999 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945861+0000", 2023-01-15T14:29:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011646000000000001 2023-01-15T14:29:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:99 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:29:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 48.671060314000002, 2023-01-15T14:29:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.54355587900000002, 2023-01-15T14:29:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 99 2023-01-15T14:29:15.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.544 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:15.693 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.693 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:15.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:29:15.693 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502262+0000", 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:15.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502270+0000", 2023-01-15T14:29:15.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1999999999999994e-06 2023-01-15T14:29:15.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:15.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502286+0000", 2023-01-15T14:29:15.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5423000000000001e-05 2023-01-15T14:29:15.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:15.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502295+0000", 2023-01-15T14:29:15.697 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4339999999999998e-06 2023-01-15T14:29:15.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.697 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:15.697 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603467+0000", 2023-01-15T14:29:15.697 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101172225 2023-01-15T14:29:15.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.697 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:15.700 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603585+0000", 2023-01-15T14:29:15.700 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117149 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843335+0000", 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23975083799999999 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843573+0000", 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023753000000000001 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.701 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.702 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:15.702 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843978+0000", 2023-01-15T14:29:15.702 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040533799999999999 2023-01-15T14:29:15.702 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.702 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.702 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:15.702 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844309+0000", 2023-01-15T14:29:15.702 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000330713 2023-01-15T14:29:15.702 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.702 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.702 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:15.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844423+0000", 2023-01-15T14:29:15.703 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114176 2023-01-15T14:29:15.703 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.703 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.703 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:15.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844510+0000", 2023-01-15T14:29:15.703 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7026000000000003e-05 2023-01-15T14:29:15.703 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.703 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.703 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:15.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943338+0000", 2023-01-15T14:29:15.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098827398999999996 2023-01-15T14:29:15.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.704 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:15.704 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945520+0000", 2023-01-15T14:29:15.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021824169999999999 2023-01-15T14:29:15.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.704 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:15.704 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045652+0000", 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013193200000001 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045733+0000", 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1435999999999998e-05 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045819+0000", 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5292999999999998e-05 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:15.705 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:100 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 48.671012390999998, 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.64425290599999996, 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:15.706 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 100 2023-01-15T14:29:15.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.707 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:15.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:15.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:29:15.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:15.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:29:15.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:15.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502310+0000", 2023-01-15T14:29:15.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:15.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:15.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502316+0000", 2023-01-15T14:29:15.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7470000000000002e-06 2023-01-15T14:29:15.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:15.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502332+0000", 2023-01-15T14:29:15.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6866000000000001e-05 2023-01-15T14:29:15.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:15.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502343+0000", 2023-01-15T14:29:15.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0913e-05 2023-01-15T14:29:15.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:15.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703594+0000", 2023-01-15T14:29:15.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20125087999999999 2023-01-15T14:29:15.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:15.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703751+0000", 2023-01-15T14:29:15.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001573 2023-01-15T14:29:15.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:15.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845262+0000", 2023-01-15T14:29:15.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.141510521 2023-01-15T14:29:15.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:15.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845391+0000", 2023-01-15T14:29:15.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012872299999999999 2023-01-15T14:29:15.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:15.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146051+0000", 2023-01-15T14:29:15.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.300660023 2023-01-15T14:29:15.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:15.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146264+0000", 2023-01-15T14:29:15.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212754 2023-01-15T14:29:15.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:15.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146564+0000", 2023-01-15T14:29:15.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030010200000000001 2023-01-15T14:29:15.920 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:15.921 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:15.921 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:15.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.922 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:203 3.a 3:52df5138:::smithi09025264-25:head [write 440350~460568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:15.922 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:29:15.923 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 47.620208996000002, 2023-01-15T14:29:15.924 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40523678899999999, 2023-01-15T14:29:15.925 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:15.925 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:15.925 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:15.925 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:15.925 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:15.925 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 203 2023-01-15T14:29:15.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.925 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:15.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.925 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:15.925 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:29:15.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:15.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:29:15.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:15.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:15.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553111+0000", 2023-01-15T14:29:15.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:15.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:15.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553483+0000", 2023-01-15T14:29:15.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037153499999999999 2023-01-15T14:29:15.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:15.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553517+0000", 2023-01-15T14:29:15.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4687000000000002e-05 2023-01-15T14:29:15.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553537+0000", 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.967e-05 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553683+0000", 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000146341 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:15.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553880+0000", 2023-01-15T14:29:15.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019671900000000001 2023-01-15T14:29:15.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:15.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554219+0000", 2023-01-15T14:29:15.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033893299999999998 2023-01-15T14:29:15.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:15.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554536+0000", 2023-01-15T14:29:15.930 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316576 2023-01-15T14:29:15.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.930 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:15.930 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554562+0000", 2023-01-15T14:29:15.930 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5905999999999999e-05 2023-01-15T14:29:15.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.930 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:15.930 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554801+0000", 2023-01-15T14:29:15.931 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000239791 2023-01-15T14:29:15.931 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.931 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:15.931 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:15.931 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555274+0000", 2023-01-15T14:29:15.931 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047294500000000001 2023-01-15T14:29:15.931 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:15.931 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.109 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757267+0000", 2023-01-15T14:29:16.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20199229399999999 2023-01-15T14:29:16.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958174+0000", 2023-01-15T14:29:16.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20090739499999999 2023-01-15T14:29:16.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:16.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958260+0000", 2023-01-15T14:29:16.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6316999999999994e-05 2023-01-15T14:29:16.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958351+0000", 2023-01-15T14:29:16.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0573000000000002e-05 2023-01-15T14:29:16.111 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.111 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.111 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.111 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:204 3.a 3:52df5138:::smithi09025264-25:head [write 1309629~542666] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:16.111 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:29:16.112 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 47.619746128000003, 2023-01-15T14:29:16.112 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40500913999999999, 2023-01-15T14:29:16.112 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.112 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:16.112 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.112 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.112 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.112 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 204 2023-01-15T14:29:16.112 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.112 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.112 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.113 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:29:16.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.116 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.116 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:29:16.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.116 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553575+0000", 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554126+0000", 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055090499999999997 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.118 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554150+0000", 2023-01-15T14:29:16.118 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3387000000000002e-05 2023-01-15T14:29:16.118 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.118 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.118 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.118 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554165+0000", 2023-01-15T14:29:16.118 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5116000000000001e-05 2023-01-15T14:29:16.118 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.118 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.118 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.119 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554626+0000", 2023-01-15T14:29:16.119 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046093499999999999 2023-01-15T14:29:16.119 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.119 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.119 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.119 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554738+0000", 2023-01-15T14:29:16.119 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112446 2023-01-15T14:29:16.119 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.119 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.119 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:16.119 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554898+0000", 2023-01-15T14:29:16.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016003100000000001 2023-01-15T14:29:16.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:16.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555182+0000", 2023-01-15T14:29:16.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028382700000000001 2023-01-15T14:29:16.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:16.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555199+0000", 2023-01-15T14:29:16.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6969e-05 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555449+0000", 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000250265 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555574+0000", 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124628 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757316+0000", 2023-01-15T14:29:16.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20174214300000001 2023-01-15T14:29:16.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958460+0000", 2023-01-15T14:29:16.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201144464 2023-01-15T14:29:16.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:16.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958534+0000", 2023-01-15T14:29:16.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3380000000000006e-05 2023-01-15T14:29:16.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.126 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.126 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958586+0000", 2023-01-15T14:29:16.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2370000000000002e-05 2023-01-15T14:29:16.126 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.126 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.126 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.126 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:205 3.a 3:52df5138:::smithi09025264-25:head [write 2323973~516377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:16.127 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:29:16.127 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 47.619128033000003, 2023-01-15T14:29:16.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404601398, 2023-01-15T14:29:16.127 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.127 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:16.127 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.127 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.128 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.128 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 205 2023-01-15T14:29:16.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.128 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:29:16.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554193+0000", 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554850+0000", 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065674799999999997 2023-01-15T14:29:16.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554873+0000", 2023-01-15T14:29:16.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2935999999999999e-05 2023-01-15T14:29:16.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.654962+0000", 2023-01-15T14:29:16.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088949 2023-01-15T14:29:16.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655107+0000", 2023-01-15T14:29:16.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144759 2023-01-15T14:29:16.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655373+0000", 2023-01-15T14:29:16.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026684899999999999 2023-01-15T14:29:16.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:16.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655720+0000", 2023-01-15T14:29:16.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034601899999999998 2023-01-15T14:29:16.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:16.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656168+0000", 2023-01-15T14:29:16.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044851899999999998 2023-01-15T14:29:16.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:16.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656190+0000", 2023-01-15T14:29:16.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1817000000000001e-05 2023-01-15T14:29:16.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:16.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656574+0000", 2023-01-15T14:29:16.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038404200000000002 2023-01-15T14:29:16.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:16.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657649+0000", 2023-01-15T14:29:16.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010746670000000001 2023-01-15T14:29:16.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757539+0000", 2023-01-15T14:29:16.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099890899000000005 2023-01-15T14:29:16.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958683+0000", 2023-01-15T14:29:16.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20114332400000001 2023-01-15T14:29:16.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958746+0000", 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3509000000000005e-05 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958796+0000", 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0176999999999997e-05 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.292 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:206 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 2840350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:16.292 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.655000+0000", 2023-01-15T14:29:16.292 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 47.518323223000003, 2023-01-15T14:29:16.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30401123800000002, 2023-01-15T14:29:16.293 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.293 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:16.293 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.293 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.293 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.293 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 206 2023-01-15T14:29:16.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.293 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.294 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655000+0000", 2023-01-15T14:29:16.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.294 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.294 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.294 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.294 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655000+0000", 2023-01-15T14:29:16.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.294 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.294 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.296 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.654998+0000", 2023-01-15T14:29:16.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:16.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655010+0000", 2023-01-15T14:29:16.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2176e-05 2023-01-15T14:29:16.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655029+0000", 2023-01-15T14:29:16.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.84e-05 2023-01-15T14:29:16.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655039+0000", 2023-01-15T14:29:16.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0332999999999999e-05 2023-01-15T14:29:16.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656357+0000", 2023-01-15T14:29:16.300 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0013186370000000001 2023-01-15T14:29:16.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.300 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656639+0000", 2023-01-15T14:29:16.300 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028126900000000001 2023-01-15T14:29:16.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.300 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:16.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656956+0000", 2023-01-15T14:29:16.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00031706600000000002 2023-01-15T14:29:16.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:16.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657312+0000", 2023-01-15T14:29:16.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035614199999999999 2023-01-15T14:29:16.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:16.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657335+0000", 2023-01-15T14:29:16.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2997e-05 2023-01-15T14:29:16.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:16.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657417+0000", 2023-01-15T14:29:16.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2311999999999994e-05 2023-01-15T14:29:16.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:16.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657663+0000", 2023-01-15T14:29:16.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024533299999999998 2023-01-15T14:29:16.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.797539+0000", 2023-01-15T14:29:16.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.13987612499999999 2023-01-15T14:29:16.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.304 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958869+0000", 2023-01-15T14:29:16.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.161330061 2023-01-15T14:29:16.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:16.304 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958935+0000", 2023-01-15T14:29:16.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6390000000000006e-05 2023-01-15T14:29:16.305 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.305 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.305 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.959011+0000", 2023-01-15T14:29:16.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5946999999999996e-05 2023-01-15T14:29:16.305 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.305 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.305 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.305 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.306 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:207 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:16.306 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.655275+0000", 2023-01-15T14:29:16.306 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 47.518048333999999, 2023-01-15T14:29:16.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30407898100000003, 2023-01-15T14:29:16.307 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.307 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:16.307 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.307 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.307 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.307 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 207 2023-01-15T14:29:16.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.307 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655275+0000", 2023-01-15T14:29:16.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655275+0000", 2023-01-15T14:29:16.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655273+0000", 2023-01-15T14:29:16.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:16.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.455 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655285+0000", 2023-01-15T14:29:16.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1392999999999999e-05 2023-01-15T14:29:16.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.455 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.456 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655317+0000", 2023-01-15T14:29:16.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2762999999999998e-05 2023-01-15T14:29:16.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655329+0000", 2023-01-15T14:29:16.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1717e-05 2023-01-15T14:29:16.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657484+0000", 2023-01-15T14:29:16.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00215496 2023-01-15T14:29:16.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:16.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657615+0000", 2023-01-15T14:29:16.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013074699999999999 2023-01-15T14:29:16.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.959096+0000", 2023-01-15T14:29:16.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30148098400000001 2023-01-15T14:29:16.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.959222+0000", 2023-01-15T14:29:16.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012578000000000001 2023-01-15T14:29:16.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.959354+0000", 2023-01-15T14:29:16.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013212 2023-01-15T14:29:16.460 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.460 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.460 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.461 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:358 3.a 3:52df5138:::smithi09025264-25:head [write 709262~697258] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:16.461 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:29:16.461 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 35.122540499999999, 2023-01-15T14:29:16.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50589529799999999, 2023-01-15T14:29:16.461 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.461 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:16.462 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.462 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.462 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.462 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 358 2023-01-15T14:29:16.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.462 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050780+0000", 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:16.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.464 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051369+0000", 2023-01-15T14:29:16.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058926200000000001 2023-01-15T14:29:16.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.464 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051390+0000", 2023-01-15T14:29:16.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1426000000000001e-05 2023-01-15T14:29:16.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051403+0000", 2023-01-15T14:29:16.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:29:16.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051520+0000", 2023-01-15T14:29:16.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117318 2023-01-15T14:29:16.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051785+0000", 2023-01-15T14:29:16.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000265018 2023-01-15T14:29:16.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:16.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052173+0000", 2023-01-15T14:29:16.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038734799999999998 2023-01-15T14:29:16.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:16.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052705+0000", 2023-01-15T14:29:16.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053209899999999996 2023-01-15T14:29:16.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:16.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052727+0000", 2023-01-15T14:29:16.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2362e-05 2023-01-15T14:29:16.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:16.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053205+0000", 2023-01-15T14:29:16.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047787200000000001 2023-01-15T14:29:16.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:16.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055711+0000", 2023-01-15T14:29:16.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0025054880000000002 2023-01-15T14:29:16.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.255549+0000", 2023-01-15T14:29:16.471 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199838285 2023-01-15T14:29:16.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556501+0000", 2023-01-15T14:29:16.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30095197499999998 2023-01-15T14:29:16.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:16.472 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556613+0000", 2023-01-15T14:29:16.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111799 2023-01-15T14:29:16.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556678+0000", 2023-01-15T14:29:16.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5220000000000002e-05 2023-01-15T14:29:16.473 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.473 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.473 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.474 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:359 3.a 3:52df5138:::smithi09025264-25:head [write 2012635~572018] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:16.474 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:29:16.474 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 35.121889906, 2023-01-15T14:29:16.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50546442499999999, 2023-01-15T14:29:16.474 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.474 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:16.474 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.475 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.475 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.475 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 359 2023-01-15T14:29:16.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.475 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:29:16.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:29:16.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051432+0000", 2023-01-15T14:29:16.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:16.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051808+0000", 2023-01-15T14:29:16.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000376074 2023-01-15T14:29:16.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051834+0000", 2023-01-15T14:29:16.478 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6018999999999999e-05 2023-01-15T14:29:16.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051869+0000", 2023-01-15T14:29:16.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4839999999999998e-05 2023-01-15T14:29:16.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052857+0000", 2023-01-15T14:29:16.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00098761399999999994 2023-01-15T14:29:16.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053071+0000", 2023-01-15T14:29:16.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000214112 2023-01-15T14:29:16.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:16.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053876+0000", 2023-01-15T14:29:16.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000804873 2023-01-15T14:29:16.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:16.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054460+0000", 2023-01-15T14:29:16.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058436100000000004 2023-01-15T14:29:16.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:16.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054486+0000", 2023-01-15T14:29:16.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069999999999999e-05 2023-01-15T14:29:16.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:16.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054773+0000", 2023-01-15T14:29:16.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028716700000000001 2023-01-15T14:29:16.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:16.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055736+0000", 2023-01-15T14:29:16.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00096295500000000004 2023-01-15T14:29:16.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.355784+0000", 2023-01-15T14:29:16.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30004825899999998 2023-01-15T14:29:16.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556757+0000", 2023-01-15T14:29:16.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20097241199999999 2023-01-15T14:29:16.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:16.624 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556830+0000", 2023-01-15T14:29:16.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3633000000000005e-05 2023-01-15T14:29:16.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556898+0000", 2023-01-15T14:29:16.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7212999999999995e-05 2023-01-15T14:29:16.625 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.626 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.626 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.627 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:360 3.a 3:52df5138:::smithi09025264-25:head [write 3046388~62874] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:16.627 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:29:16.627 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 35.121437149999998, 2023-01-15T14:29:16.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50519877000000002, 2023-01-15T14:29:16.627 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.628 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:16.628 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.628 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.628 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.629 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 360 2023-01-15T14:29:16.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.629 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:29:16.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:29:16.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051885+0000", 2023-01-15T14:29:16.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:16.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051935+0000", 2023-01-15T14:29:16.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9826e-05 2023-01-15T14:29:16.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051958+0000", 2023-01-15T14:29:16.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3003e-05 2023-01-15T14:29:16.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051965+0000", 2023-01-15T14:29:16.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6129999999999998e-06 2023-01-15T14:29:16.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.637 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.637 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054655+0000", 2023-01-15T14:29:16.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0026896860000000002 2023-01-15T14:29:16.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054869+0000", 2023-01-15T14:29:16.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021348600000000001 2023-01-15T14:29:16.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:16.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055198+0000", 2023-01-15T14:29:16.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032968199999999998 2023-01-15T14:29:16.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:16.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055605+0000", 2023-01-15T14:29:16.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040634200000000002 2023-01-15T14:29:16.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.641 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:16.641 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055622+0000", 2023-01-15T14:29:16.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7020999999999998e-05 2023-01-15T14:29:16.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.642 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:16.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055702+0000", 2023-01-15T14:29:16.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0835999999999997e-05 2023-01-15T14:29:16.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:16.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055754+0000", 2023-01-15T14:29:16.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1175999999999999e-05 2023-01-15T14:29:16.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.455996+0000", 2023-01-15T14:29:16.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40024198599999999 2023-01-15T14:29:16.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556952+0000", 2023-01-15T14:29:16.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10095644400000001 2023-01-15T14:29:16.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:16.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557019+0000", 2023-01-15T14:29:16.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6746000000000001e-05 2023-01-15T14:29:16.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557085+0000", 2023-01-15T14:29:16.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5885999999999999e-05 2023-01-15T14:29:16.647 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.647 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.647 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.792 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:361 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 3109262] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:16.792 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:29:16.792 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 35.121347243999999, 2023-01-15T14:29:16.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60469846000000005, 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 361 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:29:16.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:29:16.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051975+0000", 2023-01-15T14:29:16.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:16.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051980+0000", 2023-01-15T14:29:16.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2170000000000002e-06 2023-01-15T14:29:16.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051989+0000", 2023-01-15T14:29:16.796 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5299999999999996e-06 2023-01-15T14:29:16.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.797 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.797 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051995+0000", 2023-01-15T14:29:16.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1369999999999999e-06 2023-01-15T14:29:16.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055830+0000", 2023-01-15T14:29:16.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0038348549999999999 2023-01-15T14:29:16.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.799 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.799 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056101+0000", 2023-01-15T14:29:16.799 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027095800000000002 2023-01-15T14:29:16.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.800 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.800 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:16.800 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056560+0000", 2023-01-15T14:29:16.800 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045868599999999998 2023-01-15T14:29:16.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.800 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:16.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056954+0000", 2023-01-15T14:29:16.801 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039487600000000001 2023-01-15T14:29:16.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:16.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057004+0000", 2023-01-15T14:29:16.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9490000000000002e-05 2023-01-15T14:29:16.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:16.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057072+0000", 2023-01-15T14:29:16.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8502000000000002e-05 2023-01-15T14:29:16.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:16.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.158224+0000", 2023-01-15T14:29:16.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101151422 2023-01-15T14:29:16.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556201+0000", 2023-01-15T14:29:16.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39797714699999998 2023-01-15T14:29:16.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656421+0000", 2023-01-15T14:29:16.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100219549 2023-01-15T14:29:16.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:16.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656515+0000", 2023-01-15T14:29:16.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4048999999999993e-05 2023-01-15T14:29:16.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656674+0000", 2023-01-15T14:29:16.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000159696 2023-01-15T14:29:16.809 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.809 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.809 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.809 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:362 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:16.810 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:29:16.810 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 35.121319814000003, 2023-01-15T14:29:16.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60517504499999997, 2023-01-15T14:29:16.810 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.810 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:16.811 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.811 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.811 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.811 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 362 2023-01-15T14:29:16.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.811 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:29:16.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:29:16.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:29:16.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:16.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052007+0000", 2023-01-15T14:29:16.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.9759999999999997e-06 2023-01-15T14:29:16.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052016+0000", 2023-01-15T14:29:16.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4069999999999999e-06 2023-01-15T14:29:16.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052022+0000", 2023-01-15T14:29:16.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4670000000000001e-06 2023-01-15T14:29:16.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157159+0000", 2023-01-15T14:29:16.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.105136825 2023-01-15T14:29:16.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:16.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157325+0000", 2023-01-15T14:29:16.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000165262 2023-01-15T14:29:16.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656814+0000", 2023-01-15T14:29:16.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49948963200000002 2023-01-15T14:29:16.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656993+0000", 2023-01-15T14:29:16.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000178371 2023-01-15T14:29:16.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.657178+0000", 2023-01-15T14:29:16.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185825 2023-01-15T14:29:16.827 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.827 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.827 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.828 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:483 3.a 3:52df5138:::smithi09025264-25:head [copy-from ver 43] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2023-01-15T14:29:16.828 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:48.786893+0000", 2023-01-15T14:29:16.828 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 26.386430456999999, 2023-01-15T14:29:16.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31520366399999999, 2023-01-15T14:29:16.829 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.829 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:16.829 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.830 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.830 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.830 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 483 2023-01-15T14:29:16.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.831 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.831 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.831 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:48.786893+0000", 2023-01-15T14:29:16.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:48.786893+0000", 2023-01-15T14:29:16.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:48.786890+0000", 2023-01-15T14:29:16.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:16.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:48.786902+0000", 2023-01-15T14:29:16.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2493e-05 2023-01-15T14:29:16.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:48.786928+0000", 2023-01-15T14:29:16.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5962e-05 2023-01-15T14:29:16.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.836 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:48.887086+0000", 2023-01-15T14:29:16.836 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100157592 2023-01-15T14:29:16.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:48.887258+0000", 2023-01-15T14:29:16.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000171955 2023-01-15T14:29:16.957 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.957 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.957 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.957 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:48.887436+0000", 2023-01-15T14:29:16.957 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017777799999999999 2023-01-15T14:29:16.958 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.958 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.958 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:16.958 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:48.993782+0000", 2023-01-15T14:29:16.959 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.106346519 2023-01-15T14:29:16.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.959 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.959 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:48.999025+0000", 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0052426130000000001 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:48.999054+0000", 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8949999999999999e-05 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.000159+0000", 2023-01-15T14:29:16.962 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011055800000000001 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.099664+0000", 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099505311999999999 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.099835+0000", 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017004699999999999 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.964 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.101818+0000", 2023-01-15T14:29:16.964 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0019834800000000001 2023-01-15T14:29:16.964 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.964 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.964 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:16.964 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.101952+0000", 2023-01-15T14:29:16.964 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013358200000000001 2023-01-15T14:29:16.964 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.964 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.965 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.102096+0000", 2023-01-15T14:29:16.965 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014470499999999999 2023-01-15T14:29:16.965 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.965 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.965 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.965 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.965 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:599 3.a 3:5ba385f1:::smithi09025264-44:head [write 1593482~658027] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:29:16.965 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:59.835163+0000", 2023-01-15T14:29:16.965 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 15.338160456000001, 2023-01-15T14:29:16.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31095880199999998, 2023-01-15T14:29:16.966 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.966 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:16.966 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.966 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.966 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.966 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 599 2023-01-15T14:29:16.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.966 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.966 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.966 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.835163+0000", 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.835163+0000", 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.835160+0000", 2023-01-15T14:29:16.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:16.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.968 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.835850+0000", 2023-01-15T14:29:16.968 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00068990000000000002 2023-01-15T14:29:16.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.968 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.835889+0000", 2023-01-15T14:29:16.968 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.9759000000000003e-05 2023-01-15T14:29:16.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.835909+0000", 2023-01-15T14:29:16.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9313999999999998e-05 2023-01-15T14:29:16.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.839265+0000", 2023-01-15T14:29:16.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0033562689999999998 2023-01-15T14:29:16.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:16.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.839377+0000", 2023-01-15T14:29:16.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111711 2023-01-15T14:29:16.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.940451+0000", 2023-01-15T14:29:16.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10107456400000001 2023-01-15T14:29:16.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.971 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.940653+0000", 2023-01-15T14:29:16.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000201807 2023-01-15T14:29:16.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.971 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:16.971 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.940936+0000", 2023-01-15T14:29:16.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028313500000000001 2023-01-15T14:29:16.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.972 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:16.972 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.941423+0000", 2023-01-15T14:29:16.972 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048652299999999999 2023-01-15T14:29:16.972 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.972 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:16.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.941446+0000", 2023-01-15T14:29:16.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:29:16.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:16.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.943191+0000", 2023-01-15T14:29:16.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001744867 2023-01-15T14:29:16.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:16.974 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.943382+0000", 2023-01-15T14:29:16.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000191034 2023-01-15T14:29:16.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.974 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.974 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.974 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.044484+0000", 2023-01-15T14:29:16.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101101948 2023-01-15T14:29:16.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.974 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.975 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.975 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145850+0000", 2023-01-15T14:29:16.975 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101365749 2023-01-15T14:29:16.975 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.975 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.975 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:16.975 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145992+0000", 2023-01-15T14:29:16.975 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000142144 2023-01-15T14:29:16.975 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.975 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.975 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.976 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146121+0000", 2023-01-15T14:29:16.976 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000129305 2023-01-15T14:29:16.976 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.976 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.976 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.976 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.976 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.976 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:600 3.a 3:5ba385f1:::smithi09025264-44:head [write 2685935~559341] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:29:16.976 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:59.841660+0000", 2023-01-15T14:29:16.976 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 15.331662966, 2023-01-15T14:29:16.976 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30484809000000002, 2023-01-15T14:29:16.977 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.977 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:16.977 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.977 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.977 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.977 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 600 2023-01-15T14:29:16.977 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.977 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.977 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.977 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.977 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.841660+0000", 2023-01-15T14:29:16.978 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.978 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.978 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.978 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.978 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.841660+0000", 2023-01-15T14:29:16.981 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.981 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.982 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.982 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:16.982 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.841617+0000", 2023-01-15T14:29:16.982 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999566 2023-01-15T14:29:16.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.982 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:16.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.842158+0000", 2023-01-15T14:29:16.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054178500000000003 2023-01-15T14:29:16.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.983 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:16.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.842187+0000", 2023-01-15T14:29:16.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.887e-05 2023-01-15T14:29:16.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.983 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:16.984 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942289+0000", 2023-01-15T14:29:16.984 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100102027 2023-01-15T14:29:16.984 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.984 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.984 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:16.984 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942394+0000", 2023-01-15T14:29:16.984 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000104628 2023-01-15T14:29:16.984 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.984 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.984 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:16.984 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942552+0000", 2023-01-15T14:29:16.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157751 2023-01-15T14:29:16.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:16.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942789+0000", 2023-01-15T14:29:16.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023741600000000001 2023-01-15T14:29:16.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:16.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.943243+0000", 2023-01-15T14:29:16.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045391000000000002 2023-01-15T14:29:16.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.986 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:16.986 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.943262+0000", 2023-01-15T14:29:16.986 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.908e-05 2023-01-15T14:29:16.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.986 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:16.986 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.943554+0000", 2023-01-15T14:29:16.986 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029150600000000002 2023-01-15T14:29:16.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.943604+0000", 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0722e-05 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.144702+0000", 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201097414 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.987 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:16.988 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146271+0000", 2023-01-15T14:29:16.988 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0015695430000000001 2023-01-15T14:29:16.988 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.988 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.988 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:16.988 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146367+0000", 2023-01-15T14:29:16.988 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6026000000000005e-05 2023-01-15T14:29:16.988 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.988 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.988 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:16.988 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146508+0000", 2023-01-15T14:29:16.989 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000141018 2023-01-15T14:29:16.989 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.989 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:16.989 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:16.989 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.989 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.989 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:601 3.a 3:5ba385f1:::smithi09025264-44:head [setxattr _header (57) in=7b,truncate 3663893] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:29:16.989 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:29:16.989 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 15.230997178000001, 2023-01-15T14:29:16.989 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404742724, 2023-01-15T14:29:16.989 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:16.990 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:16.990 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:16.990 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:16.990 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:16.990 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 601 2023-01-15T14:29:16.990 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.990 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:16.990 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.990 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:16.991 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:29:16.991 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.991 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.991 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:16.991 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:16.991 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:29:16.991 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:16.991 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:16.991 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942323+0000", 2023-01-15T14:29:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942341+0000", 2023-01-15T14:29:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7659000000000002e-05 2023-01-15T14:29:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942365+0000", 2023-01-15T14:29:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4256999999999998e-05 2023-01-15T14:29:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042495+0000", 2023-01-15T14:29:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013043200000001 2023-01-15T14:29:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.144809+0000", 2023-01-15T14:29:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102314184 2023-01-15T14:29:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145098+0000", 2023-01-15T14:29:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000289036 2023-01-15T14:29:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:29:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145363+0000", 2023-01-15T14:29:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026449299999999998 2023-01-15T14:29:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145657+0000", 2023-01-15T14:29:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029433600000000002 2023-01-15T14:29:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145678+0000", 2023-01-15T14:29:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0939000000000001e-05 2023-01-15T14:29:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145853+0000", 2023-01-15T14:29:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017511800000000001 2023-01-15T14:29:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146557+0000", 2023-01-15T14:29:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00070337899999999996 2023-01-15T14:29:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146845+0000", 2023-01-15T14:29:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028790599999999999 2023-01-15T14:29:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346819+0000", 2023-01-15T14:29:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199974661 2023-01-15T14:29:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346919+0000", 2023-01-15T14:29:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9234999999999994e-05 2023-01-15T14:29:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.347069+0000", 2023-01-15T14:29:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150128 2023-01-15T14:29:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:602 3.a 3:5ba385f1:::smithi09025264-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:29:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:00.042536+0000", 2023-01-15T14:29:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 15.130787136, 2023-01-15T14:29:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.304949266, 2023-01-15T14:29:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 602 2023-01-15T14:29:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042536+0000", 2023-01-15T14:29:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042536+0000", 2023-01-15T14:29:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042534+0000", 2023-01-15T14:29:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042549+0000", 2023-01-15T14:29:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4832e-05 2023-01-15T14:29:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042576+0000", 2023-01-15T14:29:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7925000000000001e-05 2023-01-15T14:29:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042593+0000", 2023-01-15T14:29:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6575999999999999e-05 2023-01-15T14:29:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146617+0000", 2023-01-15T14:29:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104024412 2023-01-15T14:29:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146778+0000", 2023-01-15T14:29:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016040100000000001 2023-01-15T14:29:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.347185+0000", 2023-01-15T14:29:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20040733499999999 2023-01-15T14:29:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.347330+0000", 2023-01-15T14:29:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014469099999999999 2023-01-15T14:29:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.347485+0000", 2023-01-15T14:29:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015536099999999999 2023-01-15T14:29:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:17.137 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:17.138 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:29:17.138 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:29:17.139 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:29:17.140 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:17.137+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:17.143 INFO:tasks.rados.rados.0.smithi090.stdout:273: done (0 left) 2023-01-15T14:29:17.143 INFO:tasks.rados.rados.0.smithi090.stdout:274: copy_from oid 12 from oid 49 current snap is 24 2023-01-15T14:29:17.143 INFO:tasks.rados.rados.0.smithi090.stdout:275: delete oid 18 current snap is 24 2023-01-15T14:29:17.145 INFO:tasks.rados.rados.0.smithi090.stdout:275: done (1 left) 2023-01-15T14:29:17.145 INFO:tasks.rados.rados.0.smithi090.stdout:276: delete oid 45 current snap is 24 2023-01-15T14:29:17.145 INFO:tasks.rados.rados.0.smithi090.stdout:274: finishing copy_from to smithi09025264-12 2023-01-15T14:29:17.145 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 34 (ObjNum 82 snap 10 seq_num 82) dirty exists 2023-01-15T14:29:17.145 INFO:tasks.rados.rados.0.smithi090.stdout:276: done (1 left) 2023-01-15T14:29:17.145 INFO:tasks.rados.rados.0.smithi090.stdout:277: rollback oid 43 current snap is 24 2023-01-15T14:29:17.145 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 43 to 23 2023-01-15T14:29:17.145 INFO:tasks.rados.rados.0.smithi090.stdout:278: write (excl) oid 37 current snap is 24 2023-01-15T14:29:17.145 INFO:tasks.rados.rados.0.smithi090.stdout:278: seq_num 132 ranges {618524=710869,1857976=360548} 2023-01-15T14:29:17.145 INFO:tasks.rados.rados.0.smithi090.stdout:278: writing smithi09025264-37 from 618524 to 1329393 tid 1 2023-01-15T14:29:17.146 INFO:tasks.rados.rados.0.smithi090.stdout:278: writing smithi09025264-37 from 1857976 to 2218524 tid 2 2023-01-15T14:29:17.146 INFO:tasks.rados.rados.0.smithi090.stdout:279: read oid 36 snap 23 2023-01-15T14:29:17.146 INFO:tasks.rados.rados.0.smithi090.stdout:279: expect (ObjNum 35 snap 0 seq_num 35) 2023-01-15T14:29:17.146 INFO:tasks.rados.rados.0.smithi090.stdout:280: write oid 13 current snap is 24 2023-01-15T14:29:17.146 INFO:tasks.rados.rados.0.smithi090.stdout:274: finishing copy_from racing read to smithi09025264-12 2023-01-15T14:29:17.146 INFO:tasks.rados.rados.0.smithi090.stdout:280: seq_num 133 ranges {635331=786118,1916458=318873} 2023-01-15T14:29:17.146 INFO:tasks.rados.rados.0.smithi090.stdout:280: writing smithi09025264-13 from 635331 to 1421449 tid 1 2023-01-15T14:29:17.146 INFO:tasks.rados.rados.0.smithi090.stdout:280: writing smithi09025264-13 from 1916458 to 2235331 tid 2 2023-01-15T14:29:17.146 INFO:tasks.rados.rados.0.smithi090.stdout:274: done (4 left) 2023-01-15T14:29:17.146 INFO:tasks.rados.rados.0.smithi090.stdout:281: delete oid 49 current snap is 24 2023-01-15T14:29:17.147 INFO:tasks.rados.rados.0.smithi090.stdout:277: finishing rollback tid 1 to smithi09025264-43 2023-01-15T14:29:17.147 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 35 (ObjNum 129 snap 0 seq_num 3197588304) dirty dne 2023-01-15T14:29:17.147 INFO:tasks.rados.rados.0.smithi090.stdout:278: finishing write tid 1 to smithi09025264-37 2023-01-15T14:29:17.147 INFO:tasks.rados.rados.0.smithi090.stdout:278: finishing write tid 2 to smithi09025264-37 2023-01-15T14:29:17.147 INFO:tasks.rados.rados.0.smithi090.stdout:278: finishing write tid 3 to smithi09025264-37 2023-01-15T14:29:17.147 INFO:tasks.rados.rados.0.smithi090.stdout:278: finishing write tid 4 to smithi09025264-37 2023-01-15T14:29:17.147 INFO:tasks.rados.rados.0.smithi090.stdout:278: oid 37 updating version 0 to 123 2023-01-15T14:29:17.147 INFO:tasks.rados.rados.0.smithi090.stdout:278: oid 37 version 123 is already newer than 122 2023-01-15T14:29:17.147 INFO:tasks.rados.rados.0.smithi090.stdout:278: oid 37 updating version 123 to 124 2023-01-15T14:29:17.147 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 124 (ObjNum 132 snap 24 seq_num 132) dirty exists 2023-01-15T14:29:17.147 INFO:tasks.rados.rados.0.smithi090.stdout:278: left oid 37 (ObjNum 132 snap 24 seq_num 132) 2023-01-15T14:29:17.148 INFO:tasks.rados.rados.0.smithi090.stdout:279: expect (ObjNum 35 snap 0 seq_num 35) 2023-01-15T14:29:17.148 INFO:tasks.rados.rados.0.smithi090.stdout:277: done (4 left) 2023-01-15T14:29:17.148 INFO:tasks.rados.rados.0.smithi090.stdout:278: done (3 left) 2023-01-15T14:29:17.148 INFO:tasks.rados.rados.0.smithi090.stdout:279: done (2 left) 2023-01-15T14:29:17.148 INFO:tasks.rados.rados.0.smithi090.stdout:281: done (1 left) 2023-01-15T14:29:17.148 INFO:tasks.rados.rados.0.smithi090.stdout:282: copy_from oid 45 from oid 37 current snap is 24 2023-01-15T14:29:17.148 INFO:tasks.rados.rados.0.smithi090.stdout:283: snap_create 2023-01-15T14:29:17.148 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:17.148 INFO:tasks.rados.rados.0.smithi090.stdout:280: finishing write tid 1 to smithi09025264-13 2023-01-15T14:29:17.148 INFO:tasks.rados.rados.0.smithi090.stdout:280: finishing write tid 2 to smithi09025264-13 2023-01-15T14:29:17.149 INFO:tasks.rados.rados.0.smithi090.stdout:280: finishing write tid 3 to smithi09025264-13 2023-01-15T14:29:17.149 INFO:tasks.rados.rados.0.smithi090.stdout:280: finishing write tid 4 to smithi09025264-13 2023-01-15T14:29:17.149 INFO:tasks.rados.rados.0.smithi090.stdout:280: oid 13 updating version 0 to 106 2023-01-15T14:29:17.149 INFO:tasks.rados.rados.0.smithi090.stdout:280: oid 13 updating version 106 to 107 2023-01-15T14:29:17.149 INFO:tasks.rados.rados.0.smithi090.stdout:280: oid 13 updating version 107 to 108 2023-01-15T14:29:17.149 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 108 (ObjNum 133 snap 24 seq_num 133) dirty exists 2023-01-15T14:29:17.149 INFO:tasks.rados.rados.0.smithi090.stdout:280: left oid 13 (ObjNum 133 snap 24 seq_num 133) 2023-01-15T14:29:17.149 INFO:tasks.rados.rados.0.smithi090.stdout:280: done (1 left) 2023-01-15T14:29:17.149 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:17.149 INFO:tasks.rados.rados.0.smithi090.stdout:282: finishing copy_from to smithi09025264-45 2023-01-15T14:29:17.149 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 124 (ObjNum 132 snap 24 seq_num 132) dirty exists 2023-01-15T14:29:17.150 INFO:tasks.rados.rados.0.smithi090.stdout:282: finishing copy_from racing read to smithi09025264-45 2023-01-15T14:29:17.150 INFO:tasks.rados.rados.0.smithi090.stdout:282: done (0 left) 2023-01-15T14:29:17.240 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:17.233+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:17.290 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:29:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:29:17.291 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:17.291 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:29:17.295 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:17.317 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:17.334 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:17.450 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:17.450 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:17.450 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:17.450 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:17.450 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:17.451 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:29:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:29:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:29:17.451 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:17.451 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:29:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:29:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:29:17.452 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:29:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:29:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:29:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:29:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:29:17.453 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:29:17.453 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:17.441+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:17.551 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:17.545+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:17.617 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:17.617 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:17.617 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:29:17.651 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:17.645+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:17.783 INFO:tasks.rados.rados.0.smithi090.stdout:283: done (0 left) 2023-01-15T14:29:17.783 INFO:tasks.rados.rados.0.smithi090.stdout:284: delete oid 50 current snap is 25 2023-01-15T14:29:17.783 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:29:17.783 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:17.783 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:17.785 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:29:17.785 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:17.781+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:17.884 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:17.877+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:17.971 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:17.971 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:17.971 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:29:17.973 INFO:tasks.rados.rados.0.smithi090.stdout:284: done (0 left) 2023-01-15T14:29:17.973 INFO:tasks.rados.rados.0.smithi090.stdout:285: write (excl) oid 28 current snap is 25 2023-01-15T14:29:17.973 INFO:tasks.rados.rados.0.smithi090.stdout:285: seq_num 136 ranges {685752=611865,1975551=310201} 2023-01-15T14:29:17.973 INFO:tasks.rados.rados.0.smithi090.stdout:285: writing smithi09025264-28 from 685752 to 1297617 tid 1 2023-01-15T14:29:17.973 INFO:tasks.rados.rados.0.smithi090.stdout:285: writing smithi09025264-28 from 1975551 to 2285752 tid 2 2023-01-15T14:29:17.973 INFO:tasks.rados.rados.0.smithi090.stdout:286: delete oid 26 current snap is 25 2023-01-15T14:29:17.973 INFO:tasks.rados.rados.0.smithi090.stdout:285: finishing write tid 1 to smithi09025264-28 2023-01-15T14:29:17.986 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:17.981+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:18.061 INFO:tasks.rados.rados.0.smithi090.stdout:285: finishing write tid 2 to smithi09025264-28 2023-01-15T14:29:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:285: finishing write tid 3 to smithi09025264-28 2023-01-15T14:29:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:285: finishing write tid 4 to smithi09025264-28 2023-01-15T14:29:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:285: oid 28 updating version 0 to 130 2023-01-15T14:29:18.138 INFO:tasks.rados.rados.0.smithi090.stdout:285: oid 28 version 130 is already newer than 128 2023-01-15T14:29:18.138 INFO:tasks.rados.rados.0.smithi090.stdout:285: oid 28 version 130 is already newer than 129 2023-01-15T14:29:18.138 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 130 (ObjNum 136 snap 25 seq_num 136) dirty exists 2023-01-15T14:29:18.138 INFO:tasks.rados.rados.0.smithi090.stdout:285: left oid 28 (ObjNum 136 snap 25 seq_num 136) 2023-01-15T14:29:18.139 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:18.139 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:18.140 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:29:18.141 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:18.133+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:18.164 INFO:tasks.rados.rados.0.smithi090.stdout:285: done (1 left) 2023-01-15T14:29:18.179 INFO:tasks.rados.rados.0.smithi090.stdout:286: done (0 left) 2023-01-15T14:29:18.179 INFO:tasks.rados.rados.0.smithi090.stdout:287: snap_remove snap 8 2023-01-15T14:29:18.238 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:18.233+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:18.312 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:18.312 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:29:18.359 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:18.353+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:18.422 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:18.492 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:18.494 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:29:18.508 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:18.489+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:18.595 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:18.593+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:18.660 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:18.660 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:18.661 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:29:18.696 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:18.693+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:18.771 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:18.842 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:18.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:29:18.845 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:18.837+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:18.944 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:18.937+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:19.010 INFO:tasks.rados.rados.0.smithi090.stdout:287: done (0 left) 2023-01-15T14:29:19.011 INFO:tasks.rados.rados.0.smithi090.stdout:288: read oid 23 snap 23 2023-01-15T14:29:19.011 INFO:tasks.rados.rados.0.smithi090.stdout:288: expect deleted 2023-01-15T14:29:19.011 INFO:tasks.rados.rados.0.smithi090.stdout:289: snap_create 2023-01-15T14:29:19.011 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:19.012 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:19.012 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:19.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:29:19.059 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:19.053+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:19.077 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:29:19.077 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:29:19.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:29:19.199 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:19.199 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:19.199 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:29:19.200 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:19.189+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:19.297 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:19.293+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:20.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:29:20.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:29:20.520 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:20.523 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:20.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:29:20.524 INFO:tasks.rados.rados.0.smithi090.stdout:288: done (0 left) 2023-01-15T14:29:20.525 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:20.526 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:29:20.526 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2023-01-15T14:29:18.951365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":224820239,"num_objects":106,"num_object_clones":54,"num_object_copies":318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":8,"num_read":703,"num_read_kb":511959,"num_write":717,"num_write_kb":216154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":90937982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1114,"ondisk_log_size":1114,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":6532548,"kb_used_data":649415460,"kb_used_omap":5101,"kb_used_meta":0,"kb_avail":649363112,"statfs":{"total":671637155840,"available":664947826688,"internally_reserved":53604352,"allocated":665001431040,"data_stored":665001431040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5223676,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":53,"commit_latency_ns":0,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9669392,"num_objects":8,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":87,"num_read_kb":100771,"num_write":52,"num_write_kb":19769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020358"},"pg_stats":[{"pgid":"3.f","version":"67'48","reported_seq":199,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-01-15T14:29:14.118423+0000","last_change":"2023-01-15T14:29:14.219016+0000","last_active":"2023-01-15T14:29:14.118423+0000","last_peered":"2023-01-15T14:29:14.118423+0000","last_clean":"2023-01-15T14:29:14.118423+0000","last_became_active":"2023-01-15T14:28:23.846063+0000","last_became_peered":"2023-01-15T14:28:23.846063+0000","last_unstale":"2023-01-15T14:29:14.118423+0000","last_undegraded":"2023-01-15T14:29:14.118423+0000","last_fullsized":"2023-01-15T14:29:14.118423+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:52.368","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30436381499999998,"stat_sum":{"num_bytes":21344659,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":11043,"num_write":35,"num_write_kb":12817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"3"},{"start":"1f","length":"2"},{"start":"23","length":"1"}]},{"pgid":"3.e","version":"67'23","reported_seq":171,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-01-15T14:29:13.425932+0000","last_change":"2023-01-15T14:29:13.425932+0000","last_active":"2023-01-15T14:29:13.425932+0000","last_peered":"2023-01-15T14:29:13.425932+0000","last_clean":"2023-01-15T14:29:13.425932+0000","last_became_active":"2023-01-15T14:28:23.846858+0000","last_became_peered":"2023-01-15T14:28:23.846858+0000","last_unstale":"2023-01-15T14:29:13.425932+0000","last_undegraded":"2023-01-15T14:29:13.425932+0000","last_fullsized":"2023-01-15T14:29:13.425932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:44.514","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50325226999999995,"stat_sum":{"num_bytes":4756315,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":12810,"num_write":6,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"3"},{"start":"1f","length":"2"},{"start":"23","length":"1"}]},{"pgid":"3.d","version":"67'79","reported_seq":257,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-01-15T14:29:13.873753+0000","last_change":"2023-01-15T14:29:13.622183+0000","last_active":"2023-01-15T14:29:13.873753+0000","last_peered":"2023-01-15T14:29:13.873753+0000","last_clean":"2023-01-15T14:29:13.873753+0000","last_became_active":"2023-01-15T14:28:54.932913+0000","last_became_peered":"2023-01-15T14:28:54.932913+0000","last_unstale":"2023-01-15T14:29:13.873753+0000","last_undegraded":"2023-01-15T14:29:13.873753+0000","last_fullsized":"2023-01-15T14:29:13.873753+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:33.416","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70292290400000002,"stat_sum":{"num_bytes":17122428,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":48,"num_read_kb":50172,"num_write":35,"num_write_kb":14630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"3"},{"start":"1f","length":"2"},{"start":"23","length":"1"}]},{"pgid":"3.c","version":"68'31","reported_seq":112,"reported_epoch":68,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:14.692849+0000","last_change":"2023-01-15T14:29:13.710231+0000","last_active":"2023-01-15T14:29:14.692849+0000","last_peered":"2023-01-15T14:29:14.692849+0000","last_clean":"2023-01-15T14:29:14.692849+0000","last_became_active":"2023-01-15T14:28:54.833568+0000","last_became_peered":"2023-01-15T14:28:54.833568+0000","last_unstale":"2023-01-15T14:29:14.692849+0000","last_undegraded":"2023-01-15T14:29:14.692849+0000","last_fullsized":"2023-01-15T14:29:14.692849+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.421","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70323012500000004,"stat_sum":{"num_bytes":7008530,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":19,"num_write_kb":5044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2319333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"1b","length":"2"},{"start":"1f","length":"2"}]},{"pgid":"3.b","version":"67'22","reported_seq":159,"reported_epoch":68,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:14.122829+0000","last_change":"2023-01-15T14:29:14.122829+0000","last_active":"2023-01-15T14:29:14.122829+0000","last_peered":"2023-01-15T14:29:14.122829+0000","last_clean":"2023-01-15T14:29:14.122829+0000","last_became_active":"2023-01-15T14:28:23.934183+0000","last_became_peered":"2023-01-15T14:28:23.934183+0000","last_unstale":"2023-01-15T14:29:14.122829+0000","last_undegraded":"2023-01-15T14:29:14.122829+0000","last_fullsized":"2023-01-15T14:29:14.122829+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:45.586","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107785231,"stat_sum":{"num_bytes":5215740,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":14,"num_read_kb":13386,"num_write":10,"num_write_kb":3028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"1"},{"start":"1f","length":"2"}]},{"pgid":"3.a","version":"70'104","reported_seq":287,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:15.938224+0000","last_change":"2023-01-15T14:29:15.938224+0000","last_active":"2023-01-15T14:29:15.938224+0000","last_peered":"2023-01-15T14:29:15.938224+0000","last_clean":"2023-01-15T14:29:15.938224+0000","last_became_active":"2023-01-15T14:29:02.112091+0000","last_became_peered":"2023-01-15T14:29:02.112091+0000","last_unstale":"2023-01-15T14:29:15.938224+0000","last_undegraded":"2023-01-15T14:29:15.938224+0000","last_fullsized":"2023-01-15T14:29:15.938224+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:28.279","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.77001517100000005,"stat_sum":{"num_bytes":19882483,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":31,"num_read_kb":15774,"num_write":58,"num_write_kb":27397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":21341200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"3"},{"start":"1f","length":"2"},{"start":"23","length":"1"}]},{"pgid":"3.9","version":"68'86","reported_seq":247,"reported_epoch":70,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:16.037847+0000","last_change":"2023-01-15T14:29:16.037847+0000","last_active":"2023-01-15T14:29:16.037847+0000","last_peered":"2023-01-15T14:29:16.037847+0000","last_clean":"2023-01-15T14:29:16.037847+0000","last_became_active":"2023-01-15T14:28:24.109843+0000","last_became_peered":"2023-01-15T14:28:24.109843+0000","last_unstale":"2023-01-15T14:29:16.037847+0000","last_undegraded":"2023-01-15T14:29:16.037847+0000","last_fullsized":"2023-01-15T14:29:16.037847+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.059731172999999999,"stat_sum":{"num_bytes":18162385,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":59,"num_read_kb":55285,"num_write":45,"num_write_kb":18275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"3"},{"start":"1f","length":"2"},{"start":"23","length":"1"}]},{"pgid":"3.8","version":"68'85","reported_seq":265,"reported_epoch":69,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:14.933253+0000","last_change":"2023-01-15T14:29:14.122840+0000","last_active":"2023-01-15T14:29:14.933253+0000","last_peered":"2023-01-15T14:29:14.933253+0000","last_clean":"2023-01-15T14:29:14.933253+0000","last_became_active":"2023-01-15T14:28:47.937231+0000","last_became_peered":"2023-01-15T14:28:47.937231+0000","last_unstale":"2023-01-15T14:29:14.933253+0000","last_undegraded":"2023-01-15T14:29:14.933253+0000","last_fullsized":"2023-01-15T14:29:14.933253+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:29.801","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99489164699999999,"stat_sum":{"num_bytes":20507017,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":69,"num_read_kb":73604,"num_write":45,"num_write_kb":17495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"1c","length":"1"},{"start":"1f","length":"2"},{"start":"23","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":238,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-01-15T14:29:12.919039+0000","last_change":"2023-01-15T14:29:05.329442+0000","last_active":"2023-01-15T14:29:12.919039+0000","last_peered":"2023-01-15T14:29:12.919039+0000","last_clean":"2023-01-15T14:29:12.919039+0000","last_became_active":"2023-01-15T14:29:05.228004+0000","last_became_peered":"2023-01-15T14:29:05.228004+0000","last_unstale":"2023-01-15T14:29:12.919039+0000","last_undegraded":"2023-01-15T14:29:12.919039+0000","last_fullsized":"2023-01-15T14:29:12.919039+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:34.812","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":34,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-01-15T14:29:13.918759+0000","last_change":"2023-01-15T14:28:54.829192+0000","last_active":"2023-01-15T14:29:13.918759+0000","last_peered":"2023-01-15T14:29:13.918759+0000","last_clean":"2023-01-15T14:29:13.918759+0000","last_became_active":"2023-01-15T14:28:54.828857+0000","last_became_peered":"2023-01-15T14:28:54.828857+0000","last_unstale":"2023-01-15T14:29:13.918759+0000","last_undegraded":"2023-01-15T14:29:13.918759+0000","last_fullsized":"2023-01-15T14:29:13.918759+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.638","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"68'58","reported_seq":229,"reported_epoch":69,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:14.939814+0000","last_change":"2023-01-15T14:29:14.939865+0000","last_active":"2023-01-15T14:29:14.939814+0000","last_peered":"2023-01-15T14:29:14.939814+0000","last_clean":"2023-01-15T14:29:14.939814+0000","last_became_active":"2023-01-15T14:28:55.217912+0000","last_became_peered":"2023-01-15T14:28:55.217912+0000","last_unstale":"2023-01-15T14:29:14.939814+0000","last_undegraded":"2023-01-15T14:29:14.939814+0000","last_fullsized":"2023-01-15T14:29:14.939814+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.256","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0055812989999999996,"stat_sum":{"num_bytes":16833632,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":32,"num_read_kb":27661,"num_write":39,"num_write_kb":15123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10821306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"2"},{"start":"1f","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":59,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-01-15T14:29:12.918987+0000","last_change":"2023-01-15T14:28:13.013616+0000","last_active":"2023-01-15T14:29:12.918987+0000","last_peered":"2023-01-15T14:29:12.918987+0000","last_clean":"2023-01-15T14:29:12.918987+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:29:12.918987+0000","last_undegraded":"2023-01-15T14:29:12.918987+0000","last_fullsized":"2023-01-15T14:29:12.918987+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:29.249","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"68'59","reported_seq":219,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-01-15T14:29:14.722853+0000","last_change":"2023-01-15T14:29:13.617418+0000","last_active":"2023-01-15T14:29:14.722853+0000","last_peered":"2023-01-15T14:29:14.722853+0000","last_clean":"2023-01-15T14:29:14.722853+0000","last_became_active":"2023-01-15T14:28:23.842545+0000","last_became_peered":"2023-01-15T14:28:23.842545+0000","last_unstale":"2023-01-15T14:29:14.722853+0000","last_undegraded":"2023-01-15T14:29:14.722853+0000","last_fullsized":"2023-01-15T14:29:14.722853+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:30.758","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70442782699999995,"stat_sum":{"num_bytes":17344347,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":43,"num_read_kb":39237,"num_write":41,"num_write_kb":13110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"3"},{"start":"1f","length":"2"},{"start":"23","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":61,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-01-15T14:29:13.919095+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:29:13.919095+0000","last_peered":"2023-01-15T14:29:13.919095+0000","last_clean":"2023-01-15T14:29:13.919095+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:29:13.919095+0000","last_undegraded":"2023-01-15T14:29:13.919095+0000","last_fullsized":"2023-01-15T14:29:13.919095+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.689","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"70'136","reported_seq":332,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:15.937595+0000","last_change":"2023-01-15T14:29:15.937595+0000","last_active":"2023-01-15T14:29:15.937595+0000","last_peered":"2023-01-15T14:29:15.937595+0000","last_clean":"2023-01-15T14:29:15.937595+0000","last_became_active":"2023-01-15T14:29:02.007426+0000","last_became_peered":"2023-01-15T14:29:02.007426+0000","last_unstale":"2023-01-15T14:29:15.937595+0000","last_undegraded":"2023-01-15T14:29:15.937595+0000","last_fullsized":"2023-01-15T14:29:15.937595+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:48.446","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.80797126399999997,"stat_sum":{"num_bytes":21295904,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":76,"num_read_kb":66307,"num_write":62,"num_write_kb":24793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16722767,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"3"},{"start":"1f","length":"2"},{"start":"23","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-01-15T14:29:14.932537+0000","last_change":"2023-01-15T14:29:14.703116+0000","last_active":"2023-01-15T14:29:14.932537+0000","last_peered":"2023-01-15T14:29:14.932537+0000","last_clean":"2023-01-15T14:29:14.932537+0000","last_became_active":"2023-01-15T14:29:14.050302+0000","last_became_peered":"2023-01-15T14:29:14.050302+0000","last_unstale":"2023-01-15T14:29:14.932537+0000","last_undegraded":"2023-01-15T14:29:14.932537+0000","last_fullsized":"2023-01-15T14:29:14.932537+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:14.703035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:14.703035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:18.827","scrub_duration":0.44938580500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"70'120","reported_seq":297,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:16.141424+0000","last_change":"2023-01-15T14:29:15.938438+0000","last_active":"2023-01-15T14:29:16.141424+0000","last_peered":"2023-01-15T14:29:16.141424+0000","last_clean":"2023-01-15T14:29:16.141424+0000","last_became_active":"2023-01-15T14:28:23.946074+0000","last_became_peered":"2023-01-15T14:28:23.946074+0000","last_unstale":"2023-01-15T14:29:16.141424+0000","last_undegraded":"2023-01-15T14:29:16.141424+0000","last_fullsized":"2023-01-15T14:29:16.141424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:40.271","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.70791207700000003,"stat_sum":{"num_bytes":19593805,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":99,"num_read_kb":98726,"num_write":73,"num_write_kb":20526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"3"},{"start":"1f","length":"2"},{"start":"23","length":"1"}]},{"pgid":"3.5","version":"67'25","reported_seq":142,"reported_epoch":68,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:14.336020+0000","last_change":"2023-01-15T14:29:14.336020+0000","last_active":"2023-01-15T14:29:14.336020+0000","last_peered":"2023-01-15T14:29:14.336020+0000","last_clean":"2023-01-15T14:29:14.336020+0000","last_became_active":"2023-01-15T14:28:54.831872+0000","last_became_peered":"2023-01-15T14:28:54.831872+0000","last_unstale":"2023-01-15T14:29:14.336020+0000","last_undegraded":"2023-01-15T14:29:14.336020+0000","last_fullsized":"2023-01-15T14:29:14.336020+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:46.735","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42082032699999999,"stat_sum":{"num_bytes":7428595,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":2,"num_write":12,"num_write_kb":6933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4252105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"1"},{"start":"1f","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":61,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-01-15T14:29:14.994784+0000","last_change":"2023-01-15T14:28:12.897110+0000","last_active":"2023-01-15T14:29:14.994784+0000","last_peered":"2023-01-15T14:29:14.994784+0000","last_clean":"2023-01-15T14:29:14.994784+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:29:14.994784+0000","last_undegraded":"2023-01-15T14:29:14.994784+0000","last_fullsized":"2023-01-15T14:29:14.994784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:27.569","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"67'48","reported_seq":242,"reported_epoch":70,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:16.037829+0000","last_change":"2023-01-15T14:29:16.037901+0000","last_active":"2023-01-15T14:29:16.037829+0000","last_peered":"2023-01-15T14:29:16.037829+0000","last_clean":"2023-01-15T14:29:16.037829+0000","last_became_active":"2023-01-15T14:28:54.825996+0000","last_became_peered":"2023-01-15T14:28:54.825996+0000","last_unstale":"2023-01-15T14:29:16.037829+0000","last_undegraded":"2023-01-15T14:29:16.037829+0000","last_fullsized":"2023-01-15T14:29:16.037829+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.440","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.93585060499999995,"stat_sum":{"num_bytes":16377083,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":50,"num_read_kb":47841,"num_write":39,"num_write_kb":13161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12100908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"3"},{"start":"1f","length":"2"},{"start":"23","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":59,"reported_epoch":67,"state":"active+clean","last_fresh":"2023-01-15T14:29:12.919466+0000","last_change":"2023-01-15T14:28:13.318466+0000","last_active":"2023-01-15T14:29:12.919466+0000","last_peered":"2023-01-15T14:29:12.919466+0000","last_clean":"2023-01-15T14:29:12.919466+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:29:12.919466+0000","last_undegraded":"2023-01-15T14:29:12.919466+0000","last_fullsized":"2023-01-15T14:29:12.919466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:20.665","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"67'45","reported_seq":179,"reported_epoch":69,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:14.939846+0000","last_change":"2023-01-15T14:29:14.939890+0000","last_active":"2023-01-15T14:29:14.939846+0000","last_peered":"2023-01-15T14:29:14.939846+0000","last_clean":"2023-01-15T14:29:14.939846+0000","last_became_active":"2023-01-15T14:28:23.741266+0000","last_became_peered":"2023-01-15T14:28:23.741266+0000","last_unstale":"2023-01-15T14:29:14.939846+0000","last_undegraded":"2023-01-15T14:29:14.939846+0000","last_fullsized":"2023-01-15T14:29:14.939846+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50959705899999996,"stat_sum":{"num_bytes":5680700,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":8,"num_read_kb":4,"num_write":21,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1c","length":"1"},{"start":"1f","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":71,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-01-15T14:29:14.994980+0000","last_change":"2023-01-15T14:29:01.810622+0000","last_active":"2023-01-15T14:29:14.994980+0000","last_peered":"2023-01-15T14:29:14.994980+0000","last_clean":"2023-01-15T14:29:14.994980+0000","last_became_active":"2023-01-15T14:29:01.809869+0000","last_became_peered":"2023-01-15T14:29:01.809869+0000","last_unstale":"2023-01-15T14:29:14.994980+0000","last_undegraded":"2023-01-15T14:29:14.994980+0000","last_fullsized":"2023-01-15T14:29:14.994980+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:34.307","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"68'28","reported_seq":174,"reported_epoch":68,"state":"active+clean","last_fresh":"2023-01-15T14:29:14.893354+0000","last_change":"2023-01-15T14:29:13.014304+0000","last_active":"2023-01-15T14:29:14.893354+0000","last_peered":"2023-01-15T14:29:14.893354+0000","last_clean":"2023-01-15T14:29:14.893354+0000","last_became_active":"2023-01-15T14:28:23.841189+0000","last_became_peered":"2023-01-15T14:28:23.841189+0000","last_unstale":"2023-01-15T14:29:14.893354+0000","last_undegraded":"2023-01-15T14:29:14.893354+0000","last_fullsized":"2023-01-15T14:29:14.893354+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:34.816","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10065795399999999,"stat_sum":{"num_bytes":5676248,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":4,"num_write":18,"num_write_kb":7273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15","length":"1"},{"start":"19","length":"1"},{"start":"1b","length":"3"},{"start":"1f","length":"2"},{"start":"23","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":69,"state":"active+clean","last_fresh":"2023-01-15T14:29:15.033349+0000","last_change":"2023-01-15T14:28:12.896363+0000","last_active":"2023-01-15T14:29:15.033349+0000","last_peered":"2023-01-15T14:29:15.033349+0000","last_clean":"2023-01-15T14:29:15.033349+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:29:15.033349+0000","last_undegraded":"2023-01-15T14:29:15.033349+0000","last_fullsized":"2023-01-15T14:29:15.033349+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.332","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":224229871,"num_objects":104,"num_object_clones":54,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":8,"num_read":587,"num_read_kb":511859,"num_write":558,"num_write_kb":213738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":90347614,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":997,"ondisk_log_size":997,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574864,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":895284,"kb_used_data":92804948,"kb_used_omap":608,"kb_used_meta":0,"kb_avail":92804096,"statfs":{"total":95948165120,"available":95031394304,"internally_reserved":872448,"allocated":95032266752,"data_stored":95032266752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":623224,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.0700000000000003,"5min":4.0700000000000003,"15min":4.0700000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":75.893000000000001,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":9.2089999999999996,"5min":9.2089999999999996,"15min":9.2089999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":71.308999999999997,"5min":71.308999999999997,"15min":71.308999999999997},"last":0.77500000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":75.959999999999994,"5min":75.959999999999994,"15min":75.959999999999994},"last":0.73999999999999999},{"interface":"front","average":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":6.2800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":68.605000000000004,"5min":68.605000000000004,"15min":68.605000000000004},"last":0.53700000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.7910000000000004,"5min":4.7910000000000004,"15min":4.7910000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":94.733999999999995,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.84599999999999997},{"interface":"front","average":{"1min":4.1779999999999999,"5min":4.1779999999999999,"15min":4.1779999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":75.998999999999995},"last":0.67000000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.7770000000000001,"5min":3.7770000000000001,"15min":3.7770000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":76.034999999999997,"5min":76.034999999999997,"15min":76.034999999999997},"last":45.164000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":9.4160000000000004,"5min":9.4160000000000004,"15min":9.4160000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":76.658000000000001,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":3.5129999999999999,"5min":3.5129999999999999,"15min":3.5129999999999999},"last":0.96799999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.327,"5min":6.327,"15min":6.327},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":68.679000000000002,"5min":68.679000000000002,"15min":68.679000000000002},"last":0.70199999999999996},{"interface":"front","average":{"1min":4.9450000000000003,"5min":4.9450000000000003,"15min":4.9450000000000003},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":95.430000000000007,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.90900000000000003}]},{"osd":6,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":8.5239999999999991,"5min":8.5239999999999991,"15min":8.5239999999999991},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":68.635999999999996,"5min":68.635999999999996,"15min":68.635999999999996},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.6520000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.93899999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":898840,"kb_used_data":92807500,"kb_used_omap":759,"kb_used_meta":0,"kb_avail":92800540,"statfs":{"total":95948165120,"available":95027752960,"internally_reserved":7127040,"allocated":95034880000,"data_stored":95034880000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":777357,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.505},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58999999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":1.105}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.0529999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.0590000000000002,"5min":6.0590000000000002,"15min":6.0590000000000002},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":81.686999999999998,"5min":81.686999999999998,"15min":81.686999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":11.455,"5min":11.455,"15min":11.455},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":255.24299999999999,"5min":255.24299999999999,"15min":255.24299999999999},"last":1.0229999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":6.6680000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":138.197,"5min":138.197,"15min":138.197},"last":0.80000000000000004}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":1.133},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.88100000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.2800000000000002,"5min":4.2800000000000002,"15min":4.2800000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":78.888000000000005,"5min":78.888000000000005,"15min":78.888000000000005},"last":1.155},{"interface":"front","average":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":47.332000000000001,"5min":47.332000000000001,"15min":47.332000000000001},"last":1.2410000000000001}]}]},{"osd":5,"up_from":0,"seq":55834574865,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574864,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":957904,"kb_used_data":92747356,"kb_used_omap":840,"kb_used_meta":0,"kb_avail":92741476,"statfs":{"total":95948165120,"available":94967271424,"internally_reserved":6021120,"allocated":94973292544,"data_stored":94973292544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":861129,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.915,"5min":4.915,"15min":4.915},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":86.944999999999993,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.625},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.69099999999999995}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":8.5939999999999994,"5min":8.5939999999999994,"15min":8.5939999999999994},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":99.960999999999999,"5min":99.960999999999999,"15min":99.960999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.78300000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":99.884,"5min":99.884,"15min":99.884},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.999}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":12.1,"5min":12.1,"15min":12.1},"last":0.84599999999999997},{"interface":"front","average":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":6.1509999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":99.921000000000006,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.89900000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.087},{"interface":"front","average":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":30.971,"5min":30.971,"15min":30.971},"last":0.93899999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.673,"5min":5.673,"15min":5.673},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":99.780000000000001,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":5.1630000000000003,"5min":5.1630000000000003,"15min":5.1630000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":88.186999999999998,"5min":88.186999999999998,"15min":88.186999999999998},"last":1.1459999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.128,"5min":13.128,"15min":13.128},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":179.73500000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":1.04},{"interface":"front","average":{"1min":11.239000000000001,"5min":11.239000000000001,"15min":11.239000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":99.843000000000004,"5min":99.843000000000004,"15min":99.843000000000004},"last":1.022}]}]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":968004,"kb_used_data":92738716,"kb_used_omap":760,"kb_used_meta":0,"kb_avail":92731376,"statfs":{"total":95948165120,"available":94956929024,"internally_reserved":7516160,"allocated":94964445184,"data_stored":94964445184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":778983,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.8479999999999999,"5min":4.8479999999999999,"15min":4.8479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":93.632999999999996,"5min":93.632999999999996,"15min":93.632999999999996},"last":0.63400000000000001},{"interface":"front","average":{"1min":7.6420000000000003,"5min":7.6420000000000003,"15min":7.6420000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":93.674000000000007,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.624}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.532,"5min":13.532,"15min":13.532},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":159.256,"5min":159.256,"15min":159.256},"last":0.435},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.71499999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.3490000000000002,"5min":3.3490000000000002,"15min":3.3490000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":59.896999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":31.233000000000001,"5min":31.233000000000001,"15min":31.233000000000001},"last":0.80400000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":6.5620000000000003,"5min":6.5620000000000003,"15min":6.5620000000000003},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":66.311999999999998,"5min":66.311999999999998,"15min":66.311999999999998},"last":0.90400000000000003},{"interface":"front","average":{"1min":13.874000000000001,"5min":13.874000000000001,"15min":13.874000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":159.47200000000001,"5min":159.47200000000001,"15min":159.47200000000001},"last":0.69099999999999995}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":13.903,"15min":13.903},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":159.43000000000001,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.96499999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":60.098999999999997,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.94799999999999995}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.0359999999999996,"5min":4.0359999999999996,"15min":4.0359999999999996},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":60.165999999999997,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.84199999999999997},{"interface":"front","average":{"1min":18.09,"5min":18.09,"15min":18.09},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":159.512,"5min":159.512,"15min":159.512},"last":0.88}]}]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":927056,"kb_used_data":92782144,"kb_used_omap":564,"kb_used_meta":0,"kb_avail":92772324,"statfs":{"total":95948165120,"available":94998859776,"internally_reserved":10055680,"allocated":95008915456,"data_stored":95008915456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":578387,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":13,"commit_latency_ns":0,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.438}]},{"osd":1,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.5979999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":0.65200000000000002},{"interface":"front","average":{"1min":3.3559999999999999,"5min":3.3559999999999999,"15min":3.3559999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":47.518000000000001,"5min":47.518000000000001,"15min":47.518000000000001},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":10.401999999999999,"5min":10.401999999999999,"15min":10.401999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":183.53,"5min":183.53,"15min":183.53},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.70299999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.6499999999999999,"15min":3.6499999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":53.139000000000003,"5min":53.139000000000003,"15min":53.139000000000003},"last":1.0680000000000001},{"interface":"front","average":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":3.8090000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":53.186999999999998,"5min":53.186999999999998,"15min":53.186999999999998},"last":1.115}]},{"osd":5,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.93100000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":0.94799999999999995}]},{"osd":7,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":37.124000000000002,"5min":37.124000000000002,"15min":37.124000000000002},"last":1.0149999999999999},{"interface":"front","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":30.445,"5min":30.445,"15min":30.445},"last":0.89800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":926012,"kb_used_data":92784544,"kb_used_omap":766,"kb_used_meta":0,"kb_avail":92773368,"statfs":{"total":95948165120,"available":94999928832,"internally_reserved":11444224,"allocated":95011373056,"data_stored":95011373056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":784850,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":3.8929999999999998,"15min":3.8929999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":46.591000000000001,"5min":46.591000000000001,"15min":46.591000000000001},"last":4.0250000000000004},{"interface":"front","average":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":50.616,"5min":50.616,"15min":50.616},"last":0.34899999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.245},{"interface":"front","average":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":46.747999999999998,"5min":46.747999999999998,"15min":46.747999999999998},"last":0.44900000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":19.835999999999999,"5min":19.835999999999999,"15min":19.835999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":282.01400000000001,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":7.2450000000000001,"5min":7.2450000000000001,"15min":7.2450000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":53.145000000000003,"5min":53.145000000000003,"15min":53.145000000000003},"last":0.53500000000000003}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.7279999999999998,"5min":4.7279999999999998,"15min":4.7279999999999998},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":50.887999999999998,"5min":50.887999999999998,"15min":50.887999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":3.605,"5min":3.605,"15min":3.605},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":52.512,"5min":52.512,"15min":52.512},"last":0.79300000000000004}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.433},{"interface":"front","average":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":4.4379999999999997},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":46.905999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.77300000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.0019999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":50.844999999999999,"5min":50.844999999999999,"15min":50.844999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.72299999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":52.595999999999997,"5min":52.595999999999997,"15min":52.595999999999997},"last":0.69399999999999995},{"interface":"front","average":{"1min":4.75,"5min":4.75,"15min":4.75},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":50.804000000000002,"5min":50.804000000000002,"15min":50.804000000000002},"last":0.73799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":959448,"kb_used_data":92750252,"kb_used_omap":800,"kb_used_meta":0,"kb_avail":92739932,"statfs":{"total":95948165120,"available":94965690368,"internally_reserved":10567680,"allocated":94976258048,"data_stored":94976258048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":819746,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":12,"commit_latency_ns":0,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.93899999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.79800000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":16.779,"5min":16.779,"15min":16.779},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.85799999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":1.222},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.083}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":9.798,"5min":9.798,"15min":9.798},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":176.34399999999999,"5min":176.34399999999999,"15min":176.34399999999999},"last":1.113},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":0.997}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.242},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":1.175}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":15.670999999999999,"5min":15.670999999999999,"15min":15.670999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":292.61099999999999,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.127},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.147}]}]}],"pool_statfs":[]}} 2023-01-15T14:29:20.538 INFO:tasks.rados.rados.0.smithi090.stdout:289: done (0 left) 2023-01-15T14:29:21.658 INFO:tasks.rados.rados.0.smithi090.stdout:290: read oid 39 snap 24 2023-01-15T14:29:21.658 INFO:tasks.rados.rados.0.smithi090.stdout:290: expect (ObjNum 121 snap 23 seq_num 121) 2023-01-15T14:29:21.658 INFO:tasks.rados.rados.0.smithi090.stdout:291: read oid 38 snap -1 2023-01-15T14:29:21.658 INFO:tasks.rados.rados.0.smithi090.stdout:291: expect (ObjNum 37 snap 0 seq_num 37) 2023-01-15T14:29:21.658 INFO:tasks.rados.rados.0.smithi090.stdout:292: copy_from oid 3 from oid 30 current snap is 26 2023-01-15T14:29:21.658 INFO:tasks.rados.rados.0.smithi090.stdout:293: write (excl) oid 35 current snap is 26 2023-01-15T14:29:21.658 INFO:tasks.rados.rados.0.smithi090.stdout:293: seq_num 138 ranges {719366=478716,1808868=510498} 2023-01-15T14:29:21.659 INFO:tasks.rados.rados.0.smithi090.stdout:293: writing smithi09025264-35 from 719366 to 1198082 tid 1 2023-01-15T14:29:21.659 INFO:tasks.rados.rados.0.smithi090.stdout:293: writing smithi09025264-35 from 1808868 to 2319366 tid 2 2023-01-15T14:29:21.659 INFO:tasks.rados.rados.0.smithi090.stdout:294: read oid 4 snap -1 2023-01-15T14:29:21.659 INFO:tasks.rados.rados.0.smithi090.stdout:294: expect (ObjNum 120 snap 23 seq_num 120) 2023-01-15T14:29:21.661 INFO:tasks.rados.rados.0.smithi090.stdout:295: rollback oid 42 current snap is 26 2023-01-15T14:29:21.661 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 42 to 23 2023-01-15T14:29:21.661 INFO:tasks.rados.rados.0.smithi090.stdout:296: copy_from oid 15 from oid 18 current snap is 26 2023-01-15T14:29:21.661 INFO:tasks.rados.rados.0.smithi090.stdout:297: delete oid 49 current snap is 26 2023-01-15T14:29:21.661 INFO:tasks.rados.rados.0.smithi090.stdout:294: expect (ObjNum 120 snap 23 seq_num 120) 2023-01-15T14:29:21.661 INFO:tasks.rados.rados.0.smithi090.stdout:293: finishing write tid 1 to smithi09025264-35 2023-01-15T14:29:21.661 INFO:tasks.rados.rados.0.smithi090.stdout:290: expect (ObjNum 121 snap 23 seq_num 121) 2023-01-15T14:29:21.661 INFO:tasks.rados.rados.0.smithi090.stdout:291: expect (ObjNum 37 snap 0 seq_num 37) 2023-01-15T14:29:21.661 INFO:tasks.rados.rados.0.smithi090.stdout:292: finishing copy_from to smithi09025264-3 2023-01-15T14:29:21.662 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-01-15T14:29:21.662 INFO:tasks.rados.rados.0.smithi090.stdout:292: finishing copy_from racing read to smithi09025264-3 2023-01-15T14:29:21.662 INFO:tasks.rados.rados.0.smithi090.stdout:296: finishing copy_from to smithi09025264-15 2023-01-15T14:29:21.662 INFO:tasks.rados.rados.0.smithi090.stdout:296: got expected ENOENT (src dne) 2023-01-15T14:29:21.662 INFO:tasks.rados.rados.0.smithi090.stdout:296: finishing copy_from racing read to smithi09025264-15 2023-01-15T14:29:21.662 INFO:tasks.rados.rados.0.smithi090.stdout:295: finishing rollback tid 0 to smithi09025264-42 2023-01-15T14:29:21.663 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:20.509+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:21.665 INFO:tasks.rados.rados.0.smithi090.stdout:295: finishing rollback tid 1 to smithi09025264-42 2023-01-15T14:29:21.665 INFO:tasks.rados.rados.0.smithi090.stdout:295: finishing rollback tid 2 to smithi09025264-42 2023-01-15T14:29:21.665 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 42 v 103 (ObjNum 91 snap 15 seq_num 91) dirty exists 2023-01-15T14:29:21.665 INFO:tasks.rados.rados.0.smithi090.stdout:293: finishing write tid 2 to smithi09025264-35 2023-01-15T14:29:21.665 INFO:tasks.rados.rados.0.smithi090.stdout:293: finishing write tid 3 to smithi09025264-35 2023-01-15T14:29:21.665 INFO:tasks.rados.rados.0.smithi090.stdout:293: finishing write tid 4 to smithi09025264-35 2023-01-15T14:29:21.665 INFO:tasks.rados.rados.0.smithi090.stdout:293: oid 35 updating version 0 to 99 2023-01-15T14:29:21.665 INFO:tasks.rados.rados.0.smithi090.stdout:293: oid 35 updating version 99 to 104 2023-01-15T14:29:21.665 INFO:tasks.rados.rados.0.smithi090.stdout:293: oid 35 updating version 104 to 105 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 105 (ObjNum 138 snap 26 seq_num 138) dirty exists 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:293: left oid 35 (ObjNum 138 snap 26 seq_num 138) 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:290: done (7 left) 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:291: done (6 left) 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:292: done (5 left) 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:293: done (4 left) 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:294: done (3 left) 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:295: done (2 left) 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:296: done (1 left) 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:297: done (0 left) 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:298: snap_create 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:298: done (0 left) 2023-01-15T14:29:21.666 INFO:tasks.rados.rados.0.smithi090.stdout:299: snap_remove snap 26 2023-01-15T14:29:21.668 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:21.661+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:21.721 INFO:tasks.rados.rados.0.smithi090.stdout:299: done (0 left) 2023-01-15T14:29:21.863 INFO:tasks.rados.rados.0.smithi090.stdout:300: delete oid 20 current snap is 27 2023-01-15T14:29:21.865 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:21.865 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:21.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:29:21.867 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:21.867 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:21.867 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:21.867 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:21.867 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:21.857+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:21.882 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:21.917 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:22.073 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:22.073 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:22.074 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:29:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:29:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:29:22.076 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:29:22.077 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:29:22.077 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:29:22.078 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:29:22.078 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:29:22.079 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:22.080 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:29:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:29:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.081 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:166 3.8 3:1e54a7f5:::smithi09025264-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:29:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.885386492000002, 2023-01-15T14:29:22.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62664632300000001, 2023-01-15T14:29:22.083 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 166 2023-01-15T14:29:22.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:29:22.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:29:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114713+0000", 2023-01-15T14:29:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:22.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115178+0000", 2023-01-15T14:29:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046493199999999999 2023-01-15T14:29:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115208+0000", 2023-01-15T14:29:22.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0689999999999999e-05 2023-01-15T14:29:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115221+0000", 2023-01-15T14:29:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3013000000000001e-05 2023-01-15T14:29:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115334+0000", 2023-01-15T14:29:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112403 2023-01-15T14:29:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115505+0000", 2023-01-15T14:29:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170915 2023-01-15T14:29:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333496+0000", 2023-01-15T14:29:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217991667 2023-01-15T14:29:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333606+0000", 2023-01-15T14:29:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109967 2023-01-15T14:29:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:29:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333796+0000", 2023-01-15T14:29:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018928099999999999 2023-01-15T14:29:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334289+0000", 2023-01-15T14:29:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049322600000000004 2023-01-15T14:29:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334339+0000", 2023-01-15T14:29:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0439999999999998e-05 2023-01-15T14:29:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334990+0000", 2023-01-15T14:29:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065082300000000003 2023-01-15T14:29:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335100+0000", 2023-01-15T14:29:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109357 2023-01-15T14:29:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.640854+0000", 2023-01-15T14:29:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30575451300000001 2023-01-15T14:29:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741190+0000", 2023-01-15T14:29:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100335568 2023-01-15T14:29:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741263+0000", 2023-01-15T14:29:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3275999999999994e-05 2023-01-15T14:29:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741361+0000", 2023-01-15T14:29:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8435000000000002e-05 2023-01-15T14:29:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:167 3.8 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:29:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.884821803000001, 2023-01-15T14:29:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62638390099999997, 2023-01-15T14:29:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:22.103 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 167 2023-01-15T14:29:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.104 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:29:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:29:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115279+0000", 2023-01-15T14:29:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115289+0000", 2023-01-15T14:29:22.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0509999999999999e-05 2023-01-15T14:29:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:22.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115306+0000", 2023-01-15T14:29:22.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7260000000000001e-05 2023-01-15T14:29:22.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:22.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115337+0000", 2023-01-15T14:29:22.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0664000000000003e-05 2023-01-15T14:29:22.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115579+0000", 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024194400000000001 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115744+0000", 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016493700000000001 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334392+0000", 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21864830299999999 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334453+0000", 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1047e-05 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334601+0000", 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014824700000000001 2023-01-15T14:29:22.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334812+0000", 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210208 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334996+0000", 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183878 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:22.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335126+0000", 2023-01-15T14:29:22.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013001399999999999 2023-01-15T14:29:22.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:22.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335160+0000", 2023-01-15T14:29:22.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4876999999999998e-05 2023-01-15T14:29:22.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741044+0000", 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40588326000000002 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741453+0000", 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040963099999999998 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741548+0000", 2023-01-15T14:29:22.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4961000000000001e-05 2023-01-15T14:29:22.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:22.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741664+0000", 2023-01-15T14:29:22.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115333 2023-01-15T14:29:22.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.486 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:22.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.486 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:22.487 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:29:22.487 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.884729688999997, 2023-01-15T14:29:22.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:29:22.487 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:22.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:22.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:22.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:22.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:22.493 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:29:22.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.493 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:22.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:22.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:29:22.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:22.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:29:22.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:22.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:29:22.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:22.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:22.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:29:22.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:29:22.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:22.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:29:22.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:22.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:29:22.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:22.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:29:22.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:29:22.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.518 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:22.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.519 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:22.519 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:29:22.519 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.871737609999997, 2023-01-15T14:29:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:29:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:29:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:29:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:29:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:29:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:29:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:29:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:29:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:29:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:29:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:29:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:29:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:29:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:29:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:29:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:29:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:29:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:29:22.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:29:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:29:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:29:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:29:22.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:29:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:29:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:29:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:29:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:29:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:29:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.691 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:29:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.871334273999999, 2023-01-15T14:29:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:29:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:29:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:29:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:29:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:29:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:29:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:29:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:29:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:29:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:29:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:29:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:29:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:29:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:29:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:29:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:29:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:29:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:29:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:29:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:29:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:29:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:29:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:29:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:29:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:29:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:29:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:29:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:29:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:29:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:29:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:29:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:29:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:29:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:29:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:29:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.893 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.771110653999997, 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:22.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:22.895 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:29:22.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.895 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:22.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:22.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:29:22.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:22.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:29:22.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:22.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:29:22.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:22.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:22.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:29:22.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:29:22.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:22.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:29:22.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:29:22.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:22.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:29:22.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:29:22.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:29:22.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:29:22.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:22.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:29:22.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:29:22.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:29:22.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:29:22.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:22.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:29:22.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:29:22.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:29:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:29:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:193 3.8 3:1eaad226:::smithi09025264-49:head [write 406736~710070,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.294040+0000", 2023-01-15T14:29:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.706061759999997, 2023-01-15T14:29:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.54804442399999997, 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 193 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294040+0000", 2023-01-15T14:29:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294040+0000", 2023-01-15T14:29:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294037+0000", 2023-01-15T14:29:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294382+0000", 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034469500000000001 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294421+0000", 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9597000000000001e-05 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.294441+0000", 2023-01-15T14:29:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0000000000000002e-05 2023-01-15T14:29:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335292+0000", 2023-01-15T14:29:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040850274999999998 2023-01-15T14:29:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335605+0000", 2023-01-15T14:29:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031334499999999998 2023-01-15T14:29:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:29:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335831+0000", 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000225961 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.336189+0000", 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035799099999999999 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.336207+0000", 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7808e-05 2023-01-15T14:29:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.336471+0000", 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026395500000000002 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.336516+0000", 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5318999999999998e-05 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741140+0000", 2023-01-15T14:29:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40462405200000001 2023-01-15T14:29:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841833+0000", 2023-01-15T14:29:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100693281 2023-01-15T14:29:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841932+0000", 2023-01-15T14:29:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8387999999999994e-05 2023-01-15T14:29:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842084+0000", 2023-01-15T14:29:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015241699999999999 2023-01-15T14:29:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:194 3.8 3:1eaad226:::smithi09025264-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:29:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.698709411000003, 2023-01-15T14:29:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64372577200000003, 2023-01-15T14:29:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 194 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:29:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301390+0000", 2023-01-15T14:29:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301991+0000", 2023-01-15T14:29:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060122599999999995 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.302026+0000", 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5040000000000003e-05 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402147+0000", 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100121084 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402219+0000", 2023-01-15T14:29:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2045999999999994e-05 2023-01-15T14:29:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402315+0000", 2023-01-15T14:29:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5561000000000002e-05 2023-01-15T14:29:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842189+0000", 2023-01-15T14:29:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.439874129 2023-01-15T14:29:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842296+0000", 2023-01-15T14:29:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107021 2023-01-15T14:29:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:29:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842486+0000", 2023-01-15T14:29:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018965799999999999 2023-01-15T14:29:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842839+0000", 2023-01-15T14:29:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035330700000000002 2023-01-15T14:29:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842859+0000", 2023-01-15T14:29:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0347e-05 2023-01-15T14:29:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843212+0000", 2023-01-15T14:29:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352534 2023-01-15T14:29:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843835+0000", 2023-01-15T14:29:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062338599999999997 2023-01-15T14:29:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944861+0000", 2023-01-15T14:29:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10102596899999999 2023-01-15T14:29:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944908+0000", 2023-01-15T14:29:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6773000000000001e-05 2023-01-15T14:29:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945039+0000", 2023-01-15T14:29:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130684 2023-01-15T14:29:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945118+0000", 2023-01-15T14:29:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9194000000000001e-05 2023-01-15T14:29:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:195 3.8 3:1eaad226:::smithi09025264-49:head [setxattr _header (56) in=7b,truncate 2806736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.402188+0000", 2023-01-15T14:29:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.597913540999997, 2023-01-15T14:29:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60706172400000002, 2023-01-15T14:29:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:23.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:23.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:23.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:23.085 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 195 2023-01-15T14:29:23.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.085 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:23.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402188+0000", 2023-01-15T14:29:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402188+0000", 2023-01-15T14:29:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402186+0000", 2023-01-15T14:29:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:23.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402199+0000", 2023-01-15T14:29:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.24e-05 2023-01-15T14:29:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402222+0000", 2023-01-15T14:29:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-05 2023-01-15T14:29:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402233+0000", 2023-01-15T14:29:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2023-01-15T14:29:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402389+0000", 2023-01-15T14:29:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015541799999999999 2023-01-15T14:29:23.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402518+0000", 2023-01-15T14:29:23.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129534 2023-01-15T14:29:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842949+0000", 2023-01-15T14:29:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.440430722 2023-01-15T14:29:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843094+0000", 2023-01-15T14:29:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144955 2023-01-15T14:29:23.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:29:23.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843391+0000", 2023-01-15T14:29:23.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000297041 2023-01-15T14:29:23.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:23.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843731+0000", 2023-01-15T14:29:23.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033986200000000002 2023-01-15T14:29:23.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:23.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843747+0000", 2023-01-15T14:29:23.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6243999999999999e-05 2023-01-15T14:29:23.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:23.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843804+0000", 2023-01-15T14:29:23.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7240000000000001e-05 2023-01-15T14:29:23.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:23.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843883+0000", 2023-01-15T14:29:23.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9040000000000002e-05 2023-01-15T14:29:23.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:23.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945184+0000", 2023-01-15T14:29:23.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101300903 2023-01-15T14:29:23.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:23.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009101+0000", 2023-01-15T14:29:23.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063916599000000004 2023-01-15T14:29:23.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:23.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009176+0000", 2023-01-15T14:29:23.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5716999999999994e-05 2023-01-15T14:29:23.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:23.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009250+0000", 2023-01-15T14:29:23.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3213000000000005e-05 2023-01-15T14:29:23.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:23.435 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:23.435 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:23.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.435 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:196 3.8 3:1eaad226:::smithi09025264-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:23.435 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.402249+0000", 2023-01-15T14:29:23.435 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.597852695, 2023-01-15T14:29:23.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60743240399999998, 2023-01-15T14:29:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 196 2023-01-15T14:29:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:23.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402249+0000", 2023-01-15T14:29:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402249+0000", 2023-01-15T14:29:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402248+0000", 2023-01-15T14:29:23.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:23.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:23.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402253+0000", 2023-01-15T14:29:23.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8829999999999997e-06 2023-01-15T14:29:23.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:23.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402263+0000", 2023-01-15T14:29:23.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0013999999999999e-05 2023-01-15T14:29:23.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:23.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402269+0000", 2023-01-15T14:29:23.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1360000000000002e-06 2023-01-15T14:29:23.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:23.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402551+0000", 2023-01-15T14:29:23.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000281799 2023-01-15T14:29:23.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:23.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402620+0000", 2023-01-15T14:29:23.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9289999999999996e-05 2023-01-15T14:29:23.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:23.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.942361+0000", 2023-01-15T14:29:23.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.53974109800000003 2023-01-15T14:29:23.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:23.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.942474+0000", 2023-01-15T14:29:23.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112713 2023-01-15T14:29:23.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:23.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009383+0000", 2023-01-15T14:29:23.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.066909386000000001 2023-01-15T14:29:23.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:23.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009511+0000", 2023-01-15T14:29:23.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012762 2023-01-15T14:29:23.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:23.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009681+0000", 2023-01-15T14:29:23.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017044099999999999 2023-01-15T14:29:23.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:23.442 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:23.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:23.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:525 3.8 3:17d5d741:::smithi09025264-5:head [write 2555481~425187] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:29:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:29:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.181144485000001, 2023-01-15T14:29:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60814212099999998, 2023-01-15T14:29:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2023-01-15T14:29:23.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.444 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:23.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:23.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:29:23.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:23.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:23.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:29:23.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:23.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:23.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818955+0000", 2023-01-15T14:29:23.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:23.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:23.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819151+0000", 2023-01-15T14:29:23.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632699999999999 2023-01-15T14:29:23.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:23.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819175+0000", 2023-01-15T14:29:23.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3777e-05 2023-01-15T14:29:23.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:23.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819185+0000", 2023-01-15T14:29:23.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0424999999999999e-05 2023-01-15T14:29:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019555+0000", 2023-01-15T14:29:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20036944200000001 2023-01-15T14:29:23.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019726+0000", 2023-01-15T14:29:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017075099999999999 2023-01-15T14:29:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224068+0000", 2023-01-15T14:29:23.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20434244100000001 2023-01-15T14:29:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224217+0000", 2023-01-15T14:29:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014933200000000001 2023-01-15T14:29:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:29:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224571+0000", 2023-01-15T14:29:23.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035323700000000001 2023-01-15T14:29:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225080+0000", 2023-01-15T14:29:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050910500000000004 2023-01-15T14:29:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225103+0000", 2023-01-15T14:29:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3295999999999999e-05 2023-01-15T14:29:23.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225542+0000", 2023-01-15T14:29:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043897599999999999 2023-01-15T14:29:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226915+0000", 2023-01-15T14:29:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013730599999999999 2023-01-15T14:29:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.327618+0000", 2023-01-15T14:29:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100702655 2023-01-15T14:29:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:23.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426890+0000", 2023-01-15T14:29:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099272284000000002 2023-01-15T14:29:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426977+0000", 2023-01-15T14:29:23.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6500000000000002e-05 2023-01-15T14:29:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427099+0000", 2023-01-15T14:29:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122562 2023-01-15T14:29:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:23.713 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:526 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (56) in=7b,truncate 3512630] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:29:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:29:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.180896752999999, 2023-01-15T14:29:23.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60866722500000003, 2023-01-15T14:29:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2023-01-15T14:29:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:29:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:29:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:23.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819204+0000", 2023-01-15T14:29:23.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:23.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:23.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819210+0000", 2023-01-15T14:29:23.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5429999999999999e-06 2023-01-15T14:29:23.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:23.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819235+0000", 2023-01-15T14:29:23.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4114e-05 2023-01-15T14:29:23.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:23.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819246+0000", 2023-01-15T14:29:23.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1043000000000001e-05 2023-01-15T14:29:23.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:23.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019911+0000", 2023-01-15T14:29:23.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200665175 2023-01-15T14:29:23.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:23.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020085+0000", 2023-01-15T14:29:23.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000174449 2023-01-15T14:29:23.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:23.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225345+0000", 2023-01-15T14:29:23.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20526017099999999 2023-01-15T14:29:23.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:23.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225560+0000", 2023-01-15T14:29:23.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021505499999999999 2023-01-15T14:29:23.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:29:23.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225908+0000", 2023-01-15T14:29:23.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034754600000000001 2023-01-15T14:29:23.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:23.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226267+0000", 2023-01-15T14:29:23.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035936399999999998 2023-01-15T14:29:23.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:23.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226688+0000", 2023-01-15T14:29:23.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042024599999999999 2023-01-15T14:29:23.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:23.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226841+0000", 2023-01-15T14:29:23.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015324499999999999 2023-01-15T14:29:23.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:23.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226936+0000", 2023-01-15T14:29:23.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5549000000000003e-05 2023-01-15T14:29:23.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:23.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426826+0000", 2023-01-15T14:29:23.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19988924 2023-01-15T14:29:23.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:23.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427685+0000", 2023-01-15T14:29:23.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000859719 2023-01-15T14:29:23.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:23.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427781+0000", 2023-01-15T14:29:23.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5515999999999998e-05 2023-01-15T14:29:23.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:23.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:23.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:23.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427872+0000", 2023-01-15T14:29:23.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1142000000000003e-05 2023-01-15T14:29:24.000 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.000 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:24.000 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.000 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:527 3.8 3:17d5d741:::smithi09025264-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:29:24.000 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:29:24.000 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.180846265, 2023-01-15T14:29:24.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60906366599999995, 2023-01-15T14:29:24.008 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:24.008 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:24.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:24.009 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:24.009 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:24.009 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2023-01-15T14:29:24.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.009 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:24.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:24.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:29:24.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:24.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:29:24.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:24.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819259+0000", 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5940000000000004e-06 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819268+0000", 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.115e-06 2023-01-15T14:29:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:24.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.919405+0000", 2023-01-15T14:29:24.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013656999999999 2023-01-15T14:29:24.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020192+0000", 2023-01-15T14:29:24.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10078725600000001 2023-01-15T14:29:24.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:24.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020326+0000", 2023-01-15T14:29:24.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133745 2023-01-15T14:29:24.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226443+0000", 2023-01-15T14:29:24.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20611705699999999 2023-01-15T14:29:24.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:24.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226689+0000", 2023-01-15T14:29:24.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024580799999999997 2023-01-15T14:29:24.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427991+0000", 2023-01-15T14:29:24.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20130269000000001 2023-01-15T14:29:24.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.428147+0000", 2023-01-15T14:29:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155258 2023-01-15T14:29:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.428319+0000", 2023-01-15T14:29:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017226200000000001 2023-01-15T14:29:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:591 3.b 3:d85f42d9:::smithi09025264-45:head [write 1935000~798016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:29:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:59.812922+0000", 2023-01-15T14:29:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.187179045000001, 2023-01-15T14:29:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.56803569399999998, 2023-01-15T14:29:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 591 2023-01-15T14:29:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.812922+0000", 2023-01-15T14:29:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.812922+0000", 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.812920+0000", 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.813364+0000", 2023-01-15T14:29:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044398300000000001 2023-01-15T14:29:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.813388+0000", 2023-01-15T14:29:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4987e-05 2023-01-15T14:29:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913477+0000", 2023-01-15T14:29:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10008837900000001 2023-01-15T14:29:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913534+0000", 2023-01-15T14:29:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6663000000000003e-05 2023-01-15T14:29:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913645+0000", 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111069 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.010798+0000", 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097153758000000007 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.010910+0000", 2023-01-15T14:29:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111718 2023-01-15T14:29:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:29:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.011075+0000", 2023-01-15T14:29:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016526299999999999 2023-01-15T14:29:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.011404+0000", 2023-01-15T14:29:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032914000000000001 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.011420+0000", 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5973000000000001e-05 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.011755+0000", 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033445600000000001 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.012670+0000", 2023-01-15T14:29:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091509700000000002 2023-01-15T14:29:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.196470+0000", 2023-01-15T14:29:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.183800297 2023-01-15T14:29:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.380786+0000", 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.184316116 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.380883+0000", 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6671999999999999e-05 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.380958+0000", 2023-01-15T14:29:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5049000000000006e-05 2023-01-15T14:29:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:592 3.b 3:d85f42d9:::smithi09025264-45:head [write 3458512~188574] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:29:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:59.913516+0000", 2023-01-15T14:29:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.08658556, 2023-01-15T14:29:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.56750687600000005, 2023-01-15T14:29:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 592 2023-01-15T14:29:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913516+0000", 2023-01-15T14:29:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913516+0000", 2023-01-15T14:29:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913514+0000", 2023-01-15T14:29:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:24.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913613+0000", 2023-01-15T14:29:24.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9579000000000003e-05 2023-01-15T14:29:24.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:24.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913631+0000", 2023-01-15T14:29:24.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8082999999999999e-05 2023-01-15T14:29:24.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:24.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913643+0000", 2023-01-15T14:29:24.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1174e-05 2023-01-15T14:29:24.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913689+0000", 2023-01-15T14:29:24.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6672999999999999e-05 2023-01-15T14:29:24.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913755+0000", 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5283000000000005e-05 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.011782+0000", 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098027056000000001 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.011968+0000", 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018658300000000001 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.012276+0000", 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030728999999999999 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.012594+0000", 2023-01-15T14:29:24.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031828200000000003 2023-01-15T14:29:24.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:24.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.012609+0000", 2023-01-15T14:29:24.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5291999999999999e-05 2023-01-15T14:29:24.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:24.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.012854+0000", 2023-01-15T14:29:24.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024511800000000001 2023-01-15T14:29:24.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.013296+0000", 2023-01-15T14:29:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000441996 2023-01-15T14:29:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.296759+0000", 2023-01-15T14:29:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.283462936 2023-01-15T14:29:24.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.480792+0000", 2023-01-15T14:29:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18403307099999999 2023-01-15T14:29:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.480878+0000", 2023-01-15T14:29:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6186000000000005e-05 2023-01-15T14:29:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.481023+0000", 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014437200000000001 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:593 3.b 3:d85f42d9:::smithi09025264-45:head [setxattr _header (57) in=7b,truncate 3647086] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:59.913670+0000", 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.086431695999998, 2023-01-15T14:29:24.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.567873721, 2023-01-15T14:29:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 593 2023-01-15T14:29:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:24.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913670+0000", 2023-01-15T14:29:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913670+0000", 2023-01-15T14:29:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:24.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913669+0000", 2023-01-15T14:29:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:24.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913677+0000", 2023-01-15T14:29:24.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4600000000000003e-06 2023-01-15T14:29:24.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:24.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913690+0000", 2023-01-15T14:29:24.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2738000000000001e-05 2023-01-15T14:29:24.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:24.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913698+0000", 2023-01-15T14:29:24.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3440000000000001e-06 2023-01-15T14:29:24.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913791+0000", 2023-01-15T14:29:24.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2489000000000002e-05 2023-01-15T14:29:24.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:24.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913871+0000", 2023-01-15T14:29:24.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0662999999999997e-05 2023-01-15T14:29:24.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.012738+0000", 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098866409000000002 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.012910+0000", 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017177199999999999 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:29:24.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.013056+0000", 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000146766 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.013233+0000", 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017713399999999999 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.013242+0000", 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3299999999999999e-06 2023-01-15T14:29:24.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:24.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.015853+0000", 2023-01-15T14:29:24.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026108440000000002 2023-01-15T14:29:24.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:24.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.015883+0000", 2023-01-15T14:29:24.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0412e-05 2023-01-15T14:29:24.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:24.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.481234+0000", 2023-01-15T14:29:24.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.46535090699999998 2023-01-15T14:29:24.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:24.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.481323+0000", 2023-01-15T14:29:24.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9152000000000003e-05 2023-01-15T14:29:24.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:24.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.481397+0000", 2023-01-15T14:29:24.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4009000000000002e-05 2023-01-15T14:29:24.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:24.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.481544+0000", 2023-01-15T14:29:24.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000146448 2023-01-15T14:29:24.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.543 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:24.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.543 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:594 3.b 3:d85f42d9:::smithi09025264-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:29:24.543 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:59.913715+0000", 2023-01-15T14:29:24.543 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.086386796999999, 2023-01-15T14:29:24.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.568521944, 2023-01-15T14:29:24.543 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:24.543 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:24.544 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:24.544 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:24.544 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:24.544 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 594 2023-01-15T14:29:24.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.544 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:24.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:24.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913715+0000", 2023-01-15T14:29:24.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:24.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913715+0000", 2023-01-15T14:29:24.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:24.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913714+0000", 2023-01-15T14:29:24.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:24.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913720+0000", 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5250000000000002e-06 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913733+0000", 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2993000000000001e-05 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:24.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913742+0000", 2023-01-15T14:29:24.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:29:24.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913907+0000", 2023-01-15T14:29:24.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165255 2023-01-15T14:29:24.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:24.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:59.913982+0000", 2023-01-15T14:29:24.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4923000000000001e-05 2023-01-15T14:29:24.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.111572+0000", 2023-01-15T14:29:24.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.197589452 2023-01-15T14:29:24.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:24.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.111717+0000", 2023-01-15T14:29:24.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014562200000000001 2023-01-15T14:29:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.481665+0000", 2023-01-15T14:29:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36994745200000001 2023-01-15T14:29:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.481951+0000", 2023-01-15T14:29:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028675899999999998 2023-01-15T14:29:24.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:24.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.482237+0000", 2023-01-15T14:29:24.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000285297 2023-01-15T14:29:24.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.550 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:24.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.551 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:642 3.8 3:1eaad226:::smithi09025264-49:head [copy-from ver 34] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2023-01-15T14:29:24.551 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:02.735949+0000", 2023-01-15T14:29:24.551 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.264152574000001, 2023-01-15T14:29:24.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58633297799999995, 2023-01-15T14:29:24.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:24.551 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:24.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:24.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:24.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:24.552 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 642 2023-01-15T14:29:24.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.552 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:24.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:24.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.735949+0000", 2023-01-15T14:29:24.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:24.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.735949+0000", 2023-01-15T14:29:24.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:24.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.735947+0000", 2023-01-15T14:29:24.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:24.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:24.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.735956+0000", 2023-01-15T14:29:24.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1560000000000001e-06 2023-01-15T14:29:24.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:24.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.735976+0000", 2023-01-15T14:29:24.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0390999999999998e-05 2023-01-15T14:29:24.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:24.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.735988+0000", 2023-01-15T14:29:24.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1219999999999999e-05 2023-01-15T14:29:24.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736117+0000", 2023-01-15T14:29:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012902899999999999 2023-01-15T14:29:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736134+0000", 2023-01-15T14:29:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7136999999999999e-05 2023-01-15T14:29:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.014258+0000", 2023-01-15T14:29:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27812434200000002 2023-01-15T14:29:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.014439+0000", 2023-01-15T14:29:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018088100000000001 2023-01-15T14:29:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:29:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.152305+0000", 2023-01-15T14:29:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137866287 2023-01-15T14:29:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.152728+0000", 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042283000000000002 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.152749+0000", 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0553000000000001e-05 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.153897+0000", 2023-01-15T14:29:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001148505 2023-01-15T14:29:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.154045+0000", 2023-01-15T14:29:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764800000000001 2023-01-15T14:29:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.163266+0000", 2023-01-15T14:29:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0092209460000000007 2023-01-15T14:29:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.322139+0000", 2023-01-15T14:29:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.158873234 2023-01-15T14:29:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.322206+0000", 2023-01-15T14:29:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7515999999999995e-05 2023-01-15T14:29:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.322282+0000", 2023-01-15T14:29:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5425000000000004e-05 2023-01-15T14:29:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:643 3.8 3:1eaad226:::smithi09025264-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2023-01-15T14:29:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:02.736009+0000", 2023-01-15T14:29:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.264092385000001, 2023-01-15T14:29:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58661375400000004, 2023-01-15T14:29:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 643 2023-01-15T14:29:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736009+0000", 2023-01-15T14:29:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736009+0000", 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736008+0000", 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736014+0000", 2023-01-15T14:29:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7309999999999998e-06 2023-01-15T14:29:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736023+0000", 2023-01-15T14:29:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.704e-06 2023-01-15T14:29:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736028+0000", 2023-01-15T14:29:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3630000000000002e-06 2023-01-15T14:29:24.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.836277+0000", 2023-01-15T14:29:24.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100248614 2023-01-15T14:29:24.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:24.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.836291+0000", 2023-01-15T14:29:24.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2023-01-15T14:29:24.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.014657+0000", 2023-01-15T14:29:24.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.178365366 2023-01-15T14:29:24.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:29:24.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.014802+0000", 2023-01-15T14:29:24.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014527799999999999 2023-01-15T14:29:24.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.152824+0000", 2023-01-15T14:29:24.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13802230900000001 2023-01-15T14:29:24.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.152918+0000", 2023-01-15T14:29:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4104999999999994e-05 2023-01-15T14:29:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.322349+0000", 2023-01-15T14:29:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16943072200000001 2023-01-15T14:29:24.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:24.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.322513+0000", 2023-01-15T14:29:24.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016396299999999999 2023-01-15T14:29:24.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:24.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.322623+0000", 2023-01-15T14:29:24.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109978 2023-01-15T14:29:24.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.963 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:24.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.963 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:705 3.2 3:47dfe935:::smithi09025264-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2023-01-15T14:29:24.964 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:29:24.964 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 12.889517503, 2023-01-15T14:29:24.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61170208599999998, 2023-01-15T14:29:24.964 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:24.964 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:24.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:24.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:24.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:24.964 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 705 2023-01-15T14:29:24.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.965 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:24.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:24.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:29:24.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:24.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:29:24.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:24.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:24.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110583+0000", 2023-01-15T14:29:24.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:24.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:24.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110587+0000", 2023-01-15T14:29:24.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9689999999999996e-06 2023-01-15T14:29:24.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:24.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110601+0000", 2023-01-15T14:29:24.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.393e-05 2023-01-15T14:29:24.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:24.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110611+0000", 2023-01-15T14:29:24.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3999999999999998e-06 2023-01-15T14:29:24.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319174+0000", 2023-01-15T14:29:24.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20856376800000001 2023-01-15T14:29:24.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:24.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319367+0000", 2023-01-15T14:29:24.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192361 2023-01-15T14:29:24.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:24.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.721941+0000", 2023-01-15T14:29:24.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40257461300000003 2023-01-15T14:29:24.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:24.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722108+0000", 2023-01-15T14:29:24.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167054 2023-01-15T14:29:24.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:24.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:24.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:24.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722286+0000", 2023-01-15T14:29:24.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017772099999999999 2023-01-15T14:29:24.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:24.976 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:25.148 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:25.149 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:29:25.150 INFO:tasks.rados.rados.0.smithi090.stdout:300: done (0 left) 2023-01-15T14:29:25.150 INFO:tasks.rados.rados.0.smithi090.stdout:301: snap_remove snap 24 2023-01-15T14:29:25.153 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:22.065+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:25.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:29:25.156 INFO:tasks.rados.rados.0.smithi090.stdout:301: done (0 left) 2023-01-15T14:29:25.156 INFO:tasks.rados.rados.0.smithi090.stdout:302: rollback oid 31 current snap is 27 2023-01-15T14:29:25.156 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 31 to 23 2023-01-15T14:29:25.156 INFO:tasks.rados.rados.0.smithi090.stdout:303: rollback oid 19 current snap is 27 2023-01-15T14:29:25.156 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 19 to 25 2023-01-15T14:29:25.156 INFO:tasks.rados.rados.0.smithi090.stdout:304: write (excl) oid 26 current snap is 27 2023-01-15T14:29:25.157 INFO:tasks.rados.rados.0.smithi090.stdout:304: seq_num 141 ranges {769787=704463,2267961=101826} 2023-01-15T14:29:25.157 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:29:25.158 INFO:tasks.rados.rados.0.smithi090.stdout:304: writing smithi09025264-26 from 769787 to 1474250 tid 1 2023-01-15T14:29:25.158 INFO:tasks.rados.rados.0.smithi090.stdout:304: writing smithi09025264-26 from 2267961 to 2369787 tid 2 2023-01-15T14:29:25.159 INFO:tasks.rados.rados.0.smithi090.stdout:305: read oid 4 snap -1 2023-01-15T14:29:25.159 INFO:tasks.rados.rados.0.smithi090.stdout:305: expect (ObjNum 120 snap 23 seq_num 120) 2023-01-15T14:29:25.159 INFO:tasks.rados.rados.0.smithi090.stdout:306: copy_from oid 22 from oid 30 current snap is 27 2023-01-15T14:29:25.159 INFO:tasks.rados.rados.0.smithi090.stdout:302: finishing rollback tid 1 to smithi09025264-31 2023-01-15T14:29:25.362 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 0 (ObjNum 129 snap 0 seq_num 3191840240) dirty dne 2023-01-15T14:29:25.364 INFO:tasks.rados.rados.0.smithi090.stdout:302: done (4 left) 2023-01-15T14:29:25.364 INFO:tasks.rados.rados.0.smithi090.stdout:307: rollback oid 11 current snap is 27 2023-01-15T14:29:25.364 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 11 to 25 2023-01-15T14:29:25.365 INFO:tasks.rados.rados.0.smithi090.stdout:308: delete oid 3 current snap is 27 2023-01-15T14:29:25.372 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:25.565 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:29:25.566 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:29:25.566 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:25.567 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:29:25.763 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:29:25.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-01-15T14:29:25.765 INFO:tasks.rados.rados.0.smithi090.stdout:307: finishing rollback tid 1 to smithi09025264-11 2023-01-15T14:29:25.765 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 0 (ObjNum 129 snap 0 seq_num 3192865016) dirty dne 2023-01-15T14:29:25.765 INFO:tasks.rados.rados.0.smithi090.stdout:304: finishing write tid 1 to smithi09025264-26 2023-01-15T14:29:25.765 INFO:tasks.rados.rados.0.smithi090.stdout:305: expect (ObjNum 120 snap 23 seq_num 120) 2023-01-15T14:29:25.765 INFO:tasks.rados.rados.0.smithi090.stdout:303: finishing rollback tid 0 to smithi09025264-19 2023-01-15T14:29:25.765 INFO:tasks.rados.rados.0.smithi090.stdout:303: finishing rollback tid 1 to smithi09025264-19 2023-01-15T14:29:25.765 INFO:tasks.rados.rados.0.smithi090.stdout:303: finishing rollback tid 2 to smithi09025264-19 2023-01-15T14:29:25.765 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 64 (ObjNum 87 snap 12 seq_num 87) dirty exists 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:303: done (5 left) 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:305: done (4 left) 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:307: done (3 left) 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:308: done (2 left) 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:309: write oid 1 current snap is 27 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:309: seq_num 143 ranges {403401=454961,1584925=506039} 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:309: writing smithi09025264-1 from 403401 to 858362 tid 1 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:309: writing smithi09025264-1 from 1584925 to 2090964 tid 2 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:310: snap_create 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:304: finishing write tid 2 to smithi09025264-26 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:306: finishing copy_from to smithi09025264-22 2023-01-15T14:29:25.766 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-01-15T14:29:25.767 INFO:tasks.rados.rados.0.smithi090.stdout:306: finishing copy_from racing read to smithi09025264-22 2023-01-15T14:29:25.767 INFO:tasks.rados.rados.0.smithi090.stdout:306: done (2 left) 2023-01-15T14:29:25.767 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:25.767 INFO:tasks.rados.rados.0.smithi090.stdout:304: finishing write tid 3 to smithi09025264-26 2023-01-15T14:29:25.767 INFO:tasks.rados.rados.0.smithi090.stdout:304: finishing write tid 4 to smithi09025264-26 2023-01-15T14:29:25.767 INFO:tasks.rados.rados.0.smithi090.stdout:304: oid 26 updating version 0 to 145 2023-01-15T14:29:25.767 INFO:tasks.rados.rados.0.smithi090.stdout:304: oid 26 version 145 is already newer than 141 2023-01-15T14:29:25.767 INFO:tasks.rados.rados.0.smithi090.stdout:304: oid 26 version 145 is already newer than 144 2023-01-15T14:29:25.767 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 145 (ObjNum 141 snap 27 seq_num 141) dirty exists 2023-01-15T14:29:25.767 INFO:tasks.rados.rados.0.smithi090.stdout:304: left oid 26 (ObjNum 141 snap 27 seq_num 141) 2023-01-15T14:29:25.768 INFO:tasks.rados.rados.0.smithi090.stdout:304: done (1 left) 2023-01-15T14:29:25.768 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:25.769 INFO:tasks.rados.rados.0.smithi090.stdout:309: finishing write tid 1 to smithi09025264-1 2023-01-15T14:29:25.775 INFO:tasks.rados.rados.0.smithi090.stdout:309: finishing write tid 2 to smithi09025264-1 2023-01-15T14:29:25.776 INFO:tasks.rados.rados.0.smithi090.stdout:309: finishing write tid 3 to smithi09025264-1 2023-01-15T14:29:25.776 INFO:tasks.rados.rados.0.smithi090.stdout:309: finishing write tid 4 to smithi09025264-1 2023-01-15T14:29:25.776 INFO:tasks.rados.rados.0.smithi090.stdout:309: oid 1 updating version 0 to 149 2023-01-15T14:29:25.776 INFO:tasks.rados.rados.0.smithi090.stdout:309: oid 1 version 149 is already newer than 148 2023-01-15T14:29:25.776 INFO:tasks.rados.rados.0.smithi090.stdout:309: oid 1 version 149 is already newer than 147 2023-01-15T14:29:25.776 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 149 (ObjNum 143 snap 27 seq_num 143) dirty exists 2023-01-15T14:29:25.776 INFO:tasks.rados.rados.0.smithi090.stdout:309: left oid 1 (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout:309: done (0 left) 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout:310: done (0 left) 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout:311: rollback oid 8 current snap is 28 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 8 to 23 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout:312: snap_create 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout:311: finishing rollback tid 0 to smithi09025264-8 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout:311: finishing rollback tid 1 to smithi09025264-8 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout:311: finishing rollback tid 2 to smithi09025264-8 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 77 (ObjNum 79 snap 10 seq_num 79) dirty exists 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout:311: done (0 left) 2023-01-15T14:29:25.777 INFO:tasks.rados.rados.0.smithi090.stdout:312: done (0 left) 2023-01-15T14:29:25.778 INFO:tasks.rados.rados.0.smithi090.stdout:313: snap_remove snap 28 2023-01-15T14:29:25.778 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:25.149+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:25.779 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:25.361+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:25.779 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:25.757+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:25.864 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:25.857+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:25.975 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:25.975 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:25.975 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:25.976 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:25.965+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:25.976 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:25.976 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:29:25.977 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:29:25.977 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:29:25.977 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:25.978 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:29:25.982 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:26.003 INFO:tasks.rados.rados.0.smithi090.stdout:313: done (0 left) 2023-01-15T14:29:26.003 INFO:tasks.rados.rados.0.smithi090.stdout:314: copy_from oid 29 from oid 45 current snap is 29 2023-01-15T14:29:26.003 INFO:tasks.rados.rados.0.smithi090.stdout:315: rollback oid 34 current snap is 29 2023-01-15T14:29:26.003 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 34 to 23 2023-01-15T14:29:26.003 INFO:tasks.rados.rados.0.smithi090.stdout:316: read oid 13 snap -1 2023-01-15T14:29:26.003 INFO:tasks.rados.rados.0.smithi090.stdout:316: expect (ObjNum 133 snap 24 seq_num 133) 2023-01-15T14:29:26.004 INFO:tasks.rados.rados.0.smithi090.stdout:317: read oid 26 snap -1 2023-01-15T14:29:26.004 INFO:tasks.rados.rados.0.smithi090.stdout:317: expect (ObjNum 141 snap 27 seq_num 141) 2023-01-15T14:29:26.004 INFO:tasks.rados.rados.0.smithi090.stdout:318: write oid 17 current snap is 29 2023-01-15T14:29:26.004 INFO:tasks.rados.rados.0.smithi090.stdout:318: seq_num 144 ranges {420208=530210,1527054=766050} 2023-01-15T14:29:26.004 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:26.009 INFO:tasks.rados.rados.0.smithi090.stdout:318: writing smithi09025264-17 from 420208 to 950418 tid 1 2023-01-15T14:29:26.182 INFO:tasks.rados.rados.0.smithi090.stdout:318: writing smithi09025264-17 from 1527054 to 2293104 tid 2 2023-01-15T14:29:26.182 INFO:tasks.rados.rados.0.smithi090.stdout:319: snap_remove snap 25 2023-01-15T14:29:26.182 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:26.183 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:26.183 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:26.183 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:26.183 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:26.183 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:29:26.184 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:26.177+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:26.184 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:29:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:29:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:29:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:29:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:29:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:29:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:29:26.185 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:29:26.282 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:26.277+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:26.335 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:26.336 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:26.348 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:29:26.382 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:26.377+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:26.449 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:26.450 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:26.460 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:29:26.483 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:26.477+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:26.563 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:26.570 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:26.575 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:29:26.584 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:26.577+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:26.683 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:26.785 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:26.786 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:29:26.787 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:26.781+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:26.887 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:26.885+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:27.013 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:27.013 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:27.013 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:29:27.014 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:27.005+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:27.018 INFO:tasks.rados.rados.0.smithi090.stdout:319: done (5 left) 2023-01-15T14:29:27.018 INFO:tasks.rados.rados.0.smithi090.stdout:320: read oid 39 snap 23 2023-01-15T14:29:27.018 INFO:tasks.rados.rados.0.smithi090.stdout:320: expect (ObjNum 121 snap 23 seq_num 121) 2023-01-15T14:29:27.018 INFO:tasks.rados.rados.0.smithi090.stdout:321: delete oid 31 current snap is 29 2023-01-15T14:29:27.018 INFO:tasks.rados.rados.0.smithi090.stdout:316: expect (ObjNum 133 snap 24 seq_num 133) 2023-01-15T14:29:27.028 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.0 to 1 (10000) 2023-01-15T14:29:27.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-01-15T14:29:27.113 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:27.105+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:27.201 INFO:tasks.rados.rados.0.smithi090.stdout:317: expect (ObjNum 141 snap 27 seq_num 141) 2023-01-15T14:29:27.202 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:27.202 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:27.202 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:29:27.206 INFO:tasks.rados.rados.0.smithi090.stdout:314: finishing copy_from to smithi09025264-29 2023-01-15T14:29:27.206 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 124 (ObjNum 132 snap 24 seq_num 132) dirty exists 2023-01-15T14:29:27.206 INFO:tasks.rados.rados.0.smithi090.stdout:314: finishing copy_from racing read to smithi09025264-29 2023-01-15T14:29:27.206 INFO:tasks.rados.rados.0.smithi090.stdout:315: finishing rollback tid 0 to smithi09025264-34 2023-01-15T14:29:27.206 INFO:tasks.rados.rados.0.smithi090.stdout:318: finishing write tid 1 to smithi09025264-17 2023-01-15T14:29:27.206 INFO:tasks.rados.rados.0.smithi090.stdout:318: finishing write tid 2 to smithi09025264-17 2023-01-15T14:29:27.206 INFO:tasks.rados.rados.0.smithi090.stdout:315: finishing rollback tid 1 to smithi09025264-34 2023-01-15T14:29:27.206 INFO:tasks.rados.rados.0.smithi090.stdout:318: finishing write tid 3 to smithi09025264-17 2023-01-15T14:29:27.207 INFO:tasks.rados.rados.0.smithi090.stdout:315: finishing rollback tid 2 to smithi09025264-34 2023-01-15T14:29:27.207 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 62 (ObjNum 74 snap 8 seq_num 74) dirty exists 2023-01-15T14:29:27.207 INFO:tasks.rados.rados.0.smithi090.stdout:318: finishing write tid 4 to smithi09025264-17 2023-01-15T14:29:27.207 INFO:tasks.rados.rados.0.smithi090.stdout:318: oid 17 updating version 0 to 153 2023-01-15T14:29:27.207 INFO:tasks.rados.rados.0.smithi090.stdout:318: oid 17 updating version 153 to 154 2023-01-15T14:29:27.207 INFO:tasks.rados.rados.0.smithi090.stdout:318: oid 17 version 154 is already newer than 152 2023-01-15T14:29:27.207 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 154 (ObjNum 144 snap 29 seq_num 144) dirty exists 2023-01-15T14:29:27.207 INFO:tasks.rados.rados.0.smithi090.stdout:318: left oid 17 (ObjNum 144 snap 29 seq_num 144) 2023-01-15T14:29:27.216 INFO:tasks.rados.rados.0.smithi090.stdout:314: done (6 left) 2023-01-15T14:29:27.216 INFO:tasks.rados.rados.0.smithi090.stdout:315: done (5 left) 2023-01-15T14:29:27.216 INFO:tasks.rados.rados.0.smithi090.stdout:316: done (4 left) 2023-01-15T14:29:27.216 INFO:tasks.rados.rados.0.smithi090.stdout:317: done (3 left) 2023-01-15T14:29:27.216 INFO:tasks.rados.rados.0.smithi090.stdout:318: done (2 left) 2023-01-15T14:29:27.216 INFO:tasks.rados.rados.0.smithi090.stdout:321: done (1 left) 2023-01-15T14:29:27.216 INFO:tasks.rados.rados.0.smithi090.stdout:322: snap_remove snap 27 2023-01-15T14:29:27.250 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:27.245+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:27.310 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:27.310 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:27.315 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:27.309+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:27.320 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:29:27.367 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:29:27.416 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:27.409+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:27.423 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:27.423 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:27.433 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:29:27.517 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:27.513+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:27.540 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:27.540 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:27.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:29:27.618 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:27.613+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:27.658 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:27.809 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:27.811 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:29:27.811 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:27.805+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:27.911 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:27.905+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:28.021 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:28.021 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:28.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:29:28.022 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:28.013+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:28.041 INFO:tasks.rados.rados.0.smithi090.stdout:322: done (1 left) 2023-01-15T14:29:28.041 INFO:tasks.rados.rados.0.smithi090.stdout:323: delete oid 35 current snap is 29 2023-01-15T14:29:28.041 INFO:tasks.rados.rados.0.smithi090.stdout:320: expect (ObjNum 121 snap 23 seq_num 121) 2023-01-15T14:29:28.045 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.1 to 1 (10000) 2023-01-15T14:29:28.046 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:29:28.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-01-15T14:29:28.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:29:28.208 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:28.208 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:28.208 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:29:28.209 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:28.201+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:28.308 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:28.301+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:28.345 INFO:tasks.rados.rados.0.smithi090.stdout:320: done (1 left) 2023-01-15T14:29:28.408 INFO:tasks.rados.rados.0.smithi090.stdout:323: done (0 left) 2023-01-15T14:29:28.408 INFO:tasks.rados.rados.0.smithi090.stdout:324: snap_create 2023-01-15T14:29:28.410 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:28.410 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:28.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:29:28.411 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:28.411 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:28.412 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:28.405+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:28.431 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:28.451 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:28.472 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:28.501 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:28.510 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:28.505+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:28.521 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:28.535 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:29:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:29:28.789 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:28.790 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:28.790 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:29:28.790 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:29:28.790 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:29:28.790 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:29:28.790 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:29:28.791 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:29:28.791 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:29:28.791 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:29:28.792 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:29:28.793 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:28.785+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:28.891 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:28.885+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:28.971 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:28.971 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:29:28.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:29:28.972 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:28.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.972 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:68 3.0 3:0b54190a:::smithi09025264-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:28.972 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.223534+0000", 2023-01-15T14:29:28.972 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 62.723728551000001, 2023-01-15T14:29:28.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49993064500000001, 2023-01-15T14:29:28.974 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:28.974 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:28.974 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:28.974 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:28.974 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:28.974 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 68 2023-01-15T14:29:28.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.975 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:28.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:28.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223534+0000", 2023-01-15T14:29:28.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:28.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:28.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223534+0000", 2023-01-15T14:29:28.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:28.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:28.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223533+0000", 2023-01-15T14:29:28.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:28.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:28.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223539+0000", 2023-01-15T14:29:28.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9970000000000004e-06 2023-01-15T14:29:28.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:28.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223556+0000", 2023-01-15T14:29:28.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6543000000000001e-05 2023-01-15T14:29:28.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:28.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223567+0000", 2023-01-15T14:29:28.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0893999999999999e-05 2023-01-15T14:29:28.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:28.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.520962+0000", 2023-01-15T14:29:28.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29739558300000002 2023-01-15T14:29:28.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:28.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.521142+0000", 2023-01-15T14:29:28.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017972100000000001 2023-01-15T14:29:28.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:28.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.723135+0000", 2023-01-15T14:29:28.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20199320400000001 2023-01-15T14:29:28.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:28.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.723274+0000", 2023-01-15T14:29:28.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013887700000000001 2023-01-15T14:29:28.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:28.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.723465+0000", 2023-01-15T14:29:28.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019079399999999999 2023-01-15T14:29:28.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:28.985 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:28.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:28.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.986 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:94 3.0 3:07a51d33:::smithi09025264-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:28.986 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:28.986 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 62.507723308999999, 2023-01-15T14:29:28.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50760159199999999, 2023-01-15T14:29:28.986 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:28.986 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:28.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:28.987 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:28.987 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:28.987 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 94 2023-01-15T14:29:28.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.987 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:28.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:28.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:28.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:28.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:28.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:28.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:28.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:28.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439538+0000", 2023-01-15T14:29:28.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:28.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:28.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439857+0000", 2023-01-15T14:29:28.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000319086 2023-01-15T14:29:28.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:28.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439891+0000", 2023-01-15T14:29:28.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4449999999999997e-05 2023-01-15T14:29:28.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:28.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:28.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:28.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439906+0000", 2023-01-15T14:29:28.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5082999999999999e-05 2023-01-15T14:29:29.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:29.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824189+0000", 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38428238100000001 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824440+0000", 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025139800000000001 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824834+0000", 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039422200000000001 2023-01-15T14:29:29.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:29.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825347+0000", 2023-01-15T14:29:29.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051323600000000003 2023-01-15T14:29:29.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:29.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825364+0000", 2023-01-15T14:29:29.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6478e-05 2023-01-15T14:29:29.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:29.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825589+0000", 2023-01-15T14:29:29.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022497500000000001 2023-01-15T14:29:29.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:29.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826457+0000", 2023-01-15T14:29:29.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00086797600000000001 2023-01-15T14:29:29.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:29.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.846539+0000", 2023-01-15T14:29:29.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020081742 2023-01-15T14:29:29.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:29.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.946989+0000", 2023-01-15T14:29:29.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045032199999999 2023-01-15T14:29:29.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:29.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947076+0000", 2023-01-15T14:29:29.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6922999999999994e-05 2023-01-15T14:29:29.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:29.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947141+0000", 2023-01-15T14:29:29.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5357000000000005e-05 2023-01-15T14:29:29.153 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.153 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:95 3.0 3:07a51d33:::smithi09025264-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439927+0000", 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 62.507335703999999, 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50742387499999997, 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:29.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:29.155 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 95 2023-01-15T14:29:29.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.155 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:29.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439927+0000", 2023-01-15T14:29:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439927+0000", 2023-01-15T14:29:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:29.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439926+0000", 2023-01-15T14:29:29.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439933+0000", 2023-01-15T14:29:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2999999999999998e-06 2023-01-15T14:29:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439947+0000", 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4071e-05 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439953+0000", 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4160000000000001e-06 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825479+0000", 2023-01-15T14:29:29.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.385525544 2023-01-15T14:29:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825576+0000", 2023-01-15T14:29:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7342000000000004e-05 2023-01-15T14:29:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825814+0000", 2023-01-15T14:29:29.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023816700000000001 2023-01-15T14:29:29.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:29.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826048+0000", 2023-01-15T14:29:29.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000234019 2023-01-15T14:29:29.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826067+0000", 2023-01-15T14:29:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9252000000000001e-05 2023-01-15T14:29:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826137+0000", 2023-01-15T14:29:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0004000000000003e-05 2023-01-15T14:29:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:29.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.828120+0000", 2023-01-15T14:29:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019824619999999999 2023-01-15T14:29:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.846644+0000", 2023-01-15T14:29:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018524134000000001 2023-01-15T14:29:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947206+0000", 2023-01-15T14:29:29.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100562254 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947279+0000", 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2992999999999997e-05 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947351+0000", 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1847000000000005e-05 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.339 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:96 3.0 3:07a51d33:::smithi09025264-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 62.507301196, 2023-01-15T14:29:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50769745899999996, 2023-01-15T14:29:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:29.340 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 96 2023-01-15T14:29:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:29.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439961+0000", 2023-01-15T14:29:29.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:29.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:29.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439966+0000", 2023-01-15T14:29:29.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0769999999999998e-06 2023-01-15T14:29:29.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:29.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439979+0000", 2023-01-15T14:29:29.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3343000000000001e-05 2023-01-15T14:29:29.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:29.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439987+0000", 2023-01-15T14:29:29.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2400000000000001e-06 2023-01-15T14:29:29.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:29.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826191+0000", 2023-01-15T14:29:29.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38620452799999999 2023-01-15T14:29:29.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:29.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826424+0000", 2023-01-15T14:29:29.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023264800000000001 2023-01-15T14:29:29.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:29.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947394+0000", 2023-01-15T14:29:29.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120970102 2023-01-15T14:29:29.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:29.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947505+0000", 2023-01-15T14:29:29.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001105 2023-01-15T14:29:29.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:29.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947659+0000", 2023-01-15T14:29:29.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015468799999999999 2023-01-15T14:29:29.506 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.506 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:29.506 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.506 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:29.506 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:29.506 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 61.862748531999998, 2023-01-15T14:29:29.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:29.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:29.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:29.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:29.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:29.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:29.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:29.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:29.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:29.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:29:29.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:29:29.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:29.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:29:29.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:29:29.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:29.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:29:29.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:29:29.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:29.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:29:29.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:29:29.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:29.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:29:29.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:29:29.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:29.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:29:29.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:29:29.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:29.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:29:29.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:29:29.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:29.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:29:29.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:29:29.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:29.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:29:29.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:29:29.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:29.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:29:29.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:29:29.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:29.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:29:29.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:29:29.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:29.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:29:29.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:29:29.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:29.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:29:29.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:29:29.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:29.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:29:29.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:29:29.682 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.682 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:29.682 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.682 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:29.682 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:29.683 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 61.862448735000001, 2023-01-15T14:29:29.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:29:29.683 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:29.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:29.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:29.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:29.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:29.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:29:29.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:29.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:29.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:29:29.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:29:29.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:29.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:29:29.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:29:29.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:29.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:29:29.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:29:29.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:29.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:29:29.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:29:29.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:29.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:29:29.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:29:29.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:29.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:29:29.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:29:29.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:29.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:29:29.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:29:29.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:29.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:29:29.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:29:29.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:29.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:29:29.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:29:29.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:29.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:29:29.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:29:29.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:29.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:29:29.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:29:29.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:29.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:29:29.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:29:29.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:29.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:29:29.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:29:29.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:29.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:29:29.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 61.862418951000002, 2023-01-15T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:29.857 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:29.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:29.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:29.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:29.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:29:29.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:29.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:29:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:29:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:29:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:29:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:29:29.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:29:29.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:29.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:29:29.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:29:29.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:29.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:29:29.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:29:29.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:29.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:29:29.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:29:29.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:29.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:182 3.4 3:2022fdfa:::smithi09025264-46:head [write 1933572~791670] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 61.733556882000002, 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.582304865, 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 182 2023-01-15T14:29:29.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:29.866 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:29.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:29.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:29.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:30.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:30.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:30.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:30.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213704+0000", 2023-01-15T14:29:30.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:30.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:30.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214574+0000", 2023-01-15T14:29:30.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087037799999999999 2023-01-15T14:29:30.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:30.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214608+0000", 2023-01-15T14:29:30.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:29:30.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:30.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214627+0000", 2023-01-15T14:29:30.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491537+0000", 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27691020300000002 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491721+0000", 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345500000000001 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693760+0000", 2023-01-15T14:29:30.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202039252 2023-01-15T14:29:30.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:30.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693898+0000", 2023-01-15T14:29:30.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013789 2023-01-15T14:29:30.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694230+0000", 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220899999999998 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694866+0000", 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063640599999999995 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694887+0000", 2023-01-15T14:29:30.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0622999999999999e-05 2023-01-15T14:29:30.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:30.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695950+0000", 2023-01-15T14:29:30.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001062688 2023-01-15T14:29:30.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:30.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696014+0000", 2023-01-15T14:29:30.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4697999999999996e-05 2023-01-15T14:29:30.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:30.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795898+0000", 2023-01-15T14:29:30.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099883338000000002 2023-01-15T14:29:30.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:30.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795922+0000", 2023-01-15T14:29:30.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4326000000000001e-05 2023-01-15T14:29:30.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:30.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795955+0000", 2023-01-15T14:29:30.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3191000000000001e-05 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796011+0000", 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5631999999999999e-05 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:30.113 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 61.732610151000003, 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:30.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:30.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:30.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:30.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:30.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:29:30.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:30.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:30.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:29:30.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:29:30.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:30.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:29:30.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:29:30.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:30.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:29:30.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:29:30.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:29:30.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:29:30.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:30.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:29:30.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:29:30.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:29:30.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:29:30.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:30.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:29:30.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:29:30.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:30.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:29:30.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:29:30.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:30.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:29:30.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:29:30.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:30.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:29:30.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:29:30.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:30.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:29:30.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:29:30.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:30.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:29:30.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:29:30.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:30.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:29:30.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:29:30.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:30.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:29:30.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:29:30.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.227 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:30.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.227 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:30.227 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:30.227 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 61.732553453999998, 2023-01-15T14:29:30.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:29:30.228 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:30.228 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:30.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:30.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:30.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:30.228 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:29:30.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.230 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:30.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:30.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:30.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:30.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:30.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:30.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:29:30.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:30.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:30.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:29:30.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:29:30.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:30.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:29:30.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:29:30.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:30.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:29:30.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:29:30.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:29:30.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:29:30.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:30.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:29:30.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:29:30.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:29:30.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:29:30.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:30.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:29:30.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:29:30.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:29:30.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:29:30.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:30.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:29:30.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:29:30.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:30.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:29:30.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:29:30.581 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.581 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:30.581 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.581 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:29:30.581 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:30.581 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.116057281000003, 2023-01-15T14:29:30.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:29:30.582 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:30.582 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:30.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:30.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:30.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:30.582 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:29:30.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.582 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:30.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:30.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:30.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:30.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:30.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:30.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:29:30.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:30.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:29:30.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:29:30.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:29:30.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:29:30.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:30.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:29:30.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:29:30.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:29:30.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:29:30.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:30.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:29:30.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:29:30.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:30.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:29:30.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:29:30.591 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.591 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:30.591 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.592 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:333 3.0 3:0a998d04:::smithi09025264-10:head [rollback 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:29:30.592 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860002+0000", 2023-01-15T14:29:30.592 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 52.087260520000001, 2023-01-15T14:29:30.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50655092599999996, 2023-01-15T14:29:30.592 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:30.592 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:30.592 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:30.592 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:30.592 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 333 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860002+0000", 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860002+0000", 2023-01-15T14:29:30.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:30.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860001+0000", 2023-01-15T14:29:30.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:30.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:30.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860010+0000", 2023-01-15T14:29:30.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6570000000000003e-06 2023-01-15T14:29:30.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:30.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860027+0000", 2023-01-15T14:29:30.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7180000000000002e-05 2023-01-15T14:29:30.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860036+0000", 2023-01-15T14:29:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1929999999999999e-06 2023-01-15T14:29:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860266+0000", 2023-01-15T14:29:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000229769 2023-01-15T14:29:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:30.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860276+0000", 2023-01-15T14:29:30.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8800000000000003e-06 2023-01-15T14:29:30.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.163322+0000", 2023-01-15T14:29:30.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30304593099999999 2023-01-15T14:29:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.163486+0000", 2023-01-15T14:29:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016389900000000001 2023-01-15T14:29:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.163798+0000", 2023-01-15T14:29:30.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031278300000000002 2023-01-15T14:29:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164058+0000", 2023-01-15T14:29:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259609 2023-01-15T14:29:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164086+0000", 2023-01-15T14:29:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8113999999999999e-05 2023-01-15T14:29:30.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164236+0000", 2023-01-15T14:29:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014949900000000001 2023-01-15T14:29:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293740+0000", 2023-01-15T14:29:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12950431100000001 2023-01-15T14:29:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.294109+0000", 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036941200000000001 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366415+0000", 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072305164000000005 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:30.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366503+0000", 2023-01-15T14:29:30.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8101000000000002e-05 2023-01-15T14:29:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366553+0000", 2023-01-15T14:29:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0726999999999997e-05 2023-01-15T14:29:30.759 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:29:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 52.08720881, 2023-01-15T14:29:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:29:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:30.760 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:29:30.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:29:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:29:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:29:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:29:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:29:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:29:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:29:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:29:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:29:30.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:29:30.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:30.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:29:30.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:29:30.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:30.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:29:30.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:29:30.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:30.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:29:30.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:29:30.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:30.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:29:30.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:29:30.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:30.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:29:30.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:29:30.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:30.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:29:30.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:29:30.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:30.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:29:30.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:29:30.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.821 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:30.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:30.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.821 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:373 3.4 3:2029fa71:::smithi09025264-39:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:30.821 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:30.821 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.868961603000002, 2023-01-15T14:29:30.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58298356399999995, 2023-01-15T14:29:30.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:30.821 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:30.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:30.822 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:30.822 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:30.822 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-01-15T14:29:30.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.822 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:30.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:30.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:30.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:30.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:30.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:30.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:30.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078299+0000", 2023-01-15T14:29:30.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:30.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:30.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078644+0000", 2023-01-15T14:29:30.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034494099999999998 2023-01-15T14:29:30.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:30.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078676+0000", 2023-01-15T14:29:30.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1183000000000003e-05 2023-01-15T14:29:30.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:30.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078694+0000", 2023-01-15T14:29:30.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-05 2023-01-15T14:29:30.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078748+0000", 2023-01-15T14:29:30.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3634999999999997e-05 2023-01-15T14:29:30.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:30.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078753+0000", 2023-01-15T14:29:30.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6370000000000004e-06 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110841+0000", 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032087439000000002 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110967+0000", 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126029 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.457874+0000", 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.346907725 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458066+0000", 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000191592 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:30.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458321+0000", 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025454200000000002 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458918+0000", 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059740800000000001 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458945+0000", 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-05 2023-01-15T14:29:30.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:30.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459253+0000", 2023-01-15T14:29:30.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030836200000000001 2023-01-15T14:29:30.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:30.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460943+0000", 2023-01-15T14:29:30.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016903580000000001 2023-01-15T14:29:30.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:30.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:30.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:31.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461340+0000", 2023-01-15T14:29:31.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039665799999999998 2023-01-15T14:29:31.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661129+0000", 2023-01-15T14:29:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19978933500000001 2023-01-15T14:29:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:31.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661188+0000", 2023-01-15T14:29:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8687000000000002e-05 2023-01-15T14:29:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661285+0000", 2023-01-15T14:29:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779000000000003e-05 2023-01-15T14:29:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:31.118 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:374 3.4 3:2029fa71:::smithi09025264-39:head [write 3101234~41642] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.864299690999999, 2023-01-15T14:29:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57848911800000002, 2023-01-15T14:29:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:31.119 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 374 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082961+0000", 2023-01-15T14:29:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083019+0000", 2023-01-15T14:29:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999997e-05 2023-01-15T14:29:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083051+0000", 2023-01-15T14:29:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1813999999999997e-05 2023-01-15T14:29:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:31.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083069+0000", 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083143+0000", 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4363000000000006e-05 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083147+0000", 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4529999999999998e-06 2023-01-15T14:29:31.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111017+0000", 2023-01-15T14:29:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027870564 2023-01-15T14:29:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111099+0000", 2023-01-15T14:29:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1277000000000001e-05 2023-01-15T14:29:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459156+0000", 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34805700499999997 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459393+0000", 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023757499999999999 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:31.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459778+0000", 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000385335 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460020+0000", 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024141899999999999 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460037+0000", 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6665999999999999e-05 2023-01-15T14:29:31.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460220+0000", 2023-01-15T14:29:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018367599999999999 2023-01-15T14:29:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461197+0000", 2023-01-15T14:29:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00097708600000000006 2023-01-15T14:29:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461720+0000", 2023-01-15T14:29:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052235400000000005 2023-01-15T14:29:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661345+0000", 2023-01-15T14:29:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199624884 2023-01-15T14:29:31.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661405+0000", 2023-01-15T14:29:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0965999999999999e-05 2023-01-15T14:29:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:31.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661452+0000", 2023-01-15T14:29:31.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6864999999999999e-05 2023-01-15T14:29:31.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:31.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:31.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:31.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.299 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:375 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (56) in=7b,truncate 3142876] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:31.299 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:31.299 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.864157925000001, 2023-01-15T14:29:31.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57851114599999998, 2023-01-15T14:29:31.299 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:31.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:31.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:31.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:31.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:31.300 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 375 2023-01-15T14:29:31.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.300 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:31.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:31.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:31.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:31.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:31.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083104+0000", 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083113+0000", 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6500000000000008e-06 2023-01-15T14:29:31.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:31.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083130+0000", 2023-01-15T14:29:31.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2023-01-15T14:29:31.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:31.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083142+0000", 2023-01-15T14:29:31.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2252999999999999e-05 2023-01-15T14:29:31.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083173+0000", 2023-01-15T14:29:31.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0474999999999999e-05 2023-01-15T14:29:31.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:31.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083175+0000", 2023-01-15T14:29:31.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2023-01-15T14:29:31.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183469+0000", 2023-01-15T14:29:31.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10029313099999999 2023-01-15T14:29:31.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:31.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183747+0000", 2023-01-15T14:29:31.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000277998 2023-01-15T14:29:31.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460165+0000", 2023-01-15T14:29:31.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27641872899999997 2023-01-15T14:29:31.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:31.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460342+0000", 2023-01-15T14:29:31.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176792 2023-01-15T14:29:31.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:31.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460579+0000", 2023-01-15T14:29:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000237266 2023-01-15T14:29:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460833+0000", 2023-01-15T14:29:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025390900000000002 2023-01-15T14:29:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:31.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460853+0000", 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9273e-05 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461024+0000", 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017111900000000001 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461230+0000", 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020672600000000001 2023-01-15T14:29:31.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:31.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461879+0000", 2023-01-15T14:29:31.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064846999999999995 2023-01-15T14:29:31.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661524+0000", 2023-01-15T14:29:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19964509599999999 2023-01-15T14:29:31.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661560+0000", 2023-01-15T14:29:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6264999999999999e-05 2023-01-15T14:29:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661616+0000", 2023-01-15T14:29:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5952000000000002e-05 2023-01-15T14:29:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:31.474 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:376 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.864075669000002, 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57864278800000002, 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:31.475 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 376 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:31.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083186+0000", 2023-01-15T14:29:31.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083193+0000", 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083211+0000", 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7609999999999999e-05 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083223+0000", 2023-01-15T14:29:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140000000000001e-05 2023-01-15T14:29:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093534+0000", 2023-01-15T14:29:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010311529 2023-01-15T14:29:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093551+0000", 2023-01-15T14:29:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.666e-05 2023-01-15T14:29:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183834+0000", 2023-01-15T14:29:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090283063999999996 2023-01-15T14:29:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.184076+0000", 2023-01-15T14:29:31.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024241900000000001 2023-01-15T14:29:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461001+0000", 2023-01-15T14:29:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27692504299999998 2023-01-15T14:29:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:31.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461152+0000", 2023-01-15T14:29:31.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150345 2023-01-15T14:29:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661642+0000", 2023-01-15T14:29:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200489742 2023-01-15T14:29:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661720+0000", 2023-01-15T14:29:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8604999999999997e-05 2023-01-15T14:29:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661830+0000", 2023-01-15T14:29:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010992099999999999 2023-01-15T14:29:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:31.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:646 3.0 3:07a51d33:::smithi09025264-24:head [write 1886503~412692] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2023-01-15T14:29:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:03.846884+0000", 2023-01-15T14:29:31.493 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.100379108999999, 2023-01-15T14:29:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50491198400000004, 2023-01-15T14:29:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:31.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 646 2023-01-15T14:29:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:31.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:31.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.846884+0000", 2023-01-15T14:29:31.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:31.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:31.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.846884+0000", 2023-01-15T14:29:31.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:31.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:31.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.846882+0000", 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.847099+0000", 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000216975 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.847127+0000", 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8053999999999999e-05 2023-01-15T14:29:31.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:31.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.847140+0000", 2023-01-15T14:29:31.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2982e-05 2023-01-15T14:29:31.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.946268+0000", 2023-01-15T14:29:31.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099128283999999997 2023-01-15T14:29:31.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:31.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.946529+0000", 2023-01-15T14:29:31.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026035199999999999 2023-01-15T14:29:31.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:29:31.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.946948+0000", 2023-01-15T14:29:31.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041912500000000001 2023-01-15T14:29:31.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:31.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.948601+0000", 2023-01-15T14:29:31.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016531199999999999 2023-01-15T14:29:31.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:31.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.948621+0000", 2023-01-15T14:29:31.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0290999999999999e-05 2023-01-15T14:29:31.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:31.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.950494+0000", 2023-01-15T14:29:31.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001872868 2023-01-15T14:29:31.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:31.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.950860+0000", 2023-01-15T14:29:31.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036591899999999998 2023-01-15T14:29:31.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:31.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.951285+0000", 2023-01-15T14:29:31.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042471499999999999 2023-01-15T14:29:31.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:31.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:04.351228+0000", 2023-01-15T14:29:31.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39994379699999999 2023-01-15T14:29:31.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:31.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:04.351418+0000", 2023-01-15T14:29:31.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018986299999999999 2023-01-15T14:29:31.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:31.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:04.351796+0000", 2023-01-15T14:29:31.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000377582 2023-01-15T14:29:31.860 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:31.860 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:31.860 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:31.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.861 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:647 3.0 3:07a51d33:::smithi09025264-24:head [write 2761543~670951] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2023-01-15T14:29:31.861 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:03.853792+0000", 2023-01-15T14:29:31.861 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.093470699000001, 2023-01-15T14:29:31.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49845493800000001, 2023-01-15T14:29:31.861 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:31.861 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:31.861 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:31.862 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:31.862 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:31.862 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 647 2023-01-15T14:29:31.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.862 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:31.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:31.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.853792+0000", 2023-01-15T14:29:31.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:31.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:31.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.853792+0000", 2023-01-15T14:29:31.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:31.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:31.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.853790+0000", 2023-01-15T14:29:31.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:31.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:31.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.854088+0000", 2023-01-15T14:29:31.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029783799999999998 2023-01-15T14:29:31.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:31.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.854117+0000", 2023-01-15T14:29:31.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9217000000000002e-05 2023-01-15T14:29:31.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:31.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.854130+0000", 2023-01-15T14:29:31.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:29:31.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:31.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.948727+0000", 2023-01-15T14:29:31.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094596948 2023-01-15T14:29:31.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.948918+0000", 2023-01-15T14:29:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019050100000000001 2023-01-15T14:29:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:29:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.949130+0000", 2023-01-15T14:29:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021205900000000001 2023-01-15T14:29:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.949538+0000", 2023-01-15T14:29:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040862500000000003 2023-01-15T14:29:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.950499+0000", 2023-01-15T14:29:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00096102000000000004 2023-01-15T14:29:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.950990+0000", 2023-01-15T14:29:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000490442 2023-01-15T14:29:31.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:31.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.951113+0000", 2023-01-15T14:29:31.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012295399999999999 2023-01-15T14:29:31.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:31.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.951316+0000", 2023-01-15T14:29:31.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020292500000000001 2023-01-15T14:29:31.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:04.351931+0000", 2023-01-15T14:29:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40061495200000002 2023-01-15T14:29:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:04.352023+0000", 2023-01-15T14:29:31.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2519e-05 2023-01-15T14:29:31.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:31.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:31.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:31.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:04.352247+0000", 2023-01-15T14:29:31.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022394 2023-01-15T14:29:31.870 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:31.871 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:29:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:29:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.085921834000001, 2023-01-15T14:29:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:29:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:29:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:29:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:32.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:32.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:29:32.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:32.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:32.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:29:32.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:32.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:32.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:29:32.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:29:32.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:32.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:29:32.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:29:32.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:32.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:29:32.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:29:32.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:32.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:29:32.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:29:32.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:32.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:29:32.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:29:32.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:29:32.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:29:32.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:29:32.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:32.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:29:32.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:29:32.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:32.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:29:32.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:29:32.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:32.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:29:32.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:29:32.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:32.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:29:32.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:29:32.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:32.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:29:32.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:29:32.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:32.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:29:32.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:29:32.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:32.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:29:32.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:29:32.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:32.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:32.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:32.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:29:32.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:29:32.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:32.046 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:32.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:32.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:32.046 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:32.046 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:32.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:29:32.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:29:32.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:29:32.049 INFO:tasks.rados.rados.0.smithi090.stdout:324: done (0 left) 2023-01-15T14:29:32.049 INFO:tasks.rados.rados.0.smithi090.stdout:325: read oid 12 snap 29 2023-01-15T14:29:32.049 INFO:tasks.rados.rados.0.smithi090.stdout:325: expect (ObjNum 82 snap 10 seq_num 82) 2023-01-15T14:29:32.049 INFO:tasks.rados.rados.0.smithi090.stdout:326: read oid 35 snap -1 2023-01-15T14:29:32.049 INFO:tasks.rados.rados.0.smithi090.stdout:326: expect deleted 2023-01-15T14:29:32.049 INFO:tasks.rados.rados.0.smithi090.stdout:327: read oid 19 snap 23 2023-01-15T14:29:32.200 INFO:tasks.rados.rados.0.smithi090.stdout:327: expect (ObjNum 87 snap 12 seq_num 87) 2023-01-15T14:29:32.203 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.2 to 1 (10000) 2023-01-15T14:29:32.205 INFO:tasks.rados.rados.0.smithi090.stdout:328: read oid 24 snap -1 2023-01-15T14:29:32.205 INFO:tasks.rados.rados.0.smithi090.stdout:328: expect (ObjNum 34 snap 0 seq_num 34) 2023-01-15T14:29:32.208 INFO:tasks.rados.rados.0.smithi090.stdout:329: read oid 40 snap -1 2023-01-15T14:29:32.208 INFO:tasks.rados.rados.0.smithi090.stdout:329: expect (ObjNum 39 snap 0 seq_num 39) 2023-01-15T14:29:32.208 INFO:tasks.rados.rados.0.smithi090.stdout:330: write oid 28 current snap is 30 2023-01-15T14:29:32.208 INFO:tasks.rados.rados.0.smithi090.stdout:330: seq_num 147 ranges {470629=472310,1418853=513377} 2023-01-15T14:29:32.209 INFO:tasks.rados.rados.0.smithi090.stdout:330: writing smithi09025264-28 from 470629 to 942939 tid 1 2023-01-15T14:29:32.209 INFO:tasks.rados.rados.0.smithi090.stdout:330: writing smithi09025264-28 from 1418853 to 1932230 tid 2 2023-01-15T14:29:32.209 INFO:tasks.rados.rados.0.smithi090.stdout:331: read oid 26 snap 23 2023-01-15T14:29:32.209 INFO:tasks.rados.rados.0.smithi090.stdout:331: expect (ObjNum 25 snap 0 seq_num 25) 2023-01-15T14:29:32.209 INFO:tasks.rados.rados.0.smithi090.stdout:332: copy_from oid 17 from oid 44 current snap is 30 2023-01-15T14:29:32.210 INFO:tasks.rados.rados.0.smithi090.stdout:333: write oid 48 current snap is 30 2023-01-15T14:29:32.210 INFO:tasks.rados.rados.0.smithi090.stdout:333: seq_num 148 ranges {487436=547559,1760982=657035} 2023-01-15T14:29:32.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-01-15T14:29:32.218 INFO:tasks.rados.rados.0.smithi090.stdout:333: writing smithi09025264-48 from 487436 to 1034995 tid 1 2023-01-15T14:29:32.218 INFO:tasks.rados.rados.0.smithi090.stdout:333: writing smithi09025264-48 from 1760982 to 2418017 tid 2 2023-01-15T14:29:32.219 INFO:tasks.rados.rados.0.smithi090.stdout:334: write (excl) oid 33 current snap is 30 2023-01-15T14:29:32.219 INFO:tasks.rados.rados.0.smithi090.stdout:334: seq_num 149 ranges {504243=622808,1819464=517046} 2023-01-15T14:29:32.219 INFO:tasks.rados.rados.0.smithi090.stdout:334: writing smithi09025264-33 from 504243 to 1127051 tid 1 2023-01-15T14:29:32.219 INFO:tasks.rados.rados.0.smithi090.stdout:334: writing smithi09025264-33 from 1819464 to 2336510 tid 2 2023-01-15T14:29:32.219 INFO:tasks.rados.rados.0.smithi090.stdout:335: read oid 1 snap 29 2023-01-15T14:29:32.373 INFO:tasks.rados.rados.0.smithi090.stdout:335: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:29:32.373 INFO:tasks.rados.rados.0.smithi090.stdout:336: rollback oid 22 current snap is 30 2023-01-15T14:29:32.373 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 22 to 29 2023-01-15T14:29:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:337: copy_from oid 2 from oid 6 current snap is 30 2023-01-15T14:29:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:330: finishing write tid 1 to smithi09025264-28 2023-01-15T14:29:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:338: write (excl) oid 5 current snap is 30 2023-01-15T14:29:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:338: seq_num 150 ranges {521050=698057,1877946=643104} 2023-01-15T14:29:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:338: writing smithi09025264-5 from 521050 to 1219107 tid 1 2023-01-15T14:29:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:338: writing smithi09025264-5 from 1877946 to 2521050 tid 2 2023-01-15T14:29:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:339: read oid 15 snap -1 2023-01-15T14:29:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:339: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:29:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:340: rollback oid 47 current snap is 30 2023-01-15T14:29:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 47 to 23 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:327: expect (ObjNum 87 snap 12 seq_num 87) 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:336: finishing rollback tid 0 to smithi09025264-22 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:334: finishing write tid 1 to smithi09025264-33 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:325: expect (ObjNum 82 snap 10 seq_num 82) 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:329: expect (ObjNum 39 snap 0 seq_num 39) 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:328: expect (ObjNum 34 snap 0 seq_num 34) 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:336: finishing rollback tid 1 to smithi09025264-22 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:333: finishing write tid 1 to smithi09025264-48 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:330: finishing write tid 2 to smithi09025264-28 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:330: finishing write tid 3 to smithi09025264-28 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:330: finishing write tid 4 to smithi09025264-28 2023-01-15T14:29:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:330: oid 28 updating version 0 to 157 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:330: oid 28 version 157 is already newer than 155 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:330: oid 28 version 157 is already newer than 156 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 157 (ObjNum 147 snap 30 seq_num 147) dirty exists 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:330: left oid 28 (ObjNum 147 snap 30 seq_num 147) 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:331: expect (ObjNum 25 snap 0 seq_num 25) 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:334: finishing write tid 2 to smithi09025264-33 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:334: finishing write tid 3 to smithi09025264-33 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:334: finishing write tid 4 to smithi09025264-33 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:334: oid 33 updating version 0 to 48 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:334: oid 33 version 48 is already newer than 47 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:334: oid 33 version 48 is already newer than 46 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 48 (ObjNum 149 snap 30 seq_num 149) dirty exists 2023-01-15T14:29:32.376 INFO:tasks.rados.rados.0.smithi090.stdout:334: left oid 33 (ObjNum 149 snap 30 seq_num 149) 2023-01-15T14:29:32.377 INFO:tasks.rados.rados.0.smithi090.stdout:339: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:29:32.377 INFO:tasks.rados.rados.0.smithi090.stdout:338: finishing write tid 1 to smithi09025264-5 2023-01-15T14:29:32.377 INFO:tasks.rados.rados.0.smithi090.stdout:336: finishing rollback tid 2 to smithi09025264-22 2023-01-15T14:29:32.377 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 106 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-01-15T14:29:32.377 INFO:tasks.rados.rados.0.smithi090.stdout:333: finishing write tid 2 to smithi09025264-48 2023-01-15T14:29:32.377 INFO:tasks.rados.rados.0.smithi090.stdout:335: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:29:32.377 INFO:tasks.rados.rados.0.smithi090.stdout:337: finishing copy_from to smithi09025264-2 2023-01-15T14:29:32.377 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 50 (ObjNum 70 snap 8 seq_num 70) dirty exists 2023-01-15T14:29:32.377 INFO:tasks.rados.rados.0.smithi090.stdout:337: finishing copy_from racing read to smithi09025264-2 2023-01-15T14:29:32.378 INFO:tasks.rados.rados.0.smithi090.stdout:333: finishing write tid 3 to smithi09025264-48 2023-01-15T14:29:32.378 INFO:tasks.rados.rados.0.smithi090.stdout:333: finishing write tid 4 to smithi09025264-48 2023-01-15T14:29:32.378 INFO:tasks.rados.rados.0.smithi090.stdout:333: oid 48 updating version 0 to 92 2023-01-15T14:29:32.378 INFO:tasks.rados.rados.0.smithi090.stdout:333: oid 48 version 92 is already newer than 91 2023-01-15T14:29:32.378 INFO:tasks.rados.rados.0.smithi090.stdout:333: oid 48 updating version 92 to 93 2023-01-15T14:29:32.378 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 93 (ObjNum 148 snap 30 seq_num 148) dirty exists 2023-01-15T14:29:32.378 INFO:tasks.rados.rados.0.smithi090.stdout:333: left oid 48 (ObjNum 148 snap 30 seq_num 148) 2023-01-15T14:29:32.378 INFO:tasks.rados.rados.0.smithi090.stdout:340: finishing rollback tid 0 to smithi09025264-47 2023-01-15T14:29:32.378 INFO:tasks.rados.rados.0.smithi090.stdout:332: finishing copy_from to smithi09025264-17 2023-01-15T14:29:32.378 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 25 (ObjNum 78 snap 10 seq_num 78) dirty exists 2023-01-15T14:29:32.378 INFO:tasks.rados.rados.0.smithi090.stdout:332: finishing copy_from racing read to smithi09025264-17 2023-01-15T14:29:32.379 INFO:tasks.rados.rados.0.smithi090.stdout:340: finishing rollback tid 1 to smithi09025264-47 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:325: done (15 left) 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:326: done (14 left) 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:327: done (13 left) 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:328: done (12 left) 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:329: done (11 left) 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:330: done (10 left) 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:331: done (9 left) 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:332: done (8 left) 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:333: done (7 left) 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:334: done (6 left) 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:335: done (5 left) 2023-01-15T14:29:32.550 INFO:tasks.rados.rados.0.smithi090.stdout:336: done (4 left) 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:337: done (3 left) 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:339: done (2 left) 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:341: snap_remove snap 29 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:341: done (2 left) 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:342: copy_from oid 12 from oid 15 current snap is 30 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:340: finishing rollback tid 2 to smithi09025264-47 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 161 (ObjNum 110 snap 23 seq_num 110) dirty exists 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:338: finishing write tid 2 to smithi09025264-5 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:338: finishing write tid 3 to smithi09025264-5 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:338: finishing write tid 4 to smithi09025264-5 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:338: oid 5 updating version 0 to 118 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:338: oid 5 version 118 is already newer than 117 2023-01-15T14:29:32.551 INFO:tasks.rados.rados.0.smithi090.stdout:338: oid 5 updating version 118 to 119 2023-01-15T14:29:32.552 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 119 (ObjNum 150 snap 30 seq_num 150) dirty exists 2023-01-15T14:29:32.552 INFO:tasks.rados.rados.0.smithi090.stdout:338: left oid 5 (ObjNum 150 snap 30 seq_num 150) 2023-01-15T14:29:32.552 INFO:tasks.rados.rados.0.smithi090.stdout:338: done (2 left) 2023-01-15T14:29:32.552 INFO:tasks.rados.rados.0.smithi090.stdout:340: done (1 left) 2023-01-15T14:29:32.552 INFO:tasks.rados.rados.0.smithi090.stdout:343: snap_create 2023-01-15T14:29:32.552 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:32.552 INFO:tasks.rados.rados.0.smithi090.stdout:342: finishing copy_from to smithi09025264-12 2023-01-15T14:29:32.552 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 59 (ObjNum 128 snap 23 seq_num 128) dirty exists 2023-01-15T14:29:32.552 INFO:tasks.rados.rados.0.smithi090.stdout:342: finishing copy_from racing read to smithi09025264-12 2023-01-15T14:29:32.552 INFO:tasks.rados.rados.0.smithi090.stdout:342: done (0 left) 2023-01-15T14:29:32.553 INFO:tasks.rados.rados.0.smithi090.stdout:343: done (0 left) 2023-01-15T14:29:32.553 INFO:tasks.rados.rados.0.smithi090.stdout:344: copy_from oid 10 from oid 9 current snap is 31 2023-01-15T14:29:32.553 INFO:tasks.rados.rados.0.smithi090.stdout:345: snap_create 2023-01-15T14:29:32.553 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:32.553 INFO:tasks.rados.rados.0.smithi090.stdout:344: finishing copy_from to smithi09025264-10 2023-01-15T14:29:32.553 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-01-15T14:29:32.553 INFO:tasks.rados.rados.0.smithi090.stdout:344: finishing copy_from racing read to smithi09025264-10 2023-01-15T14:29:32.553 INFO:tasks.rados.rados.0.smithi090.stdout:344: done (0 left) 2023-01-15T14:29:32.553 INFO:tasks.rados.rados.0.smithi090.stdout:345: done (0 left) 2023-01-15T14:29:32.554 INFO:tasks.rados.rados.0.smithi090.stdout:346: delete oid 45 current snap is 32 2023-01-15T14:29:32.554 INFO:tasks.rados.rados.0.smithi090.stdout:346: done (0 left) 2023-01-15T14:29:32.554 INFO:tasks.rados.rados.0.smithi090.stdout:347: write oid 19 current snap is 32 2023-01-15T14:29:32.554 INFO:tasks.rados.rados.0.smithi090.stdout:347: seq_num 152 ranges {554664=448555,1478557=664373} 2023-01-15T14:29:32.554 INFO:tasks.rados.rados.0.smithi090.stdout:347: writing smithi09025264-19 from 554664 to 1003219 tid 1 2023-01-15T14:29:32.554 INFO:tasks.rados.rados.0.smithi090.stdout:347: writing smithi09025264-19 from 1478557 to 2142930 tid 2 2023-01-15T14:29:32.554 INFO:tasks.rados.rados.0.smithi090.stdout:348: snap_create 2023-01-15T14:29:32.554 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:32.554 INFO:tasks.rados.rados.0.smithi090.stdout:347: finishing write tid 1 to smithi09025264-19 2023-01-15T14:29:32.555 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:32.045+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:32.556 INFO:tasks.rados.rados.0.smithi090.stdout:347: finishing write tid 2 to smithi09025264-19 2023-01-15T14:29:32.556 INFO:tasks.rados.rados.0.smithi090.stdout:347: finishing write tid 3 to smithi09025264-19 2023-01-15T14:29:32.556 INFO:tasks.rados.rados.0.smithi090.stdout:347: finishing write tid 4 to smithi09025264-19 2023-01-15T14:29:32.557 INFO:tasks.rados.rados.0.smithi090.stdout:347: oid 19 updating version 0 to 72 2023-01-15T14:29:32.557 INFO:tasks.rados.rados.0.smithi090.stdout:347: oid 19 version 72 is already newer than 70 2023-01-15T14:29:32.557 INFO:tasks.rados.rados.0.smithi090.stdout:347: oid 19 version 72 is already newer than 71 2023-01-15T14:29:32.557 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 72 (ObjNum 152 snap 32 seq_num 152) dirty exists 2023-01-15T14:29:32.557 INFO:tasks.rados.rados.0.smithi090.stdout:347: left oid 19 (ObjNum 152 snap 32 seq_num 152) 2023-01-15T14:29:32.557 INFO:tasks.rados.rados.0.smithi090.stdout:347: done (0 left) 2023-01-15T14:29:32.559 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:32.212+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:32.560 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:32.556+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:32.660 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:32.656+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:32.722 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:32.722 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:32.722 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:29:32.722 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:32.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:29:32.723 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:32.735 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:32.754 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:32.898 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:32.900 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:32.900 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:32.902 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:32.902 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:32.903 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:29:32.903 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:32.903 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:32.904 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:29:33.024 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:29:33.024 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:29:33.024 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:29:33.024 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:29:33.025 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:29:33.025 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:29:33.025 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:29:33.026 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:32.892+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:33.027 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:33.021+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:33.093 INFO:tasks.rados.rados.0.smithi090.stdout:348: done (0 left) 2023-01-15T14:29:33.093 INFO:tasks.rados.rados.0.smithi090.stdout:349: write (excl) oid 46 current snap is 33 2023-01-15T14:29:33.093 INFO:tasks.rados.rados.0.smithi090.stdout:349: seq_num 153 ranges {571471=640157,1653392=524384} 2023-01-15T14:29:33.096 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:29:33.103 INFO:tasks.rados.rados.0.smithi090.stdout:349: writing smithi09025264-46 from 571471 to 1211628 tid 1 2023-01-15T14:29:33.103 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.3 to 1 (10000) 2023-01-15T14:29:33.110 INFO:tasks.rados.rados.0.smithi090.stdout:349: writing smithi09025264-46 from 1653392 to 2177776 tid 2 2023-01-15T14:29:33.110 INFO:tasks.rados.rados.0.smithi090.stdout:350: copy_from oid 40 from oid 14 current snap is 33 2023-01-15T14:29:33.110 INFO:tasks.rados.rados.0.smithi090.stdout:351: read oid 6 snap -1 2023-01-15T14:29:33.110 INFO:tasks.rados.rados.0.smithi090.stdout:351: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:29:33.110 INFO:tasks.rados.rados.0.smithi090.stdout:352: rollback oid 39 current snap is 33 2023-01-15T14:29:33.110 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 39 to 30 2023-01-15T14:29:33.110 INFO:tasks.rados.rados.0.smithi090.stdout:353: delete oid 42 current snap is 33 2023-01-15T14:29:33.112 INFO:tasks.rados.rados.0.smithi090.stdout:349: finishing write tid 1 to smithi09025264-46 2023-01-15T14:29:33.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-01-15T14:29:33.227 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:33.227 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:33.228 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:29:33.228 INFO:tasks.rados.rados.0.smithi090.stdout:351: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:29:33.229 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:33.221+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:33.328 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:33.321+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:33.398 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:33.398 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:33.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:29:33.410 INFO:tasks.rados.rados.0.smithi090.stdout:351: done (4 left) 2023-01-15T14:29:33.410 INFO:tasks.rados.rados.0.smithi090.stdout:353: done (3 left) 2023-01-15T14:29:33.410 INFO:tasks.rados.rados.0.smithi090.stdout:354: snap_create 2023-01-15T14:29:33.411 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:29:33.411 INFO:tasks.rados.rados.0.smithi090.stdout:350: finishing copy_from to smithi09025264-40 2023-01-15T14:29:33.411 INFO:tasks.rados.rados.0.smithi090.stdout:350: got expected ENOENT (src dne) 2023-01-15T14:29:33.411 INFO:tasks.rados.rados.0.smithi090.stdout:350: finishing copy_from racing read to smithi09025264-40 2023-01-15T14:29:33.411 INFO:tasks.rados.rados.0.smithi090.stdout:350: done (2 left) 2023-01-15T14:29:33.411 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:33.429 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:33.421+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:33.498 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:33.569 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:33.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:29:33.571 INFO:tasks.rados.rados.0.smithi090.stdout:349: finishing write tid 2 to smithi09025264-46 2023-01-15T14:29:33.571 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:33.564+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:33.614 INFO:tasks.rados.rados.0.smithi090.stdout:349: finishing write tid 3 to smithi09025264-46 2023-01-15T14:29:33.614 INFO:tasks.rados.rados.0.smithi090.stdout:349: finishing write tid 4 to smithi09025264-46 2023-01-15T14:29:33.614 INFO:tasks.rados.rados.0.smithi090.stdout:349: oid 46 updating version 0 to 69 2023-01-15T14:29:33.614 INFO:tasks.rados.rados.0.smithi090.stdout:349: oid 46 updating version 69 to 70 2023-01-15T14:29:33.614 INFO:tasks.rados.rados.0.smithi090.stdout:349: oid 46 version 70 is already newer than 68 2023-01-15T14:29:33.614 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 70 (ObjNum 153 snap 33 seq_num 153) dirty exists 2023-01-15T14:29:33.614 INFO:tasks.rados.rados.0.smithi090.stdout:349: left oid 46 (ObjNum 153 snap 33 seq_num 153) 2023-01-15T14:29:33.615 INFO:tasks.rados.rados.0.smithi090.stdout:349: done (1 left) 2023-01-15T14:29:33.615 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:33.615 INFO:tasks.rados.rados.0.smithi090.stdout:352: finishing rollback tid 0 to smithi09025264-39 2023-01-15T14:29:33.615 INFO:tasks.rados.rados.0.smithi090.stdout:352: finishing rollback tid 1 to smithi09025264-39 2023-01-15T14:29:33.616 INFO:tasks.rados.rados.0.smithi090.stdout:352: finishing rollback tid 2 to smithi09025264-39 2023-01-15T14:29:33.616 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 74 (ObjNum 121 snap 23 seq_num 121) dirty exists 2023-01-15T14:29:33.616 INFO:tasks.rados.rados.0.smithi090.stdout:352: done (0 left) 2023-01-15T14:29:33.672 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:33.665+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:33.719 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:33.750 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:33.751 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:29:33.797 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:33.792+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:33.866 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:33.867 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:33.873 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:33.868+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:33.876 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:29:33.974 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:33.969+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:33.984 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:33.984 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:33.994 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:29:34.075 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:34.068+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:34.095 INFO:tasks.rados.rados.0.smithi090.stdout:354: done (0 left) 2023-01-15T14:29:34.095 INFO:tasks.rados.rados.0.smithi090.stdout:355: read oid 33 snap -1 2023-01-15T14:29:34.095 INFO:tasks.rados.rados.0.smithi090.stdout:355: expect (ObjNum 149 snap 30 seq_num 149) 2023-01-15T14:29:34.095 INFO:tasks.rados.rados.0.smithi090.stdout:356: snap_remove snap 32 2023-01-15T14:29:34.099 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.4 to 1 (10000) 2023-01-15T14:29:34.106 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:34.106 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:34.111 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2023-01-15T14:29:34.115 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:29:34.176 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:34.168+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:34.215 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:34.216 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:34.225 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:29:34.277 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:34.272+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:34.378 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:34.369+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:34.378 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:34.378 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:34.388 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:29:34.479 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:34.477+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:34.492 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:34.492 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:34.502 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:29:34.580 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:34.572+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:34.610 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:34.610 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:34.620 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:29:34.682 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:34.676+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:34.722 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:34.722 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:34.732 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:29:34.782 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:34.781+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:34.881 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:34.881 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:34.883 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:34.876+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:34.891 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:29:34.985 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:34.981+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:34.991 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:34.991 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:35.002 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:29:35.027 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:29:35.087 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:35.080+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:35.097 INFO:tasks.rados.rados.0.smithi090.stdout:356: done (1 left) 2023-01-15T14:29:35.097 INFO:tasks.rados.rados.0.smithi090.stdout:357: copy_from oid 47 from oid 36 current snap is 34 2023-01-15T14:29:35.097 INFO:tasks.rados.rados.0.smithi090.stdout:355: expect (ObjNum 149 snap 30 seq_num 149) 2023-01-15T14:29:35.105 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:35.150 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:35.150 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:29:35.152 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.6 to 1 (10000) 2023-01-15T14:29:35.153 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:29:35.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2023-01-15T14:29:35.201 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:35.196+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:35.325 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:35.325 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:35.326 INFO:tasks.rados.rados.0.smithi090.stdout:355: done (1 left) 2023-01-15T14:29:35.326 INFO:tasks.rados.rados.0.smithi090.stdout:358: read oid 48 snap -1 2023-01-15T14:29:35.327 INFO:tasks.rados.rados.0.smithi090.stdout:358: expect (ObjNum 148 snap 30 seq_num 148) 2023-01-15T14:29:35.327 INFO:tasks.rados.rados.0.smithi090.stdout:359: write (excl) oid 38 current snap is 34 2023-01-15T14:29:35.328 INFO:tasks.rados.rados.0.smithi090.stdout:359: seq_num 155 ranges {605085=790655,2170356=434729} 2023-01-15T14:29:35.328 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:35.328 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:35.328 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:35.329 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:35.320+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:35.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:29:35.330 INFO:tasks.rados.rados.0.smithi090.stdout:359: writing smithi09025264-38 from 605085 to 1395740 tid 1 2023-01-15T14:29:35.330 INFO:tasks.rados.rados.0.smithi090.stdout:359: writing smithi09025264-38 from 2170356 to 2605085 tid 2 2023-01-15T14:29:35.330 INFO:tasks.rados.rados.0.smithi090.stdout:360: write oid 49 current snap is 34 2023-01-15T14:29:35.330 INFO:tasks.rados.rados.0.smithi090.stdout:360: seq_num 156 ranges {621892=465904,1712485=671711} 2023-01-15T14:29:35.330 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:35.331 INFO:tasks.rados.rados.0.smithi090.stdout:360: writing smithi09025264-49 from 621892 to 1087796 tid 1 2023-01-15T14:29:35.331 INFO:tasks.rados.rados.0.smithi090.stdout:360: writing smithi09025264-49 from 1712485 to 2384196 tid 2 2023-01-15T14:29:35.331 INFO:tasks.rados.rados.0.smithi090.stdout:361: write oid 6 current snap is 34 2023-01-15T14:29:35.331 INFO:tasks.rados.rados.0.smithi090.stdout:361: seq_num 157 ranges {638699=541153,1770967=531722} 2023-01-15T14:29:35.331 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:35.332 INFO:tasks.rados.rados.0.smithi090.stdout:361: writing smithi09025264-6 from 638699 to 1179852 tid 1 2023-01-15T14:29:35.332 INFO:tasks.rados.rados.0.smithi090.stdout:361: writing smithi09025264-6 from 1770967 to 2302689 tid 2 2023-01-15T14:29:35.332 INFO:tasks.rados.rados.0.smithi090.stdout:362: snap_create 2023-01-15T14:29:35.332 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:29:35.332 INFO:tasks.rados.rados.0.smithi090.stdout:358: expect (ObjNum 148 snap 30 seq_num 148) 2023-01-15T14:29:35.333 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:35.351 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:35.592 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:35.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:29:35.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:29:35.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:29:35.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:29:35.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:29:35.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:29:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:29:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:29:35.594 INFO:tasks.rados.rados.0.smithi090.stdout:360: finishing write tid 1 to smithi09025264-49 2023-01-15T14:29:35.594 INFO:tasks.rados.rados.0.smithi090.stdout:360: finishing write tid 2 to smithi09025264-49 2023-01-15T14:29:35.594 INFO:tasks.rados.rados.0.smithi090.stdout:358: done (4 left) 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:357: finishing copy_from to smithi09025264-47 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 54 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:357: finishing copy_from racing read to smithi09025264-47 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:359: finishing write tid 1 to smithi09025264-38 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:357: done (3 left) 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:361: finishing write tid 1 to smithi09025264-6 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:360: finishing write tid 3 to smithi09025264-49 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:360: finishing write tid 4 to smithi09025264-49 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:360: oid 49 updating version 0 to 126 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:360: oid 49 updating version 126 to 128 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:360: oid 49 version 128 is already newer than 127 2023-01-15T14:29:35.595 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 128 (ObjNum 156 snap 34 seq_num 156) dirty exists 2023-01-15T14:29:35.596 INFO:tasks.rados.rados.0.smithi090.stdout:360: left oid 49 (ObjNum 156 snap 34 seq_num 156) 2023-01-15T14:29:35.596 INFO:tasks.rados.rados.0.smithi090.stdout:360: done (2 left) 2023-01-15T14:29:35.596 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:35.596 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:35.597 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:29:35.597 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:35.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:29:35.598 INFO:tasks.rados.rados.0.smithi090.stdout:361: finishing write tid 2 to smithi09025264-6 2023-01-15T14:29:35.598 INFO:tasks.rados.rados.0.smithi090.stdout:361: finishing write tid 3 to smithi09025264-6 2023-01-15T14:29:35.598 INFO:tasks.rados.rados.0.smithi090.stdout:359: finishing write tid 2 to smithi09025264-38 2023-01-15T14:29:35.599 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:35.588+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:35.616 INFO:tasks.rados.rados.0.smithi090.stdout:359: finishing write tid 3 to smithi09025264-38 2023-01-15T14:29:35.616 INFO:tasks.rados.rados.0.smithi090.stdout:361: finishing write tid 4 to smithi09025264-6 2023-01-15T14:29:35.616 INFO:tasks.rados.rados.0.smithi090.stdout:361: oid 6 updating version 0 to 52 2023-01-15T14:29:35.616 INFO:tasks.rados.rados.0.smithi090.stdout:361: oid 6 updating version 52 to 53 2023-01-15T14:29:35.617 INFO:tasks.rados.rados.0.smithi090.stdout:361: oid 6 version 53 is already newer than 51 2023-01-15T14:29:35.617 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 53 (ObjNum 157 snap 34 seq_num 157) dirty exists 2023-01-15T14:29:35.617 INFO:tasks.rados.rados.0.smithi090.stdout:361: left oid 6 (ObjNum 157 snap 34 seq_num 157) 2023-01-15T14:29:35.617 INFO:tasks.rados.rados.0.smithi090.stdout:361: done (1 left) 2023-01-15T14:29:35.617 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:35.617 INFO:tasks.rados.rados.0.smithi090.stdout:359: finishing write tid 4 to smithi09025264-38 2023-01-15T14:29:35.617 INFO:tasks.rados.rados.0.smithi090.stdout:359: oid 38 updating version 0 to 97 2023-01-15T14:29:35.797 INFO:tasks.rados.rados.0.smithi090.stdout:359: oid 38 updating version 97 to 99 2023-01-15T14:29:35.797 INFO:tasks.rados.rados.0.smithi090.stdout:359: oid 38 version 99 is already newer than 98 2023-01-15T14:29:35.797 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 99 (ObjNum 155 snap 34 seq_num 155) dirty exists 2023-01-15T14:29:35.797 INFO:tasks.rados.rados.0.smithi090.stdout:359: left oid 38 (ObjNum 155 snap 34 seq_num 155) 2023-01-15T14:29:35.797 INFO:tasks.rados.rados.0.smithi090.stdout:359: done (0 left) 2023-01-15T14:29:35.799 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:35.799 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:35.799 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:29:35.800 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:29:35.800 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:35.800 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:29:35.802 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:35.792+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:35.900 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:35.896+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:36.065 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:29:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:29:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:36.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:30 3.1 3:8b0a2647:::smithi09025264-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:29:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.091153112000001, 2023-01-15T14:29:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40319718700000001, 2023-01-15T14:29:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:36.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 30 2023-01-15T14:29:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:29:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:36.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:29:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:36.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859909+0000", 2023-01-15T14:29:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:29:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.860361+0000", 2023-01-15T14:29:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045164599999999999 2023-01-15T14:29:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.860392+0000", 2023-01-15T14:29:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1331000000000002e-05 2023-01-15T14:29:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960486+0000", 2023-01-15T14:29:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100093583 2023-01-15T14:29:36.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.160839+0000", 2023-01-15T14:29:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20035291199999999 2023-01-15T14:29:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.160982+0000", 2023-01-15T14:29:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014350800000000001 2023-01-15T14:29:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161183+0000", 2023-01-15T14:29:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000200411 2023-01-15T14:29:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161525+0000", 2023-01-15T14:29:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034216500000000001 2023-01-15T14:29:36.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161587+0000", 2023-01-15T14:29:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2032999999999994e-05 2023-01-15T14:29:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:36.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161842+0000", 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000255305 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262815+0000", 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100972992 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262914+0000", 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9166999999999994e-05 2023-01-15T14:29:36.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:36.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262967+0000", 2023-01-15T14:29:36.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2244000000000003e-05 2023-01-15T14:29:36.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:36.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263050+0000", 2023-01-15T14:29:36.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2999999999999998e-05 2023-01-15T14:29:36.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:36.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263110+0000", 2023-01-15T14:29:36.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0297000000000002e-05 2023-01-15T14:29:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:31 3.1 3:8b0a2647:::smithi09025264-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:29:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.990550745999997, 2023-01-15T14:29:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405055944, 2023-01-15T14:29:36.222 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 31 2023-01-15T14:29:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:36.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:29:36.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:36.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:36.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:29:36.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:36.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:36.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960513+0000", 2023-01-15T14:29:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960524+0000", 2023-01-15T14:29:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0614e-05 2023-01-15T14:29:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960538+0000", 2023-01-15T14:29:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4226999999999999e-05 2023-01-15T14:29:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060704+0000", 2023-01-15T14:29:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166252 2023-01-15T14:29:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.261723+0000", 2023-01-15T14:29:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20101896699999999 2023-01-15T14:29:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.261869+0000", 2023-01-15T14:29:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014578099999999999 2023-01-15T14:29:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262092+0000", 2023-01-15T14:29:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223198 2023-01-15T14:29:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262397+0000", 2023-01-15T14:29:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030490500000000001 2023-01-15T14:29:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262446+0000", 2023-01-15T14:29:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8492999999999998e-05 2023-01-15T14:29:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262537+0000", 2023-01-15T14:29:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1248000000000005e-05 2023-01-15T14:29:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262847+0000", 2023-01-15T14:29:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031007799999999998 2023-01-15T14:29:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:36.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365234+0000", 2023-01-15T14:29:36.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102386856 2023-01-15T14:29:36.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:36.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365326+0000", 2023-01-15T14:29:36.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2226999999999996e-05 2023-01-15T14:29:36.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:36.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365413+0000", 2023-01-15T14:29:36.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6996999999999994e-05 2023-01-15T14:29:36.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:36.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365571+0000", 2023-01-15T14:29:36.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015794100000000001 2023-01-15T14:29:36.397 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:36.398 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:36.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:36.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.398 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:53 3.1 3:883e9a29:::smithi09025264-14:head [write 618491~694590,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:36.398 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:29:36.399 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.890242470000004, 2023-01-15T14:29:36.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659587899999998, 2023-01-15T14:29:36.399 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:36.399 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:36.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:36.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:36.400 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:36.400 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 53 2023-01-15T14:29:36.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.400 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:36.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:36.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:29:36.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:36.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:36.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:29:36.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:36.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:36.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060822+0000", 2023-01-15T14:29:36.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:36.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061225+0000", 2023-01-15T14:29:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040297199999999998 2023-01-15T14:29:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:36.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061244+0000", 2023-01-15T14:29:36.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:29:36.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:36.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061253+0000", 2023-01-15T14:29:36.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8200000000000003e-06 2023-01-15T14:29:36.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:36.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263154+0000", 2023-01-15T14:29:36.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20190054900000001 2023-01-15T14:29:36.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:36.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263511+0000", 2023-01-15T14:29:36.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035690599999999999 2023-01-15T14:29:36.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:36.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263928+0000", 2023-01-15T14:29:36.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041701200000000003 2023-01-15T14:29:36.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:36.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264611+0000", 2023-01-15T14:29:36.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068358700000000004 2023-01-15T14:29:36.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:36.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264661+0000", 2023-01-15T14:29:36.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.952e-05 2023-01-15T14:29:36.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:36.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264962+0000", 2023-01-15T14:29:36.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030116799999999999 2023-01-15T14:29:36.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:36.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365102+0000", 2023-01-15T14:29:36.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100139698 2023-01-15T14:29:36.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:36.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366354+0000", 2023-01-15T14:29:36.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001252059 2023-01-15T14:29:36.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457209+0000", 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090855667000000001 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457307+0000", 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7621000000000006e-05 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:36.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457419+0000", 2023-01-15T14:29:36.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112246 2023-01-15T14:29:36.421 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:36.421 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:36.421 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:36.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.421 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:36.421 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:29:36.421 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.889801016999996, 2023-01-15T14:29:36.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:29:36.616 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:36.616 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:36.616 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:36.616 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:36.616 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:36.616 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:29:36.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.616 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:36.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:36.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:29:36.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:36.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:36.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:29:36.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:36.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:29:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:29:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:29:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:29:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:29:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:29:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:29:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:29:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:29:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:29:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:29:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:29:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:29:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:29:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:29:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:29:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:29:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:29:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:29:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:29:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:29:37.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:29:37.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:29:37.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:29:37.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:37.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:29:37.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:29:37.026 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.889520769000001, 2023-01-15T14:29:37.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:29:37.027 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:37.027 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:37.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:37.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:37.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:37.031 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:29:37.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.031 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:37.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:37.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:29:37.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:29:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:29:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:37.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:37.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:29:37.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:29:37.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:37.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:29:37.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:29:37.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:37.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:29:37.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:29:37.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:29:37.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:29:37.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:29:37.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:29:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:29:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:29:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:29:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:29:37.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:37.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:29:37.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:29:37.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:29:37.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:29:37.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:29:37.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.040 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:37.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.040 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:37.040 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:29:37.040 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.889488151999998, 2023-01-15T14:29:37.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:29:37.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:37.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:37.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:29:37.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:29:37.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:37.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:29:37.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:29:37.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:37.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:29:37.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:29:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:29:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:29:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:29:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:29:37.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:29:37.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:29:37.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:37.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:29:37.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:29:37.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:37.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:29:37.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:29:37.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.047 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:37.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.047 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:58 3.1 3:86bba67f:::smithi09025264-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:37.047 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:29:37.047 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.788317586999995, 2023-01-15T14:29:37.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39908735699999998, 2023-01-15T14:29:37.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 58 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:37.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:29:37.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:37.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162747+0000", 2023-01-15T14:29:37.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:37.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162859+0000", 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112137 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162874+0000", 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4793999999999999e-05 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:37.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162883+0000", 2023-01-15T14:29:37.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5559999999999996e-06 2023-01-15T14:29:37.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390699+0000", 2023-01-15T14:29:37.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.227815712 2023-01-15T14:29:37.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:37.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390900+0000", 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000201518 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458887+0000", 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067986183000000006 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459000+0000", 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011291800000000001 2023-01-15T14:29:37.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459244+0000", 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024485000000000002 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459480+0000", 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023518599999999999 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459498+0000", 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8892000000000001e-05 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459822+0000", 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032378700000000002 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460258+0000", 2023-01-15T14:29:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043563500000000003 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465816+0000", 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055581099999999998 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561720+0000", 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095903587999999998 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:37.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561771+0000", 2023-01-15T14:29:37.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1433e-05 2023-01-15T14:29:37.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:37.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561836+0000", 2023-01-15T14:29:37.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4591000000000006e-05 2023-01-15T14:29:37.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.056 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:37.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.057 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:37.057 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:29:37.057 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.788170166, 2023-01-15T14:29:37.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:29:37.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:37.057 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:37.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:37.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:37.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:29:37.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:37.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:29:37.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:29:37.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:37.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:29:37.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:29:37.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:29:37.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:29:37.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:37.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:29:37.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:29:37.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:29:37.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:29:37.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:37.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:29:37.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:29:37.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:37.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:29:37.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:29:37.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:37.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:29:37.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:29:37.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:37.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:29:37.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:29:37.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:29:37.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:29:37.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:29:37.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:29:37.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:37.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:29:37.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:29:37.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.788140162999994, 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:29:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:29:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:29:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:29:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:29:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:29:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:125 3.1 3:8889e450:::smithi09025264-32:head [write 521017~798131,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:29:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.278251345000001, 2023-01-15T14:29:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40348810600000001, 2023-01-15T14:29:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 125 2023-01-15T14:29:37.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.254 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:29:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:29:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672812+0000", 2023-01-15T14:29:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673338+0000", 2023-01-15T14:29:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052553599999999995 2023-01-15T14:29:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673372+0000", 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3652999999999998e-05 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673386+0000", 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4171e-05 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773535+0000", 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148822 2023-01-15T14:29:37.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773727+0000", 2023-01-15T14:29:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192189 2023-01-15T14:29:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773918+0000", 2023-01-15T14:29:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019143099999999999 2023-01-15T14:29:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:37.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774291+0000", 2023-01-15T14:29:37.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037288399999999998 2023-01-15T14:29:37.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:37.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774341+0000", 2023-01-15T14:29:37.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0117999999999997e-05 2023-01-15T14:29:37.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774696+0000", 2023-01-15T14:29:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000355268 2023-01-15T14:29:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874892+0000", 2023-01-15T14:29:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100195981 2023-01-15T14:29:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.977868+0000", 2023-01-15T14:29:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102975649 2023-01-15T14:29:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.075846+0000", 2023-01-15T14:29:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097977838999999997 2023-01-15T14:29:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.076004+0000", 2023-01-15T14:29:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015859700000000001 2023-01-15T14:29:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.076303+0000", 2023-01-15T14:29:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029816400000000001 2023-01-15T14:29:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.273131598000006, 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:29:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:29:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:29:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:29:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:29:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:29:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:29:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:29:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:29:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:29:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:29:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:29:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:29:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:29:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:29:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:29:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:29:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:29:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:29:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:29:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:29:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:29:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:29:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:29:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:29:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:29:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:29:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:29:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.272649243999993, 2023-01-15T14:29:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:29:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:29:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:29:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:29:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:29:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:29:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:29:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:29:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:29:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:29:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:37.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:29:37.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:29:37.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:37.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:29:37.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:29:37.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:37.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:29:37.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:37.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:29:37.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:29:37.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:29:37.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:29:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:29:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:37.780 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:29:37.781 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.272596680999996, 2023-01-15T14:29:37.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:29:37.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:37.781 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:37.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:37.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:37.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:37.783 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:29:37.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.783 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:37.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:29:37.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:37.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:37.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:29:37.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:29:37.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:37.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:29:37.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:29:37.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:37.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:29:37.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:29:37.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:29:37.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:29:37.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:37.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:29:37.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:29:37.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:29:37.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:29:37.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:37.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:29:37.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:29:37.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:29:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:29:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:29:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:29:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:29:37.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:29:37.790 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.790 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:37.790 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.790 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:37.790 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:29:37.790 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.901394977000002, 2023-01-15T14:29:37.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:29:37.790 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:37.790 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:37.791 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:37.791 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:37.791 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:37.791 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:29:37.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.792 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:37.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:37.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:29:37.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:37.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:29:37.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:37.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:29:37.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:37.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:37.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:29:37.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:29:37.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:37.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:29:37.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:29:37.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:37.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:29:37.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:29:37.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:29:37.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:29:37.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:37.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:29:37.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:29:37.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:37.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:29:37.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:29:37.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:37.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:29:37.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:29:37.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:37.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:29:37.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:29:37.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:37.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:29:37.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:29:37.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:37.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:29:37.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:29:37.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:29:37.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:29:37.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:37.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:29:37.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:29:37.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:37.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:29:37.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:29:37.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:29:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:29:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:29:37.961 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 55.901278302999998, 2023-01-15T14:29:37.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:29:37.961 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:37.961 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:37.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:37.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:37.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:37.961 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:29:37.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.961 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:37.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:37.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:29:37.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:37.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:29:37.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:37.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:37.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:29:37.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:37.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:37.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:29:37.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:29:37.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:37.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:29:37.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:29:37.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:37.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:29:37.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:29:37.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:29:37.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:29:37.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:29:37.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:29:37.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:29:37.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:37.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:37.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:37.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:29:37.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:29:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:29:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:29:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:29:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:448 3.1 3:8b0a2647:::smithi09025264-8:head [write 1856548~737690] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:29:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:29:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.654020395000003, 2023-01-15T14:29:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40560468799999999, 2023-01-15T14:29:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 448 2023-01-15T14:29:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:29:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:29:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297043+0000", 2023-01-15T14:29:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297477+0000", 2023-01-15T14:29:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043336500000000001 2023-01-15T14:29:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297505+0000", 2023-01-15T14:29:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8583000000000001e-05 2023-01-15T14:29:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397598+0000", 2023-01-15T14:29:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092164 2023-01-15T14:29:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397683+0000", 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5986e-05 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397798+0000", 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114916 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498684+0000", 2023-01-15T14:29:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10088510000000001 2023-01-15T14:29:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498848+0000", 2023-01-15T14:29:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164575 2023-01-15T14:29:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499083+0000", 2023-01-15T14:29:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000235171 2023-01-15T14:29:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499670+0000", 2023-01-15T14:29:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058650200000000003 2023-01-15T14:29:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499688+0000", 2023-01-15T14:29:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8547e-05 2023-01-15T14:29:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.500037+0000", 2023-01-15T14:29:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034891399999999997 2023-01-15T14:29:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.501925+0000", 2023-01-15T14:29:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00188766 2023-01-15T14:29:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.601919+0000", 2023-01-15T14:29:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099994371999999998 2023-01-15T14:29:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702377+0000", 2023-01-15T14:29:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045810500000001 2023-01-15T14:29:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702537+0000", 2023-01-15T14:29:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015941600000000001 2023-01-15T14:29:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702650+0000", 2023-01-15T14:29:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011338899999999999 2023-01-15T14:29:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:468 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (57) in=7b,truncate 3378174] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:29:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775893+0000", 2023-01-15T14:29:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.175173309000002, 2023-01-15T14:29:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39428173999999999, 2023-01-15T14:29:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 468 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775893+0000", 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775893+0000", 2023-01-15T14:29:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775892+0000", 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775897+0000", 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4820000000000002e-06 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775906+0000", 2023-01-15T14:29:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9199999999999993e-06 2023-01-15T14:29:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775911+0000", 2023-01-15T14:29:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5179999999999998e-06 2023-01-15T14:29:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.967706+0000", 2023-01-15T14:29:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.191795189 2023-01-15T14:29:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.967988+0000", 2023-01-15T14:29:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028248100000000001 2023-01-15T14:29:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968459+0000", 2023-01-15T14:29:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000470917 2023-01-15T14:29:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968829+0000", 2023-01-15T14:29:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036951399999999999 2023-01-15T14:29:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968842+0000", 2023-01-15T14:29:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2801e-05 2023-01-15T14:29:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.968890+0000", 2023-01-15T14:29:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8775e-05 2023-01-15T14:29:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969229+0000", 2023-01-15T14:29:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033879800000000002 2023-01-15T14:29:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.169674+0000", 2023-01-15T14:29:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20044432000000001 2023-01-15T14:29:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.169983+0000", 2023-01-15T14:29:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030946800000000003 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.170076+0000", 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2872000000000001e-05 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.170174+0000", 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:29:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:29:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 48.175146026999997, 2023-01-15T14:29:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:29:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:29:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:29:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:29:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:29:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:29:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:29:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:29:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:29:38.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:38.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:29:38.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:29:38.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:38.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:29:38.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:29:38.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.328 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:38.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.328 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:29:38.328 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:29:38.329 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.958895968, 2023-01-15T14:29:38.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:29:38.329 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:38.329 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:38.329 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:38.329 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:38.329 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:38.329 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:29:38.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.329 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:38.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:29:38.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:29:38.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:38.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:29:38.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:29:38.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:38.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:29:38.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:29:38.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:38.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:29:38.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:29:38.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:38.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:29:38.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:29:38.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:38.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:29:38.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:29:38.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:38.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:29:38.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:29:38.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:38.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:29:38.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:29:38.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:38.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:29:38.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:29:38.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:38.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:29:38.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:29:38.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:38.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:29:38.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:29:38.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:38.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:29:38.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:29:38.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:38.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:29:38.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:29:38.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.338 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:38.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.338 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:29:38.338 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:29:38.338 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.958862400999999, 2023-01-15T14:29:38.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:29:38.339 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:38.339 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:38.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:38.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:38.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:38.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:29:38.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.339 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:38.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:38.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:29:38.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:38.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:29:38.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:38.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:29:38.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:38.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:38.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:29:38.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:29:38.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:38.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:29:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:29:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:29:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:29:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:29:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:29:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:29:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:29:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:29:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:29:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:29:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:29:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:29:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:29:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:29:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:29:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:29:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:29:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:29:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:29:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:29:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:38.348 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:38.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:29:38.349 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:29:38.350 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:36.056+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:38.351 INFO:tasks.rados.rados.0.smithi090.stdout:362: done (0 left) 2023-01-15T14:29:38.351 INFO:tasks.rados.rados.0.smithi090.stdout:363: read oid 45 snap -1 2023-01-15T14:29:38.351 INFO:tasks.rados.rados.0.smithi090.stdout:363: expect deleted 2023-01-15T14:29:38.519 INFO:tasks.rados.rados.0.smithi090.stdout:364: delete oid 42 current snap is 35 2023-01-15T14:29:38.520 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:29:38.520 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.7 to 1 (10000) 2023-01-15T14:29:38.522 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-01-15T14:29:38.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:29:38.526 INFO:tasks.rados.rados.0.smithi090.stdout:363: done (1 left) 2023-01-15T14:29:38.526 INFO:tasks.rados.rados.0.smithi090.stdout:364: done (0 left) 2023-01-15T14:29:38.526 INFO:tasks.rados.rados.0.smithi090.stdout:365: rollback oid 3 current snap is 35 2023-01-15T14:29:38.526 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 3 to 30 2023-01-15T14:29:38.526 INFO:tasks.rados.rados.0.smithi090.stdout:366: write (excl) oid 38 current snap is 35 2023-01-15T14:29:38.526 INFO:tasks.rados.rados.0.smithi090.stdout:366: seq_num 159 ranges {672313=691651,1887931=419038} 2023-01-15T14:29:38.526 INFO:tasks.rados.rados.0.smithi090.stdout:366: writing smithi09025264-38 from 672313 to 1363964 tid 1 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout:366: writing smithi09025264-38 from 1887931 to 2306969 tid 2 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout:367: write oid 22 current snap is 35 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout:367: seq_num 160 ranges {689120=483253,1946413=679049} 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout:367: writing smithi09025264-22 from 689120 to 1172373 tid 1 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout:367: writing smithi09025264-22 from 1946413 to 2625462 tid 2 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout:368: write (excl) oid 8 current snap is 35 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout:368: seq_num 161 ranges {705927=558502,2004895=422707} 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout:368: writing smithi09025264-8 from 705927 to 1264429 tid 1 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout:368: writing smithi09025264-8 from 2004895 to 2427602 tid 2 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout:369: snap_create 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:29:38.527 INFO:tasks.rados.rados.0.smithi090.stdout:368: finishing write tid 1 to smithi09025264-8 2023-01-15T14:29:38.528 INFO:tasks.rados.rados.0.smithi090.stdout:367: finishing write tid 1 to smithi09025264-22 2023-01-15T14:29:38.528 INFO:tasks.rados.rados.0.smithi090.stdout:366: finishing write tid 1 to smithi09025264-38 2023-01-15T14:29:38.528 INFO:tasks.rados.rados.0.smithi090.stdout:368: finishing write tid 2 to smithi09025264-8 2023-01-15T14:29:38.528 INFO:tasks.rados.rados.0.smithi090.stdout:368: finishing write tid 3 to smithi09025264-8 2023-01-15T14:29:38.528 INFO:tasks.rados.rados.0.smithi090.stdout:368: finishing write tid 4 to smithi09025264-8 2023-01-15T14:29:38.528 INFO:tasks.rados.rados.0.smithi090.stdout:368: oid 8 updating version 0 to 84 2023-01-15T14:29:38.528 INFO:tasks.rados.rados.0.smithi090.stdout:368: oid 8 version 84 is already newer than 83 2023-01-15T14:29:38.528 INFO:tasks.rados.rados.0.smithi090.stdout:368: oid 8 updating version 84 to 85 2023-01-15T14:29:38.528 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 85 (ObjNum 161 snap 35 seq_num 161) dirty exists 2023-01-15T14:29:38.528 INFO:tasks.rados.rados.0.smithi090.stdout:368: left oid 8 (ObjNum 161 snap 35 seq_num 161) 2023-01-15T14:29:38.528 INFO:tasks.rados.rados.0.smithi090.stdout:368: done (3 left) 2023-01-15T14:29:38.529 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:29:38.529 INFO:tasks.rados.rados.0.smithi090.stdout:367: finishing write tid 2 to smithi09025264-22 2023-01-15T14:29:38.529 INFO:tasks.rados.rados.0.smithi090.stdout:367: finishing write tid 3 to smithi09025264-22 2023-01-15T14:29:38.529 INFO:tasks.rados.rados.0.smithi090.stdout:367: finishing write tid 4 to smithi09025264-22 2023-01-15T14:29:38.529 INFO:tasks.rados.rados.0.smithi090.stdout:367: oid 22 updating version 0 to 110 2023-01-15T14:29:38.529 INFO:tasks.rados.rados.0.smithi090.stdout:367: oid 22 updating version 110 to 111 2023-01-15T14:29:38.529 INFO:tasks.rados.rados.0.smithi090.stdout:367: oid 22 updating version 111 to 112 2023-01-15T14:29:38.529 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 112 (ObjNum 160 snap 35 seq_num 160) dirty exists 2023-01-15T14:29:38.529 INFO:tasks.rados.rados.0.smithi090.stdout:367: left oid 22 (ObjNum 160 snap 35 seq_num 160) 2023-01-15T14:29:38.530 INFO:tasks.rados.rados.0.smithi090.stdout:367: done (2 left) 2023-01-15T14:29:38.530 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:38.530 INFO:tasks.rados.rados.0.smithi090.stdout:366: finishing write tid 2 to smithi09025264-38 2023-01-15T14:29:38.530 INFO:tasks.rados.rados.0.smithi090.stdout:366: finishing write tid 3 to smithi09025264-38 2023-01-15T14:29:38.530 INFO:tasks.rados.rados.0.smithi090.stdout:366: finishing write tid 4 to smithi09025264-38 2023-01-15T14:29:38.530 INFO:tasks.rados.rados.0.smithi090.stdout:366: oid 38 updating version 0 to 101 2023-01-15T14:29:38.530 INFO:tasks.rados.rados.0.smithi090.stdout:366: oid 38 updating version 101 to 103 2023-01-15T14:29:38.530 INFO:tasks.rados.rados.0.smithi090.stdout:366: oid 38 version 103 is already newer than 102 2023-01-15T14:29:38.530 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 103 (ObjNum 159 snap 35 seq_num 159) dirty exists 2023-01-15T14:29:38.530 INFO:tasks.rados.rados.0.smithi090.stdout:366: left oid 38 (ObjNum 159 snap 35 seq_num 159) 2023-01-15T14:29:38.530 INFO:tasks.rados.rados.0.smithi090.stdout:366: done (1 left) 2023-01-15T14:29:38.531 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:38.531 INFO:tasks.rados.rados.0.smithi090.stdout:365: finishing rollback tid 1 to smithi09025264-3 2023-01-15T14:29:38.531 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 119 (ObjNum 81 snap 0 seq_num 0) dirty dne 2023-01-15T14:29:38.531 INFO:tasks.rados.rados.0.smithi090.stdout:365: done (0 left) 2023-01-15T14:29:38.531 INFO:tasks.rados.rados.0.smithi090.stdout:369: done (0 left) 2023-01-15T14:29:38.531 INFO:tasks.rados.rados.0.smithi090.stdout:370: delete oid 30 current snap is 36 2023-01-15T14:29:38.531 INFO:tasks.rados.rados.0.smithi090.stdout:370: done (0 left) 2023-01-15T14:29:38.531 INFO:tasks.rados.rados.0.smithi090.stdout:371: snap_remove snap 33 2023-01-15T14:29:38.532 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:38.345+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:38.533 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:38.516+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:38.624 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:38.616+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:38.689 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:38.689 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:38.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGRecoveryDeleteReply(3.a e92,90 3:52c3e133:::smithi09025264-3:22,67'97)", 2023-01-15T14:29:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:38.621092+0000", 2023-01-15T14:29:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.054227148000000003, 2023-01-15T14:29:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.054256831999999998, 2023-01-15T14:29:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:29:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:38.621092+0000", 2023-01-15T14:29:38.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:38.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:38.621092+0000", 2023-01-15T14:29:38.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:38.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:38.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:38.621090+0000", 2023-01-15T14:29:38.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:38.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:38.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:38.621107+0000", 2023-01-15T14:29:38.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6903000000000001e-05 2023-01-15T14:29:38.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:38.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:38.621143+0000", 2023-01-15T14:29:38.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5707999999999998e-05 2023-01-15T14:29:38.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:38.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:38.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:38.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:38.621161+0000", 2023-01-15T14:29:38.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8240000000000002e-05 2023-01-15T14:29:38.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.693 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:38.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:38.693 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:29:38.694 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:29:38.694 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:38.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:29:38.707 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:38.725 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:38.717+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:38.882 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:38.883 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:38.884 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:38.885 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:29:38.886 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:38.886 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:38.887 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:29:38.888 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:29:38.888 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:29:38.888 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:29:38.888 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:29:38.889 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:29:38.889 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:29:38.889 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:29:38.890 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:38.873+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:38.892 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:38.893 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:29:38.899 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2023-01-15T14:29:38.866616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156797518,"num_objects":76,"num_object_clones":30,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":8,"num_read":888,"num_read_kb":707147,"num_write":802,"num_write_kb":256220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":90937982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1420,"ondisk_log_size":1420,"up":75,"acting":73,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":6229684,"kb_used_data":649680600,"kb_used_omap":6699,"kb_used_meta":0,"kb_avail":649665976,"statfs":{"total":671637155840,"available":665257959424,"internally_reserved":14974976,"allocated":665272934400,"data_stored":665272934400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860792,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":60,"commit_latency_ns":0,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7487573,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":22,"num_read_kb":22863,"num_write":14,"num_write_kb":7316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001230"},"pg_stats":[{"pgid":"3.f","version":"86'72","reported_seq":275,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.184826+0000","last_change":"2023-01-15T14:29:30.059280+0000","last_active":"2023-01-15T14:29:33.184826+0000","last_peered":"2023-01-15T14:29:33.184826+0000","last_clean":"2023-01-15T14:29:33.184826+0000","last_became_active":"2023-01-15T14:28:23.846063+0000","last_became_peered":"2023-01-15T14:28:23.846063+0000","last_unstale":"2023-01-15T14:29:33.184826+0000","last_undegraded":"2023-01-15T14:29:33.184826+0000","last_fullsized":"2023-01-15T14:29:33.184826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:52.368","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010683999999999999,"stat_sum":{"num_bytes":14524782,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":37,"num_read_kb":27958,"num_write":43,"num_write_kb":13905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"3.e","version":"73'25","reported_seq":228,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.084652+0000","last_change":"2023-01-15T14:29:30.058805+0000","last_active":"2023-01-15T14:29:33.084652+0000","last_peered":"2023-01-15T14:29:33.084652+0000","last_clean":"2023-01-15T14:29:33.084652+0000","last_became_active":"2023-01-15T14:28:23.846858+0000","last_became_peered":"2023-01-15T14:28:23.846858+0000","last_unstale":"2023-01-15T14:29:33.084652+0000","last_undegraded":"2023-01-15T14:29:33.084652+0000","last_fullsized":"2023-01-15T14:29:33.084652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:44.514","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012355630000000001,"stat_sum":{"num_bytes":2621859,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":12810,"num_write":6,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"3.d","version":"84'95","reported_seq":328,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.084308+0000","last_change":"2023-01-15T14:29:30.865788+0000","last_active":"2023-01-15T14:29:33.084308+0000","last_peered":"2023-01-15T14:29:33.084308+0000","last_clean":"2023-01-15T14:29:33.084308+0000","last_became_active":"2023-01-15T14:28:54.932913+0000","last_became_peered":"2023-01-15T14:28:54.932913+0000","last_unstale":"2023-01-15T14:29:33.084308+0000","last_undegraded":"2023-01-15T14:29:33.084308+0000","last_fullsized":"2023-01-15T14:29:33.084308+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:29:36.419","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70829518700000005,"stat_sum":{"num_bytes":11372730,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":57,"num_read_kb":60421,"num_write":39,"num_write_kb":15807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"3.c","version":"81'43","reported_seq":192,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.195340+0000","last_change":"2023-01-15T14:29:35.195572+0000","last_active":"2023-01-15T14:29:35.195340+0000","last_peered":"2023-01-15T14:29:35.195340+0000","last_clean":"2023-01-15T14:29:35.195340+0000","last_became_active":"2023-01-15T14:28:54.833568+0000","last_became_peered":"2023-01-15T14:28:54.833568+0000","last_unstale":"2023-01-15T14:29:35.195340+0000","last_undegraded":"2023-01-15T14:29:35.195340+0000","last_fullsized":"2023-01-15T14:29:35.195340+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.421","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10082179600000001,"stat_sum":{"num_bytes":2100875,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":7,"num_read_kb":3,"num_write":20,"num_write_kb":5044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2319333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"3.b","version":"86'30","reported_seq":237,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.094366+0000","last_change":"2023-01-15T14:29:35.195166+0000","last_active":"2023-01-15T14:29:35.094366+0000","last_peered":"2023-01-15T14:29:35.094366+0000","last_clean":"2023-01-15T14:29:35.094366+0000","last_became_active":"2023-01-15T14:28:23.934183+0000","last_became_peered":"2023-01-15T14:28:23.934183+0000","last_unstale":"2023-01-15T14:29:35.094366+0000","last_undegraded":"2023-01-15T14:29:35.094366+0000","last_fullsized":"2023-01-15T14:29:35.094366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:45.586","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101329776,"stat_sum":{"num_bytes":2218524,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":15,"num_read_kb":13386,"num_write":12,"num_write_kb":5195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"3.a","version":"81'121","reported_seq":341,"reported_epoch":90,"state":"remapped+peering","last_fresh":"2023-01-15T14:29:36.096806+0000","last_change":"2023-01-15T14:29:36.096806+0000","last_active":"2023-01-15T14:29:36.096527+0000","last_peered":"2023-01-15T14:29:27.218124+0000","last_clean":"2023-01-15T14:29:27.218124+0000","last_became_active":"2023-01-15T14:29:02.112091+0000","last_became_peered":"2023-01-15T14:29:02.112091+0000","last_unstale":"2023-01-15T14:29:36.096806+0000","last_undegraded":"2023-01-15T14:29:36.096806+0000","last_fullsized":"2023-01-15T14:29:36.096806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041957749999999997,"stat_sum":{"num_bytes":14852985,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":42,"num_read_kb":24511,"num_write":64,"num_write_kb":31038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":21341200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"3.9","version":"84'108","reported_seq":326,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.293416+0000","last_change":"2023-01-15T14:29:35.192159+0000","last_active":"2023-01-15T14:29:35.293416+0000","last_peered":"2023-01-15T14:29:35.293416+0000","last_clean":"2023-01-15T14:29:35.293416+0000","last_became_active":"2023-01-15T14:28:24.109843+0000","last_became_peered":"2023-01-15T14:28:24.109843+0000","last_unstale":"2023-01-15T14:29:35.293416+0000","last_undegraded":"2023-01-15T14:29:35.293416+0000","last_fullsized":"2023-01-15T14:29:35.293416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091506300000000003,"stat_sum":{"num_bytes":10773188,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":70457,"num_write":48,"num_write_kb":20836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"3.8","version":"89'125","reported_seq":375,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:35.092523+0000","last_change":"2023-01-15T14:29:35.092523+0000","last_active":"2023-01-15T14:29:35.092523+0000","last_peered":"2023-01-15T14:29:35.092523+0000","last_clean":"2023-01-15T14:29:35.092523+0000","last_became_active":"2023-01-15T14:28:47.937231+0000","last_became_peered":"2023-01-15T14:28:47.937231+0000","last_unstale":"2023-01-15T14:29:35.092523+0000","last_undegraded":"2023-01-15T14:29:35.092523+0000","last_fullsized":"2023-01-15T14:29:35.092523+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'121","last_scrub_stamp":"2023-01-15T14:29:31.589753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.589753+0000","objects_scrubbed":8,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:54.442","scrub_duration":0.28338864200000002,"objects_trimmed":0,"snaptrim_duration":0.50621750300000001,"stat_sum":{"num_bytes":12937892,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":3,"num_read":74,"num_read_kb":73606,"num_write":58,"num_write_kb":19773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":258,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.084104+0000","last_change":"2023-01-15T14:29:05.329442+0000","last_active":"2023-01-15T14:29:33.084104+0000","last_peered":"2023-01-15T14:29:33.084104+0000","last_clean":"2023-01-15T14:29:33.084104+0000","last_became_active":"2023-01-15T14:29:05.228004+0000","last_became_peered":"2023-01-15T14:29:05.228004+0000","last_unstale":"2023-01-15T14:29:33.084104+0000","last_undegraded":"2023-01-15T14:29:33.084104+0000","last_fullsized":"2023-01-15T14:29:33.084104+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:34.812","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-01-15T14:29:34.186384+0000","last_change":"2023-01-15T14:28:54.829192+0000","last_active":"2023-01-15T14:29:34.186384+0000","last_peered":"2023-01-15T14:29:34.186384+0000","last_clean":"2023-01-15T14:29:34.186384+0000","last_became_active":"2023-01-15T14:28:54.828857+0000","last_became_peered":"2023-01-15T14:28:54.828857+0000","last_unstale":"2023-01-15T14:29:34.186384+0000","last_undegraded":"2023-01-15T14:29:34.186384+0000","last_fullsized":"2023-01-15T14:29:34.186384+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.638","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"82'74","reported_seq":314,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.092585+0000","last_change":"2023-01-15T14:29:35.093000+0000","last_active":"2023-01-15T14:29:35.092585+0000","last_peered":"2023-01-15T14:29:35.092585+0000","last_clean":"2023-01-15T14:29:35.092585+0000","last_became_active":"2023-01-15T14:28:55.217912+0000","last_became_peered":"2023-01-15T14:28:55.217912+0000","last_unstale":"2023-01-15T14:29:35.092585+0000","last_undegraded":"2023-01-15T14:29:35.092585+0000","last_fullsized":"2023-01-15T14:29:35.092585+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.256","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078899999999999999,"stat_sum":{"num_bytes":11579957,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":50,"num_read_kb":45399,"num_write":40,"num_write_kb":17290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10821306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":89,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.083781+0000","last_change":"2023-01-15T14:29:31.583684+0000","last_active":"2023-01-15T14:29:33.083781+0000","last_peered":"2023-01-15T14:29:33.083781+0000","last_clean":"2023-01-15T14:29:33.083781+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:29:33.083781+0000","last_undegraded":"2023-01-15T14:29:33.083781+0000","last_fullsized":"2023-01-15T14:29:33.083781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:31.583599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.583599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:44.691","scrub_duration":0.22681161699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'81","reported_seq":310,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.194499+0000","last_change":"2023-01-15T14:29:35.195750+0000","last_active":"2023-01-15T14:29:35.194499+0000","last_peered":"2023-01-15T14:29:35.194499+0000","last_clean":"2023-01-15T14:29:35.194499+0000","last_became_active":"2023-01-15T14:28:23.842545+0000","last_became_peered":"2023-01-15T14:28:23.842545+0000","last_unstale":"2023-01-15T14:29:35.194499+0000","last_undegraded":"2023-01-15T14:29:35.194499+0000","last_fullsized":"2023-01-15T14:29:35.194499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'81","last_scrub_stamp":"2023-01-15T14:29:32.795334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:32.795334+0000","objects_scrubbed":4,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:33.824","scrub_duration":0.40728601599999997,"objects_trimmed":0,"snaptrim_duration":0.0015649730000000001,"stat_sum":{"num_bytes":9294205,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":53,"num_read_kb":49746,"num_write":43,"num_write_kb":13110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-01-15T14:29:34.186332+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:29:34.186332+0000","last_peered":"2023-01-15T14:29:34.186332+0000","last_clean":"2023-01-15T14:29:34.186332+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:29:34.186332+0000","last_undegraded":"2023-01-15T14:29:34.186332+0000","last_fullsized":"2023-01-15T14:29:34.186332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.689","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"85'165","reported_seq":425,"reported_epoch":90,"state":"remapped+peering","last_fresh":"2023-01-15T14:29:36.196307+0000","last_change":"2023-01-15T14:29:36.196638+0000","last_active":"2023-01-15T14:29:35.090958+0000","last_peered":"2023-01-15T14:29:34.622002+0000","last_clean":"2023-01-15T14:29:34.622002+0000","last_became_active":"2023-01-15T14:29:02.007426+0000","last_became_peered":"2023-01-15T14:29:02.007426+0000","last_unstale":"2023-01-15T14:29:36.196307+0000","last_undegraded":"2023-01-15T14:29:36.196307+0000","last_fullsized":"2023-01-15T14:29:36.196307+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.80652508000000001,"stat_sum":{"num_bytes":15395080,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":99,"num_read_kb":95371,"num_write":70,"num_write_kb":37255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16722767,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,6],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-01-15T14:29:34.186492+0000","last_change":"2023-01-15T14:29:14.703116+0000","last_active":"2023-01-15T14:29:34.186492+0000","last_peered":"2023-01-15T14:29:34.186492+0000","last_clean":"2023-01-15T14:29:34.186492+0000","last_became_active":"2023-01-15T14:29:14.050302+0000","last_became_peered":"2023-01-15T14:29:14.050302+0000","last_unstale":"2023-01-15T14:29:34.186492+0000","last_undegraded":"2023-01-15T14:29:34.186492+0000","last_fullsized":"2023-01-15T14:29:34.186492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:14.703035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:14.703035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:18.827","scrub_duration":0.44938580500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"89'167","reported_seq":407,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.402467+0000","last_change":"2023-01-15T14:29:35.192676+0000","last_active":"2023-01-15T14:29:35.402467+0000","last_peered":"2023-01-15T14:29:35.402467+0000","last_clean":"2023-01-15T14:29:35.402467+0000","last_became_active":"2023-01-15T14:28:23.946074+0000","last_became_peered":"2023-01-15T14:28:23.946074+0000","last_unstale":"2023-01-15T14:29:35.402467+0000","last_undegraded":"2023-01-15T14:29:35.402467+0000","last_fullsized":"2023-01-15T14:29:35.402467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:40.271","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012671379999999999,"stat_sum":{"num_bytes":22096394,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":134,"num_read_kb":129215,"num_write":97,"num_write_kb":27695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"3.5","version":"84'31","reported_seq":221,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.235074+0000","last_change":"2023-01-15T14:29:35.235386+0000","last_active":"2023-01-15T14:29:35.235074+0000","last_peered":"2023-01-15T14:29:35.235074+0000","last_clean":"2023-01-15T14:29:35.235074+0000","last_became_active":"2023-01-15T14:28:54.831872+0000","last_became_peered":"2023-01-15T14:28:54.831872+0000","last_unstale":"2023-01-15T14:29:35.235074+0000","last_undegraded":"2023-01-15T14:29:35.235074+0000","last_fullsized":"2023-01-15T14:29:35.235074+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:46.735","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100685783,"stat_sum":{"num_bytes":8521050,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":15521,"num_write":13,"num_write_kb":10036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4252105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.088856+0000","last_change":"2023-01-15T14:29:29.346425+0000","last_active":"2023-01-15T14:29:35.088856+0000","last_peered":"2023-01-15T14:29:35.088856+0000","last_clean":"2023-01-15T14:29:35.088856+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:29:35.088856+0000","last_undegraded":"2023-01-15T14:29:35.088856+0000","last_fullsized":"2023-01-15T14:29:35.088856+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:52.809","scrub_duration":0.39286881699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"89'78","reported_seq":332,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.699506+0000","last_change":"2023-01-15T14:29:35.699506+0000","last_active":"2023-01-15T14:29:35.699506+0000","last_peered":"2023-01-15T14:29:35.699506+0000","last_clean":"2023-01-15T14:29:35.699506+0000","last_became_active":"2023-01-15T14:28:54.825996+0000","last_became_peered":"2023-01-15T14:28:54.825996+0000","last_unstale":"2023-01-15T14:29:35.699506+0000","last_undegraded":"2023-01-15T14:29:35.699506+0000","last_fullsized":"2023-01-15T14:29:35.699506+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.440","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.61088218500000002,"stat_sum":{"num_bytes":11853694,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":80,"num_read_kb":81292,"num_write":47,"num_write_kb":14300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12100908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":89,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.084526+0000","last_change":"2023-01-15T14:29:25.664703+0000","last_active":"2023-01-15T14:29:33.084526+0000","last_peered":"2023-01-15T14:29:33.084526+0000","last_clean":"2023-01-15T14:29:33.084526+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:29:33.084526+0000","last_undegraded":"2023-01-15T14:29:33.084526+0000","last_fullsized":"2023-01-15T14:29:33.084526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:25.664616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:25.664616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:29.565","scrub_duration":0.40398350300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"74'50","reported_seq":253,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.235447+0000","last_change":"2023-01-15T14:29:35.235690+0000","last_active":"2023-01-15T14:29:35.235447+0000","last_peered":"2023-01-15T14:29:35.235447+0000","last_clean":"2023-01-15T14:29:35.235447+0000","last_became_active":"2023-01-15T14:28:23.741266+0000","last_became_peered":"2023-01-15T14:28:23.741266+0000","last_unstale":"2023-01-15T14:29:35.235447+0000","last_undegraded":"2023-01-15T14:29:35.235447+0000","last_fullsized":"2023-01-15T14:29:35.235447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.041319623999999999,"stat_sum":{"num_bytes":2621859,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":21,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":101,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-01-15T14:29:36.100082+0000","last_change":"2023-01-15T14:29:36.100082+0000","last_active":"2023-01-15T14:29:36.100082+0000","last_peered":"2023-01-15T14:29:36.100082+0000","last_clean":"2023-01-15T14:29:36.100082+0000","last_became_active":"2023-01-15T14:29:36.099234+0000","last_became_peered":"2023-01-15T14:29:36.099234+0000","last_unstale":"2023-01-15T14:29:36.100082+0000","last_undegraded":"2023-01-15T14:29:36.100082+0000","last_fullsized":"2023-01-15T14:29:36.100082+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"84'38","reported_seq":245,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.194622+0000","last_change":"2023-01-15T14:29:35.195430+0000","last_active":"2023-01-15T14:29:35.194622+0000","last_peered":"2023-01-15T14:29:35.194622+0000","last_clean":"2023-01-15T14:29:35.194622+0000","last_became_active":"2023-01-15T14:28:23.841189+0000","last_became_peered":"2023-01-15T14:28:23.841189+0000","last_unstale":"2023-01-15T14:29:35.194622+0000","last_undegraded":"2023-01-15T14:29:35.194622+0000","last_fullsized":"2023-01-15T14:29:35.194622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001170829,"stat_sum":{"num_bytes":3442076,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":7347,"num_write":22,"num_write_kb":8387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.095204+0000","last_change":"2023-01-15T14:28:12.896363+0000","last_active":"2023-01-15T14:29:35.095204+0000","last_peered":"2023-01-15T14:29:35.095204+0000","last_clean":"2023-01-15T14:29:35.095204+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:29:35.095204+0000","last_undegraded":"2023-01-15T14:29:35.095204+0000","last_fullsized":"2023-01-15T14:29:35.095204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.332","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":156207150,"num_objects":74,"num_object_clones":30,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":8,"num_read":772,"num_read_kb":707047,"num_write":643,"num_write_kb":253804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":90347614,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1303,"ondisk_log_size":1303,"up":48,"acting":46,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":851184,"kb_used_data":92848208,"kb_used_omap":743,"kb_used_meta":0,"kb_avail":92848196,"statfs":{"total":95948165120,"available":95076552704,"internally_reserved":12288,"allocated":95076564992,"data_stored":95076564992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":761116,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":8,"commit_latency_ns":0,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.0700000000000003,"5min":4.0700000000000003,"15min":4.0700000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":75.893000000000001,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":9.2089999999999996,"5min":9.2089999999999996,"15min":9.2089999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":71.308999999999997,"5min":71.308999999999997,"15min":71.308999999999997},"last":0.91800000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":75.959999999999994,"5min":75.959999999999994,"15min":75.959999999999994},"last":0.84499999999999997},{"interface":"front","average":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":6.2800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":68.605000000000004,"5min":68.605000000000004,"15min":68.605000000000004},"last":0.81999999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.7910000000000004,"5min":4.7910000000000004,"15min":4.7910000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":94.733999999999995,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.70499999999999996},{"interface":"front","average":{"1min":4.1779999999999999,"5min":4.1779999999999999,"15min":4.1779999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":75.998999999999995},"last":0.90000000000000002}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.7770000000000001,"5min":3.7770000000000001,"15min":3.7770000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":76.034999999999997,"5min":76.034999999999997,"15min":76.034999999999997},"last":1.169}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":9.4160000000000004,"5min":9.4160000000000004,"15min":9.4160000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":76.658000000000001,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":3.5129999999999999,"5min":3.5129999999999999,"15min":3.5129999999999999},"last":0.875}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.327,"5min":6.327,"15min":6.327},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":68.679000000000002,"5min":68.679000000000002,"15min":68.679000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":4.9450000000000003,"5min":4.9450000000000003,"15min":4.9450000000000003},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":95.430000000000007,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.999}]},{"osd":6,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":8.5239999999999991,"5min":8.5239999999999991,"15min":8.5239999999999991},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":68.635999999999996,"5min":68.635999999999996,"15min":68.635999999999996},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.6520000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.94599999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":870888,"kb_used_data":92832928,"kb_used_omap":936,"kb_used_meta":0,"kb_avail":92828492,"statfs":{"total":95948165120,"available":95056375808,"internally_reserved":4542464,"allocated":95060918272,"data_stored":95060918272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":958985,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.98499999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.67800000000000005}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.92800000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.0590000000000002,"5min":6.0590000000000002,"15min":6.0590000000000002},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":81.686999999999998,"5min":81.686999999999998,"15min":81.686999999999998},"last":0.78800000000000003},{"interface":"front","average":{"1min":11.455,"5min":11.455,"15min":11.455},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":255.24299999999999,"5min":255.24299999999999,"15min":255.24299999999999},"last":0.94999999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":6.6680000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":138.197,"5min":138.197,"15min":138.197},"last":0.75700000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.872},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.58299999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.2800000000000002,"5min":4.2800000000000002,"15min":4.2800000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":78.888000000000005,"5min":78.888000000000005,"15min":78.888000000000005},"last":0.84899999999999998},{"interface":"front","average":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":47.332000000000001,"5min":47.332000000000001,"15min":47.332000000000001},"last":0.83299999999999996}]}]},{"osd":5,"up_from":0,"seq":55834574869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":917832,"kb_used_data":92784196,"kb_used_omap":1111,"kb_used_meta":0,"kb_avail":92781548,"statfs":{"total":95948165120,"available":95008305152,"internally_reserved":2711552,"allocated":95011016704,"data_stored":95011016704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1138509,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.915,"5min":4.915,"15min":4.915},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":86.944999999999993,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.92400000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":8.5939999999999994,"5min":8.5939999999999994,"15min":8.5939999999999994},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":99.960999999999999,"5min":99.960999999999999,"15min":99.960999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":1.0069999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":99.884,"5min":99.884,"15min":99.884},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.044}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":12.1,"5min":12.1,"15min":12.1},"last":0.75},{"interface":"front","average":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":6.1509999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":99.921000000000006,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.76800000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":30.971,"5min":30.971,"15min":30.971},"last":0.80800000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.673,"5min":5.673,"15min":5.673},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":99.780000000000001,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":5.1630000000000003,"5min":5.1630000000000003,"15min":5.1630000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":88.186999999999998,"5min":88.186999999999998,"15min":88.186999999999998},"last":1.081}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.128,"5min":13.128,"15min":13.128},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":179.73500000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":11.239000000000001,"5min":11.239000000000001,"15min":11.239000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":99.843000000000004,"5min":99.843000000000004,"15min":99.843000000000004},"last":0.97799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":887720,"kb_used_data":92811740,"kb_used_omap":1049,"kb_used_meta":0,"kb_avail":92811660,"statfs":{"total":95948165120,"available":95039139840,"internally_reserved":81920,"allocated":95039221760,"data_stored":95039221760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1074873,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.8479999999999999,"5min":4.8479999999999999,"15min":4.8479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":93.632999999999996,"5min":93.632999999999996,"15min":93.632999999999996},"last":0.623},{"interface":"front","average":{"1min":7.6420000000000003,"5min":7.6420000000000003,"15min":7.6420000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":93.674000000000007,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.74399999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.532,"5min":13.532,"15min":13.532},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":159.256,"5min":159.256,"15min":159.256},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.70699999999999996}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.3490000000000002,"5min":3.3490000000000002,"15min":3.3490000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":59.896999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":31.233000000000001,"5min":31.233000000000001,"15min":31.233000000000001},"last":0.82799999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":6.5620000000000003,"5min":6.5620000000000003,"15min":6.5620000000000003},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":66.311999999999998,"5min":66.311999999999998,"15min":66.311999999999998},"last":0.85599999999999998},{"interface":"front","average":{"1min":13.874000000000001,"5min":13.874000000000001,"15min":13.874000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":159.47200000000001,"5min":159.47200000000001,"15min":159.47200000000001},"last":0.88700000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":13.903,"15min":13.903},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":159.43000000000001,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":1.1850000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":1.268},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":60.098999999999997,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.93100000000000005}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.0359999999999996,"5min":4.0359999999999996,"15min":4.0359999999999996},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":60.165999999999997,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.014},{"interface":"front","average":{"1min":18.09,"5min":18.09,"15min":18.09},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":159.512,"5min":159.512,"15min":159.512},"last":1.0369999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":886884,"kb_used_data":92813756,"kb_used_omap":759,"kb_used_meta":0,"kb_avail":92812496,"statfs":{"total":95948165120,"available":95039995904,"internally_reserved":1290240,"allocated":95041286144,"data_stored":95041286144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":777592,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":12,"commit_latency_ns":0,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.45200000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.5979999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":3.3559999999999999,"5min":3.3559999999999999,"15min":3.3559999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":47.518000000000001,"5min":47.518000000000001,"15min":47.518000000000001},"last":0.40899999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":10.401999999999999,"5min":10.401999999999999,"15min":10.401999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":183.53,"5min":183.53,"15min":183.53},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.51500000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.6499999999999999,"15min":3.6499999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":53.139000000000003,"5min":53.139000000000003,"15min":53.139000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":3.8090000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":53.186999999999998,"5min":53.186999999999998,"15min":53.186999999999998},"last":0.78900000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.77300000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":0.88800000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":37.124000000000002,"5min":37.124000000000002,"15min":37.124000000000002},"last":0.58399999999999996},{"interface":"front","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":30.445,"5min":30.445,"15min":30.445},"last":0.84399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":894384,"kb_used_data":92807396,"kb_used_omap":1008,"kb_used_meta":0,"kb_avail":92804996,"statfs":{"total":95948165120,"available":95032315904,"internally_reserved":2457600,"allocated":95034773504,"data_stored":95034773504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1032477,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":8,"commit_latency_ns":0,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":3.8929999999999998,"15min":3.8929999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":46.591000000000001,"5min":46.591000000000001,"15min":46.591000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":50.616,"5min":50.616,"15min":50.616},"last":0.53700000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":46.747999999999998,"5min":46.747999999999998,"15min":46.747999999999998},"last":0.93999999999999995}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":19.835999999999999,"5min":19.835999999999999,"15min":19.835999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":282.01400000000001,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":7.2450000000000001,"5min":7.2450000000000001,"15min":7.2450000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":53.145000000000003,"5min":53.145000000000003,"15min":53.145000000000003},"last":0.67700000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.7279999999999998,"5min":4.7279999999999998,"15min":4.7279999999999998},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":50.887999999999998,"5min":50.887999999999998,"15min":50.887999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":3.605,"5min":3.605,"15min":3.605},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":52.512,"5min":52.512,"15min":52.512},"last":0.98799999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":1.056},{"interface":"front","average":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":4.4379999999999997},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":46.905999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.1000000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.0019999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":50.844999999999999,"5min":50.844999999999999,"15min":50.844999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.873}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":52.595999999999997,"5min":52.595999999999997,"15min":52.595999999999997},"last":1.1539999999999999},{"interface":"front","average":{"1min":4.75,"5min":4.75,"15min":4.75},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":50.804000000000002,"5min":50.804000000000002,"15min":50.804000000000002},"last":0.96899999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":920792,"kb_used_data":92782376,"kb_used_omap":1091,"kb_used_meta":0,"kb_avail":92778588,"statfs":{"total":95948165120,"available":95005274112,"internally_reserved":3878912,"allocated":95009153024,"data_stored":95009153024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1117240,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.48799999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":16.779,"5min":16.779,"15min":16.779},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.45700000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.85899999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":9.798,"5min":9.798,"15min":9.798},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":176.34399999999999,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":0.752}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.78400000000000003}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":15.670999999999999,"5min":15.670999999999999,"15min":15.670999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":292.61099999999999,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":0.96099999999999997}]}]}],"pool_statfs":[]}} 2023-01-15T14:29:39.073 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:29:39.258 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:39.258 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:39.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:29:39.260 INFO:tasks.rados.rados.0.smithi090.stdout:371: done (0 left) 2023-01-15T14:29:39.260 INFO:tasks.rados.rados.0.smithi090.stdout:372: copy_from oid 46 from oid 21 current snap is 36 2023-01-15T14:29:39.260 INFO:tasks.rados.rados.0.smithi090.stdout:373: read oid 29 snap -1 2023-01-15T14:29:39.260 INFO:tasks.rados.rados.0.smithi090.stdout:373: expect (ObjNum 132 snap 24 seq_num 132) 2023-01-15T14:29:39.260 INFO:tasks.rados.rados.0.smithi090.stdout:374: write (excl) oid 33 current snap is 36 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:374: seq_num 163 ranges {739541=709000,2121859=426376} 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:374: writing smithi09025264-33 from 739541 to 1448541 tid 1 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:374: writing smithi09025264-33 from 2121859 to 2548235 tid 2 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:375: read oid 27 snap -1 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:375: expect (ObjNum 101 snap 21 seq_num 101) 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:376: write oid 41 current snap is 36 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:376: seq_num 164 ranges {756348=784249,2063988=686387} 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:376: writing smithi09025264-41 from 756348 to 1540597 tid 1 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:376: writing smithi09025264-41 from 2063988 to 2750375 tid 2 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:377: copy_from oid 18 from oid 14 current snap is 36 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:373: expect (ObjNum 132 snap 24 seq_num 132) 2023-01-15T14:29:39.261 INFO:tasks.rados.rados.0.smithi090.stdout:373: done (5 left) 2023-01-15T14:29:39.262 INFO:tasks.rados.rados.0.smithi090.stdout:378: delete oid 31 current snap is 36 2023-01-15T14:29:39.262 INFO:tasks.rados.rados.0.smithi090.stdout:374: finishing write tid 1 to smithi09025264-33 2023-01-15T14:29:39.262 INFO:tasks.rados.rados.0.smithi090.stdout:375: expect (ObjNum 101 snap 21 seq_num 101) 2023-01-15T14:29:39.262 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:39.249+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:39.359 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:39.353+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:39.422 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:39.422 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:39.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:29:39.423 INFO:tasks.rados.rados.0.smithi090.stdout:376: finishing write tid 1 to smithi09025264-41 2023-01-15T14:29:39.440 INFO:tasks.rados.rados.0.smithi090.stdout:377: finishing copy_from to smithi09025264-18 2023-01-15T14:29:39.442 INFO:tasks.rados.rados.0.smithi090.stdout:377: got expected ENOENT (src dne) 2023-01-15T14:29:39.453 INFO:tasks.rados.rados.0.smithi090.stdout:376: finishing write tid 2 to smithi09025264-41 2023-01-15T14:29:39.455 INFO:tasks.rados.rados.0.smithi090.stdout:375: done (5 left) 2023-01-15T14:29:39.455 INFO:tasks.rados.rados.0.smithi090.stdout:378: done (4 left) 2023-01-15T14:29:39.455 INFO:tasks.rados.rados.0.smithi090.stdout:379: read oid 37 snap -1 2023-01-15T14:29:39.455 INFO:tasks.rados.rados.0.smithi090.stdout:379: expect (ObjNum 132 snap 24 seq_num 132) 2023-01-15T14:29:39.455 INFO:tasks.rados.rados.0.smithi090.stdout:380: delete oid 5 current snap is 36 2023-01-15T14:29:39.460 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:39.452+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:39.528 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:39.630 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:39.631 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:29:39.632 INFO:tasks.rados.rados.0.smithi090.stdout:374: finishing write tid 2 to smithi09025264-33 2023-01-15T14:29:39.632 INFO:tasks.rados.rados.0.smithi090.stdout:374: finishing write tid 3 to smithi09025264-33 2023-01-15T14:29:39.633 INFO:tasks.rados.rados.0.smithi090.stdout:374: finishing write tid 4 to smithi09025264-33 2023-01-15T14:29:39.633 INFO:tasks.rados.rados.0.smithi090.stdout:374: oid 33 updating version 0 to 50 2023-01-15T14:29:39.633 INFO:tasks.rados.rados.0.smithi090.stdout:374: oid 33 updating version 50 to 51 2023-01-15T14:29:39.633 INFO:tasks.rados.rados.0.smithi090.stdout:374: oid 33 version 51 is already newer than 49 2023-01-15T14:29:39.633 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 51 (ObjNum 163 snap 36 seq_num 163) dirty exists 2023-01-15T14:29:39.633 INFO:tasks.rados.rados.0.smithi090.stdout:374: left oid 33 (ObjNum 163 snap 36 seq_num 163) 2023-01-15T14:29:39.633 INFO:tasks.rados.rados.0.smithi090.stdout:377: finishing copy_from racing read to smithi09025264-18 2023-01-15T14:29:39.634 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:39.637 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:29:39.637 INFO:tasks.rados.rados.0.smithi090.stdout:376: finishing write tid 3 to smithi09025264-41 2023-01-15T14:29:39.637 INFO:tasks.rados.rados.0.smithi090.stdout:376: finishing write tid 4 to smithi09025264-41 2023-01-15T14:29:39.637 INFO:tasks.rados.rados.0.smithi090.stdout:376: oid 41 updating version 0 to 116 2023-01-15T14:29:39.638 INFO:tasks.rados.rados.0.smithi090.stdout:376: oid 41 version 116 is already newer than 115 2023-01-15T14:29:39.638 INFO:tasks.rados.rados.0.smithi090.stdout:376: oid 41 updating version 116 to 117 2023-01-15T14:29:39.638 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 117 (ObjNum 164 snap 36 seq_num 164) dirty exists 2023-01-15T14:29:39.638 INFO:tasks.rados.rados.0.smithi090.stdout:376: left oid 41 (ObjNum 164 snap 36 seq_num 164) 2023-01-15T14:29:39.638 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:39.624+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:39.639 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":132,"stamp":"2023-01-15T14:29:39.107621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156797518,"num_objects":76,"num_object_clones":30,"num_object_copies":228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":8,"num_read":888,"num_read_kb":707147,"num_write":802,"num_write_kb":256220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":59,"num_bytes_recovered":90937982,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1420,"ondisk_log_size":1420,"up":75,"acting":73,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":6229684,"kb_used_data":649680600,"kb_used_omap":6699,"kb_used_meta":0,"kb_avail":649665976,"statfs":{"total":671637155840,"available":665257959424,"internally_reserved":14974976,"allocated":665272934400,"data_stored":665272934400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6860792,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":60,"commit_latency_ns":0,"apply_latency_ns":60000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6374645,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":21,"num_read_kb":22862,"num_write":10,"num_write_kb":6228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020412"},"pg_stats":[{"pgid":"3.f","version":"86'72","reported_seq":275,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.184826+0000","last_change":"2023-01-15T14:29:30.059280+0000","last_active":"2023-01-15T14:29:33.184826+0000","last_peered":"2023-01-15T14:29:33.184826+0000","last_clean":"2023-01-15T14:29:33.184826+0000","last_became_active":"2023-01-15T14:28:23.846063+0000","last_became_peered":"2023-01-15T14:28:23.846063+0000","last_unstale":"2023-01-15T14:29:33.184826+0000","last_undegraded":"2023-01-15T14:29:33.184826+0000","last_fullsized":"2023-01-15T14:29:33.184826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:52.368","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010683999999999999,"stat_sum":{"num_bytes":14524782,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":37,"num_read_kb":27958,"num_write":43,"num_write_kb":13905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"3.e","version":"73'25","reported_seq":228,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.084652+0000","last_change":"2023-01-15T14:29:30.058805+0000","last_active":"2023-01-15T14:29:33.084652+0000","last_peered":"2023-01-15T14:29:33.084652+0000","last_clean":"2023-01-15T14:29:33.084652+0000","last_became_active":"2023-01-15T14:28:23.846858+0000","last_became_peered":"2023-01-15T14:28:23.846858+0000","last_unstale":"2023-01-15T14:29:33.084652+0000","last_undegraded":"2023-01-15T14:29:33.084652+0000","last_fullsized":"2023-01-15T14:29:33.084652+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:44.514","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012355630000000001,"stat_sum":{"num_bytes":2621859,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":12810,"num_write":6,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"3.d","version":"84'95","reported_seq":328,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.084308+0000","last_change":"2023-01-15T14:29:30.865788+0000","last_active":"2023-01-15T14:29:33.084308+0000","last_peered":"2023-01-15T14:29:33.084308+0000","last_clean":"2023-01-15T14:29:33.084308+0000","last_became_active":"2023-01-15T14:28:54.932913+0000","last_became_peered":"2023-01-15T14:28:54.932913+0000","last_unstale":"2023-01-15T14:29:33.084308+0000","last_undegraded":"2023-01-15T14:29:33.084308+0000","last_fullsized":"2023-01-15T14:29:33.084308+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:29:36.419","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70829518700000005,"stat_sum":{"num_bytes":11372730,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":57,"num_read_kb":60421,"num_write":39,"num_write_kb":15807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"3.c","version":"81'43","reported_seq":192,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.195340+0000","last_change":"2023-01-15T14:29:35.195572+0000","last_active":"2023-01-15T14:29:35.195340+0000","last_peered":"2023-01-15T14:29:35.195340+0000","last_clean":"2023-01-15T14:29:35.195340+0000","last_became_active":"2023-01-15T14:28:54.833568+0000","last_became_peered":"2023-01-15T14:28:54.833568+0000","last_unstale":"2023-01-15T14:29:35.195340+0000","last_undegraded":"2023-01-15T14:29:35.195340+0000","last_fullsized":"2023-01-15T14:29:35.195340+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.421","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10082179600000001,"stat_sum":{"num_bytes":2100875,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":7,"num_read_kb":3,"num_write":20,"num_write_kb":5044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2319333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"3.b","version":"86'30","reported_seq":237,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.094366+0000","last_change":"2023-01-15T14:29:35.195166+0000","last_active":"2023-01-15T14:29:35.094366+0000","last_peered":"2023-01-15T14:29:35.094366+0000","last_clean":"2023-01-15T14:29:35.094366+0000","last_became_active":"2023-01-15T14:28:23.934183+0000","last_became_peered":"2023-01-15T14:28:23.934183+0000","last_unstale":"2023-01-15T14:29:35.094366+0000","last_undegraded":"2023-01-15T14:29:35.094366+0000","last_fullsized":"2023-01-15T14:29:35.094366+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:45.586","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101329776,"stat_sum":{"num_bytes":2218524,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":15,"num_read_kb":13386,"num_write":12,"num_write_kb":5195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"3.a","version":"81'121","reported_seq":341,"reported_epoch":90,"state":"remapped+peering","last_fresh":"2023-01-15T14:29:36.096806+0000","last_change":"2023-01-15T14:29:36.096806+0000","last_active":"2023-01-15T14:29:36.096527+0000","last_peered":"2023-01-15T14:29:27.218124+0000","last_clean":"2023-01-15T14:29:27.218124+0000","last_became_active":"2023-01-15T14:29:02.112091+0000","last_became_peered":"2023-01-15T14:29:02.112091+0000","last_unstale":"2023-01-15T14:29:36.096806+0000","last_undegraded":"2023-01-15T14:29:36.096806+0000","last_fullsized":"2023-01-15T14:29:36.096806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041957749999999997,"stat_sum":{"num_bytes":14852985,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":42,"num_read_kb":24511,"num_write":64,"num_write_kb":31038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":21341200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"3.9","version":"84'108","reported_seq":326,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.293416+0000","last_change":"2023-01-15T14:29:35.192159+0000","last_active":"2023-01-15T14:29:35.293416+0000","last_peered":"2023-01-15T14:29:35.293416+0000","last_clean":"2023-01-15T14:29:35.293416+0000","last_became_active":"2023-01-15T14:28:24.109843+0000","last_became_peered":"2023-01-15T14:28:24.109843+0000","last_unstale":"2023-01-15T14:29:35.293416+0000","last_undegraded":"2023-01-15T14:29:35.293416+0000","last_fullsized":"2023-01-15T14:29:35.293416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091506300000000003,"stat_sum":{"num_bytes":10773188,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":70457,"num_write":48,"num_write_kb":20836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"3.8","version":"89'125","reported_seq":375,"reported_epoch":89,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:35.092523+0000","last_change":"2023-01-15T14:29:35.092523+0000","last_active":"2023-01-15T14:29:35.092523+0000","last_peered":"2023-01-15T14:29:35.092523+0000","last_clean":"2023-01-15T14:29:35.092523+0000","last_became_active":"2023-01-15T14:28:47.937231+0000","last_became_peered":"2023-01-15T14:28:47.937231+0000","last_unstale":"2023-01-15T14:29:35.092523+0000","last_undegraded":"2023-01-15T14:29:35.092523+0000","last_fullsized":"2023-01-15T14:29:35.092523+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'121","last_scrub_stamp":"2023-01-15T14:29:31.589753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.589753+0000","objects_scrubbed":8,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:54.442","scrub_duration":0.28338864200000002,"objects_trimmed":0,"snaptrim_duration":0.50621750300000001,"stat_sum":{"num_bytes":12937892,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":3,"num_read":74,"num_read_kb":73606,"num_write":58,"num_write_kb":19773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":258,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.084104+0000","last_change":"2023-01-15T14:29:05.329442+0000","last_active":"2023-01-15T14:29:33.084104+0000","last_peered":"2023-01-15T14:29:33.084104+0000","last_clean":"2023-01-15T14:29:33.084104+0000","last_became_active":"2023-01-15T14:29:05.228004+0000","last_became_peered":"2023-01-15T14:29:05.228004+0000","last_unstale":"2023-01-15T14:29:33.084104+0000","last_undegraded":"2023-01-15T14:29:33.084104+0000","last_fullsized":"2023-01-15T14:29:33.084104+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:34.812","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-01-15T14:29:34.186384+0000","last_change":"2023-01-15T14:28:54.829192+0000","last_active":"2023-01-15T14:29:34.186384+0000","last_peered":"2023-01-15T14:29:34.186384+0000","last_clean":"2023-01-15T14:29:34.186384+0000","last_became_active":"2023-01-15T14:28:54.828857+0000","last_became_peered":"2023-01-15T14:28:54.828857+0000","last_unstale":"2023-01-15T14:29:34.186384+0000","last_undegraded":"2023-01-15T14:29:34.186384+0000","last_fullsized":"2023-01-15T14:29:34.186384+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.638","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"82'74","reported_seq":314,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.092585+0000","last_change":"2023-01-15T14:29:35.093000+0000","last_active":"2023-01-15T14:29:35.092585+0000","last_peered":"2023-01-15T14:29:35.092585+0000","last_clean":"2023-01-15T14:29:35.092585+0000","last_became_active":"2023-01-15T14:28:55.217912+0000","last_became_peered":"2023-01-15T14:28:55.217912+0000","last_unstale":"2023-01-15T14:29:35.092585+0000","last_undegraded":"2023-01-15T14:29:35.092585+0000","last_fullsized":"2023-01-15T14:29:35.092585+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.256","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078899999999999999,"stat_sum":{"num_bytes":11579957,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":50,"num_read_kb":45399,"num_write":40,"num_write_kb":17290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10821306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":89,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.083781+0000","last_change":"2023-01-15T14:29:31.583684+0000","last_active":"2023-01-15T14:29:33.083781+0000","last_peered":"2023-01-15T14:29:33.083781+0000","last_clean":"2023-01-15T14:29:33.083781+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:29:33.083781+0000","last_undegraded":"2023-01-15T14:29:33.083781+0000","last_fullsized":"2023-01-15T14:29:33.083781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:31.583599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.583599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:44.691","scrub_duration":0.22681161699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"82'81","reported_seq":310,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.194499+0000","last_change":"2023-01-15T14:29:35.195750+0000","last_active":"2023-01-15T14:29:35.194499+0000","last_peered":"2023-01-15T14:29:35.194499+0000","last_clean":"2023-01-15T14:29:35.194499+0000","last_became_active":"2023-01-15T14:28:23.842545+0000","last_became_peered":"2023-01-15T14:28:23.842545+0000","last_unstale":"2023-01-15T14:29:35.194499+0000","last_undegraded":"2023-01-15T14:29:35.194499+0000","last_fullsized":"2023-01-15T14:29:35.194499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'81","last_scrub_stamp":"2023-01-15T14:29:32.795334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:32.795334+0000","objects_scrubbed":4,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:33.824","scrub_duration":0.40728601599999997,"objects_trimmed":0,"snaptrim_duration":0.0015649730000000001,"stat_sum":{"num_bytes":9294205,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":53,"num_read_kb":49746,"num_write":43,"num_write_kb":13110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-01-15T14:29:34.186332+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:29:34.186332+0000","last_peered":"2023-01-15T14:29:34.186332+0000","last_clean":"2023-01-15T14:29:34.186332+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:29:34.186332+0000","last_undegraded":"2023-01-15T14:29:34.186332+0000","last_fullsized":"2023-01-15T14:29:34.186332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.689","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"85'165","reported_seq":425,"reported_epoch":90,"state":"remapped+peering","last_fresh":"2023-01-15T14:29:36.196307+0000","last_change":"2023-01-15T14:29:36.196638+0000","last_active":"2023-01-15T14:29:35.090958+0000","last_peered":"2023-01-15T14:29:34.622002+0000","last_clean":"2023-01-15T14:29:34.622002+0000","last_became_active":"2023-01-15T14:29:02.007426+0000","last_became_peered":"2023-01-15T14:29:02.007426+0000","last_unstale":"2023-01-15T14:29:36.196307+0000","last_undegraded":"2023-01-15T14:29:36.196307+0000","last_fullsized":"2023-01-15T14:29:36.196307+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.80652508000000001,"stat_sum":{"num_bytes":15395080,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":99,"num_read_kb":95371,"num_write":70,"num_write_kb":37255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16722767,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,6],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-01-15T14:29:34.186492+0000","last_change":"2023-01-15T14:29:14.703116+0000","last_active":"2023-01-15T14:29:34.186492+0000","last_peered":"2023-01-15T14:29:34.186492+0000","last_clean":"2023-01-15T14:29:34.186492+0000","last_became_active":"2023-01-15T14:29:14.050302+0000","last_became_peered":"2023-01-15T14:29:14.050302+0000","last_unstale":"2023-01-15T14:29:34.186492+0000","last_undegraded":"2023-01-15T14:29:34.186492+0000","last_fullsized":"2023-01-15T14:29:34.186492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:14.703035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:14.703035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:18.827","scrub_duration":0.44938580500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"89'167","reported_seq":407,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.402467+0000","last_change":"2023-01-15T14:29:35.192676+0000","last_active":"2023-01-15T14:29:35.402467+0000","last_peered":"2023-01-15T14:29:35.402467+0000","last_clean":"2023-01-15T14:29:35.402467+0000","last_became_active":"2023-01-15T14:28:23.946074+0000","last_became_peered":"2023-01-15T14:28:23.946074+0000","last_unstale":"2023-01-15T14:29:35.402467+0000","last_undegraded":"2023-01-15T14:29:35.402467+0000","last_fullsized":"2023-01-15T14:29:35.402467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:40.271","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012671379999999999,"stat_sum":{"num_bytes":22096394,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":134,"num_read_kb":129215,"num_write":97,"num_write_kb":27695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"3.5","version":"84'31","reported_seq":221,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.235074+0000","last_change":"2023-01-15T14:29:35.235386+0000","last_active":"2023-01-15T14:29:35.235074+0000","last_peered":"2023-01-15T14:29:35.235074+0000","last_clean":"2023-01-15T14:29:35.235074+0000","last_became_active":"2023-01-15T14:28:54.831872+0000","last_became_peered":"2023-01-15T14:28:54.831872+0000","last_unstale":"2023-01-15T14:29:35.235074+0000","last_undegraded":"2023-01-15T14:29:35.235074+0000","last_fullsized":"2023-01-15T14:29:35.235074+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:46.735","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100685783,"stat_sum":{"num_bytes":8521050,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":15521,"num_write":13,"num_write_kb":10036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4252105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.088856+0000","last_change":"2023-01-15T14:29:29.346425+0000","last_active":"2023-01-15T14:29:35.088856+0000","last_peered":"2023-01-15T14:29:35.088856+0000","last_clean":"2023-01-15T14:29:35.088856+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:29:35.088856+0000","last_undegraded":"2023-01-15T14:29:35.088856+0000","last_fullsized":"2023-01-15T14:29:35.088856+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:52.809","scrub_duration":0.39286881699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"89'78","reported_seq":332,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.699506+0000","last_change":"2023-01-15T14:29:35.699506+0000","last_active":"2023-01-15T14:29:35.699506+0000","last_peered":"2023-01-15T14:29:35.699506+0000","last_clean":"2023-01-15T14:29:35.699506+0000","last_became_active":"2023-01-15T14:28:54.825996+0000","last_became_peered":"2023-01-15T14:28:54.825996+0000","last_unstale":"2023-01-15T14:29:35.699506+0000","last_undegraded":"2023-01-15T14:29:35.699506+0000","last_fullsized":"2023-01-15T14:29:35.699506+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.440","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.61088218500000002,"stat_sum":{"num_bytes":11853694,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":80,"num_read_kb":81292,"num_write":47,"num_write_kb":14300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12100908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":89,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-01-15T14:29:33.084526+0000","last_change":"2023-01-15T14:29:25.664703+0000","last_active":"2023-01-15T14:29:33.084526+0000","last_peered":"2023-01-15T14:29:33.084526+0000","last_clean":"2023-01-15T14:29:33.084526+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:29:33.084526+0000","last_undegraded":"2023-01-15T14:29:33.084526+0000","last_fullsized":"2023-01-15T14:29:33.084526+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:25.664616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:25.664616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:29.565","scrub_duration":0.40398350300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"74'50","reported_seq":253,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.235447+0000","last_change":"2023-01-15T14:29:35.235690+0000","last_active":"2023-01-15T14:29:35.235447+0000","last_peered":"2023-01-15T14:29:35.235447+0000","last_clean":"2023-01-15T14:29:35.235447+0000","last_became_active":"2023-01-15T14:28:23.741266+0000","last_became_peered":"2023-01-15T14:28:23.741266+0000","last_unstale":"2023-01-15T14:29:35.235447+0000","last_undegraded":"2023-01-15T14:29:35.235447+0000","last_fullsized":"2023-01-15T14:29:35.235447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.041319623999999999,"stat_sum":{"num_bytes":2621859,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":21,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":101,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-01-15T14:29:36.100082+0000","last_change":"2023-01-15T14:29:36.100082+0000","last_active":"2023-01-15T14:29:36.100082+0000","last_peered":"2023-01-15T14:29:36.100082+0000","last_clean":"2023-01-15T14:29:36.100082+0000","last_became_active":"2023-01-15T14:29:36.099234+0000","last_became_peered":"2023-01-15T14:29:36.099234+0000","last_unstale":"2023-01-15T14:29:36.100082+0000","last_undegraded":"2023-01-15T14:29:36.100082+0000","last_fullsized":"2023-01-15T14:29:36.100082+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"84'38","reported_seq":245,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.194622+0000","last_change":"2023-01-15T14:29:35.195430+0000","last_active":"2023-01-15T14:29:35.194622+0000","last_peered":"2023-01-15T14:29:35.194622+0000","last_clean":"2023-01-15T14:29:35.194622+0000","last_became_active":"2023-01-15T14:28:23.841189+0000","last_became_peered":"2023-01-15T14:28:23.841189+0000","last_unstale":"2023-01-15T14:29:35.194622+0000","last_undegraded":"2023-01-15T14:29:35.194622+0000","last_fullsized":"2023-01-15T14:29:35.194622+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001170829,"stat_sum":{"num_bytes":3442076,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":7347,"num_write":22,"num_write_kb":8387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.095204+0000","last_change":"2023-01-15T14:28:12.896363+0000","last_active":"2023-01-15T14:29:35.095204+0000","last_peered":"2023-01-15T14:29:35.095204+0000","last_clean":"2023-01-15T14:29:35.095204+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:29:35.095204+0000","last_undegraded":"2023-01-15T14:29:35.095204+0000","last_fullsized":"2023-01-15T14:29:35.095204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.332","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":156207150,"num_objects":74,"num_object_clones":30,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":8,"num_read":772,"num_read_kb":707047,"num_write":643,"num_write_kb":253804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":90347614,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1303,"ondisk_log_size":1303,"up":48,"acting":46,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":851184,"kb_used_data":92848208,"kb_used_omap":743,"kb_used_meta":0,"kb_avail":92848196,"statfs":{"total":95948165120,"available":95076552704,"internally_reserved":12288,"allocated":95076564992,"data_stored":95076564992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":761116,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":8,"commit_latency_ns":0,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.0700000000000003,"5min":4.0700000000000003,"15min":4.0700000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":75.893000000000001,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":9.2089999999999996,"5min":9.2089999999999996,"15min":9.2089999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":71.308999999999997,"5min":71.308999999999997,"15min":71.308999999999997},"last":0.91800000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":75.959999999999994,"5min":75.959999999999994,"15min":75.959999999999994},"last":0.84499999999999997},{"interface":"front","average":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":6.2800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":68.605000000000004,"5min":68.605000000000004,"15min":68.605000000000004},"last":0.81999999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.7910000000000004,"5min":4.7910000000000004,"15min":4.7910000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":94.733999999999995,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.70499999999999996},{"interface":"front","average":{"1min":4.1779999999999999,"5min":4.1779999999999999,"15min":4.1779999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":75.998999999999995},"last":0.90000000000000002}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.7770000000000001,"5min":3.7770000000000001,"15min":3.7770000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":76.034999999999997,"5min":76.034999999999997,"15min":76.034999999999997},"last":1.169}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":9.4160000000000004,"5min":9.4160000000000004,"15min":9.4160000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":76.658000000000001,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":3.5129999999999999,"5min":3.5129999999999999,"15min":3.5129999999999999},"last":0.875}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.327,"5min":6.327,"15min":6.327},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":68.679000000000002,"5min":68.679000000000002,"15min":68.679000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":4.9450000000000003,"5min":4.9450000000000003,"15min":4.9450000000000003},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":95.430000000000007,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.999}]},{"osd":6,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":8.5239999999999991,"5min":8.5239999999999991,"15min":8.5239999999999991},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":68.635999999999996,"5min":68.635999999999996,"15min":68.635999999999996},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.6520000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.94599999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":870888,"kb_used_data":92832928,"kb_used_omap":936,"kb_used_meta":0,"kb_avail":92828492,"statfs":{"total":95948165120,"available":95056375808,"internally_reserved":4542464,"allocated":95060918272,"data_stored":95060918272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":958985,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.98499999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.67800000000000005}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.92800000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.0590000000000002,"5min":6.0590000000000002,"15min":6.0590000000000002},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":81.686999999999998,"5min":81.686999999999998,"15min":81.686999999999998},"last":0.78800000000000003},{"interface":"front","average":{"1min":11.455,"5min":11.455,"15min":11.455},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":255.24299999999999,"5min":255.24299999999999,"15min":255.24299999999999},"last":0.94999999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":6.6680000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":138.197,"5min":138.197,"15min":138.197},"last":0.75700000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.872},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.58299999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.2800000000000002,"5min":4.2800000000000002,"15min":4.2800000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":78.888000000000005,"5min":78.888000000000005,"15min":78.888000000000005},"last":0.84899999999999998},{"interface":"front","average":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":47.332000000000001,"5min":47.332000000000001,"15min":47.332000000000001},"last":0.83299999999999996}]}]},{"osd":5,"up_from":0,"seq":55834574869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":917832,"kb_used_data":92784196,"kb_used_omap":1111,"kb_used_meta":0,"kb_avail":92781548,"statfs":{"total":95948165120,"available":95008305152,"internally_reserved":2711552,"allocated":95011016704,"data_stored":95011016704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1138509,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.915,"5min":4.915,"15min":4.915},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":86.944999999999993,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.92400000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":8.5939999999999994,"5min":8.5939999999999994,"15min":8.5939999999999994},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":99.960999999999999,"5min":99.960999999999999,"15min":99.960999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":1.0069999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":99.884,"5min":99.884,"15min":99.884},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.044}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":12.1,"5min":12.1,"15min":12.1},"last":0.75},{"interface":"front","average":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":6.1509999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":99.921000000000006,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.76800000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":30.971,"5min":30.971,"15min":30.971},"last":0.80800000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.673,"5min":5.673,"15min":5.673},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":99.780000000000001,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":5.1630000000000003,"5min":5.1630000000000003,"15min":5.1630000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":88.186999999999998,"5min":88.186999999999998,"15min":88.186999999999998},"last":1.081}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.128,"5min":13.128,"15min":13.128},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":179.73500000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":11.239000000000001,"5min":11.239000000000001,"15min":11.239000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":99.843000000000004,"5min":99.843000000000004,"15min":99.843000000000004},"last":0.97799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":887720,"kb_used_data":92811740,"kb_used_omap":1049,"kb_used_meta":0,"kb_avail":92811660,"statfs":{"total":95948165120,"available":95039139840,"internally_reserved":81920,"allocated":95039221760,"data_stored":95039221760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1074873,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.8479999999999999,"5min":4.8479999999999999,"15min":4.8479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":93.632999999999996,"5min":93.632999999999996,"15min":93.632999999999996},"last":0.623},{"interface":"front","average":{"1min":7.6420000000000003,"5min":7.6420000000000003,"15min":7.6420000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":93.674000000000007,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.74399999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.532,"5min":13.532,"15min":13.532},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":159.256,"5min":159.256,"15min":159.256},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.70699999999999996}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.3490000000000002,"5min":3.3490000000000002,"15min":3.3490000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":59.896999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":31.233000000000001,"5min":31.233000000000001,"15min":31.233000000000001},"last":0.82799999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":6.5620000000000003,"5min":6.5620000000000003,"15min":6.5620000000000003},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":66.311999999999998,"5min":66.311999999999998,"15min":66.311999999999998},"last":0.85599999999999998},{"interface":"front","average":{"1min":13.874000000000001,"5min":13.874000000000001,"15min":13.874000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":159.47200000000001,"5min":159.47200000000001,"15min":159.47200000000001},"last":0.88700000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":13.903,"15min":13.903},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":159.43000000000001,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":1.1850000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":1.268},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":60.098999999999997,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.93100000000000005}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.0359999999999996,"5min":4.0359999999999996,"15min":4.0359999999999996},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":60.165999999999997,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.014},{"interface":"front","average":{"1min":18.09,"5min":18.09,"15min":18.09},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":159.512,"5min":159.512,"15min":159.512},"last":1.0369999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":886884,"kb_used_data":92813756,"kb_used_omap":759,"kb_used_meta":0,"kb_avail":92812496,"statfs":{"total":95948165120,"available":95039995904,"internally_reserved":1290240,"allocated":95041286144,"data_stored":95041286144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":777592,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":12,"commit_latency_ns":0,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.45200000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.5979999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":3.3559999999999999,"5min":3.3559999999999999,"15min":3.3559999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":47.518000000000001,"5min":47.518000000000001,"15min":47.518000000000001},"last":0.40899999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":10.401999999999999,"5min":10.401999999999999,"15min":10.401999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":183.53,"5min":183.53,"15min":183.53},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.51500000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.6499999999999999,"15min":3.6499999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":53.139000000000003,"5min":53.139000000000003,"15min":53.139000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":3.8090000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":53.186999999999998,"5min":53.186999999999998,"15min":53.186999999999998},"last":0.78900000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.77300000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":0.88800000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":37.124000000000002,"5min":37.124000000000002,"15min":37.124000000000002},"last":0.58399999999999996},{"interface":"front","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":30.445,"5min":30.445,"15min":30.445},"last":0.84399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":894384,"kb_used_data":92807396,"kb_used_omap":1008,"kb_used_meta":0,"kb_avail":92804996,"statfs":{"total":95948165120,"available":95032315904,"internally_reserved":2457600,"allocated":95034773504,"data_stored":95034773504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1032477,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":8,"commit_latency_ns":0,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":3.8929999999999998,"15min":3.8929999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":46.591000000000001,"5min":46.591000000000001,"15min":46.591000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":50.616,"5min":50.616,"15min":50.616},"last":0.53700000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":46.747999999999998,"5min":46.747999999999998,"15min":46.747999999999998},"last":0.93999999999999995}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":19.835999999999999,"5min":19.835999999999999,"15min":19.835999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":282.01400000000001,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":7.2450000000000001,"5min":7.2450000000000001,"15min":7.2450000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":53.145000000000003,"5min":53.145000000000003,"15min":53.145000000000003},"last":0.67700000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.7279999999999998,"5min":4.7279999999999998,"15min":4.7279999999999998},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":50.887999999999998,"5min":50.887999999999998,"15min":50.887999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":3.605,"5min":3.605,"15min":3.605},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":52.512,"5min":52.512,"15min":52.512},"last":0.98799999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":1.056},{"interface":"front","average":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":4.4379999999999997},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":46.905999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.1000000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.0019999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":50.844999999999999,"5min":50.844999999999999,"15min":50.844999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.873}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":52.595999999999997,"5min":52.595999999999997,"15min":52.595999999999997},"last":1.1539999999999999},{"interface":"front","average":{"1min":4.75,"5min":4.75,"15min":4.75},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":50.804000000000002,"5min":50.804000000000002,"15min":50.804000000000002},"last":0.96899999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":920792,"kb_used_data":92782376,"kb_used_omap":1091,"kb_used_meta":0,"kb_avail":92778588,"statfs":{"total":95948165120,"available":95005274112,"internally_reserved":3878912,"allocated":95009153024,"data_stored":95009153024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1117240,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.48799999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.28499999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":16.779,"5min":16.779,"15min":16.779},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.45700000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.85899999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":9.798,"5min":9.798,"15min":9.798},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":176.34399999999999,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":0.752}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.78400000000000003}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":15.670999999999999,"5min":15.670999999999999,"15min":15.670999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":292.61099999999999,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":0.96099999999999997}]}]}],"pool_statfs":[]}} 2023-01-15T14:29:39.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-01-15T14:29:39.820 INFO:tasks.rados.rados.0.smithi090.stdout:372: finishing copy_from to smithi09025264-46 2023-01-15T14:29:39.820 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 35 (ObjNum 83 snap 11 seq_num 83) dirty exists 2023-01-15T14:29:39.820 INFO:tasks.rados.rados.0.smithi090.stdout:372: finishing copy_from racing read to smithi09025264-46 2023-01-15T14:29:39.820 INFO:tasks.rados.rados.0.smithi090.stdout:379: expect (ObjNum 132 snap 24 seq_num 132) 2023-01-15T14:29:39.820 INFO:tasks.rados.rados.0.smithi090.stdout:372: done (5 left) 2023-01-15T14:29:39.820 INFO:tasks.rados.rados.0.smithi090.stdout:374: done (4 left) 2023-01-15T14:29:39.820 INFO:tasks.rados.rados.0.smithi090.stdout:376: done (3 left) 2023-01-15T14:29:39.820 INFO:tasks.rados.rados.0.smithi090.stdout:377: done (2 left) 2023-01-15T14:29:39.820 INFO:tasks.rados.rados.0.smithi090.stdout:379: done (1 left) 2023-01-15T14:29:39.821 INFO:tasks.rados.rados.0.smithi090.stdout:380: done (0 left) 2023-01-15T14:29:39.821 INFO:tasks.rados.rados.0.smithi090.stdout:381: rollback oid 36 current snap is 36 2023-01-15T14:29:39.821 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 36 to 30 2023-01-15T14:29:39.821 INFO:tasks.rados.rados.0.smithi090.stdout:382: rollback oid 12 current snap is 36 2023-01-15T14:29:39.821 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 12 to 35 2023-01-15T14:29:39.821 INFO:tasks.rados.rados.0.smithi090.stdout:383: write oid 34 current snap is 36 2023-01-15T14:29:39.821 INFO:tasks.rados.rados.0.smithi090.stdout:383: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2023-01-15T14:29:39.821 INFO:tasks.rados.rados.0.smithi090.stdout:383: writing smithi09025264-34 from 406769 to 1016765 tid 1 2023-01-15T14:29:39.821 INFO:tasks.rados.rados.0.smithi090.stdout:383: writing smithi09025264-34 from 1439434 to 1873148 tid 2 2023-01-15T14:29:39.821 INFO:tasks.rados.rados.0.smithi090.stdout:383: writing smithi09025264-34 from 2303522 to 2806769 tid 3 2023-01-15T14:29:39.821 INFO:tasks.rados.rados.0.smithi090.stdout:384: read oid 6 snap 35 2023-01-15T14:29:39.822 INFO:tasks.rados.rados.0.smithi090.stdout:384: expect (ObjNum 157 snap 34 seq_num 157) 2023-01-15T14:29:39.822 INFO:tasks.rados.rados.0.smithi090.stdout:385: copy_from oid 14 from oid 13 current snap is 36 2023-01-15T14:29:39.822 INFO:tasks.rados.rados.0.smithi090.stdout:382: finishing rollback tid 0 to smithi09025264-12 2023-01-15T14:29:39.822 INFO:tasks.rados.rados.0.smithi090.stdout:386: snap_remove snap 23 2023-01-15T14:29:39.823 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:39.812+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:39.920 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:39.913+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:40.009 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:40.009 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:40.010 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:29:40.021 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:40.016+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:40.119 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:40.409 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:40.410 INFO:tasks.rados.rados.0.smithi090.stdout:383: finishing write tid 1 to smithi09025264-34 2023-01-15T14:29:40.561 INFO:tasks.rados.rados.0.smithi090.stdout:386: done (5 left) 2023-01-15T14:29:40.561 INFO:tasks.rados.rados.0.smithi090.stdout:387: snap_create 2023-01-15T14:29:40.561 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:29:40.561 INFO:tasks.rados.rados.0.smithi090.stdout:381: finishing rollback tid 0 to smithi09025264-36 2023-01-15T14:29:40.561 INFO:tasks.rados.rados.0.smithi090.stdout:381: finishing rollback tid 1 to smithi09025264-36 2023-01-15T14:29:40.561 INFO:tasks.rados.rados.0.smithi090.stdout:385: finishing copy_from to smithi09025264-14 2023-01-15T14:29:40.561 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 108 (ObjNum 133 snap 24 seq_num 133) dirty exists 2023-01-15T14:29:40.561 INFO:tasks.rados.rados.0.smithi090.stdout:385: finishing copy_from racing read to smithi09025264-14 2023-01-15T14:29:40.561 INFO:tasks.rados.rados.0.smithi090.stdout:384: expect (ObjNum 157 snap 34 seq_num 157) 2023-01-15T14:29:40.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:29:40.563 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:40.563 INFO:teuthology.orchestra.run.smithi090.stdout:{"fsid":"1c77fb2b-d1a7-4884-a24e-3935e16d0b6b","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":102,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":94,"num_osds":8,"num_up_osds":8,"osd_up_since":1673792884,"num_in_osds":7,"osd_in_since":1673792933,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"remapped+peering","count":2},{"state_name":"active+clean+snaptrim","count":1}],"num_pgs":25,"num_pools":3,"num_objects":76,"data_bytes":156797518,"bytes_used":6379196416,"bytes_avail":665257959424,"bytes_total":671637155840,"inactive_pgs_ratio":0.079999998211860657,"read_bytes_sec":5851127,"write_bytes_sec":1872319,"read_op_per_sec":5,"write_op_per_sec":3},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2023-01-15T14:29:06.857309+0000","services":{}},"progress_events":{}} 2023-01-15T14:29:40.564 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2023-01-15T14:29:40.564 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:29:40.564 INFO:tasks.rados.rados.0.smithi090.stdout:381: finishing rollback tid 2 to smithi09025264-36 2023-01-15T14:29:40.564 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 121 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-01-15T14:29:40.564 INFO:tasks.rados.rados.0.smithi090.stdout:383: finishing write tid 2 to smithi09025264-34 2023-01-15T14:29:40.564 INFO:tasks.rados.rados.0.smithi090.stdout:383: finishing write tid 3 to smithi09025264-34 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:383: finishing write tid 4 to smithi09025264-34 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:383: finishing write tid 5 to smithi09025264-34 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:383: oid 34 updating version 0 to 84 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:383: oid 34 updating version 84 to 85 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:383: oid 34 version 85 is already newer than 82 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:383: oid 34 version 85 is already newer than 83 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 85 (ObjNum 167 snap 36 seq_num 167) dirty exists 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:383: left oid 34 (ObjNum 167 snap 36 seq_num 167) 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:381: done (4 left) 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:383: done (3 left) 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:384: done (2 left) 2023-01-15T14:29:40.565 INFO:tasks.rados.rados.0.smithi090.stdout:385: done (1 left) 2023-01-15T14:29:40.566 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:40.566 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:40.405+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:40.566 INFO:tasks.rados.rados.0.smithi090.stdout:382: finishing rollback tid 1 to smithi09025264-12 2023-01-15T14:29:40.566 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:29:40.607 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:40.601+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:40.739 INFO:tasks.rados.rados.0.smithi090.stdout:382: finishing rollback tid 2 to smithi09025264-12 2023-01-15T14:29:40.739 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 170 (ObjNum 128 snap 23 seq_num 128) dirty exists 2023-01-15T14:29:40.739 INFO:tasks.rados.rados.0.smithi090.stdout:382: done (0 left) 2023-01-15T14:29:40.740 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:40.740 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:40.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:29:40.741 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:40.733+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:40.839 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:40.833+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:40.844 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:40.844 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:40.854 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:29:40.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:29:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:40.918 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:29:40.918 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2023-01-15T14:29:40.867088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":158450488,"num_objects":77,"num_object_clones":32,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":7,"num_read":934,"num_read_kb":754906,"num_write":829,"num_write_kb":264916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":91528350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1467,"ondisk_log_size":1467,"up":75,"acting":73,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":74,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":6268212,"kb_used_data":649646564,"kb_used_omap":6841,"kb_used_meta":0,"kb_avail":649627448,"statfs":{"total":671637155840,"available":665218506752,"internally_reserved":19574784,"allocated":665238081536,"data_stored":665238081536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7006013,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":59,"commit_latency_ns":0,"apply_latency_ns":59000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1652970,"num_objects":1,"num_object_clones":2,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":-1,"num_read":46,"num_read_kb":47759,"num_write":27,"num_write_kb":8696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001110"},"pg_stats":[{"pgid":"3.f","version":"86'72","reported_seq":288,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-01-15T14:29:39.112222+0000","last_change":"2023-01-15T14:29:39.113673+0000","last_active":"2023-01-15T14:29:39.112222+0000","last_peered":"2023-01-15T14:29:39.112222+0000","last_clean":"2023-01-15T14:29:39.112222+0000","last_became_active":"2023-01-15T14:28:23.846063+0000","last_became_peered":"2023-01-15T14:28:23.846063+0000","last_unstale":"2023-01-15T14:29:39.112222+0000","last_undegraded":"2023-01-15T14:29:39.112222+0000","last_fullsized":"2023-01-15T14:29:39.112222+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:52.368","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016951729999999999,"stat_sum":{"num_bytes":14524782,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":37,"num_read_kb":27958,"num_write":43,"num_write_kb":13905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"3.e","version":"73'25","reported_seq":241,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-01-15T14:29:39.212191+0000","last_change":"2023-01-15T14:29:39.212757+0000","last_active":"2023-01-15T14:29:39.212191+0000","last_peered":"2023-01-15T14:29:39.212191+0000","last_clean":"2023-01-15T14:29:39.212191+0000","last_became_active":"2023-01-15T14:28:23.846858+0000","last_became_peered":"2023-01-15T14:28:23.846858+0000","last_unstale":"2023-01-15T14:29:39.212191+0000","last_undegraded":"2023-01-15T14:29:39.212191+0000","last_fullsized":"2023-01-15T14:29:39.212191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:44.514","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084241700000000004,"stat_sum":{"num_bytes":2621859,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":12810,"num_write":6,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"3.d","version":"93'105","reported_seq":357,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-01-15T14:29:39.717394+0000","last_change":"2023-01-15T14:29:39.718046+0000","last_active":"2023-01-15T14:29:39.717394+0000","last_peered":"2023-01-15T14:29:39.717394+0000","last_clean":"2023-01-15T14:29:39.717394+0000","last_became_active":"2023-01-15T14:28:54.932913+0000","last_became_peered":"2023-01-15T14:28:54.932913+0000","last_unstale":"2023-01-15T14:29:39.717394+0000","last_undegraded":"2023-01-15T14:29:39.717394+0000","last_fullsized":"2023-01-15T14:29:39.717394+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.60621071000000004,"stat_sum":{"num_bytes":13776031,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":87025,"num_write":47,"num_write_kb":18091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"3.c","version":"94'45","reported_seq":205,"reported_epoch":94,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:40.122511+0000","last_change":"2023-01-15T14:29:40.122511+0000","last_active":"2023-01-15T14:29:40.122511+0000","last_peered":"2023-01-15T14:29:40.122511+0000","last_clean":"2023-01-15T14:29:40.122511+0000","last_became_active":"2023-01-15T14:28:54.833568+0000","last_became_peered":"2023-01-15T14:28:54.833568+0000","last_unstale":"2023-01-15T14:29:40.122511+0000","last_undegraded":"2023-01-15T14:29:40.122511+0000","last_fullsized":"2023-01-15T14:29:40.122511+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.60960574099999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":20,"num_write_kb":5044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2319333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"3.b","version":"86'30","reported_seq":247,"reported_epoch":94,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:40.221680+0000","last_change":"2023-01-15T14:29:40.221680+0000","last_active":"2023-01-15T14:29:40.221680+0000","last_peered":"2023-01-15T14:29:40.221680+0000","last_clean":"2023-01-15T14:29:40.221680+0000","last_became_active":"2023-01-15T14:28:23.934183+0000","last_became_peered":"2023-01-15T14:28:23.934183+0000","last_unstale":"2023-01-15T14:29:40.221680+0000","last_undegraded":"2023-01-15T14:29:40.221680+0000","last_fullsized":"2023-01-15T14:29:40.221680+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:45.586","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51125118899999999,"stat_sum":{"num_bytes":2218524,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":15,"num_read_kb":13386,"num_write":12,"num_write_kb":5195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"3.a","version":"81'121","reported_seq":341,"reported_epoch":90,"state":"remapped+peering","last_fresh":"2023-01-15T14:29:36.096806+0000","last_change":"2023-01-15T14:29:36.096806+0000","last_active":"2023-01-15T14:29:36.096527+0000","last_peered":"2023-01-15T14:29:27.218124+0000","last_clean":"2023-01-15T14:29:27.218124+0000","last_became_active":"2023-01-15T14:29:02.112091+0000","last_became_peered":"2023-01-15T14:29:02.112091+0000","last_unstale":"2023-01-15T14:29:36.096806+0000","last_undegraded":"2023-01-15T14:29:36.096806+0000","last_fullsized":"2023-01-15T14:29:36.096806+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041957749999999997,"stat_sum":{"num_bytes":14852985,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":42,"num_read_kb":24511,"num_write":64,"num_write_kb":31038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":21341200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"3.9","version":"84'108","reported_seq":326,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.293416+0000","last_change":"2023-01-15T14:29:35.192159+0000","last_active":"2023-01-15T14:29:35.293416+0000","last_peered":"2023-01-15T14:29:35.293416+0000","last_clean":"2023-01-15T14:29:35.293416+0000","last_became_active":"2023-01-15T14:28:24.109843+0000","last_became_peered":"2023-01-15T14:28:24.109843+0000","last_unstale":"2023-01-15T14:29:35.293416+0000","last_undegraded":"2023-01-15T14:29:35.293416+0000","last_fullsized":"2023-01-15T14:29:35.293416+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091506300000000003,"stat_sum":{"num_bytes":10773188,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":70457,"num_write":48,"num_write_kb":20836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"3.8","version":"94'135","reported_seq":396,"reported_epoch":94,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:40.122433+0000","last_change":"2023-01-15T14:29:40.122433+0000","last_active":"2023-01-15T14:29:40.122433+0000","last_peered":"2023-01-15T14:29:40.122433+0000","last_clean":"2023-01-15T14:29:40.122433+0000","last_became_active":"2023-01-15T14:28:47.937231+0000","last_became_peered":"2023-01-15T14:28:47.937231+0000","last_unstale":"2023-01-15T14:29:40.122433+0000","last_undegraded":"2023-01-15T14:29:40.122433+0000","last_fullsized":"2023-01-15T14:29:40.122433+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'121","last_scrub_stamp":"2023-01-15T14:29:31.589753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.589753+0000","objects_scrubbed":8,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:54.442","scrub_duration":0.28338864200000002,"objects_trimmed":3,"snaptrim_duration":0.10082651500000001,"stat_sum":{"num_bytes":9610172,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":75,"num_read_kb":73607,"num_write":63,"num_write_kb":20884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"1.0","version":"19'117","reported_seq":289,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-01-15T14:29:39.317908+0000","last_change":"2023-01-15T14:29:39.317908+0000","last_active":"2023-01-15T14:29:39.317908+0000","last_peered":"2023-01-15T14:29:39.317908+0000","last_clean":"2023-01-15T14:29:39.317908+0000","last_became_active":"2023-01-15T14:29:39.317067+0000","last_became_peered":"2023-01-15T14:29:39.317067+0000","last_unstale":"2023-01-15T14:29:39.317908+0000","last_undegraded":"2023-01-15T14:29:39.317908+0000","last_fullsized":"2023-01-15T14:29:39.317908+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-01-15T14:29:34.186384+0000","last_change":"2023-01-15T14:28:54.829192+0000","last_active":"2023-01-15T14:29:34.186384+0000","last_peered":"2023-01-15T14:29:34.186384+0000","last_clean":"2023-01-15T14:29:34.186384+0000","last_became_active":"2023-01-15T14:28:54.828857+0000","last_became_peered":"2023-01-15T14:28:54.828857+0000","last_unstale":"2023-01-15T14:29:34.186384+0000","last_undegraded":"2023-01-15T14:29:34.186384+0000","last_fullsized":"2023-01-15T14:29:34.186384+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.638","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"82'74","reported_seq":327,"reported_epoch":94,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:40.124512+0000","last_change":"2023-01-15T14:29:40.124512+0000","last_active":"2023-01-15T14:29:40.124512+0000","last_peered":"2023-01-15T14:29:40.124512+0000","last_clean":"2023-01-15T14:29:40.124512+0000","last_became_active":"2023-01-15T14:28:55.217912+0000","last_became_peered":"2023-01-15T14:28:55.217912+0000","last_unstale":"2023-01-15T14:29:40.124512+0000","last_undegraded":"2023-01-15T14:29:40.124512+0000","last_fullsized":"2023-01-15T14:29:40.124512+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.256","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000535944,"stat_sum":{"num_bytes":11579957,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":59,"num_read_kb":56238,"num_write":40,"num_write_kb":17290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10821306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":94,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-01-15T14:29:38.299501+0000","last_change":"2023-01-15T14:29:31.583684+0000","last_active":"2023-01-15T14:29:38.299501+0000","last_peered":"2023-01-15T14:29:38.299501+0000","last_clean":"2023-01-15T14:29:38.299501+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:29:38.299501+0000","last_undegraded":"2023-01-15T14:29:38.299501+0000","last_fullsized":"2023-01-15T14:29:38.299501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:31.583599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.583599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:44.691","scrub_duration":0.22681161699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"94'90","reported_seq":331,"reported_epoch":94,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:40.118892+0000","last_change":"2023-01-15T14:29:40.118892+0000","last_active":"2023-01-15T14:29:40.118892+0000","last_peered":"2023-01-15T14:29:40.118892+0000","last_clean":"2023-01-15T14:29:40.118892+0000","last_became_active":"2023-01-15T14:28:23.842545+0000","last_became_peered":"2023-01-15T14:28:23.842545+0000","last_unstale":"2023-01-15T14:29:40.118892+0000","last_undegraded":"2023-01-15T14:29:40.118892+0000","last_fullsized":"2023-01-15T14:29:40.118892+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'81","last_scrub_stamp":"2023-01-15T14:29:32.795334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:32.795334+0000","objects_scrubbed":4,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:33.824","scrub_duration":0.40728601599999997,"objects_trimmed":1,"snaptrim_duration":0.50609028599999994,"stat_sum":{"num_bytes":12510745,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":56,"num_read_kb":49747,"num_write":48,"num_write_kb":16252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-01-15T14:29:34.186332+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:29:34.186332+0000","last_peered":"2023-01-15T14:29:34.186332+0000","last_clean":"2023-01-15T14:29:34.186332+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:29:34.186332+0000","last_undegraded":"2023-01-15T14:29:34.186332+0000","last_fullsized":"2023-01-15T14:29:34.186332+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:37.689","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"85'165","reported_seq":425,"reported_epoch":90,"state":"remapped+peering","last_fresh":"2023-01-15T14:29:36.196307+0000","last_change":"2023-01-15T14:29:36.196638+0000","last_active":"2023-01-15T14:29:35.090958+0000","last_peered":"2023-01-15T14:29:34.622002+0000","last_clean":"2023-01-15T14:29:34.622002+0000","last_became_active":"2023-01-15T14:29:02.007426+0000","last_became_peered":"2023-01-15T14:29:02.007426+0000","last_unstale":"2023-01-15T14:29:36.196307+0000","last_undegraded":"2023-01-15T14:29:36.196307+0000","last_fullsized":"2023-01-15T14:29:36.196307+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.80652508000000001,"stat_sum":{"num_bytes":15395080,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":99,"num_read_kb":95371,"num_write":70,"num_write_kb":37255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":16722767,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,6],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-01-15T14:29:34.186492+0000","last_change":"2023-01-15T14:29:14.703116+0000","last_active":"2023-01-15T14:29:34.186492+0000","last_peered":"2023-01-15T14:29:34.186492+0000","last_clean":"2023-01-15T14:29:34.186492+0000","last_became_active":"2023-01-15T14:29:14.050302+0000","last_became_peered":"2023-01-15T14:29:14.050302+0000","last_unstale":"2023-01-15T14:29:34.186492+0000","last_undegraded":"2023-01-15T14:29:34.186492+0000","last_fullsized":"2023-01-15T14:29:34.186492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:14.703035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:14.703035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:18.827","scrub_duration":0.44938580500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"89'167","reported_seq":407,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.402467+0000","last_change":"2023-01-15T14:29:35.192676+0000","last_active":"2023-01-15T14:29:35.402467+0000","last_peered":"2023-01-15T14:29:35.402467+0000","last_clean":"2023-01-15T14:29:35.402467+0000","last_became_active":"2023-01-15T14:28:23.946074+0000","last_became_peered":"2023-01-15T14:28:23.946074+0000","last_unstale":"2023-01-15T14:29:35.402467+0000","last_undegraded":"2023-01-15T14:29:35.402467+0000","last_fullsized":"2023-01-15T14:29:35.402467+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:40.271","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012671379999999999,"stat_sum":{"num_bytes":22096394,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":134,"num_read_kb":129215,"num_write":97,"num_write_kb":27695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"3.5","version":"93'37","reported_seq":240,"reported_epoch":94,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:40.121606+0000","last_change":"2023-01-15T14:29:40.121606+0000","last_active":"2023-01-15T14:29:40.121606+0000","last_peered":"2023-01-15T14:29:40.121606+0000","last_clean":"2023-01-15T14:29:40.121606+0000","last_became_active":"2023-01-15T14:28:54.831872+0000","last_became_peered":"2023-01-15T14:28:54.831872+0000","last_unstale":"2023-01-15T14:29:40.121606+0000","last_undegraded":"2023-01-15T14:29:40.121606+0000","last_fullsized":"2023-01-15T14:29:40.121606+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:46.735","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.60944599899999996,"stat_sum":{"num_bytes":9593925,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":25834,"num_write":17,"num_write_kb":11085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4252105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":91,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.088856+0000","last_change":"2023-01-15T14:29:29.346425+0000","last_active":"2023-01-15T14:29:35.088856+0000","last_peered":"2023-01-15T14:29:35.088856+0000","last_clean":"2023-01-15T14:29:35.088856+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:29:35.088856+0000","last_undegraded":"2023-01-15T14:29:35.088856+0000","last_fullsized":"2023-01-15T14:29:35.088856+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:52.809","scrub_duration":0.39286881699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"89'78","reported_seq":332,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.699506+0000","last_change":"2023-01-15T14:29:35.699506+0000","last_active":"2023-01-15T14:29:35.699506+0000","last_peered":"2023-01-15T14:29:35.699506+0000","last_clean":"2023-01-15T14:29:35.699506+0000","last_became_active":"2023-01-15T14:28:54.825996+0000","last_became_peered":"2023-01-15T14:28:54.825996+0000","last_unstale":"2023-01-15T14:29:35.699506+0000","last_undegraded":"2023-01-15T14:29:35.699506+0000","last_fullsized":"2023-01-15T14:29:35.699506+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.440","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.61088218500000002,"stat_sum":{"num_bytes":11853694,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":80,"num_read_kb":81292,"num_write":47,"num_write_kb":14300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12100908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":94,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-01-15T14:29:38.099609+0000","last_change":"2023-01-15T14:29:25.664703+0000","last_active":"2023-01-15T14:29:38.099609+0000","last_peered":"2023-01-15T14:29:38.099609+0000","last_clean":"2023-01-15T14:29:38.099609+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:29:38.099609+0000","last_undegraded":"2023-01-15T14:29:38.099609+0000","last_fullsized":"2023-01-15T14:29:38.099609+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:25.664616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:25.664616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:29.565","scrub_duration":0.40398350300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"93'54","reported_seq":266,"reported_epoch":94,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:40.121900+0000","last_change":"2023-01-15T14:29:40.121900+0000","last_active":"2023-01-15T14:29:40.121900+0000","last_peered":"2023-01-15T14:29:40.121900+0000","last_clean":"2023-01-15T14:29:40.121900+0000","last_became_active":"2023-01-15T14:28:23.741266+0000","last_became_peered":"2023-01-15T14:28:23.741266+0000","last_unstale":"2023-01-15T14:29:40.121900+0000","last_undegraded":"2023-01-15T14:29:40.121900+0000","last_fullsized":"2023-01-15T14:29:40.121900+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70310195200000003,"stat_sum":{"num_bytes":2621859,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":4,"num_write":22,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":101,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-01-15T14:29:36.100082+0000","last_change":"2023-01-15T14:29:36.100082+0000","last_active":"2023-01-15T14:29:36.100082+0000","last_peered":"2023-01-15T14:29:36.100082+0000","last_clean":"2023-01-15T14:29:36.100082+0000","last_became_active":"2023-01-15T14:29:36.099234+0000","last_became_peered":"2023-01-15T14:29:36.099234+0000","last_unstale":"2023-01-15T14:29:36.100082+0000","last_undegraded":"2023-01-15T14:29:36.100082+0000","last_fullsized":"2023-01-15T14:29:36.100082+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"94'44","reported_seq":262,"reported_epoch":94,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:40.118962+0000","last_change":"2023-01-15T14:29:40.118962+0000","last_active":"2023-01-15T14:29:40.118962+0000","last_peered":"2023-01-15T14:29:40.118962+0000","last_clean":"2023-01-15T14:29:40.118962+0000","last_became_active":"2023-01-15T14:28:23.841189+0000","last_became_peered":"2023-01-15T14:28:23.841189+0000","last_unstale":"2023-01-15T14:29:40.118962+0000","last_undegraded":"2023-01-15T14:29:40.118962+0000","last_fullsized":"2023-01-15T14:29:40.118962+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:29:41.507","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.016917674000000001,"stat_sum":{"num_bytes":3830925,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":7348,"num_write":26,"num_write_kb":9497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-01-15T14:29:35.095204+0000","last_change":"2023-01-15T14:28:12.896363+0000","last_active":"2023-01-15T14:29:35.095204+0000","last_peered":"2023-01-15T14:29:35.095204+0000","last_clean":"2023-01-15T14:29:35.095204+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:29:35.095204+0000","last_undegraded":"2023-01-15T14:29:35.095204+0000","last_fullsized":"2023-01-15T14:29:35.095204+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.332","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":157860120,"num_objects":75,"num_object_clones":32,"num_object_copies":225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":7,"num_read":818,"num_read_kb":754806,"num_write":670,"num_write_kb":262500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":90347614,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1350,"ondisk_log_size":1350,"up":48,"acting":46,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":866184,"kb_used_data":92834724,"kb_used_omap":786,"kb_used_meta":0,"kb_avail":92833196,"statfs":{"total":95948165120,"available":95061192704,"internally_reserved":1564672,"allocated":95062757376,"data_stored":95062757376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":805325,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":7,"commit_latency_ns":0,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.0700000000000003,"5min":4.0700000000000003,"15min":4.0700000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":75.893000000000001,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":9.2089999999999996,"5min":9.2089999999999996,"15min":9.2089999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":71.308999999999997,"5min":71.308999999999997,"15min":71.308999999999997},"last":0.621}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":75.959999999999994,"5min":75.959999999999994,"15min":75.959999999999994},"last":0.74099999999999999},{"interface":"front","average":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":6.2800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":68.605000000000004,"5min":68.605000000000004,"15min":68.605000000000004},"last":0.71999999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.7910000000000004,"5min":4.7910000000000004,"15min":4.7910000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":94.733999999999995,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.997},{"interface":"front","average":{"1min":4.1779999999999999,"5min":4.1779999999999999,"15min":4.1779999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":75.998999999999995},"last":1.0469999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.7770000000000001,"5min":3.7770000000000001,"15min":3.7770000000000001},"last":1.135},{"interface":"front","average":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":76.034999999999997,"5min":76.034999999999997,"15min":76.034999999999997},"last":1.1040000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":9.4160000000000004,"5min":9.4160000000000004,"15min":9.4160000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":76.658000000000001,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":3.5129999999999999,"5min":3.5129999999999999,"15min":3.5129999999999999},"last":1.0800000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.327,"5min":6.327,"15min":6.327},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":68.679000000000002,"5min":68.679000000000002,"15min":68.679000000000002},"last":0.92500000000000004},{"interface":"front","average":{"1min":4.9450000000000003,"5min":4.9450000000000003,"15min":4.9450000000000003},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":95.430000000000007,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.1919999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":8.5239999999999991,"5min":8.5239999999999991,"15min":8.5239999999999991},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":68.635999999999996,"5min":68.635999999999996,"15min":68.635999999999996},"last":1.024},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.6520000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":1.4470000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":7,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":870888,"kb_used_data":92832928,"kb_used_omap":936,"kb_used_meta":0,"kb_avail":92828492,"statfs":{"total":95948165120,"available":95056375808,"internally_reserved":4542464,"allocated":95060918272,"data_stored":95060918272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":958985,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.98499999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.67800000000000005}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.92800000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.0590000000000002,"5min":6.0590000000000002,"15min":6.0590000000000002},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":81.686999999999998,"5min":81.686999999999998,"15min":81.686999999999998},"last":0.78800000000000003},{"interface":"front","average":{"1min":11.455,"5min":11.455,"15min":11.455},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":255.24299999999999,"5min":255.24299999999999,"15min":255.24299999999999},"last":0.94999999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":6.6680000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":138.197,"5min":138.197,"15min":138.197},"last":0.75700000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.872},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.58299999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.2800000000000002,"5min":4.2800000000000002,"15min":4.2800000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":78.888000000000005,"5min":78.888000000000005,"15min":78.888000000000005},"last":0.84899999999999998},{"interface":"front","average":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":47.332000000000001,"5min":47.332000000000001,"15min":47.332000000000001},"last":0.83299999999999996}]}]},{"osd":5,"up_from":0,"seq":55834574869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":917832,"kb_used_data":92784196,"kb_used_omap":1111,"kb_used_meta":0,"kb_avail":92781548,"statfs":{"total":95948165120,"available":95008305152,"internally_reserved":2711552,"allocated":95011016704,"data_stored":95011016704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1138509,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.915,"5min":4.915,"15min":4.915},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":86.944999999999993,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.92400000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":8.5939999999999994,"5min":8.5939999999999994,"15min":8.5939999999999994},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":99.960999999999999,"5min":99.960999999999999,"15min":99.960999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":1.0069999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":99.884,"5min":99.884,"15min":99.884},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1.044}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":12.1,"5min":12.1,"15min":12.1},"last":0.75},{"interface":"front","average":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":6.1509999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":99.921000000000006,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.76800000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":30.971,"5min":30.971,"15min":30.971},"last":0.80800000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.673,"5min":5.673,"15min":5.673},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":99.780000000000001,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":5.1630000000000003,"5min":5.1630000000000003,"15min":5.1630000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":88.186999999999998,"5min":88.186999999999998,"15min":88.186999999999998},"last":1.081}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.128,"5min":13.128,"15min":13.128},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":179.73500000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":11.239000000000001,"5min":11.239000000000001,"15min":11.239000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":99.843000000000004,"5min":99.843000000000004,"15min":99.843000000000004},"last":0.97799999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":897856,"kb_used_data":92803832,"kb_used_omap":1112,"kb_used_meta":0,"kb_avail":92801524,"statfs":{"total":95948165120,"available":95028760576,"internally_reserved":2363392,"allocated":95031123968,"data_stored":95031123968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1138756,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.8479999999999999,"5min":4.8479999999999999,"15min":4.8479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":93.632999999999996,"5min":93.632999999999996,"15min":93.632999999999996},"last":0.66100000000000003},{"interface":"front","average":{"1min":7.6420000000000003,"5min":7.6420000000000003,"15min":7.6420000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":93.674000000000007,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.73599999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.532,"5min":13.532,"15min":13.532},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":159.256,"5min":159.256,"15min":159.256},"last":0.63},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.59499999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.3490000000000002,"5min":3.3490000000000002,"15min":3.3490000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":59.896999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":31.233000000000001,"5min":31.233000000000001,"15min":31.233000000000001},"last":0.81299999999999994}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":6.5620000000000003,"5min":6.5620000000000003,"15min":6.5620000000000003},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":66.311999999999998,"5min":66.311999999999998,"15min":66.311999999999998},"last":0.79700000000000004},{"interface":"front","average":{"1min":13.874000000000001,"5min":13.874000000000001,"15min":13.874000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":159.47200000000001,"5min":159.47200000000001,"15min":159.47200000000001},"last":0.86699999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":13.903,"15min":13.903},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":159.43000000000001,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.99299999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":60.098999999999997,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.93300000000000005}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.0359999999999996,"5min":4.0359999999999996,"15min":4.0359999999999996},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":60.165999999999997,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.90100000000000002},{"interface":"front","average":{"1min":18.09,"5min":18.09,"15min":18.09},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":159.512,"5min":159.512,"15min":159.512},"last":1.034}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":886884,"kb_used_data":92813756,"kb_used_omap":759,"kb_used_meta":0,"kb_avail":92812496,"statfs":{"total":95948165120,"available":95039995904,"internally_reserved":1290240,"allocated":95041286144,"data_stored":95041286144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":777592,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":12,"commit_latency_ns":0,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.45200000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.5979999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":3.3559999999999999,"5min":3.3559999999999999,"15min":3.3559999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":47.518000000000001,"5min":47.518000000000001,"15min":47.518000000000001},"last":0.40899999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":10.401999999999999,"5min":10.401999999999999,"15min":10.401999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":183.53,"5min":183.53,"15min":183.53},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.51500000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.6499999999999999,"15min":3.6499999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":53.139000000000003,"5min":53.139000000000003,"15min":53.139000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":3.8090000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":53.186999999999998,"5min":53.186999999999998,"15min":53.186999999999998},"last":0.78900000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.77300000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":0.88800000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":37.124000000000002,"5min":37.124000000000002,"15min":37.124000000000002},"last":0.58399999999999996},{"interface":"front","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":30.445,"5min":30.445,"15min":30.445},"last":0.84399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":894384,"kb_used_data":92807396,"kb_used_omap":1008,"kb_used_meta":0,"kb_avail":92804996,"statfs":{"total":95948165120,"available":95032315904,"internally_reserved":2457600,"allocated":95034773504,"data_stored":95034773504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1032477,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":8,"commit_latency_ns":0,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":3.8929999999999998,"15min":3.8929999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":46.591000000000001,"5min":46.591000000000001,"15min":46.591000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":50.616,"5min":50.616,"15min":50.616},"last":0.53700000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":46.747999999999998,"5min":46.747999999999998,"15min":46.747999999999998},"last":0.93999999999999995}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":19.835999999999999,"5min":19.835999999999999,"15min":19.835999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":282.01400000000001,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":7.2450000000000001,"5min":7.2450000000000001,"15min":7.2450000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":53.145000000000003,"5min":53.145000000000003,"15min":53.145000000000003},"last":0.67700000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.7279999999999998,"5min":4.7279999999999998,"15min":4.7279999999999998},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":50.887999999999998,"5min":50.887999999999998,"15min":50.887999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":3.605,"5min":3.605,"15min":3.605},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":52.512,"5min":52.512,"15min":52.512},"last":0.98799999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":1.056},{"interface":"front","average":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":4.4379999999999997},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":46.905999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.1000000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.0019999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":50.844999999999999,"5min":50.844999999999999,"15min":50.844999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.873}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":52.595999999999997,"5min":52.595999999999997,"15min":52.595999999999997},"last":1.1539999999999999},{"interface":"front","average":{"1min":4.75,"5min":4.75,"15min":4.75},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":50.804000000000002,"5min":50.804000000000002,"15min":50.804000000000002},"last":0.96899999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":934184,"kb_used_data":92769732,"kb_used_omap":1127,"kb_used_meta":0,"kb_avail":92765196,"statfs":{"total":95948165120,"available":94991560704,"internally_reserved":4644864,"allocated":94996205568,"data_stored":94996205568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1154369,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.39000000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.41199999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":16.779,"5min":16.779,"15min":16.779},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.45700000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":0.88900000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.70899999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":9.798,"5min":9.798,"15min":9.798},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":176.34399999999999,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":0.85899999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.90400000000000003}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":15.670999999999999,"5min":15.670999999999999,"15min":15.670999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":292.61099999999999,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":0.73399999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:29:40.977 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:40.972+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:41.097 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:41.097 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:41.098 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:29:41.100 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:41.092+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:41.110 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:41.133 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:41.138 INFO:tasks.rados.rados.0.smithi090.stdout:387: done (0 left) 2023-01-15T14:29:41.138 INFO:tasks.rados.rados.0.smithi090.stdout:388: write oid 49 current snap is 37 2023-01-15T14:29:41.138 INFO:tasks.rados.rados.0.smithi090.stdout:388: seq_num 168 ranges {423576=401598,1497916=693725} 2023-01-15T14:29:41.142 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:29:41.143 INFO:tasks.rados.rados.0.smithi090.stdout:388: writing smithi09025264-49 from 423576 to 825174 tid 1 2023-01-15T14:29:41.152 INFO:tasks.rados.rados.0.smithi090.stdout:388: writing smithi09025264-49 from 1497916 to 2191641 tid 2 2023-01-15T14:29:41.152 INFO:tasks.rados.rados.0.smithi090.stdout:389: delete oid 18 current snap is 37 2023-01-15T14:29:41.154 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:41.198 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:41.192+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:41.272 INFO:tasks.rados.rados.0.smithi090.stdout:388: finishing write tid 1 to smithi09025264-49 2023-01-15T14:29:41.272 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:41.273 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:41.273 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:29:41.281 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:41.299 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:41.293+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:41.304 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:41.326 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:41.433 INFO:tasks.rados.rados.0.smithi090.stdout:388: finishing write tid 2 to smithi09025264-49 2023-01-15T14:29:41.434 INFO:tasks.rados.rados.0.smithi090.stdout:389: done (1 left) 2023-01-15T14:29:41.434 INFO:tasks.rados.rados.0.smithi090.stdout:390: snap_remove snap 35 2023-01-15T14:29:41.435 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:41.436 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:29:41.436 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:29:41.436 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:29:41.436 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.436 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2 3.3 3:ca59e2b0:::smithi09025264-1:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:41.436 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.774074+0000", 2023-01-15T14:29:41.436 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 75.612215484999993, 2023-01-15T14:29:41.436 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.52953906799999995, 2023-01-15T14:29:41.436 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:41.436 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:41.436 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:41.437 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:41.437 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:41.466 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2 2023-01-15T14:29:41.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.467 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:41.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:41.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.774074+0000", 2023-01-15T14:29:41.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:41.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.774074+0000", 2023-01-15T14:29:41.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:41.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.774072+0000", 2023-01-15T14:29:41.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:41.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:41.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775063+0000", 2023-01-15T14:29:41.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00099123799999999997 2023-01-15T14:29:41.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:41.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775098+0000", 2023-01-15T14:29:41.471 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4379999999999999e-05 2023-01-15T14:29:41.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:41.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775113+0000", 2023-01-15T14:29:41.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5099999999999999e-05 2023-01-15T14:29:41.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.472 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.857646+0000", 2023-01-15T14:29:41.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.082533580999999995 2023-01-15T14:29:41.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:41.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.857869+0000", 2023-01-15T14:29:41.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022288000000000001 2023-01-15T14:29:41.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103036+0000", 2023-01-15T14:29:41.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24516706799999999 2023-01-15T14:29:41.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:41.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103244+0000", 2023-01-15T14:29:41.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020708799999999999 2023-01-15T14:29:41.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:29:41.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103565+0000", 2023-01-15T14:29:41.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032126799999999999 2023-01-15T14:29:41.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:41.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103895+0000", 2023-01-15T14:29:41.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033054500000000001 2023-01-15T14:29:41.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:41.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.103917+0000", 2023-01-15T14:29:41.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1416e-05 2023-01-15T14:29:41.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.104198+0000", 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028077199999999999 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.104952+0000", 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075427100000000004 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303317+0000", 2023-01-15T14:29:41.478 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.198365083 2023-01-15T14:29:41.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:41.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303455+0000", 2023-01-15T14:29:41.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013782900000000001 2023-01-15T14:29:41.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:41.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303547+0000", 2023-01-15T14:29:41.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.1879999999999994e-05 2023-01-15T14:29:41.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:41.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303613+0000", 2023-01-15T14:29:41.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6766000000000004e-05 2023-01-15T14:29:41.480 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.480 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:41.480 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.481 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3 3.3 3:ca59e2b0:::smithi09025264-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:41.481 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 75.611157351000003, 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62968594499999997, 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:41.483 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:29:41.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:41.608 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:29:41.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.609 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.609 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:41.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775131+0000", 2023-01-15T14:29:41.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:41.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:41.610 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775138+0000", 2023-01-15T14:29:41.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:29:41.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:41.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775148+0000", 2023-01-15T14:29:41.612 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7969999999999995e-06 2023-01-15T14:29:41.612 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.612 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:41.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775161+0000", 2023-01-15T14:29:41.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2469999999999999e-05 2023-01-15T14:29:41.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.614 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.857974+0000", 2023-01-15T14:29:41.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.082812887000000002 2023-01-15T14:29:41.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.615 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:41.615 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.858147+0000", 2023-01-15T14:29:41.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000173883 2023-01-15T14:29:41.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106098+0000", 2023-01-15T14:29:41.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24795019600000001 2023-01-15T14:29:41.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:41.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106181+0000", 2023-01-15T14:29:41.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3248e-05 2023-01-15T14:29:41.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:29:41.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106330+0000", 2023-01-15T14:29:41.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014918200000000001 2023-01-15T14:29:41.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:41.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106510+0000", 2023-01-15T14:29:41.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017979399999999999 2023-01-15T14:29:41.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:41.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106527+0000", 2023-01-15T14:29:41.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6643999999999999e-05 2023-01-15T14:29:41.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:41.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106605+0000", 2023-01-15T14:29:41.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8650000000000001e-05 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106644+0000", 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8686999999999997e-05 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404687+0000", 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.298042745 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:41.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404735+0000", 2023-01-15T14:29:41.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8306000000000001e-05 2023-01-15T14:29:41.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:41.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404772+0000", 2023-01-15T14:29:41.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6977000000000002e-05 2023-01-15T14:29:41.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404818+0000", 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.6439e-05 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 75.611119283999997, 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62995383500000002, 2023-01-15T14:29:41.629 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:41.630 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:41.630 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:41.630 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:41.630 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:41.630 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4 2023-01-15T14:29:41.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.630 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:41.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:41.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:29:41.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775174+0000", 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3669999999999999e-06 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775184+0000", 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5869999999999999e-06 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775190+0000", 2023-01-15T14:29:41.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:29:41.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868440+0000", 2023-01-15T14:29:41.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.093250126000000003 2023-01-15T14:29:41.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:41.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868555+0000", 2023-01-15T14:29:41.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115003 2023-01-15T14:29:41.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.302962+0000", 2023-01-15T14:29:41.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.43440729700000003 2023-01-15T14:29:41.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:41.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303197+0000", 2023-01-15T14:29:41.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023508000000000001 2023-01-15T14:29:41.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404866+0000", 2023-01-15T14:29:41.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101668723 2023-01-15T14:29:41.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:41.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404985+0000", 2023-01-15T14:29:41.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118853 2023-01-15T14:29:41.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:41.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.405124+0000", 2023-01-15T14:29:41.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013951999999999999 2023-01-15T14:29:41.649 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.649 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:41.649 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.650 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.650 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:42 3.3 3:c11c7d81:::smithi09025264-11:head [write 1552114~615956] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:41.650 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.898738+0000", 2023-01-15T14:29:41.650 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 75.487552217000001, 2023-01-15T14:29:41.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50532295100000002, 2023-01-15T14:29:41.650 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:41.650 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:41.650 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:41.650 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:41.650 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:41.650 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 42 2023-01-15T14:29:41.651 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.651 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:41.651 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.651 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:41.651 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.898738+0000", 2023-01-15T14:29:41.651 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.651 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.651 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.651 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:41.651 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.898738+0000", 2023-01-15T14:29:41.651 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.652 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.652 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.652 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:41.652 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.898720+0000", 2023-01-15T14:29:41.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999824 2023-01-15T14:29:41.652 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.652 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.652 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:41.652 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.899828+0000", 2023-01-15T14:29:41.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011084269999999999 2023-01-15T14:29:41.652 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.899855+0000", 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7237e-05 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.999992+0000", 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100136528 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.000100+0000", 2023-01-15T14:29:41.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107812 2023-01-15T14:29:41.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:41.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.000268+0000", 2023-01-15T14:29:41.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016832800000000001 2023-01-15T14:29:41.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.104202+0000", 2023-01-15T14:29:41.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10393384999999999 2023-01-15T14:29:41.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:41.655 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.104306+0000", 2023-01-15T14:29:41.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010444499999999999 2023-01-15T14:29:41.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:29:41.655 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.104474+0000", 2023-01-15T14:29:41.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016754199999999999 2023-01-15T14:29:41.656 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.656 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.656 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:41.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.104888+0000", 2023-01-15T14:29:41.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041411900000000001 2023-01-15T14:29:41.656 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.657 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.657 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:41.657 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.104902+0000", 2023-01-15T14:29:41.657 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3893999999999999e-05 2023-01-15T14:29:41.657 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.657 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.657 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:41.657 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.105253+0000", 2023-01-15T14:29:41.657 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035087900000000001 2023-01-15T14:29:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.105812+0000", 2023-01-15T14:29:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055870500000000005 2023-01-15T14:29:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303735+0000", 2023-01-15T14:29:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19792390100000001 2023-01-15T14:29:41.658 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.659 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.659 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:41.659 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403911+0000", 2023-01-15T14:29:41.659 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100175074 2023-01-15T14:29:41.659 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.659 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.659 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:41.659 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403997+0000", 2023-01-15T14:29:41.659 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6210000000000003e-05 2023-01-15T14:29:41.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.660 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:41.660 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404060+0000", 2023-01-15T14:29:41.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3759999999999999e-05 2023-01-15T14:29:41.660 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.660 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:41.660 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.661 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.661 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:87 3.9 3:9f86bdcb:::smithi09025264-22:head [setxattr _header (56) in=7b,truncate 2352947] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:41.661 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.573636+0000", 2023-01-15T14:29:41.661 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 74.812653456999996, 2023-01-15T14:29:41.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50637829499999998, 2023-01-15T14:29:41.661 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:41.661 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:41.661 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 87 2023-01-15T14:29:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573636+0000", 2023-01-15T14:29:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:41.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573636+0000", 2023-01-15T14:29:41.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:41.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573635+0000", 2023-01-15T14:29:41.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:41.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:41.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573641+0000", 2023-01-15T14:29:41.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1330000000000004e-06 2023-01-15T14:29:41.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:41.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573661+0000", 2023-01-15T14:29:41.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9692999999999999e-05 2023-01-15T14:29:41.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:41.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573667+0000", 2023-01-15T14:29:41.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3439999999999997e-06 2023-01-15T14:29:41.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.575672+0000", 2023-01-15T14:29:41.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020049310000000002 2023-01-15T14:29:41.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:41.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.575797+0000", 2023-01-15T14:29:41.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000125227 2023-01-15T14:29:41.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.768263+0000", 2023-01-15T14:29:41.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19246540600000001 2023-01-15T14:29:41.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:41.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.768456+0000", 2023-01-15T14:29:41.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019351899999999999 2023-01-15T14:29:41.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:29:41.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.768659+0000", 2023-01-15T14:29:41.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020282300000000001 2023-01-15T14:29:41.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:41.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.768972+0000", 2023-01-15T14:29:41.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00031250199999999998 2023-01-15T14:29:41.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:41.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.768997+0000", 2023-01-15T14:29:41.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4793000000000001e-05 2023-01-15T14:29:41.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:41.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.769072+0000", 2023-01-15T14:29:41.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5573000000000003e-05 2023-01-15T14:29:41.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:41.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.769201+0000", 2023-01-15T14:29:41.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012888900000000001 2023-01-15T14:29:41.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:41.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.860872+0000", 2023-01-15T14:29:41.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.091671277999999995 2023-01-15T14:29:41.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:41.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.078904+0000", 2023-01-15T14:29:41.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.21803145299999999 2023-01-15T14:29:41.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079192+0000", 2023-01-15T14:29:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028802599999999998 2023-01-15T14:29:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080015+0000", 2023-01-15T14:29:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00082284100000000005 2023-01-15T14:29:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:88 3.9 3:9f86bdcb:::smithi09025264-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.573716+0000", 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 74.812574092999995, 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50698550600000003, 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 88 2023-01-15T14:29:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573716+0000", 2023-01-15T14:29:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573716+0000", 2023-01-15T14:29:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573715+0000", 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573720+0000", 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5060000000000003e-06 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.831 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:41.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573730+0000", 2023-01-15T14:29:41.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0077e-05 2023-01-15T14:29:41.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:41.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.573737+0000", 2023-01-15T14:29:41.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:29:41.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.575851+0000", 2023-01-15T14:29:41.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021140429999999999 2023-01-15T14:29:41.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:41.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.576038+0000", 2023-01-15T14:29:41.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000187033 2023-01-15T14:29:41.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.769084+0000", 2023-01-15T14:29:41.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19304579199999999 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.769185+0000", 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101233 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080178+0000", 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31099339999999998 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:41.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080417+0000", 2023-01-15T14:29:41.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023812900000000001 2023-01-15T14:29:41.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:41.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080701+0000", 2023-01-15T14:29:41.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028463300000000001 2023-01-15T14:29:41.835 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.835 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:41.835 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:41.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.836 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:142 3.9 3:9988ac9d:::smithi09025264-36:head [write 1865105~489678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:41.836 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.865978+0000", 2023-01-15T14:29:41.836 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 74.520311438999997, 2023-01-15T14:29:41.836 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.53392634699999997, 2023-01-15T14:29:41.836 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:41.836 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:41.836 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:41.836 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:41.836 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 142 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.865978+0000", 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.865978+0000", 2023-01-15T14:29:41.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.865977+0000", 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866409+0000", 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043200999999999998 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866429+0000", 2023-01-15T14:29:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9636999999999999e-05 2023-01-15T14:29:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866437+0000", 2023-01-15T14:29:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8829999999999995e-06 2023-01-15T14:29:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.979 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867463+0000", 2023-01-15T14:29:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001025584 2023-01-15T14:29:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.981 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867556+0000", 2023-01-15T14:29:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2477000000000002e-05 2023-01-15T14:29:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.982 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096010+0000", 2023-01-15T14:29:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.228454887 2023-01-15T14:29:41.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:41.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096093+0000", 2023-01-15T14:29:41.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2503999999999995e-05 2023-01-15T14:29:41.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.986 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:29:41.986 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096231+0000", 2023-01-15T14:29:41.986 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013782599999999999 2023-01-15T14:29:41.987 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.987 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.987 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:41.987 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096643+0000", 2023-01-15T14:29:41.987 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041226199999999999 2023-01-15T14:29:41.988 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:41.988 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:41.988 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:41.988 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096696+0000", 2023-01-15T14:29:41.988 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2960000000000001e-05 2023-01-15T14:29:42.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:42.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096941+0000", 2023-01-15T14:29:42.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024520199999999997 2023-01-15T14:29:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097190+0000", 2023-01-15T14:29:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024874899999999999 2023-01-15T14:29:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.199089+0000", 2023-01-15T14:29:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101899288 2023-01-15T14:29:42.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.399771+0000", 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20068177500000001 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.399849+0000", 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7713000000000006e-05 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:42.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.399905+0000", 2023-01-15T14:29:42.008 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5949999999999998e-05 2023-01-15T14:29:42.008 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:42.008 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:42.008 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:42.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.009 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:143 3.9 3:9988ac9d:::smithi09025264-36:head [setxattr _header (56) in=7b,truncate 2588245] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:42.009 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.866459+0000", 2023-01-15T14:29:42.009 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 74.519830346000006, 2023-01-15T14:29:42.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.53372006000000005, 2023-01-15T14:29:42.009 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:42.009 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:42.009 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:42.009 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:42.010 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:42.010 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 143 2023-01-15T14:29:42.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.010 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:42.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:42.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866459+0000", 2023-01-15T14:29:42.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:42.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:42.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866459+0000", 2023-01-15T14:29:42.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:42.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:42.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866458+0000", 2023-01-15T14:29:42.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:42.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:42.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866465+0000", 2023-01-15T14:29:42.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8750000000000002e-06 2023-01-15T14:29:42.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:42.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866477+0000", 2023-01-15T14:29:42.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1571e-05 2023-01-15T14:29:42.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:42.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866484+0000", 2023-01-15T14:29:42.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2660000000000004e-06 2023-01-15T14:29:42.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:42.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867615+0000", 2023-01-15T14:29:42.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011306580000000001 2023-01-15T14:29:42.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:42.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867703+0000", 2023-01-15T14:29:42.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8418000000000006e-05 2023-01-15T14:29:42.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:42.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096735+0000", 2023-01-15T14:29:42.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.22903211800000001 2023-01-15T14:29:42.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:42.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096814+0000", 2023-01-15T14:29:42.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9022000000000003e-05 2023-01-15T14:29:42.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:29:42.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096945+0000", 2023-01-15T14:29:42.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013099499999999999 2023-01-15T14:29:42.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:42.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097142+0000", 2023-01-15T14:29:42.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019696000000000001 2023-01-15T14:29:42.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:42.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097156+0000", 2023-01-15T14:29:42.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3614e-05 2023-01-15T14:29:42.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.019 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:42.019 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097214+0000", 2023-01-15T14:29:42.020 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8202999999999998e-05 2023-01-15T14:29:42.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:42.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097312+0000", 2023-01-15T14:29:42.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7496000000000003e-05 2023-01-15T14:29:42.021 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:42.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.199130+0000", 2023-01-15T14:29:42.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101818668 2023-01-15T14:29:42.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.022 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:42.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.399988+0000", 2023-01-15T14:29:42.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20085804099999999 2023-01-15T14:29:42.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:42.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.400085+0000", 2023-01-15T14:29:42.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6204000000000003e-05 2023-01-15T14:29:42.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:42.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.400179+0000", 2023-01-15T14:29:42.159 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4926000000000006e-05 2023-01-15T14:29:42.159 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:42.159 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:42.159 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:42.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.159 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:144 3.9 3:9988ac9d:::smithi09025264-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:42.159 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:29:42.159 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 74.519780737999994, 2023-01-15T14:29:42.159 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63423976199999998, 2023-01-15T14:29:42.159 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 144 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:42.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:29:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866508+0000", 2023-01-15T14:29:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:42.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866512+0000", 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5999999999999998e-06 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866523+0000", 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0740000000000001e-05 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866531+0000", 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3310000000000004e-06 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867744+0000", 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012127399999999999 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867810+0000", 2023-01-15T14:29:42.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6506000000000003e-05 2023-01-15T14:29:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:42.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097225+0000", 2023-01-15T14:29:42.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.22941514900000001 2023-01-15T14:29:42.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:42.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097295+0000", 2023-01-15T14:29:42.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:29:42.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.186 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:42.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500386+0000", 2023-01-15T14:29:42.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40309051699999998 2023-01-15T14:29:42.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.186 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:42.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500521+0000", 2023-01-15T14:29:42.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013561700000000001 2023-01-15T14:29:42.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.187 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.187 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:42.187 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500749+0000", 2023-01-15T14:29:42.187 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022754900000000001 2023-01-15T14:29:42.187 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:42.187 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:42.187 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:42.187 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.187 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.187 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:164 3.9 3:9f7cca9d:::smithi09025264-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:42.188 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.102061+0000", 2023-01-15T14:29:42.188 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 74.284229124999996, 2023-01-15T14:29:42.188 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50348630999999999, 2023-01-15T14:29:42.188 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:42.188 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:42.188 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:42.188 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:42.188 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:42.188 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 164 2023-01-15T14:29:42.188 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.189 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:42.189 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.189 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:42.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.102061+0000", 2023-01-15T14:29:42.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:42.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:42.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.102061+0000", 2023-01-15T14:29:42.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:42.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:42.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.102060+0000", 2023-01-15T14:29:42.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:42.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:42.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.102064+0000", 2023-01-15T14:29:42.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0600000000000001e-06 2023-01-15T14:29:42.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:42.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.102073+0000", 2023-01-15T14:29:42.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8999999999999995e-06 2023-01-15T14:29:42.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:42.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.102081+0000", 2023-01-15T14:29:42.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7729999999999999e-06 2023-01-15T14:29:42.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:42.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.503640+0000", 2023-01-15T14:29:42.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40155891799999999 2023-01-15T14:29:42.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:42.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.503752+0000", 2023-01-15T14:29:42.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112722 2023-01-15T14:29:42.194 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:42.194 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.605224+0000", 2023-01-15T14:29:42.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101471688 2023-01-15T14:29:42.335 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.335 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:42.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.605362+0000", 2023-01-15T14:29:42.335 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013830599999999999 2023-01-15T14:29:42.335 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.335 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.336 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:42.336 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.605547+0000", 2023-01-15T14:29:42.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018465300000000001 2023-01-15T14:29:42.336 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:42.336 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:42.336 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:42.336 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.336 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.336 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:210 3.9 3:9e7635b9:::smithi09025264-21:head [setxattr _header (56) in=7b,truncate 2857157] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:42.336 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.510600+0000", 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 73.875690067999997, 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.59972464199999997, 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 210 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.337 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:42.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510600+0000", 2023-01-15T14:29:42.338 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:42.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.338 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:42.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510600+0000", 2023-01-15T14:29:42.338 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:42.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.338 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:42.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510598+0000", 2023-01-15T14:29:42.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:42.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:42.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510605+0000", 2023-01-15T14:29:42.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2069999999999998e-06 2023-01-15T14:29:42.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:42.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510617+0000", 2023-01-15T14:29:42.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1644000000000001e-05 2023-01-15T14:29:42.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.341 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:42.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510625+0000", 2023-01-15T14:29:42.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9109999999999993e-06 2023-01-15T14:29:42.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.362 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:42.362 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510857+0000", 2023-01-15T14:29:42.362 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023164600000000001 2023-01-15T14:29:42.362 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:42.362 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:42.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:42.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510950+0000", 2023-01-15T14:29:42.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3145000000000004e-05 2023-01-15T14:29:42.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:43.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.707435+0000", 2023-01-15T14:29:44.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19648538800000001 2023-01-15T14:29:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.707681+0000", 2023-01-15T14:29:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024560899999999999 2023-01-15T14:29:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:29:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708063+0000", 2023-01-15T14:29:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038193199999999997 2023-01-15T14:29:44.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:44.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708415+0000", 2023-01-15T14:29:44.651 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000352042 2023-01-15T14:29:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708429+0000", 2023-01-15T14:29:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.469e-05 2023-01-15T14:29:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708780+0000", 2023-01-15T14:29:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035064900000000002 2023-01-15T14:29:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708828+0000", 2023-01-15T14:29:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7756999999999997e-05 2023-01-15T14:29:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:44.655 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.809761+0000", 2023-01-15T14:29:44.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100933525 2023-01-15T14:29:44.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110150+0000", 2023-01-15T14:29:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30038901600000001 2023-01-15T14:29:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.657 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.657 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:44.657 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110226+0000", 2023-01-15T14:29:44.657 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5475999999999994e-05 2023-01-15T14:29:44.657 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110324+0000", 2023-01-15T14:29:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8583000000000001e-05 2023-01-15T14:29:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:211 3.9 3:9e7635b9:::smithi09025264-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.510639+0000", 2023-01-15T14:29:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 73.875650819000001, 2023-01-15T14:29:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60010284700000005, 2023-01-15T14:29:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 211 2023-01-15T14:29:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:44.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510639+0000", 2023-01-15T14:29:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510639+0000", 2023-01-15T14:29:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510638+0000", 2023-01-15T14:29:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510643+0000", 2023-01-15T14:29:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3340000000000003e-06 2023-01-15T14:29:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510653+0000", 2023-01-15T14:29:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0961999999999999e-05 2023-01-15T14:29:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510663+0000", 2023-01-15T14:29:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4800000000000007e-06 2023-01-15T14:29:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.601338+0000", 2023-01-15T14:29:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090674642 2023-01-15T14:29:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.601480+0000", 2023-01-15T14:29:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014286299999999999 2023-01-15T14:29:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708540+0000", 2023-01-15T14:29:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.107059404 2023-01-15T14:29:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708676+0000", 2023-01-15T14:29:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013605 2023-01-15T14:29:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110423+0000", 2023-01-15T14:29:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40174706199999999 2023-01-15T14:29:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110590+0000", 2023-01-15T14:29:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000166773 2023-01-15T14:29:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110742+0000", 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000152022 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:508 3.9 3:9f86bdcb:::smithi09025264-22:head [write 1808051~636013] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.809136+0000", 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 51.577153373999998, 2023-01-15T14:29:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60160218099999996, 2023-01-15T14:29:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 508 2023-01-15T14:29:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809136+0000", 2023-01-15T14:29:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809136+0000", 2023-01-15T14:29:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:44.944 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809135+0000", 2023-01-15T14:29:44.944 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:44.944 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.944 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.944 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:44.944 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809776+0000", 2023-01-15T14:29:44.944 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00064178800000000002 2023-01-15T14:29:44.944 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.945 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.945 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:44.945 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809801+0000", 2023-01-15T14:29:44.945 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4851999999999999e-05 2023-01-15T14:29:44.945 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.946 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.946 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:44.947 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809811+0000", 2023-01-15T14:29:44.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0285000000000001e-05 2023-01-15T14:29:44.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.948 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.948 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:44.948 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.903875+0000", 2023-01-15T14:29:44.948 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.094063472999999995 2023-01-15T14:29:44.948 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.903888+0000", 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.27e-05 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906243+0000", 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023554980000000001 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.949 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:44.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906404+0000", 2023-01-15T14:29:44.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000160705 2023-01-15T14:29:44.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:44.951 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.207137+0000", 2023-01-15T14:29:44.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30073284700000003 2023-01-15T14:29:44.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.951 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:44.952 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.207316+0000", 2023-01-15T14:29:44.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000179125 2023-01-15T14:29:44.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:29:44.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.207563+0000", 2023-01-15T14:29:44.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024729599999999999 2023-01-15T14:29:44.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:44.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208190+0000", 2023-01-15T14:29:44.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062669899999999998 2023-01-15T14:29:44.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:44.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208208+0000", 2023-01-15T14:29:44.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7759999999999999e-05 2023-01-15T14:29:44.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:44.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208503+0000", 2023-01-15T14:29:44.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000295109 2023-01-15T14:29:44.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:44.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311080+0000", 2023-01-15T14:29:44.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10257740899999999 2023-01-15T14:29:44.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:44.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311363+0000", 2023-01-15T14:29:44.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028312299999999998 2023-01-15T14:29:44.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.410431+0000", 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099068120999999995 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.410590+0000", 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015870700000000001 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:44.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:45.000 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.410739+0000", 2023-01-15T14:29:45.000 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014854899999999999 2023-01-15T14:29:45.000 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.000 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:45.000 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.000 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.000 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.000 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:509 3.9 3:9f86bdcb:::smithi09025264-22:head [write 2873212~588997] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:29:45.000 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:29:45.000 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 51.571356395000002, 2023-01-15T14:29:45.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69623497099999998, 2023-01-15T14:29:45.001 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:45.001 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:45.001 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:45.001 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:45.001 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:45.001 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 509 2023-01-15T14:29:45.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.001 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:45.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:45.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:29:45.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:45.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:29:45.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:45.008 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814931+0000", 2023-01-15T14:29:45.008 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:45.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.008 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:45.008 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815512+0000", 2023-01-15T14:29:45.008 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058082200000000004 2023-01-15T14:29:45.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:45.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815553+0000", 2023-01-15T14:29:45.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1081999999999997e-05 2023-01-15T14:29:45.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:45.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815571+0000", 2023-01-15T14:29:45.010 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8028000000000001e-05 2023-01-15T14:29:45.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906452+0000", 2023-01-15T14:29:45.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090880586999999999 2023-01-15T14:29:45.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:45.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906564+0000", 2023-01-15T14:29:45.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111976 2023-01-15T14:29:45.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208424+0000", 2023-01-15T14:29:45.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.301860199 2023-01-15T14:29:45.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:45.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208718+0000", 2023-01-15T14:29:45.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029347299999999999 2023-01-15T14:29:45.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:29:45.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209175+0000", 2023-01-15T14:29:45.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045692399999999999 2023-01-15T14:29:45.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:45.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209725+0000", 2023-01-15T14:29:45.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055027600000000004 2023-01-15T14:29:45.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:45.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209751+0000", 2023-01-15T14:29:45.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5812999999999998e-05 2023-01-15T14:29:45.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:45.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.210006+0000", 2023-01-15T14:29:45.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025564099999999997 2023-01-15T14:29:45.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:45.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311247+0000", 2023-01-15T14:29:45.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10124119199999999 2023-01-15T14:29:45.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311454+0000", 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020695800000000001 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511022+0000", 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19956729200000001 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511100+0000", 2023-01-15T14:29:45.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8589000000000003e-05 2023-01-15T14:29:45.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:45.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511168+0000", 2023-01-15T14:29:45.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8039000000000004e-05 2023-01-15T14:29:45.096 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.096 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:45.098 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.099 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:510 3.9 3:9f86bdcb:::smithi09025264-22:head [setxattr _header (57) in=7b,truncate 3462209] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:29:45.099 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:29:45.099 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 51.570692932, 2023-01-15T14:29:45.099 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69609656399999997, 2023-01-15T14:29:45.099 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:45.099 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:45.099 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:45.099 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:45.100 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:45.100 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 510 2023-01-15T14:29:45.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.101 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:45.101 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.101 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:45.101 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:29:45.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.101 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.101 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:45.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:29:45.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:45.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815596+0000", 2023-01-15T14:29:45.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:45.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:45.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815606+0000", 2023-01-15T14:29:45.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7920000000000004e-06 2023-01-15T14:29:45.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:45.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815627+0000", 2023-01-15T14:29:45.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1239000000000001e-05 2023-01-15T14:29:45.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:45.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815637+0000", 2023-01-15T14:29:45.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0349e-05 2023-01-15T14:29:45.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906600+0000", 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090963098000000006 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906717+0000", 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000116759 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310000+0000", 2023-01-15T14:29:45.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40328322599999999 2023-01-15T14:29:45.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:45.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310249+0000", 2023-01-15T14:29:45.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024874400000000002 2023-01-15T14:29:45.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:29:45.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310559+0000", 2023-01-15T14:29:45.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310382 2023-01-15T14:29:45.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:45.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310951+0000", 2023-01-15T14:29:45.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039124500000000003 2023-01-15T14:29:45.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:45.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310975+0000", 2023-01-15T14:29:45.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4496000000000001e-05 2023-01-15T14:29:45.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.175 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:45.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311041+0000", 2023-01-15T14:29:45.176 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6309000000000005e-05 2023-01-15T14:29:45.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:45.177 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311260+0000", 2023-01-15T14:29:45.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021863399999999999 2023-01-15T14:29:45.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:45.177 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311588+0000", 2023-01-15T14:29:45.180 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032813599999999998 2023-01-15T14:29:45.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:45.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511351+0000", 2023-01-15T14:29:45.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19976275700000001 2023-01-15T14:29:45.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:45.182 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511507+0000", 2023-01-15T14:29:45.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015585100000000001 2023-01-15T14:29:45.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.187 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:45.187 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511693+0000", 2023-01-15T14:29:45.188 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186627 2023-01-15T14:29:45.188 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.188 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:45.188 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.188 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.188 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.188 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:511 3.9 3:9f86bdcb:::smithi09025264-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:29:45.188 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:29:45.188 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 51.570635564, 2023-01-15T14:29:45.188 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69666647500000001, 2023-01-15T14:29:45.188 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 511 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.189 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:45.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:29:45.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:45.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815653+0000", 2023-01-15T14:29:45.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:45.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:45.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815660+0000", 2023-01-15T14:29:45.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0509999999999999e-06 2023-01-15T14:29:45.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:45.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815683+0000", 2023-01-15T14:29:45.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2742999999999999e-05 2023-01-15T14:29:45.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:45.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815698+0000", 2023-01-15T14:29:45.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5133999999999999e-05 2023-01-15T14:29:45.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908029+0000", 2023-01-15T14:29:45.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.092331448999999996 2023-01-15T14:29:45.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:45.194 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908193+0000", 2023-01-15T14:29:45.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016342799999999999 2023-01-15T14:29:45.194 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.194 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311115+0000", 2023-01-15T14:29:45.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40292222300000002 2023-01-15T14:29:45.194 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:45.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311224+0000", 2023-01-15T14:29:45.195 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010888899999999999 2023-01-15T14:29:45.195 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.195 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.195 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511800+0000", 2023-01-15T14:29:45.195 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20057596899999999 2023-01-15T14:29:45.195 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:45.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512036+0000", 2023-01-15T14:29:45.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236247 2023-01-15T14:29:45.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:45.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512321+0000", 2023-01-15T14:29:45.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028457599999999998 2023-01-15T14:29:45.278 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.278 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:45.278 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.279 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:692 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~2016807,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:29:45.279 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898357+0000", 2023-01-15T14:29:45.279 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 33.487933110999997, 2023-01-15T14:29:45.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60411950800000003, 2023-01-15T14:29:45.279 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:45.279 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:45.279 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:45.279 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:45.279 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:45.279 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 692 2023-01-15T14:29:45.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.280 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:45.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:45.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898357+0000", 2023-01-15T14:29:45.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.284 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:45.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898357+0000", 2023-01-15T14:29:45.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.284 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:45.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898356+0000", 2023-01-15T14:29:45.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:45.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898360+0000", 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2740000000000001e-06 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898373+0000", 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.216e-05 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.305 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:45.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898381+0000", 2023-01-15T14:29:45.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.225e-06 2023-01-15T14:29:45.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.500519+0000", 2023-01-15T14:29:45.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60213839400000002 2023-01-15T14:29:45.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:45.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.500676+0000", 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000156348 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.502476+0000", 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001800638 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.307 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:693 3.3 3:ca59e2b0:::smithi09025264-1:head [sparse-read 0~2016807] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:29:45.308 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:29:45.308 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 33.487902216999998, 2023-01-15T14:29:45.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70514199600000005, 2023-01-15T14:29:45.308 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:45.308 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:45.308 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:45.308 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:45.308 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:45.308 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 693 2023-01-15T14:29:45.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.308 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:45.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:45.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:29:45.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:45.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:29:45.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:45.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898387+0000", 2023-01-15T14:29:45.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:45.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:45.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898390+0000", 2023-01-15T14:29:45.368 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.252e-06 2023-01-15T14:29:45.369 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:45.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898401+0000", 2023-01-15T14:29:45.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0977e-05 2023-01-15T14:29:45.369 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:45.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.998532+0000", 2023-01-15T14:29:45.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100130443 2023-01-15T14:29:45.370 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.370 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.370 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.370 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602663+0000", 2023-01-15T14:29:45.370 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60413100500000005 2023-01-15T14:29:45.370 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.370 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.370 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:45.371 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602817+0000", 2023-01-15T14:29:45.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015440500000000001 2023-01-15T14:29:45.371 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.371 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.371 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:45.371 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.603530+0000", 2023-01-15T14:29:45.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071246299999999999 2023-01-15T14:29:45.371 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.371 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:45.371 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.371 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.372 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.372 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:864 3.3 3:c3b8cf33:::smithi09025264-26:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2023-01-15T14:29:45.372 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:22.747721+0000", 2023-01-15T14:29:45.372 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 18.638568437, 2023-01-15T14:29:45.372 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50897932499999998, 2023-01-15T14:29:45.372 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:45.372 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:45.372 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:45.372 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:45.372 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:45.372 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 864 2023-01-15T14:29:45.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.373 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:45.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:45.373 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747721+0000", 2023-01-15T14:29:45.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:45.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747721+0000", 2023-01-15T14:29:45.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:45.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747720+0000", 2023-01-15T14:29:45.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:45.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:45.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747727+0000", 2023-01-15T14:29:45.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5760000000000002e-06 2023-01-15T14:29:45.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:45.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747746+0000", 2023-01-15T14:29:45.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9202999999999999e-05 2023-01-15T14:29:45.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:45.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747753+0000", 2023-01-15T14:29:45.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8469999999999998e-06 2023-01-15T14:29:45.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.948184+0000", 2023-01-15T14:29:45.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20043086500000001 2023-01-15T14:29:45.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:45.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.948377+0000", 2023-01-15T14:29:45.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000192776 2023-01-15T14:29:45.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.609 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.609 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.610 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.952904+0000", 2023-01-15T14:29:45.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0045275389999999997 2023-01-15T14:29:45.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:45.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.953017+0000", 2023-01-15T14:29:45.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011311600000000001 2023-01-15T14:29:45.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:29:45.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.953190+0000", 2023-01-15T14:29:45.612 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000173253 2023-01-15T14:29:45.612 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.612 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.612 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:45.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.953442+0000", 2023-01-15T14:29:45.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025153300000000002 2023-01-15T14:29:45.613 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:45.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.953459+0000", 2023-01-15T14:29:45.614 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7348000000000001e-05 2023-01-15T14:29:45.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.614 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:45.615 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.953522+0000", 2023-01-15T14:29:45.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3077999999999995e-05 2023-01-15T14:29:45.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.615 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:45.616 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.953570+0000", 2023-01-15T14:29:45.616 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7477e-05 2023-01-15T14:29:45.616 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.616 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.616 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:45.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:23.055218+0000", 2023-01-15T14:29:45.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101648434 2023-01-15T14:29:45.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:45.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:23.256331+0000", 2023-01-15T14:29:45.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20111243300000001 2023-01-15T14:29:45.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:45.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:23.256485+0000", 2023-01-15T14:29:45.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015451700000000001 2023-01-15T14:29:45.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:45.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:23.256701+0000", 2023-01-15T14:29:45.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021537999999999999 2023-01-15T14:29:45.620 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.620 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:45.620 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.621 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:865 3.3 3:c3b8cf33:::smithi09025264-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2023-01-15T14:29:45.621 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:22.747782+0000", 2023-01-15T14:29:45.621 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 18.638507552, 2023-01-15T14:29:45.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50942573599999996, 2023-01-15T14:29:45.622 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:45.622 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:45.622 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:45.622 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:45.622 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:45.623 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 865 2023-01-15T14:29:45.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.623 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:45.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:45.624 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747782+0000", 2023-01-15T14:29:45.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:45.624 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747782+0000", 2023-01-15T14:29:45.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:45.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:45.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747781+0000", 2023-01-15T14:29:45.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:45.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:45.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747786+0000", 2023-01-15T14:29:45.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7099999999999998e-06 2023-01-15T14:29:45.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:45.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747801+0000", 2023-01-15T14:29:45.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.52e-05 2023-01-15T14:29:45.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:45.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.747809+0000", 2023-01-15T14:29:45.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3930000000000002e-06 2023-01-15T14:29:45.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.948624+0000", 2023-01-15T14:29:45.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20081521399999999 2023-01-15T14:29:45.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.641 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:45.641 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.948728+0000", 2023-01-15T14:29:45.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000104444 2023-01-15T14:29:45.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.642 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.953941+0000", 2023-01-15T14:29:45.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0052129050000000003 2023-01-15T14:29:45.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:45.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:22.954099+0000", 2023-01-15T14:29:45.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015783099999999999 2023-01-15T14:29:45.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:45.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:23.256843+0000", 2023-01-15T14:29:45.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30274373599999999 2023-01-15T14:29:45.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:45.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:23.256998+0000", 2023-01-15T14:29:45.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000155024 2023-01-15T14:29:45.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:45.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:45.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:45.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:23.257208+0000", 2023-01-15T14:29:45.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021001900000000001 2023-01-15T14:29:45.646 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.646 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:45.833 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.833 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:45.834 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:45.834 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:45.835 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:29:45.835 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:41.424+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:45.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:29:45.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:29:45.838 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:45.832+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:45.839 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:45.839 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:45.840 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:29:45.840 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:29:45.841 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:29:45.841 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:29:45.841 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:29:45.841 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:29:45.841 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:29:45.841 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:29:45.856 INFO:tasks.rados.rados.0.smithi090.stdout:388: finishing write tid 3 to smithi09025264-49 2023-01-15T14:29:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:388: finishing write tid 4 to smithi09025264-49 2023-01-15T14:29:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:388: oid 49 updating version 0 to 141 2023-01-15T14:29:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:388: oid 49 updating version 141 to 142 2023-01-15T14:29:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:388: oid 49 updating version 142 to 143 2023-01-15T14:29:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 143 (ObjNum 168 snap 37 seq_num 168) dirty exists 2023-01-15T14:29:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:388: left oid 49 (ObjNum 168 snap 37 seq_num 168) 2023-01-15T14:29:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:388: done (1 left) 2023-01-15T14:29:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:390: done (0 left) 2023-01-15T14:29:45.857 INFO:tasks.rados.rados.0.smithi090.stdout:391: snap_remove snap 36 2023-01-15T14:29:45.858 INFO:tasks.rados.rados.0.smithi090.stdout:391: done (0 left) 2023-01-15T14:29:45.858 INFO:tasks.rados.rados.0.smithi090.stdout:392: write oid 50 current snap is 37 2023-01-15T14:29:45.858 INFO:tasks.rados.rados.0.smithi090.stdout:392: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2023-01-15T14:29:45.858 INFO:tasks.rados.rados.0.smithi090.stdout:392: writing smithi09025264-50 from 457190 to 1009286 tid 1 2023-01-15T14:29:45.858 INFO:tasks.rados.rados.0.smithi090.stdout:392: writing smithi09025264-50 from 1614880 to 2195921 tid 2 2023-01-15T14:29:45.859 INFO:tasks.rados.rados.0.smithi090.stdout:392: writing smithi09025264-50 from 2785791 to 2857190 tid 3 2023-01-15T14:29:45.859 INFO:tasks.rados.rados.0.smithi090.stdout:393: rollback oid 32 current snap is 37 2023-01-15T14:29:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 32 to 34 2023-01-15T14:29:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:394: rollback oid 29 current snap is 37 2023-01-15T14:29:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 29 to 34 2023-01-15T14:29:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:395: read oid 37 snap 34 2023-01-15T14:29:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:395: expect (ObjNum 132 snap 24 seq_num 132) 2023-01-15T14:29:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:396: copy_from oid 7 from oid 15 current snap is 37 2023-01-15T14:29:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:397: rollback oid 42 current snap is 37 2023-01-15T14:29:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 42 to 30 2023-01-15T14:29:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:398: delete oid 10 current snap is 37 2023-01-15T14:29:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:398: done (6 left) 2023-01-15T14:29:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:399: read oid 33 snap 34 2023-01-15T14:29:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:399: expect (ObjNum 149 snap 30 seq_num 149) 2023-01-15T14:29:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:400: read oid 43 snap -1 2023-01-15T14:29:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:400: expect deleted 2023-01-15T14:29:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:401: read oid 19 snap 31 2023-01-15T14:29:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:401: expect (ObjNum 87 snap 12 seq_num 87) 2023-01-15T14:29:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:402: rollback oid 5 current snap is 37 2023-01-15T14:29:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 5 to 31 2023-01-15T14:29:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:403: rollback oid 31 current snap is 37 2023-01-15T14:29:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 31 to 34 2023-01-15T14:29:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:404: read oid 13 snap 34 2023-01-15T14:29:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:404: expect (ObjNum 133 snap 24 seq_num 133) 2023-01-15T14:29:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:405: read oid 21 snap -1 2023-01-15T14:29:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:405: expect (ObjNum 83 snap 11 seq_num 83) 2023-01-15T14:29:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:406: delete oid 34 current snap is 37 2023-01-15T14:29:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:403: finishing rollback tid 1 to smithi09025264-31 2023-01-15T14:29:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 0 (ObjNum 161 snap 0 seq_num 3201271312) dirty dne 2023-01-15T14:29:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:399: expect (ObjNum 149 snap 30 seq_num 149) 2023-01-15T14:29:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:404: expect (ObjNum 133 snap 24 seq_num 133) 2023-01-15T14:29:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:394: finishing rollback tid 0 to smithi09025264-29 2023-01-15T14:29:46.114 INFO:tasks.rados.rados.0.smithi090.stdout:394: finishing rollback tid 1 to smithi09025264-29 2023-01-15T14:29:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:395: expect (ObjNum 132 snap 24 seq_num 132) 2023-01-15T14:29:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:393: finishing rollback tid 0 to smithi09025264-32 2023-01-15T14:29:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:394: finishing rollback tid 2 to smithi09025264-29 2023-01-15T14:29:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 127 (ObjNum 132 snap 24 seq_num 132) dirty exists 2023-01-15T14:29:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:393: finishing rollback tid 1 to smithi09025264-32 2023-01-15T14:29:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:401: expect (ObjNum 87 snap 12 seq_num 87) 2023-01-15T14:29:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:405: expect (ObjNum 83 snap 11 seq_num 83) 2023-01-15T14:29:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:393: finishing rollback tid 2 to smithi09025264-32 2023-01-15T14:29:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 111 (ObjNum 108 snap 23 seq_num 108) dirty exists 2023-01-15T14:29:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:397: finishing rollback tid 1 to smithi09025264-42 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:397: finishing rollback tid 2 to smithi09025264-42 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 42 v 149 (ObjNum 91 snap 15 seq_num 91) dirty exists 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:402: finishing rollback tid 1 to smithi09025264-5 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:392: finishing write tid 1 to smithi09025264-50 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:402: finishing rollback tid 2 to smithi09025264-5 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 151 (ObjNum 150 snap 30 seq_num 150) dirty exists 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:393: done (13 left) 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:394: done (12 left) 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:395: done (11 left) 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:397: done (10 left) 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:399: done (9 left) 2023-01-15T14:29:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:400: done (8 left) 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:401: done (7 left) 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:402: done (6 left) 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:403: done (5 left) 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:404: done (4 left) 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:405: done (3 left) 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:406: done (2 left) 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:407: rollback oid 10 current snap is 37 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 10 to 31 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:408: read oid 9 snap -1 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:408: expect (ObjNum 37 snap 0 seq_num 37) 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:409: delete oid 49 current snap is 37 2023-01-15T14:29:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:407: finishing rollback tid 1 to smithi09025264-10 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:407: finishing rollback tid 2 to smithi09025264-10 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 181 (ObjNum 37 snap 0 seq_num 37) dirty exists 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:392: finishing write tid 2 to smithi09025264-50 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:392: finishing write tid 3 to smithi09025264-50 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:392: finishing write tid 4 to smithi09025264-50 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:392: finishing write tid 5 to smithi09025264-50 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:392: oid 50 updating version 0 to 83 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:392: oid 50 version 83 is already newer than 82 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:392: oid 50 version 83 is already newer than 80 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:392: oid 50 version 83 is already newer than 81 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 50 v 83 (ObjNum 170 snap 37 seq_num 170) dirty exists 2023-01-15T14:29:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:392: left oid 50 (ObjNum 170 snap 37 seq_num 170) 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:408: expect (ObjNum 37 snap 0 seq_num 37) 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:396: finishing copy_from to smithi09025264-7 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 59 (ObjNum 128 snap 23 seq_num 128) dirty exists 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:396: finishing copy_from racing read to smithi09025264-7 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:392: done (4 left) 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:396: done (3 left) 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:407: done (2 left) 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:408: done (1 left) 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:409: done (0 left) 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:410: write (excl) oid 49 current snap is 37 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:410: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2023-01-15T14:29:46.119 INFO:tasks.rados.rados.0.smithi090.stdout:410: writing smithi09025264-49 from 524418 to 977510 tid 1 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:410: writing smithi09025264-49 from 1732455 to 2320834 tid 2 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:410: writing smithi09025264-49 from 2779130 to 2924418 tid 3 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:411: delete oid 32 current snap is 37 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:411: done (1 left) 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:412: rollback oid 6 current snap is 37 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 6 to 34 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:413: write oid 24 current snap is 37 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:413: seq_num 176 ranges {558032=719943,1849419=592048} 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:413: writing smithi09025264-24 from 558032 to 1277975 tid 1 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:413: writing smithi09025264-24 from 1849419 to 2441467 tid 2 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:414: read oid 12 snap -1 2023-01-15T14:29:46.120 INFO:tasks.rados.rados.0.smithi090.stdout:414: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:29:46.121 INFO:tasks.rados.rados.0.smithi090.stdout:415: read oid 3 snap -1 2023-01-15T14:29:46.121 INFO:tasks.rados.rados.0.smithi090.stdout:415: expect deleted 2023-01-15T14:29:46.121 INFO:tasks.rados.rados.0.smithi090.stdout:416: read oid 22 snap -1 2023-01-15T14:29:46.121 INFO:tasks.rados.rados.0.smithi090.stdout:416: expect (ObjNum 160 snap 35 seq_num 160) 2023-01-15T14:29:46.121 INFO:tasks.rados.rados.0.smithi090.stdout:417: read oid 35 snap -1 2023-01-15T14:29:46.121 INFO:tasks.rados.rados.0.smithi090.stdout:417: expect deleted 2023-01-15T14:29:46.121 INFO:tasks.rados.rados.0.smithi090.stdout:418: copy_from oid 19 from oid 33 current snap is 37 2023-01-15T14:29:46.121 INFO:tasks.rados.rados.0.smithi090.stdout:419: copy_from oid 23 from oid 20 current snap is 37 2023-01-15T14:29:46.121 INFO:tasks.rados.rados.0.smithi090.stdout:420: snap_remove snap 30 2023-01-15T14:29:46.121 INFO:tasks.rados.rados.0.smithi090.stdout:419: finishing copy_from to smithi09025264-23 2023-01-15T14:29:46.121 INFO:tasks.rados.rados.0.smithi090.stdout:419: got expected ENOENT (src dne) 2023-01-15T14:29:46.122 INFO:tasks.rados.rados.0.smithi090.stdout:419: finishing copy_from racing read to smithi09025264-23 2023-01-15T14:29:46.122 INFO:tasks.rados.rados.0.smithi090.stdout:410: finishing write tid 1 to smithi09025264-49 2023-01-15T14:29:46.122 INFO:tasks.rados.rados.0.smithi090.stdout:416: expect (ObjNum 160 snap 35 seq_num 160) 2023-01-15T14:29:46.122 INFO:tasks.rados.rados.0.smithi090.stdout:412: finishing rollback tid 0 to smithi09025264-6 2023-01-15T14:29:46.122 INFO:tasks.rados.rados.0.smithi090.stdout:412: finishing rollback tid 1 to smithi09025264-6 2023-01-15T14:29:46.122 INFO:tasks.rados.rados.0.smithi090.stdout:412: finishing rollback tid 2 to smithi09025264-6 2023-01-15T14:29:46.122 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 56 (ObjNum 157 snap 34 seq_num 157) dirty exists 2023-01-15T14:29:46.122 INFO:tasks.rados.rados.0.smithi090.stdout:410: finishing write tid 2 to smithi09025264-49 2023-01-15T14:29:46.122 INFO:tasks.rados.rados.0.smithi090.stdout:410: finishing write tid 3 to smithi09025264-49 2023-01-15T14:29:46.122 INFO:tasks.rados.rados.0.smithi090.stdout:410: finishing write tid 4 to smithi09025264-49 2023-01-15T14:29:46.122 INFO:tasks.rados.rados.0.smithi090.stdout:410: finishing write tid 5 to smithi09025264-49 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:410: oid 49 updating version 0 to 156 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:410: oid 49 version 156 is already newer than 155 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:410: oid 49 updating version 156 to 158 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:410: oid 49 version 158 is already newer than 157 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 158 (ObjNum 174 snap 37 seq_num 174) dirty exists 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:410: left oid 49 (ObjNum 174 snap 37 seq_num 174) 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:410: done (9 left) 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:412: done (8 left) 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:415: done (7 left) 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:416: done (6 left) 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:417: done (5 left) 2023-01-15T14:29:46.123 INFO:tasks.rados.rados.0.smithi090.stdout:419: done (4 left) 2023-01-15T14:29:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:420: done (3 left) 2023-01-15T14:29:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:421: write (excl) oid 9 current snap is 37 2023-01-15T14:29:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:414: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:29:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:418: finishing copy_from to smithi09025264-19 2023-01-15T14:29:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 51 (ObjNum 163 snap 36 seq_num 163) dirty exists 2023-01-15T14:29:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:418: finishing copy_from racing read to smithi09025264-19 2023-01-15T14:29:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:413: finishing write tid 1 to smithi09025264-24 2023-01-15T14:29:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:413: finishing write tid 2 to smithi09025264-24 2023-01-15T14:29:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:413: finishing write tid 3 to smithi09025264-24 2023-01-15T14:29:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:413: finishing write tid 4 to smithi09025264-24 2023-01-15T14:29:46.124 INFO:tasks.rados.rados.0.smithi090.stdout:413: oid 24 updating version 0 to 184 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:413: oid 24 updating version 184 to 185 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:413: oid 24 version 185 is already newer than 183 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 185 (ObjNum 176 snap 37 seq_num 176) dirty exists 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:413: left oid 24 (ObjNum 176 snap 37 seq_num 176) 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:421: seq_num 177 ranges {574839=795192,1907901=452059} 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:421: writing smithi09025264-9 from 574839 to 1370031 tid 1 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:421: writing smithi09025264-9 from 1907901 to 2359960 tid 2 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:413: done (3 left) 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:414: done (2 left) 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:418: done (1 left) 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout:422: snap_create 2023-01-15T14:29:46.125 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:421: finishing write tid 1 to smithi09025264-9 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:421: finishing write tid 2 to smithi09025264-9 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:421: finishing write tid 3 to smithi09025264-9 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:421: finishing write tid 4 to smithi09025264-9 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:421: oid 9 updating version 0 to 30 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:421: oid 9 version 30 is already newer than 29 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:421: oid 9 version 30 is already newer than 28 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 30 (ObjNum 177 snap 37 seq_num 177) dirty exists 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:421: left oid 9 (ObjNum 177 snap 37 seq_num 177) 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:421: done (0 left) 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:422: done (0 left) 2023-01-15T14:29:46.126 INFO:tasks.rados.rados.0.smithi090.stdout:423: write oid 13 current snap is 38 2023-01-15T14:29:46.127 INFO:tasks.rados.rados.0.smithi090.stdout:423: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2023-01-15T14:29:46.127 INFO:tasks.rados.rados.0.smithi090.stdout:423: writing smithi09025264-13 from 591646 to 1062087 tid 1 2023-01-15T14:29:46.127 INFO:tasks.rados.rados.0.smithi090.stdout:423: writing smithi09025264-13 from 1566383 to 2162100 tid 2 2023-01-15T14:29:46.127 INFO:tasks.rados.rados.0.smithi090.stdout:423: writing smithi09025264-13 from 2888822 to 2991646 tid 3 2023-01-15T14:29:46.127 INFO:tasks.rados.rados.0.smithi090.stdout:424: snap_remove snap 37 2023-01-15T14:29:46.128 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:46.129 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:29:46.129 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:29:46.129 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:46.144 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:29:46.147 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:46.121+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:46.229 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:46.221+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:46.492 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:46.492 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:29:46.492 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:29:46.492 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:29:46.492 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:46.493 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:29:46.493 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:46.494 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:29:46.495 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2023-01-15T14:29:45.770485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161552227,"num_objects":74,"num_object_clones":29,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":4,"num_read":980,"num_read_kb":806083,"num_write":873,"num_write_kb":280503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":123927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1596,"ondisk_log_size":1596,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":6284924,"kb_used_data":649674824,"kb_used_omap":7546,"kb_used_meta":0,"kb_avail":649610736,"statfs":{"total":671637155840,"available":665201393664,"internally_reserved":65626112,"allocated":665267019776,"data_stored":665267019776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7727211,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":54,"commit_latency_ns":0,"apply_latency_ns":54000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3101739,"num_objects":-3,"num_object_clones":-3,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":-3,"num_read":46,"num_read_kb":51177,"num_write":44,"num_write_kb":15587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":32399361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.640356"},"pg_stats":[{"pgid":"3.f","version":"98'91","reported_seq":329,"reported_epoch":98,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:44.850312+0000","last_change":"2023-01-15T14:29:44.753271+0000","last_active":"2023-01-15T14:29:44.850312+0000","last_peered":"2023-01-15T14:29:44.850312+0000","last_clean":"2023-01-15T14:29:44.850312+0000","last_became_active":"2023-01-15T14:28:23.846063+0000","last_became_peered":"2023-01-15T14:28:23.846063+0000","last_unstale":"2023-01-15T14:29:44.850312+0000","last_undegraded":"2023-01-15T14:29:44.850312+0000","last_fullsized":"2023-01-15T14:29:44.850312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:52.368","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.22042917200000001,"stat_sum":{"num_bytes":16222355,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":48255,"num_write":50,"num_write_kb":19860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.e","version":"73'25","reported_seq":266,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-01-15T14:29:44.752651+0000","last_change":"2023-01-15T14:29:44.753307+0000","last_active":"2023-01-15T14:29:44.752651+0000","last_peered":"2023-01-15T14:29:44.752651+0000","last_clean":"2023-01-15T14:29:44.752651+0000","last_became_active":"2023-01-15T14:28:23.846858+0000","last_became_peered":"2023-01-15T14:28:23.846858+0000","last_unstale":"2023-01-15T14:29:44.752651+0000","last_undegraded":"2023-01-15T14:29:44.752651+0000","last_fullsized":"2023-01-15T14:29:44.752651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099278800000000009,"stat_sum":{"num_bytes":2621859,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":23058,"num_write":6,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.d","version":"98'111","reported_seq":382,"reported_epoch":98,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:44.852763+0000","last_change":"2023-01-15T14:29:44.852763+0000","last_active":"2023-01-15T14:29:44.852763+0000","last_peered":"2023-01-15T14:29:44.852763+0000","last_clean":"2023-01-15T14:29:44.852763+0000","last_became_active":"2023-01-15T14:28:54.932913+0000","last_became_peered":"2023-01-15T14:28:54.932913+0000","last_unstale":"2023-01-15T14:29:44.852763+0000","last_undegraded":"2023-01-15T14:29:44.852763+0000","last_fullsized":"2023-01-15T14:29:44.852763+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055415399999999995,"stat_sum":{"num_bytes":11210103,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":78,"num_read_kb":87025,"num_write":47,"num_write_kb":18091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.c","version":"94'45","reported_seq":231,"reported_epoch":98,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:44.748789+0000","last_change":"2023-01-15T14:29:44.748789+0000","last_active":"2023-01-15T14:29:44.748789+0000","last_peered":"2023-01-15T14:29:44.748789+0000","last_clean":"2023-01-15T14:29:44.748789+0000","last_became_active":"2023-01-15T14:28:54.833568+0000","last_became_peered":"2023-01-15T14:28:54.833568+0000","last_unstale":"2023-01-15T14:29:44.748789+0000","last_undegraded":"2023-01-15T14:29:44.748789+0000","last_fullsized":"2023-01-15T14:29:44.748789+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:49.726","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.18326451899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":20,"num_write_kb":5044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2319333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.b","version":"86'30","reported_seq":264,"reported_epoch":98,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:44.748645+0000","last_change":"2023-01-15T14:29:44.748645+0000","last_active":"2023-01-15T14:29:44.748645+0000","last_peered":"2023-01-15T14:29:44.748645+0000","last_clean":"2023-01-15T14:29:44.748645+0000","last_became_active":"2023-01-15T14:28:23.934183+0000","last_became_peered":"2023-01-15T14:28:23.934183+0000","last_unstale":"2023-01-15T14:29:44.748645+0000","last_undegraded":"2023-01-15T14:29:44.748645+0000","last_fullsized":"2023-01-15T14:29:44.748645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:45.586","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.18360773699999999,"stat_sum":{"num_bytes":2218524,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":15,"num_read_kb":13386,"num_write":12,"num_write_kb":5195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.a","version":"91'122","reported_seq":350,"reported_epoch":95,"state":"activating","last_fresh":"2023-01-15T14:29:41.135817+0000","last_change":"2023-01-15T14:29:41.134426+0000","last_active":"2023-01-15T14:29:41.133956+0000","last_peered":"2023-01-15T14:29:39.788425+0000","last_clean":"2023-01-15T14:29:27.218124+0000","last_became_active":"2023-01-15T14:29:37.564567+0000","last_became_peered":"2023-01-15T14:29:37.564567+0000","last_unstale":"2023-01-15T14:29:41.135817+0000","last_undegraded":"2023-01-15T14:29:41.135817+0000","last_fullsized":"2023-01-15T14:29:41.135817+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041957749999999997,"stat_sum":{"num_bytes":14852985,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":42,"num_read_kb":24511,"num_write":64,"num_write_kb":31038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":36194185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":["6","1","3","4"],"object_location_counts":[{"shards":"1,4,6","objects":6}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"3.9","version":"94'125","reported_seq":356,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-01-15T14:29:40.726995+0000","last_change":"2023-01-15T14:29:40.726995+0000","last_active":"2023-01-15T14:29:40.726995+0000","last_peered":"2023-01-15T14:29:40.726995+0000","last_clean":"2023-01-15T14:29:40.726995+0000","last_became_active":"2023-01-15T14:28:24.109843+0000","last_became_peered":"2023-01-15T14:28:24.109843+0000","last_unstale":"2023-01-15T14:29:40.726995+0000","last_undegraded":"2023-01-15T14:29:40.726995+0000","last_fullsized":"2023-01-15T14:29:40.726995+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.60682911399999995,"stat_sum":{"num_bytes":12587393,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":72,"num_read_kb":70459,"num_write":58,"num_write_kb":23409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2d","length":"1"},{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"3.8","version":"97'158","reported_seq":435,"reported_epoch":98,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:44.847546+0000","last_change":"2023-01-15T14:29:44.847546+0000","last_active":"2023-01-15T14:29:44.847546+0000","last_peered":"2023-01-15T14:29:44.847546+0000","last_clean":"2023-01-15T14:29:44.847546+0000","last_became_active":"2023-01-15T14:28:47.937231+0000","last_became_peered":"2023-01-15T14:28:47.937231+0000","last_unstale":"2023-01-15T14:29:44.847546+0000","last_undegraded":"2023-01-15T14:29:44.847546+0000","last_fullsized":"2023-01-15T14:29:44.847546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'121","last_scrub_stamp":"2023-01-15T14:29:31.589753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.589753+0000","objects_scrubbed":8,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:54.442","scrub_duration":0.28338864200000002,"objects_trimmed":0,"snaptrim_duration":1.1745734640000001,"stat_sum":{"num_bytes":17647284,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":73609,"num_write":75,"num_write_kb":23115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":294,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-01-15T14:29:42.877524+0000","last_change":"2023-01-15T14:29:39.317908+0000","last_active":"2023-01-15T14:29:42.877524+0000","last_peered":"2023-01-15T14:29:42.877524+0000","last_clean":"2023-01-15T14:29:42.877524+0000","last_became_active":"2023-01-15T14:29:39.317067+0000","last_became_peered":"2023-01-15T14:29:39.317067+0000","last_unstale":"2023-01-15T14:29:42.877524+0000","last_undegraded":"2023-01-15T14:29:42.877524+0000","last_fullsized":"2023-01-15T14:29:42.877524+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":58,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-01-15T14:29:39.210557+0000","last_change":"2023-01-15T14:28:54.829192+0000","last_active":"2023-01-15T14:29:39.210557+0000","last_peered":"2023-01-15T14:29:39.210557+0000","last_clean":"2023-01-15T14:29:39.210557+0000","last_became_active":"2023-01-15T14:28:54.828857+0000","last_became_peered":"2023-01-15T14:28:54.828857+0000","last_unstale":"2023-01-15T14:29:39.210557+0000","last_undegraded":"2023-01-15T14:29:39.210557+0000","last_fullsized":"2023-01-15T14:29:39.210557+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'82","reported_seq":352,"reported_epoch":98,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:44.747862+0000","last_change":"2023-01-15T14:29:44.747862+0000","last_active":"2023-01-15T14:29:44.747862+0000","last_peered":"2023-01-15T14:29:44.747862+0000","last_clean":"2023-01-15T14:29:44.747862+0000","last_became_active":"2023-01-15T14:28:55.217912+0000","last_became_peered":"2023-01-15T14:28:55.217912+0000","last_unstale":"2023-01-15T14:29:44.747862+0000","last_undegraded":"2023-01-15T14:29:44.747862+0000","last_fullsized":"2023-01-15T14:29:44.747862+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.18272143800000001,"stat_sum":{"num_bytes":7663959,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":59,"num_read_kb":56238,"num_write":42,"num_write_kb":17290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10821306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":99,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-01-15T14:29:42.977616+0000","last_change":"2023-01-15T14:29:31.583684+0000","last_active":"2023-01-15T14:29:42.977616+0000","last_peered":"2023-01-15T14:29:42.977616+0000","last_clean":"2023-01-15T14:29:42.977616+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:29:42.977616+0000","last_undegraded":"2023-01-15T14:29:42.977616+0000","last_fullsized":"2023-01-15T14:29:42.977616+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:31.583599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.583599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:44.691","scrub_duration":0.22681161699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"98'99","reported_seq":358,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-01-15T14:29:45.355002+0000","last_change":"2023-01-15T14:29:45.355796+0000","last_active":"2023-01-15T14:29:45.355002+0000","last_peered":"2023-01-15T14:29:45.355002+0000","last_clean":"2023-01-15T14:29:45.355002+0000","last_became_active":"2023-01-15T14:28:23.842545+0000","last_became_peered":"2023-01-15T14:28:23.842545+0000","last_unstale":"2023-01-15T14:29:45.355002+0000","last_undegraded":"2023-01-15T14:29:45.355002+0000","last_fullsized":"2023-01-15T14:29:45.355002+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'81","last_scrub_stamp":"2023-01-15T14:29:32.795334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:32.795334+0000","objects_scrubbed":4,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:33.824","scrub_duration":0.40728601599999997,"objects_trimmed":0,"snaptrim_duration":0.50506554199999998,"stat_sum":{"num_bytes":12510745,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":57,"num_read_kb":49747,"num_write":51,"num_write_kb":16252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":85,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-01-15T14:29:39.110325+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:29:39.110325+0000","last_peered":"2023-01-15T14:29:39.110325+0000","last_clean":"2023-01-15T14:29:39.110325+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:29:39.110325+0000","last_undegraded":"2023-01-15T14:29:39.110325+0000","last_fullsized":"2023-01-15T14:29:39.110325+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"94'170","reported_seq":488,"reported_epoch":95,"state":"peering","last_fresh":"2023-01-15T14:29:41.132639+0000","last_change":"2023-01-15T14:29:41.132785+0000","last_active":"2023-01-15T14:29:41.132615+0000","last_peered":"2023-01-15T14:29:40.641307+0000","last_clean":"2023-01-15T14:29:34.622002+0000","last_became_active":"2023-01-15T14:29:37.569861+0000","last_became_peered":"2023-01-15T14:29:37.569861+0000","last_unstale":"2023-01-15T14:29:41.132639+0000","last_undegraded":"2023-01-15T14:29:41.132639+0000","last_fullsized":"2023-01-15T14:29:41.132639+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.80652508000000001,"stat_sum":{"num_bytes":15395080,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":99,"num_read_kb":95371,"num_write":72,"num_write_kb":37255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":34269143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,6],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-01-15T14:29:39.210582+0000","last_change":"2023-01-15T14:29:14.703116+0000","last_active":"2023-01-15T14:29:39.210582+0000","last_peered":"2023-01-15T14:29:39.210582+0000","last_clean":"2023-01-15T14:29:39.210582+0000","last_became_active":"2023-01-15T14:29:14.050302+0000","last_became_peered":"2023-01-15T14:29:14.050302+0000","last_unstale":"2023-01-15T14:29:39.210582+0000","last_undegraded":"2023-01-15T14:29:39.210582+0000","last_fullsized":"2023-01-15T14:29:39.210582+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:14.703035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:14.703035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:18.827","scrub_duration":0.44938580500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"94'177","reported_seq":429,"reported_epoch":94,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:41.128686+0000","last_change":"2023-01-15T14:29:40.120839+0000","last_active":"2023-01-15T14:29:41.128686+0000","last_peered":"2023-01-15T14:29:41.128686+0000","last_clean":"2023-01-15T14:29:41.128686+0000","last_became_active":"2023-01-15T14:28:23.946074+0000","last_became_peered":"2023-01-15T14:28:23.946074+0000","last_unstale":"2023-01-15T14:29:41.128686+0000","last_undegraded":"2023-01-15T14:29:41.128686+0000","last_fullsized":"2023-01-15T14:29:41.128686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.72090340900000005,"stat_sum":{"num_bytes":17357301,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":143,"num_read_kb":140054,"num_write":97,"num_write_kb":27695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"3.5","version":"97'45","reported_seq":263,"reported_epoch":98,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:44.748115+0000","last_change":"2023-01-15T14:29:44.748115+0000","last_active":"2023-01-15T14:29:44.748115+0000","last_peered":"2023-01-15T14:29:44.748115+0000","last_clean":"2023-01-15T14:29:44.748115+0000","last_became_active":"2023-01-15T14:28:54.831872+0000","last_became_peered":"2023-01-15T14:28:54.831872+0000","last_unstale":"2023-01-15T14:29:44.748115+0000","last_undegraded":"2023-01-15T14:29:44.748115+0000","last_fullsized":"2023-01-15T14:29:44.748115+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:46.735","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.27521325899999999,"stat_sum":{"num_bytes":7425855,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":25834,"num_write":19,"num_write_kb":11085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4252105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-01-15T14:29:40.121833+0000","last_change":"2023-01-15T14:29:29.346425+0000","last_active":"2023-01-15T14:29:40.121833+0000","last_peered":"2023-01-15T14:29:40.121833+0000","last_clean":"2023-01-15T14:29:40.121833+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:29:40.121833+0000","last_undegraded":"2023-01-15T14:29:40.121833+0000","last_fullsized":"2023-01-15T14:29:40.121833+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:52.809","scrub_duration":0.39286881699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"94'91","reported_seq":357,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-01-15T14:29:41.329363+0000","last_change":"2023-01-15T14:29:41.430001+0000","last_active":"2023-01-15T14:29:41.329363+0000","last_peered":"2023-01-15T14:29:41.329363+0000","last_clean":"2023-01-15T14:29:41.329363+0000","last_became_active":"2023-01-15T14:28:54.825996+0000","last_became_peered":"2023-01-15T14:28:54.825996+0000","last_unstale":"2023-01-15T14:29:41.329363+0000","last_undegraded":"2023-01-15T14:29:41.329363+0000","last_fullsized":"2023-01-15T14:29:41.329363+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.440","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.29873402300000002,"stat_sum":{"num_bytes":16795632,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":82,"num_read_kb":81293,"num_write":53,"num_write_kb":19128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12100908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"1"},{"start":"31","length":"1"},{"start":"35","length":"2"},{"start":"3a","length":"1"},{"start":"3e","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":99,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-01-15T14:29:42.877904+0000","last_change":"2023-01-15T14:29:25.664703+0000","last_active":"2023-01-15T14:29:42.877904+0000","last_peered":"2023-01-15T14:29:42.877904+0000","last_clean":"2023-01-15T14:29:42.877904+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:29:42.877904+0000","last_undegraded":"2023-01-15T14:29:42.877904+0000","last_fullsized":"2023-01-15T14:29:42.877904+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:25.664616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:25.664616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:29.565","scrub_duration":0.40398350300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"98'60","reported_seq":290,"reported_epoch":98,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:44.748314+0000","last_change":"2023-01-15T14:29:44.748314+0000","last_active":"2023-01-15T14:29:44.748314+0000","last_peered":"2023-01-15T14:29:44.748314+0000","last_clean":"2023-01-15T14:29:44.748314+0000","last_became_active":"2023-01-15T14:28:23.741266+0000","last_became_peered":"2023-01-15T14:28:23.741266+0000","last_unstale":"2023-01-15T14:29:44.748314+0000","last_undegraded":"2023-01-15T14:29:44.748314+0000","last_fullsized":"2023-01-15T14:29:44.748314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.37563934599999999,"stat_sum":{"num_bytes":2621859,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":4,"num_write":22,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":106,"reported_epoch":94,"state":"active+clean","last_fresh":"2023-01-15T14:29:40.121152+0000","last_change":"2023-01-15T14:29:36.100082+0000","last_active":"2023-01-15T14:29:40.121152+0000","last_peered":"2023-01-15T14:29:40.121152+0000","last_clean":"2023-01-15T14:29:40.121152+0000","last_became_active":"2023-01-15T14:29:36.099234+0000","last_became_peered":"2023-01-15T14:29:36.099234+0000","last_unstale":"2023-01-15T14:29:40.121152+0000","last_undegraded":"2023-01-15T14:29:40.121152+0000","last_fullsized":"2023-01-15T14:29:40.121152+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"98'48","reported_seq":297,"reported_epoch":98,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:44.751583+0000","last_change":"2023-01-15T14:29:44.751583+0000","last_active":"2023-01-15T14:29:44.751583+0000","last_peered":"2023-01-15T14:29:44.751583+0000","last_clean":"2023-01-15T14:29:44.751583+0000","last_became_active":"2023-01-15T14:28:23.841189+0000","last_became_peered":"2023-01-15T14:28:23.841189+0000","last_unstale":"2023-01-15T14:29:44.751583+0000","last_undegraded":"2023-01-15T14:29:44.751583+0000","last_fullsized":"2023-01-15T14:29:44.751583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'44","last_scrub_stamp":"2023-01-15T14:29:41.838167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:41.838167+0000","objects_scrubbed":2,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:01.701","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.001156884,"stat_sum":{"num_bytes":3830925,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":17136,"num_write":26,"num_write_kb":9497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":86,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-01-15T14:29:39.110803+0000","last_change":"2023-01-15T14:28:12.896363+0000","last_active":"2023-01-15T14:29:39.110803+0000","last_peered":"2023-01-15T14:29:39.110803+0000","last_clean":"2023-01-15T14:29:39.110803+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:29:39.110803+0000","last_undegraded":"2023-01-15T14:29:39.110803+0000","last_fullsized":"2023-01-15T14:29:39.110803+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:41.332","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":160961859,"num_objects":72,"num_object_clones":29,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":4,"num_read":864,"num_read_kb":805983,"num_write":714,"num_write_kb":278087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":122746975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1479,"ondisk_log_size":1479,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":864404,"kb_used_data":92841020,"kb_used_omap":848,"kb_used_meta":0,"kb_avail":92834976,"statfs":{"total":95948165120,"available":95063015424,"internally_reserved":6189056,"allocated":95069204480,"data_stored":95069204480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":868857,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.0700000000000003,"5min":4.0700000000000003,"15min":4.0700000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":75.893000000000001,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":9.2089999999999996,"5min":9.2089999999999996,"15min":9.2089999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":71.308999999999997,"5min":71.308999999999997,"15min":71.308999999999997},"last":0.77600000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":75.959999999999994,"5min":75.959999999999994,"15min":75.959999999999994},"last":1.056},{"interface":"front","average":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":6.2800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":68.605000000000004,"5min":68.605000000000004,"15min":68.605000000000004},"last":0.96499999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.7910000000000004,"5min":4.7910000000000004,"15min":4.7910000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":94.733999999999995,"5min":94.733999999999995,"15min":94.733999999999995},"last":1.0760000000000001},{"interface":"front","average":{"1min":4.1779999999999999,"5min":4.1779999999999999,"15min":4.1779999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":75.998999999999995},"last":0.80000000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.7770000000000001,"5min":3.7770000000000001,"15min":3.7770000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":76.034999999999997,"5min":76.034999999999997,"15min":76.034999999999997},"last":0.98299999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":9.4160000000000004,"5min":9.4160000000000004,"15min":9.4160000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":76.658000000000001,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":3.5129999999999999,"5min":3.5129999999999999,"15min":3.5129999999999999},"last":82.840000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.327,"5min":6.327,"15min":6.327},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":68.679000000000002,"5min":68.679000000000002,"15min":68.679000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":4.9450000000000003,"5min":4.9450000000000003,"15min":4.9450000000000003},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":95.430000000000007,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.93400000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":8.5239999999999991,"5min":8.5239999999999991,"15min":8.5239999999999991},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":68.635999999999996,"5min":68.635999999999996,"15min":68.635999999999996},"last":1.008},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.6520000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":1.0149999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":924428,"kb_used_data":92807964,"kb_used_omap":1165,"kb_used_meta":0,"kb_avail":92774952,"statfs":{"total":95948165120,"available":95001550848,"internally_reserved":33804288,"allocated":95035355136,"data_stored":95035355136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1193042,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.53000000000000003}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":67.510000000000005},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.60999999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":67.337000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.0590000000000002,"5min":6.0590000000000002,"15min":6.0590000000000002},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":81.686999999999998,"5min":81.686999999999998,"15min":81.686999999999998},"last":67.418999999999997},{"interface":"front","average":{"1min":11.455,"5min":11.455,"15min":11.455},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":255.24299999999999,"5min":255.24299999999999,"15min":255.24299999999999},"last":0.65200000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":6.6680000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":138.197,"5min":138.197,"15min":138.197},"last":17.059999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":67.469999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.76600000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.2800000000000002,"5min":4.2800000000000002,"15min":4.2800000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":78.888000000000005,"5min":78.888000000000005,"15min":78.888000000000005},"last":0.74099999999999999},{"interface":"front","average":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":47.332000000000001,"5min":47.332000000000001,"15min":47.332000000000001},"last":121.297}]}]},{"osd":5,"up_from":0,"seq":55834574870,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":904144,"kb_used_data":92800192,"kb_used_omap":1177,"kb_used_meta":0,"kb_avail":92795236,"statfs":{"total":95948165120,"available":95022321664,"internally_reserved":5074944,"allocated":95027396608,"data_stored":95027396608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1205866,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.915,"5min":4.915,"15min":4.915},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":86.944999999999993,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.73799999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":8.5939999999999994,"5min":8.5939999999999994,"15min":8.5939999999999994},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":99.960999999999999,"5min":99.960999999999999,"15min":99.960999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.54400000000000004}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":99.884,"5min":99.884,"15min":99.884},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.75800000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":12.1,"5min":12.1,"15min":12.1},"last":0.64000000000000001},{"interface":"front","average":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":6.1509999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":99.921000000000006,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.79200000000000004}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.83199999999999996},{"interface":"front","average":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":30.971,"5min":30.971,"15min":30.971},"last":1.087}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.673,"5min":5.673,"15min":5.673},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":99.780000000000001,"5min":99.780000000000001,"15min":99.780000000000001},"last":85.721999999999994},{"interface":"front","average":{"1min":5.1630000000000003,"5min":5.1630000000000003,"15min":5.1630000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":88.186999999999998,"5min":88.186999999999998,"15min":88.186999999999998},"last":0.90600000000000003}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.128,"5min":13.128,"15min":13.128},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":179.73500000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":11.239000000000001,"5min":11.239000000000001,"15min":11.239000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":99.843000000000004,"5min":99.843000000000004,"15min":99.843000000000004},"last":0.879}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":886416,"kb_used_data":92814288,"kb_used_omap":1205,"kb_used_meta":0,"kb_avail":92812964,"statfs":{"total":95948165120,"available":95040475136,"internally_reserved":1355776,"allocated":95041830912,"data_stored":95041830912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1234410,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":16,"commit_latency_ns":0,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.8479999999999999,"5min":4.8479999999999999,"15min":4.8479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":93.632999999999996,"5min":93.632999999999996,"15min":93.632999999999996},"last":0.436},{"interface":"front","average":{"1min":7.6420000000000003,"5min":7.6420000000000003,"15min":7.6420000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":93.674000000000007,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.46200000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.532,"5min":13.532,"15min":13.532},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":159.256,"5min":159.256,"15min":159.256},"last":86.067999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.59299999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.3490000000000002,"5min":3.3490000000000002,"15min":3.3490000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":59.896999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":31.233000000000001,"5min":31.233000000000001,"15min":31.233000000000001},"last":0.51800000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":6.5620000000000003,"5min":6.5620000000000003,"15min":6.5620000000000003},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":66.311999999999998,"5min":66.311999999999998,"15min":66.311999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":13.874000000000001,"5min":13.874000000000001,"15min":13.874000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":159.47200000000001,"5min":159.47200000000001,"15min":159.47200000000001},"last":85.388999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":13.903,"15min":13.903},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":159.43000000000001,"5min":159.43000000000001,"15min":159.43000000000001},"last":85.543999999999997},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.66200000000000003}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":60.098999999999997,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.71899999999999997}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.0359999999999996,"5min":4.0359999999999996,"15min":4.0359999999999996},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":60.165999999999997,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.69099999999999995},{"interface":"front","average":{"1min":18.09,"5min":18.09,"15min":18.09},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":159.512,"5min":159.512,"15min":159.512},"last":85.484999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":913812,"kb_used_data":92792012,"kb_used_omap":813,"kb_used_meta":0,"kb_avail":92785568,"statfs":{"total":95948165120,"available":95012421632,"internally_reserved":6598656,"allocated":95019020288,"data_stored":95019020288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":832621,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":10,"commit_latency_ns":0,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.48899999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.5979999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":3.3559999999999999,"5min":3.3559999999999999,"15min":3.3559999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":47.518000000000001,"5min":47.518000000000001,"15min":47.518000000000001},"last":0.63}]},{"osd":3,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":10.401999999999999,"5min":10.401999999999999,"15min":10.401999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":183.53,"5min":183.53,"15min":183.53},"last":5.7160000000000002},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.51900000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.6499999999999999,"15min":3.6499999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":53.139000000000003,"5min":53.139000000000003,"15min":53.139000000000003},"last":1.0329999999999999},{"interface":"front","average":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":3.8090000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":53.186999999999998,"5min":53.186999999999998,"15min":53.186999999999998},"last":0.81000000000000005}]},{"osd":5,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":1}]},{"osd":6,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":0.95399999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":37.124000000000002,"5min":37.124000000000002,"15min":37.124000000000002},"last":0.97799999999999998},{"interface":"front","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":30.445,"5min":30.445,"15min":30.445},"last":1.018}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":898692,"kb_used_data":92806852,"kb_used_omap":1081,"kb_used_meta":0,"kb_avail":92800688,"statfs":{"total":95948165120,"available":95027904512,"internally_reserved":6311936,"allocated":95034216448,"data_stored":95034216448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1107205,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":7,"commit_latency_ns":0,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":3.8929999999999998,"15min":3.8929999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":46.591000000000001,"5min":46.591000000000001,"15min":46.591000000000001},"last":0.439},{"interface":"front","average":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":50.616,"5min":50.616,"15min":50.616},"last":0.57199999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":46.747999999999998,"5min":46.747999999999998,"15min":46.747999999999998},"last":0.50700000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":19.835999999999999,"5min":19.835999999999999,"15min":19.835999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":282.01400000000001,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":7.2450000000000001,"5min":7.2450000000000001,"15min":7.2450000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":53.145000000000003,"5min":53.145000000000003,"15min":53.145000000000003},"last":0.47499999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.7279999999999998,"5min":4.7279999999999998,"15min":4.7279999999999998},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":50.887999999999998,"5min":50.887999999999998,"15min":50.887999999999998},"last":0.91000000000000003},{"interface":"front","average":{"1min":3.605,"5min":3.605,"15min":3.605},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":52.512,"5min":52.512,"15min":52.512},"last":0.68600000000000005}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":4.4379999999999997},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":46.905999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.64400000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.0019999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":50.844999999999999,"5min":50.844999999999999,"15min":50.844999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.755}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":52.595999999999997,"5min":52.595999999999997,"15min":52.595999999999997},"last":0.85799999999999998},{"interface":"front","average":{"1min":4.75,"5min":4.75,"15min":4.75},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":50.804000000000002,"5min":50.804000000000002,"15min":50.804000000000002},"last":0.96299999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":893028,"kb_used_data":92812496,"kb_used_omap":1255,"kb_used_meta":0,"kb_avail":92806352,"statfs":{"total":95948165120,"available":95033704448,"internally_reserved":6291456,"allocated":95039995904,"data_stored":95039995904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285210,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.69799999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":38.192}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":16.779,"5min":16.779,"15min":16.779},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.628}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.86799999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":9.798,"5min":9.798,"15min":9.798},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":176.34399999999999,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":0.76200000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.91900000000000004}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":15.670999999999999,"5min":15.670999999999999,"15min":15.670999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":292.61099999999999,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.0549999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:29:46.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-01-15T14:29:46.498 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:46.489+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:46.591 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:46.584+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:46.668 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:46.668 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:29:46.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:29:46.669 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:29:46.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.669 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:5 3.5 3:a1c47f99:::smithi09025264-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:46.669 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:29:46.669 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.833165636000004, 2023-01-15T14:29:46.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40548699700000002, 2023-01-15T14:29:46.669 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:46.669 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:46.669 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:46.669 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:46.670 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:46.670 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 5 2023-01-15T14:29:46.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.670 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:46.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:46.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:29:46.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:46.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:46.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:29:46.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:46.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:46.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781744+0000", 2023-01-15T14:29:46.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:46.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:46.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782566+0000", 2023-01-15T14:29:46.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000821578 2023-01-15T14:29:46.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:46.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782588+0000", 2023-01-15T14:29:46.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.226e-05 2023-01-15T14:29:46.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:46.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782601+0000", 2023-01-15T14:29:46.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2998e-05 2023-01-15T14:29:46.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:46.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882850+0000", 2023-01-15T14:29:46.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100248908 2023-01-15T14:29:46.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.674 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:29:46.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882871+0000", 2023-01-15T14:29:46.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0873000000000002e-05 2023-01-15T14:29:46.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.674 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:46.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083679+0000", 2023-01-15T14:29:46.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200808083 2023-01-15T14:29:46.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:46.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083948+0000", 2023-01-15T14:29:46.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000269342 2023-01-15T14:29:46.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:29:46.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084161+0000", 2023-01-15T14:29:46.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021326900000000001 2023-01-15T14:29:46.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:46.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084655+0000", 2023-01-15T14:29:46.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049371300000000001 2023-01-15T14:29:46.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:46.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084674+0000", 2023-01-15T14:29:46.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9012999999999999e-05 2023-01-15T14:29:46.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084924+0000", 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024989899999999999 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085221+0000", 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029719600000000001 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:46.677 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186953+0000", 2023-01-15T14:29:46.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10173174 2023-01-15T14:29:46.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:46.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187026+0000", 2023-01-15T14:29:46.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2957e-05 2023-01-15T14:29:46.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:46.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187124+0000", 2023-01-15T14:29:46.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8146000000000005e-05 2023-01-15T14:29:46.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:46.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187233+0000", 2023-01-15T14:29:46.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010931599999999999 2023-01-15T14:29:46.679 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:46.679 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:46.679 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:46.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.680 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:6 3.5 3:a1c47f99:::smithi09025264-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:46.680 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:29:46.680 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.827912764000004, 2023-01-15T14:29:46.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50419954199999995, 2023-01-15T14:29:46.680 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:46.680 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:46.680 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:46.680 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:46.680 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:46.680 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 6 2023-01-15T14:29:46.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.681 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:46.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:46.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:29:46.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:46.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:46.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:29:46.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:46.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.682 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:46.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786997+0000", 2023-01-15T14:29:46.682 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:46.682 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.682 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:46.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787486+0000", 2023-01-15T14:29:46.682 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048882999999999999 2023-01-15T14:29:46.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:46.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787514+0000", 2023-01-15T14:29:46.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7591999999999999e-05 2023-01-15T14:29:46.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:46.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787526+0000", 2023-01-15T14:29:46.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1894e-05 2023-01-15T14:29:46.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:46.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882966+0000", 2023-01-15T14:29:46.684 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095440450999999996 2023-01-15T14:29:46.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:46.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882973+0000", 2023-01-15T14:29:46.684 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2570000000000001e-06 2023-01-15T14:29:46.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084809+0000", 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201835763 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084928+0000", 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118946 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:46.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187347+0000", 2023-01-15T14:29:46.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102419285 2023-01-15T14:29:46.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:46.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187450+0000", 2023-01-15T14:29:46.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010286299999999999 2023-01-15T14:29:46.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:29:46.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187735+0000", 2023-01-15T14:29:46.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028429900000000001 2023-01-15T14:29:46.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:46.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188113+0000", 2023-01-15T14:29:46.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037883200000000001 2023-01-15T14:29:46.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:46.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188138+0000", 2023-01-15T14:29:46.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4110000000000001e-05 2023-01-15T14:29:46.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:46.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188389+0000", 2023-01-15T14:29:46.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025173900000000001 2023-01-15T14:29:46.860 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.860 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:46.860 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189251+0000", 2023-01-15T14:29:46.860 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086129800000000003 2023-01-15T14:29:46.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.870 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:46.870 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290989+0000", 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101738524 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291075+0000", 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5469000000000004e-05 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291144+0000", 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:29:46.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:46.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291199+0000", 2023-01-15T14:29:46.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4413000000000002e-05 2023-01-15T14:29:46.872 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:46.872 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:46.872 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:46.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.872 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:7 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:46.872 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:29:46.873 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.827366307000005, 2023-01-15T14:29:46.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50407251399999997, 2023-01-15T14:29:46.873 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:46.873 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:46.873 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:46.873 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:46.873 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:46.873 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 7 2023-01-15T14:29:46.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.874 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:46.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:46.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:29:46.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:46.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:46.877 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:29:46.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:46.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:46.879 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787545+0000", 2023-01-15T14:29:46.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:46.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:46.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787551+0000", 2023-01-15T14:29:46.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8669999999999999e-06 2023-01-15T14:29:46.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:46.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787559+0000", 2023-01-15T14:29:46.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9780000000000005e-06 2023-01-15T14:29:46.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:46.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787564+0000", 2023-01-15T14:29:46.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5709999999999997e-06 2023-01-15T14:29:46.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:46.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883054+0000", 2023-01-15T14:29:46.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095489722999999999 2023-01-15T14:29:46.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:46.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883061+0000", 2023-01-15T14:29:46.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2760000000000003e-06 2023-01-15T14:29:46.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:46.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084987+0000", 2023-01-15T14:29:46.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20192545000000001 2023-01-15T14:29:46.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:46.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085190+0000", 2023-01-15T14:29:46.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020288899999999999 2023-01-15T14:29:46.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:46.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188374+0000", 2023-01-15T14:29:46.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103184449 2023-01-15T14:29:46.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:46.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188543+0000", 2023-01-15T14:29:46.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016925299999999999 2023-01-15T14:29:46.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:29:46.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188748+0000", 2023-01-15T14:29:46.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020473600000000001 2023-01-15T14:29:46.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:46.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188986+0000", 2023-01-15T14:29:46.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023780900000000001 2023-01-15T14:29:46.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:46.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189006+0000", 2023-01-15T14:29:46.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0604000000000001e-05 2023-01-15T14:29:46.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:46.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189084+0000", 2023-01-15T14:29:46.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7526000000000003e-05 2023-01-15T14:29:46.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:46.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189272+0000", 2023-01-15T14:29:46.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018762 2023-01-15T14:29:46.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:46.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291333+0000", 2023-01-15T14:29:46.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102061185 2023-01-15T14:29:46.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:46.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:46.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:46.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291466+0000", 2023-01-15T14:29:46.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013325699999999999 2023-01-15T14:29:46.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:47.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291535+0000", 2023-01-15T14:29:47.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9309999999999999e-05 2023-01-15T14:29:47.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.073 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.073 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.073 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291618+0000", 2023-01-15T14:29:47.073 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2773000000000002e-05 2023-01-15T14:29:47.073 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.073 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.073 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.073 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.074 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:8 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:47.074 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:29:47.074 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.827340324999994, 2023-01-15T14:29:47.074 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50442076000000002, 2023-01-15T14:29:47.074 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.074 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:47.074 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:47.074 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:47.074 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:47.074 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 8 2023-01-15T14:29:47.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.075 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.075 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:29:47.075 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.075 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:29:47.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787571+0000", 2023-01-15T14:29:47.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:47.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787575+0000", 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7890000000000001e-06 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787589+0000", 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4212999999999999e-05 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787594+0000", 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1429999999999997e-06 2023-01-15T14:29:47.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.078 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983143+0000", 2023-01-15T14:29:47.078 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.195549003 2023-01-15T14:29:47.078 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.078 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:47.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983153+0000", 2023-01-15T14:29:47.078 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4369999999999996e-06 2023-01-15T14:29:47.078 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085286+0000", 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102133064 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085401+0000", 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115176 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.079 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189116+0000", 2023-01-15T14:29:47.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103715444 2023-01-15T14:29:47.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:47.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189221+0000", 2023-01-15T14:29:47.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010438 2023-01-15T14:29:47.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291724+0000", 2023-01-15T14:29:47.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102502998 2023-01-15T14:29:47.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291824+0000", 2023-01-15T14:29:47.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010010599999999999 2023-01-15T14:29:47.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291992+0000", 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016856900000000001 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:21 3.5 3:a75958b1:::smithi09025264-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.783251410000005, 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.45851924500000002, 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.082 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:47.083 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:47.083 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:47.083 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:47.083 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 21 2023-01-15T14:29:47.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.083 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:29:47.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:29:47.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831657+0000", 2023-01-15T14:29:47.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:29:47.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832814+0000", 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011564909999999999 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832864+0000", 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0243e-05 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832903+0000", 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8704000000000001e-05 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983202+0000", 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.150299404 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983208+0000", 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0039999999999996e-06 2023-01-15T14:29:47.086 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.087 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.087 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185539+0000", 2023-01-15T14:29:47.087 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202330867 2023-01-15T14:29:47.087 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.087 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.087 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185803+0000", 2023-01-15T14:29:47.087 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026394299999999999 2023-01-15T14:29:47.087 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:29:47.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186037+0000", 2023-01-15T14:29:47.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023386300000000001 2023-01-15T14:29:47.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:47.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186637+0000", 2023-01-15T14:29:47.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060001800000000001 2023-01-15T14:29:47.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:47.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186665+0000", 2023-01-15T14:29:47.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8459999999999999e-05 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187045+0000", 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037992899999999999 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188201+0000", 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001155309 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289903+0000", 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101702601 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289985+0000", 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1617000000000001e-05 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:47.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290065+0000", 2023-01-15T14:29:47.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0276000000000002e-05 2023-01-15T14:29:47.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290180+0000", 2023-01-15T14:29:47.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011464300000000001 2023-01-15T14:29:47.093 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.093 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.093 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.094 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:22 3.5 3:a75958b1:::smithi09025264-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:47.094 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:29:47.094 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.778783722, 2023-01-15T14:29:47.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65790599599999999, 2023-01-15T14:29:47.094 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.094 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:47.094 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:47.094 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 22 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:29:47.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836126+0000", 2023-01-15T14:29:47.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:47.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836181+0000", 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4246999999999997e-05 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836212+0000", 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1843e-05 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836226+0000", 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3216e-05 2023-01-15T14:29:47.097 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.098 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.098 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.098 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186757+0000", 2023-01-15T14:29:47.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35053103899999999 2023-01-15T14:29:47.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.098 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.098 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:47.098 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186871+0000", 2023-01-15T14:29:47.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114403 2023-01-15T14:29:47.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.099 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.099 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290263+0000", 2023-01-15T14:29:47.099 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103391976 2023-01-15T14:29:47.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.099 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.099 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290413+0000", 2023-01-15T14:29:47.099 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014991200000000001 2023-01-15T14:29:47.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.100 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:29:47.100 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290627+0000", 2023-01-15T14:29:47.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021379299999999999 2023-01-15T14:29:47.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.100 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.100 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:47.100 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290861+0000", 2023-01-15T14:29:47.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023479600000000001 2023-01-15T14:29:47.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.100 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.101 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:47.101 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290888+0000", 2023-01-15T14:29:47.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6570000000000001e-05 2023-01-15T14:29:47.101 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.101 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.101 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:47.101 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290990+0000", 2023-01-15T14:29:47.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101723 2023-01-15T14:29:47.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:47.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291245+0000", 2023-01-15T14:29:47.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025543899999999999 2023-01-15T14:29:47.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392822+0000", 2023-01-15T14:29:47.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10157725500000001 2023-01-15T14:29:47.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493881+0000", 2023-01-15T14:29:47.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101058262 2023-01-15T14:29:47.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:47.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493964+0000", 2023-01-15T14:29:47.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2973000000000007e-05 2023-01-15T14:29:47.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494034+0000", 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0432999999999995e-05 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:23 3.5 3:a75958b1:::smithi09025264-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:29:47.104 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.778668593000006, 2023-01-15T14:29:47.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65796116900000001, 2023-01-15T14:29:47.105 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.105 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:47.105 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:47.105 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:47.105 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:47.105 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 23 2023-01-15T14:29:47.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.105 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.105 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:29:47.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836242+0000", 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836256+0000", 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3677e-05 2023-01-15T14:29:47.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836265+0000", 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2769999999999994e-06 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936398+0000", 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100132559 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289431+0000", 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35303322399999998 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289587+0000", 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000155873 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292052+0000", 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0024647919999999999 2023-01-15T14:29:47.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292180+0000", 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012810999999999999 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292373+0000", 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019258599999999999 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:47.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292597+0000", 2023-01-15T14:29:47.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022465500000000001 2023-01-15T14:29:47.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:47.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292618+0000", 2023-01-15T14:29:47.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0319999999999999e-05 2023-01-15T14:29:47.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:47.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292692+0000", 2023-01-15T14:29:47.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4643999999999999e-05 2023-01-15T14:29:47.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:47.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393103+0000", 2023-01-15T14:29:47.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100411156 2023-01-15T14:29:47.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493827+0000", 2023-01-15T14:29:47.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100724143 2023-01-15T14:29:47.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494085+0000", 2023-01-15T14:29:47.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000257891 2023-01-15T14:29:47.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:47.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494141+0000", 2023-01-15T14:29:47.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6132999999999999e-05 2023-01-15T14:29:47.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494204+0000", 2023-01-15T14:29:47.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2966000000000007e-05 2023-01-15T14:29:47.280 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.280 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.280 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.281 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:24 3.5 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:29:47.281 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:29:47.281 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.678487167, 2023-01-15T14:29:47.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.55808677600000001, 2023-01-15T14:29:47.281 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.281 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:47.281 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:47.281 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:47.281 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:47.281 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 24 2023-01-15T14:29:47.282 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.282 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.282 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.282 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:29:47.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.282 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.282 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.282 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:29:47.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936423+0000", 2023-01-15T14:29:47.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:47.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936433+0000", 2023-01-15T14:29:47.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0443e-05 2023-01-15T14:29:47.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.284 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936451+0000", 2023-01-15T14:29:47.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7842999999999998e-05 2023-01-15T14:29:47.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.284 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936461+0000", 2023-01-15T14:29:47.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0057e-05 2023-01-15T14:29:47.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289693+0000", 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.353231617 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289814+0000", 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120763 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392923+0000", 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10310955300000001 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393072+0000", 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000148469 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494242+0000", 2023-01-15T14:29:47.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117034499999999 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494363+0000", 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120703 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494511+0000", 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014873299999999999 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.287 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.288 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.288 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:78 3.c 3:3137f0cb:::smithi09025264-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:47.288 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:29:47.288 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.288521170999999, 2023-01-15T14:29:47.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20331801699999999, 2023-01-15T14:29:47.288 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.288 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:47.288 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:47.288 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:47.289 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:47.289 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 78 2023-01-15T14:29:47.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.289 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:29:47.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326388+0000", 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326672+0000", 2023-01-15T14:29:47.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028367900000000001 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326713+0000", 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1539999999999999e-05 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326729+0000", 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.519e-05 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326879+0000", 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015064899999999999 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327096+0000", 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021682299999999999 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:29:47.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327308+0000", 2023-01-15T14:29:47.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000211366 2023-01-15T14:29:47.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:47.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327757+0000", 2023-01-15T14:29:47.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000449009 2023-01-15T14:29:47.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:47.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327776+0000", 2023-01-15T14:29:47.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9201000000000001e-05 2023-01-15T14:29:47.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:47.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328012+0000", 2023-01-15T14:29:47.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236054 2023-01-15T14:29:47.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:47.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429534+0000", 2023-01-15T14:29:47.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10152201499999999 2023-01-15T14:29:47.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529340+0000", 2023-01-15T14:29:47.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099806249 2023-01-15T14:29:47.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.454 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.454 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.454 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529437+0000", 2023-01-15T14:29:47.454 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6949999999999998e-05 2023-01-15T14:29:47.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.454 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.454 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:47.454 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529587+0000", 2023-01-15T14:29:47.454 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015008600000000001 2023-01-15T14:29:47.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.455 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.455 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529709+0000", 2023-01-15T14:29:47.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121636 2023-01-15T14:29:47.455 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.455 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.455 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.455 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:79 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:47.456 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:29:47.456 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.288151575000001, 2023-01-15T14:29:47.456 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20348171300000001, 2023-01-15T14:29:47.456 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.456 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:47.456 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:47.456 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:47.456 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:47.456 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 79 2023-01-15T14:29:47.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.457 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:29:47.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:29:47.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326759+0000", 2023-01-15T14:29:47.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:47.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326770+0000", 2023-01-15T14:29:47.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:29:47.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326791+0000", 2023-01-15T14:29:47.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1556000000000002e-05 2023-01-15T14:29:47.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326818+0000", 2023-01-15T14:29:47.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6979999999999999e-05 2023-01-15T14:29:47.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.427993+0000", 2023-01-15T14:29:47.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117517199999999 2023-01-15T14:29:47.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428266+0000", 2023-01-15T14:29:47.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027233299999999998 2023-01-15T14:29:47.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:29:47.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428696+0000", 2023-01-15T14:29:47.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042996499999999998 2023-01-15T14:29:47.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:47.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429057+0000", 2023-01-15T14:29:47.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000360952 2023-01-15T14:29:47.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:47.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429076+0000", 2023-01-15T14:29:47.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429160+0000", 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429549+0000", 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038884599999999998 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529886+0000", 2023-01-15T14:29:47.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100336261 2023-01-15T14:29:47.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529978+0000", 2023-01-15T14:29:47.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2775999999999994e-05 2023-01-15T14:29:47.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:47.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530108+0000", 2023-01-15T14:29:47.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012958300000000001 2023-01-15T14:29:47.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.464 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530242+0000", 2023-01-15T14:29:47.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013406 2023-01-15T14:29:47.464 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.464 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.464 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.465 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:80 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:47.465 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:29:47.465 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.288013215999996, 2023-01-15T14:29:47.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20370874, 2023-01-15T14:29:47.465 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.465 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:47.465 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:47.465 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:47.465 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 80 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:29:47.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326897+0000", 2023-01-15T14:29:47.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:47.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326906+0000", 2023-01-15T14:29:47.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6929999999999996e-06 2023-01-15T14:29:47.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326931+0000", 2023-01-15T14:29:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4757e-05 2023-01-15T14:29:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326945+0000", 2023-01-15T14:29:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.402e-05 2023-01-15T14:29:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429239+0000", 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102294418 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429487+0000", 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024789300000000001 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530309+0000", 2023-01-15T14:29:47.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100821459 2023-01-15T14:29:47.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530438+0000", 2023-01-15T14:29:47.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012928700000000001 2023-01-15T14:29:47.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530607+0000", 2023-01-15T14:29:47.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169483 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:78 3.c e23/20 ondisk, result = 0)", 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 80.286121495000003, 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20094287799999999, 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:47.471 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.472 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:29:47.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.472 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:29:47.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328789+0000", 2023-01-15T14:29:47.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:47.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328799+0000", 2023-01-15T14:29:47.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8980000000000008e-06 2023-01-15T14:29:47.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328820+0000", 2023-01-15T14:29:47.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1041999999999999e-05 2023-01-15T14:29:47.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328832+0000", 2023-01-15T14:29:47.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1616e-05 2023-01-15T14:29:47.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529404+0000", 2023-01-15T14:29:47.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200572631 2023-01-15T14:29:47.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529419+0000", 2023-01-15T14:29:47.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4250000000000001e-05 2023-01-15T14:29:47.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529733+0000", 2023-01-15T14:29:47.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000314693 2023-01-15T14:29:47.623 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.623 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.624 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.624 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:201 3.5 3:a1c47f99:::smithi09025264-2:head [copy-from ver 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:47.624 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:29:47.624 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 79.208515098000007, 2023-01-15T14:29:47.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30931411399999997, 2023-01-15T14:29:47.624 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.624 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:47.624 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 201 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406394+0000", 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406408+0000", 2023-01-15T14:29:47.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4416999999999999e-05 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406446+0000", 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7339999999999998e-05 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406467+0000", 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1137000000000001e-05 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406677+0000", 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000210275 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406902+0000", 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022529000000000001 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:29:47.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.513012+0000", 2023-01-15T14:29:47.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.106109527 2023-01-15T14:29:47.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:47.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514754+0000", 2023-01-15T14:29:47.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0017419779999999999 2023-01-15T14:29:47.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:47.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514773+0000", 2023-01-15T14:29:47.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8700000000000001e-05 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.515531+0000", 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075844300000000003 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615334+0000", 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099803327999999997 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:47.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615376+0000", 2023-01-15T14:29:47.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1675999999999999e-05 2023-01-15T14:29:47.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715546+0000", 2023-01-15T14:29:47.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016992500000001 2023-01-15T14:29:47.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:47.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715626+0000", 2023-01-15T14:29:47.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0112999999999997e-05 2023-01-15T14:29:47.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715711+0000", 2023-01-15T14:29:47.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:29:47.636 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:202 3.5 3:a1c47f99:::smithi09025264-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 79.208402551999995, 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31002637300000002, 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:47.637 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:47.638 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:47.638 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 202 2023-01-15T14:29:47.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.638 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:29:47.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:29:47.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406508+0000", 2023-01-15T14:29:47.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:47.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406516+0000", 2023-01-15T14:29:47.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.413e-06 2023-01-15T14:29:47.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406534+0000", 2023-01-15T14:29:47.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8936999999999998e-05 2023-01-15T14:29:47.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406548+0000", 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3196e-05 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407271+0000", 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00072301500000000003 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.641 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407466+0000", 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000195009 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.614996+0000", 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.207530192 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615227+0000", 2023-01-15T14:29:47.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000230979 2023-01-15T14:29:47.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716282+0000", 2023-01-15T14:29:47.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101055465 2023-01-15T14:29:47.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716409+0000", 2023-01-15T14:29:47.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012653999999999999 2023-01-15T14:29:47.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716536+0000", 2023-01-15T14:29:47.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012683 2023-01-15T14:29:47.644 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.644 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:189 3.d e23/20 3:b3f111b5:::smithi09025264-48:head v 23'10, mlcod=23'6)", 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 79.163197904, 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20400475100000001, 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:29:47.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:29:47.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451712+0000", 2023-01-15T14:29:47.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:47.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452272+0000", 2023-01-15T14:29:47.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00056035899999999997 2023-01-15T14:29:47.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452310+0000", 2023-01-15T14:29:47.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8145999999999997e-05 2023-01-15T14:29:47.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452324+0000", 2023-01-15T14:29:47.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4443999999999999e-05 2023-01-15T14:29:47.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652703+0000", 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20037818199999999 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652762+0000", 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9302999999999998e-05 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:47.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653302+0000", 2023-01-15T14:29:47.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054012199999999998 2023-01-15T14:29:47.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:47.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653333+0000", 2023-01-15T14:29:47.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.112e-05 2023-01-15T14:29:47.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:47.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653667+0000", 2023-01-15T14:29:47.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033337900000000002 2023-01-15T14:29:47.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:47.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653709+0000", 2023-01-15T14:29:47.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2469999999999998e-05 2023-01-15T14:29:47.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655719+0000", 2023-01-15T14:29:47.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020096699999999999 2023-01-15T14:29:47.809 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.809 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.809 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.810 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:269 3.2 e30/20 3:47dfe935:::smithi09025264-29:head v 30'11, mlcod=23'8)", 2023-01-15T14:29:47.810 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:29:47.810 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 71.737152463000001, 2023-01-15T14:29:47.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23185540399999999, 2023-01-15T14:29:47.810 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.810 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:47.810 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:29:47.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:29:47.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:29:47.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:47.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878379+0000", 2023-01-15T14:29:47.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061979600000000004 2023-01-15T14:29:47.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878406+0000", 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978515+0000", 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100108461 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078759+0000", 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100244496 2023-01-15T14:29:47.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078796+0000", 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6758999999999998e-05 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079567+0000", 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077137199999999999 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:47.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079593+0000", 2023-01-15T14:29:47.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:29:47.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:47.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079844+0000", 2023-01-15T14:29:47.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025075899999999998 2023-01-15T14:29:47.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:47.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079910+0000", 2023-01-15T14:29:47.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6406000000000001e-05 2023-01-15T14:29:47.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.109615+0000", 2023-01-15T14:29:47.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.029704323000000001 2023-01-15T14:29:47.816 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.816 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:47.816 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:47.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.817 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:281 3.5 3:a75958b1:::smithi09025264-6:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:29:47.817 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:29:47.817 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 71.636228833999994, 2023-01-15T14:29:47.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30574310900000001, 2023-01-15T14:29:47.817 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:47.817 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:47.817 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:29:47.817 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:29:47.817 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:47.818 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 281 2023-01-15T14:29:47.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.818 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:47.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:47.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:29:47.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:47.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:29:47.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:47.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:47.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:29:47.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:47.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:47.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978686+0000", 2023-01-15T14:29:47.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4429999999999999e-06 2023-01-15T14:29:47.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:47.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978690+0000", 2023-01-15T14:29:47.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0300000000000004e-06 2023-01-15T14:29:47.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:47.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978693+0000", 2023-01-15T14:29:47.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.179e-06 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981592+0000", 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0028984140000000002 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981718+0000", 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012616600000000001 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981982+0000", 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026445199999999998 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982278+0000", 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029514100000000001 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:47.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982290+0000", 2023-01-15T14:29:47.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.271e-05 2023-01-15T14:29:47.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:47.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982343+0000", 2023-01-15T14:29:47.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2914000000000002e-05 2023-01-15T14:29:47.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:29:47.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982385+0000", 2023-01-15T14:29:47.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2114999999999999e-05 2023-01-15T14:29:47.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.181937+0000", 2023-01-15T14:29:47.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19955213099999999 2023-01-15T14:29:47.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:47.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284183+0000", 2023-01-15T14:29:47.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10224538 2023-01-15T14:29:47.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:47.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284318+0000", 2023-01-15T14:29:47.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000134843 2023-01-15T14:29:47.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:47.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:47.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:47.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284426+0000", 2023-01-15T14:29:47.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108513 2023-01-15T14:29:48.267 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:48.267 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:48.267 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:48.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.267 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:373 3.4 e34/20 3:2029fa71:::smithi09025264-39:head v 34'19, mlcod=26'16)", 2023-01-15T14:29:48.267 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:29:48.268 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 66.155246375000004, 2023-01-15T14:29:48.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20212622799999999, 2023-01-15T14:29:48.268 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:48.268 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:48.268 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:48.268 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.268 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:48.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:29:48.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:48.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.269 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:48.269 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:29:48.269 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:48.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:48.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459664+0000", 2023-01-15T14:29:48.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:48.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:48.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460099+0000", 2023-01-15T14:29:48.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043515900000000001 2023-01-15T14:29:48.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:48.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460130+0000", 2023-01-15T14:29:48.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1590000000000001e-05 2023-01-15T14:29:48.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:48.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560226+0000", 2023-01-15T14:29:48.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100095248 2023-01-15T14:29:48.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:48.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560369+0000", 2023-01-15T14:29:48.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014381499999999999 2023-01-15T14:29:48.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:48.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560404+0000", 2023-01-15T14:29:48.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4650000000000002e-05 2023-01-15T14:29:48.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:48.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560804+0000", 2023-01-15T14:29:48.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000399823 2023-01-15T14:29:48.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560827+0000", 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3612999999999999e-05 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.561101+0000", 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027355499999999998 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:48.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661710+0000", 2023-01-15T14:29:48.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100608716 2023-01-15T14:29:48.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:48.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661792+0000", 2023-01-15T14:29:48.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2006000000000001e-05 2023-01-15T14:29:48.276 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:48.276 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:48.276 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:48.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:455 3.d e41/20 3:bd982ca3:::smithi09025264-38:head v 41'24, mlcod=38'18)", 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 59.105484801999999, 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20118033699999999, 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:48.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:48.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:29:48.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:48.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:48.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509425+0000", 2023-01-15T14:29:48.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:48.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:48.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509441+0000", 2023-01-15T14:29:48.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5248e-05 2023-01-15T14:29:48.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:48.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509474+0000", 2023-01-15T14:29:48.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3531000000000001e-05 2023-01-15T14:29:48.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.282 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:48.282 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.609644+0000", 2023-01-15T14:29:48.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016994999999999 2023-01-15T14:29:48.282 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:48.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709872+0000", 2023-01-15T14:29:48.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10022840299999999 2023-01-15T14:29:48.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:48.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709906+0000", 2023-01-15T14:29:48.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3160000000000001e-05 2023-01-15T14:29:48.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:48.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710153+0000", 2023-01-15T14:29:48.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024735399999999998 2023-01-15T14:29:48.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:48.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710204+0000", 2023-01-15T14:29:48.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1075999999999997e-05 2023-01-15T14:29:48.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710280+0000", 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6202e-05 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710317+0000", 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6387000000000002e-05 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710607+0000", 2023-01-15T14:29:48.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029079699999999997 2023-01-15T14:29:48.287 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:48.287 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:48.287 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:48.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.287 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:497 3.d e44/20 3:bd982ca3:::smithi09025264-38:head v 44'31, mlcod=42'26)", 2023-01-15T14:29:48.287 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:29:48.287 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 56.840689939999997, 2023-01-15T14:29:48.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20082067100000001, 2023-01-15T14:29:48.287 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:29:48.287 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:48.288 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:29:48.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:29:48.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:29:48.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:48.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:29:48.430 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:29:48.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:29:48.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.430 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:29:48.430 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774220+0000", 2023-01-15T14:29:48.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:48.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774238+0000", 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8107000000000001e-05 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774263+0000", 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4122999999999999e-05 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774280+0000", 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7039999999999999e-05 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874060+0000", 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099780698000000001 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874130+0000", 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9852999999999997e-05 2023-01-15T14:29:48.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.433 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:48.433 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874351+0000", 2023-01-15T14:29:48.433 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022091400000000001 2023-01-15T14:29:48.433 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.433 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:48.433 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874369+0000", 2023-01-15T14:29:48.433 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7876e-05 2023-01-15T14:29:48.433 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874433+0000", 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3793000000000004e-05 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.974927+0000", 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10049393500000001 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:29:48.434 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.975043+0000", 2023-01-15T14:29:48.435 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115949 2023-01-15T14:29:48.435 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:48.435 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:48.435 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:48.435 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:29:48.435 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:29:48.435 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:48.435 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:29:48.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:29:48.437 INFO:tasks.rados.rados.0.smithi090.stdout:424: done (1 left) 2023-01-15T14:29:48.437 INFO:tasks.rados.rados.0.smithi090.stdout:425: write oid 25 current snap is 38 2023-01-15T14:29:48.437 INFO:tasks.rados.rados.0.smithi090.stdout:423: finishing write tid 1 to smithi09025264-13 2023-01-15T14:29:48.438 INFO:tasks.rados.rados.0.smithi090.stdout:425: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2023-01-15T14:29:48.438 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:29:48.438 INFO:tasks.rados.rados.0.smithi090.stdout:425: writing smithi09025264-25 from 608453 to 1154143 tid 1 2023-01-15T14:29:48.438 INFO:tasks.rados.rados.0.smithi090.stdout:425: writing smithi09025264-25 from 1624865 to 2080593 tid 2 2023-01-15T14:29:48.439 INFO:tasks.rados.rados.0.smithi090.stdout:425: writing smithi09025264-25 from 2632598 to 3008453 tid 3 2023-01-15T14:29:48.439 INFO:tasks.rados.rados.0.smithi090.stdout:426: snap_remove snap 34 2023-01-15T14:29:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:48.440 INFO:teuthology.orchestra.run.smithi090.stdout:{"fsid":"1c77fb2b-d1a7-4884-a24e-3935e16d0b6b","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":109,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":100,"num_osds":8,"num_up_osds":8,"osd_up_since":1673792884,"num_in_osds":7,"osd_in_since":1673792933,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":14},{"state_name":"active+clean+snaptrim","count":5},{"state_name":"active+clean+snaptrim_wait","count":4},{"state_name":"peering","count":1},{"state_name":"activating","count":1}],"num_pgs":25,"num_pools":3,"num_objects":77,"data_bytes":160467538,"bytes_used":6491443200,"bytes_avail":665145712640,"bytes_total":671637155840,"inactive_pgs_ratio":0.079999998211860657,"recovering_objects_per_sec":9,"recovering_bytes_per_sec":8097550,"recovering_keys_per_sec":0,"num_objects_recovered":38,"num_bytes_recovered":32399361,"num_keys_recovered":0,"read_bytes_sec":2774767,"write_bytes_sec":1894120,"read_op_per_sec":3,"write_op_per_sec":4},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2023-01-15T14:29:06.857309+0000","services":{}},"progress_events":{}} 2023-01-15T14:29:48.441 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2023-01-15T14:29:48.441 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:29:48.441 INFO:tasks.rados.rados.0.smithi090.stdout:426: done (2 left) 2023-01-15T14:29:48.441 INFO:tasks.rados.rados.0.smithi090.stdout:427: snap_remove snap 31 2023-01-15T14:29:48.442 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:48.429+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:48.537 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:48.529+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:48.595 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:48.595 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:29:48.595 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:29:48.596 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:48.596 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:29:48.596 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:48.620 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:48.750 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:29:48.750 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:29:48.915 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:29:48.915 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:29:48.915 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:29:48.915 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:48.915 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:48.915 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:48.915 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:48.915 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:48.915 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:48.916 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:29:48.916 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:48.744+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:29:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:29:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:29:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:29:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:29:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:29:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:29:48.917 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:427: done (2 left) 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:428: snap_create 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:423: finishing write tid 2 to smithi09025264-13 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:423: finishing write tid 3 to smithi09025264-13 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:423: finishing write tid 4 to smithi09025264-13 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:423: finishing write tid 5 to smithi09025264-13 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:423: oid 13 updating version 0 to 129 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:423: oid 13 updating version 129 to 130 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:423: oid 13 updating version 130 to 131 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:423: oid 13 version 131 is already newer than 128 2023-01-15T14:29:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 131 (ObjNum 178 snap 38 seq_num 178) dirty exists 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:423: left oid 13 (ObjNum 178 snap 38 seq_num 178) 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:425: finishing write tid 1 to smithi09025264-25 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:425: finishing write tid 2 to smithi09025264-25 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:425: finishing write tid 3 to smithi09025264-25 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:425: finishing write tid 4 to smithi09025264-25 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:425: finishing write tid 5 to smithi09025264-25 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:425: oid 25 updating version 0 to 136 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:425: oid 25 version 136 is already newer than 133 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:425: oid 25 version 136 is already newer than 135 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:425: oid 25 version 136 is already newer than 134 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 136 (ObjNum 179 snap 38 seq_num 179) dirty exists 2023-01-15T14:29:48.919 INFO:tasks.rados.rados.0.smithi090.stdout:425: left oid 25 (ObjNum 179 snap 38 seq_num 179) 2023-01-15T14:29:48.920 INFO:tasks.rados.rados.0.smithi090.stdout:423: done (1 left) 2023-01-15T14:29:48.920 INFO:tasks.rados.rados.0.smithi090.stdout:425: done (0 left) 2023-01-15T14:29:48.922 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:48.923 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:29:48.923 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":145,"stamp":"2023-01-15T14:29:48.745209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":153887711,"num_objects":71,"num_object_clones":27,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":4,"num_read":1025,"num_read_kb":863102,"num_write":885,"num_write_kb":282929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":123927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1645,"ondisk_log_size":1645,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":76,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":6258532,"kb_used_data":649667780,"kb_used_omap":7964,"kb_used_meta":0,"kb_avail":649637128,"statfs":{"total":671637155840,"available":665228419072,"internally_reserved":31387648,"allocated":665259806720,"data_stored":665259806720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8156025,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":59,"commit_latency_ns":0,"apply_latency_ns":59000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-6579827,"num_objects":-6,"num_object_clones":-4,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":-3,"num_read":78,"num_read_kb":97354,"num_write":38,"num_write_kb":10612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.999577"},"pg_stats":[{"pgid":"3.f","version":"98'91","reported_seq":329,"reported_epoch":98,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:44.850312+0000","last_change":"2023-01-15T14:29:44.753271+0000","last_active":"2023-01-15T14:29:44.850312+0000","last_peered":"2023-01-15T14:29:44.850312+0000","last_clean":"2023-01-15T14:29:44.850312+0000","last_became_active":"2023-01-15T14:28:23.846063+0000","last_became_peered":"2023-01-15T14:28:23.846063+0000","last_unstale":"2023-01-15T14:29:44.850312+0000","last_undegraded":"2023-01-15T14:29:44.850312+0000","last_fullsized":"2023-01-15T14:29:44.850312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:52.368","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.22042917200000001,"stat_sum":{"num_bytes":16222355,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":48255,"num_write":50,"num_write_kb":19860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.e","version":"73'25","reported_seq":266,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-01-15T14:29:44.752651+0000","last_change":"2023-01-15T14:29:44.753307+0000","last_active":"2023-01-15T14:29:44.752651+0000","last_peered":"2023-01-15T14:29:44.752651+0000","last_clean":"2023-01-15T14:29:44.752651+0000","last_became_active":"2023-01-15T14:28:23.846858+0000","last_became_peered":"2023-01-15T14:28:23.846858+0000","last_unstale":"2023-01-15T14:29:44.752651+0000","last_undegraded":"2023-01-15T14:29:44.752651+0000","last_fullsized":"2023-01-15T14:29:44.752651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099278800000000009,"stat_sum":{"num_bytes":2621859,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":23058,"num_write":6,"num_write_kb":6873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.d","version":"98'111","reported_seq":382,"reported_epoch":98,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:44.852763+0000","last_change":"2023-01-15T14:29:44.852763+0000","last_active":"2023-01-15T14:29:44.852763+0000","last_peered":"2023-01-15T14:29:44.852763+0000","last_clean":"2023-01-15T14:29:44.852763+0000","last_became_active":"2023-01-15T14:28:54.932913+0000","last_became_peered":"2023-01-15T14:28:54.932913+0000","last_unstale":"2023-01-15T14:29:44.852763+0000","last_undegraded":"2023-01-15T14:29:44.852763+0000","last_fullsized":"2023-01-15T14:29:44.852763+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055415399999999995,"stat_sum":{"num_bytes":11210103,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":78,"num_read_kb":87025,"num_write":47,"num_write_kb":18091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.c","version":"94'45","reported_seq":231,"reported_epoch":98,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:44.748789+0000","last_change":"2023-01-15T14:29:44.748789+0000","last_active":"2023-01-15T14:29:44.748789+0000","last_peered":"2023-01-15T14:29:44.748789+0000","last_clean":"2023-01-15T14:29:44.748789+0000","last_became_active":"2023-01-15T14:28:54.833568+0000","last_became_peered":"2023-01-15T14:28:54.833568+0000","last_unstale":"2023-01-15T14:29:44.748789+0000","last_undegraded":"2023-01-15T14:29:44.748789+0000","last_fullsized":"2023-01-15T14:29:44.748789+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:49.726","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.18326451899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":20,"num_write_kb":5044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2319333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.b","version":"86'30","reported_seq":264,"reported_epoch":98,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:44.748645+0000","last_change":"2023-01-15T14:29:44.748645+0000","last_active":"2023-01-15T14:29:44.748645+0000","last_peered":"2023-01-15T14:29:44.748645+0000","last_clean":"2023-01-15T14:29:44.748645+0000","last_became_active":"2023-01-15T14:28:23.934183+0000","last_became_peered":"2023-01-15T14:28:23.934183+0000","last_unstale":"2023-01-15T14:29:44.748645+0000","last_undegraded":"2023-01-15T14:29:44.748645+0000","last_fullsized":"2023-01-15T14:29:44.748645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:45.586","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.18360773699999999,"stat_sum":{"num_bytes":2218524,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":15,"num_read_kb":13386,"num_write":12,"num_write_kb":5195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.a","version":"99'131","reported_seq":396,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-01-15T14:29:46.279296+0000","last_change":"2023-01-15T14:29:44.753046+0000","last_active":"2023-01-15T14:29:46.279296+0000","last_peered":"2023-01-15T14:29:46.279296+0000","last_clean":"2023-01-15T14:29:46.279296+0000","last_became_active":"2023-01-15T14:29:41.237634+0000","last_became_peered":"2023-01-15T14:29:41.237634+0000","last_unstale":"2023-01-15T14:29:46.279296+0000","last_undegraded":"2023-01-15T14:29:46.279296+0000","last_fullsized":"2023-01-15T14:29:46.279296+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:30:41.499","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020184439999999999,"stat_sum":{"num_bytes":10612468,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":51,"num_read_kb":33248,"num_write":69,"num_write_kb":32181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":36194185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.9","version":"98'131","reported_seq":384,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-01-15T14:29:45.554625+0000","last_change":"2023-01-15T14:29:45.554625+0000","last_active":"2023-01-15T14:29:45.554625+0000","last_peered":"2023-01-15T14:29:45.554625+0000","last_clean":"2023-01-15T14:29:45.554625+0000","last_became_active":"2023-01-15T14:28:24.109843+0000","last_became_peered":"2023-01-15T14:28:24.109843+0000","last_unstale":"2023-01-15T14:29:45.554625+0000","last_undegraded":"2023-01-15T14:29:45.554625+0000","last_fullsized":"2023-01-15T14:29:45.554625+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70471951200000005,"stat_sum":{"num_bytes":12587393,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":90,"num_read_kb":99493,"num_write":58,"num_write_kb":23409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.8","version":"97'158","reported_seq":435,"reported_epoch":98,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:44.847546+0000","last_change":"2023-01-15T14:29:44.847546+0000","last_active":"2023-01-15T14:29:44.847546+0000","last_peered":"2023-01-15T14:29:44.847546+0000","last_clean":"2023-01-15T14:29:44.847546+0000","last_became_active":"2023-01-15T14:28:47.937231+0000","last_became_peered":"2023-01-15T14:28:47.937231+0000","last_unstale":"2023-01-15T14:29:44.847546+0000","last_undegraded":"2023-01-15T14:29:44.847546+0000","last_fullsized":"2023-01-15T14:29:44.847546+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'121","last_scrub_stamp":"2023-01-15T14:29:31.589753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.589753+0000","objects_scrubbed":8,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:54.442","scrub_duration":0.28338864200000002,"objects_trimmed":0,"snaptrim_duration":1.1745734640000001,"stat_sum":{"num_bytes":17647284,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":78,"num_read_kb":73609,"num_write":75,"num_write_kb":23115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":294,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-01-15T14:29:42.877524+0000","last_change":"2023-01-15T14:29:39.317908+0000","last_active":"2023-01-15T14:29:42.877524+0000","last_peered":"2023-01-15T14:29:42.877524+0000","last_clean":"2023-01-15T14:29:42.877524+0000","last_became_active":"2023-01-15T14:29:39.317067+0000","last_became_peered":"2023-01-15T14:29:39.317067+0000","last_unstale":"2023-01-15T14:29:42.877524+0000","last_undegraded":"2023-01-15T14:29:42.877524+0000","last_fullsized":"2023-01-15T14:29:42.877524+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:28:05.841513+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":72,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-01-15T14:29:44.759921+0000","last_change":"2023-01-15T14:29:44.759819+0000","last_active":"2023-01-15T14:29:44.759921+0000","last_peered":"2023-01-15T14:29:44.759921+0000","last_clean":"2023-01-15T14:29:44.759921+0000","last_became_active":"2023-01-15T14:28:54.828857+0000","last_became_peered":"2023-01-15T14:28:54.828857+0000","last_unstale":"2023-01-15T14:29:44.759921+0000","last_undegraded":"2023-01-15T14:29:44.759921+0000","last_fullsized":"2023-01-15T14:29:44.759921+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:44.759745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:44.759745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:49.078","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"98'82","reported_seq":352,"reported_epoch":98,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:44.747862+0000","last_change":"2023-01-15T14:29:44.747862+0000","last_active":"2023-01-15T14:29:44.747862+0000","last_peered":"2023-01-15T14:29:44.747862+0000","last_clean":"2023-01-15T14:29:44.747862+0000","last_became_active":"2023-01-15T14:28:55.217912+0000","last_became_peered":"2023-01-15T14:28:55.217912+0000","last_unstale":"2023-01-15T14:29:44.747862+0000","last_undegraded":"2023-01-15T14:29:44.747862+0000","last_fullsized":"2023-01-15T14:29:44.747862+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.18272143800000001,"stat_sum":{"num_bytes":7663959,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":59,"num_read_kb":56238,"num_write":42,"num_write_kb":17290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10821306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":99,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-01-15T14:29:42.977616+0000","last_change":"2023-01-15T14:29:31.583684+0000","last_active":"2023-01-15T14:29:42.977616+0000","last_peered":"2023-01-15T14:29:42.977616+0000","last_clean":"2023-01-15T14:29:42.977616+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:29:42.977616+0000","last_undegraded":"2023-01-15T14:29:42.977616+0000","last_fullsized":"2023-01-15T14:29:42.977616+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:31.583599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.583599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:44.691","scrub_duration":0.22681161699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"98'99","reported_seq":358,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-01-15T14:29:45.355002+0000","last_change":"2023-01-15T14:29:45.355796+0000","last_active":"2023-01-15T14:29:45.355002+0000","last_peered":"2023-01-15T14:29:45.355002+0000","last_clean":"2023-01-15T14:29:45.355002+0000","last_became_active":"2023-01-15T14:28:23.842545+0000","last_became_peered":"2023-01-15T14:28:23.842545+0000","last_unstale":"2023-01-15T14:29:45.355002+0000","last_undegraded":"2023-01-15T14:29:45.355002+0000","last_fullsized":"2023-01-15T14:29:45.355002+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'81","last_scrub_stamp":"2023-01-15T14:29:32.795334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:32.795334+0000","objects_scrubbed":4,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:33.824","scrub_duration":0.40728601599999997,"objects_trimmed":0,"snaptrim_duration":0.50506554199999998,"stat_sum":{"num_bytes":12510745,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":57,"num_read_kb":49747,"num_write":51,"num_write_kb":16252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":89,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-01-15T14:29:42.931270+0000","last_change":"2023-01-15T14:28:13.253852+0000","last_active":"2023-01-15T14:29:42.931270+0000","last_peered":"2023-01-15T14:29:42.931270+0000","last_clean":"2023-01-15T14:29:42.931270+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:29:42.931270+0000","last_undegraded":"2023-01-15T14:29:42.931270+0000","last_fullsized":"2023-01-15T14:29:42.931270+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:28:11.770822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"98'189","reported_seq":548,"reported_epoch":99,"state":"active+clean+scrubbing","last_fresh":"2023-01-15T14:29:46.261160+0000","last_change":"2023-01-15T14:29:45.995850+0000","last_active":"2023-01-15T14:29:46.261160+0000","last_peered":"2023-01-15T14:29:46.261160+0000","last_clean":"2023-01-15T14:29:46.261160+0000","last_became_active":"2023-01-15T14:29:42.036350+0000","last_became_peered":"2023-01-15T14:29:42.036350+0000","last_unstale":"2023-01-15T14:29:46.261160+0000","last_undegraded":"2023-01-15T14:29:46.261160+0000","last_fullsized":"2023-01-15T14:29:46.261160+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70598088199999998,"stat_sum":{"num_bytes":14975983,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":108,"num_read_kb":103780,"num_write":78,"num_write_kb":38538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":34269143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":50,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-01-15T14:29:42.731151+0000","last_change":"2023-01-15T14:29:14.703116+0000","last_active":"2023-01-15T14:29:42.731151+0000","last_peered":"2023-01-15T14:29:42.731151+0000","last_clean":"2023-01-15T14:29:42.731151+0000","last_became_active":"2023-01-15T14:29:14.050302+0000","last_became_peered":"2023-01-15T14:29:14.050302+0000","last_unstale":"2023-01-15T14:29:42.731151+0000","last_undegraded":"2023-01-15T14:29:42.731151+0000","last_fullsized":"2023-01-15T14:29:42.731151+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:14.703035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:14.703035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:18.827","scrub_duration":0.44938580500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"98'181","reported_seq":455,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-01-15T14:29:45.453873+0000","last_change":"2023-01-15T14:29:45.453873+0000","last_active":"2023-01-15T14:29:45.453873+0000","last_peered":"2023-01-15T14:29:45.453873+0000","last_clean":"2023-01-15T14:29:45.453873+0000","last_became_active":"2023-01-15T14:28:23.946074+0000","last_became_peered":"2023-01-15T14:28:23.946074+0000","last_unstale":"2023-01-15T14:29:45.453873+0000","last_undegraded":"2023-01-15T14:29:45.453873+0000","last_fullsized":"2023-01-15T14:29:45.453873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:30:41.151","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.60377411400000003,"stat_sum":{"num_bytes":15899356,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":152,"num_read_kb":150893,"num_write":97,"num_write_kb":27695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"3.5","version":"97'45","reported_seq":263,"reported_epoch":98,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:44.748115+0000","last_change":"2023-01-15T14:29:44.748115+0000","last_active":"2023-01-15T14:29:44.748115+0000","last_peered":"2023-01-15T14:29:44.748115+0000","last_clean":"2023-01-15T14:29:44.748115+0000","last_became_active":"2023-01-15T14:28:54.831872+0000","last_became_peered":"2023-01-15T14:28:54.831872+0000","last_unstale":"2023-01-15T14:29:44.748115+0000","last_undegraded":"2023-01-15T14:29:44.748115+0000","last_fullsized":"2023-01-15T14:29:44.748115+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:46.735","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.27521325899999999,"stat_sum":{"num_bytes":7425855,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":25834,"num_write":19,"num_write_kb":11085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4252105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":100,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-01-15T14:29:44.947306+0000","last_change":"2023-01-15T14:29:29.346425+0000","last_active":"2023-01-15T14:29:44.947306+0000","last_peered":"2023-01-15T14:29:44.947306+0000","last_clean":"2023-01-15T14:29:44.947306+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:29:44.947306+0000","last_undegraded":"2023-01-15T14:29:44.947306+0000","last_fullsized":"2023-01-15T14:29:44.947306+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:52.809","scrub_duration":0.39286881699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"98'102","reported_seq":383,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-01-15T14:29:46.354154+0000","last_change":"2023-01-15T14:29:46.354630+0000","last_active":"2023-01-15T14:29:46.354154+0000","last_peered":"2023-01-15T14:29:46.354154+0000","last_clean":"2023-01-15T14:29:46.354154+0000","last_became_active":"2023-01-15T14:28:54.825996+0000","last_became_peered":"2023-01-15T14:28:54.825996+0000","last_unstale":"2023-01-15T14:29:46.354154+0000","last_undegraded":"2023-01-15T14:29:46.354154+0000","last_fullsized":"2023-01-15T14:29:46.354154+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.37939830099999999,"stat_sum":{"num_bytes":15248675,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":82,"num_read_kb":81293,"num_write":54,"num_write_kb":19128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12100908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":99,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-01-15T14:29:42.877904+0000","last_change":"2023-01-15T14:29:25.664703+0000","last_active":"2023-01-15T14:29:42.877904+0000","last_peered":"2023-01-15T14:29:42.877904+0000","last_clean":"2023-01-15T14:29:42.877904+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:29:42.877904+0000","last_undegraded":"2023-01-15T14:29:42.877904+0000","last_fullsized":"2023-01-15T14:29:42.877904+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:25.664616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:25.664616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:29.565","scrub_duration":0.40398350300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"98'60","reported_seq":290,"reported_epoch":98,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:44.748314+0000","last_change":"2023-01-15T14:29:44.748314+0000","last_active":"2023-01-15T14:29:44.748314+0000","last_peered":"2023-01-15T14:29:44.748314+0000","last_clean":"2023-01-15T14:29:44.748314+0000","last_became_active":"2023-01-15T14:28:23.741266+0000","last_became_peered":"2023-01-15T14:28:23.741266+0000","last_unstale":"2023-01-15T14:29:44.748314+0000","last_undegraded":"2023-01-15T14:29:44.748314+0000","last_fullsized":"2023-01-15T14:29:44.748314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.37563934599999999,"stat_sum":{"num_bytes":2621859,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":4,"num_write":22,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":120,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-01-15T14:29:45.047706+0000","last_change":"2023-01-15T14:29:42.343333+0000","last_active":"2023-01-15T14:29:45.047706+0000","last_peered":"2023-01-15T14:29:45.047706+0000","last_clean":"2023-01-15T14:29:45.047706+0000","last_became_active":"2023-01-15T14:29:36.099234+0000","last_became_peered":"2023-01-15T14:29:36.099234+0000","last_unstale":"2023-01-15T14:29:45.047706+0000","last_undegraded":"2023-01-15T14:29:45.047706+0000","last_fullsized":"2023-01-15T14:29:45.047706+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:47.376","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"98'48","reported_seq":297,"reported_epoch":98,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:44.751583+0000","last_change":"2023-01-15T14:29:44.751583+0000","last_active":"2023-01-15T14:29:44.751583+0000","last_peered":"2023-01-15T14:29:44.751583+0000","last_clean":"2023-01-15T14:29:44.751583+0000","last_became_active":"2023-01-15T14:28:23.841189+0000","last_became_peered":"2023-01-15T14:28:23.841189+0000","last_unstale":"2023-01-15T14:29:44.751583+0000","last_undegraded":"2023-01-15T14:29:44.751583+0000","last_fullsized":"2023-01-15T14:29:44.751583+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'44","last_scrub_stamp":"2023-01-15T14:29:41.838167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:41.838167+0000","objects_scrubbed":2,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:01.701","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.001156884,"stat_sum":{"num_bytes":3830925,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":17136,"num_write":26,"num_write_kb":9497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3e","length":"2"},{"start":"42","length":"1"},{"start":"45","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":101,"reported_epoch":98,"state":"active+clean","last_fresh":"2023-01-15T14:29:44.851101+0000","last_change":"2023-01-15T14:29:42.671857+0000","last_active":"2023-01-15T14:29:44.851101+0000","last_peered":"2023-01-15T14:29:44.851101+0000","last_clean":"2023-01-15T14:29:44.851101+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:29:44.851101+0000","last_undegraded":"2023-01-15T14:29:44.851101+0000","last_fullsized":"2023-01-15T14:29:44.851101+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.671761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.671761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:45.805","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":153297343,"num_objects":69,"num_object_clones":27,"num_object_copies":207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":4,"num_read":909,"num_read_kb":863002,"num_write":726,"num_write_kb":280513,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":122746975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1528,"ondisk_log_size":1528,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":864404,"kb_used_data":92841020,"kb_used_omap":848,"kb_used_meta":0,"kb_avail":92834976,"statfs":{"total":95948165120,"available":95063015424,"internally_reserved":6189056,"allocated":95069204480,"data_stored":95069204480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":868857,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.0700000000000003,"5min":4.0700000000000003,"15min":4.0700000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":75.893000000000001,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":9.2089999999999996,"5min":9.2089999999999996,"15min":9.2089999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":71.308999999999997,"5min":71.308999999999997,"15min":71.308999999999997},"last":0.77600000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":75.959999999999994,"5min":75.959999999999994,"15min":75.959999999999994},"last":1.056},{"interface":"front","average":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":6.2800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":68.605000000000004,"5min":68.605000000000004,"15min":68.605000000000004},"last":0.96499999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.7910000000000004,"5min":4.7910000000000004,"15min":4.7910000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":94.733999999999995,"5min":94.733999999999995,"15min":94.733999999999995},"last":1.0760000000000001},{"interface":"front","average":{"1min":4.1779999999999999,"5min":4.1779999999999999,"15min":4.1779999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":75.998999999999995},"last":0.80000000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.7770000000000001,"5min":3.7770000000000001,"15min":3.7770000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":76.034999999999997,"5min":76.034999999999997,"15min":76.034999999999997},"last":0.98299999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":9.4160000000000004,"5min":9.4160000000000004,"15min":9.4160000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":76.658000000000001,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":3.5129999999999999,"5min":3.5129999999999999,"15min":3.5129999999999999},"last":82.840000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.327,"5min":6.327,"15min":6.327},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":68.679000000000002,"5min":68.679000000000002,"15min":68.679000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":4.9450000000000003,"5min":4.9450000000000003,"15min":4.9450000000000003},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":95.430000000000007,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.93400000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":8.5239999999999991,"5min":8.5239999999999991,"15min":8.5239999999999991},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":68.635999999999996,"5min":68.635999999999996,"15min":68.635999999999996},"last":1.008},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.6520000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":1.0149999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":887128,"kb_used_data":92813740,"kb_used_omap":1269,"kb_used_meta":0,"kb_avail":92812252,"statfs":{"total":95948165120,"available":95039746048,"internally_reserved":1523712,"allocated":95041269760,"data_stored":95041269760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1299637,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":7,"commit_latency_ns":0,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.85299999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":91.325000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":91.355999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.81000000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.0590000000000002,"5min":6.0590000000000002,"15min":6.0590000000000002},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":81.686999999999998,"5min":81.686999999999998,"15min":81.686999999999998},"last":0.61799999999999999},{"interface":"front","average":{"1min":11.455,"5min":11.455,"15min":11.455},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":255.24299999999999,"5min":255.24299999999999,"15min":255.24299999999999},"last":1.004}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":6.6680000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":138.197,"5min":138.197,"15min":138.197},"last":1.135}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":1.0569999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.2800000000000002,"5min":4.2800000000000002,"15min":4.2800000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":78.888000000000005,"5min":78.888000000000005,"15min":78.888000000000005},"last":1.0820000000000001},{"interface":"front","average":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":47.332000000000001,"5min":47.332000000000001,"15min":47.332000000000001},"last":1.0349999999999999}]}]},{"osd":5,"up_from":0,"seq":55834574871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":917588,"kb_used_data":92790396,"kb_used_omap":1287,"kb_used_meta":0,"kb_avail":92781792,"statfs":{"total":95948165120,"available":95008555008,"internally_reserved":8810496,"allocated":95017365504,"data_stored":95017365504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1318650,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.915,"5min":4.915,"15min":4.915},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":86.944999999999993,"5min":86.944999999999993,"15min":86.944999999999993},"last":32.625},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.63300000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":8.5939999999999994,"5min":8.5939999999999994,"15min":8.5939999999999994},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":99.960999999999999,"5min":99.960999999999999,"15min":99.960999999999999},"last":1.0529999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":1.0369999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":99.884,"5min":99.884,"15min":99.884},"last":1.0880000000000001},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.90900000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":12.1,"5min":12.1,"15min":12.1},"last":0.70899999999999996},{"interface":"front","average":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":6.1509999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":99.921000000000006,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.99099999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.96399999999999997},{"interface":"front","average":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":30.971,"5min":30.971,"15min":30.971},"last":0.82099999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.673,"5min":5.673,"15min":5.673},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":99.780000000000001,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":5.1630000000000003,"5min":5.1630000000000003,"15min":5.1630000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":88.186999999999998,"5min":88.186999999999998,"15min":88.186999999999998},"last":1.075}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.128,"5min":13.128,"15min":13.128},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":179.73500000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":11.239000000000001,"5min":11.239000000000001,"15min":11.239000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":99.843000000000004,"5min":99.843000000000004,"15min":99.843000000000004},"last":1.129}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":886416,"kb_used_data":92814288,"kb_used_omap":1205,"kb_used_meta":0,"kb_avail":92812964,"statfs":{"total":95948165120,"available":95040475136,"internally_reserved":1355776,"allocated":95041830912,"data_stored":95041830912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1234410,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":16,"commit_latency_ns":0,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.8479999999999999,"5min":4.8479999999999999,"15min":4.8479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":93.632999999999996,"5min":93.632999999999996,"15min":93.632999999999996},"last":0.436},{"interface":"front","average":{"1min":7.6420000000000003,"5min":7.6420000000000003,"15min":7.6420000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":93.674000000000007,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.46200000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.532,"5min":13.532,"15min":13.532},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":159.256,"5min":159.256,"15min":159.256},"last":86.067999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.59299999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.3490000000000002,"5min":3.3490000000000002,"15min":3.3490000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":59.896999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":31.233000000000001,"5min":31.233000000000001,"15min":31.233000000000001},"last":0.51800000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":6.5620000000000003,"5min":6.5620000000000003,"15min":6.5620000000000003},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":66.311999999999998,"5min":66.311999999999998,"15min":66.311999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":13.874000000000001,"5min":13.874000000000001,"15min":13.874000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":159.47200000000001,"5min":159.47200000000001,"15min":159.47200000000001},"last":85.388999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":13.903,"15min":13.903},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":159.43000000000001,"5min":159.43000000000001,"15min":159.43000000000001},"last":85.543999999999997},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":0.66200000000000003}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":60.098999999999997,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.71899999999999997}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.0359999999999996,"5min":4.0359999999999996,"15min":4.0359999999999996},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":60.165999999999997,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.69099999999999995},{"interface":"front","average":{"1min":18.09,"5min":18.09,"15min":18.09},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":159.512,"5min":159.512,"15min":159.512},"last":85.484999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":907640,"kb_used_data":92794396,"kb_used_omap":909,"kb_used_meta":0,"kb_avail":92791740,"statfs":{"total":95948165120,"available":95018741760,"internally_reserved":2719744,"allocated":95021461504,"data_stored":95021461504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":930928,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":13,"commit_latency_ns":0,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.91300000000000003}]},{"osd":1,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.5979999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":3.3559999999999999,"5min":3.3559999999999999,"15min":3.3559999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":47.518000000000001,"5min":47.518000000000001,"15min":47.518000000000001},"last":0.55300000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":10.401999999999999,"5min":10.401999999999999,"15min":10.401999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":183.53,"5min":183.53,"15min":183.53},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.63800000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.6499999999999999,"15min":3.6499999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":53.139000000000003,"5min":53.139000000000003,"15min":53.139000000000003},"last":0.83899999999999997},{"interface":"front","average":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":3.8090000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":53.186999999999998,"5min":53.186999999999998,"15min":53.186999999999998},"last":0.98099999999999998}]},{"osd":5,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":1.0089999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":1.1879999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":37.124000000000002,"5min":37.124000000000002,"15min":37.124000000000002},"last":1.0600000000000001},{"interface":"front","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":30.445,"5min":30.445,"15min":30.445},"last":1.1499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":902328,"kb_used_data":92801444,"kb_used_omap":1189,"kb_used_meta":0,"kb_avail":92797052,"statfs":{"total":95948165120,"available":95024181248,"internally_reserved":4497408,"allocated":95028678656,"data_stored":95028678656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1218333,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":7,"commit_latency_ns":0,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":3.8929999999999998,"15min":3.8929999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":46.591000000000001,"5min":46.591000000000001,"15min":46.591000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":50.616,"5min":50.616,"15min":50.616},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":46.747999999999998,"5min":46.747999999999998,"15min":46.747999999999998},"last":0.55700000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":19.835999999999999,"5min":19.835999999999999,"15min":19.835999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":282.01400000000001,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":7.2450000000000001,"5min":7.2450000000000001,"15min":7.2450000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":53.145000000000003,"5min":53.145000000000003,"15min":53.145000000000003},"last":0.95299999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.7279999999999998,"5min":4.7279999999999998,"15min":4.7279999999999998},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":50.887999999999998,"5min":50.887999999999998,"15min":50.887999999999998},"last":1.1679999999999999},{"interface":"front","average":{"1min":3.605,"5min":3.605,"15min":3.605},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":52.512,"5min":52.512,"15min":52.512},"last":1.0980000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":1.272},{"interface":"front","average":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":4.4379999999999997},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":46.905999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.0640000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.0019999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":50.844999999999999,"5min":50.844999999999999,"15min":50.844999999999999},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.206}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":52.595999999999997,"5min":52.595999999999997,"15min":52.595999999999997},"last":1.1819999999999999},{"interface":"front","average":{"1min":4.75,"5min":4.75,"15min":4.75},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":50.804000000000002,"5min":50.804000000000002,"15min":50.804000000000002},"last":1.131}]}]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":893028,"kb_used_data":92812496,"kb_used_omap":1255,"kb_used_meta":0,"kb_avail":92806352,"statfs":{"total":95948165120,"available":95033704448,"internally_reserved":6291456,"allocated":95039995904,"data_stored":95039995904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1285210,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.69799999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":38.192}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":16.779,"5min":16.779,"15min":16.779},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.628}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.86799999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":9.798,"5min":9.798,"15min":9.798},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":176.34399999999999,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":0.76200000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.91900000000000004}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":15.670999999999999,"5min":15.670999999999999,"15min":15.670999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":292.61099999999999,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.0549999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:29:49.101 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:48.920+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:49.103 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:49.437 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:49.438 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:29:49.440 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:49.097+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:49.441 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:49.436+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:49.541 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:49.540+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:49.635 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:49.635 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:49.636 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:29:49.641 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:49.640+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:49.743 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:49.741+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:49.833 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:49.839 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:49.839 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:29:49.840 INFO:tasks.rados.rados.0.smithi090.stdout:428: done (0 left) 2023-01-15T14:29:49.840 INFO:tasks.rados.rados.0.smithi090.stdout:429: write (excl) oid 36 current snap is 39 2023-01-15T14:29:49.840 INFO:tasks.rados.rados.0.smithi090.stdout:429: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2023-01-15T14:29:49.840 INFO:tasks.rados.rados.0.smithi090.stdout:429: writing smithi09025264-36 from 625260 to 1246199 tid 1 2023-01-15T14:29:49.840 INFO:tasks.rados.rados.0.smithi090.stdout:429: writing smithi09025264-36 from 1683347 to 2282733 tid 2 2023-01-15T14:29:49.840 INFO:tasks.rados.rados.0.smithi090.stdout:429: writing smithi09025264-36 from 2943668 to 3025260 tid 3 2023-01-15T14:29:49.840 INFO:tasks.rados.rados.0.smithi090.stdout:430: write (excl) oid 47 current snap is 39 2023-01-15T14:29:49.841 INFO:tasks.rados.rados.0.smithi090.stdout:430: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2023-01-15T14:29:49.841 INFO:tasks.rados.rados.0.smithi090.stdout:430: writing smithi09025264-47 from 642067 to 1338255 tid 1 2023-01-15T14:29:49.841 INFO:tasks.rados.rados.0.smithi090.stdout:430: writing smithi09025264-47 from 2025476 to 2484873 tid 2 2023-01-15T14:29:49.841 INFO:tasks.rados.rados.0.smithi090.stdout:430: writing smithi09025264-47 from 2971091 to 3042067 tid 3 2023-01-15T14:29:49.841 INFO:tasks.rados.rados.0.smithi090.stdout:431: snap_create 2023-01-15T14:29:49.841 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:49.877 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:49.876+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:50.011 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:50.012 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:50.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:29:50.012 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:50.005+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:50.065 INFO:tasks.rados.rados.0.smithi090.stdout:429: finishing write tid 1 to smithi09025264-36 2023-01-15T14:29:50.209 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:50.209 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:50.210 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:29:50.210 INFO:tasks.rados.rados.0.smithi090.stdout:430: finishing write tid 1 to smithi09025264-47 2023-01-15T14:29:50.210 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:50.208+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:50.309 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:50.301+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:50.392 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:50.392 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:50.392 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:29:50.393 INFO:tasks.rados.rados.0.smithi090.stdout:429: finishing write tid 2 to smithi09025264-36 2023-01-15T14:29:50.394 INFO:tasks.rados.rados.0.smithi090.stdout:429: finishing write tid 3 to smithi09025264-36 2023-01-15T14:29:50.394 INFO:tasks.rados.rados.0.smithi090.stdout:429: finishing write tid 4 to smithi09025264-36 2023-01-15T14:29:50.394 INFO:tasks.rados.rados.0.smithi090.stdout:429: finishing write tid 5 to smithi09025264-36 2023-01-15T14:29:50.394 INFO:tasks.rados.rados.0.smithi090.stdout:429: oid 36 updating version 0 to 142 2023-01-15T14:29:50.394 INFO:tasks.rados.rados.0.smithi090.stdout:429: oid 36 updating version 142 to 144 2023-01-15T14:29:50.394 INFO:tasks.rados.rados.0.smithi090.stdout:429: oid 36 version 144 is already newer than 143 2023-01-15T14:29:50.394 INFO:tasks.rados.rados.0.smithi090.stdout:429: oid 36 version 144 is already newer than 141 2023-01-15T14:29:50.394 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 144 (ObjNum 180 snap 39 seq_num 180) dirty exists 2023-01-15T14:29:50.394 INFO:tasks.rados.rados.0.smithi090.stdout:429: left oid 36 (ObjNum 180 snap 39 seq_num 180) 2023-01-15T14:29:50.394 INFO:tasks.rados.rados.0.smithi090.stdout:429: done (1 left) 2023-01-15T14:29:50.394 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:50.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:29:50.646 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:50.441+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:50.646 INFO:tasks.rados.rados.0.smithi090.stdout:430: finishing write tid 2 to smithi09025264-47 2023-01-15T14:29:50.646 INFO:tasks.rados.rados.0.smithi090.stdout:430: finishing write tid 3 to smithi09025264-47 2023-01-15T14:29:50.646 INFO:tasks.rados.rados.0.smithi090.stdout:430: finishing write tid 4 to smithi09025264-47 2023-01-15T14:29:50.647 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:50.647 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:50.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:29:50.653 INFO:tasks.rados.rados.0.smithi090.stdout:430: finishing write tid 5 to smithi09025264-47 2023-01-15T14:29:50.758 INFO:tasks.rados.rados.0.smithi090.stdout:430: oid 47 updating version 0 to 185 2023-01-15T14:29:50.758 INFO:tasks.rados.rados.0.smithi090.stdout:430: oid 47 updating version 185 to 186 2023-01-15T14:29:50.758 INFO:tasks.rados.rados.0.smithi090.stdout:430: oid 47 updating version 186 to 187 2023-01-15T14:29:50.759 INFO:tasks.rados.rados.0.smithi090.stdout:430: oid 47 updating version 187 to 188 2023-01-15T14:29:50.759 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 188 (ObjNum 181 snap 39 seq_num 181) dirty exists 2023-01-15T14:29:50.759 INFO:tasks.rados.rados.0.smithi090.stdout:430: left oid 47 (ObjNum 181 snap 39 seq_num 181) 2023-01-15T14:29:50.759 INFO:tasks.rados.rados.0.smithi090.stdout:430: done (0 left) 2023-01-15T14:29:50.759 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:50.645+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:50.761 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:50.753+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:50.799 INFO:tasks.rados.rados.0.smithi090.stdout:431: done (0 left) 2023-01-15T14:29:50.936 INFO:tasks.rados.rados.0.smithi090.stdout:432: write oid 45 current snap is 40 2023-01-15T14:29:50.936 INFO:tasks.rados.rados.0.smithi090.stdout:432: seq_num 182 ranges {658874=771437,2083958=603055} 2023-01-15T14:29:50.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:29:50.939 INFO:tasks.rados.rados.0.smithi090.stdout:432: writing smithi09025264-45 from 658874 to 1430311 tid 1 2023-01-15T14:29:50.939 INFO:tasks.rados.rados.0.smithi090.stdout:432: writing smithi09025264-45 from 2083958 to 2687013 tid 2 2023-01-15T14:29:50.939 INFO:tasks.rados.rados.0.smithi090.stdout:433: read oid 2 snap -1 2023-01-15T14:29:50.939 INFO:tasks.rados.rados.0.smithi090.stdout:433: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:29:50.940 INFO:tasks.rados.rados.0.smithi090.stdout:434: copy_from oid 5 from oid 18 current snap is 40 2023-01-15T14:29:50.940 INFO:tasks.rados.rados.0.smithi090.stdout:435: write oid 47 current snap is 40 2023-01-15T14:29:50.940 INFO:tasks.rados.rados.0.smithi090.stdout:435: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2023-01-15T14:29:50.940 INFO:tasks.rados.rados.0.smithi090.stdout:435: writing smithi09025264-47 from 675681 to 1238720 tid 1 2023-01-15T14:29:50.940 INFO:tasks.rados.rados.0.smithi090.stdout:435: writing smithi09025264-47 from 1858793 to 2605506 tid 2 2023-01-15T14:29:50.940 INFO:tasks.rados.rados.0.smithi090.stdout:435: writing smithi09025264-47 from 3025937 to 3075681 tid 3 2023-01-15T14:29:50.940 INFO:tasks.rados.rados.0.smithi090.stdout:436: write (excl) oid 40 current snap is 40 2023-01-15T14:29:50.940 INFO:tasks.rados.rados.0.smithi090.stdout:436: seq_num 184 ranges {692488=638288,1917275=606724} 2023-01-15T14:29:50.940 INFO:tasks.rados.rados.0.smithi090.stdout:436: writing smithi09025264-40 from 692488 to 1330776 tid 1 2023-01-15T14:29:50.940 INFO:tasks.rados.rados.0.smithi090.stdout:436: writing smithi09025264-40 from 1917275 to 2523999 tid 2 2023-01-15T14:29:50.941 INFO:tasks.rados.rados.0.smithi090.stdout:437: copy_from oid 16 from oid 34 current snap is 40 2023-01-15T14:29:50.941 INFO:tasks.rados.rados.0.smithi090.stdout:438: write oid 12 current snap is 40 2023-01-15T14:29:50.941 INFO:tasks.rados.rados.0.smithi090.stdout:438: seq_num 185 ranges {709295=713537,1859404=750382} 2023-01-15T14:29:50.941 INFO:tasks.rados.rados.0.smithi090.stdout:438: writing smithi09025264-12 from 709295 to 1422832 tid 1 2023-01-15T14:29:50.941 INFO:tasks.rados.rados.0.smithi090.stdout:438: writing smithi09025264-12 from 1859404 to 2609786 tid 2 2023-01-15T14:29:50.941 INFO:tasks.rados.rados.0.smithi090.stdout:439: write oid 37 current snap is 40 2023-01-15T14:29:50.960 INFO:tasks.rados.rados.0.smithi090.stdout:439: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2023-01-15T14:29:50.961 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:50.961 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:50.962 INFO:tasks.rados.rados.0.smithi090.stdout:439: writing smithi09025264-37 from 726102 to 1514888 tid 1 2023-01-15T14:29:50.962 INFO:tasks.rados.rados.0.smithi090.stdout:439: writing smithi09025264-37 from 1917886 to 2528279 tid 2 2023-01-15T14:29:50.962 INFO:tasks.rados.rados.0.smithi090.stdout:439: writing smithi09025264-37 from 3108206 to 3126102 tid 3 2023-01-15T14:29:50.962 INFO:tasks.rados.rados.0.smithi090.stdout:440: copy_from oid 14 from oid 30 current snap is 40 2023-01-15T14:29:50.962 INFO:tasks.rados.rados.0.smithi090.stdout:441: snap_remove snap 39 2023-01-15T14:29:50.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:29:50.963 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:50.933+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:51.039 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:51.036+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:51.145 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:51.145 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:51.145 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:29:51.146 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:51.146 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:51.144+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:51.154 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:51.175 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:51.313 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:51.313 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:51.313 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:51.314 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:29:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:29:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:51.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:68 3.0 3:0b54190a:::smithi09025264-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.223534+0000", 2023-01-15T14:29:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.031176193999997, 2023-01-15T14:29:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49993064500000001, 2023-01-15T14:29:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:51.315 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 68 2023-01-15T14:29:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223534+0000", 2023-01-15T14:29:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223534+0000", 2023-01-15T14:29:51.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223533+0000", 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223539+0000", 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9970000000000004e-06 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223556+0000", 2023-01-15T14:29:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6543000000000001e-05 2023-01-15T14:29:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.223567+0000", 2023-01-15T14:29:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0893999999999999e-05 2023-01-15T14:29:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.520962+0000", 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29739558300000002 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.521142+0000", 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017972100000000001 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.723135+0000", 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20199320400000001 2023-01-15T14:29:51.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.723274+0000", 2023-01-15T14:29:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013887700000000001 2023-01-15T14:29:51.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.723465+0000", 2023-01-15T14:29:51.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019079399999999999 2023-01-15T14:29:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:94 3.0 3:07a51d33:::smithi09025264-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.815170952000003, 2023-01-15T14:29:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50760159199999999, 2023-01-15T14:29:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:51.338 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 94 2023-01-15T14:29:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439538+0000", 2023-01-15T14:29:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439857+0000", 2023-01-15T14:29:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000319086 2023-01-15T14:29:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439891+0000", 2023-01-15T14:29:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4449999999999997e-05 2023-01-15T14:29:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:51.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439906+0000", 2023-01-15T14:29:51.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5082999999999999e-05 2023-01-15T14:29:51.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824189+0000", 2023-01-15T14:29:51.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38428238100000001 2023-01-15T14:29:51.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:51.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824440+0000", 2023-01-15T14:29:51.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025139800000000001 2023-01-15T14:29:51.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:51.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824834+0000", 2023-01-15T14:29:51.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039422200000000001 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825347+0000", 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051323600000000003 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825364+0000", 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6478e-05 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:51.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825589+0000", 2023-01-15T14:29:51.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022497500000000001 2023-01-15T14:29:51.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:51.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826457+0000", 2023-01-15T14:29:51.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00086797600000000001 2023-01-15T14:29:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.846539+0000", 2023-01-15T14:29:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020081742 2023-01-15T14:29:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.946989+0000", 2023-01-15T14:29:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045032199999999 2023-01-15T14:29:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947076+0000", 2023-01-15T14:29:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6922999999999994e-05 2023-01-15T14:29:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:51.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947141+0000", 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5357000000000005e-05 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:95 3.0 3:07a51d33:::smithi09025264-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439927+0000", 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.814783347000002, 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50742387499999997, 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:51.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 95 2023-01-15T14:29:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439927+0000", 2023-01-15T14:29:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439927+0000", 2023-01-15T14:29:51.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:51.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439926+0000", 2023-01-15T14:29:51.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439933+0000", 2023-01-15T14:29:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2999999999999998e-06 2023-01-15T14:29:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439947+0000", 2023-01-15T14:29:51.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4071e-05 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439953+0000", 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4160000000000001e-06 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825479+0000", 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.385525544 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825576+0000", 2023-01-15T14:29:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7342000000000004e-05 2023-01-15T14:29:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825814+0000", 2023-01-15T14:29:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023816700000000001 2023-01-15T14:29:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826048+0000", 2023-01-15T14:29:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000234019 2023-01-15T14:29:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826067+0000", 2023-01-15T14:29:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9252000000000001e-05 2023-01-15T14:29:51.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826137+0000", 2023-01-15T14:29:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0004000000000003e-05 2023-01-15T14:29:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.828120+0000", 2023-01-15T14:29:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019824619999999999 2023-01-15T14:29:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.846644+0000", 2023-01-15T14:29:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.018524134000000001 2023-01-15T14:29:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947206+0000", 2023-01-15T14:29:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100562254 2023-01-15T14:29:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947279+0000", 2023-01-15T14:29:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2992999999999997e-05 2023-01-15T14:29:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947351+0000", 2023-01-15T14:29:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1847000000000005e-05 2023-01-15T14:29:51.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:96 3.0 3:07a51d33:::smithi09025264-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.814748839000003, 2023-01-15T14:29:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50769745899999996, 2023-01-15T14:29:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 96 2023-01-15T14:29:51.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:51.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439961+0000", 2023-01-15T14:29:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439966+0000", 2023-01-15T14:29:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0769999999999998e-06 2023-01-15T14:29:51.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439979+0000", 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3343000000000001e-05 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439987+0000", 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2400000000000001e-06 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826191+0000", 2023-01-15T14:29:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38620452799999999 2023-01-15T14:29:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826424+0000", 2023-01-15T14:29:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023264800000000001 2023-01-15T14:29:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947394+0000", 2023-01-15T14:29:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120970102 2023-01-15T14:29:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947505+0000", 2023-01-15T14:29:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001105 2023-01-15T14:29:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947659+0000", 2023-01-15T14:29:51.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015468799999999999 2023-01-15T14:29:51.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.170196175000001, 2023-01-15T14:29:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:29:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:51.693 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:29:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:51.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:29:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:29:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:51.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:29:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:29:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:29:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:29:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:29:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:29:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:29:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:29:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:29:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:29:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:29:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:29:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:29:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:29:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:51.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:29:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:29:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:29:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:29:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:29:51.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:29:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:29:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:29:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:29:51.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:29:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:29:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:29:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:29:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:29:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:29:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:29:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:51.735 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.169896378000004, 2023-01-15T14:29:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:29:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:51.736 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:29:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:51.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:29:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:29:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:29:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:51.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:29:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:29:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:29:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:29:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:29:51.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:29:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:29:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:29:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:29:51.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:29:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:29:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:29:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:29:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:29:51.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:29:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:29:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:29:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:29:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:29:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:29:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:29:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:29:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:29:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:29:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:29:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:29:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:51.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:29:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:29:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:29:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:29:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:51.806 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.169866593999998, 2023-01-15T14:29:51.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:29:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:29:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:51.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:29:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:29:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:29:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:29:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:29:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:29:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:29:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:29:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:29:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:29:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:29:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:29:51.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:29:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:29:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:29:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:29:51.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:29:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:29:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:29:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:29:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:29:51.931 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:182 3.4 3:2022fdfa:::smithi09025264-46:head [write 1933572~791670] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.041004525000005, 2023-01-15T14:29:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.582304865, 2023-01-15T14:29:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:51.932 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 182 2023-01-15T14:29:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:51.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213704+0000", 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214574+0000", 2023-01-15T14:29:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087037799999999999 2023-01-15T14:29:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214608+0000", 2023-01-15T14:29:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:29:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214627+0000", 2023-01-15T14:29:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:29:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491537+0000", 2023-01-15T14:29:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27691020300000002 2023-01-15T14:29:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491721+0000", 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345500000000001 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693760+0000", 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202039252 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693898+0000", 2023-01-15T14:29:51.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013789 2023-01-15T14:29:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694230+0000", 2023-01-15T14:29:51.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220899999999998 2023-01-15T14:29:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694866+0000", 2023-01-15T14:29:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063640599999999995 2023-01-15T14:29:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694887+0000", 2023-01-15T14:29:52.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0622999999999999e-05 2023-01-15T14:29:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695950+0000", 2023-01-15T14:29:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001062688 2023-01-15T14:29:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696014+0000", 2023-01-15T14:29:52.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4697999999999996e-05 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795898+0000", 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099883338000000002 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795922+0000", 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4326000000000001e-05 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795955+0000", 2023-01-15T14:29:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3191000000000001e-05 2023-01-15T14:29:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796011+0000", 2023-01-15T14:29:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5631999999999999e-05 2023-01-15T14:29:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.086 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:52.087 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:52.088 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.040057794000006, 2023-01-15T14:29:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:29:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:29:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:29:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:29:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:29:52.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:29:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:29:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:29:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:29:52.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:29:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:29:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:29:52.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:52.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:29:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:29:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:29:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:29:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:29:52.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:29:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:29:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:29:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:29:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.113 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.040001097000001, 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:52.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:29:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:29:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:29:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:29:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:29:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:29:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:29:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:29:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:29:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:29:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:29:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:29:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:29:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:29:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:29:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:29:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:29:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:29:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.121 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:29:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.423504924, 2023-01-15T14:29:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:29:52.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:29:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:29:52.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:29:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:29:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:29:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:29:52.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:29:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:29:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:29:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:29:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:29:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:29:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:29:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:29:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:333 3.0 3:0a998d04:::smithi09025264-10:head [rollback 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:29:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860002+0000", 2023-01-15T14:29:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.394708163000004, 2023-01-15T14:29:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50655092599999996, 2023-01-15T14:29:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 333 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860002+0000", 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860002+0000", 2023-01-15T14:29:52.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860001+0000", 2023-01-15T14:29:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:52.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860010+0000", 2023-01-15T14:29:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6570000000000003e-06 2023-01-15T14:29:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860027+0000", 2023-01-15T14:29:52.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7180000000000002e-05 2023-01-15T14:29:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860036+0000", 2023-01-15T14:29:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1929999999999999e-06 2023-01-15T14:29:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860266+0000", 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000229769 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860276+0000", 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8800000000000003e-06 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.163322+0000", 2023-01-15T14:29:52.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30304593099999999 2023-01-15T14:29:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.163486+0000", 2023-01-15T14:29:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016389900000000001 2023-01-15T14:29:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:52.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.163798+0000", 2023-01-15T14:29:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031278300000000002 2023-01-15T14:29:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164058+0000", 2023-01-15T14:29:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259609 2023-01-15T14:29:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164086+0000", 2023-01-15T14:29:52.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8113999999999999e-05 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164236+0000", 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014949900000000001 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293740+0000", 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12950431100000001 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.294109+0000", 2023-01-15T14:29:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036941200000000001 2023-01-15T14:29:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366415+0000", 2023-01-15T14:29:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072305164000000005 2023-01-15T14:29:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366503+0000", 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8101000000000002e-05 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366553+0000", 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0726999999999997e-05 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:29:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.394656452999996, 2023-01-15T14:29:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:29:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:52.145 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:29:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:29:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:29:52.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:29:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:29:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:29:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:29:52.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:29:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:29:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:29:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:29:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:29:52.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:29:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:29:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:29:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:29:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:29:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:29:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:29:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:29:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:373 3.4 3:2029fa71:::smithi09025264-39:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.176409246000006, 2023-01-15T14:29:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58298356399999995, 2023-01-15T14:29:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-01-15T14:29:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078299+0000", 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078644+0000", 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034494099999999998 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078676+0000", 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1183000000000003e-05 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078694+0000", 2023-01-15T14:29:52.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-05 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078748+0000", 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3634999999999997e-05 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078753+0000", 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6370000000000004e-06 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110841+0000", 2023-01-15T14:29:52.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032087439000000002 2023-01-15T14:29:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110967+0000", 2023-01-15T14:29:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126029 2023-01-15T14:29:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.457874+0000", 2023-01-15T14:29:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.346907725 2023-01-15T14:29:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458066+0000", 2023-01-15T14:29:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000191592 2023-01-15T14:29:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458321+0000", 2023-01-15T14:29:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025454200000000002 2023-01-15T14:29:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458918+0000", 2023-01-15T14:29:52.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059740800000000001 2023-01-15T14:29:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458945+0000", 2023-01-15T14:29:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-05 2023-01-15T14:29:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459253+0000", 2023-01-15T14:29:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030836200000000001 2023-01-15T14:29:52.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460943+0000", 2023-01-15T14:29:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016903580000000001 2023-01-15T14:29:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461340+0000", 2023-01-15T14:29:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039665799999999998 2023-01-15T14:29:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661129+0000", 2023-01-15T14:29:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19978933500000001 2023-01-15T14:29:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661188+0000", 2023-01-15T14:29:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8687000000000002e-05 2023-01-15T14:29:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661285+0000", 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779000000000003e-05 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:374 3.4 3:2029fa71:::smithi09025264-39:head [write 3101234~41642] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.171747334000003, 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57848911800000002, 2023-01-15T14:29:52.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 374 2023-01-15T14:29:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:52.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082961+0000", 2023-01-15T14:29:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:52.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083019+0000", 2023-01-15T14:29:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999997e-05 2023-01-15T14:29:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083051+0000", 2023-01-15T14:29:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1813999999999997e-05 2023-01-15T14:29:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083069+0000", 2023-01-15T14:29:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:29:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083143+0000", 2023-01-15T14:29:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4363000000000006e-05 2023-01-15T14:29:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083147+0000", 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4529999999999998e-06 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111017+0000", 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027870564 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:52.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111099+0000", 2023-01-15T14:29:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1277000000000001e-05 2023-01-15T14:29:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459156+0000", 2023-01-15T14:29:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34805700499999997 2023-01-15T14:29:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459393+0000", 2023-01-15T14:29:52.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023757499999999999 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459778+0000", 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000385335 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460020+0000", 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024141899999999999 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460037+0000", 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6665999999999999e-05 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460220+0000", 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018367599999999999 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461197+0000", 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00097708600000000006 2023-01-15T14:29:52.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461720+0000", 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052235400000000005 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661345+0000", 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199624884 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:52.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661405+0000", 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0965999999999999e-05 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661452+0000", 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6864999999999999e-05 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:375 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (56) in=7b,truncate 3142876] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:52.324 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.171605568000004, 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57851114599999998, 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 375 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083104+0000", 2023-01-15T14:29:52.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083113+0000", 2023-01-15T14:29:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6500000000000008e-06 2023-01-15T14:29:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083130+0000", 2023-01-15T14:29:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2023-01-15T14:29:52.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083142+0000", 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2252999999999999e-05 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083173+0000", 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0474999999999999e-05 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083175+0000", 2023-01-15T14:29:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2023-01-15T14:29:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183469+0000", 2023-01-15T14:29:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10029313099999999 2023-01-15T14:29:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:52.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183747+0000", 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000277998 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460165+0000", 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27641872899999997 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460342+0000", 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176792 2023-01-15T14:29:52.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460579+0000", 2023-01-15T14:29:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000237266 2023-01-15T14:29:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460833+0000", 2023-01-15T14:29:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025390900000000002 2023-01-15T14:29:52.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460853+0000", 2023-01-15T14:29:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9273e-05 2023-01-15T14:29:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461024+0000", 2023-01-15T14:29:52.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017111900000000001 2023-01-15T14:29:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461230+0000", 2023-01-15T14:29:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020672600000000001 2023-01-15T14:29:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461879+0000", 2023-01-15T14:29:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064846999999999995 2023-01-15T14:29:52.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661524+0000", 2023-01-15T14:29:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19964509599999999 2023-01-15T14:29:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661560+0000", 2023-01-15T14:29:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6264999999999999e-05 2023-01-15T14:29:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661616+0000", 2023-01-15T14:29:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5952000000000002e-05 2023-01-15T14:29:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:376 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:29:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 71.171523312000005, 2023-01-15T14:29:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57864278800000002, 2023-01-15T14:29:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.611 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 376 2023-01-15T14:29:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:29:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:29:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083186+0000", 2023-01-15T14:29:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083193+0000", 2023-01-15T14:29:52.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2023-01-15T14:29:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083211+0000", 2023-01-15T14:29:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7609999999999999e-05 2023-01-15T14:29:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083223+0000", 2023-01-15T14:29:52.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140000000000001e-05 2023-01-15T14:29:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093534+0000", 2023-01-15T14:29:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010311529 2023-01-15T14:29:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:52.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093551+0000", 2023-01-15T14:29:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.666e-05 2023-01-15T14:29:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183834+0000", 2023-01-15T14:29:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090283063999999996 2023-01-15T14:29:52.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.184076+0000", 2023-01-15T14:29:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024241900000000001 2023-01-15T14:29:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461001+0000", 2023-01-15T14:29:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27692504299999998 2023-01-15T14:29:52.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461152+0000", 2023-01-15T14:29:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150345 2023-01-15T14:29:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661642+0000", 2023-01-15T14:29:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200489742 2023-01-15T14:29:52.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661720+0000", 2023-01-15T14:29:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8604999999999997e-05 2023-01-15T14:29:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661830+0000", 2023-01-15T14:29:52.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010992099999999999 2023-01-15T14:29:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:646 3.0 3:07a51d33:::smithi09025264-24:head [write 1886503~412692] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58)", 2023-01-15T14:29:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:03.846884+0000", 2023-01-15T14:29:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 47.407826751999998, 2023-01-15T14:29:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50491198400000004, 2023-01-15T14:29:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.669 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 646 2023-01-15T14:29:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.846884+0000", 2023-01-15T14:29:52.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.846884+0000", 2023-01-15T14:29:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.846882+0000", 2023-01-15T14:29:52.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.847099+0000", 2023-01-15T14:29:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000216975 2023-01-15T14:29:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.847127+0000", 2023-01-15T14:29:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8053999999999999e-05 2023-01-15T14:29:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.847140+0000", 2023-01-15T14:29:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2982e-05 2023-01-15T14:29:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.946268+0000", 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099128283999999997 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.946529+0000", 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026035199999999999 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.946948+0000", 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041912500000000001 2023-01-15T14:29:52.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.948601+0000", 2023-01-15T14:29:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016531199999999999 2023-01-15T14:29:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.948621+0000", 2023-01-15T14:29:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0290999999999999e-05 2023-01-15T14:29:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.950494+0000", 2023-01-15T14:29:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001872868 2023-01-15T14:29:52.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.950860+0000", 2023-01-15T14:29:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036591899999999998 2023-01-15T14:29:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.951285+0000", 2023-01-15T14:29:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042471499999999999 2023-01-15T14:29:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:04.351228+0000", 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39994379699999999 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:04.351418+0000", 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018986299999999999 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:04.351796+0000", 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000377582 2023-01-15T14:29:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:29:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:29:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.393369477, 2023-01-15T14:29:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:29:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:29:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:29:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:29:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:29:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:29:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:29:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:29:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:29:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:29:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:29:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:29:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:29:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:29:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:29:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:29:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:29:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:29:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:29:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:29:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:29:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:29:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:29:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:29:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:29:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:29:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:29:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:29:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:29:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:961 3.0 3:0a998d04:::smithi09025264-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2023-01-15T14:29:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:29:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 20.191600889, 2023-01-15T14:29:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51361718300000003, 2023-01-15T14:29:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 961 2023-01-15T14:29:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:29:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:29:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063109+0000", 2023-01-15T14:29:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063115+0000", 2023-01-15T14:29:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8300000000000001e-06 2023-01-15T14:29:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063127+0000", 2023-01-15T14:29:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2517000000000001e-05 2023-01-15T14:29:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063137+0000", 2023-01-15T14:29:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8509999999999994e-06 2023-01-15T14:29:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063272+0000", 2023-01-15T14:29:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013513500000000001 2023-01-15T14:29:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063277+0000", 2023-01-15T14:29:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2750000000000003e-06 2023-01-15T14:29:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.265103+0000", 2023-01-15T14:29:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201825688 2023-01-15T14:29:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:29:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.265208+0000", 2023-01-15T14:29:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104693 2023-01-15T14:29:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.375398+0000", 2023-01-15T14:29:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.110190553 2023-01-15T14:29:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.375548+0000", 2023-01-15T14:29:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149417 2023-01-15T14:29:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576187+0000", 2023-01-15T14:29:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20063947500000001 2023-01-15T14:29:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576454+0000", 2023-01-15T14:29:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026708400000000001 2023-01-15T14:29:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576727+0000", 2023-01-15T14:29:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027257699999999998 2023-01-15T14:29:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:52.836 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:52.837 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:52.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:29:52.838 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:52.839 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:29:52.839 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:29:52.840 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:29:52.840 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:29:52.840 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:29:52.840 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:29:52.840 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:29:52.840 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:29:52.840 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:29:52.841 INFO:tasks.rados.rados.0.smithi090.stdout:441: done (9 left) 2023-01-15T14:29:52.841 INFO:tasks.rados.rados.0.smithi090.stdout:442: snap_create 2023-01-15T14:29:52.841 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 9 2023-01-15T14:29:52.841 INFO:tasks.rados.rados.0.smithi090.stdout:440: finishing copy_from to smithi09025264-14 2023-01-15T14:29:52.841 INFO:tasks.rados.rados.0.smithi090.stdout:440: got expected ENOENT (src dne) 2023-01-15T14:29:52.841 INFO:tasks.rados.rados.0.smithi090.stdout:440: finishing copy_from racing read to smithi09025264-14 2023-01-15T14:29:52.841 INFO:tasks.rados.rados.0.smithi090.stdout:433: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:29:52.842 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:51.312+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:52.842 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:52.836+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:52.842 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:29:52.844 INFO:tasks.rados.rados.0.smithi090.stdout:434: finishing copy_from to smithi09025264-5 2023-01-15T14:29:52.844 INFO:tasks.rados.rados.0.smithi090.stdout:434: got expected ENOENT (src dne) 2023-01-15T14:29:52.844 INFO:tasks.rados.rados.0.smithi090.stdout:434: finishing copy_from racing read to smithi09025264-5 2023-01-15T14:29:52.844 INFO:tasks.rados.rados.0.smithi090.stdout:436: finishing write tid 1 to smithi09025264-40 2023-01-15T14:29:52.844 INFO:tasks.rados.rados.0.smithi090.stdout:437: finishing copy_from to smithi09025264-16 2023-01-15T14:29:52.888 INFO:tasks.rados.rados.0.smithi090.stdout:437: got expected ENOENT (src dne) 2023-01-15T14:29:52.888 INFO:tasks.rados.rados.0.smithi090.stdout:437: finishing copy_from racing read to smithi09025264-16 2023-01-15T14:29:52.888 INFO:tasks.rados.rados.0.smithi090.stdout:435: finishing write tid 1 to smithi09025264-47 2023-01-15T14:29:52.888 INFO:tasks.rados.rados.0.smithi090.stdout:435: finishing write tid 2 to smithi09025264-47 2023-01-15T14:29:52.888 INFO:tasks.rados.rados.0.smithi090.stdout:436: finishing write tid 2 to smithi09025264-40 2023-01-15T14:29:52.888 INFO:tasks.rados.rados.0.smithi090.stdout:435: finishing write tid 3 to smithi09025264-47 2023-01-15T14:29:52.888 INFO:tasks.rados.rados.0.smithi090.stdout:435: finishing write tid 4 to smithi09025264-47 2023-01-15T14:29:52.888 INFO:tasks.rados.rados.0.smithi090.stdout:432: finishing write tid 1 to smithi09025264-45 2023-01-15T14:29:52.888 INFO:tasks.rados.rados.0.smithi090.stdout:435: finishing write tid 5 to smithi09025264-47 2023-01-15T14:29:52.889 INFO:tasks.rados.rados.0.smithi090.stdout:435: oid 47 updating version 0 to 190 2023-01-15T14:29:52.889 INFO:tasks.rados.rados.0.smithi090.stdout:435: oid 47 updating version 190 to 192 2023-01-15T14:29:52.889 INFO:tasks.rados.rados.0.smithi090.stdout:435: oid 47 updating version 192 to 193 2023-01-15T14:29:52.889 INFO:tasks.rados.rados.0.smithi090.stdout:435: oid 47 version 193 is already newer than 191 2023-01-15T14:29:52.889 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 193 (ObjNum 183 snap 40 seq_num 183) dirty exists 2023-01-15T14:29:52.889 INFO:tasks.rados.rados.0.smithi090.stdout:435: left oid 47 (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:29:52.889 INFO:tasks.rados.rados.0.smithi090.stdout:432: finishing write tid 2 to smithi09025264-45 2023-01-15T14:29:52.889 INFO:tasks.rados.rados.0.smithi090.stdout:438: finishing write tid 1 to smithi09025264-12 2023-01-15T14:29:52.889 INFO:tasks.rados.rados.0.smithi090.stdout:432: finishing write tid 3 to smithi09025264-45 2023-01-15T14:29:52.889 INFO:tasks.rados.rados.0.smithi090.stdout:432: finishing write tid 4 to smithi09025264-45 2023-01-15T14:29:52.890 INFO:tasks.rados.rados.0.smithi090.stdout:432: oid 45 updating version 0 to 128 2023-01-15T14:29:52.890 INFO:tasks.rados.rados.0.smithi090.stdout:432: oid 45 version 128 is already newer than 127 2023-01-15T14:29:52.890 INFO:tasks.rados.rados.0.smithi090.stdout:432: oid 45 version 128 is already newer than 126 2023-01-15T14:29:52.890 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 128 (ObjNum 182 snap 40 seq_num 182) dirty exists 2023-01-15T14:29:52.890 INFO:tasks.rados.rados.0.smithi090.stdout:432: left oid 45 (ObjNum 182 snap 40 seq_num 182) 2023-01-15T14:29:52.890 INFO:tasks.rados.rados.0.smithi090.stdout:438: finishing write tid 2 to smithi09025264-12 2023-01-15T14:29:52.890 INFO:tasks.rados.rados.0.smithi090.stdout:438: finishing write tid 3 to smithi09025264-12 2023-01-15T14:29:52.890 INFO:tasks.rados.rados.0.smithi090.stdout:432: done (8 left) 2023-01-15T14:29:52.890 INFO:tasks.rados.rados.0.smithi090.stdout:433: done (7 left) 2023-01-15T14:29:52.890 INFO:tasks.rados.rados.0.smithi090.stdout:434: done (6 left) 2023-01-15T14:29:52.890 INFO:tasks.rados.rados.0.smithi090.stdout:435: done (5 left) 2023-01-15T14:29:52.891 INFO:tasks.rados.rados.0.smithi090.stdout:437: done (4 left) 2023-01-15T14:29:52.891 INFO:tasks.rados.rados.0.smithi090.stdout:440: done (3 left) 2023-01-15T14:29:52.891 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:29:52.891 INFO:tasks.rados.rados.0.smithi090.stdout:439: finishing write tid 1 to smithi09025264-37 2023-01-15T14:29:52.891 INFO:tasks.rados.rados.0.smithi090.stdout:439: finishing write tid 2 to smithi09025264-37 2023-01-15T14:29:52.891 INFO:tasks.rados.rados.0.smithi090.stdout:436: finishing write tid 3 to smithi09025264-40 2023-01-15T14:29:52.891 INFO:tasks.rados.rados.0.smithi090.stdout:436: finishing write tid 4 to smithi09025264-40 2023-01-15T14:29:52.891 INFO:tasks.rados.rados.0.smithi090.stdout:436: oid 40 updating version 0 to 114 2023-01-15T14:29:52.891 INFO:tasks.rados.rados.0.smithi090.stdout:436: oid 40 updating version 114 to 116 2023-01-15T14:29:52.891 INFO:tasks.rados.rados.0.smithi090.stdout:436: oid 40 version 116 is already newer than 115 2023-01-15T14:29:52.891 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 116 (ObjNum 184 snap 40 seq_num 184) dirty exists 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout:436: left oid 40 (ObjNum 184 snap 40 seq_num 184) 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout:438: finishing write tid 4 to smithi09025264-12 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout:438: oid 12 updating version 0 to 205 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout:438: oid 12 version 205 is already newer than 204 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout:438: oid 12 version 205 is already newer than 203 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 205 (ObjNum 185 snap 40 seq_num 185) dirty exists 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout:438: left oid 12 (ObjNum 185 snap 40 seq_num 185) 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout:439: finishing write tid 3 to smithi09025264-37 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout:439: finishing write tid 4 to smithi09025264-37 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout:436: done (2 left) 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout:438: done (1 left) 2023-01-15T14:29:52.892 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:52.893 INFO:tasks.rados.rados.0.smithi090.stdout:439: finishing write tid 5 to smithi09025264-37 2023-01-15T14:29:52.893 INFO:tasks.rados.rados.0.smithi090.stdout:439: oid 37 updating version 0 to 196 2023-01-15T14:29:52.893 INFO:tasks.rados.rados.0.smithi090.stdout:439: oid 37 updating version 196 to 197 2023-01-15T14:29:52.893 INFO:tasks.rados.rados.0.smithi090.stdout:439: oid 37 updating version 197 to 198 2023-01-15T14:29:52.893 INFO:tasks.rados.rados.0.smithi090.stdout:439: oid 37 version 198 is already newer than 195 2023-01-15T14:29:52.893 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 198 (ObjNum 186 snap 40 seq_num 186) dirty exists 2023-01-15T14:29:52.893 INFO:tasks.rados.rados.0.smithi090.stdout:439: left oid 37 (ObjNum 186 snap 40 seq_num 186) 2023-01-15T14:29:52.893 INFO:tasks.rados.rados.0.smithi090.stdout:439: done (0 left) 2023-01-15T14:29:52.893 INFO:tasks.rados.rados.0.smithi090.stdout:442: done (0 left) 2023-01-15T14:29:52.893 INFO:tasks.rados.rados.0.smithi090.stdout:443: read oid 20 snap -1 2023-01-15T14:29:52.894 INFO:tasks.rados.rados.0.smithi090.stdout:443: expect deleted 2023-01-15T14:29:52.894 INFO:tasks.rados.rados.0.smithi090.stdout:444: rollback oid 43 current snap is 41 2023-01-15T14:29:52.894 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 43 to 40 2023-01-15T14:29:52.894 INFO:tasks.rados.rados.0.smithi090.stdout:445: delete oid 42 current snap is 41 2023-01-15T14:29:52.894 INFO:tasks.rados.rados.0.smithi090.stdout:444: finishing rollback tid 1 to smithi09025264-43 2023-01-15T14:29:52.894 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 0 (ObjNum 881 snap 0 seq_num 3844009792) dirty dne 2023-01-15T14:29:52.894 INFO:tasks.rados.rados.0.smithi090.stdout:444: done (2 left) 2023-01-15T14:29:52.894 INFO:tasks.rados.rados.0.smithi090.stdout:445: done (1 left) 2023-01-15T14:29:52.894 INFO:tasks.rados.rados.0.smithi090.stdout:446: snap_remove snap 40 2023-01-15T14:29:52.941 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:52.940+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:53.086 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:53.087 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:53.087 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:29:53.087 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:53.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:29:53.088 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:53.084+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:53.187 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:53.185+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:53.275 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:29:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:29:53.275 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:53.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:29:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:53.631 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:29:53.631 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2023-01-15T14:29:52.870069+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117263756,"num_objects":48,"num_object_clones":7,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":1,"num_read":1037,"num_read_kb":863110,"num_write":922,"num_write_kb":293329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":123927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1780,"ondisk_log_size":1780,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":5958648,"kb_used_data":649944976,"kb_used_omap":8472,"kb_used_meta":0,"kb_avail":649937012,"statfs":{"total":671637155840,"available":665535500288,"internally_reserved":8155136,"allocated":665543655424,"data_stored":665543655424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8676243,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":24,"commit_latency_ns":0,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36623955,"num_objects":-23,"num_object_clones":-20,"num_object_copies":-69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":-3,"num_read":12,"num_read_kb":8,"num_write":37,"num_write_kb":10400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000808"},"pg_stats":[{"pgid":"3.f","version":"102'99","reported_seq":354,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.659957+0000","last_change":"2023-01-15T14:29:49.659957+0000","last_active":"2023-01-15T14:29:49.659957+0000","last_peered":"2023-01-15T14:29:49.659957+0000","last_clean":"2023-01-15T14:29:49.659957+0000","last_became_active":"2023-01-15T14:28:23.846063+0000","last_became_peered":"2023-01-15T14:28:23.846063+0000","last_unstale":"2023-01-15T14:29:49.659957+0000","last_undegraded":"2023-01-15T14:29:49.659957+0000","last_fullsized":"2023-01-15T14:29:49.659957+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:52.368","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.80893161099999999,"stat_sum":{"num_bytes":7748027,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":48255,"num_write":50,"num_write_kb":19860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.e","version":"102'33","reported_seq":291,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.459839+0000","last_change":"2023-01-15T14:29:49.459839+0000","last_active":"2023-01-15T14:29:49.459839+0000","last_peered":"2023-01-15T14:29:49.459839+0000","last_clean":"2023-01-15T14:29:49.459839+0000","last_became_active":"2023-01-15T14:28:23.846858+0000","last_became_peered":"2023-01-15T14:28:23.846858+0000","last_unstale":"2023-01-15T14:29:49.459839+0000","last_undegraded":"2023-01-15T14:29:49.459839+0000","last_fullsized":"2023-01-15T14:29:49.459839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.60824714800000002,"stat_sum":{"num_bytes":2974839,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23059,"num_write":10,"num_write_kb":8092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.d","version":"102'115","reported_seq":403,"reported_epoch":102,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:49.462743+0000","last_change":"2023-01-15T14:29:49.459934+0000","last_active":"2023-01-15T14:29:49.462743+0000","last_peered":"2023-01-15T14:29:49.462743+0000","last_clean":"2023-01-15T14:29:49.462743+0000","last_became_active":"2023-01-15T14:28:54.932913+0000","last_became_peered":"2023-01-15T14:28:54.932913+0000","last_unstale":"2023-01-15T14:29:49.462743+0000","last_undegraded":"2023-01-15T14:29:49.462743+0000","last_fullsized":"2023-01-15T14:29:49.462743+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.60788848100000004,"stat_sum":{"num_bytes":8857256,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":87025,"num_write":47,"num_write_kb":18091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.c","version":"94'45","reported_seq":251,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.757660+0000","last_change":"2023-01-15T14:29:49.858455+0000","last_active":"2023-01-15T14:29:49.757660+0000","last_peered":"2023-01-15T14:29:49.757660+0000","last_clean":"2023-01-15T14:29:49.757660+0000","last_became_active":"2023-01-15T14:28:54.833568+0000","last_became_peered":"2023-01-15T14:28:54.833568+0000","last_unstale":"2023-01-15T14:29:49.757660+0000","last_undegraded":"2023-01-15T14:29:49.757660+0000","last_fullsized":"2023-01-15T14:29:49.757660+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:49.726","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.107442554,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":20,"num_write_kb":5044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2319333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.b","version":"102'34","reported_seq":288,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.756823+0000","last_change":"2023-01-15T14:29:49.757556+0000","last_active":"2023-01-15T14:29:49.756823+0000","last_peered":"2023-01-15T14:29:49.756823+0000","last_clean":"2023-01-15T14:29:49.756823+0000","last_became_active":"2023-01-15T14:28:23.934183+0000","last_became_peered":"2023-01-15T14:28:23.934183+0000","last_unstale":"2023-01-15T14:29:49.756823+0000","last_undegraded":"2023-01-15T14:29:49.756823+0000","last_fullsized":"2023-01-15T14:29:49.756823+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066201799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":13386,"num_write":12,"num_write_kb":5195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.a","version":"102'146","reported_seq":426,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.748484+0000","last_change":"2023-01-15T14:29:49.665311+0000","last_active":"2023-01-15T14:29:49.748484+0000","last_peered":"2023-01-15T14:29:49.748484+0000","last_clean":"2023-01-15T14:29:49.748484+0000","last_became_active":"2023-01-15T14:29:41.237634+0000","last_became_peered":"2023-01-15T14:29:41.237634+0000","last_unstale":"2023-01-15T14:29:49.748484+0000","last_undegraded":"2023-01-15T14:29:49.748484+0000","last_fullsized":"2023-01-15T14:29:49.748484+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:30:41.499","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.91769121799999998,"stat_sum":{"num_bytes":9311045,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":33249,"num_write":74,"num_write_kb":33528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":36194185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.9","version":"103'144","reported_seq":412,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:50.369614+0000","last_change":"2023-01-15T14:29:49.452673+0000","last_active":"2023-01-15T14:29:50.369614+0000","last_peered":"2023-01-15T14:29:50.369614+0000","last_clean":"2023-01-15T14:29:50.369614+0000","last_became_active":"2023-01-15T14:28:24.109843+0000","last_became_peered":"2023-01-15T14:28:24.109843+0000","last_unstale":"2023-01-15T14:29:50.369614+0000","last_undegraded":"2023-01-15T14:29:50.369614+0000","last_fullsized":"2023-01-15T14:29:50.369614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.70611565099999996,"stat_sum":{"num_bytes":13086034,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":92,"num_read_kb":99494,"num_write":63,"num_write_kb":24682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.8","version":"102'170","reported_seq":464,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.767000+0000","last_change":"2023-01-15T14:29:49.767404+0000","last_active":"2023-01-15T14:29:49.767000+0000","last_peered":"2023-01-15T14:29:49.767000+0000","last_clean":"2023-01-15T14:29:49.767000+0000","last_became_active":"2023-01-15T14:28:47.937231+0000","last_became_peered":"2023-01-15T14:28:47.937231+0000","last_unstale":"2023-01-15T14:29:49.767000+0000","last_undegraded":"2023-01-15T14:29:49.767000+0000","last_fullsized":"2023-01-15T14:29:49.767000+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'121","last_scrub_stamp":"2023-01-15T14:29:31.589753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.589753+0000","objects_scrubbed":8,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:54.442","scrub_duration":0.28338864200000002,"objects_trimmed":0,"snaptrim_duration":0.016725107999999999,"stat_sum":{"num_bytes":8974905,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":73609,"num_write":75,"num_write_kb":23115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":309,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:48.850460+0000","last_change":"2023-01-15T14:29:45.876275+0000","last_active":"2023-01-15T14:29:48.850460+0000","last_peered":"2023-01-15T14:29:48.850460+0000","last_clean":"2023-01-15T14:29:48.850460+0000","last_became_active":"2023-01-15T14:29:39.317067+0000","last_became_peered":"2023-01-15T14:29:39.317067+0000","last_unstale":"2023-01-15T14:29:48.850460+0000","last_undegraded":"2023-01-15T14:29:48.850460+0000","last_fullsized":"2023-01-15T14:29:48.850460+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:29:45.876183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:29:45.876183+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:46.712","scrub_duration":0.30530468399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.960969+0000","last_change":"2023-01-15T14:29:44.759819+0000","last_active":"2023-01-15T14:29:49.960969+0000","last_peered":"2023-01-15T14:29:49.960969+0000","last_clean":"2023-01-15T14:29:49.960969+0000","last_became_active":"2023-01-15T14:28:54.828857+0000","last_became_peered":"2023-01-15T14:28:54.828857+0000","last_unstale":"2023-01-15T14:29:49.960969+0000","last_undegraded":"2023-01-15T14:29:49.960969+0000","last_fullsized":"2023-01-15T14:29:49.960969+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:44.759745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:44.759745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:49.078","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"103'86","reported_seq":373,"reported_epoch":103,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:49.851089+0000","last_change":"2023-01-15T14:29:49.851089+0000","last_active":"2023-01-15T14:29:49.851089+0000","last_peered":"2023-01-15T14:29:49.851089+0000","last_clean":"2023-01-15T14:29:49.851089+0000","last_became_active":"2023-01-15T14:28:55.217912+0000","last_became_peered":"2023-01-15T14:28:55.217912+0000","last_unstale":"2023-01-15T14:29:49.851089+0000","last_undegraded":"2023-01-15T14:29:49.851089+0000","last_fullsized":"2023-01-15T14:29:49.851089+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.50858874799999998,"stat_sum":{"num_bytes":7663959,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":59,"num_read_kb":56238,"num_write":42,"num_write_kb":17290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10821306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:48.850296+0000","last_change":"2023-01-15T14:29:31.583684+0000","last_active":"2023-01-15T14:29:48.850296+0000","last_peered":"2023-01-15T14:29:48.850296+0000","last_clean":"2023-01-15T14:29:48.850296+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:29:48.850296+0000","last_undegraded":"2023-01-15T14:29:48.850296+0000","last_fullsized":"2023-01-15T14:29:48.850296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:31.583599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.583599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:44.691","scrub_duration":0.22681161699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"102'107","reported_seq":381,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.352829+0000","last_change":"2023-01-15T14:29:49.352829+0000","last_active":"2023-01-15T14:29:49.352829+0000","last_peered":"2023-01-15T14:29:49.352829+0000","last_clean":"2023-01-15T14:29:49.352829+0000","last_became_active":"2023-01-15T14:28:23.842545+0000","last_became_peered":"2023-01-15T14:28:23.842545+0000","last_unstale":"2023-01-15T14:29:49.352829+0000","last_undegraded":"2023-01-15T14:29:49.352829+0000","last_fullsized":"2023-01-15T14:29:49.352829+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'81","last_scrub_stamp":"2023-01-15T14:29:32.795334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:32.795334+0000","objects_scrubbed":4,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:33.824","scrub_duration":0.40728601599999997,"objects_trimmed":3,"snaptrim_duration":0.60542346199999997,"stat_sum":{"num_bytes":7092554,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":57,"num_read_kb":49747,"num_write":51,"num_write_kb":16252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":105,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.960944+0000","last_change":"2023-01-15T14:29:47.854985+0000","last_active":"2023-01-15T14:29:49.960944+0000","last_peered":"2023-01-15T14:29:49.960944+0000","last_clean":"2023-01-15T14:29:49.960944+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:29:49.960944+0000","last_undegraded":"2023-01-15T14:29:49.960944+0000","last_fullsized":"2023-01-15T14:29:49.960944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:47.854866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:47.854866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:56.359","scrub_duration":0.63763962500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"104'205","reported_seq":585,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-01-15T14:29:51.366943+0000","last_change":"2023-01-15T14:29:50.259196+0000","last_active":"2023-01-15T14:29:51.366943+0000","last_peered":"2023-01-15T14:29:51.366943+0000","last_clean":"2023-01-15T14:29:51.366943+0000","last_became_active":"2023-01-15T14:29:42.036350+0000","last_became_peered":"2023-01-15T14:29:42.036350+0000","last_unstale":"2023-01-15T14:29:51.366943+0000","last_undegraded":"2023-01-15T14:29:51.366943+0000","last_fullsized":"2023-01-15T14:29:51.366943+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'189","last_scrub_stamp":"2023-01-15T14:29:46.834961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:46.834961+0000","objects_scrubbed":7,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:11.320","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.15519123300000001,"stat_sum":{"num_bytes":13005561,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":109,"num_read_kb":103781,"num_write":82,"num_write_kb":39968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":34269143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.860893+0000","last_change":"2023-01-15T14:29:14.703116+0000","last_active":"2023-01-15T14:29:49.860893+0000","last_peered":"2023-01-15T14:29:49.860893+0000","last_clean":"2023-01-15T14:29:49.860893+0000","last_became_active":"2023-01-15T14:29:14.050302+0000","last_became_peered":"2023-01-15T14:29:14.050302+0000","last_unstale":"2023-01-15T14:29:49.860893+0000","last_undegraded":"2023-01-15T14:29:49.860893+0000","last_fullsized":"2023-01-15T14:29:49.860893+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:14.703035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:14.703035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:18.827","scrub_duration":0.44938580500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"104'198","reported_seq":489,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-01-15T14:29:51.553734+0000","last_change":"2023-01-15T14:29:49.350947+0000","last_active":"2023-01-15T14:29:51.553734+0000","last_peered":"2023-01-15T14:29:51.553734+0000","last_clean":"2023-01-15T14:29:51.553734+0000","last_became_active":"2023-01-15T14:28:23.946074+0000","last_became_peered":"2023-01-15T14:28:23.946074+0000","last_unstale":"2023-01-15T14:29:51.553734+0000","last_undegraded":"2023-01-15T14:29:51.553734+0000","last_fullsized":"2023-01-15T14:29:51.553734+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:30:41.151","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.60403308600000005,"stat_sum":{"num_bytes":17016491,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":156,"num_read_kb":150896,"num_write":112,"num_write_kb":31609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.5","version":"103'51","reported_seq":287,"reported_epoch":103,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:49.858683+0000","last_change":"2023-01-15T14:29:49.858683+0000","last_active":"2023-01-15T14:29:49.858683+0000","last_peered":"2023-01-15T14:29:49.858683+0000","last_clean":"2023-01-15T14:29:49.858683+0000","last_became_active":"2023-01-15T14:28:54.831872+0000","last_became_peered":"2023-01-15T14:28:54.831872+0000","last_unstale":"2023-01-15T14:29:49.858683+0000","last_undegraded":"2023-01-15T14:29:49.858683+0000","last_fullsized":"2023-01-15T14:29:49.858683+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.30997538800000002,"stat_sum":{"num_bytes":5815189,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":25834,"num_write":19,"num_write_kb":11085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4252105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":105,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:50.204329+0000","last_change":"2023-01-15T14:29:29.346425+0000","last_active":"2023-01-15T14:29:50.204329+0000","last_peered":"2023-01-15T14:29:50.204329+0000","last_clean":"2023-01-15T14:29:50.204329+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:29:50.204329+0000","last_undegraded":"2023-01-15T14:29:50.204329+0000","last_fullsized":"2023-01-15T14:29:50.204329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:52.809","scrub_duration":0.39286881699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"104'116","reported_seq":412,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-01-15T14:29:51.466676+0000","last_change":"2023-01-15T14:29:50.305359+0000","last_active":"2023-01-15T14:29:51.466676+0000","last_peered":"2023-01-15T14:29:51.466676+0000","last_clean":"2023-01-15T14:29:51.466676+0000","last_became_active":"2023-01-15T14:28:54.825996+0000","last_became_peered":"2023-01-15T14:28:54.825996+0000","last_unstale":"2023-01-15T14:29:51.466676+0000","last_undegraded":"2023-01-15T14:29:51.466676+0000","last_fullsized":"2023-01-15T14:29:51.466676+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:30:48.052","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.201329761,"stat_sum":{"num_bytes":9766128,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":84,"num_read_kb":81294,"num_write":58,"num_write_kb":20345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12100908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":104,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:48.951956+0000","last_change":"2023-01-15T14:29:25.664703+0000","last_active":"2023-01-15T14:29:48.951956+0000","last_peered":"2023-01-15T14:29:48.951956+0000","last_clean":"2023-01-15T14:29:48.951956+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:29:48.951956+0000","last_undegraded":"2023-01-15T14:29:48.951956+0000","last_fullsized":"2023-01-15T14:29:48.951956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:25.664616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:25.664616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:29.565","scrub_duration":0.40398350300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"101'62","reported_seq":308,"reported_epoch":102,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:49.254023+0000","last_change":"2023-01-15T14:29:49.254023+0000","last_active":"2023-01-15T14:29:49.254023+0000","last_peered":"2023-01-15T14:29:49.254023+0000","last_clean":"2023-01-15T14:29:49.254023+0000","last_became_active":"2023-01-15T14:28:23.741266+0000","last_became_peered":"2023-01-15T14:28:23.741266+0000","last_unstale":"2023-01-15T14:29:49.254023+0000","last_undegraded":"2023-01-15T14:29:49.254023+0000","last_fullsized":"2023-01-15T14:29:49.254023+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50058042300000005,"stat_sum":{"num_bytes":2621859,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":4,"num_write":22,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":125,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:50.103748+0000","last_change":"2023-01-15T14:29:42.343333+0000","last_active":"2023-01-15T14:29:50.103748+0000","last_peered":"2023-01-15T14:29:50.103748+0000","last_clean":"2023-01-15T14:29:50.103748+0000","last_became_active":"2023-01-15T14:29:36.099234+0000","last_became_peered":"2023-01-15T14:29:36.099234+0000","last_unstale":"2023-01-15T14:29:50.103748+0000","last_undegraded":"2023-01-15T14:29:50.103748+0000","last_fullsized":"2023-01-15T14:29:50.103748+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:47.376","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":320,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.558857+0000","last_change":"2023-01-15T14:29:49.558857+0000","last_active":"2023-01-15T14:29:49.558857+0000","last_peered":"2023-01-15T14:29:49.558857+0000","last_clean":"2023-01-15T14:29:49.558857+0000","last_became_active":"2023-01-15T14:28:23.841189+0000","last_became_peered":"2023-01-15T14:28:23.841189+0000","last_unstale":"2023-01-15T14:29:49.558857+0000","last_undegraded":"2023-01-15T14:29:49.558857+0000","last_fullsized":"2023-01-15T14:29:49.558857+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'44","last_scrub_stamp":"2023-01-15T14:29:41.838167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:41.838167+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:01.701","scrub_duration":0.32316732300000001,"objects_trimmed":1,"snaptrim_duration":0.71081751100000001,"stat_sum":{"num_bytes":2739541,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":17136,"num_write":26,"num_write_kb":9497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":106,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.748727+0000","last_change":"2023-01-15T14:29:42.671857+0000","last_active":"2023-01-15T14:29:49.748727+0000","last_peered":"2023-01-15T14:29:49.748727+0000","last_clean":"2023-01-15T14:29:49.748727+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:29:49.748727+0000","last_undegraded":"2023-01-15T14:29:49.748727+0000","last_fullsized":"2023-01-15T14:29:49.748727+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.671761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.671761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:45.805","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":116673388,"num_objects":46,"num_object_clones":7,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":1,"num_read":921,"num_read_kb":863010,"num_write":763,"num_write_kb":290913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":122746975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1663,"ondisk_log_size":1663,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":830292,"kb_used_data":92869184,"kb_used_omap":908,"kb_used_meta":0,"kb_avail":92869088,"statfs":{"total":95948165120,"available":95097946112,"internally_reserved":98304,"allocated":95098044416,"data_stored":95098044416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":929978,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.0700000000000003,"5min":4.0700000000000003,"15min":4.0700000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":75.893000000000001,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":9.2089999999999996,"5min":9.2089999999999996,"15min":9.2089999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":71.308999999999997,"5min":71.308999999999997,"15min":71.308999999999997},"last":0.83699999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":75.959999999999994,"5min":75.959999999999994,"15min":75.959999999999994},"last":0.68100000000000005},{"interface":"front","average":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":6.2800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":68.605000000000004,"5min":68.605000000000004,"15min":68.605000000000004},"last":0.66100000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.7910000000000004,"5min":4.7910000000000004,"15min":4.7910000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":94.733999999999995,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.70199999999999996},{"interface":"front","average":{"1min":4.1779999999999999,"5min":4.1779999999999999,"15min":4.1779999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":75.998999999999995},"last":0.72299999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.7770000000000001,"5min":3.7770000000000001,"15min":3.7770000000000001},"last":0.874},{"interface":"front","average":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":76.034999999999997,"5min":76.034999999999997,"15min":76.034999999999997},"last":0.75600000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":9.4160000000000004,"5min":9.4160000000000004,"15min":9.4160000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":76.658000000000001,"5min":76.658000000000001,"15min":76.658000000000001},"last":1.01},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":3.5129999999999999,"5min":3.5129999999999999,"15min":3.5129999999999999},"last":1.1080000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.327,"5min":6.327,"15min":6.327},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":68.679000000000002,"5min":68.679000000000002,"15min":68.679000000000002},"last":1.04},{"interface":"front","average":{"1min":4.9450000000000003,"5min":4.9450000000000003,"15min":4.9450000000000003},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":95.430000000000007,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.139}]},{"osd":6,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":8.5239999999999991,"5min":8.5239999999999991,"15min":8.5239999999999991},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":68.635999999999996,"5min":68.635999999999996,"15min":68.635999999999996},"last":1.093},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.6520000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":1.0740000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":856032,"kb_used_data":92844724,"kb_used_omap":1342,"kb_used_meta":0,"kb_avail":92843348,"statfs":{"total":95948165120,"available":95071588352,"internally_reserved":1409024,"allocated":95072997376,"data_stored":95072997376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1374949,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.628},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68400000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.77900000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":128.173},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":114.724}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.0590000000000002,"5min":6.0590000000000002,"15min":6.0590000000000002},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":81.686999999999998,"5min":81.686999999999998,"15min":81.686999999999998},"last":0.86499999999999999},{"interface":"front","average":{"1min":11.455,"5min":11.455,"15min":11.455},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":255.24299999999999,"5min":255.24299999999999,"15min":255.24299999999999},"last":0.60599999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":6.6680000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":138.197,"5min":138.197,"15min":138.197},"last":0.79500000000000004}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.83999999999999997}]},{"osd":7,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.2800000000000002,"5min":4.2800000000000002,"15min":4.2800000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":78.888000000000005,"5min":78.888000000000005,"15min":78.888000000000005},"last":0.91200000000000003},{"interface":"front","average":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":47.332000000000001,"5min":47.332000000000001,"15min":47.332000000000001},"last":13.951000000000001}]}]},{"osd":5,"up_from":0,"seq":55834574872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":871900,"kb_used_data":92831564,"kb_used_omap":1379,"kb_used_meta":0,"kb_avail":92827480,"statfs":{"total":95948165120,"available":95055339520,"internally_reserved":4182016,"allocated":95059521536,"data_stored":95059521536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1412241,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":7,"commit_latency_ns":0,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.915,"5min":4.915,"15min":4.915},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":86.944999999999993,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.83499999999999996}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":8.5939999999999994,"5min":8.5939999999999994,"15min":8.5939999999999994},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":99.960999999999999,"5min":99.960999999999999,"15min":99.960999999999999},"last":0.505},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.68899999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":99.884,"5min":99.884,"15min":99.884},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.55600000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":12.1,"5min":12.1,"15min":12.1},"last":0.87},{"interface":"front","average":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":6.1509999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":99.921000000000006,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.78600000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":30.971,"5min":30.971,"15min":30.971},"last":0.74199999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.673,"5min":5.673,"15min":5.673},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":99.780000000000001,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":5.1630000000000003,"5min":5.1630000000000003,"15min":5.1630000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":88.186999999999998,"5min":88.186999999999998,"15min":88.186999999999998},"last":0.95799999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.128,"5min":13.128,"15min":13.128},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":179.73500000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":11.239000000000001,"5min":11.239000000000001,"15min":11.239000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":99.843000000000004,"5min":99.843000000000004,"15min":99.843000000000004},"last":0.64500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":852976,"kb_used_data":92846508,"kb_used_omap":1278,"kb_used_meta":0,"kb_avail":92846404,"statfs":{"total":95948165120,"available":95074717696,"internally_reserved":106496,"allocated":95074824192,"data_stored":95074824192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1309139,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.8479999999999999,"5min":4.8479999999999999,"15min":4.8479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":93.632999999999996,"5min":93.632999999999996,"15min":93.632999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":7.6420000000000003,"5min":7.6420000000000003,"15min":7.6420000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":93.674000000000007,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.56000000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.532,"5min":13.532,"15min":13.532},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":159.256,"5min":159.256,"15min":159.256},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.88200000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.3490000000000002,"5min":3.3490000000000002,"15min":3.3490000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":59.896999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":31.233000000000001,"5min":31.233000000000001,"15min":31.233000000000001},"last":0.94999999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":6.5620000000000003,"5min":6.5620000000000003,"15min":6.5620000000000003},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":66.311999999999998,"5min":66.311999999999998,"15min":66.311999999999998},"last":2.3010000000000002},{"interface":"front","average":{"1min":13.874000000000001,"5min":13.874000000000001,"15min":13.874000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":159.47200000000001,"5min":159.47200000000001,"15min":159.47200000000001},"last":2.5800000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":13.903,"15min":13.903},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":159.43000000000001,"5min":159.43000000000001,"15min":159.43000000000001},"last":1.355},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":1.3839999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":1.6799999999999999},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":60.098999999999997,"5min":60.098999999999997,"15min":60.098999999999997},"last":1.752}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.0359999999999996,"5min":4.0359999999999996,"15min":4.0359999999999996},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":60.165999999999997,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.0629999999999999},{"interface":"front","average":{"1min":18.09,"5min":18.09,"15min":18.09},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":159.512,"5min":159.512,"15min":159.512},"last":1.2390000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":853384,"kb_used_data":92846752,"kb_used_omap":977,"kb_used_meta":0,"kb_avail":92845996,"statfs":{"total":95948165120,"available":95074299904,"internally_reserved":774144,"allocated":95075074048,"data_stored":95075074048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000540,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.499}]},{"osd":1,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.5979999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":143.071},{"interface":"front","average":{"1min":3.3559999999999999,"5min":3.3559999999999999,"15min":3.3559999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":47.518000000000001,"5min":47.518000000000001,"15min":47.518000000000001},"last":0.65000000000000002}]},{"osd":3,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":10.401999999999999,"5min":10.401999999999999,"15min":10.401999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":183.53,"5min":183.53,"15min":183.53},"last":143.13300000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.75}]},{"osd":4,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.6499999999999999,"15min":3.6499999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":53.139000000000003,"5min":53.139000000000003,"15min":53.139000000000003},"last":1.0069999999999999},{"interface":"front","average":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":3.8090000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":53.186999999999998,"5min":53.186999999999998,"15min":53.186999999999998},"last":0.96399999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"last":143.262},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.78900000000000003}]},{"osd":6,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.877},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":143.215}]},{"osd":7,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":37.124000000000002,"5min":37.124000000000002,"15min":37.124000000000002},"last":0.92800000000000005},{"interface":"front","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":30.445,"5min":30.445,"15min":30.445},"last":143.17099999999999}]}]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":851088,"kb_used_data":92849716,"kb_used_omap":1272,"kb_used_meta":0,"kb_avail":92848292,"statfs":{"total":95948165120,"available":95076651008,"internally_reserved":1458176,"allocated":95078109184,"data_stored":95078109184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1303332,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":3.8929999999999998,"15min":3.8929999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":46.591000000000001,"5min":46.591000000000001,"15min":46.591000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":50.616,"5min":50.616,"15min":50.616},"last":0.69099999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.75800000000000001},{"interface":"front","average":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":46.747999999999998,"5min":46.747999999999998,"15min":46.747999999999998},"last":0.625}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":19.835999999999999,"5min":19.835999999999999,"15min":19.835999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":282.01400000000001,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":7.2450000000000001,"5min":7.2450000000000001,"15min":7.2450000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":53.145000000000003,"5min":53.145000000000003,"15min":53.145000000000003},"last":1.038}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.7279999999999998,"5min":4.7279999999999998,"15min":4.7279999999999998},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":50.887999999999998,"5min":50.887999999999998,"15min":50.887999999999998},"last":0.78900000000000003},{"interface":"front","average":{"1min":3.605,"5min":3.605,"15min":3.605},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":52.512,"5min":52.512,"15min":52.512},"last":0.96199999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":4.4379999999999997},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":46.905999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.1080000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.0019999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":50.844999999999999,"5min":50.844999999999999,"15min":50.844999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.022}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":52.595999999999997,"5min":52.595999999999997,"15min":52.595999999999997},"last":1.0660000000000001},{"interface":"front","average":{"1min":4.75,"5min":4.75,"15min":4.75},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":50.804000000000002,"5min":50.804000000000002,"15min":50.804000000000002},"last":0.98799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":842976,"kb_used_data":92856528,"kb_used_omap":1314,"kb_used_meta":0,"kb_avail":92856404,"statfs":{"total":95948165120,"available":95084957696,"internally_reserved":126976,"allocated":95085084672,"data_stored":95085084672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1346064,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.504}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.53700000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":16.779,"5min":16.779,"15min":16.779},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.73699999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.2}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":9.798,"5min":9.798,"15min":9.798},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":176.34399999999999,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":0.95599999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.268},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.93899999999999995}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":15.670999999999999,"5min":15.670999999999999,"15min":15.670999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":292.61099999999999,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.3089999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:29:53.696 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-01-15T14:29:53.696 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:29:53.696 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:29:53.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:29:53.698 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:53.696+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:53.798 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:53.797+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:53.870 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:29:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:29:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:38 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'3, mlcod=0'0)", 2023-01-15T14:29:53.870 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:29:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.694897126000001, 2023-01-15T14:29:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30191175999999997, 2023-01-15T14:29:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:53.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:53.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:29:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:29:53.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112976+0000", 2023-01-15T14:29:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113186+0000", 2023-01-15T14:29:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020979700000000001 2023-01-15T14:29:53.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113228+0000", 2023-01-15T14:29:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2150999999999997e-05 2023-01-15T14:29:53.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213313+0000", 2023-01-15T14:29:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100084943 2023-01-15T14:29:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313517+0000", 2023-01-15T14:29:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10020435699999999 2023-01-15T14:29:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313576+0000", 2023-01-15T14:29:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783999999999998e-05 2023-01-15T14:29:53.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313976+0000", 2023-01-15T14:29:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039960499999999998 2023-01-15T14:29:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314004+0000", 2023-01-15T14:29:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7954e-05 2023-01-15T14:29:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314175+0000", 2023-01-15T14:29:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171557 2023-01-15T14:29:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414853+0000", 2023-01-15T14:29:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100677875 2023-01-15T14:29:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414890+0000", 2023-01-15T14:29:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6803e-05 2023-01-15T14:29:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:53.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:39 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'4, mlcod=0'0)", 2023-01-15T14:29:53.888 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:29:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.594504951000005, 2023-01-15T14:29:53.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20154512699999999, 2023-01-15T14:29:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:53.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:29:53.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:29:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:53.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213369+0000", 2023-01-15T14:29:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213386+0000", 2023-01-15T14:29:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7643e-05 2023-01-15T14:29:53.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213427+0000", 2023-01-15T14:29:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0930000000000003e-05 2023-01-15T14:29:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213459+0000", 2023-01-15T14:29:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1479999999999997e-05 2023-01-15T14:29:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314106+0000", 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100646614 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314128+0000", 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2340999999999998e-05 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314224+0000", 2023-01-15T14:29:53.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6534999999999996e-05 2023-01-15T14:29:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314249+0000", 2023-01-15T14:29:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4981e-05 2023-01-15T14:29:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314337+0000", 2023-01-15T14:29:53.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7287000000000006e-05 2023-01-15T14:29:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414903+0000", 2023-01-15T14:29:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100566552 2023-01-15T14:29:53.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414916+0000", 2023-01-15T14:29:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354000000000001e-05 2023-01-15T14:29:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.902 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:46 3.0 e23/20 3:03b54904:::smithi09025264-12:head v 23'5, mlcod=23'1)", 2023-01-15T14:29:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:29:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.594001211999995, 2023-01-15T14:29:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202592668, 2023-01-15T14:29:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:29:53.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:29:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213873+0000", 2023-01-15T14:29:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:53.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214477+0000", 2023-01-15T14:29:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060481300000000005 2023-01-15T14:29:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214532+0000", 2023-01-15T14:29:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5096999999999998e-05 2023-01-15T14:29:53.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214555+0000", 2023-01-15T14:29:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.234e-05 2023-01-15T14:29:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414409+0000", 2023-01-15T14:29:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19985461199999999 2023-01-15T14:29:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414442+0000", 2023-01-15T14:29:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2987000000000001e-05 2023-01-15T14:29:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414813+0000", 2023-01-15T14:29:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037061899999999998 2023-01-15T14:29:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414832+0000", 2023-01-15T14:29:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9380000000000001e-05 2023-01-15T14:29:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415093+0000", 2023-01-15T14:29:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026032099999999999 2023-01-15T14:29:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:53.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415154+0000", 2023-01-15T14:29:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1538999999999995e-05 2023-01-15T14:29:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.416467+0000", 2023-01-15T14:29:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001312613 2023-01-15T14:29:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:281 3.5 e30/20 3:a75958b1:::smithi09025264-6:head v 30'11, mlcod=23'6)", 2023-01-15T14:29:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:34.981189+0000", 2023-01-15T14:29:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.826685913000006, 2023-01-15T14:29:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19987028400000001, 2023-01-15T14:29:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981189+0000", 2023-01-15T14:29:53.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981189+0000", 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981188+0000", 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981206+0000", 2023-01-15T14:29:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8377999999999999e-05 2023-01-15T14:29:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981231+0000", 2023-01-15T14:29:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.516e-05 2023-01-15T14:29:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981249+0000", 2023-01-15T14:29:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7425e-05 2023-01-15T14:29:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.179815+0000", 2023-01-15T14:29:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19856638300000001 2023-01-15T14:29:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:53.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.179854+0000", 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8587000000000002e-05 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180059+0000", 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020564300000000001 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180085+0000", 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5162999999999999e-05 2023-01-15T14:29:53.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180548+0000", 2023-01-15T14:29:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046301300000000002 2023-01-15T14:29:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180599+0000", 2023-01-15T14:29:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0943000000000003e-05 2023-01-15T14:29:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.181060+0000", 2023-01-15T14:29:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046124700000000002 2023-01-15T14:29:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:347 3.0 e32/20 3:03b54904:::smithi09025264-12:head v 32'40, mlcod=31'36)", 2023-01-15T14:29:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:29:53.916 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.796078312999995, 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20127003500000001, 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:53.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:29:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011795+0000", 2023-01-15T14:29:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:53.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.012239+0000", 2023-01-15T14:29:54.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000444361 2023-01-15T14:29:54.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.012261+0000", 2023-01-15T14:29:54.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2008e-05 2023-01-15T14:29:54.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.112398+0000", 2023-01-15T14:29:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013730799999999 2023-01-15T14:29:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.210584+0000", 2023-01-15T14:29:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098186184999999995 2023-01-15T14:29:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.210634+0000", 2023-01-15T14:29:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9706000000000001e-05 2023-01-15T14:29:54.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211053+0000", 2023-01-15T14:29:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041844199999999999 2023-01-15T14:29:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211117+0000", 2023-01-15T14:29:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4950000000000007e-05 2023-01-15T14:29:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211390+0000", 2023-01-15T14:29:54.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027278300000000002 2023-01-15T14:29:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211507+0000", 2023-01-15T14:29:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011642 2023-01-15T14:29:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.213067+0000", 2023-01-15T14:29:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001560402 2023-01-15T14:29:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:621 3.8 e55/41 3:17d5d741:::smithi09025264-5:head v 55'45, mlcod=53'42)", 2023-01-15T14:29:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:00.863036+0000", 2023-01-15T14:29:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 52.944839782999999, 2023-01-15T14:29:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16723332099999999, 2023-01-15T14:29:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.095 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863036+0000", 2023-01-15T14:29:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863036+0000", 2023-01-15T14:29:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863035+0000", 2023-01-15T14:29:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863346+0000", 2023-01-15T14:29:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031160999999999998 2023-01-15T14:29:54.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863374+0000", 2023-01-15T14:29:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:29:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863392+0000", 2023-01-15T14:29:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7762e-05 2023-01-15T14:29:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.963611+0000", 2023-01-15T14:29:54.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100219251 2023-01-15T14:29:54.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.963647+0000", 2023-01-15T14:29:54.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5929000000000001e-05 2023-01-15T14:29:54.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.964473+0000", 2023-01-15T14:29:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082583999999999999 2023-01-15T14:29:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.964502+0000", 2023-01-15T14:29:54.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8996000000000002e-05 2023-01-15T14:29:54.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.964751+0000", 2023-01-15T14:29:54.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024928 2023-01-15T14:29:54.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.964782+0000", 2023-01-15T14:29:54.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1202999999999999e-05 2023-01-15T14:29:54.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:01.030269+0000", 2023-01-15T14:29:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.065486775999999997 2023-01-15T14:29:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.269 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.4.0:118 3.9 e62/20 3:9f7cca9d:::smithi09025264-41:1f v 62'64, mlcod=61'58)", 2023-01-15T14:29:54.269 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:29:54.269 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.934687824999997, 2023-01-15T14:29:54.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20203982600000001, 2023-01-15T14:29:54.269 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.270 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.270 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:29:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:29:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873186+0000", 2023-01-15T14:29:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873201+0000", 2023-01-15T14:29:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5474000000000001e-05 2023-01-15T14:29:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873219+0000", 2023-01-15T14:29:54.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7677999999999999e-05 2023-01-15T14:29:54.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873234+0000", 2023-01-15T14:29:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4481000000000001e-05 2023-01-15T14:29:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.073729+0000", 2023-01-15T14:29:54.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200495595 2023-01-15T14:29:54.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.073768+0000", 2023-01-15T14:29:54.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8492999999999999e-05 2023-01-15T14:29:54.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074468+0000", 2023-01-15T14:29:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069991300000000003 2023-01-15T14:29:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074486+0000", 2023-01-15T14:29:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8669999999999999e-05 2023-01-15T14:29:54.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074563+0000", 2023-01-15T14:29:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7084999999999998e-05 2023-01-15T14:29:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074599+0000", 2023-01-15T14:29:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5453000000000001e-05 2023-01-15T14:29:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.075227+0000", 2023-01-15T14:29:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062855400000000003 2023-01-15T14:29:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:125 3.0 e70/55 3:0b54190a:::smithi09025264-17:25 v 70'136, mlcod=68'130)", 2023-01-15T14:29:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:29:54.283 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.864915101000001, 2023-01-15T14:29:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30171926799999998, 2023-01-15T14:29:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:29:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:29:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942958+0000", 2023-01-15T14:29:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942972+0000", 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3386e-05 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942990+0000", 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8113000000000001e-05 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.943002+0000", 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:29:54.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141942+0000", 2023-01-15T14:29:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19893962300000001 2023-01-15T14:29:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141978+0000", 2023-01-15T14:29:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6383e-05 2023-01-15T14:29:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142670+0000", 2023-01-15T14:29:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069211900000000005 2023-01-15T14:29:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142688+0000", 2023-01-15T14:29:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7839999999999999e-05 2023-01-15T14:29:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142768+0000", 2023-01-15T14:29:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0210999999999995e-05 2023-01-15T14:29:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244581+0000", 2023-01-15T14:29:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101813406 2023-01-15T14:29:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244680+0000", 2023-01-15T14:29:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8045000000000001e-05 2023-01-15T14:29:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.290 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:818 3.8 e70/41 3:1eaad226:::smithi09025264-49:head v 70'89, mlcod=68'85)", 2023-01-15T14:29:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:16.285706+0000", 2023-01-15T14:29:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.522169167999998, 2023-01-15T14:29:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17358161, 2023-01-15T14:29:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285706+0000", 2023-01-15T14:29:54.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285706+0000", 2023-01-15T14:29:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285706+0000", 2023-01-15T14:29:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:54.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285713+0000", 2023-01-15T14:29:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7179999999999996e-06 2023-01-15T14:29:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285728+0000", 2023-01-15T14:29:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5063e-05 2023-01-15T14:29:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285741+0000", 2023-01-15T14:29:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2847000000000001e-05 2023-01-15T14:29:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386010+0000", 2023-01-15T14:29:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100268632 2023-01-15T14:29:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386047+0000", 2023-01-15T14:29:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6637000000000002e-05 2023-01-15T14:29:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386644+0000", 2023-01-15T14:29:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059714800000000004 2023-01-15T14:29:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386673+0000", 2023-01-15T14:29:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9660000000000001e-05 2023-01-15T14:29:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386772+0000", 2023-01-15T14:29:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8550999999999999e-05 2023-01-15T14:29:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386817+0000", 2023-01-15T14:29:54.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4747999999999999e-05 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.459288+0000", 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072471181999999995 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:946 3.8 e83/41 3:17d5d741:::smithi09025264-5:head v 83'117, mlcod=81'114)", 2023-01-15T14:29:54.297 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:29.246316+0000", 2023-01-15T14:29:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.561559479, 2023-01-15T14:29:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16713046000000001, 2023-01-15T14:29:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.246316+0000", 2023-01-15T14:29:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.246316+0000", 2023-01-15T14:29:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.246315+0000", 2023-01-15T14:29:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.246875+0000", 2023-01-15T14:29:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055987799999999998 2023-01-15T14:29:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.246891+0000", 2023-01-15T14:29:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6104999999999999e-05 2023-01-15T14:29:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.346972+0000", 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100080221 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.347073+0000", 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010177 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.347137+0000", 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3880000000000005e-05 2023-01-15T14:29:54.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.348015+0000", 2023-01-15T14:29:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087809499999999996 2023-01-15T14:29:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.348058+0000", 2023-01-15T14:29:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2868999999999999e-05 2023-01-15T14:29:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.348337+0000", 2023-01-15T14:29:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027911800000000002 2023-01-15T14:29:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.348368+0000", 2023-01-15T14:29:54.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0397999999999999e-05 2023-01-15T14:29:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.413446+0000", 2023-01-15T14:29:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.065078606999999997 2023-01-15T14:29:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:948 3.8 e83/41 3:17d5d741:::smithi09025264-5:head v 83'119, mlcod=82'115)", 2023-01-15T14:29:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:29:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.457378018, 2023-01-15T14:29:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201378735, 2023-01-15T14:29:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:29:54.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:29:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350496+0000", 2023-01-15T14:29:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:54.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350511+0000", 2023-01-15T14:29:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5007e-05 2023-01-15T14:29:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350535+0000", 2023-01-15T14:29:54.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4170000000000001e-05 2023-01-15T14:29:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.450633+0000", 2023-01-15T14:29:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100098221 2023-01-15T14:29:54.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.550879+0000", 2023-01-15T14:29:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100245538 2023-01-15T14:29:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.550917+0000", 2023-01-15T14:29:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8043000000000002e-05 2023-01-15T14:29:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551222+0000", 2023-01-15T14:29:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030465400000000001 2023-01-15T14:29:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551241+0000", 2023-01-15T14:29:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9049999999999999e-05 2023-01-15T14:29:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551331+0000", 2023-01-15T14:29:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0807e-05 2023-01-15T14:29:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551356+0000", 2023-01-15T14:29:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4786999999999998e-05 2023-01-15T14:29:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551876+0000", 2023-01-15T14:29:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051983800000000001 2023-01-15T14:29:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1124 3.5 e97/48 3:a75958b1:::smithi09025264-6:head v 97'44, mlcod=94'39)", 2023-01-15T14:29:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:29:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.6704709449999999, 2023-01-15T14:29:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200993378, 2023-01-15T14:29:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:29:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:29:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:29:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137455+0000", 2023-01-15T14:29:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1187000000000003e-05 2023-01-15T14:29:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137475+0000", 2023-01-15T14:29:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9692999999999999e-05 2023-01-15T14:29:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137496+0000", 2023-01-15T14:29:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1776e-05 2023-01-15T14:29:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238363+0000", 2023-01-15T14:29:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10086711900000001 2023-01-15T14:29:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238415+0000", 2023-01-15T14:29:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1465999999999998e-05 2023-01-15T14:29:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238578+0000", 2023-01-15T14:29:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016310599999999999 2023-01-15T14:29:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238598+0000", 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9729999999999999e-05 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238673+0000", 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5302000000000005e-05 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239068+0000", 2023-01-15T14:29:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000395227 2023-01-15T14:29:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.338398+0000", 2023-01-15T14:29:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099329534999999997 2023-01-15T14:29:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1125 3.5 e97/48 3:a75958b1:::smithi09025264-6:head v 97'45, mlcod=94'39)", 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.6703537320000006, 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201211626, 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:29:54.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:29:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137521+0000", 2023-01-15T14:29:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:54.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137531+0000", 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6399999999999992e-06 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137546+0000", 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5783000000000001e-05 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137557+0000", 2023-01-15T14:29:54.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0657e-05 2023-01-15T14:29:54.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238639+0000", 2023-01-15T14:29:54.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10108166 2023-01-15T14:29:54.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238688+0000", 2023-01-15T14:29:54.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9570000000000001e-05 2023-01-15T14:29:54.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238931+0000", 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000243255 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238948+0000", 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6246e-05 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238999+0000", 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1063000000000002e-05 2023-01-15T14:29:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239101+0000", 2023-01-15T14:29:54.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010191000000000001 2023-01-15T14:29:54.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.338733+0000", 2023-01-15T14:29:54.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099632581999999997 2023-01-15T14:29:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1119 3.8 e97/41 3:1eaad226:::smithi09025264-49:head v 97'157, mlcod=97'154)", 2023-01-15T14:29:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:29:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.6687354279999997, 2023-01-15T14:29:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200961838, 2023-01-15T14:29:54.666 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.666 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.666 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:29:54.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:29:54.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139139+0000", 2023-01-15T14:29:54.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:54.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139291+0000", 2023-01-15T14:29:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015233900000000001 2023-01-15T14:29:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139318+0000", 2023-01-15T14:29:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6265999999999999e-05 2023-01-15T14:29:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139331+0000", 2023-01-15T14:29:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3757000000000001e-05 2023-01-15T14:29:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238733+0000", 2023-01-15T14:29:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099401891000000006 2023-01-15T14:29:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238778+0000", 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5188999999999997e-05 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239141+0000", 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000362911 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239155+0000", 2023-01-15T14:29:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3879e-05 2023-01-15T14:29:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239274+0000", 2023-01-15T14:29:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011916 2023-01-15T14:29:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239310+0000", 2023-01-15T14:29:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5602999999999998e-05 2023-01-15T14:29:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.340102+0000", 2023-01-15T14:29:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10079187000000001 2023-01-15T14:29:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(3.0 REQUEST e99)", 2023-01-15T14:29:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:29:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.8114614580000001, 2023-01-15T14:29:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26432840200000002, 2023-01-15T14:29:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:29:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996419+0000", 2023-01-15T14:29:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6729999999999997e-06 2023-01-15T14:29:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996449+0000", 2023-01-15T14:29:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.004e-05 2023-01-15T14:29:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996463+0000", 2023-01-15T14:29:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3607e-05 2023-01-15T14:29:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996594+0000", 2023-01-15T14:29:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013148899999999999 2023-01-15T14:29:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:29:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996618+0000", 2023-01-15T14:29:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3249000000000001e-05 2023-01-15T14:29:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260645+0000", 2023-01-15T14:29:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26402768700000001 2023-01-15T14:29:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260671+0000", 2023-01-15T14:29:54.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.546e-05 2023-01-15T14:29:54.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260742+0000", 2023-01-15T14:29:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1569999999999994e-05 2023-01-15T14:29:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:261 3.8 e98/41 3:1e54a7f5:::smithi09025264-42:3e v 98'160, mlcod=97'157)", 2023-01-15T14:29:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:29:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.7503938689999998, 2023-01-15T14:29:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20299468400000001, 2023-01-15T14:29:54.682 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.682 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:29:54.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:29:54.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057480+0000", 2023-01-15T14:29:54.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:54.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057496+0000", 2023-01-15T14:29:54.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5869999999999999e-05 2023-01-15T14:29:54.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057528+0000", 2023-01-15T14:29:54.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:29:54.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057545+0000", 2023-01-15T14:29:54.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.694e-05 2023-01-15T14:29:54.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.158971+0000", 2023-01-15T14:29:54.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101425425 2023-01-15T14:29:54.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159011+0000", 2023-01-15T14:29:54.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0058000000000002e-05 2023-01-15T14:29:54.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159858+0000", 2023-01-15T14:29:54.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084767900000000005 2023-01-15T14:29:54.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159877+0000", 2023-01-15T14:29:54.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9133000000000001e-05 2023-01-15T14:29:54.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159959+0000", 2023-01-15T14:29:54.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1693000000000005e-05 2023-01-15T14:29:54.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260430+0000", 2023-01-15T14:29:54.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10047039200000001 2023-01-15T14:29:54.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260476+0000", 2023-01-15T14:29:54.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6652e-05 2023-01-15T14:29:54.695 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.695 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.695 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.696 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:29:54.696 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:29:54.696 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.7491727130000001, 2023-01-15T14:29:54.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:29:54.697 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.697 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.697 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:29:54.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:29:54.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:29:54.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:54.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:29:54.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:29:54.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:29:54.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:29:54.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:29:54.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:29:54.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:29:54.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:29:54.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:29:54.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:29:54.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:29:54.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:29:54.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:29:54.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:29:54.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:29:54.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:29:54.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:29:54.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:29:54.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:29:54.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:29:54.709 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.709 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "replica_scrub(pg: 3.0,from:0'0,to:98'189,epoch:99/95,start:3:00000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2023-01-15T14:29:54.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:29:54.711 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.4460327719999997, 2023-01-15T14:29:54.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20057929099999999, 2023-01-15T14:29:54.711 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.711 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:54.711 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:29:54.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:29:54.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361841+0000", 2023-01-15T14:29:54.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:54.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361855+0000", 2023-01-15T14:29:54.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3464e-05 2023-01-15T14:29:54.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361886+0000", 2023-01-15T14:29:54.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1389000000000001e-05 2023-01-15T14:29:54.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.462023+0000", 2023-01-15T14:29:54.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013720800000001 2023-01-15T14:29:54.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562267+0000", 2023-01-15T14:29:54.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100243392 2023-01-15T14:29:54.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562322+0000", 2023-01-15T14:29:54.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5075999999999999e-05 2023-01-15T14:29:54.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562422+0000", 2023-01-15T14:29:54.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010015599999999999 2023-01-15T14:29:54.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.721 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(2.1 GRANT e100)", 2023-01-15T14:29:54.721 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:29:54.722 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.4902104859999996, 2023-01-15T14:29:54.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20048780599999999, 2023-01-15T14:29:54.722 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.722 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:54.722 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:29:54.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:29:54.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317663+0000", 2023-01-15T14:29:54.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:54.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317674+0000", 2023-01-15T14:29:54.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.119e-05 2023-01-15T14:29:54.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317702+0000", 2023-01-15T14:29:54.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7183e-05 2023-01-15T14:29:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.417845+0000", 2023-01-15T14:29:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100142988 2023-01-15T14:29:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518107+0000", 2023-01-15T14:29:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100262461 2023-01-15T14:29:54.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518121+0000", 2023-01-15T14:29:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.449e-05 2023-01-15T14:29:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518124+0000", 2023-01-15T14:29:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1730000000000002e-06 2023-01-15T14:29:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518153+0000", 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9043999999999998e-05 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:194 3.4 e101/48 3:2022fdfa:::smithi09025264-46:43 v 101'104, mlcod=98'100)", 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.9631311020000002, 2023-01-15T14:29:54.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20670733199999999, 2023-01-15T14:29:54.728 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:54.728 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:54.728 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:54.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:54.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:29:54.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:54.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:29:54.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:54.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:54.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844743+0000", 2023-01-15T14:29:54.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:54.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:54.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844757+0000", 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.484e-05 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844787+0000", 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9113000000000001e-05 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.944923+0000", 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013656 2023-01-15T14:29:54.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:54.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.949975+0000", 2023-01-15T14:29:54.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050521400000000001 2023-01-15T14:29:54.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:54.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950015+0000", 2023-01-15T14:29:54.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9286999999999998e-05 2023-01-15T14:29:54.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950559+0000", 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054463500000000002 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950585+0000", 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6166e-05 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:54.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950670+0000", 2023-01-15T14:29:54.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4902999999999997e-05 2023-01-15T14:29:54.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:54.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051368+0000", 2023-01-15T14:29:54.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100697913 2023-01-15T14:29:54.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:54.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:54.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:54.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051452+0000", 2023-01-15T14:29:54.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3442000000000005e-05 2023-01-15T14:29:54.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.734 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:54.734 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:54.734 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:54.735 INFO:tasks.rados.rados.0.smithi090.stdout:446: done (1 left) 2023-01-15T14:29:54.735 INFO:tasks.rados.rados.0.smithi090.stdout:447: rollback oid 36 current snap is 41 2023-01-15T14:29:54.735 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 36 to 38 2023-01-15T14:29:54.735 INFO:tasks.rados.rados.0.smithi090.stdout:443: done (1 left) 2023-01-15T14:29:54.735 INFO:tasks.rados.rados.0.smithi090.stdout:448: write (excl) oid 48 current snap is 41 2023-01-15T14:29:54.736 INFO:tasks.rados.rados.0.smithi090.stdout:448: seq_num 188 ranges {759716=539284,2034850=614062} 2023-01-15T14:29:54.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:29:54.737 INFO:tasks.rados.rados.0.smithi090.stdout:448: writing smithi09025264-48 from 759716 to 1299000 tid 1 2023-01-15T14:29:54.862 INFO:tasks.rados.rados.0.smithi090.stdout:448: writing smithi09025264-48 from 2034850 to 2648912 tid 2 2023-01-15T14:29:54.862 INFO:tasks.rados.rados.0.smithi090.stdout:449: delete oid 14 current snap is 41 2023-01-15T14:29:54.863 INFO:tasks.rados.rados.0.smithi090.stdout:449: done (2 left) 2023-01-15T14:29:54.863 INFO:tasks.rados.rados.0.smithi090.stdout:450: copy_from oid 39 from oid 21 current snap is 41 2023-01-15T14:29:54.863 INFO:tasks.rados.rados.0.smithi090.stdout:451: read oid 40 snap 38 2023-01-15T14:29:54.863 INFO:tasks.rados.rados.0.smithi090.stdout:451: expect (ObjNum 39 snap 0 seq_num 39) 2023-01-15T14:29:54.863 INFO:tasks.rados.rados.0.smithi090.stdout:452: snap_create 2023-01-15T14:29:54.863 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:29:54.864 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:29:54.865 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:29:54.866 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:54.864+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:54.866 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:29:54.867 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:29:54.868 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2023-01-15T14:29:53.815620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":117263756,"num_objects":48,"num_object_clones":7,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":1,"num_read":1037,"num_read_kb":863110,"num_write":922,"num_write_kb":293329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":99,"num_bytes_recovered":123927711,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1780,"ondisk_log_size":1780,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":75,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":5958648,"kb_used_data":649944976,"kb_used_omap":8472,"kb_used_meta":0,"kb_avail":649937012,"statfs":{"total":671637155840,"available":665535500288,"internally_reserved":8155136,"allocated":665543655424,"data_stored":665543655424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8676243,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":24,"commit_latency_ns":0,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36623955,"num_objects":-23,"num_object_clones":-20,"num_object_copies":-69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":-3,"num_read":12,"num_read_kb":8,"num_write":37,"num_write_kb":10400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.068950"},"pg_stats":[{"pgid":"3.f","version":"102'99","reported_seq":354,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.659957+0000","last_change":"2023-01-15T14:29:49.659957+0000","last_active":"2023-01-15T14:29:49.659957+0000","last_peered":"2023-01-15T14:29:49.659957+0000","last_clean":"2023-01-15T14:29:49.659957+0000","last_became_active":"2023-01-15T14:28:23.846063+0000","last_became_peered":"2023-01-15T14:28:23.846063+0000","last_unstale":"2023-01-15T14:29:49.659957+0000","last_undegraded":"2023-01-15T14:29:49.659957+0000","last_fullsized":"2023-01-15T14:29:49.659957+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:52.368","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.80893161099999999,"stat_sum":{"num_bytes":7748027,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":48255,"num_write":50,"num_write_kb":19860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.e","version":"102'33","reported_seq":291,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.459839+0000","last_change":"2023-01-15T14:29:49.459839+0000","last_active":"2023-01-15T14:29:49.459839+0000","last_peered":"2023-01-15T14:29:49.459839+0000","last_clean":"2023-01-15T14:29:49.459839+0000","last_became_active":"2023-01-15T14:28:23.846858+0000","last_became_peered":"2023-01-15T14:28:23.846858+0000","last_unstale":"2023-01-15T14:29:49.459839+0000","last_undegraded":"2023-01-15T14:29:49.459839+0000","last_fullsized":"2023-01-15T14:29:49.459839+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.60824714800000002,"stat_sum":{"num_bytes":2974839,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":23059,"num_write":10,"num_write_kb":8092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.d","version":"102'115","reported_seq":403,"reported_epoch":102,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:49.462743+0000","last_change":"2023-01-15T14:29:49.459934+0000","last_active":"2023-01-15T14:29:49.462743+0000","last_peered":"2023-01-15T14:29:49.462743+0000","last_clean":"2023-01-15T14:29:49.462743+0000","last_became_active":"2023-01-15T14:28:54.932913+0000","last_became_peered":"2023-01-15T14:28:54.932913+0000","last_unstale":"2023-01-15T14:29:49.462743+0000","last_undegraded":"2023-01-15T14:29:49.462743+0000","last_fullsized":"2023-01-15T14:29:49.462743+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.60788848100000004,"stat_sum":{"num_bytes":8857256,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":87025,"num_write":47,"num_write_kb":18091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10638699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.c","version":"94'45","reported_seq":251,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.757660+0000","last_change":"2023-01-15T14:29:49.858455+0000","last_active":"2023-01-15T14:29:49.757660+0000","last_peered":"2023-01-15T14:29:49.757660+0000","last_clean":"2023-01-15T14:29:49.757660+0000","last_became_active":"2023-01-15T14:28:54.833568+0000","last_became_peered":"2023-01-15T14:28:54.833568+0000","last_unstale":"2023-01-15T14:29:49.757660+0000","last_undegraded":"2023-01-15T14:29:49.757660+0000","last_fullsized":"2023-01-15T14:29:49.757660+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:49.726","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.107442554,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":20,"num_write_kb":5044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":2319333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.b","version":"102'34","reported_seq":288,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.756823+0000","last_change":"2023-01-15T14:29:49.757556+0000","last_active":"2023-01-15T14:29:49.756823+0000","last_peered":"2023-01-15T14:29:49.756823+0000","last_clean":"2023-01-15T14:29:49.756823+0000","last_became_active":"2023-01-15T14:28:23.934183+0000","last_became_peered":"2023-01-15T14:28:23.934183+0000","last_unstale":"2023-01-15T14:29:49.756823+0000","last_undegraded":"2023-01-15T14:29:49.756823+0000","last_fullsized":"2023-01-15T14:29:49.756823+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066201799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":13386,"num_write":12,"num_write_kb":5195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.a","version":"102'146","reported_seq":426,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.748484+0000","last_change":"2023-01-15T14:29:49.665311+0000","last_active":"2023-01-15T14:29:49.748484+0000","last_peered":"2023-01-15T14:29:49.748484+0000","last_clean":"2023-01-15T14:29:49.748484+0000","last_became_active":"2023-01-15T14:29:41.237634+0000","last_became_peered":"2023-01-15T14:29:41.237634+0000","last_unstale":"2023-01-15T14:29:49.748484+0000","last_undegraded":"2023-01-15T14:29:49.748484+0000","last_fullsized":"2023-01-15T14:29:49.748484+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:30:41.499","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.91769121799999998,"stat_sum":{"num_bytes":9311045,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":33249,"num_write":74,"num_write_kb":33528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":36194185,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.9","version":"103'144","reported_seq":412,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:50.369614+0000","last_change":"2023-01-15T14:29:49.452673+0000","last_active":"2023-01-15T14:29:50.369614+0000","last_peered":"2023-01-15T14:29:50.369614+0000","last_clean":"2023-01-15T14:29:50.369614+0000","last_became_active":"2023-01-15T14:28:24.109843+0000","last_became_peered":"2023-01-15T14:28:24.109843+0000","last_unstale":"2023-01-15T14:29:50.369614+0000","last_undegraded":"2023-01-15T14:29:50.369614+0000","last_fullsized":"2023-01-15T14:29:50.369614+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.70611565099999996,"stat_sum":{"num_bytes":13086034,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":92,"num_read_kb":99494,"num_write":63,"num_write_kb":24682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.8","version":"102'170","reported_seq":464,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.767000+0000","last_change":"2023-01-15T14:29:49.767404+0000","last_active":"2023-01-15T14:29:49.767000+0000","last_peered":"2023-01-15T14:29:49.767000+0000","last_clean":"2023-01-15T14:29:49.767000+0000","last_became_active":"2023-01-15T14:28:47.937231+0000","last_became_peered":"2023-01-15T14:28:47.937231+0000","last_unstale":"2023-01-15T14:29:49.767000+0000","last_undegraded":"2023-01-15T14:29:49.767000+0000","last_fullsized":"2023-01-15T14:29:49.767000+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'121","last_scrub_stamp":"2023-01-15T14:29:31.589753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.589753+0000","objects_scrubbed":8,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:54.442","scrub_duration":0.28338864200000002,"objects_trimmed":0,"snaptrim_duration":0.016725107999999999,"stat_sum":{"num_bytes":8974905,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":73609,"num_write":75,"num_write_kb":23115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":12151296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":309,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:48.850460+0000","last_change":"2023-01-15T14:29:45.876275+0000","last_active":"2023-01-15T14:29:48.850460+0000","last_peered":"2023-01-15T14:29:48.850460+0000","last_clean":"2023-01-15T14:29:48.850460+0000","last_became_active":"2023-01-15T14:29:39.317067+0000","last_became_peered":"2023-01-15T14:29:39.317067+0000","last_unstale":"2023-01-15T14:29:48.850460+0000","last_undegraded":"2023-01-15T14:29:48.850460+0000","last_fullsized":"2023-01-15T14:29:48.850460+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:29:45.876183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:29:45.876183+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:46.712","scrub_duration":0.30530468399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":78,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.960969+0000","last_change":"2023-01-15T14:29:44.759819+0000","last_active":"2023-01-15T14:29:49.960969+0000","last_peered":"2023-01-15T14:29:49.960969+0000","last_clean":"2023-01-15T14:29:49.960969+0000","last_became_active":"2023-01-15T14:28:54.828857+0000","last_became_peered":"2023-01-15T14:28:54.828857+0000","last_unstale":"2023-01-15T14:29:49.960969+0000","last_undegraded":"2023-01-15T14:29:49.960969+0000","last_fullsized":"2023-01-15T14:29:49.960969+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:44.759745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:44.759745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:49.078","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"103'86","reported_seq":373,"reported_epoch":103,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:49.851089+0000","last_change":"2023-01-15T14:29:49.851089+0000","last_active":"2023-01-15T14:29:49.851089+0000","last_peered":"2023-01-15T14:29:49.851089+0000","last_clean":"2023-01-15T14:29:49.851089+0000","last_became_active":"2023-01-15T14:28:55.217912+0000","last_became_peered":"2023-01-15T14:28:55.217912+0000","last_unstale":"2023-01-15T14:29:49.851089+0000","last_undegraded":"2023-01-15T14:29:49.851089+0000","last_fullsized":"2023-01-15T14:29:49.851089+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.50858874799999998,"stat_sum":{"num_bytes":7663959,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":59,"num_read_kb":56238,"num_write":42,"num_write_kb":17290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":10821306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:48.850296+0000","last_change":"2023-01-15T14:29:31.583684+0000","last_active":"2023-01-15T14:29:48.850296+0000","last_peered":"2023-01-15T14:29:48.850296+0000","last_clean":"2023-01-15T14:29:48.850296+0000","last_became_active":"2023-01-15T14:28:13.013148+0000","last_became_peered":"2023-01-15T14:28:13.013148+0000","last_unstale":"2023-01-15T14:29:48.850296+0000","last_undegraded":"2023-01-15T14:29:48.850296+0000","last_fullsized":"2023-01-15T14:29:48.850296+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:31.583599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.583599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:44.691","scrub_duration":0.22681161699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"102'107","reported_seq":381,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.352829+0000","last_change":"2023-01-15T14:29:49.352829+0000","last_active":"2023-01-15T14:29:49.352829+0000","last_peered":"2023-01-15T14:29:49.352829+0000","last_clean":"2023-01-15T14:29:49.352829+0000","last_became_active":"2023-01-15T14:28:23.842545+0000","last_became_peered":"2023-01-15T14:28:23.842545+0000","last_unstale":"2023-01-15T14:29:49.352829+0000","last_undegraded":"2023-01-15T14:29:49.352829+0000","last_fullsized":"2023-01-15T14:29:49.352829+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'81","last_scrub_stamp":"2023-01-15T14:29:32.795334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:32.795334+0000","objects_scrubbed":4,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:33.824","scrub_duration":0.40728601599999997,"objects_trimmed":3,"snaptrim_duration":0.60542346199999997,"stat_sum":{"num_bytes":7092554,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":57,"num_read_kb":49747,"num_write":51,"num_write_kb":16252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":105,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.960944+0000","last_change":"2023-01-15T14:29:47.854985+0000","last_active":"2023-01-15T14:29:49.960944+0000","last_peered":"2023-01-15T14:29:49.960944+0000","last_clean":"2023-01-15T14:29:49.960944+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:29:49.960944+0000","last_undegraded":"2023-01-15T14:29:49.960944+0000","last_fullsized":"2023-01-15T14:29:49.960944+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:47.854866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:47.854866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:56.359","scrub_duration":0.63763962500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"104'205","reported_seq":585,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-01-15T14:29:51.366943+0000","last_change":"2023-01-15T14:29:50.259196+0000","last_active":"2023-01-15T14:29:51.366943+0000","last_peered":"2023-01-15T14:29:51.366943+0000","last_clean":"2023-01-15T14:29:51.366943+0000","last_became_active":"2023-01-15T14:29:42.036350+0000","last_became_peered":"2023-01-15T14:29:42.036350+0000","last_unstale":"2023-01-15T14:29:51.366943+0000","last_undegraded":"2023-01-15T14:29:51.366943+0000","last_fullsized":"2023-01-15T14:29:51.366943+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'189","last_scrub_stamp":"2023-01-15T14:29:46.834961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:46.834961+0000","objects_scrubbed":7,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:11.320","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.15519123300000001,"stat_sum":{"num_bytes":13005561,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":109,"num_read_kb":103781,"num_write":82,"num_write_kb":39968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":34269143,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.860893+0000","last_change":"2023-01-15T14:29:14.703116+0000","last_active":"2023-01-15T14:29:49.860893+0000","last_peered":"2023-01-15T14:29:49.860893+0000","last_clean":"2023-01-15T14:29:49.860893+0000","last_became_active":"2023-01-15T14:29:14.050302+0000","last_became_peered":"2023-01-15T14:29:14.050302+0000","last_unstale":"2023-01-15T14:29:49.860893+0000","last_undegraded":"2023-01-15T14:29:49.860893+0000","last_fullsized":"2023-01-15T14:29:49.860893+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:14.703035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:14.703035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:18.827","scrub_duration":0.44938580500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"104'198","reported_seq":489,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-01-15T14:29:51.553734+0000","last_change":"2023-01-15T14:29:49.350947+0000","last_active":"2023-01-15T14:29:51.553734+0000","last_peered":"2023-01-15T14:29:51.553734+0000","last_clean":"2023-01-15T14:29:51.553734+0000","last_became_active":"2023-01-15T14:28:23.946074+0000","last_became_peered":"2023-01-15T14:28:23.946074+0000","last_unstale":"2023-01-15T14:29:51.553734+0000","last_undegraded":"2023-01-15T14:29:51.553734+0000","last_fullsized":"2023-01-15T14:29:51.553734+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:30:41.151","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.60403308600000005,"stat_sum":{"num_bytes":17016491,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":156,"num_read_kb":150896,"num_write":112,"num_write_kb":31609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"3.5","version":"103'51","reported_seq":287,"reported_epoch":103,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:29:49.858683+0000","last_change":"2023-01-15T14:29:49.858683+0000","last_active":"2023-01-15T14:29:49.858683+0000","last_peered":"2023-01-15T14:29:49.858683+0000","last_clean":"2023-01-15T14:29:49.858683+0000","last_became_active":"2023-01-15T14:28:54.831872+0000","last_became_peered":"2023-01-15T14:28:54.831872+0000","last_unstale":"2023-01-15T14:29:49.858683+0000","last_undegraded":"2023-01-15T14:29:49.858683+0000","last_fullsized":"2023-01-15T14:29:49.858683+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.30997538800000002,"stat_sum":{"num_bytes":5815189,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":25834,"num_write":19,"num_write_kb":11085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4252105,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":105,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:50.204329+0000","last_change":"2023-01-15T14:29:29.346425+0000","last_active":"2023-01-15T14:29:50.204329+0000","last_peered":"2023-01-15T14:29:50.204329+0000","last_clean":"2023-01-15T14:29:50.204329+0000","last_became_active":"2023-01-15T14:28:12.896311+0000","last_became_peered":"2023-01-15T14:28:12.896311+0000","last_unstale":"2023-01-15T14:29:50.204329+0000","last_undegraded":"2023-01-15T14:29:50.204329+0000","last_fullsized":"2023-01-15T14:29:50.204329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:52.809","scrub_duration":0.39286881699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"104'116","reported_seq":412,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-01-15T14:29:51.466676+0000","last_change":"2023-01-15T14:29:50.305359+0000","last_active":"2023-01-15T14:29:51.466676+0000","last_peered":"2023-01-15T14:29:51.466676+0000","last_clean":"2023-01-15T14:29:51.466676+0000","last_became_active":"2023-01-15T14:28:54.825996+0000","last_became_peered":"2023-01-15T14:28:54.825996+0000","last_unstale":"2023-01-15T14:29:51.466676+0000","last_undegraded":"2023-01-15T14:29:51.466676+0000","last_fullsized":"2023-01-15T14:29:51.466676+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":51,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:30:48.052","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.201329761,"stat_sum":{"num_bytes":9766128,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":84,"num_read_kb":81294,"num_write":58,"num_write_kb":20345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":12100908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":104,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:48.951956+0000","last_change":"2023-01-15T14:29:25.664703+0000","last_active":"2023-01-15T14:29:48.951956+0000","last_peered":"2023-01-15T14:29:48.951956+0000","last_clean":"2023-01-15T14:29:48.951956+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:29:48.951956+0000","last_undegraded":"2023-01-15T14:29:48.951956+0000","last_fullsized":"2023-01-15T14:29:48.951956+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:25.664616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:25.664616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:29.565","scrub_duration":0.40398350300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"101'62","reported_seq":308,"reported_epoch":102,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:29:49.254023+0000","last_change":"2023-01-15T14:29:49.254023+0000","last_active":"2023-01-15T14:29:49.254023+0000","last_peered":"2023-01-15T14:29:49.254023+0000","last_clean":"2023-01-15T14:29:49.254023+0000","last_became_active":"2023-01-15T14:28:23.741266+0000","last_became_peered":"2023-01-15T14:28:23.741266+0000","last_unstale":"2023-01-15T14:29:49.254023+0000","last_undegraded":"2023-01-15T14:29:49.254023+0000","last_fullsized":"2023-01-15T14:29:49.254023+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:29:50.987","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50058042300000005,"stat_sum":{"num_bytes":2621859,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":4,"num_write":22,"num_write_kb":7260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"1"},{"start":"3e","length":"2"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":125,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:50.103748+0000","last_change":"2023-01-15T14:29:42.343333+0000","last_active":"2023-01-15T14:29:50.103748+0000","last_peered":"2023-01-15T14:29:50.103748+0000","last_clean":"2023-01-15T14:29:50.103748+0000","last_became_active":"2023-01-15T14:29:36.099234+0000","last_became_peered":"2023-01-15T14:29:36.099234+0000","last_unstale":"2023-01-15T14:29:50.103748+0000","last_undegraded":"2023-01-15T14:29:50.103748+0000","last_fullsized":"2023-01-15T14:29:50.103748+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:47.376","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":320,"reported_epoch":102,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.558857+0000","last_change":"2023-01-15T14:29:49.558857+0000","last_active":"2023-01-15T14:29:49.558857+0000","last_peered":"2023-01-15T14:29:49.558857+0000","last_clean":"2023-01-15T14:29:49.558857+0000","last_became_active":"2023-01-15T14:28:23.841189+0000","last_became_peered":"2023-01-15T14:28:23.841189+0000","last_unstale":"2023-01-15T14:29:49.558857+0000","last_undegraded":"2023-01-15T14:29:49.558857+0000","last_fullsized":"2023-01-15T14:29:49.558857+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'44","last_scrub_stamp":"2023-01-15T14:29:41.838167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:41.838167+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:01.701","scrub_duration":0.32316732300000001,"objects_trimmed":1,"snaptrim_duration":0.71081751100000001,"stat_sum":{"num_bytes":2739541,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":17136,"num_write":26,"num_write_kb":9497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2d","length":"1"},{"start":"3c","length":"4"},{"start":"41","length":"2"},{"start":"45","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":106,"reported_epoch":103,"state":"active+clean","last_fresh":"2023-01-15T14:29:49.748727+0000","last_change":"2023-01-15T14:29:42.671857+0000","last_active":"2023-01-15T14:29:49.748727+0000","last_peered":"2023-01-15T14:29:49.748727+0000","last_clean":"2023-01-15T14:29:49.748727+0000","last_became_active":"2023-01-15T14:28:12.895506+0000","last_became_peered":"2023-01-15T14:28:12.895506+0000","last_unstale":"2023-01-15T14:29:49.748727+0000","last_undegraded":"2023-01-15T14:29:49.748727+0000","last_fullsized":"2023-01-15T14:29:49.748727+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.671761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.671761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:45.805","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":116673388,"num_objects":46,"num_object_clones":7,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":1,"num_read":921,"num_read_kb":863010,"num_write":763,"num_write_kb":290913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":122746975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1663,"ondisk_log_size":1663,"up":48,"acting":48,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":830292,"kb_used_data":92869184,"kb_used_omap":908,"kb_used_meta":0,"kb_avail":92869088,"statfs":{"total":95948165120,"available":95097946112,"internally_reserved":98304,"allocated":95098044416,"data_stored":95098044416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":929978,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.0700000000000003,"5min":4.0700000000000003,"15min":4.0700000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":75.893000000000001,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":9.2089999999999996,"5min":9.2089999999999996,"15min":9.2089999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":71.308999999999997,"5min":71.308999999999997,"15min":71.308999999999997},"last":0.83699999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.5449999999999999,"5min":6.5449999999999999,"15min":6.5449999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":75.959999999999994,"5min":75.959999999999994,"15min":75.959999999999994},"last":0.68100000000000005},{"interface":"front","average":{"1min":6.2800000000000002,"5min":6.2800000000000002,"15min":6.2800000000000002},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":68.605000000000004,"5min":68.605000000000004,"15min":68.605000000000004},"last":0.66100000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.7910000000000004,"5min":4.7910000000000004,"15min":4.7910000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":94.733999999999995,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.70199999999999996},{"interface":"front","average":{"1min":4.1779999999999999,"5min":4.1779999999999999,"15min":4.1779999999999999},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":75.998999999999995,"5min":75.998999999999995,"15min":75.998999999999995},"last":0.72299999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":3.7770000000000001,"5min":3.7770000000000001,"15min":3.7770000000000001},"last":0.874},{"interface":"front","average":{"1min":4.6520000000000001,"5min":4.6520000000000001,"15min":4.6520000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":76.034999999999997,"5min":76.034999999999997,"15min":76.034999999999997},"last":0.75600000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":9.4160000000000004,"5min":9.4160000000000004,"15min":9.4160000000000004},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":76.658000000000001,"5min":76.658000000000001,"15min":76.658000000000001},"last":1.01},{"interface":"front","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":3.5129999999999999,"5min":3.5129999999999999,"15min":3.5129999999999999},"last":1.1080000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.327,"5min":6.327,"15min":6.327},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":68.679000000000002,"5min":68.679000000000002,"15min":68.679000000000002},"last":1.04},{"interface":"front","average":{"1min":4.9450000000000003,"5min":4.9450000000000003,"15min":4.9450000000000003},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":95.430000000000007,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.139}]},{"osd":6,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":8.5239999999999991,"5min":8.5239999999999991,"15min":8.5239999999999991},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":68.635999999999996,"5min":68.635999999999996,"15min":68.635999999999996},"last":1.093},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":3.6520000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":1.0740000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":856032,"kb_used_data":92844724,"kb_used_omap":1342,"kb_used_meta":0,"kb_avail":92843348,"statfs":{"total":95948165120,"available":95071588352,"internally_reserved":1409024,"allocated":95072997376,"data_stored":95072997376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1374949,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.1579999999999999,"5min":1.1579999999999999,"15min":1.1579999999999999},"last":0.628},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68400000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.871,"5min":0.871,"15min":0.871},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.77900000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":128.173},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":114.724}]},{"osd":3,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":6.0590000000000002,"5min":6.0590000000000002,"15min":6.0590000000000002},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":81.686999999999998,"5min":81.686999999999998,"15min":81.686999999999998},"last":0.86499999999999999},{"interface":"front","average":{"1min":11.455,"5min":11.455,"15min":11.455},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":255.24299999999999,"5min":255.24299999999999,"15min":255.24299999999999},"last":0.60599999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":6.6680000000000001,"5min":6.6680000000000001,"15min":6.6680000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":138.197,"5min":138.197,"15min":138.197},"last":0.79500000000000004}]},{"osd":5,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.83999999999999997}]},{"osd":7,"last update":"Sun Jan 15 14:29:07 2023","interfaces":[{"interface":"back","average":{"1min":4.2800000000000002,"5min":4.2800000000000002,"15min":4.2800000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":78.888000000000005,"5min":78.888000000000005,"15min":78.888000000000005},"last":0.91200000000000003},{"interface":"front","average":{"1min":3.0249999999999999,"5min":3.0249999999999999,"15min":3.0249999999999999},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":47.332000000000001,"5min":47.332000000000001,"15min":47.332000000000001},"last":13.951000000000001}]}]},{"osd":5,"up_from":0,"seq":55834574872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":871900,"kb_used_data":92831564,"kb_used_omap":1379,"kb_used_meta":0,"kb_avail":92827480,"statfs":{"total":95948165120,"available":95055339520,"internally_reserved":4182016,"allocated":95059521536,"data_stored":95059521536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1412241,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":7,"commit_latency_ns":0,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.915,"5min":4.915,"15min":4.915},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":86.944999999999993,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.83499999999999996}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":8.5939999999999994,"5min":8.5939999999999994,"15min":8.5939999999999994},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":99.960999999999999,"5min":99.960999999999999,"15min":99.960999999999999},"last":0.505},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.302,"5min":1.302,"15min":1.302},"last":0.68899999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.5110000000000001,"5min":5.5110000000000001,"15min":5.5110000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":99.884,"5min":99.884,"15min":99.884},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6080000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.55600000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.339,"15min":1.339},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":12.1,"5min":12.1,"15min":12.1},"last":0.87},{"interface":"front","average":{"1min":6.1509999999999998,"5min":6.1509999999999998,"15min":6.1509999999999998},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":99.921000000000006,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.78600000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":2.2989999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":30.971,"5min":30.971,"15min":30.971},"last":0.74199999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.673,"5min":5.673,"15min":5.673},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":99.780000000000001,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":5.1630000000000003,"5min":5.1630000000000003,"15min":5.1630000000000003},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":88.186999999999998,"5min":88.186999999999998,"15min":88.186999999999998},"last":0.95799999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.128,"5min":13.128,"15min":13.128},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":179.73500000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":11.239000000000001,"5min":11.239000000000001,"15min":11.239000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":99.843000000000004,"5min":99.843000000000004,"15min":99.843000000000004},"last":0.64500000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":852976,"kb_used_data":92846508,"kb_used_omap":1278,"kb_used_meta":0,"kb_avail":92846404,"statfs":{"total":95948165120,"available":95074717696,"internally_reserved":106496,"allocated":95074824192,"data_stored":95074824192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1309139,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.8479999999999999,"5min":4.8479999999999999,"15min":4.8479999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":93.632999999999996,"5min":93.632999999999996,"15min":93.632999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":7.6420000000000003,"5min":7.6420000000000003,"15min":7.6420000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":93.674000000000007,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.56000000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.532,"5min":13.532,"15min":13.532},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":159.256,"5min":159.256,"15min":159.256},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.88200000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.3490000000000002,"5min":3.3490000000000002,"15min":3.3490000000000002},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":59.896999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":31.233000000000001,"5min":31.233000000000001,"15min":31.233000000000001},"last":0.94999999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":6.5620000000000003,"5min":6.5620000000000003,"15min":6.5620000000000003},"min":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":66.311999999999998,"5min":66.311999999999998,"15min":66.311999999999998},"last":2.3010000000000002},{"interface":"front","average":{"1min":13.874000000000001,"5min":13.874000000000001,"15min":13.874000000000001},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":159.47200000000001,"5min":159.47200000000001,"15min":159.47200000000001},"last":2.5800000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":13.903,"15min":13.903},"min":{"1min":0.746,"5min":0.746,"15min":0.746},"max":{"1min":159.43000000000001,"5min":159.43000000000001,"15min":159.43000000000001},"last":1.355},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.276,"5min":1.276,"15min":1.276},"last":1.3839999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":1.3660000000000001,"5min":1.3660000000000001,"15min":1.3660000000000001},"last":1.6799999999999999},{"interface":"front","average":{"1min":3.762,"5min":3.762,"15min":3.762},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":60.098999999999997,"5min":60.098999999999997,"15min":60.098999999999997},"last":1.752}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.0359999999999996,"5min":4.0359999999999996,"15min":4.0359999999999996},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":60.165999999999997,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.0629999999999999},{"interface":"front","average":{"1min":18.09,"5min":18.09,"15min":18.09},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":159.512,"5min":159.512,"15min":159.512},"last":1.2390000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":853384,"kb_used_data":92846752,"kb_used_omap":977,"kb_used_meta":0,"kb_avail":92845996,"statfs":{"total":95948165120,"available":95074299904,"internally_reserved":774144,"allocated":95075074048,"data_stored":95075074048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1000540,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.499}]},{"osd":1,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.5979999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":143.071},{"interface":"front","average":{"1min":3.3559999999999999,"5min":3.3559999999999999,"15min":3.3559999999999999},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":47.518000000000001,"5min":47.518000000000001,"15min":47.518000000000001},"last":0.65000000000000002}]},{"osd":3,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":10.401999999999999,"5min":10.401999999999999,"15min":10.401999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":183.53,"5min":183.53,"15min":183.53},"last":143.13300000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.75}]},{"osd":4,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":3.6499999999999999,"15min":3.6499999999999999},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":53.139000000000003,"5min":53.139000000000003,"15min":53.139000000000003},"last":1.0069999999999999},{"interface":"front","average":{"1min":3.8090000000000002,"5min":3.8090000000000002,"15min":3.8090000000000002},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":53.186999999999998,"5min":53.186999999999998,"15min":53.186999999999998},"last":0.96399999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.7489999999999997,"5min":4.7489999999999997,"15min":4.7489999999999997},"last":143.262},{"interface":"front","average":{"1min":1.161,"5min":1.161,"15min":1.161},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":4.0899999999999999,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.78900000000000003}]},{"osd":6,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":2.141,"5min":2.141,"15min":2.141},"last":0.877},{"interface":"front","average":{"1min":1.218,"5min":1.218,"15min":1.218},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":143.215}]},{"osd":7,"last update":"Sun Jan 15 14:29:04 2023","interfaces":[{"interface":"back","average":{"1min":3.0590000000000002,"5min":3.0590000000000002,"15min":3.0590000000000002},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":37.124000000000002,"5min":37.124000000000002,"15min":37.124000000000002},"last":0.92800000000000005},{"interface":"front","average":{"1min":2.7770000000000001,"5min":2.7770000000000001,"15min":2.7770000000000001},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":30.445,"5min":30.445,"15min":30.445},"last":143.17099999999999}]}]},{"osd":1,"up_from":13,"seq":55834574872,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":851088,"kb_used_data":92849716,"kb_used_omap":1272,"kb_used_meta":0,"kb_avail":92848292,"statfs":{"total":95948165120,"available":95076651008,"internally_reserved":1458176,"allocated":95078109184,"data_stored":95078109184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1303332,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":3.8929999999999998,"5min":3.8929999999999998,"15min":3.8929999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":46.591000000000001,"5min":46.591000000000001,"15min":46.591000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":50.616,"5min":50.616,"15min":50.616},"last":0.69099999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":7.3470000000000004,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.75800000000000001},{"interface":"front","average":{"1min":4.0730000000000004,"5min":4.0730000000000004,"15min":4.0730000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":46.747999999999998,"5min":46.747999999999998,"15min":46.747999999999998},"last":0.625}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":19.835999999999999,"5min":19.835999999999999,"15min":19.835999999999999},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":282.01400000000001,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":7.2450000000000001,"5min":7.2450000000000001,"15min":7.2450000000000001},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":53.145000000000003,"5min":53.145000000000003,"15min":53.145000000000003},"last":1.038}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":4.7279999999999998,"5min":4.7279999999999998,"15min":4.7279999999999998},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":50.887999999999998,"5min":50.887999999999998,"15min":50.887999999999998},"last":0.78900000000000003},{"interface":"front","average":{"1min":3.605,"5min":3.605,"15min":3.605},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":52.512,"5min":52.512,"15min":52.512},"last":0.96199999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7390000000000001,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":4.4379999999999997,"5min":4.4379999999999997,"15min":4.4379999999999997},"min":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"max":{"1min":46.905999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.1080000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":5.0019999999999998,"5min":5.0019999999999998,"15min":5.0019999999999998},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":50.844999999999999,"5min":50.844999999999999,"15min":50.844999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.4540000000000002,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.022}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":7.0170000000000003,"5min":7.0170000000000003,"15min":7.0170000000000003},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":52.595999999999997,"5min":52.595999999999997,"15min":52.595999999999997},"last":1.0660000000000001},{"interface":"front","average":{"1min":4.75,"5min":4.75,"15min":4.75},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":50.804000000000002,"5min":50.804000000000002,"15min":50.804000000000002},"last":0.98799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574872,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":842976,"kb_used_data":92856528,"kb_used_omap":1314,"kb_used_meta":0,"kb_avail":92856404,"statfs":{"total":95948165120,"available":95084957696,"internally_reserved":126976,"allocated":95085084672,"data_stored":95085084672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1346064,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.504}]},{"osd":2,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.53700000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":16.779,"5min":16.779,"15min":16.779},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":284.62799999999999,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.73699999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"min":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":1.2}]},{"osd":5,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":9.798,"5min":9.798,"15min":9.798},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":176.34399999999999,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.9450000000000001,"5min":1.9450000000000001,"15min":1.9450000000000001},"last":0.95599999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.9930000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.268},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.93899999999999995}]},{"osd":7,"last update":"Sun Jan 15 14:29:06 2023","interfaces":[{"interface":"back","average":{"1min":15.670999999999999,"5min":15.670999999999999,"15min":15.670999999999999},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":292.61099999999999,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.171,"15min":1.171},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.1070000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.3089999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:29:54.889 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:29:55.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 26 2023-01-15T14:29:55.088 INFO:tasks.rados.rados.0.smithi090.stdout:447: finishing rollback tid 0 to smithi09025264-36 2023-01-15T14:29:55.088 INFO:tasks.rados.rados.0.smithi090.stdout:451: expect (ObjNum 39 snap 0 seq_num 39) 2023-01-15T14:29:55.088 INFO:tasks.rados.rados.0.smithi090.stdout:451: done (3 left) 2023-01-15T14:29:55.088 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:29:55.088 INFO:tasks.rados.rados.0.smithi090.stdout:447: finishing rollback tid 1 to smithi09025264-36 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:447: finishing rollback tid 2 to smithi09025264-36 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 147 (ObjNum 35 snap 0 seq_num 35) dirty exists 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:447: done (2 left) 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:448: finishing write tid 1 to smithi09025264-48 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:448: finishing write tid 2 to smithi09025264-48 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:448: finishing write tid 3 to smithi09025264-48 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:448: finishing write tid 4 to smithi09025264-48 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:448: oid 48 updating version 0 to 119 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:448: oid 48 version 119 is already newer than 117 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:448: oid 48 version 119 is already newer than 118 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 119 (ObjNum 188 snap 41 seq_num 188) dirty exists 2023-01-15T14:29:55.089 INFO:tasks.rados.rados.0.smithi090.stdout:448: left oid 48 (ObjNum 188 snap 41 seq_num 188) 2023-01-15T14:29:55.090 INFO:tasks.rados.rados.0.smithi090.stdout:448: done (1 left) 2023-01-15T14:29:55.090 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:55.090 INFO:tasks.rados.rados.0.smithi090.stdout:450: finishing copy_from to smithi09025264-39 2023-01-15T14:29:55.090 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 35 (ObjNum 83 snap 11 seq_num 83) dirty exists 2023-01-15T14:29:55.090 INFO:tasks.rados.rados.0.smithi090.stdout:450: finishing copy_from racing read to smithi09025264-39 2023-01-15T14:29:55.090 INFO:tasks.rados.rados.0.smithi090.stdout:450: done (0 left) 2023-01-15T14:29:55.090 INFO:tasks.rados.rados.0.smithi090.stdout:452: done (0 left) 2023-01-15T14:29:55.090 INFO:tasks.rados.rados.0.smithi090.stdout:453: snap_remove snap 41 2023-01-15T14:29:55.092 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:55.084+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:55.253 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:29:55.258 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:55.258 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:29:55.259 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:55.259 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:55.260 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:55.248+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:55.265 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:55.283 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:55.441 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:55.441 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:55.441 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:55.441 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:55.442 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:29:55.442 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:29:55.442 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:29:55.442 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:29:55.442 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:29:55.442 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:29:55.442 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:29:55.442 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:29:55.443 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:55.443 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:55.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:29:55.445 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:55.440+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:55.542 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:55.541+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:55.625 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:55.625 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:55.625 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:29:55.643 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:55.641+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:55.741 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:55.804 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:55.805 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:29:55.807 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:55.801+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:55.838 INFO:tasks.rados.rados.0.smithi090.stdout:453: done (0 left) 2023-01-15T14:29:55.978 INFO:tasks.rados.rados.0.smithi090.stdout:454: read oid 35 snap -1 2023-01-15T14:29:55.978 INFO:tasks.rados.rados.0.smithi090.stdout:454: expect deleted 2023-01-15T14:29:55.978 INFO:tasks.rados.rados.0.smithi090.stdout:455: read oid 23 snap -1 2023-01-15T14:29:55.978 INFO:tasks.rados.rados.0.smithi090.stdout:455: expect deleted 2023-01-15T14:29:55.979 INFO:tasks.rados.rados.0.smithi090.stdout:456: snap_remove snap 38 2023-01-15T14:29:55.981 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:29:55.981 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 26 2023-01-15T14:29:55.982 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:55.983 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:55.983 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:29:55.985 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:55.977+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:56.081 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:29:56.077+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:29:56.092 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:56.093 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:56.102 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:29:56.182 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:56.181+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:56.209 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:56.209 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:56.219 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:29:56.284 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:56.280+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:56.328 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:56.328 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:56.338 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:29:56.385 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:56.381+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:56.444 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:56.445 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:56.454 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:29:56.486 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:29:56.485+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:29:56.556 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:56.556 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:56.566 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:29:56.587 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:56.584+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:56.672 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:56.672 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:56.681 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:29:56.725 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:56.724+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:56.742 INFO:tasks.rados.rados.0.smithi090.stdout:456: done (2 left) 2023-01-15T14:29:56.743 INFO:tasks.rados.rados.0.smithi090.stdout:457: write oid 6 current snap is 42 2023-01-15T14:29:56.743 INFO:tasks.rados.rados.0.smithi090.stdout:457: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2023-01-15T14:29:56.760 INFO:tasks.rados.rados.0.smithi090.stdout:457: writing smithi09025264-6 from 793330 to 1483112 tid 1 2023-01-15T14:29:56.767 INFO:tasks.rados.rados.0.smithi090.stdout:457: writing smithi09025264-6 from 2035461 to 2653192 tid 2 2023-01-15T14:29:56.771 INFO:tasks.rados.rados.0.smithi090.stdout:457: writing smithi09025264-6 from 3101545 to 3193330 tid 3 2023-01-15T14:29:56.771 INFO:tasks.rados.rados.0.smithi090.stdout:458: write (excl) oid 17 current snap is 42 2023-01-15T14:29:56.771 INFO:tasks.rados.rados.0.smithi090.stdout:458: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2023-01-15T14:29:56.776 INFO:tasks.rados.rados.0.smithi090.stdout:458: writing smithi09025264-17 from 410137 to 891521 tid 1 2023-01-15T14:29:56.783 INFO:tasks.rados.rados.0.smithi090.stdout:458: writing smithi09025264-17 from 1410296 to 2171685 tid 2 2023-01-15T14:29:56.789 INFO:tasks.rados.rados.0.smithi090.stdout:458: writing smithi09025264-17 from 2728968 to 3210137 tid 3 2023-01-15T14:29:56.789 INFO:tasks.rados.rados.0.smithi090.stdout:459: read oid 46 snap -1 2023-01-15T14:29:56.789 INFO:tasks.rados.rados.0.smithi090.stdout:459: expect (ObjNum 83 snap 11 seq_num 83) 2023-01-15T14:29:56.789 INFO:tasks.rados.rados.0.smithi090.stdout:460: read oid 49 snap -1 2023-01-15T14:29:56.790 INFO:tasks.rados.rados.0.smithi090.stdout:460: expect (ObjNum 174 snap 37 seq_num 174) 2023-01-15T14:29:56.790 INFO:tasks.rados.rados.0.smithi090.stdout:461: write oid 19 current snap is 42 2023-01-15T14:29:56.790 INFO:tasks.rados.rados.0.smithi090.stdout:461: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2023-01-15T14:29:56.790 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:56.785+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:56.795 INFO:tasks.rados.rados.0.smithi090.stdout:461: writing smithi09025264-19 from 426944 to 983577 tid 1 2023-01-15T14:29:56.801 INFO:tasks.rados.rados.0.smithi090.stdout:461: writing smithi09025264-19 from 1468778 to 2090178 tid 2 2023-01-15T14:29:56.808 INFO:tasks.rados.rados.0.smithi090.stdout:461: writing smithi09025264-19 from 2872744 to 3226944 tid 3 2023-01-15T14:29:56.808 INFO:tasks.rados.rados.0.smithi090.stdout:462: rollback oid 33 current snap is 42 2023-01-15T14:29:56.808 INFO:tasks.rados.rados.0.smithi090.stdout:462: done (7 left) 2023-01-15T14:29:56.808 INFO:tasks.rados.rados.0.smithi090.stdout:463: snap_create 2023-01-15T14:29:56.808 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:29:56.808 INFO:tasks.rados.rados.0.smithi090.stdout:455: done (6 left) 2023-01-15T14:29:56.808 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:29:56.808 INFO:tasks.rados.rados.0.smithi090.stdout:454: done (5 left) 2023-01-15T14:29:56.808 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:29:56.829 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:56.829 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:56.839 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:29:56.869 INFO:tasks.rados.rados.0.smithi090.stdout:457: finishing write tid 1 to smithi09025264-6 2023-01-15T14:29:56.879 INFO:tasks.rados.rados.0.smithi090.stdout:460: expect (ObjNum 174 snap 37 seq_num 174) 2023-01-15T14:29:56.890 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:56.888+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:56.942 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:56.942 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:56.954 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:29:56.989 INFO:tasks.rados.rados.0.smithi090.stdout:459: expect (ObjNum 83 snap 11 seq_num 83) 2023-01-15T14:29:56.997 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:56.996+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:57.091 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:29:57.089+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:29:57.254 INFO:tasks.rados.rados.0.smithi090.stdout:461: finishing write tid 1 to smithi09025264-19 2023-01-15T14:29:57.255 INFO:tasks.rados.rados.0.smithi090.stdout:458: finishing write tid 1 to smithi09025264-17 2023-01-15T14:29:57.255 INFO:tasks.rados.rados.0.smithi090.stdout:461: finishing write tid 2 to smithi09025264-19 2023-01-15T14:29:57.255 INFO:tasks.rados.rados.0.smithi090.stdout:461: finishing write tid 3 to smithi09025264-19 2023-01-15T14:29:57.255 INFO:tasks.rados.rados.0.smithi090.stdout:457: finishing write tid 2 to smithi09025264-6 2023-01-15T14:29:57.255 INFO:tasks.rados.rados.0.smithi090.stdout:461: finishing write tid 4 to smithi09025264-19 2023-01-15T14:29:57.255 INFO:tasks.rados.rados.0.smithi090.stdout:461: finishing write tid 5 to smithi09025264-19 2023-01-15T14:29:57.255 INFO:tasks.rados.rados.0.smithi090.stdout:461: oid 19 updating version 0 to 102 2023-01-15T14:29:57.255 INFO:tasks.rados.rados.0.smithi090.stdout:461: oid 19 version 102 is already newer than 101 2023-01-15T14:29:57.256 INFO:tasks.rados.rados.0.smithi090.stdout:461: oid 19 version 102 is already newer than 100 2023-01-15T14:29:57.256 INFO:tasks.rados.rados.0.smithi090.stdout:461: oid 19 updating version 102 to 103 2023-01-15T14:29:57.256 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 103 (ObjNum 192 snap 42 seq_num 192) dirty exists 2023-01-15T14:29:57.256 INFO:tasks.rados.rados.0.smithi090.stdout:461: left oid 19 (ObjNum 192 snap 42 seq_num 192) 2023-01-15T14:29:57.256 INFO:tasks.rados.rados.0.smithi090.stdout:458: finishing write tid 2 to smithi09025264-17 2023-01-15T14:29:57.256 INFO:tasks.rados.rados.0.smithi090.stdout:457: finishing write tid 3 to smithi09025264-6 2023-01-15T14:29:57.256 INFO:tasks.rados.rados.0.smithi090.stdout:459: done (4 left) 2023-01-15T14:29:57.256 INFO:tasks.rados.rados.0.smithi090.stdout:460: done (3 left) 2023-01-15T14:29:57.256 INFO:tasks.rados.rados.0.smithi090.stdout:461: done (2 left) 2023-01-15T14:29:57.256 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:29:57.256 INFO:tasks.rados.rados.0.smithi090.stdout:458: finishing write tid 3 to smithi09025264-17 2023-01-15T14:29:57.257 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:57.253+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:57.257 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:57.257 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:57.258 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:29:57.259 INFO:tasks.rados.rados.0.smithi090.stdout:457: finishing write tid 4 to smithi09025264-6 2023-01-15T14:29:57.259 INFO:tasks.rados.rados.0.smithi090.stdout:457: finishing write tid 5 to smithi09025264-6 2023-01-15T14:29:57.259 INFO:tasks.rados.rados.0.smithi090.stdout:457: oid 6 updating version 0 to 57 2023-01-15T14:29:57.259 INFO:tasks.rados.rados.0.smithi090.stdout:457: oid 6 updating version 57 to 59 2023-01-15T14:29:57.259 INFO:tasks.rados.rados.0.smithi090.stdout:457: oid 6 version 59 is already newer than 58 2023-01-15T14:29:57.259 INFO:tasks.rados.rados.0.smithi090.stdout:457: oid 6 updating version 59 to 60 2023-01-15T14:29:57.259 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 60 (ObjNum 190 snap 42 seq_num 190) dirty exists 2023-01-15T14:29:57.259 INFO:tasks.rados.rados.0.smithi090.stdout:457: left oid 6 (ObjNum 190 snap 42 seq_num 190) 2023-01-15T14:29:57.259 INFO:tasks.rados.rados.0.smithi090.stdout:457: done (1 left) 2023-01-15T14:29:57.259 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:29:57.290 INFO:tasks.rados.rados.0.smithi090.stdout:458: finishing write tid 4 to smithi09025264-17 2023-01-15T14:29:57.292 INFO:tasks.rados.rados.0.smithi090.stdout:458: finishing write tid 5 to smithi09025264-17 2023-01-15T14:29:57.292 INFO:tasks.rados.rados.0.smithi090.stdout:458: oid 17 updating version 0 to 212 2023-01-15T14:29:57.292 INFO:tasks.rados.rados.0.smithi090.stdout:458: oid 17 version 212 is already newer than 210 2023-01-15T14:29:57.292 INFO:tasks.rados.rados.0.smithi090.stdout:458: oid 17 updating version 212 to 213 2023-01-15T14:29:57.293 INFO:tasks.rados.rados.0.smithi090.stdout:458: oid 17 version 213 is already newer than 211 2023-01-15T14:29:57.293 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 213 (ObjNum 191 snap 42 seq_num 191) dirty exists 2023-01-15T14:29:57.293 INFO:tasks.rados.rados.0.smithi090.stdout:458: left oid 17 (ObjNum 191 snap 42 seq_num 191) 2023-01-15T14:29:57.293 INFO:tasks.rados.rados.0.smithi090.stdout:458: done (0 left) 2023-01-15T14:29:57.355 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:57.352+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:57.375 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:57.375 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:57.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:29:57.445 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:29:57.456 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:57.452+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:57.501 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:29:57.501 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:29:57.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:29:57.557 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:29:57.556+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:29:57.627 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:29:57.627 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:29:57.627 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:29:57.627 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:57.628 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:29:57.629 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:29:57.645 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:29:57.658 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:29:57.656+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:29:57.674 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:29:57.683 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:29:57.710 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:29:57.817 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:29:57.820 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:57.821 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:29:57.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:29:57.821 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:57.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.821 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:30 3.1 3:8b0a2647:::smithi09025264-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:57.822 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:29:57.822 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.871696513000003, 2023-01-15T14:29:57.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40319718700000001, 2023-01-15T14:29:57.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:57.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:57.822 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:57.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:57.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:57.823 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 30 2023-01-15T14:29:57.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.823 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:57.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:57.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:29:57.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:57.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:57.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:29:57.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:57.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:57.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859909+0000", 2023-01-15T14:29:57.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:29:57.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:57.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.860361+0000", 2023-01-15T14:29:57.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045164599999999999 2023-01-15T14:29:57.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:57.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.860392+0000", 2023-01-15T14:29:57.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1331000000000002e-05 2023-01-15T14:29:57.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:57.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960486+0000", 2023-01-15T14:29:57.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100093583 2023-01-15T14:29:57.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:57.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.160839+0000", 2023-01-15T14:29:57.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20035291199999999 2023-01-15T14:29:57.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:57.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.160982+0000", 2023-01-15T14:29:57.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014350800000000001 2023-01-15T14:29:57.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:57.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161183+0000", 2023-01-15T14:29:57.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000200411 2023-01-15T14:29:57.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:57.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161525+0000", 2023-01-15T14:29:57.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034216500000000001 2023-01-15T14:29:57.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:57.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161587+0000", 2023-01-15T14:29:57.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2032999999999994e-05 2023-01-15T14:29:57.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:57.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161842+0000", 2023-01-15T14:29:57.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000255305 2023-01-15T14:29:57.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:57.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262815+0000", 2023-01-15T14:29:57.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100972992 2023-01-15T14:29:57.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:57.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262914+0000", 2023-01-15T14:29:57.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9166999999999994e-05 2023-01-15T14:29:57.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:57.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262967+0000", 2023-01-15T14:29:57.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2244000000000003e-05 2023-01-15T14:29:57.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263050+0000", 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2999999999999998e-05 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263110+0000", 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0297000000000002e-05 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:57.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:57.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.984 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:31 3.1 3:8b0a2647:::smithi09025264-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:57.994 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:29:57.994 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.771094146999999, 2023-01-15T14:29:57.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405055944, 2023-01-15T14:29:57.995 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:57.995 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:57.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:57.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:57.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:57.995 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 31 2023-01-15T14:29:57.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.996 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:57.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:57.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:29:57.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:57.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:57.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:29:57.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:57.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:57.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960513+0000", 2023-01-15T14:29:57.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:57.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:57.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960524+0000", 2023-01-15T14:29:57.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0614e-05 2023-01-15T14:29:57.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:57.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:57.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:57.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960538+0000", 2023-01-15T14:29:57.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4226999999999999e-05 2023-01-15T14:29:57.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060704+0000", 2023-01-15T14:29:58.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166252 2023-01-15T14:29:58.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.261723+0000", 2023-01-15T14:29:58.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20101896699999999 2023-01-15T14:29:58.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.261869+0000", 2023-01-15T14:29:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014578099999999999 2023-01-15T14:29:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262092+0000", 2023-01-15T14:29:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223198 2023-01-15T14:29:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262397+0000", 2023-01-15T14:29:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030490500000000001 2023-01-15T14:29:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:58.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262446+0000", 2023-01-15T14:29:58.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8492999999999998e-05 2023-01-15T14:29:58.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:58.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262537+0000", 2023-01-15T14:29:58.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1248000000000005e-05 2023-01-15T14:29:58.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:58.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262847+0000", 2023-01-15T14:29:58.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031007799999999998 2023-01-15T14:29:58.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365234+0000", 2023-01-15T14:29:58.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102386856 2023-01-15T14:29:58.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365326+0000", 2023-01-15T14:29:58.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2226999999999996e-05 2023-01-15T14:29:58.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:58.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365413+0000", 2023-01-15T14:29:58.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6996999999999994e-05 2023-01-15T14:29:58.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365571+0000", 2023-01-15T14:29:58.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015794100000000001 2023-01-15T14:29:58.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.031 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.031 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:53 3.1 3:883e9a29:::smithi09025264-14:head [write 618491~694590,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:58.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:29:58.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.670785871000007, 2023-01-15T14:29:58.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659587899999998, 2023-01-15T14:29:58.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:58.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.033 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 53 2023-01-15T14:29:58.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.033 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:29:58.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060823+0000", 2023-01-15T14:29:58.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060822+0000", 2023-01-15T14:29:58.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:58.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061225+0000", 2023-01-15T14:29:58.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040297199999999998 2023-01-15T14:29:58.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061244+0000", 2023-01-15T14:29:58.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:29:58.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061253+0000", 2023-01-15T14:29:58.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8200000000000003e-06 2023-01-15T14:29:58.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263154+0000", 2023-01-15T14:29:58.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20190054900000001 2023-01-15T14:29:58.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263511+0000", 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035690599999999999 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263928+0000", 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041701200000000003 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:58.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264611+0000", 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068358700000000004 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264661+0000", 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.952e-05 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.264962+0000", 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030116799999999999 2023-01-15T14:29:58.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:58.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365102+0000", 2023-01-15T14:29:58.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100139698 2023-01-15T14:29:58.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366354+0000", 2023-01-15T14:29:58.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001252059 2023-01-15T14:29:58.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457209+0000", 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090855667000000001 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457307+0000", 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7621000000000006e-05 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457419+0000", 2023-01-15T14:29:58.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112246 2023-01-15T14:29:58.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.054 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.162 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:29:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.670344417999999, 2023-01-15T14:29:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:29:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.163 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:29:58.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:29:58.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:29:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:29:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:58.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:29:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:29:58.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:29:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:29:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:29:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:29:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:29:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:29:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:29:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:29:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:29:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:29:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:29:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:29:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:58.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:29:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:29:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:58.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:29:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:29:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:58.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:29:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:29:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:29:58.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:29:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:58.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:29:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:29:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:29:58.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:29:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:29:58.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:29:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:29:58.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:29:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:29:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:29:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:29:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.670064170000003, 2023-01-15T14:29:58.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:29:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.203 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.204 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:29:58.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.204 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:29:58.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:29:58.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:29:58.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:58.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:29:58.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:29:58.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:29:58.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:29:58.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:29:58.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:29:58.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:29:58.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:29:58.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:29:58.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:29:58.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:29:58.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:29:58.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:29:58.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:29:58.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:58.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:29:58.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:29:58.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:58.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:29:58.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:29:58.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:58.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:29:58.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:29:58.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:58.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:29:58.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:29:58.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:58.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:29:58.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:29:58.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:29:58.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:29:58.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:29:58.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:29:58.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:58.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:29:58.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:29:58.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:29:58.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:29:58.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.355 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.356 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.356 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:58.356 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:29:58.357 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.670031553000001, 2023-01-15T14:29:58.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:29:58.368 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.368 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:58.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:29:58.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:29:58.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:29:58.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:29:58.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:29:58.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:29:58.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:29:58.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:29:58.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:29:58.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:29:58.375 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.375 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.375 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.375 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:58 3.1 3:86bba67f:::smithi09025264-15:head [write 2069689~165609] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:58.375 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:29:58.375 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.568860987999997, 2023-01-15T14:29:58.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39908735699999998, 2023-01-15T14:29:58.375 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.396 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:58.396 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.396 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.396 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.396 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 58 2023-01-15T14:29:58.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.396 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:29:58.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162748+0000", 2023-01-15T14:29:58.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162747+0000", 2023-01-15T14:29:58.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:58.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162859+0000", 2023-01-15T14:29:58.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112137 2023-01-15T14:29:58.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162874+0000", 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4793999999999999e-05 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162883+0000", 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5559999999999996e-06 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390699+0000", 2023-01-15T14:29:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.227815712 2023-01-15T14:29:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390900+0000", 2023-01-15T14:29:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000201518 2023-01-15T14:29:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458887+0000", 2023-01-15T14:29:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067986183000000006 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459000+0000", 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011291800000000001 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459244+0000", 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024485000000000002 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459480+0000", 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023518599999999999 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459498+0000", 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8892000000000001e-05 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459822+0000", 2023-01-15T14:29:58.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032378700000000002 2023-01-15T14:29:58.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:58.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460258+0000", 2023-01-15T14:29:58.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043563500000000003 2023-01-15T14:29:58.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465816+0000", 2023-01-15T14:29:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055581099999999998 2023-01-15T14:29:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561720+0000", 2023-01-15T14:29:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095903587999999998 2023-01-15T14:29:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561771+0000", 2023-01-15T14:29:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1433e-05 2023-01-15T14:29:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561836+0000", 2023-01-15T14:29:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4591000000000006e-05 2023-01-15T14:29:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:58.570 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:29:58.570 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.568713567000003, 2023-01-15T14:29:58.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:29:58.570 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.570 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:58.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.570 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:29:58.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.570 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:29:58.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:29:58.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:29:58.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:58.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:29:58.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:29:58.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:29:58.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:29:58.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:29:58.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:29:58.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:29:58.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:29:58.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:29:58.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:29:58.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:29:58.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:58.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:29:58.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:29:58.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:58.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:29:58.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:29:58.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:29:58.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:29:58.579 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.568683563999997, 2023-01-15T14:29:58.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:29:58.580 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.580 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:58.580 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.580 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.580 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.580 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:29:58.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.580 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:29:58.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:29:58.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:29:58.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:29:58.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:29:58.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:29:58.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:29:58.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:29:58.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:29:58.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:29:58.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:29:58.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:29:58.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:29:58.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.587 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.587 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:125 3.1 3:8889e450:::smithi09025264-32:head [write 521017~798131,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:58.587 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:29:58.587 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.058794746000004, 2023-01-15T14:29:58.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40348810600000001, 2023-01-15T14:29:58.587 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 125 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:29:58.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672812+0000", 2023-01-15T14:29:58.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:58.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673338+0000", 2023-01-15T14:29:58.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052553599999999995 2023-01-15T14:29:58.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673372+0000", 2023-01-15T14:29:58.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3652999999999998e-05 2023-01-15T14:29:58.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673386+0000", 2023-01-15T14:29:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4171e-05 2023-01-15T14:29:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773535+0000", 2023-01-15T14:29:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148822 2023-01-15T14:29:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773727+0000", 2023-01-15T14:29:58.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192189 2023-01-15T14:29:58.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:58.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773918+0000", 2023-01-15T14:29:58.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019143099999999999 2023-01-15T14:29:58.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774291+0000", 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037288399999999998 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774341+0000", 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0117999999999997e-05 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774696+0000", 2023-01-15T14:29:58.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000355268 2023-01-15T14:29:58.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:58.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874892+0000", 2023-01-15T14:29:58.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100195981 2023-01-15T14:29:58.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.977868+0000", 2023-01-15T14:29:58.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102975649 2023-01-15T14:29:58.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.075846+0000", 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097977838999999997 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.076004+0000", 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015859700000000001 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.076303+0000", 2023-01-15T14:29:58.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029816400000000001 2023-01-15T14:29:58.596 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.596 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.596 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.596 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:58.596 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:29:58.596 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.053674998999995, 2023-01-15T14:29:58.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:29:58.597 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.597 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:58.597 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.597 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.597 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.597 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:29:58.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.597 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:29:58.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:29:58.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:29:58.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:29:58.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:29:58.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:29:58.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:29:58.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:29:58.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:29:58.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:29:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:29:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:29:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:29:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:29:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:29:58.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:29:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:29:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:29:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:29:58.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:29:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:29:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:29:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:29:58.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:29:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:29:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:29:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:29:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:29:58.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.053192644999996, 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:29:58.758 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:29:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:29:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:29:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:29:58.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:29:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:29:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:29:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:29:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:29:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:29:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:29:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:29:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:58.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:29:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:29:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:29:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:29:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:29:58.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:29:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:29:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:29:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:29:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:29:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:29:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:29:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.053140081999999, 2023-01-15T14:29:58.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:29:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:29:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:29:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:29:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:29:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:29:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:29:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:29:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:29:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:29:58.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:29:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:29:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:29:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:29:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:29:58.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:29:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:29:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:29:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:29:58.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:29:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:29:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:29:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:29:58.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:29:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:29:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:58.777 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:29:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.681938377999998, 2023-01-15T14:29:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:29:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:29:58.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:29:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:29:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:29:58.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:29:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:29:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:29:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:29:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:29:58.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:29:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:29:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:29:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:29:58.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:29:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:29:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:29:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:29:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:29:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:29:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:29:58.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:29:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:29:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:29:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:58.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:29:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:29:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:29:58.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:29:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:29:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:29:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.809 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:29:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.681821704000001, 2023-01-15T14:29:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:29:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:29:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.810 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:29:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:29:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:29:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:29:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:29:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:29:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:29:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:29:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:29:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:29:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:29:58.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:29:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:29:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:29:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:29:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:29:58.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:29:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:29:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:29:58.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:29:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:29:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:29:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:29:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:29:58.817 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:448 3.1 3:8b0a2647:::smithi09025264-8:head [write 1856548~737690] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.434563796000006, 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40560468799999999, 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:58.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:58.819 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:58.819 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 448 2023-01-15T14:29:58.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.819 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:58.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:58.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:29:58.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:58.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:29:58.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:58.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:58.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297043+0000", 2023-01-15T14:29:58.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:58.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297477+0000", 2023-01-15T14:29:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043336500000000001 2023-01-15T14:29:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297505+0000", 2023-01-15T14:29:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8583000000000001e-05 2023-01-15T14:29:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:58.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397598+0000", 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092164 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397683+0000", 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5986e-05 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397798+0000", 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114916 2023-01-15T14:29:58.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:58.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498684+0000", 2023-01-15T14:29:58.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10088510000000001 2023-01-15T14:29:58.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:58.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498848+0000", 2023-01-15T14:29:58.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164575 2023-01-15T14:29:58.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:58.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499083+0000", 2023-01-15T14:29:58.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000235171 2023-01-15T14:29:58.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:58.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499670+0000", 2023-01-15T14:29:58.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058650200000000003 2023-01-15T14:29:58.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:58.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499688+0000", 2023-01-15T14:29:58.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8547e-05 2023-01-15T14:29:58.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:58.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:58.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.500037+0000", 2023-01-15T14:29:58.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034891399999999997 2023-01-15T14:29:58.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:58.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.501925+0000", 2023-01-15T14:29:59.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00188766 2023-01-15T14:29:59.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.601919+0000", 2023-01-15T14:29:59.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099994371999999998 2023-01-15T14:29:59.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702377+0000", 2023-01-15T14:29:59.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045810500000001 2023-01-15T14:29:59.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702537+0000", 2023-01-15T14:29:59.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015941600000000001 2023-01-15T14:29:59.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702650+0000", 2023-01-15T14:29:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011338899999999999 2023-01-15T14:29:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:29:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:29:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.955689427999999, 2023-01-15T14:29:59.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:29:59.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:59.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.020 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:29:59.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.020 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:29:59.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:29:59.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:29:59.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:59.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:29:59.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:29:59.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:29:59.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:29:59.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:29:59.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:29:59.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:29:59.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:29:59.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:29:59.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:29:59.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:29:59.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:29:59.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:29:59.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:29:59.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:29:59.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:29:59.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.029 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.029 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:29:59.029 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:29:59.029 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.739439369000003, 2023-01-15T14:29:59.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:29:59.030 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.030 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.030 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.030 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.030 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.030 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:29:59.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.030 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:29:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:29:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:29:59.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:59.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:29:59.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:29:59.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:29:59.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:29:59.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:29:59.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:29:59.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:29:59.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:29:59.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:29:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:29:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:29:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:29:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:29:59.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:29:59.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:29:59.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:29:59.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:29:59.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:29:59.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:29:59.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:29:59.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:29:59.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:29:59.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:29:59.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:29:59.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:29:59.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:29:59.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:29:59.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:29:59.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.039 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:29:59.039 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:29:59.039 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.739405802, 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:29:59.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:29:59.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:29:59.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:59.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:29:59.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:29:59.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:29:59.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:29:59.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:29:59.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:29:59.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:29:59.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:29:59.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:29:59.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:29:59.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:29:59.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:29:59.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:29:59.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:29:59.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:29:59.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:29:59.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:29:59.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:29:59.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:29:59.049 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.049 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.049 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.049 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:29:59.049 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:29:59.049 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.973502256, 2023-01-15T14:29:59.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:29:59.049 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.049 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.050 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:29:59.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.050 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:29:59.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:29:59.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:29:59.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:59.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:29:59.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:29:59.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:29:59.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:29:59.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:59.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:29:59.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:29:59.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:29:59.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:29:59.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:29:59.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:29:59.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:29:59.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:29:59.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:29:59.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:29:59.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:29:59.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:29:59.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:29:59.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:29:59.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:29:59.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:29:59.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:29:59.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:29:59.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:29:59.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:29:59.059 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.059 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.059 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.059 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.059 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.059 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:59.060 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:29:59.060 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:29:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:29:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:29:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:29:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:29:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:29:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:29:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:29:59.061 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:29:59.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:29:59.063 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:57.816+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:59.066 INFO:tasks.rados.rados.0.smithi090.stdout:463: done (0 left) 2023-01-15T14:29:59.066 INFO:tasks.rados.rados.0.smithi090.stdout:464: read oid 23 snap 42 2023-01-15T14:29:59.066 INFO:tasks.rados.rados.0.smithi090.stdout:464: expect deleted 2023-01-15T14:29:59.066 INFO:tasks.rados.rados.0.smithi090.stdout:465: write oid 18 current snap is 43 2023-01-15T14:29:59.066 INFO:tasks.rados.rados.0.smithi090.stdout:465: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2023-01-15T14:29:59.066 INFO:tasks.rados.rados.0.smithi090.stdout:465: writing smithi09025264-18 from 443751 to 1075633 tid 1 2023-01-15T14:29:59.066 INFO:tasks.rados.rados.0.smithi090.stdout:465: writing smithi09025264-18 from 1810907 to 2575965 tid 2 2023-01-15T14:29:59.066 INFO:tasks.rados.rados.0.smithi090.stdout:465: writing smithi09025264-18 from 3067461 to 3243751 tid 3 2023-01-15T14:29:59.067 INFO:tasks.rados.rados.0.smithi090.stdout:466: rollback oid 34 current snap is 43 2023-01-15T14:29:59.067 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 34 to 42 2023-01-15T14:29:59.067 INFO:tasks.rados.rados.0.smithi090.stdout:467: write oid 45 current snap is 43 2023-01-15T14:29:59.067 INFO:tasks.rados.rados.0.smithi090.stdout:467: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2023-01-15T14:29:59.067 INFO:tasks.rados.rados.0.smithi090.stdout:467: writing smithi09025264-45 from 460558 to 1167689 tid 1 2023-01-15T14:29:59.067 INFO:tasks.rados.rados.0.smithi090.stdout:467: writing smithi09025264-45 from 1869389 to 2378105 tid 2 2023-01-15T14:29:59.067 INFO:tasks.rados.rados.0.smithi090.stdout:467: writing smithi09025264-45 from 3094884 to 3260558 tid 3 2023-01-15T14:29:59.067 INFO:tasks.rados.rados.0.smithi090.stdout:468: read oid 25 snap 42 2023-01-15T14:29:59.067 INFO:tasks.rados.rados.0.smithi090.stdout:468: expect (ObjNum 179 snap 38 seq_num 179) 2023-01-15T14:29:59.067 INFO:tasks.rados.rados.0.smithi090.stdout:469: write (excl) oid 38 current snap is 43 2023-01-15T14:29:59.067 INFO:tasks.rados.rados.0.smithi090.stdout:469: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2023-01-15T14:29:59.068 INFO:tasks.rados.rados.0.smithi090.stdout:469: writing smithi09025264-38 from 477365 to 1259745 tid 1 2023-01-15T14:29:59.068 INFO:tasks.rados.rados.0.smithi090.stdout:469: writing smithi09025264-38 from 1927871 to 2696598 tid 2 2023-01-15T14:29:59.068 INFO:tasks.rados.rados.0.smithi090.stdout:469: writing smithi09025264-38 from 3122307 to 3277365 tid 3 2023-01-15T14:29:59.068 INFO:tasks.rados.rados.0.smithi090.stdout:470: read oid 5 snap 42 2023-01-15T14:29:59.068 INFO:tasks.rados.rados.0.smithi090.stdout:470: expect (ObjNum 150 snap 30 seq_num 150) 2023-01-15T14:29:59.068 INFO:tasks.rados.rados.0.smithi090.stdout:471: write (excl) oid 49 current snap is 43 2023-01-15T14:29:59.068 INFO:tasks.rados.rados.0.smithi090.stdout:471: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2023-01-15T14:29:59.068 INFO:tasks.rados.rados.0.smithi090.stdout:471: writing smithi09025264-49 from 494172 to 951801 tid 1 2023-01-15T14:29:59.068 INFO:tasks.rados.rados.0.smithi090.stdout:471: writing smithi09025264-49 from 1586353 to 2098738 tid 2 2023-01-15T14:29:59.068 INFO:tasks.rados.rados.0.smithi090.stdout:471: writing smithi09025264-49 from 2749730 to 3294172 tid 3 2023-01-15T14:29:59.068 INFO:tasks.rados.rados.0.smithi090.stdout:472: delete oid 17 current snap is 43 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:467: finishing write tid 1 to smithi09025264-45 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:468: expect (ObjNum 179 snap 38 seq_num 179) 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:467: finishing write tid 2 to smithi09025264-45 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:467: finishing write tid 3 to smithi09025264-45 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:465: finishing write tid 1 to smithi09025264-18 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:465: finishing write tid 2 to smithi09025264-18 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:467: finishing write tid 4 to smithi09025264-45 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:467: finishing write tid 5 to smithi09025264-45 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:467: oid 45 updating version 0 to 130 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:467: oid 45 updating version 130 to 131 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:467: oid 45 updating version 131 to 132 2023-01-15T14:29:59.069 INFO:tasks.rados.rados.0.smithi090.stdout:467: oid 45 version 132 is already newer than 129 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 132 (ObjNum 194 snap 43 seq_num 194) dirty exists 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:467: left oid 45 (ObjNum 194 snap 43 seq_num 194) 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:469: finishing write tid 1 to smithi09025264-38 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:465: finishing write tid 3 to smithi09025264-18 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:465: finishing write tid 4 to smithi09025264-18 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:465: finishing write tid 5 to smithi09025264-18 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:465: oid 18 updating version 0 to 107 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:465: oid 18 version 107 is already newer than 106 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:465: oid 18 version 107 is already newer than 105 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:465: oid 18 version 107 is already newer than 104 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 107 (ObjNum 193 snap 43 seq_num 193) dirty exists 2023-01-15T14:29:59.070 INFO:tasks.rados.rados.0.smithi090.stdout:465: left oid 18 (ObjNum 193 snap 43 seq_num 193) 2023-01-15T14:29:59.071 INFO:tasks.rados.rados.0.smithi090.stdout:469: finishing write tid 2 to smithi09025264-38 2023-01-15T14:29:59.071 INFO:tasks.rados.rados.0.smithi090.stdout:469: finishing write tid 3 to smithi09025264-38 2023-01-15T14:29:59.071 INFO:tasks.rados.rados.0.smithi090.stdout:469: finishing write tid 4 to smithi09025264-38 2023-01-15T14:29:59.071 INFO:tasks.rados.rados.0.smithi090.stdout:469: finishing write tid 5 to smithi09025264-38 2023-01-15T14:29:59.071 INFO:tasks.rados.rados.0.smithi090.stdout:469: oid 38 updating version 0 to 124 2023-01-15T14:29:59.071 INFO:tasks.rados.rados.0.smithi090.stdout:469: oid 38 version 124 is already newer than 123 2023-01-15T14:29:59.071 INFO:tasks.rados.rados.0.smithi090.stdout:469: oid 38 updating version 124 to 125 2023-01-15T14:29:59.071 INFO:tasks.rados.rados.0.smithi090.stdout:469: oid 38 updating version 125 to 126 2023-01-15T14:29:59.071 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 126 (ObjNum 195 snap 43 seq_num 195) dirty exists 2023-01-15T14:29:59.071 INFO:tasks.rados.rados.0.smithi090.stdout:469: left oid 38 (ObjNum 195 snap 43 seq_num 195) 2023-01-15T14:29:59.072 INFO:tasks.rados.rados.0.smithi090.stdout:470: expect (ObjNum 150 snap 30 seq_num 150) 2023-01-15T14:29:59.072 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:59.069+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:59.164 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:59.160+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:59.206 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:59.206 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:59.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.206 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:29:59.207 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:29:59.207 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.3052039200000001, 2023-01-15T14:29:59.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3052349679999999, 2023-01-15T14:29:59.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.207 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:59.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.207 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:29:59.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.207 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:29:59.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:29:59.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:29:59.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:29:59.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:29:59.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:29:59.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:29:59.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:29:59.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:29:59.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:29:59.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:29:59.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:29:59.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:29:59.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:29:59.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:29:59.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.212 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.213 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:29:59.213 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:29:59.213 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.2477062370000001, 2023-01-15T14:29:59.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.247803486, 2023-01-15T14:29:59.213 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.214 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:29:59.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.214 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:29:59.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:29:59.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:29:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:29:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:29:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:29:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:29:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:29:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:29:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:29:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:29:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:29:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:29:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:29:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:29:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:29:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:29:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:29:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:29:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:29:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 2 2023-01-15T14:29:59.222 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:59.223 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:29:59.237 INFO:tasks.rados.rados.0.smithi090.stdout:464: done (8 left) 2023-01-15T14:29:59.237 INFO:tasks.rados.rados.0.smithi090.stdout:465: done (7 left) 2023-01-15T14:29:59.238 INFO:tasks.rados.rados.0.smithi090.stdout:467: done (6 left) 2023-01-15T14:29:59.238 INFO:tasks.rados.rados.0.smithi090.stdout:468: done (5 left) 2023-01-15T14:29:59.238 INFO:tasks.rados.rados.0.smithi090.stdout:469: done (4 left) 2023-01-15T14:29:59.238 INFO:tasks.rados.rados.0.smithi090.stdout:470: done (3 left) 2023-01-15T14:29:59.238 INFO:tasks.rados.rados.0.smithi090.stdout:472: done (2 left) 2023-01-15T14:29:59.238 INFO:tasks.rados.rados.0.smithi090.stdout:473: write oid 37 current snap is 43 2023-01-15T14:29:59.238 INFO:tasks.rados.rados.0.smithi090.stdout:473: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2023-01-15T14:29:59.246 INFO:tasks.rados.rados.0.smithi090.stdout:473: writing smithi09025264-37 from 527786 to 1252266 tid 1 2023-01-15T14:29:59.252 INFO:tasks.rados.rados.0.smithi090.stdout:473: writing smithi09025264-37 from 1703317 to 2219371 tid 2 2023-01-15T14:29:59.260 INFO:tasks.rados.rados.0.smithi090.stdout:473: writing smithi09025264-37 from 2804576 to 3239611 tid 3 2023-01-15T14:29:59.260 INFO:tasks.rados.rados.0.smithi090.stdout:474: copy_from oid 32 from oid 13 current snap is 43 2023-01-15T14:29:59.260 INFO:tasks.rados.rados.0.smithi090.stdout:475: rollback oid 39 current snap is 43 2023-01-15T14:29:59.260 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 39 to 42 2023-01-15T14:29:59.260 INFO:tasks.rados.rados.0.smithi090.stdout:476: read oid 16 snap -1 2023-01-15T14:29:59.260 INFO:tasks.rados.rados.0.smithi090.stdout:476: expect (ObjNum 85 snap 11 seq_num 85) 2023-01-15T14:29:59.261 INFO:tasks.rados.rados.0.smithi090.stdout:477: write (excl) oid 36 current snap is 43 2023-01-15T14:29:59.261 INFO:tasks.rados.rados.0.smithi090.stdout:477: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2023-01-15T14:29:59.265 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:29:59.264+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:29:59.270 INFO:tasks.rados.rados.0.smithi090.stdout:477: writing smithi09025264-36 from 544593 to 1344322 tid 1 2023-01-15T14:29:59.277 INFO:tasks.rados.rados.0.smithi090.stdout:477: writing smithi09025264-36 from 1761799 to 2537864 tid 2 2023-01-15T14:29:59.282 INFO:tasks.rados.rados.0.smithi090.stdout:477: writing smithi09025264-36 from 2948352 to 3344593 tid 3 2023-01-15T14:29:59.283 INFO:tasks.rados.rados.0.smithi090.stdout:478: write oid 33 current snap is 43 2023-01-15T14:29:59.283 INFO:tasks.rados.rados.0.smithi090.stdout:478: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2023-01-15T14:29:59.288 INFO:tasks.rados.rados.0.smithi090.stdout:478: writing smithi09025264-33 from 561400 to 1036378 tid 1 2023-01-15T14:29:59.295 INFO:tasks.rados.rados.0.smithi090.stdout:478: writing smithi09025264-33 from 1820281 to 2340004 tid 2 2023-01-15T14:29:59.301 INFO:tasks.rados.rados.0.smithi090.stdout:478: writing smithi09025264-33 from 2859422 to 3317001 tid 3 2023-01-15T14:29:59.301 INFO:tasks.rados.rados.0.smithi090.stdout:479: rollback oid 42 current snap is 43 2023-01-15T14:29:59.302 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 42 to 42 2023-01-15T14:29:59.302 INFO:tasks.rados.rados.0.smithi090.stdout:480: write oid 30 current snap is 43 2023-01-15T14:29:59.302 INFO:tasks.rados.rados.0.smithi090.stdout:480: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2023-01-15T14:29:59.308 INFO:tasks.rados.rados.0.smithi090.stdout:480: writing smithi09025264-30 from 578207 to 1128434 tid 1 2023-01-15T14:29:59.316 INFO:tasks.rados.rados.0.smithi090.stdout:480: writing smithi09025264-30 from 1762410 to 2542144 tid 2 2023-01-15T14:29:59.322 INFO:tasks.rados.rados.0.smithi090.stdout:480: writing smithi09025264-30 from 3286845 to 3378207 tid 3 2023-01-15T14:29:59.322 INFO:tasks.rados.rados.0.smithi090.stdout:481: read oid 40 snap -1 2023-01-15T14:29:59.322 INFO:tasks.rados.rados.0.smithi090.stdout:481: expect (ObjNum 184 snap 40 seq_num 184) 2023-01-15T14:29:59.322 INFO:tasks.rados.rados.0.smithi090.stdout:482: read oid 15 snap 42 2023-01-15T14:29:59.322 INFO:tasks.rados.rados.0.smithi090.stdout:482: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:29:59.322 INFO:tasks.rados.rados.0.smithi090.stdout:483: write (excl) oid 24 current snap is 43 2023-01-15T14:29:59.322 INFO:tasks.rados.rados.0.smithi090.stdout:483: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2023-01-15T14:29:59.328 INFO:tasks.rados.rados.0.smithi090.stdout:483: writing smithi09025264-24 from 595014 to 1220490 tid 1 2023-01-15T14:29:59.333 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:59.400 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:29:59.400 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:29:59.400 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:29:59.401 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:29:59.403 INFO:tasks.rados.rados.0.smithi090.stdout:483: writing smithi09025264-24 from 1820892 to 2344284 tid 2 2023-01-15T14:29:59.403 INFO:tasks.rados.rados.0.smithi090.stdout:483: writing smithi09025264-24 from 2797915 to 3394391 tid 3 2023-01-15T14:29:59.403 INFO:tasks.rados.rados.0.smithi090.stdout:484: snap_create 2023-01-15T14:29:59.403 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 13 2023-01-15T14:29:59.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:29:59.405 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:29:59.401+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:29:59.441 INFO:tasks.rados.rados.0.smithi090.stdout:475: finishing rollback tid 0 to smithi09025264-39 2023-01-15T14:29:59.441 INFO:tasks.rados.rados.0.smithi090.stdout:466: finishing rollback tid 1 to smithi09025264-34 2023-01-15T14:29:59.441 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2023-01-15T14:29:59.441 INFO:tasks.rados.rados.0.smithi090.stdout:466: done (12 left) 2023-01-15T14:29:59.441 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 12 2023-01-15T14:29:59.443 INFO:tasks.rados.rados.0.smithi090.stdout:481: expect (ObjNum 184 snap 40 seq_num 184) 2023-01-15T14:29:59.504 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:29:59.500+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:29:59.560 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:29:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:29:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:29:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:29:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:94 3.0 3:07a51d33:::smithi09025264-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.115002378, 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50760159199999999, 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 94 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439538+0000", 2023-01-15T14:29:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439857+0000", 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000319086 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439891+0000", 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4449999999999997e-05 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439906+0000", 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5082999999999999e-05 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824189+0000", 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38428238100000001 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824440+0000", 2023-01-15T14:29:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025139800000000001 2023-01-15T14:29:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824834+0000", 2023-01-15T14:29:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039422200000000001 2023-01-15T14:29:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825347+0000", 2023-01-15T14:29:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051323600000000003 2023-01-15T14:29:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825364+0000", 2023-01-15T14:29:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6478e-05 2023-01-15T14:29:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825589+0000", 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022497500000000001 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826457+0000", 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00086797600000000001 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.846539+0000", 2023-01-15T14:29:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020081742 2023-01-15T14:29:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.946989+0000", 2023-01-15T14:29:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045032199999999 2023-01-15T14:29:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947076+0000", 2023-01-15T14:29:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6922999999999994e-05 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947141+0000", 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5357000000000005e-05 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:96 3.0 3:07a51d33:::smithi09025264-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.114580265000001, 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50769745899999996, 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 96 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:29:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439961+0000", 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439966+0000", 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0769999999999998e-06 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439979+0000", 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3343000000000001e-05 2023-01-15T14:29:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439987+0000", 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2400000000000001e-06 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826191+0000", 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38620452799999999 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826424+0000", 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023264800000000001 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947394+0000", 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120970102 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947505+0000", 2023-01-15T14:29:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001105 2023-01-15T14:29:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947659+0000", 2023-01-15T14:29:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015468799999999999 2023-01-15T14:29:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.470027600999998, 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:29:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:29:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:29:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:29:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:29:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:29:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:29:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:29:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:29:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:29:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:29:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:29:59.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:29:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:29:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:29:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:29:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:29:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:29:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:29:59.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:29:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:29:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:29:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:29:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:29:59.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.469727804000001, 2023-01-15T14:29:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:29:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:29:59.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:29:59.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:29:59.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:29:59.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:29:59.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:29:59.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:29:59.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:29:59.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:29:59.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:29:59.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:29:59.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:29:59.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:29:59.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:29:59.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:29:59.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:29:59.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:59.595 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.469698019999996, 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:29:59.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:29:59.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:29:59.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:29:59.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:29:59.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:29:59.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:29:59.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:29:59.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:29:59.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:29:59.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:29:59.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:29:59.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:29:59.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:29:59.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:29:59.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:29:59.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:29:59.603 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.603 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.603 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.603 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:182 3.4 3:2022fdfa:::smithi09025264-46:head [write 1933572~791670] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:59.603 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:59.603 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.340835951000003, 2023-01-15T14:29:59.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.582304865, 2023-01-15T14:29:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 182 2023-01-15T14:29:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:59.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:29:59.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213704+0000", 2023-01-15T14:29:59.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:59.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214574+0000", 2023-01-15T14:29:59.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087037799999999999 2023-01-15T14:29:59.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214608+0000", 2023-01-15T14:29:59.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:29:59.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214627+0000", 2023-01-15T14:29:59.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:29:59.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491537+0000", 2023-01-15T14:29:59.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27691020300000002 2023-01-15T14:29:59.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491721+0000", 2023-01-15T14:29:59.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345500000000001 2023-01-15T14:29:59.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693760+0000", 2023-01-15T14:29:59.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202039252 2023-01-15T14:29:59.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693898+0000", 2023-01-15T14:29:59.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013789 2023-01-15T14:29:59.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:59.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694230+0000", 2023-01-15T14:29:59.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220899999999998 2023-01-15T14:29:59.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694866+0000", 2023-01-15T14:29:59.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063640599999999995 2023-01-15T14:29:59.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694887+0000", 2023-01-15T14:29:59.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0622999999999999e-05 2023-01-15T14:29:59.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695950+0000", 2023-01-15T14:29:59.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001062688 2023-01-15T14:29:59.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696014+0000", 2023-01-15T14:29:59.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4697999999999996e-05 2023-01-15T14:29:59.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795898+0000", 2023-01-15T14:29:59.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099883338000000002 2023-01-15T14:29:59.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795922+0000", 2023-01-15T14:29:59.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4326000000000001e-05 2023-01-15T14:29:59.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795955+0000", 2023-01-15T14:29:59.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3191000000000001e-05 2023-01-15T14:29:59.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796011+0000", 2023-01-15T14:29:59.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5631999999999999e-05 2023-01-15T14:29:59.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.614 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.614 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:59.614 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:59.614 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.339889220000003, 2023-01-15T14:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:59.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:29:59.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:29:59.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:29:59.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:29:59.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:29:59.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:29:59.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:29:59.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:29:59.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:29:59.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:29:59.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:29:59.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:29:59.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:29:59.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:29:59.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:29:59.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:29:59.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:29:59.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:29:59.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:29:59.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:29:59.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:29:59.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:29:59.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.624 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.624 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:29:59.624 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:59.624 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.339832522999998, 2023-01-15T14:29:59.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:29:59.625 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.625 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:59.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.625 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:29:59.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.625 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:29:59.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:59.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:29:59.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:29:59.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:29:59.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:29:59.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:29:59.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:29:59.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:29:59.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:29:59.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:29:59.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:29:59.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:29:59.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:29:59.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.631 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:29:59.632 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:59.632 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 89.723336349999997, 2023-01-15T14:29:59.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:29:59.632 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.632 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:59.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.632 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:29:59.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.632 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:29:59.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:29:59.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:29:59.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:29:59.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:29:59.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:29:59.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:29:59.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:29:59.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:29:59.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:29:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:29:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:29:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:29:59.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:29:59.637 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.638 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.638 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.638 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:29:59.638 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:59.638 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.694487878999993, 2023-01-15T14:29:59.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:29:59.638 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.638 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.638 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:29:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:29:59.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:29:59.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:29:59.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:29:59.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:29:59.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:29:59.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:29:59.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:29:59.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:29:59.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:29:59.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:29:59.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:59.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:29:59.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:29:59.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:29:59.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:29:59.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:29:59.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:29:59.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:29:59.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:29:59.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:29:59.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:29:59.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:29:59.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:29:59.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:29:59.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:29:59.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:29:59.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:29:59.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:29:59.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:29:59.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:29:59.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:29:59.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:29:59.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:29:59.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:29:59.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:29:59.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:29:59.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.648 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.648 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:373 3.4 3:2029fa71:::smithi09025264-39:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:59.648 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:59.649 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.476240672000003, 2023-01-15T14:29:59.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58298356399999995, 2023-01-15T14:29:59.649 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.649 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.649 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-01-15T14:29:59.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.649 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:59.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:29:59.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078299+0000", 2023-01-15T14:29:59.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:59.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078644+0000", 2023-01-15T14:29:59.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034494099999999998 2023-01-15T14:29:59.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078676+0000", 2023-01-15T14:29:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1183000000000003e-05 2023-01-15T14:29:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078694+0000", 2023-01-15T14:29:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-05 2023-01-15T14:29:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078748+0000", 2023-01-15T14:29:59.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3634999999999997e-05 2023-01-15T14:29:59.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:59.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078753+0000", 2023-01-15T14:29:59.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6370000000000004e-06 2023-01-15T14:29:59.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110841+0000", 2023-01-15T14:29:59.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032087439000000002 2023-01-15T14:29:59.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110967+0000", 2023-01-15T14:29:59.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126029 2023-01-15T14:29:59.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.457874+0000", 2023-01-15T14:29:59.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.346907725 2023-01-15T14:29:59.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458066+0000", 2023-01-15T14:29:59.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000191592 2023-01-15T14:29:59.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:59.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458321+0000", 2023-01-15T14:29:59.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025454200000000002 2023-01-15T14:29:59.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458918+0000", 2023-01-15T14:29:59.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059740800000000001 2023-01-15T14:29:59.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458945+0000", 2023-01-15T14:29:59.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-05 2023-01-15T14:29:59.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459253+0000", 2023-01-15T14:29:59.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030836200000000001 2023-01-15T14:29:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460943+0000", 2023-01-15T14:29:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016903580000000001 2023-01-15T14:29:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461340+0000", 2023-01-15T14:29:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039665799999999998 2023-01-15T14:29:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661129+0000", 2023-01-15T14:29:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19978933500000001 2023-01-15T14:29:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661188+0000", 2023-01-15T14:29:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8687000000000002e-05 2023-01-15T14:29:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661285+0000", 2023-01-15T14:29:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779000000000003e-05 2023-01-15T14:29:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:374 3.4 3:2029fa71:::smithi09025264-39:head [write 3101234~41642] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.47157876, 2023-01-15T14:29:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57848911800000002, 2023-01-15T14:29:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 374 2023-01-15T14:29:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:29:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082961+0000", 2023-01-15T14:29:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:29:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083019+0000", 2023-01-15T14:29:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999997e-05 2023-01-15T14:29:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083051+0000", 2023-01-15T14:29:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1813999999999997e-05 2023-01-15T14:29:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083069+0000", 2023-01-15T14:29:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:29:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083143+0000", 2023-01-15T14:29:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4363000000000006e-05 2023-01-15T14:29:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083147+0000", 2023-01-15T14:29:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4529999999999998e-06 2023-01-15T14:29:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111017+0000", 2023-01-15T14:29:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027870564 2023-01-15T14:29:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111099+0000", 2023-01-15T14:29:59.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1277000000000001e-05 2023-01-15T14:29:59.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459156+0000", 2023-01-15T14:29:59.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34805700499999997 2023-01-15T14:29:59.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459393+0000", 2023-01-15T14:29:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023757499999999999 2023-01-15T14:29:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459778+0000", 2023-01-15T14:29:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000385335 2023-01-15T14:29:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460020+0000", 2023-01-15T14:29:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024141899999999999 2023-01-15T14:29:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460037+0000", 2023-01-15T14:29:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6665999999999999e-05 2023-01-15T14:29:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460220+0000", 2023-01-15T14:29:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018367599999999999 2023-01-15T14:29:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461197+0000", 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00097708600000000006 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461720+0000", 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052235400000000005 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661345+0000", 2023-01-15T14:29:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199624884 2023-01-15T14:29:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661405+0000", 2023-01-15T14:29:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0965999999999999e-05 2023-01-15T14:29:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661452+0000", 2023-01-15T14:29:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6864999999999999e-05 2023-01-15T14:29:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:375 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (56) in=7b,truncate 3142876] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.471436994000001, 2023-01-15T14:29:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57851114599999998, 2023-01-15T14:29:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:29:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:29:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:29:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 375 2023-01-15T14:29:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:29:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:29:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:29:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:29:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:29:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:29:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083104+0000", 2023-01-15T14:29:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:29:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:29:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083113+0000", 2023-01-15T14:29:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6500000000000008e-06 2023-01-15T14:29:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:29:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083130+0000", 2023-01-15T14:29:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2023-01-15T14:29:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:29:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083142+0000", 2023-01-15T14:29:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2252999999999999e-05 2023-01-15T14:29:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083173+0000", 2023-01-15T14:29:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0474999999999999e-05 2023-01-15T14:29:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083175+0000", 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183469+0000", 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10029313099999999 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183747+0000", 2023-01-15T14:29:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000277998 2023-01-15T14:29:59.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:29:59.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460165+0000", 2023-01-15T14:29:59.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27641872899999997 2023-01-15T14:29:59.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:29:59.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460342+0000", 2023-01-15T14:29:59.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176792 2023-01-15T14:29:59.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:29:59.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460579+0000", 2023-01-15T14:29:59.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000237266 2023-01-15T14:29:59.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:29:59.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460833+0000", 2023-01-15T14:29:59.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025390900000000002 2023-01-15T14:29:59.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:29:59.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460853+0000", 2023-01-15T14:29:59.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9273e-05 2023-01-15T14:29:59.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:29:59.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461024+0000", 2023-01-15T14:29:59.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017111900000000001 2023-01-15T14:29:59.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:29:59.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461230+0000", 2023-01-15T14:29:59.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020672600000000001 2023-01-15T14:29:59.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461879+0000", 2023-01-15T14:29:59.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064846999999999995 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661524+0000", 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19964509599999999 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661560+0000", 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6264999999999999e-05 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:29:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661616+0000", 2023-01-15T14:29:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5952000000000002e-05 2023-01-15T14:29:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:29:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:29:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:29:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:29:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:376 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:29:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:29:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.471354738000002, 2023-01-15T14:29:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57864278800000002, 2023-01-15T14:29:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:29:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:29:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:29:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 376 2023-01-15T14:30:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:30:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:30:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083186+0000", 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083193+0000", 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083211+0000", 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7609999999999999e-05 2023-01-15T14:30:00.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083223+0000", 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140000000000001e-05 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093534+0000", 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010311529 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093551+0000", 2023-01-15T14:30:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.666e-05 2023-01-15T14:30:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183834+0000", 2023-01-15T14:30:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090283063999999996 2023-01-15T14:30:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.184076+0000", 2023-01-15T14:30:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024241900000000001 2023-01-15T14:30:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461001+0000", 2023-01-15T14:30:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27692504299999998 2023-01-15T14:30:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461152+0000", 2023-01-15T14:30:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150345 2023-01-15T14:30:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661642+0000", 2023-01-15T14:30:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200489742 2023-01-15T14:30:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661720+0000", 2023-01-15T14:30:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8604999999999997e-05 2023-01-15T14:30:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661830+0000", 2023-01-15T14:30:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010992099999999999 2023-01-15T14:30:00.157 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 52.693200902999997, 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:00.158 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:30:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:30:00.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:30:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:30:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:30:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:30:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:30:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:30:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:30:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:30:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:30:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:30:00.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:961 3.0 3:0a998d04:::smithi09025264-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2023-01-15T14:30:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.491432315000001, 2023-01-15T14:30:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51361718300000003, 2023-01-15T14:30:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 961 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063109+0000", 2023-01-15T14:30:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063115+0000", 2023-01-15T14:30:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8300000000000001e-06 2023-01-15T14:30:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063127+0000", 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2517000000000001e-05 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063137+0000", 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8509999999999994e-06 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063272+0000", 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013513500000000001 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063277+0000", 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2750000000000003e-06 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.265103+0000", 2023-01-15T14:30:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201825688 2023-01-15T14:30:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.265208+0000", 2023-01-15T14:30:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104693 2023-01-15T14:30:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.375398+0000", 2023-01-15T14:30:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.110190553 2023-01-15T14:30:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.375548+0000", 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149417 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576187+0000", 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20063947500000001 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576454+0000", 2023-01-15T14:30:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026708400000000001 2023-01-15T14:30:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576727+0000", 2023-01-15T14:30:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027257699999999998 2023-01-15T14:30:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:30:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.6326226530000003, 2023-01-15T14:30:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:30:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:30:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:30:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:30:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:30:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:30:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:30:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:30:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:30:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:30:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:30:00.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:30:00.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:30:00.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:30:00.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:30:00.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:30:00.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:30:00.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:30:00.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:00.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:30:00.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:30:00.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.184 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:30:00.184 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:00.184 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 5.6325639430000001, 2023-01-15T14:30:00.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:30:00.184 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.184 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:00.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:00.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:00.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:00.184 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:30:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:30:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:30:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:30:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:30:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:30:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.67614445, 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:30:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:30:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:30:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:30:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:30:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:30:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:30:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:30:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:30:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:30:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:30:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:30:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:30:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:30:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:30:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:30:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:30:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.618646767, 2023-01-15T14:30:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:30:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:30:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:30:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:30:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:30:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:30:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:30:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:30:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:30:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:30:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:30:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:30:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:30:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:30:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:30:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:30:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:30:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:30:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:30:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:30:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:30:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:30:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:30:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:30:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:30:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:30:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:30:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:30:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:30:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:30:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:30:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.207 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:00.209 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:30:00.209 INFO:tasks.rados.rados.0.smithi090.stdout:473: finishing write tid 1 to smithi09025264-37 2023-01-15T14:30:00.209 INFO:tasks.rados.rados.0.smithi090.stdout:473: finishing write tid 2 to smithi09025264-37 2023-01-15T14:30:00.209 INFO:tasks.rados.rados.0.smithi090.stdout:476: expect (ObjNum 85 snap 11 seq_num 85) 2023-01-15T14:30:00.210 INFO:tasks.rados.rados.0.smithi090.stdout:474: finishing copy_from to smithi09025264-32 2023-01-15T14:30:00.210 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 131 (ObjNum 178 snap 38 seq_num 178) dirty exists 2023-01-15T14:30:00.210 INFO:tasks.rados.rados.0.smithi090.stdout:474: finishing copy_from racing read to smithi09025264-32 2023-01-15T14:30:00.210 INFO:tasks.rados.rados.0.smithi090.stdout:482: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:30:00.210 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:30:00.211 INFO:tasks.rados.rados.0.smithi090.stdout:478: finishing write tid 1 to smithi09025264-33 2023-01-15T14:30:00.212 INFO:tasks.rados.rados.0.smithi090.stdout:475: finishing rollback tid 1 to smithi09025264-39 2023-01-15T14:30:00.212 INFO:tasks.rados.rados.0.smithi090.stdout:480: finishing write tid 1 to smithi09025264-30 2023-01-15T14:30:00.212 INFO:tasks.rados.rados.0.smithi090.stdout:474: done (11 left) 2023-01-15T14:30:00.212 INFO:tasks.rados.rados.0.smithi090.stdout:476: done (10 left) 2023-01-15T14:30:00.212 INFO:tasks.rados.rados.0.smithi090.stdout:481: done (9 left) 2023-01-15T14:30:00.212 INFO:tasks.rados.rados.0.smithi090.stdout:482: done (8 left) 2023-01-15T14:30:00.212 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:30:00.212 INFO:tasks.rados.rados.0.smithi090.stdout:475: finishing rollback tid 2 to smithi09025264-39 2023-01-15T14:30:00.212 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 128 (ObjNum 83 snap 11 seq_num 83) dirty exists 2023-01-15T14:30:00.212 INFO:tasks.rados.rados.0.smithi090.stdout:475: done (7 left) 2023-01-15T14:30:00.212 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:483: finishing write tid 1 to smithi09025264-24 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:483: finishing write tid 2 to smithi09025264-24 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:480: finishing write tid 2 to smithi09025264-30 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:473: finishing write tid 3 to smithi09025264-37 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:478: finishing write tid 2 to smithi09025264-33 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:478: finishing write tid 3 to smithi09025264-33 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:477: finishing write tid 1 to smithi09025264-36 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:483: finishing write tid 3 to smithi09025264-24 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:483: finishing write tid 4 to smithi09025264-24 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:483: finishing write tid 5 to smithi09025264-24 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:483: oid 24 updating version 0 to 218 2023-01-15T14:30:00.213 INFO:tasks.rados.rados.0.smithi090.stdout:483: oid 24 version 218 is already newer than 217 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout:483: oid 24 version 218 is already newer than 216 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout:483: oid 24 version 218 is already newer than 215 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 218 (ObjNum 202 snap 43 seq_num 202) dirty exists 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout:483: left oid 24 (ObjNum 202 snap 43 seq_num 202) 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout:480: finishing write tid 3 to smithi09025264-30 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout:477: finishing write tid 2 to smithi09025264-36 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout:477: finishing write tid 3 to smithi09025264-36 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout:483: done (6 left) 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout:478: finishing write tid 4 to smithi09025264-33 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout:478: finishing write tid 5 to smithi09025264-33 2023-01-15T14:30:00.214 INFO:tasks.rados.rados.0.smithi090.stdout:478: oid 33 updating version 0 to 57 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout:478: oid 33 version 57 is already newer than 54 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout:478: oid 33 version 57 is already newer than 55 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout:478: oid 33 version 57 is already newer than 56 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 57 (ObjNum 200 snap 43 seq_num 200) dirty exists 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout:478: left oid 33 (ObjNum 200 snap 43 seq_num 200) 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout:477: finishing write tid 4 to smithi09025264-36 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout:478: done (5 left) 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout:477: finishing write tid 5 to smithi09025264-36 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout:477: oid 36 updating version 0 to 152 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout:477: oid 36 updating version 152 to 153 2023-01-15T14:30:00.215 INFO:tasks.rados.rados.0.smithi090.stdout:477: oid 36 version 153 is already newer than 151 2023-01-15T14:30:00.216 INFO:tasks.rados.rados.0.smithi090.stdout:477: oid 36 updating version 153 to 154 2023-01-15T14:30:00.216 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 154 (ObjNum 199 snap 43 seq_num 199) dirty exists 2023-01-15T14:30:00.216 INFO:tasks.rados.rados.0.smithi090.stdout:477: left oid 36 (ObjNum 199 snap 43 seq_num 199) 2023-01-15T14:30:00.216 INFO:tasks.rados.rados.0.smithi090.stdout:480: finishing write tid 4 to smithi09025264-30 2023-01-15T14:30:00.216 INFO:tasks.rados.rados.0.smithi090.stdout:477: done (4 left) 2023-01-15T14:30:00.216 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:30:00.216 INFO:tasks.rados.rados.0.smithi090.stdout:480: finishing write tid 5 to smithi09025264-30 2023-01-15T14:30:00.216 INFO:tasks.rados.rados.0.smithi090.stdout:480: oid 30 updating version 0 to 66 2023-01-15T14:30:00.216 INFO:tasks.rados.rados.0.smithi090.stdout:480: oid 30 updating version 66 to 68 2023-01-15T14:30:00.216 INFO:tasks.rados.rados.0.smithi090.stdout:480: oid 30 version 68 is already newer than 65 2023-01-15T14:30:00.216 INFO:tasks.rados.rados.0.smithi090.stdout:480: oid 30 version 68 is already newer than 67 2023-01-15T14:30:00.217 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 68 (ObjNum 201 snap 43 seq_num 201) dirty exists 2023-01-15T14:30:00.217 INFO:tasks.rados.rados.0.smithi090.stdout:480: left oid 30 (ObjNum 201 snap 43 seq_num 201) 2023-01-15T14:30:00.217 INFO:tasks.rados.rados.0.smithi090.stdout:480: done (3 left) 2023-01-15T14:30:00.217 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:30:00.218 INFO:tasks.rados.rados.0.smithi090.stdout:473: finishing write tid 4 to smithi09025264-37 2023-01-15T14:30:00.218 INFO:tasks.rados.rados.0.smithi090.stdout:473: finishing write tid 5 to smithi09025264-37 2023-01-15T14:30:00.218 INFO:tasks.rados.rados.0.smithi090.stdout:473: oid 37 updating version 0 to 211 2023-01-15T14:30:00.218 INFO:tasks.rados.rados.0.smithi090.stdout:473: oid 37 version 211 is already newer than 208 2023-01-15T14:30:00.218 INFO:tasks.rados.rados.0.smithi090.stdout:473: oid 37 version 211 is already newer than 209 2023-01-15T14:30:00.218 INFO:tasks.rados.rados.0.smithi090.stdout:473: oid 37 version 211 is already newer than 210 2023-01-15T14:30:00.218 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 211 (ObjNum 198 snap 43 seq_num 198) dirty exists 2023-01-15T14:30:00.219 INFO:tasks.rados.rados.0.smithi090.stdout:473: left oid 37 (ObjNum 198 snap 43 seq_num 198) 2023-01-15T14:30:00.219 INFO:tasks.rados.rados.0.smithi090.stdout:473: done (2 left) 2023-01-15T14:30:00.219 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:00.219 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:00.205+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:00.309 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:00.305+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:00.325 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:00.326 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:00.326 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:30:00.326 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:00.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:30:00.410 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:00.408+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:00.435 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:00.435 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:00.435 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:00.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:30:00.511 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:00.508+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:00.555 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:38 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'3, mlcod=0'0)", 2023-01-15T14:30:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:30:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.438465023999996, 2023-01-15T14:30:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30191175999999997, 2023-01-15T14:30:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:30:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:30:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112976+0000", 2023-01-15T14:30:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113186+0000", 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020979700000000001 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113228+0000", 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2150999999999997e-05 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213313+0000", 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100084943 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313517+0000", 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10020435699999999 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313576+0000", 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783999999999998e-05 2023-01-15T14:30:00.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313976+0000", 2023-01-15T14:30:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039960499999999998 2023-01-15T14:30:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314004+0000", 2023-01-15T14:30:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7954e-05 2023-01-15T14:30:00.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314175+0000", 2023-01-15T14:30:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171557 2023-01-15T14:30:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414853+0000", 2023-01-15T14:30:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100677875 2023-01-15T14:30:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414890+0000", 2023-01-15T14:30:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6803e-05 2023-01-15T14:30:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:39 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'4, mlcod=0'0)", 2023-01-15T14:30:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:30:00.563 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.338072849, 2023-01-15T14:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20154512699999999, 2023-01-15T14:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:30:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213369+0000", 2023-01-15T14:30:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213386+0000", 2023-01-15T14:30:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7643e-05 2023-01-15T14:30:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213427+0000", 2023-01-15T14:30:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0930000000000003e-05 2023-01-15T14:30:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213459+0000", 2023-01-15T14:30:00.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1479999999999997e-05 2023-01-15T14:30:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314106+0000", 2023-01-15T14:30:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100646614 2023-01-15T14:30:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314128+0000", 2023-01-15T14:30:00.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2340999999999998e-05 2023-01-15T14:30:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314224+0000", 2023-01-15T14:30:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6534999999999996e-05 2023-01-15T14:30:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314249+0000", 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4981e-05 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314337+0000", 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7287000000000006e-05 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414903+0000", 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100566552 2023-01-15T14:30:00.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414916+0000", 2023-01-15T14:30:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354000000000001e-05 2023-01-15T14:30:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:46 3.0 e23/20 3:03b54904:::smithi09025264-12:head v 23'5, mlcod=23'1)", 2023-01-15T14:30:00.570 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.337569110000004, 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202592668, 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:30:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213873+0000", 2023-01-15T14:30:00.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214477+0000", 2023-01-15T14:30:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060481300000000005 2023-01-15T14:30:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214532+0000", 2023-01-15T14:30:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5096999999999998e-05 2023-01-15T14:30:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214555+0000", 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.234e-05 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414409+0000", 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19985461199999999 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414442+0000", 2023-01-15T14:30:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2987000000000001e-05 2023-01-15T14:30:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414813+0000", 2023-01-15T14:30:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037061899999999998 2023-01-15T14:30:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414832+0000", 2023-01-15T14:30:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9380000000000001e-05 2023-01-15T14:30:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415093+0000", 2023-01-15T14:30:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026032099999999999 2023-01-15T14:30:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415154+0000", 2023-01-15T14:30:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1538999999999995e-05 2023-01-15T14:30:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.416467+0000", 2023-01-15T14:30:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001312613 2023-01-15T14:30:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:281 3.5 e30/20 3:a75958b1:::smithi09025264-6:head v 30'11, mlcod=23'6)", 2023-01-15T14:30:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:34.981189+0000", 2023-01-15T14:30:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.570253811000001, 2023-01-15T14:30:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19987028400000001, 2023-01-15T14:30:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981189+0000", 2023-01-15T14:30:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981189+0000", 2023-01-15T14:30:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981188+0000", 2023-01-15T14:30:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981206+0000", 2023-01-15T14:30:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8377999999999999e-05 2023-01-15T14:30:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981231+0000", 2023-01-15T14:30:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.516e-05 2023-01-15T14:30:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981249+0000", 2023-01-15T14:30:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7425e-05 2023-01-15T14:30:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.179815+0000", 2023-01-15T14:30:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19856638300000001 2023-01-15T14:30:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.179854+0000", 2023-01-15T14:30:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8587000000000002e-05 2023-01-15T14:30:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180059+0000", 2023-01-15T14:30:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020564300000000001 2023-01-15T14:30:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180085+0000", 2023-01-15T14:30:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5162999999999999e-05 2023-01-15T14:30:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180548+0000", 2023-01-15T14:30:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046301300000000002 2023-01-15T14:30:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180599+0000", 2023-01-15T14:30:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0943000000000003e-05 2023-01-15T14:30:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.181060+0000", 2023-01-15T14:30:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046124700000000002 2023-01-15T14:30:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:347 3.0 e32/20 3:03b54904:::smithi09025264-12:head v 32'40, mlcod=31'36)", 2023-01-15T14:30:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:30:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.539646211000004, 2023-01-15T14:30:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20127003500000001, 2023-01-15T14:30:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:30:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011795+0000", 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.012239+0000", 2023-01-15T14:30:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000444361 2023-01-15T14:30:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.012261+0000", 2023-01-15T14:30:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2008e-05 2023-01-15T14:30:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.112398+0000", 2023-01-15T14:30:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013730799999999 2023-01-15T14:30:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.210584+0000", 2023-01-15T14:30:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098186184999999995 2023-01-15T14:30:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.210634+0000", 2023-01-15T14:30:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9706000000000001e-05 2023-01-15T14:30:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211053+0000", 2023-01-15T14:30:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041844199999999999 2023-01-15T14:30:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211117+0000", 2023-01-15T14:30:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4950000000000007e-05 2023-01-15T14:30:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211390+0000", 2023-01-15T14:30:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027278300000000002 2023-01-15T14:30:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211507+0000", 2023-01-15T14:30:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011642 2023-01-15T14:30:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.213067+0000", 2023-01-15T14:30:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001560402 2023-01-15T14:30:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:621 3.8 e55/41 3:17d5d741:::smithi09025264-5:head v 55'45, mlcod=53'42)", 2023-01-15T14:30:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:00.863036+0000", 2023-01-15T14:30:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.688407681000001, 2023-01-15T14:30:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16723332099999999, 2023-01-15T14:30:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863036+0000", 2023-01-15T14:30:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863036+0000", 2023-01-15T14:30:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863035+0000", 2023-01-15T14:30:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863346+0000", 2023-01-15T14:30:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031160999999999998 2023-01-15T14:30:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863374+0000", 2023-01-15T14:30:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:30:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.863392+0000", 2023-01-15T14:30:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7762e-05 2023-01-15T14:30:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.963611+0000", 2023-01-15T14:30:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100219251 2023-01-15T14:30:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.963647+0000", 2023-01-15T14:30:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5929000000000001e-05 2023-01-15T14:30:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.964473+0000", 2023-01-15T14:30:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082583999999999999 2023-01-15T14:30:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.964502+0000", 2023-01-15T14:30:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8996000000000002e-05 2023-01-15T14:30:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.964751+0000", 2023-01-15T14:30:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024928 2023-01-15T14:30:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:00.964782+0000", 2023-01-15T14:30:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1202999999999999e-05 2023-01-15T14:30:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:01.030269+0000", 2023-01-15T14:30:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.065486775999999997 2023-01-15T14:30:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.4.0:118 3.9 e62/20 3:9f7cca9d:::smithi09025264-41:1f v 62'64, mlcod=61'58)", 2023-01-15T14:30:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:30:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 52.678255722999999, 2023-01-15T14:30:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20203982600000001, 2023-01-15T14:30:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:30:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:30:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873186+0000", 2023-01-15T14:30:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873201+0000", 2023-01-15T14:30:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5474000000000001e-05 2023-01-15T14:30:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873219+0000", 2023-01-15T14:30:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7677999999999999e-05 2023-01-15T14:30:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873234+0000", 2023-01-15T14:30:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4481000000000001e-05 2023-01-15T14:30:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.073729+0000", 2023-01-15T14:30:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200495595 2023-01-15T14:30:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.073768+0000", 2023-01-15T14:30:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8492999999999999e-05 2023-01-15T14:30:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074468+0000", 2023-01-15T14:30:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069991300000000003 2023-01-15T14:30:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074486+0000", 2023-01-15T14:30:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8669999999999999e-05 2023-01-15T14:30:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074563+0000", 2023-01-15T14:30:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7084999999999998e-05 2023-01-15T14:30:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074599+0000", 2023-01-15T14:30:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5453000000000001e-05 2023-01-15T14:30:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.075227+0000", 2023-01-15T14:30:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062855400000000003 2023-01-15T14:30:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:125 3.0 e70/55 3:0b54190a:::smithi09025264-17:25 v 70'136, mlcod=68'130)", 2023-01-15T14:30:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:30:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.608482999000003, 2023-01-15T14:30:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30171926799999998, 2023-01-15T14:30:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:30:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:30:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942958+0000", 2023-01-15T14:30:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942972+0000", 2023-01-15T14:30:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3386e-05 2023-01-15T14:30:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942990+0000", 2023-01-15T14:30:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8113000000000001e-05 2023-01-15T14:30:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.943002+0000", 2023-01-15T14:30:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:30:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141942+0000", 2023-01-15T14:30:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19893962300000001 2023-01-15T14:30:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141978+0000", 2023-01-15T14:30:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6383e-05 2023-01-15T14:30:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142670+0000", 2023-01-15T14:30:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069211900000000005 2023-01-15T14:30:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142688+0000", 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7839999999999999e-05 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142768+0000", 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0210999999999995e-05 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244581+0000", 2023-01-15T14:30:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101813406 2023-01-15T14:30:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244680+0000", 2023-01-15T14:30:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8045000000000001e-05 2023-01-15T14:30:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:818 3.8 e70/41 3:1eaad226:::smithi09025264-49:head v 70'89, mlcod=68'85)", 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:16.285706+0000", 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.265737066, 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17358161, 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285706+0000", 2023-01-15T14:30:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285706+0000", 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285706+0000", 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285713+0000", 2023-01-15T14:30:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7179999999999996e-06 2023-01-15T14:30:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285728+0000", 2023-01-15T14:30:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5063e-05 2023-01-15T14:30:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.285741+0000", 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2847000000000001e-05 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386010+0000", 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100268632 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386047+0000", 2023-01-15T14:30:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6637000000000002e-05 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386644+0000", 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059714800000000004 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386673+0000", 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9660000000000001e-05 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386772+0000", 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8550999999999999e-05 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.386817+0000", 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4747999999999999e-05 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.459288+0000", 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072471181999999995 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:948 3.8 e83/41 3:17d5d741:::smithi09025264-5:head v 83'119, mlcod=82'115)", 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.200945915999998, 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201378735, 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:30:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:30:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350496+0000", 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350511+0000", 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5007e-05 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350535+0000", 2023-01-15T14:30:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4170000000000001e-05 2023-01-15T14:30:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.450633+0000", 2023-01-15T14:30:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100098221 2023-01-15T14:30:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.550879+0000", 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100245538 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.550917+0000", 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8043000000000002e-05 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551222+0000", 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030465400000000001 2023-01-15T14:30:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551241+0000", 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9049999999999999e-05 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551331+0000", 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0807e-05 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551356+0000", 2023-01-15T14:30:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4786999999999998e-05 2023-01-15T14:30:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551876+0000", 2023-01-15T14:30:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051983800000000001 2023-01-15T14:30:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1124 3.5 e97/48 3:a75958b1:::smithi09025264-6:head v 97'44, mlcod=94'39)", 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.414038843, 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200993378, 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137455+0000", 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1187000000000003e-05 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137475+0000", 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9692999999999999e-05 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137496+0000", 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1776e-05 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238363+0000", 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10086711900000001 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238415+0000", 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1465999999999998e-05 2023-01-15T14:30:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238578+0000", 2023-01-15T14:30:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016310599999999999 2023-01-15T14:30:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238598+0000", 2023-01-15T14:30:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9729999999999999e-05 2023-01-15T14:30:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238673+0000", 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5302000000000005e-05 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239068+0000", 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000395227 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.338398+0000", 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099329534999999997 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1125 3.5 e97/48 3:a75958b1:::smithi09025264-6:head v 97'45, mlcod=94'39)", 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.413921630000001, 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201211626, 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137521+0000", 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137531+0000", 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6399999999999992e-06 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137546+0000", 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5783000000000001e-05 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137557+0000", 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0657e-05 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238639+0000", 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10108166 2023-01-15T14:30:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238688+0000", 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9570000000000001e-05 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238931+0000", 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000243255 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238948+0000", 2023-01-15T14:30:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6246e-05 2023-01-15T14:30:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238999+0000", 2023-01-15T14:30:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1063000000000002e-05 2023-01-15T14:30:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239101+0000", 2023-01-15T14:30:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010191000000000001 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.338733+0000", 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099632581999999997 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1119 3.8 e97/41 3:1eaad226:::smithi09025264-49:head v 97'157, mlcod=97'154)", 2023-01-15T14:30:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.412303326, 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200961838, 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:30:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139139+0000", 2023-01-15T14:30:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139291+0000", 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015233900000000001 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139318+0000", 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6265999999999999e-05 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139331+0000", 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3757000000000001e-05 2023-01-15T14:30:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238733+0000", 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099401891000000006 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238778+0000", 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5188999999999997e-05 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239141+0000", 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000362911 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239155+0000", 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3879e-05 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239274+0000", 2023-01-15T14:30:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011916 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239310+0000", 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5602999999999998e-05 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.340102+0000", 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10079187000000001 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(3.0 REQUEST e99)", 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.555029356, 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26432840200000002, 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996419+0000", 2023-01-15T14:30:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6729999999999997e-06 2023-01-15T14:30:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996449+0000", 2023-01-15T14:30:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.004e-05 2023-01-15T14:30:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996463+0000", 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3607e-05 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996594+0000", 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013148899999999999 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996618+0000", 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3249000000000001e-05 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260645+0000", 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26402768700000001 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260671+0000", 2023-01-15T14:30:00.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.546e-05 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260742+0000", 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1569999999999994e-05 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:261 3.8 e98/41 3:1e54a7f5:::smithi09025264-42:3e v 98'160, mlcod=97'157)", 2023-01-15T14:30:00.664 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.493961767, 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20299468400000001, 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057480+0000", 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057496+0000", 2023-01-15T14:30:00.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5869999999999999e-05 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057528+0000", 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057545+0000", 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.694e-05 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.158971+0000", 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101425425 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159011+0000", 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0058000000000002e-05 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159858+0000", 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084767900000000005 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159877+0000", 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9133000000000001e-05 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159959+0000", 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1693000000000005e-05 2023-01-15T14:30:00.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260430+0000", 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10047039200000001 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260476+0000", 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6652e-05 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.492740611, 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:00.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:30:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:30:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:30:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:30:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:30:00.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:30:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:30:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:30:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:30:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "replica_scrub(pg: 3.0,from:0'0,to:98'189,epoch:99/95,start:3:00000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.189600670000001, 2023-01-15T14:30:00.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20057929099999999, 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:30:00.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361841+0000", 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361855+0000", 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3464e-05 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361886+0000", 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1389000000000001e-05 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.462023+0000", 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013720800000001 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562267+0000", 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100243392 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562322+0000", 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5075999999999999e-05 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562422+0000", 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010015599999999999 2023-01-15T14:30:00.681 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(2.1 GRANT e100)", 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.233778384000001, 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20048780599999999, 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:30:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:30:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317663+0000", 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317674+0000", 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.119e-05 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317702+0000", 2023-01-15T14:30:00.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7183e-05 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.417845+0000", 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100142988 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518107+0000", 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100262461 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518121+0000", 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.449e-05 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518124+0000", 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1730000000000002e-06 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518153+0000", 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9043999999999998e-05 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:194 3.4 e101/48 3:2022fdfa:::smithi09025264-46:43 v 101'104, mlcod=98'100)", 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 12.706699, 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20670733199999999, 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.687 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844743+0000", 2023-01-15T14:30:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844757+0000", 2023-01-15T14:30:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.484e-05 2023-01-15T14:30:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844787+0000", 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9113000000000001e-05 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.944923+0000", 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013656 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.949975+0000", 2023-01-15T14:30:00.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050521400000000001 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950015+0000", 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9286999999999998e-05 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950559+0000", 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054463500000000002 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950585+0000", 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6166e-05 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950670+0000", 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4902999999999997e-05 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051368+0000", 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100697913 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051452+0000", 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3442000000000005e-05 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.693 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1224 3.4 e107/48 3:2029fa71:::smithi09025264-39:head v 107'120, mlcod=104'116)", 2023-01-15T14:30:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:30:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.3196703599999999, 2023-01-15T14:30:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22601005900000001, 2023-01-15T14:30:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:30:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231771+0000", 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:00.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233829+0000", 2023-01-15T14:30:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002057836 2023-01-15T14:30:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233881+0000", 2023-01-15T14:30:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1856e-05 2023-01-15T14:30:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.333996+0000", 2023-01-15T14:30:00.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100115315 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434199+0000", 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100203373 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434236+0000", 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6563000000000002e-05 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.435979+0000", 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017426570000000001 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.436036+0000", 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7716000000000001e-05 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437140+0000", 2023-01-15T14:30:00.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011036399999999999 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437251+0000", 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110869 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.457783+0000", 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020532181 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.699 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.700 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.700 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:00.700 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:00.700 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:00.701 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:30:00.702 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:00.700+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:00.802 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:00.801+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:00.812 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:30:00.813 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:00.822 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:30:00.903 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:00.901+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:00.930 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:00.931 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:00.931 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:00.940 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:30:00.985 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:30:00.985 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:30:00.985 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.5779683302832527 2023-01-15T14:30:00.985 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 0.5779683302832527 2023-01-15T14:30:01.004 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:01.001+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:01.063 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:30:01.084 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:01.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:166 3.8 3:1e54a7f5:::smithi09025264-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:30:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.966483385000004, 2023-01-15T14:30:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62664632300000001, 2023-01-15T14:30:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 166 2023-01-15T14:30:01.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:30:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114713+0000", 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115178+0000", 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046493199999999999 2023-01-15T14:30:01.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115208+0000", 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0689999999999999e-05 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115221+0000", 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3013000000000001e-05 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115334+0000", 2023-01-15T14:30:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112403 2023-01-15T14:30:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115505+0000", 2023-01-15T14:30:01.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170915 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333496+0000", 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217991667 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333606+0000", 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109967 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:30:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333796+0000", 2023-01-15T14:30:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018928099999999999 2023-01-15T14:30:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334289+0000", 2023-01-15T14:30:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049322600000000004 2023-01-15T14:30:01.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334339+0000", 2023-01-15T14:30:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0439999999999998e-05 2023-01-15T14:30:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334990+0000", 2023-01-15T14:30:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065082300000000003 2023-01-15T14:30:01.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335100+0000", 2023-01-15T14:30:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109357 2023-01-15T14:30:01.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.640854+0000", 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30575451300000001 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741190+0000", 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100335568 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741263+0000", 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3275999999999994e-05 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741361+0000", 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8435000000000002e-05 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:167 3.8 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:30:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.965918696000003, 2023-01-15T14:30:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62638390099999997, 2023-01-15T14:30:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 167 2023-01-15T14:30:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:30:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:30:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115279+0000", 2023-01-15T14:30:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115289+0000", 2023-01-15T14:30:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0509999999999999e-05 2023-01-15T14:30:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115306+0000", 2023-01-15T14:30:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7260000000000001e-05 2023-01-15T14:30:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115337+0000", 2023-01-15T14:30:01.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0664000000000003e-05 2023-01-15T14:30:01.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115579+0000", 2023-01-15T14:30:01.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024194400000000001 2023-01-15T14:30:01.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115744+0000", 2023-01-15T14:30:01.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016493700000000001 2023-01-15T14:30:01.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334392+0000", 2023-01-15T14:30:01.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21864830299999999 2023-01-15T14:30:01.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334453+0000", 2023-01-15T14:30:01.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1047e-05 2023-01-15T14:30:01.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:30:01.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334601+0000", 2023-01-15T14:30:01.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014824700000000001 2023-01-15T14:30:01.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:01.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334812+0000", 2023-01-15T14:30:01.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210208 2023-01-15T14:30:01.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:01.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334996+0000", 2023-01-15T14:30:01.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183878 2023-01-15T14:30:01.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:01.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335126+0000", 2023-01-15T14:30:01.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013001399999999999 2023-01-15T14:30:01.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:01.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335160+0000", 2023-01-15T14:30:01.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4876999999999998e-05 2023-01-15T14:30:01.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741044+0000", 2023-01-15T14:30:01.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40588326000000002 2023-01-15T14:30:01.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741453+0000", 2023-01-15T14:30:01.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040963099999999998 2023-01-15T14:30:01.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:01.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741548+0000", 2023-01-15T14:30:01.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4961000000000001e-05 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741664+0000", 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115333 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:01.110 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.965826582000005, 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:30:01.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:30:01.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:30:01.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:30:01.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:30:01.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:30:01.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:30:01.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:30:01.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:30:01.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:30:01.117 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.952834503000005, 2023-01-15T14:30:01.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:30:01.118 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.118 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:01.118 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.118 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.118 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.118 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:30:01.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.118 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:30:01.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:30:01.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:30:01.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:01.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:30:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:30:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:30:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:30:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:30:01.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:30:01.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:30:01.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:30:01.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:30:01.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:30:01.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:30:01.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:30:01.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:01.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:30:01.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:30:01.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:01.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:01.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:30:01.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:30:01.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:30:01.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:30:01.127 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.128 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.128 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:01.128 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:30:01.128 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.952431167, 2023-01-15T14:30:01.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:30:01.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:30:01.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:30:01.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:01.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:30:01.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:30:01.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:30:01.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:30:01.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:30:01.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:30:01.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:30:01.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:30:01.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:30:01.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:30:01.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:30:01.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:30:01.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:30:01.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:30:01.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:30:01.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:30:01.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:30:01.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:01.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:30:01.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:30:01.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:01.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:30:01.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:30:01.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:01.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:30:01.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:30:01.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:01.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:30:01.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:30:01.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:30:01.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:30:01.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:30:01.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:30:01.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.139 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.139 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:01.139 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:30:01.139 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.852207547000006, 2023-01-15T14:30:01.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:30:01.139 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.140 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:01.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.140 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:30:01.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.140 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:30:01.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:30:01.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:30:01.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:01.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:30:01.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:30:01.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:30:01.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:30:01.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:30:01.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:30:01.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:30:01.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:30:01.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:30:01.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:30:01.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:30:01.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:30:01.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.147 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:194 3.8 3:1eaad226:::smithi09025264-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:01.147 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:30:01.147 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.779806304000005, 2023-01-15T14:30:01.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64372577200000003, 2023-01-15T14:30:01.147 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.147 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:01.147 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.147 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.147 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.147 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 194 2023-01-15T14:30:01.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.148 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:30:01.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:30:01.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301390+0000", 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301991+0000", 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060122599999999995 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.302026+0000", 2023-01-15T14:30:01.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5040000000000003e-05 2023-01-15T14:30:01.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402147+0000", 2023-01-15T14:30:01.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100121084 2023-01-15T14:30:01.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402219+0000", 2023-01-15T14:30:01.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2045999999999994e-05 2023-01-15T14:30:01.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402315+0000", 2023-01-15T14:30:01.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5561000000000002e-05 2023-01-15T14:30:01.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842189+0000", 2023-01-15T14:30:01.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.439874129 2023-01-15T14:30:01.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842296+0000", 2023-01-15T14:30:01.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107021 2023-01-15T14:30:01.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:30:01.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842486+0000", 2023-01-15T14:30:01.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018965799999999999 2023-01-15T14:30:01.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:01.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842839+0000", 2023-01-15T14:30:01.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035330700000000002 2023-01-15T14:30:01.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:01.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842859+0000", 2023-01-15T14:30:01.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0347e-05 2023-01-15T14:30:01.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:01.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843212+0000", 2023-01-15T14:30:01.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352534 2023-01-15T14:30:01.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:01.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843835+0000", 2023-01-15T14:30:01.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062338599999999997 2023-01-15T14:30:01.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944861+0000", 2023-01-15T14:30:01.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10102596899999999 2023-01-15T14:30:01.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944908+0000", 2023-01-15T14:30:01.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6773000000000001e-05 2023-01-15T14:30:01.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:01.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945039+0000", 2023-01-15T14:30:01.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130684 2023-01-15T14:30:01.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945118+0000", 2023-01-15T14:30:01.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9194000000000001e-05 2023-01-15T14:30:01.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.156 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.157 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:195 3.8 3:1eaad226:::smithi09025264-49:head [setxattr _header (56) in=7b,truncate 2806736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:01.157 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.402188+0000", 2023-01-15T14:30:01.157 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.679010434000006, 2023-01-15T14:30:01.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60706172400000002, 2023-01-15T14:30:01.157 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.157 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:01.157 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.157 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.157 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 195 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402188+0000", 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402188+0000", 2023-01-15T14:30:01.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402186+0000", 2023-01-15T14:30:01.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:01.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402199+0000", 2023-01-15T14:30:01.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.24e-05 2023-01-15T14:30:01.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402222+0000", 2023-01-15T14:30:01.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-05 2023-01-15T14:30:01.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402233+0000", 2023-01-15T14:30:01.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1253000000000001e-05 2023-01-15T14:30:01.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402389+0000", 2023-01-15T14:30:01.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015541799999999999 2023-01-15T14:30:01.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402518+0000", 2023-01-15T14:30:01.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129534 2023-01-15T14:30:01.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842949+0000", 2023-01-15T14:30:01.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.440430722 2023-01-15T14:30:01.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843094+0000", 2023-01-15T14:30:01.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000144955 2023-01-15T14:30:01.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:30:01.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843391+0000", 2023-01-15T14:30:01.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000297041 2023-01-15T14:30:01.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:01.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843731+0000", 2023-01-15T14:30:01.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033986200000000002 2023-01-15T14:30:01.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:01.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843747+0000", 2023-01-15T14:30:01.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6243999999999999e-05 2023-01-15T14:30:01.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:01.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843804+0000", 2023-01-15T14:30:01.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7240000000000001e-05 2023-01-15T14:30:01.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:01.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843883+0000", 2023-01-15T14:30:01.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9040000000000002e-05 2023-01-15T14:30:01.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945184+0000", 2023-01-15T14:30:01.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101300903 2023-01-15T14:30:01.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009101+0000", 2023-01-15T14:30:01.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063916599000000004 2023-01-15T14:30:01.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:01.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009176+0000", 2023-01-15T14:30:01.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5716999999999994e-05 2023-01-15T14:30:01.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009250+0000", 2023-01-15T14:30:01.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3213000000000005e-05 2023-01-15T14:30:01.166 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.166 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.167 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.167 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:196 3.8 3:1eaad226:::smithi09025264-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:01.167 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.402249+0000", 2023-01-15T14:30:01.167 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.678949587999995, 2023-01-15T14:30:01.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60743240399999998, 2023-01-15T14:30:01.167 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.167 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:01.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.167 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.168 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.168 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 196 2023-01-15T14:30:01.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.168 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402249+0000", 2023-01-15T14:30:01.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402249+0000", 2023-01-15T14:30:01.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402248+0000", 2023-01-15T14:30:01.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:01.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402253+0000", 2023-01-15T14:30:01.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8829999999999997e-06 2023-01-15T14:30:01.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402263+0000", 2023-01-15T14:30:01.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0013999999999999e-05 2023-01-15T14:30:01.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402269+0000", 2023-01-15T14:30:01.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1360000000000002e-06 2023-01-15T14:30:01.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402551+0000", 2023-01-15T14:30:01.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000281799 2023-01-15T14:30:01.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402620+0000", 2023-01-15T14:30:01.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9289999999999996e-05 2023-01-15T14:30:01.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.942361+0000", 2023-01-15T14:30:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.53974109800000003 2023-01-15T14:30:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.942474+0000", 2023-01-15T14:30:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112713 2023-01-15T14:30:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009383+0000", 2023-01-15T14:30:01.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.066909386000000001 2023-01-15T14:30:01.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009511+0000", 2023-01-15T14:30:01.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012762 2023-01-15T14:30:01.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:28.009681+0000", 2023-01-15T14:30:01.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017044099999999999 2023-01-15T14:30:01.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.174 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.174 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:525 3.8 3:17d5d741:::smithi09025264-5:head [write 2555481~425187] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:30:01.174 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:30:01.174 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.262241377999999, 2023-01-15T14:30:01.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60814212099999998, 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:30:01.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:30:01.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818955+0000", 2023-01-15T14:30:01.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:01.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819151+0000", 2023-01-15T14:30:01.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632699999999999 2023-01-15T14:30:01.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819175+0000", 2023-01-15T14:30:01.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3777e-05 2023-01-15T14:30:01.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819185+0000", 2023-01-15T14:30:01.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0424999999999999e-05 2023-01-15T14:30:01.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019555+0000", 2023-01-15T14:30:01.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20036944200000001 2023-01-15T14:30:01.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019726+0000", 2023-01-15T14:30:01.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017075099999999999 2023-01-15T14:30:01.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224068+0000", 2023-01-15T14:30:01.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20434244100000001 2023-01-15T14:30:01.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224217+0000", 2023-01-15T14:30:01.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014933200000000001 2023-01-15T14:30:01.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:30:01.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224571+0000", 2023-01-15T14:30:01.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035323700000000001 2023-01-15T14:30:01.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:01.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225080+0000", 2023-01-15T14:30:01.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050910500000000004 2023-01-15T14:30:01.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:01.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225103+0000", 2023-01-15T14:30:01.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3295999999999999e-05 2023-01-15T14:30:01.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:01.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225542+0000", 2023-01-15T14:30:01.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043897599999999999 2023-01-15T14:30:01.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:01.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226915+0000", 2023-01-15T14:30:01.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013730599999999999 2023-01-15T14:30:01.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.327618+0000", 2023-01-15T14:30:01.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100702655 2023-01-15T14:30:01.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426890+0000", 2023-01-15T14:30:01.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099272284000000002 2023-01-15T14:30:01.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:01.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426977+0000", 2023-01-15T14:30:01.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6500000000000002e-05 2023-01-15T14:30:01.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427099+0000", 2023-01-15T14:30:01.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122562 2023-01-15T14:30:01.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.185 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.185 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:526 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (56) in=7b,truncate 3512630] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:30:01.185 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:30:01.185 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.261993645999993, 2023-01-15T14:30:01.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60866722500000003, 2023-01-15T14:30:01.185 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.186 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:01.186 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.186 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.186 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.186 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2023-01-15T14:30:01.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.186 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:30:01.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:30:01.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819204+0000", 2023-01-15T14:30:01.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:01.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819210+0000", 2023-01-15T14:30:01.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5429999999999999e-06 2023-01-15T14:30:01.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819235+0000", 2023-01-15T14:30:01.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4114e-05 2023-01-15T14:30:01.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819246+0000", 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1043000000000001e-05 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019911+0000", 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200665175 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020085+0000", 2023-01-15T14:30:01.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000174449 2023-01-15T14:30:01.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225345+0000", 2023-01-15T14:30:01.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20526017099999999 2023-01-15T14:30:01.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225560+0000", 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021505499999999999 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225908+0000", 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034754600000000001 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226267+0000", 2023-01-15T14:30:01.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035936399999999998 2023-01-15T14:30:01.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:01.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226688+0000", 2023-01-15T14:30:01.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042024599999999999 2023-01-15T14:30:01.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:01.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226841+0000", 2023-01-15T14:30:01.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015324499999999999 2023-01-15T14:30:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226936+0000", 2023-01-15T14:30:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5549000000000003e-05 2023-01-15T14:30:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426826+0000", 2023-01-15T14:30:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19988924 2023-01-15T14:30:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427685+0000", 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000859719 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427781+0000", 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5515999999999998e-05 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427872+0000", 2023-01-15T14:30:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1142000000000003e-05 2023-01-15T14:30:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:527 3.8 3:17d5d741:::smithi09025264-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:30:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.261943157999994, 2023-01-15T14:30:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60906366599999995, 2023-01-15T14:30:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2023-01-15T14:30:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819259+0000", 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5940000000000004e-06 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819268+0000", 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.115e-06 2023-01-15T14:30:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.919405+0000", 2023-01-15T14:30:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013656999999999 2023-01-15T14:30:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020192+0000", 2023-01-15T14:30:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10078725600000001 2023-01-15T14:30:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020326+0000", 2023-01-15T14:30:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133745 2023-01-15T14:30:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226443+0000", 2023-01-15T14:30:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20611705699999999 2023-01-15T14:30:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226689+0000", 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024580799999999997 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427991+0000", 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20130269000000001 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.428147+0000", 2023-01-15T14:30:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155258 2023-01-15T14:30:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.428319+0000", 2023-01-15T14:30:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017226200000000001 2023-01-15T14:30:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:643 3.8 3:1eaad226:::smithi09025264-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e57)", 2023-01-15T14:30:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:02.736009+0000", 2023-01-15T14:30:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.345189277999999, 2023-01-15T14:30:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58661375400000004, 2023-01-15T14:30:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 643 2023-01-15T14:30:01.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736009+0000", 2023-01-15T14:30:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736009+0000", 2023-01-15T14:30:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736008+0000", 2023-01-15T14:30:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736014+0000", 2023-01-15T14:30:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7309999999999998e-06 2023-01-15T14:30:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736023+0000", 2023-01-15T14:30:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.704e-06 2023-01-15T14:30:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.736028+0000", 2023-01-15T14:30:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3630000000000002e-06 2023-01-15T14:30:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.836277+0000", 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100248614 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:02.836291+0000", 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.428e-05 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.014657+0000", 2023-01-15T14:30:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.178365366 2023-01-15T14:30:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.014802+0000", 2023-01-15T14:30:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014527799999999999 2023-01-15T14:30:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.152824+0000", 2023-01-15T14:30:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13802230900000001 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.152918+0000", 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4104999999999994e-05 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.322349+0000", 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16943072200000001 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.322513+0000", 2023-01-15T14:30:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016396299999999999 2023-01-15T14:30:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:03.322623+0000", 2023-01-15T14:30:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109978 2023-01-15T14:30:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:705 3.2 3:47dfe935:::smithi09025264-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2023-01-15T14:30:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:30:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.970614396000002, 2023-01-15T14:30:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61170208599999998, 2023-01-15T14:30:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 705 2023-01-15T14:30:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:30:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:30:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110583+0000", 2023-01-15T14:30:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110587+0000", 2023-01-15T14:30:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9689999999999996e-06 2023-01-15T14:30:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110601+0000", 2023-01-15T14:30:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.393e-05 2023-01-15T14:30:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110611+0000", 2023-01-15T14:30:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3999999999999998e-06 2023-01-15T14:30:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319174+0000", 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20856376800000001 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319367+0000", 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192361 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.721941+0000", 2023-01-15T14:30:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40257461300000003 2023-01-15T14:30:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722108+0000", 2023-01-15T14:30:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167054 2023-01-15T14:30:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722286+0000", 2023-01-15T14:30:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017772099999999999 2023-01-15T14:30:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:30:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.1588923219999998, 2023-01-15T14:30:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:30:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:30:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:30:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:30:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:30:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:30:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:30:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:30:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:30:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:30:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:30:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:30:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:30:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:30:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:30:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:30:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:30:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:30:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:30:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:30:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.152430968, 2023-01-15T14:30:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:30:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:30:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:30:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:30:01.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:30:01.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:01.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:30:01.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:30:01.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:30:01.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:30:01.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:30:01.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:30:01.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:30:01.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:30:01.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:01.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:30:01.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:30:01.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:30:01.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:30:01.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:30:01.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:30:01.229 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.229 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.229 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.229 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:01.229 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:30:01.230 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.052009043, 2023-01-15T14:30:01.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:30:01.230 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.230 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:01.230 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.230 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.230 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.230 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:30:01.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.230 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:30:01.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:30:01.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:30:01.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:01.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:30:01.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:30:01.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:30:01.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:30:01.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:30:01.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:30:01.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:30:01.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:30:01.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:01.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:30:01.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:30:01.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:30:01.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:30:01.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:30:01.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:30:01.235 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.235 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.235 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.235 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:01.236 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:30:01.236 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.0516715570000001, 2023-01-15T14:30:01.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:30:01.236 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.236 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:01.236 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.236 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.236 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.236 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:30:01.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.236 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:30:01.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:30:01.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:30:01.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:01.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:30:01.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:30:01.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:30:01.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:30:01.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:30:01.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:30:01.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:30:01.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:30:01.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:01.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:30:01.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:30:01.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:30:01.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:30:01.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:30:01.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:30:01.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.241 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.242 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:01.242 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:30:01.242 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.051615661, 2023-01-15T14:30:01.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:30:01.242 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.242 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:01.242 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.242 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.242 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.242 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:30:01.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.243 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:30:01.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:30:01.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:30:01.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:01.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:30:01.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:30:01.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:30:01.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:30:01.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:01.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:30:01.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:30:01.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:30:01.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:30:01.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:30:01.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:30:01.247 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.247 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.247 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.247 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1314 3.18 3.afe52a78 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:30:01.248 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:30:01.248 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.7213205299999998, 2023-01-15T14:30:01.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62560938700000002, 2023-01-15T14:30:01.248 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:01.248 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:01.248 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:01.248 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:01.248 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:01.248 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1314 2023-01-15T14:30:01.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:01.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359877+0000", 2023-01-15T14:30:01.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:01.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:01.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359882+0000", 2023-01-15T14:30:01.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0050000000000004e-06 2023-01-15T14:30:01.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:01.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359893+0000", 2023-01-15T14:30:01.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0676000000000001e-05 2023-01-15T14:30:01.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:01.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359904+0000", 2023-01-15T14:30:01.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0788000000000001e-05 2023-01-15T14:30:01.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460094+0000", 2023-01-15T14:30:01.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10019024999999999 2023-01-15T14:30:01.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:01.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460106+0000", 2023-01-15T14:30:01.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354999999999999e-05 2023-01-15T14:30:01.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:01.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985477+0000", 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52537039100000005 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985487+0000", 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:01.253 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:01.255 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:30:01.256 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:01.256 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:01.256 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:01.256 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:01.257 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:01.252+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:01.281 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:01.296 INFO:tasks.rados.rados.0.smithi090.stdout:471: finishing write tid 1 to smithi09025264-49 2023-01-15T14:30:01.300 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:01.322 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:01.340 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:01.342 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:30:01.342 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:30:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:30:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:30:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:30:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:30:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:30:01.343 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:30:01.369 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:01.369 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:01.379 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:30:01.398 INFO:tasks.rados.rados.0.smithi090.stdout:471: finishing write tid 2 to smithi09025264-49 2023-01-15T14:30:01.398 INFO:tasks.rados.rados.0.smithi090.stdout:471: finishing write tid 3 to smithi09025264-49 2023-01-15T14:30:01.399 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:01.397+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:01.457 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:01.453+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:01.483 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:01.483 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:01.492 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:30:01.499 INFO:tasks.rados.rados.0.smithi090.stdout:471: finishing write tid 4 to smithi09025264-49 2023-01-15T14:30:01.500 INFO:tasks.rados.rados.0.smithi090.stdout:471: finishing write tid 5 to smithi09025264-49 2023-01-15T14:30:01.500 INFO:tasks.rados.rados.0.smithi090.stdout:471: oid 49 updating version 0 to 180 2023-01-15T14:30:01.500 INFO:tasks.rados.rados.0.smithi090.stdout:471: oid 49 updating version 180 to 181 2023-01-15T14:30:01.500 INFO:tasks.rados.rados.0.smithi090.stdout:471: oid 49 version 181 is already newer than 179 2023-01-15T14:30:01.500 INFO:tasks.rados.rados.0.smithi090.stdout:471: oid 49 version 181 is already newer than 178 2023-01-15T14:30:01.500 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 181 (ObjNum 196 snap 43 seq_num 196) dirty exists 2023-01-15T14:30:01.501 INFO:tasks.rados.rados.0.smithi090.stdout:471: left oid 49 (ObjNum 196 snap 43 seq_num 196) 2023-01-15T14:30:01.501 INFO:tasks.rados.rados.0.smithi090.stdout:471: done (1 left) 2023-01-15T14:30:01.501 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:01.557 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:01.553+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:01.600 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:01.600 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:01.610 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:30:01.658 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:01.656+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:01.702 INFO:tasks.rados.rados.0.smithi090.stdout:479: finishing rollback tid 1 to smithi09025264-42 2023-01-15T14:30:01.702 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 42 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2023-01-15T14:30:01.702 INFO:tasks.rados.rados.0.smithi090.stdout:479: done (0 left) 2023-01-15T14:30:01.715 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:01.715 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:01.724 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:30:01.758 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:01.757+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:01.860 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:01.857+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:01.900 INFO:tasks.rados.rados.0.smithi090.stdout:484: done (0 left) 2023-01-15T14:30:01.980 INFO:tasks.rados.rados.0.smithi090.stdout:485: delete oid 24 current snap is 44 2023-01-15T14:30:01.983 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:01.983 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:01.984 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.7 to 0.577968 (93f5) 2023-01-15T14:30:01.984 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:30:01.985 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:01.980+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:02.082 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:02.080+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:02.141 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:02.141 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:02.142 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:30:02.143 INFO:tasks.rados.rados.0.smithi090.stdout:485: done (0 left) 2023-01-15T14:30:02.143 INFO:tasks.rados.rados.0.smithi090.stdout:486: write oid 5 current snap is 44 2023-01-15T14:30:02.143 INFO:tasks.rados.rados.0.smithi090.stdout:486: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2023-01-15T14:30:02.143 INFO:tasks.rados.rados.0.smithi090.stdout:486: writing smithi09025264-5 from 628628 to 1404602 tid 1 2023-01-15T14:30:02.143 INFO:tasks.rados.rados.0.smithi090.stdout:486: writing smithi09025264-5 from 1937856 to 2464917 tid 2 2023-01-15T14:30:02.143 INFO:tasks.rados.rados.0.smithi090.stdout:486: writing smithi09025264-5 from 3252761 to 3428628 tid 3 2023-01-15T14:30:02.143 INFO:tasks.rados.rados.0.smithi090.stdout:487: read oid 17 snap -1 2023-01-15T14:30:02.143 INFO:tasks.rados.rados.0.smithi090.stdout:487: expect deleted 2023-01-15T14:30:02.143 INFO:tasks.rados.rados.0.smithi090.stdout:488: read oid 6 snap -1 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:488: expect (ObjNum 190 snap 42 seq_num 190) 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:489: read oid 48 snap -1 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:489: expect (ObjNum 188 snap 41 seq_num 188) 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:490: write oid 10 current snap is 44 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:490: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:490: writing smithi09025264-10 from 645435 to 1096658 tid 1 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:490: writing smithi09025264-10 from 1879985 to 2667057 tid 2 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:490: writing smithi09025264-10 from 3280184 to 3445435 tid 3 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:491: read oid 43 snap 42 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:491: expect deleted 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:492: read oid 1 snap -1 2023-01-15T14:30:02.144 INFO:tasks.rados.rados.0.smithi090.stdout:492: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:30:02.145 INFO:tasks.rados.rados.0.smithi090.stdout:493: rollback oid 3 current snap is 44 2023-01-15T14:30:02.145 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 3 to 42 2023-01-15T14:30:02.145 INFO:tasks.rados.rados.0.smithi090.stdout:488: expect (ObjNum 190 snap 42 seq_num 190) 2023-01-15T14:30:02.189 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:02.188+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:02.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:30:02.317 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:02.317 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:02.318 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:30:02.318 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:02.316+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:02.342 INFO:tasks.rados.rados.0.smithi090.stdout:489: expect (ObjNum 188 snap 41 seq_num 188) 2023-01-15T14:30:02.417 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:02.413+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:02.428 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:02.428 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:02.438 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:30:02.537 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:02.536+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:02.660 INFO:tasks.rados.rados.0.smithi090.stdout:492: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:30:02.660 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:02.657+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:02.660 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:02.661 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:02.661 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:30:02.673 INFO:tasks.rados.rados.0.smithi090.stdout:493: finishing rollback tid 1 to smithi09025264-3 2023-01-15T14:30:02.686 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 0 (ObjNum 65 snap 0 seq_num 3182663320) dirty dne 2023-01-15T14:30:02.686 INFO:tasks.rados.rados.0.smithi090.stdout:487: done (7 left) 2023-01-15T14:30:02.686 INFO:tasks.rados.rados.0.smithi090.stdout:488: done (6 left) 2023-01-15T14:30:02.686 INFO:tasks.rados.rados.0.smithi090.stdout:489: done (5 left) 2023-01-15T14:30:02.686 INFO:tasks.rados.rados.0.smithi090.stdout:491: done (4 left) 2023-01-15T14:30:02.687 INFO:tasks.rados.rados.0.smithi090.stdout:492: done (3 left) 2023-01-15T14:30:02.687 INFO:tasks.rados.rados.0.smithi090.stdout:493: done (2 left) 2023-01-15T14:30:02.687 INFO:tasks.rados.rados.0.smithi090.stdout:494: snap_remove snap 42 2023-01-15T14:30:02.760 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:02.757+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:02.838 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:02.839 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:02.841 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:30:02.862 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:02.857+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:02.902 INFO:tasks.rados.rados.0.smithi090.stdout:494: done (2 left) 2023-01-15T14:30:03.030 INFO:tasks.rados.rados.0.smithi090.stdout:495: rollback oid 35 current snap is 44 2023-01-15T14:30:03.031 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 35 to 43 2023-01-15T14:30:03.031 INFO:tasks.rados.rados.0.smithi090.stdout:496: write oid 30 current snap is 44 2023-01-15T14:30:03.031 INFO:tasks.rados.rados.0.smithi090.stdout:496: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2023-01-15T14:30:03.036 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:30:03.037 INFO:tasks.rados.rados.0.smithi090.stdout:496: writing smithi09025264-30 from 662242 to 1305067 tid 1 2023-01-15T14:30:03.037 INFO:tasks.rados.rados.0.smithi090.stdout:496: writing smithi09025264-30 from 2054820 to 2585550 tid 2 2023-01-15T14:30:03.038 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:03.149 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:03.149 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:30:03.150 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:03.032+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:03.151 INFO:tasks.rados.rados.0.smithi090.stdout:496: writing smithi09025264-30 from 3307607 to 3462242 tid 3 2023-01-15T14:30:03.151 INFO:tasks.rados.rados.0.smithi090.stdout:497: delete oid 9 current snap is 44 2023-01-15T14:30:03.151 INFO:tasks.rados.rados.0.smithi090.stdout:490: finishing write tid 1 to smithi09025264-10 2023-01-15T14:30:03.151 INFO:tasks.rados.rados.0.smithi090.stdout:490: finishing write tid 2 to smithi09025264-10 2023-01-15T14:30:03.152 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:03.148+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:03.157 INFO:tasks.rados.rados.0.smithi090.stdout:490: finishing write tid 3 to smithi09025264-10 2023-01-15T14:30:03.253 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:03.249+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:03.312 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:03.312 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:03.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:30:03.354 INFO:tasks.rados.rados.0.smithi090.stdout:495: finishing rollback tid 1 to smithi09025264-35 2023-01-15T14:30:03.354 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 0 (ObjNum 657 snap 0 seq_num 3844009568) dirty dne 2023-01-15T14:30:03.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:30:03.355 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:03.353+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:03.356 INFO:tasks.rados.rados.0.smithi090.stdout:490: finishing write tid 4 to smithi09025264-10 2023-01-15T14:30:03.357 INFO:tasks.rados.rados.0.smithi090.stdout:490: finishing write tid 5 to smithi09025264-10 2023-01-15T14:30:03.357 INFO:tasks.rados.rados.0.smithi090.stdout:490: oid 10 updating version 0 to 219 2023-01-15T14:30:03.357 INFO:tasks.rados.rados.0.smithi090.stdout:490: oid 10 updating version 219 to 220 2023-01-15T14:30:03.357 INFO:tasks.rados.rados.0.smithi090.stdout:490: oid 10 version 220 is already newer than 218 2023-01-15T14:30:03.357 INFO:tasks.rados.rados.0.smithi090.stdout:490: oid 10 version 220 is already newer than 217 2023-01-15T14:30:03.358 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 220 (ObjNum 205 snap 44 seq_num 205) dirty exists 2023-01-15T14:30:03.358 INFO:tasks.rados.rados.0.smithi090.stdout:490: left oid 10 (ObjNum 205 snap 44 seq_num 205) 2023-01-15T14:30:03.450 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:03.544 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:03.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:30:03.546 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:03.546 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:03.546 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:03.546 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:03.541+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:03.565 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:03.567 INFO:tasks.rados.rados.0.smithi090.stdout:490: done (4 left) 2023-01-15T14:30:03.567 INFO:tasks.rados.rados.0.smithi090.stdout:495: done (3 left) 2023-01-15T14:30:03.567 INFO:tasks.rados.rados.0.smithi090.stdout:497: done (2 left) 2023-01-15T14:30:03.567 INFO:tasks.rados.rados.0.smithi090.stdout:498: snap_create 2023-01-15T14:30:03.567 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:03.589 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:03.705 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:03.705 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:03.705 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:03.706 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:03.706 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:03.706 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:03.706 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:30:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:30:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:30:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:30:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:30:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:30:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:30:03.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:30:03.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:30:03.709 INFO:tasks.rados.rados.0.smithi090.stdout:496: finishing write tid 1 to smithi09025264-30 2023-01-15T14:30:03.709 INFO:tasks.rados.rados.0.smithi090.stdout:496: finishing write tid 2 to smithi09025264-30 2023-01-15T14:30:03.710 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:03.704+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:03.795 INFO:tasks.rados.rados.0.smithi090.stdout:496: finishing write tid 3 to smithi09025264-30 2023-01-15T14:30:03.877 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.878 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:30 3.1 3:8b0a2647:::smithi09025264-8:head [write 1776668~340981] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:30:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.962152353999997, 2023-01-15T14:30:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40319718700000001, 2023-01-15T14:30:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:03.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 30 2023-01-15T14:30:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:03.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:30:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859913+0000", 2023-01-15T14:30:03.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.859909+0000", 2023-01-15T14:30:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:30:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.860361+0000", 2023-01-15T14:30:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045164599999999999 2023-01-15T14:30:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:03.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.860392+0000", 2023-01-15T14:30:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1331000000000002e-05 2023-01-15T14:30:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960486+0000", 2023-01-15T14:30:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100093583 2023-01-15T14:30:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.160839+0000", 2023-01-15T14:30:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20035291199999999 2023-01-15T14:30:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.160982+0000", 2023-01-15T14:30:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014350800000000001 2023-01-15T14:30:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161183+0000", 2023-01-15T14:30:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000200411 2023-01-15T14:30:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:03.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161525+0000", 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034216500000000001 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161587+0000", 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2032999999999994e-05 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161842+0000", 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000255305 2023-01-15T14:30:03.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262815+0000", 2023-01-15T14:30:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100972992 2023-01-15T14:30:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:03.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262914+0000", 2023-01-15T14:30:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9166999999999994e-05 2023-01-15T14:30:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262967+0000", 2023-01-15T14:30:03.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2244000000000003e-05 2023-01-15T14:30:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263050+0000", 2023-01-15T14:30:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2999999999999998e-05 2023-01-15T14:30:03.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.263110+0000", 2023-01-15T14:30:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0297000000000002e-05 2023-01-15T14:30:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:03.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:31 3.1 3:8b0a2647:::smithi09025264-8:head [setxattr _header (55) in=7b,truncate 2117649] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:30:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.861549987999993, 2023-01-15T14:30:03.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405055944, 2023-01-15T14:30:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:03.893 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 31 2023-01-15T14:30:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:30:03.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960515+0000", 2023-01-15T14:30:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:03.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960513+0000", 2023-01-15T14:30:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960524+0000", 2023-01-15T14:30:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0614e-05 2023-01-15T14:30:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:03.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:25.960538+0000", 2023-01-15T14:30:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4226999999999999e-05 2023-01-15T14:30:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.060704+0000", 2023-01-15T14:30:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166252 2023-01-15T14:30:03.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.261723+0000", 2023-01-15T14:30:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20101896699999999 2023-01-15T14:30:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.261869+0000", 2023-01-15T14:30:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014578099999999999 2023-01-15T14:30:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262092+0000", 2023-01-15T14:30:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223198 2023-01-15T14:30:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:03.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262397+0000", 2023-01-15T14:30:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030490500000000001 2023-01-15T14:30:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262446+0000", 2023-01-15T14:30:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8492999999999998e-05 2023-01-15T14:30:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262537+0000", 2023-01-15T14:30:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1248000000000005e-05 2023-01-15T14:30:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.262847+0000", 2023-01-15T14:30:03.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031007799999999998 2023-01-15T14:30:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365234+0000", 2023-01-15T14:30:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102386856 2023-01-15T14:30:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365326+0000", 2023-01-15T14:30:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2226999999999996e-05 2023-01-15T14:30:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365413+0000", 2023-01-15T14:30:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6996999999999994e-05 2023-01-15T14:30:03.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365571+0000", 2023-01-15T14:30:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015794100000000001 2023-01-15T14:30:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:03.906 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:30:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.760800259000007, 2023-01-15T14:30:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:30:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:03.907 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:30:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:03.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:30:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:03.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:30:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:30:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:30:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:30:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:03.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:30:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:30:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:30:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:30:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:30:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:30:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:30:03.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:30:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:30:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:30:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:30:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:30:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:30:03.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:30:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:30:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:30:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:03.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:30:03.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:30:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:30:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:30:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:30:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:30:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:30:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:30:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:30:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:03.919 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:30:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.760520010999997, 2023-01-15T14:30:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:30:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:30:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:30:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:30:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:30:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:30:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:30:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:30:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:30:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:30:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:30:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:30:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:30:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:30:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:30:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:30:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:30:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:30:03.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:30:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:30:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:30:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:30:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:30:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:30:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:30:04.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:30:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:30:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:30:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:30:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.760487393999995, 2023-01-15T14:30:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:30:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:30:04.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:30:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:30:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:30:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:30:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:30:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:30:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:30:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:30:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:30:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:30:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:30:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:30:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:30:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:30:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:30:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:30:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:30:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:30:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:30:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:30:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.659169407999997, 2023-01-15T14:30:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:30:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:30:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:30:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:30:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:30:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:30:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:30:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:30:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:30:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:30:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:30:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:30:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:30:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:30:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:30:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:30:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:30:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:30:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:30:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:30:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:30:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:30:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:30:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:30:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:30:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:30:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:30:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:30:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:30:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.659139405000005, 2023-01-15T14:30:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:30:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:30:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:30:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:30:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:30:04.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:30:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:30:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:30:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:30:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:30:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:30:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:30:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:30:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:30:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:30:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:30:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:30:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:125 3.1 3:8889e450:::smithi09025264-32:head [write 521017~798131,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:30:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.149250586999997, 2023-01-15T14:30:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40348810600000001, 2023-01-15T14:30:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.070 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 125 2023-01-15T14:30:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:30:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672815+0000", 2023-01-15T14:30:04.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.672812+0000", 2023-01-15T14:30:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673338+0000", 2023-01-15T14:30:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052553599999999995 2023-01-15T14:30:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673372+0000", 2023-01-15T14:30:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3652999999999998e-05 2023-01-15T14:30:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.673386+0000", 2023-01-15T14:30:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4171e-05 2023-01-15T14:30:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773535+0000", 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148822 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773727+0000", 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192189 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.773918+0000", 2023-01-15T14:30:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019143099999999999 2023-01-15T14:30:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774291+0000", 2023-01-15T14:30:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037288399999999998 2023-01-15T14:30:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774341+0000", 2023-01-15T14:30:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0117999999999997e-05 2023-01-15T14:30:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.774696+0000", 2023-01-15T14:30:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000355268 2023-01-15T14:30:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874892+0000", 2023-01-15T14:30:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100195981 2023-01-15T14:30:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.977868+0000", 2023-01-15T14:30:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102975649 2023-01-15T14:30:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.075846+0000", 2023-01-15T14:30:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097977838999999997 2023-01-15T14:30:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.076004+0000", 2023-01-15T14:30:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015859700000000001 2023-01-15T14:30:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.076303+0000", 2023-01-15T14:30:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029816400000000001 2023-01-15T14:30:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:30:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.144130840000003, 2023-01-15T14:30:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:30:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:30:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:30:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:30:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:30:04.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:30:04.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:30:04.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:30:04.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:30:04.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:30:04.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:30:04.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:30:04.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:30:04.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:30:04.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:30:04.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:04.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:30:04.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:30:04.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:04.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:30:04.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:30:04.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:04.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:30:04.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:30:04.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:30:04.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:30:04.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:30:04.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:30:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:30:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:30:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:30:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:30:04.459 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.460 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.460 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.460 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:04.460 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:30:04.460 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.143648486000004, 2023-01-15T14:30:04.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:30:04.461 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.461 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:04.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.462 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.462 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:30:04.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.462 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:30:04.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:30:04.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:30:04.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:04.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:30:04.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:30:04.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:30:04.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:30:04.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:30:04.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:30:04.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:30:04.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:30:04.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:30:04.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:30:04.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:30:04.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:30:04.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:30:04.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:30:04.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:04.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:30:04.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:30:04.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:04.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:30:04.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:30:04.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:04.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:30:04.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:30:04.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:04.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:30:04.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:30:04.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:04.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:30:04.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:30:04.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:30:04.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:30:04.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:30:04.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:30:04.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:04.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:30:04.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:30:04.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:30:04.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:30:04.483 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.483 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.483 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.484 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:04.484 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:30:04.484 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.143595923000007, 2023-01-15T14:30:04.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:30:04.484 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.485 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:04.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.485 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:30:04.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.486 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:30:04.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:30:04.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:30:04.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:04.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:30:04.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:30:04.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:30:04.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:30:04.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:30:04.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:30:04.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:30:04.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:30:04.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:30:04.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:30:04.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:30:04.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:30:04.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:30:04.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:30:04.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:30:04.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:30:04.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:30:04.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:30:04.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:30:04.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:30:04.498 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.498 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.498 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.499 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:04.499 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:30:04.499 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 83.772394219000006, 2023-01-15T14:30:04.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:30:04.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:04.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.500 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.501 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:30:04.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.501 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:30:04.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:30:04.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:30:04.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:30:04.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:30:04.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:30:04.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:30:04.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:30:04.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:30:04.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:30:04.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:30:04.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:30:04.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:30:04.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:30:04.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:04.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:30:04.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:30:04.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:04.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:30:04.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:30:04.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:04.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:30:04.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:30:04.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:04.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:30:04.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:30:04.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:04.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:30:04.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:30:04.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:30:04.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:30:04.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:30:04.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:30:04.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:04.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:30:04.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:30:04.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:30:04.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:30:04.517 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.517 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.517 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.517 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:04.518 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:30:04.518 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 83.772277544999994, 2023-01-15T14:30:04.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:30:04.518 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.519 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:04.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.519 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:30:04.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.520 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:30:04.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:30:04.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:30:04.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:04.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:30:04.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:30:04.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:30:04.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:30:04.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:30:04.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:30:04.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:30:04.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:30:04.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:04.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:30:04.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:30:04.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:30:04.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:30:04.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:30:04.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:30:04.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:30:04.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:30:04.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:30:04.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:30:04.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:30:04.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:30:04.530 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.530 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.530 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.531 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:448 3.1 3:8b0a2647:::smithi09025264-8:head [write 1856548~737690] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:30:04.531 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:30:04.531 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.525019637, 2023-01-15T14:30:04.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40560468799999999, 2023-01-15T14:30:04.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.531 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:04.531 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.531 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.531 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.531 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 448 2023-01-15T14:30:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:30:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297043+0000", 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297477+0000", 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043336500000000001 2023-01-15T14:30:04.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297505+0000", 2023-01-15T14:30:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8583000000000001e-05 2023-01-15T14:30:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397598+0000", 2023-01-15T14:30:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092164 2023-01-15T14:30:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397683+0000", 2023-01-15T14:30:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5986e-05 2023-01-15T14:30:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397798+0000", 2023-01-15T14:30:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114916 2023-01-15T14:30:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498684+0000", 2023-01-15T14:30:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10088510000000001 2023-01-15T14:30:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498848+0000", 2023-01-15T14:30:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164575 2023-01-15T14:30:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499083+0000", 2023-01-15T14:30:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000235171 2023-01-15T14:30:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499670+0000", 2023-01-15T14:30:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058650200000000003 2023-01-15T14:30:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:04.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499688+0000", 2023-01-15T14:30:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8547e-05 2023-01-15T14:30:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.500037+0000", 2023-01-15T14:30:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034891399999999997 2023-01-15T14:30:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.501925+0000", 2023-01-15T14:30:04.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00188766 2023-01-15T14:30:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.601919+0000", 2023-01-15T14:30:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099994371999999998 2023-01-15T14:30:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702377+0000", 2023-01-15T14:30:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045810500000001 2023-01-15T14:30:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702537+0000", 2023-01-15T14:30:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015941600000000001 2023-01-15T14:30:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702650+0000", 2023-01-15T14:30:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011338899999999999 2023-01-15T14:30:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:30:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:30:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.046145268999993, 2023-01-15T14:30:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:30:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:30:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:30:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:30:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:30:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:30:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:30:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:30:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:30:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:30:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:30:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:30:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:30:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:30:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:30:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:30:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:30:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:30:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:30:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:30:04.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.829895209999997, 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:04.662 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:30:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:30:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:30:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:30:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:30:04.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:30:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:30:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:30:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:30:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:30:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:30:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:30:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:30:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:30:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:30:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:30:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:30:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:04.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:30:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:30:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:30:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:30:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:30:04.671 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:30:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.829861643000001, 2023-01-15T14:30:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:30:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:04.672 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:30:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:30:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:30:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:30:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:30:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:30:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:30:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:30:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:30:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:04.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:30:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:30:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:30:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:30:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:30:04.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:30:04.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:30:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:30:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:30:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:30:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:30:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:30:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:30:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:30:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:30:04.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.063958097, 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:04.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:30:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:30:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:30:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:30:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:30:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:30:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:30:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:30:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:30:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:30:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:30:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:30:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:30:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:30:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:30:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:30:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:30:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:30:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:30:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:30:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:30:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:30:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:30:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:30:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:30:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:30:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:30:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:30:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:30:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:30:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:30:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:30:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:30:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:30:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.5207792969999998, 2023-01-15T14:30:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:30:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:30:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:30:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:30:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:30:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:30:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:30:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:30:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:30:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:30:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:30:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:30:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:30:04.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:30:04.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:30:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:30:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:30:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:30:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:30:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:30:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:30:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:30:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:30:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:30:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:30:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:30:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:30:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.5207401220000003, 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:30:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:30:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:30:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:30:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:30:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:30:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:30:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:30:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:30:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:30:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:30:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:30:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:30:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:30:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:05.080 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:05.080 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:30:05.081 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:03.872+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:05.082 INFO:tasks.rados.rados.0.smithi090.stdout:496: finishing write tid 4 to smithi09025264-30 2023-01-15T14:30:05.083 INFO:tasks.rados.rados.0.smithi090.stdout:496: finishing write tid 5 to smithi09025264-30 2023-01-15T14:30:05.083 INFO:tasks.rados.rados.0.smithi090.stdout:496: oid 30 updating version 0 to 71 2023-01-15T14:30:05.083 INFO:tasks.rados.rados.0.smithi090.stdout:496: oid 30 updating version 71 to 73 2023-01-15T14:30:05.083 INFO:tasks.rados.rados.0.smithi090.stdout:496: oid 30 version 73 is already newer than 72 2023-01-15T14:30:05.083 INFO:tasks.rados.rados.0.smithi090.stdout:496: oid 30 version 73 is already newer than 70 2023-01-15T14:30:05.083 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 73 (ObjNum 206 snap 44 seq_num 206) dirty exists 2023-01-15T14:30:05.083 INFO:tasks.rados.rados.0.smithi090.stdout:496: left oid 30 (ObjNum 206 snap 44 seq_num 206) 2023-01-15T14:30:05.083 INFO:tasks.rados.rados.0.smithi090.stdout:496: done (1 left) 2023-01-15T14:30:05.083 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:05.083 INFO:tasks.rados.rados.0.smithi090.stdout:486: finishing write tid 1 to smithi09025264-5 2023-01-15T14:30:05.083 INFO:tasks.rados.rados.0.smithi090.stdout:486: finishing write tid 2 to smithi09025264-5 2023-01-15T14:30:05.084 INFO:tasks.rados.rados.0.smithi090.stdout:486: finishing write tid 3 to smithi09025264-5 2023-01-15T14:30:05.084 INFO:tasks.rados.rados.0.smithi090.stdout:486: finishing write tid 4 to smithi09025264-5 2023-01-15T14:30:05.084 INFO:tasks.rados.rados.0.smithi090.stdout:486: finishing write tid 5 to smithi09025264-5 2023-01-15T14:30:05.084 INFO:tasks.rados.rados.0.smithi090.stdout:486: oid 5 updating version 0 to 181 2023-01-15T14:30:05.084 INFO:tasks.rados.rados.0.smithi090.stdout:486: oid 5 version 181 is already newer than 180 2023-01-15T14:30:05.084 INFO:tasks.rados.rados.0.smithi090.stdout:486: oid 5 version 181 is already newer than 179 2023-01-15T14:30:05.084 INFO:tasks.rados.rados.0.smithi090.stdout:486: oid 5 version 181 is already newer than 178 2023-01-15T14:30:05.084 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 181 (ObjNum 204 snap 44 seq_num 204) dirty exists 2023-01-15T14:30:05.084 INFO:tasks.rados.rados.0.smithi090.stdout:486: left oid 5 (ObjNum 204 snap 44 seq_num 204) 2023-01-15T14:30:05.084 INFO:tasks.rados.rados.0.smithi090.stdout:486: done (0 left) 2023-01-15T14:30:05.084 INFO:tasks.rados.rados.0.smithi090.stdout:498: done (0 left) 2023-01-15T14:30:05.085 INFO:tasks.rados.rados.0.smithi090.stdout:499: delete oid 11 current snap is 45 2023-01-15T14:30:05.085 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:05.080+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:05.145 INFO:tasks.rados.rados.0.smithi090.stdout:499: done (0 left) 2023-01-15T14:30:05.269 INFO:tasks.rados.rados.0.smithi090.stdout:500: snap_create 2023-01-15T14:30:05.269 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:30:05.271 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:05.271 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:05.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.271 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.19 118/114 [PushOp(3:9988ac9d:::smithi09025264-36:head, version: 112'154, data_included: [544593~799729,1761799~248847], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(3:9988ac9d:::smithi09025264-36:head@112'154, size: 3344593, copy_subset: [544593~799729,1761799~776065,2588245~756348], clone_subset: {3:9988ac9d:::smithi09025264-36:55=[0~544593,1344322~417477,2537864~50381]}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2010646, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-01-15T14:30:05.271 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:05.132758+0000", 2023-01-15T14:30:05.271 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.068496743999999998, 2023-01-15T14:30:05.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068592976999999999, 2023-01-15T14:30:05.272 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:05.272 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "no flag points reached", 2023-01-15T14:30:05.272 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:05.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:05.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:05.132758+0000", 2023-01-15T14:30:05.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:05.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:05.132758+0000", 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:05.132757+0000", 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:05.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:05.133346+0000", 2023-01-15T14:30:05.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058814499999999999 2023-01-15T14:30:05.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:05.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:05.133398+0000", 2023-01-15T14:30:05.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2796000000000001e-05 2023-01-15T14:30:05.274 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:05.274 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:05.274 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:05.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(3.16 118/115 [PushReplyOp(3:68fb3263:::smithi09025264-33:head)])", 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:05.190770+0000", 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.010485255000000001, 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010641281000000001, 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:05.190770+0000", 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:05.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:05.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:05.190770+0000", 2023-01-15T14:30:05.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:05.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:05.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:05.190767+0000", 2023-01-15T14:30:05.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:05.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:05.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:05.190784+0000", 2023-01-15T14:30:05.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6616000000000002e-05 2023-01-15T14:30:05.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:05.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:05.190818+0000", 2023-01-15T14:30:05.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4558999999999999e-05 2023-01-15T14:30:05.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:05.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:05.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:05.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:05.190835+0000", 2023-01-15T14:30:05.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6464e-05 2023-01-15T14:30:05.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:05.278 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:05.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:05.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:05.278 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:30:05.278 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 2 2023-01-15T14:30:05.278 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:05.279 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:30:05.279 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:05.268+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:05.372 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:05.368+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:05.477 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:05.478 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:05.478 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:05.478 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:05.478 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:05.478 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:30:05.479 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:05.476+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:05.578 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:05.576+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:07.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:30:07.328 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 6, 7] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:30:07.333 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:30:07.333 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 6, 7] 2023-01-15T14:30:07.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 6 2023-01-15T14:30:07.334 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:07.328+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:07.336 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:08.709 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:94 3.0 3:07a51d33:::smithi09025264-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:30:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 99.144259051999995, 2023-01-15T14:30:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50760159199999999, 2023-01-15T14:30:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:08.710 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 94 2023-01-15T14:30:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:30:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:30:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:08.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439538+0000", 2023-01-15T14:30:08.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:08.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439857+0000", 2023-01-15T14:30:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000319086 2023-01-15T14:30:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439891+0000", 2023-01-15T14:30:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4449999999999997e-05 2023-01-15T14:30:08.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439906+0000", 2023-01-15T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5082999999999999e-05 2023-01-15T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824189+0000", 2023-01-15T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38428238100000001 2023-01-15T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824440+0000", 2023-01-15T14:30:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025139800000000001 2023-01-15T14:30:08.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:30:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824834+0000", 2023-01-15T14:30:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039422200000000001 2023-01-15T14:30:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825347+0000", 2023-01-15T14:30:08.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051323600000000003 2023-01-15T14:30:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825364+0000", 2023-01-15T14:30:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6478e-05 2023-01-15T14:30:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825589+0000", 2023-01-15T14:30:08.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022497500000000001 2023-01-15T14:30:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:08.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:09.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826457+0000", 2023-01-15T14:30:09.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00086797600000000001 2023-01-15T14:30:09.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:09.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.846539+0000", 2023-01-15T14:30:09.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020081742 2023-01-15T14:30:09.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:09.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.946989+0000", 2023-01-15T14:30:09.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045032199999999 2023-01-15T14:30:09.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:09.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947076+0000", 2023-01-15T14:30:09.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6922999999999994e-05 2023-01-15T14:30:09.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:09.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947141+0000", 2023-01-15T14:30:09.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5357000000000005e-05 2023-01-15T14:30:09.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.038 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:09.038 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.038 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:96 3.0 3:07a51d33:::smithi09025264-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:09.038 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:30:09.038 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 99.143836938999996, 2023-01-15T14:30:09.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50769745899999996, 2023-01-15T14:30:09.038 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:09.039 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:09.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:09.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:09.039 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:09.039 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 96 2023-01-15T14:30:09.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.039 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:09.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:09.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:30:09.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:09.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:30:09.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:09.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439961+0000", 2023-01-15T14:30:09.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:09.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:09.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439966+0000", 2023-01-15T14:30:09.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0769999999999998e-06 2023-01-15T14:30:09.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:09.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439979+0000", 2023-01-15T14:30:09.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3343000000000001e-05 2023-01-15T14:30:09.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:09.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439987+0000", 2023-01-15T14:30:09.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2400000000000001e-06 2023-01-15T14:30:09.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:09.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826191+0000", 2023-01-15T14:30:09.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38620452799999999 2023-01-15T14:30:09.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:09.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826424+0000", 2023-01-15T14:30:09.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023264800000000001 2023-01-15T14:30:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947394+0000", 2023-01-15T14:30:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120970102 2023-01-15T14:30:09.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947505+0000", 2023-01-15T14:30:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001105 2023-01-15T14:30:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947659+0000", 2023-01-15T14:30:09.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015468799999999999 2023-01-15T14:30:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.499284274999994, 2023-01-15T14:30:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:30:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:09.207 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:09.208 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:09.208 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:09.208 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:09.208 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:30:09.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.208 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:09.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:09.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:09.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:09.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:09.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:30:09.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:30:09.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:09.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:30:09.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:30:09.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:09.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:30:09.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:30:09.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:30:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:30:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:09.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:30:09.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:30:09.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:09.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:30:09.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:30:09.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:09.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:30:09.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:30:09.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:09.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:30:09.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:30:09.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:09.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:30:09.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:30:09.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:09.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:30:09.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:30:09.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:09.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:30:09.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:30:09.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:09.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:30:09.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:30:09.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:09.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:30:09.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:30:09.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:09.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:30:09.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:30:09.368 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.368 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:09.368 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.368 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:09.368 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:30:09.368 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.498984477999997, 2023-01-15T14:30:09.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:30:09.368 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:09.369 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:09.369 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:09.369 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:09.369 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:09.369 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:30:09.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.569 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:09.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:09.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:30:09.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:09.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:30:09.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:09.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:30:09.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:09.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:09.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:30:09.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:30:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:30:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:30:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:30:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:30:09.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:30:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:30:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:30:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:30:09.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:09.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:30:09.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:30:09.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:09.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:30:09.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:30:09.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:09.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:30:09.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:30:09.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:09.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:30:09.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:30:09.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:09.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:30:09.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:30:09.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:09.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:30:09.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:30:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:30:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:30:09.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:30:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:30:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:30:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:30:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:30:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:30:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:30:09.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:30:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.935 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:09.936 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:30:09.936 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.498954694000005, 2023-01-15T14:30:09.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:30:09.936 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:09.936 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:09.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:09.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:09.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:09.936 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:30:09.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.936 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:09.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:30:09.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:09.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:09.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:30:09.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:30:09.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:09.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:30:09.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:30:09.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:09.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:30:09.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:30:09.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:09.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:30:09.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:30:09.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:09.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:30:09.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:30:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:30:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:30:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:30:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:30:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:30:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:30:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:30:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:30:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:30:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:30:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:182 3.4 3:2022fdfa:::smithi09025264-46:head [write 1933572~791670] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:30:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.370092624999998, 2023-01-15T14:30:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.582304865, 2023-01-15T14:30:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 182 2023-01-15T14:30:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:30:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:30:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213704+0000", 2023-01-15T14:30:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214574+0000", 2023-01-15T14:30:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087037799999999999 2023-01-15T14:30:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214608+0000", 2023-01-15T14:30:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:30:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214627+0000", 2023-01-15T14:30:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:30:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491537+0000", 2023-01-15T14:30:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27691020300000002 2023-01-15T14:30:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491721+0000", 2023-01-15T14:30:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345500000000001 2023-01-15T14:30:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693760+0000", 2023-01-15T14:30:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202039252 2023-01-15T14:30:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693898+0000", 2023-01-15T14:30:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013789 2023-01-15T14:30:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694230+0000", 2023-01-15T14:30:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220899999999998 2023-01-15T14:30:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694866+0000", 2023-01-15T14:30:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063640599999999995 2023-01-15T14:30:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694887+0000", 2023-01-15T14:30:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0622999999999999e-05 2023-01-15T14:30:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695950+0000", 2023-01-15T14:30:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001062688 2023-01-15T14:30:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696014+0000", 2023-01-15T14:30:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4697999999999996e-05 2023-01-15T14:30:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795898+0000", 2023-01-15T14:30:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099883338000000002 2023-01-15T14:30:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795922+0000", 2023-01-15T14:30:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4326000000000001e-05 2023-01-15T14:30:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795955+0000", 2023-01-15T14:30:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3191000000000001e-05 2023-01-15T14:30:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796011+0000", 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5631999999999999e-05 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.369145893999999, 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:30:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:30:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:30:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:30:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:30:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:30:09.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:30:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:30:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:30:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:30:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:30:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:30:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:30:10.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:30:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:30:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:30:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:30:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:30:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:30:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:30:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:30:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:30:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:30:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:30:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:30:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:30:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:30:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:30:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:30:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:30:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:30:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:30:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:30:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:30:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:30:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:30:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.369089196999994, 2023-01-15T14:30:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:30:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:30:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:30:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:30:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:30:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:30:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:30:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:30:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:30:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:30:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:30:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:30:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:30:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:30:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:30:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:30:10.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:30:10.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:10.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:30:10.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:30:10.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:30:10.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.130 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:10.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.131 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:30:10.131 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:30:10.131 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.752593024000006, 2023-01-15T14:30:10.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:30:10.131 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:10.131 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:10.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:10.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:10.132 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:10.132 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:30:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:30:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:30:10.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:30:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:10.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:30:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:30:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:30:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:30:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:30:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:30:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:10.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:30:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:30:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:30:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:30:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:30:10.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:30:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:30:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:30:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 88.723744553000003, 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:10.267 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:30:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:30:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:30:10.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:30:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:30:10.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:30:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:30:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:30:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:30:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:30:10.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:30:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:30:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:30:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:30:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:30:10.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:30:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:30:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:10.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:30:10.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:30:10.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:10.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:30:10.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:30:10.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:10.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:30:10.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:30:10.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.434 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:10.434 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.435 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:373 3.4 3:2029fa71:::smithi09025264-39:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:10.435 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:30:10.435 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.505497345999999, 2023-01-15T14:30:10.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58298356399999995, 2023-01-15T14:30:10.435 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:10.435 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:10.435 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:10.435 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:10.435 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:10.435 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-01-15T14:30:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:30:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:30:10.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078299+0000", 2023-01-15T14:30:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078644+0000", 2023-01-15T14:30:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034494099999999998 2023-01-15T14:30:10.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078676+0000", 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1183000000000003e-05 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078694+0000", 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-05 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078748+0000", 2023-01-15T14:30:10.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3634999999999997e-05 2023-01-15T14:30:10.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:10.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078753+0000", 2023-01-15T14:30:10.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6370000000000004e-06 2023-01-15T14:30:10.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110841+0000", 2023-01-15T14:30:10.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032087439000000002 2023-01-15T14:30:10.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:10.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110967+0000", 2023-01-15T14:30:10.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126029 2023-01-15T14:30:10.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.457874+0000", 2023-01-15T14:30:10.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.346907725 2023-01-15T14:30:10.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:10.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458066+0000", 2023-01-15T14:30:10.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000191592 2023-01-15T14:30:10.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:10.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458321+0000", 2023-01-15T14:30:10.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025454200000000002 2023-01-15T14:30:10.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:10.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458918+0000", 2023-01-15T14:30:10.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059740800000000001 2023-01-15T14:30:10.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:10.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458945+0000", 2023-01-15T14:30:10.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-05 2023-01-15T14:30:10.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:10.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459253+0000", 2023-01-15T14:30:10.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030836200000000001 2023-01-15T14:30:10.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:10.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460943+0000", 2023-01-15T14:30:10.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016903580000000001 2023-01-15T14:30:10.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:10.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461340+0000", 2023-01-15T14:30:10.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039665799999999998 2023-01-15T14:30:10.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:10.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661129+0000", 2023-01-15T14:30:10.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19978933500000001 2023-01-15T14:30:10.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:10.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661188+0000", 2023-01-15T14:30:10.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8687000000000002e-05 2023-01-15T14:30:10.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:10.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661285+0000", 2023-01-15T14:30:10.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779000000000003e-05 2023-01-15T14:30:10.600 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.600 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:10.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.601 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:374 3.4 3:2029fa71:::smithi09025264-39:head [write 3101234~41642] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:10.601 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:30:10.601 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.500835433999995, 2023-01-15T14:30:10.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57848911800000002, 2023-01-15T14:30:10.601 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:10.601 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:10.601 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:10.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:10.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:10.602 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 374 2023-01-15T14:30:10.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.602 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:10.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:10.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:30:10.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:10.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:30:10.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:10.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082961+0000", 2023-01-15T14:30:10.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:10.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:10.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083019+0000", 2023-01-15T14:30:10.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999997e-05 2023-01-15T14:30:10.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:10.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083051+0000", 2023-01-15T14:30:10.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1813999999999997e-05 2023-01-15T14:30:10.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:10.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083069+0000", 2023-01-15T14:30:10.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:30:10.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083143+0000", 2023-01-15T14:30:10.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4363000000000006e-05 2023-01-15T14:30:10.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:10.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083147+0000", 2023-01-15T14:30:10.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4529999999999998e-06 2023-01-15T14:30:10.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111017+0000", 2023-01-15T14:30:10.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027870564 2023-01-15T14:30:10.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:10.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111099+0000", 2023-01-15T14:30:10.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1277000000000001e-05 2023-01-15T14:30:10.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459156+0000", 2023-01-15T14:30:10.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34805700499999997 2023-01-15T14:30:10.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:10.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459393+0000", 2023-01-15T14:30:10.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023757499999999999 2023-01-15T14:30:10.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:10.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459778+0000", 2023-01-15T14:30:10.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000385335 2023-01-15T14:30:10.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:10.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460020+0000", 2023-01-15T14:30:10.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024141899999999999 2023-01-15T14:30:10.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:10.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460037+0000", 2023-01-15T14:30:10.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6665999999999999e-05 2023-01-15T14:30:10.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:10.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460220+0000", 2023-01-15T14:30:10.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018367599999999999 2023-01-15T14:30:10.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:10.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461197+0000", 2023-01-15T14:30:10.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00097708600000000006 2023-01-15T14:30:10.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:10.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461720+0000", 2023-01-15T14:30:10.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052235400000000005 2023-01-15T14:30:10.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:10.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661345+0000", 2023-01-15T14:30:10.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199624884 2023-01-15T14:30:10.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:10.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661405+0000", 2023-01-15T14:30:10.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0965999999999999e-05 2023-01-15T14:30:10.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:10.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661452+0000", 2023-01-15T14:30:10.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6864999999999999e-05 2023-01-15T14:30:10.613 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.613 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:10.613 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.613 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:375 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (56) in=7b,truncate 3142876] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:10.613 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:30:10.613 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.500693667999997, 2023-01-15T14:30:10.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57851114599999998, 2023-01-15T14:30:10.614 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:10.614 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:10.614 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:10.614 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:10.614 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:10.614 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 375 2023-01-15T14:30:10.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.614 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:10.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:10.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:30:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:30:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083104+0000", 2023-01-15T14:30:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083113+0000", 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6500000000000008e-06 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083130+0000", 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083142+0000", 2023-01-15T14:30:10.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2252999999999999e-05 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083173+0000", 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0474999999999999e-05 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083175+0000", 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183469+0000", 2023-01-15T14:30:10.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10029313099999999 2023-01-15T14:30:10.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:10.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183747+0000", 2023-01-15T14:30:10.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000277998 2023-01-15T14:30:10.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:10.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460165+0000", 2023-01-15T14:30:10.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27641872899999997 2023-01-15T14:30:10.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:10.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460342+0000", 2023-01-15T14:30:10.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176792 2023-01-15T14:30:10.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460579+0000", 2023-01-15T14:30:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000237266 2023-01-15T14:30:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460833+0000", 2023-01-15T14:30:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025390900000000002 2023-01-15T14:30:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:10.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460853+0000", 2023-01-15T14:30:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9273e-05 2023-01-15T14:30:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461024+0000", 2023-01-15T14:30:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017111900000000001 2023-01-15T14:30:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461230+0000", 2023-01-15T14:30:10.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020672600000000001 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461879+0000", 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064846999999999995 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661524+0000", 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19964509599999999 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661560+0000", 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6264999999999999e-05 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661616+0000", 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5952000000000002e-05 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:10.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:10.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:10.811 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:376 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:10.811 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:30:10.811 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.500611411999998, 2023-01-15T14:30:10.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57864278800000002, 2023-01-15T14:30:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 376 2023-01-15T14:30:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:30:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:30:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083186+0000", 2023-01-15T14:30:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083193+0000", 2023-01-15T14:30:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2023-01-15T14:30:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083211+0000", 2023-01-15T14:30:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7609999999999999e-05 2023-01-15T14:30:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083223+0000", 2023-01-15T14:30:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140000000000001e-05 2023-01-15T14:30:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093534+0000", 2023-01-15T14:30:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010311529 2023-01-15T14:30:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093551+0000", 2023-01-15T14:30:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.666e-05 2023-01-15T14:30:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183834+0000", 2023-01-15T14:30:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090283063999999996 2023-01-15T14:30:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.184076+0000", 2023-01-15T14:30:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024241900000000001 2023-01-15T14:30:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461001+0000", 2023-01-15T14:30:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27692504299999998 2023-01-15T14:30:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461152+0000", 2023-01-15T14:30:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150345 2023-01-15T14:30:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661642+0000", 2023-01-15T14:30:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200489742 2023-01-15T14:30:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661720+0000", 2023-01-15T14:30:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8604999999999997e-05 2023-01-15T14:30:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661830+0000", 2023-01-15T14:30:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010992099999999999 2023-01-15T14:30:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:30:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.722457577, 2023-01-15T14:30:11.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:30:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:30:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:11.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:11.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:11.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:11.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:11.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:11.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:11.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:11.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:30:11.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:30:11.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:11.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:30:11.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:30:11.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:30:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:30:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:30:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:30:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:30:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:30:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:30:11.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:30:11.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:11.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:30:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:30:11.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:30:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:30:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:11.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:30:11.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:30:11.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:11.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:30:11.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:30:11.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:11.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:30:11.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:30:11.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:30:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:30:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:961 3.0 3:0a998d04:::smithi09025264-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2023-01-15T14:30:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.520688989, 2023-01-15T14:30:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51361718300000003, 2023-01-15T14:30:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 961 2023-01-15T14:30:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063109+0000", 2023-01-15T14:30:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063115+0000", 2023-01-15T14:30:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8300000000000001e-06 2023-01-15T14:30:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:11.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063127+0000", 2023-01-15T14:30:11.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2517000000000001e-05 2023-01-15T14:30:11.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:11.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063137+0000", 2023-01-15T14:30:11.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8509999999999994e-06 2023-01-15T14:30:11.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063272+0000", 2023-01-15T14:30:11.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013513500000000001 2023-01-15T14:30:11.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:11.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063277+0000", 2023-01-15T14:30:11.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2750000000000003e-06 2023-01-15T14:30:11.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.265103+0000", 2023-01-15T14:30:11.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201825688 2023-01-15T14:30:11.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:11.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.265208+0000", 2023-01-15T14:30:11.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104693 2023-01-15T14:30:11.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.375398+0000", 2023-01-15T14:30:11.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.110190553 2023-01-15T14:30:11.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:11.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.375548+0000", 2023-01-15T14:30:11.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149417 2023-01-15T14:30:11.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576187+0000", 2023-01-15T14:30:11.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20063947500000001 2023-01-15T14:30:11.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:11.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576454+0000", 2023-01-15T14:30:11.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026708400000000001 2023-01-15T14:30:11.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:11.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576727+0000", 2023-01-15T14:30:11.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027257699999999998 2023-01-15T14:30:11.238 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.238 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:11.238 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.239 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:30:11.239 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:11.239 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.661879326999999, 2023-01-15T14:30:11.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:30:11.240 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:11.240 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:11.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:11.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:11.240 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:11.241 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:30:11.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.241 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:11.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:11.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:11.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:11.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:11.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:11.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:30:11.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:11.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:11.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:30:11.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:30:11.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:11.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:30:11.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:30:11.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:11.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:30:11.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:30:11.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:30:11.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:30:11.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:30:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:30:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:30:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:30:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:30:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:11.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:30:11.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:30:11.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:11.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:30:11.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:30:11.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:11.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:30:11.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:30:11.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:11.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:30:11.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:30:11.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:11.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:30:11.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:30:11.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:11.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:30:11.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:30:11.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:11.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:30:11.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:30:11.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:11.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:30:11.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:30:11.382 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.382 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:11.382 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.383 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:30:11.383 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:11.383 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.661820617, 2023-01-15T14:30:11.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:30:11.384 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:11.384 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:11.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:11.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:11.385 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:11.386 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:30:11.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.387 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:11.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:11.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:11.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:11.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:11.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:11.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:30:11.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:11.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:11.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:30:11.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:30:11.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:11.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:30:11.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:30:11.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:11.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:30:11.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:30:11.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:30:11.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:30:11.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:11.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:30:11.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:30:11.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:30:11.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:30:11.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:11.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:30:11.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:30:11.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:30:11.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:30:11.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:11.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:30:11.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:30:11.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:11.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:30:11.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:30:11.393 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.393 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:11.393 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.393 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:11.393 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:11.393 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.7054011239999998, 2023-01-15T14:30:11.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:30:11.393 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:11.394 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:11.394 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:11.394 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:11.394 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:11.394 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:30:11.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.394 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:11.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:11.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:11.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:11.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:30:11.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:30:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:30:11.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:11.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:30:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:30:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:30:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:30:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:30:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:30:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:30:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:30:11.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:30:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:30:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:30:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:30:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:30:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:30:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:11.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.6479034410000004, 2023-01-15T14:30:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:30:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:11.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:30:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:11.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:11.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:30:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:30:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:30:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:30:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:30:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:30:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:30:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:30:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:30:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:30:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:30:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:30:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:11.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:30:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:30:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:30:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:30:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:30:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:30:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:30:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:30:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:30:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:30:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:30:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:30:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:30:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:11.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:30:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:30:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:30:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:30:11.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:30:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:30:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:30:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:30:11.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:11.562 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:11.562 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:11.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:30:11.567 INFO:tasks.rados.rados.0.smithi090.stdout:500: done (0 left) 2023-01-15T14:30:11.567 INFO:tasks.rados.rados.0.smithi090.stdout:501: snap_create 2023-01-15T14:30:11.569 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:30:11.576 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:11.568+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:11.704 INFO:tasks.rados.rados.0.smithi090.stdout:501: done (0 left) 2023-01-15T14:30:11.705 INFO:tasks.rados.rados.0.smithi090.stdout:502: write (excl) oid 16 current snap is 47 2023-01-15T14:30:11.705 INFO:tasks.rados.rados.0.smithi090.stdout:502: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2023-01-15T14:30:11.705 INFO:tasks.rados.rados.0.smithi090.stdout:502: writing smithi09025264-16 from 712663 to 1181235 tid 1 2023-01-15T14:30:11.705 INFO:tasks.rados.rados.0.smithi090.stdout:502: writing smithi09025264-16 from 1713913 to 2391970 tid 2 2023-01-15T14:30:11.705 INFO:tasks.rados.rados.0.smithi090.stdout:502: writing smithi09025264-16 from 2873523 to 3465256 tid 3 2023-01-15T14:30:11.705 INFO:tasks.rados.rados.0.smithi090.stdout:503: write oid 40 current snap is 47 2023-01-15T14:30:11.705 INFO:tasks.rados.rados.0.smithi090.stdout:503: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2023-01-15T14:30:11.705 INFO:tasks.rados.rados.0.smithi090.stdout:503: writing smithi09025264-40 from 729470 to 1273291 tid 1 2023-01-15T14:30:11.705 INFO:tasks.rados.rados.0.smithi090.stdout:503: writing smithi09025264-40 from 1772395 to 2310463 tid 2 2023-01-15T14:30:11.705 INFO:tasks.rados.rados.0.smithi090.stdout:503: writing smithi09025264-40 from 2900946 to 3303951 tid 3 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:504: snap_remove snap 46 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:504: done (2 left) 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:505: snap_remove snap 44 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:505: done (2 left) 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:506: copy_from oid 35 from oid 15 current snap is 47 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:507: write oid 49 current snap is 47 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:507: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:507: writing smithi09025264-49 from 746277 to 1365347 tid 1 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:507: writing smithi09025264-49 from 1830877 to 2512603 tid 2 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:507: writing smithi09025264-49 from 2928369 to 3546277 tid 3 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:508: write (excl) oid 23 current snap is 47 2023-01-15T14:30:11.706 INFO:tasks.rados.rados.0.smithi090.stdout:508: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:508: writing smithi09025264-23 from 763084 to 1457403 tid 1 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:508: writing smithi09025264-23 from 1889359 to 2431096 tid 2 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:508: writing smithi09025264-23 from 2955792 to 3497694 tid 3 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:509: delete oid 38 current snap is 47 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:503: finishing write tid 1 to smithi09025264-40 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:503: finishing write tid 2 to smithi09025264-40 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:503: finishing write tid 3 to smithi09025264-40 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:503: finishing write tid 4 to smithi09025264-40 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:503: finishing write tid 5 to smithi09025264-40 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:503: oid 40 updating version 0 to 133 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:503: oid 40 updating version 133 to 134 2023-01-15T14:30:11.707 INFO:tasks.rados.rados.0.smithi090.stdout:503: oid 40 version 134 is already newer than 132 2023-01-15T14:30:11.708 INFO:tasks.rados.rados.0.smithi090.stdout:503: oid 40 updating version 134 to 135 2023-01-15T14:30:11.708 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 135 (ObjNum 210 snap 47 seq_num 210) dirty exists 2023-01-15T14:30:11.708 INFO:tasks.rados.rados.0.smithi090.stdout:503: left oid 40 (ObjNum 210 snap 47 seq_num 210) 2023-01-15T14:30:11.708 INFO:tasks.rados.rados.0.smithi090.stdout:502: finishing write tid 1 to smithi09025264-16 2023-01-15T14:30:11.708 INFO:tasks.rados.rados.0.smithi090.stdout:502: finishing write tid 2 to smithi09025264-16 2023-01-15T14:30:11.708 INFO:tasks.rados.rados.0.smithi090.stdout:502: finishing write tid 3 to smithi09025264-16 2023-01-15T14:30:11.708 INFO:tasks.rados.rados.0.smithi090.stdout:502: finishing write tid 4 to smithi09025264-16 2023-01-15T14:30:11.708 INFO:tasks.rados.rados.0.smithi090.stdout:502: finishing write tid 5 to smithi09025264-16 2023-01-15T14:30:11.710 INFO:tasks.rados.rados.0.smithi090.stdout:502: oid 16 updating version 0 to 130 2023-01-15T14:30:11.710 INFO:tasks.rados.rados.0.smithi090.stdout:502: oid 16 updating version 130 to 131 2023-01-15T14:30:11.710 INFO:tasks.rados.rados.0.smithi090.stdout:502: oid 16 version 131 is already newer than 129 2023-01-15T14:30:11.710 INFO:tasks.rados.rados.0.smithi090.stdout:502: oid 16 version 131 is already newer than 128 2023-01-15T14:30:11.710 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 131 (ObjNum 209 snap 47 seq_num 209) dirty exists 2023-01-15T14:30:11.710 INFO:tasks.rados.rados.0.smithi090.stdout:502: left oid 16 (ObjNum 209 snap 47 seq_num 209) 2023-01-15T14:30:11.710 INFO:tasks.rados.rados.0.smithi090.stdout:506: finishing copy_from to smithi09025264-35 2023-01-15T14:30:11.711 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 59 (ObjNum 128 snap 23 seq_num 128) dirty exists 2023-01-15T14:30:11.711 INFO:tasks.rados.rados.0.smithi090.stdout:506: finishing copy_from racing read to smithi09025264-35 2023-01-15T14:30:11.711 INFO:tasks.rados.rados.0.smithi090.stdout:507: finishing write tid 1 to smithi09025264-49 2023-01-15T14:30:11.711 INFO:tasks.rados.rados.0.smithi090.stdout:507: finishing write tid 2 to smithi09025264-49 2023-01-15T14:30:11.711 INFO:tasks.rados.rados.0.smithi090.stdout:507: finishing write tid 3 to smithi09025264-49 2023-01-15T14:30:11.711 INFO:tasks.rados.rados.0.smithi090.stdout:507: finishing write tid 4 to smithi09025264-49 2023-01-15T14:30:11.711 INFO:tasks.rados.rados.0.smithi090.stdout:507: finishing write tid 5 to smithi09025264-49 2023-01-15T14:30:11.712 INFO:tasks.rados.rados.0.smithi090.stdout:507: oid 49 updating version 0 to 189 2023-01-15T14:30:11.712 INFO:tasks.rados.rados.0.smithi090.stdout:507: oid 49 updating version 189 to 191 2023-01-15T14:30:11.712 INFO:tasks.rados.rados.0.smithi090.stdout:507: oid 49 version 191 is already newer than 188 2023-01-15T14:30:11.712 INFO:tasks.rados.rados.0.smithi090.stdout:507: oid 49 version 191 is already newer than 190 2023-01-15T14:30:11.712 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 191 (ObjNum 211 snap 47 seq_num 211) dirty exists 2023-01-15T14:30:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:507: left oid 49 (ObjNum 211 snap 47 seq_num 211) 2023-01-15T14:30:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:508: finishing write tid 1 to smithi09025264-23 2023-01-15T14:30:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:502: done (5 left) 2023-01-15T14:30:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:503: done (4 left) 2023-01-15T14:30:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:506: done (3 left) 2023-01-15T14:30:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:507: done (2 left) 2023-01-15T14:30:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:509: done (1 left) 2023-01-15T14:30:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:510: delete oid 10 current snap is 47 2023-01-15T14:30:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:510: done (1 left) 2023-01-15T14:30:11.713 INFO:tasks.rados.rados.0.smithi090.stdout:511: copy_from oid 48 from oid 29 current snap is 47 2023-01-15T14:30:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:512: snap_create 2023-01-15T14:30:11.714 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:508: finishing write tid 2 to smithi09025264-23 2023-01-15T14:30:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:508: finishing write tid 3 to smithi09025264-23 2023-01-15T14:30:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:508: finishing write tid 4 to smithi09025264-23 2023-01-15T14:30:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:508: finishing write tid 5 to smithi09025264-23 2023-01-15T14:30:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:508: oid 23 updating version 0 to 129 2023-01-15T14:30:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:508: oid 23 updating version 129 to 132 2023-01-15T14:30:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:508: oid 23 updating version 132 to 133 2023-01-15T14:30:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:508: oid 23 updating version 133 to 134 2023-01-15T14:30:11.714 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 134 (ObjNum 212 snap 47 seq_num 212) dirty exists 2023-01-15T14:30:11.715 INFO:tasks.rados.rados.0.smithi090.stdout:508: left oid 23 (ObjNum 212 snap 47 seq_num 212) 2023-01-15T14:30:11.715 INFO:tasks.rados.rados.0.smithi090.stdout:508: done (1 left) 2023-01-15T14:30:11.715 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:11.715 INFO:tasks.rados.rados.0.smithi090.stdout:511: finishing copy_from to smithi09025264-48 2023-01-15T14:30:11.715 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 127 (ObjNum 132 snap 24 seq_num 132) dirty exists 2023-01-15T14:30:11.715 INFO:tasks.rados.rados.0.smithi090.stdout:511: finishing copy_from racing read to smithi09025264-48 2023-01-15T14:30:11.715 INFO:tasks.rados.rados.0.smithi090.stdout:511: done (0 left) 2023-01-15T14:30:11.715 INFO:tasks.rados.rados.0.smithi090.stdout:512: done (0 left) 2023-01-15T14:30:11.715 INFO:tasks.rados.rados.0.smithi090.stdout:513: write (excl) oid 26 current snap is 48 2023-01-15T14:30:11.715 INFO:tasks.rados.rados.0.smithi090.stdout:513: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2023-01-15T14:30:11.715 INFO:tasks.rados.rados.0.smithi090.stdout:513: writing smithi09025264-26 from 413505 to 1049924 tid 1 2023-01-15T14:30:11.716 INFO:tasks.rados.rados.0.smithi090.stdout:513: writing smithi09025264-26 from 1664805 to 2353869 tid 2 2023-01-15T14:30:11.716 INFO:tasks.rados.rados.0.smithi090.stdout:513: writing smithi09025264-26 from 3038061 to 3530132 tid 3 2023-01-15T14:30:11.716 INFO:tasks.rados.rados.0.smithi090.stdout:514: read oid 36 snap -1 2023-01-15T14:30:11.716 INFO:tasks.rados.rados.0.smithi090.stdout:514: expect (ObjNum 199 snap 43 seq_num 199) 2023-01-15T14:30:11.716 INFO:tasks.rados.rados.0.smithi090.stdout:515: read oid 8 snap -1 2023-01-15T14:30:11.716 INFO:tasks.rados.rados.0.smithi090.stdout:515: expect (ObjNum 161 snap 35 seq_num 161) 2023-01-15T14:30:11.716 INFO:tasks.rados.rados.0.smithi090.stdout:516: snap_remove snap 45 2023-01-15T14:30:11.716 INFO:tasks.rados.rados.0.smithi090.stdout:513: finishing write tid 1 to smithi09025264-26 2023-01-15T14:30:11.716 INFO:tasks.rados.rados.0.smithi090.stdout:514: expect (ObjNum 199 snap 43 seq_num 199) 2023-01-15T14:30:11.716 INFO:tasks.rados.rados.0.smithi090.stdout:513: finishing write tid 2 to smithi09025264-26 2023-01-15T14:30:11.716 INFO:tasks.rados.rados.0.smithi090.stdout:513: finishing write tid 3 to smithi09025264-26 2023-01-15T14:30:11.717 INFO:tasks.rados.rados.0.smithi090.stdout:515: expect (ObjNum 161 snap 35 seq_num 161) 2023-01-15T14:30:11.717 INFO:tasks.rados.rados.0.smithi090.stdout:513: finishing write tid 4 to smithi09025264-26 2023-01-15T14:30:11.717 INFO:tasks.rados.rados.0.smithi090.stdout:513: finishing write tid 5 to smithi09025264-26 2023-01-15T14:30:11.717 INFO:tasks.rados.rados.0.smithi090.stdout:513: oid 26 updating version 0 to 209 2023-01-15T14:30:11.717 INFO:tasks.rados.rados.0.smithi090.stdout:513: oid 26 updating version 209 to 211 2023-01-15T14:30:11.717 INFO:tasks.rados.rados.0.smithi090.stdout:513: oid 26 updating version 211 to 212 2023-01-15T14:30:11.717 INFO:tasks.rados.rados.0.smithi090.stdout:513: oid 26 version 212 is already newer than 210 2023-01-15T14:30:11.717 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 212 (ObjNum 215 snap 48 seq_num 215) dirty exists 2023-01-15T14:30:11.717 INFO:tasks.rados.rados.0.smithi090.stdout:513: left oid 26 (ObjNum 215 snap 48 seq_num 215) 2023-01-15T14:30:11.717 INFO:tasks.rados.rados.0.smithi090.stdout:513: done (3 left) 2023-01-15T14:30:11.717 INFO:tasks.rados.rados.0.smithi090.stdout:514: done (2 left) 2023-01-15T14:30:11.718 INFO:tasks.rados.rados.0.smithi090.stdout:515: done (1 left) 2023-01-15T14:30:11.718 INFO:tasks.rados.rados.0.smithi090.stdout:516: done (0 left) 2023-01-15T14:30:11.718 INFO:tasks.rados.rados.0.smithi090.stdout:517: read oid 7 snap 47 2023-01-15T14:30:11.718 INFO:tasks.rados.rados.0.smithi090.stdout:517: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:30:11.718 INFO:tasks.rados.rados.0.smithi090.stdout:518: snap_remove snap 43 2023-01-15T14:30:11.732 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:11.852 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:11.728+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:11.853 INFO:tasks.rados.rados.0.smithi090.stdout:518: done (1 left) 2023-01-15T14:30:11.853 INFO:tasks.rados.rados.0.smithi090.stdout:519: read oid 20 snap 47 2023-01-15T14:30:11.853 INFO:tasks.rados.rados.0.smithi090.stdout:519: expect deleted 2023-01-15T14:30:11.853 INFO:tasks.rados.rados.0.smithi090.stdout:520: snap_create 2023-01-15T14:30:11.853 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:11.853 INFO:tasks.rados.rados.0.smithi090.stdout:517: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:30:11.854 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:11.854 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:11.854 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:11.854 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:11.855 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:11.855 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:11.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:30:11.857 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:11.857 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:11.852+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:11.870 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:11.879 INFO:tasks.rados.rados.0.smithi090.stdout:517: done (1 left) 2023-01-15T14:30:11.879 INFO:tasks.rados.rados.0.smithi090.stdout:519: done (0 left) 2023-01-15T14:30:11.888 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:30:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:30:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:30:12.010 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:30:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:30:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:30:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:30:12.011 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:30:12.011 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-01-15T14:30:11.924+0000 7fb90e2b4700 -1 mon.a@0(leader).osd e125 definitely_dead 0 2023-01-15T14:30:12.012 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:12.012 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:12.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:30:12.013 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:12.008+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:12.111 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:12.108+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:12.179 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:12.179 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:12.179 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:30:12.226 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:12.224+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:12.280 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:12.280 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:12.289 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:30:12.313 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:12.312+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:12.387 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:12.387 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:12.396 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:30:12.414 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:12.412+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:12.496 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:12.496 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:12.506 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:30:12.553 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:12.552+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:12.615 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:12.612+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:12.622 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:12.622 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:12.633 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:30:12.716 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:12.712+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:12.749 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:12.750 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:12.759 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:30:12.798 INFO:tasks.rados.rados.0.smithi090.stdout:520: done (0 left) 2023-01-15T14:30:12.798 INFO:tasks.rados.rados.0.smithi090.stdout:521: write oid 13 current snap is 49 2023-01-15T14:30:12.798 INFO:tasks.rados.rados.0.smithi090.stdout:521: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2023-01-15T14:30:12.803 INFO:teuthology.orchestra.run.smithi090.stderr:marked out osd.6. 2023-01-15T14:30:12.812 INFO:tasks.rados.rados.0.smithi090.stdout:521: writing smithi09025264-13 from 430312 to 1141980 tid 1 2023-01-15T14:30:12.812 INFO:tasks.rados.rados.0.smithi090.stdout:521: writing smithi09025264-13 from 1723287 to 2272362 tid 2 2023-01-15T14:30:12.818 INFO:tasks.rados.rados.0.smithi090.stdout:521: writing smithi09025264-13 from 2781837 to 3485180 tid 3 2023-01-15T14:30:12.818 INFO:tasks.rados.rados.0.smithi090.stdout:522: snap_remove snap 47 2023-01-15T14:30:12.818 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:12.816+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:12.868 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:12.868 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:12.877 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:30:12.918 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:12.916+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:12.977 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:12.977 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:12.987 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:30:13.018 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:13.016+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:13.089 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:13.089 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:13.099 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:30:13.120 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:13.116+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:13.220 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:13.220+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:13.249 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:13.249 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:13.258 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:30:13.321 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:13.320+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:13.364 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:13.364 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:13.374 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:30:13.421 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:13.420+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:13.523 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:13.520+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:13.524 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:13.524 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:13.534 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:30:13.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:30:13.624 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:13.620+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:13.642 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:13.642 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:13.652 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:30:13.724 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:13.720+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:13.759 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:13.759 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:13.769 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:30:13.807 INFO:tasks.rados.rados.0.smithi090.stdout:522: done (1 left) 2023-01-15T14:30:13.807 INFO:tasks.rados.rados.0.smithi090.stdout:523: read oid 35 snap -1 2023-01-15T14:30:13.807 INFO:tasks.rados.rados.0.smithi090.stdout:523: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:30:13.807 INFO:tasks.rados.rados.0.smithi090.stdout:524: snap_remove snap 48 2023-01-15T14:30:13.825 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:13.820+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:13.876 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:13.876 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:13.885 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:30:13.926 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:13.924+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:13.989 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:14.023 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:14.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:30:14.025 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:30:14.027 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:14.024+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:14.127 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:14.124+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:14.137 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:14.138 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:14.152 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:30:14.168 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:14.190 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:14.207 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:14.228 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:14.224+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:14.246 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:14.258 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:14.258 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:14.258 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:14.258 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:14.258 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:14.264 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:14.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:30:14.291 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:14.310 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:14.329 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:14.324+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:14.334 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:14.336 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:30:14.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:30:14.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:30:14.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:30:14.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:30:14.337 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:30:14.338 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:30:14.338 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:30:14.375 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:14.375 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:14.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:14.375 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:14.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.375 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.376 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:30:14.376 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.307540278, 2023-01-15T14:30:14.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:30:14.376 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.376 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.376 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.376 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.376 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.376 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:30:14.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.377 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:30:14.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:30:14.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:30:14.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:30:14.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:30:14.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:30:14.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:30:14.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:30:14.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:30:14.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:30:14.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:30:14.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:30:14.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:30:14.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:30:14.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:30:14.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:30:14.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:30:14.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:30:14.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:30:14.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:30:14.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:30:14.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:30:14.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:30:14.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:30:14.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:30:14.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:30:14.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:30:14.386 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.386 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.387 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.387 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.387 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:30:14.387 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.30726002999999, 2023-01-15T14:30:14.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:30:14.387 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.387 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.387 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.388 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.388 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.388 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:30:14.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.388 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:30:14.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:30:14.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:30:14.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:30:14.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:30:14.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:30:14.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:30:14.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:30:14.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:30:14.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:30:14.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:30:14.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:30:14.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:30:14.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:30:14.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:30:14.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:30:14.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:30:14.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:14.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:30:14.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:30:14.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:30:14.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:30:14.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:30:14.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:30:14.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:30:14.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:30:14.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:30:14.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:30:14.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:30:14.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.398 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.30722741300001, 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:30:14.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:14.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:30:14.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:30:14.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:30:14.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:30:14.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:30:14.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:30:14.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:30:14.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:30:14.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:30:14.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:30:14.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:30:14.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:30:14.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:30:14.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:30:14.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:30:14.406 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.406 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.406 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.406 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.406 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:30:14.406 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.20590942699999, 2023-01-15T14:30:14.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:30:14.407 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.407 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.407 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.407 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:30:14.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.407 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:30:14.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:30:14.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:30:14.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:30:14.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:30:14.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:30:14.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:30:14.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:30:14.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:30:14.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:30:14.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:30:14.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:30:14.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.205879424, 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.416 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:14.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.417 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:30:14.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.417 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:30:14.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:30:14.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:30:14.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:30:14.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:30:14.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:30:14.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:30:14.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.690870859, 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:30:14.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.424 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:30:14.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:30:14.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:30:14.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:30:14.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:30:14.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:30:14.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:30:14.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:30:14.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:30:14.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:30:14.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:30:14.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:30:14.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:30:14.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:30:14.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:14.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:30:14.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:30:14.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:30:14.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:30:14.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:30:14.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.432 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.433 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.433 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.433 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:30:14.433 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.690388505, 2023-01-15T14:30:14.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:30:14.433 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.433 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:30:14.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:30:14.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:14.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:30:14.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:30:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:30:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:30:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:30:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:30:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:30:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:30:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:30:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:30:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:30:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:30:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:30:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:30:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:30:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:30:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.690335942, 2023-01-15T14:30:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:30:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:30:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:30:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:30:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:30:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:30:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:30:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:30:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:30:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:30:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:30:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:30:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:30:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:30:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:30:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:30:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:30:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:30:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:30:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:30:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:30:14.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:30:14.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:30:14.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:30:14.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:30:14.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.451 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.451 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:14.451 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:30:14.451 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.319134238000004, 2023-01-15T14:30:14.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:30:14.452 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.452 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.452 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.452 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.452 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.452 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:30:14.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.452 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:30:14.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:30:14.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:30:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:30:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:30:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:30:14.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:30:14.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:30:14.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:30:14.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:30:14.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:30:14.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:30:14.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:30:14.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:30:14.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:30:14.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:30:14.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:30:14.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:30:14.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:30:14.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:30:14.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:30:14.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:30:14.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:30:14.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:30:14.460 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.460 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.461 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.461 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:14.461 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:30:14.461 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.319017564000006, 2023-01-15T14:30:14.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:30:14.461 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.461 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:14.461 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.462 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.462 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.462 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:30:14.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.462 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:30:14.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:30:14.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:30:14.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:30:14.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:30:14.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:30:14.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:30:14.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:14.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:30:14.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:30:14.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:30:14.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:30:14.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:30:14.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:30:14.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:30:14.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:30:14.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:30:14.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:30:14.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:30:14.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:30:14.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.468 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.468 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:448 3.1 3:8b0a2647:::smithi09025264-8:head [write 1856548~737690] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e41)", 2023-01-15T14:30:14.469 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:30:14.469 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.071759655999998, 2023-01-15T14:30:14.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40560468799999999, 2023-01-15T14:30:14.469 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.469 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.469 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 448 2023-01-15T14:30:14.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.470 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:30:14.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297045+0000", 2023-01-15T14:30:14.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297043+0000", 2023-01-15T14:30:14.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:14.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297477+0000", 2023-01-15T14:30:14.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043336500000000001 2023-01-15T14:30:14.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.297505+0000", 2023-01-15T14:30:14.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8583000000000001e-05 2023-01-15T14:30:14.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397598+0000", 2023-01-15T14:30:14.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092164 2023-01-15T14:30:14.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397683+0000", 2023-01-15T14:30:14.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5986e-05 2023-01-15T14:30:14.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.397798+0000", 2023-01-15T14:30:14.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114916 2023-01-15T14:30:14.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498684+0000", 2023-01-15T14:30:14.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10088510000000001 2023-01-15T14:30:14.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.498848+0000", 2023-01-15T14:30:14.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164575 2023-01-15T14:30:14.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:14.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499083+0000", 2023-01-15T14:30:14.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000235171 2023-01-15T14:30:14.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499670+0000", 2023-01-15T14:30:14.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058650200000000003 2023-01-15T14:30:14.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.499688+0000", 2023-01-15T14:30:14.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8547e-05 2023-01-15T14:30:14.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.500037+0000", 2023-01-15T14:30:14.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034891399999999997 2023-01-15T14:30:14.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.501925+0000", 2023-01-15T14:30:14.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00188766 2023-01-15T14:30:14.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.601919+0000", 2023-01-15T14:30:14.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099994371999999998 2023-01-15T14:30:14.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702377+0000", 2023-01-15T14:30:14.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045810500000001 2023-01-15T14:30:14.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702537+0000", 2023-01-15T14:30:14.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015941600000000001 2023-01-15T14:30:14.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.702650+0000", 2023-01-15T14:30:14.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011338899999999999 2023-01-15T14:30:14.479 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.479 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.479 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.479 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:30:14.479 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:30:14.480 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.592885288000005, 2023-01-15T14:30:14.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:30:14.480 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.480 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:14.480 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.480 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.480 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:30:14.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.481 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:30:14.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:30:14.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:30:14.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:14.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:30:14.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:30:14.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:30:14.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:30:14.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:30:14.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:30:14.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:30:14.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:30:14.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:30:14.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:30:14.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:30:14.486 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.376635229000001, 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:30:14.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:30:14.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:30:14.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:30:14.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:30:14.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:30:14.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:30:14.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:14.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:30:14.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:30:14.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:30:14.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:30:14.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:30:14.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:30:14.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:14.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:30:14.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:30:14.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:30:14.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:30:14.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:30:14.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:30:14.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:30:14.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:30:14.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:30:14.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:30:14.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:30:14.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:30:14.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:30:14.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.496 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.496 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.497 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:30:14.497 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:14.497 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.376601661999999, 2023-01-15T14:30:14.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:30:14.497 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.497 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.497 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.498 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.498 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.498 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:30:14.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.498 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:14.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:30:14.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:30:14.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:30:14.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:30:14.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:30:14.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:30:14.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:30:14.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:30:14.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:14.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:30:14.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:30:14.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:30:14.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:30:14.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:30:14.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:30:14.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:30:14.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:30:14.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:30:14.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:30:14.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:30:14.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:30:14.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:30:14.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:30:14.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:30:14.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:30:14.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:30:14.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:30:14.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:30:14.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:30:14.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.507 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.508 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:14.508 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:30:14.508 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.610698116000002, 2023-01-15T14:30:14.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:30:14.508 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.508 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.508 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:30:14.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:30:14.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:30:14.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:14.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:30:14.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:30:14.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:30:14.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:30:14.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:30:14.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:30:14.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:14.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:30:14.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:30:14.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:30:14.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:30:14.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:30:14.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:30:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:30:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:30:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:30:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:30:14.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:30:14.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:30:14.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:30:14.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.518 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.519 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:30:14.519 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:30:14.519 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.067519316, 2023-01-15T14:30:14.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:30:14.519 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.519 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.520 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:30:14.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.520 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:30:14.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:30:14.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:30:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:30:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:30:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:30:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:14.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:30:14.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:30:14.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:30:14.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:30:14.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:30:14.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:30:14.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:30:14.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:30:14.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:30:14.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:30:14.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.527 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:30:14.528 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:30:14.528 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.067480141000001, 2023-01-15T14:30:14.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:30:14.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:14.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.528 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:30:14.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.528 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:30:14.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.533 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1373 3.12 3:495d4bbc:::smithi09025264-16:head [write 2873523~591733] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.6753638149999999, 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40949580099999999, 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1373 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693440+0000", 2023-01-15T14:30:14.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693808+0000", 2023-01-15T14:30:14.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000367742 2023-01-15T14:30:14.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693831+0000", 2023-01-15T14:30:14.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3213e-05 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693846+0000", 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4909000000000001e-05 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.695833+0000", 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019869240000000002 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696121+0000", 2023-01-15T14:30:14.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028753300000000003 2023-01-15T14:30:14.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798577+0000", 2023-01-15T14:30:14.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102456808 2023-01-15T14:30:14.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798849+0000", 2023-01-15T14:30:14.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027138999999999999 2023-01-15T14:30:14.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:30:14.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799155+0000", 2023-01-15T14:30:14.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030594800000000001 2023-01-15T14:30:14.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799614+0000", 2023-01-15T14:30:14.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045960700000000001 2023-01-15T14:30:14.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799737+0000", 2023-01-15T14:30:14.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012218900000000001 2023-01-15T14:30:14.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799998+0000", 2023-01-15T14:30:14.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026129700000000002 2023-01-15T14:30:14.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800639+0000", 2023-01-15T14:30:14.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064108800000000005 2023-01-15T14:30:14.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102023+0000", 2023-01-15T14:30:14.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30138417200000001 2023-01-15T14:30:14.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102198+0000", 2023-01-15T14:30:14.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017453600000000001 2023-01-15T14:30:14.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102398+0000", 2023-01-15T14:30:14.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020078000000000001 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102937+0000", 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053874499999999998 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:30:14.543 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.6749392209999998, 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:30:14.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:30:14.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:30:14.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:30:14.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:30:14.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:30:14.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:30:14.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:30:14.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:30:14.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:30:14.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:30:14.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:30:14.552 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.552 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.552 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.674886892, 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.553 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:30:14.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:30:14.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:30:14.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:30:14.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:30:14.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.559 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:14.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:30:14.561 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:14.560+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:14.662 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:14.660+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:14.718 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:14.718 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:14.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.719 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGRecoveryDelete(3.5 e127,126 3:a75958b1:::smithi09025264-6:47,101'48)", 2023-01-15T14:30:14.719 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:14.640436+0000", 2023-01-15T14:30:14.719 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.077061678999999994, 2023-01-15T14:30:14.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.077101425000000001, 2023-01-15T14:30:14.719 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.719 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:14.719 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:14.640436+0000", 2023-01-15T14:30:14.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:14.640436+0000", 2023-01-15T14:30:14.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:14.640434+0000", 2023-01-15T14:30:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:14.640448+0000", 2023-01-15T14:30:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3923e-05 2023-01-15T14:30:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:14.640473+0000", 2023-01-15T14:30:14.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4867000000000001e-05 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:14.640488+0000", 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4822e-05 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:30:14.722 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:14.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:30:14.763 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:14.760+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:14.814 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:30:14.815 INFO:tasks.rados.rados.0.smithi090.stdout:524: done (2 left) 2023-01-15T14:30:14.815 INFO:tasks.rados.rados.0.smithi090.stdout:525: copy_from oid 25 from oid 7 current snap is 49 2023-01-15T14:30:14.815 INFO:tasks.rados.rados.0.smithi090.stdout:526: snap_create 2023-01-15T14:30:14.816 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:30:14.816 INFO:tasks.rados.rados.0.smithi090.stdout:523: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:30:14.842 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:14.842 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:14.842 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:14.852 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:30:14.864 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:14.860+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:14.939 INFO:tasks.rados.rados.0.smithi090.stdout:523: done (2 left) 2023-01-15T14:30:14.939 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:14.959 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:14.959 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:14.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:14.959 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:14.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.959 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:94 3.0 3:07a51d33:::smithi09025264-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.960 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:30:14.960 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.514339393, 2023-01-15T14:30:14.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50760159199999999, 2023-01-15T14:30:14.960 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.960 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.960 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.960 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.960 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.961 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 94 2023-01-15T14:30:14.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.961 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:30:14.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439540+0000", 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439538+0000", 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439857+0000", 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000319086 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439891+0000", 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4449999999999997e-05 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439906+0000", 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5082999999999999e-05 2023-01-15T14:30:14.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824189+0000", 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38428238100000001 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824440+0000", 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025139800000000001 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:30:14.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.824834+0000", 2023-01-15T14:30:14.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039422200000000001 2023-01-15T14:30:14.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825347+0000", 2023-01-15T14:30:14.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051323600000000003 2023-01-15T14:30:14.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825364+0000", 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6478e-05 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.825589+0000", 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022497500000000001 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826457+0000", 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00086797600000000001 2023-01-15T14:30:14.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.846539+0000", 2023-01-15T14:30:14.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020081742 2023-01-15T14:30:14.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.946989+0000", 2023-01-15T14:30:14.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10045032199999999 2023-01-15T14:30:14.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947076+0000", 2023-01-15T14:30:14.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6922999999999994e-05 2023-01-15T14:30:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947141+0000", 2023-01-15T14:30:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5357000000000005e-05 2023-01-15T14:30:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.969 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:96 3.0 3:07a51d33:::smithi09025264-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.970 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:30:14.970 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.51391728, 2023-01-15T14:30:14.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50769745899999996, 2023-01-15T14:30:14.970 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.970 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:14.970 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.970 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.970 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.970 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 96 2023-01-15T14:30:14.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.970 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:30:14.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439962+0000", 2023-01-15T14:30:14.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439961+0000", 2023-01-15T14:30:14.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:14.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439966+0000", 2023-01-15T14:30:14.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0769999999999998e-06 2023-01-15T14:30:14.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439979+0000", 2023-01-15T14:30:14.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3343000000000001e-05 2023-01-15T14:30:14.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.439987+0000", 2023-01-15T14:30:14.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2400000000000001e-06 2023-01-15T14:30:14.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826191+0000", 2023-01-15T14:30:14.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.38620452799999999 2023-01-15T14:30:14.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.826424+0000", 2023-01-15T14:30:14.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023264800000000001 2023-01-15T14:30:14.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947394+0000", 2023-01-15T14:30:14.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.120970102 2023-01-15T14:30:14.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947505+0000", 2023-01-15T14:30:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001105 2023-01-15T14:30:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.947659+0000", 2023-01-15T14:30:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015468799999999999 2023-01-15T14:30:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.977 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.977 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:14.977 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.869364616, 2023-01-15T14:30:14.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:30:14.977 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.977 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.978 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:30:14.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.978 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:14.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:14.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:14.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:30:14.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:30:14.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:30:14.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:30:14.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:30:14.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:30:14.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:30:14.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:30:14.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:30:14.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:30:14.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:30:14.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:30:14.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:30:14.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:30:14.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:14.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:30:14.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:30:14.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:30:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:30:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:30:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:30:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:30:14.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:30:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:30:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:30:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:30:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:30:14.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:30:14.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:30:14.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:30:14.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:30:14.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:30:14.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:30:14.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.987 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.987 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.987 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:30:14.988 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.869064819, 2023-01-15T14:30:14.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:30:14.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.988 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:14.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.988 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:30:14.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.988 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:30:14.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:30:14.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:14.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:30:14.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:30:14.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:14.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:30:14.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:30:14.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:30:14.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:30:14.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:14.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:30:14.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:30:14.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:14.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:30:14.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:14.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:30:14.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:30:14.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:14.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:30:14.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:30:14.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:30:14.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:30:14.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:14.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:30:14.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:30:14.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:14.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:30:14.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:30:14.997 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.869035035, 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:14.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:30:15.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:30:15.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:30:15.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:30:15.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:30:15.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:30:15.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:30:15.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:30:15.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:30:15.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:30:15.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:30:15.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:30:15.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:30:15.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:30:15.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:30:15.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.005 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.005 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:182 3.4 3:2022fdfa:::smithi09025264-46:head [write 1933572~791670] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:15.005 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:30:15.005 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.740172966, 2023-01-15T14:30:15.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.582304865, 2023-01-15T14:30:15.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.005 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 182 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:30:15.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213704+0000", 2023-01-15T14:30:15.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:15.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214574+0000", 2023-01-15T14:30:15.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087037799999999999 2023-01-15T14:30:15.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214608+0000", 2023-01-15T14:30:15.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:30:15.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214627+0000", 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491537+0000", 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27691020300000002 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491721+0000", 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345500000000001 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693760+0000", 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202039252 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693898+0000", 2023-01-15T14:30:15.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013789 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694230+0000", 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220899999999998 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694866+0000", 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063640599999999995 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694887+0000", 2023-01-15T14:30:15.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0622999999999999e-05 2023-01-15T14:30:15.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695950+0000", 2023-01-15T14:30:15.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001062688 2023-01-15T14:30:15.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:15.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696014+0000", 2023-01-15T14:30:15.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4697999999999996e-05 2023-01-15T14:30:15.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795898+0000", 2023-01-15T14:30:15.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099883338000000002 2023-01-15T14:30:15.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795922+0000", 2023-01-15T14:30:15.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4326000000000001e-05 2023-01-15T14:30:15.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795955+0000", 2023-01-15T14:30:15.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3191000000000001e-05 2023-01-15T14:30:15.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796011+0000", 2023-01-15T14:30:15.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5631999999999999e-05 2023-01-15T14:30:15.015 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.015 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.015 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.015 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:15.016 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:30:15.016 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.739226235, 2023-01-15T14:30:15.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:30:15.016 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.016 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.016 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.016 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.016 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.016 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:30:15.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.017 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:30:15.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:30:15.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:30:15.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:15.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:30:15.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:30:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:30:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:30:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:30:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:30:15.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:30:15.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:30:15.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:30:15.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:15.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:30:15.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:30:15.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:30:15.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:30:15.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:30:15.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:30:15.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:30:15.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:30:15.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:30:15.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.026 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:15.026 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:30:15.026 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.739169538, 2023-01-15T14:30:15.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:30:15.026 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.026 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:15.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.027 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:30:15.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.027 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:30:15.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:30:15.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:30:15.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:15.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:30:15.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:30:15.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:30:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:30:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:30:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:30:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:30:15.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:30:15.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:30:15.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:30:15.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:30:15.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:30:15.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:30:15.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:30:15.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:30:15.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:30:15.034 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.034 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.034 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.034 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:30:15.034 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:30:15.034 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 105.122673365, 2023-01-15T14:30:15.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:30:15.034 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.034 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:30:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:30:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:30:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:30:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:15.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:30:15.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:30:15.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:30:15.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:30:15.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:30:15.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:30:15.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:30:15.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:30:15.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:30:15.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:30:15.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:30:15.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:30:15.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:30:15.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:30:15.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:30:15.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:30:15.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.041 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.041 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.041 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:30:15.041 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:30:15.041 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.093824893999994, 2023-01-15T14:30:15.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:30:15.041 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:30:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:30:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:30:15.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:30:15.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:15.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:30:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:30:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:30:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:30:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:30:15.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:30:15.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:30:15.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:30:15.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:30:15.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:30:15.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:30:15.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:30:15.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:30:15.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:30:15.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:30:15.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.050 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.051 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:373 3.4 3:2029fa71:::smithi09025264-39:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:15.051 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:30:15.051 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.875577687000003, 2023-01-15T14:30:15.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58298356399999995, 2023-01-15T14:30:15.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.051 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.052 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-01-15T14:30:15.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.052 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:30:15.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078299+0000", 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078644+0000", 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034494099999999998 2023-01-15T14:30:15.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078676+0000", 2023-01-15T14:30:15.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1183000000000003e-05 2023-01-15T14:30:15.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078694+0000", 2023-01-15T14:30:15.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-05 2023-01-15T14:30:15.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078748+0000", 2023-01-15T14:30:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3634999999999997e-05 2023-01-15T14:30:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078753+0000", 2023-01-15T14:30:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6370000000000004e-06 2023-01-15T14:30:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110841+0000", 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032087439000000002 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110967+0000", 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126029 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.457874+0000", 2023-01-15T14:30:15.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.346907725 2023-01-15T14:30:15.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458066+0000", 2023-01-15T14:30:15.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000191592 2023-01-15T14:30:15.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:15.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458321+0000", 2023-01-15T14:30:15.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025454200000000002 2023-01-15T14:30:15.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458918+0000", 2023-01-15T14:30:15.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059740800000000001 2023-01-15T14:30:15.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458945+0000", 2023-01-15T14:30:15.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-05 2023-01-15T14:30:15.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459253+0000", 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030836200000000001 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460943+0000", 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016903580000000001 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461340+0000", 2023-01-15T14:30:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039665799999999998 2023-01-15T14:30:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661129+0000", 2023-01-15T14:30:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19978933500000001 2023-01-15T14:30:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661188+0000", 2023-01-15T14:30:15.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8687000000000002e-05 2023-01-15T14:30:15.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661285+0000", 2023-01-15T14:30:15.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779000000000003e-05 2023-01-15T14:30:15.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.061 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.061 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.061 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:374 3.4 3:2029fa71:::smithi09025264-39:head [write 3101234~41642] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:30:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.870915775, 2023-01-15T14:30:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57848911800000002, 2023-01-15T14:30:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 374 2023-01-15T14:30:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.062 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082961+0000", 2023-01-15T14:30:15.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:15.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083019+0000", 2023-01-15T14:30:15.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999997e-05 2023-01-15T14:30:15.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083051+0000", 2023-01-15T14:30:15.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1813999999999997e-05 2023-01-15T14:30:15.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083069+0000", 2023-01-15T14:30:15.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:30:15.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083143+0000", 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4363000000000006e-05 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083147+0000", 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4529999999999998e-06 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111017+0000", 2023-01-15T14:30:15.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027870564 2023-01-15T14:30:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111099+0000", 2023-01-15T14:30:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1277000000000001e-05 2023-01-15T14:30:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459156+0000", 2023-01-15T14:30:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34805700499999997 2023-01-15T14:30:15.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459393+0000", 2023-01-15T14:30:15.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023757499999999999 2023-01-15T14:30:15.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:15.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459778+0000", 2023-01-15T14:30:15.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000385335 2023-01-15T14:30:15.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460020+0000", 2023-01-15T14:30:15.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024141899999999999 2023-01-15T14:30:15.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460037+0000", 2023-01-15T14:30:15.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6665999999999999e-05 2023-01-15T14:30:15.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460220+0000", 2023-01-15T14:30:15.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018367599999999999 2023-01-15T14:30:15.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:15.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461197+0000", 2023-01-15T14:30:15.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00097708600000000006 2023-01-15T14:30:15.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461720+0000", 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052235400000000005 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661345+0000", 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199624884 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661405+0000", 2023-01-15T14:30:15.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0965999999999999e-05 2023-01-15T14:30:15.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661452+0000", 2023-01-15T14:30:15.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6864999999999999e-05 2023-01-15T14:30:15.072 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.072 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.072 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.072 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:375 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (56) in=7b,truncate 3142876] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.870774009000002, 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57851114599999998, 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 375 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:30:15.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:30:15.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083104+0000", 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083113+0000", 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6500000000000008e-06 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083130+0000", 2023-01-15T14:30:15.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2023-01-15T14:30:15.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083142+0000", 2023-01-15T14:30:15.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2252999999999999e-05 2023-01-15T14:30:15.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083173+0000", 2023-01-15T14:30:15.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0474999999999999e-05 2023-01-15T14:30:15.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083175+0000", 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183469+0000", 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10029313099999999 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183747+0000", 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000277998 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460165+0000", 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27641872899999997 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460342+0000", 2023-01-15T14:30:15.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176792 2023-01-15T14:30:15.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:15.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460579+0000", 2023-01-15T14:30:15.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000237266 2023-01-15T14:30:15.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460833+0000", 2023-01-15T14:30:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025390900000000002 2023-01-15T14:30:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460853+0000", 2023-01-15T14:30:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9273e-05 2023-01-15T14:30:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461024+0000", 2023-01-15T14:30:15.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017111900000000001 2023-01-15T14:30:15.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:15.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461230+0000", 2023-01-15T14:30:15.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020672600000000001 2023-01-15T14:30:15.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461879+0000", 2023-01-15T14:30:15.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064846999999999995 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661524+0000", 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19964509599999999 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661560+0000", 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6264999999999999e-05 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661616+0000", 2023-01-15T14:30:15.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5952000000000002e-05 2023-01-15T14:30:15.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.083 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.083 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:376 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:15.083 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:30:15.083 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.870691753000003, 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57864278800000002, 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 376 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:30:15.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:30:15.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083186+0000", 2023-01-15T14:30:15.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083193+0000", 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083211+0000", 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7609999999999999e-05 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083223+0000", 2023-01-15T14:30:15.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140000000000001e-05 2023-01-15T14:30:15.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093534+0000", 2023-01-15T14:30:15.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010311529 2023-01-15T14:30:15.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:15.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093551+0000", 2023-01-15T14:30:15.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.666e-05 2023-01-15T14:30:15.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183834+0000", 2023-01-15T14:30:15.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090283063999999996 2023-01-15T14:30:15.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.184076+0000", 2023-01-15T14:30:15.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024241900000000001 2023-01-15T14:30:15.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461001+0000", 2023-01-15T14:30:15.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27692504299999998 2023-01-15T14:30:15.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461152+0000", 2023-01-15T14:30:15.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150345 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661642+0000", 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200489742 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661720+0000", 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8604999999999997e-05 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661830+0000", 2023-01-15T14:30:15.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010992099999999999 2023-01-15T14:30:15.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.091 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:30:15.091 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:15.091 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.092537918000005, 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:15.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:15.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:15.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:15.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:30:15.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:30:15.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:30:15.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:30:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:30:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:30:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:30:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:30:15.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:30:15.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:30:15.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:30:15.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:30:15.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:30:15.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:30:15.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:30:15.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.100 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.100 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:961 3.0 3:0a998d04:::smithi09025264-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2023-01-15T14:30:15.100 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:15.100 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.890769329999998, 2023-01-15T14:30:15.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51361718300000003, 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 961 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:15.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063109+0000", 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063115+0000", 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8300000000000001e-06 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063127+0000", 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2517000000000001e-05 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063137+0000", 2023-01-15T14:30:15.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8509999999999994e-06 2023-01-15T14:30:15.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063272+0000", 2023-01-15T14:30:15.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013513500000000001 2023-01-15T14:30:15.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:15.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063277+0000", 2023-01-15T14:30:15.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2750000000000003e-06 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.265103+0000", 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201825688 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.265208+0000", 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104693 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.375398+0000", 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.110190553 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.375548+0000", 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149417 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576187+0000", 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20063947500000001 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576454+0000", 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026708400000000001 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576727+0000", 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027257699999999998 2023-01-15T14:30:15.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.031959667999999, 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.109 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.109 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:30:15.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.109 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:15.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:30:15.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:30:15.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:30:15.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:30:15.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:30:15.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:30:15.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:30:15.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:30:15.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:30:15.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:30:15.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.116 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.031900958000001, 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.117 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:15.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:15.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:30:15.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:30:15.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:15.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:30:15.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:30:15.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:30:15.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.075481464999999, 2023-01-15T14:30:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:30:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:30:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:30:15.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:30:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:30:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:30:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:30:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:30:15.129 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 17.017983782000002, 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:15.130 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:30:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:30:15.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:30:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:30:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:30:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:30:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:30:15.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:30:15.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:30:15.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:30:15.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:30:15.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:30:15.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:30:15.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:30:15.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:15.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:30:15.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:30:15.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:30:15.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:30:15.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:30:15.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:30:15.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.139 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.139 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.139 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:15.141 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:30:15.141 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:15.136+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:15.245 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:15.240+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:15.301 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:15.301 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:15.302 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:30:15.302 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:15.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:30:15.345 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:15.340+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:15.547 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:15.547 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:15.547 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:15.547 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:15.547 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:15.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:30:15.549 INFO:tasks.rados.rados.0.smithi090.stdout:521: finishing write tid 1 to smithi09025264-13 2023-01-15T14:30:15.550 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:15.544+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:15.622 INFO:tasks.rados.rados.0.smithi090.stdout:521: finishing write tid 2 to smithi09025264-13 2023-01-15T14:30:15.716 INFO:tasks.rados.rados.0.smithi090.stdout:521: finishing write tid 3 to smithi09025264-13 2023-01-15T14:30:15.716 INFO:tasks.rados.rados.0.smithi090.stdout:521: finishing write tid 4 to smithi09025264-13 2023-01-15T14:30:15.717 INFO:tasks.rados.rados.0.smithi090.stdout:521: finishing write tid 5 to smithi09025264-13 2023-01-15T14:30:15.717 INFO:tasks.rados.rados.0.smithi090.stdout:521: oid 13 updating version 0 to 147 2023-01-15T14:30:15.717 INFO:tasks.rados.rados.0.smithi090.stdout:521: oid 13 updating version 147 to 149 2023-01-15T14:30:15.718 INFO:tasks.rados.rados.0.smithi090.stdout:521: oid 13 updating version 149 to 150 2023-01-15T14:30:15.718 INFO:tasks.rados.rados.0.smithi090.stdout:521: oid 13 version 150 is already newer than 148 2023-01-15T14:30:15.718 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 150 (ObjNum 216 snap 49 seq_num 216) dirty exists 2023-01-15T14:30:15.718 INFO:tasks.rados.rados.0.smithi090.stdout:521: left oid 13 (ObjNum 216 snap 49 seq_num 216) 2023-01-15T14:30:15.718 INFO:tasks.rados.rados.0.smithi090.stdout:521: done (1 left) 2023-01-15T14:30:15.718 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:15.719 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:15.722 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:15.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:15.722 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:15.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.722 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:38 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'3, mlcod=0'0)", 2023-01-15T14:30:15.723 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:30:15.723 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.590878104, 2023-01-15T14:30:15.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30191175999999997, 2023-01-15T14:30:15.723 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.723 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.724 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:30:15.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:30:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112976+0000", 2023-01-15T14:30:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113186+0000", 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020979700000000001 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113228+0000", 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2150999999999997e-05 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213313+0000", 2023-01-15T14:30:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100084943 2023-01-15T14:30:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313517+0000", 2023-01-15T14:30:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10020435699999999 2023-01-15T14:30:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313576+0000", 2023-01-15T14:30:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783999999999998e-05 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313976+0000", 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039960499999999998 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314004+0000", 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7954e-05 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314175+0000", 2023-01-15T14:30:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171557 2023-01-15T14:30:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414853+0000", 2023-01-15T14:30:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100677875 2023-01-15T14:30:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414890+0000", 2023-01-15T14:30:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6803e-05 2023-01-15T14:30:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:39 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'4, mlcod=0'0)", 2023-01-15T14:30:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:30:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.490485929, 2023-01-15T14:30:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20154512699999999, 2023-01-15T14:30:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:30:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:30:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213369+0000", 2023-01-15T14:30:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213386+0000", 2023-01-15T14:30:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7643e-05 2023-01-15T14:30:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213427+0000", 2023-01-15T14:30:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0930000000000003e-05 2023-01-15T14:30:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213459+0000", 2023-01-15T14:30:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1479999999999997e-05 2023-01-15T14:30:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314106+0000", 2023-01-15T14:30:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100646614 2023-01-15T14:30:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314128+0000", 2023-01-15T14:30:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2340999999999998e-05 2023-01-15T14:30:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314224+0000", 2023-01-15T14:30:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6534999999999996e-05 2023-01-15T14:30:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314249+0000", 2023-01-15T14:30:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4981e-05 2023-01-15T14:30:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314337+0000", 2023-01-15T14:30:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7287000000000006e-05 2023-01-15T14:30:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414903+0000", 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100566552 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414916+0000", 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354000000000001e-05 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:46 3.0 e23/20 3:03b54904:::smithi09025264-12:head v 23'5, mlcod=23'1)", 2023-01-15T14:30:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:30:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.48998219000001, 2023-01-15T14:30:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202592668, 2023-01-15T14:30:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:30:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:30:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213873+0000", 2023-01-15T14:30:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214477+0000", 2023-01-15T14:30:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060481300000000005 2023-01-15T14:30:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214532+0000", 2023-01-15T14:30:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5096999999999998e-05 2023-01-15T14:30:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214555+0000", 2023-01-15T14:30:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.234e-05 2023-01-15T14:30:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414409+0000", 2023-01-15T14:30:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19985461199999999 2023-01-15T14:30:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414442+0000", 2023-01-15T14:30:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2987000000000001e-05 2023-01-15T14:30:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414813+0000", 2023-01-15T14:30:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037061899999999998 2023-01-15T14:30:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414832+0000", 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9380000000000001e-05 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415093+0000", 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026032099999999999 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415154+0000", 2023-01-15T14:30:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1538999999999995e-05 2023-01-15T14:30:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.416467+0000", 2023-01-15T14:30:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001312613 2023-01-15T14:30:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:281 3.5 e30/20 3:a75958b1:::smithi09025264-6:head v 30'11, mlcod=23'6)", 2023-01-15T14:30:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:34.981189+0000", 2023-01-15T14:30:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.722666891, 2023-01-15T14:30:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19987028400000001, 2023-01-15T14:30:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981189+0000", 2023-01-15T14:30:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981189+0000", 2023-01-15T14:30:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981188+0000", 2023-01-15T14:30:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981206+0000", 2023-01-15T14:30:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8377999999999999e-05 2023-01-15T14:30:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981231+0000", 2023-01-15T14:30:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.516e-05 2023-01-15T14:30:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:34.981249+0000", 2023-01-15T14:30:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7425e-05 2023-01-15T14:30:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.179815+0000", 2023-01-15T14:30:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19856638300000001 2023-01-15T14:30:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.179854+0000", 2023-01-15T14:30:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8587000000000002e-05 2023-01-15T14:30:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180059+0000", 2023-01-15T14:30:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020564300000000001 2023-01-15T14:30:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180085+0000", 2023-01-15T14:30:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5162999999999999e-05 2023-01-15T14:30:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180548+0000", 2023-01-15T14:30:15.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046301300000000002 2023-01-15T14:30:15.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.180599+0000", 2023-01-15T14:30:15.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0943000000000003e-05 2023-01-15T14:30:15.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:35.181060+0000", 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046124700000000002 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:347 3.0 e32/20 3:03b54904:::smithi09025264-12:head v 32'40, mlcod=31'36)", 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.692059291000007, 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20127003500000001, 2023-01-15T14:30:15.789 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011795+0000", 2023-01-15T14:30:15.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:15.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.012239+0000", 2023-01-15T14:30:15.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000444361 2023-01-15T14:30:15.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.012261+0000", 2023-01-15T14:30:15.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2008e-05 2023-01-15T14:30:15.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.112398+0000", 2023-01-15T14:30:15.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013730799999999 2023-01-15T14:30:15.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.210584+0000", 2023-01-15T14:30:15.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098186184999999995 2023-01-15T14:30:15.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.210634+0000", 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9706000000000001e-05 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211053+0000", 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041844199999999999 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211117+0000", 2023-01-15T14:30:15.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4950000000000007e-05 2023-01-15T14:30:15.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211390+0000", 2023-01-15T14:30:15.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027278300000000002 2023-01-15T14:30:15.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211507+0000", 2023-01-15T14:30:15.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011642 2023-01-15T14:30:15.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.213067+0000", 2023-01-15T14:30:15.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001560402 2023-01-15T14:30:15.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.877 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.878 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.4.0:118 3.9 e62/20 3:9f7cca9d:::smithi09025264-41:1f v 62'64, mlcod=61'58)", 2023-01-15T14:30:15.878 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:30:15.878 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 67.830668802999995, 2023-01-15T14:30:15.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20203982600000001, 2023-01-15T14:30:15.878 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.878 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.878 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:30:15.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:30:15.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873186+0000", 2023-01-15T14:30:15.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:15.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873201+0000", 2023-01-15T14:30:15.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5474000000000001e-05 2023-01-15T14:30:15.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873219+0000", 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7677999999999999e-05 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873234+0000", 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4481000000000001e-05 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.073729+0000", 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200495595 2023-01-15T14:30:15.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.073768+0000", 2023-01-15T14:30:15.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8492999999999999e-05 2023-01-15T14:30:15.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074468+0000", 2023-01-15T14:30:15.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069991300000000003 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074486+0000", 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8669999999999999e-05 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074563+0000", 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7084999999999998e-05 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074599+0000", 2023-01-15T14:30:15.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5453000000000001e-05 2023-01-15T14:30:15.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.075227+0000", 2023-01-15T14:30:15.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062855400000000003 2023-01-15T14:30:15.884 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.884 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.884 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.885 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:125 3.0 e70/55 3:0b54190a:::smithi09025264-17:25 v 70'136, mlcod=68'130)", 2023-01-15T14:30:15.885 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:30:15.885 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.760896078999998, 2023-01-15T14:30:15.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30171926799999998, 2023-01-15T14:30:15.885 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.886 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.886 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:30:15.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:30:15.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942958+0000", 2023-01-15T14:30:15.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:15.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942972+0000", 2023-01-15T14:30:15.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3386e-05 2023-01-15T14:30:15.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942990+0000", 2023-01-15T14:30:15.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8113000000000001e-05 2023-01-15T14:30:15.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.943002+0000", 2023-01-15T14:30:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:30:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141942+0000", 2023-01-15T14:30:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19893962300000001 2023-01-15T14:30:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141978+0000", 2023-01-15T14:30:15.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6383e-05 2023-01-15T14:30:15.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142670+0000", 2023-01-15T14:30:15.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069211900000000005 2023-01-15T14:30:15.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142688+0000", 2023-01-15T14:30:15.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7839999999999999e-05 2023-01-15T14:30:15.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142768+0000", 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0210999999999995e-05 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244581+0000", 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101813406 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244680+0000", 2023-01-15T14:30:15.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8045000000000001e-05 2023-01-15T14:30:15.892 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.892 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.892 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.892 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:948 3.8 e83/41 3:17d5d741:::smithi09025264-5:head v 83'119, mlcod=82'115)", 2023-01-15T14:30:15.892 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:30:15.892 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.353358995999997, 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201378735, 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:30:15.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350496+0000", 2023-01-15T14:30:15.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:15.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350511+0000", 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5007e-05 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350535+0000", 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4170000000000001e-05 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.450633+0000", 2023-01-15T14:30:15.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100098221 2023-01-15T14:30:15.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.550879+0000", 2023-01-15T14:30:15.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100245538 2023-01-15T14:30:15.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.550917+0000", 2023-01-15T14:30:15.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8043000000000002e-05 2023-01-15T14:30:15.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551222+0000", 2023-01-15T14:30:15.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030465400000000001 2023-01-15T14:30:15.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:15.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551241+0000", 2023-01-15T14:30:15.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9049999999999999e-05 2023-01-15T14:30:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551331+0000", 2023-01-15T14:30:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0807e-05 2023-01-15T14:30:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551356+0000", 2023-01-15T14:30:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4786999999999998e-05 2023-01-15T14:30:15.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:15.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551876+0000", 2023-01-15T14:30:15.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051983800000000001 2023-01-15T14:30:15.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.899 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:15.899 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:15.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.899 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1124 3.5 e97/48 3:a75958b1:::smithi09025264-6:head v 97'44, mlcod=94'39)", 2023-01-15T14:30:15.899 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:15.900 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.566451922999999, 2023-01-15T14:30:15.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200993378, 2023-01-15T14:30:15.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:15.900 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:15.900 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:15.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:15.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:15.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137455+0000", 2023-01-15T14:30:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1187000000000003e-05 2023-01-15T14:30:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137475+0000", 2023-01-15T14:30:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9692999999999999e-05 2023-01-15T14:30:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137496+0000", 2023-01-15T14:30:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1776e-05 2023-01-15T14:30:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238363+0000", 2023-01-15T14:30:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10086711900000001 2023-01-15T14:30:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238415+0000", 2023-01-15T14:30:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1465999999999998e-05 2023-01-15T14:30:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238578+0000", 2023-01-15T14:30:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016310599999999999 2023-01-15T14:30:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:16.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238598+0000", 2023-01-15T14:30:16.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9729999999999999e-05 2023-01-15T14:30:16.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:16.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238673+0000", 2023-01-15T14:30:16.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5302000000000005e-05 2023-01-15T14:30:16.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:16.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239068+0000", 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000395227 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.338398+0000", 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099329534999999997 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1125 3.5 e97/48 3:a75958b1:::smithi09025264-6:head v 97'45, mlcod=94'39)", 2023-01-15T14:30:16.039 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.56633471, 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201211626, 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:30:16.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137521+0000", 2023-01-15T14:30:16.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:16.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137531+0000", 2023-01-15T14:30:16.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6399999999999992e-06 2023-01-15T14:30:16.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137546+0000", 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5783000000000001e-05 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137557+0000", 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0657e-05 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238639+0000", 2023-01-15T14:30:16.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10108166 2023-01-15T14:30:16.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238688+0000", 2023-01-15T14:30:16.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9570000000000001e-05 2023-01-15T14:30:16.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:16.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238931+0000", 2023-01-15T14:30:16.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000243255 2023-01-15T14:30:16.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:16.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238948+0000", 2023-01-15T14:30:16.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6246e-05 2023-01-15T14:30:16.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:16.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238999+0000", 2023-01-15T14:30:16.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1063000000000002e-05 2023-01-15T14:30:16.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:16.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239101+0000", 2023-01-15T14:30:16.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010191000000000001 2023-01-15T14:30:16.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.338733+0000", 2023-01-15T14:30:16.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099632581999999997 2023-01-15T14:30:16.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.055 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.055 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1119 3.8 e97/41 3:1eaad226:::smithi09025264-49:head v 97'157, mlcod=97'154)", 2023-01-15T14:30:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:30:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.564716405999999, 2023-01-15T14:30:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200961838, 2023-01-15T14:30:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:30:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:30:16.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139139+0000", 2023-01-15T14:30:16.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:16.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139291+0000", 2023-01-15T14:30:16.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015233900000000001 2023-01-15T14:30:16.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139318+0000", 2023-01-15T14:30:16.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6265999999999999e-05 2023-01-15T14:30:16.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:16.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139331+0000", 2023-01-15T14:30:16.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3757000000000001e-05 2023-01-15T14:30:16.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238733+0000", 2023-01-15T14:30:16.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099401891000000006 2023-01-15T14:30:16.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238778+0000", 2023-01-15T14:30:16.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5188999999999997e-05 2023-01-15T14:30:16.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:16.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239141+0000", 2023-01-15T14:30:16.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000362911 2023-01-15T14:30:16.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:16.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239155+0000", 2023-01-15T14:30:16.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3879e-05 2023-01-15T14:30:16.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:16.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239274+0000", 2023-01-15T14:30:16.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011916 2023-01-15T14:30:16.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:16.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239310+0000", 2023-01-15T14:30:16.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5602999999999998e-05 2023-01-15T14:30:16.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.340102+0000", 2023-01-15T14:30:16.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10079187000000001 2023-01-15T14:30:16.205 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.205 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.205 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.205 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(3.0 REQUEST e99)", 2023-01-15T14:30:16.205 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:16.206 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.707442436000001, 2023-01-15T14:30:16.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26432840200000002, 2023-01-15T14:30:16.206 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.206 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:16.207 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:16.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996419+0000", 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6729999999999997e-06 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996449+0000", 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.004e-05 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:16.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996463+0000", 2023-01-15T14:30:16.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3607e-05 2023-01-15T14:30:16.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996594+0000", 2023-01-15T14:30:16.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013148899999999999 2023-01-15T14:30:16.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996618+0000", 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3249000000000001e-05 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260645+0000", 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26402768700000001 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260671+0000", 2023-01-15T14:30:16.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.546e-05 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260742+0000", 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1569999999999994e-05 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:261 3.8 e98/41 3:1e54a7f5:::smithi09025264-42:3e v 98'160, mlcod=97'157)", 2023-01-15T14:30:16.212 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:30:16.213 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.646374847000001, 2023-01-15T14:30:16.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20299468400000001, 2023-01-15T14:30:16.213 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:16.213 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:30:16.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:30:16.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057480+0000", 2023-01-15T14:30:16.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:16.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057496+0000", 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5869999999999999e-05 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057528+0000", 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:16.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057545+0000", 2023-01-15T14:30:16.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.694e-05 2023-01-15T14:30:16.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.158971+0000", 2023-01-15T14:30:16.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101425425 2023-01-15T14:30:16.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159011+0000", 2023-01-15T14:30:16.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0058000000000002e-05 2023-01-15T14:30:16.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:16.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159858+0000", 2023-01-15T14:30:16.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084767900000000005 2023-01-15T14:30:16.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:16.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159877+0000", 2023-01-15T14:30:16.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9133000000000001e-05 2023-01-15T14:30:16.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:16.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159959+0000", 2023-01-15T14:30:16.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1693000000000005e-05 2023-01-15T14:30:16.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:16.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260430+0000", 2023-01-15T14:30:16.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10047039200000001 2023-01-15T14:30:16.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260476+0000", 2023-01-15T14:30:16.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6652e-05 2023-01-15T14:30:16.219 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.219 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.219 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.645153691000001, 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:30:16.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:30:16.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:30:16.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:30:16.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:30:16.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:16.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:30:16.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:30:16.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:16.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:30:16.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:30:16.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:16.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:30:16.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:30:16.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:16.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:30:16.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:30:16.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:30:16.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:30:16.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.226 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.227 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "replica_scrub(pg: 3.0,from:0'0,to:98'189,epoch:99/95,start:3:00000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2023-01-15T14:30:16.227 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:30:16.227 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.34201375, 2023-01-15T14:30:16.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20057929099999999, 2023-01-15T14:30:16.227 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.227 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:16.228 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:30:16.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:30:16.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361841+0000", 2023-01-15T14:30:16.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:16.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361855+0000", 2023-01-15T14:30:16.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3464e-05 2023-01-15T14:30:16.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361886+0000", 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1389000000000001e-05 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.462023+0000", 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013720800000001 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562267+0000", 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100243392 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562322+0000", 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5075999999999999e-05 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562422+0000", 2023-01-15T14:30:16.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010015599999999999 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(2.1 GRANT e100)", 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.386191463999999, 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20048780599999999, 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:16.232 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317663+0000", 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317674+0000", 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.119e-05 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317702+0000", 2023-01-15T14:30:16.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7183e-05 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.417845+0000", 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100142988 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518107+0000", 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100262461 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518121+0000", 2023-01-15T14:30:16.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.449e-05 2023-01-15T14:30:16.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518124+0000", 2023-01-15T14:30:16.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1730000000000002e-06 2023-01-15T14:30:16.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518153+0000", 2023-01-15T14:30:16.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9043999999999998e-05 2023-01-15T14:30:16.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.237 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.238 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:194 3.4 e101/48 3:2022fdfa:::smithi09025264-46:43 v 101'104, mlcod=98'100)", 2023-01-15T14:30:16.238 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:30:16.238 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.859112079999999, 2023-01-15T14:30:16.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20670733199999999, 2023-01-15T14:30:16.238 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.238 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:16.238 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:30:16.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:30:16.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844743+0000", 2023-01-15T14:30:16.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:16.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844757+0000", 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.484e-05 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844787+0000", 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9113000000000001e-05 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.944923+0000", 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013656 2023-01-15T14:30:16.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.949975+0000", 2023-01-15T14:30:16.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050521400000000001 2023-01-15T14:30:16.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950015+0000", 2023-01-15T14:30:16.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9286999999999998e-05 2023-01-15T14:30:16.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950559+0000", 2023-01-15T14:30:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054463500000000002 2023-01-15T14:30:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950585+0000", 2023-01-15T14:30:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6166e-05 2023-01-15T14:30:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:16.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950670+0000", 2023-01-15T14:30:16.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4902999999999997e-05 2023-01-15T14:30:16.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:16.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051368+0000", 2023-01-15T14:30:16.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100697913 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051452+0000", 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3442000000000005e-05 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1224 3.4 e107/48 3:2029fa71:::smithi09025264-39:head v 107'120, mlcod=104'116)", 2023-01-15T14:30:16.245 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.472083439999999, 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22601005900000001, 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:30:16.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231771+0000", 2023-01-15T14:30:16.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:16.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233829+0000", 2023-01-15T14:30:16.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002057836 2023-01-15T14:30:16.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233881+0000", 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1856e-05 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.333996+0000", 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100115315 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434199+0000", 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100203373 2023-01-15T14:30:16.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434236+0000", 2023-01-15T14:30:16.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6563000000000002e-05 2023-01-15T14:30:16.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:16.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.435979+0000", 2023-01-15T14:30:16.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017426570000000001 2023-01-15T14:30:16.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:16.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.436036+0000", 2023-01-15T14:30:16.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7716000000000001e-05 2023-01-15T14:30:16.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:16.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437140+0000", 2023-01-15T14:30:16.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011036399999999999 2023-01-15T14:30:16.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:16.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437251+0000", 2023-01-15T14:30:16.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110869 2023-01-15T14:30:16.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.457783+0000", 2023-01-15T14:30:16.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020532181 2023-01-15T14:30:16.253 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.253 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.255 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.13 120/114 [PushOp(3:cbfd2d71:::smithi09025264-37:head, version: 112'211, data_included: [3145728~182058], data_size: 182058, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:cbfd2d71:::smithi09025264-37:head@112'211, size: 3327786, copy_subset: [0~3327786], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3327786, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:30:16.255 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:06.903352+0000", 2023-01-15T14:30:16.255 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.8005046910000004, 2023-01-15T14:30:16.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20099204800000001, 2023-01-15T14:30:16.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.256 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:16.256 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903352+0000", 2023-01-15T14:30:16.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903352+0000", 2023-01-15T14:30:16.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903350+0000", 2023-01-15T14:30:16.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:16.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903498+0000", 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000147918 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903537+0000", 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8983000000000003e-05 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903564+0000", 2023-01-15T14:30:16.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7253000000000001e-05 2023-01-15T14:30:16.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.103937+0000", 2023-01-15T14:30:16.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200373464 2023-01-15T14:30:16.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.103953+0000", 2023-01-15T14:30:16.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5243e-05 2023-01-15T14:30:16.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.104344+0000", 2023-01-15T14:30:16.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039117399999999999 2023-01-15T14:30:16.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.260 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.260 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(3.9 RELEASE e121)", 2023-01-15T14:30:16.260 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:08.681807+0000", 2023-01-15T14:30:16.261 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.0220493079999997, 2023-01-15T14:30:16.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20186040499999999, 2023-01-15T14:30:16.261 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.261 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:16.261 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.681807+0000", 2023-01-15T14:30:16.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.681807+0000", 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.681805+0000", 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.681816+0000", 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0523000000000001e-05 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.681847+0000", 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0818999999999997e-05 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.781924+0000", 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100077638 2023-01-15T14:30:16.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:16.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.883538+0000", 2023-01-15T14:30:16.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10161368899999999 2023-01-15T14:30:16.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:16.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.883574+0000", 2023-01-15T14:30:16.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6273999999999999e-05 2023-01-15T14:30:16.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:16.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.883667+0000", 2023-01-15T14:30:16.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3119000000000001e-05 2023-01-15T14:30:16.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.265 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.266 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.266 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:16.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:30:16.267 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:15.716+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:16.268 INFO:tasks.rados.rados.0.smithi090.stdout:525: finishing copy_from to smithi09025264-25 2023-01-15T14:30:16.268 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 59 (ObjNum 128 snap 23 seq_num 128) dirty exists 2023-01-15T14:30:16.268 INFO:tasks.rados.rados.0.smithi090.stdout:525: finishing copy_from racing read to smithi09025264-25 2023-01-15T14:30:16.268 INFO:tasks.rados.rados.0.smithi090.stdout:525: done (0 left) 2023-01-15T14:30:16.268 INFO:tasks.rados.rados.0.smithi090.stdout:526: done (0 left) 2023-01-15T14:30:16.268 INFO:tasks.rados.rados.0.smithi090.stdout:527: snap_remove snap 49 2023-01-15T14:30:16.269 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:16.264+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:16.348 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:30:16.384 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:16.384 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:16.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.385 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGRecoveryDeleteReply(3.2 e128,126 3:47dfe935:::smithi09025264-29:47,103'85)", 2023-01-15T14:30:16.385 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:16.346881+0000", 2023-01-15T14:30:16.385 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.036147317999999998, 2023-01-15T14:30:16.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.036189458000000001, 2023-01-15T14:30:16.385 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:16.385 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "no flag points reached", 2023-01-15T14:30:16.385 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:16.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:16.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:16.346881+0000", 2023-01-15T14:30:16.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:16.346881+0000", 2023-01-15T14:30:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:16.346880+0000", 2023-01-15T14:30:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:16.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:16.346894+0000", 2023-01-15T14:30:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.376e-05 2023-01-15T14:30:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:16.346925+0000", 2023-01-15T14:30:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1055e-05 2023-01-15T14:30:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:16.388 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:30:16.389 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:30:16.389 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:16.389 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:16.380+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:16.395 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:30:16.470 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:16.468+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:16.539 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:16.552 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:16.561 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:16.561 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:16.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:30:16.572 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:16.567+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:16.574 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:16.601 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:16.744 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:16.744 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:16.745 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:30:16.745 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:30:16.745 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:30:16.745 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:30:16.745 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:30:16.745 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:30:16.745 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:30:16.746 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:30:16.749 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:16.749 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:16.749 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:30:16.750 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:16.739+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:16.760 INFO:tasks.rados.rados.0.smithi090.stdout:527: done (0 left) 2023-01-15T14:30:16.760 INFO:tasks.rados.rados.0.smithi090.stdout:528: read oid 7 snap -1 2023-01-15T14:30:16.760 INFO:tasks.rados.rados.0.smithi090.stdout:528: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:30:16.760 INFO:tasks.rados.rados.0.smithi090.stdout:529: rollback oid 4 current snap is 50 2023-01-15T14:30:16.760 INFO:tasks.rados.rados.0.smithi090.stdout:529: done (1 left) 2023-01-15T14:30:16.760 INFO:tasks.rados.rados.0.smithi090.stdout:530: read oid 39 snap -1 2023-01-15T14:30:16.761 INFO:tasks.rados.rados.0.smithi090.stdout:530: expect (ObjNum 83 snap 11 seq_num 83) 2023-01-15T14:30:16.761 INFO:tasks.rados.rados.0.smithi090.stdout:531: delete oid 43 current snap is 50 2023-01-15T14:30:16.828 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:30:17.154 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:17.154 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:17.154 INFO:tasks.rados.rados.0.smithi090.stdout:531: done (2 left) 2023-01-15T14:30:17.155 INFO:tasks.rados.rados.0.smithi090.stdout:532: rollback oid 10 current snap is 50 2023-01-15T14:30:17.155 INFO:tasks.rados.rados.0.smithi090.stdout:532: done (2 left) 2023-01-15T14:30:17.155 INFO:tasks.rados.rados.0.smithi090.stdout:533: write oid 25 current snap is 50 2023-01-15T14:30:17.155 INFO:tasks.rados.rados.0.smithi090.stdout:533: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2023-01-15T14:30:17.155 INFO:tasks.rados.rados.0.smithi090.stdout:533: writing smithi09025264-25 from 463926 to 926092 tid 1 2023-01-15T14:30:17.155 INFO:tasks.rados.rados.0.smithi090.stdout:533: writing smithi09025264-25 from 1723898 to 2160289 tid 2 2023-01-15T14:30:17.155 INFO:tasks.rados.rados.0.smithi090.stdout:533: writing smithi09025264-25 from 2603977 to 3329864 tid 3 2023-01-15T14:30:17.155 INFO:tasks.rados.rados.0.smithi090.stdout:534: copy_from oid 46 from oid 10 current snap is 50 2023-01-15T14:30:17.155 INFO:tasks.rados.rados.0.smithi090.stdout:528: expect (ObjNum 128 snap 23 seq_num 128) 2023-01-15T14:30:17.155 INFO:tasks.rados.rados.0.smithi090.stdout:530: expect (ObjNum 83 snap 11 seq_num 83) 2023-01-15T14:30:17.156 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:30:17.156 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:17.148+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:17.215 INFO:tasks.rados.rados.0.smithi090.stdout:528: done (3 left) 2023-01-15T14:30:17.557 INFO:tasks.rados.rados.0.smithi090.stdout:530: done (2 left) 2023-01-15T14:30:17.557 INFO:tasks.rados.rados.0.smithi090.stdout:535: write oid 26 current snap is 50 2023-01-15T14:30:17.557 INFO:tasks.rados.rados.0.smithi090.stdout:535: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2023-01-15T14:30:17.561 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:17.561 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:17.562 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:30:17.565 INFO:tasks.rados.rados.0.smithi090.stdout:535: writing smithi09025264-26 from 480733 to 1018148 tid 1 2023-01-15T14:30:17.565 INFO:tasks.rados.rados.0.smithi090.stdout:535: writing smithi09025264-26 from 1782380 to 2478782 tid 2 2023-01-15T14:30:17.565 INFO:tasks.rados.rados.0.smithi090.stdout:535: writing smithi09025264-26 from 3031400 to 3568559 tid 3 2023-01-15T14:30:17.761 INFO:tasks.rados.rados.0.smithi090.stdout:536: delete oid 5 current snap is 50 2023-01-15T14:30:17.761 INFO:tasks.rados.rados.0.smithi090.stdout:534: finishing copy_from to smithi09025264-46 2023-01-15T14:30:17.761 INFO:tasks.rados.rados.0.smithi090.stdout:534: got expected ENOENT (src dne) 2023-01-15T14:30:17.761 INFO:tasks.rados.rados.0.smithi090.stdout:534: finishing copy_from racing read to smithi09025264-46 2023-01-15T14:30:17.761 INFO:tasks.rados.rados.0.smithi090.stdout:533: finishing write tid 1 to smithi09025264-25 2023-01-15T14:30:17.761 INFO:tasks.rados.rados.0.smithi090.stdout:533: finishing write tid 2 to smithi09025264-25 2023-01-15T14:30:17.761 INFO:tasks.rados.rados.0.smithi090.stdout:533: finishing write tid 3 to smithi09025264-25 2023-01-15T14:30:17.761 INFO:tasks.rados.rados.0.smithi090.stdout:533: finishing write tid 4 to smithi09025264-25 2023-01-15T14:30:17.761 INFO:tasks.rados.rados.0.smithi090.stdout:533: finishing write tid 5 to smithi09025264-25 2023-01-15T14:30:17.762 INFO:tasks.rados.rados.0.smithi090.stdout:533: oid 25 updating version 0 to 153 2023-01-15T14:30:17.941 INFO:tasks.rados.rados.0.smithi090.stdout:533: oid 25 version 153 is already newer than 152 2023-01-15T14:30:17.941 INFO:tasks.rados.rados.0.smithi090.stdout:533: oid 25 updating version 153 to 154 2023-01-15T14:30:17.942 INFO:tasks.rados.rados.0.smithi090.stdout:533: oid 25 updating version 154 to 155 2023-01-15T14:30:17.942 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 155 (ObjNum 218 snap 50 seq_num 218) dirty exists 2023-01-15T14:30:17.942 INFO:tasks.rados.rados.0.smithi090.stdout:533: left oid 25 (ObjNum 218 snap 50 seq_num 218) 2023-01-15T14:30:17.942 INFO:tasks.rados.rados.0.smithi090.stdout:533: done (3 left) 2023-01-15T14:30:17.942 INFO:tasks.rados.rados.0.smithi090.stdout:534: done (2 left) 2023-01-15T14:30:17.942 INFO:tasks.rados.rados.0.smithi090.stdout:536: done (1 left) 2023-01-15T14:30:17.944 INFO:tasks.rados.rados.0.smithi090.stdout:537: read oid 32 snap -1 2023-01-15T14:30:17.944 INFO:tasks.rados.rados.0.smithi090.stdout:537: expect (ObjNum 178 snap 38 seq_num 178) 2023-01-15T14:30:17.944 INFO:tasks.rados.rados.0.smithi090.stdout:538: write oid 17 current snap is 50 2023-01-15T14:30:17.944 INFO:tasks.rados.rados.0.smithi090.stdout:538: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2023-01-15T14:30:17.945 INFO:tasks.rados.rados.0.smithi090.stdout:538: writing smithi09025264-17 from 514347 to 918613 tid 1 2023-01-15T14:30:17.945 INFO:tasks.rados.rados.0.smithi090.stdout:538: writing smithi09025264-17 from 1499344 to 2199415 tid 2 2023-01-15T14:30:17.945 INFO:tasks.rados.rados.0.smithi090.stdout:538: writing smithi09025264-17 from 2686246 to 3362302 tid 3 2023-01-15T14:30:17.945 INFO:tasks.rados.rados.0.smithi090.stdout:539: snap_create 2023-01-15T14:30:17.945 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:30:17.945 INFO:tasks.rados.rados.0.smithi090.stdout:537: expect (ObjNum 178 snap 38 seq_num 178) 2023-01-15T14:30:17.946 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:17.556+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:17.947 INFO:tasks.rados.rados.0.smithi090.stdout:535: finishing write tid 1 to smithi09025264-26 2023-01-15T14:30:17.948 INFO:tasks.rados.rados.0.smithi090.stdout:535: finishing write tid 2 to smithi09025264-26 2023-01-15T14:30:17.948 INFO:tasks.rados.rados.0.smithi090.stdout:535: finishing write tid 3 to smithi09025264-26 2023-01-15T14:30:17.948 INFO:tasks.rados.rados.0.smithi090.stdout:535: finishing write tid 4 to smithi09025264-26 2023-01-15T14:30:17.948 INFO:tasks.rados.rados.0.smithi090.stdout:535: finishing write tid 5 to smithi09025264-26 2023-01-15T14:30:17.948 INFO:tasks.rados.rados.0.smithi090.stdout:535: oid 26 updating version 0 to 218 2023-01-15T14:30:17.948 INFO:tasks.rados.rados.0.smithi090.stdout:535: oid 26 updating version 218 to 220 2023-01-15T14:30:17.948 INFO:tasks.rados.rados.0.smithi090.stdout:535: oid 26 version 220 is already newer than 219 2023-01-15T14:30:17.948 INFO:tasks.rados.rados.0.smithi090.stdout:535: oid 26 version 220 is already newer than 217 2023-01-15T14:30:17.948 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 220 (ObjNum 219 snap 50 seq_num 219) dirty exists 2023-01-15T14:30:17.948 INFO:tasks.rados.rados.0.smithi090.stdout:535: left oid 26 (ObjNum 219 snap 50 seq_num 219) 2023-01-15T14:30:17.948 INFO:tasks.rados.rados.0.smithi090.stdout:535: done (2 left) 2023-01-15T14:30:17.949 INFO:tasks.rados.rados.0.smithi090.stdout:537: done (1 left) 2023-01-15T14:30:17.949 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:17.949 INFO:tasks.rados.rados.0.smithi090.stdout:538: finishing write tid 1 to smithi09025264-17 2023-01-15T14:30:17.949 INFO:tasks.rados.rados.0.smithi090.stdout:538: finishing write tid 2 to smithi09025264-17 2023-01-15T14:30:17.952 INFO:tasks.rados.rados.0.smithi090.stdout:538: finishing write tid 3 to smithi09025264-17 2023-01-15T14:30:17.952 INFO:tasks.rados.rados.0.smithi090.stdout:538: finishing write tid 4 to smithi09025264-17 2023-01-15T14:30:17.952 INFO:tasks.rados.rados.0.smithi090.stdout:538: finishing write tid 5 to smithi09025264-17 2023-01-15T14:30:17.952 INFO:tasks.rados.rados.0.smithi090.stdout:538: oid 17 updating version 0 to 231 2023-01-15T14:30:17.952 INFO:tasks.rados.rados.0.smithi090.stdout:538: oid 17 updating version 231 to 232 2023-01-15T14:30:17.952 INFO:tasks.rados.rados.0.smithi090.stdout:538: oid 17 updating version 232 to 233 2023-01-15T14:30:17.953 INFO:tasks.rados.rados.0.smithi090.stdout:538: oid 17 updating version 233 to 234 2023-01-15T14:30:17.953 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 234 (ObjNum 221 snap 50 seq_num 221) dirty exists 2023-01-15T14:30:17.953 INFO:tasks.rados.rados.0.smithi090.stdout:538: left oid 17 (ObjNum 221 snap 50 seq_num 221) 2023-01-15T14:30:17.953 INFO:tasks.rados.rados.0.smithi090.stdout:538: done (0 left) 2023-01-15T14:30:17.954 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:30:17.954 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:30:17.954 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2023-01-15T14:30:17.954 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-01-15T14:30:17.954 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-01-15T14:30:17.954 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2023-01-15T14:30:17.956 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:17.956 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:17.957 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:30:17.957 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:17.951+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:17.984 INFO:tasks.rados.rados.0.smithi090.stdout:539: done (0 left) 2023-01-15T14:30:18.108 INFO:tasks.rados.rados.0.smithi090.stdout:540: snap_remove snap 50 2023-01-15T14:30:18.111 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:18.111 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-01-15T14:30:18.111 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:18.112 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:18.107+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:18.112 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:30:18.211 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:18.208+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:18.272 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:18.272 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:18.272 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:30:18.311 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:18.308+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:18.432 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:18.432 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:18.433 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:30:18.433 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:18.427+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:18.533 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:18.531+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:18.588 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:18.588 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:18.588 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:30:18.634 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:18.631+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:18.793 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:30:18.794 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:18.794 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:18.795 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:30:18.795 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:18.791+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:18.895 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:18.891+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:18.951 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:18.951 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:18.952 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:18.959 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:30:18.968 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:19.112 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:19.113 INFO:tasks.rados.rados.0.smithi090.stdout:540: done (0 left) 2023-01-15T14:30:19.113 INFO:tasks.rados.rados.0.smithi090.stdout:541: write (excl) oid 28 current snap is 51 2023-01-15T14:30:19.113 INFO:tasks.rados.rados.0.smithi090.stdout:541: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2023-01-15T14:30:19.113 INFO:tasks.rados.rados.0.smithi090.stdout:541: writing smithi09025264-28 from 531154 to 1010669 tid 1 2023-01-15T14:30:19.113 INFO:tasks.rados.rados.0.smithi090.stdout:541: writing smithi09025264-28 from 1557826 to 2001555 tid 2 2023-01-15T14:30:19.113 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:19.113 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:19.114 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:19.114 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:30:19.115 INFO:tasks.rados.rados.0.smithi090.stdout:541: writing smithi09025264-28 from 2713669 to 3200997 tid 3 2023-01-15T14:30:19.116 INFO:tasks.rados.rados.0.smithi090.stdout:541: writing smithi09025264-28 from 3679295 to 3731154 tid 4 2023-01-15T14:30:19.116 INFO:tasks.rados.rados.0.smithi090.stdout:542: delete oid 25 current snap is 51 2023-01-15T14:30:19.116 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:19.123 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:19.123 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:19.123 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:30:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:30:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:30:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:30:19.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:30:19.124 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:30:19.124 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:30:19.124 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:30:19.124 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:19.108+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:19.213 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:19.211+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:19.272 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:19.272 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:30:19.272 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:30:19.272 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:19.273 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:30:19.306 INFO:tasks.rados.rados.0.smithi090.stdout:542: done (1 left) 2023-01-15T14:30:19.452 INFO:tasks.rados.rados.0.smithi090.stdout:543: copy_from oid 3 from oid 11 current snap is 51 2023-01-15T14:30:19.452 INFO:tasks.rados.rados.0.smithi090.stdout:544: rollback oid 46 current snap is 51 2023-01-15T14:30:19.452 INFO:tasks.rados.rados.0.smithi090.stdout:544: done (2 left) 2023-01-15T14:30:19.452 INFO:tasks.rados.rados.0.smithi090.stdout:545: copy_from oid 21 from oid 47 current snap is 51 2023-01-15T14:30:19.453 INFO:tasks.rados.rados.0.smithi090.stdout:546: write oid 5 current snap is 51 2023-01-15T14:30:19.453 INFO:tasks.rados.rados.0.smithi090.stdout:546: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2023-01-15T14:30:19.456 INFO:tasks.rados.rados.0.smithi090.stdout:546: writing smithi09025264-5 from 564768 to 1194781 tid 1 2023-01-15T14:30:19.607 INFO:tasks.rados.rados.0.smithi090.stdout:546: writing smithi09025264-5 from 1674790 to 2122188 tid 2 2023-01-15T14:30:19.607 INFO:tasks.rados.rados.0.smithi090.stdout:546: writing smithi09025264-5 from 2768515 to 3394740 tid 3 2023-01-15T14:30:19.607 INFO:tasks.rados.rados.0.smithi090.stdout:547: write (excl) oid 37 current snap is 51 2023-01-15T14:30:19.607 INFO:tasks.rados.rados.0.smithi090.stdout:547: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2023-01-15T14:30:19.607 INFO:tasks.rados.rados.0.smithi090.stdout:547: writing smithi09025264-37 from 581575 to 1286837 tid 1 2023-01-15T14:30:19.607 INFO:tasks.rados.rados.0.smithi090.stdout:547: writing smithi09025264-37 from 1733272 to 2440681 tid 2 2023-01-15T14:30:19.607 INFO:tasks.rados.rados.0.smithi090.stdout:547: writing smithi09025264-37 from 3195938 to 3633435 tid 3 2023-01-15T14:30:19.607 INFO:tasks.rados.rados.0.smithi090.stdout:548: write (excl) oid 23 current snap is 51 2023-01-15T14:30:19.607 INFO:tasks.rados.rados.0.smithi090.stdout:548: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2023-01-15T14:30:19.608 INFO:tasks.rados.rados.0.smithi090.stdout:548: writing smithi09025264-23 from 598382 to 1378893 tid 1 2023-01-15T14:30:19.608 INFO:tasks.rados.rados.0.smithi090.stdout:548: writing smithi09025264-23 from 2075401 to 2526468 tid 2 2023-01-15T14:30:19.608 INFO:tasks.rados.rados.0.smithi090.stdout:548: writing smithi09025264-23 from 3107008 to 3755777 tid 3 2023-01-15T14:30:19.608 INFO:tasks.rados.rados.0.smithi090.stdout:549: delete oid 6 current snap is 51 2023-01-15T14:30:19.608 INFO:tasks.rados.rados.0.smithi090.stdout:543: finishing copy_from to smithi09025264-3 2023-01-15T14:30:19.608 INFO:tasks.rados.rados.0.smithi090.stdout:543: got expected ENOENT (src dne) 2023-01-15T14:30:19.608 INFO:tasks.rados.rados.0.smithi090.stdout:543: finishing copy_from racing read to smithi09025264-3 2023-01-15T14:30:19.608 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:19.452+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:19.611 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:19.611 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:30:19.611 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:30:19.614 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:30:19.617 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:19.617 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:30:19.618 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:19.608+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:19.635 INFO:tasks.rados.rados.0.smithi090.stdout:546: finishing write tid 1 to smithi09025264-5 2023-01-15T14:30:19.815 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:19.815 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:30:19.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:30:19.815 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:30:19.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:19.815 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:70 3.f 3:fe04f54c:::smithi09025264-18:head [write 1845135~440584] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:19.815 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252308+0000", 2023-01-15T14:30:19.816 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.47241421, 2023-01-15T14:30:19.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.72689689099999999, 2023-01-15T14:30:19.816 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:19.816 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:19.816 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:19.816 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:19.816 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:19.816 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 70 2023-01-15T14:30:19.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:19.817 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:19.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:19.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:19.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252308+0000", 2023-01-15T14:30:19.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:19.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:19.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:19.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:19.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252308+0000", 2023-01-15T14:30:19.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:19.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:19.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:19.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:19.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252307+0000", 2023-01-15T14:30:19.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:19.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:19.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:19.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:19.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252613+0000", 2023-01-15T14:30:20.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000306929 2023-01-15T14:30:20.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.019 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.019 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252634+0000", 2023-01-15T14:30:20.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0276e-05 2023-01-15T14:30:20.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252647+0000", 2023-01-15T14:30:20.020 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3114e-05 2023-01-15T14:30:20.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355188+0000", 2023-01-15T14:30:20.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102540914 2023-01-15T14:30:20.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355319+0000", 2023-01-15T14:30:20.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013152699999999999 2023-01-15T14:30:20.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.657662+0000", 2023-01-15T14:30:20.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30234277199999998 2023-01-15T14:30:20.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:20.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.657813+0000", 2023-01-15T14:30:20.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015124600000000001 2023-01-15T14:30:20.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:20.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658070+0000", 2023-01-15T14:30:20.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025707600000000002 2023-01-15T14:30:20.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:20.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658459+0000", 2023-01-15T14:30:20.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038831500000000003 2023-01-15T14:30:20.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:20.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658480+0000", 2023-01-15T14:30:20.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1610000000000001e-05 2023-01-15T14:30:20.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:20.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.658736+0000", 2023-01-15T14:30:20.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025607899999999999 2023-01-15T14:30:20.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:20.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760108+0000", 2023-01-15T14:30:20.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10137170199999999 2023-01-15T14:30:20.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760226+0000", 2023-01-15T14:30:20.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011748600000000001 2023-01-15T14:30:20.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978988+0000", 2023-01-15T14:30:20.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.218762238 2023-01-15T14:30:20.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:20.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.979116+0000", 2023-01-15T14:30:20.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000128021 2023-01-15T14:30:20.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:20.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.979205+0000", 2023-01-15T14:30:20.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9248999999999999e-05 2023-01-15T14:30:20.230 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.230 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:20.230 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.230 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:71 3.f 3:fe04f54c:::smithi09025264-18:head [setxattr _header (56) in=7b,truncate 2285719] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:20.230 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:30:20.230 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.47205755100001, 2023-01-15T14:30:20.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.82719834800000003, 2023-01-15T14:30:20.231 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:20.231 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:20.231 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:20.231 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:20.231 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:20.231 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 71 2023-01-15T14:30:20.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.231 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:20.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:20.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:30:20.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:20.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:30:20.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:20.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252664+0000", 2023-01-15T14:30:20.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:20.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:20.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252675+0000", 2023-01-15T14:30:20.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1296e-05 2023-01-15T14:30:20.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.234 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.234 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252697+0000", 2023-01-15T14:30:20.234 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1874000000000001e-05 2023-01-15T14:30:20.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.234 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.234 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252708+0000", 2023-01-15T14:30:20.234 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.15e-05 2023-01-15T14:30:20.235 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.235 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.235 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.235 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355413+0000", 2023-01-15T14:30:20.235 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10270497100000001 2023-01-15T14:30:20.235 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.235 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.235 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.235 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355622+0000", 2023-01-15T14:30:20.235 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000208982 2023-01-15T14:30:20.235 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.236 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.236 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.236 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.758709+0000", 2023-01-15T14:30:20.236 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40308671800000001 2023-01-15T14:30:20.236 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.236 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.236 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:20.236 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.758958+0000", 2023-01-15T14:30:20.236 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024949 2023-01-15T14:30:20.236 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.236 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:20.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759375+0000", 2023-01-15T14:30:20.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000416208 2023-01-15T14:30:20.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:20.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759708+0000", 2023-01-15T14:30:20.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033326900000000002 2023-01-15T14:30:20.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:20.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759744+0000", 2023-01-15T14:30:20.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6477000000000003e-05 2023-01-15T14:30:20.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:20.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759822+0000", 2023-01-15T14:30:20.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7440000000000004e-05 2023-01-15T14:30:20.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760139+0000", 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316996 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760440+0000", 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000300945 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079578+0000", 2023-01-15T14:30:20.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31913862599999998 2023-01-15T14:30:20.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:20.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079709+0000", 2023-01-15T14:30:20.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013108 2023-01-15T14:30:20.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:20.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079863+0000", 2023-01-15T14:30:20.241 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000153819 2023-01-15T14:30:20.241 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.241 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:20.241 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.241 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.241 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:72 3.f 3:fe04f54c:::smithi09025264-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:20.241 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:30:20.241 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.471998214, 2023-01-15T14:30:20.242 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.82782374599999997, 2023-01-15T14:30:20.242 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:20.242 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:20.242 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:20.242 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:20.242 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:20.242 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 72 2023-01-15T14:30:20.242 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.242 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.243 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:20.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252723+0000", 2023-01-15T14:30:20.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:20.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.244 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:20.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252730+0000", 2023-01-15T14:30:20.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.703e-06 2023-01-15T14:30:20.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.244 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252747+0000", 2023-01-15T14:30:20.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6739999999999999e-05 2023-01-15T14:30:20.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.245 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.245 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252758+0000", 2023-01-15T14:30:20.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1389999999999999e-05 2023-01-15T14:30:20.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.245 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.245 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.455798+0000", 2023-01-15T14:30:20.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20303954199999999 2023-01-15T14:30:20.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.456092+0000", 2023-01-15T14:30:20.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029407199999999999 2023-01-15T14:30:20.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759852+0000", 2023-01-15T14:30:20.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.303760525 2023-01-15T14:30:20.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760038+0000", 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018584600000000001 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080005+0000", 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31996668299999997 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:20.248 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080286+0000", 2023-01-15T14:30:20.248 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028087899999999999 2023-01-15T14:30:20.248 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.248 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.248 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:20.248 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080548+0000", 2023-01-15T14:30:20.248 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026241300000000002 2023-01-15T14:30:20.248 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.248 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:20.248 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.248 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.249 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.249 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:74 3.f 3:f8edd34e:::smithi09025264-19:head [write 1903617~398909] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:20.249 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356537+0000", 2023-01-15T14:30:20.249 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.36818566700001, 2023-01-15T14:30:20.249 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.72430009100000003, 2023-01-15T14:30:20.249 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:20.249 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:20.249 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:20.249 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:20.249 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:20.249 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 74 2023-01-15T14:30:20.250 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.250 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:20.250 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.250 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:20.250 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356537+0000", 2023-01-15T14:30:20.250 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.250 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.250 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.251 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:20.428 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356537+0000", 2023-01-15T14:30:20.428 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.428 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.428 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.428 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:20.428 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356535+0000", 2023-01-15T14:30:20.429 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:20.429 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.429 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.429 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:20.429 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356854+0000", 2023-01-15T14:30:20.429 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000318979 2023-01-15T14:30:20.429 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.429 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.429 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.429 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356871+0000", 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6509999999999999e-05 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356881+0000", 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0159999999999999e-05 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457503+0000", 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100622054 2023-01-15T14:30:20.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.431 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457677+0000", 2023-01-15T14:30:20.431 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017399200000000001 2023-01-15T14:30:20.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.431 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.777230+0000", 2023-01-15T14:30:20.431 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.319552742 2023-01-15T14:30:20.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:20.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.777471+0000", 2023-01-15T14:30:20.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024115200000000001 2023-01-15T14:30:20.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:20.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.777842+0000", 2023-01-15T14:30:20.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037174199999999999 2023-01-15T14:30:20.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.433 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:20.433 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778275+0000", 2023-01-15T14:30:20.433 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043252900000000002 2023-01-15T14:30:20.433 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.433 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:20.433 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778295+0000", 2023-01-15T14:30:20.433 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9572000000000002e-05 2023-01-15T14:30:20.433 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.434 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:20.434 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778511+0000", 2023-01-15T14:30:20.434 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021633700000000001 2023-01-15T14:30:20.434 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.434 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.434 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:20.434 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:30:20.434 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060300800000000004 2023-01-15T14:30:20.434 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.434 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.435 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.435 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079412+0000", 2023-01-15T14:30:20.435 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30029809899999999 2023-01-15T14:30:20.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.438 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080661+0000", 2023-01-15T14:30:20.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001249497 2023-01-15T14:30:20.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:20.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080737+0000", 2023-01-15T14:30:20.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:30:20.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:20.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080837+0000", 2023-01-15T14:30:20.440 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9653000000000003e-05 2023-01-15T14:30:20.440 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.440 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:20.440 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.440 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:75 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:20.440 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:30:20.440 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.367825864, 2023-01-15T14:30:20.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92335155400000002, 2023-01-15T14:30:20.441 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:20.441 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:20.441 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:20.441 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:20.441 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:20.441 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 75 2023-01-15T14:30:20.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.441 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:20.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:20.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:30:20.442 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.442 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.442 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:20.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:30:20.442 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.442 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.442 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:20.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356896+0000", 2023-01-15T14:30:20.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:20.443 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:20.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356903+0000", 2023-01-15T14:30:20.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4619999999999999e-06 2023-01-15T14:30:20.443 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356917+0000", 2023-01-15T14:30:20.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4195999999999999e-05 2023-01-15T14:30:20.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.444 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.444 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356926+0000", 2023-01-15T14:30:20.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:30:20.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.444 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.444 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457743+0000", 2023-01-15T14:30:20.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100817668 2023-01-15T14:30:20.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.445 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.445 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457841+0000", 2023-01-15T14:30:20.445 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7079999999999999e-05 2023-01-15T14:30:20.445 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.445 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.445 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778474+0000", 2023-01-15T14:30:20.445 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32063301799999999 2023-01-15T14:30:20.445 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:20.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778627+0000", 2023-01-15T14:30:20.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015389599999999999 2023-01-15T14:30:20.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:20.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778815+0000", 2023-01-15T14:30:20.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018726600000000001 2023-01-15T14:30:20.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:20.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779033+0000", 2023-01-15T14:30:20.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021819999999999999 2023-01-15T14:30:20.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:20.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779051+0000", 2023-01-15T14:30:20.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7640000000000001e-05 2023-01-15T14:30:20.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:20.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:30:20.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3590999999999995e-05 2023-01-15T14:30:20.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:20.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779163+0000", 2023-01-15T14:30:20.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:30:20.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.081002+0000", 2023-01-15T14:30:20.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30183937799999999 2023-01-15T14:30:20.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280043+0000", 2023-01-15T14:30:20.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19904084699999999 2023-01-15T14:30:20.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:20.450 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280138+0000", 2023-01-15T14:30:20.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4915999999999997e-05 2023-01-15T14:30:20.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:20.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280248+0000", 2023-01-15T14:30:20.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110233 2023-01-15T14:30:20.634 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.634 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:20.634 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.634 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:76 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:20.634 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:30:20.635 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.36778425599999, 2023-01-15T14:30:20.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92420393899999997, 2023-01-15T14:30:20.635 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:20.635 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:20.635 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:20.635 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:20.635 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:20.635 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 76 2023-01-15T14:30:20.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.635 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:20.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:20.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:30:20.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:20.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:30:20.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356937+0000", 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356948+0000", 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.094e-05 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356959+0000", 2023-01-15T14:30:20.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1058e-05 2023-01-15T14:30:20.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356968+0000", 2023-01-15T14:30:20.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5339999999999999e-06 2023-01-15T14:30:20.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459524+0000", 2023-01-15T14:30:20.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102556147 2023-01-15T14:30:20.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459662+0000", 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000137756 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978621+0000", 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51895887600000001 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978870+0000", 2023-01-15T14:30:20.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024928999999999999 2023-01-15T14:30:20.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280761+0000", 2023-01-15T14:30:20.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30189113400000001 2023-01-15T14:30:20.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:20.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281023+0000", 2023-01-15T14:30:20.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026172800000000003 2023-01-15T14:30:20.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.641 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:20.641 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281142+0000", 2023-01-15T14:30:20.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000119296 2023-01-15T14:30:20.641 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.641 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:20.641 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.642 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:91 3.d 3:bc1c6b44:::smithi09025264-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:20.642 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:30:20.642 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.321295875, 2023-01-15T14:30:20.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.844937151, 2023-01-15T14:30:20.642 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:20.642 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:20.642 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:20.642 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:20.642 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:20.642 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 91 2023-01-15T14:30:20.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.643 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:20.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:20.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:30:20.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:20.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:30:20.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:20.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403425+0000", 2023-01-15T14:30:20.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:20.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:20.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403434+0000", 2023-01-15T14:30:20.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3299999999999999e-06 2023-01-15T14:30:20.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403446+0000", 2023-01-15T14:30:20.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2176e-05 2023-01-15T14:30:20.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403453+0000", 2023-01-15T14:30:20.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5830000000000003e-06 2023-01-15T14:30:20.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403654+0000", 2023-01-15T14:30:20.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020136000000000001 2023-01-15T14:30:20.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403807+0000", 2023-01-15T14:30:20.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015342599999999999 2023-01-15T14:30:20.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907836+0000", 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50402854699999999 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907934+0000", 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8472000000000002e-05 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908096+0000", 2023-01-15T14:30:20.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000162139 2023-01-15T14:30:20.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:20.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908335+0000", 2023-01-15T14:30:20.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000238891 2023-01-15T14:30:20.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:20.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908350+0000", 2023-01-15T14:30:20.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4868000000000001e-05 2023-01-15T14:30:20.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:20.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908416+0000", 2023-01-15T14:30:20.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5502999999999999e-05 2023-01-15T14:30:20.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:20.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.952370+0000", 2023-01-15T14:30:20.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.043954663999999997 2023-01-15T14:30:20.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.650 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.650 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.008749+0000", 2023-01-15T14:30:20.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.056378696999999998 2023-01-15T14:30:20.650 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.650 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.650 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.650 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248164+0000", 2023-01-15T14:30:20.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23941479500000001 2023-01-15T14:30:20.650 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.650 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.650 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:20.651 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248244+0000", 2023-01-15T14:30:20.651 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9616000000000005e-05 2023-01-15T14:30:20.651 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.651 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.651 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:20.651 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248364+0000", 2023-01-15T14:30:20.651 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120167 2023-01-15T14:30:20.651 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.651 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:20.651 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.652 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.652 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.652 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:92 3.d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:20.652 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:30:20.652 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.321260182, 2023-01-15T14:30:20.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.84526834699999998, 2023-01-15T14:30:20.652 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:20.652 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:20.652 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:20.652 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:20.653 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:20.653 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 92 2023-01-15T14:30:20.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.653 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:20.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:20.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:30:20.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403461+0000", 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403466+0000", 2023-01-15T14:30:20.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.34e-06 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403476+0000", 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8530000000000004e-06 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403492+0000", 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6646999999999998e-05 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501333+0000", 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097840420999999997 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501483+0000", 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150123 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.007528+0000", 2023-01-15T14:30:20.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50604486299999996 2023-01-15T14:30:20.657 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.657 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.657 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.657 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.007649+0000", 2023-01-15T14:30:20.657 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001217 2023-01-15T14:30:20.657 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.657 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.657 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.657 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248441+0000", 2023-01-15T14:30:20.657 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.240791267 2023-01-15T14:30:20.657 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:20.658 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248561+0000", 2023-01-15T14:30:20.658 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120516 2023-01-15T14:30:20.658 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:20.658 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248731+0000", 2023-01-15T14:30:20.658 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016937000000000001 2023-01-15T14:30:20.658 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.658 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:20.659 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.659 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.659 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.659 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:107 3.d 3:b2a32959:::smithi09025264-27:head [setxattr _header (56) in=7b,truncate 2436982] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:20.659 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.508410+0000", 2023-01-15T14:30:20.659 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.216312008, 2023-01-15T14:30:20.659 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63911129799999999, 2023-01-15T14:30:20.659 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:20.659 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:20.659 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:20.659 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:20.660 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:20.660 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 107 2023-01-15T14:30:20.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.660 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:20.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.660 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:20.660 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508410+0000", 2023-01-15T14:30:20.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.661 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:20.661 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508410+0000", 2023-01-15T14:30:20.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.661 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:20.661 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508409+0000", 2023-01-15T14:30:20.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:20.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:20.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508418+0000", 2023-01-15T14:30:20.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4830000000000005e-06 2023-01-15T14:30:20.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508436+0000", 2023-01-15T14:30:20.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7419999999999999e-05 2023-01-15T14:30:20.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508448+0000", 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2274e-05 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508766+0000", 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00031751199999999999 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508906+0000", 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014061999999999999 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906456+0000", 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39754930500000002 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906561+0000", 2023-01-15T14:30:20.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000105159 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906729+0000", 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016813900000000001 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906908+0000", 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000179049 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.906974+0000", 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6360999999999997e-05 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907147+0000", 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000172991 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:20.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.952355+0000", 2023-01-15T14:30:20.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.045207642999999999 2023-01-15T14:30:20.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.008697+0000", 2023-01-15T14:30:20.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.056342148000000002 2023-01-15T14:30:20.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.147257+0000", 2023-01-15T14:30:20.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.13856006800000001 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.147356+0000", 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9083e-05 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.147522+0000", 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016548999999999999 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.668 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:108 3.d 3:b2a32959:::smithi09025264-27:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.508466+0000", 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.216255965, 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.73959223500000004, 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:20.669 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:20.670 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 108 2023-01-15T14:30:20.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.670 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:20.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:20.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508466+0000", 2023-01-15T14:30:20.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:20.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508466+0000", 2023-01-15T14:30:20.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:20.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508465+0000", 2023-01-15T14:30:20.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:20.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:20.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508473+0000", 2023-01-15T14:30:20.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.374e-06 2023-01-15T14:30:20.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508502+0000", 2023-01-15T14:30:20.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.9709e-05 2023-01-15T14:30:20.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508513+0000", 2023-01-15T14:30:20.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0200000000000001e-05 2023-01-15T14:30:20.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.508947+0000", 2023-01-15T14:30:20.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043390699999999998 2023-01-15T14:30:20.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.509045+0000", 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8400000000000007e-05 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907035+0000", 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39798985799999997 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907172+0000", 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013714599999999999 2023-01-15T14:30:20.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.247729+0000", 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.34055681599999998 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.247863+0000", 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013370999999999999 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:20.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248059+0000", 2023-01-15T14:30:20.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019614900000000001 2023-01-15T14:30:20.676 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.676 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:20.676 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.676 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:191 3.d 3:b3f111b5:::smithi09025264-48:head [setxattr _header (56) in=7b,truncate 2789929] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:20.676 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.289388+0000", 2023-01-15T14:30:20.676 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 112.43533394400001, 2023-01-15T14:30:20.677 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67091844199999995, 2023-01-15T14:30:20.677 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:20.677 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:20.677 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:20.677 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:20.677 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:20.677 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 191 2023-01-15T14:30:20.677 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.677 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:20.677 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.677 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289388+0000", 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289388+0000", 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289387+0000", 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:20.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:20.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289396+0000", 2023-01-15T14:30:20.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6000000000000007e-06 2023-01-15T14:30:20.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.289410+0000", 2023-01-15T14:30:20.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3933e-05 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.389520+0000", 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100110488 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.552712+0000", 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.16319123799999999 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.552851+0000", 2023-01-15T14:30:20.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000139128 2023-01-15T14:30:20.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.658429+0000", 2023-01-15T14:30:20.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10557828399999999 2023-01-15T14:30:20.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.658606+0000", 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017746 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.658814+0000", 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020789199999999999 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659030+0000", 2023-01-15T14:30:20.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021556 2023-01-15T14:30:20.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:20.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659054+0000", 2023-01-15T14:30:20.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4182999999999999e-05 2023-01-15T14:30:20.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:20.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659131+0000", 2023-01-15T14:30:20.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.729e-05 2023-01-15T14:30:20.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.659442+0000", 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310575 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.817131+0000", 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.15768874399999999 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.960024+0000", 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.14289289399999999 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.960168+0000", 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144186 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.960307+0000", 2023-01-15T14:30:20.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013917000000000001 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:200 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.491107+0000", 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 112.23361520900001, 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81997309299999999, 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:20.830 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:20.831 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:20.831 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 200 2023-01-15T14:30:20.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.831 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:20.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.831 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:20.831 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491107+0000", 2023-01-15T14:30:20.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.831 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:20.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491107+0000", 2023-01-15T14:30:20.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:20.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:20.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491106+0000", 2023-01-15T14:30:20.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:20.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:20.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491113+0000", 2023-01-15T14:30:20.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9129999999999997e-06 2023-01-15T14:30:20.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:20.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491132+0000", 2023-01-15T14:30:20.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8593999999999998e-05 2023-01-15T14:30:20.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:20.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.491146+0000", 2023-01-15T14:30:20.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4036000000000001e-05 2023-01-15T14:30:20.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.692622+0000", 2023-01-15T14:30:20.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20147609 2023-01-15T14:30:20.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.692862+0000", 2023-01-15T14:30:20.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024064600000000001 2023-01-15T14:30:20.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:20.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.909332+0000", 2023-01-15T14:30:20.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.216469364 2023-01-15T14:30:20.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:20.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:20.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:20.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.909470+0000", 2023-01-15T14:30:20.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013843300000000001 2023-01-15T14:30:21.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:21.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.310780+0000", 2023-01-15T14:30:21.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.401309939 2023-01-15T14:30:21.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:21.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.310914+0000", 2023-01-15T14:30:21.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013376600000000001 2023-01-15T14:30:21.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:21.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.311080+0000", 2023-01-15T14:30:21.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000166405 2023-01-15T14:30:21.149 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:21.149 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1074 3.f 3:f87c0183:::smithi09025264-50:head [write 1614880~581041] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.754686+0000", 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 36.970036647999997, 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.78829256299999995, 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:21.150 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1074 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.754686+0000", 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.754686+0000", 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:21.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:21.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.754683+0000", 2023-01-15T14:30:21.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:21.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:21.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.755225+0000", 2023-01-15T14:30:21.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054107299999999999 2023-01-15T14:30:21.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:21.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.755258+0000", 2023-01-15T14:30:21.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3056000000000003e-05 2023-01-15T14:30:21.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:21.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.755271+0000", 2023-01-15T14:30:21.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3879999999999999e-05 2023-01-15T14:30:21.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:21.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.847741+0000", 2023-01-15T14:30:21.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.092469853000000005 2023-01-15T14:30:21.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:21.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.847751+0000", 2023-01-15T14:30:21.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7070000000000004e-06 2023-01-15T14:30:21.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:21.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.949502+0000", 2023-01-15T14:30:21.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101750668 2023-01-15T14:30:21.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:21.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.949684+0000", 2023-01-15T14:30:21.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018261300000000001 2023-01-15T14:30:21.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:21.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.949910+0000", 2023-01-15T14:30:21.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000225479 2023-01-15T14:30:21.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:21.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.950463+0000", 2023-01-15T14:30:21.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055321600000000004 2023-01-15T14:30:21.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.160 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.160 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:21.160 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.950488+0000", 2023-01-15T14:30:21.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5316e-05 2023-01-15T14:30:21.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.162 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:21.162 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.950744+0000", 2023-01-15T14:30:21.162 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025602900000000001 2023-01-15T14:30:21.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.162 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.162 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:21.162 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951524+0000", 2023-01-15T14:30:21.162 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000779588 2023-01-15T14:30:21.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.162 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.163 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:21.163 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065753+0000", 2023-01-15T14:30:21.163 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.114229573 2023-01-15T14:30:21.163 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.163 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:21.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.542850+0000", 2023-01-15T14:30:21.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.47709690100000002 2023-01-15T14:30:21.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:21.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.542918+0000", 2023-01-15T14:30:21.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7331999999999998e-05 2023-01-15T14:30:21.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:21.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.542978+0000", 2023-01-15T14:30:21.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0582999999999999e-05 2023-01-15T14:30:21.165 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:21.165 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1075 3.f 3:f87c0183:::smithi09025264-50:head [write 2785791~71399] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758416+0000", 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 36.966306873999997, 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.78485753899999999, 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:21.166 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:21.167 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1075 2023-01-15T14:30:21.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.167 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:21.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:21.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758416+0000", 2023-01-15T14:30:21.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:21.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:21.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758416+0000", 2023-01-15T14:30:21.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:21.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:21.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758413+0000", 2023-01-15T14:30:21.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:21.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.169 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:21.169 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758540+0000", 2023-01-15T14:30:21.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000126752 2023-01-15T14:30:21.169 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.169 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.169 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:21.169 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758610+0000", 2023-01-15T14:30:21.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9721000000000006e-05 2023-01-15T14:30:21.169 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.169 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.169 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:21.170 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758626+0000", 2023-01-15T14:30:21.170 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6209999999999999e-05 2023-01-15T14:30:21.170 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.170 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.170 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:21.170 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.847786+0000", 2023-01-15T14:30:21.170 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089159840000000004 2023-01-15T14:30:21.170 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.170 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.170 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:21.171 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.847792+0000", 2023-01-15T14:30:21.171 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5270000000000004e-06 2023-01-15T14:30:21.171 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.171 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.171 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:21.171 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.950627+0000", 2023-01-15T14:30:21.171 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102834561 2023-01-15T14:30:21.171 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.171 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.171 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:21.172 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.950798+0000", 2023-01-15T14:30:21.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017075700000000001 2023-01-15T14:30:21.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:21.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951014+0000", 2023-01-15T14:30:21.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021627899999999999 2023-01-15T14:30:21.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:21.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951443+0000", 2023-01-15T14:30:21.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042904400000000001 2023-01-15T14:30:21.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:21.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951458+0000", 2023-01-15T14:30:21.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5079000000000001e-05 2023-01-15T14:30:21.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:21.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951750+0000", 2023-01-15T14:30:21.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000291515 2023-01-15T14:30:21.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:21.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952258+0000", 2023-01-15T14:30:21.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00050820100000000001 2023-01-15T14:30:21.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:21.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065797+0000", 2023-01-15T14:30:21.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.113538917 2023-01-15T14:30:21.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:21.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.543119+0000", 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.47732227999999999 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.543207+0000", 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8127999999999994e-05 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.543273+0000", 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5845000000000004e-05 2023-01-15T14:30:21.333 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:21.334 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:21.334 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:21.490 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.490 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.490 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1076 3.f 3:f87c0183:::smithi09025264-50:head [setxattr _header (57) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:21.490 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:30:21.490 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 36.966075500999999, 2023-01-15T14:30:21.491 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.88473693200000003, 2023-01-15T14:30:21.491 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:21.491 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:21.491 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:21.492 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:21.492 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:21.711 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1076 2023-01-15T14:30:21.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.711 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:21.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:21.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:30:21.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:21.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:21.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:30:21.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:21.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:21.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758646+0000", 2023-01-15T14:30:21.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:21.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:21.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758653+0000", 2023-01-15T14:30:21.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8029999999999997e-06 2023-01-15T14:30:21.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:21.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758670+0000", 2023-01-15T14:30:21.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6158999999999999e-05 2023-01-15T14:30:21.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:21.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758677+0000", 2023-01-15T14:30:21.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9840000000000004e-06 2023-01-15T14:30:21.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.718 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:21.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848049+0000", 2023-01-15T14:30:21.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089372163000000004 2023-01-15T14:30:21.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:21.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848056+0000", 2023-01-15T14:30:21.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0360000000000001e-06 2023-01-15T14:30:21.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:21.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951562+0000", 2023-01-15T14:30:21.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10350670100000001 2023-01-15T14:30:21.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:21.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951726+0000", 2023-01-15T14:30:21.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000163311 2023-01-15T14:30:21.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:21.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951958+0000", 2023-01-15T14:30:21.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000232525 2023-01-15T14:30:21.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:21.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952168+0000", 2023-01-15T14:30:21.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020949799999999999 2023-01-15T14:30:21.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:21.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952186+0000", 2023-01-15T14:30:21.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7714e-05 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952254+0000", 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7998999999999997e-05 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952278+0000", 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4909000000000002e-05 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:21.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065836+0000", 2023-01-15T14:30:21.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.113557081 2023-01-15T14:30:21.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:21.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643196+0000", 2023-01-15T14:30:21.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.57736012999999997 2023-01-15T14:30:21.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643278+0000", 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2671999999999997e-05 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643384+0000", 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010551400000000001 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.877 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1077 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:21.877 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:30:21.877 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 36.966036682000002, 2023-01-15T14:30:21.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.98557123499999999, 2023-01-15T14:30:21.877 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:21.877 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:21.877 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:21.877 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:21.877 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:21.877 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1077 2023-01-15T14:30:21.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.878 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:21.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:21.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:30:21.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:21.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:21.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:30:21.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:21.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.879 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.879 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:21.879 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758685+0000", 2023-01-15T14:30:21.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:21.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.879 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.879 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:21.879 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758689+0000", 2023-01-15T14:30:21.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2760000000000002e-06 2023-01-15T14:30:21.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:21.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758704+0000", 2023-01-15T14:30:21.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4187e-05 2023-01-15T14:30:21.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:21.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758710+0000", 2023-01-15T14:30:21.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4330000000000001e-06 2023-01-15T14:30:21.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:21.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848086+0000", 2023-01-15T14:30:21.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089376466000000002 2023-01-15T14:30:21.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:21.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848093+0000", 2023-01-15T14:30:21.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4269999999999996e-06 2023-01-15T14:30:21.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:21.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952346+0000", 2023-01-15T14:30:21.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104252758 2023-01-15T14:30:21.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:21.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952427+0000", 2023-01-15T14:30:21.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1786000000000006e-05 2023-01-15T14:30:21.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:21.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743602+0000", 2023-01-15T14:30:21.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.79117428099999998 2023-01-15T14:30:21.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:21.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743892+0000", 2023-01-15T14:30:21.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029067200000000001 2023-01-15T14:30:21.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:21.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.744257+0000", 2023-01-15T14:30:21.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000364619 2023-01-15T14:30:21.885 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:21.885 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:21.885 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:21.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.885 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1087 3.f 3:fd983206:::smithi09025264-7:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:21.885 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:30:21.885 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 36.966002199999998, 2023-01-15T14:30:21.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0859964360000001, 2023-01-15T14:30:21.886 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:21.886 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:21.886 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:21.886 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:21.886 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:21.886 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1087 2023-01-15T14:30:21.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.886 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:21.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:21.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:30:21.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:21.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:21.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:30:21.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:21.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:21.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:30:21.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:21.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:21.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758724+0000", 2023-01-15T14:30:21.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7740000000000004e-06 2023-01-15T14:30:21.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:21.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758738+0000", 2023-01-15T14:30:21.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.33e-05 2023-01-15T14:30:21.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:21.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758744+0000", 2023-01-15T14:30:21.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2859999999999996e-06 2023-01-15T14:30:21.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:21.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848121+0000", 2023-01-15T14:30:21.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089377097000000003 2023-01-15T14:30:21.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:21.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:21.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:21.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848127+0000", 2023-01-15T14:30:21.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3500000000000002e-06 2023-01-15T14:30:22.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.037 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:22.037 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952454+0000", 2023-01-15T14:30:22.037 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104326167 2023-01-15T14:30:22.037 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.037 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:22.037 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952520+0000", 2023-01-15T14:30:22.037 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6166000000000003e-05 2023-01-15T14:30:22.037 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.038 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.038 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:22.038 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.063003+0000", 2023-01-15T14:30:22.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.110482916 2023-01-15T14:30:22.038 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.038 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.038 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:22.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065265+0000", 2023-01-15T14:30:22.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022626370000000001 2023-01-15T14:30:22.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:22.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065291+0000", 2023-01-15T14:30:22.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6095e-05 2023-01-15T14:30:22.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:22.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.066153+0000", 2023-01-15T14:30:22.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086127100000000004 2023-01-15T14:30:22.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:22.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166533+0000", 2023-01-15T14:30:22.044 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100380177 2023-01-15T14:30:22.044 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.044 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.044 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:22.044 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166646+0000", 2023-01-15T14:30:22.044 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112717 2023-01-15T14:30:22.044 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.045 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:22.045 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844498+0000", 2023-01-15T14:30:22.045 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67785237399999998 2023-01-15T14:30:22.045 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.045 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:22.045 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844605+0000", 2023-01-15T14:30:22.045 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106691 2023-01-15T14:30:22.045 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.046 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:22.046 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844717+0000", 2023-01-15T14:30:22.046 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011203499999999999 2023-01-15T14:30:22.046 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:22.046 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:22.046 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:22.046 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.046 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.046 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1088 3.f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:22.047 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:30:22.047 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 36.965946533999997, 2023-01-15T14:30:22.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.086312696, 2023-01-15T14:30:22.047 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:22.047 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:22.047 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:22.047 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:22.047 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:22.047 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1088 2023-01-15T14:30:22.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.047 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:22.048 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.048 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:22.048 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:30:22.048 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:22.048 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.048 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.048 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:22.048 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:30:22.048 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:22.048 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.048 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.049 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:22.049 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758775+0000", 2023-01-15T14:30:22.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:22.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.049 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:22.049 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758786+0000", 2023-01-15T14:30:22.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.076e-05 2023-01-15T14:30:22.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.049 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:22.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758809+0000", 2023-01-15T14:30:22.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3706999999999999e-05 2023-01-15T14:30:22.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.050 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:22.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758821+0000", 2023-01-15T14:30:22.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1976e-05 2023-01-15T14:30:22.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.050 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:22.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878534+0000", 2023-01-15T14:30:22.051 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.119712966 2023-01-15T14:30:22.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:22.051 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878540+0000", 2023-01-15T14:30:22.051 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0680000000000002e-06 2023-01-15T14:30:22.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:22.051 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952693+0000", 2023-01-15T14:30:22.051 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.074153067000000003 2023-01-15T14:30:22.052 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.052 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.052 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:22.052 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952735+0000", 2023-01-15T14:30:22.052 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1792999999999998e-05 2023-01-15T14:30:22.052 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.052 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.052 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:22.052 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065518+0000", 2023-01-15T14:30:22.052 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.11278334499999999 2023-01-15T14:30:22.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:22.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065687+0000", 2023-01-15T14:30:22.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016807700000000001 2023-01-15T14:30:22.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:22.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844786+0000", 2023-01-15T14:30:22.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.77909980300000004 2023-01-15T14:30:22.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:22.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844944+0000", 2023-01-15T14:30:22.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157424 2023-01-15T14:30:22.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:22.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.845089+0000", 2023-01-15T14:30:22.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144814 2023-01-15T14:30:22.054 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:22.055 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:22.055 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:22.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.055 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1280 3.d 3:bd982ca3:::smithi09025264-38:head [setxattr _header (57) in=7b,truncate 3277365] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:22.055 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:57.917587+0000", 2023-01-15T14:30:22.055 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 21.807135561999999, 2023-01-15T14:30:22.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67358660299999995, 2023-01-15T14:30:22.055 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:22.056 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:22.056 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:22.056 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:22.056 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:22.056 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1280 2023-01-15T14:30:22.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.056 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:22.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:22.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917587+0000", 2023-01-15T14:30:22.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:22.057 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.057 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:22.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917587+0000", 2023-01-15T14:30:22.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:22.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.058 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.058 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:22.058 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917585+0000", 2023-01-15T14:30:22.058 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:22.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.058 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.058 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:22.058 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917597+0000", 2023-01-15T14:30:22.059 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1514e-05 2023-01-15T14:30:22.059 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.059 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.059 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:22.059 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917614+0000", 2023-01-15T14:30:22.059 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6813999999999999e-05 2023-01-15T14:30:22.059 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.059 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.059 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:22.059 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917624+0000", 2023-01-15T14:30:22.060 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0648e-05 2023-01-15T14:30:22.060 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.060 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.060 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:22.060 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.110787+0000", 2023-01-15T14:30:22.060 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19316277800000001 2023-01-15T14:30:22.060 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.060 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.060 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:22.060 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.110895+0000", 2023-01-15T14:30:22.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107404 2023-01-15T14:30:22.061 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.061 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.061 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:22.061 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.389463+0000", 2023-01-15T14:30:22.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.27856870099999997 2023-01-15T14:30:22.061 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:22.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.389647+0000", 2023-01-15T14:30:22.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018348299999999999 2023-01-15T14:30:22.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:30:22.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.389910+0000", 2023-01-15T14:30:22.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026346099999999999 2023-01-15T14:30:22.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:22.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.390321+0000", 2023-01-15T14:30:22.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041041899999999997 2023-01-15T14:30:22.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:22.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.390342+0000", 2023-01-15T14:30:22.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1886999999999999e-05 2023-01-15T14:30:22.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:22.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.390436+0000", 2023-01-15T14:30:22.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3210999999999999e-05 2023-01-15T14:30:22.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:22.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.390905+0000", 2023-01-15T14:30:22.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046920799999999998 2023-01-15T14:30:22.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:22.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.491531+0000", 2023-01-15T14:30:22.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100626021 2023-01-15T14:30:22.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:22.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.590824+0000", 2023-01-15T14:30:22.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099292907999999999 2023-01-15T14:30:22.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.590994+0000", 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017012700000000001 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.591173+0000", 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017946999999999999 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:22.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.136 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1281 3.d 3:bd982ca3:::smithi09025264-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:22.136 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:57.917650+0000", 2023-01-15T14:30:22.136 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 21.807072817000002, 2023-01-15T14:30:22.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67396394199999998, 2023-01-15T14:30:22.136 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:22.136 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:22.136 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:22.136 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:22.136 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1281 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917650+0000", 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917650+0000", 2023-01-15T14:30:22.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:22.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:22.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917648+0000", 2023-01-15T14:30:22.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:22.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:22.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917655+0000", 2023-01-15T14:30:22.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4570000000000002e-06 2023-01-15T14:30:22.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917677+0000", 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2425e-05 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:57.917688+0000", 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0339999999999999e-05 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:22.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.211087+0000", 2023-01-15T14:30:22.140 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.29339880600000001 2023-01-15T14:30:22.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.140 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:22.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.211362+0000", 2023-01-15T14:30:22.140 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027563400000000002 2023-01-15T14:30:22.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.140 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:22.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.390690+0000", 2023-01-15T14:30:22.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17932827700000001 2023-01-15T14:30:22.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:22.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.390847+0000", 2023-01-15T14:30:22.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015621 2023-01-15T14:30:22.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:22.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.591265+0000", 2023-01-15T14:30:22.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20041834 2023-01-15T14:30:22.142 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.199 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.199 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:22.199 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.591430+0000", 2023-01-15T14:30:22.199 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016505400000000001 2023-01-15T14:30:22.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:22.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:22.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:22.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:58.591614+0000", 2023-01-15T14:30:22.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000183499 2023-01-15T14:30:22.202 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:22.202 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:22.203 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:22.203 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:22.203 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:22.203 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:22.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:30:22.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:30:22.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:30:22.205 INFO:tasks.rados.rados.0.smithi090.stdout:546: finishing write tid 2 to smithi09025264-5 2023-01-15T14:30:22.206 INFO:tasks.rados.rados.0.smithi090.stdout:546: finishing write tid 3 to smithi09025264-5 2023-01-15T14:30:22.206 INFO:tasks.rados.rados.0.smithi090.stdout:543: done (6 left) 2023-01-15T14:30:22.206 INFO:tasks.rados.rados.0.smithi090.stdout:549: done (5 left) 2023-01-15T14:30:22.206 INFO:tasks.rados.rados.0.smithi090.stdout:550: copy_from oid 44 from oid 10 current snap is 51 2023-01-15T14:30:22.206 INFO:tasks.rados.rados.0.smithi090.stdout:551: read oid 9 snap -1 2023-01-15T14:30:22.206 INFO:tasks.rados.rados.0.smithi090.stdout:551: expect deleted 2023-01-15T14:30:22.206 INFO:tasks.rados.rados.0.smithi090.stdout:552: read oid 32 snap -1 2023-01-15T14:30:22.206 INFO:tasks.rados.rados.0.smithi090.stdout:552: expect (ObjNum 178 snap 38 seq_num 178) 2023-01-15T14:30:22.206 INFO:tasks.rados.rados.0.smithi090.stdout:553: snap_create 2023-01-15T14:30:22.206 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:30:22.206 INFO:tasks.rados.rados.0.smithi090.stdout:546: finishing write tid 4 to smithi09025264-5 2023-01-15T14:30:22.207 INFO:tasks.rados.rados.0.smithi090.stdout:546: finishing write tid 5 to smithi09025264-5 2023-01-15T14:30:22.207 INFO:tasks.rados.rados.0.smithi090.stdout:546: oid 5 updating version 0 to 186 2023-01-15T14:30:22.207 INFO:tasks.rados.rados.0.smithi090.stdout:546: oid 5 updating version 186 to 188 2023-01-15T14:30:22.207 INFO:tasks.rados.rados.0.smithi090.stdout:546: oid 5 version 188 is already newer than 187 2023-01-15T14:30:22.207 INFO:tasks.rados.rados.0.smithi090.stdout:546: oid 5 version 188 is already newer than 185 2023-01-15T14:30:22.361 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 188 (ObjNum 224 snap 51 seq_num 224) dirty exists 2023-01-15T14:30:22.361 INFO:tasks.rados.rados.0.smithi090.stdout:546: left oid 5 (ObjNum 224 snap 51 seq_num 224) 2023-01-15T14:30:22.361 INFO:tasks.rados.rados.0.smithi090.stdout:546: done (7 left) 2023-01-15T14:30:22.361 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:30:22.362 INFO:tasks.rados.rados.0.smithi090.stdout:551: done (6 left) 2023-01-15T14:30:22.362 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:30:22.362 INFO:tasks.rados.rados.0.smithi090.stdout:550: finishing copy_from to smithi09025264-44 2023-01-15T14:30:22.362 INFO:tasks.rados.rados.0.smithi090.stdout:550: got expected ENOENT (src dne) 2023-01-15T14:30:22.362 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:19.808+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:22.363 INFO:tasks.rados.rados.0.smithi090.stdout:550: finishing copy_from racing read to smithi09025264-44 2023-01-15T14:30:22.363 INFO:tasks.rados.rados.0.smithi090.stdout:550: done (5 left) 2023-01-15T14:30:22.363 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:30:22.363 INFO:tasks.rados.rados.0.smithi090.stdout:541: finishing write tid 1 to smithi09025264-28 2023-01-15T14:30:22.364 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:22.200+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:22.520 INFO:tasks.rados.rados.0.smithi090.stdout:541: finishing write tid 2 to smithi09025264-28 2023-01-15T14:30:22.685 INFO:tasks.rados.rados.0.smithi090.stdout:541: finishing write tid 3 to smithi09025264-28 2023-01-15T14:30:22.685 INFO:tasks.rados.rados.0.smithi090.stdout:541: finishing write tid 4 to smithi09025264-28 2023-01-15T14:30:22.685 INFO:tasks.rados.rados.0.smithi090.stdout:541: finishing write tid 5 to smithi09025264-28 2023-01-15T14:30:22.686 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:22.686 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:30:22.687 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:22.687 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:30:22.688 INFO:tasks.rados.rados.0.smithi090.stdout:541: finishing write tid 6 to smithi09025264-28 2023-01-15T14:30:22.689 INFO:tasks.rados.rados.0.smithi090.stdout:541: oid 28 updating version 0 to 218 2023-01-15T14:30:22.689 INFO:tasks.rados.rados.0.smithi090.stdout:541: oid 28 version 218 is already newer than 217 2023-01-15T14:30:22.689 INFO:tasks.rados.rados.0.smithi090.stdout:541: oid 28 version 218 is already newer than 216 2023-01-15T14:30:22.689 INFO:tasks.rados.rados.0.smithi090.stdout:541: oid 28 version 218 is already newer than 215 2023-01-15T14:30:22.689 INFO:tasks.rados.rados.0.smithi090.stdout:541: oid 28 version 218 is already newer than 214 2023-01-15T14:30:22.690 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 218 (ObjNum 222 snap 51 seq_num 222) dirty exists 2023-01-15T14:30:22.690 INFO:tasks.rados.rados.0.smithi090.stdout:541: left oid 28 (ObjNum 222 snap 51 seq_num 222) 2023-01-15T14:30:22.690 INFO:tasks.rados.rados.0.smithi090.stdout:541: done (4 left) 2023-01-15T14:30:22.690 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:30:22.692 INFO:tasks.rados.rados.0.smithi090.stdout:545: finishing copy_from to smithi09025264-21 2023-01-15T14:30:22.692 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 193 (ObjNum 183 snap 40 seq_num 183) dirty exists 2023-01-15T14:30:22.692 INFO:tasks.rados.rados.0.smithi090.stdout:552: expect (ObjNum 178 snap 38 seq_num 178) 2023-01-15T14:30:22.692 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:22.515+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:22.693 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:22.696 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:22.697 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:22.692+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:22.727 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:22.846 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:22.847 INFO:tasks.rados.rados.0.smithi090.stdout:545: finishing copy_from racing read to smithi09025264-21 2023-01-15T14:30:22.847 INFO:tasks.rados.rados.0.smithi090.stdout:547: finishing write tid 1 to smithi09025264-37 2023-01-15T14:30:22.847 INFO:tasks.rados.rados.0.smithi090.stdout:545: done (3 left) 2023-01-15T14:30:22.847 INFO:tasks.rados.rados.0.smithi090.stdout:552: done (2 left) 2023-01-15T14:30:22.848 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:22.849 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:22.849 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:22.850 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:30:22.850 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:22.844+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:22.850 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:22.878 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:22.992 INFO:tasks.rados.rados.0.smithi090.stdout:548: finishing write tid 1 to smithi09025264-23 2023-01-15T14:30:22.992 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:22.993 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:22.988+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:23.013 INFO:tasks.rados.rados.0.smithi090.stdout:547: finishing write tid 2 to smithi09025264-37 2023-01-15T14:30:23.013 INFO:tasks.rados.rados.0.smithi090.stdout:547: finishing write tid 3 to smithi09025264-37 2023-01-15T14:30:23.013 INFO:tasks.rados.rados.0.smithi090.stdout:547: finishing write tid 4 to smithi09025264-37 2023-01-15T14:30:23.013 INFO:tasks.rados.rados.0.smithi090.stdout:548: finishing write tid 2 to smithi09025264-23 2023-01-15T14:30:23.016 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:23.020 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:30:23.020 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:30:23.020 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:30:23.020 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:30:23.020 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:30:23.020 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:30:23.020 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:30:23.020 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: filestore_inject_stall = '0' (not observed, change may require restart) osd_enable_op_tracker = 'false' 2023-01-15T14:30:23.021 INFO:tasks.rados.rados.0.smithi090.stdout:547: finishing write tid 5 to smithi09025264-37 2023-01-15T14:30:23.021 INFO:tasks.rados.rados.0.smithi090.stdout:547: oid 37 updating version 0 to 221 2023-01-15T14:30:23.021 INFO:tasks.rados.rados.0.smithi090.stdout:547: oid 37 version 221 is already newer than 219 2023-01-15T14:30:23.021 INFO:tasks.rados.rados.0.smithi090.stdout:547: oid 37 updating version 221 to 222 2023-01-15T14:30:23.021 INFO:tasks.rados.rados.0.smithi090.stdout:547: oid 37 version 222 is already newer than 220 2023-01-15T14:30:23.022 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 222 (ObjNum 225 snap 51 seq_num 225) dirty exists 2023-01-15T14:30:23.022 INFO:tasks.rados.rados.0.smithi090.stdout:547: left oid 37 (ObjNum 225 snap 51 seq_num 225) 2023-01-15T14:30:23.022 INFO:tasks.rados.rados.0.smithi090.stdout:547: done (1 left) 2023-01-15T14:30:23.022 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:23.040 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:30:23.161 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:30:23.161 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:30:23.161 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-01-15T14:30:23.161 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-01-15T14:30:23.162 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:23.159+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:23.163 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:23.163 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:23.164 INFO:tasks.rados.rados.0.smithi090.stdout:548: finishing write tid 3 to smithi09025264-23 2023-01-15T14:30:23.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:30:23.221 INFO:tasks.rados.rados.0.smithi090.stdout:548: finishing write tid 4 to smithi09025264-23 2023-01-15T14:30:23.242 INFO:tasks.rados.rados.0.smithi090.stdout:548: finishing write tid 5 to smithi09025264-23 2023-01-15T14:30:23.243 INFO:tasks.rados.rados.0.smithi090.stdout:548: oid 23 updating version 0 to 147 2023-01-15T14:30:23.243 INFO:tasks.rados.rados.0.smithi090.stdout:548: oid 23 updating version 147 to 148 2023-01-15T14:30:23.243 INFO:tasks.rados.rados.0.smithi090.stdout:548: oid 23 version 148 is already newer than 146 2023-01-15T14:30:23.243 INFO:tasks.rados.rados.0.smithi090.stdout:548: oid 23 version 148 is already newer than 145 2023-01-15T14:30:23.243 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 148 (ObjNum 226 snap 51 seq_num 226) dirty exists 2023-01-15T14:30:23.244 INFO:tasks.rados.rados.0.smithi090.stdout:548: left oid 23 (ObjNum 226 snap 51 seq_num 226) 2023-01-15T14:30:23.244 INFO:tasks.rados.rados.0.smithi090.stdout:548: done (0 left) 2023-01-15T14:30:23.262 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:23.260+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:23.332 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:23.333 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:23.333 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:30:23.364 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:23.359+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:23.485 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:23.485 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:23.486 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:30:23.486 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:23.480+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:30:23.504 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 136, 2023-01-15T14:30:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "1c77fb2b-d1a7-4884-a24e-3935e16d0b6b", 2023-01-15T14:30:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2023-01-15T14:27:57.735717+0000", 2023-01-15T14:30:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2023-01-15T14:30:23.014788+0000", 2023-01-15T14:30:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2023-01-15T14:28:04.837522+0000", 2023-01-15T14:30:23.504 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2023-01-15T14:30:12.782283+0000", 2023-01-15T14:30:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-01-15T14:30:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5799936, 2023-01-15T14:30:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2023-01-15T14:30:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2023-01-15T14:30:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2023-01-15T14:30:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2023-01-15T14:30:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2023-01-15T14:30:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:30:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2023-01-15T14:30:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2023-01-15T14:30:23.505 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-01-15T14:30:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2023-01-15T14:30:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2023-01-15T14:30:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 3, 2023-01-15T14:30:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2023-01-15T14:30:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2023-01-15T14:30:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2023-01-15T14:30:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "reef", 2023-01-15T14:30:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2023-01-15T14:30:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.506 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2023-01-15T14:30:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2023-01-15T14:30:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:04.899282+0000", 2023-01-15T14:30:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:30:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:30:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:30:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:30:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:30:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:30:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:30:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:30:23.507 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:30:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:30:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:30:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:30:23.508 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2023-01-15T14:30:23.622 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2023-01-15T14:30:23.623 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2023-01-15T14:30:23.623 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2023-01-15T14:30:23.623 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2023-01-15T14:30:23.623 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:30:23.623 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:30:23.623 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:30:23.623 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:30:23.623 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:30:23.623 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:30:23.624 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:30:23.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.624 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "16", 2023-01-15T14:30:23.624 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:30:23.624 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:30:23.624 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:30:23.624 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:30:23.624 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:30:23.624 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:30:23.625 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:30:23.625 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:30:23.625 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:30:23.627 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:30:23.627 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:30:23.627 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:30:23.627 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:30:23.627 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:30:23.627 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:30:23.628 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:30:23.628 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:30:23.628 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:30:23.628 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:30:23.628 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:30:23.628 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:30:23.628 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:30:23.629 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:30:23.629 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:30:23.629 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:30:23.722 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:30:23.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.722 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:30:23.722 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:30:23.722 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2023-01-15T14:30:23.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.724 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:30:23.724 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2023-01-15T14:30:23.724 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.724 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2023-01-15T14:30:23.725 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2023-01-15T14:30:23.725 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:11.115002+0000", 2023-01-15T14:30:23.725 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:30:23.725 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:30:23.725 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:30:23.725 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:30:23.805 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:30:23.805 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:30:23.806 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:30:23.806 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:30:23.807 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:30:23.807 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:30:23.808 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:30:23.808 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:30:23.808 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2023-01-15T14:30:23.809 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2023-01-15T14:30:23.809 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2023-01-15T14:30:23.809 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2023-01-15T14:30:23.810 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2023-01-15T14:30:23.810 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:30:23.810 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:30:23.810 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:30:23.810 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:30:23.810 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:30:23.810 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:30:23.810 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:30:23.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.811 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "19", 2023-01-15T14:30:23.811 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:30:23.811 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:30:23.811 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:30:23.811 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:30:23.812 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:30:23.812 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:30:23.812 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:30:23.812 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:30:23.812 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:30:23.812 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:30:23.812 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:30:23.812 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:30:23.812 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:30:23.812 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:30:23.812 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:30:23.817 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:30:23.817 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:30:23.817 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:30:23.817 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:30:23.817 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:30:23.817 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:30:23.817 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:30:23.818 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:30:23.818 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:30:23.818 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:30:23.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:30:23.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.818 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:30:23.818 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:30:23.819 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:30:23.819 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:30:23.819 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:30:23.819 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:30:23.819 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:30:23.819 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:30:23.820 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:30:23.820 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:30:23.820 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:30:23.820 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:30:23.820 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:30:23.821 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2023-01-15T14:30:23.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.822 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:30:23.822 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "unique_pool_0", 2023-01-15T14:30:23.822 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:22.434903+0000", 2023-01-15T14:30:23.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 16385, 2023-01-15T14:30:23.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,pool_snaps", 2023-01-15T14:30:23.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:30:23.823 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:30:23.823 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:30:23.823 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:30:23.823 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:30:23.823 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:30:23.823 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:30:23.823 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:30:23.823 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:30:23.823 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:30:23.823 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 26, 2023-01-15T14:30:23.823 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 26, 2023-01-15T14:30:23.824 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 26, 2023-01-15T14:30:23.824 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 26, 2023-01-15T14:30:23.824 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 26, 2023-01-15T14:30:23.824 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:30:23.824 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:30:23.824 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:30:23.824 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:30:23.881 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:30:23.881 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:30:23.881 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:30:23.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.882 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "132", 2023-01-15T14:30:23.882 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:30:23.882 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:30:23.882 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "111", 2023-01-15T14:30:23.883 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:30:23.883 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "pool", 2023-01-15T14:30:23.883 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 102, 2023-01-15T14:30:23.883 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 132, 2023-01-15T14:30:23.883 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:30:23.883 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:30:23.883 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:30:23.883 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:30:23.883 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:30:23.883 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:30:23.884 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:30:23.884 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:30:23.884 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:30:23.884 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:30:23.884 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:30:23.884 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:30:23.884 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:30:23.885 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:30:23.885 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:30:23.885 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:30:23.885 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:30:23.885 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:30:23.885 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:30:23.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.885 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:30:23.886 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:30:23.886 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:30:23.886 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:30:23.886 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:30:23.886 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:30:23.886 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:30:23.886 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:30:23.886 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:30:23.887 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:30:23.887 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:30:23.887 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:30:23.887 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:30:23.887 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2023-01-15T14:30:23.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.887 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:30:23.887 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:30:23.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.888 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:30:23.888 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e55c5ac4-3cff-4ee5-8719-373648f83d8d", 2023-01-15T14:30:23.888 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:30:23.890 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:30:23.890 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:30:23.890 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:30:23.890 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:30:23.890 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:30:23.890 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:30:23.890 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 131, 2023-01-15T14:30:23.890 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:30:23.890 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:30:23.890 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:30:23.891 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:23.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.891 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:23.891 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2023-01-15T14:30:23.891 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:30:23.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.891 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:23.891 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2023-01-15T14:30:23.891 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.892 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:23.893 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2023-01-15T14:30:23.893 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:30:23.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.967 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2023-01-15T14:30:23.968 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.969 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:23.970 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2023-01-15T14:30:23.970 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:30:23.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.970 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:23.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.970 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6811/23473", 2023-01-15T14:30:23.970 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6815/23473", 2023-01-15T14:30:23.972 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/23473", 2023-01-15T14:30:23.972 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6820/23473", 2023-01-15T14:30:23.972 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:30:23.973 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:30:23.973 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:30:23.973 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:23.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.973 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:30:23.973 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "2c1b33c9-0a1e-4734-936d-d51d64fe997a", 2023-01-15T14:30:23.973 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:30:23.973 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:30:23.973 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:30:23.974 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:30:23.974 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:30:23.974 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:30:23.974 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:30:23.974 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 133, 2023-01-15T14:30:23.974 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:30:23.974 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:30:23.975 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:30:23.975 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:23.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.975 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:23.975 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2023-01-15T14:30:23.975 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:30:23.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.975 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:23.975 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2023-01-15T14:30:23.975 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:30:23.976 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.976 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:23.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.976 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:30:23.976 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:23.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:23.976 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2023-01-15T14:30:23.977 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:30:23.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.977 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:23.977 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2023-01-15T14:30:23.977 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:30:23.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.977 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:23.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.977 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:30:23.977 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:23.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.979 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:30:23.979 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:23.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:23.979 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2023-01-15T14:30:23.979 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:30:23.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:23.979 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2023-01-15T14:30:23.979 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:30:23.980 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.980 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:23.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.980 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/23474", 2023-01-15T14:30:23.980 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/23474", 2023-01-15T14:30:23.980 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/23474", 2023-01-15T14:30:23.980 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/23474", 2023-01-15T14:30:23.981 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:30:23.981 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:30:23.981 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:30:23.981 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:23.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.981 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:30:23.981 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "b314174a-eca9-4baa-a60c-2589f1fc9a8e", 2023-01-15T14:30:23.981 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:30:23.981 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:30:23.982 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:30:23.982 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:30:23.982 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:30:23.982 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:30:23.982 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:30:23.982 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 126, 2023-01-15T14:30:23.982 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:30:23.982 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:30:23.982 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:30:23.982 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:23.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.983 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:23.983 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2023-01-15T14:30:23.983 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:30:23.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.988 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:23.988 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2023-01-15T14:30:23.988 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:30:23.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:23.988 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:23.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:23.988 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:30:23.989 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:23.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:23.989 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:23.989 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2023-01-15T14:30:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:30:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:30:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2023-01-15T14:30:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:30:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2023-01-15T14:30:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:30:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6810/23475", 2023-01-15T14:30:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/23475", 2023-01-15T14:30:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6821/23475", 2023-01-15T14:30:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6817/23475", 2023-01-15T14:30:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:30:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:30:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:30:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:30:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "29eb0b17-ed5d-4665-a347-2fb8406d9c62", 2023-01-15T14:30:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:30:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:30:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:30:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:30:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:30:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:30:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:30:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 126, 2023-01-15T14:30:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:30:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:30:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:30:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6824", 2023-01-15T14:30:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:30:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6825", 2023-01-15T14:30:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:30:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:30:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6826", 2023-01-15T14:30:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:30:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6827", 2023-01-15T14:30:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:30:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:30:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6830", 2023-01-15T14:30:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:30:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6831", 2023-01-15T14:30:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:30:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:30:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6828", 2023-01-15T14:30:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:30:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6829", 2023-01-15T14:30:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:30:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6825/23778", 2023-01-15T14:30:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6827/23778", 2023-01-15T14:30:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6831/23778", 2023-01-15T14:30:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6829/23778", 2023-01-15T14:30:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:30:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:30:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:30:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:30:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "59362feb-caa9-44e3-b6d0-b79397dacbfc", 2023-01-15T14:30:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:30:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:30:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:30:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:30:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:30:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:30:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:30:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 134, 2023-01-15T14:30:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:30:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:30:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:30:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6800", 2023-01-15T14:30:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:30:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6801", 2023-01-15T14:30:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:30:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:30:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6802", 2023-01-15T14:30:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:30:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6803", 2023-01-15T14:30:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:30:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:30:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6806", 2023-01-15T14:30:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:30:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6807", 2023-01-15T14:30:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:30:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:30:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6804", 2023-01-15T14:30:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:30:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.261 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6805", 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6801/23523", 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6803/23523", 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6807/23523", 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6805/23523", 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:30:24.262 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:30:24.263 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.263 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:30:24.263 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "fbc8821c-8d78-40f8-a571-b18a70d42000", 2023-01-15T14:30:24.263 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:30:24.263 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:30:24.263 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:30:24.263 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:30:24.263 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:30:24.263 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:30:24.264 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:30:24.264 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:30:24.264 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:30:24.264 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:30:24.264 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:30:24.264 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.264 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.264 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6824", 2023-01-15T14:30:24.264 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:30:24.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.265 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.265 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6825", 2023-01-15T14:30:24.265 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:30:24.308 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:30:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6826", 2023-01-15T14:30:24.309 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:30:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6827", 2023-01-15T14:30:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:30:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.439 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:30:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6830", 2023-01-15T14:30:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:30:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.440 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6831", 2023-01-15T14:30:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:30:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:30:24.441 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6828", 2023-01-15T14:30:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:30:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6829", 2023-01-15T14:30:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:30:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6825/23525", 2023-01-15T14:30:24.443 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6827/23525", 2023-01-15T14:30:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6831/23525", 2023-01-15T14:30:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6829/23525", 2023-01-15T14:30:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:30:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:30:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:30:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.444 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:30:24.557 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "34550470-3561-492f-b605-b5f0e8052c1e", 2023-01-15T14:30:24.557 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:30:24.557 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:30:24.558 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:30:24.558 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:30:24.558 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:30:24.558 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:30:24.558 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:30:24.558 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 118, 2023-01-15T14:30:24.558 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:30:24.558 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:30:24.558 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:30:24.558 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.559 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.559 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6808", 2023-01-15T14:30:24.559 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:30:24.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.559 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.559 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6809", 2023-01-15T14:30:24.559 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:30:24.559 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.559 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.560 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:30:24.560 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.560 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.560 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6810", 2023-01-15T14:30:24.560 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:30:24.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.560 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6811", 2023-01-15T14:30:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:30:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:30:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6814", 2023-01-15T14:30:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:30:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6815", 2023-01-15T14:30:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:30:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:30:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6812", 2023-01-15T14:30:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:30:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6813", 2023-01-15T14:30:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:30:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6809/23524", 2023-01-15T14:30:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6811/23524", 2023-01-15T14:30:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6815/23524", 2023-01-15T14:30:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6813/23524", 2023-01-15T14:30:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:30:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:30:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:30:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:30:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1f123d0d-d617-4771-b516-21aa5a29b734", 2023-01-15T14:30:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:30:24.602 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:30:24.602 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.5779571533203125, 2023-01-15T14:30:24.602 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:30:24.603 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:30:24.603 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:30:24.603 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:30:24.603 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 131, 2023-01-15T14:30:24.603 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:30:24.603 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:30:24.603 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:30:24.603 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.603 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.604 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6816", 2023-01-15T14:30:24.604 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:30:24.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.604 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.604 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6817", 2023-01-15T14:30:24.604 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:30:24.604 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.604 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.605 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:30:24.605 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.605 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.605 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6818", 2023-01-15T14:30:24.605 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:30:24.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.605 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.605 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6819", 2023-01-15T14:30:24.605 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:30:24.606 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.606 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.606 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:30:24.606 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.606 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.606 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6822", 2023-01-15T14:30:24.606 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:30:24.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.607 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.607 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6823", 2023-01-15T14:30:24.607 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:30:24.607 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.607 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.608 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:30:24.608 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:30:24.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.608 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:30:24.608 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6820", 2023-01-15T14:30:24.608 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:30:24.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.609 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:30:24.609 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6821", 2023-01-15T14:30:24.609 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:30:24.609 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.609 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.609 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6817/23526", 2023-01-15T14:30:24.624 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6819/23526", 2023-01-15T14:30:24.624 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6823/23526", 2023-01-15T14:30:24.624 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6821/23526", 2023-01-15T14:30:24.624 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:30:24.624 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:30:24.624 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:30:24.624 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.624 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:30:24.625 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2023-01-15T14:30:24.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.625 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:30:24.625 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:30:24.625 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:30:24.625 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:30:24.625 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:30:24.625 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:30:24.625 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:29:26.125016+0000", 2023-01-15T14:30:24.625 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:30:24.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:29:30.078140+0000", 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.626 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:30:24.627 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:30:24.627 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:30:24.627 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:30:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:30:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:30:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:29:29.124231+0000", 2023-01-15T14:30:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:30:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:30:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:30:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:30:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:30:24.751 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:30:24.751 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:30:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:30:15.421866+0000", 2023-01-15T14:30:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:30:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:30:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:30:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:30:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:30:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:30:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:30:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:29:20.829475+0000", 2023-01-15T14:30:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:30:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:30:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:30:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:30:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:30:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:30:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:30:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:30:08.564732+0000", 2023-01-15T14:30:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:30:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:30:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:30:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:30:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:30:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:30:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:30:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:29:17.150234+0000", 2023-01-15T14:30:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:30:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:30:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:30:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:30:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:30:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:30:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:30:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:30:11.543592+0000", 2023-01-15T14:30:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:30:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:30:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [ 2023-01-15T14:30:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "2.2", 2023-01-15T14:30:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:30:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: 2, 2023-01-15T14:30:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: 1, 2023-01-15T14:30:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: 6 2023-01-15T14:30:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:30:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2023-01-15T14:30:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.8", 2023-01-15T14:30:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:30:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 3, 2023-01-15T14:30:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 1 2023-01-15T14:30:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:30:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2023-01-15T14:30:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2023-01-15T14:30:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2023-01-15T14:30:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2023-01-15T14:30:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2023-01-15T14:30:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2023-01-15T14:30:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2023-01-15T14:30:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2023-01-15T14:30:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2023-01-15T14:30:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2023-01-15T14:30:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2023-01-15T14:30:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2023-01-15T14:30:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2023-01-15T14:30:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:30:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2023-01-15T14:30:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 86, 2023-01-15T14:30:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:30:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 93, 2023-01-15T14:30:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:30:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 96, 2023-01-15T14:30:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:30:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 99, 2023-01-15T14:30:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:30:24.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:24.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:24.920 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 101, 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2023-01-15T14:30:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2023-01-15T14:30:24.922 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2023-01-15T14:30:24.922 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2023-01-15T14:30:24.922 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2023-01-15T14:30:24.922 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:24.922 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:24.924 INFO:tasks.thrashosds.thrasher:j is {'epoch': 136, 'fsid': '1c77fb2b-d1a7-4884-a24e-3935e16d0b6b', 'created': '2023-01-15T14:27:57.735717+0000', 'modified': '2023-01-15T14:30:23.014788+0000', 'last_up_change': '2023-01-15T14:28:04.837522+0000', 'last_in_change': '2023-01-15T14:30:12.782283+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-01-15T14:28:04.899282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-01-15T14:28:11.115002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-01-15T14:28:22.434903+0000', 'flags': 16385, 'flags_names': 'hashpspool,pool_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '132', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '111', 'auid': 0, 'snap_mode': 'pool', 'snap_seq': 102, 'snap_epoch': 132, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e55c5ac4-3cff-4ee5-8719-373648f83d8d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 131, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 23473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 23473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 23473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6820', 'nonce': 23473}]}, 'public_addr': '172.21.15.90:6811/23473', 'cluster_addr': '172.21.15.90:6815/23473', 'heartbeat_back_addr': '172.21.15.90:6823/23473', 'heartbeat_front_addr': '172.21.15.90:6820/23473', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2c1b33c9-0a1e-4734-936d-d51d64fe997a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 133, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 23474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 23474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 23474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 23474}]}, 'public_addr': '172.21.15.90:6801/23474', 'cluster_addr': '172.21.15.90:6803/23474', 'heartbeat_back_addr': '172.21.15.90:6807/23474', 'heartbeat_front_addr': '172.21.15.90:6805/23474', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b314174a-eca9-4baa-a60c-2589f1fc9a8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 126, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 23475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 23475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6819', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 23475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 23475}]}, 'public_addr': '172.21.15.90:6810/23475', 'cluster_addr': '172.21.15.90:6814/23475', 'heartbeat_back_addr': '172.21.15.90:6821/23475', 'heartbeat_front_addr': '172.21.15.90:6817/23475', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '29eb0b17-ed5d-4665-a347-2fb8406d9c62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 126, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6824', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6825', 'nonce': 23778}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6826', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6827', 'nonce': 23778}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6830', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6831', 'nonce': 23778}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6828', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6829', 'nonce': 23778}]}, 'public_addr': '172.21.15.90:6825/23778', 'cluster_addr': '172.21.15.90:6827/23778', 'heartbeat_back_addr': '172.21.15.90:6831/23778', 'heartbeat_front_addr': '172.21.15.90:6829/23778', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '59362feb-caa9-44e3-b6d0-b79397dacbfc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 134, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6800', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6801', 'nonce': 23523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6802', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6803', 'nonce': 23523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6806', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6807', 'nonce': 23523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6804', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6805', 'nonce': 23523}]}, 'public_addr': '172.21.15.141:6801/23523', 'cluster_addr': '172.21.15.141:6803/23523', 'heartbeat_back_addr': '172.21.15.141:6807/23523', 'heartbeat_front_addr': '172.21.15.141:6805/23523', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fbc8821c-8d78-40f8-a571-b18a70d42000', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6824', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6825', 'nonce': 23525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6826', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6827', 'nonce': 23525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6830', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6831', 'nonce': 23525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6828', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6829', 'nonce': 23525}]}, 'public_addr': '172.21.15.141:6825/23525', 'cluster_addr': '172.21.15.141:6827/23525', 'heartbeat_back_addr': '172.21.15.141:6831/23525', 'heartbeat_front_addr': '172.21.15.141:6829/23525', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '34550470-3561-492f-b605-b5f0e8052c1e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6808', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6809', 'nonce': 23524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6810', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6811', 'nonce': 23524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6814', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6815', 'nonce': 23524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6812', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6813', 'nonce': 23524}]}, 'public_addr': '172.21.15.141:6809/23524', 'cluster_addr': '172.21.15.141:6811/23524', 'heartbeat_back_addr': '172.21.15.141:6815/23524', 'heartbeat_front_addr': '172.21.15.141:6813/23524', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f123d0d-d617-4771-b516-21aa5a29b734', 'up': 1, 'in': 1, 'weight': 0.5779571533203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 131, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6816', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6817', 'nonce': 23526}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6818', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6819', 'nonce': 23526}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6822', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6823', 'nonce': 23526}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6820', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6821', 'nonce': 23526}]}, 'public_addr': '172.21.15.141:6817/23526', 'cluster_addr': '172.21.15.141:6819/23526', 'heartbeat_back_addr': '172.21.15.141:6823/23526', 'heartbeat_front_addr': '172.21.15.141:6821/23526', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:29:26.125016+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:29:30.078140+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:29:29.124231+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:30:15.421866+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:29:20.829475+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:30:08.564732+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:29:17.150234+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:30:11.543592+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.2', 'osds': [2, 1, 6]}], 'pg_upmap_items': [{'pgid': '3.8', 'mappings': [{'from': 3, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 86, 'length': 1}, {'begin': 93, 'length': 1}, {'begin': 96, 'length': 1}, {'begin': 99, 'length': 1}, {'begin': 101, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-01-15T14:30:24.925 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.2 2023-01-15T14:30:24.925 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap 2.2 2023-01-15T14:30:24.926 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:24.926 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:24.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:30:24.929 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:24.919+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:25.070 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:30:25.071 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:25.067+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:25.093 INFO:tasks.rados.rados.0.smithi090.stdout:553: done (0 left) 2023-01-15T14:30:25.093 INFO:tasks.rados.rados.0.smithi090.stdout:554: read oid 31 snap 51 2023-01-15T14:30:25.093 INFO:tasks.rados.rados.0.smithi090.stdout:554: expect deleted 2023-01-15T14:30:25.093 INFO:tasks.rados.rados.0.smithi090.stdout:555: read oid 13 snap -1 2023-01-15T14:30:25.093 INFO:tasks.rados.rados.0.smithi090.stdout:555: expect (ObjNum 216 snap 49 seq_num 216) 2023-01-15T14:30:25.093 INFO:tasks.rados.rados.0.smithi090.stdout:556: copy_from oid 32 from oid 5 current snap is 52 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:557: read oid 21 snap -1 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:557: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:558: copy_from oid 14 from oid 28 current snap is 52 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:559: read oid 38 snap -1 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:559: expect deleted 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:560: write oid 34 current snap is 52 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:560: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:560: writing smithi09025264-34 from 631996 to 1163005 tid 1 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:560: writing smithi09025264-34 from 1792365 to 2247101 tid 2 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:560: writing smithi09025264-34 from 2761854 to 3433167 tid 3 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:561: write oid 24 current snap is 52 2023-01-15T14:30:25.094 INFO:tasks.rados.rados.0.smithi090.stdout:561: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2023-01-15T14:30:25.095 INFO:tasks.rados.rados.0.smithi090.stdout:561: writing smithi09025264-24 from 648803 to 1371414 tid 1 2023-01-15T14:30:25.095 INFO:tasks.rados.rados.0.smithi090.stdout:561: writing smithi09025264-24 from 1967200 to 2681947 tid 2 2023-01-15T14:30:25.095 INFO:tasks.rados.rados.0.smithi090.stdout:561: writing smithi09025264-24 from 3421983 to 3848803 tid 3 2023-01-15T14:30:25.095 INFO:tasks.rados.rados.0.smithi090.stdout:562: write (excl) oid 40 current snap is 52 2023-01-15T14:30:25.095 INFO:tasks.rados.rados.0.smithi090.stdout:562: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2023-01-15T14:30:25.095 INFO:tasks.rados.rados.0.smithi090.stdout:562: writing smithi09025264-40 from 665610 to 1463470 tid 1 2023-01-15T14:30:25.095 INFO:tasks.rados.rados.0.smithi090.stdout:562: writing smithi09025264-40 from 1909329 to 2367734 tid 2 2023-01-15T14:30:25.095 INFO:tasks.rados.rados.0.smithi090.stdout:562: writing smithi09025264-40 from 2816700 to 3226910 tid 3 2023-01-15T14:30:25.095 INFO:tasks.rados.rados.0.smithi090.stdout:563: delete oid 37 current snap is 52 2023-01-15T14:30:25.095 INFO:tasks.rados.rados.0.smithi090.stdout:555: expect (ObjNum 216 snap 49 seq_num 216) 2023-01-15T14:30:25.096 INFO:tasks.rados.rados.0.smithi090.stdout:558: finishing copy_from to smithi09025264-14 2023-01-15T14:30:25.096 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 218 (ObjNum 222 snap 51 seq_num 222) dirty exists 2023-01-15T14:30:25.096 INFO:tasks.rados.rados.0.smithi090.stdout:558: finishing copy_from racing read to smithi09025264-14 2023-01-15T14:30:25.096 INFO:tasks.rados.rados.0.smithi090.stdout:560: finishing write tid 1 to smithi09025264-34 2023-01-15T14:30:25.096 INFO:tasks.rados.rados.0.smithi090.stdout:560: finishing write tid 2 to smithi09025264-34 2023-01-15T14:30:25.096 INFO:tasks.rados.rados.0.smithi090.stdout:555: done (9 left) 2023-01-15T14:30:25.096 INFO:tasks.rados.rados.0.smithi090.stdout:558: done (8 left) 2023-01-15T14:30:25.096 INFO:tasks.rados.rados.0.smithi090.stdout:559: done (7 left) 2023-01-15T14:30:25.096 INFO:tasks.rados.rados.0.smithi090.stdout:563: done (6 left) 2023-01-15T14:30:25.096 INFO:tasks.rados.rados.0.smithi090.stdout:564: write oid 44 current snap is 52 2023-01-15T14:30:25.096 INFO:tasks.rados.rados.0.smithi090.stdout:564: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2023-01-15T14:30:25.097 INFO:tasks.rados.rados.0.smithi090.stdout:564: writing smithi09025264-44 from 699224 to 1247582 tid 1 2023-01-15T14:30:25.097 INFO:tasks.rados.rados.0.smithi090.stdout:564: writing smithi09025264-44 from 2026293 to 2488367 tid 2 2023-01-15T14:30:25.097 INFO:tasks.rados.rados.0.smithi090.stdout:564: writing smithi09025264-44 from 3271546 to 3704300 tid 3 2023-01-15T14:30:25.097 INFO:tasks.rados.rados.0.smithi090.stdout:565: snap_create 2023-01-15T14:30:25.097 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:30:25.097 INFO:tasks.rados.rados.0.smithi090.stdout:556: finishing copy_from to smithi09025264-32 2023-01-15T14:30:25.097 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 188 (ObjNum 224 snap 51 seq_num 224) dirty exists 2023-01-15T14:30:25.097 INFO:tasks.rados.rados.0.smithi090.stdout:556: finishing copy_from racing read to smithi09025264-32 2023-01-15T14:30:25.100 INFO:tasks.rados.rados.0.smithi090.stdout:556: done (6 left) 2023-01-15T14:30:25.100 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:30:25.100 INFO:tasks.rados.rados.0.smithi090.stdout:562: finishing write tid 1 to smithi09025264-40 2023-01-15T14:30:25.100 INFO:tasks.rados.rados.0.smithi090.stdout:561: finishing write tid 1 to smithi09025264-24 2023-01-15T14:30:25.100 INFO:tasks.rados.rados.0.smithi090.stdout:561: finishing write tid 2 to smithi09025264-24 2023-01-15T14:30:25.100 INFO:tasks.rados.rados.0.smithi090.stdout:561: finishing write tid 3 to smithi09025264-24 2023-01-15T14:30:25.100 INFO:tasks.rados.rados.0.smithi090.stdout:561: finishing write tid 4 to smithi09025264-24 2023-01-15T14:30:25.100 INFO:tasks.rados.rados.0.smithi090.stdout:557: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:30:25.100 INFO:tasks.rados.rados.0.smithi090.stdout:561: finishing write tid 5 to smithi09025264-24 2023-01-15T14:30:25.100 INFO:tasks.rados.rados.0.smithi090.stdout:561: oid 24 updating version 0 to 228 2023-01-15T14:30:25.101 INFO:tasks.rados.rados.0.smithi090.stdout:561: oid 24 version 228 is already newer than 225 2023-01-15T14:30:25.101 INFO:tasks.rados.rados.0.smithi090.stdout:561: oid 24 version 228 is already newer than 226 2023-01-15T14:30:25.101 INFO:tasks.rados.rados.0.smithi090.stdout:561: oid 24 version 228 is already newer than 227 2023-01-15T14:30:25.101 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 228 (ObjNum 229 snap 52 seq_num 229) dirty exists 2023-01-15T14:30:25.101 INFO:tasks.rados.rados.0.smithi090.stdout:561: left oid 24 (ObjNum 229 snap 52 seq_num 229) 2023-01-15T14:30:25.101 INFO:tasks.rados.rados.0.smithi090.stdout:560: finishing write tid 3 to smithi09025264-34 2023-01-15T14:30:25.101 INFO:tasks.rados.rados.0.smithi090.stdout:560: finishing write tid 4 to smithi09025264-34 2023-01-15T14:30:25.101 INFO:tasks.rados.rados.0.smithi090.stdout:560: finishing write tid 5 to smithi09025264-34 2023-01-15T14:30:25.101 INFO:tasks.rados.rados.0.smithi090.stdout:560: oid 34 updating version 0 to 127 2023-01-15T14:30:25.101 INFO:tasks.rados.rados.0.smithi090.stdout:560: oid 34 updating version 127 to 129 2023-01-15T14:30:25.101 INFO:tasks.rados.rados.0.smithi090.stdout:560: oid 34 version 129 is already newer than 128 2023-01-15T14:30:25.102 INFO:tasks.rados.rados.0.smithi090.stdout:560: oid 34 version 129 is already newer than 126 2023-01-15T14:30:25.102 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 129 (ObjNum 228 snap 52 seq_num 228) dirty exists 2023-01-15T14:30:25.102 INFO:tasks.rados.rados.0.smithi090.stdout:560: left oid 34 (ObjNum 228 snap 52 seq_num 228) 2023-01-15T14:30:25.102 INFO:tasks.rados.rados.0.smithi090.stdout:562: finishing write tid 2 to smithi09025264-40 2023-01-15T14:30:25.102 INFO:tasks.rados.rados.0.smithi090.stdout:562: finishing write tid 3 to smithi09025264-40 2023-01-15T14:30:25.102 INFO:tasks.rados.rados.0.smithi090.stdout:562: finishing write tid 4 to smithi09025264-40 2023-01-15T14:30:25.102 INFO:tasks.rados.rados.0.smithi090.stdout:562: finishing write tid 5 to smithi09025264-40 2023-01-15T14:30:25.102 INFO:tasks.rados.rados.0.smithi090.stdout:562: oid 40 updating version 0 to 146 2023-01-15T14:30:25.102 INFO:tasks.rados.rados.0.smithi090.stdout:562: oid 40 updating version 146 to 148 2023-01-15T14:30:25.102 INFO:tasks.rados.rados.0.smithi090.stdout:562: oid 40 version 148 is already newer than 147 2023-01-15T14:30:25.102 INFO:tasks.rados.rados.0.smithi090.stdout:562: oid 40 version 148 is already newer than 145 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 148 (ObjNum 230 snap 52 seq_num 230) dirty exists 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:562: left oid 40 (ObjNum 230 snap 52 seq_num 230) 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:554: done (5 left) 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:557: done (4 left) 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:560: done (3 left) 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:561: done (2 left) 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:562: done (1 left) 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:564: finishing write tid 1 to smithi09025264-44 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:564: finishing write tid 2 to smithi09025264-44 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:564: finishing write tid 3 to smithi09025264-44 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:564: finishing write tid 4 to smithi09025264-44 2023-01-15T14:30:25.103 INFO:tasks.rados.rados.0.smithi090.stdout:564: finishing write tid 5 to smithi09025264-44 2023-01-15T14:30:25.104 INFO:tasks.rados.rados.0.smithi090.stdout:564: oid 44 updating version 0 to 160 2023-01-15T14:30:25.104 INFO:tasks.rados.rados.0.smithi090.stdout:564: oid 44 updating version 160 to 161 2023-01-15T14:30:25.104 INFO:tasks.rados.rados.0.smithi090.stdout:564: oid 44 version 161 is already newer than 159 2023-01-15T14:30:25.104 INFO:tasks.rados.rados.0.smithi090.stdout:564: oid 44 version 161 is already newer than 158 2023-01-15T14:30:25.104 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 161 (ObjNum 232 snap 52 seq_num 232) dirty exists 2023-01-15T14:30:25.104 INFO:tasks.rados.rados.0.smithi090.stdout:564: left oid 44 (ObjNum 232 snap 52 seq_num 232) 2023-01-15T14:30:25.104 INFO:tasks.rados.rados.0.smithi090.stdout:564: done (0 left) 2023-01-15T14:30:25.104 INFO:tasks.rados.rados.0.smithi090.stdout:565: done (0 left) 2023-01-15T14:30:25.104 INFO:tasks.rados.rados.0.smithi090.stdout:566: read oid 22 snap 52 2023-01-15T14:30:25.104 INFO:tasks.rados.rados.0.smithi090.stdout:566: expect (ObjNum 160 snap 35 seq_num 160) 2023-01-15T14:30:25.104 INFO:tasks.rados.rados.0.smithi090.stdout:567: rollback oid 7 current snap is 53 2023-01-15T14:30:25.105 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 7 to 52 2023-01-15T14:30:25.105 INFO:tasks.rados.rados.0.smithi090.stdout:568: copy_from oid 28 from oid 40 current snap is 53 2023-01-15T14:30:25.105 INFO:tasks.rados.rados.0.smithi090.stdout:569: copy_from oid 26 from oid 2 current snap is 53 2023-01-15T14:30:25.105 INFO:tasks.rados.rados.0.smithi090.stdout:570: read oid 17 snap -1 2023-01-15T14:30:25.105 INFO:tasks.rados.rados.0.smithi090.stdout:570: expect (ObjNum 221 snap 50 seq_num 221) 2023-01-15T14:30:25.105 INFO:tasks.rados.rados.0.smithi090.stdout:571: rollback oid 50 current snap is 53 2023-01-15T14:30:25.105 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 50 to 52 2023-01-15T14:30:25.105 INFO:tasks.rados.rados.0.smithi090.stdout:572: write (excl) oid 49 current snap is 53 2023-01-15T14:30:25.105 INFO:tasks.rados.rados.0.smithi090.stdout:572: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2023-01-15T14:30:25.105 INFO:tasks.rados.rados.0.smithi090.stdout:572: writing smithi09025264-49 from 716031 to 1339638 tid 1 2023-01-15T14:30:25.105 INFO:tasks.rados.rados.0.smithi090.stdout:572: writing smithi09025264-49 from 2084775 to 2690507 tid 2 2023-01-15T14:30:25.106 INFO:tasks.rados.rados.0.smithi090.stdout:572: writing smithi09025264-49 from 3298969 to 3916031 tid 3 2023-01-15T14:30:25.106 INFO:tasks.rados.rados.0.smithi090.stdout:573: snap_create 2023-01-15T14:30:25.106 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:30:25.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:30:25.238 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:25.239 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:25.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:30:25.239 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:25.236+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:25.289 INFO:tasks.rados.rados.0.smithi090.stdout:572: finishing write tid 1 to smithi09025264-49 2023-01-15T14:30:25.403 INFO:tasks.rados.rados.0.smithi090.stdout:572: finishing write tid 2 to smithi09025264-49 2023-01-15T14:30:25.403 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:25.404 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:25.404 INFO:tasks.rados.rados.0.smithi090.stdout:566: expect (ObjNum 160 snap 35 seq_num 160) 2023-01-15T14:30:25.404 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:25.404 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:25.404 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:25.405 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:25.405 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:25.405 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:25.405 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:25.399+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:25.406 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:25.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:30:25.540 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:25.540 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:25.540 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:25.540 INFO:tasks.rados.rados.0.smithi090.stdout:570: expect (ObjNum 221 snap 50 seq_num 221) 2023-01-15T14:30:25.541 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:25.541 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:30:25.541 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:30:25.541 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:30:25.542 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:30:25.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:30:25.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:30:25.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:30:25.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:30:25.543 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:25.535+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:25.635 INFO:tasks.rados.rados.0.smithi090.stdout:568: finishing copy_from to smithi09025264-28 2023-01-15T14:30:25.635 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 148 (ObjNum 230 snap 52 seq_num 230) dirty exists 2023-01-15T14:30:25.635 INFO:tasks.rados.rados.0.smithi090.stdout:568: finishing copy_from racing read to smithi09025264-28 2023-01-15T14:30:25.635 INFO:tasks.rados.rados.0.smithi090.stdout:569: finishing copy_from to smithi09025264-26 2023-01-15T14:30:25.635 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 50 (ObjNum 70 snap 8 seq_num 70) dirty exists 2023-01-15T14:30:25.635 INFO:tasks.rados.rados.0.smithi090.stdout:569: finishing copy_from racing read to smithi09025264-26 2023-01-15T14:30:25.635 INFO:tasks.rados.rados.0.smithi090.stdout:572: finishing write tid 3 to smithi09025264-49 2023-01-15T14:30:25.636 INFO:tasks.rados.rados.0.smithi090.stdout:572: finishing write tid 4 to smithi09025264-49 2023-01-15T14:30:25.636 INFO:tasks.rados.rados.0.smithi090.stdout:572: finishing write tid 5 to smithi09025264-49 2023-01-15T14:30:25.636 INFO:tasks.rados.rados.0.smithi090.stdout:572: oid 49 updating version 0 to 200 2023-01-15T14:30:25.636 INFO:tasks.rados.rados.0.smithi090.stdout:572: oid 49 version 200 is already newer than 199 2023-01-15T14:30:25.703 INFO:tasks.rados.rados.0.smithi090.stdout:572: oid 49 version 200 is already newer than 198 2023-01-15T14:30:25.704 INFO:tasks.rados.rados.0.smithi090.stdout:572: oid 49 version 200 is already newer than 197 2023-01-15T14:30:25.704 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 200 (ObjNum 233 snap 53 seq_num 233) dirty exists 2023-01-15T14:30:25.704 INFO:tasks.rados.rados.0.smithi090.stdout:572: left oid 49 (ObjNum 233 snap 53 seq_num 233) 2023-01-15T14:30:25.704 INFO:tasks.rados.rados.0.smithi090.stdout:567: finishing rollback tid 0 to smithi09025264-7 2023-01-15T14:30:25.704 INFO:tasks.rados.rados.0.smithi090.stdout:567: finishing rollback tid 1 to smithi09025264-7 2023-01-15T14:30:25.704 INFO:tasks.rados.rados.0.smithi090.stdout:566: done (6 left) 2023-01-15T14:30:25.704 INFO:tasks.rados.rados.0.smithi090.stdout:568: done (5 left) 2023-01-15T14:30:25.704 INFO:tasks.rados.rados.0.smithi090.stdout:569: done (4 left) 2023-01-15T14:30:25.704 INFO:tasks.rados.rados.0.smithi090.stdout:570: done (3 left) 2023-01-15T14:30:25.704 INFO:tasks.rados.rados.0.smithi090.stdout:572: done (2 left) 2023-01-15T14:30:25.704 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:25.705 INFO:tasks.rados.rados.0.smithi090.stdout:567: finishing rollback tid 2 to smithi09025264-7 2023-01-15T14:30:25.705 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 111 (ObjNum 128 snap 23 seq_num 128) dirty exists 2023-01-15T14:30:25.705 INFO:tasks.rados.rados.0.smithi090.stdout:571: finishing rollback tid 0 to smithi09025264-50 2023-01-15T14:30:25.705 INFO:tasks.rados.rados.0.smithi090.stdout:567: done (1 left) 2023-01-15T14:30:25.705 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:25.707 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:25.707 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:25.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:25.707 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:25.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.708 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:38 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'3, mlcod=0'0)", 2023-01-15T14:30:25.708 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:30:25.708 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 119.53127198599999, 2023-01-15T14:30:25.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30191175999999997, 2023-01-15T14:30:25.708 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:25.708 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:25.708 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:25.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:25.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:30:25.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:25.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:25.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:30:25.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:25.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:25.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112976+0000", 2023-01-15T14:30:25.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:25.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:25.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113186+0000", 2023-01-15T14:30:25.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020979700000000001 2023-01-15T14:30:25.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:25.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113228+0000", 2023-01-15T14:30:25.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2150999999999997e-05 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213313+0000", 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100084943 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313517+0000", 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10020435699999999 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:25.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313576+0000", 2023-01-15T14:30:25.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783999999999998e-05 2023-01-15T14:30:25.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:25.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313976+0000", 2023-01-15T14:30:25.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039960499999999998 2023-01-15T14:30:25.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:25.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314004+0000", 2023-01-15T14:30:25.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7954e-05 2023-01-15T14:30:25.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:25.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314175+0000", 2023-01-15T14:30:25.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171557 2023-01-15T14:30:25.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414853+0000", 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100677875 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414890+0000", 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6803e-05 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.715 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:39 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'4, mlcod=0'0)", 2023-01-15T14:30:25.715 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:30:25.715 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 119.430879811, 2023-01-15T14:30:25.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20154512699999999, 2023-01-15T14:30:25.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:25.715 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:25.715 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:25.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:25.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:30:25.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:25.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:25.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213370+0000", 2023-01-15T14:30:25.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:25.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:25.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213369+0000", 2023-01-15T14:30:25.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:25.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:25.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213386+0000", 2023-01-15T14:30:25.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7643e-05 2023-01-15T14:30:25.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:25.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213427+0000", 2023-01-15T14:30:25.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0930000000000003e-05 2023-01-15T14:30:25.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:25.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213459+0000", 2023-01-15T14:30:25.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1479999999999997e-05 2023-01-15T14:30:25.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:25.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314106+0000", 2023-01-15T14:30:25.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100646614 2023-01-15T14:30:25.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:25.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314128+0000", 2023-01-15T14:30:25.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2340999999999998e-05 2023-01-15T14:30:25.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:25.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314224+0000", 2023-01-15T14:30:25.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6534999999999996e-05 2023-01-15T14:30:25.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:25.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314249+0000", 2023-01-15T14:30:25.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4981e-05 2023-01-15T14:30:25.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:25.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314337+0000", 2023-01-15T14:30:25.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7287000000000006e-05 2023-01-15T14:30:25.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:25.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414903+0000", 2023-01-15T14:30:25.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100566552 2023-01-15T14:30:25.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:25.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414916+0000", 2023-01-15T14:30:25.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354000000000001e-05 2023-01-15T14:30:25.735 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:25.735 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:25.735 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:25.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.735 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:46 3.0 e23/20 3:03b54904:::smithi09025264-12:head v 23'5, mlcod=23'1)", 2023-01-15T14:30:25.735 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:30:25.735 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 119.430376072, 2023-01-15T14:30:25.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202592668, 2023-01-15T14:30:25.735 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:25.735 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:30:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:30:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:25.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213873+0000", 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214477+0000", 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060481300000000005 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:25.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214532+0000", 2023-01-15T14:30:25.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5096999999999998e-05 2023-01-15T14:30:25.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:25.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214555+0000", 2023-01-15T14:30:25.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.234e-05 2023-01-15T14:30:25.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:25.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414409+0000", 2023-01-15T14:30:25.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19985461199999999 2023-01-15T14:30:25.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:25.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414442+0000", 2023-01-15T14:30:25.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2987000000000001e-05 2023-01-15T14:30:25.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:25.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414813+0000", 2023-01-15T14:30:25.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037061899999999998 2023-01-15T14:30:25.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:25.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414832+0000", 2023-01-15T14:30:25.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9380000000000001e-05 2023-01-15T14:30:25.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:25.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415093+0000", 2023-01-15T14:30:25.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026032099999999999 2023-01-15T14:30:25.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:25.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415154+0000", 2023-01-15T14:30:25.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1538999999999995e-05 2023-01-15T14:30:25.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:25.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.416467+0000", 2023-01-15T14:30:25.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001312613 2023-01-15T14:30:25.741 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:25.741 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:25.742 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:25.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.742 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:347 3.0 e32/20 3:03b54904:::smithi09025264-12:head v 32'40, mlcod=31'36)", 2023-01-15T14:30:25.742 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:30:25.742 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.632453173, 2023-01-15T14:30:25.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20127003500000001, 2023-01-15T14:30:25.742 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:25.742 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:25.742 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:25.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011797+0000", 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:25.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.011795+0000", 2023-01-15T14:30:25.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:25.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:25.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.012239+0000", 2023-01-15T14:30:25.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000444361 2023-01-15T14:30:25.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:25.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.012261+0000", 2023-01-15T14:30:25.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2008e-05 2023-01-15T14:30:25.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:25.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.112398+0000", 2023-01-15T14:30:25.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013730799999999 2023-01-15T14:30:25.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:25.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.210584+0000", 2023-01-15T14:30:25.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098186184999999995 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.210634+0000", 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9706000000000001e-05 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211053+0000", 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041844199999999999 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:25.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211117+0000", 2023-01-15T14:30:25.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4950000000000007e-05 2023-01-15T14:30:25.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:25.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211390+0000", 2023-01-15T14:30:25.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027278300000000002 2023-01-15T14:30:25.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:25.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.211507+0000", 2023-01-15T14:30:25.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011642 2023-01-15T14:30:25.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:25.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:38.213067+0000", 2023-01-15T14:30:25.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001560402 2023-01-15T14:30:25.894 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:25.895 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:25.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:25.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.895 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.4.0:118 3.9 e62/20 3:9f7cca9d:::smithi09025264-41:1f v 62'64, mlcod=61'58)", 2023-01-15T14:30:25.895 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:30:25.895 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.771062685000004, 2023-01-15T14:30:25.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20203982600000001, 2023-01-15T14:30:25.895 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:25.895 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:25.895 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:25.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:25.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:30:25.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:25.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873188+0000", 2023-01-15T14:30:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873186+0000", 2023-01-15T14:30:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:25.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873201+0000", 2023-01-15T14:30:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5474000000000001e-05 2023-01-15T14:30:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873219+0000", 2023-01-15T14:30:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7677999999999999e-05 2023-01-15T14:30:25.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.873234+0000", 2023-01-15T14:30:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4481000000000001e-05 2023-01-15T14:30:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.073729+0000", 2023-01-15T14:30:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200495595 2023-01-15T14:30:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.073768+0000", 2023-01-15T14:30:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8492999999999999e-05 2023-01-15T14:30:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074468+0000", 2023-01-15T14:30:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069991300000000003 2023-01-15T14:30:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074486+0000", 2023-01-15T14:30:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8669999999999999e-05 2023-01-15T14:30:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074563+0000", 2023-01-15T14:30:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7084999999999998e-05 2023-01-15T14:30:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.074599+0000", 2023-01-15T14:30:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5453000000000001e-05 2023-01-15T14:30:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:08.075227+0000", 2023-01-15T14:30:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062855400000000003 2023-01-15T14:30:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:25.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.015 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:125 3.0 e70/55 3:0b54190a:::smithi09025264-17:25 v 70'136, mlcod=68'130)", 2023-01-15T14:30:26.015 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:30:26.016 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.701289961000001, 2023-01-15T14:30:26.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30171926799999998, 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:30:26.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942958+0000", 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942972+0000", 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3386e-05 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942990+0000", 2023-01-15T14:30:26.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8113000000000001e-05 2023-01-15T14:30:26.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.943002+0000", 2023-01-15T14:30:26.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:30:26.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141942+0000", 2023-01-15T14:30:26.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19893962300000001 2023-01-15T14:30:26.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141978+0000", 2023-01-15T14:30:26.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6383e-05 2023-01-15T14:30:26.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:26.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142670+0000", 2023-01-15T14:30:26.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069211900000000005 2023-01-15T14:30:26.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:26.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142688+0000", 2023-01-15T14:30:26.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7839999999999999e-05 2023-01-15T14:30:26.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:26.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142768+0000", 2023-01-15T14:30:26.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0210999999999995e-05 2023-01-15T14:30:26.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:26.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244581+0000", 2023-01-15T14:30:26.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101813406 2023-01-15T14:30:26.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244680+0000", 2023-01-15T14:30:26.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8045000000000001e-05 2023-01-15T14:30:26.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.062 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.062 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:948 3.8 e83/41 3:17d5d741:::smithi09025264-5:head v 83'119, mlcod=82'115)", 2023-01-15T14:30:26.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:30:26.063 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 56.293752877999999, 2023-01-15T14:30:26.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201378735, 2023-01-15T14:30:26.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:26.063 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:30:26.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350497+0000", 2023-01-15T14:30:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350496+0000", 2023-01-15T14:30:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350511+0000", 2023-01-15T14:30:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5007e-05 2023-01-15T14:30:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.350535+0000", 2023-01-15T14:30:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4170000000000001e-05 2023-01-15T14:30:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.450633+0000", 2023-01-15T14:30:26.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100098221 2023-01-15T14:30:26.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.550879+0000", 2023-01-15T14:30:26.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100245538 2023-01-15T14:30:26.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.550917+0000", 2023-01-15T14:30:26.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8043000000000002e-05 2023-01-15T14:30:26.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551222+0000", 2023-01-15T14:30:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030465400000000001 2023-01-15T14:30:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551241+0000", 2023-01-15T14:30:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9049999999999999e-05 2023-01-15T14:30:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551331+0000", 2023-01-15T14:30:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0807e-05 2023-01-15T14:30:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551356+0000", 2023-01-15T14:30:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4786999999999998e-05 2023-01-15T14:30:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:29.551876+0000", 2023-01-15T14:30:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051983800000000001 2023-01-15T14:30:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1124 3.5 e97/48 3:a75958b1:::smithi09025264-6:head v 97'44, mlcod=94'39)", 2023-01-15T14:30:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.506845804999998, 2023-01-15T14:30:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200993378, 2023-01-15T14:30:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137404+0000", 2023-01-15T14:30:26.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:26.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137455+0000", 2023-01-15T14:30:26.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1187000000000003e-05 2023-01-15T14:30:26.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137475+0000", 2023-01-15T14:30:26.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9692999999999999e-05 2023-01-15T14:30:26.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137496+0000", 2023-01-15T14:30:26.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1776e-05 2023-01-15T14:30:26.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238363+0000", 2023-01-15T14:30:26.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10086711900000001 2023-01-15T14:30:26.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238415+0000", 2023-01-15T14:30:26.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1465999999999998e-05 2023-01-15T14:30:26.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:26.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238578+0000", 2023-01-15T14:30:26.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016310599999999999 2023-01-15T14:30:26.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:26.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238598+0000", 2023-01-15T14:30:26.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9729999999999999e-05 2023-01-15T14:30:26.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:26.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238673+0000", 2023-01-15T14:30:26.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5302000000000005e-05 2023-01-15T14:30:26.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:26.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239068+0000", 2023-01-15T14:30:26.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000395227 2023-01-15T14:30:26.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.338398+0000", 2023-01-15T14:30:26.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099329534999999997 2023-01-15T14:30:26.221 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.221 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.221 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.221 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1125 3.5 e97/48 3:a75958b1:::smithi09025264-6:head v 97'45, mlcod=94'39)", 2023-01-15T14:30:26.221 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:30:26.221 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.506728592000002, 2023-01-15T14:30:26.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201211626, 2023-01-15T14:30:26.221 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.222 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:26.222 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:30:26.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137522+0000", 2023-01-15T14:30:26.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137521+0000", 2023-01-15T14:30:26.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:26.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137531+0000", 2023-01-15T14:30:26.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6399999999999992e-06 2023-01-15T14:30:26.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137546+0000", 2023-01-15T14:30:26.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5783000000000001e-05 2023-01-15T14:30:26.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.137557+0000", 2023-01-15T14:30:26.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0657e-05 2023-01-15T14:30:26.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238639+0000", 2023-01-15T14:30:26.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10108166 2023-01-15T14:30:26.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238688+0000", 2023-01-15T14:30:26.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9570000000000001e-05 2023-01-15T14:30:26.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:26.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238931+0000", 2023-01-15T14:30:26.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000243255 2023-01-15T14:30:26.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:26.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238948+0000", 2023-01-15T14:30:26.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6246e-05 2023-01-15T14:30:26.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:26.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238999+0000", 2023-01-15T14:30:26.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1063000000000002e-05 2023-01-15T14:30:26.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:26.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239101+0000", 2023-01-15T14:30:26.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010191000000000001 2023-01-15T14:30:26.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.338733+0000", 2023-01-15T14:30:26.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099632581999999997 2023-01-15T14:30:26.234 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.234 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.234 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.234 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1119 3.8 e97/41 3:1eaad226:::smithi09025264-49:head v 97'157, mlcod=97'154)", 2023-01-15T14:30:26.234 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:30:26.234 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.505110287999997, 2023-01-15T14:30:26.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200961838, 2023-01-15T14:30:26.234 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.235 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:26.235 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:30:26.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139140+0000", 2023-01-15T14:30:26.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139139+0000", 2023-01-15T14:30:26.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:26.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139291+0000", 2023-01-15T14:30:26.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015233900000000001 2023-01-15T14:30:26.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139318+0000", 2023-01-15T14:30:26.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6265999999999999e-05 2023-01-15T14:30:26.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.139331+0000", 2023-01-15T14:30:26.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3757000000000001e-05 2023-01-15T14:30:26.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238733+0000", 2023-01-15T14:30:26.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099401891000000006 2023-01-15T14:30:26.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.238778+0000", 2023-01-15T14:30:26.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5188999999999997e-05 2023-01-15T14:30:26.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:26.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239141+0000", 2023-01-15T14:30:26.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000362911 2023-01-15T14:30:26.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:26.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239155+0000", 2023-01-15T14:30:26.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3879e-05 2023-01-15T14:30:26.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:26.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239274+0000", 2023-01-15T14:30:26.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011916 2023-01-15T14:30:26.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:26.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.239310+0000", 2023-01-15T14:30:26.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5602999999999998e-05 2023-01-15T14:30:26.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:44.340102+0000", 2023-01-15T14:30:26.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10079187000000001 2023-01-15T14:30:26.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.241 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.242 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(3.0 REQUEST e99)", 2023-01-15T14:30:26.242 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:26.242 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.647836318000003, 2023-01-15T14:30:26.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26432840200000002, 2023-01-15T14:30:26.242 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.242 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:26.242 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:26.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996419+0000", 2023-01-15T14:30:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6729999999999997e-06 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996449+0000", 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.004e-05 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996463+0000", 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3607e-05 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996594+0000", 2023-01-15T14:30:26.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013148899999999999 2023-01-15T14:30:26.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:30:26.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996618+0000", 2023-01-15T14:30:26.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3249000000000001e-05 2023-01-15T14:30:26.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260645+0000", 2023-01-15T14:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26402768700000001 2023-01-15T14:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260671+0000", 2023-01-15T14:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.546e-05 2023-01-15T14:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260742+0000", 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1569999999999994e-05 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:261 3.8 e98/41 3:1e54a7f5:::smithi09025264-42:3e v 98'160, mlcod=97'157)", 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.586768728999999, 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20299468400000001, 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.365 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:26.366 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:30:26.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:30:26.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057480+0000", 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057496+0000", 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5869999999999999e-05 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057528+0000", 2023-01-15T14:30:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:30:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057545+0000", 2023-01-15T14:30:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.694e-05 2023-01-15T14:30:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.158971+0000", 2023-01-15T14:30:26.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101425425 2023-01-15T14:30:26.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159011+0000", 2023-01-15T14:30:26.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0058000000000002e-05 2023-01-15T14:30:26.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:26.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159858+0000", 2023-01-15T14:30:26.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084767900000000005 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159877+0000", 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9133000000000001e-05 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159959+0000", 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1693000000000005e-05 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260430+0000", 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10047039200000001 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260476+0000", 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6652e-05 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.372 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:30:26.372 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:30:26.372 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.585547572999999, 2023-01-15T14:30:26.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:30:26.372 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.372 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:26.372 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:30:26.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:30:26.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:30:26.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:26.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:30:26.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:30:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:30:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:30:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:30:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:30:26.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:30:26.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:30:26.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:30:26.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:26.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:30:26.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:30:26.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:26.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:30:26.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:30:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:30:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:30:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "replica_scrub(pg: 3.0,from:0'0,to:98'189,epoch:99/95,start:3:00000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2023-01-15T14:30:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:30:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.282407632000002, 2023-01-15T14:30:26.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20057929099999999, 2023-01-15T14:30:26.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:26.528 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:30:26.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361843+0000", 2023-01-15T14:30:26.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361841+0000", 2023-01-15T14:30:26.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:26.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361855+0000", 2023-01-15T14:30:26.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3464e-05 2023-01-15T14:30:26.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.361886+0000", 2023-01-15T14:30:26.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1389000000000001e-05 2023-01-15T14:30:26.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.462023+0000", 2023-01-15T14:30:26.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013720800000001 2023-01-15T14:30:26.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562267+0000", 2023-01-15T14:30:26.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100243392 2023-01-15T14:30:26.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562322+0000", 2023-01-15T14:30:26.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5075999999999999e-05 2023-01-15T14:30:26.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.562422+0000", 2023-01-15T14:30:26.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010015599999999999 2023-01-15T14:30:26.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.532 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.533 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(2.1 GRANT e100)", 2023-01-15T14:30:26.533 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:30:26.533 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.326585346000002, 2023-01-15T14:30:26.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20048780599999999, 2023-01-15T14:30:26.533 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.533 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:26.534 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:30:26.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317665+0000", 2023-01-15T14:30:26.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317663+0000", 2023-01-15T14:30:26.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:26.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317674+0000", 2023-01-15T14:30:26.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.119e-05 2023-01-15T14:30:26.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.317702+0000", 2023-01-15T14:30:26.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7183e-05 2023-01-15T14:30:26.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.417845+0000", 2023-01-15T14:30:26.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100142988 2023-01-15T14:30:26.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518107+0000", 2023-01-15T14:30:26.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100262461 2023-01-15T14:30:26.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518121+0000", 2023-01-15T14:30:26.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.449e-05 2023-01-15T14:30:26.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518124+0000", 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1730000000000002e-06 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.518153+0000", 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9043999999999998e-05 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.549 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:194 3.4 e101/48 3:2022fdfa:::smithi09025264-46:43 v 101'104, mlcod=98'100)", 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.799505961999998, 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20670733199999999, 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:30:26.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844743+0000", 2023-01-15T14:30:26.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:26.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844757+0000", 2023-01-15T14:30:26.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.484e-05 2023-01-15T14:30:26.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844787+0000", 2023-01-15T14:30:26.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9113000000000001e-05 2023-01-15T14:30:26.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.944923+0000", 2023-01-15T14:30:26.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013656 2023-01-15T14:30:26.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.949975+0000", 2023-01-15T14:30:26.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050521400000000001 2023-01-15T14:30:26.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950015+0000", 2023-01-15T14:30:26.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9286999999999998e-05 2023-01-15T14:30:26.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:26.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950559+0000", 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054463500000000002 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950585+0000", 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6166e-05 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950670+0000", 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4902999999999997e-05 2023-01-15T14:30:26.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:26.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051368+0000", 2023-01-15T14:30:26.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100697913 2023-01-15T14:30:26.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051452+0000", 2023-01-15T14:30:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3442000000000005e-05 2023-01-15T14:30:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1224 3.4 e107/48 3:2029fa71:::smithi09025264-39:head v 107'120, mlcod=104'116)", 2023-01-15T14:30:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:30:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.412477322000001, 2023-01-15T14:30:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22601005900000001, 2023-01-15T14:30:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:30:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:30:26.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231771+0000", 2023-01-15T14:30:26.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:26.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233829+0000", 2023-01-15T14:30:26.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002057836 2023-01-15T14:30:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233881+0000", 2023-01-15T14:30:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1856e-05 2023-01-15T14:30:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.333996+0000", 2023-01-15T14:30:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100115315 2023-01-15T14:30:26.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434199+0000", 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100203373 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434236+0000", 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6563000000000002e-05 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.435979+0000", 2023-01-15T14:30:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017426570000000001 2023-01-15T14:30:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.436036+0000", 2023-01-15T14:30:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7716000000000001e-05 2023-01-15T14:30:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437140+0000", 2023-01-15T14:30:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011036399999999999 2023-01-15T14:30:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437251+0000", 2023-01-15T14:30:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110869 2023-01-15T14:30:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.457783+0000", 2023-01-15T14:30:26.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020532181 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.13 120/114 [PushOp(3:cbfd2d71:::smithi09025264-37:head, version: 112'211, data_included: [3145728~182058], data_size: 182058, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:cbfd2d71:::smithi09025264-37:head@112'211, size: 3327786, copy_subset: [0~3327786], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3327786, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:06.903352+0000", 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.740898572999999, 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20099204800000001, 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:26.686 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903352+0000", 2023-01-15T14:30:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903352+0000", 2023-01-15T14:30:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903350+0000", 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903498+0000", 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000147918 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903537+0000", 2023-01-15T14:30:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8983000000000003e-05 2023-01-15T14:30:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:06.903564+0000", 2023-01-15T14:30:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7253000000000001e-05 2023-01-15T14:30:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.103937+0000", 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200373464 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.103953+0000", 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5243e-05 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.104344+0000", 2023-01-15T14:30:26.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039117399999999999 2023-01-15T14:30:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.832 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(3.9 RELEASE e121)", 2023-01-15T14:30:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:08.681807+0000", 2023-01-15T14:30:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.962443189999998, 2023-01-15T14:30:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20186040499999999, 2023-01-15T14:30:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.681807+0000", 2023-01-15T14:30:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.681807+0000", 2023-01-15T14:30:26.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.681805+0000", 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.681816+0000", 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0523000000000001e-05 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.681847+0000", 2023-01-15T14:30:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0818999999999997e-05 2023-01-15T14:30:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.781924+0000", 2023-01-15T14:30:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100077638 2023-01-15T14:30:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.883538+0000", 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10161368899999999 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.883574+0000", 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6273999999999999e-05 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:26.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.883667+0000", 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3119000000000001e-05 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "replica_scrub(pg: 3.4,from:0'0,to:125'143,epoch:135/111,start:3:20000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:22.255733+0000", 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.3885174569999998, 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200467808, 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:26.910 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.255733+0000", 2023-01-15T14:30:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.255733+0000", 2023-01-15T14:30:26.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.255731+0000", 2023-01-15T14:30:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.255747+0000", 2023-01-15T14:30:26.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:30:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:26.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.255783+0000", 2023-01-15T14:30:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6139000000000001e-05 2023-01-15T14:30:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.255798+0000", 2023-01-15T14:30:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5630000000000001e-05 2023-01-15T14:30:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.456132+0000", 2023-01-15T14:30:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20033369000000001 2023-01-15T14:30:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.456152+0000", 2023-01-15T14:30:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0211999999999999e-05 2023-01-15T14:30:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.456201+0000", 2023-01-15T14:30:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8257000000000002e-05 2023-01-15T14:30:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:27.008 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:27.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:30:27.009 INFO:tasks.rados.rados.0.smithi090.stdout:571: finishing rollback tid 1 to smithi09025264-50 2023-01-15T14:30:27.009 INFO:tasks.rados.rados.0.smithi090.stdout:571: finishing rollback tid 2 to smithi09025264-50 2023-01-15T14:30:27.009 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 50 v 115 (ObjNum 170 snap 37 seq_num 170) dirty exists 2023-01-15T14:30:27.009 INFO:tasks.rados.rados.0.smithi090.stdout:571: done (0 left) 2023-01-15T14:30:27.010 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:25.704+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:27.010 INFO:tasks.rados.rados.0.smithi090.stdout:573: done (0 left) 2023-01-15T14:30:27.010 INFO:tasks.rados.rados.0.smithi090.stdout:574: rollback oid 25 current snap is 54 2023-01-15T14:30:27.011 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 25 to 52 2023-01-15T14:30:27.011 INFO:tasks.rados.rados.0.smithi090.stdout:575: copy_from oid 49 from oid 36 current snap is 54 2023-01-15T14:30:27.011 INFO:tasks.rados.rados.0.smithi090.stdout:576: delete oid 9 current snap is 54 2023-01-15T14:30:27.012 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:30:27.013 INFO:teuthology.orchestra.run.smithi090.stderr:clear 2.2 pg_upmap mapping 2023-01-15T14:30:27.021 INFO:tasks.rados.rados.0.smithi090.stdout:576: done (2 left) 2023-01-15T14:30:27.021 INFO:tasks.rados.rados.0.smithi090.stdout:577: write oid 43 current snap is 54 2023-01-15T14:30:27.023 INFO:tasks.rados.rados.0.smithi090.stdout:577: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2023-01-15T14:30:27.023 INFO:tasks.rados.rados.0.smithi090.stdout:577: writing smithi09025264-43 from 749645 to 1523750 tid 1 2023-01-15T14:30:27.023 INFO:tasks.rados.rados.0.smithi090.stdout:577: writing smithi09025264-43 from 2085386 to 2694787 tid 2 2023-01-15T14:30:27.023 INFO:tasks.rados.rados.0.smithi090.stdout:577: writing smithi09025264-43 from 3237462 to 3949645 tid 3 2023-01-15T14:30:27.024 INFO:tasks.rados.rados.0.smithi090.stdout:578: copy_from oid 11 from oid 12 current snap is 54 2023-01-15T14:30:27.024 INFO:tasks.rados.rados.0.smithi090.stdout:579: rollback oid 16 current snap is 54 2023-01-15T14:30:27.024 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 16 to 52 2023-01-15T14:30:27.024 INFO:tasks.rados.rados.0.smithi090.stdout:580: read oid 1 snap 51 2023-01-15T14:30:27.024 INFO:tasks.rados.rados.0.smithi090.stdout:580: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:30:27.025 INFO:tasks.rados.rados.0.smithi090.stdout:581: read oid 34 snap -1 2023-01-15T14:30:27.025 INFO:tasks.rados.rados.0.smithi090.stdout:581: expect (ObjNum 228 snap 52 seq_num 228) 2023-01-15T14:30:27.025 INFO:tasks.rados.rados.0.smithi090.stdout:582: read oid 48 snap -1 2023-01-15T14:30:27.026 INFO:tasks.rados.rados.0.smithi090.stdout:582: expect (ObjNum 132 snap 24 seq_num 132) 2023-01-15T14:30:27.027 INFO:tasks.rados.rados.0.smithi090.stdout:583: delete oid 7 current snap is 54 2023-01-15T14:30:27.027 INFO:tasks.rados.rados.0.smithi090.stdout:581: expect (ObjNum 228 snap 52 seq_num 228) 2023-01-15T14:30:27.027 INFO:tasks.rados.rados.0.smithi090.stdout:574: finishing rollback tid 1 to smithi09025264-25 2023-01-15T14:30:27.028 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 0 (ObjNum 129 snap 0 seq_num 3192864672) dirty dne 2023-01-15T14:30:27.028 INFO:tasks.rados.rados.0.smithi090.stdout:575: finishing copy_from to smithi09025264-49 2023-01-15T14:30:27.028 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 154 (ObjNum 199 snap 43 seq_num 199) dirty exists 2023-01-15T14:30:27.028 INFO:tasks.rados.rados.0.smithi090.stdout:575: finishing copy_from racing read to smithi09025264-49 2023-01-15T14:30:27.028 INFO:tasks.rados.rados.0.smithi090.stdout:574: done (8 left) 2023-01-15T14:30:27.028 INFO:tasks.rados.rados.0.smithi090.stdout:575: done (7 left) 2023-01-15T14:30:27.028 INFO:tasks.rados.rados.0.smithi090.stdout:581: done (6 left) 2023-01-15T14:30:27.028 INFO:tasks.rados.rados.0.smithi090.stdout:583: done (5 left) 2023-01-15T14:30:27.028 INFO:tasks.rados.rados.0.smithi090.stdout:584: read oid 22 snap 53 2023-01-15T14:30:27.028 INFO:tasks.rados.rados.0.smithi090.stdout:584: expect (ObjNum 160 snap 35 seq_num 160) 2023-01-15T14:30:27.029 INFO:tasks.rados.rados.0.smithi090.stdout:585: write oid 18 current snap is 54 2023-01-15T14:30:27.029 INFO:tasks.rados.rados.0.smithi090.stdout:585: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2023-01-15T14:30:27.029 INFO:tasks.rados.rados.0.smithi090.stdout:585: writing smithi09025264-18 from 783259 to 1424215 tid 1 2023-01-15T14:30:27.029 INFO:tasks.rados.rados.0.smithi090.stdout:585: writing smithi09025264-18 from 1918703 to 2531773 tid 2 2023-01-15T14:30:27.029 INFO:tasks.rados.rados.0.smithi090.stdout:585: writing smithi09025264-18 from 3008661 to 3414128 tid 3 2023-01-15T14:30:27.029 INFO:tasks.rados.rados.0.smithi090.stdout:585: writing smithi09025264-18 from 3821939 to 3983259 tid 4 2023-01-15T14:30:27.029 INFO:tasks.rados.rados.0.smithi090.stdout:586: copy_from oid 28 from oid 27 current snap is 54 2023-01-15T14:30:27.029 INFO:tasks.rados.rados.0.smithi090.stdout:587: write oid 46 current snap is 54 2023-01-15T14:30:27.029 INFO:tasks.rados.rados.0.smithi090.stdout:587: seq_num 238 ranges {400066=716205,1860832=139234} 2023-01-15T14:30:27.029 INFO:tasks.rados.rados.0.smithi090.stdout:587: writing smithi09025264-46 from 400066 to 1116271 tid 1 2023-01-15T14:30:27.030 INFO:tasks.rados.rados.0.smithi090.stdout:587: writing smithi09025264-46 from 1860832 to 2000066 tid 2 2023-01-15T14:30:27.030 INFO:tasks.rados.rados.0.smithi090.stdout:588: write oid 26 current snap is 54 2023-01-15T14:30:27.030 INFO:tasks.rados.rados.0.smithi090.stdout:588: seq_num 239 ranges {416873=791454,1919314=97559} 2023-01-15T14:30:27.030 INFO:tasks.rados.rados.0.smithi090.stdout:588: writing smithi09025264-26 from 416873 to 1208327 tid 1 2023-01-15T14:30:27.030 INFO:tasks.rados.rados.0.smithi090.stdout:588: writing smithi09025264-26 from 1919314 to 2016873 tid 2 2023-01-15T14:30:27.030 INFO:tasks.rados.rados.0.smithi090.stdout:589: snap_create 2023-01-15T14:30:27.030 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 10 2023-01-15T14:30:27.030 INFO:tasks.rados.rados.0.smithi090.stdout:580: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:30:27.030 INFO:tasks.rados.rados.0.smithi090.stdout:582: expect (ObjNum 132 snap 24 seq_num 132) 2023-01-15T14:30:27.030 INFO:tasks.rados.rados.0.smithi090.stdout:584: expect (ObjNum 160 snap 35 seq_num 160) 2023-01-15T14:30:27.030 INFO:tasks.rados.rados.0.smithi090.stdout:578: finishing copy_from to smithi09025264-11 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 205 (ObjNum 185 snap 40 seq_num 185) dirty exists 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout:578: finishing copy_from racing read to smithi09025264-11 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout:586: finishing copy_from to smithi09025264-28 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 41 (ObjNum 101 snap 21 seq_num 101) dirty exists 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout:578: done (9 left) 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout:580: done (8 left) 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout:582: done (7 left) 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout:584: done (6 left) 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout:586: finishing copy_from racing read to smithi09025264-28 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout:586: done (5 left) 2023-01-15T14:30:27.031 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:30:27.032 INFO:tasks.rados.rados.0.smithi090.stdout:577: finishing write tid 1 to smithi09025264-43 2023-01-15T14:30:27.032 INFO:tasks.rados.rados.0.smithi090.stdout:577: finishing write tid 2 to smithi09025264-43 2023-01-15T14:30:27.032 INFO:tasks.rados.rados.0.smithi090.stdout:577: finishing write tid 3 to smithi09025264-43 2023-01-15T14:30:27.032 INFO:tasks.rados.rados.0.smithi090.stdout:579: finishing rollback tid 0 to smithi09025264-16 2023-01-15T14:30:27.032 INFO:tasks.rados.rados.0.smithi090.stdout:587: finishing write tid 1 to smithi09025264-46 2023-01-15T14:30:27.032 INFO:tasks.rados.rados.0.smithi090.stdout:587: finishing write tid 2 to smithi09025264-46 2023-01-15T14:30:27.032 INFO:tasks.rados.rados.0.smithi090.stdout:577: finishing write tid 4 to smithi09025264-43 2023-01-15T14:30:27.032 INFO:tasks.rados.rados.0.smithi090.stdout:579: finishing rollback tid 1 to smithi09025264-16 2023-01-15T14:30:27.032 INFO:tasks.rados.rados.0.smithi090.stdout:579: finishing rollback tid 2 to smithi09025264-16 2023-01-15T14:30:27.032 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 134 (ObjNum 209 snap 47 seq_num 209) dirty exists 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout:585: finishing write tid 1 to smithi09025264-18 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout:579: done (4 left) 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout:587: finishing write tid 3 to smithi09025264-46 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout:587: finishing write tid 4 to smithi09025264-46 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout:587: oid 46 updating version 0 to 151 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout:587: oid 46 updating version 151 to 152 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout:587: oid 46 version 152 is already newer than 150 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 152 (ObjNum 238 snap 54 seq_num 238) dirty exists 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout:587: left oid 46 (ObjNum 238 snap 54 seq_num 238) 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout:587: done (3 left) 2023-01-15T14:30:27.033 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:585: finishing write tid 2 to smithi09025264-18 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:585: finishing write tid 3 to smithi09025264-18 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:585: finishing write tid 4 to smithi09025264-18 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:585: finishing write tid 5 to smithi09025264-18 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:585: finishing write tid 6 to smithi09025264-18 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:585: oid 18 updating version 0 to 119 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:585: oid 18 updating version 119 to 123 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:585: oid 18 version 123 is already newer than 121 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:585: oid 18 version 123 is already newer than 120 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:585: oid 18 version 123 is already newer than 122 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 123 (ObjNum 237 snap 54 seq_num 237) dirty exists 2023-01-15T14:30:27.034 INFO:tasks.rados.rados.0.smithi090.stdout:585: left oid 18 (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout:585: done (2 left) 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout:577: finishing write tid 5 to smithi09025264-43 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout:577: oid 43 updating version 0 to 68 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout:577: oid 43 version 68 is already newer than 66 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout:577: oid 43 updating version 68 to 69 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout:577: oid 43 version 69 is already newer than 67 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 69 (ObjNum 235 snap 54 seq_num 235) dirty exists 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout:577: left oid 43 (ObjNum 235 snap 54 seq_num 235) 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout:577: done (1 left) 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:27.035 INFO:tasks.rados.rados.0.smithi090.stdout:588: finishing write tid 1 to smithi09025264-26 2023-01-15T14:30:27.036 INFO:tasks.rados.rados.0.smithi090.stdout:588: finishing write tid 2 to smithi09025264-26 2023-01-15T14:30:27.036 INFO:tasks.rados.rados.0.smithi090.stdout:588: finishing write tid 3 to smithi09025264-26 2023-01-15T14:30:27.036 INFO:tasks.rados.rados.0.smithi090.stdout:588: finishing write tid 4 to smithi09025264-26 2023-01-15T14:30:27.036 INFO:tasks.rados.rados.0.smithi090.stdout:588: oid 26 updating version 0 to 227 2023-01-15T14:30:27.036 INFO:tasks.rados.rados.0.smithi090.stdout:588: oid 26 updating version 227 to 228 2023-01-15T14:30:27.036 INFO:tasks.rados.rados.0.smithi090.stdout:588: oid 26 updating version 228 to 229 2023-01-15T14:30:27.036 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 229 (ObjNum 239 snap 54 seq_num 239) dirty exists 2023-01-15T14:30:27.036 INFO:tasks.rados.rados.0.smithi090.stdout:588: left oid 26 (ObjNum 239 snap 54 seq_num 239) 2023-01-15T14:30:27.036 INFO:tasks.rados.rados.0.smithi090.stdout:588: done (0 left) 2023-01-15T14:30:27.037 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:27.012+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:27.116 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:27.111+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:27.171 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:27.171 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:27.171 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:30:27.171 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:27.181 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:30:27.216 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:27.215+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:27.297 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:27.297 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:27.298 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:27.298 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:27.298 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:27.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:30:27.317 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:27.315+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:27.406 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:166 3.8 3:1e54a7f5:::smithi09025264-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:27.493 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.28682181400001, 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62664632300000001, 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 166 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114713+0000", 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:27.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115178+0000", 2023-01-15T14:30:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046493199999999999 2023-01-15T14:30:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115208+0000", 2023-01-15T14:30:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0689999999999999e-05 2023-01-15T14:30:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115221+0000", 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3013000000000001e-05 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115334+0000", 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112403 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:27.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115505+0000", 2023-01-15T14:30:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170915 2023-01-15T14:30:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333496+0000", 2023-01-15T14:30:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217991667 2023-01-15T14:30:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333606+0000", 2023-01-15T14:30:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109967 2023-01-15T14:30:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:30:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333796+0000", 2023-01-15T14:30:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018928099999999999 2023-01-15T14:30:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334289+0000", 2023-01-15T14:30:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049322600000000004 2023-01-15T14:30:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334339+0000", 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0439999999999998e-05 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334990+0000", 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065082300000000003 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335100+0000", 2023-01-15T14:30:27.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109357 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.640854+0000", 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30575451300000001 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741190+0000", 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100335568 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741263+0000", 2023-01-15T14:30:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3275999999999994e-05 2023-01-15T14:30:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741361+0000", 2023-01-15T14:30:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8435000000000002e-05 2023-01-15T14:30:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.503 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:167 3.8 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:30:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.28625712500001, 2023-01-15T14:30:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62638390099999997, 2023-01-15T14:30:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:27.504 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 167 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:27.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:30:27.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:27.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115279+0000", 2023-01-15T14:30:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115289+0000", 2023-01-15T14:30:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0509999999999999e-05 2023-01-15T14:30:27.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115306+0000", 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7260000000000001e-05 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115337+0000", 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0664000000000003e-05 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115579+0000", 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024194400000000001 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115744+0000", 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016493700000000001 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334392+0000", 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21864830299999999 2023-01-15T14:30:27.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334453+0000", 2023-01-15T14:30:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1047e-05 2023-01-15T14:30:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:30:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334601+0000", 2023-01-15T14:30:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014824700000000001 2023-01-15T14:30:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334812+0000", 2023-01-15T14:30:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210208 2023-01-15T14:30:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334996+0000", 2023-01-15T14:30:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183878 2023-01-15T14:30:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335126+0000", 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013001399999999999 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335160+0000", 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4876999999999998e-05 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741044+0000", 2023-01-15T14:30:27.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40588326000000002 2023-01-15T14:30:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741453+0000", 2023-01-15T14:30:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040963099999999998 2023-01-15T14:30:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741548+0000", 2023-01-15T14:30:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4961000000000001e-05 2023-01-15T14:30:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741664+0000", 2023-01-15T14:30:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115333 2023-01-15T14:30:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:30:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.28616501099999, 2023-01-15T14:30:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:30:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:27.550 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:30:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:30:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:30:27.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:30:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:30:27.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:30:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:30:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:30:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:30:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:30:27.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:30:27.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:27.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:30:27.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:30:27.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:27.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:30:27.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:30:27.649 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.649 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:27.649 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.649 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:27.649 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:30:27.649 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.27317293199999, 2023-01-15T14:30:27.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:30:27.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:27.650 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:27.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:27.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:27.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:27.650 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:30:27.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.650 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:27.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:27.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:30:27.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:27.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:30:27.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:27.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:30:27.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:27.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:27.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:30:27.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:30:27.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:27.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:30:27.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:30:27.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:27.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:30:27.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:30:27.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:30:27.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:30:27.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:27.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:30:27.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:30:27.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:30:27.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:30:27.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:27.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:30:27.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:30:27.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:30:27.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:30:27.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:30:27.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:27.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:30:27.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:30:27.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:27.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:30:27.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:30:27.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:27.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:30:27.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:30:27.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:27.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:30:27.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:30:27.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:30:27.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:30:27.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:30:27.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:30:27.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:27.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:30:27.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:30:27.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:27.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:30:27.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:30:27.660 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.660 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:27.660 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.660 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:27.660 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:30:27.661 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.272769596, 2023-01-15T14:30:27.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:30:27.661 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:27.661 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:27.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:27.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:27.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:27.661 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:30:27.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.661 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:27.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:27.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:30:27.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:27.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:30:27.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:27.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:30:27.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:27.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:27.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:30:27.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:30:27.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:27.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:30:27.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:30:27.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:27.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:30:27.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:30:27.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:30:27.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:30:27.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:27.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:30:27.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:30:27.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:30:27.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:30:27.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:27.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:30:27.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:30:27.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:27.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:30:27.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:30:27.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:30:27.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:30:27.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:30:27.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:30:27.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:27.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:30:27.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:30:27.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:27.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:30:27.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:30:27.671 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.671 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:27.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.672 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:27.672 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:30:27.672 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.17254597599999, 2023-01-15T14:30:27.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:30:27.673 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:27.673 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:27.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:27.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:27.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:27.673 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:30:27.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.674 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:27.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:27.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:30:27.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:27.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:30:27.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:27.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:30:27.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:27.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:27.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:30:27.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:30:27.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:27.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:30:27.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:30:27.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:27.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:30:27.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:30:27.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:30:27.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:30:27.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:27.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:30:27.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:30:27.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:30:27.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:30:27.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:27.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:30:27.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:30:27.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:30:27.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:30:27.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:27.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:30:27.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:30:27.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:27.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:30:27.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:30:27.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.787 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:27.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.787 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:194 3.8 3:1eaad226:::smithi09025264-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:27.788 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:30:27.788 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.10014473299999, 2023-01-15T14:30:27.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64372577200000003, 2023-01-15T14:30:27.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:27.813 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:27.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:27.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:27.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:27.814 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 194 2023-01-15T14:30:27.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.814 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:27.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:27.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:30:27.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:27.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:30:27.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:27.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301390+0000", 2023-01-15T14:30:27.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:27.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:27.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301991+0000", 2023-01-15T14:30:27.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060122599999999995 2023-01-15T14:30:27.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:27.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.302026+0000", 2023-01-15T14:30:27.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5040000000000003e-05 2023-01-15T14:30:27.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:27.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402147+0000", 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100121084 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402219+0000", 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2045999999999994e-05 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402315+0000", 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5561000000000002e-05 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842189+0000", 2023-01-15T14:30:27.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.439874129 2023-01-15T14:30:27.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:27.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842296+0000", 2023-01-15T14:30:27.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107021 2023-01-15T14:30:27.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:30:27.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842486+0000", 2023-01-15T14:30:27.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018965799999999999 2023-01-15T14:30:27.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:27.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842839+0000", 2023-01-15T14:30:27.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035330700000000002 2023-01-15T14:30:27.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:27.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842859+0000", 2023-01-15T14:30:27.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0347e-05 2023-01-15T14:30:27.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:27.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843212+0000", 2023-01-15T14:30:27.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352534 2023-01-15T14:30:27.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:27.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843835+0000", 2023-01-15T14:30:27.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062338599999999997 2023-01-15T14:30:27.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944861+0000", 2023-01-15T14:30:27.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10102596899999999 2023-01-15T14:30:27.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944908+0000", 2023-01-15T14:30:27.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6773000000000001e-05 2023-01-15T14:30:27.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:27.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945039+0000", 2023-01-15T14:30:27.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130684 2023-01-15T14:30:27.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:27.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945118+0000", 2023-01-15T14:30:27.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9194000000000001e-05 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:525 3.8 3:17d5d741:::smithi09025264-5:head [write 2555481~425187] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.582579807000002, 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60814212099999998, 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:27.826 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:27.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:27.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:27.827 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2023-01-15T14:30:27.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.827 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:27.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:27.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:30:27.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:27.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:30:27.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818955+0000", 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819151+0000", 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632699999999999 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:27.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819175+0000", 2023-01-15T14:30:27.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3777e-05 2023-01-15T14:30:27.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:27.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819185+0000", 2023-01-15T14:30:27.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0424999999999999e-05 2023-01-15T14:30:27.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019555+0000", 2023-01-15T14:30:27.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20036944200000001 2023-01-15T14:30:27.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:27.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019726+0000", 2023-01-15T14:30:27.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017075099999999999 2023-01-15T14:30:27.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224068+0000", 2023-01-15T14:30:27.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20434244100000001 2023-01-15T14:30:27.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:27.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224217+0000", 2023-01-15T14:30:27.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014933200000000001 2023-01-15T14:30:27.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:30:27.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224571+0000", 2023-01-15T14:30:27.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035323700000000001 2023-01-15T14:30:27.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:27.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225080+0000", 2023-01-15T14:30:27.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050910500000000004 2023-01-15T14:30:27.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:27.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225103+0000", 2023-01-15T14:30:27.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3295999999999999e-05 2023-01-15T14:30:27.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:27.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225542+0000", 2023-01-15T14:30:27.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043897599999999999 2023-01-15T14:30:27.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:27.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226915+0000", 2023-01-15T14:30:27.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013730599999999999 2023-01-15T14:30:27.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.327618+0000", 2023-01-15T14:30:27.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100702655 2023-01-15T14:30:27.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426890+0000", 2023-01-15T14:30:27.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099272284000000002 2023-01-15T14:30:27.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:27.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426977+0000", 2023-01-15T14:30:27.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6500000000000002e-05 2023-01-15T14:30:27.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:27.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427099+0000", 2023-01-15T14:30:27.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122562 2023-01-15T14:30:27.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.945 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:27.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.945 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:526 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (56) in=7b,truncate 3512630] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:30:27.945 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:30:27.945 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.582332074999997, 2023-01-15T14:30:27.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60866722500000003, 2023-01-15T14:30:27.946 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:27.946 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:27.946 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:27.946 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:27.946 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:27.946 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2023-01-15T14:30:27.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.946 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:27.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:27.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:30:27.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:27.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:30:27.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:27.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:27.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819204+0000", 2023-01-15T14:30:27.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:27.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:27.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819210+0000", 2023-01-15T14:30:27.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5429999999999999e-06 2023-01-15T14:30:27.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819235+0000", 2023-01-15T14:30:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4114e-05 2023-01-15T14:30:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819246+0000", 2023-01-15T14:30:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1043000000000001e-05 2023-01-15T14:30:27.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019911+0000", 2023-01-15T14:30:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200665175 2023-01-15T14:30:27.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020085+0000", 2023-01-15T14:30:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000174449 2023-01-15T14:30:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225345+0000", 2023-01-15T14:30:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20526017099999999 2023-01-15T14:30:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225560+0000", 2023-01-15T14:30:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021505499999999999 2023-01-15T14:30:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:30:27.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225908+0000", 2023-01-15T14:30:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034754600000000001 2023-01-15T14:30:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226267+0000", 2023-01-15T14:30:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035936399999999998 2023-01-15T14:30:27.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226688+0000", 2023-01-15T14:30:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042024599999999999 2023-01-15T14:30:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226841+0000", 2023-01-15T14:30:27.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015324499999999999 2023-01-15T14:30:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226936+0000", 2023-01-15T14:30:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5549000000000003e-05 2023-01-15T14:30:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426826+0000", 2023-01-15T14:30:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19988924 2023-01-15T14:30:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:27.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427685+0000", 2023-01-15T14:30:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000859719 2023-01-15T14:30:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427781+0000", 2023-01-15T14:30:27.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5515999999999998e-05 2023-01-15T14:30:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427872+0000", 2023-01-15T14:30:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1142000000000003e-05 2023-01-15T14:30:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:27.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:527 3.8 3:17d5d741:::smithi09025264-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:30:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.582281586999997, 2023-01-15T14:30:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60906366599999995, 2023-01-15T14:30:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:27.987 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:27.988 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:28.096 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2023-01-15T14:30:28.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.096 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819259+0000", 2023-01-15T14:30:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5940000000000004e-06 2023-01-15T14:30:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819268+0000", 2023-01-15T14:30:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.115e-06 2023-01-15T14:30:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.919405+0000", 2023-01-15T14:30:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013656999999999 2023-01-15T14:30:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020192+0000", 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10078725600000001 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020326+0000", 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133745 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226443+0000", 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20611705699999999 2023-01-15T14:30:28.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:28.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226689+0000", 2023-01-15T14:30:28.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024580799999999997 2023-01-15T14:30:28.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427991+0000", 2023-01-15T14:30:28.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20130269000000001 2023-01-15T14:30:28.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:28.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.428147+0000", 2023-01-15T14:30:28.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155258 2023-01-15T14:30:28.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:28.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.428319+0000", 2023-01-15T14:30:28.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017226200000000001 2023-01-15T14:30:28.102 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.102 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.132 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:705 3.2 3:47dfe935:::smithi09025264-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2023-01-15T14:30:28.133 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:30:28.133 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.290952825000005, 2023-01-15T14:30:28.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61170208599999998, 2023-01-15T14:30:28.133 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:28.133 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:28.133 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:28.133 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:28.133 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:28.133 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 705 2023-01-15T14:30:28.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.134 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:28.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:28.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:30:28.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:28.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:30:28.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110583+0000", 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110587+0000", 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9689999999999996e-06 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:28.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110601+0000", 2023-01-15T14:30:28.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.393e-05 2023-01-15T14:30:28.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:28.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110611+0000", 2023-01-15T14:30:28.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3999999999999998e-06 2023-01-15T14:30:28.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319174+0000", 2023-01-15T14:30:28.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20856376800000001 2023-01-15T14:30:28.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:28.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319367+0000", 2023-01-15T14:30:28.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192361 2023-01-15T14:30:28.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.721941+0000", 2023-01-15T14:30:28.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40257461300000003 2023-01-15T14:30:28.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:28.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722108+0000", 2023-01-15T14:30:28.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167054 2023-01-15T14:30:28.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:28.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722286+0000", 2023-01-15T14:30:28.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017772099999999999 2023-01-15T14:30:28.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.139 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.139 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:28.139 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:30:28.139 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.479230750999999, 2023-01-15T14:30:28.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:30:28.139 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:28.139 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:28.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:28.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:28.140 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:28.140 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:30:28.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.140 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:28.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:28.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:30:28.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:28.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:30:28.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:28.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:30:28.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:28.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:28.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:30:28.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:30:28.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:30:28.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:30:28.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:28.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:30:28.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:30:28.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:30:28.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:30:28.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:28.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:30:28.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:30:28.153 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.153 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.153 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.154 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:28.154 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:30:28.154 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.472769397, 2023-01-15T14:30:28.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:30:28.154 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:28.155 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:28.155 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:28.155 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:28.155 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:28.155 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:30:28.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.156 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:28.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:28.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:30:28.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:28.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:30:28.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:28.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:30:28.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:28.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:28.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:30:28.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:30:28.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:28.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:30:28.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:30:28.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:28.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:30:28.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:30:28.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:30:28.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:30:28.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:28.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:30:28.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:30:28.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:30:28.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:30:28.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:28.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:30:28.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:30:28.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.163 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.164 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:28.164 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:30:28.164 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.372347472000001, 2023-01-15T14:30:28.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:30:28.164 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:28.164 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:28.164 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:28.164 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:28.164 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:28.164 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:30:28.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.165 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:28.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:28.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:30:28.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:28.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:30:28.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:28.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:30:28.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:28.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:28.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:30:28.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:30:28.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:30:28.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:28.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:30:28.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:30:28.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:30:28.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:30:28.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:28.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:30:28.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:30:28.274 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.274 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.274 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.274 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:28.275 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:30:28.275 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.372009985999998, 2023-01-15T14:30:28.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:30:28.275 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:28.275 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:28.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:28.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:28.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:28.276 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:30:28.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.277 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:28.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:28.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:30:28.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:28.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:30:28.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:28.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:30:28.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:30:28.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:28.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:30:28.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:30:28.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:30:28.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:30:28.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:28.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:30:28.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:30:28.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.282 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:28.282 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:30:28.282 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.371954089999999, 2023-01-15T14:30:28.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:30:28.282 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:28.282 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:28.282 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:28.282 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:28.283 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:28.283 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:30:28.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.283 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:28.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:28.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:30:28.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:28.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:30:28.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:28.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:30:28.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:28.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:28.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:30:28.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:30:28.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:28.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:30:28.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:30:28.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:28.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:30:28.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:30:28.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:30:28.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:30:28.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:28.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:30:28.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:30:28.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:30:28.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:30:28.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:28.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:30:28.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:30:28.300 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.300 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.300 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.300 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1314 3.18 3.afe52a78 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:30:28.300 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:30:28.300 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.041658958999999, 2023-01-15T14:30:28.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62560938700000002, 2023-01-15T14:30:28.301 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:28.301 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:28.301 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:28.301 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:28.301 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:28.301 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1314 2023-01-15T14:30:28.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.301 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:28.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:28.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:30:28.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:28.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:30:28.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:28.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359877+0000", 2023-01-15T14:30:28.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:28.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:28.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359882+0000", 2023-01-15T14:30:28.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0050000000000004e-06 2023-01-15T14:30:28.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:28.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359893+0000", 2023-01-15T14:30:28.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0676000000000001e-05 2023-01-15T14:30:28.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:28.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359904+0000", 2023-01-15T14:30:28.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0788000000000001e-05 2023-01-15T14:30:28.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460094+0000", 2023-01-15T14:30:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10019024999999999 2023-01-15T14:30:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460106+0000", 2023-01-15T14:30:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354999999999999e-05 2023-01-15T14:30:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985477+0000", 2023-01-15T14:30:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52537039100000005 2023-01-15T14:30:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985487+0000", 2023-01-15T14:30:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:30:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.307 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1335 3.8 3:17d5d741:::smithi09025264-5:head [write 3252761~175867] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:30:28.307 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:30:28.307 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.469639698000002, 2023-01-15T14:30:28.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66052243499999996, 2023-01-15T14:30:28.307 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:28.307 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:28.307 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:28.307 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:28.307 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:28.307 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1335 2023-01-15T14:30:28.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.308 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:28.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:28.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:30:28.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:28.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:30:28.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931896+0000", 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931983+0000", 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6470000000000004e-05 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:28.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931995+0000", 2023-01-15T14:30:28.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2075e-05 2023-01-15T14:30:28.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:28.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932003+0000", 2023-01-15T14:30:28.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5059999999999995e-06 2023-01-15T14:30:28.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932346+0000", 2023-01-15T14:30:28.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034280299999999998 2023-01-15T14:30:28.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:28.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932357+0000", 2023-01-15T14:30:28.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.13e-05 2023-01-15T14:30:28.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189006+0000", 2023-01-15T14:30:28.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25664859899999998 2023-01-15T14:30:28.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:28.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189336+0000", 2023-01-15T14:30:28.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033030899999999998 2023-01-15T14:30:28.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:28.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189481+0000", 2023-01-15T14:30:28.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014484400000000001 2023-01-15T14:30:28.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:28.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189768+0000", 2023-01-15T14:30:28.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028704300000000001 2023-01-15T14:30:28.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:28.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189787+0000", 2023-01-15T14:30:28.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9343000000000001e-05 2023-01-15T14:30:28.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:28.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191361+0000", 2023-01-15T14:30:28.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015734029999999999 2023-01-15T14:30:28.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:28.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191469+0000", 2023-01-15T14:30:28.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010859899999999999 2023-01-15T14:30:28.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:28.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391274+0000", 2023-01-15T14:30:28.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19980450899999999 2023-01-15T14:30:28.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:28.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592067+0000", 2023-01-15T14:30:28.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20079332 2023-01-15T14:30:28.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:28.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592190+0000", 2023-01-15T14:30:28.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012291400000000001 2023-01-15T14:30:28.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:28.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592420+0000", 2023-01-15T14:30:28.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000229371 2023-01-15T14:30:28.378 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.378 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.378 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.378 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1336 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (56) in=7b,truncate 3428628] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:30:28.378 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.469522608999998, 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66113910799999998, 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1336 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:28.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:30:28.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:28.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:30:28.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:28.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932013+0000", 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932019+0000", 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5380000000000002e-06 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932027+0000", 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5150000000000006e-06 2023-01-15T14:30:28.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:28.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932034+0000", 2023-01-15T14:30:28.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2023-01-15T14:30:28.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032497+0000", 2023-01-15T14:30:28.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100463001 2023-01-15T14:30:28.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:28.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032518+0000", 2023-01-15T14:30:28.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1359999999999999e-05 2023-01-15T14:30:28.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189826+0000", 2023-01-15T14:30:28.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15730834299999999 2023-01-15T14:30:28.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:28.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189951+0000", 2023-01-15T14:30:28.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124583 2023-01-15T14:30:28.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:28.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190093+0000", 2023-01-15T14:30:28.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014221399999999999 2023-01-15T14:30:28.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190253+0000", 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015953000000000001 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191365+0000", 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011123190000000001 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:28.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191439+0000", 2023-01-15T14:30:28.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3648999999999999e-05 2023-01-15T14:30:28.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:28.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191491+0000", 2023-01-15T14:30:28.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2444999999999997e-05 2023-01-15T14:30:28.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:28.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391472+0000", 2023-01-15T14:30:28.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199981193 2023-01-15T14:30:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592619+0000", 2023-01-15T14:30:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20114642499999999 2023-01-15T14:30:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:28.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592748+0000", 2023-01-15T14:30:28.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012950200000000001 2023-01-15T14:30:28.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:28.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593153+0000", 2023-01-15T14:30:28.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040503700000000002 2023-01-15T14:30:28.444 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.445 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.445 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1337 3.8 3:17d5d741:::smithi09025264-5:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:30:28.445 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:30:28.445 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.46949438, 2023-01-15T14:30:28.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66179168200000005, 2023-01-15T14:30:28.445 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:28.445 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:28.445 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:28.446 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:28.446 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:28.446 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1337 2023-01-15T14:30:28.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.446 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:28.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:28.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:30:28.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:28.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:30:28.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:28.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:28.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:30:28.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:28.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:28.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932046+0000", 2023-01-15T14:30:28.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8089999999999999e-06 2023-01-15T14:30:28.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:28.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932055+0000", 2023-01-15T14:30:28.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2199999999999998e-06 2023-01-15T14:30:28.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:28.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932061+0000", 2023-01-15T14:30:28.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2779999999999999e-06 2023-01-15T14:30:28.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190318+0000", 2023-01-15T14:30:28.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.258256967 2023-01-15T14:30:28.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:28.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190413+0000", 2023-01-15T14:30:28.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4627000000000001e-05 2023-01-15T14:30:28.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:28.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593288+0000", 2023-01-15T14:30:28.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.402875339 2023-01-15T14:30:28.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:28.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593515+0000", 2023-01-15T14:30:28.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022700699999999999 2023-01-15T14:30:28.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:28.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:28.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:28.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593834+0000", 2023-01-15T14:30:28.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031906799999999999 2023-01-15T14:30:28.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.620 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:28.620 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:28.620 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:28.621 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:30:28.622 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:28.619+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:28.623 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:30:28.626 INFO:tasks.rados.rados.0.smithi090.stdout:589: done (0 left) 2023-01-15T14:30:28.626 INFO:tasks.rados.rados.0.smithi090.stdout:590: rollback oid 20 current snap is 55 2023-01-15T14:30:28.626 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 20 to 51 2023-01-15T14:30:28.627 INFO:tasks.rados.rados.0.smithi090.stdout:591: copy_from oid 5 from oid 34 current snap is 55 2023-01-15T14:30:28.627 INFO:tasks.rados.rados.0.smithi090.stdout:592: read oid 26 snap -1 2023-01-15T14:30:28.627 INFO:tasks.rados.rados.0.smithi090.stdout:592: expect (ObjNum 239 snap 54 seq_num 239) 2023-01-15T14:30:28.627 INFO:tasks.rados.rados.0.smithi090.stdout:593: write (excl) oid 40 current snap is 55 2023-01-15T14:30:28.627 INFO:tasks.rados.rados.0.smithi090.stdout:593: seq_num 240 ranges {433680=466703,1577796=455884} 2023-01-15T14:30:28.627 INFO:tasks.rados.rados.0.smithi090.stdout:593: writing smithi09025264-40 from 433680 to 900383 tid 1 2023-01-15T14:30:28.627 INFO:tasks.rados.rados.0.smithi090.stdout:593: writing smithi09025264-40 from 1577796 to 2033680 tid 2 2023-01-15T14:30:28.627 INFO:tasks.rados.rados.0.smithi090.stdout:594: delete oid 46 current snap is 55 2023-01-15T14:30:28.627 INFO:tasks.rados.rados.0.smithi090.stdout:590: finishing rollback tid 1 to smithi09025264-20 2023-01-15T14:30:28.627 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 0 (ObjNum 65 snap 0 seq_num 3204318592) dirty dne 2023-01-15T14:30:28.628 INFO:tasks.rados.rados.0.smithi090.stdout:592: expect (ObjNum 239 snap 54 seq_num 239) 2023-01-15T14:30:28.628 INFO:tasks.rados.rados.0.smithi090.stdout:593: finishing write tid 1 to smithi09025264-40 2023-01-15T14:30:28.628 INFO:tasks.rados.rados.0.smithi090.stdout:593: finishing write tid 2 to smithi09025264-40 2023-01-15T14:30:28.628 INFO:tasks.rados.rados.0.smithi090.stdout:593: finishing write tid 3 to smithi09025264-40 2023-01-15T14:30:28.628 INFO:tasks.rados.rados.0.smithi090.stdout:591: finishing copy_from to smithi09025264-5 2023-01-15T14:30:28.628 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 129 (ObjNum 228 snap 52 seq_num 228) dirty exists 2023-01-15T14:30:28.628 INFO:tasks.rados.rados.0.smithi090.stdout:591: finishing copy_from racing read to smithi09025264-5 2023-01-15T14:30:28.628 INFO:tasks.rados.rados.0.smithi090.stdout:593: finishing write tid 4 to smithi09025264-40 2023-01-15T14:30:28.628 INFO:tasks.rados.rados.0.smithi090.stdout:593: oid 40 updating version 0 to 154 2023-01-15T14:30:28.628 INFO:tasks.rados.rados.0.smithi090.stdout:593: oid 40 updating version 154 to 156 2023-01-15T14:30:28.629 INFO:tasks.rados.rados.0.smithi090.stdout:593: oid 40 version 156 is already newer than 155 2023-01-15T14:30:28.629 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 156 (ObjNum 240 snap 55 seq_num 240) dirty exists 2023-01-15T14:30:28.629 INFO:tasks.rados.rados.0.smithi090.stdout:593: left oid 40 (ObjNum 240 snap 55 seq_num 240) 2023-01-15T14:30:28.629 INFO:tasks.rados.rados.0.smithi090.stdout:590: done (4 left) 2023-01-15T14:30:28.629 INFO:tasks.rados.rados.0.smithi090.stdout:591: done (3 left) 2023-01-15T14:30:28.629 INFO:tasks.rados.rados.0.smithi090.stdout:592: done (2 left) 2023-01-15T14:30:28.629 INFO:tasks.rados.rados.0.smithi090.stdout:593: done (1 left) 2023-01-15T14:30:28.629 INFO:tasks.rados.rados.0.smithi090.stdout:594: done (0 left) 2023-01-15T14:30:28.629 INFO:tasks.rados.rados.0.smithi090.stdout:595: write oid 6 current snap is 55 2023-01-15T14:30:28.630 INFO:tasks.rados.rados.0.smithi090.stdout:595: seq_num 242 ranges {467294=617201,1578407=488887} 2023-01-15T14:30:28.630 INFO:tasks.rados.rados.0.smithi090.stdout:595: writing smithi09025264-6 from 467294 to 1084495 tid 1 2023-01-15T14:30:28.630 INFO:tasks.rados.rados.0.smithi090.stdout:595: writing smithi09025264-6 from 1578407 to 2067294 tid 2 2023-01-15T14:30:28.630 INFO:tasks.rados.rados.0.smithi090.stdout:596: rollback oid 36 current snap is 55 2023-01-15T14:30:28.630 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 36 to 51 2023-01-15T14:30:28.630 INFO:tasks.rados.rados.0.smithi090.stdout:597: snap_create 2023-01-15T14:30:28.630 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:28.630 INFO:tasks.rados.rados.0.smithi090.stdout:596: finishing rollback tid 0 to smithi09025264-36 2023-01-15T14:30:28.630 INFO:tasks.rados.rados.0.smithi090.stdout:595: finishing write tid 1 to smithi09025264-6 2023-01-15T14:30:28.631 INFO:tasks.rados.rados.0.smithi090.stdout:595: finishing write tid 2 to smithi09025264-6 2023-01-15T14:30:28.631 INFO:tasks.rados.rados.0.smithi090.stdout:596: finishing rollback tid 1 to smithi09025264-36 2023-01-15T14:30:28.631 INFO:tasks.rados.rados.0.smithi090.stdout:595: finishing write tid 3 to smithi09025264-6 2023-01-15T14:30:28.631 INFO:tasks.rados.rados.0.smithi090.stdout:596: finishing rollback tid 2 to smithi09025264-36 2023-01-15T14:30:28.631 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 196 (ObjNum 199 snap 43 seq_num 199) dirty exists 2023-01-15T14:30:28.631 INFO:tasks.rados.rados.0.smithi090.stdout:596: done (1 left) 2023-01-15T14:30:28.631 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:28.631 INFO:tasks.rados.rados.0.smithi090.stdout:595: finishing write tid 4 to smithi09025264-6 2023-01-15T14:30:28.631 INFO:tasks.rados.rados.0.smithi090.stdout:595: oid 6 updating version 0 to 63 2023-01-15T14:30:28.631 INFO:tasks.rados.rados.0.smithi090.stdout:595: oid 6 version 63 is already newer than 62 2023-01-15T14:30:28.632 INFO:tasks.rados.rados.0.smithi090.stdout:595: oid 6 updating version 63 to 64 2023-01-15T14:30:28.632 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 64 (ObjNum 242 snap 55 seq_num 242) dirty exists 2023-01-15T14:30:28.632 INFO:tasks.rados.rados.0.smithi090.stdout:595: left oid 6 (ObjNum 242 snap 55 seq_num 242) 2023-01-15T14:30:28.632 INFO:tasks.rados.rados.0.smithi090.stdout:595: done (0 left) 2023-01-15T14:30:28.722 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:28.720+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:28.824 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:28.824 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:28.825 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:28.825 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:30:28.825 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:30:28.825 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:28.825 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:30:28.826 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:28.819+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:28.828 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:28.852 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:28.989 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:28.989 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:28.989 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:28.989 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:28.990 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:28.990 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:28.990 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:30:28.991 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:28.987+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:28.991 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:30:28.991 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:30:28.992 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:30:28.992 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:30:28.992 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:30:28.992 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:30:28.992 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:30:28.992 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:30:29.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:30:29.154 INFO:tasks.rados.rados.0.smithi090.stdout:597: done (0 left) 2023-01-15T14:30:29.154 INFO:tasks.rados.rados.0.smithi090.stdout:598: snap_create 2023-01-15T14:30:29.154 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:29.152+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:29.255 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:29.252+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:29.312 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:29.312 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:29.313 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:30:29.361 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:29.359+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:29.441 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:29.441 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:29.441 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:30:29.457 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:29.455+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:29.548 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:30.670 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:30.673 INFO:tasks.rados.rados.0.smithi090.stdout:598: done (0 left) 2023-01-15T14:30:30.673 INFO:tasks.rados.rados.0.smithi090.stdout:599: rollback oid 7 current snap is 57 2023-01-15T14:30:30.673 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 7 to 55 2023-01-15T14:30:30.673 INFO:tasks.rados.rados.0.smithi090.stdout:600: write oid 4 current snap is 57 2023-01-15T14:30:30.680 INFO:tasks.rados.rados.0.smithi090.stdout:600: seq_num 243 ranges {484101=692450,1636889=447212} 2023-01-15T14:30:30.681 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:30:30.681 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:30:30.682 INFO:tasks.rados.rados.0.smithi090.stdout:600: writing smithi09025264-4 from 484101 to 1176551 tid 1 2023-01-15T14:30:30.682 INFO:tasks.rados.rados.0.smithi090.stdout:600: writing smithi09025264-4 from 1636889 to 2084101 tid 2 2023-01-15T14:30:30.683 INFO:tasks.rados.rados.0.smithi090.stdout:599: finishing rollback tid 1 to smithi09025264-7 2023-01-15T14:30:30.683 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 117 (ObjNum 97 snap 0 seq_num 3189890720) dirty dne 2023-01-15T14:30:30.683 INFO:tasks.rados.rados.0.smithi090.stdout:599: done (1 left) 2023-01-15T14:30:30.683 INFO:tasks.rados.rados.0.smithi090.stdout:601: rollback oid 44 current snap is 57 2023-01-15T14:30:30.683 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 44 to 55 2023-01-15T14:30:30.683 INFO:tasks.rados.rados.0.smithi090.stdout:602: snap_remove snap 52 2023-01-15T14:30:30.684 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:30.668+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:30.773 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:30.768+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:31.689 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:30:31.691 INFO:tasks.rados.rados.0.smithi090.stdout:602: done (2 left) 2023-01-15T14:30:31.691 INFO:tasks.rados.rados.0.smithi090.stdout:603: snap_create 2023-01-15T14:30:31.691 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:31.691 INFO:tasks.rados.rados.0.smithi090.stdout:600: finishing write tid 1 to smithi09025264-4 2023-01-15T14:30:31.692 INFO:tasks.rados.rados.0.smithi090.stdout:601: finishing rollback tid 0 to smithi09025264-44 2023-01-15T14:30:31.692 INFO:tasks.rados.rados.0.smithi090.stdout:601: finishing rollback tid 1 to smithi09025264-44 2023-01-15T14:30:31.692 INFO:tasks.rados.rados.0.smithi090.stdout:600: finishing write tid 2 to smithi09025264-4 2023-01-15T14:30:31.692 INFO:tasks.rados.rados.0.smithi090.stdout:600: finishing write tid 3 to smithi09025264-4 2023-01-15T14:30:31.692 INFO:tasks.rados.rados.0.smithi090.stdout:601: finishing rollback tid 2 to smithi09025264-44 2023-01-15T14:30:31.692 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 166 (ObjNum 232 snap 52 seq_num 232) dirty exists 2023-01-15T14:30:31.694 INFO:tasks.rados.rados.0.smithi090.stdout:600: finishing write tid 4 to smithi09025264-4 2023-01-15T14:30:31.798 INFO:tasks.rados.rados.0.smithi090.stdout:600: oid 4 updating version 0 to 137 2023-01-15T14:30:31.799 INFO:tasks.rados.rados.0.smithi090.stdout:600: oid 4 version 137 is already newer than 135 2023-01-15T14:30:31.799 INFO:tasks.rados.rados.0.smithi090.stdout:600: oid 4 version 137 is already newer than 136 2023-01-15T14:30:31.799 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 4 v 137 (ObjNum 243 snap 57 seq_num 243) dirty exists 2023-01-15T14:30:31.799 INFO:tasks.rados.rados.0.smithi090.stdout:600: left oid 4 (ObjNum 243 snap 57 seq_num 243) 2023-01-15T14:30:31.799 INFO:tasks.rados.rados.0.smithi090.stdout:600: done (1 left) 2023-01-15T14:30:31.799 INFO:tasks.rados.rados.0.smithi090.stdout:601: done (0 left) 2023-01-15T14:30:31.803 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:31.804 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:31.804 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:30:31.805 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:31.687+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:31.806 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:31.803+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:31.808 INFO:tasks.rados.rados.0.smithi090.stdout:603: done (0 left) 2023-01-15T14:30:31.851 INFO:tasks.rados.rados.0.smithi090.stdout:604: read oid 21 snap -1 2023-01-15T14:30:31.851 INFO:tasks.rados.rados.0.smithi090.stdout:604: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:30:31.851 INFO:tasks.rados.rados.0.smithi090.stdout:605: write (excl) oid 29 current snap is 58 2023-01-15T14:30:31.851 INFO:tasks.rados.rados.0.smithi090.stdout:605: seq_num 244 ranges {500908=484052,1411724=689184} 2023-01-15T14:30:31.851 INFO:tasks.rados.rados.0.smithi090.stdout:605: writing smithi09025264-29 from 500908 to 984960 tid 1 2023-01-15T14:30:31.851 INFO:tasks.rados.rados.0.smithi090.stdout:605: writing smithi09025264-29 from 1411724 to 2100908 tid 2 2023-01-15T14:30:31.851 INFO:tasks.rados.rados.0.smithi090.stdout:606: snap_remove snap 54 2023-01-15T14:30:31.907 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:31.903+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:31.958 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:31.959 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:31.959 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:31.960 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:30:31.975 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:31.995 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:32.012 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:32.007+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:32.015 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:32.015 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:30:32.015 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:30:32.015 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 7] 2023-01-15T14:30:32.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2023-01-15T14:30:32.072 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:32.118 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:32.120 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:32.120 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:32.121 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:30:32.121 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:32.115+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:32.126 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:32.174 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:32.175 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:30:32.175 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:30:32.175 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:30:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:30:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:30:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:30:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:30:32.176 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:30:32.221 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:32.219+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:32.293 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:30:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:30:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:30:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:97 3.a 3:52df5138:::smithi09025264-25:head [write 403368~555035,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:30:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 125.78662018199999, 2023-01-15T14:30:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.340758171, 2023-01-15T14:30:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 97 2023-01-15T14:30:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:30:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:30:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501140+0000", 2023-01-15T14:30:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:30:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501684+0000", 2023-01-15T14:30:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054441799999999996 2023-01-15T14:30:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501713+0000", 2023-01-15T14:30:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8833e-05 2023-01-15T14:30:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:32.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501726+0000", 2023-01-15T14:30:32.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3137e-05 2023-01-15T14:30:32.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601982+0000", 2023-01-15T14:30:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100256518 2023-01-15T14:30:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602426+0000", 2023-01-15T14:30:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044401500000000002 2023-01-15T14:30:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602703+0000", 2023-01-15T14:30:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000276889 2023-01-15T14:30:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603066+0000", 2023-01-15T14:30:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036278599999999998 2023-01-15T14:30:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603098+0000", 2023-01-15T14:30:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1433000000000002e-05 2023-01-15T14:30:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:32.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603382+0000", 2023-01-15T14:30:32.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028402899999999999 2023-01-15T14:30:32.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:32.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603632+0000", 2023-01-15T14:30:32.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025033 2023-01-15T14:30:32.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805354+0000", 2023-01-15T14:30:32.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20172168700000001 2023-01-15T14:30:32.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841465+0000", 2023-01-15T14:30:32.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.036111852999999999 2023-01-15T14:30:32.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:32.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841682+0000", 2023-01-15T14:30:32.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000216949 2023-01-15T14:30:32.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.341 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:32.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841901+0000", 2023-01-15T14:30:32.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021858700000000001 2023-01-15T14:30:32.341 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.341 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:32.341 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.342 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:98 3.a 3:52df5138:::smithi09025264-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:32.342 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:30:32.342 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 125.786010511, 2023-01-15T14:30:32.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.44410839299999999, 2023-01-15T14:30:32.342 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:32.342 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:32.342 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:32.342 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 98 2023-01-15T14:30:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:30:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:30:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501752+0000", 2023-01-15T14:30:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502223+0000", 2023-01-15T14:30:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000471843 2023-01-15T14:30:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502240+0000", 2023-01-15T14:30:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6663e-05 2023-01-15T14:30:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502248+0000", 2023-01-15T14:30:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5669999999999997e-06 2023-01-15T14:30:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603236+0000", 2023-01-15T14:30:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100988439 2023-01-15T14:30:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603384+0000", 2023-01-15T14:30:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014844200000000001 2023-01-15T14:30:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842016+0000", 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23863163800000001 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842267+0000", 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025101300000000002 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842561+0000", 2023-01-15T14:30:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029432900000000002 2023-01-15T14:30:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843120+0000", 2023-01-15T14:30:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055900500000000001 2023-01-15T14:30:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843146+0000", 2023-01-15T14:30:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5273e-05 2023-01-15T14:30:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843512+0000", 2023-01-15T14:30:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036621599999999999 2023-01-15T14:30:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943324+0000", 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099812008999999993 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945437+0000", 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021126069999999999 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945619+0000", 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018280600000000001 2023-01-15T14:30:32.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:32.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945744+0000", 2023-01-15T14:30:32.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012511299999999999 2023-01-15T14:30:32.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:32.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945861+0000", 2023-01-15T14:30:32.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011646000000000001 2023-01-15T14:30:32.351 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.352 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:32.352 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.352 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:99 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:32.352 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:30:32.352 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 125.785500255, 2023-01-15T14:30:32.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.54355587900000002, 2023-01-15T14:30:32.352 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:32.352 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:32.352 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:32.353 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:32.353 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:32.353 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 99 2023-01-15T14:30:32.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.353 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:32.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:32.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:30:32.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:32.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:30:32.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:32.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502262+0000", 2023-01-15T14:30:32.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:32.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:32.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502270+0000", 2023-01-15T14:30:32.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1999999999999994e-06 2023-01-15T14:30:32.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:32.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502286+0000", 2023-01-15T14:30:32.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5423000000000001e-05 2023-01-15T14:30:32.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:32.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502295+0000", 2023-01-15T14:30:32.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4339999999999998e-06 2023-01-15T14:30:32.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603467+0000", 2023-01-15T14:30:32.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101172225 2023-01-15T14:30:32.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:32.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603585+0000", 2023-01-15T14:30:32.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117149 2023-01-15T14:30:32.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843335+0000", 2023-01-15T14:30:32.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23975083799999999 2023-01-15T14:30:32.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:32.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843573+0000", 2023-01-15T14:30:32.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023753000000000001 2023-01-15T14:30:32.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:32.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843978+0000", 2023-01-15T14:30:32.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040533799999999999 2023-01-15T14:30:32.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:32.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844309+0000", 2023-01-15T14:30:32.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000330713 2023-01-15T14:30:32.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.538 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:32.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844423+0000", 2023-01-15T14:30:32.539 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114176 2023-01-15T14:30:32.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:32.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844510+0000", 2023-01-15T14:30:32.539 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7026000000000003e-05 2023-01-15T14:30:32.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:32.541 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943338+0000", 2023-01-15T14:30:32.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098827398999999996 2023-01-15T14:30:32.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945520+0000", 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021824169999999999 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045652+0000", 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013193200000001 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.542 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045733+0000", 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1435999999999998e-05 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045819+0000", 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5292999999999998e-05 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:100 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:30:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 125.78545233200001, 2023-01-15T14:30:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.64425290599999996, 2023-01-15T14:30:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 100 2023-01-15T14:30:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:30:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:32.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:30:32.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:32.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502310+0000", 2023-01-15T14:30:32.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:32.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:32.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502316+0000", 2023-01-15T14:30:32.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7470000000000002e-06 2023-01-15T14:30:32.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:32.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502332+0000", 2023-01-15T14:30:32.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6866000000000001e-05 2023-01-15T14:30:32.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502343+0000", 2023-01-15T14:30:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0913e-05 2023-01-15T14:30:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703594+0000", 2023-01-15T14:30:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20125087999999999 2023-01-15T14:30:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703751+0000", 2023-01-15T14:30:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001573 2023-01-15T14:30:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845262+0000", 2023-01-15T14:30:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.141510521 2023-01-15T14:30:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845391+0000", 2023-01-15T14:30:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012872299999999999 2023-01-15T14:30:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146051+0000", 2023-01-15T14:30:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.300660023 2023-01-15T14:30:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146264+0000", 2023-01-15T14:30:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212754 2023-01-15T14:30:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146564+0000", 2023-01-15T14:30:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030010200000000001 2023-01-15T14:30:32.553 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.553 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:32.553 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.553 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:203 3.a 3:52df5138:::smithi09025264-25:head [write 440350~460568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:32.553 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:30:32.553 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 124.734648937, 2023-01-15T14:30:32.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40523678899999999, 2023-01-15T14:30:32.553 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:32.553 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:32.554 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:32.554 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:32.554 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:32.554 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 203 2023-01-15T14:30:32.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.554 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:32.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:32.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:30:32.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:32.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:30:32.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:32.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553111+0000", 2023-01-15T14:30:32.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:30:32.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:32.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553483+0000", 2023-01-15T14:30:32.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037153499999999999 2023-01-15T14:30:32.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:32.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553517+0000", 2023-01-15T14:30:32.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4687000000000002e-05 2023-01-15T14:30:32.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:32.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553537+0000", 2023-01-15T14:30:32.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.967e-05 2023-01-15T14:30:32.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553683+0000", 2023-01-15T14:30:32.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000146341 2023-01-15T14:30:32.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:32.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553880+0000", 2023-01-15T14:30:32.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019671900000000001 2023-01-15T14:30:32.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:32.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554219+0000", 2023-01-15T14:30:32.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033893299999999998 2023-01-15T14:30:32.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:32.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554536+0000", 2023-01-15T14:30:32.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316576 2023-01-15T14:30:32.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:32.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554562+0000", 2023-01-15T14:30:32.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5905999999999999e-05 2023-01-15T14:30:32.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:32.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554801+0000", 2023-01-15T14:30:32.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000239791 2023-01-15T14:30:32.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:32.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555274+0000", 2023-01-15T14:30:32.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047294500000000001 2023-01-15T14:30:32.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757267+0000", 2023-01-15T14:30:32.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20199229399999999 2023-01-15T14:30:32.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958174+0000", 2023-01-15T14:30:32.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20090739499999999 2023-01-15T14:30:32.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:32.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958260+0000", 2023-01-15T14:30:32.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6316999999999994e-05 2023-01-15T14:30:32.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:32.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958351+0000", 2023-01-15T14:30:32.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0573000000000002e-05 2023-01-15T14:30:32.562 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.562 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:32.562 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.562 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:204 3.a 3:52df5138:::smithi09025264-25:head [write 1309629~542666] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:32.562 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:30:32.562 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 124.734186069, 2023-01-15T14:30:32.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40500913999999999, 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 204 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:30:32.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:32.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:30:32.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:32.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553575+0000", 2023-01-15T14:30:32.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:32.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:32.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554126+0000", 2023-01-15T14:30:32.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055090499999999997 2023-01-15T14:30:32.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:32.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554150+0000", 2023-01-15T14:30:32.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3387000000000002e-05 2023-01-15T14:30:32.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:32.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554165+0000", 2023-01-15T14:30:32.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5116000000000001e-05 2023-01-15T14:30:32.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554626+0000", 2023-01-15T14:30:32.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046093499999999999 2023-01-15T14:30:32.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:32.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554738+0000", 2023-01-15T14:30:32.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112446 2023-01-15T14:30:32.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:32.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554898+0000", 2023-01-15T14:30:32.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016003100000000001 2023-01-15T14:30:32.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:32.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555182+0000", 2023-01-15T14:30:32.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028382700000000001 2023-01-15T14:30:32.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:32.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555199+0000", 2023-01-15T14:30:32.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6969e-05 2023-01-15T14:30:32.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:32.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555449+0000", 2023-01-15T14:30:32.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000250265 2023-01-15T14:30:32.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:32.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555574+0000", 2023-01-15T14:30:32.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124628 2023-01-15T14:30:32.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757316+0000", 2023-01-15T14:30:32.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20174214300000001 2023-01-15T14:30:32.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958460+0000", 2023-01-15T14:30:32.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201144464 2023-01-15T14:30:32.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:32.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958534+0000", 2023-01-15T14:30:32.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3380000000000006e-05 2023-01-15T14:30:32.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:32.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958586+0000", 2023-01-15T14:30:32.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2370000000000002e-05 2023-01-15T14:30:32.571 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.571 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:32.571 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.571 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:205 3.a 3:52df5138:::smithi09025264-25:head [write 2323973~516377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:32.572 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:30:32.572 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 124.733567974, 2023-01-15T14:30:32.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404601398, 2023-01-15T14:30:32.572 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:32.572 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:32.572 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:32.572 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:32.572 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:32.572 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 205 2023-01-15T14:30:32.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.572 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:32.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554193+0000", 2023-01-15T14:30:32.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:32.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:32.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554850+0000", 2023-01-15T14:30:32.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065674799999999997 2023-01-15T14:30:32.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:32.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554873+0000", 2023-01-15T14:30:32.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2935999999999999e-05 2023-01-15T14:30:32.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:32.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.654962+0000", 2023-01-15T14:30:32.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088949 2023-01-15T14:30:32.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655107+0000", 2023-01-15T14:30:32.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144759 2023-01-15T14:30:32.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:32.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655373+0000", 2023-01-15T14:30:32.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026684899999999999 2023-01-15T14:30:32.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:32.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655720+0000", 2023-01-15T14:30:32.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034601899999999998 2023-01-15T14:30:32.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:32.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656168+0000", 2023-01-15T14:30:32.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044851899999999998 2023-01-15T14:30:32.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:32.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656190+0000", 2023-01-15T14:30:32.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1817000000000001e-05 2023-01-15T14:30:32.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656574+0000", 2023-01-15T14:30:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038404200000000002 2023-01-15T14:30:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657649+0000", 2023-01-15T14:30:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010746670000000001 2023-01-15T14:30:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757539+0000", 2023-01-15T14:30:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099890899000000005 2023-01-15T14:30:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958683+0000", 2023-01-15T14:30:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20114332400000001 2023-01-15T14:30:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958746+0000", 2023-01-15T14:30:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3509000000000005e-05 2023-01-15T14:30:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958796+0000", 2023-01-15T14:30:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0176999999999997e-05 2023-01-15T14:30:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:358 3.a 3:52df5138:::smithi09025264-25:head [write 709262~697258] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:30:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 112.236980441, 2023-01-15T14:30:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50589529799999999, 2023-01-15T14:30:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 358 2023-01-15T14:30:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:30:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:32.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:30:32.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:32.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050780+0000", 2023-01-15T14:30:32.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:30:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051369+0000", 2023-01-15T14:30:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058926200000000001 2023-01-15T14:30:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051390+0000", 2023-01-15T14:30:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1426000000000001e-05 2023-01-15T14:30:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051403+0000", 2023-01-15T14:30:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:30:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051520+0000", 2023-01-15T14:30:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117318 2023-01-15T14:30:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051785+0000", 2023-01-15T14:30:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000265018 2023-01-15T14:30:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052173+0000", 2023-01-15T14:30:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038734799999999998 2023-01-15T14:30:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052705+0000", 2023-01-15T14:30:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053209899999999996 2023-01-15T14:30:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052727+0000", 2023-01-15T14:30:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2362e-05 2023-01-15T14:30:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053205+0000", 2023-01-15T14:30:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047787200000000001 2023-01-15T14:30:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055711+0000", 2023-01-15T14:30:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0025054880000000002 2023-01-15T14:30:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.255549+0000", 2023-01-15T14:30:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199838285 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556501+0000", 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30095197499999998 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556613+0000", 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111799 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556678+0000", 2023-01-15T14:30:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5220000000000002e-05 2023-01-15T14:30:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:359 3.a 3:52df5138:::smithi09025264-25:head [write 2012635~572018] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:30:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 112.23632984699999, 2023-01-15T14:30:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50546442499999999, 2023-01-15T14:30:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 359 2023-01-15T14:30:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:30:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:30:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051432+0000", 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051808+0000", 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000376074 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051834+0000", 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6018999999999999e-05 2023-01-15T14:30:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051869+0000", 2023-01-15T14:30:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4839999999999998e-05 2023-01-15T14:30:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052857+0000", 2023-01-15T14:30:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00098761399999999994 2023-01-15T14:30:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053071+0000", 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000214112 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053876+0000", 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000804873 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054460+0000", 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058436100000000004 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054486+0000", 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069999999999999e-05 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054773+0000", 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028716700000000001 2023-01-15T14:30:32.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:32.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055736+0000", 2023-01-15T14:30:32.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00096295500000000004 2023-01-15T14:30:32.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.732 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.355784+0000", 2023-01-15T14:30:32.732 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30004825899999998 2023-01-15T14:30:32.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556757+0000", 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20097241199999999 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556830+0000", 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3633000000000005e-05 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:32.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556898+0000", 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7212999999999995e-05 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:360 3.a 3:52df5138:::smithi09025264-25:head [write 3046388~62874] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 112.23587709100001, 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50519877000000002, 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:32.734 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:32.735 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:32.736 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:32.736 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 360 2023-01-15T14:30:32.736 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.737 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:32.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:32.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:30:32.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:32.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:30:32.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:32.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:32.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051885+0000", 2023-01-15T14:30:32.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:32.738 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:32.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051935+0000", 2023-01-15T14:30:32.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9826e-05 2023-01-15T14:30:32.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.739 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.739 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:32.739 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051958+0000", 2023-01-15T14:30:32.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3003e-05 2023-01-15T14:30:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051965+0000", 2023-01-15T14:30:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6129999999999998e-06 2023-01-15T14:30:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054655+0000", 2023-01-15T14:30:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0026896860000000002 2023-01-15T14:30:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054869+0000", 2023-01-15T14:30:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021348600000000001 2023-01-15T14:30:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055198+0000", 2023-01-15T14:30:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032968199999999998 2023-01-15T14:30:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055605+0000", 2023-01-15T14:30:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040634200000000002 2023-01-15T14:30:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055622+0000", 2023-01-15T14:30:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7020999999999998e-05 2023-01-15T14:30:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:33.000 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055702+0000", 2023-01-15T14:30:33.000 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0835999999999997e-05 2023-01-15T14:30:33.000 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.000 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.000 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:33.000 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055754+0000", 2023-01-15T14:30:33.000 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1175999999999999e-05 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.455996+0000", 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40024198599999999 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556952+0000", 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10095644400000001 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:33.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557019+0000", 2023-01-15T14:30:33.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6746000000000001e-05 2023-01-15T14:30:33.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:33.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557085+0000", 2023-01-15T14:30:33.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5885999999999999e-05 2023-01-15T14:30:33.003 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:33.003 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:33.003 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:33.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.003 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:361 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 3109262] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:33.003 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:30:33.003 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 112.23578718500001, 2023-01-15T14:30:33.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60469846000000005, 2023-01-15T14:30:33.004 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:33.004 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:33.005 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:33.005 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:33.005 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:33.005 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 361 2023-01-15T14:30:33.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.006 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:33.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:33.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:30:33.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:33.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:33.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:30:33.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:33.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:33.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051975+0000", 2023-01-15T14:30:33.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:33.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:33.008 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051980+0000", 2023-01-15T14:30:33.008 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2170000000000002e-06 2023-01-15T14:30:33.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.008 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:33.008 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051989+0000", 2023-01-15T14:30:33.008 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5299999999999996e-06 2023-01-15T14:30:33.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.008 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:33.008 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051995+0000", 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1369999999999999e-06 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055830+0000", 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0038348549999999999 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056101+0000", 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027095800000000002 2023-01-15T14:30:33.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:33.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056560+0000", 2023-01-15T14:30:33.010 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045868599999999998 2023-01-15T14:30:33.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:33.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056954+0000", 2023-01-15T14:30:33.010 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039487600000000001 2023-01-15T14:30:33.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:33.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057004+0000", 2023-01-15T14:30:33.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9490000000000002e-05 2023-01-15T14:30:33.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057072+0000", 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8502000000000002e-05 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.158224+0000", 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101151422 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:33.334 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556201+0000", 2023-01-15T14:30:33.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39797714699999998 2023-01-15T14:30:33.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.334 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:33.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656421+0000", 2023-01-15T14:30:33.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100219549 2023-01-15T14:30:33.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.342 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:33.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656515+0000", 2023-01-15T14:30:33.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4048999999999993e-05 2023-01-15T14:30:33.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:33.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656674+0000", 2023-01-15T14:30:33.343 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000159696 2023-01-15T14:30:33.343 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:33.343 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:33.343 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:33.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.343 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:362 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:33.343 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:30:33.344 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 112.235759755, 2023-01-15T14:30:33.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60517504499999997, 2023-01-15T14:30:33.344 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:33.344 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:33.344 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:33.344 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:33.344 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:33.344 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 362 2023-01-15T14:30:33.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.344 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:33.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:33.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:30:33.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:33.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:33.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:30:33.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:33.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:33.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:30:33.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:33.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:33.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052007+0000", 2023-01-15T14:30:33.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.9759999999999997e-06 2023-01-15T14:30:33.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:33.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052016+0000", 2023-01-15T14:30:33.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4069999999999999e-06 2023-01-15T14:30:33.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.379 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:33.380 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052022+0000", 2023-01-15T14:30:33.380 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4670000000000001e-06 2023-01-15T14:30:33.380 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.380 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:33.381 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157159+0000", 2023-01-15T14:30:33.382 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.105136825 2023-01-15T14:30:33.382 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.382 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.382 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:33.382 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157325+0000", 2023-01-15T14:30:33.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000165262 2023-01-15T14:30:33.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.383 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:33.383 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656814+0000", 2023-01-15T14:30:33.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49948963200000002 2023-01-15T14:30:33.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.384 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:33.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656993+0000", 2023-01-15T14:30:33.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000178371 2023-01-15T14:30:33.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:33.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.657178+0000", 2023-01-15T14:30:33.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185825 2023-01-15T14:30:33.385 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:33.385 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:33.385 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:33.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.386 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:601 3.a 3:5ba385f1:::smithi09025264-44:head [setxattr _header (57) in=7b,truncate 3663893] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:30:33.386 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:30:33.386 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 92.345437118999996, 2023-01-15T14:30:33.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404742724, 2023-01-15T14:30:33.386 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:33.386 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 601 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:33.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:30:33.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:33.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:33.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942323+0000", 2023-01-15T14:30:33.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:30:33.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942341+0000", 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7659000000000002e-05 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942365+0000", 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4256999999999998e-05 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:33.389 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042495+0000", 2023-01-15T14:30:33.390 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013043200000001 2023-01-15T14:30:33.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.390 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.390 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:33.390 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.144809+0000", 2023-01-15T14:30:33.390 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102314184 2023-01-15T14:30:33.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.390 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:33.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145098+0000", 2023-01-15T14:30:33.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000289036 2023-01-15T14:30:33.391 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:33.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145363+0000", 2023-01-15T14:30:33.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026449299999999998 2023-01-15T14:30:33.391 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:33.392 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145657+0000", 2023-01-15T14:30:33.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029433600000000002 2023-01-15T14:30:33.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.392 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.392 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:33.392 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145678+0000", 2023-01-15T14:30:33.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0939000000000001e-05 2023-01-15T14:30:33.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.392 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145853+0000", 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017511800000000001 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146557+0000", 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00070337899999999996 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:33.393 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146845+0000", 2023-01-15T14:30:33.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028790599999999999 2023-01-15T14:30:33.394 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.394 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.394 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:33.394 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346819+0000", 2023-01-15T14:30:33.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199974661 2023-01-15T14:30:33.394 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.394 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.394 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:33.394 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346919+0000", 2023-01-15T14:30:33.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9234999999999994e-05 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.347069+0000", 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150128 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1153 3.a 3:5919d83e:::smithi09025264-13:head [write 1566383~595717] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:30:33.395 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:30:33.396 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 46.497212064999999, 2023-01-15T14:30:33.396 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.38805106299999997, 2023-01-15T14:30:33.396 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:33.396 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:33.396 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:33.396 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:33.396 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:33.396 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1153 2023-01-15T14:30:33.396 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.396 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:33.396 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.397 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:33.397 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:30:33.397 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:33.397 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.397 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.397 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:33.397 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:30:33.397 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:33.397 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.397 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.397 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:33.398 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790531+0000", 2023-01-15T14:30:33.398 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999795 2023-01-15T14:30:33.398 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.398 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:33.398 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791189+0000", 2023-01-15T14:30:33.398 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065897000000000004 2023-01-15T14:30:33.398 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.398 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:33.398 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791215+0000", 2023-01-15T14:30:33.399 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5700000000000001e-05 2023-01-15T14:30:33.399 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.399 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.399 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:33.399 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791232+0000", 2023-01-15T14:30:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6546e-05 2023-01-15T14:30:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791392+0000", 2023-01-15T14:30:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016051699999999999 2023-01-15T14:30:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791408+0000", 2023-01-15T14:30:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5566999999999999e-05 2023-01-15T14:30:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.974825+0000", 2023-01-15T14:30:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18341675599999999 2023-01-15T14:30:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975000+0000", 2023-01-15T14:30:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017548299999999999 2023-01-15T14:30:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975255+0000", 2023-01-15T14:30:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254658 2023-01-15T14:30:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975914+0000", 2023-01-15T14:30:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065881800000000003 2023-01-15T14:30:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975931+0000", 2023-01-15T14:30:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7523000000000001e-05 2023-01-15T14:30:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:33.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976267+0000", 2023-01-15T14:30:33.539 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033545299999999999 2023-01-15T14:30:33.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:33.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978320+0000", 2023-01-15T14:30:33.540 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020537110000000002 2023-01-15T14:30:33.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.540 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.540 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:33.540 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978647+0000", 2023-01-15T14:30:33.540 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032660700000000003 2023-01-15T14:30:33.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.540 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.540 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:33.540 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178434+0000", 2023-01-15T14:30:33.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19978705699999999 2023-01-15T14:30:33.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:33.541 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178527+0000", 2023-01-15T14:30:33.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3523999999999994e-05 2023-01-15T14:30:33.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:33.542 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178602+0000", 2023-01-15T14:30:33.542 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4622000000000005e-05 2023-01-15T14:30:33.542 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:33.542 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:33.542 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:33.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.543 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1154 3.a 3:5919d83e:::smithi09025264-13:head [write 2888822~102824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:30:33.543 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:30:33.543 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 46.491870738000003, 2023-01-15T14:30:33.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48318778800000001, 2023-01-15T14:30:33.544 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:33.544 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:33.544 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:33.544 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:33.544 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:33.545 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1154 2023-01-15T14:30:33.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.545 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:33.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:33.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:30:33.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:33.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:33.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:30:33.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:33.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:33.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795890+0000", 2023-01-15T14:30:33.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:33.742 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:33.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:33.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:33.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795967+0000", 2023-01-15T14:30:33.742 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6304999999999995e-05 2023-01-15T14:30:33.744 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:34.161 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796000+0000", 2023-01-15T14:30:34.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3593000000000002e-05 2023-01-15T14:30:34.161 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.161 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:34.161 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796019+0000", 2023-01-15T14:30:34.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8955999999999999e-05 2023-01-15T14:30:34.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.162 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.162 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:34.162 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796189+0000", 2023-01-15T14:30:34.162 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017032700000000001 2023-01-15T14:30:34.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.162 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.162 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:34.162 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796208+0000", 2023-01-15T14:30:34.163 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8389e-05 2023-01-15T14:30:34.163 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.163 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.163 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:34.163 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976335+0000", 2023-01-15T14:30:34.163 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18012763100000001 2023-01-15T14:30:34.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:34.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976487+0000", 2023-01-15T14:30:34.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015173000000000001 2023-01-15T14:30:34.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:34.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976763+0000", 2023-01-15T14:30:34.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027597400000000001 2023-01-15T14:30:34.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:34.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977085+0000", 2023-01-15T14:30:34.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032233400000000002 2023-01-15T14:30:34.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:34.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977103+0000", 2023-01-15T14:30:34.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7727000000000002e-05 2023-01-15T14:30:34.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:34.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977238+0000", 2023-01-15T14:30:34.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013525399999999999 2023-01-15T14:30:34.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:34.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978347+0000", 2023-01-15T14:30:34.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011084350000000001 2023-01-15T14:30:34.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:34.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.078223+0000", 2023-01-15T14:30:34.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099875830999999998 2023-01-15T14:30:34.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:34.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.278897+0000", 2023-01-15T14:30:34.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20067421599999999 2023-01-15T14:30:34.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:34.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279021+0000", 2023-01-15T14:30:34.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124354 2023-01-15T14:30:34.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:34.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279080+0000", 2023-01-15T14:30:34.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8811999999999999e-05 2023-01-15T14:30:34.169 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:34.169 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:34.169 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:34.169 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.169 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.169 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1155 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 2991646] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:30:34.169 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:30:34.169 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 46.491697719000001, 2023-01-15T14:30:34.170 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48334445999999998, 2023-01-15T14:30:34.170 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:34.170 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:34.170 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:34.170 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:34.170 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:34.170 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1155 2023-01-15T14:30:34.170 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.170 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:34.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:34.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:30:34.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:34.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:34.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:30:34.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:34.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:34.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796064+0000", 2023-01-15T14:30:34.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:34.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:34.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796074+0000", 2023-01-15T14:30:34.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:30:34.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:34.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796096+0000", 2023-01-15T14:30:34.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2422999999999999e-05 2023-01-15T14:30:34.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:34.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796109+0000", 2023-01-15T14:30:34.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796274+0000", 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000164976 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796283+0000", 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9539999999999993e-06 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:34.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977269+0000", 2023-01-15T14:30:34.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18098651700000001 2023-01-15T14:30:34.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:34.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977438+0000", 2023-01-15T14:30:34.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168913 2023-01-15T14:30:34.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:34.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977865+0000", 2023-01-15T14:30:34.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042707500000000001 2023-01-15T14:30:34.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:34.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978241+0000", 2023-01-15T14:30:34.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037593000000000001 2023-01-15T14:30:34.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:34.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978253+0000", 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1817000000000001e-05 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978288+0000", 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4586999999999999e-05 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978370+0000", 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1712999999999995e-05 2023-01-15T14:30:34.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279124+0000", 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30075418799999998 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279208+0000", 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4500999999999996e-05 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.362 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:34.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279294+0000", 2023-01-15T14:30:34.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5680999999999995e-05 2023-01-15T14:30:34.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:34.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279410+0000", 2023-01-15T14:30:34.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115878 2023-01-15T14:30:34.363 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:34.363 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:34.363 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:34.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1156 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 46.491586194, 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.58375825000000003, 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1156 2023-01-15T14:30:34.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796176+0000", 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796182+0000", 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6799999999999998e-06 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:34.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796200+0000", 2023-01-15T14:30:34.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8136999999999999e-05 2023-01-15T14:30:34.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:34.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796217+0000", 2023-01-15T14:30:34.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7036000000000001e-05 2023-01-15T14:30:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796337+0000", 2023-01-15T14:30:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120674 2023-01-15T14:30:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796345+0000", 2023-01-15T14:30:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4540000000000001e-06 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978423+0000", 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18207789599999999 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978577+0000", 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000153805 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379596+0000", 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.401019348 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379754+0000", 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015840400000000001 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379935+0000", 2023-01-15T14:30:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018081299999999999 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1160 3.a 3:52df5138:::smithi09025264-25:head [write 2632598~375855] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 45.532779607000002, 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.398337832, 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:34.561 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:34.561 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:34.561 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1160 2023-01-15T14:30:34.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.561 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:34.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:34.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:30:34.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:34.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:34.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:30:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754981+0000", 2023-01-15T14:30:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755360+0000", 2023-01-15T14:30:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037899899999999998 2023-01-15T14:30:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755392+0000", 2023-01-15T14:30:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1977000000000002e-05 2023-01-15T14:30:34.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:34.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755407+0000", 2023-01-15T14:30:34.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4446e-05 2023-01-15T14:30:34.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:34.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949294+0000", 2023-01-15T14:30:34.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.193887483 2023-01-15T14:30:34.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:34.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949478+0000", 2023-01-15T14:30:34.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018400900000000001 2023-01-15T14:30:34.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:34.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949904+0000", 2023-01-15T14:30:34.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042552200000000001 2023-01-15T14:30:34.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:34.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950531+0000", 2023-01-15T14:30:34.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062732399999999996 2023-01-15T14:30:34.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:34.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950552+0000", 2023-01-15T14:30:34.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1229999999999998e-05 2023-01-15T14:30:34.770 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:34.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951094+0000", 2023-01-15T14:30:34.771 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054169000000000005 2023-01-15T14:30:34.771 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.771 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.771 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:34.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951256+0000", 2023-01-15T14:30:34.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016176299999999999 2023-01-15T14:30:34.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:34.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052665+0000", 2023-01-15T14:30:34.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101409257 2023-01-15T14:30:34.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:34.776 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153132+0000", 2023-01-15T14:30:34.776 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100466783 2023-01-15T14:30:34.776 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.776 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.776 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:34.776 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153223+0000", 2023-01-15T14:30:34.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.1415000000000005e-05 2023-01-15T14:30:34.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:34.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153321+0000", 2023-01-15T14:30:34.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8181000000000001e-05 2023-01-15T14:30:34.778 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:34.778 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:34.778 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:34.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.779 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1161 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (57) in=7b,truncate 3008453] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:30:34.779 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:30:34.779 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 45.532336018999999, 2023-01-15T14:30:34.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49840109900000001, 2023-01-15T14:30:34.780 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:34.780 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:34.780 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:34.780 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:34.781 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:34.781 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1161 2023-01-15T14:30:34.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.781 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:34.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:34.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:30:34.782 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:34.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:34.968 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:30:34.968 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:34.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:34.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755426+0000", 2023-01-15T14:30:34.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:34.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:34.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755433+0000", 2023-01-15T14:30:34.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.793e-06 2023-01-15T14:30:34.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.971 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:34.971 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755448+0000", 2023-01-15T14:30:34.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5047e-05 2023-01-15T14:30:34.972 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.972 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.972 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:34.972 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755455+0000", 2023-01-15T14:30:34.972 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3590000000000002e-06 2023-01-15T14:30:34.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:34.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951466+0000", 2023-01-15T14:30:34.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.196011038 2023-01-15T14:30:34.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:34.974 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:34.974 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:35.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951611+0000", 2023-01-15T14:30:35.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000145184 2023-01-15T14:30:35.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:35.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951921+0000", 2023-01-15T14:30:35.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030989100000000002 2023-01-15T14:30:35.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:35.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952187+0000", 2023-01-15T14:30:35.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026623100000000002 2023-01-15T14:30:35.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:35.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952207+0000", 2023-01-15T14:30:35.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9510000000000001e-05 2023-01-15T14:30:35.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:35.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952271+0000", 2023-01-15T14:30:35.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4511999999999995e-05 2023-01-15T14:30:35.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:35.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052716+0000", 2023-01-15T14:30:35.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10044497300000001 2023-01-15T14:30:35.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:35.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253531+0000", 2023-01-15T14:30:35.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20081490699999999 2023-01-15T14:30:35.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:35.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253635+0000", 2023-01-15T14:30:35.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000103838 2023-01-15T14:30:35.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:35.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253727+0000", 2023-01-15T14:30:35.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2298000000000003e-05 2023-01-15T14:30:35.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:35.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253828+0000", 2023-01-15T14:30:35.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000100655 2023-01-15T14:30:35.033 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:35.034 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:35.034 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:35.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.034 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1162 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:30:35.034 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:30:35.035 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 45.532296623999997, 2023-01-15T14:30:35.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.498775839, 2023-01-15T14:30:35.035 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:35.035 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:35.035 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:35.036 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:35.036 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:35.036 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1162 2023-01-15T14:30:35.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.036 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:35.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.037 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:35.037 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:30:35.037 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:35.037 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.038 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:35.038 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:30:35.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:35.038 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.038 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:35.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:30:35.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:35.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:35.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755470+0000", 2023-01-15T14:30:35.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0570000000000003e-06 2023-01-15T14:30:35.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.041 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:35.041 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755484+0000", 2023-01-15T14:30:35.041 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4229999999999999e-05 2023-01-15T14:30:35.041 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.041 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:35.042 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755492+0000", 2023-01-15T14:30:35.042 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6070000000000002e-06 2023-01-15T14:30:35.042 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.042 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:35.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.152842+0000", 2023-01-15T14:30:35.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39734988799999998 2023-01-15T14:30:35.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153014+0000", 2023-01-15T14:30:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017236000000000001 2023-01-15T14:30:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253921+0000", 2023-01-15T14:30:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100907497 2023-01-15T14:30:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254060+0000", 2023-01-15T14:30:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013815400000000001 2023-01-15T14:30:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254242+0000", 2023-01-15T14:30:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000182726 2023-01-15T14:30:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:35.187 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:35.187 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:30:35.188 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:30:35.188 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-01-15T14:30:32.324+0000 7fb90e2b4700 -1 mon.a@0(leader).osd e145 definitely_dead 0 2023-01-15T14:30:35.189 INFO:tasks.rados.rados.0.smithi090.stdout:606: done (2 left) 2023-01-15T14:30:35.189 INFO:tasks.rados.rados.0.smithi090.stdout:607: delete oid 43 current snap is 58 2023-01-15T14:30:35.189 INFO:tasks.rados.rados.0.smithi090.stdout:604: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:30:35.190 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:30:35.190 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:35.183+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:35.191 INFO:teuthology.orchestra.run.smithi090.stderr:marked out osd.1. 2023-01-15T14:30:35.194 INFO:tasks.rados.rados.0.smithi090.stdout:605: finishing write tid 1 to smithi09025264-29 2023-01-15T14:30:35.194 INFO:tasks.rados.rados.0.smithi090.stdout:605: finishing write tid 2 to smithi09025264-29 2023-01-15T14:30:35.194 INFO:tasks.rados.rados.0.smithi090.stdout:605: finishing write tid 3 to smithi09025264-29 2023-01-15T14:30:35.194 INFO:tasks.rados.rados.0.smithi090.stdout:605: finishing write tid 4 to smithi09025264-29 2023-01-15T14:30:35.195 INFO:tasks.rados.rados.0.smithi090.stdout:605: oid 29 updating version 0 to 130 2023-01-15T14:30:35.195 INFO:tasks.rados.rados.0.smithi090.stdout:605: oid 29 version 130 is already newer than 129 2023-01-15T14:30:35.195 INFO:tasks.rados.rados.0.smithi090.stdout:605: oid 29 version 130 is already newer than 128 2023-01-15T14:30:35.195 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 130 (ObjNum 244 snap 58 seq_num 244) dirty exists 2023-01-15T14:30:35.195 INFO:tasks.rados.rados.0.smithi090.stdout:605: left oid 29 (ObjNum 244 snap 58 seq_num 244) 2023-01-15T14:30:35.195 INFO:tasks.rados.rados.0.smithi090.stdout:604: done (2 left) 2023-01-15T14:30:35.195 INFO:tasks.rados.rados.0.smithi090.stdout:605: done (1 left) 2023-01-15T14:30:35.196 INFO:tasks.rados.rados.0.smithi090.stdout:607: done (0 left) 2023-01-15T14:30:35.196 INFO:tasks.rados.rados.0.smithi090.stdout:608: rollback oid 28 current snap is 58 2023-01-15T14:30:35.196 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 28 to 51 2023-01-15T14:30:35.196 INFO:tasks.rados.rados.0.smithi090.stdout:609: read oid 21 snap 57 2023-01-15T14:30:35.196 INFO:tasks.rados.rados.0.smithi090.stdout:609: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:30:35.197 INFO:tasks.rados.rados.0.smithi090.stdout:610: write oid 39 current snap is 58 2023-01-15T14:30:35.197 INFO:tasks.rados.rados.0.smithi090.stdout:610: seq_num 246 ranges {534522=634550,1812335=322187} 2023-01-15T14:30:35.197 INFO:tasks.rados.rados.0.smithi090.stdout:610: writing smithi09025264-39 from 534522 to 1169072 tid 1 2023-01-15T14:30:35.197 INFO:tasks.rados.rados.0.smithi090.stdout:610: writing smithi09025264-39 from 1812335 to 2134522 tid 2 2023-01-15T14:30:35.198 INFO:tasks.rados.rados.0.smithi090.stdout:611: snap_remove snap 56 2023-01-15T14:30:35.198 INFO:tasks.rados.rados.0.smithi090.stdout:611: done (3 left) 2023-01-15T14:30:35.198 INFO:tasks.rados.rados.0.smithi090.stdout:612: copy_from oid 3 from oid 11 current snap is 58 2023-01-15T14:30:35.198 INFO:tasks.rados.rados.0.smithi090.stdout:608: finishing rollback tid 0 to smithi09025264-28 2023-01-15T14:30:35.198 INFO:tasks.rados.rados.0.smithi090.stdout:608: finishing rollback tid 1 to smithi09025264-28 2023-01-15T14:30:35.198 INFO:tasks.rados.rados.0.smithi090.stdout:608: finishing rollback tid 2 to smithi09025264-28 2023-01-15T14:30:35.199 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 234 (ObjNum 222 snap 51 seq_num 222) dirty exists 2023-01-15T14:30:35.199 INFO:tasks.rados.rados.0.smithi090.stdout:608: done (3 left) 2023-01-15T14:30:35.199 INFO:tasks.rados.rados.0.smithi090.stdout:613: snap_remove snap 55 2023-01-15T14:30:35.289 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:35.287+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:35.381 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:35.381 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:35.381 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:35.382 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:35.382 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:30:35.382 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:30:35.382 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:35.383 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:30:35.389 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:35.408 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:35.588 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:35.427+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:35.589 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:35.589 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:35.589 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:35.591 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:35.591 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:35.592 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:30:35.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:30:35.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:30:35.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:30:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:30:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:30:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:30:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:30:35.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:30:35.595 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:35.584+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:35.689 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:35.687+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:35.788 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:35.788 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:35.788 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:30:35.797 INFO:tasks.rados.rados.0.smithi090.stdout:613: done (3 left) 2023-01-15T14:30:35.798 INFO:tasks.rados.rados.0.smithi090.stdout:614: delete oid 45 current snap is 58 2023-01-15T14:30:35.798 INFO:tasks.rados.rados.0.smithi090.stdout:609: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:30:35.801 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:30:35.833 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:35.831+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:35.993 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:35.993 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:35.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:30:35.994 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:35.988+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:35.999 INFO:tasks.rados.rados.0.smithi090.stdout:610: finishing write tid 1 to smithi09025264-39 2023-01-15T14:30:35.999 INFO:tasks.rados.rados.0.smithi090.stdout:610: finishing write tid 2 to smithi09025264-39 2023-01-15T14:30:35.999 INFO:tasks.rados.rados.0.smithi090.stdout:610: finishing write tid 3 to smithi09025264-39 2023-01-15T14:30:35.999 INFO:tasks.rados.rados.0.smithi090.stdout:610: finishing write tid 4 to smithi09025264-39 2023-01-15T14:30:35.999 INFO:tasks.rados.rados.0.smithi090.stdout:610: oid 39 updating version 0 to 165 2023-01-15T14:30:35.999 INFO:tasks.rados.rados.0.smithi090.stdout:610: oid 39 version 165 is already newer than 164 2023-01-15T14:30:36.000 INFO:tasks.rados.rados.0.smithi090.stdout:610: oid 39 updating version 165 to 166 2023-01-15T14:30:36.000 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 166 (ObjNum 246 snap 58 seq_num 246) dirty exists 2023-01-15T14:30:36.000 INFO:tasks.rados.rados.0.smithi090.stdout:610: left oid 39 (ObjNum 246 snap 58 seq_num 246) 2023-01-15T14:30:36.000 INFO:tasks.rados.rados.0.smithi090.stdout:612: finishing copy_from to smithi09025264-3 2023-01-15T14:30:36.000 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 205 (ObjNum 185 snap 40 seq_num 185) dirty exists 2023-01-15T14:30:36.000 INFO:tasks.rados.rados.0.smithi090.stdout:612: finishing copy_from racing read to smithi09025264-3 2023-01-15T14:30:36.001 INFO:tasks.rados.rados.0.smithi090.stdout:609: done (3 left) 2023-01-15T14:30:36.001 INFO:tasks.rados.rados.0.smithi090.stdout:610: done (2 left) 2023-01-15T14:30:36.001 INFO:tasks.rados.rados.0.smithi090.stdout:612: done (1 left) 2023-01-15T14:30:36.001 INFO:tasks.rados.rados.0.smithi090.stdout:614: done (0 left) 2023-01-15T14:30:36.001 INFO:tasks.rados.rados.0.smithi090.stdout:615: write oid 29 current snap is 58 2023-01-15T14:30:36.001 INFO:tasks.rados.rados.0.smithi090.stdout:615: seq_num 248 ranges {568136=785048,1929299=238837} 2023-01-15T14:30:36.018 INFO:tasks.rados.rados.0.smithi090.stdout:615: writing smithi09025264-29 from 568136 to 1353184 tid 1 2023-01-15T14:30:36.198 INFO:tasks.rados.rados.0.smithi090.stdout:615: writing smithi09025264-29 from 1929299 to 2168136 tid 2 2023-01-15T14:30:36.198 INFO:tasks.rados.rados.0.smithi090.stdout:616: read oid 28 snap -1 2023-01-15T14:30:36.198 INFO:tasks.rados.rados.0.smithi090.stdout:616: expect (ObjNum 222 snap 51 seq_num 222) 2023-01-15T14:30:36.198 INFO:tasks.rados.rados.0.smithi090.stdout:617: delete oid 34 current snap is 58 2023-01-15T14:30:36.199 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:36.199 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:36.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:30:36.201 INFO:tasks.rados.rados.0.smithi090.stdout:617: done (2 left) 2023-01-15T14:30:36.201 INFO:tasks.rados.rados.0.smithi090.stdout:618: read oid 4 snap -1 2023-01-15T14:30:36.201 INFO:tasks.rados.rados.0.smithi090.stdout:618: expect (ObjNum 243 snap 57 seq_num 243) 2023-01-15T14:30:36.201 INFO:tasks.rados.rados.0.smithi090.stdout:619: delete oid 11 current snap is 58 2023-01-15T14:30:36.201 INFO:tasks.rados.rados.0.smithi090.stdout:619: done (3 left) 2023-01-15T14:30:36.201 INFO:tasks.rados.rados.0.smithi090.stdout:620: copy_from oid 25 from oid 27 current snap is 58 2023-01-15T14:30:36.201 INFO:tasks.rados.rados.0.smithi090.stdout:621: copy_from oid 15 from oid 23 current snap is 58 2023-01-15T14:30:36.201 INFO:tasks.rados.rados.0.smithi090.stdout:622: snap_remove snap 53 2023-01-15T14:30:36.201 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:36.196+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:36.299 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:36.296+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:36.400 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:36.400 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:36.401 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:30:36.401 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:36.396+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:36.501 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:36.499+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:36.611 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:36.611 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:36.612 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:30:36.613 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:36.607+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:36.712 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:36.708+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:36.815 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:36.815 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:36.816 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:30:36.816 INFO:tasks.rados.rados.0.smithi090.stdout:622: done (5 left) 2023-01-15T14:30:36.817 INFO:tasks.rados.rados.0.smithi090.stdout:623: snap_create 2023-01-15T14:30:36.817 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:30:36.817 INFO:tasks.rados.rados.0.smithi090.stdout:616: expect (ObjNum 222 snap 51 seq_num 222) 2023-01-15T14:30:36.817 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:36.812+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:36.915 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:36.911+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:37.021 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:37.021 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:37.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:30:37.218 INFO:tasks.rados.rados.0.smithi090.stdout:618: expect (ObjNum 243 snap 57 seq_num 243) 2023-01-15T14:30:37.219 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:37.016+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:37.219 INFO:tasks.rados.rados.0.smithi090.stdout:621: finishing copy_from to smithi09025264-15 2023-01-15T14:30:37.220 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 148 (ObjNum 226 snap 51 seq_num 226) dirty exists 2023-01-15T14:30:37.220 INFO:tasks.rados.rados.0.smithi090.stdout:621: finishing copy_from racing read to smithi09025264-15 2023-01-15T14:30:37.220 INFO:tasks.rados.rados.0.smithi090.stdout:615: finishing write tid 1 to smithi09025264-29 2023-01-15T14:30:37.220 INFO:tasks.rados.rados.0.smithi090.stdout:615: finishing write tid 2 to smithi09025264-29 2023-01-15T14:30:37.220 INFO:tasks.rados.rados.0.smithi090.stdout:615: finishing write tid 3 to smithi09025264-29 2023-01-15T14:30:37.220 INFO:tasks.rados.rados.0.smithi090.stdout:615: finishing write tid 4 to smithi09025264-29 2023-01-15T14:30:37.220 INFO:tasks.rados.rados.0.smithi090.stdout:615: oid 29 updating version 0 to 131 2023-01-15T14:30:37.221 INFO:tasks.rados.rados.0.smithi090.stdout:615: oid 29 updating version 131 to 133 2023-01-15T14:30:37.221 INFO:tasks.rados.rados.0.smithi090.stdout:615: oid 29 version 133 is already newer than 132 2023-01-15T14:30:37.221 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 133 (ObjNum 248 snap 58 seq_num 248) dirty exists 2023-01-15T14:30:37.221 INFO:tasks.rados.rados.0.smithi090.stdout:615: left oid 29 (ObjNum 248 snap 58 seq_num 248) 2023-01-15T14:30:37.221 INFO:tasks.rados.rados.0.smithi090.stdout:620: finishing copy_from to smithi09025264-25 2023-01-15T14:30:37.221 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 41 (ObjNum 101 snap 21 seq_num 101) dirty exists 2023-01-15T14:30:37.221 INFO:tasks.rados.rados.0.smithi090.stdout:620: finishing copy_from racing read to smithi09025264-25 2023-01-15T14:30:37.221 INFO:tasks.rados.rados.0.smithi090.stdout:615: done (4 left) 2023-01-15T14:30:37.221 INFO:tasks.rados.rados.0.smithi090.stdout:616: done (3 left) 2023-01-15T14:30:37.221 INFO:tasks.rados.rados.0.smithi090.stdout:618: done (2 left) 2023-01-15T14:30:37.222 INFO:tasks.rados.rados.0.smithi090.stdout:620: done (1 left) 2023-01-15T14:30:37.222 INFO:tasks.rados.rados.0.smithi090.stdout:621: done (0 left) 2023-01-15T14:30:37.224 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:37.219+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:37.324 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:37.320+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:37.428 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:37.428 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:37.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:30:37.429 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:37.424+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:37.528 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:37.524+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:37.620 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:30:37.621 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:37.621 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:37.622 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:30:37.631 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:37.627+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:37.725 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:37.833 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:37.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:30:37.836 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:30:37.837 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:37.837 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:37.838 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:37.838 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:37.838 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:37.832+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:37.839 INFO:tasks.rados.rados.0.smithi090.stdout:623: done (0 left) 2023-01-15T14:30:37.839 INFO:tasks.rados.rados.0.smithi090.stdout:624: delete oid 39 current snap is 59 2023-01-15T14:30:37.852 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:37.885 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:37.936 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:37.931+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:37.994 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:37.994 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:37.994 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:30:37.994 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:37.994 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:37.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:30:37.996 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:37.996 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:30:37.996 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:30:37.997 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:30:37.997 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:30:37.997 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:30:37.997 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:30:37.997 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:30:37.997 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:30:38.037 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:38.035+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:38.162 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:38.162 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:38.162 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:30:38.163 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:38.159+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:38.188 INFO:tasks.rados.rados.0.smithi090.stdout:624: done (0 left) 2023-01-15T14:30:38.189 INFO:tasks.rados.rados.0.smithi090.stdout:625: snap_create 2023-01-15T14:30:38.262 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:38.260+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:38.342 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:38.342 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:166 3.8 3:1e54a7f5:::smithi09025264-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:38.343 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:30:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.15528468599999, 2023-01-15T14:30:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62664632300000001, 2023-01-15T14:30:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:38.344 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 166 2023-01-15T14:30:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.345 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:30:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:30:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114713+0000", 2023-01-15T14:30:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:38.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115178+0000", 2023-01-15T14:30:38.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046493199999999999 2023-01-15T14:30:38.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:38.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115208+0000", 2023-01-15T14:30:38.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0689999999999999e-05 2023-01-15T14:30:38.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:38.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115221+0000", 2023-01-15T14:30:38.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3013000000000001e-05 2023-01-15T14:30:38.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115334+0000", 2023-01-15T14:30:38.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112403 2023-01-15T14:30:38.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:38.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115505+0000", 2023-01-15T14:30:38.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170915 2023-01-15T14:30:38.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333496+0000", 2023-01-15T14:30:38.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217991667 2023-01-15T14:30:38.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:38.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333606+0000", 2023-01-15T14:30:38.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109967 2023-01-15T14:30:38.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:30:38.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333796+0000", 2023-01-15T14:30:38.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018928099999999999 2023-01-15T14:30:38.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:38.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334289+0000", 2023-01-15T14:30:38.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049322600000000004 2023-01-15T14:30:38.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:38.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334339+0000", 2023-01-15T14:30:38.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0439999999999998e-05 2023-01-15T14:30:38.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:38.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334990+0000", 2023-01-15T14:30:38.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065082300000000003 2023-01-15T14:30:38.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:38.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335100+0000", 2023-01-15T14:30:38.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109357 2023-01-15T14:30:38.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.640854+0000", 2023-01-15T14:30:38.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30575451300000001 2023-01-15T14:30:38.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741190+0000", 2023-01-15T14:30:38.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100335568 2023-01-15T14:30:38.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:38.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741263+0000", 2023-01-15T14:30:38.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3275999999999994e-05 2023-01-15T14:30:38.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:38.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741361+0000", 2023-01-15T14:30:38.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8435000000000002e-05 2023-01-15T14:30:38.700 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.700 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:38.700 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.701 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:167 3.8 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:38.701 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:30:38.701 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.154719997, 2023-01-15T14:30:38.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62638390099999997, 2023-01-15T14:30:38.702 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:38.702 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:38.702 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:38.702 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:38.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:38.703 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 167 2023-01-15T14:30:38.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.703 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:38.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:38.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:30:38.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:38.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:30:38.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:38.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115279+0000", 2023-01-15T14:30:38.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:38.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:38.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115289+0000", 2023-01-15T14:30:38.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0509999999999999e-05 2023-01-15T14:30:38.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:38.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115306+0000", 2023-01-15T14:30:38.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7260000000000001e-05 2023-01-15T14:30:38.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:38.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115337+0000", 2023-01-15T14:30:38.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0664000000000003e-05 2023-01-15T14:30:38.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115579+0000", 2023-01-15T14:30:38.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024194400000000001 2023-01-15T14:30:38.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:38.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115744+0000", 2023-01-15T14:30:38.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016493700000000001 2023-01-15T14:30:38.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334392+0000", 2023-01-15T14:30:38.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21864830299999999 2023-01-15T14:30:38.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:38.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334453+0000", 2023-01-15T14:30:38.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1047e-05 2023-01-15T14:30:38.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:30:38.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334601+0000", 2023-01-15T14:30:38.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014824700000000001 2023-01-15T14:30:38.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:38.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334812+0000", 2023-01-15T14:30:38.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210208 2023-01-15T14:30:38.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334996+0000", 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183878 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335126+0000", 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013001399999999999 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:38.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335160+0000", 2023-01-15T14:30:38.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4876999999999998e-05 2023-01-15T14:30:38.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741044+0000", 2023-01-15T14:30:38.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40588326000000002 2023-01-15T14:30:38.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741453+0000", 2023-01-15T14:30:38.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040963099999999998 2023-01-15T14:30:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741548+0000", 2023-01-15T14:30:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4961000000000001e-05 2023-01-15T14:30:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741664+0000", 2023-01-15T14:30:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115333 2023-01-15T14:30:38.716 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.15462788299999, 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:38.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:38.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:30:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:30:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:30:38.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:30:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:30:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:30:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:30:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:30:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:30:38.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:38.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:30:38.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:30:38.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:38.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:30:38.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:30:38.724 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.724 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:38.724 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.725 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:38.725 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:30:38.725 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.141635804, 2023-01-15T14:30:38.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:30:38.725 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:38.726 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:38.726 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:38.726 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:38.726 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:38.726 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:30:38.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.726 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:38.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:38.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:30:38.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:38.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:30:38.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:30:38.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:30:38.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:38.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:30:38.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:30:38.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:30:38.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:30:38.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:30:38.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:38.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:30:38.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:30:38.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:38.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:30:38.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:30:38.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:38.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:30:38.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:30:38.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:38.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:30:38.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:38.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:30:38.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:30:38.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:38.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:30:38.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:30:38.735 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.735 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:38.736 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.736 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:38.736 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:30:38.736 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.141232468, 2023-01-15T14:30:38.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:30:38.737 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:38.737 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:38.737 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:38.737 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:38.737 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:38.737 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:30:38.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.738 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:38.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:38.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:30:38.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:38.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:30:38.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:38.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:30:38.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:38.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:38.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:30:38.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:30:38.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:38.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:30:38.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:30:38.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:38.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:30:38.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:30:38.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:30:38.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:30:38.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:38.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:30:38.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:30:38.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:30:38.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:30:38.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:30:38.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:38.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:30:38.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:30:38.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:30:38.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:38.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:30:38.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:30:38.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:38.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:30:38.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:30:38.900 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.900 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:38.900 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.900 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:38.900 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:30:38.900 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.04100884799999, 2023-01-15T14:30:38.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:30:38.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:38.900 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:38.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:30:38.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:38.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:30:38.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:38.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:38.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:30:38.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:30:38.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:38.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:30:38.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:30:38.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:38.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:30:38.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:30:38.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:30:38.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:30:38.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:38.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:30:38.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:30:38.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:30:38.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:30:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:30:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:30:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:30:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:30:38.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:38.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.908 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:194 3.8 3:1eaad226:::smithi09025264-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:38.908 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:30:38.908 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.96860760499999, 2023-01-15T14:30:38.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64372577200000003, 2023-01-15T14:30:38.908 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:38.908 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:38.908 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:38.908 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:38.909 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:38.909 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 194 2023-01-15T14:30:38.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.909 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:38.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:38.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:30:38.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:38.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:30:38.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:38.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301390+0000", 2023-01-15T14:30:38.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:38.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:38.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301991+0000", 2023-01-15T14:30:38.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060122599999999995 2023-01-15T14:30:38.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:38.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.302026+0000", 2023-01-15T14:30:38.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5040000000000003e-05 2023-01-15T14:30:38.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:38.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402147+0000", 2023-01-15T14:30:38.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100121084 2023-01-15T14:30:38.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402219+0000", 2023-01-15T14:30:38.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2045999999999994e-05 2023-01-15T14:30:38.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:38.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402315+0000", 2023-01-15T14:30:38.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5561000000000002e-05 2023-01-15T14:30:38.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842189+0000", 2023-01-15T14:30:38.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.439874129 2023-01-15T14:30:38.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:38.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842296+0000", 2023-01-15T14:30:38.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107021 2023-01-15T14:30:38.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:30:38.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842486+0000", 2023-01-15T14:30:38.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018965799999999999 2023-01-15T14:30:38.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:38.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842839+0000", 2023-01-15T14:30:38.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035330700000000002 2023-01-15T14:30:38.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842859+0000", 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0347e-05 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843212+0000", 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352534 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:38.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843835+0000", 2023-01-15T14:30:38.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062338599999999997 2023-01-15T14:30:38.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944861+0000", 2023-01-15T14:30:38.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10102596899999999 2023-01-15T14:30:38.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944908+0000", 2023-01-15T14:30:38.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6773000000000001e-05 2023-01-15T14:30:38.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:38.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945039+0000", 2023-01-15T14:30:38.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130684 2023-01-15T14:30:38.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:38.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945118+0000", 2023-01-15T14:30:38.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9194000000000001e-05 2023-01-15T14:30:38.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.918 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:38.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:38.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.918 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:525 3.8 3:17d5d741:::smithi09025264-5:head [write 2555481~425187] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:30:38.918 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:30:38.918 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 106.451042679, 2023-01-15T14:30:38.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60814212099999998, 2023-01-15T14:30:38.918 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:38.918 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 525 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818957+0000", 2023-01-15T14:30:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.818955+0000", 2023-01-15T14:30:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819151+0000", 2023-01-15T14:30:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019632699999999999 2023-01-15T14:30:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819175+0000", 2023-01-15T14:30:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3777e-05 2023-01-15T14:30:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:38.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819185+0000", 2023-01-15T14:30:38.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0424999999999999e-05 2023-01-15T14:30:38.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019555+0000", 2023-01-15T14:30:38.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20036944200000001 2023-01-15T14:30:38.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:38.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019726+0000", 2023-01-15T14:30:38.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017075099999999999 2023-01-15T14:30:38.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:38.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224068+0000", 2023-01-15T14:30:38.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20434244100000001 2023-01-15T14:30:38.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224217+0000", 2023-01-15T14:30:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014933200000000001 2023-01-15T14:30:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:30:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.224571+0000", 2023-01-15T14:30:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035323700000000001 2023-01-15T14:30:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225080+0000", 2023-01-15T14:30:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050910500000000004 2023-01-15T14:30:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225103+0000", 2023-01-15T14:30:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3295999999999999e-05 2023-01-15T14:30:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:38.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225542+0000", 2023-01-15T14:30:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043897599999999999 2023-01-15T14:30:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226915+0000", 2023-01-15T14:30:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013730599999999999 2023-01-15T14:30:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.327618+0000", 2023-01-15T14:30:38.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100702655 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426890+0000", 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099272284000000002 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426977+0000", 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6500000000000002e-05 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:38.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:38.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:38.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427099+0000", 2023-01-15T14:30:38.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000122562 2023-01-15T14:30:38.928 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:526 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (56) in=7b,truncate 3512630] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 106.45079494700001, 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60866722500000003, 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.104 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 526 2023-01-15T14:30:39.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.105 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:30:39.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819205+0000", 2023-01-15T14:30:39.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819204+0000", 2023-01-15T14:30:39.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:39.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819210+0000", 2023-01-15T14:30:39.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5429999999999999e-06 2023-01-15T14:30:39.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819235+0000", 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4114e-05 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819246+0000", 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1043000000000001e-05 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.019911+0000", 2023-01-15T14:30:39.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200665175 2023-01-15T14:30:39.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:39.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020085+0000", 2023-01-15T14:30:39.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000174449 2023-01-15T14:30:39.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225345+0000", 2023-01-15T14:30:39.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20526017099999999 2023-01-15T14:30:39.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:39.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225560+0000", 2023-01-15T14:30:39.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021505499999999999 2023-01-15T14:30:39.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:30:39.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.225908+0000", 2023-01-15T14:30:39.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034754600000000001 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226267+0000", 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035936399999999998 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226688+0000", 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042024599999999999 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226841+0000", 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015324499999999999 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226936+0000", 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5549000000000003e-05 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.426826+0000", 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19988924 2023-01-15T14:30:39.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:39.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427685+0000", 2023-01-15T14:30:39.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000859719 2023-01-15T14:30:39.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:39.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427781+0000", 2023-01-15T14:30:39.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5515999999999998e-05 2023-01-15T14:30:39.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:39.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427872+0000", 2023-01-15T14:30:39.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1142000000000003e-05 2023-01-15T14:30:39.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.113 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.113 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:527 3.8 3:17d5d741:::smithi09025264-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e46)", 2023-01-15T14:30:39.113 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:39.113 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 106.45074445900001, 2023-01-15T14:30:39.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60906366599999995, 2023-01-15T14:30:39.114 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.114 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:39.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.114 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 527 2023-01-15T14:30:39.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.114 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:39.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:39.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819255+0000", 2023-01-15T14:30:39.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:39.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819259+0000", 2023-01-15T14:30:39.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5940000000000004e-06 2023-01-15T14:30:39.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.819268+0000", 2023-01-15T14:30:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.115e-06 2023-01-15T14:30:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:51.919405+0000", 2023-01-15T14:30:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013656999999999 2023-01-15T14:30:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020192+0000", 2023-01-15T14:30:39.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10078725600000001 2023-01-15T14:30:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.020326+0000", 2023-01-15T14:30:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133745 2023-01-15T14:30:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226443+0000", 2023-01-15T14:30:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20611705699999999 2023-01-15T14:30:39.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.226689+0000", 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024580799999999997 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.427991+0000", 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20130269000000001 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:39.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.428147+0000", 2023-01-15T14:30:39.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155258 2023-01-15T14:30:39.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:39.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:52.428319+0000", 2023-01-15T14:30:39.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017226200000000001 2023-01-15T14:30:39.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.120 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.121 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:705 3.2 3:47dfe935:::smithi09025264-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2023-01-15T14:30:39.121 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:30:39.121 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 89.159415697, 2023-01-15T14:30:39.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61170208599999998, 2023-01-15T14:30:39.121 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.121 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:39.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.121 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 705 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110583+0000", 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110587+0000", 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9689999999999996e-06 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110601+0000", 2023-01-15T14:30:39.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.393e-05 2023-01-15T14:30:39.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110611+0000", 2023-01-15T14:30:39.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3999999999999998e-06 2023-01-15T14:30:39.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319174+0000", 2023-01-15T14:30:39.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20856376800000001 2023-01-15T14:30:39.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:39.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319367+0000", 2023-01-15T14:30:39.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192361 2023-01-15T14:30:39.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.721941+0000", 2023-01-15T14:30:39.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40257461300000003 2023-01-15T14:30:39.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:39.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722108+0000", 2023-01-15T14:30:39.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167054 2023-01-15T14:30:39.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:39.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722286+0000", 2023-01-15T14:30:39.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017772099999999999 2023-01-15T14:30:39.294 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.295 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.295 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.295 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:39.296 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:30:39.296 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.347693622999998, 2023-01-15T14:30:39.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:30:39.296 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.296 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:39.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.297 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:30:39.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.298 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:30:39.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:30:39.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:30:39.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:39.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:30:39.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:30:39.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:30:39.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:30:39.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:30:39.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:30:39.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:30:39.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:30:39.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:39.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:30:39.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:30:39.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:30:39.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:30:39.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:39.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:30:39.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:30:39.307 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.307 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.307 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.308 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:39.308 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:30:39.308 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.341232269000002, 2023-01-15T14:30:39.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:30:39.309 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.309 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:39.309 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.309 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.310 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:30:39.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.310 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:30:39.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:30:39.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:30:39.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:39.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:30:39.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:30:39.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:30:39.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:30:39.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:30:39.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:30:39.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:30:39.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:30:39.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:39.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:30:39.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:30:39.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:30:39.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:30:39.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:39.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:30:39.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:30:39.320 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.320 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.320 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.321 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:39.321 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:30:39.321 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.240810344000003, 2023-01-15T14:30:39.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:30:39.321 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.321 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:39.322 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.322 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.322 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.322 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:30:39.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.323 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:30:39.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:30:39.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:30:39.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:39.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:30:39.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:30:39.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:30:39.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:30:39.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:30:39.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:30:39.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:30:39.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:30:39.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:39.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:30:39.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:30:39.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:30:39.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:30:39.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:39.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:30:39.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:30:39.512 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.512 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.512 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.513 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:39.513 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:30:39.513 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.240472857999997, 2023-01-15T14:30:39.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:30:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:30:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.514 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:30:39.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:30:39.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:30:39.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:39.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:30:39.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:30:39.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:30:39.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:30:39.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:39.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:30:39.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:30:39.519 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.519 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.519 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.519 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:39.519 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:30:39.519 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.240416961999998, 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:39.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:30:39.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:30:39.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:30:39.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:30:39.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:30:39.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:30:39.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:30:39.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:30:39.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:39.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:30:39.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:30:39.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:30:39.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:30:39.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:39.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:30:39.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:30:39.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.525 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.526 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1314 3.18 3.afe52a78 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:30:39.526 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:30:39.526 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.910121830999998, 2023-01-15T14:30:39.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62560938700000002, 2023-01-15T14:30:39.527 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.527 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:30:39.527 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.527 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.527 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.528 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1314 2023-01-15T14:30:39.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.528 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:30:39.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:30:39.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359877+0000", 2023-01-15T14:30:39.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:39.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359882+0000", 2023-01-15T14:30:39.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0050000000000004e-06 2023-01-15T14:30:39.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359893+0000", 2023-01-15T14:30:39.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0676000000000001e-05 2023-01-15T14:30:39.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359904+0000", 2023-01-15T14:30:39.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0788000000000001e-05 2023-01-15T14:30:39.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460094+0000", 2023-01-15T14:30:39.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10019024999999999 2023-01-15T14:30:39.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:39.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460106+0000", 2023-01-15T14:30:39.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354999999999999e-05 2023-01-15T14:30:39.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985477+0000", 2023-01-15T14:30:39.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52537039100000005 2023-01-15T14:30:39.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:39.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985487+0000", 2023-01-15T14:30:39.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:30:39.535 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.535 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.535 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.535 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1335 3.8 3:17d5d741:::smithi09025264-5:head [write 3252761~175867] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:30:39.536 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:30:39.536 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.338102569999997, 2023-01-15T14:30:39.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66052243499999996, 2023-01-15T14:30:39.536 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.536 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:39.536 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.536 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.537 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.537 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1335 2023-01-15T14:30:39.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.537 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:30:39.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:30:39.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931896+0000", 2023-01-15T14:30:39.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:39.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931983+0000", 2023-01-15T14:30:39.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6470000000000004e-05 2023-01-15T14:30:39.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931995+0000", 2023-01-15T14:30:39.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2075e-05 2023-01-15T14:30:39.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932003+0000", 2023-01-15T14:30:39.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5059999999999995e-06 2023-01-15T14:30:39.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932346+0000", 2023-01-15T14:30:39.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034280299999999998 2023-01-15T14:30:39.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:39.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932357+0000", 2023-01-15T14:30:39.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.13e-05 2023-01-15T14:30:39.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189006+0000", 2023-01-15T14:30:39.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25664859899999998 2023-01-15T14:30:39.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:39.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189336+0000", 2023-01-15T14:30:39.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033030899999999998 2023-01-15T14:30:39.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:39.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189481+0000", 2023-01-15T14:30:39.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014484400000000001 2023-01-15T14:30:39.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:39.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189768+0000", 2023-01-15T14:30:39.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028704300000000001 2023-01-15T14:30:39.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:39.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189787+0000", 2023-01-15T14:30:39.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9343000000000001e-05 2023-01-15T14:30:39.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:39.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191361+0000", 2023-01-15T14:30:39.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015734029999999999 2023-01-15T14:30:39.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:39.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191469+0000", 2023-01-15T14:30:39.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010859899999999999 2023-01-15T14:30:39.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:39.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391274+0000", 2023-01-15T14:30:39.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19980450899999999 2023-01-15T14:30:39.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:39.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592067+0000", 2023-01-15T14:30:39.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20079332 2023-01-15T14:30:39.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:39.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592190+0000", 2023-01-15T14:30:39.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012291400000000001 2023-01-15T14:30:39.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:39.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592420+0000", 2023-01-15T14:30:39.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000229371 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1336 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (56) in=7b,truncate 3428628] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.337985480999997, 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66113910799999998, 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1336 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:30:39.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932013+0000", 2023-01-15T14:30:39.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:39.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932019+0000", 2023-01-15T14:30:39.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5380000000000002e-06 2023-01-15T14:30:39.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932027+0000", 2023-01-15T14:30:39.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5150000000000006e-06 2023-01-15T14:30:39.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932034+0000", 2023-01-15T14:30:39.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2023-01-15T14:30:39.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032497+0000", 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100463001 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032518+0000", 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1359999999999999e-05 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189826+0000", 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15730834299999999 2023-01-15T14:30:39.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:39.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189951+0000", 2023-01-15T14:30:39.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124583 2023-01-15T14:30:39.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:39.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190093+0000", 2023-01-15T14:30:39.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014221399999999999 2023-01-15T14:30:39.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:39.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190253+0000", 2023-01-15T14:30:39.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015953000000000001 2023-01-15T14:30:39.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:39.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191365+0000", 2023-01-15T14:30:39.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011123190000000001 2023-01-15T14:30:39.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:39.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191439+0000", 2023-01-15T14:30:39.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3648999999999999e-05 2023-01-15T14:30:39.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:39.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191491+0000", 2023-01-15T14:30:39.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2444999999999997e-05 2023-01-15T14:30:39.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:39.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391472+0000", 2023-01-15T14:30:39.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199981193 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592619+0000", 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20114642499999999 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592748+0000", 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012950200000000001 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:39.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593153+0000", 2023-01-15T14:30:39.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040503700000000002 2023-01-15T14:30:39.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.919 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:39.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:39.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.919 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1337 3.8 3:17d5d741:::smithi09025264-5:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:30:39.919 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:30:39.919 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.337957252000002, 2023-01-15T14:30:39.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66179168200000005, 2023-01-15T14:30:39.920 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:39.920 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:39.920 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:39.920 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:39.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:39.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1337 2023-01-15T14:30:39.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.921 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:39.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:39.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:30:39.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:39.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:30:39.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:39.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:39.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:30:39.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:39.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:39.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932046+0000", 2023-01-15T14:30:39.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8089999999999999e-06 2023-01-15T14:30:39.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:39.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932055+0000", 2023-01-15T14:30:39.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2199999999999998e-06 2023-01-15T14:30:39.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:39.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932061+0000", 2023-01-15T14:30:39.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2779999999999999e-06 2023-01-15T14:30:39.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:39.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190318+0000", 2023-01-15T14:30:39.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.258256967 2023-01-15T14:30:39.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:39.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:39.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190413+0000", 2023-01-15T14:30:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4627000000000001e-05 2023-01-15T14:30:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:40.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593288+0000", 2023-01-15T14:30:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.402875339 2023-01-15T14:30:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593515+0000", 2023-01-15T14:30:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022700699999999999 2023-01-15T14:30:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:40.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593834+0000", 2023-01-15T14:30:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031906799999999999 2023-01-15T14:30:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:40.100 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:40.100 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:40.101 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:30:40.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:30:40.102 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:40.099+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:40.103 INFO:tasks.rados.rados.0.smithi090.stdout:625: done (0 left) 2023-01-15T14:30:40.103 INFO:tasks.rados.rados.0.smithi090.stdout:626: copy_from oid 1 from oid 20 current snap is 60 2023-01-15T14:30:40.103 INFO:tasks.rados.rados.0.smithi090.stdout:627: read oid 48 snap 57 2023-01-15T14:30:40.103 INFO:tasks.rados.rados.0.smithi090.stdout:627: expect (ObjNum 132 snap 24 seq_num 132) 2023-01-15T14:30:40.103 INFO:tasks.rados.rados.0.smithi090.stdout:628: copy_from oid 45 from oid 30 current snap is 60 2023-01-15T14:30:40.103 INFO:tasks.rados.rados.0.smithi090.stdout:629: rollback oid 14 current snap is 60 2023-01-15T14:30:40.104 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 14 to 58 2023-01-15T14:30:40.104 INFO:tasks.rados.rados.0.smithi090.stdout:630: snap_remove snap 51 2023-01-15T14:30:40.105 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:30:40.107 INFO:tasks.rados.rados.0.smithi090.stdout:630: done (4 left) 2023-01-15T14:30:40.107 INFO:tasks.rados.rados.0.smithi090.stdout:631: write (excl) oid 33 current snap is 60 2023-01-15T14:30:40.107 INFO:tasks.rados.rados.0.smithi090.stdout:631: seq_num 252 ranges {635364=402397,1763227=472137} 2023-01-15T14:30:40.107 INFO:tasks.rados.rados.0.smithi090.stdout:631: writing smithi09025264-33 from 635364 to 1037761 tid 1 2023-01-15T14:30:40.107 INFO:tasks.rados.rados.0.smithi090.stdout:631: writing smithi09025264-33 from 1763227 to 2235364 tid 2 2023-01-15T14:30:40.107 INFO:tasks.rados.rados.0.smithi090.stdout:632: write oid 3 current snap is 60 2023-01-15T14:30:40.108 INFO:tasks.rados.rados.0.smithi090.stdout:626: finishing copy_from to smithi09025264-1 2023-01-15T14:30:40.108 INFO:tasks.rados.rados.0.smithi090.stdout:626: got expected ENOENT (src dne) 2023-01-15T14:30:40.108 INFO:tasks.rados.rados.0.smithi090.stdout:632: seq_num 253 ranges {652171=477646,1821709=430462} 2023-01-15T14:30:40.108 INFO:tasks.rados.rados.0.smithi090.stdout:632: writing smithi09025264-3 from 652171 to 1129817 tid 1 2023-01-15T14:30:40.108 INFO:tasks.rados.rados.0.smithi090.stdout:632: writing smithi09025264-3 from 1821709 to 2252171 tid 2 2023-01-15T14:30:40.108 INFO:tasks.rados.rados.0.smithi090.stdout:633: read oid 16 snap 58 2023-01-15T14:30:40.108 INFO:tasks.rados.rados.0.smithi090.stdout:633: expect (ObjNum 209 snap 47 seq_num 209) 2023-01-15T14:30:40.108 INFO:tasks.rados.rados.0.smithi090.stdout:634: write oid 25 current snap is 60 2023-01-15T14:30:40.108 INFO:tasks.rados.rados.0.smithi090.stdout:626: finishing copy_from racing read to smithi09025264-1 2023-01-15T14:30:40.108 INFO:tasks.rados.rados.0.smithi090.stdout:634: seq_num 254 ranges {668978=552895,1763838=505140} 2023-01-15T14:30:40.108 INFO:tasks.rados.rados.0.smithi090.stdout:634: writing smithi09025264-25 from 668978 to 1221873 tid 1 2023-01-15T14:30:40.109 INFO:tasks.rados.rados.0.smithi090.stdout:634: writing smithi09025264-25 from 1763838 to 2268978 tid 2 2023-01-15T14:30:40.109 INFO:tasks.rados.rados.0.smithi090.stdout:626: done (7 left) 2023-01-15T14:30:40.109 INFO:tasks.rados.rados.0.smithi090.stdout:635: write oid 32 current snap is 60 2023-01-15T14:30:40.109 INFO:tasks.rados.rados.0.smithi090.stdout:635: seq_num 255 ranges {685785=628144,1822320=463465} 2023-01-15T14:30:40.109 INFO:tasks.rados.rados.0.smithi090.stdout:635: writing smithi09025264-32 from 685785 to 1313929 tid 1 2023-01-15T14:30:40.109 INFO:tasks.rados.rados.0.smithi090.stdout:635: writing smithi09025264-32 from 1822320 to 2285785 tid 2 2023-01-15T14:30:40.109 INFO:tasks.rados.rados.0.smithi090.stdout:636: read oid 31 snap -1 2023-01-15T14:30:40.109 INFO:tasks.rados.rados.0.smithi090.stdout:636: expect deleted 2023-01-15T14:30:40.109 INFO:tasks.rados.rados.0.smithi090.stdout:637: write oid 46 current snap is 60 2023-01-15T14:30:40.109 INFO:tasks.rados.rados.0.smithi090.stdout:637: seq_num 256 ranges {702592=703393,1880802=421790} 2023-01-15T14:30:40.109 INFO:tasks.rados.rados.0.smithi090.stdout:637: writing smithi09025264-46 from 702592 to 1405985 tid 1 2023-01-15T14:30:40.110 INFO:tasks.rados.rados.0.smithi090.stdout:637: writing smithi09025264-46 from 1880802 to 2302592 tid 2 2023-01-15T14:30:40.110 INFO:tasks.rados.rados.0.smithi090.stdout:638: delete oid 19 current snap is 60 2023-01-15T14:30:40.110 INFO:tasks.rados.rados.0.smithi090.stdout:628: finishing copy_from to smithi09025264-45 2023-01-15T14:30:40.110 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 73 (ObjNum 206 snap 44 seq_num 206) dirty exists 2023-01-15T14:30:40.110 INFO:tasks.rados.rados.0.smithi090.stdout:628: finishing copy_from racing read to smithi09025264-45 2023-01-15T14:30:40.110 INFO:tasks.rados.rados.0.smithi090.stdout:627: expect (ObjNum 132 snap 24 seq_num 132) 2023-01-15T14:30:40.110 INFO:tasks.rados.rados.0.smithi090.stdout:629: finishing rollback tid 0 to smithi09025264-14 2023-01-15T14:30:40.110 INFO:tasks.rados.rados.0.smithi090.stdout:629: finishing rollback tid 1 to smithi09025264-14 2023-01-15T14:30:40.110 INFO:tasks.rados.rados.0.smithi090.stdout:629: finishing rollback tid 2 to smithi09025264-14 2023-01-15T14:30:40.110 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 221 (ObjNum 222 snap 51 seq_num 222) dirty exists 2023-01-15T14:30:40.110 INFO:tasks.rados.rados.0.smithi090.stdout:633: expect (ObjNum 209 snap 47 seq_num 209) 2023-01-15T14:30:40.194 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7] out_osds: [5, 6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:30:40.298 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:30:40.298 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:30:40.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:30:40.302 INFO:tasks.rados.rados.0.smithi090.stdout:632: finishing write tid 1 to smithi09025264-3 2023-01-15T14:30:40.302 INFO:tasks.rados.rados.0.smithi090.stdout:632: finishing write tid 2 to smithi09025264-3 2023-01-15T14:30:40.302 INFO:tasks.rados.rados.0.smithi090.stdout:637: finishing write tid 1 to smithi09025264-46 2023-01-15T14:30:40.302 INFO:tasks.rados.rados.0.smithi090.stdout:637: finishing write tid 2 to smithi09025264-46 2023-01-15T14:30:40.302 INFO:tasks.rados.rados.0.smithi090.stdout:632: finishing write tid 3 to smithi09025264-3 2023-01-15T14:30:40.302 INFO:tasks.rados.rados.0.smithi090.stdout:632: finishing write tid 4 to smithi09025264-3 2023-01-15T14:30:40.302 INFO:tasks.rados.rados.0.smithi090.stdout:632: oid 3 updating version 0 to 206 2023-01-15T14:30:40.303 INFO:tasks.rados.rados.0.smithi090.stdout:632: oid 3 updating version 206 to 207 2023-01-15T14:30:40.303 INFO:tasks.rados.rados.0.smithi090.stdout:632: oid 3 updating version 207 to 208 2023-01-15T14:30:40.303 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 208 (ObjNum 253 snap 60 seq_num 253) dirty exists 2023-01-15T14:30:40.303 INFO:tasks.rados.rados.0.smithi090.stdout:632: left oid 3 (ObjNum 253 snap 60 seq_num 253) 2023-01-15T14:30:40.303 INFO:tasks.rados.rados.0.smithi090.stdout:634: finishing write tid 1 to smithi09025264-25 2023-01-15T14:30:40.303 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:40.303 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:40.304 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:40.304 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:30:40.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:1704 3.11 e153/131 3:8889e450:::smithi09025264-32:head v 153'121, mlcod=152'118)", 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.199236+0000", 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.019564372, 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.019592274999999999, 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.199236+0000", 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.305 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:40.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.199236+0000", 2023-01-15T14:30:40.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:40.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.199232+0000", 2023-01-15T14:30:40.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999962 2023-01-15T14:30:40.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.199791+0000", 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000559769 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.199819+0000", 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7682e-05 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:40.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.199833+0000", 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3623e-05 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.199939+0000", 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106612 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.199980+0000", 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0914000000000002e-05 2023-01-15T14:30:40.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.200746+0000", 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00076533899999999997 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.200766+0000", 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9995000000000001e-05 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:40.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.201151+0000", 2023-01-15T14:30:40.310 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000385941 2023-01-15T14:30:40.310 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.310 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.310 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:40.310 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.201217+0000", 2023-01-15T14:30:40.310 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5382999999999994e-05 2023-01-15T14:30:40.310 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:40.311 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:40.311 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:40.311 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.311 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.311 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:1705 3.11 e153/131 3:8889e450:::smithi09025264-32:head v 153'122, mlcod=152'118)", 2023-01-15T14:30:40.311 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.201540+0000", 2023-01-15T14:30:40.311 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.017260059000000001, 2023-01-15T14:30:40.311 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.017379319000000001, 2023-01-15T14:30:40.311 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:40.312 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:40.312 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:40.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:40.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.201540+0000", 2023-01-15T14:30:40.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:40.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.201540+0000", 2023-01-15T14:30:40.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:40.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.201539+0000", 2023-01-15T14:30:40.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:40.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:40.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.201895+0000", 2023-01-15T14:30:40.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035609999999999998 2023-01-15T14:30:40.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:40.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.201914+0000", 2023-01-15T14:30:40.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9306000000000001e-05 2023-01-15T14:30:40.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:40.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.201922+0000", 2023-01-15T14:30:40.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2500000000000006e-06 2023-01-15T14:30:40.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:40.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202004+0000", 2023-01-15T14:30:40.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2174000000000003e-05 2023-01-15T14:30:40.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:40.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202026+0000", 2023-01-15T14:30:40.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1078999999999999e-05 2023-01-15T14:30:40.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:40.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202251+0000", 2023-01-15T14:30:40.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022551099999999999 2023-01-15T14:30:40.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:40.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202271+0000", 2023-01-15T14:30:40.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0049999999999999e-05 2023-01-15T14:30:40.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:40.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202511+0000", 2023-01-15T14:30:40.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024016700000000001 2023-01-15T14:30:40.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:40.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202549+0000", 2023-01-15T14:30:40.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7766999999999999e-05 2023-01-15T14:30:40.318 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:40.318 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:40.319 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:40.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.319 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:1706 3.11 e153/131 3:8889e450:::smithi09025264-32:head v 153'123, mlcod=152'118)", 2023-01-15T14:30:40.319 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.202849+0000", 2023-01-15T14:30:40.319 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.015950672999999999, 2023-01-15T14:30:40.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.016152126999999999, 2023-01-15T14:30:40.319 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:40.319 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:40.319 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:40.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:40.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202849+0000", 2023-01-15T14:30:40.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:40.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202849+0000", 2023-01-15T14:30:40.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:40.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202848+0000", 2023-01-15T14:30:40.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:40.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:40.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202856+0000", 2023-01-15T14:30:40.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5050000000000001e-06 2023-01-15T14:30:40.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:40.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202872+0000", 2023-01-15T14:30:40.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.573e-05 2023-01-15T14:30:40.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:40.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202879+0000", 2023-01-15T14:30:40.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8719999999999996e-06 2023-01-15T14:30:40.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:40.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.202966+0000", 2023-01-15T14:30:40.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7658000000000007e-05 2023-01-15T14:30:40.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:40.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.203011+0000", 2023-01-15T14:30:40.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.5127999999999999e-05 2023-01-15T14:30:40.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:40.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.203236+0000", 2023-01-15T14:30:40.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000224281 2023-01-15T14:30:40.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:40.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.203253+0000", 2023-01-15T14:30:40.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7413000000000001e-05 2023-01-15T14:30:40.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:40.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.203311+0000", 2023-01-15T14:30:40.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7855000000000001e-05 2023-01-15T14:30:40.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:40.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.203340+0000", 2023-01-15T14:30:40.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.9362000000000001e-05 2023-01-15T14:30:40.662 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:40.662 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:40.662 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:40.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.662 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:1701 3.a e153/147)", 2023-01-15T14:30:40.662 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.205300+0000", 2023-01-15T14:30:40.662 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.013499651999999999, 2023-01-15T14:30:40.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.013781455999999999, 2023-01-15T14:30:40.663 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:40.663 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "no flag points reached", 2023-01-15T14:30:40.663 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:40.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:40.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.205300+0000", 2023-01-15T14:30:40.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:40.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.205300+0000", 2023-01-15T14:30:40.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:40.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.205298+0000", 2023-01-15T14:30:40.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:40.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:40.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.205312+0000", 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.314e-05 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.205346+0000", 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4792000000000002e-05 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.665 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1695 3.a 3:52c3e133:::smithi09025264-3:head [setxattr _header (56) in=63b,truncate 2252171] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:30:40.666 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:39.971046+0000", 2023-01-15T14:30:40.666 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.24775412099999999, 2023-01-15T14:30:40.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24808766400000001, 2023-01-15T14:30:40.666 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:40.666 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:30:40.666 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:40.666 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:40.666 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:40.666 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1695 2023-01-15T14:30:40.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.666 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:40.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:40.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971046+0000", 2023-01-15T14:30:40.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:40.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971046+0000", 2023-01-15T14:30:40.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:40.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971045+0000", 2023-01-15T14:30:40.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:40.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:40.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971052+0000", 2023-01-15T14:30:40.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7070000000000003e-06 2023-01-15T14:30:40.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:40.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971061+0000", 2023-01-15T14:30:40.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6569999999999992e-06 2023-01-15T14:30:40.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:40.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971069+0000", 2023-01-15T14:30:40.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3390000000000004e-06 2023-01-15T14:30:40.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.973899+0000", 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0028304099999999998 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.974173+0000", 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027367800000000002 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,3", 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.974510+0000", 2023-01-15T14:30:40.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033783700000000001 2023-01-15T14:30:40.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:40.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.974928+0000", 2023-01-15T14:30:40.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000417575 2023-01-15T14:30:40.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:40.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.974943+0000", 2023-01-15T14:30:40.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5113000000000001e-05 2023-01-15T14:30:40.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:40.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.974999+0000", 2023-01-15T14:30:40.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5664000000000001e-05 2023-01-15T14:30:40.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:40.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.975058+0000", 2023-01-15T14:30:40.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9401000000000003e-05 2023-01-15T14:30:40.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:40.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.147948+0000", 2023-01-15T14:30:40.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17288969400000001 2023-01-15T14:30:40.673 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:40.673 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:40.673 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:40.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.673 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1696 3.a 3:52c3e133:::smithi09025264-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:30:40.673 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:39.971078+0000", 2023-01-15T14:30:40.673 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.24772164599999999, 2023-01-15T14:30:40.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24816023100000001, 2023-01-15T14:30:40.674 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:40.674 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:30:40.674 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:40.845 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:40.845 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:40.845 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1696 2023-01-15T14:30:40.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.846 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:40.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:40.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971078+0000", 2023-01-15T14:30:40.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971078+0000", 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971078+0000", 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971082+0000", 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2950000000000004e-06 2023-01-15T14:30:40.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:40.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971092+0000", 2023-01-15T14:30:40.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6900000000000004e-06 2023-01-15T14:30:40.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:40.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971099+0000", 2023-01-15T14:30:40.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3409999999999997e-06 2023-01-15T14:30:40.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:40.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:40.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:40.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.975137+0000", 2023-01-15T14:30:40.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0040379220000000002 2023-01-15T14:30:41.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:41.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.975456+0000", 2023-01-15T14:30:41.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000318697 2023-01-15T14:30:41.029 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.029 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1700 3.a 3:52df5138:::smithi09025264-25:head [write 668978~552895 in=552895b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:30:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:39.971108+0000", 2023-01-15T14:30:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.24769182200000001, 2023-01-15T14:30:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24819938799999999, 2023-01-15T14:30:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:30:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:41.030 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:41.031 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:41.031 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1700 2023-01-15T14:30:41.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.031 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971108+0000", 2023-01-15T14:30:41.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971108+0000", 2023-01-15T14:30:41.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.037 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971108+0000", 2023-01-15T14:30:41.037 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:41.037 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.037 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.038 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971505+0000", 2023-01-15T14:30:41.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039754599999999998 2023-01-15T14:30:41.038 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.038 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.038 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.038 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971522+0000", 2023-01-15T14:30:41.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6832000000000001e-05 2023-01-15T14:30:41.038 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.038 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.038 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.038 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971533+0000", 2023-01-15T14:30:41.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1418e-05 2023-01-15T14:30:41.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:41.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.975553+0000", 2023-01-15T14:30:41.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.004019636 2023-01-15T14:30:41.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:41.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.975742+0000", 2023-01-15T14:30:41.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018876 2023-01-15T14:30:41.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,3", 2023-01-15T14:30:41.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.976096+0000", 2023-01-15T14:30:41.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000354459 2023-01-15T14:30:41.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:41.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.977182+0000", 2023-01-15T14:30:41.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010854230000000001 2023-01-15T14:30:41.041 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.041 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.041 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:41.041 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.977199+0000", 2023-01-15T14:30:41.041 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7269999999999999e-05 2023-01-15T14:30:41.041 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.041 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.041 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:41.042 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.977554+0000", 2023-01-15T14:30:41.042 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035539300000000002 2023-01-15T14:30:41.042 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.042 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:41.042 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.979711+0000", 2023-01-15T14:30:41.042 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021571139999999999 2023-01-15T14:30:41.042 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.042 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:41.042 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152563+0000", 2023-01-15T14:30:41.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17285186399999999 2023-01-15T14:30:41.043 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.043 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.043 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.043 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1701 3.a 3:52df5138:::smithi09025264-25:head [write 1763838~505140 in=505140b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:30:41.215 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:39.971549+0000", 2023-01-15T14:30:41.215 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.247250474, 2023-01-15T14:30:41.215 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24785738800000001, 2023-01-15T14:30:41.215 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.215 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:30:41.215 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:41.215 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:41.215 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:41.215 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1701 2023-01-15T14:30:41.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.216 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971549+0000", 2023-01-15T14:30:41.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971549+0000", 2023-01-15T14:30:41.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.971548+0000", 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972474+0000", 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00092583400000000005 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972497+0000", 2023-01-15T14:30:41.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2464999999999999e-05 2023-01-15T14:30:41.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972533+0000", 2023-01-15T14:30:41.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6497e-05 2023-01-15T14:30:41.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:41.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.977321+0000", 2023-01-15T14:30:41.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0047876869999999997 2023-01-15T14:30:41.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:41.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.977566+0000", 2023-01-15T14:30:41.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024559199999999999 2023-01-15T14:30:41.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,3", 2023-01-15T14:30:41.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.977844+0000", 2023-01-15T14:30:41.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027753100000000002 2023-01-15T14:30:41.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:41.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.978496+0000", 2023-01-15T14:30:41.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065247099999999995 2023-01-15T14:30:41.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:41.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.978513+0000", 2023-01-15T14:30:41.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6399e-05 2023-01-15T14:30:41.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:41.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.978877+0000", 2023-01-15T14:30:41.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036386099999999999 2023-01-15T14:30:41.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:41.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.979729+0000", 2023-01-15T14:30:41.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00085273399999999996 2023-01-15T14:30:41.221 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.221 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1702 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (57) in=64b,truncate 2268978] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:39.972553+0000", 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.246246729, 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.246949115, 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:41.222 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:41.223 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1702 2023-01-15T14:30:41.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.223 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972553+0000", 2023-01-15T14:30:41.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972553+0000", 2023-01-15T14:30:41.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972552+0000", 2023-01-15T14:30:41.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:41.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972561+0000", 2023-01-15T14:30:41.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0799999999999995e-06 2023-01-15T14:30:41.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972575+0000", 2023-01-15T14:30:41.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4384e-05 2023-01-15T14:30:41.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972602+0000", 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6755e-05 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.978651+0000", 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0060486940000000003 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.978865+0000", 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021482200000000001 2023-01-15T14:30:41.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,3", 2023-01-15T14:30:41.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.979247+0000", 2023-01-15T14:30:41.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000381999 2023-01-15T14:30:41.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:41.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.979636+0000", 2023-01-15T14:30:41.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000388658 2023-01-15T14:30:41.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:41.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.979647+0000", 2023-01-15T14:30:41.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1467999999999999e-05 2023-01-15T14:30:41.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:41.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.979675+0000", 2023-01-15T14:30:41.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7761999999999999e-05 2023-01-15T14:30:41.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:41.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.979745+0000", 2023-01-15T14:30:41.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9422e-05 2023-01-15T14:30:41.229 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.229 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.229 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.229 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1703 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:30:41.229 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:39.972629+0000", 2023-01-15T14:30:41.400 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.246171376, 2023-01-15T14:30:41.400 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24697533199999999, 2023-01-15T14:30:41.400 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.400 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1703 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972629+0000", 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972629+0000", 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972627+0000", 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972635+0000", 2023-01-15T14:30:41.403 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4449999999999999e-06 2023-01-15T14:30:41.405 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.405 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.405 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.405 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972651+0000", 2023-01-15T14:30:41.406 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5863e-05 2023-01-15T14:30:41.406 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.406 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.406 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972661+0000", 2023-01-15T14:30:41.406 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0171e-05 2023-01-15T14:30:41.406 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.406 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:41.407 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.079923+0000", 2023-01-15T14:30:41.407 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.107261708 2023-01-15T14:30:41.407 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.407 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.407 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:41.407 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.080115+0000", 2023-01-15T14:30:41.407 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019224300000000001 2023-01-15T14:30:41.407 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.408 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.408 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.408 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.408 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.408 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1715 3.f 3:f8edd34e:::smithi09025264-19:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:30:41.409 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:39.972674+0000", 2023-01-15T14:30:41.409 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.24612569100000001, 2023-01-15T14:30:41.409 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24699553399999999, 2023-01-15T14:30:41.409 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.409 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:30:41.410 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:41.410 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:41.410 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:41.410 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1715 2023-01-15T14:30:41.410 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.411 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.411 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.411 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.411 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972674+0000", 2023-01-15T14:30:41.411 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.412 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.412 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.412 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.412 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972674+0000", 2023-01-15T14:30:41.412 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.413 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.413 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.413 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.413 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972673+0000", 2023-01-15T14:30:41.413 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:41.413 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.414 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.414 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.414 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972687+0000", 2023-01-15T14:30:41.414 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4002000000000001e-05 2023-01-15T14:30:41.414 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.415 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.415 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.415 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972703+0000", 2023-01-15T14:30:41.415 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5454000000000001e-05 2023-01-15T14:30:41.415 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.415 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.416 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.416 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972713+0000", 2023-01-15T14:30:41.416 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0377000000000001e-05 2023-01-15T14:30:41.416 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:41.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972816+0000", 2023-01-15T14:30:41.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000102528 2023-01-15T14:30:41.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:30:41.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.972833+0000", 2023-01-15T14:30:41.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7672e-05 2023-01-15T14:30:41.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.974498+0000", 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001664579 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.974794+0000", 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029569399999999999 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,3", 2023-01-15T14:30:41.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.975112+0000", 2023-01-15T14:30:41.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00031858400000000001 2023-01-15T14:30:41.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:41.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.976014+0000", 2023-01-15T14:30:41.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00090148799999999999 2023-01-15T14:30:41.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:41.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.976028+0000", 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4219999999999999e-05 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.976626+0000", 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059789200000000004 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:39.977086+0000", 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046015699999999999 2023-01-15T14:30:41.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:41.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.080311+0000", 2023-01-15T14:30:41.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103225057 2023-01-15T14:30:41.586 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.586 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.586 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.586 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:1715 3.f e153/152)", 2023-01-15T14:30:41.586 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.058422+0000", 2023-01-15T14:30:41.587 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.160377723, 2023-01-15T14:30:41.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.16136014200000001, 2023-01-15T14:30:41.587 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.587 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:41.587 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.058422+0000", 2023-01-15T14:30:41.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.058422+0000", 2023-01-15T14:30:41.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.058421+0000", 2023-01-15T14:30:41.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:41.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.058432+0000", 2023-01-15T14:30:41.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1454000000000001e-05 2023-01-15T14:30:41.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.058459+0000", 2023-01-15T14:30:41.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973000000000001e-05 2023-01-15T14:30:41.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.058472+0000", 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2397e-05 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.4.0:349 3.f e153/152)", 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.076217+0000", 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.14258343000000001, 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.14362688500000001, 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.590 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076217+0000", 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076217+0000", 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076216+0000", 2023-01-15T14:30:41.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:41.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076223+0000", 2023-01-15T14:30:41.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5530000000000004e-06 2023-01-15T14:30:41.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076242+0000", 2023-01-15T14:30:41.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.872e-05 2023-01-15T14:30:41.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076253+0000", 2023-01-15T14:30:41.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1430000000000001e-05 2023-01-15T14:30:41.593 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.593 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.593 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.4.0:351 3.f e153/152)", 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.076630+0000", 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.14216954800000001, 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.14325681800000001, 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076630+0000", 2023-01-15T14:30:41.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076630+0000", 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076629+0000", 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.595 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.596 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076641+0000", 2023-01-15T14:30:41.596 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2435999999999999e-05 2023-01-15T14:30:41.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076662+0000", 2023-01-15T14:30:41.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0829999999999999e-05 2023-01-15T14:30:41.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.076679+0000", 2023-01-15T14:30:41.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6668999999999999e-05 2023-01-15T14:30:41.766 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.766 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.766 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:1695 3.a e153/147)", 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.152550+0000", 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.066249789000000003, 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.067381457000000006, 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152550+0000", 2023-01-15T14:30:41.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152550+0000", 2023-01-15T14:30:41.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152550+0000", 2023-01-15T14:30:41.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:41.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152553+0000", 2023-01-15T14:30:41.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1599999999999998e-06 2023-01-15T14:30:41.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152560+0000", 2023-01-15T14:30:41.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2969999999999998e-06 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152565+0000", 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0069999999999997e-06 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:1700 3.a e153/147)", 2023-01-15T14:30:41.770 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.152622+0000", 2023-01-15T14:30:41.771 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.066177997000000002, 2023-01-15T14:30:41.771 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.067359927999999999, 2023-01-15T14:30:41.771 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.771 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:41.771 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.771 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.771 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.771 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152622+0000", 2023-01-15T14:30:41.771 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.771 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.771 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.772 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.772 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152622+0000", 2023-01-15T14:30:41.772 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.772 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.772 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.772 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.772 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152622+0000", 2023-01-15T14:30:41.772 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152625+0000", 2023-01-15T14:30:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5140000000000002e-06 2023-01-15T14:30:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152634+0000", 2023-01-15T14:30:41.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8729999999999996e-06 2023-01-15T14:30:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152640+0000", 2023-01-15T14:30:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0390000000000001e-06 2023-01-15T14:30:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:1701 3.a e153/147)", 2023-01-15T14:30:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.152654+0000", 2023-01-15T14:30:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.066145615000000005, 2023-01-15T14:30:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.067378797000000004, 2023-01-15T14:30:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152654+0000", 2023-01-15T14:30:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152654+0000", 2023-01-15T14:30:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152654+0000", 2023-01-15T14:30:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152657+0000", 2023-01-15T14:30:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2569999999999999e-06 2023-01-15T14:30:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152664+0000", 2023-01-15T14:30:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1559999999999998e-06 2023-01-15T14:30:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.152670+0000", 2023-01-15T14:30:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2270000000000001e-06 2023-01-15T14:30:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.3.0:348 3.b e153/20)", 2023-01-15T14:30:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.155200+0000", 2023-01-15T14:30:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.063599638, 2023-01-15T14:30:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.064876469000000006, 2023-01-15T14:30:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.155200+0000", 2023-01-15T14:30:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.155200+0000", 2023-01-15T14:30:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.155199+0000", 2023-01-15T14:30:41.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.155210+0000", 2023-01-15T14:30:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0876999999999999e-05 2023-01-15T14:30:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.155227+0000", 2023-01-15T14:30:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6929999999999999e-05 2023-01-15T14:30:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.155235+0000", 2023-01-15T14:30:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2989999999999997e-06 2023-01-15T14:30:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.4.0:351 3.f e153/152)", 2023-01-15T14:30:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:40.157386+0000", 2023-01-15T14:30:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.061414153999999999, 2023-01-15T14:30:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.062736933999999994, 2023-01-15T14:30:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:41.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.157386+0000", 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.157386+0000", 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.157385+0000", 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:41.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.157392+0000", 2023-01-15T14:30:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5300000000000002e-06 2023-01-15T14:30:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.157408+0000", 2023-01-15T14:30:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6049000000000002e-05 2023-01-15T14:30:41.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:40.157415+0000", 2023-01-15T14:30:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6869999999999994e-06 2023-01-15T14:30:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2023-01-15T14:30:41.830 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 19 2023-01-15T14:30:41.830 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:41.831 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:30:41.832 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:40.299+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:627: done (10 left) 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:628: done (9 left) 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:629: done (8 left) 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:632: done (7 left) 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:633: done (6 left) 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:636: done (5 left) 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:638: done (4 left) 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:639: copy_from oid 20 from oid 38 current snap is 60 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:640: copy_from oid 45 from oid 19 current snap is 60 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:641: read oid 49 snap 57 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:641: expect (ObjNum 199 snap 43 seq_num 199) 2023-01-15T14:30:41.833 INFO:tasks.rados.rados.0.smithi090.stdout:642: delete oid 4 current snap is 60 2023-01-15T14:30:41.834 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:41.835 INFO:tasks.rados.rados.0.smithi090.stdout:634: finishing write tid 2 to smithi09025264-25 2023-01-15T14:30:42.030 INFO:tasks.rados.rados.0.smithi090.stdout:634: finishing write tid 3 to smithi09025264-25 2023-01-15T14:30:42.030 INFO:tasks.rados.rados.0.smithi090.stdout:634: finishing write tid 4 to smithi09025264-25 2023-01-15T14:30:42.031 INFO:tasks.rados.rados.0.smithi090.stdout:634: oid 25 updating version 0 to 209 2023-01-15T14:30:42.031 INFO:tasks.rados.rados.0.smithi090.stdout:634: oid 25 updating version 209 to 211 2023-01-15T14:30:42.031 INFO:tasks.rados.rados.0.smithi090.stdout:634: oid 25 version 211 is already newer than 210 2023-01-15T14:30:42.031 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 211 (ObjNum 254 snap 60 seq_num 254) dirty exists 2023-01-15T14:30:42.031 INFO:tasks.rados.rados.0.smithi090.stdout:634: left oid 25 (ObjNum 254 snap 60 seq_num 254) 2023-01-15T14:30:42.031 INFO:tasks.rados.rados.0.smithi090.stdout:637: finishing write tid 3 to smithi09025264-46 2023-01-15T14:30:42.032 INFO:tasks.rados.rados.0.smithi090.stdout:637: finishing write tid 4 to smithi09025264-46 2023-01-15T14:30:42.032 INFO:tasks.rados.rados.0.smithi090.stdout:637: oid 46 updating version 0 to 169 2023-01-15T14:30:42.032 INFO:tasks.rados.rados.0.smithi090.stdout:637: oid 46 updating version 169 to 171 2023-01-15T14:30:42.032 INFO:tasks.rados.rados.0.smithi090.stdout:637: oid 46 version 171 is already newer than 170 2023-01-15T14:30:42.033 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 171 (ObjNum 256 snap 60 seq_num 256) dirty exists 2023-01-15T14:30:42.033 INFO:tasks.rados.rados.0.smithi090.stdout:637: left oid 46 (ObjNum 256 snap 60 seq_num 256) 2023-01-15T14:30:42.033 INFO:tasks.rados.rados.0.smithi090.stdout:631: finishing write tid 1 to smithi09025264-33 2023-01-15T14:30:42.033 INFO:tasks.rados.rados.0.smithi090.stdout:634: done (7 left) 2023-01-15T14:30:42.035 INFO:tasks.rados.rados.0.smithi090.stdout:637: done (6 left) 2023-01-15T14:30:42.035 INFO:tasks.rados.rados.0.smithi090.stdout:642: done (5 left) 2023-01-15T14:30:42.036 INFO:tasks.rados.rados.0.smithi090.stdout:643: snap_remove snap 58 2023-01-15T14:30:42.037 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:42.037 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:42.037 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:42.038 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:42.038 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:42.039 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:30:42.039 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:30:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:30:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:30:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:30:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:30:42.040 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:30:42.041 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:30:42.042 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:41.831+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:42.047 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:42.047 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:42.050 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:30:42.052 INFO:tasks.rados.rados.0.smithi090.stdout:643: done (5 left) 2023-01-15T14:30:42.052 INFO:tasks.rados.rados.0.smithi090.stdout:644: delete oid 15 current snap is 60 2023-01-15T14:30:42.053 INFO:tasks.rados.rados.0.smithi090.stdout:639: finishing copy_from to smithi09025264-20 2023-01-15T14:30:42.053 INFO:tasks.rados.rados.0.smithi090.stdout:639: got expected ENOENT (src dne) 2023-01-15T14:30:42.053 INFO:tasks.rados.rados.0.smithi090.stdout:641: expect (ObjNum 199 snap 43 seq_num 199) 2023-01-15T14:30:42.053 INFO:tasks.rados.rados.0.smithi090.stdout:640: finishing copy_from to smithi09025264-45 2023-01-15T14:30:42.053 INFO:tasks.rados.rados.0.smithi090.stdout:640: got expected ENOENT (src dne) 2023-01-15T14:30:42.054 INFO:tasks.rados.rados.0.smithi090.stdout:640: finishing copy_from racing read to smithi09025264-45 2023-01-15T14:30:42.054 INFO:tasks.rados.rados.0.smithi090.stdout:635: finishing write tid 1 to smithi09025264-32 2023-01-15T14:30:42.054 INFO:tasks.rados.rados.0.smithi090.stdout:635: finishing write tid 2 to smithi09025264-32 2023-01-15T14:30:42.054 INFO:tasks.rados.rados.0.smithi090.stdout:635: finishing write tid 3 to smithi09025264-32 2023-01-15T14:30:42.054 INFO:tasks.rados.rados.0.smithi090.stdout:639: finishing copy_from racing read to smithi09025264-20 2023-01-15T14:30:42.054 INFO:tasks.rados.rados.0.smithi090.stdout:631: finishing write tid 2 to smithi09025264-33 2023-01-15T14:30:42.055 INFO:tasks.rados.rados.0.smithi090.stdout:631: finishing write tid 3 to smithi09025264-33 2023-01-15T14:30:42.055 INFO:tasks.rados.rados.0.smithi090.stdout:631: finishing write tid 4 to smithi09025264-33 2023-01-15T14:30:42.055 INFO:tasks.rados.rados.0.smithi090.stdout:631: oid 33 updating version 0 to 61 2023-01-15T14:30:42.055 INFO:tasks.rados.rados.0.smithi090.stdout:631: oid 33 updating version 61 to 62 2023-01-15T14:30:42.055 INFO:tasks.rados.rados.0.smithi090.stdout:631: oid 33 updating version 62 to 63 2023-01-15T14:30:42.056 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 63 (ObjNum 252 snap 60 seq_num 252) dirty exists 2023-01-15T14:30:42.056 INFO:tasks.rados.rados.0.smithi090.stdout:631: left oid 33 (ObjNum 252 snap 60 seq_num 252) 2023-01-15T14:30:42.056 INFO:tasks.rados.rados.0.smithi090.stdout:635: finishing write tid 4 to smithi09025264-32 2023-01-15T14:30:42.218 INFO:tasks.rados.rados.0.smithi090.stdout:635: oid 32 updating version 0 to 224 2023-01-15T14:30:42.218 INFO:tasks.rados.rados.0.smithi090.stdout:635: oid 32 version 224 is already newer than 222 2023-01-15T14:30:42.218 INFO:tasks.rados.rados.0.smithi090.stdout:635: oid 32 version 224 is already newer than 223 2023-01-15T14:30:42.218 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 224 (ObjNum 255 snap 60 seq_num 255) dirty exists 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:635: left oid 32 (ObjNum 255 snap 60 seq_num 255) 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:631: done (5 left) 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:635: done (4 left) 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:639: done (3 left) 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:640: done (2 left) 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:641: done (1 left) 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:644: done (0 left) 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:645: write oid 48 current snap is 60 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:645: seq_num 260 ranges {769820=720742,2114730=255090} 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:645: writing smithi09025264-48 from 769820 to 1490562 tid 1 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:645: writing smithi09025264-48 from 2114730 to 2369820 tid 2 2023-01-15T14:30:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:646: write oid 19 current snap is 60 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:646: seq_num 261 ranges {786627=795991,2173212=213415} 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:646: writing smithi09025264-19 from 786627 to 1582618 tid 1 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:646: writing smithi09025264-19 from 2173212 to 2386627 tid 2 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:647: snap_remove snap 57 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:647: done (2 left) 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:648: snap_create 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:646: finishing write tid 1 to smithi09025264-19 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:645: finishing write tid 1 to smithi09025264-48 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:645: finishing write tid 2 to smithi09025264-48 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:646: finishing write tid 2 to smithi09025264-19 2023-01-15T14:30:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:645: finishing write tid 3 to smithi09025264-48 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:645: finishing write tid 4 to smithi09025264-48 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:645: oid 48 updating version 0 to 150 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:645: oid 48 updating version 150 to 152 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:645: oid 48 version 152 is already newer than 151 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 152 (ObjNum 260 snap 60 seq_num 260) dirty exists 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:645: left oid 48 (ObjNum 260 snap 60 seq_num 260) 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:646: finishing write tid 3 to smithi09025264-19 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:646: finishing write tid 4 to smithi09025264-19 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:646: oid 19 updating version 0 to 139 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:646: oid 19 updating version 139 to 140 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:646: oid 19 updating version 140 to 141 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 141 (ObjNum 261 snap 60 seq_num 261) dirty exists 2023-01-15T14:30:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:646: left oid 19 (ObjNum 261 snap 60 seq_num 261) 2023-01-15T14:30:42.222 INFO:tasks.rados.rados.0.smithi090.stdout:645: done (1 left) 2023-01-15T14:30:42.222 INFO:tasks.rados.rados.0.smithi090.stdout:646: done (0 left) 2023-01-15T14:30:42.223 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:42.080+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:42.224 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:30:42.225 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:30:42.225 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2023-01-15T14:30:41.764830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":183352670,"num_objects":74,"num_object_clones":28,"num_object_copies":222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":7,"num_read":662,"num_read_kb":518814,"num_write":645,"num_write_kb":211421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":288,"num_bytes_recovered":193307445,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3501,"ondisk_log_size":3501,"up":105,"acting":100,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":105,"num_osds":5,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":468496900,"kb_used":4633764,"kb_used_data":463925468,"kb_used_omap":12448,"kb_used_meta":0,"kb_avail":463863136,"statfs":{"total":479740825600,"available":474995851264,"internally_reserved":63827968,"allocated":475059679232,"data_stored":475059679232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12747188,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,3,8,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":35,"commit_latency_ns":0,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11972996,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-87,"num_objects_misplaced":-10,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":36,"num_read_kb":37153,"num_write":28,"num_write_kb":16105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":101,"num_bytes_recovered":74160666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.936940"},"pg_stats":[{"pgid":"3.19","version":"144'163","reported_seq":626,"reported_epoch":153,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:30:40.483448+0000","last_change":"2023-01-15T14:30:38.659212+0000","last_active":"2023-01-15T14:30:40.483448+0000","last_peered":"2023-01-15T14:30:40.483448+0000","last_clean":"2023-01-15T14:30:31.264358+0000","last_became_active":"2023-01-15T14:30:34.992893+0000","last_became_peered":"2023-01-15T14:30:34.992893+0000","last_unstale":"2023-01-15T14:30:40.483448+0000","last_undegraded":"2023-01-15T14:30:40.483448+0000","last_fullsized":"2023-01-15T14:30:34.791342+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'157","last_scrub_stamp":"2023-01-15T14:30:26.792736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:26.792736+0000","objects_scrubbed":4,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:44.706","scrub_duration":0.61998826900000004,"objects_trimmed":0,"snaptrim_duration":0.0013192939999999999,"stat_sum":{"num_bytes":11865742,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":15024,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":2756348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4],"avail_no_missing":["3","1","4"],"object_location_counts":[{"shards":"1,3,4","objects":4},{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"68","length":"1"}]},{"pgid":"3.18","version":"144'204","reported_seq":691,"reported_epoch":153,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:30:40.384854+0000","last_change":"2023-01-15T14:30:35.303281+0000","last_active":"2023-01-15T14:30:40.384854+0000","last_peered":"2023-01-15T14:30:40.384854+0000","last_clean":"2023-01-15T14:30:31.766553+0000","last_became_active":"2023-01-15T14:30:35.202986+0000","last_became_peered":"2023-01-15T14:30:35.202986+0000","last_unstale":"2023-01-15T14:30:40.384854+0000","last_undegraded":"2023-01-15T14:30:34.791832+0000","last_fullsized":"2023-01-15T14:30:34.791011+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":118,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:43.073","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.0044165389999999997,"stat_sum":{"num_bytes":10888567,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":18214,"num_write":6,"num_write_kb":5071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0],"avail_no_missing":["3","0","1"],"object_location_counts":[{"shards":"0,1,3","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"68","length":"1"}]},{"pgid":"3.17","version":"149'75","reported_seq":496,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:39.953747+0000","last_change":"2023-01-15T14:30:39.955141+0000","last_active":"2023-01-15T14:30:39.953747+0000","last_peered":"2023-01-15T14:30:39.953747+0000","last_clean":"2023-01-15T14:30:39.953747+0000","last_became_active":"2023-01-15T14:30:33.899849+0000","last_became_peered":"2023-01-15T14:30:33.899849+0000","last_unstale":"2023-01-15T14:30:39.953747+0000","last_undegraded":"2023-01-15T14:30:39.953747+0000","last_fullsized":"2023-01-15T14:30:39.953747+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'65","last_scrub_stamp":"2023-01-15T14:30:24.156798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.156798+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:44.469","scrub_duration":0.30368223799999999,"objects_trimmed":0,"snaptrim_duration":0.101444514,"stat_sum":{"num_bytes":3949645,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3949645,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"6"}]},{"pgid":"3.16","version":"153'61","reported_seq":451,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:40.153668+0000","last_change":"2023-01-15T14:30:40.153668+0000","last_active":"2023-01-15T14:30:40.153668+0000","last_peered":"2023-01-15T14:30:40.153668+0000","last_clean":"2023-01-15T14:30:40.153668+0000","last_became_active":"2023-01-15T14:30:33.898057+0000","last_became_peered":"2023-01-15T14:30:33.898057+0000","last_unstale":"2023-01-15T14:30:40.153668+0000","last_undegraded":"2023-01-15T14:30:40.153668+0000","last_fullsized":"2023-01-15T14:30:40.153668+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":5,"last_scrub":"118'59","last_scrub_stamp":"2023-01-15T14:30:25.417530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:25.417530+0000","objects_scrubbed":2,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.190316714,"stat_sum":{"num_bytes":2394027,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":8568,"num_write":19,"num_write_kb":6561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":9462341,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"6"}]},{"pgid":"3.15","version":"110'55","reported_seq":455,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:39.942376+0000","last_change":"2023-01-15T14:30:40.044826+0000","last_active":"2023-01-15T14:30:39.942376+0000","last_peered":"2023-01-15T14:30:39.942376+0000","last_clean":"2023-01-15T14:30:39.942376+0000","last_became_active":"2023-01-15T14:30:03.169969+0000","last_became_peered":"2023-01-15T14:30:03.169969+0000","last_unstale":"2023-01-15T14:30:39.942376+0000","last_undegraded":"2023-01-15T14:30:39.942376+0000","last_fullsized":"2023-01-15T14:30:39.942376+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:30:24.721082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.721082+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:48.096","scrub_duration":0.109000768,"objects_trimmed":0,"snaptrim_duration":0.102545838,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"6"}]},{"pgid":"3.14","version":"150'133","reported_seq":610,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-01-15T14:30:40.858094+0000","last_change":"2023-01-15T14:30:40.859196+0000","last_active":"2023-01-15T14:30:40.858094+0000","last_peered":"2023-01-15T14:30:40.858094+0000","last_clean":"2023-01-15T14:30:40.858094+0000","last_became_active":"2023-01-15T14:30:16.972329+0000","last_became_peered":"2023-01-15T14:30:16.972329+0000","last_unstale":"2023-01-15T14:30:40.858094+0000","last_undegraded":"2023-01-15T14:30:40.858094+0000","last_fullsized":"2023-01-15T14:30:40.858094+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:31.572","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.001802094,"stat_sum":{"num_bytes":3831996,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":8,"num_read_kb":11234,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"6"},{"start":"73","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":169,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:39.850595+0000","last_change":"2023-01-15T14:30:34.107421+0000","last_active":"2023-01-15T14:30:39.850595+0000","last_peered":"2023-01-15T14:30:39.850595+0000","last_clean":"2023-01-15T14:30:39.850595+0000","last_became_active":"2023-01-15T14:30:34.006917+0000","last_became_peered":"2023-01-15T14:30:34.006917+0000","last_unstale":"2023-01-15T14:30:39.850595+0000","last_undegraded":"2023-01-15T14:30:39.850595+0000","last_fullsized":"2023-01-15T14:30:39.850595+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.671761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.671761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:09.542","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":481,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:39.952469+0000","last_change":"2023-01-15T14:30:39.953562+0000","last_active":"2023-01-15T14:30:39.952469+0000","last_peered":"2023-01-15T14:30:39.952469+0000","last_clean":"2023-01-15T14:30:39.952469+0000","last_became_active":"2023-01-15T14:30:33.993887+0000","last_became_peered":"2023-01-15T14:30:33.993887+0000","last_unstale":"2023-01-15T14:30:39.952469+0000","last_undegraded":"2023-01-15T14:30:39.952469+0000","last_fullsized":"2023-01-15T14:30:39.952469+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:30:21.421951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:21.421951+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:45.807","scrub_duration":0.569620185,"objects_trimmed":0,"snaptrim_duration":0.099852045,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"6"}]},{"pgid":"2.6","version":"0'0","reported_seq":170,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:39.848490+0000","last_change":"2023-01-15T14:30:34.094812+0000","last_active":"2023-01-15T14:30:39.848490+0000","last_peered":"2023-01-15T14:30:39.848490+0000","last_clean":"2023-01-15T14:30:39.848490+0000","last_became_active":"2023-01-15T14:30:33.993841+0000","last_became_peered":"2023-01-15T14:30:33.993841+0000","last_unstale":"2023-01-15T14:30:39.848490+0000","last_undegraded":"2023-01-15T14:30:39.848490+0000","last_fullsized":"2023-01-15T14:30:39.848490+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:58.163","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"125'75","reported_seq":499,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:40.044920+0000","last_change":"2023-01-15T14:30:40.075377+0000","last_active":"2023-01-15T14:30:40.044920+0000","last_peered":"2023-01-15T14:30:40.044920+0000","last_clean":"2023-01-15T14:30:40.044920+0000","last_became_active":"2023-01-15T14:30:03.070234+0000","last_became_peered":"2023-01-15T14:30:03.070234+0000","last_unstale":"2023-01-15T14:30:40.044920+0000","last_undegraded":"2023-01-15T14:30:40.044920+0000","last_fullsized":"2023-01-15T14:30:40.044920+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:25.827","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.125070498,"stat_sum":{"num_bytes":3462242,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"6"}]},{"pgid":"2.5","version":"0'0","reported_seq":150,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-01-15T14:30:35.850757+0000","last_change":"2023-01-15T14:29:25.664703+0000","last_active":"2023-01-15T14:30:35.850757+0000","last_peered":"2023-01-15T14:30:35.850757+0000","last_clean":"2023-01-15T14:30:35.850757+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:30:35.850757+0000","last_undegraded":"2023-01-15T14:30:35.850757+0000","last_fullsized":"2023-01-15T14:30:35.850757+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:25.664616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:25.664616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:30:37.368","scrub_duration":0.40398350300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"154'177","reported_seq":673,"reported_epoch":154,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:30:41.023453+0000","last_change":"2023-01-15T14:30:40.020263+0000","last_active":"2023-01-15T14:30:41.023453+0000","last_peered":"2023-01-15T14:30:41.023453+0000","last_clean":"2023-01-15T14:30:41.023453+0000","last_became_active":"2023-01-15T14:30:40.015688+0000","last_became_peered":"2023-01-15T14:30:40.015688+0000","last_unstale":"2023-01-15T14:30:41.023453+0000","last_undegraded":"2023-01-15T14:30:41.023453+0000","last_fullsized":"2023-01-15T14:30:41.023453+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:37.418","scrub_duration":0.50739213100000002,"objects_trimmed":1,"snaptrim_duration":0.10240964,"stat_sum":{"num_bytes":13442573,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":8,"num_read_kb":5,"num_write":23,"num_write_kb":5401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":17942309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"68","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-01-15T14:30:38.837086+0000","last_change":"2023-01-15T14:30:33.997641+0000","last_active":"2023-01-15T14:30:38.837086+0000","last_peered":"2023-01-15T14:30:38.837086+0000","last_clean":"2023-01-15T14:30:38.837086+0000","last_became_active":"2023-01-15T14:30:33.996972+0000","last_became_peered":"2023-01-15T14:30:33.996972+0000","last_unstale":"2023-01-15T14:30:38.837086+0000","last_undegraded":"2023-01-15T14:30:38.837086+0000","last_fullsized":"2023-01-15T14:30:38.837086+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"141'59","reported_seq":496,"reported_epoch":153,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:30:40.042525+0000","last_change":"2023-01-15T14:30:40.042525+0000","last_active":"2023-01-15T14:30:40.042525+0000","last_peered":"2023-01-15T14:30:40.042525+0000","last_clean":"2023-01-15T14:30:40.042525+0000","last_became_active":"2023-01-15T14:30:33.898976+0000","last_became_peered":"2023-01-15T14:30:33.898976+0000","last_unstale":"2023-01-15T14:30:40.042525+0000","last_undegraded":"2023-01-15T14:30:40.042525+0000","last_fullsized":"2023-01-15T14:30:40.042525+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:31:12.510","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81608054600000002,"stat_sum":{"num_bytes":2067294,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":30038,"num_write":17,"num_write_kb":7307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":13739656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"68","length":"5"}]},{"pgid":"3.3","version":"149'233","reported_seq":711,"reported_epoch":154,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:30:41.591068+0000","last_change":"2023-01-15T14:30:41.170122+0000","last_active":"2023-01-15T14:30:41.591068+0000","last_peered":"2023-01-15T14:30:41.591068+0000","last_clean":"2023-01-15T14:30:32.473614+0000","last_became_active":"2023-01-15T14:30:34.999152+0000","last_became_peered":"2023-01-15T14:30:34.999152+0000","last_unstale":"2023-01-15T14:30:41.591068+0000","last_undegraded":"2023-01-15T14:30:34.891382+0000","last_fullsized":"2023-01-15T14:30:34.890493+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:31:11.242","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42022305999999998,"stat_sum":{"num_bytes":15208544,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":93,"num_read_kb":87274,"num_write":73,"num_write_kb":26320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":4938988,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":3},{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"68","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":64,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-01-15T14:30:38.935991+0000","last_change":"2023-01-15T14:30:34.402628+0000","last_active":"2023-01-15T14:30:38.935991+0000","last_peered":"2023-01-15T14:30:38.935991+0000","last_clean":"2023-01-15T14:30:38.935991+0000","last_became_active":"2023-01-15T14:30:34.401861+0000","last_became_peered":"2023-01-15T14:30:34.401861+0000","last_unstale":"2023-01-15T14:30:38.935991+0000","last_undegraded":"2023-01-15T14:30:38.935991+0000","last_fullsized":"2023-01-15T14:30:38.935991+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:14.703035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:14.703035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"137'228","reported_seq":817,"reported_epoch":154,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:30:40.968272+0000","last_change":"2023-01-15T14:30:41.068829+0000","last_active":"2023-01-15T14:30:40.968272+0000","last_peered":"2023-01-15T14:30:40.968272+0000","last_clean":"2023-01-15T14:30:24.089045+0000","last_became_active":"2023-01-15T14:30:35.000933+0000","last_became_peered":"2023-01-15T14:30:35.000933+0000","last_unstale":"2023-01-15T14:30:40.968272+0000","last_undegraded":"2023-01-15T14:30:40.968272+0000","last_fullsized":"2023-01-15T14:30:34.790066+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:29.362","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.30441851800000003,"stat_sum":{"num_bytes":6958098,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6958098,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0],"avail_no_missing":["2","0","1","3"],"object_location_counts":[{"shards":"0,2","objects":2}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"68","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":155,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:39.850310+0000","last_change":"2023-01-15T14:29:47.854985+0000","last_active":"2023-01-15T14:30:39.850310+0000","last_peered":"2023-01-15T14:30:39.850310+0000","last_clean":"2023-01-15T14:30:39.850310+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:30:39.850310+0000","last_undegraded":"2023-01-15T14:30:39.850310+0000","last_fullsized":"2023-01-15T14:30:39.850310+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:47.854866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:47.854866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:56.359","scrub_duration":0.63763962500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"153'117","reported_seq":547,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:40.644141+0000","last_change":"2023-01-15T14:30:40.645811+0000","last_active":"2023-01-15T14:30:40.644141+0000","last_peered":"2023-01-15T14:30:40.644141+0000","last_clean":"2023-01-15T14:30:40.644141+0000","last_became_active":"2023-01-15T14:29:59.063638+0000","last_became_peered":"2023-01-15T14:29:59.063638+0000","last_unstale":"2023-01-15T14:30:40.644141+0000","last_undegraded":"2023-01-15T14:30:40.644141+0000","last_fullsized":"2023-01-15T14:30:40.644141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'111","last_scrub_stamp":"2023-01-15T14:30:23.145968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.145968+0000","objects_scrubbed":1,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:35.973","scrub_duration":0.28528039199999999,"objects_trimmed":0,"snaptrim_duration":0.794360391,"stat_sum":{"num_bytes":5949678,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":3710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"6"}]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":149,"state":"active+clean","last_fresh":"2023-01-15T14:30:35.850282+0000","last_change":"2023-01-15T14:30:33.997492+0000","last_active":"2023-01-15T14:30:35.850282+0000","last_peered":"2023-01-15T14:30:35.850282+0000","last_clean":"2023-01-15T14:30:35.850282+0000","last_became_active":"2023-01-15T14:30:33.996499+0000","last_became_peered":"2023-01-15T14:30:33.996499+0000","last_unstale":"2023-01-15T14:30:35.850282+0000","last_undegraded":"2023-01-15T14:30:35.850282+0000","last_fullsized":"2023-01-15T14:30:35.850282+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:31.583599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.583599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22681161699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"153'99","reported_seq":595,"reported_epoch":153,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:30:39.942845+0000","last_change":"2023-01-15T14:30:39.942845+0000","last_active":"2023-01-15T14:30:39.942845+0000","last_peered":"2023-01-15T14:30:39.942845+0000","last_clean":"2023-01-15T14:30:39.942845+0000","last_became_active":"2023-01-15T14:30:34.299864+0000","last_became_peered":"2023-01-15T14:30:34.299864+0000","last_unstale":"2023-01-15T14:30:39.942845+0000","last_undegraded":"2023-01-15T14:30:39.942845+0000","last_fullsized":"2023-01-15T14:30:39.942845+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:18.119","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.41825241299999999,"stat_sum":{"num_bytes":3827212,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":8,"num_write_kb":2148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6537956,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"68","length":"5"}]},{"pgid":"2.3","version":"0'0","reported_seq":147,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:39.850269+0000","last_change":"2023-01-15T14:30:13.907259+0000","last_active":"2023-01-15T14:30:39.850269+0000","last_peered":"2023-01-15T14:30:39.850269+0000","last_clean":"2023-01-15T14:30:39.850269+0000","last_became_active":"2023-01-15T14:30:13.906504+0000","last_became_peered":"2023-01-15T14:30:13.906504+0000","last_unstale":"2023-01-15T14:30:39.850269+0000","last_undegraded":"2023-01-15T14:30:39.850269+0000","last_fullsized":"2023-01-15T14:30:39.850269+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:44.759745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:44.759745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:55.162","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":281,"reported_epoch":152,"state":"active+clean","last_fresh":"2023-01-15T14:30:39.116593+0000","last_change":"2023-01-15T14:30:39.116057+0000","last_active":"2023-01-15T14:30:39.116593+0000","last_peered":"2023-01-15T14:30:39.116593+0000","last_clean":"2023-01-15T14:30:39.116593+0000","last_became_active":"2023-01-15T14:30:39.115312+0000","last_became_peered":"2023-01-15T14:30:39.115312+0000","last_unstale":"2023-01-15T14:30:39.116593+0000","last_undegraded":"2023-01-15T14:30:39.116593+0000","last_fullsized":"2023-01-15T14:30:39.116593+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:29:45.876183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:29:45.876183+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:52.357","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"144'192","reported_seq":701,"reported_epoch":153,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:30:40.074724+0000","last_change":"2023-01-15T14:30:40.074724+0000","last_active":"2023-01-15T14:30:40.074724+0000","last_peered":"2023-01-15T14:30:40.074724+0000","last_clean":"2023-01-15T14:30:40.074724+0000","last_became_active":"2023-01-15T14:30:40.073925+0000","last_became_peered":"2023-01-15T14:30:40.073925+0000","last_unstale":"2023-01-15T14:30:40.074724+0000","last_undegraded":"2023-01-15T14:30:40.074724+0000","last_fullsized":"2023-01-15T14:30:40.074724+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:30.681","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.090394347,"stat_sum":{"num_bytes":7596764,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":3743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7596764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"68","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":572,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-01-15T14:30:40.957293+0000","last_change":"2023-01-15T14:30:40.960468+0000","last_active":"2023-01-15T14:30:40.957293+0000","last_peered":"2023-01-15T14:30:40.957293+0000","last_clean":"2023-01-15T14:30:40.957293+0000","last_became_active":"2023-01-15T14:29:58.963354+0000","last_became_peered":"2023-01-15T14:29:58.963354+0000","last_unstale":"2023-01-15T14:30:40.957293+0000","last_undegraded":"2023-01-15T14:30:40.957293+0000","last_fullsized":"2023-01-15T14:30:40.957293+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:30:08.681869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:08.681869+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:19.959","scrub_duration":0.40398525099999999,"objects_trimmed":0,"snaptrim_duration":0.0034406860000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"6"},{"start":"73","length":"1"}]},{"pgid":"3.a","version":"153'178","reported_seq":614,"reported_epoch":154,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:30:41.439740+0000","last_change":"2023-01-15T14:30:39.578237+0000","last_active":"2023-01-15T14:30:41.439740+0000","last_peered":"2023-01-15T14:30:41.439740+0000","last_clean":"2023-01-15T14:30:32.052973+0000","last_became_active":"2023-01-15T14:30:35.093414+0000","last_became_peered":"2023-01-15T14:30:35.093414+0000","last_unstale":"2023-01-15T14:30:41.439740+0000","last_undegraded":"2023-01-15T14:30:41.439740+0000","last_fullsized":"2023-01-15T14:30:34.794673+0000","mapping_epoch":147,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.60436995599999999,"stat_sum":{"num_bytes":17352313,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":47442,"num_write":103,"num_write_kb":49117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":50076701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3],"avail_no_missing":["4","1","3"],"object_location_counts":[{"shards":"1,3,4","objects":2},{"shards":"3,4","objects":6}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"68","length":"1"}]},{"pgid":"3.b","version":"153'51","reported_seq":436,"reported_epoch":153,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:30:40.483258+0000","last_change":"2023-01-15T14:30:40.153821+0000","last_active":"2023-01-15T14:30:40.483258+0000","last_peered":"2023-01-15T14:30:40.483258+0000","last_clean":"2023-01-15T14:30:40.483258+0000","last_became_active":"2023-01-15T14:28:23.934183+0000","last_became_peered":"2023-01-15T14:28:23.934183+0000","last_unstale":"2023-01-15T14:30:40.483258+0000","last_undegraded":"2023-01-15T14:30:40.483258+0000","last_fullsized":"2023-01-15T14:30:40.483258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81253016899999997,"stat_sum":{"num_bytes":6722800,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":13388,"num_write":23,"num_write_kb":11270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"68","length":"5"}]},{"pgid":"3.c","version":"141'46","reported_seq":422,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:39.942244+0000","last_change":"2023-01-15T14:30:39.942668+0000","last_active":"2023-01-15T14:30:39.942244+0000","last_peered":"2023-01-15T14:30:39.942244+0000","last_clean":"2023-01-15T14:30:39.942244+0000","last_became_active":"2023-01-15T14:30:34.099580+0000","last_became_peered":"2023-01-15T14:30:34.099580+0000","last_unstale":"2023-01-15T14:30:39.942244+0000","last_undegraded":"2023-01-15T14:30:39.942244+0000","last_fullsized":"2023-01-15T14:30:39.942244+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:02.867","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.00059510200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":20,"num_write_kb":5044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2319333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"6"}]},{"pgid":"3.d","version":"135'148","reported_seq":633,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-01-15T14:30:36.905516+0000","last_change":"2023-01-15T14:30:36.906213+0000","last_active":"2023-01-15T14:30:36.905516+0000","last_peered":"2023-01-15T14:30:36.905516+0000","last_clean":"2023-01-15T14:30:36.905516+0000","last_became_active":"2023-01-15T14:30:22.602655+0000","last_became_peered":"2023-01-15T14:30:22.602655+0000","last_unstale":"2023-01-15T14:30:36.905516+0000","last_undegraded":"2023-01-15T14:30:36.905516+0000","last_fullsized":"2023-01-15T14:30:36.905516+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091815899999999999,"stat_sum":{"num_bytes":9714413,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":113302,"num_write":68,"num_write_kb":26630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":20117814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"68","length":"5"}]},{"pgid":"3.e","version":"139'38","reported_seq":450,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-01-15T14:30:40.857134+0000","last_change":"2023-01-15T14:30:40.857867+0000","last_active":"2023-01-15T14:30:40.857134+0000","last_peered":"2023-01-15T14:30:40.857134+0000","last_clean":"2023-01-15T14:30:40.857134+0000","last_became_active":"2023-01-15T14:30:33.996232+0000","last_became_peered":"2023-01-15T14:30:33.996232+0000","last_unstale":"2023-01-15T14:30:40.857134+0000","last_undegraded":"2023-01-15T14:30:40.857134+0000","last_fullsized":"2023-01-15T14:30:40.857134+0000","mapping_epoch":146,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'37","last_scrub_stamp":"2023-01-15T14:30:25.793922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:25.793922+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:44.713","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.00095089100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":23059,"num_write":11,"num_write_kb":8092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2974839,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"6"},{"start":"73","length":"1"}]},{"pgid":"3.f","version":"154'141","reported_seq":618,"reported_epoch":154,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:30:41.561711+0000","last_change":"2023-01-15T14:30:41.561769+0000","last_active":"2023-01-15T14:30:41.561711+0000","last_peered":"2023-01-15T14:30:41.561711+0000","last_clean":"2023-01-15T14:30:41.561711+0000","last_became_active":"2023-01-15T14:30:39.973590+0000","last_became_peered":"2023-01-15T14:30:39.973590+0000","last_unstale":"2023-01-15T14:30:41.561711+0000","last_undegraded":"2023-01-15T14:30:41.561711+0000","last_fullsized":"2023-01-15T14:30:41.561711+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.60397762099999996,"stat_sum":{"num_bytes":14605316,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":71,"num_read_kb":67175,"num_write":76,"num_write_kb":25662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":29798811,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"2"}]},{"pgid":"3.10","version":"130'234","reported_seq":873,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-01-15T14:30:39.852351+0000","last_change":"2023-01-15T14:30:39.952116+0000","last_active":"2023-01-15T14:30:39.852351+0000","last_peered":"2023-01-15T14:30:39.852351+0000","last_clean":"2023-01-15T14:30:39.852351+0000","last_became_active":"2023-01-15T14:30:19.199769+0000","last_became_peered":"2023-01-15T14:30:19.199769+0000","last_unstale":"2023-01-15T14:30:39.852351+0000","last_undegraded":"2023-01-15T14:30:39.852351+0000","last_fullsized":"2023-01-15T14:30:39.852351+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":1,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:44.768","scrub_duration":0.20710088300000001,"objects_trimmed":0,"snaptrim_duration":0.099897995000000003,"stat_sum":{"num_bytes":3714347,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"6"}]},{"pgid":"3.11","version":"137'115","reported_seq":599,"reported_epoch":150,"state":"active+clean","last_fresh":"2023-01-15T14:30:37.005437+0000","last_change":"2023-01-15T14:30:37.006881+0000","last_active":"2023-01-15T14:30:37.005437+0000","last_peered":"2023-01-15T14:30:37.005437+0000","last_clean":"2023-01-15T14:30:37.005437+0000","last_became_active":"2023-01-15T14:30:22.499879+0000","last_became_peered":"2023-01-15T14:30:22.499879+0000","last_unstale":"2023-01-15T14:30:37.005437+0000","last_undegraded":"2023-01-15T14:30:37.005437+0000","last_fullsized":"2023-01-15T14:30:37.005437+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'112","last_scrub_stamp":"2023-01-15T14:30:09.593275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.593275+0000","objects_scrubbed":2,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:27.977","scrub_duration":0.28909560899999998,"objects_trimmed":0,"snaptrim_duration":0.0016419679999999999,"stat_sum":{"num_bytes":13193495,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":33960,"num_write":2,"num_write_kb":7321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5697573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"68","length":"5"}]},{"pgid":"3.12","version":"153'121","reported_seq":226,"reported_epoch":153,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:30:40.267487+0000","last_change":"2023-01-15T14:30:40.000988+0000","last_active":"2023-01-15T14:30:40.267487+0000","last_peered":"2023-01-15T14:30:40.267487+0000","last_clean":"2023-01-15T14:30:40.267487+0000","last_became_active":"2023-01-15T14:30:37.932190+0000","last_became_peered":"2023-01-15T14:30:37.932190+0000","last_unstale":"2023-01-15T14:30:40.267487+0000","last_undegraded":"2023-01-15T14:30:40.267487+0000","last_fullsized":"2023-01-15T14:30:40.267487+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"139'103","last_scrub_stamp":"2023-01-15T14:30:30.045615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:30.045615+0000","objects_scrubbed":3,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:47.875","scrub_duration":0.207128334,"objects_trimmed":2,"snaptrim_duration":0.10963814,"stat_sum":{"num_bytes":5596764,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":19,"num_read_kb":27344,"num_write":5,"num_write_kb":1114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":6669165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"68","length":"5"}]},{"pgid":"3.13","version":"150'240","reported_seq":680,"reported_epoch":150,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:30:36.905803+0000","last_change":"2023-01-15T14:30:36.905803+0000","last_active":"2023-01-15T14:30:36.905803+0000","last_peered":"2023-01-15T14:30:36.905803+0000","last_clean":"2023-01-15T14:30:36.905803+0000","last_became_active":"2023-01-15T14:30:08.974612+0000","last_became_peered":"2023-01-15T14:30:08.974612+0000","last_unstale":"2023-01-15T14:30:36.905803+0000","last_undegraded":"2023-01-15T14:30:36.905803+0000","last_fullsized":"2023-01-15T14:30:36.905803+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:29.544","scrub_duration":0.50670270299999998,"objects_trimmed":1,"snaptrim_duration":0.77605900000000005,"stat_sum":{"num_bytes":20420472,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":23,"num_read_kb":22682,"num_write":16,"num_write_kb":10625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"68","length":"1"},{"start":"6a","length":"3"}]}],"pool_stats":[{"poolid":3,"num_pg":26,"stat_sum":{"num_bytes":182762302,"num_objects":72,"num_object_clones":28,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":7,"num_read":546,"num_read_kb":518714,"num_write":486,"num_write_kb":209005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":191536341,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3384,"ondisk_log_size":3384,"up":78,"acting":73,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":853136,"kb_used_data":92848564,"kb_used_omap":1627,"kb_used_meta":0,"kb_avail":92846244,"statfs":{"total":95948165120,"available":95074553856,"internally_reserved":2375680,"allocated":95076929536,"data_stored":95076929536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1666335,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":3.4119999999999999,"15min":3.8500000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.522,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":9.1829999999999998,"5min":9.2029999999999994,"15min":9.2070000000000007},"min":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":73.769000000000005,"5min":73.769000000000005,"15min":73.769000000000005},"last":0.42899999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":6.3079999999999998,"5min":6.4969999999999999,"15min":6.5289999999999999},"min":{"1min":0.56200000000000006,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":88.003,"5min":88.003,"15min":88.003},"last":0.76700000000000002},{"interface":"front","average":{"1min":5.3300000000000001,"5min":6.0899999999999999,"15min":6.2160000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":73.704999999999998,"5min":73.704999999999998,"15min":73.704999999999998},"last":0.72499999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":13.926,"5min":6.6180000000000003,"15min":5.4000000000000004},"min":{"1min":0.69799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":74.010999999999996,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.90600000000000003,"5min":3.5230000000000001,"15min":3.9590000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.8169999999999999,"5min":75.998999999999995,"15min":75.998999999999995},"last":0.88500000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":14.441000000000001,"5min":3.6459999999999999,"15min":1.847},"min":{"1min":0.71799999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":144.65600000000001,"5min":144.65600000000001,"15min":144.65600000000001},"last":0.82699999999999996},{"interface":"front","average":{"1min":4.0670000000000002,"5min":4.5350000000000001,"15min":4.6130000000000004},"min":{"1min":0.71799999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":45.164000000000001,"5min":76.034999999999997,"15min":76.034999999999997},"last":31.751000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":5.3949999999999996,"5min":8.6110000000000007,"15min":9.1470000000000002},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":73.459999999999994,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":16.405000000000001,"5min":4.0529999999999999,"15min":1.994},"min":{"1min":0.67000000000000004,"5min":0.498,"15min":0.498},"max":{"1min":164.21199999999999,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.92400000000000004}]},{"osd":5,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":5.4480000000000004,"5min":6.1509999999999998,"15min":6.2679999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":73.561000000000007,"5min":73.561000000000007,"15min":73.561000000000007},"last":0.95099999999999996},{"interface":"front","average":{"1min":5.3739999999999997,"5min":5.0300000000000002,"15min":4.9729999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":70.387,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.97899999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":5.6429999999999998,"5min":7.9470000000000001,"15min":8.3309999999999995},"min":{"1min":0.68700000000000006,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":73.638999999999996,"5min":73.638999999999996,"15min":73.638999999999996},"last":1.0089999999999999},{"interface":"front","average":{"1min":5.5419999999999998,"5min":1.988,"15min":1.3959999999999999},"min":{"1min":0.93899999999999995,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":70.530000000000001,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.0309999999999999}]}]},{"osd":6,"up_from":0,"seq":55834574882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":22,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":981420,"kb_used_data":92736960,"kb_used_omap":2837,"kb_used_meta":0,"kb_avail":92717960,"statfs":{"total":95948165120,"available":94943191040,"internally_reserved":19456000,"allocated":94962647040,"data_stored":94962647040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2905534,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":7.6550000000000002,"5min":5.4630000000000001,"15min":5.0970000000000004},"min":{"1min":0.47599999999999998,"5min":0.44,"15min":0.44},"max":{"1min":86.519000000000005,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.83499999999999996},{"interface":"front","average":{"1min":3.6139999999999999,"5min":1.347,"15min":0.96899999999999997},"min":{"1min":0.502,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":54.201000000000001,"5min":54.201000000000001,"15min":54.201000000000001},"last":0.42599999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":8.8239999999999998,"15min":8.6699999999999999},"min":{"1min":0.502,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":171.91499999999999,"5min":171.91499999999999,"15min":171.91499999999999},"last":24.489999999999998},{"interface":"front","average":{"1min":5.726,"5min":1.78,"15min":1.1220000000000001},"min":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":93.156000000000006,"5min":93.156000000000006,"15min":93.156000000000006},"last":0.71299999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":5.2720000000000002,"5min":5.4630000000000001,"15min":5.4950000000000001},"min":{"1min":0.57299999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":85.185000000000002,"5min":99.884,"15min":99.884},"last":97.974000000000004},{"interface":"front","average":{"1min":0.875,"5min":0.90300000000000002,"15min":0.90700000000000003},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1830000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.877}]},{"osd":3,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.361,"15min":1.3460000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":12.878,"5min":12.878,"15min":12.878},"last":0.85999999999999999},{"interface":"front","average":{"1min":6.7469999999999999,"5min":6.2699999999999996,"15min":6.1900000000000004},"min":{"1min":0.73299999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":85.150000000000006,"5min":99.921000000000006,"15min":99.921000000000006},"last":24.649999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.91700000000000004},"min":{"1min":0.60899999999999999,"5min":0.496,"15min":0.496},"max":{"1min":1.2749999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.92700000000000005,"5min":2.024,"15min":2.2069999999999999},"min":{"1min":0.65500000000000003,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.153,"5min":30.971,"15min":30.971},"last":0.94399999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":9.8130000000000006,"5min":6.5010000000000003,"15min":5.9489999999999998},"min":{"1min":0.69699999999999995,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":85.721999999999994,"5min":99.780000000000001,"15min":99.780000000000001},"last":24.553999999999998},{"interface":"front","average":{"1min":1.042,"5min":4.3380000000000001,"15min":4.8879999999999999},"min":{"1min":0.88100000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.373,"5min":88.186999999999998,"15min":88.186999999999998},"last":1.234}]},{"osd":7,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":10.694000000000001,"15min":12.316000000000001},"min":{"1min":0.66600000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.216,"5min":179.73500000000001,"15min":179.73500000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":5.46,"5min":10.083,"15min":10.853},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":85.382999999999996,"5min":99.843000000000004,"15min":99.843000000000004},"last":24.603999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":27,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":943892,"kb_used_data":92767976,"kb_used_omap":2777,"kb_used_meta":0,"kb_avail":92755488,"statfs":{"total":95948165120,"available":94981619712,"internally_reserved":12787712,"allocated":94994407424,"data_stored":94994407424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2843796,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,7,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":8,"commit_latency_ns":0,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":9.1959999999999997,"5min":5.7169999999999996,"15min":5.1369999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":174.52199999999999,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":3.3130000000000002,"5min":6.7759999999999998,"15min":7.3529999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":43.481999999999999,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.60999999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":10.188000000000001,"5min":12.863,"15min":13.308999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":86.067999999999998,"5min":159.256,"15min":159.256},"last":10.398999999999999},{"interface":"front","average":{"1min":8.5890000000000004,"5min":2.3100000000000001,"15min":1.264},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":174.578,"5min":174.578,"15min":174.578},"last":0.51500000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":2.5779999999999998,"5min":3.194,"15min":3.2970000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":43.594000000000001,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":8.9700000000000006,"5min":3.548,"15min":2.6440000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":174.45099999999999,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.56999999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":2.9089999999999998,"5min":5.8310000000000004,"15min":6.3179999999999996},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":43.689,"5min":66.311999999999998,"15min":66.311999999999998},"last":47.247},{"interface":"front","average":{"1min":14.416,"5min":13.981999999999999,"15min":13.91},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":88.355999999999995,"5min":159.47200000000001,"15min":159.47200000000001},"last":10.49}]},{"osd":5,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":10.44,"5min":13.210000000000001,"15min":13.672000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":85.543999999999997,"5min":159.43000000000001,"15min":159.43000000000001},"last":10.638},{"interface":"front","average":{"1min":8.8780000000000001,"5min":2.593,"15min":1.5449999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":174.61000000000001,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.76300000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":8.8729999999999993,"5min":2.6000000000000001,"15min":1.554},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":174.63999999999999,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":2.9790000000000001,"5min":3.605,"15min":3.7090000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":43.661000000000001,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.81599999999999995}]},{"osd":7,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":2.944,"5min":3.8170000000000002,"15min":3.9630000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":43.712000000000003,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.77400000000000002},{"interface":"front","average":{"1min":10.481999999999999,"5min":16.568000000000001,"15min":17.582000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":85.484999999999999,"5min":159.512,"15min":159.512},"last":11.417}]}]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":19,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":904368,"kb_used_data":92819964,"kb_used_omap":2332,"kb_used_meta":0,"kb_avail":92795012,"statfs":{"total":95948165120,"available":95022092288,"internally_reserved":25550848,"allocated":95047643136,"data_stored":95047643136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2388459,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":3.008,"5min":1.1140000000000001,"15min":0.79800000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":45.466999999999999,"5min":45.466999999999999,"15min":45.466999999999999},"last":46.652999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.111,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.495}]},{"osd":1,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":11.958,"5min":3.0299999999999998,"15min":1.542},"min":{"1min":0.371,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":143.071,"5min":143.071,"15min":143.071},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":2.831,"15min":3.181},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.952,"5min":47.518000000000001,"15min":47.518000000000001},"last":0.56200000000000006}]},{"osd":3,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":20.398,"5min":12.401,"15min":11.068},"min":{"1min":0.41399999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":143.13300000000001,"5min":183.53,"15min":183.53},"last":0.63300000000000001},{"interface":"front","average":{"1min":11.003,"5min":2.8140000000000001,"15min":1.4490000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":195.16399999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":46.759}]},{"osd":4,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":12.813000000000001,"5min":5.4820000000000002,"15min":4.2599999999999998},"min":{"1min":0.67500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":175.34700000000001,"5min":175.34700000000001,"15min":175.34700000000001},"last":46.828000000000003},{"interface":"front","average":{"1min":12.773999999999999,"5min":5.6020000000000003,"15min":4.4059999999999997},"min":{"1min":0.75800000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":175.30500000000001,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.73999999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":8.4290000000000003,"5min":2.6259999999999999,"15min":1.659},"min":{"1min":0.67300000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":143.262,"5min":143.262,"15min":143.262},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.006,"5min":1.1299999999999999,"15min":1.1499999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":2.2280000000000002,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":46.798000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.0960000000000001,"15min":1.0529999999999999},"min":{"1min":0.72499999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":7.8849999999999998,"5min":7.8849999999999998,"15min":7.8849999999999998},"last":0.92400000000000004},{"interface":"front","average":{"1min":8.8740000000000006,"5min":2.7490000000000001,"15min":1.728},"min":{"1min":0.75800000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":143.215,"5min":143.215,"15min":143.215},"last":0.73199999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":2.637,"15min":2.9180000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.073,"5min":37.124000000000002,"15min":37.124000000000002},"last":0.98099999999999998},{"interface":"front","average":{"1min":8.4359999999999999,"5min":3.9079999999999999,"15min":3.1539999999999999},"min":{"1min":0.78400000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":143.17099999999999,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.89900000000000002}]}]},{"osd":1,"up_from":0,"seq":55834574882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":13,"seq":55834574882,"num_pgs":27,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":950948,"kb_used_data":92752004,"kb_used_omap":2874,"kb_used_meta":0,"kb_avail":92748432,"statfs":{"total":95948165120,"available":94974394368,"internally_reserved":3657728,"allocated":94978052096,"data_stored":94978052096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2943064,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":10,"commit_latency_ns":0,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":4.3659999999999997,"5min":1.371,"15min":0.872},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":69.872,"5min":69.872,"15min":69.872},"last":0.503},{"interface":"front","average":{"1min":5.5919999999999996,"5min":1.702,"15min":1.054},"min":{"1min":0.38700000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":47.795999999999999,"5min":47.795999999999999,"15min":47.795999999999999},"last":0.60399999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":9.25,"5min":2.4049999999999998,"15min":1.264},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":65.825000000000003,"5min":65.825000000000003,"15min":65.825000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":8.2590000000000003,"5min":2.2189999999999999,"15min":1.212},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":71.406999999999996,"5min":71.406999999999996,"15min":71.406999999999996},"last":48.893000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":4.5759999999999996,"5min":14.337999999999999,"15min":15.965},"min":{"1min":0.47599999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":69.968000000000004,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":15.317,"5min":3.7040000000000002,"15min":1.768},"min":{"1min":0.45700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":174.09399999999999,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.68400000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":7.6420000000000003,"5min":2.3170000000000002,"15min":1.429},"min":{"1min":0.54900000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":71.483000000000004,"5min":71.483000000000004,"15min":71.483000000000004},"last":0.93899999999999995},{"interface":"front","average":{"1min":7.6029999999999998,"5min":2.3399999999999999,"15min":1.4630000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":70.046999999999997,"5min":70.046999999999997,"15min":70.046999999999997},"last":0.90600000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":5.7960000000000003,"5min":8.9969999999999999,"15min":9.5310000000000006},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":47.969000000000001,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":4.7880000000000003,"5min":1.788,"15min":1.288},"min":{"1min":0.63400000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":71.563999999999993,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.88800000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":4.8070000000000004,"5min":1.897,"15min":1.4119999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":70.189999999999998,"5min":70.189999999999998,"15min":70.189999999999998},"last":0.97199999999999998},{"interface":"front","average":{"1min":5.8959999999999999,"5min":2.0859999999999999,"15min":1.4510000000000001},"min":{"1min":0.66700000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":47.920999999999999,"5min":47.920999999999999,"15min":47.920999999999999},"last":0.77400000000000002}]},{"osd":7,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":8.3680000000000003,"5min":14.210000000000001,"15min":15.183999999999999},"min":{"1min":0.66800000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":71.629999999999995,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":4.8170000000000002,"5min":1.8999999999999999,"15min":1.4139999999999999},"min":{"1min":0.72499999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":70.126000000000005,"5min":70.126000000000005,"15min":70.126000000000005},"last":1.0129999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:30:42.412 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:30:42.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 36 2023-01-15T14:30:42.412 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:30:42.414 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:42.412+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:42.515 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:42.512+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:42.599 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:42.599 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:42.600 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:30:42.615 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:42.612+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:42.779 INFO:tasks.rados.rados.0.smithi090.stdout:648: done (0 left) 2023-01-15T14:30:42.780 INFO:tasks.rados.rados.0.smithi090.stdout:649: read oid 10 snap 59 2023-01-15T14:30:42.780 INFO:tasks.rados.rados.0.smithi090.stdout:649: expect deleted 2023-01-15T14:30:42.780 INFO:tasks.rados.rados.0.smithi090.stdout:650: snap_remove snap 60 2023-01-15T14:30:42.780 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:42.776+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:42.780 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:42.781 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:42.781 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:30:42.781 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:30:42.787 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 36 2023-01-15T14:30:42.880 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:42.876+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:42.986 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:42.987 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:42.987 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:30:42.988 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:42.984+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:43.088 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:43.083+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:43.187 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:43.187 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:43.187 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:30:43.188 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:43.184+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:43.290 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:43.288+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:43.393 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:43.394 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:43.394 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:30:43.394 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:43.391+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:43.494 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:43.492+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:43.582 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:43.582 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:43.585 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:30:43.595 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:43.592+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:43.696 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:43.691+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:43.775 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:43.775 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:43.776 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:30:43.779 INFO:tasks.rados.rados.0.smithi090.stdout:650: done (1 left) 2023-01-15T14:30:43.780 INFO:tasks.rados.rados.0.smithi090.stdout:651: copy_from oid 5 from oid 35 current snap is 61 2023-01-15T14:30:43.780 INFO:tasks.rados.rados.0.smithi090.stdout:649: done (1 left) 2023-01-15T14:30:43.780 INFO:tasks.rados.rados.0.smithi090.stdout:652: rollback oid 13 current snap is 61 2023-01-15T14:30:43.780 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 13 to 59 2023-01-15T14:30:43.780 INFO:tasks.rados.rados.0.smithi090.stdout:653: read oid 9 snap -1 2023-01-15T14:30:43.780 INFO:tasks.rados.rados.0.smithi090.stdout:653: expect deleted 2023-01-15T14:30:43.780 INFO:tasks.rados.rados.0.smithi090.stdout:654: write oid 21 current snap is 61 2023-01-15T14:30:43.781 INFO:tasks.rados.rados.0.smithi090.stdout:654: seq_num 262 ranges {403434=471240,1315341=400756} 2023-01-15T14:30:43.789 INFO:tasks.rados.rados.0.smithi090.stdout:654: writing smithi09025264-21 from 403434 to 874674 tid 1 2023-01-15T14:30:43.794 INFO:tasks.rados.rados.0.smithi090.stdout:654: writing smithi09025264-21 from 1315341 to 1716097 tid 2 2023-01-15T14:30:43.794 INFO:tasks.rados.rados.0.smithi090.stdout:655: snap_create 2023-01-15T14:30:43.794 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:30:43.821 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:43.819+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:43.956 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:43.956 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:43.956 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:30:43.957 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:43.952+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:44.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:30:44.156 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:44.156 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:44.156 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:30:44.157 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:44.151+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:44.256 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:44.252+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:44.344 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:44.344 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:44.345 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:44.345 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:44.345 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:44.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:30:44.357 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:44.355+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:44.532 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:44.532 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:44.533 INFO:tasks.rados.rados.0.smithi090.stdout:651: finishing copy_from to smithi09025264-5 2023-01-15T14:30:44.533 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 59 (ObjNum 128 snap 23 seq_num 128) dirty exists 2023-01-15T14:30:44.533 INFO:tasks.rados.rados.0.smithi090.stdout:654: finishing write tid 1 to smithi09025264-21 2023-01-15T14:30:44.533 INFO:tasks.rados.rados.0.smithi090.stdout:654: finishing write tid 2 to smithi09025264-21 2023-01-15T14:30:44.533 INFO:tasks.rados.rados.0.smithi090.stdout:651: finishing copy_from racing read to smithi09025264-5 2023-01-15T14:30:44.533 INFO:tasks.rados.rados.0.smithi090.stdout:651: done (3 left) 2023-01-15T14:30:44.534 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:30:44.534 INFO:tasks.rados.rados.0.smithi090.stdout:654: finishing write tid 3 to smithi09025264-21 2023-01-15T14:30:44.534 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:44.534 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:44.534 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:44.535 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:44.535 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:30:44.535 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:44.535 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:44.536 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:30:44.536 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:30:44.538 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:30:44.538 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:30:44.538 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:30:44.538 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:30:44.538 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:30:44.539 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:30:44.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:30:44.774 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:44.527+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:44.775 INFO:tasks.rados.rados.0.smithi090.stdout:654: finishing write tid 4 to smithi09025264-21 2023-01-15T14:30:44.775 INFO:tasks.rados.rados.0.smithi090.stdout:654: oid 21 updating version 0 to 199 2023-01-15T14:30:44.775 INFO:tasks.rados.rados.0.smithi090.stdout:654: oid 21 version 199 is already newer than 198 2023-01-15T14:30:44.775 INFO:tasks.rados.rados.0.smithi090.stdout:654: oid 21 version 199 is already newer than 197 2023-01-15T14:30:44.775 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 199 (ObjNum 262 snap 61 seq_num 262) dirty exists 2023-01-15T14:30:44.775 INFO:tasks.rados.rados.0.smithi090.stdout:654: left oid 21 (ObjNum 262 snap 61 seq_num 262) 2023-01-15T14:30:44.775 INFO:tasks.rados.rados.0.smithi090.stdout:654: done (2 left) 2023-01-15T14:30:44.776 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:44.776 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:44.771+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:44.798 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:30:44.917 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:44.912+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:44.934 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:44.935 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:44.944 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:30:45.017 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:45.015+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:45.105 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:45.106 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:45.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:45.107 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:45.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.108 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:45.108 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:30:45.108 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 139.02187434000001, 2023-01-15T14:30:45.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:30:45.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:45.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.109 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:30:45.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.109 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:30:45.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:30:45.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:30:45.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:45.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:30:45.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:30:45.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:30:45.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:30:45.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:30:45.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:30:45.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:30:45.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:30:45.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:30:45.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:30:45.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:30:45.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:30:45.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:30:45.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:30:45.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:45.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:30:45.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:30:45.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:45.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:30:45.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:30:45.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:45.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:30:45.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:30:45.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:45.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:30:45.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:30:45.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:45.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:30:45.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:30:45.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:30:45.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:30:45.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:30:45.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:30:45.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:45.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:30:45.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:30:45.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:30:45.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:30:45.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.119 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.119 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.120 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:45.120 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:30:45.120 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 139.02159409199999, 2023-01-15T14:30:45.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:30:45.120 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.121 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:45.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.121 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.121 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:30:45.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.122 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:30:45.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:30:45.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:30:45.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:45.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:30:45.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:30:45.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:30:45.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:30:45.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:30:45.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:30:45.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:30:45.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:30:45.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:30:45.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:30:45.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:45.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:30:45.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:30:45.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:45.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:30:45.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:30:45.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:45.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:30:45.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:30:45.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:45.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:30:45.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:30:45.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:45.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:30:45.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:30:45.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:30:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:30:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:30:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:30:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.131 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:45.131 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:30:45.131 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 139.021561475, 2023-01-15T14:30:45.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:30:45.131 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.131 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:45.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.131 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.131 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:30:45.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.132 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:30:45.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:30:45.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:30:45.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:45.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:30:45.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:30:45.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:30:45.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:30:45.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:30:45.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:30:45.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:30:45.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:30:45.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:30:45.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:30:45.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:30:45.138 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.138 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.138 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.138 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:45.138 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:30:45.138 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 138.920243489, 2023-01-15T14:30:45.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:30:45.139 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.139 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:45.139 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.139 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.139 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.139 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:30:45.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.139 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:30:45.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:30:45.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:30:45.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:45.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:30:45.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:30:45.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:30:45.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:30:45.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:30:45.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:30:45.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:30:45.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:30:45.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:30:45.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:30:45.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:45.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:30:45.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:30:45.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:45.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:30:45.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:30:45.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:45.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:30:45.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:30:45.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:30:45.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:30:45.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:30:45.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:30:45.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:45.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.148 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 138.92021348599999, 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:30:45.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:45.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:30:45.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:30:45.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:30:45.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:30:45.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:30:45.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:30:45.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:30:45.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:30:45.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:30:45.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:30:45.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:30:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:30:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:30:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:30:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:30:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:30:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:30:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:30:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:30:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:30:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:30:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 138.40520492100001, 2023-01-15T14:30:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:30:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:30:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:30:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:30:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:30:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:30:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:30:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:30:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:30:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:30:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:30:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:30:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:30:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:30:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:30:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:30:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:30:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:30:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:30:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:30:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:45.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:30:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:30:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:30:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:30:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:30:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:30:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:30:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:30:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:30:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:30:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:30:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:30:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:30:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:30:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:30:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:30:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:30:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:30:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:30:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 138.40472256699999, 2023-01-15T14:30:45.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:30:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:30:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:30:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:30:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:30:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:30:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:30:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:30:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:30:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:30:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:30:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:30:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:30:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:30:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:30:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:30:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:30:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:30:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:30:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:45.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:30:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:30:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:30:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:30:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:30:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:30:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:45.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:30:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:30:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:30:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:30:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:30:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:30:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:30:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:30:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:30:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:30:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:30:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:30:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:30:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 138.404670004, 2023-01-15T14:30:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:30:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:30:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:30:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:30:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:30:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:30:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:30:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:30:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:30:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:30:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:30:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:30:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:30:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:30:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:30:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:30:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:30:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:30:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:30:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:30:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:30:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:30:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:30:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:30:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:30:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:30:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.0334683, 2023-01-15T14:30:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:30:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:30:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:30:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:30:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:30:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:30:45.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:30:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:30:45.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:30:45.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:30:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:30:45.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:30:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:30:45.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:30:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:30:45.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:30:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:30:45.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:30:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:30:45.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:30:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:45.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:30:45.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:30:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:30:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:30:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:30:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:30:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:30:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:30:45.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:30:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:30:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:30:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:30:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:30:45.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:30:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:30:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.033351626, 2023-01-15T14:30:45.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:30:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:30:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:30:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:30:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:30:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:45.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:30:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:30:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:30:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:30:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:30:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:30:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:30:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:30:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:30:45.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:30:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:30:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:30:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:30:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:30:45.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:30:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:30:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:30:45.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:30:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:30:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:30:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.862 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:30:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:30:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 117.30721935, 2023-01-15T14:30:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:30:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:45.863 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.864 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.864 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.864 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:30:45.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.865 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:30:45.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:30:45.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:30:45.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:45.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:30:45.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:30:45.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:30:45.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:30:45.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:30:45.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:30:45.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:30:45.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:30:45.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:45.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:30:45.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:30:45.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:30:45.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:30:45.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:30:45.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:30:45.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:30:45.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:30:45.870 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.871 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.871 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:30:45.871 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:30:45.871 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.090969290999993, 2023-01-15T14:30:45.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:30:45.871 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.871 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:45.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:30:45.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.872 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:30:45.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:30:45.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:30:45.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:45.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:30:45.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:30:45.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:30:45.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:30:45.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:30:45.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:30:45.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:30:45.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:30:45.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:45.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:30:45.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:30:45.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:45.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:30:45.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:30:45.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:45.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:30:45.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:30:45.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:45.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:30:45.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:30:45.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:45.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:30:45.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:30:45.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:45.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:30:45.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:30:45.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:45.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:30:45.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:30:45.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:45.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:30:45.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:30:45.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:30:45.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:30:45.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:45.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:30:45.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:30:45.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:45.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:30:45.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:30:45.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:45.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:30:45.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:30:45.883 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.883 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:45.883 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:45.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.883 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:30:45.883 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:45.883 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.090935724000005, 2023-01-15T14:30:45.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:30:45.884 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:45.884 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:45.884 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:45.884 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:45.884 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:45.884 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:30:45.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.884 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:45.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:45.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:45.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:45.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:45.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:45.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:45.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:30:45.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:45.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:45.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:30:45.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:30:45.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:45.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:30:45.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:30:45.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:45.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:45.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:45.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:30:45.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:46.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:30:46.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:30:46.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:46.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:46.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:46.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:46.023 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 62.325032178000001, 2023-01-15T14:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:46.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:30:46.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:46.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:30:46.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:30:46.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:46.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:30:46.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:30:46.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:46.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:30:46.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:30:46.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:46.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:46.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.781853378000001, 2023-01-15T14:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:30:46.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:46.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:46.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:30:46.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:30:46.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:30:46.043 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.781814203000003, 2023-01-15T14:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:46.044 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:30:46.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:30:46.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:30:46.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.142 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1373 3.12 3:495d4bbc:::smithi09025264-16:head [write 2873523~591733] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:30:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:30:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.389697877000003, 2023-01-15T14:30:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40949580099999999, 2023-01-15T14:30:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1373 2023-01-15T14:30:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.143 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:30:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:30:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693440+0000", 2023-01-15T14:30:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693808+0000", 2023-01-15T14:30:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000367742 2023-01-15T14:30:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693831+0000", 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3213e-05 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693846+0000", 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4909000000000001e-05 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.695833+0000", 2023-01-15T14:30:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019869240000000002 2023-01-15T14:30:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696121+0000", 2023-01-15T14:30:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028753300000000003 2023-01-15T14:30:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798577+0000", 2023-01-15T14:30:46.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102456808 2023-01-15T14:30:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798849+0000", 2023-01-15T14:30:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027138999999999999 2023-01-15T14:30:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:30:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799155+0000", 2023-01-15T14:30:46.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030594800000000001 2023-01-15T14:30:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799614+0000", 2023-01-15T14:30:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045960700000000001 2023-01-15T14:30:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799737+0000", 2023-01-15T14:30:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012218900000000001 2023-01-15T14:30:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799998+0000", 2023-01-15T14:30:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026129700000000002 2023-01-15T14:30:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800639+0000", 2023-01-15T14:30:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064108800000000005 2023-01-15T14:30:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102023+0000", 2023-01-15T14:30:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30138417200000001 2023-01-15T14:30:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102198+0000", 2023-01-15T14:30:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017453600000000001 2023-01-15T14:30:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102398+0000", 2023-01-15T14:30:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020078000000000001 2023-01-15T14:30:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102937+0000", 2023-01-15T14:30:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053874499999999998 2023-01-15T14:30:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:46.179 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:30:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:30:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.389273283000001, 2023-01-15T14:30:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:30:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:46.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:30:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:30:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:30:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:30:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:30:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:30:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:30:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:30:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:30:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:30:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:30:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:30:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:30:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:30:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:30:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:30:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:30:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:30:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:30:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:30:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:30:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:30:46.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:30:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:30:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:30:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:46.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:30:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:30:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:30:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:30:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:46.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:30:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:30:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:30:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:30:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:30:46.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:30:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:30:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:30:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.191 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:30:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:30:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.389220954000002, 2023-01-15T14:30:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:30:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:30:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.192 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:30:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:30:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:30:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:30:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:30:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:30:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:30:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:30:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:30:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:30:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:30:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:46.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:30:46.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:30:46.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:30:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:30:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:30:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:30:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:30:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:30:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:30:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:30:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:30:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:30:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:30:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:30:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.792307755, 2023-01-15T14:30:46.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:30:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:30:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:30:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:30:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:46.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:30:46.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:46.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:46.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:30:46.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:30:46.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:46.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:30:46.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:30:46.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:46.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:30:46.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:30:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:30:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:30:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:30:46.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:30:46.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:46.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:30:46.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:30:46.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:30:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:30:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:46.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:46.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:30:46.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:30:46.350 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.350 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:46.350 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.350 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:46.351 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:46.351 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:46.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:30:46.354 INFO:tasks.rados.rados.0.smithi090.stdout:653: done (1 left) 2023-01-15T14:30:46.354 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:46.354 INFO:tasks.rados.rados.0.smithi090.stdout:652: finishing rollback tid 0 to smithi09025264-13 2023-01-15T14:30:46.354 INFO:tasks.rados.rados.0.smithi090.stdout:652: finishing rollback tid 1 to smithi09025264-13 2023-01-15T14:30:46.355 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:30:46.402 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:46.351+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:46.455 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:46.451+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:46.515 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:46.515 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:46.516 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:30:46.516 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:46.527 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:30:46.658 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:46.655+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:46.759 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:46.755+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:46.851 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:30:46.852 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:30:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:30:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:30:46.853 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:46.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:30:46.860 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:46.855+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:46.961 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:46.956+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:47.026 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:47.026 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:30:47.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:30:47.026 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:47.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.026 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:47.026 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:47.026 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 139.914526129, 2023-01-15T14:30:47.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:30:47.027 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:47.027 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:47.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:47.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:47.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:47.027 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:30:47.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.027 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:47.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:47.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:47.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:47.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:47.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:47.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:30:47.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:47.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:47.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:30:47.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:30:47.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:47.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:30:47.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:30:47.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:47.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:30:47.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:30:47.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:30:47.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:30:47.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:47.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:30:47.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:30:47.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:47.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:30:47.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:30:47.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:47.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:30:47.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:30:47.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:47.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:30:47.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:30:47.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:47.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:30:47.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:30:47.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:47.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:30:47.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:30:47.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:30:47.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:30:47.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:47.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:30:47.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:30:47.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:47.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.036 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:47.037 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:30:47.037 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 139.914226332, 2023-01-15T14:30:47.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:30:47.037 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:47.037 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:47.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:47.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:47.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:47.037 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:30:47.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.037 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:47.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:47.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:30:47.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:47.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:30:47.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:47.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:30:47.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:47.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:47.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:30:47.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:30:47.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:47.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:30:47.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:30:47.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:47.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:30:47.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:30:47.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:30:47.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:30:47.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:47.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:30:47.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:30:47.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:30:47.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:30:47.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:47.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:30:47.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:30:47.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:47.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:30:47.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:30:47.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:47.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:30:47.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:30:47.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:47.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:30:47.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:30:47.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:47.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:30:47.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:30:47.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:47.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:30:47.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:30:47.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:30:47.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:30:47.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:30:47.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:30:47.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:47.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:30:47.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:30:47.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:47.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:30:47.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:30:47.186 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.186 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:47.186 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.187 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:47.187 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:30:47.287 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 139.91419654800001, 2023-01-15T14:30:47.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:30:47.287 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:47.288 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:47.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:47.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:47.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:47.288 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:30:47.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.288 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:47.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:47.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:30:47.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:47.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:30:47.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:47.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:30:47.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:47.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:47.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:30:47.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:30:47.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:47.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:30:47.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:30:47.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:47.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:30:47.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:30:47.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:30:47.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:30:47.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:47.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:30:47.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:30:47.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:30:47.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:30:47.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:47.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:30:47.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:30:47.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:30:47.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:30:47.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:47.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:30:47.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:30:47.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:47.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:30:47.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:30:47.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:47.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.299 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:182 3.4 3:2022fdfa:::smithi09025264-46:head [write 1933572~791670] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:47.299 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:30:47.299 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 139.785334479, 2023-01-15T14:30:47.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.582304865, 2023-01-15T14:30:47.300 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:47.300 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:47.300 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:47.300 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:47.300 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:47.301 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 182 2023-01-15T14:30:47.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.301 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:47.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:47.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:30:47.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:47.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:30:47.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:47.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213704+0000", 2023-01-15T14:30:47.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:47.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:47.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214574+0000", 2023-01-15T14:30:47.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087037799999999999 2023-01-15T14:30:47.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:47.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214608+0000", 2023-01-15T14:30:47.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:30:47.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:47.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214627+0000", 2023-01-15T14:30:47.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:30:47.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491537+0000", 2023-01-15T14:30:47.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27691020300000002 2023-01-15T14:30:47.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:47.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491721+0000", 2023-01-15T14:30:47.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345500000000001 2023-01-15T14:30:47.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693760+0000", 2023-01-15T14:30:47.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202039252 2023-01-15T14:30:47.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:47.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693898+0000", 2023-01-15T14:30:47.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013789 2023-01-15T14:30:47.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:47.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694230+0000", 2023-01-15T14:30:47.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220899999999998 2023-01-15T14:30:47.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:47.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694866+0000", 2023-01-15T14:30:47.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063640599999999995 2023-01-15T14:30:47.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:47.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694887+0000", 2023-01-15T14:30:47.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0622999999999999e-05 2023-01-15T14:30:47.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:47.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695950+0000", 2023-01-15T14:30:47.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001062688 2023-01-15T14:30:47.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:47.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696014+0000", 2023-01-15T14:30:47.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4697999999999996e-05 2023-01-15T14:30:47.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795898+0000", 2023-01-15T14:30:47.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099883338000000002 2023-01-15T14:30:47.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795922+0000", 2023-01-15T14:30:47.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4326000000000001e-05 2023-01-15T14:30:47.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:47.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795955+0000", 2023-01-15T14:30:47.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3191000000000001e-05 2023-01-15T14:30:47.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:47.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796011+0000", 2023-01-15T14:30:47.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5631999999999999e-05 2023-01-15T14:30:47.485 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.485 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:47.485 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.486 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:47.486 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:30:47.486 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 139.784387748, 2023-01-15T14:30:47.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:30:47.487 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:47.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:47.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:47.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:47.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:47.488 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:30:47.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.488 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:47.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:47.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:30:47.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:30:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:30:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:30:47.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:30:47.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:47.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:30:47.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:30:47.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:30:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:30:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:30:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:30:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:30:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:30:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:30:47.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:30:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:30:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:30:47.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:30:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:30:47.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:30:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:30:47.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:30:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:30:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:30:47.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:30:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:30:47.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:30:47.501 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 139.78433105100001, 2023-01-15T14:30:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:30:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:30:47.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:30:47.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:47.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:30:47.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:47.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:30:47.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:47.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:47.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:30:47.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:30:47.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:47.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:30:47.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:30:47.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:47.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:30:47.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:30:47.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:30:47.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:30:47.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:47.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:30:47.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:30:47.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:30:47.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:30:47.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:47.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:30:47.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:30:47.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:30:47.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:30:47.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:47.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:30:47.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:30:47.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:47.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:30:47.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:30:47.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.510 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:47.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.511 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:30:47.511 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:30:47.511 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 137.16783487800001, 2023-01-15T14:30:47.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:30:47.511 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:47.511 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:47.511 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:47.511 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:47.512 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:47.512 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:30:47.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.512 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:47.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:47.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:30:47.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:47.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:30:47.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:47.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:30:47.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:47.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:47.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:30:47.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:30:47.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:47.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:30:47.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:30:47.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:47.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:30:47.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:30:47.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:30:47.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:30:47.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:47.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:30:47.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:30:47.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:30:47.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:30:47.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:47.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:30:47.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:30:47.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:47.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:30:47.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:30:47.703 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.703 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:47.703 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.703 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:30:47.703 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:30:47.703 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.138986407, 2023-01-15T14:30:47.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:30:47.703 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:47.703 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:47.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:47.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:47.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:47.704 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:30:47.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.704 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:47.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:47.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:30:47.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:47.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:30:47.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:47.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:30:47.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:47.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:47.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:30:47.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:30:47.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:30:47.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:30:47.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:47.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:30:47.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:30:47.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:30:47.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:30:47.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:47.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:30:47.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:47.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:30:47.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:30:47.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:47.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:30:47.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:30:47.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:30:47.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:30:47.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:47.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:30:47.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:30:47.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:47.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:30:47.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:30:47.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.713 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:47.713 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.713 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:373 3.4 3:2029fa71:::smithi09025264-39:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:47.713 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:30:47.713 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 126.9207392, 2023-01-15T14:30:47.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58298356399999995, 2023-01-15T14:30:47.713 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:47.713 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:47.713 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:47.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:30:47.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:47.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078299+0000", 2023-01-15T14:30:47.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:47.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:47.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078644+0000", 2023-01-15T14:30:47.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034494099999999998 2023-01-15T14:30:47.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:47.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078676+0000", 2023-01-15T14:30:47.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1183000000000003e-05 2023-01-15T14:30:47.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078694+0000", 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-05 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078748+0000", 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3634999999999997e-05 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078753+0000", 2023-01-15T14:30:47.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6370000000000004e-06 2023-01-15T14:30:47.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110841+0000", 2023-01-15T14:30:47.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032087439000000002 2023-01-15T14:30:47.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:47.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110967+0000", 2023-01-15T14:30:47.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126029 2023-01-15T14:30:47.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.457874+0000", 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.346907725 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458066+0000", 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000191592 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458321+0000", 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025454200000000002 2023-01-15T14:30:47.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458918+0000", 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059740800000000001 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458945+0000", 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-05 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:47.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459253+0000", 2023-01-15T14:30:47.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030836200000000001 2023-01-15T14:30:47.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:47.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460943+0000", 2023-01-15T14:30:47.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016903580000000001 2023-01-15T14:30:47.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461340+0000", 2023-01-15T14:30:47.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039665799999999998 2023-01-15T14:30:47.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661129+0000", 2023-01-15T14:30:47.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19978933500000001 2023-01-15T14:30:47.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:47.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661188+0000", 2023-01-15T14:30:47.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8687000000000002e-05 2023-01-15T14:30:47.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:47.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661285+0000", 2023-01-15T14:30:47.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779000000000003e-05 2023-01-15T14:30:47.916 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.916 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:47.917 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.917 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:374 3.4 3:2029fa71:::smithi09025264-39:head [write 3101234~41642] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:47.917 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:30:47.917 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 126.916077288, 2023-01-15T14:30:47.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57848911800000002, 2023-01-15T14:30:47.918 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:47.918 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:47.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:47.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:47.918 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 374 2023-01-15T14:30:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:30:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082963+0000", 2023-01-15T14:30:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.082961+0000", 2023-01-15T14:30:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083019+0000", 2023-01-15T14:30:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7769999999999997e-05 2023-01-15T14:30:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083051+0000", 2023-01-15T14:30:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1813999999999997e-05 2023-01-15T14:30:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083069+0000", 2023-01-15T14:30:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:30:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083143+0000", 2023-01-15T14:30:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4363000000000006e-05 2023-01-15T14:30:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083147+0000", 2023-01-15T14:30:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4529999999999998e-06 2023-01-15T14:30:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111017+0000", 2023-01-15T14:30:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027870564 2023-01-15T14:30:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.111099+0000", 2023-01-15T14:30:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1277000000000001e-05 2023-01-15T14:30:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459156+0000", 2023-01-15T14:30:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34805700499999997 2023-01-15T14:30:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459393+0000", 2023-01-15T14:30:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023757499999999999 2023-01-15T14:30:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459778+0000", 2023-01-15T14:30:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000385335 2023-01-15T14:30:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460020+0000", 2023-01-15T14:30:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024141899999999999 2023-01-15T14:30:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460037+0000", 2023-01-15T14:30:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6665999999999999e-05 2023-01-15T14:30:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460220+0000", 2023-01-15T14:30:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018367599999999999 2023-01-15T14:30:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461197+0000", 2023-01-15T14:30:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00097708600000000006 2023-01-15T14:30:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461720+0000", 2023-01-15T14:30:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052235400000000005 2023-01-15T14:30:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661345+0000", 2023-01-15T14:30:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199624884 2023-01-15T14:30:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661405+0000", 2023-01-15T14:30:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0965999999999999e-05 2023-01-15T14:30:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661452+0000", 2023-01-15T14:30:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6864999999999999e-05 2023-01-15T14:30:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:375 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (56) in=7b,truncate 3142876] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 126.915935522, 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57851114599999998, 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 375 2023-01-15T14:30:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:30:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:30:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083104+0000", 2023-01-15T14:30:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083113+0000", 2023-01-15T14:30:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6500000000000008e-06 2023-01-15T14:30:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:48.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083130+0000", 2023-01-15T14:30:48.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2023-01-15T14:30:48.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:48.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083142+0000", 2023-01-15T14:30:48.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2252999999999999e-05 2023-01-15T14:30:48.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083173+0000", 2023-01-15T14:30:48.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0474999999999999e-05 2023-01-15T14:30:48.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:48.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083175+0000", 2023-01-15T14:30:48.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2023-01-15T14:30:48.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183469+0000", 2023-01-15T14:30:48.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10029313099999999 2023-01-15T14:30:48.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:48.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183747+0000", 2023-01-15T14:30:48.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000277998 2023-01-15T14:30:48.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460165+0000", 2023-01-15T14:30:48.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27641872899999997 2023-01-15T14:30:48.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:48.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460342+0000", 2023-01-15T14:30:48.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176792 2023-01-15T14:30:48.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:30:48.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460579+0000", 2023-01-15T14:30:48.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000237266 2023-01-15T14:30:48.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:48.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460833+0000", 2023-01-15T14:30:48.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025390900000000002 2023-01-15T14:30:48.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:48.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460853+0000", 2023-01-15T14:30:48.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9273e-05 2023-01-15T14:30:48.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:48.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461024+0000", 2023-01-15T14:30:48.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017111900000000001 2023-01-15T14:30:48.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:48.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461230+0000", 2023-01-15T14:30:48.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020672600000000001 2023-01-15T14:30:48.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:48.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461879+0000", 2023-01-15T14:30:48.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064846999999999995 2023-01-15T14:30:48.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:48.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661524+0000", 2023-01-15T14:30:48.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19964509599999999 2023-01-15T14:30:48.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:48.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661560+0000", 2023-01-15T14:30:48.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6264999999999999e-05 2023-01-15T14:30:48.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:48.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661616+0000", 2023-01-15T14:30:48.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5952000000000002e-05 2023-01-15T14:30:48.111 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.111 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:48.111 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.111 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:376 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:48.111 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:30:48.112 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 126.915853266, 2023-01-15T14:30:48.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57864278800000002, 2023-01-15T14:30:48.112 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:48.112 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:48.112 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:48.112 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:48.112 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:48.112 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 376 2023-01-15T14:30:48.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.112 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:48.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:48.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:30:48.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:48.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:30:48.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:48.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083186+0000", 2023-01-15T14:30:48.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:48.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:48.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083193+0000", 2023-01-15T14:30:48.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2023-01-15T14:30:48.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:48.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083211+0000", 2023-01-15T14:30:48.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7609999999999999e-05 2023-01-15T14:30:48.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:48.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083223+0000", 2023-01-15T14:30:48.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140000000000001e-05 2023-01-15T14:30:48.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093534+0000", 2023-01-15T14:30:48.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010311529 2023-01-15T14:30:48.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:48.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093551+0000", 2023-01-15T14:30:48.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.666e-05 2023-01-15T14:30:48.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183834+0000", 2023-01-15T14:30:48.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090283063999999996 2023-01-15T14:30:48.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:48.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.184076+0000", 2023-01-15T14:30:48.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024241900000000001 2023-01-15T14:30:48.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461001+0000", 2023-01-15T14:30:48.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27692504299999998 2023-01-15T14:30:48.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:48.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461152+0000", 2023-01-15T14:30:48.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150345 2023-01-15T14:30:48.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661642+0000", 2023-01-15T14:30:48.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200489742 2023-01-15T14:30:48.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:48.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661720+0000", 2023-01-15T14:30:48.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8604999999999997e-05 2023-01-15T14:30:48.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:48.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661830+0000", 2023-01-15T14:30:48.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010992099999999999 2023-01-15T14:30:48.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.121 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:48.121 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.121 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:30:48.121 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:48.121 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.137699431, 2023-01-15T14:30:48.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:30:48.122 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:48.122 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:48.122 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:48.122 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:48.122 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:48.122 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:30:48.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.122 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:48.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:48.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:48.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:48.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:48.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:48.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:30:48.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:48.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:48.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:30:48.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:30:48.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:48.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:30:48.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:30:48.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:48.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:30:48.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:30:48.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:30:48.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:30:48.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:48.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:30:48.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:30:48.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:30:48.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:30:48.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:30:48.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:48.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:30:48.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:30:48.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:48.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:30:48.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:30:48.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:48.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:30:48.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:30:48.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:48.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:30:48.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:30:48.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:48.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:30:48.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:30:48.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:48.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:30:48.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:30:48.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:48.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:30:48.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:30:48.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:48.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:30:48.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:30:48.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.131 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:48.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.278 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:961 3.0 3:0a998d04:::smithi09025264-10:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e85)", 2023-01-15T14:30:48.278 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:48.279 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.935930842999994, 2023-01-15T14:30:48.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51361718300000003, 2023-01-15T14:30:48.279 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:48.279 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:48.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:48.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:48.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:48.279 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 961 2023-01-15T14:30:48.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.279 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:48.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:48.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:48.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:48.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063110+0000", 2023-01-15T14:30:48.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:48.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063109+0000", 2023-01-15T14:30:48.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:48.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:48.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063115+0000", 2023-01-15T14:30:48.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8300000000000001e-06 2023-01-15T14:30:48.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:48.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063127+0000", 2023-01-15T14:30:48.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2517000000000001e-05 2023-01-15T14:30:48.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:48.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063137+0000", 2023-01-15T14:30:48.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8509999999999994e-06 2023-01-15T14:30:48.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063272+0000", 2023-01-15T14:30:48.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013513500000000001 2023-01-15T14:30:48.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:48.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.063277+0000", 2023-01-15T14:30:48.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2750000000000003e-06 2023-01-15T14:30:48.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.265103+0000", 2023-01-15T14:30:48.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201825688 2023-01-15T14:30:48.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:48.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.265208+0000", 2023-01-15T14:30:48.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104693 2023-01-15T14:30:48.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.375398+0000", 2023-01-15T14:30:48.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.110190553 2023-01-15T14:30:48.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:48.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.375548+0000", 2023-01-15T14:30:48.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000149417 2023-01-15T14:30:48.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576187+0000", 2023-01-15T14:30:48.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20063947500000001 2023-01-15T14:30:48.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:48.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576454+0000", 2023-01-15T14:30:48.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026708400000000001 2023-01-15T14:30:48.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:31.576727+0000", 2023-01-15T14:30:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027257699999999998 2023-01-15T14:30:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:30:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:48.287 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.077121181000003, 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:48.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:48.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:48.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:30:48.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:30:48.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:48.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:30:48.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:30:48.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:30:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:30:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:30:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:30:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:48.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:30:48.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:30:48.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:30:48.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:30:48.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:30:48.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:48.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:30:48.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:30:48.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:48.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:30:48.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:30:48.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:48.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:30:48.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:30:48.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:48.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:30:48.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:30:48.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:48.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:30:48.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:30:48.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:48.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:30:48.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:30:48.296 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.296 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:48.296 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.296 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:30:48.296 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:48.296 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.077062470999998, 2023-01-15T14:30:48.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:30:48.297 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:48.297 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:48.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:48.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:48.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:48.297 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:30:48.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.297 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:48.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:48.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:48.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:48.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:30:48.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:30:48.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:48.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:30:48.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:30:48.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:48.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:30:48.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:30:48.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:30:48.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:30:48.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:48.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:30:48.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:30:48.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:30:48.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:30:48.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:48.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:30:48.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:30:48.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:48.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:30:48.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:30:48.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:48.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.303 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:48.303 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:48.304 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.120642977999999, 2023-01-15T14:30:48.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:30:48.304 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:48.304 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:48.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:48.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:48.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:48.304 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:30:48.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.304 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:48.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:48.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:48.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:48.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:30:48.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:48.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:30:48.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:48.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:48.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:30:48.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:30:48.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:48.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:30:48.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:48.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:30:48.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:30:48.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:30:48.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:30:48.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:48.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:30:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:30:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:30:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:30:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.483 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:30:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.063145294999998, 2023-01-15T14:30:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:30:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:48.484 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:30:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:30:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:30:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:48.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:30:48.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:30:48.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:30:48.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:30:48.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:30:48.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:30:48.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:30:48.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:30:48.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:30:48.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:48.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:30:48.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:30:48.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:48.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:30:48.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:48.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:30:48.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:30:48.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:48.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:30:48.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:30:48.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.494 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:48.494 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:30:48.494 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.694493789999999, 2023-01-15T14:30:48.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:30:48.494 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:48.494 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:48.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:48.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:48.494 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:48.494 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:30:48.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:48.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:30:48.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:30:48.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:48.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:30:48.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:30:48.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:48.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:30:48.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:30:48.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:30:48.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:30:48.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:30:48.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:48.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:30:48.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:30:48.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:48.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:30:48.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:30:48.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:30:48.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:30:48.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:30:48.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:48.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:30:48.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:30:48.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.502 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:48.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.502 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:48.503 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:30:48.503 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 27.694391149000001, 2023-01-15T14:30:48.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:30:48.503 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:48.503 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:30:48.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:30:48.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:30:48.503 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:48.503 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:30:48.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:30:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:30:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:48.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:48.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:30:48.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:30:48.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:48.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:30:48.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:30:48.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:30:48.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:30:48.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:48.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:30:48.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:30:48.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:30:48.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:30:48.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:48.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:30:48.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:48.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:48.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:30:48.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:30:48.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:30:48.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.510 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:48.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:48.510 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:48.510 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:48.511 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:30:48.511 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7] out_osds: [5, 6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:30:48.511 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:30:48.512 INFO:tasks.thrashosds.thrasher:Adding osd 1 2023-01-15T14:30:48.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2023-01-15T14:30:48.512 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:30:48.513 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:48.507+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:48.514 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:48.514 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:48.514 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:48.514 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:48.514 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:48.514 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:48.514 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:48.515 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:48.515 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:30:48.515 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:30:48.516 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:30:48.516 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:30:48.516 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:30:48.516 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:30:48.516 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:30:48.516 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:30:48.517 INFO:tasks.rados.rados.0.smithi090.stdout:652: finishing rollback tid 2 to smithi09025264-13 2023-01-15T14:30:48.517 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 214 (ObjNum 216 snap 49 seq_num 216) dirty exists 2023-01-15T14:30:48.517 INFO:tasks.rados.rados.0.smithi090.stdout:652: done (0 left) 2023-01-15T14:30:48.618 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:48.615+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:48.698 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:48.699 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:48.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:30:48.718 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:48.715+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:48.819 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:48.815+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:48.897 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:48.898 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:48.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:30:48.917 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-01-15T14:30:48.912+0000 7fb90baaf700 -1 mon.a@0(leader).osd e162 definitely_dead 0 2023-01-15T14:30:48.917 INFO:tasks.rados.rados.0.smithi090.stdout:655: done (0 left) 2023-01-15T14:30:48.917 INFO:tasks.rados.rados.0.smithi090.stdout:656: copy_from oid 12 from oid 9 current snap is 62 2023-01-15T14:30:48.918 INFO:tasks.rados.rados.0.smithi090.stdout:657: snap_create 2023-01-15T14:30:48.918 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:48.920 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:48.916+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:48.923 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:30:48.996 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:49.108 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:49.110 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:30:49.110 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:49.108+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:49.211 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:49.207+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:49.272 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:49.273 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:49.273 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:30:49.312 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:49.311+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:49.323 INFO:tasks.rados.rados.0.smithi090.stdout:656: finishing copy_from to smithi09025264-12 2023-01-15T14:30:49.323 INFO:tasks.rados.rados.0.smithi090.stdout:656: got expected ENOENT (src dne) 2023-01-15T14:30:49.323 INFO:tasks.rados.rados.0.smithi090.stdout:656: finishing copy_from racing read to smithi09025264-12 2023-01-15T14:30:49.323 INFO:tasks.rados.rados.0.smithi090.stdout:656: done (0 left) 2023-01-15T14:30:49.387 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:49.387 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:49.397 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:30:49.413 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:49.411+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:49.500 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:49.501 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:49.511 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:30:49.514 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:49.512+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:49.615 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:49.612+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:49.626 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:49.626 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:49.637 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:30:49.715 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:49.712+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:49.745 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:49.746 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:49.755 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:30:49.818 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:49.816+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:49.860 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:49.860 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:49.870 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:30:49.917 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:49.915+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:49.931 INFO:tasks.rados.rados.0.smithi090.stdout:657: done (0 left) 2023-01-15T14:30:49.931 INFO:tasks.rados.rados.0.smithi090.stdout:658: copy_from oid 1 from oid 42 current snap is 63 2023-01-15T14:30:49.931 INFO:tasks.rados.rados.0.smithi090.stdout:659: read oid 44 snap -1 2023-01-15T14:30:49.931 INFO:tasks.rados.rados.0.smithi090.stdout:659: expect (ObjNum 232 snap 52 seq_num 232) 2023-01-15T14:30:49.931 INFO:tasks.rados.rados.0.smithi090.stdout:660: snap_create 2023-01-15T14:30:49.931 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:49.934 INFO:teuthology.orchestra.run.smithi090.stderr:marked in osd.1. 2023-01-15T14:30:49.948 INFO:tasks.thrashosds.thrasher:Added osd 1 2023-01-15T14:30:49.981 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:49.981 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:49.991 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:30:50.018 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:50.016+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:50.119 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:50.116+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:50.138 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:50.265 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:50.267 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:30:50.267 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:50.263+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:50.368 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:50.364+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:50.419 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:50.419 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:50.419 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:30:50.468 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:50.463+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:50.517 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:30:50.623 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:50.624 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:50.624 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:30:50.625 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:50.620+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:50.724 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:50.720+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:50.738 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:50.738 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:50.748 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:30:50.826 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:50.823+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:50.837 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:50.853 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:50.854 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:50.854 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:50.864 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:30:50.873 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:50.892 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:50.913 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:50.926 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:50.924+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:50.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:30:50.943 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:50.961 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:50.972 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:50.972 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:50.981 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:50.982 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:30:50.982 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:30:50.982 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:30:50.982 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:30:50.983 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:30:50.983 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:30:50.983 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:30:50.983 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:30:50.983 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:30:51.027 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:51.023+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:51.085 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:51.096 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:30:51.096 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:30:51.096 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:30:51.096 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:51.097 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:30:51.128 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:51.123+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:51.203 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:51.294 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:30:51.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:30:51.294 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:30:51.294 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.294 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:71 3.f 3:fe04f54c:::smithi09025264-18:head [setxattr _header (56) in=7b,truncate 2285719] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:51.294 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:30:51.294 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 144.94552616799999, 2023-01-15T14:30:51.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.82719834800000003, 2023-01-15T14:30:51.295 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.295 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:51.295 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.295 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.295 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.295 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 71 2023-01-15T14:30:51.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.295 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:30:51.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.296 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.296 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:30:51.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.296 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.296 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252664+0000", 2023-01-15T14:30:51.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:51.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252675+0000", 2023-01-15T14:30:51.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1296e-05 2023-01-15T14:30:51.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252697+0000", 2023-01-15T14:30:51.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1874000000000001e-05 2023-01-15T14:30:51.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252708+0000", 2023-01-15T14:30:51.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.15e-05 2023-01-15T14:30:51.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355413+0000", 2023-01-15T14:30:51.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10270497100000001 2023-01-15T14:30:51.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355622+0000", 2023-01-15T14:30:51.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000208982 2023-01-15T14:30:51.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.758709+0000", 2023-01-15T14:30:51.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40308671800000001 2023-01-15T14:30:51.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.300 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.758958+0000", 2023-01-15T14:30:51.300 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024949 2023-01-15T14:30:51.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.300 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:51.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759375+0000", 2023-01-15T14:30:51.300 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000416208 2023-01-15T14:30:51.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:51.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759708+0000", 2023-01-15T14:30:51.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033326900000000002 2023-01-15T14:30:51.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:51.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759744+0000", 2023-01-15T14:30:51.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6477000000000003e-05 2023-01-15T14:30:51.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:51.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759822+0000", 2023-01-15T14:30:51.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7440000000000004e-05 2023-01-15T14:30:51.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:51.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760139+0000", 2023-01-15T14:30:51.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316996 2023-01-15T14:30:51.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760440+0000", 2023-01-15T14:30:51.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000300945 2023-01-15T14:30:51.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079578+0000", 2023-01-15T14:30:51.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31913862599999998 2023-01-15T14:30:51.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:51.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079709+0000", 2023-01-15T14:30:51.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013108 2023-01-15T14:30:51.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.304 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079863+0000", 2023-01-15T14:30:51.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000153819 2023-01-15T14:30:51.304 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.304 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.304 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.305 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:72 3.f 3:fe04f54c:::smithi09025264-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:51.305 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:30:51.305 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 144.945466831, 2023-01-15T14:30:51.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.82782374599999997, 2023-01-15T14:30:51.305 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.305 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:51.305 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.305 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.305 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.305 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 72 2023-01-15T14:30:51.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.306 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:30:51.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:30:51.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252723+0000", 2023-01-15T14:30:51.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:51.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252730+0000", 2023-01-15T14:30:51.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.703e-06 2023-01-15T14:30:51.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.308 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.308 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252747+0000", 2023-01-15T14:30:51.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6739999999999999e-05 2023-01-15T14:30:51.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.308 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.308 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252758+0000", 2023-01-15T14:30:51.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1389999999999999e-05 2023-01-15T14:30:51.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.455798+0000", 2023-01-15T14:30:51.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20303954199999999 2023-01-15T14:30:51.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.456092+0000", 2023-01-15T14:30:51.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029407199999999999 2023-01-15T14:30:51.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.310 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.310 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759852+0000", 2023-01-15T14:30:51.310 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.303760525 2023-01-15T14:30:51.310 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.310 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.310 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.310 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760038+0000", 2023-01-15T14:30:51.310 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018584600000000001 2023-01-15T14:30:51.310 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.310 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.310 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.311 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080005+0000", 2023-01-15T14:30:51.311 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31996668299999997 2023-01-15T14:30:51.311 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.311 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.311 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.311 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080286+0000", 2023-01-15T14:30:51.311 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028087899999999999 2023-01-15T14:30:51.311 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.311 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.311 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.311 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080548+0000", 2023-01-15T14:30:51.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026241300000000002 2023-01-15T14:30:51.312 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.312 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.312 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.312 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:75 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:51.312 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:30:51.312 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 144.84129448100001, 2023-01-15T14:30:51.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92335155400000002, 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 75 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:30:51.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:30:51.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356896+0000", 2023-01-15T14:30:51.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:51.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356903+0000", 2023-01-15T14:30:51.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4619999999999999e-06 2023-01-15T14:30:51.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356917+0000", 2023-01-15T14:30:51.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4195999999999999e-05 2023-01-15T14:30:51.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356926+0000", 2023-01-15T14:30:51.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:30:51.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457743+0000", 2023-01-15T14:30:51.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100817668 2023-01-15T14:30:51.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457841+0000", 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7079999999999999e-05 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778474+0000", 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32063301799999999 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778627+0000", 2023-01-15T14:30:51.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015389599999999999 2023-01-15T14:30:51.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:51.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778815+0000", 2023-01-15T14:30:51.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018726600000000001 2023-01-15T14:30:51.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:51.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779033+0000", 2023-01-15T14:30:51.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021819999999999999 2023-01-15T14:30:51.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:51.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779051+0000", 2023-01-15T14:30:51.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7640000000000001e-05 2023-01-15T14:30:51.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:51.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:30:51.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3590999999999995e-05 2023-01-15T14:30:51.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:51.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779163+0000", 2023-01-15T14:30:51.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:30:51.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.081002+0000", 2023-01-15T14:30:51.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30183937799999999 2023-01-15T14:30:51.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280043+0000", 2023-01-15T14:30:51.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19904084699999999 2023-01-15T14:30:51.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:51.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280138+0000", 2023-01-15T14:30:51.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4915999999999997e-05 2023-01-15T14:30:51.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280248+0000", 2023-01-15T14:30:51.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110233 2023-01-15T14:30:51.322 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.322 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.322 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.322 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:76 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:51.322 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:30:51.322 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 144.841252873, 2023-01-15T14:30:51.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92420393899999997, 2023-01-15T14:30:51.323 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.323 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:51.323 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.323 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.323 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.323 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 76 2023-01-15T14:30:51.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.323 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:30:51.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:30:51.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356937+0000", 2023-01-15T14:30:51.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:51.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356948+0000", 2023-01-15T14:30:51.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.094e-05 2023-01-15T14:30:51.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356959+0000", 2023-01-15T14:30:51.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1058e-05 2023-01-15T14:30:51.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356968+0000", 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5339999999999999e-06 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459524+0000", 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102556147 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459662+0000", 2023-01-15T14:30:51.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000137756 2023-01-15T14:30:51.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978621+0000", 2023-01-15T14:30:51.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51895887600000001 2023-01-15T14:30:51.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.454 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.454 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.455 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978870+0000", 2023-01-15T14:30:51.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024928999999999999 2023-01-15T14:30:51.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.455 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.455 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280761+0000", 2023-01-15T14:30:51.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30189113400000001 2023-01-15T14:30:51.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.455 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.455 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281023+0000", 2023-01-15T14:30:51.456 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026172800000000003 2023-01-15T14:30:51.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.456 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.456 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281142+0000", 2023-01-15T14:30:51.456 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000119296 2023-01-15T14:30:51.456 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.456 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.456 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.457 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:91 3.d 3:bc1c6b44:::smithi09025264-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:51.457 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:30:51.457 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 144.79476449200001, 2023-01-15T14:30:51.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.844937151, 2023-01-15T14:30:51.457 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.457 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:51.457 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.457 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.457 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.457 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 91 2023-01-15T14:30:51.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.458 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:30:51.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:30:51.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403425+0000", 2023-01-15T14:30:51.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:51.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403434+0000", 2023-01-15T14:30:51.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3299999999999999e-06 2023-01-15T14:30:51.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403446+0000", 2023-01-15T14:30:51.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2176e-05 2023-01-15T14:30:51.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403453+0000", 2023-01-15T14:30:51.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5830000000000003e-06 2023-01-15T14:30:51.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403654+0000", 2023-01-15T14:30:51.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020136000000000001 2023-01-15T14:30:51.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403807+0000", 2023-01-15T14:30:51.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015342599999999999 2023-01-15T14:30:51.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907836+0000", 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50402854699999999 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907934+0000", 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8472000000000002e-05 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908096+0000", 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000162139 2023-01-15T14:30:51.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908335+0000", 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000238891 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908350+0000", 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4868000000000001e-05 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:51.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908416+0000", 2023-01-15T14:30:51.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5502999999999999e-05 2023-01-15T14:30:51.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.464 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:51.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.952370+0000", 2023-01-15T14:30:51.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.043954663999999997 2023-01-15T14:30:51.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.464 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.008749+0000", 2023-01-15T14:30:51.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.056378696999999998 2023-01-15T14:30:51.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248164+0000", 2023-01-15T14:30:51.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23941479500000001 2023-01-15T14:30:51.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:51.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248244+0000", 2023-01-15T14:30:51.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9616000000000005e-05 2023-01-15T14:30:51.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248364+0000", 2023-01-15T14:30:51.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120167 2023-01-15T14:30:51.466 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.466 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.466 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.466 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:92 3.d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:51.466 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:30:51.467 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 144.79472879900001, 2023-01-15T14:30:51.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.84526834699999998, 2023-01-15T14:30:51.467 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.467 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:51.467 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.467 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.467 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.467 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 92 2023-01-15T14:30:51.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.467 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:30:51.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:30:51.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403461+0000", 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403466+0000", 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.34e-06 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403476+0000", 2023-01-15T14:30:51.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8530000000000004e-06 2023-01-15T14:30:51.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403492+0000", 2023-01-15T14:30:51.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6646999999999998e-05 2023-01-15T14:30:51.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501333+0000", 2023-01-15T14:30:51.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097840420999999997 2023-01-15T14:30:51.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501483+0000", 2023-01-15T14:30:51.471 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150123 2023-01-15T14:30:51.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.007528+0000", 2023-01-15T14:30:51.471 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50604486299999996 2023-01-15T14:30:51.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.472 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.007649+0000", 2023-01-15T14:30:51.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001217 2023-01-15T14:30:51.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.472 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248441+0000", 2023-01-15T14:30:51.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.240791267 2023-01-15T14:30:51.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248561+0000", 2023-01-15T14:30:51.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120516 2023-01-15T14:30:51.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248731+0000", 2023-01-15T14:30:51.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016937000000000001 2023-01-15T14:30:51.473 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.473 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.473 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1076 3.f 3:f87c0183:::smithi09025264-50:head [setxattr _header (57) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 68.439544118000001, 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.88473693200000003, 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1076 2023-01-15T14:30:51.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.475 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:30:51.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:30:51.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758646+0000", 2023-01-15T14:30:51.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:51.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758653+0000", 2023-01-15T14:30:51.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8029999999999997e-06 2023-01-15T14:30:51.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758670+0000", 2023-01-15T14:30:51.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6158999999999999e-05 2023-01-15T14:30:51.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758677+0000", 2023-01-15T14:30:51.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9840000000000004e-06 2023-01-15T14:30:51.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848049+0000", 2023-01-15T14:30:51.478 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089372163000000004 2023-01-15T14:30:51.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:51.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848056+0000", 2023-01-15T14:30:51.478 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0360000000000001e-06 2023-01-15T14:30:51.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951562+0000", 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10350670100000001 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951726+0000", 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000163311 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951958+0000", 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000232525 2023-01-15T14:30:51.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:51.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952168+0000", 2023-01-15T14:30:51.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020949799999999999 2023-01-15T14:30:51.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:51.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952186+0000", 2023-01-15T14:30:51.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7714e-05 2023-01-15T14:30:51.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:51.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952254+0000", 2023-01-15T14:30:51.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7998999999999997e-05 2023-01-15T14:30:51.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:51.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952278+0000", 2023-01-15T14:30:51.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4909000000000002e-05 2023-01-15T14:30:51.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065836+0000", 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.113557081 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643196+0000", 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.57736012999999997 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643278+0000", 2023-01-15T14:30:51.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2671999999999997e-05 2023-01-15T14:30:51.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.483 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643384+0000", 2023-01-15T14:30:51.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010551400000000001 2023-01-15T14:30:51.483 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.483 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.483 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.483 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1077 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 68.439505299000004, 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.98557123499999999, 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1077 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.484 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:30:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:30:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758685+0000", 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758689+0000", 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2760000000000002e-06 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758704+0000", 2023-01-15T14:30:51.486 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4187e-05 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758710+0000", 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4330000000000001e-06 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848086+0000", 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089376466000000002 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.487 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:51.488 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848093+0000", 2023-01-15T14:30:51.488 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4269999999999996e-06 2023-01-15T14:30:51.488 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.488 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952346+0000", 2023-01-15T14:30:51.488 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104252758 2023-01-15T14:30:51.488 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.489 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952427+0000", 2023-01-15T14:30:51.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1786000000000006e-05 2023-01-15T14:30:51.489 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.489 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.489 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.489 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743602+0000", 2023-01-15T14:30:51.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.79117428099999998 2023-01-15T14:30:51.489 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.489 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.489 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.489 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743892+0000", 2023-01-15T14:30:51.490 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029067200000000001 2023-01-15T14:30:51.490 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.490 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.490 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.490 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.744257+0000", 2023-01-15T14:30:51.490 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000364619 2023-01-15T14:30:51.490 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.490 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.490 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.490 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.490 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.617 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1087 3.f 3:fd983206:::smithi09025264-7:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:51.618 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:30:51.618 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 68.439470817, 2023-01-15T14:30:51.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0859964360000001, 2023-01-15T14:30:51.618 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.618 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:51.618 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.618 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.618 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.618 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1087 2023-01-15T14:30:51.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.619 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:30:51.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758724+0000", 2023-01-15T14:30:51.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7740000000000004e-06 2023-01-15T14:30:51.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758738+0000", 2023-01-15T14:30:51.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.33e-05 2023-01-15T14:30:51.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758744+0000", 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2859999999999996e-06 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848121+0000", 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089377097000000003 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848127+0000", 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3500000000000002e-06 2023-01-15T14:30:51.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952454+0000", 2023-01-15T14:30:51.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104326167 2023-01-15T14:30:51.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952520+0000", 2023-01-15T14:30:51.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6166000000000003e-05 2023-01-15T14:30:51.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:30:51.624 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.063003+0000", 2023-01-15T14:30:51.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.110482916 2023-01-15T14:30:51.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:51.624 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065265+0000", 2023-01-15T14:30:51.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022626370000000001 2023-01-15T14:30:51.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:51.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065291+0000", 2023-01-15T14:30:51.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6095e-05 2023-01-15T14:30:51.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:51.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.066153+0000", 2023-01-15T14:30:51.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086127100000000004 2023-01-15T14:30:51.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:51.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166533+0000", 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100380177 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166646+0000", 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112717 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844498+0000", 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67785237399999998 2023-01-15T14:30:51.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:51.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844605+0000", 2023-01-15T14:30:51.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106691 2023-01-15T14:30:51.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844717+0000", 2023-01-15T14:30:51.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011203499999999999 2023-01-15T14:30:51.627 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.627 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.628 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.628 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1088 3.f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:30:51.628 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:30:51.628 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 68.439415151000006, 2023-01-15T14:30:51.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.086312696, 2023-01-15T14:30:51.628 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.628 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:51.628 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.628 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.629 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.629 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1088 2023-01-15T14:30:51.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.629 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:30:51.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758775+0000", 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758786+0000", 2023-01-15T14:30:51.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.076e-05 2023-01-15T14:30:51.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758809+0000", 2023-01-15T14:30:51.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3706999999999999e-05 2023-01-15T14:30:51.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758821+0000", 2023-01-15T14:30:51.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1976e-05 2023-01-15T14:30:51.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878534+0000", 2023-01-15T14:30:51.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.119712966 2023-01-15T14:30:51.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:51.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878540+0000", 2023-01-15T14:30:51.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0680000000000002e-06 2023-01-15T14:30:51.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952693+0000", 2023-01-15T14:30:51.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.074153067000000003 2023-01-15T14:30:51.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:51.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952735+0000", 2023-01-15T14:30:51.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1792999999999998e-05 2023-01-15T14:30:51.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065518+0000", 2023-01-15T14:30:51.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.11278334499999999 2023-01-15T14:30:51.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065687+0000", 2023-01-15T14:30:51.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016807700000000001 2023-01-15T14:30:51.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844786+0000", 2023-01-15T14:30:51.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.77909980300000004 2023-01-15T14:30:51.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.635 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844944+0000", 2023-01-15T14:30:51.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157424 2023-01-15T14:30:51.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.635 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.845089+0000", 2023-01-15T14:30:51.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144814 2023-01-15T14:30:51.636 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.636 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.636 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.636 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:16 3.13 3:ca393a9e:::smithi09025264-47:head [assert-version v193,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:51.636 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:30:51.636 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 31.892437726000001, 2023-01-15T14:30:51.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8970339479999998, 2023-01-15T14:30:51.636 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.636 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305751+0000", 2023-01-15T14:30:51.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:51.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305763+0000", 2023-01-15T14:30:51.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2309e-05 2023-01-15T14:30:51.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305788+0000", 2023-01-15T14:30:51.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4388000000000001e-05 2023-01-15T14:30:51.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305799+0000", 2023-01-15T14:30:51.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1304000000000001e-05 2023-01-15T14:30:51.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196425+0000", 2023-01-15T14:30:51.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8906259830000001 2023-01-15T14:30:51.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196673+0000", 2023-01-15T14:30:51.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024747499999999998 2023-01-15T14:30:51.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.202787+0000", 2023-01-15T14:30:51.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0061146509999999996 2023-01-15T14:30:51.641 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.641 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.641 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.641 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1480 3.13 3:cbfd2d71:::smithi09025264-37:head [write 581575~705262,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:51.641 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:30:51.641 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 31.859169997999999, 2023-01-15T14:30:51.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.375425221, 2023-01-15T14:30:51.641 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.641 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1480 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:30:51.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339019+0000", 2023-01-15T14:30:51.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:51.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339659+0000", 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000639742 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339690+0000", 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1594000000000003e-05 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339703+0000", 2023-01-15T14:30:51.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2828e-05 2023-01-15T14:30:51.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.510837+0000", 2023-01-15T14:30:51.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1711338759999999 2023-01-15T14:30:51.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511048+0000", 2023-01-15T14:30:51.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021057899999999999 2023-01-15T14:30:51.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:30:51.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511325+0000", 2023-01-15T14:30:51.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027690400000000002 2023-01-15T14:30:51.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:51.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512092+0000", 2023-01-15T14:30:51.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00076720800000000002 2023-01-15T14:30:51.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:51.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512113+0000", 2023-01-15T14:30:51.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0917e-05 2023-01-15T14:30:51.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:51.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512439+0000", 2023-01-15T14:30:51.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032636299999999998 2023-01-15T14:30:51.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:51.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512871+0000", 2023-01-15T14:30:51.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000432087 2023-01-15T14:30:51.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.514165+0000", 2023-01-15T14:30:51.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001293726 2023-01-15T14:30:51.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714137+0000", 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19997229499999999 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714233+0000", 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6062000000000003e-05 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.650 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714446+0000", 2023-01-15T14:30:51.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212952 2023-01-15T14:30:51.650 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.650 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.650 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.650 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.650 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.650 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1481 3.13 3:cbfd2d71:::smithi09025264-37:head [write 1733272~707409] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:51.650 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:30:51.650 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 31.852320715000001, 2023-01-15T14:30:51.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.572522502, 2023-01-15T14:30:51.651 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.651 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:51.651 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.651 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.651 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.651 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1481 2023-01-15T14:30:51.651 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.651 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.651 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.651 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.651 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:30:51.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.652 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.652 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.652 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.652 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:30:51.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.652 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.652 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.652 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.652 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345868+0000", 2023-01-15T14:30:51.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:51.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346316+0000", 2023-01-15T14:30:51.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044813399999999999 2023-01-15T14:30:51.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346348+0000", 2023-01-15T14:30:51.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:30:51.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346362+0000", 2023-01-15T14:30:51.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.375e-05 2023-01-15T14:30:51.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512276+0000", 2023-01-15T14:30:51.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1659136829999999 2023-01-15T14:30:51.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:51.655 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512427+0000", 2023-01-15T14:30:51.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015088699999999999 2023-01-15T14:30:51.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:51.655 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714565+0000", 2023-01-15T14:30:51.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20213841599999999 2023-01-15T14:30:51.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:51.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714732+0000", 2023-01-15T14:30:51.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016676700000000001 2023-01-15T14:30:51.656 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.656 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.656 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:30:51.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714964+0000", 2023-01-15T14:30:51.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000231678 2023-01-15T14:30:51.656 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.656 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.656 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:51.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715499+0000", 2023-01-15T14:30:51.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053513300000000003 2023-01-15T14:30:51.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:51.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715523+0000", 2023-01-15T14:30:51.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4266000000000001e-05 2023-01-15T14:30:51.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715867+0000", 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034419399999999998 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.816932+0000", 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10106498999999999 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906374+0000", 2023-01-15T14:30:51.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089441442999999995 2023-01-15T14:30:51.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:51.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918050+0000", 2023-01-15T14:30:51.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.011676677999999999 2023-01-15T14:30:51.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.951 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:51.951 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918218+0000", 2023-01-15T14:30:51.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016802000000000001 2023-01-15T14:30:51.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:51.952 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918393+0000", 2023-01-15T14:30:51.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017447 2023-01-15T14:30:51.952 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.952 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:51.952 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:51.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1482 3.13 3:cbfd2d71:::smithi09025264-37:head [write 3195938~437497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:30:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 31.845884857000001, 2023-01-15T14:30:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5664780289999998, 2023-01-15T14:30:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:51.954 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1482 2023-01-15T14:30:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.955 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:30:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:51.957 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:30:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352275+0000", 2023-01-15T14:30:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999695 2023-01-15T14:30:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:51.958 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352644+0000", 2023-01-15T14:30:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036870300000000001 2023-01-15T14:30:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352676+0000", 2023-01-15T14:30:51.959 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1726000000000001e-05 2023-01-15T14:30:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352689+0000", 2023-01-15T14:30:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3617e-05 2023-01-15T14:30:51.960 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:52.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512602+0000", 2023-01-15T14:30:52.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1599126690000001 2023-01-15T14:30:52.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:52.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512839+0000", 2023-01-15T14:30:52.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023676900000000001 2023-01-15T14:30:52.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715649+0000", 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20281044600000001 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715777+0000", 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012738800000000001 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:30:52.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715995+0000", 2023-01-15T14:30:52.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000217858 2023-01-15T14:30:52.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:52.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716694+0000", 2023-01-15T14:30:52.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00069956700000000001 2023-01-15T14:30:52.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:52.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716709+0000", 2023-01-15T14:30:52.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4362000000000001e-05 2023-01-15T14:30:52.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:52.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716936+0000", 2023-01-15T14:30:52.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022744200000000001 2023-01-15T14:30:52.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:52.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.818876+0000", 2023-01-15T14:30:52.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101939635 2023-01-15T14:30:52.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:52.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906390+0000", 2023-01-15T14:30:52.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087513987000000001 2023-01-15T14:30:52.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:52.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918577+0000", 2023-01-15T14:30:52.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012187066 2023-01-15T14:30:52.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:52.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918697+0000", 2023-01-15T14:30:52.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120326 2023-01-15T14:30:52.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:52.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918784+0000", 2023-01-15T14:30:52.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7200999999999994e-05 2023-01-15T14:30:52.155 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:52.155 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:52.155 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:52.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.155 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1483 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3781575] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:52.155 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:30:52.155 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 31.845476704999999, 2023-01-15T14:30:52.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5663522429999999, 2023-01-15T14:30:52.155 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:52.155 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:52.156 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:52.156 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:52.156 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:52.156 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1483 2023-01-15T14:30:52.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.156 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:52.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:52.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:30:52.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:52.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.163 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:52.163 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:30:52.163 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:52.163 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.163 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.163 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:52.163 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352713+0000", 2023-01-15T14:30:52.163 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:52.163 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.163 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:52.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352720+0000", 2023-01-15T14:30:52.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7959999999999998e-06 2023-01-15T14:30:52.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:52.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352735+0000", 2023-01-15T14:30:52.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.455e-05 2023-01-15T14:30:52.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:52.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352741+0000", 2023-01-15T14:30:52.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8360000000000003e-06 2023-01-15T14:30:52.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:52.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512901+0000", 2023-01-15T14:30:52.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1601594980000001 2023-01-15T14:30:52.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:52.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.513067+0000", 2023-01-15T14:30:52.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016626 2023-01-15T14:30:52.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:52.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716977+0000", 2023-01-15T14:30:52.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20391002799999999 2023-01-15T14:30:52.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717146+0000", 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168851 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717467+0000", 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032135600000000002 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:52.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717800+0000", 2023-01-15T14:30:52.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033217400000000002 2023-01-15T14:30:52.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:52.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717822+0000", 2023-01-15T14:30:52.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2308999999999999e-05 2023-01-15T14:30:52.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:52.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717970+0000", 2023-01-15T14:30:52.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014788099999999999 2023-01-15T14:30:52.169 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.169 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.169 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:52.169 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.819135+0000", 2023-01-15T14:30:52.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116478 2023-01-15T14:30:52.169 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.169 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.169 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:52.169 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906408+0000", 2023-01-15T14:30:52.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087272902999999999 2023-01-15T14:30:52.170 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.170 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.170 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:52.170 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918887+0000", 2023-01-15T14:30:52.170 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012479965000000001 2023-01-15T14:30:52.170 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.170 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.170 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:52.170 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918959+0000", 2023-01-15T14:30:52.170 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1978999999999996e-05 2023-01-15T14:30:52.170 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.171 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.171 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:52.171 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.919067+0000", 2023-01-15T14:30:52.171 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107083 2023-01-15T14:30:52.171 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:52.171 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:52.171 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:52.171 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.171 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.171 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1485 3.d 3:bc1c6b44:::smithi09025264-23:head [write 598382~780511,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:52.171 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:30:52.172 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 31.838006271000001, 2023-01-15T14:30:52.172 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5474121890000001, 2023-01-15T14:30:52.172 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:52.172 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:52.172 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:52.172 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:52.172 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:52.172 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1485 2023-01-15T14:30:52.172 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.172 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:52.172 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.173 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:52.173 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:30:52.173 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:52.173 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.173 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.173 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:52.173 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:30:52.173 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:52.173 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.174 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.174 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:52.174 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360168+0000", 2023-01-15T14:30:52.174 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999828 2023-01-15T14:30:52.174 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.174 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.174 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:52.174 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360790+0000", 2023-01-15T14:30:52.174 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062269099999999998 2023-01-15T14:30:52.174 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.174 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.175 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:52.175 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360822+0000", 2023-01-15T14:30:52.175 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1903000000000003e-05 2023-01-15T14:30:52.175 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.175 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.175 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:52.175 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360836+0000", 2023-01-15T14:30:52.175 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3380000000000001e-05 2023-01-15T14:30:52.175 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.176 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:52.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096664+0000", 2023-01-15T14:30:52.176 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7358282320000002 2023-01-15T14:30:52.176 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.176 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:30:52.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096683+0000", 2023-01-15T14:30:52.176 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9063999999999999e-05 2023-01-15T14:30:52.176 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.176 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:52.177 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704561+0000", 2023-01-15T14:30:52.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60787808399999999 2023-01-15T14:30:52.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:52.177 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704846+0000", 2023-01-15T14:30:52.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028527400000000002 2023-01-15T14:30:52.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:52.178 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705135+0000", 2023-01-15T14:30:52.178 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028875400000000001 2023-01-15T14:30:52.178 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.178 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.178 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:52.178 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705735+0000", 2023-01-15T14:30:52.178 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059956899999999999 2023-01-15T14:30:52.178 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.178 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.178 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:52.178 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705761+0000", 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.706129+0000", 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036822299999999999 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906331+0000", 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200202091 2023-01-15T14:30:52.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.180 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.180 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:52.180 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907111+0000", 2023-01-15T14:30:52.180 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077990799999999995 2023-01-15T14:30:52.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.180 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.180 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:52.180 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907379+0000", 2023-01-15T14:30:52.180 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026804800000000001 2023-01-15T14:30:52.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.180 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:52.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907529+0000", 2023-01-15T14:30:52.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015022700000000001 2023-01-15T14:30:52.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:52.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907597+0000", 2023-01-15T14:30:52.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7785000000000002e-05 2023-01-15T14:30:52.181 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:52.181 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:52.181 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1486 3.d 3:bc1c6b44:::smithi09025264-23:head [write 2075401~451067] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 31.831610222999998, 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6461318330000001, 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:52.182 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1486 2023-01-15T14:30:52.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.183 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:52.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:52.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:30:52.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:52.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:52.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:30:52.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:52.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:52.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366579+0000", 2023-01-15T14:30:52.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:52.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.362 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.362 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:52.362 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366845+0000", 2023-01-15T14:30:52.362 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026610499999999999 2023-01-15T14:30:52.362 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.362 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.362 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:52.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366878+0000", 2023-01-15T14:30:52.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2750000000000003e-05 2023-01-15T14:30:52.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:52.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.467009+0000", 2023-01-15T14:30:52.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100131071 2023-01-15T14:30:52.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:52.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096793+0000", 2023-01-15T14:30:52.364 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6297846869999999 2023-01-15T14:30:52.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.364 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:30:52.364 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096806+0000", 2023-01-15T14:30:52.364 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3077e-05 2023-01-15T14:30:52.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.364 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:52.364 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907957+0000", 2023-01-15T14:30:52.364 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81115055400000002 2023-01-15T14:30:52.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:52.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908229+0000", 2023-01-15T14:30:52.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027187499999999998 2023-01-15T14:30:52.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:30:52.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908496+0000", 2023-01-15T14:30:52.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026754200000000001 2023-01-15T14:30:52.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:52.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909025+0000", 2023-01-15T14:30:52.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052865200000000001 2023-01-15T14:30:52.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:52.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909046+0000", 2023-01-15T14:30:52.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0713e-05 2023-01-15T14:30:52.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:52.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909294+0000", 2023-01-15T14:30:52.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024864899999999998 2023-01-15T14:30:52.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:52.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.911607+0000", 2023-01-15T14:30:52.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023128939999999998 2023-01-15T14:30:52.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:52.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012076+0000", 2023-01-15T14:30:52.368 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100468293 2023-01-15T14:30:52.368 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:52.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012426+0000", 2023-01-15T14:30:52.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035044299999999998 2023-01-15T14:30:52.369 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:52.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012532+0000", 2023-01-15T14:30:52.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106142 2023-01-15T14:30:52.369 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:52.370 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012713+0000", 2023-01-15T14:30:52.370 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018048599999999999 2023-01-15T14:30:52.370 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:52.370 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:52.370 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:52.370 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.370 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.370 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1496 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:52.370 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:30:52.370 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 31.461967374, 2023-01-15T14:30:52.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870616037, 2023-01-15T14:30:52.371 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:52.371 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:52.371 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:52.371 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:52.371 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:52.371 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1496 2023-01-15T14:30:52.371 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.371 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:52.371 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.371 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:52.372 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:30:52.372 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:52.372 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.372 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.372 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:52.372 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:30:52.372 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:52.372 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736220+0000", 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736238+0000", 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7156e-05 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:52.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736275+0000", 2023-01-15T14:30:52.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7876000000000001e-05 2023-01-15T14:30:52.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:52.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736290+0000", 2023-01-15T14:30:52.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4449e-05 2023-01-15T14:30:52.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:52.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602217+0000", 2023-01-15T14:30:52.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8659270299999999 2023-01-15T14:30:52.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:52.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602468+0000", 2023-01-15T14:30:52.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000251315 2023-01-15T14:30:52.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:52.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.606840+0000", 2023-01-15T14:30:52.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0043715009999999999 2023-01-15T14:30:52.376 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:52.376 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:52.376 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:52.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:52.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:52.376 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1497 3.11 3:8889e450:::smithi09025264-32:head [read 0~2991646,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:52.376 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:30:52.376 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 31.461879920000001, 2023-01-15T14:30:52.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738359249999998, 2023-01-15T14:30:52.376 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:52.376 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:53.799 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:53.799 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:53.800 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:53.800 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1497 2023-01-15T14:30:53.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:53.800 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:53.800 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:53.800 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:53.800 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:30:53.800 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:53.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:53.800 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:54.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:30:54.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:54.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:54.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736310+0000", 2023-01-15T14:30:54.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:54.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:54.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736317+0000", 2023-01-15T14:30:54.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7240000000000003e-06 2023-01-15T14:30:54.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:54.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736334+0000", 2023-01-15T14:30:54.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6878999999999999e-05 2023-01-15T14:30:54.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736341+0000", 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0129999999999996e-06 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607070+0000", 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870728873 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:54.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607226+0000", 2023-01-15T14:30:54.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015595 2023-01-15T14:30:54.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.022 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:54.022 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610147+0000", 2023-01-15T14:30:54.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.002921416 2023-01-15T14:30:54.022 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:54.022 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:54.022 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:54.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.023 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1498 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:30:54.023 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:30:54.023 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 31.461839814000001, 2023-01-15T14:30:54.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8752667839999999, 2023-01-15T14:30:54.023 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:54.023 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:54.024 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:54.024 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:54.025 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:54.025 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1498 2023-01-15T14:30:54.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.025 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:54.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:54.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:30:54.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:54.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:54.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:30:54.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:54.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:54.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736350+0000", 2023-01-15T14:30:54.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:54.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:54.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736355+0000", 2023-01-15T14:30:54.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4900000000000002e-06 2023-01-15T14:30:54.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:54.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736363+0000", 2023-01-15T14:30:54.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2770000000000005e-06 2023-01-15T14:30:54.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:54.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736369+0000", 2023-01-15T14:30:54.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7400000000000001e-06 2023-01-15T14:30:54.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:54.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610234+0000", 2023-01-15T14:30:54.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738651439999998 2023-01-15T14:30:54.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:54.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610402+0000", 2023-01-15T14:30:54.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168283 2023-01-15T14:30:54.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:54.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:54.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:54.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.611618+0000", 2023-01-15T14:30:54.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001215647 2023-01-15T14:30:54.029 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:54.029 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:54.030 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:54.030 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:54.030 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:54.030 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:54.031 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:30:54.035 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:30:54.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:30:54.037 INFO:tasks.rados.rados.0.smithi090.stdout:659: expect (ObjNum 232 snap 52 seq_num 232) 2023-01-15T14:30:54.038 INFO:tasks.rados.rados.0.smithi090.stdout:658: finishing copy_from to smithi09025264-1 2023-01-15T14:30:54.038 INFO:tasks.rados.rados.0.smithi090.stdout:658: got expected ENOENT (src dne) 2023-01-15T14:30:54.038 INFO:tasks.rados.rados.0.smithi090.stdout:658: finishing copy_from racing read to smithi09025264-1 2023-01-15T14:30:54.038 INFO:tasks.rados.rados.0.smithi090.stdout:658: done (1 left) 2023-01-15T14:30:54.038 INFO:tasks.rados.rados.0.smithi090.stdout:659: done (0 left) 2023-01-15T14:30:54.039 INFO:tasks.rados.rados.0.smithi090.stdout:660: done (0 left) 2023-01-15T14:30:54.039 INFO:tasks.rados.rados.0.smithi090.stdout:661: snap_create 2023-01-15T14:30:54.039 INFO:tasks.rados.rados.0.smithi090.stdout:661: done (0 left) 2023-01-15T14:30:54.039 INFO:tasks.rados.rados.0.smithi090.stdout:662: read oid 27 snap -1 2023-01-15T14:30:54.039 INFO:tasks.rados.rados.0.smithi090.stdout:662: expect (ObjNum 101 snap 21 seq_num 101) 2023-01-15T14:30:54.040 INFO:tasks.rados.rados.0.smithi090.stdout:663: write oid 49 current snap is 65 2023-01-15T14:30:54.040 INFO:tasks.rados.rados.0.smithi090.stdout:663: seq_num 263 ranges {420241=546489,1373823=544414} 2023-01-15T14:30:54.040 INFO:tasks.rados.rados.0.smithi090.stdout:663: writing smithi09025264-49 from 420241 to 966730 tid 1 2023-01-15T14:30:54.041 INFO:tasks.rados.rados.0.smithi090.stdout:663: writing smithi09025264-49 from 1373823 to 1918237 tid 2 2023-01-15T14:30:54.041 INFO:tasks.rados.rados.0.smithi090.stdout:664: delete oid 10 current snap is 65 2023-01-15T14:30:54.043 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:30:54.028+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:30:54.107 INFO:tasks.rados.rados.0.smithi090.stdout:664: done (2 left) 2023-01-15T14:30:54.233 INFO:tasks.rados.rados.0.smithi090.stdout:665: rollback oid 15 current snap is 65 2023-01-15T14:30:54.233 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 15 to 62 2023-01-15T14:30:54.233 INFO:tasks.rados.rados.0.smithi090.stdout:666: snap_remove snap 59 2023-01-15T14:30:54.235 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:30:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:30:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1768 3.18 e167/164)", 2023-01-15T14:30:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:54.106700+0000", 2023-01-15T14:30:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.055471621999999998, 2023-01-15T14:30:54.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055505116, 2023-01-15T14:30:54.238 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:54.238 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:54.238 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.106700+0000", 2023-01-15T14:30:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.106700+0000", 2023-01-15T14:30:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:54.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.106699+0000", 2023-01-15T14:30:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.106708+0000", 2023-01-15T14:30:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8570000000000001e-06 2023-01-15T14:30:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.106722+0000", 2023-01-15T14:30:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3560000000000001e-05 2023-01-15T14:30:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.106730+0000", 2023-01-15T14:30:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8480000000000001e-06 2023-01-15T14:30:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:54.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(3.0 167/164 [PushReplyOp(3:03b54904:::smithi09025264-12:head)])", 2023-01-15T14:30:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:54.112448+0000", 2023-01-15T14:30:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.049723986999999997, 2023-01-15T14:30:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.049844687999999998, 2023-01-15T14:30:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:30:54.242 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:30:54.244 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:30:54.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:30:54.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.112448+0000", 2023-01-15T14:30:54.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:54.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:54.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.112448+0000", 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.112446+0000", 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:30:54.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.112457+0000", 2023-01-15T14:30:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1009000000000001e-05 2023-01-15T14:30:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:30:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.112479+0000", 2023-01-15T14:30:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2084999999999999e-05 2023-01-15T14:30:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:30:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:30:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:30:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:54.112491+0000", 2023-01-15T14:30:54.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1664e-05 2023-01-15T14:30:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:30:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:30:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:30:54.248 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 2 2023-01-15T14:30:54.248 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:30:54.249 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:30:54.249 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:54.249 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:54.250 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:54.250 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:54.232+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:54.250 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:54.268 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:54.424 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:54.424 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:54.424 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:54.425 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:30:54.425 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:30:54.425 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:30:54.425 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:30:54.425 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:30:54.426 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:30:54.426 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:30:54.426 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:30:54.427 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:54.427 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:54.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:30:54.427 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:54.420+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:54.525 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:54.520+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:54.630 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:54.631 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:54.632 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:30:54.633 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:54.627+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:54.730 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:54.727+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:54.833 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:54.833 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:54.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:30:54.834 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:54.831+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:54.934 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:54.932+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:55.037 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:30:55.037 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:30:55.037 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2023-01-15T14:30:55.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:55.039 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:55.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:30:55.040 INFO:tasks.rados.rados.0.smithi090.stdout:666: done (3 left) 2023-01-15T14:30:55.040 INFO:tasks.rados.rados.0.smithi090.stdout:667: read oid 33 snap -1 2023-01-15T14:30:55.040 INFO:tasks.rados.rados.0.smithi090.stdout:667: expect (ObjNum 252 snap 60 seq_num 252) 2023-01-15T14:30:55.040 INFO:tasks.rados.rados.0.smithi090.stdout:668: snap_remove snap 61 2023-01-15T14:30:55.041 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:55.035+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:55.139 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:55.136+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:55.206 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:55.206 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:55.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:30:55.241 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:55.239+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:55.358 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:55.358 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:55.359 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:30:55.359 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:55.355+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:55.384 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:30:55.521 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:30:55.522 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2023-01-15T14:30:54.985352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":131489322,"num_objects":55,"num_object_clones":17,"num_object_copies":165,"num_objects_missing_on_primary":5,"num_objects_missing":5,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":3,"num_read":619,"num_read_kb":468669,"num_write":629,"num_write_kb":201211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":242810538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4890,"ondisk_log_size":4890,"up":135,"acting":129,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":5323520,"kb_used_data":556912416,"kb_used_omap":18428,"kb_used_meta":0,"kb_avail":556872760,"statfs":{"total":575688990720,"available":570237706240,"internally_reserved":40607744,"allocated":570278313984,"data_stored":570278313984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18870966,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-328168,"num_objects":0,"num_object_clones":1,"num_object_copies":0,"num_objects_missing_on_primary":5,"num_objects_missing":5,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-59,"num_read_kb":-63155,"num_write":-36,"num_write_kb":-13792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-5277921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.066434"},"pg_stats":[{"pgid":"3.23","version":"149'233","reported_seq":736,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-01-15T14:30:50.020753+0000","last_change":"2023-01-15T14:30:47.017286+0000","last_active":"2023-01-15T14:30:50.020753+0000","last_peered":"2023-01-15T14:30:50.020753+0000","last_clean":"2023-01-15T14:30:50.020753+0000","last_became_active":"2023-01-15T14:30:47.012501+0000","last_became_peered":"2023-01-15T14:30:47.012501+0000","last_unstale":"2023-01-15T14:30:50.020753+0000","last_undegraded":"2023-01-15T14:30:50.020753+0000","last_fullsized":"2023-01-15T14:30:50.020753+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037266930000000001,"stat_sum":{"num_bytes":7604272,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":47,"num_read_kb":43637,"num_write":37,"num_write_kb":13160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2469494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.22","version":"155'101","reported_seq":663,"reported_epoch":163,"state":"peering","last_fresh":"2023-01-15T14:30:49.915995+0000","last_change":"2023-01-15T14:30:49.916862+0000","last_active":"2023-01-15T14:30:49.915892+0000","last_peered":"2023-01-15T14:30:49.749222+0000","last_clean":"2023-01-15T14:30:45.607652+0000","last_became_active":"2023-01-15T14:30:47.984777+0000","last_became_peered":"2023-01-15T14:30:47.984777+0000","last_unstale":"2023-01-15T14:30:49.915995+0000","last_undegraded":"2023-01-15T14:30:49.915995+0000","last_fullsized":"2023-01-15T14:30:49.915995+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.101204921,"stat_sum":{"num_bytes":1084068,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":5437114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.21","version":"155'121","reported_seq":591,"reported_epoch":164,"state":"peering","last_fresh":"2023-01-15T14:30:50.929305+0000","last_change":"2023-01-15T14:30:50.929305+0000","last_active":"2023-01-15T14:30:46.877177+0000","last_peered":"2023-01-15T14:30:45.882272+0000","last_clean":"2023-01-15T14:30:45.882272+0000","last_became_active":"2023-01-15T14:30:45.090193+0000","last_became_peered":"2023-01-15T14:30:45.090193+0000","last_unstale":"2023-01-15T14:30:50.929305+0000","last_undegraded":"2023-01-15T14:30:50.929305+0000","last_fullsized":"2023-01-15T14:30:50.929305+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'111","last_scrub_stamp":"2023-01-15T14:30:23.145968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.145968+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.28528039199999999,"objects_trimmed":0,"snaptrim_duration":0.10264011100000001,"stat_sum":{"num_bytes":1899191,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.20","version":"137'228","reported_seq":794,"reported_epoch":163,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:30:50.702467+0000","last_change":"2023-01-15T14:30:50.486477+0000","last_active":"2023-01-15T14:30:50.702467+0000","last_peered":"2023-01-15T14:30:50.702467+0000","last_clean":"2023-01-15T14:30:24.089045+0000","last_became_active":"2023-01-15T14:30:48.085229+0000","last_became_peered":"2023-01-15T14:30:48.085229+0000","last_unstale":"2023-01-15T14:30:50.702467+0000","last_undegraded":"2023-01-15T14:30:50.702467+0000","last_fullsized":"2023-01-15T14:30:47.776598+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3479049,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,2],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.1f","version":"155'145","reported_seq":694,"reported_epoch":163,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:30:50.381574+0000","last_change":"2023-01-15T14:30:50.381574+0000","last_active":"2023-01-15T14:30:50.381574+0000","last_peered":"2023-01-15T14:30:50.381574+0000","last_clean":"2023-01-15T14:30:42.438205+0000","last_became_active":"2023-01-15T14:30:47.980571+0000","last_became_peered":"2023-01-15T14:30:47.980571+0000","last_unstale":"2023-01-15T14:30:50.381574+0000","last_undegraded":"2023-01-15T14:30:50.381574+0000","last_fullsized":"2023-01-15T14:30:47.779696+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6227010,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":36,"num_read_kb":33588,"num_write":38,"num_write_kb":12831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":27353426,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,4],"avail_no_missing":["0","2","3","4"],"object_location_counts":[{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.1e","version":"139'38","reported_seq":434,"reported_epoch":163,"state":"peering","last_fresh":"2023-01-15T14:30:49.920445+0000","last_change":"2023-01-15T14:30:49.920445+0000","last_active":"2023-01-15T14:30:49.916977+0000","last_peered":"2023-01-15T14:30:49.037511+0000","last_clean":"2023-01-15T14:30:49.037511+0000","last_became_active":"2023-01-15T14:30:46.882632+0000","last_became_peered":"2023-01-15T14:30:46.882632+0000","last_unstale":"2023-01-15T14:30:49.920445+0000","last_undegraded":"2023-01-15T14:30:49.920445+0000","last_fullsized":"2023-01-15T14:30:49.920445+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'37","last_scrub_stamp":"2023-01-15T14:30:25.793922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:25.793922+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":11530,"num_write":6,"num_write_kb":4046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1487420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.1d","version":"155'154","reported_seq":678,"reported_epoch":164,"state":"remapped+peering","last_fresh":"2023-01-15T14:30:50.919452+0000","last_change":"2023-01-15T14:30:50.919790+0000","last_active":"2023-01-15T14:30:49.924214+0000","last_peered":"2023-01-15T14:30:49.011988+0000","last_clean":"2023-01-15T14:30:49.011988+0000","last_became_active":"2023-01-15T14:30:46.874479+0000","last_became_peered":"2023-01-15T14:30:46.874479+0000","last_unstale":"2023-01-15T14:30:50.919452+0000","last_undegraded":"2023-01-15T14:30:50.919452+0000","last_fullsized":"2023-01-15T14:30:50.919452+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5345123,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":63155,"num_write":36,"num_write_kb":13792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":10058907,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,7],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":25,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-01-15T14:30:51.235412+0000","last_change":"2023-01-15T14:30:51.235412+0000","last_active":"2023-01-15T14:30:51.235412+0000","last_peered":"2023-01-15T14:30:51.235412+0000","last_clean":"2023-01-15T14:30:51.235412+0000","last_became_active":"2023-01-15T14:30:51.234588+0000","last_became_peered":"2023-01-15T14:30:51.234588+0000","last_unstale":"2023-01-15T14:30:51.235412+0000","last_undegraded":"2023-01-15T14:30:51.235412+0000","last_fullsized":"2023-01-15T14:30:51.235412+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":10,"num_write_kb":2522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1159667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.1b","version":"155'53","reported_seq":478,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-01-15T14:30:50.020651+0000","last_change":"2023-01-15T14:30:48.314187+0000","last_active":"2023-01-15T14:30:50.020651+0000","last_peered":"2023-01-15T14:30:50.020651+0000","last_clean":"2023-01-15T14:30:50.020651+0000","last_became_active":"2023-01-15T14:30:47.013378+0000","last_became_peered":"2023-01-15T14:30:47.013378+0000","last_unstale":"2023-01-15T14:30:50.020651+0000","last_undegraded":"2023-01-15T14:30:50.020651+0000","last_fullsized":"2023-01-15T14:30:50.020651+0000","mapping_epoch":159,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1731121,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":6694,"num_write":12,"num_write_kb":5635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3462242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.1a","version":"158'182","reported_seq":686,"reported_epoch":164,"state":"remapped+peering","last_fresh":"2023-01-15T14:30:50.923729+0000","last_change":"2023-01-15T14:30:50.923918+0000","last_active":"2023-01-15T14:30:49.920064+0000","last_peered":"2023-01-15T14:30:49.363712+0000","last_clean":"2023-01-15T14:30:45.901798+0000","last_became_active":"2023-01-15T14:30:48.076511+0000","last_became_peered":"2023-01-15T14:30:48.076511+0000","last_unstale":"2023-01-15T14:30:50.923729+0000","last_undegraded":"2023-01-15T14:30:50.923729+0000","last_fullsized":"2023-01-15T14:30:50.923729+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8676157,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":23721,"num_write":54,"num_write_kb":24559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":36386386,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"157'169","reported_seq":688,"reported_epoch":164,"state":"remapped+peering","last_fresh":"2023-01-15T14:30:50.928108+0000","last_change":"2023-01-15T14:30:50.928108+0000","last_active":"2023-01-15T14:30:50.927763+0000","last_peered":"2023-01-15T14:30:44.093474+0000","last_clean":"2023-01-15T14:30:44.093474+0000","last_became_active":"2023-01-15T14:30:42.081286+0000","last_became_peered":"2023-01-15T14:30:42.081286+0000","last_unstale":"2023-01-15T14:30:50.928108+0000","last_undegraded":"2023-01-15T14:30:50.928108+0000","last_fullsized":"2023-01-15T14:30:50.928108+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'157","last_scrub_stamp":"2023-01-15T14:30:26.792736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:26.792736+0000","objects_scrubbed":4,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:27.117","scrub_duration":0.61998826900000004,"objects_trimmed":0,"snaptrim_duration":0.73421570899999999,"stat_sum":{"num_bytes":12737738,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":15024,"num_write":6,"num_write_kb":853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15210335,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.18","version":"160'208","reported_seq":789,"reported_epoch":163,"state":"peering","last_fresh":"2023-01-15T14:30:49.917386+0000","last_change":"2023-01-15T14:30:49.922302+0000","last_active":"2023-01-15T14:30:49.917291+0000","last_peered":"2023-01-15T14:30:49.037259+0000","last_clean":"2023-01-15T14:30:49.037259+0000","last_became_active":"2023-01-15T14:30:46.879013+0000","last_became_peered":"2023-01-15T14:30:46.879013+0000","last_unstale":"2023-01-15T14:30:49.917386+0000","last_undegraded":"2023-01-15T14:30:49.917386+0000","last_fullsized":"2023-01-15T14:30:49.917386+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":160,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:29.922","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.61292552199999994,"stat_sum":{"num_bytes":5495889,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":18214,"num_write":6,"num_write_kb":5071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":12958197,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.17","version":"155'77","reported_seq":523,"reported_epoch":163,"state":"peering","last_fresh":"2023-01-15T14:30:49.931337+0000","last_change":"2023-01-15T14:30:49.931655+0000","last_active":"2023-01-15T14:30:49.931303+0000","last_peered":"2023-01-15T14:30:49.126287+0000","last_clean":"2023-01-15T14:30:49.126287+0000","last_became_active":"2023-01-15T14:30:33.899849+0000","last_became_peered":"2023-01-15T14:30:33.899849+0000","last_unstale":"2023-01-15T14:30:49.931337+0000","last_undegraded":"2023-01-15T14:30:49.931337+0000","last_fullsized":"2023-01-15T14:30:49.931337+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'65","last_scrub_stamp":"2023-01-15T14:30:24.156798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.156798+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:37.577","scrub_duration":0.30368223799999999,"objects_trimmed":0,"snaptrim_duration":0.101041797,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3949645,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.16","version":"155'67","reported_seq":17,"reported_epoch":164,"state":"active+recovering","last_fresh":"2023-01-15T14:30:51.532175+0000","last_change":"2023-01-15T14:30:51.386755+0000","last_active":"2023-01-15T14:30:51.532175+0000","last_peered":"2023-01-15T14:30:51.532175+0000","last_clean":"2023-01-15T14:30:41.774899+0000","last_became_active":"2023-01-15T14:30:51.036725+0000","last_became_peered":"2023-01-15T14:30:51.036725+0000","last_unstale":"2023-01-15T14:30:51.532175+0000","last_undegraded":"2023-01-15T14:30:51.532175+0000","last_fullsized":"2023-01-15T14:30:51.532175+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":5,"last_scrub":"118'59","last_scrub_stamp":"2023-01-15T14:30:25.417530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:25.417530+0000","objects_scrubbed":2,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.53679152699999999,"stat_sum":{"num_bytes":2866164,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":5,"num_objects_missing":5,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":8569,"num_write":22,"num_write_kb":7023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":9462341,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2,3","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":478,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-01-15T14:30:49.037433+0000","last_change":"2023-01-15T14:30:43.893366+0000","last_active":"2023-01-15T14:30:49.037433+0000","last_peered":"2023-01-15T14:30:49.037433+0000","last_clean":"2023-01-15T14:30:49.037433+0000","last_became_active":"2023-01-15T14:30:03.169969+0000","last_became_peered":"2023-01-15T14:30:03.169969+0000","last_unstale":"2023-01-15T14:30:49.037433+0000","last_undegraded":"2023-01-15T14:30:49.037433+0000","last_fullsized":"2023-01-15T14:30:49.037433+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:30:24.721082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.721082+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:48.096","scrub_duration":0.109000768,"objects_trimmed":0,"snaptrim_duration":0.003449677,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.14","version":"155'135","reported_seq":631,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-01-15T14:30:49.923569+0000","last_change":"2023-01-15T14:30:43.781003+0000","last_active":"2023-01-15T14:30:49.923569+0000","last_peered":"2023-01-15T14:30:49.923569+0000","last_clean":"2023-01-15T14:30:49.923569+0000","last_became_active":"2023-01-15T14:30:16.972329+0000","last_became_peered":"2023-01-15T14:30:16.972329+0000","last_unstale":"2023-01-15T14:30:49.923569+0000","last_undegraded":"2023-01-15T14:30:49.923569+0000","last_fullsized":"2023-01-15T14:30:49.923569+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:31.572","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.0013454160000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":11234,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":189,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-01-15T14:30:51.228396+0000","last_change":"2023-01-15T14:30:51.228396+0000","last_active":"2023-01-15T14:30:51.228396+0000","last_peered":"2023-01-15T14:30:51.228396+0000","last_clean":"2023-01-15T14:30:51.228396+0000","last_became_active":"2023-01-15T14:30:51.127504+0000","last_became_peered":"2023-01-15T14:30:51.127504+0000","last_unstale":"2023-01-15T14:30:51.228396+0000","last_undegraded":"2023-01-15T14:30:51.228396+0000","last_fullsized":"2023-01-15T14:30:51.228396+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.671761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.671761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":506,"reported_epoch":163,"state":"peering","last_fresh":"2023-01-15T14:30:49.928124+0000","last_change":"2023-01-15T14:30:49.928378+0000","last_active":"2023-01-15T14:30:49.928087+0000","last_peered":"2023-01-15T14:30:49.126586+0000","last_clean":"2023-01-15T14:30:49.126586+0000","last_became_active":"2023-01-15T14:30:33.993887+0000","last_became_peered":"2023-01-15T14:30:33.993887+0000","last_unstale":"2023-01-15T14:30:49.928124+0000","last_undegraded":"2023-01-15T14:30:49.928124+0000","last_fullsized":"2023-01-15T14:30:49.928124+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:30:21.421951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:21.421951+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:31.852","scrub_duration":0.569620185,"objects_trimmed":0,"snaptrim_duration":0.100593706,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,4],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-01-15T14:30:51.035957+0000","last_change":"2023-01-15T14:30:51.035825+0000","last_active":"2023-01-15T14:30:51.035957+0000","last_peered":"2023-01-15T14:30:51.035957+0000","last_clean":"2023-01-15T14:30:51.035957+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:30:51.035957+0000","last_undegraded":"2023-01-15T14:30:51.035957+0000","last_fullsized":"2023-01-15T14:30:51.035957+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"125'75","reported_seq":522,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-01-15T14:30:49.038528+0000","last_change":"2023-01-15T14:30:43.893941+0000","last_active":"2023-01-15T14:30:49.038528+0000","last_peered":"2023-01-15T14:30:49.038528+0000","last_clean":"2023-01-15T14:30:49.038528+0000","last_became_active":"2023-01-15T14:30:03.070234+0000","last_became_peered":"2023-01-15T14:30:03.070234+0000","last_unstale":"2023-01-15T14:30:49.038528+0000","last_undegraded":"2023-01-15T14:30:49.038528+0000","last_fullsized":"2023-01-15T14:30:49.038528+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:25.827","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.003591768,"stat_sum":{"num_bytes":3462242,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":168,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-01-15T14:30:50.927194+0000","last_change":"2023-01-15T14:30:40.671483+0000","last_active":"2023-01-15T14:30:50.927194+0000","last_peered":"2023-01-15T14:30:50.927194+0000","last_clean":"2023-01-15T14:30:50.927194+0000","last_became_active":"2023-01-15T14:28:13.317872+0000","last_became_peered":"2023-01-15T14:28:13.317872+0000","last_unstale":"2023-01-15T14:30:50.927194+0000","last_undegraded":"2023-01-15T14:30:50.927194+0000","last_fullsized":"2023-01-15T14:30:50.927194+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:25.664616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:25.664616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40398350300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"156'185","reported_seq":717,"reported_epoch":158,"state":"active+clean","last_fresh":"2023-01-15T14:30:44.782418+0000","last_change":"2023-01-15T14:30:44.560544+0000","last_active":"2023-01-15T14:30:44.782418+0000","last_peered":"2023-01-15T14:30:44.782418+0000","last_clean":"2023-01-15T14:30:44.782418+0000","last_became_active":"2023-01-15T14:30:40.015688+0000","last_became_peered":"2023-01-15T14:30:40.015688+0000","last_unstale":"2023-01-15T14:30:44.782418+0000","last_undegraded":"2023-01-15T14:30:44.782418+0000","last_fullsized":"2023-01-15T14:30:44.782418+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:37.418","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.76729663400000003,"stat_sum":{"num_bytes":4336272,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":23,"num_write_kb":5401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":17942309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-01-15T14:30:51.238223+0000","last_change":"2023-01-15T14:30:51.237821+0000","last_active":"2023-01-15T14:30:51.238223+0000","last_peered":"2023-01-15T14:30:51.238223+0000","last_clean":"2023-01-15T14:30:51.238223+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:30:51.238223+0000","last_undegraded":"2023-01-15T14:30:51.238223+0000","last_fullsized":"2023-01-15T14:30:51.238223+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"141'59","reported_seq":524,"reported_epoch":163,"state":"peering","last_fresh":"2023-01-15T14:30:49.921968+0000","last_change":"2023-01-15T14:30:49.922261+0000","last_active":"2023-01-15T14:30:49.921875+0000","last_peered":"2023-01-15T14:30:49.037695+0000","last_clean":"2023-01-15T14:30:49.037695+0000","last_became_active":"2023-01-15T14:30:33.898976+0000","last_became_peered":"2023-01-15T14:30:33.898976+0000","last_unstale":"2023-01-15T14:30:49.921968+0000","last_undegraded":"2023-01-15T14:30:49.921968+0000","last_fullsized":"2023-01-15T14:30:49.921968+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":148,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:31:12.510","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.101457347,"stat_sum":{"num_bytes":2067294,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":30038,"num_write":17,"num_write_kb":7307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":13739656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.3","version":"160'239","reported_seq":790,"reported_epoch":164,"state":"remapped+peering","last_fresh":"2023-01-15T14:30:51.022193+0000","last_change":"2023-01-15T14:30:51.022523+0000","last_active":"2023-01-15T14:30:49.919203+0000","last_peered":"2023-01-15T14:30:48.912268+0000","last_clean":"2023-01-15T14:30:48.912268+0000","last_became_active":"2023-01-15T14:30:44.886874+0000","last_became_peered":"2023-01-15T14:30:44.886874+0000","last_unstale":"2023-01-15T14:30:51.022193+0000","last_undegraded":"2023-01-15T14:30:51.022193+0000","last_fullsized":"2023-01-15T14:30:51.022193+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:31:11.242","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.60962510999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":46,"num_read_kb":43637,"num_write":36,"num_write_kb":13160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7897259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-01-15T14:30:51.337760+0000","last_change":"2023-01-15T14:30:51.337447+0000","last_active":"2023-01-15T14:30:51.337760+0000","last_peered":"2023-01-15T14:30:51.337760+0000","last_clean":"2023-01-15T14:30:51.337760+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:30:51.337760+0000","last_undegraded":"2023-01-15T14:30:51.337760+0000","last_fullsized":"2023-01-15T14:30:51.337760+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:14.703035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:14.703035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"137'228","reported_seq":876,"reported_epoch":159,"state":"active+clean+scrubbing","last_fresh":"2023-01-15T14:30:45.900756+0000","last_change":"2023-01-15T14:30:46.059998+0000","last_active":"2023-01-15T14:30:45.900756+0000","last_peered":"2023-01-15T14:30:45.900756+0000","last_clean":"2023-01-15T14:30:45.900756+0000","last_became_active":"2023-01-15T14:30:45.085956+0000","last_became_peered":"2023-01-15T14:30:45.085956+0000","last_unstale":"2023-01-15T14:30:45.900756+0000","last_undegraded":"2023-01-15T14:30:45.900756+0000","last_fullsized":"2023-01-15T14:30:45.900756+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.0061536079999999996,"stat_sum":{"num_bytes":3479049,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3479049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":165,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-01-15T14:30:50.126795+0000","last_change":"2023-01-15T14:29:47.854985+0000","last_active":"2023-01-15T14:30:50.126795+0000","last_peered":"2023-01-15T14:30:50.126795+0000","last_clean":"2023-01-15T14:30:50.126795+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:30:50.126795+0000","last_undegraded":"2023-01-15T14:30:50.126795+0000","last_fullsized":"2023-01-15T14:30:50.126795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:47.854866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:47.854866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:56.359","scrub_duration":0.63763962500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":593,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-01-15T14:30:49.125686+0000","last_change":"2023-01-15T14:30:47.349738+0000","last_active":"2023-01-15T14:30:49.125686+0000","last_peered":"2023-01-15T14:30:49.125686+0000","last_clean":"2023-01-15T14:30:49.125686+0000","last_became_active":"2023-01-15T14:30:44.984034+0000","last_became_peered":"2023-01-15T14:30:44.984034+0000","last_unstale":"2023-01-15T14:30:49.125686+0000","last_undegraded":"2023-01-15T14:30:49.125686+0000","last_fullsized":"2023-01-15T14:30:49.125686+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:30:47.349633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.349633+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:54.439","scrub_duration":0.50340890699999996,"objects_trimmed":0,"snaptrim_duration":0.000849955,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":187,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-01-15T14:30:51.033924+0000","last_change":"2023-01-15T14:30:51.033716+0000","last_active":"2023-01-15T14:30:51.033924+0000","last_peered":"2023-01-15T14:30:51.033924+0000","last_clean":"2023-01-15T14:30:51.033924+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:30:51.033924+0000","last_undegraded":"2023-01-15T14:30:51.033924+0000","last_fullsized":"2023-01-15T14:30:51.033924+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:31.583599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:31.583599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.22681161699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":634,"reported_epoch":163,"state":"peering","last_fresh":"2023-01-15T14:30:49.930176+0000","last_change":"2023-01-15T14:30:49.930422+0000","last_active":"2023-01-15T14:30:49.930110+0000","last_peered":"2023-01-15T14:30:49.207644+0000","last_clean":"2023-01-15T14:30:49.207644+0000","last_became_active":"2023-01-15T14:30:45.190741+0000","last_became_peered":"2023-01-15T14:30:45.190741+0000","last_unstale":"2023-01-15T14:30:49.930176+0000","last_undegraded":"2023-01-15T14:30:49.930176+0000","last_fullsized":"2023-01-15T14:30:49.930176+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.00076053600000000002,"stat_sum":{"num_bytes":1084068,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3268978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":157,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-01-15T14:30:50.026551+0000","last_change":"2023-01-15T14:30:13.907259+0000","last_active":"2023-01-15T14:30:50.026551+0000","last_peered":"2023-01-15T14:30:50.026551+0000","last_clean":"2023-01-15T14:30:50.026551+0000","last_became_active":"2023-01-15T14:30:13.906504+0000","last_became_peered":"2023-01-15T14:30:13.906504+0000","last_unstale":"2023-01-15T14:30:50.026551+0000","last_undegraded":"2023-01-15T14:30:50.026551+0000","last_fullsized":"2023-01-15T14:30:50.026551+0000","mapping_epoch":126,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:44.759745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:44.759745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:30:55.162","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":293,"reported_epoch":163,"state":"peering","last_fresh":"2023-01-15T14:30:50.128195+0000","last_change":"2023-01-15T14:30:50.128844+0000","last_active":"2023-01-15T14:30:50.128095+0000","last_peered":"2023-01-15T14:30:49.225537+0000","last_clean":"2023-01-15T14:30:49.225537+0000","last_became_active":"2023-01-15T14:30:39.115312+0000","last_became_peered":"2023-01-15T14:30:39.115312+0000","last_unstale":"2023-01-15T14:30:50.128195+0000","last_undegraded":"2023-01-15T14:30:50.128195+0000","last_fullsized":"2023-01-15T14:30:50.128195+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:29:45.876183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:29:45.876183+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"157'196","reported_seq":749,"reported_epoch":163,"state":"peering","last_fresh":"2023-01-15T14:30:49.928658+0000","last_change":"2023-01-15T14:30:49.928924+0000","last_active":"2023-01-15T14:30:49.928617+0000","last_peered":"2023-01-15T14:30:49.137481+0000","last_clean":"2023-01-15T14:30:49.137481+0000","last_became_active":"2023-01-15T14:30:40.073925+0000","last_became_peered":"2023-01-15T14:30:40.073925+0000","last_unstale":"2023-01-15T14:30:49.928658+0000","last_undegraded":"2023-01-15T14:30:49.928658+0000","last_fullsized":"2023-01-15T14:30:49.928658+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:48.277","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.30662587099999999,"stat_sum":{"num_bytes":5983292,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":5844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7596764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":591,"reported_epoch":163,"state":"active+clean","last_fresh":"2023-01-15T14:30:50.021968+0000","last_change":"2023-01-15T14:30:43.882014+0000","last_active":"2023-01-15T14:30:50.021968+0000","last_peered":"2023-01-15T14:30:50.021968+0000","last_clean":"2023-01-15T14:30:50.021968+0000","last_became_active":"2023-01-15T14:29:58.963354+0000","last_became_peered":"2023-01-15T14:29:58.963354+0000","last_unstale":"2023-01-15T14:30:50.021968+0000","last_undegraded":"2023-01-15T14:30:50.021968+0000","last_fullsized":"2023-01-15T14:30:50.021968+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:30:08.681869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:08.681869+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:19.959","scrub_duration":0.40398525099999999,"objects_trimmed":0,"snaptrim_duration":0.0019274520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.a","version":"158'182","reported_seq":682,"reported_epoch":164,"state":"remapped+peering","last_fresh":"2023-01-15T14:30:51.022132+0000","last_change":"2023-01-15T14:30:51.022811+0000","last_active":"2023-01-15T14:30:49.924264+0000","last_peered":"2023-01-15T14:30:49.114456+0000","last_clean":"2023-01-15T14:30:49.114456+0000","last_became_active":"2023-01-15T14:30:44.888318+0000","last_became_peered":"2023-01-15T14:30:44.888318+0000","last_unstale":"2023-01-15T14:30:51.022132+0000","last_undegraded":"2023-01-15T14:30:51.022132+0000","last_fullsized":"2023-01-15T14:30:51.022132+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30212452699999998,"stat_sum":{"num_bytes":8676156,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":23721,"num_write":51,"num_write_kb":24558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":27807017,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":473,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-01-15T14:30:49.137409+0000","last_change":"2023-01-15T14:30:45.190726+0000","last_active":"2023-01-15T14:30:49.137409+0000","last_peered":"2023-01-15T14:30:49.137409+0000","last_clean":"2023-01-15T14:30:49.137409+0000","last_became_active":"2023-01-15T14:30:45.089409+0000","last_became_peered":"2023-01-15T14:30:45.089409+0000","last_unstale":"2023-01-15T14:30:49.137409+0000","last_undegraded":"2023-01-15T14:30:49.137409+0000","last_fullsized":"2023-01-15T14:30:49.137409+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100756206,"stat_sum":{"num_bytes":1731121,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":6694,"num_write":11,"num_write_kb":5635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.c","version":"141'46","reported_seq":458,"reported_epoch":163,"state":"peering","last_fresh":"2023-01-15T14:30:49.920934+0000","last_change":"2023-01-15T14:30:49.921690+0000","last_active":"2023-01-15T14:30:49.920832+0000","last_peered":"2023-01-15T14:30:49.037417+0000","last_clean":"2023-01-15T14:30:49.037417+0000","last_became_active":"2023-01-15T14:30:44.982366+0000","last_became_peered":"2023-01-15T14:30:44.982366+0000","last_unstale":"2023-01-15T14:30:49.920934+0000","last_undegraded":"2023-01-15T14:30:49.920934+0000","last_fullsized":"2023-01-15T14:30:49.920934+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.00074144800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":10,"num_write_kb":2522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1159666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.d","version":"155'154","reported_seq":696,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-01-15T14:30:50.926976+0000","last_change":"2023-01-15T14:30:47.381751+0000","last_active":"2023-01-15T14:30:50.926976+0000","last_peered":"2023-01-15T14:30:50.926976+0000","last_clean":"2023-01-15T14:30:50.926976+0000","last_became_active":"2023-01-15T14:30:44.889199+0000","last_became_peered":"2023-01-15T14:30:44.889199+0000","last_unstale":"2023-01-15T14:30:50.926976+0000","last_undegraded":"2023-01-15T14:30:50.926976+0000","last_fullsized":"2023-01-15T14:30:50.926976+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:01.864","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.002552487,"stat_sum":{"num_bytes":6891863,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":489,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-01-15T14:30:51.031681+0000","last_change":"2023-01-15T14:30:51.031681+0000","last_active":"2023-01-15T14:30:51.031681+0000","last_peered":"2023-01-15T14:30:51.031681+0000","last_clean":"2023-01-15T14:30:51.031681+0000","last_became_active":"2023-01-15T14:30:51.030850+0000","last_became_peered":"2023-01-15T14:30:51.030850+0000","last_unstale":"2023-01-15T14:30:51.031681+0000","last_undegraded":"2023-01-15T14:30:51.031681+0000","last_fullsized":"2023-01-15T14:30:51.031681+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'37","last_scrub_stamp":"2023-01-15T14:30:25.793922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:25.793922+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.61105603799999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":11529,"num_write":5,"num_write_kb":4046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1487419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":672,"reported_epoch":164,"state":"remapped+peering","last_fresh":"2023-01-15T14:30:50.922000+0000","last_change":"2023-01-15T14:30:50.922222+0000","last_active":"2023-01-15T14:30:49.923444+0000","last_peered":"2023-01-15T14:30:48.912715+0000","last_clean":"2023-01-15T14:30:48.912715+0000","last_became_active":"2023-01-15T14:30:45.187515+0000","last_became_peered":"2023-01-15T14:30:45.187515+0000","last_unstale":"2023-01-15T14:30:50.922000+0000","last_undegraded":"2023-01-15T14:30:50.922000+0000","last_fullsized":"2023-01-15T14:30:50.922000+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50980899000000002,"stat_sum":{"num_bytes":6227010,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":33587,"num_write":38,"num_write_kb":12831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":14899405,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,3],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.10","version":"130'234","reported_seq":896,"reported_epoch":162,"state":"active+clean","last_fresh":"2023-01-15T14:30:49.225466+0000","last_change":"2023-01-15T14:30:43.985880+0000","last_active":"2023-01-15T14:30:49.225466+0000","last_peered":"2023-01-15T14:30:49.225466+0000","last_clean":"2023-01-15T14:30:49.225466+0000","last_became_active":"2023-01-15T14:30:19.199769+0000","last_became_peered":"2023-01-15T14:30:19.199769+0000","last_unstale":"2023-01-15T14:30:49.225466+0000","last_undegraded":"2023-01-15T14:30:49.225466+0000","last_fullsized":"2023-01-15T14:30:49.225466+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":1,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:44.768","scrub_duration":0.20710088300000001,"objects_trimmed":0,"snaptrim_duration":0.103327712,"stat_sum":{"num_bytes":3714347,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.11","version":"155'133","reported_seq":647,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-01-15T14:30:51.024698+0000","last_change":"2023-01-15T14:30:43.882812+0000","last_active":"2023-01-15T14:30:51.024698+0000","last_peered":"2023-01-15T14:30:51.024698+0000","last_clean":"2023-01-15T14:30:51.024698+0000","last_became_active":"2023-01-15T14:30:22.499879+0000","last_became_peered":"2023-01-15T14:30:22.499879+0000","last_unstale":"2023-01-15T14:30:51.024698+0000","last_undegraded":"2023-01-15T14:30:51.024698+0000","last_fullsized":"2023-01-15T14:30:51.024698+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":135,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'112","last_scrub_stamp":"2023-01-15T14:30:09.593275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.593275+0000","objects_scrubbed":2,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:27.977","scrub_duration":0.28909560899999998,"objects_trimmed":0,"snaptrim_duration":0.0016113919999999999,"stat_sum":{"num_bytes":11293458,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":27,"num_read_kb":33961,"num_write":8,"num_write_kb":8388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5697573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.12","version":"155'125","reported_seq":257,"reported_epoch":163,"state":"peering","last_fresh":"2023-01-15T14:30:49.928100+0000","last_change":"2023-01-15T14:30:49.928365+0000","last_active":"2023-01-15T14:30:49.928065+0000","last_peered":"2023-01-15T14:30:49.225940+0000","last_clean":"2023-01-15T14:30:49.225940+0000","last_became_active":"2023-01-15T14:30:37.932190+0000","last_became_peered":"2023-01-15T14:30:37.932190+0000","last_unstale":"2023-01-15T14:30:49.928100+0000","last_undegraded":"2023-01-15T14:30:49.928100+0000","last_fullsized":"2023-01-15T14:30:49.928100+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":5,"last_scrub":"139'103","last_scrub_stamp":"2023-01-15T14:30:30.045615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:30.045615+0000","objects_scrubbed":3,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:52.334","scrub_duration":0.207128334,"objects_trimmed":0,"snaptrim_duration":0.0079632239999999996,"stat_sum":{"num_bytes":5596764,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":19,"num_read_kb":27344,"num_write":5,"num_write_kb":1114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":6669165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]},{"pgid":"3.13","version":"155'246","reported_seq":720,"reported_epoch":164,"state":"active+clean","last_fresh":"2023-01-15T14:30:50.925031+0000","last_change":"2023-01-15T14:30:43.982488+0000","last_active":"2023-01-15T14:30:50.925031+0000","last_peered":"2023-01-15T14:30:50.925031+0000","last_clean":"2023-01-15T14:30:50.925031+0000","last_became_active":"2023-01-15T14:30:08.974612+0000","last_became_peered":"2023-01-15T14:30:08.974612+0000","last_unstale":"2023-01-15T14:30:50.925031+0000","last_undegraded":"2023-01-15T14:30:50.925031+0000","last_fullsized":"2023-01-15T14:30:50.925031+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":246,"log_dups_size":0,"ondisk_log_size":246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:29.544","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.001116758,"stat_sum":{"num_bytes":9210236,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":22682,"num_write":16,"num_write_kb":10625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"1"},{"start":"78","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":130898954,"num_objects":53,"num_object_clones":17,"num_object_copies":159,"num_objects_missing_on_primary":5,"num_objects_missing":5,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":3,"num_read":503,"num_read_kb":468569,"num_write":470,"num_write_kb":198795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":337,"num_bytes_recovered":241039434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4773,"ondisk_log_size":4773,"up":108,"acting":102,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574883,"num_pgs":16,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":860352,"kb_used_data":92850336,"kb_used_omap":1966,"kb_used_meta":0,"kb_avail":92839028,"statfs":{"total":95948165120,"available":95067164672,"internally_reserved":11579392,"allocated":95078744064,"data_stored":95078744064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2013521,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":3.4119999999999999,"15min":3.8500000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.522,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":9.1829999999999998,"5min":9.2029999999999994,"15min":9.2070000000000007},"min":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":73.769000000000005,"5min":73.769000000000005,"15min":73.769000000000005},"last":0.56799999999999995}]},{"osd":1,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":6.3079999999999998,"5min":6.4969999999999999,"15min":6.5289999999999999},"min":{"1min":0.56200000000000006,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":88.003,"5min":88.003,"15min":88.003},"last":0.54000000000000004},{"interface":"front","average":{"1min":5.3300000000000001,"5min":6.0899999999999999,"15min":6.2160000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":73.704999999999998,"5min":73.704999999999998,"15min":73.704999999999998},"last":0.72899999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":13.926,"5min":6.6180000000000003,"15min":5.4000000000000004},"min":{"1min":0.69799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":74.010999999999996,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":3.5230000000000001,"15min":3.9590000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.8169999999999999,"5min":75.998999999999995,"15min":75.998999999999995},"last":0.81899999999999995}]},{"osd":3,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":14.441000000000001,"5min":3.6459999999999999,"15min":1.847},"min":{"1min":0.71799999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":144.65600000000001,"5min":144.65600000000001,"15min":144.65600000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":4.0670000000000002,"5min":4.5350000000000001,"15min":4.6130000000000004},"min":{"1min":0.71799999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":45.164000000000001,"5min":76.034999999999997,"15min":76.034999999999997},"last":0.77100000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":5.3949999999999996,"5min":8.6110000000000007,"15min":9.1470000000000002},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":73.459999999999994,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":16.405000000000001,"5min":4.0529999999999999,"15min":1.994},"min":{"1min":0.67000000000000004,"5min":0.498,"15min":0.498},"max":{"1min":164.21199999999999,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.98299999999999998}]},{"osd":5,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":5.4480000000000004,"5min":6.1509999999999998,"15min":6.2679999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":73.561000000000007,"5min":73.561000000000007,"15min":73.561000000000007},"last":0.92500000000000004},{"interface":"front","average":{"1min":5.3739999999999997,"5min":5.0300000000000002,"15min":4.9729999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":70.387,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.028}]},{"osd":6,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":5.6429999999999998,"5min":7.9470000000000001,"15min":8.3309999999999995},"min":{"1min":0.68700000000000006,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":73.638999999999996,"5min":73.638999999999996,"15min":73.638999999999996},"last":1.008},{"interface":"front","average":{"1min":5.5419999999999998,"5min":1.988,"15min":1.3959999999999999},"min":{"1min":0.93899999999999995,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":70.530000000000001,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.0620000000000001}]}]},{"osd":6,"up_from":0,"seq":55834574884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":30,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":924308,"kb_used_data":92775384,"kb_used_omap":3476,"kb_used_meta":0,"kb_avail":92775072,"statfs":{"total":95948165120,"available":95001673728,"internally_reserved":319488,"allocated":95001993216,"data_stored":95001993216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3559911,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":7.6550000000000002,"5min":5.4630000000000001,"15min":5.0970000000000004},"min":{"1min":0.47599999999999998,"5min":0.44,"15min":0.44},"max":{"1min":86.519000000000005,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.92800000000000005},{"interface":"front","average":{"1min":3.6139999999999999,"5min":1.347,"15min":0.96899999999999997},"min":{"1min":0.502,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":54.201000000000001,"5min":54.201000000000001,"15min":54.201000000000001},"last":0.64900000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":8.8239999999999998,"15min":8.6699999999999999},"min":{"1min":0.502,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":171.91499999999999,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":5.726,"5min":1.78,"15min":1.1220000000000001},"min":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":93.156000000000006,"5min":93.156000000000006,"15min":93.156000000000006},"last":0.77400000000000002}]},{"osd":2,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":5.2720000000000002,"5min":5.4630000000000001,"15min":5.4950000000000001},"min":{"1min":0.57299999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":85.185000000000002,"5min":99.884,"15min":99.884},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.875,"5min":0.90300000000000002,"15min":0.90700000000000003},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.1830000000000001,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":1}]},{"osd":3,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.361,"15min":1.3460000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":12.878,"5min":12.878,"15min":12.878},"last":0.83499999999999996},{"interface":"front","average":{"1min":6.7469999999999999,"5min":6.2699999999999996,"15min":6.1900000000000004},"min":{"1min":0.73299999999999998,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":85.150000000000006,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.755}]},{"osd":5,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.91300000000000003,"15min":0.91700000000000004},"min":{"1min":0.60899999999999999,"5min":0.496,"15min":0.496},"max":{"1min":1.2749999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.92700000000000005,"5min":2.024,"15min":2.2069999999999999},"min":{"1min":0.65500000000000003,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.153,"5min":30.971,"15min":30.971},"last":0.72799999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":9.8130000000000006,"5min":6.5010000000000003,"15min":5.9489999999999998},"min":{"1min":0.69699999999999995,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":85.721999999999994,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.042,"5min":4.3380000000000001,"15min":4.8879999999999999},"min":{"1min":0.88100000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.373,"5min":88.186999999999998,"15min":88.186999999999998},"last":0.85599999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":10.694000000000001,"15min":12.316000000000001},"min":{"1min":0.66600000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.216,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":5.46,"5min":10.083,"15min":10.853},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":85.382999999999996,"5min":99.843000000000004,"15min":99.843000000000004},"last":0.81599999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":36,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":915892,"kb_used_data":92783768,"kb_used_omap":3622,"kb_used_meta":0,"kb_avail":92783488,"statfs":{"total":95948165120,"available":95010291712,"internally_reserved":286720,"allocated":95010578432,"data_stored":95010578432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3709390,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":9.1959999999999997,"5min":5.7169999999999996,"15min":5.1369999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":174.52199999999999,"5min":174.52199999999999,"15min":174.52199999999999},"last":36.137999999999998},{"interface":"front","average":{"1min":3.3130000000000002,"5min":6.7759999999999998,"15min":7.3529999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":43.481999999999999,"5min":93.674000000000007,"15min":93.674000000000007},"last":36.207000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":10.188000000000001,"5min":12.863,"15min":13.308999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":86.067999999999998,"5min":159.256,"15min":159.256},"last":0.755},{"interface":"front","average":{"1min":8.5890000000000004,"5min":2.3100000000000001,"15min":1.264},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":174.578,"5min":174.578,"15min":174.578},"last":0.65600000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":2.5779999999999998,"5min":3.194,"15min":3.2970000000000002},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":43.594000000000001,"5min":59.896999999999998,"15min":59.896999999999998},"last":40.692999999999998},{"interface":"front","average":{"1min":8.9700000000000006,"5min":3.548,"15min":2.6440000000000001},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":174.45099999999999,"5min":174.45099999999999,"15min":174.45099999999999},"last":42.853999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":2.9089999999999998,"5min":5.8310000000000004,"15min":6.3179999999999996},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":43.689,"5min":66.311999999999998,"15min":66.311999999999998},"last":0.99099999999999999},{"interface":"front","average":{"1min":14.416,"5min":13.981999999999999,"15min":13.91},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":88.355999999999995,"5min":159.47200000000001,"15min":159.47200000000001},"last":0.93400000000000005}]},{"osd":5,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":10.44,"5min":13.210000000000001,"15min":13.672000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":85.543999999999997,"5min":159.43000000000001,"15min":159.43000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":8.8780000000000001,"5min":2.593,"15min":1.5449999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":174.61000000000001,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.82999999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":8.8729999999999993,"5min":2.6000000000000001,"15min":1.554},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":174.63999999999999,"5min":174.63999999999999,"15min":174.63999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":2.9790000000000001,"5min":3.605,"15min":3.7090000000000001},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":43.661000000000001,"5min":60.098999999999997,"15min":60.098999999999997},"last":1.077}]},{"osd":7,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":2.944,"5min":3.8170000000000002,"15min":3.9630000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":43.712000000000003,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.1439999999999999},{"interface":"front","average":{"1min":10.481999999999999,"5min":16.568000000000001,"15min":17.582000000000001},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":85.484999999999999,"5min":159.512,"15min":159.512},"last":1.248}]}]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":25,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":919548,"kb_used_data":92804900,"kb_used_omap":2800,"kb_used_meta":0,"kb_avail":92779832,"statfs":{"total":95948165120,"available":95006547968,"internally_reserved":25669632,"allocated":95032217600,"data_stored":95032217600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2868083,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":3.008,"5min":1.1140000000000001,"15min":0.79800000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":45.466999999999999,"5min":45.466999999999999,"15min":45.466999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.111,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":88.299999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":11.958,"5min":3.0299999999999998,"15min":1.542},"min":{"1min":0.371,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":143.071,"5min":143.071,"15min":143.071},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.73299999999999998,"5min":2.831,"15min":3.181},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":2.952,"5min":47.518000000000001,"15min":47.518000000000001},"last":88.186000000000007}]},{"osd":3,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":20.398,"5min":12.401,"15min":11.068},"min":{"1min":0.41399999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":143.13300000000001,"5min":183.53,"15min":183.53},"last":0.72699999999999998},{"interface":"front","average":{"1min":11.003,"5min":2.8140000000000001,"15min":1.4490000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":195.16399999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.67600000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":12.813000000000001,"5min":5.4820000000000002,"15min":4.2599999999999998},"min":{"1min":0.67500000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":175.34700000000001,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":12.773999999999999,"5min":5.6020000000000003,"15min":4.4059999999999997},"min":{"1min":0.75800000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":175.30500000000001,"5min":175.30500000000001,"15min":175.30500000000001},"last":88.459999999999994}]},{"osd":5,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":8.4290000000000003,"5min":2.6259999999999999,"15min":1.659},"min":{"1min":0.67300000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":143.262,"5min":143.262,"15min":143.262},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.006,"5min":1.1299999999999999,"15min":1.1499999999999999},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":2.2280000000000002,"5min":4.0899999999999999,"15min":4.0899999999999999},"last":0.93000000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.0960000000000001,"15min":1.0529999999999999},"min":{"1min":0.72499999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":7.8849999999999998,"5min":7.8849999999999998,"15min":7.8849999999999998},"last":88.542000000000002},{"interface":"front","average":{"1min":8.8740000000000006,"5min":2.7490000000000001,"15min":1.728},"min":{"1min":0.75800000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":143.215,"5min":143.215,"15min":143.215},"last":0.90800000000000003}]},{"osd":7,"last update":"Sun Jan 15 14:30:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":2.637,"15min":2.9180000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.073,"5min":37.124000000000002,"15min":37.124000000000002},"last":88.623999999999995},{"interface":"front","average":{"1min":8.4359999999999999,"5min":3.9079999999999999,"15min":3.1539999999999999},"min":{"1min":0.78400000000000003,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":143.17099999999999,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.96699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798300,"kb_used_data":92901420,"kb_used_omap":2986,"kb_used_meta":0,"kb_avail":92901080,"statfs":{"total":95948165120,"available":95130705920,"internally_reserved":348160,"allocated":95131054080,"data_stored":95131054080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3058187,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":7.5800000000000001,"5min":4.6299999999999999,"15min":4.1379999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":144.12799999999999,"5min":144.12799999999999,"15min":144.12799999999999},"last":1.147},{"interface":"front","average":{"1min":0.61199999999999999,"5min":3.5800000000000001,"15min":4.0750000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.456,"5min":50.616,"15min":50.616},"last":0.497}]},{"osd":2,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.88900000000000001,"15min":0.94099999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.82999999999999996,"5min":7.3470000000000004,"15min":7.3470000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":3.0779999999999998,"5min":3.8740000000000001,"15min":4.0060000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":48.119999999999997,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.80500000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":16.004999999999999,"15min":18.559000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.91600000000000004,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":2.6269999999999998,"5min":6.3209999999999997,"15min":6.9370000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":38.292999999999999,"5min":53.145000000000003,"15min":53.145000000000003},"last":0.83799999999999997}]},{"osd":4,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":3.4329999999999998,"5min":4.4690000000000003,"15min":4.641},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":53.667000000000002,"5min":53.667000000000002,"15min":53.667000000000002},"last":0.93500000000000005},{"interface":"front","average":{"1min":5.3319999999999999,"5min":3.9500000000000002,"15min":3.7200000000000002},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":93.989000000000004,"5min":93.989000000000004,"15min":93.989000000000004},"last":0.92500000000000004}]},{"osd":5,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":1.01,"15min":1.0249999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.5,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.94799999999999995,"5min":3.7400000000000002,"15min":4.2050000000000001},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.2629999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.86199999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":4.1929999999999996,"15min":4.7320000000000002},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.6639999999999999,"5min":50.844999999999999,"15min":50.844999999999999},"last":1.121},{"interface":"front","average":{"1min":0.93000000000000005,"5min":1.085,"15min":1.111},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2629999999999999,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.0409999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":6.2160000000000002,"5min":6.8559999999999999,"15min":6.9630000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":72.632999999999996,"5min":72.632999999999996,"15min":72.632999999999996},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":4.0039999999999996,"15min":4.5010000000000003},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.7390000000000001,"5min":50.804000000000002,"15min":50.804000000000002},"last":1.0680000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":35,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":905120,"kb_used_data":92796608,"kb_used_omap":3576,"kb_used_meta":0,"kb_avail":92794260,"statfs":{"total":95948165120,"available":95021322240,"internally_reserved":2404352,"allocated":95023726592,"data_stored":95023726592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3661874,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":4.3659999999999997,"5min":1.371,"15min":0.872},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":69.872,"5min":69.872,"15min":69.872},"last":0.54300000000000004},{"interface":"front","average":{"1min":5.5919999999999996,"5min":1.702,"15min":1.054},"min":{"1min":0.38700000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":47.795999999999999,"5min":47.795999999999999,"15min":47.795999999999999},"last":0.71799999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":9.25,"5min":2.4049999999999998,"15min":1.264},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":65.825000000000003,"5min":65.825000000000003,"15min":65.825000000000003},"last":0.60499999999999998},{"interface":"front","average":{"1min":8.2590000000000003,"5min":2.2189999999999999,"15min":1.212},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":71.406999999999996,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.67300000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":4.5759999999999996,"5min":14.337999999999999,"15min":15.965},"min":{"1min":0.47599999999999998,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":69.968000000000004,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.624},{"interface":"front","average":{"1min":15.317,"5min":3.7040000000000002,"15min":1.768},"min":{"1min":0.45700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":174.09399999999999,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.79400000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":7.6420000000000003,"5min":2.3170000000000002,"15min":1.429},"min":{"1min":0.54900000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":71.483000000000004,"5min":71.483000000000004,"15min":71.483000000000004},"last":0.81499999999999995},{"interface":"front","average":{"1min":7.6029999999999998,"5min":2.3399999999999999,"15min":1.4630000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":70.046999999999997,"5min":70.046999999999997,"15min":70.046999999999997},"last":0.91400000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":5.7960000000000003,"5min":8.9969999999999999,"15min":9.5310000000000006},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":47.969000000000001,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":4.7880000000000003,"5min":1.788,"15min":1.288},"min":{"1min":0.63400000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":71.563999999999993,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.94899999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":4.8070000000000004,"5min":1.897,"15min":1.4119999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":70.189999999999998,"5min":70.189999999999998,"15min":70.189999999999998},"last":1.0169999999999999},{"interface":"front","average":{"1min":5.8959999999999999,"5min":2.0859999999999999,"15min":1.4510000000000001},"min":{"1min":0.66700000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":47.920999999999999,"5min":47.920999999999999,"15min":47.920999999999999},"last":0.99099999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:30:06 2023","interfaces":[{"interface":"back","average":{"1min":8.3680000000000003,"5min":14.210000000000001,"15min":15.183999999999999},"min":{"1min":0.66800000000000004,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":71.629999999999995,"5min":292.61099999999999,"15min":292.61099999999999},"last":73.864999999999995},{"interface":"front","average":{"1min":4.8170000000000002,"5min":1.8999999999999999,"15min":1.4139999999999999},"min":{"1min":0.72499999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":70.126000000000005,"5min":70.126000000000005,"15min":70.126000000000005},"last":0.77400000000000002}]}]}],"pool_statfs":[]}} 2023-01-15T14:30:55.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 36 2023-01-15T14:30:55.554 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:55.554 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:55.555 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:30:55.555 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:55.516+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:55.622 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:55.619+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:55.708 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:55.708 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:55.724 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:30:55.728 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:55.723+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:55.829 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:55.824+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:55.852 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:55.853 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:55.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:30:55.930 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:55.928+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:55.995 INFO:tasks.rados.rados.0.smithi090.stdout:668: done (4 left) 2023-01-15T14:30:55.996 INFO:tasks.rados.rados.0.smithi090.stdout:669: read oid 40 snap -1 2023-01-15T14:30:55.996 INFO:tasks.rados.rados.0.smithi090.stdout:669: expect (ObjNum 240 snap 55 seq_num 240) 2023-01-15T14:30:55.998 INFO:tasks.rados.rados.0.smithi090.stdout:670: copy_from oid 39 from oid 47 current snap is 65 2023-01-15T14:30:55.998 INFO:tasks.rados.rados.0.smithi090.stdout:671: copy_from oid 5 from oid 29 current snap is 65 2023-01-15T14:30:55.998 INFO:tasks.rados.rados.0.smithi090.stdout:672: copy_from oid 4 from oid 1 current snap is 65 2023-01-15T14:30:55.998 INFO:tasks.rados.rados.0.smithi090.stdout:673: write (excl) oid 34 current snap is 65 2023-01-15T14:30:55.999 INFO:tasks.rados.rados.0.smithi090.stdout:673: seq_num 265 ranges {453855=696987,1890787=548083} 2023-01-15T14:30:56.000 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:56.065 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:56.066 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:30:56.067 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pgp_num to 36 2023-01-15T14:30:56.067 INFO:tasks.rados.rados.0.smithi090.stdout:673: writing smithi09025264-34 from 453855 to 1150842 tid 1 2023-01-15T14:30:56.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:30:56.069 INFO:tasks.rados.rados.0.smithi090.stdout:673: writing smithi09025264-34 from 1890787 to 2438870 tid 2 2023-01-15T14:30:56.069 INFO:tasks.rados.rados.0.smithi090.stdout:674: snap_create 2023-01-15T14:30:56.069 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 9 2023-01-15T14:30:56.069 INFO:tasks.rados.rados.0.smithi090.stdout:663: finishing write tid 1 to smithi09025264-49 2023-01-15T14:30:56.069 INFO:tasks.rados.rados.0.smithi090.stdout:662: expect (ObjNum 101 snap 21 seq_num 101) 2023-01-15T14:30:56.070 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:56.064+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:56.168 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:30:56.164+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:30:56.178 INFO:tasks.rados.rados.0.smithi090.stdout:665: finishing rollback tid 1 to smithi09025264-15 2023-01-15T14:30:56.178 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 149 (ObjNum 273 snap 0 seq_num 0) dirty dne 2023-01-15T14:30:56.178 INFO:tasks.rados.rados.0.smithi090.stdout:663: finishing write tid 2 to smithi09025264-49 2023-01-15T14:30:56.179 INFO:tasks.rados.rados.0.smithi090.stdout:663: finishing write tid 3 to smithi09025264-49 2023-01-15T14:30:56.179 INFO:tasks.rados.rados.0.smithi090.stdout:663: finishing write tid 4 to smithi09025264-49 2023-01-15T14:30:56.179 INFO:tasks.rados.rados.0.smithi090.stdout:663: oid 49 updating version 0 to 210 2023-01-15T14:30:56.179 INFO:tasks.rados.rados.0.smithi090.stdout:663: oid 49 updating version 210 to 211 2023-01-15T14:30:56.179 INFO:tasks.rados.rados.0.smithi090.stdout:663: oid 49 updating version 211 to 212 2023-01-15T14:30:56.179 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 212 (ObjNum 263 snap 65 seq_num 263) dirty exists 2023-01-15T14:30:56.180 INFO:tasks.rados.rados.0.smithi090.stdout:663: left oid 49 (ObjNum 263 snap 65 seq_num 263) 2023-01-15T14:30:56.180 INFO:tasks.rados.rados.0.smithi090.stdout:667: expect (ObjNum 252 snap 60 seq_num 252) 2023-01-15T14:30:56.210 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:56.210 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:56.221 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:30:56.269 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:30:56.267+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:30:56.474 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:30:56.476 INFO:tasks.rados.rados.0.smithi090.stdout:662: done (8 left) 2023-01-15T14:30:56.476 INFO:tasks.rados.rados.0.smithi090.stdout:663: done (7 left) 2023-01-15T14:30:56.476 INFO:tasks.rados.rados.0.smithi090.stdout:665: done (6 left) 2023-01-15T14:30:56.476 INFO:tasks.rados.rados.0.smithi090.stdout:667: done (5 left) 2023-01-15T14:30:56.477 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:30:56.477 INFO:tasks.rados.rados.0.smithi090.stdout:672: finishing copy_from to smithi09025264-4 2023-01-15T14:30:56.477 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 4 v 149 (ObjNum 143 snap 27 seq_num 143) dirty exists 2023-01-15T14:30:56.477 INFO:tasks.rados.rados.0.smithi090.stdout:672: finishing copy_from racing read to smithi09025264-4 2023-01-15T14:30:56.477 INFO:tasks.rados.rados.0.smithi090.stdout:671: finishing copy_from to smithi09025264-5 2023-01-15T14:30:56.478 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 133 (ObjNum 248 snap 58 seq_num 248) dirty exists 2023-01-15T14:30:56.478 INFO:tasks.rados.rados.0.smithi090.stdout:671: finishing copy_from racing read to smithi09025264-5 2023-01-15T14:30:56.478 INFO:tasks.rados.rados.0.smithi090.stdout:671: done (4 left) 2023-01-15T14:30:56.478 INFO:tasks.rados.rados.0.smithi090.stdout:672: done (3 left) 2023-01-15T14:30:56.478 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:30:56.480 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:56.480 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:56.481 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:30:56.481 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:56.471+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:56.499 INFO:tasks.rados.rados.0.smithi090.stdout:673: finishing write tid 1 to smithi09025264-34 2023-01-15T14:30:56.576 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:56.571+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:56.596 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:56.596 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:56.606 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:30:56.620 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:30:56.639 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:30:56.656 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:30:56.677 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:56.672+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:56.678 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:30:56.702 INFO:tasks.rados.rados.0.smithi090.stdout:673: finishing write tid 2 to smithi09025264-34 2023-01-15T14:30:56.702 INFO:tasks.rados.rados.0.smithi090.stdout:673: finishing write tid 3 to smithi09025264-34 2023-01-15T14:30:56.703 INFO:tasks.rados.rados.0.smithi090.stdout:673: finishing write tid 4 to smithi09025264-34 2023-01-15T14:30:56.703 INFO:tasks.rados.rados.0.smithi090.stdout:673: oid 34 updating version 0 to 137 2023-01-15T14:30:56.703 INFO:tasks.rados.rados.0.smithi090.stdout:673: oid 34 version 137 is already newer than 136 2023-01-15T14:30:56.703 INFO:tasks.rados.rados.0.smithi090.stdout:673: oid 34 updating version 137 to 138 2023-01-15T14:30:56.703 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 138 (ObjNum 265 snap 65 seq_num 265) dirty exists 2023-01-15T14:30:56.704 INFO:tasks.rados.rados.0.smithi090.stdout:673: left oid 34 (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:30:56.704 INFO:tasks.rados.rados.0.smithi090.stdout:673: done (2 left) 2023-01-15T14:30:56.704 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:30:56.712 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:30:56.712 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:30:56.723 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:30:56.886 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:30:56.886 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:30:56.886 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:30:56.887 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:30:56.887 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:30:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:30:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:30:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:30:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:30:56.888 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:30:56.889 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:30:56.889 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:30:56.889 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:30:56.890 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:56.887+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:56.894 INFO:tasks.rados.rados.0.smithi090.stdout:669: expect (ObjNum 240 snap 55 seq_num 240) 2023-01-15T14:30:56.983 INFO:tasks.rados.rados.0.smithi090.stdout:669: done (1 left) 2023-01-15T14:30:57.087 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:30:57.089 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:57.090 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:30:57.090 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:30:57.090 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:57.090 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:30:57.091 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:57.084+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:57.190 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:30:57.187+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:30:57.200 INFO:tasks.rados.rados.0.smithi090.stdout:670: finishing copy_from to smithi09025264-39 2023-01-15T14:30:57.201 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 193 (ObjNum 183 snap 40 seq_num 183) dirty exists 2023-01-15T14:30:57.201 INFO:tasks.rados.rados.0.smithi090.stdout:670: finishing copy_from racing read to smithi09025264-39 2023-01-15T14:30:57.201 INFO:tasks.rados.rados.0.smithi090.stdout:670: done (0 left) 2023-01-15T14:30:57.206 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:57.206 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:30:57.206 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:30:57.206 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:30:57.206 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:57.216 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:30:57.292 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:57.288+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:57.371 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:57.371 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:30:57.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:30:57.371 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:30:57.371 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.371 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:5 3.5 3:a1c47f99:::smithi09025264-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:30:57.371 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:30:57.371 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.584170801, 2023-01-15T14:30:57.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40548699700000002, 2023-01-15T14:30:57.372 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.372 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:57.372 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.372 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.372 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.372 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 5 2023-01-15T14:30:57.372 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.372 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.372 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.372 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.372 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:30:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:30:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781744+0000", 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782566+0000", 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000821578 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782588+0000", 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.226e-05 2023-01-15T14:30:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782601+0000", 2023-01-15T14:30:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2998e-05 2023-01-15T14:30:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882850+0000", 2023-01-15T14:30:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100248908 2023-01-15T14:30:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:30:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882871+0000", 2023-01-15T14:30:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0873000000000002e-05 2023-01-15T14:30:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083679+0000", 2023-01-15T14:30:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200808083 2023-01-15T14:30:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083948+0000", 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000269342 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084161+0000", 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021326900000000001 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:57.378 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084655+0000", 2023-01-15T14:30:57.378 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049371300000000001 2023-01-15T14:30:57.378 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.378 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.378 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:57.378 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084674+0000", 2023-01-15T14:30:57.378 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9012999999999999e-05 2023-01-15T14:30:57.378 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.379 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:57.379 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084924+0000", 2023-01-15T14:30:57.379 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024989899999999999 2023-01-15T14:30:57.379 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.379 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:57.379 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085221+0000", 2023-01-15T14:30:57.379 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029719600000000001 2023-01-15T14:30:57.379 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.380 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186953+0000", 2023-01-15T14:30:57.380 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10173174 2023-01-15T14:30:57.380 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.380 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.380 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187026+0000", 2023-01-15T14:30:57.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2957e-05 2023-01-15T14:30:57.381 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.381 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.381 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:57.381 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187124+0000", 2023-01-15T14:30:57.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8146000000000005e-05 2023-01-15T14:30:57.381 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.381 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.381 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.382 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187233+0000", 2023-01-15T14:30:57.382 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010931599999999999 2023-01-15T14:30:57.382 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.382 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.382 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.382 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:6 3.5 3:a1c47f99:::smithi09025264-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:30:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:30:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.578917929, 2023-01-15T14:30:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50419954199999995, 2023-01-15T14:30:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 6 2023-01-15T14:30:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:30:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:30:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786997+0000", 2023-01-15T14:30:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787486+0000", 2023-01-15T14:30:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048882999999999999 2023-01-15T14:30:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787514+0000", 2023-01-15T14:30:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7591999999999999e-05 2023-01-15T14:30:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787526+0000", 2023-01-15T14:30:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1894e-05 2023-01-15T14:30:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882966+0000", 2023-01-15T14:30:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095440450999999996 2023-01-15T14:30:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882973+0000", 2023-01-15T14:30:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2570000000000001e-06 2023-01-15T14:30:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084809+0000", 2023-01-15T14:30:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201835763 2023-01-15T14:30:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084928+0000", 2023-01-15T14:30:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118946 2023-01-15T14:30:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187347+0000", 2023-01-15T14:30:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102419285 2023-01-15T14:30:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187450+0000", 2023-01-15T14:30:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010286299999999999 2023-01-15T14:30:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:30:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187735+0000", 2023-01-15T14:30:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028429900000000001 2023-01-15T14:30:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188113+0000", 2023-01-15T14:30:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037883200000000001 2023-01-15T14:30:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188138+0000", 2023-01-15T14:30:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4110000000000001e-05 2023-01-15T14:30:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188389+0000", 2023-01-15T14:30:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025173900000000001 2023-01-15T14:30:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189251+0000", 2023-01-15T14:30:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086129800000000003 2023-01-15T14:30:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290989+0000", 2023-01-15T14:30:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101738524 2023-01-15T14:30:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.395 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.395 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291075+0000", 2023-01-15T14:30:57.395 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5469000000000004e-05 2023-01-15T14:30:57.395 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.395 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.395 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:57.396 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291144+0000", 2023-01-15T14:30:57.396 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:30:57.396 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.396 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.396 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.397 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291199+0000", 2023-01-15T14:30:57.397 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4413000000000002e-05 2023-01-15T14:30:57.397 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.397 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.397 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.398 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.398 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:7 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:30:57.398 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:30:57.398 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.57837147199999, 2023-01-15T14:30:57.398 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50407251399999997, 2023-01-15T14:30:57.398 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.399 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:57.399 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.399 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.399 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.399 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 7 2023-01-15T14:30:57.399 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.400 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.400 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.400 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.400 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:30:57.400 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.400 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.400 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.400 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.400 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:30:57.400 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.400 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.401 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787545+0000", 2023-01-15T14:30:57.401 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:57.401 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.401 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787551+0000", 2023-01-15T14:30:57.401 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8669999999999999e-06 2023-01-15T14:30:57.402 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.402 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.402 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787559+0000", 2023-01-15T14:30:57.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9780000000000005e-06 2023-01-15T14:30:57.402 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.402 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.402 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787564+0000", 2023-01-15T14:30:57.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5709999999999997e-06 2023-01-15T14:30:57.402 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.403 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.403 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.403 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883054+0000", 2023-01-15T14:30:57.403 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095489722999999999 2023-01-15T14:30:57.403 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.403 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.403 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:57.403 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883061+0000", 2023-01-15T14:30:57.403 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2760000000000003e-06 2023-01-15T14:30:57.403 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.403 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.404 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.404 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084987+0000", 2023-01-15T14:30:57.404 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20192545000000001 2023-01-15T14:30:57.404 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.404 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.404 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:57.404 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085190+0000", 2023-01-15T14:30:57.404 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020288899999999999 2023-01-15T14:30:57.405 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.405 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.405 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.405 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188374+0000", 2023-01-15T14:30:57.405 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103184449 2023-01-15T14:30:57.405 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.405 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.405 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.405 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188543+0000", 2023-01-15T14:30:57.405 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016925299999999999 2023-01-15T14:30:57.406 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.406 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:30:57.406 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188748+0000", 2023-01-15T14:30:57.406 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020473600000000001 2023-01-15T14:30:57.406 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.406 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:57.406 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188986+0000", 2023-01-15T14:30:57.406 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023780900000000001 2023-01-15T14:30:57.406 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189006+0000", 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0604000000000001e-05 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189084+0000", 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7526000000000003e-05 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.407 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:57.408 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189272+0000", 2023-01-15T14:30:57.408 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018762 2023-01-15T14:30:57.408 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.408 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.408 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.408 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291333+0000", 2023-01-15T14:30:57.408 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102061185 2023-01-15T14:30:57.408 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.408 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.408 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.408 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291466+0000", 2023-01-15T14:30:57.409 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013325699999999999 2023-01-15T14:30:57.409 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.409 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.409 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:57.409 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291535+0000", 2023-01-15T14:30:57.409 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9309999999999999e-05 2023-01-15T14:30:57.409 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.409 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.409 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.409 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291618+0000", 2023-01-15T14:30:57.410 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2773000000000002e-05 2023-01-15T14:30:57.410 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.410 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.410 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.410 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.410 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.410 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:8 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:30:57.410 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:30:57.410 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.57834549, 2023-01-15T14:30:57.410 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50442076000000002, 2023-01-15T14:30:57.410 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.411 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:57.411 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.411 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.411 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.411 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 8 2023-01-15T14:30:57.411 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.411 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.411 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.411 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.411 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:30:57.412 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.412 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.412 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.412 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.412 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:30:57.412 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.412 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.413 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.413 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.413 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787571+0000", 2023-01-15T14:30:57.413 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:57.413 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.413 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.414 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.414 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787575+0000", 2023-01-15T14:30:57.414 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7890000000000001e-06 2023-01-15T14:30:57.414 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.414 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.414 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.415 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787589+0000", 2023-01-15T14:30:57.415 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4212999999999999e-05 2023-01-15T14:30:57.415 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.415 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.415 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.416 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787594+0000", 2023-01-15T14:30:57.416 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1429999999999997e-06 2023-01-15T14:30:57.416 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.416 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.417 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.417 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983143+0000", 2023-01-15T14:30:57.417 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.195549003 2023-01-15T14:30:57.417 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.417 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.417 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:57.418 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983153+0000", 2023-01-15T14:30:57.418 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4369999999999996e-06 2023-01-15T14:30:57.418 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.418 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.418 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.418 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085286+0000", 2023-01-15T14:30:57.418 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102133064 2023-01-15T14:30:57.418 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.418 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.419 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:57.419 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085401+0000", 2023-01-15T14:30:57.419 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115176 2023-01-15T14:30:57.419 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.419 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.420 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.420 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189116+0000", 2023-01-15T14:30:57.420 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103715444 2023-01-15T14:30:57.420 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.420 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.420 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:57.420 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189221+0000", 2023-01-15T14:30:57.420 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010438 2023-01-15T14:30:57.420 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.420 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.421 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.421 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291724+0000", 2023-01-15T14:30:57.421 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102502998 2023-01-15T14:30:57.421 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.421 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.421 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.421 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291824+0000", 2023-01-15T14:30:57.421 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010010599999999999 2023-01-15T14:30:57.421 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.421 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.421 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.422 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291992+0000", 2023-01-15T14:30:57.422 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016856900000000001 2023-01-15T14:30:57.422 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.422 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.422 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.422 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.422 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.422 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:21 3.5 3:a75958b1:::smithi09025264-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:30:57.423 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:30:57.423 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.534256575, 2023-01-15T14:30:57.423 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.45851924500000002, 2023-01-15T14:30:57.423 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.423 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:57.423 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.423 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.424 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.424 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 21 2023-01-15T14:30:57.424 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.424 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.424 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.424 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.424 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:30:57.424 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.424 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.424 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.424 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.425 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:30:57.425 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.425 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.425 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.425 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.425 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831657+0000", 2023-01-15T14:30:57.425 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:30:57.425 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.425 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.425 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832814+0000", 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011564909999999999 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832864+0000", 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0243e-05 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832903+0000", 2023-01-15T14:30:57.426 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8704000000000001e-05 2023-01-15T14:30:57.427 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.427 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.427 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.427 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983202+0000", 2023-01-15T14:30:57.427 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.150299404 2023-01-15T14:30:57.427 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.427 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.427 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:30:57.427 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983208+0000", 2023-01-15T14:30:57.427 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0039999999999996e-06 2023-01-15T14:30:57.428 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.428 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.428 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.428 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185539+0000", 2023-01-15T14:30:57.428 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202330867 2023-01-15T14:30:57.428 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.428 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.428 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.428 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185803+0000", 2023-01-15T14:30:57.428 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026394299999999999 2023-01-15T14:30:57.428 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.429 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.429 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:30:57.429 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186037+0000", 2023-01-15T14:30:57.429 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023386300000000001 2023-01-15T14:30:57.429 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.429 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.429 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:57.429 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186637+0000", 2023-01-15T14:30:57.429 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060001800000000001 2023-01-15T14:30:57.429 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.429 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:57.430 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186665+0000", 2023-01-15T14:30:57.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8459999999999999e-05 2023-01-15T14:30:57.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.430 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:57.430 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187045+0000", 2023-01-15T14:30:57.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037992899999999999 2023-01-15T14:30:57.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.430 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:57.431 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188201+0000", 2023-01-15T14:30:57.431 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001155309 2023-01-15T14:30:57.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.431 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289903+0000", 2023-01-15T14:30:57.431 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101702601 2023-01-15T14:30:57.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289985+0000", 2023-01-15T14:30:57.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1617000000000001e-05 2023-01-15T14:30:57.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:57.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290065+0000", 2023-01-15T14:30:57.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0276000000000002e-05 2023-01-15T14:30:57.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.433 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290180+0000", 2023-01-15T14:30:57.433 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011464300000000001 2023-01-15T14:30:57.433 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.433 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.433 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.433 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.433 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:22 3.5 3:a75958b1:::smithi09025264-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:30:57.433 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:30:57.433 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.529788887, 2023-01-15T14:30:57.433 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65790599599999999, 2023-01-15T14:30:57.434 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.434 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:57.434 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.434 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.434 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.434 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 22 2023-01-15T14:30:57.434 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.434 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.434 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.434 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.435 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:30:57.435 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.435 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.435 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.435 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.435 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:30:57.435 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.435 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.435 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.435 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.436 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836126+0000", 2023-01-15T14:30:57.436 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:57.436 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.436 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.436 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.436 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836181+0000", 2023-01-15T14:30:57.436 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4246999999999997e-05 2023-01-15T14:30:57.436 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.436 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.436 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.436 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836212+0000", 2023-01-15T14:30:57.437 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1843e-05 2023-01-15T14:30:57.437 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.437 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.437 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.437 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836226+0000", 2023-01-15T14:30:57.437 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3216e-05 2023-01-15T14:30:57.437 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.437 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.437 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.437 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186757+0000", 2023-01-15T14:30:57.438 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35053103899999999 2023-01-15T14:30:57.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.438 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.438 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:57.438 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186871+0000", 2023-01-15T14:30:57.438 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114403 2023-01-15T14:30:57.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.438 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.438 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.438 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290263+0000", 2023-01-15T14:30:57.438 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103391976 2023-01-15T14:30:57.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290413+0000", 2023-01-15T14:30:57.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014991200000000001 2023-01-15T14:30:57.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:30:57.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290627+0000", 2023-01-15T14:30:57.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021379299999999999 2023-01-15T14:30:57.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.440 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:57.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290861+0000", 2023-01-15T14:30:57.440 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023479600000000001 2023-01-15T14:30:57.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.440 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:57.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290888+0000", 2023-01-15T14:30:57.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6570000000000001e-05 2023-01-15T14:30:57.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:57.441 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290990+0000", 2023-01-15T14:30:57.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101723 2023-01-15T14:30:57.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.442 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:57.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291245+0000", 2023-01-15T14:30:57.442 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025543899999999999 2023-01-15T14:30:57.442 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.442 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392822+0000", 2023-01-15T14:30:57.442 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10157725500000001 2023-01-15T14:30:57.442 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.442 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493881+0000", 2023-01-15T14:30:57.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101058262 2023-01-15T14:30:57.443 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:57.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493964+0000", 2023-01-15T14:30:57.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2973000000000007e-05 2023-01-15T14:30:57.443 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494034+0000", 2023-01-15T14:30:57.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0432999999999995e-05 2023-01-15T14:30:57.444 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.444 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.444 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.444 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:23 3.5 3:a75958b1:::smithi09025264-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:30:57.444 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:30:57.444 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.529673758, 2023-01-15T14:30:57.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65796116900000001, 2023-01-15T14:30:57.444 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.445 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:57.445 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.445 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.445 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.445 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 23 2023-01-15T14:30:57.445 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.445 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.445 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:30:57.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:30:57.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836242+0000", 2023-01-15T14:30:57.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:57.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836256+0000", 2023-01-15T14:30:57.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3677e-05 2023-01-15T14:30:57.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836265+0000", 2023-01-15T14:30:57.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2769999999999994e-06 2023-01-15T14:30:57.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936398+0000", 2023-01-15T14:30:57.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100132559 2023-01-15T14:30:57.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289431+0000", 2023-01-15T14:30:57.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35303322399999998 2023-01-15T14:30:57.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:57.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289587+0000", 2023-01-15T14:30:57.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000155873 2023-01-15T14:30:57.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292052+0000", 2023-01-15T14:30:57.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0024647919999999999 2023-01-15T14:30:57.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.450 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292180+0000", 2023-01-15T14:30:57.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012810999999999999 2023-01-15T14:30:57.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:30:57.450 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292373+0000", 2023-01-15T14:30:57.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019258599999999999 2023-01-15T14:30:57.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:57.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292597+0000", 2023-01-15T14:30:57.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022465500000000001 2023-01-15T14:30:57.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:57.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292618+0000", 2023-01-15T14:30:57.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0319999999999999e-05 2023-01-15T14:30:57.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:57.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292692+0000", 2023-01-15T14:30:57.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4643999999999999e-05 2023-01-15T14:30:57.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:57.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393103+0000", 2023-01-15T14:30:57.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100411156 2023-01-15T14:30:57.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493827+0000", 2023-01-15T14:30:57.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100724143 2023-01-15T14:30:57.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494085+0000", 2023-01-15T14:30:57.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000257891 2023-01-15T14:30:57.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:57.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494141+0000", 2023-01-15T14:30:57.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6132999999999999e-05 2023-01-15T14:30:57.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.454 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.454 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.454 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494204+0000", 2023-01-15T14:30:57.454 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2966000000000007e-05 2023-01-15T14:30:57.454 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.454 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.454 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.455 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:24 3.5 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:30:57.455 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:30:57.455 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.429492332, 2023-01-15T14:30:57.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.55808677600000001, 2023-01-15T14:30:57.455 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.455 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:57.455 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.455 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.455 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.456 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 24 2023-01-15T14:30:57.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.456 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.456 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.456 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:30:57.456 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.456 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.456 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:30:57.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936423+0000", 2023-01-15T14:30:57.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:57.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936433+0000", 2023-01-15T14:30:57.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0443e-05 2023-01-15T14:30:57.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936451+0000", 2023-01-15T14:30:57.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7842999999999998e-05 2023-01-15T14:30:57.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936461+0000", 2023-01-15T14:30:57.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0057e-05 2023-01-15T14:30:57.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289693+0000", 2023-01-15T14:30:57.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.353231617 2023-01-15T14:30:57.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:57.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289814+0000", 2023-01-15T14:30:57.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120763 2023-01-15T14:30:57.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392923+0000", 2023-01-15T14:30:57.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10310955300000001 2023-01-15T14:30:57.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:57.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393072+0000", 2023-01-15T14:30:57.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000148469 2023-01-15T14:30:57.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494242+0000", 2023-01-15T14:30:57.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117034499999999 2023-01-15T14:30:57.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494363+0000", 2023-01-15T14:30:57.498 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120703 2023-01-15T14:30:57.500 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.500 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.500 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.501 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494511+0000", 2023-01-15T14:30:57.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014873299999999999 2023-01-15T14:30:57.501 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.501 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.501 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.501 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.501 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.501 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:78 3.c 3:3137f0cb:::smithi09025264-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:57.501 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:30:57.501 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.03952633599999, 2023-01-15T14:30:57.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20331801699999999, 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 78 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:30:57.502 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326388+0000", 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.503 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326672+0000", 2023-01-15T14:30:57.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028367900000000001 2023-01-15T14:30:57.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326713+0000", 2023-01-15T14:30:57.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1539999999999999e-05 2023-01-15T14:30:57.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326729+0000", 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.519e-05 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326879+0000", 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015064899999999999 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327096+0000", 2023-01-15T14:30:57.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021682299999999999 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327308+0000", 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000211366 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327757+0000", 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000449009 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.506 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327776+0000", 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9201000000000001e-05 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328012+0000", 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236054 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429534+0000", 2023-01-15T14:30:57.507 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10152201499999999 2023-01-15T14:30:57.508 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.508 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.508 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.508 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529340+0000", 2023-01-15T14:30:57.508 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099806249 2023-01-15T14:30:57.508 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.508 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.508 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.508 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529437+0000", 2023-01-15T14:30:57.508 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6949999999999998e-05 2023-01-15T14:30:57.508 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.509 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.509 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:57.509 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529587+0000", 2023-01-15T14:30:57.509 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015008600000000001 2023-01-15T14:30:57.509 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.509 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.509 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.509 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529709+0000", 2023-01-15T14:30:57.509 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121636 2023-01-15T14:30:57.509 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.509 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:79 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.03915674000001, 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20348171300000001, 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.510 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.511 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 79 2023-01-15T14:30:57.511 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.511 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.511 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.511 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.511 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:30:57.511 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.511 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.511 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.511 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.511 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326759+0000", 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326770+0000", 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:30:57.512 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326791+0000", 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1556000000000002e-05 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326818+0000", 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6979999999999999e-05 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.513 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.427993+0000", 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117517199999999 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428266+0000", 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027233299999999998 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428696+0000", 2023-01-15T14:30:57.514 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042996499999999998 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429057+0000", 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000360952 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429076+0000", 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429160+0000", 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429549+0000", 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038884599999999998 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529886+0000", 2023-01-15T14:30:57.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100336261 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529978+0000", 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2775999999999994e-05 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530108+0000", 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012958300000000001 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530242+0000", 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013406 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:80 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.03901838100001, 2023-01-15T14:30:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20370874, 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 80 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:30:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:30:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326897+0000", 2023-01-15T14:30:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326906+0000", 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6929999999999996e-06 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326931+0000", 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4757e-05 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326945+0000", 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.402e-05 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429239+0000", 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102294418 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429487+0000", 2023-01-15T14:30:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024789300000000001 2023-01-15T14:30:57.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530309+0000", 2023-01-15T14:30:57.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100821459 2023-01-15T14:30:57.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530438+0000", 2023-01-15T14:30:57.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012928700000000001 2023-01-15T14:30:57.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530607+0000", 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169483 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:78 3.c e23/20 ondisk, result = 0)", 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:30:57.524 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.03712666000001, 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20094287799999999, 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:30:57.526 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.526 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.526 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328789+0000", 2023-01-15T14:30:57.526 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:57.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.526 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.526 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328799+0000", 2023-01-15T14:30:57.526 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8980000000000008e-06 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328820+0000", 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1041999999999999e-05 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328832+0000", 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1616e-05 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529404+0000", 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200572631 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529419+0000", 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4250000000000001e-05 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529733+0000", 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000314693 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:201 3.5 3:a1c47f99:::smithi09025264-2:head [copy-from ver 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 149.959520263, 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30931411399999997, 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.529 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 201 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.531 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:30:57.531 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.531 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406394+0000", 2023-01-15T14:30:57.531 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:30:57.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406408+0000", 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4416999999999999e-05 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406446+0000", 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7339999999999998e-05 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406467+0000", 2023-01-15T14:30:57.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1137000000000001e-05 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406677+0000", 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000210275 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406902+0000", 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022529000000000001 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.513012+0000", 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.106109527 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514754+0000", 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0017419779999999999 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514773+0000", 2023-01-15T14:30:57.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8700000000000001e-05 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.515531+0000", 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075844300000000003 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615334+0000", 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099803327999999997 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:57.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615376+0000", 2023-01-15T14:30:57.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1675999999999999e-05 2023-01-15T14:30:57.536 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.536 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:57.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715546+0000", 2023-01-15T14:30:57.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016992500000001 2023-01-15T14:30:57.536 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.701 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.702 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:57.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715626+0000", 2023-01-15T14:30:57.703 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0112999999999997e-05 2023-01-15T14:30:57.703 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.704 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.704 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715711+0000", 2023-01-15T14:30:57.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:30:57.704 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.704 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.705 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.705 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.705 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:202 3.5 3:a1c47f99:::smithi09025264-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:57.705 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:30:57.706 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 149.959407717, 2023-01-15T14:30:57.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31002637300000002, 2023-01-15T14:30:57.706 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.706 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:57.706 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:57.706 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:57.706 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:57.706 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 202 2023-01-15T14:30:57.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.706 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:30:57.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:30:57.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406508+0000", 2023-01-15T14:30:57.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:57.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406516+0000", 2023-01-15T14:30:57.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.413e-06 2023-01-15T14:30:57.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406534+0000", 2023-01-15T14:30:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8936999999999998e-05 2023-01-15T14:30:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406548+0000", 2023-01-15T14:30:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3196e-05 2023-01-15T14:30:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407271+0000", 2023-01-15T14:30:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00072301500000000003 2023-01-15T14:30:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:30:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407466+0000", 2023-01-15T14:30:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000195009 2023-01-15T14:30:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.614996+0000", 2023-01-15T14:30:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.207530192 2023-01-15T14:30:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615227+0000", 2023-01-15T14:30:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000230979 2023-01-15T14:30:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716282+0000", 2023-01-15T14:30:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101055465 2023-01-15T14:30:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716409+0000", 2023-01-15T14:30:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012653999999999999 2023-01-15T14:30:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716536+0000", 2023-01-15T14:30:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012683 2023-01-15T14:30:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:189 3.d e23/20 3:b3f111b5:::smithi09025264-48:head v 23'10, mlcod=23'6)", 2023-01-15T14:30:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:30:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 149.914203069, 2023-01-15T14:30:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20400475100000001, 2023-01-15T14:30:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:30:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:30:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451712+0000", 2023-01-15T14:30:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:30:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452272+0000", 2023-01-15T14:30:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00056035899999999997 2023-01-15T14:30:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452310+0000", 2023-01-15T14:30:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8145999999999997e-05 2023-01-15T14:30:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452324+0000", 2023-01-15T14:30:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4443999999999999e-05 2023-01-15T14:30:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652703+0000", 2023-01-15T14:30:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20037818199999999 2023-01-15T14:30:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652762+0000", 2023-01-15T14:30:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9302999999999998e-05 2023-01-15T14:30:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653302+0000", 2023-01-15T14:30:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054012199999999998 2023-01-15T14:30:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653333+0000", 2023-01-15T14:30:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.112e-05 2023-01-15T14:30:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653667+0000", 2023-01-15T14:30:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033337900000000002 2023-01-15T14:30:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653709+0000", 2023-01-15T14:30:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2469999999999998e-05 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655719+0000", 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020096699999999999 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:269 3.2 e30/20 3:47dfe935:::smithi09025264-29:head v 30'11, mlcod=23'8)", 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:30:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 142.48815762800001, 2023-01-15T14:30:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23185540399999999, 2023-01-15T14:30:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:30:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:30:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:30:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878379+0000", 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061979600000000004 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878406+0000", 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978515+0000", 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100108461 2023-01-15T14:30:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078759+0000", 2023-01-15T14:30:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100244496 2023-01-15T14:30:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078796+0000", 2023-01-15T14:30:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6758999999999998e-05 2023-01-15T14:30:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079567+0000", 2023-01-15T14:30:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077137199999999999 2023-01-15T14:30:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079593+0000", 2023-01-15T14:30:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:30:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079844+0000", 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025075899999999998 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079910+0000", 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6406000000000001e-05 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.109615+0000", 2023-01-15T14:30:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.029704323000000001 2023-01-15T14:30:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:281 3.5 3:a75958b1:::smithi09025264-6:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:30:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:30:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 142.38723399899999, 2023-01-15T14:30:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30574310900000001, 2023-01-15T14:30:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 281 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.119 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:30:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:30:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978686+0000", 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4429999999999999e-06 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978690+0000", 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0300000000000004e-06 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978693+0000", 2023-01-15T14:30:58.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.179e-06 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981592+0000", 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0028984140000000002 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981718+0000", 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012616600000000001 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:30:58.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981982+0000", 2023-01-15T14:30:58.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026445199999999998 2023-01-15T14:30:58.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:58.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982278+0000", 2023-01-15T14:30:58.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029514100000000001 2023-01-15T14:30:58.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982290+0000", 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.271e-05 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982343+0000", 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2914000000000002e-05 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:58.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982385+0000", 2023-01-15T14:30:58.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2114999999999999e-05 2023-01-15T14:30:58.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:58.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.181937+0000", 2023-01-15T14:30:58.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19955213099999999 2023-01-15T14:30:58.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:58.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284183+0000", 2023-01-15T14:30:58.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10224538 2023-01-15T14:30:58.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.126 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:58.126 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284318+0000", 2023-01-15T14:30:58.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000134843 2023-01-15T14:30:58.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.126 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:58.126 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284426+0000", 2023-01-15T14:30:58.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108513 2023-01-15T14:30:58.126 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.127 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:58.127 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.128 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:373 3.4 e34/20 3:2029fa71:::smithi09025264-39:head v 34'19, mlcod=26'16)", 2023-01-15T14:30:58.128 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:30:58.128 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 136.90625154, 2023-01-15T14:30:58.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20212622799999999, 2023-01-15T14:30:58.129 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:58.129 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:58.129 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:58.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:58.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:30:58.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:58.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459664+0000", 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460099+0000", 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043515900000000001 2023-01-15T14:30:58.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460130+0000", 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1590000000000001e-05 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560226+0000", 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100095248 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560369+0000", 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014381499999999999 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560404+0000", 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4650000000000002e-05 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560804+0000", 2023-01-15T14:30:58.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000399823 2023-01-15T14:30:58.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:58.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560827+0000", 2023-01-15T14:30:58.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3612999999999999e-05 2023-01-15T14:30:58.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:58.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.561101+0000", 2023-01-15T14:30:58.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027355499999999998 2023-01-15T14:30:58.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661710+0000", 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100608716 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661792+0000", 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2006000000000001e-05 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:58.134 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:455 3.d e41/20 3:bd982ca3:::smithi09025264-38:head v 41'24, mlcod=38'18)", 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 129.85648996699999, 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20118033699999999, 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:58.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:30:58.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:58.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:30:58.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:58.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509425+0000", 2023-01-15T14:30:58.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:58.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:58.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509441+0000", 2023-01-15T14:30:58.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5248e-05 2023-01-15T14:30:58.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:58.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509474+0000", 2023-01-15T14:30:58.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3531000000000001e-05 2023-01-15T14:30:58.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:58.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.609644+0000", 2023-01-15T14:30:58.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016994999999999 2023-01-15T14:30:58.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:58.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709872+0000", 2023-01-15T14:30:58.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10022840299999999 2023-01-15T14:30:58.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709906+0000", 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3160000000000001e-05 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710153+0000", 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024735399999999998 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:58.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710204+0000", 2023-01-15T14:30:58.140 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1075999999999997e-05 2023-01-15T14:30:58.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.140 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:58.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710280+0000", 2023-01-15T14:30:58.140 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6202e-05 2023-01-15T14:30:58.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:58.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710317+0000", 2023-01-15T14:30:58.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6387000000000002e-05 2023-01-15T14:30:58.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:58.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710607+0000", 2023-01-15T14:30:58.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029079699999999997 2023-01-15T14:30:58.320 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.320 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:58.321 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.321 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:497 3.d e44/20 3:bd982ca3:::smithi09025264-38:head v 44'31, mlcod=42'26)", 2023-01-15T14:30:58.321 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:30:58.321 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 127.591695105, 2023-01-15T14:30:58.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20082067100000001, 2023-01-15T14:30:58.321 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:58.321 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:58.321 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:58.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:58.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:30:58.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:58.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:30:58.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774220+0000", 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774238+0000", 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8107000000000001e-05 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774263+0000", 2023-01-15T14:30:58.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4122999999999999e-05 2023-01-15T14:30:58.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:58.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774280+0000", 2023-01-15T14:30:58.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7039999999999999e-05 2023-01-15T14:30:58.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:58.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874060+0000", 2023-01-15T14:30:58.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099780698000000001 2023-01-15T14:30:58.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874130+0000", 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9852999999999997e-05 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874351+0000", 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022091400000000001 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874369+0000", 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7876e-05 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874433+0000", 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3793000000000004e-05 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.974927+0000", 2023-01-15T14:30:58.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10049393500000001 2023-01-15T14:30:58.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:58.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.975043+0000", 2023-01-15T14:30:58.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115949 2023-01-15T14:30:58.327 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.327 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:58.327 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.327 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.327 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:58.327 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:58.329 INFO:tasks.rados.rados.0.smithi090.stdout:674: done (0 left) 2023-01-15T14:30:58.329 INFO:tasks.rados.rados.0.smithi090.stdout:675: write (excl) oid 24 current snap is 66 2023-01-15T14:30:58.329 INFO:tasks.rados.rados.0.smithi090.stdout:675: seq_num 266 ranges {470662=772236,1832916=408094} 2023-01-15T14:30:58.329 INFO:tasks.rados.rados.0.smithi090.stdout:675: writing smithi09025264-24 from 470662 to 1242898 tid 1 2023-01-15T14:30:58.329 INFO:tasks.rados.rados.0.smithi090.stdout:675: writing smithi09025264-24 from 1832916 to 2241010 tid 2 2023-01-15T14:30:58.329 INFO:tasks.rados.rados.0.smithi090.stdout:676: snap_remove snap 65 2023-01-15T14:30:58.329 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:58.324+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:58.330 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:30:58.430 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:58.427+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:30:58.519 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:58.520 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:30:58.520 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:30:58.520 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:58.520 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:30:58.561 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:30:58.559+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:30:58.725 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:58.725 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:30:58.725 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:30:58.726 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:30:58.726 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:30:58.726 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:30:58.727 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:30:58.720+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:30:58.826 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:30:58.824+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:30:58.933 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:30:58.934 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:30:58.936 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:30:58.936 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:30:58.936 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:30:58.937 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:30:58.937 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.937 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:97 3.a 3:52df5138:::smithi09025264-25:head [write 403368~555035,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:58.937 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:30:58.937 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 152.332786258, 2023-01-15T14:30:58.937 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.340758171, 2023-01-15T14:30:58.937 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:58.937 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:58.937 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:58.938 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:58.938 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:58.938 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 97 2023-01-15T14:30:58.938 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.938 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:58.938 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.938 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:58.938 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:30:58.938 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.938 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.938 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.939 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:58.939 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:30:58.939 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.939 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.939 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.939 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:58.939 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501140+0000", 2023-01-15T14:30:58.939 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:30:58.939 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.939 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.939 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:58.940 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501684+0000", 2023-01-15T14:30:58.940 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054441799999999996 2023-01-15T14:30:58.940 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.940 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.940 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:58.940 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501713+0000", 2023-01-15T14:30:58.940 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8833e-05 2023-01-15T14:30:58.940 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.940 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.940 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:58.940 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501726+0000", 2023-01-15T14:30:58.941 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3137e-05 2023-01-15T14:30:58.941 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.941 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.941 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:58.941 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601982+0000", 2023-01-15T14:30:58.941 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100256518 2023-01-15T14:30:58.941 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.941 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.941 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:58.941 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602426+0000", 2023-01-15T14:30:58.941 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044401500000000002 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602703+0000", 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000276889 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603066+0000", 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036278599999999998 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.942 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.943 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:58.943 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603098+0000", 2023-01-15T14:30:58.943 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1433000000000002e-05 2023-01-15T14:30:58.943 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.943 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.943 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:58.943 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603382+0000", 2023-01-15T14:30:58.943 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028402899999999999 2023-01-15T14:30:58.943 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.943 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.943 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:58.944 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603632+0000", 2023-01-15T14:30:58.944 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025033 2023-01-15T14:30:58.944 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.944 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.944 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:58.944 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805354+0000", 2023-01-15T14:30:58.944 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20172168700000001 2023-01-15T14:30:58.944 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.944 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.944 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:58.944 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841465+0000", 2023-01-15T14:30:58.945 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.036111852999999999 2023-01-15T14:30:58.945 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.945 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.945 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:58.945 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841682+0000", 2023-01-15T14:30:58.945 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000216949 2023-01-15T14:30:58.945 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.945 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.945 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:58.945 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841901+0000", 2023-01-15T14:30:58.945 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021858700000000001 2023-01-15T14:30:58.946 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.946 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:58.946 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:58.946 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.946 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.946 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:98 3.a 3:52df5138:::smithi09025264-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:58.946 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:30:58.946 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 152.33217658699999, 2023-01-15T14:30:58.946 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.44410839299999999, 2023-01-15T14:30:58.946 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:58.946 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:58.947 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:58.947 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:58.947 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:58.947 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 98 2023-01-15T14:30:58.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.947 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:58.947 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.947 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:58.947 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:30:58.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.948 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.948 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.948 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:58.948 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:30:58.948 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:58.948 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.948 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.948 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:58.948 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501752+0000", 2023-01-15T14:30:58.949 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:58.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.949 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:58.949 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502223+0000", 2023-01-15T14:30:58.949 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000471843 2023-01-15T14:30:58.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.949 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:58.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502240+0000", 2023-01-15T14:30:58.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6663e-05 2023-01-15T14:30:58.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:58.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502248+0000", 2023-01-15T14:30:58.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5669999999999997e-06 2023-01-15T14:30:58.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:58.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603236+0000", 2023-01-15T14:30:58.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100988439 2023-01-15T14:30:58.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.951 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:58.951 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603384+0000", 2023-01-15T14:30:58.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014844200000000001 2023-01-15T14:30:58.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.951 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:58.951 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842016+0000", 2023-01-15T14:30:58.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23863163800000001 2023-01-15T14:30:58.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.952 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:58.952 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842267+0000", 2023-01-15T14:30:58.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025101300000000002 2023-01-15T14:30:58.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:58.952 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:58.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:59.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842561+0000", 2023-01-15T14:30:59.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029432900000000002 2023-01-15T14:30:59.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:59.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843120+0000", 2023-01-15T14:30:59.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055900500000000001 2023-01-15T14:30:59.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:59.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843146+0000", 2023-01-15T14:30:59.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5273e-05 2023-01-15T14:30:59.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:59.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843512+0000", 2023-01-15T14:30:59.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036621599999999999 2023-01-15T14:30:59.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:59.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943324+0000", 2023-01-15T14:30:59.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099812008999999993 2023-01-15T14:30:59.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945437+0000", 2023-01-15T14:30:59.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021126069999999999 2023-01-15T14:30:59.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945619+0000", 2023-01-15T14:30:59.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018280600000000001 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945744+0000", 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012511299999999999 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945861+0000", 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011646000000000001 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.136 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:59.137 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.137 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:99 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:59.137 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:30:59.137 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 152.33166633100001, 2023-01-15T14:30:59.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.54355587900000002, 2023-01-15T14:30:59.137 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:59.137 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:59.137 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:59.137 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 99 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:59.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502262+0000", 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502270+0000", 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1999999999999994e-06 2023-01-15T14:30:59.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.140 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:59.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502286+0000", 2023-01-15T14:30:59.140 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5423000000000001e-05 2023-01-15T14:30:59.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.140 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:59.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502295+0000", 2023-01-15T14:30:59.140 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4339999999999998e-06 2023-01-15T14:30:59.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603467+0000", 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101172225 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603585+0000", 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117149 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:59.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843335+0000", 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23975083799999999 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843573+0000", 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023753000000000001 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843978+0000", 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040533799999999999 2023-01-15T14:30:59.142 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844309+0000", 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000330713 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844423+0000", 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114176 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.143 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844510+0000", 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7026000000000003e-05 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943338+0000", 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098827398999999996 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945520+0000", 2023-01-15T14:30:59.144 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021824169999999999 2023-01-15T14:30:59.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045652+0000", 2023-01-15T14:30:59.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013193200000001 2023-01-15T14:30:59.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:59.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045733+0000", 2023-01-15T14:30:59.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1435999999999998e-05 2023-01-15T14:30:59.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.146 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:59.146 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045819+0000", 2023-01-15T14:30:59.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5292999999999998e-05 2023-01-15T14:30:59.146 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.146 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:59.146 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.147 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:100 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:59.147 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:30:59.147 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 152.331618408, 2023-01-15T14:30:59.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.64425290599999996, 2023-01-15T14:30:59.147 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:59.147 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:30:59.147 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:59.147 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:59.147 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:59.147 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 100 2023-01-15T14:30:59.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.148 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:59.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:59.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:30:59.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:59.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:30:59.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:59.541 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502310+0000", 2023-01-15T14:30:59.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:59.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.542 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:59.542 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502316+0000", 2023-01-15T14:30:59.542 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7470000000000002e-06 2023-01-15T14:30:59.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.542 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:59.543 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502332+0000", 2023-01-15T14:30:59.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6866000000000001e-05 2023-01-15T14:30:59.543 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.543 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:59.543 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502343+0000", 2023-01-15T14:30:59.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0913e-05 2023-01-15T14:30:59.543 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.544 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:59.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703594+0000", 2023-01-15T14:30:59.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20125087999999999 2023-01-15T14:30:59.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.544 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.544 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:59.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703751+0000", 2023-01-15T14:30:59.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001573 2023-01-15T14:30:59.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.544 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:59.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845262+0000", 2023-01-15T14:30:59.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.141510521 2023-01-15T14:30:59.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:30:59.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845391+0000", 2023-01-15T14:30:59.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012872299999999999 2023-01-15T14:30:59.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146051+0000", 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.300660023 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146264+0000", 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212754 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146564+0000", 2023-01-15T14:30:59.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030010200000000001 2023-01-15T14:30:59.547 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.547 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:59.547 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.547 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:203 3.a 3:52df5138:::smithi09025264-25:head [write 440350~460568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:59.547 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:30:59.547 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.28081501299999, 2023-01-15T14:30:59.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40523678899999999, 2023-01-15T14:30:59.547 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:59.548 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:59.548 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:59.548 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:59.548 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:59.548 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 203 2023-01-15T14:30:59.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.548 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:59.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:59.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:30:59.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:59.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:30:59.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:59.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553111+0000", 2023-01-15T14:30:59.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:30:59.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:59.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553483+0000", 2023-01-15T14:30:59.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037153499999999999 2023-01-15T14:30:59.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:59.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553517+0000", 2023-01-15T14:30:59.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4687000000000002e-05 2023-01-15T14:30:59.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:59.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553537+0000", 2023-01-15T14:30:59.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.967e-05 2023-01-15T14:30:59.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:59.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553683+0000", 2023-01-15T14:30:59.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000146341 2023-01-15T14:30:59.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:59.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553880+0000", 2023-01-15T14:30:59.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019671900000000001 2023-01-15T14:30:59.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:59.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554219+0000", 2023-01-15T14:30:59.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033893299999999998 2023-01-15T14:30:59.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:59.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554536+0000", 2023-01-15T14:30:59.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316576 2023-01-15T14:30:59.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:59.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554562+0000", 2023-01-15T14:30:59.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5905999999999999e-05 2023-01-15T14:30:59.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:59.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554801+0000", 2023-01-15T14:30:59.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000239791 2023-01-15T14:30:59.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:59.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555274+0000", 2023-01-15T14:30:59.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047294500000000001 2023-01-15T14:30:59.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757267+0000", 2023-01-15T14:30:59.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20199229399999999 2023-01-15T14:30:59.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958174+0000", 2023-01-15T14:30:59.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20090739499999999 2023-01-15T14:30:59.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:59.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958260+0000", 2023-01-15T14:30:59.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6316999999999994e-05 2023-01-15T14:30:59.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:59.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958351+0000", 2023-01-15T14:30:59.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0573000000000002e-05 2023-01-15T14:30:59.557 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.557 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:59.557 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.557 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:204 3.a 3:52df5138:::smithi09025264-25:head [write 1309629~542666] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:59.557 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:30:59.557 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.28035214499999, 2023-01-15T14:30:59.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40500913999999999, 2023-01-15T14:30:59.557 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 204 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:59.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:30:59.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:59.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553575+0000", 2023-01-15T14:30:59.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:59.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:59.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554126+0000", 2023-01-15T14:30:59.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055090499999999997 2023-01-15T14:30:59.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:59.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554150+0000", 2023-01-15T14:30:59.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3387000000000002e-05 2023-01-15T14:30:59.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:59.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554165+0000", 2023-01-15T14:30:59.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5116000000000001e-05 2023-01-15T14:30:59.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:59.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554626+0000", 2023-01-15T14:30:59.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046093499999999999 2023-01-15T14:30:59.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:59.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554738+0000", 2023-01-15T14:30:59.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112446 2023-01-15T14:30:59.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:59.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554898+0000", 2023-01-15T14:30:59.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016003100000000001 2023-01-15T14:30:59.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:59.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555182+0000", 2023-01-15T14:30:59.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028382700000000001 2023-01-15T14:30:59.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:59.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555199+0000", 2023-01-15T14:30:59.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6969e-05 2023-01-15T14:30:59.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:59.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555449+0000", 2023-01-15T14:30:59.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000250265 2023-01-15T14:30:59.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:59.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555574+0000", 2023-01-15T14:30:59.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124628 2023-01-15T14:30:59.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757316+0000", 2023-01-15T14:30:59.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20174214300000001 2023-01-15T14:30:59.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958460+0000", 2023-01-15T14:30:59.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201144464 2023-01-15T14:30:59.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958534+0000", 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3380000000000006e-05 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958586+0000", 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2370000000000002e-05 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:59.566 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:205 3.a 3:52df5138:::smithi09025264-25:head [write 2323973~516377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.27973405, 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404601398, 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:59.567 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 205 2023-01-15T14:30:59.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.568 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:59.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:59.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:30:59.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:59.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:30:59.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:59.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554193+0000", 2023-01-15T14:30:59.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:30:59.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:59.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554850+0000", 2023-01-15T14:30:59.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065674799999999997 2023-01-15T14:30:59.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:59.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554873+0000", 2023-01-15T14:30:59.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2935999999999999e-05 2023-01-15T14:30:59.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:59.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.654962+0000", 2023-01-15T14:30:59.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088949 2023-01-15T14:30:59.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:59.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655107+0000", 2023-01-15T14:30:59.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144759 2023-01-15T14:30:59.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:59.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655373+0000", 2023-01-15T14:30:59.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026684899999999999 2023-01-15T14:30:59.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:59.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655720+0000", 2023-01-15T14:30:59.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034601899999999998 2023-01-15T14:30:59.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:59.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656168+0000", 2023-01-15T14:30:59.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044851899999999998 2023-01-15T14:30:59.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:59.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656190+0000", 2023-01-15T14:30:59.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1817000000000001e-05 2023-01-15T14:30:59.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:59.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656574+0000", 2023-01-15T14:30:59.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038404200000000002 2023-01-15T14:30:59.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:59.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657649+0000", 2023-01-15T14:30:59.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010746670000000001 2023-01-15T14:30:59.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757539+0000", 2023-01-15T14:30:59.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099890899000000005 2023-01-15T14:30:59.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958683+0000", 2023-01-15T14:30:59.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20114332400000001 2023-01-15T14:30:59.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:59.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958746+0000", 2023-01-15T14:30:59.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3509000000000005e-05 2023-01-15T14:30:59.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:59.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958796+0000", 2023-01-15T14:30:59.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0176999999999997e-05 2023-01-15T14:30:59.580 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.580 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:59.580 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.581 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:358 3.a 3:52df5138:::smithi09025264-25:head [write 709262~697258] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:59.581 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:30:59.581 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 138.78314651700001, 2023-01-15T14:30:59.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50589529799999999, 2023-01-15T14:30:59.581 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:59.581 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:59.581 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:59.581 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:59.582 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:59.582 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 358 2023-01-15T14:30:59.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.582 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:59.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:59.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:30:59.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:59.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:30:59.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:59.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050780+0000", 2023-01-15T14:30:59.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:30:59.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:59.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051369+0000", 2023-01-15T14:30:59.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058926200000000001 2023-01-15T14:30:59.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:59.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051390+0000", 2023-01-15T14:30:59.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1426000000000001e-05 2023-01-15T14:30:59.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:59.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051403+0000", 2023-01-15T14:30:59.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:30:59.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:59.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051520+0000", 2023-01-15T14:30:59.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117318 2023-01-15T14:30:59.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:59.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051785+0000", 2023-01-15T14:30:59.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000265018 2023-01-15T14:30:59.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:59.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052173+0000", 2023-01-15T14:30:59.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038734799999999998 2023-01-15T14:30:59.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:59.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052705+0000", 2023-01-15T14:30:59.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053209899999999996 2023-01-15T14:30:59.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:59.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052727+0000", 2023-01-15T14:30:59.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2362e-05 2023-01-15T14:30:59.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:59.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053205+0000", 2023-01-15T14:30:59.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047787200000000001 2023-01-15T14:30:59.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:59.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055711+0000", 2023-01-15T14:30:59.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0025054880000000002 2023-01-15T14:30:59.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.255549+0000", 2023-01-15T14:30:59.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199838285 2023-01-15T14:30:59.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556501+0000", 2023-01-15T14:30:59.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30095197499999998 2023-01-15T14:30:59.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:59.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556613+0000", 2023-01-15T14:30:59.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111799 2023-01-15T14:30:59.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:59.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556678+0000", 2023-01-15T14:30:59.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5220000000000002e-05 2023-01-15T14:30:59.591 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.591 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:59.591 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.591 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:359 3.a 3:52df5138:::smithi09025264-25:head [write 2012635~572018] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:59.591 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:30:59.592 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 138.782495923, 2023-01-15T14:30:59.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50546442499999999, 2023-01-15T14:30:59.592 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:59.592 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:59.592 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:59.592 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:59.592 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:59.592 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 359 2023-01-15T14:30:59.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.593 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:59.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:59.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:30:59.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:59.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:30:59.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:59.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051432+0000", 2023-01-15T14:30:59.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:59.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:59.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051808+0000", 2023-01-15T14:30:59.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000376074 2023-01-15T14:30:59.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:59.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051834+0000", 2023-01-15T14:30:59.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6018999999999999e-05 2023-01-15T14:30:59.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:59.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051869+0000", 2023-01-15T14:30:59.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4839999999999998e-05 2023-01-15T14:30:59.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:59.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052857+0000", 2023-01-15T14:30:59.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00098761399999999994 2023-01-15T14:30:59.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:59.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053071+0000", 2023-01-15T14:30:59.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000214112 2023-01-15T14:30:59.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:59.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053876+0000", 2023-01-15T14:30:59.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000804873 2023-01-15T14:30:59.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:59.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054460+0000", 2023-01-15T14:30:59.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058436100000000004 2023-01-15T14:30:59.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:59.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054486+0000", 2023-01-15T14:30:59.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069999999999999e-05 2023-01-15T14:30:59.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:59.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054773+0000", 2023-01-15T14:30:59.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028716700000000001 2023-01-15T14:30:59.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.753 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:59.753 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055736+0000", 2023-01-15T14:30:59.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00096295500000000004 2023-01-15T14:30:59.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.753 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.355784+0000", 2023-01-15T14:30:59.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30004825899999998 2023-01-15T14:30:59.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556757+0000", 2023-01-15T14:30:59.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20097241199999999 2023-01-15T14:30:59.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:30:59.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556830+0000", 2023-01-15T14:30:59.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3633000000000005e-05 2023-01-15T14:30:59.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:30:59.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556898+0000", 2023-01-15T14:30:59.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7212999999999995e-05 2023-01-15T14:30:59.759 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.760 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:30:59.760 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:30:59.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.760 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:360 3.a 3:52df5138:::smithi09025264-25:head [write 3046388~62874] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:30:59.761 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:30:59.761 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 138.78204316700001, 2023-01-15T14:30:59.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50519877000000002, 2023-01-15T14:30:59.761 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:30:59.761 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:30:59.762 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:30:59.762 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:30:59.762 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:30:59.762 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 360 2023-01-15T14:30:59.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.763 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:30:59.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:30:59.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:30:59.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:30:59.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:30:59.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:30:59.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:30:59.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051885+0000", 2023-01-15T14:30:59.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:30:59.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:30:59.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051935+0000", 2023-01-15T14:30:59.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9826e-05 2023-01-15T14:30:59.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:30:59.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051958+0000", 2023-01-15T14:30:59.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3003e-05 2023-01-15T14:30:59.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:30:59.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051965+0000", 2023-01-15T14:30:59.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6129999999999998e-06 2023-01-15T14:30:59.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:30:59.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054655+0000", 2023-01-15T14:30:59.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0026896860000000002 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054869+0000", 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021348600000000001 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055198+0000", 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032968199999999998 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.954 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:30:59.954 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055605+0000", 2023-01-15T14:30:59.954 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040634200000000002 2023-01-15T14:30:59.954 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.954 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.954 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:30:59.954 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055622+0000", 2023-01-15T14:30:59.954 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7020999999999998e-05 2023-01-15T14:30:59.954 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.954 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.954 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055702+0000", 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0835999999999997e-05 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055754+0000", 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1175999999999999e-05 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.455996+0000", 2023-01-15T14:30:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40024198599999999 2023-01-15T14:31:00.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556952+0000", 2023-01-15T14:31:00.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10095644400000001 2023-01-15T14:31:00.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557019+0000", 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6746000000000001e-05 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557085+0000", 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5885999999999999e-05 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.142 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.143 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:361 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 3109262] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:00.143 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:31:00.143 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 138.78195326100001, 2023-01-15T14:31:00.143 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60469846000000005, 2023-01-15T14:31:00.143 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:00.143 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:00.143 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:00.143 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:00.144 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:00.144 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 361 2023-01-15T14:31:00.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.144 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:00.144 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.144 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:00.144 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:31:00.144 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051975+0000", 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:00.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051980+0000", 2023-01-15T14:31:00.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2170000000000002e-06 2023-01-15T14:31:00.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.146 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:00.146 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051989+0000", 2023-01-15T14:31:00.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5299999999999996e-06 2023-01-15T14:31:00.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.146 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:00.146 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051995+0000", 2023-01-15T14:31:00.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1369999999999999e-06 2023-01-15T14:31:00.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055830+0000", 2023-01-15T14:31:00.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0038348549999999999 2023-01-15T14:31:00.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:00.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056101+0000", 2023-01-15T14:31:00.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027095800000000002 2023-01-15T14:31:00.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:00.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056560+0000", 2023-01-15T14:31:00.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045868599999999998 2023-01-15T14:31:00.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:00.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056954+0000", 2023-01-15T14:31:00.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039487600000000001 2023-01-15T14:31:00.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057004+0000", 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9490000000000002e-05 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057072+0000", 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8502000000000002e-05 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.158224+0000", 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101151422 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556201+0000", 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39797714699999998 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656421+0000", 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100219549 2023-01-15T14:31:00.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656515+0000", 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4048999999999993e-05 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656674+0000", 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000159696 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.151 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.152 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:362 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:00.152 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:31:00.152 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 138.781925831, 2023-01-15T14:31:00.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60517504499999997, 2023-01-15T14:31:00.152 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:00.152 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:00.152 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:00.152 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:00.152 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:00.153 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 362 2023-01-15T14:31:00.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.153 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:00.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:00.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:31:00.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:00.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:31:00.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:00.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:31:00.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:00.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:00.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052007+0000", 2023-01-15T14:31:00.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.9759999999999997e-06 2023-01-15T14:31:00.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:00.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052016+0000", 2023-01-15T14:31:00.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4069999999999999e-06 2023-01-15T14:31:00.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:00.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052022+0000", 2023-01-15T14:31:00.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4670000000000001e-06 2023-01-15T14:31:00.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157159+0000", 2023-01-15T14:31:00.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.105136825 2023-01-15T14:31:00.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:00.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157325+0000", 2023-01-15T14:31:00.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000165262 2023-01-15T14:31:00.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656814+0000", 2023-01-15T14:31:00.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49948963200000002 2023-01-15T14:31:00.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:00.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656993+0000", 2023-01-15T14:31:00.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000178371 2023-01-15T14:31:00.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:00.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.657178+0000", 2023-01-15T14:31:00.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185825 2023-01-15T14:31:00.362 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.362 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.362 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.363 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:601 3.a 3:5ba385f1:::smithi09025264-44:head [setxattr _header (57) in=7b,truncate 3663893] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:31:00.363 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:31:00.363 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 118.891603195, 2023-01-15T14:31:00.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404742724, 2023-01-15T14:31:00.363 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:00.363 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:00.363 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:00.363 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:00.363 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:00.364 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 601 2023-01-15T14:31:00.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.364 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:00.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.364 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:00.364 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:31:00.364 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.364 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:00.364 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:31:00.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:00.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942323+0000", 2023-01-15T14:31:00.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:00.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:00.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942341+0000", 2023-01-15T14:31:00.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7659000000000002e-05 2023-01-15T14:31:00.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:00.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942365+0000", 2023-01-15T14:31:00.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4256999999999998e-05 2023-01-15T14:31:00.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:00.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042495+0000", 2023-01-15T14:31:00.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013043200000001 2023-01-15T14:31:00.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.144809+0000", 2023-01-15T14:31:00.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102314184 2023-01-15T14:31:00.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:00.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145098+0000", 2023-01-15T14:31:00.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000289036 2023-01-15T14:31:00.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:00.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145363+0000", 2023-01-15T14:31:00.368 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026449299999999998 2023-01-15T14:31:00.368 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:00.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145657+0000", 2023-01-15T14:31:00.368 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029433600000000002 2023-01-15T14:31:00.368 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:00.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145678+0000", 2023-01-15T14:31:00.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0939000000000001e-05 2023-01-15T14:31:00.369 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:00.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145853+0000", 2023-01-15T14:31:00.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017511800000000001 2023-01-15T14:31:00.369 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:00.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146557+0000", 2023-01-15T14:31:00.370 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00070337899999999996 2023-01-15T14:31:00.370 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.370 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.370 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.370 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146845+0000", 2023-01-15T14:31:00.370 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028790599999999999 2023-01-15T14:31:00.370 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.370 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.370 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.370 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346819+0000", 2023-01-15T14:31:00.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199974661 2023-01-15T14:31:00.371 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.371 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.371 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:00.371 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346919+0000", 2023-01-15T14:31:00.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9234999999999994e-05 2023-01-15T14:31:00.371 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.371 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.371 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:00.371 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.347069+0000", 2023-01-15T14:31:00.372 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150128 2023-01-15T14:31:00.372 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.372 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.372 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.372 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.372 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.372 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1153 3.a 3:5919d83e:::smithi09025264-13:head [write 1566383~595717] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:31:00.372 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:31:00.372 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 73.043378141000005, 2023-01-15T14:31:00.372 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.38805106299999997, 2023-01-15T14:31:00.372 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:00.373 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:00.373 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:00.373 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:00.373 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:00.373 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1153 2023-01-15T14:31:00.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.373 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:00.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:00.373 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:31:00.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790531+0000", 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999795 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791189+0000", 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065897000000000004 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791215+0000", 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5700000000000001e-05 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:00.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791232+0000", 2023-01-15T14:31:00.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6546e-05 2023-01-15T14:31:00.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791392+0000", 2023-01-15T14:31:00.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016051699999999999 2023-01-15T14:31:00.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:00.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791408+0000", 2023-01-15T14:31:00.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5566999999999999e-05 2023-01-15T14:31:00.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.974825+0000", 2023-01-15T14:31:00.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18341675599999999 2023-01-15T14:31:00.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:00.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975000+0000", 2023-01-15T14:31:00.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017548299999999999 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975255+0000", 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254658 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975914+0000", 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065881800000000003 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.378 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.379 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:00.379 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975931+0000", 2023-01-15T14:31:00.379 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7523000000000001e-05 2023-01-15T14:31:00.379 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.379 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:00.379 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976267+0000", 2023-01-15T14:31:00.379 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033545299999999999 2023-01-15T14:31:00.379 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.379 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:00.380 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978320+0000", 2023-01-15T14:31:00.380 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020537110000000002 2023-01-15T14:31:00.380 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.380 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.380 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978647+0000", 2023-01-15T14:31:00.380 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032660700000000003 2023-01-15T14:31:00.380 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.380 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.381 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178434+0000", 2023-01-15T14:31:00.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19978705699999999 2023-01-15T14:31:00.381 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.381 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.381 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:00.381 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178527+0000", 2023-01-15T14:31:00.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3523999999999994e-05 2023-01-15T14:31:00.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:00.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178602+0000", 2023-01-15T14:31:00.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4622000000000005e-05 2023-01-15T14:31:00.570 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.570 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.570 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.570 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1154 3.a 3:5919d83e:::smithi09025264-13:head [write 2888822~102824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:31:00.570 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:31:00.570 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 73.038036813999994, 2023-01-15T14:31:00.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48318778800000001, 2023-01-15T14:31:00.570 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:00.570 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1154 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:00.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:31:00.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:00.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795890+0000", 2023-01-15T14:31:00.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:00.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:00.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795967+0000", 2023-01-15T14:31:00.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6304999999999995e-05 2023-01-15T14:31:00.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:00.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796000+0000", 2023-01-15T14:31:00.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3593000000000002e-05 2023-01-15T14:31:00.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:00.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796019+0000", 2023-01-15T14:31:00.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8955999999999999e-05 2023-01-15T14:31:00.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796189+0000", 2023-01-15T14:31:00.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017032700000000001 2023-01-15T14:31:00.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:00.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796208+0000", 2023-01-15T14:31:00.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8389e-05 2023-01-15T14:31:00.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976335+0000", 2023-01-15T14:31:00.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18012763100000001 2023-01-15T14:31:00.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:00.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976487+0000", 2023-01-15T14:31:00.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015173000000000001 2023-01-15T14:31:00.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:00.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976763+0000", 2023-01-15T14:31:00.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027597400000000001 2023-01-15T14:31:00.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977085+0000", 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032233400000000002 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977103+0000", 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7727000000000002e-05 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:00.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977238+0000", 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013525399999999999 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978347+0000", 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011084350000000001 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.078223+0000", 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099875830999999998 2023-01-15T14:31:00.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.278897+0000", 2023-01-15T14:31:00.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20067421599999999 2023-01-15T14:31:00.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:00.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279021+0000", 2023-01-15T14:31:00.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124354 2023-01-15T14:31:00.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:00.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279080+0000", 2023-01-15T14:31:00.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8811999999999999e-05 2023-01-15T14:31:00.584 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.584 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.585 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.585 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1155 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 2991646] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:31:00.585 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:31:00.585 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 73.037863795000007, 2023-01-15T14:31:00.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48334445999999998, 2023-01-15T14:31:00.585 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:00.586 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:00.586 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:00.586 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:00.586 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:00.586 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1155 2023-01-15T14:31:00.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.586 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:00.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:00.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:31:00.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:00.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:31:00.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:00.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796064+0000", 2023-01-15T14:31:00.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:00.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:00.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796074+0000", 2023-01-15T14:31:00.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:31:00.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:00.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796096+0000", 2023-01-15T14:31:00.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2422999999999999e-05 2023-01-15T14:31:00.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:00.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796109+0000", 2023-01-15T14:31:00.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:31:00.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796274+0000", 2023-01-15T14:31:00.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000164976 2023-01-15T14:31:00.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:00.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796283+0000", 2023-01-15T14:31:00.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9539999999999993e-06 2023-01-15T14:31:00.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977269+0000", 2023-01-15T14:31:00.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18098651700000001 2023-01-15T14:31:00.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:00.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977438+0000", 2023-01-15T14:31:00.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168913 2023-01-15T14:31:00.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:00.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977865+0000", 2023-01-15T14:31:00.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042707500000000001 2023-01-15T14:31:00.594 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:00.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978241+0000", 2023-01-15T14:31:00.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037593000000000001 2023-01-15T14:31:00.594 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.595 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:00.595 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978253+0000", 2023-01-15T14:31:00.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1817000000000001e-05 2023-01-15T14:31:00.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.595 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.595 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:00.595 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978288+0000", 2023-01-15T14:31:00.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4586999999999999e-05 2023-01-15T14:31:00.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:00.596 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978370+0000", 2023-01-15T14:31:00.596 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1712999999999995e-05 2023-01-15T14:31:00.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.596 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279124+0000", 2023-01-15T14:31:00.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30075418799999998 2023-01-15T14:31:00.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279208+0000", 2023-01-15T14:31:00.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4500999999999996e-05 2023-01-15T14:31:00.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:00.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279294+0000", 2023-01-15T14:31:00.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5680999999999995e-05 2023-01-15T14:31:00.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:00.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279410+0000", 2023-01-15T14:31:00.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115878 2023-01-15T14:31:00.599 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.599 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.599 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.599 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1156 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:31:00.599 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:31:00.600 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 73.037752269999999, 2023-01-15T14:31:00.600 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.58375825000000003, 2023-01-15T14:31:00.600 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:00.600 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:00.600 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:00.600 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:00.600 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:00.600 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1156 2023-01-15T14:31:00.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.601 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:00.601 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.601 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:00.601 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:31:00.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.601 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.601 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:00.602 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:31:00.602 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.602 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:00.602 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796176+0000", 2023-01-15T14:31:00.602 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:00.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.602 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:00.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796182+0000", 2023-01-15T14:31:00.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6799999999999998e-06 2023-01-15T14:31:00.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:00.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796200+0000", 2023-01-15T14:31:00.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8136999999999999e-05 2023-01-15T14:31:00.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:00.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796217+0000", 2023-01-15T14:31:00.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7036000000000001e-05 2023-01-15T14:31:00.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796337+0000", 2023-01-15T14:31:00.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120674 2023-01-15T14:31:00.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:00.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796345+0000", 2023-01-15T14:31:00.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4540000000000001e-06 2023-01-15T14:31:00.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978423+0000", 2023-01-15T14:31:00.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18207789599999999 2023-01-15T14:31:00.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:00.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978577+0000", 2023-01-15T14:31:00.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000153805 2023-01-15T14:31:00.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379596+0000", 2023-01-15T14:31:00.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.401019348 2023-01-15T14:31:00.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:00.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379754+0000", 2023-01-15T14:31:00.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015840400000000001 2023-01-15T14:31:00.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:00.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379935+0000", 2023-01-15T14:31:00.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018081299999999999 2023-01-15T14:31:00.735 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.735 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.735 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.735 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.735 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1160 3.a 3:52df5138:::smithi09025264-25:head [write 2632598~375855] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:31:00.735 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:31:00.735 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 72.078945683000001, 2023-01-15T14:31:00.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.398337832, 2023-01-15T14:31:00.735 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:00.736 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:00.736 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:00.736 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:00.736 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:00.736 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1160 2023-01-15T14:31:00.736 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.736 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:00.736 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:00.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:31:00.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:00.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:31:00.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:00.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754981+0000", 2023-01-15T14:31:00.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:00.738 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:00.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755360+0000", 2023-01-15T14:31:00.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037899899999999998 2023-01-15T14:31:00.738 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:00.739 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755392+0000", 2023-01-15T14:31:00.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1977000000000002e-05 2023-01-15T14:31:00.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.739 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.739 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:00.739 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755407+0000", 2023-01-15T14:31:00.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4446e-05 2023-01-15T14:31:00.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.739 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.739 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.739 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949294+0000", 2023-01-15T14:31:00.740 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.193887483 2023-01-15T14:31:00.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.740 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:00.740 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949478+0000", 2023-01-15T14:31:00.740 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018400900000000001 2023-01-15T14:31:00.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.740 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:00.740 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949904+0000", 2023-01-15T14:31:00.740 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042552200000000001 2023-01-15T14:31:00.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:00.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950531+0000", 2023-01-15T14:31:00.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062732399999999996 2023-01-15T14:31:00.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:00.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950552+0000", 2023-01-15T14:31:00.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1229999999999998e-05 2023-01-15T14:31:00.742 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:00.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951094+0000", 2023-01-15T14:31:00.742 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054169000000000005 2023-01-15T14:31:00.742 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:00.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951256+0000", 2023-01-15T14:31:00.742 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016176299999999999 2023-01-15T14:31:00.742 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052665+0000", 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101409257 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153132+0000", 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100466783 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.743 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:00.744 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153223+0000", 2023-01-15T14:31:00.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.1415000000000005e-05 2023-01-15T14:31:00.744 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.744 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.744 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:00.744 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153321+0000", 2023-01-15T14:31:00.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8181000000000001e-05 2023-01-15T14:31:00.744 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.744 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.744 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.745 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1161 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (57) in=7b,truncate 3008453] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:31:00.745 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:31:00.745 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 72.078502095000005, 2023-01-15T14:31:00.745 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49840109900000001, 2023-01-15T14:31:00.745 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:00.745 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:00.745 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:00.745 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:00.746 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:00.746 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1161 2023-01-15T14:31:00.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.746 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:00.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:00.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:31:00.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755426+0000", 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755433+0000", 2023-01-15T14:31:00.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.793e-06 2023-01-15T14:31:00.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:00.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755448+0000", 2023-01-15T14:31:00.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5047e-05 2023-01-15T14:31:00.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:00.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755455+0000", 2023-01-15T14:31:00.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3590000000000002e-06 2023-01-15T14:31:00.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951466+0000", 2023-01-15T14:31:00.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.196011038 2023-01-15T14:31:00.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:00.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951611+0000", 2023-01-15T14:31:00.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000145184 2023-01-15T14:31:00.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:00.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951921+0000", 2023-01-15T14:31:00.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030989100000000002 2023-01-15T14:31:00.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:00.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952187+0000", 2023-01-15T14:31:00.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026623100000000002 2023-01-15T14:31:00.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:00.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952207+0000", 2023-01-15T14:31:00.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9510000000000001e-05 2023-01-15T14:31:00.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:00.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952271+0000", 2023-01-15T14:31:00.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4511999999999995e-05 2023-01-15T14:31:00.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:00.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052716+0000", 2023-01-15T14:31:00.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10044497300000001 2023-01-15T14:31:00.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253531+0000", 2023-01-15T14:31:00.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20081490699999999 2023-01-15T14:31:00.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253635+0000", 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000103838 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253727+0000", 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2298000000000003e-05 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253828+0000", 2023-01-15T14:31:00.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000100655 2023-01-15T14:31:00.754 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.754 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.754 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.754 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1162 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:31:00.754 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:31:00.754 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 72.078462700000003, 2023-01-15T14:31:00.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.498775839, 2023-01-15T14:31:00.754 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:00.754 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:00.755 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:00.755 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:00.755 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:00.755 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1162 2023-01-15T14:31:00.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.755 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:00.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.755 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:00.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:31:00.755 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:00.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:31:00.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:00.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:00.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:31:00.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:00.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755470+0000", 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0570000000000003e-06 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755484+0000", 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4229999999999999e-05 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:00.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755492+0000", 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6070000000000002e-06 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.152842+0000", 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39734988799999998 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153014+0000", 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017236000000000001 2023-01-15T14:31:00.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:00.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253921+0000", 2023-01-15T14:31:00.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100907497 2023-01-15T14:31:00.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:00.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254060+0000", 2023-01-15T14:31:00.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013815400000000001 2023-01-15T14:31:00.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:00.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:00.760 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:00.760 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254242+0000", 2023-01-15T14:31:00.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000182726 2023-01-15T14:31:00.760 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.760 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.760 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.760 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:00.761 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:00.761 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:00.761 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:31:00.762 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:30:58.931+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:00.764 INFO:tasks.rados.rados.0.smithi090.stdout:676: done (1 left) 2023-01-15T14:31:00.764 INFO:tasks.rados.rados.0.smithi090.stdout:677: write oid 29 current snap is 66 2023-01-15T14:31:00.764 INFO:tasks.rados.rados.0.smithi090.stdout:677: seq_num 267 ranges {487469=563838,1607751=551752} 2023-01-15T14:31:00.764 INFO:tasks.rados.rados.0.smithi090.stdout:677: writing smithi09025264-29 from 487469 to 1051307 tid 1 2023-01-15T14:31:00.764 INFO:tasks.rados.rados.0.smithi090.stdout:677: writing smithi09025264-29 from 1607751 to 2159503 tid 2 2023-01-15T14:31:00.765 INFO:tasks.rados.rados.0.smithi090.stdout:678: delete oid 16 current snap is 66 2023-01-15T14:31:00.765 INFO:tasks.rados.rados.0.smithi090.stdout:675: finishing write tid 1 to smithi09025264-24 2023-01-15T14:31:00.765 INFO:tasks.rados.rados.0.smithi090.stdout:675: finishing write tid 2 to smithi09025264-24 2023-01-15T14:31:00.765 INFO:tasks.rados.rados.0.smithi090.stdout:675: finishing write tid 3 to smithi09025264-24 2023-01-15T14:31:00.765 INFO:tasks.rados.rados.0.smithi090.stdout:675: finishing write tid 4 to smithi09025264-24 2023-01-15T14:31:00.765 INFO:tasks.rados.rados.0.smithi090.stdout:675: oid 24 updating version 0 to 230 2023-01-15T14:31:00.765 INFO:tasks.rados.rados.0.smithi090.stdout:675: oid 24 updating version 230 to 231 2023-01-15T14:31:00.765 INFO:tasks.rados.rados.0.smithi090.stdout:675: oid 24 updating version 231 to 232 2023-01-15T14:31:00.765 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 232 (ObjNum 266 snap 66 seq_num 266) dirty exists 2023-01-15T14:31:00.766 INFO:tasks.rados.rados.0.smithi090.stdout:675: left oid 24 (ObjNum 266 snap 66 seq_num 266) 2023-01-15T14:31:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:00.767 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:00.768 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:00.759+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:31:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:31:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:31:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:31:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:31:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:31:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:31:00.769 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:31:00.772 INFO:tasks.rados.rados.0.smithi090.stdout:677: finishing write tid 1 to smithi09025264-29 2023-01-15T14:31:00.772 INFO:tasks.rados.rados.0.smithi090.stdout:677: finishing write tid 2 to smithi09025264-29 2023-01-15T14:31:00.772 INFO:tasks.rados.rados.0.smithi090.stdout:677: finishing write tid 3 to smithi09025264-29 2023-01-15T14:31:00.772 INFO:tasks.rados.rados.0.smithi090.stdout:677: finishing write tid 4 to smithi09025264-29 2023-01-15T14:31:00.772 INFO:tasks.rados.rados.0.smithi090.stdout:677: oid 29 updating version 0 to 135 2023-01-15T14:31:00.772 INFO:tasks.rados.rados.0.smithi090.stdout:677: oid 29 updating version 135 to 136 2023-01-15T14:31:00.772 INFO:tasks.rados.rados.0.smithi090.stdout:677: oid 29 version 136 is already newer than 134 2023-01-15T14:31:00.772 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 136 (ObjNum 267 snap 66 seq_num 267) dirty exists 2023-01-15T14:31:00.773 INFO:tasks.rados.rados.0.smithi090.stdout:677: left oid 29 (ObjNum 267 snap 66 seq_num 267) 2023-01-15T14:31:00.773 INFO:tasks.rados.rados.0.smithi090.stdout:675: done (2 left) 2023-01-15T14:31:00.773 INFO:tasks.rados.rados.0.smithi090.stdout:677: done (1 left) 2023-01-15T14:31:00.773 INFO:tasks.rados.rados.0.smithi090.stdout:678: done (0 left) 2023-01-15T14:31:00.773 INFO:tasks.rados.rados.0.smithi090.stdout:679: rollback oid 6 current snap is 66 2023-01-15T14:31:00.773 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 6 to 64 2023-01-15T14:31:00.773 INFO:tasks.rados.rados.0.smithi090.stdout:680: write oid 35 current snap is 66 2023-01-15T14:31:00.773 INFO:tasks.rados.rados.0.smithi090.stdout:680: seq_num 269 ranges {521083=714336,1724715=555421} 2023-01-15T14:31:00.773 INFO:tasks.rados.rados.0.smithi090.stdout:680: writing smithi09025264-35 from 521083 to 1235419 tid 1 2023-01-15T14:31:00.773 INFO:tasks.rados.rados.0.smithi090.stdout:680: writing smithi09025264-35 from 1724715 to 2280136 tid 2 2023-01-15T14:31:00.897 INFO:tasks.rados.rados.0.smithi090.stdout:681: snap_create 2023-01-15T14:31:00.897 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:00.897 INFO:tasks.rados.rados.0.smithi090.stdout:679: finishing rollback tid 0 to smithi09025264-6 2023-01-15T14:31:00.897 INFO:tasks.rados.rados.0.smithi090.stdout:679: finishing rollback tid 1 to smithi09025264-6 2023-01-15T14:31:00.897 INFO:tasks.rados.rados.0.smithi090.stdout:679: finishing rollback tid 2 to smithi09025264-6 2023-01-15T14:31:00.898 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 67 (ObjNum 242 snap 55 seq_num 242) dirty exists 2023-01-15T14:31:00.898 INFO:tasks.rados.rados.0.smithi090.stdout:679: done (1 left) 2023-01-15T14:31:00.898 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:00.898 INFO:tasks.rados.rados.0.smithi090.stdout:680: finishing write tid 1 to smithi09025264-35 2023-01-15T14:31:00.898 INFO:tasks.rados.rados.0.smithi090.stdout:680: finishing write tid 2 to smithi09025264-35 2023-01-15T14:31:00.899 INFO:tasks.rados.rados.0.smithi090.stdout:680: finishing write tid 3 to smithi09025264-35 2023-01-15T14:31:00.899 INFO:tasks.rados.rados.0.smithi090.stdout:680: finishing write tid 4 to smithi09025264-35 2023-01-15T14:31:00.899 INFO:tasks.rados.rados.0.smithi090.stdout:680: oid 35 updating version 0 to 215 2023-01-15T14:31:00.899 INFO:tasks.rados.rados.0.smithi090.stdout:680: oid 35 version 215 is already newer than 214 2023-01-15T14:31:00.899 INFO:tasks.rados.rados.0.smithi090.stdout:680: oid 35 updating version 215 to 216 2023-01-15T14:31:00.900 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 216 (ObjNum 269 snap 66 seq_num 269) dirty exists 2023-01-15T14:31:00.901 INFO:tasks.rados.rados.0.smithi090.stdout:680: left oid 35 (ObjNum 269 snap 66 seq_num 269) 2023-01-15T14:31:00.901 INFO:tasks.rados.rados.0.smithi090.stdout:680: done (0 left) 2023-01-15T14:31:00.901 INFO:tasks.rados.rados.0.smithi090.stdout:681: done (0 left) 2023-01-15T14:31:00.901 INFO:tasks.rados.rados.0.smithi090.stdout:682: read oid 1 snap -1 2023-01-15T14:31:00.901 INFO:tasks.rados.rados.0.smithi090.stdout:682: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:31:00.902 INFO:tasks.rados.rados.0.smithi090.stdout:683: snap_create 2023-01-15T14:31:00.902 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:00.903 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:31:01.046 INFO:tasks.rados.rados.0.smithi090.stdout:682: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:31:01.046 INFO:tasks.rados.rados.0.smithi090.stdout:682: done (0 left) 2023-01-15T14:31:01.048 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:01.048 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:01.049 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:31:01.049 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:01.043+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:01.050 INFO:tasks.rados.rados.0.smithi090.stdout:683: done (0 left) 2023-01-15T14:31:01.050 INFO:tasks.rados.rados.0.smithi090.stdout:684: snap_create 2023-01-15T14:31:01.070 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:31:01.070 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:31:01.071 INFO:tasks.thrashosds.thrasher:Reweighting osd 0 to 0.3336354712820637 2023-01-15T14:31:01.071 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 0.3336354712820637 2023-01-15T14:31:01.149 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:01.147+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:01.209 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:01.209 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:01.219 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:31:01.250 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:01.248+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:01.320 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:01.321 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:01.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:31:01.351 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:01.347+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:01.439 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:01.439 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:01.449 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:31:01.452 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:01.451+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:01.549 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:01.591 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:01.593 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:31:01.593 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:01.588+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:01.694 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:01.692+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:01.702 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:01.702 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:01.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:31:01.785 INFO:tasks.rados.rados.0.smithi090.stdout:684: done (0 left) 2023-01-15T14:31:01.796 INFO:tasks.rados.rados.0.smithi090.stdout:685: snap_create 2023-01-15T14:31:01.798 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.0 to 0.333635 (5569) 2023-01-15T14:31:01.798 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:01.796+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:01.824 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:01.824 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:01.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:31:01.967 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:01.963+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:02.067 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:02.063+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:02.131 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:02.132 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:02.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:31:02.168 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:02.164+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:02.240 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:02.407 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:02.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:31:02.409 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:02.407+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:02.510 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:02.507+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:02.520 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:02.520 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:02.530 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:31:02.612 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:02.607+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:02.635 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:02.820 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:02.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:31:02.822 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:02.818+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:02.823 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:31:02.823 INFO:tasks.rados.rados.0.smithi090.stdout:685: done (0 left) 2023-01-15T14:31:02.823 INFO:tasks.rados.rados.0.smithi090.stdout:686: snap_remove snap 66 2023-01-15T14:31:02.922 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:02.920+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:02.938 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:02.938 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:02.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:31:02.966 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:02.988 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:03.009 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:03.023 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:03.020+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:03.026 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:03.046 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:03.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:31:03.069 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:03.075 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:03.075 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:03.076 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:03.083 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:31:03.095 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:03.125 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:03.122+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:03.126 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:03.128 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:31:03.128 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:31:03.128 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:31:03.128 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:31:03.129 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:31:03.129 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:31:03.129 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:31:03.129 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:31:03.207 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:03.230 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:03.230 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:03.234 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:31:03.235 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:03.228+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:03.334 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:03.332+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:03.643 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:166 3.8 3:1e54a7f5:::smithi09025264-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 156.22314623899999, 2023-01-15T14:31:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62664632300000001, 2023-01-15T14:31:03.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 166 2023-01-15T14:31:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:03.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114713+0000", 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:03.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115178+0000", 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046493199999999999 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115208+0000", 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0689999999999999e-05 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115221+0000", 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3013000000000001e-05 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115334+0000", 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112403 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115505+0000", 2023-01-15T14:31:03.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170915 2023-01-15T14:31:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333496+0000", 2023-01-15T14:31:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217991667 2023-01-15T14:31:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:03.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333606+0000", 2023-01-15T14:31:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109967 2023-01-15T14:31:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333796+0000", 2023-01-15T14:31:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018928099999999999 2023-01-15T14:31:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:03.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334289+0000", 2023-01-15T14:31:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049322600000000004 2023-01-15T14:31:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334339+0000", 2023-01-15T14:31:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0439999999999998e-05 2023-01-15T14:31:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334990+0000", 2023-01-15T14:31:03.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065082300000000003 2023-01-15T14:31:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335100+0000", 2023-01-15T14:31:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109357 2023-01-15T14:31:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.640854+0000", 2023-01-15T14:31:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30575451300000001 2023-01-15T14:31:03.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741190+0000", 2023-01-15T14:31:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100335568 2023-01-15T14:31:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741263+0000", 2023-01-15T14:31:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3275999999999994e-05 2023-01-15T14:31:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741361+0000", 2023-01-15T14:31:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8435000000000002e-05 2023-01-15T14:31:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:167 3.8 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:03.653 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 156.22258155, 2023-01-15T14:31:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62638390099999997, 2023-01-15T14:31:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 167 2023-01-15T14:31:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.654 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:03.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115279+0000", 2023-01-15T14:31:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115289+0000", 2023-01-15T14:31:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0509999999999999e-05 2023-01-15T14:31:03.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115306+0000", 2023-01-15T14:31:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7260000000000001e-05 2023-01-15T14:31:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115337+0000", 2023-01-15T14:31:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0664000000000003e-05 2023-01-15T14:31:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115579+0000", 2023-01-15T14:31:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024194400000000001 2023-01-15T14:31:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115744+0000", 2023-01-15T14:31:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016493700000000001 2023-01-15T14:31:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334392+0000", 2023-01-15T14:31:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21864830299999999 2023-01-15T14:31:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334453+0000", 2023-01-15T14:31:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1047e-05 2023-01-15T14:31:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334601+0000", 2023-01-15T14:31:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014824700000000001 2023-01-15T14:31:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334812+0000", 2023-01-15T14:31:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210208 2023-01-15T14:31:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334996+0000", 2023-01-15T14:31:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183878 2023-01-15T14:31:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335126+0000", 2023-01-15T14:31:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013001399999999999 2023-01-15T14:31:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:03.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335160+0000", 2023-01-15T14:31:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4876999999999998e-05 2023-01-15T14:31:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741044+0000", 2023-01-15T14:31:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40588326000000002 2023-01-15T14:31:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741453+0000", 2023-01-15T14:31:03.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040963099999999998 2023-01-15T14:31:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741548+0000", 2023-01-15T14:31:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4961000000000001e-05 2023-01-15T14:31:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741664+0000", 2023-01-15T14:31:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115333 2023-01-15T14:31:03.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:03.664 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 156.22248943599999, 2023-01-15T14:31:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:31:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:31:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:03.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:31:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:31:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:31:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:03.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:31:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:31:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:31:03.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:03.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:31:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:31:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 156.209497357, 2023-01-15T14:31:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:31:03.845 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:03.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:31:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:31:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:31:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:31:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:03.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:31:03.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:31:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:31:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:31:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:31:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:31:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:31:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:31:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:31:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:31:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:31:03.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:31:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:31:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:31:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:31:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:03.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 156.209094021, 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:03.855 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:31:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:03.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:03.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:31:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:31:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:31:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:31:04.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:31:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:31:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:31:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:31:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:31:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:31:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:31:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:31:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:31:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:31:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:31:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:31:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:31:04.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:31:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:31:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:31:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:04.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:31:04.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:31:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:31:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:31:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:31:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:31:04.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:31:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:31:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:31:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 156.10887040099999, 2023-01-15T14:31:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:31:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:04.057 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:31:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:31:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:31:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:31:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:31:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:31:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:31:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:31:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:31:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:31:04.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:31:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:31:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:31:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:31:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:31:04.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:31:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:31:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:04.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:194 3.8 3:1eaad226:::smithi09025264-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 156.03646915799999, 2023-01-15T14:31:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64372577200000003, 2023-01-15T14:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 194 2023-01-15T14:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301390+0000", 2023-01-15T14:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301991+0000", 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060122599999999995 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.302026+0000", 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5040000000000003e-05 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402147+0000", 2023-01-15T14:31:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100121084 2023-01-15T14:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402219+0000", 2023-01-15T14:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2045999999999994e-05 2023-01-15T14:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402315+0000", 2023-01-15T14:31:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5561000000000002e-05 2023-01-15T14:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842189+0000", 2023-01-15T14:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.439874129 2023-01-15T14:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842296+0000", 2023-01-15T14:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107021 2023-01-15T14:31:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842486+0000", 2023-01-15T14:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018965799999999999 2023-01-15T14:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842839+0000", 2023-01-15T14:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035330700000000002 2023-01-15T14:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842859+0000", 2023-01-15T14:31:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0347e-05 2023-01-15T14:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843212+0000", 2023-01-15T14:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352534 2023-01-15T14:31:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843835+0000", 2023-01-15T14:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062338599999999997 2023-01-15T14:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944861+0000", 2023-01-15T14:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10102596899999999 2023-01-15T14:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944908+0000", 2023-01-15T14:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6773000000000001e-05 2023-01-15T14:31:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945039+0000", 2023-01-15T14:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130684 2023-01-15T14:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945118+0000", 2023-01-15T14:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9194000000000001e-05 2023-01-15T14:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:705 3.2 3:47dfe935:::smithi09025264-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2023-01-15T14:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 114.22727725, 2023-01-15T14:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61170208599999998, 2023-01-15T14:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:04.272 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 705 2023-01-15T14:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110583+0000", 2023-01-15T14:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:04.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110587+0000", 2023-01-15T14:31:04.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9689999999999996e-06 2023-01-15T14:31:04.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:04.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110601+0000", 2023-01-15T14:31:04.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.393e-05 2023-01-15T14:31:04.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110611+0000", 2023-01-15T14:31:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3999999999999998e-06 2023-01-15T14:31:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319174+0000", 2023-01-15T14:31:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20856376800000001 2023-01-15T14:31:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319367+0000", 2023-01-15T14:31:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192361 2023-01-15T14:31:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.721941+0000", 2023-01-15T14:31:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40257461300000003 2023-01-15T14:31:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722108+0000", 2023-01-15T14:31:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167054 2023-01-15T14:31:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722286+0000", 2023-01-15T14:31:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017772099999999999 2023-01-15T14:31:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.415555175999998, 2023-01-15T14:31:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:31:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:31:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:31:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:31:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:31:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:31:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:31:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:31:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:31:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.409093822000003, 2023-01-15T14:31:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:31:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:31:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:31:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:31:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:31:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:31:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:31:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:31:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.308671896999996, 2023-01-15T14:31:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:31:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:31:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:31:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:31:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:31:04.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:31:04.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:04.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.308334411000004, 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:04.903 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:31:04.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:31:04.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:31:04.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:04.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:31:04.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:31:04.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.308278514999998, 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.909 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:04.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:04.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:04.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:04.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:04.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:04.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:31:04.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:31:04.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:04.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:31:04.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:31:04.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:31:04.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:31:04.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:04.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:31:04.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:31:04.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:04.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:31:04.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:31:04.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1314 3.18 3.afe52a78 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:04.914 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 63.977983383999998, 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62560938700000002, 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1314 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:04.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:04.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:04.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:04.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:04.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:04.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359877+0000", 2023-01-15T14:31:04.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:04.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:04.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359882+0000", 2023-01-15T14:31:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0050000000000004e-06 2023-01-15T14:31:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359893+0000", 2023-01-15T14:31:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0676000000000001e-05 2023-01-15T14:31:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359904+0000", 2023-01-15T14:31:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0788000000000001e-05 2023-01-15T14:31:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460094+0000", 2023-01-15T14:31:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10019024999999999 2023-01-15T14:31:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460106+0000", 2023-01-15T14:31:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354999999999999e-05 2023-01-15T14:31:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985477+0000", 2023-01-15T14:31:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52537039100000005 2023-01-15T14:31:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985487+0000", 2023-01-15T14:31:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:31:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1335 3.8 3:17d5d741:::smithi09025264-5:head [write 3252761~175867] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.405964122999997, 2023-01-15T14:31:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66052243499999996, 2023-01-15T14:31:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1335 2023-01-15T14:31:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931896+0000", 2023-01-15T14:31:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931983+0000", 2023-01-15T14:31:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6470000000000004e-05 2023-01-15T14:31:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931995+0000", 2023-01-15T14:31:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2075e-05 2023-01-15T14:31:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932003+0000", 2023-01-15T14:31:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5059999999999995e-06 2023-01-15T14:31:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932346+0000", 2023-01-15T14:31:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034280299999999998 2023-01-15T14:31:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932357+0000", 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.13e-05 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189006+0000", 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25664859899999998 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189336+0000", 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033030899999999998 2023-01-15T14:31:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189481+0000", 2023-01-15T14:31:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014484400000000001 2023-01-15T14:31:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189768+0000", 2023-01-15T14:31:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028704300000000001 2023-01-15T14:31:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189787+0000", 2023-01-15T14:31:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9343000000000001e-05 2023-01-15T14:31:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191361+0000", 2023-01-15T14:31:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015734029999999999 2023-01-15T14:31:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191469+0000", 2023-01-15T14:31:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010859899999999999 2023-01-15T14:31:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391274+0000", 2023-01-15T14:31:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19980450899999999 2023-01-15T14:31:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592067+0000", 2023-01-15T14:31:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20079332 2023-01-15T14:31:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592190+0000", 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012291400000000001 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592420+0000", 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000229371 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1336 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (56) in=7b,truncate 3428628] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.405847033999997, 2023-01-15T14:31:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66113910799999998, 2023-01-15T14:31:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:06.722 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1336 2023-01-15T14:31:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:06.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:06.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:06.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:06.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932013+0000", 2023-01-15T14:31:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932019+0000", 2023-01-15T14:31:06.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5380000000000002e-06 2023-01-15T14:31:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:06.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:07.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932027+0000", 2023-01-15T14:31:07.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5150000000000006e-06 2023-01-15T14:31:07.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:07.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932034+0000", 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032497+0000", 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100463001 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032518+0000", 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1359999999999999e-05 2023-01-15T14:31:07.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189826+0000", 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15730834299999999 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189951+0000", 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124583 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:07.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190093+0000", 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014221399999999999 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190253+0000", 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015953000000000001 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191365+0000", 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011123190000000001 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:07.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191439+0000", 2023-01-15T14:31:07.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3648999999999999e-05 2023-01-15T14:31:07.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:07.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191491+0000", 2023-01-15T14:31:07.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2444999999999997e-05 2023-01-15T14:31:07.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391472+0000", 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199981193 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592619+0000", 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20114642499999999 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592748+0000", 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012950200000000001 2023-01-15T14:31:07.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:07.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593153+0000", 2023-01-15T14:31:07.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040503700000000002 2023-01-15T14:31:07.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:07.110 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:07.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:07.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.110 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1337 3.8 3:17d5d741:::smithi09025264-5:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:07.110 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.405818805000003, 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66179168200000005, 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1337 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:07.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:07.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:07.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:07.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:07.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:07.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:07.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932046+0000", 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8089999999999999e-06 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932055+0000", 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2199999999999998e-06 2023-01-15T14:31:07.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:07.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932061+0000", 2023-01-15T14:31:07.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2779999999999999e-06 2023-01-15T14:31:07.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:07.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190318+0000", 2023-01-15T14:31:07.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.258256967 2023-01-15T14:31:07.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190413+0000", 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4627000000000001e-05 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593288+0000", 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.402875339 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:07.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593515+0000", 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022700699999999999 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593834+0000", 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031906799999999999 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.116 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1690 3.16 3:68fb3263:::smithi09025264-33:head [write 1763227~472137] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.475345834999999, 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72570748100000004, 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1690 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:07.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:07.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:07.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:07.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:07.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:07.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:07.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862514+0000", 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862746+0000", 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000232301 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862770+0000", 2023-01-15T14:31:07.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3286e-05 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862779+0000", 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4199999999999996e-06 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862844+0000", 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5036999999999994e-05 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862860+0000", 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6426000000000002e-05 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051251+0000", 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18839023299999999 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:07.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051466+0000", 2023-01-15T14:31:07.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021523199999999999 2023-01-15T14:31:07.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:07.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368326+0000", 2023-01-15T14:31:07.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31686002699999999 2023-01-15T14:31:07.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:07.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368559+0000", 2023-01-15T14:31:07.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023305400000000001 2023-01-15T14:31:07.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:07.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368935+0000", 2023-01-15T14:31:07.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037579099999999999 2023-01-15T14:31:07.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:07.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369531+0000", 2023-01-15T14:31:07.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059593199999999997 2023-01-15T14:31:07.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:07.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369571+0000", 2023-01-15T14:31:07.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0649999999999999e-05 2023-01-15T14:31:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370331+0000", 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075969899999999996 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371230+0000", 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000899134 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472895+0000", 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101665191 2023-01-15T14:31:07.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.587914+0000", 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115018671 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588013+0000", 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8494999999999998e-05 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588223+0000", 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210294 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1691 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 2235364] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.475061580999999, 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72593841699999995, 2023-01-15T14:31:07.326 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1691 2023-01-15T14:31:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:07.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862799+0000", 2023-01-15T14:31:07.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862807+0000", 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1280000000000008e-06 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862821+0000", 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.345e-05 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:07.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862826+0000", 2023-01-15T14:31:07.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3720000000000004e-06 2023-01-15T14:31:07.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:07.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862883+0000", 2023-01-15T14:31:07.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6589000000000003e-05 2023-01-15T14:31:07.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862888+0000", 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4060000000000004e-06 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151601+0000", 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28871354100000002 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151732+0000", 2023-01-15T14:31:07.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013032300000000001 2023-01-15T14:31:07.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:07.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369729+0000", 2023-01-15T14:31:07.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217997515 2023-01-15T14:31:07.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:07.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369949+0000", 2023-01-15T14:31:07.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219664 2023-01-15T14:31:07.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370367+0000", 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041822799999999998 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370771+0000", 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040358400000000002 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370794+0000", 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-05 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370880+0000", 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6020999999999995e-05 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371241+0000", 2023-01-15T14:31:07.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000360956 2023-01-15T14:31:07.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:07.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472994+0000", 2023-01-15T14:31:07.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101753089 2023-01-15T14:31:07.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:07.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588348+0000", 2023-01-15T14:31:07.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11535382299999999 2023-01-15T14:31:07.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588539+0000", 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019090399999999999 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588738+0000", 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019910800000000001 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:07.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.475028096999999, 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:07.337 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:31:07.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.339 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:07.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:07.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:07.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:07.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:07.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:07.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:07.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:07.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:07.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:07.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:07.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:31:07.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:31:07.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:07.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:07.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:07.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:31:07.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:31:07.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:08.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:31:08.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:08.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:31:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:31:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:31:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:31:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:08.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:31:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:31:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:31:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:31:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:08.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:31:08.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:31:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:08.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:08.842 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:08.843 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:08.844 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:31:08.844 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:03.640+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:08.845 INFO:tasks.rados.rados.0.smithi090.stdout:686: done (0 left) 2023-01-15T14:31:08.845 INFO:tasks.rados.rados.0.smithi090.stdout:687: rollback oid 33 current snap is 70 2023-01-15T14:31:08.845 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 33 to 68 2023-01-15T14:31:08.845 INFO:tasks.rados.rados.0.smithi090.stdout:688: copy_from oid 7 from oid 16 current snap is 70 2023-01-15T14:31:08.845 INFO:tasks.rados.rados.0.smithi090.stdout:689: write oid 38 current snap is 70 2023-01-15T14:31:08.845 INFO:tasks.rados.rados.0.smithi090.stdout:689: seq_num 270 ranges {537890=789585,1783197=699079} 2023-01-15T14:31:08.846 INFO:tasks.rados.rados.0.smithi090.stdout:689: writing smithi09025264-38 from 537890 to 1327475 tid 1 2023-01-15T14:31:08.847 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:31:08.847 INFO:tasks.rados.rados.0.smithi090.stdout:689: writing smithi09025264-38 from 1783197 to 2482276 tid 2 2023-01-15T14:31:08.847 INFO:tasks.rados.rados.0.smithi090.stdout:690: copy_from oid 41 from oid 15 current snap is 70 2023-01-15T14:31:08.847 INFO:tasks.rados.rados.0.smithi090.stdout:691: write (excl) oid 26 current snap is 70 2023-01-15T14:31:08.848 INFO:tasks.rados.rados.0.smithi090.stdout:691: seq_num 271 ranges {554697=464834,1725326=559090} 2023-01-15T14:31:08.848 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:31:08.853 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:31:08.853 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:31:08.853 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2023-01-15T14:31:08.853 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-01-15T14:31:08.853 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-01-15T14:31:08.853 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2023-01-15T14:31:08.854 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:08.846+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:08.860 INFO:tasks.rados.rados.0.smithi090.stdout:691: writing smithi09025264-26 from 554697 to 1019531 tid 1 2023-01-15T14:31:08.860 INFO:tasks.rados.rados.0.smithi090.stdout:691: writing smithi09025264-26 from 1725326 to 2284416 tid 2 2023-01-15T14:31:08.861 INFO:tasks.rados.rados.0.smithi090.stdout:692: read oid 30 snap 68 2023-01-15T14:31:08.861 INFO:tasks.rados.rados.0.smithi090.stdout:692: expect (ObjNum 206 snap 44 seq_num 206) 2023-01-15T14:31:08.861 INFO:tasks.rados.rados.0.smithi090.stdout:693: rollback oid 40 current snap is 70 2023-01-15T14:31:08.861 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 40 to 62 2023-01-15T14:31:08.861 INFO:tasks.rados.rados.0.smithi090.stdout:694: delete oid 14 current snap is 70 2023-01-15T14:31:08.861 INFO:tasks.rados.rados.0.smithi090.stdout:692: expect (ObjNum 206 snap 44 seq_num 206) 2023-01-15T14:31:08.861 INFO:tasks.rados.rados.0.smithi090.stdout:690: finishing copy_from to smithi09025264-41 2023-01-15T14:31:08.861 INFO:tasks.rados.rados.0.smithi090.stdout:690: got expected ENOENT (src dne) 2023-01-15T14:31:08.861 INFO:tasks.rados.rados.0.smithi090.stdout:690: finishing copy_from racing read to smithi09025264-41 2023-01-15T14:31:08.861 INFO:tasks.rados.rados.0.smithi090.stdout:691: finishing write tid 1 to smithi09025264-26 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:687: finishing rollback tid 0 to smithi09025264-33 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:689: finishing write tid 1 to smithi09025264-38 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:689: finishing write tid 2 to smithi09025264-38 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:687: finishing rollback tid 1 to smithi09025264-33 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:693: finishing rollback tid 0 to smithi09025264-40 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:690: done (7 left) 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:692: done (6 left) 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:694: done (5 left) 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:695: read oid 23 snap -1 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:695: expect (ObjNum 226 snap 51 seq_num 226) 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:696: write oid 44 current snap is 70 2023-01-15T14:31:08.862 INFO:tasks.rados.rados.0.smithi090.stdout:696: seq_num 273 ranges {588311=615332,1842290=562759} 2023-01-15T14:31:08.863 INFO:tasks.rados.rados.0.smithi090.stdout:696: writing smithi09025264-44 from 588311 to 1203643 tid 1 2023-01-15T14:31:08.863 INFO:tasks.rados.rados.0.smithi090.stdout:696: writing smithi09025264-44 from 1842290 to 2405049 tid 2 2023-01-15T14:31:08.863 INFO:tasks.rados.rados.0.smithi090.stdout:697: read oid 29 snap -1 2023-01-15T14:31:08.863 INFO:tasks.rados.rados.0.smithi090.stdout:697: expect (ObjNum 267 snap 66 seq_num 267) 2023-01-15T14:31:08.863 INFO:tasks.rados.rados.0.smithi090.stdout:698: read oid 46 snap -1 2023-01-15T14:31:08.863 INFO:tasks.rados.rados.0.smithi090.stdout:698: expect (ObjNum 256 snap 60 seq_num 256) 2023-01-15T14:31:08.863 INFO:tasks.rados.rados.0.smithi090.stdout:699: delete oid 12 current snap is 70 2023-01-15T14:31:08.863 INFO:tasks.rados.rados.0.smithi090.stdout:691: finishing write tid 2 to smithi09025264-26 2023-01-15T14:31:08.863 INFO:tasks.rados.rados.0.smithi090.stdout:688: finishing copy_from to smithi09025264-7 2023-01-15T14:31:08.863 INFO:tasks.rados.rados.0.smithi090.stdout:688: got expected ENOENT (src dne) 2023-01-15T14:31:08.994 INFO:tasks.rados.rados.0.smithi090.stdout:688: finishing copy_from racing read to smithi09025264-7 2023-01-15T14:31:08.994 INFO:tasks.rados.rados.0.smithi090.stdout:691: finishing write tid 3 to smithi09025264-26 2023-01-15T14:31:08.994 INFO:tasks.rados.rados.0.smithi090.stdout:691: finishing write tid 4 to smithi09025264-26 2023-01-15T14:31:08.994 INFO:tasks.rados.rados.0.smithi090.stdout:691: oid 26 updating version 0 to 241 2023-01-15T14:31:08.994 INFO:tasks.rados.rados.0.smithi090.stdout:691: oid 26 updating version 241 to 242 2023-01-15T14:31:08.994 INFO:tasks.rados.rados.0.smithi090.stdout:691: oid 26 updating version 242 to 243 2023-01-15T14:31:08.994 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 243 (ObjNum 271 snap 70 seq_num 271) dirty exists 2023-01-15T14:31:08.994 INFO:tasks.rados.rados.0.smithi090.stdout:691: left oid 26 (ObjNum 271 snap 70 seq_num 271) 2023-01-15T14:31:08.994 INFO:tasks.rados.rados.0.smithi090.stdout:689: finishing write tid 3 to smithi09025264-38 2023-01-15T14:31:08.995 INFO:tasks.rados.rados.0.smithi090.stdout:689: finishing write tid 4 to smithi09025264-38 2023-01-15T14:31:08.995 INFO:tasks.rados.rados.0.smithi090.stdout:689: oid 38 updating version 0 to 157 2023-01-15T14:31:08.995 INFO:tasks.rados.rados.0.smithi090.stdout:689: oid 38 version 157 is already newer than 155 2023-01-15T14:31:08.995 INFO:tasks.rados.rados.0.smithi090.stdout:689: oid 38 version 157 is already newer than 156 2023-01-15T14:31:08.995 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 157 (ObjNum 270 snap 70 seq_num 270) dirty exists 2023-01-15T14:31:08.995 INFO:tasks.rados.rados.0.smithi090.stdout:689: left oid 38 (ObjNum 270 snap 70 seq_num 270) 2023-01-15T14:31:08.995 INFO:tasks.rados.rados.0.smithi090.stdout:697: expect (ObjNum 267 snap 66 seq_num 267) 2023-01-15T14:31:08.995 INFO:tasks.rados.rados.0.smithi090.stdout:695: expect (ObjNum 226 snap 51 seq_num 226) 2023-01-15T14:31:08.995 INFO:tasks.rados.rados.0.smithi090.stdout:687: finishing rollback tid 2 to smithi09025264-33 2023-01-15T14:31:08.995 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 73 (ObjNum 252 snap 60 seq_num 252) dirty exists 2023-01-15T14:31:08.995 INFO:tasks.rados.rados.0.smithi090.stdout:687: done (9 left) 2023-01-15T14:31:08.996 INFO:tasks.rados.rados.0.smithi090.stdout:688: done (8 left) 2023-01-15T14:31:08.996 INFO:tasks.rados.rados.0.smithi090.stdout:689: done (7 left) 2023-01-15T14:31:08.996 INFO:tasks.rados.rados.0.smithi090.stdout:691: done (6 left) 2023-01-15T14:31:08.996 INFO:tasks.rados.rados.0.smithi090.stdout:695: done (5 left) 2023-01-15T14:31:08.996 INFO:tasks.rados.rados.0.smithi090.stdout:697: done (4 left) 2023-01-15T14:31:08.996 INFO:tasks.rados.rados.0.smithi090.stdout:699: done (3 left) 2023-01-15T14:31:08.996 INFO:tasks.rados.rados.0.smithi090.stdout:700: snap_remove snap 64 2023-01-15T14:31:08.996 INFO:tasks.rados.rados.0.smithi090.stdout:700: done (3 left) 2023-01-15T14:31:08.996 INFO:tasks.rados.rados.0.smithi090.stdout:701: write (excl) oid 28 current snap is 70 2023-01-15T14:31:08.996 INFO:tasks.rados.rados.0.smithi090.stdout:696: finishing write tid 1 to smithi09025264-44 2023-01-15T14:31:08.997 INFO:tasks.rados.rados.0.smithi090.stdout:698: expect (ObjNum 256 snap 60 seq_num 256) 2023-01-15T14:31:08.997 INFO:tasks.rados.rados.0.smithi090.stdout:696: finishing write tid 2 to smithi09025264-44 2023-01-15T14:31:08.997 INFO:tasks.rados.rados.0.smithi090.stdout:696: finishing write tid 3 to smithi09025264-44 2023-01-15T14:31:08.997 INFO:tasks.rados.rados.0.smithi090.stdout:696: finishing write tid 4 to smithi09025264-44 2023-01-15T14:31:08.997 INFO:tasks.rados.rados.0.smithi090.stdout:696: oid 44 updating version 0 to 217 2023-01-15T14:31:08.997 INFO:tasks.rados.rados.0.smithi090.stdout:696: oid 44 version 217 is already newer than 216 2023-01-15T14:31:08.997 INFO:tasks.rados.rados.0.smithi090.stdout:696: oid 44 version 217 is already newer than 215 2023-01-15T14:31:08.997 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 217 (ObjNum 273 snap 70 seq_num 273) dirty exists 2023-01-15T14:31:08.997 INFO:tasks.rados.rados.0.smithi090.stdout:696: left oid 44 (ObjNum 273 snap 70 seq_num 273) 2023-01-15T14:31:08.997 INFO:tasks.rados.rados.0.smithi090.stdout:701: seq_num 275 ranges {621925=482183,1559254=566428} 2023-01-15T14:31:08.997 INFO:tasks.rados.rados.0.smithi090.stdout:701: writing smithi09025264-28 from 621925 to 1104108 tid 1 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:701: writing smithi09025264-28 from 1559254 to 2125682 tid 2 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:696: done (3 left) 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:698: done (2 left) 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:702: delete oid 1 current snap is 70 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:701: finishing write tid 1 to smithi09025264-28 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:702: done (2 left) 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:703: read oid 18 snap -1 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:703: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:704: write (excl) oid 31 current snap is 70 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:704: seq_num 277 ranges {655539=632681,2076218=570097} 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:704: writing smithi09025264-31 from 655539 to 1288220 tid 1 2023-01-15T14:31:08.998 INFO:tasks.rados.rados.0.smithi090.stdout:704: writing smithi09025264-31 from 2076218 to 2646315 tid 2 2023-01-15T14:31:08.999 INFO:tasks.rados.rados.0.smithi090.stdout:705: snap_remove snap 63 2023-01-15T14:31:08.999 INFO:tasks.rados.rados.0.smithi090.stdout:705: done (4 left) 2023-01-15T14:31:08.999 INFO:tasks.rados.rados.0.smithi090.stdout:706: snap_remove snap 68 2023-01-15T14:31:08.999 INFO:tasks.rados.rados.0.smithi090.stdout:706: done (4 left) 2023-01-15T14:31:08.999 INFO:tasks.rados.rados.0.smithi090.stdout:707: snap_create 2023-01-15T14:31:08.999 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:31:08.999 INFO:tasks.rados.rados.0.smithi090.stdout:701: finishing write tid 2 to smithi09025264-28 2023-01-15T14:31:08.999 INFO:tasks.rados.rados.0.smithi090.stdout:703: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:08.999 INFO:tasks.rados.rados.0.smithi090.stdout:701: finishing write tid 3 to smithi09025264-28 2023-01-15T14:31:08.999 INFO:tasks.rados.rados.0.smithi090.stdout:701: finishing write tid 4 to smithi09025264-28 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:701: oid 28 updating version 0 to 252 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:701: oid 28 version 252 is already newer than 251 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:701: oid 28 version 252 is already newer than 248 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 252 (ObjNum 275 snap 70 seq_num 275) dirty exists 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:701: left oid 28 (ObjNum 275 snap 70 seq_num 275) 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:704: finishing write tid 1 to smithi09025264-31 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:704: finishing write tid 2 to smithi09025264-31 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:704: finishing write tid 3 to smithi09025264-31 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:704: finishing write tid 4 to smithi09025264-31 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:704: oid 31 updating version 0 to 238 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:704: oid 31 version 238 is already newer than 236 2023-01-15T14:31:09.000 INFO:tasks.rados.rados.0.smithi090.stdout:704: oid 31 version 238 is already newer than 237 2023-01-15T14:31:09.001 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 238 (ObjNum 277 snap 70 seq_num 277) dirty exists 2023-01-15T14:31:09.001 INFO:tasks.rados.rados.0.smithi090.stdout:704: left oid 31 (ObjNum 277 snap 70 seq_num 277) 2023-01-15T14:31:09.001 INFO:tasks.rados.rados.0.smithi090.stdout:701: done (3 left) 2023-01-15T14:31:09.001 INFO:tasks.rados.rados.0.smithi090.stdout:703: done (2 left) 2023-01-15T14:31:09.001 INFO:tasks.rados.rados.0.smithi090.stdout:704: done (1 left) 2023-01-15T14:31:09.001 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:09.001 INFO:tasks.rados.rados.0.smithi090.stdout:693: finishing rollback tid 1 to smithi09025264-40 2023-01-15T14:31:09.001 INFO:tasks.rados.rados.0.smithi090.stdout:693: finishing rollback tid 2 to smithi09025264-40 2023-01-15T14:31:09.001 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 196 (ObjNum 240 snap 55 seq_num 240) dirty exists 2023-01-15T14:31:09.001 INFO:tasks.rados.rados.0.smithi090.stdout:693: done (0 left) 2023-01-15T14:31:09.001 INFO:tasks.rados.rados.0.smithi090.stdout:707: done (0 left) 2023-01-15T14:31:09.002 INFO:tasks.rados.rados.0.smithi090.stdout:708: snap_create 2023-01-15T14:31:09.002 INFO:tasks.rados.rados.0.smithi090.stdout:708: done (0 left) 2023-01-15T14:31:09.002 INFO:tasks.rados.rados.0.smithi090.stdout:709: read oid 22 snap -1 2023-01-15T14:31:09.002 INFO:tasks.rados.rados.0.smithi090.stdout:709: expect (ObjNum 160 snap 35 seq_num 160) 2023-01-15T14:31:09.002 INFO:tasks.rados.rados.0.smithi090.stdout:710: write (excl) oid 38 current snap is 72 2023-01-15T14:31:09.002 INFO:tasks.rados.rados.0.smithi090.stdout:710: seq_num 278 ranges {672346=707930,2134700=537646} 2023-01-15T14:31:09.002 INFO:tasks.rados.rados.0.smithi090.stdout:710: writing smithi09025264-38 from 672346 to 1380276 tid 1 2023-01-15T14:31:09.002 INFO:tasks.rados.rados.0.smithi090.stdout:710: writing smithi09025264-38 from 2134700 to 2672346 tid 2 2023-01-15T14:31:09.002 INFO:tasks.rados.rados.0.smithi090.stdout:711: snap_create 2023-01-15T14:31:09.002 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:09.003 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:09.004 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:31:09.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.004 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1872 3.1d 3:bd982ca3:::smithi09025264-38:head [write 672346~707930 in=707930b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2023-01-15T14:31:09.004 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:08.817076+0000", 2023-01-15T14:31:09.004 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.170501598, 2023-01-15T14:31:09.004 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17054039300000001, 2023-01-15T14:31:09.004 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:09.004 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:31:09.004 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:09.004 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:09.005 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:09.005 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1872 2023-01-15T14:31:09.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.005 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:09.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:09.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.817076+0000", 2023-01-15T14:31:09.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:09.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.817076+0000", 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.817073+0000", 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.817963+0000", 2023-01-15T14:31:09.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089014599999999997 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.818000+0000", 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7552000000000002e-05 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.818017+0000", 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6656000000000001e-05 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.008 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:09.008 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.918216+0000", 2023-01-15T14:31:09.008 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100198626 2023-01-15T14:31:09.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.008 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:09.008 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.918427+0000", 2023-01-15T14:31:09.008 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000211292 2023-01-15T14:31:09.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.008 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.918824+0000", 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039734299999999998 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.920788+0000", 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0019640209999999998 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.920813+0000", 2023-01-15T14:31:09.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4896e-05 2023-01-15T14:31:09.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:09.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.921234+0000", 2023-01-15T14:31:09.010 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042049500000000001 2023-01-15T14:31:09.010 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:09.010 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:09.010 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:09.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.010 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1873 3.1d 3.c53419bd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e182)", 2023-01-15T14:31:09.011 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:08.824646+0000", 2023-01-15T14:31:09.011 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.162931561, 2023-01-15T14:31:09.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.163070884, 2023-01-15T14:31:09.011 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:09.011 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:09.011 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:09.011 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:09.011 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:09.011 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1873 2023-01-15T14:31:09.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.011 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:09.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:09.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.824646+0000", 2023-01-15T14:31:09.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:09.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:09.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.824646+0000", 2023-01-15T14:31:09.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:09.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:09.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.824643+0000", 2023-01-15T14:31:09.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:09.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:09.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825067+0000", 2023-01-15T14:31:09.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042346099999999998 2023-01-15T14:31:09.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:09.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825100+0000", 2023-01-15T14:31:09.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3373e-05 2023-01-15T14:31:09.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:09.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825114+0000", 2023-01-15T14:31:09.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3849000000000001e-05 2023-01-15T14:31:09.014 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:09.014 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:09.014 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:09.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1874 3.1d 3.c53419bd (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e182)", 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:08.825139+0000", 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.16243844399999999, 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.16263148799999999, 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1874 2023-01-15T14:31:09.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.016 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:09.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:09.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825139+0000", 2023-01-15T14:31:09.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:09.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:09.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825139+0000", 2023-01-15T14:31:09.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:09.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825138+0000", 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825145+0000", 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9140000000000002e-06 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:09.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825154+0000", 2023-01-15T14:31:09.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7829999999999993e-06 2023-01-15T14:31:09.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:09.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825161+0000", 2023-01-15T14:31:09.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3690000000000003e-06 2023-01-15T14:31:09.018 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:09.018 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:09.018 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:09.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1875 3.1d 3.c53419bd (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e182)", 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:08.825196+0000", 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.162380685, 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.162627034, 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1875 2023-01-15T14:31:09.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825196+0000", 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825196+0000", 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:09.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825196+0000", 2023-01-15T14:31:09.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:09.021 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:09.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825201+0000", 2023-01-15T14:31:09.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0429999999999998e-06 2023-01-15T14:31:09.021 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825216+0000", 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5452999999999999e-05 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.825223+0000", 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:1872 3.1d e182/166)", 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:08.922768+0000", 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.064809404000000001, 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.065108037999999993, 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "no flag points reached", 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.922768+0000", 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:09.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:09.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.922768+0000", 2023-01-15T14:31:09.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:09.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:09.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.922766+0000", 2023-01-15T14:31:09.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:09.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.922778+0000", 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2577e-05 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:08.922805+0000", 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6118999999999998e-05 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:09.025 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:09.026 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2023-01-15T14:31:09.026 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 5 2023-01-15T14:31:09.026 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:09.026 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:09.026 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-01-15T14:31:09.026 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:09.027 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:31:09.029 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:09.000+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:09.105 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:09.100+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:09.158 INFO:tasks.rados.rados.0.smithi090.stdout:709: expect (ObjNum 160 snap 35 seq_num 160) 2023-01-15T14:31:09.158 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:09.159 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:09.159 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:31:09.159 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:31:09.159 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:31:09.159 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:09.159 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:31:09.173 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:09.191 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:09.303 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:09.303 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:09.303 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:09.304 INFO:tasks.rados.rados.0.smithi090.stdout:710: finishing write tid 1 to smithi09025264-38 2023-01-15T14:31:09.304 INFO:tasks.rados.rados.0.smithi090.stdout:709: done (1 left) 2023-01-15T14:31:09.304 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:09.304 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:09.304 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:09.305 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:31:09.305 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:09.305 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:09.306 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:09.300+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:09.306 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:31:09.306 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:31:09.306 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:31:09.306 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:31:09.306 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:31:09.306 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:31:09.317 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:31:09.317 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:31:09.404 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:09.400+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:10.600 INFO:tasks.rados.rados.0.smithi090.stdout:710: finishing write tid 2 to smithi09025264-38 2023-01-15T14:31:10.600 INFO:tasks.rados.rados.0.smithi090.stdout:710: finishing write tid 3 to smithi09025264-38 2023-01-15T14:31:10.601 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:10.601 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:10.601 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:31:10.602 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:10.598+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:10.701 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:10.696+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:10.757 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:10.758 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:10.768 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:31:10.802 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:10.798+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:10.856 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:31:10.925 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:10.926 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:10.926 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:31:10.926 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:10.922+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:11.027 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:11.022+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:11.103 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:11.103 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:11.104 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:31:11.127 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:11.124+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:11.210 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:11.243 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:11.245 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:31:11.245 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:11.240+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:11.320 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:31:11.403 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:11.403 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:11.404 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:31:11.404 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:11.398+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:11.504 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:11.498+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:11.510 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:11.510 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:11.520 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:31:11.549 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:11.570 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:11.589 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:11.605 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:11.600+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:11.606 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:11.668 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:11.668 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:11.678 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:31:11.705 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:11.700+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:11.783 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:11.832 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:11.834 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:31:11.835 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:11.832+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:11.844 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:31:11.935 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:11.930+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:11.938 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:11.939 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:11.939 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:11.939 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:11.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:31:12.023 INFO:tasks.rados.rados.0.smithi090.stdout:710: finishing write tid 4 to smithi09025264-38 2023-01-15T14:31:12.023 INFO:tasks.rados.rados.0.smithi090.stdout:710: oid 38 updating version 0 to 160 2023-01-15T14:31:12.023 INFO:tasks.rados.rados.0.smithi090.stdout:710: oid 38 version 160 is already newer than 159 2023-01-15T14:31:12.023 INFO:tasks.rados.rados.0.smithi090.stdout:710: oid 38 updating version 160 to 161 2023-01-15T14:31:12.023 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 161 (ObjNum 278 snap 72 seq_num 278) dirty exists 2023-01-15T14:31:12.023 INFO:tasks.rados.rados.0.smithi090.stdout:710: left oid 38 (ObjNum 278 snap 72 seq_num 278) 2023-01-15T14:31:12.024 INFO:tasks.rados.rados.0.smithi090.stdout:710: done (0 left) 2023-01-15T14:31:12.036 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:12.030+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:12.059 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:12.059 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:12.059 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:12.068 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:31:12.137 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:12.132+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:12.160 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:12.182 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 166.116965139, 2023-01-15T14:31:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:31:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:31:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:31:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:31:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:31:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:31:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:31:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:31:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:31:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:31:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:31:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:31:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:31:12.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:31:12.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:31:12.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:31:12.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:31:12.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:31:12.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:31:12.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:31:12.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:31:12.192 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.192 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 166.11668489100001, 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.193 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.194 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:31:12.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.194 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:12.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:31:12.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:31:12.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:31:12.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:31:12.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:31:12.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:31:12.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:31:12.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:31:12.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:31:12.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:31:12.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:31:12.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:31:12.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:31:12.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:31:12.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:31:12.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:31:12.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:31:12.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:31:12.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:31:12.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:31:12.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:31:12.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:31:12.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:31:12.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.202 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.202 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.203 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.203 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:12.203 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 166.11665227399999, 2023-01-15T14:31:12.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:31:12.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.203 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.203 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.203 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.203 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.203 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:31:12.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.204 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:12.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:31:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:31:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:31:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:31:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:31:12.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:31:12.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:31:12.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:31:12.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:31:12.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:31:12.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:31:12.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:31:12.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:31:12.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:31:12.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:31:12.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:31:12.210 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.210 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.210 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.210 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.210 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:12.210 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 166.01533428799999, 2023-01-15T14:31:12.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:31:12.210 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.211 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.211 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.211 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.211 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.211 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:31:12.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.211 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:12.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:31:12.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:31:12.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:31:12.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:31:12.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:31:12.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:31:12.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:31:12.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:31:12.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:31:12.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:31:12.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:12.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:31:12.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:31:12.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:31:12.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:31:12.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:31:12.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:31:12.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:31:12.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:31:12.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:31:12.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:31:12.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:31:12.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:31:12.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:31:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:31:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:31:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:31:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:31:12.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:31:12.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.220 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.220 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 166.01530428500001, 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:12.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:12.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:31:12.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:31:12.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:31:12.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:31:12.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:31:12.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:31:12.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:31:12.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:31:12.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:31:12.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:31:12.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:31:12.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:31:12.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:31:12.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:31:12.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.50029572, 2023-01-15T14:31:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:31:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.228 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:31:12.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.229 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:12.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:12.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:31:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:31:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:31:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:31:12.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:31:12.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:31:12.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:31:12.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:31:12.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:31:12.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:31:12.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:31:12.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:31:12.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:12.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:31:12.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:31:12.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:31:12.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:31:12.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:31:12.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:31:12.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:31:12.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:31:12.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:31:12.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.237 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.237 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.238 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.238 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:12.238 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.49981336600001, 2023-01-15T14:31:12.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:31:12.238 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.238 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.238 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.238 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.238 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.238 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:31:12.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.239 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:12.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:12.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:31:12.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:12.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:31:12.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:31:12.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:31:12.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:31:12.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:31:12.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:31:12.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:31:12.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:31:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:31:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:31:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:31:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:31:12.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:31:12.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:31:12.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:31:12.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:31:12.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:31:12.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:31:12.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:31:12.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:31:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:31:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:31:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:31:12.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:31:12.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.248 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.248 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.248 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:12.248 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.49976080299999, 2023-01-15T14:31:12.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:31:12.248 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.248 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.249 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:31:12.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.249 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:12.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:12.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:31:12.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:31:12.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:31:12.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:31:12.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:31:12.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:31:12.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:31:12.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:31:12.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:31:12.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:31:12.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:31:12.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:31:12.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:31:12.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:31:12.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:31:12.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:31:12.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:31:12.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:31:12.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:31:12.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:31:12.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:31:12.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.255 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.256 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:12.256 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:12.256 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.128559099, 2023-01-15T14:31:12.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:31:12.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.256 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.256 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:31:12.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.257 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:12.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:12.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:31:12.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:12.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:31:12.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:31:12.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:31:12.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:31:12.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:31:12.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:12.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:31:12.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:31:12.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:31:12.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:31:12.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:31:12.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:31:12.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:31:12.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:31:12.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:31:12.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:31:12.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:31:12.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:31:12.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:31:12.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:31:12.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.264 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.265 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:12.265 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:12.265 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.128442425, 2023-01-15T14:31:12.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:31:12.265 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.265 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.265 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.265 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:12.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:31:12.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:31:12.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:31:12.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:31:12.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:31:12.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:31:12.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:31:12.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:31:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:31:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:31:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:31:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:31:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:31:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:31:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:31:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:31:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:31:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:31:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:31:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:31:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:31:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:31:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.40231014899999, 2023-01-15T14:31:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:31:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:31:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:31:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:31:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:31:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:31:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:31:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:31:12.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:31:12.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:31:12.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:31:12.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.278 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.279 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:31:12.279 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:12.279 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.18606009, 2023-01-15T14:31:12.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:31:12.279 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.279 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.279 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:31:12.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.280 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:12.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:12.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:31:12.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:31:12.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:12.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:31:12.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:31:12.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:31:12.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:31:12.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:31:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:31:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:31:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:31:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:31:12.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:31:12.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:31:12.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:31:12.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:31:12.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:31:12.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:31:12.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:31:12.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:31:12.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:31:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:31:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:31:12.288 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:12.289 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.186026523, 2023-01-15T14:31:12.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:31:12.289 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.289 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.289 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.289 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.289 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.289 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:31:12.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.289 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:31:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:31:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:31:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:31:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:31:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:31:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:31:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:31:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:31:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:31:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:31:12.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:31:12.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:31:12.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:31:12.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:12.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:31:12.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:31:12.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:31:12.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:31:12.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:31:12.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:31:12.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:31:12.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:31:12.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:31:12.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:31:12.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:31:12.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:31:12.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:31:12.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:31:12.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.298 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:31:12.299 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:12.299 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 89.420122977000005, 2023-01-15T14:31:12.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:31:12.299 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.299 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.299 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:31:12.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.299 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:12.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:12.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:31:12.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:31:12.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:31:12.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:31:12.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:31:12.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:31:12.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:12.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:31:12.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:31:12.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:31:12.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:31:12.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:31:12.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:31:12.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:12.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:31:12.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:31:12.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:31:12.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:31:12.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:31:12.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:31:12.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:31:12.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:31:12.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:31:12.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:31:12.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:31:12.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:31:12.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:31:12.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:31:12.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.309 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.309 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:12.309 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:31:12.309 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 72.876944176999999, 2023-01-15T14:31:12.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:31:12.309 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:31:12.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:31:12.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:12.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:31:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:31:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:31:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:31:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:31:12.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:31:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:31:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:31:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:31:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:31:12.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:31:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:31:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:31:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:31:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:31:12.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 72.876905002000001, 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.318 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:31:12.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:31:12.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:12.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:31:12.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:31:12.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:31:12.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:31:12.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:31:12.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:31:12.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:31:12.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:31:12.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:31:12.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:31:12.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:31:12.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.324 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.324 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1373 3.12 3:495d4bbc:::smithi09025264-16:head [write 2873523~591733] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:31:12.324 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:31:12.324 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.484788675999994, 2023-01-15T14:31:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40949580099999999, 2023-01-15T14:31:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1373 2023-01-15T14:31:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:31:12.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:31:12.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693440+0000", 2023-01-15T14:31:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693808+0000", 2023-01-15T14:31:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000367742 2023-01-15T14:31:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693831+0000", 2023-01-15T14:31:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3213e-05 2023-01-15T14:31:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693846+0000", 2023-01-15T14:31:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4909000000000001e-05 2023-01-15T14:31:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.695833+0000", 2023-01-15T14:31:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019869240000000002 2023-01-15T14:31:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696121+0000", 2023-01-15T14:31:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028753300000000003 2023-01-15T14:31:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798577+0000", 2023-01-15T14:31:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102456808 2023-01-15T14:31:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798849+0000", 2023-01-15T14:31:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027138999999999999 2023-01-15T14:31:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799155+0000", 2023-01-15T14:31:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030594800000000001 2023-01-15T14:31:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799614+0000", 2023-01-15T14:31:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045960700000000001 2023-01-15T14:31:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799737+0000", 2023-01-15T14:31:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012218900000000001 2023-01-15T14:31:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799998+0000", 2023-01-15T14:31:12.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026129700000000002 2023-01-15T14:31:12.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800639+0000", 2023-01-15T14:31:12.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064108800000000005 2023-01-15T14:31:12.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102023+0000", 2023-01-15T14:31:12.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30138417200000001 2023-01-15T14:31:12.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102198+0000", 2023-01-15T14:31:12.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017453600000000001 2023-01-15T14:31:12.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102398+0000", 2023-01-15T14:31:12.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020078000000000001 2023-01-15T14:31:12.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102937+0000", 2023-01-15T14:31:12.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053874499999999998 2023-01-15T14:31:12.334 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.334 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.334 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.335 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:31:12.335 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:31:12.335 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.484364081999999, 2023-01-15T14:31:12.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:31:12.335 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.335 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.335 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.335 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.335 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.335 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:31:12.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.336 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:31:12.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:31:12.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:31:12.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:12.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:31:12.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:31:12.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:31:12.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:31:12.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:31:12.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:31:12.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:31:12.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:31:12.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:31:12.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:31:12.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:31:12.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:31:12.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:31:12.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:31:12.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:12.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:31:12.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:31:12.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:31:12.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:31:12.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:31:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:31:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:31:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:31:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:31:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:31:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:31:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:31:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:31:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:31:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:31:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:31:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:31:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:31:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:31:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:31:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 64.484311753, 2023-01-15T14:31:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:31:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:31:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:31:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:31:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:31:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:31:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:31:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:31:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:31:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:31:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:31:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:31:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:31:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:31:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:31:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:31:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:31:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:31:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:31:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:31:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:31:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.887398554000001, 2023-01-15T14:31:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:31:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:31:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:31:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:31:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:31:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:31:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:31:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:31:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:31:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:31:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:31:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:31:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:31:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.359 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.359 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.359 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:12.360 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:12.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:31:12.361 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:12.361 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:12.362 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:31:12.362 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:31:12.362 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:31:12.362 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:31:12.362 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: filestore_inject_stall = '0' (not observed, change may require restart) osd_enable_op_tracker = 'true' 2023-01-15T14:31:12.362 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:31:12.363 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:31:12.363 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:31:12.363 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:12.356+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:12.461 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:12.456+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:12.515 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:12.516 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:12.526 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:31:12.561 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:12.558+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:12.629 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:12.629 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:12.630 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:12.630 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:12.639 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:31:12.662 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:12.658+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:12.752 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:12.753 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:12.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:12.753 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:12.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.753 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.753 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:12.753 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.66234058000001, 2023-01-15T14:31:12.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:31:12.753 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.754 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:31:12.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.754 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:12.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:31:12.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:31:12.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:31:12.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:31:12.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:31:12.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:31:12.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:31:12.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:31:12.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:31:12.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:12.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:31:12.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:31:12.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:31:12.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:31:12.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:31:12.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:31:12.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:31:12.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:31:12.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:31:12.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:31:12.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:31:12.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:31:12.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:31:12.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:31:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:31:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:31:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:31:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:31:12.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.66204078300001, 2023-01-15T14:31:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:31:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:31:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:31:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:31:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:31:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:31:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:31:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:31:12.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:31:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:31:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:31:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:31:12.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:31:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:31:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:31:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:31:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:31:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:31:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:31:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:31:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:31:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:31:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:31:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:31:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:31:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:31:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:31:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:31:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.66201099899999, 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.773 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:31:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:31:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:31:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:31:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:31:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:31:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:31:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:31:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:31:12.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:31:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:31:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:31:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:31:12.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:31:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:31:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:31:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:31:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:31:12.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:31:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:31:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:31:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:31:12.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:182 3.4 3:2022fdfa:::smithi09025264-46:head [write 1933572~791670] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.53314893000001, 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.582304865, 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 182 2023-01-15T14:31:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213704+0000", 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214574+0000", 2023-01-15T14:31:12.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087037799999999999 2023-01-15T14:31:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214608+0000", 2023-01-15T14:31:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:31:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214627+0000", 2023-01-15T14:31:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:31:12.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491537+0000", 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27691020300000002 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491721+0000", 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345500000000001 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693760+0000", 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202039252 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693898+0000", 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013789 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694230+0000", 2023-01-15T14:31:12.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220899999999998 2023-01-15T14:31:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694866+0000", 2023-01-15T14:31:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063640599999999995 2023-01-15T14:31:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694887+0000", 2023-01-15T14:31:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0622999999999999e-05 2023-01-15T14:31:12.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695950+0000", 2023-01-15T14:31:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001062688 2023-01-15T14:31:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696014+0000", 2023-01-15T14:31:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4697999999999996e-05 2023-01-15T14:31:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795898+0000", 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099883338000000002 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795922+0000", 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4326000000000001e-05 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795955+0000", 2023-01-15T14:31:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3191000000000001e-05 2023-01-15T14:31:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796011+0000", 2023-01-15T14:31:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5631999999999999e-05 2023-01-15T14:31:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.53220219900001, 2023-01-15T14:31:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:31:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:31:12.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:31:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:31:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:31:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:31:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:31:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:31:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:31:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:31:12.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:31:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:31:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:31:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:31:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:31:12.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:31:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:31:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:31:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:31:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:31:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:31:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:31:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:31:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:31:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:31:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:31:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:31:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:31:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:31:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:31:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:31:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.53214550199999, 2023-01-15T14:31:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:31:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:31:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:31:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:31:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:31:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:31:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:31:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:31:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:31:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:31:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:31:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:31:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 162.91564932899999, 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:31:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:31:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:31:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:31:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:31:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:31:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:31:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:31:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:31:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:31:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:31:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:31:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 155.88680085799999, 2023-01-15T14:31:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:31:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:31:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:31:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:31:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:31:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:31:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:31:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:31:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:31:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:31:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:31:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:31:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:31:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:31:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:31:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:31:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:31:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:31:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:31:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:31:12.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:31:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:31:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:31:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:31:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:373 3.4 3:2029fa71:::smithi09025264-39:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.668553651, 2023-01-15T14:31:12.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58298356399999995, 2023-01-15T14:31:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-01-15T14:31:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078299+0000", 2023-01-15T14:31:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078644+0000", 2023-01-15T14:31:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034494099999999998 2023-01-15T14:31:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078676+0000", 2023-01-15T14:31:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1183000000000003e-05 2023-01-15T14:31:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078694+0000", 2023-01-15T14:31:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-05 2023-01-15T14:31:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078748+0000", 2023-01-15T14:31:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3634999999999997e-05 2023-01-15T14:31:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078753+0000", 2023-01-15T14:31:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6370000000000004e-06 2023-01-15T14:31:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110841+0000", 2023-01-15T14:31:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032087439000000002 2023-01-15T14:31:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110967+0000", 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126029 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.457874+0000", 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.346907725 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458066+0000", 2023-01-15T14:31:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000191592 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458321+0000", 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025454200000000002 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458918+0000", 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059740800000000001 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458945+0000", 2023-01-15T14:31:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-05 2023-01-15T14:31:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459253+0000", 2023-01-15T14:31:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030836200000000001 2023-01-15T14:31:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460943+0000", 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016903580000000001 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461340+0000", 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039665799999999998 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661129+0000", 2023-01-15T14:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19978933500000001 2023-01-15T14:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661188+0000", 2023-01-15T14:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8687000000000002e-05 2023-01-15T14:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661285+0000", 2023-01-15T14:31:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779000000000003e-05 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:375 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (56) in=7b,truncate 3142876] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.66374997299999, 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57851114599999998, 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 375 2023-01-15T14:31:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:31:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083104+0000", 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083113+0000", 2023-01-15T14:31:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6500000000000008e-06 2023-01-15T14:31:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083130+0000", 2023-01-15T14:31:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2023-01-15T14:31:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083142+0000", 2023-01-15T14:31:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2252999999999999e-05 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083173+0000", 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0474999999999999e-05 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083175+0000", 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183469+0000", 2023-01-15T14:31:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10029313099999999 2023-01-15T14:31:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183747+0000", 2023-01-15T14:31:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000277998 2023-01-15T14:31:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460165+0000", 2023-01-15T14:31:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27641872899999997 2023-01-15T14:31:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460342+0000", 2023-01-15T14:31:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176792 2023-01-15T14:31:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460579+0000", 2023-01-15T14:31:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000237266 2023-01-15T14:31:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460833+0000", 2023-01-15T14:31:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025390900000000002 2023-01-15T14:31:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460853+0000", 2023-01-15T14:31:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9273e-05 2023-01-15T14:31:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461024+0000", 2023-01-15T14:31:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017111900000000001 2023-01-15T14:31:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461230+0000", 2023-01-15T14:31:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020672600000000001 2023-01-15T14:31:12.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461879+0000", 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064846999999999995 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661524+0000", 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19964509599999999 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661560+0000", 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6264999999999999e-05 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661616+0000", 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5952000000000002e-05 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:376 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.66366771700001, 2023-01-15T14:31:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57864278800000002, 2023-01-15T14:31:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 376 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083186+0000", 2023-01-15T14:31:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083193+0000", 2023-01-15T14:31:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2023-01-15T14:31:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083211+0000", 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7609999999999999e-05 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083223+0000", 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140000000000001e-05 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093534+0000", 2023-01-15T14:31:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010311529 2023-01-15T14:31:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093551+0000", 2023-01-15T14:31:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.666e-05 2023-01-15T14:31:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183834+0000", 2023-01-15T14:31:12.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090283063999999996 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.184076+0000", 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024241900000000001 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461001+0000", 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27692504299999998 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461152+0000", 2023-01-15T14:31:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150345 2023-01-15T14:31:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661642+0000", 2023-01-15T14:31:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200489742 2023-01-15T14:31:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661720+0000", 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8604999999999997e-05 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661830+0000", 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010992099999999999 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:31:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.885513882, 2023-01-15T14:31:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:31:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:31:12.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:31:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:31:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:31:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:31:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:31:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:31:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:31:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:31:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:31:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:31:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:12.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:31:12.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:31:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:31:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:31:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:31:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:31:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:31:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:31:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:31:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:31:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.824935632000006, 2023-01-15T14:31:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:31:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:31:12.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:31:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:31:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:31:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:31:12.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:31:12.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:31:12.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:31:12.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:12.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:31:12.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:31:12.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:31:12.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:31:12.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:31:12.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:31:12.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:31:12.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:31:12.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:31:12.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:31:12.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:31:12.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:31:12.871 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:12.872 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.824876922000001, 2023-01-15T14:31:12.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:31:12.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.872 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:31:12.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.872 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:12.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:12.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:31:12.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:12.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:31:12.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:31:12.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:31:12.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:31:12.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:31:12.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:31:12.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:31:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:31:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:31:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:31:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:31:12.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:31:12.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:31:12.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:31:12.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:31:12.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:31:12.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:31:12.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:31:12.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:31:12.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:31:12.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.878 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.879 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.879 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:12.879 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:12.879 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.868457429000003, 2023-01-15T14:31:12.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:31:12.879 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.879 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.880 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:31:12.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.880 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:12.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:12.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:31:12.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:12.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:31:12.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:31:12.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:31:12.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:31:12.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:31:12.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:31:12.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:31:12.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:31:12.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:12.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:31:12.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:31:12.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:31:12.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:31:12.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:31:12.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:31:12.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:31:12.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:31:12.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.885 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.886 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.886 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:12.886 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:12.886 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 74.810959745999995, 2023-01-15T14:31:12.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:31:12.886 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.886 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.887 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.887 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:31:12.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.887 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:12.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:12.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:31:12.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:12.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:31:12.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:31:12.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:31:12.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:12.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:31:12.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:31:12.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:31:12.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:31:12.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:31:12.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:31:12.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:31:12.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:31:12.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:31:12.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:31:12.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:31:12.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:31:12.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:31:12.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:31:12.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:31:12.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:31:12.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:31:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:31:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:31:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:31:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:31:12.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.442308240999999, 2023-01-15T14:31:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:31:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.897 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:31:12.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.897 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:12.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:12.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:31:12.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:12.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:31:12.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:31:12.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:31:12.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:31:12.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:31:12.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:31:12.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:31:12.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:31:12.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:31:12.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:31:12.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:12.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:31:12.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:31:12.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:31:12.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:31:12.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:31:12.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:31:12.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:31:12.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:31:12.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:31:12.904 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.904 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 53.442205600000001, 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.905 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:12.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:31:12.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:31:12.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:31:12.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:31:12.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:31:12.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.912 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:12.912 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:12.912 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.8412236049999997, 2023-01-15T14:31:12.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:31:12.912 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.912 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:12.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:12.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:31:12.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:31:12.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:31:12.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:31:12.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:31:12.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:31:12.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:31:12.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:31:12.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:12.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:31:12.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:31:12.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:31:12.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:31:12.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:31:12.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:31:12.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:31:12.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:31:12.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:31:12.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:31:12.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:31:12.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:31:12.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:31:12.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:31:12.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:31:12.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:31:12.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:31:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:31:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:31:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:31:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:31:12.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:31:12.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:31:12.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:31:12.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:31:12.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:31:12.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:31:12.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:31:12.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.8411981809999993, 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:12.925 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:12.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:31:12.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:31:12.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:12.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:31:12.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:31:12.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:31:12.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:12.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:31:12.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:12.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:31:12.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:31:12.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:31:12.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:31:12.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:12.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:31:12.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:12.936 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:12.936 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:12.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:31:12.939 INFO:tasks.rados.rados.0.smithi090.stdout:711: done (0 left) 2023-01-15T14:31:12.939 INFO:tasks.rados.rados.0.smithi090.stdout:712: write oid 14 current snap is 73 2023-01-15T14:31:12.939 INFO:tasks.rados.rados.0.smithi090.stdout:712: seq_num 279 ranges {689153=783179,2076829=457413} 2023-01-15T14:31:12.939 INFO:tasks.rados.rados.0.smithi090.stdout:712: writing smithi09025264-14 from 689153 to 1472332 tid 1 2023-01-15T14:31:12.939 INFO:tasks.rados.rados.0.smithi090.stdout:712: writing smithi09025264-14 from 2076829 to 2534242 tid 2 2023-01-15T14:31:12.939 INFO:tasks.rados.rados.0.smithi090.stdout:713: read oid 32 snap -1 2023-01-15T14:31:12.939 INFO:tasks.rados.rados.0.smithi090.stdout:713: expect (ObjNum 255 snap 60 seq_num 255) 2023-01-15T14:31:12.939 INFO:tasks.rados.rados.0.smithi090.stdout:714: read oid 39 snap -1 2023-01-15T14:31:12.939 INFO:tasks.rados.rados.0.smithi090.stdout:714: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:31:12.939 INFO:tasks.rados.rados.0.smithi090.stdout:715: delete oid 26 current snap is 73 2023-01-15T14:31:12.939 INFO:tasks.rados.rados.0.smithi090.stdout:715: done (3 left) 2023-01-15T14:31:12.940 INFO:tasks.rados.rados.0.smithi090.stdout:716: read oid 24 snap 72 2023-01-15T14:31:12.940 INFO:tasks.rados.rados.0.smithi090.stdout:716: expect (ObjNum 266 snap 66 seq_num 266) 2023-01-15T14:31:12.940 INFO:tasks.rados.rados.0.smithi090.stdout:717: read oid 36 snap -1 2023-01-15T14:31:12.940 INFO:tasks.rados.rados.0.smithi090.stdout:717: expect (ObjNum 199 snap 43 seq_num 199) 2023-01-15T14:31:12.940 INFO:tasks.rados.rados.0.smithi090.stdout:718: snap_create 2023-01-15T14:31:12.940 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:31:12.940 INFO:tasks.rados.rados.0.smithi090.stdout:713: expect (ObjNum 255 snap 60 seq_num 255) 2023-01-15T14:31:12.941 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:12.934+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:13.003 INFO:tasks.rados.rados.0.smithi090.stdout:712: finishing write tid 1 to smithi09025264-14 2023-01-15T14:31:13.003 INFO:tasks.rados.rados.0.smithi090.stdout:714: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:31:13.038 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:13.034+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:13.065 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:13.065 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:13.065 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:13.065 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:31:13.139 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:13.134+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:13.168 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:13.265 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:13.265 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:13.265 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:13.265 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:13.267 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:31:13.268 INFO:tasks.rados.rados.0.smithi090.stdout:712: finishing write tid 2 to smithi09025264-14 2023-01-15T14:31:13.268 INFO:tasks.rados.rados.0.smithi090.stdout:713: done (4 left) 2023-01-15T14:31:13.268 INFO:tasks.rados.rados.0.smithi090.stdout:714: done (3 left) 2023-01-15T14:31:13.268 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:31:13.268 INFO:tasks.rados.rados.0.smithi090.stdout:712: finishing write tid 3 to smithi09025264-14 2023-01-15T14:31:13.268 INFO:tasks.rados.rados.0.smithi090.stdout:712: finishing write tid 4 to smithi09025264-14 2023-01-15T14:31:13.269 INFO:tasks.rados.rados.0.smithi090.stdout:712: oid 14 updating version 0 to 226 2023-01-15T14:31:13.269 INFO:tasks.rados.rados.0.smithi090.stdout:712: oid 14 updating version 226 to 227 2023-01-15T14:31:13.269 INFO:tasks.rados.rados.0.smithi090.stdout:712: oid 14 updating version 227 to 228 2023-01-15T14:31:13.269 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 228 (ObjNum 279 snap 73 seq_num 279) dirty exists 2023-01-15T14:31:13.269 INFO:tasks.rados.rados.0.smithi090.stdout:712: left oid 14 (ObjNum 279 snap 73 seq_num 279) 2023-01-15T14:31:13.269 INFO:tasks.rados.rados.0.smithi090.stdout:716: expect (ObjNum 266 snap 66 seq_num 266) 2023-01-15T14:31:13.270 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:13.262+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:13.311 INFO:tasks.rados.rados.0.smithi090.stdout:717: expect (ObjNum 199 snap 43 seq_num 199) 2023-01-15T14:31:13.367 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:13.364+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:13.461 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:13.461 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:13.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:13.462 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:13.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.462 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:38 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'3, mlcod=0'0)", 2023-01-15T14:31:13.462 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:13.462 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.26166281299999, 2023-01-15T14:31:13.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30191175999999997, 2023-01-15T14:31:13.462 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.463 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.463 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:13.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:13.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112976+0000", 2023-01-15T14:31:13.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:13.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113186+0000", 2023-01-15T14:31:13.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020979700000000001 2023-01-15T14:31:13.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113228+0000", 2023-01-15T14:31:13.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2150999999999997e-05 2023-01-15T14:31:13.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213313+0000", 2023-01-15T14:31:13.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100084943 2023-01-15T14:31:13.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313517+0000", 2023-01-15T14:31:13.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10020435699999999 2023-01-15T14:31:13.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313576+0000", 2023-01-15T14:31:13.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783999999999998e-05 2023-01-15T14:31:13.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313976+0000", 2023-01-15T14:31:13.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039960499999999998 2023-01-15T14:31:13.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314004+0000", 2023-01-15T14:31:13.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7954e-05 2023-01-15T14:31:13.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314175+0000", 2023-01-15T14:31:13.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171557 2023-01-15T14:31:13.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414853+0000", 2023-01-15T14:31:13.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100677875 2023-01-15T14:31:13.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414890+0000", 2023-01-15T14:31:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6803e-05 2023-01-15T14:31:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:46 3.0 e23/20 3:03b54904:::smithi09025264-12:head v 23'5, mlcod=23'1)", 2023-01-15T14:31:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:31:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.16076689900001, 2023-01-15T14:31:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202592668, 2023-01-15T14:31:13.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.476 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.476 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:31:13.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:31:13.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213873+0000", 2023-01-15T14:31:13.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:13.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214477+0000", 2023-01-15T14:31:13.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060481300000000005 2023-01-15T14:31:13.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214532+0000", 2023-01-15T14:31:13.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5096999999999998e-05 2023-01-15T14:31:13.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214555+0000", 2023-01-15T14:31:13.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.234e-05 2023-01-15T14:31:13.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414409+0000", 2023-01-15T14:31:13.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19985461199999999 2023-01-15T14:31:13.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414442+0000", 2023-01-15T14:31:13.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2987000000000001e-05 2023-01-15T14:31:13.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414813+0000", 2023-01-15T14:31:13.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037061899999999998 2023-01-15T14:31:13.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414832+0000", 2023-01-15T14:31:13.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9380000000000001e-05 2023-01-15T14:31:13.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415093+0000", 2023-01-15T14:31:13.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026032099999999999 2023-01-15T14:31:13.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415154+0000", 2023-01-15T14:31:13.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1538999999999995e-05 2023-01-15T14:31:13.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.416467+0000", 2023-01-15T14:31:13.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001312613 2023-01-15T14:31:13.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.488 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.489 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.489 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:125 3.0 e70/55 3:0b54190a:::smithi09025264-17:25 v 70'136, mlcod=68'130)", 2023-01-15T14:31:13.489 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:13.489 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 117.43168078799999, 2023-01-15T14:31:13.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30171926799999998, 2023-01-15T14:31:13.490 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.490 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.490 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:13.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:13.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942958+0000", 2023-01-15T14:31:13.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:13.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942972+0000", 2023-01-15T14:31:13.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3386e-05 2023-01-15T14:31:13.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942990+0000", 2023-01-15T14:31:13.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8113000000000001e-05 2023-01-15T14:31:13.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.943002+0000", 2023-01-15T14:31:13.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:31:13.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141942+0000", 2023-01-15T14:31:13.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19893962300000001 2023-01-15T14:31:13.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141978+0000", 2023-01-15T14:31:13.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6383e-05 2023-01-15T14:31:13.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142670+0000", 2023-01-15T14:31:13.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069211900000000005 2023-01-15T14:31:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142688+0000", 2023-01-15T14:31:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7839999999999999e-05 2023-01-15T14:31:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142768+0000", 2023-01-15T14:31:13.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0210999999999995e-05 2023-01-15T14:31:13.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244581+0000", 2023-01-15T14:31:13.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101813406 2023-01-15T14:31:13.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244680+0000", 2023-01-15T14:31:13.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8045000000000001e-05 2023-01-15T14:31:13.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.500 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.500 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.500 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(3.0 REQUEST e99)", 2023-01-15T14:31:13.500 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:13.501 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.378227144999997, 2023-01-15T14:31:13.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26432840200000002, 2023-01-15T14:31:13.501 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.501 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:13.501 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:13.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:13.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:13.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:13.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996419+0000", 2023-01-15T14:31:13.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6729999999999997e-06 2023-01-15T14:31:13.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996449+0000", 2023-01-15T14:31:13.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.004e-05 2023-01-15T14:31:13.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996463+0000", 2023-01-15T14:31:13.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3607e-05 2023-01-15T14:31:13.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996594+0000", 2023-01-15T14:31:13.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013148899999999999 2023-01-15T14:31:13.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:31:13.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996618+0000", 2023-01-15T14:31:13.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3249000000000001e-05 2023-01-15T14:31:13.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260645+0000", 2023-01-15T14:31:13.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26402768700000001 2023-01-15T14:31:13.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260671+0000", 2023-01-15T14:31:13.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.546e-05 2023-01-15T14:31:13.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260742+0000", 2023-01-15T14:31:13.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1569999999999994e-05 2023-01-15T14:31:13.509 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.510 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.510 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:261 3.8 e98/41 3:1e54a7f5:::smithi09025264-42:3e v 98'160, mlcod=97'157)", 2023-01-15T14:31:13.511 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:31:13.511 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.317159556000007, 2023-01-15T14:31:13.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20299468400000001, 2023-01-15T14:31:13.511 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.511 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.512 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:31:13.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:31:13.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057480+0000", 2023-01-15T14:31:13.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:13.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057496+0000", 2023-01-15T14:31:13.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5869999999999999e-05 2023-01-15T14:31:13.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057528+0000", 2023-01-15T14:31:13.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:31:13.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057545+0000", 2023-01-15T14:31:13.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.694e-05 2023-01-15T14:31:13.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.158971+0000", 2023-01-15T14:31:13.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101425425 2023-01-15T14:31:13.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159011+0000", 2023-01-15T14:31:13.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0058000000000002e-05 2023-01-15T14:31:13.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159858+0000", 2023-01-15T14:31:13.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084767900000000005 2023-01-15T14:31:13.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159877+0000", 2023-01-15T14:31:13.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9133000000000001e-05 2023-01-15T14:31:13.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159959+0000", 2023-01-15T14:31:13.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1693000000000005e-05 2023-01-15T14:31:13.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260430+0000", 2023-01-15T14:31:13.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10047039200000001 2023-01-15T14:31:13.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260476+0000", 2023-01-15T14:31:13.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6652e-05 2023-01-15T14:31:13.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.525 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.525 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:31:13.525 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:13.525 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.315938399999993, 2023-01-15T14:31:13.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:31:13.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.526 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.526 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:13.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:13.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:31:13.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:13.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:31:13.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:31:13.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:31:13.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:31:13.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:31:13.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:31:13.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:31:13.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:31:13.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:31:13.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:31:13.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:31:13.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:31:13.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:31:13.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:31:13.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:31:13.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:31:13.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:31:13.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:31:13.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:31:13.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:31:13.533 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.534 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.534 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:194 3.4 e101/48 3:2022fdfa:::smithi09025264-46:43 v 101'104, mlcod=98'100)", 2023-01-15T14:31:13.534 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:31:13.534 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.529896789000006, 2023-01-15T14:31:13.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20670733199999999, 2023-01-15T14:31:13.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.534 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844743+0000", 2023-01-15T14:31:13.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:13.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844757+0000", 2023-01-15T14:31:13.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.484e-05 2023-01-15T14:31:13.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844787+0000", 2023-01-15T14:31:13.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9113000000000001e-05 2023-01-15T14:31:13.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.944923+0000", 2023-01-15T14:31:13.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013656 2023-01-15T14:31:13.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.949975+0000", 2023-01-15T14:31:13.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050521400000000001 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950015+0000", 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9286999999999998e-05 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950559+0000", 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054463500000000002 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950585+0000", 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6166e-05 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950670+0000", 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4902999999999997e-05 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051368+0000", 2023-01-15T14:31:13.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100697913 2023-01-15T14:31:13.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051452+0000", 2023-01-15T14:31:13.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3442000000000005e-05 2023-01-15T14:31:13.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.540 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.540 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.541 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1224 3.4 e107/48 3:2029fa71:::smithi09025264-39:head v 107'120, mlcod=104'116)", 2023-01-15T14:31:13.541 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:31:13.541 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.142868148999995, 2023-01-15T14:31:13.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22601005900000001, 2023-01-15T14:31:13.541 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.541 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.541 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231771+0000", 2023-01-15T14:31:13.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:13.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233829+0000", 2023-01-15T14:31:13.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002057836 2023-01-15T14:31:13.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233881+0000", 2023-01-15T14:31:13.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1856e-05 2023-01-15T14:31:13.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.333996+0000", 2023-01-15T14:31:13.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100115315 2023-01-15T14:31:13.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434199+0000", 2023-01-15T14:31:13.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100203373 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434236+0000", 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6563000000000002e-05 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.435979+0000", 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017426570000000001 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.436036+0000", 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7716000000000001e-05 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437140+0000", 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011036399999999999 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437251+0000", 2023-01-15T14:31:13.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110869 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.457783+0000", 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020532181 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1649 3.4 3:2029fa71:::smithi09025264-39:head [write 534522~634550] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.893806+0000", 2023-01-15T14:31:13.547 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.480835071999998, 2023-01-15T14:31:13.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205815042, 2023-01-15T14:31:13.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.548 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:13.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:13.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:13.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1649 2023-01-15T14:31:13.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.548 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.893806+0000", 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.893806+0000", 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.893803+0000", 2023-01-15T14:31:13.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:31:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894135+0000", 2023-01-15T14:31:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033239899999999999 2023-01-15T14:31:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894168+0000", 2023-01-15T14:31:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2490000000000002e-05 2023-01-15T14:31:13.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894195+0000", 2023-01-15T14:31:13.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7576000000000002e-05 2023-01-15T14:31:13.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894997+0000", 2023-01-15T14:31:13.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080147300000000003 2023-01-15T14:31:13.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:13.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895010+0000", 2023-01-15T14:31:13.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611000000000001e-05 2023-01-15T14:31:13.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.991782+0000", 2023-01-15T14:31:13.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096771757 2023-01-15T14:31:13.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.992255+0000", 2023-01-15T14:31:13.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047275099999999999 2023-01-15T14:31:13.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:13.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.992633+0000", 2023-01-15T14:31:13.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037850299999999998 2023-01-15T14:31:13.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.995684+0000", 2023-01-15T14:31:13.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030508610000000002 2023-01-15T14:31:13.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.995715+0000", 2023-01-15T14:31:13.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0759000000000001e-05 2023-01-15T14:31:13.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.996025+0000", 2023-01-15T14:31:13.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003102 2023-01-15T14:31:13.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.996334+0000", 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030889699999999998 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098878+0000", 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10254395299999999 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099030+0000", 2023-01-15T14:31:13.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152192 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099198+0000", 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016808400000000001 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099621+0000", 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042263899999999999 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.678 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.679 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1650 3.4 3:2029fa71:::smithi09025264-39:head [write 1812335~322187] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:13.679 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894235+0000", 2023-01-15T14:31:13.679 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.480405664000003, 2023-01-15T14:31:13.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20642070600000001, 2023-01-15T14:31:13.679 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.679 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.679 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:13.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:13.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:13.680 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1650 2023-01-15T14:31:13.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.680 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894235+0000", 2023-01-15T14:31:13.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894235+0000", 2023-01-15T14:31:13.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894234+0000", 2023-01-15T14:31:13.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:13.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894404+0000", 2023-01-15T14:31:13.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017082499999999999 2023-01-15T14:31:13.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894426+0000", 2023-01-15T14:31:13.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1387e-05 2023-01-15T14:31:13.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894440+0000", 2023-01-15T14:31:13.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3753e-05 2023-01-15T14:31:13.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895046+0000", 2023-01-15T14:31:13.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060607599999999999 2023-01-15T14:31:13.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:13.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895054+0000", 2023-01-15T14:31:13.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6130000000000004e-06 2023-01-15T14:31:13.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.096846+0000", 2023-01-15T14:31:13.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201791888 2023-01-15T14:31:13.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.097087+0000", 2023-01-15T14:31:13.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024123800000000001 2023-01-15T14:31:13.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:13.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.097486+0000", 2023-01-15T14:31:13.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039807900000000002 2023-01-15T14:31:13.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098073+0000", 2023-01-15T14:31:13.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058732299999999995 2023-01-15T14:31:13.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098097+0000", 2023-01-15T14:31:13.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3773000000000001e-05 2023-01-15T14:31:13.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098309+0000", 2023-01-15T14:31:13.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021264999999999999 2023-01-15T14:31:13.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:13.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098383+0000", 2023-01-15T14:31:13.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3819999999999995e-05 2023-01-15T14:31:13.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099809+0000", 2023-01-15T14:31:13.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014256360000000001 2023-01-15T14:31:13.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100004+0000", 2023-01-15T14:31:13.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019514099999999999 2023-01-15T14:31:13.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100220+0000", 2023-01-15T14:31:13.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021621800000000001 2023-01-15T14:31:13.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100656+0000", 2023-01-15T14:31:13.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000436036 2023-01-15T14:31:13.690 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.690 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.690 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.691 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:13.691 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:13.691 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.480181010000003, 2023-01-15T14:31:13.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:31:13.691 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.691 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.691 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:13.691 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:13.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:13.692 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:31:13.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.692 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:13.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:13.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:31:13.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:13.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:31:13.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:31:13.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:31:13.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:31:13.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:31:13.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:31:13.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:31:13.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:31:13.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:13.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:31:13.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:31:13.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:31:13.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:31:13.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:31:13.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:31:13.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:13.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:31:13.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:31:13.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:31:13.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:31:13.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:31:13.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:31:13.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:31:13.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:31:13.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:13.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:31:13.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:31:13.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:31:13.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:31:13.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:31:13.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:31:13.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:31:13.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:31:13.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:31:13.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:31:13.702 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.702 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.702 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.702 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:13.702 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:13.703 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.480130066999997, 2023-01-15T14:31:13.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:31:13.703 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.703 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:13.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:13.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:13.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:13.703 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:31:13.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.704 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:13.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:13.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:31:13.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:13.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:31:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:31:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:31:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:31:13.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:31:13.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:31:13.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:31:13.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:31:13.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:13.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:31:13.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:31:13.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:31:13.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:31:13.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:13.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:31:13.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:31:13.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:31:13.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:31:13.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:31:13.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:31:13.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:31:13.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:31:13.711 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.711 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.711 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.711 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:31:13.711 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:13.712 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.539230826000001, 2023-01-15T14:31:13.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:31:13.712 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.712 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.712 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:13.712 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:13.712 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:13.712 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:31:13.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.713 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:13.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:13.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:31:13.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:13.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:31:13.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:31:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:31:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:31:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:31:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:31:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:31:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:31:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:31:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:31:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:31:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:31:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:31:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:31:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:31:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:31:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:31:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:31:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:31:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:31:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:31:13.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:31:13.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:31:13.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:31:13.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.721 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.465456652999997, 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:13.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:13.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:31:13.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:13.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:31:13.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:31:13.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:31:13.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:31:13.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:31:13.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:31:13.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:31:13.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:31:13.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:13.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:31:13.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:31:13.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:31:13.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:31:13.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:31:13.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:31:13.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:31:13.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:31:13.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:31:13.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:31:13.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:13.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:31:13.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:31:13.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:31:13.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:31:13.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:31:13.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:31:13.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:31:13.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:31:13.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:31:13.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:31:13.731 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.731 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.732 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1712 3.4 3:2022fdfa:::smithi09025264-46:head [write 1880802~421790] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:13.732 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:13.732 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 33.460317961999998, 2023-01-15T14:31:13.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30396041800000001, 2023-01-15T14:31:13.732 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.732 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:13.732 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1712 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914322+0000", 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914561+0000", 2023-01-15T14:31:13.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023884700000000001 2023-01-15T14:31:13.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914572+0000", 2023-01-15T14:31:13.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0499999999999999e-05 2023-01-15T14:31:13.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914576+0000", 2023-01-15T14:31:13.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6820000000000002e-06 2023-01-15T14:31:13.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014941+0000", 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100364994 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014959+0000", 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7416000000000001e-05 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018171+0000", 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032121239999999998 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018333+0000", 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016180900000000001 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018581+0000", 2023-01-15T14:31:13.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024773699999999999 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018897+0000", 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031622099999999999 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018907+0000", 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0061000000000001e-05 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:13.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.019108+0000", 2023-01-15T14:31:13.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020080099999999999 2023-01-15T14:31:13.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:13.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.020057+0000", 2023-01-15T14:31:13.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094954499999999999 2023-01-15T14:31:13.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150472+0000", 2023-01-15T14:31:13.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.130414958 2023-01-15T14:31:13.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:13.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218188+0000", 2023-01-15T14:31:13.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067715656999999999 2023-01-15T14:31:13.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:13.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218242+0000", 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4651999999999998e-05 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218284+0000", 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1084999999999997e-05 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.741 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.18 157/147 [PushOp(3:1eaad226:::smithi09025264-49:69, version: 139'201, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:1eaad226:::smithi09025264-49:69@139'201, size: 3916031, copy_subset: [0~3916031], clone_subset: {}, snapset: 69=[]:{68=[67],69=[69]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:31:13.742 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:13.742 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 28.66540741, 2023-01-15T14:31:13.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27553999000000001, 2023-01-15T14:31:13.742 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.742 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:13.742 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:13.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:13.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709232+0000", 2023-01-15T14:31:13.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:13.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709949+0000", 2023-01-15T14:31:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071723000000000002 2023-01-15T14:31:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709990+0000", 2023-01-15T14:31:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.015e-05 2023-01-15T14:31:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.810156+0000", 2023-01-15T14:31:13.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10016643 2023-01-15T14:31:13.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984513+0000", 2023-01-15T14:31:13.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.174357128 2023-01-15T14:31:13.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984522+0000", 2023-01-15T14:31:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0350000000000007e-06 2023-01-15T14:31:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984774+0000", 2023-01-15T14:31:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025144700000000002 2023-01-15T14:31:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.747 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:13.747 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:13.747 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.5715012329999993, 2023-01-15T14:31:13.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:31:13.747 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.747 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:13.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:13.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:13.747 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:13.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:31:13.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:31:13.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:13.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:31:13.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:31:13.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:31:13.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:31:13.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:13.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:31:13.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:31:13.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:31:13.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:31:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:31:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:31:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:31:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:31:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1820 3.1f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.4711997480000001, 2023-01-15T14:31:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306504938, 2023-01-15T14:31:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:13.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:13.882 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1820 2023-01-15T14:31:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903440+0000", 2023-01-15T14:31:13.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903448+0000", 2023-01-15T14:31:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9430000000000006e-06 2023-01-15T14:31:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903469+0000", 2023-01-15T14:31:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0305e-05 2023-01-15T14:31:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903481+0000", 2023-01-15T14:31:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:31:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903616+0000", 2023-01-15T14:31:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013477199999999999 2023-01-15T14:31:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:13.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903643+0000", 2023-01-15T14:31:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7098999999999999e-05 2023-01-15T14:31:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209081+0000", 2023-01-15T14:31:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30543790300000001 2023-01-15T14:31:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:13.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209213+0000", 2023-01-15T14:31:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013155400000000001 2023-01-15T14:31:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209600+0000", 2023-01-15T14:31:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038751699999999998 2023-01-15T14:31:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209946+0000", 2023-01-15T14:31:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034601 2023-01-15T14:31:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.897 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(3.1f 177/176 [PushReplyOp(3:f87c0183:::smithi09025264-50:head)])", 2023-01-15T14:31:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:04.427925+0000", 2023-01-15T14:31:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.9467163480000007, 2023-01-15T14:31:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202446562, 2023-01-15T14:31:13.898 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.427925+0000", 2023-01-15T14:31:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.427925+0000", 2023-01-15T14:31:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.427923+0000", 2023-01-15T14:31:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.427937+0000", 2023-01-15T14:31:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4389999999999999e-05 2023-01-15T14:31:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.427969+0000", 2023-01-15T14:31:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2030000000000003e-05 2023-01-15T14:31:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.528100+0000", 2023-01-15T14:31:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013047999999999 2023-01-15T14:31:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.628327+0000", 2023-01-15T14:31:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100227102 2023-01-15T14:31:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.630371+0000", 2023-01-15T14:31:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002044424 2023-01-15T14:31:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.907 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.908 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.12 178/175 [PushOp(3:495d4bbc:::smithi09025264-16:81, version: 172'127, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:495d4bbc:::smithi09025264-16:81@172'127, size: 3512663, copy_subset: [0~3512663], clone_subset: {}, snapset: 81=[]:{81=[7d,7c,7b]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-01-15T14:31:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:05.000831+0000", 2023-01-15T14:31:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.3738098369999996, 2023-01-15T14:31:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20204813899999999, 2023-01-15T14:31:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:13.909 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:05.000831+0000", 2023-01-15T14:31:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:05.000831+0000", 2023-01-15T14:31:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:13.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:05.000829+0000", 2023-01-15T14:31:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:13.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:05.001903+0000", 2023-01-15T14:31:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001074204 2023-01-15T14:31:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:05.001943+0000", 2023-01-15T14:31:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0275999999999999e-05 2023-01-15T14:31:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:05.001968+0000", 2023-01-15T14:31:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4712999999999999e-05 2023-01-15T14:31:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:05.202343+0000", 2023-01-15T14:31:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20037522399999999 2023-01-15T14:31:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:05.202358+0000", 2023-01-15T14:31:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.415e-05 2023-01-15T14:31:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:13.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:05.202879+0000", 2023-01-15T14:31:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052185800000000004 2023-01-15T14:31:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:13.919 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:13.919 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:13.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:31:13.923 INFO:tasks.rados.rados.0.smithi090.stdout:712: done (2 left) 2023-01-15T14:31:13.923 INFO:tasks.rados.rados.0.smithi090.stdout:716: done (1 left) 2023-01-15T14:31:13.923 INFO:tasks.rados.rados.0.smithi090.stdout:717: done (0 left) 2023-01-15T14:31:13.923 INFO:tasks.rados.rados.0.smithi090.stdout:718: done (0 left) 2023-01-15T14:31:13.923 INFO:tasks.rados.rados.0.smithi090.stdout:719: read oid 18 snap -1 2023-01-15T14:31:13.924 INFO:tasks.rados.rados.0.smithi090.stdout:719: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:13.924 INFO:tasks.rados.rados.0.smithi090.stdout:720: write oid 16 current snap is 74 2023-01-15T14:31:13.924 INFO:tasks.rados.rados.0.smithi090.stdout:720: seq_num 281 ranges {722767=533677,1793793=461082} 2023-01-15T14:31:13.924 INFO:tasks.rados.rados.0.smithi090.stdout:720: writing smithi09025264-16 from 722767 to 1256444 tid 1 2023-01-15T14:31:13.924 INFO:tasks.rados.rados.0.smithi090.stdout:720: writing smithi09025264-16 from 1793793 to 2254875 tid 2 2023-01-15T14:31:13.925 INFO:tasks.rados.rados.0.smithi090.stdout:721: write oid 23 current snap is 74 2023-01-15T14:31:13.925 INFO:tasks.rados.rados.0.smithi090.stdout:721: seq_num 282 ranges {739574=725279,1968628=721093} 2023-01-15T14:31:13.925 INFO:tasks.rados.rados.0.smithi090.stdout:721: writing smithi09025264-23 from 739574 to 1464853 tid 1 2023-01-15T14:31:13.925 INFO:tasks.rados.rados.0.smithi090.stdout:721: writing smithi09025264-23 from 1968628 to 2689721 tid 2 2023-01-15T14:31:13.925 INFO:tasks.rados.rados.0.smithi090.stdout:722: delete oid 19 current snap is 74 2023-01-15T14:31:13.926 INFO:tasks.rados.rados.0.smithi090.stdout:719: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:13.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:31:13.929 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:13.923+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:14.028 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:31:14.084 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:31:14.084 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:31:14.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:31:14.086 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:14.023+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:14.088 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:14.088 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:14.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.088 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1898 3.12 3:495d4bbc:::smithi09025264-16:head [write 722767~533677 in=533677b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2023-01-15T14:31:14.088 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:13.863908+0000", 2023-01-15T14:31:14.089 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.19535400999999999, 2023-01-15T14:31:14.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19537728800000001, 2023-01-15T14:31:14.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:14.089 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:31:14.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:14.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:14.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:14.090 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1898 2023-01-15T14:31:14.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.090 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:14.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:14.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.863908+0000", 2023-01-15T14:31:14.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:14.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.863908+0000", 2023-01-15T14:31:14.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:14.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.863905+0000", 2023-01-15T14:31:14.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:14.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:14.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.864105+0000", 2023-01-15T14:31:14.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020021700000000001 2023-01-15T14:31:14.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:14.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.864135+0000", 2023-01-15T14:31:14.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9614000000000001e-05 2023-01-15T14:31:14.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:14.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.864148+0000", 2023-01-15T14:31:14.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3371e-05 2023-01-15T14:31:14.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:14.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.964337+0000", 2023-01-15T14:31:14.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100188074 2023-01-15T14:31:14.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:14.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.964474+0000", 2023-01-15T14:31:14.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013736299999999999 2023-01-15T14:31:14.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:31:14.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.964700+0000", 2023-01-15T14:31:14.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022608400000000001 2023-01-15T14:31:14.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:14.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.965181+0000", 2023-01-15T14:31:14.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048140699999999999 2023-01-15T14:31:14.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:14.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.965195+0000", 2023-01-15T14:31:14.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3521e-05 2023-01-15T14:31:14.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:14.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.967496+0000", 2023-01-15T14:31:14.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023014509999999999 2023-01-15T14:31:14.099 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.099 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:14.099 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.099 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1899 3.12 3:495d4bbc:::smithi09025264-16:head [write 1793793~461082 in=461082b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2023-01-15T14:31:14.100 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:13.869736+0000", 2023-01-15T14:31:14.100 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.18952596999999999, 2023-01-15T14:31:14.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189606045, 2023-01-15T14:31:14.100 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:14.100 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:31:14.100 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:14.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:14.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:14.101 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1899 2023-01-15T14:31:14.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.101 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:14.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:14.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.869736+0000", 2023-01-15T14:31:14.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:14.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.869736+0000", 2023-01-15T14:31:14.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:14.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.869733+0000", 2023-01-15T14:31:14.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:31:14.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:14.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870098+0000", 2023-01-15T14:31:14.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036468299999999998 2023-01-15T14:31:14.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:14.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870132+0000", 2023-01-15T14:31:14.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4431000000000003e-05 2023-01-15T14:31:14.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:14.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870149+0000", 2023-01-15T14:31:14.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6674e-05 2023-01-15T14:31:14.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:14.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.965308+0000", 2023-01-15T14:31:14.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095159429000000004 2023-01-15T14:31:14.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:14.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.965431+0000", 2023-01-15T14:31:14.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012306200000000001 2023-01-15T14:31:14.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:31:14.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.965602+0000", 2023-01-15T14:31:14.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017129800000000001 2023-01-15T14:31:14.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:14.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.965916+0000", 2023-01-15T14:31:14.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000313956 2023-01-15T14:31:14.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:14.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.967502+0000", 2023-01-15T14:31:14.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015854409999999999 2023-01-15T14:31:14.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.967727+0000", 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022498900000000001 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1900 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=64b,truncate 2722767] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:13.870185+0000", 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.189076621, 2023-01-15T14:31:14.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189207761, 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1900 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870185+0000", 2023-01-15T14:31:14.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870185+0000", 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870184+0000", 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:14.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870194+0000", 2023-01-15T14:31:14.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0233e-05 2023-01-15T14:31:14.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:14.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870211+0000", 2023-01-15T14:31:14.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6560000000000001e-05 2023-01-15T14:31:14.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870217+0000", 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8499999999999999e-06 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.966013+0000", 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095796531000000004 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.966147+0000", 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013324100000000001 2023-01-15T14:31:14.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.966318+0000", 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171361 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.966595+0000", 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027689099999999998 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:14.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.967503+0000", 2023-01-15T14:31:14.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090768900000000004 2023-01-15T14:31:14.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:14.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.967727+0000", 2023-01-15T14:31:14.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000224785 2023-01-15T14:31:14.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.116 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:14.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1901 3.12 3.3dd2ba92 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e187)", 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:13.870224+0000", 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.189037651, 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18921739500000001, 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:14.117 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1901 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870224+0000", 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870224+0000", 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870224+0000", 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870229+0000", 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4539999999999998e-06 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:14.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870239+0000", 2023-01-15T14:31:14.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9990000000000003e-06 2023-01-15T14:31:14.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:14.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.870243+0000", 2023-01-15T14:31:14.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1160000000000001e-06 2023-01-15T14:31:14.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.120 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:14.120 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.4530.0:1898 3.12 e187/175)", 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:14.051302+0000", 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0079596949999999993, 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.008166731, 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051302+0000", 2023-01-15T14:31:14.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051302+0000", 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051301+0000", 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:14.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051319+0000", 2023-01-15T14:31:14.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7966999999999999e-05 2023-01-15T14:31:14.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:14.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051337+0000", 2023-01-15T14:31:14.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8400999999999999e-05 2023-01-15T14:31:14.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:14.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051348+0000", 2023-01-15T14:31:14.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0526000000000001e-05 2023-01-15T14:31:14.124 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.124 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:14.124 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.124 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.4530.0:1899 3.12 e187/175)", 2023-01-15T14:31:14.124 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:14.051560+0000", 2023-01-15T14:31:14.124 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0077023739999999997, 2023-01-15T14:31:14.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079326770000000008, 2023-01-15T14:31:14.125 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:14.125 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:14.125 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:14.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:14.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051560+0000", 2023-01-15T14:31:14.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:14.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051560+0000", 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051559+0000", 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051563+0000", 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7330000000000001e-06 2023-01-15T14:31:14.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051571+0000", 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.551e-06 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051575+0000", 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9199999999999997e-06 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.128 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.4530.0:1900 3.12 e187/175)", 2023-01-15T14:31:14.128 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:14.051590+0000", 2023-01-15T14:31:14.128 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.007671654, 2023-01-15T14:31:14.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0079249990000000003, 2023-01-15T14:31:14.128 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:14.128 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:14.128 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:14.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:14.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051590+0000", 2023-01-15T14:31:14.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:14.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051590+0000", 2023-01-15T14:31:14.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:14.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051590+0000", 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967296 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051592+0000", 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.137e-06 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051596+0000", 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3390000000000001e-06 2023-01-15T14:31:14.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:14.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.051598+0000", 2023-01-15T14:31:14.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.159e-06 2023-01-15T14:31:14.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.131 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:14.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.131 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.131 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:31:14.131 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 7 2023-01-15T14:31:14.131 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:14.132 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:31:14.133 INFO:tasks.rados.rados.0.smithi090.stdout:721: finishing write tid 1 to smithi09025264-23 2023-01-15T14:31:14.133 INFO:tasks.rados.rados.0.smithi090.stdout:719: done (3 left) 2023-01-15T14:31:14.133 INFO:tasks.rados.rados.0.smithi090.stdout:722: done (2 left) 2023-01-15T14:31:14.133 INFO:tasks.rados.rados.0.smithi090.stdout:723: read oid 18 snap -1 2023-01-15T14:31:14.134 INFO:tasks.rados.rados.0.smithi090.stdout:723: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:14.134 INFO:tasks.rados.rados.0.smithi090.stdout:724: snap_remove snap 73 2023-01-15T14:31:14.135 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:14.130+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:14.236 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:14.231+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:14.254 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:14.254 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:14.254 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:14.254 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:14.254 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:14.263 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:31:14.337 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:14.334+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:14.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:31:14.436 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.436 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:166 3.8 3:1e54a7f5:::smithi09025264-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.25280671799999, 2023-01-15T14:31:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62664632300000001, 2023-01-15T14:31:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 166 2023-01-15T14:31:14.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114713+0000", 2023-01-15T14:31:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115178+0000", 2023-01-15T14:31:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046493199999999999 2023-01-15T14:31:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115208+0000", 2023-01-15T14:31:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0689999999999999e-05 2023-01-15T14:31:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115221+0000", 2023-01-15T14:31:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3013000000000001e-05 2023-01-15T14:31:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:14.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115334+0000", 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112403 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115505+0000", 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170915 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333496+0000", 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217991667 2023-01-15T14:31:14.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333606+0000", 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109967 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333796+0000", 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018928099999999999 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334289+0000", 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049322600000000004 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334339+0000", 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0439999999999998e-05 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334990+0000", 2023-01-15T14:31:14.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065082300000000003 2023-01-15T14:31:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335100+0000", 2023-01-15T14:31:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109357 2023-01-15T14:31:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.640854+0000", 2023-01-15T14:31:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30575451300000001 2023-01-15T14:31:14.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741190+0000", 2023-01-15T14:31:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100335568 2023-01-15T14:31:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741263+0000", 2023-01-15T14:31:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3275999999999994e-05 2023-01-15T14:31:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741361+0000", 2023-01-15T14:31:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8435000000000002e-05 2023-01-15T14:31:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:167 3.8 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:14.446 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.252242029, 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62638390099999997, 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 167 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:14.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115279+0000", 2023-01-15T14:31:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:14.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115289+0000", 2023-01-15T14:31:15.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0509999999999999e-05 2023-01-15T14:31:15.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115306+0000", 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7260000000000001e-05 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115337+0000", 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0664000000000003e-05 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115579+0000", 2023-01-15T14:31:15.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024194400000000001 2023-01-15T14:31:15.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115744+0000", 2023-01-15T14:31:15.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016493700000000001 2023-01-15T14:31:15.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334392+0000", 2023-01-15T14:31:15.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21864830299999999 2023-01-15T14:31:15.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334453+0000", 2023-01-15T14:31:15.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1047e-05 2023-01-15T14:31:15.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:15.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334601+0000", 2023-01-15T14:31:15.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014824700000000001 2023-01-15T14:31:15.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:15.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334812+0000", 2023-01-15T14:31:15.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210208 2023-01-15T14:31:15.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:15.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334996+0000", 2023-01-15T14:31:15.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183878 2023-01-15T14:31:15.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:15.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335126+0000", 2023-01-15T14:31:15.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013001399999999999 2023-01-15T14:31:15.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:15.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335160+0000", 2023-01-15T14:31:15.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4876999999999998e-05 2023-01-15T14:31:15.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741044+0000", 2023-01-15T14:31:15.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40588326000000002 2023-01-15T14:31:15.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741453+0000", 2023-01-15T14:31:15.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040963099999999998 2023-01-15T14:31:15.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:15.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741548+0000", 2023-01-15T14:31:15.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4961000000000001e-05 2023-01-15T14:31:15.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741664+0000", 2023-01-15T14:31:15.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115333 2023-01-15T14:31:15.267 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.267 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.267 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.268 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:15.268 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:15.268 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.25214991499999, 2023-01-15T14:31:15.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:31:15.268 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.268 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:15.268 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.268 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.268 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.269 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:31:15.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.269 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:15.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:15.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:31:15.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:15.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:31:15.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:31:15.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:31:15.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:31:15.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:31:15.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:31:15.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:31:15.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:31:15.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:31:15.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:31:15.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:31:15.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:31:15.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:31:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:31:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:31:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:31:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.275 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.239157836, 2023-01-15T14:31:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:31:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:31:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.276 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:31:15.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:15.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:31:15.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:31:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:31:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:31:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:31:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:31:15.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:31:15.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:31:15.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:31:15.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:31:15.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:31:15.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:31:15.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:15.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:31:15.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:31:15.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:15.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:31:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:31:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:31:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:31:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:31:15.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:31:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:31:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:31:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:31:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:31:15.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:15.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:31:15.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:31:15.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:31:15.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:31:15.456 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.457 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.457 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:15.457 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:15.457 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.2387545, 2023-01-15T14:31:15.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:31:15.457 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.457 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:31:15.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:31:15.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:31:15.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:31:15.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:31:15.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:31:15.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:31:15.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:31:15.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:31:15.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:31:15.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:31:15.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:15.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:31:15.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:31:15.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:15.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:31:15.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:31:15.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:15.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:31:15.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:31:15.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:15.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:31:15.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:31:15.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:31:15.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:31:15.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:31:15.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:31:15.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:15.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:31:15.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:31:15.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:31:15.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:31:15.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.490 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.490 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.13853087999999, 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.491 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:31:15.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.492 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:15.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:15.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:31:15.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:15.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:31:15.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:31:15.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:31:15.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:31:15.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:31:15.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:31:15.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:31:15.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:31:15.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:31:15.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:31:15.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:31:15.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:31:15.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:31:15.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:31:15.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:31:15.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:31:15.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:31:15.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:31:15.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:31:15.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:31:15.584 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.585 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.585 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:194 3.8 3:1eaad226:::smithi09025264-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:15.585 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:15.585 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.06612963699999, 2023-01-15T14:31:15.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64372577200000003, 2023-01-15T14:31:15.585 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.585 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:15.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.586 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 194 2023-01-15T14:31:15.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.586 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:15.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:15.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301390+0000", 2023-01-15T14:31:15.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:15.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301991+0000", 2023-01-15T14:31:15.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060122599999999995 2023-01-15T14:31:15.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.302026+0000", 2023-01-15T14:31:15.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5040000000000003e-05 2023-01-15T14:31:15.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402147+0000", 2023-01-15T14:31:15.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100121084 2023-01-15T14:31:15.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402219+0000", 2023-01-15T14:31:15.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2045999999999994e-05 2023-01-15T14:31:15.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402315+0000", 2023-01-15T14:31:15.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5561000000000002e-05 2023-01-15T14:31:15.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842189+0000", 2023-01-15T14:31:15.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.439874129 2023-01-15T14:31:15.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842296+0000", 2023-01-15T14:31:15.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107021 2023-01-15T14:31:15.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:15.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842486+0000", 2023-01-15T14:31:15.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018965799999999999 2023-01-15T14:31:15.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:15.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842839+0000", 2023-01-15T14:31:15.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035330700000000002 2023-01-15T14:31:15.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:15.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842859+0000", 2023-01-15T14:31:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0347e-05 2023-01-15T14:31:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843212+0000", 2023-01-15T14:31:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352534 2023-01-15T14:31:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:15.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843835+0000", 2023-01-15T14:31:15.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062338599999999997 2023-01-15T14:31:15.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944861+0000", 2023-01-15T14:31:15.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10102596899999999 2023-01-15T14:31:15.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944908+0000", 2023-01-15T14:31:15.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6773000000000001e-05 2023-01-15T14:31:15.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:15.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945039+0000", 2023-01-15T14:31:15.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130684 2023-01-15T14:31:15.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945118+0000", 2023-01-15T14:31:15.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9194000000000001e-05 2023-01-15T14:31:15.605 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.606 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.606 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.606 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:705 3.2 3:47dfe935:::smithi09025264-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2023-01-15T14:31:15.606 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:15.606 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.256937729, 2023-01-15T14:31:15.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61170208599999998, 2023-01-15T14:31:15.606 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.607 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:15.607 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.607 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.607 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.607 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 705 2023-01-15T14:31:15.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.607 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:15.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:15.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110583+0000", 2023-01-15T14:31:15.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:15.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110587+0000", 2023-01-15T14:31:15.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9689999999999996e-06 2023-01-15T14:31:15.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110601+0000", 2023-01-15T14:31:15.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.393e-05 2023-01-15T14:31:15.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110611+0000", 2023-01-15T14:31:15.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3999999999999998e-06 2023-01-15T14:31:15.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319174+0000", 2023-01-15T14:31:15.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20856376800000001 2023-01-15T14:31:15.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319367+0000", 2023-01-15T14:31:15.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192361 2023-01-15T14:31:15.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.721941+0000", 2023-01-15T14:31:15.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40257461300000003 2023-01-15T14:31:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722108+0000", 2023-01-15T14:31:15.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167054 2023-01-15T14:31:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722286+0000", 2023-01-15T14:31:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017772099999999999 2023-01-15T14:31:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.445215654999998, 2023-01-15T14:31:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:31:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.729 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:31:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:15.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:31:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:31:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:31:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:31:15.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:31:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:31:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:31:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:31:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:31:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:15.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:31:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:31:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:31:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:31:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:31:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:31:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:15.737 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.438754301000003, 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:31:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:15.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:31:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:31:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:31:15.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:31:15.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:31:15.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:31:15.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:31:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:31:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:31:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:31:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:31:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:31:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:31:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:31:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.745 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.338332375999997, 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:31:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:31:15.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:31:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:31:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:31:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:31:15.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:31:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:31:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:31:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:31:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.337994890000004, 2023-01-15T14:31:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:31:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.752 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:31:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:31:15.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:31:15.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:31:15.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:31:15.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:31:15.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:31:15.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:31:15.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:31:15.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:15.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:31:15.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:31:15.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:31:15.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:31:15.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:31:15.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:31:15.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.758 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.758 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:15.759 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:15.759 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.337938993999998, 2023-01-15T14:31:15.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:31:15.759 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.760 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:15.760 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.760 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.760 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.760 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:31:15.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.760 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:15.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:15.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:31:15.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:15.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:31:15.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:31:15.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:31:15.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:31:15.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:31:15.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:31:15.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:31:15.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:31:15.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:31:15.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:31:15.765 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.765 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.765 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.765 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1314 3.18 3.afe52a78 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:15.766 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:15.766 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.007643862999998, 2023-01-15T14:31:15.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62560938700000002, 2023-01-15T14:31:15.766 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.766 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:15.766 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.766 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.766 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.766 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1314 2023-01-15T14:31:15.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.766 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359877+0000", 2023-01-15T14:31:15.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:15.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359882+0000", 2023-01-15T14:31:15.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0050000000000004e-06 2023-01-15T14:31:15.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359893+0000", 2023-01-15T14:31:15.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0676000000000001e-05 2023-01-15T14:31:15.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359904+0000", 2023-01-15T14:31:15.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0788000000000001e-05 2023-01-15T14:31:15.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460094+0000", 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10019024999999999 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460106+0000", 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354999999999999e-05 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985477+0000", 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52537039100000005 2023-01-15T14:31:15.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985487+0000", 2023-01-15T14:31:15.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:31:15.771 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.771 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.771 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.771 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1335 3.8 3:17d5d741:::smithi09025264-5:head [write 3252761~175867] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:15.771 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.435624602000004, 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66052243499999996, 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1335 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:15.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:15.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931896+0000", 2023-01-15T14:31:15.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:15.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931983+0000", 2023-01-15T14:31:15.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6470000000000004e-05 2023-01-15T14:31:15.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931995+0000", 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2075e-05 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932003+0000", 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5059999999999995e-06 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932346+0000", 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034280299999999998 2023-01-15T14:31:15.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:15.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932357+0000", 2023-01-15T14:31:15.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.13e-05 2023-01-15T14:31:15.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189006+0000", 2023-01-15T14:31:15.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25664859899999998 2023-01-15T14:31:15.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189336+0000", 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033030899999999998 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189481+0000", 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014484400000000001 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:15.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189768+0000", 2023-01-15T14:31:15.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028704300000000001 2023-01-15T14:31:15.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:15.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189787+0000", 2023-01-15T14:31:15.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9343000000000001e-05 2023-01-15T14:31:15.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:15.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191361+0000", 2023-01-15T14:31:15.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015734029999999999 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191469+0000", 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010859899999999999 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391274+0000", 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19980450899999999 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592067+0000", 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20079332 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592190+0000", 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012291400000000001 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592420+0000", 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000229371 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1336 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (56) in=7b,truncate 3428628] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.435507513000005, 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66113910799999998, 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.781 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1336 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932013+0000", 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932019+0000", 2023-01-15T14:31:15.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5380000000000002e-06 2023-01-15T14:31:15.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932027+0000", 2023-01-15T14:31:15.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5150000000000006e-06 2023-01-15T14:31:15.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932034+0000", 2023-01-15T14:31:15.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2023-01-15T14:31:15.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032497+0000", 2023-01-15T14:31:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100463001 2023-01-15T14:31:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032518+0000", 2023-01-15T14:31:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1359999999999999e-05 2023-01-15T14:31:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189826+0000", 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15730834299999999 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189951+0000", 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124583 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:15.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190093+0000", 2023-01-15T14:31:15.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014221399999999999 2023-01-15T14:31:15.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190253+0000", 2023-01-15T14:31:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015953000000000001 2023-01-15T14:31:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191365+0000", 2023-01-15T14:31:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011123190000000001 2023-01-15T14:31:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191439+0000", 2023-01-15T14:31:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3648999999999999e-05 2023-01-15T14:31:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191491+0000", 2023-01-15T14:31:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2444999999999997e-05 2023-01-15T14:31:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391472+0000", 2023-01-15T14:31:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199981193 2023-01-15T14:31:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592619+0000", 2023-01-15T14:31:15.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20114642499999999 2023-01-15T14:31:15.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:15.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592748+0000", 2023-01-15T14:31:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012950200000000001 2023-01-15T14:31:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593153+0000", 2023-01-15T14:31:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040503700000000002 2023-01-15T14:31:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1337 3.8 3:17d5d741:::smithi09025264-5:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.435479283999996, 2023-01-15T14:31:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66179168200000005, 2023-01-15T14:31:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1337 2023-01-15T14:31:15.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:15.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:15.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:15.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932046+0000", 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8089999999999999e-06 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932055+0000", 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2199999999999998e-06 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932061+0000", 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2779999999999999e-06 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190318+0000", 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.258256967 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190413+0000", 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4627000000000001e-05 2023-01-15T14:31:15.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593288+0000", 2023-01-15T14:31:15.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.402875339 2023-01-15T14:31:15.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593515+0000", 2023-01-15T14:31:15.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022700699999999999 2023-01-15T14:31:15.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593834+0000", 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031906799999999999 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1690 3.16 3:68fb3263:::smithi09025264-33:head [write 1763227~472137] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.505006313999999, 2023-01-15T14:31:15.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72570748100000004, 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1690 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:15.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862514+0000", 2023-01-15T14:31:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:15.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862746+0000", 2023-01-15T14:31:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000232301 2023-01-15T14:31:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862770+0000", 2023-01-15T14:31:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3286e-05 2023-01-15T14:31:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862779+0000", 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4199999999999996e-06 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862844+0000", 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5036999999999994e-05 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862860+0000", 2023-01-15T14:31:15.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6426000000000002e-05 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051251+0000", 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18839023299999999 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051466+0000", 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021523199999999999 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368326+0000", 2023-01-15T14:31:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31686002699999999 2023-01-15T14:31:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368559+0000", 2023-01-15T14:31:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023305400000000001 2023-01-15T14:31:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368935+0000", 2023-01-15T14:31:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037579099999999999 2023-01-15T14:31:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369531+0000", 2023-01-15T14:31:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059593199999999997 2023-01-15T14:31:15.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369571+0000", 2023-01-15T14:31:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0649999999999999e-05 2023-01-15T14:31:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370331+0000", 2023-01-15T14:31:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075969899999999996 2023-01-15T14:31:15.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:15.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371230+0000", 2023-01-15T14:31:15.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000899134 2023-01-15T14:31:15.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472895+0000", 2023-01-15T14:31:15.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101665191 2023-01-15T14:31:15.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.587914+0000", 2023-01-15T14:31:15.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115018671 2023-01-15T14:31:15.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:15.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588013+0000", 2023-01-15T14:31:15.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8494999999999998e-05 2023-01-15T14:31:15.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588223+0000", 2023-01-15T14:31:15.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210294 2023-01-15T14:31:15.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.926 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.926 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1691 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 2235364] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:15.926 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.504722059999999, 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72593841699999995, 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1691 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:15.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:15.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862799+0000", 2023-01-15T14:31:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862807+0000", 2023-01-15T14:31:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1280000000000008e-06 2023-01-15T14:31:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862821+0000", 2023-01-15T14:31:15.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.345e-05 2023-01-15T14:31:15.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862826+0000", 2023-01-15T14:31:15.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3720000000000004e-06 2023-01-15T14:31:15.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862883+0000", 2023-01-15T14:31:15.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6589000000000003e-05 2023-01-15T14:31:15.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:15.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862888+0000", 2023-01-15T14:31:15.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4060000000000004e-06 2023-01-15T14:31:15.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151601+0000", 2023-01-15T14:31:15.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28871354100000002 2023-01-15T14:31:15.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151732+0000", 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013032300000000001 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369729+0000", 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217997515 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369949+0000", 2023-01-15T14:31:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219664 2023-01-15T14:31:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370367+0000", 2023-01-15T14:31:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041822799999999998 2023-01-15T14:31:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370771+0000", 2023-01-15T14:31:15.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040358400000000002 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370794+0000", 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-05 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370880+0000", 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6020999999999995e-05 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371241+0000", 2023-01-15T14:31:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000360956 2023-01-15T14:31:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472994+0000", 2023-01-15T14:31:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101753089 2023-01-15T14:31:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588348+0000", 2023-01-15T14:31:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11535382299999999 2023-01-15T14:31:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588539+0000", 2023-01-15T14:31:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019090399999999999 2023-01-15T14:31:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588738+0000", 2023-01-15T14:31:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019910800000000001 2023-01-15T14:31:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.504688576, 2023-01-15T14:31:15.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:15.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:15.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:31:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:31:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:31:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:31:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:15.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:31:15.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:31:15.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:31:15.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:31:15.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:31:15.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:31:15.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:31:15.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:31:15.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:15.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:31:15.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:31:15.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:15.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:31:15.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:31:15.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:15.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:31:15.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:31:15.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:15.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:15.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:15.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:31:15.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:31:15.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.946 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:15.946 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:15.946 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:15.947 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:31:15.949 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:31:15.951 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:31:15.951 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2023-01-15T14:31:13.848535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":136869993,"num_objects":64,"num_object_clones":21,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":5,"num_read":768,"num_read_kb":640010,"num_write":719,"num_write_kb":240244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":604,"num_bytes_recovered":392054164,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5031,"ondisk_log_size":5031,"up":135,"acting":135,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":5326580,"kb_used_data":556893672,"kb_used_omap":22213,"kb_used_meta":0,"kb_avail":556869700,"statfs":{"total":575688990720,"available":570234572800,"internally_reserved":24547328,"allocated":570259120128,"data_stored":570259120128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22747093,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":334,"commit_latency_ns":0,"apply_latency_ns":334000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2932833,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-24,"num_objects_misplaced":-5,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":-13,"num_read_kb":-8974,"num_write":-16,"num_write_kb":-5248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":11545653,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030846"},"pg_stats":[{"pgid":"3.23","version":"149'233","reported_seq":738,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-01-15T14:31:08.813624+0000","last_change":"2023-01-15T14:31:07.155448+0000","last_active":"2023-01-15T14:31:08.813624+0000","last_peered":"2023-01-15T14:31:08.813624+0000","last_clean":"2023-01-15T14:31:08.813624+0000","last_became_active":"2023-01-15T14:31:06.029716+0000","last_became_peered":"2023-01-15T14:31:06.029716+0000","last_unstale":"2023-01-15T14:31:08.813624+0000","last_undegraded":"2023-01-15T14:31:08.813624+0000","last_fullsized":"2023-01-15T14:31:08.813624+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.37220384099999998,"stat_sum":{"num_bytes":7604272,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":47,"num_read_kb":43637,"num_write":37,"num_write_kb":13160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2469494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.22","version":"181'107","reported_seq":797,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.095304+0000","last_change":"2023-01-15T14:31:09.095304+0000","last_active":"2023-01-15T14:31:09.095304+0000","last_peered":"2023-01-15T14:31:09.095304+0000","last_clean":"2023-01-15T14:31:09.095304+0000","last_became_active":"2023-01-15T14:31:08.084483+0000","last_became_peered":"2023-01-15T14:31:08.084483+0000","last_unstale":"2023-01-15T14:31:09.095304+0000","last_undegraded":"2023-01-15T14:31:09.095304+0000","last_fullsized":"2023-01-15T14:31:09.095304+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.28145420599999998,"stat_sum":{"num_bytes":2518991,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":12156,"num_write":8,"num_write_kb":2164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":11208309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.21","version":"168'124","reported_seq":658,"reported_epoch":184,"state":"active+clean+scrubbing","last_fresh":"2023-01-15T14:31:11.310892+0000","last_change":"2023-01-15T14:31:11.209353+0000","last_active":"2023-01-15T14:31:11.310892+0000","last_peered":"2023-01-15T14:31:11.310892+0000","last_clean":"2023-01-15T14:31:11.310892+0000","last_became_active":"2023-01-15T14:30:52.226939+0000","last_became_peered":"2023-01-15T14:30:52.226939+0000","last_unstale":"2023-01-15T14:31:11.310892+0000","last_undegraded":"2023-01-15T14:31:11.310892+0000","last_fullsized":"2023-01-15T14:31:11.310892+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'111","last_scrub_stamp":"2023-01-15T14:30:23.145968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.145968+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":0.28528039199999999,"objects_trimmed":0,"snaptrim_duration":0.101276437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3798382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.20","version":"181'236","reported_seq":898,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.843299+0000","last_change":"2023-01-15T14:31:08.613365+0000","last_active":"2023-01-15T14:31:09.843299+0000","last_peered":"2023-01-15T14:31:09.843299+0000","last_clean":"2023-01-15T14:31:09.843299+0000","last_became_active":"2023-01-15T14:31:07.906191+0000","last_became_peered":"2023-01-15T14:31:07.906191+0000","last_unstale":"2023-01-15T14:31:09.843299+0000","last_undegraded":"2023-01-15T14:31:09.843299+0000","last_fullsized":"2023-01-15T14:31:09.843299+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.70514344299999998,"stat_sum":{"num_bytes":4659379,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":7,"num_write_kb":2065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8877936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.1f","version":"168'147","reported_seq":774,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.821962+0000","last_change":"2023-01-15T14:31:08.063615+0000","last_active":"2023-01-15T14:31:09.821962+0000","last_peered":"2023-01-15T14:31:09.821962+0000","last_clean":"2023-01-15T14:31:09.821962+0000","last_became_active":"2023-01-15T14:31:07.960304+0000","last_became_peered":"2023-01-15T14:31:07.960304+0000","last_unstale":"2023-01-15T14:31:09.821962+0000","last_undegraded":"2023-01-15T14:31:09.821962+0000","last_fullsized":"2023-01-15T14:31:09.821962+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020815930000000001,"stat_sum":{"num_bytes":3000066,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":44,"num_read_kb":49152,"num_write":38,"num_write_kb":12831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":36580502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.1e","version":"139'38","reported_seq":502,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-01-15T14:31:08.813793+0000","last_change":"2023-01-15T14:31:06.942272+0000","last_active":"2023-01-15T14:31:08.813793+0000","last_peered":"2023-01-15T14:31:08.813793+0000","last_clean":"2023-01-15T14:31:08.813793+0000","last_became_active":"2023-01-15T14:30:51.134845+0000","last_became_peered":"2023-01-15T14:30:51.134845+0000","last_unstale":"2023-01-15T14:31:08.813793+0000","last_undegraded":"2023-01-15T14:31:08.813793+0000","last_fullsized":"2023-01-15T14:31:08.813793+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'37","last_scrub_stamp":"2023-01-15T14:30:25.793922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:25.793922+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.15943829300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":11530,"num_write":6,"num_write_kb":4046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1487420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.1d","version":"182'161","reported_seq":774,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-01-15T14:31:12.027026+0000","last_change":"2023-01-15T14:31:07.001157+0000","last_active":"2023-01-15T14:31:12.027026+0000","last_peered":"2023-01-15T14:31:12.027026+0000","last_clean":"2023-01-15T14:31:12.027026+0000","last_became_active":"2023-01-15T14:30:54.085528+0000","last_became_peered":"2023-01-15T14:30:54.085528+0000","last_unstale":"2023-01-15T14:31:12.027026+0000","last_undegraded":"2023-01-15T14:31:12.027026+0000","last_fullsized":"2023-01-15T14:31:12.027026+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012662813,"stat_sum":{"num_bytes":9128589,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":71,"num_read_kb":81711,"num_write":44,"num_write_kb":16465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":13857289,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":74,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-01-15T14:31:12.027350+0000","last_change":"2023-01-15T14:31:06.894125+0000","last_active":"2023-01-15T14:31:12.027350+0000","last_peered":"2023-01-15T14:31:12.027350+0000","last_clean":"2023-01-15T14:31:12.027350+0000","last_became_active":"2023-01-15T14:30:51.234588+0000","last_became_peered":"2023-01-15T14:30:51.234588+0000","last_unstale":"2023-01-15T14:31:12.027350+0000","last_undegraded":"2023-01-15T14:31:12.027350+0000","last_fullsized":"2023-01-15T14:31:12.027350+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.006168438,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":10,"num_write_kb":2522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1159667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.1b","version":"155'53","reported_seq":500,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-01-15T14:31:12.026878+0000","last_change":"2023-01-15T14:31:06.895777+0000","last_active":"2023-01-15T14:31:12.026878+0000","last_peered":"2023-01-15T14:31:12.026878+0000","last_clean":"2023-01-15T14:31:12.026878+0000","last_became_active":"2023-01-15T14:31:02.994801+0000","last_became_peered":"2023-01-15T14:31:02.994801+0000","last_unstale":"2023-01-15T14:31:12.026878+0000","last_undegraded":"2023-01-15T14:31:12.026878+0000","last_fullsized":"2023-01-15T14:31:12.026878+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007588287,"stat_sum":{"num_bytes":1731121,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":6694,"num_write":12,"num_write_kb":5635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3462242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.1a","version":"182'188","reported_seq":747,"reported_epoch":184,"state":"peering","last_fresh":"2023-01-15T14:31:12.026820+0000","last_change":"2023-01-15T14:31:12.024100+0000","last_active":"2023-01-15T14:31:12.023291+0000","last_peered":"2023-01-15T14:31:09.778411+0000","last_clean":"2023-01-15T14:31:09.778411+0000","last_became_active":"2023-01-15T14:30:52.132739+0000","last_became_peered":"2023-01-15T14:30:52.132739+0000","last_unstale":"2023-01-15T14:31:12.026820+0000","last_undegraded":"2023-01-15T14:31:12.026820+0000","last_fullsized":"2023-01-15T14:31:12.026820+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":8804416,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":38958,"num_write":58,"num_write_kb":25710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":56061752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"172'171","reported_seq":749,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.842648+0000","last_change":"2023-01-15T14:31:06.789236+0000","last_active":"2023-01-15T14:31:09.842648+0000","last_peered":"2023-01-15T14:31:09.842648+0000","last_clean":"2023-01-15T14:31:09.842648+0000","last_became_active":"2023-01-15T14:30:59.222826+0000","last_became_peered":"2023-01-15T14:30:59.222826+0000","last_unstale":"2023-01-15T14:31:09.842648+0000","last_undegraded":"2023-01-15T14:31:09.842648+0000","last_fullsized":"2023-01-15T14:31:09.842648+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'157","last_scrub_stamp":"2023-01-15T14:30:26.792736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:26.792736+0000","objects_scrubbed":4,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:38.155","scrub_duration":0.61998826900000004,"objects_trimmed":0,"snaptrim_duration":0.0012967129999999999,"stat_sum":{"num_bytes":11193495,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":25536,"num_write":6,"num_write_kb":853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":29479511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.18","version":"182'220","reported_seq":917,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:10.135295+0000","last_change":"2023-01-15T14:31:10.136878+0000","last_active":"2023-01-15T14:31:10.135295+0000","last_peered":"2023-01-15T14:31:10.135295+0000","last_clean":"2023-01-15T14:31:10.135295+0000","last_became_active":"2023-01-15T14:31:04.108363+0000","last_became_peered":"2023-01-15T14:31:04.108363+0000","last_unstale":"2023-01-15T14:31:10.135295+0000","last_undegraded":"2023-01-15T14:31:10.135295+0000","last_fullsized":"2023-01-15T14:31:10.135295+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:19.367","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.110692465,"stat_sum":{"num_bytes":8097496,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":18216,"num_write":14,"num_write_kb":7378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":22066072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.17","version":"155'77","reported_seq":554,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-01-15T14:31:08.814429+0000","last_change":"2023-01-15T14:31:07.050342+0000","last_active":"2023-01-15T14:31:08.814429+0000","last_peered":"2023-01-15T14:31:08.814429+0000","last_clean":"2023-01-15T14:31:08.814429+0000","last_became_active":"2023-01-15T14:31:02.993865+0000","last_became_peered":"2023-01-15T14:31:02.993865+0000","last_unstale":"2023-01-15T14:31:08.814429+0000","last_undegraded":"2023-01-15T14:31:08.814429+0000","last_fullsized":"2023-01-15T14:31:08.814429+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'65","last_scrub_stamp":"2023-01-15T14:30:24.156798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.156798+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:34.293","scrub_duration":0.30368223799999999,"objects_trimmed":0,"snaptrim_duration":0.167115861,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3949645,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.16","version":"180'79","reported_seq":103,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-01-15T14:31:11.311590+0000","last_change":"2023-01-15T14:31:11.311464+0000","last_active":"2023-01-15T14:31:11.311590+0000","last_peered":"2023-01-15T14:31:11.311590+0000","last_clean":"2023-01-15T14:31:11.311590+0000","last_became_active":"2023-01-15T14:30:51.036725+0000","last_became_peered":"2023-01-15T14:30:51.036725+0000","last_unstale":"2023-01-15T14:31:11.311590+0000","last_undegraded":"2023-01-15T14:31:11.311590+0000","last_fullsized":"2023-01-15T14:31:11.311590+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.70770021100000002,"stat_sum":{"num_bytes":2235364,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":526,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-01-15T14:31:08.813753+0000","last_change":"2023-01-15T14:31:06.943900+0000","last_active":"2023-01-15T14:31:08.813753+0000","last_peered":"2023-01-15T14:31:08.813753+0000","last_clean":"2023-01-15T14:31:08.813753+0000","last_became_active":"2023-01-15T14:30:03.169969+0000","last_became_peered":"2023-01-15T14:30:03.169969+0000","last_unstale":"2023-01-15T14:31:08.813753+0000","last_undegraded":"2023-01-15T14:31:08.813753+0000","last_fullsized":"2023-01-15T14:31:08.813753+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:30:24.721082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.721082+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:48.096","scrub_duration":0.109000768,"objects_trimmed":0,"snaptrim_duration":0.16055111599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.14","version":"169'138","reported_seq":684,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-01-15T14:31:12.027487+0000","last_change":"2023-01-15T14:31:06.998733+0000","last_active":"2023-01-15T14:31:12.027487+0000","last_peered":"2023-01-15T14:31:12.027487+0000","last_clean":"2023-01-15T14:31:12.027487+0000","last_became_active":"2023-01-15T14:30:16.972329+0000","last_became_peered":"2023-01-15T14:30:16.972329+0000","last_unstale":"2023-01-15T14:31:12.027487+0000","last_undegraded":"2023-01-15T14:31:12.027487+0000","last_fullsized":"2023-01-15T14:31:12.027487+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:31.572","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.11030556800000001,"stat_sum":{"num_bytes":2453855,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":11235,"num_write":5,"num_write_kb":1217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":209,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.822093+0000","last_change":"2023-01-15T14:30:51.228396+0000","last_active":"2023-01-15T14:31:09.822093+0000","last_peered":"2023-01-15T14:31:09.822093+0000","last_clean":"2023-01-15T14:31:09.822093+0000","last_became_active":"2023-01-15T14:30:51.127504+0000","last_became_peered":"2023-01-15T14:30:51.127504+0000","last_unstale":"2023-01-15T14:31:09.822093+0000","last_undegraded":"2023-01-15T14:31:09.822093+0000","last_fullsized":"2023-01-15T14:31:09.822093+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.671761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.671761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":544,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.943361+0000","last_change":"2023-01-15T14:31:06.891931+0000","last_active":"2023-01-15T14:31:09.943361+0000","last_peered":"2023-01-15T14:31:09.943361+0000","last_clean":"2023-01-15T14:31:09.943361+0000","last_became_active":"2023-01-15T14:31:02.895949+0000","last_became_peered":"2023-01-15T14:31:02.895949+0000","last_unstale":"2023-01-15T14:31:09.943361+0000","last_undegraded":"2023-01-15T14:31:09.943361+0000","last_fullsized":"2023-01-15T14:31:09.943361+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:30:21.421951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:21.421951+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:50.589","scrub_duration":0.569620185,"objects_trimmed":0,"snaptrim_duration":0.10035348400000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":31,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:10.043709+0000","last_change":"2023-01-15T14:30:51.035825+0000","last_active":"2023-01-15T14:31:10.043709+0000","last_peered":"2023-01-15T14:31:10.043709+0000","last_clean":"2023-01-15T14:31:10.043709+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:31:10.043709+0000","last_undegraded":"2023-01-15T14:31:10.043709+0000","last_fullsized":"2023-01-15T14:31:10.043709+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"125'75","reported_seq":595,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-01-15T14:31:08.814658+0000","last_change":"2023-01-15T14:31:07.048677+0000","last_active":"2023-01-15T14:31:08.814658+0000","last_peered":"2023-01-15T14:31:08.814658+0000","last_clean":"2023-01-15T14:31:08.814658+0000","last_became_active":"2023-01-15T14:31:02.994339+0000","last_became_peered":"2023-01-15T14:31:02.994339+0000","last_unstale":"2023-01-15T14:31:08.814658+0000","last_undegraded":"2023-01-15T14:31:08.814658+0000","last_fullsized":"2023-01-15T14:31:08.814658+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:20.645","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.16579828999999999,"stat_sum":{"num_bytes":3462242,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":16914,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3462242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":200,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.943604+0000","last_change":"2023-01-15T14:31:08.573925+0000","last_active":"2023-01-15T14:31:09.943604+0000","last_peered":"2023-01-15T14:31:09.943604+0000","last_clean":"2023-01-15T14:31:09.943604+0000","last_became_active":"2023-01-15T14:31:03.088392+0000","last_became_peered":"2023-01-15T14:31:03.088392+0000","last_unstale":"2023-01-15T14:31:09.943604+0000","last_undegraded":"2023-01-15T14:31:09.943604+0000","last_fullsized":"2023-01-15T14:31:09.943604+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:25.664616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:25.664616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:31:13.573","scrub_duration":0.40398350300000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"182'192","reported_seq":829,"reported_epoch":183,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:31:09.843162+0000","last_change":"2023-01-15T14:31:09.222582+0000","last_active":"2023-01-15T14:31:09.843162+0000","last_peered":"2023-01-15T14:31:09.843162+0000","last_clean":"2023-01-15T14:31:09.843162+0000","last_became_active":"2023-01-15T14:31:09.119865+0000","last_became_peered":"2023-01-15T14:31:09.119865+0000","last_unstale":"2023-01-15T14:31:09.843162+0000","last_undegraded":"2023-01-15T14:31:09.843162+0000","last_fullsized":"2023-01-15T14:31:09.843162+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:50.609","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.011167300999999999,"stat_sum":{"num_bytes":7411953,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":21193,"num_write":26,"num_write_kb":8405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":31724214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"80","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.843293+0000","last_change":"2023-01-15T14:30:51.237821+0000","last_active":"2023-01-15T14:31:09.843293+0000","last_peered":"2023-01-15T14:31:09.843293+0000","last_clean":"2023-01-15T14:31:09.843293+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:31:09.843293+0000","last_undegraded":"2023-01-15T14:31:09.843293+0000","last_fullsized":"2023-01-15T14:31:09.843293+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"179'67","reported_seq":604,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-01-15T14:31:08.814005+0000","last_change":"2023-01-15T14:31:07.154062+0000","last_active":"2023-01-15T14:31:08.814005+0000","last_peered":"2023-01-15T14:31:08.814005+0000","last_clean":"2023-01-15T14:31:08.814005+0000","last_became_active":"2023-01-15T14:30:51.138174+0000","last_became_peered":"2023-01-15T14:30:51.138174+0000","last_unstale":"2023-01-15T14:31:08.814005+0000","last_undegraded":"2023-01-15T14:31:08.814005+0000","last_fullsized":"2023-01-15T14:31:08.814005+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:31:12.510","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.37036040100000001,"stat_sum":{"num_bytes":2067294,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":30038,"num_write":19,"num_write_kb":7307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":18983440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.3","version":"180'249","reported_seq":901,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-01-15T14:31:12.027291+0000","last_change":"2023-01-15T14:31:07.703029+0000","last_active":"2023-01-15T14:31:12.027291+0000","last_peered":"2023-01-15T14:31:12.027291+0000","last_clean":"2023-01-15T14:31:12.027291+0000","last_became_active":"2023-01-15T14:31:03.093753+0000","last_became_peered":"2023-01-15T14:31:03.093753+0000","last_unstale":"2023-01-15T14:31:12.027291+0000","last_undegraded":"2023-01-15T14:31:12.027291+0000","last_fullsized":"2023-01-15T14:31:12.027291+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81405111399999996,"stat_sum":{"num_bytes":1294205,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":43638,"num_write":40,"num_write_kb":14160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":9914132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.976507+0000","last_change":"2023-01-15T14:30:51.337447+0000","last_active":"2023-01-15T14:31:09.976507+0000","last_peered":"2023-01-15T14:31:09.976507+0000","last_clean":"2023-01-15T14:31:09.976507+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:31:09.976507+0000","last_undegraded":"2023-01-15T14:31:09.976507+0000","last_fullsized":"2023-01-15T14:31:09.976507+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:14.703035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:14.703035+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"180'236","reported_seq":833,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.843223+0000","last_change":"2023-01-15T14:31:07.604546+0000","last_active":"2023-01-15T14:31:09.843223+0000","last_peered":"2023-01-15T14:31:09.843223+0000","last_clean":"2023-01-15T14:31:09.843223+0000","last_became_active":"2023-01-15T14:30:56.101261+0000","last_became_peered":"2023-01-15T14:30:56.101261+0000","last_unstale":"2023-01-15T14:31:09.843223+0000","last_undegraded":"2023-01-15T14:31:09.843223+0000","last_fullsized":"2023-01-15T14:31:09.843223+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"137'228","last_scrub_stamp":"2023-01-15T14:30:46.603760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:46.603760+0000","objects_scrubbed":1,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.61616140500000005,"stat_sum":{"num_bytes":3479049,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3109295,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":185,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.820589+0000","last_change":"2023-01-15T14:29:47.854985+0000","last_active":"2023-01-15T14:31:09.820589+0000","last_peered":"2023-01-15T14:31:09.820589+0000","last_clean":"2023-01-15T14:31:09.820589+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:31:09.820589+0000","last_undegraded":"2023-01-15T14:31:09.820589+0000","last_fullsized":"2023-01-15T14:31:09.820589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:47.854866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:47.854866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.63763962500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":605,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.823007+0000","last_change":"2023-01-15T14:31:06.891901+0000","last_active":"2023-01-15T14:31:09.823007+0000","last_peered":"2023-01-15T14:31:09.823007+0000","last_clean":"2023-01-15T14:31:09.823007+0000","last_became_active":"2023-01-15T14:31:06.028849+0000","last_became_peered":"2023-01-15T14:31:06.028849+0000","last_unstale":"2023-01-15T14:31:09.823007+0000","last_undegraded":"2023-01-15T14:31:09.823007+0000","last_fullsized":"2023-01-15T14:31:09.823007+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:30:47.349633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.349633+0000","objects_scrubbed":1,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.28528039199999999,"objects_trimmed":0,"snaptrim_duration":0.0041291469999999997,"stat_sum":{"num_bytes":1899191,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":219,"reported_epoch":184,"state":"active+clean","last_fresh":"2023-01-15T14:31:11.136740+0000","last_change":"2023-01-15T14:31:11.136609+0000","last_active":"2023-01-15T14:31:11.136740+0000","last_peered":"2023-01-15T14:31:11.136740+0000","last_clean":"2023-01-15T14:31:11.136740+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:31:11.136740+0000","last_undegraded":"2023-01-15T14:31:11.136740+0000","last_fullsized":"2023-01-15T14:31:11.136740+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:11.136490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.136490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.39969354600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":700,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-01-15T14:31:08.914439+0000","last_change":"2023-01-15T14:31:07.045039+0000","last_active":"2023-01-15T14:31:08.914439+0000","last_peered":"2023-01-15T14:31:08.914439+0000","last_clean":"2023-01-15T14:31:08.914439+0000","last_became_active":"2023-01-15T14:30:54.184242+0000","last_became_peered":"2023-01-15T14:30:54.184242+0000","last_unstale":"2023-01-15T14:31:08.914439+0000","last_undegraded":"2023-01-15T14:31:08.914439+0000","last_fullsized":"2023-01-15T14:31:08.914439+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.25962200699999999,"stat_sum":{"num_bytes":1084068,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3268978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":188,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.822184+0000","last_change":"2023-01-15T14:31:02.793835+0000","last_active":"2023-01-15T14:31:09.822184+0000","last_peered":"2023-01-15T14:31:09.822184+0000","last_clean":"2023-01-15T14:31:09.822184+0000","last_became_active":"2023-01-15T14:31:02.793531+0000","last_became_peered":"2023-01-15T14:31:02.793531+0000","last_unstale":"2023-01-15T14:31:09.822184+0000","last_undegraded":"2023-01-15T14:31:09.822184+0000","last_fullsized":"2023-01-15T14:31:09.822184+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:44.759745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:44.759745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":319,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.842768+0000","last_change":"2023-01-15T14:31:03.091584+0000","last_active":"2023-01-15T14:31:09.842768+0000","last_peered":"2023-01-15T14:31:09.842768+0000","last_clean":"2023-01-15T14:31:09.842768+0000","last_became_active":"2023-01-15T14:31:03.091071+0000","last_became_peered":"2023-01-15T14:31:03.091071+0000","last_unstale":"2023-01-15T14:31:09.842768+0000","last_undegraded":"2023-01-15T14:31:09.842768+0000","last_fullsized":"2023-01-15T14:31:09.842768+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:29:45.876183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:29:45.876183+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"179'204","reported_seq":845,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-01-15T14:31:08.813171+0000","last_change":"2023-01-15T14:31:07.046967+0000","last_active":"2023-01-15T14:31:08.813171+0000","last_peered":"2023-01-15T14:31:08.813171+0000","last_clean":"2023-01-15T14:31:08.813171+0000","last_became_active":"2023-01-15T14:31:03.288830+0000","last_became_peered":"2023-01-15T14:31:03.288830+0000","last_unstale":"2023-01-15T14:31:08.813171+0000","last_undegraded":"2023-01-15T14:31:08.813171+0000","last_fullsized":"2023-01-15T14:31:08.813171+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:34.102","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.16444709299999999,"stat_sum":{"num_bytes":4319432,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":0,"num_write":3,"num_write_kb":7962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":15748192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":639,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:12.022162+0000","last_change":"2023-01-15T14:31:06.891030+0000","last_active":"2023-01-15T14:31:12.022162+0000","last_peered":"2023-01-15T14:31:12.022162+0000","last_clean":"2023-01-15T14:31:12.022162+0000","last_became_active":"2023-01-15T14:29:58.963354+0000","last_became_peered":"2023-01-15T14:29:58.963354+0000","last_unstale":"2023-01-15T14:31:12.022162+0000","last_undegraded":"2023-01-15T14:31:12.022162+0000","last_fullsized":"2023-01-15T14:31:12.022162+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:30:08.681869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:08.681869+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:19.959","scrub_duration":0.40398525099999999,"objects_trimmed":0,"snaptrim_duration":0.003849983,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.a","version":"174'186","reported_seq":775,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:12.022811+0000","last_change":"2023-01-15T14:31:06.999741+0000","last_active":"2023-01-15T14:31:12.022811+0000","last_peered":"2023-01-15T14:31:12.022811+0000","last_clean":"2023-01-15T14:31:12.022811+0000","last_became_active":"2023-01-15T14:31:01.355624+0000","last_became_peered":"2023-01-15T14:31:01.355624+0000","last_unstale":"2023-01-15T14:31:12.022811+0000","last_undegraded":"2023-01-15T14:31:12.022811+0000","last_fullsized":"2023-01-15T14:31:12.022811+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011980332999999999,"stat_sum":{"num_bytes":4424360,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":23721,"num_write":51,"num_write_kb":24558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":37790905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":531,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-01-15T14:31:08.814218+0000","last_change":"2023-01-15T14:31:07.052045+0000","last_active":"2023-01-15T14:31:08.814218+0000","last_peered":"2023-01-15T14:31:08.814218+0000","last_clean":"2023-01-15T14:31:08.814218+0000","last_became_active":"2023-01-15T14:31:02.994679+0000","last_became_peered":"2023-01-15T14:31:02.994679+0000","last_unstale":"2023-01-15T14:31:08.814218+0000","last_undegraded":"2023-01-15T14:31:08.814218+0000","last_fullsized":"2023-01-15T14:31:08.814218+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.16826560300000001,"stat_sum":{"num_bytes":1731121,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":6694,"num_write":11,"num_write_kb":5635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.c","version":"141'46","reported_seq":545,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.823633+0000","last_change":"2023-01-15T14:31:06.940669+0000","last_active":"2023-01-15T14:31:09.823633+0000","last_peered":"2023-01-15T14:31:09.823633+0000","last_clean":"2023-01-15T14:31:09.823633+0000","last_became_active":"2023-01-15T14:31:02.991074+0000","last_became_peered":"2023-01-15T14:31:02.991074+0000","last_unstale":"2023-01-15T14:31:09.823633+0000","last_undegraded":"2023-01-15T14:31:09.823633+0000","last_fullsized":"2023-01-15T14:31:09.823633+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.15848641299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":10,"num_write_kb":2522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1159666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.d","version":"168'156","reported_seq":748,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.843658+0000","last_change":"2023-01-15T14:31:06.887209+0000","last_active":"2023-01-15T14:31:09.843658+0000","last_peered":"2023-01-15T14:31:09.843658+0000","last_clean":"2023-01-15T14:31:09.843658+0000","last_became_active":"2023-01-15T14:30:44.889199+0000","last_became_peered":"2023-01-15T14:30:44.889199+0000","last_unstale":"2023-01-15T14:31:09.843658+0000","last_undegraded":"2023-01-15T14:31:09.843658+0000","last_fullsized":"2023-01-15T14:31:09.843658+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:01.864","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.0023694190000000002,"stat_sum":{"num_bytes":6067327,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":10837,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":547,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:12.022224+0000","last_change":"2023-01-15T14:31:06.993676+0000","last_active":"2023-01-15T14:31:12.022224+0000","last_peered":"2023-01-15T14:31:12.022224+0000","last_clean":"2023-01-15T14:31:12.022224+0000","last_became_active":"2023-01-15T14:31:03.094506+0000","last_became_peered":"2023-01-15T14:31:03.094506+0000","last_unstale":"2023-01-15T14:31:12.022224+0000","last_undegraded":"2023-01-15T14:31:12.022224+0000","last_fullsized":"2023-01-15T14:31:12.022224+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'37","last_scrub_stamp":"2023-01-15T14:30:25.793922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:25.793922+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.105734274,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":11529,"num_write":5,"num_write_kb":4046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1487419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":756,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:12.022257+0000","last_change":"2023-01-15T14:31:06.997512+0000","last_active":"2023-01-15T14:31:12.022257+0000","last_peered":"2023-01-15T14:31:12.022257+0000","last_clean":"2023-01-15T14:31:12.022257+0000","last_became_active":"2023-01-15T14:31:06.028460+0000","last_became_peered":"2023-01-15T14:31:06.028460+0000","last_unstale":"2023-01-15T14:31:12.022257+0000","last_undegraded":"2023-01-15T14:31:12.022257+0000","last_fullsized":"2023-01-15T14:31:12.022257+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10853452299999999,"stat_sum":{"num_bytes":6227010,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":33587,"num_write":38,"num_write_kb":12831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":14899405,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.10","version":"179'238","reported_seq":838,"reported_epoch":183,"state":"peering","last_fresh":"2023-01-15T14:31:09.924846+0000","last_change":"2023-01-15T14:31:09.925306+0000","last_active":"2023-01-15T14:31:09.924773+0000","last_peered":"2023-01-15T14:31:09.237424+0000","last_clean":"2023-01-15T14:30:11.752178+0000","last_became_active":"2023-01-15T14:31:04.109129+0000","last_became_peered":"2023-01-15T14:31:04.109129+0000","last_unstale":"2023-01-15T14:31:09.924846+0000","last_undegraded":"2023-01-15T14:31:09.924846+0000","last_fullsized":"2023-01-15T14:31:09.924846+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":2,"snaptrim_duration":0.60414505699999999,"stat_sum":{"num_bytes":6487552,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":51893,"num_write":60,"num_write_kb":25114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":26680915,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"80","length":"1"}]},{"pgid":"3.11","version":"180'141","reported_seq":754,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.843679+0000","last_change":"2023-01-15T14:31:07.705076+0000","last_active":"2023-01-15T14:31:09.843679+0000","last_peered":"2023-01-15T14:31:09.843679+0000","last_clean":"2023-01-15T14:31:09.843679+0000","last_became_active":"2023-01-15T14:31:06.888501+0000","last_became_peered":"2023-01-15T14:31:06.888501+0000","last_unstale":"2023-01-15T14:31:09.843679+0000","last_undegraded":"2023-01-15T14:31:09.843679+0000","last_fullsized":"2023-01-15T14:31:09.843679+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":5,"last_scrub":"112'112","last_scrub_stamp":"2023-01-15T14:30:09.593275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.593275+0000","objects_scrubbed":2,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:13.409","scrub_duration":0.28909560899999998,"objects_trimmed":0,"snaptrim_duration":0.81362778700000005,"stat_sum":{"num_bytes":8722866,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":27,"num_read_kb":33961,"num_write":9,"num_write_kb":8388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":14420439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.12","version":"179'132","reported_seq":342,"reported_epoch":182,"state":"active+clean","last_fresh":"2023-01-15T14:31:08.914241+0000","last_change":"2023-01-15T14:31:07.275277+0000","last_active":"2023-01-15T14:31:08.914241+0000","last_peered":"2023-01-15T14:31:08.914241+0000","last_clean":"2023-01-15T14:31:08.914241+0000","last_became_active":"2023-01-15T14:31:03.093394+0000","last_became_peered":"2023-01-15T14:31:03.093394+0000","last_unstale":"2023-01-15T14:31:08.914241+0000","last_undegraded":"2023-01-15T14:31:08.914241+0000","last_fullsized":"2023-01-15T14:31:08.914241+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"139'103","last_scrub_stamp":"2023-01-15T14:30:30.045615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:30.045615+0000","objects_scrubbed":3,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:36.166","scrub_duration":0.207128334,"objects_trimmed":0,"snaptrim_duration":0.49023117199999999,"stat_sum":{"num_bytes":5916064,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":21,"num_read_kb":27344,"num_write":7,"num_write_kb":3462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":12585229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.13","version":"180'260","reported_seq":784,"reported_epoch":183,"state":"active+clean","last_fresh":"2023-01-15T14:31:09.843671+0000","last_change":"2023-01-15T14:31:07.495302+0000","last_active":"2023-01-15T14:31:09.843671+0000","last_peered":"2023-01-15T14:31:09.843671+0000","last_clean":"2023-01-15T14:31:09.843671+0000","last_became_active":"2023-01-15T14:30:08.974612+0000","last_became_peered":"2023-01-15T14:30:08.974612+0000","last_unstale":"2023-01-15T14:31:09.843671+0000","last_undegraded":"2023-01-15T14:31:09.843671+0000","last_fullsized":"2023-01-15T14:31:09.843671+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:29.544","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.61062312900000004,"stat_sum":{"num_bytes":10258847,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":34,"num_read_kb":29731,"num_write":21,"num_write_kb":11650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":36,"stat_sum":{"num_bytes":136279625,"num_objects":62,"num_object_clones":21,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":5,"num_read":652,"num_read_kb":639910,"num_write":560,"num_write_kb":237828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":596,"num_bytes_recovered":389692692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4914,"ondisk_log_size":4914,"up":108,"acting":108,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":841000,"kb_used_data":92858516,"kb_used_omap":2214,"kb_used_meta":0,"kb_avail":92858380,"statfs":{"total":95948165120,"available":95086981120,"internally_reserved":139264,"allocated":95087120384,"data_stored":95087120384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2267340,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":3.4119999999999999,"15min":3.8500000000000001},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.522,"5min":75.893000000000001,"15min":75.893000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":9.1829999999999998,"5min":9.2029999999999994,"15min":9.2070000000000007},"min":{"1min":0.53000000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":73.769000000000005,"5min":73.769000000000005,"15min":73.769000000000005},"last":0.97999999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":6.3079999999999998,"5min":6.4969999999999999,"15min":6.5289999999999999},"min":{"1min":0.56200000000000006,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":88.003,"5min":88.003,"15min":88.003},"last":0.81899999999999995},{"interface":"front","average":{"1min":5.3300000000000001,"5min":6.0899999999999999,"15min":6.2160000000000002},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":73.704999999999998,"5min":73.704999999999998,"15min":73.704999999999998},"last":1.1339999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":13.926,"5min":6.6180000000000003,"15min":5.4000000000000004},"min":{"1min":0.69799999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":74.010999999999996,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.90600000000000003,"5min":3.5230000000000001,"15min":3.9590000000000001},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.8169999999999999,"5min":75.998999999999995,"15min":75.998999999999995},"last":0.88}]},{"osd":3,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":14.441000000000001,"5min":3.6459999999999999,"15min":1.847},"min":{"1min":0.71799999999999997,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":144.65600000000001,"5min":144.65600000000001,"15min":144.65600000000001},"last":15.58},{"interface":"front","average":{"1min":4.0670000000000002,"5min":4.5350000000000001,"15min":4.6130000000000004},"min":{"1min":0.71799999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":45.164000000000001,"5min":76.034999999999997,"15min":76.034999999999997},"last":1.01}]},{"osd":4,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":5.3949999999999996,"5min":8.6110000000000007,"15min":9.1470000000000002},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":73.459999999999994,"5min":76.658000000000001,"15min":76.658000000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":16.405000000000001,"5min":4.0529999999999999,"15min":1.994},"min":{"1min":0.67000000000000004,"5min":0.498,"15min":0.498},"max":{"1min":164.21199999999999,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.94399999999999995}]},{"osd":5,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":5.4480000000000004,"5min":6.1509999999999998,"15min":6.2679999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":73.561000000000007,"5min":73.561000000000007,"15min":73.561000000000007},"last":1.1799999999999999},{"interface":"front","average":{"1min":5.3739999999999997,"5min":5.0300000000000002,"15min":4.9729999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":70.387,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.034}]},{"osd":6,"last update":"Sun Jan 15 14:30:08 2023","interfaces":[{"interface":"back","average":{"1min":5.6429999999999998,"5min":7.9470000000000001,"15min":8.3309999999999995},"min":{"1min":0.68700000000000006,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":73.638999999999996,"5min":73.638999999999996,"15min":73.638999999999996},"last":1.216},{"interface":"front","average":{"1min":5.5419999999999998,"5min":1.988,"15min":1.3959999999999999},"min":{"1min":0.93899999999999995,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":70.530000000000001,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.1080000000000001}]}]},{"osd":6,"up_from":0,"seq":55834574888,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574888,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":27,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":918836,"kb_used_data":92781880,"kb_used_omap":4032,"kb_used_meta":0,"kb_avail":92780544,"statfs":{"total":95948165120,"available":95007277056,"internally_reserved":1368064,"allocated":95008645120,"data_stored":95008645120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4129631,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,3],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":329,"commit_latency_ns":0,"apply_latency_ns":329000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":4.633,"15min":4.8209999999999997},"min":{"1min":0.47099999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.282,"5min":86.944999999999993,"15min":86.944999999999993},"last":2165.8420000000001},{"interface":"front","average":{"1min":4.0869999999999997,"5min":2.008,"15min":1.1899999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":70.352999999999994,"5min":70.352999999999994,"15min":70.352999999999994},"last":0.95099999999999996}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":2.145,"5min":7.5339999999999998,"15min":8.2400000000000002},"min":{"1min":0.60399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":24.489999999999998,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":12.505000000000001,"5min":4.1219999999999999,"15min":1.903},"min":{"1min":0.52900000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":110.06999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.83999999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":5.4859999999999998,"15min":5.5019999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":97.974000000000004,"5min":99.884,"15min":99.884},"last":2165.3220000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.88500000000000001,"15min":0.90100000000000002},"min":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.254,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.91200000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.2649999999999999,"15min":1.3140000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.3280000000000001,"5min":12.878,"15min":12.878},"last":0.86499999999999999},{"interface":"front","average":{"1min":2.6320000000000001,"5min":5.5659999999999998,"15min":5.9560000000000004},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":24.649999999999999,"5min":99.921000000000006,"15min":99.921000000000006},"last":2165.4690000000001}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.88600000000000001,"15min":0.90800000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.109,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.746,"15min":2.1139999999999999},"min":{"1min":0.65200000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3879999999999999,"5min":30.971,"15min":30.971},"last":0.68200000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":2.2149999999999999,"5min":5.8090000000000002,"15min":5.718},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":24.553999999999998,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.92600000000000005,"5min":3.4910000000000001,"15min":4.6050000000000004},"min":{"1min":0.66700000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.234,"5min":88.186999999999998,"15min":88.186999999999998},"last":2165.7330000000002}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":8.266,"15min":11.507},"min":{"1min":0.63400000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.4510000000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":2.2770000000000001,"5min":8.2899999999999991,"15min":10.256},"min":{"1min":0.65600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":24.603999999999999,"5min":99.843000000000004,"15min":99.843000000000004},"last":2165.5309999999999}]}]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":33,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":904836,"kb_used_data":92795952,"kb_used_omap":4338,"kb_used_meta":0,"kb_avail":92794544,"statfs":{"total":95948165120,"available":95021613056,"internally_reserved":1441792,"allocated":95023054848,"data_stored":95023054848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442872,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":8.657,"5min":6.4790000000000001,"15min":5.391},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":64.275999999999996,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":4.7889999999999997,"5min":6.2050000000000001,"15min":7.1630000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":36.207000000000001,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.82099999999999995}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.3579999999999997,"5min":11.628,"15min":12.897},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":60.664000000000001,"5min":159.256,"15min":159.256},"last":0.64400000000000002},{"interface":"front","average":{"1min":19.719999999999999,"5min":6.1059999999999999,"15min":2.5289999999999999},"min":{"1min":0.40200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":155.44,"5min":174.578,"15min":174.578},"last":0.89700000000000002}]},{"osd":2,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":3.1429999999999998,"5min":3.153,"15min":3.2829999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":40.692999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":7.3019999999999996,"5min":4.5700000000000003,"15min":2.9849999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":64.381,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.72899999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":8.7050000000000001,"5min":6.2599999999999998,"15min":6.4610000000000003},"min":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":72.438999999999993,"5min":72.438999999999993,"15min":72.438999999999993},"last":72.438999999999993},{"interface":"front","average":{"1min":8.8840000000000003,"5min":12.984,"15min":13.577},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":60.857999999999997,"5min":159.47200000000001,"15min":159.47200000000001},"last":0.79600000000000004}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.226,"5min":11.475,"15min":13.093},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":60.912999999999997,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.95099999999999996},{"interface":"front","average":{"1min":4.8529999999999998,"5min":3.359,"15min":1.8009999999999999},"min":{"1min":0.752,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":64.513000000000005,"5min":174.61000000000001,"15min":174.61000000000001},"last":1.1120000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.8799999999999999,"5min":3.3690000000000002,"15min":1.8109999999999999},"min":{"1min":0.69999999999999996,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":64.590000000000003,"5min":174.63999999999999,"15min":174.63999999999999},"last":1.022},{"interface":"front","average":{"1min":1,"5min":3.0529999999999999,"15min":3.5249999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.304,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.99399999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":3.206,"15min":3.7589999999999999},"min":{"1min":0.69699999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.2110000000000001,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.0589999999999999},{"interface":"front","average":{"1min":5.3739999999999997,"5min":14.025,"15min":16.734999999999999},"min":{"1min":0.72599999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":60.951999999999998,"5min":159.512,"15min":159.512},"last":1.0900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":23,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":901192,"kb_used_data":92803108,"kb_used_omap":3273,"kb_used_meta":0,"kb_avail":92798188,"statfs":{"total":95948165120,"available":95025344512,"internally_reserved":5038080,"allocated":95030382592,"data_stored":95030382592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3351882,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.5739999999999998,"5min":2.5009999999999999,"15min":1.2609999999999999},"min":{"1min":0.439,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":66.277000000000001,"5min":66.277000000000001,"15min":66.277000000000001},"last":40.423000000000002},{"interface":"front","average":{"1min":4.617,"5min":1.4670000000000001,"15min":0.96499999999999997},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":88.299999999999997,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.54200000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.2610000000000001,"5min":3.7229999999999999,"15min":1.7729999999999999},"min":{"1min":0.46500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":80.233999999999995,"5min":143.071,"15min":143.071},"last":0.60899999999999999},{"interface":"front","average":{"1min":4.6989999999999998,"5min":3.1000000000000001,"15min":3.27},"min":{"1min":0.505,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":88.186000000000007,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.72899999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":10.462,"15min":10.422000000000001},"min":{"1min":0.40999999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.94899999999999995,"5min":183.53,"15min":183.53},"last":0.68500000000000005},{"interface":"front","average":{"1min":5.7729999999999997,"5min":3.8149999999999999,"15min":1.7829999999999999},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":66.335999999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.52000000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":5.9489999999999998,"15min":4.4160000000000004},"min":{"1min":0.624,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":66.491,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":4.9290000000000003,"5min":5.8259999999999996,"15min":4.4809999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":88.459999999999994,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.83999999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":2.577,"15min":1.643},"min":{"1min":0.65800000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2829999999999999,"5min":143.262,"15min":143.262},"last":0.75900000000000001},{"interface":"front","average":{"1min":6.0339999999999998,"5min":2.1040000000000001,"15min":1.4750000000000001},"min":{"1min":0.71899999999999997,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":66.456999999999994,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.88600000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.9889999999999999,"5min":1.8879999999999999,"15min":1.3169999999999999},"min":{"1min":0.70699999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":88.542000000000002,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.028,"5min":2.7109999999999999,"15min":1.7150000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.585,"5min":143.215,"15min":143.215},"last":0.79900000000000004}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.6230000000000002,"5min":3.5489999999999999,"15min":3.222},"min":{"1min":0.81100000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":88.623999999999995,"5min":88.623999999999995,"15min":88.623999999999995},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.048,"5min":3.5630000000000002,"15min":3.0390000000000001},"min":{"1min":0.79900000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3859999999999999,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.93200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":34,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":937996,"kb_used_data":92777436,"kb_used_omap":4266,"kb_used_meta":0,"kb_avail":92761384,"statfs":{"total":95948165120,"available":94987657216,"internally_reserved":16437248,"allocated":95004094464,"data_stored":95004094464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4369010,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3609999999999998,"5min":4.9240000000000004,"15min":4.2359999999999998},"min":{"1min":0.39900000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":100.533,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":4.6509999999999998,"5min":3.6459999999999999,"15min":4.0970000000000004},"min":{"1min":0.373,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":85.751000000000005,"5min":85.751000000000005,"15min":85.751000000000005},"last":0.53100000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":2.6600000000000001,"5min":1.2270000000000001,"15min":1.0529999999999999},"min":{"1min":0.32500000000000001,"5min":0.245,"15min":0.245},"max":{"1min":43.759999999999998,"5min":43.759999999999998,"15min":43.759999999999998},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":3.1869999999999998,"15min":3.7770000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82899999999999996,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.65500000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":17.001000000000001,"5min":15.438000000000001,"15min":18.370000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":245.637,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":5.6310000000000002,"5min":5.9980000000000002,"15min":6.8289999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":85.814999999999998,"5min":85.814999999999998,"15min":85.814999999999998},"last":0.60799999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:30:09 2023","interfaces":[{"interface":"back","average":{"1min":3.4329999999999998,"5min":4.4690000000000003,"15min":4.641},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":53.667000000000002,"5min":53.667000000000002,"15min":53.667000000000002},"last":0.876},{"interface":"front","average":{"1min":5.3319999999999999,"5min":3.9500000000000002,"15min":3.7200000000000002},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":93.989000000000004,"5min":93.989000000000004,"15min":93.989000000000004},"last":1.0860000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.98099999999999998,"15min":1.0149999999999999},"min":{"1min":0.58099999999999996,"5min":0.433,"15min":0.433},"max":{"1min":1.244,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.92200000000000004,"5min":3.036,"15min":3.9700000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.1339999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.90100000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":4.2060000000000004,"15min":4.7359999999999998},"min":{"1min":0.745,"5min":0.63,"15min":0.63},"max":{"1min":85.884,"5min":85.884,"15min":85.884},"last":0.96399999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.056,"15min":1.101},"min":{"1min":0.69599999999999995,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.298,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.77100000000000002}]},{"osd":7,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":8.3879999999999999,"5min":7.1310000000000002,"15min":7.0549999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":157.03700000000001,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":5.0919999999999996,"5min":4.0730000000000004,"15min":4.524},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":85.852999999999994,"5min":85.852999999999994,"15min":85.852999999999994},"last":1.0109999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574888,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":822720,"kb_used_data":92876780,"kb_used_omap":4088,"kb_used_meta":0,"kb_avail":92876660,"statfs":{"total":95948165120,"available":95105699840,"internally_reserved":122880,"allocated":95105822720,"data_stored":95105822720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4186358,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":7.2519999999999998,"5min":2.6970000000000001,"15min":1.3140000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":69.816999999999993,"5min":69.872,"15min":69.872},"last":0.55700000000000005},{"interface":"front","average":{"1min":5.6130000000000004,"5min":2.6789999999999998,"15min":1.379},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":98.203999999999994,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.89300000000000002}]},{"osd":2,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.7039999999999997,"5min":3.407,"15min":1.5980000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":98.286000000000001,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":3.0270000000000001,"5min":2.6819999999999999,"15min":1.3660000000000001},"min":{"1min":0.38400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":48.893000000000001,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.628}]},{"osd":3,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":12.154999999999999,"5min":13.413,"15min":15.657},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":94.712999999999994,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":10.484,"5min":5.6399999999999997,"15min":2.4140000000000001},"min":{"1min":0.42399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":98.352999999999994,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.65900000000000003}]},{"osd":4,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.5570000000000004,"5min":3.2309999999999999,"15min":1.734},"min":{"1min":0.63800000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":61.536000000000001,"5min":71.483000000000004,"15min":71.483000000000004},"last":0.84499999999999997},{"interface":"front","average":{"1min":9.2989999999999995,"5min":3.9950000000000001,"15min":2.0150000000000001},"min":{"1min":0.67400000000000004,"5min":0.439,"15min":0.439},"max":{"1min":70.179000000000002,"5min":70.179000000000002,"15min":70.179000000000002},"last":1.2}]},{"osd":5,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.9009999999999998,"5min":8.218,"15min":9.2710000000000008},"min":{"1min":0.68500000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":98.411000000000001,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.7709999999999999,"15min":1.2829999999999999},"min":{"1min":0.63400000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.2959999999999998,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.81299999999999994}]},{"osd":6,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":7.7030000000000003,"5min":3.2040000000000002,"15min":1.8480000000000001},"min":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":70.259,"5min":70.259,"15min":70.259},"last":0.87},{"interface":"front","average":{"1min":5.984,"5min":3.056,"15min":1.774},"min":{"1min":0.749,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":98.484999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":0.93400000000000005}]},{"osd":7,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":4.633,"5min":12.002000000000001,"15min":14.448},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":73.864999999999995,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":7.7199999999999998,"5min":3.21,"15min":1.8500000000000001},"min":{"1min":0.70399999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":70.322000000000003,"5min":70.322000000000003,"15min":70.322000000000003},"last":1.01}]}]}],"pool_statfs":[]}} 2023-01-15T14:31:16.076 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:31:16.077 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 46 2023-01-15T14:31:16.077 INFO:tasks.rados.rados.0.smithi090.stdout:724: done (3 left) 2023-01-15T14:31:16.077 INFO:tasks.rados.rados.0.smithi090.stdout:725: snap_remove snap 70 2023-01-15T14:31:16.078 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:31:16.079 INFO:tasks.rados.rados.0.smithi090.stdout:725: done (3 left) 2023-01-15T14:31:16.079 INFO:tasks.rados.rados.0.smithi090.stdout:726: rollback oid 46 current snap is 74 2023-01-15T14:31:16.080 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 46 to 72 2023-01-15T14:31:16.080 INFO:tasks.rados.rados.0.smithi090.stdout:721: finishing write tid 2 to smithi09025264-23 2023-01-15T14:31:16.080 INFO:tasks.rados.rados.0.smithi090.stdout:721: finishing write tid 3 to smithi09025264-23 2023-01-15T14:31:16.080 INFO:tasks.rados.rados.0.smithi090.stdout:721: finishing write tid 4 to smithi09025264-23 2023-01-15T14:31:16.080 INFO:tasks.rados.rados.0.smithi090.stdout:721: oid 23 updating version 0 to 163 2023-01-15T14:31:16.081 INFO:tasks.rados.rados.0.smithi090.stdout:721: oid 23 updating version 163 to 164 2023-01-15T14:31:16.081 INFO:tasks.rados.rados.0.smithi090.stdout:721: oid 23 updating version 164 to 165 2023-01-15T14:31:16.081 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 165 (ObjNum 282 snap 74 seq_num 282) dirty exists 2023-01-15T14:31:16.081 INFO:tasks.rados.rados.0.smithi090.stdout:721: left oid 23 (ObjNum 282 snap 74 seq_num 282) 2023-01-15T14:31:16.082 INFO:tasks.rados.rados.0.smithi090.stdout:720: finishing write tid 1 to smithi09025264-16 2023-01-15T14:31:16.082 INFO:tasks.rados.rados.0.smithi090.stdout:720: finishing write tid 2 to smithi09025264-16 2023-01-15T14:31:16.082 INFO:tasks.rados.rados.0.smithi090.stdout:720: finishing write tid 3 to smithi09025264-16 2023-01-15T14:31:16.082 INFO:tasks.rados.rados.0.smithi090.stdout:720: finishing write tid 4 to smithi09025264-16 2023-01-15T14:31:16.082 INFO:tasks.rados.rados.0.smithi090.stdout:720: oid 16 updating version 0 to 152 2023-01-15T14:31:16.082 INFO:tasks.rados.rados.0.smithi090.stdout:720: oid 16 version 152 is already newer than 151 2023-01-15T14:31:16.082 INFO:tasks.rados.rados.0.smithi090.stdout:720: oid 16 updating version 152 to 153 2023-01-15T14:31:16.082 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 153 (ObjNum 281 snap 74 seq_num 281) dirty exists 2023-01-15T14:31:16.082 INFO:tasks.rados.rados.0.smithi090.stdout:720: left oid 16 (ObjNum 281 snap 74 seq_num 281) 2023-01-15T14:31:16.082 INFO:tasks.rados.rados.0.smithi090.stdout:723: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:16.083 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:15.947+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:16.228 INFO:tasks.rados.rados.0.smithi090.stdout:720: done (3 left) 2023-01-15T14:31:16.229 INFO:tasks.rados.rados.0.smithi090.stdout:721: done (2 left) 2023-01-15T14:31:16.229 INFO:tasks.rados.rados.0.smithi090.stdout:723: done (1 left) 2023-01-15T14:31:16.229 INFO:tasks.rados.rados.0.smithi090.stdout:727: write oid 40 current snap is 74 2023-01-15T14:31:16.230 INFO:tasks.rados.rados.0.smithi090.stdout:727: seq_num 284 ranges {773188=475777,1969239=724762} 2023-01-15T14:31:16.230 INFO:tasks.rados.rados.0.smithi090.stdout:727: writing smithi09025264-40 from 773188 to 1248965 tid 1 2023-01-15T14:31:16.230 INFO:tasks.rados.rados.0.smithi090.stdout:727: writing smithi09025264-40 from 1969239 to 2694001 tid 2 2023-01-15T14:31:16.230 INFO:tasks.rados.rados.0.smithi090.stdout:728: write (excl) oid 2 current snap is 74 2023-01-15T14:31:16.230 INFO:tasks.rados.rados.0.smithi090.stdout:728: seq_num 285 ranges {789995=551026,2027721=468420} 2023-01-15T14:31:16.231 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:16.232 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:16.232 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:31:16.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.232 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:1912 3.4 e189/181)", 2023-01-15T14:31:16.233 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:15.986750+0000", 2023-01-15T14:31:16.233 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.078296959999999999, 2023-01-15T14:31:16.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.078330536000000006, 2023-01-15T14:31:16.233 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "no flag points reached", 2023-01-15T14:31:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:15.986750+0000", 2023-01-15T14:31:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:15.986750+0000", 2023-01-15T14:31:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:15.986748+0000", 2023-01-15T14:31:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:15.986761+0000", 2023-01-15T14:31:16.241 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3006e-05 2023-01-15T14:31:16.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:16.241 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.241 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:16.241 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:15.986802+0000", 2023-01-15T14:31:16.242 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0703e-05 2023-01-15T14:31:16.242 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:16.243 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:16.243 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:16.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:16.243 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.244 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.4.0:398 3.1d e189/166)", 2023-01-15T14:31:16.244 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:15.998096+0000", 2023-01-15T14:31:16.244 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.066950130999999996, 2023-01-15T14:31:16.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.067055192, 2023-01-15T14:31:16.244 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:16.245 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:16.245 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:16.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:16.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:15.998096+0000", 2023-01-15T14:31:16.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:16.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:16.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:16.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:15.998096+0000", 2023-01-15T14:31:16.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:16.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:16.248 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.248 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:16.248 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:15.998096+0000", 2023-01-15T14:31:16.248 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:16.248 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:16.249 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.249 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:16.249 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:15.998099+0000", 2023-01-15T14:31:16.249 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1700000000000001e-06 2023-01-15T14:31:16.249 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:16.250 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.250 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:16.250 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:15.998116+0000", 2023-01-15T14:31:16.250 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6708e-05 2023-01-15T14:31:16.251 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:16.251 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:16.251 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:16.251 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:15.998121+0000", 2023-01-15T14:31:16.252 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4040000000000003e-06 2023-01-15T14:31:16.252 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:16.252 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:16.253 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:16.253 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:16.253 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2023-01-15T14:31:16.253 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 2 2023-01-15T14:31:16.253 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:16.254 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:31:16.255 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:16.222+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:16.256 INFO:tasks.rados.rados.0.smithi090.stdout:728: writing smithi09025264-2 from 789995 to 1341021 tid 1 2023-01-15T14:31:16.256 INFO:tasks.rados.rados.0.smithi090.stdout:728: writing smithi09025264-2 from 2027721 to 2496141 tid 2 2023-01-15T14:31:16.256 INFO:tasks.rados.rados.0.smithi090.stdout:729: delete oid 44 current snap is 74 2023-01-15T14:31:16.257 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:16.257 INFO:tasks.rados.rados.0.smithi090.stdout:728: finishing write tid 1 to smithi09025264-2 2023-01-15T14:31:16.258 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:16.259 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:16.260 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:16.278 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:16.301 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:16.380 INFO:tasks.rados.rados.0.smithi090.stdout:728: finishing write tid 2 to smithi09025264-2 2023-01-15T14:31:16.381 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:31:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:31:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:31:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:31:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:31:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:31:16.382 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:31:16.383 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:31:16.383 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:16.375+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:16.385 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:16.385 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:16.391 INFO:tasks.rados.rados.0.smithi090.stdout:726: finishing rollback tid 0 to smithi09025264-46 2023-01-15T14:31:16.392 INFO:tasks.rados.rados.0.smithi090.stdout:729: done (3 left) 2023-01-15T14:31:16.448 INFO:tasks.rados.rados.0.smithi090.stdout:730: write (excl) oid 35 current snap is 74 2023-01-15T14:31:16.559 INFO:tasks.rados.rados.0.smithi090.stdout:730: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2023-01-15T14:31:16.562 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:31:16.563 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:16.558+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:16.564 INFO:tasks.rados.rados.0.smithi090.stdout:730: writing smithi09025264-35 from 423609 to 1125133 tid 1 2023-01-15T14:31:16.564 INFO:tasks.rados.rados.0.smithi090.stdout:730: writing smithi09025264-35 from 1628332 to 2100421 tid 2 2023-01-15T14:31:16.564 INFO:tasks.rados.rados.0.smithi090.stdout:730: writing smithi09025264-35 from 2648987 to 2823609 tid 3 2023-01-15T14:31:16.564 INFO:tasks.rados.rados.0.smithi090.stdout:731: read oid 15 snap 67 2023-01-15T14:31:16.564 INFO:tasks.rados.rados.0.smithi090.stdout:731: expect deleted 2023-01-15T14:31:16.565 INFO:tasks.rados.rados.0.smithi090.stdout:732: read oid 13 snap -1 2023-01-15T14:31:16.565 INFO:tasks.rados.rados.0.smithi090.stdout:732: expect (ObjNum 216 snap 49 seq_num 216) 2023-01-15T14:31:16.565 INFO:tasks.rados.rados.0.smithi090.stdout:733: snap_create 2023-01-15T14:31:16.565 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:31:16.565 INFO:tasks.rados.rados.0.smithi090.stdout:728: finishing write tid 3 to smithi09025264-2 2023-01-15T14:31:16.565 INFO:tasks.rados.rados.0.smithi090.stdout:728: finishing write tid 4 to smithi09025264-2 2023-01-15T14:31:16.565 INFO:tasks.rados.rados.0.smithi090.stdout:728: oid 2 updating version 0 to 71 2023-01-15T14:31:16.565 INFO:tasks.rados.rados.0.smithi090.stdout:728: oid 2 version 71 is already newer than 69 2023-01-15T14:31:16.565 INFO:tasks.rados.rados.0.smithi090.stdout:728: oid 2 version 71 is already newer than 70 2023-01-15T14:31:16.566 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 71 (ObjNum 285 snap 74 seq_num 285) dirty exists 2023-01-15T14:31:16.566 INFO:tasks.rados.rados.0.smithi090.stdout:728: left oid 2 (ObjNum 285 snap 74 seq_num 285) 2023-01-15T14:31:16.566 INFO:tasks.rados.rados.0.smithi090.stdout:728: done (5 left) 2023-01-15T14:31:16.566 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:31:16.566 INFO:tasks.rados.rados.0.smithi090.stdout:732: expect (ObjNum 216 snap 49 seq_num 216) 2023-01-15T14:31:16.661 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:16.658+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:16.711 INFO:tasks.rados.rados.0.smithi090.stdout:732: done (4 left) 2023-01-15T14:31:16.711 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:31:16.711 INFO:tasks.rados.rados.0.smithi090.stdout:726: finishing rollback tid 1 to smithi09025264-46 2023-01-15T14:31:16.711 INFO:tasks.rados.rados.0.smithi090.stdout:726: finishing rollback tid 2 to smithi09025264-46 2023-01-15T14:31:16.711 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 199 (ObjNum 256 snap 60 seq_num 256) dirty exists 2023-01-15T14:31:16.711 INFO:tasks.rados.rados.0.smithi090.stdout:727: finishing write tid 1 to smithi09025264-40 2023-01-15T14:31:16.711 INFO:tasks.rados.rados.0.smithi090.stdout:726: done (3 left) 2023-01-15T14:31:16.711 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:31:16.711 INFO:tasks.rados.rados.0.smithi090.stdout:727: finishing write tid 2 to smithi09025264-40 2023-01-15T14:31:16.712 INFO:tasks.rados.rados.0.smithi090.stdout:731: done (2 left) 2023-01-15T14:31:16.712 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:16.712 INFO:tasks.rados.rados.0.smithi090.stdout:730: finishing write tid 1 to smithi09025264-35 2023-01-15T14:31:16.712 INFO:tasks.rados.rados.0.smithi090.stdout:727: finishing write tid 3 to smithi09025264-40 2023-01-15T14:31:16.712 INFO:tasks.rados.rados.0.smithi090.stdout:727: finishing write tid 4 to smithi09025264-40 2023-01-15T14:31:16.712 INFO:tasks.rados.rados.0.smithi090.stdout:727: oid 40 updating version 0 to 201 2023-01-15T14:31:16.712 INFO:tasks.rados.rados.0.smithi090.stdout:727: oid 40 updating version 201 to 202 2023-01-15T14:31:16.712 INFO:tasks.rados.rados.0.smithi090.stdout:727: oid 40 version 202 is already newer than 200 2023-01-15T14:31:16.712 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 202 (ObjNum 284 snap 74 seq_num 284) dirty exists 2023-01-15T14:31:16.712 INFO:tasks.rados.rados.0.smithi090.stdout:727: left oid 40 (ObjNum 284 snap 74 seq_num 284) 2023-01-15T14:31:16.712 INFO:tasks.rados.rados.0.smithi090.stdout:727: done (1 left) 2023-01-15T14:31:16.713 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:16.713 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:16.713 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:16.713 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:31:16.762 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:16.758+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:16.798 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 46 2023-01-15T14:31:16.819 INFO:tasks.rados.rados.0.smithi090.stdout:730: finishing write tid 2 to smithi09025264-35 2023-01-15T14:31:16.819 INFO:tasks.rados.rados.0.smithi090.stdout:730: finishing write tid 3 to smithi09025264-35 2023-01-15T14:31:16.819 INFO:tasks.rados.rados.0.smithi090.stdout:730: finishing write tid 4 to smithi09025264-35 2023-01-15T14:31:16.820 INFO:tasks.rados.rados.0.smithi090.stdout:730: finishing write tid 5 to smithi09025264-35 2023-01-15T14:31:16.820 INFO:tasks.rados.rados.0.smithi090.stdout:730: oid 35 updating version 0 to 222 2023-01-15T14:31:16.820 INFO:tasks.rados.rados.0.smithi090.stdout:730: oid 35 updating version 222 to 224 2023-01-15T14:31:16.820 INFO:tasks.rados.rados.0.smithi090.stdout:730: oid 35 updating version 224 to 225 2023-01-15T14:31:16.820 INFO:tasks.rados.rados.0.smithi090.stdout:730: oid 35 version 225 is already newer than 223 2023-01-15T14:31:16.821 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 225 (ObjNum 287 snap 74 seq_num 287) dirty exists 2023-01-15T14:31:16.821 INFO:tasks.rados.rados.0.smithi090.stdout:730: left oid 35 (ObjNum 287 snap 74 seq_num 287) 2023-01-15T14:31:16.821 INFO:tasks.rados.rados.0.smithi090.stdout:730: done (0 left) 2023-01-15T14:31:16.824 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:16.824 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:16.833 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:31:16.877 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:16.874+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:16.964 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:16.962+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:16.983 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:16.983 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:16.993 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:31:17.066 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:17.063+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:17.097 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:17.098 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:17.107 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:31:17.167 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:17.163+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:17.218 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:17.219 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:17.228 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:31:17.268 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:17.262+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:17.336 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:17.698 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:17.700 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:31:17.700 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:17.698+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:17.802 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:17.799+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:17.830 INFO:tasks.rados.rados.0.smithi090.stdout:733: done (0 left) 2023-01-15T14:31:17.831 INFO:tasks.rados.rados.0.smithi090.stdout:734: write (excl) oid 30 current snap is 75 2023-01-15T14:31:17.831 INFO:tasks.rados.rados.0.smithi090.stdout:734: seq_num 288 ranges {440416=776773,1686814=732100} 2023-01-15T14:31:17.831 INFO:tasks.rados.rados.0.smithi090.stdout:734: writing smithi09025264-30 from 440416 to 1217189 tid 1 2023-01-15T14:31:17.831 INFO:tasks.rados.rados.0.smithi090.stdout:734: writing smithi09025264-30 from 1686814 to 2418914 tid 2 2023-01-15T14:31:17.831 INFO:tasks.rados.rados.0.smithi090.stdout:735: write (excl) oid 3 current snap is 75 2023-01-15T14:31:17.831 INFO:tasks.rados.rados.0.smithi090.stdout:735: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2023-01-15T14:31:17.831 INFO:tasks.rados.rados.0.smithi090.stdout:735: writing smithi09025264-3 from 457223 to 1025598 tid 1 2023-01-15T14:31:17.831 INFO:tasks.rados.rados.0.smithi090.stdout:735: writing smithi09025264-3 from 1461649 to 1937407 tid 2 2023-01-15T14:31:17.833 INFO:tasks.rados.rados.0.smithi090.stdout:735: writing smithi09025264-3 from 2420186 to 2857223 tid 3 2023-01-15T14:31:17.833 INFO:tasks.rados.rados.0.smithi090.stdout:736: delete oid 10 current snap is 75 2023-01-15T14:31:17.859 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:17.859 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:17.870 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:31:17.900 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:17.898+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:17.975 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:17.976 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:17.986 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:31:18.004 INFO:tasks.rados.rados.0.smithi090.stdout:735: finishing write tid 1 to smithi09025264-3 2023-01-15T14:31:18.020 INFO:tasks.rados.rados.0.smithi090.stdout:734: finishing write tid 1 to smithi09025264-30 2023-01-15T14:31:18.021 INFO:tasks.rados.rados.0.smithi090.stdout:736: done (2 left) 2023-01-15T14:31:18.022 INFO:tasks.rados.rados.0.smithi090.stdout:737: read oid 11 snap -1 2023-01-15T14:31:18.022 INFO:tasks.rados.rados.0.smithi090.stdout:737: expect deleted 2023-01-15T14:31:18.022 INFO:tasks.rados.rados.0.smithi090.stdout:738: copy_from oid 41 from oid 24 current snap is 75 2023-01-15T14:31:18.022 INFO:tasks.rados.rados.0.smithi090.stdout:739: rollback oid 28 current snap is 75 2023-01-15T14:31:18.022 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 28 to 69 2023-01-15T14:31:18.022 INFO:tasks.rados.rados.0.smithi090.stdout:740: read oid 42 snap -1 2023-01-15T14:31:18.022 INFO:tasks.rados.rados.0.smithi090.stdout:740: expect deleted 2023-01-15T14:31:18.022 INFO:tasks.rados.rados.0.smithi090.stdout:741: snap_create 2023-01-15T14:31:18.022 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:31:18.023 INFO:tasks.rados.rados.0.smithi090.stdout:737: done (5 left) 2023-01-15T14:31:18.023 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:31:18.029 INFO:tasks.rados.rados.0.smithi090.stdout:739: finishing rollback tid 0 to smithi09025264-28 2023-01-15T14:31:18.029 INFO:tasks.rados.rados.0.smithi090.stdout:739: finishing rollback tid 1 to smithi09025264-28 2023-01-15T14:31:18.033 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:18.030+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:18.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:31:18.124 INFO:tasks.rados.rados.0.smithi090.stdout:734: finishing write tid 2 to smithi09025264-30 2023-01-15T14:31:18.124 INFO:tasks.rados.rados.0.smithi090.stdout:740: done (4 left) 2023-01-15T14:31:18.124 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:31:18.124 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:18.119+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:18.125 INFO:tasks.rados.rados.0.smithi090.stdout:734: finishing write tid 3 to smithi09025264-30 2023-01-15T14:31:18.125 INFO:tasks.rados.rados.0.smithi090.stdout:734: finishing write tid 4 to smithi09025264-30 2023-01-15T14:31:18.125 INFO:tasks.rados.rados.0.smithi090.stdout:734: oid 30 updating version 0 to 77 2023-01-15T14:31:18.125 INFO:tasks.rados.rados.0.smithi090.stdout:734: oid 30 updating version 77 to 78 2023-01-15T14:31:18.125 INFO:tasks.rados.rados.0.smithi090.stdout:734: oid 30 updating version 78 to 79 2023-01-15T14:31:18.125 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 79 (ObjNum 288 snap 75 seq_num 288) dirty exists 2023-01-15T14:31:18.125 INFO:tasks.rados.rados.0.smithi090.stdout:734: left oid 30 (ObjNum 288 snap 75 seq_num 288) 2023-01-15T14:31:18.126 INFO:tasks.rados.rados.0.smithi090.stdout:734: done (3 left) 2023-01-15T14:31:18.126 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:31:18.126 INFO:tasks.rados.rados.0.smithi090.stdout:735: finishing write tid 2 to smithi09025264-3 2023-01-15T14:31:18.127 INFO:tasks.rados.rados.0.smithi090.stdout:735: finishing write tid 3 to smithi09025264-3 2023-01-15T14:31:18.139 INFO:tasks.rados.rados.0.smithi090.stdout:738: finishing copy_from to smithi09025264-41 2023-01-15T14:31:18.139 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 232 (ObjNum 266 snap 66 seq_num 266) dirty exists 2023-01-15T14:31:18.139 INFO:tasks.rados.rados.0.smithi090.stdout:738: finishing copy_from racing read to smithi09025264-41 2023-01-15T14:31:18.139 INFO:tasks.rados.rados.0.smithi090.stdout:738: done (2 left) 2023-01-15T14:31:18.139 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:18.143 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:18.143 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:18.153 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:31:18.203 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:18.199+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:18.209 INFO:tasks.rados.rados.0.smithi090.stdout:735: finishing write tid 4 to smithi09025264-3 2023-01-15T14:31:18.210 INFO:tasks.rados.rados.0.smithi090.stdout:735: finishing write tid 5 to smithi09025264-3 2023-01-15T14:31:18.210 INFO:tasks.rados.rados.0.smithi090.stdout:735: oid 3 updating version 0 to 215 2023-01-15T14:31:18.210 INFO:tasks.rados.rados.0.smithi090.stdout:735: oid 3 version 215 is already newer than 213 2023-01-15T14:31:18.210 INFO:tasks.rados.rados.0.smithi090.stdout:735: oid 3 version 215 is already newer than 214 2023-01-15T14:31:18.211 INFO:tasks.rados.rados.0.smithi090.stdout:735: oid 3 version 215 is already newer than 212 2023-01-15T14:31:18.211 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 215 (ObjNum 289 snap 75 seq_num 289) dirty exists 2023-01-15T14:31:18.211 INFO:tasks.rados.rados.0.smithi090.stdout:735: left oid 3 (ObjNum 289 snap 75 seq_num 289) 2023-01-15T14:31:18.211 INFO:tasks.rados.rados.0.smithi090.stdout:735: done (1 left) 2023-01-15T14:31:18.211 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:18.224 INFO:tasks.rados.rados.0.smithi090.stdout:739: finishing rollback tid 2 to smithi09025264-28 2023-01-15T14:31:18.224 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 264 (ObjNum 222 snap 51 seq_num 222) dirty exists 2023-01-15T14:31:18.224 INFO:tasks.rados.rados.0.smithi090.stdout:739: done (0 left) 2023-01-15T14:31:18.255 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:18.256 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:18.266 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:31:18.310 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:18.307+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:18.374 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:18.375 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:18.383 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:31:18.385 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:31:18.405 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:18.399+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:18.485 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:18.487 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:18.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:31:18.518 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:18.518 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:18.515+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:18.536 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:18.555 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:18.593 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:18.595 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:18.607 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:18.603+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:18.617 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:18.623 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:18.623 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:18.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:18.623 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:18.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.624 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:18.624 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:18.624 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.55451087500001, 2023-01-15T14:31:18.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:31:18.624 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.624 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.624 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.624 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.624 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.624 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:31:18.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:18.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:31:18.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:31:18.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:31:18.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:31:18.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:31:18.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:31:18.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:31:18.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:31:18.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:31:18.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:31:18.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:18.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:31:18.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:31:18.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:31:18.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:31:18.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:31:18.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.633 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:18.633 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:18.633 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.55423062700001, 2023-01-15T14:31:18.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:18.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:31:18.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:31:18.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:31:18.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:31:18.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:31:18.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:31:18.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:31:18.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:31:18.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:31:18.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:31:18.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:18.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:31:18.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:31:18.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:31:18.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:31:18.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:31:18.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:31:18.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:31:18.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:31:18.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:31:18.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:31:18.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:31:18.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:31:18.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:31:18.642 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.55419800999999, 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.643 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.644 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.644 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:31:18.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.644 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:18.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:31:18.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:31:18.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:31:18.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:31:18.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:31:18.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:31:18.649 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.452880024, 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:18.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:31:18.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:18.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:31:18.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:31:18.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:31:18.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:31:18.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:31:18.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:31:18.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:31:18.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:31:18.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:31:18.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:31:18.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:31:18.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:18.659 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:18.660 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.45285002099999, 2023-01-15T14:31:18.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:31:18.660 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.660 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:18.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.660 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:31:18.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.660 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:31:18.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:18.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:31:18.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:31:18.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:31:18.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:31:18.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:31:18.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:31:18.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:31:18.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.666 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.937841456, 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:18.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:18.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:31:18.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:31:18.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:31:18.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:31:18.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:31:18.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:31:18.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:31:18.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.676 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.676 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:18.676 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:18.676 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.93735910199999, 2023-01-15T14:31:18.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:31:18.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.676 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.797 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:31:18.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.798 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:18.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:18.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:31:18.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:18.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:31:18.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:31:18.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:31:18.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:31:18.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:31:18.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:31:18.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:31:18.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:31:18.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:31:18.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:31:18.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:31:18.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:31:18.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:31:18.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:31:18.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:18.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:31:18.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:31:18.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:31:18.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:31:18.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:31:18.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:31:18.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:31:18.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:31:18.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:31:18.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:31:18.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:31:18.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:31:18.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:31:18.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:31:18.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:31:18.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:31:18.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:31:18.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.93730653899999, 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:18.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.814 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:31:18.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.814 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:18.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:31:18.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:31:18.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:31:18.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:31:18.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:31:18.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:31:18.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:31:18.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:31:18.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:31:18.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:31:18.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:31:18.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:31:18.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:31:18.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:31:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:31:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 158.566104835, 2023-01-15T14:31:18.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:18.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:18.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:31:18.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:18.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:31:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:31:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:31:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:31:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:31:18.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:31:18.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:18.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:31:18.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:31:18.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:31:18.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:31:18.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:31:18.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:31:18.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:31:18.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:31:18.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:31:18.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:31:18.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:31:18.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:31:18.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:31:18.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:31:18.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:31:18.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:31:18.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:31:18.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:31:18.832 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.832 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.833 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.833 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:18.833 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:18.833 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 158.56598816100001, 2023-01-15T14:31:18.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:31:18.833 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.834 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:18.834 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.834 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.834 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.834 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:31:18.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.834 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:18.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:18.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:31:18.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:18.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:31:18.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:31:18.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:31:18.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:31:18.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:31:18.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:31:18.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:31:18.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:31:18.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:18.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:31:18.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:31:18.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:31:18.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:31:18.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:31:18.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:31:18.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:31:18.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:31:18.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:31:18.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:31:18.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:31:18.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:31:18.842 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.842 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.842 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.843 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:31:18.843 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:18.843 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 150.83985588499999, 2023-01-15T14:31:18.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:31:18.843 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.843 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:18.843 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.844 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.844 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.844 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:31:18.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.844 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:18.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:18.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:31:18.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:18.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:31:18.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:31:18.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:31:18.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:31:18.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:31:18.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:31:18.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:31:18.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:31:18.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:31:18.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:31:18.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:31:18.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:31:18.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:31:18.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:31:18.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:31:18.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:31:18.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.852 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.852 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:31:18.853 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:18.853 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 114.623605826, 2023-01-15T14:31:18.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:31:18.853 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.853 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:31:18.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:18.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:18.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:31:18.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:18.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:31:18.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:31:18.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:31:18.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:31:18.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:31:18.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:31:18.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:31:18.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:31:18.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:31:18.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:18.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:31:18.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:31:18.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:31:18.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:31:18.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:31:18.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:31:18.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:31:18.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:31:18.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:31:18.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.864 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:31:18.864 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:18.864 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 114.623572259, 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:18.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:31:18.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:31:18.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:31:18.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:31:18.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:18.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:31:18.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:31:18.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:31:18.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:31:18.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:31:18.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:31:18.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:31:18.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.873 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.857668712999995, 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.874 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:31:18.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:18.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:31:18.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:31:18.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:31:18.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:31:18.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:31:18.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:31:18.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:31:18.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:31:18.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:31:18.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.314489913000003, 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:31:18.883 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:31:18.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:31:18.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:31:18.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:31:18.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:31:18.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:31:18.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:18.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:31:18.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:31:18.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:31:18.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:31:18.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:31:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:31:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:31:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:31:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:31:18.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:31:18.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:31:18.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:31:18.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:31:18.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:31:18.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.892 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.892 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.892 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:18.892 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:31:18.892 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 79.314450738000005, 2023-01-15T14:31:18.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:31:18.892 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.892 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:18.892 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.893 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.893 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.893 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:31:18.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.893 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:31:18.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:31:18.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:31:18.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:31:18.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:31:18.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:31:18.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:31:18.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:31:18.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:31:18.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:31:18.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:31:18.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:31:18.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:31:18.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:31:18.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:31:18.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:31:18.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:31:18.959 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.960 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.960 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.960 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1373 3.12 3:495d4bbc:::smithi09025264-16:head [write 2873523~591733] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:31:18.960 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:31:18.960 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.922334411999998, 2023-01-15T14:31:18.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40949580099999999, 2023-01-15T14:31:18.960 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.960 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.961 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.961 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1373 2023-01-15T14:31:18.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.961 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:31:18.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693440+0000", 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693808+0000", 2023-01-15T14:31:18.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000367742 2023-01-15T14:31:18.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693831+0000", 2023-01-15T14:31:18.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3213e-05 2023-01-15T14:31:18.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693846+0000", 2023-01-15T14:31:18.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4909000000000001e-05 2023-01-15T14:31:18.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.695833+0000", 2023-01-15T14:31:18.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019869240000000002 2023-01-15T14:31:18.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696121+0000", 2023-01-15T14:31:18.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028753300000000003 2023-01-15T14:31:18.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798577+0000", 2023-01-15T14:31:18.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102456808 2023-01-15T14:31:18.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798849+0000", 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027138999999999999 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799155+0000", 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030594800000000001 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799614+0000", 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045960700000000001 2023-01-15T14:31:18.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799737+0000", 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012218900000000001 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799998+0000", 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026129700000000002 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800639+0000", 2023-01-15T14:31:18.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064108800000000005 2023-01-15T14:31:18.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102023+0000", 2023-01-15T14:31:18.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30138417200000001 2023-01-15T14:31:18.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102198+0000", 2023-01-15T14:31:18.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017453600000000001 2023-01-15T14:31:18.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102398+0000", 2023-01-15T14:31:18.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020078000000000001 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102937+0000", 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053874499999999998 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:31:18.970 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:31:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.921909818000003, 2023-01-15T14:31:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:31:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:31:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:31:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:31:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:31:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:31:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:31:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:31:18.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:31:18.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:31:18.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:31:18.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:31:18.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:31:18.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:31:18.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:31:18.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:31:18.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:31:18.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:31:18.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:31:18.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:18.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:31:18.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:31:18.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:31:18.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:18.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:31:18.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:31:18.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:18.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:31:18.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:31:18.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:31:18.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:31:18.980 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.980 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.980 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.980 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:31:18.981 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:31:18.981 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.921857489000004, 2023-01-15T14:31:18.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:31:18.981 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.981 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:18.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.981 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:31:18.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.981 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:31:18.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:31:18.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:31:18.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:18.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:31:18.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:31:18.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:31:18.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:31:18.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:31:18.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:31:18.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:31:18.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:31:18.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:31:18.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:31:18.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:31:18.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:31:18.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:31:18.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:31:18.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:31:18.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.988 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.988 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:31:18.988 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:18.989 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 52.324944289999998, 2023-01-15T14:31:18.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:31:18.989 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:18.989 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:18.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:18.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:18.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:18.990 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:31:18.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.990 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:18.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:18.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:18.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:18.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:18.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:18.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:18.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:18.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:18.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:18.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:31:18.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:31:18.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:18.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:31:18.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:31:18.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:18.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:31:18.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:31:18.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:31:18.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:31:18.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:18.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:31:18.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:31:18.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:18.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:31:18.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:31:18.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:18.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:31:18.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:31:18.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:18.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:18.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:18.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:31:18.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:31:18.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.997 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:18.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:18.998 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:19.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:31:19.001 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:19.001 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:19.002 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:31:19.002 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:31:19.002 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:31:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:31:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:31:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:31:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:31:19.003 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:31:19.004 INFO:tasks.rados.rados.0.smithi090.stdout:741: done (0 left) 2023-01-15T14:31:19.004 INFO:tasks.rados.rados.0.smithi090.stdout:742: copy_from oid 6 from oid 24 current snap is 76 2023-01-15T14:31:19.004 INFO:tasks.rados.rados.0.smithi090.stdout:743: snap_create 2023-01-15T14:31:19.004 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:19.005 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:31:19.005 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:18.995+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:19.101 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:19.098+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:19.118 INFO:tasks.rados.rados.0.smithi090.stdout:742: finishing copy_from to smithi09025264-6 2023-01-15T14:31:19.119 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 232 (ObjNum 266 snap 66 seq_num 266) dirty exists 2023-01-15T14:31:19.119 INFO:tasks.rados.rados.0.smithi090.stdout:742: finishing copy_from racing read to smithi09025264-6 2023-01-15T14:31:19.119 INFO:tasks.rados.rados.0.smithi090.stdout:742: done (0 left) 2023-01-15T14:31:19.149 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:19.149 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:19.149 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:19.149 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:19.159 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:31:19.202 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:19.198+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:19.264 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:19.264 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:19.264 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:19.264 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:19.265 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:19.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:31:19.302 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:19.299+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:19.386 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:19.386 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:19.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.296203432, 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.387 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:31:19.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.388 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:19.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:19.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:19.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:19.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:31:19.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:31:19.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:31:19.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:31:19.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:31:19.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:31:19.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:31:19.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:31:19.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:31:19.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:31:19.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:31:19.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:31:19.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:19.396 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.295903635, 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:19.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:31:19.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:31:19.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:31:19.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:31:19.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:31:19.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:31:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:31:19.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:31:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:31:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:31:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:31:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:31:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:31:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:31:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:31:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:31:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:31:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:31:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:31:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.29587385100001, 2023-01-15T14:31:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:31:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.458 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:31:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:31:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:31:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:31:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:31:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:31:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:31:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:31:19.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:31:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:31:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:31:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:31:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:31:19.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:31:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:31:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:31:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:31:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.464 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:182 3.4 3:2022fdfa:::smithi09025264-46:head [write 1933572~791670] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.167011782, 2023-01-15T14:31:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.582304865, 2023-01-15T14:31:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.465 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 182 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213704+0000", 2023-01-15T14:31:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214574+0000", 2023-01-15T14:31:19.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087037799999999999 2023-01-15T14:31:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214608+0000", 2023-01-15T14:31:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:31:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214627+0000", 2023-01-15T14:31:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:31:19.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491537+0000", 2023-01-15T14:31:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27691020300000002 2023-01-15T14:31:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491721+0000", 2023-01-15T14:31:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345500000000001 2023-01-15T14:31:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693760+0000", 2023-01-15T14:31:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202039252 2023-01-15T14:31:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693898+0000", 2023-01-15T14:31:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013789 2023-01-15T14:31:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694230+0000", 2023-01-15T14:31:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220899999999998 2023-01-15T14:31:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694866+0000", 2023-01-15T14:31:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063640599999999995 2023-01-15T14:31:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694887+0000", 2023-01-15T14:31:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0622999999999999e-05 2023-01-15T14:31:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695950+0000", 2023-01-15T14:31:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001062688 2023-01-15T14:31:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696014+0000", 2023-01-15T14:31:19.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4697999999999996e-05 2023-01-15T14:31:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795898+0000", 2023-01-15T14:31:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099883338000000002 2023-01-15T14:31:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795922+0000", 2023-01-15T14:31:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4326000000000001e-05 2023-01-15T14:31:19.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795955+0000", 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3191000000000001e-05 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796011+0000", 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5631999999999999e-05 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.166065051, 2023-01-15T14:31:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:31:19.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:31:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:19.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:31:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:31:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:31:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:31:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:31:19.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:31:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:31:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:31:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:31:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:31:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:31:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:31:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:31:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:31:19.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:31:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:31:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:31:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:31:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:31:19.484 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.16600835400001, 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:31:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:31:19.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:31:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:31:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:31:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:31:19.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:31:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:31:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:31:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:31:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:31:19.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:31:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:31:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:31:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:31:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:31:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:31:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:31:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 169.54951218100001, 2023-01-15T14:31:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:31:19.492 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:31:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:31:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:31:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:31:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:31:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:31:19.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:31:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:31:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:31:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:31:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:31:19.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:19.498 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 162.52066371000001, 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:31:19.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:31:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:31:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:31:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:31:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:31:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:31:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:31:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:31:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:31:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:31:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:31:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:31:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:31:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:31:19.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:31:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:31:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:31:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:31:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:31:19.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:31:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:31:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:31:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:31:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:31:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:31:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:373 3.4 3:2029fa71:::smithi09025264-39:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 159.30241650299999, 2023-01-15T14:31:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58298356399999995, 2023-01-15T14:31:19.508 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-01-15T14:31:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:19.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078299+0000", 2023-01-15T14:31:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:19.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078644+0000", 2023-01-15T14:31:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034494099999999998 2023-01-15T14:31:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078676+0000", 2023-01-15T14:31:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1183000000000003e-05 2023-01-15T14:31:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078694+0000", 2023-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-05 2023-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078748+0000", 2023-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3634999999999997e-05 2023-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078753+0000", 2023-01-15T14:31:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6370000000000004e-06 2023-01-15T14:31:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110841+0000", 2023-01-15T14:31:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032087439000000002 2023-01-15T14:31:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110967+0000", 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126029 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.457874+0000", 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.346907725 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458066+0000", 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000191592 2023-01-15T14:31:19.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458321+0000", 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025454200000000002 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458918+0000", 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059740800000000001 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458945+0000", 2023-01-15T14:31:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-05 2023-01-15T14:31:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459253+0000", 2023-01-15T14:31:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030836200000000001 2023-01-15T14:31:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460943+0000", 2023-01-15T14:31:19.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016903580000000001 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461340+0000", 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039665799999999998 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661129+0000", 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19978933500000001 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661188+0000", 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8687000000000002e-05 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661285+0000", 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779000000000003e-05 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:375 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (56) in=7b,truncate 3142876] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:31:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 159.29761282499999, 2023-01-15T14:31:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57851114599999998, 2023-01-15T14:31:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.519 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 375 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083104+0000", 2023-01-15T14:31:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083113+0000", 2023-01-15T14:31:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6500000000000008e-06 2023-01-15T14:31:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083130+0000", 2023-01-15T14:31:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2023-01-15T14:31:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083142+0000", 2023-01-15T14:31:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2252999999999999e-05 2023-01-15T14:31:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083173+0000", 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0474999999999999e-05 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083175+0000", 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183469+0000", 2023-01-15T14:31:19.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10029313099999999 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183747+0000", 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000277998 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460165+0000", 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27641872899999997 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460342+0000", 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176792 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460579+0000", 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000237266 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460833+0000", 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025390900000000002 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460853+0000", 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9273e-05 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461024+0000", 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017111900000000001 2023-01-15T14:31:19.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461230+0000", 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020672600000000001 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461879+0000", 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064846999999999995 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661524+0000", 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19964509599999999 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661560+0000", 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6264999999999999e-05 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661616+0000", 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5952000000000002e-05 2023-01-15T14:31:19.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:376 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 159.297530569, 2023-01-15T14:31:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57864278800000002, 2023-01-15T14:31:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:19.529 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 376 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083186+0000", 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083193+0000", 2023-01-15T14:31:19.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083211+0000", 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7609999999999999e-05 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083223+0000", 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140000000000001e-05 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093534+0000", 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010311529 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093551+0000", 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.666e-05 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183834+0000", 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090283063999999996 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.184076+0000", 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024241900000000001 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461001+0000", 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27692504299999998 2023-01-15T14:31:19.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461152+0000", 2023-01-15T14:31:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150345 2023-01-15T14:31:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661642+0000", 2023-01-15T14:31:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200489742 2023-01-15T14:31:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661720+0000", 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8604999999999997e-05 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661830+0000", 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010992099999999999 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.51937673399999, 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:31:19.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:31:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:31:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:31:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:31:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:31:19.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:31:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:31:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:31:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:31:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.458798483999999, 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:31:19.545 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:31:19.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:31:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:31:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:31:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:31:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:31:19.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:31:19.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:31:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:31:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:31:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:31:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:31:19.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.553 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.458739773999994, 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.554 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:31:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:19.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:31:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:31:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:31:19.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:31:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:31:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:31:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:31:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:31:19.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:31:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:31:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:31:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:31:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.560 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.502320280999996, 2023-01-15T14:31:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:31:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:31:19.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:31:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:31:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:31:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:31:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:31:19.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:31:19.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:31:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.444822598000002, 2023-01-15T14:31:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:31:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.567 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:31:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:31:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:31:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:31:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:31:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:31:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:31:19.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:31:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:31:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:31:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:31:19.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:31:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:31:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:31:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:31:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:31:19.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:31:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:31:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:31:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:31:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:31:19.576 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.076171092999999, 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.577 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:19.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:31:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:31:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:31:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:31:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:31:19.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:31:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:31:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:31:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:31:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:31:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:31:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:31:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:31:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:31:19.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:19.585 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.076068452000001, 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:31:19.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:31:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:31:19.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:31:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:31:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:31:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:31:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:31:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:31:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:31:19.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:31:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:31:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:31:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:31:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:31:19.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:19.621 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.475086457, 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:19.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:31:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:31:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:31:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:31:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:19.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:31:19.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:31:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:31:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:31:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:31:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:31:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:31:19.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:31:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:31:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:31:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:31:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:31:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:31:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:31:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:31:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:31:19.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:31:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.475061032999999, 2023-01-15T14:31:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:31:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:19.633 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:19.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:31:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:31:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:31:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:31:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:31:19.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:31:19.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:19.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:31:19.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:31:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:31:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:31:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:19.643 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:19.643 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:19.645 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:31:19.645 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:19.639+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:19.745 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:19.742+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:19.802 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:19.802 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:19.803 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:19.803 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:19.813 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:31:19.846 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:19.843+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:19.897 INFO:tasks.rados.rados.0.smithi090.stdout:743: done (0 left) 2023-01-15T14:31:20.025 INFO:tasks.rados.rados.0.smithi090.stdout:744: write (excl) oid 13 current snap is 77 2023-01-15T14:31:20.025 INFO:tasks.rados.rados.0.smithi090.stdout:744: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2023-01-15T14:31:20.026 INFO:tasks.rados.rados.0.smithi090.stdout:744: writing smithi09025264-13 from 490837 to 1209710 tid 1 2023-01-15T14:31:20.026 INFO:tasks.rados.rados.0.smithi090.stdout:744: writing smithi09025264-13 from 1862260 to 2341687 tid 2 2023-01-15T14:31:20.026 INFO:tasks.rados.rados.0.smithi090.stdout:744: writing smithi09025264-13 from 2875032 to 2890837 tid 3 2023-01-15T14:31:20.027 INFO:tasks.rados.rados.0.smithi090.stdout:745: rollback oid 12 current snap is 77 2023-01-15T14:31:20.027 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 12 to 69 2023-01-15T14:31:20.027 INFO:tasks.rados.rados.0.smithi090.stdout:746: delete oid 49 current snap is 77 2023-01-15T14:31:20.027 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:20.028 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:20.028 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:20.028 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:20.028 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:20.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:31:20.029 INFO:tasks.rados.rados.0.smithi090.stdout:745: finishing rollback tid 1 to smithi09025264-12 2023-01-15T14:31:20.029 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:20.023+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:20.128 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:20.123+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:20.186 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:20.186 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:20.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:20.186 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:20.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.186 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:38 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'3, mlcod=0'0)", 2023-01-15T14:31:20.186 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:20.187 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 174.06776643399999, 2023-01-15T14:31:20.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30191175999999997, 2023-01-15T14:31:20.187 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.187 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.187 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:20.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:20.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112976+0000", 2023-01-15T14:31:20.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:20.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113186+0000", 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020979700000000001 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113228+0000", 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2150999999999997e-05 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213313+0000", 2023-01-15T14:31:20.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100084943 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313517+0000", 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10020435699999999 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313576+0000", 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783999999999998e-05 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313976+0000", 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039960499999999998 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314004+0000", 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7954e-05 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314175+0000", 2023-01-15T14:31:20.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171557 2023-01-15T14:31:20.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414853+0000", 2023-01-15T14:31:20.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100677875 2023-01-15T14:31:20.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414890+0000", 2023-01-15T14:31:20.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6803e-05 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:46 3.0 e23/20 3:03b54904:::smithi09025264-12:head v 23'5, mlcod=23'1)", 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 173.96687051999999, 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202592668, 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.194 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:31:20.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:31:20.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213873+0000", 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214477+0000", 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060481300000000005 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214532+0000", 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5096999999999998e-05 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214555+0000", 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.234e-05 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414409+0000", 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19985461199999999 2023-01-15T14:31:20.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414442+0000", 2023-01-15T14:31:20.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2987000000000001e-05 2023-01-15T14:31:20.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414813+0000", 2023-01-15T14:31:20.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037061899999999998 2023-01-15T14:31:20.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414832+0000", 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9380000000000001e-05 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415093+0000", 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026032099999999999 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415154+0000", 2023-01-15T14:31:20.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1538999999999995e-05 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.416467+0000", 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001312613 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:125 3.0 e70/55 3:0b54190a:::smithi09025264-17:25 v 70'136, mlcod=68'130)", 2023-01-15T14:31:20.200 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 124.237784409, 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30171926799999998, 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942958+0000", 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942972+0000", 2023-01-15T14:31:20.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3386e-05 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942990+0000", 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8113000000000001e-05 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.943002+0000", 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141942+0000", 2023-01-15T14:31:20.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19893962300000001 2023-01-15T14:31:20.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141978+0000", 2023-01-15T14:31:20.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6383e-05 2023-01-15T14:31:20.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142670+0000", 2023-01-15T14:31:20.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069211900000000005 2023-01-15T14:31:20.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142688+0000", 2023-01-15T14:31:20.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7839999999999999e-05 2023-01-15T14:31:20.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142768+0000", 2023-01-15T14:31:20.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0210999999999995e-05 2023-01-15T14:31:20.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244581+0000", 2023-01-15T14:31:20.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101813406 2023-01-15T14:31:20.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244680+0000", 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8045000000000001e-05 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(3.0 REQUEST e99)", 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.184330766000002, 2023-01-15T14:31:20.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26432840200000002, 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:20.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996419+0000", 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6729999999999997e-06 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996449+0000", 2023-01-15T14:31:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.004e-05 2023-01-15T14:31:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996463+0000", 2023-01-15T14:31:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3607e-05 2023-01-15T14:31:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996594+0000", 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013148899999999999 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996618+0000", 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3249000000000001e-05 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260645+0000", 2023-01-15T14:31:20.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26402768700000001 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260671+0000", 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.546e-05 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260742+0000", 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1569999999999994e-05 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.234 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:261 3.8 e98/41 3:1e54a7f5:::smithi09025264-42:3e v 98'160, mlcod=97'157)", 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.123263176999998, 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20299468400000001, 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057480+0000", 2023-01-15T14:31:20.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057496+0000", 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5869999999999999e-05 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057528+0000", 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057545+0000", 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.694e-05 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.158971+0000", 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101425425 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159011+0000", 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0058000000000002e-05 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159858+0000", 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084767900000000005 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159877+0000", 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9133000000000001e-05 2023-01-15T14:31:20.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159959+0000", 2023-01-15T14:31:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1693000000000005e-05 2023-01-15T14:31:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260430+0000", 2023-01-15T14:31:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10047039200000001 2023-01-15T14:31:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260476+0000", 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6652e-05 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.122042020999999, 2023-01-15T14:31:20.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:31:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:31:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:31:20.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:31:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:31:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:31:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:31:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:31:20.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:31:20.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:31:20.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:31:20.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:31:20.247 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.247 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:194 3.4 e101/48 3:2022fdfa:::smithi09025264-46:43 v 101'104, mlcod=98'100)", 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.336000409999997, 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20670733199999999, 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844743+0000", 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844757+0000", 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.484e-05 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844787+0000", 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9113000000000001e-05 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.944923+0000", 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013656 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.949975+0000", 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050521400000000001 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950015+0000", 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9286999999999998e-05 2023-01-15T14:31:20.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950559+0000", 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054463500000000002 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950585+0000", 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6166e-05 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950670+0000", 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4902999999999997e-05 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051368+0000", 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100697913 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051452+0000", 2023-01-15T14:31:20.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3442000000000005e-05 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1224 3.4 e107/48 3:2029fa71:::smithi09025264-39:head v 107'120, mlcod=104'116)", 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.94897177, 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22601005900000001, 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.254 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231771+0000", 2023-01-15T14:31:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233829+0000", 2023-01-15T14:31:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002057836 2023-01-15T14:31:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233881+0000", 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1856e-05 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.333996+0000", 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100115315 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434199+0000", 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100203373 2023-01-15T14:31:20.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434236+0000", 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6563000000000002e-05 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.435979+0000", 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017426570000000001 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.436036+0000", 2023-01-15T14:31:20.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7716000000000001e-05 2023-01-15T14:31:20.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437140+0000", 2023-01-15T14:31:20.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011036399999999999 2023-01-15T14:31:20.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437251+0000", 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110869 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.457783+0000", 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020532181 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.260 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1649 3.4 3:2029fa71:::smithi09025264-39:head [write 534522~634550] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.893806+0000", 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.286938693000003, 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205815042, 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1649 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.893806+0000", 2023-01-15T14:31:20.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.893806+0000", 2023-01-15T14:31:20.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.893803+0000", 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894135+0000", 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033239899999999999 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894168+0000", 2023-01-15T14:31:20.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2490000000000002e-05 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894195+0000", 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7576000000000002e-05 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894997+0000", 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080147300000000003 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895010+0000", 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611000000000001e-05 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.991782+0000", 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096771757 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.992255+0000", 2023-01-15T14:31:20.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047275099999999999 2023-01-15T14:31:20.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:20.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.992633+0000", 2023-01-15T14:31:20.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037850299999999998 2023-01-15T14:31:20.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.995684+0000", 2023-01-15T14:31:20.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030508610000000002 2023-01-15T14:31:20.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.995715+0000", 2023-01-15T14:31:20.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0759000000000001e-05 2023-01-15T14:31:20.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.996025+0000", 2023-01-15T14:31:20.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003102 2023-01-15T14:31:20.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.996334+0000", 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030889699999999998 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098878+0000", 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10254395299999999 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099030+0000", 2023-01-15T14:31:20.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152192 2023-01-15T14:31:20.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099198+0000", 2023-01-15T14:31:20.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016808400000000001 2023-01-15T14:31:20.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099621+0000", 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042263899999999999 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1650 3.4 3:2029fa71:::smithi09025264-39:head [write 1812335~322187] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894235+0000", 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.286509285000001, 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20642070600000001, 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.270 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.271 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.271 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.271 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.271 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1650 2023-01-15T14:31:20.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.271 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894235+0000", 2023-01-15T14:31:20.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894235+0000", 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894234+0000", 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894404+0000", 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017082499999999999 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894426+0000", 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1387e-05 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894440+0000", 2023-01-15T14:31:20.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3753e-05 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895046+0000", 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060607599999999999 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895054+0000", 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6130000000000004e-06 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.096846+0000", 2023-01-15T14:31:20.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201791888 2023-01-15T14:31:20.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.097087+0000", 2023-01-15T14:31:20.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024123800000000001 2023-01-15T14:31:20.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.097486+0000", 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039807900000000002 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098073+0000", 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058732299999999995 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098097+0000", 2023-01-15T14:31:20.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3773000000000001e-05 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098309+0000", 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021264999999999999 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098383+0000", 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3819999999999995e-05 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099809+0000", 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014256360000000001 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100004+0000", 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019514099999999999 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100220+0000", 2023-01-15T14:31:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021621800000000001 2023-01-15T14:31:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100656+0000", 2023-01-15T14:31:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000436036 2023-01-15T14:31:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.286284631000001, 2023-01-15T14:31:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:31:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:31:20.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:31:20.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:31:20.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:31:20.288 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.286233688000003, 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.289 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:31:20.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:31:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:31:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:31:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:31:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:31:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:31:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:31:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:31:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:31:20.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.295 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:31:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.345334446999999, 2023-01-15T14:31:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:31:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.296 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:31:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:31:20.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:31:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:31:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:31:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:31:20.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:31:20.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:31:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:31:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:31:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:31:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:31:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:31:20.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:31:20.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:31:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:31:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:31:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:31:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:31:20.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:31:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:31:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:20.397 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.271560274000002, 2023-01-15T14:31:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:31:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:31:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:31:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:31:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:31:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:31:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:31:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:31:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:31:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:31:20.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:31:20.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:31:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:31:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:31:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:31:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1712 3.4 3:2022fdfa:::smithi09025264-46:head [write 1880802~421790] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.266421583000003, 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30396041800000001, 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.407 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1712 2023-01-15T14:31:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914322+0000", 2023-01-15T14:31:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914561+0000", 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023884700000000001 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914572+0000", 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0499999999999999e-05 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914576+0000", 2023-01-15T14:31:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6820000000000002e-06 2023-01-15T14:31:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014941+0000", 2023-01-15T14:31:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100364994 2023-01-15T14:31:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014959+0000", 2023-01-15T14:31:20.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7416000000000001e-05 2023-01-15T14:31:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018171+0000", 2023-01-15T14:31:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032121239999999998 2023-01-15T14:31:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018333+0000", 2023-01-15T14:31:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016180900000000001 2023-01-15T14:31:20.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018581+0000", 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024773699999999999 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018897+0000", 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031622099999999999 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018907+0000", 2023-01-15T14:31:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0061000000000001e-05 2023-01-15T14:31:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.019108+0000", 2023-01-15T14:31:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020080099999999999 2023-01-15T14:31:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.020057+0000", 2023-01-15T14:31:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094954499999999999 2023-01-15T14:31:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150472+0000", 2023-01-15T14:31:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.130414958 2023-01-15T14:31:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218188+0000", 2023-01-15T14:31:20.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067715656999999999 2023-01-15T14:31:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218242+0000", 2023-01-15T14:31:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4651999999999998e-05 2023-01-15T14:31:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218284+0000", 2023-01-15T14:31:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1084999999999997e-05 2023-01-15T14:31:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.18 157/147 [PushOp(3:1eaad226:::smithi09025264-49:69, version: 139'201, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:1eaad226:::smithi09025264-49:69@139'201, size: 3916031, copy_subset: [0~3916031], clone_subset: {}, snapset: 69=[]:{68=[67],69=[69]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:31:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 35.471511030999999, 2023-01-15T14:31:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27553999000000001, 2023-01-15T14:31:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709232+0000", 2023-01-15T14:31:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709949+0000", 2023-01-15T14:31:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071723000000000002 2023-01-15T14:31:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709990+0000", 2023-01-15T14:31:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.015e-05 2023-01-15T14:31:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.810156+0000", 2023-01-15T14:31:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10016643 2023-01-15T14:31:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984513+0000", 2023-01-15T14:31:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.174357128 2023-01-15T14:31:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984522+0000", 2023-01-15T14:31:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0350000000000007e-06 2023-01-15T14:31:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984774+0000", 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025144700000000002 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.377604854000001, 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:31:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:31:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:31:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:31:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:31:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:31:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:31:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:31:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:31:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:31:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:31:20.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:31:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:31:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1820 3.1f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:20.428 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.277303368999998, 2023-01-15T14:31:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306504938, 2023-01-15T14:31:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1820 2023-01-15T14:31:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903440+0000", 2023-01-15T14:31:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903448+0000", 2023-01-15T14:31:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9430000000000006e-06 2023-01-15T14:31:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903469+0000", 2023-01-15T14:31:20.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0305e-05 2023-01-15T14:31:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903481+0000", 2023-01-15T14:31:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:31:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903616+0000", 2023-01-15T14:31:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013477199999999999 2023-01-15T14:31:20.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903643+0000", 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7098999999999999e-05 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209081+0000", 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30543790300000001 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209213+0000", 2023-01-15T14:31:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013155400000000001 2023-01-15T14:31:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209600+0000", 2023-01-15T14:31:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038751699999999998 2023-01-15T14:31:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209946+0000", 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034601 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1906 3.1f 3:f8edd34e:::smithi09025264-19:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:13.885370+0000", 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.2953749370000001, 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20444454000000001, 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.435 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1906 2023-01-15T14:31:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885370+0000", 2023-01-15T14:31:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885370+0000", 2023-01-15T14:31:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885367+0000", 2023-01-15T14:31:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885380+0000", 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2413e-05 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885406+0000", 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6542000000000001e-05 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885418+0000", 2023-01-15T14:31:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1219999999999999e-05 2023-01-15T14:31:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885507+0000", 2023-01-15T14:31:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9738e-05 2023-01-15T14:31:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885782+0000", 2023-01-15T14:31:20.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027431999999999999 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.886171+0000", 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038963000000000002 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.887617+0000", 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001445986 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.887634+0000", 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6633000000000002e-05 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.887686+0000", 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2346999999999999e-05 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.887721+0000", 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4427000000000001e-05 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.089581+0000", 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201860013 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.089642+0000", 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0859000000000001e-05 2023-01-15T14:31:20.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.089711+0000", 2023-01-15T14:31:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9648999999999997e-05 2023-01-15T14:31:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.089814+0000", 2023-01-15T14:31:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010310900000000001 2023-01-15T14:31:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.443 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1912 3.4 e189/181 3:2022fdfa:::smithi09025264-46:head v 189'194, mlcod=180'190)", 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:15.986052+0000", 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.194693011, 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22134975400000001, 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:15.986052+0000", 2023-01-15T14:31:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:15.986052+0000", 2023-01-15T14:31:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:15.986051+0000", 2023-01-15T14:31:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:15.986063+0000", 2023-01-15T14:31:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2472e-05 2023-01-15T14:31:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:15.986095+0000", 2023-01-15T14:31:20.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1613999999999999e-05 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.086194+0000", 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009912999999999 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.190517+0000", 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104323034 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.190558+0000", 2023-01-15T14:31:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0535999999999999e-05 2023-01-15T14:31:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.191286+0000", 2023-01-15T14:31:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072818899999999996 2023-01-15T14:31:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.191347+0000", 2023-01-15T14:31:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0748999999999997e-05 2023-01-15T14:31:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.191417+0000", 2023-01-15T14:31:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0908999999999995e-05 2023-01-15T14:31:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.192443+0000", 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010255570000000001 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.207401+0000", 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014958447 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.450 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:20.452 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:31:20.453 INFO:tasks.rados.rados.0.smithi090.stdout:744: finishing write tid 1 to smithi09025264-13 2023-01-15T14:31:20.453 INFO:tasks.rados.rados.0.smithi090.stdout:745: finishing rollback tid 2 to smithi09025264-12 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 238 (ObjNum 185 snap 40 seq_num 185) dirty exists 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:744: finishing write tid 2 to smithi09025264-13 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:744: finishing write tid 3 to smithi09025264-13 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:744: finishing write tid 4 to smithi09025264-13 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:745: done (2 left) 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:746: done (1 left) 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:747: delete oid 38 current snap is 77 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:744: finishing write tid 5 to smithi09025264-13 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:744: oid 13 updating version 0 to 222 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:744: oid 13 version 222 is already newer than 220 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:744: oid 13 version 222 is already newer than 221 2023-01-15T14:31:20.454 INFO:tasks.rados.rados.0.smithi090.stdout:744: oid 13 version 222 is already newer than 219 2023-01-15T14:31:20.455 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 222 (ObjNum 291 snap 77 seq_num 291) dirty exists 2023-01-15T14:31:20.455 INFO:tasks.rados.rados.0.smithi090.stdout:744: left oid 13 (ObjNum 291 snap 77 seq_num 291) 2023-01-15T14:31:20.455 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:20.446+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:20.553 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:20.547+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:20.569 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:20.569 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:20.570 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:20.570 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:20.579 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:31:20.609 INFO:tasks.rados.rados.0.smithi090.stdout:744: done (1 left) 2023-01-15T14:31:20.609 INFO:tasks.rados.rados.0.smithi090.stdout:747: done (0 left) 2023-01-15T14:31:20.609 INFO:tasks.rados.rados.0.smithi090.stdout:748: read oid 5 snap -1 2023-01-15T14:31:20.609 INFO:tasks.rados.rados.0.smithi090.stdout:748: expect (ObjNum 248 snap 58 seq_num 248) 2023-01-15T14:31:20.609 INFO:tasks.rados.rados.0.smithi090.stdout:749: read oid 48 snap 71 2023-01-15T14:31:20.609 INFO:tasks.rados.rados.0.smithi090.stdout:749: expect (ObjNum 260 snap 60 seq_num 260) 2023-01-15T14:31:20.610 INFO:tasks.rados.rados.0.smithi090.stdout:750: write oid 41 current snap is 77 2023-01-15T14:31:20.610 INFO:tasks.rados.rados.0.smithi090.stdout:750: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2023-01-15T14:31:20.621 INFO:tasks.rados.rados.0.smithi090.stdout:750: writing smithi09025264-41 from 541258 to 1085878 tid 1 2023-01-15T14:31:20.628 INFO:tasks.rados.rados.0.smithi090.stdout:750: writing smithi09025264-41 from 1637706 to 2264460 tid 2 2023-01-15T14:31:20.632 INFO:tasks.rados.rados.0.smithi090.stdout:750: writing smithi09025264-41 from 2840948 to 2941258 tid 3 2023-01-15T14:31:20.632 INFO:tasks.rados.rados.0.smithi090.stdout:751: snap_remove snap 69 2023-01-15T14:31:20.653 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:20.650+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:20.683 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:20.684 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:20.684 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:20.684 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:20.684 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:20.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:31:20.754 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:20.750+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:20.804 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:166 3.8 3:1e54a7f5:::smithi09025264-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:20.804 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 173.684114577, 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62664632300000001, 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 166 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114713+0000", 2023-01-15T14:31:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115178+0000", 2023-01-15T14:31:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046493199999999999 2023-01-15T14:31:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115208+0000", 2023-01-15T14:31:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0689999999999999e-05 2023-01-15T14:31:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115221+0000", 2023-01-15T14:31:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3013000000000001e-05 2023-01-15T14:31:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115334+0000", 2023-01-15T14:31:20.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112403 2023-01-15T14:31:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115505+0000", 2023-01-15T14:31:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170915 2023-01-15T14:31:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333496+0000", 2023-01-15T14:31:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217991667 2023-01-15T14:31:20.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333606+0000", 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109967 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333796+0000", 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018928099999999999 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334289+0000", 2023-01-15T14:31:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049322600000000004 2023-01-15T14:31:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334339+0000", 2023-01-15T14:31:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0439999999999998e-05 2023-01-15T14:31:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334990+0000", 2023-01-15T14:31:20.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065082300000000003 2023-01-15T14:31:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335100+0000", 2023-01-15T14:31:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109357 2023-01-15T14:31:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.640854+0000", 2023-01-15T14:31:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30575451300000001 2023-01-15T14:31:20.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741190+0000", 2023-01-15T14:31:20.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100335568 2023-01-15T14:31:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741263+0000", 2023-01-15T14:31:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3275999999999994e-05 2023-01-15T14:31:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741361+0000", 2023-01-15T14:31:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8435000000000002e-05 2023-01-15T14:31:20.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:167 3.8 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 173.68354988799999, 2023-01-15T14:31:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62638390099999997, 2023-01-15T14:31:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 167 2023-01-15T14:31:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115279+0000", 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115289+0000", 2023-01-15T14:31:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0509999999999999e-05 2023-01-15T14:31:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115306+0000", 2023-01-15T14:31:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7260000000000001e-05 2023-01-15T14:31:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115337+0000", 2023-01-15T14:31:20.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0664000000000003e-05 2023-01-15T14:31:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115579+0000", 2023-01-15T14:31:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024194400000000001 2023-01-15T14:31:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115744+0000", 2023-01-15T14:31:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016493700000000001 2023-01-15T14:31:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334392+0000", 2023-01-15T14:31:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21864830299999999 2023-01-15T14:31:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334453+0000", 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1047e-05 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334601+0000", 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014824700000000001 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334812+0000", 2023-01-15T14:31:20.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210208 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334996+0000", 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183878 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335126+0000", 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013001399999999999 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335160+0000", 2023-01-15T14:31:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4876999999999998e-05 2023-01-15T14:31:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741044+0000", 2023-01-15T14:31:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40588326000000002 2023-01-15T14:31:20.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741453+0000", 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040963099999999998 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741548+0000", 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4961000000000001e-05 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741664+0000", 2023-01-15T14:31:20.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115333 2023-01-15T14:31:20.825 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.825 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.825 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.825 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:20.825 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:20.825 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 173.683457774, 2023-01-15T14:31:20.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:31:20.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.826 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:20.826 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.826 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.826 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.826 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:31:20.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.826 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:31:20.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:31:20.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:31:20.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:31:20.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:31:20.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:31:20.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:31:20.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:31:20.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:31:20.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:31:20.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:31:20.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:31:20.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:31:20.832 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.832 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.832 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 173.67046569499999, 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.833 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:31:20.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:20.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:31:20.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:31:20.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:31:20.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:31:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:31:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:31:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:31:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:31:20.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:31:20.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:31:20.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:31:20.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:31:20.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:31:20.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:31:20.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:31:20.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:31:20.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:31:20.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:31:20.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:31:20.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:31:20.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:31:20.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:31:20.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:31:20.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:31:20.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:31:20.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:31:20.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:31:20.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:31:20.842 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.842 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.842 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.842 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 173.67006235900001, 2023-01-15T14:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.843 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:31:20.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:20.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:31:20.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:31:20.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:31:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:31:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:31:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:31:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:31:20.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:31:20.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:20.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:31:20.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:31:20.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:31:20.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:31:20.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:31:20.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:31:20.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:31:20.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:31:20.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:31:20.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:31:20.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:31:20.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:31:20.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:31:20.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:31:20.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:31:20.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:31:20.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:31:20.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:31:20.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:31:20.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:31:20.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:31:20.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 173.56983873900001, 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:31:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:31:20.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:31:20.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:31:20.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:31:20.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:31:20.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:31:20.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:31:20.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:31:20.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:20.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:31:20.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:31:20.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:31:20.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:31:20.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:20.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:31:20.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:31:20.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:31:20.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:31:20.875 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.875 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.875 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.875 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:194 3.8 3:1eaad226:::smithi09025264-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:20.875 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:20.875 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 173.497437496, 2023-01-15T14:31:20.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64372577200000003, 2023-01-15T14:31:20.875 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 194 2023-01-15T14:31:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301390+0000", 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301991+0000", 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060122599999999995 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.302026+0000", 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5040000000000003e-05 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402147+0000", 2023-01-15T14:31:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100121084 2023-01-15T14:31:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402219+0000", 2023-01-15T14:31:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2045999999999994e-05 2023-01-15T14:31:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402315+0000", 2023-01-15T14:31:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5561000000000002e-05 2023-01-15T14:31:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842189+0000", 2023-01-15T14:31:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.439874129 2023-01-15T14:31:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842296+0000", 2023-01-15T14:31:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107021 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842486+0000", 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018965799999999999 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842839+0000", 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035330700000000002 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842859+0000", 2023-01-15T14:31:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0347e-05 2023-01-15T14:31:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843212+0000", 2023-01-15T14:31:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352534 2023-01-15T14:31:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843835+0000", 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062338599999999997 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944861+0000", 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10102596899999999 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944908+0000", 2023-01-15T14:31:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6773000000000001e-05 2023-01-15T14:31:20.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:20.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945039+0000", 2023-01-15T14:31:20.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130684 2023-01-15T14:31:20.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945118+0000", 2023-01-15T14:31:20.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9194000000000001e-05 2023-01-15T14:31:20.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.885 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.885 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:705 3.2 3:47dfe935:::smithi09025264-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2023-01-15T14:31:20.885 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:20.885 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 131.688245588, 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61170208599999998, 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 705 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110583+0000", 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:20.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110587+0000", 2023-01-15T14:31:20.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9689999999999996e-06 2023-01-15T14:31:20.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110601+0000", 2023-01-15T14:31:20.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.393e-05 2023-01-15T14:31:20.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110611+0000", 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3999999999999998e-06 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319174+0000", 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20856376800000001 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:20.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319367+0000", 2023-01-15T14:31:20.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192361 2023-01-15T14:31:20.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.721941+0000", 2023-01-15T14:31:20.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40257461300000003 2023-01-15T14:31:20.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:20.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722108+0000", 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167054 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722286+0000", 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017772099999999999 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.891 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.876523513999999, 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:31:20.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:31:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:31:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.870062160000003, 2023-01-15T14:31:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:31:20.897 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:31:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:31:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:31:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:31:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:31:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:31:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:31:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:31:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:31:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:31:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:31:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:31:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:31:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:31:20.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.769640234999997, 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:20.903 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.904 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.904 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.904 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:31:20.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.904 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:20.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:31:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:31:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:31:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:31:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:31:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:31:20.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:31:20.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:31:20.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:31:20.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:31:20.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:31:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:31:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:31:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:31:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:31:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.769302749000005, 2023-01-15T14:31:20.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:31:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:31:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:20.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:20.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:31:20.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:20.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:31:20.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:31:20.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:31:20.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:31:20.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:31:20.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:31:20.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:31:20.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:31:20.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:31:20.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:31:20.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:31:20.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:31:20.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:31:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:31:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.769246852999999, 2023-01-15T14:31:20.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:31:20.916 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.916 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:20.916 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.916 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.916 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.916 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:31:20.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.916 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:20.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:31:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:31:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:31:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:31:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:20.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:31:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:31:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:31:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:31:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:20.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1314 3.18 3.afe52a78 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 81.438951721999999, 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62560938700000002, 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:20.921 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1314 2023-01-15T14:31:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:20.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359877+0000", 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:20.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:20.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359882+0000", 2023-01-15T14:31:20.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0050000000000004e-06 2023-01-15T14:31:20.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:20.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:20.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359893+0000", 2023-01-15T14:31:20.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0676000000000001e-05 2023-01-15T14:31:20.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:20.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359904+0000", 2023-01-15T14:31:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0788000000000001e-05 2023-01-15T14:31:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460094+0000", 2023-01-15T14:31:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10019024999999999 2023-01-15T14:31:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460106+0000", 2023-01-15T14:31:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354999999999999e-05 2023-01-15T14:31:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985477+0000", 2023-01-15T14:31:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52537039100000005 2023-01-15T14:31:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985487+0000", 2023-01-15T14:31:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:31:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1335 3.8 3:17d5d741:::smithi09025264-5:head [write 3252761~175867] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:21.039 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.866932461000005, 2023-01-15T14:31:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66052243499999996, 2023-01-15T14:31:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1335 2023-01-15T14:31:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:21.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931896+0000", 2023-01-15T14:31:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931983+0000", 2023-01-15T14:31:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6470000000000004e-05 2023-01-15T14:31:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:21.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931995+0000", 2023-01-15T14:31:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2075e-05 2023-01-15T14:31:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932003+0000", 2023-01-15T14:31:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5059999999999995e-06 2023-01-15T14:31:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932346+0000", 2023-01-15T14:31:21.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034280299999999998 2023-01-15T14:31:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932357+0000", 2023-01-15T14:31:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.13e-05 2023-01-15T14:31:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189006+0000", 2023-01-15T14:31:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25664859899999998 2023-01-15T14:31:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:21.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189336+0000", 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033030899999999998 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189481+0000", 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014484400000000001 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189768+0000", 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028704300000000001 2023-01-15T14:31:21.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189787+0000", 2023-01-15T14:31:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9343000000000001e-05 2023-01-15T14:31:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:21.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191361+0000", 2023-01-15T14:31:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015734029999999999 2023-01-15T14:31:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191469+0000", 2023-01-15T14:31:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010859899999999999 2023-01-15T14:31:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:21.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391274+0000", 2023-01-15T14:31:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19980450899999999 2023-01-15T14:31:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592067+0000", 2023-01-15T14:31:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20079332 2023-01-15T14:31:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592190+0000", 2023-01-15T14:31:21.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012291400000000001 2023-01-15T14:31:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592420+0000", 2023-01-15T14:31:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000229371 2023-01-15T14:31:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.050 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1336 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (56) in=7b,truncate 3428628] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.866815372000005, 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66113910799999998, 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:21.051 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1336 2023-01-15T14:31:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932013+0000", 2023-01-15T14:31:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932019+0000", 2023-01-15T14:31:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5380000000000002e-06 2023-01-15T14:31:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932027+0000", 2023-01-15T14:31:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5150000000000006e-06 2023-01-15T14:31:21.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932034+0000", 2023-01-15T14:31:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2023-01-15T14:31:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032497+0000", 2023-01-15T14:31:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100463001 2023-01-15T14:31:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032518+0000", 2023-01-15T14:31:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1359999999999999e-05 2023-01-15T14:31:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189826+0000", 2023-01-15T14:31:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15730834299999999 2023-01-15T14:31:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189951+0000", 2023-01-15T14:31:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124583 2023-01-15T14:31:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190093+0000", 2023-01-15T14:31:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014221399999999999 2023-01-15T14:31:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190253+0000", 2023-01-15T14:31:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015953000000000001 2023-01-15T14:31:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191365+0000", 2023-01-15T14:31:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011123190000000001 2023-01-15T14:31:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:21.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191439+0000", 2023-01-15T14:31:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3648999999999999e-05 2023-01-15T14:31:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191491+0000", 2023-01-15T14:31:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2444999999999997e-05 2023-01-15T14:31:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391472+0000", 2023-01-15T14:31:21.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199981193 2023-01-15T14:31:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592619+0000", 2023-01-15T14:31:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20114642499999999 2023-01-15T14:31:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592748+0000", 2023-01-15T14:31:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012950200000000001 2023-01-15T14:31:21.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593153+0000", 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040503700000000002 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1337 3.8 3:17d5d741:::smithi09025264-5:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:21.062 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.866787142999996, 2023-01-15T14:31:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66179168200000005, 2023-01-15T14:31:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1337 2023-01-15T14:31:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:21.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932046+0000", 2023-01-15T14:31:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8089999999999999e-06 2023-01-15T14:31:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:21.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932055+0000", 2023-01-15T14:31:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2199999999999998e-06 2023-01-15T14:31:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932061+0000", 2023-01-15T14:31:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2779999999999999e-06 2023-01-15T14:31:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190318+0000", 2023-01-15T14:31:21.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.258256967 2023-01-15T14:31:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190413+0000", 2023-01-15T14:31:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4627000000000001e-05 2023-01-15T14:31:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593288+0000", 2023-01-15T14:31:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.402875339 2023-01-15T14:31:21.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593515+0000", 2023-01-15T14:31:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022700699999999999 2023-01-15T14:31:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593834+0000", 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031906799999999999 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1690 3.16 3:68fb3263:::smithi09025264-33:head [write 1763227~472137] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.936314173, 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72570748100000004, 2023-01-15T14:31:21.069 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1690 2023-01-15T14:31:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:21.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862514+0000", 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862746+0000", 2023-01-15T14:31:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000232301 2023-01-15T14:31:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862770+0000", 2023-01-15T14:31:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3286e-05 2023-01-15T14:31:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862779+0000", 2023-01-15T14:31:21.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4199999999999996e-06 2023-01-15T14:31:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862844+0000", 2023-01-15T14:31:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5036999999999994e-05 2023-01-15T14:31:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862860+0000", 2023-01-15T14:31:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6426000000000002e-05 2023-01-15T14:31:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051251+0000", 2023-01-15T14:31:21.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18839023299999999 2023-01-15T14:31:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051466+0000", 2023-01-15T14:31:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021523199999999999 2023-01-15T14:31:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368326+0000", 2023-01-15T14:31:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31686002699999999 2023-01-15T14:31:21.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368559+0000", 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023305400000000001 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368935+0000", 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037579099999999999 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369531+0000", 2023-01-15T14:31:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059593199999999997 2023-01-15T14:31:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369571+0000", 2023-01-15T14:31:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0649999999999999e-05 2023-01-15T14:31:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370331+0000", 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075969899999999996 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371230+0000", 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000899134 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:21.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472895+0000", 2023-01-15T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101665191 2023-01-15T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.587914+0000", 2023-01-15T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115018671 2023-01-15T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588013+0000", 2023-01-15T14:31:21.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8494999999999998e-05 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588223+0000", 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210294 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1691 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 2235364] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:21.132 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:21.133 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.936029918999999, 2023-01-15T14:31:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72593841699999995, 2023-01-15T14:31:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1691 2023-01-15T14:31:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862799+0000", 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862807+0000", 2023-01-15T14:31:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1280000000000008e-06 2023-01-15T14:31:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862821+0000", 2023-01-15T14:31:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.345e-05 2023-01-15T14:31:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862826+0000", 2023-01-15T14:31:21.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3720000000000004e-06 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862883+0000", 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6589000000000003e-05 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862888+0000", 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4060000000000004e-06 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151601+0000", 2023-01-15T14:31:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28871354100000002 2023-01-15T14:31:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151732+0000", 2023-01-15T14:31:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013032300000000001 2023-01-15T14:31:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369729+0000", 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217997515 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369949+0000", 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219664 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370367+0000", 2023-01-15T14:31:21.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041822799999999998 2023-01-15T14:31:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370771+0000", 2023-01-15T14:31:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040358400000000002 2023-01-15T14:31:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370794+0000", 2023-01-15T14:31:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-05 2023-01-15T14:31:21.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370880+0000", 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6020999999999995e-05 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371241+0000", 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000360956 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472994+0000", 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101753089 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588348+0000", 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11535382299999999 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588539+0000", 2023-01-15T14:31:21.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019090399999999999 2023-01-15T14:31:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588738+0000", 2023-01-15T14:31:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019910800000000001 2023-01-15T14:31:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.209 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.935996435, 2023-01-15T14:31:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:31:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:31:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.210 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:31:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:31:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:31:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:31:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:31:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:31:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:31:21.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:21.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:21.217 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:21.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:31:21.218 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:31:21.220 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:31:21.220 INFO:tasks.rados.rados.0.smithi090.stdout:751: done (3 left) 2023-01-15T14:31:21.221 INFO:tasks.rados.rados.0.smithi090.stdout:752: snap_create 2023-01-15T14:31:21.221 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:31:21.221 INFO:tasks.rados.rados.0.smithi090.stdout:750: finishing write tid 1 to smithi09025264-41 2023-01-15T14:31:21.221 INFO:tasks.rados.rados.0.smithi090.stdout:750: finishing write tid 2 to smithi09025264-41 2023-01-15T14:31:21.221 INFO:tasks.rados.rados.0.smithi090.stdout:750: finishing write tid 3 to smithi09025264-41 2023-01-15T14:31:21.221 INFO:tasks.rados.rados.0.smithi090.stdout:750: finishing write tid 4 to smithi09025264-41 2023-01-15T14:31:21.221 INFO:tasks.rados.rados.0.smithi090.stdout:750: finishing write tid 5 to smithi09025264-41 2023-01-15T14:31:21.221 INFO:tasks.rados.rados.0.smithi090.stdout:750: oid 41 updating version 0 to 236 2023-01-15T14:31:21.221 INFO:tasks.rados.rados.0.smithi090.stdout:750: oid 41 version 236 is already newer than 235 2023-01-15T14:31:21.221 INFO:tasks.rados.rados.0.smithi090.stdout:750: oid 41 version 236 is already newer than 233 2023-01-15T14:31:21.222 INFO:tasks.rados.rados.0.smithi090.stdout:750: oid 41 version 236 is already newer than 234 2023-01-15T14:31:21.222 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 236 (ObjNum 294 snap 77 seq_num 294) dirty exists 2023-01-15T14:31:21.222 INFO:tasks.rados.rados.0.smithi090.stdout:750: left oid 41 (ObjNum 294 snap 77 seq_num 294) 2023-01-15T14:31:21.222 INFO:tasks.rados.rados.0.smithi090.stdout:750: done (2 left) 2023-01-15T14:31:21.222 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:21.222 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:21.215+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:21.319 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:21.315+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:21.349 INFO:tasks.rados.rados.0.smithi090.stdout:749: expect (ObjNum 260 snap 60 seq_num 260) 2023-01-15T14:31:21.349 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:21.350 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:31:21.350 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:31:21.350 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:21.350 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:31:21.368 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:21.421 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:21.418+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:21.509 INFO:tasks.rados.rados.0.smithi090.stdout:749: done (1 left) 2023-01-15T14:31:21.509 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:21.510 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:21.510 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:31:21.510 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:31:21.510 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:31:21.510 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:21.510 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:31:21.555 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:21.561 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:21.558+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:21.681 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:21.681 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:21.681 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:21.681 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:21.681 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:21.681 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:21.682 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:21.682 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:21.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:31:21.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:31:21.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:31:21.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:31:21.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:31:21.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:31:21.683 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:31:21.683 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:31:21.683 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:31:21.683 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:21.675+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:21.781 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:21.778+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:21.840 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:31:21.840 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:31:21.840 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:31:21.840 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:31:21.842 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:31:21.842 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:21.843 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:21.843 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:31:21.883 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:21.879+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:21.954 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:21.954 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:21.964 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:31:21.983 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:21.978+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:22.082 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:22.082 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:22.085 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:22.079+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:22.091 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:31:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:31:22.159 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:31:22.172 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2023-01-15T14:31:21.786828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":156006337,"num_objects":73,"num_object_clones":30,"num_object_copies":219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":6,"num_read":773,"num_read_kb":678678,"num_write":701,"num_write_kb":233161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":538,"num_bytes_recovered":359370248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5563,"ondisk_log_size":5563,"up":150,"acting":150,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":5428292,"kb_used_data":556796308,"kb_used_omap":23478,"kb_used_meta":0,"kb_avail":556767988,"statfs":{"total":575688990720,"available":570130419712,"internally_reserved":28999680,"allocated":570159419392,"data_stored":570159419392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":24042284,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":49,"commit_latency_ns":0,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9351207,"num_objects":4,"num_object_clones":5,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":-10,"num_read_kb":-18105,"num_write":6,"num_write_kb":5070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-31,"num_bytes_recovered":-15862107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.992790"},"pg_stats":[{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:31:19.886177+0000","last_change":"2023-01-15T14:31:19.886177+0000","last_active":"2023-01-15T14:31:19.886177+0000","last_peered":"2023-01-15T14:31:19.886177+0000","last_clean":"2023-01-15T14:31:19.886177+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:31:19.886177+0000","last_undegraded":"2023-01-15T14:31:19.886177+0000","last_fullsized":"2023-01-15T14:31:19.886177+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:19.886177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:31:19.886177+0000","last_clean_scrub_stamp":"2023-01-15T14:31:19.886177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"141'46","reported_seq":565,"reported_epoch":193,"state":"unknown","last_fresh":"2023-01-15T14:31:19.898532+0000","last_change":"2023-01-15T14:31:19.898459+0000","last_active":"2023-01-15T14:31:19.898459+0000","last_peered":"2023-01-15T14:31:18.991410+0000","last_clean":"2023-01-15T14:31:18.991410+0000","last_became_active":"2023-01-15T14:31:02.991074+0000","last_became_peered":"2023-01-15T14:31:02.991074+0000","last_unstale":"2023-01-15T14:31:19.898532+0000","last_undegraded":"2023-01-15T14:31:19.898532+0000","last_fullsized":"2023-01-15T14:31:19.898532+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":504,"reported_epoch":194,"state":"peering","last_fresh":"2023-01-15T14:31:21.020649+0000","last_change":"2023-01-15T14:31:21.020649+0000","last_active":"2023-01-15T14:31:01.781002+0000","last_peered":"2023-01-15T14:31:01.045759+0000","last_clean":"2023-01-15T14:31:01.045759+0000","last_became_active":"2023-01-15T14:30:45.089409+0000","last_became_peered":"2023-01-15T14:30:45.089409+0000","last_unstale":"2023-01-15T14:31:21.020649+0000","last_undegraded":"2023-01-15T14:31:21.020649+0000","last_fullsized":"2023-01-15T14:31:21.020649+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":865561,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3347,"num_write":6,"num_write_kb":2818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:31:19.886177+0000","last_change":"2023-01-15T14:31:19.886177+0000","last_active":"2023-01-15T14:31:19.886177+0000","last_peered":"2023-01-15T14:31:19.886177+0000","last_clean":"2023-01-15T14:31:19.886177+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:31:19.886177+0000","last_undegraded":"2023-01-15T14:31:19.886177+0000","last_fullsized":"2023-01-15T14:31:19.886177+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:19.886177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:31:19.886177+0000","last_clean_scrub_stamp":"2023-01-15T14:31:19.886177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:31:19.886177+0000","last_change":"2023-01-15T14:31:19.886177+0000","last_active":"2023-01-15T14:31:19.886177+0000","last_peered":"2023-01-15T14:31:19.886177+0000","last_clean":"2023-01-15T14:31:19.886177+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:31:19.886177+0000","last_undegraded":"2023-01-15T14:31:19.886177+0000","last_fullsized":"2023-01-15T14:31:19.886177+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:19.886177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:31:19.886177+0000","last_clean_scrub_stamp":"2023-01-15T14:31:19.886177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"179'204","reported_seq":865,"reported_epoch":193,"state":"unknown","last_fresh":"2023-01-15T14:31:19.898018+0000","last_change":"2023-01-15T14:31:19.897975+0000","last_active":"2023-01-15T14:31:19.897975+0000","last_peered":"2023-01-15T14:31:19.000534+0000","last_clean":"2023-01-15T14:31:19.000534+0000","last_became_active":"2023-01-15T14:31:03.288830+0000","last_became_peered":"2023-01-15T14:31:03.288830+0000","last_unstale":"2023-01-15T14:31:19.898018+0000","last_undegraded":"2023-01-15T14:31:19.898018+0000","last_fullsized":"2023-01-15T14:31:19.898018+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2159716,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":2,"num_write_kb":3981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":7874096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":619,"reported_epoch":193,"state":"unknown","last_fresh":"2023-01-15T14:31:19.902084+0000","last_change":"2023-01-15T14:31:19.902044+0000","last_active":"2023-01-15T14:31:19.902044+0000","last_peered":"2023-01-15T14:31:18.994400+0000","last_clean":"2023-01-15T14:31:18.994400+0000","last_became_active":"2023-01-15T14:31:02.994339+0000","last_became_peered":"2023-01-15T14:31:02.994339+0000","last_unstale":"2023-01-15T14:31:19.902084+0000","last_undegraded":"2023-01-15T14:31:19.902084+0000","last_fullsized":"2023-01-15T14:31:19.902084+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2485558,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":8458,"num_write":2,"num_write_kb":737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1731121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:31:19.886177+0000","last_change":"2023-01-15T14:31:19.886177+0000","last_active":"2023-01-15T14:31:19.886177+0000","last_peered":"2023-01-15T14:31:19.886177+0000","last_clean":"2023-01-15T14:31:19.886177+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:31:19.886177+0000","last_undegraded":"2023-01-15T14:31:19.886177+0000","last_fullsized":"2023-01-15T14:31:19.886177+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:19.886177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:31:19.886177+0000","last_clean_scrub_stamp":"2023-01-15T14:31:19.886177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"192'73","reported_seq":630,"reported_epoch":194,"state":"peering","last_fresh":"2023-01-15T14:31:21.021023+0000","last_change":"2023-01-15T14:31:21.019503+0000","last_active":"2023-01-15T14:31:21.018620+0000","last_peered":"2023-01-15T14:31:19.001178+0000","last_clean":"2023-01-15T14:31:19.001178+0000","last_became_active":"2023-01-15T14:30:51.138174+0000","last_became_peered":"2023-01-15T14:30:51.138174+0000","last_unstale":"2023-01-15T14:31:21.021023+0000","last_undegraded":"2023-01-15T14:31:21.021023+0000","last_fullsized":"2023-01-15T14:31:21.021023+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2778701,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":15020,"num_write":12,"num_write_kb":5359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9491720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:31:19.886177+0000","last_change":"2023-01-15T14:31:19.886177+0000","last_active":"2023-01-15T14:31:19.886177+0000","last_peered":"2023-01-15T14:31:19.886177+0000","last_clean":"2023-01-15T14:31:19.886177+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:31:19.886177+0000","last_undegraded":"2023-01-15T14:31:19.886177+0000","last_fullsized":"2023-01-15T14:31:19.886177+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:19.886177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:31:19.886177+0000","last_clean_scrub_stamp":"2023-01-15T14:31:19.886177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"149'233","reported_seq":756,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.100542+0000","last_change":"2023-01-15T14:31:15.992990+0000","last_active":"2023-01-15T14:31:19.100542+0000","last_peered":"2023-01-15T14:31:19.100542+0000","last_clean":"2023-01-15T14:31:19.100542+0000","last_became_active":"2023-01-15T14:31:06.029716+0000","last_became_peered":"2023-01-15T14:31:06.029716+0000","last_unstale":"2023-01-15T14:31:19.100542+0000","last_undegraded":"2023-01-15T14:31:19.100542+0000","last_fullsized":"2023-01-15T14:31:19.100542+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.107855534,"stat_sum":{"num_bytes":7604272,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":47,"num_read_kb":43637,"num_write":37,"num_write_kb":13160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2469494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.22","version":"181'107","reported_seq":815,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.000402+0000","last_change":"2023-01-15T14:31:15.987554+0000","last_active":"2023-01-15T14:31:19.000402+0000","last_peered":"2023-01-15T14:31:19.000402+0000","last_clean":"2023-01-15T14:31:19.000402+0000","last_became_active":"2023-01-15T14:31:08.084483+0000","last_became_peered":"2023-01-15T14:31:08.084483+0000","last_unstale":"2023-01-15T14:31:19.000402+0000","last_undegraded":"2023-01-15T14:31:19.000402+0000","last_fullsized":"2023-01-15T14:31:19.000402+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.10600380299999999,"stat_sum":{"num_bytes":2518991,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":12156,"num_write":8,"num_write_kb":2164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":11208309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.21","version":"168'124","reported_seq":686,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-01-15T14:31:20.923997+0000","last_change":"2023-01-15T14:31:20.927980+0000","last_active":"2023-01-15T14:31:20.923997+0000","last_peered":"2023-01-15T14:31:20.923997+0000","last_clean":"2023-01-15T14:31:20.923997+0000","last_became_active":"2023-01-15T14:30:52.226939+0000","last_became_peered":"2023-01-15T14:30:52.226939+0000","last_unstale":"2023-01-15T14:31:20.923997+0000","last_undegraded":"2023-01-15T14:31:20.923997+0000","last_fullsized":"2023-01-15T14:31:20.923997+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"168'124","last_scrub_stamp":"2023-01-15T14:31:11.414546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.414546+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:23.043","scrub_duration":0.20479377300000001,"objects_trimmed":0,"snaptrim_duration":0.0044579930000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.20","version":"181'236","reported_seq":914,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-01-15T14:31:16.081552+0000","last_change":"2023-01-15T14:31:16.183111+0000","last_active":"2023-01-15T14:31:16.081552+0000","last_peered":"2023-01-15T14:31:16.081552+0000","last_clean":"2023-01-15T14:31:16.081552+0000","last_became_active":"2023-01-15T14:31:07.906191+0000","last_became_peered":"2023-01-15T14:31:07.906191+0000","last_unstale":"2023-01-15T14:31:16.081552+0000","last_undegraded":"2023-01-15T14:31:16.081552+0000","last_fullsized":"2023-01-15T14:31:16.081552+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.20220396299999999,"stat_sum":{"num_bytes":4659379,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9657,"num_write":7,"num_write_kb":2065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8877936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.1f","version":"194'155","reported_seq":807,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:31:20.926850+0000","last_change":"2023-01-15T14:31:20.926850+0000","last_active":"2023-01-15T14:31:20.926850+0000","last_peered":"2023-01-15T14:31:20.926850+0000","last_clean":"2023-01-15T14:31:20.926850+0000","last_became_active":"2023-01-15T14:31:07.960304+0000","last_became_peered":"2023-01-15T14:31:07.960304+0000","last_unstale":"2023-01-15T14:31:20.926850+0000","last_undegraded":"2023-01-15T14:31:20.926850+0000","last_fullsized":"2023-01-15T14:31:20.926850+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:15.241","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70535094099999995,"stat_sum":{"num_bytes":3000066,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":64,"num_read_kb":91950,"num_write":39,"num_write_kb":12831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":36580502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.1e","version":"139'38","reported_seq":520,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.000694+0000","last_change":"2023-01-15T14:31:15.991987+0000","last_active":"2023-01-15T14:31:19.000694+0000","last_peered":"2023-01-15T14:31:19.000694+0000","last_clean":"2023-01-15T14:31:19.000694+0000","last_became_active":"2023-01-15T14:30:51.134845+0000","last_became_peered":"2023-01-15T14:30:51.134845+0000","last_unstale":"2023-01-15T14:31:19.000694+0000","last_undegraded":"2023-01-15T14:31:19.000694+0000","last_fullsized":"2023-01-15T14:31:19.000694+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'37","last_scrub_stamp":"2023-01-15T14:30:25.793922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:25.793922+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.107643237,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":11530,"num_write":6,"num_write_kb":4046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1487420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.1d","version":"189'171","reported_seq":796,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-01-15T14:31:16.790551+0000","last_change":"2023-01-15T14:31:16.792335+0000","last_active":"2023-01-15T14:31:16.790551+0000","last_peered":"2023-01-15T14:31:16.790551+0000","last_clean":"2023-01-15T14:31:16.790551+0000","last_became_active":"2023-01-15T14:30:54.085528+0000","last_became_peered":"2023-01-15T14:30:54.085528+0000","last_unstale":"2023-01-15T14:31:16.790551+0000","last_undegraded":"2023-01-15T14:31:16.790551+0000","last_fullsized":"2023-01-15T14:31:16.790551+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062386840000000004,"stat_sum":{"num_bytes":10574961,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":81712,"num_write":48,"num_write_kb":17879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":13857289,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":87,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-01-15T14:31:15.991230+0000","last_change":"2023-01-15T14:31:15.991230+0000","last_active":"2023-01-15T14:31:15.991230+0000","last_peered":"2023-01-15T14:31:15.991230+0000","last_clean":"2023-01-15T14:31:15.991230+0000","last_became_active":"2023-01-15T14:30:51.234588+0000","last_became_peered":"2023-01-15T14:30:51.234588+0000","last_unstale":"2023-01-15T14:31:15.991230+0000","last_undegraded":"2023-01-15T14:31:15.991230+0000","last_fullsized":"2023-01-15T14:31:15.991230+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.0024925659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":10,"num_write_kb":2522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1159667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.1b","version":"155'53","reported_seq":513,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-01-15T14:31:15.993607+0000","last_change":"2023-01-15T14:31:15.993607+0000","last_active":"2023-01-15T14:31:15.993607+0000","last_peered":"2023-01-15T14:31:15.993607+0000","last_clean":"2023-01-15T14:31:15.993607+0000","last_became_active":"2023-01-15T14:31:02.994801+0000","last_became_peered":"2023-01-15T14:31:02.994801+0000","last_unstale":"2023-01-15T14:31:15.993607+0000","last_undegraded":"2023-01-15T14:31:15.993607+0000","last_fullsized":"2023-01-15T14:31:15.993607+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046473870000000002,"stat_sum":{"num_bytes":1731121,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":6694,"num_write":12,"num_write_kb":5635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3462242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.1a","version":"189'196","reported_seq":838,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-01-15T14:31:16.425593+0000","last_change":"2023-01-15T14:31:15.991676+0000","last_active":"2023-01-15T14:31:16.425593+0000","last_peered":"2023-01-15T14:31:16.425593+0000","last_clean":"2023-01-15T14:31:16.425593+0000","last_became_active":"2023-01-15T14:31:13.049050+0000","last_became_peered":"2023-01-15T14:31:13.049050+0000","last_unstale":"2023-01-15T14:31:16.425593+0000","last_undegraded":"2023-01-15T14:31:16.425593+0000","last_fullsized":"2023-01-15T14:31:16.425593+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034543640000000001,"stat_sum":{"num_bytes":8804416,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":52,"num_read_kb":53146,"num_write":59,"num_write_kb":25710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":56061752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.19","version":"194'180","reported_seq":783,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-01-15T14:31:21.631521+0000","last_change":"2023-01-15T14:31:21.633150+0000","last_active":"2023-01-15T14:31:21.631521+0000","last_peered":"2023-01-15T14:31:21.631521+0000","last_clean":"2023-01-15T14:31:21.631521+0000","last_became_active":"2023-01-15T14:30:59.222826+0000","last_became_peered":"2023-01-15T14:30:59.222826+0000","last_unstale":"2023-01-15T14:31:21.631521+0000","last_undegraded":"2023-01-15T14:31:21.631521+0000","last_fullsized":"2023-01-15T14:31:21.631521+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'157","last_scrub_stamp":"2023-01-15T14:30:26.792736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:26.792736+0000","objects_scrubbed":4,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:38.155","scrub_duration":0.61998826900000004,"objects_trimmed":0,"snaptrim_duration":0.71602835300000001,"stat_sum":{"num_bytes":15306127,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":27,"num_read_kb":35342,"num_write":12,"num_write_kb":4509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":29479511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.18","version":"193'227","reported_seq":941,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.897247+0000","last_change":"2023-01-15T14:31:15.881353+0000","last_active":"2023-01-15T14:31:19.897247+0000","last_peered":"2023-01-15T14:31:19.897247+0000","last_clean":"2023-01-15T14:31:19.897247+0000","last_became_active":"2023-01-15T14:31:04.108363+0000","last_became_peered":"2023-01-15T14:31:04.108363+0000","last_unstale":"2023-01-15T14:31:19.897247+0000","last_undegraded":"2023-01-15T14:31:19.897247+0000","last_fullsized":"2023-01-15T14:31:19.897247+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.0014788259999999999,"stat_sum":{"num_bytes":9573635,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":23,"num_read_kb":18217,"num_write":20,"num_write_kb":8697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":22066072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.17","version":"155'77","reported_seq":572,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-01-15T14:31:18.994441+0000","last_change":"2023-01-15T14:31:15.994688+0000","last_active":"2023-01-15T14:31:18.994441+0000","last_peered":"2023-01-15T14:31:18.994441+0000","last_clean":"2023-01-15T14:31:18.994441+0000","last_became_active":"2023-01-15T14:31:02.993865+0000","last_became_peered":"2023-01-15T14:31:02.993865+0000","last_unstale":"2023-01-15T14:31:18.994441+0000","last_undegraded":"2023-01-15T14:31:18.994441+0000","last_fullsized":"2023-01-15T14:31:18.994441+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'65","last_scrub_stamp":"2023-01-15T14:30:24.156798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.156798+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:34.293","scrub_duration":0.30368223799999999,"objects_trimmed":0,"snaptrim_duration":0.0097492510000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3949645,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.16","version":"194'81","reported_seq":127,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-01-15T14:31:21.530973+0000","last_change":"2023-01-15T14:31:21.532578+0000","last_active":"2023-01-15T14:31:21.530973+0000","last_peered":"2023-01-15T14:31:21.530973+0000","last_clean":"2023-01-15T14:31:21.530973+0000","last_became_active":"2023-01-15T14:30:51.036725+0000","last_became_peered":"2023-01-15T14:30:51.036725+0000","last_unstale":"2023-01-15T14:31:21.530973+0000","last_undegraded":"2023-01-15T14:31:21.530973+0000","last_fullsized":"2023-01-15T14:31:21.530973+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:36.231","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.611276712,"stat_sum":{"num_bytes":2235364,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":544,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.000967+0000","last_change":"2023-01-15T14:31:15.986805+0000","last_active":"2023-01-15T14:31:19.000967+0000","last_peered":"2023-01-15T14:31:19.000967+0000","last_clean":"2023-01-15T14:31:19.000967+0000","last_became_active":"2023-01-15T14:30:03.169969+0000","last_became_peered":"2023-01-15T14:30:03.169969+0000","last_unstale":"2023-01-15T14:31:19.000967+0000","last_undegraded":"2023-01-15T14:31:19.000967+0000","last_fullsized":"2023-01-15T14:31:19.000967+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:30:24.721082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.721082+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:48.096","scrub_duration":0.109000768,"objects_trimmed":0,"snaptrim_duration":0.10637988399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.14","version":"169'138","reported_seq":697,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-01-15T14:31:15.992989+0000","last_change":"2023-01-15T14:31:15.992989+0000","last_active":"2023-01-15T14:31:15.992989+0000","last_peered":"2023-01-15T14:31:15.992989+0000","last_clean":"2023-01-15T14:31:15.992989+0000","last_became_active":"2023-01-15T14:30:16.972329+0000","last_became_peered":"2023-01-15T14:30:16.972329+0000","last_unstale":"2023-01-15T14:31:15.992989+0000","last_undegraded":"2023-01-15T14:31:15.992989+0000","last_fullsized":"2023-01-15T14:31:15.992989+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:31.572","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.0040927469999999999,"stat_sum":{"num_bytes":2453855,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":11235,"num_write":5,"num_write_kb":1217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":219,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.902691+0000","last_change":"2023-01-15T14:30:51.228396+0000","last_active":"2023-01-15T14:31:19.902691+0000","last_peered":"2023-01-15T14:31:19.902691+0000","last_clean":"2023-01-15T14:31:19.902691+0000","last_became_active":"2023-01-15T14:30:51.127504+0000","last_became_peered":"2023-01-15T14:30:51.127504+0000","last_unstale":"2023-01-15T14:31:19.902691+0000","last_undegraded":"2023-01-15T14:31:19.902691+0000","last_fullsized":"2023-01-15T14:31:19.902691+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.671761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.671761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:31:23.347","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":576,"reported_epoch":194,"state":"active+clean","last_fresh":"2023-01-15T14:31:21.633406+0000","last_change":"2023-01-15T14:31:21.634563+0000","last_active":"2023-01-15T14:31:21.633406+0000","last_peered":"2023-01-15T14:31:21.633406+0000","last_clean":"2023-01-15T14:31:21.633406+0000","last_became_active":"2023-01-15T14:31:21.030929+0000","last_became_peered":"2023-01-15T14:31:21.030929+0000","last_unstale":"2023-01-15T14:31:21.633406+0000","last_undegraded":"2023-01-15T14:31:21.633406+0000","last_fullsized":"2023-01-15T14:31:21.633406+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:30:21.421951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:21.421951+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.569620185,"objects_trimmed":0,"snaptrim_duration":0.60308367900000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.905713+0000","last_change":"2023-01-15T14:30:51.035825+0000","last_active":"2023-01-15T14:31:19.905713+0000","last_peered":"2023-01-15T14:31:19.905713+0000","last_clean":"2023-01-15T14:31:19.905713+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:31:19.905713+0000","last_undegraded":"2023-01-15T14:31:19.905713+0000","last_fullsized":"2023-01-15T14:31:19.905713+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:31:22.068","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"191'79","reported_seq":620,"reported_epoch":193,"state":"peering","last_fresh":"2023-01-15T14:31:19.902876+0000","last_change":"2023-01-15T14:31:19.902876+0000","last_active":"2023-01-15T14:31:19.902044+0000","last_peered":"2023-01-15T14:31:18.994400+0000","last_clean":"2023-01-15T14:31:18.994400+0000","last_became_active":"2023-01-15T14:31:02.994339+0000","last_became_peered":"2023-01-15T14:31:02.994339+0000","last_unstale":"2023-01-15T14:31:19.902876+0000","last_undegraded":"2023-01-15T14:31:19.902876+0000","last_fullsized":"2023-01-15T14:31:19.902876+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.10751015799999999,"stat_sum":{"num_bytes":2485557,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":8457,"num_write":2,"num_write_kb":737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1731121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":215,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-01-15T14:31:15.089238+0000","last_change":"2023-01-15T14:31:15.088518+0000","last_active":"2023-01-15T14:31:15.089238+0000","last_peered":"2023-01-15T14:31:15.089238+0000","last_clean":"2023-01-15T14:31:15.089238+0000","last_became_active":"2023-01-15T14:31:03.088392+0000","last_became_peered":"2023-01-15T14:31:03.088392+0000","last_unstale":"2023-01-15T14:31:15.089238+0000","last_undegraded":"2023-01-15T14:31:15.089238+0000","last_fullsized":"2023-01-15T14:31:15.089238+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:15.088397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.088397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:32.351","scrub_duration":0.40518630900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"189'200","reported_seq":886,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:31:21.634609+0000","last_change":"2023-01-15T14:31:21.634670+0000","last_active":"2023-01-15T14:31:21.634609+0000","last_peered":"2023-01-15T14:31:21.634609+0000","last_clean":"2023-01-15T14:31:21.634609+0000","last_became_active":"2023-01-15T14:31:21.226293+0000","last_became_peered":"2023-01-15T14:31:21.226293+0000","last_unstale":"2023-01-15T14:31:21.634609+0000","last_undegraded":"2023-01-15T14:31:21.634609+0000","last_fullsized":"2023-01-15T14:31:21.634609+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.102625073,"stat_sum":{"num_bytes":4345839,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":18109,"num_write":16,"num_write_kb":4789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":15862107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.908257+0000","last_change":"2023-01-15T14:30:51.237821+0000","last_active":"2023-01-15T14:31:19.908257+0000","last_peered":"2023-01-15T14:31:19.908257+0000","last_clean":"2023-01-15T14:31:19.908257+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:31:19.908257+0000","last_undegraded":"2023-01-15T14:31:19.908257+0000","last_fullsized":"2023-01-15T14:31:19.908257+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:31:21.077","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"192'73","reported_seq":631,"reported_epoch":193,"state":"peering","last_fresh":"2023-01-15T14:31:19.904092+0000","last_change":"2023-01-15T14:31:19.904092+0000","last_active":"2023-01-15T14:31:19.903194+0000","last_peered":"2023-01-15T14:31:19.118544+0000","last_clean":"2023-01-15T14:31:19.118544+0000","last_became_active":"2023-01-15T14:30:51.138174+0000","last_became_peered":"2023-01-15T14:30:51.138174+0000","last_unstale":"2023-01-15T14:31:19.904092+0000","last_undegraded":"2023-01-15T14:31:19.904092+0000","last_fullsized":"2023-01-15T14:31:19.904092+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.106173789,"stat_sum":{"num_bytes":2778701,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15019,"num_write":12,"num_write_kb":5358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9491720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.3","version":"189'253","reported_seq":927,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-01-15T14:31:16.891519+0000","last_change":"2023-01-15T14:31:16.992917+0000","last_active":"2023-01-15T14:31:16.891519+0000","last_peered":"2023-01-15T14:31:16.891519+0000","last_clean":"2023-01-15T14:31:16.891519+0000","last_became_active":"2023-01-15T14:31:03.093753+0000","last_became_peered":"2023-01-15T14:31:03.093753+0000","last_unstale":"2023-01-15T14:31:16.891519+0000","last_undegraded":"2023-01-15T14:31:16.891519+0000","last_fullsized":"2023-01-15T14:31:16.891519+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'251","last_scrub_stamp":"2023-01-15T14:31:15.363400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.363400+0000","objects_scrubbed":3,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:42.073","scrub_duration":0.208136242,"objects_trimmed":0,"snaptrim_duration":0.106217222,"stat_sum":{"num_bytes":3311078,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-01-15T14:31:20.007514+0000","last_change":"2023-01-15T14:31:13.764052+0000","last_active":"2023-01-15T14:31:20.007514+0000","last_peered":"2023-01-15T14:31:20.007514+0000","last_clean":"2023-01-15T14:31:20.007514+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:31:20.007514+0000","last_undegraded":"2023-01-15T14:31:20.007514+0000","last_fullsized":"2023-01-15T14:31:20.007514+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:13.763945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:13.763945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:39.419","scrub_duration":0.58613919699999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"194'240","reported_seq":867,"reported_epoch":194,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:31:21.532743+0000","last_change":"2023-01-15T14:31:21.532743+0000","last_active":"2023-01-15T14:31:21.532743+0000","last_peered":"2023-01-15T14:31:21.532743+0000","last_clean":"2023-01-15T14:31:21.532743+0000","last_became_active":"2023-01-15T14:30:56.101261+0000","last_became_peered":"2023-01-15T14:30:56.101261+0000","last_unstale":"2023-01-15T14:31:21.532743+0000","last_undegraded":"2023-01-15T14:31:21.532743+0000","last_fullsized":"2023-01-15T14:31:21.532743+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:31.088","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.00077503299999999995,"stat_sum":{"num_bytes":6218590,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":195,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.902266+0000","last_change":"2023-01-15T14:29:47.854985+0000","last_active":"2023-01-15T14:31:19.902266+0000","last_peered":"2023-01-15T14:31:19.902266+0000","last_clean":"2023-01-15T14:31:19.902266+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:31:19.902266+0000","last_undegraded":"2023-01-15T14:31:19.902266+0000","last_fullsized":"2023-01-15T14:31:19.902266+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:47.854866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:47.854866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.63763962500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":619,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-01-15T14:31:15.996755+0000","last_change":"2023-01-15T14:31:15.996755+0000","last_active":"2023-01-15T14:31:15.996755+0000","last_peered":"2023-01-15T14:31:15.996755+0000","last_clean":"2023-01-15T14:31:15.996755+0000","last_became_active":"2023-01-15T14:31:06.028849+0000","last_became_peered":"2023-01-15T14:31:06.028849+0000","last_unstale":"2023-01-15T14:31:15.996755+0000","last_undegraded":"2023-01-15T14:31:15.996755+0000","last_fullsized":"2023-01-15T14:31:15.996755+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:30:47.349633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.349633+0000","objects_scrubbed":1,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.28528039199999999,"objects_trimmed":0,"snaptrim_duration":0.0066563689999999997,"stat_sum":{"num_bytes":1899191,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":224,"reported_epoch":188,"state":"active+clean","last_fresh":"2023-01-15T14:31:14.987493+0000","last_change":"2023-01-15T14:31:11.136609+0000","last_active":"2023-01-15T14:31:14.987493+0000","last_peered":"2023-01-15T14:31:14.987493+0000","last_clean":"2023-01-15T14:31:14.987493+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:31:14.987493+0000","last_undegraded":"2023-01-15T14:31:14.987493+0000","last_fullsized":"2023-01-15T14:31:14.987493+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:11.136490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.136490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:36.593","scrub_duration":0.39969354600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":718,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.101397+0000","last_change":"2023-01-15T14:31:15.989456+0000","last_active":"2023-01-15T14:31:19.101397+0000","last_peered":"2023-01-15T14:31:19.101397+0000","last_clean":"2023-01-15T14:31:19.101397+0000","last_became_active":"2023-01-15T14:30:54.184242+0000","last_became_peered":"2023-01-15T14:30:54.184242+0000","last_unstale":"2023-01-15T14:31:19.101397+0000","last_undegraded":"2023-01-15T14:31:19.101397+0000","last_fullsized":"2023-01-15T14:31:19.101397+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.106412838,"stat_sum":{"num_bytes":1084068,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3268978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":198,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.902295+0000","last_change":"2023-01-15T14:31:02.793835+0000","last_active":"2023-01-15T14:31:19.902295+0000","last_peered":"2023-01-15T14:31:19.902295+0000","last_clean":"2023-01-15T14:31:19.902295+0000","last_became_active":"2023-01-15T14:31:02.793531+0000","last_became_peered":"2023-01-15T14:31:02.793531+0000","last_unstale":"2023-01-15T14:31:19.902295+0000","last_undegraded":"2023-01-15T14:31:19.902295+0000","last_fullsized":"2023-01-15T14:31:19.902295+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:44.759745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:44.759745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":329,"reported_epoch":193,"state":"active+clean","last_fresh":"2023-01-15T14:31:20.006696+0000","last_change":"2023-01-15T14:31:03.091584+0000","last_active":"2023-01-15T14:31:20.006696+0000","last_peered":"2023-01-15T14:31:20.006696+0000","last_clean":"2023-01-15T14:31:20.006696+0000","last_became_active":"2023-01-15T14:31:03.091071+0000","last_became_peered":"2023-01-15T14:31:03.091071+0000","last_unstale":"2023-01-15T14:31:20.006696+0000","last_undegraded":"2023-01-15T14:31:20.006696+0000","last_fullsized":"2023-01-15T14:31:20.006696+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:29:45.876183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:29:45.876183+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"179'204","reported_seq":865,"reported_epoch":193,"state":"peering","last_fresh":"2023-01-15T14:31:19.898018+0000","last_change":"2023-01-15T14:31:19.899006+0000","last_active":"2023-01-15T14:31:19.897975+0000","last_peered":"2023-01-15T14:31:19.000534+0000","last_clean":"2023-01-15T14:31:19.000534+0000","last_became_active":"2023-01-15T14:31:03.288830+0000","last_became_peered":"2023-01-15T14:31:03.288830+0000","last_unstale":"2023-01-15T14:31:19.898018+0000","last_undegraded":"2023-01-15T14:31:19.898018+0000","last_fullsized":"2023-01-15T14:31:19.898018+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:50.574","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.0026066169999999999,"stat_sum":{"num_bytes":2159716,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":3981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":7874096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":652,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-01-15T14:31:16.792562+0000","last_change":"2023-01-15T14:31:16.794219+0000","last_active":"2023-01-15T14:31:16.792562+0000","last_peered":"2023-01-15T14:31:16.792562+0000","last_clean":"2023-01-15T14:31:16.792562+0000","last_became_active":"2023-01-15T14:29:58.963354+0000","last_became_peered":"2023-01-15T14:29:58.963354+0000","last_unstale":"2023-01-15T14:31:16.792562+0000","last_undegraded":"2023-01-15T14:31:16.792562+0000","last_fullsized":"2023-01-15T14:31:16.792562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:30:08.681869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:08.681869+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:19.959","scrub_duration":0.40398525099999999,"objects_trimmed":0,"snaptrim_duration":0.0084814030000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.a","version":"174'186","reported_seq":789,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-01-15T14:31:16.091222+0000","last_change":"2023-01-15T14:31:16.091222+0000","last_active":"2023-01-15T14:31:16.091222+0000","last_peered":"2023-01-15T14:31:16.091222+0000","last_clean":"2023-01-15T14:31:16.091222+0000","last_became_active":"2023-01-15T14:31:01.355624+0000","last_became_peered":"2023-01-15T14:31:01.355624+0000","last_unstale":"2023-01-15T14:31:16.091222+0000","last_undegraded":"2023-01-15T14:31:16.091222+0000","last_fullsized":"2023-01-15T14:31:16.091222+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023146339999999999,"stat_sum":{"num_bytes":4424360,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":23721,"num_write":51,"num_write_kb":24558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":37790905,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":551,"reported_epoch":193,"state":"peering","last_fresh":"2023-01-15T14:31:19.898863+0000","last_change":"2023-01-15T14:31:19.899411+0000","last_active":"2023-01-15T14:31:19.898819+0000","last_peered":"2023-01-15T14:31:19.000831+0000","last_clean":"2023-01-15T14:31:19.000831+0000","last_became_active":"2023-01-15T14:31:02.994679+0000","last_became_peered":"2023-01-15T14:31:02.994679+0000","last_unstale":"2023-01-15T14:31:19.898863+0000","last_undegraded":"2023-01-15T14:31:19.898863+0000","last_fullsized":"2023-01-15T14:31:19.898863+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.108807084,"stat_sum":{"num_bytes":865560,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3347,"num_write":5,"num_write_kb":2817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.c","version":"141'46","reported_seq":566,"reported_epoch":193,"state":"peering","last_fresh":"2023-01-15T14:31:19.899794+0000","last_change":"2023-01-15T14:31:19.899794+0000","last_active":"2023-01-15T14:31:19.898459+0000","last_peered":"2023-01-15T14:31:18.991410+0000","last_clean":"2023-01-15T14:31:18.991410+0000","last_became_active":"2023-01-15T14:31:02.991074+0000","last_became_peered":"2023-01-15T14:31:02.991074+0000","last_unstale":"2023-01-15T14:31:19.899794+0000","last_undegraded":"2023-01-15T14:31:19.899794+0000","last_fullsized":"2023-01-15T14:31:19.899794+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.106918217,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.d","version":"168'156","reported_seq":761,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-01-15T14:31:15.882567+0000","last_change":"2023-01-15T14:31:15.983621+0000","last_active":"2023-01-15T14:31:15.882567+0000","last_peered":"2023-01-15T14:31:15.882567+0000","last_clean":"2023-01-15T14:31:15.882567+0000","last_became_active":"2023-01-15T14:30:44.889199+0000","last_became_peered":"2023-01-15T14:30:44.889199+0000","last_unstale":"2023-01-15T14:31:15.882567+0000","last_undegraded":"2023-01-15T14:31:15.882567+0000","last_fullsized":"2023-01-15T14:31:15.882567+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:01.864","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.102458037,"stat_sum":{"num_bytes":6067327,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":10837,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":560,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-01-15T14:31:15.995574+0000","last_change":"2023-01-15T14:31:15.995574+0000","last_active":"2023-01-15T14:31:15.995574+0000","last_peered":"2023-01-15T14:31:15.995574+0000","last_clean":"2023-01-15T14:31:15.995574+0000","last_became_active":"2023-01-15T14:31:03.094506+0000","last_became_peered":"2023-01-15T14:31:03.094506+0000","last_unstale":"2023-01-15T14:31:15.995574+0000","last_undegraded":"2023-01-15T14:31:15.995574+0000","last_fullsized":"2023-01-15T14:31:15.995574+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'37","last_scrub_stamp":"2023-01-15T14:30:25.793922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:25.793922+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.0058998289999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":11529,"num_write":5,"num_write_kb":4046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1487419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":769,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-01-15T14:31:16.794369+0000","last_change":"2023-01-15T14:31:16.795120+0000","last_active":"2023-01-15T14:31:16.794369+0000","last_peered":"2023-01-15T14:31:16.794369+0000","last_clean":"2023-01-15T14:31:16.794369+0000","last_became_active":"2023-01-15T14:31:06.028460+0000","last_became_peered":"2023-01-15T14:31:06.028460+0000","last_unstale":"2023-01-15T14:31:16.794369+0000","last_undegraded":"2023-01-15T14:31:16.794369+0000","last_fullsized":"2023-01-15T14:31:16.794369+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0082030499999999999,"stat_sum":{"num_bytes":6227010,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":33587,"num_write":38,"num_write_kb":12831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":14899405,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.10","version":"191'239","reported_seq":881,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-01-15T14:31:18.999998+0000","last_change":"2023-01-15T14:31:16.080775+0000","last_active":"2023-01-15T14:31:18.999998+0000","last_peered":"2023-01-15T14:31:18.999998+0000","last_clean":"2023-01-15T14:31:18.999998+0000","last_became_active":"2023-01-15T14:31:11.135827+0000","last_became_peered":"2023-01-15T14:31:11.135827+0000","last_unstale":"2023-01-15T14:31:18.999998+0000","last_undegraded":"2023-01-15T14:31:18.999998+0000","last_fullsized":"2023-01-15T14:31:18.999998+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.00081317599999999998,"stat_sum":{"num_bytes":6487552,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":59,"num_read_kb":51893,"num_write":60,"num_write_kb":25114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":26680915,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.11","version":"186'144","reported_seq":784,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-01-15T14:31:15.881511+0000","last_change":"2023-01-15T14:31:15.883645+0000","last_active":"2023-01-15T14:31:15.881511+0000","last_peered":"2023-01-15T14:31:15.881511+0000","last_clean":"2023-01-15T14:31:15.881511+0000","last_became_active":"2023-01-15T14:31:06.888501+0000","last_became_peered":"2023-01-15T14:31:06.888501+0000","last_unstale":"2023-01-15T14:31:15.881511+0000","last_undegraded":"2023-01-15T14:31:15.881511+0000","last_fullsized":"2023-01-15T14:31:15.881511+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'144","last_scrub_stamp":"2023-01-15T14:31:13.806973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:13.806973+0000","objects_scrubbed":4,"log_size":144,"log_dups_size":0,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:29.247","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.0024732389999999999,"stat_sum":{"num_bytes":11412019,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":42898,"num_write":13,"num_write_kb":9600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":14420439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.12","version":"187'135","reported_seq":364,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-01-15T14:31:19.000993+0000","last_change":"2023-01-15T14:31:15.988415+0000","last_active":"2023-01-15T14:31:19.000993+0000","last_peered":"2023-01-15T14:31:19.000993+0000","last_clean":"2023-01-15T14:31:19.000993+0000","last_became_active":"2023-01-15T14:31:03.093394+0000","last_became_peered":"2023-01-15T14:31:03.093394+0000","last_unstale":"2023-01-15T14:31:19.000993+0000","last_undegraded":"2023-01-15T14:31:19.000993+0000","last_fullsized":"2023-01-15T14:31:19.000993+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"139'103","last_scrub_stamp":"2023-01-15T14:30:30.045615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:30.045615+0000","objects_scrubbed":3,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:36.166","scrub_duration":0.207128334,"objects_trimmed":0,"snaptrim_duration":0.10592916500000001,"stat_sum":{"num_bytes":8638831,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":27345,"num_write":11,"num_write_kb":4435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":12585229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.13","version":"180'260","reported_seq":797,"reported_epoch":189,"state":"active+clean","last_fresh":"2023-01-15T14:31:15.881404+0000","last_change":"2023-01-15T14:31:15.882353+0000","last_active":"2023-01-15T14:31:15.881404+0000","last_peered":"2023-01-15T14:31:15.881404+0000","last_clean":"2023-01-15T14:31:15.881404+0000","last_became_active":"2023-01-15T14:30:08.974612+0000","last_became_peered":"2023-01-15T14:30:08.974612+0000","last_unstale":"2023-01-15T14:31:15.881404+0000","last_undegraded":"2023-01-15T14:31:15.881404+0000","last_fullsized":"2023-01-15T14:31:15.881404+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:29.544","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.001192165,"stat_sum":{"num_bytes":10258847,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":34,"num_read_kb":29731,"num_write":21,"num_write_kb":11650,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":155415969,"num_objects":71,"num_object_clones":30,"num_object_copies":213,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":6,"num_read":657,"num_read_kb":678578,"num_write":542,"num_write_kb":230745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":357008776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5446,"ondisk_log_size":5446,"up":123,"acting":123,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":843536,"kb_used_data":92857220,"kb_used_omap":2245,"kb_used_meta":0,"kb_avail":92855844,"statfs":{"total":95948165120,"available":95084384256,"internally_reserved":1409024,"allocated":95085793280,"data_stored":95085793280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2299536,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":14.18,"5min":5.4340000000000002,"15min":4.524},"min":{"1min":0.57099999999999995,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":148.10900000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.753,"5min":7.5119999999999996,"15min":8.6430000000000007},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0569999999999999,"5min":73.769000000000005,"15min":73.769000000000005},"last":0.70399999999999996}]},{"osd":1,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":5.3419999999999996,"15min":6.1440000000000001},"min":{"1min":0.47699999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":88.003,"15min":88.003},"last":0.92200000000000004},{"interface":"front","average":{"1min":6.2370000000000001,"5min":6.0810000000000004,"15min":6.2130000000000001},"min":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":97.704999999999998,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.82999999999999996}]},{"osd":2,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":5.8250000000000002,"15min":5.1349999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.032,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.79900000000000004},{"interface":"front","average":{"1min":5.8869999999999996,"5min":3.8650000000000002,"15min":4.0730000000000004},"min":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":91.837999999999994,"5min":91.837999999999994,"15min":91.837999999999994},"last":0.66600000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":3.782,"15min":1.8919999999999999},"min":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":15.58,"5min":144.65600000000001,"15min":144.65600000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":2.8439999999999999,"5min":4.173,"15min":4.492},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":31.751000000000001,"5min":76.034999999999997,"15min":76.034999999999997},"last":0.85699999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":6.8899999999999997,"15min":8.5739999999999998},"min":{"1min":0.60499999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0609999999999999,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.042,"5min":4.0679999999999996,"15min":1.9990000000000001},"min":{"1min":0.61899999999999999,"5min":0.498,"15min":0.498},"max":{"1min":3.944,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.73999999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":5.0599999999999996,"15min":5.9039999999999999},"min":{"1min":0.51200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1799999999999999,"5min":73.561000000000007,"15min":73.561000000000007},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.92100000000000004,"5min":4.226,"15min":4.7050000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.3029999999999999,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.69299999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":6.4340000000000002,"15min":7.827},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.216,"5min":73.638999999999996,"15min":73.638999999999996},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.9690000000000001,"15min":1.389},"min":{"1min":0.74399999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1559999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":0.876}]}]},{"osd":6,"up_from":0,"seq":55834574890,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574890,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":27,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":925376,"kb_used_data":92778284,"kb_used_omap":4182,"kb_used_meta":0,"kb_avail":92774004,"statfs":{"total":95948165120,"available":95000580096,"internally_reserved":4382720,"allocated":95004962816,"data_stored":95004962816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4283095,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":4.633,"15min":4.8209999999999997},"min":{"1min":0.47099999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.282,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.77900000000000003},{"interface":"front","average":{"1min":4.0869999999999997,"5min":2.008,"15min":1.1899999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":70.352999999999994,"5min":70.352999999999994,"15min":70.352999999999994},"last":0.86199999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":2.145,"5min":7.5339999999999998,"15min":8.2400000000000002},"min":{"1min":0.60399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":24.489999999999998,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":12.505000000000001,"5min":4.1219999999999999,"15min":1.903},"min":{"1min":0.52900000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":110.06999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.59099999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":5.4859999999999998,"15min":5.5019999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":97.974000000000004,"5min":99.884,"15min":99.884},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.88500000000000001,"15min":0.90100000000000002},"min":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.254,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.90100000000000002}]},{"osd":3,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.2649999999999999,"15min":1.3140000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.3280000000000001,"5min":12.878,"15min":12.878},"last":0.65900000000000003},{"interface":"front","average":{"1min":2.6320000000000001,"5min":5.5659999999999998,"15min":5.9560000000000004},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":24.649999999999999,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.83099999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.88600000000000001,"15min":0.90800000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.109,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.008},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.746,"15min":2.1139999999999999},"min":{"1min":0.65200000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3879999999999999,"5min":30.971,"15min":30.971},"last":1.0509999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":2.2149999999999999,"5min":5.8090000000000002,"15min":5.718},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":24.553999999999998,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.92600000000000005,"5min":3.4910000000000001,"15min":4.6050000000000004},"min":{"1min":0.66700000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.234,"5min":88.186999999999998,"15min":88.186999999999998},"last":1.081}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":8.266,"15min":11.507},"min":{"1min":0.63400000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.4510000000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":2.2770000000000001,"5min":8.2899999999999991,"15min":10.256},"min":{"1min":0.65600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":24.603999999999999,"5min":99.843000000000004,"15min":99.843000000000004},"last":1.034}]}]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":32,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":939512,"kb_used_data":92767908,"kb_used_omap":4672,"kb_used_meta":0,"kb_avail":92759868,"statfs":{"total":95948165120,"available":94986104832,"internally_reserved":8232960,"allocated":94994337792,"data_stored":94994337792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784350,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":14,"commit_latency_ns":0,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":8.657,"5min":6.4790000000000001,"15min":5.391},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":64.275999999999996,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":4.7889999999999997,"5min":6.2050000000000001,"15min":7.1630000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":36.207000000000001,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.79700000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.3579999999999997,"5min":11.628,"15min":12.897},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":60.664000000000001,"5min":159.256,"15min":159.256},"last":0.65200000000000002},{"interface":"front","average":{"1min":19.719999999999999,"5min":6.1059999999999999,"15min":2.5289999999999999},"min":{"1min":0.40200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":155.44,"5min":174.578,"15min":174.578},"last":0.72499999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":3.1429999999999998,"5min":3.153,"15min":3.2829999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":40.692999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.69199999999999995},{"interface":"front","average":{"1min":7.3019999999999996,"5min":4.5700000000000003,"15min":2.9849999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":64.381,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.92600000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":8.7050000000000001,"5min":6.2599999999999998,"15min":6.4610000000000003},"min":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":72.438999999999993,"5min":72.438999999999993,"15min":72.438999999999993},"last":0.91300000000000003},{"interface":"front","average":{"1min":8.8840000000000003,"5min":12.984,"15min":13.577},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":60.857999999999997,"5min":159.47200000000001,"15min":159.47200000000001},"last":0.97599999999999998}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.226,"5min":11.475,"15min":13.093},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":60.912999999999997,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":4.8529999999999998,"5min":3.359,"15min":1.8009999999999999},"min":{"1min":0.752,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":64.513000000000005,"5min":174.61000000000001,"15min":174.61000000000001},"last":1.1379999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.8799999999999999,"5min":3.3690000000000002,"15min":1.8109999999999999},"min":{"1min":0.69999999999999996,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":64.590000000000003,"5min":174.63999999999999,"15min":174.63999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":1,"5min":3.0529999999999999,"15min":3.5249999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.304,"5min":60.098999999999997,"15min":60.098999999999997},"last":1.25}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":3.206,"15min":3.7589999999999999},"min":{"1min":0.69699999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.2110000000000001,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.1639999999999999},{"interface":"front","average":{"1min":5.3739999999999997,"5min":14.025,"15min":16.734999999999999},"min":{"1min":0.72599999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":60.951999999999998,"5min":159.512,"15min":159.512},"last":1.073}]}]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":23,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":934036,"kb_used_data":92769260,"kb_used_omap":3556,"kb_used_meta":0,"kb_avail":92765344,"statfs":{"total":95948165120,"available":94991712256,"internally_reserved":4009984,"allocated":94995722240,"data_stored":94995722240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3642223,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.5739999999999998,"5min":2.5009999999999999,"15min":1.2609999999999999},"min":{"1min":0.439,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":66.277000000000001,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":4.617,"5min":1.4670000000000001,"15min":0.96499999999999997},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":88.299999999999997,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.495}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.2610000000000001,"5min":3.7229999999999999,"15min":1.7729999999999999},"min":{"1min":0.46500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":80.233999999999995,"5min":143.071,"15min":143.071},"last":0.67000000000000004},{"interface":"front","average":{"1min":4.6989999999999998,"5min":3.1000000000000001,"15min":3.27},"min":{"1min":0.505,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":88.186000000000007,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.59999999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":10.462,"15min":10.422000000000001},"min":{"1min":0.40999999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.94899999999999995,"5min":183.53,"15min":183.53},"last":0.56000000000000005},{"interface":"front","average":{"1min":5.7729999999999997,"5min":3.8149999999999999,"15min":1.7829999999999999},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":66.335999999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.83799999999999997}]},{"osd":4,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":5.9489999999999998,"15min":4.4160000000000004},"min":{"1min":0.624,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":66.491,"5min":175.34700000000001,"15min":175.34700000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":4.9290000000000003,"5min":5.8259999999999996,"15min":4.4809999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":88.459999999999994,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.97899999999999998}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":2.577,"15min":1.643},"min":{"1min":0.65800000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2829999999999999,"5min":143.262,"15min":143.262},"last":0.82199999999999995},{"interface":"front","average":{"1min":6.0339999999999998,"5min":2.1040000000000001,"15min":1.4750000000000001},"min":{"1min":0.71899999999999997,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":66.456999999999994,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.96499999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.9889999999999999,"5min":1.8879999999999999,"15min":1.3169999999999999},"min":{"1min":0.70699999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":88.542000000000002,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.028,"5min":2.7109999999999999,"15min":1.7150000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.585,"5min":143.215,"15min":143.215},"last":1.147}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.6230000000000002,"5min":3.5489999999999999,"15min":3.222},"min":{"1min":0.81100000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":88.623999999999995,"5min":88.623999999999995,"15min":88.623999999999995},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.048,"5min":3.5630000000000002,"15min":3.0390000000000001},"min":{"1min":0.79900000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3859999999999999,"5min":143.17099999999999,"15min":143.17099999999999},"last":1.089}]}]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":42,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":979276,"kb_used_data":92730756,"kb_used_omap":4694,"kb_used_meta":0,"kb_avail":92720104,"statfs":{"total":95948165120,"available":94945386496,"internally_reserved":10907648,"allocated":94956294144,"data_stored":94956294144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4806907,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,2,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3609999999999998,"5min":4.9240000000000004,"15min":4.2359999999999998},"min":{"1min":0.39900000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":100.533,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.501},{"interface":"front","average":{"1min":4.6509999999999998,"5min":3.6459999999999999,"15min":4.0970000000000004},"min":{"1min":0.373,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":85.751000000000005,"5min":85.751000000000005,"15min":85.751000000000005},"last":0.84699999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":2.6600000000000001,"5min":1.2270000000000001,"15min":1.0529999999999999},"min":{"1min":0.32500000000000001,"5min":0.245,"15min":0.245},"max":{"1min":43.759999999999998,"5min":43.759999999999998,"15min":43.759999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":3.1869999999999998,"15min":3.7770000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82899999999999996,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.71599999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":17.001000000000001,"5min":15.438000000000001,"15min":18.370000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":245.637,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":5.6310000000000002,"5min":5.9980000000000002,"15min":6.8289999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":85.814999999999998,"5min":85.814999999999998,"15min":85.814999999999998},"last":0.80500000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":63.935000000000002,"5min":16.309999999999999,"15min":8.5879999999999992},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1181.3389999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":57.106000000000002,"5min":14.65,"15min":7.2859999999999996},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1181.8720000000001,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.96399999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.98099999999999998,"15min":1.0149999999999999},"min":{"1min":0.58099999999999996,"5min":0.433,"15min":0.433},"max":{"1min":1.244,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":1.2030000000000001},{"interface":"front","average":{"1min":0.92200000000000004,"5min":3.036,"15min":3.9700000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.1339999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.0129999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":4.2060000000000004,"15min":4.7359999999999998},"min":{"1min":0.745,"5min":0.63,"15min":0.63},"max":{"1min":85.884,"5min":85.884,"15min":85.884},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.056,"15min":1.101},"min":{"1min":0.69599999999999995,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.298,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.044}]},{"osd":7,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":8.3879999999999999,"5min":7.1310000000000002,"15min":7.0549999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":157.03700000000001,"5min":157.03700000000001,"15min":157.03700000000001},"last":1.097},{"interface":"front","average":{"1min":5.0919999999999996,"5min":4.0730000000000004,"15min":4.524},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":85.852999999999994,"5min":85.852999999999994,"15min":85.852999999999994},"last":1.2250000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":806556,"kb_used_data":92892880,"kb_used_omap":4127,"kb_used_meta":0,"kb_avail":92892824,"statfs":{"total":95948165120,"available":95122251776,"internally_reserved":57344,"allocated":95122309120,"data_stored":95122309120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4226173,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":7.2519999999999998,"5min":2.6970000000000001,"15min":1.3140000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":69.816999999999993,"5min":69.872,"15min":69.872},"last":0.629},{"interface":"front","average":{"1min":5.6130000000000004,"5min":2.6789999999999998,"15min":1.379},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":98.203999999999994,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.75900000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.7039999999999997,"5min":3.407,"15min":1.5980000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":98.286000000000001,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":3.0270000000000001,"5min":2.6819999999999999,"15min":1.3660000000000001},"min":{"1min":0.38400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":48.893000000000001,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.90800000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":12.154999999999999,"5min":13.413,"15min":15.657},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":94.712999999999994,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.85399999999999998},{"interface":"front","average":{"1min":10.484,"5min":5.6399999999999997,"15min":2.4140000000000001},"min":{"1min":0.42399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":98.352999999999994,"5min":174.09399999999999,"15min":174.09399999999999},"last":1.0780000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.5570000000000004,"5min":3.2309999999999999,"15min":1.734},"min":{"1min":0.63800000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":61.536000000000001,"5min":71.483000000000004,"15min":71.483000000000004},"last":1.0509999999999999},{"interface":"front","average":{"1min":9.2989999999999995,"5min":3.9950000000000001,"15min":2.0150000000000001},"min":{"1min":0.67400000000000004,"5min":0.439,"15min":0.439},"max":{"1min":70.179000000000002,"5min":70.179000000000002,"15min":70.179000000000002},"last":0.94499999999999995}]},{"osd":5,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.9009999999999998,"5min":8.218,"15min":9.2710000000000008},"min":{"1min":0.68500000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":98.411000000000001,"5min":176.34399999999999,"15min":176.34399999999999},"last":1.169},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.7709999999999999,"15min":1.2829999999999999},"min":{"1min":0.63400000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.2959999999999998,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.997}]},{"osd":6,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":7.7030000000000003,"5min":3.2040000000000002,"15min":1.8480000000000001},"min":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":70.259,"5min":70.259,"15min":70.259},"last":1.034},{"interface":"front","average":{"1min":5.984,"5min":3.056,"15min":1.774},"min":{"1min":0.749,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":98.484999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":1.2190000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":4.633,"5min":12.002000000000001,"15min":14.448},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":73.864999999999995,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.105},{"interface":"front","average":{"1min":7.7199999999999998,"5min":3.21,"15min":1.8500000000000001},"min":{"1min":0.70399999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":70.322000000000003,"5min":70.322000000000003,"15min":70.322000000000003},"last":1.1399999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:31:22.309 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:31:22.309 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 56 2023-01-15T14:31:22.311 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:22.311 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:22.312 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:31:22.312 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:22.306+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:22.411 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:22.407+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:22.481 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:22.481 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:22.481 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:31:22.512 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:22.507+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:22.593 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:22.593 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:22.603 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:31:22.613 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:22.610+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:22.709 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:22.709 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:22.713 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:22.707+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:22.718 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:31:22.815 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:22.810+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:22.821 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 56 2023-01-15T14:31:22.829 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:22.829 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:22.838 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:31:22.914 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:22.911+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:22.990 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:22.990 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:23.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:31:23.016 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:23.010+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:23.114 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:23.114 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:23.117 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:23.114+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:23.125 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:31:23.220 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:23.214+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:23.230 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:23.230 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:23.239 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:31:23.320 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:23.315+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:23.347 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:23.347 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:23.357 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:31:23.421 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:23.418+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:23.464 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:23.464 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:23.473 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:31:23.522 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:23.518+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:23.527 INFO:tasks.rados.rados.0.smithi090.stdout:748: expect (ObjNum 248 snap 58 seq_num 248) 2023-01-15T14:31:23.581 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:23.581 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:23.590 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:31:23.623 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:23.618+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:23.653 INFO:tasks.rados.rados.0.smithi090.stdout:748: done (0 left) 2023-01-15T14:31:23.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:31:23.695 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:23.695 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:23.706 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:31:23.724 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:23.718+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:23.811 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:23.812 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:23.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:31:23.822 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:23.824 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:23.818+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:23.844 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:31:23.845 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:23.868 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:23.886 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:23.907 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:23.924 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:23.924 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:23.925 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:23.925 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:23.925 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:23.919+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:23.932 INFO:tasks.rados.rados.0.smithi090.stdout:752: done (0 left) 2023-01-15T14:31:23.932 INFO:tasks.rados.rados.0.smithi090.stdout:753: read oid 10 snap 62 2023-01-15T14:31:23.932 INFO:tasks.rados.rados.0.smithi090.stdout:753: expect deleted 2023-01-15T14:31:23.932 INFO:tasks.rados.rados.0.smithi090.stdout:754: write oid 26 current snap is 78 2023-01-15T14:31:23.932 INFO:tasks.rados.rados.0.smithi090.stdout:754: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2023-01-15T14:31:23.933 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:23.938 INFO:tasks.rados.rados.0.smithi090.stdout:754: writing smithi09025264-26 from 558065 to 1177934 tid 1 2023-01-15T14:31:23.939 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:31:23.944 INFO:tasks.rados.rados.0.smithi090.stdout:754: writing smithi09025264-26 from 1579835 to 2066600 tid 2 2023-01-15T14:31:23.950 INFO:tasks.rados.rados.0.smithi090.stdout:754: writing smithi09025264-26 from 2468371 to 2958065 tid 3 2023-01-15T14:31:23.950 INFO:tasks.rados.rados.0.smithi090.stdout:755: rollback oid 48 current snap is 78 2023-01-15T14:31:23.951 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 48 to 62 2023-01-15T14:31:23.951 INFO:tasks.rados.rados.0.smithi090.stdout:756: rollback oid 39 current snap is 78 2023-01-15T14:31:23.951 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 39 to 62 2023-01-15T14:31:23.951 INFO:tasks.rados.rados.0.smithi090.stdout:757: delete oid 6 current snap is 78 2023-01-15T14:31:23.954 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:24.026 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:24.022+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:24.043 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:38 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'3, mlcod=0'0)", 2023-01-15T14:31:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.92600567599999, 2023-01-15T14:31:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30191175999999997, 2023-01-15T14:31:24.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:24.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112976+0000", 2023-01-15T14:31:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113186+0000", 2023-01-15T14:31:24.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020979700000000001 2023-01-15T14:31:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113228+0000", 2023-01-15T14:31:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2150999999999997e-05 2023-01-15T14:31:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213313+0000", 2023-01-15T14:31:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100084943 2023-01-15T14:31:24.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313517+0000", 2023-01-15T14:31:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10020435699999999 2023-01-15T14:31:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313576+0000", 2023-01-15T14:31:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783999999999998e-05 2023-01-15T14:31:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313976+0000", 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039960499999999998 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314004+0000", 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7954e-05 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314175+0000", 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171557 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414853+0000", 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100677875 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414890+0000", 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6803e-05 2023-01-15T14:31:24.050 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:46 3.0 e23/20 3:03b54904:::smithi09025264-12:head v 23'5, mlcod=23'1)", 2023-01-15T14:31:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:31:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.82510976200001, 2023-01-15T14:31:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202592668, 2023-01-15T14:31:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.051 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:31:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213874+0000", 2023-01-15T14:31:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213873+0000", 2023-01-15T14:31:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214477+0000", 2023-01-15T14:31:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060481300000000005 2023-01-15T14:31:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214532+0000", 2023-01-15T14:31:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5096999999999998e-05 2023-01-15T14:31:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.214555+0000", 2023-01-15T14:31:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.234e-05 2023-01-15T14:31:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414409+0000", 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19985461199999999 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414442+0000", 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2987000000000001e-05 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414813+0000", 2023-01-15T14:31:24.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037061899999999998 2023-01-15T14:31:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414832+0000", 2023-01-15T14:31:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9380000000000001e-05 2023-01-15T14:31:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415093+0000", 2023-01-15T14:31:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026032099999999999 2023-01-15T14:31:24.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.415154+0000", 2023-01-15T14:31:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1538999999999995e-05 2023-01-15T14:31:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.416467+0000", 2023-01-15T14:31:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001312613 2023-01-15T14:31:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.057 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:125 3.0 e70/55 3:0b54190a:::smithi09025264-17:25 v 70'136, mlcod=68'130)", 2023-01-15T14:31:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 128.096023651, 2023-01-15T14:31:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30171926799999998, 2023-01-15T14:31:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942958+0000", 2023-01-15T14:31:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942972+0000", 2023-01-15T14:31:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3386e-05 2023-01-15T14:31:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942990+0000", 2023-01-15T14:31:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8113000000000001e-05 2023-01-15T14:31:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.943002+0000", 2023-01-15T14:31:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:31:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141942+0000", 2023-01-15T14:31:24.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19893962300000001 2023-01-15T14:31:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141978+0000", 2023-01-15T14:31:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6383e-05 2023-01-15T14:31:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142670+0000", 2023-01-15T14:31:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069211900000000005 2023-01-15T14:31:24.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142688+0000", 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7839999999999999e-05 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142768+0000", 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0210999999999995e-05 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244581+0000", 2023-01-15T14:31:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101813406 2023-01-15T14:31:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244680+0000", 2023-01-15T14:31:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8045000000000001e-05 2023-01-15T14:31:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(3.0 REQUEST e99)", 2023-01-15T14:31:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.042570007999998, 2023-01-15T14:31:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26432840200000002, 2023-01-15T14:31:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996414+0000", 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:24.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996419+0000", 2023-01-15T14:31:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6729999999999997e-06 2023-01-15T14:31:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996449+0000", 2023-01-15T14:31:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.004e-05 2023-01-15T14:31:24.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996463+0000", 2023-01-15T14:31:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3607e-05 2023-01-15T14:31:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996594+0000", 2023-01-15T14:31:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013148899999999999 2023-01-15T14:31:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:31:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:45.996618+0000", 2023-01-15T14:31:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3249000000000001e-05 2023-01-15T14:31:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260645+0000", 2023-01-15T14:31:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26402768700000001 2023-01-15T14:31:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260671+0000", 2023-01-15T14:31:24.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.546e-05 2023-01-15T14:31:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260742+0000", 2023-01-15T14:31:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1569999999999994e-05 2023-01-15T14:31:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.075 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:261 3.8 e98/41 3:1e54a7f5:::smithi09025264-42:3e v 98'160, mlcod=97'157)", 2023-01-15T14:31:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:31:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.981502418999995, 2023-01-15T14:31:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20299468400000001, 2023-01-15T14:31:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:31:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057482+0000", 2023-01-15T14:31:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057480+0000", 2023-01-15T14:31:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057496+0000", 2023-01-15T14:31:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5869999999999999e-05 2023-01-15T14:31:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057528+0000", 2023-01-15T14:31:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:31:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.057545+0000", 2023-01-15T14:31:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.694e-05 2023-01-15T14:31:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.158971+0000", 2023-01-15T14:31:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101425425 2023-01-15T14:31:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159011+0000", 2023-01-15T14:31:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0058000000000002e-05 2023-01-15T14:31:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159858+0000", 2023-01-15T14:31:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00084767900000000005 2023-01-15T14:31:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159877+0000", 2023-01-15T14:31:24.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9133000000000001e-05 2023-01-15T14:31:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.159959+0000", 2023-01-15T14:31:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1693000000000005e-05 2023-01-15T14:31:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260430+0000", 2023-01-15T14:31:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10047039200000001 2023-01-15T14:31:24.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260476+0000", 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6652e-05 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:24.082 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 97.980281262999995, 2023-01-15T14:31:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:31:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:31:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:31:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:31:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:31:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:31:24.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:31:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:31:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:31:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:31:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:31:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:31:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:31:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:31:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:31:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:31:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:31:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:31:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:31:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:31:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:31:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:31:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:194 3.4 e101/48 3:2022fdfa:::smithi09025264-46:43 v 101'104, mlcod=98'100)", 2023-01-15T14:31:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:31:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 96.194239651999993, 2023-01-15T14:31:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20670733199999999, 2023-01-15T14:31:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:31:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844744+0000", 2023-01-15T14:31:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844743+0000", 2023-01-15T14:31:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844757+0000", 2023-01-15T14:31:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.484e-05 2023-01-15T14:31:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.844787+0000", 2023-01-15T14:31:24.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9113000000000001e-05 2023-01-15T14:31:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.944923+0000", 2023-01-15T14:31:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10013656 2023-01-15T14:31:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.949975+0000", 2023-01-15T14:31:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0050521400000000001 2023-01-15T14:31:24.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950015+0000", 2023-01-15T14:31:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9286999999999998e-05 2023-01-15T14:31:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950559+0000", 2023-01-15T14:31:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054463500000000002 2023-01-15T14:31:24.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950585+0000", 2023-01-15T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6166e-05 2023-01-15T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:47.950670+0000", 2023-01-15T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4902999999999997e-05 2023-01-15T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051368+0000", 2023-01-15T14:31:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100697913 2023-01-15T14:31:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:48.051452+0000", 2023-01-15T14:31:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3442000000000005e-05 2023-01-15T14:31:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1224 3.4 e107/48 3:2029fa71:::smithi09025264-39:head v 107'120, mlcod=104'116)", 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 89.807211011999996, 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22601005900000001, 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:31:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231773+0000", 2023-01-15T14:31:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231771+0000", 2023-01-15T14:31:24.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233829+0000", 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002057836 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.233881+0000", 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1856e-05 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.333996+0000", 2023-01-15T14:31:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100115315 2023-01-15T14:31:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434199+0000", 2023-01-15T14:31:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100203373 2023-01-15T14:31:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.434236+0000", 2023-01-15T14:31:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6563000000000002e-05 2023-01-15T14:31:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.435979+0000", 2023-01-15T14:31:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017426570000000001 2023-01-15T14:31:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.436036+0000", 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7716000000000001e-05 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437140+0000", 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011036399999999999 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.437251+0000", 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110869 2023-01-15T14:31:24.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.457783+0000", 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.020532181 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1649 3.4 3:2029fa71:::smithi09025264-39:head [write 534522~634550] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.893806+0000", 2023-01-15T14:31:24.103 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.145177935, 2023-01-15T14:31:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205815042, 2023-01-15T14:31:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1649 2023-01-15T14:31:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.893806+0000", 2023-01-15T14:31:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.893806+0000", 2023-01-15T14:31:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.893803+0000", 2023-01-15T14:31:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:31:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894135+0000", 2023-01-15T14:31:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033239899999999999 2023-01-15T14:31:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894168+0000", 2023-01-15T14:31:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2490000000000002e-05 2023-01-15T14:31:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894195+0000", 2023-01-15T14:31:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7576000000000002e-05 2023-01-15T14:31:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894997+0000", 2023-01-15T14:31:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00080147300000000003 2023-01-15T14:31:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895010+0000", 2023-01-15T14:31:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611000000000001e-05 2023-01-15T14:31:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.991782+0000", 2023-01-15T14:31:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096771757 2023-01-15T14:31:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.992255+0000", 2023-01-15T14:31:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047275099999999999 2023-01-15T14:31:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:24.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.992633+0000", 2023-01-15T14:31:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037850299999999998 2023-01-15T14:31:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.995684+0000", 2023-01-15T14:31:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0030508610000000002 2023-01-15T14:31:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.995715+0000", 2023-01-15T14:31:24.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0759000000000001e-05 2023-01-15T14:31:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.996025+0000", 2023-01-15T14:31:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003102 2023-01-15T14:31:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.996334+0000", 2023-01-15T14:31:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030889699999999998 2023-01-15T14:31:24.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098878+0000", 2023-01-15T14:31:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10254395299999999 2023-01-15T14:31:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099030+0000", 2023-01-15T14:31:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152192 2023-01-15T14:31:24.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099198+0000", 2023-01-15T14:31:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016808400000000001 2023-01-15T14:31:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099621+0000", 2023-01-15T14:31:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042263899999999999 2023-01-15T14:31:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.113 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1650 3.4 3:2029fa71:::smithi09025264-39:head [write 1812335~322187] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894235+0000", 2023-01-15T14:31:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.144748526999997, 2023-01-15T14:31:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20642070600000001, 2023-01-15T14:31:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.114 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1650 2023-01-15T14:31:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894235+0000", 2023-01-15T14:31:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894235+0000", 2023-01-15T14:31:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894234+0000", 2023-01-15T14:31:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894404+0000", 2023-01-15T14:31:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017082499999999999 2023-01-15T14:31:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894426+0000", 2023-01-15T14:31:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1387e-05 2023-01-15T14:31:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894440+0000", 2023-01-15T14:31:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3753e-05 2023-01-15T14:31:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895046+0000", 2023-01-15T14:31:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060607599999999999 2023-01-15T14:31:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895054+0000", 2023-01-15T14:31:24.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6130000000000004e-06 2023-01-15T14:31:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.096846+0000", 2023-01-15T14:31:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201791888 2023-01-15T14:31:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.097087+0000", 2023-01-15T14:31:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024123800000000001 2023-01-15T14:31:24.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.097486+0000", 2023-01-15T14:31:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039807900000000002 2023-01-15T14:31:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098073+0000", 2023-01-15T14:31:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058732299999999995 2023-01-15T14:31:24.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098097+0000", 2023-01-15T14:31:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3773000000000001e-05 2023-01-15T14:31:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098309+0000", 2023-01-15T14:31:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021264999999999999 2023-01-15T14:31:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.098383+0000", 2023-01-15T14:31:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3819999999999995e-05 2023-01-15T14:31:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.099809+0000", 2023-01-15T14:31:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014256360000000001 2023-01-15T14:31:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100004+0000", 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019514099999999999 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100220+0000", 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021621800000000001 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100656+0000", 2023-01-15T14:31:24.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000436036 2023-01-15T14:31:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.144523872999997, 2023-01-15T14:31:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:31:24.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:31:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:31:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:31:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:31:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:31:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:31:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:31:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:31:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:31:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:31:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:31:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:31:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:31:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:31:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:31:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:31:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:31:24.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:31:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:31:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:31:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:31:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:31:24.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:31:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:31:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:31:24.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:31:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:31:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:31:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:31:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:31:24.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.144472929999999, 2023-01-15T14:31:24.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:31:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:31:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:31:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:31:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:31:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:31:24.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:31:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:31:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:31:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:31:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:31:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:31:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:31:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:31:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:31:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:31:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:31:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:31:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:31:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:31:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.203573689000002, 2023-01-15T14:31:24.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:24.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:31:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:31:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:31:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:31:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:31:24.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:31:24.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:31:24.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:24.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:31:24.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:31:24.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:31:24.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:31:24.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:31:24.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:31:24.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:31:24.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:31:24.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:31:24.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:31:24.152 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.152 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.152 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.152 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:24.152 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:24.152 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.129799515999999, 2023-01-15T14:31:24.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:31:24.152 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:24.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:31:24.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:24.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:31:24.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:31:24.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:31:24.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:31:24.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:31:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:31:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:31:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:31:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:31:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:31:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:31:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:31:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:31:24.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:31:24.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:31:24.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:31:24.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:31:24.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:31:24.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:31:24.162 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.162 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.162 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.162 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1712 3.4 3:2022fdfa:::smithi09025264-46:head [write 1880802~421790] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:24.162 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.124660824999999, 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30396041800000001, 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1712 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914322+0000", 2023-01-15T14:31:24.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:24.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914561+0000", 2023-01-15T14:31:24.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023884700000000001 2023-01-15T14:31:24.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914572+0000", 2023-01-15T14:31:24.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0499999999999999e-05 2023-01-15T14:31:24.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914576+0000", 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6820000000000002e-06 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014941+0000", 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100364994 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014959+0000", 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7416000000000001e-05 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018171+0000", 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032121239999999998 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018333+0000", 2023-01-15T14:31:24.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016180900000000001 2023-01-15T14:31:24.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:24.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018581+0000", 2023-01-15T14:31:24.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024773699999999999 2023-01-15T14:31:24.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018897+0000", 2023-01-15T14:31:24.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031622099999999999 2023-01-15T14:31:24.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018907+0000", 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0061000000000001e-05 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.019108+0000", 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020080099999999999 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.020057+0000", 2023-01-15T14:31:24.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094954499999999999 2023-01-15T14:31:24.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150472+0000", 2023-01-15T14:31:24.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.130414958 2023-01-15T14:31:24.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218188+0000", 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067715656999999999 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218242+0000", 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4651999999999998e-05 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218284+0000", 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1084999999999997e-05 2023-01-15T14:31:24.171 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.18 157/147 [PushOp(3:1eaad226:::smithi09025264-49:69, version: 139'201, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:1eaad226:::smithi09025264-49:69@139'201, size: 3916031, copy_subset: [0~3916031], clone_subset: {}, snapset: 69=[]:{68=[67],69=[69]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.329750273000002, 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27553999000000001, 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:24.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:24.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709232+0000", 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709949+0000", 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071723000000000002 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709990+0000", 2023-01-15T14:31:24.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.015e-05 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.810156+0000", 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10016643 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984513+0000", 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.174357128 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984522+0000", 2023-01-15T14:31:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0350000000000007e-06 2023-01-15T14:31:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984774+0000", 2023-01-15T14:31:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025144700000000002 2023-01-15T14:31:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 20.235844096000001, 2023-01-15T14:31:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:31:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.177 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.178 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:31:24.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.178 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:24.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:31:24.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:31:24.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:31:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:31:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:31:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:31:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:31:24.182 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1820 3.1f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 20.135542611000002, 2023-01-15T14:31:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306504938, 2023-01-15T14:31:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:24.183 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1820 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903440+0000", 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903448+0000", 2023-01-15T14:31:24.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9430000000000006e-06 2023-01-15T14:31:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903469+0000", 2023-01-15T14:31:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0305e-05 2023-01-15T14:31:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903481+0000", 2023-01-15T14:31:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:31:24.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903616+0000", 2023-01-15T14:31:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013477199999999999 2023-01-15T14:31:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903643+0000", 2023-01-15T14:31:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7098999999999999e-05 2023-01-15T14:31:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209081+0000", 2023-01-15T14:31:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30543790300000001 2023-01-15T14:31:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209213+0000", 2023-01-15T14:31:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013155400000000001 2023-01-15T14:31:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209600+0000", 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038751699999999998 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209946+0000", 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034601 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1906 3.1f 3:f8edd34e:::smithi09025264-19:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:13.885370+0000", 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.153614179, 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20444454000000001, 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1906 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885370+0000", 2023-01-15T14:31:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885370+0000", 2023-01-15T14:31:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885367+0000", 2023-01-15T14:31:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885380+0000", 2023-01-15T14:31:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2413e-05 2023-01-15T14:31:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885406+0000", 2023-01-15T14:31:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6542000000000001e-05 2023-01-15T14:31:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885418+0000", 2023-01-15T14:31:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1219999999999999e-05 2023-01-15T14:31:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885507+0000", 2023-01-15T14:31:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9738e-05 2023-01-15T14:31:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.885782+0000", 2023-01-15T14:31:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027431999999999999 2023-01-15T14:31:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.886171+0000", 2023-01-15T14:31:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038963000000000002 2023-01-15T14:31:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.887617+0000", 2023-01-15T14:31:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001445986 2023-01-15T14:31:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.887634+0000", 2023-01-15T14:31:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6633000000000002e-05 2023-01-15T14:31:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.887686+0000", 2023-01-15T14:31:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2346999999999999e-05 2023-01-15T14:31:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:13.887721+0000", 2023-01-15T14:31:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4427000000000001e-05 2023-01-15T14:31:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.089581+0000", 2023-01-15T14:31:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201860013 2023-01-15T14:31:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.089642+0000", 2023-01-15T14:31:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0859000000000001e-05 2023-01-15T14:31:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.089711+0000", 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9648999999999997e-05 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:14.089814+0000", 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010310900000000001 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1912 3.4 e189/181 3:2022fdfa:::smithi09025264-46:head v 189'194, mlcod=180'190)", 2023-01-15T14:31:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:15.986052+0000", 2023-01-15T14:31:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 8.0529322529999998, 2023-01-15T14:31:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22134975400000001, 2023-01-15T14:31:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:15.986052+0000", 2023-01-15T14:31:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:15.986052+0000", 2023-01-15T14:31:24.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:15.986051+0000", 2023-01-15T14:31:24.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:24.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:15.986063+0000", 2023-01-15T14:31:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2472e-05 2023-01-15T14:31:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:15.986095+0000", 2023-01-15T14:31:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1613999999999999e-05 2023-01-15T14:31:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.086194+0000", 2023-01-15T14:31:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009912999999999 2023-01-15T14:31:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.190517+0000", 2023-01-15T14:31:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104323034 2023-01-15T14:31:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.190558+0000", 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0535999999999999e-05 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.191286+0000", 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072818899999999996 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.191347+0000", 2023-01-15T14:31:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0748999999999997e-05 2023-01-15T14:31:24.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.191417+0000", 2023-01-15T14:31:24.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0908999999999995e-05 2023-01-15T14:31:24.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.192443+0000", 2023-01-15T14:31:24.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010255570000000001 2023-01-15T14:31:24.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:16.207401+0000", 2023-01-15T14:31:24.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.014958447 2023-01-15T14:31:24.205 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.205 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.205 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.205 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.205 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.205 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:24.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:31:24.208 INFO:tasks.rados.rados.0.smithi090.stdout:755: finishing rollback tid 0 to smithi09025264-48 2023-01-15T14:31:24.208 INFO:tasks.rados.rados.0.smithi090.stdout:755: finishing rollback tid 1 to smithi09025264-48 2023-01-15T14:31:24.208 INFO:tasks.rados.rados.0.smithi090.stdout:755: finishing rollback tid 2 to smithi09025264-48 2023-01-15T14:31:24.208 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 160 (ObjNum 260 snap 60 seq_num 260) dirty exists 2023-01-15T14:31:24.208 INFO:tasks.rados.rados.0.smithi090.stdout:754: finishing write tid 1 to smithi09025264-26 2023-01-15T14:31:24.208 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:31:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:31:24.209 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:31:24.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:31:24.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:31:24.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:31:24.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:31:24.210 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:31:24.210 INFO:tasks.rados.rados.0.smithi090.stdout:755: done (4 left) 2023-01-15T14:31:24.211 INFO:tasks.rados.rados.0.smithi090.stdout:757: done (3 left) 2023-01-15T14:31:24.211 INFO:tasks.rados.rados.0.smithi090.stdout:758: rollback oid 36 current snap is 78 2023-01-15T14:31:24.211 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 36 to 72 2023-01-15T14:31:24.211 INFO:tasks.rados.rados.0.smithi090.stdout:759: delete oid 23 current snap is 78 2023-01-15T14:31:24.211 INFO:tasks.rados.rados.0.smithi090.stdout:756: finishing rollback tid 0 to smithi09025264-39 2023-01-15T14:31:24.211 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:24.202+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:24.229 INFO:tasks.rados.rados.0.smithi090.stdout:754: finishing write tid 2 to smithi09025264-26 2023-01-15T14:31:24.229 INFO:tasks.rados.rados.0.smithi090.stdout:754: finishing write tid 3 to smithi09025264-26 2023-01-15T14:31:24.265 INFO:tasks.rados.rados.0.smithi090.stdout:758: finishing rollback tid 0 to smithi09025264-36 2023-01-15T14:31:24.266 INFO:tasks.rados.rados.0.smithi090.stdout:758: finishing rollback tid 1 to smithi09025264-36 2023-01-15T14:31:24.266 INFO:tasks.rados.rados.0.smithi090.stdout:758: finishing rollback tid 2 to smithi09025264-36 2023-01-15T14:31:24.266 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 239 (ObjNum 199 snap 43 seq_num 199) dirty exists 2023-01-15T14:31:24.275 INFO:tasks.rados.rados.0.smithi090.stdout:756: finishing rollback tid 1 to smithi09025264-39 2023-01-15T14:31:24.275 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 205 (ObjNum 65 snap 0 seq_num 3188165184) dirty dne 2023-01-15T14:31:24.307 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:24.303+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:24.327 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:24.327 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:24.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.327 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:1989 3.3 e197/175)", 2023-01-15T14:31:24.327 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:24.036322+0000", 2023-01-15T14:31:24.328 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.289658211, 2023-01-15T14:31:24.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28969799800000001, 2023-01-15T14:31:24.328 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.328 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:24.328 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:24.036322+0000", 2023-01-15T14:31:24.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:24.036322+0000", 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:24.036321+0000", 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:24.036331+0000", 2023-01-15T14:31:24.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0465999999999999e-05 2023-01-15T14:31:24.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:24.036353+0000", 2023-01-15T14:31:24.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1376e-05 2023-01-15T14:31:24.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:24.036362+0000", 2023-01-15T14:31:24.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0140000000000004e-06 2023-01-15T14:31:24.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.331 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.331 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:31:24.331 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:31:24.331 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:24.337 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:31:24.366 INFO:tasks.rados.rados.0.smithi090.stdout:754: finishing write tid 4 to smithi09025264-26 2023-01-15T14:31:24.369 INFO:tasks.rados.rados.0.smithi090.stdout:753: done (4 left) 2023-01-15T14:31:24.369 INFO:tasks.rados.rados.0.smithi090.stdout:756: done (3 left) 2023-01-15T14:31:24.369 INFO:tasks.rados.rados.0.smithi090.stdout:758: done (2 left) 2023-01-15T14:31:24.369 INFO:tasks.rados.rados.0.smithi090.stdout:759: done (1 left) 2023-01-15T14:31:24.369 INFO:tasks.rados.rados.0.smithi090.stdout:760: write oid 10 current snap is 78 2023-01-15T14:31:24.370 INFO:tasks.rados.rados.0.smithi090.stdout:760: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2023-01-15T14:31:24.385 INFO:tasks.rados.rados.0.smithi090.stdout:760: writing smithi09025264-10 from 608486 to 1170455 tid 1 2023-01-15T14:31:24.393 INFO:tasks.rados.rados.0.smithi090.stdout:760: writing smithi09025264-10 from 1871634 to 2505726 tid 2 2023-01-15T14:31:24.397 INFO:tasks.rados.rados.0.smithi090.stdout:760: writing smithi09025264-10 from 2950640 to 3008486 tid 3 2023-01-15T14:31:24.397 INFO:tasks.rados.rados.0.smithi090.stdout:761: read oid 45 snap -1 2023-01-15T14:31:24.397 INFO:tasks.rados.rados.0.smithi090.stdout:761: expect (ObjNum 206 snap 44 seq_num 206) 2023-01-15T14:31:24.397 INFO:tasks.rados.rados.0.smithi090.stdout:762: write (excl) oid 2 current snap is 78 2023-01-15T14:31:24.397 INFO:tasks.rados.rados.0.smithi090.stdout:762: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2023-01-15T14:31:24.405 INFO:tasks.rados.rados.0.smithi090.stdout:762: writing smithi09025264-2 from 625293 to 1262511 tid 1 2023-01-15T14:31:24.409 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:24.403+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:24.412 INFO:tasks.rados.rados.0.smithi090.stdout:762: writing smithi09025264-2 from 1813763 to 2307866 tid 2 2023-01-15T14:31:24.417 INFO:tasks.rados.rados.0.smithi090.stdout:762: writing smithi09025264-2 from 2978063 to 3025293 tid 3 2023-01-15T14:31:24.417 INFO:tasks.rados.rados.0.smithi090.stdout:763: snap_remove snap 77 2023-01-15T14:31:24.477 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:24.477 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:24.477 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:24.487 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:31:24.510 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:24.506+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:24.611 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:24.606+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:24.649 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:24.649 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:24.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:24.649 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:24.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.649 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:166 3.8 3:1e54a7f5:::smithi09025264-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:24.649 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:24.650 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.52929430399999, 2023-01-15T14:31:24.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62664632300000001, 2023-01-15T14:31:24.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.650 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.650 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 166 2023-01-15T14:31:24.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.650 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114713+0000", 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115178+0000", 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046493199999999999 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115208+0000", 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0689999999999999e-05 2023-01-15T14:31:24.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115221+0000", 2023-01-15T14:31:24.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3013000000000001e-05 2023-01-15T14:31:24.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115334+0000", 2023-01-15T14:31:24.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112403 2023-01-15T14:31:24.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115505+0000", 2023-01-15T14:31:24.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170915 2023-01-15T14:31:24.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333496+0000", 2023-01-15T14:31:24.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217991667 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333606+0000", 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109967 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333796+0000", 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018928099999999999 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334289+0000", 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049322600000000004 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334339+0000", 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0439999999999998e-05 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334990+0000", 2023-01-15T14:31:24.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065082300000000003 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335100+0000", 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109357 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.640854+0000", 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30575451300000001 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741190+0000", 2023-01-15T14:31:24.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100335568 2023-01-15T14:31:24.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741263+0000", 2023-01-15T14:31:24.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3275999999999994e-05 2023-01-15T14:31:24.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741361+0000", 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8435000000000002e-05 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:167 3.8 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.528729615, 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62638390099999997, 2023-01-15T14:31:24.659 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 167 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115279+0000", 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115289+0000", 2023-01-15T14:31:24.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0509999999999999e-05 2023-01-15T14:31:24.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115306+0000", 2023-01-15T14:31:24.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7260000000000001e-05 2023-01-15T14:31:24.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115337+0000", 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0664000000000003e-05 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115579+0000", 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024194400000000001 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115744+0000", 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016493700000000001 2023-01-15T14:31:24.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334392+0000", 2023-01-15T14:31:24.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21864830299999999 2023-01-15T14:31:24.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334453+0000", 2023-01-15T14:31:24.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1047e-05 2023-01-15T14:31:24.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:24.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334601+0000", 2023-01-15T14:31:24.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014824700000000001 2023-01-15T14:31:24.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334812+0000", 2023-01-15T14:31:24.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210208 2023-01-15T14:31:24.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334996+0000", 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183878 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335126+0000", 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013001399999999999 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335160+0000", 2023-01-15T14:31:24.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4876999999999998e-05 2023-01-15T14:31:24.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741044+0000", 2023-01-15T14:31:24.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40588326000000002 2023-01-15T14:31:24.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741453+0000", 2023-01-15T14:31:24.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040963099999999998 2023-01-15T14:31:24.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741548+0000", 2023-01-15T14:31:24.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4961000000000001e-05 2023-01-15T14:31:24.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741664+0000", 2023-01-15T14:31:24.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115333 2023-01-15T14:31:24.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.669 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.669 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.669 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:24.669 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:24.669 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.52863750099999, 2023-01-15T14:31:24.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:31:24.669 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.670 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:24.670 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.670 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.670 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.670 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:31:24.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.670 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:24.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:24.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:31:24.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:31:24.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:31:24.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:31:24.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:31:24.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:31:24.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:31:24.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:31:24.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:31:24.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:31:24.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:31:24.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:31:24.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:31:24.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.677 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:24.677 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:24.677 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.51564542200001, 2023-01-15T14:31:24.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:31:24.677 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.677 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.677 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:31:24.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.678 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:24.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:24.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:31:24.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:24.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:31:24.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:31:24.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:31:24.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:31:24.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:31:24.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:31:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:31:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:31:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:31:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:31:24.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:31:24.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:31:24.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:31:24.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:31:24.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:31:24.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:31:24.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:31:24.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:31:24.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:31:24.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:31:24.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:31:24.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:31:24.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:31:24.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:31:24.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:31:24.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:31:24.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:31:24.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:31:24.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.686 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.687 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.687 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:24.687 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:24.687 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.515242086, 2023-01-15T14:31:24.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:31:24.687 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.687 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.688 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.688 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:31:24.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.688 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:24.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:31:24.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:31:24.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:31:24.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:31:24.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:31:24.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:31:24.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:31:24.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:31:24.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:31:24.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:31:24.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:31:24.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:31:24.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:31:24.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:31:24.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.696 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.696 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:24.697 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:24.697 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.41501846599999, 2023-01-15T14:31:24.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:31:24.697 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.697 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:24.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.698 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:31:24.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.698 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:24.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:24.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:31:24.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:24.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:31:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:31:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:31:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:31:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:31:24.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:31:24.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:31:24.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:31:24.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:31:24.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:31:24.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:31:24.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.704 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.704 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:194 3.8 3:1eaad226:::smithi09025264-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:24.704 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:24.704 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.34261722299999, 2023-01-15T14:31:24.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64372577200000003, 2023-01-15T14:31:24.705 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.705 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.705 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 194 2023-01-15T14:31:24.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301390+0000", 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301991+0000", 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060122599999999995 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.302026+0000", 2023-01-15T14:31:24.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5040000000000003e-05 2023-01-15T14:31:24.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402147+0000", 2023-01-15T14:31:24.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100121084 2023-01-15T14:31:24.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402219+0000", 2023-01-15T14:31:24.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2045999999999994e-05 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402315+0000", 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5561000000000002e-05 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842189+0000", 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.439874129 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842296+0000", 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107021 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842486+0000", 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018965799999999999 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842839+0000", 2023-01-15T14:31:24.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035330700000000002 2023-01-15T14:31:24.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842859+0000", 2023-01-15T14:31:24.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0347e-05 2023-01-15T14:31:24.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843212+0000", 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352534 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843835+0000", 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062338599999999997 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944861+0000", 2023-01-15T14:31:24.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10102596899999999 2023-01-15T14:31:24.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944908+0000", 2023-01-15T14:31:24.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6773000000000001e-05 2023-01-15T14:31:24.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945039+0000", 2023-01-15T14:31:24.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130684 2023-01-15T14:31:24.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945118+0000", 2023-01-15T14:31:24.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9194000000000001e-05 2023-01-15T14:31:24.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.714 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.714 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.714 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:705 3.2 3:47dfe935:::smithi09025264-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 135.53342531499999, 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61170208599999998, 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 705 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:24.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110583+0000", 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110587+0000", 2023-01-15T14:31:24.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9689999999999996e-06 2023-01-15T14:31:24.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110601+0000", 2023-01-15T14:31:24.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.393e-05 2023-01-15T14:31:24.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110611+0000", 2023-01-15T14:31:24.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3999999999999998e-06 2023-01-15T14:31:24.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319174+0000", 2023-01-15T14:31:24.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20856376800000001 2023-01-15T14:31:24.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319367+0000", 2023-01-15T14:31:24.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192361 2023-01-15T14:31:24.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.721941+0000", 2023-01-15T14:31:24.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40257461300000003 2023-01-15T14:31:24.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722108+0000", 2023-01-15T14:31:24.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167054 2023-01-15T14:31:24.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722286+0000", 2023-01-15T14:31:24.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017772099999999999 2023-01-15T14:31:24.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.721 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.721 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:24.721 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:24.721 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.721703241, 2023-01-15T14:31:24.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:31:24.722 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.722 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:24.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.722 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:31:24.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.722 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:24.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:24.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:31:24.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:31:24.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:31:24.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:31:24.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:31:24.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:31:24.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:31:24.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.727 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.715241887000005, 2023-01-15T14:31:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:31:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:31:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.728 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:31:24.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:31:24.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:31:24.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:31:24.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:31:24.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:31:24.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:31:24.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:31:24.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.614819961999999, 2023-01-15T14:31:24.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:24.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:31:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:31:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:31:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:31:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:31:24.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.738 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.614482476000006, 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:31:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:31:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:31:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:31:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:31:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:31:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:31:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:31:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:31:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:31:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:31:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:31:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:31:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:31:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:31:24.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:31:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.61442658, 2023-01-15T14:31:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:31:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:31:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:31:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:31:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:31:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:31:24.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:31:24.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:31:24.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:31:24.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:31:24.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:31:24.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:31:24.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:31:24.751 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.751 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.751 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1314 3.18 3.afe52a78 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.284131449, 2023-01-15T14:31:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62560938700000002, 2023-01-15T14:31:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:24.752 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.753 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1314 2023-01-15T14:31:24.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.753 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:24.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:24.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359877+0000", 2023-01-15T14:31:24.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:24.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359882+0000", 2023-01-15T14:31:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0050000000000004e-06 2023-01-15T14:31:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359893+0000", 2023-01-15T14:31:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0676000000000001e-05 2023-01-15T14:31:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359904+0000", 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0788000000000001e-05 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460094+0000", 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10019024999999999 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460106+0000", 2023-01-15T14:31:24.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354999999999999e-05 2023-01-15T14:31:24.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985477+0000", 2023-01-15T14:31:24.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52537039100000005 2023-01-15T14:31:24.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985487+0000", 2023-01-15T14:31:24.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:31:24.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.758 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.758 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1335 3.8 3:17d5d741:::smithi09025264-5:head [write 3252761~175867] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:24.758 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:24.758 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.712112188000006, 2023-01-15T14:31:24.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66052243499999996, 2023-01-15T14:31:24.759 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.759 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.759 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1335 2023-01-15T14:31:24.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.759 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:24.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:24.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931896+0000", 2023-01-15T14:31:24.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:24.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931983+0000", 2023-01-15T14:31:24.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6470000000000004e-05 2023-01-15T14:31:24.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931995+0000", 2023-01-15T14:31:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2075e-05 2023-01-15T14:31:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932003+0000", 2023-01-15T14:31:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5059999999999995e-06 2023-01-15T14:31:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932346+0000", 2023-01-15T14:31:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034280299999999998 2023-01-15T14:31:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932357+0000", 2023-01-15T14:31:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.13e-05 2023-01-15T14:31:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189006+0000", 2023-01-15T14:31:24.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25664859899999998 2023-01-15T14:31:24.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189336+0000", 2023-01-15T14:31:24.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033030899999999998 2023-01-15T14:31:24.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189481+0000", 2023-01-15T14:31:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014484400000000001 2023-01-15T14:31:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189768+0000", 2023-01-15T14:31:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028704300000000001 2023-01-15T14:31:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189787+0000", 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9343000000000001e-05 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191361+0000", 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015734029999999999 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191469+0000", 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010859899999999999 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391274+0000", 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19980450899999999 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592067+0000", 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20079332 2023-01-15T14:31:24.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592190+0000", 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012291400000000001 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592420+0000", 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000229371 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1336 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (56) in=7b,truncate 3428628] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.711995099000006, 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66113910799999998, 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.769 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1336 2023-01-15T14:31:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:24.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932013+0000", 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932019+0000", 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5380000000000002e-06 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932027+0000", 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5150000000000006e-06 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932034+0000", 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032497+0000", 2023-01-15T14:31:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100463001 2023-01-15T14:31:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032518+0000", 2023-01-15T14:31:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1359999999999999e-05 2023-01-15T14:31:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189826+0000", 2023-01-15T14:31:24.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15730834299999999 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189951+0000", 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124583 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190093+0000", 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014221399999999999 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190253+0000", 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015953000000000001 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191365+0000", 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011123190000000001 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191439+0000", 2023-01-15T14:31:24.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3648999999999999e-05 2023-01-15T14:31:24.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191491+0000", 2023-01-15T14:31:24.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2444999999999997e-05 2023-01-15T14:31:24.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391472+0000", 2023-01-15T14:31:24.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199981193 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592619+0000", 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20114642499999999 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592748+0000", 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012950200000000001 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593153+0000", 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040503700000000002 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1337 3.8 3:17d5d741:::smithi09025264-5:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 80.711966869999998, 2023-01-15T14:31:24.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66179168200000005, 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1337 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:24.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932046+0000", 2023-01-15T14:31:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8089999999999999e-06 2023-01-15T14:31:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932055+0000", 2023-01-15T14:31:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2199999999999998e-06 2023-01-15T14:31:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932061+0000", 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2779999999999999e-06 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190318+0000", 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.258256967 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190413+0000", 2023-01-15T14:31:24.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4627000000000001e-05 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593288+0000", 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.402875339 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593515+0000", 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022700699999999999 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593834+0000", 2023-01-15T14:31:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031906799999999999 2023-01-15T14:31:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.784 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1690 3.16 3:68fb3263:::smithi09025264-33:head [write 1763227~472137] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.781493900000001, 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72570748100000004, 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1690 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862514+0000", 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862746+0000", 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000232301 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862770+0000", 2023-01-15T14:31:24.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3286e-05 2023-01-15T14:31:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862779+0000", 2023-01-15T14:31:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4199999999999996e-06 2023-01-15T14:31:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862844+0000", 2023-01-15T14:31:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5036999999999994e-05 2023-01-15T14:31:24.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862860+0000", 2023-01-15T14:31:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6426000000000002e-05 2023-01-15T14:31:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051251+0000", 2023-01-15T14:31:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18839023299999999 2023-01-15T14:31:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051466+0000", 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021523199999999999 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368326+0000", 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31686002699999999 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:24.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368559+0000", 2023-01-15T14:31:24.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023305400000000001 2023-01-15T14:31:24.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:24.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368935+0000", 2023-01-15T14:31:24.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037579099999999999 2023-01-15T14:31:24.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:24.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369531+0000", 2023-01-15T14:31:24.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059593199999999997 2023-01-15T14:31:24.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:24.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369571+0000", 2023-01-15T14:31:24.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0649999999999999e-05 2023-01-15T14:31:24.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:24.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370331+0000", 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075969899999999996 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371230+0000", 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000899134 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472895+0000", 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101665191 2023-01-15T14:31:24.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.587914+0000", 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115018671 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588013+0000", 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8494999999999998e-05 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588223+0000", 2023-01-15T14:31:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210294 2023-01-15T14:31:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1691 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 2235364] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.781209646000001, 2023-01-15T14:31:24.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72593841699999995, 2023-01-15T14:31:24.796 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:24.796 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:24.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:24.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:24.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:24.796 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1691 2023-01-15T14:31:24.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.796 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:24.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:24.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862799+0000", 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:24.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862807+0000", 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1280000000000008e-06 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862821+0000", 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.345e-05 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862826+0000", 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3720000000000004e-06 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862883+0000", 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6589000000000003e-05 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862888+0000", 2023-01-15T14:31:24.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4060000000000004e-06 2023-01-15T14:31:24.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:24.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:24.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:25.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151601+0000", 2023-01-15T14:31:25.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28871354100000002 2023-01-15T14:31:25.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:25.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151732+0000", 2023-01-15T14:31:25.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013032300000000001 2023-01-15T14:31:25.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369729+0000", 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217997515 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369949+0000", 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219664 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:25.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370367+0000", 2023-01-15T14:31:25.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041822799999999998 2023-01-15T14:31:25.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:25.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370771+0000", 2023-01-15T14:31:25.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040358400000000002 2023-01-15T14:31:25.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370794+0000", 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-05 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370880+0000", 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6020999999999995e-05 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371241+0000", 2023-01-15T14:31:25.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000360956 2023-01-15T14:31:25.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:25.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472994+0000", 2023-01-15T14:31:25.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101753089 2023-01-15T14:31:25.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:25.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588348+0000", 2023-01-15T14:31:25.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11535382299999999 2023-01-15T14:31:25.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588539+0000", 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019090399999999999 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588738+0000", 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019910800000000001 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:25.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 44.781176162000001, 2023-01-15T14:31:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:31:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:25.148 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:25.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:25.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:25.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:25.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:25.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:25.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:25.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:31:25.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:31:25.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:25.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:31:25.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:31:25.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:25.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:31:25.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:31:25.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:25.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:31:25.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:31:25.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:25.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:31:25.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:31:25.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:25.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:31:25.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:31:25.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:25.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:31:25.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:31:25.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:25.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:31:25.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:31:25.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:25.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:31:25.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:31:25.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:25.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:31:25.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:31:25.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:25.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:31:25.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:31:25.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:25.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:25.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:25.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:31:25.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:31:25.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:25.156 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:25.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:25.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:25.156 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:25.156 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:25.158 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:31:25.352 INFO:tasks.rados.rados.0.smithi090.stdout:761: expect (ObjNum 206 snap 44 seq_num 206) 2023-01-15T14:31:25.352 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:31:25.354 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:25.154+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:25.355 INFO:tasks.rados.rados.0.smithi090.stdout:754: finishing write tid 5 to smithi09025264-26 2023-01-15T14:31:25.355 INFO:tasks.rados.rados.0.smithi090.stdout:754: oid 26 updating version 0 to 257 2023-01-15T14:31:25.356 INFO:tasks.rados.rados.0.smithi090.stdout:754: oid 26 updating version 257 to 258 2023-01-15T14:31:25.356 INFO:tasks.rados.rados.0.smithi090.stdout:754: oid 26 version 258 is already newer than 256 2023-01-15T14:31:25.356 INFO:tasks.rados.rados.0.smithi090.stdout:754: oid 26 updating version 258 to 259 2023-01-15T14:31:25.356 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 259 (ObjNum 295 snap 78 seq_num 295) dirty exists 2023-01-15T14:31:25.356 INFO:tasks.rados.rados.0.smithi090.stdout:754: left oid 26 (ObjNum 295 snap 78 seq_num 295) 2023-01-15T14:31:25.356 INFO:tasks.rados.rados.0.smithi090.stdout:754: done (4 left) 2023-01-15T14:31:25.356 INFO:tasks.rados.rados.0.smithi090.stdout:761: done (3 left) 2023-01-15T14:31:25.356 INFO:tasks.rados.rados.0.smithi090.stdout:763: done (2 left) 2023-01-15T14:31:25.356 INFO:tasks.rados.rados.0.smithi090.stdout:764: write oid 13 current snap is 78 2023-01-15T14:31:25.356 INFO:tasks.rados.rados.0.smithi090.stdout:760: finishing write tid 1 to smithi09025264-10 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:762: finishing write tid 1 to smithi09025264-2 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:760: finishing write tid 2 to smithi09025264-10 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:762: finishing write tid 2 to smithi09025264-2 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:760: finishing write tid 3 to smithi09025264-10 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:760: finishing write tid 4 to smithi09025264-10 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:762: finishing write tid 3 to smithi09025264-2 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:764: seq_num 300 ranges {642100=712467,1872245=637761} 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:764: writing smithi09025264-13 from 642100 to 1354567 tid 1 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:764: writing smithi09025264-13 from 1872245 to 2510006 tid 2 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:765: delete oid 20 current snap is 78 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:760: finishing write tid 5 to smithi09025264-10 2023-01-15T14:31:25.357 INFO:tasks.rados.rados.0.smithi090.stdout:760: oid 10 updating version 0 to 241 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:760: oid 10 version 241 is already newer than 240 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:760: oid 10 updating version 241 to 242 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:760: oid 10 updating version 242 to 243 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 243 (ObjNum 298 snap 78 seq_num 298) dirty exists 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:760: left oid 10 (ObjNum 298 snap 78 seq_num 298) 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:762: finishing write tid 4 to smithi09025264-2 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:762: finishing write tid 5 to smithi09025264-2 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:762: oid 2 updating version 0 to 235 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:762: oid 2 version 235 is already newer than 233 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:762: oid 2 version 235 is already newer than 234 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:762: oid 2 updating version 235 to 236 2023-01-15T14:31:25.358 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 236 (ObjNum 299 snap 78 seq_num 299) dirty exists 2023-01-15T14:31:25.359 INFO:tasks.rados.rados.0.smithi090.stdout:762: left oid 2 (ObjNum 299 snap 78 seq_num 299) 2023-01-15T14:31:25.359 INFO:tasks.rados.rados.0.smithi090.stdout:760: done (3 left) 2023-01-15T14:31:25.359 INFO:tasks.rados.rados.0.smithi090.stdout:762: done (2 left) 2023-01-15T14:31:25.359 INFO:tasks.rados.rados.0.smithi090.stdout:765: done (1 left) 2023-01-15T14:31:25.359 INFO:tasks.rados.rados.0.smithi090.stdout:766: write (excl) oid 11 current snap is 78 2023-01-15T14:31:25.359 INFO:tasks.rados.rados.0.smithi090.stdout:766: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2023-01-15T14:31:25.359 INFO:tasks.rados.rados.0.smithi090.stdout:766: writing smithi09025264-11 from 675714 to 1138679 tid 1 2023-01-15T14:31:25.359 INFO:tasks.rados.rados.0.smithi090.stdout:766: writing smithi09025264-11 from 1589209 to 2230639 tid 2 2023-01-15T14:31:25.359 INFO:tasks.rados.rados.0.smithi090.stdout:766: writing smithi09025264-11 from 2943979 to 3075714 tid 3 2023-01-15T14:31:25.359 INFO:tasks.rados.rados.0.smithi090.stdout:767: delete oid 38 current snap is 78 2023-01-15T14:31:25.359 INFO:tasks.rados.rados.0.smithi090.stdout:764: finishing write tid 1 to smithi09025264-13 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:764: finishing write tid 2 to smithi09025264-13 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:764: finishing write tid 3 to smithi09025264-13 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:764: finishing write tid 4 to smithi09025264-13 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:764: oid 13 updating version 0 to 224 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:764: oid 13 updating version 224 to 225 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:764: oid 13 version 225 is already newer than 223 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 225 (ObjNum 300 snap 78 seq_num 300) dirty exists 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:764: left oid 13 (ObjNum 300 snap 78 seq_num 300) 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:764: done (2 left) 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:767: done (1 left) 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:768: read oid 7 snap -1 2023-01-15T14:31:25.360 INFO:tasks.rados.rados.0.smithi090.stdout:768: expect deleted 2023-01-15T14:31:25.361 INFO:tasks.rados.rados.0.smithi090.stdout:769: delete oid 27 current snap is 78 2023-01-15T14:31:25.361 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:25.350+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:25.441 INFO:tasks.rados.rados.0.smithi090.stdout:766: finishing write tid 1 to smithi09025264-11 2023-01-15T14:31:25.759 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:25.759 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:31:25.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.759 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2024 3.3 3:c11c7d81:::smithi09025264-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:25.760 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:25.151319+0000", 2023-01-15T14:31:25.760 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.31666621900000003, 2023-01-15T14:31:25.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31671185099999999, 2023-01-15T14:31:25.760 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:25.760 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:31:25.760 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:25.760 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:25.760 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:25.760 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2024 2023-01-15T14:31:25.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.760 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:25.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:25.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151319+0000", 2023-01-15T14:31:25.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:25.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151319+0000", 2023-01-15T14:31:25.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151318+0000", 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151324+0000", 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.3750000000000002e-06 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:25.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151338+0000", 2023-01-15T14:31:25.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4600000000000001e-05 2023-01-15T14:31:25.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:25.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151347+0000", 2023-01-15T14:31:25.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7490000000000004e-06 2023-01-15T14:31:25.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:25.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.239481+0000", 2023-01-15T14:31:25.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.088134303999999997 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.239757+0000", 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027616099999999997 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444963+0000", 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.205205792 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.445242+0000", 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027832699999999998 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:2029 3.d e198/193 3:b2a32959:::smithi09025264-27:head v 198'164, mlcod=197'160)", 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:25.444500+0000", 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.023484927999999999, 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.023678899999999999, 2023-01-15T14:31:25.765 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:25.766 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:25.766 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:25.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:25.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444500+0000", 2023-01-15T14:31:25.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:25.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444500+0000", 2023-01-15T14:31:25.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444499+0000", 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444511+0000", 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1596999999999999e-05 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:25.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444535+0000", 2023-01-15T14:31:25.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4269000000000001e-05 2023-01-15T14:31:25.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:25.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444549+0000", 2023-01-15T14:31:25.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3859999999999999e-05 2023-01-15T14:31:25.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:25.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444682+0000", 2023-01-15T14:31:25.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000133079 2023-01-15T14:31:25.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:25.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444750+0000", 2023-01-15T14:31:25.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8221999999999998e-05 2023-01-15T14:31:25.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.445487+0000", 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073716199999999997 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.445505+0000", 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7638e-05 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:25.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.445629+0000", 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000123831 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.445664+0000", 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5398999999999998e-05 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.771 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:2022 3.3 e198/175)", 2023-01-15T14:31:25.772 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:25.444793+0000", 2023-01-15T14:31:25.772 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.023192154999999999, 2023-01-15T14:31:25.772 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.023505325000000001, 2023-01-15T14:31:25.772 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:25.772 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:25.772 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:25.772 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.772 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:25.772 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444793+0000", 2023-01-15T14:31:25.772 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.772 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.773 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.773 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:25.773 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444793+0000", 2023-01-15T14:31:25.773 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.773 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.773 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.773 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:25.773 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444792+0000", 2023-01-15T14:31:25.773 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:25.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:25.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444799+0000", 2023-01-15T14:31:25.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1689999999999997e-06 2023-01-15T14:31:25.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:25.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444831+0000", 2023-01-15T14:31:25.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2005000000000003e-05 2023-01-15T14:31:25.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444843+0000", 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2184999999999999e-05 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2021 3.3 3:c11c7d81:::smithi09025264-11:head [write 1589209~641430 in=641430b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:25.146302+0000", 2023-01-15T14:31:25.775 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.32168349200000002, 2023-01-15T14:31:25.776 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32207422400000002, 2023-01-15T14:31:25.776 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:25.776 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:31:25.776 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:25.776 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:25.776 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:25.776 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2021 2023-01-15T14:31:25.776 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.776 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:25.776 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.776 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.146302+0000", 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.146302+0000", 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.146301+0000", 2023-01-15T14:31:25.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:25.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:25.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.146889+0000", 2023-01-15T14:31:25.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058781899999999995 2023-01-15T14:31:25.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:25.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.146906+0000", 2023-01-15T14:31:25.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6925000000000001e-05 2023-01-15T14:31:25.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.146916+0000", 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9299999999999998e-06 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.238141+0000", 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.091225424999999999 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:25.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.238394+0000", 2023-01-15T14:31:25.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025304899999999998 2023-01-15T14:31:25.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.959 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.959 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:25.959 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.441312+0000", 2023-01-15T14:31:25.960 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20291837300000001 2023-01-15T14:31:25.960 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.960 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.960 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:25.960 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.441496+0000", 2023-01-15T14:31:25.960 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018317999999999999 2023-01-15T14:31:25.960 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.960 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.960 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:25.960 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.441811+0000", 2023-01-15T14:31:25.960 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00031523 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.442300+0000", 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048933099999999997 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.442317+0000", 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6611000000000001e-05 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.961 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.962 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:25.962 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.442621+0000", 2023-01-15T14:31:25.962 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030382 2023-01-15T14:31:25.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.962 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.443332+0000", 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071171400000000001 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.445363+0000", 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020302969999999999 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:25.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.964 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.964 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2022 3.3 3:c11c7d81:::smithi09025264-11:head [write 2943979~131735 in=131735b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:25.964 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:25.151091+0000", 2023-01-15T14:31:25.964 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.31689403199999999, 2023-01-15T14:31:25.964 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31745154399999997, 2023-01-15T14:31:25.964 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:25.964 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:31:25.964 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:25.964 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:25.964 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:25.964 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2022 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151091+0000", 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151091+0000", 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151090+0000", 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151208+0000", 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118596 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.966 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151238+0000", 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.9629e-05 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151253+0000", 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4581e-05 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.238782+0000", 2023-01-15T14:31:25.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087529261999999997 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.239154+0000", 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000371753 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.442421+0000", 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20326712999999999 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:25.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.442588+0000", 2023-01-15T14:31:25.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016757299999999999 2023-01-15T14:31:25.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:25.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.442904+0000", 2023-01-15T14:31:25.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00031584400000000002 2023-01-15T14:31:25.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.443219+0000", 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000314751 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.443271+0000", 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1761999999999997e-05 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.443390+0000", 2023-01-15T14:31:25.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011937700000000001 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444197+0000", 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00080639800000000005 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2023 3.3 3:c11c7d81:::smithi09025264-11:head [setxattr _header (57) in=64b,truncate 3075714] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:25.971 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:25.151272+0000", 2023-01-15T14:31:25.972 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.316713089, 2023-01-15T14:31:25.972 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31744327100000003, 2023-01-15T14:31:25.972 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:25.972 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:31:25.972 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:25.972 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:25.972 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:25.972 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2023 2023-01-15T14:31:25.972 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.972 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:25.972 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151272+0000", 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151272+0000", 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:25.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151271+0000", 2023-01-15T14:31:25.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:25.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.974 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.974 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:25.974 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151279+0000", 2023-01-15T14:31:25.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3459999999999994e-06 2023-01-15T14:31:25.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:25.974 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:25.974 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:25.974 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151296+0000", 2023-01-15T14:31:25.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6653000000000001e-05 2023-01-15T14:31:25.975 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:26.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.151306+0000", 2023-01-15T14:31:26.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8179999999999999e-06 2023-01-15T14:31:26.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.239250+0000", 2023-01-15T14:31:26.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087944226 2023-01-15T14:31:26.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.239411+0000", 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016123400000000001 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.443387+0000", 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.203975931 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.443546+0000", 2023-01-15T14:31:26.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000158617 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.443763+0000", 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021765200000000001 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444098+0000", 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033508300000000002 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:26.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444679+0000", 2023-01-15T14:31:26.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058073200000000004 2023-01-15T14:31:26.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:26.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444769+0000", 2023-01-15T14:31:26.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0094999999999997e-05 2023-01-15T14:31:26.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:26.169 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.444864+0000", 2023-01-15T14:31:26.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4884999999999997e-05 2023-01-15T14:31:26.169 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.169 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:26.169 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.169 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.169 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2023-01-15T14:31:26.169 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 6 2023-01-15T14:31:26.169 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:26.170 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:31:26.170 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:25.754+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:26.176 INFO:tasks.rados.rados.0.smithi090.stdout:766: finishing write tid 2 to smithi09025264-11 2023-01-15T14:31:26.176 INFO:tasks.rados.rados.0.smithi090.stdout:766: finishing write tid 3 to smithi09025264-11 2023-01-15T14:31:26.176 INFO:tasks.rados.rados.0.smithi090.stdout:768: done (2 left) 2023-01-15T14:31:26.176 INFO:tasks.rados.rados.0.smithi090.stdout:769: done (1 left) 2023-01-15T14:31:26.176 INFO:tasks.rados.rados.0.smithi090.stdout:770: rollback oid 30 current snap is 78 2023-01-15T14:31:26.176 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 30 to 72 2023-01-15T14:31:26.176 INFO:tasks.rados.rados.0.smithi090.stdout:771: read oid 39 snap -1 2023-01-15T14:31:26.176 INFO:tasks.rados.rados.0.smithi090.stdout:771: expect deleted 2023-01-15T14:31:26.176 INFO:tasks.rados.rados.0.smithi090.stdout:772: delete oid 49 current snap is 78 2023-01-15T14:31:26.176 INFO:tasks.rados.rados.0.smithi090.stdout:766: finishing write tid 4 to smithi09025264-11 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:766: finishing write tid 5 to smithi09025264-11 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:766: oid 11 updating version 0 to 261 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:766: oid 11 updating version 261 to 263 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:766: oid 11 version 263 is already newer than 262 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:766: oid 11 version 263 is already newer than 260 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 263 (ObjNum 302 snap 78 seq_num 302) dirty exists 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:766: left oid 11 (ObjNum 302 snap 78 seq_num 302) 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:766: done (3 left) 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:771: done (2 left) 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:772: done (1 left) 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:773: copy_from oid 8 from oid 35 current snap is 78 2023-01-15T14:31:26.177 INFO:tasks.rados.rados.0.smithi090.stdout:774: rollback oid 46 current snap is 78 2023-01-15T14:31:26.178 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 46 to 76 2023-01-15T14:31:26.178 INFO:tasks.rados.rados.0.smithi090.stdout:775: write (excl) oid 3 current snap is 78 2023-01-15T14:31:26.178 INFO:tasks.rados.rados.0.smithi090.stdout:775: seq_num 306 ranges {742942=480314,1823137=648768} 2023-01-15T14:31:26.178 INFO:tasks.rados.rados.0.smithi090.stdout:775: writing smithi09025264-3 from 742942 to 1223256 tid 1 2023-01-15T14:31:26.178 INFO:tasks.rados.rados.0.smithi090.stdout:775: writing smithi09025264-3 from 1823137 to 2471905 tid 2 2023-01-15T14:31:26.178 INFO:tasks.rados.rados.0.smithi090.stdout:776: write (excl) oid 19 current snap is 78 2023-01-15T14:31:26.178 INFO:tasks.rados.rados.0.smithi090.stdout:770: finishing rollback tid 0 to smithi09025264-30 2023-01-15T14:31:26.178 INFO:tasks.rados.rados.0.smithi090.stdout:776: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2023-01-15T14:31:26.178 INFO:tasks.rados.rados.0.smithi090.stdout:776: writing smithi09025264-19 from 759749 to 1315312 tid 1 2023-01-15T14:31:26.178 INFO:tasks.rados.rados.0.smithi090.stdout:776: writing smithi09025264-19 from 1765266 to 2557692 tid 2 2023-01-15T14:31:26.178 INFO:tasks.rados.rados.0.smithi090.stdout:776: writing smithi09025264-19 from 2964741 to 3159749 tid 3 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:777: copy_from oid 7 from oid 38 current snap is 78 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:770: finishing rollback tid 1 to smithi09025264-30 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:778: rollback oid 50 current snap is 78 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 50 to 72 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:779: delete oid 16 current snap is 78 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:779: done (7 left) 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:780: write oid 13 current snap is 78 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:780: seq_num 309 ranges {793363=706061,2282230=796095} 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:780: writing smithi09025264-13 from 793363 to 1499424 tid 1 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:780: writing smithi09025264-13 from 2282230 to 3078325 tid 2 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:781: read oid 1 snap -1 2023-01-15T14:31:26.179 INFO:tasks.rados.rados.0.smithi090.stdout:781: expect deleted 2023-01-15T14:31:26.180 INFO:tasks.rados.rados.0.smithi090.stdout:782: read oid 16 snap 71 2023-01-15T14:31:26.180 INFO:tasks.rados.rados.0.smithi090.stdout:782: expect deleted 2023-01-15T14:31:26.180 INFO:tasks.rados.rados.0.smithi090.stdout:783: read oid 43 snap -1 2023-01-15T14:31:26.180 INFO:tasks.rados.rados.0.smithi090.stdout:783: expect deleted 2023-01-15T14:31:26.180 INFO:tasks.rados.rados.0.smithi090.stdout:784: read oid 40 snap -1 2023-01-15T14:31:26.180 INFO:tasks.rados.rados.0.smithi090.stdout:784: expect (ObjNum 284 snap 74 seq_num 284) 2023-01-15T14:31:26.180 INFO:tasks.rados.rados.0.smithi090.stdout:785: snap_create 2023-01-15T14:31:26.180 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 12 2023-01-15T14:31:26.180 INFO:tasks.rados.rados.0.smithi090.stdout:777: finishing copy_from to smithi09025264-7 2023-01-15T14:31:26.180 INFO:tasks.rados.rados.0.smithi090.stdout:777: got expected ENOENT (src dne) 2023-01-15T14:31:26.180 INFO:tasks.rados.rados.0.smithi090.stdout:777: finishing copy_from racing read to smithi09025264-7 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout:777: done (11 left) 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 11 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout:774: finishing rollback tid 0 to smithi09025264-46 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout:774: finishing rollback tid 1 to smithi09025264-46 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout:776: finishing write tid 1 to smithi09025264-19 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout:778: finishing rollback tid 0 to smithi09025264-50 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout:778: finishing rollback tid 1 to smithi09025264-50 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout:782: done (10 left) 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 10 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout:781: done (9 left) 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 9 2023-01-15T14:31:26.181 INFO:tasks.rados.rados.0.smithi090.stdout:770: finishing rollback tid 2 to smithi09025264-30 2023-01-15T14:31:26.182 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 85 (ObjNum 206 snap 44 seq_num 206) dirty exists 2023-01-15T14:31:26.182 INFO:tasks.rados.rados.0.smithi090.stdout:770: done (8 left) 2023-01-15T14:31:26.182 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:31:26.182 INFO:tasks.rados.rados.0.smithi090.stdout:783: done (7 left) 2023-01-15T14:31:26.182 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:31:26.182 INFO:tasks.rados.rados.0.smithi090.stdout:775: finishing write tid 1 to smithi09025264-3 2023-01-15T14:31:26.182 INFO:tasks.rados.rados.0.smithi090.stdout:775: finishing write tid 2 to smithi09025264-3 2023-01-15T14:31:26.182 INFO:tasks.rados.rados.0.smithi090.stdout:775: finishing write tid 3 to smithi09025264-3 2023-01-15T14:31:26.182 INFO:tasks.rados.rados.0.smithi090.stdout:780: finishing write tid 1 to smithi09025264-13 2023-01-15T14:31:26.183 INFO:tasks.rados.rados.0.smithi090.stdout:778: finishing rollback tid 2 to smithi09025264-50 2023-01-15T14:31:26.183 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 50 v 160 (ObjNum 170 snap 37 seq_num 170) dirty exists 2023-01-15T14:31:26.183 INFO:tasks.rados.rados.0.smithi090.stdout:778: done (6 left) 2023-01-15T14:31:26.183 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:31:26.183 INFO:tasks.rados.rados.0.smithi090.stdout:774: finishing rollback tid 2 to smithi09025264-46 2023-01-15T14:31:26.183 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 211 (ObjNum 256 snap 60 seq_num 256) dirty exists 2023-01-15T14:31:26.183 INFO:tasks.rados.rados.0.smithi090.stdout:774: done (5 left) 2023-01-15T14:31:26.183 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:31:26.183 INFO:tasks.rados.rados.0.smithi090.stdout:773: finishing copy_from to smithi09025264-8 2023-01-15T14:31:26.183 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 225 (ObjNum 287 snap 74 seq_num 287) dirty exists 2023-01-15T14:31:26.183 INFO:tasks.rados.rados.0.smithi090.stdout:775: finishing write tid 4 to smithi09025264-3 2023-01-15T14:31:26.184 INFO:tasks.rados.rados.0.smithi090.stdout:775: oid 3 updating version 0 to 218 2023-01-15T14:31:26.184 INFO:tasks.rados.rados.0.smithi090.stdout:775: oid 3 version 218 is already newer than 217 2023-01-15T14:31:26.184 INFO:tasks.rados.rados.0.smithi090.stdout:775: oid 3 version 218 is already newer than 216 2023-01-15T14:31:26.184 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 218 (ObjNum 306 snap 78 seq_num 306) dirty exists 2023-01-15T14:31:26.184 INFO:tasks.rados.rados.0.smithi090.stdout:775: left oid 3 (ObjNum 306 snap 78 seq_num 306) 2023-01-15T14:31:26.184 INFO:tasks.rados.rados.0.smithi090.stdout:775: done (4 left) 2023-01-15T14:31:26.184 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:31:26.184 INFO:tasks.rados.rados.0.smithi090.stdout:780: finishing write tid 2 to smithi09025264-13 2023-01-15T14:31:26.184 INFO:tasks.rados.rados.0.smithi090.stdout:776: finishing write tid 2 to smithi09025264-19 2023-01-15T14:31:26.184 INFO:tasks.rados.rados.0.smithi090.stdout:776: finishing write tid 3 to smithi09025264-19 2023-01-15T14:31:26.184 INFO:tasks.rados.rados.0.smithi090.stdout:773: finishing copy_from racing read to smithi09025264-8 2023-01-15T14:31:26.373 INFO:tasks.rados.rados.0.smithi090.stdout:773: done (3 left) 2023-01-15T14:31:26.373 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:31:26.373 INFO:tasks.rados.rados.0.smithi090.stdout:776: finishing write tid 4 to smithi09025264-19 2023-01-15T14:31:26.373 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:26.167+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:26.374 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:31:26.376 INFO:tasks.rados.rados.0.smithi090.stdout:780: finishing write tid 3 to smithi09025264-13 2023-01-15T14:31:26.376 INFO:tasks.rados.rados.0.smithi090.stdout:776: finishing write tid 5 to smithi09025264-19 2023-01-15T14:31:26.376 INFO:tasks.rados.rados.0.smithi090.stdout:776: oid 19 updating version 0 to 163 2023-01-15T14:31:26.376 INFO:tasks.rados.rados.0.smithi090.stdout:776: oid 19 version 163 is already newer than 162 2023-01-15T14:31:26.376 INFO:tasks.rados.rados.0.smithi090.stdout:776: oid 19 version 163 is already newer than 156 2023-01-15T14:31:26.376 INFO:tasks.rados.rados.0.smithi090.stdout:776: oid 19 version 163 is already newer than 161 2023-01-15T14:31:26.376 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 163 (ObjNum 307 snap 78 seq_num 307) dirty exists 2023-01-15T14:31:26.377 INFO:tasks.rados.rados.0.smithi090.stdout:776: left oid 19 (ObjNum 307 snap 78 seq_num 307) 2023-01-15T14:31:26.377 INFO:tasks.rados.rados.0.smithi090.stdout:776: done (2 left) 2023-01-15T14:31:26.377 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:26.377 INFO:tasks.rados.rados.0.smithi090.stdout:780: finishing write tid 4 to smithi09025264-13 2023-01-15T14:31:26.377 INFO:tasks.rados.rados.0.smithi090.stdout:780: oid 13 updating version 0 to 227 2023-01-15T14:31:26.377 INFO:tasks.rados.rados.0.smithi090.stdout:780: oid 13 version 227 is already newer than 226 2023-01-15T14:31:26.377 INFO:tasks.rados.rados.0.smithi090.stdout:780: oid 13 updating version 227 to 228 2023-01-15T14:31:26.377 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 228 (ObjNum 309 snap 78 seq_num 309) dirty exists 2023-01-15T14:31:26.377 INFO:tasks.rados.rados.0.smithi090.stdout:780: left oid 13 (ObjNum 309 snap 78 seq_num 309) 2023-01-15T14:31:26.377 INFO:tasks.rados.rados.0.smithi090.stdout:780: done (1 left) 2023-01-15T14:31:26.377 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:26.378 INFO:tasks.rados.rados.0.smithi090.stdout:784: expect (ObjNum 284 snap 74 seq_num 284) 2023-01-15T14:31:26.378 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:26.378 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:31:26.378 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:31:26.379 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:31:26.379 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:26.379 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:31:26.379 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:26.374+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:26.440 INFO:tasks.rados.rados.0.smithi090.stdout:784: done (0 left) 2023-01-15T14:31:26.577 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:26.577 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:26.577 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:26.577 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:26.571+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:26.578 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:26.579 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:31:26.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:31:26.579 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:31:26.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.579 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3 3.3 3:ca59e2b0:::smithi09025264-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:31:26.579 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:31:26.579 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 180.75667448900001, 2023-01-15T14:31:26.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62968594499999997, 2023-01-15T14:31:26.579 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:26.579 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775131+0000", 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:26.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775138+0000", 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775148+0000", 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7969999999999995e-06 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775161+0000", 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2469999999999999e-05 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.857974+0000", 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.082812887000000002 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.858147+0000", 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000173883 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106098+0000", 2023-01-15T14:31:26.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24795019600000001 2023-01-15T14:31:26.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:26.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106181+0000", 2023-01-15T14:31:26.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3248e-05 2023-01-15T14:31:26.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:31:26.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106330+0000", 2023-01-15T14:31:26.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014918200000000001 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106510+0000", 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017979399999999999 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106527+0000", 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6643999999999999e-05 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106605+0000", 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8650000000000001e-05 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106644+0000", 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8686999999999997e-05 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:26.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404687+0000", 2023-01-15T14:31:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.298042745 2023-01-15T14:31:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404735+0000", 2023-01-15T14:31:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8306000000000001e-05 2023-01-15T14:31:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404772+0000", 2023-01-15T14:31:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6977000000000002e-05 2023-01-15T14:31:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404818+0000", 2023-01-15T14:31:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.6439e-05 2023-01-15T14:31:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:31:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:31:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 180.75663642200001, 2023-01-15T14:31:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62995383500000002, 2023-01-15T14:31:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:31:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:31:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775174+0000", 2023-01-15T14:31:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3669999999999999e-06 2023-01-15T14:31:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775184+0000", 2023-01-15T14:31:26.782 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5869999999999999e-06 2023-01-15T14:31:26.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:26.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775190+0000", 2023-01-15T14:31:26.782 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:31:26.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868440+0000", 2023-01-15T14:31:26.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.093250126000000003 2023-01-15T14:31:26.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868555+0000", 2023-01-15T14:31:26.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115003 2023-01-15T14:31:26.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.302962+0000", 2023-01-15T14:31:26.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.43440729700000003 2023-01-15T14:31:26.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303197+0000", 2023-01-15T14:31:26.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023508000000000001 2023-01-15T14:31:26.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404866+0000", 2023-01-15T14:31:26.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101668723 2023-01-15T14:31:26.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:26.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404985+0000", 2023-01-15T14:31:26.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118853 2023-01-15T14:31:26.786 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.786 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.786 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:26.786 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.405124+0000", 2023-01-15T14:31:26.786 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013951999999999999 2023-01-15T14:31:26.786 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.786 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:26.786 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.787 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:142 3.9 3:9988ac9d:::smithi09025264-36:head [write 1865105~489678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:26.787 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.865978+0000", 2023-01-15T14:31:26.787 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 179.66582857700001, 2023-01-15T14:31:26.787 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.53392634699999997, 2023-01-15T14:31:26.787 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:26.787 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:26.788 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:26.788 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:26.788 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:26.788 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 142 2023-01-15T14:31:26.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.788 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:26.788 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.788 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:26.788 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.865978+0000", 2023-01-15T14:31:26.788 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:26.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.865978+0000", 2023-01-15T14:31:26.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:26.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.865977+0000", 2023-01-15T14:31:26.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:26.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866409+0000", 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043200999999999998 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866429+0000", 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9636999999999999e-05 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:26.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866437+0000", 2023-01-15T14:31:26.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8829999999999995e-06 2023-01-15T14:31:26.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867463+0000", 2023-01-15T14:31:26.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001025584 2023-01-15T14:31:26.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867556+0000", 2023-01-15T14:31:26.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2477000000000002e-05 2023-01-15T14:31:26.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096010+0000", 2023-01-15T14:31:26.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.228454887 2023-01-15T14:31:26.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096093+0000", 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2503999999999995e-05 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096231+0000", 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013782599999999999 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096643+0000", 2023-01-15T14:31:26.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041226199999999999 2023-01-15T14:31:26.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:26.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096696+0000", 2023-01-15T14:31:26.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2960000000000001e-05 2023-01-15T14:31:26.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:26.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.096941+0000", 2023-01-15T14:31:26.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024520199999999997 2023-01-15T14:31:26.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:26.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097190+0000", 2023-01-15T14:31:26.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024874899999999999 2023-01-15T14:31:26.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:26.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.199089+0000", 2023-01-15T14:31:26.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101899288 2023-01-15T14:31:26.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.796 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:26.796 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.399771+0000", 2023-01-15T14:31:26.796 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20068177500000001 2023-01-15T14:31:26.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.796 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:26.796 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.399849+0000", 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7713000000000006e-05 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.399905+0000", 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5949999999999998e-05 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.797 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:144 3.9 3:9988ac9d:::smithi09025264-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:26.798 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:31:26.798 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 179.66529787600001, 2023-01-15T14:31:26.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63423976199999998, 2023-01-15T14:31:26.798 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:26.798 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:26.798 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:26.798 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:26.798 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:26.798 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 144 2023-01-15T14:31:26.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.798 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:26.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.799 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:26.799 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:31:26.799 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.799 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:26.799 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:31:26.799 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.800 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:26.800 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866508+0000", 2023-01-15T14:31:26.800 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:26.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.800 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.800 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:26.800 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866512+0000", 2023-01-15T14:31:26.800 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5999999999999998e-06 2023-01-15T14:31:26.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.800 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.800 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:26.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866523+0000", 2023-01-15T14:31:26.801 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0740000000000001e-05 2023-01-15T14:31:26.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:26.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866531+0000", 2023-01-15T14:31:26.801 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3310000000000004e-06 2023-01-15T14:31:26.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867744+0000", 2023-01-15T14:31:26.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012127399999999999 2023-01-15T14:31:26.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867810+0000", 2023-01-15T14:31:26.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6506000000000003e-05 2023-01-15T14:31:26.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097225+0000", 2023-01-15T14:31:26.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.22941514900000001 2023-01-15T14:31:26.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097295+0000", 2023-01-15T14:31:26.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:31:26.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500386+0000", 2023-01-15T14:31:26.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40309051699999998 2023-01-15T14:31:26.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:26.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500521+0000", 2023-01-15T14:31:26.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013561700000000001 2023-01-15T14:31:26.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:26.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500749+0000", 2023-01-15T14:31:26.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022754900000000001 2023-01-15T14:31:26.805 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.805 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:26.805 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.805 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:210 3.9 3:9e7635b9:::smithi09025264-21:head [setxattr _header (56) in=7b,truncate 2857157] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:26.805 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.510600+0000", 2023-01-15T14:31:26.805 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 179.02120720600001, 2023-01-15T14:31:26.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.59972464199999997, 2023-01-15T14:31:26.806 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:26.806 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:26.806 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:26.806 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:26.806 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:26.806 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 210 2023-01-15T14:31:26.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.806 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:26.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:26.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510600+0000", 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510600+0000", 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510598+0000", 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:26.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:26.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510605+0000", 2023-01-15T14:31:26.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2069999999999998e-06 2023-01-15T14:31:26.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:26.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510617+0000", 2023-01-15T14:31:26.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1644000000000001e-05 2023-01-15T14:31:26.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510625+0000", 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9109999999999993e-06 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510857+0000", 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023164600000000001 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510950+0000", 2023-01-15T14:31:26.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3145000000000004e-05 2023-01-15T14:31:26.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.707435+0000", 2023-01-15T14:31:26.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19648538800000001 2023-01-15T14:31:26.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:26.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.707681+0000", 2023-01-15T14:31:26.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024560899999999999 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708063+0000", 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038193199999999997 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708415+0000", 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000352042 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:26.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708429+0000", 2023-01-15T14:31:26.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.469e-05 2023-01-15T14:31:26.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:26.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708780+0000", 2023-01-15T14:31:26.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035064900000000002 2023-01-15T14:31:26.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:26.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708828+0000", 2023-01-15T14:31:26.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7756999999999997e-05 2023-01-15T14:31:26.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:26.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.809761+0000", 2023-01-15T14:31:26.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100933525 2023-01-15T14:31:26.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:26.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110150+0000", 2023-01-15T14:31:26.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30038901600000001 2023-01-15T14:31:26.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:26.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110226+0000", 2023-01-15T14:31:26.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5475999999999994e-05 2023-01-15T14:31:26.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:26.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110324+0000", 2023-01-15T14:31:26.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8583000000000001e-05 2023-01-15T14:31:26.815 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.815 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:26.815 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.815 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:211 3.9 3:9e7635b9:::smithi09025264-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:26.815 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.510639+0000", 2023-01-15T14:31:26.816 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 179.02116795699999, 2023-01-15T14:31:26.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60010284700000005, 2023-01-15T14:31:26.816 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:26.816 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:26.816 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:26.816 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:26.816 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:26.816 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 211 2023-01-15T14:31:26.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510639+0000", 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510639+0000", 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:26.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510638+0000", 2023-01-15T14:31:26.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:26.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:26.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510643+0000", 2023-01-15T14:31:26.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3340000000000003e-06 2023-01-15T14:31:26.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510653+0000", 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0961999999999999e-05 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510663+0000", 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4800000000000007e-06 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.601338+0000", 2023-01-15T14:31:26.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090674642 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.601480+0000", 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014286299999999999 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708540+0000", 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.107059404 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708676+0000", 2023-01-15T14:31:26.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013605 2023-01-15T14:31:26.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110423+0000", 2023-01-15T14:31:26.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40174706199999999 2023-01-15T14:31:26.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:26.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110590+0000", 2023-01-15T14:31:26.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000166773 2023-01-15T14:31:26.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:26.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110742+0000", 2023-01-15T14:31:26.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000152022 2023-01-15T14:31:26.823 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.823 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:26.823 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.823 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:508 3.9 3:9f86bdcb:::smithi09025264-22:head [write 1808051~636013] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.809136+0000", 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 156.72267051200001, 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60160218099999996, 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 508 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:26.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809136+0000", 2023-01-15T14:31:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809136+0000", 2023-01-15T14:31:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:26.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809135+0000", 2023-01-15T14:31:26.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:26.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:26.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809776+0000", 2023-01-15T14:31:26.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00064178800000000002 2023-01-15T14:31:26.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:26.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809801+0000", 2023-01-15T14:31:26.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4851999999999999e-05 2023-01-15T14:31:26.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:26.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809811+0000", 2023-01-15T14:31:26.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0285000000000001e-05 2023-01-15T14:31:26.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.903875+0000", 2023-01-15T14:31:26.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.094063472999999995 2023-01-15T14:31:26.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.984 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.984 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:26.984 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.903888+0000", 2023-01-15T14:31:26.984 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.27e-05 2023-01-15T14:31:26.984 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.984 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.984 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906243+0000", 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023554980000000001 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906404+0000", 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000160705 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.207137+0000", 2023-01-15T14:31:26.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30073284700000003 2023-01-15T14:31:26.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.986 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:26.986 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.207316+0000", 2023-01-15T14:31:26.986 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000179125 2023-01-15T14:31:26.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.986 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:31:26.986 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.207563+0000", 2023-01-15T14:31:26.986 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024729599999999999 2023-01-15T14:31:26.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.987 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.987 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:26.987 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208190+0000", 2023-01-15T14:31:26.987 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062669899999999998 2023-01-15T14:31:26.987 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.987 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.987 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:26.988 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208208+0000", 2023-01-15T14:31:26.988 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7759999999999999e-05 2023-01-15T14:31:26.989 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.989 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.989 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:26.989 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208503+0000", 2023-01-15T14:31:26.989 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000295109 2023-01-15T14:31:26.989 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.989 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.989 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:26.989 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311080+0000", 2023-01-15T14:31:26.989 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10257740899999999 2023-01-15T14:31:26.990 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.990 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.990 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:26.990 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311363+0000", 2023-01-15T14:31:26.990 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028312299999999998 2023-01-15T14:31:26.990 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.990 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.990 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:26.990 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.410431+0000", 2023-01-15T14:31:26.991 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099068120999999995 2023-01-15T14:31:26.991 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.991 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.991 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.410590+0000", 2023-01-15T14:31:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015870700000000001 2023-01-15T14:31:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.410739+0000", 2023-01-15T14:31:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014854899999999999 2023-01-15T14:31:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:509 3.9 3:9f86bdcb:::smithi09025264-22:head [write 2873212~588997] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:31:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:31:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 156.71687353300001, 2023-01-15T14:31:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69623497099999998, 2023-01-15T14:31:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 509 2023-01-15T14:31:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:31:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:31:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814931+0000", 2023-01-15T14:31:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:26.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815512+0000", 2023-01-15T14:31:26.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058082200000000004 2023-01-15T14:31:26.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:26.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815553+0000", 2023-01-15T14:31:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1081999999999997e-05 2023-01-15T14:31:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815571+0000", 2023-01-15T14:31:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8028000000000001e-05 2023-01-15T14:31:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906452+0000", 2023-01-15T14:31:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090880586999999999 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906564+0000", 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111976 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208424+0000", 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.301860199 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:26.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208718+0000", 2023-01-15T14:31:26.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029347299999999999 2023-01-15T14:31:26.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:31:26.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209175+0000", 2023-01-15T14:31:26.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045692399999999999 2023-01-15T14:31:26.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:26.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:26.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:27.000 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209725+0000", 2023-01-15T14:31:27.000 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055027600000000004 2023-01-15T14:31:27.000 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.000 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.000 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:27.000 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209751+0000", 2023-01-15T14:31:27.000 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5812999999999998e-05 2023-01-15T14:31:27.000 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.000 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.000 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:27.000 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.210006+0000", 2023-01-15T14:31:27.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025564099999999997 2023-01-15T14:31:27.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:27.001 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311247+0000", 2023-01-15T14:31:27.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10124119199999999 2023-01-15T14:31:27.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:27.001 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311454+0000", 2023-01-15T14:31:27.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020695800000000001 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511022+0000", 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19956729200000001 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511100+0000", 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8589000000000003e-05 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:27.003 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511168+0000", 2023-01-15T14:31:27.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8039000000000004e-05 2023-01-15T14:31:27.003 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.003 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:27.003 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.003 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:510 3.9 3:9f86bdcb:::smithi09025264-22:head [setxattr _header (57) in=7b,truncate 3462209] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:31:27.003 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:31:27.003 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 156.71621006999999, 2023-01-15T14:31:27.004 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69609656399999997, 2023-01-15T14:31:27.004 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:27.004 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:27.004 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:27.004 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:27.004 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:27.189 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 510 2023-01-15T14:31:27.189 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.189 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:27.189 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.189 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:27.189 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:31:27.189 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.189 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.189 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.189 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:27.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:31:27.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:27.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815596+0000", 2023-01-15T14:31:27.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:27.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:27.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815606+0000", 2023-01-15T14:31:27.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7920000000000004e-06 2023-01-15T14:31:27.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:27.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815627+0000", 2023-01-15T14:31:27.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1239000000000001e-05 2023-01-15T14:31:27.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:27.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815637+0000", 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0349e-05 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906600+0000", 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090963098000000006 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906717+0000", 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000116759 2023-01-15T14:31:27.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310000+0000", 2023-01-15T14:31:27.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40328322599999999 2023-01-15T14:31:27.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:27.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310249+0000", 2023-01-15T14:31:27.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024874400000000002 2023-01-15T14:31:27.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:31:27.194 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310559+0000", 2023-01-15T14:31:27.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310382 2023-01-15T14:31:27.194 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:27.194 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310951+0000", 2023-01-15T14:31:27.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039124500000000003 2023-01-15T14:31:27.194 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:27.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310975+0000", 2023-01-15T14:31:27.195 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4496000000000001e-05 2023-01-15T14:31:27.195 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.195 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.195 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:27.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311041+0000", 2023-01-15T14:31:27.195 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6309000000000005e-05 2023-01-15T14:31:27.195 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.195 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.195 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:27.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311260+0000", 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021863399999999999 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311588+0000", 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032813599999999998 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511351+0000", 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19976275700000001 2023-01-15T14:31:27.196 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511507+0000", 2023-01-15T14:31:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015585100000000001 2023-01-15T14:31:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511693+0000", 2023-01-15T14:31:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186627 2023-01-15T14:31:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:511 3.9 3:9f86bdcb:::smithi09025264-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:31:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:31:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 156.71615270199999, 2023-01-15T14:31:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69666647500000001, 2023-01-15T14:31:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 511 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:31:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815653+0000", 2023-01-15T14:31:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815660+0000", 2023-01-15T14:31:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0509999999999999e-06 2023-01-15T14:31:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815683+0000", 2023-01-15T14:31:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2742999999999999e-05 2023-01-15T14:31:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815698+0000", 2023-01-15T14:31:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5133999999999999e-05 2023-01-15T14:31:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908029+0000", 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.092331448999999996 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908193+0000", 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016342799999999999 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311115+0000", 2023-01-15T14:31:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40292222300000002 2023-01-15T14:31:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311224+0000", 2023-01-15T14:31:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010888899999999999 2023-01-15T14:31:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511800+0000", 2023-01-15T14:31:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20057596899999999 2023-01-15T14:31:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512036+0000", 2023-01-15T14:31:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236247 2023-01-15T14:31:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512321+0000", 2023-01-15T14:31:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028457599999999998 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:692 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~2016807,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898357+0000", 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 138.63345024899999, 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60411950800000003, 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 692 2023-01-15T14:31:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898357+0000", 2023-01-15T14:31:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898357+0000", 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898356+0000", 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898360+0000", 2023-01-15T14:31:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2740000000000001e-06 2023-01-15T14:31:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898373+0000", 2023-01-15T14:31:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.216e-05 2023-01-15T14:31:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898381+0000", 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.225e-06 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.500519+0000", 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60213839400000002 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.500676+0000", 2023-01-15T14:31:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000156348 2023-01-15T14:31:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.502476+0000", 2023-01-15T14:31:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001800638 2023-01-15T14:31:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:693 3.3 3:ca59e2b0:::smithi09025264-1:head [sparse-read 0~2016807] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 138.633419355, 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70514199600000005, 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 693 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:31:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:31:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898387+0000", 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898390+0000", 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.252e-06 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898401+0000", 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0977e-05 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.998532+0000", 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100130443 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602663+0000", 2023-01-15T14:31:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60413100500000005 2023-01-15T14:31:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602817+0000", 2023-01-15T14:31:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015440500000000001 2023-01-15T14:31:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.603530+0000", 2023-01-15T14:31:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071246299999999999 2023-01-15T14:31:27.216 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.216 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:27.216 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.216 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1294 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3327786] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:27.395 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:59.360408+0000", 2023-01-15T14:31:27.395 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 87.171399307000001, 2023-01-15T14:31:27.395 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60454960700000004, 2023-01-15T14:31:27.395 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:27.395 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:27.395 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:27.396 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:27.396 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:27.396 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1294 2023-01-15T14:31:27.396 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.396 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:27.396 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.396 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:27.396 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.360408+0000", 2023-01-15T14:31:27.396 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.396 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.397 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.397 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:27.397 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.360408+0000", 2023-01-15T14:31:27.397 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.397 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.397 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.397 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:27.397 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.360405+0000", 2023-01-15T14:31:27.397 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:27.398 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.398 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:27.398 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.360417+0000", 2023-01-15T14:31:27.398 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1953000000000001e-05 2023-01-15T14:31:27.398 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.398 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:27.398 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.360434+0000", 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6443000000000002e-05 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.460556+0000", 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100121896 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.461178+0000", 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062218799999999997 2023-01-15T14:31:27.399 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.400 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.400 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:27.400 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.461390+0000", 2023-01-15T14:31:27.400 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021255499999999999 2023-01-15T14:31:27.400 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.400 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.400 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:31:27.400 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.461724+0000", 2023-01-15T14:31:27.400 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033382699999999997 2023-01-15T14:31:27.400 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.462215+0000", 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049096100000000004 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.462237+0000", 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2235999999999999e-05 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:27.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.462342+0000", 2023-01-15T14:31:27.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000104411 2023-01-15T14:31:27.402 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.402 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.402 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:27.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.462488+0000", 2023-01-15T14:31:27.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000145914 2023-01-15T14:31:27.402 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.402 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.402 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:27.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.680400+0000", 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.217912509 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.964546+0000", 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.28414540500000002 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.964742+0000", 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019644700000000001 2023-01-15T14:31:27.403 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.404 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.404 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:27.404 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.964957+0000", 2023-01-15T14:31:27.404 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021501199999999999 2023-01-15T14:31:27.404 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.404 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:27.404 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.404 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.404 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.404 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1421 3.a 3:5919d83e:::smithi09025264-13:head [write 1723287~549075] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:31:27.404 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:31:27.405 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 71.619504845999998, 2023-01-15T14:31:27.405 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61118175699999999, 2023-01-15T14:31:27.405 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:27.405 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:27.405 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:27.405 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:27.405 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:27.405 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1421 2023-01-15T14:31:27.405 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.405 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:27.405 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:27.406 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:31:27.406 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.406 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.406 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:27.406 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:31:27.406 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.406 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.406 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912300+0000", 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913127+0000", 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00082704000000000002 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913158+0000", 2023-01-15T14:31:27.407 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1146999999999998e-05 2023-01-15T14:31:27.408 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.408 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.408 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:27.408 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913177+0000", 2023-01-15T14:31:27.408 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:31:27.408 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.408 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.408 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.408 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913384+0000", 2023-01-15T14:31:27.408 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000206589 2023-01-15T14:31:27.408 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.409 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.409 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:31:27.409 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913408+0000", 2023-01-15T14:31:27.409 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3493e-05 2023-01-15T14:31:27.409 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.409 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.409 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.409 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318579+0000", 2023-01-15T14:31:27.409 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40517134399999999 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318806+0000", 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022680899999999999 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319060+0000", 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254069 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.410 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.411 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:27.411 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319527+0000", 2023-01-15T14:31:27.411 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000467081 2023-01-15T14:31:27.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:27.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319544+0000", 2023-01-15T14:31:27.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6949999999999999e-05 2023-01-15T14:31:27.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:27.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319778+0000", 2023-01-15T14:31:27.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023391500000000001 2023-01-15T14:31:27.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:27.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421126+0000", 2023-01-15T14:31:27.612 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101348626 2023-01-15T14:31:27.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522834+0000", 2023-01-15T14:31:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10170761 2023-01-15T14:31:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523142+0000", 2023-01-15T14:31:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854800000000001 2023-01-15T14:31:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523251+0000", 2023-01-15T14:31:27.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108411 2023-01-15T14:31:27.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:27.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523484+0000", 2023-01-15T14:31:27.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023292500000000001 2023-01-15T14:31:27.814 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.814 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:27.814 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.815 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1422 3.a 3:5919d83e:::smithi09025264-13:head [write 2781837~703343] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:31:27.815 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:31:27.815 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 71.618566049999998, 2023-01-15T14:31:27.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61062001700000001, 2023-01-15T14:31:27.815 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:27.815 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:27.815 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:27.815 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:27.815 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:27.815 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1422 2023-01-15T14:31:27.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:27.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913239+0000", 2023-01-15T14:31:27.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:27.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:27.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914075+0000", 2023-01-15T14:31:27.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00083594099999999998 2023-01-15T14:31:27.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:27.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914104+0000", 2023-01-15T14:31:27.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8938999999999999e-05 2023-01-15T14:31:27.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:27.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914121+0000", 2023-01-15T14:31:27.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7306999999999999e-05 2023-01-15T14:31:27.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914234+0000", 2023-01-15T14:31:27.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112737 2023-01-15T14:31:27.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:31:27.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914257+0000", 2023-01-15T14:31:27.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3156e-05 2023-01-15T14:31:27.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:27.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319798+0000", 2023-01-15T14:31:27.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40554153100000001 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319991+0000", 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019217999999999999 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320237+0000", 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024621099999999998 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320759+0000", 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052174700000000003 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320776+0000", 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7173e-05 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:27.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.321080+0000", 2023-01-15T14:31:27.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030424400000000001 2023-01-15T14:31:27.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:27.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421162+0000", 2023-01-15T14:31:27.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100082085 2023-01-15T14:31:27.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:27.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522963+0000", 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10180109399999999 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523576+0000", 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061277499999999997 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523664+0000", 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8002000000000002e-05 2023-01-15T14:31:27.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:27.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523861+0000", 2023-01-15T14:31:27.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019685899999999999 2023-01-15T14:31:27.824 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.824 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:27.824 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:27.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.825 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1423 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 3630312] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:31:27.825 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:31:27.825 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 71.617643514999997, 2023-01-15T14:31:27.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70862486899999999, 2023-01-15T14:31:27.825 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:27.825 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:27.825 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:27.825 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:27.825 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:27.826 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1423 2023-01-15T14:31:27.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.826 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:27.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:27.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:31:27.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:27.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:31:27.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:27.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:27.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914162+0000", 2023-01-15T14:31:27.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:27.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:27.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914172+0000", 2023-01-15T14:31:27.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0267e-05 2023-01-15T14:31:27.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:27.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914188+0000", 2023-01-15T14:31:27.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6212999999999999e-05 2023-01-15T14:31:27.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:27.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:27.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:27.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914201+0000", 2023-01-15T14:31:27.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3073e-05 2023-01-15T14:31:28.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914315+0000", 2023-01-15T14:31:28.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011357399999999999 2023-01-15T14:31:28.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:31:28.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914334+0000", 2023-01-15T14:31:28.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9004e-05 2023-01-15T14:31:28.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421258+0000", 2023-01-15T14:31:28.010 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.506924606 2023-01-15T14:31:28.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:28.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421518+0000", 2023-01-15T14:31:28.010 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025950499999999999 2023-01-15T14:31:28.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:28.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421843+0000", 2023-01-15T14:31:28.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032498299999999999 2023-01-15T14:31:28.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:28.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422222+0000", 2023-01-15T14:31:28.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037867599999999999 2023-01-15T14:31:28.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:28.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422245+0000", 2023-01-15T14:31:28.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3096999999999999e-05 2023-01-15T14:31:28.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:28.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422343+0000", 2023-01-15T14:31:28.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8487999999999996e-05 2023-01-15T14:31:28.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:28.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522890+0000", 2023-01-15T14:31:28.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10054658900000001 2023-01-15T14:31:28.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:28.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523075+0000", 2023-01-15T14:31:28.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185386 2023-01-15T14:31:28.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:28.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622151+0000", 2023-01-15T14:31:28.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099075724000000004 2023-01-15T14:31:28.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622244+0000", 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3387999999999994e-05 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622788+0000", 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000543973 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:28.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1424 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 71.617588143000006, 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70984700199999995, 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1424 2023-01-15T14:31:28.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.016 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:28.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:28.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:31:28.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:28.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:28.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:31:28.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:28.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914218+0000", 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914224+0000", 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6769999999999998e-06 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914242+0000", 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7759000000000001e-05 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014331+0000", 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088819 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114642+0000", 2023-01-15T14:31:28.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100311027 2023-01-15T14:31:28.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.019 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:31:28.019 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114682+0000", 2023-01-15T14:31:28.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0028999999999999e-05 2023-01-15T14:31:28.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.019 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.019 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523918+0000", 2023-01-15T14:31:28.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40923631500000002 2023-01-15T14:31:28.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524124+0000", 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020524900000000001 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622921+0000", 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098797069000000001 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:28.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.623328+0000", 2023-01-15T14:31:28.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000407539 2023-01-15T14:31:28.021 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:28.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.624066+0000", 2023-01-15T14:31:28.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073762899999999995 2023-01-15T14:31:28.021 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:28.021 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:28.021 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1430 3.a 3:52df5138:::smithi09025264-25:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 71.517428382000006, 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81872653100000004, 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:28.022 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1430 2023-01-15T14:31:28.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.023 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:28.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:28.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:31:28.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:28.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:28.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:31:28.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014376+0000", 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014391+0000", 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4922999999999999e-05 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014417+0000", 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5698999999999999e-05 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014435+0000", 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7827000000000001e-05 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114759+0000", 2023-01-15T14:31:28.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100324859 2023-01-15T14:31:28.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:31:28.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114771+0000", 2023-01-15T14:31:28.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:31:28.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524222+0000", 2023-01-15T14:31:28.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40945100800000001 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524545+0000", 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000322367 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.626981+0000", 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102436308 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:28.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627874+0000", 2023-01-15T14:31:28.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089274899999999997 2023-01-15T14:31:28.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:28.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627930+0000", 2023-01-15T14:31:28.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6122000000000002e-05 2023-01-15T14:31:28.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:28.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628609+0000", 2023-01-15T14:31:28.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00067958700000000005 2023-01-15T14:31:28.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:28.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628798+0000", 2023-01-15T14:31:28.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018809 2023-01-15T14:31:28.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628870+0000", 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2844999999999998e-05 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832404+0000", 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20353310299999999 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832606+0000", 2023-01-15T14:31:28.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020207999999999999 2023-01-15T14:31:28.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:28.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833105+0000", 2023-01-15T14:31:28.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049945399999999998 2023-01-15T14:31:28.031 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:28.031 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:28.031 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:28.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.031 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1431 3.a 3:52df5138:::smithi09025264-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:31:28.032 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:31:28.032 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 71.517337093999998, 2023-01-15T14:31:28.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81967516600000001, 2023-01-15T14:31:28.032 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:28.032 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:28.032 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:28.032 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:28.032 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:28.032 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1431 2023-01-15T14:31:28.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.032 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:28.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014469+0000", 2023-01-15T14:31:28.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:28.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:28.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014477+0000", 2023-01-15T14:31:28.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9300000000000003e-06 2023-01-15T14:31:28.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:28.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014495+0000", 2023-01-15T14:31:28.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8363999999999999e-05 2023-01-15T14:31:28.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:28.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114610+0000", 2023-01-15T14:31:28.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10011521700000001 2023-01-15T14:31:28.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214948+0000", 2023-01-15T14:31:28.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100337739 2023-01-15T14:31:28.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:31:28.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214962+0000", 2023-01-15T14:31:28.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4516e-05 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524874+0000", 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30991196599999998 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.525013+0000", 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013873999999999999 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628222+0000", 2023-01-15T14:31:28.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103208721 2023-01-15T14:31:28.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:28.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628504+0000", 2023-01-15T14:31:28.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028245700000000001 2023-01-15T14:31:28.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833214+0000", 2023-01-15T14:31:28.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.204709691 2023-01-15T14:31:28.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:28.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833555+0000", 2023-01-15T14:31:28.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034094000000000003 2023-01-15T14:31:28.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:28.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.834145+0000", 2023-01-15T14:31:28.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059006900000000003 2023-01-15T14:31:28.224 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:28.224 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:28.224 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:28.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.224 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2060 3.1a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:28.224 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:25.714569+0000", 2023-01-15T14:31:28.224 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.81723829599999998, 2023-01-15T14:31:28.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.56541817800000005, 2023-01-15T14:31:28.224 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2060 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.714569+0000", 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:28.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.714569+0000", 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.714568+0000", 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:28.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.714572+0000", 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4129999999999999e-06 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.714587+0000", 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4747e-05 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.714594+0000", 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0859999999999996e-06 2023-01-15T14:31:28.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.778092+0000", 2023-01-15T14:31:28.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.063497709999999999 2023-01-15T14:31:28.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:28.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:25.778305+0000", 2023-01-15T14:31:28.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000213432 2023-01-15T14:31:28.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:26.279565+0000", 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50126025699999999 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:26.279757+0000", 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019174599999999999 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:28.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:26.279987+0000", 2023-01-15T14:31:28.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022954 2023-01-15T14:31:28.230 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:28.230 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:28.230 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:28.230 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:28.230 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:28.230 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:28.231 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:31:28.231 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:31:28.232 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:31:28.232 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:31:28.232 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-01-15T14:31:28.232 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-01-15T14:31:28.233 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:28.234 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:28.235 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:31:28.235 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:31:28.235 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:31:28.235 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:31:28.235 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:31:28.235 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:31:28.235 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:31:28.235 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:31:28.238 INFO:tasks.rados.rados.0.smithi090.stdout:785: done (0 left) 2023-01-15T14:31:28.238 INFO:tasks.rados.rados.0.smithi090.stdout:786: delete oid 5 current snap is 79 2023-01-15T14:31:28.238 INFO:tasks.rados.rados.0.smithi090.stdout:786: done (0 left) 2023-01-15T14:31:28.238 INFO:tasks.rados.rados.0.smithi090.stdout:787: rollback oid 43 current snap is 79 2023-01-15T14:31:28.238 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 43 to 76 2023-01-15T14:31:28.238 INFO:tasks.rados.rados.0.smithi090.stdout:788: write (excl) oid 28 current snap is 79 2023-01-15T14:31:28.238 INFO:tasks.rados.rados.0.smithi090.stdout:788: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout:788: writing smithi09025264-28 from 426977 to 883536 tid 1 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout:788: writing smithi09025264-28 from 1482841 to 2282605 tid 2 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout:788: writing smithi09025264-28 from 2958080 to 3226977 tid 3 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout:789: snap_create 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout:787: finishing rollback tid 1 to smithi09025264-43 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 0 (ObjNum -1971531936 snap 32764 seq_num 304) dirty dne 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout:787: done (1 left) 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout:788: finishing write tid 1 to smithi09025264-28 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout:788: finishing write tid 2 to smithi09025264-28 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout:788: finishing write tid 3 to smithi09025264-28 2023-01-15T14:31:28.239 INFO:tasks.rados.rados.0.smithi090.stdout:788: finishing write tid 4 to smithi09025264-28 2023-01-15T14:31:28.240 INFO:tasks.rados.rados.0.smithi090.stdout:788: finishing write tid 5 to smithi09025264-28 2023-01-15T14:31:28.240 INFO:tasks.rados.rados.0.smithi090.stdout:788: oid 28 updating version 0 to 271 2023-01-15T14:31:28.240 INFO:tasks.rados.rados.0.smithi090.stdout:788: oid 28 updating version 271 to 273 2023-01-15T14:31:28.240 INFO:tasks.rados.rados.0.smithi090.stdout:788: oid 28 version 273 is already newer than 272 2023-01-15T14:31:28.240 INFO:tasks.rados.rados.0.smithi090.stdout:788: oid 28 version 273 is already newer than 270 2023-01-15T14:31:28.240 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 273 (ObjNum 311 snap 79 seq_num 311) dirty exists 2023-01-15T14:31:28.240 INFO:tasks.rados.rados.0.smithi090.stdout:788: left oid 28 (ObjNum 311 snap 79 seq_num 311) 2023-01-15T14:31:28.240 INFO:tasks.rados.rados.0.smithi090.stdout:788: done (0 left) 2023-01-15T14:31:28.240 INFO:tasks.rados.rados.0.smithi090.stdout:789: done (0 left) 2023-01-15T14:31:28.240 INFO:tasks.rados.rados.0.smithi090.stdout:790: read oid 47 snap 75 2023-01-15T14:31:28.240 INFO:tasks.rados.rados.0.smithi090.stdout:790: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:31:28.241 INFO:tasks.rados.rados.0.smithi090.stdout:791: write (excl) oid 12 current snap is 80 2023-01-15T14:31:28.241 INFO:tasks.rados.rados.0.smithi090.stdout:791: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2023-01-15T14:31:28.241 INFO:tasks.rados.rados.0.smithi090.stdout:791: writing smithi09025264-12 from 443784 to 1091945 tid 1 2023-01-15T14:31:28.241 INFO:tasks.rados.rados.0.smithi090.stdout:791: writing smithi09025264-12 from 1657676 to 2317451 tid 2 2023-01-15T14:31:28.241 INFO:tasks.rados.rados.0.smithi090.stdout:791: writing smithi09025264-12 from 2818209 to 3243784 tid 3 2023-01-15T14:31:28.241 INFO:tasks.rados.rados.0.smithi090.stdout:792: read oid 29 snap -1 2023-01-15T14:31:28.241 INFO:tasks.rados.rados.0.smithi090.stdout:792: expect (ObjNum 267 snap 66 seq_num 267) 2023-01-15T14:31:28.241 INFO:tasks.rados.rados.0.smithi090.stdout:793: read oid 39 snap 62 2023-01-15T14:31:28.241 INFO:tasks.rados.rados.0.smithi090.stdout:793: expect deleted 2023-01-15T14:31:28.241 INFO:tasks.rados.rados.0.smithi090.stdout:794: snap_remove snap 79 2023-01-15T14:31:28.242 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:28.227+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:28.334 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:28.330+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:28.420 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:28.420 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:28.421 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:31:28.435 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:28.431+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:28.527 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:28.747 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:28.749 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:31:28.749 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:28.746+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:28.750 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:31:28.750 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:28.750 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 201, 2023-01-15T14:31:28.750 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "1c77fb2b-d1a7-4884-a24e-3935e16d0b6b", 2023-01-15T14:31:28.750 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2023-01-15T14:27:57.735717+0000", 2023-01-15T14:31:28.750 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2023-01-15T14:31:27.959324+0000", 2023-01-15T14:31:28.750 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2023-01-15T14:28:04.837522+0000", 2023-01-15T14:31:28.750 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2023-01-15T14:30:49.906949+0000", 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5799936, 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-01-15T14:31:28.751 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2023-01-15T14:31:28.752 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2023-01-15T14:31:28.752 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 3, 2023-01-15T14:31:28.752 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2023-01-15T14:31:28.752 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2023-01-15T14:31:28.752 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "jewel", 2023-01-15T14:31:28.752 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "reef", 2023-01-15T14:31:28.752 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2023-01-15T14:31:28.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.752 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2023-01-15T14:31:28.752 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2023-01-15T14:31:28.752 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:04.899282+0000", 2023-01-15T14:31:28.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:31:28.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:31:28.753 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:31:28.753 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:31:28.753 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:31:28.753 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:31:28.753 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:31:28.753 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:31:28.753 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:31:28.753 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:31:28.753 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:31:28.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:31:28.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2023-01-15T14:31:28.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2023-01-15T14:31:28.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2023-01-15T14:31:28.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2023-01-15T14:31:28.754 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2023-01-15T14:31:28.754 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:31:28.754 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:31:28.754 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:31:28.754 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:31:28.754 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "16", 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:31:28.755 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:31:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:31:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:31:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:31:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:31:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:31:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:31:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:31:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:31:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:31:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:31:28.756 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:31:28.757 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:31:28.757 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:31:28.757 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:31:28.757 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:31:28.757 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:31:28.757 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:31:28.757 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:31:28.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.757 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:31:28.757 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:31:28.757 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:31:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:31:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:31:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:31:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:31:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:31:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:31:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:31:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:31:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2023-01-15T14:31:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2023-01-15T14:31:28.758 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:11.115002+0000", 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:31:28.759 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:31:28.760 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:31:28.760 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:31:28.760 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:31:28.760 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:31:28.760 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:31:28.760 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:31:28.760 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:31:28.760 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:31:28.760 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:31:28.760 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2023-01-15T14:31:28.760 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.761 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "19", 2023-01-15T14:31:28.762 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:31:28.762 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:31:28.762 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:31:28.762 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:31:28.762 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:31:28.762 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:31:28.762 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:31:28.762 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:31:28.762 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:31:28.762 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:31:28.763 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:31:28.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:31:28.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:31:28.763 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:31:28.763 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:31:28.763 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:31:28.763 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:31:28.763 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:31:28.763 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:31:28.763 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:31:28.763 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:31:28.764 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:31:28.764 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:31:28.764 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:31:28.764 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:31:28.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:31:28.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.764 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:31:28.764 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:31:28.764 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:31:28.764 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:31:28.764 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:31:28.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "unique_pool_0", 2023-01-15T14:31:28.766 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:22.434903+0000", 2023-01-15T14:31:28.766 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 16385, 2023-01-15T14:31:28.766 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,pool_snaps", 2023-01-15T14:31:28.766 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:31:28.766 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:31:28.766 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:31:28.766 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:31:28.766 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:31:28.766 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 46, 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 46, 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 56, 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 56, 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 46, 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:31:28.767 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "201", 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "193", 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "pool", 2023-01-15T14:31:28.768 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 151, 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 201, 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [ 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 123, 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:30:49.323163+0000", 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap63" 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 131, 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:31:00.276854+0000", 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap68" 2023-01-15T14:31:28.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.770 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 139, 2023-01-15T14:31:28.770 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:31:07.800445+0000", 2023-01-15T14:31:28.770 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap72" 2023-01-15T14:31:28.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.770 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 140, 2023-01-15T14:31:28.770 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:31:12.022478+0000", 2023-01-15T14:31:28.770 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap73" 2023-01-15T14:31:28.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.771 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 144, 2023-01-15T14:31:28.771 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:31:16.819985+0000", 2023-01-15T14:31:28.771 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap75" 2023-01-15T14:31:28.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.771 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 145, 2023-01-15T14:31:28.771 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:31:18.223868+0000", 2023-01-15T14:31:28.771 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap76" 2023-01-15T14:31:28.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.771 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 146, 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:31:19.119143+0000", 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap77" 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 150, 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:31:26.439714+0000", 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap79" 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 151, 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:31:27.526984+0000", 2023-01-15T14:31:28.772 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap80" 2023-01-15T14:31:28.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.773 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:31:28.773 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:31:28.773 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:31:28.773 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:31:28.773 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:31:28.773 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:31:28.773 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:31:28.773 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:31:28.773 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:31:28.773 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:31:28.774 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:31:28.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:31:28.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:31:28.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:31:28.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:31:28.774 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:31:28.774 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:31:28.774 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:31:28.774 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:31:28.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.774 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:31:28.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:31:28.775 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:31:28.775 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:31:28.775 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:31:28.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:31:28.775 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:31:28.775 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:31:28.775 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:31:28.775 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:31:28.775 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:31:28.775 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e55c5ac4-3cff-4ee5-8719-373648f83d8d", 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:31:28.776 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.3336334228515625, 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 174, 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.777 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.779 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.779 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2023-01-15T14:31:28.779 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:31:28.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.779 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.779 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2023-01-15T14:31:28.779 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:31:28.779 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.779 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:31:28.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.781 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.781 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:31:28.781 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.781 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2023-01-15T14:31:28.781 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:31:28.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.782 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2023-01-15T14:31:28.782 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:31:28.782 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.782 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.782 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6811/23473", 2023-01-15T14:31:28.782 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6815/23473", 2023-01-15T14:31:28.782 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/23473", 2023-01-15T14:31:28.782 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6820/23473", 2023-01-15T14:31:28.782 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:31:28.782 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:31:28.783 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:31:28.783 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.783 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:31:28.783 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "2c1b33c9-0a1e-4734-936d-d51d64fe997a", 2023-01-15T14:31:28.783 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:31:28.783 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:31:28.783 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:31:28.783 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:31:28.783 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 197, 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:31:28.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.785 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2023-01-15T14:31:28.785 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:31:28.785 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.785 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.785 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:31:28.785 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:31:28.786 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:31:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2023-01-15T14:31:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:31:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2023-01-15T14:31:28.788 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:31:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/23474", 2023-01-15T14:31:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/23474", 2023-01-15T14:31:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/23474", 2023-01-15T14:31:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/23474", 2023-01-15T14:31:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:31:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:31:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:31:28.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "b314174a-eca9-4baa-a60c-2589f1fc9a8e", 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:31:28.790 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 196, 2023-01-15T14:31:28.791 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:31:28.791 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:31:28.791 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:31:28.791 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.791 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.791 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2023-01-15T14:31:28.791 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:31:28.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.791 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.792 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2023-01-15T14:31:28.792 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:31:28.792 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.792 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.792 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:31:28.792 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.793 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2023-01-15T14:31:28.793 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:31:28.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.793 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.793 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2023-01-15T14:31:28.793 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:31:28.793 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.793 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.793 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:31:28.794 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.794 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.794 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2023-01-15T14:31:28.794 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:31:28.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.794 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.794 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2023-01-15T14:31:28.794 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:31:28.794 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.795 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.795 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:31:28.795 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.795 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.795 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2023-01-15T14:31:28.795 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:31:28.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.796 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.796 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2023-01-15T14:31:28.796 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:31:28.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.796 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.796 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6810/23475", 2023-01-15T14:31:28.796 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/23475", 2023-01-15T14:31:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6821/23475", 2023-01-15T14:31:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6817/23475", 2023-01-15T14:31:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:31:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:31:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:31:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:31:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "29eb0b17-ed5d-4665-a347-2fb8406d9c62", 2023-01-15T14:31:28.797 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:31:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:31:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:31:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:31:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:31:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:31:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:31:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 198, 2023-01-15T14:31:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:31:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:31:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:31:28.798 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.799 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6824", 2023-01-15T14:31:28.799 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:31:28.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.799 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6825", 2023-01-15T14:31:28.799 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:31:28.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.799 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.800 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:31:28.800 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.800 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6826", 2023-01-15T14:31:28.800 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:31:28.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.801 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6827", 2023-01-15T14:31:28.801 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:31:28.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.801 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.801 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:31:28.801 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.801 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.801 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6830", 2023-01-15T14:31:28.801 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6831", 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.802 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.803 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6828", 2023-01-15T14:31:28.803 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:31:28.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.803 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.803 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6829", 2023-01-15T14:31:28.803 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:31:28.803 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.803 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.803 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6825/23778", 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6827/23778", 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6831/23778", 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6829/23778", 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "59362feb-caa9-44e3-b6d0-b79397dacbfc", 2023-01-15T14:31:28.804 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:31:28.805 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:31:28.805 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:31:28.805 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:31:28.805 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:31:28.805 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:31:28.805 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:31:28.805 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 194, 2023-01-15T14:31:28.805 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:31:28.805 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:31:28.805 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:31:28.805 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6800", 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6801", 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.807 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:31:28.807 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.807 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.807 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6802", 2023-01-15T14:31:28.807 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:31:28.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.807 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.807 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6803", 2023-01-15T14:31:28.807 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6806", 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.808 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.809 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6807", 2023-01-15T14:31:28.809 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:31:28.809 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.809 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.809 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:31:28.809 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.809 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.809 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6804", 2023-01-15T14:31:28.809 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:31:28.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.810 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.810 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6805", 2023-01-15T14:31:28.810 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:31:28.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.810 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.810 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6801/23523", 2023-01-15T14:31:28.810 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6803/23523", 2023-01-15T14:31:28.810 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6807/23523", 2023-01-15T14:31:28.811 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6805/23523", 2023-01-15T14:31:28.811 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:31:28.811 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:31:28.811 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:31:28.811 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.811 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:31:28.811 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "fbc8821c-8d78-40f8-a571-b18a70d42000", 2023-01-15T14:31:28.811 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:31:28.811 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:31:28.812 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:31:28.812 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:31:28.812 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:31:28.812 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:31:28.812 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:31:28.812 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:31:28.812 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:31:28.812 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:31:28.812 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:31:28.812 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6824", 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6825", 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.813 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6826", 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6827", 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.814 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.815 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:31:28.815 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.815 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.815 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6830", 2023-01-15T14:31:28.815 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:31:28.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.815 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.815 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6831", 2023-01-15T14:31:28.816 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:31:28.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.816 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.816 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:31:28.816 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.816 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6828", 2023-01-15T14:31:28.816 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:31:28.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.817 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.817 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6829", 2023-01-15T14:31:28.817 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:31:28.817 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.817 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.817 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6825/23525", 2023-01-15T14:31:28.817 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6827/23525", 2023-01-15T14:31:28.817 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6831/23525", 2023-01-15T14:31:28.818 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6829/23525", 2023-01-15T14:31:28.818 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:31:28.818 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:31:28.818 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:31:28.818 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.818 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:31:28.818 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "34550470-3561-492f-b605-b5f0e8052c1e", 2023-01-15T14:31:28.818 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:31:28.818 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 118, 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.819 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.820 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6808", 2023-01-15T14:31:28.820 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:31:28.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.820 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.820 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6809", 2023-01-15T14:31:28.820 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:31:28.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.820 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.820 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6810", 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6811", 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.821 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.822 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:31:28.822 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.822 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6814", 2023-01-15T14:31:28.822 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:31:28.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.822 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6815", 2023-01-15T14:31:28.823 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:31:28.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.823 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.823 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:31:28.823 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.823 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6812", 2023-01-15T14:31:28.823 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:31:28.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6813", 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6809/23524", 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6811/23524", 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6815/23524", 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6813/23524", 2023-01-15T14:31:28.824 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:31:28.825 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:31:28.825 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:31:28.825 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.825 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:31:28.825 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1f123d0d-d617-4771-b516-21aa5a29b734", 2023-01-15T14:31:28.825 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:31:28.825 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:31:28.825 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.5779571533203125, 2023-01-15T14:31:28.825 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 197, 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6816", 2023-01-15T14:31:28.826 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:31:28.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.827 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6817", 2023-01-15T14:31:28.827 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:31:28.827 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.827 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.911 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:31:28.911 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.911 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6818", 2023-01-15T14:31:28.912 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:31:28.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.912 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.912 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6819", 2023-01-15T14:31:28.912 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:31:28.912 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.912 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.912 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:31:28.913 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.913 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.913 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6822", 2023-01-15T14:31:28.913 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:31:28.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.913 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.913 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6823", 2023-01-15T14:31:28.913 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:31:28.913 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6820", 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:31:28.914 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6821", 2023-01-15T14:31:28.915 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:31:28.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.915 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.915 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6817/23526", 2023-01-15T14:31:28.915 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6819/23526", 2023-01-15T14:31:28.915 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6823/23526", 2023-01-15T14:31:28.915 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6821/23526", 2023-01-15T14:31:28.915 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:31:28.915 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:31:28.915 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:31:28.916 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.916 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.916 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:31:28.916 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2023-01-15T14:31:28.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.916 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:31:28.916 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:31:28.916 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:31:28.916 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:31:28.916 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:31:28.916 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:31:28.917 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:30:29.938905+0000", 2023-01-15T14:31:28.917 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:31:28.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.917 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:31:28.917 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:31:28.917 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:31:28.917 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:31:28.917 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:31:28.917 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:31:28.917 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:30:52.712517+0000", 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:30:51.841996+0000", 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:31:28.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.919 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:31:28.919 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:31:28.919 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:31:28.919 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:31:28.919 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:31:28.919 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:31:28.919 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:30:15.421866+0000", 2023-01-15T14:31:28.919 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:31:28.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.920 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:31:28.920 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:31:28.920 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:31:28.920 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:31:28.920 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:31:28.920 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:31:28.920 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:30:26.274383+0000", 2023-01-15T14:31:28.920 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:31:28.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.921 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:31:28.921 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:31:28.921 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:31:28.921 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:31:28.921 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:31:28.921 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:31:28.921 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:27.525459+0000", 2023-01-15T14:31:28.921 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:31:28.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.921 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:30:39.598015+0000", 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:31:28.922 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:30:11.543592+0000", 2023-01-15T14:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2023-01-15T14:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [], 2023-01-15T14:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2023-01-15T14:31:28.923 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2023-01-15T14:31:28.924 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2023-01-15T14:31:28.924 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2023-01-15T14:31:28.924 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2023-01-15T14:31:28.924 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2023-01-15T14:31:28.924 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2023-01-15T14:31:28.924 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2023-01-15T14:31:28.924 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2023-01-15T14:31:28.924 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2023-01-15T14:31:28.924 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2023-01-15T14:31:28.924 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2023-01-15T14:31:28.924 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 103, 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 105, 2023-01-15T14:31:28.925 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 4 2023-01-15T14:31:28.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.926 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 110, 2023-01-15T14:31:28.926 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:31:28.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.926 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 115, 2023-01-15T14:31:28.926 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2023-01-15T14:31:28.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.926 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 120, 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 122, 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 124, 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.927 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 128, 2023-01-15T14:31:28.928 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:31:28.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.928 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 130, 2023-01-15T14:31:28.928 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:31:28.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.928 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 132, 2023-01-15T14:31:28.928 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2023-01-15T14:31:28.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.929 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 138, 2023-01-15T14:31:28.929 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:31:28.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.929 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 141, 2023-01-15T14:31:28.929 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:31:28.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:28.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:28.929 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 148, 2023-01-15T14:31:28.929 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:31:28.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.930 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:28.930 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:28.930 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:31:29.068 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2023-01-15T14:31:29.070 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2023-01-15T14:31:29.070 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2023-01-15T14:31:29.070 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2023-01-15T14:31:29.070 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2023-01-15T14:31:29.070 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2023-01-15T14:31:29.070 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2023-01-15T14:31:29.070 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2023-01-15T14:31:29.070 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2023-01-15T14:31:29.071 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2023-01-15T14:31:29.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:29.071 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:29.072 INFO:tasks.thrashosds.thrasher:j is {'epoch': 201, 'fsid': '1c77fb2b-d1a7-4884-a24e-3935e16d0b6b', 'created': '2023-01-15T14:27:57.735717+0000', 'modified': '2023-01-15T14:31:27.959324+0000', 'last_up_change': '2023-01-15T14:28:04.837522+0000', 'last_in_change': '2023-01-15T14:30:49.906949+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-01-15T14:28:04.899282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-01-15T14:28:11.115002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-01-15T14:28:22.434903+0000', 'flags': 16385, 'flags_names': 'hashpspool,pool_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 46, 'pg_placement_num': 46, 'pg_placement_num_target': 56, 'pg_num_target': 56, 'pg_num_pending': 46, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '201', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '193', 'auid': 0, 'snap_mode': 'pool', 'snap_seq': 151, 'snap_epoch': 201, 'pool_snaps': [{'snapid': 123, 'stamp': '2023-01-15T14:30:49.323163+0000', 'name': 'smithi09025264-snap63'}, {'snapid': 131, 'stamp': '2023-01-15T14:31:00.276854+0000', 'name': 'smithi09025264-snap68'}, {'snapid': 139, 'stamp': '2023-01-15T14:31:07.800445+0000', 'name': 'smithi09025264-snap72'}, {'snapid': 140, 'stamp': '2023-01-15T14:31:12.022478+0000', 'name': 'smithi09025264-snap73'}, {'snapid': 144, 'stamp': '2023-01-15T14:31:16.819985+0000', 'name': 'smithi09025264-snap75'}, {'snapid': 145, 'stamp': '2023-01-15T14:31:18.223868+0000', 'name': 'smithi09025264-snap76'}, {'snapid': 146, 'stamp': '2023-01-15T14:31:19.119143+0000', 'name': 'smithi09025264-snap77'}, {'snapid': 150, 'stamp': '2023-01-15T14:31:26.439714+0000', 'name': 'smithi09025264-snap79'}, {'snapid': 151, 'stamp': '2023-01-15T14:31:27.526984+0000', 'name': 'smithi09025264-snap80'}], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e55c5ac4-3cff-4ee5-8719-373648f83d8d', 'up': 1, 'in': 1, 'weight': 0.3336334228515625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 174, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 23473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 23473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 23473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6820', 'nonce': 23473}]}, 'public_addr': '172.21.15.90:6811/23473', 'cluster_addr': '172.21.15.90:6815/23473', 'heartbeat_back_addr': '172.21.15.90:6823/23473', 'heartbeat_front_addr': '172.21.15.90:6820/23473', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2c1b33c9-0a1e-4734-936d-d51d64fe997a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 197, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 23474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 23474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 23474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 23474}]}, 'public_addr': '172.21.15.90:6801/23474', 'cluster_addr': '172.21.15.90:6803/23474', 'heartbeat_back_addr': '172.21.15.90:6807/23474', 'heartbeat_front_addr': '172.21.15.90:6805/23474', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b314174a-eca9-4baa-a60c-2589f1fc9a8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 196, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 23475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 23475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6819', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 23475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 23475}]}, 'public_addr': '172.21.15.90:6810/23475', 'cluster_addr': '172.21.15.90:6814/23475', 'heartbeat_back_addr': '172.21.15.90:6821/23475', 'heartbeat_front_addr': '172.21.15.90:6817/23475', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '29eb0b17-ed5d-4665-a347-2fb8406d9c62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6824', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6825', 'nonce': 23778}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6826', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6827', 'nonce': 23778}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6830', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6831', 'nonce': 23778}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6828', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6829', 'nonce': 23778}]}, 'public_addr': '172.21.15.90:6825/23778', 'cluster_addr': '172.21.15.90:6827/23778', 'heartbeat_back_addr': '172.21.15.90:6831/23778', 'heartbeat_front_addr': '172.21.15.90:6829/23778', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '59362feb-caa9-44e3-b6d0-b79397dacbfc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 194, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6800', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6801', 'nonce': 23523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6802', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6803', 'nonce': 23523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6806', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6807', 'nonce': 23523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6804', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6805', 'nonce': 23523}]}, 'public_addr': '172.21.15.141:6801/23523', 'cluster_addr': '172.21.15.141:6803/23523', 'heartbeat_back_addr': '172.21.15.141:6807/23523', 'heartbeat_front_addr': '172.21.15.141:6805/23523', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fbc8821c-8d78-40f8-a571-b18a70d42000', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6824', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6825', 'nonce': 23525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6826', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6827', 'nonce': 23525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6830', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6831', 'nonce': 23525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6828', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6829', 'nonce': 23525}]}, 'public_addr': '172.21.15.141:6825/23525', 'cluster_addr': '172.21.15.141:6827/23525', 'heartbeat_back_addr': '172.21.15.141:6831/23525', 'heartbeat_front_addr': '172.21.15.141:6829/23525', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '34550470-3561-492f-b605-b5f0e8052c1e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6808', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6809', 'nonce': 23524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6810', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6811', 'nonce': 23524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6814', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6815', 'nonce': 23524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6812', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6813', 'nonce': 23524}]}, 'public_addr': '172.21.15.141:6809/23524', 'cluster_addr': '172.21.15.141:6811/23524', 'heartbeat_back_addr': '172.21.15.141:6815/23524', 'heartbeat_front_addr': '172.21.15.141:6813/23524', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f123d0d-d617-4771-b516-21aa5a29b734', 'up': 1, 'in': 1, 'weight': 0.5779571533203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 197, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6816', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6817', 'nonce': 23526}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6818', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6819', 'nonce': 23526}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6822', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6823', 'nonce': 23526}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6820', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6821', 'nonce': 23526}]}, 'public_addr': '172.21.15.141:6817/23526', 'cluster_addr': '172.21.15.141:6819/23526', 'heartbeat_back_addr': '172.21.15.141:6823/23526', 'heartbeat_front_addr': '172.21.15.141:6821/23526', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:30:29.938905+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:30:52.712517+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:30:51.841996+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:30:15.421866+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:30:26.274383+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:31:27.525459+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:30:39.598015+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:30:11.543592+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 103, 'length': 1}, {'begin': 105, 'length': 4}, {'begin': 110, 'length': 1}, {'begin': 115, 'length': 2}, {'begin': 120, 'length': 1}, {'begin': 122, 'length': 1}, {'begin': 124, 'length': 2}, {'begin': 128, 'length': 1}, {'begin': 130, 'length': 1}, {'begin': 132, 'length': 2}, {'begin': 138, 'length': 1}, {'begin': 141, 'length': 1}, {'begin': 148, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-01-15T14:31:29.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:31:29.074 INFO:tasks.rados.rados.0.smithi090.stdout:794: done (4 left) 2023-01-15T14:31:29.074 INFO:tasks.rados.rados.0.smithi090.stdout:795: rollback oid 4 current snap is 80 2023-01-15T14:31:29.074 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 4 to 75 2023-01-15T14:31:29.074 INFO:tasks.rados.rados.0.smithi090.stdout:796: read oid 15 snap -1 2023-01-15T14:31:29.074 INFO:tasks.rados.rados.0.smithi090.stdout:796: expect deleted 2023-01-15T14:31:29.074 INFO:tasks.rados.rados.0.smithi090.stdout:797: write oid 36 current snap is 80 2023-01-15T14:31:29.074 INFO:tasks.rados.rados.0.smithi090.stdout:797: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2023-01-15T14:31:29.074 INFO:tasks.rados.rados.0.smithi090.stdout:797: writing smithi09025264-36 from 460591 to 1184001 tid 1 2023-01-15T14:31:29.075 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:31:29.076 INFO:tasks.rados.rados.0.smithi090.stdout:797: writing smithi09025264-36 from 1716158 to 2119591 tid 2 2023-01-15T14:31:29.076 INFO:tasks.rados.rados.0.smithi090.stdout:797: writing smithi09025264-36 from 2729279 to 3169419 tid 3 2023-01-15T14:31:29.076 INFO:tasks.rados.rados.0.smithi090.stdout:798: read oid 6 snap -1 2023-01-15T14:31:29.076 INFO:tasks.rados.rados.0.smithi090.stdout:798: expect deleted 2023-01-15T14:31:29.076 INFO:tasks.rados.rados.0.smithi090.stdout:799: delete oid 38 current snap is 80 2023-01-15T14:31:29.076 INFO:tasks.rados.rados.0.smithi090.stdout:792: expect (ObjNum 267 snap 66 seq_num 267) 2023-01-15T14:31:29.077 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:29.067+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:29.175 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:29.171+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:29.317 INFO:tasks.rados.rados.0.smithi090.stdout:790: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:31:29.318 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:29.318 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:29.318 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:31:29.319 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:29.311+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:29.379 INFO:tasks.rados.rados.0.smithi090.stdout:791: finishing write tid 1 to smithi09025264-12 2023-01-15T14:31:29.477 INFO:tasks.rados.rados.0.smithi090.stdout:791: finishing write tid 2 to smithi09025264-12 2023-01-15T14:31:29.477 INFO:tasks.rados.rados.0.smithi090.stdout:791: finishing write tid 3 to smithi09025264-12 2023-01-15T14:31:29.477 INFO:tasks.rados.rados.0.smithi090.stdout:791: finishing write tid 4 to smithi09025264-12 2023-01-15T14:31:29.477 INFO:tasks.rados.rados.0.smithi090.stdout:791: finishing write tid 5 to smithi09025264-12 2023-01-15T14:31:29.477 INFO:tasks.rados.rados.0.smithi090.stdout:791: oid 12 updating version 0 to 244 2023-01-15T14:31:29.477 INFO:tasks.rados.rados.0.smithi090.stdout:791: oid 12 version 244 is already newer than 242 2023-01-15T14:31:29.477 INFO:tasks.rados.rados.0.smithi090.stdout:791: oid 12 version 244 is already newer than 243 2023-01-15T14:31:29.477 INFO:tasks.rados.rados.0.smithi090.stdout:791: oid 12 updating version 244 to 245 2023-01-15T14:31:29.478 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 245 (ObjNum 312 snap 80 seq_num 312) dirty exists 2023-01-15T14:31:29.478 INFO:tasks.rados.rados.0.smithi090.stdout:791: left oid 12 (ObjNum 312 snap 80 seq_num 312) 2023-01-15T14:31:29.478 INFO:tasks.rados.rados.0.smithi090.stdout:797: finishing write tid 1 to smithi09025264-36 2023-01-15T14:31:29.478 INFO:tasks.rados.rados.0.smithi090.stdout:797: finishing write tid 2 to smithi09025264-36 2023-01-15T14:31:29.478 INFO:tasks.rados.rados.0.smithi090.stdout:797: finishing write tid 3 to smithi09025264-36 2023-01-15T14:31:29.480 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:31:29.482 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:31:29.482 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2023-01-15T14:31:28.987994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":183426568,"num_objects":95,"num_object_clones":50,"num_object_copies":285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":10,"num_read":773,"num_read_kb":671710,"num_write":704,"num_write_kb":221968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":313937716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6436,"ondisk_log_size":6436,"up":165,"acting":165,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":169,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":5595428,"kb_used_data":556648740,"kb_used_omap":24849,"kb_used_meta":0,"kb_avail":556600852,"statfs":{"total":575688990720,"available":569959272448,"internally_reserved":49037312,"allocated":570008309760,"data_stored":570008309760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25446236,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1,0,0,3,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":53,"commit_latency_ns":0,"apply_latency_ns":53000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25532663,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":3,"num_read":58,"num_read_kb":48773,"num_write":77,"num_write_kb":24884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":29524673,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.063792"},"pg_stats":[{"pgid":"3.2d","version":"168'156","reported_seq":802,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.999100+0000","last_change":"2023-01-15T14:31:25.241893+0000","last_active":"2023-01-15T14:31:25.999100+0000","last_peered":"2023-01-15T14:31:25.999100+0000","last_clean":"2023-01-15T14:31:25.999100+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:31:25.999100+0000","last_undegraded":"2023-01-15T14:31:25.999100+0000","last_fullsized":"2023-01-15T14:31:25.999100+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.20526107099999999,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":24,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.971235+0000","last_change":"2023-01-15T14:31:25.046610+0000","last_active":"2023-01-15T14:31:25.971235+0000","last_peered":"2023-01-15T14:31:25.971235+0000","last_clean":"2023-01-15T14:31:25.971235+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:31:25.971235+0000","last_undegraded":"2023-01-15T14:31:25.971235+0000","last_fullsized":"2023-01-15T14:31:25.971235+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.011062684,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":525,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.036091+0000","last_change":"2023-01-15T14:31:25.036091+0000","last_active":"2023-01-15T14:31:25.036091+0000","last_peered":"2023-01-15T14:31:25.036091+0000","last_clean":"2023-01-15T14:31:25.036091+0000","last_became_active":"2023-01-15T14:31:21.896494+0000","last_became_peered":"2023-01-15T14:31:21.896494+0000","last_unstale":"2023-01-15T14:31:25.036091+0000","last_undegraded":"2023-01-15T14:31:25.036091+0000","last_fullsized":"2023-01-15T14:31:25.036091+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00245598,"stat_sum":{"num_bytes":865561,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3347,"num_write":6,"num_write_kb":2818,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.2a","version":"191'191","reported_seq":819,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-01-15T14:31:27.114610+0000","last_change":"2023-01-15T14:31:25.245524+0000","last_active":"2023-01-15T14:31:27.114610+0000","last_peered":"2023-01-15T14:31:27.114610+0000","last_clean":"2023-01-15T14:31:27.114610+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:31:27.114610+0000","last_undegraded":"2023-01-15T14:31:27.114610+0000","last_fullsized":"2023-01-15T14:31:27.114610+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078854300000000006,"stat_sum":{"num_bytes":3036773,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":11861,"num_write":28,"num_write_kb":13003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":18895453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.29","version":"110'149","reported_seq":464,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.348793+0000","last_change":"2023-01-15T14:31:25.348793+0000","last_active":"2023-01-15T14:31:25.348793+0000","last_peered":"2023-01-15T14:31:25.348793+0000","last_clean":"2023-01-15T14:31:25.348793+0000","last_became_active":"2023-01-15T14:31:25.136376+0000","last_became_peered":"2023-01-15T14:31:25.136376+0000","last_unstale":"2023-01-15T14:31:25.348793+0000","last_undegraded":"2023-01-15T14:31:25.348793+0000","last_fullsized":"2023-01-15T14:31:25.348793+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:30:08.681869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:08.681869+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21182261099999999,"stat_sum":{"num_bytes":2857174,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":24874,"num_write":16,"num_write_kb":6171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.28","version":"179'204","reported_seq":908,"reported_epoch":198,"state":"peering","last_fresh":"2023-01-15T14:31:24.932969+0000","last_change":"2023-01-15T14:31:24.933219+0000","last_active":"2023-01-15T14:31:24.932896+0000","last_peered":"2023-01-15T14:31:24.465236+0000","last_clean":"2023-01-15T14:31:19.000534+0000","last_became_active":"2023-01-15T14:31:22.920741+0000","last_became_peered":"2023-01-15T14:31:22.920741+0000","last_unstale":"2023-01-15T14:31:24.932969+0000","last_undegraded":"2023-01-15T14:31:24.932969+0000","last_fullsized":"2023-01-15T14:31:24.932969+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2159716,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10594,"num_write":2,"num_write_kb":3981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":12193528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":642,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:24.931281+0000","last_change":"2023-01-15T14:31:24.934388+0000","last_active":"2023-01-15T14:31:24.931281+0000","last_peered":"2023-01-15T14:31:24.931281+0000","last_clean":"2023-01-15T14:31:24.931281+0000","last_became_active":"2023-01-15T14:31:22.091646+0000","last_became_peered":"2023-01-15T14:31:22.091646+0000","last_unstale":"2023-01-15T14:31:24.931281+0000","last_undegraded":"2023-01-15T14:31:24.931281+0000","last_fullsized":"2023-01-15T14:31:24.931281+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.0034568870000000001,"stat_sum":{"num_bytes":2485558,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":8458,"num_write":2,"num_write_kb":737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1731121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.26","version":"102'52","reported_seq":24,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.999079+0000","last_change":"2023-01-15T14:31:25.038585+0000","last_active":"2023-01-15T14:31:25.999079+0000","last_peered":"2023-01-15T14:31:25.999079+0000","last_clean":"2023-01-15T14:31:25.999079+0000","last_became_active":"2023-01-15T14:31:21.901622+0000","last_became_peered":"2023-01-15T14:31:21.901622+0000","last_unstale":"2023-01-15T14:31:25.999079+0000","last_undegraded":"2023-01-15T14:31:25.999079+0000","last_fullsized":"2023-01-15T14:31:25.999079+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:30:21.421951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:21.421951+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.569620185,"objects_trimmed":0,"snaptrim_duration":0.0038919150000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.25","version":"198'79","reported_seq":656,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:26.050839+0000","last_change":"2023-01-15T14:31:26.050839+0000","last_active":"2023-01-15T14:31:26.050839+0000","last_peered":"2023-01-15T14:31:26.050839+0000","last_clean":"2023-01-15T14:31:26.050839+0000","last_became_active":"2023-01-15T14:31:22.000386+0000","last_became_peered":"2023-01-15T14:31:22.000386+0000","last_unstale":"2023-01-15T14:31:26.050839+0000","last_undegraded":"2023-01-15T14:31:26.050839+0000","last_fullsized":"2023-01-15T14:31:26.050839+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0151982669999999,"stat_sum":{"num_bytes":2778701,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":14,"num_read_kb":15020,"num_write":13,"num_write_kb":5359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9491720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.24","version":"189'200","reported_seq":882,"reported_epoch":198,"state":"peering","last_fresh":"2023-01-15T14:31:24.546538+0000","last_change":"2023-01-15T14:31:25.914503+0000","last_active":"2023-01-15T14:31:25.914503+0000","last_peered":"2023-01-15T14:31:24.546538+0000","last_clean":"2023-01-15T14:31:19.000125+0000","last_became_active":"2023-01-15T14:31:23.218704+0000","last_became_peered":"2023-01-15T14:31:23.218704+0000","last_unstale":"2023-01-15T14:31:24.546538+0000","last_undegraded":"2023-01-15T14:31:24.546538+0000","last_fullsized":"2023-01-15T14:31:22.811693+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4345840,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":18109,"num_write":16,"num_write_kb":4789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":21209193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"1"},{"start":"8a","length":"1"},{"start":"8d","length":"1"}]},{"pgid":"3.23","version":"149'233","reported_seq":769,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:24.931194+0000","last_change":"2023-01-15T14:31:25.033337+0000","last_active":"2023-01-15T14:31:24.931194+0000","last_peered":"2023-01-15T14:31:24.931194+0000","last_clean":"2023-01-15T14:31:24.931194+0000","last_became_active":"2023-01-15T14:31:06.029716+0000","last_became_peered":"2023-01-15T14:31:06.029716+0000","last_unstale":"2023-01-15T14:31:24.931194+0000","last_undegraded":"2023-01-15T14:31:24.931194+0000","last_fullsized":"2023-01-15T14:31:24.931194+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.102591155,"stat_sum":{"num_bytes":7604272,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":47,"num_read_kb":43637,"num_write":37,"num_write_kb":13160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2469494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.22","version":"181'107","reported_seq":828,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:24.937357+0000","last_change":"2023-01-15T14:31:25.133475+0000","last_active":"2023-01-15T14:31:24.937357+0000","last_peered":"2023-01-15T14:31:24.937357+0000","last_clean":"2023-01-15T14:31:24.937357+0000","last_became_active":"2023-01-15T14:31:08.084483+0000","last_became_peered":"2023-01-15T14:31:08.084483+0000","last_unstale":"2023-01-15T14:31:24.937357+0000","last_undegraded":"2023-01-15T14:31:24.937357+0000","last_fullsized":"2023-01-15T14:31:24.937357+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.201232249,"stat_sum":{"num_bytes":2518991,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":12156,"num_write":8,"num_write_kb":2164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":11208309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.21","version":"168'124","reported_seq":695,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.031569+0000","last_change":"2023-01-15T14:31:25.031569+0000","last_active":"2023-01-15T14:31:25.031569+0000","last_peered":"2023-01-15T14:31:25.031569+0000","last_clean":"2023-01-15T14:31:25.031569+0000","last_became_active":"2023-01-15T14:30:52.226939+0000","last_became_peered":"2023-01-15T14:30:52.226939+0000","last_unstale":"2023-01-15T14:31:25.031569+0000","last_undegraded":"2023-01-15T14:31:25.031569+0000","last_fullsized":"2023-01-15T14:31:25.031569+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":6,"last_scrub":"168'124","last_scrub_stamp":"2023-01-15T14:31:11.414546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.414546+0000","objects_scrubbed":0,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:23.043","scrub_duration":0.20479377300000001,"objects_trimmed":0,"snaptrim_duration":0.0047361490000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.20","version":"181'236","reported_seq":935,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.998190+0000","last_change":"2023-01-15T14:31:25.038129+0000","last_active":"2023-01-15T14:31:25.998190+0000","last_peered":"2023-01-15T14:31:25.998190+0000","last_clean":"2023-01-15T14:31:25.998190+0000","last_became_active":"2023-01-15T14:31:07.906191+0000","last_became_peered":"2023-01-15T14:31:07.906191+0000","last_unstale":"2023-01-15T14:31:25.998190+0000","last_undegraded":"2023-01-15T14:31:25.998190+0000","last_fullsized":"2023-01-15T14:31:25.998190+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.0039860410000000001,"stat_sum":{"num_bytes":4659379,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":9657,"num_write":7,"num_write_kb":2065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8877936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.1f","version":"198'163","reported_seq":826,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:26.097472+0000","last_change":"2023-01-15T14:31:25.031380+0000","last_active":"2023-01-15T14:31:26.097472+0000","last_peered":"2023-01-15T14:31:26.097472+0000","last_clean":"2023-01-15T14:31:26.097472+0000","last_became_active":"2023-01-15T14:31:07.960304+0000","last_became_peered":"2023-01-15T14:31:07.960304+0000","last_unstale":"2023-01-15T14:31:26.097472+0000","last_undegraded":"2023-01-15T14:31:26.097472+0000","last_fullsized":"2023-01-15T14:31:26.097472+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:15.241","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037336230000000001,"stat_sum":{"num_bytes":6159815,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":66,"num_read_kb":91951,"num_write":46,"num_write_kb":14339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":36580502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.1e","version":"139'38","reported_seq":533,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.244310+0000","last_change":"2023-01-15T14:31:25.246388+0000","last_active":"2023-01-15T14:31:25.244310+0000","last_peered":"2023-01-15T14:31:25.244310+0000","last_clean":"2023-01-15T14:31:25.244310+0000","last_became_active":"2023-01-15T14:30:51.134845+0000","last_became_peered":"2023-01-15T14:30:51.134845+0000","last_unstale":"2023-01-15T14:31:25.244310+0000","last_undegraded":"2023-01-15T14:31:25.244310+0000","last_fullsized":"2023-01-15T14:31:25.244310+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'37","last_scrub_stamp":"2023-01-15T14:30:25.793922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:25.793922+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.215445631,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":11530,"num_write":6,"num_write_kb":4046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1487420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.1d","version":"198'180","reported_seq":821,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.971240+0000","last_change":"2023-01-15T14:31:25.643427+0000","last_active":"2023-01-15T14:31:25.971240+0000","last_peered":"2023-01-15T14:31:25.971240+0000","last_clean":"2023-01-15T14:31:25.971240+0000","last_became_active":"2023-01-15T14:30:54.085528+0000","last_became_peered":"2023-01-15T14:30:54.085528+0000","last_unstale":"2023-01-15T14:31:25.971240+0000","last_undegraded":"2023-01-15T14:31:25.971240+0000","last_fullsized":"2023-01-15T14:31:25.971240+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70737571799999999,"stat_sum":{"num_bytes":10574961,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":73,"num_read_kb":81712,"num_write":50,"num_write_kb":17879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":13857289,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":105,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.971532+0000","last_change":"2023-01-15T14:31:25.045095+0000","last_active":"2023-01-15T14:31:25.971532+0000","last_peered":"2023-01-15T14:31:25.971532+0000","last_clean":"2023-01-15T14:31:25.971532+0000","last_became_active":"2023-01-15T14:30:51.234588+0000","last_became_peered":"2023-01-15T14:30:51.234588+0000","last_unstale":"2023-01-15T14:31:25.971532+0000","last_undegraded":"2023-01-15T14:31:25.971532+0000","last_fullsized":"2023-01-15T14:31:25.971532+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.107281846,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":10,"num_write_kb":2522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1159667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.1b","version":"155'53","reported_seq":534,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.971286+0000","last_change":"2023-01-15T14:31:24.942673+0000","last_active":"2023-01-15T14:31:25.971286+0000","last_peered":"2023-01-15T14:31:25.971286+0000","last_clean":"2023-01-15T14:31:25.971286+0000","last_became_active":"2023-01-15T14:31:02.994801+0000","last_became_peered":"2023-01-15T14:31:02.994801+0000","last_unstale":"2023-01-15T14:31:25.971286+0000","last_undegraded":"2023-01-15T14:31:25.971286+0000","last_fullsized":"2023-01-15T14:31:25.971286+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065772770000000003,"stat_sum":{"num_bytes":1731121,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":23608,"num_write":12,"num_write_kb":5635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3462242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.1a","version":"198'211","reported_seq":882,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:26.279914+0000","last_change":"2023-01-15T14:31:24.939433+0000","last_active":"2023-01-15T14:31:26.279914+0000","last_peered":"2023-01-15T14:31:26.279914+0000","last_clean":"2023-01-15T14:31:26.279914+0000","last_became_active":"2023-01-15T14:31:13.049050+0000","last_became_peered":"2023-01-15T14:31:13.049050+0000","last_unstale":"2023-01-15T14:31:26.279914+0000","last_undegraded":"2023-01-15T14:31:26.279914+0000","last_fullsized":"2023-01-15T14:31:26.279914+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'196","last_scrub_stamp":"2023-01-15T14:31:19.373635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:19.373635+0000","objects_scrubbed":4,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:37.846","scrub_duration":0.20803622599999999,"objects_trimmed":0,"snaptrim_duration":0.0042777259999999999,"stat_sum":{"num_bytes":11047317,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":4,"num_read_kb":3,"num_write":13,"num_write_kb":3975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.19","version":"198'186","reported_seq":797,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:26.351449+0000","last_change":"2023-01-15T14:31:26.457559+0000","last_active":"2023-01-15T14:31:26.351449+0000","last_peered":"2023-01-15T14:31:26.351449+0000","last_clean":"2023-01-15T14:31:26.351449+0000","last_became_active":"2023-01-15T14:30:59.222826+0000","last_became_peered":"2023-01-15T14:30:59.222826+0000","last_unstale":"2023-01-15T14:31:26.351449+0000","last_undegraded":"2023-01-15T14:31:26.351449+0000","last_fullsized":"2023-01-15T14:31:26.351449+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'157","last_scrub_stamp":"2023-01-15T14:30:26.792736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:26.792736+0000","objects_scrubbed":4,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:38.155","scrub_duration":0.61998826900000004,"objects_trimmed":0,"snaptrim_duration":0.50412762200000005,"stat_sum":{"num_bytes":15306127,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":27,"num_read_kb":35342,"num_write":14,"num_write_kb":4509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":29479511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.18","version":"198'232","reported_seq":959,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.649345+0000","last_change":"2023-01-15T14:31:25.241948+0000","last_active":"2023-01-15T14:31:25.649345+0000","last_peered":"2023-01-15T14:31:25.649345+0000","last_clean":"2023-01-15T14:31:25.649345+0000","last_became_active":"2023-01-15T14:31:04.108363+0000","last_became_peered":"2023-01-15T14:31:04.108363+0000","last_unstale":"2023-01-15T14:31:25.649345+0000","last_undegraded":"2023-01-15T14:31:25.649345+0000","last_fullsized":"2023-01-15T14:31:25.649345+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.30839879999999997,"stat_sum":{"num_bytes":9573635,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":23,"num_read_kb":18217,"num_write":20,"num_write_kb":8697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":22066072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.17","version":"155'77","reported_seq":585,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.236430+0000","last_change":"2023-01-15T14:31:25.238715+0000","last_active":"2023-01-15T14:31:25.236430+0000","last_peered":"2023-01-15T14:31:25.236430+0000","last_clean":"2023-01-15T14:31:25.236430+0000","last_became_active":"2023-01-15T14:31:02.993865+0000","last_became_peered":"2023-01-15T14:31:02.993865+0000","last_unstale":"2023-01-15T14:31:25.236430+0000","last_undegraded":"2023-01-15T14:31:25.236430+0000","last_fullsized":"2023-01-15T14:31:25.236430+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'65","last_scrub_stamp":"2023-01-15T14:30:24.156798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.156798+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:34.293","scrub_duration":0.30368223799999999,"objects_trimmed":0,"snaptrim_duration":0.30541575799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":2048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3949645,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.16","version":"194'81","reported_seq":136,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.138641+0000","last_change":"2023-01-15T14:31:25.138641+0000","last_active":"2023-01-15T14:31:25.138641+0000","last_peered":"2023-01-15T14:31:25.138641+0000","last_clean":"2023-01-15T14:31:25.138641+0000","last_became_active":"2023-01-15T14:30:51.036725+0000","last_became_peered":"2023-01-15T14:30:51.036725+0000","last_unstale":"2023-01-15T14:31:25.138641+0000","last_undegraded":"2023-01-15T14:31:25.138641+0000","last_fullsized":"2023-01-15T14:31:25.138641+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:36.231","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.103754101,"stat_sum":{"num_bytes":2235364,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":557,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.033523+0000","last_change":"2023-01-15T14:31:25.034365+0000","last_active":"2023-01-15T14:31:25.033523+0000","last_peered":"2023-01-15T14:31:25.033523+0000","last_clean":"2023-01-15T14:31:25.033523+0000","last_became_active":"2023-01-15T14:30:03.169969+0000","last_became_peered":"2023-01-15T14:30:03.169969+0000","last_unstale":"2023-01-15T14:31:25.033523+0000","last_undegraded":"2023-01-15T14:31:25.033523+0000","last_fullsized":"2023-01-15T14:31:25.033523+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:30:24.721082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.721082+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:48.096","scrub_duration":0.109000768,"objects_trimmed":0,"snaptrim_duration":0.102046607,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.14","version":"169'138","reported_seq":715,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.971713+0000","last_change":"2023-01-15T14:31:25.238448+0000","last_active":"2023-01-15T14:31:25.971713+0000","last_peered":"2023-01-15T14:31:25.971713+0000","last_clean":"2023-01-15T14:31:25.971713+0000","last_became_active":"2023-01-15T14:30:16.972329+0000","last_became_peered":"2023-01-15T14:30:16.972329+0000","last_unstale":"2023-01-15T14:31:25.971713+0000","last_undegraded":"2023-01-15T14:31:25.971713+0000","last_fullsized":"2023-01-15T14:31:25.971713+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":5,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:31.572","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.19998875099999999,"stat_sum":{"num_bytes":2453855,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":11235,"num_write":5,"num_write_kb":1217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":224,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.127698+0000","last_change":"2023-01-15T14:30:51.228396+0000","last_active":"2023-01-15T14:31:25.127698+0000","last_peered":"2023-01-15T14:31:25.127698+0000","last_clean":"2023-01-15T14:31:25.127698+0000","last_became_active":"2023-01-15T14:30:51.127504+0000","last_became_peered":"2023-01-15T14:30:51.127504+0000","last_unstale":"2023-01-15T14:31:25.127698+0000","last_undegraded":"2023-01-15T14:31:25.127698+0000","last_fullsized":"2023-01-15T14:31:25.127698+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.671761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.671761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":595,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:26.051047+0000","last_change":"2023-01-15T14:31:26.058975+0000","last_active":"2023-01-15T14:31:26.051047+0000","last_peered":"2023-01-15T14:31:26.051047+0000","last_clean":"2023-01-15T14:31:26.051047+0000","last_became_active":"2023-01-15T14:31:21.030929+0000","last_became_peered":"2023-01-15T14:31:21.030929+0000","last_unstale":"2023-01-15T14:31:26.051047+0000","last_undegraded":"2023-01-15T14:31:26.051047+0000","last_fullsized":"2023-01-15T14:31:26.051047+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:31:25.342830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:25.342830+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:44.671","scrub_duration":0.20886623600000001,"objects_trimmed":0,"snaptrim_duration":0.106605676,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":46,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.035219+0000","last_change":"2023-01-15T14:30:51.035825+0000","last_active":"2023-01-15T14:31:25.035219+0000","last_peered":"2023-01-15T14:31:25.035219+0000","last_clean":"2023-01-15T14:31:25.035219+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:31:25.035219+0000","last_undegraded":"2023-01-15T14:31:25.035219+0000","last_fullsized":"2023-01-15T14:31:25.035219+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"198'85","reported_seq":646,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.653723+0000","last_change":"2023-01-15T14:31:25.242899+0000","last_active":"2023-01-15T14:31:25.653723+0000","last_peered":"2023-01-15T14:31:25.653723+0000","last_clean":"2023-01-15T14:31:25.653723+0000","last_became_active":"2023-01-15T14:31:21.133504+0000","last_became_peered":"2023-01-15T14:31:21.133504+0000","last_unstale":"2023-01-15T14:31:25.653723+0000","last_undegraded":"2023-01-15T14:31:25.653723+0000","last_fullsized":"2023-01-15T14:31:25.653723+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.30930753799999999,"stat_sum":{"num_bytes":4616256,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":8457,"num_write":4,"num_write_kb":737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1731121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":226,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.999331+0000","last_change":"2023-01-15T14:31:15.088518+0000","last_active":"2023-01-15T14:31:25.999331+0000","last_peered":"2023-01-15T14:31:25.999331+0000","last_clean":"2023-01-15T14:31:25.999331+0000","last_became_active":"2023-01-15T14:31:03.088392+0000","last_became_peered":"2023-01-15T14:31:03.088392+0000","last_unstale":"2023-01-15T14:31:25.999331+0000","last_undegraded":"2023-01-15T14:31:25.999331+0000","last_fullsized":"2023-01-15T14:31:25.999331+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:15.088397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.088397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:32.351","scrub_duration":0.40518630900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"198'211","reported_seq":905,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.954460+0000","last_change":"2023-01-15T14:31:25.640918+0000","last_active":"2023-01-15T14:31:25.954460+0000","last_peered":"2023-01-15T14:31:25.954460+0000","last_clean":"2023-01-15T14:31:25.954460+0000","last_became_active":"2023-01-15T14:31:21.226293+0000","last_became_peered":"2023-01-15T14:31:21.226293+0000","last_unstale":"2023-01-15T14:31:25.954460+0000","last_undegraded":"2023-01-15T14:31:25.954460+0000","last_fullsized":"2023-01-15T14:31:25.954460+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.60614811899999999,"stat_sum":{"num_bytes":4345839,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":18,"num_read_kb":18109,"num_write":20,"num_write_kb":4789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":15862107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":46,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.035045+0000","last_change":"2023-01-15T14:30:51.237821+0000","last_active":"2023-01-15T14:31:25.035045+0000","last_peered":"2023-01-15T14:31:25.035045+0000","last_clean":"2023-01-15T14:31:25.035045+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:31:25.035045+0000","last_undegraded":"2023-01-15T14:31:25.035045+0000","last_fullsized":"2023-01-15T14:31:25.035045+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"198'82","reported_seq":661,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.741334+0000","last_change":"2023-01-15T14:31:25.741949+0000","last_active":"2023-01-15T14:31:25.741334+0000","last_peered":"2023-01-15T14:31:25.741334+0000","last_clean":"2023-01-15T14:31:25.741334+0000","last_became_active":"2023-01-15T14:31:21.132693+0000","last_became_peered":"2023-01-15T14:31:21.132693+0000","last_unstale":"2023-01-15T14:31:25.741334+0000","last_undegraded":"2023-01-15T14:31:25.741334+0000","last_fullsized":"2023-01-15T14:31:25.741334+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:25.367","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.809278198,"stat_sum":{"num_bytes":4145320,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":15020,"num_write":17,"num_write_kb":6511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9491720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.3","version":"198'263","reported_seq":957,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:26.071793+0000","last_change":"2023-01-15T14:31:25.236172+0000","last_active":"2023-01-15T14:31:26.071793+0000","last_peered":"2023-01-15T14:31:26.071793+0000","last_clean":"2023-01-15T14:31:26.071793+0000","last_became_active":"2023-01-15T14:31:03.093753+0000","last_became_peered":"2023-01-15T14:31:03.093753+0000","last_unstale":"2023-01-15T14:31:26.071793+0000","last_undegraded":"2023-01-15T14:31:26.071793+0000","last_fullsized":"2023-01-15T14:31:26.071793+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'251","last_scrub_stamp":"2023-01-15T14:31:15.363400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.363400+0000","objects_scrubbed":3,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:42.073","scrub_duration":0.208136242,"objects_trimmed":0,"snaptrim_duration":0.19927066700000001,"stat_sum":{"num_bytes":9344857,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":10,"num_write_kb":2770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":56,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.034341+0000","last_change":"2023-01-15T14:31:13.764052+0000","last_active":"2023-01-15T14:31:25.034341+0000","last_peered":"2023-01-15T14:31:25.034341+0000","last_clean":"2023-01-15T14:31:25.034341+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:31:25.034341+0000","last_undegraded":"2023-01-15T14:31:25.034341+0000","last_fullsized":"2023-01-15T14:31:25.034341+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:13.763945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:13.763945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:39.419","scrub_duration":0.58613919699999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"194'240","reported_seq":878,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.137233+0000","last_change":"2023-01-15T14:31:25.137233+0000","last_active":"2023-01-15T14:31:25.137233+0000","last_peered":"2023-01-15T14:31:25.137233+0000","last_clean":"2023-01-15T14:31:25.137233+0000","last_became_active":"2023-01-15T14:30:56.101261+0000","last_became_peered":"2023-01-15T14:30:56.101261+0000","last_unstale":"2023-01-15T14:31:25.137233+0000","last_undegraded":"2023-01-15T14:31:25.137233+0000","last_fullsized":"2023-01-15T14:31:25.137233+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:31.088","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.103312742,"stat_sum":{"num_bytes":6218590,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":200,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.027166+0000","last_change":"2023-01-15T14:29:47.854985+0000","last_active":"2023-01-15T14:31:25.027166+0000","last_peered":"2023-01-15T14:31:25.027166+0000","last_clean":"2023-01-15T14:31:25.027166+0000","last_became_active":"2023-01-15T14:28:13.253291+0000","last_became_peered":"2023-01-15T14:28:13.253291+0000","last_unstale":"2023-01-15T14:31:25.027166+0000","last_undegraded":"2023-01-15T14:31:25.027166+0000","last_fullsized":"2023-01-15T14:31:25.027166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:47.854866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:47.854866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.63763962500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":637,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.971413+0000","last_change":"2023-01-15T14:31:25.240260+0000","last_active":"2023-01-15T14:31:25.971413+0000","last_peered":"2023-01-15T14:31:25.971413+0000","last_clean":"2023-01-15T14:31:25.971413+0000","last_became_active":"2023-01-15T14:31:06.028849+0000","last_became_peered":"2023-01-15T14:31:06.028849+0000","last_unstale":"2023-01-15T14:31:25.971413+0000","last_undegraded":"2023-01-15T14:31:25.971413+0000","last_fullsized":"2023-01-15T14:31:25.971413+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:30:47.349633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.349633+0000","objects_scrubbed":1,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.28528039199999999,"objects_trimmed":0,"snaptrim_duration":0.104608611,"stat_sum":{"num_bytes":1899191,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":235,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:26.098177+0000","last_change":"2023-01-15T14:31:11.136609+0000","last_active":"2023-01-15T14:31:26.098177+0000","last_peered":"2023-01-15T14:31:26.098177+0000","last_clean":"2023-01-15T14:31:26.098177+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:31:26.098177+0000","last_undegraded":"2023-01-15T14:31:26.098177+0000","last_fullsized":"2023-01-15T14:31:26.098177+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:11.136490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.136490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:36.593","scrub_duration":0.39969354600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":731,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.246456+0000","last_change":"2023-01-15T14:31:25.344730+0000","last_active":"2023-01-15T14:31:25.246456+0000","last_peered":"2023-01-15T14:31:25.246456+0000","last_clean":"2023-01-15T14:31:25.246456+0000","last_became_active":"2023-01-15T14:30:54.184242+0000","last_became_peered":"2023-01-15T14:30:54.184242+0000","last_unstale":"2023-01-15T14:31:25.246456+0000","last_undegraded":"2023-01-15T14:31:25.246456+0000","last_fullsized":"2023-01-15T14:31:25.246456+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.31301576199999998,"stat_sum":{"num_bytes":1084068,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3268978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":203,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.027215+0000","last_change":"2023-01-15T14:31:02.793835+0000","last_active":"2023-01-15T14:31:25.027215+0000","last_peered":"2023-01-15T14:31:25.027215+0000","last_clean":"2023-01-15T14:31:25.027215+0000","last_became_active":"2023-01-15T14:31:02.793531+0000","last_became_peered":"2023-01-15T14:31:02.793531+0000","last_unstale":"2023-01-15T14:31:25.027215+0000","last_undegraded":"2023-01-15T14:31:25.027215+0000","last_fullsized":"2023-01-15T14:31:25.027215+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:44.759745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:44.759745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":334,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.133335+0000","last_change":"2023-01-15T14:31:03.091584+0000","last_active":"2023-01-15T14:31:25.133335+0000","last_peered":"2023-01-15T14:31:25.133335+0000","last_clean":"2023-01-15T14:31:25.133335+0000","last_became_active":"2023-01-15T14:31:03.091071+0000","last_became_peered":"2023-01-15T14:31:03.091071+0000","last_unstale":"2023-01-15T14:31:25.133335+0000","last_undegraded":"2023-01-15T14:31:25.133335+0000","last_fullsized":"2023-01-15T14:31:25.133335+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:29:45.876183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:29:45.876183+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"179'204","reported_seq":886,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.243047+0000","last_change":"2023-01-15T14:31:25.244169+0000","last_active":"2023-01-15T14:31:25.243047+0000","last_peered":"2023-01-15T14:31:25.243047+0000","last_clean":"2023-01-15T14:31:25.243047+0000","last_became_active":"2023-01-15T14:31:21.332760+0000","last_became_peered":"2023-01-15T14:31:21.332760+0000","last_unstale":"2023-01-15T14:31:25.243047+0000","last_undegraded":"2023-01-15T14:31:25.243047+0000","last_fullsized":"2023-01-15T14:31:25.243047+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.31042893300000002,"stat_sum":{"num_bytes":2159716,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":3981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":7874096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":680,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.971665+0000","last_change":"2023-01-15T14:31:25.242417+0000","last_active":"2023-01-15T14:31:25.971665+0000","last_peered":"2023-01-15T14:31:25.971665+0000","last_clean":"2023-01-15T14:31:25.971665+0000","last_became_active":"2023-01-15T14:31:21.122856+0000","last_became_peered":"2023-01-15T14:31:21.122856+0000","last_unstale":"2023-01-15T14:31:25.971665+0000","last_undegraded":"2023-01-15T14:31:25.971665+0000","last_fullsized":"2023-01-15T14:31:25.971665+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:30:08.681869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:08.681869+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40398525099999999,"objects_trimmed":0,"snaptrim_duration":0.10616210399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.a","version":"198'197","reported_seq":829,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.989741+0000","last_change":"2023-01-15T14:31:25.043860+0000","last_active":"2023-01-15T14:31:25.989741+0000","last_peered":"2023-01-15T14:31:25.989741+0000","last_clean":"2023-01-15T14:31:25.989741+0000","last_became_active":"2023-01-15T14:31:21.032574+0000","last_became_peered":"2023-01-15T14:31:21.032574+0000","last_unstale":"2023-01-15T14:31:25.989741+0000","last_undegraded":"2023-01-15T14:31:25.989741+0000","last_fullsized":"2023-01-15T14:31:25.989741+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":197,"log_dups_size":0,"ondisk_log_size":197,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10771856,"stat_sum":{"num_bytes":4451573,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":11862,"num_write":32,"num_write_kb":14107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":18895452,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":572,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.135080+0000","last_change":"2023-01-15T14:31:25.236091+0000","last_active":"2023-01-15T14:31:25.135080+0000","last_peered":"2023-01-15T14:31:25.135080+0000","last_clean":"2023-01-15T14:31:25.135080+0000","last_became_active":"2023-01-15T14:31:21.230392+0000","last_became_peered":"2023-01-15T14:31:21.230392+0000","last_unstale":"2023-01-15T14:31:25.135080+0000","last_undegraded":"2023-01-15T14:31:25.135080+0000","last_fullsized":"2023-01-15T14:31:25.135080+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30330885699999999,"stat_sum":{"num_bytes":865560,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3347,"num_write":5,"num_write_kb":2817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.c","version":"198'47","reported_seq":588,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.133782+0000","last_change":"2023-01-15T14:31:25.134898+0000","last_active":"2023-01-15T14:31:25.133782+0000","last_peered":"2023-01-15T14:31:25.133782+0000","last_clean":"2023-01-15T14:31:25.133782+0000","last_became_active":"2023-01-15T14:31:21.227945+0000","last_became_peered":"2023-01-15T14:31:21.227945+0000","last_unstale":"2023-01-15T14:31:25.133782+0000","last_undegraded":"2023-01-15T14:31:25.133782+0000","last_fullsized":"2023-01-15T14:31:25.133782+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.20216606200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.d","version":"198'164","reported_seq":800,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.999099+0000","last_change":"2023-01-15T14:31:25.741606+0000","last_active":"2023-01-15T14:31:25.999099+0000","last_peered":"2023-01-15T14:31:25.999099+0000","last_clean":"2023-01-15T14:31:25.999099+0000","last_became_active":"2023-01-15T14:31:21.132225+0000","last_became_peered":"2023-01-15T14:31:21.132225+0000","last_unstale":"2023-01-15T14:31:25.999099+0000","last_undegraded":"2023-01-15T14:31:25.999099+0000","last_fullsized":"2023-01-15T14:31:25.999099+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.60687154099999996,"stat_sum":{"num_bytes":3033663,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":12,"num_read_kb":14682,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":578,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:26.070968+0000","last_change":"2023-01-15T14:31:25.045842+0000","last_active":"2023-01-15T14:31:26.070968+0000","last_peered":"2023-01-15T14:31:26.070968+0000","last_clean":"2023-01-15T14:31:26.070968+0000","last_became_active":"2023-01-15T14:31:03.094506+0000","last_became_peered":"2023-01-15T14:31:03.094506+0000","last_unstale":"2023-01-15T14:31:26.070968+0000","last_undegraded":"2023-01-15T14:31:26.070968+0000","last_fullsized":"2023-01-15T14:31:26.070968+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'37","last_scrub_stamp":"2023-01-15T14:30:25.793922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:25.793922+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.10705419300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":11529,"num_write":5,"num_write_kb":4046,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1487419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":798,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.989749+0000","last_change":"2023-01-15T14:31:25.047333+0000","last_active":"2023-01-15T14:31:25.989749+0000","last_peered":"2023-01-15T14:31:25.989749+0000","last_clean":"2023-01-15T14:31:25.989749+0000","last_became_active":"2023-01-15T14:31:06.028460+0000","last_became_peered":"2023-01-15T14:31:06.028460+0000","last_unstale":"2023-01-15T14:31:25.989749+0000","last_undegraded":"2023-01-15T14:31:25.989749+0000","last_fullsized":"2023-01-15T14:31:25.989749+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:31:22.549180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:22.549180+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:23.696","scrub_duration":0.50470070300000003,"objects_trimmed":0,"snaptrim_duration":0.010859807000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.10","version":"197'243","reported_seq":899,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.035352+0000","last_change":"2023-01-15T14:31:25.038319+0000","last_active":"2023-01-15T14:31:25.035352+0000","last_peered":"2023-01-15T14:31:25.035352+0000","last_clean":"2023-01-15T14:31:25.035352+0000","last_became_active":"2023-01-15T14:31:11.135827+0000","last_became_peered":"2023-01-15T14:31:11.135827+0000","last_unstale":"2023-01-15T14:31:25.035352+0000","last_undegraded":"2023-01-15T14:31:25.035352+0000","last_fullsized":"2023-01-15T14:31:25.035352+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.105996643,"stat_sum":{"num_bytes":9496038,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":60,"num_read_kb":51894,"num_write":65,"num_write_kb":26340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61,"num_bytes_recovered":26680915,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.11","version":"198'148","reported_seq":807,"reported_epoch":199,"state":"active+clean","last_fresh":"2023-01-15T14:31:26.098543+0000","last_change":"2023-01-15T14:31:25.136696+0000","last_active":"2023-01-15T14:31:26.098543+0000","last_peered":"2023-01-15T14:31:26.098543+0000","last_clean":"2023-01-15T14:31:26.098543+0000","last_became_active":"2023-01-15T14:31:06.888501+0000","last_became_peered":"2023-01-15T14:31:06.888501+0000","last_unstale":"2023-01-15T14:31:26.098543+0000","last_undegraded":"2023-01-15T14:31:26.098543+0000","last_fullsized":"2023-01-15T14:31:26.098543+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":180,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'144","last_scrub_stamp":"2023-01-15T14:31:13.806973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:13.806973+0000","objects_scrubbed":4,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:29.247","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.0027069189999999999,"stat_sum":{"num_bytes":14235628,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":42898,"num_write":14,"num_write_kb":12358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":14420439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.12","version":"198'137","reported_seq":378,"reported_epoch":198,"state":"active+clean","last_fresh":"2023-01-15T14:31:25.346538+0000","last_change":"2023-01-15T14:31:25.346538+0000","last_active":"2023-01-15T14:31:25.346538+0000","last_peered":"2023-01-15T14:31:25.346538+0000","last_clean":"2023-01-15T14:31:25.346538+0000","last_became_active":"2023-01-15T14:31:03.093394+0000","last_became_peered":"2023-01-15T14:31:03.093394+0000","last_unstale":"2023-01-15T14:31:25.346538+0000","last_undegraded":"2023-01-15T14:31:25.346538+0000","last_fullsized":"2023-01-15T14:31:25.346538+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"139'103","last_scrub_stamp":"2023-01-15T14:30:30.045615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:30.045615+0000","objects_scrubbed":3,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:36.166","scrub_duration":0.207128334,"objects_trimmed":0,"snaptrim_duration":0.31365726199999999,"stat_sum":{"num_bytes":8638831,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":22,"num_read_kb":27345,"num_write":12,"num_write_kb":4435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":12585229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]},{"pgid":"3.13","version":"200'270","reported_seq":823,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-01-15T14:31:27.115055+0000","last_change":"2023-01-15T14:31:25.036950+0000","last_active":"2023-01-15T14:31:27.115055+0000","last_peered":"2023-01-15T14:31:27.115055+0000","last_clean":"2023-01-15T14:31:27.115055+0000","last_became_active":"2023-01-15T14:30:08.974612+0000","last_became_peered":"2023-01-15T14:30:08.974612+0000","last_unstale":"2023-01-15T14:31:27.115055+0000","last_undegraded":"2023-01-15T14:31:27.115055+0000","last_fullsized":"2023-01-15T14:31:27.115055+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":5,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:31:29.544","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.0027070969999999999,"stat_sum":{"num_bytes":12873246,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":35,"num_read_kb":29731,"num_write":24,"num_write_kb":12096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"67","length":"1"},{"start":"69","length":"4"},{"start":"6e","length":"1"},{"start":"73","length":"2"},{"start":"78","length":"1"},{"start":"7a","length":"1"},{"start":"7c","length":"2"},{"start":"80","length":"1"},{"start":"82","length":"1"},{"start":"84","length":"2"},{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"94","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":46,"stat_sum":{"num_bytes":182836200,"num_objects":93,"num_object_clones":50,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":10,"num_read":657,"num_read_kb":671610,"num_write":545,"num_write_kb":219552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":508,"num_bytes_recovered":311576244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6319,"ondisk_log_size":6319,"up":138,"acting":138,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":17,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":862804,"kb_used_data":92839560,"kb_used_omap":2398,"kb_used_meta":0,"kb_avail":92836576,"statfs":{"total":95948165120,"available":95064653824,"internally_reserved":3055616,"allocated":95067709440,"data_stored":95067709440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2456258,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":14.18,"5min":5.4340000000000002,"15min":4.524},"min":{"1min":0.57099999999999995,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":148.10900000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.753,"5min":7.5119999999999996,"15min":8.6430000000000007},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0569999999999999,"5min":73.769000000000005,"15min":73.769000000000005},"last":2.0099999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":5.3419999999999996,"15min":6.1440000000000001},"min":{"1min":0.47699999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":88.003,"15min":88.003},"last":0.92100000000000004},{"interface":"front","average":{"1min":6.2370000000000001,"5min":6.0810000000000004,"15min":6.2130000000000001},"min":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":97.704999999999998,"5min":97.704999999999998,"15min":97.704999999999998},"last":1.948}]},{"osd":2,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":5.8250000000000002,"15min":5.1349999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.032,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.76800000000000002},{"interface":"front","average":{"1min":5.8869999999999996,"5min":3.8650000000000002,"15min":4.0730000000000004},"min":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":91.837999999999994,"5min":91.837999999999994,"15min":91.837999999999994},"last":1.0349999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":3.782,"15min":1.8919999999999999},"min":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":15.58,"5min":144.65600000000001,"15min":144.65600000000001},"last":1.056},{"interface":"front","average":{"1min":2.8439999999999999,"5min":4.173,"15min":4.492},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":31.751000000000001,"5min":76.034999999999997,"15min":76.034999999999997},"last":0.99399999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":6.8899999999999997,"15min":8.5739999999999998},"min":{"1min":0.60499999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0609999999999999,"5min":76.658000000000001,"15min":76.658000000000001},"last":1.591},{"interface":"front","average":{"1min":1.042,"5min":4.0679999999999996,"15min":1.9990000000000001},"min":{"1min":0.61899999999999999,"5min":0.498,"15min":0.498},"max":{"1min":3.944,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.96399999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":5.0599999999999996,"15min":5.9039999999999999},"min":{"1min":0.51200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1799999999999999,"5min":73.561000000000007,"15min":73.561000000000007},"last":1.714},{"interface":"front","average":{"1min":0.92100000000000004,"5min":4.226,"15min":4.7050000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.3029999999999999,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.115}]},{"osd":6,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":6.4340000000000002,"15min":7.827},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.216,"5min":73.638999999999996,"15min":73.638999999999996},"last":2.0670000000000002},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.9690000000000001,"15min":1.389},"min":{"1min":0.74399999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1559999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.234}]}]},{"osd":6,"up_from":0,"seq":55834574891,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574891,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":35,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":982500,"kb_used_data":92728364,"kb_used_omap":4686,"kb_used_meta":0,"kb_avail":92716880,"statfs":{"total":95948165120,"available":94942085120,"internally_reserved":11759616,"allocated":94953844736,"data_stored":94953844736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4799077,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":4.633,"15min":4.8209999999999997},"min":{"1min":0.47099999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.282,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.83599999999999997},{"interface":"front","average":{"1min":4.0869999999999997,"5min":2.008,"15min":1.1899999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":70.352999999999994,"5min":70.352999999999994,"15min":70.352999999999994},"last":0.56999999999999995}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":2.145,"5min":7.5339999999999998,"15min":8.2400000000000002},"min":{"1min":0.60399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":24.489999999999998,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":12.505000000000001,"5min":4.1219999999999999,"15min":1.903},"min":{"1min":0.52900000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":110.06999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.60499999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":5.4859999999999998,"15min":5.5019999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":97.974000000000004,"5min":99.884,"15min":99.884},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.88500000000000001,"15min":0.90100000000000002},"min":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.254,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.68000000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.2649999999999999,"15min":1.3140000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.3280000000000001,"5min":12.878,"15min":12.878},"last":0.69799999999999995},{"interface":"front","average":{"1min":2.6320000000000001,"5min":5.5659999999999998,"15min":5.9560000000000004},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":24.649999999999999,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.71699999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.88600000000000001,"15min":0.90800000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.109,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.746,"15min":2.1139999999999999},"min":{"1min":0.65200000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3879999999999999,"5min":30.971,"15min":30.971},"last":0.74299999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":2.2149999999999999,"5min":5.8090000000000002,"15min":5.718},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":24.553999999999998,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.875},{"interface":"front","average":{"1min":0.92600000000000005,"5min":3.4910000000000001,"15min":4.6050000000000004},"min":{"1min":0.66700000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.234,"5min":88.186999999999998,"15min":88.186999999999998},"last":1.107}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":8.266,"15min":11.507},"min":{"1min":0.63400000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.4510000000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":2.2770000000000001,"5min":8.2899999999999991,"15min":10.256},"min":{"1min":0.65600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":24.603999999999999,"5min":99.843000000000004,"15min":99.843000000000004},"last":0.93200000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":40,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":971728,"kb_used_data":92737796,"kb_used_omap":4953,"kb_used_meta":0,"kb_avail":92727652,"statfs":{"total":95948165120,"available":94953115648,"internally_reserved":10387456,"allocated":94963503104,"data_stored":94963503104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5072094,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":8.657,"5min":6.4790000000000001,"15min":5.391},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":64.275999999999996,"5min":174.52199999999999,"15min":174.52199999999999},"last":122.065},{"interface":"front","average":{"1min":4.7889999999999997,"5min":6.2050000000000001,"15min":7.1630000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":36.207000000000001,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.49299999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.3579999999999997,"5min":11.628,"15min":12.897},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":60.664000000000001,"5min":159.256,"15min":159.256},"last":0.47299999999999998},{"interface":"front","average":{"1min":19.719999999999999,"5min":6.1059999999999999,"15min":2.5289999999999999},"min":{"1min":0.40200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":155.44,"5min":174.578,"15min":174.578},"last":121.203}]},{"osd":2,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":3.1429999999999998,"5min":3.153,"15min":3.2829999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":40.692999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":7.3019999999999996,"5min":4.5700000000000003,"15min":2.9849999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":64.381,"5min":174.45099999999999,"15min":174.45099999999999},"last":122.13}]},{"osd":4,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":8.7050000000000001,"5min":6.2599999999999998,"15min":6.4610000000000003},"min":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":72.438999999999993,"5min":72.438999999999993,"15min":72.438999999999993},"last":0.79700000000000004},{"interface":"front","average":{"1min":8.8840000000000003,"5min":12.984,"15min":13.577},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":60.857999999999997,"5min":159.47200000000001,"15min":159.47200000000001},"last":0.59699999999999998}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.226,"5min":11.475,"15min":13.093},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":60.912999999999997,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.625},{"interface":"front","average":{"1min":4.8529999999999998,"5min":3.359,"15min":1.8009999999999999},"min":{"1min":0.752,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":64.513000000000005,"5min":174.61000000000001,"15min":174.61000000000001},"last":122.208}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.8799999999999999,"5min":3.3690000000000002,"15min":1.8109999999999999},"min":{"1min":0.69999999999999996,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":64.590000000000003,"5min":174.63999999999999,"15min":174.63999999999999},"last":122.26600000000001},{"interface":"front","average":{"1min":1,"5min":3.0529999999999999,"15min":3.5249999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.304,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.879}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":3.206,"15min":3.7589999999999999},"min":{"1min":0.69699999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.2110000000000001,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.94199999999999995},{"interface":"front","average":{"1min":5.3739999999999997,"5min":14.025,"15min":16.734999999999999},"min":{"1min":0.72599999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":60.951999999999998,"5min":159.512,"15min":159.512},"last":0.82699999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":30,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":948832,"kb_used_data":92758896,"kb_used_omap":3728,"kb_used_meta":0,"kb_avail":92750548,"statfs":{"total":95948165120,"available":94976561152,"internally_reserved":8548352,"allocated":94985109504,"data_stored":94985109504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3818192,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":12,"commit_latency_ns":0,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.5739999999999998,"5min":2.5009999999999999,"15min":1.2609999999999999},"min":{"1min":0.439,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":66.277000000000001,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":4.617,"5min":1.4670000000000001,"15min":0.96499999999999997},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":88.299999999999997,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.52900000000000003}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.2610000000000001,"5min":3.7229999999999999,"15min":1.7729999999999999},"min":{"1min":0.46500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":80.233999999999995,"5min":143.071,"15min":143.071},"last":0.41699999999999998},{"interface":"front","average":{"1min":4.6989999999999998,"5min":3.1000000000000001,"15min":3.27},"min":{"1min":0.505,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":88.186000000000007,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.59599999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":10.462,"15min":10.422000000000001},"min":{"1min":0.40999999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.94899999999999995,"5min":183.53,"15min":183.53},"last":0.61499999999999999},{"interface":"front","average":{"1min":5.7729999999999997,"5min":3.8149999999999999,"15min":1.7829999999999999},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":66.335999999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.56799999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":5.9489999999999998,"15min":4.4160000000000004},"min":{"1min":0.624,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":66.491,"5min":175.34700000000001,"15min":175.34700000000001},"last":24.875},{"interface":"front","average":{"1min":4.9290000000000003,"5min":5.8259999999999996,"15min":4.4809999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":88.459999999999994,"5min":175.30500000000001,"15min":175.30500000000001},"last":24.859999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":2.577,"15min":1.643},"min":{"1min":0.65800000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2829999999999999,"5min":143.262,"15min":143.262},"last":0.76200000000000001},{"interface":"front","average":{"1min":6.0339999999999998,"5min":2.1040000000000001,"15min":1.4750000000000001},"min":{"1min":0.71899999999999997,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":66.456999999999994,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.78000000000000003}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.9889999999999999,"5min":1.8879999999999999,"15min":1.3169999999999999},"min":{"1min":0.70699999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":88.542000000000002,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.028,"5min":2.7109999999999999,"15min":1.7150000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.585,"5min":143.215,"15min":143.215},"last":0.80600000000000005}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.6230000000000002,"5min":3.5489999999999999,"15min":3.222},"min":{"1min":0.81100000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":88.623999999999995,"5min":88.623999999999995,"15min":88.623999999999995},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.048,"5min":3.5630000000000002,"15min":3.0390000000000001},"min":{"1min":0.79900000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3859999999999999,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.69099999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574891,"num_pgs":39,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1010928,"kb_used_data":92697996,"kb_used_omap":4908,"kb_used_meta":0,"kb_avail":92688452,"statfs":{"total":95948165120,"available":94912974848,"internally_reserved":9773056,"allocated":94922747904,"data_stored":94922747904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5026579,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":12,"commit_latency_ns":0,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3609999999999998,"5min":4.9240000000000004,"15min":4.2359999999999998},"min":{"1min":0.39900000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":100.533,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":4.6509999999999998,"5min":3.6459999999999999,"15min":4.0970000000000004},"min":{"1min":0.373,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":85.751000000000005,"5min":85.751000000000005,"15min":85.751000000000005},"last":0.72399999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":2.6600000000000001,"5min":1.2270000000000001,"15min":1.0529999999999999},"min":{"1min":0.32500000000000001,"5min":0.245,"15min":0.245},"max":{"1min":43.759999999999998,"5min":43.759999999999998,"15min":43.759999999999998},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":3.1869999999999998,"15min":3.7770000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82899999999999996,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.94699999999999995}]},{"osd":3,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":17.001000000000001,"5min":15.438000000000001,"15min":18.370000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":245.637,"5min":282.01400000000001,"15min":282.01400000000001},"last":1.0740000000000001},{"interface":"front","average":{"1min":5.6310000000000002,"5min":5.9980000000000002,"15min":6.8289999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":85.814999999999998,"5min":85.814999999999998,"15min":85.814999999999998},"last":0.62}]},{"osd":4,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":63.935000000000002,"5min":16.309999999999999,"15min":8.5879999999999992},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1181.3389999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":57.106000000000002,"5min":14.65,"15min":7.2859999999999996},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1181.8720000000001,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":1.091}]},{"osd":5,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.98099999999999998,"15min":1.0149999999999999},"min":{"1min":0.58099999999999996,"5min":0.433,"15min":0.433},"max":{"1min":1.244,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":1.266},{"interface":"front","average":{"1min":0.92200000000000004,"5min":3.036,"15min":3.9700000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.1339999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.1220000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":4.2060000000000004,"15min":4.7359999999999998},"min":{"1min":0.745,"5min":0.63,"15min":0.63},"max":{"1min":85.884,"5min":85.884,"15min":85.884},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.056,"15min":1.101},"min":{"1min":0.69599999999999995,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.298,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.169}]},{"osd":7,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":8.3879999999999999,"5min":7.1310000000000002,"15min":7.0549999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":157.03700000000001,"5min":157.03700000000001,"15min":157.03700000000001},"last":1.196},{"interface":"front","average":{"1min":5.0919999999999996,"5min":4.0730000000000004,"15min":4.524},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":85.852999999999994,"5min":85.852999999999994,"15min":85.852999999999994},"last":0.92800000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":818636,"kb_used_data":92886128,"kb_used_omap":4173,"kb_used_meta":0,"kb_avail":92880744,"statfs":{"total":95948165120,"available":95109881856,"internally_reserved":5513216,"allocated":95115395072,"data_stored":95115395072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4274036,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":7.2519999999999998,"5min":2.6970000000000001,"15min":1.3140000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":69.816999999999993,"5min":69.872,"15min":69.872},"last":0.51400000000000001},{"interface":"front","average":{"1min":5.6130000000000004,"5min":2.6789999999999998,"15min":1.379},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":98.203999999999994,"5min":98.203999999999994,"15min":98.203999999999994},"last":3.569}]},{"osd":2,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.7039999999999997,"5min":3.407,"15min":1.5980000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":98.286000000000001,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":3.0270000000000001,"5min":2.6819999999999999,"15min":1.3660000000000001},"min":{"1min":0.38400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":48.893000000000001,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.66800000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":12.154999999999999,"5min":13.413,"15min":15.657},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":94.712999999999994,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":10.484,"5min":5.6399999999999997,"15min":2.4140000000000001},"min":{"1min":0.42399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":98.352999999999994,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.62}]},{"osd":4,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.5570000000000004,"5min":3.2309999999999999,"15min":1.734},"min":{"1min":0.63800000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":61.536000000000001,"5min":71.483000000000004,"15min":71.483000000000004},"last":0.82999999999999996},{"interface":"front","average":{"1min":9.2989999999999995,"5min":3.9950000000000001,"15min":2.0150000000000001},"min":{"1min":0.67400000000000004,"5min":0.439,"15min":0.439},"max":{"1min":70.179000000000002,"5min":70.179000000000002,"15min":70.179000000000002},"last":0.877}]},{"osd":5,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.9009999999999998,"5min":8.218,"15min":9.2710000000000008},"min":{"1min":0.68500000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":98.411000000000001,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.7709999999999999,"15min":1.2829999999999999},"min":{"1min":0.63400000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.2959999999999998,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.89800000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":7.7030000000000003,"5min":3.2040000000000002,"15min":1.8480000000000001},"min":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":70.259,"5min":70.259,"15min":70.259},"last":0.92100000000000004},{"interface":"front","average":{"1min":5.984,"5min":3.056,"15min":1.774},"min":{"1min":0.749,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":98.484999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":0.85699999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":4.633,"5min":12.002000000000001,"15min":14.448},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":73.864999999999995,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":7.7199999999999998,"5min":3.21,"15min":1.8500000000000001},"min":{"1min":0.70399999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":70.322000000000003,"5min":70.322000000000003,"15min":70.322000000000003},"last":0.80100000000000005}]}]}],"pool_statfs":[]}} 2023-01-15T14:31:29.857 INFO:tasks.thrashosds.thrasher:Setting 3.2d to [2, 3, 5, 4, 1, 6] 2023-01-15T14:31:29.857 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.2d', '2', '3', '5', '4', '1', '6'] 2023-01-15T14:31:29.858 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.2d 2 3 5 4 1 6 2023-01-15T14:31:29.860 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:29.860 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:29.860 INFO:tasks.rados.rados.0.smithi090.stdout:797: finishing write tid 4 to smithi09025264-36 2023-01-15T14:31:29.860 INFO:tasks.rados.rados.0.smithi090.stdout:797: finishing write tid 5 to smithi09025264-36 2023-01-15T14:31:29.860 INFO:tasks.rados.rados.0.smithi090.stdout:797: oid 36 updating version 0 to 243 2023-01-15T14:31:29.860 INFO:tasks.rados.rados.0.smithi090.stdout:797: oid 36 version 243 is already newer than 241 2023-01-15T14:31:30.491 INFO:tasks.rados.rados.0.smithi090.stdout:797: oid 36 version 243 is already newer than 242 2023-01-15T14:31:30.491 INFO:tasks.rados.rados.0.smithi090.stdout:797: oid 36 version 243 is already newer than 240 2023-01-15T14:31:30.491 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 243 (ObjNum 313 snap 80 seq_num 313) dirty exists 2023-01-15T14:31:30.491 INFO:tasks.rados.rados.0.smithi090.stdout:797: left oid 36 (ObjNum 313 snap 80 seq_num 313) 2023-01-15T14:31:30.492 INFO:tasks.rados.rados.0.smithi090.stdout:795: finishing rollback tid 0 to smithi09025264-4 2023-01-15T14:31:30.492 INFO:tasks.rados.rados.0.smithi090.stdout:790: done (8 left) 2023-01-15T14:31:30.492 INFO:tasks.rados.rados.0.smithi090.stdout:791: done (7 left) 2023-01-15T14:31:30.492 INFO:tasks.rados.rados.0.smithi090.stdout:792: done (6 left) 2023-01-15T14:31:30.492 INFO:tasks.rados.rados.0.smithi090.stdout:793: done (5 left) 2023-01-15T14:31:30.492 INFO:tasks.rados.rados.0.smithi090.stdout:796: done (4 left) 2023-01-15T14:31:30.492 INFO:tasks.rados.rados.0.smithi090.stdout:797: done (3 left) 2023-01-15T14:31:30.492 INFO:tasks.rados.rados.0.smithi090.stdout:798: done (2 left) 2023-01-15T14:31:30.492 INFO:tasks.rados.rados.0.smithi090.stdout:799: done (1 left) 2023-01-15T14:31:30.492 INFO:tasks.rados.rados.0.smithi090.stdout:800: snap_create 2023-01-15T14:31:30.493 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:30.493 INFO:tasks.rados.rados.0.smithi090.stdout:795: finishing rollback tid 1 to smithi09025264-4 2023-01-15T14:31:30.493 INFO:tasks.rados.rados.0.smithi090.stdout:795: finishing rollback tid 2 to smithi09025264-4 2023-01-15T14:31:30.493 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 4 v 157 (ObjNum 143 snap 27 seq_num 143) dirty exists 2023-01-15T14:31:30.493 INFO:tasks.rados.rados.0.smithi090.stdout:795: done (0 left) 2023-01-15T14:31:30.494 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:31:30.495 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:29.855+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:30.497 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:31:30.498 INFO:tasks.rados.rados.0.smithi090.stdout:800: done (0 left) 2023-01-15T14:31:30.499 INFO:tasks.rados.rados.0.smithi090.stdout:801: read oid 19 snap -1 2023-01-15T14:31:30.499 INFO:tasks.rados.rados.0.smithi090.stdout:801: expect (ObjNum 307 snap 78 seq_num 307) 2023-01-15T14:31:30.499 INFO:tasks.rados.rados.0.smithi090.stdout:802: read oid 42 snap -1 2023-01-15T14:31:30.499 INFO:tasks.rados.rados.0.smithi090.stdout:802: expect deleted 2023-01-15T14:31:30.500 INFO:tasks.rados.rados.0.smithi090.stdout:803: read oid 26 snap 78 2023-01-15T14:31:30.500 INFO:tasks.rados.rados.0.smithi090.stdout:803: expect (ObjNum 295 snap 78 seq_num 295) 2023-01-15T14:31:30.500 INFO:tasks.rados.rados.0.smithi090.stdout:804: rollback oid 3 current snap is 81 2023-01-15T14:31:30.500 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 3 to 67 2023-01-15T14:31:30.500 INFO:tasks.rados.rados.0.smithi090.stdout:805: copy_from oid 33 from oid 43 current snap is 81 2023-01-15T14:31:30.501 INFO:tasks.rados.rados.0.smithi090.stdout:806: copy_from oid 36 from oid 38 current snap is 81 2023-01-15T14:31:30.501 INFO:tasks.rados.rados.0.smithi090.stdout:807: write oid 17 current snap is 81 2023-01-15T14:31:30.501 INFO:tasks.rados.rados.0.smithi090.stdout:807: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2023-01-15T14:31:30.501 INFO:tasks.rados.rados.0.smithi090.stdout:807: writing smithi09025264-17 from 494205 to 968113 tid 1 2023-01-15T14:31:30.502 INFO:tasks.rados.rados.0.smithi090.stdout:807: writing smithi09025264-17 from 1433122 to 1840224 tid 2 2023-01-15T14:31:30.502 INFO:tasks.rados.rados.0.smithi090.stdout:807: writing smithi09025264-17 from 2384125 to 2846809 tid 3 2023-01-15T14:31:30.502 INFO:tasks.rados.rados.0.smithi090.stdout:808: snap_remove snap 74 2023-01-15T14:31:30.503 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:30.495+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:30.599 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:30.594+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:30.697 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:30.698 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:30.698 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:30.698 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:31:30.726 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:31.120 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:31:31.121 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:30.738+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:31.122 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:31.122 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:31.122 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:31.122 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:31.122 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:31.122 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:31.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:31:31.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:31:31.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:31:31.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:31:31.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:31:31.124 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:31:31.124 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:31:31.124 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:31:31.124 INFO:tasks.rados.rados.0.smithi090.stdout:808: done (7 left) 2023-01-15T14:31:31.124 INFO:tasks.rados.rados.0.smithi090.stdout:809: copy_from oid 11 from oid 21 current snap is 81 2023-01-15T14:31:31.125 INFO:tasks.rados.rados.0.smithi090.stdout:806: finishing copy_from to smithi09025264-36 2023-01-15T14:31:31.125 INFO:tasks.rados.rados.0.smithi090.stdout:806: got expected ENOENT (src dne) 2023-01-15T14:31:31.125 INFO:tasks.rados.rados.0.smithi090.stdout:806: finishing copy_from racing read to smithi09025264-36 2023-01-15T14:31:31.125 INFO:tasks.rados.rados.0.smithi090.stdout:804: finishing rollback tid 0 to smithi09025264-3 2023-01-15T14:31:31.125 INFO:tasks.rados.rados.0.smithi090.stdout:804: finishing rollback tid 1 to smithi09025264-3 2023-01-15T14:31:31.125 INFO:tasks.rados.rados.0.smithi090.stdout:803: expect (ObjNum 295 snap 78 seq_num 295) 2023-01-15T14:31:31.126 INFO:teuthology.orchestra.run.smithi090.stderr:set 3.2d pg_upmap_items mapping to [2->3,5->4,1->6] 2023-01-15T14:31:31.127 INFO:tasks.rados.rados.0.smithi090.stdout:807: finishing write tid 1 to smithi09025264-17 2023-01-15T14:31:31.127 INFO:tasks.rados.rados.0.smithi090.stdout:807: finishing write tid 2 to smithi09025264-17 2023-01-15T14:31:31.127 INFO:tasks.rados.rados.0.smithi090.stdout:805: finishing copy_from to smithi09025264-33 2023-01-15T14:31:31.127 INFO:tasks.rados.rados.0.smithi090.stdout:805: got expected ENOENT (src dne) 2023-01-15T14:31:31.127 INFO:tasks.rados.rados.0.smithi090.stdout:807: finishing write tid 3 to smithi09025264-17 2023-01-15T14:31:31.127 INFO:tasks.rados.rados.0.smithi090.stdout:801: expect (ObjNum 307 snap 78 seq_num 307) 2023-01-15T14:31:31.128 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:31.115+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:31.215 INFO:tasks.rados.rados.0.smithi090.stdout:807: finishing write tid 4 to smithi09025264-17 2023-01-15T14:31:31.322 INFO:tasks.rados.rados.0.smithi090.stdout:804: finishing rollback tid 2 to smithi09025264-3 2023-01-15T14:31:31.322 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 221 (ObjNum 253 snap 60 seq_num 253) dirty exists 2023-01-15T14:31:31.322 INFO:tasks.rados.rados.0.smithi090.stdout:805: finishing copy_from racing read to smithi09025264-33 2023-01-15T14:31:31.322 INFO:tasks.rados.rados.0.smithi090.stdout:807: finishing write tid 5 to smithi09025264-17 2023-01-15T14:31:31.322 INFO:tasks.rados.rados.0.smithi090.stdout:807: oid 17 updating version 0 to 245 2023-01-15T14:31:31.322 INFO:tasks.rados.rados.0.smithi090.stdout:807: oid 17 updating version 245 to 246 2023-01-15T14:31:31.323 INFO:tasks.rados.rados.0.smithi090.stdout:807: oid 17 updating version 246 to 247 2023-01-15T14:31:31.323 INFO:tasks.rados.rados.0.smithi090.stdout:807: oid 17 updating version 247 to 248 2023-01-15T14:31:31.323 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 248 (ObjNum 315 snap 81 seq_num 315) dirty exists 2023-01-15T14:31:31.323 INFO:tasks.rados.rados.0.smithi090.stdout:807: left oid 17 (ObjNum 315 snap 81 seq_num 315) 2023-01-15T14:31:31.323 INFO:tasks.rados.rados.0.smithi090.stdout:801: done (7 left) 2023-01-15T14:31:31.323 INFO:tasks.rados.rados.0.smithi090.stdout:802: done (6 left) 2023-01-15T14:31:31.323 INFO:tasks.rados.rados.0.smithi090.stdout:803: done (5 left) 2023-01-15T14:31:31.323 INFO:tasks.rados.rados.0.smithi090.stdout:804: done (4 left) 2023-01-15T14:31:31.323 INFO:tasks.rados.rados.0.smithi090.stdout:805: done (3 left) 2023-01-15T14:31:31.323 INFO:tasks.rados.rados.0.smithi090.stdout:806: done (2 left) 2023-01-15T14:31:31.323 INFO:tasks.rados.rados.0.smithi090.stdout:807: done (1 left) 2023-01-15T14:31:31.324 INFO:tasks.rados.rados.0.smithi090.stdout:810: read oid 28 snap 72 2023-01-15T14:31:31.324 INFO:tasks.rados.rados.0.smithi090.stdout:810: expect (ObjNum 275 snap 70 seq_num 275) 2023-01-15T14:31:31.324 INFO:tasks.rados.rados.0.smithi090.stdout:811: snap_create 2023-01-15T14:31:31.324 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:31.326 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:31.326 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:31:31.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:31:31.326 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:31:31.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.327 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:97 3.a 3:52df5138:::smithi09025264-25:head [write 403368~555035,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:31.327 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:31:31.327 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 184.73686080300001, 2023-01-15T14:31:31.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.340758171, 2023-01-15T14:31:31.327 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:31.327 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:31.327 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:31.327 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:31.327 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:31.327 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 97 2023-01-15T14:31:31.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.328 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:31.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:31.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:31:31.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:31.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:31.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:31:31.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:31.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:31.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501140+0000", 2023-01-15T14:31:31.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:31:31.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:31.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501684+0000", 2023-01-15T14:31:31.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054441799999999996 2023-01-15T14:31:31.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:31.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501713+0000", 2023-01-15T14:31:31.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8833e-05 2023-01-15T14:31:31.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:31.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501726+0000", 2023-01-15T14:31:31.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3137e-05 2023-01-15T14:31:31.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:31.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601982+0000", 2023-01-15T14:31:31.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100256518 2023-01-15T14:31:31.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:31.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602426+0000", 2023-01-15T14:31:31.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044401500000000002 2023-01-15T14:31:31.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602703+0000", 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000276889 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603066+0000", 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036278599999999998 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:31.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603098+0000", 2023-01-15T14:31:31.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1433000000000002e-05 2023-01-15T14:31:31.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:31.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603382+0000", 2023-01-15T14:31:31.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028402899999999999 2023-01-15T14:31:31.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:31.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603632+0000", 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025033 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805354+0000", 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20172168700000001 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841465+0000", 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.036111852999999999 2023-01-15T14:31:31.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:31.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:31.930 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:31.930 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841682+0000", 2023-01-15T14:31:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000216949 2023-01-15T14:31:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841901+0000", 2023-01-15T14:31:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021858700000000001 2023-01-15T14:31:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:32.317 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:98 3.a 3:52df5138:::smithi09025264-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:31:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 184.73625113200001, 2023-01-15T14:31:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.44410839299999999, 2023-01-15T14:31:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:32.318 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 98 2023-01-15T14:31:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:31:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:31:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501752+0000", 2023-01-15T14:31:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:32.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502223+0000", 2023-01-15T14:31:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000471843 2023-01-15T14:31:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502240+0000", 2023-01-15T14:31:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6663e-05 2023-01-15T14:31:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502248+0000", 2023-01-15T14:31:32.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5669999999999997e-06 2023-01-15T14:31:32.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:32.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603236+0000", 2023-01-15T14:31:32.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100988439 2023-01-15T14:31:32.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603384+0000", 2023-01-15T14:31:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014844200000000001 2023-01-15T14:31:32.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842016+0000", 2023-01-15T14:31:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23863163800000001 2023-01-15T14:31:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842267+0000", 2023-01-15T14:31:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025101300000000002 2023-01-15T14:31:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842561+0000", 2023-01-15T14:31:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029432900000000002 2023-01-15T14:31:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843120+0000", 2023-01-15T14:31:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055900500000000001 2023-01-15T14:31:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:32.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843146+0000", 2023-01-15T14:31:32.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5273e-05 2023-01-15T14:31:32.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:32.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843512+0000", 2023-01-15T14:31:32.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036621599999999999 2023-01-15T14:31:32.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:32.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943324+0000", 2023-01-15T14:31:32.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099812008999999993 2023-01-15T14:31:32.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:32.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945437+0000", 2023-01-15T14:31:32.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021126069999999999 2023-01-15T14:31:32.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:32.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945619+0000", 2023-01-15T14:31:32.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018280600000000001 2023-01-15T14:31:32.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:32.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945744+0000", 2023-01-15T14:31:32.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012511299999999999 2023-01-15T14:31:32.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:32.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945861+0000", 2023-01-15T14:31:32.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011646000000000001 2023-01-15T14:31:32.331 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.515 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:32.515 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.516 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:99 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:32.519 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:31:32.519 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 184.73574087599999, 2023-01-15T14:31:32.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.54355587900000002, 2023-01-15T14:31:32.519 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:32.519 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:32.519 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:32.519 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:32.519 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:32.519 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 99 2023-01-15T14:31:32.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.519 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:32.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:32.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:31:32.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:32.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:31:32.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502262+0000", 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502270+0000", 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1999999999999994e-06 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:32.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502286+0000", 2023-01-15T14:31:32.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5423000000000001e-05 2023-01-15T14:31:32.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:32.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502295+0000", 2023-01-15T14:31:32.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4339999999999998e-06 2023-01-15T14:31:32.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:32.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603467+0000", 2023-01-15T14:31:32.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101172225 2023-01-15T14:31:32.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:32.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603585+0000", 2023-01-15T14:31:32.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117149 2023-01-15T14:31:32.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:32.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843335+0000", 2023-01-15T14:31:32.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23975083799999999 2023-01-15T14:31:32.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843573+0000", 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023753000000000001 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843978+0000", 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040533799999999999 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844309+0000", 2023-01-15T14:31:32.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000330713 2023-01-15T14:31:32.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.526 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:32.526 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844423+0000", 2023-01-15T14:31:32.526 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114176 2023-01-15T14:31:32.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.526 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844510+0000", 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7026000000000003e-05 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943338+0000", 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098827398999999996 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945520+0000", 2023-01-15T14:31:32.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021824169999999999 2023-01-15T14:31:32.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:32.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045652+0000", 2023-01-15T14:31:32.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013193200000001 2023-01-15T14:31:32.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:32.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045733+0000", 2023-01-15T14:31:32.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1435999999999998e-05 2023-01-15T14:31:32.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:32.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045819+0000", 2023-01-15T14:31:32.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5292999999999998e-05 2023-01-15T14:31:32.529 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.529 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:32.529 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.529 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:100 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:32.529 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 184.73569295300001, 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.64425290599999996, 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 100 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.530 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502310+0000", 2023-01-15T14:31:32.531 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:32.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:32.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502316+0000", 2023-01-15T14:31:32.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7470000000000002e-06 2023-01-15T14:31:32.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:32.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502332+0000", 2023-01-15T14:31:32.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6866000000000001e-05 2023-01-15T14:31:32.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:32.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502343+0000", 2023-01-15T14:31:32.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0913e-05 2023-01-15T14:31:32.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:32.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703594+0000", 2023-01-15T14:31:32.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20125087999999999 2023-01-15T14:31:32.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:32.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703751+0000", 2023-01-15T14:31:32.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001573 2023-01-15T14:31:32.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:32.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845262+0000", 2023-01-15T14:31:32.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.141510521 2023-01-15T14:31:32.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:32.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845391+0000", 2023-01-15T14:31:32.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012872299999999999 2023-01-15T14:31:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146051+0000", 2023-01-15T14:31:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.300660023 2023-01-15T14:31:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146264+0000", 2023-01-15T14:31:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212754 2023-01-15T14:31:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:32.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146564+0000", 2023-01-15T14:31:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030010200000000001 2023-01-15T14:31:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:203 3.a 3:52df5138:::smithi09025264-25:head [write 440350~460568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:32.716 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:31:32.717 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 183.68488955800001, 2023-01-15T14:31:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40523678899999999, 2023-01-15T14:31:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 203 2023-01-15T14:31:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.718 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:31:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:31:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553111+0000", 2023-01-15T14:31:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553483+0000", 2023-01-15T14:31:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037153499999999999 2023-01-15T14:31:32.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553517+0000", 2023-01-15T14:31:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4687000000000002e-05 2023-01-15T14:31:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553537+0000", 2023-01-15T14:31:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.967e-05 2023-01-15T14:31:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553683+0000", 2023-01-15T14:31:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000146341 2023-01-15T14:31:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553880+0000", 2023-01-15T14:31:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019671900000000001 2023-01-15T14:31:32.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554219+0000", 2023-01-15T14:31:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033893299999999998 2023-01-15T14:31:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554536+0000", 2023-01-15T14:31:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316576 2023-01-15T14:31:32.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554562+0000", 2023-01-15T14:31:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5905999999999999e-05 2023-01-15T14:31:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554801+0000", 2023-01-15T14:31:32.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000239791 2023-01-15T14:31:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555274+0000", 2023-01-15T14:31:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047294500000000001 2023-01-15T14:31:32.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:32.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757267+0000", 2023-01-15T14:31:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20199229399999999 2023-01-15T14:31:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:32.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958174+0000", 2023-01-15T14:31:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20090739499999999 2023-01-15T14:31:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958260+0000", 2023-01-15T14:31:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6316999999999994e-05 2023-01-15T14:31:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:32.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958351+0000", 2023-01-15T14:31:32.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0573000000000002e-05 2023-01-15T14:31:32.915 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.915 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:32.915 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.915 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:204 3.a 3:52df5138:::smithi09025264-25:head [write 1309629~542666] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:32.915 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:31:32.915 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 183.68442669000001, 2023-01-15T14:31:32.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40500913999999999, 2023-01-15T14:31:32.916 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:32.916 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:32.916 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:32.916 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:32.916 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:32.916 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 204 2023-01-15T14:31:32.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.916 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:32.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:32.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:31:32.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:32.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:31:32.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:32.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553575+0000", 2023-01-15T14:31:32.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:32.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:32.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554126+0000", 2023-01-15T14:31:32.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055090499999999997 2023-01-15T14:31:32.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:32.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554150+0000", 2023-01-15T14:31:32.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3387000000000002e-05 2023-01-15T14:31:32.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:32.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554165+0000", 2023-01-15T14:31:32.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5116000000000001e-05 2023-01-15T14:31:32.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:32.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554626+0000", 2023-01-15T14:31:32.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046093499999999999 2023-01-15T14:31:32.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:32.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554738+0000", 2023-01-15T14:31:32.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112446 2023-01-15T14:31:32.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:32.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554898+0000", 2023-01-15T14:31:32.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016003100000000001 2023-01-15T14:31:32.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.921 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:32.921 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555182+0000", 2023-01-15T14:31:32.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028382700000000001 2023-01-15T14:31:32.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.921 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:32.921 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555199+0000", 2023-01-15T14:31:32.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6969e-05 2023-01-15T14:31:32.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:32.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555449+0000", 2023-01-15T14:31:32.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000250265 2023-01-15T14:31:32.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:32.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555574+0000", 2023-01-15T14:31:32.923 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124628 2023-01-15T14:31:32.923 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.923 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.923 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:32.923 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757316+0000", 2023-01-15T14:31:32.923 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20174214300000001 2023-01-15T14:31:32.923 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.923 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.924 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:32.924 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958460+0000", 2023-01-15T14:31:32.924 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201144464 2023-01-15T14:31:32.924 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.924 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.924 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:32.924 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958534+0000", 2023-01-15T14:31:32.924 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3380000000000006e-05 2023-01-15T14:31:32.924 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.924 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.924 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:32.925 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958586+0000", 2023-01-15T14:31:32.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2370000000000002e-05 2023-01-15T14:31:32.925 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.925 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:32.925 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:32.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.925 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:205 3.a 3:52df5138:::smithi09025264-25:head [write 2323973~516377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:32.925 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:31:32.925 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 183.68380859499999, 2023-01-15T14:31:32.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404601398, 2023-01-15T14:31:32.926 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:32.926 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:32.926 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:32.926 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:32.926 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:32.926 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 205 2023-01-15T14:31:32.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.926 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:32.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:32.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:31:32.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:32.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:31:32.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:32.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:32.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554193+0000", 2023-01-15T14:31:32.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:32.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:32.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554850+0000", 2023-01-15T14:31:32.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065674799999999997 2023-01-15T14:31:32.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:32.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554873+0000", 2023-01-15T14:31:32.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2935999999999999e-05 2023-01-15T14:31:32.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:32.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.654962+0000", 2023-01-15T14:31:32.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088949 2023-01-15T14:31:32.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:32.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655107+0000", 2023-01-15T14:31:32.930 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144759 2023-01-15T14:31:32.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.930 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:32.930 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655373+0000", 2023-01-15T14:31:32.930 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026684899999999999 2023-01-15T14:31:32.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.931 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:32.931 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655720+0000", 2023-01-15T14:31:32.931 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034601899999999998 2023-01-15T14:31:32.931 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.931 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.931 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:32.931 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656168+0000", 2023-01-15T14:31:32.931 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044851899999999998 2023-01-15T14:31:32.931 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.932 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.932 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:32.932 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656190+0000", 2023-01-15T14:31:32.932 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1817000000000001e-05 2023-01-15T14:31:32.932 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:32.932 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:32.932 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:32.932 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656574+0000", 2023-01-15T14:31:32.932 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038404200000000002 2023-01-15T14:31:32.932 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.115 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:33.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657649+0000", 2023-01-15T14:31:33.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010746670000000001 2023-01-15T14:31:33.116 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.116 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757539+0000", 2023-01-15T14:31:33.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099890899000000005 2023-01-15T14:31:33.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958683+0000", 2023-01-15T14:31:33.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20114332400000001 2023-01-15T14:31:33.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:33.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958746+0000", 2023-01-15T14:31:33.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3509000000000005e-05 2023-01-15T14:31:33.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:33.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958796+0000", 2023-01-15T14:31:33.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0176999999999997e-05 2023-01-15T14:31:33.123 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.124 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:33.124 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.125 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:358 3.a 3:52df5138:::smithi09025264-25:head [write 709262~697258] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:33.125 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:31:33.125 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 171.18722106199999, 2023-01-15T14:31:33.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50589529799999999, 2023-01-15T14:31:33.126 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:33.127 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:33.127 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:33.127 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:33.127 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:33.127 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 358 2023-01-15T14:31:33.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.127 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:33.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:33.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:31:33.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:33.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:31:33.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050780+0000", 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051369+0000", 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058926200000000001 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:33.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051390+0000", 2023-01-15T14:31:33.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1426000000000001e-05 2023-01-15T14:31:33.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:33.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051403+0000", 2023-01-15T14:31:33.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:31:33.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:33.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051520+0000", 2023-01-15T14:31:33.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117318 2023-01-15T14:31:33.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051785+0000", 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000265018 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052173+0000", 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038734799999999998 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052705+0000", 2023-01-15T14:31:33.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053209899999999996 2023-01-15T14:31:33.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:33.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052727+0000", 2023-01-15T14:31:33.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2362e-05 2023-01-15T14:31:33.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:33.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053205+0000", 2023-01-15T14:31:33.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047787200000000001 2023-01-15T14:31:33.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:33.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055711+0000", 2023-01-15T14:31:33.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0025054880000000002 2023-01-15T14:31:33.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.255549+0000", 2023-01-15T14:31:33.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199838285 2023-01-15T14:31:33.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556501+0000", 2023-01-15T14:31:33.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30095197499999998 2023-01-15T14:31:33.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:33.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556613+0000", 2023-01-15T14:31:33.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111799 2023-01-15T14:31:33.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:33.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556678+0000", 2023-01-15T14:31:33.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5220000000000002e-05 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:359 3.a 3:52df5138:::smithi09025264-25:head [write 2012635~572018] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 171.18657046800001, 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50546442499999999, 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:33.322 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:33.323 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:33.323 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:33.323 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 359 2023-01-15T14:31:33.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.323 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:33.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:33.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:31:33.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:33.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:31:33.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:33.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051432+0000", 2023-01-15T14:31:33.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:33.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051808+0000", 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000376074 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051834+0000", 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6018999999999999e-05 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051869+0000", 2023-01-15T14:31:33.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4839999999999998e-05 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052857+0000", 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00098761399999999994 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053071+0000", 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000214112 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:33.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053876+0000", 2023-01-15T14:31:33.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000804873 2023-01-15T14:31:33.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:33.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054460+0000", 2023-01-15T14:31:33.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058436100000000004 2023-01-15T14:31:33.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:33.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054486+0000", 2023-01-15T14:31:33.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069999999999999e-05 2023-01-15T14:31:33.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:33.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054773+0000", 2023-01-15T14:31:33.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028716700000000001 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055736+0000", 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00096295500000000004 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.355784+0000", 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30004825899999998 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556757+0000", 2023-01-15T14:31:33.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20097241199999999 2023-01-15T14:31:33.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:33.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556830+0000", 2023-01-15T14:31:33.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3633000000000005e-05 2023-01-15T14:31:33.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556898+0000", 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7212999999999995e-05 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:360 3.a 3:52df5138:::smithi09025264-25:head [write 3046388~62874] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 171.186117712, 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50519877000000002, 2023-01-15T14:31:33.331 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:33.332 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:33.332 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:33.332 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:33.520 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:33.521 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 360 2023-01-15T14:31:33.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.521 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:33.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:33.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:31:33.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:33.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:31:33.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:33.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051885+0000", 2023-01-15T14:31:33.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:33.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:33.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051935+0000", 2023-01-15T14:31:33.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9826e-05 2023-01-15T14:31:33.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:33.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051958+0000", 2023-01-15T14:31:33.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3003e-05 2023-01-15T14:31:33.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:33.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051965+0000", 2023-01-15T14:31:33.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6129999999999998e-06 2023-01-15T14:31:33.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:33.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054655+0000", 2023-01-15T14:31:33.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0026896860000000002 2023-01-15T14:31:33.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:33.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054869+0000", 2023-01-15T14:31:33.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021348600000000001 2023-01-15T14:31:33.525 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.525 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:33.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055198+0000", 2023-01-15T14:31:33.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032968199999999998 2023-01-15T14:31:33.525 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.525 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:33.526 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055605+0000", 2023-01-15T14:31:33.526 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040634200000000002 2023-01-15T14:31:33.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.526 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:33.526 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055622+0000", 2023-01-15T14:31:33.526 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7020999999999998e-05 2023-01-15T14:31:33.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.526 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:33.526 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055702+0000", 2023-01-15T14:31:33.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0835999999999997e-05 2023-01-15T14:31:33.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:33.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055754+0000", 2023-01-15T14:31:33.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1175999999999999e-05 2023-01-15T14:31:33.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.455996+0000", 2023-01-15T14:31:33.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40024198599999999 2023-01-15T14:31:33.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556952+0000", 2023-01-15T14:31:33.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10095644400000001 2023-01-15T14:31:33.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:33.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557019+0000", 2023-01-15T14:31:33.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6746000000000001e-05 2023-01-15T14:31:33.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.530 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:33.530 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557085+0000", 2023-01-15T14:31:33.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5885999999999999e-05 2023-01-15T14:31:33.530 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.530 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:33.530 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.531 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:361 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 3109262] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:33.531 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:31:33.531 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 171.186027806, 2023-01-15T14:31:33.531 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60469846000000005, 2023-01-15T14:31:33.531 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:33.531 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:33.531 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:33.531 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:33.531 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:33.531 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 361 2023-01-15T14:31:33.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:31:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:31:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051975+0000", 2023-01-15T14:31:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051980+0000", 2023-01-15T14:31:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2170000000000002e-06 2023-01-15T14:31:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051989+0000", 2023-01-15T14:31:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5299999999999996e-06 2023-01-15T14:31:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051995+0000", 2023-01-15T14:31:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1369999999999999e-06 2023-01-15T14:31:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055830+0000", 2023-01-15T14:31:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0038348549999999999 2023-01-15T14:31:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056101+0000", 2023-01-15T14:31:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027095800000000002 2023-01-15T14:31:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056560+0000", 2023-01-15T14:31:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045868599999999998 2023-01-15T14:31:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056954+0000", 2023-01-15T14:31:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039487600000000001 2023-01-15T14:31:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:33.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057004+0000", 2023-01-15T14:31:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9490000000000002e-05 2023-01-15T14:31:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057072+0000", 2023-01-15T14:31:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8502000000000002e-05 2023-01-15T14:31:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.158224+0000", 2023-01-15T14:31:33.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101151422 2023-01-15T14:31:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556201+0000", 2023-01-15T14:31:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39797714699999998 2023-01-15T14:31:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656421+0000", 2023-01-15T14:31:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100219549 2023-01-15T14:31:33.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:33.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656515+0000", 2023-01-15T14:31:33.701 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4048999999999993e-05 2023-01-15T14:31:33.703 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656674+0000", 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000159696 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:362 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:31:33.704 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 171.18600037600001, 2023-01-15T14:31:33.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60517504499999997, 2023-01-15T14:31:33.705 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:33.705 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:33.705 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:33.705 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:33.705 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:33.705 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 362 2023-01-15T14:31:33.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.706 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:33.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:33.706 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:31:33.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:33.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:31:33.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:33.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:31:33.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:33.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:33.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052007+0000", 2023-01-15T14:31:33.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.9759999999999997e-06 2023-01-15T14:31:33.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:33.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052016+0000", 2023-01-15T14:31:33.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4069999999999999e-06 2023-01-15T14:31:33.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:33.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052022+0000", 2023-01-15T14:31:33.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4670000000000001e-06 2023-01-15T14:31:33.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:33.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157159+0000", 2023-01-15T14:31:33.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.105136825 2023-01-15T14:31:33.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157325+0000", 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000165262 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656814+0000", 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49948963200000002 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656993+0000", 2023-01-15T14:31:33.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000178371 2023-01-15T14:31:33.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:33.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.657178+0000", 2023-01-15T14:31:33.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185825 2023-01-15T14:31:33.711 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.711 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:33.711 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.711 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:601 3.a 3:5ba385f1:::smithi09025264-44:head [setxattr _header (57) in=7b,truncate 3663893] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.29567774, 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404742724, 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 601 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:33.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:33.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942323+0000", 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942341+0000", 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7659000000000002e-05 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942365+0000", 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4256999999999998e-05 2023-01-15T14:31:33.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:33.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042495+0000", 2023-01-15T14:31:33.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013043200000001 2023-01-15T14:31:33.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:33.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.144809+0000", 2023-01-15T14:31:33.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102314184 2023-01-15T14:31:33.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145098+0000", 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000289036 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145363+0000", 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026449299999999998 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:33.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145657+0000", 2023-01-15T14:31:33.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029433600000000002 2023-01-15T14:31:33.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:33.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145678+0000", 2023-01-15T14:31:33.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0939000000000001e-05 2023-01-15T14:31:33.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:33.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145853+0000", 2023-01-15T14:31:33.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017511800000000001 2023-01-15T14:31:33.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:33.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146557+0000", 2023-01-15T14:31:33.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00070337899999999996 2023-01-15T14:31:33.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146845+0000", 2023-01-15T14:31:33.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028790599999999999 2023-01-15T14:31:33.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346819+0000", 2023-01-15T14:31:33.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199974661 2023-01-15T14:31:33.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:33.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346919+0000", 2023-01-15T14:31:33.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9234999999999994e-05 2023-01-15T14:31:33.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:33.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.347069+0000", 2023-01-15T14:31:33.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150128 2023-01-15T14:31:33.894 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.894 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:33.894 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.894 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1153 3.a 3:5919d83e:::smithi09025264-13:head [write 1566383~595717] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:31:33.894 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:31:33.894 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 105.44745268600001, 2023-01-15T14:31:33.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.38805106299999997, 2023-01-15T14:31:33.894 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:33.895 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:33.895 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:33.895 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:33.895 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:33.895 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1153 2023-01-15T14:31:33.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.895 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:33.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:33.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:31:33.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:33.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:31:33.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:33.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790531+0000", 2023-01-15T14:31:33.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999795 2023-01-15T14:31:33.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:33.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791189+0000", 2023-01-15T14:31:33.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065897000000000004 2023-01-15T14:31:33.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:33.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791215+0000", 2023-01-15T14:31:33.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5700000000000001e-05 2023-01-15T14:31:33.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:33.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791232+0000", 2023-01-15T14:31:33.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6546e-05 2023-01-15T14:31:33.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:33.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791392+0000", 2023-01-15T14:31:33.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016051699999999999 2023-01-15T14:31:33.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:33.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791408+0000", 2023-01-15T14:31:33.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5566999999999999e-05 2023-01-15T14:31:33.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:33.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.974825+0000", 2023-01-15T14:31:33.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18341675599999999 2023-01-15T14:31:33.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:33.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975000+0000", 2023-01-15T14:31:33.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017548299999999999 2023-01-15T14:31:33.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:33.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975255+0000", 2023-01-15T14:31:33.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254658 2023-01-15T14:31:33.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:33.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975914+0000", 2023-01-15T14:31:33.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065881800000000003 2023-01-15T14:31:33.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:33.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975931+0000", 2023-01-15T14:31:33.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7523000000000001e-05 2023-01-15T14:31:33.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:33.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976267+0000", 2023-01-15T14:31:33.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033545299999999999 2023-01-15T14:31:33.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978320+0000", 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020537110000000002 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978647+0000", 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032660700000000003 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:33.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178434+0000", 2023-01-15T14:31:33.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19978705699999999 2023-01-15T14:31:33.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:33.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178527+0000", 2023-01-15T14:31:33.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3523999999999994e-05 2023-01-15T14:31:33.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:33.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178602+0000", 2023-01-15T14:31:33.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4622000000000005e-05 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1154 3.a 3:5919d83e:::smithi09025264-13:head [write 2888822~102824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 105.44211135899999, 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48318778800000001, 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:33.906 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:33.907 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:33.907 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:33.907 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1154 2023-01-15T14:31:33.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.907 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:33.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:33.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:31:33.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:33.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:31:33.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:33.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:33.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795890+0000", 2023-01-15T14:31:33.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:33.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795967+0000", 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6304999999999995e-05 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796000+0000", 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3593000000000002e-05 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:33.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796019+0000", 2023-01-15T14:31:33.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8955999999999999e-05 2023-01-15T14:31:33.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:33.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796189+0000", 2023-01-15T14:31:33.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017032700000000001 2023-01-15T14:31:33.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:33.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796208+0000", 2023-01-15T14:31:33.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8389e-05 2023-01-15T14:31:33.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:33.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:33.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:33.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976335+0000", 2023-01-15T14:31:33.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18012763100000001 2023-01-15T14:31:33.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.070 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:34.070 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976487+0000", 2023-01-15T14:31:34.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015173000000000001 2023-01-15T14:31:34.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.071 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:34.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976763+0000", 2023-01-15T14:31:34.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027597400000000001 2023-01-15T14:31:34.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:34.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977085+0000", 2023-01-15T14:31:34.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032233400000000002 2023-01-15T14:31:34.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:34.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977103+0000", 2023-01-15T14:31:34.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7727000000000002e-05 2023-01-15T14:31:34.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:34.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977238+0000", 2023-01-15T14:31:34.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013525399999999999 2023-01-15T14:31:34.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978347+0000", 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011084350000000001 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.078223+0000", 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099875830999999998 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:34.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.278897+0000", 2023-01-15T14:31:34.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20067421599999999 2023-01-15T14:31:34.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:34.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279021+0000", 2023-01-15T14:31:34.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124354 2023-01-15T14:31:34.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:34.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279080+0000", 2023-01-15T14:31:34.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8811999999999999e-05 2023-01-15T14:31:34.096 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.096 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:34.096 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.096 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1155 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 2991646] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:31:34.096 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:31:34.096 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 105.44193833999999, 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48334445999999998, 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1155 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:34.097 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796064+0000", 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:34.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.099 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:34.099 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796074+0000", 2023-01-15T14:31:34.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:31:34.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:34.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796096+0000", 2023-01-15T14:31:34.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2422999999999999e-05 2023-01-15T14:31:34.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:34.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796109+0000", 2023-01-15T14:31:34.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:31:34.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:34.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796274+0000", 2023-01-15T14:31:34.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000164976 2023-01-15T14:31:34.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:34.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796283+0000", 2023-01-15T14:31:34.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9539999999999993e-06 2023-01-15T14:31:34.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:34.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977269+0000", 2023-01-15T14:31:34.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18098651700000001 2023-01-15T14:31:34.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:34.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977438+0000", 2023-01-15T14:31:34.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168913 2023-01-15T14:31:34.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:34.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977865+0000", 2023-01-15T14:31:34.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042707500000000001 2023-01-15T14:31:34.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:34.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978241+0000", 2023-01-15T14:31:34.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037593000000000001 2023-01-15T14:31:34.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:34.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978253+0000", 2023-01-15T14:31:34.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1817000000000001e-05 2023-01-15T14:31:34.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:34.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978288+0000", 2023-01-15T14:31:34.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4586999999999999e-05 2023-01-15T14:31:34.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:34.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978370+0000", 2023-01-15T14:31:34.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1712999999999995e-05 2023-01-15T14:31:34.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.268 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.270 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:34.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279124+0000", 2023-01-15T14:31:34.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30075418799999998 2023-01-15T14:31:34.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279208+0000", 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4500999999999996e-05 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279294+0000", 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5680999999999995e-05 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:34.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279410+0000", 2023-01-15T14:31:34.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115878 2023-01-15T14:31:34.272 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.272 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:34.272 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.272 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1156 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:31:34.272 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:31:34.272 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 105.441826815, 2023-01-15T14:31:34.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.58375825000000003, 2023-01-15T14:31:34.272 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1156 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:34.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796176+0000", 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:34.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796182+0000", 2023-01-15T14:31:34.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6799999999999998e-06 2023-01-15T14:31:34.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.281 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:34.281 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796200+0000", 2023-01-15T14:31:34.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8136999999999999e-05 2023-01-15T14:31:34.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.281 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:34.281 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796217+0000", 2023-01-15T14:31:34.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7036000000000001e-05 2023-01-15T14:31:34.282 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.282 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:34.282 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796337+0000", 2023-01-15T14:31:34.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120674 2023-01-15T14:31:34.282 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.282 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:34.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796345+0000", 2023-01-15T14:31:34.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4540000000000001e-06 2023-01-15T14:31:34.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:34.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978423+0000", 2023-01-15T14:31:34.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18207789599999999 2023-01-15T14:31:34.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:34.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978577+0000", 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000153805 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379596+0000", 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.401019348 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379754+0000", 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015840400000000001 2023-01-15T14:31:34.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379935+0000", 2023-01-15T14:31:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018081299999999999 2023-01-15T14:31:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1160 3.a 3:52df5138:::smithi09025264-25:head [write 2632598~375855] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:31:34.285 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 104.483020228, 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.398337832, 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1160 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:34.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:31:34.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:34.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:34.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:31:34.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:34.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:34.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754981+0000", 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755360+0000", 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037899899999999998 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755392+0000", 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1977000000000002e-05 2023-01-15T14:31:34.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755407+0000", 2023-01-15T14:31:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4446e-05 2023-01-15T14:31:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949294+0000", 2023-01-15T14:31:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.193887483 2023-01-15T14:31:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949478+0000", 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018400900000000001 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949904+0000", 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042552200000000001 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:34.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950531+0000", 2023-01-15T14:31:34.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062732399999999996 2023-01-15T14:31:34.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:34.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950552+0000", 2023-01-15T14:31:34.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1229999999999998e-05 2023-01-15T14:31:34.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:34.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951094+0000", 2023-01-15T14:31:34.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054169000000000005 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951256+0000", 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016176299999999999 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052665+0000", 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101409257 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:34.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153132+0000", 2023-01-15T14:31:34.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100466783 2023-01-15T14:31:34.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:34.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153223+0000", 2023-01-15T14:31:34.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.1415000000000005e-05 2023-01-15T14:31:34.294 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.294 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.294 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:34.294 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153321+0000", 2023-01-15T14:31:34.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8181000000000001e-05 2023-01-15T14:31:34.294 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.454 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:34.454 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1161 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (57) in=7b,truncate 3008453] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 104.48257664, 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49840109900000001, 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1161 2023-01-15T14:31:34.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.456 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755426+0000", 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755433+0000", 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.793e-06 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755448+0000", 2023-01-15T14:31:34.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5047e-05 2023-01-15T14:31:34.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:34.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755455+0000", 2023-01-15T14:31:34.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3590000000000002e-06 2023-01-15T14:31:34.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:34.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951466+0000", 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.196011038 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951611+0000", 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000145184 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951921+0000", 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030989100000000002 2023-01-15T14:31:34.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:34.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952187+0000", 2023-01-15T14:31:34.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026623100000000002 2023-01-15T14:31:34.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:34.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952207+0000", 2023-01-15T14:31:34.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9510000000000001e-05 2023-01-15T14:31:34.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:34.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952271+0000", 2023-01-15T14:31:34.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4511999999999995e-05 2023-01-15T14:31:34.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:34.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052716+0000", 2023-01-15T14:31:34.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10044497300000001 2023-01-15T14:31:34.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253531+0000", 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20081490699999999 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253635+0000", 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000103838 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253727+0000", 2023-01-15T14:31:34.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2298000000000003e-05 2023-01-15T14:31:34.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:34.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253828+0000", 2023-01-15T14:31:34.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000100655 2023-01-15T14:31:34.463 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.463 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:34.463 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.463 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1162 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 104.482537245, 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.498775839, 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1162 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:34.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:34.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755470+0000", 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0570000000000003e-06 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755484+0000", 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4229999999999999e-05 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755492+0000", 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6070000000000002e-06 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.152842+0000", 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39734988799999998 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:34.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153014+0000", 2023-01-15T14:31:34.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017236000000000001 2023-01-15T14:31:34.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:34.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253921+0000", 2023-01-15T14:31:34.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100907497 2023-01-15T14:31:34.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:34.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254060+0000", 2023-01-15T14:31:34.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013815400000000001 2023-01-15T14:31:34.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:34.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:34.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:34.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254242+0000", 2023-01-15T14:31:34.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000182726 2023-01-15T14:31:34.470 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.470 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:34.470 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.470 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:34.470 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:34.470 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:34.471 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:31:34.471 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:31.322+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:34.472 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:31:34.473 INFO:tasks.rados.rados.0.smithi090.stdout:810: expect (ObjNum 275 snap 70 seq_num 275) 2023-01-15T14:31:34.473 INFO:tasks.rados.rados.0.smithi090.stdout:810: done (1 left) 2023-01-15T14:31:34.473 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:34.473 INFO:tasks.rados.rados.0.smithi090.stdout:809: finishing copy_from to smithi09025264-11 2023-01-15T14:31:34.473 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 199 (ObjNum 262 snap 61 seq_num 262) dirty exists 2023-01-15T14:31:34.473 INFO:tasks.rados.rados.0.smithi090.stdout:809: finishing copy_from racing read to smithi09025264-11 2023-01-15T14:31:34.473 INFO:tasks.rados.rados.0.smithi090.stdout:809: done (0 left) 2023-01-15T14:31:34.473 INFO:tasks.rados.rados.0.smithi090.stdout:811: done (0 left) 2023-01-15T14:31:34.473 INFO:tasks.rados.rados.0.smithi090.stdout:812: read oid 38 snap -1 2023-01-15T14:31:34.473 INFO:tasks.rados.rados.0.smithi090.stdout:812: expect deleted 2023-01-15T14:31:34.474 INFO:tasks.rados.rados.0.smithi090.stdout:813: snap_remove snap 80 2023-01-15T14:31:34.474 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:31:34.476 INFO:tasks.rados.rados.0.smithi090.stdout:813: done (1 left) 2023-01-15T14:31:34.476 INFO:tasks.rados.rados.0.smithi090.stdout:814: read oid 24 snap -1 2023-01-15T14:31:34.476 INFO:tasks.rados.rados.0.smithi090.stdout:814: expect (ObjNum 266 snap 66 seq_num 266) 2023-01-15T14:31:34.476 INFO:tasks.rados.rados.0.smithi090.stdout:815: snap_create 2023-01-15T14:31:34.476 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:34.476 INFO:tasks.rados.rados.0.smithi090.stdout:812: done (1 left) 2023-01-15T14:31:34.476 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:34.476 INFO:tasks.rados.rados.0.smithi090.stdout:814: expect (ObjNum 266 snap 66 seq_num 266) 2023-01-15T14:31:34.476 INFO:tasks.rados.rados.0.smithi090.stdout:814: done (0 left) 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:815: done (0 left) 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:816: rollback oid 25 current snap is 83 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 25 to 76 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:817: rollback oid 19 current snap is 83 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 19 to 72 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:818: read oid 39 snap 76 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:818: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:819: rollback oid 37 current snap is 83 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 37 to 62 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:820: read oid 16 snap 75 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:820: expect (ObjNum 281 snap 74 seq_num 281) 2023-01-15T14:31:34.477 INFO:tasks.rados.rados.0.smithi090.stdout:821: snap_remove snap 78 2023-01-15T14:31:34.478 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:34.467+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:34.573 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:34.570+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:34.645 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:34.646 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:34.646 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:34.647 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:31:34.647 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:31:34.647 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:34.648 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:31:34.661 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:34.680 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:34.835 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:34.686+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:34.835 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:34.830+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:34.836 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:34.836 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:34.836 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:34.836 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:31:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:31:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:31:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:31:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:31:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:31:34.837 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:31:34.838 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:31:34.838 INFO:tasks.rados.rados.0.smithi090.stdout:821: done (5 left) 2023-01-15T14:31:34.838 INFO:tasks.rados.rados.0.smithi090.stdout:822: write oid 40 current snap is 83 2023-01-15T14:31:34.838 INFO:tasks.rados.rados.0.smithi090.stdout:822: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2023-01-15T14:31:34.838 INFO:tasks.rados.rados.0.smithi090.stdout:822: writing smithi09025264-40 from 511012 to 1060169 tid 1 2023-01-15T14:31:34.844 INFO:tasks.rados.rados.0.smithi090.stdout:822: writing smithi09025264-40 from 1775251 to 2326011 tid 2 2023-01-15T14:31:34.850 INFO:tasks.rados.rados.0.smithi090.stdout:822: writing smithi09025264-40 from 3095195 to 3311012 tid 3 2023-01-15T14:31:35.009 INFO:tasks.rados.rados.0.smithi090.stdout:823: snap_create 2023-01-15T14:31:35.009 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:31:35.009 INFO:tasks.rados.rados.0.smithi090.stdout:818: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:31:35.011 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:35.013 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:35.013 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:31:35.014 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:35.007+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:35.050 INFO:tasks.rados.rados.0.smithi090.stdout:816: finishing rollback tid 0 to smithi09025264-25 2023-01-15T14:31:35.196 INFO:tasks.rados.rados.0.smithi090.stdout:816: finishing rollback tid 1 to smithi09025264-25 2023-01-15T14:31:35.196 INFO:tasks.rados.rados.0.smithi090.stdout:817: finishing rollback tid 0 to smithi09025264-19 2023-01-15T14:31:35.196 INFO:tasks.rados.rados.0.smithi090.stdout:817: finishing rollback tid 1 to smithi09025264-19 2023-01-15T14:31:35.196 INFO:tasks.rados.rados.0.smithi090.stdout:816: finishing rollback tid 2 to smithi09025264-25 2023-01-15T14:31:35.196 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 224 (ObjNum 254 snap 60 seq_num 254) dirty exists 2023-01-15T14:31:35.197 INFO:tasks.rados.rados.0.smithi090.stdout:817: finishing rollback tid 2 to smithi09025264-19 2023-01-15T14:31:35.197 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 169 (ObjNum 261 snap 60 seq_num 261) dirty exists 2023-01-15T14:31:35.197 INFO:tasks.rados.rados.0.smithi090.stdout:822: finishing write tid 1 to smithi09025264-40 2023-01-15T14:31:35.197 INFO:tasks.rados.rados.0.smithi090.stdout:820: expect (ObjNum 281 snap 74 seq_num 281) 2023-01-15T14:31:35.198 INFO:tasks.rados.rados.0.smithi090.stdout:816: done (5 left) 2023-01-15T14:31:35.198 INFO:tasks.rados.rados.0.smithi090.stdout:817: done (4 left) 2023-01-15T14:31:35.198 INFO:tasks.rados.rados.0.smithi090.stdout:818: done (3 left) 2023-01-15T14:31:35.198 INFO:tasks.rados.rados.0.smithi090.stdout:820: done (2 left) 2023-01-15T14:31:35.198 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:35.199 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:35.199 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:35.199 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:35.194+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:35.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:31:35.237 INFO:tasks.rados.rados.0.smithi090.stdout:822: finishing write tid 2 to smithi09025264-40 2023-01-15T14:31:35.606 INFO:tasks.rados.rados.0.smithi090.stdout:822: finishing write tid 3 to smithi09025264-40 2023-01-15T14:31:35.607 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:35.607 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:35.608 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:31:35.609 INFO:tasks.rados.rados.0.smithi090.stdout:822: finishing write tid 4 to smithi09025264-40 2023-01-15T14:31:35.609 INFO:tasks.rados.rados.0.smithi090.stdout:819: finishing rollback tid 1 to smithi09025264-37 2023-01-15T14:31:35.609 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 0 (ObjNum 33 snap 0 seq_num 3192395872) dirty dne 2023-01-15T14:31:35.611 INFO:tasks.rados.rados.0.smithi090.stdout:819: done (1 left) 2023-01-15T14:31:35.611 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:35.611 INFO:tasks.rados.rados.0.smithi090.stdout:822: finishing write tid 5 to smithi09025264-40 2023-01-15T14:31:35.612 INFO:tasks.rados.rados.0.smithi090.stdout:822: oid 40 updating version 0 to 205 2023-01-15T14:31:35.612 INFO:tasks.rados.rados.0.smithi090.stdout:822: oid 40 updating version 205 to 206 2023-01-15T14:31:35.612 INFO:tasks.rados.rados.0.smithi090.stdout:822: oid 40 updating version 206 to 207 2023-01-15T14:31:35.612 INFO:tasks.rados.rados.0.smithi090.stdout:822: oid 40 version 207 is already newer than 204 2023-01-15T14:31:35.612 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 207 (ObjNum 316 snap 83 seq_num 316) dirty exists 2023-01-15T14:31:35.612 INFO:tasks.rados.rados.0.smithi090.stdout:822: left oid 40 (ObjNum 316 snap 83 seq_num 316) 2023-01-15T14:31:35.612 INFO:tasks.rados.rados.0.smithi090.stdout:822: done (0 left) 2023-01-15T14:31:35.613 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:35.602+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:35.707 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:35.702+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:35.809 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:35.809 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:35.810 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:31:35.810 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:35.803+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:35.859 INFO:tasks.rados.rados.0.smithi090.stdout:823: done (0 left) 2023-01-15T14:31:35.988 INFO:tasks.rados.rados.0.smithi090.stdout:824: rollback oid 15 current snap is 84 2023-01-15T14:31:35.989 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 15 to 83 2023-01-15T14:31:35.989 INFO:tasks.rados.rados.0.smithi090.stdout:825: write oid 31 current snap is 84 2023-01-15T14:31:35.989 INFO:tasks.rados.rados.0.smithi090.stdout:825: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2023-01-15T14:31:35.992 INFO:tasks.rados.rados.0.smithi090.stdout:825: writing smithi09025264-31 from 527819 to 1152225 tid 1 2023-01-15T14:31:35.992 INFO:tasks.rados.rados.0.smithi090.stdout:825: writing smithi09025264-31 from 1833733 to 2244504 tid 2 2023-01-15T14:31:35.992 INFO:tasks.rados.rados.0.smithi090.stdout:825: writing smithi09025264-31 from 2722618 to 3324199 tid 3 2023-01-15T14:31:35.992 INFO:tasks.rados.rados.0.smithi090.stdout:826: write (excl) oid 10 current snap is 84 2023-01-15T14:31:35.992 INFO:tasks.rados.rados.0.smithi090.stdout:824: finishing rollback tid 1 to smithi09025264-15 2023-01-15T14:31:35.992 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 0 (ObjNum 289 snap 0 seq_num 3195795104) dirty dne 2023-01-15T14:31:35.992 INFO:tasks.rados.rados.0.smithi090.stdout:826: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2023-01-15T14:31:35.993 INFO:tasks.rados.rados.0.smithi090.stdout:826: writing smithi09025264-10 from 544626 to 1244281 tid 1 2023-01-15T14:31:35.993 INFO:tasks.rados.rados.0.smithi090.stdout:826: writing smithi09025264-10 from 1892215 to 2446644 tid 2 2023-01-15T14:31:35.993 INFO:tasks.rados.rados.0.smithi090.stdout:826: writing smithi09025264-10 from 3150041 to 3344626 tid 3 2023-01-15T14:31:35.993 INFO:tasks.rados.rados.0.smithi090.stdout:824: done (2 left) 2023-01-15T14:31:35.993 INFO:tasks.rados.rados.0.smithi090.stdout:827: rollback oid 13 current snap is 84 2023-01-15T14:31:35.993 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 13 to 71 2023-01-15T14:31:35.993 INFO:tasks.rados.rados.0.smithi090.stdout:828: read oid 8 snap -1 2023-01-15T14:31:35.993 INFO:tasks.rados.rados.0.smithi090.stdout:828: expect (ObjNum 287 snap 74 seq_num 287) 2023-01-15T14:31:35.993 INFO:tasks.rados.rados.0.smithi090.stdout:829: read oid 25 snap 82 2023-01-15T14:31:35.993 INFO:tasks.rados.rados.0.smithi090.stdout:829: expect (ObjNum 254 snap 60 seq_num 254) 2023-01-15T14:31:35.994 INFO:tasks.rados.rados.0.smithi090.stdout:830: delete oid 29 current snap is 84 2023-01-15T14:31:35.994 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:35.994 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:35.995 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:31:35.995 INFO:tasks.rados.rados.0.smithi090.stdout:828: expect (ObjNum 287 snap 74 seq_num 287) 2023-01-15T14:31:35.996 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:35.987+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:36.092 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:36.086+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:36.206 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:31:36.206 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:31:36.207 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-01-15T14:31:36.211 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.000000 2023-01-15T14:31:36.211 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 0 2023-01-15T14:31:36.212 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:36.213 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:36.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:31:36.214 INFO:tasks.rados.rados.0.smithi090.stdout:828: done (5 left) 2023-01-15T14:31:36.214 INFO:tasks.rados.rados.0.smithi090.stdout:830: done (4 left) 2023-01-15T14:31:36.214 INFO:tasks.rados.rados.0.smithi090.stdout:831: read oid 28 snap -1 2023-01-15T14:31:36.214 INFO:tasks.rados.rados.0.smithi090.stdout:831: expect (ObjNum 311 snap 79 seq_num 311) 2023-01-15T14:31:36.214 INFO:tasks.rados.rados.0.smithi090.stdout:832: snap_remove snap 75 2023-01-15T14:31:36.214 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:36.207+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:36.314 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:36.310+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:36.421 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:36.421 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:36.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:31:36.422 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:36.415+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:36.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:31:36.523 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:36.519+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:36.548 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:36.621 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:36.623 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:31:36.624 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:36.622+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:36.725 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:36.722+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:36.729 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:36.729 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:36.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:31:36.814 INFO:tasks.rados.rados.0.smithi090.stdout:832: done (5 left) 2023-01-15T14:31:36.817 INFO:tasks.rados.rados.0.smithi090.stdout:833: snap_create 2023-01-15T14:31:36.817 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:31:36.817 INFO:tasks.rados.rados.0.smithi090.stdout:831: expect (ObjNum 311 snap 79 seq_num 311) 2023-01-15T14:31:36.824 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.2 primary-affinity to 0 (802) 2023-01-15T14:31:36.825 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:36.819+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:36.829 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:31:36.838 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:31:36.859 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:36.859 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:36.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:31:36.926 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:36.922+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:36.982 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:37.002 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:37.017 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:37.017 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:37.024 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:37.027 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:37.022+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:37.029 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:31:37.140 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:37.140 INFO:tasks.rados.rados.0.smithi090.stdout:829: expect (ObjNum 254 snap 60 seq_num 254) 2023-01-15T14:31:37.141 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:37.141 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:37.141 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:37.141 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:37.135+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:37.151 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:37.153 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:31:37.153 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:31:37.153 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:31:37.153 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:31:37.153 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:31:37.153 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:31:37.153 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:31:37.154 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:31:37.171 INFO:tasks.rados.rados.0.smithi090.stdout:827: finishing rollback tid 0 to smithi09025264-13 2023-01-15T14:31:37.300 INFO:tasks.rados.rados.0.smithi090.stdout:827: finishing rollback tid 1 to smithi09025264-13 2023-01-15T14:31:37.300 INFO:tasks.rados.rados.0.smithi090.stdout:827: finishing rollback tid 2 to smithi09025264-13 2023-01-15T14:31:37.300 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 231 (ObjNum 216 snap 49 seq_num 216) dirty exists 2023-01-15T14:31:37.300 INFO:tasks.rados.rados.0.smithi090.stdout:825: finishing write tid 1 to smithi09025264-31 2023-01-15T14:31:37.300 INFO:tasks.rados.rados.0.smithi090.stdout:825: finishing write tid 2 to smithi09025264-31 2023-01-15T14:31:37.300 INFO:tasks.rados.rados.0.smithi090.stdout:825: finishing write tid 3 to smithi09025264-31 2023-01-15T14:31:37.302 INFO:tasks.rados.rados.0.smithi090.stdout:825: finishing write tid 4 to smithi09025264-31 2023-01-15T14:31:37.302 INFO:tasks.rados.rados.0.smithi090.stdout:825: finishing write tid 5 to smithi09025264-31 2023-01-15T14:31:37.302 INFO:tasks.rados.rados.0.smithi090.stdout:825: oid 31 updating version 0 to 256 2023-01-15T14:31:37.302 INFO:tasks.rados.rados.0.smithi090.stdout:825: oid 31 updating version 256 to 258 2023-01-15T14:31:37.303 INFO:tasks.rados.rados.0.smithi090.stdout:825: oid 31 updating version 258 to 259 2023-01-15T14:31:37.303 INFO:tasks.rados.rados.0.smithi090.stdout:825: oid 31 version 259 is already newer than 257 2023-01-15T14:31:37.303 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 259 (ObjNum 317 snap 84 seq_num 317) dirty exists 2023-01-15T14:31:37.303 INFO:tasks.rados.rados.0.smithi090.stdout:825: left oid 31 (ObjNum 317 snap 84 seq_num 317) 2023-01-15T14:31:37.303 INFO:tasks.rados.rados.0.smithi090.stdout:826: finishing write tid 1 to smithi09025264-10 2023-01-15T14:31:37.303 INFO:tasks.rados.rados.0.smithi090.stdout:825: done (4 left) 2023-01-15T14:31:37.303 INFO:tasks.rados.rados.0.smithi090.stdout:827: done (3 left) 2023-01-15T14:31:37.304 INFO:tasks.rados.rados.0.smithi090.stdout:829: done (2 left) 2023-01-15T14:31:37.304 INFO:tasks.rados.rados.0.smithi090.stdout:831: done (1 left) 2023-01-15T14:31:37.304 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:37.305 INFO:tasks.rados.rados.0.smithi090.stdout:826: finishing write tid 2 to smithi09025264-10 2023-01-15T14:31:37.306 INFO:tasks.rados.rados.0.smithi090.stdout:826: finishing write tid 3 to smithi09025264-10 2023-01-15T14:31:37.306 INFO:tasks.rados.rados.0.smithi090.stdout:826: finishing write tid 4 to smithi09025264-10 2023-01-15T14:31:37.307 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.3.0:459 3.12 e210/207)", 2023-01-15T14:31:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:37.206923+0000", 2023-01-15T14:31:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.035567077000000002, 2023-01-15T14:31:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.035598280000000003, 2023-01-15T14:31:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:37.206923+0000", 2023-01-15T14:31:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:37.206923+0000", 2023-01-15T14:31:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:37.206922+0000", 2023-01-15T14:31:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:37.206926+0000", 2023-01-15T14:31:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3759999999999999e-06 2023-01-15T14:31:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:37.206937+0000", 2023-01-15T14:31:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1151e-05 2023-01-15T14:31:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:37.206942+0000", 2023-01-15T14:31:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8250000000000004e-06 2023-01-15T14:31:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:31:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:31:37.315 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:37.316 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:31:37.316 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:37.302+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:37.325 INFO:tasks.rados.rados.0.smithi090.stdout:826: finishing write tid 5 to smithi09025264-10 2023-01-15T14:31:37.325 INFO:tasks.rados.rados.0.smithi090.stdout:826: oid 10 updating version 0 to 262 2023-01-15T14:31:37.325 INFO:tasks.rados.rados.0.smithi090.stdout:826: oid 10 updating version 262 to 264 2023-01-15T14:31:37.325 INFO:tasks.rados.rados.0.smithi090.stdout:826: oid 10 version 264 is already newer than 263 2023-01-15T14:31:37.325 INFO:tasks.rados.rados.0.smithi090.stdout:826: oid 10 version 264 is already newer than 261 2023-01-15T14:31:37.326 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 264 (ObjNum 318 snap 84 seq_num 318) dirty exists 2023-01-15T14:31:37.326 INFO:tasks.rados.rados.0.smithi090.stdout:826: left oid 10 (ObjNum 318 snap 84 seq_num 318) 2023-01-15T14:31:37.326 INFO:tasks.rados.rados.0.smithi090.stdout:826: done (0 left) 2023-01-15T14:31:37.406 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:37.403+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:37.474 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:37.475 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:37.475 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:37.475 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:37.475 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:37.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:31:37.508 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:37.503+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:37.592 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:37.592 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:37.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:37.592 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:37.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.592 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:166 3.8 3:1e54a7f5:::smithi09025264-42:head [write 1699644~784332] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:37.592 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:37.592 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 190.47186864299999, 2023-01-15T14:31:37.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62664632300000001, 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 166 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:37.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114715+0000", 2023-01-15T14:31:37.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.114713+0000", 2023-01-15T14:31:37.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:37.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115178+0000", 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046493199999999999 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115208+0000", 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0689999999999999e-05 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115221+0000", 2023-01-15T14:31:37.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3013000000000001e-05 2023-01-15T14:31:37.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115334+0000", 2023-01-15T14:31:37.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112403 2023-01-15T14:31:37.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:37.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115505+0000", 2023-01-15T14:31:37.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170915 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333496+0000", 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217991667 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333606+0000", 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109967 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:37.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.333796+0000", 2023-01-15T14:31:37.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018928099999999999 2023-01-15T14:31:37.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:37.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334289+0000", 2023-01-15T14:31:37.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049322600000000004 2023-01-15T14:31:37.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334339+0000", 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0439999999999998e-05 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334990+0000", 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065082300000000003 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335100+0000", 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109357 2023-01-15T14:31:37.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.640854+0000", 2023-01-15T14:31:37.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30575451300000001 2023-01-15T14:31:37.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741190+0000", 2023-01-15T14:31:37.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100335568 2023-01-15T14:31:37.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741263+0000", 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3275999999999994e-05 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741361+0000", 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8435000000000002e-05 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:167 3.8 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 190.47130395400001, 2023-01-15T14:31:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62638390099999997, 2023-01-15T14:31:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.603 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 167 2023-01-15T14:31:37.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.603 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:37.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115280+0000", 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115279+0000", 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115289+0000", 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0509999999999999e-05 2023-01-15T14:31:37.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115306+0000", 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7260000000000001e-05 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115337+0000", 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0664000000000003e-05 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115579+0000", 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024194400000000001 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115744+0000", 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016493700000000001 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334392+0000", 2023-01-15T14:31:37.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21864830299999999 2023-01-15T14:31:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334453+0000", 2023-01-15T14:31:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1047e-05 2023-01-15T14:31:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334601+0000", 2023-01-15T14:31:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014824700000000001 2023-01-15T14:31:37.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334812+0000", 2023-01-15T14:31:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210208 2023-01-15T14:31:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334996+0000", 2023-01-15T14:31:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000183878 2023-01-15T14:31:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335126+0000", 2023-01-15T14:31:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013001399999999999 2023-01-15T14:31:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335160+0000", 2023-01-15T14:31:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4876999999999998e-05 2023-01-15T14:31:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741044+0000", 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40588326000000002 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741453+0000", 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040963099999999998 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741548+0000", 2023-01-15T14:31:37.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4961000000000001e-05 2023-01-15T14:31:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.741664+0000", 2023-01-15T14:31:37.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115333 2023-01-15T14:31:37.632 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 190.47121184, 2023-01-15T14:31:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:31:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:37.633 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.634 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:31:37.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.634 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:37.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:31:37.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:31:37.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:37.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:31:37.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:31:37.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:31:37.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:31:37.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:37.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:31:37.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:31:37.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:31:37.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:31:37.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:37.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:31:37.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:31:37.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:31:37.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:31:37.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:37.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:31:37.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:31:37.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:31:37.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:31:37.640 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.640 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.640 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.640 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:37.640 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:37.640 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 190.45821976100001, 2023-01-15T14:31:37.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:31:37.641 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.641 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:37.641 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.641 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.641 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.641 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:31:37.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.641 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:31:37.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:37.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:31:37.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:31:37.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:31:37.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:31:37.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:31:37.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:31:37.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:31:37.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:31:37.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:37.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:31:37.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:31:37.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:31:37.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:31:37.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:37.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:31:37.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:31:37.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:31:37.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:31:37.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:31:37.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:37.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:31:37.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:31:37.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:37.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:31:37.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:31:37.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:37.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:31:37.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:31:37.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:31:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:31:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:31:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:31:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:31:37.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:31:37.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.651 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.651 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:37.651 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:37.651 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 190.457816425, 2023-01-15T14:31:37.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:31:37.651 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:31:37.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:31:37.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:37.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:31:37.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:31:37.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:31:37.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:37.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:31:37.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:31:37.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:31:37.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:31:37.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:31:37.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:37.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:31:37.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:31:37.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:37.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:31:37.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:31:37.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:37.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:31:37.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:31:37.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:37.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:31:37.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.660 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 190.357592805, 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.661 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:31:37.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:37.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:31:37.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:31:37.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:31:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:31:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:31:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:31:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:31:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:31:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:31:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:31:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:31:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:31:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:194 3.8 3:1eaad226:::smithi09025264-49:head [write 1709018~655350] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 190.28519156199999, 2023-01-15T14:31:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.64372577200000003, 2023-01-15T14:31:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 194 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301392+0000", 2023-01-15T14:31:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301390+0000", 2023-01-15T14:31:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.301991+0000", 2023-01-15T14:31:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060122599999999995 2023-01-15T14:31:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.302026+0000", 2023-01-15T14:31:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5040000000000003e-05 2023-01-15T14:31:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402147+0000", 2023-01-15T14:31:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100121084 2023-01-15T14:31:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402219+0000", 2023-01-15T14:31:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2045999999999994e-05 2023-01-15T14:31:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.402315+0000", 2023-01-15T14:31:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5561000000000002e-05 2023-01-15T14:31:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842189+0000", 2023-01-15T14:31:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.439874129 2023-01-15T14:31:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842296+0000", 2023-01-15T14:31:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107021 2023-01-15T14:31:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:31:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842486+0000", 2023-01-15T14:31:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018965799999999999 2023-01-15T14:31:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842839+0000", 2023-01-15T14:31:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035330700000000002 2023-01-15T14:31:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.842859+0000", 2023-01-15T14:31:37.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0347e-05 2023-01-15T14:31:37.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:37.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843212+0000", 2023-01-15T14:31:37.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352534 2023-01-15T14:31:37.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:37.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.843835+0000", 2023-01-15T14:31:37.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062338599999999997 2023-01-15T14:31:37.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944861+0000", 2023-01-15T14:31:37.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10102596899999999 2023-01-15T14:31:37.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.944908+0000", 2023-01-15T14:31:37.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6773000000000001e-05 2023-01-15T14:31:37.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:37.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945039+0000", 2023-01-15T14:31:37.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000130684 2023-01-15T14:31:37.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.945118+0000", 2023-01-15T14:31:37.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9194000000000001e-05 2023-01-15T14:31:37.678 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.679 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.679 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.679 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:705 3.2 3:47dfe935:::smithi09025264-29:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e63)", 2023-01-15T14:31:37.679 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:37.679 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 148.47599965399999, 2023-01-15T14:31:37.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61170208599999998, 2023-01-15T14:31:37.679 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.680 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:37.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.680 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.680 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 705 2023-01-15T14:31:37.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.680 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:37.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110584+0000", 2023-01-15T14:31:37.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110583+0000", 2023-01-15T14:31:37.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:37.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110587+0000", 2023-01-15T14:31:37.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9689999999999996e-06 2023-01-15T14:31:37.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110601+0000", 2023-01-15T14:31:37.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.393e-05 2023-01-15T14:31:37.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.110611+0000", 2023-01-15T14:31:37.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3999999999999998e-06 2023-01-15T14:31:37.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319174+0000", 2023-01-15T14:31:37.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20856376800000001 2023-01-15T14:31:37.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:37.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.319367+0000", 2023-01-15T14:31:37.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192361 2023-01-15T14:31:37.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.721941+0000", 2023-01-15T14:31:37.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40257461300000003 2023-01-15T14:31:37.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:37.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722108+0000", 2023-01-15T14:31:37.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000167054 2023-01-15T14:31:37.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:09.722286+0000", 2023-01-15T14:31:37.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017772099999999999 2023-01-15T14:31:37.685 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.685 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.685 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.686 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:37.686 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:37.686 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 99.664277580000004, 2023-01-15T14:31:37.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:31:37.686 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.686 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:37.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.687 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:31:37.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.687 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:37.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:31:37.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:31:37.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:37.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:31:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:31:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:31:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:31:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:31:37.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:31:37.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:31:37.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:31:37.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:37.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:31:37.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:31:37.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:31:37.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:31:37.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:31:37.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:31:37.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.691 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.692 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:37.692 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:37.692 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 99.657816225999994, 2023-01-15T14:31:37.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:31:37.692 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.692 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:37.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.693 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:31:37.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.693 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:37.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:31:37.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:31:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:31:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:31:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:31:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:31:37.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:31:37.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:31:37.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:37.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:31:37.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:31:37.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 99.557394301000002, 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:31:37.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.699 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:37.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:31:37.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:31:37.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:37.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:31:37.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:31:37.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:31:37.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:31:37.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:31:37.843 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.843 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.843 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.844 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:37.844 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:37.844 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 99.557056814999996, 2023-01-15T14:31:37.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:31:37.844 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.844 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:37.844 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.844 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.844 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.844 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:31:37.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:37.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:31:37.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:31:37.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:31:37.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:31:37.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:31:37.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:31:37.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:31:37.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:31:37.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:37.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:31:37.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:31:37.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:31:37.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:31:37.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:31:37.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:31:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 99.557000919000004, 2023-01-15T14:31:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:31:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.850 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:37.851 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.851 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.851 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.851 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:31:37.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.851 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:37.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:31:37.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:31:37.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:37.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:31:37.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:31:37.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:31:37.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:31:37.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:31:37.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:31:37.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:31:37.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:31:37.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:31:37.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:31:37.855 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.856 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.856 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.856 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1314 3.18 3.afe52a78 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:37.856 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:37.856 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.226705788000004, 2023-01-15T14:31:37.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62560938700000002, 2023-01-15T14:31:37.856 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.856 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:37.856 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1314 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359878+0000", 2023-01-15T14:31:37.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359877+0000", 2023-01-15T14:31:37.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:37.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359882+0000", 2023-01-15T14:31:37.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0050000000000004e-06 2023-01-15T14:31:37.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359893+0000", 2023-01-15T14:31:37.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0676000000000001e-05 2023-01-15T14:31:37.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.359904+0000", 2023-01-15T14:31:37.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0788000000000001e-05 2023-01-15T14:31:37.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460094+0000", 2023-01-15T14:31:37.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10019024999999999 2023-01-15T14:31:37.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:37.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.460106+0000", 2023-01-15T14:31:37.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2354999999999999e-05 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985477+0000", 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52537039100000005 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985487+0000", 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.861 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.862 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.862 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1335 3.8 3:17d5d741:::smithi09025264-5:head [write 3252761~175867] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:37.862 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:37.862 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.654686526999996, 2023-01-15T14:31:37.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66052243499999996, 2023-01-15T14:31:37.862 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.862 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:37.862 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.862 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.863 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.863 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1335 2023-01-15T14:31:37.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.863 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:37.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931897+0000", 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931896+0000", 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931983+0000", 2023-01-15T14:31:37.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6470000000000004e-05 2023-01-15T14:31:37.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.931995+0000", 2023-01-15T14:31:37.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2075e-05 2023-01-15T14:31:37.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932003+0000", 2023-01-15T14:31:37.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5059999999999995e-06 2023-01-15T14:31:37.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932346+0000", 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034280299999999998 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932357+0000", 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.13e-05 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189006+0000", 2023-01-15T14:31:37.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.25664859899999998 2023-01-15T14:31:37.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:37.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189336+0000", 2023-01-15T14:31:37.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033030899999999998 2023-01-15T14:31:37.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:37.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189481+0000", 2023-01-15T14:31:37.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014484400000000001 2023-01-15T14:31:37.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:37.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189768+0000", 2023-01-15T14:31:37.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028704300000000001 2023-01-15T14:31:37.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:37.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189787+0000", 2023-01-15T14:31:37.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9343000000000001e-05 2023-01-15T14:31:37.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:37.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191361+0000", 2023-01-15T14:31:37.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015734029999999999 2023-01-15T14:31:37.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:37.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191469+0000", 2023-01-15T14:31:37.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010859899999999999 2023-01-15T14:31:37.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391274+0000", 2023-01-15T14:31:37.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19980450899999999 2023-01-15T14:31:37.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592067+0000", 2023-01-15T14:31:37.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20079332 2023-01-15T14:31:37.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592190+0000", 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012291400000000001 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592420+0000", 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000229371 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.872 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.873 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1336 3.8 3:17d5d741:::smithi09025264-5:head [setxattr _header (56) in=7b,truncate 3428628] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:37.873 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:37.873 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.654569437999996, 2023-01-15T14:31:37.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66113910799999998, 2023-01-15T14:31:37.873 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.873 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:37.873 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.873 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.873 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1336 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932014+0000", 2023-01-15T14:31:37.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932013+0000", 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932019+0000", 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5380000000000002e-06 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932027+0000", 2023-01-15T14:31:37.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5150000000000006e-06 2023-01-15T14:31:37.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932034+0000", 2023-01-15T14:31:37.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2829999999999998e-06 2023-01-15T14:31:37.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032497+0000", 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100463001 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.032518+0000", 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1359999999999999e-05 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189826+0000", 2023-01-15T14:31:37.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15730834299999999 2023-01-15T14:31:37.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:37.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.189951+0000", 2023-01-15T14:31:37.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124583 2023-01-15T14:31:37.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:37.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190093+0000", 2023-01-15T14:31:37.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014221399999999999 2023-01-15T14:31:37.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190253+0000", 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015953000000000001 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191365+0000", 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011123190000000001 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:37.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191439+0000", 2023-01-15T14:31:37.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3648999999999999e-05 2023-01-15T14:31:37.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:37.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.191491+0000", 2023-01-15T14:31:37.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2444999999999997e-05 2023-01-15T14:31:37.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.391472+0000", 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199981193 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592619+0000", 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20114642499999999 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.592748+0000", 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012950200000000001 2023-01-15T14:31:37.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593153+0000", 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040503700000000002 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1337 3.8 3:17d5d741:::smithi09025264-5:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:37.882 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 93.654541209000001, 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66179168200000005, 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1337 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:37.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:37.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:37.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:37.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932042+0000", 2023-01-15T14:31:37.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:37.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:37.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932046+0000", 2023-01-15T14:31:37.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8089999999999999e-06 2023-01-15T14:31:37.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:37.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932055+0000", 2023-01-15T14:31:37.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2199999999999998e-06 2023-01-15T14:31:37.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:37.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:03.932061+0000", 2023-01-15T14:31:37.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2779999999999999e-06 2023-01-15T14:31:37.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190318+0000", 2023-01-15T14:31:37.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.258256967 2023-01-15T14:31:37.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.190413+0000", 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4627000000000001e-05 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593288+0000", 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.402875339 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:37.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593515+0000", 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022700699999999999 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:04.593834+0000", 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031906799999999999 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.888 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1690 3.16 3:68fb3263:::smithi09025264-33:head [write 1763227~472137] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.724068238999998, 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72570748100000004, 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1690 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:37.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:37.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:37.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:37.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:37.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:37.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:31:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:38.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862514+0000", 2023-01-15T14:31:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862746+0000", 2023-01-15T14:31:38.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000232301 2023-01-15T14:31:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862770+0000", 2023-01-15T14:31:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3286e-05 2023-01-15T14:31:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:38.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862779+0000", 2023-01-15T14:31:38.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4199999999999996e-06 2023-01-15T14:31:38.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:38.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862844+0000", 2023-01-15T14:31:38.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5036999999999994e-05 2023-01-15T14:31:38.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:38.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862860+0000", 2023-01-15T14:31:38.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6426000000000002e-05 2023-01-15T14:31:38.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:38.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051251+0000", 2023-01-15T14:31:38.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18839023299999999 2023-01-15T14:31:38.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:38.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051466+0000", 2023-01-15T14:31:38.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021523199999999999 2023-01-15T14:31:38.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:38.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368326+0000", 2023-01-15T14:31:38.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31686002699999999 2023-01-15T14:31:38.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:38.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368559+0000", 2023-01-15T14:31:38.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023305400000000001 2023-01-15T14:31:38.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:38.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368935+0000", 2023-01-15T14:31:38.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037579099999999999 2023-01-15T14:31:38.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:38.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369531+0000", 2023-01-15T14:31:38.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059593199999999997 2023-01-15T14:31:38.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:38.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369571+0000", 2023-01-15T14:31:38.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0649999999999999e-05 2023-01-15T14:31:38.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:38.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370331+0000", 2023-01-15T14:31:38.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075969899999999996 2023-01-15T14:31:38.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:38.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371230+0000", 2023-01-15T14:31:38.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000899134 2023-01-15T14:31:38.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:38.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472895+0000", 2023-01-15T14:31:38.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101665191 2023-01-15T14:31:38.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:38.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.587914+0000", 2023-01-15T14:31:38.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115018671 2023-01-15T14:31:38.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:38.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588013+0000", 2023-01-15T14:31:38.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8494999999999998e-05 2023-01-15T14:31:38.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:38.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588223+0000", 2023-01-15T14:31:38.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210294 2023-01-15T14:31:38.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:38.058 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:38.058 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:38.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.058 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1691 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 2235364] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:38.058 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:38.058 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.723783984999997, 2023-01-15T14:31:38.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72593841699999995, 2023-01-15T14:31:38.059 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:38.059 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:38.059 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:38.059 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:38.059 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:38.059 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1691 2023-01-15T14:31:38.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.059 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:38.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:38.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:38.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:38.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:38.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:31:38.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:38.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:38.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862799+0000", 2023-01-15T14:31:38.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:38.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:38.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862807+0000", 2023-01-15T14:31:38.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1280000000000008e-06 2023-01-15T14:31:38.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:38.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862821+0000", 2023-01-15T14:31:38.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.345e-05 2023-01-15T14:31:38.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:38.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862826+0000", 2023-01-15T14:31:38.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3720000000000004e-06 2023-01-15T14:31:38.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:38.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862883+0000", 2023-01-15T14:31:38.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6589000000000003e-05 2023-01-15T14:31:38.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:38.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862888+0000", 2023-01-15T14:31:38.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4060000000000004e-06 2023-01-15T14:31:38.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:38.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151601+0000", 2023-01-15T14:31:38.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28871354100000002 2023-01-15T14:31:38.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:38.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151732+0000", 2023-01-15T14:31:38.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013032300000000001 2023-01-15T14:31:38.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:38.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369729+0000", 2023-01-15T14:31:38.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217997515 2023-01-15T14:31:38.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:38.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369949+0000", 2023-01-15T14:31:38.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219664 2023-01-15T14:31:38.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:38.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370367+0000", 2023-01-15T14:31:38.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041822799999999998 2023-01-15T14:31:38.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:38.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370771+0000", 2023-01-15T14:31:38.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040358400000000002 2023-01-15T14:31:38.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:38.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370794+0000", 2023-01-15T14:31:38.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-05 2023-01-15T14:31:38.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:38.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370880+0000", 2023-01-15T14:31:38.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6020999999999995e-05 2023-01-15T14:31:38.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:38.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371241+0000", 2023-01-15T14:31:38.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000360956 2023-01-15T14:31:38.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:38.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472994+0000", 2023-01-15T14:31:38.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101753089 2023-01-15T14:31:38.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:38.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588348+0000", 2023-01-15T14:31:38.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11535382299999999 2023-01-15T14:31:38.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:38.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588539+0000", 2023-01-15T14:31:38.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019090399999999999 2023-01-15T14:31:38.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:38.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588738+0000", 2023-01-15T14:31:38.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019910800000000001 2023-01-15T14:31:38.070 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:38.070 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:38.070 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:38.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.071 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:38.071 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:38.071 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 57.723750500999998, 2023-01-15T14:31:38.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:31:38.071 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:38.071 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:38.071 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:38.071 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:38.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:38.253 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:31:38.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.254 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:38.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:38.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:38.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:38.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:38.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:38.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:38.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:38.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:31:38.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:38.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:38.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:31:38.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:31:38.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:38.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:31:38.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:31:38.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:38.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:31:38.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:31:38.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:38.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:31:38.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:31:38.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:38.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:31:38.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:31:38.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:38.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:31:38.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:31:38.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:38.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:31:38.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:31:38.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:38.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:31:38.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:31:38.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:38.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:31:38.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:31:38.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:38.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:31:38.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:31:38.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:38.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:31:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:31:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:31:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:31:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:38.266 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:38.267 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:38.268 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:31:38.269 INFO:tasks.rados.rados.0.smithi090.stdout:833: done (0 left) 2023-01-15T14:31:38.269 INFO:tasks.rados.rados.0.smithi090.stdout:834: write (excl) oid 46 current snap is 85 2023-01-15T14:31:38.269 INFO:tasks.rados.rados.0.smithi090.stdout:834: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2023-01-15T14:31:38.269 INFO:tasks.rados.rados.0.smithi090.stdout:834: writing smithi09025264-46 from 578240 to 1144746 tid 1 2023-01-15T14:31:38.269 INFO:tasks.rados.rados.0.smithi090.stdout:834: writing smithi09025264-46 from 1609179 to 2167277 tid 2 2023-01-15T14:31:38.270 INFO:tasks.rados.rados.0.smithi090.stdout:834: writing smithi09025264-46 from 2804887 to 3356637 tid 3 2023-01-15T14:31:38.270 INFO:tasks.rados.rados.0.smithi090.stdout:835: copy_from oid 23 from oid 45 current snap is 85 2023-01-15T14:31:38.270 INFO:tasks.rados.rados.0.smithi090.stdout:836: snap_remove snap 67 2023-01-15T14:31:38.270 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:38.262+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:38.370 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:38.366+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:40.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:31:40.036 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:31:40.038 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:40.038 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:31:40.195 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:31:40.195 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:40.196 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:31:40.197 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:40.030+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:40.206 INFO:tasks.rados.rados.0.smithi090.stdout:836: done (2 left) 2023-01-15T14:31:40.206 INFO:tasks.rados.rados.0.smithi090.stdout:837: rollback oid 10 current snap is 85 2023-01-15T14:31:40.206 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 10 to 72 2023-01-15T14:31:40.206 INFO:tasks.rados.rados.0.smithi090.stdout:834: finishing write tid 1 to smithi09025264-46 2023-01-15T14:31:40.206 INFO:tasks.rados.rados.0.smithi090.stdout:834: finishing write tid 2 to smithi09025264-46 2023-01-15T14:31:40.207 INFO:tasks.rados.rados.0.smithi090.stdout:834: finishing write tid 3 to smithi09025264-46 2023-01-15T14:31:40.207 INFO:tasks.rados.rados.0.smithi090.stdout:835: finishing copy_from to smithi09025264-23 2023-01-15T14:31:40.207 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 73 (ObjNum 206 snap 44 seq_num 206) dirty exists 2023-01-15T14:31:40.207 INFO:tasks.rados.rados.0.smithi090.stdout:835: finishing copy_from racing read to smithi09025264-23 2023-01-15T14:31:40.207 INFO:tasks.rados.rados.0.smithi090.stdout:834: finishing write tid 4 to smithi09025264-46 2023-01-15T14:31:40.207 INFO:tasks.rados.rados.0.smithi090.stdout:834: finishing write tid 5 to smithi09025264-46 2023-01-15T14:31:40.207 INFO:tasks.rados.rados.0.smithi090.stdout:834: oid 46 updating version 0 to 223 2023-01-15T14:31:40.207 INFO:tasks.rados.rados.0.smithi090.stdout:834: oid 46 updating version 223 to 224 2023-01-15T14:31:40.208 INFO:tasks.rados.rados.0.smithi090.stdout:834: oid 46 version 224 is already newer than 222 2023-01-15T14:31:40.208 INFO:tasks.rados.rados.0.smithi090.stdout:834: oid 46 version 224 is already newer than 221 2023-01-15T14:31:40.208 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 224 (ObjNum 320 snap 85 seq_num 320) dirty exists 2023-01-15T14:31:40.208 INFO:tasks.rados.rados.0.smithi090.stdout:834: left oid 46 (ObjNum 320 snap 85 seq_num 320) 2023-01-15T14:31:40.208 INFO:tasks.rados.rados.0.smithi090.stdout:834: done (2 left) 2023-01-15T14:31:40.208 INFO:tasks.rados.rados.0.smithi090.stdout:835: done (1 left) 2023-01-15T14:31:40.208 INFO:tasks.rados.rados.0.smithi090.stdout:838: write (excl) oid 2 current snap is 85 2023-01-15T14:31:40.208 INFO:tasks.rados.rados.0.smithi090.stdout:838: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2023-01-15T14:31:40.208 INFO:tasks.rados.rados.0.smithi090.stdout:838: writing smithi09025264-2 from 595047 to 1236802 tid 1 2023-01-15T14:31:40.208 INFO:tasks.rados.rados.0.smithi090.stdout:838: writing smithi09025264-2 from 1667661 to 2085770 tid 2 2023-01-15T14:31:40.208 INFO:tasks.rados.rados.0.smithi090.stdout:838: writing smithi09025264-2 from 2548663 to 3311685 tid 3 2023-01-15T14:31:40.209 INFO:tasks.rados.rados.0.smithi090.stdout:839: copy_from oid 12 from oid 33 current snap is 85 2023-01-15T14:31:40.209 INFO:tasks.rados.rados.0.smithi090.stdout:840: read oid 8 snap 84 2023-01-15T14:31:40.209 INFO:tasks.rados.rados.0.smithi090.stdout:840: expect (ObjNum 287 snap 74 seq_num 287) 2023-01-15T14:31:40.209 INFO:tasks.rados.rados.0.smithi090.stdout:841: rollback oid 39 current snap is 85 2023-01-15T14:31:40.209 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 39 to 76 2023-01-15T14:31:40.209 INFO:tasks.rados.rados.0.smithi090.stdout:842: read oid 18 snap -1 2023-01-15T14:31:40.209 INFO:tasks.rados.rados.0.smithi090.stdout:842: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:40.209 INFO:tasks.rados.rados.0.smithi090.stdout:843: write (excl) oid 20 current snap is 85 2023-01-15T14:31:40.209 INFO:tasks.rados.rados.0.smithi090.stdout:843: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2023-01-15T14:31:40.209 INFO:tasks.rados.rados.0.smithi090.stdout:843: writing smithi09025264-20 from 611854 to 1328858 tid 1 2023-01-15T14:31:40.209 INFO:tasks.rados.rados.0.smithi090.stdout:843: writing smithi09025264-20 from 2126143 to 2687910 tid 2 2023-01-15T14:31:40.210 INFO:tasks.rados.rados.0.smithi090.stdout:843: writing smithi09025264-20 from 3259733 to 3411854 tid 3 2023-01-15T14:31:40.210 INFO:tasks.rados.rados.0.smithi090.stdout:844: write oid 19 current snap is 85 2023-01-15T14:31:40.210 INFO:tasks.rados.rados.0.smithi090.stdout:844: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2023-01-15T14:31:40.210 INFO:tasks.rados.rados.0.smithi090.stdout:844: writing smithi09025264-19 from 628661 to 1420914 tid 1 2023-01-15T14:31:40.210 INFO:tasks.rados.rados.0.smithi090.stdout:844: writing smithi09025264-19 from 2184625 to 2606403 tid 2 2023-01-15T14:31:40.210 INFO:tasks.rados.rados.0.smithi090.stdout:844: writing smithi09025264-19 from 3403509 to 3428661 tid 3 2023-01-15T14:31:40.210 INFO:tasks.rados.rados.0.smithi090.stdout:845: snap_create 2023-01-15T14:31:40.210 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:31:40.210 INFO:tasks.rados.rados.0.smithi090.stdout:842: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:40.210 INFO:tasks.rados.rados.0.smithi090.stdout:840: expect (ObjNum 287 snap 74 seq_num 287) 2023-01-15T14:31:40.210 INFO:tasks.rados.rados.0.smithi090.stdout:841: finishing rollback tid 1 to smithi09025264-39 2023-01-15T14:31:40.211 INFO:tasks.rados.rados.0.smithi090.stdout:841: finishing rollback tid 2 to smithi09025264-39 2023-01-15T14:31:40.211 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 226 (ObjNum 183 snap 40 seq_num 183) dirty exists 2023-01-15T14:31:40.211 INFO:tasks.rados.rados.0.smithi090.stdout:844: finishing write tid 1 to smithi09025264-19 2023-01-15T14:31:40.211 INFO:tasks.rados.rados.0.smithi090.stdout:837: finishing rollback tid 0 to smithi09025264-10 2023-01-15T14:31:40.211 INFO:tasks.rados.rados.0.smithi090.stdout:838: finishing write tid 1 to smithi09025264-2 2023-01-15T14:31:40.211 INFO:tasks.rados.rados.0.smithi090.stdout:843: finishing write tid 1 to smithi09025264-20 2023-01-15T14:31:40.211 INFO:tasks.rados.rados.0.smithi090.stdout:839: finishing copy_from to smithi09025264-12 2023-01-15T14:31:40.211 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 73 (ObjNum 252 snap 60 seq_num 252) dirty exists 2023-01-15T14:31:40.211 INFO:tasks.rados.rados.0.smithi090.stdout:840: done (7 left) 2023-01-15T14:31:40.211 INFO:tasks.rados.rados.0.smithi090.stdout:841: done (6 left) 2023-01-15T14:31:40.211 INFO:tasks.rados.rados.0.smithi090.stdout:842: done (5 left) 2023-01-15T14:31:40.212 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:31:40.212 INFO:tasks.rados.rados.0.smithi090.stdout:839: finishing copy_from racing read to smithi09025264-12 2023-01-15T14:31:40.212 INFO:tasks.rados.rados.0.smithi090.stdout:844: finishing write tid 2 to smithi09025264-19 2023-01-15T14:31:40.212 INFO:tasks.rados.rados.0.smithi090.stdout:844: finishing write tid 3 to smithi09025264-19 2023-01-15T14:31:40.212 INFO:tasks.rados.rados.0.smithi090.stdout:844: finishing write tid 4 to smithi09025264-19 2023-01-15T14:31:40.212 INFO:tasks.rados.rados.0.smithi090.stdout:844: finishing write tid 5 to smithi09025264-19 2023-01-15T14:31:40.212 INFO:tasks.rados.rados.0.smithi090.stdout:844: oid 19 updating version 0 to 175 2023-01-15T14:31:40.212 INFO:tasks.rados.rados.0.smithi090.stdout:844: oid 19 updating version 175 to 177 2023-01-15T14:31:40.212 INFO:tasks.rados.rados.0.smithi090.stdout:844: oid 19 version 177 is already newer than 176 2023-01-15T14:31:40.212 INFO:tasks.rados.rados.0.smithi090.stdout:844: oid 19 updating version 177 to 178 2023-01-15T14:31:40.212 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 178 (ObjNum 323 snap 85 seq_num 323) dirty exists 2023-01-15T14:31:40.213 INFO:tasks.rados.rados.0.smithi090.stdout:844: left oid 19 (ObjNum 323 snap 85 seq_num 323) 2023-01-15T14:31:40.213 INFO:tasks.rados.rados.0.smithi090.stdout:839: done (4 left) 2023-01-15T14:31:40.213 INFO:tasks.rados.rados.0.smithi090.stdout:844: done (3 left) 2023-01-15T14:31:40.213 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:31:40.213 INFO:tasks.rados.rados.0.smithi090.stdout:837: finishing rollback tid 1 to smithi09025264-10 2023-01-15T14:31:40.213 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 267 (ObjNum 129 snap 0 seq_num 3197583904) dirty dne 2023-01-15T14:31:40.213 INFO:tasks.rados.rados.0.smithi090.stdout:837: done (2 left) 2023-01-15T14:31:40.213 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:40.213 INFO:tasks.rados.rados.0.smithi090.stdout:843: finishing write tid 2 to smithi09025264-20 2023-01-15T14:31:40.213 INFO:tasks.rados.rados.0.smithi090.stdout:838: finishing write tid 2 to smithi09025264-2 2023-01-15T14:31:40.213 INFO:tasks.rados.rados.0.smithi090.stdout:838: finishing write tid 3 to smithi09025264-2 2023-01-15T14:31:40.214 INFO:tasks.rados.rados.0.smithi090.stdout:843: finishing write tid 3 to smithi09025264-20 2023-01-15T14:31:40.214 INFO:tasks.rados.rados.0.smithi090.stdout:838: finishing write tid 4 to smithi09025264-2 2023-01-15T14:31:40.214 INFO:tasks.rados.rados.0.smithi090.stdout:838: finishing write tid 5 to smithi09025264-2 2023-01-15T14:31:40.214 INFO:tasks.rados.rados.0.smithi090.stdout:838: oid 2 updating version 0 to 238 2023-01-15T14:31:40.214 INFO:tasks.rados.rados.0.smithi090.stdout:838: oid 2 updating version 238 to 240 2023-01-15T14:31:40.214 INFO:tasks.rados.rados.0.smithi090.stdout:838: oid 2 version 240 is already newer than 239 2023-01-15T14:31:40.214 INFO:tasks.rados.rados.0.smithi090.stdout:838: oid 2 version 240 is already newer than 237 2023-01-15T14:31:40.214 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 240 (ObjNum 321 snap 85 seq_num 321) dirty exists 2023-01-15T14:31:40.215 INFO:tasks.rados.rados.0.smithi090.stdout:838: left oid 2 (ObjNum 321 snap 85 seq_num 321) 2023-01-15T14:31:40.215 INFO:tasks.rados.rados.0.smithi090.stdout:838: done (1 left) 2023-01-15T14:31:40.215 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:40.215 INFO:tasks.rados.rados.0.smithi090.stdout:843: finishing write tid 4 to smithi09025264-20 2023-01-15T14:31:40.215 INFO:tasks.rados.rados.0.smithi090.stdout:843: finishing write tid 5 to smithi09025264-20 2023-01-15T14:31:40.215 INFO:tasks.rados.rados.0.smithi090.stdout:843: oid 20 updating version 0 to 49 2023-01-15T14:31:40.215 INFO:tasks.rados.rados.0.smithi090.stdout:843: oid 20 updating version 49 to 50 2023-01-15T14:31:40.215 INFO:tasks.rados.rados.0.smithi090.stdout:843: oid 20 version 50 is already newer than 48 2023-01-15T14:31:40.215 INFO:tasks.rados.rados.0.smithi090.stdout:843: oid 20 updating version 50 to 51 2023-01-15T14:31:40.215 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 51 (ObjNum 322 snap 85 seq_num 322) dirty exists 2023-01-15T14:31:40.216 INFO:tasks.rados.rados.0.smithi090.stdout:843: left oid 20 (ObjNum 322 snap 85 seq_num 322) 2023-01-15T14:31:40.217 INFO:tasks.rados.rados.0.smithi090.stdout:843: done (0 left) 2023-01-15T14:31:40.217 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:40.202+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:40.306 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:40.299+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:40.389 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:40.390 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:40.390 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:40.390 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:31:40.390 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:31:40.390 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:31:40.390 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:40.391 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:31:40.397 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:40.407 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:40.402+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:40.554 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:40.554 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:40.555 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:40.555 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:40.555 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:40.556 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:40.556 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:40.557 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:31:40.558 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:31:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:31:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:31:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:31:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:31:40.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:31:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:31:40.560 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:31:40.560 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:40.547+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:40.653 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:40.651+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:40.715 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:40.716 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:40.719 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:31:40.754 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:40.750+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:40.888 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:40.892 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:40.893 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:31:40.894 INFO:tasks.rados.rados.0.smithi090.stdout:845: done (0 left) 2023-01-15T14:31:40.894 INFO:tasks.rados.rados.0.smithi090.stdout:846: read oid 1 snap -1 2023-01-15T14:31:40.894 INFO:tasks.rados.rados.0.smithi090.stdout:846: expect deleted 2023-01-15T14:31:40.894 INFO:tasks.rados.rados.0.smithi090.stdout:847: read oid 37 snap 62 2023-01-15T14:31:40.894 INFO:tasks.rados.rados.0.smithi090.stdout:847: expect deleted 2023-01-15T14:31:40.894 INFO:tasks.rados.rados.0.smithi090.stdout:848: read oid 42 snap -1 2023-01-15T14:31:40.894 INFO:tasks.rados.rados.0.smithi090.stdout:848: expect deleted 2023-01-15T14:31:40.894 INFO:tasks.rados.rados.0.smithi090.stdout:849: copy_from oid 50 from oid 29 current snap is 86 2023-01-15T14:31:40.895 INFO:tasks.rados.rados.0.smithi090.stdout:850: rollback oid 32 current snap is 86 2023-01-15T14:31:40.895 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 32 to 72 2023-01-15T14:31:40.895 INFO:tasks.rados.rados.0.smithi090.stdout:851: read oid 40 snap -1 2023-01-15T14:31:40.895 INFO:tasks.rados.rados.0.smithi090.stdout:851: expect (ObjNum 316 snap 83 seq_num 316) 2023-01-15T14:31:40.895 INFO:tasks.rados.rados.0.smithi090.stdout:852: snap_remove snap 85 2023-01-15T14:31:40.896 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:31:40.896 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:40.883+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:40.988 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:40.983+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:41.055 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:41.055 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:41.055 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:31:41.089 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:41.083+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:41.211 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:41.211 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:41.211 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:31:41.212 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:41.207+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:41.312 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:41.307+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:41.367 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:41.368 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:41.368 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:31:41.413 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:41.410+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:41.469 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:41.542 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:41.544 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:41.539+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:41.545 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:31:41.644 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:41.638+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:41.709 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:41.709 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:41.710 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:31:41.745 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:41.739+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:41.822 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:41.901 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:41.901 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:31:41.902 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:31:41.902 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-01-15T14:31:41.902 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.904422 2023-01-15T14:31:41.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0.9044220502430209 2023-01-15T14:31:41.905 INFO:tasks.rados.rados.0.smithi090.stdout:852: done (6 left) 2023-01-15T14:31:41.905 INFO:tasks.rados.rados.0.smithi090.stdout:853: rollback oid 9 current snap is 86 2023-01-15T14:31:41.905 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 9 to 81 2023-01-15T14:31:41.906 INFO:tasks.rados.rados.0.smithi090.stdout:849: finishing copy_from to smithi09025264-50 2023-01-15T14:31:41.906 INFO:tasks.rados.rados.0.smithi090.stdout:849: got expected ENOENT (src dne) 2023-01-15T14:31:41.906 INFO:tasks.rados.rados.0.smithi090.stdout:849: finishing copy_from racing read to smithi09025264-50 2023-01-15T14:31:41.906 INFO:tasks.rados.rados.0.smithi090.stdout:851: expect (ObjNum 316 snap 83 seq_num 316) 2023-01-15T14:31:41.907 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:31:41.908 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:41.898+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:42.005 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:42.002+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:42.168 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:42.169 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:42.169 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:42.162+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:42.170 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:31:42.171 INFO:tasks.rados.rados.0.smithi090.stdout:850: finishing rollback tid 0 to smithi09025264-32 2023-01-15T14:31:42.212 INFO:tasks.rados.rados.0.smithi090.stdout:850: finishing rollback tid 1 to smithi09025264-32 2023-01-15T14:31:42.219 INFO:tasks.rados.rados.0.smithi090.stdout:850: finishing rollback tid 2 to smithi09025264-32 2023-01-15T14:31:42.220 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 231 (ObjNum 255 snap 60 seq_num 255) dirty exists 2023-01-15T14:31:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:853: finishing rollback tid 1 to smithi09025264-9 2023-01-15T14:31:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 0 (ObjNum 49 snap 0 seq_num 3191846144) dirty dne 2023-01-15T14:31:42.221 INFO:tasks.rados.rados.0.smithi090.stdout:846: done (6 left) 2023-01-15T14:31:42.222 INFO:tasks.rados.rados.0.smithi090.stdout:847: done (5 left) 2023-01-15T14:31:42.222 INFO:tasks.rados.rados.0.smithi090.stdout:848: done (4 left) 2023-01-15T14:31:42.222 INFO:tasks.rados.rados.0.smithi090.stdout:849: done (3 left) 2023-01-15T14:31:42.222 INFO:tasks.rados.rados.0.smithi090.stdout:850: done (2 left) 2023-01-15T14:31:42.222 INFO:tasks.rados.rados.0.smithi090.stdout:851: done (1 left) 2023-01-15T14:31:42.222 INFO:tasks.rados.rados.0.smithi090.stdout:853: done (0 left) 2023-01-15T14:31:42.223 INFO:tasks.rados.rados.0.smithi090.stdout:854: rollback oid 44 current snap is 86 2023-01-15T14:31:42.223 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 44 to 71 2023-01-15T14:31:42.223 INFO:tasks.rados.rados.0.smithi090.stdout:855: read oid 27 snap -1 2023-01-15T14:31:42.223 INFO:tasks.rados.rados.0.smithi090.stdout:855: expect deleted 2023-01-15T14:31:42.223 INFO:tasks.rados.rados.0.smithi090.stdout:856: write oid 2 current snap is 86 2023-01-15T14:31:42.223 INFO:tasks.rados.rados.0.smithi090.stdout:856: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2023-01-15T14:31:42.223 INFO:tasks.rados.rados.0.smithi090.stdout:856: writing smithi09025264-2 from 645468 to 1112970 tid 1 2023-01-15T14:31:42.224 INFO:tasks.rados.rados.0.smithi090.stdout:856: writing smithi09025264-2 from 1726754 to 2292190 tid 2 2023-01-15T14:31:42.224 INFO:tasks.rados.rados.0.smithi090.stdout:856: writing smithi09025264-2 from 2798226 to 3395064 tid 3 2023-01-15T14:31:42.224 INFO:tasks.rados.rados.0.smithi090.stdout:857: read oid 11 snap -1 2023-01-15T14:31:42.224 INFO:tasks.rados.rados.0.smithi090.stdout:857: expect (ObjNum 262 snap 61 seq_num 262) 2023-01-15T14:31:42.224 INFO:tasks.rados.rados.0.smithi090.stdout:858: read oid 49 snap 62 2023-01-15T14:31:42.224 INFO:tasks.rados.rados.0.smithi090.stdout:858: expect (ObjNum 199 snap 43 seq_num 199) 2023-01-15T14:31:42.224 INFO:tasks.rados.rados.0.smithi090.stdout:859: snap_create 2023-01-15T14:31:42.225 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:31:42.225 INFO:tasks.rados.rados.0.smithi090.stdout:855: done (4 left) 2023-01-15T14:31:42.225 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:31:42.225 INFO:tasks.rados.rados.0.smithi090.stdout:854: finishing rollback tid 1 to smithi09025264-44 2023-01-15T14:31:42.225 INFO:tasks.rados.rados.0.smithi090.stdout:854: finishing rollback tid 2 to smithi09025264-44 2023-01-15T14:31:42.225 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 233 (ObjNum 273 snap 70 seq_num 273) dirty exists 2023-01-15T14:31:42.226 INFO:tasks.rados.rados.0.smithi090.stdout:854: done (3 left) 2023-01-15T14:31:42.226 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:31:42.227 INFO:tasks.rados.rados.0.smithi090.stdout:858: expect (ObjNum 199 snap 43 seq_num 199) 2023-01-15T14:31:42.268 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:42.263+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:42.383 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:42.384 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:42.384 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:31:42.384 INFO:tasks.rados.rados.0.smithi090.stdout:857: expect (ObjNum 262 snap 61 seq_num 262) 2023-01-15T14:31:42.385 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:42.379+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:42.484 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:42.478+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:42.543 INFO:tasks.rados.rados.0.smithi090.stdout:856: finishing write tid 1 to smithi09025264-2 2023-01-15T14:31:42.544 INFO:tasks.rados.rados.0.smithi090.stdout:856: finishing write tid 2 to smithi09025264-2 2023-01-15T14:31:42.544 INFO:tasks.rados.rados.0.smithi090.stdout:856: finishing write tid 3 to smithi09025264-2 2023-01-15T14:31:42.544 INFO:tasks.rados.rados.0.smithi090.stdout:857: done (2 left) 2023-01-15T14:31:42.544 INFO:tasks.rados.rados.0.smithi090.stdout:858: done (1 left) 2023-01-15T14:31:42.544 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:42.545 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:42.545 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:42.545 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:31:42.561 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:31:42.757 INFO:tasks.rados.rados.0.smithi090.stdout:856: finishing write tid 4 to smithi09025264-2 2023-01-15T14:31:42.757 INFO:tasks.rados.rados.0.smithi090.stdout:856: finishing write tid 5 to smithi09025264-2 2023-01-15T14:31:42.757 INFO:tasks.rados.rados.0.smithi090.stdout:856: oid 2 updating version 0 to 243 2023-01-15T14:31:42.757 INFO:tasks.rados.rados.0.smithi090.stdout:856: oid 2 version 243 is already newer than 241 2023-01-15T14:31:42.757 INFO:tasks.rados.rados.0.smithi090.stdout:856: oid 2 version 243 is already newer than 242 2023-01-15T14:31:42.758 INFO:tasks.rados.rados.0.smithi090.stdout:856: oid 2 updating version 243 to 244 2023-01-15T14:31:42.758 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 244 (ObjNum 324 snap 86 seq_num 324) dirty exists 2023-01-15T14:31:42.758 INFO:tasks.rados.rados.0.smithi090.stdout:856: left oid 2 (ObjNum 324 snap 86 seq_num 324) 2023-01-15T14:31:42.758 INFO:tasks.rados.rados.0.smithi090.stdout:856: done (0 left) 2023-01-15T14:31:42.759 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:42.759 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:42.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:31:42.760 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:42.754+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:42.858 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:42.854+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:42.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:31:42.920 INFO:tasks.rados.rados.0.smithi090.stdout:859: done (0 left) 2023-01-15T14:31:42.920 INFO:tasks.rados.rados.0.smithi090.stdout:860: read oid 34 snap -1 2023-01-15T14:31:42.920 INFO:tasks.rados.rados.0.smithi090.stdout:860: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:31:42.920 INFO:tasks.rados.rados.0.smithi090.stdout:861: read oid 35 snap 71 2023-01-15T14:31:42.920 INFO:tasks.rados.rados.0.smithi090.stdout:861: expect (ObjNum 269 snap 66 seq_num 269) 2023-01-15T14:31:42.921 INFO:tasks.rados.rados.0.smithi090.stdout:862: delete oid 24 current snap is 87 2023-01-15T14:31:42.921 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.3 primary-affinity to 0.904422 (8592722) 2023-01-15T14:31:42.922 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:42.922 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:42.930 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:31:42.946 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:43.079 INFO:tasks.rados.rados.0.smithi090.stdout:862: done (2 left) 2023-01-15T14:31:43.079 INFO:tasks.rados.rados.0.smithi090.stdout:863: rollback oid 15 current snap is 87 2023-01-15T14:31:43.080 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 15 to 84 2023-01-15T14:31:43.080 INFO:tasks.rados.rados.0.smithi090.stdout:864: read oid 11 snap 71 2023-01-15T14:31:43.080 INFO:tasks.rados.rados.0.smithi090.stdout:864: expect deleted 2023-01-15T14:31:43.080 INFO:tasks.rados.rados.0.smithi090.stdout:865: read oid 46 snap -1 2023-01-15T14:31:43.081 INFO:tasks.rados.rados.0.smithi090.stdout:865: expect (ObjNum 320 snap 85 seq_num 320) 2023-01-15T14:31:43.081 INFO:tasks.rados.rados.0.smithi090.stdout:866: copy_from oid 21 from oid 9 current snap is 87 2023-01-15T14:31:43.081 INFO:tasks.rados.rados.0.smithi090.stdout:867: rollback oid 31 current snap is 87 2023-01-15T14:31:43.081 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 31 to 76 2023-01-15T14:31:43.082 INFO:tasks.rados.rados.0.smithi090.stdout:868: read oid 23 snap -1 2023-01-15T14:31:43.082 INFO:tasks.rados.rados.0.smithi090.stdout:868: expect (ObjNum 206 snap 44 seq_num 206) 2023-01-15T14:31:43.082 INFO:tasks.rados.rados.0.smithi090.stdout:869: read oid 6 snap 72 2023-01-15T14:31:43.082 INFO:tasks.rados.rados.0.smithi090.stdout:869: expect (ObjNum 242 snap 55 seq_num 242) 2023-01-15T14:31:43.082 INFO:tasks.rados.rados.0.smithi090.stdout:870: write oid 30 current snap is 87 2023-01-15T14:31:43.083 INFO:tasks.rados.rados.0.smithi090.stdout:870: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2023-01-15T14:31:43.083 INFO:tasks.rados.rados.0.smithi090.stdout:870: writing smithi09025264-30 from 679082 to 1297082 tid 1 2023-01-15T14:31:43.083 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:43.085 INFO:tasks.rados.rados.0.smithi090.stdout:870: writing smithi09025264-30 from 1843718 to 2412823 tid 2 2023-01-15T14:31:43.085 INFO:tasks.rados.rados.0.smithi090.stdout:870: writing smithi09025264-30 from 2853072 to 3479082 tid 3 2023-01-15T14:31:43.085 INFO:tasks.rados.rados.0.smithi090.stdout:871: write (excl) oid 33 current snap is 87 2023-01-15T14:31:43.085 INFO:tasks.rados.rados.0.smithi090.stdout:871: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2023-01-15T14:31:43.085 INFO:tasks.rados.rados.0.smithi090.stdout:871: writing smithi09025264-33 from 695889 to 1105491 tid 1 2023-01-15T14:31:43.086 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:43.087 INFO:tasks.rados.rados.0.smithi090.stdout:871: writing smithi09025264-33 from 1618553 to 2331316 tid 2 2023-01-15T14:31:43.087 INFO:tasks.rados.rados.0.smithi090.stdout:871: writing smithi09025264-33 from 2996848 to 3495889 tid 3 2023-01-15T14:31:43.088 INFO:tasks.rados.rados.0.smithi090.stdout:872: write oid 50 current snap is 87 2023-01-15T14:31:43.088 INFO:tasks.rados.rados.0.smithi090.stdout:872: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2023-01-15T14:31:43.088 INFO:tasks.rados.rados.0.smithi090.stdout:872: writing smithi09025264-50 from 712696 to 1197547 tid 1 2023-01-15T14:31:43.088 INFO:tasks.rados.rados.0.smithi090.stdout:872: writing smithi09025264-50 from 1960682 to 2533456 tid 2 2023-01-15T14:31:43.088 INFO:tasks.rados.rados.0.smithi090.stdout:872: writing smithi09025264-50 from 3307918 to 3512696 tid 3 2023-01-15T14:31:43.088 INFO:tasks.rados.rados.0.smithi090.stdout:864: done (11 left) 2023-01-15T14:31:43.088 INFO:tasks.rados.rados.0.smithi090.stdout:873: copy_from oid 14 from oid 2 current snap is 87 2023-01-15T14:31:43.089 INFO:tasks.rados.rados.0.smithi090.stdout:874: read oid 8 snap -1 2023-01-15T14:31:43.089 INFO:tasks.rados.rados.0.smithi090.stdout:874: expect (ObjNum 287 snap 74 seq_num 287) 2023-01-15T14:31:43.089 INFO:tasks.rados.rados.0.smithi090.stdout:875: read oid 11 snap -1 2023-01-15T14:31:43.089 INFO:tasks.rados.rados.0.smithi090.stdout:875: expect (ObjNum 262 snap 61 seq_num 262) 2023-01-15T14:31:43.089 INFO:tasks.rados.rados.0.smithi090.stdout:876: read oid 5 snap 83 2023-01-15T14:31:43.089 INFO:tasks.rados.rados.0.smithi090.stdout:876: expect deleted 2023-01-15T14:31:43.089 INFO:tasks.rados.rados.0.smithi090.stdout:877: read oid 40 snap -1 2023-01-15T14:31:43.090 INFO:tasks.rados.rados.0.smithi090.stdout:877: expect (ObjNum 316 snap 83 seq_num 316) 2023-01-15T14:31:43.090 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:31:43.091 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:43.075+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:43.104 INFO:tasks.rados.rados.0.smithi090.stdout:860: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:31:43.180 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:43.175+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:43.238 INFO:tasks.rados.rados.0.smithi090.stdout:860: done (15 left) 2023-01-15T14:31:43.238 INFO:tasks.rados.rados.0.smithi090.stdout:878: read oid 16 snap -1 2023-01-15T14:31:43.238 INFO:tasks.rados.rados.0.smithi090.stdout:878: expect deleted 2023-01-15T14:31:43.238 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:31:43.238 INFO:tasks.rados.rados.0.smithi090.stdout:868: expect (ObjNum 206 snap 44 seq_num 206) 2023-01-15T14:31:43.239 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:43.239 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:43.240 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:43.240 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:43.240 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:43.240 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:31:43.281 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:43.278+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:43.413 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:43.413 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:43.414 INFO:tasks.rados.rados.0.smithi090.stdout:872: finishing write tid 1 to smithi09025264-50 2023-01-15T14:31:43.414 INFO:tasks.rados.rados.0.smithi090.stdout:875: expect (ObjNum 262 snap 61 seq_num 262) 2023-01-15T14:31:43.414 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:43.414 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:43.414 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:43.415 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:43.407+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:43.415 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:31:43.415 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:31:43.415 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:31:43.415 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:31:43.415 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:31:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:31:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:31:43.416 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:31:43.495 INFO:tasks.rados.rados.0.smithi090.stdout:865: expect (ObjNum 320 snap 85 seq_num 320) 2023-01-15T14:31:43.576 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:43.577 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:43.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:43.577 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:43.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.577 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:43.577 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:43.577 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.43046799999999, 2023-01-15T14:31:43.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:31:43.577 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:43.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:43.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:43.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:43.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:43.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:43.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:43.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:43.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:31:43.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:31:43.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:43.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:31:43.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:31:43.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:43.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:31:43.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:31:43.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:43.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:31:43.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:31:43.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:43.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:31:43.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:31:43.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:43.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:31:43.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:31:43.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:43.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:31:43.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:31:43.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:43.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:31:43.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:31:43.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:43.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:31:43.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:31:43.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:43.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:31:43.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:31:43.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:43.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:31:43.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:31:43.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:43.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:31:43.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:31:43.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:43.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:31:43.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:31:43.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:43.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:31:43.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:31:43.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:43.587 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:43.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:43.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.588 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:43.588 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:43.588 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.43016820299999, 2023-01-15T14:31:43.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:31:43.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:43.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:43.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:43.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:43.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:43.588 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:31:43.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.589 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:43.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:43.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:43.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:43.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:43.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:43.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:31:43.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:43.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:31:43.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:31:43.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:43.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:31:43.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:31:43.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:43.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:31:43.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:31:43.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:43.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:31:43.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:31:43.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:43.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:31:43.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:31:43.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:43.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:31:43.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:31:43.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:43.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:31:43.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.430138419, 2023-01-15T14:31:43.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:31:43.784 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:43.784 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:43.784 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:43.784 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:43.784 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:43.784 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:31:43.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.785 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:43.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:43.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:43.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:43.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:43.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:43.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:43.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:43.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:31:43.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:43.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:43.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:31:43.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:31:43.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:43.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:31:43.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:31:43.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:43.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:31:43.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:31:43.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:43.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:31:43.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:31:43.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:43.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:31:43.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:31:43.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:43.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:31:43.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:31:43.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:43.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:31:43.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:31:43.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:43.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:31:43.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:31:43.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:43.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:31:43.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:31:43.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:43.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:31:43.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:31:43.984 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:43.984 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:43.984 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:43.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.984 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:182 3.4 3:2022fdfa:::smithi09025264-46:head [write 1933572~791670] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:43.984 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:43.985 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.30127634999999, 2023-01-15T14:31:43.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.582304865, 2023-01-15T14:31:43.985 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:43.985 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:43.985 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:43.985 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:43.985 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:43.985 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 182 2023-01-15T14:31:43.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.985 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:43.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213704+0000", 2023-01-15T14:31:43.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:43.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:43.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214574+0000", 2023-01-15T14:31:43.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087037799999999999 2023-01-15T14:31:43.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:43.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214608+0000", 2023-01-15T14:31:43.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:31:43.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:43.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214627+0000", 2023-01-15T14:31:43.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:31:43.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:43.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491537+0000", 2023-01-15T14:31:43.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27691020300000002 2023-01-15T14:31:43.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:43.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491721+0000", 2023-01-15T14:31:43.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345500000000001 2023-01-15T14:31:43.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:43.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693760+0000", 2023-01-15T14:31:43.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202039252 2023-01-15T14:31:43.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:43.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693898+0000", 2023-01-15T14:31:43.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013789 2023-01-15T14:31:43.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:43.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694230+0000", 2023-01-15T14:31:43.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220899999999998 2023-01-15T14:31:43.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:43.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694866+0000", 2023-01-15T14:31:43.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063640599999999995 2023-01-15T14:31:43.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:43.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694887+0000", 2023-01-15T14:31:43.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0622999999999999e-05 2023-01-15T14:31:43.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:43.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695950+0000", 2023-01-15T14:31:43.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001062688 2023-01-15T14:31:43.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:43.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696014+0000", 2023-01-15T14:31:43.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4697999999999996e-05 2023-01-15T14:31:43.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:43.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795898+0000", 2023-01-15T14:31:43.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099883338000000002 2023-01-15T14:31:43.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:43.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795922+0000", 2023-01-15T14:31:43.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4326000000000001e-05 2023-01-15T14:31:43.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:43.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795955+0000", 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3191000000000001e-05 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796011+0000", 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5631999999999999e-05 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.994 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:43.995 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:43.995 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.300329619, 2023-01-15T14:31:43.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:31:43.995 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:43.995 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:43.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:43.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:43.995 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:43.995 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:31:43.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.995 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:43.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:43.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:43.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:43.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:43.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:43.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:43.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:31:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:31:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:31:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:31:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:31:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:31:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:31:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:31:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:31:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:31:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:31:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:31:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:31:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:31:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:31:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:31:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:31:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:31:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:31:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:31:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:31:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:31:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:31:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:31:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:31:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:31:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:31:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:31:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:31:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:31:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:31:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:31:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:31:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.300272922, 2023-01-15T14:31:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:31:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:31:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:31:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:31:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:31:44.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:31:44.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:31:44.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:31:44.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:31:44.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:31:44.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:31:44.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:44.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:31:44.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:31:44.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:31:44.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:31:44.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:31:44.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:31:44.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:31:44.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 193.683776749, 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:44.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:44.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:31:44.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:44.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:31:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:31:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:31:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:31:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:31:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:31:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:31:44.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:31:44.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:31:44.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:31:44.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 186.654928278, 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.197 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:44.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.198 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:31:44.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.198 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:31:44.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:31:44.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:31:44.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:31:44.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:31:44.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:31:44.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:31:44.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:31:44.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:31:44.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:44.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:31:44.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:31:44.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:31:44.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:373 3.4 3:2029fa71:::smithi09025264-39:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 183.43668107100001, 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58298356399999995, 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:44.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078299+0000", 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078644+0000", 2023-01-15T14:31:44.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034494099999999998 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078676+0000", 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1183000000000003e-05 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078694+0000", 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-05 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078748+0000", 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3634999999999997e-05 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078753+0000", 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6370000000000004e-06 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110841+0000", 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032087439000000002 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110967+0000", 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126029 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.457874+0000", 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.346907725 2023-01-15T14:31:44.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458066+0000", 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000191592 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458321+0000", 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025454200000000002 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458918+0000", 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059740800000000001 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458945+0000", 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-05 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459253+0000", 2023-01-15T14:31:44.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030836200000000001 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460943+0000", 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016903580000000001 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461340+0000", 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039665799999999998 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661129+0000", 2023-01-15T14:31:44.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19978933500000001 2023-01-15T14:31:44.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:44.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661188+0000", 2023-01-15T14:31:44.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8687000000000002e-05 2023-01-15T14:31:44.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661285+0000", 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779000000000003e-05 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:375 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (56) in=7b,truncate 3142876] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 183.43187739300001, 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57851114599999998, 2023-01-15T14:31:44.217 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 375 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083105+0000", 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083104+0000", 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083113+0000", 2023-01-15T14:31:44.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6500000000000008e-06 2023-01-15T14:31:44.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083130+0000", 2023-01-15T14:31:44.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6860000000000001e-05 2023-01-15T14:31:44.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083142+0000", 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2252999999999999e-05 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083173+0000", 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0474999999999999e-05 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083175+0000", 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4990000000000001e-06 2023-01-15T14:31:44.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183469+0000", 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10029313099999999 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183747+0000", 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000277998 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460165+0000", 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27641872899999997 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460342+0000", 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000176792 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460579+0000", 2023-01-15T14:31:44.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000237266 2023-01-15T14:31:44.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:44.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460833+0000", 2023-01-15T14:31:44.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025390900000000002 2023-01-15T14:31:44.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:44.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460853+0000", 2023-01-15T14:31:44.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9273e-05 2023-01-15T14:31:44.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461024+0000", 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017111900000000001 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461230+0000", 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020672600000000001 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461879+0000", 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064846999999999995 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661524+0000", 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19964509599999999 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661560+0000", 2023-01-15T14:31:44.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6264999999999999e-05 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661616+0000", 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5952000000000002e-05 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:376 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:44.227 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:44.228 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 183.43179513699999, 2023-01-15T14:31:44.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57864278800000002, 2023-01-15T14:31:44.228 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.228 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:44.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.228 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.228 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 376 2023-01-15T14:31:44.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.228 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:44.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:44.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083186+0000", 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083193+0000", 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083211+0000", 2023-01-15T14:31:44.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7609999999999999e-05 2023-01-15T14:31:44.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083223+0000", 2023-01-15T14:31:44.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140000000000001e-05 2023-01-15T14:31:44.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093534+0000", 2023-01-15T14:31:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010311529 2023-01-15T14:31:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093551+0000", 2023-01-15T14:31:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.666e-05 2023-01-15T14:31:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183834+0000", 2023-01-15T14:31:44.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090283063999999996 2023-01-15T14:31:44.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:44.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.184076+0000", 2023-01-15T14:31:44.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024241900000000001 2023-01-15T14:31:44.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461001+0000", 2023-01-15T14:31:44.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27692504299999998 2023-01-15T14:31:44.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:44.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461152+0000", 2023-01-15T14:31:44.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150345 2023-01-15T14:31:44.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661642+0000", 2023-01-15T14:31:44.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200489742 2023-01-15T14:31:44.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661720+0000", 2023-01-15T14:31:44.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8604999999999997e-05 2023-01-15T14:31:44.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661830+0000", 2023-01-15T14:31:44.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010992099999999999 2023-01-15T14:31:44.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.574 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.575 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:31:44.575 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:44.575 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 156.65364130200001, 2023-01-15T14:31:44.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:31:44.575 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:44.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.576 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:31:44.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.576 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:44.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:44.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:44.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:44.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:31:44.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:31:44.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:31:44.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:31:44.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:31:44.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:31:44.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:31:44.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:31:44.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:31:44.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:31:44.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:44.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:31:44.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:31:44.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:44.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:31:44.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:31:44.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:44.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:31:44.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:31:44.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:44.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:31:44.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:31:44.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:44.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:31:44.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:31:44.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:31:44.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:31:44.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:31:44.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:31:44.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:44.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:31:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:31:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:31:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:31:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.755 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.59306305200001, 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:31:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:31:44.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:31:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:31:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:31:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:31:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:31:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:31:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:31:44.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:31:44.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:44.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:31:44.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:31:44.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:44.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:31:44.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:31:44.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:31:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:31:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:31:44.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:31:44.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:31:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:31:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:31:44.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:31:44.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:31:44.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:31:44.767 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:31:44.768 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.593004342, 2023-01-15T14:31:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:31:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.770 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:31:44.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.770 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:44.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:31:44.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:31:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:31:44.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:31:44.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:31:44.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:31:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:31:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:31:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:31:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:31:44.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:31:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:31:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:31:44.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:31:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:31:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:31:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:31:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:31:44.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:31:44.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:31:44.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:31:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:44.781 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 105.636584849, 2023-01-15T14:31:44.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:31:44.782 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.782 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:44.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:31:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:44.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:31:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:31:44.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:31:44.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:31:44.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:31:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:31:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:31:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:31:44.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:31:44.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:31:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:31:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:31:44.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:31:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:31:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:31:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:31:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:31:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 105.57908716599999, 2023-01-15T14:31:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:31:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.792 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:31:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:31:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:44.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:31:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:31:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:31:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:31:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:31:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:31:44.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:31:44.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:31:44.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:31:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:31:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:31:44.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:31:44.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:31:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:31:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:31:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:31:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:31:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:31:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:31:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:31:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:31:44.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:44.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:31:44.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:31:44.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:31:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:31:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:31:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:31:44.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:31:44.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:31:44.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:31:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:31:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:31:44.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:31:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.210435661000005, 2023-01-15T14:31:44.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:31:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:31:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:31:44.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:31:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:31:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:31:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:31:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:31:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:31:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:31:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:31:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:31:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:31:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:31:44.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:31:44.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:31:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:31:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:44.936 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.210333019999993, 2023-01-15T14:31:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:31:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:31:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:31:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:31:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:31:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:31:44.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:31:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:31:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:31:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:31:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:31:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:31:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:31:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:31:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:31:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:31:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:31:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:44.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:31:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:31:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:31:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:31:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:31:44.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:31:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.609351025000002, 2023-01-15T14:31:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:31:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:44.944 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:31:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:31:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:31:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:31:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:31:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:44.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:44.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:31:45.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:31:45.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:31:45.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:31:45.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:45.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:31:45.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:31:45.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:31:45.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:31:45.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:31:45.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:31:45.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:31:45.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:45.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:31:45.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:31:45.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:31:45.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:31:45.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:31:45.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:31:45.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:45.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:31:45.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:31:45.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:31:45.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:31:45.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:31:45.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:31:45.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:31:45.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:31:45.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:31:45.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:31:45.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:31:45.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:31:45.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:31:45.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:31:45.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:31:45.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:31:45.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:31:45.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:31:45.092 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.092 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.092 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.092 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:45.092 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:45.092 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.609325601000002, 2023-01-15T14:31:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:31:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:31:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:31:45.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:31:45.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:31:45.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:31:45.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:31:45.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:45.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:31:45.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:31:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:31:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:31:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:31:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:45.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:31:45.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:31:45.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:31:45.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:31:45.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:31:45.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.103 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:45.104 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:31:45.105 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:43.571+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:45.105 INFO:tasks.rados.rados.0.smithi090.stdout:869: expect (ObjNum 242 snap 55 seq_num 242) 2023-01-15T14:31:45.105 INFO:tasks.rados.rados.0.smithi090.stdout:872: finishing write tid 2 to smithi09025264-50 2023-01-15T14:31:45.105 INFO:tasks.rados.rados.0.smithi090.stdout:865: done (15 left) 2023-01-15T14:31:45.106 INFO:tasks.rados.rados.0.smithi090.stdout:868: done (14 left) 2023-01-15T14:31:45.106 INFO:tasks.rados.rados.0.smithi090.stdout:869: done (13 left) 2023-01-15T14:31:45.106 INFO:tasks.rados.rados.0.smithi090.stdout:875: done (12 left) 2023-01-15T14:31:45.106 INFO:tasks.rados.rados.0.smithi090.stdout:879: write (excl) oid 49 current snap is 87 2023-01-15T14:31:45.106 INFO:tasks.rados.rados.0.smithi090.stdout:872: finishing write tid 3 to smithi09025264-50 2023-01-15T14:31:45.106 INFO:tasks.rados.rados.0.smithi090.stdout:861: expect (ObjNum 269 snap 66 seq_num 269) 2023-01-15T14:31:45.106 INFO:tasks.rados.rados.0.smithi090.stdout:877: expect (ObjNum 316 snap 83 seq_num 316) 2023-01-15T14:31:45.107 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:31:45.112 INFO:tasks.rados.rados.0.smithi090.stdout:872: finishing write tid 4 to smithi09025264-50 2023-01-15T14:31:45.113 INFO:tasks.rados.rados.0.smithi090.stdout:872: finishing write tid 5 to smithi09025264-50 2023-01-15T14:31:45.113 INFO:tasks.rados.rados.0.smithi090.stdout:872: oid 50 updating version 0 to 186 2023-01-15T14:31:45.113 INFO:tasks.rados.rados.0.smithi090.stdout:872: oid 50 version 186 is already newer than 184 2023-01-15T14:31:45.113 INFO:tasks.rados.rados.0.smithi090.stdout:872: oid 50 version 186 is already newer than 185 2023-01-15T14:31:45.113 INFO:tasks.rados.rados.0.smithi090.stdout:872: oid 50 updating version 186 to 187 2023-01-15T14:31:45.113 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 50 v 187 (ObjNum 328 snap 87 seq_num 328) dirty exists 2023-01-15T14:31:45.113 INFO:tasks.rados.rados.0.smithi090.stdout:872: left oid 50 (ObjNum 328 snap 87 seq_num 328) 2023-01-15T14:31:45.113 INFO:tasks.rados.rados.0.smithi090.stdout:866: finishing copy_from to smithi09025264-21 2023-01-15T14:31:45.113 INFO:tasks.rados.rados.0.smithi090.stdout:866: got expected ENOENT (src dne) 2023-01-15T14:31:45.114 INFO:tasks.rados.rados.0.smithi090.stdout:866: finishing copy_from racing read to smithi09025264-21 2023-01-15T14:31:45.114 INFO:tasks.rados.rados.0.smithi090.stdout:863: finishing rollback tid 1 to smithi09025264-15 2023-01-15T14:31:45.114 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 0 (ObjNum 33 snap 0 seq_num 3182663392) dirty dne 2023-01-15T14:31:45.114 INFO:tasks.rados.rados.0.smithi090.stdout:867: finishing rollback tid 0 to smithi09025264-31 2023-01-15T14:31:45.114 INFO:tasks.rados.rados.0.smithi090.stdout:867: finishing rollback tid 1 to smithi09025264-31 2023-01-15T14:31:45.114 INFO:tasks.rados.rados.0.smithi090.stdout:867: finishing rollback tid 2 to smithi09025264-31 2023-01-15T14:31:45.114 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 275 (ObjNum 277 snap 70 seq_num 277) dirty exists 2023-01-15T14:31:45.114 INFO:tasks.rados.rados.0.smithi090.stdout:871: finishing write tid 1 to smithi09025264-33 2023-01-15T14:31:45.114 INFO:tasks.rados.rados.0.smithi090.stdout:874: expect (ObjNum 287 snap 74 seq_num 287) 2023-01-15T14:31:45.114 INFO:tasks.rados.rados.0.smithi090.stdout:873: finishing copy_from to smithi09025264-14 2023-01-15T14:31:45.115 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 244 (ObjNum 324 snap 86 seq_num 324) dirty exists 2023-01-15T14:31:45.115 INFO:tasks.rados.rados.0.smithi090.stdout:873: finishing copy_from racing read to smithi09025264-14 2023-01-15T14:31:45.115 INFO:tasks.rados.rados.0.smithi090.stdout:871: finishing write tid 2 to smithi09025264-33 2023-01-15T14:31:45.115 INFO:tasks.rados.rados.0.smithi090.stdout:871: finishing write tid 3 to smithi09025264-33 2023-01-15T14:31:45.115 INFO:tasks.rados.rados.0.smithi090.stdout:871: finishing write tid 4 to smithi09025264-33 2023-01-15T14:31:45.115 INFO:tasks.rados.rados.0.smithi090.stdout:871: finishing write tid 5 to smithi09025264-33 2023-01-15T14:31:45.115 INFO:tasks.rados.rados.0.smithi090.stdout:871: oid 33 updating version 0 to 85 2023-01-15T14:31:45.115 INFO:tasks.rados.rados.0.smithi090.stdout:871: oid 33 updating version 85 to 86 2023-01-15T14:31:45.115 INFO:tasks.rados.rados.0.smithi090.stdout:871: oid 33 updating version 86 to 87 2023-01-15T14:31:45.116 INFO:tasks.rados.rados.0.smithi090.stdout:871: oid 33 updating version 87 to 88 2023-01-15T14:31:45.116 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 88 (ObjNum 327 snap 87 seq_num 327) dirty exists 2023-01-15T14:31:45.116 INFO:tasks.rados.rados.0.smithi090.stdout:871: left oid 33 (ObjNum 327 snap 87 seq_num 327) 2023-01-15T14:31:45.116 INFO:tasks.rados.rados.0.smithi090.stdout:879: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2023-01-15T14:31:45.116 INFO:tasks.rados.rados.0.smithi090.stdout:879: writing smithi09025264-49 from 729503 to 1289603 tid 1 2023-01-15T14:31:45.116 INFO:tasks.rados.rados.0.smithi090.stdout:879: writing smithi09025264-49 from 2019164 to 2735596 tid 2 2023-01-15T14:31:45.116 INFO:tasks.rados.rados.0.smithi090.stdout:879: writing smithi09025264-49 from 3335341 to 3529503 tid 3 2023-01-15T14:31:45.116 INFO:tasks.rados.rados.0.smithi090.stdout:861: done (12 left) 2023-01-15T14:31:45.116 INFO:tasks.rados.rados.0.smithi090.stdout:863: done (11 left) 2023-01-15T14:31:45.117 INFO:tasks.rados.rados.0.smithi090.stdout:866: done (10 left) 2023-01-15T14:31:45.117 INFO:tasks.rados.rados.0.smithi090.stdout:867: done (9 left) 2023-01-15T14:31:45.117 INFO:tasks.rados.rados.0.smithi090.stdout:871: done (8 left) 2023-01-15T14:31:45.117 INFO:tasks.rados.rados.0.smithi090.stdout:872: done (7 left) 2023-01-15T14:31:45.117 INFO:tasks.rados.rados.0.smithi090.stdout:873: done (6 left) 2023-01-15T14:31:45.117 INFO:tasks.rados.rados.0.smithi090.stdout:874: done (5 left) 2023-01-15T14:31:45.117 INFO:tasks.rados.rados.0.smithi090.stdout:876: done (4 left) 2023-01-15T14:31:45.117 INFO:tasks.rados.rados.0.smithi090.stdout:877: done (3 left) 2023-01-15T14:31:45.117 INFO:tasks.rados.rados.0.smithi090.stdout:878: done (2 left) 2023-01-15T14:31:45.117 INFO:tasks.rados.rados.0.smithi090.stdout:880: write oid 7 current snap is 87 2023-01-15T14:31:45.118 INFO:tasks.rados.rados.0.smithi090.stdout:880: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2023-01-15T14:31:45.118 INFO:tasks.rados.rados.0.smithi090.stdout:880: writing smithi09025264-7 from 746310 to 1381659 tid 1 2023-01-15T14:31:45.118 INFO:tasks.rados.rados.0.smithi090.stdout:880: writing smithi09025264-7 from 2077646 to 2654089 tid 2 2023-01-15T14:31:45.118 INFO:tasks.rados.rados.0.smithi090.stdout:880: writing smithi09025264-7 from 3362764 to 3546310 tid 3 2023-01-15T14:31:45.118 INFO:tasks.rados.rados.0.smithi090.stdout:881: copy_from oid 10 from oid 38 current snap is 87 2023-01-15T14:31:45.118 INFO:tasks.rados.rados.0.smithi090.stdout:870: finishing write tid 1 to smithi09025264-30 2023-01-15T14:31:45.118 INFO:tasks.rados.rados.0.smithi090.stdout:880: finishing write tid 1 to smithi09025264-7 2023-01-15T14:31:45.118 INFO:tasks.rados.rados.0.smithi090.stdout:882: copy_from oid 8 from oid 23 current snap is 87 2023-01-15T14:31:45.118 INFO:tasks.rados.rados.0.smithi090.stdout:883: snap_remove snap 71 2023-01-15T14:31:45.118 INFO:tasks.rados.rados.0.smithi090.stdout:883: done (5 left) 2023-01-15T14:31:45.119 INFO:tasks.rados.rados.0.smithi090.stdout:884: read oid 9 snap -1 2023-01-15T14:31:45.119 INFO:tasks.rados.rados.0.smithi090.stdout:884: expect deleted 2023-01-15T14:31:45.119 INFO:tasks.rados.rados.0.smithi090.stdout:885: copy_from oid 20 from oid 14 current snap is 87 2023-01-15T14:31:45.119 INFO:tasks.rados.rados.0.smithi090.stdout:870: finishing write tid 2 to smithi09025264-30 2023-01-15T14:31:45.119 INFO:tasks.rados.rados.0.smithi090.stdout:880: finishing write tid 2 to smithi09025264-7 2023-01-15T14:31:45.119 INFO:tasks.rados.rados.0.smithi090.stdout:870: finishing write tid 3 to smithi09025264-30 2023-01-15T14:31:45.119 INFO:tasks.rados.rados.0.smithi090.stdout:870: finishing write tid 4 to smithi09025264-30 2023-01-15T14:31:45.119 INFO:tasks.rados.rados.0.smithi090.stdout:870: finishing write tid 5 to smithi09025264-30 2023-01-15T14:31:45.119 INFO:tasks.rados.rados.0.smithi090.stdout:870: oid 30 updating version 0 to 94 2023-01-15T14:31:45.120 INFO:tasks.rados.rados.0.smithi090.stdout:870: oid 30 updating version 94 to 96 2023-01-15T14:31:45.120 INFO:tasks.rados.rados.0.smithi090.stdout:870: oid 30 version 96 is already newer than 93 2023-01-15T14:31:45.120 INFO:tasks.rados.rados.0.smithi090.stdout:870: oid 30 version 96 is already newer than 95 2023-01-15T14:31:45.120 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 96 (ObjNum 326 snap 87 seq_num 326) dirty exists 2023-01-15T14:31:45.120 INFO:tasks.rados.rados.0.smithi090.stdout:870: left oid 30 (ObjNum 326 snap 87 seq_num 326) 2023-01-15T14:31:45.120 INFO:tasks.rados.rados.0.smithi090.stdout:879: finishing write tid 1 to smithi09025264-49 2023-01-15T14:31:45.120 INFO:tasks.rados.rados.0.smithi090.stdout:881: finishing copy_from to smithi09025264-10 2023-01-15T14:31:45.120 INFO:tasks.rados.rados.0.smithi090.stdout:881: got expected ENOENT (src dne) 2023-01-15T14:31:45.120 INFO:tasks.rados.rados.0.smithi090.stdout:882: finishing copy_from to smithi09025264-8 2023-01-15T14:31:45.120 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 73 (ObjNum 206 snap 44 seq_num 206) dirty exists 2023-01-15T14:31:45.121 INFO:tasks.rados.rados.0.smithi090.stdout:882: finishing copy_from racing read to smithi09025264-8 2023-01-15T14:31:45.121 INFO:tasks.rados.rados.0.smithi090.stdout:879: finishing write tid 2 to smithi09025264-49 2023-01-15T14:31:45.121 INFO:tasks.rados.rados.0.smithi090.stdout:870: done (6 left) 2023-01-15T14:31:45.121 INFO:tasks.rados.rados.0.smithi090.stdout:882: done (5 left) 2023-01-15T14:31:45.121 INFO:tasks.rados.rados.0.smithi090.stdout:886: rollback oid 41 current snap is 87 2023-01-15T14:31:45.121 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 41 to 62 2023-01-15T14:31:45.121 INFO:tasks.rados.rados.0.smithi090.stdout:879: finishing write tid 3 to smithi09025264-49 2023-01-15T14:31:45.121 INFO:tasks.rados.rados.0.smithi090.stdout:879: finishing write tid 4 to smithi09025264-49 2023-01-15T14:31:45.122 INFO:tasks.rados.rados.0.smithi090.stdout:881: finishing copy_from racing read to smithi09025264-10 2023-01-15T14:31:45.122 INFO:tasks.rados.rados.0.smithi090.stdout:880: finishing write tid 3 to smithi09025264-7 2023-01-15T14:31:45.122 INFO:tasks.rados.rados.0.smithi090.stdout:880: finishing write tid 4 to smithi09025264-7 2023-01-15T14:31:45.122 INFO:tasks.rados.rados.0.smithi090.stdout:880: finishing write tid 5 to smithi09025264-7 2023-01-15T14:31:45.122 INFO:tasks.rados.rados.0.smithi090.stdout:880: oid 7 updating version 0 to 190 2023-01-15T14:31:45.122 INFO:tasks.rados.rados.0.smithi090.stdout:880: oid 7 updating version 190 to 191 2023-01-15T14:31:45.122 INFO:tasks.rados.rados.0.smithi090.stdout:880: oid 7 version 191 is already newer than 189 2023-01-15T14:31:45.122 INFO:tasks.rados.rados.0.smithi090.stdout:880: oid 7 version 191 is already newer than 188 2023-01-15T14:31:45.123 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 191 (ObjNum 330 snap 87 seq_num 330) dirty exists 2023-01-15T14:31:45.123 INFO:tasks.rados.rados.0.smithi090.stdout:880: left oid 7 (ObjNum 330 snap 87 seq_num 330) 2023-01-15T14:31:45.123 INFO:tasks.rados.rados.0.smithi090.stdout:879: finishing write tid 5 to smithi09025264-49 2023-01-15T14:31:45.123 INFO:tasks.rados.rados.0.smithi090.stdout:879: oid 49 updating version 0 to 241 2023-01-15T14:31:45.123 INFO:tasks.rados.rados.0.smithi090.stdout:879: oid 49 updating version 241 to 242 2023-01-15T14:31:45.123 INFO:tasks.rados.rados.0.smithi090.stdout:879: oid 49 updating version 242 to 243 2023-01-15T14:31:45.123 INFO:tasks.rados.rados.0.smithi090.stdout:879: oid 49 updating version 243 to 244 2023-01-15T14:31:45.123 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 244 (ObjNum 329 snap 87 seq_num 329) dirty exists 2023-01-15T14:31:45.123 INFO:tasks.rados.rados.0.smithi090.stdout:879: left oid 49 (ObjNum 329 snap 87 seq_num 329) 2023-01-15T14:31:45.124 INFO:tasks.rados.rados.0.smithi090.stdout:879: done (5 left) 2023-01-15T14:31:45.124 INFO:tasks.rados.rados.0.smithi090.stdout:880: done (4 left) 2023-01-15T14:31:45.124 INFO:tasks.rados.rados.0.smithi090.stdout:881: done (3 left) 2023-01-15T14:31:45.124 INFO:tasks.rados.rados.0.smithi090.stdout:887: read oid 11 snap -1 2023-01-15T14:31:45.124 INFO:tasks.rados.rados.0.smithi090.stdout:887: expect (ObjNum 262 snap 61 seq_num 262) 2023-01-15T14:31:45.124 INFO:tasks.rados.rados.0.smithi090.stdout:888: snap_create 2023-01-15T14:31:45.124 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:31:45.124 INFO:tasks.rados.rados.0.smithi090.stdout:885: finishing copy_from to smithi09025264-20 2023-01-15T14:31:45.124 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 244 (ObjNum 324 snap 86 seq_num 324) dirty exists 2023-01-15T14:31:45.125 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:45.103+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:45.129 INFO:tasks.rados.rados.0.smithi090.stdout:884: done (3 left) 2023-01-15T14:31:45.242 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:31:45.243 INFO:tasks.rados.rados.0.smithi090.stdout:886: finishing rollback tid 0 to smithi09025264-41 2023-01-15T14:31:45.243 INFO:tasks.rados.rados.0.smithi090.stdout:886: finishing rollback tid 1 to smithi09025264-41 2023-01-15T14:31:45.243 INFO:tasks.rados.rados.0.smithi090.stdout:885: finishing copy_from racing read to smithi09025264-20 2023-01-15T14:31:45.243 INFO:tasks.rados.rados.0.smithi090.stdout:885: done (2 left) 2023-01-15T14:31:45.244 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:45.244 INFO:tasks.rados.rados.0.smithi090.stdout:887: expect (ObjNum 262 snap 61 seq_num 262) 2023-01-15T14:31:45.244 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:45.245 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:45.245 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:45.245 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:31:45.246 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:45.239+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:45.286 INFO:tasks.rados.rados.0.smithi090.stdout:886: finishing rollback tid 2 to smithi09025264-41 2023-01-15T14:31:45.286 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 246 (ObjNum 164 snap 36 seq_num 164) dirty exists 2023-01-15T14:31:45.286 INFO:tasks.rados.rados.0.smithi090.stdout:886: done (1 left) 2023-01-15T14:31:45.287 INFO:tasks.rados.rados.0.smithi090.stdout:887: done (0 left) 2023-01-15T14:31:45.345 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:45.342+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:45.392 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:45.404 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:45.405 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:45.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:31:45.427 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:31:45.445 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:45.442+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:45.516 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:38 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'3, mlcod=0'0)", 2023-01-15T14:31:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 199.39811877599999, 2023-01-15T14:31:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30191175999999997, 2023-01-15T14:31:45.517 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112976+0000", 2023-01-15T14:31:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113186+0000", 2023-01-15T14:31:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020979700000000001 2023-01-15T14:31:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113228+0000", 2023-01-15T14:31:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2150999999999997e-05 2023-01-15T14:31:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213313+0000", 2023-01-15T14:31:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100084943 2023-01-15T14:31:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313517+0000", 2023-01-15T14:31:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10020435699999999 2023-01-15T14:31:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313576+0000", 2023-01-15T14:31:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783999999999998e-05 2023-01-15T14:31:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313976+0000", 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039960499999999998 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314004+0000", 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7954e-05 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314175+0000", 2023-01-15T14:31:45.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171557 2023-01-15T14:31:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414853+0000", 2023-01-15T14:31:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100677875 2023-01-15T14:31:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414890+0000", 2023-01-15T14:31:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6803e-05 2023-01-15T14:31:45.523 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:125 3.0 e70/55 3:0b54190a:::smithi09025264-17:25 v 70'136, mlcod=68'130)", 2023-01-15T14:31:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 149.568136751, 2023-01-15T14:31:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30171926799999998, 2023-01-15T14:31:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.524 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942958+0000", 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942972+0000", 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3386e-05 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942990+0000", 2023-01-15T14:31:45.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8113000000000001e-05 2023-01-15T14:31:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.943002+0000", 2023-01-15T14:31:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:31:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141942+0000", 2023-01-15T14:31:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19893962300000001 2023-01-15T14:31:45.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141978+0000", 2023-01-15T14:31:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6383e-05 2023-01-15T14:31:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142670+0000", 2023-01-15T14:31:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069211900000000005 2023-01-15T14:31:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142688+0000", 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7839999999999999e-05 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142768+0000", 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0210999999999995e-05 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244581+0000", 2023-01-15T14:31:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101813406 2023-01-15T14:31:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244680+0000", 2023-01-15T14:31:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8045000000000001e-05 2023-01-15T14:31:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:31:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 119.452394363, 2023-01-15T14:31:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:31:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:45.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:31:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:45.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:31:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:31:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:31:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:31:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:31:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:31:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:31:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:31:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:31:45.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:31:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:31:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:31:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.537 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.616636972999999, 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:31:45.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:31:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:31:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:31:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:31:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:31:45.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:31:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:31:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:31:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:31:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:31:45.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:31:45.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.616586029999993, 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:45.547 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:31:45.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:31:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:31:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:31:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:31:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:31:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:31:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:31:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:31:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:31:45.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:31:45.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.556 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:31:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 67.675686788999997, 2023-01-15T14:31:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:31:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:31:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:31:45.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:31:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:31:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:31:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:31:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:31:45.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:31:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:31:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:31:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:31:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:45.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:31:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:31:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:31:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:31:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:31:45.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:31:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:31:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:31:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:31:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:31:45.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.601912616000007, 2023-01-15T14:31:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:31:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:31:45.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:31:45.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:31:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:31:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:31:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:31:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:31:45.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:31:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:31:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:31:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:31:45.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:31:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:31:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:31:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:31:45.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:31:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:31:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:31:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:31:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:31:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:31:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:31:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:31:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1712 3.4 3:2022fdfa:::smithi09025264-46:head [write 1880802~421790] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.596773924999994, 2023-01-15T14:31:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30396041800000001, 2023-01-15T14:31:45.577 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1712 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914322+0000", 2023-01-15T14:31:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:45.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914561+0000", 2023-01-15T14:31:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023884700000000001 2023-01-15T14:31:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914572+0000", 2023-01-15T14:31:45.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0499999999999999e-05 2023-01-15T14:31:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914576+0000", 2023-01-15T14:31:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6820000000000002e-06 2023-01-15T14:31:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014941+0000", 2023-01-15T14:31:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100364994 2023-01-15T14:31:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014959+0000", 2023-01-15T14:31:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7416000000000001e-05 2023-01-15T14:31:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018171+0000", 2023-01-15T14:31:45.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032121239999999998 2023-01-15T14:31:45.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018333+0000", 2023-01-15T14:31:45.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016180900000000001 2023-01-15T14:31:45.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:45.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018581+0000", 2023-01-15T14:31:45.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024773699999999999 2023-01-15T14:31:45.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018897+0000", 2023-01-15T14:31:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031622099999999999 2023-01-15T14:31:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018907+0000", 2023-01-15T14:31:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0061000000000001e-05 2023-01-15T14:31:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.019108+0000", 2023-01-15T14:31:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020080099999999999 2023-01-15T14:31:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.020057+0000", 2023-01-15T14:31:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094954499999999999 2023-01-15T14:31:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150472+0000", 2023-01-15T14:31:45.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.130414958 2023-01-15T14:31:45.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218188+0000", 2023-01-15T14:31:45.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067715656999999999 2023-01-15T14:31:45.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218242+0000", 2023-01-15T14:31:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4651999999999998e-05 2023-01-15T14:31:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218284+0000", 2023-01-15T14:31:45.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1084999999999997e-05 2023-01-15T14:31:45.591 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.591 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.591 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.592 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.18 157/147 [PushOp(3:1eaad226:::smithi09025264-49:69, version: 139'201, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:1eaad226:::smithi09025264-49:69@139'201, size: 3916031, copy_subset: [0~3916031], clone_subset: {}, snapset: 69=[]:{68=[67],69=[69]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:31:45.592 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:45.592 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 60.801863373000003, 2023-01-15T14:31:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27553999000000001, 2023-01-15T14:31:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709232+0000", 2023-01-15T14:31:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709949+0000", 2023-01-15T14:31:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071723000000000002 2023-01-15T14:31:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709990+0000", 2023-01-15T14:31:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.015e-05 2023-01-15T14:31:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.810156+0000", 2023-01-15T14:31:45.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10016643 2023-01-15T14:31:45.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984513+0000", 2023-01-15T14:31:45.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.174357128 2023-01-15T14:31:45.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984522+0000", 2023-01-15T14:31:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0350000000000007e-06 2023-01-15T14:31:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984774+0000", 2023-01-15T14:31:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025144700000000002 2023-01-15T14:31:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.707957196000002, 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:31:45.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:31:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:31:45.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:31:45.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:31:45.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:31:45.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:31:45.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:31:45.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:31:45.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:31:45.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:45.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:31:45.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:31:45.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:31:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:31:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:31:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:31:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:31:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:31:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1820 3.1f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.607655711, 2023-01-15T14:31:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306504938, 2023-01-15T14:31:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1820 2023-01-15T14:31:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:45.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903440+0000", 2023-01-15T14:31:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903448+0000", 2023-01-15T14:31:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9430000000000006e-06 2023-01-15T14:31:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903469+0000", 2023-01-15T14:31:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0305e-05 2023-01-15T14:31:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903481+0000", 2023-01-15T14:31:45.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903616+0000", 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013477199999999999 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903643+0000", 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7098999999999999e-05 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209081+0000", 2023-01-15T14:31:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30543790300000001 2023-01-15T14:31:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209213+0000", 2023-01-15T14:31:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013155400000000001 2023-01-15T14:31:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209600+0000", 2023-01-15T14:31:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038751699999999998 2023-01-15T14:31:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209946+0000", 2023-01-15T14:31:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034601 2023-01-15T14:31:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.824550514999999, 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.617 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:31:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:31:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:31:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:31:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:31:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:31:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:31:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:31:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:31:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:31:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:31:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:31:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:31:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:31:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:31:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:31:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:31:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:31:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:31:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:31:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:31:45.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:31:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:31:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:31:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:31:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:31:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:31:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:31:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:31:45.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:31:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:31:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:31:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:31:45.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:31:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:31:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:31:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.630 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:31:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.821143392, 2023-01-15T14:31:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:31:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:31:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.631 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:31:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:31:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:31:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:31:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:31:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:31:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:31:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:31:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:31:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:31:45.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:31:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:31:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:31:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:45.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:31:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:31:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:31:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:31:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:31:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:31:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:31:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:31:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:31:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:31:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:31:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:31:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:31:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:31:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:31:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:31:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:31:45.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:31:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:45.643 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:31:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.820927786999999, 2023-01-15T14:31:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:31:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:31:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.644 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:31:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:31:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:31:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:31:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:31:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:31:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:31:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:31:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:31:45.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:31:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:31:45.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:31:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:31:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:31:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:31:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:31:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:31:45.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:31:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:31:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:31:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:31:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:31:45.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:31:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:31:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:31:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:31:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:31:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:31:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:31:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:31:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:31:45.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:31:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:31:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:31:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:31:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:31:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.820820713, 2023-01-15T14:31:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:31:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.658 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:31:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:31:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:31:45.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:31:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:31:45.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:31:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:31:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:31:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:31:45.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:31:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:31:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:31:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:31:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:31:45.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:31:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:31:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:31:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:31:45.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:31:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:31:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:31:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:31:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:31:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:31:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:31:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:31:45.665 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.690581663, 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:31:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:31:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:31:45.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:31:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:31:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:31:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:31:45.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:31:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:31:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:31:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:31:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:31:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:31:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:31:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:31:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:31:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:31:45.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:31:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:31:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:31:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:31:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:31:45.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:31:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:31:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:31:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:31:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:31:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:31:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:31:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:31:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.690477125999999, 2023-01-15T14:31:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:31:45.675 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:31:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:31:45.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:31:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:31:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:45.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:31:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:31:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:31:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:31:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:31:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:31:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:31:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:31:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:31:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:31:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:31:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:31:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:31:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:31:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:31:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:31:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:31:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:31:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:31:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:31:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:31:45.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:31:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:31:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:31:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:31:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:31:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:31:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:31:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:31:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:31:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.690407195000001, 2023-01-15T14:31:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:31:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:45.720 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:31:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:31:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:31:45.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:31:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:31:45.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:31:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:31:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:31:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:31:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:31:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:31:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:31:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:31:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:45.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:31:45.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:31:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:31:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:31:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:31:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:31:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:31:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:31:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:31:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2284 3.10 e216/207 3:094fbf13:::smithi09025264-31:head v 216'274, mlcod=212'269)", 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.3448421530000001, 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28055179600000002, 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:31:45.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166254+0000", 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166261+0000", 2023-01-15T14:31:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4080000000000003e-06 2023-01-15T14:31:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166273+0000", 2023-01-15T14:31:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.221e-05 2023-01-15T14:31:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166279+0000", 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8170000000000004e-06 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365575+0000", 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199295791 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365598+0000", 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3913e-05 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365737+0000", 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138312 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365763+0000", 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6596000000000001e-05 2023-01-15T14:31:45.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365803+0000", 2023-01-15T14:31:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9544e-05 2023-01-15T14:31:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365842+0000", 2023-01-15T14:31:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9050000000000001e-05 2023-01-15T14:31:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.446807+0000", 2023-01-15T14:31:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.080964750000000002 2023-01-15T14:31:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2285 3.10 e216/207 3:094fbf13:::smithi09025264-31:head v 216'275, mlcod=212'269)", 2023-01-15T14:31:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:31:45.736 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.344294122, 2023-01-15T14:31:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28017387399999999, 2023-01-15T14:31:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:31:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166802+0000", 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166810+0000", 2023-01-15T14:31:45.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6650000000000003e-06 2023-01-15T14:31:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166821+0000", 2023-01-15T14:31:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1129e-05 2023-01-15T14:31:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166827+0000", 2023-01-15T14:31:45.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4640000000000003e-06 2023-01-15T14:31:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365957+0000", 2023-01-15T14:31:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199129158 2023-01-15T14:31:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:45.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366005+0000", 2023-01-15T14:31:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8822000000000001e-05 2023-01-15T14:31:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366301+0000", 2023-01-15T14:31:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029544199999999998 2023-01-15T14:31:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:45.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366316+0000", 2023-01-15T14:31:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5078e-05 2023-01-15T14:31:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366350+0000", 2023-01-15T14:31:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3575000000000003e-05 2023-01-15T14:31:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366368+0000", 2023-01-15T14:31:45.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8267e-05 2023-01-15T14:31:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.446977+0000", 2023-01-15T14:31:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.080608964000000005 2023-01-15T14:31:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:45.743 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:45.743 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:45.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:31:45.746 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:45.746 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:45.746 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:45.746 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:45.746 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:45.747 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:45.747 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:45.747 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:45.748 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:31:45.748 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:31:45.748 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:31:45.748 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:31:45.748 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:31:45.748 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:31:45.748 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:31:45.748 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:31:45.749 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:45.742+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:45.846 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:45.842+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:45.905 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:45.905 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:45.915 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:31:45.947 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:45.943+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:46.045 INFO:tasks.rados.rados.0.smithi090.stdout:888: done (0 left) 2023-01-15T14:31:46.045 INFO:tasks.rados.rados.0.smithi090.stdout:889: rollback oid 15 current snap is 88 2023-01-15T14:31:46.045 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 15 to 82 2023-01-15T14:31:46.046 INFO:tasks.rados.rados.0.smithi090.stdout:890: delete oid 24 current snap is 88 2023-01-15T14:31:46.047 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:46.047 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:46.048 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:31:46.075 INFO:tasks.rados.rados.0.smithi090.stdout:889: finishing rollback tid 1 to smithi09025264-15 2023-01-15T14:31:46.075 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 0 (ObjNum 561 snap 0 seq_num 3844009472) dirty dne 2023-01-15T14:31:46.076 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:46.071+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:46.150 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:46.147+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:46.228 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:46.229 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:46.229 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:31:46.252 INFO:tasks.rados.rados.0.smithi090.stdout:889: done (1 left) 2023-01-15T14:31:46.252 INFO:tasks.rados.rados.0.smithi090.stdout:890: done (0 left) 2023-01-15T14:31:46.252 INFO:tasks.rados.rados.0.smithi090.stdout:891: write oid 11 current snap is 88 2023-01-15T14:31:46.252 INFO:tasks.rados.rados.0.smithi090.stdout:891: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2023-01-15T14:31:46.263 INFO:tasks.rados.rados.0.smithi090.stdout:891: writing smithi09025264-11 from 779924 to 1565771 tid 1 2023-01-15T14:31:46.269 INFO:tasks.rados.rados.0.smithi090.stdout:891: writing smithi09025264-11 from 2078257 to 2658369 tid 2 2023-01-15T14:31:46.273 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:46.270+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:46.274 INFO:tasks.rados.rados.0.smithi090.stdout:891: writing smithi09025264-11 from 3301257 to 3579924 tid 3 2023-01-15T14:31:46.274 INFO:tasks.rados.rados.0.smithi090.stdout:892: read oid 44 snap -1 2023-01-15T14:31:46.274 INFO:tasks.rados.rados.0.smithi090.stdout:892: expect (ObjNum 273 snap 70 seq_num 273) 2023-01-15T14:31:46.274 INFO:tasks.rados.rados.0.smithi090.stdout:893: write (excl) oid 49 current snap is 88 2023-01-15T14:31:46.275 INFO:tasks.rados.rados.0.smithi090.stdout:893: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2023-01-15T14:31:46.281 INFO:tasks.rados.rados.0.smithi090.stdout:893: writing smithi09025264-49 from 796731 to 1257827 tid 1 2023-01-15T14:31:46.369 INFO:tasks.rados.rados.0.smithi090.stdout:893: writing smithi09025264-49 from 1736739 to 2460509 tid 2 2023-01-15T14:31:46.369 INFO:tasks.rados.rados.0.smithi090.stdout:893: writing smithi09025264-49 from 2928680 to 3596731 tid 3 2023-01-15T14:31:46.369 INFO:tasks.rados.rados.0.smithi090.stdout:894: snap_remove snap 76 2023-01-15T14:31:46.370 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:46.370 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:46.370 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:31:46.371 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:46.366+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:46.470 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:46.467+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:46.477 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:46.477 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:46.487 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:31:46.570 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:46.566+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:46.591 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:46.592 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:46.601 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:31:46.672 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:46.667+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:46.752 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:46.752 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:46.762 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:31:46.773 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:46.770+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:46.813 INFO:tasks.rados.rados.0.smithi090.stdout:894: done (3 left) 2023-01-15T14:31:46.813 INFO:tasks.rados.rados.0.smithi090.stdout:895: read oid 18 snap 72 2023-01-15T14:31:46.813 INFO:tasks.rados.rados.0.smithi090.stdout:895: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:46.813 INFO:tasks.rados.rados.0.smithi090.stdout:893: finishing write tid 1 to smithi09025264-49 2023-01-15T14:31:46.813 INFO:tasks.rados.rados.0.smithi090.stdout:892: expect (ObjNum 273 snap 70 seq_num 273) 2023-01-15T14:31:46.861 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:46.862 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:46.871 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:31:46.874 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:46.870+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:46.975 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:46.971+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:46.979 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:46.979 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:46.990 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:31:47.015 INFO:tasks.rados.rados.0.smithi090.stdout:893: finishing write tid 2 to smithi09025264-49 2023-01-15T14:31:47.015 INFO:tasks.rados.rados.0.smithi090.stdout:892: done (3 left) 2023-01-15T14:31:47.016 INFO:tasks.rados.rados.0.smithi090.stdout:896: read oid 45 snap -1 2023-01-15T14:31:47.016 INFO:tasks.rados.rados.0.smithi090.stdout:896: expect (ObjNum 206 snap 44 seq_num 206) 2023-01-15T14:31:47.016 INFO:tasks.rados.rados.0.smithi090.stdout:897: delete oid 32 current snap is 88 2023-01-15T14:31:47.016 INFO:tasks.rados.rados.0.smithi090.stdout:893: finishing write tid 3 to smithi09025264-49 2023-01-15T14:31:47.016 INFO:tasks.rados.rados.0.smithi090.stdout:891: finishing write tid 1 to smithi09025264-11 2023-01-15T14:31:47.016 INFO:tasks.rados.rados.0.smithi090.stdout:893: finishing write tid 4 to smithi09025264-49 2023-01-15T14:31:47.017 INFO:tasks.rados.rados.0.smithi090.stdout:893: finishing write tid 5 to smithi09025264-49 2023-01-15T14:31:47.017 INFO:tasks.rados.rados.0.smithi090.stdout:893: oid 49 updating version 0 to 252 2023-01-15T14:31:47.017 INFO:tasks.rados.rados.0.smithi090.stdout:893: oid 49 updating version 252 to 253 2023-01-15T14:31:47.017 INFO:tasks.rados.rados.0.smithi090.stdout:893: oid 49 version 253 is already newer than 251 2023-01-15T14:31:47.017 INFO:tasks.rados.rados.0.smithi090.stdout:893: oid 49 version 253 is already newer than 250 2023-01-15T14:31:47.017 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 253 (ObjNum 333 snap 88 seq_num 333) dirty exists 2023-01-15T14:31:47.017 INFO:tasks.rados.rados.0.smithi090.stdout:893: left oid 49 (ObjNum 333 snap 88 seq_num 333) 2023-01-15T14:31:47.018 INFO:tasks.rados.rados.0.smithi090.stdout:891: finishing write tid 2 to smithi09025264-11 2023-01-15T14:31:47.018 INFO:tasks.rados.rados.0.smithi090.stdout:891: finishing write tid 3 to smithi09025264-11 2023-01-15T14:31:47.018 INFO:tasks.rados.rados.0.smithi090.stdout:891: finishing write tid 4 to smithi09025264-11 2023-01-15T14:31:47.018 INFO:tasks.rados.rados.0.smithi090.stdout:891: finishing write tid 5 to smithi09025264-11 2023-01-15T14:31:47.018 INFO:tasks.rados.rados.0.smithi090.stdout:891: oid 11 updating version 0 to 275 2023-01-15T14:31:47.018 INFO:tasks.rados.rados.0.smithi090.stdout:891: oid 11 updating version 275 to 276 2023-01-15T14:31:47.018 INFO:tasks.rados.rados.0.smithi090.stdout:891: oid 11 version 276 is already newer than 273 2023-01-15T14:31:47.018 INFO:tasks.rados.rados.0.smithi090.stdout:891: oid 11 version 276 is already newer than 274 2023-01-15T14:31:47.018 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 276 (ObjNum 332 snap 88 seq_num 332) dirty exists 2023-01-15T14:31:47.018 INFO:tasks.rados.rados.0.smithi090.stdout:891: left oid 11 (ObjNum 332 snap 88 seq_num 332) 2023-01-15T14:31:47.076 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:47.071+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:47.158 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:31:47.159 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:47.159 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:47.160 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:31:47.177 INFO:tasks.rados.rados.0.smithi090.stdout:895: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:47.178 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:47.171+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:47.269 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:47.318 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:47.320 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:31:47.320 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:47.315+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:47.386 INFO:tasks.rados.rados.0.smithi090.stdout:896: expect (ObjNum 206 snap 44 seq_num 206) 2023-01-15T14:31:47.480 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:47.481 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:47.481 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:31:47.482 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:47.475+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:47.554 INFO:tasks.rados.rados.0.smithi090.stdout:891: done (4 left) 2023-01-15T14:31:47.644 INFO:tasks.rados.rados.0.smithi090.stdout:893: done (3 left) 2023-01-15T14:31:47.644 INFO:tasks.rados.rados.0.smithi090.stdout:895: done (2 left) 2023-01-15T14:31:47.644 INFO:tasks.rados.rados.0.smithi090.stdout:896: done (1 left) 2023-01-15T14:31:47.644 INFO:tasks.rados.rados.0.smithi090.stdout:897: done (0 left) 2023-01-15T14:31:47.644 INFO:tasks.rados.rados.0.smithi090.stdout:898: copy_from oid 37 from oid 11 current snap is 88 2023-01-15T14:31:47.644 INFO:tasks.rados.rados.0.smithi090.stdout:899: write (excl) oid 25 current snap is 88 2023-01-15T14:31:47.645 INFO:tasks.rados.rados.0.smithi090.stdout:899: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2023-01-15T14:31:47.646 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:47.647 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:47.647 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:31:47.647 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:47.642+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:47.648 INFO:tasks.rados.rados.0.smithi090.stdout:899: writing smithi09025264-25 from 430345 to 1158292 tid 1 2023-01-15T14:31:47.669 INFO:tasks.rados.rados.0.smithi090.stdout:899: writing smithi09025264-25 from 1570056 to 2297495 tid 2 2023-01-15T14:31:47.670 INFO:tasks.rados.rados.0.smithi090.stdout:899: writing smithi09025264-25 from 2699879 to 3169768 tid 3 2023-01-15T14:31:47.670 INFO:tasks.rados.rados.0.smithi090.stdout:900: copy_from oid 48 from oid 27 current snap is 88 2023-01-15T14:31:47.670 INFO:tasks.rados.rados.0.smithi090.stdout:901: rollback oid 46 current snap is 88 2023-01-15T14:31:47.670 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 46 to 62 2023-01-15T14:31:47.670 INFO:tasks.rados.rados.0.smithi090.stdout:902: write oid 45 current snap is 88 2023-01-15T14:31:47.670 INFO:tasks.rados.rados.0.smithi090.stdout:902: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2023-01-15T14:31:47.670 INFO:tasks.rados.rados.0.smithi090.stdout:902: writing smithi09025264-45 from 447152 to 850348 tid 1 2023-01-15T14:31:47.671 INFO:tasks.rados.rados.0.smithi090.stdout:902: writing smithi09025264-45 from 1512185 to 2099635 tid 2 2023-01-15T14:31:47.671 INFO:tasks.rados.rados.0.smithi090.stdout:902: writing smithi09025264-45 from 2727302 to 3408463 tid 3 2023-01-15T14:31:47.671 INFO:tasks.rados.rados.0.smithi090.stdout:903: delete oid 17 current snap is 88 2023-01-15T14:31:47.671 INFO:tasks.rados.rados.0.smithi090.stdout:900: finishing copy_from to smithi09025264-48 2023-01-15T14:31:47.671 INFO:tasks.rados.rados.0.smithi090.stdout:900: got expected ENOENT (src dne) 2023-01-15T14:31:47.671 INFO:tasks.rados.rados.0.smithi090.stdout:900: finishing copy_from racing read to smithi09025264-48 2023-01-15T14:31:47.704 INFO:tasks.rados.rados.0.smithi090.stdout:900: done (5 left) 2023-01-15T14:31:47.801 INFO:tasks.rados.rados.0.smithi090.stdout:903: done (4 left) 2023-01-15T14:31:47.801 INFO:tasks.rados.rados.0.smithi090.stdout:904: snap_remove snap 84 2023-01-15T14:31:47.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:31:47.803 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:47.803 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:47.804 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:31:47.804 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:47.799+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:47.903 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:47.898+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:47.964 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:31:47.964 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:31:47.964 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:31:47.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:31:47.966 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:47.967 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:47.967 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:47.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:31:47.976 INFO:tasks.rados.rados.0.smithi090.stdout:904: done (4 left) 2023-01-15T14:31:47.999 INFO:tasks.rados.rados.0.smithi090.stdout:905: write oid 13 current snap is 88 2023-01-15T14:31:47.999 INFO:tasks.rados.rados.0.smithi090.stdout:905: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2023-01-15T14:31:48.000 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:48.000 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:31:48.001 INFO:tasks.rados.rados.0.smithi090.stdout:905: writing smithi09025264-13 from 480766 to 1034460 tid 1 2023-01-15T14:31:48.001 INFO:tasks.rados.rados.0.smithi090.stdout:905: writing smithi09025264-13 from 1629149 to 2220268 tid 2 2023-01-15T14:31:48.002 INFO:tasks.rados.rados.0.smithi090.stdout:905: writing smithi09025264-13 from 2782148 to 3202206 tid 3 2023-01-15T14:31:48.002 INFO:tasks.rados.rados.0.smithi090.stdout:906: read oid 21 snap 81 2023-01-15T14:31:48.002 INFO:tasks.rados.rados.0.smithi090.stdout:906: expect (ObjNum 262 snap 61 seq_num 262) 2023-01-15T14:31:48.002 INFO:tasks.rados.rados.0.smithi090.stdout:907: snap_remove snap 86 2023-01-15T14:31:48.002 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:48.004 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:47.998+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:48.017 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:48.087 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:48.125 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:48.125 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:48.125 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:48.126 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:31:48.127 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:48.123+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:48.134 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:48.165 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:48.196 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:48.287 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:31:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:31:48.288 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:31:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:31:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:31:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:31:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:31:48.289 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:31:48.291 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:48.291 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:48.292 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:48.292 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:31:48.294 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:48.282+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:48.316 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:31:48.319 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:31:48.333 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2023-01-15T14:31:47.968153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":265163352,"num_objects":131,"num_object_clones":79,"num_object_copies":393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":131,"num_whiteouts":11,"num_read":885,"num_read_kb":841524,"num_write":734,"num_write_kb":222332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":417,"num_bytes_recovered":264419291,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8192,"ondisk_log_size":8192,"up":195,"acting":195,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":195,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":5936648,"kb_used_data":556317468,"kb_used_omap":29390,"kb_used_meta":0,"kb_avail":556259632,"statfs":{"total":575688990720,"available":569609863168,"internally_reserved":59224064,"allocated":569669087232,"data_stored":569669087232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":30096102,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":37,"commit_latency_ns":0,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27012016,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":124821,"num_write":29,"num_write_kb":13869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-21,"num_bytes_recovered":-9754858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.063830"},"pg_stats":[{"pgid":"3.37","version":"200'78","reported_seq":53,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.082921+0000","last_change":"2023-01-15T14:31:47.082921+0000","last_active":"2023-01-15T14:31:47.082921+0000","last_peered":"2023-01-15T14:31:47.082921+0000","last_clean":"2023-01-15T14:31:47.082921+0000","last_became_active":"2023-01-15T14:31:37.024453+0000","last_became_peered":"2023-01-15T14:31:37.024453+0000","last_unstale":"2023-01-15T14:31:47.082921+0000","last_undegraded":"2023-01-15T14:31:47.082921+0000","last_fullsized":"2023-01-15T14:31:47.082921+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:09.695","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.672596313,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.36","version":"194'81","reported_seq":210,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.637148+0000","last_change":"2023-01-15T14:31:45.637148+0000","last_active":"2023-01-15T14:31:45.637148+0000","last_peered":"2023-01-15T14:31:45.637148+0000","last_clean":"2023-01-15T14:31:45.637148+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:31:45.637148+0000","last_undegraded":"2023-01-15T14:31:45.637148+0000","last_fullsized":"2023-01-15T14:31:45.637148+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.629658728,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.35","version":"110'55","reported_seq":628,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:45.123286+0000","last_change":"2023-01-15T14:31:45.123194+0000","last_active":"2023-01-15T14:31:45.123286+0000","last_peered":"2023-01-15T14:31:45.123286+0000","last_clean":"2023-01-15T14:31:45.123286+0000","last_became_active":"2023-01-15T14:31:38.033824+0000","last_became_peered":"2023-01-15T14:31:38.033824+0000","last_unstale":"2023-01-15T14:31:45.123286+0000","last_undegraded":"2023-01-15T14:31:45.123286+0000","last_fullsized":"2023-01-15T14:31:45.123286+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:30:24.721082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.721082+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.109000768,"objects_trimmed":0,"snaptrim_duration":0.58429463999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":800,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.060696+0000","last_change":"2023-01-15T14:31:47.060696+0000","last_active":"2023-01-15T14:31:47.060696+0000","last_peered":"2023-01-15T14:31:47.060696+0000","last_clean":"2023-01-15T14:31:47.060696+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:31:47.060696+0000","last_undegraded":"2023-01-15T14:31:47.060696+0000","last_fullsized":"2023-01-15T14:31:47.060696+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.20345640400000001,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5618,"num_write":3,"num_write_kb":609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.33","version":"204'275","reported_seq":880,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:45.022378+0000","last_change":"2023-01-15T14:31:45.022344+0000","last_active":"2023-01-15T14:31:45.022378+0000","last_peered":"2023-01-15T14:31:45.022378+0000","last_clean":"2023-01-15T14:31:45.022378+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:31:45.022378+0000","last_undegraded":"2023-01-15T14:31:45.022378+0000","last_fullsized":"2023-01-15T14:31:45.022378+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.48632392200000002,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.32","version":"218'153","reported_seq":478,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.426912+0000","last_change":"2023-01-15T14:31:45.426912+0000","last_active":"2023-01-15T14:31:45.426912+0000","last_peered":"2023-01-15T14:31:45.426912+0000","last_clean":"2023-01-15T14:31:45.426912+0000","last_became_active":"2023-01-15T14:31:36.921952+0000","last_became_peered":"2023-01-15T14:31:36.921952+0000","last_unstale":"2023-01-15T14:31:45.426912+0000","last_undegraded":"2023-01-15T14:31:45.426912+0000","last_fullsized":"2023-01-15T14:31:45.426912+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:56.390","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.50615451,"stat_sum":{"num_bytes":2403401,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.13","version":"218'286","reported_seq":924,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.082267+0000","last_change":"2023-01-15T14:31:47.082267+0000","last_active":"2023-01-15T14:31:47.082267+0000","last_peered":"2023-01-15T14:31:47.082267+0000","last_clean":"2023-01-15T14:31:47.082267+0000","last_became_active":"2023-01-15T14:31:35.097414+0000","last_became_peered":"2023-01-15T14:31:35.097414+0000","last_unstale":"2023-01-15T14:31:47.082267+0000","last_undegraded":"2023-01-15T14:31:47.082267+0000","last_fullsized":"2023-01-15T14:31:47.082267+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":286,"log_dups_size":0,"ondisk_log_size":286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:45.688","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.81318309600000005,"stat_sum":{"num_bytes":13941907,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.12","version":"210'147","reported_seq":451,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:44.922338+0000","last_change":"2023-01-15T14:31:44.922284+0000","last_active":"2023-01-15T14:31:44.922338+0000","last_peered":"2023-01-15T14:31:44.922338+0000","last_clean":"2023-01-15T14:31:44.922338+0000","last_became_active":"2023-01-15T14:31:35.028565+0000","last_became_peered":"2023-01-15T14:31:35.028565+0000","last_unstale":"2023-01-15T14:31:44.922338+0000","last_undegraded":"2023-01-15T14:31:44.922338+0000","last_fullsized":"2023-01-15T14:31:44.922338+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"139'103","last_scrub_stamp":"2023-01-15T14:30:30.045615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:30.045615+0000","objects_scrubbed":3,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.207128334,"objects_trimmed":0,"snaptrim_duration":0.39820908100000002,"stat_sum":{"num_bytes":4319415,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":26971,"num_write":7,"num_write_kb":2217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6292614,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.11","version":"220'176","reported_seq":917,"reported_epoch":220,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:31:47.317867+0000","last_change":"2023-01-15T14:31:47.083124+0000","last_active":"2023-01-15T14:31:47.317867+0000","last_peered":"2023-01-15T14:31:47.317867+0000","last_clean":"2023-01-15T14:31:47.317867+0000","last_became_active":"2023-01-15T14:31:35.026209+0000","last_became_peered":"2023-01-15T14:31:35.026209+0000","last_unstale":"2023-01-15T14:31:47.317867+0000","last_undegraded":"2023-01-15T14:31:47.317867+0000","last_fullsized":"2023-01-15T14:31:47.317867+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:06.167","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.57615515100000003,"stat_sum":{"num_bytes":21143338,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":12,"num_read_kb":13794,"num_write":5,"num_write_kb":6747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.10","version":"218'279","reported_seq":1000,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.926020+0000","last_change":"2023-01-15T14:31:45.926020+0000","last_active":"2023-01-15T14:31:45.926020+0000","last_peered":"2023-01-15T14:31:45.926020+0000","last_clean":"2023-01-15T14:31:45.926020+0000","last_became_active":"2023-01-15T14:31:35.135778+0000","last_became_peered":"2023-01-15T14:31:35.135778+0000","last_unstale":"2023-01-15T14:31:45.926020+0000","last_undegraded":"2023-01-15T14:31:45.926020+0000","last_fullsized":"2023-01-15T14:31:45.926020+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":1.0049318389999999,"stat_sum":{"num_bytes":9752724,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":33,"num_read_kb":25949,"num_write":49,"num_write_kb":16843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":13340457,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":863,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.163901+0000","last_change":"2023-01-15T14:31:47.163901+0000","last_active":"2023-01-15T14:31:47.163901+0000","last_peered":"2023-01-15T14:31:47.163901+0000","last_clean":"2023-01-15T14:31:47.163901+0000","last_became_active":"2023-01-15T14:31:35.032038+0000","last_became_peered":"2023-01-15T14:31:35.032038+0000","last_unstale":"2023-01-15T14:31:47.163901+0000","last_undegraded":"2023-01-15T14:31:47.163901+0000","last_fullsized":"2023-01-15T14:31:47.163901+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:31:22.549180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:22.549180+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50470070300000003,"objects_trimmed":0,"snaptrim_duration":0.91546465099999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":653,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.160570+0000","last_change":"2023-01-15T14:31:47.160570+0000","last_active":"2023-01-15T14:31:47.160570+0000","last_peered":"2023-01-15T14:31:47.160570+0000","last_clean":"2023-01-15T14:31:47.160570+0000","last_became_active":"2023-01-15T14:31:35.237663+0000","last_became_peered":"2023-01-15T14:31:35.237663+0000","last_unstale":"2023-01-15T14:31:47.160570+0000","last_undegraded":"2023-01-15T14:31:47.160570+0000","last_fullsized":"2023-01-15T14:31:47.160570+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:29.609173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:29.609173+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.54285148999999999,"objects_trimmed":0,"snaptrim_duration":0.30670275000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.d","version":"218'180","reported_seq":877,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.083091+0000","last_change":"2023-01-15T14:31:47.083091+0000","last_active":"2023-01-15T14:31:47.083091+0000","last_peered":"2023-01-15T14:31:47.083091+0000","last_clean":"2023-01-15T14:31:47.083091+0000","last_became_active":"2023-01-15T14:31:21.132225+0000","last_became_peered":"2023-01-15T14:31:21.132225+0000","last_unstale":"2023-01-15T14:31:47.083091+0000","last_undegraded":"2023-01-15T14:31:47.083091+0000","last_fullsized":"2023-01-15T14:31:47.083091+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:41.817","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.81187748199999998,"stat_sum":{"num_bytes":6067327,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.c","version":"218'53","reported_seq":646,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:45.144335+0000","last_change":"2023-01-15T14:31:44.922527+0000","last_active":"2023-01-15T14:31:45.144335+0000","last_peered":"2023-01-15T14:31:45.144335+0000","last_clean":"2023-01-15T14:31:45.144335+0000","last_became_active":"2023-01-15T14:31:21.227945+0000","last_became_peered":"2023-01-15T14:31:21.227945+0000","last_unstale":"2023-01-15T14:31:45.144335+0000","last_undegraded":"2023-01-15T14:31:45.144335+0000","last_fullsized":"2023-01-15T14:31:45.144335+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:42.336","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.37574644600000001,"stat_sum":{"num_bytes":6857322,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":11,"num_write_kb":6025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":623,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:44.927090+0000","last_change":"2023-01-15T14:31:44.927076+0000","last_active":"2023-01-15T14:31:44.927090+0000","last_peered":"2023-01-15T14:31:44.927090+0000","last_clean":"2023-01-15T14:31:44.927090+0000","last_became_active":"2023-01-15T14:31:21.230392+0000","last_became_peered":"2023-01-15T14:31:21.230392+0000","last_unstale":"2023-01-15T14:31:44.927090+0000","last_undegraded":"2023-01-15T14:31:44.927090+0000","last_fullsized":"2023-01-15T14:31:44.927090+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:38.384","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.378664161,"stat_sum":{"num_bytes":865560,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3347,"num_write":5,"num_write_kb":2817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.a","version":"218'225","reported_seq":909,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.060974+0000","last_change":"2023-01-15T14:31:47.060974+0000","last_active":"2023-01-15T14:31:47.060974+0000","last_peered":"2023-01-15T14:31:47.060974+0000","last_clean":"2023-01-15T14:31:47.060974+0000","last_became_active":"2023-01-15T14:31:21.032574+0000","last_became_peered":"2023-01-15T14:31:21.032574+0000","last_unstale":"2023-01-15T14:31:47.060974+0000","last_undegraded":"2023-01-15T14:31:47.060974+0000","last_fullsized":"2023-01-15T14:31:47.060974+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.71883680699999997,"stat_sum":{"num_bytes":4593327,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":20730,"num_write":36,"num_write_kb":14107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":18895452,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":745,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.173635+0000","last_change":"2023-01-15T14:31:47.173635+0000","last_active":"2023-01-15T14:31:47.173635+0000","last_peered":"2023-01-15T14:31:47.173635+0000","last_clean":"2023-01-15T14:31:47.173635+0000","last_became_active":"2023-01-15T14:31:21.122856+0000","last_became_peered":"2023-01-15T14:31:21.122856+0000","last_unstale":"2023-01-15T14:31:47.173635+0000","last_undegraded":"2023-01-15T14:31:47.173635+0000","last_fullsized":"2023-01-15T14:31:47.173635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:31:44.640345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.640345+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:46.259","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":1.0060140479999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.8","version":"179'204","reported_seq":940,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.928212+0000","last_change":"2023-01-15T14:31:45.928212+0000","last_active":"2023-01-15T14:31:45.928212+0000","last_peered":"2023-01-15T14:31:45.928212+0000","last_clean":"2023-01-15T14:31:45.928212+0000","last_became_active":"2023-01-15T14:31:21.332760+0000","last_became_peered":"2023-01-15T14:31:21.332760+0000","last_unstale":"2023-01-15T14:31:45.928212+0000","last_undegraded":"2023-01-15T14:31:45.928212+0000","last_fullsized":"2023-01-15T14:31:45.928212+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":1.0066607030000001,"stat_sum":{"num_bytes":2159716,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":3981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":7874096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":354,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.021584+0000","last_change":"2023-01-15T14:31:03.091584+0000","last_active":"2023-01-15T14:31:45.021584+0000","last_peered":"2023-01-15T14:31:45.021584+0000","last_clean":"2023-01-15T14:31:45.021584+0000","last_became_active":"2023-01-15T14:31:03.091071+0000","last_became_peered":"2023-01-15T14:31:03.091071+0000","last_unstale":"2023-01-15T14:31:45.021584+0000","last_undegraded":"2023-01-15T14:31:45.021584+0000","last_fullsized":"2023-01-15T14:31:45.021584+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:29:45.876183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:29:45.876183+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":193,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:44.926980+0000","last_change":"2023-01-15T14:31:38.018850+0000","last_active":"2023-01-15T14:31:44.926980+0000","last_peered":"2023-01-15T14:31:44.926980+0000","last_clean":"2023-01-15T14:31:44.926980+0000","last_became_active":"2023-01-15T14:31:38.018145+0000","last_became_peered":"2023-01-15T14:31:38.018145+0000","last_unstale":"2023-01-15T14:31:44.926980+0000","last_undegraded":"2023-01-15T14:31:44.926980+0000","last_fullsized":"2023-01-15T14:31:44.926980+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:43.428","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":633,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.060634+0000","last_change":"2023-01-15T14:31:47.060547+0000","last_active":"2023-01-15T14:31:47.060634+0000","last_peered":"2023-01-15T14:31:47.060634+0000","last_clean":"2023-01-15T14:31:47.060634+0000","last_became_active":"2023-01-15T14:31:44.311861+0000","last_became_peered":"2023-01-15T14:31:44.311861+0000","last_unstale":"2023-01-15T14:31:47.060634+0000","last_undegraded":"2023-01-15T14:31:47.060634+0000","last_fullsized":"2023-01-15T14:31:47.060634+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.205719648,"stat_sum":{"num_bytes":1084068,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3268978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":255,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.971432+0000","last_change":"2023-01-15T14:31:11.136609+0000","last_active":"2023-01-15T14:31:45.971432+0000","last_peered":"2023-01-15T14:31:45.971432+0000","last_clean":"2023-01-15T14:31:45.971432+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:31:45.971432+0000","last_undegraded":"2023-01-15T14:31:45.971432+0000","last_fullsized":"2023-01-15T14:31:45.971432+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:11.136490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.136490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:36.593","scrub_duration":0.39969354600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":702,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.073343+0000","last_change":"2023-01-15T14:31:47.073343+0000","last_active":"2023-01-15T14:31:47.073343+0000","last_peered":"2023-01-15T14:31:47.073343+0000","last_clean":"2023-01-15T14:31:47.073343+0000","last_became_active":"2023-01-15T14:31:06.028849+0000","last_became_peered":"2023-01-15T14:31:06.028849+0000","last_unstale":"2023-01-15T14:31:47.073343+0000","last_undegraded":"2023-01-15T14:31:47.073343+0000","last_fullsized":"2023-01-15T14:31:47.073343+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:31:36.200118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:36.200118+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:45.734","scrub_duration":0.103601373,"objects_trimmed":0,"snaptrim_duration":0.308806372,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:44.927018+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:31:44.927018+0000","last_peered":"2023-01-15T14:31:44.927018+0000","last_clean":"2023-01-15T14:31:44.927018+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:31:44.927018+0000","last_undegraded":"2023-01-15T14:31:44.927018+0000","last_fullsized":"2023-01-15T14:31:44.927018+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:38.960","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"213'253","reported_seq":944,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-01-15T14:31:46.545078+0000","last_change":"2023-01-15T14:31:46.546232+0000","last_active":"2023-01-15T14:31:46.545078+0000","last_peered":"2023-01-15T14:31:46.545078+0000","last_clean":"2023-01-15T14:31:46.545078+0000","last_became_active":"2023-01-15T14:30:56.101261+0000","last_became_peered":"2023-01-15T14:30:56.101261+0000","last_unstale":"2023-01-15T14:31:46.545078+0000","last_undegraded":"2023-01-15T14:31:46.545078+0000","last_fullsized":"2023-01-15T14:31:46.545078+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:31.088","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.31031341099999998,"stat_sum":{"num_bytes":8588443,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":3877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":76,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:44.922391+0000","last_change":"2023-01-15T14:31:13.764052+0000","last_active":"2023-01-15T14:31:44.922391+0000","last_peered":"2023-01-15T14:31:44.922391+0000","last_clean":"2023-01-15T14:31:44.922391+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:31:44.922391+0000","last_undegraded":"2023-01-15T14:31:44.922391+0000","last_fullsized":"2023-01-15T14:31:44.922391+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:13.763945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:13.763945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:39.419","scrub_duration":0.58613919699999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"219'278","reported_seq":1036,"reported_epoch":220,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:31:47.060973+0000","last_change":"2023-01-15T14:31:46.955082+0000","last_active":"2023-01-15T14:31:47.060973+0000","last_peered":"2023-01-15T14:31:47.060973+0000","last_clean":"2023-01-15T14:31:47.060973+0000","last_became_active":"2023-01-15T14:31:03.093753+0000","last_became_peered":"2023-01-15T14:31:03.093753+0000","last_unstale":"2023-01-15T14:31:47.060973+0000","last_undegraded":"2023-01-15T14:31:47.060973+0000","last_fullsized":"2023-01-15T14:31:47.060973+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'251","last_scrub_stamp":"2023-01-15T14:31:15.363400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.363400+0000","objects_scrubbed":3,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:42.073","scrub_duration":0.208136242,"objects_trimmed":0,"snaptrim_duration":0.0051660559999999996,"stat_sum":{"num_bytes":10960091,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":44988,"num_write":16,"num_write_kb":6726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.5","version":"215'97","reported_seq":729,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:45.123964+0000","last_change":"2023-01-15T14:31:45.123883+0000","last_active":"2023-01-15T14:31:45.123964+0000","last_peered":"2023-01-15T14:31:45.123964+0000","last_clean":"2023-01-15T14:31:45.123964+0000","last_became_active":"2023-01-15T14:31:21.132693+0000","last_became_peered":"2023-01-15T14:31:21.132693+0000","last_unstale":"2023-01-15T14:31:45.123964+0000","last_undegraded":"2023-01-15T14:31:45.123964+0000","last_fullsized":"2023-01-15T14:31:45.123964+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:25.367","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.400399964,"stat_sum":{"num_bytes":6308409,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":15022,"num_write":27,"num_write_kb":9886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9491720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":66,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:44.921535+0000","last_change":"2023-01-15T14:30:51.237821+0000","last_active":"2023-01-15T14:31:44.921535+0000","last_peered":"2023-01-15T14:31:44.921535+0000","last_clean":"2023-01-15T14:31:44.921535+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:31:44.921535+0000","last_undegraded":"2023-01-15T14:31:44.921535+0000","last_fullsized":"2023-01-15T14:31:44.921535+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"218'234","reported_seq":981,"reported_epoch":219,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:31:46.253432+0000","last_change":"2023-01-15T14:31:46.131589+0000","last_active":"2023-01-15T14:31:46.253432+0000","last_peered":"2023-01-15T14:31:46.253432+0000","last_clean":"2023-01-15T14:31:46.253432+0000","last_became_active":"2023-01-15T14:31:21.226293+0000","last_became_peered":"2023-01-15T14:31:21.226293+0000","last_unstale":"2023-01-15T14:31:46.253432+0000","last_undegraded":"2023-01-15T14:31:46.253432+0000","last_fullsized":"2023-01-15T14:31:46.253432+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":234,"log_dups_size":0,"ondisk_log_size":234,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.01313098,"stat_sum":{"num_bytes":9621772,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":46338,"num_write":26,"num_write_kb":6428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":15862107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":246,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.971865+0000","last_change":"2023-01-15T14:31:15.088518+0000","last_active":"2023-01-15T14:31:45.971865+0000","last_peered":"2023-01-15T14:31:45.971865+0000","last_clean":"2023-01-15T14:31:45.971865+0000","last_became_active":"2023-01-15T14:31:03.088392+0000","last_became_peered":"2023-01-15T14:31:03.088392+0000","last_unstale":"2023-01-15T14:31:45.971865+0000","last_undegraded":"2023-01-15T14:31:45.971865+0000","last_fullsized":"2023-01-15T14:31:45.971865+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:15.088397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.088397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:32.351","scrub_duration":0.40518630900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"218'98","reported_seq":728,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.063220+0000","last_change":"2023-01-15T14:31:47.063177+0000","last_active":"2023-01-15T14:31:47.063220+0000","last_peered":"2023-01-15T14:31:47.063220+0000","last_clean":"2023-01-15T14:31:47.063220+0000","last_became_active":"2023-01-15T14:31:44.117230+0000","last_became_peered":"2023-01-15T14:31:44.117230+0000","last_unstale":"2023-01-15T14:31:47.063220+0000","last_undegraded":"2023-01-15T14:31:47.063220+0000","last_fullsized":"2023-01-15T14:31:47.063220+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.41566773299999998,"stat_sum":{"num_bytes":6429371,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":8458,"num_write":9,"num_write_kb":2509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1731121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":66,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.009176+0000","last_change":"2023-01-15T14:30:51.035825+0000","last_active":"2023-01-15T14:31:45.009176+0000","last_peered":"2023-01-15T14:31:45.009176+0000","last_clean":"2023-01-15T14:31:45.009176+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:31:45.009176+0000","last_undegraded":"2023-01-15T14:31:45.009176+0000","last_fullsized":"2023-01-15T14:31:45.009176+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":648,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.638894+0000","last_change":"2023-01-15T14:31:45.638894+0000","last_active":"2023-01-15T14:31:45.638894+0000","last_peered":"2023-01-15T14:31:45.638894+0000","last_clean":"2023-01-15T14:31:45.638894+0000","last_became_active":"2023-01-15T14:31:21.030929+0000","last_became_peered":"2023-01-15T14:31:21.030929+0000","last_unstale":"2023-01-15T14:31:45.638894+0000","last_undegraded":"2023-01-15T14:31:45.638894+0000","last_fullsized":"2023-01-15T14:31:45.638894+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:31:25.342830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:25.342830+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:44.671","scrub_duration":0.20886623600000001,"objects_trimmed":0,"snaptrim_duration":0.630460201,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":202,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:44.927735+0000","last_change":"2023-01-15T14:31:38.112378+0000","last_active":"2023-01-15T14:31:44.927735+0000","last_peered":"2023-01-15T14:31:44.927735+0000","last_clean":"2023-01-15T14:31:44.927735+0000","last_became_active":"2023-01-15T14:31:38.111590+0000","last_became_peered":"2023-01-15T14:31:38.111590+0000","last_unstale":"2023-01-15T14:31:44.927735+0000","last_undegraded":"2023-01-15T14:31:44.927735+0000","last_fullsized":"2023-01-15T14:31:44.927735+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:51.156","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"169'138","reported_seq":784,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.253563+0000","last_change":"2023-01-15T14:31:47.253628+0000","last_active":"2023-01-15T14:31:47.253563+0000","last_peered":"2023-01-15T14:31:47.253563+0000","last_clean":"2023-01-15T14:31:47.253563+0000","last_became_active":"2023-01-15T14:31:35.134196+0000","last_became_peered":"2023-01-15T14:31:35.134196+0000","last_unstale":"2023-01-15T14:31:47.253563+0000","last_undegraded":"2023-01-15T14:31:47.253563+0000","last_fullsized":"2023-01-15T14:31:47.253563+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.09218208,"stat_sum":{"num_bytes":1226927,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":15209,"num_write":2,"num_write_kb":608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":618,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:44.927170+0000","last_change":"2023-01-15T14:31:44.927138+0000","last_active":"2023-01-15T14:31:44.927170+0000","last_peered":"2023-01-15T14:31:44.927170+0000","last_clean":"2023-01-15T14:31:44.927170+0000","last_became_active":"2023-01-15T14:31:34.951721+0000","last_became_peered":"2023-01-15T14:31:34.951721+0000","last_unstale":"2023-01-15T14:31:44.927170+0000","last_undegraded":"2023-01-15T14:31:44.927170+0000","last_fullsized":"2023-01-15T14:31:44.927170+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:30:24.721082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.721082+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.109000768,"objects_trimmed":0,"snaptrim_duration":0.40139308099999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.16","version":"218'90","reported_seq":221,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-01-15T14:31:46.442397+0000","last_change":"2023-01-15T14:31:46.444160+0000","last_active":"2023-01-15T14:31:46.442397+0000","last_peered":"2023-01-15T14:31:46.442397+0000","last_clean":"2023-01-15T14:31:46.442397+0000","last_became_active":"2023-01-15T14:31:35.032807+0000","last_became_peered":"2023-01-15T14:31:35.032807+0000","last_unstale":"2023-01-15T14:31:46.442397+0000","last_undegraded":"2023-01-15T14:31:46.442397+0000","last_fullsized":"2023-01-15T14:31:46.442397+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:40.949","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.107062621,"stat_sum":{"num_bytes":4522302,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.17","version":"200'78","reported_seq":648,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:44.927902+0000","last_change":"2023-01-15T14:31:44.927879+0000","last_active":"2023-01-15T14:31:44.927902+0000","last_peered":"2023-01-15T14:31:44.927902+0000","last_clean":"2023-01-15T14:31:44.927902+0000","last_became_active":"2023-01-15T14:31:34.932309+0000","last_became_peered":"2023-01-15T14:31:34.932309+0000","last_unstale":"2023-01-15T14:31:44.927902+0000","last_undegraded":"2023-01-15T14:31:44.927902+0000","last_fullsized":"2023-01-15T14:31:44.927902+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'65","last_scrub_stamp":"2023-01-15T14:30:24.156798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.156798+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30368223799999999,"objects_trimmed":0,"snaptrim_duration":0.48021021899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1974822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.18","version":"219'250","reported_seq":1033,"reported_epoch":219,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:31:46.243547+0000","last_change":"2023-01-15T14:31:45.926206+0000","last_active":"2023-01-15T14:31:46.243547+0000","last_peered":"2023-01-15T14:31:46.243547+0000","last_clean":"2023-01-15T14:31:46.243547+0000","last_became_active":"2023-01-15T14:31:04.108363+0000","last_became_peered":"2023-01-15T14:31:04.108363+0000","last_unstale":"2023-01-15T14:31:46.243547+0000","last_undegraded":"2023-01-15T14:31:46.243547+0000","last_fullsized":"2023-01-15T14:31:46.243547+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.386570156,"stat_sum":{"num_bytes":13564234,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":37875,"num_write":26,"num_write_kb":10585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":22066072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.19","version":"218'213","reported_seq":875,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.635901+0000","last_change":"2023-01-15T14:31:45.635901+0000","last_active":"2023-01-15T14:31:45.635901+0000","last_peered":"2023-01-15T14:31:45.635901+0000","last_clean":"2023-01-15T14:31:45.635901+0000","last_became_active":"2023-01-15T14:30:59.222826+0000","last_became_peered":"2023-01-15T14:30:59.222826+0000","last_unstale":"2023-01-15T14:31:45.635901+0000","last_undegraded":"2023-01-15T14:31:45.635901+0000","last_fullsized":"2023-01-15T14:31:45.635901+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":172,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'157","last_scrub_stamp":"2023-01-15T14:30:26.792736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:26.792736+0000","objects_scrubbed":4,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61998826900000004,"objects_trimmed":0,"snaptrim_duration":0.714732649,"stat_sum":{"num_bytes":19629458,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":35343,"num_write":21,"num_write_kb":6040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":29479511,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.1a","version":"220'231","reported_seq":958,"reported_epoch":220,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:31:47.253757+0000","last_change":"2023-01-15T14:31:47.253757+0000","last_active":"2023-01-15T14:31:47.253757+0000","last_peered":"2023-01-15T14:31:47.253757+0000","last_clean":"2023-01-15T14:31:47.253757+0000","last_became_active":"2023-01-15T14:31:13.049050+0000","last_became_peered":"2023-01-15T14:31:13.049050+0000","last_unstale":"2023-01-15T14:31:47.253757+0000","last_undegraded":"2023-01-15T14:31:47.253757+0000","last_fullsized":"2023-01-15T14:31:47.253757+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'196","last_scrub_stamp":"2023-01-15T14:31:19.373635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:19.373635+0000","objects_scrubbed":4,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:37.846","scrub_duration":0.20803622599999999,"objects_trimmed":0,"snaptrim_duration":0.90671364499999996,"stat_sum":{"num_bytes":16412267,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":12,"num_read_kb":10119,"num_write":16,"num_write_kb":3975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.1b","version":"155'53","reported_seq":604,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.365657+0000","last_change":"2023-01-15T14:31:47.060552+0000","last_active":"2023-01-15T14:31:47.365657+0000","last_peered":"2023-01-15T14:31:47.365657+0000","last_clean":"2023-01-15T14:31:47.365657+0000","last_became_active":"2023-01-15T14:31:02.994801+0000","last_became_peered":"2023-01-15T14:31:02.994801+0000","last_unstale":"2023-01-15T14:31:47.365657+0000","last_undegraded":"2023-01-15T14:31:47.365657+0000","last_fullsized":"2023-01-15T14:31:47.365657+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:06.174","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.30808192299999998,"stat_sum":{"num_bytes":3462242,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":13532,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":160,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.252367+0000","last_change":"2023-01-15T14:31:47.252457+0000","last_active":"2023-01-15T14:31:47.252367+0000","last_peered":"2023-01-15T14:31:47.252367+0000","last_clean":"2023-01-15T14:31:47.252367+0000","last_became_active":"2023-01-15T14:30:51.234588+0000","last_became_peered":"2023-01-15T14:30:51.234588+0000","last_unstale":"2023-01-15T14:31:47.252367+0000","last_undegraded":"2023-01-15T14:31:47.252367+0000","last_fullsized":"2023-01-15T14:31:47.252367+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:33.090","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.190940363,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":10,"num_write_kb":2522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1159667,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.1d","version":"219'196","reported_seq":894,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.250516+0000","last_change":"2023-01-15T14:31:47.250654+0000","last_active":"2023-01-15T14:31:47.250516+0000","last_peered":"2023-01-15T14:31:47.250516+0000","last_clean":"2023-01-15T14:31:47.250516+0000","last_became_active":"2023-01-15T14:30:54.085528+0000","last_became_peered":"2023-01-15T14:30:54.085528+0000","last_unstale":"2023-01-15T14:31:47.250516+0000","last_undegraded":"2023-01-15T14:31:47.250516+0000","last_fullsized":"2023-01-15T14:31:47.250516+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:32.052","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.189569816,"stat_sum":{"num_bytes":12926083,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":82,"num_read_kb":95244,"num_write":51,"num_write_kb":21261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":13857289,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.1e","version":"215'39","reported_seq":595,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:45.028070+0000","last_change":"2023-01-15T14:31:45.028051+0000","last_active":"2023-01-15T14:31:45.028070+0000","last_peered":"2023-01-15T14:31:45.028070+0000","last_clean":"2023-01-15T14:31:45.028070+0000","last_became_active":"2023-01-15T14:30:51.134845+0000","last_became_peered":"2023-01-15T14:30:51.134845+0000","last_unstale":"2023-01-15T14:31:45.028070+0000","last_undegraded":"2023-01-15T14:31:45.028070+0000","last_fullsized":"2023-01-15T14:31:45.028070+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:31:44.906253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.906253+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:59.426","scrub_duration":0.55943661099999997,"objects_trimmed":0,"snaptrim_duration":0.47276694000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.1f","version":"218'195","reported_seq":929,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.168517+0000","last_change":"2023-01-15T14:31:47.062325+0000","last_active":"2023-01-15T14:31:47.168517+0000","last_peered":"2023-01-15T14:31:47.168517+0000","last_clean":"2023-01-15T14:31:47.168517+0000","last_became_active":"2023-01-15T14:31:38.214614+0000","last_became_peered":"2023-01-15T14:31:38.214614+0000","last_unstale":"2023-01-15T14:31:47.168517+0000","last_undegraded":"2023-01-15T14:31:47.168517+0000","last_fullsized":"2023-01-15T14:31:47.168517+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.91451565499999998,"stat_sum":{"num_bytes":15842172,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":97,"num_read_kb":146296,"num_write":63,"num_write_kb":18148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":36580502,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.20","version":"220'243","reported_seq":1000,"reported_epoch":220,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:31:47.083264+0000","last_change":"2023-01-15T14:31:47.083264+0000","last_active":"2023-01-15T14:31:47.083264+0000","last_peered":"2023-01-15T14:31:47.083264+0000","last_clean":"2023-01-15T14:31:47.083264+0000","last_became_active":"2023-01-15T14:31:07.906191+0000","last_became_peered":"2023-01-15T14:31:07.906191+0000","last_unstale":"2023-01-15T14:31:47.083264+0000","last_undegraded":"2023-01-15T14:31:47.083264+0000","last_fullsized":"2023-01-15T14:31:47.083264+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.77607770099999995,"stat_sum":{"num_bytes":4659379,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":20,"num_read_kb":21726,"num_write":8,"num_write_kb":2065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8877936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.21","version":"219'126","reported_seq":649,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.081860+0000","last_change":"2023-01-15T14:31:47.081860+0000","last_active":"2023-01-15T14:31:47.081860+0000","last_peered":"2023-01-15T14:31:47.081860+0000","last_clean":"2023-01-15T14:31:47.081860+0000","last_became_active":"2023-01-15T14:31:38.233481+0000","last_became_peered":"2023-01-15T14:31:38.233481+0000","last_unstale":"2023-01-15T14:31:47.081860+0000","last_undegraded":"2023-01-15T14:31:47.081860+0000","last_fullsized":"2023-01-15T14:31:47.081860+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"168'124","last_scrub_stamp":"2023-01-15T14:31:11.414546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.414546+0000","objects_scrubbed":0,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.28528039199999999,"objects_trimmed":0,"snaptrim_duration":0.003757491,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3798382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.22","version":"218'115","reported_seq":899,"reported_epoch":218,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:31:45.928296+0000","last_change":"2023-01-15T14:31:45.928296+0000","last_active":"2023-01-15T14:31:45.928296+0000","last_peered":"2023-01-15T14:31:45.928296+0000","last_clean":"2023-01-15T14:31:45.928296+0000","last_became_active":"2023-01-15T14:31:08.084483+0000","last_became_peered":"2023-01-15T14:31:08.084483+0000","last_unstale":"2023-01-15T14:31:45.928296+0000","last_undegraded":"2023-01-15T14:31:45.928296+0000","last_fullsized":"2023-01-15T14:31:45.928296+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:31.869","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.37375956100000002,"stat_sum":{"num_bytes":3603059,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.23","version":"149'233","reported_seq":820,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:44.922339+0000","last_change":"2023-01-15T14:31:44.922284+0000","last_active":"2023-01-15T14:31:44.922339+0000","last_peered":"2023-01-15T14:31:44.922339+0000","last_clean":"2023-01-15T14:31:44.922339+0000","last_became_active":"2023-01-15T14:31:06.029716+0000","last_became_peered":"2023-01-15T14:31:06.029716+0000","last_unstale":"2023-01-15T14:31:44.922339+0000","last_undegraded":"2023-01-15T14:31:44.922339+0000","last_fullsized":"2023-01-15T14:31:44.922339+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.47787588600000003,"stat_sum":{"num_bytes":7604272,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":47,"num_read_kb":43637,"num_write":37,"num_write_kb":13160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2469494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.24","version":"214'211","reported_seq":969,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:44.922069+0000","last_change":"2023-01-15T14:31:44.922003+0000","last_active":"2023-01-15T14:31:44.922069+0000","last_peered":"2023-01-15T14:31:44.922069+0000","last_clean":"2023-01-15T14:31:44.922069+0000","last_became_active":"2023-01-15T14:31:26.168777+0000","last_became_peered":"2023-01-15T14:31:26.168777+0000","last_unstale":"2023-01-15T14:31:44.922069+0000","last_undegraded":"2023-01-15T14:31:44.922069+0000","last_fullsized":"2023-01-15T14:31:44.922069+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":211,"log_dups_size":0,"ondisk_log_size":211,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.60493616100000003,"stat_sum":{"num_bytes":5983581,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":47,"num_read_kb":63482,"num_write":21,"num_write_kb":6075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":21209193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.25","version":"218'87","reported_seq":730,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:46.131398+0000","last_change":"2023-01-15T14:31:46.131398+0000","last_active":"2023-01-15T14:31:46.131398+0000","last_peered":"2023-01-15T14:31:46.131398+0000","last_clean":"2023-01-15T14:31:46.131398+0000","last_became_active":"2023-01-15T14:31:22.000386+0000","last_became_peered":"2023-01-15T14:31:22.000386+0000","last_unstale":"2023-01-15T14:31:46.131398+0000","last_undegraded":"2023-01-15T14:31:46.131398+0000","last_fullsized":"2023-01-15T14:31:46.131398+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:51.596","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":1.208409718,"stat_sum":{"num_bytes":4537956,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":7,"num_read_kb":6061,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.26","version":"102'52","reported_seq":89,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.082483+0000","last_change":"2023-01-15T14:31:47.082483+0000","last_active":"2023-01-15T14:31:47.082483+0000","last_peered":"2023-01-15T14:31:47.082483+0000","last_clean":"2023-01-15T14:31:47.082483+0000","last_became_active":"2023-01-15T14:31:21.901622+0000","last_became_peered":"2023-01-15T14:31:21.901622+0000","last_unstale":"2023-01-15T14:31:47.082483+0000","last_undegraded":"2023-01-15T14:31:47.082483+0000","last_fullsized":"2023-01-15T14:31:47.082483+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:31:29.194049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:29.194049+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:33.783","scrub_duration":0.55352289399999999,"objects_trimmed":0,"snaptrim_duration":0.001521536,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":693,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:44.927545+0000","last_change":"2023-01-15T14:31:44.927521+0000","last_active":"2023-01-15T14:31:44.927545+0000","last_peered":"2023-01-15T14:31:44.927545+0000","last_clean":"2023-01-15T14:31:44.927545+0000","last_became_active":"2023-01-15T14:31:22.091646+0000","last_became_peered":"2023-01-15T14:31:22.091646+0000","last_unstale":"2023-01-15T14:31:44.927545+0000","last_undegraded":"2023-01-15T14:31:44.927545+0000","last_fullsized":"2023-01-15T14:31:44.927545+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.47756979799999999,"stat_sum":{"num_bytes":2485558,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":8458,"num_write":2,"num_write_kb":737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1731121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.28","version":"218'216","reported_seq":989,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.926133+0000","last_change":"2023-01-15T14:31:45.926133+0000","last_active":"2023-01-15T14:31:45.926133+0000","last_peered":"2023-01-15T14:31:45.926133+0000","last_clean":"2023-01-15T14:31:45.926133+0000","last_became_active":"2023-01-15T14:31:26.054878+0000","last_became_peered":"2023-01-15T14:31:26.054878+0000","last_unstale":"2023-01-15T14:31:45.926133+0000","last_undegraded":"2023-01-15T14:31:45.926133+0000","last_fullsized":"2023-01-15T14:31:45.926133+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":1.0053566949999999,"stat_sum":{"num_bytes":2159716,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":10594,"num_write":3,"num_write_kb":3981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":12193528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.29","version":"110'149","reported_seq":515,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:45.027447+0000","last_change":"2023-01-15T14:31:45.027411+0000","last_active":"2023-01-15T14:31:45.027447+0000","last_peered":"2023-01-15T14:31:45.027447+0000","last_clean":"2023-01-15T14:31:45.027447+0000","last_became_active":"2023-01-15T14:31:25.136376+0000","last_became_peered":"2023-01-15T14:31:25.136376+0000","last_unstale":"2023-01-15T14:31:45.027447+0000","last_undegraded":"2023-01-15T14:31:45.027447+0000","last_fullsized":"2023-01-15T14:31:45.027447+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:30:08.681869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:08.681869+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.37319031600000002,"stat_sum":{"num_bytes":2857174,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":24874,"num_write":16,"num_write_kb":6171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.2a","version":"191'191","reported_seq":884,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.182098+0000","last_change":"2023-01-15T14:31:47.182098+0000","last_active":"2023-01-15T14:31:47.182098+0000","last_peered":"2023-01-15T14:31:47.182098+0000","last_clean":"2023-01-15T14:31:47.182098+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:31:47.182098+0000","last_undegraded":"2023-01-15T14:31:47.182098+0000","last_fullsized":"2023-01-15T14:31:47.182098+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:44.953","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.0027159319999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":588,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-01-15T14:31:46.444318+0000","last_change":"2023-01-15T14:31:46.544867+0000","last_active":"2023-01-15T14:31:46.444318+0000","last_peered":"2023-01-15T14:31:46.444318+0000","last_clean":"2023-01-15T14:31:46.444318+0000","last_became_active":"2023-01-15T14:31:21.896494+0000","last_became_peered":"2023-01-15T14:31:21.896494+0000","last_unstale":"2023-01-15T14:31:46.444318+0000","last_undegraded":"2023-01-15T14:31:46.444318+0000","last_fullsized":"2023-01-15T14:31:46.444318+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:30.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.975433+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:33.669","scrub_duration":0.0040286109999999997,"objects_trimmed":0,"snaptrim_duration":0.39267342,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":79,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.060595+0000","last_change":"2023-01-15T14:31:47.060595+0000","last_active":"2023-01-15T14:31:47.060595+0000","last_peered":"2023-01-15T14:31:47.060595+0000","last_clean":"2023-01-15T14:31:47.060595+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:31:47.060595+0000","last_undegraded":"2023-01-15T14:31:47.060595+0000","last_fullsized":"2023-01-15T14:31:47.060595+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.20483100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.2d","version":"168'156","reported_seq":857,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.082601+0000","last_change":"2023-01-15T14:31:47.082601+0000","last_active":"2023-01-15T14:31:47.082601+0000","last_peered":"2023-01-15T14:31:47.082601+0000","last_clean":"2023-01-15T14:31:47.082601+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:31:47.082601+0000","last_undegraded":"2023-01-15T14:31:47.082601+0000","last_fullsized":"2023-01-15T14:31:47.082601+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.81366173600000002,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.2e","version":"139'38","reported_seq":523,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.637676+0000","last_change":"2023-01-15T14:31:45.637676+0000","last_active":"2023-01-15T14:31:45.637676+0000","last_peered":"2023-01-15T14:31:45.637676+0000","last_clean":"2023-01-15T14:31:45.637676+0000","last_became_active":"2023-01-15T14:31:38.116756+0000","last_became_peered":"2023-01-15T14:31:38.116756+0000","last_unstale":"2023-01-15T14:31:45.637676+0000","last_undegraded":"2023-01-15T14:31:45.637676+0000","last_fullsized":"2023-01-15T14:31:45.637676+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:29.609173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:29.609173+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.62936287599999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":5765,"num_write":3,"num_write_kb":2023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":743710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":720,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.083034+0000","last_change":"2023-01-15T14:31:47.083034+0000","last_active":"2023-01-15T14:31:47.083034+0000","last_peered":"2023-01-15T14:31:47.083034+0000","last_clean":"2023-01-15T14:31:47.083034+0000","last_became_active":"2023-01-15T14:31:40.024531+0000","last_became_peered":"2023-01-15T14:31:40.024531+0000","last_unstale":"2023-01-15T14:31:47.083034+0000","last_undegraded":"2023-01-15T14:31:47.083034+0000","last_fullsized":"2023-01-15T14:31:47.083034+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:31:22.549180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:22.549180+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.91420409700000005,"stat_sum":{"num_bytes":3113505,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":16794,"num_write":19,"num_write_kb":6416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7449703,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.30","version":"206'252","reported_seq":977,"reported_epoch":220,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:31:47.181682+0000","last_change":"2023-01-15T14:31:47.181682+0000","last_active":"2023-01-15T14:31:47.181682+0000","last_peered":"2023-01-15T14:31:47.181682+0000","last_clean":"2023-01-15T14:31:47.181682+0000","last_became_active":"2023-01-15T14:31:39.915702+0000","last_became_peered":"2023-01-15T14:31:39.915702+0000","last_unstale":"2023-01-15T14:31:47.181682+0000","last_undegraded":"2023-01-15T14:31:47.181682+0000","last_fullsized":"2023-01-15T14:31:47.181682+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.0017977850000000001,"stat_sum":{"num_bytes":5419866,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":25948,"num_write":35,"num_write_kb":13827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":13340458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"83","length":"1"},{"start":"8b","length":"1"},{"start":"91","length":"1"},{"start":"a3","length":"1"}]},{"pgid":"3.31","version":"204'150","reported_seq":857,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-01-15T14:31:45.427771+0000","last_change":"2023-01-15T14:31:45.427771+0000","last_active":"2023-01-15T14:31:45.427771+0000","last_peered":"2023-01-15T14:31:45.427771+0000","last_clean":"2023-01-15T14:31:45.427771+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:31:45.427771+0000","last_undegraded":"2023-01-15T14:31:45.427771+0000","last_fullsized":"2023-01-15T14:31:45.427771+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'144","last_scrub_stamp":"2023-01-15T14:31:13.806973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:13.806973+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.50567314500000005,"stat_sum":{"num_bytes":7117814,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":21449,"num_write":7,"num_write_kb":6179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":7210220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"83","length":"1"},{"start":"85","length":"1"},{"start":"8b","length":"1"},{"start":"90","length":"2"},{"start":"94","length":"1"},{"start":"96","length":"2"},{"start":"99","length":"1"},{"start":"a3","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":56,"stat_sum":{"num_bytes":264572984,"num_objects":129,"num_object_clones":79,"num_object_copies":387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":129,"num_whiteouts":11,"num_read":769,"num_read_kb":841424,"num_write":575,"num_write_kb":219916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":409,"num_bytes_recovered":262057819,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8075,"ondisk_log_size":8075,"up":168,"acting":168,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574894,"num_pgs":21,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":888408,"kb_used_data":92817904,"kb_used_omap":2905,"kb_used_meta":0,"kb_avail":92810972,"statfs":{"total":95948165120,"available":95038435328,"internally_reserved":7098368,"allocated":95045533696,"data_stored":95045533696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2975007,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":14.18,"5min":5.4340000000000002,"15min":4.524},"min":{"1min":0.57099999999999995,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":148.10900000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.753,"5min":7.5119999999999996,"15min":8.6430000000000007},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0569999999999999,"5min":73.769000000000005,"15min":73.769000000000005},"last":0.58699999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":5.3419999999999996,"15min":6.1440000000000001},"min":{"1min":0.47699999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":88.003,"15min":88.003},"last":0.72099999999999997},{"interface":"front","average":{"1min":6.2370000000000001,"5min":6.0810000000000004,"15min":6.2130000000000001},"min":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":97.704999999999998,"5min":97.704999999999998,"15min":97.704999999999998},"last":96.581999999999994}]},{"osd":2,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":5.8250000000000002,"15min":5.1349999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.032,"5min":94.733999999999995,"15min":94.733999999999995},"last":0.85499999999999998},{"interface":"front","average":{"1min":5.8869999999999996,"5min":3.8650000000000002,"15min":4.0730000000000004},"min":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":91.837999999999994,"5min":91.837999999999994,"15min":91.837999999999994},"last":0.86699999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":3.782,"15min":1.8919999999999999},"min":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":15.58,"5min":144.65600000000001,"15min":144.65600000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":2.8439999999999999,"5min":4.173,"15min":4.492},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":31.751000000000001,"5min":76.034999999999997,"15min":76.034999999999997},"last":0.81200000000000006}]},{"osd":4,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":6.8899999999999997,"15min":8.5739999999999998},"min":{"1min":0.60499999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0609999999999999,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.745},{"interface":"front","average":{"1min":1.042,"5min":4.0679999999999996,"15min":1.9990000000000001},"min":{"1min":0.61899999999999999,"5min":0.498,"15min":0.498},"max":{"1min":3.944,"5min":164.21199999999999,"15min":164.21199999999999},"last":1.044}]},{"osd":5,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":5.0599999999999996,"15min":5.9039999999999999},"min":{"1min":0.51200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1799999999999999,"5min":73.561000000000007,"15min":73.561000000000007},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.92100000000000004,"5min":4.226,"15min":4.7050000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.3029999999999999,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.92400000000000004}]},{"osd":6,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":6.4340000000000002,"15min":7.827},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.216,"5min":73.638999999999996,"15min":73.638999999999996},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.9690000000000001,"15min":1.389},"min":{"1min":0.74399999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1559999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.1160000000000001}]}]},{"osd":6,"up_from":0,"seq":55834574895,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574895,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574894,"num_pgs":42,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1036528,"kb_used_data":92676296,"kb_used_omap":5691,"kb_used_meta":0,"kb_avail":92662852,"statfs":{"total":95948165120,"available":94886760448,"internally_reserved":13766656,"allocated":94900527104,"data_stored":94900527104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5827961,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":4.633,"15min":4.8209999999999997},"min":{"1min":0.47099999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.282,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.55100000000000005},{"interface":"front","average":{"1min":4.0869999999999997,"5min":2.008,"15min":1.1899999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":70.352999999999994,"5min":70.352999999999994,"15min":70.352999999999994},"last":0.61699999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":2.145,"5min":7.5339999999999998,"15min":8.2400000000000002},"min":{"1min":0.60399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":24.489999999999998,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":12.505000000000001,"5min":4.1219999999999999,"15min":1.903},"min":{"1min":0.52900000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":110.06999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.72899999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":5.4859999999999998,"15min":5.5019999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":97.974000000000004,"5min":99.884,"15min":99.884},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.88500000000000001,"15min":0.90100000000000002},"min":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.254,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.91200000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.2649999999999999,"15min":1.3140000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.3280000000000001,"5min":12.878,"15min":12.878},"last":0.69999999999999996},{"interface":"front","average":{"1min":2.6320000000000001,"5min":5.5659999999999998,"15min":5.9560000000000004},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":24.649999999999999,"5min":99.921000000000006,"15min":99.921000000000006},"last":0.96699999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.88600000000000001,"15min":0.90800000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.109,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.746,"15min":2.1139999999999999},"min":{"1min":0.65200000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3879999999999999,"5min":30.971,"15min":30.971},"last":0.79300000000000004}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":2.2149999999999999,"5min":5.8090000000000002,"15min":5.718},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":24.553999999999998,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.92600000000000005,"5min":3.4910000000000001,"15min":4.6050000000000004},"min":{"1min":0.66700000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.234,"5min":88.186999999999998,"15min":88.186999999999998},"last":1.0329999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":8.266,"15min":11.507},"min":{"1min":0.63400000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.4510000000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":76.713999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":8.2899999999999991,"15min":10.256},"min":{"1min":0.65600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":24.603999999999999,"5min":99.843000000000004,"15min":99.843000000000004},"last":1.0589999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574894,"num_pgs":45,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1095052,"kb_used_data":92622256,"kb_used_omap":6042,"kb_used_meta":0,"kb_avail":92604328,"statfs":{"total":95948165120,"available":94826831872,"internally_reserved":18358272,"allocated":94845190144,"data_stored":94845190144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6187422,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":8,"commit_latency_ns":0,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":8.657,"5min":6.4790000000000001,"15min":5.391},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":64.275999999999996,"5min":174.52199999999999,"15min":174.52199999999999},"last":57.651000000000003},{"interface":"front","average":{"1min":4.7889999999999997,"5min":6.2050000000000001,"15min":7.1630000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":36.207000000000001,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.41299999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.3579999999999997,"5min":11.628,"15min":12.897},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":60.664000000000001,"5min":159.256,"15min":159.256},"last":0.46500000000000002},{"interface":"front","average":{"1min":19.719999999999999,"5min":6.1059999999999999,"15min":2.5289999999999999},"min":{"1min":0.40200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":155.44,"5min":174.578,"15min":174.578},"last":57.725000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":3.1429999999999998,"5min":3.153,"15min":3.2829999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":40.692999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":7.3019999999999996,"5min":4.5700000000000003,"15min":2.9849999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":64.381,"5min":174.45099999999999,"15min":174.45099999999999},"last":57.789999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":8.7050000000000001,"5min":6.2599999999999998,"15min":6.4610000000000003},"min":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":72.438999999999993,"5min":72.438999999999993,"15min":72.438999999999993},"last":0.70399999999999996},{"interface":"front","average":{"1min":8.8840000000000003,"5min":12.984,"15min":13.577},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":60.857999999999997,"5min":159.47200000000001,"15min":159.47200000000001},"last":0.83599999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.226,"5min":11.475,"15min":13.093},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":60.912999999999997,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.875},{"interface":"front","average":{"1min":4.8529999999999998,"5min":3.359,"15min":1.8009999999999999},"min":{"1min":0.752,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":64.513000000000005,"5min":174.61000000000001,"15min":174.61000000000001},"last":57.935000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.8799999999999999,"5min":3.3690000000000002,"15min":1.8109999999999999},"min":{"1min":0.69999999999999996,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":64.590000000000003,"5min":174.63999999999999,"15min":174.63999999999999},"last":58.006},{"interface":"front","average":{"1min":1,"5min":3.0529999999999999,"15min":3.5249999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.304,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.93100000000000005}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":3.206,"15min":3.7589999999999999},"min":{"1min":0.69699999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.2110000000000001,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.044},{"interface":"front","average":{"1min":5.3739999999999997,"5min":14.025,"15min":16.734999999999999},"min":{"1min":0.72599999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":60.951999999999998,"5min":159.512,"15min":159.512},"last":0.95399999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574894,"num_pgs":32,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1026812,"kb_used_data":92681912,"kb_used_omap":4636,"kb_used_meta":0,"kb_avail":92672568,"statfs":{"total":95948165120,"available":94896709632,"internally_reserved":9568256,"allocated":94906277888,"data_stored":94906277888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4748258,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":10,"commit_latency_ns":0,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.5739999999999998,"5min":2.5009999999999999,"15min":1.2609999999999999},"min":{"1min":0.439,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":66.277000000000001,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.62},{"interface":"front","average":{"1min":4.617,"5min":1.4670000000000001,"15min":0.96499999999999997},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":88.299999999999997,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.66700000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.2610000000000001,"5min":3.7229999999999999,"15min":1.7729999999999999},"min":{"1min":0.46500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":80.233999999999995,"5min":143.071,"15min":143.071},"last":53.942},{"interface":"front","average":{"1min":4.6989999999999998,"5min":3.1000000000000001,"15min":3.27},"min":{"1min":0.505,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":88.186000000000007,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.47099999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":10.462,"15min":10.422000000000001},"min":{"1min":0.40999999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.94899999999999995,"5min":183.53,"15min":183.53},"last":0.53900000000000003},{"interface":"front","average":{"1min":5.7729999999999997,"5min":3.8149999999999999,"15min":1.7829999999999999},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":66.335999999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.79100000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":5.9489999999999998,"15min":4.4160000000000004},"min":{"1min":0.624,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":66.491,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":4.9290000000000003,"5min":5.8259999999999996,"15min":4.4809999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":88.459999999999994,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.81999999999999995}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":2.577,"15min":1.643},"min":{"1min":0.65800000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2829999999999999,"5min":143.262,"15min":143.262},"last":0.76800000000000002},{"interface":"front","average":{"1min":6.0339999999999998,"5min":2.1040000000000001,"15min":1.4750000000000001},"min":{"1min":0.71899999999999997,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":66.456999999999994,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.97799999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.9889999999999999,"5min":1.8879999999999999,"15min":1.3169999999999999},"min":{"1min":0.70699999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":88.542000000000002,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.872},{"interface":"front","average":{"1min":1.028,"5min":2.7109999999999999,"15min":1.7150000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.585,"5min":143.215,"15min":143.215},"last":0.85199999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.6230000000000002,"5min":3.5489999999999999,"15min":3.222},"min":{"1min":0.81100000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":88.623999999999995,"5min":88.623999999999995,"15min":88.623999999999995},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.048,"5min":3.5630000000000002,"15min":3.0390000000000001},"min":{"1min":0.79900000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3859999999999999,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.92100000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":45,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1069780,"kb_used_data":92639760,"kb_used_omap":5815,"kb_used_meta":0,"kb_avail":92629600,"statfs":{"total":95948165120,"available":94852710400,"internally_reserved":10403840,"allocated":94863114240,"data_stored":94863114240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5954885,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":12,"commit_latency_ns":0,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3609999999999998,"5min":4.9240000000000004,"15min":4.2359999999999998},"min":{"1min":0.39900000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":100.533,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":4.6509999999999998,"5min":3.6459999999999999,"15min":4.0970000000000004},"min":{"1min":0.373,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":85.751000000000005,"5min":85.751000000000005,"15min":85.751000000000005},"last":0.55900000000000005}]},{"osd":2,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":2.6600000000000001,"5min":1.2270000000000001,"15min":1.0529999999999999},"min":{"1min":0.32500000000000001,"5min":0.245,"15min":0.245},"max":{"1min":43.759999999999998,"5min":43.759999999999998,"15min":43.759999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":3.1869999999999998,"15min":3.7770000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82899999999999996,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.76100000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":17.001000000000001,"5min":15.438000000000001,"15min":18.370000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":245.637,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":5.6310000000000002,"5min":5.9980000000000002,"15min":6.8289999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":85.814999999999998,"5min":85.814999999999998,"15min":85.814999999999998},"last":0.83399999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":63.935000000000002,"5min":16.309999999999999,"15min":8.5879999999999992},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1181.3389999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":57.106000000000002,"5min":14.65,"15min":7.2859999999999996},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1181.8720000000001,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.91100000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.98099999999999998,"15min":1.0149999999999999},"min":{"1min":0.58099999999999996,"5min":0.433,"15min":0.433},"max":{"1min":1.244,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.92200000000000004,"5min":3.036,"15min":3.9700000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.1339999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.95899999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":4.2060000000000004,"15min":4.7359999999999998},"min":{"1min":0.745,"5min":0.63,"15min":0.63},"max":{"1min":85.884,"5min":85.884,"15min":85.884},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.056,"15min":1.101},"min":{"1min":0.69599999999999995,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.298,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":0.99299999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":8.3879999999999999,"5min":7.1310000000000002,"15min":7.0549999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":157.03700000000001,"5min":157.03700000000001,"15min":157.03700000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":5.0919999999999996,"5min":4.0730000000000004,"15min":4.524},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":85.852999999999994,"5min":85.852999999999994,"15min":85.852999999999994},"last":1.028}]}]},{"osd":0,"up_from":13,"seq":55834574895,"num_pgs":10,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":820068,"kb_used_data":92879340,"kb_used_omap":4299,"kb_used_meta":0,"kb_avail":92879312,"statfs":{"total":95948165120,"available":95108415488,"internally_reserved":28672,"allocated":95108444160,"data_stored":95108444160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4402569,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":7.2519999999999998,"5min":2.6970000000000001,"15min":1.3140000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":69.816999999999993,"5min":69.872,"15min":69.872},"last":0.35899999999999999},{"interface":"front","average":{"1min":5.6130000000000004,"5min":2.6789999999999998,"15min":1.379},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":98.203999999999994,"5min":98.203999999999994,"15min":98.203999999999994},"last":61.447000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.7039999999999997,"5min":3.407,"15min":1.5980000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":98.286000000000001,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":3.0270000000000001,"5min":2.6819999999999999,"15min":1.3660000000000001},"min":{"1min":0.38400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":48.893000000000001,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.32400000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":12.154999999999999,"5min":13.413,"15min":15.657},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":94.712999999999994,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":10.484,"5min":5.6399999999999997,"15min":2.4140000000000001},"min":{"1min":0.42399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":98.352999999999994,"5min":174.09399999999999,"15min":174.09399999999999},"last":162.61600000000001}]},{"osd":4,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.5570000000000004,"5min":3.2309999999999999,"15min":1.734},"min":{"1min":0.63800000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":61.536000000000001,"5min":71.483000000000004,"15min":71.483000000000004},"last":0.63900000000000001},{"interface":"front","average":{"1min":9.2989999999999995,"5min":3.9950000000000001,"15min":2.0150000000000001},"min":{"1min":0.67400000000000004,"5min":0.439,"15min":0.439},"max":{"1min":70.179000000000002,"5min":70.179000000000002,"15min":70.179000000000002},"last":0.98399999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.9009999999999998,"5min":8.218,"15min":9.2710000000000008},"min":{"1min":0.68500000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":98.411000000000001,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.7709999999999999,"15min":1.2829999999999999},"min":{"1min":0.63400000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.2959999999999998,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.68100000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":7.7030000000000003,"5min":3.2040000000000002,"15min":1.8480000000000001},"min":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":70.259,"5min":70.259,"15min":70.259},"last":1.036},{"interface":"front","average":{"1min":5.984,"5min":3.056,"15min":1.774},"min":{"1min":0.749,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":98.484999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":0.755}]},{"osd":7,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":4.633,"5min":12.002000000000001,"15min":14.448},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":73.864999999999995,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":7.7199999999999998,"5min":3.21,"15min":1.8500000000000001},"min":{"1min":0.70399999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":70.322000000000003,"5min":70.322000000000003,"15min":70.322000000000003},"last":0.92000000000000004}]}]}],"pool_statfs":[]}} 2023-01-15T14:31:48.483 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:31:48.614 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 66 2023-01-15T14:31:48.618 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:48.618 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:48.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:48.619 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:48.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.619 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:48.619 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:48.619 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.34943890599999, 2023-01-15T14:31:48.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:31:48.620 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.620 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:48.620 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.620 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.620 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.621 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:31:48.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.621 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:48.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:48.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:31:48.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:48.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:31:48.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:31:48.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:31:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:31:48.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:31:48.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:31:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:31:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:31:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:31:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:31:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:31:48.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:31:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:31:48.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:31:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:31:48.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:31:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:31:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:31:48.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:31:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:31:48.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:31:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:31:48.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:31:48.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:31:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:31:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:31:48.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:31:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:31:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:31:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:31:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:31:48.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:31:48.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.657 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.657 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:48.657 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:48.658 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.34915865799999, 2023-01-15T14:31:48.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:31:48.658 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.658 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:48.658 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.658 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.658 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.658 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:31:48.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.658 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:48.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:48.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:31:48.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:48.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:31:48.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:31:48.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:31:48.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:31:48.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:31:48.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:31:48.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:31:48.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:31:48.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:31:48.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:31:48.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:48.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:31:48.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:31:48.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:48.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:31:48.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:31:48.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:48.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:31:48.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:31:48.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:48.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:31:48.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:31:48.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:31:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:31:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:31:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:31:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:31:48.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:31:48.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.668 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.668 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:48.668 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:48.668 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.34912604100001, 2023-01-15T14:31:48.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:31:48.668 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.668 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:48.669 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.669 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.669 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.669 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:31:48.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.669 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:48.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:31:48.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:31:48.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:31:48.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:31:48.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:31:48.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:31:48.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:31:48.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:31:48.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:31:48.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:31:48.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:31:48.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:31:48.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:31:48.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:31:48.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:31:48.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:31:48.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.675 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.675 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:48.763 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:48.764 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.24780805500001, 2023-01-15T14:31:48.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:31:48.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.764 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:48.764 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.765 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.765 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.769 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:31:48.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.770 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:48.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:48.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:31:48.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:48.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:31:48.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:31:48.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:31:48.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:31:48.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:31:48.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:31:48.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:31:48.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:31:48.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:31:48.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:31:48.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:31:48.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:31:48.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:31:48.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:31:48.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:48.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:31:48.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:31:48.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:48.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:31:48.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:31:48.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:48.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:31:48.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:31:48.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:48.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:31:48.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:31:48.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:31:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:31:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:31:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:31:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:31:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:31:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:31:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:31:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:31:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:31:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.247778052, 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:31:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:31:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:31:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:31:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:31:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:31:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:31:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:31:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 201.73276948700001, 2023-01-15T14:31:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:31:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:31:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:31:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:31:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:31:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:31:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:31:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:31:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:31:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:31:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:31:48.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:31:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:31:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:31:48.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:31:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:31:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:31:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:31:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:48.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:31:48.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:31:48.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:48.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:31:48.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:31:48.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:31:48.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:31:48.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:31:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:31:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:31:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:31:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:31:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:31:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:48.812 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 201.732287133, 2023-01-15T14:31:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:31:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:31:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:31:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:31:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:31:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:31:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:31:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:31:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:31:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:31:48.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:31:48.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:31:48.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:31:48.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:31:48.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:31:48.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:31:48.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:31:48.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:48.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:31:48.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:31:48.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:48.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:31:48.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:31:48.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:31:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:31:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:48.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:31:48.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:31:48.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:48.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:31:48.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:31:48.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:31:48.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:31:48.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:31:48.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:31:48.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:48.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:31:48.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:31:48.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:31:48.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:31:48.827 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.827 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.827 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.827 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:48.827 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:48.828 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 201.73223457, 2023-01-15T14:31:48.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:31:48.828 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.828 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:48.828 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.828 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.828 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.828 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:31:48.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.828 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:31:48.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:31:48.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:31:48.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:31:48.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:31:48.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:31:48.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:31:48.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:31:48.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:31:48.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:31:48.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:31:48.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:31:48.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:31:48.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:31:48.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:31:48.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:31:48.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:31:48.834 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.834 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.834 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 188.36103286599999, 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.835 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:31:48.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.836 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:48.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:48.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:31:48.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:48.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:31:48.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:31:48.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:31:48.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:31:48.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:48.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:31:48.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:31:48.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:31:48.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:31:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:31:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:31:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:31:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:31:48.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.928 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:48.928 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:48.928 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 188.36091619199999, 2023-01-15T14:31:48.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:31:48.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:48.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:31:48.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:31:48.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:31:48.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:31:48.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:31:48.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:31:48.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:48.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:31:48.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:31:48.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:31:48.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:31:48.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:31:48.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:31:48.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:31:48.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.935 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.935 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:31:48.935 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:48.935 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 180.634783916, 2023-01-15T14:31:48.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:31:48.935 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.935 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:48.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.936 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.936 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:31:48.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.936 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:48.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:48.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:31:48.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:48.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:31:48.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:31:48.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:31:48.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:31:48.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:31:48.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:31:48.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:31:48.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:31:48.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.941 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.941 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:31:48.941 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:48.942 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.418533857, 2023-01-15T14:31:48.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:31:48.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:48.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.942 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:31:48.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.942 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:48.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:48.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:31:48.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:48.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:31:48.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:31:48.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:31:48.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:31:48.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:31:48.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:31:48.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:31:48.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:31:48.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:48.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:31:48.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:48.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:31:48.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:31:48.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:31:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:31:48.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:31:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:31:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:31:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:31:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:31:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:31:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:31:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:31:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:31:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:31:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:31:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:48.965 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.41850029, 2023-01-15T14:31:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:31:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:31:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:31:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:31:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:31:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:31:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:31:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:31:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:31:48.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:31:48.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:48.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:31:48.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:31:48.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:31:48.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:31:48.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:31:48.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:31:48.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:48.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:31:48.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:31:48.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:48.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:31:48.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:31:48.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:48.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:31:48.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:31:48.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:31:48.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:31:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:31:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:31:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:31:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:31:48.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:48.975 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.65259674399999, 2023-01-15T14:31:48.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:31:48.976 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.976 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:48.976 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.976 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.976 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.976 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:31:48.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.976 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:31:48.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:48.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:31:48.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:31:48.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:48.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:31:48.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:31:48.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:48.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:31:48.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:31:48.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:48.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:31:48.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:31:48.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:48.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:31:48.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:31:48.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:48.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:31:48.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:31:48.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:48.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:31:48.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:31:48.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:31:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:31:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:31:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:31:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:31:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:31:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:31:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:31:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:31:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:31:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:31:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.109417944, 2023-01-15T14:31:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:31:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:31:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:31:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:31:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:31:49.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:31:49.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:31:49.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:31:49.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:31:49.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:31:49.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:31:49.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:31:49.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:31:49.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:31:49.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:31:49.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:31:49.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:31:49.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:31:49.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:31:49.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:31:49.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:31:49.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:31:49.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:31:49.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:31:49.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:31:49.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:31:49.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:31:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:31:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:31:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:31:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.109378769, 2023-01-15T14:31:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:31:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:31:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:31:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:31:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:31:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:31:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:31:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:31:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:31:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:31:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:31:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:31:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1373 3.12 3:495d4bbc:::smithi09025264-16:head [write 2873523~591733] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.717262443, 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40949580099999999, 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1373 2023-01-15T14:31:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:31:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:31:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693440+0000", 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693808+0000", 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000367742 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693831+0000", 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3213e-05 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693846+0000", 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4909000000000001e-05 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.695833+0000", 2023-01-15T14:31:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019869240000000002 2023-01-15T14:31:49.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696121+0000", 2023-01-15T14:31:49.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028753300000000003 2023-01-15T14:31:49.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798577+0000", 2023-01-15T14:31:49.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102456808 2023-01-15T14:31:49.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798849+0000", 2023-01-15T14:31:49.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027138999999999999 2023-01-15T14:31:49.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799155+0000", 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030594800000000001 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799614+0000", 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045960700000000001 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799737+0000", 2023-01-15T14:31:49.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012218900000000001 2023-01-15T14:31:49.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799998+0000", 2023-01-15T14:31:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026129700000000002 2023-01-15T14:31:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800639+0000", 2023-01-15T14:31:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064108800000000005 2023-01-15T14:31:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102023+0000", 2023-01-15T14:31:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30138417200000001 2023-01-15T14:31:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102198+0000", 2023-01-15T14:31:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017453600000000001 2023-01-15T14:31:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102398+0000", 2023-01-15T14:31:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020078000000000001 2023-01-15T14:31:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102937+0000", 2023-01-15T14:31:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053874499999999998 2023-01-15T14:31:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:31:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:31:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.716837849, 2023-01-15T14:31:49.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:31:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:31:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:31:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:31:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:31:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:31:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:31:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:31:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:31:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:31:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:31:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:31:49.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:31:49.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:31:49.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:31:49.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:31:49.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:31:49.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:31:49.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:31:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:31:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:31:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:31:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:31:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:31:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:31:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:31:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:31:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:31:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:31:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:31:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:31:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:31:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.71678552, 2023-01-15T14:31:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:31:49.136 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.136 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:49.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.136 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:31:49.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.136 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:49.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:31:49.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:31:49.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:31:49.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:31:49.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:31:49.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:31:49.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:31:49.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:31:49.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:31:49.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:31:49.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:31:49.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:31:49.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:31:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 82.119872321000003, 2023-01-15T14:31:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:31:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:49.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:31:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:31:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:31:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:31:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:31:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:31:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:31:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:31:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:31:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:31:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:31:49.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:31:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:31:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.249 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.249 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:49.250 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:31:49.250 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:48.611+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:49.251 INFO:tasks.rados.rados.0.smithi090.stdout:907: done (6 left) 2023-01-15T14:31:49.251 INFO:tasks.rados.rados.0.smithi090.stdout:908: snap_remove snap 83 2023-01-15T14:31:49.253 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:49.247+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:49.355 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:49.349+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:49.362 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:49.362 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:49.363 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:49.363 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:49.372 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:31:49.455 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:49.451+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:49.505 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:49.506 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:49.522 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:31:49.556 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:49.551+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:49.629 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.53865662600001, 2023-01-15T14:31:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:31:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:31:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:31:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:31:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:31:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:31:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:31:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:31:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:31:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:31:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:31:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:31:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:31:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:31:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:31:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:31:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:31:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:31:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:31:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:31:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:31:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:31:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:31:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:31:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:31:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:31:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:31:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:31:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:31:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:31:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.53835682900001, 2023-01-15T14:31:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:31:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:31:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:31:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:31:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:31:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:31:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:31:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:31:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:31:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:31:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:31:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:31:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:31:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:31:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:31:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:31:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:31:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:31:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:31:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.53832704499999, 2023-01-15T14:31:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:31:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:31:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:31:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:31:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:31:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:31:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:31:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:31:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:31:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:31:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:31:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:31:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:31:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:31:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:182 3.4 3:2022fdfa:::smithi09025264-46:head [write 1933572~791670] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.40946497600001, 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.582304865, 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 182 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213706+0000", 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.213704+0000", 2023-01-15T14:31:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214574+0000", 2023-01-15T14:31:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087037799999999999 2023-01-15T14:31:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214608+0000", 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214627+0000", 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491537+0000", 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27691020300000002 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.491721+0000", 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018345500000000001 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693760+0000", 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202039252 2023-01-15T14:31:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.693898+0000", 2023-01-15T14:31:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013789 2023-01-15T14:31:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694230+0000", 2023-01-15T14:31:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220899999999998 2023-01-15T14:31:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694866+0000", 2023-01-15T14:31:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063640599999999995 2023-01-15T14:31:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.694887+0000", 2023-01-15T14:31:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0622999999999999e-05 2023-01-15T14:31:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695950+0000", 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001062688 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696014+0000", 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4697999999999996e-05 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795898+0000", 2023-01-15T14:31:49.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099883338000000002 2023-01-15T14:31:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795922+0000", 2023-01-15T14:31:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4326000000000001e-05 2023-01-15T14:31:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795955+0000", 2023-01-15T14:31:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3191000000000001e-05 2023-01-15T14:31:49.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796011+0000", 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5631999999999999e-05 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:49.666 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.40851824500001, 2023-01-15T14:31:49.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:31:49.667 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.667 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.667 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:31:49.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.667 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:49.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:31:49.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:31:49.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:31:49.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:31:49.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:31:49.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:31:49.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:31:49.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:31:49.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:31:49.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:49.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:31:49.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:31:49.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:31:49.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:31:49.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:31:49.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:31:49.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:31:49.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:31:49.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.676 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.676 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:49.676 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:49.676 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 202.40846154799999, 2023-01-15T14:31:49.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:31:49.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.676 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:49.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.677 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:31:49.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.677 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:49.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:31:49.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:31:49.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:31:49.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:31:49.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:31:49.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:31:49.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:31:49.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:31:49.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:31:49.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:31:49.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:31:49.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.683 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.683 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:31:49.683 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:49.683 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 199.79196537499999, 2023-01-15T14:31:49.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:31:49.683 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.684 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:49.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.684 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:31:49.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.684 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:49.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:31:49.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:31:49.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:31:49.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:31:49.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:31:49.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 192.76311690399999, 2023-01-15T14:31:49.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:31:49.690 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.690 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.690 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:31:49.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.690 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:31:49.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:31:49.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:31:49.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:31:49.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:49.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:31:49.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:31:49.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:31:49.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:31:49.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:31:49.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:31:49.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:31:49.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:31:49.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:31:49.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:31:49.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:31:49.698 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.698 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.698 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.699 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:373 3.4 3:2029fa71:::smithi09025264-39:head [write 1729599~575687] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:49.699 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:49.699 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 189.544869697, 2023-01-15T14:31:49.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58298356399999995, 2023-01-15T14:31:49.699 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.699 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.700 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 373 2023-01-15T14:31:49.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.700 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:49.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078301+0000", 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078299+0000", 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078644+0000", 2023-01-15T14:31:49.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034494099999999998 2023-01-15T14:31:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078676+0000", 2023-01-15T14:31:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1183000000000003e-05 2023-01-15T14:31:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078694+0000", 2023-01-15T14:31:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.84e-05 2023-01-15T14:31:49.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078748+0000", 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3634999999999997e-05 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.078753+0000", 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6370000000000004e-06 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110841+0000", 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032087439000000002 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.110967+0000", 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000126029 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.457874+0000", 2023-01-15T14:31:49.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.346907725 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458066+0000", 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000191592 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458321+0000", 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025454200000000002 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458918+0000", 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059740800000000001 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.458945+0000", 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6699999999999998e-05 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.459253+0000", 2023-01-15T14:31:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030836200000000001 2023-01-15T14:31:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.460943+0000", 2023-01-15T14:31:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016903580000000001 2023-01-15T14:31:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461340+0000", 2023-01-15T14:31:49.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039665799999999998 2023-01-15T14:31:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661129+0000", 2023-01-15T14:31:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19978933500000001 2023-01-15T14:31:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661188+0000", 2023-01-15T14:31:49.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8687000000000002e-05 2023-01-15T14:31:49.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661285+0000", 2023-01-15T14:31:49.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6779000000000003e-05 2023-01-15T14:31:49.709 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.709 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.709 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:376 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 189.53998376300001, 2023-01-15T14:31:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.57864278800000002, 2023-01-15T14:31:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.710 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 376 2023-01-15T14:31:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083187+0000", 2023-01-15T14:31:49.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083186+0000", 2023-01-15T14:31:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083193+0000", 2023-01-15T14:31:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8199999999999999e-06 2023-01-15T14:31:49.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083211+0000", 2023-01-15T14:31:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7609999999999999e-05 2023-01-15T14:31:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.083223+0000", 2023-01-15T14:31:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2140000000000001e-05 2023-01-15T14:31:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093534+0000", 2023-01-15T14:31:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.010311529 2023-01-15T14:31:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.093551+0000", 2023-01-15T14:31:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.666e-05 2023-01-15T14:31:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.183834+0000", 2023-01-15T14:31:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.090283063999999996 2023-01-15T14:31:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.184076+0000", 2023-01-15T14:31:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024241900000000001 2023-01-15T14:31:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461001+0000", 2023-01-15T14:31:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27692504299999998 2023-01-15T14:31:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.461152+0000", 2023-01-15T14:31:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150345 2023-01-15T14:31:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661642+0000", 2023-01-15T14:31:49.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200489742 2023-01-15T14:31:49.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661720+0000", 2023-01-15T14:31:49.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8604999999999997e-05 2023-01-15T14:31:49.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.661830+0000", 2023-01-15T14:31:49.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010992099999999999 2023-01-15T14:31:49.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.718 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.718 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:31:49.718 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:49.718 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 162.761829928, 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:49.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:49.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:31:49.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:49.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:31:49.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:31:49.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:31:49.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:31:49.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:31:49.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:31:49.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:31:49.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:31:49.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:31:49.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:31:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:31:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:31:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:31:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:31:49.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:31:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:31:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:31:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:31:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:31:49.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:31:49.727 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.727 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.727 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.727 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:31:49.727 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:49.727 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 115.70125167800001, 2023-01-15T14:31:49.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:31:49.728 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.728 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.728 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.728 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.728 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.728 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:31:49.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.728 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:31:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:31:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:31:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:31:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:31:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:31:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:31:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:31:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:31:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:31:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:31:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:31:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:31:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:31:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:31:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:31:49.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:31:49.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:31:49.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:31:49.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:31:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:31:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:31:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:31:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:31:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:31:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:31:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:31:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:31:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:31:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:31:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:31:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:31:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 115.701192968, 2023-01-15T14:31:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:31:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:31:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:31:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:31:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:49.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:31:49.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:31:49.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:31:49.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:31:49.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:31:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:31:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:31:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:31:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:31:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:31:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:31:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 111.744773475, 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:31:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:31:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:31:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:31:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:31:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:31:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:31:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:31:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:31:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:31:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:31:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:31:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:31:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:31:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:31:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 111.68727579199999, 2023-01-15T14:31:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:31:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.752 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:31:49.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:31:49.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:49.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:31:49.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:31:49.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:31:49.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:31:49.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:49.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:31:49.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:31:49.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:31:49.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:31:49.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:31:49.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:31:49.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:31:49.761 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.761 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.761 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.761 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.318624287000006, 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:31:49.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:49.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:31:49.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:31:49.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:31:49.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:31:49.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:31:49.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:31:49.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:31:49.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:31:49.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.318521645999994, 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.770 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:31:49.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:31:49.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:31:49.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:31:49.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:31:49.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:31:49.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.717539651000003, 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.777 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:31:49.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:31:49.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:31:49.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:31:49.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:31:49.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:31:49.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:31:49.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:31:49.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:31:49.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:31:49.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:31:49.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:31:49.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:31:49.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:31:49.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:31:49.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:31:49.788 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.788 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.788 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.788 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:49.788 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.717514227000002, 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:49.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:31:49.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:31:49.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:49.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:31:49.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:31:49.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:31:49.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:31:49.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:31:49.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:49.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:31:49.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:31:49.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:49.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:31:49.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:31:49.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:49.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:31:49.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:31:49.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:49.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:31:49.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:31:49.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:49.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:31:49.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:31:49.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:31:49.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:31:49.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:49.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:31:49.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:31:49.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:49.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:31:49.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:31:49.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:31:49.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:31:49.799 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.799 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.800 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.800 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2301 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e216)", 2023-01-15T14:31:49.800 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:31:49.800 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.4276397599999999, 2023-01-15T14:31:49.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61754771200000003, 2023-01-15T14:31:49.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:49.800 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:49.800 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:49.800 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2301 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:49.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195530+0000", 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195535+0000", 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3660000000000002e-06 2023-01-15T14:31:49.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:49.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195546+0000", 2023-01-15T14:31:49.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1062e-05 2023-01-15T14:31:49.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:49.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195551+0000", 2023-01-15T14:31:49.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2023-01-15T14:31:49.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195999+0000", 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044799800000000002 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.196106+0000", 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106172 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510631+0000", 2023-01-15T14:31:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31452575999999999 2023-01-15T14:31:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510875+0000", 2023-01-15T14:31:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024415199999999998 2023-01-15T14:31:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812445+0000", 2023-01-15T14:31:49.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30156923000000002 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812727+0000", 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028227800000000001 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.813079+0000", 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199599999999998 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:49.807 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:49.807 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:49.808 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:31:49.808 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:49.805+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:49.909 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:49.905+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:49.967 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:49.992 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:49.992 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:49.993 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:31:50.002 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:31:50.010 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:50.005+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:50.013 INFO:tasks.rados.rados.0.smithi090.stdout:908: done (6 left) 2023-01-15T14:31:50.014 INFO:tasks.rados.rados.0.smithi090.stdout:909: snap_create 2023-01-15T14:31:50.014 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:31:50.014 INFO:tasks.rados.rados.0.smithi090.stdout:901: finishing rollback tid 0 to smithi09025264-46 2023-01-15T14:31:50.014 INFO:tasks.rados.rados.0.smithi090.stdout:901: finishing rollback tid 1 to smithi09025264-46 2023-01-15T14:31:50.014 INFO:tasks.rados.rados.0.smithi090.stdout:901: finishing rollback tid 2 to smithi09025264-46 2023-01-15T14:31:50.014 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 238 (ObjNum 256 snap 60 seq_num 256) dirty exists 2023-01-15T14:31:50.014 INFO:tasks.rados.rados.0.smithi090.stdout:898: finishing copy_from to smithi09025264-37 2023-01-15T14:31:50.014 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 276 (ObjNum 332 snap 88 seq_num 332) dirty exists 2023-01-15T14:31:50.014 INFO:tasks.rados.rados.0.smithi090.stdout:898: finishing copy_from racing read to smithi09025264-37 2023-01-15T14:31:50.014 INFO:tasks.rados.rados.0.smithi090.stdout:902: finishing write tid 1 to smithi09025264-45 2023-01-15T14:31:50.015 INFO:tasks.rados.rados.0.smithi090.stdout:902: finishing write tid 2 to smithi09025264-45 2023-01-15T14:31:50.015 INFO:tasks.rados.rados.0.smithi090.stdout:899: finishing write tid 1 to smithi09025264-25 2023-01-15T14:31:50.015 INFO:tasks.rados.rados.0.smithi090.stdout:902: finishing write tid 3 to smithi09025264-45 2023-01-15T14:31:50.015 INFO:tasks.rados.rados.0.smithi090.stdout:902: finishing write tid 4 to smithi09025264-45 2023-01-15T14:31:50.015 INFO:tasks.rados.rados.0.smithi090.stdout:902: finishing write tid 5 to smithi09025264-45 2023-01-15T14:31:50.015 INFO:tasks.rados.rados.0.smithi090.stdout:902: oid 45 updating version 0 to 136 2023-01-15T14:31:50.015 INFO:tasks.rados.rados.0.smithi090.stdout:902: oid 45 version 136 is already newer than 135 2023-01-15T14:31:50.015 INFO:tasks.rados.rados.0.smithi090.stdout:902: oid 45 version 136 is already newer than 134 2023-01-15T14:31:50.015 INFO:tasks.rados.rados.0.smithi090.stdout:902: oid 45 updating version 136 to 137 2023-01-15T14:31:50.015 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 137 (ObjNum 336 snap 88 seq_num 336) dirty exists 2023-01-15T14:31:50.015 INFO:tasks.rados.rados.0.smithi090.stdout:902: left oid 45 (ObjNum 336 snap 88 seq_num 336) 2023-01-15T14:31:50.016 INFO:tasks.rados.rados.0.smithi090.stdout:906: expect (ObjNum 262 snap 61 seq_num 262) 2023-01-15T14:31:50.023 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 66 2023-01-15T14:31:50.112 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:50.107+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:50.121 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:50.121 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:50.122 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:50.122 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:50.131 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:31:50.191 INFO:tasks.rados.rados.0.smithi090.stdout:899: finishing write tid 2 to smithi09025264-25 2023-01-15T14:31:50.191 INFO:tasks.rados.rados.0.smithi090.stdout:905: finishing write tid 1 to smithi09025264-13 2023-01-15T14:31:50.191 INFO:tasks.rados.rados.0.smithi090.stdout:905: finishing write tid 2 to smithi09025264-13 2023-01-15T14:31:50.192 INFO:tasks.rados.rados.0.smithi090.stdout:899: finishing write tid 3 to smithi09025264-25 2023-01-15T14:31:50.192 INFO:tasks.rados.rados.0.smithi090.stdout:905: finishing write tid 3 to smithi09025264-13 2023-01-15T14:31:50.192 INFO:tasks.rados.rados.0.smithi090.stdout:899: finishing write tid 4 to smithi09025264-25 2023-01-15T14:31:50.192 INFO:tasks.rados.rados.0.smithi090.stdout:899: finishing write tid 5 to smithi09025264-25 2023-01-15T14:31:50.192 INFO:tasks.rados.rados.0.smithi090.stdout:899: oid 25 updating version 0 to 231 2023-01-15T14:31:50.192 INFO:tasks.rados.rados.0.smithi090.stdout:899: oid 25 updating version 231 to 232 2023-01-15T14:31:50.192 INFO:tasks.rados.rados.0.smithi090.stdout:899: oid 25 updating version 232 to 233 2023-01-15T14:31:50.192 INFO:tasks.rados.rados.0.smithi090.stdout:899: oid 25 updating version 233 to 234 2023-01-15T14:31:50.192 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 234 (ObjNum 335 snap 88 seq_num 335) dirty exists 2023-01-15T14:31:50.192 INFO:tasks.rados.rados.0.smithi090.stdout:899: left oid 25 (ObjNum 335 snap 88 seq_num 335) 2023-01-15T14:31:50.192 INFO:tasks.rados.rados.0.smithi090.stdout:905: finishing write tid 4 to smithi09025264-13 2023-01-15T14:31:50.193 INFO:tasks.rados.rados.0.smithi090.stdout:905: finishing write tid 5 to smithi09025264-13 2023-01-15T14:31:50.193 INFO:tasks.rados.rados.0.smithi090.stdout:905: oid 13 updating version 0 to 237 2023-01-15T14:31:50.193 INFO:tasks.rados.rados.0.smithi090.stdout:905: oid 13 version 237 is already newer than 235 2023-01-15T14:31:50.193 INFO:tasks.rados.rados.0.smithi090.stdout:905: oid 13 version 237 is already newer than 234 2023-01-15T14:31:50.193 INFO:tasks.rados.rados.0.smithi090.stdout:905: oid 13 version 237 is already newer than 236 2023-01-15T14:31:50.193 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 237 (ObjNum 338 snap 88 seq_num 338) dirty exists 2023-01-15T14:31:50.193 INFO:tasks.rados.rados.0.smithi090.stdout:905: left oid 13 (ObjNum 338 snap 88 seq_num 338) 2023-01-15T14:31:50.193 INFO:tasks.rados.rados.0.smithi090.stdout:898: done (5 left) 2023-01-15T14:31:50.193 INFO:tasks.rados.rados.0.smithi090.stdout:899: done (4 left) 2023-01-15T14:31:50.193 INFO:tasks.rados.rados.0.smithi090.stdout:901: done (3 left) 2023-01-15T14:31:50.193 INFO:tasks.rados.rados.0.smithi090.stdout:902: done (2 left) 2023-01-15T14:31:50.194 INFO:tasks.rados.rados.0.smithi090.stdout:905: done (1 left) 2023-01-15T14:31:50.194 INFO:tasks.rados.rados.0.smithi090.stdout:906: done (0 left) 2023-01-15T14:31:50.212 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:50.207+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:50.236 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:38 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'3, mlcod=0'0)", 2023-01-15T14:31:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 204.117628682, 2023-01-15T14:31:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30191175999999997, 2023-01-15T14:31:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:31:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112976+0000", 2023-01-15T14:31:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113186+0000", 2023-01-15T14:31:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020979700000000001 2023-01-15T14:31:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113228+0000", 2023-01-15T14:31:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2150999999999997e-05 2023-01-15T14:31:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213313+0000", 2023-01-15T14:31:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100084943 2023-01-15T14:31:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313517+0000", 2023-01-15T14:31:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10020435699999999 2023-01-15T14:31:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313576+0000", 2023-01-15T14:31:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783999999999998e-05 2023-01-15T14:31:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313976+0000", 2023-01-15T14:31:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039960499999999998 2023-01-15T14:31:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314004+0000", 2023-01-15T14:31:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7954e-05 2023-01-15T14:31:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314175+0000", 2023-01-15T14:31:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171557 2023-01-15T14:31:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414853+0000", 2023-01-15T14:31:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100677875 2023-01-15T14:31:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414890+0000", 2023-01-15T14:31:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6803e-05 2023-01-15T14:31:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:125 3.0 e70/55 3:0b54190a:::smithi09025264-17:25 v 70'136, mlcod=68'130)", 2023-01-15T14:31:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 154.28764665700001, 2023-01-15T14:31:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30171926799999998, 2023-01-15T14:31:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:31:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942958+0000", 2023-01-15T14:31:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942972+0000", 2023-01-15T14:31:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3386e-05 2023-01-15T14:31:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942990+0000", 2023-01-15T14:31:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8113000000000001e-05 2023-01-15T14:31:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.943002+0000", 2023-01-15T14:31:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:31:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141942+0000", 2023-01-15T14:31:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19893962300000001 2023-01-15T14:31:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141978+0000", 2023-01-15T14:31:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6383e-05 2023-01-15T14:31:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142670+0000", 2023-01-15T14:31:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069211900000000005 2023-01-15T14:31:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142688+0000", 2023-01-15T14:31:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7839999999999999e-05 2023-01-15T14:31:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142768+0000", 2023-01-15T14:31:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0210999999999995e-05 2023-01-15T14:31:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244581+0000", 2023-01-15T14:31:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101813406 2023-01-15T14:31:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244680+0000", 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8045000000000001e-05 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 124.171904269, 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:31:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:31:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:31:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:31:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:31:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:31:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:31:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:31:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:31:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:31:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:31:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:31:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:31:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:31:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:31:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.336146878999998, 2023-01-15T14:31:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:31:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:31:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:31:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:31:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:31:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:31:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:31:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:31:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:31:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:31:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:31:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:31:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:31:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:31:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:31:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:31:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:31:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:31:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:31:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.336095936000007, 2023-01-15T14:31:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:31:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:31:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:31:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:31:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:31:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:31:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:31:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:31:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:31:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:31:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:31:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:31:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:31:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:31:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:31:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:31:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:31:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:31:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:31:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:31:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:31:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:31:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:31:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:31:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 72.395196694999996, 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:31:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:31:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:31:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:31:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:31:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:31:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:31:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:31:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:31:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:31:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:31:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:31:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:31:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:31:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:31:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:31:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:31:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:31:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:31:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:31:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:31:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:31:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:31:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.321422522000006, 2023-01-15T14:31:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:31:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:31:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:31:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:31:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:31:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:31:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:31:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:31:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:31:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:31:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:31:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:31:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:31:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:31:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:31:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:31:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:31:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:31:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:31:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:31:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:31:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:31:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:31:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:31:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:31:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:31:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1712 3.4 3:2022fdfa:::smithi09025264-46:head [write 1880802~421790] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.316283831000007, 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30396041800000001, 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1712 2023-01-15T14:31:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:31:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914322+0000", 2023-01-15T14:31:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914561+0000", 2023-01-15T14:31:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023884700000000001 2023-01-15T14:31:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914572+0000", 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0499999999999999e-05 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914576+0000", 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6820000000000002e-06 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014941+0000", 2023-01-15T14:31:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100364994 2023-01-15T14:31:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014959+0000", 2023-01-15T14:31:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7416000000000001e-05 2023-01-15T14:31:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018171+0000", 2023-01-15T14:31:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032121239999999998 2023-01-15T14:31:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018333+0000", 2023-01-15T14:31:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016180900000000001 2023-01-15T14:31:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018581+0000", 2023-01-15T14:31:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024773699999999999 2023-01-15T14:31:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018897+0000", 2023-01-15T14:31:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031622099999999999 2023-01-15T14:31:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018907+0000", 2023-01-15T14:31:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0061000000000001e-05 2023-01-15T14:31:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.019108+0000", 2023-01-15T14:31:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020080099999999999 2023-01-15T14:31:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.020057+0000", 2023-01-15T14:31:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094954499999999999 2023-01-15T14:31:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150472+0000", 2023-01-15T14:31:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.130414958 2023-01-15T14:31:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218188+0000", 2023-01-15T14:31:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067715656999999999 2023-01-15T14:31:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218242+0000", 2023-01-15T14:31:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4651999999999998e-05 2023-01-15T14:31:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218284+0000", 2023-01-15T14:31:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1084999999999997e-05 2023-01-15T14:31:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.18 157/147 [PushOp(3:1eaad226:::smithi09025264-49:69, version: 139'201, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:1eaad226:::smithi09025264-49:69@139'201, size: 3916031, copy_subset: [0~3916031], clone_subset: {}, snapset: 69=[]:{68=[67],69=[69]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:31:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 65.521373279000002, 2023-01-15T14:31:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.27553999000000001, 2023-01-15T14:31:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709234+0000", 2023-01-15T14:31:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709232+0000", 2023-01-15T14:31:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709949+0000", 2023-01-15T14:31:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071723000000000002 2023-01-15T14:31:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.709990+0000", 2023-01-15T14:31:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.015e-05 2023-01-15T14:31:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.810156+0000", 2023-01-15T14:31:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10016643 2023-01-15T14:31:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984513+0000", 2023-01-15T14:31:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.174357128 2023-01-15T14:31:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984522+0000", 2023-01-15T14:31:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0350000000000007e-06 2023-01-15T14:31:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:44.984774+0000", 2023-01-15T14:31:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025144700000000002 2023-01-15T14:31:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.427467102000001, 2023-01-15T14:31:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:31:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:31:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:31:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:31:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:31:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:31:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:31:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:31:50.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:31:50.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:31:50.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:31:50.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:50.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:31:50.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:31:50.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:31:50.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:31:50.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:31:50.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:31:50.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:31:50.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:31:50.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.324 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.324 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.324 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1820 3.1f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:31:50.325 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:50.325 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.327165616999999, 2023-01-15T14:31:50.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306504938, 2023-01-15T14:31:50.325 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.325 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:31:50.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.325 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.325 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1820 2023-01-15T14:31:50.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.325 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:50.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:31:50.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903440+0000", 2023-01-15T14:31:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903448+0000", 2023-01-15T14:31:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9430000000000006e-06 2023-01-15T14:31:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903469+0000", 2023-01-15T14:31:50.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0305e-05 2023-01-15T14:31:50.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903481+0000", 2023-01-15T14:31:50.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:31:50.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903616+0000", 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013477199999999999 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903643+0000", 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7098999999999999e-05 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209081+0000", 2023-01-15T14:31:50.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30543790300000001 2023-01-15T14:31:50.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:50.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209213+0000", 2023-01-15T14:31:50.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013155400000000001 2023-01-15T14:31:50.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209600+0000", 2023-01-15T14:31:50.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038751699999999998 2023-01-15T14:31:50.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209946+0000", 2023-01-15T14:31:50.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034601 2023-01-15T14:31:50.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.331 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.331 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:50.332 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:31:50.332 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.544060421000001, 2023-01-15T14:31:50.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:31:50.332 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.332 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.332 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.332 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:31:50.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:31:50.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:31:50.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:31:50.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:31:50.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:31:50.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:31:50.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:31:50.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:50.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:31:50.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:31:50.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:31:50.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:31:50.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:50.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:31:50.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:31:50.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:31:50.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:31:50.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:31:50.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:31:50.341 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.341 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.341 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.341 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:50.341 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:31:50.341 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.540653297999999, 2023-01-15T14:31:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:31:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:31:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:31:50.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:50.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:31:50.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:31:50.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:31:50.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:31:50.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:31:50.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:31:50.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:31:50.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:31:50.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:50.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:31:50.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:31:50.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:31:50.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:31:50.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:31:50.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:31:50.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:50.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:31:50.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:31:50.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:31:50.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:31:50.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:31:50.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:31:50.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:31:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:31:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:31:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:31:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:31:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:31:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:31:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:31:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.540437693000001, 2023-01-15T14:31:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:31:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:31:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:31:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:31:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:31:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:31:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:31:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:31:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:31:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:31:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:31:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:31:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:31:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:31:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:31:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:31:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:31:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:31:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:31:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:31:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:31:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:31:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:31:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:31:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:31:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:31:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:31:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:31:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:31:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:31:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:31:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.540330618999999, 2023-01-15T14:31:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:31:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:31:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:31:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:31:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:31:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:31:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:31:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:31:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:31:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:31:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:31:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:31:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:31:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:31:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:31:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:31:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:31:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:31:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:31:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:31:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:31:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.410091568999999, 2023-01-15T14:31:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:31:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:31:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:31:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:31:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:31:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:31:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:31:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:31:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:31:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:31:50.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:31:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:31:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:31:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:31:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:31:50.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:31:50.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:31:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:31:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:31:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:31:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:31:50.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:31:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:31:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:31:50.378 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.409987032, 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:31:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:31:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:31:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:31:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:31:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:31:50.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:31:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:31:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:31:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:31:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:31:50.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:31:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:31:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:31:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:31:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:31:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:31:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:31:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:31:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.409917101000001, 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:31:50.388 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:31:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:31:50.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:31:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:31:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:31:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:31:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:31:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:31:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:31:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:31:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:50.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:31:50.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:31:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:31:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:31:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:31:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:31:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:31:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:31:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:31:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:31:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:31:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:31:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:31:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:50.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:31:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:31:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:31:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:31:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:31:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:31:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2284 3.10 e216/207 3:094fbf13:::smithi09025264-31:head v 216'274, mlcod=212'269)", 2023-01-15T14:31:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:31:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.064352059, 2023-01-15T14:31:50.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28055179600000002, 2023-01-15T14:31:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:31:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:31:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166254+0000", 2023-01-15T14:31:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166261+0000", 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4080000000000003e-06 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166273+0000", 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.221e-05 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166279+0000", 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8170000000000004e-06 2023-01-15T14:31:50.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365575+0000", 2023-01-15T14:31:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199295791 2023-01-15T14:31:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365598+0000", 2023-01-15T14:31:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3913e-05 2023-01-15T14:31:50.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365737+0000", 2023-01-15T14:31:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138312 2023-01-15T14:31:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365763+0000", 2023-01-15T14:31:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6596000000000001e-05 2023-01-15T14:31:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365803+0000", 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9544e-05 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365842+0000", 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9050000000000001e-05 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.446807+0000", 2023-01-15T14:31:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.080964750000000002 2023-01-15T14:31:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2285 3.10 e216/207 3:094fbf13:::smithi09025264-31:head v 216'275, mlcod=212'269)", 2023-01-15T14:31:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:31:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.0638040279999998, 2023-01-15T14:31:50.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28017387399999999, 2023-01-15T14:31:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:31:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:31:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166802+0000", 2023-01-15T14:31:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166810+0000", 2023-01-15T14:31:50.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6650000000000003e-06 2023-01-15T14:31:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166821+0000", 2023-01-15T14:31:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1129e-05 2023-01-15T14:31:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166827+0000", 2023-01-15T14:31:50.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4640000000000003e-06 2023-01-15T14:31:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365957+0000", 2023-01-15T14:31:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199129158 2023-01-15T14:31:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366005+0000", 2023-01-15T14:31:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8822000000000001e-05 2023-01-15T14:31:50.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366301+0000", 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029544199999999998 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366316+0000", 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5078e-05 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366350+0000", 2023-01-15T14:31:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3575000000000003e-05 2023-01-15T14:31:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366368+0000", 2023-01-15T14:31:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8267e-05 2023-01-15T14:31:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:50.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.446977+0000", 2023-01-15T14:31:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.080608964000000005 2023-01-15T14:31:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.412 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.412 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:50.413 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:31:50.414 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:31:50.415 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:50.409+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:50.515 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:50.509+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:50.573 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.3.0:519 3.10 e223/207)", 2023-01-15T14:31:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:50.522808+0000", 2023-01-15T14:31:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.048790418000000002, 2023-01-15T14:31:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.048812809999999998, 2023-01-15T14:31:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:31:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:50.522808+0000", 2023-01-15T14:31:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:50.522808+0000", 2023-01-15T14:31:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:50.522807+0000", 2023-01-15T14:31:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:50.522814+0000", 2023-01-15T14:31:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2219999999999996e-06 2023-01-15T14:31:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:50.522825+0000", 2023-01-15T14:31:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0518e-05 2023-01-15T14:31:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:50.522832+0000", 2023-01-15T14:31:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0339999999999999e-06 2023-01-15T14:31:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:31:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:31:50.577 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:50.583 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:31:50.601 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:50.615 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:50.609+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:50.617 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:50.636 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:50.654 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:50.674 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:50.679 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:50.679 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:50.690 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:31:50.694 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:50.717 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:50.711+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:50.724 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:50.758 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:50.761 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:31:50.761 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:31:50.761 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:31:50.762 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:31:50.762 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:31:50.762 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:31:50.762 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:31:50.762 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:31:50.801 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:50.801 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:50.811 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:31:50.854 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:50.851+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:50.918 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:50.915+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:50.922 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:50.922 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:50.932 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:31:51.019 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:51.013+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:51.024 INFO:tasks.rados.rados.0.smithi090.stdout:909: done (0 left) 2023-01-15T14:31:51.025 INFO:tasks.rados.rados.0.smithi090.stdout:910: read oid 33 snap -1 2023-01-15T14:31:51.025 INFO:tasks.rados.rados.0.smithi090.stdout:910: expect (ObjNum 327 snap 87 seq_num 327) 2023-01-15T14:31:51.025 INFO:tasks.rados.rados.0.smithi090.stdout:911: read oid 14 snap -1 2023-01-15T14:31:51.025 INFO:tasks.rados.rados.0.smithi090.stdout:911: expect (ObjNum 324 snap 86 seq_num 324) 2023-01-15T14:31:51.025 INFO:tasks.rados.rados.0.smithi090.stdout:912: write (excl) oid 21 current snap is 89 2023-01-15T14:31:51.025 INFO:tasks.rados.rados.0.smithi090.stdout:912: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2023-01-15T14:31:51.032 INFO:tasks.rados.rados.0.smithi090.stdout:912: writing smithi09025264-21 from 497573 to 1126516 tid 1 2023-01-15T14:31:51.040 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:51.041 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:51.041 INFO:tasks.rados.rados.0.smithi090.stdout:912: writing smithi09025264-21 from 1687631 to 2422408 tid 2 2023-01-15T14:31:51.043 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:31:51.049 INFO:tasks.rados.rados.0.smithi090.stdout:912: writing smithi09025264-21 from 3093218 to 3697573 tid 3 2023-01-15T14:31:51.049 INFO:tasks.rados.rados.0.smithi090.stdout:913: rollback oid 2 current snap is 89 2023-01-15T14:31:51.049 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 2 to 62 2023-01-15T14:31:51.049 INFO:tasks.rados.rados.0.smithi090.stdout:911: expect (ObjNum 324 snap 86 seq_num 324) 2023-01-15T14:31:51.050 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:31:51.120 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:51.115+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:51.152 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:51.153 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:51.162 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:31:51.222 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:51.217+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:51.263 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:51.264 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:51.273 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:31:51.322 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:51.319+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:31:51.347 INFO:tasks.rados.rados.0.smithi090.stdout:911: done (3 left) 2023-01-15T14:31:51.347 INFO:tasks.rados.rados.0.smithi090.stdout:914: read oid 18 snap 72 2023-01-15T14:31:51.347 INFO:tasks.rados.rados.0.smithi090.stdout:914: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:51.347 INFO:tasks.rados.rados.0.smithi090.stdout:915: rollback oid 38 current snap is 89 2023-01-15T14:31:51.347 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 38 to 82 2023-01-15T14:31:51.347 INFO:tasks.rados.rados.0.smithi090.stdout:913: finishing rollback tid 0 to smithi09025264-2 2023-01-15T14:31:51.347 INFO:tasks.rados.rados.0.smithi090.stdout:913: finishing rollback tid 1 to smithi09025264-2 2023-01-15T14:31:51.347 INFO:tasks.rados.rados.0.smithi090.stdout:910: expect (ObjNum 327 snap 87 seq_num 327) 2023-01-15T14:31:51.423 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:51.568 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:51.571 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:51.417+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:51.571 INFO:tasks.rados.rados.0.smithi090.stdout:913: finishing rollback tid 2 to smithi09025264-2 2023-01-15T14:31:51.571 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 247 (ObjNum 70 snap 8 seq_num 70) dirty exists 2023-01-15T14:31:51.572 INFO:tasks.rados.rados.0.smithi090.stdout:910: done (4 left) 2023-01-15T14:31:51.572 INFO:tasks.rados.rados.0.smithi090.stdout:913: done (3 left) 2023-01-15T14:31:51.572 INFO:tasks.rados.rados.0.smithi090.stdout:916: snap_create 2023-01-15T14:31:51.572 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:31:51.573 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:31:51.573 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:51.565+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:51.672 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:51.665+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:51.688 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:51.688 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:51.698 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:31:51.974 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:31:51.969+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:31:52.075 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:31:52.071+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:31:52.184 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:52.184 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:52.184 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:31:52.185 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:52.179+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:52.285 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:52.279+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:52.340 INFO:tasks.rados.rados.0.smithi090.stdout:915: finishing rollback tid 1 to smithi09025264-38 2023-01-15T14:31:52.341 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 173 (ObjNum 33 snap 0 seq_num 3195000560) dirty dne 2023-01-15T14:31:52.341 INFO:tasks.rados.rados.0.smithi090.stdout:915: done (2 left) 2023-01-15T14:31:52.341 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:31:52.341 INFO:tasks.rados.rados.0.smithi090.stdout:912: finishing write tid 1 to smithi09025264-21 2023-01-15T14:31:52.341 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:52.341 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:52.342 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:31:52.342 INFO:tasks.rados.rados.0.smithi090.stdout:914: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:31:52.387 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:52.383+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:52.552 INFO:tasks.rados.rados.0.smithi090.stdout:912: finishing write tid 2 to smithi09025264-21 2023-01-15T14:31:52.552 INFO:tasks.rados.rados.0.smithi090.stdout:912: finishing write tid 3 to smithi09025264-21 2023-01-15T14:31:52.552 INFO:tasks.rados.rados.0.smithi090.stdout:914: done (1 left) 2023-01-15T14:31:52.553 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:31:52.553 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:52.553 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:52.554 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:31:52.555 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:52.545+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:52.641 INFO:tasks.rados.rados.0.smithi090.stdout:912: finishing write tid 4 to smithi09025264-21 2023-01-15T14:31:52.998 INFO:tasks.rados.rados.0.smithi090.stdout:912: finishing write tid 5 to smithi09025264-21 2023-01-15T14:31:52.999 INFO:tasks.rados.rados.0.smithi090.stdout:912: oid 21 updating version 0 to 248 2023-01-15T14:31:52.999 INFO:tasks.rados.rados.0.smithi090.stdout:912: oid 21 updating version 248 to 249 2023-01-15T14:31:52.999 INFO:tasks.rados.rados.0.smithi090.stdout:912: oid 21 updating version 249 to 250 2023-01-15T14:31:52.999 INFO:tasks.rados.rados.0.smithi090.stdout:912: oid 21 version 250 is already newer than 247 2023-01-15T14:31:52.999 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 250 (ObjNum 339 snap 89 seq_num 339) dirty exists 2023-01-15T14:31:52.999 INFO:tasks.rados.rados.0.smithi090.stdout:912: left oid 21 (ObjNum 339 snap 89 seq_num 339) 2023-01-15T14:31:52.999 INFO:tasks.rados.rados.0.smithi090.stdout:912: done (0 left) 2023-01-15T14:31:53.000 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:31:53.001 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:52.995+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:53.002 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:53.002 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:53.003 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:31:53.027 INFO:tasks.rados.rados.0.smithi090.stdout:916: done (0 left) 2023-01-15T14:31:53.200 INFO:tasks.rados.rados.0.smithi090.stdout:917: snap_create 2023-01-15T14:31:53.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:31:53.203 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:31:53.203 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:31:53.203 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:31:53.203 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:31:53.203 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:31:53.204 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:31:53.204 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:31:53.197+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:31:53.210 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:31:53.237 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:31:53.406 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:31:53.406 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:31:53.406 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:31:53.407 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:53.408 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:31:53.408 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:31:53.408 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:31:53.408 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.408 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:71 3.f 3:fe04f54c:::smithi09025264-18:head [setxattr _header (56) in=7b,truncate 2285719] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:53.408 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:31:53.408 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 207.06507016800001, 2023-01-15T14:31:53.409 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.82719834800000003, 2023-01-15T14:31:53.409 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:53.409 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:53.409 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:53.409 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:53.409 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:53.409 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 71 2023-01-15T14:31:53.409 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.409 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:53.409 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.409 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:53.410 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:31:53.410 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:53.612 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252665+0000", 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252664+0000", 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:53.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252675+0000", 2023-01-15T14:31:53.614 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1296e-05 2023-01-15T14:31:53.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.614 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:53.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252697+0000", 2023-01-15T14:31:53.614 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1874000000000001e-05 2023-01-15T14:31:53.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.614 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:53.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252708+0000", 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.15e-05 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355413+0000", 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10270497100000001 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.355622+0000", 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000208982 2023-01-15T14:31:53.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.616 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.616 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:53.616 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.758709+0000", 2023-01-15T14:31:53.616 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40308671800000001 2023-01-15T14:31:53.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:53.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.758958+0000", 2023-01-15T14:31:53.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024949 2023-01-15T14:31:53.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:53.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759375+0000", 2023-01-15T14:31:53.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000416208 2023-01-15T14:31:53.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:53.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759708+0000", 2023-01-15T14:31:53.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033326900000000002 2023-01-15T14:31:53.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:53.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759744+0000", 2023-01-15T14:31:53.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6477000000000003e-05 2023-01-15T14:31:53.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:53.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759822+0000", 2023-01-15T14:31:53.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7440000000000004e-05 2023-01-15T14:31:53.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:53.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760139+0000", 2023-01-15T14:31:53.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316996 2023-01-15T14:31:53.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:53.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:53.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:53.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760440+0000", 2023-01-15T14:31:53.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000300945 2023-01-15T14:31:54.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:54.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079578+0000", 2023-01-15T14:31:54.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31913862599999998 2023-01-15T14:31:54.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:54.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079709+0000", 2023-01-15T14:31:54.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013108 2023-01-15T14:31:54.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:54.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.079863+0000", 2023-01-15T14:31:54.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000153819 2023-01-15T14:31:54.007 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.007 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:54.007 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.007 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:72 3.f 3:fe04f54c:::smithi09025264-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:54.007 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:31:54.007 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 207.065010831, 2023-01-15T14:31:54.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.82782374599999997, 2023-01-15T14:31:54.008 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:54.008 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:54.008 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:54.008 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:54.008 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:54.008 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 72 2023-01-15T14:31:54.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.008 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:54.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:54.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:31:54.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:54.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252724+0000", 2023-01-15T14:31:54.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:54.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252723+0000", 2023-01-15T14:31:54.010 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:54.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:54.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252730+0000", 2023-01-15T14:31:54.010 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.703e-06 2023-01-15T14:31:54.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:54.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252747+0000", 2023-01-15T14:31:54.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6739999999999999e-05 2023-01-15T14:31:54.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:54.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.252758+0000", 2023-01-15T14:31:54.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1389999999999999e-05 2023-01-15T14:31:54.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.455798+0000", 2023-01-15T14:31:54.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20303954199999999 2023-01-15T14:31:54.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:54.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.456092+0000", 2023-01-15T14:31:54.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029407199999999999 2023-01-15T14:31:54.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.759852+0000", 2023-01-15T14:31:54.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.303760525 2023-01-15T14:31:54.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:54.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.760038+0000", 2023-01-15T14:31:54.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018584600000000001 2023-01-15T14:31:54.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080005+0000", 2023-01-15T14:31:54.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31996668299999997 2023-01-15T14:31:54.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:54.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080286+0000", 2023-01-15T14:31:54.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028087899999999999 2023-01-15T14:31:54.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:54.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.080548+0000", 2023-01-15T14:31:54.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026241300000000002 2023-01-15T14:31:54.224 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.224 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:54.224 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:75 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 206.960838481, 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92335155400000002, 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:54.225 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 75 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:54.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356896+0000", 2023-01-15T14:31:54.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:54.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356903+0000", 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4619999999999999e-06 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356917+0000", 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4195999999999999e-05 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:54.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356926+0000", 2023-01-15T14:31:54.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:31:54.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457743+0000", 2023-01-15T14:31:54.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100817668 2023-01-15T14:31:54.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:54.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457841+0000", 2023-01-15T14:31:54.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7079999999999999e-05 2023-01-15T14:31:54.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778474+0000", 2023-01-15T14:31:54.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32063301799999999 2023-01-15T14:31:54.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:54.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778627+0000", 2023-01-15T14:31:54.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015389599999999999 2023-01-15T14:31:54.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:54.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778815+0000", 2023-01-15T14:31:54.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018726600000000001 2023-01-15T14:31:54.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:54.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779033+0000", 2023-01-15T14:31:54.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021819999999999999 2023-01-15T14:31:54.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:54.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779051+0000", 2023-01-15T14:31:54.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7640000000000001e-05 2023-01-15T14:31:54.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:54.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:31:54.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3590999999999995e-05 2023-01-15T14:31:54.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:54.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779163+0000", 2023-01-15T14:31:54.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:31:54.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.234 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:54.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.081002+0000", 2023-01-15T14:31:54.234 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30183937799999999 2023-01-15T14:31:54.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.234 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.234 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:54.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280043+0000", 2023-01-15T14:31:54.234 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19904084699999999 2023-01-15T14:31:54.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.234 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.235 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:54.235 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280138+0000", 2023-01-15T14:31:54.235 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4915999999999997e-05 2023-01-15T14:31:54.235 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.235 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.235 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:54.235 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280248+0000", 2023-01-15T14:31:54.235 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110233 2023-01-15T14:31:54.236 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.236 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:54.236 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.236 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.236 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.236 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:76 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:54.236 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:31:54.237 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 206.96079687299999, 2023-01-15T14:31:54.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92420393899999997, 2023-01-15T14:31:54.237 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:54.237 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:54.237 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:54.237 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:54.237 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:54.238 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 76 2023-01-15T14:31:54.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.238 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:54.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:54.430 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:31:54.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.430 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:54.430 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:31:54.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.430 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:54.431 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356937+0000", 2023-01-15T14:31:54.431 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:54.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:54.431 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356948+0000", 2023-01-15T14:31:54.431 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.094e-05 2023-01-15T14:31:54.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:54.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356959+0000", 2023-01-15T14:31:54.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1058e-05 2023-01-15T14:31:54.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:54.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356968+0000", 2023-01-15T14:31:54.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5339999999999999e-06 2023-01-15T14:31:54.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.433 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459524+0000", 2023-01-15T14:31:54.433 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102556147 2023-01-15T14:31:54.433 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.433 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:54.433 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459662+0000", 2023-01-15T14:31:54.433 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000137756 2023-01-15T14:31:54.433 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.433 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.434 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978621+0000", 2023-01-15T14:31:54.434 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51895887600000001 2023-01-15T14:31:54.434 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.434 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.434 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:54.434 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978870+0000", 2023-01-15T14:31:54.434 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024928999999999999 2023-01-15T14:31:54.434 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.434 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.434 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.435 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280761+0000", 2023-01-15T14:31:54.435 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30189113400000001 2023-01-15T14:31:54.435 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.435 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.435 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:54.435 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281023+0000", 2023-01-15T14:31:54.435 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026172800000000003 2023-01-15T14:31:54.436 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.436 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.436 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:54.436 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281142+0000", 2023-01-15T14:31:54.436 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000119296 2023-01-15T14:31:54.436 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.436 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:54.436 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.436 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.436 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.437 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:91 3.d 3:bc1c6b44:::smithi09025264-23:head [setxattr _header (56) in=7b,truncate 2369754] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:54.437 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:31:54.437 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 206.914308492, 2023-01-15T14:31:54.437 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.844937151, 2023-01-15T14:31:54.437 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:54.437 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:54.437 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:54.437 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:54.437 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:54.437 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 91 2023-01-15T14:31:54.437 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403427+0000", 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.438 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:54.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403425+0000", 2023-01-15T14:31:54.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:54.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:54.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403434+0000", 2023-01-15T14:31:54.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3299999999999999e-06 2023-01-15T14:31:54.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.440 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:54.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403446+0000", 2023-01-15T14:31:54.440 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2176e-05 2023-01-15T14:31:54.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.440 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:54.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403453+0000", 2023-01-15T14:31:54.440 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5830000000000003e-06 2023-01-15T14:31:54.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.441 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403654+0000", 2023-01-15T14:31:54.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020136000000000001 2023-01-15T14:31:54.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:54.441 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403807+0000", 2023-01-15T14:31:54.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015342599999999999 2023-01-15T14:31:54.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907836+0000", 2023-01-15T14:31:54.442 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50402854699999999 2023-01-15T14:31:54.442 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.442 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:54.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.907934+0000", 2023-01-15T14:31:54.442 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8472000000000002e-05 2023-01-15T14:31:54.442 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 5,6", 2023-01-15T14:31:54.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908096+0000", 2023-01-15T14:31:54.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000162139 2023-01-15T14:31:54.443 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:54.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908335+0000", 2023-01-15T14:31:54.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000238891 2023-01-15T14:31:54.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.444 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:54.444 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908350+0000", 2023-01-15T14:31:54.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4868000000000001e-05 2023-01-15T14:31:54.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.444 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:54.444 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.908416+0000", 2023-01-15T14:31:54.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5502999999999999e-05 2023-01-15T14:31:54.445 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.445 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:54.445 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.952370+0000", 2023-01-15T14:31:54.445 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.043954663999999997 2023-01-15T14:31:54.445 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.445 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:54.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.008749+0000", 2023-01-15T14:31:54.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.056378696999999998 2023-01-15T14:31:54.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:54.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248164+0000", 2023-01-15T14:31:54.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23941479500000001 2023-01-15T14:31:54.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:54.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248244+0000", 2023-01-15T14:31:54.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9616000000000005e-05 2023-01-15T14:31:54.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:54.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248364+0000", 2023-01-15T14:31:54.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120167 2023-01-15T14:31:54.838 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.838 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:54.838 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.839 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:92 3.d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:54.839 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:31:54.839 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 206.914272799, 2023-01-15T14:31:54.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.84526834699999998, 2023-01-15T14:31:54.839 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:54.839 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:54.839 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:54.839 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:54.839 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:54.840 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 92 2023-01-15T14:31:54.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.840 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:54.840 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:54.840 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:31:54.840 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.840 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:54.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403462+0000", 2023-01-15T14:31:54.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:54.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403461+0000", 2023-01-15T14:31:54.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:54.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:54.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403466+0000", 2023-01-15T14:31:54.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.34e-06 2023-01-15T14:31:54.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:54.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403476+0000", 2023-01-15T14:31:54.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8530000000000004e-06 2023-01-15T14:31:54.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:54.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.403492+0000", 2023-01-15T14:31:54.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6646999999999998e-05 2023-01-15T14:31:54.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501333+0000", 2023-01-15T14:31:54.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097840420999999997 2023-01-15T14:31:54.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:54.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501483+0000", 2023-01-15T14:31:54.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150123 2023-01-15T14:31:54.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.007528+0000", 2023-01-15T14:31:54.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50604486299999996 2023-01-15T14:31:54.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:54.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.007649+0000", 2023-01-15T14:31:54.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001217 2023-01-15T14:31:54.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248441+0000", 2023-01-15T14:31:54.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.240791267 2023-01-15T14:31:54.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:54.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248561+0000", 2023-01-15T14:31:54.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120516 2023-01-15T14:31:54.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:54.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.248731+0000", 2023-01-15T14:31:54.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016937000000000001 2023-01-15T14:31:54.846 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.847 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:54.847 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:54.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.847 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1076 3.f 3:f87c0183:::smithi09025264-50:head [setxattr _header (57) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:31:54.847 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:31:54.847 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 130.55908811800001, 2023-01-15T14:31:54.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.88473693200000003, 2023-01-15T14:31:54.847 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:54.847 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:54.847 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:54.848 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:54.848 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:54.848 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1076 2023-01-15T14:31:54.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.848 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:54.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:54.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:31:54.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:54.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:31:54.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:54.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:54.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758646+0000", 2023-01-15T14:31:54.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:54.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758653+0000", 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8029999999999997e-06 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758670+0000", 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6158999999999999e-05 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:54.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758677+0000", 2023-01-15T14:31:54.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9840000000000004e-06 2023-01-15T14:31:54.851 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.851 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.851 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:54.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848049+0000", 2023-01-15T14:31:54.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089372163000000004 2023-01-15T14:31:54.851 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:54.851 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:54.851 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:54.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848056+0000", 2023-01-15T14:31:54.852 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0360000000000001e-06 2023-01-15T14:31:54.852 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.046 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.047 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951562+0000", 2023-01-15T14:31:55.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10350670100000001 2023-01-15T14:31:55.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:55.047 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951726+0000", 2023-01-15T14:31:55.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000163311 2023-01-15T14:31:55.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:55.048 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951958+0000", 2023-01-15T14:31:55.048 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000232525 2023-01-15T14:31:55.048 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.048 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.048 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:55.048 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952168+0000", 2023-01-15T14:31:55.048 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020949799999999999 2023-01-15T14:31:55.048 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.048 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.048 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:55.048 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952186+0000", 2023-01-15T14:31:55.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7714e-05 2023-01-15T14:31:55.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.049 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:55.049 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952254+0000", 2023-01-15T14:31:55.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7998999999999997e-05 2023-01-15T14:31:55.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.049 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952278+0000", 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4909000000000002e-05 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065836+0000", 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.113557081 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643196+0000", 2023-01-15T14:31:55.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.57736012999999997 2023-01-15T14:31:55.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:55.051 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643278+0000", 2023-01-15T14:31:55.051 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2671999999999997e-05 2023-01-15T14:31:55.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:55.052 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643384+0000", 2023-01-15T14:31:55.052 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010551400000000001 2023-01-15T14:31:55.052 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.052 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:55.052 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.052 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.052 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.052 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1077 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:31:55.052 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:31:55.052 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 130.55904929900001, 2023-01-15T14:31:55.052 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.98557123499999999, 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1077 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:31:55.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:55.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:31:55.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:55.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758685+0000", 2023-01-15T14:31:55.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:55.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:55.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758689+0000", 2023-01-15T14:31:55.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2760000000000002e-06 2023-01-15T14:31:55.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:55.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758704+0000", 2023-01-15T14:31:55.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4187e-05 2023-01-15T14:31:55.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758710+0000", 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4330000000000001e-06 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848086+0000", 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089376466000000002 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:55.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848093+0000", 2023-01-15T14:31:55.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4269999999999996e-06 2023-01-15T14:31:55.057 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.057 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952346+0000", 2023-01-15T14:31:55.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104252758 2023-01-15T14:31:55.057 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.057 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:55.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952427+0000", 2023-01-15T14:31:55.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1786000000000006e-05 2023-01-15T14:31:55.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.058 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.058 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.058 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743602+0000", 2023-01-15T14:31:55.058 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.79117428099999998 2023-01-15T14:31:55.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.058 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.058 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:55.058 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743892+0000", 2023-01-15T14:31:55.058 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029067200000000001 2023-01-15T14:31:55.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.059 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.059 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:55.059 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.744257+0000", 2023-01-15T14:31:55.059 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000364619 2023-01-15T14:31:55.059 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.059 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:55.059 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.059 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.060 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.060 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1087 3.f 3:fd983206:::smithi09025264-7:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:31:55.060 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:31:55.060 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 130.55901481699999, 2023-01-15T14:31:55.060 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0859964360000001, 2023-01-15T14:31:55.060 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:55.060 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:55.060 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:55.060 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:55.060 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:55.060 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1087 2023-01-15T14:31:55.061 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.061 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:55.061 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.061 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:55.061 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:31:55.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.061 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.061 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.061 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:55.061 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:31:55.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758724+0000", 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7740000000000004e-06 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.062 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.063 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:55.063 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758738+0000", 2023-01-15T14:31:55.063 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.33e-05 2023-01-15T14:31:55.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.063 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:55.063 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758744+0000", 2023-01-15T14:31:55.063 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2859999999999996e-06 2023-01-15T14:31:55.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.063 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.064 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848121+0000", 2023-01-15T14:31:55.064 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089377097000000003 2023-01-15T14:31:55.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.064 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.064 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:55.064 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848127+0000", 2023-01-15T14:31:55.064 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3500000000000002e-06 2023-01-15T14:31:55.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.064 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.064 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.064 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952454+0000", 2023-01-15T14:31:55.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104326167 2023-01-15T14:31:55.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:55.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952520+0000", 2023-01-15T14:31:55.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6166000000000003e-05 2023-01-15T14:31:55.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:31:55.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.063003+0000", 2023-01-15T14:31:55.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.110482916 2023-01-15T14:31:55.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:55.250 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065265+0000", 2023-01-15T14:31:55.250 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022626370000000001 2023-01-15T14:31:55.250 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.250 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.250 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:55.251 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065291+0000", 2023-01-15T14:31:55.251 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6095e-05 2023-01-15T14:31:55.251 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.251 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.251 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:55.252 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.066153+0000", 2023-01-15T14:31:55.252 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086127100000000004 2023-01-15T14:31:55.252 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.252 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.252 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:55.253 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166533+0000", 2023-01-15T14:31:55.253 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100380177 2023-01-15T14:31:55.253 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.253 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.253 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:55.254 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166646+0000", 2023-01-15T14:31:55.254 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112717 2023-01-15T14:31:55.254 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.254 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.254 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:55.254 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844498+0000", 2023-01-15T14:31:55.255 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67785237399999998 2023-01-15T14:31:55.255 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.255 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.255 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:55.255 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844605+0000", 2023-01-15T14:31:55.256 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106691 2023-01-15T14:31:55.256 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:55.256 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844717+0000", 2023-01-15T14:31:55.256 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011203499999999999 2023-01-15T14:31:55.257 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.257 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:55.257 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.257 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.257 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.258 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1088 3.f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:31:55.258 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:31:55.258 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 130.55895915100001, 2023-01-15T14:31:55.258 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.086312696, 2023-01-15T14:31:55.258 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:55.258 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:55.259 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:55.259 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:55.259 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:55.259 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1088 2023-01-15T14:31:55.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.260 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:55.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:55.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:31:55.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:55.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:31:55.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:55.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758775+0000", 2023-01-15T14:31:55.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:55.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:55.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758786+0000", 2023-01-15T14:31:55.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.076e-05 2023-01-15T14:31:55.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:55.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758809+0000", 2023-01-15T14:31:55.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3706999999999999e-05 2023-01-15T14:31:55.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.454 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.454 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:55.454 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758821+0000", 2023-01-15T14:31:55.454 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1976e-05 2023-01-15T14:31:55.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.455 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.455 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878534+0000", 2023-01-15T14:31:55.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.119712966 2023-01-15T14:31:55.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.455 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:55.455 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878540+0000", 2023-01-15T14:31:55.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0680000000000002e-06 2023-01-15T14:31:55.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.456 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.456 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952693+0000", 2023-01-15T14:31:55.456 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.074153067000000003 2023-01-15T14:31:55.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.456 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:55.456 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952735+0000", 2023-01-15T14:31:55.456 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1792999999999998e-05 2023-01-15T14:31:55.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065518+0000", 2023-01-15T14:31:55.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.11278334499999999 2023-01-15T14:31:55.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:55.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065687+0000", 2023-01-15T14:31:55.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016807700000000001 2023-01-15T14:31:55.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844786+0000", 2023-01-15T14:31:55.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.77909980300000004 2023-01-15T14:31:55.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:55.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844944+0000", 2023-01-15T14:31:55.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157424 2023-01-15T14:31:55.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:55.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.845089+0000", 2023-01-15T14:31:55.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144814 2023-01-15T14:31:55.460 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.460 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:55.461 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.461 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:16 3.13 3:ca393a9e:::smithi09025264-47:head [assert-version v193,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:55.461 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:31:55.462 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 94.011981726000002, 2023-01-15T14:31:55.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8970339479999998, 2023-01-15T14:31:55.462 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:55.462 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:55.462 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:55.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:55.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:31:55.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.464 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:55.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:31:55.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:55.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305751+0000", 2023-01-15T14:31:55.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:55.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:55.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305763+0000", 2023-01-15T14:31:55.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2309e-05 2023-01-15T14:31:55.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:55.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305788+0000", 2023-01-15T14:31:55.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4388000000000001e-05 2023-01-15T14:31:55.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:55.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305799+0000", 2023-01-15T14:31:55.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1304000000000001e-05 2023-01-15T14:31:55.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196425+0000", 2023-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8906259830000001 2023-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196673+0000", 2023-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024747499999999998 2023-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.202787+0000", 2023-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0061146509999999996 2023-01-15T14:31:55.662 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.665 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1480 3.13 3:cbfd2d71:::smithi09025264-37:head [write 581575~705262,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:55.665 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:31:55.665 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 93.978713998000003, 2023-01-15T14:31:55.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.375425221, 2023-01-15T14:31:55.665 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:55.665 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:55.666 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:55.666 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:55.666 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:55.666 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1480 2023-01-15T14:31:55.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.666 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:55.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:55.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:31:55.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:55.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:31:55.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:55.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339019+0000", 2023-01-15T14:31:55.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:55.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:55.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339659+0000", 2023-01-15T14:31:55.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000639742 2023-01-15T14:31:55.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:55.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339690+0000", 2023-01-15T14:31:55.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1594000000000003e-05 2023-01-15T14:31:55.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:55.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339703+0000", 2023-01-15T14:31:55.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2828e-05 2023-01-15T14:31:55.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.510837+0000", 2023-01-15T14:31:55.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1711338759999999 2023-01-15T14:31:55.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:55.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511048+0000", 2023-01-15T14:31:55.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021057899999999999 2023-01-15T14:31:55.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:55.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511325+0000", 2023-01-15T14:31:55.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027690400000000002 2023-01-15T14:31:55.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:55.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512092+0000", 2023-01-15T14:31:55.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00076720800000000002 2023-01-15T14:31:55.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:55.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512113+0000", 2023-01-15T14:31:55.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0917e-05 2023-01-15T14:31:55.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:55.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512439+0000", 2023-01-15T14:31:55.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032636299999999998 2023-01-15T14:31:55.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:55.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512871+0000", 2023-01-15T14:31:55.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000432087 2023-01-15T14:31:55.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:55.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.514165+0000", 2023-01-15T14:31:55.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001293726 2023-01-15T14:31:55.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:55.862 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714137+0000", 2023-01-15T14:31:55.862 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19997229499999999 2023-01-15T14:31:55.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714233+0000", 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6062000000000003e-05 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714446+0000", 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212952 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:55.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.865 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1481 3.13 3:cbfd2d71:::smithi09025264-37:head [write 1733272~707409] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:55.866 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:31:55.866 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 93.971864714999995, 2023-01-15T14:31:55.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.572522502, 2023-01-15T14:31:55.866 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:55.866 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:55.866 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:55.866 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:55.866 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:55.866 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1481 2023-01-15T14:31:55.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.866 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:55.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:55.867 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:31:55.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:55.867 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:31:55.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:55.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:55.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345868+0000", 2023-01-15T14:31:55.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:31:55.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:55.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346316+0000", 2023-01-15T14:31:55.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044813399999999999 2023-01-15T14:31:55.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:55.869 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346348+0000", 2023-01-15T14:31:55.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:31:55.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.869 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.869 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:55.869 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346362+0000", 2023-01-15T14:31:55.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.375e-05 2023-01-15T14:31:55.870 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.870 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.870 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.870 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512276+0000", 2023-01-15T14:31:55.870 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1659136829999999 2023-01-15T14:31:55.870 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.870 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.870 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512427+0000", 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015088699999999999 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714565+0000", 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20213841599999999 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714732+0000", 2023-01-15T14:31:55.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016676700000000001 2023-01-15T14:31:55.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:55.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714964+0000", 2023-01-15T14:31:55.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000231678 2023-01-15T14:31:55.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:55.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:55.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:55.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715499+0000", 2023-01-15T14:31:55.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053513300000000003 2023-01-15T14:31:56.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:56.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715523+0000", 2023-01-15T14:31:56.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4266000000000001e-05 2023-01-15T14:31:56.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:56.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715867+0000", 2023-01-15T14:31:56.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034419399999999998 2023-01-15T14:31:56.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:56.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.816932+0000", 2023-01-15T14:31:56.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10106498999999999 2023-01-15T14:31:56.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.068 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:56.068 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906374+0000", 2023-01-15T14:31:56.068 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089441442999999995 2023-01-15T14:31:56.068 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.068 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.068 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:56.068 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918050+0000", 2023-01-15T14:31:56.068 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.011676677999999999 2023-01-15T14:31:56.068 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.068 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.068 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:56.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918218+0000", 2023-01-15T14:31:56.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016802000000000001 2023-01-15T14:31:56.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.069 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.070 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:56.070 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918393+0000", 2023-01-15T14:31:56.070 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017447 2023-01-15T14:31:56.070 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:56.070 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:56.070 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:56.070 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.070 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1482 3.13 3:cbfd2d71:::smithi09025264-37:head [write 3195938~437497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:56.070 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:31:56.071 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 93.965428857000006, 2023-01-15T14:31:56.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5664780289999998, 2023-01-15T14:31:56.071 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:56.071 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:56.071 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:56.071 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:56.071 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:56.071 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1482 2023-01-15T14:31:56.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.071 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:56.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:56.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:31:56.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:56.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:56.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:31:56.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:56.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.073 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:56.073 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352275+0000", 2023-01-15T14:31:56.073 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999695 2023-01-15T14:31:56.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.073 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.073 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:56.073 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352644+0000", 2023-01-15T14:31:56.073 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036870300000000001 2023-01-15T14:31:56.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.073 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.074 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:56.074 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352676+0000", 2023-01-15T14:31:56.074 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1726000000000001e-05 2023-01-15T14:31:56.074 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.074 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.074 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:56.074 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352689+0000", 2023-01-15T14:31:56.074 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3617e-05 2023-01-15T14:31:56.074 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.074 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.074 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512602+0000", 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1599126690000001 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512839+0000", 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023676900000000001 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715649+0000", 2023-01-15T14:31:56.075 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20281044600000001 2023-01-15T14:31:56.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:56.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715777+0000", 2023-01-15T14:31:56.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012738800000000001 2023-01-15T14:31:56.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:56.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715995+0000", 2023-01-15T14:31:56.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000217858 2023-01-15T14:31:56.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:56.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716694+0000", 2023-01-15T14:31:56.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00069956700000000001 2023-01-15T14:31:56.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:56.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716709+0000", 2023-01-15T14:31:56.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4362000000000001e-05 2023-01-15T14:31:56.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716936+0000", 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022744200000000001 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.818876+0000", 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101939635 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:56.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906390+0000", 2023-01-15T14:31:56.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087513987000000001 2023-01-15T14:31:56.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.079 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:56.079 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918577+0000", 2023-01-15T14:31:56.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012187066 2023-01-15T14:31:56.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.079 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:56.079 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918697+0000", 2023-01-15T14:31:56.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120326 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918784+0000", 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7200999999999994e-05 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1483 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3781575] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:56.080 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:31:56.081 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 93.965020705000001, 2023-01-15T14:31:56.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5663522429999999, 2023-01-15T14:31:56.081 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:56.081 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:56.081 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:56.081 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:56.081 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:56.081 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1483 2023-01-15T14:31:56.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.081 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:56.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:56.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:31:56.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:56.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:56.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:31:56.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:56.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:56.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352713+0000", 2023-01-15T14:31:56.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:56.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:56.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352720+0000", 2023-01-15T14:31:56.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7959999999999998e-06 2023-01-15T14:31:56.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:56.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352735+0000", 2023-01-15T14:31:56.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.455e-05 2023-01-15T14:31:56.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:56.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352741+0000", 2023-01-15T14:31:56.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8360000000000003e-06 2023-01-15T14:31:56.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:56.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512901+0000", 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1601594980000001 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.513067+0000", 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016626 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716977+0000", 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20391002799999999 2023-01-15T14:31:56.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.086 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:56.086 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717146+0000", 2023-01-15T14:31:56.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168851 2023-01-15T14:31:56.086 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:31:56.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717467+0000", 2023-01-15T14:31:56.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032135600000000002 2023-01-15T14:31:56.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:56.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717800+0000", 2023-01-15T14:31:56.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033217400000000002 2023-01-15T14:31:56.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:56.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717822+0000", 2023-01-15T14:31:56.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2308999999999999e-05 2023-01-15T14:31:56.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:56.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717970+0000", 2023-01-15T14:31:56.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014788099999999999 2023-01-15T14:31:56.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:56.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.819135+0000", 2023-01-15T14:31:56.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116478 2023-01-15T14:31:56.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:56.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906408+0000", 2023-01-15T14:31:56.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087272902999999999 2023-01-15T14:31:56.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:56.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918887+0000", 2023-01-15T14:31:56.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012479965000000001 2023-01-15T14:31:56.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:56.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918959+0000", 2023-01-15T14:31:56.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1978999999999996e-05 2023-01-15T14:31:56.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.281 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:56.281 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.919067+0000", 2023-01-15T14:31:56.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107083 2023-01-15T14:31:56.281 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:56.281 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:56.281 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:56.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.282 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1485 3.d 3:bc1c6b44:::smithi09025264-23:head [write 598382~780511,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:56.282 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:31:56.282 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 93.957550271000002, 2023-01-15T14:31:56.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5474121890000001, 2023-01-15T14:31:56.282 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:56.282 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:56.282 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:56.282 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:56.282 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:56.283 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1485 2023-01-15T14:31:56.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.283 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:56.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:56.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:31:56.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:56.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:56.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:31:56.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:56.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.284 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:56.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360168+0000", 2023-01-15T14:31:56.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999828 2023-01-15T14:31:56.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360790+0000", 2023-01-15T14:31:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062269099999999998 2023-01-15T14:31:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360822+0000", 2023-01-15T14:31:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1903000000000003e-05 2023-01-15T14:31:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360836+0000", 2023-01-15T14:31:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3380000000000001e-05 2023-01-15T14:31:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096664+0000", 2023-01-15T14:31:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7358282320000002 2023-01-15T14:31:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:31:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096683+0000", 2023-01-15T14:31:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9063999999999999e-05 2023-01-15T14:31:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:56.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704561+0000", 2023-01-15T14:31:56.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60787808399999999 2023-01-15T14:31:56.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:56.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704846+0000", 2023-01-15T14:31:56.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028527400000000002 2023-01-15T14:31:56.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:56.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705135+0000", 2023-01-15T14:31:56.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028875400000000001 2023-01-15T14:31:56.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:56.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705735+0000", 2023-01-15T14:31:56.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059956899999999999 2023-01-15T14:31:56.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:56.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705761+0000", 2023-01-15T14:31:56.478 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:31:56.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:56.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.706129+0000", 2023-01-15T14:31:56.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036822299999999999 2023-01-15T14:31:56.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:56.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906331+0000", 2023-01-15T14:31:56.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200202091 2023-01-15T14:31:56.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:56.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907111+0000", 2023-01-15T14:31:56.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077990799999999995 2023-01-15T14:31:56.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:56.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907379+0000", 2023-01-15T14:31:56.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026804800000000001 2023-01-15T14:31:56.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:56.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907529+0000", 2023-01-15T14:31:56.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015022700000000001 2023-01-15T14:31:56.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:56.484 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907597+0000", 2023-01-15T14:31:56.484 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7785000000000002e-05 2023-01-15T14:31:56.484 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:56.484 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:56.484 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:56.484 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.484 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.484 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1486 3.d 3:bc1c6b44:::smithi09025264-23:head [write 2075401~451067] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:56.485 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:31:56.485 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 93.951154223000003, 2023-01-15T14:31:56.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6461318330000001, 2023-01-15T14:31:56.485 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:56.485 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:56.485 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:56.485 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:56.485 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:56.485 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1486 2023-01-15T14:31:56.486 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.683 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:56.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:56.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:31:56.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:56.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:56.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:31:56.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:56.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:56.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366579+0000", 2023-01-15T14:31:56.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:56.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:56.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366845+0000", 2023-01-15T14:31:56.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026610499999999999 2023-01-15T14:31:56.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:56.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366878+0000", 2023-01-15T14:31:56.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2750000000000003e-05 2023-01-15T14:31:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.467009+0000", 2023-01-15T14:31:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100131071 2023-01-15T14:31:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096793+0000", 2023-01-15T14:31:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6297846869999999 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096806+0000", 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3077e-05 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907957+0000", 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81115055400000002 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908229+0000", 2023-01-15T14:31:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027187499999999998 2023-01-15T14:31:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908496+0000", 2023-01-15T14:31:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026754200000000001 2023-01-15T14:31:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909025+0000", 2023-01-15T14:31:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052865200000000001 2023-01-15T14:31:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909046+0000", 2023-01-15T14:31:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0713e-05 2023-01-15T14:31:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909294+0000", 2023-01-15T14:31:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024864899999999998 2023-01-15T14:31:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:31:57.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.911607+0000", 2023-01-15T14:31:57.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023128939999999998 2023-01-15T14:31:57.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:57.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012076+0000", 2023-01-15T14:31:57.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100468293 2023-01-15T14:31:57.097 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.097 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:57.097 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012426+0000", 2023-01-15T14:31:57.099 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035044299999999998 2023-01-15T14:31:57.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012532+0000", 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106142 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012713+0000", 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018048599999999999 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:57.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.101 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.101 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1496 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:57.101 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:31:57.101 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 93.581511374000002, 2023-01-15T14:31:57.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870616037, 2023-01-15T14:31:57.101 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:57.101 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:57.101 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:57.101 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:57.101 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:57.101 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1496 2023-01-15T14:31:57.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.103 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:57.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:57.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:31:57.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:57.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:57.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:31:57.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:57.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.104 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.104 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:57.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736220+0000", 2023-01-15T14:31:57.104 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:31:57.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.104 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.104 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:57.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736238+0000", 2023-01-15T14:31:57.104 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7156e-05 2023-01-15T14:31:57.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736275+0000", 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7876000000000001e-05 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736290+0000", 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4449e-05 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.105 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:57.106 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602217+0000", 2023-01-15T14:31:57.106 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8659270299999999 2023-01-15T14:31:57.106 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.106 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.106 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:57.106 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602468+0000", 2023-01-15T14:31:57.106 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000251315 2023-01-15T14:31:57.106 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.106 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.106 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:57.106 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.606840+0000", 2023-01-15T14:31:57.107 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0043715009999999999 2023-01-15T14:31:57.107 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:57.107 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:57.107 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:57.107 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.107 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.107 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1497 3.11 3:8889e450:::smithi09025264-32:head [read 0~2991646,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:57.107 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:31:57.107 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 93.581423920000006, 2023-01-15T14:31:57.107 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738359249999998, 2023-01-15T14:31:57.107 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1497 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:57.108 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736310+0000", 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:57.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736317+0000", 2023-01-15T14:31:57.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7240000000000003e-06 2023-01-15T14:31:57.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:57.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736334+0000", 2023-01-15T14:31:57.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6878999999999999e-05 2023-01-15T14:31:57.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:57.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736341+0000", 2023-01-15T14:31:57.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0129999999999996e-06 2023-01-15T14:31:57.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:57.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607070+0000", 2023-01-15T14:31:57.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870728873 2023-01-15T14:31:57.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:57.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607226+0000", 2023-01-15T14:31:57.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015595 2023-01-15T14:31:57.112 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:57.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610147+0000", 2023-01-15T14:31:57.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.002921416 2023-01-15T14:31:57.298 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:57.298 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:57.298 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:57.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1498 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 93.581383814000006, 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8752667839999999, 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1498 2023-01-15T14:31:57.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.300 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:31:57.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:31:57.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:31:57.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736350+0000", 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:31:57.304 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736355+0000", 2023-01-15T14:31:57.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4900000000000002e-06 2023-01-15T14:31:57.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:31:57.304 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736363+0000", 2023-01-15T14:31:57.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2770000000000005e-06 2023-01-15T14:31:57.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:31:57.305 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736369+0000", 2023-01-15T14:31:57.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7400000000000001e-06 2023-01-15T14:31:57.305 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.305 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:31:57.305 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610234+0000", 2023-01-15T14:31:57.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738651439999998 2023-01-15T14:31:57.305 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.305 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610402+0000", 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168283 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.611618+0000", 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001215647 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:31:57.306 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:31:57.307 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:57.307 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:53.401+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:57.307 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:31:57.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:31:57.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:31:57.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:31:57.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:31:57.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:31:57.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:31:57.308 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:31:57.308 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:31:57.309 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:31:57.309 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:31:57.310 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2023-01-15T14:31:57.310 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-01-15T14:31:57.310 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-01-15T14:31:57.310 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2023-01-15T14:31:57.310 INFO:tasks.rados.rados.0.smithi090.stdout:917: done (0 left) 2023-01-15T14:31:57.310 INFO:tasks.rados.rados.0.smithi090.stdout:918: write oid 13 current snap is 91 2023-01-15T14:31:57.311 INFO:tasks.rados.rados.0.smithi090.stdout:918: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2023-01-15T14:31:57.311 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:31:57.311 INFO:tasks.rados.rados.0.smithi090.stdout:918: writing smithi09025264-13 from 514380 to 1218572 tid 1 2023-01-15T14:31:57.311 INFO:tasks.rados.rados.0.smithi090.stdout:918: writing smithi09025264-13 from 1629760 to 2108195 tid 2 2023-01-15T14:31:57.314 INFO:tasks.rados.rados.0.smithi090.stdout:918: writing smithi09025264-13 from 2604288 to 3046890 tid 3 2023-01-15T14:31:57.500 INFO:tasks.rados.rados.0.smithi090.stdout:919: snap_remove snap 90 2023-01-15T14:31:57.501 INFO:tasks.rados.rados.0.smithi090.stdout:918: finishing write tid 1 to smithi09025264-13 2023-01-15T14:31:57.501 INFO:tasks.rados.rados.0.smithi090.stdout:918: finishing write tid 2 to smithi09025264-13 2023-01-15T14:31:57.501 INFO:tasks.rados.rados.0.smithi090.stdout:918: finishing write tid 3 to smithi09025264-13 2023-01-15T14:31:57.501 INFO:tasks.rados.rados.0.smithi090.stdout:918: finishing write tid 4 to smithi09025264-13 2023-01-15T14:31:57.501 INFO:tasks.rados.rados.0.smithi090.stdout:918: finishing write tid 5 to smithi09025264-13 2023-01-15T14:31:57.501 INFO:tasks.rados.rados.0.smithi090.stdout:918: oid 13 updating version 0 to 242 2023-01-15T14:31:57.501 INFO:tasks.rados.rados.0.smithi090.stdout:918: oid 13 updating version 242 to 243 2023-01-15T14:31:57.501 INFO:tasks.rados.rados.0.smithi090.stdout:918: oid 13 updating version 243 to 244 2023-01-15T14:31:57.501 INFO:tasks.rados.rados.0.smithi090.stdout:918: oid 13 updating version 244 to 245 2023-01-15T14:31:57.501 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 245 (ObjNum 340 snap 91 seq_num 340) dirty exists 2023-01-15T14:31:57.501 INFO:tasks.rados.rados.0.smithi090.stdout:918: left oid 13 (ObjNum 340 snap 91 seq_num 340) 2023-01-15T14:31:57.502 INFO:tasks.rados.rados.0.smithi090.stdout:918: done (1 left) 2023-01-15T14:31:57.502 INFO:tasks.rados.rados.0.smithi090.stdout:919: done (0 left) 2023-01-15T14:31:57.502 INFO:tasks.rados.rados.0.smithi090.stdout:920: write (excl) oid 43 current snap is 91 2023-01-15T14:31:57.502 INFO:tasks.rados.rados.0.smithi090.stdout:920: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2023-01-15T14:31:57.502 INFO:tasks.rados.rados.0.smithi090.stdout:920: writing smithi09025264-43 from 531187 to 1310628 tid 1 2023-01-15T14:31:57.502 INFO:tasks.rados.rados.0.smithi090.stdout:920: writing smithi09025264-43 from 2088242 to 2826688 tid 2 2023-01-15T14:31:57.502 INFO:tasks.rados.rados.0.smithi090.stdout:920: writing smithi09025264-43 from 3431711 to 3731187 tid 3 2023-01-15T14:31:57.502 INFO:tasks.rados.rados.0.smithi090.stdout:921: write (excl) oid 11 current snap is 91 2023-01-15T14:31:57.502 INFO:tasks.rados.rados.0.smithi090.stdout:921: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2023-01-15T14:31:57.502 INFO:tasks.rados.rados.0.smithi090.stdout:921: writing smithi09025264-11 from 547994 to 1002684 tid 1 2023-01-15T14:31:57.503 INFO:tasks.rados.rados.0.smithi090.stdout:921: writing smithi09025264-11 from 1746724 to 2228828 tid 2 2023-01-15T14:31:57.503 INFO:tasks.rados.rados.0.smithi090.stdout:921: writing smithi09025264-11 from 2659134 to 3124280 tid 3 2023-01-15T14:31:57.503 INFO:tasks.rados.rados.0.smithi090.stdout:921: writing smithi09025264-11 from 3605976 to 3747994 tid 4 2023-01-15T14:31:57.503 INFO:tasks.rados.rados.0.smithi090.stdout:922: snap_remove snap 82 2023-01-15T14:31:57.503 INFO:tasks.rados.rados.0.smithi090.stdout:922: done (2 left) 2023-01-15T14:31:57.503 INFO:tasks.rados.rados.0.smithi090.stdout:923: read oid 24 snap -1 2023-01-15T14:31:57.503 INFO:tasks.rados.rados.0.smithi090.stdout:923: expect deleted 2023-01-15T14:31:57.503 INFO:tasks.rados.rados.0.smithi090.stdout:924: snap_remove snap 62 2023-01-15T14:31:57.503 INFO:tasks.rados.rados.0.smithi090.stdout:924: done (3 left) 2023-01-15T14:31:57.503 INFO:tasks.rados.rados.0.smithi090.stdout:925: write oid 3 current snap is 91 2023-01-15T14:31:57.504 INFO:tasks.rados.rados.0.smithi090.stdout:925: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2023-01-15T14:31:57.504 INFO:tasks.rados.rados.0.smithi090.stdout:925: writing smithi09025264-3 from 564801 to 1211093 tid 1 2023-01-15T14:31:57.504 INFO:tasks.rados.rados.0.smithi090.stdout:925: writing smithi09025264-3 from 1921559 to 2663674 tid 2 2023-01-15T14:31:57.504 INFO:tasks.rados.rados.0.smithi090.stdout:925: writing smithi09025264-3 from 3202910 to 3764801 tid 3 2023-01-15T14:31:57.504 INFO:tasks.rados.rados.0.smithi090.stdout:926: snap_remove snap 89 2023-01-15T14:31:57.505 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:57.305+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:57.506 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:31:57.503+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:31:57.606 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:31:57.601+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:31:57.671 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:57.671 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:31:57.671 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:57.672 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:31:57.672 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-01-15T14:31:57.672 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:31:57.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:31:57.722 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:31:57.719+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:31:57.835 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:57.836 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:31:57.836 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:31:57.836 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:31:57.836 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:31:57.837 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:31:57.837 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:31:57.829+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:31:57.844 INFO:tasks.rados.rados.0.smithi090.stdout:926: done (4 left) 2023-01-15T14:31:57.995 INFO:tasks.rados.rados.0.smithi090.stdout:927: rollback oid 29 current snap is 91 2023-01-15T14:31:57.995 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 29 to 88 2023-01-15T14:31:57.995 INFO:tasks.rados.rados.0.smithi090.stdout:920: finishing write tid 1 to smithi09025264-43 2023-01-15T14:31:57.995 INFO:tasks.rados.rados.0.smithi090.stdout:921: finishing write tid 1 to smithi09025264-11 2023-01-15T14:31:57.995 INFO:tasks.rados.rados.0.smithi090.stdout:920: finishing write tid 2 to smithi09025264-43 2023-01-15T14:31:57.995 INFO:tasks.rados.rados.0.smithi090.stdout:920: finishing write tid 3 to smithi09025264-43 2023-01-15T14:31:57.995 INFO:tasks.rados.rados.0.smithi090.stdout:921: finishing write tid 2 to smithi09025264-11 2023-01-15T14:31:57.995 INFO:tasks.rados.rados.0.smithi090.stdout:921: finishing write tid 3 to smithi09025264-11 2023-01-15T14:31:57.996 INFO:tasks.rados.rados.0.smithi090.stdout:921: finishing write tid 4 to smithi09025264-11 2023-01-15T14:31:57.996 INFO:tasks.rados.rados.0.smithi090.stdout:920: finishing write tid 4 to smithi09025264-43 2023-01-15T14:31:57.996 INFO:tasks.rados.rados.0.smithi090.stdout:920: finishing write tid 5 to smithi09025264-43 2023-01-15T14:31:57.996 INFO:tasks.rados.rados.0.smithi090.stdout:920: oid 43 updating version 0 to 81 2023-01-15T14:31:57.996 INFO:tasks.rados.rados.0.smithi090.stdout:920: oid 43 updating version 81 to 82 2023-01-15T14:31:57.996 INFO:tasks.rados.rados.0.smithi090.stdout:920: oid 43 version 82 is already newer than 79 2023-01-15T14:31:57.996 INFO:tasks.rados.rados.0.smithi090.stdout:920: oid 43 version 82 is already newer than 80 2023-01-15T14:31:57.997 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 82 (ObjNum 341 snap 91 seq_num 341) dirty exists 2023-01-15T14:31:57.997 INFO:tasks.rados.rados.0.smithi090.stdout:920: left oid 43 (ObjNum 341 snap 91 seq_num 341) 2023-01-15T14:31:57.997 INFO:tasks.rados.rados.0.smithi090.stdout:921: finishing write tid 5 to smithi09025264-11 2023-01-15T14:31:57.997 INFO:tasks.rados.rados.0.smithi090.stdout:920: done (4 left) 2023-01-15T14:31:57.997 INFO:tasks.rados.rados.0.smithi090.stdout:928: read oid 27 snap -1 2023-01-15T14:31:57.997 INFO:tasks.rados.rados.0.smithi090.stdout:928: expect deleted 2023-01-15T14:31:57.998 INFO:tasks.rados.rados.0.smithi090.stdout:929: write oid 19 current snap is 91 2023-01-15T14:31:57.998 INFO:tasks.rados.rados.0.smithi090.stdout:929: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2023-01-15T14:31:57.999 INFO:tasks.rados.rados.0.smithi090.stdout:929: writing smithi09025264-19 from 581608 to 1303149 tid 1 2023-01-15T14:31:57.999 INFO:tasks.rados.rados.0.smithi090.stdout:929: writing smithi09025264-19 from 1863688 to 2349461 tid 2 2023-01-15T14:31:57.999 INFO:tasks.rados.rados.0.smithi090.stdout:929: writing smithi09025264-19 from 3113980 to 3718023 tid 3 2023-01-15T14:31:57.999 INFO:tasks.rados.rados.0.smithi090.stdout:930: snap_remove snap 87 2023-01-15T14:31:58.001 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:31:58.001 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:31:58.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:31:58.001 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:31:58.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 211.87418484899999, 2023-01-15T14:31:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:31:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:31:58.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:58.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:31:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:31:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:58.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:31:58.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:31:58.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:58.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:31:58.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:31:58.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:58.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:31:58.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:31:58.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:58.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:31:58.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:31:58.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:58.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:31:58.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:31:58.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:58.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:31:58.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:31:58.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:58.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:31:58.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:31:58.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:58.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:31:58.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:31:58.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:58.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:31:58.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:31:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:31:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:31:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:31:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:31:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:31:58.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:31:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:31:58.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:31:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:58.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:31:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:31:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:58.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.560 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:58.560 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:58.561 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 211.87390460099999, 2023-01-15T14:31:58.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:31:58.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:58.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:58.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:58.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:58.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:58.561 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:31:58.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.561 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:58.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:58.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:58.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:58.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:58.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:31:58.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:58.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:58.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:31:58.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:31:58.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:58.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:31:58.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:31:58.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:58.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:31:58.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:31:58.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:58.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:31:58.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:31:58.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:58.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:31:58.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:31:58.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:31:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:31:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:31:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:31:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:31:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:31:58.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:31:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:31:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:31:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:31:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:31:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:31:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:31:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:31:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:31:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:31:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:31:58.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:31:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:31:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:31:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 211.873871984, 2023-01-15T14:31:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:31:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:58.743 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:31:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:31:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:31:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:31:58.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:31:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:31:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:31:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:31:58.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:31:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:31:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:31:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:31:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:31:58.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:31:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:31:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:31:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:31:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 211.77255399800001, 2023-01-15T14:31:58.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:31:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:31:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:58.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:58.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:31:59.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:59.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:31:59.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:59.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:59.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:31:59.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:31:59.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:59.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:31:59.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:31:59.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:59.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:31:59.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:31:59.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:31:59.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:31:59.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:31:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:31:59.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:31:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:31:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:31:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:31:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:31:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:31:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:31:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:31:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:31:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:31:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:31:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:31:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:31:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:31:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:59.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:31:59.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:31:59.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:59.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:31:59.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:31:59.346 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.346 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:59.346 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.347 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:59.347 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:59.347 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 211.772523995, 2023-01-15T14:31:59.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:31:59.348 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:59.348 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:59.348 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:59.348 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:59.348 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:59.348 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:31:59.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.348 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:59.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:59.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:59.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:59.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:31:59.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:31:59.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:31:59.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:59.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:31:59.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:31:59.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:31:59.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:31:59.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:59.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:31:59.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:31:59.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:31:59.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:31:59.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:59.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:31:59.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:31:59.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:31:59.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:31:59.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:59.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:31:59.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:31:59.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:59.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:31:59.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:31:59.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.355 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:59.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.355 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:59.355 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:59.355 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 211.25751543000001, 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:59.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:59.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:59.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:31:59.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:59.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:31:59.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:31:59.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:59.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:31:59.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:31:59.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:59.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:31:59.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:31:59.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:59.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:31:59.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:31:59.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:31:59.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:31:59.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:59.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:31:59.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:31:59.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:31:59.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:31:59.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:31:59.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:31:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:31:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:31:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:31:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:31:59.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:59.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:31:59.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:31:59.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:31:59.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:31:59.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:31:59.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:31:59.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:59.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:31:59.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:31:59.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 211.257033076, 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:31:59.365 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:59.366 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:59.366 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:59.366 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:59.366 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:59.366 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:31:59.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.366 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:59.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:59.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:59.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:59.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:31:59.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:59.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:31:59.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:31:59.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:59.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:31:59.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:31:59.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:31:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:31:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:31:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:31:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:31:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:31:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:31:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:31:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:31:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:31:59.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:31:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:31:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:31:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:31:59.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:31:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:31:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:31:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:31:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:31:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:31:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:31:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:31:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.557 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:59.558 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.558 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:31:59.559 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 211.256980513, 2023-01-15T14:31:59.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:31:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:59.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:31:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.562 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:59.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:31:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:59.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:31:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:59.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:31:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:31:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:31:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:31:59.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:31:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:31:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:31:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:31:59.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:31:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:31:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:31:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:31:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:59.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:31:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:31:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:31:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:31:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 197.88577880899999, 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.573 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:31:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:59.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:31:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:31:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:31:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:31:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:31:59.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:31:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:59.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:31:59.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:31:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:31:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:31:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:31:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:31:59.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:31:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:31:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:31:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:31:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:31:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:31:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:31:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:31:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:59.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 197.88566213499999, 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:59.758 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:31:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:31:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:31:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:31:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:31:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:31:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:31:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:31:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:31:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:31:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:31:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:31:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:31:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:31:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:31:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:31:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:31:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:31:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:31:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:31:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:31:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:31:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:31:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:31:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:31:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 190.159529859, 2023-01-15T14:31:59.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:31:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:31:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:31:59.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:59.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:31:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:31:59.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:31:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:31:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:59.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:31:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:31:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:31:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:31:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:31:59.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:31:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:31:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:31:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:31:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:31:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:31:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:31:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:31:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:31:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:31:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:59.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:31:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:59.966 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 153.9432798, 2023-01-15T14:31:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:31:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:59.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:31:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:59.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:31:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:31:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:31:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:31:59.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:31:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:31:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:31:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:31:59.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:31:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:31:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:31:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:31:59.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:31:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:31:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:31:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:31:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:31:59.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:31:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:31:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:31:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:31:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:59.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:31:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:31:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:31:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:31:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:31:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:31:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:31:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:31:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:59.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:31:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:31:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:31:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 153.943246233, 2023-01-15T14:31:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:31:59.978 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:31:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:59.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:31:59.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:31:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:31:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:31:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:31:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:31:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:31:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:31:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:31:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:31:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:31:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:31:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:31:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:31:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:31:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:31:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:31:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:31:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:31:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:31:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:31:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:31:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:31:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:31:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:31:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:31:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:31:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:31:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:31:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:31:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:31:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:31:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:31:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:31:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:31:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:31:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:31:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:31:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:31:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:31:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:31:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:31:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 135.17734268699999, 2023-01-15T14:31:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:31:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:31:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:31:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:31:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:31:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:31:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:31:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:31:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:31:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:31:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:31:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:31:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:31:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:31:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:31:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:31:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:31:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:31:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:31:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:31:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:31:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:31:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:31:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:31:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:31:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:31:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:31:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:31:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:31:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:32:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:32:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:32:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:32:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:32:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:32:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:32:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:32:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:32:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:32:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 118.634163887, 2023-01-15T14:32:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:32:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:32:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:32:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:32:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:32:00.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:32:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:32:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:32:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:32:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:32:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:32:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:32:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:32:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:00.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:32:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:32:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:32:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:32:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:32:00.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:32:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:32:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:32:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:32:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:32:00.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.011 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:32:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:32:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 118.634124712, 2023-01-15T14:32:00.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:32:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:32:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:00.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:00.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:32:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:32:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:32:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:32:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:32:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:32:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:32:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:32:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:00.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:32:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:32:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:32:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:32:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:00.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.019 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1373 3.12 3:495d4bbc:::smithi09025264-16:head [write 2873523~591733] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:32:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:32:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.24200838599999, 2023-01-15T14:32:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40949580099999999, 2023-01-15T14:32:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1373 2023-01-15T14:32:00.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:32:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:00.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:32:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693440+0000", 2023-01-15T14:32:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693808+0000", 2023-01-15T14:32:00.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000367742 2023-01-15T14:32:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693831+0000", 2023-01-15T14:32:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3213e-05 2023-01-15T14:32:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693846+0000", 2023-01-15T14:32:00.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4909000000000001e-05 2023-01-15T14:32:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.695833+0000", 2023-01-15T14:32:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019869240000000002 2023-01-15T14:32:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696121+0000", 2023-01-15T14:32:00.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028753300000000003 2023-01-15T14:32:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798577+0000", 2023-01-15T14:32:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102456808 2023-01-15T14:32:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798849+0000", 2023-01-15T14:32:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027138999999999999 2023-01-15T14:32:00.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:32:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799155+0000", 2023-01-15T14:32:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030594800000000001 2023-01-15T14:32:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799614+0000", 2023-01-15T14:32:00.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045960700000000001 2023-01-15T14:32:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799737+0000", 2023-01-15T14:32:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012218900000000001 2023-01-15T14:32:00.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799998+0000", 2023-01-15T14:32:00.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026129700000000002 2023-01-15T14:32:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800639+0000", 2023-01-15T14:32:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064108800000000005 2023-01-15T14:32:00.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102023+0000", 2023-01-15T14:32:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30138417200000001 2023-01-15T14:32:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:00.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102198+0000", 2023-01-15T14:32:00.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017453600000000001 2023-01-15T14:32:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102398+0000", 2023-01-15T14:32:00.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020078000000000001 2023-01-15T14:32:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102937+0000", 2023-01-15T14:32:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053874499999999998 2023-01-15T14:32:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:32:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:32:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.241583792, 2023-01-15T14:32:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:32:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:32:00.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:32:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:32:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:32:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:32:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:32:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:32:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:32:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:32:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:32:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:32:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:32:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:32:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:32:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:32:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:32:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:32:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:32:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:32:00.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:32:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:32:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:32:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:32:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:32:00.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:32:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:32:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:32:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:32:00.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:32:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:32:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:32:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:32:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:32:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:32:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:32:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:32:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:32:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:32:00.586 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:32:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.241531463, 2023-01-15T14:32:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:32:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:32:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:32:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:32:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:32:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:32:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:32:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:32:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:32:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:32:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:32:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:32:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:32:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:32:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:32:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:32:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:32:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:32:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:32:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:32:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:32:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:32:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:32:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:32:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:32:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:32:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.644618264000002, 2023-01-15T14:32:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:32:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:32:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:32:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:32:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:32:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:32:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:32:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:32:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:32:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:32:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:32:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:32:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:32:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:32:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:32:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:32:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:32:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:32:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:00.776 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:00.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:32:00.778 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:31:57.995+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:00.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:32:00.779 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:32:00.780 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:00.777+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:00.782 INFO:tasks.rados.rados.0.smithi090.stdout:930: done (6 left) 2023-01-15T14:32:00.782 INFO:tasks.rados.rados.0.smithi090.stdout:931: copy_from oid 9 from oid 48 current snap is 91 2023-01-15T14:32:00.782 INFO:tasks.rados.rados.0.smithi090.stdout:921: finishing write tid 6 to smithi09025264-11 2023-01-15T14:32:00.782 INFO:tasks.rados.rados.0.smithi090.stdout:921: oid 11 updating version 0 to 285 2023-01-15T14:32:00.782 INFO:tasks.rados.rados.0.smithi090.stdout:921: oid 11 version 285 is already newer than 283 2023-01-15T14:32:00.782 INFO:tasks.rados.rados.0.smithi090.stdout:921: oid 11 updating version 285 to 286 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:921: oid 11 version 286 is already newer than 282 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:921: oid 11 version 286 is already newer than 284 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 286 (ObjNum 342 snap 91 seq_num 342) dirty exists 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:921: left oid 11 (ObjNum 342 snap 91 seq_num 342) 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:925: finishing write tid 1 to smithi09025264-3 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:925: finishing write tid 2 to smithi09025264-3 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:925: finishing write tid 3 to smithi09025264-3 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:925: finishing write tid 4 to smithi09025264-3 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:925: finishing write tid 5 to smithi09025264-3 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:925: oid 3 updating version 0 to 242 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:925: oid 3 version 242 is already newer than 241 2023-01-15T14:32:00.783 INFO:tasks.rados.rados.0.smithi090.stdout:925: oid 3 version 242 is already newer than 240 2023-01-15T14:32:00.784 INFO:tasks.rados.rados.0.smithi090.stdout:925: oid 3 updating version 242 to 243 2023-01-15T14:32:00.784 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 243 (ObjNum 343 snap 91 seq_num 343) dirty exists 2023-01-15T14:32:00.784 INFO:tasks.rados.rados.0.smithi090.stdout:925: left oid 3 (ObjNum 343 snap 91 seq_num 343) 2023-01-15T14:32:00.784 INFO:tasks.rados.rados.0.smithi090.stdout:927: finishing rollback tid 1 to smithi09025264-29 2023-01-15T14:32:00.784 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 137 (ObjNum 33 snap 0 seq_num 3197587168) dirty dne 2023-01-15T14:32:00.784 INFO:tasks.rados.rados.0.smithi090.stdout:929: finishing write tid 1 to smithi09025264-19 2023-01-15T14:32:00.784 INFO:tasks.rados.rados.0.smithi090.stdout:929: finishing write tid 2 to smithi09025264-19 2023-01-15T14:32:00.784 INFO:tasks.rados.rados.0.smithi090.stdout:929: finishing write tid 3 to smithi09025264-19 2023-01-15T14:32:00.784 INFO:tasks.rados.rados.0.smithi090.stdout:929: finishing write tid 4 to smithi09025264-19 2023-01-15T14:32:00.784 INFO:tasks.rados.rados.0.smithi090.stdout:929: finishing write tid 5 to smithi09025264-19 2023-01-15T14:32:00.784 INFO:tasks.rados.rados.0.smithi090.stdout:929: oid 19 updating version 0 to 212 2023-01-15T14:32:00.785 INFO:tasks.rados.rados.0.smithi090.stdout:929: oid 19 updating version 212 to 214 2023-01-15T14:32:00.785 INFO:tasks.rados.rados.0.smithi090.stdout:929: oid 19 version 214 is already newer than 211 2023-01-15T14:32:00.785 INFO:tasks.rados.rados.0.smithi090.stdout:929: oid 19 version 214 is already newer than 213 2023-01-15T14:32:00.785 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 214 (ObjNum 344 snap 91 seq_num 344) dirty exists 2023-01-15T14:32:00.785 INFO:tasks.rados.rados.0.smithi090.stdout:929: left oid 19 (ObjNum 344 snap 91 seq_num 344) 2023-01-15T14:32:00.785 INFO:tasks.rados.rados.0.smithi090.stdout:921: done (6 left) 2023-01-15T14:32:00.785 INFO:tasks.rados.rados.0.smithi090.stdout:923: done (5 left) 2023-01-15T14:32:00.785 INFO:tasks.rados.rados.0.smithi090.stdout:925: done (4 left) 2023-01-15T14:32:00.785 INFO:tasks.rados.rados.0.smithi090.stdout:927: done (3 left) 2023-01-15T14:32:00.785 INFO:tasks.rados.rados.0.smithi090.stdout:928: done (2 left) 2023-01-15T14:32:00.785 INFO:tasks.rados.rados.0.smithi090.stdout:929: done (1 left) 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:932: write oid 17 current snap is 91 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:932: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:932: writing smithi09025264-17 from 598415 to 1395205 tid 1 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:932: writing smithi09025264-17 from 1922170 to 2667954 tid 2 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:932: writing smithi09025264-17 from 3141403 to 3556718 tid 3 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:933: delete oid 15 current snap is 91 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:932: finishing write tid 1 to smithi09025264-17 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:932: finishing write tid 2 to smithi09025264-17 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:932: finishing write tid 3 to smithi09025264-17 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:932: finishing write tid 4 to smithi09025264-17 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:932: finishing write tid 5 to smithi09025264-17 2023-01-15T14:32:00.786 INFO:tasks.rados.rados.0.smithi090.stdout:932: oid 17 updating version 0 to 305 2023-01-15T14:32:00.787 INFO:tasks.rados.rados.0.smithi090.stdout:932: oid 17 version 305 is already newer than 304 2023-01-15T14:32:00.787 INFO:tasks.rados.rados.0.smithi090.stdout:932: oid 17 version 305 is already newer than 302 2023-01-15T14:32:00.787 INFO:tasks.rados.rados.0.smithi090.stdout:932: oid 17 version 305 is already newer than 303 2023-01-15T14:32:00.787 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 305 (ObjNum 345 snap 91 seq_num 345) dirty exists 2023-01-15T14:32:00.787 INFO:tasks.rados.rados.0.smithi090.stdout:932: left oid 17 (ObjNum 345 snap 91 seq_num 345) 2023-01-15T14:32:00.881 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:00.877+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:00.985 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:00.985 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:32:00.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.985 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2462 3.3e 3:7e8e668e:::smithi09025264-9:head [copy-from ver 160 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2023-01-15T14:32:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:58.848179+0000", 2023-01-15T14:32:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.051417421, 2023-01-15T14:32:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.051444456, 2023-01-15T14:32:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2462 2023-01-15T14:32:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.986 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848179+0000", 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848179+0000", 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848177+0000", 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848190+0000", 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.279e-05 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848205+0000", 2023-01-15T14:32:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5492e-05 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848218+0000", 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2132e-05 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.948740+0000", 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100522872 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.949184+0000", 2023-01-15T14:32:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044310500000000001 2023-01-15T14:32:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2463 3.3e 3:7e8e668e:::smithi09025264-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e232)", 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:58.848231+0000", 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 2.0513652800000002, 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.051437022, 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "delayed", 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2463 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848231+0000", 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848231+0000", 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848231+0000", 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848235+0000", 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.036e-06 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848244+0000", 2023-01-15T14:32:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0040000000000005e-06 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.848250+0000", 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1210000000000003e-06 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.949592+0000", 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10134187 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:58.949662+0000", 2023-01-15T14:32:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0154000000000006e-05 2023-01-15T14:32:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 2 2023-01-15T14:32:00.995 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:00.996 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:32:00.996 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:00.978+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:00.997 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:01.190 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:01.191 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:01.191 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:01.191 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:01.192 INFO:tasks.rados.rados.0.smithi090.stdout:931: finishing copy_from to smithi09025264-9 2023-01-15T14:32:01.192 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 160 (ObjNum 260 snap 60 seq_num 260) dirty exists 2023-01-15T14:32:01.193 INFO:tasks.rados.rados.0.smithi090.stdout:931: finishing copy_from racing read to smithi09025264-9 2023-01-15T14:32:01.193 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:01.194 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:01.194 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:01.195 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:32:01.195 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:32:01.196 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:32:01.196 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:32:01.196 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:32:01.196 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:32:01.196 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:32:01.196 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: filestore_inject_stall = '0' (not observed, change may require restart) osd_enable_op_tracker = 'false' 2023-01-15T14:32:01.198 INFO:tasks.rados.rados.0.smithi090.stdout:931: done (2 left) 2023-01-15T14:32:01.198 INFO:tasks.rados.rados.0.smithi090.stdout:932: done (1 left) 2023-01-15T14:32:01.199 INFO:tasks.rados.rados.0.smithi090.stdout:933: done (0 left) 2023-01-15T14:32:01.199 INFO:tasks.rados.rados.0.smithi090.stdout:934: read oid 43 snap 72 2023-01-15T14:32:01.199 INFO:tasks.rados.rados.0.smithi090.stdout:934: expect deleted 2023-01-15T14:32:01.199 INFO:tasks.rados.rados.0.smithi090.stdout:935: read oid 12 snap -1 2023-01-15T14:32:01.199 INFO:tasks.rados.rados.0.smithi090.stdout:935: expect (ObjNum 252 snap 60 seq_num 252) 2023-01-15T14:32:01.199 INFO:tasks.rados.rados.0.smithi090.stdout:936: read oid 3 snap -1 2023-01-15T14:32:01.200 INFO:tasks.rados.rados.0.smithi090.stdout:936: expect (ObjNum 343 snap 91 seq_num 343) 2023-01-15T14:32:01.200 INFO:tasks.rados.rados.0.smithi090.stdout:937: snap_create 2023-01-15T14:32:01.200 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:32:01.200 INFO:tasks.rados.rados.0.smithi090.stdout:934: done (2 left) 2023-01-15T14:32:01.200 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:01.202 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:01.186+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:01.291 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:01.286+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:01.370 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:01.370 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:01.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:32:01.371 INFO:tasks.rados.rados.0.smithi090.stdout:936: expect (ObjNum 343 snap 91 seq_num 343) 2023-01-15T14:32:01.392 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:01.386+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:02.959 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:32:02.959 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:32:02.959 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-01-15T14:32:02.959 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-01-15T14:32:02.960 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:02.953+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:02.961 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:02.961 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:02.962 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:32:02.970 INFO:tasks.rados.rados.0.smithi090.stdout:936: done (1 left) 2023-01-15T14:32:02.970 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:03.123 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:32:03.125 INFO:tasks.rados.rados.0.smithi090.stdout:935: expect (ObjNum 252 snap 60 seq_num 252) 2023-01-15T14:32:03.136 INFO:tasks.rados.rados.0.smithi090.stdout:935: done (0 left) 2023-01-15T14:32:03.136 INFO:tasks.rados.rados.0.smithi090.stdout:937: done (0 left) 2023-01-15T14:32:03.136 INFO:tasks.rados.rados.0.smithi090.stdout:938: write (excl) oid 16 current snap is 92 2023-01-15T14:32:03.136 INFO:tasks.rados.rados.0.smithi090.stdout:938: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2023-01-15T14:32:03.137 INFO:tasks.rados.rados.0.smithi090.stdout:938: writing smithi09025264-16 from 632029 to 1179317 tid 1 2023-01-15T14:32:03.137 INFO:tasks.rados.rados.0.smithi090.stdout:938: writing smithi09025264-16 from 1639134 to 2388587 tid 2 2023-01-15T14:32:03.137 INFO:tasks.rados.rados.0.smithi090.stdout:938: writing smithi09025264-16 from 2796249 to 3234108 tid 3 2023-01-15T14:32:03.137 INFO:tasks.rados.rados.0.smithi090.stdout:938: writing smithi09025264-16 from 3653524 to 3832029 tid 4 2023-01-15T14:32:03.137 INFO:tasks.rados.rados.0.smithi090.stdout:939: write (excl) oid 25 current snap is 92 2023-01-15T14:32:03.137 INFO:tasks.rados.rados.0.smithi090.stdout:939: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2023-01-15T14:32:03.137 INFO:tasks.rados.rados.0.smithi090.stdout:939: writing smithi09025264-25 from 648836 to 1271373 tid 1 2023-01-15T14:32:03.137 INFO:tasks.rados.rados.0.smithi090.stdout:939: writing smithi09025264-25 from 1981263 to 2474374 tid 2 2023-01-15T14:32:03.137 INFO:tasks.rados.rados.0.smithi090.stdout:939: writing smithi09025264-25 from 3107319 to 3848836 tid 3 2023-01-15T14:32:03.137 INFO:tasks.rados.rados.0.smithi090.stdout:940: delete oid 29 current snap is 92 2023-01-15T14:32:03.138 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:03.118+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:03.151 INFO:tasks.rados.rados.0.smithi090.stdout:938: finishing write tid 1 to smithi09025264-16 2023-01-15T14:32:03.271 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:32:03.272 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:03.272 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:03.273 INFO:tasks.rados.rados.0.smithi090.stdout:939: finishing write tid 1 to smithi09025264-25 2023-01-15T14:32:03.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:32:03.274 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:03.269+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:03.304 INFO:tasks.rados.rados.0.smithi090.stdout:940: done (2 left) 2023-01-15T14:32:03.304 INFO:tasks.rados.rados.0.smithi090.stdout:941: snap_create 2023-01-15T14:32:03.304 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:03.355 INFO:tasks.rados.rados.0.smithi090.stdout:938: finishing write tid 2 to smithi09025264-16 2023-01-15T14:32:03.463 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:03.463 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:03.464 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:32:03.464 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:03.465 INFO:tasks.rados.rados.0.smithi090.stdout:938: finishing write tid 3 to smithi09025264-16 2023-01-15T14:32:03.465 INFO:tasks.rados.rados.0.smithi090.stdout:938: finishing write tid 4 to smithi09025264-16 2023-01-15T14:32:03.465 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:32:03.466 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 236, 2023-01-15T14:32:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "1c77fb2b-d1a7-4884-a24e-3935e16d0b6b", 2023-01-15T14:32:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2023-01-15T14:27:57.735717+0000", 2023-01-15T14:32:03.466 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2023-01-15T14:32:02.824953+0000", 2023-01-15T14:32:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2023-01-15T14:28:04.837522+0000", 2023-01-15T14:32:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2023-01-15T14:30:49.906949+0000", 2023-01-15T14:32:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-01-15T14:32:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5801984, 2023-01-15T14:32:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2023-01-15T14:32:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2023-01-15T14:32:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2023-01-15T14:32:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2023-01-15T14:32:03.467 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2023-01-15T14:32:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2023-01-15T14:32:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2023-01-15T14:32:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2023-01-15T14:32:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-01-15T14:32:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2023-01-15T14:32:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2023-01-15T14:32:03.596 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 3, 2023-01-15T14:32:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2023-01-15T14:32:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2023-01-15T14:32:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2023-01-15T14:32:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "reef", 2023-01-15T14:32:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2023-01-15T14:32:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2023-01-15T14:32:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2023-01-15T14:32:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:04.899282+0000", 2023-01-15T14:32:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:32:03.597 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:32:03.598 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2023-01-15T14:32:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2023-01-15T14:32:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2023-01-15T14:32:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2023-01-15T14:32:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2023-01-15T14:32:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:32:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:32:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:32:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:32:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:32:03.599 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:32:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:32:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "16", 2023-01-15T14:32:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:32:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:32:03.600 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:32:03.606 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:32:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:32:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:32:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:32:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:32:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:32:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:32:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:32:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:32:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:32:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:32:03.607 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:32:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:32:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:32:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:32:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:32:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:32:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:32:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:32:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:32:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:32:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:32:03.608 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:32:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:32:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:32:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:32:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:32:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:32:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:32:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:32:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:32:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:32:03.609 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:32:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:32:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2023-01-15T14:32:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2023-01-15T14:32:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2023-01-15T14:32:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:32:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2023-01-15T14:32:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:03.610 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2023-01-15T14:32:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2023-01-15T14:32:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:11.115002+0000", 2023-01-15T14:32:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:32:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:32:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:32:03.611 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:32:03.619 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:32:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:32:03.620 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:32:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:32:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:32:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:32:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:32:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:32:03.621 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2023-01-15T14:32:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2023-01-15T14:32:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2023-01-15T14:32:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2023-01-15T14:32:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2023-01-15T14:32:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:32:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:32:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:32:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:32:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:32:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:32:03.622 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "19", 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:32:03.623 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:32:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:32:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:32:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:32:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:32:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:32:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:32:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:32:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:32:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:32:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:32:03.624 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:32:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:32:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:32:03.625 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:32:03.762 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:32:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:32:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:32:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:32:03.763 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:32:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:32:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:32:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:32:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:32:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:32:03.764 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:32:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:32:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:32:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:32:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:32:03.765 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2023-01-15T14:32:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:32:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "unique_pool_0", 2023-01-15T14:32:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:22.434903+0000", 2023-01-15T14:32:03.766 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 16385, 2023-01-15T14:32:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,pool_snaps", 2023-01-15T14:32:03.767 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:32:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:32:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:32:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:32:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:32:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:32:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:32:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:32:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:32:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:32:03.925 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 66, 2023-01-15T14:32:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 66, 2023-01-15T14:32:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 66, 2023-01-15T14:32:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 66, 2023-01-15T14:32:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 66, 2023-01-15T14:32:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:32:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:32:03.926 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:32:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:32:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:32:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:32:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:32:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "236", 2023-01-15T14:32:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:32:03.929 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:32:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "224", 2023-01-15T14:32:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:32:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "pool", 2023-01-15T14:32:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 180, 2023-01-15T14:32:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 236, 2023-01-15T14:32:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [ 2023-01-15T14:32:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 140, 2023-01-15T14:32:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:31:12.022478+0000", 2023-01-15T14:32:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap73" 2023-01-15T14:32:03.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 155, 2023-01-15T14:32:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:31:31.526941+0000", 2023-01-15T14:32:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap82" 2023-01-15T14:32:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 172, 2023-01-15T14:32:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:31:50.190514+0000", 2023-01-15T14:32:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap89" 2023-01-15T14:32:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:03.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 180, 2023-01-15T14:32:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:02.193761+0000", 2023-01-15T14:32:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap92" 2023-01-15T14:32:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:32:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:32:03.932 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:32:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:32:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:32:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:32:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:32:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:32:03.933 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:32:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:32:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:32:03.934 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:32:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:32:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:32:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:32:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:32:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:32:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:32:03.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.091 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:32:04.091 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:32:04.091 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:32:04.092 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:32:04.092 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:32:04.092 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:32:04.092 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:32:04.092 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:32:04.092 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:32:04.092 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:32:04.092 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:32:04.092 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:32:04.092 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e55c5ac4-3cff-4ee5-8719-373648f83d8d", 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.3336334228515625, 2023-01-15T14:32:04.093 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:04.094 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:04.094 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:04.094 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:04.094 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 233, 2023-01-15T14:32:04.094 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:04.094 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:04.094 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:04.094 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.094 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.094 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2023-01-15T14:32:04.095 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:04.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.095 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.095 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2023-01-15T14:32:04.100 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:04.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.101 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.101 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:04.101 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.101 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.101 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2023-01-15T14:32:04.101 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:04.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2023-01-15T14:32:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2023-01-15T14:32:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6811/23473", 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6815/23473", 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/23473", 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6820/23473", 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:32:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "2c1b33c9-0a1e-4734-936d-d51d64fe997a", 2023-01-15T14:32:04.106 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 231, 2023-01-15T14:32:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2023-01-15T14:32:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2023-01-15T14:32:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2023-01-15T14:32:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2023-01-15T14:32:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2023-01-15T14:32:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2023-01-15T14:32:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2023-01-15T14:32:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2023-01-15T14:32:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/23474", 2023-01-15T14:32:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/23474", 2023-01-15T14:32:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/23474", 2023-01-15T14:32:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/23474", 2023-01-15T14:32:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:32:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "b314174a-eca9-4baa-a60c-2589f1fc9a8e", 2023-01-15T14:32:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0, 2023-01-15T14:32:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 234, 2023-01-15T14:32:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.262 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2023-01-15T14:32:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2023-01-15T14:32:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.340 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2023-01-15T14:32:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.341 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2023-01-15T14:32:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.342 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:04.343 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2023-01-15T14:32:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:04.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2023-01-15T14:32:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:04.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6810/23475", 2023-01-15T14:32:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/23475", 2023-01-15T14:32:04.346 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6821/23475", 2023-01-15T14:32:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6817/23475", 2023-01-15T14:32:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:04.347 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:32:04.348 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "29eb0b17-ed5d-4665-a347-2fb8406d9c62", 2023-01-15T14:32:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.9044189453125, 2023-01-15T14:32:04.349 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:04.350 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 234, 2023-01-15T14:32:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:04.416 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6824", 2023-01-15T14:32:04.417 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6825", 2023-01-15T14:32:04.418 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6826", 2023-01-15T14:32:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6827", 2023-01-15T14:32:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6830", 2023-01-15T14:32:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6831", 2023-01-15T14:32:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.428 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.428 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6828", 2023-01-15T14:32:04.429 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:04.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.429 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.429 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6829", 2023-01-15T14:32:04.429 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:04.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.430 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.430 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6825/23778", 2023-01-15T14:32:04.430 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6827/23778", 2023-01-15T14:32:04.431 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6831/23778", 2023-01-15T14:32:04.431 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6829/23778", 2023-01-15T14:32:04.431 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:04.431 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:04.431 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:04.432 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.432 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:32:04.432 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "59362feb-caa9-44e3-b6d0-b79397dacbfc", 2023-01-15T14:32:04.433 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:04.433 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:04.433 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:04.433 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:04.433 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:04.433 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:04.434 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:04.434 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 229, 2023-01-15T14:32:04.434 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:04.434 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:04.434 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:04.434 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.435 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.435 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6800", 2023-01-15T14:32:04.435 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:04.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.436 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.436 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6801", 2023-01-15T14:32:04.436 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:04.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.437 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.437 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:04.437 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.438 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.438 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6802", 2023-01-15T14:32:04.438 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:04.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.438 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.439 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6803", 2023-01-15T14:32:04.439 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:04.439 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.439 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.440 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:04.440 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.440 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.573 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6806", 2023-01-15T14:32:04.573 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:04.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.573 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6807", 2023-01-15T14:32:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6804", 2023-01-15T14:32:04.574 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6805", 2023-01-15T14:32:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6801/23523", 2023-01-15T14:32:04.575 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6803/23523", 2023-01-15T14:32:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6807/23523", 2023-01-15T14:32:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6805/23523", 2023-01-15T14:32:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.576 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "fbc8821c-8d78-40f8-a571-b18a70d42000", 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:04.577 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6824", 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6825", 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.578 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6826", 2023-01-15T14:32:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.579 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6827", 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6830", 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6831", 2023-01-15T14:32:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.581 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6828", 2023-01-15T14:32:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6829", 2023-01-15T14:32:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.582 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6825/23525", 2023-01-15T14:32:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6827/23525", 2023-01-15T14:32:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6831/23525", 2023-01-15T14:32:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6829/23525", 2023-01-15T14:32:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:32:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "34550470-3561-492f-b605-b5f0e8052c1e", 2023-01-15T14:32:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:32:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:32:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:04.584 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 118, 2023-01-15T14:32:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6808", 2023-01-15T14:32:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6809", 2023-01-15T14:32:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.586 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6810", 2023-01-15T14:32:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6811", 2023-01-15T14:32:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.587 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6814", 2023-01-15T14:32:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6815", 2023-01-15T14:32:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:04.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6812", 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.589 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6813", 2023-01-15T14:32:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6809/23524", 2023-01-15T14:32:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6811/23524", 2023-01-15T14:32:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6815/23524", 2023-01-15T14:32:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6813/23524", 2023-01-15T14:32:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:04.590 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:04.591 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.591 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:32:04.591 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1f123d0d-d617-4771-b516-21aa5a29b734", 2023-01-15T14:32:04.591 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:04.732 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:04.732 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.5779571533203125, 2023-01-15T14:32:04.732 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 235, 2023-01-15T14:32:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6816", 2023-01-15T14:32:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6817", 2023-01-15T14:32:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.734 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6818", 2023-01-15T14:32:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.735 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6819", 2023-01-15T14:32:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.736 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6822", 2023-01-15T14:32:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6823", 2023-01-15T14:32:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:04.737 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6820", 2023-01-15T14:32:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6821", 2023-01-15T14:32:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:04.738 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6817/23526", 2023-01-15T14:32:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6819/23526", 2023-01-15T14:32:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6823/23526", 2023-01-15T14:32:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6821/23526", 2023-01-15T14:32:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.739 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2023-01-15T14:32:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:32:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:47.556519+0000", 2023-01-15T14:32:04.740 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:32:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:54.007074+0000", 2023-01-15T14:32:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:04.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:53.308067+0000", 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.742 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:43.268238+0000", 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:04.743 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:51.428540+0000", 2023-01-15T14:32:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:32:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:04.744 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:27.525459+0000", 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:04.745 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:55.585270+0000", 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:39.052183+0000", 2023-01-15T14:32:04.746 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.2d", 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 2, 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 3 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.747 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2023-01-15T14:32:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [ 2023-01-15T14:32:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.39", 2023-01-15T14:32:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:32:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: 1, 2023-01-15T14:32:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: 4 2023-01-15T14:32:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.748 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2023-01-15T14:32:04.749 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2023-01-15T14:32:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:32:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2023-01-15T14:32:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 123, 2023-01-15T14:32:04.750 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 157, 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 159, 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 165, 2023-01-15T14:32:04.751 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:32:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 167, 2023-01-15T14:32:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:32:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 173, 2023-01-15T14:32:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2023-01-15T14:32:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2023-01-15T14:32:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2023-01-15T14:32:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2023-01-15T14:32:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2023-01-15T14:32:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2023-01-15T14:32:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2023-01-15T14:32:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2023-01-15T14:32:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2023-01-15T14:32:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2023-01-15T14:32:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2023-01-15T14:32:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:04.754 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:04.754 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:04.900 INFO:tasks.thrashosds.thrasher:j is {'epoch': 236, 'fsid': '1c77fb2b-d1a7-4884-a24e-3935e16d0b6b', 'created': '2023-01-15T14:27:57.735717+0000', 'modified': '2023-01-15T14:32:02.824953+0000', 'last_up_change': '2023-01-15T14:28:04.837522+0000', 'last_in_change': '2023-01-15T14:30:49.906949+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-01-15T14:28:04.899282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-01-15T14:28:11.115002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-01-15T14:28:22.434903+0000', 'flags': 16385, 'flags_names': 'hashpspool,pool_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 66, 'pg_placement_num': 66, 'pg_placement_num_target': 66, 'pg_num_target': 66, 'pg_num_pending': 66, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '236', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '224', 'auid': 0, 'snap_mode': 'pool', 'snap_seq': 180, 'snap_epoch': 236, 'pool_snaps': [{'snapid': 140, 'stamp': '2023-01-15T14:31:12.022478+0000', 'name': 'smithi09025264-snap73'}, {'snapid': 155, 'stamp': '2023-01-15T14:31:31.526941+0000', 'name': 'smithi09025264-snap82'}, {'snapid': 172, 'stamp': '2023-01-15T14:31:50.190514+0000', 'name': 'smithi09025264-snap89'}, {'snapid': 180, 'stamp': '2023-01-15T14:32:02.193761+0000', 'name': 'smithi09025264-snap92'}], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e55c5ac4-3cff-4ee5-8719-373648f83d8d', 'up': 1, 'in': 1, 'weight': 0.3336334228515625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 233, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 23473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 23473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 23473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6820', 'nonce': 23473}]}, 'public_addr': '172.21.15.90:6811/23473', 'cluster_addr': '172.21.15.90:6815/23473', 'heartbeat_back_addr': '172.21.15.90:6823/23473', 'heartbeat_front_addr': '172.21.15.90:6820/23473', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2c1b33c9-0a1e-4734-936d-d51d64fe997a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 231, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 23474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 23474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 23474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 23474}]}, 'public_addr': '172.21.15.90:6801/23474', 'cluster_addr': '172.21.15.90:6803/23474', 'heartbeat_back_addr': '172.21.15.90:6807/23474', 'heartbeat_front_addr': '172.21.15.90:6805/23474', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b314174a-eca9-4baa-a60c-2589f1fc9a8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 234, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 23475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 23475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6819', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 23475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 23475}]}, 'public_addr': '172.21.15.90:6810/23475', 'cluster_addr': '172.21.15.90:6814/23475', 'heartbeat_back_addr': '172.21.15.90:6821/23475', 'heartbeat_front_addr': '172.21.15.90:6817/23475', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '29eb0b17-ed5d-4665-a347-2fb8406d9c62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9044189453125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 234, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6824', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6825', 'nonce': 23778}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6826', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6827', 'nonce': 23778}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6830', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6831', 'nonce': 23778}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6828', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6829', 'nonce': 23778}]}, 'public_addr': '172.21.15.90:6825/23778', 'cluster_addr': '172.21.15.90:6827/23778', 'heartbeat_back_addr': '172.21.15.90:6831/23778', 'heartbeat_front_addr': '172.21.15.90:6829/23778', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '59362feb-caa9-44e3-b6d0-b79397dacbfc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 229, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6800', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6801', 'nonce': 23523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6802', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6803', 'nonce': 23523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6806', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6807', 'nonce': 23523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6804', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6805', 'nonce': 23523}]}, 'public_addr': '172.21.15.141:6801/23523', 'cluster_addr': '172.21.15.141:6803/23523', 'heartbeat_back_addr': '172.21.15.141:6807/23523', 'heartbeat_front_addr': '172.21.15.141:6805/23523', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fbc8821c-8d78-40f8-a571-b18a70d42000', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6824', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6825', 'nonce': 23525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6826', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6827', 'nonce': 23525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6830', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6831', 'nonce': 23525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6828', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6829', 'nonce': 23525}]}, 'public_addr': '172.21.15.141:6825/23525', 'cluster_addr': '172.21.15.141:6827/23525', 'heartbeat_back_addr': '172.21.15.141:6831/23525', 'heartbeat_front_addr': '172.21.15.141:6829/23525', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '34550470-3561-492f-b605-b5f0e8052c1e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6808', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6809', 'nonce': 23524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6810', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6811', 'nonce': 23524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6814', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6815', 'nonce': 23524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6812', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6813', 'nonce': 23524}]}, 'public_addr': '172.21.15.141:6809/23524', 'cluster_addr': '172.21.15.141:6811/23524', 'heartbeat_back_addr': '172.21.15.141:6815/23524', 'heartbeat_front_addr': '172.21.15.141:6813/23524', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f123d0d-d617-4771-b516-21aa5a29b734', 'up': 1, 'in': 1, 'weight': 0.5779571533203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 235, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6816', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6817', 'nonce': 23526}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6818', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6819', 'nonce': 23526}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6822', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6823', 'nonce': 23526}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6820', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6821', 'nonce': 23526}]}, 'public_addr': '172.21.15.141:6817/23526', 'cluster_addr': '172.21.15.141:6819/23526', 'heartbeat_back_addr': '172.21.15.141:6823/23526', 'heartbeat_front_addr': '172.21.15.141:6821/23526', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:47.556519+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:54.007074+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:53.308067+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:43.268238+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:51.428540+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:31:27.525459+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:31:55.585270+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:39.052183+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.2d', 'mappings': [{'from': 2, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.39', 'osds': [1, 4]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 123, 'length': 1}, {'begin': 157, 'length': 1}, {'begin': 159, 'length': 1}, {'begin': 165, 'length': 1}, {'begin': 167, 'length': 1}, {'begin': 173, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-01-15T14:32:04.900 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:32:04.901 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:03.458+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:04.902 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:04.902 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:04.903 INFO:tasks.rados.rados.0.smithi090.stdout:938: finishing write tid 5 to smithi09025264-16 2023-01-15T14:32:04.903 INFO:tasks.rados.rados.0.smithi090.stdout:938: finishing write tid 6 to smithi09025264-16 2023-01-15T14:32:04.903 INFO:tasks.rados.rados.0.smithi090.stdout:938: oid 16 updating version 0 to 160 2023-01-15T14:32:04.903 INFO:tasks.rados.rados.0.smithi090.stdout:938: oid 16 version 160 is already newer than 158 2023-01-15T14:32:04.903 INFO:tasks.rados.rados.0.smithi090.stdout:938: oid 16 updating version 160 to 162 2023-01-15T14:32:04.904 INFO:tasks.rados.rados.0.smithi090.stdout:938: oid 16 version 162 is already newer than 161 2023-01-15T14:32:04.904 INFO:tasks.rados.rados.0.smithi090.stdout:938: oid 16 version 162 is already newer than 159 2023-01-15T14:32:04.904 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 162 (ObjNum 347 snap 92 seq_num 347) dirty exists 2023-01-15T14:32:04.904 INFO:tasks.rados.rados.0.smithi090.stdout:938: left oid 16 (ObjNum 347 snap 92 seq_num 347) 2023-01-15T14:32:04.904 INFO:tasks.rados.rados.0.smithi090.stdout:938: done (1 left) 2023-01-15T14:32:04.904 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:04.904 INFO:tasks.rados.rados.0.smithi090.stdout:939: finishing write tid 2 to smithi09025264-25 2023-01-15T14:32:04.904 INFO:tasks.rados.rados.0.smithi090.stdout:939: finishing write tid 3 to smithi09025264-25 2023-01-15T14:32:04.904 INFO:tasks.rados.rados.0.smithi090.stdout:939: finishing write tid 4 to smithi09025264-25 2023-01-15T14:32:04.905 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:04.905 INFO:tasks.rados.rados.0.smithi090.stdout:939: finishing write tid 5 to smithi09025264-25 2023-01-15T14:32:04.905 INFO:tasks.rados.rados.0.smithi090.stdout:939: oid 25 updating version 0 to 255 2023-01-15T14:32:04.905 INFO:tasks.rados.rados.0.smithi090.stdout:939: oid 25 updating version 255 to 256 2023-01-15T14:32:04.905 INFO:tasks.rados.rados.0.smithi090.stdout:939: oid 25 version 256 is already newer than 253 2023-01-15T14:32:04.905 INFO:tasks.rados.rados.0.smithi090.stdout:939: oid 25 version 256 is already newer than 254 2023-01-15T14:32:04.905 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 256 (ObjNum 348 snap 92 seq_num 348) dirty exists 2023-01-15T14:32:04.906 INFO:tasks.rados.rados.0.smithi090.stdout:939: left oid 25 (ObjNum 348 snap 92 seq_num 348) 2023-01-15T14:32:04.906 INFO:tasks.rados.rados.0.smithi090.stdout:939: done (0 left) 2023-01-15T14:32:04.906 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:04.906 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:32:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:32:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:32:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:32:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:32:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:32:04.907 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:32:04.908 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:32:04.908 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:04.897+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:04.909 INFO:tasks.rados.rados.0.smithi090.stdout:941: done (0 left) 2023-01-15T14:32:05.049 INFO:tasks.rados.rados.0.smithi090.stdout:942: read oid 2 snap -1 2023-01-15T14:32:05.049 INFO:tasks.rados.rados.0.smithi090.stdout:942: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:32:05.050 INFO:tasks.rados.rados.0.smithi090.stdout:943: delete oid 20 current snap is 93 2023-01-15T14:32:05.050 INFO:tasks.rados.rados.0.smithi090.stdout:942: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:32:05.050 INFO:tasks.rados.rados.0.smithi090.stdout:942: done (1 left) 2023-01-15T14:32:05.050 INFO:tasks.rados.rados.0.smithi090.stdout:943: done (0 left) 2023-01-15T14:32:05.050 INFO:tasks.rados.rados.0.smithi090.stdout:944: copy_from oid 46 from oid 43 current snap is 93 2023-01-15T14:32:05.050 INFO:tasks.rados.rados.0.smithi090.stdout:945: read oid 6 snap -1 2023-01-15T14:32:05.051 INFO:tasks.rados.rados.0.smithi090.stdout:945: expect deleted 2023-01-15T14:32:05.051 INFO:tasks.rados.rados.0.smithi090.stdout:946: read oid 15 snap 92 2023-01-15T14:32:05.051 INFO:tasks.rados.rados.0.smithi090.stdout:946: expect deleted 2023-01-15T14:32:05.051 INFO:tasks.rados.rados.0.smithi090.stdout:947: delete oid 24 current snap is 93 2023-01-15T14:32:05.051 INFO:tasks.rados.rados.0.smithi090.stdout:944: finishing copy_from to smithi09025264-46 2023-01-15T14:32:05.051 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 82 (ObjNum 341 snap 91 seq_num 341) dirty exists 2023-01-15T14:32:05.051 INFO:tasks.rados.rados.0.smithi090.stdout:944: finishing copy_from racing read to smithi09025264-46 2023-01-15T14:32:05.051 INFO:tasks.rados.rados.0.smithi090.stdout:944: done (3 left) 2023-01-15T14:32:05.051 INFO:tasks.rados.rados.0.smithi090.stdout:945: done (2 left) 2023-01-15T14:32:05.051 INFO:tasks.rados.rados.0.smithi090.stdout:946: done (1 left) 2023-01-15T14:32:05.051 INFO:tasks.rados.rados.0.smithi090.stdout:947: done (0 left) 2023-01-15T14:32:05.052 INFO:tasks.rados.rados.0.smithi090.stdout:948: read oid 19 snap -1 2023-01-15T14:32:05.052 INFO:tasks.rados.rados.0.smithi090.stdout:948: expect (ObjNum 344 snap 91 seq_num 344) 2023-01-15T14:32:05.052 INFO:tasks.rados.rados.0.smithi090.stdout:949: snap_remove snap 92 2023-01-15T14:32:05.053 INFO:tasks.rados.rados.0.smithi090.stdout:949: done (1 left) 2023-01-15T14:32:05.054 INFO:tasks.rados.rados.0.smithi090.stdout:950: write oid 9 current snap is 93 2023-01-15T14:32:05.054 INFO:tasks.rados.rados.0.smithi090.stdout:950: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2023-01-15T14:32:05.054 INFO:tasks.rados.rados.0.smithi090.stdout:950: writing smithi09025264-9 from 716064 to 1355950 tid 1 2023-01-15T14:32:05.054 INFO:tasks.rados.rados.0.smithi090.stdout:950: writing smithi09025264-9 from 1815191 to 2315640 tid 2 2023-01-15T14:32:05.054 INFO:tasks.rados.rados.0.smithi090.stdout:950: writing smithi09025264-9 from 2817011 to 3343936 tid 3 2023-01-15T14:32:05.054 INFO:tasks.rados.rados.0.smithi090.stdout:951: read oid 39 snap -1 2023-01-15T14:32:05.054 INFO:tasks.rados.rados.0.smithi090.stdout:951: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:32:05.054 INFO:tasks.rados.rados.0.smithi090.stdout:952: rollback oid 34 current snap is 93 2023-01-15T14:32:05.054 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 34 to 91 2023-01-15T14:32:05.054 INFO:tasks.rados.rados.0.smithi090.stdout:948: expect (ObjNum 344 snap 91 seq_num 344) 2023-01-15T14:32:05.056 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:05.056 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:32:05.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:32:05.057 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:32:05.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.057 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:38 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'3, mlcod=0'0)", 2023-01-15T14:32:05.057 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:32:05.057 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 218.89472612399999, 2023-01-15T14:32:05.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30191175999999997, 2023-01-15T14:32:05.061 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:05.061 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:05.061 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:05.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:05.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:32:05.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:05.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:32:05.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:05.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112976+0000", 2023-01-15T14:32:05.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113186+0000", 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020979700000000001 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113228+0000", 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2150999999999997e-05 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213313+0000", 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100084943 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313517+0000", 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10020435699999999 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:05.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313576+0000", 2023-01-15T14:32:05.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783999999999998e-05 2023-01-15T14:32:05.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:05.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313976+0000", 2023-01-15T14:32:05.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039960499999999998 2023-01-15T14:32:05.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:05.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314004+0000", 2023-01-15T14:32:05.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7954e-05 2023-01-15T14:32:05.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:05.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314175+0000", 2023-01-15T14:32:05.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171557 2023-01-15T14:32:05.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:05.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414853+0000", 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100677875 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414890+0000", 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6803e-05 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.067 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:125 3.0 e70/55 3:0b54190a:::smithi09025264-17:25 v 70'136, mlcod=68'130)", 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 169.064744099, 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30171926799999998, 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942958+0000", 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:05.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942972+0000", 2023-01-15T14:32:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3386e-05 2023-01-15T14:32:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942990+0000", 2023-01-15T14:32:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8113000000000001e-05 2023-01-15T14:32:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.943002+0000", 2023-01-15T14:32:05.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:32:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141942+0000", 2023-01-15T14:32:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19893962300000001 2023-01-15T14:32:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:05.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141978+0000", 2023-01-15T14:32:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6383e-05 2023-01-15T14:32:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142670+0000", 2023-01-15T14:32:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069211900000000005 2023-01-15T14:32:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142688+0000", 2023-01-15T14:32:05.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7839999999999999e-05 2023-01-15T14:32:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142768+0000", 2023-01-15T14:32:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0210999999999995e-05 2023-01-15T14:32:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244581+0000", 2023-01-15T14:32:05.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101813406 2023-01-15T14:32:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244680+0000", 2023-01-15T14:32:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8045000000000001e-05 2023-01-15T14:32:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.074 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 138.94900171099999, 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:32:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:32:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:32:05.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:32:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:32:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:32:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:32:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:32:05.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:32:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:32:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:32:05.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:32:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:32:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:32:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:32:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:32:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:32:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:32:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:32:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:32:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.113244320999996, 2023-01-15T14:32:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:32:05.212 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:32:05.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:32:05.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:32:05.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:32:05.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:05.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:32:05.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:32:05.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:05.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:32:05.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:32:05.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:05.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:32:05.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:32:05.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:05.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:32:05.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:05.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:32:05.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:32:05.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:05.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:32:05.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:32:05.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.222 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:32:05.222 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:32:05.222 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.113193378000005, 2023-01-15T14:32:05.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:32:05.222 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:05.222 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:05.222 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:05.222 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:05.222 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:05.222 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:32:05.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.223 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:05.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:05.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:32:05.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:05.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:32:05.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:05.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:32:05.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:32:05.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:05.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:32:05.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:32:05.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:32:05.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:32:05.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:05.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:32:05.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:32:05.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:32:05.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:05.228 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.172294136999994, 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:05.229 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:05.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:32:05.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:32:05.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:05.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:32:05.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:32:05.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:32:05.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:32:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:32:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:32:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:32:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:32:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:32:05.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:05.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:32:05.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:32:05.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:05.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:32:05.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:32:05.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:05.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:32:05.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:32:05.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:05.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:32:05.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:32:05.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:05.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:32:05.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:32:05.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:05.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:32:05.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:32:05.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:05.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:32:05.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:05.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:32:05.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:32:05.238 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.238 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:05.238 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.238 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:05.238 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.098519964000005, 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:32:05.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:05.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:05.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:32:05.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:32:05.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:05.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:32:05.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:32:05.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:05.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:32:05.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:32:05.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:32:05.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:32:05.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:05.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:32:05.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:32:05.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:32:05.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:32:05.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:05.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:32:05.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:32:05.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:05.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:32:05.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:05.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:32:05.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:32:05.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:05.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:32:05.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:32:05.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:05.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:32:05.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:32:05.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:05.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:32:05.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:32:05.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:05.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:32:05.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:32:05.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:05.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:32:05.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:32:05.383 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.383 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:05.383 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.384 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1712 3.4 3:2022fdfa:::smithi09025264-46:head [write 1880802~421790] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:05.384 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:32:05.384 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 85.093381273000006, 2023-01-15T14:32:05.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30396041800000001, 2023-01-15T14:32:05.384 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:05.384 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:05.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:05.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:05.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:05.384 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1712 2023-01-15T14:32:05.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:05.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914322+0000", 2023-01-15T14:32:05.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:05.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:05.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914561+0000", 2023-01-15T14:32:05.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023884700000000001 2023-01-15T14:32:05.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914572+0000", 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0499999999999999e-05 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914576+0000", 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6820000000000002e-06 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014941+0000", 2023-01-15T14:32:05.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100364994 2023-01-15T14:32:05.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:05.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014959+0000", 2023-01-15T14:32:05.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7416000000000001e-05 2023-01-15T14:32:05.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018171+0000", 2023-01-15T14:32:05.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032121239999999998 2023-01-15T14:32:05.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:05.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018333+0000", 2023-01-15T14:32:05.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016180900000000001 2023-01-15T14:32:05.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:05.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018581+0000", 2023-01-15T14:32:05.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024773699999999999 2023-01-15T14:32:05.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:05.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018897+0000", 2023-01-15T14:32:05.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031622099999999999 2023-01-15T14:32:05.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:05.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018907+0000", 2023-01-15T14:32:05.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0061000000000001e-05 2023-01-15T14:32:05.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.019108+0000", 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020080099999999999 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.020057+0000", 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094954499999999999 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150472+0000", 2023-01-15T14:32:05.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.130414958 2023-01-15T14:32:05.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:05.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218188+0000", 2023-01-15T14:32:05.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067715656999999999 2023-01-15T14:32:05.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:05.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218242+0000", 2023-01-15T14:32:05.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4651999999999998e-05 2023-01-15T14:32:05.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:05.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218284+0000", 2023-01-15T14:32:05.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1084999999999997e-05 2023-01-15T14:32:05.394 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.394 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:05.394 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.528 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:32:05.529 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:32:05.529 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 61.204564544, 2023-01-15T14:32:05.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:32:05.529 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:05.529 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:05.529 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:05.529 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:05.529 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:05.529 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:32:05.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.529 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:05.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:05.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:32:05.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:05.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:32:05.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:05.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:32:05.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:05.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:05.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:32:05.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:32:05.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:05.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:32:05.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:32:05.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:05.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:32:05.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:32:05.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:32:05.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:32:05.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:05.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:32:05.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:32:05.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:32:05.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:32:05.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:05.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:32:05.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:32:05.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:05.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:32:05.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:32:05.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.689 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:05.689 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.690 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1820 3.1f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:32:05.690 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:32:05.690 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 61.104263058999997, 2023-01-15T14:32:05.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306504938, 2023-01-15T14:32:05.690 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:05.690 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:05.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:05.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:05.690 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:05.691 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1820 2023-01-15T14:32:05.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.691 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:05.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:32:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:32:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:05.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903440+0000", 2023-01-15T14:32:05.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:05.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:05.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903448+0000", 2023-01-15T14:32:05.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9430000000000006e-06 2023-01-15T14:32:05.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903469+0000", 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0305e-05 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903481+0000", 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903616+0000", 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013477199999999999 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903643+0000", 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7098999999999999e-05 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209081+0000", 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30543790300000001 2023-01-15T14:32:05.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:05.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209213+0000", 2023-01-15T14:32:05.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013155400000000001 2023-01-15T14:32:05.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209600+0000", 2023-01-15T14:32:05.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038751699999999998 2023-01-15T14:32:05.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:05.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209946+0000", 2023-01-15T14:32:05.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034601 2023-01-15T14:32:05.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:05.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:05.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.865 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:05.865 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:32:05.865 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.321157863000003, 2023-01-15T14:32:05.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:32:05.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:05.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:05.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:05.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:05.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:05.866 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:32:05.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.866 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:05.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:05.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:32:05.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:05.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:32:05.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:05.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:32:05.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:32:05.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:05.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:32:05.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:32:05.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:32:05.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:32:05.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:05.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:32:05.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:32:05.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:05.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:32:05.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:32:05.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:05.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:32:05.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:32:05.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:05.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:32:05.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:05.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:32:05.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:32:05.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:05.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:05.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:05.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:32:06.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:32:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:32:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:32:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:32:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:32:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:32:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:32:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:32:06.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:32:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:32:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.317750740000001, 2023-01-15T14:32:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:32:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:06.010 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:32:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:32:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:06.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:32:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:32:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:32:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:32:06.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:32:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:32:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:32:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:32:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:32:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:32:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:32:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:32:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:32:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:32:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:32:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:32:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:06.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:32:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:32:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:32:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:32:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:32:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:32:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:32:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:32:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:06.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:32:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:32:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:32:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:32:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:06.021 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:32:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.317535135, 2023-01-15T14:32:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:32:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:32:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:06.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:32:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:32:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:32:06.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:32:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:32:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:32:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:32:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:32:06.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:32:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:32:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:32:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:32:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:06.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:32:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:32:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:32:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:32:06.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:32:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:32:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:32:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:32:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:32:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:32:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 39.317428061000001, 2023-01-15T14:32:06.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:32:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:32:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:06.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:06.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:06.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:32:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:32:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:32:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:32:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:06.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:32:06.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:32:06.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:32:06.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:32:06.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:06.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:32:06.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:32:06.179 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.187189011000001, 2023-01-15T14:32:06.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:32:06.180 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:06.180 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:06.181 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:06.181 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:06.181 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:06.181 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:32:06.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.181 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:06.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:06.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:32:06.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:06.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:32:06.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:32:06.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:06.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:32:06.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:32:06.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:06.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:32:06.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:32:06.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:32:06.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:32:06.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:32:06.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:32:06.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:32:06.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:32:06.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:32:06.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:06.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:32:06.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:32:06.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:06.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:32:06.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:32:06.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:06.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:32:06.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:32:06.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:06.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:32:06.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:32:06.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:06.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:32:06.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:32:06.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:06.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:32:06.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:32:06.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:32:06.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:32:06.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:32:06.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:32:06.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:06.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:32:06.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:32:06.190 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.187084473999999, 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:06.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:32:06.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:06.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:32:06.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:32:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:32:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:32:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:06.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:32:06.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:06.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:32:06.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:32:06.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:32:06.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:32:06.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:06.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:32:06.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:32:06.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:06.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:32:06.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:32:06.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:06.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:32:06.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:32:06.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:06.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:32:06.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:32:06.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:32:06.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:32:06.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:06.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.187014543, 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:06.327 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:32:06.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:32:06.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:32:06.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:06.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:32:06.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:06.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:32:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:32:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:32:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:32:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:06.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:32:06.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:32:06.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:32:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:32:06.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:06.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:32:06.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:32:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:32:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:32:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:32:06.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:32:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:32:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:32:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:06.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:32:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:32:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:32:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:32:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:06.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:32:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:32:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:32:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:32:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:32:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:32:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:32:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:32:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:32:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:32:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:32:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:32:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:06.346 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2284 3.10 e216/207 3:094fbf13:::smithi09025264-31:head v 216'274, mlcod=212'269)", 2023-01-15T14:32:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:32:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.841449501, 2023-01-15T14:32:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28055179600000002, 2023-01-15T14:32:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:06.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:32:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:32:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166254+0000", 2023-01-15T14:32:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166261+0000", 2023-01-15T14:32:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4080000000000003e-06 2023-01-15T14:32:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166273+0000", 2023-01-15T14:32:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.221e-05 2023-01-15T14:32:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166279+0000", 2023-01-15T14:32:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8170000000000004e-06 2023-01-15T14:32:06.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365575+0000", 2023-01-15T14:32:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199295791 2023-01-15T14:32:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365598+0000", 2023-01-15T14:32:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3913e-05 2023-01-15T14:32:06.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365737+0000", 2023-01-15T14:32:06.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138312 2023-01-15T14:32:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365763+0000", 2023-01-15T14:32:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6596000000000001e-05 2023-01-15T14:32:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365803+0000", 2023-01-15T14:32:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9544e-05 2023-01-15T14:32:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365842+0000", 2023-01-15T14:32:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9050000000000001e-05 2023-01-15T14:32:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.446807+0000", 2023-01-15T14:32:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.080964750000000002 2023-01-15T14:32:06.364 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2285 3.10 e216/207 3:094fbf13:::smithi09025264-31:head v 216'275, mlcod=212'269)", 2023-01-15T14:32:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:32:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 21.840901469999999, 2023-01-15T14:32:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28017387399999999, 2023-01-15T14:32:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:06.365 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:32:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:06.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:32:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166802+0000", 2023-01-15T14:32:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:06.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166810+0000", 2023-01-15T14:32:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6650000000000003e-06 2023-01-15T14:32:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166821+0000", 2023-01-15T14:32:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1129e-05 2023-01-15T14:32:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:06.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166827+0000", 2023-01-15T14:32:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4640000000000003e-06 2023-01-15T14:32:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365957+0000", 2023-01-15T14:32:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199129158 2023-01-15T14:32:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:06.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366005+0000", 2023-01-15T14:32:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8822000000000001e-05 2023-01-15T14:32:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366301+0000", 2023-01-15T14:32:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029544199999999998 2023-01-15T14:32:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:06.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366316+0000", 2023-01-15T14:32:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5078e-05 2023-01-15T14:32:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366350+0000", 2023-01-15T14:32:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3575000000000003e-05 2023-01-15T14:32:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:06.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366368+0000", 2023-01-15T14:32:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8267e-05 2023-01-15T14:32:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.446977+0000", 2023-01-15T14:32:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.080608964000000005 2023-01-15T14:32:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.4.0:534 3.1a e226/224 3:5919d83e:::smithi09025264-13:a9 v 226'240, mlcod=0'0)", 2023-01-15T14:32:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:32:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.472119867, 2023-01-15T14:32:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30202553599999998, 2023-01-15T14:32:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:06.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:32:06.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:06.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:32:06.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:06.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:06.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535584+0000", 2023-01-15T14:32:06.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:06.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:06.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535592+0000", 2023-01-15T14:32:06.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3010000000000007e-06 2023-01-15T14:32:06.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:06.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535604+0000", 2023-01-15T14:32:06.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1800000000000001e-05 2023-01-15T14:32:06.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:06.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.635708+0000", 2023-01-15T14:32:06.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10010366800000001 2023-01-15T14:32:06.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:06.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836079+0000", 2023-01-15T14:32:06.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20037171500000001 2023-01-15T14:32:06.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:06.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836113+0000", 2023-01-15T14:32:06.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:32:06.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:06.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836714+0000", 2023-01-15T14:32:06.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060093400000000002 2023-01-15T14:32:06.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:06.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836737+0000", 2023-01-15T14:32:06.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3073000000000001e-05 2023-01-15T14:32:06.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:06.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836816+0000", 2023-01-15T14:32:06.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8740999999999997e-05 2023-01-15T14:32:06.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:06.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836898+0000", 2023-01-15T14:32:06.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2731999999999993e-05 2023-01-15T14:32:06.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:06.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:06.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:06.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.837610+0000", 2023-01-15T14:32:06.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071176200000000001 2023-01-15T14:32:06.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:06.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:06.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:06.549 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:06.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:32:06.551 INFO:tasks.rados.rados.0.smithi090.stdout:951: expect (ObjNum 183 snap 40 seq_num 183) 2023-01-15T14:32:06.553 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:32:06.554 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:05.049+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:06.554 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:32:06.556 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:32:06.556 INFO:tasks.rados.rados.0.smithi090.stdout:952: finishing rollback tid 0 to smithi09025264-34 2023-01-15T14:32:06.993 INFO:tasks.rados.rados.0.smithi090.stdout:948: done (3 left) 2023-01-15T14:32:06.994 INFO:tasks.rados.rados.0.smithi090.stdout:951: done (2 left) 2023-01-15T14:32:06.994 INFO:tasks.rados.rados.0.smithi090.stdout:953: read oid 2 snap -1 2023-01-15T14:32:06.994 INFO:tasks.rados.rados.0.smithi090.stdout:953: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:32:06.994 INFO:tasks.rados.rados.0.smithi090.stdout:954: write (excl) oid 14 current snap is 93 2023-01-15T14:32:06.994 INFO:tasks.rados.rados.0.smithi090.stdout:954: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2023-01-15T14:32:06.995 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2023-01-15T14:32:04.910682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":247769335,"num_objects":111,"num_object_clones":59,"num_object_copies":333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":111,"num_whiteouts":10,"num_read":907,"num_read_kb":876520,"num_write":761,"num_write_kb":233112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":286947731,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10155,"ondisk_log_size":10155,"up":225,"acting":224,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":232,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":5798804,"kb_used_data":556442748,"kb_used_omap":34983,"kb_used_meta":0,"kb_avail":556397476,"statfs":{"total":575688990720,"available":569751015424,"internally_reserved":46358528,"allocated":569797373952,"data_stored":569797373952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35822907,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,3,4],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":533,"commit_latency_ns":0,"apply_latency_ns":533000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15036250,"num_objects":-16,"num_object_clones":-15,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-40,"num_objects_misplaced":-25,"num_objects_unfound":0,"num_objects_dirty":-16,"num_whiteouts":-2,"num_read":13,"num_read_kb":22068,"num_write":11,"num_write_kb":5999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":32070516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001146"},"pg_stats":[{"pgid":"3.41","version":"155'121","reported_seq":782,"reported_epoch":235,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:02.256085+0000","last_change":"2023-01-15T14:32:02.256226+0000","last_active":"2023-01-15T14:32:02.256085+0000","last_peered":"2023-01-15T14:32:02.256085+0000","last_clean":"2023-01-15T14:32:02.256085+0000","last_became_active":"2023-01-15T14:31:56.927677+0000","last_became_peered":"2023-01-15T14:31:56.927677+0000","last_unstale":"2023-01-15T14:32:02.256085+0000","last_undegraded":"2023-01-15T14:32:02.256085+0000","last_fullsized":"2023-01-15T14:32:02.256085+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:31:36.200118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:36.200118+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103601373,"objects_trimmed":0,"snaptrim_duration":0.42123817899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"}]},{"pgid":"3.40","version":"223'257","reported_seq":976,"reported_epoch":234,"state":"peering","last_fresh":"2023-01-15T14:32:00.308589+0000","last_change":"2023-01-15T14:32:01.524128+0000","last_active":"2023-01-15T14:32:01.524128+0000","last_peered":"2023-01-15T14:32:00.308589+0000","last_clean":"2023-01-15T14:31:51.035992+0000","last_became_active":"2023-01-15T14:31:55.447457+0000","last_became_peered":"2023-01-15T14:31:55.447457+0000","last_unstale":"2023-01-15T14:32:00.308589+0000","last_undegraded":"2023-01-15T14:32:00.308589+0000","last_fullsized":"2023-01-15T14:31:55.042520+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":169,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4294222,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":8588443,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"}]},{"pgid":"3.3f","version":"222'201","reported_seq":948,"reported_epoch":235,"state":"peering","last_fresh":"2023-01-15T14:32:01.073512+0000","last_change":"2023-01-15T14:32:02.524855+0000","last_active":"2023-01-15T14:32:02.524855+0000","last_peered":"2023-01-15T14:32:01.073512+0000","last_clean":"2023-01-15T14:31:49.660654+0000","last_became_active":"2023-01-15T14:31:55.712363+0000","last_became_peered":"2023-01-15T14:31:55.712363+0000","last_unstale":"2023-01-15T14:32:01.073512+0000","last_undegraded":"2023-01-15T14:32:01.073512+0000","last_fullsized":"2023-01-15T14:31:55.043184+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7921086,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":49,"num_read_kb":73148,"num_write":32,"num_write_kb":9074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":25819820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"}]},{"pgid":"3.3e","version":"233'40","reported_seq":502,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.071426+0000","last_change":"2023-01-15T14:31:58.951015+0000","last_active":"2023-01-15T14:32:01.071426+0000","last_peered":"2023-01-15T14:32:01.071426+0000","last_clean":"2023-01-15T14:32:01.071426+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:32:01.071426+0000","last_undegraded":"2023-01-15T14:32:01.071426+0000","last_fullsized":"2023-01-15T14:32:01.071426+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:31:44.906253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.906253+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.10427038299999999,"stat_sum":{"num_bytes":2369820,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":5765,"num_write":4,"num_write_kb":4338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":743710,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.3d","version":"232'203","reported_seq":981,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.971577+0000","last_change":"2023-01-15T14:31:59.971577+0000","last_active":"2023-01-15T14:31:59.971577+0000","last_peered":"2023-01-15T14:31:59.971577+0000","last_clean":"2023-01-15T14:31:59.971577+0000","last_became_active":"2023-01-15T14:31:59.249241+0000","last_became_peered":"2023-01-15T14:31:59.249241+0000","last_unstale":"2023-01-15T14:31:59.971577+0000","last_undegraded":"2023-01-15T14:31:59.971577+0000","last_fullsized":"2023-01-15T14:31:59.971577+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.112134794,"stat_sum":{"num_bytes":5739856,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":41,"num_read_kb":47622,"num_write":26,"num_write_kb":10631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":16861615,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.3c","version":"141'46","reported_seq":42,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.176043+0000","last_change":"2023-01-15T14:32:01.176668+0000","last_active":"2023-01-15T14:32:01.176043+0000","last_peered":"2023-01-15T14:32:01.176043+0000","last_clean":"2023-01-15T14:32:01.176043+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:32:01.176043+0000","last_undegraded":"2023-01-15T14:32:01.176043+0000","last_fullsized":"2023-01-15T14:32:01.176043+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.30933827899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":579834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":688,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.230836+0000","last_change":"2023-01-15T14:32:02.238883+0000","last_active":"2023-01-15T14:32:02.230836+0000","last_peered":"2023-01-15T14:32:02.230836+0000","last_clean":"2023-01-15T14:32:02.230836+0000","last_became_active":"2023-01-15T14:31:54.237985+0000","last_became_peered":"2023-01-15T14:31:54.237985+0000","last_unstale":"2023-01-15T14:32:02.230836+0000","last_undegraded":"2023-01-15T14:32:02.230836+0000","last_fullsized":"2023-01-15T14:32:02.230836+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.40471285899999998,"stat_sum":{"num_bytes":2659480,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6767,"num_write":3,"num_write_kb":817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1029,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.859266+0000","last_change":"2023-01-15T14:31:58.957927+0000","last_active":"2023-01-15T14:31:59.859266+0000","last_peered":"2023-01-15T14:31:59.859266+0000","last_clean":"2023-01-15T14:31:59.859266+0000","last_became_active":"2023-01-15T14:31:57.025706+0000","last_became_peered":"2023-01-15T14:31:57.025706+0000","last_unstale":"2023-01-15T14:31:59.859266+0000","last_undegraded":"2023-01-15T14:31:59.859266+0000","last_fullsized":"2023-01-15T14:31:59.859266+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":6,"last_scrub":"189'196","last_scrub_stamp":"2023-01-15T14:31:19.373635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:19.373635+0000","objects_scrubbed":4,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20803622599999999,"objects_trimmed":0,"snaptrim_duration":0.013899682999999999,"stat_sum":{"num_bytes":9013796,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":5060,"num_write":11,"num_write_kb":2753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.39","version":"225'228","reported_seq":962,"reported_epoch":234,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:32:01.689317+0000","last_change":"2023-01-15T14:32:01.689504+0000","last_active":"2023-01-15T14:32:01.689317+0000","last_peered":"2023-01-15T14:32:01.689317+0000","last_clean":"2023-01-15T14:31:52.640562+0000","last_became_active":"2023-01-15T14:31:55.615840+0000","last_became_peered":"2023-01-15T14:31:55.615840+0000","last_unstale":"2023-01-15T14:32:01.689317+0000","last_undegraded":"2023-01-15T14:32:01.689317+0000","last_fullsized":"2023-01-15T14:31:55.031006+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'157","last_scrub_stamp":"2023-01-15T14:30:26.792736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:26.792736+0000","objects_scrubbed":4,"log_size":228,"log_dups_size":0,"ondisk_log_size":228,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61998826900000004,"objects_trimmed":0,"snaptrim_duration":0.205956941,"stat_sum":{"num_bytes":11867927,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":23544,"num_write":16,"num_write_kb":4944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":14739756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[1,4],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"1,3,4","objects":6},{"shards":"1,4","objects":-1}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"}]},{"pgid":"3.38","version":"233'257","reported_seq":1146,"reported_epoch":234,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:32:00.968424+0000","last_change":"2023-01-15T14:32:00.772968+0000","last_active":"2023-01-15T14:32:00.968424+0000","last_peered":"2023-01-15T14:32:00.968424+0000","last_clean":"2023-01-15T14:32:00.968424+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:32:00.968424+0000","last_undegraded":"2023-01-15T14:32:00.968424+0000","last_fullsized":"2023-01-15T14:32:00.968424+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5462773,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":18938,"num_write":15,"num_write_kb":5973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"a1","length":"1"}]},{"pgid":"3.37","version":"200'78","reported_seq":104,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.920964+0000","last_change":"2023-01-15T14:32:01.351733+0000","last_active":"2023-01-15T14:32:01.920964+0000","last_peered":"2023-01-15T14:32:01.920964+0000","last_clean":"2023-01-15T14:32:01.920964+0000","last_became_active":"2023-01-15T14:31:37.024453+0000","last_became_peered":"2023-01-15T14:31:37.024453+0000","last_unstale":"2023-01-15T14:32:01.920964+0000","last_undegraded":"2023-01-15T14:32:01.920964+0000","last_fullsized":"2023-01-15T14:32:01.920964+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:09.695","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.309770143,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":263,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.859935+0000","last_change":"2023-01-15T14:31:58.955086+0000","last_active":"2023-01-15T14:31:59.859935+0000","last_peered":"2023-01-15T14:31:59.859935+0000","last_clean":"2023-01-15T14:31:59.859935+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:31:59.859935+0000","last_undegraded":"2023-01-15T14:31:59.859935+0000","last_fullsized":"2023-01-15T14:31:59.859935+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.107459233,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":681,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.741207+0000","last_change":"2023-01-15T14:32:00.741207+0000","last_active":"2023-01-15T14:32:00.741207+0000","last_peered":"2023-01-15T14:32:00.741207+0000","last_clean":"2023-01-15T14:32:00.741207+0000","last_became_active":"2023-01-15T14:31:38.033824+0000","last_became_peered":"2023-01-15T14:31:38.033824+0000","last_unstale":"2023-01-15T14:32:00.741207+0000","last_undegraded":"2023-01-15T14:32:00.741207+0000","last_fullsized":"2023-01-15T14:32:00.741207+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:30:24.721082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.721082+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.109000768,"objects_trimmed":0,"snaptrim_duration":0.88182932999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":852,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.834589+0000","last_change":"2023-01-15T14:32:01.249983+0000","last_active":"2023-01-15T14:32:01.834589+0000","last_peered":"2023-01-15T14:32:01.834589+0000","last_clean":"2023-01-15T14:32:01.834589+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:32:01.834589+0000","last_undegraded":"2023-01-15T14:32:01.834589+0000","last_fullsized":"2023-01-15T14:32:01.834589+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:48.327","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.34775705099999998,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5618,"num_write":3,"num_write_kb":609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":933,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.734854+0000","last_change":"2023-01-15T14:32:00.734854+0000","last_active":"2023-01-15T14:32:00.734854+0000","last_peered":"2023-01-15T14:32:00.734854+0000","last_clean":"2023-01-15T14:32:00.734854+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:32:00.734854+0000","last_undegraded":"2023-01-15T14:32:00.734854+0000","last_fullsized":"2023-01-15T14:32:00.734854+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.87739016000000003,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.32","version":"220'155","reported_seq":533,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:32:00.059656+0000","last_change":"2023-01-15T14:31:58.955921+0000","last_active":"2023-01-15T14:32:00.059656+0000","last_peered":"2023-01-15T14:32:00.059656+0000","last_clean":"2023-01-15T14:32:00.059656+0000","last_became_active":"2023-01-15T14:31:36.921952+0000","last_became_peered":"2023-01-15T14:31:36.921952+0000","last_unstale":"2023-01-15T14:32:00.059656+0000","last_undegraded":"2023-01-15T14:32:00.059656+0000","last_fullsized":"2023-01-15T14:32:00.059656+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:56.390","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.107769082,"stat_sum":{"num_bytes":2403401,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.13","version":"230'293","reported_seq":983,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.920389+0000","last_change":"2023-01-15T14:32:01.047031+0000","last_active":"2023-01-15T14:32:01.920389+0000","last_peered":"2023-01-15T14:32:01.920389+0000","last_clean":"2023-01-15T14:32:01.920389+0000","last_became_active":"2023-01-15T14:31:35.097414+0000","last_became_peered":"2023-01-15T14:31:35.097414+0000","last_unstale":"2023-01-15T14:32:01.920389+0000","last_undegraded":"2023-01-15T14:32:01.920389+0000","last_fullsized":"2023-01-15T14:32:01.920389+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:45.688","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.074697693999999995,"stat_sum":{"num_bytes":11359477,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":3497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.12","version":"230'151","reported_seq":508,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.943770+0000","last_change":"2023-01-15T14:32:00.943770+0000","last_active":"2023-01-15T14:32:00.943770+0000","last_peered":"2023-01-15T14:32:00.943770+0000","last_clean":"2023-01-15T14:32:00.943770+0000","last_became_active":"2023-01-15T14:31:35.028565+0000","last_became_peered":"2023-01-15T14:31:35.028565+0000","last_unstale":"2023-01-15T14:32:00.943770+0000","last_undegraded":"2023-01-15T14:32:00.943770+0000","last_fullsized":"2023-01-15T14:32:00.943770+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"139'103","last_scrub_stamp":"2023-01-15T14:30:30.045615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:30.045615+0000","objects_scrubbed":3,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.207128334,"objects_trimmed":0,"snaptrim_duration":0.88622885399999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":26971,"num_write":7,"num_write_kb":2217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6292614,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.11","version":"235'206","reported_seq":991,"reported_epoch":235,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:32:02.023381+0000","last_change":"2023-01-15T14:32:02.023381+0000","last_active":"2023-01-15T14:32:02.023381+0000","last_peered":"2023-01-15T14:32:02.023381+0000","last_clean":"2023-01-15T14:32:02.023381+0000","last_became_active":"2023-01-15T14:31:35.026209+0000","last_became_peered":"2023-01-15T14:31:35.026209+0000","last_unstale":"2023-01-15T14:32:02.023381+0000","last_undegraded":"2023-01-15T14:32:02.023381+0000","last_fullsized":"2023-01-15T14:32:02.023381+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":206,"log_dups_size":0,"ondisk_log_size":206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:06.167","scrub_duration":0.30779894200000002,"objects_trimmed":1,"snaptrim_duration":0.0017121459999999999,"stat_sum":{"num_bytes":17412184,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":20,"num_read_kb":27258,"num_write":5,"num_write_kb":6747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.10","version":"234'309","reported_seq":1073,"reported_epoch":234,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:32:01.176936+0000","last_change":"2023-01-15T14:32:01.176936+0000","last_active":"2023-01-15T14:32:01.176936+0000","last_peered":"2023-01-15T14:32:01.176936+0000","last_clean":"2023-01-15T14:32:01.176936+0000","last_became_active":"2023-01-15T14:31:35.135778+0000","last_became_peered":"2023-01-15T14:31:35.135778+0000","last_unstale":"2023-01-15T14:32:01.176936+0000","last_undegraded":"2023-01-15T14:32:01.176936+0000","last_fullsized":"2023-01-15T14:32:01.176936+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:50.443","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.57626714599999995,"stat_sum":{"num_bytes":10253458,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":35,"num_read_kb":25950,"num_write":55,"num_write_kb":18757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":13340457,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":915,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.240274+0000","last_change":"2023-01-15T14:32:02.241434+0000","last_active":"2023-01-15T14:32:02.240274+0000","last_peered":"2023-01-15T14:32:02.240274+0000","last_clean":"2023-01-15T14:32:02.240274+0000","last_became_active":"2023-01-15T14:31:35.032038+0000","last_became_peered":"2023-01-15T14:31:35.032038+0000","last_unstale":"2023-01-15T14:32:02.240274+0000","last_undegraded":"2023-01-15T14:32:02.240274+0000","last_fullsized":"2023-01-15T14:32:02.240274+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:31:22.549180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:22.549180+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50470070300000003,"objects_trimmed":0,"snaptrim_duration":0.30905149199999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.e","version":"139'38","reported_seq":715,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.239109+0000","last_change":"2023-01-15T14:32:02.240079+0000","last_active":"2023-01-15T14:32:02.239109+0000","last_peered":"2023-01-15T14:32:02.239109+0000","last_clean":"2023-01-15T14:32:02.239109+0000","last_became_active":"2023-01-15T14:31:35.237663+0000","last_became_peered":"2023-01-15T14:31:35.237663+0000","last_unstale":"2023-01-15T14:32:02.239109+0000","last_undegraded":"2023-01-15T14:32:02.239109+0000","last_fullsized":"2023-01-15T14:32:02.239109+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:16.384","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.30521432900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.d","version":"230'188","reported_seq":936,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.920411+0000","last_change":"2023-01-15T14:32:01.355874+0000","last_active":"2023-01-15T14:32:01.920411+0000","last_peered":"2023-01-15T14:32:01.920411+0000","last_clean":"2023-01-15T14:32:01.920411+0000","last_became_active":"2023-01-15T14:31:21.132225+0000","last_became_peered":"2023-01-15T14:31:21.132225+0000","last_unstale":"2023-01-15T14:32:01.920411+0000","last_undegraded":"2023-01-15T14:32:01.920411+0000","last_fullsized":"2023-01-15T14:32:01.920411+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:41.817","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.38341130600000001,"stat_sum":{"num_bytes":6067327,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.c","version":"225'55","reported_seq":701,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.739859+0000","last_change":"2023-01-15T14:32:00.739859+0000","last_active":"2023-01-15T14:32:00.739859+0000","last_peered":"2023-01-15T14:32:00.739859+0000","last_clean":"2023-01-15T14:32:00.739859+0000","last_became_active":"2023-01-15T14:31:21.227945+0000","last_became_peered":"2023-01-15T14:31:21.227945+0000","last_unstale":"2023-01-15T14:32:00.739859+0000","last_undegraded":"2023-01-15T14:32:00.739859+0000","last_fullsized":"2023-01-15T14:32:00.739859+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:42.336","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.880896805,"stat_sum":{"num_bytes":3445468,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":11,"num_write_kb":6025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":676,"reported_epoch":234,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:01.074338+0000","last_change":"2023-01-15T14:32:01.074369+0000","last_active":"2023-01-15T14:32:01.074338+0000","last_peered":"2023-01-15T14:32:01.074338+0000","last_clean":"2023-01-15T14:32:01.074338+0000","last_became_active":"2023-01-15T14:31:21.230392+0000","last_became_peered":"2023-01-15T14:31:21.230392+0000","last_unstale":"2023-01-15T14:32:01.074338+0000","last_undegraded":"2023-01-15T14:32:01.074338+0000","last_fullsized":"2023-01-15T14:32:01.074338+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:38.384","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105396515,"stat_sum":{"num_bytes":865560,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3347,"num_write":5,"num_write_kb":2817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.a","version":"234'251","reported_seq":984,"reported_epoch":235,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:02.153871+0000","last_change":"2023-01-15T14:32:02.154007+0000","last_active":"2023-01-15T14:32:02.153871+0000","last_peered":"2023-01-15T14:32:02.153871+0000","last_clean":"2023-01-15T14:32:02.153871+0000","last_became_active":"2023-01-15T14:31:21.032574+0000","last_became_peered":"2023-01-15T14:31:21.032574+0000","last_unstale":"2023-01-15T14:32:02.153871+0000","last_undegraded":"2023-01-15T14:32:02.153871+0000","last_fullsized":"2023-01-15T14:32:02.153871+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.23217431099999999,"stat_sum":{"num_bytes":8221912,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":41,"num_read_kb":42798,"num_write":46,"num_write_kb":17894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":18895452,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":797,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.136157+0000","last_change":"2023-01-15T14:32:01.422102+0000","last_active":"2023-01-15T14:32:02.136157+0000","last_peered":"2023-01-15T14:32:02.136157+0000","last_clean":"2023-01-15T14:32:02.136157+0000","last_became_active":"2023-01-15T14:31:21.122856+0000","last_became_peered":"2023-01-15T14:31:21.122856+0000","last_unstale":"2023-01-15T14:32:02.136157+0000","last_undegraded":"2023-01-15T14:32:02.136157+0000","last_fullsized":"2023-01-15T14:32:02.136157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:31:44.640345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.640345+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:46.259","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.51982624600000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":990,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.535072+0000","last_change":"2023-01-15T14:32:00.535072+0000","last_active":"2023-01-15T14:32:00.535072+0000","last_peered":"2023-01-15T14:32:00.535072+0000","last_clean":"2023-01-15T14:32:00.535072+0000","last_became_active":"2023-01-15T14:31:21.332760+0000","last_became_peered":"2023-01-15T14:31:21.332760+0000","last_unstale":"2023-01-15T14:32:00.535072+0000","last_undegraded":"2023-01-15T14:32:00.535072+0000","last_fullsized":"2023-01-15T14:32:00.535072+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.47867674999999998,"stat_sum":{"num_bytes":2159716,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":3981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":7874096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":369,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.958602+0000","last_change":"2023-01-15T14:31:03.091584+0000","last_active":"2023-01-15T14:31:59.958602+0000","last_peered":"2023-01-15T14:31:59.958602+0000","last_clean":"2023-01-15T14:31:59.958602+0000","last_became_active":"2023-01-15T14:31:03.091071+0000","last_became_peered":"2023-01-15T14:31:03.091071+0000","last_unstale":"2023-01-15T14:31:59.958602+0000","last_undegraded":"2023-01-15T14:31:59.958602+0000","last_fullsized":"2023-01-15T14:31:59.958602+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:29:45.876183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:29:45.876183+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.860213+0000","last_change":"2023-01-15T14:31:38.018850+0000","last_active":"2023-01-15T14:31:59.860213+0000","last_peered":"2023-01-15T14:31:59.860213+0000","last_clean":"2023-01-15T14:31:59.860213+0000","last_became_active":"2023-01-15T14:31:38.018145+0000","last_became_peered":"2023-01-15T14:31:38.018145+0000","last_unstale":"2023-01-15T14:31:59.860213+0000","last_undegraded":"2023-01-15T14:31:59.860213+0000","last_fullsized":"2023-01-15T14:31:59.860213+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:43.428","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":685,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.153954+0000","last_change":"2023-01-15T14:32:02.255179+0000","last_active":"2023-01-15T14:32:02.153954+0000","last_peered":"2023-01-15T14:32:02.153954+0000","last_clean":"2023-01-15T14:32:02.153954+0000","last_became_active":"2023-01-15T14:31:44.311861+0000","last_became_peered":"2023-01-15T14:31:44.311861+0000","last_unstale":"2023-01-15T14:32:02.153954+0000","last_undegraded":"2023-01-15T14:32:02.153954+0000","last_fullsized":"2023-01-15T14:32:02.153954+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.33355411699999998,"stat_sum":{"num_bytes":1084068,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3268978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":269,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.041409+0000","last_change":"2023-01-15T14:31:11.136609+0000","last_active":"2023-01-15T14:32:01.041409+0000","last_peered":"2023-01-15T14:32:01.041409+0000","last_clean":"2023-01-15T14:32:01.041409+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:32:01.041409+0000","last_undegraded":"2023-01-15T14:32:01.041409+0000","last_fullsized":"2023-01-15T14:32:01.041409+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:11.136490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.136490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:36.593","scrub_duration":0.39969354600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":766,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.035499+0000","last_change":"2023-01-15T14:32:01.352299+0000","last_active":"2023-01-15T14:32:02.035499+0000","last_peered":"2023-01-15T14:32:02.035499+0000","last_clean":"2023-01-15T14:32:02.035499+0000","last_became_active":"2023-01-15T14:31:52.074023+0000","last_became_peered":"2023-01-15T14:31:52.074023+0000","last_unstale":"2023-01-15T14:32:02.035499+0000","last_undegraded":"2023-01-15T14:32:02.035499+0000","last_fullsized":"2023-01-15T14:32:02.035499+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:31:36.200118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:36.200118+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103601373,"objects_trimmed":0,"snaptrim_duration":0.45034833800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":37,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.956874+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:31:59.956874+0000","last_peered":"2023-01-15T14:31:59.956874+0000","last_clean":"2023-01-15T14:31:59.956874+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:31:59.956874+0000","last_undegraded":"2023-01-15T14:31:59.956874+0000","last_fullsized":"2023-01-15T14:31:59.956874+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:38.960","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1013,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.959108+0000","last_change":"2023-01-15T14:31:59.050447+0000","last_active":"2023-01-15T14:31:59.959108+0000","last_peered":"2023-01-15T14:31:59.959108+0000","last_clean":"2023-01-15T14:31:59.959108+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:31:59.959108+0000","last_undegraded":"2023-01-15T14:31:59.959108+0000","last_fullsized":"2023-01-15T14:31:59.959108+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.103129159,"stat_sum":{"num_bytes":4294221,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":91,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.957090+0000","last_change":"2023-01-15T14:31:13.764052+0000","last_active":"2023-01-15T14:31:59.957090+0000","last_peered":"2023-01-15T14:31:59.957090+0000","last_clean":"2023-01-15T14:31:59.957090+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:31:59.957090+0000","last_undegraded":"2023-01-15T14:31:59.957090+0000","last_fullsized":"2023-01-15T14:31:59.957090+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:13.763945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:13.763945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:39.419","scrub_duration":0.58613919699999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"235'294","reported_seq":1106,"reported_epoch":235,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:32:02.242946+0000","last_change":"2023-01-15T14:32:02.242946+0000","last_active":"2023-01-15T14:32:02.242946+0000","last_peered":"2023-01-15T14:32:02.242946+0000","last_clean":"2023-01-15T14:32:02.242946+0000","last_became_active":"2023-01-15T14:31:03.093753+0000","last_became_peered":"2023-01-15T14:31:03.093753+0000","last_unstale":"2023-01-15T14:32:02.242946+0000","last_undegraded":"2023-01-15T14:32:02.242946+0000","last_fullsized":"2023-01-15T14:32:02.242946+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'251","last_scrub_stamp":"2023-01-15T14:31:15.363400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.363400+0000","objects_scrubbed":3,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:42.073","scrub_duration":0.208136242,"objects_trimmed":0,"snaptrim_duration":0.39660716200000001,"stat_sum":{"num_bytes":11773720,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":41,"num_read_kb":44989,"num_write":22,"num_write_kb":8236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.5","version":"233'111","reported_seq":791,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.772943+0000","last_change":"2023-01-15T14:32:00.772943+0000","last_active":"2023-01-15T14:32:00.772943+0000","last_peered":"2023-01-15T14:32:00.772943+0000","last_clean":"2023-01-15T14:32:00.772943+0000","last_became_active":"2023-01-15T14:31:21.132693+0000","last_became_peered":"2023-01-15T14:31:21.132693+0000","last_unstale":"2023-01-15T14:32:00.772943+0000","last_undegraded":"2023-01-15T14:32:00.772943+0000","last_fullsized":"2023-01-15T14:32:00.772943+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:25.367","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81542173699999998,"stat_sum":{"num_bytes":7758029,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":15022,"num_write":29,"num_write_kb":9886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9491720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":81,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.859161+0000","last_change":"2023-01-15T14:30:51.237821+0000","last_active":"2023-01-15T14:31:59.859161+0000","last_peered":"2023-01-15T14:31:59.859161+0000","last_clean":"2023-01-15T14:31:59.859161+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:31:59.859161+0000","last_undegraded":"2023-01-15T14:31:59.859161+0000","last_fullsized":"2023-01-15T14:31:59.859161+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"232'254","reported_seq":1054,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.859083+0000","last_change":"2023-01-15T14:31:59.549531+0000","last_active":"2023-01-15T14:31:59.859083+0000","last_peered":"2023-01-15T14:31:59.859083+0000","last_clean":"2023-01-15T14:31:59.859083+0000","last_became_active":"2023-01-15T14:31:21.226293+0000","last_became_peered":"2023-01-15T14:31:21.226293+0000","last_unstale":"2023-01-15T14:31:59.859083+0000","last_undegraded":"2023-01-15T14:31:59.859083+0000","last_fullsized":"2023-01-15T14:31:59.859083+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":1,"snaptrim_duration":0.70269059300000003,"stat_sum":{"num_bytes":8546124,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":46338,"num_write":28,"num_write_kb":6428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":15862107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":261,"reported_epoch":234,"state":"active+clean","last_fresh":"2023-01-15T14:32:00.972724+0000","last_change":"2023-01-15T14:31:15.088518+0000","last_active":"2023-01-15T14:32:00.972724+0000","last_peered":"2023-01-15T14:32:00.972724+0000","last_clean":"2023-01-15T14:32:00.972724+0000","last_became_active":"2023-01-15T14:31:03.088392+0000","last_became_peered":"2023-01-15T14:31:03.088392+0000","last_unstale":"2023-01-15T14:32:00.972724+0000","last_undegraded":"2023-01-15T14:32:00.972724+0000","last_fullsized":"2023-01-15T14:32:00.972724+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:15.088397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.088397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:32.351","scrub_duration":0.40518630900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"232'108","reported_seq":790,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.241647+0000","last_change":"2023-01-15T14:32:02.242824+0000","last_active":"2023-01-15T14:32:02.241647+0000","last_peered":"2023-01-15T14:32:02.241647+0000","last_clean":"2023-01-15T14:32:02.241647+0000","last_became_active":"2023-01-15T14:31:44.117230+0000","last_became_peered":"2023-01-15T14:31:44.117230+0000","last_unstale":"2023-01-15T14:32:02.241647+0000","last_undegraded":"2023-01-15T14:32:02.241647+0000","last_fullsized":"2023-01-15T14:32:02.241647+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.31098066299999999,"stat_sum":{"num_bytes":4920498,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":8458,"num_write":9,"num_write_kb":2509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1731121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.961133+0000","last_change":"2023-01-15T14:30:51.035825+0000","last_active":"2023-01-15T14:31:59.961133+0000","last_peered":"2023-01-15T14:31:59.961133+0000","last_clean":"2023-01-15T14:31:59.961133+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:31:59.961133+0000","last_undegraded":"2023-01-15T14:31:59.961133+0000","last_fullsized":"2023-01-15T14:31:59.961133+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":701,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.860196+0000","last_change":"2023-01-15T14:31:59.051775+0000","last_active":"2023-01-15T14:31:59.860196+0000","last_peered":"2023-01-15T14:31:59.860196+0000","last_clean":"2023-01-15T14:31:59.860196+0000","last_became_active":"2023-01-15T14:31:21.030929+0000","last_became_peered":"2023-01-15T14:31:21.030929+0000","last_unstale":"2023-01-15T14:31:59.860196+0000","last_undegraded":"2023-01-15T14:31:59.860196+0000","last_fullsized":"2023-01-15T14:31:59.860196+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:31:25.342830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:25.342830+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:44.671","scrub_duration":0.20886623600000001,"objects_trimmed":0,"snaptrim_duration":0.104070417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":217,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.857880+0000","last_change":"2023-01-15T14:31:38.112378+0000","last_active":"2023-01-15T14:31:59.857880+0000","last_peered":"2023-01-15T14:31:59.857880+0000","last_clean":"2023-01-15T14:31:59.857880+0000","last_became_active":"2023-01-15T14:31:38.111590+0000","last_became_peered":"2023-01-15T14:31:38.111590+0000","last_unstale":"2023-01-15T14:31:59.857880+0000","last_undegraded":"2023-01-15T14:31:59.857880+0000","last_fullsized":"2023-01-15T14:31:59.857880+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:51.156","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"169'138","reported_seq":836,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.127090+0000","last_change":"2023-01-15T14:32:02.128203+0000","last_active":"2023-01-15T14:32:02.127090+0000","last_peered":"2023-01-15T14:32:02.127090+0000","last_clean":"2023-01-15T14:32:02.127090+0000","last_became_active":"2023-01-15T14:31:35.134196+0000","last_became_peered":"2023-01-15T14:31:35.134196+0000","last_unstale":"2023-01-15T14:32:02.127090+0000","last_undegraded":"2023-01-15T14:32:02.127090+0000","last_fullsized":"2023-01-15T14:32:02.127090+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.19238931100000001,"stat_sum":{"num_bytes":1226927,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":15209,"num_write":2,"num_write_kb":608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":671,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.742447+0000","last_change":"2023-01-15T14:32:00.742447+0000","last_active":"2023-01-15T14:32:00.742447+0000","last_peered":"2023-01-15T14:32:00.742447+0000","last_clean":"2023-01-15T14:32:00.742447+0000","last_became_active":"2023-01-15T14:31:34.951721+0000","last_became_peered":"2023-01-15T14:31:34.951721+0000","last_unstale":"2023-01-15T14:32:00.742447+0000","last_undegraded":"2023-01-15T14:32:00.742447+0000","last_fullsized":"2023-01-15T14:32:00.742447+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:30:24.721082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.721082+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.109000768,"objects_trimmed":0,"snaptrim_duration":0.78208967799999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.16","version":"230'100","reported_seq":287,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.860489+0000","last_change":"2023-01-15T14:31:58.956909+0000","last_active":"2023-01-15T14:31:59.860489+0000","last_peered":"2023-01-15T14:31:59.860489+0000","last_clean":"2023-01-15T14:31:59.860489+0000","last_became_active":"2023-01-15T14:31:35.032807+0000","last_became_peered":"2023-01-15T14:31:35.032807+0000","last_unstale":"2023-01-15T14:31:59.860489+0000","last_undegraded":"2023-01-15T14:31:59.860489+0000","last_fullsized":"2023-01-15T14:31:59.860489+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:40.949","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.10862221800000001,"stat_sum":{"num_bytes":4522302,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":13661,"num_write":5,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.17","version":"228'82","reported_seq":706,"reported_epoch":234,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:01.175866+0000","last_change":"2023-01-15T14:32:01.176022+0000","last_active":"2023-01-15T14:32:01.175866+0000","last_peered":"2023-01-15T14:32:01.175866+0000","last_clean":"2023-01-15T14:32:01.175866+0000","last_became_active":"2023-01-15T14:31:34.932309+0000","last_became_peered":"2023-01-15T14:31:34.932309+0000","last_unstale":"2023-01-15T14:32:01.175866+0000","last_undegraded":"2023-01-15T14:32:01.175866+0000","last_fullsized":"2023-01-15T14:32:01.175866+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'65","last_scrub_stamp":"2023-01-15T14:30:24.156798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.156798+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30368223799999999,"objects_trimmed":0,"snaptrim_duration":0.30744262500000002,"stat_sum":{"num_bytes":3731187,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":8,"num_write_kb":2801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1974822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.18","version":"231'257","reported_seq":1106,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.633330+0000","last_change":"2023-01-15T14:32:00.633330+0000","last_active":"2023-01-15T14:32:00.633330+0000","last_peered":"2023-01-15T14:32:00.633330+0000","last_clean":"2023-01-15T14:32:00.633330+0000","last_became_active":"2023-01-15T14:31:52.330386+0000","last_became_peered":"2023-01-15T14:31:52.330386+0000","last_unstale":"2023-01-15T14:32:00.633330+0000","last_undegraded":"2023-01-15T14:32:00.633330+0000","last_fullsized":"2023-01-15T14:32:00.633330+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.67683554999999995,"stat_sum":{"num_bytes":6337110,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":18938,"num_write":15,"num_write_kb":5972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":11033036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.19","version":"230'225","reported_seq":955,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.858872+0000","last_change":"2023-01-15T14:31:58.953335+0000","last_active":"2023-01-15T14:31:59.858872+0000","last_peered":"2023-01-15T14:31:59.858872+0000","last_clean":"2023-01-15T14:31:59.858872+0000","last_became_active":"2023-01-15T14:31:51.927440+0000","last_became_peered":"2023-01-15T14:31:51.927440+0000","last_unstale":"2023-01-15T14:31:59.858872+0000","last_undegraded":"2023-01-15T14:31:59.858872+0000","last_fullsized":"2023-01-15T14:31:59.858872+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'157","last_scrub_stamp":"2023-01-15T14:30:26.792736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:26.792736+0000","objects_scrubbed":4,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61998826900000004,"objects_trimmed":0,"snaptrim_duration":0.106481008,"stat_sum":{"num_bytes":9229042,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":23543,"num_write":10,"num_write_kb":3020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":14739755,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.1a","version":"235'257","reported_seq":1045,"reported_epoch":235,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:32:02.256306+0000","last_change":"2023-01-15T14:32:02.256306+0000","last_active":"2023-01-15T14:32:02.256306+0000","last_peered":"2023-01-15T14:32:02.256306+0000","last_clean":"2023-01-15T14:32:02.256306+0000","last_became_active":"2023-01-15T14:31:52.124026+0000","last_became_peered":"2023-01-15T14:31:52.124026+0000","last_unstale":"2023-01-15T14:32:02.256306+0000","last_undegraded":"2023-01-15T14:32:02.256306+0000","last_fullsized":"2023-01-15T14:32:02.256306+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'196","last_scrub_stamp":"2023-01-15T14:31:19.373635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:19.373635+0000","objects_scrubbed":4,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20803622599999999,"objects_trimmed":0,"snaptrim_duration":0.205838667,"stat_sum":{"num_bytes":6861946,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":5061,"num_write":15,"num_write_kb":4341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.1b","version":"233'68","reported_seq":684,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.128310+0000","last_change":"2023-01-15T14:32:02.129466+0000","last_active":"2023-01-15T14:32:02.128310+0000","last_peered":"2023-01-15T14:32:02.128310+0000","last_clean":"2023-01-15T14:32:02.128310+0000","last_became_active":"2023-01-15T14:31:52.321399+0000","last_became_peered":"2023-01-15T14:31:52.321399+0000","last_unstale":"2023-01-15T14:32:02.128310+0000","last_undegraded":"2023-01-15T14:32:02.128310+0000","last_fullsized":"2023-01-15T14:32:02.128310+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.29478042700000001,"stat_sum":{"num_bytes":2659479,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":6766,"num_write":2,"num_write_kb":817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":224,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.835267+0000","last_change":"2023-01-15T14:32:01.348789+0000","last_active":"2023-01-15T14:32:01.835267+0000","last_peered":"2023-01-15T14:32:01.835267+0000","last_clean":"2023-01-15T14:32:01.835267+0000","last_became_active":"2023-01-15T14:31:51.926652+0000","last_became_peered":"2023-01-15T14:31:51.926652+0000","last_unstale":"2023-01-15T14:32:01.835267+0000","last_undegraded":"2023-01-15T14:32:01.835267+0000","last_fullsized":"2023-01-15T14:32:01.835267+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:33.090","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.44579417199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":961,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.835098+0000","last_change":"2023-01-15T14:32:01.350091+0000","last_active":"2023-01-15T14:32:01.835098+0000","last_peered":"2023-01-15T14:32:01.835098+0000","last_clean":"2023-01-15T14:32:01.835098+0000","last_became_active":"2023-01-15T14:31:52.025227+0000","last_became_peered":"2023-01-15T14:31:52.025227+0000","last_unstale":"2023-01-15T14:32:01.835098+0000","last_undegraded":"2023-01-15T14:32:01.835098+0000","last_fullsized":"2023-01-15T14:32:01.835098+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:32.052","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.347090589,"stat_sum":{"num_bytes":5739855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":47622,"num_write":25,"num_write_kb":10630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6928644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":660,"reported_epoch":234,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.969023+0000","last_change":"2023-01-15T14:32:00.969104+0000","last_active":"2023-01-15T14:32:00.969023+0000","last_peered":"2023-01-15T14:32:00.969023+0000","last_clean":"2023-01-15T14:32:00.969023+0000","last_became_active":"2023-01-15T14:31:52.329815+0000","last_became_peered":"2023-01-15T14:31:52.329815+0000","last_unstale":"2023-01-15T14:32:00.969023+0000","last_undegraded":"2023-01-15T14:32:00.969023+0000","last_fullsized":"2023-01-15T14:32:00.969023+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:31:44.906253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.906253+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55943661099999997,"objects_trimmed":0,"snaptrim_duration":0.001054744,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.1f","version":"233'220","reported_seq":1015,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.129525+0000","last_change":"2023-01-15T14:32:02.230645+0000","last_active":"2023-01-15T14:32:02.129525+0000","last_peered":"2023-01-15T14:32:02.129525+0000","last_clean":"2023-01-15T14:32:02.129525+0000","last_became_active":"2023-01-15T14:31:52.124821+0000","last_became_peered":"2023-01-15T14:31:52.124821+0000","last_unstale":"2023-01-15T14:32:02.129525+0000","last_undegraded":"2023-01-15T14:32:02.129525+0000","last_fullsized":"2023-01-15T14:32:02.129525+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.29881852800000003,"stat_sum":{"num_bytes":8801773,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":73149,"num_write":36,"num_write_kb":10844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":18290251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.20","version":"230'253","reported_seq":1060,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.919865+0000","last_change":"2023-01-15T14:32:01.142999+0000","last_active":"2023-01-15T14:32:01.919865+0000","last_peered":"2023-01-15T14:32:01.919865+0000","last_clean":"2023-01-15T14:32:01.919865+0000","last_became_active":"2023-01-15T14:31:07.906191+0000","last_became_peered":"2023-01-15T14:31:07.906191+0000","last_unstale":"2023-01-15T14:32:01.919865+0000","last_undegraded":"2023-01-15T14:32:01.919865+0000","last_fullsized":"2023-01-15T14:32:01.919865+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":253,"log_dups_size":0,"ondisk_log_size":253,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.0013828460000000001,"stat_sum":{"num_bytes":2100908,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":20,"num_read_kb":21726,"num_write":8,"num_write_kb":2065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8877936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.21","version":"232'127","reported_seq":702,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.020773+0000","last_change":"2023-01-15T14:32:01.354452+0000","last_active":"2023-01-15T14:32:02.020773+0000","last_peered":"2023-01-15T14:32:02.020773+0000","last_clean":"2023-01-15T14:32:02.020773+0000","last_became_active":"2023-01-15T14:31:38.233481+0000","last_became_peered":"2023-01-15T14:31:38.233481+0000","last_unstale":"2023-01-15T14:32:02.020773+0000","last_undegraded":"2023-01-15T14:32:02.020773+0000","last_fullsized":"2023-01-15T14:32:02.020773+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"168'124","last_scrub_stamp":"2023-01-15T14:31:11.414546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.414546+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:48.531","scrub_duration":0.28528039199999999,"objects_trimmed":0,"snaptrim_duration":0.209368896,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":1855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3798382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.22","version":"232'124","reported_seq":961,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.737331+0000","last_change":"2023-01-15T14:32:00.737331+0000","last_active":"2023-01-15T14:32:00.737331+0000","last_peered":"2023-01-15T14:32:00.737331+0000","last_clean":"2023-01-15T14:32:00.737331+0000","last_became_active":"2023-01-15T14:31:08.084483+0000","last_became_peered":"2023-01-15T14:31:08.084483+0000","last_unstale":"2023-01-15T14:32:00.737331+0000","last_undegraded":"2023-01-15T14:32:00.737331+0000","last_fullsized":"2023-01-15T14:32:00.737331+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:31.869","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.87615301599999995,"stat_sum":{"num_bytes":2487469,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.23","version":"149'233","reported_seq":873,"reported_epoch":234,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.970430+0000","last_change":"2023-01-15T14:32:00.970546+0000","last_active":"2023-01-15T14:32:00.970430+0000","last_peered":"2023-01-15T14:32:00.970430+0000","last_clean":"2023-01-15T14:32:00.970430+0000","last_became_active":"2023-01-15T14:31:06.029716+0000","last_became_peered":"2023-01-15T14:31:06.029716+0000","last_unstale":"2023-01-15T14:32:00.970430+0000","last_undegraded":"2023-01-15T14:32:00.970430+0000","last_fullsized":"2023-01-15T14:32:00.970430+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:48.395","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.102122195,"stat_sum":{"num_bytes":7604272,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":47,"num_read_kb":43637,"num_write":37,"num_write_kb":13160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2469494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.24","version":"232'219","reported_seq":1030,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.738710+0000","last_change":"2023-01-15T14:32:00.738710+0000","last_active":"2023-01-15T14:32:00.738710+0000","last_peered":"2023-01-15T14:32:00.738710+0000","last_clean":"2023-01-15T14:32:00.738710+0000","last_became_active":"2023-01-15T14:31:26.168777+0000","last_became_peered":"2023-01-15T14:31:26.168777+0000","last_unstale":"2023-01-15T14:32:00.738710+0000","last_undegraded":"2023-01-15T14:32:00.738710+0000","last_fullsized":"2023-01-15T14:32:00.738710+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.88005697900000002,"stat_sum":{"num_bytes":5983581,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":47,"num_read_kb":63482,"num_write":21,"num_write_kb":6075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":21209193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.25","version":"230'91","reported_seq":787,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.859551+0000","last_change":"2023-01-15T14:31:58.958830+0000","last_active":"2023-01-15T14:31:59.859551+0000","last_peered":"2023-01-15T14:31:59.859551+0000","last_clean":"2023-01-15T14:31:59.859551+0000","last_became_active":"2023-01-15T14:31:22.000386+0000","last_became_peered":"2023-01-15T14:31:22.000386+0000","last_unstale":"2023-01-15T14:31:59.859551+0000","last_undegraded":"2023-01-15T14:31:59.859551+0000","last_fullsized":"2023-01-15T14:31:59.859551+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:51.596","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.014386312,"stat_sum":{"num_bytes":2067294,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":7,"num_read_kb":6061,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":141,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:02.016953+0000","last_change":"2023-01-15T14:32:01.147229+0000","last_active":"2023-01-15T14:32:02.016953+0000","last_peered":"2023-01-15T14:32:02.016953+0000","last_clean":"2023-01-15T14:32:02.016953+0000","last_became_active":"2023-01-15T14:31:21.901622+0000","last_became_peered":"2023-01-15T14:31:21.901622+0000","last_unstale":"2023-01-15T14:32:02.016953+0000","last_undegraded":"2023-01-15T14:32:02.016953+0000","last_fullsized":"2023-01-15T14:32:02.016953+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:31:29.194049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:29.194049+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:33.783","scrub_duration":0.55352289399999999,"objects_trimmed":0,"snaptrim_duration":0.103149489,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":746,"reported_epoch":234,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.973374+0000","last_change":"2023-01-15T14:32:00.973498+0000","last_active":"2023-01-15T14:32:00.973374+0000","last_peered":"2023-01-15T14:32:00.973374+0000","last_clean":"2023-01-15T14:32:00.973374+0000","last_became_active":"2023-01-15T14:31:22.091646+0000","last_became_peered":"2023-01-15T14:31:22.091646+0000","last_unstale":"2023-01-15T14:32:00.973374+0000","last_undegraded":"2023-01-15T14:32:00.973374+0000","last_fullsized":"2023-01-15T14:32:00.973374+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.1057995,"stat_sum":{"num_bytes":2485558,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":8458,"num_write":2,"num_write_kb":737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1731121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.28","version":"231'220","reported_seq":1043,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.533959+0000","last_change":"2023-01-15T14:32:00.533959+0000","last_active":"2023-01-15T14:32:00.533959+0000","last_peered":"2023-01-15T14:32:00.533959+0000","last_clean":"2023-01-15T14:32:00.533959+0000","last_became_active":"2023-01-15T14:31:26.054878+0000","last_became_peered":"2023-01-15T14:31:26.054878+0000","last_unstale":"2023-01-15T14:32:00.533959+0000","last_undegraded":"2023-01-15T14:32:00.533959+0000","last_fullsized":"2023-01-15T14:32:00.533959+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.57795091799999998,"stat_sum":{"num_bytes":8420,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":12,"num_read_kb":10594,"num_write":3,"num_write_kb":3981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":12193528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.29","version":"110'149","reported_seq":568,"reported_epoch":233,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:32:00.736108+0000","last_change":"2023-01-15T14:32:00.736108+0000","last_active":"2023-01-15T14:32:00.736108+0000","last_peered":"2023-01-15T14:32:00.736108+0000","last_clean":"2023-01-15T14:32:00.736108+0000","last_became_active":"2023-01-15T14:31:25.136376+0000","last_became_peered":"2023-01-15T14:31:25.136376+0000","last_unstale":"2023-01-15T14:32:00.736108+0000","last_undegraded":"2023-01-15T14:32:00.736108+0000","last_fullsized":"2023-01-15T14:32:00.736108+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:30:08.681869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:08.681869+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.87946543799999999,"stat_sum":{"num_bytes":2857174,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":24874,"num_write":16,"num_write_kb":6171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ae","length":"1"}]},{"pgid":"3.2a","version":"191'191","reported_seq":936,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.919894+0000","last_change":"2023-01-15T14:32:01.144114+0000","last_active":"2023-01-15T14:32:01.919894+0000","last_peered":"2023-01-15T14:32:01.919894+0000","last_clean":"2023-01-15T14:32:01.919894+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:32:01.919894+0000","last_undegraded":"2023-01-15T14:32:01.919894+0000","last_fullsized":"2023-01-15T14:32:01.919894+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:44.953","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.17191088800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":641,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.860876+0000","last_change":"2023-01-15T14:31:58.952284+0000","last_active":"2023-01-15T14:31:59.860876+0000","last_peered":"2023-01-15T14:31:59.860876+0000","last_clean":"2023-01-15T14:31:59.860876+0000","last_became_active":"2023-01-15T14:31:21.896494+0000","last_became_peered":"2023-01-15T14:31:21.896494+0000","last_unstale":"2023-01-15T14:31:59.860876+0000","last_undegraded":"2023-01-15T14:31:59.860876+0000","last_fullsized":"2023-01-15T14:31:59.860876+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:30.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.975433+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:33.669","scrub_duration":0.0040286109999999997,"objects_trimmed":0,"snaptrim_duration":0.105409479,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.2c","version":"141'46","reported_seq":131,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.834572+0000","last_change":"2023-01-15T14:32:01.248806+0000","last_active":"2023-01-15T14:32:01.834572+0000","last_peered":"2023-01-15T14:32:01.834572+0000","last_clean":"2023-01-15T14:32:01.834572+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:32:01.834572+0000","last_undegraded":"2023-01-15T14:32:01.834572+0000","last_fullsized":"2023-01-15T14:32:01.834572+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:47.318","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.34608389699999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":909,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.921497+0000","last_change":"2023-01-15T14:32:01.145671+0000","last_active":"2023-01-15T14:32:01.921497+0000","last_peered":"2023-01-15T14:32:01.921497+0000","last_clean":"2023-01-15T14:32:01.921497+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:32:01.921497+0000","last_undegraded":"2023-01-15T14:32:01.921497+0000","last_fullsized":"2023-01-15T14:32:01.921497+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.173066782,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":586,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.860066+0000","last_change":"2023-01-15T14:31:58.954112+0000","last_active":"2023-01-15T14:31:59.860066+0000","last_peered":"2023-01-15T14:31:59.860066+0000","last_clean":"2023-01-15T14:31:59.860066+0000","last_became_active":"2023-01-15T14:31:38.116756+0000","last_became_peered":"2023-01-15T14:31:38.116756+0000","last_unstale":"2023-01-15T14:31:59.860066+0000","last_undegraded":"2023-01-15T14:31:59.860066+0000","last_fullsized":"2023-01-15T14:31:59.860066+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:51.292","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.10687018600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.2f","version":"155'145","reported_seq":774,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.920829+0000","last_change":"2023-01-15T14:32:01.349523+0000","last_active":"2023-01-15T14:32:01.920829+0000","last_peered":"2023-01-15T14:32:01.920829+0000","last_clean":"2023-01-15T14:32:01.920829+0000","last_became_active":"2023-01-15T14:31:40.024531+0000","last_became_peered":"2023-01-15T14:31:40.024531+0000","last_unstale":"2023-01-15T14:32:01.920829+0000","last_undegraded":"2023-01-15T14:32:01.920829+0000","last_fullsized":"2023-01-15T14:32:01.920829+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:31:22.549180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:22.549180+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30725682300000001,"stat_sum":{"num_bytes":3113505,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":16794,"num_write":19,"num_write_kb":6416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7449703,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":1037,"reported_epoch":235,"state":"active+clean","last_fresh":"2023-01-15T14:32:01.919749+0000","last_change":"2023-01-15T14:32:01.353223+0000","last_active":"2023-01-15T14:32:01.919749+0000","last_peered":"2023-01-15T14:32:01.919749+0000","last_clean":"2023-01-15T14:32:01.919749+0000","last_became_active":"2023-01-15T14:31:39.915702+0000","last_became_peered":"2023-01-15T14:31:39.915702+0000","last_unstale":"2023-01-15T14:32:01.919749+0000","last_undegraded":"2023-01-15T14:32:01.919749+0000","last_fullsized":"2023-01-15T14:32:01.919749+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:58.379","scrub_duration":1.1539330249999999,"objects_trimmed":0,"snaptrim_duration":0.31219691100000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]},{"pgid":"3.31","version":"204'150","reported_seq":910,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-01-15T14:31:59.858505+0000","last_change":"2023-01-15T14:31:58.848092+0000","last_active":"2023-01-15T14:31:59.858505+0000","last_peered":"2023-01-15T14:31:59.858505+0000","last_clean":"2023-01-15T14:31:59.858505+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:31:59.858505+0000","last_undegraded":"2023-01-15T14:31:59.858505+0000","last_fullsized":"2023-01-15T14:31:59.858505+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"186'144","last_scrub_stamp":"2023-01-15T14:31:13.806973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:13.806973+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.001723584,"stat_sum":{"num_bytes":7117814,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":21449,"num_write":7,"num_write_kb":6179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":7210220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7b","length":"1"},{"start":"8b","length":"1"},{"start":"92","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a1","length":"1"},{"start":"a5","length":"1"},{"start":"a7","length":"1"},{"start":"ad","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":66,"stat_sum":{"num_bytes":247178967,"num_objects":109,"num_object_clones":59,"num_object_copies":327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":109,"num_whiteouts":10,"num_read":791,"num_read_kb":876420,"num_write":602,"num_write_kb":230696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":284586259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10038,"ondisk_log_size":10038,"up":198,"acting":197,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574897,"num_pgs":24,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":871084,"kb_used_data":92829328,"kb_used_omap":3443,"kb_used_meta":0,"kb_avail":92828296,"statfs":{"total":95948165120,"available":95056175104,"internally_reserved":1056768,"allocated":95057231872,"data_stored":95057231872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3525868,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,3,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":525,"commit_latency_ns":0,"apply_latency_ns":525000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":14.18,"5min":5.4340000000000002,"15min":4.524},"min":{"1min":0.57099999999999995,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":148.10900000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.753,"5min":7.5119999999999996,"15min":8.6430000000000007},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0569999999999999,"5min":73.769000000000005,"15min":73.769000000000005},"last":0.70999999999999996}]},{"osd":1,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":5.3419999999999996,"15min":6.1440000000000001},"min":{"1min":0.47699999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.089,"5min":88.003,"15min":88.003},"last":0.79600000000000004},{"interface":"front","average":{"1min":6.2370000000000001,"5min":6.0810000000000004,"15min":6.2130000000000001},"min":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":97.704999999999998,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.88800000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":5.8250000000000002,"15min":5.1349999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.032,"5min":94.733999999999995,"15min":94.733999999999995},"last":1.0369999999999999},{"interface":"front","average":{"1min":5.8869999999999996,"5min":3.8650000000000002,"15min":4.0730000000000004},"min":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":91.837999999999994,"5min":91.837999999999994,"15min":91.837999999999994},"last":0.85099999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":3.782,"15min":1.8919999999999999},"min":{"1min":0.57299999999999995,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":15.58,"5min":144.65600000000001,"15min":144.65600000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":2.8439999999999999,"5min":4.173,"15min":4.492},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":31.751000000000001,"5min":76.034999999999997,"15min":76.034999999999997},"last":0.94099999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":6.8899999999999997,"15min":8.5739999999999998},"min":{"1min":0.60499999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0609999999999999,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.625},{"interface":"front","average":{"1min":1.042,"5min":4.0679999999999996,"15min":1.9990000000000001},"min":{"1min":0.61899999999999999,"5min":0.498,"15min":0.498},"max":{"1min":3.944,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.66400000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":5.0599999999999996,"15min":5.9039999999999999},"min":{"1min":0.51200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1799999999999999,"5min":73.561000000000007,"15min":73.561000000000007},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.92100000000000004,"5min":4.226,"15min":4.7050000000000001},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.3029999999999999,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.96499999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":6.4340000000000002,"15min":7.827},"min":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.216,"5min":73.638999999999996,"15min":73.638999999999996},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.9690000000000001,"15min":1.389},"min":{"1min":0.74399999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.1559999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":0.873}]}]},{"osd":6,"up_from":0,"seq":55834574898,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574898,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":51,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1050280,"kb_used_data":92670364,"kb_used_omap":7026,"kb_used_meta":0,"kb_avail":92649100,"statfs":{"total":95948165120,"available":94872678400,"internally_reserved":21774336,"allocated":94894452736,"data_stored":94894452736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7194695,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":4.633,"15min":4.8209999999999997},"min":{"1min":0.47099999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.282,"5min":86.944999999999993,"15min":86.944999999999993},"last":0.91500000000000004},{"interface":"front","average":{"1min":4.0869999999999997,"5min":2.008,"15min":1.1899999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":70.352999999999994,"5min":70.352999999999994,"15min":70.352999999999994},"last":0.66500000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":2.145,"5min":7.5339999999999998,"15min":8.2400000000000002},"min":{"1min":0.60399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":24.489999999999998,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":12.505000000000001,"5min":4.1219999999999999,"15min":1.903},"min":{"1min":0.52900000000000003,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":110.06999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.97599999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.6289999999999996,"5min":5.4859999999999998,"15min":5.5019999999999998},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":97.974000000000004,"5min":99.884,"15min":99.884},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.88500000000000001,"15min":0.90100000000000002},"min":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.254,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.83899999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":1.2649999999999999,"15min":1.3140000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.3280000000000001,"5min":12.878,"15min":12.878},"last":0.82999999999999996},{"interface":"front","average":{"1min":2.6320000000000001,"5min":5.5659999999999998,"15min":5.9560000000000004},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":24.649999999999999,"5min":99.921000000000006,"15min":99.921000000000006},"last":28.521000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.88600000000000001,"15min":0.90800000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.109,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.115},{"interface":"front","average":{"1min":0.90700000000000003,"5min":1.746,"15min":2.1139999999999999},"min":{"1min":0.65200000000000002,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3879999999999999,"5min":30.971,"15min":30.971},"last":0.76000000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":2.2149999999999999,"5min":5.8090000000000002,"15min":5.718},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":24.553999999999998,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.998},{"interface":"front","average":{"1min":0.92600000000000005,"5min":3.4910000000000001,"15min":4.6050000000000004},"min":{"1min":0.66700000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.234,"5min":88.186999999999998,"15min":88.186999999999998},"last":1.0669999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":8.266,"15min":11.507},"min":{"1min":0.63400000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.4510000000000001,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":2.2770000000000001,"5min":8.2899999999999991,"15min":10.256},"min":{"1min":0.65600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":24.603999999999999,"5min":99.843000000000004,"15min":99.843000000000004},"last":0.95699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":53,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1054084,"kb_used_data":92658196,"kb_used_omap":7322,"kb_used_meta":0,"kb_avail":92645296,"statfs":{"total":95948165120,"available":94868783104,"internally_reserved":13209600,"allocated":94881992704,"data_stored":94881992704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7497747,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":8.657,"5min":6.4790000000000001,"15min":5.391},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":64.275999999999996,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":4.7889999999999997,"5min":6.2050000000000001,"15min":7.1630000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":36.207000000000001,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.54700000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.3579999999999997,"5min":11.628,"15min":12.897},"min":{"1min":0.42699999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":60.664000000000001,"5min":159.256,"15min":159.256},"last":0.51900000000000002},{"interface":"front","average":{"1min":19.719999999999999,"5min":6.1059999999999999,"15min":2.5289999999999999},"min":{"1min":0.40200000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":155.44,"5min":174.578,"15min":174.578},"last":0.85299999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":3.1429999999999998,"5min":3.153,"15min":3.2829999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":40.692999999999998,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":7.3019999999999996,"5min":4.5700000000000003,"15min":2.9849999999999999},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":64.381,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.91800000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":8.7050000000000001,"5min":6.2599999999999998,"15min":6.4610000000000003},"min":{"1min":0.65800000000000003,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":72.438999999999993,"5min":72.438999999999993,"15min":72.438999999999993},"last":0.64200000000000002},{"interface":"front","average":{"1min":8.8840000000000003,"5min":12.984,"15min":13.577},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":60.857999999999997,"5min":159.47200000000001,"15min":159.47200000000001},"last":0.94299999999999995}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.226,"5min":11.475,"15min":13.093},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":60.912999999999997,"5min":159.43000000000001,"15min":159.43000000000001},"last":1.024},{"interface":"front","average":{"1min":4.8529999999999998,"5min":3.359,"15min":1.8009999999999999},"min":{"1min":0.752,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":64.513000000000005,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.999}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.8799999999999999,"5min":3.3690000000000002,"15min":1.8109999999999999},"min":{"1min":0.69999999999999996,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":64.590000000000003,"5min":174.63999999999999,"15min":174.63999999999999},"last":1.0669999999999999},{"interface":"front","average":{"1min":1,"5min":3.0529999999999999,"15min":3.5249999999999999},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.304,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.72699999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":3.206,"15min":3.7589999999999999},"min":{"1min":0.69699999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.2110000000000001,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.75800000000000001},{"interface":"front","average":{"1min":5.3739999999999997,"5min":14.025,"15min":16.734999999999999},"min":{"1min":0.72599999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":60.951999999999998,"5min":159.512,"15min":159.512},"last":0.81499999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":40,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":993744,"kb_used_data":92713096,"kb_used_omap":5672,"kb_used_meta":0,"kb_avail":92705636,"statfs":{"total":95948165120,"available":94930571264,"internally_reserved":7639040,"allocated":94938210304,"data_stored":94938210304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5808437,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.5739999999999998,"5min":2.5009999999999999,"15min":1.2609999999999999},"min":{"1min":0.439,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":66.277000000000001,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":4.617,"5min":1.4670000000000001,"15min":0.96499999999999997},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":88.299999999999997,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.48599999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.2610000000000001,"5min":3.7229999999999999,"15min":1.7729999999999999},"min":{"1min":0.46500000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":80.233999999999995,"5min":143.071,"15min":143.071},"last":0.69799999999999995},{"interface":"front","average":{"1min":4.6989999999999998,"5min":3.1000000000000001,"15min":3.27},"min":{"1min":0.505,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":88.186000000000007,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.55600000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":10.462,"15min":10.422000000000001},"min":{"1min":0.40999999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.94899999999999995,"5min":183.53,"15min":183.53},"last":1.0580000000000001},{"interface":"front","average":{"1min":5.7729999999999997,"5min":3.8149999999999999,"15min":1.7829999999999999},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":66.335999999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.67400000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":5.9859999999999998,"5min":5.9489999999999998,"15min":4.4160000000000004},"min":{"1min":0.624,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":66.491,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":4.9290000000000003,"5min":5.8259999999999996,"15min":4.4809999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":88.459999999999994,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.85799999999999998}]},{"osd":5,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":2.577,"15min":1.643},"min":{"1min":0.65800000000000003,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2829999999999999,"5min":143.262,"15min":143.262},"last":0.81999999999999995},{"interface":"front","average":{"1min":6.0339999999999998,"5min":2.1040000000000001,"15min":1.4750000000000001},"min":{"1min":0.71899999999999997,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":66.456999999999994,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.89500000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":4.9889999999999999,"5min":1.8879999999999999,"15min":1.3169999999999999},"min":{"1min":0.70699999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":88.542000000000002,"5min":88.542000000000002,"15min":88.542000000000002},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.028,"5min":2.7109999999999999,"15min":1.7150000000000001},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.585,"5min":143.215,"15min":143.215},"last":1.1699999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:31:09 2023","interfaces":[{"interface":"back","average":{"1min":7.6230000000000002,"5min":3.5489999999999999,"15min":3.222},"min":{"1min":0.81100000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":88.623999999999995,"5min":88.623999999999995,"15min":88.623999999999995},"last":1.087},{"interface":"front","average":{"1min":1.048,"5min":3.5630000000000002,"15min":3.0390000000000001},"min":{"1min":0.79900000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.3859999999999999,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.95399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":52,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1011240,"kb_used_data":92690668,"kb_used_omap":7040,"kb_used_meta":0,"kb_avail":92688140,"statfs":{"total":95948165120,"available":94912655360,"internally_reserved":2588672,"allocated":94915244032,"data_stored":94915244032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7209220,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,2],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3609999999999998,"5min":4.9240000000000004,"15min":4.2359999999999998},"min":{"1min":0.39900000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":100.533,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":4.6509999999999998,"5min":3.6459999999999999,"15min":4.0970000000000004},"min":{"1min":0.373,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":85.751000000000005,"5min":85.751000000000005,"15min":85.751000000000005},"last":0.55900000000000005}]},{"osd":2,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":2.6600000000000001,"5min":1.2270000000000001,"15min":1.0529999999999999},"min":{"1min":0.32500000000000001,"5min":0.245,"15min":0.245},"max":{"1min":43.759999999999998,"5min":43.759999999999998,"15min":43.759999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":3.1869999999999998,"15min":3.7770000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82899999999999996,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.84899999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":17.001000000000001,"5min":15.438000000000001,"15min":18.370000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":245.637,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":5.6310000000000002,"5min":5.9980000000000002,"15min":6.8289999999999997},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":85.814999999999998,"5min":85.814999999999998,"15min":85.814999999999998},"last":0.71099999999999997}]},{"osd":4,"last update":"Sun Jan 15 14:31:12 2023","interfaces":[{"interface":"back","average":{"1min":63.935000000000002,"5min":16.309999999999999,"15min":8.5879999999999992},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1181.3389999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":57.106000000000002,"5min":14.65,"15min":7.2859999999999996},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1181.8720000000001,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":1.1519999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.98099999999999998,"15min":1.0149999999999999},"min":{"1min":0.58099999999999996,"5min":0.433,"15min":0.433},"max":{"1min":1.244,"5min":1.7390000000000001,"15min":1.7390000000000001},"last":1.081},{"interface":"front","average":{"1min":0.92200000000000004,"5min":3.036,"15min":3.9700000000000002},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.1339999999999999,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.05}]},{"osd":6,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":5.069,"5min":4.2060000000000004,"15min":4.7359999999999998},"min":{"1min":0.745,"5min":0.63,"15min":0.63},"max":{"1min":85.884,"5min":85.884,"15min":85.884},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.056,"15min":1.101},"min":{"1min":0.69599999999999995,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.298,"5min":2.4540000000000002,"15min":2.4540000000000002},"last":1.1919999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:31:10 2023","interfaces":[{"interface":"back","average":{"1min":8.3879999999999999,"5min":7.1310000000000002,"15min":7.0549999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":157.03700000000001,"5min":157.03700000000001,"15min":157.03700000000001},"last":1.234},{"interface":"front","average":{"1min":5.0919999999999996,"5min":4.0730000000000004,"15min":4.524},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":85.852999999999994,"5min":85.852999999999994,"15min":85.852999999999994},"last":1.1040000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":818372,"kb_used_data":92881096,"kb_used_omap":4479,"kb_used_meta":0,"kb_avail":92881008,"statfs":{"total":95948165120,"available":95110152192,"internally_reserved":90112,"allocated":95110242304,"data_stored":95110242304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4586940,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":7.2519999999999998,"5min":2.6970000000000001,"15min":1.3140000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":69.816999999999993,"5min":69.872,"15min":69.872},"last":0.54400000000000004},{"interface":"front","average":{"1min":5.6130000000000004,"5min":2.6789999999999998,"15min":1.379},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":98.203999999999994,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.66900000000000004}]},{"osd":2,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.7039999999999997,"5min":3.407,"15min":1.5980000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":98.286000000000001,"5min":98.286000000000001,"15min":98.286000000000001},"last":1.155},{"interface":"front","average":{"1min":3.0270000000000001,"5min":2.6819999999999999,"15min":1.3660000000000001},"min":{"1min":0.38400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":48.893000000000001,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.57799999999999996}]},{"osd":3,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":12.154999999999999,"5min":13.413,"15min":15.657},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":94.712999999999994,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":10.484,"5min":5.6399999999999997,"15min":2.4140000000000001},"min":{"1min":0.42399999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":98.352999999999994,"5min":174.09399999999999,"15min":174.09399999999999},"last":1.0069999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.5570000000000004,"5min":3.2309999999999999,"15min":1.734},"min":{"1min":0.63800000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":61.536000000000001,"5min":71.483000000000004,"15min":71.483000000000004},"last":0.871},{"interface":"front","average":{"1min":9.2989999999999995,"5min":3.9950000000000001,"15min":2.0150000000000001},"min":{"1min":0.67400000000000004,"5min":0.439,"15min":0.439},"max":{"1min":70.179000000000002,"5min":70.179000000000002,"15min":70.179000000000002},"last":0.91200000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":5.9009999999999998,"5min":8.218,"15min":9.2710000000000008},"min":{"1min":0.68500000000000005,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":98.411000000000001,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.95399999999999996,"5min":1.7709999999999999,"15min":1.2829999999999999},"min":{"1min":0.63400000000000001,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":2.2959999999999998,"5min":71.563999999999993,"15min":71.563999999999993},"last":1.04}]},{"osd":6,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":7.7030000000000003,"5min":3.2040000000000002,"15min":1.8480000000000001},"min":{"1min":0.66700000000000004,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":70.259,"5min":70.259,"15min":70.259},"last":1.1839999999999999},{"interface":"front","average":{"1min":5.984,"5min":3.056,"15min":1.774},"min":{"1min":0.749,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":98.484999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":1.0840000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:31:07 2023","interfaces":[{"interface":"back","average":{"1min":4.633,"5min":12.002000000000001,"15min":14.448},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":73.864999999999995,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":7.7199999999999998,"5min":3.21,"15min":1.8500000000000001},"min":{"1min":0.70399999999999996,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":70.322000000000003,"5min":70.322000000000003,"15min":70.322000000000003},"last":1.1259999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:32:07.335 INFO:tasks.thrashosds.thrasher:Setting 3.1 to [1, 7, 2] 2023-01-15T14:32:07.336 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.1', '1', '7', '2'] 2023-01-15T14:32:07.336 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.1 1 7 2 2023-01-15T14:32:07.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:32:07.538 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:07.333+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:07.541 INFO:tasks.rados.rados.0.smithi090.stdout:954: writing smithi09025264-14 from 732871 to 1448006 tid 1 2023-01-15T14:32:07.541 INFO:tasks.rados.rados.0.smithi090.stdout:954: writing smithi09025264-14 from 1873673 to 2517780 tid 2 2023-01-15T14:32:07.541 INFO:tasks.rados.rados.0.smithi090.stdout:954: writing smithi09025264-14 from 3244434 to 3932871 tid 3 2023-01-15T14:32:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:955: delete oid 33 current snap is 93 2023-01-15T14:32:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:952: finishing rollback tid 1 to smithi09025264-34 2023-01-15T14:32:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:952: finishing rollback tid 2 to smithi09025264-34 2023-01-15T14:32:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 142 (ObjNum 265 snap 65 seq_num 265) dirty exists 2023-01-15T14:32:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:953: expect (ObjNum 70 snap 8 seq_num 70) 2023-01-15T14:32:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:950: finishing write tid 1 to smithi09025264-9 2023-01-15T14:32:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:950: finishing write tid 2 to smithi09025264-9 2023-01-15T14:32:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:950: finishing write tid 3 to smithi09025264-9 2023-01-15T14:32:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:950: finishing write tid 4 to smithi09025264-9 2023-01-15T14:32:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:950: finishing write tid 5 to smithi09025264-9 2023-01-15T14:32:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:950: oid 9 updating version 0 to 162 2023-01-15T14:32:07.543 INFO:tasks.rados.rados.0.smithi090.stdout:950: oid 9 version 162 is already newer than 161 2023-01-15T14:32:07.744 INFO:tasks.rados.rados.0.smithi090.stdout:950: oid 9 updating version 162 to 164 2023-01-15T14:32:07.744 INFO:tasks.rados.rados.0.smithi090.stdout:950: oid 9 version 164 is already newer than 163 2023-01-15T14:32:07.744 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 164 (ObjNum 352 snap 93 seq_num 352) dirty exists 2023-01-15T14:32:07.744 INFO:tasks.rados.rados.0.smithi090.stdout:950: left oid 9 (ObjNum 352 snap 93 seq_num 352) 2023-01-15T14:32:07.744 INFO:tasks.rados.rados.0.smithi090.stdout:954: finishing write tid 1 to smithi09025264-14 2023-01-15T14:32:07.744 INFO:tasks.rados.rados.0.smithi090.stdout:954: finishing write tid 2 to smithi09025264-14 2023-01-15T14:32:07.744 INFO:tasks.rados.rados.0.smithi090.stdout:950: done (4 left) 2023-01-15T14:32:07.744 INFO:tasks.rados.rados.0.smithi090.stdout:952: done (3 left) 2023-01-15T14:32:07.745 INFO:tasks.rados.rados.0.smithi090.stdout:953: done (2 left) 2023-01-15T14:32:07.745 INFO:tasks.rados.rados.0.smithi090.stdout:955: done (1 left) 2023-01-15T14:32:07.745 INFO:tasks.rados.rados.0.smithi090.stdout:956: rollback oid 21 current snap is 93 2023-01-15T14:32:07.745 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 21 to 91 2023-01-15T14:32:07.745 INFO:tasks.rados.rados.0.smithi090.stdout:957: snap_remove snap 88 2023-01-15T14:32:07.745 INFO:tasks.rados.rados.0.smithi090.stdout:957: done (2 left) 2023-01-15T14:32:07.745 INFO:tasks.rados.rados.0.smithi090.stdout:958: rollback oid 25 current snap is 93 2023-01-15T14:32:07.745 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 25 to 91 2023-01-15T14:32:07.746 INFO:tasks.rados.rados.0.smithi090.stdout:954: finishing write tid 3 to smithi09025264-14 2023-01-15T14:32:07.746 INFO:tasks.rados.rados.0.smithi090.stdout:954: finishing write tid 4 to smithi09025264-14 2023-01-15T14:32:07.746 INFO:tasks.rados.rados.0.smithi090.stdout:954: finishing write tid 5 to smithi09025264-14 2023-01-15T14:32:07.746 INFO:tasks.rados.rados.0.smithi090.stdout:954: oid 14 updating version 0 to 249 2023-01-15T14:32:07.946 INFO:tasks.rados.rados.0.smithi090.stdout:954: oid 14 version 249 is already newer than 246 2023-01-15T14:32:07.946 INFO:tasks.rados.rados.0.smithi090.stdout:954: oid 14 version 249 is already newer than 248 2023-01-15T14:32:07.946 INFO:tasks.rados.rados.0.smithi090.stdout:954: oid 14 version 249 is already newer than 247 2023-01-15T14:32:07.946 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 249 (ObjNum 353 snap 93 seq_num 353) dirty exists 2023-01-15T14:32:07.946 INFO:tasks.rados.rados.0.smithi090.stdout:954: left oid 14 (ObjNum 353 snap 93 seq_num 353) 2023-01-15T14:32:07.946 INFO:tasks.rados.rados.0.smithi090.stdout:954: done (2 left) 2023-01-15T14:32:07.947 INFO:tasks.rados.rados.0.smithi090.stdout:959: write oid 35 current snap is 93 2023-01-15T14:32:07.947 INFO:tasks.rados.rados.0.smithi090.stdout:959: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2023-01-15T14:32:07.947 INFO:tasks.rados.rados.0.smithi090.stdout:959: writing smithi09025264-35 from 766485 to 1232118 tid 1 2023-01-15T14:32:07.947 INFO:tasks.rados.rados.0.smithi090.stdout:959: writing smithi09025264-35 from 1990637 to 2638413 tid 2 2023-01-15T14:32:07.947 INFO:tasks.rados.rados.0.smithi090.stdout:959: writing smithi09025264-35 from 3299280 to 3966485 tid 3 2023-01-15T14:32:07.947 INFO:tasks.rados.rados.0.smithi090.stdout:960: snap_create 2023-01-15T14:32:07.947 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:32:07.947 INFO:tasks.rados.rados.0.smithi090.stdout:958: finishing rollback tid 0 to smithi09025264-25 2023-01-15T14:32:07.947 INFO:tasks.rados.rados.0.smithi090.stdout:958: finishing rollback tid 1 to smithi09025264-25 2023-01-15T14:32:07.947 INFO:tasks.rados.rados.0.smithi090.stdout:958: finishing rollback tid 2 to smithi09025264-25 2023-01-15T14:32:07.947 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 259 (ObjNum 335 snap 88 seq_num 335) dirty exists 2023-01-15T14:32:07.948 INFO:tasks.rados.rados.0.smithi090.stdout:958: done (2 left) 2023-01-15T14:32:07.948 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:07.948 INFO:tasks.rados.rados.0.smithi090.stdout:956: finishing rollback tid 0 to smithi09025264-21 2023-01-15T14:32:07.948 INFO:tasks.rados.rados.0.smithi090.stdout:959: finishing write tid 1 to smithi09025264-35 2023-01-15T14:32:07.948 INFO:tasks.rados.rados.0.smithi090.stdout:959: finishing write tid 2 to smithi09025264-35 2023-01-15T14:32:07.948 INFO:tasks.rados.rados.0.smithi090.stdout:956: finishing rollback tid 1 to smithi09025264-21 2023-01-15T14:32:07.948 INFO:tasks.rados.rados.0.smithi090.stdout:959: finishing write tid 3 to smithi09025264-35 2023-01-15T14:32:07.948 INFO:tasks.rados.rados.0.smithi090.stdout:956: finishing rollback tid 2 to smithi09025264-21 2023-01-15T14:32:07.948 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 253 (ObjNum 339 snap 89 seq_num 339) dirty exists 2023-01-15T14:32:07.948 INFO:tasks.rados.rados.0.smithi090.stdout:956: done (1 left) 2023-01-15T14:32:07.949 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:07.949 INFO:tasks.rados.rados.0.smithi090.stdout:959: finishing write tid 4 to smithi09025264-35 2023-01-15T14:32:07.949 INFO:tasks.rados.rados.0.smithi090.stdout:959: finishing write tid 5 to smithi09025264-35 2023-01-15T14:32:07.949 INFO:tasks.rados.rados.0.smithi090.stdout:959: oid 35 updating version 0 to 261 2023-01-15T14:32:07.949 INFO:tasks.rados.rados.0.smithi090.stdout:959: oid 35 updating version 261 to 262 2023-01-15T14:32:07.949 INFO:tasks.rados.rados.0.smithi090.stdout:959: oid 35 version 262 is already newer than 260 2023-01-15T14:32:07.949 INFO:tasks.rados.rados.0.smithi090.stdout:959: oid 35 version 262 is already newer than 259 2023-01-15T14:32:07.949 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 262 (ObjNum 355 snap 93 seq_num 355) dirty exists 2023-01-15T14:32:07.949 INFO:tasks.rados.rados.0.smithi090.stdout:959: left oid 35 (ObjNum 355 snap 93 seq_num 355) 2023-01-15T14:32:07.949 INFO:tasks.rados.rados.0.smithi090.stdout:959: done (0 left) 2023-01-15T14:32:07.950 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:06.550+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:07.954 INFO:tasks.rados.rados.0.smithi090.stdout:960: done (0 left) 2023-01-15T14:32:07.954 INFO:tasks.rados.rados.0.smithi090.stdout:961: delete oid 8 current snap is 94 2023-01-15T14:32:07.954 INFO:tasks.rados.rados.0.smithi090.stdout:961: done (0 left) 2023-01-15T14:32:07.954 INFO:tasks.rados.rados.0.smithi090.stdout:962: delete oid 12 current snap is 94 2023-01-15T14:32:07.954 INFO:tasks.rados.rados.0.smithi090.stdout:962: done (0 left) 2023-01-15T14:32:07.954 INFO:tasks.rados.rados.0.smithi090.stdout:963: delete oid 7 current snap is 94 2023-01-15T14:32:07.954 INFO:tasks.rados.rados.0.smithi090.stdout:963: done (0 left) 2023-01-15T14:32:07.954 INFO:tasks.rados.rados.0.smithi090.stdout:964: snap_create 2023-01-15T14:32:07.955 INFO:tasks.rados.rados.0.smithi090.stdout:964: done (0 left) 2023-01-15T14:32:07.955 INFO:tasks.rados.rados.0.smithi090.stdout:965: copy_from oid 22 from oid 47 current snap is 95 2023-01-15T14:32:07.955 INFO:tasks.rados.rados.0.smithi090.stdout:966: read oid 27 snap 94 2023-01-15T14:32:07.955 INFO:tasks.rados.rados.0.smithi090.stdout:966: expect deleted 2023-01-15T14:32:07.955 INFO:tasks.rados.rados.0.smithi090.stdout:967: write (excl) oid 44 current snap is 95 2023-01-15T14:32:07.955 INFO:tasks.rados.rados.0.smithi090.stdout:967: seq_num 359 ranges {433713=482982,1424565=609148} 2023-01-15T14:32:07.955 INFO:tasks.rados.rados.0.smithi090.stdout:967: writing smithi09025264-44 from 433713 to 916695 tid 1 2023-01-15T14:32:07.955 INFO:tasks.rados.rados.0.smithi090.stdout:967: writing smithi09025264-44 from 1424565 to 2033713 tid 2 2023-01-15T14:32:07.955 INFO:tasks.rados.rados.0.smithi090.stdout:968: write oid 10 current snap is 95 2023-01-15T14:32:07.955 INFO:tasks.rados.rados.0.smithi090.stdout:968: seq_num 360 ranges {450520=558231,1483047=515125} 2023-01-15T14:32:07.956 INFO:tasks.rados.rados.0.smithi090.stdout:968: writing smithi09025264-10 from 450520 to 1008751 tid 1 2023-01-15T14:32:07.958 INFO:tasks.rados.rados.0.smithi090.stdout:968: writing smithi09025264-10 from 1483047 to 1998172 tid 2 2023-01-15T14:32:07.958 INFO:tasks.rados.rados.0.smithi090.stdout:969: snap_remove snap 91 2023-01-15T14:32:07.959 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:07.953+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:08.058 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:08.054+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:08.152 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:08.152 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:32:08.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.153 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2556 3.10 3:0a998d04:::smithi09025264-10:head [write 450520~558231 in=558231b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2023-01-15T14:32:08.153 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:07.947647+0000", 2023-01-15T14:32:08.153 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.138593409, 2023-01-15T14:32:08.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.138643246, 2023-01-15T14:32:08.153 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:08.154 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:32:08.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:08.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:08.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:08.154 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2556 2023-01-15T14:32:08.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.155 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:08.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:08.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.947647+0000", 2023-01-15T14:32:08.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:08.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.947647+0000", 2023-01-15T14:32:08.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:08.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.947645+0000", 2023-01-15T14:32:08.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:08.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:08.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.947882+0000", 2023-01-15T14:32:08.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:32:08.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:08.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.947906+0000", 2023-01-15T14:32:08.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3904999999999999e-05 2023-01-15T14:32:08.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:08.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.947919+0000", 2023-01-15T14:32:08.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2432000000000001e-05 2023-01-15T14:32:08.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:08.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.948059+0000", 2023-01-15T14:32:08.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014005199999999999 2023-01-15T14:32:08.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:32:08.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.948072+0000", 2023-01-15T14:32:08.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3040999999999999e-05 2023-01-15T14:32:08.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:08.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.049892+0000", 2023-01-15T14:32:08.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101820139 2023-01-15T14:32:08.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:08.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.049971+0000", 2023-01-15T14:32:08.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8987999999999997e-05 2023-01-15T14:32:08.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:08.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.050128+0000", 2023-01-15T14:32:08.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015741099999999999 2023-01-15T14:32:08.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.050553+0000", 2023-01-15T14:32:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042437799999999999 2023-01-15T14:32:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.050570+0000", 2023-01-15T14:32:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7152000000000001e-05 2023-01-15T14:32:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.052691+0000", 2023-01-15T14:32:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021210579999999999 2023-01-15T14:32:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:08.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.052741+0000", 2023-01-15T14:32:08.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.9892999999999998e-05 2023-01-15T14:32:08.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:08.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.053575+0000", 2023-01-15T14:32:08.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083396100000000003 2023-01-15T14:32:08.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.171 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:08.171 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.171 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2557 3.10 3:0a998d04:::smithi09025264-10:head [write 1483047~515125 in=515125b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2023-01-15T14:32:08.172 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:07.953250+0000", 2023-01-15T14:32:08.172 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.132990372, 2023-01-15T14:32:08.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13320605099999999, 2023-01-15T14:32:08.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:08.172 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:32:08.173 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:08.173 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:08.173 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:08.173 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2557 2023-01-15T14:32:08.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.173 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:08.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:08.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953250+0000", 2023-01-15T14:32:08.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:08.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953250+0000", 2023-01-15T14:32:08.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:08.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953248+0000", 2023-01-15T14:32:08.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:32:08.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:08.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953538+0000", 2023-01-15T14:32:08.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029001800000000001 2023-01-15T14:32:08.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:08.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953575+0000", 2023-01-15T14:32:08.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7951000000000003e-05 2023-01-15T14:32:08.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:08.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953595+0000", 2023-01-15T14:32:08.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9502000000000001e-05 2023-01-15T14:32:08.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:08.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.050615+0000", 2023-01-15T14:32:08.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097020304000000002 2023-01-15T14:32:08.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:08.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.050729+0000", 2023-01-15T14:32:08.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113686 2023-01-15T14:32:08.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:08.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.050882+0000", 2023-01-15T14:32:08.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015315899999999999 2023-01-15T14:32:08.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:08.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.051202+0000", 2023-01-15T14:32:08.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032006800000000001 2023-01-15T14:32:08.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:08.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.052696+0000", 2023-01-15T14:32:08.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001493383 2023-01-15T14:32:08.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:08.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.052995+0000", 2023-01-15T14:32:08.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000299468 2023-01-15T14:32:08.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:08.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.053030+0000", 2023-01-15T14:32:08.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4822e-05 2023-01-15T14:32:08.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:08.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.053661+0000", 2023-01-15T14:32:08.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063067400000000001 2023-01-15T14:32:08.187 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.187 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:08.187 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.188 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2558 3.10 3:0a998d04:::smithi09025264-10:head [setxattr _header (57) in=64b,truncate 2050520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e241)", 2023-01-15T14:32:08.188 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:07.953625+0000", 2023-01-15T14:32:08.188 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.13261597999999999, 2023-01-15T14:32:08.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.132909262, 2023-01-15T14:32:08.189 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:08.189 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:32:08.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:08.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:08.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2558 2023-01-15T14:32:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:08.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:08.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953625+0000", 2023-01-15T14:32:08.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:08.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953625+0000", 2023-01-15T14:32:08.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:08.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953623+0000", 2023-01-15T14:32:08.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:08.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:08.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953632+0000", 2023-01-15T14:32:08.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1499999999999999e-06 2023-01-15T14:32:08.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:08.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953651+0000", 2023-01-15T14:32:08.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9000000000000001e-05 2023-01-15T14:32:08.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:08.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953666+0000", 2023-01-15T14:32:08.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5682999999999999e-05 2023-01-15T14:32:08.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:08.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.051250+0000", 2023-01-15T14:32:08.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097583965999999994 2023-01-15T14:32:08.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:08.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.051349+0000", 2023-01-15T14:32:08.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8608000000000002e-05 2023-01-15T14:32:08.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:08.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.051489+0000", 2023-01-15T14:32:08.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014006700000000001 2023-01-15T14:32:08.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.051698+0000", 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000209342 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.052696+0000", 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00099816000000000011 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:08.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.052996+0000", 2023-01-15T14:32:08.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029967500000000001 2023-01-15T14:32:08.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:08.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.053038+0000", 2023-01-15T14:32:08.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1557000000000002e-05 2023-01-15T14:32:08.364 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.364 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:08.364 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2559 3.10 3:0a998d04:::smithi09025264-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e241)", 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:07.953686+0000", 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.13255502299999999, 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13291639799999999, 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "delayed", 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2559 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.365 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:08.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:08.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953686+0000", 2023-01-15T14:32:08.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:08.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953686+0000", 2023-01-15T14:32:08.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:08.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953684+0000", 2023-01-15T14:32:08.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:08.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:08.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953690+0000", 2023-01-15T14:32:08.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4759999999999997e-06 2023-01-15T14:32:08.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:08.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953708+0000", 2023-01-15T14:32:08.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7638999999999998e-05 2023-01-15T14:32:08.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:08.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:07.953721+0000", 2023-01-15T14:32:08.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3604000000000001e-05 2023-01-15T14:32:08.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:08.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.051744+0000", 2023-01-15T14:32:08.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098022460000000006 2023-01-15T14:32:08.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:08.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.051826+0000", 2023-01-15T14:32:08.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2179e-05 2023-01-15T14:32:08.370 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.370 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:08.370 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.370 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "replica_scrub(pg: 3.3a,from:0'0,to:0'0,epoch:241/229,start:3:5c000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2023-01-15T14:32:08.370 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:08.028062+0000", 2023-01-15T14:32:08.370 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.058178976, 2023-01-15T14:32:08.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.058587976999999999, 2023-01-15T14:32:08.370 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:08.371 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "no flag points reached", 2023-01-15T14:32:08.371 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:08.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:08.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.028062+0000", 2023-01-15T14:32:08.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:08.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.028062+0000", 2023-01-15T14:32:08.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:08.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.028061+0000", 2023-01-15T14:32:08.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:08.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:08.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.028068+0000", 2023-01-15T14:32:08.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6919999999999996e-06 2023-01-15T14:32:08.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.028084+0000", 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6331e-05 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.4530.0:2558 3.10 e241/207)", 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:08.053584+0000", 2023-01-15T14:32:08.373 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.032656487999999997, 2023-01-15T14:32:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.033094788999999999, 2023-01-15T14:32:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "no flag points reached", 2023-01-15T14:32:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.053584+0000", 2023-01-15T14:32:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:08.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.053584+0000", 2023-01-15T14:32:08.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:08.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:08.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.053584+0000", 2023-01-15T14:32:08.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:08.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:08.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.053587+0000", 2023-01-15T14:32:08.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5080000000000001e-06 2023-01-15T14:32:08.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:08.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:08.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:08.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:08.053593+0000", 2023-01-15T14:32:08.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6690000000000003e-06 2023-01-15T14:32:08.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.377 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:08.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:08.377 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:08.377 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 6 2023-01-15T14:32:08.377 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:08.378 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:08.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:32:08.380 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:08.380 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:08.380 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:08.380 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:08.380 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:08.380 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:08.380 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:08.381 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:32:08.381 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:32:08.381 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:32:08.562 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:32:08.562 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:32:08.562 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:32:08.562 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:32:08.563 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:32:08.565 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:08.558+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:08.666 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:08.661+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:08.776 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:08.777 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:08.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:32:08.779 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:08.770+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:08.878 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:08.873+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:08.973 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:08.974 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:08.974 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:32:08.974 INFO:tasks.rados.rados.0.smithi090.stdout:969: done (4 left) 2023-01-15T14:32:08.975 INFO:tasks.rados.rados.0.smithi090.stdout:970: snap_remove snap 72 2023-01-15T14:32:08.975 INFO:teuthology.orchestra.run.smithi090.stderr:set 3.1 pg_upmap mapping to [1,7,2] 2023-01-15T14:32:08.975 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:32:08.978 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:08.974+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:09.079 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:09.074+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:09.185 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:09.185 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:09.185 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:32:09.186 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:09.178+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:09.286 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:09.281+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:09.501 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:09.501 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:09.501 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:32:09.502 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:09.494+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:09.601 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:09.594+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:09.685 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:09.685 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:09.686 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:32:09.702 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:09.697+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:09.899 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:09.899 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:09.899 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:32:09.900 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:09.893+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:09.941 INFO:tasks.rados.rados.0.smithi090.stdout:970: done (4 left) 2023-01-15T14:32:10.089 INFO:tasks.rados.rados.0.smithi090.stdout:971: snap_create 2023-01-15T14:32:10.089 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:32:10.089 INFO:tasks.rados.rados.0.smithi090.stdout:968: finishing write tid 1 to smithi09025264-10 2023-01-15T14:32:10.089 INFO:tasks.rados.rados.0.smithi090.stdout:968: finishing write tid 2 to smithi09025264-10 2023-01-15T14:32:10.089 INFO:tasks.rados.rados.0.smithi090.stdout:968: finishing write tid 3 to smithi09025264-10 2023-01-15T14:32:10.089 INFO:tasks.rados.rados.0.smithi090.stdout:965: finishing copy_from to smithi09025264-22 2023-01-15T14:32:10.089 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 193 (ObjNum 183 snap 40 seq_num 183) dirty exists 2023-01-15T14:32:10.090 INFO:tasks.rados.rados.0.smithi090.stdout:967: finishing write tid 1 to smithi09025264-44 2023-01-15T14:32:10.090 INFO:tasks.rados.rados.0.smithi090.stdout:965: finishing copy_from racing read to smithi09025264-22 2023-01-15T14:32:10.090 INFO:tasks.rados.rados.0.smithi090.stdout:968: finishing write tid 4 to smithi09025264-10 2023-01-15T14:32:10.090 INFO:tasks.rados.rados.0.smithi090.stdout:968: oid 10 updating version 0 to 312 2023-01-15T14:32:10.090 INFO:tasks.rados.rados.0.smithi090.stdout:968: oid 10 version 312 is already newer than 310 2023-01-15T14:32:10.091 INFO:tasks.rados.rados.0.smithi090.stdout:968: oid 10 version 312 is already newer than 311 2023-01-15T14:32:10.091 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 312 (ObjNum 360 snap 95 seq_num 360) dirty exists 2023-01-15T14:32:10.091 INFO:tasks.rados.rados.0.smithi090.stdout:968: left oid 10 (ObjNum 360 snap 95 seq_num 360) 2023-01-15T14:32:10.091 INFO:tasks.rados.rados.0.smithi090.stdout:967: finishing write tid 2 to smithi09025264-44 2023-01-15T14:32:10.091 INFO:tasks.rados.rados.0.smithi090.stdout:967: finishing write tid 3 to smithi09025264-44 2023-01-15T14:32:10.091 INFO:tasks.rados.rados.0.smithi090.stdout:967: finishing write tid 4 to smithi09025264-44 2023-01-15T14:32:10.091 INFO:tasks.rados.rados.0.smithi090.stdout:967: oid 44 updating version 0 to 261 2023-01-15T14:32:10.091 INFO:tasks.rados.rados.0.smithi090.stdout:967: oid 44 updating version 261 to 263 2023-01-15T14:32:10.091 INFO:tasks.rados.rados.0.smithi090.stdout:967: oid 44 version 263 is already newer than 262 2023-01-15T14:32:10.091 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 263 (ObjNum 359 snap 95 seq_num 359) dirty exists 2023-01-15T14:32:10.092 INFO:tasks.rados.rados.0.smithi090.stdout:967: left oid 44 (ObjNum 359 snap 95 seq_num 359) 2023-01-15T14:32:10.092 INFO:tasks.rados.rados.0.smithi090.stdout:965: done (3 left) 2023-01-15T14:32:10.092 INFO:tasks.rados.rados.0.smithi090.stdout:966: done (2 left) 2023-01-15T14:32:10.092 INFO:tasks.rados.rados.0.smithi090.stdout:967: done (1 left) 2023-01-15T14:32:10.092 INFO:tasks.rados.rados.0.smithi090.stdout:968: done (0 left) 2023-01-15T14:32:10.096 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:10.097 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:10.097 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:32:10.097 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:10.089+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:10.197 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:10.190+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:10.277 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:10.277 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:10.277 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:32:10.298 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:10.294+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:10.464 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:10.464 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:10.465 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:32:10.465 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:10.457+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:10.563 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:32:10.649 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:10.557+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:10.652 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:10.653 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:10.655 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:32:10.702 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:10.698+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:10.836 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:10.836 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:10.836 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:10.837 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:10.837 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:10.837 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:32:10.838 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:10.829+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:10.851 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:10.879 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:11.018 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:32:11.019 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:11.013+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:11.020 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:11.021 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:11.021 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:32:11.021 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:32:11.021 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:32:11.021 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:32:11.021 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:32:11.022 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:32:11.022 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:32:11.022 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:32:11.022 INFO:tasks.rados.rados.0.smithi090.stdout:971: done (0 left) 2023-01-15T14:32:11.022 INFO:tasks.rados.rados.0.smithi090.stdout:972: write (excl) oid 15 current snap is 96 2023-01-15T14:32:11.022 INFO:tasks.rados.rados.0.smithi090.stdout:972: seq_num 361 ranges {467327=633480,1825176=242151} 2023-01-15T14:32:11.025 INFO:tasks.rados.rados.0.smithi090.stdout:972: writing smithi09025264-15 from 467327 to 1100807 tid 1 2023-01-15T14:32:11.025 INFO:tasks.rados.rados.0.smithi090.stdout:972: writing smithi09025264-15 from 1825176 to 2067327 tid 2 2023-01-15T14:32:11.025 INFO:tasks.rados.rados.0.smithi090.stdout:973: read oid 42 snap -1 2023-01-15T14:32:11.025 INFO:tasks.rados.rados.0.smithi090.stdout:973: expect deleted 2023-01-15T14:32:11.025 INFO:tasks.rados.rados.0.smithi090.stdout:974: write oid 41 current snap is 96 2023-01-15T14:32:11.025 INFO:tasks.rados.rados.0.smithi090.stdout:974: seq_num 362 ranges {484134=708729,1883658=200476} 2023-01-15T14:32:11.025 INFO:tasks.rados.rados.0.smithi090.stdout:974: writing smithi09025264-41 from 484134 to 1192863 tid 1 2023-01-15T14:32:11.025 INFO:tasks.rados.rados.0.smithi090.stdout:974: writing smithi09025264-41 from 1883658 to 2084134 tid 2 2023-01-15T14:32:11.025 INFO:tasks.rados.rados.0.smithi090.stdout:975: read oid 43 snap -1 2023-01-15T14:32:11.025 INFO:tasks.rados.rados.0.smithi090.stdout:975: expect (ObjNum 341 snap 91 seq_num 341) 2023-01-15T14:32:11.026 INFO:tasks.rados.rados.0.smithi090.stdout:976: write oid 49 current snap is 96 2023-01-15T14:32:11.026 INFO:tasks.rados.rados.0.smithi090.stdout:976: seq_num 363 ranges {500941=783978,1942140=158801} 2023-01-15T14:32:11.026 INFO:tasks.rados.rados.0.smithi090.stdout:976: writing smithi09025264-49 from 500941 to 1284919 tid 1 2023-01-15T14:32:11.026 INFO:tasks.rados.rados.0.smithi090.stdout:976: writing smithi09025264-49 from 1942140 to 2100941 tid 2 2023-01-15T14:32:11.026 INFO:tasks.rados.rados.0.smithi090.stdout:977: read oid 37 snap -1 2023-01-15T14:32:11.026 INFO:tasks.rados.rados.0.smithi090.stdout:977: expect (ObjNum 332 snap 88 seq_num 332) 2023-01-15T14:32:11.026 INFO:tasks.rados.rados.0.smithi090.stdout:978: rollback oid 27 current snap is 96 2023-01-15T14:32:11.026 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 27 to 94 2023-01-15T14:32:11.026 INFO:tasks.rados.rados.0.smithi090.stdout:979: copy_from oid 2 from oid 50 current snap is 96 2023-01-15T14:32:11.026 INFO:tasks.rados.rados.0.smithi090.stdout:980: delete oid 47 current snap is 96 2023-01-15T14:32:11.026 INFO:tasks.rados.rados.0.smithi090.stdout:977: expect (ObjNum 332 snap 88 seq_num 332) 2023-01-15T14:32:11.120 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:11.113+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:11.210 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:11.211 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:32:11.211 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.211 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2573 3.39 3:9f7cca9d:::smithi09025264-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e244)", 2023-01-15T14:32:11.211 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:32:10.985496+0000", 2023-01-15T14:32:11.212 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.15438663, 2023-01-15T14:32:11.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.15443653199999999, 2023-01-15T14:32:11.212 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:11.212 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:32:11.212 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:11.212 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:11.212 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:11.212 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2573 2023-01-15T14:32:11.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.212 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:11.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:11.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985496+0000", 2023-01-15T14:32:11.213 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:11.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985496+0000", 2023-01-15T14:32:11.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:11.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985495+0000", 2023-01-15T14:32:11.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:11.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:11.215 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985499+0000", 2023-01-15T14:32:11.215 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.087e-06 2023-01-15T14:32:11.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:11.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985509+0000", 2023-01-15T14:32:11.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4169999999999998e-06 2023-01-15T14:32:11.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:11.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985515+0000", 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0560000000000001e-06 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.076478+0000", 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090962941000000005 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.076706+0000", 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022790899999999999 2023-01-15T14:32:11.217 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.218 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:11.218 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.218 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2584 3.d 3:b2a32959:::smithi09025264-27:head [rollback b9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2023-01-15T14:32:11.218 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:32:10.997415+0000", 2023-01-15T14:32:11.218 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.14246729499999999, 2023-01-15T14:32:11.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.14263614399999999, 2023-01-15T14:32:11.218 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:11.218 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:11.218 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:11.219 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:11.219 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:11.219 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2584 2023-01-15T14:32:11.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.219 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:11.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:11.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997415+0000", 2023-01-15T14:32:11.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:11.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997415+0000", 2023-01-15T14:32:11.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:11.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997415+0000", 2023-01-15T14:32:11.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:11.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:11.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997418+0000", 2023-01-15T14:32:11.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6729999999999998e-06 2023-01-15T14:32:11.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:11.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997427+0000", 2023-01-15T14:32:11.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9630000000000004e-06 2023-01-15T14:32:11.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:11.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997434+0000", 2023-01-15T14:32:11.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5699999999999998e-06 2023-01-15T14:32:11.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:11.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.097620+0000", 2023-01-15T14:32:11.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10018611500000001 2023-01-15T14:32:11.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:11.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.097775+0000", 2023-01-15T14:32:11.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015472700000000001 2023-01-15T14:32:11.224 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.224 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:11.224 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.224 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2587 3.13 3:ca393a9e:::smithi09025264-47:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e244)", 2023-01-15T14:32:11.224 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:32:10.997454+0000", 2023-01-15T14:32:11.224 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.14242871600000001, 2023-01-15T14:32:11.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.142699086, 2023-01-15T14:32:11.224 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:11.225 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:32:11.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:11.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:11.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:11.225 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2587 2023-01-15T14:32:11.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.225 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:11.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:11.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997454+0000", 2023-01-15T14:32:11.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:11.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997454+0000", 2023-01-15T14:32:11.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:11.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997453+0000", 2023-01-15T14:32:11.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:11.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:11.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997457+0000", 2023-01-15T14:32:11.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8569999999999997e-06 2023-01-15T14:32:11.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:11.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997467+0000", 2023-01-15T14:32:11.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8630000000000003e-06 2023-01-15T14:32:11.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:11.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.997473+0000", 2023-01-15T14:32:11.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3759999999999996e-06 2023-01-15T14:32:11.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:11.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.006705+0000", 2023-01-15T14:32:11.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0092319280000000004 2023-01-15T14:32:11.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:11.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.006902+0000", 2023-01-15T14:32:11.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019652999999999999 2023-01-15T14:32:11.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:11.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.007317+0000", 2023-01-15T14:32:11.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000415055 2023-01-15T14:32:11.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:11.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.008523+0000", 2023-01-15T14:32:11.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012058629999999999 2023-01-15T14:32:11.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:11.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.008548+0000", 2023-01-15T14:32:11.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4992999999999999e-05 2023-01-15T14:32:11.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:11.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.008682+0000", 2023-01-15T14:32:11.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013409399999999999 2023-01-15T14:32:11.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:11.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.008756+0000", 2023-01-15T14:32:11.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3910000000000002e-05 2023-01-15T14:32:11.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:11.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.109233+0000", 2023-01-15T14:32:11.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100477605 2023-01-15T14:32:11.232 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.232 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:11.232 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.232 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2563 3.21 3:86bba67f:::smithi09025264-15:head [write 467327~633480 in=633480b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2023-01-15T14:32:11.232 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:32:10.966132+0000", 2023-01-15T14:32:11.232 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.17375075000000001, 2023-01-15T14:32:11.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17417545700000001, 2023-01-15T14:32:11.233 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:11.233 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:32:11.233 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:11.233 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:11.233 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:11.233 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2563 2023-01-15T14:32:11.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.233 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:11.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:11.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.966132+0000", 2023-01-15T14:32:11.234 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.234 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.234 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:11.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.966132+0000", 2023-01-15T14:32:11.234 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.235 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.235 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:11.235 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.966130+0000", 2023-01-15T14:32:11.235 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:11.235 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.235 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.235 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:11.235 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.966888+0000", 2023-01-15T14:32:11.235 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075808099999999997 2023-01-15T14:32:11.235 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.236 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.236 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:11.236 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.966923+0000", 2023-01-15T14:32:11.236 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5615999999999999e-05 2023-01-15T14:32:11.236 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.236 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.236 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:11.236 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.966940+0000", 2023-01-15T14:32:11.236 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6160000000000001e-05 2023-01-15T14:32:11.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:11.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.967030+0000", 2023-01-15T14:32:11.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0110000000000003e-05 2023-01-15T14:32:11.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:11.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.967175+0000", 2023-01-15T14:32:11.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014515900000000001 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.967406+0000", 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000231642 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.967815+0000", 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040854500000000001 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:11.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.967830+0000", 2023-01-15T14:32:11.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4904e-05 2023-01-15T14:32:11.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:11.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.968190+0000", 2023-01-15T14:32:11.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036026100000000001 2023-01-15T14:32:11.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:11.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.968272+0000", 2023-01-15T14:32:11.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1392999999999998e-05 2023-01-15T14:32:11.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:11.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.070685+0000", 2023-01-15T14:32:11.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102413853 2023-01-15T14:32:11.240 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.240 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:11.240 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.241 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.241 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2564 3.21 3:86bba67f:::smithi09025264-15:head [write 1825176~242151 in=242151b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2023-01-15T14:32:11.241 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:32:10.973438+0000", 2023-01-15T14:32:11.241 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.16644424199999999, 2023-01-15T14:32:11.241 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.16701690599999999, 2023-01-15T14:32:11.241 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:11.241 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:32:11.241 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:11.242 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:11.242 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:11.242 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2564 2023-01-15T14:32:11.242 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.242 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:11.242 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.242 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:11.242 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973438+0000", 2023-01-15T14:32:11.242 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.242 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.242 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.243 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:11.243 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973438+0000", 2023-01-15T14:32:11.243 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.243 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.243 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:11.243 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973436+0000", 2023-01-15T14:32:11.243 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:11.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.243 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.243 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:11.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973639+0000", 2023-01-15T14:32:11.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020275600000000001 2023-01-15T14:32:11.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.244 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:11.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973673+0000", 2023-01-15T14:32:11.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4079999999999999e-05 2023-01-15T14:32:11.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.244 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:11.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973687+0000", 2023-01-15T14:32:11.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4375999999999999e-05 2023-01-15T14:32:11.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.245 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:11.245 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.070822+0000", 2023-01-15T14:32:11.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097134688999999996 2023-01-15T14:32:11.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.245 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:11.245 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.071013+0000", 2023-01-15T14:32:11.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019053200000000001 2023-01-15T14:32:11.246 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.246 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:11.246 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.246 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2565 3.21 3:86bba67f:::smithi09025264-15:head [setxattr _header (57) in=64b,truncate 2067327] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2023-01-15T14:32:11.246 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:32:10.973707+0000", 2023-01-15T14:32:11.246 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.16617559100000001, 2023-01-15T14:32:11.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.16684917299999999, 2023-01-15T14:32:11.428 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:11.428 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:32:11.428 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:11.428 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:11.429 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:11.429 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2565 2023-01-15T14:32:11.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.431 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:11.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:11.431 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973707+0000", 2023-01-15T14:32:11.431 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:11.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973707+0000", 2023-01-15T14:32:11.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:11.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973706+0000", 2023-01-15T14:32:11.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:11.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.433 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:11.433 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973712+0000", 2023-01-15T14:32:11.657 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1739999999999997e-06 2023-01-15T14:32:11.657 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:11.658 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973723+0000", 2023-01-15T14:32:11.658 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0253e-05 2023-01-15T14:32:11.658 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:11.658 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973729+0000", 2023-01-15T14:32:11.659 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3230000000000003e-06 2023-01-15T14:32:11.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.071117+0000", 2023-01-15T14:32:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0973884 2023-01-15T14:32:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.071401+0000", 2023-01-15T14:32:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028399799999999999 2023-01-15T14:32:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2566 3.21 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e244)", 2023-01-15T14:32:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:32:10.973738+0000", 2023-01-15T14:32:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.166144818, 2023-01-15T14:32:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.16694753500000001, 2023-01-15T14:32:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:32:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2566 2023-01-15T14:32:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973738+0000", 2023-01-15T14:32:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973738+0000", 2023-01-15T14:32:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973737+0000", 2023-01-15T14:32:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973741+0000", 2023-01-15T14:32:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.16e-06 2023-01-15T14:32:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973751+0000", 2023-01-15T14:32:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4329999999999993e-06 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.973757+0000", 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.767e-06 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.071495+0000", 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097738740000000005 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:11.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.071617+0000", 2023-01-15T14:32:11.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121784 2023-01-15T14:32:11.670 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.670 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:11.670 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.670 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2570 3.39 3:9f7cca9d:::smithi09025264-41:head [write 484134~708729 in=708729b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2023-01-15T14:32:11.670 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:32:10.980281+0000", 2023-01-15T14:32:11.670 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.15960169599999999, 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.160503284, 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2570 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:11.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.980281+0000", 2023-01-15T14:32:11.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:11.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.980281+0000", 2023-01-15T14:32:11.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:11.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.980279+0000", 2023-01-15T14:32:11.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.980927+0000", 2023-01-15T14:32:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00064790400000000004 2023-01-15T14:32:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.980960+0000", 2023-01-15T14:32:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3485999999999998e-05 2023-01-15T14:32:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.980974+0000", 2023-01-15T14:32:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3779e-05 2023-01-15T14:32:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.071487+0000", 2023-01-15T14:32:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090513108999999994 2023-01-15T14:32:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.071783+0000", 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029632399999999999 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.072403+0000", 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061931400000000004 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.073544+0000", 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011409580000000001 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.073564+0000", 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0710999999999999e-05 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.073978+0000", 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041376299999999999 2023-01-15T14:32:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.677 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.677 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:11.677 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.076353+0000", 2023-01-15T14:32:11.677 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023744730000000002 2023-01-15T14:32:11.677 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.677 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:11.677 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.677 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.677 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.677 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2571 3.39 3:9f7cca9d:::smithi09025264-41:head [write 1883658~200476 in=200476b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2023-01-15T14:32:11.677 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:32:10.985221+0000", 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.154661878, 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.15570245999999999, 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2571 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985221+0000", 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985221+0000", 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985219+0000", 2023-01-15T14:32:11.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:11.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:11.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985398+0000", 2023-01-15T14:32:11.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000179333 2023-01-15T14:32:11.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:11.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985431+0000", 2023-01-15T14:32:11.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3246e-05 2023-01-15T14:32:11.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:11.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985446+0000", 2023-01-15T14:32:11.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4523e-05 2023-01-15T14:32:11.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.073736+0000", 2023-01-15T14:32:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.088290076999999995 2023-01-15T14:32:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.073952+0000", 2023-01-15T14:32:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021655599999999999 2023-01-15T14:32:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.074254+0000", 2023-01-15T14:32:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000301351 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.074808+0000", 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055445500000000003 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.074825+0000", 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6779999999999999e-05 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.075079+0000", 2023-01-15T14:32:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025444900000000002 2023-01-15T14:32:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.076392+0000", 2023-01-15T14:32:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0013121020000000001 2023-01-15T14:32:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:11.869 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:11.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.869 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.869 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2572 3.39 3:9f7cca9d:::smithi09025264-41:head [setxattr _header (57) in=64b,truncate 2084134] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e244)", 2023-01-15T14:32:11.869 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:32:10.985465+0000", 2023-01-15T14:32:11.869 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.15441775599999999, 2023-01-15T14:32:11.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.15559885200000001, 2023-01-15T14:32:11.870 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:11.870 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:32:11.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:11.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:11.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:11.870 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2572 2023-01-15T14:32:11.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.871 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:11.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:11.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985465+0000", 2023-01-15T14:32:11.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:11.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985465+0000", 2023-01-15T14:32:11.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:11.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:11.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985464+0000", 2023-01-15T14:32:11.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:11.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:11.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:11.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:11.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985471+0000", 2023-01-15T14:32:11.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6900000000000003e-06 2023-01-15T14:32:11.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:12.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:12.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:12.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985481+0000", 2023-01-15T14:32:12.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0146000000000001e-05 2023-01-15T14:32:12.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:12.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:12.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:12.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:10.985487+0000", 2023-01-15T14:32:12.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3470000000000004e-06 2023-01-15T14:32:12.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:12.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:12.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:12.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.075017+0000", 2023-01-15T14:32:12.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089529421999999997 2023-01-15T14:32:12.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:12.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:12.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:12.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.075304+0000", 2023-01-15T14:32:12.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028779700000000001 2023-01-15T14:32:12.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:12.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:12.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:12.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.075789+0000", 2023-01-15T14:32:12.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048446399999999999 2023-01-15T14:32:12.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:12.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:12.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:12.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.076224+0000", 2023-01-15T14:32:12.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000435637 2023-01-15T14:32:12.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:12.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:12.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:12.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.076246+0000", 2023-01-15T14:32:12.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1263999999999998e-05 2023-01-15T14:32:12.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:12.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:12.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:12.304 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.076322+0000", 2023-01-15T14:32:12.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6689999999999999e-05 2023-01-15T14:32:12.305 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:12.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:12.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:12.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:11.076422+0000", 2023-01-15T14:32:12.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9559e-05 2023-01-15T14:32:12.459 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:12.459 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:12.459 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:12.459 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:12.459 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2023-01-15T14:32:12.459 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 10 2023-01-15T14:32:12.459 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:12.460 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:32:12.462 INFO:tasks.rados.rados.0.smithi090.stdout:975: expect (ObjNum 341 snap 91 seq_num 341) 2023-01-15T14:32:12.463 INFO:tasks.rados.rados.0.smithi090.stdout:973: done (8 left) 2023-01-15T14:32:12.463 INFO:tasks.rados.rados.0.smithi090.stdout:975: done (7 left) 2023-01-15T14:32:12.463 INFO:tasks.rados.rados.0.smithi090.stdout:977: done (6 left) 2023-01-15T14:32:12.463 INFO:tasks.rados.rados.0.smithi090.stdout:980: done (5 left) 2023-01-15T14:32:12.463 INFO:tasks.rados.rados.0.smithi090.stdout:981: snap_remove snap 81 2023-01-15T14:32:12.463 INFO:tasks.rados.rados.0.smithi090.stdout:981: done (5 left) 2023-01-15T14:32:12.463 INFO:tasks.rados.rados.0.smithi090.stdout:982: read oid 35 snap -1 2023-01-15T14:32:12.463 INFO:tasks.rados.rados.0.smithi090.stdout:982: expect (ObjNum 355 snap 93 seq_num 355) 2023-01-15T14:32:12.463 INFO:tasks.rados.rados.0.smithi090.stdout:983: delete oid 16 current snap is 96 2023-01-15T14:32:12.464 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:32:12.659 INFO:tasks.rados.rados.0.smithi090.stdout:972: finishing write tid 1 to smithi09025264-15 2023-01-15T14:32:12.659 INFO:tasks.rados.rados.0.smithi090.stdout:976: finishing write tid 1 to smithi09025264-49 2023-01-15T14:32:12.661 INFO:tasks.rados.rados.0.smithi090.stdout:976: finishing write tid 2 to smithi09025264-49 2023-01-15T14:32:12.661 INFO:tasks.rados.rados.0.smithi090.stdout:976: finishing write tid 3 to smithi09025264-49 2023-01-15T14:32:12.661 INFO:tasks.rados.rados.0.smithi090.stdout:976: finishing write tid 4 to smithi09025264-49 2023-01-15T14:32:12.662 INFO:tasks.rados.rados.0.smithi090.stdout:976: oid 49 updating version 0 to 261 2023-01-15T14:32:12.662 INFO:tasks.rados.rados.0.smithi090.stdout:976: oid 49 updating version 261 to 263 2023-01-15T14:32:12.662 INFO:tasks.rados.rados.0.smithi090.stdout:976: oid 49 version 263 is already newer than 262 2023-01-15T14:32:12.662 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 263 (ObjNum 363 snap 96 seq_num 363) dirty exists 2023-01-15T14:32:12.662 INFO:tasks.rados.rados.0.smithi090.stdout:976: left oid 49 (ObjNum 363 snap 96 seq_num 363) 2023-01-15T14:32:12.662 INFO:tasks.rados.rados.0.smithi090.stdout:974: finishing write tid 1 to smithi09025264-41 2023-01-15T14:32:12.663 INFO:tasks.rados.rados.0.smithi090.stdout:978: finishing rollback tid 1 to smithi09025264-27 2023-01-15T14:32:12.663 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 164 (ObjNum 593 snap 0 seq_num 3202430232) dirty dne 2023-01-15T14:32:12.663 INFO:tasks.rados.rados.0.smithi090.stdout:974: finishing write tid 2 to smithi09025264-41 2023-01-15T14:32:12.663 INFO:tasks.rados.rados.0.smithi090.stdout:974: finishing write tid 3 to smithi09025264-41 2023-01-15T14:32:12.663 INFO:tasks.rados.rados.0.smithi090.stdout:972: finishing write tid 2 to smithi09025264-15 2023-01-15T14:32:12.663 INFO:tasks.rados.rados.0.smithi090.stdout:974: finishing write tid 4 to smithi09025264-41 2023-01-15T14:32:12.664 INFO:tasks.rados.rados.0.smithi090.stdout:974: oid 41 updating version 0 to 254 2023-01-15T14:32:12.664 INFO:tasks.rados.rados.0.smithi090.stdout:974: oid 41 updating version 254 to 255 2023-01-15T14:32:12.664 INFO:tasks.rados.rados.0.smithi090.stdout:974: oid 41 updating version 255 to 256 2023-01-15T14:32:12.664 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 256 (ObjNum 362 snap 96 seq_num 362) dirty exists 2023-01-15T14:32:12.664 INFO:tasks.rados.rados.0.smithi090.stdout:974: left oid 41 (ObjNum 362 snap 96 seq_num 362) 2023-01-15T14:32:12.665 INFO:tasks.rados.rados.0.smithi090.stdout:979: finishing copy_from to smithi09025264-2 2023-01-15T14:32:12.665 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 187 (ObjNum 328 snap 87 seq_num 328) dirty exists 2023-01-15T14:32:12.665 INFO:tasks.rados.rados.0.smithi090.stdout:979: finishing copy_from racing read to smithi09025264-2 2023-01-15T14:32:12.665 INFO:tasks.rados.rados.0.smithi090.stdout:972: finishing write tid 3 to smithi09025264-15 2023-01-15T14:32:12.665 INFO:tasks.rados.rados.0.smithi090.stdout:972: finishing write tid 4 to smithi09025264-15 2023-01-15T14:32:12.665 INFO:tasks.rados.rados.0.smithi090.stdout:972: oid 15 updating version 0 to 152 2023-01-15T14:32:12.666 INFO:tasks.rados.rados.0.smithi090.stdout:972: oid 15 version 152 is already newer than 151 2023-01-15T14:32:12.666 INFO:tasks.rados.rados.0.smithi090.stdout:972: oid 15 version 152 is already newer than 150 2023-01-15T14:32:12.666 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 152 (ObjNum 361 snap 96 seq_num 361) dirty exists 2023-01-15T14:32:12.666 INFO:tasks.rados.rados.0.smithi090.stdout:972: left oid 15 (ObjNum 361 snap 96 seq_num 361) 2023-01-15T14:32:12.668 INFO:tasks.rados.rados.0.smithi090.stdout:972: done (6 left) 2023-01-15T14:32:12.668 INFO:tasks.rados.rados.0.smithi090.stdout:974: done (5 left) 2023-01-15T14:32:12.668 INFO:tasks.rados.rados.0.smithi090.stdout:976: done (4 left) 2023-01-15T14:32:12.668 INFO:tasks.rados.rados.0.smithi090.stdout:978: done (3 left) 2023-01-15T14:32:12.669 INFO:tasks.rados.rados.0.smithi090.stdout:979: done (2 left) 2023-01-15T14:32:12.669 INFO:tasks.rados.rados.0.smithi090.stdout:983: done (1 left) 2023-01-15T14:32:12.669 INFO:tasks.rados.rados.0.smithi090.stdout:984: snap_remove snap 94 2023-01-15T14:32:12.670 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:12.457+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:12.672 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:12.673 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:32:12.673 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:32:12.673 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:32:12.673 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:12.674 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:32:12.674 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:12.666+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:12.771 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:12.765+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:13.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:32:13.069 INFO:tasks.rados.rados.0.smithi090.stdout:984: done (1 left) 2023-01-15T14:32:13.069 INFO:tasks.rados.rados.0.smithi090.stdout:985: delete oid 11 current snap is 96 2023-01-15T14:32:13.069 INFO:tasks.rados.rados.0.smithi090.stdout:982: expect (ObjNum 355 snap 93 seq_num 355) 2023-01-15T14:32:13.071 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:13.272 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:32:13.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:32:13.272 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:32:13.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:75 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 226.43129658199999, 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92335155400000002, 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 75 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.273 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356896+0000", 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356903+0000", 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4619999999999999e-06 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356917+0000", 2023-01-15T14:32:13.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4195999999999999e-05 2023-01-15T14:32:13.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:13.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356926+0000", 2023-01-15T14:32:13.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:32:13.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:13.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457743+0000", 2023-01-15T14:32:13.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100817668 2023-01-15T14:32:13.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:13.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457841+0000", 2023-01-15T14:32:13.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7079999999999999e-05 2023-01-15T14:32:13.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778474+0000", 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32063301799999999 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778627+0000", 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015389599999999999 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778815+0000", 2023-01-15T14:32:13.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018726600000000001 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779033+0000", 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021819999999999999 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779051+0000", 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7640000000000001e-05 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3590999999999995e-05 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779163+0000", 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:13.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.081002+0000", 2023-01-15T14:32:13.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30183937799999999 2023-01-15T14:32:13.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:13.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280043+0000", 2023-01-15T14:32:13.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19904084699999999 2023-01-15T14:32:13.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:13.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280138+0000", 2023-01-15T14:32:13.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4915999999999997e-05 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280248+0000", 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110233 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:76 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:13.481 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 226.43125497400001, 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92420393899999997, 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 76 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:13.483 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:32:13.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:13.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:13.483 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:32:13.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:13.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356937+0000", 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356948+0000", 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.094e-05 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356959+0000", 2023-01-15T14:32:13.484 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1058e-05 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356968+0000", 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5339999999999999e-06 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459524+0000", 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102556147 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.486 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:13.486 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459662+0000", 2023-01-15T14:32:13.486 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000137756 2023-01-15T14:32:13.486 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.486 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.486 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:13.486 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978621+0000", 2023-01-15T14:32:13.486 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51895887600000001 2023-01-15T14:32:13.486 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.486 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.486 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978870+0000", 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024928999999999999 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280761+0000", 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30189113400000001 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281023+0000", 2023-01-15T14:32:13.487 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026172800000000003 2023-01-15T14:32:13.488 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:13.488 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281142+0000", 2023-01-15T14:32:13.488 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000119296 2023-01-15T14:32:13.488 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:13.488 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:13.488 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:13.488 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.488 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1076 3.f 3:f87c0183:::smithi09025264-50:head [setxattr _header (57) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:13.489 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:32:13.489 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 150.029546219, 2023-01-15T14:32:13.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.88473693200000003, 2023-01-15T14:32:13.489 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:13.680 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:13.681 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:13.681 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:13.681 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:13.681 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1076 2023-01-15T14:32:13.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.681 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:13.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:13.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:32:13.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:13.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758646+0000", 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.682 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758653+0000", 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8029999999999997e-06 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758670+0000", 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6158999999999999e-05 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758677+0000", 2023-01-15T14:32:13.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9840000000000004e-06 2023-01-15T14:32:13.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:13.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848049+0000", 2023-01-15T14:32:13.684 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089372163000000004 2023-01-15T14:32:13.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:13.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848056+0000", 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0360000000000001e-06 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951562+0000", 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10350670100000001 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951726+0000", 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000163311 2023-01-15T14:32:13.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:13.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951958+0000", 2023-01-15T14:32:13.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000232525 2023-01-15T14:32:13.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:13.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952168+0000", 2023-01-15T14:32:13.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020949799999999999 2023-01-15T14:32:13.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952186+0000", 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7714e-05 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952254+0000", 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7998999999999997e-05 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:13.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952278+0000", 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4909000000000002e-05 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065836+0000", 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.113557081 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643196+0000", 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.57736012999999997 2023-01-15T14:32:13.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.689 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:13.689 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643278+0000", 2023-01-15T14:32:13.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2671999999999997e-05 2023-01-15T14:32:13.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.689 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:13.689 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643384+0000", 2023-01-15T14:32:13.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010551400000000001 2023-01-15T14:32:13.689 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:13.689 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1077 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 150.0295074, 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.98557123499999999, 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:13.690 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1077 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:32:13.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758685+0000", 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758689+0000", 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2760000000000002e-06 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.693 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:13.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758704+0000", 2023-01-15T14:32:13.693 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4187e-05 2023-01-15T14:32:13.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:13.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758710+0000", 2023-01-15T14:32:13.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4330000000000001e-06 2023-01-15T14:32:13.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:13.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848086+0000", 2023-01-15T14:32:13.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089376466000000002 2023-01-15T14:32:13.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848093+0000", 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4269999999999996e-06 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952346+0000", 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104252758 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952427+0000", 2023-01-15T14:32:13.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1786000000000006e-05 2023-01-15T14:32:13.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:13.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743602+0000", 2023-01-15T14:32:13.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.79117428099999998 2023-01-15T14:32:13.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:13.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743892+0000", 2023-01-15T14:32:13.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029067200000000001 2023-01-15T14:32:13.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.744257+0000", 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000364619 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1087 3.f 3:fd983206:::smithi09025264-7:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:13.896 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 150.02947291800001, 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0859964360000001, 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1087 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:13.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:13.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:13.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:13.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:13.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:13.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:13.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:13.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:13.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:13.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758724+0000", 2023-01-15T14:32:13.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7740000000000004e-06 2023-01-15T14:32:13.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:13.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758738+0000", 2023-01-15T14:32:13.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.33e-05 2023-01-15T14:32:13.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:13.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758744+0000", 2023-01-15T14:32:13.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2859999999999996e-06 2023-01-15T14:32:13.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:13.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:13.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848121+0000", 2023-01-15T14:32:13.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089377097000000003 2023-01-15T14:32:13.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:13.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:14.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848127+0000", 2023-01-15T14:32:14.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3500000000000002e-06 2023-01-15T14:32:14.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:14.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952454+0000", 2023-01-15T14:32:14.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104326167 2023-01-15T14:32:14.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:14.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952520+0000", 2023-01-15T14:32:14.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6166000000000003e-05 2023-01-15T14:32:14.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:14.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.063003+0000", 2023-01-15T14:32:14.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.110482916 2023-01-15T14:32:14.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:14.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065265+0000", 2023-01-15T14:32:14.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022626370000000001 2023-01-15T14:32:14.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:14.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065291+0000", 2023-01-15T14:32:14.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6095e-05 2023-01-15T14:32:14.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:14.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.066153+0000", 2023-01-15T14:32:14.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086127100000000004 2023-01-15T14:32:14.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:14.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166533+0000", 2023-01-15T14:32:14.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100380177 2023-01-15T14:32:14.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:14.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166646+0000", 2023-01-15T14:32:14.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112717 2023-01-15T14:32:14.057 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844498+0000", 2023-01-15T14:32:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67785237399999998 2023-01-15T14:32:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844605+0000", 2023-01-15T14:32:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106691 2023-01-15T14:32:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:14.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844717+0000", 2023-01-15T14:32:14.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011203499999999999 2023-01-15T14:32:14.443 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:14.443 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:14.444 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:14.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.444 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1088 3.f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:14.444 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:32:14.444 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 150.029417252, 2023-01-15T14:32:14.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.086312696, 2023-01-15T14:32:14.444 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:14.444 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:14.444 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:14.445 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:14.445 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1088 2023-01-15T14:32:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:32:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:32:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758775+0000", 2023-01-15T14:32:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758786+0000", 2023-01-15T14:32:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.076e-05 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758809+0000", 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3706999999999999e-05 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758821+0000", 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1976e-05 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:14.450 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878534+0000", 2023-01-15T14:32:14.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.119712966 2023-01-15T14:32:14.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:14.450 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878540+0000", 2023-01-15T14:32:14.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0680000000000002e-06 2023-01-15T14:32:14.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:14.601 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952693+0000", 2023-01-15T14:32:14.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.074153067000000003 2023-01-15T14:32:14.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.601 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.601 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:14.601 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952735+0000", 2023-01-15T14:32:14.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1792999999999998e-05 2023-01-15T14:32:14.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.602 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:14.602 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065518+0000", 2023-01-15T14:32:14.602 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.11278334499999999 2023-01-15T14:32:14.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.602 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:14.602 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065687+0000", 2023-01-15T14:32:14.602 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016807700000000001 2023-01-15T14:32:14.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844786+0000", 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.77909980300000004 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844944+0000", 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157424 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:14.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.845089+0000", 2023-01-15T14:32:14.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144814 2023-01-15T14:32:14.604 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:14.604 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:14.604 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:14.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.604 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:16 3.13 3:ca393a9e:::smithi09025264-47:head [assert-version v193,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:14.604 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:32:14.604 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.48243982699999, 2023-01-15T14:32:14.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8970339479999998, 2023-01-15T14:32:14.605 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:14.605 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:14.605 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:14.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:14.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:32:14.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:14.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:14.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:32:14.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:14.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:14.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305751+0000", 2023-01-15T14:32:14.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:14.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:14.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305763+0000", 2023-01-15T14:32:14.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2309e-05 2023-01-15T14:32:14.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:14.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305788+0000", 2023-01-15T14:32:14.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4388000000000001e-05 2023-01-15T14:32:14.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:14.608 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305799+0000", 2023-01-15T14:32:14.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1304000000000001e-05 2023-01-15T14:32:14.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:14.608 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196425+0000", 2023-01-15T14:32:14.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8906259830000001 2023-01-15T14:32:14.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:14.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196673+0000", 2023-01-15T14:32:14.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024747499999999998 2023-01-15T14:32:14.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:14.610 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.202787+0000", 2023-01-15T14:32:14.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0061146509999999996 2023-01-15T14:32:14.610 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:14.610 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:14.610 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:14.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.611 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1480 3.13 3:cbfd2d71:::smithi09025264-37:head [write 581575~705262,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:14.611 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:32:14.611 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.44917209899999, 2023-01-15T14:32:14.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.375425221, 2023-01-15T14:32:14.611 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:14.611 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:14.786 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:14.787 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:14.787 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:14.787 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1480 2023-01-15T14:32:14.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.787 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:14.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.787 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:14.787 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:32:14.787 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:14.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.788 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.788 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:14.788 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:32:14.788 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:14.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.788 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339019+0000", 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339659+0000", 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000639742 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:14.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339690+0000", 2023-01-15T14:32:14.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1594000000000003e-05 2023-01-15T14:32:14.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:14.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339703+0000", 2023-01-15T14:32:14.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2828e-05 2023-01-15T14:32:14.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:14.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.510837+0000", 2023-01-15T14:32:14.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1711338759999999 2023-01-15T14:32:14.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:14.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511048+0000", 2023-01-15T14:32:14.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021057899999999999 2023-01-15T14:32:14.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:14.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511325+0000", 2023-01-15T14:32:14.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027690400000000002 2023-01-15T14:32:14.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:14.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512092+0000", 2023-01-15T14:32:14.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00076720800000000002 2023-01-15T14:32:14.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:14.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512113+0000", 2023-01-15T14:32:14.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0917e-05 2023-01-15T14:32:14.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:14.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512439+0000", 2023-01-15T14:32:14.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032636299999999998 2023-01-15T14:32:14.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:14.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512871+0000", 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000432087 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.514165+0000", 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001293726 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714137+0000", 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19997229499999999 2023-01-15T14:32:14.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:14.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714233+0000", 2023-01-15T14:32:14.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6062000000000003e-05 2023-01-15T14:32:14.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.987 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.987 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:14.987 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714446+0000", 2023-01-15T14:32:14.987 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212952 2023-01-15T14:32:14.987 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:14.987 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:14.988 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:14.990 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.990 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.990 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1481 3.13 3:cbfd2d71:::smithi09025264-37:head [write 1733272~707409] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:14.990 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:32:14.990 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.442322816, 2023-01-15T14:32:14.991 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.572522502, 2023-01-15T14:32:14.991 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:14.991 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:14.991 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:14.991 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:14.991 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:14.991 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1481 2023-01-15T14:32:14.991 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.992 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:14.992 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.992 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:14.992 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:32:14.992 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:14.992 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.992 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.992 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:14.992 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:32:14.992 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:14.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.994 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:14.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345868+0000", 2023-01-15T14:32:14.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:14.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:14.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346316+0000", 2023-01-15T14:32:14.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044813399999999999 2023-01-15T14:32:14.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:14.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346348+0000", 2023-01-15T14:32:14.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:32:14.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:14.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346362+0000", 2023-01-15T14:32:14.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.375e-05 2023-01-15T14:32:14.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:14.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512276+0000", 2023-01-15T14:32:14.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1659136829999999 2023-01-15T14:32:14.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:14.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:14.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:14.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512427+0000", 2023-01-15T14:32:14.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015088699999999999 2023-01-15T14:32:15.000 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.000 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.000 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:15.000 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714565+0000", 2023-01-15T14:32:15.000 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20213841599999999 2023-01-15T14:32:15.000 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.000 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.000 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:15.000 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714732+0000", 2023-01-15T14:32:15.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016676700000000001 2023-01-15T14:32:15.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:15.003 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714964+0000", 2023-01-15T14:32:15.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000231678 2023-01-15T14:32:15.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:15.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715499+0000", 2023-01-15T14:32:15.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053513300000000003 2023-01-15T14:32:15.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:15.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715523+0000", 2023-01-15T14:32:15.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4266000000000001e-05 2023-01-15T14:32:15.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:15.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715867+0000", 2023-01-15T14:32:15.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034419399999999998 2023-01-15T14:32:15.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:15.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.816932+0000", 2023-01-15T14:32:15.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10106498999999999 2023-01-15T14:32:15.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906374+0000", 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089441442999999995 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918050+0000", 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.011676677999999999 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918218+0000", 2023-01-15T14:32:15.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016802000000000001 2023-01-15T14:32:15.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:15.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918393+0000", 2023-01-15T14:32:15.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017447 2023-01-15T14:32:15.150 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:15.151 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:15.151 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:15.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.151 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1482 3.13 3:cbfd2d71:::smithi09025264-37:head [write 3195938~437497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:15.152 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:32:15.152 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.435886958, 2023-01-15T14:32:15.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5664780289999998, 2023-01-15T14:32:15.154 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1482 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:15.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:32:15.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:15.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:15.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352275+0000", 2023-01-15T14:32:15.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999695 2023-01-15T14:32:15.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:15.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352644+0000", 2023-01-15T14:32:15.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036870300000000001 2023-01-15T14:32:15.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:15.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352676+0000", 2023-01-15T14:32:15.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1726000000000001e-05 2023-01-15T14:32:15.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:15.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352689+0000", 2023-01-15T14:32:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3617e-05 2023-01-15T14:32:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512602+0000", 2023-01-15T14:32:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1599126690000001 2023-01-15T14:32:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512839+0000", 2023-01-15T14:32:15.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023676900000000001 2023-01-15T14:32:15.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:15.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715649+0000", 2023-01-15T14:32:15.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20281044600000001 2023-01-15T14:32:15.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.311 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:15.311 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715777+0000", 2023-01-15T14:32:15.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012738800000000001 2023-01-15T14:32:15.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:15.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715995+0000", 2023-01-15T14:32:15.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000217858 2023-01-15T14:32:15.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:15.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716694+0000", 2023-01-15T14:32:15.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00069956700000000001 2023-01-15T14:32:15.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:15.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716709+0000", 2023-01-15T14:32:15.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4362000000000001e-05 2023-01-15T14:32:15.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:15.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716936+0000", 2023-01-15T14:32:15.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022744200000000001 2023-01-15T14:32:15.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:15.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.818876+0000", 2023-01-15T14:32:15.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101939635 2023-01-15T14:32:15.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:15.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906390+0000", 2023-01-15T14:32:15.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087513987000000001 2023-01-15T14:32:15.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:15.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918577+0000", 2023-01-15T14:32:15.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012187066 2023-01-15T14:32:15.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:15.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918697+0000", 2023-01-15T14:32:15.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120326 2023-01-15T14:32:15.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:15.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918784+0000", 2023-01-15T14:32:15.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7200999999999994e-05 2023-01-15T14:32:15.475 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:15.475 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:15.476 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:15.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.476 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1483 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3781575] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:15.477 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:32:15.477 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.43547880600001, 2023-01-15T14:32:15.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5663522429999999, 2023-01-15T14:32:15.477 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:15.477 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:15.478 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:15.478 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:15.481 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:15.481 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1483 2023-01-15T14:32:15.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.481 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:15.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:15.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:32:15.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:15.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:15.483 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:32:15.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:15.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.484 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:15.484 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352713+0000", 2023-01-15T14:32:15.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:15.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:15.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352720+0000", 2023-01-15T14:32:15.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7959999999999998e-06 2023-01-15T14:32:15.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:15.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352735+0000", 2023-01-15T14:32:15.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.455e-05 2023-01-15T14:32:15.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:15.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352741+0000", 2023-01-15T14:32:15.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8360000000000003e-06 2023-01-15T14:32:15.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:15.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512901+0000", 2023-01-15T14:32:15.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1601594980000001 2023-01-15T14:32:15.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:15.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.513067+0000", 2023-01-15T14:32:15.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016626 2023-01-15T14:32:15.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:15.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716977+0000", 2023-01-15T14:32:15.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20391002799999999 2023-01-15T14:32:15.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:15.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717146+0000", 2023-01-15T14:32:15.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168851 2023-01-15T14:32:15.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:15.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717467+0000", 2023-01-15T14:32:15.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032135600000000002 2023-01-15T14:32:15.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:15.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717800+0000", 2023-01-15T14:32:15.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033217400000000002 2023-01-15T14:32:15.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:15.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717822+0000", 2023-01-15T14:32:15.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2308999999999999e-05 2023-01-15T14:32:15.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:15.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717970+0000", 2023-01-15T14:32:15.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014788099999999999 2023-01-15T14:32:15.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:15.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.819135+0000", 2023-01-15T14:32:15.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116478 2023-01-15T14:32:15.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:15.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906408+0000", 2023-01-15T14:32:15.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087272902999999999 2023-01-15T14:32:15.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:15.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918887+0000", 2023-01-15T14:32:15.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012479965000000001 2023-01-15T14:32:15.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:15.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918959+0000", 2023-01-15T14:32:15.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1978999999999996e-05 2023-01-15T14:32:15.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:15.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.919067+0000", 2023-01-15T14:32:15.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107083 2023-01-15T14:32:15.814 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:15.814 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:15.814 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:15.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.814 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1485 3.d 3:bc1c6b44:::smithi09025264-23:head [write 598382~780511,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:15.814 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:32:15.814 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.42800837199999, 2023-01-15T14:32:15.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5474121890000001, 2023-01-15T14:32:15.814 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:15.815 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:15.815 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:15.815 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:15.815 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:15.815 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1485 2023-01-15T14:32:15.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.815 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:15.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:15.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:32:15.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360168+0000", 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999828 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:15.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360790+0000", 2023-01-15T14:32:15.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062269099999999998 2023-01-15T14:32:15.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:15.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360822+0000", 2023-01-15T14:32:15.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1903000000000003e-05 2023-01-15T14:32:15.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:15.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360836+0000", 2023-01-15T14:32:15.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3380000000000001e-05 2023-01-15T14:32:15.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:15.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096664+0000", 2023-01-15T14:32:15.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7358282320000002 2023-01-15T14:32:15.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.971 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:15.971 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096683+0000", 2023-01-15T14:32:15.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9063999999999999e-05 2023-01-15T14:32:15.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.971 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:15.971 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704561+0000", 2023-01-15T14:32:15.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60787808399999999 2023-01-15T14:32:15.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.972 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.972 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:15.972 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704846+0000", 2023-01-15T14:32:15.972 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028527400000000002 2023-01-15T14:32:15.972 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.972 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.972 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:15.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705135+0000", 2023-01-15T14:32:15.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028875400000000001 2023-01-15T14:32:15.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:15.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705735+0000", 2023-01-15T14:32:15.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059956899999999999 2023-01-15T14:32:15.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.974 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:15.974 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705761+0000", 2023-01-15T14:32:15.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:32:15.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.974 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.974 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:15.974 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.706129+0000", 2023-01-15T14:32:15.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036822299999999999 2023-01-15T14:32:15.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.974 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.975 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:15.975 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906331+0000", 2023-01-15T14:32:15.975 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200202091 2023-01-15T14:32:15.975 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.975 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.975 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:15.975 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907111+0000", 2023-01-15T14:32:15.975 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077990799999999995 2023-01-15T14:32:15.975 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.976 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.976 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:15.976 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907379+0000", 2023-01-15T14:32:15.976 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026804800000000001 2023-01-15T14:32:15.976 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.976 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.976 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:15.976 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907529+0000", 2023-01-15T14:32:15.976 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015022700000000001 2023-01-15T14:32:15.977 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.978 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.978 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:15.978 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907597+0000", 2023-01-15T14:32:15.978 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7785000000000002e-05 2023-01-15T14:32:15.978 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:15.978 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:15.979 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:15.979 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.979 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.980 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1486 3.d 3:bc1c6b44:::smithi09025264-23:head [write 2075401~451067] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:15.980 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:32:15.980 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.42161232399999, 2023-01-15T14:32:15.980 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6461318330000001, 2023-01-15T14:32:15.980 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:15.981 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:15.981 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:15.981 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:15.981 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:15.981 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1486 2023-01-15T14:32:15.981 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:15.981 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:15.981 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:15.982 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:16.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:32:16.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:16.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:16.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:32:16.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:16.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:16.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366579+0000", 2023-01-15T14:32:16.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:16.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:16.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366845+0000", 2023-01-15T14:32:16.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026610499999999999 2023-01-15T14:32:16.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:16.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366878+0000", 2023-01-15T14:32:16.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2750000000000003e-05 2023-01-15T14:32:16.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:16.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.467009+0000", 2023-01-15T14:32:16.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100131071 2023-01-15T14:32:16.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:16.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096793+0000", 2023-01-15T14:32:16.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6297846869999999 2023-01-15T14:32:16.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:16.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096806+0000", 2023-01-15T14:32:16.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3077e-05 2023-01-15T14:32:16.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:16.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907957+0000", 2023-01-15T14:32:16.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81115055400000002 2023-01-15T14:32:16.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:16.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908229+0000", 2023-01-15T14:32:16.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027187499999999998 2023-01-15T14:32:16.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:16.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908496+0000", 2023-01-15T14:32:16.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026754200000000001 2023-01-15T14:32:16.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:16.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909025+0000", 2023-01-15T14:32:16.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052865200000000001 2023-01-15T14:32:16.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:16.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909046+0000", 2023-01-15T14:32:16.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0713e-05 2023-01-15T14:32:16.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:16.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909294+0000", 2023-01-15T14:32:16.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024864899999999998 2023-01-15T14:32:16.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:16.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.911607+0000", 2023-01-15T14:32:16.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023128939999999998 2023-01-15T14:32:16.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:16.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012076+0000", 2023-01-15T14:32:16.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100468293 2023-01-15T14:32:16.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012426+0000", 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035044299999999998 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012532+0000", 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106142 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:16.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012713+0000", 2023-01-15T14:32:16.378 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018048599999999999 2023-01-15T14:32:16.378 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:16.378 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:16.378 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:16.378 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.379 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1496 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:16.379 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:32:16.379 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.05196947500001, 2023-01-15T14:32:16.379 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870616037, 2023-01-15T14:32:16.380 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:16.380 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:16.380 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:16.380 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:16.380 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:16.380 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1496 2023-01-15T14:32:16.380 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.380 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:16.380 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:16.380 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736220+0000", 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:32:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736238+0000", 2023-01-15T14:32:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7156e-05 2023-01-15T14:32:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736275+0000", 2023-01-15T14:32:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7876000000000001e-05 2023-01-15T14:32:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736290+0000", 2023-01-15T14:32:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4449e-05 2023-01-15T14:32:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602217+0000", 2023-01-15T14:32:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8659270299999999 2023-01-15T14:32:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602468+0000", 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000251315 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.606840+0000", 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0043715009999999999 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.384 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1497 3.11 3:8889e450:::smithi09025264-32:head [read 0~2991646,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:32:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.051882021, 2023-01-15T14:32:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738359249999998, 2023-01-15T14:32:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:16.386 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:16.386 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1497 2023-01-15T14:32:16.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.387 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:16.544 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.544 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:16.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:32:16.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:16.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:16.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:32:16.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:16.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:16.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736310+0000", 2023-01-15T14:32:16.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:16.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:16.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736317+0000", 2023-01-15T14:32:16.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7240000000000003e-06 2023-01-15T14:32:16.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:16.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736334+0000", 2023-01-15T14:32:16.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6878999999999999e-05 2023-01-15T14:32:16.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:16.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736341+0000", 2023-01-15T14:32:16.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0129999999999996e-06 2023-01-15T14:32:16.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:16.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607070+0000", 2023-01-15T14:32:16.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870728873 2023-01-15T14:32:16.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:16.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607226+0000", 2023-01-15T14:32:16.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015595 2023-01-15T14:32:16.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:16.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610147+0000", 2023-01-15T14:32:16.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.002921416 2023-01-15T14:32:16.557 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:16.557 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:16.557 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:16.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.557 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1498 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:16.557 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:32:16.557 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 113.051841915, 2023-01-15T14:32:16.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8752667839999999, 2023-01-15T14:32:16.557 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:16.557 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1498 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:16.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:32:16.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:16.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:16.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736350+0000", 2023-01-15T14:32:16.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:16.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:16.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736355+0000", 2023-01-15T14:32:16.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4900000000000002e-06 2023-01-15T14:32:16.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:16.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736363+0000", 2023-01-15T14:32:16.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2770000000000005e-06 2023-01-15T14:32:16.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:16.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736369+0000", 2023-01-15T14:32:16.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7400000000000001e-06 2023-01-15T14:32:16.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:16.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610234+0000", 2023-01-15T14:32:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738651439999998 2023-01-15T14:32:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610402+0000", 2023-01-15T14:32:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168283 2023-01-15T14:32:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.611618+0000", 2023-01-15T14:32:16.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001215647 2023-01-15T14:32:16.709 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:16.709 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:16.709 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:16.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.7.0:440 3.11 e231/207 ondisk, result = 0)", 2023-01-15T14:32:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:32:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.544060257, 2023-01-15T14:32:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973962139999998, 2023-01-15T14:32:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:16.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:32:16.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:16.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:16.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:32:16.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:16.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:16.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244132+0000", 2023-01-15T14:32:16.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:16.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:16.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244143+0000", 2023-01-15T14:32:16.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1647e-05 2023-01-15T14:32:16.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:16.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244165+0000", 2023-01-15T14:32:16.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1501999999999998e-05 2023-01-15T14:32:16.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:16.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244180+0000", 2023-01-15T14:32:16.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5434000000000002e-05 2023-01-15T14:32:16.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:16.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:16.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:17.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941489+0000", 2023-01-15T14:32:17.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973091899999999 2023-01-15T14:32:17.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:17.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941503+0000", 2023-01-15T14:32:17.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.414e-05 2023-01-15T14:32:17.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:17.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941529+0000", 2023-01-15T14:32:17.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5622999999999998e-05 2023-01-15T14:32:17.029 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:17.029 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:17.029 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:17.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.030 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.4.0:562 3.1b e231/224 3:d85f42d9:::smithi09025264-45:a8 v 231'66, mlcod=225'62)", 2023-01-15T14:32:17.030 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:17.030 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 14.353709061, 2023-01-15T14:32:17.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2578820149999999, 2023-01-15T14:32:17.030 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:17.031 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:17.031 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:17.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:17.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:17.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:17.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:17.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:17.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:17.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:17.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:17.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:17.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:17.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434492+0000", 2023-01-15T14:32:17.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4360000000000002e-06 2023-01-15T14:32:17.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:17.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434517+0000", 2023-01-15T14:32:17.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.48e-05 2023-01-15T14:32:17.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:17.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434530+0000", 2023-01-15T14:32:17.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2982999999999999e-05 2023-01-15T14:32:17.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:17.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434663+0000", 2023-01-15T14:32:17.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013307 2023-01-15T14:32:17.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:17.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434700+0000", 2023-01-15T14:32:17.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7020000000000001e-05 2023-01-15T14:32:17.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:17.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435438+0000", 2023-01-15T14:32:17.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073808900000000004 2023-01-15T14:32:17.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:17.036 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435456+0000", 2023-01-15T14:32:17.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8091999999999999e-05 2023-01-15T14:32:17.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.036 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:17.038 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435526+0000", 2023-01-15T14:32:17.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9806000000000002e-05 2023-01-15T14:32:17.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:17.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435570+0000", 2023-01-15T14:32:17.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4091999999999997e-05 2023-01-15T14:32:17.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:17.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.692366+0000", 2023-01-15T14:32:17.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2567961539999999 2023-01-15T14:32:17.040 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:17.188 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:17.188 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:17.188 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.188 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.189 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:24 3.d 3:b3f111b5:::smithi09025264-48:head [assert-version v160,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e232)", 2023-01-15T14:32:17.189 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:32:17.189 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 13.837872845, 2023-01-15T14:32:17.189 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9963031440000001, 2023-01-15T14:32:17.189 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:17.189 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:17.189 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:17.189 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.189 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:17.189 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:32:17.189 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:17.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:17.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:32:17.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:17.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:17.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950318+0000", 2023-01-15T14:32:17.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:17.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:17.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950333+0000", 2023-01-15T14:32:17.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.467e-05 2023-01-15T14:32:17.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:17.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950368+0000", 2023-01-15T14:32:17.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4895999999999999e-05 2023-01-15T14:32:17.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:17.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050461+0000", 2023-01-15T14:32:17.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10009304400000001 2023-01-15T14:32:17.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:17.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050593+0000", 2023-01-15T14:32:17.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000132764 2023-01-15T14:32:17.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050615+0000", 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1483000000000001e-05 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945049+0000", 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.894434138 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945326+0000", 2023-01-15T14:32:17.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000277368 2023-01-15T14:32:17.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:17.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.946623+0000", 2023-01-15T14:32:17.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012970340000000001 2023-01-15T14:32:17.447 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:17.447 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:17.447 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:17.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.447 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.3.0:547 3.22 e232/180 3:47dfe935:::smithi09025264-29:81 v 232'124, mlcod=229'121)", 2023-01-15T14:32:17.447 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:32:17.447 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 13.737643096999999, 2023-01-15T14:32:17.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.643153431, 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:32:17.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050549+0000", 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050561+0000", 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.278e-05 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:17.450 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050583+0000", 2023-01-15T14:32:17.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1053e-05 2023-01-15T14:32:17.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:17.450 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050596+0000", 2023-01-15T14:32:17.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3946e-05 2023-01-15T14:32:17.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050708+0000", 2023-01-15T14:32:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011129 2023-01-15T14:32:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050750+0000", 2023-01-15T14:32:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2725000000000002e-05 2023-01-15T14:32:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:17.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051324+0000", 2023-01-15T14:32:17.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00057335700000000003 2023-01-15T14:32:17.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:17.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051340+0000", 2023-01-15T14:32:17.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5968999999999999e-05 2023-01-15T14:32:17.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:17.655 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051405+0000", 2023-01-15T14:32:17.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5419000000000005e-05 2023-01-15T14:32:17.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:17.659 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051471+0000", 2023-01-15T14:32:17.659 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5671000000000002e-05 2023-01-15T14:32:17.659 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:17.659 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:17.659 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:17.659 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.693703+0000", 2023-01-15T14:32:17.659 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.642232554 2023-01-15T14:32:17.659 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:17.659 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:17.659 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:17.659 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:17.660 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:17.660 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:17.660 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:13.061+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:17.814 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:32:17.815 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:32:17.815 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:32:17.819 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:32:17.819 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:32:17.820 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:32:17.821 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:17.809+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:17.822 INFO:tasks.rados.rados.0.smithi090.stdout:982: done (1 left) 2023-01-15T14:32:17.822 INFO:tasks.rados.rados.0.smithi090.stdout:985: done (0 left) 2023-01-15T14:32:17.822 INFO:tasks.rados.rados.0.smithi090.stdout:986: snap_create 2023-01-15T14:32:17.823 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:17.823 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:17.824 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:17.824 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:17.824 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:17.824 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:17.824 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:17.824 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:17.825 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:32:17.825 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:32:17.825 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:32:17.825 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:32:17.825 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:32:17.825 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:32:17.825 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:32:17.825 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:32:17.830 INFO:tasks.rados.rados.0.smithi090.stdout:986: done (0 left) 2023-01-15T14:32:17.974 INFO:tasks.rados.rados.0.smithi090.stdout:987: copy_from oid 17 from oid 11 current snap is 97 2023-01-15T14:32:17.974 INFO:tasks.rados.rados.0.smithi090.stdout:988: delete oid 3 current snap is 97 2023-01-15T14:32:17.974 INFO:tasks.rados.rados.0.smithi090.stdout:988: done (1 left) 2023-01-15T14:32:17.974 INFO:tasks.rados.rados.0.smithi090.stdout:989: write (excl) oid 35 current snap is 97 2023-01-15T14:32:17.974 INFO:tasks.rados.rados.0.smithi090.stdout:989: seq_num 368 ranges {584976=476576,1834550=350426} 2023-01-15T14:32:17.974 INFO:tasks.rados.rados.0.smithi090.stdout:989: writing smithi09025264-35 from 584976 to 1061552 tid 1 2023-01-15T14:32:17.975 INFO:tasks.rados.rados.0.smithi090.stdout:989: writing smithi09025264-35 from 1834550 to 2184976 tid 2 2023-01-15T14:32:17.975 INFO:tasks.rados.rados.0.smithi090.stdout:990: copy_from oid 28 from oid 9 current snap is 97 2023-01-15T14:32:17.975 INFO:tasks.rados.rados.0.smithi090.stdout:991: snap_create 2023-01-15T14:32:17.975 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:32:17.975 INFO:tasks.rados.rados.0.smithi090.stdout:990: finishing copy_from to smithi09025264-28 2023-01-15T14:32:17.975 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 164 (ObjNum 352 snap 93 seq_num 352) dirty exists 2023-01-15T14:32:17.975 INFO:tasks.rados.rados.0.smithi090.stdout:990: finishing copy_from racing read to smithi09025264-28 2023-01-15T14:32:17.975 INFO:tasks.rados.rados.0.smithi090.stdout:990: done (2 left) 2023-01-15T14:32:17.975 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:17.976 INFO:tasks.rados.rados.0.smithi090.stdout:987: finishing copy_from to smithi09025264-17 2023-01-15T14:32:17.976 INFO:tasks.rados.rados.0.smithi090.stdout:987: got expected ENOENT (src dne) 2023-01-15T14:32:17.976 INFO:tasks.rados.rados.0.smithi090.stdout:987: finishing copy_from racing read to smithi09025264-17 2023-01-15T14:32:17.977 INFO:tasks.rados.rados.0.smithi090.stdout:987: done (1 left) 2023-01-15T14:32:17.977 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:17.986 INFO:tasks.rados.rados.0.smithi090.stdout:989: finishing write tid 1 to smithi09025264-35 2023-01-15T14:32:17.986 INFO:tasks.rados.rados.0.smithi090.stdout:989: finishing write tid 2 to smithi09025264-35 2023-01-15T14:32:17.986 INFO:tasks.rados.rados.0.smithi090.stdout:989: finishing write tid 3 to smithi09025264-35 2023-01-15T14:32:17.986 INFO:tasks.rados.rados.0.smithi090.stdout:989: finishing write tid 4 to smithi09025264-35 2023-01-15T14:32:17.986 INFO:tasks.rados.rados.0.smithi090.stdout:989: oid 35 updating version 0 to 268 2023-01-15T14:32:17.986 INFO:tasks.rados.rados.0.smithi090.stdout:989: oid 35 updating version 268 to 269 2023-01-15T14:32:17.986 INFO:tasks.rados.rados.0.smithi090.stdout:989: oid 35 updating version 269 to 270 2023-01-15T14:32:17.987 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 270 (ObjNum 368 snap 97 seq_num 368) dirty exists 2023-01-15T14:32:17.987 INFO:tasks.rados.rados.0.smithi090.stdout:989: left oid 35 (ObjNum 368 snap 97 seq_num 368) 2023-01-15T14:32:18.130 INFO:tasks.rados.rados.0.smithi090.stdout:989: done (0 left) 2023-01-15T14:32:18.130 INFO:tasks.rados.rados.0.smithi090.stdout:991: done (0 left) 2023-01-15T14:32:18.130 INFO:tasks.rados.rados.0.smithi090.stdout:992: copy_from oid 37 from oid 31 current snap is 98 2023-01-15T14:32:18.131 INFO:tasks.rados.rados.0.smithi090.stdout:993: read oid 19 snap -1 2023-01-15T14:32:18.131 INFO:tasks.rados.rados.0.smithi090.stdout:993: expect (ObjNum 344 snap 91 seq_num 344) 2023-01-15T14:32:18.131 INFO:tasks.rados.rados.0.smithi090.stdout:994: snap_create 2023-01-15T14:32:18.131 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:18.132 INFO:tasks.rados.rados.0.smithi090.stdout:993: expect (ObjNum 344 snap 91 seq_num 344) 2023-01-15T14:32:18.132 INFO:tasks.rados.rados.0.smithi090.stdout:992: finishing copy_from to smithi09025264-37 2023-01-15T14:32:18.132 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 275 (ObjNum 277 snap 70 seq_num 277) dirty exists 2023-01-15T14:32:18.132 INFO:tasks.rados.rados.0.smithi090.stdout:992: finishing copy_from racing read to smithi09025264-37 2023-01-15T14:32:18.132 INFO:tasks.rados.rados.0.smithi090.stdout:992: done (1 left) 2023-01-15T14:32:18.133 INFO:tasks.rados.rados.0.smithi090.stdout:993: done (0 left) 2023-01-15T14:32:18.133 INFO:tasks.rados.rados.0.smithi090.stdout:994: done (0 left) 2023-01-15T14:32:18.133 INFO:tasks.rados.rados.0.smithi090.stdout:995: rollback oid 27 current snap is 99 2023-01-15T14:32:18.133 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 27 to 95 2023-01-15T14:32:18.133 INFO:tasks.rados.rados.0.smithi090.stdout:996: snap_remove snap 98 2023-01-15T14:32:18.133 INFO:tasks.rados.rados.0.smithi090.stdout:996: done (1 left) 2023-01-15T14:32:18.133 INFO:tasks.rados.rados.0.smithi090.stdout:997: rollback oid 35 current snap is 99 2023-01-15T14:32:18.133 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 35 to 97 2023-01-15T14:32:18.133 INFO:tasks.rados.rados.0.smithi090.stdout:995: finishing rollback tid 1 to smithi09025264-27 2023-01-15T14:32:18.133 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 0 (ObjNum 417 snap 0 seq_num 3200695008) dirty dne 2023-01-15T14:32:18.133 INFO:tasks.rados.rados.0.smithi090.stdout:995: done (1 left) 2023-01-15T14:32:18.134 INFO:tasks.rados.rados.0.smithi090.stdout:998: rollback oid 31 current snap is 99 2023-01-15T14:32:18.134 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 31 to 97 2023-01-15T14:32:18.134 INFO:tasks.rados.rados.0.smithi090.stdout:999: write oid 13 current snap is 99 2023-01-15T14:32:18.134 INFO:tasks.rados.rados.0.smithi090.stdout:999: seq_num 369 ranges {601783=551825,1776679=425104} 2023-01-15T14:32:18.134 INFO:tasks.rados.rados.0.smithi090.stdout:999: writing smithi09025264-13 from 601783 to 1153608 tid 1 2023-01-15T14:32:18.134 INFO:tasks.rados.rados.0.smithi090.stdout:999: writing smithi09025264-13 from 1776679 to 2201783 tid 2 2023-01-15T14:32:18.134 INFO:tasks.rados.rados.0.smithi090.stdout:1000: snap_remove snap 96 2023-01-15T14:32:18.136 INFO:tasks.rados.rados.0.smithi090.stdout:1000: done (3 left) 2023-01-15T14:32:18.136 INFO:tasks.rados.rados.0.smithi090.stdout:1001: delete oid 37 current snap is 99 2023-01-15T14:32:18.136 INFO:tasks.rados.rados.0.smithi090.stdout:999: finishing write tid 1 to smithi09025264-13 2023-01-15T14:32:18.136 INFO:tasks.rados.rados.0.smithi090.stdout:997: finishing rollback tid 0 to smithi09025264-35 2023-01-15T14:32:18.136 INFO:tasks.rados.rados.0.smithi090.stdout:997: finishing rollback tid 1 to smithi09025264-35 2023-01-15T14:32:18.136 INFO:tasks.rados.rados.0.smithi090.stdout:999: finishing write tid 2 to smithi09025264-13 2023-01-15T14:32:18.136 INFO:tasks.rados.rados.0.smithi090.stdout:999: finishing write tid 3 to smithi09025264-13 2023-01-15T14:32:18.136 INFO:tasks.rados.rados.0.smithi090.stdout:999: finishing write tid 4 to smithi09025264-13 2023-01-15T14:32:18.136 INFO:tasks.rados.rados.0.smithi090.stdout:999: oid 13 updating version 0 to 275 2023-01-15T14:32:18.136 INFO:tasks.rados.rados.0.smithi090.stdout:999: oid 13 updating version 275 to 277 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:999: oid 13 version 277 is already newer than 276 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 277 (ObjNum 369 snap 99 seq_num 369) dirty exists 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:999: left oid 13 (ObjNum 369 snap 99 seq_num 369) 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:998: finishing rollback tid 0 to smithi09025264-31 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:997: finishing rollback tid 2 to smithi09025264-35 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 276 (ObjNum 368 snap 97 seq_num 368) dirty exists 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:998: finishing rollback tid 1 to smithi09025264-31 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:998: finishing rollback tid 2 to smithi09025264-31 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 326 (ObjNum 277 snap 70 seq_num 277) dirty exists 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:997: done (3 left) 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:998: done (2 left) 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:999: done (1 left) 2023-01-15T14:32:18.137 INFO:tasks.rados.rados.0.smithi090.stdout:1001: done (0 left) 2023-01-15T14:32:18.138 INFO:tasks.rados.rados.0.smithi090.stdout:1002: snap_remove snap 93 2023-01-15T14:32:18.140 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:18.129+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:18.237 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:18.229+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:18.293 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:18.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:32:18.337 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:18.330+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:18.449 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:18.450 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:18.450 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:32:18.453 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:32:18.454 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:32:18.454 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:18.445+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:18.457 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2023-01-15T14:32:17.845835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":144076680,"num_objects":59,"num_object_clones":24,"num_object_copies":177,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":9,"num_read":895,"num_read_kb":895794,"num_write":747,"num_write_kb":224519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":376,"num_bytes_recovered":247581592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10450,"ondisk_log_size":10450,"up":225,"acting":225,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":226,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":5300044,"kb_used_data":556921680,"kb_used_omap":38423,"kb_used_meta":0,"kb_avail":556896236,"statfs":{"total":575688990720,"available":570261745664,"internally_reserved":26054656,"allocated":570287800320,"data_stored":570287800320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":39345991,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":10,"commit_latency_ns":0,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-36497142,"num_objects":-17,"num_object_clones":-11,"num_object_copies":-51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-17,"num_whiteouts":0,"num_read":14,"num_read_kb":27314,"num_write":3,"num_write_kb":4910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-12,"num_bytes_recovered":-2474831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.879893"},"pg_stats":[{"pgid":"3.41","version":"155'121","reported_seq":834,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.933301+0000","last_change":"2023-01-15T14:32:16.937853+0000","last_active":"2023-01-15T14:32:16.933301+0000","last_peered":"2023-01-15T14:32:16.933301+0000","last_clean":"2023-01-15T14:32:16.933301+0000","last_became_active":"2023-01-15T14:31:56.927677+0000","last_became_peered":"2023-01-15T14:31:56.927677+0000","last_unstale":"2023-01-15T14:32:16.933301+0000","last_undegraded":"2023-01-15T14:32:16.933301+0000","last_fullsized":"2023-01-15T14:32:16.933301+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:31:36.200118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:36.200118+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103601373,"objects_trimmed":0,"snaptrim_duration":0.0051477729999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.40","version":"246'267","reported_seq":1060,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.785826+0000","last_change":"2023-01-15T14:32:15.785826+0000","last_active":"2023-01-15T14:32:15.785826+0000","last_peered":"2023-01-15T14:32:15.785826+0000","last_clean":"2023-01-15T14:32:15.785826+0000","last_became_active":"2023-01-15T14:32:02.030960+0000","last_became_peered":"2023-01-15T14:32:02.030960+0000","last_unstale":"2023-01-15T14:32:15.785826+0000","last_undegraded":"2023-01-15T14:32:15.785826+0000","last_fullsized":"2023-01-15T14:32:15.785826+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.80739877500000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":11,"num_read_kb":8737,"num_write":5,"num_write_kb":1939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":8588443,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.3f","version":"242'205","reported_seq":1028,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.991590+0000","last_change":"2023-01-15T14:32:16.992751+0000","last_active":"2023-01-15T14:32:16.991590+0000","last_peered":"2023-01-15T14:32:16.991590+0000","last_clean":"2023-01-15T14:32:16.991590+0000","last_became_active":"2023-01-15T14:32:02.943089+0000","last_became_peered":"2023-01-15T14:32:02.943089+0000","last_unstale":"2023-01-15T14:32:16.991590+0000","last_undegraded":"2023-01-15T14:32:16.991590+0000","last_fullsized":"2023-01-15T14:32:16.991590+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:09.339","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070746719999999997,"stat_sum":{"num_bytes":7921086,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":49,"num_read_kb":73148,"num_write":33,"num_write_kb":9074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":25819820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.3e","version":"242'47","reported_seq":561,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.150733+0000","last_change":"2023-01-15T14:32:16.150253+0000","last_active":"2023-01-15T14:32:16.150733+0000","last_peered":"2023-01-15T14:32:16.150733+0000","last_clean":"2023-01-15T14:32:16.150733+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:32:16.150733+0000","last_undegraded":"2023-01-15T14:32:16.150733+0000","last_fullsized":"2023-01-15T14:32:16.150733+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:42.278","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.017364599000000001,"stat_sum":{"num_bytes":3916064,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.3d","version":"244'207","reported_seq":1035,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.979227+0000","last_change":"2023-01-15T14:32:12.852723+0000","last_active":"2023-01-15T14:32:14.979227+0000","last_peered":"2023-01-15T14:32:14.979227+0000","last_clean":"2023-01-15T14:32:14.979227+0000","last_became_active":"2023-01-15T14:31:59.249241+0000","last_became_peered":"2023-01-15T14:31:59.249241+0000","last_unstale":"2023-01-15T14:32:14.979227+0000","last_undegraded":"2023-01-15T14:32:14.979227+0000","last_fullsized":"2023-01-15T14:32:14.979227+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:12.136","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.0043861200000000003,"stat_sum":{"num_bytes":3462242,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":83,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.305161+0000","last_change":"2023-01-15T14:32:15.305161+0000","last_active":"2023-01-15T14:32:15.305161+0000","last_peered":"2023-01-15T14:32:15.305161+0000","last_clean":"2023-01-15T14:32:15.305161+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:32:15.305161+0000","last_undegraded":"2023-01-15T14:32:15.305161+0000","last_fullsized":"2023-01-15T14:32:15.305161+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.32414921200000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":579834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.3b","version":"221'60","reported_seq":733,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.933468+0000","last_change":"2023-01-15T14:32:16.935180+0000","last_active":"2023-01-15T14:32:16.933468+0000","last_peered":"2023-01-15T14:32:16.933468+0000","last_clean":"2023-01-15T14:32:16.933468+0000","last_became_active":"2023-01-15T14:31:54.237985+0000","last_became_peered":"2023-01-15T14:31:54.237985+0000","last_unstale":"2023-01-15T14:32:16.933468+0000","last_undegraded":"2023-01-15T14:32:16.933468+0000","last_fullsized":"2023-01-15T14:32:16.933468+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.0023482949999999998,"stat_sum":{"num_bytes":2659480,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6767,"num_write":3,"num_write_kb":817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1080,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.978397+0000","last_change":"2023-01-15T14:32:12.850809+0000","last_active":"2023-01-15T14:32:14.978397+0000","last_peered":"2023-01-15T14:32:14.978397+0000","last_clean":"2023-01-15T14:32:14.978397+0000","last_became_active":"2023-01-15T14:31:57.025706+0000","last_became_peered":"2023-01-15T14:31:57.025706+0000","last_unstale":"2023-01-15T14:32:14.978397+0000","last_undegraded":"2023-01-15T14:32:14.978397+0000","last_fullsized":"2023-01-15T14:32:14.978397+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:30.639","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.0026835349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.39","version":"248'262","reported_seq":1037,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.987447+0000","last_change":"2023-01-15T14:32:17.046615+0000","last_active":"2023-01-15T14:32:16.987447+0000","last_peered":"2023-01-15T14:32:16.987447+0000","last_clean":"2023-01-15T14:32:16.987447+0000","last_became_active":"2023-01-15T14:32:06.420117+0000","last_became_peered":"2023-01-15T14:32:06.420117+0000","last_unstale":"2023-01-15T14:32:16.987447+0000","last_undegraded":"2023-01-15T14:32:16.987447+0000","last_fullsized":"2023-01-15T14:32:16.987447+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'157","last_scrub_stamp":"2023-01-15T14:30:26.792736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:26.792736+0000","objects_scrubbed":4,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:12.259","scrub_duration":0.61998826900000004,"objects_trimmed":0,"snaptrim_duration":0.061717367000000002,"stat_sum":{"num_bytes":8810461,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":23545,"num_write":23,"num_write_kb":8837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":31983837,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.38","version":"247'267","reported_seq":1223,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.584641+0000","last_change":"2023-01-15T14:32:15.584641+0000","last_active":"2023-01-15T14:32:15.584641+0000","last_peered":"2023-01-15T14:32:15.584641+0000","last_clean":"2023-01-15T14:32:15.584641+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:32:15.584641+0000","last_undegraded":"2023-01-15T14:32:15.584641+0000","last_fullsized":"2023-01-15T14:32:15.584641+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.606024644,"stat_sum":{"num_bytes":2199622,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":22,"num_read_kb":18939,"num_write":19,"num_write_kb":6895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.37","version":"200'78","reported_seq":149,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.046742+0000","last_change":"2023-01-15T14:32:17.048468+0000","last_active":"2023-01-15T14:32:17.046742+0000","last_peered":"2023-01-15T14:32:17.046742+0000","last_clean":"2023-01-15T14:32:17.046742+0000","last_became_active":"2023-01-15T14:31:37.024453+0000","last_became_peered":"2023-01-15T14:31:37.024453+0000","last_unstale":"2023-01-15T14:32:17.046742+0000","last_undegraded":"2023-01-15T14:32:17.046742+0000","last_fullsized":"2023-01-15T14:32:17.046742+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:09.695","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.062236405000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.36","version":"194'81","reported_seq":304,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.979686+0000","last_change":"2023-01-15T14:32:12.959142+0000","last_active":"2023-01-15T14:32:14.979686+0000","last_peered":"2023-01-15T14:32:14.979686+0000","last_clean":"2023-01-15T14:32:14.979686+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:32:14.979686+0000","last_undegraded":"2023-01-15T14:32:14.979686+0000","last_fullsized":"2023-01-15T14:32:14.979686+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.11025191199999999,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.35","version":"110'55","reported_seq":725,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.079638+0000","last_change":"2023-01-15T14:32:14.796398+0000","last_active":"2023-01-15T14:32:15.079638+0000","last_peered":"2023-01-15T14:32:15.079638+0000","last_clean":"2023-01-15T14:32:15.079638+0000","last_became_active":"2023-01-15T14:31:38.033824+0000","last_became_peered":"2023-01-15T14:31:38.033824+0000","last_unstale":"2023-01-15T14:32:15.079638+0000","last_undegraded":"2023-01-15T14:32:15.079638+0000","last_fullsized":"2023-01-15T14:32:15.079638+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:30:24.721082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.721082+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.109000768,"objects_trimmed":0,"snaptrim_duration":0.72637969099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":897,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.264271+0000","last_change":"2023-01-15T14:32:17.265569+0000","last_active":"2023-01-15T14:32:17.264271+0000","last_peered":"2023-01-15T14:32:17.264271+0000","last_clean":"2023-01-15T14:32:17.264271+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:32:17.264271+0000","last_undegraded":"2023-01-15T14:32:17.264271+0000","last_fullsized":"2023-01-15T14:32:17.264271+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:48.327","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.23083030399999999,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5618,"num_write":3,"num_write_kb":609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.33","version":"204'275","reported_seq":977,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.979641+0000","last_change":"2023-01-15T14:32:14.901842+0000","last_active":"2023-01-15T14:32:14.979641+0000","last_peered":"2023-01-15T14:32:14.979641+0000","last_clean":"2023-01-15T14:32:14.979641+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:32:14.979641+0000","last_undegraded":"2023-01-15T14:32:14.979641+0000","last_fullsized":"2023-01-15T14:32:14.979641+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.93327276100000001,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.32","version":"243'157","reported_seq":576,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.978563+0000","last_change":"2023-01-15T14:32:12.851547+0000","last_active":"2023-01-15T14:32:14.978563+0000","last_peered":"2023-01-15T14:32:14.978563+0000","last_clean":"2023-01-15T14:32:14.978563+0000","last_became_active":"2023-01-15T14:31:36.921952+0000","last_became_peered":"2023-01-15T14:31:36.921952+0000","last_unstale":"2023-01-15T14:32:14.978563+0000","last_undegraded":"2023-01-15T14:32:14.978563+0000","last_fullsized":"2023-01-15T14:32:14.978563+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:56.390","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.003395393,"stat_sum":{"num_bytes":2403401,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.13","version":"248'305","reported_seq":1043,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.988488+0000","last_change":"2023-01-15T14:32:16.989763+0000","last_active":"2023-01-15T14:32:16.988488+0000","last_peered":"2023-01-15T14:32:16.988488+0000","last_clean":"2023-01-15T14:32:16.988488+0000","last_became_active":"2023-01-15T14:31:35.097414+0000","last_became_peered":"2023-01-15T14:31:35.097414+0000","last_unstale":"2023-01-15T14:32:16.988488+0000","last_undegraded":"2023-01-15T14:32:16.988488+0000","last_fullsized":"2023-01-15T14:32:16.988488+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:45.688","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.0047135839999999998,"stat_sum":{"num_bytes":16454185,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":12,"num_read_kb":17489,"num_write":4,"num_write_kb":9916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.12","version":"248'160","reported_seq":561,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.909502+0000","last_change":"2023-01-15T14:32:15.909502+0000","last_active":"2023-01-15T14:32:15.909502+0000","last_peered":"2023-01-15T14:32:15.909502+0000","last_clean":"2023-01-15T14:32:15.909502+0000","last_became_active":"2023-01-15T14:31:35.028565+0000","last_became_peered":"2023-01-15T14:31:35.028565+0000","last_unstale":"2023-01-15T14:32:15.909502+0000","last_undegraded":"2023-01-15T14:32:15.909502+0000","last_fullsized":"2023-01-15T14:32:15.909502+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"139'103","last_scrub_stamp":"2023-01-15T14:30:30.045615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:30.045615+0000","objects_scrubbed":3,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.207128334,"objects_trimmed":0,"snaptrim_duration":0.82746519600000001,"stat_sum":{"num_bytes":3832029,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":23,"num_read_kb":26972,"num_write":14,"num_write_kb":4087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6292614,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.11","version":"246'229","reported_seq":1062,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.049420+0000","last_change":"2023-01-15T14:32:17.049420+0000","last_active":"2023-01-15T14:32:17.049420+0000","last_peered":"2023-01-15T14:32:17.049420+0000","last_clean":"2023-01-15T14:32:17.049420+0000","last_became_active":"2023-01-15T14:31:35.026209+0000","last_became_peered":"2023-01-15T14:31:35.026209+0000","last_unstale":"2023-01-15T14:32:17.049420+0000","last_undegraded":"2023-01-15T14:32:17.049420+0000","last_fullsized":"2023-01-15T14:32:17.049420+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:06.167","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.005959621,"stat_sum":{"num_bytes":7395113,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":22,"num_read_kb":27259,"num_write":11,"num_write_kb":8749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.10","version":"246'322","reported_seq":1134,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.090904+0000","last_change":"2023-01-15T14:32:14.904195+0000","last_active":"2023-01-15T14:32:15.090904+0000","last_peered":"2023-01-15T14:32:15.090904+0000","last_clean":"2023-01-15T14:32:15.090904+0000","last_became_active":"2023-01-15T14:31:35.135778+0000","last_became_peered":"2023-01-15T14:31:35.135778+0000","last_unstale":"2023-01-15T14:32:15.090904+0000","last_undegraded":"2023-01-15T14:32:15.090904+0000","last_fullsized":"2023-01-15T14:32:15.090904+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:50.443","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.83518732699999998,"stat_sum":{"num_bytes":3202274,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":25951,"num_write":59,"num_write_kb":19807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":13340457,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":960,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.188716+0000","last_change":"2023-01-15T14:32:17.190901+0000","last_active":"2023-01-15T14:32:17.188716+0000","last_peered":"2023-01-15T14:32:17.188716+0000","last_clean":"2023-01-15T14:32:17.188716+0000","last_became_active":"2023-01-15T14:31:35.032038+0000","last_became_peered":"2023-01-15T14:31:35.032038+0000","last_unstale":"2023-01-15T14:32:17.188716+0000","last_undegraded":"2023-01-15T14:32:17.188716+0000","last_fullsized":"2023-01-15T14:32:17.188716+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:31:22.549180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:22.549180+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:15.089","scrub_duration":0.50470070300000003,"objects_trimmed":0,"snaptrim_duration":0.25385522500000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":760,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.265805+0000","last_change":"2023-01-15T14:32:17.266961+0000","last_active":"2023-01-15T14:32:17.265805+0000","last_peered":"2023-01-15T14:32:17.265805+0000","last_clean":"2023-01-15T14:32:17.265805+0000","last_became_active":"2023-01-15T14:31:35.237663+0000","last_became_peered":"2023-01-15T14:31:35.237663+0000","last_unstale":"2023-01-15T14:32:17.265805+0000","last_undegraded":"2023-01-15T14:32:17.265805+0000","last_fullsized":"2023-01-15T14:32:17.265805+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:16.384","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.16671492299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.d","version":"244'193","reported_seq":986,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.989901+0000","last_change":"2023-01-15T14:32:16.991437+0000","last_active":"2023-01-15T14:32:16.989901+0000","last_peered":"2023-01-15T14:32:16.989901+0000","last_clean":"2023-01-15T14:32:16.989901+0000","last_became_active":"2023-01-15T14:31:21.132225+0000","last_became_peered":"2023-01-15T14:31:21.132225+0000","last_unstale":"2023-01-15T14:32:16.989901+0000","last_undegraded":"2023-01-15T14:32:16.989901+0000","last_fullsized":"2023-01-15T14:32:16.989901+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:41.817","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.0059726099999999997,"stat_sum":{"num_bytes":2369820,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.c","version":"242'63","reported_seq":752,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.992884+0000","last_change":"2023-01-15T14:32:15.992884+0000","last_active":"2023-01-15T14:32:15.992884+0000","last_peered":"2023-01-15T14:32:15.992884+0000","last_clean":"2023-01-15T14:32:15.992884+0000","last_became_active":"2023-01-15T14:31:21.227945+0000","last_became_peered":"2023-01-15T14:31:21.227945+0000","last_unstale":"2023-01-15T14:32:15.992884+0000","last_undegraded":"2023-01-15T14:32:15.992884+0000","last_fullsized":"2023-01-15T14:32:15.992884+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:42.336","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":1.010929352,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":12,"num_write_kb":6025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":720,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.891400+0000","last_change":"2023-01-15T14:32:15.891400+0000","last_active":"2023-01-15T14:32:15.891400+0000","last_peered":"2023-01-15T14:32:15.891400+0000","last_clean":"2023-01-15T14:32:15.891400+0000","last_became_active":"2023-01-15T14:31:21.230392+0000","last_became_peered":"2023-01-15T14:31:21.230392+0000","last_unstale":"2023-01-15T14:32:15.891400+0000","last_undegraded":"2023-01-15T14:32:15.891400+0000","last_fullsized":"2023-01-15T14:32:15.891400+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:38.384","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81035930700000003,"stat_sum":{"num_bytes":865560,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3347,"num_write":5,"num_write_kb":2817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.a","version":"247'275","reported_seq":1052,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.191018+0000","last_change":"2023-01-15T14:32:17.263379+0000","last_active":"2023-01-15T14:32:17.191018+0000","last_peered":"2023-01-15T14:32:17.191018+0000","last_clean":"2023-01-15T14:32:17.191018+0000","last_became_active":"2023-01-15T14:31:21.032574+0000","last_became_peered":"2023-01-15T14:31:21.032574+0000","last_unstale":"2023-01-15T14:32:17.191018+0000","last_undegraded":"2023-01-15T14:32:17.191018+0000","last_fullsized":"2023-01-15T14:32:17.191018+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.308854609,"stat_sum":{"num_bytes":4261584,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":43,"num_read_kb":42799,"num_write":54,"num_write_kb":19709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":18895452,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":842,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.937337+0000","last_change":"2023-01-15T14:32:16.938658+0000","last_active":"2023-01-15T14:32:16.937337+0000","last_peered":"2023-01-15T14:32:16.937337+0000","last_clean":"2023-01-15T14:32:16.937337+0000","last_became_active":"2023-01-15T14:31:21.122856+0000","last_became_peered":"2023-01-15T14:31:21.122856+0000","last_unstale":"2023-01-15T14:32:16.937337+0000","last_undegraded":"2023-01-15T14:32:16.937337+0000","last_fullsized":"2023-01-15T14:32:16.937337+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:31:44.640345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.640345+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:46.259","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.0048535729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.8","version":"179'204","reported_seq":1044,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.979230+0000","last_change":"2023-01-15T14:32:14.900669+0000","last_active":"2023-01-15T14:32:14.979230+0000","last_peered":"2023-01-15T14:32:14.979230+0000","last_clean":"2023-01-15T14:32:14.979230+0000","last_became_active":"2023-01-15T14:31:21.332760+0000","last_became_peered":"2023-01-15T14:31:21.332760+0000","last_unstale":"2023-01-15T14:32:14.979230+0000","last_undegraded":"2023-01-15T14:32:14.979230+0000","last_fullsized":"2023-01-15T14:32:14.979230+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:14.668","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.73115200000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":384,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.001487+0000","last_change":"2023-01-15T14:31:03.091584+0000","last_active":"2023-01-15T14:32:15.001487+0000","last_peered":"2023-01-15T14:32:15.001487+0000","last_clean":"2023-01-15T14:32:15.001487+0000","last_became_active":"2023-01-15T14:31:03.091071+0000","last_became_peered":"2023-01-15T14:31:03.091071+0000","last_unstale":"2023-01-15T14:32:15.001487+0000","last_undegraded":"2023-01-15T14:32:15.001487+0000","last_fullsized":"2023-01-15T14:32:15.001487+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:29:45.876183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:29:45.876183+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":223,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.980666+0000","last_change":"2023-01-15T14:31:38.018850+0000","last_active":"2023-01-15T14:32:14.980666+0000","last_peered":"2023-01-15T14:32:14.980666+0000","last_clean":"2023-01-15T14:32:14.980666+0000","last_became_active":"2023-01-15T14:31:38.018145+0000","last_became_peered":"2023-01-15T14:31:38.018145+0000","last_unstale":"2023-01-15T14:32:14.980666+0000","last_undegraded":"2023-01-15T14:32:14.980666+0000","last_fullsized":"2023-01-15T14:32:14.980666+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:43.428","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":730,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.263514+0000","last_change":"2023-01-15T14:32:17.264764+0000","last_active":"2023-01-15T14:32:17.263514+0000","last_peered":"2023-01-15T14:32:17.263514+0000","last_clean":"2023-01-15T14:32:17.263514+0000","last_became_active":"2023-01-15T14:31:44.311861+0000","last_became_peered":"2023-01-15T14:31:44.311861+0000","last_unstale":"2023-01-15T14:32:17.263514+0000","last_undegraded":"2023-01-15T14:32:17.263514+0000","last_fullsized":"2023-01-15T14:32:17.263514+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.310061331,"stat_sum":{"num_bytes":1084068,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3268978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":285,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.985065+0000","last_change":"2023-01-15T14:31:11.136609+0000","last_active":"2023-01-15T14:32:16.985065+0000","last_peered":"2023-01-15T14:32:16.985065+0000","last_clean":"2023-01-15T14:32:16.985065+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:32:16.985065+0000","last_undegraded":"2023-01-15T14:32:16.985065+0000","last_fullsized":"2023-01-15T14:32:16.985065+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:11.136490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.136490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:36.593","scrub_duration":0.39969354600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":595,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.149539+0000","last_change":"2023-01-15T14:32:13.151160+0000","last_active":"2023-01-15T14:32:16.149539+0000","last_peered":"2023-01-15T14:32:16.149539+0000","last_clean":"2023-01-15T14:32:16.149539+0000","last_became_active":"2023-01-15T14:32:13.047395+0000","last_became_peered":"2023-01-15T14:32:13.047395+0000","last_unstale":"2023-01-15T14:32:16.149539+0000","last_undegraded":"2023-01-15T14:32:16.149539+0000","last_fullsized":"2023-01-15T14:32:16.149539+0000","mapping_epoch":245,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":246,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:31:36.200118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:36.200118+0000","objects_scrubbed":1,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.28528039199999999,"objects_trimmed":0,"snaptrim_duration":0.102660846,"stat_sum":{"num_bytes":949595,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":52,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.079203+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:32:15.079203+0000","last_peered":"2023-01-15T14:32:15.079203+0000","last_clean":"2023-01-15T14:32:15.079203+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:32:15.079203+0000","last_undegraded":"2023-01-15T14:32:15.079203+0000","last_fullsized":"2023-01-15T14:32:15.079203+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:38.960","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1064,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.078336+0000","last_change":"2023-01-15T14:32:12.960864+0000","last_active":"2023-01-15T14:32:15.078336+0000","last_peered":"2023-01-15T14:32:15.078336+0000","last_clean":"2023-01-15T14:32:15.078336+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:32:15.078336+0000","last_undegraded":"2023-01-15T14:32:15.078336+0000","last_fullsized":"2023-01-15T14:32:15.078336+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:31.185","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.012264713,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":106,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.979045+0000","last_change":"2023-01-15T14:31:13.764052+0000","last_active":"2023-01-15T14:32:14.979045+0000","last_peered":"2023-01-15T14:32:14.979045+0000","last_clean":"2023-01-15T14:32:14.979045+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:32:14.979045+0000","last_undegraded":"2023-01-15T14:32:14.979045+0000","last_fullsized":"2023-01-15T14:32:14.979045+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:13.763945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:13.763945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:39.419","scrub_duration":0.58613919699999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"247'302","reported_seq":1160,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.939748+0000","last_change":"2023-01-15T14:32:17.140903+0000","last_active":"2023-01-15T14:32:16.939748+0000","last_peered":"2023-01-15T14:32:16.939748+0000","last_clean":"2023-01-15T14:32:16.939748+0000","last_became_active":"2023-01-15T14:31:03.093753+0000","last_became_peered":"2023-01-15T14:31:03.093753+0000","last_unstale":"2023-01-15T14:32:16.939748+0000","last_undegraded":"2023-01-15T14:32:16.939748+0000","last_fullsized":"2023-01-15T14:32:16.939748+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'251","last_scrub_stamp":"2023-01-15T14:31:15.363400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.363400+0000","objects_scrubbed":3,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:42.073","scrub_duration":0.208136242,"objects_trimmed":0,"snaptrim_duration":0.20559697800000001,"stat_sum":{"num_bytes":6706059,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":42,"num_read_kb":44989,"num_write":23,"num_write_kb":8236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.5","version":"248'123","reported_seq":858,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.414336+0000","last_change":"2023-01-15T14:32:16.414336+0000","last_active":"2023-01-15T14:32:16.414336+0000","last_peered":"2023-01-15T14:32:16.414336+0000","last_clean":"2023-01-15T14:32:16.414336+0000","last_became_active":"2023-01-15T14:31:21.132693+0000","last_became_peered":"2023-01-15T14:31:21.132693+0000","last_unstale":"2023-01-15T14:32:16.414336+0000","last_undegraded":"2023-01-15T14:32:16.414336+0000","last_fullsized":"2023-01-15T14:32:16.414336+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:25.367","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.3337499720000001,"stat_sum":{"num_bytes":5271951,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":35,"num_read_kb":39854,"num_write":30,"num_write_kb":13317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9491720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.179757+0000","last_change":"2023-01-15T14:30:51.237821+0000","last_active":"2023-01-15T14:32:15.179757+0000","last_peered":"2023-01-15T14:32:15.179757+0000","last_clean":"2023-01-15T14:32:15.179757+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:32:15.179757+0000","last_undegraded":"2023-01-15T14:32:15.179757+0000","last_fullsized":"2023-01-15T14:32:15.179757+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:29.346364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:29.346364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"245'268","reported_seq":1111,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.079616+0000","last_change":"2023-01-15T14:32:12.960326+0000","last_active":"2023-01-15T14:32:15.079616+0000","last_peered":"2023-01-15T14:32:15.079616+0000","last_clean":"2023-01-15T14:32:15.079616+0000","last_became_active":"2023-01-15T14:31:21.226293+0000","last_became_peered":"2023-01-15T14:31:21.226293+0000","last_unstale":"2023-01-15T14:32:15.079616+0000","last_undegraded":"2023-01-15T14:32:15.079616+0000","last_fullsized":"2023-01-15T14:32:15.079616+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.094589099999999995,"stat_sum":{"num_bytes":5774434,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":61362,"num_write":29,"num_write_kb":10072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":15862107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":276,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.094711+0000","last_change":"2023-01-15T14:31:15.088518+0000","last_active":"2023-01-15T14:32:16.094711+0000","last_peered":"2023-01-15T14:32:16.094711+0000","last_clean":"2023-01-15T14:32:16.094711+0000","last_became_active":"2023-01-15T14:31:03.088392+0000","last_became_peered":"2023-01-15T14:31:03.088392+0000","last_unstale":"2023-01-15T14:32:16.094711+0000","last_undegraded":"2023-01-15T14:32:16.094711+0000","last_fullsized":"2023-01-15T14:32:16.094711+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:15.088397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.088397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:32.351","scrub_duration":0.40518630900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"245'112","reported_seq":839,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.141039+0000","last_change":"2023-01-15T14:32:17.189703+0000","last_active":"2023-01-15T14:32:17.141039+0000","last_peered":"2023-01-15T14:32:17.141039+0000","last_clean":"2023-01-15T14:32:17.141039+0000","last_became_active":"2023-01-15T14:31:44.117230+0000","last_became_peered":"2023-01-15T14:31:44.117230+0000","last_unstale":"2023-01-15T14:32:17.141039+0000","last_undegraded":"2023-01-15T14:32:17.141039+0000","last_fullsized":"2023-01-15T14:32:17.141039+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.25298987499999998,"stat_sum":{"num_bytes":993524,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":8458,"num_write":9,"num_write_kb":2509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1731121,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":96,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.981365+0000","last_change":"2023-01-15T14:30:51.035825+0000","last_active":"2023-01-15T14:32:14.981365+0000","last_peered":"2023-01-15T14:32:14.981365+0000","last_clean":"2023-01-15T14:32:14.981365+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:32:14.981365+0000","last_undegraded":"2023-01-15T14:32:14.981365+0000","last_fullsized":"2023-01-15T14:32:14.981365+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":742,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.080156+0000","last_change":"2023-01-15T14:32:12.954204+0000","last_active":"2023-01-15T14:32:15.080156+0000","last_peered":"2023-01-15T14:32:15.080156+0000","last_clean":"2023-01-15T14:32:15.080156+0000","last_became_active":"2023-01-15T14:31:21.030929+0000","last_became_peered":"2023-01-15T14:31:21.030929+0000","last_unstale":"2023-01-15T14:32:15.080156+0000","last_undegraded":"2023-01-15T14:32:15.080156+0000","last_fullsized":"2023-01-15T14:32:15.080156+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:31:25.342830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:25.342830+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:44.671","scrub_duration":0.20886623600000001,"objects_trimmed":0,"snaptrim_duration":0.104864254,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":232,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.980029+0000","last_change":"2023-01-15T14:31:38.112378+0000","last_active":"2023-01-15T14:32:14.980029+0000","last_peered":"2023-01-15T14:32:14.980029+0000","last_clean":"2023-01-15T14:32:14.980029+0000","last_became_active":"2023-01-15T14:31:38.111590+0000","last_became_peered":"2023-01-15T14:31:38.111590+0000","last_unstale":"2023-01-15T14:32:14.980029+0000","last_undegraded":"2023-01-15T14:32:14.980029+0000","last_fullsized":"2023-01-15T14:32:14.980029+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:51.156","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"245'148","reported_seq":890,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.264863+0000","last_change":"2023-01-15T14:32:17.266412+0000","last_active":"2023-01-15T14:32:17.264863+0000","last_peered":"2023-01-15T14:32:17.264863+0000","last_clean":"2023-01-15T14:32:17.264863+0000","last_became_active":"2023-01-15T14:31:35.134196+0000","last_became_peered":"2023-01-15T14:31:35.134196+0000","last_unstale":"2023-01-15T14:32:17.264863+0000","last_undegraded":"2023-01-15T14:32:17.264863+0000","last_fullsized":"2023-01-15T14:32:17.264863+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.22999705100000001,"stat_sum":{"num_bytes":1226927,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":13,"num_read_kb":15209,"num_write":4,"num_write_kb":608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":715,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.787216+0000","last_change":"2023-01-15T14:32:15.787216+0000","last_active":"2023-01-15T14:32:15.787216+0000","last_peered":"2023-01-15T14:32:15.787216+0000","last_clean":"2023-01-15T14:32:15.787216+0000","last_became_active":"2023-01-15T14:31:34.951721+0000","last_became_peered":"2023-01-15T14:31:34.951721+0000","last_unstale":"2023-01-15T14:32:15.787216+0000","last_undegraded":"2023-01-15T14:32:15.787216+0000","last_fullsized":"2023-01-15T14:32:15.787216+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:30:24.721082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.721082+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.109000768,"objects_trimmed":0,"snaptrim_duration":0.70696439700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.16","version":"245'110","reported_seq":337,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.981131+0000","last_change":"2023-01-15T14:32:12.959732+0000","last_active":"2023-01-15T14:32:14.981131+0000","last_peered":"2023-01-15T14:32:14.981131+0000","last_clean":"2023-01-15T14:32:14.981131+0000","last_became_active":"2023-01-15T14:31:35.032807+0000","last_became_peered":"2023-01-15T14:31:35.032807+0000","last_unstale":"2023-01-15T14:32:14.981131+0000","last_undegraded":"2023-01-15T14:32:14.981131+0000","last_fullsized":"2023-01-15T14:32:14.981131+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:40.949","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.11007618499999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":13661,"num_write":6,"num_write_kb":1586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.17","version":"228'82","reported_seq":754,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.888815+0000","last_change":"2023-01-15T14:32:15.888815+0000","last_active":"2023-01-15T14:32:15.888815+0000","last_peered":"2023-01-15T14:32:15.888815+0000","last_clean":"2023-01-15T14:32:15.888815+0000","last_became_active":"2023-01-15T14:31:34.932309+0000","last_became_peered":"2023-01-15T14:31:34.932309+0000","last_unstale":"2023-01-15T14:32:15.888815+0000","last_undegraded":"2023-01-15T14:32:15.888815+0000","last_fullsized":"2023-01-15T14:32:15.888815+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'65","last_scrub_stamp":"2023-01-15T14:30:24.156798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.156798+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30368223799999999,"objects_trimmed":0,"snaptrim_duration":0.90730483200000001,"stat_sum":{"num_bytes":3731187,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":21869,"num_write":8,"num_write_kb":2801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1974822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.18","version":"247'272","reported_seq":1169,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.094180+0000","last_change":"2023-01-15T14:32:16.094180+0000","last_active":"2023-01-15T14:32:16.094180+0000","last_peered":"2023-01-15T14:32:16.094180+0000","last_clean":"2023-01-15T14:32:16.094180+0000","last_became_active":"2023-01-15T14:31:52.330386+0000","last_became_peered":"2023-01-15T14:31:52.330386+0000","last_unstale":"2023-01-15T14:32:16.094180+0000","last_undegraded":"2023-01-15T14:32:16.094180+0000","last_fullsized":"2023-01-15T14:32:16.094180+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":1.115073449,"stat_sum":{"num_bytes":7133375,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":38314,"num_write":24,"num_write_kb":8521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":11033036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.19","version":"243'227","reported_seq":1008,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.978541+0000","last_change":"2023-01-15T14:32:12.849617+0000","last_active":"2023-01-15T14:32:14.978541+0000","last_peered":"2023-01-15T14:32:14.978541+0000","last_clean":"2023-01-15T14:32:14.978541+0000","last_became_active":"2023-01-15T14:31:51.927440+0000","last_became_peered":"2023-01-15T14:31:51.927440+0000","last_unstale":"2023-01-15T14:32:14.978541+0000","last_undegraded":"2023-01-15T14:32:14.978541+0000","last_fullsized":"2023-01-15T14:32:14.978541+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:07.960","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.0021039309999999999,"stat_sum":{"num_bytes":3260591,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.1a","version":"250'277","reported_seq":1111,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.262301+0000","last_change":"2023-01-15T14:32:17.264153+0000","last_active":"2023-01-15T14:32:17.262301+0000","last_peered":"2023-01-15T14:32:17.262301+0000","last_clean":"2023-01-15T14:32:17.262301+0000","last_became_active":"2023-01-15T14:31:52.124026+0000","last_became_peered":"2023-01-15T14:31:52.124026+0000","last_unstale":"2023-01-15T14:32:17.262301+0000","last_undegraded":"2023-01-15T14:32:17.262301+0000","last_fullsized":"2023-01-15T14:32:17.262301+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'196","last_scrub_stamp":"2023-01-15T14:31:19.373635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:19.373635+0000","objects_scrubbed":4,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20803622599999999,"objects_trimmed":0,"snaptrim_duration":0.31001330999999999,"stat_sum":{"num_bytes":976929,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":5063,"num_write":23,"num_write_kb":6363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.1b","version":"246'72","reported_seq":733,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.935590+0000","last_change":"2023-01-15T14:32:16.937227+0000","last_active":"2023-01-15T14:32:16.935590+0000","last_peered":"2023-01-15T14:32:16.935590+0000","last_clean":"2023-01-15T14:32:16.935590+0000","last_became_active":"2023-01-15T14:31:52.321399+0000","last_became_peered":"2023-01-15T14:31:52.321399+0000","last_unstale":"2023-01-15T14:32:16.935590+0000","last_undegraded":"2023-01-15T14:32:16.935590+0000","last_fullsized":"2023-01-15T14:32:16.935590+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.0035368589999999998,"stat_sum":{"num_bytes":987672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":6766,"num_write":2,"num_write_kb":817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":269,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.036427+0000","last_change":"2023-01-15T14:32:17.037634+0000","last_active":"2023-01-15T14:32:17.036427+0000","last_peered":"2023-01-15T14:32:17.036427+0000","last_clean":"2023-01-15T14:32:17.036427+0000","last_became_active":"2023-01-15T14:31:51.926652+0000","last_became_peered":"2023-01-15T14:31:51.926652+0000","last_unstale":"2023-01-15T14:32:17.036427+0000","last_undegraded":"2023-01-15T14:32:17.036427+0000","last_fullsized":"2023-01-15T14:32:17.036427+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:33.090","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.10189623,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1006,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.938772+0000","last_change":"2023-01-15T14:32:16.939610+0000","last_active":"2023-01-15T14:32:16.938772+0000","last_peered":"2023-01-15T14:32:16.938772+0000","last_clean":"2023-01-15T14:32:16.938772+0000","last_became_active":"2023-01-15T14:31:52.025227+0000","last_became_peered":"2023-01-15T14:31:52.025227+0000","last_unstale":"2023-01-15T14:32:16.938772+0000","last_undegraded":"2023-01-15T14:32:16.938772+0000","last_fullsized":"2023-01-15T14:32:16.938772+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:32.052","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044734739999999999,"stat_sum":{"num_bytes":5739855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":47622,"num_write":25,"num_write_kb":10630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6928644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.1e","version":"215'39","reported_seq":704,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.080551+0000","last_change":"2023-01-15T14:32:14.905507+0000","last_active":"2023-01-15T14:32:15.080551+0000","last_peered":"2023-01-15T14:32:15.080551+0000","last_clean":"2023-01-15T14:32:15.080551+0000","last_became_active":"2023-01-15T14:31:52.329815+0000","last_became_peered":"2023-01-15T14:31:52.329815+0000","last_unstale":"2023-01-15T14:32:15.080551+0000","last_undegraded":"2023-01-15T14:32:15.080551+0000","last_fullsized":"2023-01-15T14:32:15.080551+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:31:44.906253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.906253+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55943661099999997,"objects_trimmed":0,"snaptrim_duration":0.93571720300000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.1f","version":"247'230","reported_seq":1075,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.037750+0000","last_change":"2023-01-15T14:32:17.188542+0000","last_active":"2023-01-15T14:32:17.037750+0000","last_peered":"2023-01-15T14:32:17.037750+0000","last_clean":"2023-01-15T14:32:17.037750+0000","last_became_active":"2023-01-15T14:31:52.124821+0000","last_became_peered":"2023-01-15T14:31:52.124821+0000","last_unstale":"2023-01-15T14:32:17.037750+0000","last_undegraded":"2023-01-15T14:32:17.037750+0000","last_fullsized":"2023-01-15T14:32:17.037750+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":230,"log_dups_size":0,"ondisk_log_size":230,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.25226622900000001,"stat_sum":{"num_bytes":675777,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":68,"num_read_kb":113780,"num_write":36,"num_write_kb":10844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":18290251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.20","version":"245'258","reported_seq":1110,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.985323+0000","last_change":"2023-01-15T14:32:16.988386+0000","last_active":"2023-01-15T14:32:16.985323+0000","last_peered":"2023-01-15T14:32:16.985323+0000","last_clean":"2023-01-15T14:32:16.985323+0000","last_became_active":"2023-01-15T14:31:07.906191+0000","last_became_peered":"2023-01-15T14:31:07.906191+0000","last_unstale":"2023-01-15T14:32:16.985323+0000","last_undegraded":"2023-01-15T14:32:16.985323+0000","last_fullsized":"2023-01-15T14:32:16.985323+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.0036881510000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":21726,"num_write":8,"num_write_kb":2065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8877936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.21","version":"244'130","reported_seq":751,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.993772+0000","last_change":"2023-01-15T14:32:17.045353+0000","last_active":"2023-01-15T14:32:16.993772+0000","last_peered":"2023-01-15T14:32:16.993772+0000","last_clean":"2023-01-15T14:32:16.993772+0000","last_became_active":"2023-01-15T14:31:38.233481+0000","last_became_peered":"2023-01-15T14:31:38.233481+0000","last_unstale":"2023-01-15T14:32:16.993772+0000","last_undegraded":"2023-01-15T14:32:16.993772+0000","last_fullsized":"2023-01-15T14:32:16.993772+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"168'124","last_scrub_stamp":"2023-01-15T14:31:11.414546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.414546+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:48.531","scrub_duration":0.28528039199999999,"objects_trimmed":0,"snaptrim_duration":0.059568045,"stat_sum":{"num_bytes":2067327,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":2711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3798382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.22","version":"246'129","reported_seq":1010,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.304404+0000","last_change":"2023-01-15T14:32:15.304404+0000","last_active":"2023-01-15T14:32:15.304404+0000","last_peered":"2023-01-15T14:32:15.304404+0000","last_clean":"2023-01-15T14:32:15.304404+0000","last_became_active":"2023-01-15T14:31:08.084483+0000","last_became_peered":"2023-01-15T14:31:08.084483+0000","last_unstale":"2023-01-15T14:32:15.304404+0000","last_undegraded":"2023-01-15T14:32:15.304404+0000","last_fullsized":"2023-01-15T14:32:15.304404+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:31.869","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.32262083499999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.23","version":"149'233","reported_seq":917,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.890123+0000","last_change":"2023-01-15T14:32:15.890123+0000","last_active":"2023-01-15T14:32:15.890123+0000","last_peered":"2023-01-15T14:32:15.890123+0000","last_clean":"2023-01-15T14:32:15.890123+0000","last_became_active":"2023-01-15T14:31:06.029716+0000","last_became_peered":"2023-01-15T14:31:06.029716+0000","last_unstale":"2023-01-15T14:32:15.890123+0000","last_undegraded":"2023-01-15T14:32:15.890123+0000","last_fullsized":"2023-01-15T14:32:15.890123+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:48.395","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.91007939199999999,"stat_sum":{"num_bytes":7604272,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":47,"num_read_kb":43637,"num_write":37,"num_write_kb":13160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2469494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.24","version":"245'223","reported_seq":1083,"reported_epoch":249,"state":"active+clean+scrubbing","last_fresh":"2023-01-15T14:32:16.526478+0000","last_change":"2023-01-15T14:32:16.422020+0000","last_active":"2023-01-15T14:32:16.526478+0000","last_peered":"2023-01-15T14:32:16.526478+0000","last_clean":"2023-01-15T14:32:16.526478+0000","last_became_active":"2023-01-15T14:31:26.168777+0000","last_became_peered":"2023-01-15T14:31:26.168777+0000","last_unstale":"2023-01-15T14:32:16.526478+0000","last_undegraded":"2023-01-15T14:32:16.526478+0000","last_fullsized":"2023-01-15T14:32:16.526478+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"scrubbing for 0s","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.93308668400000006,"stat_sum":{"num_bytes":4343446,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":47,"num_read_kb":63482,"num_write":21,"num_write_kb":6075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":21209193,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.25","version":"243'93","reported_seq":830,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.979459+0000","last_change":"2023-01-15T14:32:12.953397+0000","last_active":"2023-01-15T14:32:14.979459+0000","last_peered":"2023-01-15T14:32:14.979459+0000","last_clean":"2023-01-15T14:32:14.979459+0000","last_became_active":"2023-01-15T14:31:22.000386+0000","last_became_peered":"2023-01-15T14:31:22.000386+0000","last_unstale":"2023-01-15T14:32:14.979459+0000","last_undegraded":"2023-01-15T14:32:14.979459+0000","last_fullsized":"2023-01-15T14:32:14.979459+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:51.596","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.104560242,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6061,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.26","version":"102'52","reported_seq":186,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.085845+0000","last_change":"2023-01-15T14:32:17.187280+0000","last_active":"2023-01-15T14:32:17.085845+0000","last_peered":"2023-01-15T14:32:17.085845+0000","last_clean":"2023-01-15T14:32:17.085845+0000","last_became_active":"2023-01-15T14:31:21.901622+0000","last_became_peered":"2023-01-15T14:31:21.901622+0000","last_unstale":"2023-01-15T14:32:17.085845+0000","last_undegraded":"2023-01-15T14:32:17.085845+0000","last_fullsized":"2023-01-15T14:32:17.085845+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:31:29.194049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:29.194049+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:33.783","scrub_duration":0.55352289399999999,"objects_trimmed":0,"snaptrim_duration":0.101638597,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":801,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.979666+0000","last_change":"2023-01-15T14:32:14.906547+0000","last_active":"2023-01-15T14:32:14.979666+0000","last_peered":"2023-01-15T14:32:14.979666+0000","last_clean":"2023-01-15T14:32:14.979666+0000","last_became_active":"2023-01-15T14:31:22.091646+0000","last_became_peered":"2023-01-15T14:31:22.091646+0000","last_unstale":"2023-01-15T14:32:14.979666+0000","last_undegraded":"2023-01-15T14:32:14.979666+0000","last_fullsized":"2023-01-15T14:32:14.979666+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:41.261","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.93641192799999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.28","version":"243'222","reported_seq":1089,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.978735+0000","last_change":"2023-01-15T14:32:14.899505+0000","last_active":"2023-01-15T14:32:14.978735+0000","last_peered":"2023-01-15T14:32:14.978735+0000","last_clean":"2023-01-15T14:32:14.978735+0000","last_became_active":"2023-01-15T14:31:26.054878+0000","last_became_peered":"2023-01-15T14:31:26.054878+0000","last_unstale":"2023-01-15T14:32:14.978735+0000","last_undegraded":"2023-01-15T14:32:14.978735+0000","last_fullsized":"2023-01-15T14:32:14.978735+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:01.279","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.83038681800000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":10594,"num_write":3,"num_write_kb":3981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":12193528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.29","version":"110'149","reported_seq":612,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:15.079078+0000","last_change":"2023-01-15T14:32:14.898182+0000","last_active":"2023-01-15T14:32:15.079078+0000","last_peered":"2023-01-15T14:32:15.079078+0000","last_clean":"2023-01-15T14:32:15.079078+0000","last_became_active":"2023-01-15T14:31:25.136376+0000","last_became_peered":"2023-01-15T14:31:25.136376+0000","last_unstale":"2023-01-15T14:32:15.079078+0000","last_undegraded":"2023-01-15T14:32:15.079078+0000","last_fullsized":"2023-01-15T14:32:15.079078+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:30:08.681869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:08.681869+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:14.346","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.82874932499999998,"stat_sum":{"num_bytes":2857174,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":24874,"num_write":16,"num_write_kb":6171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.2a","version":"191'191","reported_seq":981,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.985474+0000","last_change":"2023-01-15T14:32:16.987284+0000","last_active":"2023-01-15T14:32:16.985474+0000","last_peered":"2023-01-15T14:32:16.985474+0000","last_clean":"2023-01-15T14:32:16.985474+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:32:16.985474+0000","last_undegraded":"2023-01-15T14:32:16.985474+0000","last_fullsized":"2023-01-15T14:32:16.985474+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:44.953","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.002469443,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":682,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.979960+0000","last_change":"2023-01-15T14:32:12.852157+0000","last_active":"2023-01-15T14:32:14.979960+0000","last_peered":"2023-01-15T14:32:14.979960+0000","last_clean":"2023-01-15T14:32:14.979960+0000","last_became_active":"2023-01-15T14:31:21.896494+0000","last_became_peered":"2023-01-15T14:31:21.896494+0000","last_unstale":"2023-01-15T14:32:14.979960+0000","last_undegraded":"2023-01-15T14:32:14.979960+0000","last_fullsized":"2023-01-15T14:32:14.979960+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:30.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.975433+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:33.669","scrub_duration":0.0040286109999999997,"objects_trimmed":0,"snaptrim_duration":0.0039173469999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":176,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.937957+0000","last_change":"2023-01-15T14:32:17.036291+0000","last_active":"2023-01-15T14:32:16.937957+0000","last_peered":"2023-01-15T14:32:16.937957+0000","last_clean":"2023-01-15T14:32:16.937957+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:32:16.937957+0000","last_undegraded":"2023-01-15T14:32:16.937957+0000","last_fullsized":"2023-01-15T14:32:16.937957+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:47.318","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.101556549,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.2d","version":"168'156","reported_seq":954,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:16.992874+0000","last_change":"2023-01-15T14:32:16.993563+0000","last_active":"2023-01-15T14:32:16.992874+0000","last_peered":"2023-01-15T14:32:16.992874+0000","last_clean":"2023-01-15T14:32:16.992874+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:32:16.992874+0000","last_undegraded":"2023-01-15T14:32:16.992874+0000","last_fullsized":"2023-01-15T14:32:16.992874+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.0078529619999999998,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.2e","version":"139'38","reported_seq":627,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.979856+0000","last_change":"2023-01-15T14:32:12.966879+0000","last_active":"2023-01-15T14:32:14.979856+0000","last_peered":"2023-01-15T14:32:14.979856+0000","last_clean":"2023-01-15T14:32:14.979856+0000","last_became_active":"2023-01-15T14:31:38.116756+0000","last_became_peered":"2023-01-15T14:31:38.116756+0000","last_unstale":"2023-01-15T14:32:14.979856+0000","last_undegraded":"2023-01-15T14:32:14.979856+0000","last_fullsized":"2023-01-15T14:32:14.979856+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:51.292","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.117749649,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":829,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.045463+0000","last_change":"2023-01-15T14:32:17.047580+0000","last_active":"2023-01-15T14:32:17.045463+0000","last_peered":"2023-01-15T14:32:17.045463+0000","last_clean":"2023-01-15T14:32:17.045463+0000","last_became_active":"2023-01-15T14:31:40.024531+0000","last_became_peered":"2023-01-15T14:31:40.024531+0000","last_unstale":"2023-01-15T14:32:17.045463+0000","last_undegraded":"2023-01-15T14:32:17.045463+0000","last_fullsized":"2023-01-15T14:32:17.045463+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:34.781","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.061717853000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.30","version":"206'252","reported_seq":1082,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-01-15T14:32:17.085042+0000","last_change":"2023-01-15T14:32:17.086382+0000","last_active":"2023-01-15T14:32:17.085042+0000","last_peered":"2023-01-15T14:32:17.085042+0000","last_clean":"2023-01-15T14:32:17.085042+0000","last_became_active":"2023-01-15T14:31:39.915702+0000","last_became_peered":"2023-01-15T14:31:39.915702+0000","last_unstale":"2023-01-15T14:32:17.085042+0000","last_undegraded":"2023-01-15T14:32:17.085042+0000","last_fullsized":"2023-01-15T14:32:17.085042+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:58.379","scrub_duration":1.1539330249999999,"objects_trimmed":0,"snaptrim_duration":0.00168052,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"b4","length":"1"},{"start":"b9","length":"1"},{"start":"c1","length":"1"}]},{"pgid":"3.31","version":"204'150","reported_seq":961,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-01-15T14:32:14.978214+0000","last_change":"2023-01-15T14:32:12.848386+0000","last_active":"2023-01-15T14:32:14.978214+0000","last_peered":"2023-01-15T14:32:14.978214+0000","last_clean":"2023-01-15T14:32:14.978214+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:32:14.978214+0000","last_undegraded":"2023-01-15T14:32:14.978214+0000","last_fullsized":"2023-01-15T14:32:14.978214+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:16.599","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.0012558960000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8c","length":"1"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a7","length":"1"},{"start":"ac","length":"3"},{"start":"b4","length":"2"},{"start":"b9","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":66,"stat_sum":{"num_bytes":143486312,"num_objects":57,"num_object_clones":24,"num_object_copies":171,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":9,"num_read":779,"num_read_kb":895694,"num_write":588,"num_write_kb":222103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":368,"num_bytes_recovered":245220120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10333,"ondisk_log_size":10333,"up":198,"acting":198,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574900,"num_pgs":24,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":870072,"kb_used_data":92836192,"kb_used_omap":3837,"kb_used_meta":0,"kb_avail":92829308,"statfs":{"total":95948165120,"available":95057211392,"internally_reserved":7049216,"allocated":95064260608,"data_stored":95064260608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3930028,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":4.774,"15min":4.3040000000000003},"min":{"1min":0.55500000000000005,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.98199999999999998,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":2.9169999999999998,"5min":6.2539999999999996,"15min":8.2240000000000002},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":43.125,"5min":73.769000000000005,"15min":73.769000000000005},"last":1.032}]},{"osd":1,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":2.2909999999999999,"5min":4.4909999999999997,"15min":5.8600000000000003},"min":{"1min":0.56899999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":31.518999999999998,"5min":88.003,"15min":88.003},"last":0.93500000000000005},{"interface":"front","average":{"1min":9.8260000000000005,"5min":6.79,"15min":6.4500000000000002},"min":{"1min":0.59399999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.581999999999994,"5min":97.704999999999998,"15min":97.704999999999998},"last":1.202}]},{"osd":2,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":14.355,"5min":7.7380000000000004,"15min":5.7729999999999997},"min":{"1min":0.51400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":151.31800000000001,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.0089999999999999,"5min":3.2309999999999999,"15min":3.8620000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.633,"5min":91.837999999999994,"15min":91.837999999999994},"last":1.0549999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":18.995000000000001,"5min":7.3920000000000003,"15min":3.0960000000000001},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":209.05500000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":5.7030000000000003,"5min":4.383,"15min":4.5620000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":98.825000000000003,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.83699999999999997}]},{"osd":4,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":2.9430000000000001,"5min":5.5949999999999998,"15min":8.1419999999999995},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":42.893999999999998,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.871},{"interface":"front","average":{"1min":8.5739999999999998,"5min":5.5899999999999999,"15min":2.5059999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":154.999,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.77200000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":3.0379999999999998,"5min":4.4029999999999996,"15min":5.6849999999999996},"min":{"1min":0.52100000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":43.029000000000003,"5min":73.561000000000007,"15min":73.561000000000007},"last":1.1639999999999999},{"interface":"front","average":{"1min":3.7930000000000001,"5min":3.9950000000000001,"15min":4.6280000000000001},"min":{"1min":0.48399999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":58.134999999999998,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.143}]},{"osd":6,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":3.109,"5min":5.351,"15min":7.4660000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":42.978999999999999,"5min":73.638999999999996,"15min":73.638999999999996},"last":1.1200000000000001},{"interface":"front","average":{"1min":3.8410000000000002,"5min":2.5169999999999999,"15min":1.5720000000000001},"min":{"1min":0.60299999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":58.204999999999998,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.087}]}]},{"osd":6,"up_from":0,"seq":55834574901,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574901,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":50,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":940084,"kb_used_data":92767540,"kb_used_omap":7816,"kb_used_meta":0,"kb_avail":92759296,"statfs":{"total":95948165120,"available":94985519104,"internally_reserved":8441856,"allocated":94993960960,"data_stored":94993960960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8004455,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":103.827,"5min":24.414999999999999,"15min":11.414999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2165.8420000000001,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":5.7409999999999997,"5min":3,"15min":1.52},"min":{"1min":0.56999999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":88.600999999999999,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.64900000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":6.6109999999999998,"15min":7.9329999999999998},"min":{"1min":0.50700000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":69.415000000000006,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":1.863,"5min":4.3360000000000003,"15min":1.974},"min":{"1min":0.58999999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":22.411999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.621}]},{"osd":2,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":103.90300000000001,"5min":25.164999999999999,"15min":12.061999999999999},"min":{"1min":0.59899999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2165.3220000000001,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.871,"15min":0.89700000000000002},"min":{"1min":0.623,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0349999999999999,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.71499999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":10.015000000000001,"5min":3,"15min":1.8919999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":88.823999999999998,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.72599999999999998},{"interface":"front","average":{"1min":105.232,"5min":25.382000000000001,"15min":12.561},"min":{"1min":0.628,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2165.4690000000001,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":14.961}]},{"osd":5,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.88100000000000001,"15min":0.90700000000000003},"min":{"1min":0.58299999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.302,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":5.867,"5min":2.4590000000000001,"15min":2.3519999999999999},"min":{"1min":0.68200000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":88.911000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.81899999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":4.8620000000000001,"15min":5.4020000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.341,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":104.14,"5min":23.286000000000001,"15min":11.204000000000001},"min":{"1min":0.84599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2165.7330000000002,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":0.78700000000000003}]},{"osd":7,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":9.5570000000000004,"5min":7.5519999999999996,"15min":11.269},"min":{"1min":0.76100000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":89.061000000000007,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":104.11,"5min":26.864999999999998,"15min":16.446999999999999},"min":{"1min":0.78600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2165.5309999999999,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":0.877}]}]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":50,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":882852,"kb_used_data":92817832,"kb_used_omap":8069,"kb_used_meta":0,"kb_avail":92816528,"statfs":{"total":95948165120,"available":95044124672,"internally_reserved":1335296,"allocated":95045459968,"data_stored":95045459968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8262867,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":12.667,"5min":8.0429999999999993,"15min":5.9130000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":122.065,"5min":174.52199999999999,"15min":174.52199999999999},"last":4.3559999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":4.8109999999999999,"15min":6.6980000000000004},"min":{"1min":0.41299999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.0469999999999999,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.50900000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":4.1790000000000003,"5min":9.7569999999999997,"15min":12.273},"min":{"1min":0.45000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":51.162999999999997,"5min":159.256,"15min":159.256},"last":0.47899999999999998},{"interface":"front","average":{"1min":20.414000000000001,"5min":10.041,"15min":3.8410000000000002},"min":{"1min":0.42799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":121.203,"5min":174.578,"15min":174.578},"last":4.4710000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":2.6139999999999999,"15min":3.1040000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3700000000000001,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":13.534000000000001,"5min":6.8380000000000001,"15min":3.7410000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":122.13,"5min":174.45099999999999,"15min":174.45099999999999},"last":4.4260000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":122.569,"5min":29.460999999999999,"15min":14.195},"min":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2059.6680000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":84.640000000000001},{"interface":"front","average":{"1min":122.634,"5min":34.735999999999997,"15min":20.827999999999999},"min":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2058.4229999999998,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":0.621}]},{"osd":5,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":2.0019999999999998,"5min":9.0939999999999994,"15min":12.300000000000001},"min":{"1min":0.625,"5min":0.622,"15min":0.622},"max":{"1min":21.789999999999999,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":13.038,"5min":5.7619999999999996,"15min":2.6019999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":122.208,"5min":174.61000000000001,"15min":174.61000000000001},"last":4.5229999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":5.766,"15min":2.6099999999999999},"min":{"1min":0.65900000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":122.26600000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":4.5659999999999998},{"interface":"front","average":{"1min":0.98399999999999999,"5min":2.4969999999999999,"15min":3.3399999999999999},"min":{"1min":0.70999999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.49,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.73699999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.5990000000000002,"15min":3.5569999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.522,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.77700000000000002},{"interface":"front","average":{"1min":5.444,"5min":11.496,"15min":15.891999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":55.337000000000003,"5min":159.512,"15min":159.512},"last":0.82999999999999996}]}]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":41,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":925220,"kb_used_data":92781864,"kb_used_omap":6367,"kb_used_meta":0,"kb_avail":92774160,"statfs":{"total":95948165120,"available":95000739840,"internally_reserved":7888896,"allocated":95008628736,"data_stored":95008628736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6519822,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":3.2869999999999999,"5min":3.0299999999999998,"15min":1.4370000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":55.649999999999999,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.4410000000000001,"15min":0.95699999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.85199999999999998,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.50700000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":7.359,"5min":5.0350000000000001,"15min":2.2109999999999999},"min":{"1min":0.27700000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":83.007999999999996,"5min":143.071,"15min":143.071},"last":91.525999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.5489999999999999,"15min":3.0870000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.0109999999999999,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.57199999999999995}]},{"osd":3,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":8.6809999999999992,"5min":10.118,"15min":10.307},"min":{"1min":0.308,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":86.191999999999993,"5min":183.53,"15min":183.53},"last":0.55100000000000005},{"interface":"front","average":{"1min":3.3500000000000001,"5min":4.3319999999999999,"15min":1.9550000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":54.307000000000002,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.81299999999999994}]},{"osd":4,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":4.718,"5min":6.1630000000000003,"15min":4.4870000000000001},"min":{"1min":0.66300000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":54.383000000000003,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.96399999999999997},{"interface":"front","average":{"1min":2.077,"5min":5.4790000000000001,"15min":4.3650000000000002},"min":{"1min":0.59299999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":24.859999999999999,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.90500000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":2.5139999999999998,"15min":1.6220000000000001},"min":{"1min":0.622,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2709999999999999,"5min":143.262,"15min":143.262},"last":0.86899999999999999},{"interface":"front","average":{"1min":3.5920000000000001,"5min":2.5899999999999999,"15min":1.637},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":54.353999999999999,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.89100000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.8660000000000001,"15min":1.3100000000000001},"min":{"1min":0.59599999999999997,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4259999999999999,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.94199999999999995,"5min":2.6560000000000001,"15min":1.6970000000000001},"min":{"1min":0.64300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.319,"5min":143.215,"15min":143.215},"last":0.78800000000000003}]},{"osd":7,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":4.306,"15min":3.4740000000000002},"min":{"1min":0.66900000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":118.652,"5min":118.652,"15min":118.652},"last":0.747},{"interface":"front","average":{"1min":0.95299999999999996,"5min":3.198,"15min":2.9169999999999998},"min":{"1min":0.67300000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.278,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.93700000000000006}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":50,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":876704,"kb_used_data":92823940,"kb_used_omap":7747,"kb_used_meta":0,"kb_avail":92822676,"statfs":{"total":95948165120,"available":95050420224,"internally_reserved":1294336,"allocated":95051714560,"data_stored":95051714560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7933320,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":4.258,"15min":4.0140000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.91300000000000003,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":5.7370000000000001,"5min":3.9289999999999998,"15min":4.1909999999999998},"min":{"1min":0.39800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":86.927000000000007,"5min":86.927000000000007,"15min":86.927000000000007},"last":0.82599999999999996}]},{"osd":2,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":1.764,"15min":1.232},"min":{"1min":0.25700000000000001,"5min":0.245,"15min":0.245},"max":{"1min":64.941999999999993,"5min":64.941999999999993,"15min":64.941999999999993},"last":0.629},{"interface":"front","average":{"1min":0.66400000000000003,"5min":2.5049999999999999,"15min":3.5499999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.044,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.67500000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":4.9669999999999996,"5min":12.464,"15min":17.378},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":90.161000000000001,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.752},{"interface":"front","average":{"1min":11.167999999999999,"5min":6.7830000000000004,"15min":7.0910000000000002},"min":{"1min":0.33100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":86.894999999999996,"5min":86.894999999999996,"15min":86.894999999999996},"last":0.61199999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:32:12 2023","interfaces":[{"interface":"back","average":{"1min":5.6849999999999996,"5min":16.501000000000001,"15min":8.6519999999999992},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":87.128,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":1.056},{"interface":"front","average":{"1min":9.8059999999999992,"5min":15.890000000000001,"15min":7.7000000000000002},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":84.629999999999995,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":6.9800000000000004}]},{"osd":5,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":3.9940000000000002,"5min":1.5740000000000001,"15min":1.2130000000000001},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":65.120999999999995,"5min":65.120999999999995,"15min":65.120999999999995},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.89200000000000002,"5min":2.327,"15min":3.734},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.2270000000000001,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.95199999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":6.0309999999999997,"5min":4.4119999999999999,"15min":4.8049999999999997},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":87.153000000000006,"5min":87.153000000000006,"15min":87.153000000000006},"last":0.97999999999999998},{"interface":"front","average":{"1min":4.0220000000000002,"5min":1.6359999999999999,"15min":1.294},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":65.156000000000006,"5min":65.156000000000006,"15min":65.156000000000006},"last":0.88500000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":5.3310000000000004,"5min":6.7930000000000001,"15min":6.9420000000000002},"min":{"1min":0.73799999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":91.843999999999994,"5min":157.03700000000001,"15min":157.03700000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":6.0880000000000001,"5min":4.3399999999999999,"15min":4.6130000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":87.069999999999993,"5min":87.069999999999993,"15min":87.069999999999993},"last":1.1040000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":805112,"kb_used_data":92894312,"kb_used_omap":4585,"kb_used_meta":0,"kb_avail":92894268,"statfs":{"total":95948165120,"available":95123730432,"internally_reserved":45056,"allocated":95123775488,"data_stored":95123775488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4695499,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":2.7509999999999999,"5min":3.1230000000000002,"15min":1.456},"min":{"1min":0.29499999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":49.725000000000001,"5min":69.872,"15min":69.872},"last":0.71099999999999997},{"interface":"front","average":{"1min":3.5139999999999998,"5min":3.2349999999999999,"15min":1.5649999999999999},"min":{"1min":0.33700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":61.447000000000003,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.67300000000000004}]},{"osd":2,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":3.4159999999999999,"15min":1.601},"min":{"1min":0.52700000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.2,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":2.6499999999999999,"15min":1.3560000000000001},"min":{"1min":0.32400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.90800000000000003,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.624}]},{"osd":3,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":2.863,"5min":10.630000000000001,"15min":14.728999999999999},"min":{"1min":0.372,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":49.331000000000003,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":8.0960000000000001,"5min":7.0990000000000002,"15min":2.8999999999999999},"min":{"1min":0.45800000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":162.61600000000001,"5min":174.09399999999999,"15min":174.09399999999999},"last":1.077}]},{"osd":4,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":68.361999999999995,"5min":16.706,"15min":6.226},"min":{"1min":0.63900000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1486.309,"5min":1486.309,"15min":1486.309},"last":0.92300000000000004},{"interface":"front","average":{"1min":70.664000000000001,"5min":17.922999999999998,"15min":6.657},"min":{"1min":0.56599999999999995,"5min":0.439,"15min":0.439},"max":{"1min":1486.3789999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":1.046}]},{"osd":5,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":6.4420000000000002,"15min":8.6790000000000003},"min":{"1min":0.64700000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4910000000000001,"5min":176.34399999999999,"15min":176.34399999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.7370000000000001,"15min":1.2709999999999999},"min":{"1min":0.60699999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3220000000000001,"5min":71.563999999999993,"15min":71.563999999999993},"last":1.3220000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":3.198,"5min":3.609,"15min":1.9830000000000001},"min":{"1min":0.68999999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":49.616,"5min":70.259,"15min":70.259},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.006,"5min":3.0299999999999998,"15min":1.766},"min":{"1min":0.755,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.4419999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":1.3879999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":9.0449999999999999,"15min":13.462},"min":{"1min":0.57199999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.137,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":3.21,"5min":3.617,"15min":1.986},"min":{"1min":0.67100000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":49.670000000000002,"5min":70.322000000000003,"15min":70.322000000000003},"last":1.353}]}]}],"pool_statfs":[]}} 2023-01-15T14:32:18.786 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:32:18.786 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 76 2023-01-15T14:32:18.949 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:18.949 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:18.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:32:18.950 INFO:tasks.rados.rados.0.smithi090.stdout:1002: done (0 left) 2023-01-15T14:32:18.950 INFO:tasks.rados.rados.0.smithi090.stdout:1003: snap_create 2023-01-15T14:32:18.951 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:32:18.951 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:18.942+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:19.095 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:19.086+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:19.194 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:19.190+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:19.254 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:19.254 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:19.255 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:32:19.295 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:19.289+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:19.405 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:19.405 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:19.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:32:19.406 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:19.398+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:19.506 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:19.505+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:19.575 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:19.575 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:19.576 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:32:19.733 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:19.726+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:19.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:32:19.883 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:19.884 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:19.884 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:32:19.885 INFO:tasks.rados.rados.0.smithi090.stdout:1003: done (0 left) 2023-01-15T14:32:19.885 INFO:tasks.rados.rados.0.smithi090.stdout:1004: delete oid 14 current snap is 100 2023-01-15T14:32:19.887 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 76 2023-01-15T14:32:20.050 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:19.877+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:20.050 INFO:tasks.rados.rados.0.smithi090.stdout:1004: done (0 left) 2023-01-15T14:32:20.050 INFO:tasks.rados.rados.0.smithi090.stdout:1005: delete oid 43 current snap is 100 2023-01-15T14:32:20.052 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:20.046+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:20.151 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:20.149+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:20.201 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:20.201 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:20.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:32:20.202 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:20.202 INFO:tasks.rados.rados.0.smithi090.stdout:1005: done (0 left) 2023-01-15T14:32:20.202 INFO:tasks.rados.rados.0.smithi090.stdout:1006: snap_remove snap 99 2023-01-15T14:32:20.205 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:20.222 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:20.240 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:20.252 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:20.246+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:20.367 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:20.367 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:20.367 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:20.368 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:32:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:32:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:32:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:38 3.0 e23/20 3:0a998d04:::smithi09025264-10:head v 23'3, mlcod=0'0)", 2023-01-15T14:32:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:32:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 234.19617514399999, 2023-01-15T14:32:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30191175999999997, 2023-01-15T14:32:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:20.369 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:20.370 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:20.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:20.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:32:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112978+0000", 2023-01-15T14:32:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.112976+0000", 2023-01-15T14:32:20.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113186+0000", 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020979700000000001 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.113228+0000", 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2150999999999997e-05 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.213313+0000", 2023-01-15T14:32:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100084943 2023-01-15T14:32:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313517+0000", 2023-01-15T14:32:20.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10020435699999999 2023-01-15T14:32:20.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:20.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313576+0000", 2023-01-15T14:32:20.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8783999999999998e-05 2023-01-15T14:32:20.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.313976+0000", 2023-01-15T14:32:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039960499999999998 2023-01-15T14:32:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314004+0000", 2023-01-15T14:32:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7954e-05 2023-01-15T14:32:20.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.314175+0000", 2023-01-15T14:32:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171557 2023-01-15T14:32:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414853+0000", 2023-01-15T14:32:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100677875 2023-01-15T14:32:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.414890+0000", 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6803e-05 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:125 3.0 e70/55 3:0b54190a:::smithi09025264-17:25 v 70'136, mlcod=68'130)", 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 184.366193119, 2023-01-15T14:32:20.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30171926799999998, 2023-01-15T14:32:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:32:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942960+0000", 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942958+0000", 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:20.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942972+0000", 2023-01-15T14:32:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3386e-05 2023-01-15T14:32:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.942990+0000", 2023-01-15T14:32:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8113000000000001e-05 2023-01-15T14:32:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:15.943002+0000", 2023-01-15T14:32:20.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141942+0000", 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19893962300000001 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.141978+0000", 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6383e-05 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142670+0000", 2023-01-15T14:32:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069211900000000005 2023-01-15T14:32:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142688+0000", 2023-01-15T14:32:20.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7839999999999999e-05 2023-01-15T14:32:20.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.142768+0000", 2023-01-15T14:32:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0210999999999995e-05 2023-01-15T14:32:20.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244581+0000", 2023-01-15T14:32:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101813406 2023-01-15T14:32:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:16.244680+0000", 2023-01-15T14:32:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8045000000000001e-05 2023-01-15T14:32:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:20.418 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:32:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:32:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 154.250450731, 2023-01-15T14:32:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:32:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:20.419 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:32:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:32:20.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:32:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:32:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:32:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:32:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:32:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:32:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:32:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:32:20.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:32:20.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:20.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:32:20.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:32:20.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:20.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:32:20.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:32:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:32:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:32:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:32:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:32:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:32:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:32:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:32:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:32:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:32:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:32:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 105.414693341, 2023-01-15T14:32:20.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:32:20.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:20.537 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:20.537 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:20.547 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:20.547 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:20.548 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:32:20.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.548 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:32:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:32:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:32:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:32:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:32:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:32:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:32:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:32:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:32:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:32:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:32:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:32:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:32:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:20.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:32:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:32:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:32:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:32:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:32:20.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:32:20.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:20.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:32:20.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:32:20.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:20.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:32:20.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:32:20.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:20.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:32:20.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:32:20.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:20.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:32:20.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:32:20.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:20.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:32:20.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:32:20.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:20.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:32:20.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:32:20.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:32:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:32:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:32:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:32:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:20.854 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:32:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:32:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 105.414642398, 2023-01-15T14:32:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:32:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:20.855 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:20.856 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:20.856 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:32:20.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.856 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:20.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:20.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:32:20.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:20.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:32:20.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:20.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:32:20.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:20.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:20.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:32:20.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:32:20.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:20.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:32:20.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:32:20.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:20.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:32:20.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:32:20.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:20.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:32:20.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:32:20.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:20.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:32:20.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:32:20.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:20.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:32:20.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:32:20.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:20.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:32:20.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:32:20.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:20.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:32:20.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:32:20.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:20.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:20.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:20.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.864 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:32:20.864 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:32:20.864 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 102.473743157, 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:20.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:32:20.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:20.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:32:20.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:20.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:20.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:32:20.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:20.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:20.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:32:20.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:32:20.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:20.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:32:20.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:32:20.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:20.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:32:20.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:32:20.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:20.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:32:20.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:32:20.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:32:20.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:32:20.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:32:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:32:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:32:20.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:32:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:32:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:32:20.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:32:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:20.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:32:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:32:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:20.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:32:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:32:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:20.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:32:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:32:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:20.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:32:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:32:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:32:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:32:21.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:32:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:32:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:32:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:32:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:32:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:32:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:32:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.399968984, 2023-01-15T14:32:21.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:32:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:32:21.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:32:21.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:32:21.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:32:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:32:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:32:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:32:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:32:21.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:32:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:32:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:32:21.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:32:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1712 3.4 3:2022fdfa:::smithi09025264-46:head [write 1880802~421790] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:32:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.394830293, 2023-01-15T14:32:21.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30396041800000001, 2023-01-15T14:32:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1712 2023-01-15T14:32:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914322+0000", 2023-01-15T14:32:21.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914561+0000", 2023-01-15T14:32:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023884700000000001 2023-01-15T14:32:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914572+0000", 2023-01-15T14:32:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0499999999999999e-05 2023-01-15T14:32:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914576+0000", 2023-01-15T14:32:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6820000000000002e-06 2023-01-15T14:32:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014941+0000", 2023-01-15T14:32:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100364994 2023-01-15T14:32:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014959+0000", 2023-01-15T14:32:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7416000000000001e-05 2023-01-15T14:32:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018171+0000", 2023-01-15T14:32:21.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032121239999999998 2023-01-15T14:32:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018333+0000", 2023-01-15T14:32:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016180900000000001 2023-01-15T14:32:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018581+0000", 2023-01-15T14:32:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024773699999999999 2023-01-15T14:32:21.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018897+0000", 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031622099999999999 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018907+0000", 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0061000000000001e-05 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.019108+0000", 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020080099999999999 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.020057+0000", 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094954499999999999 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150472+0000", 2023-01-15T14:32:21.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.130414958 2023-01-15T14:32:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218188+0000", 2023-01-15T14:32:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067715656999999999 2023-01-15T14:32:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218242+0000", 2023-01-15T14:32:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4651999999999998e-05 2023-01-15T14:32:21.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218284+0000", 2023-01-15T14:32:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1084999999999997e-05 2023-01-15T14:32:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:32:21.201 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:32:21.334 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.506013564, 2023-01-15T14:32:21.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:32:21.335 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.336 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:21.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:21.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:21.337 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:21.337 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:32:21.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.337 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:32:21.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:32:21.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:32:21.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:32:21.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:32:21.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:32:21.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:32:21.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:21.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:32:21.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:32:21.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:32:21.343 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.343 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.343 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.343 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1820 3.1f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:32:21.343 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:32:21.343 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.405712078999997, 2023-01-15T14:32:21.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306504938, 2023-01-15T14:32:21.344 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.344 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:21.344 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:21.344 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:21.344 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:21.344 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1820 2023-01-15T14:32:21.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.344 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903440+0000", 2023-01-15T14:32:21.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:21.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903448+0000", 2023-01-15T14:32:21.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9430000000000006e-06 2023-01-15T14:32:21.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903469+0000", 2023-01-15T14:32:21.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0305e-05 2023-01-15T14:32:21.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903481+0000", 2023-01-15T14:32:21.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:32:21.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903616+0000", 2023-01-15T14:32:21.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013477199999999999 2023-01-15T14:32:21.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:21.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903643+0000", 2023-01-15T14:32:21.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7098999999999999e-05 2023-01-15T14:32:21.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209081+0000", 2023-01-15T14:32:21.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30543790300000001 2023-01-15T14:32:21.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:21.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209213+0000", 2023-01-15T14:32:21.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013155400000000001 2023-01-15T14:32:21.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209600+0000", 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038751699999999998 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209946+0000", 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034601 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.356 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.357 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:21.357 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:32:21.357 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.622606883000003, 2023-01-15T14:32:21.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:32:21.357 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.357 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:21.357 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:21.358 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:21.358 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:21.358 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:32:21.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.358 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:32:21.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:32:21.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:32:21.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:32:21.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:21.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:32:21.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:32:21.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:21.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:32:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:32:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:32:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:32:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:32:21.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:32:21.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:32:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:32:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:32:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:32:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:32:21.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:32:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:32:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:32:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:32:21.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:32:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:32:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:32:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.549 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:32:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.619199760000001, 2023-01-15T14:32:21.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:32:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:21.553 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:32:21.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:32:21.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:32:21.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:32:21.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:32:21.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:32:21.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:21.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:32:21.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:32:21.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:21.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:32:21.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:32:21.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:32:21.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.618984155, 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:32:21.562 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:32:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:32:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:32:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:32:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:32:21.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:21.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:32:21.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:32:21.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:21.571 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:32:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.618877081000001, 2023-01-15T14:32:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:32:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:32:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:32:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:32:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:32:21.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:21.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:32:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:32:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:32:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:32:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:32:21.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:32:21.578 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.488638031000001, 2023-01-15T14:32:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:32:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:32:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:32:21.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:32:21.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:32:21.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:32:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:32:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:32:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:32:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:21.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:32:21.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.488533494000002, 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:32:21.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:32:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:32:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:32:21.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:32:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:32:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:32:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:32:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:32:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:32:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:32:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:32:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:21.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:32:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:32:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:32:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:32:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:32:21.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:32:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:32:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:32:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:32:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 46.488463563000003, 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.598 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:32:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:32:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:32:21.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:32:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:32:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:32:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:32:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:21.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:32:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:32:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:32:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:32:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:32:21.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:32:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:32:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:32:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:32:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2284 3.10 e216/207 3:094fbf13:::smithi09025264-31:head v 216'274, mlcod=212'269)", 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.142898520999999, 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28055179600000002, 2023-01-15T14:32:21.607 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166255+0000", 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166254+0000", 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166261+0000", 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4080000000000003e-06 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166273+0000", 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.221e-05 2023-01-15T14:32:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166279+0000", 2023-01-15T14:32:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8170000000000004e-06 2023-01-15T14:32:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365575+0000", 2023-01-15T14:32:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199295791 2023-01-15T14:32:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365598+0000", 2023-01-15T14:32:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3913e-05 2023-01-15T14:32:21.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365737+0000", 2023-01-15T14:32:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138312 2023-01-15T14:32:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365763+0000", 2023-01-15T14:32:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6596000000000001e-05 2023-01-15T14:32:21.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365803+0000", 2023-01-15T14:32:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9544e-05 2023-01-15T14:32:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365842+0000", 2023-01-15T14:32:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9050000000000001e-05 2023-01-15T14:32:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.446807+0000", 2023-01-15T14:32:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.080964750000000002 2023-01-15T14:32:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2285 3.10 e216/207 3:094fbf13:::smithi09025264-31:head v 216'275, mlcod=212'269)", 2023-01-15T14:32:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:32:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 37.142350489999998, 2023-01-15T14:32:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28017387399999999, 2023-01-15T14:32:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:32:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166803+0000", 2023-01-15T14:32:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166802+0000", 2023-01-15T14:32:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166810+0000", 2023-01-15T14:32:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6650000000000003e-06 2023-01-15T14:32:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166821+0000", 2023-01-15T14:32:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1129e-05 2023-01-15T14:32:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.166827+0000", 2023-01-15T14:32:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4640000000000003e-06 2023-01-15T14:32:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.365957+0000", 2023-01-15T14:32:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199129158 2023-01-15T14:32:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366005+0000", 2023-01-15T14:32:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8822000000000001e-05 2023-01-15T14:32:21.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366301+0000", 2023-01-15T14:32:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029544199999999998 2023-01-15T14:32:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366316+0000", 2023-01-15T14:32:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5078e-05 2023-01-15T14:32:21.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366350+0000", 2023-01-15T14:32:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3575000000000003e-05 2023-01-15T14:32:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.366368+0000", 2023-01-15T14:32:21.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8267e-05 2023-01-15T14:32:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.446977+0000", 2023-01-15T14:32:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.080608964000000005 2023-01-15T14:32:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.4.0:534 3.1a e226/224 3:5919d83e:::smithi09025264-13:a9 v 226'240, mlcod=0'0)", 2023-01-15T14:32:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:32:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.773568887, 2023-01-15T14:32:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30202553599999998, 2023-01-15T14:32:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:32:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:32:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:21.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535584+0000", 2023-01-15T14:32:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535592+0000", 2023-01-15T14:32:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3010000000000007e-06 2023-01-15T14:32:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535604+0000", 2023-01-15T14:32:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1800000000000001e-05 2023-01-15T14:32:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.635708+0000", 2023-01-15T14:32:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10010366800000001 2023-01-15T14:32:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836079+0000", 2023-01-15T14:32:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20037171500000001 2023-01-15T14:32:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836113+0000", 2023-01-15T14:32:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:32:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836714+0000", 2023-01-15T14:32:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060093400000000002 2023-01-15T14:32:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836737+0000", 2023-01-15T14:32:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3073000000000001e-05 2023-01-15T14:32:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836816+0000", 2023-01-15T14:32:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8740999999999997e-05 2023-01-15T14:32:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836898+0000", 2023-01-15T14:32:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2731999999999993e-05 2023-01-15T14:32:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:21.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.837610+0000", 2023-01-15T14:32:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071176200000000001 2023-01-15T14:32:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:21.725 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:21.725 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:21.726 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:32:21.726 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:21.727 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:32:21.727 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:32:21.728 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:32:21.728 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:32:21.728 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:32:21.728 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:32:21.728 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:32:21.728 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:32:21.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:32:21.730 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:20.361+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:21.730 INFO:tasks.rados.rados.0.smithi090.stdout:1006: done (0 left) 2023-01-15T14:32:21.730 INFO:tasks.rados.rados.0.smithi090.stdout:1007: write (excl) oid 35 current snap is 100 2023-01-15T14:32:21.731 INFO:tasks.rados.rados.0.smithi090.stdout:1007: seq_num 373 ranges {669011=569174,2010607=258404} 2023-01-15T14:32:21.731 INFO:tasks.rados.rados.0.smithi090.stdout:1007: writing smithi09025264-35 from 669011 to 1238185 tid 1 2023-01-15T14:32:21.731 INFO:tasks.rados.rados.0.smithi090.stdout:1007: writing smithi09025264-35 from 2010607 to 2269011 tid 2 2023-01-15T14:32:21.731 INFO:tasks.rados.rados.0.smithi090.stdout:1008: snap_create 2023-01-15T14:32:21.731 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:21.731 INFO:tasks.rados.rados.0.smithi090.stdout:1007: finishing write tid 1 to smithi09025264-35 2023-01-15T14:32:21.731 INFO:tasks.rados.rados.0.smithi090.stdout:1007: finishing write tid 2 to smithi09025264-35 2023-01-15T14:32:21.731 INFO:tasks.rados.rados.0.smithi090.stdout:1007: finishing write tid 3 to smithi09025264-35 2023-01-15T14:32:21.731 INFO:tasks.rados.rados.0.smithi090.stdout:1007: finishing write tid 4 to smithi09025264-35 2023-01-15T14:32:21.731 INFO:tasks.rados.rados.0.smithi090.stdout:1007: oid 35 updating version 0 to 281 2023-01-15T14:32:21.732 INFO:tasks.rados.rados.0.smithi090.stdout:1007: oid 35 updating version 281 to 283 2023-01-15T14:32:21.732 INFO:tasks.rados.rados.0.smithi090.stdout:1007: oid 35 version 283 is already newer than 282 2023-01-15T14:32:21.732 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 283 (ObjNum 373 snap 100 seq_num 373) dirty exists 2023-01-15T14:32:21.732 INFO:tasks.rados.rados.0.smithi090.stdout:1007: left oid 35 (ObjNum 373 snap 100 seq_num 373) 2023-01-15T14:32:21.732 INFO:tasks.rados.rados.0.smithi090.stdout:1007: done (0 left) 2023-01-15T14:32:21.772 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:21.766+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:21.831 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:21.825+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:21.948 INFO:tasks.rados.rados.0.smithi090.stdout:1008: done (0 left) 2023-01-15T14:32:21.948 INFO:tasks.rados.rados.0.smithi090.stdout:1009: snap_create 2023-01-15T14:32:21.949 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:21.949 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:21.949 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:32:21.949 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:21.949 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:32:21.950 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:21.941+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:22.051 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:22.045+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:22.127 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:32:22.127 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:32:22.128 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:22.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:32:22.150 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:22.145+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:22.233 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:32:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:32:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:32:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:32:22.283 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 235.11374002400001, 2023-01-15T14:32:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:32:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:32:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:32:22.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:32:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:32:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:32:22.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:32:22.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:32:22.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:32:22.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:32:22.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:32:22.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:32:22.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:32:22.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:32:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:32:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:32:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:32:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:32:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:32:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 235.10074794499999, 2023-01-15T14:32:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:32:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.304 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.304 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:32:22.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.304 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:32:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:32:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:32:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:32:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:32:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:32:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:32:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:32:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:32:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:32:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:32:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:32:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:32:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:32:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:32:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:32:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:32:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:32:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:32:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:32:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:22.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:32:22.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:32:22.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:22.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:32:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:32:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:32:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:32:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:32:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:32:22.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:32:22.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:32:22.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:32:22.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:32:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:32:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:32:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:32:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:32:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:32:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 235.10034460899999, 2023-01-15T14:32:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:32:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:32:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:32:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:32:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:32:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:32:22.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:32:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:32:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:32:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:32:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:22.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:32:22.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:32:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:32:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:32:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:32:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:32:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 235.00012098900001, 2023-01-15T14:32:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:32:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.701 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:32:22.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:32:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:32:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:32:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:32:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:32:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.30680576399999, 2023-01-15T14:32:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:32:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:32:22.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:32:22.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:32:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:32:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:32:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:32:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.30034441000001, 2023-01-15T14:32:22.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:32:22.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:32:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:32:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:32:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:32:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:32:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:32:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:32:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:32:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:32:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:32:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.199922485, 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:32:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:32:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:32:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:32:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:32:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:32:22.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.748 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:22.748 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:32:22.748 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.199584999, 2023-01-15T14:32:22.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:32:22.748 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.748 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:22.748 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.748 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.748 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.748 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:32:22.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:32:22.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:32:22.753 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.753 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.753 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.753 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:22.753 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:32:22.753 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.199529103, 2023-01-15T14:32:22.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:32:22.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:32:22.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:32:22.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:32:22.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:32:22.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:32:22.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.758 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1690 3.16 3:68fb3263:::smithi09025264-33:head [write 1763227~472137] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 102.366596423, 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72570748100000004, 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1690 2023-01-15T14:32:22.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862514+0000", 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862746+0000", 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000232301 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862770+0000", 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3286e-05 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862779+0000", 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4199999999999996e-06 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862844+0000", 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5036999999999994e-05 2023-01-15T14:32:22.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862860+0000", 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6426000000000002e-05 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051251+0000", 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18839023299999999 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051466+0000", 2023-01-15T14:32:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021523199999999999 2023-01-15T14:32:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368326+0000", 2023-01-15T14:32:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31686002699999999 2023-01-15T14:32:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368559+0000", 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023305400000000001 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368935+0000", 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037579099999999999 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369531+0000", 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059593199999999997 2023-01-15T14:32:22.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369571+0000", 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0649999999999999e-05 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370331+0000", 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075969899999999996 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371230+0000", 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000899134 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472895+0000", 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101665191 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.587914+0000", 2023-01-15T14:32:22.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115018671 2023-01-15T14:32:22.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:22.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588013+0000", 2023-01-15T14:32:22.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8494999999999998e-05 2023-01-15T14:32:22.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588223+0000", 2023-01-15T14:32:22.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210294 2023-01-15T14:32:22.768 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1691 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 2235364] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 102.366312169, 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72593841699999995, 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.769 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.770 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1691 2023-01-15T14:32:22.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.770 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:32:22.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862799+0000", 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862807+0000", 2023-01-15T14:32:22.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1280000000000008e-06 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862821+0000", 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.345e-05 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862826+0000", 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3720000000000004e-06 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862883+0000", 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6589000000000003e-05 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862888+0000", 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4060000000000004e-06 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151601+0000", 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28871354100000002 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151732+0000", 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013032300000000001 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369729+0000", 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217997515 2023-01-15T14:32:22.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369949+0000", 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219664 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370367+0000", 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041822799999999998 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370771+0000", 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040358400000000002 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370794+0000", 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-05 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370880+0000", 2023-01-15T14:32:22.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6020999999999995e-05 2023-01-15T14:32:22.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:22.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371241+0000", 2023-01-15T14:32:22.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000360956 2023-01-15T14:32:22.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472994+0000", 2023-01-15T14:32:22.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101753089 2023-01-15T14:32:22.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588348+0000", 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11535382299999999 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588539+0000", 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019090399999999999 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588738+0000", 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019910800000000001 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 102.366278685, 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:32:22.779 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:32:22.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:32:22.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:32:22.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:32:22.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:32:22.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:32:22.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:32:22.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:32:22.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:32:22.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:32:22.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:32:22.786 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2206 3.5 3:a1c47f99:::smithi09025264-2:head [write 1667661~418109] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.830633+0000", 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.398478701999998, 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70338831700000004, 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.787 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2206 2023-01-15T14:32:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830633+0000", 2023-01-15T14:32:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830633+0000", 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830631+0000", 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830800+0000", 2023-01-15T14:32:22.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016843000000000001 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830813+0000", 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3392e-05 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830819+0000", 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6099999999999997e-06 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927297+0000", 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096477775000000002 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927462+0000", 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016500799999999999 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.224878+0000", 2023-01-15T14:32:22.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29741604399999999 2023-01-15T14:32:22.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.224989+0000", 2023-01-15T14:32:22.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011082300000000001 2023-01-15T14:32:22.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:22.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225160+0000", 2023-01-15T14:32:22.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017161699999999999 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225399+0000", 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238645 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226023+0000", 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062401000000000004 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226469+0000", 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044639400000000002 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227365+0000", 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089538200000000001 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429122+0000", 2023-01-15T14:32:22.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20175698 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.533830+0000", 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104708068 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.533947+0000", 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117218 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.534021+0000", 2023-01-15T14:32:22.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4608000000000001e-05 2023-01-15T14:32:22.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.796 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.796 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2207 3.5 3:a1c47f99:::smithi09025264-2:head [write 2548663~763022] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:22.796 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:32:22.796 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.391520956999997, 2023-01-15T14:32:22.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79657873300000004, 2023-01-15T14:32:22.797 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.797 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:22.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2207 2023-01-15T14:32:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837590+0000", 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:22.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837960+0000", 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037055900000000001 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837980+0000", 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9953e-05 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837988+0000", 2023-01-15T14:32:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2800000000000003e-06 2023-01-15T14:32:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927512+0000", 2023-01-15T14:32:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089523563 2023-01-15T14:32:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927645+0000", 2023-01-15T14:32:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013285799999999999 2023-01-15T14:32:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225466+0000", 2023-01-15T14:32:22.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29782133100000002 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225553+0000", 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6866000000000005e-05 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225708+0000", 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015455199999999999 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225971+0000", 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026317099999999998 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226027+0000", 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6601000000000002e-05 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:22.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226470+0000", 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044267500000000001 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227372+0000", 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090215200000000003 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429314+0000", 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20194183499999999 2023-01-15T14:32:22.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.633999+0000", 2023-01-15T14:32:22.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20468505400000001 2023-01-15T14:32:22.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:22.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634075+0000", 2023-01-15T14:32:22.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:32:22.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634170+0000", 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4821999999999994e-05 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2208 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (56) in=7b,truncate 3395047] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.391098364000001, 2023-01-15T14:32:22.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79650439900000003, 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2208 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:32:22.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838018+0000", 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2549999999999997e-06 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838031+0000", 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2717999999999999e-05 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838036+0000", 2023-01-15T14:32:22.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2240000000000003e-06 2023-01-15T14:32:22.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927676+0000", 2023-01-15T14:32:22.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089639620000000003 2023-01-15T14:32:22.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927804+0000", 2023-01-15T14:32:22.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012832300000000001 2023-01-15T14:32:22.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226096+0000", 2023-01-15T14:32:22.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29829207400000002 2023-01-15T14:32:22.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226347+0000", 2023-01-15T14:32:22.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025084800000000002 2023-01-15T14:32:22.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:22.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226750+0000", 2023-01-15T14:32:22.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040343200000000001 2023-01-15T14:32:22.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:22.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227147+0000", 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039682900000000002 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227600+0000", 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045321199999999998 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227859+0000", 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259041 2023-01-15T14:32:22.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227895+0000", 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5710999999999997e-05 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429405+0000", 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201509415 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634300+0000", 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20489528200000001 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634409+0000", 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109524 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634518+0000", 2023-01-15T14:32:22.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108548 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2209 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.391069549000001, 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79724995499999995, 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.920 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2209 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:32:22.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:32:22.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:22.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838044+0000", 2023-01-15T14:32:22.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6120000000000001e-06 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838056+0000", 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1236000000000001e-05 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838059+0000", 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6940000000000001e-06 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.027908+0000", 2023-01-15T14:32:22.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189848287 2023-01-15T14:32:22.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.028049+0000", 2023-01-15T14:32:22.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014132500000000001 2023-01-15T14:32:22.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227252+0000", 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19920324 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227341+0000", 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8683000000000005e-05 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634637+0000", 2023-01-15T14:32:22.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729567300000002 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634980+0000", 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034335000000000002 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.635292+0000", 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003123 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.926 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.927 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2221 3.c 3:3137f0cb:::smithi09025264-20:head [write 2126143~561767] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:22.927 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.853714+0000", 2023-01-15T14:32:22.927 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.375398019999999, 2023-01-15T14:32:22.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.67651091299999999, 2023-01-15T14:32:22.927 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.927 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:22.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.927 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2221 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.853714+0000", 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.853714+0000", 2023-01-15T14:32:22.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.853711+0000", 2023-01-15T14:32:22.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:22.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.853912+0000", 2023-01-15T14:32:22.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020081299999999999 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.853945+0000", 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2845000000000001e-05 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.853958+0000", 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2838e-05 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025255+0000", 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17129745399999999 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025351+0000", 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5748000000000006e-05 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225456+0000", 2023-01-15T14:32:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20010534799999999 2023-01-15T14:32:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225690+0000", 2023-01-15T14:32:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233909 2023-01-15T14:32:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226008+0000", 2023-01-15T14:32:22.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031801699999999997 2023-01-15T14:32:22.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:22.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226628+0000", 2023-01-15T14:32:22.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00061952700000000003 2023-01-15T14:32:22.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:22.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226661+0000", 2023-01-15T14:32:22.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3015999999999997e-05 2023-01-15T14:32:22.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227598+0000", 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093671100000000001 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228528+0000", 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093064000000000005 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429314+0000", 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20078611399999999 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.530028+0000", 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100714044 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.530146+0000", 2023-01-15T14:32:22.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117211 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.530225+0000", 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9145000000000002e-05 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2222 3.c 3:3137f0cb:::smithi09025264-20:head [write 3259733~152121] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:22.936 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:32:22.937 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.371217989999998, 2023-01-15T14:32:22.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77651572400000002, 2023-01-15T14:32:22.937 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.937 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:22.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.937 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2222 2023-01-15T14:32:22.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.937 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857891+0000", 2023-01-15T14:32:22.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:32:22.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857993+0000", 2023-01-15T14:32:22.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010276399999999999 2023-01-15T14:32:22.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858021+0000", 2023-01-15T14:32:22.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8022e-05 2023-01-15T14:32:22.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858034+0000", 2023-01-15T14:32:22.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2549e-05 2023-01-15T14:32:22.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025371+0000", 2023-01-15T14:32:22.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16733653100000001 2023-01-15T14:32:22.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025429+0000", 2023-01-15T14:32:22.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8922000000000003e-05 2023-01-15T14:32:22.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226869+0000", 2023-01-15T14:32:22.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20143931100000001 2023-01-15T14:32:22.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227088+0000", 2023-01-15T14:32:22.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219328 2023-01-15T14:32:22.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:22.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227443+0000", 2023-01-15T14:32:22.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035481 2023-01-15T14:32:22.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227830+0000", 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000386702 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227861+0000", 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1584000000000001e-05 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227939+0000", 2023-01-15T14:32:22.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7354999999999994e-05 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228548+0000", 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060954800000000001 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429817+0000", 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20126904600000001 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634125+0000", 2023-01-15T14:32:22.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20430780300000001 2023-01-15T14:32:22.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:22.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634259+0000", 2023-01-15T14:32:22.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134144 2023-01-15T14:32:22.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634410+0000", 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150447 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2223 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (57) in=7b,truncate 3411854] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.371059203000002, 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87671107599999998, 2023-01-15T14:32:22.946 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2223 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:32:22.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858052+0000", 2023-01-15T14:32:22.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:22.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:22.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858056+0000", 2023-01-15T14:32:22.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5619999999999997e-06 2023-01-15T14:32:22.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:22.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858067+0000", 2023-01-15T14:32:22.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0328e-05 2023-01-15T14:32:22.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858071+0000", 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6439999999999999e-06 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025446+0000", 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16737407200000001 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025503+0000", 2023-01-15T14:32:22.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7952999999999999e-05 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227970+0000", 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202466866 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228139+0000", 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016889100000000001 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:22.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228284+0000", 2023-01-15T14:32:22.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014493700000000001 2023-01-15T14:32:22.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:22.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228470+0000", 2023-01-15T14:32:22.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185541 2023-01-15T14:32:22.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228481+0000", 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.167e-05 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228537+0000", 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5912999999999998e-05 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228563+0000", 2023-01-15T14:32:22.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5613e-05 2023-01-15T14:32:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.433181+0000", 2023-01-15T14:32:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20461838099999999 2023-01-15T14:32:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734620+0000", 2023-01-15T14:32:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301438386 2023-01-15T14:32:22.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734682+0000", 2023-01-15T14:32:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2373999999999996e-05 2023-01-15T14:32:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734764+0000", 2023-01-15T14:32:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1656999999999994e-05 2023-01-15T14:32:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.955 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:22.956 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:22.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.956 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2224 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:22.956 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:32:22.956 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.371034152999997, 2023-01-15T14:32:22.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97704279199999999, 2023-01-15T14:32:22.956 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:22.956 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:22.956 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:22.956 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2224 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:22.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:32:22.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:22.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:22.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:22.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:22.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858077+0000", 2023-01-15T14:32:22.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:22.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:23.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:23.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:23.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858080+0000", 2023-01-15T14:32:23.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858089+0000", 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3449999999999995e-06 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858093+0000", 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9820000000000002e-06 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:23.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:23.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:23.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025520+0000", 2023-01-15T14:32:23.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.167426556 2023-01-15T14:32:23.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:23.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:23.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:23.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025594+0000", 2023-01-15T14:32:23.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4339999999999996e-05 2023-01-15T14:32:23.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:23.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228608+0000", 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20301395699999999 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228752+0000", 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014385 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:23.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834878+0000", 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60612591500000002 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834987+0000", 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108733 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.835120+0000", 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133688 2023-01-15T14:32:23.106 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:23.107 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:23.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:23.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:23.107 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:23.107 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:23.109 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:32:23.109 INFO:tasks.rados.rados.0.smithi090.stdout:1009: done (0 left) 2023-01-15T14:32:23.110 INFO:tasks.rados.rados.0.smithi090.stdout:1010: write oid 41 current snap is 102 2023-01-15T14:32:23.110 INFO:tasks.rados.rados.0.smithi090.stdout:1010: seq_num 374 ranges {685818=644423,2069089=216729} 2023-01-15T14:32:23.110 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:32:23.111 INFO:tasks.rados.rados.0.smithi090.stdout:1010: writing smithi09025264-41 from 685818 to 1330241 tid 1 2023-01-15T14:32:23.111 INFO:tasks.rados.rados.0.smithi090.stdout:1010: writing smithi09025264-41 from 2069089 to 2285818 tid 2 2023-01-15T14:32:23.111 INFO:tasks.rados.rados.0.smithi090.stdout:1011: write (excl) oid 32 current snap is 102 2023-01-15T14:32:23.111 INFO:tasks.rados.rados.0.smithi090.stdout:1011: seq_num 375 ranges {702625=719672,2127571=175054} 2023-01-15T14:32:23.112 INFO:tasks.rados.rados.0.smithi090.stdout:1011: writing smithi09025264-32 from 702625 to 1422297 tid 1 2023-01-15T14:32:23.112 INFO:tasks.rados.rados.0.smithi090.stdout:1011: writing smithi09025264-32 from 2127571 to 2302625 tid 2 2023-01-15T14:32:23.112 INFO:tasks.rados.rados.0.smithi090.stdout:1012: read oid 7 snap -1 2023-01-15T14:32:23.112 INFO:tasks.rados.rados.0.smithi090.stdout:1012: expect deleted 2023-01-15T14:32:23.112 INFO:tasks.rados.rados.0.smithi090.stdout:1013: snap_remove snap 101 2023-01-15T14:32:23.113 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:23.105+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:23.210 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:23.205+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:23.313 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:23.313 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:32:23.313 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:32:23.314 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:23.314 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:32:23.314 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:23.306+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:23.414 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:23.409+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:23.513 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:23.513 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:32:23.514 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:32:23.514 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:32:23.514 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:23.515 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:32:23.517 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:23.510+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:23.618 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:23.613+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:23.717 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:23.717 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:32:23.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:32:23.718 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:32:23.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.718 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3 3.3 3:ca59e2b0:::smithi09025264-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:32:23.718 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:32:23.718 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 237.89354315400001, 2023-01-15T14:32:23.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62968594499999997, 2023-01-15T14:32:23.718 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:23.718 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:23.718 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:23.718 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:23.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:32:23.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:23.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:23.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775131+0000", 2023-01-15T14:32:23.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:23.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:23.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775138+0000", 2023-01-15T14:32:23.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:32:23.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:23.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775148+0000", 2023-01-15T14:32:23.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7969999999999995e-06 2023-01-15T14:32:23.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:23.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775161+0000", 2023-01-15T14:32:23.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2469999999999999e-05 2023-01-15T14:32:23.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:23.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.857974+0000", 2023-01-15T14:32:23.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.082812887000000002 2023-01-15T14:32:23.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:23.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.858147+0000", 2023-01-15T14:32:23.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000173883 2023-01-15T14:32:23.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:23.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106098+0000", 2023-01-15T14:32:23.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24795019600000001 2023-01-15T14:32:23.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:23.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106181+0000", 2023-01-15T14:32:23.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3248e-05 2023-01-15T14:32:23.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:32:23.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106330+0000", 2023-01-15T14:32:23.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014918200000000001 2023-01-15T14:32:23.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:23.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106510+0000", 2023-01-15T14:32:23.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017979399999999999 2023-01-15T14:32:23.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:23.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106527+0000", 2023-01-15T14:32:23.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6643999999999999e-05 2023-01-15T14:32:23.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:23.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106605+0000", 2023-01-15T14:32:23.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8650000000000001e-05 2023-01-15T14:32:23.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:23.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106644+0000", 2023-01-15T14:32:23.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8686999999999997e-05 2023-01-15T14:32:23.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:23.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404687+0000", 2023-01-15T14:32:23.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.298042745 2023-01-15T14:32:23.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:23.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404735+0000", 2023-01-15T14:32:23.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8306000000000001e-05 2023-01-15T14:32:23.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:23.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404772+0000", 2023-01-15T14:32:23.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6977000000000002e-05 2023-01-15T14:32:23.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.732 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:23.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404818+0000", 2023-01-15T14:32:23.732 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.6439e-05 2023-01-15T14:32:23.732 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:23.732 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:23.732 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:23.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.733 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:32:23.733 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:32:23.733 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 237.89350508699999, 2023-01-15T14:32:23.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62995383500000002, 2023-01-15T14:32:23.733 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:23.733 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:23.733 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:23.733 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:23.734 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:23.734 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4 2023-01-15T14:32:23.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.734 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:23.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.734 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:23.734 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:32:23.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:23.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.735 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.735 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:23.735 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:32:23.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:23.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.735 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.735 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:23.736 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:32:23.736 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:23.736 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.736 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.736 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:23.736 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775174+0000", 2023-01-15T14:32:23.736 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3669999999999999e-06 2023-01-15T14:32:23.736 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.736 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.736 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:23.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775184+0000", 2023-01-15T14:32:23.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5869999999999999e-06 2023-01-15T14:32:23.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:23.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775190+0000", 2023-01-15T14:32:23.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:32:23.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:23.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868440+0000", 2023-01-15T14:32:23.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.093250126000000003 2023-01-15T14:32:23.738 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:23.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:23.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:23.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868555+0000", 2023-01-15T14:32:23.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115003 2023-01-15T14:32:24.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.302962+0000", 2023-01-15T14:32:24.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.43440729700000003 2023-01-15T14:32:24.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:24.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303197+0000", 2023-01-15T14:32:24.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023508000000000001 2023-01-15T14:32:24.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404866+0000", 2023-01-15T14:32:24.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101668723 2023-01-15T14:32:24.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:24.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404985+0000", 2023-01-15T14:32:24.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118853 2023-01-15T14:32:24.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:24.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.405124+0000", 2023-01-15T14:32:24.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013951999999999999 2023-01-15T14:32:24.134 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.134 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:24.135 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.136 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:144 3.9 3:9988ac9d:::smithi09025264-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:24.136 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:32:24.136 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 236.80216654099999, 2023-01-15T14:32:24.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63423976199999998, 2023-01-15T14:32:24.136 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:24.136 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:24.136 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:24.136 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:24.137 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:24.137 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 144 2023-01-15T14:32:24.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.137 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:24.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:24.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:32:24.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:24.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:32:24.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:24.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866508+0000", 2023-01-15T14:32:24.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:24.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:24.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866512+0000", 2023-01-15T14:32:24.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5999999999999998e-06 2023-01-15T14:32:24.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:24.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866523+0000", 2023-01-15T14:32:24.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0740000000000001e-05 2023-01-15T14:32:24.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.140 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:24.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866531+0000", 2023-01-15T14:32:24.140 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3310000000000004e-06 2023-01-15T14:32:24.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.140 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867744+0000", 2023-01-15T14:32:24.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012127399999999999 2023-01-15T14:32:24.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:24.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867810+0000", 2023-01-15T14:32:24.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6506000000000003e-05 2023-01-15T14:32:24.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097225+0000", 2023-01-15T14:32:24.142 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.22941514900000001 2023-01-15T14:32:24.142 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.142 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.142 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:24.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097295+0000", 2023-01-15T14:32:24.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:32:24.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.146 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.146 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500386+0000", 2023-01-15T14:32:24.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40309051699999998 2023-01-15T14:32:24.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.146 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:24.146 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500521+0000", 2023-01-15T14:32:24.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013561700000000001 2023-01-15T14:32:24.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:24.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500749+0000", 2023-01-15T14:32:24.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022754900000000001 2023-01-15T14:32:24.147 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.147 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:24.147 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.148 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:211 3.9 3:9e7635b9:::smithi09025264-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:24.148 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.510639+0000", 2023-01-15T14:32:24.148 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 236.158036622, 2023-01-15T14:32:24.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60010284700000005, 2023-01-15T14:32:24.148 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:24.148 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:24.148 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:24.148 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:24.149 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:24.149 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 211 2023-01-15T14:32:24.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.149 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:24.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:24.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510639+0000", 2023-01-15T14:32:24.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:24.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510639+0000", 2023-01-15T14:32:24.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:24.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510638+0000", 2023-01-15T14:32:24.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:24.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:24.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510643+0000", 2023-01-15T14:32:24.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3340000000000003e-06 2023-01-15T14:32:24.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:24.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510653+0000", 2023-01-15T14:32:24.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0961999999999999e-05 2023-01-15T14:32:24.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:24.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.510663+0000", 2023-01-15T14:32:24.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4800000000000007e-06 2023-01-15T14:32:24.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.601338+0000", 2023-01-15T14:32:24.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090674642 2023-01-15T14:32:24.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:24.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.601480+0000", 2023-01-15T14:32:24.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014286299999999999 2023-01-15T14:32:24.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708540+0000", 2023-01-15T14:32:24.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.107059404 2023-01-15T14:32:24.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:24.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.708676+0000", 2023-01-15T14:32:24.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013605 2023-01-15T14:32:24.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110423+0000", 2023-01-15T14:32:24.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40174706199999999 2023-01-15T14:32:24.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:24.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110590+0000", 2023-01-15T14:32:24.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000166773 2023-01-15T14:32:24.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:24.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:28.110742+0000", 2023-01-15T14:32:24.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000152022 2023-01-15T14:32:24.156 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.156 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:24.156 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.156 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:508 3.9 3:9f86bdcb:::smithi09025264-22:head [write 1808051~636013] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:32:24.156 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.809136+0000", 2023-01-15T14:32:24.157 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 213.85953917699999, 2023-01-15T14:32:24.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60160218099999996, 2023-01-15T14:32:24.157 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:24.157 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:24.157 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:24.157 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:24.157 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:24.157 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 508 2023-01-15T14:32:24.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.157 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:24.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:24.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809136+0000", 2023-01-15T14:32:24.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:24.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809136+0000", 2023-01-15T14:32:24.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:24.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809135+0000", 2023-01-15T14:32:24.311 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:24.311 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.311 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.311 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:24.311 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809776+0000", 2023-01-15T14:32:24.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00064178800000000002 2023-01-15T14:32:24.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:24.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809801+0000", 2023-01-15T14:32:24.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4851999999999999e-05 2023-01-15T14:32:24.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:24.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.809811+0000", 2023-01-15T14:32:24.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0285000000000001e-05 2023-01-15T14:32:24.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.903875+0000", 2023-01-15T14:32:24.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.094063472999999995 2023-01-15T14:32:24.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:24.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.903888+0000", 2023-01-15T14:32:24.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.27e-05 2023-01-15T14:32:24.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906243+0000", 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023554980000000001 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906404+0000", 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000160705 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.207137+0000", 2023-01-15T14:32:24.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30073284700000003 2023-01-15T14:32:24.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:24.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.207316+0000", 2023-01-15T14:32:24.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000179125 2023-01-15T14:32:24.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:32:24.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.207563+0000", 2023-01-15T14:32:24.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024729599999999999 2023-01-15T14:32:24.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:24.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208190+0000", 2023-01-15T14:32:24.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062669899999999998 2023-01-15T14:32:24.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:24.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208208+0000", 2023-01-15T14:32:24.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7759999999999999e-05 2023-01-15T14:32:24.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:24.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208503+0000", 2023-01-15T14:32:24.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000295109 2023-01-15T14:32:24.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311080+0000", 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10257740899999999 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311363+0000", 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028312299999999998 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.410431+0000", 2023-01-15T14:32:24.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099068120999999995 2023-01-15T14:32:24.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:24.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.410590+0000", 2023-01-15T14:32:24.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015870700000000001 2023-01-15T14:32:24.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:24.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.410739+0000", 2023-01-15T14:32:24.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014854899999999999 2023-01-15T14:32:24.320 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.321 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:24.321 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.321 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:509 3.9 3:9f86bdcb:::smithi09025264-22:head [write 2873212~588997] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:32:24.321 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:32:24.321 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 213.85374219799999, 2023-01-15T14:32:24.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69623497099999998, 2023-01-15T14:32:24.321 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:24.321 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:24.321 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 509 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:24.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:32:24.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:24.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814931+0000", 2023-01-15T14:32:24.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:24.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:24.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815512+0000", 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058082200000000004 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815553+0000", 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1081999999999997e-05 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815571+0000", 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8028000000000001e-05 2023-01-15T14:32:24.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906452+0000", 2023-01-15T14:32:24.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090880586999999999 2023-01-15T14:32:24.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:24.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906564+0000", 2023-01-15T14:32:24.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111976 2023-01-15T14:32:24.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208424+0000", 2023-01-15T14:32:24.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.301860199 2023-01-15T14:32:24.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:24.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208718+0000", 2023-01-15T14:32:24.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029347299999999999 2023-01-15T14:32:24.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:32:24.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209175+0000", 2023-01-15T14:32:24.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045692399999999999 2023-01-15T14:32:24.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:24.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209725+0000", 2023-01-15T14:32:24.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055027600000000004 2023-01-15T14:32:24.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:24.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209751+0000", 2023-01-15T14:32:24.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5812999999999998e-05 2023-01-15T14:32:24.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:24.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.210006+0000", 2023-01-15T14:32:24.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025564099999999997 2023-01-15T14:32:24.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:24.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311247+0000", 2023-01-15T14:32:24.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10124119199999999 2023-01-15T14:32:24.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:24.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311454+0000", 2023-01-15T14:32:24.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020695800000000001 2023-01-15T14:32:24.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:24.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511022+0000", 2023-01-15T14:32:24.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19956729200000001 2023-01-15T14:32:24.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:24.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511100+0000", 2023-01-15T14:32:24.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8589000000000003e-05 2023-01-15T14:32:24.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:24.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511168+0000", 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8039000000000004e-05 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:510 3.9 3:9f86bdcb:::smithi09025264-22:head [setxattr _header (57) in=7b,truncate 3462209] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 213.853078735, 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69609656399999997, 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:24.477 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 510 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:32:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815596+0000", 2023-01-15T14:32:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:24.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815606+0000", 2023-01-15T14:32:24.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7920000000000004e-06 2023-01-15T14:32:24.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:24.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815627+0000", 2023-01-15T14:32:24.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1239000000000001e-05 2023-01-15T14:32:24.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:24.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815637+0000", 2023-01-15T14:32:24.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0349e-05 2023-01-15T14:32:24.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906600+0000", 2023-01-15T14:32:24.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090963098000000006 2023-01-15T14:32:24.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:24.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906717+0000", 2023-01-15T14:32:24.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000116759 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310000+0000", 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40328322599999999 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310249+0000", 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024874400000000002 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:32:24.483 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310559+0000", 2023-01-15T14:32:24.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310382 2023-01-15T14:32:24.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.484 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.484 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:24.484 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310951+0000", 2023-01-15T14:32:24.484 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039124500000000003 2023-01-15T14:32:24.484 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.484 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.484 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:24.484 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310975+0000", 2023-01-15T14:32:24.484 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4496000000000001e-05 2023-01-15T14:32:24.484 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.484 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311041+0000", 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6309000000000005e-05 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311260+0000", 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021863399999999999 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311588+0000", 2023-01-15T14:32:24.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032813599999999998 2023-01-15T14:32:24.486 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.486 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.486 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:24.486 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511351+0000", 2023-01-15T14:32:24.486 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19976275700000001 2023-01-15T14:32:24.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.487 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.487 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:24.487 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511507+0000", 2023-01-15T14:32:24.487 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015585100000000001 2023-01-15T14:32:24.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511693+0000", 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186627 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:511 3.9 3:9f86bdcb:::smithi09025264-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:32:24.488 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 213.853021367, 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69666647500000001, 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 511 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:32:24.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.490 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.490 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.490 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:24.490 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:32:24.490 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.490 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.490 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.490 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:24.490 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815653+0000", 2023-01-15T14:32:24.490 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:24.491 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.491 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.491 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:24.491 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815660+0000", 2023-01-15T14:32:24.492 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0509999999999999e-06 2023-01-15T14:32:24.492 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.492 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.492 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:24.493 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815683+0000", 2023-01-15T14:32:24.493 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2742999999999999e-05 2023-01-15T14:32:24.493 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.493 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.493 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:24.493 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815698+0000", 2023-01-15T14:32:24.493 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5133999999999999e-05 2023-01-15T14:32:24.493 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.493 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.493 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908029+0000", 2023-01-15T14:32:24.494 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.092331448999999996 2023-01-15T14:32:24.494 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.494 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:24.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908193+0000", 2023-01-15T14:32:24.494 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016342799999999999 2023-01-15T14:32:24.494 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.494 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311115+0000", 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40292222300000002 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311224+0000", 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010888899999999999 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511800+0000", 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20057596899999999 2023-01-15T14:32:24.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.496 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.496 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:24.496 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512036+0000", 2023-01-15T14:32:24.496 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236247 2023-01-15T14:32:24.496 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.496 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.496 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:24.496 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512321+0000", 2023-01-15T14:32:24.496 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028457599999999998 2023-01-15T14:32:24.496 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.496 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:692 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~2016807,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898357+0000", 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 195.770318914, 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60411950800000003, 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:24.497 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:24.498 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 692 2023-01-15T14:32:24.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.498 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:24.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.498 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:24.498 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898357+0000", 2023-01-15T14:32:24.498 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.498 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:24.498 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898357+0000", 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898356+0000", 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898360+0000", 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2740000000000001e-06 2023-01-15T14:32:24.499 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.500 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.500 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:24.500 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898373+0000", 2023-01-15T14:32:24.500 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.216e-05 2023-01-15T14:32:24.500 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.500 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.500 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:24.500 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898381+0000", 2023-01-15T14:32:24.500 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.225e-06 2023-01-15T14:32:24.500 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.500 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.500519+0000", 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60213839400000002 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.500676+0000", 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000156348 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:24.501 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.502476+0000", 2023-01-15T14:32:24.502 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001800638 2023-01-15T14:32:24.502 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.502 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:24.502 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.502 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.502 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:693 3.3 3:ca59e2b0:::smithi09025264-1:head [sparse-read 0~2016807] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:32:24.502 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:32:24.502 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 195.77028802000001, 2023-01-15T14:32:24.502 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70514199600000005, 2023-01-15T14:32:24.502 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 693 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:24.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:32:24.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:24.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898387+0000", 2023-01-15T14:32:24.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:24.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:24.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898390+0000", 2023-01-15T14:32:24.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.252e-06 2023-01-15T14:32:24.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:24.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898401+0000", 2023-01-15T14:32:24.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0977e-05 2023-01-15T14:32:24.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:24.637 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.998532+0000", 2023-01-15T14:32:24.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100130443 2023-01-15T14:32:24.637 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.637 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.637 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602663+0000", 2023-01-15T14:32:24.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60413100500000005 2023-01-15T14:32:24.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:24.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602817+0000", 2023-01-15T14:32:24.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015440500000000001 2023-01-15T14:32:24.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:24.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.603530+0000", 2023-01-15T14:32:24.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071246299999999999 2023-01-15T14:32:24.639 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.640 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:24.640 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:24.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.640 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1294 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3327786] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:32:24.641 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:59.360408+0000", 2023-01-15T14:32:24.641 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 144.30826797200001, 2023-01-15T14:32:24.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60454960700000004, 2023-01-15T14:32:24.641 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:24.641 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:24.641 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:24.642 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:24.642 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:24.642 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1294 2023-01-15T14:32:24.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.642 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:24.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:24.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.360408+0000", 2023-01-15T14:32:24.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:24.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.360408+0000", 2023-01-15T14:32:24.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:24.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:24.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.360405+0000", 2023-01-15T14:32:24.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:24.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:24.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.360417+0000", 2023-01-15T14:32:24.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1953000000000001e-05 2023-01-15T14:32:24.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:24.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.360434+0000", 2023-01-15T14:32:24.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6443000000000002e-05 2023-01-15T14:32:24.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:24.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.460556+0000", 2023-01-15T14:32:24.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100121896 2023-01-15T14:32:24.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:24.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.461178+0000", 2023-01-15T14:32:24.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062218799999999997 2023-01-15T14:32:24.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:24.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.461390+0000", 2023-01-15T14:32:24.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021255499999999999 2023-01-15T14:32:24.650 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.650 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.651 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:32:24.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.461724+0000", 2023-01-15T14:32:24.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033382699999999997 2023-01-15T14:32:24.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:24.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.462215+0000", 2023-01-15T14:32:24.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049096100000000004 2023-01-15T14:32:24.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:24.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:24.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:24.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.462237+0000", 2023-01-15T14:32:25.975 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2235999999999999e-05 2023-01-15T14:32:25.975 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:25.976 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:25.976 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:25.976 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.462342+0000", 2023-01-15T14:32:25.976 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000104411 2023-01-15T14:32:25.976 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:25.976 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:25.976 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:25.976 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.462488+0000", 2023-01-15T14:32:25.977 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000145914 2023-01-15T14:32:25.977 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:25.977 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:25.977 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:25.977 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.680400+0000", 2023-01-15T14:32:25.977 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.217912509 2023-01-15T14:32:25.977 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:25.977 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:25.977 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:25.978 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.964546+0000", 2023-01-15T14:32:25.978 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.28414540500000002 2023-01-15T14:32:25.978 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:25.978 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:25.978 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:25.978 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.964742+0000", 2023-01-15T14:32:25.978 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019644700000000001 2023-01-15T14:32:25.978 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:25.978 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:25.979 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:25.979 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:59.964957+0000", 2023-01-15T14:32:25.979 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021501199999999999 2023-01-15T14:32:25.979 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:25.979 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:25.979 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:26.176 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.176 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.176 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1421 3.a 3:5919d83e:::smithi09025264-13:head [write 1723287~549075] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:32:26.176 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:32:26.176 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 128.75637351099999, 2023-01-15T14:32:26.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61118175699999999, 2023-01-15T14:32:26.177 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:26.177 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:26.177 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:26.177 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:26.177 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:26.178 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1421 2023-01-15T14:32:26.178 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.178 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:26.179 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.179 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:26.179 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:32:26.179 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:26.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.179 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.180 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:26.180 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:32:26.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:26.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:26.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912300+0000", 2023-01-15T14:32:26.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913127+0000", 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00082704000000000002 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913158+0000", 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1146999999999998e-05 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:26.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913177+0000", 2023-01-15T14:32:26.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:32:26.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:26.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913384+0000", 2023-01-15T14:32:26.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000206589 2023-01-15T14:32:26.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:26.184 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913408+0000", 2023-01-15T14:32:26.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3493e-05 2023-01-15T14:32:26.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.184 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.184 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:26.379 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318579+0000", 2023-01-15T14:32:26.379 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40517134399999999 2023-01-15T14:32:26.379 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:26.380 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318806+0000", 2023-01-15T14:32:26.380 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022680899999999999 2023-01-15T14:32:26.380 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.380 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:32:26.383 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319060+0000", 2023-01-15T14:32:26.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254069 2023-01-15T14:32:26.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:26.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319527+0000", 2023-01-15T14:32:26.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000467081 2023-01-15T14:32:26.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.384 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:26.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319544+0000", 2023-01-15T14:32:26.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6949999999999999e-05 2023-01-15T14:32:26.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:26.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319778+0000", 2023-01-15T14:32:26.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023391500000000001 2023-01-15T14:32:26.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:26.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421126+0000", 2023-01-15T14:32:26.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101348626 2023-01-15T14:32:26.386 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.386 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:26.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522834+0000", 2023-01-15T14:32:26.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10170761 2023-01-15T14:32:26.386 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.386 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:26.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523142+0000", 2023-01-15T14:32:26.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854800000000001 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523251+0000", 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108411 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523484+0000", 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023292500000000001 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:26.387 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:26.388 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:26.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.388 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1422 3.a 3:5919d83e:::smithi09025264-13:head [write 2781837~703343] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:32:26.388 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:32:26.388 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 128.75543471500001, 2023-01-15T14:32:26.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61062001700000001, 2023-01-15T14:32:26.388 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:26.388 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:26.388 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:26.388 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1422 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:26.389 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:32:26.390 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:26.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.390 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.390 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:26.390 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913239+0000", 2023-01-15T14:32:26.390 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:26.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.390 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.390 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:26.390 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914075+0000", 2023-01-15T14:32:26.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00083594099999999998 2023-01-15T14:32:26.391 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:26.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914104+0000", 2023-01-15T14:32:26.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8938999999999999e-05 2023-01-15T14:32:26.391 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:26.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914121+0000", 2023-01-15T14:32:26.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7306999999999999e-05 2023-01-15T14:32:26.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.392 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.392 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:26.392 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914234+0000", 2023-01-15T14:32:26.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112737 2023-01-15T14:32:26.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.392 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.392 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:26.393 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914257+0000", 2023-01-15T14:32:26.393 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3156e-05 2023-01-15T14:32:26.393 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.393 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.393 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:26.393 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319798+0000", 2023-01-15T14:32:26.393 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40554153100000001 2023-01-15T14:32:26.393 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.393 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.393 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:26.393 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319991+0000", 2023-01-15T14:32:26.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019217999999999999 2023-01-15T14:32:26.394 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.394 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.394 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:32:26.394 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320237+0000", 2023-01-15T14:32:26.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024621099999999998 2023-01-15T14:32:26.394 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.394 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.394 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:26.394 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320759+0000", 2023-01-15T14:32:26.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052174700000000003 2023-01-15T14:32:26.395 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.395 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.395 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:26.395 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320776+0000", 2023-01-15T14:32:26.395 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7173e-05 2023-01-15T14:32:26.395 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.395 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.395 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:26.395 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.321080+0000", 2023-01-15T14:32:26.395 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030424400000000001 2023-01-15T14:32:26.395 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.396 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.396 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:26.396 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421162+0000", 2023-01-15T14:32:26.396 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100082085 2023-01-15T14:32:26.396 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.396 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.396 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:26.396 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522963+0000", 2023-01-15T14:32:26.396 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10180109399999999 2023-01-15T14:32:26.396 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.396 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.397 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:26.397 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523576+0000", 2023-01-15T14:32:26.397 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061277499999999997 2023-01-15T14:32:26.397 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.397 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.397 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:26.397 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523664+0000", 2023-01-15T14:32:26.397 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8002000000000002e-05 2023-01-15T14:32:26.397 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.397 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.397 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:26.398 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523861+0000", 2023-01-15T14:32:26.398 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019685899999999999 2023-01-15T14:32:26.398 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:26.398 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:26.398 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:26.398 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.398 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1423 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 3630312] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:32:26.398 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:32:26.398 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 128.75451218000001, 2023-01-15T14:32:26.399 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70862486899999999, 2023-01-15T14:32:26.399 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:26.399 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:26.399 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:26.399 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:26.399 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:26.399 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1423 2023-01-15T14:32:26.399 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.399 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:26.399 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.400 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:26.400 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:32:26.400 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:26.400 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.400 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.400 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:26.400 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:32:26.400 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:26.400 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.400 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914162+0000", 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914172+0000", 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0267e-05 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:26.401 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914188+0000", 2023-01-15T14:32:26.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6212999999999999e-05 2023-01-15T14:32:26.402 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914201+0000", 2023-01-15T14:32:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3073e-05 2023-01-15T14:32:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914315+0000", 2023-01-15T14:32:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011357399999999999 2023-01-15T14:32:26.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914334+0000", 2023-01-15T14:32:26.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9004e-05 2023-01-15T14:32:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421258+0000", 2023-01-15T14:32:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.506924606 2023-01-15T14:32:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:26.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421518+0000", 2023-01-15T14:32:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025950499999999999 2023-01-15T14:32:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:32:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421843+0000", 2023-01-15T14:32:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032498299999999999 2023-01-15T14:32:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422222+0000", 2023-01-15T14:32:26.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037867599999999999 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422245+0000", 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3096999999999999e-05 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422343+0000", 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8487999999999996e-05 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:26.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522890+0000", 2023-01-15T14:32:26.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10054658900000001 2023-01-15T14:32:26.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:26.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523075+0000", 2023-01-15T14:32:26.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185386 2023-01-15T14:32:26.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:26.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622151+0000", 2023-01-15T14:32:26.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099075724000000004 2023-01-15T14:32:26.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:26.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622244+0000", 2023-01-15T14:32:26.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3387999999999994e-05 2023-01-15T14:32:26.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:26.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622788+0000", 2023-01-15T14:32:26.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000543973 2023-01-15T14:32:26.594 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:26.594 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:26.594 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:26.594 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.595 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1424 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:32:26.595 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:32:26.595 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 128.75445680799999, 2023-01-15T14:32:26.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70984700199999995, 2023-01-15T14:32:26.595 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:26.595 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:26.595 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:26.596 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:26.596 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:26.596 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1424 2023-01-15T14:32:26.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.596 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:26.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:26.602 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:32:26.602 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:26.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.602 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914218+0000", 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914224+0000", 2023-01-15T14:32:26.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6769999999999998e-06 2023-01-15T14:32:26.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:26.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914242+0000", 2023-01-15T14:32:26.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7759000000000001e-05 2023-01-15T14:32:26.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:26.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014331+0000", 2023-01-15T14:32:26.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088819 2023-01-15T14:32:26.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114642+0000", 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100311027 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114682+0000", 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0028999999999999e-05 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:26.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523918+0000", 2023-01-15T14:32:26.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40923631500000002 2023-01-15T14:32:26.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:26.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524124+0000", 2023-01-15T14:32:26.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020524900000000001 2023-01-15T14:32:26.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:26.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622921+0000", 2023-01-15T14:32:26.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098797069000000001 2023-01-15T14:32:26.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:26.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.623328+0000", 2023-01-15T14:32:26.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000407539 2023-01-15T14:32:26.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.776 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.776 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:26.776 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.624066+0000", 2023-01-15T14:32:26.776 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073762899999999995 2023-01-15T14:32:26.776 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:26.776 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:26.776 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:26.776 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.776 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.776 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1430 3.a 3:52df5138:::smithi09025264-25:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:32:26.776 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:32:26.777 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 128.654297047, 2023-01-15T14:32:26.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81872653100000004, 2023-01-15T14:32:26.777 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:26.777 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:26.777 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:26.777 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:26.777 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:26.778 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1430 2023-01-15T14:32:26.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.778 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:26.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:26.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:32:26.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:26.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014376+0000", 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014391+0000", 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4922999999999999e-05 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014417+0000", 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5698999999999999e-05 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014435+0000", 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7827000000000001e-05 2023-01-15T14:32:26.780 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114759+0000", 2023-01-15T14:32:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100324859 2023-01-15T14:32:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114771+0000", 2023-01-15T14:32:26.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:32:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524222+0000", 2023-01-15T14:32:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40945100800000001 2023-01-15T14:32:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524545+0000", 2023-01-15T14:32:26.992 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000322367 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.626981+0000", 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102436308 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627874+0000", 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089274899999999997 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.993 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627930+0000", 2023-01-15T14:32:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6122000000000002e-05 2023-01-15T14:32:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628609+0000", 2023-01-15T14:32:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00067958700000000005 2023-01-15T14:32:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.994 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628798+0000", 2023-01-15T14:32:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018809 2023-01-15T14:32:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628870+0000", 2023-01-15T14:32:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2844999999999998e-05 2023-01-15T14:32:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:26.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832404+0000", 2023-01-15T14:32:26.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20353310299999999 2023-01-15T14:32:26.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832606+0000", 2023-01-15T14:32:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020207999999999999 2023-01-15T14:32:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833105+0000", 2023-01-15T14:32:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049945399999999998 2023-01-15T14:32:26.997 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1431 3.a 3:52df5138:::smithi09025264-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:32:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:32:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 128.65420575900001, 2023-01-15T14:32:26.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81967516600000001, 2023-01-15T14:32:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:27.197 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1431 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:32:27.198 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014469+0000", 2023-01-15T14:32:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014477+0000", 2023-01-15T14:32:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9300000000000003e-06 2023-01-15T14:32:27.199 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014495+0000", 2023-01-15T14:32:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8363999999999999e-05 2023-01-15T14:32:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114610+0000", 2023-01-15T14:32:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10011521700000001 2023-01-15T14:32:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.200 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214948+0000", 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100337739 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214962+0000", 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4516e-05 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.201 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524874+0000", 2023-01-15T14:32:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30991196599999998 2023-01-15T14:32:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.525013+0000", 2023-01-15T14:32:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013873999999999999 2023-01-15T14:32:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628222+0000", 2023-01-15T14:32:27.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103208721 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628504+0000", 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028245700000000001 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833214+0000", 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.204709691 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833555+0000", 2023-01-15T14:32:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034094000000000003 2023-01-15T14:32:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.834145+0000", 2023-01-15T14:32:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059006900000000003 2023-01-15T14:32:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:27.204 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2439 3.3 3:c11c7d81:::smithi09025264-11:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:32:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:32:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 28.585301254000001, 2023-01-15T14:32:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70719632099999996, 2023-01-15T14:32:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:27.205 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2439 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:27.206 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:32:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083373+0000", 2023-01-15T14:32:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083380+0000", 2023-01-15T14:32:27.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:32:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083391+0000", 2023-01-15T14:32:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0366e-05 2023-01-15T14:32:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083397+0000", 2023-01-15T14:32:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2740000000000004e-06 2023-01-15T14:32:27.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083648+0000", 2023-01-15T14:32:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025052599999999999 2023-01-15T14:32:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083666+0000", 2023-01-15T14:32:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8848999999999999e-05 2023-01-15T14:32:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.209 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237725+0000", 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.154058361 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237945+0000", 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021978300000000001 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.210 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.386967+0000", 2023-01-15T14:32:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.149022562 2023-01-15T14:32:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387230+0000", 2023-01-15T14:32:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026256100000000002 2023-01-15T14:32:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:32:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387695+0000", 2023-01-15T14:32:27.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046480099999999998 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388112+0000", 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041713200000000002 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388130+0000", 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8315999999999999e-05 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:27.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388189+0000", 2023-01-15T14:32:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8665999999999997e-05 2023-01-15T14:32:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388829+0000", 2023-01-15T14:32:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00063992699999999999 2023-01-15T14:32:27.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:27.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.489989+0000", 2023-01-15T14:32:27.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116059199999999 2023-01-15T14:32:27.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:27.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790300+0000", 2023-01-15T14:32:27.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30031108699999998 2023-01-15T14:32:27.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:27.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790396+0000", 2023-01-15T14:32:27.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5721000000000001e-05 2023-01-15T14:32:27.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:27.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790571+0000", 2023-01-15T14:32:27.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017464999999999999 2023-01-15T14:32:27.218 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:27.218 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:27.218 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:27.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.219 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2440 3.3 3:c11c7d81:::smithi09025264-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:32:27.219 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:32:27.219 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 28.585209523, 2023-01-15T14:32:27.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70759506000000005, 2023-01-15T14:32:27.219 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:27.219 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:27.219 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:27.219 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:27.219 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:27.219 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2440 2023-01-15T14:32:27.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.220 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:27.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:27.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:32:27.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:27.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:27.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:32:27.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:27.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:27.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083465+0000", 2023-01-15T14:32:27.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:27.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:27.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083470+0000", 2023-01-15T14:32:27.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2630000000000003e-06 2023-01-15T14:32:27.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083481+0000", 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0699999999999999e-05 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083488+0000", 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8870000000000003e-06 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180331+0000", 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.096843296999999995 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180342+0000", 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.031e-05 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238012+0000", 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.057670829 2023-01-15T14:32:27.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238257+0000", 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000244959 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590126+0000", 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35186851699999999 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:27.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590389+0000", 2023-01-15T14:32:27.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026276799999999998 2023-01-15T14:32:27.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790692+0000", 2023-01-15T14:32:27.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20030350299999999 2023-01-15T14:32:27.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:27.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790891+0000", 2023-01-15T14:32:27.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019893999999999999 2023-01-15T14:32:27.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:27.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.791061+0000", 2023-01-15T14:32:27.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169913 2023-01-15T14:32:27.226 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:27.226 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:27.226 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:27.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2450 3.a 3:52c3e133:::smithi09025264-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e230)", 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 26.821356910999999, 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.68251353100000001, 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2450 2023-01-15T14:32:27.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.228 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:27.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:27.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:32:27.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:27.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:27.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:32:27.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:27.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:27.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847318+0000", 2023-01-15T14:32:27.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:27.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:27.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847324+0000", 2023-01-15T14:32:27.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:32:27.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847339+0000", 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5582000000000001e-05 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847349+0000", 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0139999999999999e-05 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947644+0000", 2023-01-15T14:32:27.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10029455600000001 2023-01-15T14:32:27.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:27.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947657+0000", 2023-01-15T14:32:27.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2796000000000001e-05 2023-01-15T14:32:27.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125638+0000", 2023-01-15T14:32:27.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17798127799999999 2023-01-15T14:32:27.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:27.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125780+0000", 2023-01-15T14:32:27.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014173000000000001 2023-01-15T14:32:27.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:27.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529035+0000", 2023-01-15T14:32:27.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40325542199999997 2023-01-15T14:32:27.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:27.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529366+0000", 2023-01-15T14:32:27.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033111700000000002 2023-01-15T14:32:27.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:27.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:27.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:27.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529832+0000", 2023-01-15T14:32:27.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046592399999999999 2023-01-15T14:32:27.400 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:27.401 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:27.401 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:27.401 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:32:27.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:32:27.403 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:32:27.403 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:32:27.403 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2023-01-15T14:32:27.403 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-01-15T14:32:27.403 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-01-15T14:32:27.404 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok config set filestore_inject_stall 3 2023-01-15T14:32:27.404 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:32:27.409 INFO:tasks.rados.rados.0.smithi090.stdout:1013: done (3 left) 2023-01-15T14:32:27.410 INFO:tasks.rados.rados.0.smithi090.stdout:1014: read oid 42 snap 97 2023-01-15T14:32:27.410 INFO:tasks.rados.rados.0.smithi090.stdout:1014: expect deleted 2023-01-15T14:32:27.410 INFO:tasks.rados.rados.0.smithi090.stdout:1010: finishing write tid 1 to smithi09025264-41 2023-01-15T14:32:27.410 INFO:tasks.rados.rados.0.smithi090.stdout:1010: finishing write tid 2 to smithi09025264-41 2023-01-15T14:32:27.410 INFO:tasks.rados.rados.0.smithi090.stdout:1010: finishing write tid 3 to smithi09025264-41 2023-01-15T14:32:27.410 INFO:tasks.rados.rados.0.smithi090.stdout:1010: finishing write tid 4 to smithi09025264-41 2023-01-15T14:32:27.410 INFO:tasks.rados.rados.0.smithi090.stdout:1010: oid 41 updating version 0 to 270 2023-01-15T14:32:27.410 INFO:tasks.rados.rados.0.smithi090.stdout:1010: oid 41 version 270 is already newer than 268 2023-01-15T14:32:27.410 INFO:tasks.rados.rados.0.smithi090.stdout:1010: oid 41 version 270 is already newer than 269 2023-01-15T14:32:27.410 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 270 (ObjNum 374 snap 102 seq_num 374) dirty exists 2023-01-15T14:32:27.410 INFO:tasks.rados.rados.0.smithi090.stdout:1010: left oid 41 (ObjNum 374 snap 102 seq_num 374) 2023-01-15T14:32:27.411 INFO:tasks.rados.rados.0.smithi090.stdout:1011: finishing write tid 1 to smithi09025264-32 2023-01-15T14:32:27.411 INFO:tasks.rados.rados.0.smithi090.stdout:1011: finishing write tid 2 to smithi09025264-32 2023-01-15T14:32:27.411 INFO:tasks.rados.rados.0.smithi090.stdout:1011: finishing write tid 3 to smithi09025264-32 2023-01-15T14:32:27.411 INFO:tasks.rados.rados.0.smithi090.stdout:1011: finishing write tid 4 to smithi09025264-32 2023-01-15T14:32:27.411 INFO:tasks.rados.rados.0.smithi090.stdout:1011: oid 32 updating version 0 to 253 2023-01-15T14:32:27.411 INFO:tasks.rados.rados.0.smithi090.stdout:1011: oid 32 version 253 is already newer than 252 2023-01-15T14:32:27.411 INFO:tasks.rados.rados.0.smithi090.stdout:1011: oid 32 updating version 253 to 254 2023-01-15T14:32:27.411 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 254 (ObjNum 375 snap 102 seq_num 375) dirty exists 2023-01-15T14:32:27.411 INFO:tasks.rados.rados.0.smithi090.stdout:1011: left oid 32 (ObjNum 375 snap 102 seq_num 375) 2023-01-15T14:32:27.411 INFO:tasks.rados.rados.0.smithi090.stdout:1010: done (3 left) 2023-01-15T14:32:27.411 INFO:tasks.rados.rados.0.smithi090.stdout:1011: done (2 left) 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1012: done (1 left) 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1015: snap_remove snap 95 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1015: done (1 left) 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1016: delete oid 46 current snap is 102 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1014: done (1 left) 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1016: done (0 left) 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1017: write oid 31 current snap is 102 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1017: seq_num 377 ranges {736239=470170,1728182=571782} 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1017: writing smithi09025264-31 from 736239 to 1206409 tid 1 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1017: writing smithi09025264-31 from 1728182 to 2299964 tid 2 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1018: copy_from oid 29 from oid 21 current snap is 102 2023-01-15T14:32:27.412 INFO:tasks.rados.rados.0.smithi090.stdout:1019: snap_remove snap 97 2023-01-15T14:32:27.413 INFO:tasks.rados.rados.0.smithi090.stdout:1019: done (2 left) 2023-01-15T14:32:27.413 INFO:tasks.rados.rados.0.smithi090.stdout:1020: read oid 34 snap -1 2023-01-15T14:32:27.413 INFO:tasks.rados.rados.0.smithi090.stdout:1020: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:32:27.413 INFO:tasks.rados.rados.0.smithi090.stdout:1021: write (excl) oid 33 current snap is 102 2023-01-15T14:32:27.413 INFO:tasks.rados.rados.0.smithi090.stdout:1021: seq_num 378 ranges {753046=545419,1786664=431793} 2023-01-15T14:32:27.413 INFO:tasks.rados.rados.0.smithi090.stdout:1021: writing smithi09025264-33 from 753046 to 1298465 tid 1 2023-01-15T14:32:27.413 INFO:tasks.rados.rados.0.smithi090.stdout:1021: writing smithi09025264-33 from 1786664 to 2218457 tid 2 2023-01-15T14:32:27.413 INFO:tasks.rados.rados.0.smithi090.stdout:1022: snap_create 2023-01-15T14:32:27.413 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:32:27.413 INFO:tasks.rados.rados.0.smithi090.stdout:1017: finishing write tid 1 to smithi09025264-31 2023-01-15T14:32:27.413 INFO:tasks.rados.rados.0.smithi090.stdout:1018: finishing copy_from to smithi09025264-29 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 253 (ObjNum 339 snap 89 seq_num 339) dirty exists 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:1017: finishing write tid 2 to smithi09025264-31 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:1017: finishing write tid 3 to smithi09025264-31 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:1017: finishing write tid 4 to smithi09025264-31 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:1017: oid 31 updating version 0 to 334 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:1017: oid 31 version 334 is already newer than 333 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:1017: oid 31 version 334 is already newer than 332 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 334 (ObjNum 377 snap 102 seq_num 377) dirty exists 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:1017: left oid 31 (ObjNum 377 snap 102 seq_num 377) 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:1018: finishing copy_from racing read to smithi09025264-29 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:1017: done (3 left) 2023-01-15T14:32:27.414 INFO:tasks.rados.rados.0.smithi090.stdout:1018: done (2 left) 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout:1020: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout:1020: done (1 left) 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout:1021: finishing write tid 1 to smithi09025264-33 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout:1021: finishing write tid 2 to smithi09025264-33 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout:1021: finishing write tid 3 to smithi09025264-33 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout:1021: finishing write tid 4 to smithi09025264-33 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout:1021: oid 33 updating version 0 to 113 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout:1021: oid 33 version 113 is already newer than 112 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout:1021: oid 33 version 113 is already newer than 111 2023-01-15T14:32:27.415 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 113 (ObjNum 378 snap 102 seq_num 378) dirty exists 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:1021: left oid 33 (ObjNum 378 snap 102 seq_num 378) 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:1021: done (0 left) 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:1022: done (0 left) 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:1023: rollback oid 37 current snap is 103 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 37 to 100 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:1024: read oid 32 snap -1 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:1024: expect (ObjNum 375 snap 102 seq_num 375) 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:1025: rollback oid 25 current snap is 103 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 25 to 100 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:1026: write (excl) oid 29 current snap is 103 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:1026: seq_num 379 ranges {769853=620668,1845146=524707} 2023-01-15T14:32:27.416 INFO:tasks.rados.rados.0.smithi090.stdout:1026: writing smithi09025264-29 from 769853 to 1390521 tid 1 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:1026: writing smithi09025264-29 from 1845146 to 2369853 tid 2 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:1027: write oid 20 current snap is 103 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:1027: seq_num 380 ranges {786660=695917,1903628=435462} 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:1027: writing smithi09025264-20 from 786660 to 1482577 tid 1 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:1027: writing smithi09025264-20 from 1903628 to 2339090 tid 2 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:1028: delete oid 26 current snap is 103 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:1023: finishing rollback tid 1 to smithi09025264-37 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 0 (ObjNum 305 snap 0 seq_num 3200686864) dirty dne 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:1024: expect (ObjNum 375 snap 102 seq_num 375) 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:1027: finishing write tid 1 to smithi09025264-20 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:1026: finishing write tid 1 to smithi09025264-29 2023-01-15T14:32:27.417 INFO:tasks.rados.rados.0.smithi090.stdout:1027: finishing write tid 2 to smithi09025264-20 2023-01-15T14:32:27.418 INFO:tasks.rados.rados.0.smithi090.stdout:1027: finishing write tid 3 to smithi09025264-20 2023-01-15T14:32:27.418 INFO:tasks.rados.rados.0.smithi090.stdout:1027: finishing write tid 4 to smithi09025264-20 2023-01-15T14:32:27.418 INFO:tasks.rados.rados.0.smithi090.stdout:1027: oid 20 updating version 0 to 247 2023-01-15T14:32:27.418 INFO:tasks.rados.rados.0.smithi090.stdout:1027: oid 20 updating version 247 to 248 2023-01-15T14:32:27.418 INFO:tasks.rados.rados.0.smithi090.stdout:1027: oid 20 version 248 is already newer than 246 2023-01-15T14:32:27.418 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 248 (ObjNum 380 snap 103 seq_num 380) dirty exists 2023-01-15T14:32:27.418 INFO:tasks.rados.rados.0.smithi090.stdout:1027: left oid 20 (ObjNum 380 snap 103 seq_num 380) 2023-01-15T14:32:27.418 INFO:tasks.rados.rados.0.smithi090.stdout:1026: finishing write tid 2 to smithi09025264-29 2023-01-15T14:32:27.418 INFO:tasks.rados.rados.0.smithi090.stdout:1026: finishing write tid 3 to smithi09025264-29 2023-01-15T14:32:27.418 INFO:tasks.rados.rados.0.smithi090.stdout:1025: finishing rollback tid 0 to smithi09025264-25 2023-01-15T14:32:27.419 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:27.398+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:27.419 INFO:tasks.rados.rados.0.smithi090.stdout:1026: finishing write tid 4 to smithi09025264-29 2023-01-15T14:32:27.419 INFO:tasks.rados.rados.0.smithi090.stdout:1026: oid 29 updating version 0 to 255 2023-01-15T14:32:27.419 INFO:tasks.rados.rados.0.smithi090.stdout:1026: oid 29 updating version 255 to 256 2023-01-15T14:32:27.420 INFO:tasks.rados.rados.0.smithi090.stdout:1026: oid 29 version 256 is already newer than 254 2023-01-15T14:32:27.420 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 256 (ObjNum 379 snap 103 seq_num 379) dirty exists 2023-01-15T14:32:27.420 INFO:tasks.rados.rados.0.smithi090.stdout:1026: left oid 29 (ObjNum 379 snap 103 seq_num 379) 2023-01-15T14:32:27.504 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:27.498+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:27.593 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:27.593 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:32:27.593 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:32:27.594 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:27.594 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:27.595 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:27.595 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-01-15T14:32:27.596 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:27.596 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:32:27.605 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:27.601+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:27.660 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:27.812 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:27.812 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:32:27.812 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:32:27.813 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:32:27.813 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:27.813 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:27.813 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:27.813 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:27.813 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:27.813 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:27.813 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:27.814 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:32:27.815 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:27.805+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:27.815 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:32:27.815 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:32:27.815 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:32:27.815 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:32:27.816 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:32:27.816 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:32:27.816 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:32:27.816 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:32:27.823 INFO:tasks.rados.rados.0.smithi090.stdout:1023: done (5 left) 2023-01-15T14:32:28.017 INFO:tasks.rados.rados.0.smithi090.stdout:1024: done (4 left) 2023-01-15T14:32:28.017 INFO:tasks.rados.rados.0.smithi090.stdout:1026: done (3 left) 2023-01-15T14:32:28.017 INFO:tasks.rados.rados.0.smithi090.stdout:1027: done (2 left) 2023-01-15T14:32:28.017 INFO:tasks.rados.rados.0.smithi090.stdout:1028: done (1 left) 2023-01-15T14:32:28.017 INFO:tasks.rados.rados.0.smithi090.stdout:1029: write oid 36 current snap is 103 2023-01-15T14:32:28.017 INFO:tasks.rados.rados.0.smithi090.stdout:1029: seq_num 382 ranges {420274=562768,1620592=439131} 2023-01-15T14:32:28.020 INFO:tasks.rados.rados.0.smithi090.stdout:1029: writing smithi09025264-36 from 420274 to 983042 tid 1 2023-01-15T14:32:28.020 INFO:tasks.rados.rados.0.smithi090.stdout:1029: writing smithi09025264-36 from 1620592 to 2059723 tid 2 2023-01-15T14:32:28.020 INFO:tasks.rados.rados.0.smithi090.stdout:1030: copy_from oid 48 from oid 27 current snap is 103 2023-01-15T14:32:28.021 INFO:tasks.rados.rados.0.smithi090.stdout:1031: copy_from oid 10 from oid 5 current snap is 103 2023-01-15T14:32:28.021 INFO:tasks.rados.rados.0.smithi090.stdout:1032: rollback oid 17 current snap is 103 2023-01-15T14:32:28.021 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 17 to 102 2023-01-15T14:32:28.021 INFO:tasks.rados.rados.0.smithi090.stdout:1033: write (excl) oid 19 current snap is 103 2023-01-15T14:32:28.021 INFO:tasks.rados.rados.0.smithi090.stdout:1033: seq_num 383 ranges {437081=638017,1679074=582789} 2023-01-15T14:32:28.021 INFO:tasks.rados.rados.0.smithi090.stdout:1033: writing smithi09025264-19 from 437081 to 1075098 tid 1 2023-01-15T14:32:28.021 INFO:tasks.rados.rados.0.smithi090.stdout:1033: writing smithi09025264-19 from 1679074 to 2261863 tid 2 2023-01-15T14:32:28.021 INFO:tasks.rados.rados.0.smithi090.stdout:1034: rollback oid 32 current snap is 103 2023-01-15T14:32:28.021 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 32 to 102 2023-01-15T14:32:28.021 INFO:tasks.rados.rados.0.smithi090.stdout:1029: finishing write tid 1 to smithi09025264-36 2023-01-15T14:32:28.021 INFO:tasks.rados.rados.0.smithi090.stdout:1031: finishing copy_from to smithi09025264-10 2023-01-15T14:32:28.022 INFO:tasks.rados.rados.0.smithi090.stdout:1031: got expected ENOENT (src dne) 2023-01-15T14:32:28.022 INFO:tasks.rados.rados.0.smithi090.stdout:1035: read oid 18 snap -1 2023-01-15T14:32:28.022 INFO:tasks.rados.rados.0.smithi090.stdout:1035: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:32:28.022 INFO:tasks.rados.rados.0.smithi090.stdout:1036: rollback oid 49 current snap is 103 2023-01-15T14:32:28.022 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 49 to 102 2023-01-15T14:32:28.022 INFO:tasks.rados.rados.0.smithi090.stdout:1037: read oid 21 snap -1 2023-01-15T14:32:28.022 INFO:tasks.rados.rados.0.smithi090.stdout:1037: expect (ObjNum 339 snap 89 seq_num 339) 2023-01-15T14:32:28.022 INFO:tasks.rados.rados.0.smithi090.stdout:1038: read oid 45 snap 102 2023-01-15T14:32:28.022 INFO:tasks.rados.rados.0.smithi090.stdout:1038: expect (ObjNum 336 snap 88 seq_num 336) 2023-01-15T14:32:28.022 INFO:tasks.rados.rados.0.smithi090.stdout:1039: copy_from oid 23 from oid 15 current snap is 103 2023-01-15T14:32:28.429 INFO:tasks.rados.rados.0.smithi090.stdout:1040: rollback oid 3 current snap is 103 2023-01-15T14:32:28.429 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 3 to 102 2023-01-15T14:32:28.429 INFO:tasks.rados.rados.0.smithi090.stdout:1041: write (excl) oid 39 current snap is 103 2023-01-15T14:32:28.429 INFO:tasks.rados.rados.0.smithi090.stdout:1041: seq_num 384 ranges {453888=713266,1737556=442800} 2023-01-15T14:32:28.429 INFO:tasks.rados.rados.0.smithi090.stdout:1041: writing smithi09025264-39 from 453888 to 1167154 tid 1 2023-01-15T14:32:28.430 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:32:28.431 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:28.013+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:28.432 INFO:tasks.rados.rados.0.smithi090.stdout:1041: writing smithi09025264-39 from 1737556 to 2180356 tid 2 2023-01-15T14:32:28.432 INFO:tasks.rados.rados.0.smithi090.stdout:1042: rollback oid 29 current snap is 103 2023-01-15T14:32:28.432 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 29 to 102 2023-01-15T14:32:28.432 INFO:tasks.rados.rados.0.smithi090.stdout:1043: write (excl) oid 40 current snap is 103 2023-01-15T14:32:28.432 INFO:tasks.rados.rados.0.smithi090.stdout:1043: seq_num 385 ranges {470695=788515,1679685=586458} 2023-01-15T14:32:28.433 INFO:tasks.rados.rados.0.smithi090.stdout:1043: writing smithi09025264-40 from 470695 to 1259210 tid 1 2023-01-15T14:32:28.434 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:28.434 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:28.434 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:32:28.436 INFO:tasks.rados.rados.0.smithi090.stdout:1043: writing smithi09025264-40 from 1679685 to 2266143 tid 2 2023-01-15T14:32:28.436 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:32:28.436 INFO:tasks.rados.rados.0.smithi090.stdout:1031: finishing copy_from racing read to smithi09025264-10 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout:1032: finishing rollback tid 0 to smithi09025264-17 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout:1031: done (15 left) 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout:1044: read oid 41 snap -1 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout:1044: expect (ObjNum 374 snap 102 seq_num 374) 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout:1030: finishing copy_from to smithi09025264-48 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout:1030: got expected ENOENT (src dne) 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout:1030: finishing copy_from racing read to smithi09025264-48 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout:1030: done (15 left) 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout:1045: write (excl) oid 31 current snap is 103 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout:1045: seq_num 386 ranges {487502=463764,1738167=446469} 2023-01-15T14:32:28.437 INFO:tasks.rados.rados.0.smithi090.stdout:1045: writing smithi09025264-31 from 487502 to 951266 tid 1 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout:1045: writing smithi09025264-31 from 1738167 to 2184636 tid 2 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 16 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout:1035: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout:1035: done (15 left) 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout:1046: rollback oid 8 current snap is 103 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 8 to 102 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout:1033: finishing write tid 1 to smithi09025264-19 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout:1032: finishing rollback tid 1 to smithi09025264-17 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout:1029: finishing write tid 2 to smithi09025264-36 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout:1029: finishing write tid 3 to smithi09025264-36 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout:1029: finishing write tid 4 to smithi09025264-36 2023-01-15T14:32:28.438 INFO:tasks.rados.rados.0.smithi090.stdout:1029: oid 36 updating version 0 to 247 2023-01-15T14:32:28.439 INFO:tasks.rados.rados.0.smithi090.stdout:1029: oid 36 updating version 247 to 248 2023-01-15T14:32:28.439 INFO:tasks.rados.rados.0.smithi090.stdout:1029: oid 36 updating version 248 to 249 2023-01-15T14:32:28.439 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 249 (ObjNum 382 snap 103 seq_num 382) dirty exists 2023-01-15T14:32:28.439 INFO:tasks.rados.rados.0.smithi090.stdout:1029: left oid 36 (ObjNum 382 snap 103 seq_num 382) 2023-01-15T14:32:28.439 INFO:tasks.rados.rados.0.smithi090.stdout:1040: finishing rollback tid 1 to smithi09025264-3 2023-01-15T14:32:28.440 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 275 (ObjNum 33 snap 0 seq_num 3182657928) dirty dne 2023-01-15T14:32:28.440 INFO:tasks.rados.rados.0.smithi090.stdout:1032: finishing rollback tid 2 to smithi09025264-17 2023-01-15T14:32:28.440 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 342 (ObjNum 345 snap 91 seq_num 345) dirty exists 2023-01-15T14:32:28.440 INFO:tasks.rados.rados.0.smithi090.stdout:1037: expect (ObjNum 339 snap 89 seq_num 339) 2023-01-15T14:32:28.440 INFO:tasks.rados.rados.0.smithi090.stdout:1041: finishing write tid 1 to smithi09025264-39 2023-01-15T14:32:28.440 INFO:tasks.rados.rados.0.smithi090.stdout:1036: finishing rollback tid 0 to smithi09025264-49 2023-01-15T14:32:28.440 INFO:tasks.rados.rados.0.smithi090.stdout:1042: finishing rollback tid 0 to smithi09025264-29 2023-01-15T14:32:28.440 INFO:tasks.rados.rados.0.smithi090.stdout:1038: expect (ObjNum 336 snap 88 seq_num 336) 2023-01-15T14:32:28.441 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:28.426+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:28.532 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:28.525+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:28.536 INFO:tasks.rados.rados.0.smithi090.stdout:1042: finishing rollback tid 1 to smithi09025264-29 2023-01-15T14:32:28.536 INFO:tasks.rados.rados.0.smithi090.stdout:1036: finishing rollback tid 1 to smithi09025264-49 2023-01-15T14:32:28.536 INFO:tasks.rados.rados.0.smithi090.stdout:1033: finishing write tid 2 to smithi09025264-19 2023-01-15T14:32:28.536 INFO:tasks.rados.rados.0.smithi090.stdout:1034: finishing rollback tid 0 to smithi09025264-32 2023-01-15T14:32:28.537 INFO:tasks.rados.rados.0.smithi090.stdout:1034: finishing rollback tid 1 to smithi09025264-32 2023-01-15T14:32:28.537 INFO:tasks.rados.rados.0.smithi090.stdout:1034: finishing rollback tid 2 to smithi09025264-32 2023-01-15T14:32:28.537 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 257 (ObjNum 375 snap 102 seq_num 375) dirty exists 2023-01-15T14:32:28.537 INFO:tasks.rados.rados.0.smithi090.stdout:1044: expect (ObjNum 374 snap 102 seq_num 374) 2023-01-15T14:32:28.553 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:28.553 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:28.564 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:32:28.633 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:28.629+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:28.658 INFO:tasks.rados.rados.0.smithi090.stdout:1042: finishing rollback tid 2 to smithi09025264-29 2023-01-15T14:32:28.658 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 259 (ObjNum 339 snap 89 seq_num 339) dirty exists 2023-01-15T14:32:28.658 INFO:tasks.rados.rados.0.smithi090.stdout:1045: finishing write tid 1 to smithi09025264-31 2023-01-15T14:32:28.658 INFO:tasks.rados.rados.0.smithi090.stdout:1041: finishing write tid 2 to smithi09025264-39 2023-01-15T14:32:28.658 INFO:tasks.rados.rados.0.smithi090.stdout:1041: finishing write tid 3 to smithi09025264-39 2023-01-15T14:32:28.658 INFO:tasks.rados.rados.0.smithi090.stdout:1041: finishing write tid 4 to smithi09025264-39 2023-01-15T14:32:28.658 INFO:tasks.rados.rados.0.smithi090.stdout:1041: oid 39 updating version 0 to 276 2023-01-15T14:32:28.658 INFO:tasks.rados.rados.0.smithi090.stdout:1041: oid 39 version 276 is already newer than 274 2023-01-15T14:32:28.659 INFO:tasks.rados.rados.0.smithi090.stdout:1041: oid 39 version 276 is already newer than 275 2023-01-15T14:32:28.659 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 276 (ObjNum 384 snap 103 seq_num 384) dirty exists 2023-01-15T14:32:28.659 INFO:tasks.rados.rados.0.smithi090.stdout:1041: left oid 39 (ObjNum 384 snap 103 seq_num 384) 2023-01-15T14:32:28.659 INFO:tasks.rados.rados.0.smithi090.stdout:1043: finishing write tid 1 to smithi09025264-40 2023-01-15T14:32:28.659 INFO:tasks.rados.rados.0.smithi090.stdout:1033: finishing write tid 3 to smithi09025264-19 2023-01-15T14:32:28.659 INFO:tasks.rados.rados.0.smithi090.stdout:1029: done (15 left) 2023-01-15T14:32:28.659 INFO:tasks.rados.rados.0.smithi090.stdout:1032: done (14 left) 2023-01-15T14:32:28.659 INFO:tasks.rados.rados.0.smithi090.stdout:1034: done (13 left) 2023-01-15T14:32:28.659 INFO:tasks.rados.rados.0.smithi090.stdout:1037: done (12 left) 2023-01-15T14:32:28.659 INFO:tasks.rados.rados.0.smithi090.stdout:1038: done (11 left) 2023-01-15T14:32:28.659 INFO:tasks.rados.rados.0.smithi090.stdout:1040: done (10 left) 2023-01-15T14:32:28.660 INFO:tasks.rados.rados.0.smithi090.stdout:1041: done (9 left) 2023-01-15T14:32:28.660 INFO:tasks.rados.rados.0.smithi090.stdout:1042: done (8 left) 2023-01-15T14:32:28.660 INFO:tasks.rados.rados.0.smithi090.stdout:1044: done (7 left) 2023-01-15T14:32:28.660 INFO:tasks.rados.rados.0.smithi090.stdout:1047: write (excl) oid 5 current snap is 103 2023-01-15T14:32:28.660 INFO:tasks.rados.rados.0.smithi090.stdout:1033: finishing write tid 4 to smithi09025264-19 2023-01-15T14:32:28.660 INFO:tasks.rados.rados.0.smithi090.stdout:1033: oid 19 updating version 0 to 232 2023-01-15T14:32:28.660 INFO:tasks.rados.rados.0.smithi090.stdout:1033: oid 19 updating version 232 to 234 2023-01-15T14:32:28.660 INFO:tasks.rados.rados.0.smithi090.stdout:1033: oid 19 version 234 is already newer than 233 2023-01-15T14:32:28.660 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 234 (ObjNum 383 snap 103 seq_num 383) dirty exists 2023-01-15T14:32:28.660 INFO:tasks.rados.rados.0.smithi090.stdout:1033: left oid 19 (ObjNum 383 snap 103 seq_num 383) 2023-01-15T14:32:28.661 INFO:tasks.rados.rados.0.smithi090.stdout:1046: finishing rollback tid 1 to smithi09025264-8 2023-01-15T14:32:28.661 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 0 (ObjNum 321 snap 0 seq_num 3844009232) dirty dne 2023-01-15T14:32:28.661 INFO:tasks.rados.rados.0.smithi090.stdout:1036: finishing rollback tid 2 to smithi09025264-49 2023-01-15T14:32:28.661 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 275 (ObjNum 363 snap 96 seq_num 363) dirty exists 2023-01-15T14:32:28.661 INFO:tasks.rados.rados.0.smithi090.stdout:1043: finishing write tid 2 to smithi09025264-40 2023-01-15T14:32:28.661 INFO:tasks.rados.rados.0.smithi090.stdout:1043: finishing write tid 3 to smithi09025264-40 2023-01-15T14:32:28.661 INFO:tasks.rados.rados.0.smithi090.stdout:1039: finishing copy_from to smithi09025264-23 2023-01-15T14:32:28.661 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 152 (ObjNum 361 snap 96 seq_num 361) dirty exists 2023-01-15T14:32:28.661 INFO:tasks.rados.rados.0.smithi090.stdout:1039: finishing copy_from racing read to smithi09025264-23 2023-01-15T14:32:28.661 INFO:tasks.rados.rados.0.smithi090.stdout:1047: seq_num 387 ranges {504309=539013,1796649=590127} 2023-01-15T14:32:28.667 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:28.667 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:28.672 INFO:tasks.rados.rados.0.smithi090.stdout:1047: writing smithi09025264-5 from 504309 to 1043322 tid 1 2023-01-15T14:32:28.677 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:32:28.680 INFO:tasks.rados.rados.0.smithi090.stdout:1047: writing smithi09025264-5 from 1796649 to 2386776 tid 2 2023-01-15T14:32:28.680 INFO:tasks.rados.rados.0.smithi090.stdout:1033: done (7 left) 2023-01-15T14:32:28.680 INFO:tasks.rados.rados.0.smithi090.stdout:1036: done (6 left) 2023-01-15T14:32:28.680 INFO:tasks.rados.rados.0.smithi090.stdout:1039: done (5 left) 2023-01-15T14:32:28.680 INFO:tasks.rados.rados.0.smithi090.stdout:1046: done (4 left) 2023-01-15T14:32:28.681 INFO:tasks.rados.rados.0.smithi090.stdout:1048: write (excl) oid 12 current snap is 103 2023-01-15T14:32:28.681 INFO:tasks.rados.rados.0.smithi090.stdout:1045: finishing write tid 2 to smithi09025264-31 2023-01-15T14:32:28.681 INFO:tasks.rados.rados.0.smithi090.stdout:1048: seq_num 388 ranges {521116=730615,1971484=549632} 2023-01-15T14:32:28.687 INFO:tasks.rados.rados.0.smithi090.stdout:1048: writing smithi09025264-12 from 521116 to 1251731 tid 1 2023-01-15T14:32:28.695 INFO:tasks.rados.rados.0.smithi090.stdout:1048: writing smithi09025264-12 from 1971484 to 2521116 tid 2 2023-01-15T14:32:28.695 INFO:tasks.rados.rados.0.smithi090.stdout:1049: snap_create 2023-01-15T14:32:28.695 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:32:28.695 INFO:tasks.rados.rados.0.smithi090.stdout:1045: finishing write tid 3 to smithi09025264-31 2023-01-15T14:32:28.695 INFO:tasks.rados.rados.0.smithi090.stdout:1045: finishing write tid 4 to smithi09025264-31 2023-01-15T14:32:28.695 INFO:tasks.rados.rados.0.smithi090.stdout:1045: oid 31 updating version 0 to 344 2023-01-15T14:32:28.695 INFO:tasks.rados.rados.0.smithi090.stdout:1045: oid 31 updating version 344 to 346 2023-01-15T14:32:28.696 INFO:tasks.rados.rados.0.smithi090.stdout:1045: oid 31 version 346 is already newer than 345 2023-01-15T14:32:28.696 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 346 (ObjNum 386 snap 103 seq_num 386) dirty exists 2023-01-15T14:32:28.696 INFO:tasks.rados.rados.0.smithi090.stdout:1045: left oid 31 (ObjNum 386 snap 103 seq_num 386) 2023-01-15T14:32:28.696 INFO:tasks.rados.rados.0.smithi090.stdout:1045: done (4 left) 2023-01-15T14:32:28.696 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:32:28.696 INFO:tasks.rados.rados.0.smithi090.stdout:1043: finishing write tid 4 to smithi09025264-40 2023-01-15T14:32:28.696 INFO:tasks.rados.rados.0.smithi090.stdout:1043: oid 40 updating version 0 to 226 2023-01-15T14:32:28.696 INFO:tasks.rados.rados.0.smithi090.stdout:1043: oid 40 version 226 is already newer than 225 2023-01-15T14:32:28.696 INFO:tasks.rados.rados.0.smithi090.stdout:1043: oid 40 updating version 226 to 227 2023-01-15T14:32:28.696 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 227 (ObjNum 385 snap 103 seq_num 385) dirty exists 2023-01-15T14:32:28.696 INFO:tasks.rados.rados.0.smithi090.stdout:1043: left oid 40 (ObjNum 385 snap 103 seq_num 385) 2023-01-15T14:32:28.697 INFO:tasks.rados.rados.0.smithi090.stdout:1043: done (3 left) 2023-01-15T14:32:28.697 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:32:28.697 INFO:tasks.rados.rados.0.smithi090.stdout:1047: finishing write tid 1 to smithi09025264-5 2023-01-15T14:32:28.734 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:28.726+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:28.830 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:29.045 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:29.250 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:32:29.250 INFO:tasks.rados.rados.0.smithi090.stdout:1047: finishing write tid 2 to smithi09025264-5 2023-01-15T14:32:29.251 INFO:tasks.rados.rados.0.smithi090.stdout:1047: finishing write tid 3 to smithi09025264-5 2023-01-15T14:32:29.251 INFO:tasks.rados.rados.0.smithi090.stdout:1047: finishing write tid 4 to smithi09025264-5 2023-01-15T14:32:29.251 INFO:tasks.rados.rados.0.smithi090.stdout:1047: oid 5 updating version 0 to 225 2023-01-15T14:32:29.251 INFO:tasks.rados.rados.0.smithi090.stdout:1047: oid 5 version 225 is already newer than 224 2023-01-15T14:32:29.251 INFO:tasks.rados.rados.0.smithi090.stdout:1047: oid 5 version 225 is already newer than 223 2023-01-15T14:32:29.251 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 225 (ObjNum 387 snap 103 seq_num 387) dirty exists 2023-01-15T14:32:29.251 INFO:tasks.rados.rados.0.smithi090.stdout:1047: left oid 5 (ObjNum 387 snap 103 seq_num 387) 2023-01-15T14:32:29.251 INFO:tasks.rados.rados.0.smithi090.stdout:1047: done (2 left) 2023-01-15T14:32:29.251 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:29.251 INFO:tasks.rados.rados.0.smithi090.stdout:1048: finishing write tid 1 to smithi09025264-12 2023-01-15T14:32:29.252 INFO:tasks.rados.rados.0.smithi090.stdout:1048: finishing write tid 2 to smithi09025264-12 2023-01-15T14:32:29.252 INFO:tasks.rados.rados.0.smithi090.stdout:1048: finishing write tid 3 to smithi09025264-12 2023-01-15T14:32:29.252 INFO:tasks.rados.rados.0.smithi090.stdout:1048: finishing write tid 4 to smithi09025264-12 2023-01-15T14:32:29.252 INFO:tasks.rados.rados.0.smithi090.stdout:1048: oid 12 updating version 0 to 270 2023-01-15T14:32:29.252 INFO:tasks.rados.rados.0.smithi090.stdout:1048: oid 12 updating version 270 to 272 2023-01-15T14:32:29.252 INFO:tasks.rados.rados.0.smithi090.stdout:1048: oid 12 version 272 is already newer than 271 2023-01-15T14:32:29.252 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 272 (ObjNum 388 snap 103 seq_num 388) dirty exists 2023-01-15T14:32:29.252 INFO:tasks.rados.rados.0.smithi090.stdout:1048: left oid 12 (ObjNum 388 snap 103 seq_num 388) 2023-01-15T14:32:29.252 INFO:tasks.rados.rados.0.smithi090.stdout:1048: done (1 left) 2023-01-15T14:32:29.252 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:29.253 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:29.242+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:29.350 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:29.346+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:29.451 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:29.451 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:29.452 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:32:29.452 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:29.445+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:29.552 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:29.545+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:29.657 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:29.864 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:29.865 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:32:29.866 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:29.649+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:29.866 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:32:29.868 INFO:tasks.rados.rados.0.smithi090.stdout:1025: finishing rollback tid 1 to smithi09025264-25 2023-01-15T14:32:29.868 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:29.862+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:29.933 INFO:tasks.rados.rados.0.smithi090.stdout:1025: finishing rollback tid 2 to smithi09025264-25 2023-01-15T14:32:30.064 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 284 (ObjNum 335 snap 88 seq_num 335) dirty exists 2023-01-15T14:32:30.064 INFO:tasks.rados.rados.0.smithi090.stdout:1025: done (0 left) 2023-01-15T14:32:30.065 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:30.065 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:30.066 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:30.066 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:30.066 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:30.066 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:30.061+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:30.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:32:30.073 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:30.094 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:30.166 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:30.165+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:30.276 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:30.276 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:30.276 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:32:30.276 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:30.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:32:30.277 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:30.273+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:30.376 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:30.370+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:30.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:32:30.481 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:30.481 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:30.482 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:32:30.482 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:32:30.482 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:30.482 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:32:30.483 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:30.474+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:30.582 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:30.574+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:30.643 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:32:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:32:30.643 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:32:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:32:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 244.53247738799999, 2023-01-15T14:32:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:32:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:30.644 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:32:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:32:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:30.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:32:30.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:30.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:32:30.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:30.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:30.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:32:30.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:32:30.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:30.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:32:30.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:32:30.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:30.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:32:30.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:32:30.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:32:30.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:32:30.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:30.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:32:30.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:32:30.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:32:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:32:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:32:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:32:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:30.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:32:30.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:32:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:32:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:32:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:32:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:32:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:32:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:32:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:32:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:30.653 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 244.53219713999999, 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:30.654 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:32:30.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:30.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:32:30.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:32:30.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:30.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:32:30.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:32:30.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:30.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:32:30.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:32:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:32:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:32:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:32:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 244.53216452300001, 2023-01-15T14:32:30.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:32:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:32:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:30.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:32:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:32:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:32:30.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:32:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:32:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:32:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:32:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:32:30.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:32:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:32:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:32:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:32:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:32:30.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:32:30.670 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 244.43084653700001, 2023-01-15T14:32:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:32:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:32:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:32:30.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:32:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:32:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:32:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:32:30.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:32:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:32:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:32:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:32:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:32:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:32:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:32:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:32:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:32:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:32:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:32:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:32:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:30.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:32:30.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:32:30.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:30.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:32:30.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:32:30.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:30.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:32:30.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:32:30.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:30.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:32:30.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:32:30.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:30.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:32:30.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:32:30.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:30.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:32:30.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:32:30.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:30.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:32:30.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:32:30.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:30.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:32:30.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:32:30.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.864 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:30.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.864 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:30.864 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:32:30.864 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 244.430816534, 2023-01-15T14:32:30.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:32:30.864 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:30.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:32:30.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:30.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:32:30.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:30.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:30.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:32:30.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:32:30.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:32:30.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:32:30.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:30.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:32:30.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:32:30.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:32:30.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:32:30.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:30.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:32:30.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:32:30.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:32:30.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:32:30.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.872 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:30.872 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:32:30.872 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.91580796900001, 2023-01-15T14:32:30.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:32:30.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:30.872 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:30.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:30.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:30.872 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:30.872 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:32:30.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.873 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:30.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:30.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:32:30.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:30.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:32:30.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:30.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:32:30.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:32:30.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:30.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:32:30.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:30.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:32:30.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:32:30.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:30.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:32:30.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:32:30.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:30.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:32:30.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.882 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:30.882 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:32:30.882 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.915325615, 2023-01-15T14:32:30.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:32:30.882 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:30.882 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:30.882 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:30.882 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:30.882 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:30.882 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:32:30.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:30.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:32:30.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:30.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:30.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:32:30.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:32:30.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:30.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:32:30.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:32:30.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:30.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:32:30.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:32:30.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:30.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:32:30.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:32:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:32:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:32:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:32:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:30.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:32:30.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:30.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:32:30.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:32:30.892 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.893 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:30.893 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:30.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.893 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:30.893 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:32:30.893 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.915273052, 2023-01-15T14:32:30.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:32:30.893 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:30.893 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:30.893 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:30.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:30.894 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:30.894 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:32:30.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:30.894 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:30.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:30.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:30.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:32:31.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:32:31.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:32:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:32:31.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:32:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:32:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:32:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:32:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:32:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:32:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:32:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:32:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:32:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:32:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:31.027 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.184 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:32:31.184 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:32:31.185 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 230.54407134799999, 2023-01-15T14:32:31.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:32:31.185 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:31.185 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:31.186 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:31.186 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:31.186 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:31.186 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:32:31.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.187 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:31.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:31.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:32:31.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:32:31.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:32:31.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:32:31.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:32:31.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:32:31.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:32:31.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:32:31.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:32:31.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:32:31.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:32:31.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:32:31.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:31.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:32:31.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:32:31.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:31.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:32:31.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:32:31.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:31.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:32:31.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:32:31.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:31.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:32:31.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:32:31.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:31.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:32:31.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:32:31.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:31.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:32:31.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:32:31.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:32:31.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:32:31.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:32:31.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:32:31.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:31.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:32:31.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:32:31.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:31.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:32:31.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:32:31.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.203 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:31.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.203 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:32:31.204 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:32:31.204 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 230.54395467399999, 2023-01-15T14:32:31.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:32:31.204 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:31.204 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:31.205 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:31.205 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:31.205 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:31.205 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:32:31.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.206 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:31.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:31.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:32:31.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:32:31.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:32:31.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:31.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:32:31.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:32:31.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:32:31.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:32:31.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:32:31.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:32:31.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:32:31.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:32:31.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:31.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:32:31.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:32:31.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:32:31.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:32:31.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:31.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:32:31.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:32:31.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:32:31.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:32:31.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:31.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:32:31.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:32:31.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:31.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:32:31.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:32:31.346 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.346 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:31.346 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.346 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:32:31.346 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 222.817822398, 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:32:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:32:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:32:31.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:31.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:32:31.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:32:31.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:32:31.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:32:31.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:31.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.352 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:32:31.353 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:32:31.353 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 186.601572339, 2023-01-15T14:32:31.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:32:31.353 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:31.353 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:31.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:31.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:31.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:31.353 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:32:31.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.353 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:32:31.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:32:31.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:32:31.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:32:31.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:32:31.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:32:31.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:31.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:32:31.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:32:31.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:32:31.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:32:31.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:31.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:32:31.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:32:31.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:31.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:32:31.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:32:31.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:31.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:32:31.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:32:31.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:31.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:32:31.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:32:31.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:32:31.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:32:31.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:32:31.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:32:31.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:31.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:32:31.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:32:31.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:31.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:32:31.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:32:31.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.545 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:31.546 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.546 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:32:31.546 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:32:31.546 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 186.601538772, 2023-01-15T14:32:31.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:32:31.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:31.546 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:31.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:31.546 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:31.547 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:31.547 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:32:31.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.547 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:31.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:31.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:32:31.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:32:31.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:32:31.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:31.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:32:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:32:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:32:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:32:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:32:31.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:32:31.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:32:31.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:32:31.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:31.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:32:31.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:32:31.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:31.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:32:31.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:32:31.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:31.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:32:31.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:32:31.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:31.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:32:31.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:31.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:32:31.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:32:31.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:31.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:32:31.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:32:31.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.555 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:31.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.556 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:31.556 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:32:31.556 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 167.83563522599999, 2023-01-15T14:32:31.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:32:31.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:31.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:31.556 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:32:31.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:32:31.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:32:31.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:32:31.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:32:31.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:31.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:32:31.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:31.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:32:31.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:32:31.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:32:31.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:32:31.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:31.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:32:31.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:32:31.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:32:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:32:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.704 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:32:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:32:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 151.292456426, 2023-01-15T14:32:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:32:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:31.705 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:32:31.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:32:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:32:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:32:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:32:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:32:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:32:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:32:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:32:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:32:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:32:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:32:31.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:32:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:32:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:32:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:32:31.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:32:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:32:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:32:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:32:31.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:32:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:32:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:32:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:32:31.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:32:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:32:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:32:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:32:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 151.29241725099999, 2023-01-15T14:32:31.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:32:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:32:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.719 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:32:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:32:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:32:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:32:31.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:32:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:32:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:32:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:32:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:32:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:32:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:32:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:31.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:32:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:32:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:32:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:32:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:32:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:32:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:32:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:32:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1373 3.12 3:495d4bbc:::smithi09025264-16:head [write 2873523~591733] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:32:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:32:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 142.90030092500001, 2023-01-15T14:32:31.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40949580099999999, 2023-01-15T14:32:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1373 2023-01-15T14:32:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:31.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:32:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:32:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693440+0000", 2023-01-15T14:32:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693808+0000", 2023-01-15T14:32:31.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000367742 2023-01-15T14:32:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693831+0000", 2023-01-15T14:32:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3213e-05 2023-01-15T14:32:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693846+0000", 2023-01-15T14:32:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4909000000000001e-05 2023-01-15T14:32:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.695833+0000", 2023-01-15T14:32:31.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019869240000000002 2023-01-15T14:32:31.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:31.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696121+0000", 2023-01-15T14:32:31.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028753300000000003 2023-01-15T14:32:31.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798577+0000", 2023-01-15T14:32:31.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102456808 2023-01-15T14:32:31.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:31.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798849+0000", 2023-01-15T14:32:31.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027138999999999999 2023-01-15T14:32:31.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:32:31.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799155+0000", 2023-01-15T14:32:31.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030594800000000001 2023-01-15T14:32:31.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799614+0000", 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045960700000000001 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799737+0000", 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012218900000000001 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799998+0000", 2023-01-15T14:32:31.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026129700000000002 2023-01-15T14:32:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800639+0000", 2023-01-15T14:32:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064108800000000005 2023-01-15T14:32:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102023+0000", 2023-01-15T14:32:31.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30138417200000001 2023-01-15T14:32:31.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102198+0000", 2023-01-15T14:32:31.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017453600000000001 2023-01-15T14:32:31.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:31.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102398+0000", 2023-01-15T14:32:31.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020078000000000001 2023-01-15T14:32:31.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:31.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102937+0000", 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053874499999999998 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 142.899876331, 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:31.878 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:31.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:31.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:31.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:31.879 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:32:31.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.879 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:31.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:31.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:32:31.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:32:31.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:32:31.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:31.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:32:31.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:32:31.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:32:31.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:32:31.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:31.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:32:31.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:32:31.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:31.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:32:31.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:32:31.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:31.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:32:31.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:31.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:32:31.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:32:31.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:31.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:32:31.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:32:31.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:31.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:32:31.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:32:31.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:32:31.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:32:31.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:31.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:32:31.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:32:31.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:31.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:32:31.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:32:31.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:31.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:32:31.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 142.899824002, 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:31.889 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:31.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:32:31.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:31.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:31.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:32:31.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:31.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:31.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:32:31.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:32:31.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:31.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:32:31.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:32:31.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:31.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:31.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:31.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:32:31.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:32:32.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:32:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:32:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:32:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:32:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:32:32.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 124.302910803, 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:32.030 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:32.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:32:32.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:32:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:32:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:32:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:32:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:32:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:32:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:32:32.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:32:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:32:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:32.036 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:32.037 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:32:32.038 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:32.038 INFO:tasks.rados.rados.0.smithi090.stdout:1049: done (0 left) 2023-01-15T14:32:32.038 INFO:tasks.rados.rados.0.smithi090.stdout:1050: write (excl) oid 23 current snap is 104 2023-01-15T14:32:32.038 INFO:tasks.rados.rados.0.smithi090.stdout:1050: seq_num 389 ranges {537923=405864,1513613=593796} 2023-01-15T14:32:32.038 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:32:32.039 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:32.039 INFO:tasks.rados.rados.0.smithi090.stdout:1050: writing smithi09025264-23 from 537923 to 943787 tid 1 2023-01-15T14:32:32.040 INFO:tasks.rados.rados.0.smithi090.stdout:1050: writing smithi09025264-23 from 1513613 to 2107409 tid 2 2023-01-15T14:32:32.040 INFO:tasks.rados.rados.0.smithi090.stdout:1051: write oid 22 current snap is 104 2023-01-15T14:32:32.040 INFO:tasks.rados.rados.0.smithi090.stdout:1051: seq_num 390 ranges {554730=481113,1572095=737454} 2023-01-15T14:32:32.041 INFO:tasks.rados.rados.0.smithi090.stdout:1051: writing smithi09025264-22 from 554730 to 1035843 tid 1 2023-01-15T14:32:32.041 INFO:tasks.rados.rados.0.smithi090.stdout:1051: writing smithi09025264-22 from 1572095 to 2309549 tid 2 2023-01-15T14:32:32.041 INFO:tasks.rados.rados.0.smithi090.stdout:1052: read oid 10 snap 100 2023-01-15T14:32:32.041 INFO:tasks.rados.rados.0.smithi090.stdout:1052: expect (ObjNum 360 snap 95 seq_num 360) 2023-01-15T14:32:32.041 INFO:tasks.rados.rados.0.smithi090.stdout:1053: rollback oid 5 current snap is 104 2023-01-15T14:32:32.041 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 5 to 103 2023-01-15T14:32:32.041 INFO:tasks.rados.rados.0.smithi090.stdout:1054: delete oid 3 current snap is 104 2023-01-15T14:32:32.041 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:32.042 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:32:32.042 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:32:32.042 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:32:32.042 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:32:32.043 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:32:32.043 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: filestore_inject_stall = '0' (not observed, change may require restart) osd_enable_op_tracker = 'true' 2023-01-15T14:32:32.043 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:32:32.043 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:32:32.045 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:32.033+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:32.192 INFO:tasks.rados.rados.0.smithi090.stdout:1052: expect (ObjNum 360 snap 95 seq_num 360) 2023-01-15T14:32:32.192 INFO:tasks.rados.rados.0.smithi090.stdout:1050: finishing write tid 1 to smithi09025264-23 2023-01-15T14:32:32.192 INFO:tasks.rados.rados.0.smithi090.stdout:1053: finishing rollback tid 0 to smithi09025264-5 2023-01-15T14:32:32.192 INFO:tasks.rados.rados.0.smithi090.stdout:1051: finishing write tid 1 to smithi09025264-22 2023-01-15T14:32:32.192 INFO:tasks.rados.rados.0.smithi090.stdout:1053: finishing rollback tid 1 to smithi09025264-5 2023-01-15T14:32:32.192 INFO:tasks.rados.rados.0.smithi090.stdout:1053: finishing rollback tid 2 to smithi09025264-5 2023-01-15T14:32:32.192 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 229 (ObjNum 387 snap 103 seq_num 387) dirty exists 2023-01-15T14:32:32.192 INFO:tasks.rados.rados.0.smithi090.stdout:1052: done (4 left) 2023-01-15T14:32:32.193 INFO:tasks.rados.rados.0.smithi090.stdout:1053: done (3 left) 2023-01-15T14:32:32.193 INFO:tasks.rados.rados.0.smithi090.stdout:1054: done (2 left) 2023-01-15T14:32:32.193 INFO:tasks.rados.rados.0.smithi090.stdout:1055: write oid 12 current snap is 104 2023-01-15T14:32:32.193 INFO:tasks.rados.rados.0.smithi090.stdout:1055: seq_num 392 ranges {588344=631611,1689059=741123} 2023-01-15T14:32:32.193 INFO:tasks.rados.rados.0.smithi090.stdout:1055: writing smithi09025264-12 from 588344 to 1219955 tid 1 2023-01-15T14:32:32.193 INFO:tasks.rados.rados.0.smithi090.stdout:1055: writing smithi09025264-12 from 1689059 to 2430182 tid 2 2023-01-15T14:32:32.193 INFO:tasks.rados.rados.0.smithi090.stdout:1056: snap_remove snap 102 2023-01-15T14:32:32.193 INFO:tasks.rados.rados.0.smithi090.stdout:1051: finishing write tid 2 to smithi09025264-22 2023-01-15T14:32:32.193 INFO:tasks.rados.rados.0.smithi090.stdout:1051: finishing write tid 3 to smithi09025264-22 2023-01-15T14:32:32.193 INFO:tasks.rados.rados.0.smithi090.stdout:1051: finishing write tid 4 to smithi09025264-22 2023-01-15T14:32:32.193 INFO:tasks.rados.rados.0.smithi090.stdout:1051: oid 22 updating version 0 to 280 2023-01-15T14:32:32.194 INFO:tasks.rados.rados.0.smithi090.stdout:1051: oid 22 version 280 is already newer than 278 2023-01-15T14:32:32.194 INFO:tasks.rados.rados.0.smithi090.stdout:1051: oid 22 version 280 is already newer than 279 2023-01-15T14:32:32.194 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 280 (ObjNum 390 snap 104 seq_num 390) dirty exists 2023-01-15T14:32:32.194 INFO:tasks.rados.rados.0.smithi090.stdout:1051: left oid 22 (ObjNum 390 snap 104 seq_num 390) 2023-01-15T14:32:32.194 INFO:tasks.rados.rados.0.smithi090.stdout:1050: finishing write tid 2 to smithi09025264-23 2023-01-15T14:32:32.194 INFO:tasks.rados.rados.0.smithi090.stdout:1050: finishing write tid 3 to smithi09025264-23 2023-01-15T14:32:32.194 INFO:tasks.rados.rados.0.smithi090.stdout:1055: finishing write tid 1 to smithi09025264-12 2023-01-15T14:32:32.194 INFO:tasks.rados.rados.0.smithi090.stdout:1055: finishing write tid 2 to smithi09025264-12 2023-01-15T14:32:32.194 INFO:tasks.rados.rados.0.smithi090.stdout:1055: finishing write tid 3 to smithi09025264-12 2023-01-15T14:32:32.194 INFO:tasks.rados.rados.0.smithi090.stdout:1055: finishing write tid 4 to smithi09025264-12 2023-01-15T14:32:32.194 INFO:tasks.rados.rados.0.smithi090.stdout:1055: oid 12 updating version 0 to 274 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:1055: oid 12 updating version 274 to 275 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:1055: oid 12 updating version 275 to 276 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 276 (ObjNum 392 snap 104 seq_num 392) dirty exists 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:1055: left oid 12 (ObjNum 392 snap 104 seq_num 392) 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:1050: finishing write tid 4 to smithi09025264-23 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:1050: oid 23 updating version 0 to 213 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:1050: oid 23 version 213 is already newer than 211 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:1050: oid 23 version 213 is already newer than 212 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 213 (ObjNum 389 snap 104 seq_num 389) dirty exists 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:1050: left oid 23 (ObjNum 389 snap 104 seq_num 389) 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:1050: done (3 left) 2023-01-15T14:32:32.195 INFO:tasks.rados.rados.0.smithi090.stdout:1051: done (2 left) 2023-01-15T14:32:32.196 INFO:tasks.rados.rados.0.smithi090.stdout:1055: done (1 left) 2023-01-15T14:32:32.196 INFO:tasks.rados.rados.0.smithi090.stdout:1056: done (0 left) 2023-01-15T14:32:32.196 INFO:tasks.rados.rados.0.smithi090.stdout:1057: rollback oid 39 current snap is 104 2023-01-15T14:32:32.196 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 39 to 100 2023-01-15T14:32:32.196 INFO:tasks.rados.rados.0.smithi090.stdout:1058: write (excl) oid 25 current snap is 104 2023-01-15T14:32:32.196 INFO:tasks.rados.rados.0.smithi090.stdout:1058: seq_num 393 ranges {605151=706860,2031188=573963} 2023-01-15T14:32:32.196 INFO:tasks.rados.rados.0.smithi090.stdout:1058: writing smithi09025264-25 from 605151 to 1312011 tid 1 2023-01-15T14:32:32.196 INFO:tasks.rados.rados.0.smithi090.stdout:1058: writing smithi09025264-25 from 2031188 to 2605151 tid 2 2023-01-15T14:32:32.196 INFO:tasks.rados.rados.0.smithi090.stdout:1059: read oid 18 snap -1 2023-01-15T14:32:32.196 INFO:tasks.rados.rados.0.smithi090.stdout:1059: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:32:32.196 INFO:tasks.rados.rados.0.smithi090.stdout:1060: write oid 9 current snap is 104 2023-01-15T14:32:32.197 INFO:tasks.rados.rados.0.smithi090.stdout:1060: seq_num 394 ranges {621958=782109,2089670=532288} 2023-01-15T14:32:32.197 INFO:tasks.rados.rados.0.smithi090.stdout:1060: writing smithi09025264-9 from 621958 to 1404067 tid 1 2023-01-15T14:32:32.197 INFO:tasks.rados.rados.0.smithi090.stdout:1060: writing smithi09025264-9 from 2089670 to 2621958 tid 2 2023-01-15T14:32:32.197 INFO:tasks.rados.rados.0.smithi090.stdout:1061: copy_from oid 49 from oid 31 current snap is 104 2023-01-15T14:32:32.197 INFO:tasks.rados.rados.0.smithi090.stdout:1062: read oid 27 snap -1 2023-01-15T14:32:32.197 INFO:tasks.rados.rados.0.smithi090.stdout:1062: expect deleted 2023-01-15T14:32:32.197 INFO:tasks.rados.rados.0.smithi090.stdout:1063: rollback oid 13 current snap is 104 2023-01-15T14:32:32.197 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 13 to 103 2023-01-15T14:32:32.197 INFO:tasks.rados.rados.0.smithi090.stdout:1064: snap_create 2023-01-15T14:32:32.197 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:32:32.197 INFO:tasks.rados.rados.0.smithi090.stdout:1057: finishing rollback tid 0 to smithi09025264-39 2023-01-15T14:32:32.198 INFO:tasks.rados.rados.0.smithi090.stdout:1059: expect (ObjNum 237 snap 54 seq_num 237) 2023-01-15T14:32:32.200 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:32.194+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:32.279 INFO:tasks.rados.rados.0.smithi090.stdout:1059: done (6 left) 2023-01-15T14:32:32.366 INFO:tasks.rados.rados.0.smithi090.stdout:1062: done (5 left) 2023-01-15T14:32:32.367 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:32:32.369 INFO:tasks.rados.rados.0.smithi090.stdout:1058: finishing write tid 1 to smithi09025264-25 2023-01-15T14:32:32.369 INFO:tasks.rados.rados.0.smithi090.stdout:1057: finishing rollback tid 1 to smithi09025264-39 2023-01-15T14:32:32.373 INFO:tasks.rados.rados.0.smithi090.stdout:1057: finishing rollback tid 2 to smithi09025264-39 2023-01-15T14:32:32.373 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 280 (ObjNum 183 snap 40 seq_num 183) dirty exists 2023-01-15T14:32:32.373 INFO:tasks.rados.rados.0.smithi090.stdout:1060: finishing write tid 1 to smithi09025264-9 2023-01-15T14:32:32.373 INFO:tasks.rados.rados.0.smithi090.stdout:1057: done (4 left) 2023-01-15T14:32:32.373 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:32:32.373 INFO:tasks.rados.rados.0.smithi090.stdout:1060: finishing write tid 2 to smithi09025264-9 2023-01-15T14:32:32.373 INFO:tasks.rados.rados.0.smithi090.stdout:1060: finishing write tid 3 to smithi09025264-9 2023-01-15T14:32:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:1060: finishing write tid 4 to smithi09025264-9 2023-01-15T14:32:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:1060: oid 9 updating version 0 to 166 2023-01-15T14:32:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:1060: oid 9 version 166 is already newer than 165 2023-01-15T14:32:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:1060: oid 9 updating version 166 to 167 2023-01-15T14:32:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 167 (ObjNum 394 snap 104 seq_num 394) dirty exists 2023-01-15T14:32:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:1060: left oid 9 (ObjNum 394 snap 104 seq_num 394) 2023-01-15T14:32:32.374 INFO:tasks.rados.rados.0.smithi090.stdout:1060: done (3 left) 2023-01-15T14:32:32.374 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:32:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:1058: finishing write tid 2 to smithi09025264-25 2023-01-15T14:32:32.375 INFO:tasks.rados.rados.0.smithi090.stdout:1063: finishing rollback tid 0 to smithi09025264-13 2023-01-15T14:32:32.376 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:32:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2785 3.1a e265/224 3:5919d83e:::smithi09025264-13:head v 265'289, mlcod=258'285)", 2023-01-15T14:32:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:32.277752+0000", 2023-01-15T14:32:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.017554388000000001, 2023-01-15T14:32:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017581624000000001, 2023-01-15T14:32:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277752+0000", 2023-01-15T14:32:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277752+0000", 2023-01-15T14:32:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277751+0000", 2023-01-15T14:32:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277775+0000", 2023-01-15T14:32:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3745999999999999e-05 2023-01-15T14:32:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277791+0000", 2023-01-15T14:32:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.596e-05 2023-01-15T14:32:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277806+0000", 2023-01-15T14:32:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4822e-05 2023-01-15T14:32:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277847+0000", 2023-01-15T14:32:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0970999999999998e-05 2023-01-15T14:32:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277863+0000", 2023-01-15T14:32:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5971e-05 2023-01-15T14:32:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278332+0000", 2023-01-15T14:32:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046920300000000001 2023-01-15T14:32:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278340+0000", 2023-01-15T14:32:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8369999999999997e-06 2023-01-15T14:32:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:32.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278369+0000", 2023-01-15T14:32:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8595999999999999e-05 2023-01-15T14:32:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278388+0000", 2023-01-15T14:32:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9660000000000002e-05 2023-01-15T14:32:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2786 3.1a e265/224 3:5919d83e:::smithi09025264-13:head v 265'290, mlcod=258'285)", 2023-01-15T14:32:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:32.277975+0000", 2023-01-15T14:32:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.017331335999999999, 2023-01-15T14:32:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017437695999999999, 2023-01-15T14:32:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:32.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277975+0000", 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277975+0000", 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277975+0000", 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.277993+0000", 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8223000000000002e-05 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278008+0000", 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5002e-05 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:32.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278031+0000", 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2717e-05 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278578+0000", 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054779600000000003 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278619+0000", 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0652999999999999e-05 2023-01-15T14:32:32.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278854+0000", 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000234956 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278867+0000", 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2594e-05 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278896+0000", 2023-01-15T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8892000000000001e-05 2023-01-15T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.279188+0000", 2023-01-15T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029206399999999997 2023-01-15T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2787 3.1a e265/224 3:5919d83e:::smithi09025264-13:head v 265'291, mlcod=258'285)", 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:32.278045+0000", 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.017261477000000001, 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.017439883999999999, 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278045+0000", 2023-01-15T14:32:32.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278045+0000", 2023-01-15T14:32:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278045+0000", 2023-01-15T14:32:32.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278053+0000", 2023-01-15T14:32:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9330000000000001e-06 2023-01-15T14:32:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278067+0000", 2023-01-15T14:32:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3976e-05 2023-01-15T14:32:32.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278077+0000", 2023-01-15T14:32:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0169999999999999e-05 2023-01-15T14:32:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278914+0000", 2023-01-15T14:32:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083687099999999999 2023-01-15T14:32:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.278951+0000", 2023-01-15T14:32:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7639999999999999e-05 2023-01-15T14:32:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.279152+0000", 2023-01-15T14:32:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020117799999999999 2023-01-15T14:32:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.279236+0000", 2023-01-15T14:32:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3088000000000001e-05 2023-01-15T14:32:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.279268+0000", 2023-01-15T14:32:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2088000000000002e-05 2023-01-15T14:32:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:32.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.279283+0000", 2023-01-15T14:32:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5158e-05 2023-01-15T14:32:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:32.541 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 3 2023-01-15T14:32:32.541 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:32.542 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:32:32.542 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:32.362+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:32.543 INFO:tasks.rados.rados.0.smithi090.stdout:1063: finishing rollback tid 1 to smithi09025264-13 2023-01-15T14:32:32.543 INFO:tasks.rados.rados.0.smithi090.stdout:1061: finishing copy_from to smithi09025264-49 2023-01-15T14:32:32.543 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 346 (ObjNum 386 snap 103 seq_num 386) dirty exists 2023-01-15T14:32:32.543 INFO:tasks.rados.rados.0.smithi090.stdout:1061: finishing copy_from racing read to smithi09025264-49 2023-01-15T14:32:32.543 INFO:tasks.rados.rados.0.smithi090.stdout:1061: done (2 left) 2023-01-15T14:32:32.544 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:32.544 INFO:tasks.rados.rados.0.smithi090.stdout:1063: finishing rollback tid 2 to smithi09025264-13 2023-01-15T14:32:32.544 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 291 (ObjNum 369 snap 99 seq_num 369) dirty exists 2023-01-15T14:32:32.544 INFO:tasks.rados.rados.0.smithi090.stdout:1063: done (1 left) 2023-01-15T14:32:32.544 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:32.544 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:32.538+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:32.585 INFO:tasks.rados.rados.0.smithi090.stdout:1058: finishing write tid 3 to smithi09025264-25 2023-01-15T14:32:32.731 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:32:32.731 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:32:32.732 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2023-01-15T14:32:32.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-01-15T14:32:32.733 INFO:tasks.rados.rados.0.smithi090.stdout:1058: finishing write tid 4 to smithi09025264-25 2023-01-15T14:32:32.733 INFO:tasks.rados.rados.0.smithi090.stdout:1058: oid 25 updating version 0 to 289 2023-01-15T14:32:32.734 INFO:tasks.rados.rados.0.smithi090.stdout:1058: oid 25 updating version 289 to 291 2023-01-15T14:32:32.734 INFO:tasks.rados.rados.0.smithi090.stdout:1058: oid 25 version 291 is already newer than 290 2023-01-15T14:32:32.734 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 291 (ObjNum 393 snap 104 seq_num 393) dirty exists 2023-01-15T14:32:32.734 INFO:tasks.rados.rados.0.smithi090.stdout:1058: left oid 25 (ObjNum 393 snap 104 seq_num 393) 2023-01-15T14:32:32.734 INFO:tasks.rados.rados.0.smithi090.stdout:1058: done (0 left) 2023-01-15T14:32:32.735 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:32.735 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:32.735 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:32:32.735 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:32:32.735 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:32.736 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:32:32.738 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:32.726+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:32.835 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:32.830+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:32.939 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:32:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.79787270599999, 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:32:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:32:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:32:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:32:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:32:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:32:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:32:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:32:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:32:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:32:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:32:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:32:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:32:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:32:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:32:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:32:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:32:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:32:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:32:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:32:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:32:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:32:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:32:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:32:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:32:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:32:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:32:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:32:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:32:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:32:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:32:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:32:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:32:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:32:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:32:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:32:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:32:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:32:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.797572909, 2023-01-15T14:32:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:32:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:32:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:32:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:32:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:32:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:32:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:32:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:32:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:32:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:32:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:33.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:32:33.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:32:33.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:32:33.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:32:33.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:33.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:32:33.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:32:33.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:32:33.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:32:33.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:32:33.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:33.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:32:33.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:32:33.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:33.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:32:33.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:32:33.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:33.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:32:33.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:32:33.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:33.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:32:33.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:32:33.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:33.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:32:33.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:32:33.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:33.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:32:33.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:32:33.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:33.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:32:33.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:32:33.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:33.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:32:33.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:32:33.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.149 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:33.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.149 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:33.149 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:32:33.149 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.797543125, 2023-01-15T14:32:33.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:32:33.149 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:33.150 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:33.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:33.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:33.150 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:33.150 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:32:33.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.150 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:33.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:33.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:32:33.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:33.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:32:33.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:33.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:32:33.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:33.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:33.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:32:33.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:32:33.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:33.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:32:33.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:32:33.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:33.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:32:33.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:32:33.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:32:33.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:32:33.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:33.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:32:33.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:32:33.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:32:33.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:32:33.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:33.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:32:33.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:32:33.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:32:33.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:32:33.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:33.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:32:33.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:32:33.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:33.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:32:33.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:32:33.157 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.158 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:33.158 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.158 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:33.158 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:32:33.158 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.667734325, 2023-01-15T14:32:33.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:32:33.158 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:33.158 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:33.159 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:33.159 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:33.159 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:33.159 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:32:33.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.159 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:33.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:33.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:32:33.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:33.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:32:33.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:33.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:32:33.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:33.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:33.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:32:33.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:32:33.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:33.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:32:33.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:32:33.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:33.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:32:33.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:32:33.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:32:33.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:32:33.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:33.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:32:33.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:32:33.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:32:33.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:32:33.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:33.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:32:33.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:32:33.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:32:33.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:32:33.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:32:33.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:33.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:32:33.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:32:33.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:33.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:32:33.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:32:33.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:32:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:32:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:32:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:32:33.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:33.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:32:33.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:32:33.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:33.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:32:33.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:32:33.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:33.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:32:33.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:32:33.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:33.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:32:33.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:32:33.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.176 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:33.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.176 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:33.176 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:32:33.176 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 245.66767762800001, 2023-01-15T14:32:33.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:32:33.177 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:33.177 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:33.177 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:33.177 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:33.177 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:33.177 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:32:33.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.178 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:33.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:32:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:32:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:32:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:32:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:32:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:32:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:32:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:32:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:32:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:32:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:32:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:32:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:32:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:32:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:32:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:32:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:32:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:32:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:32:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:32:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:32:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:32:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:32:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:32:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:32:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.05118145500001, 2023-01-15T14:32:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:32:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:32:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:32:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:32:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:32:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:33.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:32:33.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:32:33.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:33.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:32:33.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:32:33.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:33.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:32:33.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:32:33.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:32:33.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:32:33.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:33.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:32:33.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:32:33.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:32:33.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:32:33.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:33.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:32:33.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:32:33.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:33.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:32:33.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:32:33.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.676 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:33.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.677 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:32:33.677 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:32:33.677 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 236.022332984, 2023-01-15T14:32:33.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:32:33.677 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:33.678 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:33.678 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:33.678 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:33.715 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:33.716 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:32:33.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.716 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:33.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:33.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:32:33.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:33.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:32:33.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:33.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:32:33.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:33.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:33.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:32:33.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:32:33.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:33.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:32:33.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:32:33.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:32:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:32:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:32:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:32:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:33.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:32:33.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:32:33.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:32:33.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:32:33.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:33.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:32:33.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:32:33.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:32:33.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:32:33.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:32:33.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:33.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:32:33.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:32:33.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:33.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:32:33.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:32:33.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:33.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:32:33.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:32:33.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:33.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:32:33.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:32:33.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:33.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:32:33.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:32:33.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:33.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:32:33.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:32:33.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:33.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:32:33.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:32:33.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:33.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:32:33.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:32:33.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.828 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:33.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.828 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:32:33.829 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:32:33.829 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 206.02104600800001, 2023-01-15T14:32:33.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:32:33.829 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:33.829 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:33.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:33.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:33.829 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:33.829 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:32:33.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.829 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:33.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:33.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:32:33.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:33.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:32:33.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:33.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:32:33.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:32:33.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:33.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:32:33.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:32:33.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:33.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:32:33.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:32:33.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:33.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:32:33.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:32:33.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:32:33.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:32:33.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:32:33.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:33.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:32:33.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:32:33.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:33.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:32:33.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:32:33.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:33.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:32:33.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:32:33.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:33.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:32:33.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:32:33.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:33.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:32:33.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:33.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:32:33.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:32:33.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.840 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:33.840 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:33.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.840 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:32:33.840 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:32:33.840 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 158.96046775799999, 2023-01-15T14:32:33.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:32:33.841 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:33.841 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:33.841 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:33.841 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:33.841 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:33.841 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:32:33.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.841 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:33.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:33.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:32:33.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:33.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:33.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:33.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:33.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:32:33.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:34.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:32:34.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:34.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:34.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:32:34.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:32:34.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:32:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:32:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:32:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:32:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:32:34.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:32:34.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:32:34.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:32:34.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:32:34.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:32:34.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:32:34.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:34.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:32:34.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:32:34.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:34.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:32:34.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:32:34.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:34.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:32:34.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:32:34.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:34.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:32:34.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:32:34.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:32:34.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:32:34.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:32:34.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:32:34.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:34.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.207 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:32:34.365 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:32:34.365 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 158.960409048, 2023-01-15T14:32:34.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:32:34.366 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:34.366 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:34.366 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:34.366 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:34.366 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:34.366 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:32:34.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.366 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:34.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:34.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:32:34.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:34.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:32:34.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:34.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:32:34.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:34.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:34.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:32:34.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:32:34.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:34.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:32:34.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:32:34.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:34.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:32:34.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:32:34.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:32:34.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:32:34.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:34.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:32:34.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:32:34.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:32:34.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:32:34.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:32:34.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:32:34.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:34.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:32:34.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:32:34.524 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.524 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:34.524 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.524 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 155.003989555, 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:34.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:34.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:32:34.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:34.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:32:34.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:34.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:32:34.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:34.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:34.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:32:34.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:32:34.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:34.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:32:34.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:32:34.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:34.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:32:34.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:32:34.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:32:34.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:32:34.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:32:34.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:32:34.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:34.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 154.946491872, 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:34.531 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:34.532 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:34.532 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:34.532 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:34.532 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:32:34.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.532 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:34.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:34.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:32:34.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:34.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:32:34.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:32:34.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:34.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:32:34.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:32:34.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:34.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:32:34.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:32:34.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:32:34.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:32:34.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:34.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:32:34.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:32:34.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:32:34.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:32:34.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:32:34.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:34.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:32:34.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:32:34.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:34.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:32:34.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:32:34.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:34.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:32:34.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:32:34.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:32:34.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:32:34.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:32:34.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:32:34.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:34.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:32:34.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:32:34.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:34.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:32:34.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:32:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:32:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 133.57784036699999, 2023-01-15T14:32:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:32:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:34.614 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:34.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:32:34.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:34.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:32:34.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:32:34.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:34.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:32:34.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:32:34.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:34.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:32:34.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:32:34.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:34.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:32:34.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:32:34.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:32:34.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:32:34.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:32:34.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:32:34.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:34.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:32:34.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:32:34.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:34.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:32:34.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:32:34.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:34.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:32:34.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:32:34.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:34.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:32:34.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:32:34.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:34.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:32:34.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:32:34.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:32:34.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:32:34.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:32:34.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:32:34.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:34.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:32:34.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:32:34.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:34.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:32:34.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:32:34.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.691 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:34.691 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.691 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:34.692 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:32:34.692 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 133.57773772600001, 2023-01-15T14:32:34.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:32:34.692 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:34.692 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:34.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:34.692 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:34.693 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:34.693 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:32:34.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.693 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:34.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:34.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:32:34.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:34.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:32:34.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:34.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:32:34.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:34.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:34.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:32:34.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:32:34.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:34.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:32:34.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:32:34.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:34.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:32:34.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:32:34.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:32:34.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:32:34.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:34.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:32:34.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:32:34.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:32:34.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:32:34.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:34.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:32:34.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:32:34.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:32:34.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:32:34.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:32:34.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:32:34.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:34.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:32:34.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:32:34.701 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.701 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:34.701 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.701 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:32:34.702 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:32:34.702 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 88.976755730999997, 2023-01-15T14:32:34.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:32:34.702 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:34.702 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:34.702 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:34.702 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:34.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:34.703 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:32:34.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.703 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:34.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:34.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:32:34.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:34.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:32:34.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:34.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:32:34.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:34.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:34.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:32:34.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:32:34.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:34.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:32:34.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:32:34.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:34.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:32:34.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:32:34.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:32:34.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:32:34.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:34.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:32:34.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:32:34.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:32:34.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:32:34.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:32:34.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:32:34.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:32:34.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:32:34.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:32:34.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:32:34.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:32:34.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:34.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:32:34.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:32:34.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:32:34.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:32:34.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:32:34.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:32:34.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:34.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:32:34.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:32:34.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:34.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:32:34.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:32:34.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:34.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:32:34.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:32:34.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:34.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:32:34.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:32:34.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:34.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:32:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:32:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:32:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:32:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:32:34.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.846 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:34.847 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.847 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:32:34.847 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:32:34.847 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 88.976730306999997, 2023-01-15T14:32:34.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:32:34.847 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:34.847 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:34.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:34.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:34.848 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:34.848 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:32:34.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.848 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:34.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:34.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:32:34.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:34.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:32:34.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:34.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:32:34.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:34.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:34.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:32:34.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:32:34.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:34.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:32:34.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:32:34.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:32:34.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:32:34.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:32:34.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:32:34.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:34.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:32:34.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:32:34.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:32:34.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:32:34.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:32:34.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:32:34.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:34.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:32:34.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:32:34.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:34.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:32:34.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:32:34.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:34.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:32:34.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:32:34.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:34.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:32:34.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:32:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:32:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:32:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:32:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:32:34.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:32:34.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:32:34.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:34.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:32:34.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:32:34.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2301 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e216)", 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 49.68685584, 2023-01-15T14:32:34.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61754771200000003, 2023-01-15T14:32:34.860 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:34.860 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:34.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:34.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:34.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:34.860 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2301 2023-01-15T14:32:34.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.860 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:34.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:34.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:32:34.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:34.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:32:34.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:34.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195530+0000", 2023-01-15T14:32:34.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195535+0000", 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3660000000000002e-06 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195546+0000", 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1062e-05 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:34.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195551+0000", 2023-01-15T14:32:34.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2023-01-15T14:32:34.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195999+0000", 2023-01-15T14:32:34.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044799800000000002 2023-01-15T14:32:34.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:34.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.196106+0000", 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106172 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510631+0000", 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31452575999999999 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510875+0000", 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024415199999999998 2023-01-15T14:32:34.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812445+0000", 2023-01-15T14:32:34.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30156923000000002 2023-01-15T14:32:34.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812727+0000", 2023-01-15T14:32:34.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028227800000000001 2023-01-15T14:32:34.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.813079+0000", 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199599999999998 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2667 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e259)", 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 6.9979659449999998, 2023-01-15T14:32:34.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60917505800000005, 2023-01-15T14:32:34.867 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:34.867 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:34.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:34.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:34.867 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:34.867 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2667 2023-01-15T14:32:34.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.867 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:34.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:34.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:32:34.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:34.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967296 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884423+0000", 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.905e-06 2023-01-15T14:32:34.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:34.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884427+0000", 2023-01-15T14:32:34.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7689999999999998e-06 2023-01-15T14:32:34.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:34.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884429+0000", 2023-01-15T14:32:34.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5509999999999998e-06 2023-01-15T14:32:34.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.080096+0000", 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19566640099999999 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.080290+0000", 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194346 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.275924+0000", 2023-01-15T14:32:34.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19563417299999999 2023-01-15T14:32:34.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:34.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.276184+0000", 2023-01-15T14:32:34.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025966800000000001 2023-01-15T14:32:34.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493228+0000", 2023-01-15T14:32:34.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21704441599999999 2023-01-15T14:32:34.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493378+0000", 2023-01-15T14:32:34.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014980000000000001 2023-01-15T14:32:34.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:34.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493596+0000", 2023-01-15T14:32:34.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021825300000000001 2023-01-15T14:32:34.873 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.874 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:34.874 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.874 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2748 3.3d 3:bc1c6b44:::smithi09025264-23:head [setxattr _header (58) in=7b,truncate 2537923] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:32:34.874 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:32:34.874 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.9609928679999999, 2023-01-15T14:32:34.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59151328599999997, 2023-01-15T14:32:34.874 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:34.874 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:34.874 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:34.875 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:34.875 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:34.875 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2748 2023-01-15T14:32:34.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.875 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:34.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:34.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:32:34.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:34.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:32:34.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:34.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:34.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:32:34.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:34.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921398+0000", 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3139999999999997e-06 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921407+0000", 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7050000000000005e-06 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921410+0000", 2023-01-15T14:32:34.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7459999999999998e-06 2023-01-15T14:32:34.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.007186+0000", 2023-01-15T14:32:34.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.085775343000000004 2023-01-15T14:32:34.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:34.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.007335+0000", 2023-01-15T14:32:34.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014914400000000001 2023-01-15T14:32:34.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.209977+0000", 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20264216900000001 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.210228+0000", 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025044800000000001 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:34.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.210911+0000", 2023-01-15T14:32:34.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068313300000000005 2023-01-15T14:32:34.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:34.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.211407+0000", 2023-01-15T14:32:34.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049586300000000003 2023-01-15T14:32:34.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:34.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.211425+0000", 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8403e-05 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212523+0000", 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010975710000000001 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.213095+0000", 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057295100000000004 2023-01-15T14:32:34.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.250158+0000", 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037062401000000002 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.512701+0000", 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26254333400000002 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:34.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.512801+0000", 2023-01-15T14:32:34.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9333999999999995e-05 2023-01-15T14:32:34.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:34.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.512908+0000", 2023-01-15T14:32:34.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106995 2023-01-15T14:32:34.883 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.883 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:34.883 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:34.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:34.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:34.884 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2749 3.3d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:32:34.884 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:32:34.884 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.960971064, 2023-01-15T14:32:34.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.69206326600000001, 2023-01-15T14:32:34.884 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:34.884 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2749 2023-01-15T14:32:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.003 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:35.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:32:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:32:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:32:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921418+0000", 2023-01-15T14:32:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1670000000000002e-06 2023-01-15T14:32:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921426+0000", 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0569999999999993e-06 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921429+0000", 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9629999999999998e-06 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107302+0000", 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.185873327 2023-01-15T14:32:35.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107657+0000", 2023-01-15T14:32:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035504299999999998 2023-01-15T14:32:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212694+0000", 2023-01-15T14:32:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105037095 2023-01-15T14:32:35.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212936+0000", 2023-01-15T14:32:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241492 2023-01-15T14:32:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613125+0000", 2023-01-15T14:32:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40018927900000001 2023-01-15T14:32:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613294+0000", 2023-01-15T14:32:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016838000000000001 2023-01-15T14:32:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613479+0000", 2023-01-15T14:32:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018577599999999999 2023-01-15T14:32:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.010 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.011 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.011 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:35.012 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:32:35.012 INFO:tasks.rados.rados.0.smithi090.stdout:1064: done (0 left) 2023-01-15T14:32:35.012 INFO:tasks.rados.rados.0.smithi090.stdout:1065: snap_create 2023-01-15T14:32:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:32:35.013 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:35.013 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 266, 2023-01-15T14:32:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "1c77fb2b-d1a7-4884-a24e-3935e16d0b6b", 2023-01-15T14:32:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2023-01-15T14:27:57.735717+0000", 2023-01-15T14:32:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2023-01-15T14:32:32.905287+0000", 2023-01-15T14:32:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2023-01-15T14:28:04.837522+0000", 2023-01-15T14:32:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2023-01-15T14:30:49.906949+0000", 2023-01-15T14:32:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-01-15T14:32:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5804032, 2023-01-15T14:32:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2023-01-15T14:32:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2023-01-15T14:32:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2023-01-15T14:32:35.014 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2023-01-15T14:32:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2023-01-15T14:32:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2023-01-15T14:32:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2023-01-15T14:32:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2023-01-15T14:32:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-01-15T14:32:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2023-01-15T14:32:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2023-01-15T14:32:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 3, 2023-01-15T14:32:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2023-01-15T14:32:35.015 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2023-01-15T14:32:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2023-01-15T14:32:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "reef", 2023-01-15T14:32:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2023-01-15T14:32:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2023-01-15T14:32:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2023-01-15T14:32:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:04.899282+0000", 2023-01-15T14:32:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:32:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:32:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:32:35.016 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:32:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:32:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:32:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:32:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:32:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:32:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:32:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:32:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:32:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2023-01-15T14:32:35.017 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2023-01-15T14:32:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2023-01-15T14:32:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2023-01-15T14:32:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2023-01-15T14:32:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:32:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:32:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:32:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:32:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:32:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:32:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:32:35.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "16", 2023-01-15T14:32:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:32:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:32:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:32:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:32:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:32:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:32:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:32:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:32:35.019 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:32:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:32:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:32:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:32:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:32:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:32:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:32:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:32:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:32:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:32:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:32:35.020 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:32:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:32:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:32:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:32:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:32:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:32:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:32:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:32:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:32:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:32:35.021 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:32:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:32:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:32:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:32:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:32:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:32:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:32:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:32:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2023-01-15T14:32:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2023-01-15T14:32:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2023-01-15T14:32:35.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:32:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2023-01-15T14:32:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2023-01-15T14:32:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2023-01-15T14:32:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:11.115002+0000", 2023-01-15T14:32:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:32:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:32:35.023 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:32:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:32:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:32:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:32:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:32:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:32:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:32:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:32:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:32:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:32:35.024 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:32:35.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "19", 2023-01-15T14:32:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:32:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:32:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:32:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:32:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:32:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:32:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:32:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:32:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:32:35.026 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:32:35.027 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:32:35.028 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:32:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:32:35.029 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:32:35.161 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:32:35.162 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:32:35.162 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:32:35.162 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:32:35.162 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2023-01-15T14:32:35.162 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.162 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:32:35.162 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "unique_pool_0", 2023-01-15T14:32:35.162 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:22.434903+0000", 2023-01-15T14:32:35.162 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 16385, 2023-01-15T14:32:35.163 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,pool_snaps", 2023-01-15T14:32:35.163 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:32:35.163 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:32:35.163 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:32:35.163 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:32:35.163 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:32:35.163 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:32:35.163 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:32:35.163 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 76, 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 76, 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 76, 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 76, 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 76, 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:32:35.164 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:32:35.165 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:32:35.165 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:32:35.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "266", 2023-01-15T14:32:35.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:32:35.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:32:35.165 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "254", 2023-01-15T14:32:35.165 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:32:35.165 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "pool", 2023-01-15T14:32:35.165 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 207, 2023-01-15T14:32:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 266, 2023-01-15T14:32:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [ 2023-01-15T14:32:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 199, 2023-01-15T14:32:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:21.535495+0000", 2023-01-15T14:32:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap101" 2023-01-15T14:32:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 205, 2023-01-15T14:32:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:29.932293+0000", 2023-01-15T14:32:35.166 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap104" 2023-01-15T14:32:35.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.167 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 207, 2023-01-15T14:32:35.167 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:32.584611+0000", 2023-01-15T14:32:35.167 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap105" 2023-01-15T14:32:35.167 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.167 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:35.167 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:32:35.167 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:32:35.167 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:32:35.167 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:32:35.168 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:32:35.168 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:32:35.168 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:32:35.168 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:32:35.168 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:32:35.168 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:32:35.168 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:32:35.168 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:32:35.168 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:32:35.168 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:32:35.168 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:32:35.169 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:32:35.169 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:32:35.169 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:32:35.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.169 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:32:35.169 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:32:35.169 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:32:35.169 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:32:35.169 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:32:35.169 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:32:35.169 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:32:35.170 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:32:35.170 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:32:35.170 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:32:35.170 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:32:35.170 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:32:35.170 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:32:35.170 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2023-01-15T14:32:35.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.170 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:35.170 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:32:35.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.171 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:32:35.171 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e55c5ac4-3cff-4ee5-8719-373648f83d8d", 2023-01-15T14:32:35.171 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:35.171 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:35.171 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.3336334228515625, 2023-01-15T14:32:35.171 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:35.171 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:35.171 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:35.172 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:35.172 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 261, 2023-01-15T14:32:35.172 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:35.172 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:35.172 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:35.172 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.172 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.172 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2023-01-15T14:32:35.172 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:35.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.173 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.173 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2023-01-15T14:32:35.173 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:35.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.173 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.173 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:35.173 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.173 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:35.174 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.176 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:35.176 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.176 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.176 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2023-01-15T14:32:35.176 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:35.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.176 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.176 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2023-01-15T14:32:35.176 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:35.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.177 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.177 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6811/23473", 2023-01-15T14:32:35.177 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6815/23473", 2023-01-15T14:32:35.177 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/23473", 2023-01-15T14:32:35.177 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6820/23473", 2023-01-15T14:32:35.177 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:35.177 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:35.177 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:35.177 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.178 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:32:35.178 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "2c1b33c9-0a1e-4734-936d-d51d64fe997a", 2023-01-15T14:32:35.178 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:35.178 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:35.178 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:35.178 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:35.178 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:35.178 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:35.179 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:35.179 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 262, 2023-01-15T14:32:35.179 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:35.179 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:35.179 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:35.179 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.179 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.179 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2023-01-15T14:32:35.179 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:35.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.180 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.180 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2023-01-15T14:32:35.180 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:35.180 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.180 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.180 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:35.180 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.181 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.181 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2023-01-15T14:32:35.181 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:35.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.181 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.181 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2023-01-15T14:32:35.181 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:35.181 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.181 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.182 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:35.182 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.182 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.182 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2023-01-15T14:32:35.182 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:35.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.182 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.182 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2023-01-15T14:32:35.182 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:35.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.183 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.183 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:35.183 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.183 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.183 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2023-01-15T14:32:35.183 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:35.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/23474", 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/23474", 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/23474", 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/23474", 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:35.184 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:35.185 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:35.185 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.185 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:32:35.185 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "b314174a-eca9-4baa-a60c-2589f1fc9a8e", 2023-01-15T14:32:35.185 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:35.185 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:35.185 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:35.185 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0, 2023-01-15T14:32:35.185 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:35.186 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:35.186 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:35.186 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 234, 2023-01-15T14:32:35.186 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:35.186 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:35.186 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:35.186 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.186 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.186 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2023-01-15T14:32:35.186 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:35.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.187 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.187 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2023-01-15T14:32:35.187 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:35.187 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.187 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.187 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:35.187 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.188 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.189 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2023-01-15T14:32:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.190 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2023-01-15T14:32:35.191 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:35.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.191 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.191 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6810/23475", 2023-01-15T14:32:35.191 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/23475", 2023-01-15T14:32:35.191 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6821/23475", 2023-01-15T14:32:35.191 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6817/23475", 2023-01-15T14:32:35.191 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:35.191 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:35.191 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:32:35.192 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "29eb0b17-ed5d-4665-a347-2fb8406d9c62", 2023-01-15T14:32:35.326 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:35.326 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:35.326 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:35.326 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.9044189453125, 2023-01-15T14:32:35.326 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:35.326 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:35.326 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:35.326 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 260, 2023-01-15T14:32:35.326 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:35.326 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:35.327 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:35.327 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.327 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.327 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6824", 2023-01-15T14:32:35.327 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:35.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.327 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.327 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6825", 2023-01-15T14:32:35.327 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:35.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.328 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.328 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:35.328 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.328 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.328 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6826", 2023-01-15T14:32:35.328 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:35.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6827", 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6830", 2023-01-15T14:32:35.329 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:35.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.330 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.330 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6831", 2023-01-15T14:32:35.330 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:35.330 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.330 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.330 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:35.330 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6828", 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6829", 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.331 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6825/23778", 2023-01-15T14:32:35.332 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6827/23778", 2023-01-15T14:32:35.332 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6831/23778", 2023-01-15T14:32:35.332 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6829/23778", 2023-01-15T14:32:35.332 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:35.332 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:35.332 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:35.332 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.332 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:32:35.332 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "59362feb-caa9-44e3-b6d0-b79397dacbfc", 2023-01-15T14:32:35.333 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:35.333 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:35.333 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:35.333 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:35.333 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:35.333 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:35.333 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:35.333 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 260, 2023-01-15T14:32:35.333 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:35.333 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:35.333 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6800", 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6801", 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.334 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.335 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:35.335 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.335 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.335 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6802", 2023-01-15T14:32:35.335 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:35.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.335 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.335 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6803", 2023-01-15T14:32:35.336 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:35.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.336 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.336 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:35.336 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.336 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.336 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6806", 2023-01-15T14:32:35.336 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:35.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.337 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.337 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6807", 2023-01-15T14:32:35.337 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:35.337 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.337 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.337 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:35.337 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.337 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6804", 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6805", 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6801/23523", 2023-01-15T14:32:35.338 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6803/23523", 2023-01-15T14:32:35.339 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6807/23523", 2023-01-15T14:32:35.339 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6805/23523", 2023-01-15T14:32:35.339 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:35.339 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:35.339 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:35.339 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.339 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:32:35.339 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "fbc8821c-8d78-40f8-a571-b18a70d42000", 2023-01-15T14:32:35.339 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:35.340 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:32:35.340 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:32:35.340 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:35.340 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:35.340 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:35.340 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:35.340 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:32:35.340 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:35.340 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:35.340 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:35.340 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.341 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.341 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6824", 2023-01-15T14:32:35.341 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:35.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.341 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.341 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6825", 2023-01-15T14:32:35.341 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:35.341 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.341 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.342 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:35.342 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.342 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.342 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6826", 2023-01-15T14:32:35.342 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:35.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.342 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.342 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6827", 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6830", 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6831", 2023-01-15T14:32:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.344 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6828", 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6829", 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6825/23525", 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6827/23525", 2023-01-15T14:32:35.345 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6831/23525", 2023-01-15T14:32:35.346 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6829/23525", 2023-01-15T14:32:35.346 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:35.346 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:35.346 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:35.346 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.346 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:32:35.346 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "34550470-3561-492f-b605-b5f0e8052c1e", 2023-01-15T14:32:35.346 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:35.346 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 118, 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.347 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.348 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6808", 2023-01-15T14:32:35.348 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:35.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.348 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.348 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6809", 2023-01-15T14:32:35.348 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:35.348 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.348 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.348 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6810", 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6811", 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.349 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.350 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:35.350 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.350 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.350 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6814", 2023-01-15T14:32:35.350 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:35.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.350 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.350 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6815", 2023-01-15T14:32:35.351 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:35.351 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.351 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.351 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:35.351 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.351 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.351 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6812", 2023-01-15T14:32:35.351 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:35.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6813", 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6809/23524", 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6811/23524", 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6815/23524", 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6813/23524", 2023-01-15T14:32:35.352 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:35.353 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:35.353 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:35.353 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.353 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:32:35.353 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1f123d0d-d617-4771-b516-21aa5a29b734", 2023-01-15T14:32:35.353 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:35.353 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:35.353 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.5779571533203125, 2023-01-15T14:32:35.353 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:35.354 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:35.354 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:35.354 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:35.354 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 263, 2023-01-15T14:32:35.354 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:35.354 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:35.354 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:35.354 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.354 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.354 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6816", 2023-01-15T14:32:35.355 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:35.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.355 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.355 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6817", 2023-01-15T14:32:35.355 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:35.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.355 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.355 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:35.355 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6818", 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6819", 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.357 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:35.357 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.357 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.357 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6822", 2023-01-15T14:32:35.357 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:35.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.357 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.357 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6823", 2023-01-15T14:32:35.357 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:35.358 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.358 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.358 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:35.358 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:35.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.358 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:35.358 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6820", 2023-01-15T14:32:35.358 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:35.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6821", 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6817/23526", 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6819/23526", 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6823/23526", 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6821/23526", 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:35.359 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:35.360 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:35.361 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:47.556519+0000", 2023-01-15T14:32:35.361 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:35.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.361 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:32:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:35.486 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:54.007074+0000", 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:35.487 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:53.308067+0000", 2023-01-15T14:32:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:32:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:35.488 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:43.268238+0000", 2023-01-15T14:32:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:32:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:35.489 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:51.428540+0000", 2023-01-15T14:32:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:32:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:32:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:27.525459+0000", 2023-01-15T14:32:35.490 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.491 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:32:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:35.492 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:32:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:55.585270+0000", 2023-01-15T14:32:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:32:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:35.493 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:39.052183+0000", 2023-01-15T14:32:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [ 2023-01-15T14:32:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.1", 2023-01-15T14:32:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:32:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: 1, 2023-01-15T14:32:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: 7, 2023-01-15T14:32:35.494 INFO:teuthology.orchestra.run.smithi090.stdout: 2 2023-01-15T14:32:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2023-01-15T14:32:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.2d", 2023-01-15T14:32:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:32:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.495 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 2, 2023-01-15T14:32:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 3 2023-01-15T14:32:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2023-01-15T14:32:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2023-01-15T14:32:35.503 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2023-01-15T14:32:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2023-01-15T14:32:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2023-01-15T14:32:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2023-01-15T14:32:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2023-01-15T14:32:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2023-01-15T14:32:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2023-01-15T14:32:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2023-01-15T14:32:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2023-01-15T14:32:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2023-01-15T14:32:35.504 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2023-01-15T14:32:35.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.648 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:32:35.648 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2023-01-15T14:32:35.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 188, 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 192, 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 200, 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:35.649 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 204, 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2023-01-15T14:32:35.650 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2023-01-15T14:32:35.651 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2023-01-15T14:32:35.651 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2023-01-15T14:32:35.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:35.651 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:35.652 INFO:tasks.thrashosds.thrasher:j is {'epoch': 266, 'fsid': '1c77fb2b-d1a7-4884-a24e-3935e16d0b6b', 'created': '2023-01-15T14:27:57.735717+0000', 'modified': '2023-01-15T14:32:32.905287+0000', 'last_up_change': '2023-01-15T14:28:04.837522+0000', 'last_in_change': '2023-01-15T14:30:49.906949+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-01-15T14:28:04.899282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-01-15T14:28:11.115002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-01-15T14:28:22.434903+0000', 'flags': 16385, 'flags_names': 'hashpspool,pool_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 76, 'pg_placement_num': 76, 'pg_placement_num_target': 76, 'pg_num_target': 76, 'pg_num_pending': 76, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '266', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '254', 'auid': 0, 'snap_mode': 'pool', 'snap_seq': 207, 'snap_epoch': 266, 'pool_snaps': [{'snapid': 199, 'stamp': '2023-01-15T14:32:21.535495+0000', 'name': 'smithi09025264-snap101'}, {'snapid': 205, 'stamp': '2023-01-15T14:32:29.932293+0000', 'name': 'smithi09025264-snap104'}, {'snapid': 207, 'stamp': '2023-01-15T14:32:32.584611+0000', 'name': 'smithi09025264-snap105'}], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e55c5ac4-3cff-4ee5-8719-373648f83d8d', 'up': 1, 'in': 1, 'weight': 0.3336334228515625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 23473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 23473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 23473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6820', 'nonce': 23473}]}, 'public_addr': '172.21.15.90:6811/23473', 'cluster_addr': '172.21.15.90:6815/23473', 'heartbeat_back_addr': '172.21.15.90:6823/23473', 'heartbeat_front_addr': '172.21.15.90:6820/23473', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2c1b33c9-0a1e-4734-936d-d51d64fe997a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 262, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 23474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 23474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 23474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 23474}]}, 'public_addr': '172.21.15.90:6801/23474', 'cluster_addr': '172.21.15.90:6803/23474', 'heartbeat_back_addr': '172.21.15.90:6807/23474', 'heartbeat_front_addr': '172.21.15.90:6805/23474', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b314174a-eca9-4baa-a60c-2589f1fc9a8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 234, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 23475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 23475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6819', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 23475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 23475}]}, 'public_addr': '172.21.15.90:6810/23475', 'cluster_addr': '172.21.15.90:6814/23475', 'heartbeat_back_addr': '172.21.15.90:6821/23475', 'heartbeat_front_addr': '172.21.15.90:6817/23475', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '29eb0b17-ed5d-4665-a347-2fb8406d9c62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9044189453125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 260, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6824', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6825', 'nonce': 23778}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6826', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6827', 'nonce': 23778}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6830', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6831', 'nonce': 23778}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6828', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6829', 'nonce': 23778}]}, 'public_addr': '172.21.15.90:6825/23778', 'cluster_addr': '172.21.15.90:6827/23778', 'heartbeat_back_addr': '172.21.15.90:6831/23778', 'heartbeat_front_addr': '172.21.15.90:6829/23778', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '59362feb-caa9-44e3-b6d0-b79397dacbfc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 260, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6800', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6801', 'nonce': 23523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6802', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6803', 'nonce': 23523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6806', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6807', 'nonce': 23523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6804', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6805', 'nonce': 23523}]}, 'public_addr': '172.21.15.141:6801/23523', 'cluster_addr': '172.21.15.141:6803/23523', 'heartbeat_back_addr': '172.21.15.141:6807/23523', 'heartbeat_front_addr': '172.21.15.141:6805/23523', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fbc8821c-8d78-40f8-a571-b18a70d42000', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6824', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6825', 'nonce': 23525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6826', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6827', 'nonce': 23525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6830', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6831', 'nonce': 23525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6828', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6829', 'nonce': 23525}]}, 'public_addr': '172.21.15.141:6825/23525', 'cluster_addr': '172.21.15.141:6827/23525', 'heartbeat_back_addr': '172.21.15.141:6831/23525', 'heartbeat_front_addr': '172.21.15.141:6829/23525', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '34550470-3561-492f-b605-b5f0e8052c1e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6808', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6809', 'nonce': 23524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6810', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6811', 'nonce': 23524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6814', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6815', 'nonce': 23524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6812', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6813', 'nonce': 23524}]}, 'public_addr': '172.21.15.141:6809/23524', 'cluster_addr': '172.21.15.141:6811/23524', 'heartbeat_back_addr': '172.21.15.141:6815/23524', 'heartbeat_front_addr': '172.21.15.141:6813/23524', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f123d0d-d617-4771-b516-21aa5a29b734', 'up': 1, 'in': 1, 'weight': 0.5779571533203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 263, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6816', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6817', 'nonce': 23526}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6818', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6819', 'nonce': 23526}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6822', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6823', 'nonce': 23526}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6820', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6821', 'nonce': 23526}]}, 'public_addr': '172.21.15.141:6817/23526', 'cluster_addr': '172.21.15.141:6819/23526', 'heartbeat_back_addr': '172.21.15.141:6823/23526', 'heartbeat_front_addr': '172.21.15.141:6821/23526', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:47.556519+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:54.007074+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:53.308067+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:43.268238+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:51.428540+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:31:27.525459+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:31:55.585270+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:39.052183+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.1', 'osds': [1, 7, 2]}], 'pg_upmap_items': [{'pgid': '3.2d', 'mappings': [{'from': 2, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 188, 'length': 1}, {'begin': 192, 'length': 1}, {'begin': 200, 'length': 1}, {'begin': 204, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-01-15T14:32:35.652 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.1 2023-01-15T14:32:35.652 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap 3.1 2023-01-15T14:32:35.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:32:35.653 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:32:35.654 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:32.933+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:35.655 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:35.648+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:35.657 INFO:tasks.rados.rados.0.smithi090.stdout:1065: done (0 left) 2023-01-15T14:32:35.657 INFO:tasks.rados.rados.0.smithi090.stdout:1066: write oid 14 current snap is 106 2023-01-15T14:32:35.658 INFO:tasks.rados.rados.0.smithi090.stdout:1066: seq_num 395 ranges {638765=457358,1748152=604803} 2023-01-15T14:32:35.658 INFO:tasks.rados.rados.0.smithi090.stdout:1066: writing smithi09025264-14 from 638765 to 1096123 tid 1 2023-01-15T14:32:35.659 INFO:tasks.rados.rados.0.smithi090.stdout:1066: writing smithi09025264-14 from 1748152 to 2352955 tid 2 2023-01-15T14:32:35.659 INFO:tasks.rados.rados.0.smithi090.stdout:1067: write (excl) oid 31 current snap is 106 2023-01-15T14:32:35.659 INFO:tasks.rados.rados.0.smithi090.stdout:1067: seq_num 396 ranges {655572=648960,1922987=732585} 2023-01-15T14:32:35.659 INFO:tasks.rados.rados.0.smithi090.stdout:1067: writing smithi09025264-31 from 655572 to 1304532 tid 1 2023-01-15T14:32:35.659 INFO:tasks.rados.rados.0.smithi090.stdout:1067: writing smithi09025264-31 from 1922987 to 2655572 tid 2 2023-01-15T14:32:35.659 INFO:tasks.rados.rados.0.smithi090.stdout:1068: copy_from oid 41 from oid 17 current snap is 106 2023-01-15T14:32:35.659 INFO:tasks.rados.rados.0.smithi090.stdout:1069: write oid 48 current snap is 106 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1069: seq_num 397 ranges {672379=724209,1865116=608472} 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1069: writing smithi09025264-48 from 672379 to 1396588 tid 1 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1069: writing smithi09025264-48 from 1865116 to 2473588 tid 2 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1070: read oid 37 snap -1 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1070: expect deleted 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1071: snap_remove snap 100 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1071: done (5 left) 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1072: write oid 29 current snap is 106 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1067: finishing write tid 1 to smithi09025264-31 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1072: seq_num 398 ranges {689186=799458,1923598=752130} 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1072: writing smithi09025264-29 from 689186 to 1488644 tid 1 2023-01-15T14:32:35.660 INFO:tasks.rados.rados.0.smithi090.stdout:1072: writing smithi09025264-29 from 1923598 to 2675728 tid 2 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1073: rollback oid 35 current snap is 106 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 35 to 104 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1066: finishing write tid 1 to smithi09025264-14 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1067: finishing write tid 2 to smithi09025264-31 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1067: finishing write tid 3 to smithi09025264-31 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1066: finishing write tid 2 to smithi09025264-14 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1069: finishing write tid 1 to smithi09025264-48 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1070: done (6 left) 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1074: delete oid 18 current snap is 106 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1066: finishing write tid 3 to smithi09025264-14 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1066: finishing write tid 4 to smithi09025264-14 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1066: oid 14 updating version 0 to 258 2023-01-15T14:32:35.661 INFO:tasks.rados.rados.0.smithi090.stdout:1066: oid 14 updating version 258 to 259 2023-01-15T14:32:35.662 INFO:tasks.rados.rados.0.smithi090.stdout:1066: oid 14 updating version 259 to 260 2023-01-15T14:32:35.662 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 260 (ObjNum 395 snap 106 seq_num 395) dirty exists 2023-01-15T14:32:35.662 INFO:tasks.rados.rados.0.smithi090.stdout:1066: left oid 14 (ObjNum 395 snap 106 seq_num 395) 2023-01-15T14:32:35.662 INFO:tasks.rados.rados.0.smithi090.stdout:1068: finishing copy_from to smithi09025264-41 2023-01-15T14:32:35.662 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 342 (ObjNum 345 snap 91 seq_num 345) dirty exists 2023-01-15T14:32:35.662 INFO:tasks.rados.rados.0.smithi090.stdout:1068: finishing copy_from racing read to smithi09025264-41 2023-01-15T14:32:35.662 INFO:tasks.rados.rados.0.smithi090.stdout:1067: finishing write tid 4 to smithi09025264-31 2023-01-15T14:32:35.662 INFO:tasks.rados.rados.0.smithi090.stdout:1067: oid 31 updating version 0 to 352 2023-01-15T14:32:35.662 INFO:tasks.rados.rados.0.smithi090.stdout:1067: oid 31 updating version 352 to 354 2023-01-15T14:32:35.662 INFO:tasks.rados.rados.0.smithi090.stdout:1067: oid 31 version 354 is already newer than 353 2023-01-15T14:32:35.662 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 354 (ObjNum 396 snap 106 seq_num 396) dirty exists 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:1067: left oid 31 (ObjNum 396 snap 106 seq_num 396) 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:1069: finishing write tid 2 to smithi09025264-48 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:1069: finishing write tid 3 to smithi09025264-48 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:1069: finishing write tid 4 to smithi09025264-48 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:1069: oid 48 updating version 0 to 197 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:1069: oid 48 version 197 is already newer than 196 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:1069: oid 48 version 197 is already newer than 195 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 197 (ObjNum 397 snap 106 seq_num 397) dirty exists 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:1069: left oid 48 (ObjNum 397 snap 106 seq_num 397) 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:1073: finishing rollback tid 0 to smithi09025264-35 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:1073: finishing rollback tid 1 to smithi09025264-35 2023-01-15T14:32:35.663 INFO:tasks.rados.rados.0.smithi090.stdout:1073: finishing rollback tid 2 to smithi09025264-35 2023-01-15T14:32:35.664 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 291 (ObjNum 373 snap 100 seq_num 373) dirty exists 2023-01-15T14:32:35.664 INFO:tasks.rados.rados.0.smithi090.stdout:1066: done (6 left) 2023-01-15T14:32:35.664 INFO:tasks.rados.rados.0.smithi090.stdout:1067: done (5 left) 2023-01-15T14:32:35.664 INFO:tasks.rados.rados.0.smithi090.stdout:1068: done (4 left) 2023-01-15T14:32:35.664 INFO:tasks.rados.rados.0.smithi090.stdout:1069: done (3 left) 2023-01-15T14:32:35.664 INFO:tasks.rados.rados.0.smithi090.stdout:1073: done (2 left) 2023-01-15T14:32:35.664 INFO:tasks.rados.rados.0.smithi090.stdout:1074: done (1 left) 2023-01-15T14:32:35.664 INFO:tasks.rados.rados.0.smithi090.stdout:1075: copy_from oid 38 from oid 48 current snap is 106 2023-01-15T14:32:35.664 INFO:tasks.rados.rados.0.smithi090.stdout:1076: read oid 5 snap -1 2023-01-15T14:32:35.664 INFO:tasks.rados.rados.0.smithi090.stdout:1076: expect (ObjNum 387 snap 103 seq_num 387) 2023-01-15T14:32:35.664 INFO:tasks.rados.rados.0.smithi090.stdout:1077: snap_remove snap 105 2023-01-15T14:32:35.755 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:35.750+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:35.815 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:35.815 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:35.815 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:32:35.815 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:35.815 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:32:35.818 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:35.836 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:35.969 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:35.969 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:35.970 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:35.970 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:35.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:32:36.132 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:36.133 INFO:tasks.rados.rados.0.smithi090.stdout:1077: done (3 left) 2023-01-15T14:32:36.133 INFO:tasks.rados.rados.0.smithi090.stdout:1078: read oid 40 snap -1 2023-01-15T14:32:36.133 INFO:tasks.rados.rados.0.smithi090.stdout:1078: expect (ObjNum 385 snap 103 seq_num 385) 2023-01-15T14:32:36.133 INFO:tasks.rados.rados.0.smithi090.stdout:1079: write (excl) oid 45 current snap is 106 2023-01-15T14:32:36.133 INFO:tasks.rados.rados.0.smithi090.stdout:1079: seq_num 400 ranges {722800=549956,2040562=682238} 2023-01-15T14:32:36.134 INFO:tasks.rados.rados.0.smithi090.stdout:1079: writing smithi09025264-45 from 722800 to 1272756 tid 1 2023-01-15T14:32:36.134 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:36.137 INFO:tasks.rados.rados.0.smithi090.stdout:1079: writing smithi09025264-45 from 2040562 to 2722800 tid 2 2023-01-15T14:32:36.137 INFO:tasks.rados.rados.0.smithi090.stdout:1080: write oid 37 current snap is 106 2023-01-15T14:32:36.137 INFO:tasks.rados.rados.0.smithi090.stdout:1080: seq_num 401 ranges {739607=625205,1982691=499457} 2023-01-15T14:32:36.137 INFO:tasks.rados.rados.0.smithi090.stdout:1080: writing smithi09025264-37 from 739607 to 1364812 tid 1 2023-01-15T14:32:36.138 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:36.139 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:35.964+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:36.139 INFO:tasks.rados.rados.0.smithi090.stdout:1080: writing smithi09025264-37 from 1982691 to 2482148 tid 2 2023-01-15T14:32:36.146 INFO:tasks.rados.rados.0.smithi090.stdout:1081: read oid 25 snap -1 2023-01-15T14:32:36.146 INFO:tasks.rados.rados.0.smithi090.stdout:1081: expect (ObjNum 393 snap 104 seq_num 393) 2023-01-15T14:32:36.146 INFO:tasks.rados.rados.0.smithi090.stdout:1082: snap_create 2023-01-15T14:32:36.146 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:32:36.147 INFO:tasks.rados.rados.0.smithi090.stdout:1076: expect (ObjNum 387 snap 103 seq_num 387) 2023-01-15T14:32:36.147 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:36.148 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:32:36.148 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:32:36.148 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:32:36.148 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:32:36.148 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:32:36.148 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:32:36.148 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:32:36.149 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:32:36.149 INFO:tasks.rados.rados.0.smithi090.stdout:1072: finishing write tid 1 to smithi09025264-29 2023-01-15T14:32:36.149 INFO:tasks.rados.rados.0.smithi090.stdout:1072: finishing write tid 2 to smithi09025264-29 2023-01-15T14:32:36.150 INFO:tasks.rados.rados.0.smithi090.stdout:1072: finishing write tid 3 to smithi09025264-29 2023-01-15T14:32:36.150 INFO:tasks.rados.rados.0.smithi090.stdout:1072: finishing write tid 4 to smithi09025264-29 2023-01-15T14:32:36.150 INFO:tasks.rados.rados.0.smithi090.stdout:1072: oid 29 updating version 0 to 262 2023-01-15T14:32:36.150 INFO:tasks.rados.rados.0.smithi090.stdout:1072: oid 29 version 262 is already newer than 260 2023-01-15T14:32:36.150 INFO:tasks.rados.rados.0.smithi090.stdout:1072: oid 29 version 262 is already newer than 261 2023-01-15T14:32:36.151 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 262 (ObjNum 398 snap 106 seq_num 398) dirty exists 2023-01-15T14:32:36.151 INFO:tasks.rados.rados.0.smithi090.stdout:1072: left oid 29 (ObjNum 398 snap 106 seq_num 398) 2023-01-15T14:32:36.151 INFO:tasks.rados.rados.0.smithi090.stdout:1075: finishing copy_from to smithi09025264-38 2023-01-15T14:32:36.151 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 197 (ObjNum 397 snap 106 seq_num 397) dirty exists 2023-01-15T14:32:36.151 INFO:tasks.rados.rados.0.smithi090.stdout:1075: finishing copy_from racing read to smithi09025264-38 2023-01-15T14:32:36.151 INFO:tasks.rados.rados.0.smithi090.stdout:1078: expect (ObjNum 385 snap 103 seq_num 385) 2023-01-15T14:32:36.152 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:36.132+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:36.240 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:36.234+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:36.299 INFO:tasks.rados.rados.0.smithi090.stdout:1081: expect (ObjNum 393 snap 104 seq_num 393) 2023-01-15T14:32:36.303 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:36.304 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:36.313 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:32:36.341 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:36.336+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:36.445 INFO:tasks.rados.rados.0.smithi090.stdout:1080: finishing write tid 1 to smithi09025264-37 2023-01-15T14:32:36.445 INFO:tasks.rados.rados.0.smithi090.stdout:1079: finishing write tid 1 to smithi09025264-45 2023-01-15T14:32:36.445 INFO:tasks.rados.rados.0.smithi090.stdout:1072: done (6 left) 2023-01-15T14:32:36.445 INFO:tasks.rados.rados.0.smithi090.stdout:1075: done (5 left) 2023-01-15T14:32:36.445 INFO:tasks.rados.rados.0.smithi090.stdout:1076: done (4 left) 2023-01-15T14:32:36.445 INFO:tasks.rados.rados.0.smithi090.stdout:1078: done (3 left) 2023-01-15T14:32:36.445 INFO:tasks.rados.rados.0.smithi090.stdout:1081: done (2 left) 2023-01-15T14:32:36.445 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:36.446 INFO:tasks.rados.rados.0.smithi090.stdout:1080: finishing write tid 2 to smithi09025264-37 2023-01-15T14:32:36.446 INFO:tasks.rados.rados.0.smithi090.stdout:1080: finishing write tid 3 to smithi09025264-37 2023-01-15T14:32:36.447 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:36.447 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:36.447 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:32:36.449 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:36.440+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:36.471 INFO:tasks.rados.rados.0.smithi090.stdout:1080: finishing write tid 4 to smithi09025264-37 2023-01-15T14:32:36.471 INFO:tasks.rados.rados.0.smithi090.stdout:1080: oid 37 updating version 0 to 323 2023-01-15T14:32:36.471 INFO:tasks.rados.rados.0.smithi090.stdout:1080: oid 37 updating version 323 to 324 2023-01-15T14:32:36.472 INFO:tasks.rados.rados.0.smithi090.stdout:1080: oid 37 updating version 324 to 325 2023-01-15T14:32:36.472 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 325 (ObjNum 401 snap 106 seq_num 401) dirty exists 2023-01-15T14:32:36.472 INFO:tasks.rados.rados.0.smithi090.stdout:1080: left oid 37 (ObjNum 401 snap 106 seq_num 401) 2023-01-15T14:32:36.472 INFO:tasks.rados.rados.0.smithi090.stdout:1080: done (1 left) 2023-01-15T14:32:36.472 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:36.476 INFO:tasks.rados.rados.0.smithi090.stdout:1079: finishing write tid 2 to smithi09025264-45 2023-01-15T14:32:36.476 INFO:tasks.rados.rados.0.smithi090.stdout:1079: finishing write tid 3 to smithi09025264-45 2023-01-15T14:32:36.477 INFO:tasks.rados.rados.0.smithi090.stdout:1079: finishing write tid 4 to smithi09025264-45 2023-01-15T14:32:36.477 INFO:tasks.rados.rados.0.smithi090.stdout:1079: oid 45 updating version 0 to 138 2023-01-15T14:32:36.477 INFO:tasks.rados.rados.0.smithi090.stdout:1079: oid 45 updating version 138 to 139 2023-01-15T14:32:36.477 INFO:tasks.rados.rados.0.smithi090.stdout:1079: oid 45 updating version 139 to 140 2023-01-15T14:32:36.477 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 140 (ObjNum 400 snap 106 seq_num 400) dirty exists 2023-01-15T14:32:36.477 INFO:tasks.rados.rados.0.smithi090.stdout:1079: left oid 45 (ObjNum 400 snap 106 seq_num 400) 2023-01-15T14:32:36.477 INFO:tasks.rados.rados.0.smithi090.stdout:1079: done (0 left) 2023-01-15T14:32:36.546 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:36.538+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:36.609 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:36.610 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:36.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:32:36.647 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:36.644+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:36.720 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:36.772 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:36.774 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:32:36.774 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:36.766+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:36.848 INFO:tasks.rados.rados.0.smithi090.stdout:1082: done (0 left) 2023-01-15T14:32:36.848 INFO:tasks.rados.rados.0.smithi090.stdout:1083: snap_remove snap 104 2023-01-15T14:32:36.856 INFO:teuthology.orchestra.run.smithi090.stderr:clear 3.1 pg_upmap mapping 2023-01-15T14:32:36.863 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:32:36.875 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:36.870+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:36.934 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:36.934 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:36.934 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:32:36.975 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:36.968+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:37.060 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:37.060 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:37.061 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:32:37.077 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:37.070+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:37.164 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:37.216 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:37.217 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:32:37.218 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:37.212+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:37.318 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:37.314+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:37.327 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:37.328 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:37.337 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:32:37.419 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:37.414+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:37.438 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:37.438 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:37.448 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:32:37.520 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:37.514+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:37.642 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:37.642 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:37.642 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:32:37.643 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:37.634+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:37.743 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:37.738+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:37.847 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:37.847 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:37.847 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:32:37.849 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:37.840+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:37.854 INFO:tasks.rados.rados.0.smithi090.stdout:1083: done (0 left) 2023-01-15T14:32:37.854 INFO:tasks.rados.rados.0.smithi090.stdout:1084: snap_create 2023-01-15T14:32:37.947 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:37.940+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:38.008 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:38.008 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:38.008 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:32:38.049 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:38.044+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:38.149 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:38.144+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:38.177 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:32:38.179 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:38.179 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:38.180 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:32:38.250 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:38.244+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:38.343 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:38.343 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:38.344 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:38.344 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:32:38.351 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:38.351 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:38.346+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:38.370 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:38.389 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:38.502 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:38.502 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:38.503 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:38.503 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:38.503 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:38.503 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:38.504 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:32:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:32:38.504 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:32:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:32:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:32:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:32:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:32:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:32:38.505 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:32:38.505 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:38.498+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:38.603 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:38.596+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:38.682 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:38.682 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:32:38.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:32:38.683 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:32:38.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.683 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:75 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:38.683 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:32:38.683 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 252.25651924600001, 2023-01-15T14:32:38.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92335155400000002, 2023-01-15T14:32:38.684 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:38.684 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:38.684 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:38.684 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:38.684 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:38.684 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 75 2023-01-15T14:32:38.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.685 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:38.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:38.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:32:38.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:38.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:38.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:32:38.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:38.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:38.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356896+0000", 2023-01-15T14:32:38.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:38.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356903+0000", 2023-01-15T14:32:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4619999999999999e-06 2023-01-15T14:32:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356917+0000", 2023-01-15T14:32:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4195999999999999e-05 2023-01-15T14:32:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356926+0000", 2023-01-15T14:32:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:32:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457743+0000", 2023-01-15T14:32:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100817668 2023-01-15T14:32:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457841+0000", 2023-01-15T14:32:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7079999999999999e-05 2023-01-15T14:32:38.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.693 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.693 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778474+0000", 2023-01-15T14:32:38.693 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32063301799999999 2023-01-15T14:32:38.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:38.694 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778627+0000", 2023-01-15T14:32:38.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015389599999999999 2023-01-15T14:32:38.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:38.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778815+0000", 2023-01-15T14:32:38.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018726600000000001 2023-01-15T14:32:38.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:38.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779033+0000", 2023-01-15T14:32:38.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021819999999999999 2023-01-15T14:32:38.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.696 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:38.696 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779051+0000", 2023-01-15T14:32:38.696 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7640000000000001e-05 2023-01-15T14:32:38.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.700 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:38.700 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:32:38.700 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3590999999999995e-05 2023-01-15T14:32:38.701 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.701 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.701 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:38.701 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779163+0000", 2023-01-15T14:32:38.701 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:32:38.702 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.702 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.702 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:38.702 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.081002+0000", 2023-01-15T14:32:38.702 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30183937799999999 2023-01-15T14:32:38.702 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.702 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.703 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:38.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280043+0000", 2023-01-15T14:32:38.703 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19904084699999999 2023-01-15T14:32:38.703 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.703 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.703 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:38.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280138+0000", 2023-01-15T14:32:38.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4915999999999997e-05 2023-01-15T14:32:38.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.704 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:38.704 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280248+0000", 2023-01-15T14:32:38.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110233 2023-01-15T14:32:38.705 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:38.705 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:38.705 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:38.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.705 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.705 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:76 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:38.873 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:32:38.873 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 252.25647763800001, 2023-01-15T14:32:38.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92420393899999997, 2023-01-15T14:32:38.873 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:38.873 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:38.873 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 76 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356937+0000", 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356948+0000", 2023-01-15T14:32:38.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.094e-05 2023-01-15T14:32:38.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:38.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356959+0000", 2023-01-15T14:32:38.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1058e-05 2023-01-15T14:32:38.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:38.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356968+0000", 2023-01-15T14:32:38.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5339999999999999e-06 2023-01-15T14:32:38.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.879 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.879 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459524+0000", 2023-01-15T14:32:38.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102556147 2023-01-15T14:32:38.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.879 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.879 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:38.879 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459662+0000", 2023-01-15T14:32:38.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000137756 2023-01-15T14:32:38.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978621+0000", 2023-01-15T14:32:38.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51895887600000001 2023-01-15T14:32:38.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:38.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978870+0000", 2023-01-15T14:32:38.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024928999999999999 2023-01-15T14:32:38.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280761+0000", 2023-01-15T14:32:38.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30189113400000001 2023-01-15T14:32:38.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:38.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281023+0000", 2023-01-15T14:32:38.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026172800000000003 2023-01-15T14:32:38.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:38.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281142+0000", 2023-01-15T14:32:38.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000119296 2023-01-15T14:32:38.882 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:38.883 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:38.883 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:38.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.883 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1076 3.f 3:f87c0183:::smithi09025264-50:head [setxattr _header (57) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:38.883 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:32:38.883 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 175.85476888299999, 2023-01-15T14:32:38.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.88473693200000003, 2023-01-15T14:32:38.883 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:38.883 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1076 2023-01-15T14:32:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:32:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:38.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:38.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:32:38.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:38.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:38.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758646+0000", 2023-01-15T14:32:38.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:38.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:38.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758653+0000", 2023-01-15T14:32:38.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8029999999999997e-06 2023-01-15T14:32:38.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:38.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758670+0000", 2023-01-15T14:32:38.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6158999999999999e-05 2023-01-15T14:32:38.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:38.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758677+0000", 2023-01-15T14:32:38.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9840000000000004e-06 2023-01-15T14:32:38.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848049+0000", 2023-01-15T14:32:38.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089372163000000004 2023-01-15T14:32:38.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:38.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848056+0000", 2023-01-15T14:32:38.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0360000000000001e-06 2023-01-15T14:32:38.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951562+0000", 2023-01-15T14:32:38.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10350670100000001 2023-01-15T14:32:38.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:38.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951726+0000", 2023-01-15T14:32:38.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000163311 2023-01-15T14:32:38.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:38.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951958+0000", 2023-01-15T14:32:38.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000232525 2023-01-15T14:32:38.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:38.891 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952168+0000", 2023-01-15T14:32:38.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020949799999999999 2023-01-15T14:32:38.891 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:38.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952186+0000", 2023-01-15T14:32:38.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7714e-05 2023-01-15T14:32:38.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:38.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952254+0000", 2023-01-15T14:32:38.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7998999999999997e-05 2023-01-15T14:32:38.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952278+0000", 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4909000000000002e-05 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065836+0000", 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.113557081 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643196+0000", 2023-01-15T14:32:38.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.57736012999999997 2023-01-15T14:32:38.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:38.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643278+0000", 2023-01-15T14:32:38.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2671999999999997e-05 2023-01-15T14:32:38.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:38.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643384+0000", 2023-01-15T14:32:38.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010551400000000001 2023-01-15T14:32:38.894 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:38.895 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:38.895 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:38.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.895 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1077 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:38.895 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:32:38.895 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 175.85473006399999, 2023-01-15T14:32:38.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.98557123499999999, 2023-01-15T14:32:38.895 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:38.895 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:38.895 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:38.896 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:38.896 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:38.896 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1077 2023-01-15T14:32:38.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.896 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:38.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:38.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:32:38.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:38.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:38.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:32:38.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:38.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:38.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758685+0000", 2023-01-15T14:32:38.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:38.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:38.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758689+0000", 2023-01-15T14:32:38.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2760000000000002e-06 2023-01-15T14:32:38.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:38.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758704+0000", 2023-01-15T14:32:38.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4187e-05 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758710+0000", 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4330000000000001e-06 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848086+0000", 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089376466000000002 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848093+0000", 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4269999999999996e-06 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952346+0000", 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104252758 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952427+0000", 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1786000000000006e-05 2023-01-15T14:32:38.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743602+0000", 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.79117428099999998 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743892+0000", 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029067200000000001 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:38.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.744257+0000", 2023-01-15T14:32:38.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000364619 2023-01-15T14:32:38.904 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:38.904 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:38.904 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:38.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.904 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1087 3.f 3:fd983206:::smithi09025264-7:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:38.904 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:38.904 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 175.85469558200001, 2023-01-15T14:32:38.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0859964360000001, 2023-01-15T14:32:38.905 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:38.905 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:38.905 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:38.905 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:38.905 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:38.905 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1087 2023-01-15T14:32:38.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.905 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:38.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:38.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:38.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:38.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:38.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:38.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:38.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:38.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:38.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:38.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:38.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758724+0000", 2023-01-15T14:32:38.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7740000000000004e-06 2023-01-15T14:32:38.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:38.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758738+0000", 2023-01-15T14:32:38.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.33e-05 2023-01-15T14:32:38.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:38.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758744+0000", 2023-01-15T14:32:38.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2859999999999996e-06 2023-01-15T14:32:38.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848121+0000", 2023-01-15T14:32:38.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089377097000000003 2023-01-15T14:32:38.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:38.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848127+0000", 2023-01-15T14:32:38.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3500000000000002e-06 2023-01-15T14:32:38.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:38.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952454+0000", 2023-01-15T14:32:38.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104326167 2023-01-15T14:32:38.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:38.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:38.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:38.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952520+0000", 2023-01-15T14:32:38.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6166000000000003e-05 2023-01-15T14:32:39.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:39.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.063003+0000", 2023-01-15T14:32:39.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.110482916 2023-01-15T14:32:39.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:39.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065265+0000", 2023-01-15T14:32:39.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022626370000000001 2023-01-15T14:32:39.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:39.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065291+0000", 2023-01-15T14:32:39.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6095e-05 2023-01-15T14:32:39.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:39.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.066153+0000", 2023-01-15T14:32:39.078 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086127100000000004 2023-01-15T14:32:39.078 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.078 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:39.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166533+0000", 2023-01-15T14:32:39.078 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100380177 2023-01-15T14:32:39.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.079 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.079 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166646+0000", 2023-01-15T14:32:39.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112717 2023-01-15T14:32:39.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844498+0000", 2023-01-15T14:32:39.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67785237399999998 2023-01-15T14:32:39.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:39.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844605+0000", 2023-01-15T14:32:39.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106691 2023-01-15T14:32:39.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844717+0000", 2023-01-15T14:32:39.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011203499999999999 2023-01-15T14:32:39.082 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.082 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.082 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.083 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1088 3.f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:39.083 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:32:39.083 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 175.854639916, 2023-01-15T14:32:39.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.086312696, 2023-01-15T14:32:39.084 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.084 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:39.084 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:39.084 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:39.085 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:39.085 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1088 2023-01-15T14:32:39.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.085 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.086 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:32:39.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.086 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.086 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.086 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:32:39.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.086 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758775+0000", 2023-01-15T14:32:39.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:39.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758786+0000", 2023-01-15T14:32:39.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.076e-05 2023-01-15T14:32:39.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758809+0000", 2023-01-15T14:32:39.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3706999999999999e-05 2023-01-15T14:32:39.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758821+0000", 2023-01-15T14:32:39.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1976e-05 2023-01-15T14:32:39.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878534+0000", 2023-01-15T14:32:39.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.119712966 2023-01-15T14:32:39.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:39.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878540+0000", 2023-01-15T14:32:39.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0680000000000002e-06 2023-01-15T14:32:39.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952693+0000", 2023-01-15T14:32:39.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.074153067000000003 2023-01-15T14:32:39.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:39.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952735+0000", 2023-01-15T14:32:39.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1792999999999998e-05 2023-01-15T14:32:39.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065518+0000", 2023-01-15T14:32:39.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.11278334499999999 2023-01-15T14:32:39.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:39.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065687+0000", 2023-01-15T14:32:39.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016807700000000001 2023-01-15T14:32:39.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.097 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844786+0000", 2023-01-15T14:32:39.097 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.77909980300000004 2023-01-15T14:32:39.097 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.097 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.097 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844944+0000", 2023-01-15T14:32:39.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157424 2023-01-15T14:32:39.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.098 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.098 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.098 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.845089+0000", 2023-01-15T14:32:39.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144814 2023-01-15T14:32:39.099 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.099 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.099 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.099 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:16 3.13 3:ca393a9e:::smithi09025264-47:head [assert-version v193,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:39.099 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:32:39.100 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 139.307662491, 2023-01-15T14:32:39.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8970339479999998, 2023-01-15T14:32:39.100 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.100 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:39.100 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.101 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.101 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.101 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:32:39.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.101 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.101 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.101 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.101 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:32:39.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305751+0000", 2023-01-15T14:32:39.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:39.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305763+0000", 2023-01-15T14:32:39.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2309e-05 2023-01-15T14:32:39.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305788+0000", 2023-01-15T14:32:39.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4388000000000001e-05 2023-01-15T14:32:39.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305799+0000", 2023-01-15T14:32:39.104 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1304000000000001e-05 2023-01-15T14:32:39.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.104 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.104 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196425+0000", 2023-01-15T14:32:39.104 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8906259830000001 2023-01-15T14:32:39.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.104 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.104 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196673+0000", 2023-01-15T14:32:39.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024747499999999998 2023-01-15T14:32:39.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.105 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.105 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.202787+0000", 2023-01-15T14:32:39.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0061146509999999996 2023-01-15T14:32:39.105 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.105 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.105 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.106 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.106 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1480 3.13 3:cbfd2d71:::smithi09025264-37:head [write 581575~705262,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:39.106 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:32:39.106 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 139.274394763, 2023-01-15T14:32:39.106 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.375425221, 2023-01-15T14:32:39.106 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.106 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:39.106 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:39.106 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:39.106 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:39.107 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1480 2023-01-15T14:32:39.107 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.107 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.107 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.107 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.107 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:32:39.107 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.107 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.107 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.107 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.107 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:32:39.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.108 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.108 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.108 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339019+0000", 2023-01-15T14:32:39.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:39.108 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.108 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.108 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339659+0000", 2023-01-15T14:32:39.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000639742 2023-01-15T14:32:39.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.109 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339690+0000", 2023-01-15T14:32:39.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1594000000000003e-05 2023-01-15T14:32:39.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.109 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339703+0000", 2023-01-15T14:32:39.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2828e-05 2023-01-15T14:32:39.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.510837+0000", 2023-01-15T14:32:39.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1711338759999999 2023-01-15T14:32:39.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511048+0000", 2023-01-15T14:32:39.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021057899999999999 2023-01-15T14:32:39.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:39.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511325+0000", 2023-01-15T14:32:39.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027690400000000002 2023-01-15T14:32:39.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:39.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512092+0000", 2023-01-15T14:32:39.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00076720800000000002 2023-01-15T14:32:39.112 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.112 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.112 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:39.112 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512113+0000", 2023-01-15T14:32:39.112 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0917e-05 2023-01-15T14:32:39.112 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.112 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.112 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:39.112 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512439+0000", 2023-01-15T14:32:39.112 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032636299999999998 2023-01-15T14:32:39.112 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:39.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512871+0000", 2023-01-15T14:32:39.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000432087 2023-01-15T14:32:39.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.487 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.514165+0000", 2023-01-15T14:32:39.487 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001293726 2023-01-15T14:32:39.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.488 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714137+0000", 2023-01-15T14:32:39.488 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19997229499999999 2023-01-15T14:32:39.488 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:39.488 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714233+0000", 2023-01-15T14:32:39.488 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6062000000000003e-05 2023-01-15T14:32:39.489 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.489 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.489 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.489 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714446+0000", 2023-01-15T14:32:39.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212952 2023-01-15T14:32:39.489 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.490 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.490 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.490 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.490 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.490 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1481 3.13 3:cbfd2d71:::smithi09025264-37:head [write 1733272~707409] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:39.490 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:32:39.490 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 139.26754548, 2023-01-15T14:32:39.490 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.572522502, 2023-01-15T14:32:39.490 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.490 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:39.491 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:39.491 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:39.491 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:39.491 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1481 2023-01-15T14:32:39.491 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.491 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.492 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.492 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.492 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:32:39.492 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.492 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.492 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.493 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.493 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:32:39.493 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.493 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.493 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.493 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345868+0000", 2023-01-15T14:32:39.494 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:39.494 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.494 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346316+0000", 2023-01-15T14:32:39.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044813399999999999 2023-01-15T14:32:39.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.495 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.495 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.495 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346348+0000", 2023-01-15T14:32:39.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:32:39.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.496 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.496 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.496 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346362+0000", 2023-01-15T14:32:39.496 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.375e-05 2023-01-15T14:32:39.496 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.496 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.496 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.497 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512276+0000", 2023-01-15T14:32:39.497 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1659136829999999 2023-01-15T14:32:39.497 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.497 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.497 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:39.497 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512427+0000", 2023-01-15T14:32:39.497 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015088699999999999 2023-01-15T14:32:39.497 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.498 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.498 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714565+0000", 2023-01-15T14:32:39.498 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20213841599999999 2023-01-15T14:32:39.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.498 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.498 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714732+0000", 2023-01-15T14:32:39.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016676700000000001 2023-01-15T14:32:39.499 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.499 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.499 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:39.499 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714964+0000", 2023-01-15T14:32:39.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000231678 2023-01-15T14:32:39.499 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.500 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.500 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:39.500 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715499+0000", 2023-01-15T14:32:39.500 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053513300000000003 2023-01-15T14:32:39.500 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.500 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.500 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:39.500 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715523+0000", 2023-01-15T14:32:39.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4266000000000001e-05 2023-01-15T14:32:39.501 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.501 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.501 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:39.501 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715867+0000", 2023-01-15T14:32:39.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034419399999999998 2023-01-15T14:32:39.501 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.502 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.502 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.816932+0000", 2023-01-15T14:32:39.502 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10106498999999999 2023-01-15T14:32:39.502 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.502 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:39.502 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906374+0000", 2023-01-15T14:32:39.503 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089441442999999995 2023-01-15T14:32:39.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.503 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.503 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.503 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918050+0000", 2023-01-15T14:32:39.503 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.011676677999999999 2023-01-15T14:32:39.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:39.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918218+0000", 2023-01-15T14:32:39.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016802000000000001 2023-01-15T14:32:39.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918393+0000", 2023-01-15T14:32:39.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017447 2023-01-15T14:32:39.505 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.505 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.505 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.505 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1482 3.13 3:cbfd2d71:::smithi09025264-37:head [write 3195938~437497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:39.506 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:32:39.506 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 139.26110962199999, 2023-01-15T14:32:39.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5664780289999998, 2023-01-15T14:32:39.506 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.506 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:39.506 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:39.506 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:39.506 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:39.507 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1482 2023-01-15T14:32:39.507 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.507 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.507 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.507 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.507 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:32:39.507 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.508 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.508 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.508 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.508 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:32:39.508 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.508 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.509 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.509 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.509 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352275+0000", 2023-01-15T14:32:39.509 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999695 2023-01-15T14:32:39.509 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.509 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.510 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.510 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352644+0000", 2023-01-15T14:32:39.510 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036870300000000001 2023-01-15T14:32:39.510 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.510 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.510 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.511 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352676+0000", 2023-01-15T14:32:39.511 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1726000000000001e-05 2023-01-15T14:32:39.511 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.511 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.511 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.512 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352689+0000", 2023-01-15T14:32:39.512 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3617e-05 2023-01-15T14:32:39.512 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.512 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.512 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.512 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512602+0000", 2023-01-15T14:32:39.513 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1599126690000001 2023-01-15T14:32:39.513 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.513 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.513 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:39.513 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512839+0000", 2023-01-15T14:32:39.513 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023676900000000001 2023-01-15T14:32:39.513 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.514 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.514 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.514 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715649+0000", 2023-01-15T14:32:39.514 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20281044600000001 2023-01-15T14:32:39.514 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715777+0000", 2023-01-15T14:32:39.515 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012738800000000001 2023-01-15T14:32:39.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.516 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:39.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715995+0000", 2023-01-15T14:32:39.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000217858 2023-01-15T14:32:39.516 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.516 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.517 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:39.517 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716694+0000", 2023-01-15T14:32:39.517 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00069956700000000001 2023-01-15T14:32:39.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:39.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716709+0000", 2023-01-15T14:32:39.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4362000000000001e-05 2023-01-15T14:32:39.518 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:39.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716936+0000", 2023-01-15T14:32:39.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022744200000000001 2023-01-15T14:32:39.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.818876+0000", 2023-01-15T14:32:39.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101939635 2023-01-15T14:32:39.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:39.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906390+0000", 2023-01-15T14:32:39.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087513987000000001 2023-01-15T14:32:39.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918577+0000", 2023-01-15T14:32:39.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012187066 2023-01-15T14:32:39.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:39.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918697+0000", 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120326 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918784+0000", 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7200999999999994e-05 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1483 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3781575] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:39.521 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:32:39.522 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 139.26070146999999, 2023-01-15T14:32:39.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5663522429999999, 2023-01-15T14:32:39.522 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.522 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:39.522 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:39.522 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:39.522 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:39.522 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1483 2023-01-15T14:32:39.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.522 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:32:39.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:32:39.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352713+0000", 2023-01-15T14:32:39.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:39.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352720+0000", 2023-01-15T14:32:39.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7959999999999998e-06 2023-01-15T14:32:39.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352735+0000", 2023-01-15T14:32:39.526 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.455e-05 2023-01-15T14:32:39.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.526 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.526 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352741+0000", 2023-01-15T14:32:39.526 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8360000000000003e-06 2023-01-15T14:32:39.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.526 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.526 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512901+0000", 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1601594980000001 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.513067+0000", 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016626 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716977+0000", 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20391002799999999 2023-01-15T14:32:39.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717146+0000", 2023-01-15T14:32:39.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168851 2023-01-15T14:32:39.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:39.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717467+0000", 2023-01-15T14:32:39.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032135600000000002 2023-01-15T14:32:39.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717800+0000", 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033217400000000002 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717822+0000", 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2308999999999999e-05 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:39.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717970+0000", 2023-01-15T14:32:39.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014788099999999999 2023-01-15T14:32:39.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.530 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.530 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.819135+0000", 2023-01-15T14:32:39.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116478 2023-01-15T14:32:39.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.530 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:39.530 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906408+0000", 2023-01-15T14:32:39.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087272902999999999 2023-01-15T14:32:39.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.531 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918887+0000", 2023-01-15T14:32:39.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012479965000000001 2023-01-15T14:32:39.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:39.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918959+0000", 2023-01-15T14:32:39.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1978999999999996e-05 2023-01-15T14:32:39.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.919067+0000", 2023-01-15T14:32:39.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107083 2023-01-15T14:32:39.691 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.691 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.691 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.691 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1485 3.d 3:bc1c6b44:::smithi09025264-23:head [write 598382~780511,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:39.691 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:32:39.691 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 139.25323103599999, 2023-01-15T14:32:39.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5474121890000001, 2023-01-15T14:32:39.692 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.692 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:39.692 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:39.692 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:39.692 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:39.692 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1485 2023-01-15T14:32:39.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.692 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360168+0000", 2023-01-15T14:32:39.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999828 2023-01-15T14:32:39.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.694 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360790+0000", 2023-01-15T14:32:39.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062269099999999998 2023-01-15T14:32:39.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.694 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360822+0000", 2023-01-15T14:32:39.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1903000000000003e-05 2023-01-15T14:32:39.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360836+0000", 2023-01-15T14:32:39.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3380000000000001e-05 2023-01-15T14:32:39.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096664+0000", 2023-01-15T14:32:39.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7358282320000002 2023-01-15T14:32:39.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.696 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:39.696 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096683+0000", 2023-01-15T14:32:39.696 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9063999999999999e-05 2023-01-15T14:32:39.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.696 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.696 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704561+0000", 2023-01-15T14:32:39.696 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60787808399999999 2023-01-15T14:32:39.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.697 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.697 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704846+0000", 2023-01-15T14:32:39.697 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028527400000000002 2023-01-15T14:32:39.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.697 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:39.697 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705135+0000", 2023-01-15T14:32:39.697 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028875400000000001 2023-01-15T14:32:39.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.698 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:39.698 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705735+0000", 2023-01-15T14:32:39.698 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059956899999999999 2023-01-15T14:32:39.698 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.698 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.698 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:39.698 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705761+0000", 2023-01-15T14:32:39.699 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:32:39.699 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.699 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.699 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:39.699 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.706129+0000", 2023-01-15T14:32:39.699 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036822299999999999 2023-01-15T14:32:39.699 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.699 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.699 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:39.699 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906331+0000", 2023-01-15T14:32:39.700 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200202091 2023-01-15T14:32:39.700 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.700 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.700 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.700 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907111+0000", 2023-01-15T14:32:39.700 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077990799999999995 2023-01-15T14:32:39.700 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.700 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.700 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.700 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907379+0000", 2023-01-15T14:32:39.700 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026804800000000001 2023-01-15T14:32:39.701 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.701 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.701 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:39.701 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907529+0000", 2023-01-15T14:32:39.701 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015022700000000001 2023-01-15T14:32:39.701 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.701 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.701 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.701 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907597+0000", 2023-01-15T14:32:39.701 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7785000000000002e-05 2023-01-15T14:32:39.701 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1486 3.d 3:bc1c6b44:::smithi09025264-23:head [write 2075401~451067] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 139.24683498799999, 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6461318330000001, 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:39.702 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:39.703 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:39.703 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1486 2023-01-15T14:32:39.703 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.703 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.703 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.703 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:32:39.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:32:39.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366579+0000", 2023-01-15T14:32:39.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:39.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366845+0000", 2023-01-15T14:32:39.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026610499999999999 2023-01-15T14:32:39.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366878+0000", 2023-01-15T14:32:39.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2750000000000003e-05 2023-01-15T14:32:39.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.467009+0000", 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100131071 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096793+0000", 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6297846869999999 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096806+0000", 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3077e-05 2023-01-15T14:32:39.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907957+0000", 2023-01-15T14:32:39.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81115055400000002 2023-01-15T14:32:39.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908229+0000", 2023-01-15T14:32:39.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027187499999999998 2023-01-15T14:32:39.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:39.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908496+0000", 2023-01-15T14:32:39.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026754200000000001 2023-01-15T14:32:39.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:39.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909025+0000", 2023-01-15T14:32:39.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052865200000000001 2023-01-15T14:32:39.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:39.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909046+0000", 2023-01-15T14:32:39.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0713e-05 2023-01-15T14:32:39.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:39.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909294+0000", 2023-01-15T14:32:39.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024864899999999998 2023-01-15T14:32:39.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.911607+0000", 2023-01-15T14:32:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023128939999999998 2023-01-15T14:32:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012076+0000", 2023-01-15T14:32:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100468293 2023-01-15T14:32:39.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012426+0000", 2023-01-15T14:32:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035044299999999998 2023-01-15T14:32:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012532+0000", 2023-01-15T14:32:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106142 2023-01-15T14:32:39.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012713+0000", 2023-01-15T14:32:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018048599999999999 2023-01-15T14:32:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1496 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:39.715 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:32:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 138.87719213899999, 2023-01-15T14:32:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870616037, 2023-01-15T14:32:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1496 2023-01-15T14:32:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:32:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:32:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736220+0000", 2023-01-15T14:32:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:32:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736238+0000", 2023-01-15T14:32:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7156e-05 2023-01-15T14:32:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.718 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736275+0000", 2023-01-15T14:32:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7876000000000001e-05 2023-01-15T14:32:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736290+0000", 2023-01-15T14:32:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4449e-05 2023-01-15T14:32:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602217+0000", 2023-01-15T14:32:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8659270299999999 2023-01-15T14:32:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602468+0000", 2023-01-15T14:32:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000251315 2023-01-15T14:32:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.606840+0000", 2023-01-15T14:32:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0043715009999999999 2023-01-15T14:32:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1497 3.11 3:8889e450:::smithi09025264-32:head [read 0~2991646,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:32:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 138.87710468500001, 2023-01-15T14:32:39.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738359249999998, 2023-01-15T14:32:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1497 2023-01-15T14:32:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:32:39.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:32:39.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736310+0000", 2023-01-15T14:32:39.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:39.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736317+0000", 2023-01-15T14:32:39.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7240000000000003e-06 2023-01-15T14:32:39.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736334+0000", 2023-01-15T14:32:39.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6878999999999999e-05 2023-01-15T14:32:39.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736341+0000", 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0129999999999996e-06 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607070+0000", 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870728873 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607226+0000", 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015595 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610147+0000", 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.002921416 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.727 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1498 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:39.727 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:32:39.727 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 138.87706457900001, 2023-01-15T14:32:39.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8752667839999999, 2023-01-15T14:32:39.727 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.727 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:39.727 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:39.727 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:39.727 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:39.727 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1498 2023-01-15T14:32:39.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.728 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:32:39.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:32:39.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736350+0000", 2023-01-15T14:32:39.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:39.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736355+0000", 2023-01-15T14:32:39.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4900000000000002e-06 2023-01-15T14:32:39.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736363+0000", 2023-01-15T14:32:39.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2770000000000005e-06 2023-01-15T14:32:39.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736369+0000", 2023-01-15T14:32:39.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7400000000000001e-06 2023-01-15T14:32:39.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610234+0000", 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738651439999998 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610402+0000", 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168283 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.611618+0000", 2023-01-15T14:32:39.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001215647 2023-01-15T14:32:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.7.0:440 3.11 e231/207 ondisk, result = 0)", 2023-01-15T14:32:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:32:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 40.369282921, 2023-01-15T14:32:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973962139999998, 2023-01-15T14:32:39.732 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:32:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:32:39.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244132+0000", 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244143+0000", 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1647e-05 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244165+0000", 2023-01-15T14:32:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1501999999999998e-05 2023-01-15T14:32:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244180+0000", 2023-01-15T14:32:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5434000000000002e-05 2023-01-15T14:32:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.735 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941489+0000", 2023-01-15T14:32:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973091899999999 2023-01-15T14:32:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941503+0000", 2023-01-15T14:32:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.414e-05 2023-01-15T14:32:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.736 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941529+0000", 2023-01-15T14:32:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5622999999999998e-05 2023-01-15T14:32:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.4.0:562 3.1b e231/224 3:d85f42d9:::smithi09025264-45:a8 v 231'66, mlcod=225'62)", 2023-01-15T14:32:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 40.178931724999998, 2023-01-15T14:32:39.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2578820149999999, 2023-01-15T14:32:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434492+0000", 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4360000000000002e-06 2023-01-15T14:32:39.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434517+0000", 2023-01-15T14:32:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.48e-05 2023-01-15T14:32:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434530+0000", 2023-01-15T14:32:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2982999999999999e-05 2023-01-15T14:32:39.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434663+0000", 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013307 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434700+0000", 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7020000000000001e-05 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435438+0000", 2023-01-15T14:32:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073808900000000004 2023-01-15T14:32:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435456+0000", 2023-01-15T14:32:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8091999999999999e-05 2023-01-15T14:32:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:39.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435526+0000", 2023-01-15T14:32:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9806000000000002e-05 2023-01-15T14:32:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435570+0000", 2023-01-15T14:32:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4091999999999997e-05 2023-01-15T14:32:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.692366+0000", 2023-01-15T14:32:39.743 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2567961539999999 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:24 3.d 3:b3f111b5:::smithi09025264-48:head [assert-version v160,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e232)", 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 39.663095509000001, 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9963031440000001, 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:39.744 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.745 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950318+0000", 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950333+0000", 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.467e-05 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950368+0000", 2023-01-15T14:32:39.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4895999999999999e-05 2023-01-15T14:32:39.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050461+0000", 2023-01-15T14:32:39.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10009304400000001 2023-01-15T14:32:39.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050593+0000", 2023-01-15T14:32:39.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000132764 2023-01-15T14:32:39.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:32:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050615+0000", 2023-01-15T14:32:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1483000000000001e-05 2023-01-15T14:32:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945049+0000", 2023-01-15T14:32:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.894434138 2023-01-15T14:32:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945326+0000", 2023-01-15T14:32:39.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000277368 2023-01-15T14:32:39.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.946623+0000", 2023-01-15T14:32:39.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012970340000000001 2023-01-15T14:32:39.749 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.749 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.749 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.750 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.3.0:547 3.22 e232/180 3:47dfe935:::smithi09025264-29:81 v 232'124, mlcod=229'121)", 2023-01-15T14:32:39.750 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:32:39.750 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 39.562865760999998, 2023-01-15T14:32:39.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.643153431, 2023-01-15T14:32:39.750 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:39.750 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:39.750 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:39.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050549+0000", 2023-01-15T14:32:39.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:39.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:39.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050561+0000", 2023-01-15T14:32:39.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.278e-05 2023-01-15T14:32:39.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:39.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050583+0000", 2023-01-15T14:32:39.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1053e-05 2023-01-15T14:32:39.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.753 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:39.753 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050596+0000", 2023-01-15T14:32:39.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3946e-05 2023-01-15T14:32:39.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.753 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:39.753 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050708+0000", 2023-01-15T14:32:39.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011129 2023-01-15T14:32:39.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:39.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050750+0000", 2023-01-15T14:32:39.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2725000000000002e-05 2023-01-15T14:32:39.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:39.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051324+0000", 2023-01-15T14:32:39.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00057335700000000003 2023-01-15T14:32:39.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:39.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051340+0000", 2023-01-15T14:32:39.755 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5968999999999999e-05 2023-01-15T14:32:39.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.755 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:39.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051405+0000", 2023-01-15T14:32:39.755 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5419000000000005e-05 2023-01-15T14:32:39.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.755 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:39.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051471+0000", 2023-01-15T14:32:39.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5671000000000002e-05 2023-01-15T14:32:39.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:39.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:39.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:39.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.693703+0000", 2023-01-15T14:32:39.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.642232554 2023-01-15T14:32:39.756 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.756 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.756 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.756 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:39.756 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:39.757 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:39.757 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:32:39.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:32:39.760 INFO:tasks.rados.rados.0.smithi090.stdout:1084: done (0 left) 2023-01-15T14:32:39.760 INFO:tasks.rados.rados.0.smithi090.stdout:1085: rollback oid 17 current snap is 108 2023-01-15T14:32:39.760 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 17 to 103 2023-01-15T14:32:39.760 INFO:tasks.rados.rados.0.smithi090.stdout:1086: read oid 42 snap -1 2023-01-15T14:32:39.760 INFO:tasks.rados.rados.0.smithi090.stdout:1086: expect deleted 2023-01-15T14:32:39.760 INFO:tasks.rados.rados.0.smithi090.stdout:1087: write oid 29 current snap is 108 2023-01-15T14:32:39.760 INFO:tasks.rados.rados.0.smithi090.stdout:1087: seq_num 402 ranges {756414=700454,2041173=715241} 2023-01-15T14:32:39.760 INFO:tasks.rados.rados.0.smithi090.stdout:1087: writing smithi09025264-29 from 756414 to 1456868 tid 1 2023-01-15T14:32:39.760 INFO:tasks.rados.rados.0.smithi090.stdout:1087: writing smithi09025264-29 from 2041173 to 2756414 tid 2 2023-01-15T14:32:39.760 INFO:tasks.rados.rados.0.smithi090.stdout:1088: write (excl) oid 46 current snap is 108 2023-01-15T14:32:39.761 INFO:tasks.rados.rados.0.smithi090.stdout:1088: seq_num 403 ranges {773221=492056,1816008=503126} 2023-01-15T14:32:39.761 INFO:tasks.rados.rados.0.smithi090.stdout:1088: writing smithi09025264-46 from 773221 to 1265277 tid 1 2023-01-15T14:32:39.761 INFO:tasks.rados.rados.0.smithi090.stdout:1088: writing smithi09025264-46 from 1816008 to 2319134 tid 2 2023-01-15T14:32:39.761 INFO:tasks.rados.rados.0.smithi090.stdout:1089: read oid 36 snap -1 2023-01-15T14:32:39.761 INFO:tasks.rados.rados.0.smithi090.stdout:1089: expect (ObjNum 382 snap 103 seq_num 382) 2023-01-15T14:32:39.761 INFO:tasks.rados.rados.0.smithi090.stdout:1090: write oid 8 current snap is 108 2023-01-15T14:32:39.761 INFO:tasks.rados.rados.0.smithi090.stdout:1090: seq_num 404 ranges {790028=567305,1874490=763137} 2023-01-15T14:32:39.761 INFO:tasks.rados.rados.0.smithi090.stdout:1090: writing smithi09025264-8 from 790028 to 1357333 tid 1 2023-01-15T14:32:39.761 INFO:tasks.rados.rados.0.smithi090.stdout:1090: writing smithi09025264-8 from 1874490 to 2637627 tid 2 2023-01-15T14:32:39.761 INFO:tasks.rados.rados.0.smithi090.stdout:1091: snap_remove snap 106 2023-01-15T14:32:39.762 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:39.754+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:39.859 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:39.854+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:39.899 INFO:tasks.rados.rados.0.smithi090.stdout:1091: done (6 left) 2023-01-15T14:32:39.899 INFO:tasks.rados.rados.0.smithi090.stdout:1092: snap_remove snap 107 2023-01-15T14:32:39.901 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:39.901 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:39.901 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:32:39.901 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:39.902 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:32:39.961 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:39.954+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:40.087 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:32:40.087 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:32:40.087 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:40.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:32:40.088 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:40.080+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:40.188 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:40.182+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:40.309 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:32:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:32:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:32:40.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 254.177715484, 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.310 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:32:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:32:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:32:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:32:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:32:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:32:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:32:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:32:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:32:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:32:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:32:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:32:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:32:40.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:32:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:32:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:32:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:32:40.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:32:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:32:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:32:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:32:40.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:32:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:32:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:32:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:32:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:32:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:32:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:32:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:32:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:32:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:32:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:32:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 254.17743523600001, 2023-01-15T14:32:40.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:32:40.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:32:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:32:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:40.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:32:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:32:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:40.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:32:40.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:32:40.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:40.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:32:40.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:32:40.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:32:40.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:32:40.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:40.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:32:40.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:32:40.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:32:40.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:32:40.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:40.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:32:40.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:32:40.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:40.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:32:40.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:32:40.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:40.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:32:40.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:32:40.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:40.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:32:40.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:32:40.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:40.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:32:40.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:32:40.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:40.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:32:40.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:32:40.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:40.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:32:40.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:32:40.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:40.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:32:40.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:32:40.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:40.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:32:40.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:32:40.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:40.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:32:40.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:32:40.517 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:40.517 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:40.517 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 254.17740261899999, 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:40.518 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:32:40.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.519 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:40.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:40.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:32:40.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:40.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:40.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:32:40.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:40.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:40.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:32:40.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:40.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:40.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:32:40.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:32:40.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:40.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:32:40.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:32:40.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:40.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:32:40.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:32:40.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:32:40.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:32:40.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:40.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:32:40.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:32:40.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:32:40.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:32:40.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:40.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:32:40.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:32:40.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:32:40.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:32:40.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:40.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:32:40.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:32:40.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:40.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:32:40.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:32:40.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:40.528 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:40.529 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:40.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.529 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:40.529 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:32:40.529 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 254.07608463299999, 2023-01-15T14:32:40.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:32:40.529 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:40.529 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:40.529 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:40.530 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:40.530 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:40.530 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:32:40.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.530 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:40.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:40.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:32:40.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:40.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:40.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:32:40.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:40.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:40.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:32:40.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:40.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:32:40.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:32:40.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:32:40.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:32:40.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:32:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:32:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:32:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:32:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:32:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:32:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:40.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:32:40.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:32:40.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:40.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:32:40.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:32:40.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:32:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:32:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:32:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:32:40.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:40.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:32:40.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:32:40.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:32:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:32:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:40.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:32:40.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:32:40.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:32:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:32:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:40.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:32:40.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:32:40.695 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:40.695 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:32:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 254.07605462999999, 2023-01-15T14:32:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:32:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:32:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:32:40.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:40.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:40.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:32:40.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:40.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:40.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:32:40.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:40.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:40.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:32:40.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:32:40.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:32:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:32:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:32:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:32:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:32:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:32:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:32:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:32:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:32:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:32:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:32:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:32:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:32:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:32:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:32:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:32:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:32:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:32:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:41.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.100 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 253.561046065, 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:32:41.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:41.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:32:41.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:32:41.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:41.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:32:41.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:32:41.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:41.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:32:41.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:32:41.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:32:41.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:32:41.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:41.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:32:41.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:32:41.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:32:41.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:32:41.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:41.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:32:41.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:32:41.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:41.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:32:41.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:32:41.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:41.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:32:41.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:32:41.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:41.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:32:41.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:32:41.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:32:41.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:32:41.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:32:41.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:41.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:32:41.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:32:41.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:41.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:32:41.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:32:41.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.514 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:41.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.515 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:41.515 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:32:41.516 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 253.56056371099999, 2023-01-15T14:32:41.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:32:41.516 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:41.516 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:41.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:41.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:41.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:41.516 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:32:41.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.516 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:41.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:41.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:32:41.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:41.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:32:41.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:41.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:32:41.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:41.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:41.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:32:41.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:32:41.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:41.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:32:41.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:32:41.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:41.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:32:41.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:32:41.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:32:41.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:32:41.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:41.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:32:41.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:32:41.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:32:41.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:32:41.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:41.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:32:41.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:32:41.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:32:41.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:32:41.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:41.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:32:41.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:32:41.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:41.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:32:41.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:32:41.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.526 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:41.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.526 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:41.526 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:32:41.526 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 253.56051114799999, 2023-01-15T14:32:41.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:32:41.526 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:41.526 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:41.526 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:41.527 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:41.527 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:41.527 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:32:41.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.527 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:41.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:41.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:32:41.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:41.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:32:41.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:41.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:32:41.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:41.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:41.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:32:41.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:32:41.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:41.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:32:41.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:32:41.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:41.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:32:41.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:32:41.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:32:41.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:32:41.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:41.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:32:41.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:32:41.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:32:41.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:32:41.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:41.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:32:41.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:32:41.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:32:41.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:32:41.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:41.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:32:41.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:32:41.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:41.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 240.189309444, 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:41.534 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:41.535 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:41.535 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:41.535 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:32:41.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.535 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:41.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:41.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:32:41.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:41.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:32:41.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:41.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:32:41.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:32:41.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:41.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:32:41.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:32:41.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:32:41.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:32:41.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:41.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:32:41.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:32:41.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:41.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:32:41.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:32:41.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:41.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:32:41.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:32:41.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:41.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:32:41.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:32:41.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:41.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:32:41.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:32:41.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:41.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:32:41.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:32:41.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:41.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:32:41.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:32:41.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:41.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:32:41.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:32:41.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.543 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:41.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:32:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:32:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 240.18919277000001, 2023-01-15T14:32:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:32:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:41.544 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:32:41.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.545 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:41.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:41.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:32:41.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:41.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:32:41.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:41.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:32:41.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:32:41.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:41.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:32:41.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:32:41.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:32:41.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:32:41.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:41.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:32:41.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:32:41.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:32:41.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:32:41.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:41.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:32:41.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:32:41.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:32:41.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:32:41.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:41.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:32:41.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:32:41.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:41.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:32:41.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:32:41.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.551 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:41.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.551 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:32:41.551 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:32:41.551 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 232.46306049399999, 2023-01-15T14:32:41.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:32:41.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:41.551 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:41.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:41.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:41.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:41.552 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:32:41.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.552 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:41.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:41.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:32:41.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:41.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:32:41.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:41.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:32:41.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:41.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:41.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:32:41.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:32:41.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:41.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:32:41.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:32:41.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:32:41.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:32:41.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:32:41.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:32:41.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:41.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:32:41.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:32:41.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:41.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:32:41.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:32:41.557 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.557 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:41.557 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.557 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:32:41.557 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:32:41.558 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.24681043499999, 2023-01-15T14:32:41.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:32:41.558 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:41.558 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:41.558 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:41.558 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:41.558 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:41.558 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:32:41.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:32:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:32:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:41.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:32:41.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:41.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:41.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:32:41.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:32:41.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:41.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:32:41.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:32:41.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:41.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:32:41.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:32:41.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:32:41.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:41.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:32:41.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:32:41.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:41.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:32:41.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:32:41.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:41.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:32:41.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:32:41.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:41.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:32:41.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:32:41.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:32:41.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:32:41.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:32:41.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:32:41.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.715 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:32:41.716 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:32:41.716 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.24677686800001, 2023-01-15T14:32:41.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:32:41.716 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:41.716 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:41.716 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:41.716 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:41.716 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:41.716 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:32:41.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.716 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:32:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:32:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:41.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:32:41.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:41.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:41.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:32:41.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:32:41.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:32:41.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:41.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:32:41.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:32:41.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:41.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:32:41.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:32:41.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:41.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:32:41.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:41.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:32:41.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:32:41.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:32:41.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:32:41.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:32:41.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:32:41.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:41.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:32:41.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:32:41.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.48087332200001, 2023-01-15T14:32:41.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:32:41.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:41.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:32:41.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:41.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:32:41.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:41.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:41.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:32:41.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:32:41.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:41.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:32:41.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:32:41.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:41.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:32:41.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:41.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:32:41.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:32:41.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:32:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:32:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:32:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:32:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:32:41.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:32:41.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:32:41.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:32:41.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:41.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:32:41.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:32:41.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:32:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:32:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:32:41.736 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 160.93769452199999, 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:41.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:32:41.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:41.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:32:41.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:41.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:32:41.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:41.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:32:41.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:32:41.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:32:41.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:32:41.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:32:41.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:41.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:32:41.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:32:41.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:41.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:32:41.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:32:41.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:41.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:32:41.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:32:41.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:41.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:32:41.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:32:41.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:32:41.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:32:41.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:32:41.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:32:41.931 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.931 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:41.931 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:41.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.931 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:32:41.931 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:32:41.931 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 160.937655347, 2023-01-15T14:32:41.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:32:41.932 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:41.932 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:41.932 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:41.932 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:41.932 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:41.932 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:32:41.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.932 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:41.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:41.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:41.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:32:41.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:32:41.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:41.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:41.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:41.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:32:41.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:32:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:32:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:32:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:32:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:32:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:32:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:32:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:32:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:32:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:42.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:32:42.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:32:42.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:42.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:32:42.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:32:42.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:42.090 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:42.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:42.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.091 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1373 3.12 3:495d4bbc:::smithi09025264-16:head [write 2873523~591733] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:32:42.091 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:32:42.091 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.545539021, 2023-01-15T14:32:42.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40949580099999999, 2023-01-15T14:32:42.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:42.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:42.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:42.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:42.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:42.093 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1373 2023-01-15T14:32:42.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.093 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:42.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:42.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:32:42.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:42.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:42.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:32:42.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:42.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:42.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693440+0000", 2023-01-15T14:32:42.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:42.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:42.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693808+0000", 2023-01-15T14:32:42.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000367742 2023-01-15T14:32:42.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:42.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693831+0000", 2023-01-15T14:32:42.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3213e-05 2023-01-15T14:32:42.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:42.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693846+0000", 2023-01-15T14:32:42.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4909000000000001e-05 2023-01-15T14:32:42.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:42.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.695833+0000", 2023-01-15T14:32:42.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019869240000000002 2023-01-15T14:32:42.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:42.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696121+0000", 2023-01-15T14:32:42.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028753300000000003 2023-01-15T14:32:42.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:42.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798577+0000", 2023-01-15T14:32:42.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102456808 2023-01-15T14:32:42.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:42.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798849+0000", 2023-01-15T14:32:42.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027138999999999999 2023-01-15T14:32:42.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:32:42.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799155+0000", 2023-01-15T14:32:42.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030594800000000001 2023-01-15T14:32:42.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:42.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799614+0000", 2023-01-15T14:32:42.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045960700000000001 2023-01-15T14:32:42.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:42.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799737+0000", 2023-01-15T14:32:42.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012218900000000001 2023-01-15T14:32:42.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:42.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799998+0000", 2023-01-15T14:32:42.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026129700000000002 2023-01-15T14:32:42.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:42.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800639+0000", 2023-01-15T14:32:42.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064108800000000005 2023-01-15T14:32:42.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:42.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102023+0000", 2023-01-15T14:32:42.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30138417200000001 2023-01-15T14:32:42.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:42.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102198+0000", 2023-01-15T14:32:42.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017453600000000001 2023-01-15T14:32:42.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:42.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102398+0000", 2023-01-15T14:32:42.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020078000000000001 2023-01-15T14:32:42.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:42.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102937+0000", 2023-01-15T14:32:42.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053874499999999998 2023-01-15T14:32:42.108 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:42.108 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:42.108 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:42.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.108 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:32:42.108 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:32:42.108 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.54511442699999, 2023-01-15T14:32:42.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:32:42.109 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:42.109 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:42.109 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:42.109 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:42.109 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:42.109 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:32:42.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.109 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:42.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:42.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:32:42.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:42.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:42.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:32:42.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:42.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:32:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:32:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:32:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:42.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:32:42.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:32:42.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:42.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:32:42.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:32:42.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:42.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:32:42.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:32:42.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:42.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:32:42.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:32:42.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:42.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:32:42.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:32:42.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:42.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:32:42.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:32:42.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:32:42.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:32:42.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:32:42.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:42.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:32:42.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:32:42.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:42.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:32:42.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:32:42.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:42.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:32:42.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:32:42.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:42.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:32:42.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:32:42.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:42.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:32:42.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:32:42.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:42.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:32:42.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:32:42.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:42.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:32:42.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:32:42.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:42.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:32:42.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:32:42.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:42.123 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:42.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:42.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.123 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:32:42.123 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:32:42.124 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.54506209799999, 2023-01-15T14:32:42.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:32:42.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:42.124 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:42.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:42.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:42.124 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:42.124 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:32:42.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.124 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:42.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:42.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:32:42.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:42.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:42.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:32:42.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:42.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:32:42.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:42.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:32:42.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:32:42.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:42.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:32:42.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:32:42.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:42.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:32:42.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:32:42.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:42.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:32:42.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:32:42.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.255 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 133.94814889899999, 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:42.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:42.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:32:42.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:42.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:42.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:32:42.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:32:42.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:42.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:32:42.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:32:42.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:42.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:32:42.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:32:42.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:42.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:32:42.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:32:42.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:42.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:32:42.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:32:42.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:42.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:32:42.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:32:42.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:42.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:32:42.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:32:42.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:42.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:42.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:42.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:32:42.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:32:42.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:42.264 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:42.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:42.264 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:42.264 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:42.265 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:42.265 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:32:42.266 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:40.302+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:42.267 INFO:tasks.rados.rados.0.smithi090.stdout:1092: done (6 left) 2023-01-15T14:32:42.267 INFO:tasks.rados.rados.0.smithi090.stdout:1093: write oid 48 current snap is 108 2023-01-15T14:32:42.267 INFO:tasks.rados.rados.0.smithi090.stdout:1093: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2023-01-15T14:32:42.268 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:32:42.269 INFO:tasks.rados.rados.0.smithi090.stdout:1093: writing smithi09025264-48 from 406835 to 1049389 tid 1 2023-01-15T14:32:42.269 INFO:tasks.rados.rados.0.smithi090.stdout:1093: writing smithi09025264-48 from 1532972 to 2039767 tid 2 2023-01-15T14:32:42.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:32:42.270 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:32:42.423 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:32:42.423 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2023-01-15T14:32:42.423 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-01-15T14:32:42.424 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-01-15T14:32:42.424 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok config set filestore_inject_stall 3 2023-01-15T14:32:42.429 INFO:tasks.rados.rados.0.smithi090.stdout:1093: writing smithi09025264-48 from 2772312 to 2806835 tid 3 2023-01-15T14:32:42.430 INFO:tasks.rados.rados.0.smithi090.stdout:1094: rollback oid 12 current snap is 108 2023-01-15T14:32:42.430 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 12 to 103 2023-01-15T14:32:42.430 INFO:tasks.rados.rados.0.smithi090.stdout:1095: copy_from oid 35 from oid 5 current snap is 108 2023-01-15T14:32:42.430 INFO:tasks.rados.rados.0.smithi090.stdout:1096: write (excl) oid 23 current snap is 108 2023-01-15T14:32:42.430 INFO:tasks.rados.rados.0.smithi090.stdout:1096: seq_num 406 ranges {423642=717803,1875101=766806} 2023-01-15T14:32:42.430 INFO:tasks.rados.rados.0.smithi090.stdout:1096: writing smithi09025264-23 from 423642 to 1141445 tid 1 2023-01-15T14:32:42.430 INFO:tasks.rados.rados.0.smithi090.stdout:1096: writing smithi09025264-23 from 1875101 to 2641907 tid 2 2023-01-15T14:32:42.430 INFO:tasks.rados.rados.0.smithi090.stdout:1097: write (excl) oid 11 current snap is 108 2023-01-15T14:32:42.430 INFO:tasks.rados.rados.0.smithi090.stdout:1097: seq_num 407 ranges {440449=793052,1933583=510464} 2023-01-15T14:32:42.430 INFO:tasks.rados.rados.0.smithi090.stdout:1097: writing smithi09025264-11 from 440449 to 1233501 tid 1 2023-01-15T14:32:42.430 INFO:tasks.rados.rados.0.smithi090.stdout:1097: writing smithi09025264-11 from 1933583 to 2444047 tid 2 2023-01-15T14:32:42.431 INFO:tasks.rados.rados.0.smithi090.stdout:1098: snap_create 2023-01-15T14:32:42.431 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 11 2023-01-15T14:32:42.431 INFO:tasks.rados.rados.0.smithi090.stdout:1089: expect (ObjNum 382 snap 103 seq_num 382) 2023-01-15T14:32:42.431 INFO:tasks.rados.rados.0.smithi090.stdout:1085: finishing rollback tid 0 to smithi09025264-17 2023-01-15T14:32:42.431 INFO:tasks.rados.rados.0.smithi090.stdout:1087: finishing write tid 1 to smithi09025264-29 2023-01-15T14:32:42.431 INFO:tasks.rados.rados.0.smithi090.stdout:1086: done (10 left) 2023-01-15T14:32:42.431 INFO:tasks.rados.rados.0.smithi090.stdout:1089: done (9 left) 2023-01-15T14:32:42.431 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 9 2023-01-15T14:32:42.431 INFO:tasks.rados.rados.0.smithi090.stdout:1088: finishing write tid 1 to smithi09025264-46 2023-01-15T14:32:42.432 INFO:tasks.rados.rados.0.smithi090.stdout:1087: finishing write tid 2 to smithi09025264-29 2023-01-15T14:32:42.432 INFO:tasks.rados.rados.0.smithi090.stdout:1087: finishing write tid 3 to smithi09025264-29 2023-01-15T14:32:42.432 INFO:tasks.rados.rados.0.smithi090.stdout:1085: finishing rollback tid 1 to smithi09025264-17 2023-01-15T14:32:42.432 INFO:tasks.rados.rados.0.smithi090.stdout:1088: finishing write tid 2 to smithi09025264-46 2023-01-15T14:32:42.432 INFO:tasks.rados.rados.0.smithi090.stdout:1087: finishing write tid 4 to smithi09025264-29 2023-01-15T14:32:42.432 INFO:tasks.rados.rados.0.smithi090.stdout:1087: oid 29 updating version 0 to 264 2023-01-15T14:32:42.432 INFO:tasks.rados.rados.0.smithi090.stdout:1087: oid 29 version 264 is already newer than 263 2023-01-15T14:32:42.432 INFO:tasks.rados.rados.0.smithi090.stdout:1087: oid 29 updating version 264 to 265 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 265 (ObjNum 402 snap 108 seq_num 402) dirty exists 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:1087: left oid 29 (ObjNum 402 snap 108 seq_num 402) 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:1088: finishing write tid 3 to smithi09025264-46 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:1088: finishing write tid 4 to smithi09025264-46 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:1088: oid 46 updating version 0 to 289 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:1088: oid 46 version 289 is already newer than 287 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:1088: oid 46 version 289 is already newer than 288 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 289 (ObjNum 403 snap 108 seq_num 403) dirty exists 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:1088: left oid 46 (ObjNum 403 snap 108 seq_num 403) 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:1085: finishing rollback tid 2 to smithi09025264-17 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 366 (ObjNum 345 snap 91 seq_num 345) dirty exists 2023-01-15T14:32:42.433 INFO:tasks.rados.rados.0.smithi090.stdout:1090: finishing write tid 1 to smithi09025264-8 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:1090: finishing write tid 2 to smithi09025264-8 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:1090: finishing write tid 3 to smithi09025264-8 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:1090: finishing write tid 4 to smithi09025264-8 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:1090: oid 8 updating version 0 to 261 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:1090: oid 8 updating version 261 to 262 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:1090: oid 8 updating version 262 to 263 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 263 (ObjNum 404 snap 108 seq_num 404) dirty exists 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:1090: left oid 8 (ObjNum 404 snap 108 seq_num 404) 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:1085: done (8 left) 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:1087: done (7 left) 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:1088: done (6 left) 2023-01-15T14:32:42.434 INFO:tasks.rados.rados.0.smithi090.stdout:1090: done (5 left) 2023-01-15T14:32:42.435 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:32:42.435 INFO:tasks.rados.rados.0.smithi090.stdout:1097: finishing write tid 1 to smithi09025264-11 2023-01-15T14:32:42.435 INFO:tasks.rados.rados.0.smithi090.stdout:1093: finishing write tid 1 to smithi09025264-48 2023-01-15T14:32:42.435 INFO:tasks.rados.rados.0.smithi090.stdout:1093: finishing write tid 2 to smithi09025264-48 2023-01-15T14:32:42.435 INFO:tasks.rados.rados.0.smithi090.stdout:1096: finishing write tid 1 to smithi09025264-23 2023-01-15T14:32:42.435 INFO:tasks.rados.rados.0.smithi090.stdout:1096: finishing write tid 2 to smithi09025264-23 2023-01-15T14:32:42.435 INFO:tasks.rados.rados.0.smithi090.stdout:1093: finishing write tid 3 to smithi09025264-48 2023-01-15T14:32:42.436 INFO:tasks.rados.rados.0.smithi090.stdout:1093: finishing write tid 4 to smithi09025264-48 2023-01-15T14:32:42.436 INFO:tasks.rados.rados.0.smithi090.stdout:1093: finishing write tid 5 to smithi09025264-48 2023-01-15T14:32:42.436 INFO:tasks.rados.rados.0.smithi090.stdout:1093: oid 48 updating version 0 to 206 2023-01-15T14:32:42.436 INFO:tasks.rados.rados.0.smithi090.stdout:1093: oid 48 updating version 206 to 207 2023-01-15T14:32:42.436 INFO:tasks.rados.rados.0.smithi090.stdout:1093: oid 48 version 207 is already newer than 205 2023-01-15T14:32:42.436 INFO:tasks.rados.rados.0.smithi090.stdout:1093: oid 48 version 207 is already newer than 204 2023-01-15T14:32:42.437 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 207 (ObjNum 405 snap 108 seq_num 405) dirty exists 2023-01-15T14:32:42.437 INFO:tasks.rados.rados.0.smithi090.stdout:1093: left oid 48 (ObjNum 405 snap 108 seq_num 405) 2023-01-15T14:32:42.437 INFO:tasks.rados.rados.0.smithi090.stdout:1093: done (4 left) 2023-01-15T14:32:42.437 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:32:42.437 INFO:tasks.rados.rados.0.smithi090.stdout:1094: finishing rollback tid 0 to smithi09025264-12 2023-01-15T14:32:42.437 INFO:tasks.rados.rados.0.smithi090.stdout:1094: finishing rollback tid 1 to smithi09025264-12 2023-01-15T14:32:42.437 INFO:tasks.rados.rados.0.smithi090.stdout:1097: finishing write tid 2 to smithi09025264-11 2023-01-15T14:32:42.438 INFO:tasks.rados.rados.0.smithi090.stdout:1097: finishing write tid 3 to smithi09025264-11 2023-01-15T14:32:42.438 INFO:tasks.rados.rados.0.smithi090.stdout:1096: finishing write tid 3 to smithi09025264-23 2023-01-15T14:32:42.438 INFO:tasks.rados.rados.0.smithi090.stdout:1096: finishing write tid 4 to smithi09025264-23 2023-01-15T14:32:42.438 INFO:tasks.rados.rados.0.smithi090.stdout:1096: oid 23 updating version 0 to 219 2023-01-15T14:32:42.438 INFO:tasks.rados.rados.0.smithi090.stdout:1096: oid 23 updating version 219 to 220 2023-01-15T14:32:42.438 INFO:tasks.rados.rados.0.smithi090.stdout:1096: oid 23 updating version 220 to 221 2023-01-15T14:32:42.439 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 221 (ObjNum 406 snap 108 seq_num 406) dirty exists 2023-01-15T14:32:42.439 INFO:tasks.rados.rados.0.smithi090.stdout:1096: left oid 23 (ObjNum 406 snap 108 seq_num 406) 2023-01-15T14:32:42.439 INFO:tasks.rados.rados.0.smithi090.stdout:1096: done (3 left) 2023-01-15T14:32:42.439 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:32:42.439 INFO:tasks.rados.rados.0.smithi090.stdout:1097: finishing write tid 4 to smithi09025264-11 2023-01-15T14:32:42.439 INFO:tasks.rados.rados.0.smithi090.stdout:1097: oid 11 updating version 0 to 307 2023-01-15T14:32:42.439 INFO:tasks.rados.rados.0.smithi090.stdout:1097: oid 11 updating version 307 to 309 2023-01-15T14:32:42.440 INFO:tasks.rados.rados.0.smithi090.stdout:1097: oid 11 version 309 is already newer than 308 2023-01-15T14:32:42.440 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 309 (ObjNum 407 snap 108 seq_num 407) dirty exists 2023-01-15T14:32:42.440 INFO:tasks.rados.rados.0.smithi090.stdout:1097: left oid 11 (ObjNum 407 snap 108 seq_num 407) 2023-01-15T14:32:42.440 INFO:tasks.rados.rados.0.smithi090.stdout:1097: done (2 left) 2023-01-15T14:32:42.440 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:42.441 INFO:tasks.rados.rados.0.smithi090.stdout:1094: finishing rollback tid 2 to smithi09025264-12 2023-01-15T14:32:42.441 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 279 (ObjNum 388 snap 103 seq_num 388) dirty exists 2023-01-15T14:32:42.441 INFO:tasks.rados.rados.0.smithi090.stdout:1094: done (1 left) 2023-01-15T14:32:42.441 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:42.441 INFO:tasks.rados.rados.0.smithi090.stdout:1095: finishing copy_from to smithi09025264-35 2023-01-15T14:32:42.441 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 229 (ObjNum 387 snap 103 seq_num 387) dirty exists 2023-01-15T14:32:42.442 INFO:tasks.rados.rados.0.smithi090.stdout:1095: finishing copy_from racing read to smithi09025264-35 2023-01-15T14:32:42.442 INFO:tasks.rados.rados.0.smithi090.stdout:1095: done (0 left) 2023-01-15T14:32:42.442 INFO:tasks.rados.rados.0.smithi090.stdout:1098: done (0 left) 2023-01-15T14:32:42.442 INFO:tasks.rados.rados.0.smithi090.stdout:1099: rollback oid 44 current snap is 109 2023-01-15T14:32:42.442 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 44 to 103 2023-01-15T14:32:42.442 INFO:tasks.rados.rados.0.smithi090.stdout:1100: read oid 19 snap 108 2023-01-15T14:32:42.443 INFO:tasks.rados.rados.0.smithi090.stdout:1100: expect (ObjNum 383 snap 103 seq_num 383) 2023-01-15T14:32:42.443 INFO:tasks.rados.rados.0.smithi090.stdout:1101: read oid 2 snap -1 2023-01-15T14:32:42.443 INFO:tasks.rados.rados.0.smithi090.stdout:1101: expect (ObjNum 328 snap 87 seq_num 328) 2023-01-15T14:32:42.443 INFO:tasks.rados.rados.0.smithi090.stdout:1102: rollback oid 14 current snap is 109 2023-01-15T14:32:42.443 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 14 to 103 2023-01-15T14:32:42.443 INFO:tasks.rados.rados.0.smithi090.stdout:1103: snap_create 2023-01-15T14:32:42.443 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:32:42.444 INFO:tasks.rados.rados.0.smithi090.stdout:1102: finishing rollback tid 0 to smithi09025264-14 2023-01-15T14:32:42.444 INFO:tasks.rados.rados.0.smithi090.stdout:1102: finishing rollback tid 1 to smithi09025264-14 2023-01-15T14:32:42.444 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 265 (ObjNum 481 snap 0 seq_num 3194989072) dirty dne 2023-01-15T14:32:42.444 INFO:tasks.rados.rados.0.smithi090.stdout:1102: done (3 left) 2023-01-15T14:32:42.445 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:32:42.445 INFO:tasks.rados.rados.0.smithi090.stdout:1101: expect (ObjNum 328 snap 87 seq_num 328) 2023-01-15T14:32:42.445 INFO:tasks.rados.rados.0.smithi090.stdout:1100: expect (ObjNum 383 snap 103 seq_num 383) 2023-01-15T14:32:42.446 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:42.266+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:42.447 INFO:tasks.rados.rados.0.smithi090.stdout:1099: finishing rollback tid 0 to smithi09025264-44 2023-01-15T14:32:42.447 INFO:tasks.rados.rados.0.smithi090.stdout:1099: finishing rollback tid 1 to smithi09025264-44 2023-01-15T14:32:42.447 INFO:tasks.rados.rados.0.smithi090.stdout:1099: finishing rollback tid 2 to smithi09025264-44 2023-01-15T14:32:42.447 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 297 (ObjNum 359 snap 95 seq_num 359) dirty exists 2023-01-15T14:32:42.448 INFO:tasks.rados.rados.0.smithi090.stdout:1099: done (2 left) 2023-01-15T14:32:42.448 INFO:tasks.rados.rados.0.smithi090.stdout:1100: done (1 left) 2023-01-15T14:32:42.448 INFO:tasks.rados.rados.0.smithi090.stdout:1101: done (0 left) 2023-01-15T14:32:42.491 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:42.486+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:42.588 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:42.588 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-01-15T14:32:42.588 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:42.588 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:32:42.589 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:42.589 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:32:42.590 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:42.590 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:42.582+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:42.599 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:42.620 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:42.645 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:42.761 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:42.761 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:42.762 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:42.762 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:42.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:32:42.763 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:42.763 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:42.764 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:32:42.764 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:32:42.764 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:32:42.764 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:32:42.764 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:32:42.764 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:32:42.765 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:32:42.765 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:32:42.765 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:42.756+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:42.863 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:42.858+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:42.936 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:42.936 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:42.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:32:42.987 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:42.982+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:43.094 INFO:tasks.rados.rados.0.smithi090.stdout:1103: done (0 left) 2023-01-15T14:32:43.094 INFO:tasks.rados.rados.0.smithi090.stdout:1104: read oid 28 snap -1 2023-01-15T14:32:43.095 INFO:tasks.rados.rados.0.smithi090.stdout:1104: expect (ObjNum 352 snap 93 seq_num 352) 2023-01-15T14:32:43.095 INFO:tasks.rados.rados.0.smithi090.stdout:1105: delete oid 35 current snap is 110 2023-01-15T14:32:43.097 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:43.097 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:43.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:32:43.099 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:43.088+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:43.099 INFO:tasks.rados.rados.0.smithi090.stdout:1104: expect (ObjNum 352 snap 93 seq_num 352) 2023-01-15T14:32:43.195 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:43.188+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:43.257 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:43.257 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:43.257 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:32:43.296 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:43.290+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:43.420 INFO:tasks.rados.rados.0.smithi090.stdout:1104: done (1 left) 2023-01-15T14:32:43.420 INFO:tasks.rados.rados.0.smithi090.stdout:1105: done (0 left) 2023-01-15T14:32:43.420 INFO:tasks.rados.rados.0.smithi090.stdout:1106: rollback oid 32 current snap is 110 2023-01-15T14:32:43.420 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 32 to 108 2023-01-15T14:32:43.421 INFO:tasks.rados.rados.0.smithi090.stdout:1107: delete oid 39 current snap is 110 2023-01-15T14:32:43.421 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:43.422 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:43.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:32:43.422 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:43.414+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:43.517 INFO:tasks.rados.rados.0.smithi090.stdout:1106: finishing rollback tid 0 to smithi09025264-32 2023-01-15T14:32:43.587 INFO:tasks.rados.rados.0.smithi090.stdout:1106: finishing rollback tid 1 to smithi09025264-32 2023-01-15T14:32:43.592 INFO:tasks.rados.rados.0.smithi090.stdout:1106: finishing rollback tid 2 to smithi09025264-32 2023-01-15T14:32:43.592 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 268 (ObjNum 375 snap 102 seq_num 375) dirty exists 2023-01-15T14:32:43.592 INFO:tasks.rados.rados.0.smithi090.stdout:1106: done (1 left) 2023-01-15T14:32:43.592 INFO:tasks.rados.rados.0.smithi090.stdout:1107: done (0 left) 2023-01-15T14:32:43.593 INFO:tasks.rados.rados.0.smithi090.stdout:1108: write oid 35 current snap is 110 2023-01-15T14:32:43.593 INFO:tasks.rados.rados.0.smithi090.stdout:1108: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2023-01-15T14:32:43.593 INFO:tasks.rados.rados.0.smithi090.stdout:1108: writing smithi09025264-35 from 490870 to 1109669 tid 1 2023-01-15T14:32:43.593 INFO:tasks.rados.rados.0.smithi090.stdout:1108: writing smithi09025264-35 from 1592676 to 2366820 tid 2 2023-01-15T14:32:43.593 INFO:tasks.rados.rados.0.smithi090.stdout:1108: writing smithi09025264-35 from 2793074 to 2890870 tid 3 2023-01-15T14:32:43.594 INFO:tasks.rados.rados.0.smithi090.stdout:1109: read oid 32 snap 108 2023-01-15T14:32:43.594 INFO:tasks.rados.rados.0.smithi090.stdout:1109: expect (ObjNum 375 snap 102 seq_num 375) 2023-01-15T14:32:43.594 INFO:tasks.rados.rados.0.smithi090.stdout:1110: copy_from oid 50 from oid 3 current snap is 110 2023-01-15T14:32:43.594 INFO:tasks.rados.rados.0.smithi090.stdout:1111: rollback oid 16 current snap is 110 2023-01-15T14:32:43.594 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 16 to 109 2023-01-15T14:32:43.594 INFO:tasks.rados.rados.0.smithi090.stdout:1112: snap_create 2023-01-15T14:32:43.595 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:32:43.595 INFO:tasks.rados.rados.0.smithi090.stdout:1111: finishing rollback tid 1 to smithi09025264-16 2023-01-15T14:32:43.595 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 0 (ObjNum 33 snap 0 seq_num 3) dirty dne 2023-01-15T14:32:43.595 INFO:tasks.rados.rados.0.smithi090.stdout:1111: done (3 left) 2023-01-15T14:32:43.595 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:32:43.596 INFO:tasks.rados.rados.0.smithi090.stdout:1110: finishing copy_from to smithi09025264-50 2023-01-15T14:32:43.596 INFO:tasks.rados.rados.0.smithi090.stdout:1110: got expected ENOENT (src dne) 2023-01-15T14:32:43.596 INFO:tasks.rados.rados.0.smithi090.stdout:1110: finishing copy_from racing read to smithi09025264-50 2023-01-15T14:32:43.596 INFO:tasks.rados.rados.0.smithi090.stdout:1110: done (2 left) 2023-01-15T14:32:43.596 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:43.598 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:43.598 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:43.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:32:43.786 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:43.588+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:43.788 INFO:tasks.rados.rados.0.smithi090.stdout:1109: expect (ObjNum 375 snap 102 seq_num 375) 2023-01-15T14:32:43.788 INFO:tasks.rados.rados.0.smithi090.stdout:1108: finishing write tid 1 to smithi09025264-35 2023-01-15T14:32:43.788 INFO:tasks.rados.rados.0.smithi090.stdout:1108: finishing write tid 2 to smithi09025264-35 2023-01-15T14:32:43.788 INFO:tasks.rados.rados.0.smithi090.stdout:1109: done (1 left) 2023-01-15T14:32:43.788 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:43.789 INFO:tasks.rados.rados.0.smithi090.stdout:1108: finishing write tid 3 to smithi09025264-35 2023-01-15T14:32:43.789 INFO:tasks.rados.rados.0.smithi090.stdout:1108: finishing write tid 4 to smithi09025264-35 2023-01-15T14:32:43.789 INFO:tasks.rados.rados.0.smithi090.stdout:1108: finishing write tid 5 to smithi09025264-35 2023-01-15T14:32:43.789 INFO:tasks.rados.rados.0.smithi090.stdout:1108: oid 35 updating version 0 to 301 2023-01-15T14:32:43.789 INFO:tasks.rados.rados.0.smithi090.stdout:1108: oid 35 version 301 is already newer than 299 2023-01-15T14:32:43.790 INFO:tasks.rados.rados.0.smithi090.stdout:1108: oid 35 version 301 is already newer than 300 2023-01-15T14:32:43.790 INFO:tasks.rados.rados.0.smithi090.stdout:1108: oid 35 updating version 301 to 302 2023-01-15T14:32:43.790 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 302 (ObjNum 410 snap 110 seq_num 410) dirty exists 2023-01-15T14:32:43.790 INFO:tasks.rados.rados.0.smithi090.stdout:1108: left oid 35 (ObjNum 410 snap 110 seq_num 410) 2023-01-15T14:32:43.790 INFO:tasks.rados.rados.0.smithi090.stdout:1108: done (0 left) 2023-01-15T14:32:43.791 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:43.784+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:43.887 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:43.880+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:43.897 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:43.898 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:43.907 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:32:43.988 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:43.980+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:44.000 INFO:tasks.rados.rados.0.smithi090.stdout:1112: done (0 left) 2023-01-15T14:32:44.001 INFO:tasks.rados.rados.0.smithi090.stdout:1113: snap_create 2023-01-15T14:32:44.016 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:44.016 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:44.026 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:32:44.089 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:44.082+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:44.140 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:44.140 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:44.150 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:32:44.191 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:44.184+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:44.252 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:44.252 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:44.264 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:32:44.293 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:44.288+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:44.366 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:44.367 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:44.377 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:32:44.421 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:44.416+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:44.426 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:32:44.495 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:44.488+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:44.611 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:44.612 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:44.612 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:32:44.613 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:44.604+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:44.712 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:44.706+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:44.976 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:32:44.978 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:44.978 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:44.980 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:32:44.980 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:44.972+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:45.010 INFO:tasks.rados.rados.0.smithi090.stdout:1113: done (0 left) 2023-01-15T14:32:45.132 INFO:tasks.rados.rados.0.smithi090.stdout:1114: rollback oid 43 current snap is 112 2023-01-15T14:32:45.132 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 43 to 103 2023-01-15T14:32:45.132 INFO:tasks.rados.rados.0.smithi090.stdout:1115: delete oid 9 current snap is 112 2023-01-15T14:32:45.133 INFO:tasks.rados.rados.0.smithi090.stdout:1114: finishing rollback tid 1 to smithi09025264-43 2023-01-15T14:32:45.134 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 0 (ObjNum 81 snap 0 seq_num 0) dirty dne 2023-01-15T14:32:45.135 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:45.135 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:45.135 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:32:45.136 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:32:45.137 INFO:tasks.rados.rados.0.smithi090.stdout:1114: done (1 left) 2023-01-15T14:32:45.137 INFO:tasks.rados.rados.0.smithi090.stdout:1115: done (0 left) 2023-01-15T14:32:45.137 INFO:tasks.rados.rados.0.smithi090.stdout:1116: write (excl) oid 13 current snap is 112 2023-01-15T14:32:45.137 INFO:tasks.rados.rados.0.smithi090.stdout:1116: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2023-01-15T14:32:45.137 INFO:tasks.rados.rados.0.smithi090.stdout:1116: writing smithi09025264-13 from 524484 to 1010134 tid 1 2023-01-15T14:32:45.137 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:45.138 INFO:tasks.rados.rados.0.smithi090.stdout:1116: writing smithi09025264-13 from 1425993 to 2087453 tid 2 2023-01-15T14:32:45.138 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:45.126+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:45.139 INFO:tasks.rados.rados.0.smithi090.stdout:1116: writing smithi09025264-13 from 2847920 to 2924484 tid 3 2023-01-15T14:32:45.139 INFO:tasks.rados.rados.0.smithi090.stdout:1117: read oid 50 snap 109 2023-01-15T14:32:45.139 INFO:tasks.rados.rados.0.smithi090.stdout:1117: expect (ObjNum 328 snap 87 seq_num 328) 2023-01-15T14:32:45.140 INFO:tasks.rados.rados.0.smithi090.stdout:1118: read oid 35 snap 111 2023-01-15T14:32:45.140 INFO:tasks.rados.rados.0.smithi090.stdout:1118: expect (ObjNum 410 snap 110 seq_num 410) 2023-01-15T14:32:45.140 INFO:tasks.rados.rados.0.smithi090.stdout:1119: write oid 37 current snap is 112 2023-01-15T14:32:45.140 INFO:tasks.rados.rados.0.smithi090.stdout:1119: seq_num 413 ranges {541291=560899,1884475=521471} 2023-01-15T14:32:45.143 INFO:tasks.rados.rados.0.smithi090.stdout:1119: writing smithi09025264-37 from 541291 to 1102190 tid 1 2023-01-15T14:32:45.152 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:45.153 INFO:tasks.rados.rados.0.smithi090.stdout:1119: writing smithi09025264-37 from 1884475 to 2405946 tid 2 2023-01-15T14:32:45.154 INFO:tasks.rados.rados.0.smithi090.stdout:1120: delete oid 25 current snap is 112 2023-01-15T14:32:45.166 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:45.295 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:45.298 INFO:tasks.rados.rados.0.smithi090.stdout:1118: expect (ObjNum 410 snap 110 seq_num 410) 2023-01-15T14:32:45.299 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:45.290+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:45.299 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:45.299 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:45.300 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:32:45.321 INFO:tasks.rados.rados.0.smithi090.stdout:1117: expect (ObjNum 328 snap 87 seq_num 328) 2023-01-15T14:32:45.458 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:45.458 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:45.459 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:45.459 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:45.460 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:32:45.461 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:45.452+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:45.481 INFO:tasks.rados.rados.0.smithi090.stdout:1117: done (4 left) 2023-01-15T14:32:45.949 INFO:tasks.rados.rados.0.smithi090.stdout:1118: done (3 left) 2023-01-15T14:32:45.950 INFO:tasks.rados.rados.0.smithi090.stdout:1120: done (2 left) 2023-01-15T14:32:45.950 INFO:tasks.rados.rados.0.smithi090.stdout:1121: read oid 27 snap -1 2023-01-15T14:32:45.950 INFO:tasks.rados.rados.0.smithi090.stdout:1121: expect deleted 2023-01-15T14:32:45.950 INFO:tasks.rados.rados.0.smithi090.stdout:1122: write (excl) oid 35 current snap is 112 2023-01-15T14:32:45.950 INFO:tasks.rados.rados.0.smithi090.stdout:1116: finishing write tid 1 to smithi09025264-13 2023-01-15T14:32:45.950 INFO:tasks.rados.rados.0.smithi090.stdout:1122: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2023-01-15T14:32:45.957 INFO:tasks.rados.rados.0.smithi090.stdout:1122: writing smithi09025264-35 from 574905 to 1286302 tid 1 2023-01-15T14:32:45.958 INFO:tasks.rados.rados.0.smithi090.stdout:1122: writing smithi09025264-35 from 1885086 to 2410226 tid 2 2023-01-15T14:32:45.958 INFO:tasks.rados.rados.0.smithi090.stdout:1122: writing smithi09025264-35 from 2930189 to 2974905 tid 3 2023-01-15T14:32:45.958 INFO:tasks.rados.rados.0.smithi090.stdout:1123: read oid 31 snap -1 2023-01-15T14:32:45.958 INFO:tasks.rados.rados.0.smithi090.stdout:1123: expect (ObjNum 396 snap 106 seq_num 396) 2023-01-15T14:32:45.958 INFO:tasks.rados.rados.0.smithi090.stdout:1124: write (excl) oid 25 current snap is 112 2023-01-15T14:32:45.958 INFO:tasks.rados.rados.0.smithi090.stdout:1119: finishing write tid 1 to smithi09025264-37 2023-01-15T14:32:45.958 INFO:tasks.rados.rados.0.smithi090.stdout:1124: seq_num 416 ranges {591712=786646,1943568=668798} 2023-01-15T14:32:45.958 INFO:tasks.rados.rados.0.smithi090.stdout:1124: writing smithi09025264-25 from 591712 to 1378358 tid 1 2023-01-15T14:32:45.958 INFO:tasks.rados.rados.0.smithi090.stdout:1124: writing smithi09025264-25 from 1943568 to 2612366 tid 2 2023-01-15T14:32:45.958 INFO:tasks.rados.rados.0.smithi090.stdout:1125: write (excl) oid 36 current snap is 112 2023-01-15T14:32:45.959 INFO:tasks.rados.rados.0.smithi090.stdout:1119: finishing write tid 2 to smithi09025264-37 2023-01-15T14:32:45.959 INFO:tasks.rados.rados.0.smithi090.stdout:1125: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2023-01-15T14:32:45.959 INFO:tasks.rados.rados.0.smithi090.stdout:1125: writing smithi09025264-36 from 608519 to 1070414 tid 1 2023-01-15T14:32:45.959 INFO:tasks.rados.rados.0.smithi090.stdout:1125: writing smithi09025264-36 from 1602050 to 2130859 tid 2 2023-01-15T14:32:45.959 INFO:tasks.rados.rados.0.smithi090.stdout:1125: writing smithi09025264-36 from 2585035 to 3008519 tid 3 2023-01-15T14:32:45.959 INFO:tasks.rados.rados.0.smithi090.stdout:1126: write (excl) oid 43 current snap is 112 2023-01-15T14:32:45.959 INFO:tasks.rados.rados.0.smithi090.stdout:1119: finishing write tid 3 to smithi09025264-37 2023-01-15T14:32:45.959 INFO:tasks.rados.rados.0.smithi090.stdout:1126: seq_num 418 ranges {625326=537144,1944179=672467} 2023-01-15T14:32:45.959 INFO:tasks.rados.rados.0.smithi090.stdout:1126: writing smithi09025264-43 from 625326 to 1162470 tid 1 2023-01-15T14:32:45.960 INFO:tasks.rados.rados.0.smithi090.stdout:1126: writing smithi09025264-43 from 1944179 to 2616646 tid 2 2023-01-15T14:32:45.960 INFO:tasks.rados.rados.0.smithi090.stdout:1127: copy_from oid 32 from oid 8 current snap is 112 2023-01-15T14:32:45.960 INFO:tasks.rados.rados.0.smithi090.stdout:1119: finishing write tid 4 to smithi09025264-37 2023-01-15T14:32:45.960 INFO:tasks.rados.rados.0.smithi090.stdout:1119: oid 37 updating version 0 to 327 2023-01-15T14:32:45.960 INFO:tasks.rados.rados.0.smithi090.stdout:1119: oid 37 updating version 327 to 328 2023-01-15T14:32:45.960 INFO:tasks.rados.rados.0.smithi090.stdout:1119: oid 37 updating version 328 to 329 2023-01-15T14:32:45.960 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 329 (ObjNum 413 snap 112 seq_num 413) dirty exists 2023-01-15T14:32:45.960 INFO:tasks.rados.rados.0.smithi090.stdout:1119: left oid 37 (ObjNum 413 snap 112 seq_num 413) 2023-01-15T14:32:45.960 INFO:tasks.rados.rados.0.smithi090.stdout:1122: finishing write tid 1 to smithi09025264-35 2023-01-15T14:32:45.960 INFO:tasks.rados.rados.0.smithi090.stdout:1119: done (8 left) 2023-01-15T14:32:45.961 INFO:tasks.rados.rados.0.smithi090.stdout:1121: done (7 left) 2023-01-15T14:32:45.961 INFO:tasks.rados.rados.0.smithi090.stdout:1128: read oid 42 snap -1 2023-01-15T14:32:45.961 INFO:tasks.rados.rados.0.smithi090.stdout:1128: expect deleted 2023-01-15T14:32:45.961 INFO:tasks.rados.rados.0.smithi090.stdout:1129: copy_from oid 41 from oid 45 current snap is 112 2023-01-15T14:32:45.961 INFO:tasks.rados.rados.0.smithi090.stdout:1123: expect (ObjNum 396 snap 106 seq_num 396) 2023-01-15T14:32:45.961 INFO:tasks.rados.rados.0.smithi090.stdout:1124: finishing write tid 1 to smithi09025264-25 2023-01-15T14:32:45.961 INFO:tasks.rados.rados.0.smithi090.stdout:1123: done (8 left) 2023-01-15T14:32:45.961 INFO:tasks.rados.rados.0.smithi090.stdout:1130: write oid 40 current snap is 112 2023-01-15T14:32:45.961 INFO:tasks.rados.rados.0.smithi090.stdout:1116: finishing write tid 2 to smithi09025264-13 2023-01-15T14:32:45.961 INFO:tasks.rados.rados.0.smithi090.stdout:1116: finishing write tid 3 to smithi09025264-13 2023-01-15T14:32:45.962 INFO:tasks.rados.rados.0.smithi090.stdout:1125: finishing write tid 1 to smithi09025264-36 2023-01-15T14:32:45.962 INFO:tasks.rados.rados.0.smithi090.stdout:1122: finishing write tid 2 to smithi09025264-35 2023-01-15T14:32:45.962 INFO:tasks.rados.rados.0.smithi090.stdout:1130: seq_num 419 ranges {642133=728746,2119014=532478} 2023-01-15T14:32:45.962 INFO:tasks.rados.rados.0.smithi090.stdout:1130: writing smithi09025264-40 from 642133 to 1370879 tid 1 2023-01-15T14:32:45.962 INFO:tasks.rados.rados.0.smithi090.stdout:1130: writing smithi09025264-40 from 2119014 to 2651492 tid 2 2023-01-15T14:32:45.962 INFO:tasks.rados.rados.0.smithi090.stdout:1128: done (8 left) 2023-01-15T14:32:45.962 INFO:tasks.rados.rados.0.smithi090.stdout:1131: rollback oid 44 current snap is 112 2023-01-15T14:32:45.962 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 44 to 111 2023-01-15T14:32:45.962 INFO:tasks.rados.rados.0.smithi090.stdout:1132: write oid 2 current snap is 112 2023-01-15T14:32:45.963 INFO:tasks.rados.rados.0.smithi090.stdout:1132: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2023-01-15T14:32:45.963 INFO:tasks.rados.rados.0.smithi090.stdout:1132: writing smithi09025264-2 from 658940 to 1062935 tid 1 2023-01-15T14:32:45.963 INFO:tasks.rados.rados.0.smithi090.stdout:1132: writing smithi09025264-2 from 1777496 to 2453632 tid 2 2023-01-15T14:32:45.963 INFO:tasks.rados.rados.0.smithi090.stdout:1132: writing smithi09025264-2 from 2950951 to 3058940 tid 3 2023-01-15T14:32:45.963 INFO:tasks.rados.rados.0.smithi090.stdout:1133: copy_from oid 19 from oid 30 current snap is 112 2023-01-15T14:32:45.963 INFO:tasks.rados.rados.0.smithi090.stdout:1132: finishing write tid 1 to smithi09025264-2 2023-01-15T14:32:45.963 INFO:tasks.rados.rados.0.smithi090.stdout:1134: write oid 27 current snap is 112 2023-01-15T14:32:45.963 INFO:tasks.rados.rados.0.smithi090.stdout:1134: seq_num 421 ranges {675747=479244,1835978=536147} 2023-01-15T14:32:45.963 INFO:tasks.rados.rados.0.smithi090.stdout:1134: writing smithi09025264-27 from 675747 to 1154991 tid 1 2023-01-15T14:32:45.964 INFO:tasks.rados.rados.0.smithi090.stdout:1134: writing smithi09025264-27 from 1835978 to 2372125 tid 2 2023-01-15T14:32:45.964 INFO:tasks.rados.rados.0.smithi090.stdout:1135: write (excl) oid 18 current snap is 112 2023-01-15T14:32:45.964 INFO:tasks.rados.rados.0.smithi090.stdout:1132: finishing write tid 2 to smithi09025264-2 2023-01-15T14:32:45.964 INFO:tasks.rados.rados.0.smithi090.stdout:1135: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2023-01-15T14:32:45.964 INFO:tasks.rados.rados.0.smithi090.stdout:1135: writing smithi09025264-18 from 692554 to 1247047 tid 1 2023-01-15T14:32:45.964 INFO:tasks.rados.rados.0.smithi090.stdout:1135: writing smithi09025264-18 from 1778107 to 2457912 tid 2 2023-01-15T14:32:45.964 INFO:tasks.rados.rados.0.smithi090.stdout:1135: writing smithi09025264-18 from 2889444 to 3092554 tid 3 2023-01-15T14:32:45.964 INFO:tasks.rados.rados.0.smithi090.stdout:1136: read oid 15 snap 108 2023-01-15T14:32:45.964 INFO:tasks.rados.rados.0.smithi090.stdout:1136: expect (ObjNum 361 snap 96 seq_num 361) 2023-01-15T14:32:45.964 INFO:tasks.rados.rados.0.smithi090.stdout:1137: copy_from oid 46 from oid 10 current snap is 112 2023-01-15T14:32:45.965 INFO:tasks.rados.rados.0.smithi090.stdout:1133: finishing copy_from to smithi09025264-19 2023-01-15T14:32:45.965 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 96 (ObjNum 326 snap 87 seq_num 326) dirty exists 2023-01-15T14:32:45.965 INFO:tasks.rados.rados.0.smithi090.stdout:1138: snap_create 2023-01-15T14:32:45.965 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 15 2023-01-15T14:32:45.965 INFO:tasks.rados.rados.0.smithi090.stdout:1133: finishing copy_from racing read to smithi09025264-19 2023-01-15T14:32:45.965 INFO:tasks.rados.rados.0.smithi090.stdout:1133: done (14 left) 2023-01-15T14:32:45.965 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 14 2023-01-15T14:32:45.965 INFO:tasks.rados.rados.0.smithi090.stdout:1124: finishing write tid 2 to smithi09025264-25 2023-01-15T14:32:45.965 INFO:tasks.rados.rados.0.smithi090.stdout:1126: finishing write tid 1 to smithi09025264-43 2023-01-15T14:32:45.966 INFO:tasks.rados.rados.0.smithi090.stdout:1130: finishing write tid 1 to smithi09025264-40 2023-01-15T14:32:45.966 INFO:tasks.rados.rados.0.smithi090.stdout:1116: finishing write tid 4 to smithi09025264-13 2023-01-15T14:32:45.966 INFO:tasks.rados.rados.0.smithi090.stdout:1116: finishing write tid 5 to smithi09025264-13 2023-01-15T14:32:45.966 INFO:tasks.rados.rados.0.smithi090.stdout:1116: oid 13 updating version 0 to 300 2023-01-15T14:32:45.966 INFO:tasks.rados.rados.0.smithi090.stdout:1116: oid 13 updating version 300 to 302 2023-01-15T14:32:45.966 INFO:tasks.rados.rados.0.smithi090.stdout:1116: oid 13 version 302 is already newer than 299 2023-01-15T14:32:45.966 INFO:tasks.rados.rados.0.smithi090.stdout:1116: oid 13 version 302 is already newer than 301 2023-01-15T14:32:45.966 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 302 (ObjNum 412 snap 112 seq_num 412) dirty exists 2023-01-15T14:32:45.966 INFO:tasks.rados.rados.0.smithi090.stdout:1116: left oid 13 (ObjNum 412 snap 112 seq_num 412) 2023-01-15T14:32:45.967 INFO:tasks.rados.rados.0.smithi090.stdout:1116: done (13 left) 2023-01-15T14:32:45.967 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 13 2023-01-15T14:32:45.967 INFO:tasks.rados.rados.0.smithi090.stdout:1132: finishing write tid 3 to smithi09025264-2 2023-01-15T14:32:45.967 INFO:tasks.rados.rados.0.smithi090.stdout:1122: finishing write tid 3 to smithi09025264-35 2023-01-15T14:32:45.967 INFO:tasks.rados.rados.0.smithi090.stdout:1124: finishing write tid 3 to smithi09025264-25 2023-01-15T14:32:45.967 INFO:tasks.rados.rados.0.smithi090.stdout:1124: finishing write tid 4 to smithi09025264-25 2023-01-15T14:32:45.967 INFO:tasks.rados.rados.0.smithi090.stdout:1124: oid 25 updating version 0 to 298 2023-01-15T14:32:45.967 INFO:tasks.rados.rados.0.smithi090.stdout:1124: oid 25 updating version 298 to 300 2023-01-15T14:32:45.967 INFO:tasks.rados.rados.0.smithi090.stdout:1124: oid 25 version 300 is already newer than 299 2023-01-15T14:32:45.968 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 300 (ObjNum 416 snap 112 seq_num 416) dirty exists 2023-01-15T14:32:45.968 INFO:tasks.rados.rados.0.smithi090.stdout:1124: left oid 25 (ObjNum 416 snap 112 seq_num 416) 2023-01-15T14:32:45.968 INFO:tasks.rados.rados.0.smithi090.stdout:1124: done (12 left) 2023-01-15T14:32:45.968 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 12 2023-01-15T14:32:45.968 INFO:tasks.rados.rados.0.smithi090.stdout:1129: finishing copy_from to smithi09025264-41 2023-01-15T14:32:45.968 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 140 (ObjNum 400 snap 106 seq_num 400) dirty exists 2023-01-15T14:32:45.968 INFO:tasks.rados.rados.0.smithi090.stdout:1129: finishing copy_from racing read to smithi09025264-41 2023-01-15T14:32:45.968 INFO:tasks.rados.rados.0.smithi090.stdout:1127: finishing copy_from to smithi09025264-32 2023-01-15T14:32:45.968 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 263 (ObjNum 404 snap 108 seq_num 404) dirty exists 2023-01-15T14:32:45.968 INFO:tasks.rados.rados.0.smithi090.stdout:1127: finishing copy_from racing read to smithi09025264-32 2023-01-15T14:32:45.969 INFO:tasks.rados.rados.0.smithi090.stdout:1127: done (11 left) 2023-01-15T14:32:45.969 INFO:tasks.rados.rados.0.smithi090.stdout:1129: done (10 left) 2023-01-15T14:32:45.969 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 10 2023-01-15T14:32:45.969 INFO:tasks.rados.rados.0.smithi090.stdout:1126: finishing write tid 2 to smithi09025264-43 2023-01-15T14:32:45.969 INFO:tasks.rados.rados.0.smithi090.stdout:1122: finishing write tid 4 to smithi09025264-35 2023-01-15T14:32:45.969 INFO:tasks.rados.rados.0.smithi090.stdout:1125: finishing write tid 2 to smithi09025264-36 2023-01-15T14:32:45.969 INFO:tasks.rados.rados.0.smithi090.stdout:1125: finishing write tid 3 to smithi09025264-36 2023-01-15T14:32:45.969 INFO:tasks.rados.rados.0.smithi090.stdout:1125: finishing write tid 4 to smithi09025264-36 2023-01-15T14:32:45.970 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:45.945+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:45.971 INFO:tasks.rados.rados.0.smithi090.stdout:1135: finishing write tid 1 to smithi09025264-18 2023-01-15T14:32:46.109 INFO:tasks.rados.rados.0.smithi090.stdout:1130: finishing write tid 2 to smithi09025264-40 2023-01-15T14:32:46.109 INFO:tasks.rados.rados.0.smithi090.stdout:1130: finishing write tid 3 to smithi09025264-40 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:1132: finishing write tid 4 to smithi09025264-2 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:1130: finishing write tid 4 to smithi09025264-40 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:1130: oid 40 updating version 0 to 235 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:1130: oid 40 version 235 is already newer than 233 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:1130: oid 40 version 235 is already newer than 234 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 235 (ObjNum 419 snap 112 seq_num 419) dirty exists 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:1130: left oid 40 (ObjNum 419 snap 112 seq_num 419) 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:1132: finishing write tid 5 to smithi09025264-2 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:1132: oid 2 updating version 0 to 248 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:1132: oid 2 updating version 248 to 249 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:1132: oid 2 updating version 249 to 250 2023-01-15T14:32:46.110 INFO:tasks.rados.rados.0.smithi090.stdout:1132: oid 2 updating version 250 to 251 2023-01-15T14:32:46.111 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 251 (ObjNum 420 snap 112 seq_num 420) dirty exists 2023-01-15T14:32:46.111 INFO:tasks.rados.rados.0.smithi090.stdout:1132: left oid 2 (ObjNum 420 snap 112 seq_num 420) 2023-01-15T14:32:46.111 INFO:tasks.rados.rados.0.smithi090.stdout:1130: done (9 left) 2023-01-15T14:32:46.111 INFO:tasks.rados.rados.0.smithi090.stdout:1132: done (8 left) 2023-01-15T14:32:46.111 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:32:46.111 INFO:tasks.rados.rados.0.smithi090.stdout:1136: expect (ObjNum 361 snap 96 seq_num 361) 2023-01-15T14:32:46.114 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:46.114 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:46.114 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:32:46.115 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:46.104+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:1136: done (7 left) 2023-01-15T14:32:46.115 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:32:46.115 INFO:tasks.rados.rados.0.smithi090.stdout:1122: finishing write tid 5 to smithi09025264-35 2023-01-15T14:32:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:1122: oid 35 updating version 0 to 304 2023-01-15T14:32:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:1122: oid 35 updating version 304 to 307 2023-01-15T14:32:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:1122: oid 35 version 307 is already newer than 306 2023-01-15T14:32:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:1122: oid 35 version 307 is already newer than 305 2023-01-15T14:32:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 307 (ObjNum 415 snap 112 seq_num 415) dirty exists 2023-01-15T14:32:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:1122: left oid 35 (ObjNum 415 snap 112 seq_num 415) 2023-01-15T14:32:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:1122: done (6 left) 2023-01-15T14:32:46.116 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:32:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:1125: finishing write tid 5 to smithi09025264-36 2023-01-15T14:32:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:1125: oid 36 updating version 0 to 253 2023-01-15T14:32:46.116 INFO:tasks.rados.rados.0.smithi090.stdout:1125: oid 36 version 253 is already newer than 251 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:1125: oid 36 version 253 is already newer than 252 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:1125: oid 36 version 253 is already newer than 250 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 253 (ObjNum 417 snap 112 seq_num 417) dirty exists 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:1125: left oid 36 (ObjNum 417 snap 112 seq_num 417) 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:1125: done (5 left) 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:1131: finishing rollback tid 0 to smithi09025264-44 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:1131: finishing rollback tid 1 to smithi09025264-44 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:1126: finishing write tid 3 to smithi09025264-43 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:1126: finishing write tid 4 to smithi09025264-43 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:1126: oid 43 updating version 0 to 92 2023-01-15T14:32:46.117 INFO:tasks.rados.rados.0.smithi090.stdout:1126: oid 43 updating version 92 to 93 2023-01-15T14:32:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:1126: oid 43 version 93 is already newer than 91 2023-01-15T14:32:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 93 (ObjNum 418 snap 112 seq_num 418) dirty exists 2023-01-15T14:32:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:1126: left oid 43 (ObjNum 418 snap 112 seq_num 418) 2023-01-15T14:32:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:1126: done (4 left) 2023-01-15T14:32:46.118 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:32:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:1134: finishing write tid 1 to smithi09025264-27 2023-01-15T14:32:46.118 INFO:tasks.rados.rados.0.smithi090.stdout:1135: finishing write tid 2 to smithi09025264-18 2023-01-15T14:32:46.118 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:46.121 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:46.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:32:46.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:32:46.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:32:46.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:32:46.124 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:32:46.124 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:32:46.124 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: filestore_inject_stall = '0' (not observed, change may require restart) osd_enable_op_tracker = 'true' 2023-01-15T14:32:46.124 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:32:46.165 INFO:tasks.rados.rados.0.smithi090.stdout:1131: finishing rollback tid 2 to smithi09025264-44 2023-01-15T14:32:46.273 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 306 (ObjNum 359 snap 95 seq_num 359) dirty exists 2023-01-15T14:32:46.273 INFO:tasks.rados.rados.0.smithi090.stdout:1131: done (3 left) 2023-01-15T14:32:46.273 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:32:46.273 INFO:tasks.rados.rados.0.smithi090.stdout:1134: finishing write tid 2 to smithi09025264-27 2023-01-15T14:32:46.274 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:46.274 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:32:46.274 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:32:46.276 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:46.277 INFO:tasks.rados.rados.0.smithi090.stdout:1134: finishing write tid 3 to smithi09025264-27 2023-01-15T14:32:46.277 INFO:tasks.rados.rados.0.smithi090.stdout:1135: finishing write tid 3 to smithi09025264-18 2023-01-15T14:32:46.277 INFO:tasks.rados.rados.0.smithi090.stdout:1134: finishing write tid 4 to smithi09025264-27 2023-01-15T14:32:46.277 INFO:tasks.rados.rados.0.smithi090.stdout:1134: oid 27 updating version 0 to 209 2023-01-15T14:32:46.277 INFO:tasks.rados.rados.0.smithi090.stdout:1134: oid 27 updating version 209 to 210 2023-01-15T14:32:46.277 INFO:tasks.rados.rados.0.smithi090.stdout:1134: oid 27 version 210 is already newer than 208 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 210 (ObjNum 421 snap 112 seq_num 421) dirty exists 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout:1134: left oid 27 (ObjNum 421 snap 112 seq_num 421) 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout:1134: done (2 left) 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout:1137: finishing copy_from to smithi09025264-46 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 312 (ObjNum 360 snap 95 seq_num 360) dirty exists 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout:1137: finishing copy_from racing read to smithi09025264-46 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout:1137: done (1 left) 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout:1135: finishing write tid 4 to smithi09025264-18 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout:1135: finishing write tid 5 to smithi09025264-18 2023-01-15T14:32:46.278 INFO:tasks.rados.rados.0.smithi090.stdout:1135: oid 18 updating version 0 to 216 2023-01-15T14:32:46.279 INFO:tasks.rados.rados.0.smithi090.stdout:1135: oid 18 version 216 is already newer than 214 2023-01-15T14:32:46.279 INFO:tasks.rados.rados.0.smithi090.stdout:1135: oid 18 updating version 216 to 217 2023-01-15T14:32:46.279 INFO:tasks.rados.rados.0.smithi090.stdout:1135: oid 18 version 217 is already newer than 215 2023-01-15T14:32:46.279 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 217 (ObjNum 422 snap 112 seq_num 422) dirty exists 2023-01-15T14:32:46.279 INFO:tasks.rados.rados.0.smithi090.stdout:1135: left oid 18 (ObjNum 422 snap 112 seq_num 422) 2023-01-15T14:32:46.279 INFO:tasks.rados.rados.0.smithi090.stdout:1135: done (0 left) 2023-01-15T14:32:46.280 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:46.268+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:46.282 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:32:46.376 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:46.368+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:46.436 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:46.436 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:32:46.436 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:32:46.437 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:32:46.437 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:46.437 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:32:46.477 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:46.469+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:46.547 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:46.548 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:32:46.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:32:46.548 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:32:46.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.548 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:75 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:46.548 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:32:46.548 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 260.187928339, 2023-01-15T14:32:46.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92335155400000002, 2023-01-15T14:32:46.548 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.549 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:46.549 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.549 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.549 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.549 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 75 2023-01-15T14:32:46.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.549 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:32:46.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356896+0000", 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356903+0000", 2023-01-15T14:32:46.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4619999999999999e-06 2023-01-15T14:32:46.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356917+0000", 2023-01-15T14:32:46.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4195999999999999e-05 2023-01-15T14:32:46.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356926+0000", 2023-01-15T14:32:46.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:32:46.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457743+0000", 2023-01-15T14:32:46.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100817668 2023-01-15T14:32:46.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:46.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457841+0000", 2023-01-15T14:32:46.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7079999999999999e-05 2023-01-15T14:32:46.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778474+0000", 2023-01-15T14:32:46.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32063301799999999 2023-01-15T14:32:46.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778627+0000", 2023-01-15T14:32:46.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015389599999999999 2023-01-15T14:32:46.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:46.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778815+0000", 2023-01-15T14:32:46.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018726600000000001 2023-01-15T14:32:46.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:46.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779033+0000", 2023-01-15T14:32:46.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021819999999999999 2023-01-15T14:32:46.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779051+0000", 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7640000000000001e-05 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3590999999999995e-05 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:46.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779163+0000", 2023-01-15T14:32:46.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:32:46.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.081002+0000", 2023-01-15T14:32:46.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30183937799999999 2023-01-15T14:32:46.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280043+0000", 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19904084699999999 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280138+0000", 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4915999999999997e-05 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280248+0000", 2023-01-15T14:32:46.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110233 2023-01-15T14:32:46.558 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.558 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.558 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.558 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:76 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:46.558 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:32:46.558 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 260.18788673099999, 2023-01-15T14:32:46.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92420393899999997, 2023-01-15T14:32:46.558 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.558 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 76 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356937+0000", 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356948+0000", 2023-01-15T14:32:46.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.094e-05 2023-01-15T14:32:46.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356959+0000", 2023-01-15T14:32:46.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1058e-05 2023-01-15T14:32:46.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356968+0000", 2023-01-15T14:32:46.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5339999999999999e-06 2023-01-15T14:32:46.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459524+0000", 2023-01-15T14:32:46.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102556147 2023-01-15T14:32:46.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:46.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459662+0000", 2023-01-15T14:32:46.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000137756 2023-01-15T14:32:46.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978621+0000", 2023-01-15T14:32:46.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51895887600000001 2023-01-15T14:32:46.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:46.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978870+0000", 2023-01-15T14:32:46.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024928999999999999 2023-01-15T14:32:46.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280761+0000", 2023-01-15T14:32:46.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30189113400000001 2023-01-15T14:32:46.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281023+0000", 2023-01-15T14:32:46.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026172800000000003 2023-01-15T14:32:46.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281142+0000", 2023-01-15T14:32:46.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000119296 2023-01-15T14:32:46.568 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.568 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.568 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.569 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1076 3.f 3:f87c0183:::smithi09025264-50:head [setxattr _header (57) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:46.569 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:32:46.569 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 183.786177976, 2023-01-15T14:32:46.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.88473693200000003, 2023-01-15T14:32:46.569 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.569 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:46.569 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.569 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.569 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.569 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1076 2023-01-15T14:32:46.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.570 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:32:46.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:32:46.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758646+0000", 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758653+0000", 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8029999999999997e-06 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758670+0000", 2023-01-15T14:32:46.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6158999999999999e-05 2023-01-15T14:32:46.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758677+0000", 2023-01-15T14:32:46.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9840000000000004e-06 2023-01-15T14:32:46.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848049+0000", 2023-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089372163000000004 2023-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848056+0000", 2023-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0360000000000001e-06 2023-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951562+0000", 2023-01-15T14:32:46.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10350670100000001 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951726+0000", 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000163311 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951958+0000", 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000232525 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:46.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952168+0000", 2023-01-15T14:32:46.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020949799999999999 2023-01-15T14:32:46.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:46.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952186+0000", 2023-01-15T14:32:46.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7714e-05 2023-01-15T14:32:46.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:46.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952254+0000", 2023-01-15T14:32:46.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7998999999999997e-05 2023-01-15T14:32:46.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:46.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952278+0000", 2023-01-15T14:32:46.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4909000000000002e-05 2023-01-15T14:32:46.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065836+0000", 2023-01-15T14:32:46.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.113557081 2023-01-15T14:32:46.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643196+0000", 2023-01-15T14:32:46.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.57736012999999997 2023-01-15T14:32:46.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:46.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643278+0000", 2023-01-15T14:32:46.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2671999999999997e-05 2023-01-15T14:32:46.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643384+0000", 2023-01-15T14:32:46.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010551400000000001 2023-01-15T14:32:46.578 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.578 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.578 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1077 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 183.78613915700001, 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.98557123499999999, 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.579 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1077 2023-01-15T14:32:46.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.580 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:32:46.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:32:46.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758685+0000", 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758689+0000", 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2760000000000002e-06 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758704+0000", 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4187e-05 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758710+0000", 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4330000000000001e-06 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848086+0000", 2023-01-15T14:32:46.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089376466000000002 2023-01-15T14:32:46.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:46.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848093+0000", 2023-01-15T14:32:46.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4269999999999996e-06 2023-01-15T14:32:46.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952346+0000", 2023-01-15T14:32:46.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104252758 2023-01-15T14:32:46.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:46.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952427+0000", 2023-01-15T14:32:46.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1786000000000006e-05 2023-01-15T14:32:46.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743602+0000", 2023-01-15T14:32:46.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.79117428099999998 2023-01-15T14:32:46.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743892+0000", 2023-01-15T14:32:46.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029067200000000001 2023-01-15T14:32:46.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.744257+0000", 2023-01-15T14:32:46.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000364619 2023-01-15T14:32:46.585 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.586 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.586 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.586 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1087 3.f 3:fd983206:::smithi09025264-7:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:46.586 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:46.586 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 183.78610467499999, 2023-01-15T14:32:46.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0859964360000001, 2023-01-15T14:32:46.586 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.586 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:46.586 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1087 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:46.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:32:46.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:46.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758724+0000", 2023-01-15T14:32:46.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7740000000000004e-06 2023-01-15T14:32:46.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758738+0000", 2023-01-15T14:32:46.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.33e-05 2023-01-15T14:32:46.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758744+0000", 2023-01-15T14:32:46.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2859999999999996e-06 2023-01-15T14:32:46.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848121+0000", 2023-01-15T14:32:46.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089377097000000003 2023-01-15T14:32:46.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848127+0000", 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3500000000000002e-06 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952454+0000", 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104326167 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952520+0000", 2023-01-15T14:32:46.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6166000000000003e-05 2023-01-15T14:32:46.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:46.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.063003+0000", 2023-01-15T14:32:46.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.110482916 2023-01-15T14:32:46.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065265+0000", 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022626370000000001 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065291+0000", 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6095e-05 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.066153+0000", 2023-01-15T14:32:46.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086127100000000004 2023-01-15T14:32:46.594 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:46.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166533+0000", 2023-01-15T14:32:46.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100380177 2023-01-15T14:32:46.594 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166646+0000", 2023-01-15T14:32:46.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112717 2023-01-15T14:32:46.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.595 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.595 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.595 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844498+0000", 2023-01-15T14:32:46.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67785237399999998 2023-01-15T14:32:46.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.595 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.595 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:46.595 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844605+0000", 2023-01-15T14:32:46.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106691 2023-01-15T14:32:46.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844717+0000", 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011203499999999999 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1088 3.f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:32:46.596 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 183.78604900900001, 2023-01-15T14:32:46.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.086312696, 2023-01-15T14:32:46.597 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.597 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:46.597 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.597 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.597 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.597 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1088 2023-01-15T14:32:46.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.597 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:32:46.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:32:46.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758775+0000", 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758786+0000", 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.076e-05 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758809+0000", 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3706999999999999e-05 2023-01-15T14:32:46.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758821+0000", 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1976e-05 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878534+0000", 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.119712966 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.600 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878540+0000", 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0680000000000002e-06 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952693+0000", 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.074153067000000003 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952735+0000", 2023-01-15T14:32:46.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1792999999999998e-05 2023-01-15T14:32:46.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.602 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.602 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065518+0000", 2023-01-15T14:32:46.602 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.11278334499999999 2023-01-15T14:32:46.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.602 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:46.602 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065687+0000", 2023-01-15T14:32:46.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016807700000000001 2023-01-15T14:32:46.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844786+0000", 2023-01-15T14:32:46.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.77909980300000004 2023-01-15T14:32:46.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844944+0000", 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157424 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.845089+0000", 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144814 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.605 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:16 3.13 3:ca393a9e:::smithi09025264-47:head [assert-version v193,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:46.605 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:32:46.605 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 147.23907158399999, 2023-01-15T14:32:46.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8970339479999998, 2023-01-15T14:32:46.605 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.605 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:46.605 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:32:46.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:32:46.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305751+0000", 2023-01-15T14:32:46.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:46.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305763+0000", 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2309e-05 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305788+0000", 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4388000000000001e-05 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.608 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305799+0000", 2023-01-15T14:32:46.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1304000000000001e-05 2023-01-15T14:32:46.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.608 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196425+0000", 2023-01-15T14:32:46.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8906259830000001 2023-01-15T14:32:46.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196673+0000", 2023-01-15T14:32:46.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024747499999999998 2023-01-15T14:32:46.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.609 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.609 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.202787+0000", 2023-01-15T14:32:46.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0061146509999999996 2023-01-15T14:32:46.609 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.609 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.610 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.610 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1480 3.13 3:cbfd2d71:::smithi09025264-37:head [write 581575~705262,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:46.610 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:32:46.610 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 147.20580385599999, 2023-01-15T14:32:46.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.375425221, 2023-01-15T14:32:46.610 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.610 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:46.610 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.610 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.611 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.611 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1480 2023-01-15T14:32:46.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.611 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:32:46.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:32:46.612 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.612 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.612 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.612 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339019+0000", 2023-01-15T14:32:46.612 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:46.612 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.612 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.612 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339659+0000", 2023-01-15T14:32:46.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000639742 2023-01-15T14:32:46.613 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339690+0000", 2023-01-15T14:32:46.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1594000000000003e-05 2023-01-15T14:32:46.613 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339703+0000", 2023-01-15T14:32:46.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2828e-05 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.510837+0000", 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1711338759999999 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511048+0000", 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021057899999999999 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.615 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:46.615 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511325+0000", 2023-01-15T14:32:46.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027690400000000002 2023-01-15T14:32:46.616 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.616 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.616 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:46.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512092+0000", 2023-01-15T14:32:46.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00076720800000000002 2023-01-15T14:32:46.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:46.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512113+0000", 2023-01-15T14:32:46.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0917e-05 2023-01-15T14:32:46.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:46.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512439+0000", 2023-01-15T14:32:46.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032636299999999998 2023-01-15T14:32:46.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:46.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512871+0000", 2023-01-15T14:32:46.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000432087 2023-01-15T14:32:46.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.514165+0000", 2023-01-15T14:32:46.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001293726 2023-01-15T14:32:46.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714137+0000", 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19997229499999999 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714233+0000", 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6062000000000003e-05 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714446+0000", 2023-01-15T14:32:46.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212952 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1481 3.13 3:cbfd2d71:::smithi09025264-37:head [write 1733272~707409] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 147.19895457300001, 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.572522502, 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.622 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1481 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:32:46.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.624 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345868+0000", 2023-01-15T14:32:46.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:46.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.624 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346316+0000", 2023-01-15T14:32:46.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044813399999999999 2023-01-15T14:32:46.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346348+0000", 2023-01-15T14:32:46.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:32:46.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346362+0000", 2023-01-15T14:32:46.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.375e-05 2023-01-15T14:32:46.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512276+0000", 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1659136829999999 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512427+0000", 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015088699999999999 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714565+0000", 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20213841599999999 2023-01-15T14:32:46.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714732+0000", 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016676700000000001 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714964+0000", 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000231678 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715499+0000", 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053513300000000003 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715523+0000", 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4266000000000001e-05 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715867+0000", 2023-01-15T14:32:46.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034419399999999998 2023-01-15T14:32:46.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.816932+0000", 2023-01-15T14:32:46.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10106498999999999 2023-01-15T14:32:46.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:46.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906374+0000", 2023-01-15T14:32:46.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089441442999999995 2023-01-15T14:32:46.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918050+0000", 2023-01-15T14:32:46.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.011676677999999999 2023-01-15T14:32:46.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:46.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918218+0000", 2023-01-15T14:32:46.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016802000000000001 2023-01-15T14:32:46.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918393+0000", 2023-01-15T14:32:46.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017447 2023-01-15T14:32:46.631 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.631 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.631 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.631 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1482 3.13 3:cbfd2d71:::smithi09025264-37:head [write 3195938~437497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:46.631 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:32:46.632 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 147.19251871500001, 2023-01-15T14:32:46.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5664780289999998, 2023-01-15T14:32:46.632 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.632 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:46.632 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.632 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.632 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.632 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1482 2023-01-15T14:32:46.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.632 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352275+0000", 2023-01-15T14:32:46.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999695 2023-01-15T14:32:46.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352644+0000", 2023-01-15T14:32:46.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036870300000000001 2023-01-15T14:32:46.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352676+0000", 2023-01-15T14:32:46.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1726000000000001e-05 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352689+0000", 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3617e-05 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512602+0000", 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1599126690000001 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:46.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512839+0000", 2023-01-15T14:32:46.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023676900000000001 2023-01-15T14:32:46.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715649+0000", 2023-01-15T14:32:46.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20281044600000001 2023-01-15T14:32:46.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715777+0000", 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012738800000000001 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715995+0000", 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000217858 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:46.637 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716694+0000", 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00069956700000000001 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716709+0000", 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4362000000000001e-05 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716936+0000", 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022744200000000001 2023-01-15T14:32:46.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.818876+0000", 2023-01-15T14:32:46.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101939635 2023-01-15T14:32:46.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:46.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906390+0000", 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087513987000000001 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918577+0000", 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012187066 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918697+0000", 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120326 2023-01-15T14:32:46.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918784+0000", 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7200999999999994e-05 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1483 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3781575] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:32:46.641 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 147.192110563, 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5663522429999999, 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1483 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:32:46.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:32:46.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352713+0000", 2023-01-15T14:32:46.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:46.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352720+0000", 2023-01-15T14:32:46.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7959999999999998e-06 2023-01-15T14:32:46.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.760 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352735+0000", 2023-01-15T14:32:46.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.455e-05 2023-01-15T14:32:46.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352741+0000", 2023-01-15T14:32:46.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8360000000000003e-06 2023-01-15T14:32:46.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512901+0000", 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1601594980000001 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.513067+0000", 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016626 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716977+0000", 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20391002799999999 2023-01-15T14:32:46.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717146+0000", 2023-01-15T14:32:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168851 2023-01-15T14:32:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:32:46.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717467+0000", 2023-01-15T14:32:46.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032135600000000002 2023-01-15T14:32:46.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:46.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717800+0000", 2023-01-15T14:32:46.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033217400000000002 2023-01-15T14:32:46.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:46.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717822+0000", 2023-01-15T14:32:46.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2308999999999999e-05 2023-01-15T14:32:46.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:46.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717970+0000", 2023-01-15T14:32:46.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014788099999999999 2023-01-15T14:32:46.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.819135+0000", 2023-01-15T14:32:46.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116478 2023-01-15T14:32:46.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:46.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906408+0000", 2023-01-15T14:32:46.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087272902999999999 2023-01-15T14:32:46.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918887+0000", 2023-01-15T14:32:46.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012479965000000001 2023-01-15T14:32:46.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:46.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918959+0000", 2023-01-15T14:32:46.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1978999999999996e-05 2023-01-15T14:32:46.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.919067+0000", 2023-01-15T14:32:46.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107083 2023-01-15T14:32:46.770 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.770 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.770 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.770 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.770 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1485 3.d 3:bc1c6b44:::smithi09025264-23:head [write 598382~780511,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:46.770 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:32:46.770 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 147.184640129, 2023-01-15T14:32:46.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5474121890000001, 2023-01-15T14:32:46.770 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.771 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:46.771 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.771 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.771 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.771 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1485 2023-01-15T14:32:46.771 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.771 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.771 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.771 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.771 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:32:46.771 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.772 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.772 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.772 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.772 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:32:46.772 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.772 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.772 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.772 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.772 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360168+0000", 2023-01-15T14:32:46.772 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999828 2023-01-15T14:32:46.773 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.773 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.773 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.773 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360790+0000", 2023-01-15T14:32:46.773 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062269099999999998 2023-01-15T14:32:46.773 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.773 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.773 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.773 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360822+0000", 2023-01-15T14:32:46.773 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1903000000000003e-05 2023-01-15T14:32:46.773 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360836+0000", 2023-01-15T14:32:46.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3380000000000001e-05 2023-01-15T14:32:46.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096664+0000", 2023-01-15T14:32:46.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7358282320000002 2023-01-15T14:32:46.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:46.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096683+0000", 2023-01-15T14:32:46.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9063999999999999e-05 2023-01-15T14:32:46.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704561+0000", 2023-01-15T14:32:46.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60787808399999999 2023-01-15T14:32:46.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.776 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.776 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704846+0000", 2023-01-15T14:32:46.776 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028527400000000002 2023-01-15T14:32:46.776 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.776 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.776 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:46.776 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705135+0000", 2023-01-15T14:32:46.776 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028875400000000001 2023-01-15T14:32:46.776 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:46.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705735+0000", 2023-01-15T14:32:46.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059956899999999999 2023-01-15T14:32:46.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:46.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705761+0000", 2023-01-15T14:32:46.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:32:46.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:46.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.706129+0000", 2023-01-15T14:32:46.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036822299999999999 2023-01-15T14:32:46.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:46.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906331+0000", 2023-01-15T14:32:46.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200202091 2023-01-15T14:32:46.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907111+0000", 2023-01-15T14:32:46.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077990799999999995 2023-01-15T14:32:46.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907379+0000", 2023-01-15T14:32:46.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026804800000000001 2023-01-15T14:32:46.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.780 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.780 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:46.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907529+0000", 2023-01-15T14:32:46.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015022700000000001 2023-01-15T14:32:46.780 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.780 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.780 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907597+0000", 2023-01-15T14:32:46.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7785000000000002e-05 2023-01-15T14:32:46.781 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.781 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.781 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.781 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1486 3.d 3:bc1c6b44:::smithi09025264-23:head [write 2075401~451067] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:46.781 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:32:46.781 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 147.178244081, 2023-01-15T14:32:46.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6461318330000001, 2023-01-15T14:32:46.782 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.782 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:46.782 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.782 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.782 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.782 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1486 2023-01-15T14:32:46.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.782 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366579+0000", 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:46.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366845+0000", 2023-01-15T14:32:46.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026610499999999999 2023-01-15T14:32:46.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366878+0000", 2023-01-15T14:32:46.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2750000000000003e-05 2023-01-15T14:32:46.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.467009+0000", 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100131071 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096793+0000", 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6297846869999999 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:32:46.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096806+0000", 2023-01-15T14:32:46.786 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3077e-05 2023-01-15T14:32:46.786 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.786 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.786 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.786 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907957+0000", 2023-01-15T14:32:46.786 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81115055400000002 2023-01-15T14:32:46.786 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.786 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.786 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.786 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908229+0000", 2023-01-15T14:32:46.786 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027187499999999998 2023-01-15T14:32:46.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.787 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:46.787 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908496+0000", 2023-01-15T14:32:46.787 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026754200000000001 2023-01-15T14:32:46.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.787 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:46.787 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909025+0000", 2023-01-15T14:32:46.787 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052865200000000001 2023-01-15T14:32:46.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.788 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.788 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:46.788 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909046+0000", 2023-01-15T14:32:46.788 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0713e-05 2023-01-15T14:32:46.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.788 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.788 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:46.788 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909294+0000", 2023-01-15T14:32:46.788 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024864899999999998 2023-01-15T14:32:46.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.788 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.911607+0000", 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023128939999999998 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012076+0000", 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100468293 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:46.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012426+0000", 2023-01-15T14:32:46.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035044299999999998 2023-01-15T14:32:46.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:46.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012532+0000", 2023-01-15T14:32:46.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106142 2023-01-15T14:32:46.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012713+0000", 2023-01-15T14:32:46.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018048599999999999 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1496 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 146.808601232, 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870616037, 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:46.791 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.943 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.943 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.943 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1496 2023-01-15T14:32:46.943 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.943 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.943 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.944 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.944 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:32:46.944 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.944 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.944 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.945 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.945 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:32:46.945 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.945 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.945 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.945 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.946 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736220+0000", 2023-01-15T14:32:46.946 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:32:46.946 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.946 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.946 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.947 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736238+0000", 2023-01-15T14:32:46.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7156e-05 2023-01-15T14:32:46.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.947 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.947 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.948 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736275+0000", 2023-01-15T14:32:46.948 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7876000000000001e-05 2023-01-15T14:32:46.948 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.949 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.949 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736290+0000", 2023-01-15T14:32:46.949 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4449e-05 2023-01-15T14:32:46.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602217+0000", 2023-01-15T14:32:46.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8659270299999999 2023-01-15T14:32:46.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.951 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.951 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602468+0000", 2023-01-15T14:32:46.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000251315 2023-01-15T14:32:46.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.953 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.953 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.606840+0000", 2023-01-15T14:32:46.953 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0043715009999999999 2023-01-15T14:32:46.953 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.953 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.953 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.954 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.954 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.954 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1497 3.11 3:8889e450:::smithi09025264-32:head [read 0~2991646,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:46.954 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:32:46.954 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 146.80851377799999, 2023-01-15T14:32:46.954 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738359249999998, 2023-01-15T14:32:46.954 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.954 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:46.954 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.955 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.955 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.955 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1497 2023-01-15T14:32:46.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.955 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.955 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.955 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:32:46.955 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.956 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:32:46.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736310+0000", 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736317+0000", 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7240000000000003e-06 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.957 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:46.958 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736334+0000", 2023-01-15T14:32:46.958 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6878999999999999e-05 2023-01-15T14:32:46.958 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.958 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.958 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:46.958 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736341+0000", 2023-01-15T14:32:46.959 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0129999999999996e-06 2023-01-15T14:32:46.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.959 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.959 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:46.959 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607070+0000", 2023-01-15T14:32:46.959 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870728873 2023-01-15T14:32:46.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.959 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.960 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:46.960 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607226+0000", 2023-01-15T14:32:46.960 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015595 2023-01-15T14:32:46.960 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.960 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.960 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:46.960 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610147+0000", 2023-01-15T14:32:46.960 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.002921416 2023-01-15T14:32:46.960 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.960 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:46.960 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:46.961 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.961 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.961 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1498 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:32:46.961 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:32:46.961 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 146.80847367199999, 2023-01-15T14:32:46.961 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8752667839999999, 2023-01-15T14:32:46.961 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:46.961 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:46.961 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:32:46.961 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:32:46.961 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:46.962 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1498 2023-01-15T14:32:46.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.962 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:46.962 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.962 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:46.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:32:46.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:46.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:46.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:32:46.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:46.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:46.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:47.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736350+0000", 2023-01-15T14:32:47.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:32:47.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:47.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736355+0000", 2023-01-15T14:32:47.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4900000000000002e-06 2023-01-15T14:32:47.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736363+0000", 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2770000000000005e-06 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736369+0000", 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7400000000000001e-06 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610234+0000", 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738651439999998 2023-01-15T14:32:47.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610402+0000", 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168283 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.611618+0000", 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001215647 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:47.147 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:47.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.148 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.7.0:440 3.11 e231/207 ondisk, result = 0)", 2023-01-15T14:32:47.148 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:32:47.148 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 48.300692013999999, 2023-01-15T14:32:47.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973962139999998, 2023-01-15T14:32:47.148 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:47.148 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:47.148 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:47.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:47.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:32:47.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:47.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:47.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:32:47.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:47.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:47.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244132+0000", 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244143+0000", 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1647e-05 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244165+0000", 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1501999999999998e-05 2023-01-15T14:32:47.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:47.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244180+0000", 2023-01-15T14:32:47.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5434000000000002e-05 2023-01-15T14:32:47.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:47.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941489+0000", 2023-01-15T14:32:47.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973091899999999 2023-01-15T14:32:47.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:47.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941503+0000", 2023-01-15T14:32:47.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.414e-05 2023-01-15T14:32:47.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:47.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941529+0000", 2023-01-15T14:32:47.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5622999999999998e-05 2023-01-15T14:32:47.152 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:47.152 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:47.152 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.4.0:562 3.1b e231/224 3:d85f42d9:::smithi09025264-45:a8 v 231'66, mlcod=225'62)", 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 48.110340817999997, 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2578820149999999, 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:47.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:47.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:47.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:47.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:47.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:47.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:47.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:32:47.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434492+0000", 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4360000000000002e-06 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434517+0000", 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.48e-05 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:47.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434530+0000", 2023-01-15T14:32:47.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2982999999999999e-05 2023-01-15T14:32:47.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:47.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434663+0000", 2023-01-15T14:32:47.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013307 2023-01-15T14:32:47.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:47.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434700+0000", 2023-01-15T14:32:47.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7020000000000001e-05 2023-01-15T14:32:47.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:47.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435438+0000", 2023-01-15T14:32:47.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073808900000000004 2023-01-15T14:32:47.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:47.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435456+0000", 2023-01-15T14:32:47.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8091999999999999e-05 2023-01-15T14:32:47.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:47.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435526+0000", 2023-01-15T14:32:47.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9806000000000002e-05 2023-01-15T14:32:47.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:47.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435570+0000", 2023-01-15T14:32:47.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4091999999999997e-05 2023-01-15T14:32:47.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:47.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:47.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:47.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.692366+0000", 2023-01-15T14:32:47.159 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2567961539999999 2023-01-15T14:32:47.159 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:47.159 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:47.159 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:47.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:24 3.d 3:b3f111b5:::smithi09025264-48:head [assert-version v160,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e232)", 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 47.594504602000001, 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9963031440000001, 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950318+0000", 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:49.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950333+0000", 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.467e-05 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950368+0000", 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4895999999999999e-05 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050461+0000", 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10009304400000001 2023-01-15T14:32:49.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:49.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050593+0000", 2023-01-15T14:32:49.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000132764 2023-01-15T14:32:49.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:32:49.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050615+0000", 2023-01-15T14:32:49.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1483000000000001e-05 2023-01-15T14:32:49.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.036 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:49.036 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945049+0000", 2023-01-15T14:32:49.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.894434138 2023-01-15T14:32:49.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.036 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:49.036 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945326+0000", 2023-01-15T14:32:49.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000277368 2023-01-15T14:32:49.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.036 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.946623+0000", 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012970340000000001 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.3.0:547 3.22 e232/180 3:47dfe935:::smithi09025264-29:81 v 232'124, mlcod=229'121)", 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 47.494274853999997, 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.643153431, 2023-01-15T14:32:49.037 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:32:49.038 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:32:49.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050549+0000", 2023-01-15T14:32:49.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:49.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:32:49.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050561+0000", 2023-01-15T14:32:49.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.278e-05 2023-01-15T14:32:49.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:32:49.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050583+0000", 2023-01-15T14:32:49.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1053e-05 2023-01-15T14:32:49.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:32:49.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050596+0000", 2023-01-15T14:32:49.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3946e-05 2023-01-15T14:32:49.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.041 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.041 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:32:49.041 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050708+0000", 2023-01-15T14:32:49.041 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011129 2023-01-15T14:32:49.041 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.041 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.041 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:32:49.041 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050750+0000", 2023-01-15T14:32:49.041 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2725000000000002e-05 2023-01-15T14:32:49.042 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:49.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051324+0000", 2023-01-15T14:32:49.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00057335700000000003 2023-01-15T14:32:49.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:49.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051340+0000", 2023-01-15T14:32:49.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5968999999999999e-05 2023-01-15T14:32:49.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.044 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:49.044 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051405+0000", 2023-01-15T14:32:49.044 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5419000000000005e-05 2023-01-15T14:32:49.044 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.044 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.187 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:32:49.187 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051471+0000", 2023-01-15T14:32:49.187 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5671000000000002e-05 2023-01-15T14:32:49.187 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:32:49.187 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:32:49.187 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:32:49.187 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.693703+0000", 2023-01-15T14:32:49.187 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.642232554 2023-01-15T14:32:49.188 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:49.188 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:49.188 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:49.188 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:32:49.188 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:32:49.188 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:49.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:32:49.189 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:32:49.189 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:32:49.189 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-01-15T14:32:49.189 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-01-15T14:32:49.190 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:32:49.192 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:49.184+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:49.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:32:49.193 INFO:tasks.rados.rados.0.smithi090.stdout:1138: done (0 left) 2023-01-15T14:32:49.194 INFO:tasks.rados.rados.0.smithi090.stdout:1139: delete oid 49 current snap is 113 2023-01-15T14:32:49.195 INFO:tasks.rados.rados.0.smithi090.stdout:1139: done (0 left) 2023-01-15T14:32:49.195 INFO:tasks.rados.rados.0.smithi090.stdout:1140: snap_create 2023-01-15T14:32:49.195 INFO:tasks.rados.rados.0.smithi090.stdout:1140: done (0 left) 2023-01-15T14:32:49.195 INFO:tasks.rados.rados.0.smithi090.stdout:1141: snap_create 2023-01-15T14:32:49.195 INFO:tasks.rados.rados.0.smithi090.stdout:1141: done (0 left) 2023-01-15T14:32:49.195 INFO:tasks.rados.rados.0.smithi090.stdout:1142: read oid 40 snap -1 2023-01-15T14:32:49.195 INFO:tasks.rados.rados.0.smithi090.stdout:1142: expect (ObjNum 419 snap 112 seq_num 419) 2023-01-15T14:32:49.195 INFO:tasks.rados.rados.0.smithi090.stdout:1143: write (excl) oid 10 current snap is 115 2023-01-15T14:32:49.195 INFO:tasks.rados.rados.0.smithi090.stdout:1143: seq_num 424 ranges {726168=704991,1895071=683474} 2023-01-15T14:32:49.196 INFO:tasks.rados.rados.0.smithi090.stdout:1143: writing smithi09025264-10 from 726168 to 1431159 tid 1 2023-01-15T14:32:49.196 INFO:tasks.rados.rados.0.smithi090.stdout:1143: writing smithi09025264-10 from 1895071 to 2578545 tid 2 2023-01-15T14:32:49.196 INFO:tasks.rados.rados.0.smithi090.stdout:1144: snap_create 2023-01-15T14:32:49.196 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:49.290 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:49.284+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:49.368 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:32:49.368 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:49.369 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:49.371 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:32:49.371 INFO:tasks.rados.rados.0.smithi090.stdout:1142: expect (ObjNum 419 snap 112 seq_num 419) 2023-01-15T14:32:49.529 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:49.529 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:49.529 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:49.529 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:49.530 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:49.530 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:49.530 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:49.531 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:32:49.532 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:32:49.532 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:32:49.532 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:32:49.532 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:32:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:32:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:32:49.533 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:32:49.535 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:49.528+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:49.561 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:32:49.729 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 281, 2023-01-15T14:32:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "1c77fb2b-d1a7-4884-a24e-3935e16d0b6b", 2023-01-15T14:32:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2023-01-15T14:27:57.735717+0000", 2023-01-15T14:32:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2023-01-15T14:32:49.011445+0000", 2023-01-15T14:32:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2023-01-15T14:28:04.837522+0000", 2023-01-15T14:32:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2023-01-15T14:30:49.906949+0000", 2023-01-15T14:32:49.729 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-01-15T14:32:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5806080, 2023-01-15T14:32:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2023-01-15T14:32:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2023-01-15T14:32:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2023-01-15T14:32:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2023-01-15T14:32:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2023-01-15T14:32:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2023-01-15T14:32:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2023-01-15T14:32:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2023-01-15T14:32:49.730 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2023-01-15T14:32:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-01-15T14:32:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2023-01-15T14:32:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2023-01-15T14:32:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 3, 2023-01-15T14:32:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2023-01-15T14:32:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2023-01-15T14:32:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2023-01-15T14:32:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "reef", 2023-01-15T14:32:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2023-01-15T14:32:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.731 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2023-01-15T14:32:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2023-01-15T14:32:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:04.899282+0000", 2023-01-15T14:32:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:32:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:32:49.732 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:32:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:32:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:32:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:32:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:32:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:32:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:32:49.733 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:32:49.734 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:32:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:32:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2023-01-15T14:32:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2023-01-15T14:32:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2023-01-15T14:32:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2023-01-15T14:32:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2023-01-15T14:32:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:32:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:32:49.735 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:32:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:32:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:32:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:32:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:32:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "16", 2023-01-15T14:32:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:32:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:32:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:32:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:32:49.736 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:32:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:32:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:32:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:32:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:32:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:32:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:32:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:32:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:32:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:32:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:32:49.737 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:32:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:32:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:32:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:32:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:32:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:32:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:32:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:32:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:32:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:32:49.738 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:32:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:32:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:32:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:32:49.739 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:32:49.740 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:32:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:32:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:32:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:32:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:32:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:32:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:32:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2023-01-15T14:32:49.742 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2023-01-15T14:32:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2023-01-15T14:32:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:32:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2023-01-15T14:32:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2023-01-15T14:32:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2023-01-15T14:32:49.743 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:11.115002+0000", 2023-01-15T14:32:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:32:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:32:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:32:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:32:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:32:49.744 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:32:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:32:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:32:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:32:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:32:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:32:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:32:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2023-01-15T14:32:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2023-01-15T14:32:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2023-01-15T14:32:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2023-01-15T14:32:49.745 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2023-01-15T14:32:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:32:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:32:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:32:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:32:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:32:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:32:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:32:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "19", 2023-01-15T14:32:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:32:49.746 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:32:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:32:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:32:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:32:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:32:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:32:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:32:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:32:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:32:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:32:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:32:49.747 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:32:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:32:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:32:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:32:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:32:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:32:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:32:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:32:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:32:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:32:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:32:49.748 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:32:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:32:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:32:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:32:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:32:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:32:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:32:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:32:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:32:49.749 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:32:49.750 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "unique_pool_0", 2023-01-15T14:32:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:22.434903+0000", 2023-01-15T14:32:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 16385, 2023-01-15T14:32:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,pool_snaps", 2023-01-15T14:32:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:32:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:32:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:32:49.751 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:32:49.931 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:32:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:32:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:32:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:32:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:32:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:32:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 76, 2023-01-15T14:32:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 76, 2023-01-15T14:32:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 76, 2023-01-15T14:32:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 76, 2023-01-15T14:32:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 76, 2023-01-15T14:32:49.932 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:32:49.933 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:32:49.933 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:32:49.933 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:32:49.933 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:32:49.933 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:32:49.933 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:32:49.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.933 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "281", 2023-01-15T14:32:49.934 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:32:49.934 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:32:49.934 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "254", 2023-01-15T14:32:49.934 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:32:49.934 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "pool", 2023-01-15T14:32:49.935 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 222, 2023-01-15T14:32:49.935 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 281, 2023-01-15T14:32:49.935 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [ 2023-01-15T14:32:49.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.935 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 205, 2023-01-15T14:32:49.935 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:29.932293+0000", 2023-01-15T14:32:49.936 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap104" 2023-01-15T14:32:49.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.936 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 216, 2023-01-15T14:32:49.936 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:41.495642+0000", 2023-01-15T14:32:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap109" 2023-01-15T14:32:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 217, 2023-01-15T14:32:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:42.425395+0000", 2023-01-15T14:32:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap110" 2023-01-15T14:32:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 218, 2023-01-15T14:32:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:43.736677+0000", 2023-01-15T14:32:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap111" 2023-01-15T14:32:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 219, 2023-01-15T14:32:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:43.999592+0000", 2023-01-15T14:32:49.940 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap112" 2023-01-15T14:32:49.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.940 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 220, 2023-01-15T14:32:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:46.266030+0000", 2023-01-15T14:32:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap113" 2023-01-15T14:32:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 221, 2023-01-15T14:32:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:47.161506+0000", 2023-01-15T14:32:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap114" 2023-01-15T14:32:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 222, 2023-01-15T14:32:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:48.022241+0000", 2023-01-15T14:32:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap115" 2023-01-15T14:32:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:32:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:32:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:32:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:32:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:32:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:32:49.945 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:32:49.945 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:32:49.945 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:32:49.945 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:32:49.945 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:32:49.945 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:32:49.946 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:32:49.946 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:32:49.946 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:32:49.946 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:32:49.946 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:32:49.947 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:32:49.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:49.947 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:32:49.947 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:32:49.947 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:32:49.948 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:32:49.948 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:32:49.948 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:32:49.948 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:32:49.948 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:32:49.949 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:32:49.949 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:32:49.949 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:32:49.949 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:32:49.949 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:32:49.950 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2023-01-15T14:32:49.950 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:49.950 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:49.950 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:49.950 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:32:49.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.951 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:32:49.951 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e55c5ac4-3cff-4ee5-8719-373648f83d8d", 2023-01-15T14:32:49.951 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:49.951 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:49.952 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.3336334228515625, 2023-01-15T14:32:49.952 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:49.952 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:49.952 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:49.952 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 261, 2023-01-15T14:32:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:49.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:49.954 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2023-01-15T14:32:50.137 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2023-01-15T14:32:50.138 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:50.139 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2023-01-15T14:32:50.140 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.141 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2023-01-15T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.142 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2023-01-15T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2023-01-15T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.144 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6811/23473", 2023-01-15T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6815/23473", 2023-01-15T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/23473", 2023-01-15T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6820/23473", 2023-01-15T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:50.145 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "2c1b33c9-0a1e-4734-936d-d51d64fe997a", 2023-01-15T14:32:50.146 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 272, 2023-01-15T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:50.147 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2023-01-15T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2023-01-15T14:32:50.148 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2023-01-15T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2023-01-15T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.150 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2023-01-15T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2023-01-15T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2023-01-15T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.152 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2023-01-15T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/23474", 2023-01-15T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/23474", 2023-01-15T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/23474", 2023-01-15T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/23474", 2023-01-15T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:50.153 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "b314174a-eca9-4baa-a60c-2589f1fc9a8e", 2023-01-15T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:50.154 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0, 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 234, 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.155 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2023-01-15T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2023-01-15T14:32:50.156 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2023-01-15T14:32:50.157 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2023-01-15T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:50.158 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2023-01-15T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.160 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2023-01-15T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6810/23475", 2023-01-15T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/23475", 2023-01-15T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6821/23475", 2023-01-15T14:32:50.161 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6817/23475", 2023-01-15T14:32:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:50.349 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:32:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "29eb0b17-ed5d-4665-a347-2fb8406d9c62", 2023-01-15T14:32:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.9044189453125, 2023-01-15T14:32:50.350 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 260, 2023-01-15T14:32:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.351 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6824", 2023-01-15T14:32:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6825", 2023-01-15T14:32:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6826", 2023-01-15T14:32:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.353 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6827", 2023-01-15T14:32:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6830", 2023-01-15T14:32:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6831", 2023-01-15T14:32:50.355 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.356 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6828", 2023-01-15T14:32:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6829", 2023-01-15T14:32:50.358 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:32:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6825/23778", 2023-01-15T14:32:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6827/23778", 2023-01-15T14:32:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6831/23778", 2023-01-15T14:32:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6829/23778", 2023-01-15T14:32:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:50.359 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:32:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "59362feb-caa9-44e3-b6d0-b79397dacbfc", 2023-01-15T14:32:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:32:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:50.360 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 273, 2023-01-15T14:32:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:50.361 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6800", 2023-01-15T14:32:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6801", 2023-01-15T14:32:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:50.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6802", 2023-01-15T14:32:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.363 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6803", 2023-01-15T14:32:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6806", 2023-01-15T14:32:50.364 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6807", 2023-01-15T14:32:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6804", 2023-01-15T14:32:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6805", 2023-01-15T14:32:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:32:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.366 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.367 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6801/23523", 2023-01-15T14:32:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6803/23523", 2023-01-15T14:32:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6807/23523", 2023-01-15T14:32:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6805/23523", 2023-01-15T14:32:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.368 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:32:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "fbc8821c-8d78-40f8-a571-b18a70d42000", 2023-01-15T14:32:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:32:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:32:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:50.369 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:32:50.370 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:50.546 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6824", 2023-01-15T14:32:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.547 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6825", 2023-01-15T14:32:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6826", 2023-01-15T14:32:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:50.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6827", 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.549 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6830", 2023-01-15T14:32:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6831", 2023-01-15T14:32:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:50.550 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6828", 2023-01-15T14:32:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6829", 2023-01-15T14:32:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:32:50.551 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.707 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6825/23525", 2023-01-15T14:32:50.707 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6827/23525", 2023-01-15T14:32:50.707 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6831/23525", 2023-01-15T14:32:50.707 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6829/23525", 2023-01-15T14:32:50.707 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:50.707 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:50.708 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:50.708 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.713 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:32:50.714 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "34550470-3561-492f-b605-b5f0e8052c1e", 2023-01-15T14:32:50.714 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:50.714 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:32:50.714 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:32:50.714 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:50.714 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:50.714 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:50.714 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:50.714 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 118, 2023-01-15T14:32:50.714 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:50.714 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:50.715 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:50.715 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.715 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6808", 2023-01-15T14:32:50.715 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:50.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.715 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6809", 2023-01-15T14:32:50.717 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:50.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.717 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.717 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:50.717 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.718 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.718 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6810", 2023-01-15T14:32:50.718 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:50.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.718 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.718 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6811", 2023-01-15T14:32:50.719 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:50.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.719 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.719 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:50.719 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.719 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.719 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6814", 2023-01-15T14:32:50.719 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:50.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.720 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.720 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6815", 2023-01-15T14:32:50.720 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:50.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:50.865 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.866 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.866 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6812", 2023-01-15T14:32:50.866 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:50.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.868 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.868 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6813", 2023-01-15T14:32:50.868 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:32:50.868 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.868 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.868 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6809/23524", 2023-01-15T14:32:50.868 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6811/23524", 2023-01-15T14:32:50.869 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6815/23524", 2023-01-15T14:32:50.869 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6813/23524", 2023-01-15T14:32:50.869 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:50.869 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:50.870 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:50.870 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.870 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:32:50.870 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1f123d0d-d617-4771-b516-21aa5a29b734", 2023-01-15T14:32:50.870 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:32:50.870 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:32:50.870 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.5779571533203125, 2023-01-15T14:32:50.870 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:32:50.870 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:32:50.871 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:32:50.871 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:32:50.871 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 263, 2023-01-15T14:32:50.871 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:32:50.871 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:32:50.871 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:32:50.871 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.871 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.871 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6816", 2023-01-15T14:32:50.872 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:50.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.872 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.872 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6817", 2023-01-15T14:32:50.872 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:50.872 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.872 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.872 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:32:50.873 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.873 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.873 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6818", 2023-01-15T14:32:50.873 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:50.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.873 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.873 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6819", 2023-01-15T14:32:50.873 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:50.874 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.874 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.874 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:32:50.874 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.874 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.874 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6822", 2023-01-15T14:32:50.874 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:50.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.875 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.875 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6823", 2023-01-15T14:32:50.875 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:50.875 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.875 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.875 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:32:50.875 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:32:50.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.876 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:32:50.876 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6820", 2023-01-15T14:32:50.876 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:50.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.876 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:32:50.876 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6821", 2023-01-15T14:32:50.876 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:32:50.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.877 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.877 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6817/23526", 2023-01-15T14:32:50.877 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6819/23526", 2023-01-15T14:32:50.877 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6823/23526", 2023-01-15T14:32:50.877 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6821/23526", 2023-01-15T14:32:50.877 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:32:50.877 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:32:50.877 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:32:50.877 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:47.556519+0000", 2023-01-15T14:32:50.878 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:54.007074+0000", 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.880 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:32:50.880 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:50.880 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:50.880 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:50.880 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:50.880 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:50.880 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:53.308067+0000", 2023-01-15T14:32:50.880 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:50.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.881 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:32:50.881 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:50.881 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:50.881 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:50.881 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:50.881 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:50.881 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:43.268238+0000", 2023-01-15T14:32:50.881 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:50.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:51.428540+0000", 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:32:50.882 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:50.883 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:50.883 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:50.883 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:50.883 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:32:50.883 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:32:32.921684+0000", 2023-01-15T14:32:50.883 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:50.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.883 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:32:50.883 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:50.884 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:50.884 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:50.884 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:50.884 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:32:50.884 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:55.585270+0000", 2023-01-15T14:32:50.884 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:50.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.884 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:32:50.884 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:32:50.885 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:32:50.885 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:32:50.885 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:32:50.885 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:32:50.885 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:39.052183+0000", 2023-01-15T14:32:50.885 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:32:50.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.885 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:50.885 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2023-01-15T14:32:50.885 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2023-01-15T14:32:50.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.886 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.2d", 2023-01-15T14:32:50.886 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:32:50.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.886 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 2, 2023-01-15T14:32:50.886 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 3 2023-01-15T14:32:50.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.887 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.887 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:50.887 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2023-01-15T14:32:50.887 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [], 2023-01-15T14:32:50.887 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2023-01-15T14:32:50.887 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2023-01-15T14:32:50.887 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2023-01-15T14:32:50.887 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2023-01-15T14:32:50.888 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2023-01-15T14:32:50.888 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2023-01-15T14:32:50.888 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2023-01-15T14:32:50.888 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2023-01-15T14:32:50.888 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2023-01-15T14:32:50.888 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2023-01-15T14:32:50.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:50.888 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2023-01-15T14:32:50.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.889 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:32:50.889 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2023-01-15T14:32:50.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:50.889 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 213, 2023-01-15T14:32:50.889 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:32:50.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.889 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:50.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:50.889 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:32:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2023-01-15T14:32:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2023-01-15T14:32:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2023-01-15T14:32:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2023-01-15T14:32:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2023-01-15T14:32:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2023-01-15T14:32:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2023-01-15T14:32:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2023-01-15T14:32:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2023-01-15T14:32:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2023-01-15T14:32:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:51.037 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:51.039 INFO:tasks.thrashosds.thrasher:j is {'epoch': 281, 'fsid': '1c77fb2b-d1a7-4884-a24e-3935e16d0b6b', 'created': '2023-01-15T14:27:57.735717+0000', 'modified': '2023-01-15T14:32:49.011445+0000', 'last_up_change': '2023-01-15T14:28:04.837522+0000', 'last_in_change': '2023-01-15T14:30:49.906949+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-01-15T14:28:04.899282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-01-15T14:28:11.115002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-01-15T14:28:22.434903+0000', 'flags': 16385, 'flags_names': 'hashpspool,pool_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 76, 'pg_placement_num': 76, 'pg_placement_num_target': 76, 'pg_num_target': 76, 'pg_num_pending': 76, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '281', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '254', 'auid': 0, 'snap_mode': 'pool', 'snap_seq': 222, 'snap_epoch': 281, 'pool_snaps': [{'snapid': 205, 'stamp': '2023-01-15T14:32:29.932293+0000', 'name': 'smithi09025264-snap104'}, {'snapid': 216, 'stamp': '2023-01-15T14:32:41.495642+0000', 'name': 'smithi09025264-snap109'}, {'snapid': 217, 'stamp': '2023-01-15T14:32:42.425395+0000', 'name': 'smithi09025264-snap110'}, {'snapid': 218, 'stamp': '2023-01-15T14:32:43.736677+0000', 'name': 'smithi09025264-snap111'}, {'snapid': 219, 'stamp': '2023-01-15T14:32:43.999592+0000', 'name': 'smithi09025264-snap112'}, {'snapid': 220, 'stamp': '2023-01-15T14:32:46.266030+0000', 'name': 'smithi09025264-snap113'}, {'snapid': 221, 'stamp': '2023-01-15T14:32:47.161506+0000', 'name': 'smithi09025264-snap114'}, {'snapid': 222, 'stamp': '2023-01-15T14:32:48.022241+0000', 'name': 'smithi09025264-snap115'}], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e55c5ac4-3cff-4ee5-8719-373648f83d8d', 'up': 1, 'in': 1, 'weight': 0.3336334228515625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 261, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 23473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 23473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 23473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6820', 'nonce': 23473}]}, 'public_addr': '172.21.15.90:6811/23473', 'cluster_addr': '172.21.15.90:6815/23473', 'heartbeat_back_addr': '172.21.15.90:6823/23473', 'heartbeat_front_addr': '172.21.15.90:6820/23473', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2c1b33c9-0a1e-4734-936d-d51d64fe997a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 272, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 23474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 23474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 23474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 23474}]}, 'public_addr': '172.21.15.90:6801/23474', 'cluster_addr': '172.21.15.90:6803/23474', 'heartbeat_back_addr': '172.21.15.90:6807/23474', 'heartbeat_front_addr': '172.21.15.90:6805/23474', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b314174a-eca9-4baa-a60c-2589f1fc9a8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 234, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 23475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 23475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6819', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 23475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 23475}]}, 'public_addr': '172.21.15.90:6810/23475', 'cluster_addr': '172.21.15.90:6814/23475', 'heartbeat_back_addr': '172.21.15.90:6821/23475', 'heartbeat_front_addr': '172.21.15.90:6817/23475', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '29eb0b17-ed5d-4665-a347-2fb8406d9c62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9044189453125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 260, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6824', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6825', 'nonce': 23778}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6826', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6827', 'nonce': 23778}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6830', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6831', 'nonce': 23778}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6828', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6829', 'nonce': 23778}]}, 'public_addr': '172.21.15.90:6825/23778', 'cluster_addr': '172.21.15.90:6827/23778', 'heartbeat_back_addr': '172.21.15.90:6831/23778', 'heartbeat_front_addr': '172.21.15.90:6829/23778', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '59362feb-caa9-44e3-b6d0-b79397dacbfc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 273, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6800', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6801', 'nonce': 23523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6802', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6803', 'nonce': 23523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6806', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6807', 'nonce': 23523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6804', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6805', 'nonce': 23523}]}, 'public_addr': '172.21.15.141:6801/23523', 'cluster_addr': '172.21.15.141:6803/23523', 'heartbeat_back_addr': '172.21.15.141:6807/23523', 'heartbeat_front_addr': '172.21.15.141:6805/23523', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fbc8821c-8d78-40f8-a571-b18a70d42000', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6824', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6825', 'nonce': 23525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6826', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6827', 'nonce': 23525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6830', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6831', 'nonce': 23525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6828', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6829', 'nonce': 23525}]}, 'public_addr': '172.21.15.141:6825/23525', 'cluster_addr': '172.21.15.141:6827/23525', 'heartbeat_back_addr': '172.21.15.141:6831/23525', 'heartbeat_front_addr': '172.21.15.141:6829/23525', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '34550470-3561-492f-b605-b5f0e8052c1e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6808', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6809', 'nonce': 23524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6810', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6811', 'nonce': 23524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6814', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6815', 'nonce': 23524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6812', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6813', 'nonce': 23524}]}, 'public_addr': '172.21.15.141:6809/23524', 'cluster_addr': '172.21.15.141:6811/23524', 'heartbeat_back_addr': '172.21.15.141:6815/23524', 'heartbeat_front_addr': '172.21.15.141:6813/23524', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f123d0d-d617-4771-b516-21aa5a29b734', 'up': 1, 'in': 1, 'weight': 0.5779571533203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 263, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6816', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6817', 'nonce': 23526}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6818', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6819', 'nonce': 23526}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6822', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6823', 'nonce': 23526}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6820', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6821', 'nonce': 23526}]}, 'public_addr': '172.21.15.141:6817/23526', 'cluster_addr': '172.21.15.141:6819/23526', 'heartbeat_back_addr': '172.21.15.141:6823/23526', 'heartbeat_front_addr': '172.21.15.141:6821/23526', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:47.556519+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:54.007074+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:53.308067+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:43.268238+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:51.428540+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:32:32.921684+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:31:55.585270+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:39.052183+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.2d', 'mappings': [{'from': 2, 'to': 3}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 213, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-01-15T14:32:51.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:32:51.040 INFO:tasks.rados.rados.0.smithi090.stdout:1143: finishing write tid 1 to smithi09025264-10 2023-01-15T14:32:51.040 INFO:tasks.rados.rados.0.smithi090.stdout:1142: done (1 left) 2023-01-15T14:32:51.040 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:51.041 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:51.041 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:51.042 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:32:51.042 INFO:tasks.rados.rados.0.smithi090.stdout:1143: finishing write tid 2 to smithi09025264-10 2023-01-15T14:32:51.042 INFO:tasks.rados.rados.0.smithi090.stdout:1143: finishing write tid 3 to smithi09025264-10 2023-01-15T14:32:51.043 INFO:tasks.rados.rados.0.smithi090.stdout:1143: finishing write tid 4 to smithi09025264-10 2023-01-15T14:32:51.043 INFO:tasks.rados.rados.0.smithi090.stdout:1143: oid 10 updating version 0 to 372 2023-01-15T14:32:51.043 INFO:tasks.rados.rados.0.smithi090.stdout:1143: oid 10 updating version 372 to 374 2023-01-15T14:32:51.043 INFO:tasks.rados.rados.0.smithi090.stdout:1143: oid 10 version 374 is already newer than 373 2023-01-15T14:32:51.043 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 374 (ObjNum 424 snap 115 seq_num 424) dirty exists 2023-01-15T14:32:51.043 INFO:tasks.rados.rados.0.smithi090.stdout:1143: left oid 10 (ObjNum 424 snap 115 seq_num 424) 2023-01-15T14:32:51.043 INFO:tasks.rados.rados.0.smithi090.stdout:1143: done (0 left) 2023-01-15T14:32:51.043 INFO:tasks.rados.rados.0.smithi090.stdout:1144: done (0 left) 2023-01-15T14:32:51.043 INFO:tasks.rados.rados.0.smithi090.stdout:1145: write (excl) oid 48 current snap is 116 2023-01-15T14:32:51.043 INFO:tasks.rados.rados.0.smithi090.stdout:1145: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2023-01-15T14:32:51.044 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:32:51.044 INFO:tasks.rados.rados.0.smithi090.stdout:1145: writing smithi09025264-48 from 742975 to 1523215 tid 1 2023-01-15T14:32:51.044 INFO:tasks.rados.rados.0.smithi090.stdout:1145: writing smithi09025264-48 from 1953553 to 2380685 tid 2 2023-01-15T14:32:51.046 INFO:tasks.rados.rados.0.smithi090.stdout:1145: writing smithi09025264-48 from 2971713 to 3142975 tid 3 2023-01-15T14:32:51.046 INFO:tasks.rados.rados.0.smithi090.stdout:1146: read oid 20 snap 114 2023-01-15T14:32:51.046 INFO:tasks.rados.rados.0.smithi090.stdout:1146: expect (ObjNum 380 snap 103 seq_num 380) 2023-01-15T14:32:51.046 INFO:tasks.rados.rados.0.smithi090.stdout:1147: read oid 41 snap -1 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1147: expect (ObjNum 400 snap 106 seq_num 400) 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1148: write (excl) oid 28 current snap is 116 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1148: seq_num 426 ranges {759782=571842,2012035=687143} 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1148: writing smithi09025264-28 from 759782 to 1331624 tid 1 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1148: writing smithi09025264-28 from 2012035 to 2699178 tid 2 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1149: write oid 17 current snap is 116 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1149: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1149: writing smithi09025264-17 from 776589 to 1423680 tid 1 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1149: writing smithi09025264-17 from 2070517 to 2501318 tid 2 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1149: writing smithi09025264-17 from 3026559 to 3176589 tid 3 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1150: read oid 15 snap 112 2023-01-15T14:32:51.047 INFO:tasks.rados.rados.0.smithi090.stdout:1150: expect (ObjNum 361 snap 96 seq_num 361) 2023-01-15T14:32:51.048 INFO:tasks.rados.rados.0.smithi090.stdout:1151: read oid 31 snap -1 2023-01-15T14:32:51.048 INFO:tasks.rados.rados.0.smithi090.stdout:1151: expect (ObjNum 396 snap 106 seq_num 396) 2023-01-15T14:32:51.048 INFO:tasks.rados.rados.0.smithi090.stdout:1152: snap_remove snap 103 2023-01-15T14:32:51.048 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:51.033+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:51.049 INFO:tasks.rados.rados.0.smithi090.stdout:1152: done (7 left) 2023-01-15T14:32:51.049 INFO:tasks.rados.rados.0.smithi090.stdout:1153: delete oid 36 current snap is 116 2023-01-15T14:32:51.049 INFO:tasks.rados.rados.0.smithi090.stdout:1145: finishing write tid 1 to smithi09025264-48 2023-01-15T14:32:51.049 INFO:tasks.rados.rados.0.smithi090.stdout:1147: expect (ObjNum 400 snap 106 seq_num 400) 2023-01-15T14:32:51.139 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:51.136+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:51.190 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:51.190 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:51.191 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:32:51.202 INFO:tasks.rados.rados.0.smithi090.stdout:1151: expect (ObjNum 396 snap 106 seq_num 396) 2023-01-15T14:32:51.240 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:51.232+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:51.292 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:51.293 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:51.302 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:32:51.341 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:51.333+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:51.345 INFO:tasks.rados.rados.0.smithi090.stdout:1149: finishing write tid 1 to smithi09025264-17 2023-01-15T14:32:51.345 INFO:tasks.rados.rados.0.smithi090.stdout:1146: expect (ObjNum 380 snap 103 seq_num 380) 2023-01-15T14:32:51.352 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:32:51.354 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:32:51.364 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2023-01-15T14:32:51.043228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":184697642,"num_objects":87,"num_object_clones":44,"num_object_copies":261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":5,"num_read":897,"num_read_kb":822105,"num_write":793,"num_write_kb":240104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":309,"num_bytes_recovered":198268601,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12543,"ondisk_log_size":12543,"up":255,"acting":255,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":255,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":5586496,"kb_used_data":556683440,"kb_used_omap":46196,"kb_used_meta":0,"kb_avail":556609784,"statfs":{"total":575688990720,"available":569968418816,"internally_reserved":75423744,"allocated":570043842560,"data_stored":570043842560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":47305086,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":98,"commit_latency_ns":0,"apply_latency_ns":98000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22275777,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":46,"num_read_kb":44801,"num_write":35,"num_write_kb":17426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.120326"},"pg_stats":[{"pgid":"3.4b","version":"155'53","reported_seq":571,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.147850+0000","last_change":"2023-01-15T14:32:40.893260+0000","last_active":"2023-01-15T14:32:44.147850+0000","last_peered":"2023-01-15T14:32:44.147850+0000","last_clean":"2023-01-15T14:32:44.147850+0000","last_became_active":"2023-01-15T14:32:25.161408+0000","last_became_peered":"2023-01-15T14:32:25.161408+0000","last_unstale":"2023-01-15T14:32:44.147850+0000","last_undegraded":"2023-01-15T14:32:44.147850+0000","last_fullsized":"2023-01-15T14:32:44.147850+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:40.131","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046513570000000001,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.4a","version":"278'300","reported_seq":1232,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.056643+0000","last_change":"2023-01-15T14:32:40.972252+0000","last_active":"2023-01-15T14:32:47.056643+0000","last_peered":"2023-01-15T14:32:47.056643+0000","last_clean":"2023-01-15T14:32:47.056643+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:32:47.056643+0000","last_undegraded":"2023-01-15T14:32:47.056643+0000","last_fullsized":"2023-01-15T14:32:47.056643+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:33.185","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.104828955,"stat_sum":{"num_bytes":4272535,"num_objects":1,"num_object_clones":2,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":34131,"num_write":38,"num_write_kb":12530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.49","version":"110'149","reported_seq":952,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.059726+0000","last_change":"2023-01-15T14:32:40.869831+0000","last_active":"2023-01-15T14:32:47.059726+0000","last_peered":"2023-01-15T14:32:47.059726+0000","last_clean":"2023-01-15T14:32:47.059726+0000","last_became_active":"2023-01-15T14:32:28.259491+0000","last_became_peered":"2023-01-15T14:32:28.259491+0000","last_unstale":"2023-01-15T14:32:47.059726+0000","last_undegraded":"2023-01-15T14:32:47.059726+0000","last_fullsized":"2023-01-15T14:32:47.059726+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:31:44.640345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.640345+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.00486989,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.48","version":"179'204","reported_seq":1158,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.994863+0000","last_change":"2023-01-15T14:32:41.274216+0000","last_active":"2023-01-15T14:32:43.994863+0000","last_peered":"2023-01-15T14:32:43.994863+0000","last_clean":"2023-01-15T14:32:43.994863+0000","last_became_active":"2023-01-15T14:32:28.187902+0000","last_became_peered":"2023-01-15T14:32:28.187902+0000","last_unstale":"2023-01-15T14:32:43.994863+0000","last_undegraded":"2023-01-15T14:32:43.994863+0000","last_fullsized":"2023-01-15T14:32:43.994863+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.40903318599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.47","version":"245'112","reported_seq":949,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.964990+0000","last_change":"2023-01-15T14:32:40.974928+0000","last_active":"2023-01-15T14:32:46.964990+0000","last_peered":"2023-01-15T14:32:46.964990+0000","last_clean":"2023-01-15T14:32:46.964990+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:32:46.964990+0000","last_undegraded":"2023-01-15T14:32:46.964990+0000","last_fullsized":"2023-01-15T14:32:46.964990+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.107014655,"stat_sum":{"num_bytes":496762,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":5,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.46","version":"102'52","reported_seq":846,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.249419+0000","last_change":"2023-01-15T14:32:41.400284+0000","last_active":"2023-01-15T14:32:44.249419+0000","last_peered":"2023-01-15T14:32:44.249419+0000","last_clean":"2023-01-15T14:32:44.249419+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:32:44.249419+0000","last_undegraded":"2023-01-15T14:32:44.249419+0000","last_fullsized":"2023-01-15T14:32:44.249419+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:31:25.342830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:25.342830+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:32.011","scrub_duration":0.20886623600000001,"objects_trimmed":0,"snaptrim_duration":0.50981663600000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.45","version":"253'125","reported_seq":959,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.956642+0000","last_change":"2023-01-15T14:32:40.974206+0000","last_active":"2023-01-15T14:32:46.956642+0000","last_peered":"2023-01-15T14:32:46.956642+0000","last_clean":"2023-01-15T14:32:46.956642+0000","last_became_active":"2023-01-15T14:32:28.162971+0000","last_became_peered":"2023-01-15T14:32:28.162971+0000","last_unstale":"2023-01-15T14:32:46.956642+0000","last_undegraded":"2023-01-15T14:32:46.956642+0000","last_fullsized":"2023-01-15T14:32:46.956642+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:50.736","scrub_duration":0.206392353,"objects_trimmed":0,"snaptrim_duration":0.104457859,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.44","version":"245'268","reported_seq":1183,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.963739+0000","last_change":"2023-01-15T14:32:40.871635+0000","last_active":"2023-01-15T14:32:46.963739+0000","last_peered":"2023-01-15T14:32:46.963739+0000","last_clean":"2023-01-15T14:32:46.963739+0000","last_became_active":"2023-01-15T14:32:28.189610+0000","last_became_peered":"2023-01-15T14:32:28.189610+0000","last_unstale":"2023-01-15T14:32:46.963739+0000","last_undegraded":"2023-01-15T14:32:46.963739+0000","last_fullsized":"2023-01-15T14:32:46.963739+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.0063993339999999996,"stat_sum":{"num_bytes":2887217,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":30681,"num_write":15,"num_write_kb":5036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7931054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.43","version":"268'310","reported_seq":1295,"reported_epoch":280,"state":"active+clean","last_fresh":"2023-01-15T14:32:48.019859+0000","last_change":"2023-01-15T14:32:40.994022+0000","last_active":"2023-01-15T14:32:48.019859+0000","last_peered":"2023-01-15T14:32:48.019859+0000","last_clean":"2023-01-15T14:32:48.019859+0000","last_became_active":"2023-01-15T14:32:28.977084+0000","last_became_peered":"2023-01-15T14:32:28.977084+0000","last_unstale":"2023-01-15T14:32:48.019859+0000","last_undegraded":"2023-01-15T14:32:48.019859+0000","last_fullsized":"2023-01-15T14:32:48.019859+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:02.589","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.0060336340000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":845,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.097801+0000","last_change":"2023-01-15T14:32:41.276782+0000","last_active":"2023-01-15T14:32:44.097801+0000","last_peered":"2023-01-15T14:32:44.097801+0000","last_clean":"2023-01-15T14:32:44.097801+0000","last_became_active":"2023-01-15T14:32:25.163911+0000","last_became_peered":"2023-01-15T14:32:25.163911+0000","last_unstale":"2023-01-15T14:32:44.097801+0000","last_undegraded":"2023-01-15T14:32:44.097801+0000","last_fullsized":"2023-01-15T14:32:44.097801+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.41046229299999998,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.41","version":"155'121","reported_seq":925,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.060202+0000","last_change":"2023-01-15T14:32:41.704256+0000","last_active":"2023-01-15T14:32:47.060202+0000","last_peered":"2023-01-15T14:32:47.060202+0000","last_clean":"2023-01-15T14:32:47.060202+0000","last_became_active":"2023-01-15T14:31:56.927677+0000","last_became_peered":"2023-01-15T14:31:56.927677+0000","last_unstale":"2023-01-15T14:32:47.060202+0000","last_undegraded":"2023-01-15T14:32:47.060202+0000","last_fullsized":"2023-01-15T14:32:47.060202+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:56.100","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.0051031929999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.40","version":"274'279","reported_seq":1158,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.995268+0000","last_change":"2023-01-15T14:32:41.174822+0000","last_active":"2023-01-15T14:32:43.995268+0000","last_peered":"2023-01-15T14:32:43.995268+0000","last_clean":"2023-01-15T14:32:43.995268+0000","last_became_active":"2023-01-15T14:32:02.030960+0000","last_became_peered":"2023-01-15T14:32:02.030960+0000","last_unstale":"2023-01-15T14:32:43.995268+0000","last_undegraded":"2023-01-15T14:32:43.995268+0000","last_fullsized":"2023-01-15T14:32:43.995268+0000","mapping_epoch":234,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.30911992799999999,"stat_sum":{"num_bytes":3893850,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":8739,"num_write":15,"num_write_kb":4531,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":8588443,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.3f","version":"278'217","reported_seq":1127,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.058136+0000","last_change":"2023-01-15T14:32:41.075663+0000","last_active":"2023-01-15T14:32:47.058136+0000","last_peered":"2023-01-15T14:32:47.058136+0000","last_clean":"2023-01-15T14:32:47.058136+0000","last_became_active":"2023-01-15T14:32:02.943089+0000","last_became_peered":"2023-01-15T14:32:02.943089+0000","last_unstale":"2023-01-15T14:32:47.058136+0000","last_undegraded":"2023-01-15T14:32:47.058136+0000","last_fullsized":"2023-01-15T14:32:47.058136+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:09.339","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20571985700000001,"stat_sum":{"num_bytes":7467330,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":108167,"num_write":39,"num_write_kb":10479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":25819820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.3e","version":"278'55","reported_seq":653,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:45.005393+0000","last_change":"2023-01-15T14:32:41.482821+0000","last_active":"2023-01-15T14:32:45.005393+0000","last_peered":"2023-01-15T14:32:45.005393+0000","last_clean":"2023-01-15T14:32:45.005393+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:32:45.005393+0000","last_undegraded":"2023-01-15T14:32:45.005393+0000","last_fullsized":"2023-01-15T14:32:45.005393+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:42.278","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.59282380800000001,"stat_sum":{"num_bytes":5230461,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.3d","version":"274'221","reported_seq":1136,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.148227+0000","last_change":"2023-01-15T14:32:41.105159+0000","last_active":"2023-01-15T14:32:44.148227+0000","last_peered":"2023-01-15T14:32:44.148227+0000","last_clean":"2023-01-15T14:32:44.148227+0000","last_became_active":"2023-01-15T14:31:59.249241+0000","last_became_peered":"2023-01-15T14:31:59.249241+0000","last_unstale":"2023-01-15T14:32:44.148227+0000","last_undegraded":"2023-01-15T14:32:44.148227+0000","last_fullsized":"2023-01-15T14:32:44.148227+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:12.136","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.216456234,"stat_sum":{"num_bytes":6767538,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":10,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":168,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.995757+0000","last_change":"2023-01-15T14:32:41.379593+0000","last_active":"2023-01-15T14:32:43.995757+0000","last_peered":"2023-01-15T14:32:43.995757+0000","last_clean":"2023-01-15T14:32:43.995757+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:32:43.995757+0000","last_undegraded":"2023-01-15T14:32:43.995757+0000","last_fullsized":"2023-01-15T14:32:43.995757+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.41244196599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":579834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.3b","version":"221'60","reported_seq":824,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.963687+0000","last_change":"2023-01-15T14:32:44.563445+0000","last_active":"2023-01-15T14:32:46.963687+0000","last_peered":"2023-01-15T14:32:46.963687+0000","last_clean":"2023-01-15T14:32:46.963687+0000","last_became_active":"2023-01-15T14:31:54.237985+0000","last_became_peered":"2023-01-15T14:31:54.237985+0000","last_unstale":"2023-01-15T14:32:46.963687+0000","last_undegraded":"2023-01-15T14:32:46.963687+0000","last_fullsized":"2023-01-15T14:32:46.963687+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:52.139","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.003664732,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1165,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.147228+0000","last_change":"2023-01-15T14:32:41.505251+0000","last_active":"2023-01-15T14:32:44.147228+0000","last_peered":"2023-01-15T14:32:44.147228+0000","last_clean":"2023-01-15T14:32:44.147228+0000","last_became_active":"2023-01-15T14:31:57.025706+0000","last_became_peered":"2023-01-15T14:31:57.025706+0000","last_unstale":"2023-01-15T14:32:44.147228+0000","last_undegraded":"2023-01-15T14:32:44.147228+0000","last_fullsized":"2023-01-15T14:32:44.147228+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:30.639","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.52097700300000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.39","version":"278'294","reported_seq":1155,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.956379+0000","last_change":"2023-01-15T14:32:40.873094+0000","last_active":"2023-01-15T14:32:46.956379+0000","last_peered":"2023-01-15T14:32:46.956379+0000","last_clean":"2023-01-15T14:32:46.956379+0000","last_became_active":"2023-01-15T14:32:06.420117+0000","last_became_peered":"2023-01-15T14:32:06.420117+0000","last_unstale":"2023-01-15T14:32:46.956379+0000","last_undegraded":"2023-01-15T14:32:46.956379+0000","last_fullsized":"2023-01-15T14:32:46.956379+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'157","last_scrub_stamp":"2023-01-15T14:30:26.792736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:26.792736+0000","objects_scrubbed":4,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:12.259","scrub_duration":0.61998826900000004,"objects_trimmed":0,"snaptrim_duration":0.00456011,"stat_sum":{"num_bytes":12481388,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":51920,"num_write":33,"num_write_kb":17239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":31983837,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.38","version":"268'281","reported_seq":1321,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.195388+0000","last_change":"2023-01-15T14:32:41.175581+0000","last_active":"2023-01-15T14:32:44.195388+0000","last_peered":"2023-01-15T14:32:44.195388+0000","last_clean":"2023-01-15T14:32:44.195388+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:32:44.195388+0000","last_undegraded":"2023-01-15T14:32:44.195388+0000","last_fullsized":"2023-01-15T14:32:44.195388+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.30983241700000003,"stat_sum":{"num_bytes":2487502,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":18939,"num_write":22,"num_write_kb":9325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.37","version":"200'78","reported_seq":230,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.958513+0000","last_change":"2023-01-15T14:32:41.178292+0000","last_active":"2023-01-15T14:32:46.958513+0000","last_peered":"2023-01-15T14:32:46.958513+0000","last_clean":"2023-01-15T14:32:46.958513+0000","last_became_active":"2023-01-15T14:31:37.024453+0000","last_became_peered":"2023-01-15T14:31:37.024453+0000","last_unstale":"2023-01-15T14:32:46.958513+0000","last_undegraded":"2023-01-15T14:32:46.958513+0000","last_fullsized":"2023-01-15T14:32:46.958513+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:09.695","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.30811052500000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.36","version":"194'81","reported_seq":389,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.348826+0000","last_change":"2023-01-15T14:32:41.275905+0000","last_active":"2023-01-15T14:32:44.348826+0000","last_peered":"2023-01-15T14:32:44.348826+0000","last_clean":"2023-01-15T14:32:44.348826+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:32:44.348826+0000","last_undegraded":"2023-01-15T14:32:44.348826+0000","last_fullsized":"2023-01-15T14:32:44.348826+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.38641219599999999,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.35","version":"110'55","reported_seq":820,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:45.943960+0000","last_change":"2023-01-15T14:32:45.943865+0000","last_active":"2023-01-15T14:32:45.943960+0000","last_peered":"2023-01-15T14:32:45.943960+0000","last_clean":"2023-01-15T14:32:45.943960+0000","last_became_active":"2023-01-15T14:31:38.033824+0000","last_became_peered":"2023-01-15T14:31:38.033824+0000","last_unstale":"2023-01-15T14:32:45.943960+0000","last_undegraded":"2023-01-15T14:32:45.943960+0000","last_fullsized":"2023-01-15T14:32:45.943960+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.282936307,"objects_trimmed":0,"snaptrim_duration":0.50968611799999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":978,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.964090+0000","last_change":"2023-01-15T14:32:40.973364+0000","last_active":"2023-01-15T14:32:46.964090+0000","last_peered":"2023-01-15T14:32:46.964090+0000","last_clean":"2023-01-15T14:32:46.964090+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:32:46.964090+0000","last_undegraded":"2023-01-15T14:32:46.964090+0000","last_fullsized":"2023-01-15T14:32:46.964090+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"137'129","last_scrub_stamp":"2023-01-15T14:30:24.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:24.902895+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:48.327","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.107522328,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5618,"num_write":3,"num_write_kb":609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.33","version":"204'275","reported_seq":1062,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.997261+0000","last_change":"2023-01-15T14:32:41.276165+0000","last_active":"2023-01-15T14:32:43.997261+0000","last_peered":"2023-01-15T14:32:43.997261+0000","last_clean":"2023-01-15T14:32:43.997261+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:32:43.997261+0000","last_undegraded":"2023-01-15T14:32:43.997261+0000","last_fullsized":"2023-01-15T14:32:43.997261+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.408575943,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.32","version":"243'157","reported_seq":661,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.147278+0000","last_change":"2023-01-15T14:32:41.489486+0000","last_active":"2023-01-15T14:32:44.147278+0000","last_peered":"2023-01-15T14:32:44.147278+0000","last_clean":"2023-01-15T14:32:44.147278+0000","last_became_active":"2023-01-15T14:31:36.921952+0000","last_became_peered":"2023-01-15T14:31:36.921952+0000","last_unstale":"2023-01-15T14:32:44.147278+0000","last_undegraded":"2023-01-15T14:32:44.147278+0000","last_fullsized":"2023-01-15T14:32:44.147278+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:56.390","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.50486321499999998,"stat_sum":{"num_bytes":2403401,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.13","version":"278'329","reported_seq":1151,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.956635+0000","last_change":"2023-01-15T14:32:40.874869+0000","last_active":"2023-01-15T14:32:46.956635+0000","last_peered":"2023-01-15T14:32:46.956635+0000","last_clean":"2023-01-15T14:32:46.956635+0000","last_became_active":"2023-01-15T14:31:35.097414+0000","last_became_peered":"2023-01-15T14:31:35.097414+0000","last_unstale":"2023-01-15T14:32:46.956635+0000","last_undegraded":"2023-01-15T14:32:46.956635+0000","last_fullsized":"2023-01-15T14:32:46.956635+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.0059965160000000003,"stat_sum":{"num_bytes":7939725,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":32795,"num_write":13,"num_write_kb":12073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.12","version":"259'164","reported_seq":661,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.097982+0000","last_change":"2023-01-15T14:32:41.378738+0000","last_active":"2023-01-15T14:32:44.097982+0000","last_peered":"2023-01-15T14:32:44.097982+0000","last_clean":"2023-01-15T14:32:44.097982+0000","last_became_active":"2023-01-15T14:31:35.028565+0000","last_became_peered":"2023-01-15T14:31:35.028565+0000","last_unstale":"2023-01-15T14:32:44.097982+0000","last_undegraded":"2023-01-15T14:32:44.097982+0000","last_fullsized":"2023-01-15T14:32:44.097982+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:27.929","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.51196809899999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.11","version":"278'263","reported_seq":1184,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.957563+0000","last_change":"2023-01-15T14:32:40.875511+0000","last_active":"2023-01-15T14:32:46.957563+0000","last_peered":"2023-01-15T14:32:46.957563+0000","last_clean":"2023-01-15T14:32:46.957563+0000","last_became_active":"2023-01-15T14:31:35.026209+0000","last_became_peered":"2023-01-15T14:31:35.026209+0000","last_unstale":"2023-01-15T14:32:46.957563+0000","last_undegraded":"2023-01-15T14:32:46.957563+0000","last_fullsized":"2023-01-15T14:32:46.957563+0000","mapping_epoch":179,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:06.167","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.0063708870000000004,"stat_sum":{"num_bytes":10521446,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":45,"num_read_kb":47511,"num_write":31,"num_write_kb":14687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.10","version":"274'370","reported_seq":1272,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:45.504787+0000","last_change":"2023-01-15T14:32:42.085757+0000","last_active":"2023-01-15T14:32:45.504787+0000","last_peered":"2023-01-15T14:32:45.504787+0000","last_clean":"2023-01-15T14:32:45.504787+0000","last_became_active":"2023-01-15T14:31:35.135778+0000","last_became_peered":"2023-01-15T14:31:35.135778+0000","last_unstale":"2023-01-15T14:32:45.504787+0000","last_undegraded":"2023-01-15T14:32:45.504787+0000","last_fullsized":"2023-01-15T14:32:45.504787+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:50.443","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.10747293600000001,"stat_sum":{"num_bytes":4415782,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":61,"num_read_kb":48947,"num_write":77,"num_write_kb":23066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":13340457,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1041,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.066586+0000","last_change":"2023-01-15T14:32:41.080398+0000","last_active":"2023-01-15T14:32:47.066586+0000","last_peered":"2023-01-15T14:32:47.066586+0000","last_clean":"2023-01-15T14:32:47.066586+0000","last_became_active":"2023-01-15T14:31:35.032038+0000","last_became_peered":"2023-01-15T14:31:35.032038+0000","last_unstale":"2023-01-15T14:32:47.066586+0000","last_undegraded":"2023-01-15T14:32:47.066586+0000","last_fullsized":"2023-01-15T14:32:47.066586+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:31:22.549180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:22.549180+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:15.089","scrub_duration":0.50470070300000003,"objects_trimmed":0,"snaptrim_duration":0.11006817200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":842,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.965006+0000","last_change":"2023-01-15T14:32:40.975398+0000","last_active":"2023-01-15T14:32:46.965006+0000","last_peered":"2023-01-15T14:32:46.965006+0000","last_clean":"2023-01-15T14:32:46.965006+0000","last_became_active":"2023-01-15T14:31:35.237663+0000","last_became_peered":"2023-01-15T14:31:35.237663+0000","last_unstale":"2023-01-15T14:32:46.965006+0000","last_undegraded":"2023-01-15T14:32:46.965006+0000","last_fullsized":"2023-01-15T14:32:46.965006+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:16.384","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.107557877,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.d","version":"278'210","reported_seq":1088,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.056820+0000","last_change":"2023-01-15T14:32:41.076700+0000","last_active":"2023-01-15T14:32:47.056820+0000","last_peered":"2023-01-15T14:32:47.056820+0000","last_clean":"2023-01-15T14:32:47.056820+0000","last_became_active":"2023-01-15T14:31:21.132225+0000","last_became_peered":"2023-01-15T14:31:21.132225+0000","last_unstale":"2023-01-15T14:32:47.056820+0000","last_undegraded":"2023-01-15T14:32:47.056820+0000","last_fullsized":"2023-01-15T14:32:47.056820+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":210,"log_dups_size":0,"ondisk_log_size":210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:32:48.284","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.20665525800000001,"stat_sum":{"num_bytes":7709183,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":3453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.c","version":"260'66","reported_seq":851,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.997506+0000","last_change":"2023-01-15T14:32:43.604003+0000","last_active":"2023-01-15T14:32:43.997506+0000","last_peered":"2023-01-15T14:32:43.997506+0000","last_clean":"2023-01-15T14:32:43.997506+0000","last_became_active":"2023-01-15T14:31:21.227945+0000","last_became_peered":"2023-01-15T14:31:21.227945+0000","last_unstale":"2023-01-15T14:32:43.997506+0000","last_undegraded":"2023-01-15T14:32:43.997506+0000","last_fullsized":"2023-01-15T14:32:43.997506+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:02.585","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.0011215800000000001,"stat_sum":{"num_bytes":2386660,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":824,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.999066+0000","last_change":"2023-01-15T14:32:42.179828+0000","last_active":"2023-01-15T14:32:43.999066+0000","last_peered":"2023-01-15T14:32:43.999066+0000","last_clean":"2023-01-15T14:32:43.999066+0000","last_became_active":"2023-01-15T14:32:22.246088+0000","last_became_peered":"2023-01-15T14:32:22.246088+0000","last_unstale":"2023-01-15T14:32:43.999066+0000","last_undegraded":"2023-01-15T14:32:43.999066+0000","last_fullsized":"2023-01-15T14:32:43.999066+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:06.346","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.101489038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.a","version":"252'279","reported_seq":1146,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.964341+0000","last_change":"2023-01-15T14:32:40.978176+0000","last_active":"2023-01-15T14:32:46.964341+0000","last_peered":"2023-01-15T14:32:46.964341+0000","last_clean":"2023-01-15T14:32:46.964341+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:32:46.964341+0000","last_undegraded":"2023-01-15T14:32:46.964341+0000","last_fullsized":"2023-01-15T14:32:46.964341+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:25.185","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012484599000000001,"stat_sum":{"num_bytes":2130792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21399,"num_write":27,"num_write_kb":9854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9447726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":933,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.061028+0000","last_change":"2023-01-15T14:32:40.872918+0000","last_active":"2023-01-15T14:32:47.061028+0000","last_peered":"2023-01-15T14:32:47.061028+0000","last_clean":"2023-01-15T14:32:47.061028+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:32:47.061028+0000","last_undegraded":"2023-01-15T14:32:47.061028+0000","last_fullsized":"2023-01-15T14:32:47.061028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:31:44.640345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.640345+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.0060332019999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.8","version":"179'204","reported_seq":1139,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:45.205498+0000","last_change":"2023-01-15T14:32:42.245885+0000","last_active":"2023-01-15T14:32:45.205498+0000","last_peered":"2023-01-15T14:32:45.205498+0000","last_clean":"2023-01-15T14:32:45.205498+0000","last_became_active":"2023-01-15T14:32:22.142296+0000","last_became_peered":"2023-01-15T14:32:22.142296+0000","last_unstale":"2023-01-15T14:32:45.205498+0000","last_undegraded":"2023-01-15T14:32:45.205498+0000","last_fullsized":"2023-01-15T14:32:45.205498+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.16799726800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":413,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.147338+0000","last_change":"2023-01-15T14:31:03.091584+0000","last_active":"2023-01-15T14:32:44.147338+0000","last_peered":"2023-01-15T14:32:44.147338+0000","last_clean":"2023-01-15T14:32:44.147338+0000","last_became_active":"2023-01-15T14:31:03.091071+0000","last_became_peered":"2023-01-15T14:31:03.091071+0000","last_unstale":"2023-01-15T14:32:44.147338+0000","last_undegraded":"2023-01-15T14:32:44.147338+0000","last_fullsized":"2023-01-15T14:32:44.147338+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:29:45.876183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:29:45.876183+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:32:47.144","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":252,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.998630+0000","last_change":"2023-01-15T14:31:38.018850+0000","last_active":"2023-01-15T14:32:43.998630+0000","last_peered":"2023-01-15T14:32:43.998630+0000","last_clean":"2023-01-15T14:32:43.998630+0000","last_became_active":"2023-01-15T14:31:38.018145+0000","last_became_peered":"2023-01-15T14:31:38.018145+0000","last_unstale":"2023-01-15T14:32:43.998630+0000","last_undegraded":"2023-01-15T14:32:43.998630+0000","last_fullsized":"2023-01-15T14:32:43.998630+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":830,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.964023+0000","last_change":"2023-01-15T14:32:40.977865+0000","last_active":"2023-01-15T14:32:46.964023+0000","last_peered":"2023-01-15T14:32:46.964023+0000","last_clean":"2023-01-15T14:32:46.964023+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:32:46.964023+0000","last_undegraded":"2023-01-15T14:32:46.964023+0000","last_fullsized":"2023-01-15T14:32:46.964023+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:35.297","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.012479229,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":314,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:45.245350+0000","last_change":"2023-01-15T14:31:11.136609+0000","last_active":"2023-01-15T14:32:45.245350+0000","last_peered":"2023-01-15T14:32:45.245350+0000","last_clean":"2023-01-15T14:32:45.245350+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:32:45.245350+0000","last_undegraded":"2023-01-15T14:32:45.245350+0000","last_fullsized":"2023-01-15T14:32:45.245350+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:11.136490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.136490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:32:47.246","scrub_duration":0.39969354600000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":703,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.059574+0000","last_change":"2023-01-15T14:32:41.122001+0000","last_active":"2023-01-15T14:32:47.059574+0000","last_peered":"2023-01-15T14:32:47.059574+0000","last_clean":"2023-01-15T14:32:47.059574+0000","last_became_active":"2023-01-15T14:32:40.967729+0000","last_became_peered":"2023-01-15T14:32:40.967729+0000","last_unstale":"2023-01-15T14:32:47.059574+0000","last_undegraded":"2023-01-15T14:32:47.059574+0000","last_fullsized":"2023-01-15T14:32:47.059574+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:43.783","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.14819659600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":81,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.995406+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:32:43.995406+0000","last_peered":"2023-01-15T14:32:43.995406+0000","last_clean":"2023-01-15T14:32:43.995406+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:32:43.995406+0000","last_undegraded":"2023-01-15T14:32:43.995406+0000","last_fullsized":"2023-01-15T14:32:43.995406+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1149,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.147804+0000","last_change":"2023-01-15T14:32:41.401105+0000","last_active":"2023-01-15T14:32:44.147804+0000","last_peered":"2023-01-15T14:32:44.147804+0000","last_clean":"2023-01-15T14:32:44.147804+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:32:44.147804+0000","last_undegraded":"2023-01-15T14:32:44.147804+0000","last_fullsized":"2023-01-15T14:32:44.147804+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:31.185","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.42909233400000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":135,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.148074+0000","last_change":"2023-01-15T14:31:13.764052+0000","last_active":"2023-01-15T14:32:44.148074+0000","last_peered":"2023-01-15T14:32:44.148074+0000","last_clean":"2023-01-15T14:32:44.148074+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:32:44.148074+0000","last_undegraded":"2023-01-15T14:32:44.148074+0000","last_fullsized":"2023-01-15T14:32:44.148074+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:13.763945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:13.763945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58613919699999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"274'309","reported_seq":1260,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.165241+0000","last_change":"2023-01-15T14:32:40.873739+0000","last_active":"2023-01-15T14:32:47.165241+0000","last_peered":"2023-01-15T14:32:47.165241+0000","last_clean":"2023-01-15T14:32:47.165241+0000","last_became_active":"2023-01-15T14:32:22.247115+0000","last_became_peered":"2023-01-15T14:32:22.247115+0000","last_unstale":"2023-01-15T14:32:47.165241+0000","last_undegraded":"2023-01-15T14:32:47.165241+0000","last_fullsized":"2023-01-15T14:32:47.165241+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'251","last_scrub_stamp":"2023-01-15T14:31:15.363400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.363400+0000","objects_scrubbed":3,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.208136242,"objects_trimmed":0,"snaptrim_duration":0.0068437819999999996,"stat_sum":{"num_bytes":2840449,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":22495,"num_write":15,"num_write_kb":5392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.5","version":"278'132","reported_seq":975,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:45.956836+0000","last_change":"2023-01-15T14:32:42.181830+0000","last_active":"2023-01-15T14:32:45.956836+0000","last_peered":"2023-01-15T14:32:45.956836+0000","last_clean":"2023-01-15T14:32:45.956836+0000","last_became_active":"2023-01-15T14:32:21.948278+0000","last_became_peered":"2023-01-15T14:32:21.948278+0000","last_unstale":"2023-01-15T14:32:45.956836+0000","last_undegraded":"2023-01-15T14:32:45.956836+0000","last_fullsized":"2023-01-15T14:32:45.956836+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:38.550","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.108803759,"stat_sum":{"num_bytes":4700816,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":13729,"num_write":5,"num_write_kb":1162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.148478+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:32:44.148478+0000","last_peered":"2023-01-15T14:32:44.148478+0000","last_clean":"2023-01-15T14:32:44.148478+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:32:44.148478+0000","last_undegraded":"2023-01-15T14:32:44.148478+0000","last_fullsized":"2023-01-15T14:32:44.148478+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:37.670","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"278'293","reported_seq":1228,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.168321+0000","last_change":"2023-01-15T14:32:41.502400+0000","last_active":"2023-01-15T14:32:46.168321+0000","last_peered":"2023-01-15T14:32:46.168321+0000","last_clean":"2023-01-15T14:32:46.168321+0000","last_became_active":"2023-01-15T14:32:22.247573+0000","last_became_peered":"2023-01-15T14:32:22.247573+0000","last_unstale":"2023-01-15T14:32:46.168321+0000","last_undegraded":"2023-01-15T14:32:46.168321+0000","last_fullsized":"2023-01-15T14:32:46.168321+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.51273273900000005,"stat_sum":{"num_bytes":5135837,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":29,"num_read_kb":30683,"num_write":27,"num_write_kb":9142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7931053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":316,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:45.246043+0000","last_change":"2023-01-15T14:32:34.578760+0000","last_active":"2023-01-15T14:32:45.246043+0000","last_peered":"2023-01-15T14:32:45.246043+0000","last_clean":"2023-01-15T14:32:45.246043+0000","last_became_active":"2023-01-15T14:31:03.088392+0000","last_became_peered":"2023-01-15T14:31:03.088392+0000","last_unstale":"2023-01-15T14:32:45.246043+0000","last_undegraded":"2023-01-15T14:32:45.246043+0000","last_fullsized":"2023-01-15T14:32:45.246043+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:38.541","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"245'112","reported_seq":941,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.066380+0000","last_change":"2023-01-15T14:32:40.976307+0000","last_active":"2023-01-15T14:32:47.066380+0000","last_peered":"2023-01-15T14:32:47.066380+0000","last_clean":"2023-01-15T14:32:47.066380+0000","last_became_active":"2023-01-15T14:32:22.146367+0000","last_became_peered":"2023-01-15T14:32:22.146367+0000","last_unstale":"2023-01-15T14:32:47.066380+0000","last_undegraded":"2023-01-15T14:32:47.066380+0000","last_fullsized":"2023-01-15T14:32:47.066380+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:03.357","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.10669975499999999,"stat_sum":{"num_bytes":3479082,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":125,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.249547+0000","last_change":"2023-01-15T14:30:51.035825+0000","last_active":"2023-01-15T14:32:44.249547+0000","last_peered":"2023-01-15T14:32:44.249547+0000","last_clean":"2023-01-15T14:32:44.249547+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:32:44.249547+0000","last_undegraded":"2023-01-15T14:32:44.249547+0000","last_fullsized":"2023-01-15T14:32:44.249547+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:29:42.343239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:29:42.343239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:32:45.110","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":846,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.248787+0000","last_change":"2023-01-15T14:32:41.392846+0000","last_active":"2023-01-15T14:32:44.248787+0000","last_peered":"2023-01-15T14:32:44.248787+0000","last_clean":"2023-01-15T14:32:44.248787+0000","last_became_active":"2023-01-15T14:32:22.247696+0000","last_became_peered":"2023-01-15T14:32:22.247696+0000","last_unstale":"2023-01-15T14:32:44.248787+0000","last_undegraded":"2023-01-15T14:32:44.248787+0000","last_fullsized":"2023-01-15T14:32:44.248787+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:09.533","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.502985299,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":261,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.996148+0000","last_change":"2023-01-15T14:31:38.112378+0000","last_active":"2023-01-15T14:32:43.996148+0000","last_peered":"2023-01-15T14:32:43.996148+0000","last_clean":"2023-01-15T14:32:43.996148+0000","last_became_active":"2023-01-15T14:31:38.111590+0000","last_became_peered":"2023-01-15T14:31:38.111590+0000","last_unstale":"2023-01-15T14:32:43.996148+0000","last_undegraded":"2023-01-15T14:32:43.996148+0000","last_fullsized":"2023-01-15T14:32:43.996148+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:51.156","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"245'148","reported_seq":985,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.066020+0000","last_change":"2023-01-15T14:32:40.977554+0000","last_active":"2023-01-15T14:32:47.066020+0000","last_peered":"2023-01-15T14:32:47.066020+0000","last_clean":"2023-01-15T14:32:47.066020+0000","last_became_active":"2023-01-15T14:31:35.134196+0000","last_became_peered":"2023-01-15T14:31:35.134196+0000","last_unstale":"2023-01-15T14:32:47.066020+0000","last_undegraded":"2023-01-15T14:32:47.066020+0000","last_fullsized":"2023-01-15T14:32:47.066020+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:32:19.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:19.303243+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:46.971","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.10646768700000001,"stat_sum":{"num_bytes":2453855,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7195,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":811,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.996497+0000","last_change":"2023-01-15T14:32:42.183049+0000","last_active":"2023-01-15T14:32:43.996497+0000","last_peered":"2023-01-15T14:32:43.996497+0000","last_clean":"2023-01-15T14:32:43.996497+0000","last_became_active":"2023-01-15T14:31:34.951721+0000","last_became_peered":"2023-01-15T14:31:34.951721+0000","last_unstale":"2023-01-15T14:32:43.996497+0000","last_undegraded":"2023-01-15T14:32:43.996497+0000","last_fullsized":"2023-01-15T14:32:43.996497+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:41.128","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.204621519,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.16","version":"259'113","reported_seq":426,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.249214+0000","last_change":"2023-01-15T14:32:41.397876+0000","last_active":"2023-01-15T14:32:44.249214+0000","last_peered":"2023-01-15T14:32:44.249214+0000","last_clean":"2023-01-15T14:32:44.249214+0000","last_became_active":"2023-01-15T14:31:35.032807+0000","last_became_peered":"2023-01-15T14:31:35.032807+0000","last_unstale":"2023-01-15T14:32:44.249214+0000","last_undegraded":"2023-01-15T14:32:44.249214+0000","last_fullsized":"2023-01-15T14:32:44.249214+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.50767293899999999,"stat_sum":{"num_bytes":2353046,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":13662,"num_write":10,"num_write_kb":2541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":861,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.058402+0000","last_change":"2023-01-15T14:32:41.279396+0000","last_active":"2023-01-15T14:32:46.058402+0000","last_peered":"2023-01-15T14:32:46.058402+0000","last_clean":"2023-01-15T14:32:46.058402+0000","last_became_active":"2023-01-15T14:31:34.932309+0000","last_became_peered":"2023-01-15T14:31:34.932309+0000","last_unstale":"2023-01-15T14:32:46.058402+0000","last_undegraded":"2023-01-15T14:32:46.058402+0000","last_fullsized":"2023-01-15T14:32:46.058402+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:43.141","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.40918083900000002,"stat_sum":{"num_bytes":3025326,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.18","version":"278'307","reported_seq":1293,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:45.940629+0000","last_change":"2023-01-15T14:32:42.279048+0000","last_active":"2023-01-15T14:32:45.940629+0000","last_peered":"2023-01-15T14:32:45.940629+0000","last_clean":"2023-01-15T14:32:45.940629+0000","last_became_active":"2023-01-15T14:31:52.330386+0000","last_became_peered":"2023-01-15T14:31:52.330386+0000","last_unstale":"2023-01-15T14:32:45.940629+0000","last_undegraded":"2023-01-15T14:32:45.940629+0000","last_fullsized":"2023-01-15T14:32:45.940629+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.200358801,"stat_sum":{"num_bytes":11324650,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":49,"num_read_kb":55265,"num_write":44,"num_write_kb":14485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":11033036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.19","version":"278'240","reported_seq":1109,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:45.942756+0000","last_change":"2023-01-15T14:32:41.109191+0000","last_active":"2023-01-15T14:32:45.942756+0000","last_peered":"2023-01-15T14:32:45.942756+0000","last_clean":"2023-01-15T14:32:45.942756+0000","last_became_active":"2023-01-15T14:31:51.927440+0000","last_became_peered":"2023-01-15T14:31:51.927440+0000","last_unstale":"2023-01-15T14:32:45.942756+0000","last_undegraded":"2023-01-15T14:32:45.942756+0000","last_fullsized":"2023-01-15T14:32:45.942756+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:07.960","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.21979875099999999,"stat_sum":{"num_bytes":3999223,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":7098,"num_write":9,"num_write_kb":2362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.1a","version":"278'306","reported_seq":1227,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.963677+0000","last_change":"2023-01-15T14:32:40.979294+0000","last_active":"2023-01-15T14:32:46.963677+0000","last_peered":"2023-01-15T14:32:46.963677+0000","last_clean":"2023-01-15T14:32:46.963677+0000","last_became_active":"2023-01-15T14:31:52.124026+0000","last_became_peered":"2023-01-15T14:31:52.124026+0000","last_unstale":"2023-01-15T14:32:46.963677+0000","last_undegraded":"2023-01-15T14:32:46.963677+0000","last_fullsized":"2023-01-15T14:32:46.963677+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:49.729","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.012730775999999999,"stat_sum":{"num_bytes":6105307,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":9,"num_write_kb":1196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.1b","version":"271'78","reported_seq":824,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.064694+0000","last_change":"2023-01-15T14:32:41.018171+0000","last_active":"2023-01-15T14:32:47.064694+0000","last_peered":"2023-01-15T14:32:47.064694+0000","last_clean":"2023-01-15T14:32:47.064694+0000","last_became_active":"2023-01-15T14:31:52.321399+0000","last_became_peered":"2023-01-15T14:31:52.321399+0000","last_unstale":"2023-01-15T14:32:47.064694+0000","last_undegraded":"2023-01-15T14:32:47.064694+0000","last_fullsized":"2023-01-15T14:32:47.064694+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.051823785999999997,"stat_sum":{"num_bytes":2219866,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":21019,"num_write":6,"num_write_kb":2022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":350,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.065366+0000","last_change":"2023-01-15T14:32:40.976762+0000","last_active":"2023-01-15T14:32:47.065366+0000","last_peered":"2023-01-15T14:32:47.065366+0000","last_clean":"2023-01-15T14:32:47.065366+0000","last_became_active":"2023-01-15T14:31:51.926652+0000","last_became_peered":"2023-01-15T14:31:51.926652+0000","last_unstale":"2023-01-15T14:32:47.065366+0000","last_undegraded":"2023-01-15T14:32:47.065366+0000","last_fullsized":"2023-01-15T14:32:47.065366+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:35.143","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.10589860500000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1087,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.964618+0000","last_change":"2023-01-15T14:32:40.872384+0000","last_active":"2023-01-15T14:32:46.964618+0000","last_peered":"2023-01-15T14:32:46.964618+0000","last_clean":"2023-01-15T14:32:46.964618+0000","last_became_active":"2023-01-15T14:31:52.025227+0000","last_became_peered":"2023-01-15T14:31:52.025227+0000","last_unstale":"2023-01-15T14:32:46.964618+0000","last_undegraded":"2023-01-15T14:32:46.964618+0000","last_fullsized":"2023-01-15T14:32:46.964618+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:32.121","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061298740000000004,"stat_sum":{"num_bytes":5739855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":47622,"num_write":25,"num_write_kb":10630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6928644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.1e","version":"215'39","reported_seq":789,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.996868+0000","last_change":"2023-01-15T14:32:41.277957+0000","last_active":"2023-01-15T14:32:43.996868+0000","last_peered":"2023-01-15T14:32:43.996868+0000","last_clean":"2023-01-15T14:32:43.996868+0000","last_became_active":"2023-01-15T14:31:52.329815+0000","last_became_peered":"2023-01-15T14:31:52.329815+0000","last_unstale":"2023-01-15T14:32:43.996868+0000","last_undegraded":"2023-01-15T14:32:43.996868+0000","last_fullsized":"2023-01-15T14:32:43.996868+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:31:44.906253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.906253+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55943661099999997,"objects_trimmed":0,"snaptrim_duration":0.40922820100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.1f","version":"278'240","reported_seq":1186,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.066135+0000","last_change":"2023-01-15T14:32:40.975891+0000","last_active":"2023-01-15T14:32:47.066135+0000","last_peered":"2023-01-15T14:32:47.066135+0000","last_clean":"2023-01-15T14:32:47.066135+0000","last_became_active":"2023-01-15T14:31:52.124821+0000","last_became_peered":"2023-01-15T14:31:52.124821+0000","last_unstale":"2023-01-15T14:32:47.066135+0000","last_undegraded":"2023-01-15T14:32:47.066135+0000","last_fullsized":"2023-01-15T14:32:47.066135+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:55.650","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.106626101,"stat_sum":{"num_bytes":9428859,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":26683,"num_write":1,"num_write_kb":3398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.20","version":"245'258","reported_seq":1191,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.956468+0000","last_change":"2023-01-15T14:32:40.871316+0000","last_active":"2023-01-15T14:32:46.956468+0000","last_peered":"2023-01-15T14:32:46.956468+0000","last_clean":"2023-01-15T14:32:46.956468+0000","last_became_active":"2023-01-15T14:31:07.906191+0000","last_became_peered":"2023-01-15T14:31:07.906191+0000","last_unstale":"2023-01-15T14:32:46.956468+0000","last_undegraded":"2023-01-15T14:32:46.956468+0000","last_fullsized":"2023-01-15T14:32:46.956468+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.0033047879999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":21726,"num_write":8,"num_write_kb":2065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8877936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.21","version":"244'130","reported_seq":836,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.956999+0000","last_change":"2023-01-15T14:32:40.873693+0000","last_active":"2023-01-15T14:32:46.956999+0000","last_peered":"2023-01-15T14:32:46.956999+0000","last_clean":"2023-01-15T14:32:46.956999+0000","last_became_active":"2023-01-15T14:31:38.233481+0000","last_became_peered":"2023-01-15T14:31:38.233481+0000","last_unstale":"2023-01-15T14:32:46.956999+0000","last_undegraded":"2023-01-15T14:32:46.956999+0000","last_fullsized":"2023-01-15T14:32:46.956999+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"168'124","last_scrub_stamp":"2023-01-15T14:31:11.414546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.414546+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:48.531","scrub_duration":0.28528039199999999,"objects_trimmed":0,"snaptrim_duration":0.0048887089999999998,"stat_sum":{"num_bytes":2067327,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10100,"num_write":5,"num_write_kb":2711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3798382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.22","version":"274'155","reported_seq":1122,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.997478+0000","last_change":"2023-01-15T14:32:41.978822+0000","last_active":"2023-01-15T14:32:43.997478+0000","last_peered":"2023-01-15T14:32:43.997478+0000","last_clean":"2023-01-15T14:32:43.997478+0000","last_became_active":"2023-01-15T14:31:08.084483+0000","last_became_peered":"2023-01-15T14:31:08.084483+0000","last_unstale":"2023-01-15T14:32:43.997478+0000","last_undegraded":"2023-01-15T14:32:43.997478+0000","last_fullsized":"2023-01-15T14:32:43.997478+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":1,"snaptrim_duration":1.1129136310000001,"stat_sum":{"num_bytes":5329847,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":16,"num_write_kb":7631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.23","version":"149'233","reported_seq":1002,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.997676+0000","last_change":"2023-01-15T14:32:41.277104+0000","last_active":"2023-01-15T14:32:43.997676+0000","last_peered":"2023-01-15T14:32:43.997676+0000","last_clean":"2023-01-15T14:32:43.997676+0000","last_became_active":"2023-01-15T14:31:06.029716+0000","last_became_peered":"2023-01-15T14:31:06.029716+0000","last_unstale":"2023-01-15T14:32:43.997676+0000","last_undegraded":"2023-01-15T14:32:43.997676+0000","last_fullsized":"2023-01-15T14:32:43.997676+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:48.395","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40930412999999999,"stat_sum":{"num_bytes":7604272,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":47,"num_read_kb":43637,"num_write":37,"num_write_kb":13160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2469494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.24","version":"278'235","reported_seq":1189,"reported_epoch":278,"state":"active+clean","last_fresh":"2023-01-15T14:32:45.956774+0000","last_change":"2023-01-15T14:32:41.274980+0000","last_active":"2023-01-15T14:32:45.956774+0000","last_peered":"2023-01-15T14:32:45.956774+0000","last_clean":"2023-01-15T14:32:45.956774+0000","last_became_active":"2023-01-15T14:31:26.168777+0000","last_became_peered":"2023-01-15T14:31:26.168777+0000","last_unstale":"2023-01-15T14:32:45.956774+0000","last_undegraded":"2023-01-15T14:32:45.956774+0000","last_fullsized":"2023-01-15T14:32:45.956774+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:35.100","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.40751849499999998,"stat_sum":{"num_bytes":4122560,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9658,"num_write":8,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.25","version":"243'93","reported_seq":915,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.148308+0000","last_change":"2023-01-15T14:32:41.101041+0000","last_active":"2023-01-15T14:32:44.148308+0000","last_peered":"2023-01-15T14:32:44.148308+0000","last_clean":"2023-01-15T14:32:44.148308+0000","last_became_active":"2023-01-15T14:31:22.000386+0000","last_became_peered":"2023-01-15T14:31:22.000386+0000","last_unstale":"2023-01-15T14:32:44.148308+0000","last_undegraded":"2023-01-15T14:32:44.148308+0000","last_fullsized":"2023-01-15T14:32:44.148308+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:51.596","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.21167341100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6061,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.26","version":"102'52","reported_seq":277,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.957202+0000","last_change":"2023-01-15T14:32:41.072854+0000","last_active":"2023-01-15T14:32:46.957202+0000","last_peered":"2023-01-15T14:32:46.957202+0000","last_clean":"2023-01-15T14:32:46.957202+0000","last_became_active":"2023-01-15T14:31:21.901622+0000","last_became_peered":"2023-01-15T14:31:21.901622+0000","last_unstale":"2023-01-15T14:32:46.957202+0000","last_undegraded":"2023-01-15T14:32:46.957202+0000","last_fullsized":"2023-01-15T14:32:46.957202+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:08.119","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.20333895399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":886,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.996052+0000","last_change":"2023-01-15T14:32:41.278761+0000","last_active":"2023-01-15T14:32:43.996052+0000","last_peered":"2023-01-15T14:32:43.996052+0000","last_clean":"2023-01-15T14:32:43.996052+0000","last_became_active":"2023-01-15T14:31:22.091646+0000","last_became_peered":"2023-01-15T14:31:22.091646+0000","last_unstale":"2023-01-15T14:32:43.996052+0000","last_undegraded":"2023-01-15T14:32:43.996052+0000","last_fullsized":"2023-01-15T14:32:43.996052+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:41.261","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.40948396599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.28","version":"264'229","reported_seq":1185,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:43.997539+0000","last_change":"2023-01-15T14:32:42.347139+0000","last_active":"2023-01-15T14:32:43.997539+0000","last_peered":"2023-01-15T14:32:43.997539+0000","last_clean":"2023-01-15T14:32:43.997539+0000","last_became_active":"2023-01-15T14:31:26.054878+0000","last_became_peered":"2023-01-15T14:31:26.054878+0000","last_unstale":"2023-01-15T14:32:43.997539+0000","last_undegraded":"2023-01-15T14:32:43.997539+0000","last_fullsized":"2023-01-15T14:32:43.997539+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:01.279","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.36957786100000001,"stat_sum":{"num_bytes":2504309,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":17937,"num_write":9,"num_write_kb":5085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":12193528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.29","version":"110'149","reported_seq":697,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.095542+0000","last_change":"2023-01-15T14:32:42.180717+0000","last_active":"2023-01-15T14:32:44.095542+0000","last_peered":"2023-01-15T14:32:44.095542+0000","last_clean":"2023-01-15T14:32:44.095542+0000","last_became_active":"2023-01-15T14:31:25.136376+0000","last_became_peered":"2023-01-15T14:31:25.136376+0000","last_unstale":"2023-01-15T14:32:44.095542+0000","last_undegraded":"2023-01-15T14:32:44.095542+0000","last_fullsized":"2023-01-15T14:32:44.095542+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:30:08.681869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:08.681869+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:14.346","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030698370000000002,"stat_sum":{"num_bytes":2857174,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":24874,"num_write":16,"num_write_kb":6171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1062,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.956492+0000","last_change":"2023-01-15T14:32:40.971188+0000","last_active":"2023-01-15T14:32:46.956492+0000","last_peered":"2023-01-15T14:32:46.956492+0000","last_clean":"2023-01-15T14:32:46.956492+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:32:46.956492+0000","last_undegraded":"2023-01-15T14:32:46.956492+0000","last_fullsized":"2023-01-15T14:32:46.956492+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:32:49.295","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.101931944,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":767,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.148598+0000","last_change":"2023-01-15T14:32:41.283499+0000","last_active":"2023-01-15T14:32:44.148598+0000","last_peered":"2023-01-15T14:32:44.148598+0000","last_clean":"2023-01-15T14:32:44.148598+0000","last_became_active":"2023-01-15T14:31:21.896494+0000","last_became_peered":"2023-01-15T14:31:21.896494+0000","last_unstale":"2023-01-15T14:32:44.148598+0000","last_undegraded":"2023-01-15T14:32:44.148598+0000","last_fullsized":"2023-01-15T14:32:44.148598+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:30.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.975433+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040286109999999997,"objects_trimmed":0,"snaptrim_duration":0.39389282199999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":257,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.064544+0000","last_change":"2023-01-15T14:32:40.978717+0000","last_active":"2023-01-15T14:32:47.064544+0000","last_peered":"2023-01-15T14:32:47.064544+0000","last_clean":"2023-01-15T14:32:47.064544+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:32:47.064544+0000","last_undegraded":"2023-01-15T14:32:47.064544+0000","last_fullsized":"2023-01-15T14:32:47.064544+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:32:47.318","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.012374417,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1035,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.957117+0000","last_change":"2023-01-15T14:32:41.074286+0000","last_active":"2023-01-15T14:32:46.957117+0000","last_peered":"2023-01-15T14:32:46.957117+0000","last_clean":"2023-01-15T14:32:46.957117+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:32:46.957117+0000","last_undegraded":"2023-01-15T14:32:46.957117+0000","last_fullsized":"2023-01-15T14:32:46.957117+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.20428043000000001,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.2e","version":"139'38","reported_seq":712,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.248583+0000","last_change":"2023-01-15T14:32:41.287515+0000","last_active":"2023-01-15T14:32:44.248583+0000","last_peered":"2023-01-15T14:32:44.248583+0000","last_clean":"2023-01-15T14:32:44.248583+0000","last_became_active":"2023-01-15T14:31:38.116756+0000","last_became_peered":"2023-01-15T14:31:38.116756+0000","last_unstale":"2023-01-15T14:32:44.248583+0000","last_undegraded":"2023-01-15T14:32:44.248583+0000","last_fullsized":"2023-01-15T14:32:44.248583+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:51.292","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.39784351099999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":910,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:46.958008+0000","last_change":"2023-01-15T14:32:41.077663+0000","last_active":"2023-01-15T14:32:46.958008+0000","last_peered":"2023-01-15T14:32:46.958008+0000","last_clean":"2023-01-15T14:32:46.958008+0000","last_became_active":"2023-01-15T14:31:40.024531+0000","last_became_peered":"2023-01-15T14:31:40.024531+0000","last_unstale":"2023-01-15T14:32:46.958008+0000","last_undegraded":"2023-01-15T14:32:46.958008+0000","last_fullsized":"2023-01-15T14:32:46.958008+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:34.781","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.20760838200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.30","version":"206'252","reported_seq":1163,"reported_epoch":279,"state":"active+clean","last_fresh":"2023-01-15T14:32:47.056338+0000","last_change":"2023-01-15T14:32:41.078797+0000","last_active":"2023-01-15T14:32:47.056338+0000","last_peered":"2023-01-15T14:32:47.056338+0000","last_clean":"2023-01-15T14:32:47.056338+0000","last_became_active":"2023-01-15T14:31:39.915702+0000","last_became_peered":"2023-01-15T14:31:39.915702+0000","last_unstale":"2023-01-15T14:32:47.056338+0000","last_undegraded":"2023-01-15T14:32:47.056338+0000","last_fullsized":"2023-01-15T14:32:47.056338+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:32:58.379","scrub_duration":1.1539330249999999,"objects_trimmed":0,"snaptrim_duration":0.111191213,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d5","length":"1"}]},{"pgid":"3.31","version":"204'150","reported_seq":1046,"reported_epoch":277,"state":"active+clean","last_fresh":"2023-01-15T14:32:44.147302+0000","last_change":"2023-01-15T14:32:41.497184+0000","last_active":"2023-01-15T14:32:44.147302+0000","last_peered":"2023-01-15T14:32:44.147302+0000","last_clean":"2023-01-15T14:32:44.147302+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:32:44.147302+0000","last_undegraded":"2023-01-15T14:32:44.147302+0000","last_fullsized":"2023-01-15T14:32:44.147302+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:16.599","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.50773348600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c0","length":"1"},{"start":"c7","length":"2"},{"start":"cc","length":"1"},{"start":"cf","length":"2"},{"start":"d3","length":"1"},{"start":"d5","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":76,"stat_sum":{"num_bytes":184107274,"num_objects":85,"num_object_clones":44,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":5,"num_read":781,"num_read_kb":822005,"num_write":634,"num_write_kb":237688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":301,"num_bytes_recovered":195907129,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12426,"ondisk_log_size":12426,"up":228,"acting":228,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574906,"num_pgs":28,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":897972,"kb_used_data":92811936,"kb_used_omap":4662,"kb_used_meta":0,"kb_avail":92801408,"statfs":{"total":95948165120,"available":95028641792,"internally_reserved":10780672,"allocated":95039422464,"data_stored":95039422464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4773892,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":4.774,"15min":4.3040000000000003},"min":{"1min":0.55500000000000005,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.98199999999999998,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.54000000000000004},{"interface":"front","average":{"1min":2.9169999999999998,"5min":6.2539999999999996,"15min":8.2240000000000002},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":43.125,"5min":73.769000000000005,"15min":73.769000000000005},"last":0.68700000000000006}]},{"osd":1,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":2.2909999999999999,"5min":4.4909999999999997,"15min":5.8600000000000003},"min":{"1min":0.56899999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":31.518999999999998,"5min":88.003,"15min":88.003},"last":1.373},{"interface":"front","average":{"1min":9.8260000000000005,"5min":6.79,"15min":6.4500000000000002},"min":{"1min":0.59399999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.581999999999994,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.58799999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":14.355,"5min":7.7380000000000004,"15min":5.7729999999999997},"min":{"1min":0.51400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":151.31800000000001,"5min":151.31800000000001,"15min":151.31800000000001},"last":1.179},{"interface":"front","average":{"1min":1.0089999999999999,"5min":3.2309999999999999,"15min":3.8620000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.633,"5min":91.837999999999994,"15min":91.837999999999994},"last":1.2050000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":18.995000000000001,"5min":7.3920000000000003,"15min":3.0960000000000001},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":209.05500000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":1.2370000000000001},{"interface":"front","average":{"1min":5.7030000000000003,"5min":4.383,"15min":4.5620000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":98.825000000000003,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.65000000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":2.9430000000000001,"5min":5.5949999999999998,"15min":8.1419999999999995},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":42.893999999999998,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":8.5739999999999998,"5min":5.5899999999999999,"15min":2.5059999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":154.999,"5min":164.21199999999999,"15min":164.21199999999999},"last":1.3169999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":3.0379999999999998,"5min":4.4029999999999996,"15min":5.6849999999999996},"min":{"1min":0.52100000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":43.029000000000003,"5min":73.561000000000007,"15min":73.561000000000007},"last":1.0269999999999999},{"interface":"front","average":{"1min":3.7930000000000001,"5min":3.9950000000000001,"15min":4.6280000000000001},"min":{"1min":0.48399999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":58.134999999999998,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.355}]},{"osd":6,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":3.109,"5min":5.351,"15min":7.4660000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":42.978999999999999,"5min":73.638999999999996,"15min":73.638999999999996},"last":1.1379999999999999},{"interface":"front","average":{"1min":3.8410000000000002,"5min":2.5169999999999999,"15min":1.5720000000000001},"min":{"1min":0.60299999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":58.204999999999998,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.2789999999999999}]}]},{"osd":6,"up_from":0,"seq":55834574907,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574907,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574906,"num_pgs":58,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":991384,"kb_used_data":92726496,"kb_used_omap":9594,"kb_used_meta":0,"kb_avail":92707996,"statfs":{"total":95948165120,"available":94932987904,"internally_reserved":18944000,"allocated":94951931904,"data_stored":94951931904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9824947,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":12,"commit_latency_ns":0,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":103.827,"5min":24.414999999999999,"15min":11.414999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2165.8420000000001,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":5.7409999999999997,"5min":3,"15min":1.52},"min":{"1min":0.56999999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":88.600999999999999,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.95899999999999996}]},{"osd":1,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":6.6109999999999998,"15min":7.9329999999999998},"min":{"1min":0.50700000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":69.415000000000006,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.863,"5min":4.3360000000000003,"15min":1.974},"min":{"1min":0.58999999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":22.411999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.98699999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":103.90300000000001,"5min":25.164999999999999,"15min":12.061999999999999},"min":{"1min":0.59899999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2165.3220000000001,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.871,"15min":0.89700000000000002},"min":{"1min":0.623,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0349999999999999,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.89600000000000002}]},{"osd":3,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":10.015000000000001,"5min":3,"15min":1.8919999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":88.823999999999998,"5min":88.823999999999998,"15min":88.823999999999998},"last":1.0269999999999999},{"interface":"front","average":{"1min":105.232,"5min":25.382000000000001,"15min":12.561},"min":{"1min":0.628,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2165.4690000000001,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":0.83799999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.88100000000000001,"15min":0.90700000000000003},"min":{"1min":0.58299999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.302,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":5.867,"5min":2.4590000000000001,"15min":2.3519999999999999},"min":{"1min":0.68200000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":88.911000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":1.0900000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":4.8620000000000001,"15min":5.4020000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.341,"5min":99.780000000000001,"15min":99.780000000000001},"last":1.006},{"interface":"front","average":{"1min":104.14,"5min":23.286000000000001,"15min":11.204000000000001},"min":{"1min":0.84599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2165.7330000000002,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.038}]},{"osd":7,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":9.5570000000000004,"5min":7.5519999999999996,"15min":11.269},"min":{"1min":0.76100000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":89.061000000000007,"5min":179.73500000000001,"15min":179.73500000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":104.11,"5min":26.864999999999998,"15min":16.446999999999999},"min":{"1min":0.78600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2165.5309999999999,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.0529999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574906,"num_pgs":57,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":951676,"kb_used_data":92764176,"kb_used_omap":9766,"kb_used_meta":0,"kb_avail":92747704,"statfs":{"total":95948165120,"available":94973648896,"internally_reserved":16867328,"allocated":94990516224,"data_stored":94990516224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10001168,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":24,"commit_latency_ns":0,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":12.667,"5min":8.0429999999999993,"15min":5.9130000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":122.065,"5min":174.52199999999999,"15min":174.52199999999999},"last":17.731000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":4.8109999999999999,"15min":6.6980000000000004},"min":{"1min":0.41299999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.0469999999999999,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.46899999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":4.1790000000000003,"5min":9.7569999999999997,"15min":12.273},"min":{"1min":0.45000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":51.162999999999997,"5min":159.256,"15min":159.256},"last":0.54200000000000004},{"interface":"front","average":{"1min":20.414000000000001,"5min":10.041,"15min":3.8410000000000002},"min":{"1min":0.42799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":121.203,"5min":174.578,"15min":174.578},"last":17.826000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":2.6139999999999999,"15min":3.1040000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3700000000000001,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":13.534000000000001,"5min":6.8380000000000001,"15min":3.7410000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":122.13,"5min":174.45099999999999,"15min":174.45099999999999},"last":17.876000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":122.569,"5min":29.460999999999999,"15min":14.195},"min":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2059.6680000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":30.596},{"interface":"front","average":{"1min":122.634,"5min":34.735999999999997,"15min":20.827999999999999},"min":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2058.4229999999998,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":0.81299999999999994}]},{"osd":5,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":2.0019999999999998,"5min":9.0939999999999994,"15min":12.300000000000001},"min":{"1min":0.625,"5min":0.622,"15min":0.622},"max":{"1min":21.789999999999999,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":13.038,"5min":5.7619999999999996,"15min":2.6019999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":122.208,"5min":174.61000000000001,"15min":174.61000000000001},"last":17.917000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":5.766,"15min":2.6099999999999999},"min":{"1min":0.65900000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":122.26600000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":17.989000000000001},{"interface":"front","average":{"1min":0.98399999999999999,"5min":2.4969999999999999,"15min":3.3399999999999999},"min":{"1min":0.70999999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.49,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.90400000000000003}]},{"osd":7,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.5990000000000002,"15min":3.5569999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.522,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.04},{"interface":"front","average":{"1min":5.444,"5min":11.496,"15min":15.891999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":55.337000000000003,"5min":159.512,"15min":159.512},"last":0.97599999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574906,"num_pgs":46,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":992180,"kb_used_data":92725452,"kb_used_omap":7881,"kb_used_meta":0,"kb_avail":92707200,"statfs":{"total":95948165120,"available":94932172800,"internally_reserved":18690048,"allocated":94950862848,"data_stored":94950862848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8070444,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":30,"commit_latency_ns":0,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":3.2869999999999999,"5min":3.0299999999999998,"15min":1.4370000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":55.649999999999999,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":1.4410000000000001,"15min":0.95699999999999996},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.85199999999999998,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.64600000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":7.359,"5min":5.0350000000000001,"15min":2.2109999999999999},"min":{"1min":0.27700000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":83.007999999999996,"5min":143.071,"15min":143.071},"last":0.752},{"interface":"front","average":{"1min":0.60199999999999998,"5min":2.5489999999999999,"15min":3.0870000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.0109999999999999,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.52400000000000002}]},{"osd":3,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":8.6809999999999992,"5min":10.118,"15min":10.307},"min":{"1min":0.308,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":86.191999999999993,"5min":183.53,"15min":183.53},"last":0.70199999999999996},{"interface":"front","average":{"1min":3.3500000000000001,"5min":4.3319999999999999,"15min":1.9550000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":54.307000000000002,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.71999999999999997}]},{"osd":4,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":4.718,"5min":6.1630000000000003,"15min":4.4870000000000001},"min":{"1min":0.66300000000000003,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":54.383000000000003,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":2.077,"5min":5.4790000000000001,"15min":4.3650000000000002},"min":{"1min":0.59299999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":24.859999999999999,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.86599999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":2.5139999999999998,"15min":1.6220000000000001},"min":{"1min":0.622,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2709999999999999,"5min":143.262,"15min":143.262},"last":1.1659999999999999},{"interface":"front","average":{"1min":3.5920000000000001,"5min":2.5899999999999999,"15min":1.637},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":54.353999999999999,"5min":66.456999999999994,"15min":66.456999999999994},"last":1.1120000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.8660000000000001,"15min":1.3100000000000001},"min":{"1min":0.59599999999999997,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.4259999999999999,"5min":88.542000000000002,"15min":88.542000000000002},"last":1.1339999999999999},{"interface":"front","average":{"1min":0.94199999999999995,"5min":2.6560000000000001,"15min":1.6970000000000001},"min":{"1min":0.64300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.319,"5min":143.215,"15min":143.215},"last":0.95599999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":6.8410000000000002,"5min":4.306,"15min":3.4740000000000002},"min":{"1min":0.66900000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":118.652,"5min":118.652,"15min":118.652},"last":1.2190000000000001},{"interface":"front","average":{"1min":0.95299999999999996,"5min":3.198,"15min":2.9169999999999998},"min":{"1min":0.67300000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.278,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.999}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":54,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":941524,"kb_used_data":92767716,"kb_used_omap":9435,"kb_used_meta":0,"kb_avail":92757856,"statfs":{"total":95948165120,"available":94984044544,"internally_reserved":10096640,"allocated":94994141184,"data_stored":94994141184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9662080,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":23,"commit_latency_ns":0,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":4.258,"15min":4.0140000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.91300000000000003,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":5.7370000000000001,"5min":3.9289999999999998,"15min":4.1909999999999998},"min":{"1min":0.39800000000000002,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":86.927000000000007,"5min":86.927000000000007,"15min":86.927000000000007},"last":0.66100000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":3.6499999999999999,"5min":1.764,"15min":1.232},"min":{"1min":0.25700000000000001,"5min":0.245,"15min":0.245},"max":{"1min":64.941999999999993,"5min":64.941999999999993,"15min":64.941999999999993},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":2.5049999999999999,"15min":3.5499999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.044,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.745}]},{"osd":3,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":4.9669999999999996,"5min":12.464,"15min":17.378},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":90.161000000000001,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":11.167999999999999,"5min":6.7830000000000004,"15min":7.0910000000000002},"min":{"1min":0.33100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":86.894999999999996,"5min":86.894999999999996,"15min":86.894999999999996},"last":0.93999999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:32:12 2023","interfaces":[{"interface":"back","average":{"1min":5.6849999999999996,"5min":16.501000000000001,"15min":8.6519999999999992},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":87.128,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":9.8059999999999992,"5min":15.890000000000001,"15min":7.7000000000000002},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":84.629999999999995,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.70999999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":3.9940000000000002,"5min":1.5740000000000001,"15min":1.2130000000000001},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":65.120999999999995,"5min":65.120999999999995,"15min":65.120999999999995},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.89200000000000002,"5min":2.327,"15min":3.734},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.2270000000000001,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.80900000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":6.0309999999999997,"5min":4.4119999999999999,"15min":4.8049999999999997},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":87.153000000000006,"5min":87.153000000000006,"15min":87.153000000000006},"last":1.042},{"interface":"front","average":{"1min":4.0220000000000002,"5min":1.6359999999999999,"15min":1.294},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":65.156000000000006,"5min":65.156000000000006,"15min":65.156000000000006},"last":0.84199999999999997}]},{"osd":7,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":5.3310000000000004,"5min":6.7930000000000001,"15min":6.9420000000000002},"min":{"1min":0.73799999999999999,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":91.843999999999994,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":6.0880000000000001,"5min":4.3399999999999999,"15min":4.6130000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":87.069999999999993,"5min":87.069999999999993,"15min":87.069999999999993},"last":1.079}]}]},{"osd":0,"up_from":13,"seq":55834574908,"num_pgs":12,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":811760,"kb_used_data":92887664,"kb_used_omap":4856,"kb_used_meta":0,"kb_avail":92887620,"statfs":{"total":95948165120,"available":95116922880,"internally_reserved":45056,"allocated":95116967936,"data_stored":95116967936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4972555,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":2.7509999999999999,"5min":3.1230000000000002,"15min":1.456},"min":{"1min":0.29499999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":49.725000000000001,"5min":69.872,"15min":69.872},"last":0.70599999999999996},{"interface":"front","average":{"1min":3.5139999999999998,"5min":3.2349999999999999,"15min":1.5649999999999999},"min":{"1min":0.33700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":61.447000000000003,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.98899999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":3.4159999999999999,"15min":1.601},"min":{"1min":0.52700000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.2,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.629},{"interface":"front","average":{"1min":0.54800000000000004,"5min":2.6499999999999999,"15min":1.3560000000000001},"min":{"1min":0.32400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.90800000000000003,"5min":71.406999999999996,"15min":71.406999999999996},"last":1.302}]},{"osd":3,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":2.863,"5min":10.630000000000001,"15min":14.728999999999999},"min":{"1min":0.372,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":49.331000000000003,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":8.0960000000000001,"5min":7.0990000000000002,"15min":2.8999999999999999},"min":{"1min":0.45800000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":162.61600000000001,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.90300000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":68.361999999999995,"5min":16.706,"15min":6.226},"min":{"1min":0.63900000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1486.309,"5min":1486.309,"15min":1486.309},"last":1.135},{"interface":"front","average":{"1min":70.664000000000001,"5min":17.922999999999998,"15min":6.657},"min":{"1min":0.56599999999999995,"5min":0.439,"15min":0.439},"max":{"1min":1486.3789999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":1.075}]},{"osd":5,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":6.4420000000000002,"15min":8.6790000000000003},"min":{"1min":0.64700000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4910000000000001,"5min":176.34399999999999,"15min":176.34399999999999},"last":1.107},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.7370000000000001,"15min":1.2709999999999999},"min":{"1min":0.60699999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3220000000000001,"5min":71.563999999999993,"15min":71.563999999999993},"last":1.2529999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":3.198,"5min":3.609,"15min":1.9830000000000001},"min":{"1min":0.68999999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":49.616,"5min":70.259,"15min":70.259},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.006,"5min":3.0299999999999998,"15min":1.766},"min":{"1min":0.755,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.4419999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":1.2250000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":9.0449999999999999,"15min":13.462},"min":{"1min":0.57199999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.137,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":3.21,"5min":3.617,"15min":1.986},"min":{"1min":0.67100000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":49.670000000000002,"5min":70.322000000000003,"15min":70.322000000000003},"last":1.179}]}]}],"pool_statfs":[]}} 2023-01-15T14:32:51.605 INFO:tasks.thrashosds.thrasher:Setting 3.39 to [4, 2, 3, 5, 6, 7] 2023-01-15T14:32:51.605 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.39', '4', '2', '3', '5', '6', '7'] 2023-01-15T14:32:51.605 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.39 4 2 3 5 6 7 2023-01-15T14:32:51.606 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:32:51.607 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:51.607 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:51.607 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:32:51.609 INFO:tasks.rados.rados.0.smithi090.stdout:1145: finishing write tid 2 to smithi09025264-48 2023-01-15T14:32:51.609 INFO:tasks.rados.rados.0.smithi090.stdout:1145: finishing write tid 3 to smithi09025264-48 2023-01-15T14:32:51.609 INFO:tasks.rados.rados.0.smithi090.stdout:1145: finishing write tid 4 to smithi09025264-48 2023-01-15T14:32:51.609 INFO:tasks.rados.rados.0.smithi090.stdout:1145: finishing write tid 5 to smithi09025264-48 2023-01-15T14:32:51.609 INFO:tasks.rados.rados.0.smithi090.stdout:1145: oid 48 updating version 0 to 214 2023-01-15T14:32:51.609 INFO:tasks.rados.rados.0.smithi090.stdout:1145: oid 48 updating version 214 to 215 2023-01-15T14:32:51.609 INFO:tasks.rados.rados.0.smithi090.stdout:1145: oid 48 version 215 is already newer than 213 2023-01-15T14:32:51.609 INFO:tasks.rados.rados.0.smithi090.stdout:1145: oid 48 version 215 is already newer than 212 2023-01-15T14:32:51.609 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 215 (ObjNum 425 snap 116 seq_num 425) dirty exists 2023-01-15T14:32:51.609 INFO:tasks.rados.rados.0.smithi090.stdout:1145: left oid 48 (ObjNum 425 snap 116 seq_num 425) 2023-01-15T14:32:51.610 INFO:tasks.rados.rados.0.smithi090.stdout:1148: finishing write tid 1 to smithi09025264-28 2023-01-15T14:32:51.610 INFO:tasks.rados.rados.0.smithi090.stdout:1150: expect (ObjNum 361 snap 96 seq_num 361) 2023-01-15T14:32:51.610 INFO:tasks.rados.rados.0.smithi090.stdout:1149: finishing write tid 2 to smithi09025264-17 2023-01-15T14:32:51.610 INFO:tasks.rados.rados.0.smithi090.stdout:1149: finishing write tid 3 to smithi09025264-17 2023-01-15T14:32:51.610 INFO:tasks.rados.rados.0.smithi090.stdout:1149: finishing write tid 4 to smithi09025264-17 2023-01-15T14:32:51.610 INFO:tasks.rados.rados.0.smithi090.stdout:1149: finishing write tid 5 to smithi09025264-17 2023-01-15T14:32:51.610 INFO:tasks.rados.rados.0.smithi090.stdout:1149: oid 17 updating version 0 to 378 2023-01-15T14:32:51.610 INFO:tasks.rados.rados.0.smithi090.stdout:1149: oid 17 version 378 is already newer than 376 2023-01-15T14:32:51.610 INFO:tasks.rados.rados.0.smithi090.stdout:1149: oid 17 updating version 378 to 379 2023-01-15T14:32:51.610 INFO:tasks.rados.rados.0.smithi090.stdout:1149: oid 17 version 379 is already newer than 377 2023-01-15T14:32:51.610 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 379 (ObjNum 427 snap 116 seq_num 427) dirty exists 2023-01-15T14:32:51.611 INFO:tasks.rados.rados.0.smithi090.stdout:1149: left oid 17 (ObjNum 427 snap 116 seq_num 427) 2023-01-15T14:32:51.611 INFO:tasks.rados.rados.0.smithi090.stdout:1148: finishing write tid 2 to smithi09025264-28 2023-01-15T14:32:51.611 INFO:tasks.rados.rados.0.smithi090.stdout:1148: finishing write tid 3 to smithi09025264-28 2023-01-15T14:32:51.611 INFO:tasks.rados.rados.0.smithi090.stdout:1148: finishing write tid 4 to smithi09025264-28 2023-01-15T14:32:51.611 INFO:tasks.rados.rados.0.smithi090.stdout:1148: oid 28 updating version 0 to 333 2023-01-15T14:32:51.611 INFO:tasks.rados.rados.0.smithi090.stdout:1148: oid 28 version 333 is already newer than 331 2023-01-15T14:32:51.611 INFO:tasks.rados.rados.0.smithi090.stdout:1148: oid 28 version 333 is already newer than 332 2023-01-15T14:32:51.611 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 333 (ObjNum 426 snap 116 seq_num 426) dirty exists 2023-01-15T14:32:51.611 INFO:tasks.rados.rados.0.smithi090.stdout:1148: left oid 28 (ObjNum 426 snap 116 seq_num 426) 2023-01-15T14:32:51.611 INFO:tasks.rados.rados.0.smithi090.stdout:1145: done (7 left) 2023-01-15T14:32:51.611 INFO:tasks.rados.rados.0.smithi090.stdout:1146: done (6 left) 2023-01-15T14:32:51.612 INFO:tasks.rados.rados.0.smithi090.stdout:1147: done (5 left) 2023-01-15T14:32:51.612 INFO:tasks.rados.rados.0.smithi090.stdout:1148: done (4 left) 2023-01-15T14:32:51.612 INFO:tasks.rados.rados.0.smithi090.stdout:1149: done (3 left) 2023-01-15T14:32:51.612 INFO:tasks.rados.rados.0.smithi090.stdout:1150: done (2 left) 2023-01-15T14:32:51.612 INFO:tasks.rados.rados.0.smithi090.stdout:1151: done (1 left) 2023-01-15T14:32:51.612 INFO:tasks.rados.rados.0.smithi090.stdout:1153: done (0 left) 2023-01-15T14:32:51.612 INFO:tasks.rados.rados.0.smithi090.stdout:1154: write oid 22 current snap is 116 2023-01-15T14:32:51.612 INFO:tasks.rados.rados.0.smithi090.stdout:1154: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2023-01-15T14:32:51.612 INFO:tasks.rados.rados.0.smithi090.stdout:1154: writing smithi09025264-22 from 410203 to 1207792 tid 1 2023-01-15T14:32:51.612 INFO:tasks.rados.rados.0.smithi090.stdout:1154: writing smithi09025264-22 from 1787481 to 2221951 tid 2 2023-01-15T14:32:51.612 INFO:tasks.rados.rados.0.smithi090.stdout:1154: writing smithi09025264-22 from 2681405 to 3210203 tid 3 2023-01-15T14:32:51.613 INFO:tasks.rados.rados.0.smithi090.stdout:1155: snap_remove snap 112 2023-01-15T14:32:51.613 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:51.600+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:51.707 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:51.700+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:51.783 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:51.783 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:51.794 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:32:51.808 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:51.800+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:51.932 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:51.933 INFO:tasks.rados.rados.0.smithi090.stdout:1155: done (1 left) 2023-01-15T14:32:51.933 INFO:tasks.rados.rados.0.smithi090.stdout:1156: read oid 13 snap -1 2023-01-15T14:32:51.933 INFO:tasks.rados.rados.0.smithi090.stdout:1156: expect (ObjNum 412 snap 112 seq_num 412) 2023-01-15T14:32:51.933 INFO:tasks.rados.rados.0.smithi090.stdout:1154: finishing write tid 1 to smithi09025264-22 2023-01-15T14:32:51.933 INFO:tasks.rados.rados.0.smithi090.stdout:1154: finishing write tid 2 to smithi09025264-22 2023-01-15T14:32:51.933 INFO:tasks.rados.rados.0.smithi090.stdout:1154: finishing write tid 3 to smithi09025264-22 2023-01-15T14:32:51.933 INFO:tasks.rados.rados.0.smithi090.stdout:1157: read oid 11 snap 113 2023-01-15T14:32:51.933 INFO:tasks.rados.rados.0.smithi090.stdout:1157: expect (ObjNum 407 snap 108 seq_num 407) 2023-01-15T14:32:51.933 INFO:tasks.rados.rados.0.smithi090.stdout:1158: snap_remove snap 109 2023-01-15T14:32:51.934 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:51.934 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:51.935 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:32:51.935 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:51.935 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:51.925+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:51.943 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:51.984 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:52.162 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:52.162 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:52.163 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:32:52.163 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:32:52.163 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:52.164 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:32:52.164 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:52.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:32:52.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:32:52.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:32:52.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:32:52.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:32:52.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:32:52.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:32:52.166 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:32:52.166 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:52.160+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:52.263 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:52.257+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:52.330 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 186.257136527, 2023-01-15T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:52.331 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:52.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:32:52.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:52.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 137.421379137, 2023-01-15T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:52.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:52.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:52.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:52.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:32:52.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:32:52.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:32:52.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:52.348 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 137.42132819400001, 2023-01-15T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:52.349 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:52.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:32:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:32:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:52.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:32:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:32:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:32:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:32:52.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:32:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:32:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:32:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:32:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:32:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:32:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:32:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:32:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:32:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:32:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:32:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:32:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:52.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:32:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:32:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:32:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:32:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:52.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.480428953, 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:32:52.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:32:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:32:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:52.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:32:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:32:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:32:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:32:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:32:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:32:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:32:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:32:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:32:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:32:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:32:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:32:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:52.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:32:52.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:32:53.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:32:53.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:32:53.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:53.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:32:53.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:32:53.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:32:53.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:32:53.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:32:53.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:32:53.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:32:53.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:32:53.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:32:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:32:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:32:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:32:53.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:32:53.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:32:53.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:32:53.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:32:53.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:32:53.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:32:53.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.011 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.011 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:53.012 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:32:53.012 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.40665478, 2023-01-15T14:32:53.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:32:53.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:53.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:53.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:53.012 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:32:53.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.013 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:32:53.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:32:53.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:32:53.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:32:53.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:32:53.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:32:53.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:32:53.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:32:53.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:53.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:32:53.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:32:53.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:32:53.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:32:53.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:32:53.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:32:53.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:53.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:32:53.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:32:53.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:32:53.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:32:53.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:32:53.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:32:53.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:32:53.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:32:53.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:53.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:32:53.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:32:53.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:32:53.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:32:53.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:32:53.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:32:53.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:32:53.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:32:53.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:32:53.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:32:53.021 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.022 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.022 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.022 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1712 3.4 3:2022fdfa:::smithi09025264-46:head [write 1880802~421790] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:53.022 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:32:53.022 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.40151608900001, 2023-01-15T14:32:53.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30396041800000001, 2023-01-15T14:32:53.022 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.022 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.022 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1712 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:32:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914322+0000", 2023-01-15T14:32:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914561+0000", 2023-01-15T14:32:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023884700000000001 2023-01-15T14:32:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914572+0000", 2023-01-15T14:32:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0499999999999999e-05 2023-01-15T14:32:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914576+0000", 2023-01-15T14:32:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6820000000000002e-06 2023-01-15T14:32:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014941+0000", 2023-01-15T14:32:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100364994 2023-01-15T14:32:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014959+0000", 2023-01-15T14:32:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7416000000000001e-05 2023-01-15T14:32:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018171+0000", 2023-01-15T14:32:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032121239999999998 2023-01-15T14:32:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018333+0000", 2023-01-15T14:32:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016180900000000001 2023-01-15T14:32:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:32:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018581+0000", 2023-01-15T14:32:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024773699999999999 2023-01-15T14:32:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018897+0000", 2023-01-15T14:32:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031622099999999999 2023-01-15T14:32:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018907+0000", 2023-01-15T14:32:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0061000000000001e-05 2023-01-15T14:32:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.019108+0000", 2023-01-15T14:32:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020080099999999999 2023-01-15T14:32:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.020057+0000", 2023-01-15T14:32:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094954499999999999 2023-01-15T14:32:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150472+0000", 2023-01-15T14:32:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.130414958 2023-01-15T14:32:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218188+0000", 2023-01-15T14:32:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067715656999999999 2023-01-15T14:32:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218242+0000", 2023-01-15T14:32:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4651999999999998e-05 2023-01-15T14:32:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218284+0000", 2023-01-15T14:32:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1084999999999997e-05 2023-01-15T14:32:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:32:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:32:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.51269936, 2023-01-15T14:32:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:32:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:32:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:32:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:32:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:32:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:32:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:32:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:32:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:32:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:32:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:32:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:32:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:32:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:32:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:32:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:32:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1820 3.1f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.412397875, 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306504938, 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1820 2023-01-15T14:32:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:32:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903440+0000", 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903448+0000", 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9430000000000006e-06 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903469+0000", 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0305e-05 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903481+0000", 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:32:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903616+0000", 2023-01-15T14:32:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013477199999999999 2023-01-15T14:32:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903643+0000", 2023-01-15T14:32:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7098999999999999e-05 2023-01-15T14:32:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209081+0000", 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30543790300000001 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209213+0000", 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013155400000000001 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209600+0000", 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038751699999999998 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209946+0000", 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034601 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.046 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.629292679000002, 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:32:53.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:53.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:32:53.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:32:53.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:32:53.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:32:53.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:32:53.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:32:53.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:53.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:32:53.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:32:53.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:32:53.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:32:53.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:32:53.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.056 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.056 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:53.056 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:32:53.057 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.625885556, 2023-01-15T14:32:53.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:32:53.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.057 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:53.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:53.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:53.057 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:32:53.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.058 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:32:53.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:32:53.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:32:53.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:53.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:32:53.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:32:53.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:32:53.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:32:53.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:53.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:32:53.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:32:53.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:32:53.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:32:53.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:32:53.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:32:53.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:53.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:32:53.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:32:53.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:32:53.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:32:53.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:32:53.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:32:53.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:32:53.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:32:53.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:53.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:32:53.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:32:53.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:32:53.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:32:53.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:32:53.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:32:53.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:32:53.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:32:53.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:32:53.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:32:53.066 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.066 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.066 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.625669951000006, 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:53.067 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:32:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:32:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:32:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:32:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:32:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:32:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:32:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:32:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:32:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:32:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:32:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:32:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:53.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:32:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:32:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:32:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:32:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:32:53.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:32:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:32:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:32:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:32:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:53.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:32:53.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:32:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:32:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:32:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:32:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.214 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:32:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:32:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.625562876999993, 2023-01-15T14:32:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:32:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:53.215 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:32:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:32:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:32:53.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:32:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:32:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:32:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:32:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:32:53.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:32:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:32:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:32:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:32:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:32:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:32:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:32:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:32:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:32:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:32:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:32:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:32:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:32:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:32:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:32:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:32:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.495323826999993, 2023-01-15T14:32:53.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:32:53.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:32:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:32:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:32:53.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:32:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:32:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:32:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:32:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:32:53.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:32:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:32:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:32:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:32:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:32:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:32:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:32:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:32:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:32:53.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:32:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:32:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:32:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:32:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:32:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:32:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:32:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:32:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.495219289999994, 2023-01-15T14:32:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:32:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:32:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:32:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:32:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:32:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:32:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:32:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:32:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:32:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:32:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:32:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:32:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:32:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:32:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:32:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:32:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:32:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:32:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:32:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.495149358999996, 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:32:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:32:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:32:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:32:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:32:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:32:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:32:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:32:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:32:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:32:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:32:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:32:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:32:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:32:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:32:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:32:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:32:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.4.0:534 3.1a e226/224 3:5919d83e:::smithi09025264-13:a9 v 226'240, mlcod=0'0)", 2023-01-15T14:32:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:32:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 58.780254683000003, 2023-01-15T14:32:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30202553599999998, 2023-01-15T14:32:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:32:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:32:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535584+0000", 2023-01-15T14:32:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:53.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535592+0000", 2023-01-15T14:32:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3010000000000007e-06 2023-01-15T14:32:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535604+0000", 2023-01-15T14:32:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1800000000000001e-05 2023-01-15T14:32:53.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.635708+0000", 2023-01-15T14:32:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10010366800000001 2023-01-15T14:32:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836079+0000", 2023-01-15T14:32:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20037171500000001 2023-01-15T14:32:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836113+0000", 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836714+0000", 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060093400000000002 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836737+0000", 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3073000000000001e-05 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836816+0000", 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8740999999999997e-05 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836898+0000", 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2731999999999993e-05 2023-01-15T14:32:53.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.837610+0000", 2023-01-15T14:32:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071176200000000001 2023-01-15T14:32:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.258 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2780 3.38 e265/232 3:1eaad226:::smithi09025264-49:head v 265'279, mlcod=261'274)", 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 20.323978232999998, 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32005591799999999, 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991860+0000", 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993927+0000", 2023-01-15T14:32:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020673229999999998 2023-01-15T14:32:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993946+0000", 2023-01-15T14:32:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9429999999999999e-05 2023-01-15T14:32:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.094039+0000", 2023-01-15T14:32:53.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092898 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285071+0000", 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.191031904 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285106+0000", 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5227e-05 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286323+0000", 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012163829999999999 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286335+0000", 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2224e-05 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287218+0000", 2023-01-15T14:32:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088265000000000001 2023-01-15T14:32:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287250+0000", 2023-01-15T14:32:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2209000000000003e-05 2023-01-15T14:32:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.311917+0000", 2023-01-15T14:32:53.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024667062999999999 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2868 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'278, mlcod=264'275)", 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.322087292000001, 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31268252400000002, 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.265 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993751+0000", 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993762+0000", 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0822000000000001e-05 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993777+0000", 2023-01-15T14:32:53.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5540999999999999e-05 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993787+0000", 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0173999999999999e-05 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290198+0000", 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29641022099999997 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290212+0000", 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4214e-05 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290300+0000", 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8300000000000005e-05 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290312+0000", 2023-01-15T14:32:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2344000000000001e-05 2023-01-15T14:32:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290357+0000", 2023-01-15T14:32:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4367999999999999e-05 2023-01-15T14:32:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290389+0000", 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1983000000000002e-05 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.306434+0000", 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016045667 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.271 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:32:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:32:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.322037867000001, 2023-01-15T14:32:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:32:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:32:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:32:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:32:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:32:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:32:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:32:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:32:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:32:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:32:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:32:53.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.31822618, 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:53.278 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:32:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:32:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:32:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:32:53.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:32:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:32:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:32:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:32:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:53.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:53.284 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:53.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:32:53.285 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:32:53.286 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:53.280+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:53.287 INFO:tasks.rados.rados.0.smithi090.stdout:1158: done (3 left) 2023-01-15T14:32:53.287 INFO:tasks.rados.rados.0.smithi090.stdout:1159: write oid 25 current snap is 116 2023-01-15T14:32:53.287 INFO:tasks.rados.rados.0.smithi090.stdout:1154: finishing write tid 4 to smithi09025264-22 2023-01-15T14:32:53.287 INFO:tasks.rados.rados.0.smithi090.stdout:1159: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2023-01-15T14:32:53.287 INFO:teuthology.orchestra.run.smithi090.stderr:set 3.39 pg_upmap_items mapping to [4->2,3->5,6->7] 2023-01-15T14:32:53.288 INFO:tasks.rados.rados.0.smithi090.stdout:1159: writing smithi09025264-25 from 427010 to 899848 tid 1 2023-01-15T14:32:53.288 INFO:tasks.rados.rados.0.smithi090.stdout:1159: writing smithi09025264-25 from 1329610 to 2024091 tid 2 2023-01-15T14:32:53.289 INFO:tasks.rados.rados.0.smithi090.stdout:1159: writing smithi09025264-25 from 2708828 to 3227010 tid 3 2023-01-15T14:32:53.289 INFO:tasks.rados.rados.0.smithi090.stdout:1160: snap_remove snap 114 2023-01-15T14:32:53.387 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:53.381+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:53.625 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:53.626 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:53.626 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:32:53.626 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:53.627 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:32:53.628 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:53.620+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:53.725 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:53.724+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:53.828 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:32:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:32:53.828 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:32:53.828 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:53.829 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:32:53.829 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:53.821+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:53.871 INFO:tasks.rados.rados.0.smithi090.stdout:1160: done (4 left) 2023-01-15T14:32:54.027 INFO:tasks.rados.rados.0.smithi090.stdout:1161: copy_from oid 31 from oid 19 current snap is 116 2023-01-15T14:32:54.028 INFO:tasks.rados.rados.0.smithi090.stdout:1162: delete oid 3 current snap is 116 2023-01-15T14:32:54.028 INFO:tasks.rados.rados.0.smithi090.stdout:1154: finishing write tid 5 to smithi09025264-22 2023-01-15T14:32:54.028 INFO:tasks.rados.rados.0.smithi090.stdout:1154: oid 22 updating version 0 to 344 2023-01-15T14:32:54.028 INFO:tasks.rados.rados.0.smithi090.stdout:1154: oid 22 updating version 344 to 346 2023-01-15T14:32:54.028 INFO:tasks.rados.rados.0.smithi090.stdout:1154: oid 22 version 346 is already newer than 343 2023-01-15T14:32:54.028 INFO:tasks.rados.rados.0.smithi090.stdout:1154: oid 22 version 346 is already newer than 345 2023-01-15T14:32:54.028 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 346 (ObjNum 429 snap 116 seq_num 429) dirty exists 2023-01-15T14:32:54.028 INFO:tasks.rados.rados.0.smithi090.stdout:1154: left oid 22 (ObjNum 429 snap 116 seq_num 429) 2023-01-15T14:32:54.028 INFO:tasks.rados.rados.0.smithi090.stdout:1156: expect (ObjNum 412 snap 112 seq_num 412) 2023-01-15T14:32:54.030 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:32:54.031 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:54.032 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:32:54.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:32:54.032 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:32:54.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.032 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:168 3.8 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:54.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:32:54.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 266.82283494400002, 2023-01-15T14:32:54.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72621873100000001, 2023-01-15T14:32:54.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:54.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:54.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:54.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:54.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:54.033 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 168 2023-01-15T14:32:54.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.033 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:54.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:54.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:32:54.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:54.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115372+0000", 2023-01-15T14:32:54.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:54.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115371+0000", 2023-01-15T14:32:54.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:54.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:54.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115378+0000", 2023-01-15T14:32:54.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.007e-06 2023-01-15T14:32:54.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:54.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115401+0000", 2023-01-15T14:32:54.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3713000000000001e-05 2023-01-15T14:32:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115411+0000", 2023-01-15T14:32:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8130000000000007e-06 2023-01-15T14:32:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115800+0000", 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038879600000000001 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.115915+0000", 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011554599999999999 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334898+0000", 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218982238 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334999+0000", 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100869 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841296+0000", 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50629754900000001 2023-01-15T14:32:54.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841435+0000", 2023-01-15T14:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138789 2023-01-15T14:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.841591+0000", 2023-01-15T14:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155661 2023-01-15T14:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:54.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 266.80984286500001, 2023-01-15T14:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:54.439 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:54.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:54.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:32:54.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:54.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 266.80943952899997, 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:54.449 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:54.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:32:54.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:32:54.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:32:54.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:32:54.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:54.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:32:54.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:32:54.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:32:54.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:32:54.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:32:54.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:54.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:32:54.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:32:54.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:54.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:32:54.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:32:54.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:54.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:32:54.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:32:54.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:54.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:32:54.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:32:54.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:54.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:32:54.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:32:54.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:54.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:32:54.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:32:54.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:54.648 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:54.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:54.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.649 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:32:54.649 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:32:54.649 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 266.70921590900002, 2023-01-15T14:32:54.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:32:54.649 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:54.649 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:54.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:54.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:54.650 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:54.650 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:32:54.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.650 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:54.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:54.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:32:54.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:54.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:32:54.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:54.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:32:54.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:54.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:54.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:32:54.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:32:54.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:54.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:32:54.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:32:54.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:54.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:32:54.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:32:54.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:32:54.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:32:54.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:54.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:32:54.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:32:54.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:32:54.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:32:54.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:54.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:32:54.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:32:54.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:32:54.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:32:54.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:54.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:32:54.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:32:54.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:54.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:32:54.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:32:54.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:54.656 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:54.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:54.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.657 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:54.657 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:32:54.657 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 176.015900684, 2023-01-15T14:32:54.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:32:54.658 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:54.658 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:54.658 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:54.659 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:54.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:54.662 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:32:54.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.663 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:54.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:54.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:32:54.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:54.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:32:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:54.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:32:54.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:32:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:32:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:32:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:32:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:32:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:32:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:32:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:32:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:32:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:32:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:32:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:32:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:32:54.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:32:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.675 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:32:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 176.00943932999999, 2023-01-15T14:32:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:32:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:54.676 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:32:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:54.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:32:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:32:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:54.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:54.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:32:54.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:54.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:32:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:32:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:54.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:32:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:32:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:32:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:32:54.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:54.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:54.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:54.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:32:54.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:32:54.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:55.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:32:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:32:55.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:32:55.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 175.90901740499999, 2023-01-15T14:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:55.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:32:55.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:32:55.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:32:55.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:32:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.254 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 175.90867991900001, 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:55.255 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:32:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:32:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:55.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:32:55.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:55.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:32:55.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:55.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:55.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:32:55.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:32:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:32:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:32:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:32:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:32:55.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:32:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:32:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:32:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:32:55.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:32:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:32:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:32:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:32:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.260 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:32:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:32:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 175.90862402299999, 2023-01-15T14:32:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:32:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:32:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:55.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:32:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:32:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:32:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:32:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:32:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:55.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:32:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:32:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:32:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:32:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:32:55.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:32:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:32:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:32:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:32:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:32:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:32:55.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:55.266 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1690 3.16 3:68fb3263:::smithi09025264-33:head [write 1763227~472137] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:32:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.07569134299999, 2023-01-15T14:32:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72570748100000004, 2023-01-15T14:32:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:55.267 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1690 2023-01-15T14:32:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:32:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862515+0000", 2023-01-15T14:32:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862514+0000", 2023-01-15T14:32:55.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862746+0000", 2023-01-15T14:32:55.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000232301 2023-01-15T14:32:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862770+0000", 2023-01-15T14:32:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3286e-05 2023-01-15T14:32:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862779+0000", 2023-01-15T14:32:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4199999999999996e-06 2023-01-15T14:32:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862844+0000", 2023-01-15T14:32:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5036999999999994e-05 2023-01-15T14:32:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862860+0000", 2023-01-15T14:32:55.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6426000000000002e-05 2023-01-15T14:32:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051251+0000", 2023-01-15T14:32:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18839023299999999 2023-01-15T14:32:55.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.051466+0000", 2023-01-15T14:32:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021523199999999999 2023-01-15T14:32:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368326+0000", 2023-01-15T14:32:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31686002699999999 2023-01-15T14:32:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368559+0000", 2023-01-15T14:32:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023305400000000001 2023-01-15T14:32:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:32:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.368935+0000", 2023-01-15T14:32:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037579099999999999 2023-01-15T14:32:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:55.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369531+0000", 2023-01-15T14:32:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00059593199999999997 2023-01-15T14:32:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369571+0000", 2023-01-15T14:32:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0649999999999999e-05 2023-01-15T14:32:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:55.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370331+0000", 2023-01-15T14:32:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075969899999999996 2023-01-15T14:32:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371230+0000", 2023-01-15T14:32:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000899134 2023-01-15T14:32:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472895+0000", 2023-01-15T14:32:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101665191 2023-01-15T14:32:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.587914+0000", 2023-01-15T14:32:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.115018671 2023-01-15T14:32:55.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:55.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588013+0000", 2023-01-15T14:32:55.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8494999999999998e-05 2023-01-15T14:32:55.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:55.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588223+0000", 2023-01-15T14:32:55.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210294 2023-01-15T14:32:55.463 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.463 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:55.463 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.464 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1691 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 2235364] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:55.464 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:32:55.464 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.07540708900001, 2023-01-15T14:32:55.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72593841699999995, 2023-01-15T14:32:55.464 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:55.464 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:55.464 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1691 2023-01-15T14:32:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862800+0000", 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862799+0000", 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:55.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862807+0000", 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1280000000000008e-06 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862821+0000", 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.345e-05 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:55.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862826+0000", 2023-01-15T14:32:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3720000000000004e-06 2023-01-15T14:32:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862883+0000", 2023-01-15T14:32:55.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6589000000000003e-05 2023-01-15T14:32:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862888+0000", 2023-01-15T14:32:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4060000000000004e-06 2023-01-15T14:32:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151601+0000", 2023-01-15T14:32:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28871354100000002 2023-01-15T14:32:55.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.151732+0000", 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013032300000000001 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369729+0000", 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.217997515 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.369949+0000", 2023-01-15T14:32:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219664 2023-01-15T14:32:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:32:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370367+0000", 2023-01-15T14:32:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041822799999999998 2023-01-15T14:32:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370771+0000", 2023-01-15T14:32:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040358400000000002 2023-01-15T14:32:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370794+0000", 2023-01-15T14:32:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3410000000000001e-05 2023-01-15T14:32:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:55.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370880+0000", 2023-01-15T14:32:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6020999999999995e-05 2023-01-15T14:32:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371241+0000", 2023-01-15T14:32:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000360956 2023-01-15T14:32:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:55.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.472994+0000", 2023-01-15T14:32:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101753089 2023-01-15T14:32:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588348+0000", 2023-01-15T14:32:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.11535382299999999 2023-01-15T14:32:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588539+0000", 2023-01-15T14:32:55.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019090399999999999 2023-01-15T14:32:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588738+0000", 2023-01-15T14:32:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019910800000000001 2023-01-15T14:32:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.684 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:32:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:32:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.07537360500001, 2023-01-15T14:32:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:32:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:32:55.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:55.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:32:55.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:32:55.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:32:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:32:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:32:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:32:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:32:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:32:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:32:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:32:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:32:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:55.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:32:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:32:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:32:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:32:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:32:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:32:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:32:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:32:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:32:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:32:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:32:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:32:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2206 3.5 3:a1c47f99:::smithi09025264-2:head [write 1667661~418109] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.830633+0000", 2023-01-15T14:32:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.107573622000004, 2023-01-15T14:32:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70338831700000004, 2023-01-15T14:32:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:55.878 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2206 2023-01-15T14:32:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830633+0000", 2023-01-15T14:32:55.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830633+0000", 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830631+0000", 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830800+0000", 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016843000000000001 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830813+0000", 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3392e-05 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:55.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.830819+0000", 2023-01-15T14:32:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6099999999999997e-06 2023-01-15T14:32:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927297+0000", 2023-01-15T14:32:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.096477775000000002 2023-01-15T14:32:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927462+0000", 2023-01-15T14:32:55.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016500799999999999 2023-01-15T14:32:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.224878+0000", 2023-01-15T14:32:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29741604399999999 2023-01-15T14:32:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.224989+0000", 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011082300000000001 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225160+0000", 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017161699999999999 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225399+0000", 2023-01-15T14:32:55.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238645 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226023+0000", 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062401000000000004 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226469+0000", 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044639400000000002 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227365+0000", 2023-01-15T14:32:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00089538200000000001 2023-01-15T14:32:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429122+0000", 2023-01-15T14:32:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20175698 2023-01-15T14:32:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.533830+0000", 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104708068 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.533947+0000", 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000117218 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:55.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.534021+0000", 2023-01-15T14:32:55.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4608000000000001e-05 2023-01-15T14:32:55.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.888 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:55.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:55.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:55.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:55.888 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2207 3.5 3:a1c47f99:::smithi09025264-2:head [write 2548663~763022] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:55.888 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:32:55.888 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.100615876999996, 2023-01-15T14:32:55.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79657873300000004, 2023-01-15T14:32:55.889 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:55.889 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:56.278 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:56.278 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:56.278 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:56.278 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2207 2023-01-15T14:32:56.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.279 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:56.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:56.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:32:56.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:56.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:56.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:32:56.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:56.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:56.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837590+0000", 2023-01-15T14:32:56.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:32:56.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:56.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837960+0000", 2023-01-15T14:32:56.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037055900000000001 2023-01-15T14:32:56.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:56.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837980+0000", 2023-01-15T14:32:56.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9953e-05 2023-01-15T14:32:56.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:56.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837988+0000", 2023-01-15T14:32:56.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2800000000000003e-06 2023-01-15T14:32:56.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:56.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927512+0000", 2023-01-15T14:32:56.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089523563 2023-01-15T14:32:56.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:56.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927645+0000", 2023-01-15T14:32:56.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013285799999999999 2023-01-15T14:32:56.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:56.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225466+0000", 2023-01-15T14:32:56.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29782133100000002 2023-01-15T14:32:56.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:56.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225553+0000", 2023-01-15T14:32:56.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6866000000000005e-05 2023-01-15T14:32:56.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:56.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225708+0000", 2023-01-15T14:32:56.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015455199999999999 2023-01-15T14:32:56.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:56.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225971+0000", 2023-01-15T14:32:56.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026317099999999998 2023-01-15T14:32:56.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:56.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226027+0000", 2023-01-15T14:32:56.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6601000000000002e-05 2023-01-15T14:32:56.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:56.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226470+0000", 2023-01-15T14:32:56.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044267500000000001 2023-01-15T14:32:56.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:56.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227372+0000", 2023-01-15T14:32:56.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090215200000000003 2023-01-15T14:32:56.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:56.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429314+0000", 2023-01-15T14:32:56.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20194183499999999 2023-01-15T14:32:56.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:56.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.633999+0000", 2023-01-15T14:32:56.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20468505400000001 2023-01-15T14:32:56.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:56.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634075+0000", 2023-01-15T14:32:56.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:32:56.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:56.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634170+0000", 2023-01-15T14:32:56.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4821999999999994e-05 2023-01-15T14:32:56.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:56.292 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:56.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:56.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.293 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2208 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (56) in=7b,truncate 3395047] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:56.293 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:32:56.293 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.100193283999999, 2023-01-15T14:32:56.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79650439900000003, 2023-01-15T14:32:56.294 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:56.294 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:56.295 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:56.295 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:56.295 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:56.295 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2208 2023-01-15T14:32:56.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.296 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:56.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:56.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:32:56.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:56.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:56.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:32:56.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:56.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:56.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:32:56.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:56.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:56.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838018+0000", 2023-01-15T14:32:56.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2549999999999997e-06 2023-01-15T14:32:56.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:56.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838031+0000", 2023-01-15T14:32:56.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2717999999999999e-05 2023-01-15T14:32:56.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:56.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838036+0000", 2023-01-15T14:32:56.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2240000000000003e-06 2023-01-15T14:32:56.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:56.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927676+0000", 2023-01-15T14:32:56.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089639620000000003 2023-01-15T14:32:56.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:56.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927804+0000", 2023-01-15T14:32:56.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012832300000000001 2023-01-15T14:32:56.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:56.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226096+0000", 2023-01-15T14:32:56.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29829207400000002 2023-01-15T14:32:56.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:56.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226347+0000", 2023-01-15T14:32:56.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025084800000000002 2023-01-15T14:32:56.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:56.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226750+0000", 2023-01-15T14:32:56.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040343200000000001 2023-01-15T14:32:56.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:56.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227147+0000", 2023-01-15T14:32:56.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039682900000000002 2023-01-15T14:32:56.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227600+0000", 2023-01-15T14:32:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045321199999999998 2023-01-15T14:32:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227859+0000", 2023-01-15T14:32:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259041 2023-01-15T14:32:56.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227895+0000", 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5710999999999997e-05 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429405+0000", 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201509415 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634300+0000", 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20489528200000001 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634409+0000", 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109524 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634518+0000", 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108548 2023-01-15T14:32:56.494 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:56.495 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:56.495 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:56.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.495 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2209 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:56.495 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:32:56.495 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.100164469000006, 2023-01-15T14:32:56.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79724995499999995, 2023-01-15T14:32:56.496 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:56.496 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:56.496 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:56.496 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2209 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:56.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838044+0000", 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6120000000000001e-06 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:56.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838056+0000", 2023-01-15T14:32:56.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1236000000000001e-05 2023-01-15T14:32:56.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:56.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838059+0000", 2023-01-15T14:32:56.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6940000000000001e-06 2023-01-15T14:32:56.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:56.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.027908+0000", 2023-01-15T14:32:56.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189848287 2023-01-15T14:32:56.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:56.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.028049+0000", 2023-01-15T14:32:56.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014132500000000001 2023-01-15T14:32:56.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:56.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227252+0000", 2023-01-15T14:32:56.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19920324 2023-01-15T14:32:56.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:56.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227341+0000", 2023-01-15T14:32:56.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8683000000000005e-05 2023-01-15T14:32:56.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:56.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634637+0000", 2023-01-15T14:32:56.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729567300000002 2023-01-15T14:32:56.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:56.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634980+0000", 2023-01-15T14:32:56.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034335000000000002 2023-01-15T14:32:56.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:56.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.635292+0000", 2023-01-15T14:32:56.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003123 2023-01-15T14:32:56.504 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:56.504 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:56.504 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:56.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.505 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2222 3.c 3:3137f0cb:::smithi09025264-20:head [write 3259733~152121] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:56.505 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:32:56.505 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.080312910000004, 2023-01-15T14:32:56.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77651572400000002, 2023-01-15T14:32:56.505 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:56.506 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:56.506 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:56.506 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:56.506 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:56.506 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2222 2023-01-15T14:32:56.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.506 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:56.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857891+0000", 2023-01-15T14:32:56.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:32:56.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:56.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857993+0000", 2023-01-15T14:32:56.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010276399999999999 2023-01-15T14:32:56.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:56.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858021+0000", 2023-01-15T14:32:56.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8022e-05 2023-01-15T14:32:56.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:56.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858034+0000", 2023-01-15T14:32:56.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2549e-05 2023-01-15T14:32:56.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:56.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025371+0000", 2023-01-15T14:32:56.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16733653100000001 2023-01-15T14:32:56.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025429+0000", 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8922000000000003e-05 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226869+0000", 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20143931100000001 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227088+0000", 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219328 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:56.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227443+0000", 2023-01-15T14:32:56.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035481 2023-01-15T14:32:56.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:56.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227830+0000", 2023-01-15T14:32:56.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000386702 2023-01-15T14:32:56.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:56.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227861+0000", 2023-01-15T14:32:56.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1584000000000001e-05 2023-01-15T14:32:56.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:56.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227939+0000", 2023-01-15T14:32:56.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7354999999999994e-05 2023-01-15T14:32:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228548+0000", 2023-01-15T14:32:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060954800000000001 2023-01-15T14:32:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429817+0000", 2023-01-15T14:32:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20126904600000001 2023-01-15T14:32:56.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634125+0000", 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20430780300000001 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634259+0000", 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134144 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:56.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634410+0000", 2023-01-15T14:32:56.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150447 2023-01-15T14:32:56.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:56.696 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:56.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:56.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.696 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2223 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (57) in=7b,truncate 3411854] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:56.696 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:32:56.696 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.080154123, 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87671107599999998, 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2223 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:56.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858052+0000", 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:56.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:56.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858056+0000", 2023-01-15T14:32:56.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5619999999999997e-06 2023-01-15T14:32:56.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:56.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858067+0000", 2023-01-15T14:32:56.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0328e-05 2023-01-15T14:32:56.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:56.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858071+0000", 2023-01-15T14:32:56.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6439999999999999e-06 2023-01-15T14:32:56.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:56.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025446+0000", 2023-01-15T14:32:56.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16737407200000001 2023-01-15T14:32:56.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025503+0000", 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7952999999999999e-05 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227970+0000", 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202466866 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:56.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228139+0000", 2023-01-15T14:32:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016889100000000001 2023-01-15T14:32:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:32:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228284+0000", 2023-01-15T14:32:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014493700000000001 2023-01-15T14:32:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228470+0000", 2023-01-15T14:32:56.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185541 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228481+0000", 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.167e-05 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228537+0000", 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5912999999999998e-05 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:56.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228563+0000", 2023-01-15T14:32:56.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5613e-05 2023-01-15T14:32:56.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:56.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.433181+0000", 2023-01-15T14:32:56.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20461838099999999 2023-01-15T14:32:56.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:56.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734620+0000", 2023-01-15T14:32:56.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301438386 2023-01-15T14:32:56.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:56.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734682+0000", 2023-01-15T14:32:56.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2373999999999996e-05 2023-01-15T14:32:56.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:56.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:56.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:56.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734764+0000", 2023-01-15T14:32:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1656999999999994e-05 2023-01-15T14:32:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:57.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2224 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 75.080129072999995, 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97704279199999999, 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2224 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.100 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:57.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:32:57.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:32:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858077+0000", 2023-01-15T14:32:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:32:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858080+0000", 2023-01-15T14:32:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:32:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:57.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858089+0000", 2023-01-15T14:32:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3449999999999995e-06 2023-01-15T14:32:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858093+0000", 2023-01-15T14:32:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9820000000000002e-06 2023-01-15T14:32:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:57.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025520+0000", 2023-01-15T14:32:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.167426556 2023-01-15T14:32:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025594+0000", 2023-01-15T14:32:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4339999999999996e-05 2023-01-15T14:32:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:57.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228608+0000", 2023-01-15T14:32:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20301395699999999 2023-01-15T14:32:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:32:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228752+0000", 2023-01-15T14:32:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014385 2023-01-15T14:32:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834878+0000", 2023-01-15T14:32:57.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60612591500000002 2023-01-15T14:32:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834987+0000", 2023-01-15T14:32:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108733 2023-01-15T14:32:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.835120+0000", 2023-01-15T14:32:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133688 2023-01-15T14:32:57.111 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2808 3.22 3:47dfe935:::smithi09025264-29:head [write 689186~799458] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e268)", 2023-01-15T14:32:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:34.952339+0000", 2023-01-15T14:32:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.985867860999999, 2023-01-15T14:32:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70634859500000002, 2023-01-15T14:32:57.112 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:32:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:32:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:32:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:32:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:32:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2808 2023-01-15T14:32:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:32:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:32:57.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:34.952339+0000", 2023-01-15T14:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:34.952339+0000", 2023-01-15T14:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:34.952336+0000", 2023-01-15T14:32:57.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:32:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:32:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:34.952793+0000", 2023-01-15T14:32:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045684300000000002 2023-01-15T14:32:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:32:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:34.952815+0000", 2023-01-15T14:32:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1869e-05 2023-01-15T14:32:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:32:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:34.952825+0000", 2023-01-15T14:32:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9969999999999993e-06 2023-01-15T14:32:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:34.952885+0000", 2023-01-15T14:32:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0132e-05 2023-01-15T14:32:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:32:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:34.952891+0000", 2023-01-15T14:32:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0979999999999999e-06 2023-01-15T14:32:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:32:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:35.351027+0000", 2023-01-15T14:32:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39813578799999999 2023-01-15T14:32:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:32:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:35.351247+0000", 2023-01-15T14:32:57.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000220542 2023-01-15T14:32:57.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:32:57.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:35.351690+0000", 2023-01-15T14:32:57.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044308100000000001 2023-01-15T14:32:57.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:32:57.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:35.353000+0000", 2023-01-15T14:32:57.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001309354 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:35.353060+0000", 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9967999999999999e-05 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:35.353470+0000", 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041066799999999999 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:35.354108+0000", 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063795400000000004 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:35.367928+0000", 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.01382012 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:35.658519+0000", 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.290591194 2023-01-15T14:32:57.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:35.658609+0000", 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9781e-05 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:35.658687+0000", 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8249000000000002e-05 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:57.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:57.513 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:32:57.513 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:32:57.514 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:57.514 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:32:57.515 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:54.021+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:57.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:32:57.516 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:32:57.518 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:57.516+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:57.524 INFO:tasks.rados.rados.0.smithi090.stdout:1157: expect (ObjNum 407 snap 108 seq_num 407) 2023-01-15T14:32:57.524 INFO:tasks.rados.rados.0.smithi090.stdout:1159: finishing write tid 1 to smithi09025264-25 2023-01-15T14:32:57.524 INFO:tasks.rados.rados.0.smithi090.stdout:1159: finishing write tid 2 to smithi09025264-25 2023-01-15T14:32:57.524 INFO:tasks.rados.rados.0.smithi090.stdout:1159: finishing write tid 3 to smithi09025264-25 2023-01-15T14:32:57.524 INFO:tasks.rados.rados.0.smithi090.stdout:1159: finishing write tid 4 to smithi09025264-25 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:1159: finishing write tid 5 to smithi09025264-25 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:1159: oid 25 updating version 0 to 304 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:1159: oid 25 updating version 304 to 307 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:1159: oid 25 version 307 is already newer than 306 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:1159: oid 25 version 307 is already newer than 305 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 307 (ObjNum 430 snap 116 seq_num 430) dirty exists 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:1159: left oid 25 (ObjNum 430 snap 116 seq_num 430) 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:1154: done (5 left) 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:1156: done (4 left) 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:1157: done (3 left) 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:1159: done (2 left) 2023-01-15T14:32:57.525 INFO:tasks.rados.rados.0.smithi090.stdout:1162: done (1 left) 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout:1163: snap_create 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout:1161: finishing copy_from to smithi09025264-31 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 96 (ObjNum 326 snap 87 seq_num 326) dirty exists 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout:1161: finishing copy_from racing read to smithi09025264-31 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout:1161: done (0 left) 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout:1163: done (0 left) 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout:1164: delete oid 37 current snap is 117 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout:1164: done (0 left) 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout:1165: read oid 28 snap 110 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout:1165: expect (ObjNum 352 snap 93 seq_num 352) 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout:1166: snap_create 2023-01-15T14:32:57.526 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:57.527 INFO:tasks.rados.rados.0.smithi090.stdout:1165: expect (ObjNum 352 snap 93 seq_num 352) 2023-01-15T14:32:57.527 INFO:tasks.rados.rados.0.smithi090.stdout:1165: done (0 left) 2023-01-15T14:32:57.527 INFO:tasks.rados.rados.0.smithi090.stdout:1166: done (0 left) 2023-01-15T14:32:57.527 INFO:tasks.rados.rados.0.smithi090.stdout:1167: delete oid 44 current snap is 118 2023-01-15T14:32:57.527 INFO:tasks.rados.rados.0.smithi090.stdout:1167: done (0 left) 2023-01-15T14:32:57.527 INFO:tasks.rados.rados.0.smithi090.stdout:1168: rollback oid 32 current snap is 118 2023-01-15T14:32:57.527 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 32 to 117 2023-01-15T14:32:57.527 INFO:tasks.rados.rados.0.smithi090.stdout:1169: write (excl) oid 24 current snap is 118 2023-01-15T14:32:57.527 INFO:tasks.rados.rados.0.smithi090.stdout:1169: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2023-01-15T14:32:57.527 INFO:tasks.rados.rados.0.smithi090.stdout:1169: writing smithi09025264-24 from 494238 to 984425 tid 1 2023-01-15T14:32:57.527 INFO:tasks.rados.rados.0.smithi090.stdout:1169: writing smithi09025264-24 from 1563538 to 2265357 tid 2 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:1169: writing smithi09025264-24 from 2818520 to 3294238 tid 3 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:1170: write (excl) oid 12 current snap is 118 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:1169: finishing write tid 1 to smithi09025264-24 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:1170: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:1170: writing smithi09025264-12 from 511045 to 1076481 tid 1 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:1170: writing smithi09025264-12 from 1622020 to 2067497 tid 2 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:1170: writing smithi09025264-12 from 2729590 to 3286445 tid 3 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:1171: copy_from oid 37 from oid 3 current snap is 118 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:1172: rollback oid 6 current snap is 118 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 6 to 110 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:1173: write oid 48 current snap is 118 2023-01-15T14:32:57.528 INFO:tasks.rados.rados.0.smithi090.stdout:1173: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:1173: writing smithi09025264-48 from 527852 to 1168537 tid 1 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:1173: writing smithi09025264-48 from 1680502 to 2269637 tid 2 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:1173: writing smithi09025264-48 from 2757013 to 3327852 tid 3 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:1174: write oid 47 current snap is 118 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:1174: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:1174: writing smithi09025264-47 from 544659 to 1260593 tid 1 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:1174: writing smithi09025264-47 from 1738984 to 2188130 tid 2 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:1174: writing smithi09025264-47 from 2784436 to 3344659 tid 3 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:1175: snap_create 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:1172: finishing rollback tid 1 to smithi09025264-6 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 0 (ObjNum 33 snap 0 seq_num 3182657928) dirty dne 2023-01-15T14:32:57.529 INFO:tasks.rados.rados.0.smithi090.stdout:1172: done (6 left) 2023-01-15T14:32:57.530 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:32:57.530 INFO:tasks.rados.rados.0.smithi090.stdout:1170: finishing write tid 1 to smithi09025264-12 2023-01-15T14:32:57.530 INFO:tasks.rados.rados.0.smithi090.stdout:1170: finishing write tid 2 to smithi09025264-12 2023-01-15T14:32:57.530 INFO:tasks.rados.rados.0.smithi090.stdout:1170: finishing write tid 3 to smithi09025264-12 2023-01-15T14:32:57.530 INFO:tasks.rados.rados.0.smithi090.stdout:1170: finishing write tid 4 to smithi09025264-12 2023-01-15T14:32:57.530 INFO:tasks.rados.rados.0.smithi090.stdout:1170: finishing write tid 5 to smithi09025264-12 2023-01-15T14:32:57.672 INFO:tasks.rados.rados.0.smithi090.stdout:1170: oid 12 updating version 0 to 284 2023-01-15T14:32:57.672 INFO:tasks.rados.rados.0.smithi090.stdout:1170: oid 12 updating version 284 to 285 2023-01-15T14:32:57.672 INFO:tasks.rados.rados.0.smithi090.stdout:1170: oid 12 updating version 285 to 286 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout:1170: oid 12 version 286 is already newer than 283 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 286 (ObjNum 435 snap 118 seq_num 435) dirty exists 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout:1170: left oid 12 (ObjNum 435 snap 118 seq_num 435) 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout:1170: done (5 left) 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout:1168: finishing rollback tid 0 to smithi09025264-32 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout:1168: finishing rollback tid 1 to smithi09025264-32 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout:1168: finishing rollback tid 2 to smithi09025264-32 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 271 (ObjNum 404 snap 108 seq_num 404) dirty exists 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout:1168: done (4 left) 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:32:57.673 INFO:tasks.rados.rados.0.smithi090.stdout:1173: finishing write tid 1 to smithi09025264-48 2023-01-15T14:32:57.674 INFO:tasks.rados.rados.0.smithi090.stdout:1171: finishing copy_from to smithi09025264-37 2023-01-15T14:32:57.674 INFO:tasks.rados.rados.0.smithi090.stdout:1171: got expected ENOENT (src dne) 2023-01-15T14:32:57.674 INFO:tasks.rados.rados.0.smithi090.stdout:1171: finishing copy_from racing read to smithi09025264-37 2023-01-15T14:32:57.674 INFO:tasks.rados.rados.0.smithi090.stdout:1171: done (3 left) 2023-01-15T14:32:57.674 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:32:57.674 INFO:tasks.rados.rados.0.smithi090.stdout:1173: finishing write tid 2 to smithi09025264-48 2023-01-15T14:32:57.674 INFO:tasks.rados.rados.0.smithi090.stdout:1173: finishing write tid 3 to smithi09025264-48 2023-01-15T14:32:57.674 INFO:tasks.rados.rados.0.smithi090.stdout:1173: finishing write tid 4 to smithi09025264-48 2023-01-15T14:32:57.674 INFO:tasks.rados.rados.0.smithi090.stdout:1173: finishing write tid 5 to smithi09025264-48 2023-01-15T14:32:57.674 INFO:tasks.rados.rados.0.smithi090.stdout:1173: oid 48 updating version 0 to 224 2023-01-15T14:32:57.674 INFO:tasks.rados.rados.0.smithi090.stdout:1173: oid 48 version 224 is already newer than 222 2023-01-15T14:32:57.675 INFO:tasks.rados.rados.0.smithi090.stdout:1173: oid 48 version 224 is already newer than 223 2023-01-15T14:32:57.675 INFO:tasks.rados.rados.0.smithi090.stdout:1173: oid 48 version 224 is already newer than 221 2023-01-15T14:32:57.675 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 224 (ObjNum 436 snap 118 seq_num 436) dirty exists 2023-01-15T14:32:57.675 INFO:tasks.rados.rados.0.smithi090.stdout:1173: left oid 48 (ObjNum 436 snap 118 seq_num 436) 2023-01-15T14:32:57.675 INFO:tasks.rados.rados.0.smithi090.stdout:1173: done (2 left) 2023-01-15T14:32:57.675 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:32:57.675 INFO:tasks.rados.rados.0.smithi090.stdout:1169: finishing write tid 2 to smithi09025264-24 2023-01-15T14:32:57.675 INFO:tasks.rados.rados.0.smithi090.stdout:1169: finishing write tid 3 to smithi09025264-24 2023-01-15T14:32:57.675 INFO:tasks.rados.rados.0.smithi090.stdout:1169: finishing write tid 4 to smithi09025264-24 2023-01-15T14:32:57.675 INFO:tasks.rados.rados.0.smithi090.stdout:1174: finishing write tid 1 to smithi09025264-47 2023-01-15T14:32:57.675 INFO:tasks.rados.rados.0.smithi090.stdout:1174: finishing write tid 2 to smithi09025264-47 2023-01-15T14:32:57.676 INFO:tasks.rados.rados.0.smithi090.stdout:1174: finishing write tid 3 to smithi09025264-47 2023-01-15T14:32:57.676 INFO:tasks.rados.rados.0.smithi090.stdout:1169: finishing write tid 5 to smithi09025264-24 2023-01-15T14:32:57.676 INFO:tasks.rados.rados.0.smithi090.stdout:1169: oid 24 updating version 0 to 262 2023-01-15T14:32:57.676 INFO:tasks.rados.rados.0.smithi090.stdout:1169: oid 24 version 262 is already newer than 260 2023-01-15T14:32:57.676 INFO:tasks.rados.rados.0.smithi090.stdout:1169: oid 24 version 262 is already newer than 261 2023-01-15T14:32:57.676 INFO:tasks.rados.rados.0.smithi090.stdout:1169: oid 24 version 262 is already newer than 259 2023-01-15T14:32:57.676 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 262 (ObjNum 434 snap 118 seq_num 434) dirty exists 2023-01-15T14:32:57.676 INFO:tasks.rados.rados.0.smithi090.stdout:1169: left oid 24 (ObjNum 434 snap 118 seq_num 434) 2023-01-15T14:32:57.676 INFO:tasks.rados.rados.0.smithi090.stdout:1169: done (1 left) 2023-01-15T14:32:57.676 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:32:57.676 INFO:tasks.rados.rados.0.smithi090.stdout:1174: finishing write tid 4 to smithi09025264-47 2023-01-15T14:32:57.677 INFO:tasks.rados.rados.0.smithi090.stdout:1174: finishing write tid 5 to smithi09025264-47 2023-01-15T14:32:57.677 INFO:tasks.rados.rados.0.smithi090.stdout:1174: oid 47 updating version 0 to 343 2023-01-15T14:32:57.677 INFO:tasks.rados.rados.0.smithi090.stdout:1174: oid 47 updating version 343 to 344 2023-01-15T14:32:57.677 INFO:tasks.rados.rados.0.smithi090.stdout:1174: oid 47 version 344 is already newer than 342 2023-01-15T14:32:57.677 INFO:tasks.rados.rados.0.smithi090.stdout:1174: oid 47 updating version 344 to 345 2023-01-15T14:32:57.677 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 345 (ObjNum 437 snap 118 seq_num 437) dirty exists 2023-01-15T14:32:57.677 INFO:tasks.rados.rados.0.smithi090.stdout:1174: left oid 47 (ObjNum 437 snap 118 seq_num 437) 2023-01-15T14:32:57.677 INFO:tasks.rados.rados.0.smithi090.stdout:1174: done (0 left) 2023-01-15T14:32:57.679 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:57.679 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:32:57.679 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:32:57.679 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:57.679 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:32:57.680 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:32:57.676+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:32:57.779 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:57.776+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:57.905 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:32:57.906 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:32:57.906 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:32:57.906 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:32:57.907 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:32:58.066 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:32:58.066 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:32:58.066 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:32:58.066 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:32:58.067 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:32:58.067 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:57.900+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:58.068 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:32:58.068 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:32:58.069 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:32:58.069 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:32:58.069 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:32:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:32:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:32:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:32:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:32:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:32:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:32:58.070 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:32:58.070 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:58.061+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1175: done (0 left) 2023-01-15T14:32:58.224 INFO:tasks.rados.rados.0.smithi090.stdout:1176: copy_from oid 6 from oid 33 current snap is 119 2023-01-15T14:32:58.224 INFO:tasks.rados.rados.0.smithi090.stdout:1177: snap_remove snap 110 2023-01-15T14:32:58.226 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:58.226 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:58.226 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:32:58.227 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:32:58.227 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:58.224+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:58.290 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:32:58.570 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:32:58.570 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2023-01-15T14:32:58.570 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-01-15T14:32:58.570 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-01-15T14:32:58.571 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2023-01-15T14:32:58.573 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:58.573 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:58.573 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:32:58.574 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:58.565+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:58.673 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:58.665+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:58.733 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:58.733 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:58.733 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:32:58.733 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-01-15T14:32:58.733 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:32:58.734 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:32:58.781 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:58.780+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:58.891 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:58.891 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:58.891 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:32:58.892 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:58.885+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:58.992 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:58.989+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:32:59.051 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:59.051 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:59.052 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:32:59.101 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:59.100+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:59.213 INFO:tasks.rados.rados.0.smithi090.stdout:1177: done (1 left) 2023-01-15T14:32:59.213 INFO:tasks.rados.rados.0.smithi090.stdout:1178: snap_remove snap 113 2023-01-15T14:32:59.214 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:59.215 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:59.215 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:32:59.215 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:32:59.212+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:32:59.314 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:59.305+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:59.373 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:59.373 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:59.373 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:32:59.415 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:32:59.412+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:32:59.557 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:59.557 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:59.558 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:32:59.558 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:32:59.552+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:32:59.657 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:32:59.649+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:32:59.763 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:59.763 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:59.763 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:32:59.764 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:32:59.757+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:32:59.864 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:32:59.857+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:32:59.926 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:32:59.926 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:32:59.926 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:32:59.979 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:32:59.973+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:00.088 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:33:00.089 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:00.088+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:00.090 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:00.090 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:00.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:33:00.109 INFO:tasks.rados.rados.0.smithi090.stdout:1178: done (1 left) 2023-01-15T14:33:00.398 INFO:tasks.rados.rados.0.smithi090.stdout:1179: read oid 20 snap -1 2023-01-15T14:33:00.398 INFO:tasks.rados.rados.0.smithi090.stdout:1179: expect (ObjNum 380 snap 103 seq_num 380) 2023-01-15T14:33:00.398 INFO:tasks.rados.rados.0.smithi090.stdout:1180: snap_remove snap 117 2023-01-15T14:33:00.399 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:33:00.401 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:00.401 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:00.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:33:00.403 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:00.403 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:00.403 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:00.404 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:00.393+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:00.501 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:00.493+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:00.587 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:33:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.48859446699998, 2023-01-15T14:33:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:33:00.587 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.588 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:33:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:33:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:33:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:33:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:33:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:33:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:33:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:33:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:33:00.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:33:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:33:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:33:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:33:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:33:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:33:00.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:33:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:33:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:33:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:33:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:33:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:33:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:33:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:33:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:33:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:33:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:33:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:33:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:33:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:33:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:33:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:33:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:33:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:33:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.48831421900002, 2023-01-15T14:33:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:33:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:33:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:33:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:33:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:33:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:33:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:33:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:33:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:33:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:33:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:33:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:33:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:33:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:33:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:33:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:33:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:33:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:33:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:33:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:33:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:33:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:33:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:33:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:33:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:33:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:33:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:33:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:33:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:33:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:33:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:33:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:33:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:33:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:33:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:33:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:33:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:33:00.616 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:33:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.48828160199997, 2023-01-15T14:33:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:33:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:00.617 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:33:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:33:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:33:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:33:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:33:00.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:33:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:33:00.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:33:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:33:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:33:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:33:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:33:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:33:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:33:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:33:00.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:33:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:33:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:33:00.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:33:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:33:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:33:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:33:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:33:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:33:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:33:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.386963616, 2023-01-15T14:33:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:33:00.630 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.631 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:33:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:33:00.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:33:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:33:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:33:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:33:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:33:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:33:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:33:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:33:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:33:00.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:33:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:33:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:33:00.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:33:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:33:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:33:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:33:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:00.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:33:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:33:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:33:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:33:00.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:33:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:33:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:33:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:33:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:00.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:33:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:33:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:33:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:33:00.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:33:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:33:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:33:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:33:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:33:00.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:33:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:00.647 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:33:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.386933613, 2023-01-15T14:33:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:33:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.648 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:33:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:33:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:33:00.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:33:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:33:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:33:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:33:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:33:00.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 273.87192504799998, 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.655 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:33:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:33:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:33:00.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:33:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:33:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:33:00.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:33:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:33:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:33:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:33:00.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:33:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:33:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:33:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:33:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:33:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:33:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:33:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:33:00.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:33:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:33:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:33:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:33:00.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:33:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:33:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 273.871442694, 2023-01-15T14:33:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:33:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:33:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:33:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:33:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:33:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:33:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:33:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:33:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:33:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:33:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:33:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:33:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:33:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:33:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:33:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:33:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:33:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:33:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:33:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:33:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:33:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:33:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:33:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:33:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:33:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:33:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:33:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 273.871390131, 2023-01-15T14:33:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:33:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:33:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:33:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:33:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:33:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:33:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:33:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:33:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:33:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:33:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:33:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:33:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:33:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:33:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:33:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:33:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:33:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 260.50018842700001, 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.795 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:00.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.796 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.796 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:33:00.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.796 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:33:00.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:33:00.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:33:00.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:33:00.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:33:00.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:33:00.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:33:00.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:33:00.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:33:00.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:33:00.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:33:00.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:00.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:33:00.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:33:00.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:00.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:33:00.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:33:00.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:00.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:33:00.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:33:00.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:00.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:33:00.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:33:00.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:00.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:33:00.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:33:00.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:33:00.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:33:00.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:33:00.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:33:00.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:00.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:33:00.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:33:00.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:33:00.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:33:00.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.807 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.807 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:00.807 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:33:00.807 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 260.50007175299999, 2023-01-15T14:33:00.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:33:00.808 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.808 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:00.808 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.808 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.808 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.808 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:33:00.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.809 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:33:00.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:33:00.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:33:00.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:00.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:33:00.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:33:00.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:33:00.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:33:00.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:33:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:33:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:33:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:33:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:33:00.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:33:00.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:33:00.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:33:00.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:33:00.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:33:00.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:33:00.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:33:00.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.816 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.817 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:33:00.817 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:33:00.817 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 252.773939477, 2023-01-15T14:33:00.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:33:00.817 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.818 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:00.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.818 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.818 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:33:00.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.818 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:33:00.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:33:00.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 216.557689418, 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:33:00.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.824 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:00.824 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.824 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.824 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.824 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:33:00.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.824 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:00.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:33:00.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:00.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:33:00.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:00.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:33:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:33:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:33:00.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:33:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:33:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:33:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:33:00.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:33:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:33:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:33:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:33:00.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:33:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:33:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:33:00.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:33:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:00.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:33:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:33:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:33:00.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:33:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:33:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:33:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:33:00.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:33:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:33:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:33:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:33:00.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:33:00.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:00.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:33:00.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:33:00.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:00.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:33:00.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:33:00.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:00.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:33:00.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:33:00.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:00.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:00.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:00.998 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:33:00.998 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:00.998 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 216.55765585099999, 2023-01-15T14:33:00.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:33:00.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:00.999 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:00.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:00.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:00.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:00.999 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:33:00.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:00.999 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:00.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:01.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:01.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:01.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:01.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:01.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:01.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:01.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:01.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:33:01.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:33:01.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:01.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:33:01.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:33:01.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:01.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:33:01.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:33:01.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:33:01.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:33:01.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:01.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:33:01.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:33:01.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:33:01.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:33:01.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:01.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:33:01.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:33:01.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:01.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:33:01.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:33:01.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:01.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:33:01.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:33:01.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:01.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:33:01.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:33:01.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:01.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:33:01.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:33:01.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:01.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:33:01.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:33:01.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:33:01.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:33:01.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:33:01.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:33:01.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:01.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:33:01.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:33:01.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:01.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:33:01.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:33:01.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.011 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:01.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.011 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:33:01.011 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:33:01.012 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 197.79175230499999, 2023-01-15T14:33:01.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:33:01.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:01.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:01.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:01.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:01.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:01.012 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:33:01.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.012 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:01.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:01.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:33:01.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:01.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:33:01.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:01.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:33:01.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:01.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:01.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:33:01.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:33:01.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:01.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:33:01.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:33:01.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:01.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:33:01.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:33:01.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:33:01.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:33:01.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:01.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:33:01.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:33:01.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:33:01.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:33:01.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:01.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:33:01.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:33:01.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:01.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:33:01.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:33:01.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:01.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:33:01.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:33:01.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:01.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:33:01.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:33:01.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:01.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:33:01.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:33:01.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:01.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:33:01.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:33:01.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:33:01.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:33:01.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:33:01.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:33:01.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:01.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:33:01.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:33:01.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:01.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:33:01.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:33:01.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.023 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:01.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.024 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:33:01.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:33:01.024 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 181.248573505, 2023-01-15T14:33:01.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:33:01.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:01.024 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:01.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:01.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:01.025 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:01.025 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:33:01.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.025 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:01.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:01.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:33:01.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:01.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:33:01.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:33:01.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:33:01.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:33:01.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:33:01.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:33:01.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:33:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:33:01.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:33:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:33:01.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:33:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:33:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:33:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 181.24853433000001, 2023-01-15T14:33:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:33:01.196 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:33:01.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:33:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:33:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:33:01.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:33:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:33:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:33:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:33:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:33:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:33:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:33:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:33:01.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:33:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:33:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:33:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:33:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:33:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:33:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1373 3.12 3:495d4bbc:::smithi09025264-16:head [write 2873523~591733] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.85641800400001, 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40949580099999999, 2023-01-15T14:33:01.206 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1373 2023-01-15T14:33:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:33:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:33:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693440+0000", 2023-01-15T14:33:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693808+0000", 2023-01-15T14:33:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000367742 2023-01-15T14:33:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:01.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693831+0000", 2023-01-15T14:33:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3213e-05 2023-01-15T14:33:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693846+0000", 2023-01-15T14:33:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4909000000000001e-05 2023-01-15T14:33:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.695833+0000", 2023-01-15T14:33:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019869240000000002 2023-01-15T14:33:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696121+0000", 2023-01-15T14:33:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028753300000000003 2023-01-15T14:33:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798577+0000", 2023-01-15T14:33:01.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102456808 2023-01-15T14:33:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798849+0000", 2023-01-15T14:33:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027138999999999999 2023-01-15T14:33:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:33:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799155+0000", 2023-01-15T14:33:01.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030594800000000001 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799614+0000", 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045960700000000001 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799737+0000", 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012218900000000001 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799998+0000", 2023-01-15T14:33:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026129700000000002 2023-01-15T14:33:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800639+0000", 2023-01-15T14:33:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064108800000000005 2023-01-15T14:33:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102023+0000", 2023-01-15T14:33:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30138417200000001 2023-01-15T14:33:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102198+0000", 2023-01-15T14:33:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017453600000000001 2023-01-15T14:33:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:01.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102398+0000", 2023-01-15T14:33:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020078000000000001 2023-01-15T14:33:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102937+0000", 2023-01-15T14:33:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053874499999999998 2023-01-15T14:33:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:33:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:33:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.85599341, 2023-01-15T14:33:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:33:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:01.217 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:33:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:33:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:01.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:33:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:33:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:01.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:33:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:33:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:33:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:33:01.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:33:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:33:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:33:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:33:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:01.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:33:01.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:01.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:33:01.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:33:01.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:01.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:33:01.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:33:01.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:01.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:33:01.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:33:01.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:33:01.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:33:01.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:33:01.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:33:01.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:01.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:33:01.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:33:01.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:01.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:33:01.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:33:01.403 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.404 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:01.404 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.404 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:33:01.404 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:33:01.404 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.855941081, 2023-01-15T14:33:01.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:33:01.404 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:01.404 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:01.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:01.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:01.405 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:01.405 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:33:01.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.406 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:01.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:01.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:33:01.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:01.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:33:01.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:01.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:33:01.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:01.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:01.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:33:01.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:33:01.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:01.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:33:01.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:33:01.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:01.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:33:01.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:33:01.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:33:01.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:33:01.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:01.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:33:01.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:33:01.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:33:01.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:33:01.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:01.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:33:01.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:33:01.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:33:01.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:33:01.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:01.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:33:01.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:33:01.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:01.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:33:01.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:33:01.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.412 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:01.412 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.412 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 154.259027882, 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:33:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:33:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:33:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:33:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:33:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:33:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:33:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:33:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:33:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:33:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:33:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:33:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:33:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:33:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:33:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:33:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:01.419 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:01.419 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:33:01.421 INFO:tasks.rados.rados.0.smithi090.stdout:1180: done (2 left) 2023-01-15T14:33:01.421 INFO:tasks.rados.rados.0.smithi090.stdout:1181: read oid 37 snap -1 2023-01-15T14:33:01.421 INFO:tasks.rados.rados.0.smithi090.stdout:1181: expect deleted 2023-01-15T14:33:01.421 INFO:tasks.rados.rados.0.smithi090.stdout:1182: delete oid 19 current snap is 119 2023-01-15T14:33:01.421 INFO:tasks.rados.rados.0.smithi090.stdout:1176: finishing copy_from to smithi09025264-6 2023-01-15T14:33:01.421 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 113 (ObjNum 378 snap 102 seq_num 378) dirty exists 2023-01-15T14:33:01.421 INFO:tasks.rados.rados.0.smithi090.stdout:1176: finishing copy_from racing read to smithi09025264-6 2023-01-15T14:33:01.422 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:33:01.423 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:01.413+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:01.429 INFO:tasks.rados.rados.0.smithi090.stdout:1176: done (3 left) 2023-01-15T14:33:01.429 INFO:tasks.rados.rados.0.smithi090.stdout:1181: done (2 left) 2023-01-15T14:33:01.429 INFO:tasks.rados.rados.0.smithi090.stdout:1182: done (1 left) 2023-01-15T14:33:01.429 INFO:tasks.rados.rados.0.smithi090.stdout:1183: snap_remove snap 115 2023-01-15T14:33:01.522 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:01.513+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:01.610 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:01.610 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:01.611 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:33:01.611 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:01.611 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:33:01.623 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:01.620+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:01.721 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:01.775 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:01.776 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:33:01.776 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:33:01.776 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:01.777 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:33:01.778 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:01.776+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:01.863 INFO:tasks.rados.rados.0.smithi090.stdout:1183: done (1 left) 2023-01-15T14:33:01.939 INFO:tasks.rados.rados.0.smithi090.stdout:1184: snap_remove snap 108 2023-01-15T14:33:01.941 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:01.941 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:01.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:01.941 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:01.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.942 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:01.942 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:01.942 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.80150873299999, 2023-01-15T14:33:01.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:33:01.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:01.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:01.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:01.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:01.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:01.942 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:33:01.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.943 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:01.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:01.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:01.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:01.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:01.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:01.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:01.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:01.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:01.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:33:01.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:33:01.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:01.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:33:01.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:33:01.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:01.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:33:01.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:33:01.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:33:01.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:33:01.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:01.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:33:01.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:33:01.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:01.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:33:01.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:33:01.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:01.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:33:01.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:33:01.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:33:01.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:33:01.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:33:01.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:01.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:33:01.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:33:01.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:01.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:33:01.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:33:01.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:01.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:33:01.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:33:01.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:01.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:33:01.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:33:01.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:33:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:33:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:33:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:33:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:01.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.968 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:33:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.80120893600002, 2023-01-15T14:33:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:33:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:33:01.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:33:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:33:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:01.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:33:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:33:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:33:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:01.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:01.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:01.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:33:02.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:33:02.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:33:02.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:33:02.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:33:02.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:33:02.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:33:02.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:33:02.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:33:02.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:33:02.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:02.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:33:02.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:33:02.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:33:02.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:02.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:33:02.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:33:02.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:33:02.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:33:02.092 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.092 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.092 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.093 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:02.093 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:33:02.093 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.80117915199997, 2023-01-15T14:33:02.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:33:02.093 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.093 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:02.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:33:02.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:33:02.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:02.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:33:02.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:33:02.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:33:02.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:33:02.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:33:02.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:33:02.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:33:02.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:02.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:33:02.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:33:02.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:33:02.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:33:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:33:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:33:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:33:02.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:33:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:33:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.671370352, 2023-01-15T14:33:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:33:02.125 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:33:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:33:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:02.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:33:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:33:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:33:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:33:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:02.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:33:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:33:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:33:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:33:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:33:02.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:33:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:33:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:33:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:33:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:33:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:33:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:33:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:33:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:33:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:33:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:33:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:33:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:33:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:33:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:33:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:33:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:33:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:33:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:33:02.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:33:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:33:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.67131365500001, 2023-01-15T14:33:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:33:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.136 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:33:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:33:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:33:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:33:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:33:02.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:33:02.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 272.05481748199998, 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:02.143 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:33:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:33:02.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:33:02.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:33:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:33:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:33:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:33:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:33:02.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.150 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.150 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:33:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:33:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 265.02596901099997, 2023-01-15T14:33:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:33:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.281 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:33:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:33:02.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:33:02.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:33:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:33:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:33:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:33:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:33:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:33:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:33:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:33:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:33:02.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:33:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:02.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:33:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:33:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:33:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:33:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:33:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:33:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:33:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:33:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:33:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:33:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:33:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:33:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:33:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 235.02468203500001, 2023-01-15T14:33:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:33:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:33:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:33:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:33:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:33:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:33:02.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:33:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:33:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:33:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:33:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:33:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:33:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:33:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:33:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:33:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:02.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:33:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:33:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:33:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:33:02.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:33:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:33:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:33:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:33:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:33:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:33:02.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:33:02.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:33:02.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:33:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:33:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:33:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:33:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:33:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:33:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 187.96410378499999, 2023-01-15T14:33:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:33:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:33:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:33:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:33:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:33:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:33:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:33:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:33:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:33:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:33:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:33:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:33:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:33:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:33:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:33:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:33:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:33:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:33:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:33:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:33:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:33:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:33:02.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:02.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:33:02.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:33:02.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:33:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:33:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:33:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:33:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:33:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:33:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:33:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:33:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:33:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:33:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:33:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:33:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 187.964045075, 2023-01-15T14:33:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:33:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:33:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.628 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:33:02.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:33:02.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:33:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:33:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:33:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:33:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:33:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:33:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:33:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:33:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:33:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:33:02.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:33:02.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:33:02.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:33:02.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:02.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:33:02.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:33:02.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:33:02.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:33:02.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:33:02.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:33:02.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:33:02.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:33:02.634 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.634 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.634 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.635 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:02.635 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:33:02.635 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 184.007625582, 2023-01-15T14:33:02.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:33:02.635 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.635 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:02.635 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.635 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.636 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.636 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:33:02.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.636 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:33:02.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:33:02.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:33:02.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:02.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:33:02.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:33:02.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:33:02.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:33:02.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:33:02.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:33:02.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:33:02.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:33:02.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:02.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:33:02.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:33:02.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:33:02.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:33:02.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:33:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:33:02.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:33:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:33:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.771 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:33:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 183.95012789899999, 2023-01-15T14:33:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:33:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:33:02.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:33:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:33:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:33:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:33:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:33:02.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:33:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:33:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:33:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:33:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:33:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:33:02.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:33:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:33:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:33:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:33:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:33:02.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:33:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:33:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:33:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:33:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:33:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:33:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:33:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:33:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:33:02.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:33:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:33:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:33:02.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:33:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:33:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:33:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:33:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:33:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:33:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:33:02.787 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 162.58147639399999, 2023-01-15T14:33:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:33:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:33:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:33:02.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:33:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:33:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:33:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:33:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:33:02.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:33:02.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:33:02.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:33:02.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:33:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:33:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:33:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:33:02.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:33:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:33:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:33:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:33:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:33:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:33:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:33:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:33:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:33:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:33:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:33:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:33:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:33:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:33:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 162.58137375300001, 2023-01-15T14:33:02.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:33:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:33:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:33:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:33:02.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:33:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:33:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:33:02.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:33:02.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:33:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 117.980391758, 2023-01-15T14:33:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:33:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:33:02.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:02.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:33:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:33:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:33:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:33:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:33:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:33:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:33:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:33:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:33:02.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:33:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:33:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:33:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:33:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:33:02.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:33:02.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:33:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:33:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:33:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:33:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:33:02.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:33:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:33:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:33:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:33:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:02.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:33:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:33:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:33:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:33:02.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:33:02.945 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 117.980366334, 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:33:02.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:33:02.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:33:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:33:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:33:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:33:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:33:02.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:33:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:33:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:33:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:33:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:33:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:02.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:33:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:33:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:33:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:33:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:33:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:33:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:33:02.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:33:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:33:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:33:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:02.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:33:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:33:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:02.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:33:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:33:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:33:03.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:33:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:33:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:33:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:33:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:33:03.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2301 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e216)", 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:33:03.065 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.690491867000006, 2023-01-15T14:33:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61754771200000003, 2023-01-15T14:33:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2301 2023-01-15T14:33:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:33:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:33:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:03.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195530+0000", 2023-01-15T14:33:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:03.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195535+0000", 2023-01-15T14:33:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3660000000000002e-06 2023-01-15T14:33:03.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195546+0000", 2023-01-15T14:33:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1062e-05 2023-01-15T14:33:03.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195551+0000", 2023-01-15T14:33:03.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2023-01-15T14:33:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195999+0000", 2023-01-15T14:33:03.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044799800000000002 2023-01-15T14:33:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.196106+0000", 2023-01-15T14:33:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106172 2023-01-15T14:33:03.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510631+0000", 2023-01-15T14:33:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31452575999999999 2023-01-15T14:33:03.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510875+0000", 2023-01-15T14:33:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024415199999999998 2023-01-15T14:33:03.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812445+0000", 2023-01-15T14:33:03.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30156923000000002 2023-01-15T14:33:03.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812727+0000", 2023-01-15T14:33:03.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028227800000000001 2023-01-15T14:33:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.813079+0000", 2023-01-15T14:33:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199599999999998 2023-01-15T14:33:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2667 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e259)", 2023-01-15T14:33:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 36.001601972000003, 2023-01-15T14:33:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60917505800000005, 2023-01-15T14:33:03.263 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2667 2023-01-15T14:33:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.264 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967296 2023-01-15T14:33:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884423+0000", 2023-01-15T14:33:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.905e-06 2023-01-15T14:33:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884427+0000", 2023-01-15T14:33:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7689999999999998e-06 2023-01-15T14:33:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:03.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884429+0000", 2023-01-15T14:33:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5509999999999998e-06 2023-01-15T14:33:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.080096+0000", 2023-01-15T14:33:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19566640099999999 2023-01-15T14:33:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.080290+0000", 2023-01-15T14:33:03.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194346 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.275924+0000", 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19563417299999999 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.276184+0000", 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025966800000000001 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493228+0000", 2023-01-15T14:33:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21704441599999999 2023-01-15T14:33:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493378+0000", 2023-01-15T14:33:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014980000000000001 2023-01-15T14:33:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493596+0000", 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021825300000000001 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2748 3.3d 3:bc1c6b44:::smithi09025264-23:head [setxattr _header (58) in=7b,truncate 2537923] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.964628895000001, 2023-01-15T14:33:03.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59151328599999997, 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2748 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:33:03.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:33:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:33:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:03.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921398+0000", 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3139999999999997e-06 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921407+0000", 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7050000000000005e-06 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921410+0000", 2023-01-15T14:33:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7459999999999998e-06 2023-01-15T14:33:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.007186+0000", 2023-01-15T14:33:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.085775343000000004 2023-01-15T14:33:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:03.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.007335+0000", 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014914400000000001 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.209977+0000", 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20264216900000001 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.210228+0000", 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025044800000000001 2023-01-15T14:33:03.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.210911+0000", 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068313300000000005 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.211407+0000", 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049586300000000003 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.211425+0000", 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8403e-05 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212523+0000", 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010975710000000001 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.213095+0000", 2023-01-15T14:33:03.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057295100000000004 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.250158+0000", 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037062401000000002 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.512701+0000", 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26254333400000002 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.512801+0000", 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9333999999999995e-05 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.512908+0000", 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106995 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2749 3.3d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:33:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.964607091000001, 2023-01-15T14:33:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.69206326600000001, 2023-01-15T14:33:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:03.283 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2749 2023-01-15T14:33:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:03.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921418+0000", 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1670000000000002e-06 2023-01-15T14:33:03.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921426+0000", 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0569999999999993e-06 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921429+0000", 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9629999999999998e-06 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107302+0000", 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.185873327 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107657+0000", 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035504299999999998 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212694+0000", 2023-01-15T14:33:03.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105037095 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212936+0000", 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241492 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613125+0000", 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40018927900000001 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613294+0000", 2023-01-15T14:33:03.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016838000000000001 2023-01-15T14:33:03.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613479+0000", 2023-01-15T14:33:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018577599999999999 2023-01-15T14:33:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.430 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:03.430 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:03.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:33:03.432 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:33:03.434 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:01.936+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:03.434 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:03.434 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:03.435 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:03.435 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:03.435 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:33:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:33:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:33:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: filestore_inject_stall = '0' (not observed, change may require restart) osd_enable_op_tracker = 'true' 2023-01-15T14:33:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:33:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:33:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:33:03.436 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:33:03.438 INFO:tasks.rados.rados.0.smithi090.stdout:1184: done (1 left) 2023-01-15T14:33:03.438 INFO:tasks.rados.rados.0.smithi090.stdout:1185: read oid 15 snap -1 2023-01-15T14:33:03.438 INFO:tasks.rados.rados.0.smithi090.stdout:1185: expect (ObjNum 361 snap 96 seq_num 361) 2023-01-15T14:33:03.438 INFO:tasks.rados.rados.0.smithi090.stdout:1186: copy_from oid 26 from oid 9 current snap is 119 2023-01-15T14:33:03.438 INFO:tasks.rados.rados.0.smithi090.stdout:1179: expect (ObjNum 380 snap 103 seq_num 380) 2023-01-15T14:33:03.438 INFO:tasks.rados.rados.0.smithi090.stdout:1179: done (2 left) 2023-01-15T14:33:03.438 INFO:tasks.rados.rados.0.smithi090.stdout:1187: read oid 30 snap -1 2023-01-15T14:33:03.438 INFO:tasks.rados.rados.0.smithi090.stdout:1187: expect (ObjNum 326 snap 87 seq_num 326) 2023-01-15T14:33:03.438 INFO:tasks.rados.rados.0.smithi090.stdout:1188: snap_create 2023-01-15T14:33:03.439 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:03.439 INFO:tasks.rados.rados.0.smithi090.stdout:1186: finishing copy_from to smithi09025264-26 2023-01-15T14:33:03.439 INFO:tasks.rados.rados.0.smithi090.stdout:1186: got expected ENOENT (src dne) 2023-01-15T14:33:03.439 INFO:tasks.rados.rados.0.smithi090.stdout:1186: finishing copy_from racing read to smithi09025264-26 2023-01-15T14:33:03.439 INFO:tasks.rados.rados.0.smithi090.stdout:1186: done (2 left) 2023-01-15T14:33:03.439 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:03.439 INFO:tasks.rados.rados.0.smithi090.stdout:1185: expect (ObjNum 361 snap 96 seq_num 361) 2023-01-15T14:33:03.440 INFO:tasks.rados.rados.0.smithi090.stdout:1185: done (1 left) 2023-01-15T14:33:03.440 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:03.440 INFO:tasks.rados.rados.0.smithi090.stdout:1187: expect (ObjNum 326 snap 87 seq_num 326) 2023-01-15T14:33:03.440 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:03.425+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:03.449 INFO:tasks.rados.rados.0.smithi090.stdout:1187: done (0 left) 2023-01-15T14:33:03.534 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:03.532+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:03.564 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.4.0:675 3.1a e294/224)", 2023-01-15T14:33:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:03.467987+0000", 2023-01-15T14:33:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.071167357000000001, 2023-01-15T14:33:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.071192070999999996, 2023-01-15T14:33:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:03.565 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:03.467987+0000", 2023-01-15T14:33:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:03.467987+0000", 2023-01-15T14:33:03.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:03.467986+0000", 2023-01-15T14:33:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:03.467994+0000", 2023-01-15T14:33:03.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0730000000000005e-06 2023-01-15T14:33:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:03.468018+0000", 2023-01-15T14:33:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.37e-05 2023-01-15T14:33:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:03.468025+0000", 2023-01-15T14:33:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1369999999999996e-06 2023-01-15T14:33:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:03.569 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:33:03.569 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:03.570 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:33:03.636 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:03.629+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:03.739 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:33:03.739 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:33:03.739 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2023-01-15T14:33:03.739 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-01-15T14:33:03.739 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-01-15T14:33:03.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2023-01-15T14:33:03.741 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:33:03.741 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:33:03.741 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:03.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:33:03.743 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:03.733+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:03.842 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:03.833+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:03.918 INFO:tasks.rados.rados.0.smithi090.stdout:1188: done (0 left) 2023-01-15T14:33:03.918 INFO:tasks.rados.rados.0.smithi090.stdout:1189: snap_create 2023-01-15T14:33:03.919 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:33:03.919 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:03.920 INFO:teuthology.orchestra.run.smithi090.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-01-15T14:33:03.920 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.3.0:262 3.8 e98/41 3:17d5d741:::smithi09025264-5:43 v 98'162, mlcod=97'157)", 2023-01-15T14:33:03.921 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 197.84535598100001, 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302066416, 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058703+0000", 2023-01-15T14:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058701+0000", 2023-01-15T14:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:03.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058717+0000", 2023-01-15T14:33:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6350000000000001e-05 2023-01-15T14:33:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058759+0000", 2023-01-15T14:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1502999999999999e-05 2023-01-15T14:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:03.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.058776+0000", 2023-01-15T14:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6558999999999998e-05 2023-01-15T14:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260016+0000", 2023-01-15T14:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20124061600000001 2023-01-15T14:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260036+0000", 2023-01-15T14:33:03.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0032000000000001e-05 2023-01-15T14:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260411+0000", 2023-01-15T14:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037480899999999998 2023-01-15T14:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260426+0000", 2023-01-15T14:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5167999999999999e-05 2023-01-15T14:33:03.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.260526+0000", 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9692999999999996e-05 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360700+0000", 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100174374 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:46.360769+0000", 2023-01-15T14:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8842000000000003e-05 2023-01-15T14:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 149.00959859100001, 2023-01-15T14:33:03.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:03.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:03.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:03.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:33:03.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:33:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:03.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:33:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:33:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:33:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:33:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:04.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:33:04.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:04.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:33:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:33:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:33:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:33:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:33:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 149.00954764799999, 2023-01-15T14:33:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:33:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:33:04.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:33:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:33:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:33:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:33:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:33:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:33:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:04.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:33:04.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:33:04.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:33:04.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:33:04.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:04.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:33:04.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:33:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:33:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:33:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:33:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:33:04.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:33:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:33:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:33:04.103 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 146.06864840700001, 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:04.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:33:04.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:04.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:33:04.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:33:04.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:33:04.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:33:04.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:33:04.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:33:04.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:33:04.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:33:04.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:33:04.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:04.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:33:04.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:04.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:33:04.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:33:04.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:04.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:33:04.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:33:04.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:33:04.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:33:04.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:04.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:33:04.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:33:04.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:33:04.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:33:04.114 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.114 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.114 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.114 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:33:04.114 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:04.114 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 143.99487423400001, 2023-01-15T14:33:04.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:33:04.115 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.115 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:04.115 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.115 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.115 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.115 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:33:04.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.115 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:04.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:04.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:33:04.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:04.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:33:04.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:33:04.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:33:04.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:33:04.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:33:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:33:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:33:04.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:33:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:33:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:33:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:33:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:04.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:33:04.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:33:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:33:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:33:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:33:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:33:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:33:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1712 3.4 3:2022fdfa:::smithi09025264-46:head [write 1880802~421790] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:33:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:33:04.259 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 143.98973554299999, 2023-01-15T14:33:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30396041800000001, 2023-01-15T14:33:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1712 2023-01-15T14:33:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:33:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914323+0000", 2023-01-15T14:33:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914322+0000", 2023-01-15T14:33:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914561+0000", 2023-01-15T14:33:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023884700000000001 2023-01-15T14:33:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914572+0000", 2023-01-15T14:33:04.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0499999999999999e-05 2023-01-15T14:33:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.914576+0000", 2023-01-15T14:33:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6820000000000002e-06 2023-01-15T14:33:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014941+0000", 2023-01-15T14:33:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100364994 2023-01-15T14:33:04.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014959+0000", 2023-01-15T14:33:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7416000000000001e-05 2023-01-15T14:33:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018171+0000", 2023-01-15T14:33:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032121239999999998 2023-01-15T14:33:04.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018333+0000", 2023-01-15T14:33:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016180900000000001 2023-01-15T14:33:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018581+0000", 2023-01-15T14:33:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024773699999999999 2023-01-15T14:33:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018897+0000", 2023-01-15T14:33:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031622099999999999 2023-01-15T14:33:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018907+0000", 2023-01-15T14:33:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0061000000000001e-05 2023-01-15T14:33:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.019108+0000", 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020080099999999999 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.020057+0000", 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00094954499999999999 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150472+0000", 2023-01-15T14:33:04.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.130414958 2023-01-15T14:33:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218188+0000", 2023-01-15T14:33:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067715656999999999 2023-01-15T14:33:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218242+0000", 2023-01-15T14:33:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4651999999999998e-05 2023-01-15T14:33:04.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218284+0000", 2023-01-15T14:33:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1084999999999997e-05 2023-01-15T14:33:04.271 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.100918814, 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.391 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:33:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:33:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:33:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:33:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:33:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:33:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:33:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:33:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:33:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:33:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:33:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:33:04.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:33:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:33:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:33:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:33:04.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:33:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1820 3.1f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:33:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:33:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 120.00061732899999, 2023-01-15T14:33:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306504938, 2023-01-15T14:33:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.397 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:33:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1820 2023-01-15T14:33:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:33:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903441+0000", 2023-01-15T14:33:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903440+0000", 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903448+0000", 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9430000000000006e-06 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903469+0000", 2023-01-15T14:33:04.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0305e-05 2023-01-15T14:33:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903481+0000", 2023-01-15T14:33:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:33:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903616+0000", 2023-01-15T14:33:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013477199999999999 2023-01-15T14:33:04.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903643+0000", 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7098999999999999e-05 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209081+0000", 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30543790300000001 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209213+0000", 2023-01-15T14:33:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013155400000000001 2023-01-15T14:33:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209600+0000", 2023-01-15T14:33:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038751699999999998 2023-01-15T14:33:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209946+0000", 2023-01-15T14:33:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034601 2023-01-15T14:33:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.217512133, 2023-01-15T14:33:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:33:04.425 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:33:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:33:04.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:04.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:33:04.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:33:04.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:33:04.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:33:04.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:33:04.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.546 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.547 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.547 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:04.547 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:04.547 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.214105009999997, 2023-01-15T14:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:33:04.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:33:04.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:04.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:33:04.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:33:04.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:33:04.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.213889405000003, 2023-01-15T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:33:04.685 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:04.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:33:04.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:33:04.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:04.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:33:04.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 98.213782331000004, 2023-01-15T14:33:04.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:33:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.696 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:04.703 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:33:04.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:33:04.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.083543281000004, 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:04.710 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:04.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:04.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:33:04.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:33:04.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:33:04.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:33:04.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:04.720 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:04.732 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.083438744000006, 2023-01-15T14:33:04.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:33:04.948 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.948 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:04.948 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.948 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:04.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:33:04.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:33:04.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:04.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:33:04.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:33:04.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:04.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:33:04.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:04.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:33:04.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:33:04.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:04.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:33:04.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:33:04.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:04.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:33:04.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:33:04.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:04.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:33:04.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:33:04.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:33:04.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:33:04.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:04.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:33:04.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:33:04.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:04.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.957 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:04.958 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:04.958 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.083368813000007, 2023-01-15T14:33:04.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:33:04.958 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:04.958 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:04.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:04.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:04.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:04.958 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:33:04.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:04.959 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:04.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:04.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:04.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:04.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:05.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:05.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:05.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:33:05.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:33:05.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:05.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:33:05.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:33:05.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:05.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:33:05.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:33:05.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:05.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:33:05.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:33:05.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:05.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:33:05.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:33:05.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:05.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:33:05.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:33:05.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:33:05.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:33:05.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:05.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:33:05.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:33:05.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:05.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:33:05.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:33:05.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:05.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:33:05.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:33:05.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:05.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:33:05.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:33:05.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.299 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.4.0:534 3.1a e226/224 3:5919d83e:::smithi09025264-13:a9 v 226'240, mlcod=0'0)", 2023-01-15T14:33:05.299 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:33:05.299 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 70.368474137000007, 2023-01-15T14:33:05.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30202553599999998, 2023-01-15T14:33:05.299 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:05.299 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:05.299 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:05.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:05.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:33:05.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:05.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535585+0000", 2023-01-15T14:33:05.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:05.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535584+0000", 2023-01-15T14:33:05.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535592+0000", 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3010000000000007e-06 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.535604+0000", 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1800000000000001e-05 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.635708+0000", 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10010366800000001 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836079+0000", 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20037171500000001 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:05.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836113+0000", 2023-01-15T14:33:05.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.345e-05 2023-01-15T14:33:05.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:05.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836714+0000", 2023-01-15T14:33:05.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060093400000000002 2023-01-15T14:33:05.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:05.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836737+0000", 2023-01-15T14:33:05.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3073000000000001e-05 2023-01-15T14:33:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836816+0000", 2023-01-15T14:33:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8740999999999997e-05 2023-01-15T14:33:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.836898+0000", 2023-01-15T14:33:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2731999999999993e-05 2023-01-15T14:33:05.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:05.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:53.837610+0000", 2023-01-15T14:33:05.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071176200000000001 2023-01-15T14:33:05.305 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.305 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.305 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.305 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2780 3.38 e265/232 3:1eaad226:::smithi09025264-49:head v 265'279, mlcod=261'274)", 2023-01-15T14:33:05.305 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.912197686999999, 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32005591799999999, 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:05.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:05.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:05.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991860+0000", 2023-01-15T14:33:05.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:05.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:05.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993927+0000", 2023-01-15T14:33:05.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020673229999999998 2023-01-15T14:33:05.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:05.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993946+0000", 2023-01-15T14:33:05.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9429999999999999e-05 2023-01-15T14:33:05.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:05.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.094039+0000", 2023-01-15T14:33:05.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092898 2023-01-15T14:33:05.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:05.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285071+0000", 2023-01-15T14:33:05.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.191031904 2023-01-15T14:33:05.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:05.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285106+0000", 2023-01-15T14:33:05.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5227e-05 2023-01-15T14:33:05.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:05.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286323+0000", 2023-01-15T14:33:05.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012163829999999999 2023-01-15T14:33:05.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:05.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286335+0000", 2023-01-15T14:33:05.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2224e-05 2023-01-15T14:33:05.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:05.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287218+0000", 2023-01-15T14:33:05.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088265000000000001 2023-01-15T14:33:05.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287250+0000", 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2209000000000003e-05 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.311917+0000", 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024667062999999999 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.500 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2868 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'278, mlcod=264'275)", 2023-01-15T14:33:05.500 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:05.500 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.910306746, 2023-01-15T14:33:05.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31268252400000002, 2023-01-15T14:33:05.500 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:05.500 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:05.500 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:05.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:05.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:05.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:05.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:05.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:05.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993751+0000", 2023-01-15T14:33:05.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:05.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993762+0000", 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0822000000000001e-05 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993777+0000", 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5540999999999999e-05 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993787+0000", 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0173999999999999e-05 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290198+0000", 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29641022099999997 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290212+0000", 2023-01-15T14:33:05.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4214e-05 2023-01-15T14:33:05.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:05.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290300+0000", 2023-01-15T14:33:05.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8300000000000005e-05 2023-01-15T14:33:05.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:05.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290312+0000", 2023-01-15T14:33:05.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2344000000000001e-05 2023-01-15T14:33:05.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:05.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290357+0000", 2023-01-15T14:33:05.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4367999999999999e-05 2023-01-15T14:33:05.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:05.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290389+0000", 2023-01-15T14:33:05.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1983000000000002e-05 2023-01-15T14:33:05.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.306434+0000", 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016045667 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.910257321, 2023-01-15T14:33:05.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:33:05.507 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:05.507 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:05.507 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:05.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:05.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:05.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:05.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:05.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:05.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:05.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:05.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:05.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:33:05.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:33:05.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:05.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:33:05.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:33:05.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:05.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:33:05.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:33:05.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:33:05.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:05.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:33:05.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:33:05.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:05.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:33:05.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:33:05.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:05.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:33:05.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:33:05.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:05.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:33:05.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:33:05.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.514 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.514 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:33:05.514 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.906445634000001, 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:33:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:05.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:33:05.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:33:05.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:33:05.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.521 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.521 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.521 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.521 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:05.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:33:05.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:33:05.524 INFO:tasks.rados.rados.0.smithi090.stdout:1189: done (0 left) 2023-01-15T14:33:05.524 INFO:tasks.rados.rados.0.smithi090.stdout:1190: rollback oid 32 current snap is 121 2023-01-15T14:33:05.524 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 32 to 119 2023-01-15T14:33:05.524 INFO:tasks.rados.rados.0.smithi090.stdout:1191: read oid 19 snap -1 2023-01-15T14:33:05.524 INFO:tasks.rados.rados.0.smithi090.stdout:1191: expect deleted 2023-01-15T14:33:05.524 INFO:tasks.rados.rados.0.smithi090.stdout:1192: copy_from oid 10 from oid 11 current snap is 121 2023-01-15T14:33:05.524 INFO:tasks.rados.rados.0.smithi090.stdout:1193: snap_remove snap 118 2023-01-15T14:33:05.525 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:05.517+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:05.624 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:05.620+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:05.705 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.7.0:647 3.d e295/193 3:b3f111b5:::smithi09025264-48:df v 295'230, mlcod=290'226)", 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:04.033753+0000", 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 1.6023128820000001, 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.60233314, 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:05.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.033753+0000", 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.033753+0000", 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.033751+0000", 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.033765+0000", 2023-01-15T14:33:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3286e-05 2023-01-15T14:33:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.033793+0000", 2023-01-15T14:33:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8039999999999999e-05 2023-01-15T14:33:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:05.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.033806+0000", 2023-01-15T14:33:05.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3399e-05 2023-01-15T14:33:05.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:05.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.033886+0000", 2023-01-15T14:33:05.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9832000000000004e-05 2023-01-15T14:33:05.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:05.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.033925+0000", 2023-01-15T14:33:05.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8835999999999999e-05 2023-01-15T14:33:05.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:05.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.034555+0000", 2023-01-15T14:33:05.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063018000000000004 2023-01-15T14:33:05.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:05.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.034606+0000", 2023-01-15T14:33:05.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1316000000000001e-05 2023-01-15T14:33:05.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:05.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.034677+0000", 2023-01-15T14:33:05.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0388000000000004e-05 2023-01-15T14:33:05.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:05.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.034707+0000", 2023-01-15T14:33:05.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9980000000000001e-05 2023-01-15T14:33:05.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.712 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.712 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.7.0:649 3.4a e295/263 3:52df5138:::smithi09025264-25:e2 v 295'320, mlcod=292'316)", 2023-01-15T14:33:05.712 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:04.638090+0000", 2023-01-15T14:33:05.712 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.99797575199999999, 2023-01-15T14:33:05.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.99804404400000002, 2023-01-15T14:33:05.712 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:05.712 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:05.713 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:05.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:05.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.638090+0000", 2023-01-15T14:33:05.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:05.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.638090+0000", 2023-01-15T14:33:05.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:05.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.638088+0000", 2023-01-15T14:33:05.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:05.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:05.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.638101+0000", 2023-01-15T14:33:05.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3040000000000001e-05 2023-01-15T14:33:05.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:05.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.638131+0000", 2023-01-15T14:33:05.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9862999999999999e-05 2023-01-15T14:33:05.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:05.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.638143+0000", 2023-01-15T14:33:05.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1953000000000001e-05 2023-01-15T14:33:05.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:05.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.638290+0000", 2023-01-15T14:33:05.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014719000000000001 2023-01-15T14:33:05.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:05.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.638335+0000", 2023-01-15T14:33:05.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4036000000000003e-05 2023-01-15T14:33:05.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.638997+0000", 2023-01-15T14:33:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000662176 2023-01-15T14:33:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.639049+0000", 2023-01-15T14:33:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2145999999999998e-05 2023-01-15T14:33:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:05.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.639122+0000", 2023-01-15T14:33:05.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3365e-05 2023-01-15T14:33:05.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:05.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.639191+0000", 2023-01-15T14:33:05.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8435000000000005e-05 2023-01-15T14:33:05.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.718 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.719 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3117 3.10 3.20b19950 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e296)", 2023-01-15T14:33:05.719 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:04.879628+0000", 2023-01-15T14:33:05.719 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.75643825200000003, 2023-01-15T14:33:05.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.75655100600000003, 2023-01-15T14:33:05.719 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:05.719 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "delayed", 2023-01-15T14:33:05.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:05.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:05.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:05.719 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3117 2023-01-15T14:33:05.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879628+0000", 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879628+0000", 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:05.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879625+0000", 2023-01-15T14:33:05.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:05.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:05.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879640+0000", 2023-01-15T14:33:05.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5489e-05 2023-01-15T14:33:05.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879668+0000", 2023-01-15T14:33:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7319000000000001e-05 2023-01-15T14:33:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879685+0000", 2023-01-15T14:33:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7856999999999999e-05 2023-01-15T14:33:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:05.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.980162+0000", 2023-01-15T14:33:05.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100476725 2023-01-15T14:33:05.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:33:05.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.980175+0000", 2023-01-15T14:33:05.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3173000000000001e-05 2023-01-15T14:33:05.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.723 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.724 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3118 3.10 3.20b19950 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e296)", 2023-01-15T14:33:05.724 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:04.879828+0000", 2023-01-15T14:33:05.724 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.75623807099999996, 2023-01-15T14:33:05.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.75638692799999996, 2023-01-15T14:33:05.724 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:05.724 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "delayed", 2023-01-15T14:33:05.724 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:05.724 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:05.724 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:05.724 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3118 2023-01-15T14:33:05.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.725 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:05.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:05.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879828+0000", 2023-01-15T14:33:05.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:05.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879828+0000", 2023-01-15T14:33:05.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:05.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879826+0000", 2023-01-15T14:33:05.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:05.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:05.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879835+0000", 2023-01-15T14:33:05.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5280000000000003e-06 2023-01-15T14:33:05.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:05.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879855+0000", 2023-01-15T14:33:05.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0279999999999999e-05 2023-01-15T14:33:05.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:05.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.879868+0000", 2023-01-15T14:33:05.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2771e-05 2023-01-15T14:33:05.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:05.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.980338+0000", 2023-01-15T14:33:05.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100470351 2023-01-15T14:33:05.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:05.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:04.980363+0000", 2023-01-15T14:33:05.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5123e-05 2023-01-15T14:33:05.728 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.728 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.728 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.729 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.7.0:650 3.d e296/193)", 2023-01-15T14:33:05.729 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:05.590721+0000", 2023-01-15T14:33:05.729 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.045345110000000001, 2023-01-15T14:33:05.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045525253000000002, 2023-01-15T14:33:05.729 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:05.729 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:05.729 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:05.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:05.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:05.590721+0000", 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:05.590721+0000", 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:05.590719+0000", 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:05.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:05.590736+0000", 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7677e-05 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:05.590765+0000", 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8819e-05 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:05.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:05.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:05.590781+0000", 2023-01-15T14:33:05.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5449999999999999e-05 2023-01-15T14:33:05.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.732 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:05.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.732 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:05.732 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:05.732 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 5 2023-01-15T14:33:05.732 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:05.733 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:33:05.734 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:05.734 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:05.734 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:05.734 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:05.736 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:05.729+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:05.742 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:05.762 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:05.784 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:05.813 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:05.815 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:33:05.815 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:33:05.815 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:33:05.816 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:33:05.816 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:33:05.816 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:33:05.816 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:33:05.816 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:33:05.836 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:05.829+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:05.843 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:05.843 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:05.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:33:05.883 INFO:tasks.rados.rados.0.smithi090.stdout:1193: done (3 left) 2023-01-15T14:33:05.883 INFO:tasks.rados.rados.0.smithi090.stdout:1194: write oid 12 current snap is 121 2023-01-15T14:33:05.883 INFO:tasks.rados.rados.0.smithi090.stdout:1194: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2023-01-15T14:33:05.894 INFO:tasks.rados.rados.0.smithi090.stdout:1194: writing smithi09025264-12 from 578273 to 1044705 tid 1 2023-01-15T14:33:05.902 INFO:tasks.rados.rados.0.smithi090.stdout:1194: writing smithi09025264-12 from 1739595 to 2192410 tid 2 2023-01-15T14:33:05.909 INFO:tasks.rados.rados.0.smithi090.stdout:1194: writing smithi09025264-12 from 2722929 to 3378273 tid 3 2023-01-15T14:33:05.909 INFO:tasks.rados.rados.0.smithi090.stdout:1195: read oid 33 snap 116 2023-01-15T14:33:05.909 INFO:tasks.rados.rados.0.smithi090.stdout:1195: expect (ObjNum 378 snap 102 seq_num 378) 2023-01-15T14:33:05.909 INFO:tasks.rados.rados.0.smithi090.stdout:1196: write oid 26 current snap is 121 2023-01-15T14:33:05.909 INFO:tasks.rados.rados.0.smithi090.stdout:1196: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2023-01-15T14:33:05.915 INFO:tasks.rados.rados.0.smithi090.stdout:1196: writing smithi09025264-26 from 595080 to 1136761 tid 1 2023-01-15T14:33:05.920 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:33:05.923 INFO:tasks.rados.rados.0.smithi090.stdout:1196: writing smithi09025264-26 from 1798077 to 2394550 tid 2 2023-01-15T14:33:05.929 INFO:tasks.rados.rados.0.smithi090.stdout:1196: writing smithi09025264-26 from 3150352 to 3395080 tid 3 2023-01-15T14:33:05.929 INFO:tasks.rados.rados.0.smithi090.stdout:1197: copy_from oid 30 from oid 23 current snap is 121 2023-01-15T14:33:05.929 INFO:tasks.rados.rados.0.smithi090.stdout:1191: done (6 left) 2023-01-15T14:33:05.929 INFO:tasks.rados.rados.0.smithi090.stdout:1198: delete oid 7 current snap is 121 2023-01-15T14:33:05.937 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:05.936+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:05.989 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:06.115 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:06.117 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:33:06.117 INFO:tasks.rados.rados.0.smithi090.stdout:1196: finishing write tid 1 to smithi09025264-26 2023-01-15T14:33:06.117 INFO:tasks.rados.rados.0.smithi090.stdout:1196: finishing write tid 2 to smithi09025264-26 2023-01-15T14:33:06.118 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:06.112+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:06.215 INFO:tasks.rados.rados.0.smithi090.stdout:1195: expect (ObjNum 378 snap 102 seq_num 378) 2023-01-15T14:33:06.217 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:06.216+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:06.230 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:06.230 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:06.241 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:33:06.305 INFO:tasks.rados.rados.0.smithi090.stdout:1196: finishing write tid 3 to smithi09025264-26 2023-01-15T14:33:06.306 INFO:tasks.rados.rados.0.smithi090.stdout:1196: finishing write tid 4 to smithi09025264-26 2023-01-15T14:33:06.306 INFO:tasks.rados.rados.0.smithi090.stdout:1195: done (6 left) 2023-01-15T14:33:06.306 INFO:tasks.rados.rados.0.smithi090.stdout:1198: done (5 left) 2023-01-15T14:33:06.306 INFO:tasks.rados.rados.0.smithi090.stdout:1199: write oid 3 current snap is 121 2023-01-15T14:33:06.306 INFO:tasks.rados.rados.0.smithi090.stdout:1199: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2023-01-15T14:33:06.317 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:06.309+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:06.319 INFO:tasks.rados.rados.0.smithi090.stdout:1199: writing smithi09025264-3 from 628694 to 1037226 tid 1 2023-01-15T14:33:06.325 INFO:tasks.rados.rados.0.smithi090.stdout:1199: writing smithi09025264-3 from 1515041 to 2115183 tid 2 2023-01-15T14:33:06.332 INFO:tasks.rados.rados.0.smithi090.stdout:1199: writing smithi09025264-3 from 2805198 to 3357310 tid 3 2023-01-15T14:33:06.333 INFO:tasks.rados.rados.0.smithi090.stdout:1200: snap_create 2023-01-15T14:33:06.333 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:33:06.333 INFO:tasks.rados.rados.0.smithi090.stdout:1196: finishing write tid 5 to smithi09025264-26 2023-01-15T14:33:06.333 INFO:tasks.rados.rados.0.smithi090.stdout:1196: oid 26 updating version 0 to 312 2023-01-15T14:33:06.333 INFO:tasks.rados.rados.0.smithi090.stdout:1196: oid 26 updating version 312 to 314 2023-01-15T14:33:06.333 INFO:tasks.rados.rados.0.smithi090.stdout:1196: oid 26 version 314 is already newer than 313 2023-01-15T14:33:06.333 INFO:tasks.rados.rados.0.smithi090.stdout:1196: oid 26 version 314 is already newer than 311 2023-01-15T14:33:06.333 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 314 (ObjNum 440 snap 121 seq_num 440) dirty exists 2023-01-15T14:33:06.333 INFO:tasks.rados.rados.0.smithi090.stdout:1196: left oid 26 (ObjNum 440 snap 121 seq_num 440) 2023-01-15T14:33:06.334 INFO:tasks.rados.rados.0.smithi090.stdout:1196: done (5 left) 2023-01-15T14:33:06.334 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:33:06.342 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:06.342 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:06.352 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:33:06.418 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:06.413+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:06.460 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:06.460 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:06.470 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:33:06.519 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:06.513+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:06.593 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:06.594 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:06.594 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:33:06.620 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:06.616+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:06.696 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:06.696 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:06.705 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:33:06.720 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:06.716+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:06.811 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:06.811 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:06.821 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:33:06.821 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:06.820+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:06.875 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:33:06.937 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:06.929+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:06.979 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:06.979 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:06.989 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:33:07.022 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:07.020+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:07.097 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:07.098 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:07.107 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:33:07.149 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:07.148+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:07.224 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:07.220+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:07.257 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:07.257 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:07.267 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:33:07.325 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:07.317+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:07.411 INFO:tasks.rados.rados.0.smithi090.stdout:1197: finishing copy_from to smithi09025264-30 2023-01-15T14:33:07.411 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 221 (ObjNum 406 snap 108 seq_num 406) dirty exists 2023-01-15T14:33:07.412 INFO:tasks.rados.rados.0.smithi090.stdout:1197: finishing copy_from racing read to smithi09025264-30 2023-01-15T14:33:07.412 INFO:tasks.rados.rados.0.smithi090.stdout:1197: done (4 left) 2023-01-15T14:33:07.412 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:33:07.412 INFO:tasks.rados.rados.0.smithi090.stdout:1199: finishing write tid 1 to smithi09025264-3 2023-01-15T14:33:07.412 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:07.412 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:07.413 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:33:07.426 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:07.417+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:07.445 INFO:tasks.rados.rados.0.smithi090.stdout:1199: finishing write tid 2 to smithi09025264-3 2023-01-15T14:33:07.537 INFO:tasks.rados.rados.0.smithi090.stdout:1199: finishing write tid 3 to smithi09025264-3 2023-01-15T14:33:07.538 INFO:tasks.rados.rados.0.smithi090.stdout:1190: finishing rollback tid 0 to smithi09025264-32 2023-01-15T14:33:07.539 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:07.529+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:07.540 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:07.540 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:07.540 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:33:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:1199: finishing write tid 4 to smithi09025264-3 2023-01-15T14:33:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:1199: finishing write tid 5 to smithi09025264-3 2023-01-15T14:33:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:1199: oid 3 updating version 0 to 324 2023-01-15T14:33:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:1199: oid 3 version 324 is already newer than 321 2023-01-15T14:33:07.542 INFO:tasks.rados.rados.0.smithi090.stdout:1199: oid 3 version 324 is already newer than 323 2023-01-15T14:33:07.543 INFO:tasks.rados.rados.0.smithi090.stdout:1199: oid 3 version 324 is already newer than 322 2023-01-15T14:33:07.543 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 324 (ObjNum 442 snap 121 seq_num 442) dirty exists 2023-01-15T14:33:07.543 INFO:tasks.rados.rados.0.smithi090.stdout:1199: left oid 3 (ObjNum 442 snap 121 seq_num 442) 2023-01-15T14:33:07.543 INFO:tasks.rados.rados.0.smithi090.stdout:1199: done (3 left) 2023-01-15T14:33:07.543 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:07.639 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:07.633+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:07.651 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:07.652 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:07.662 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:33:07.741 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:07.740+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:07.741 INFO:tasks.rados.rados.0.smithi090.stdout:1192: finishing copy_from to smithi09025264-10 2023-01-15T14:33:07.741 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 309 (ObjNum 407 snap 108 seq_num 407) dirty exists 2023-01-15T14:33:07.742 INFO:tasks.rados.rados.0.smithi090.stdout:1192: finishing copy_from racing read to smithi09025264-10 2023-01-15T14:33:07.742 INFO:tasks.rados.rados.0.smithi090.stdout:1192: done (2 left) 2023-01-15T14:33:07.742 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:07.770 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:07.770 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:07.780 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:33:07.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:33:07.841 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:07.840+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:07.883 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:07.883 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:07.894 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:33:07.942 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:07.937+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:07.966 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:07.983 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:08.068 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:08.070 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:08.070 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:08.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:08.070 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:08.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.070 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:08.070 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:33:08.070 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 281.93620059900002, 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:33:08.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:33:08.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:33:08.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:08.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:33:08.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:33:08.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:33:08.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:33:08.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:33:08.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:33:08.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:33:08.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:33:08.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:33:08.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:33:08.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:33:08.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:33:08.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:33:08.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:33:08.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:08.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:33:08.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:33:08.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:08.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:33:08.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:33:08.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:08.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:33:08.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:33:08.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:08.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:33:08.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:33:08.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:08.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:33:08.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:33:08.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:33:08.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:33:08.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:33:08.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:33:08.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:08.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:33:08.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:33:08.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:33:08.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:33:08.081 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.081 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.081 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.081 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:08.081 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:33:08.081 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 281.93592035099999, 2023-01-15T14:33:08.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:33:08.081 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.082 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:08.082 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.082 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.082 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.082 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:33:08.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.082 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:33:08.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:33:08.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:33:08.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:33:08.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:33:08.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:33:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:33:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:33:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:33:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:33:08.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:33:08.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:33:08.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:33:08.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:33:08.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:33:08.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:08.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:33:08.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:33:08.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:08.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:33:08.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:33:08.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:08.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:33:08.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:33:08.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:08.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:33:08.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:33:08.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:08.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:33:08.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:33:08.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:33:08.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:33:08.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:33:08.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:33:08.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:08.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:33:08.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:33:08.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:33:08.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:33:08.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.091 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:08.092 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:33:08.092 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 281.935887734, 2023-01-15T14:33:08.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:33:08.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:08.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:33:08.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.094 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:08.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:33:08.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:33:08.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:33:08.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:33:08.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:33:08.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:33:08.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:33:08.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:33:08.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:33:08.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:33:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:33:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:33:08.100 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 281.83456974799998, 2023-01-15T14:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:33:08.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:33:08.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:33:08.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:33:08.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:33:08.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:33:08.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:33:08.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:08.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:33:08.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:33:08.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:08.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:33:08.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:33:08.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:08.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:33:08.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:33:08.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:08.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:33:08.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:33:08.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:33:08.109 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.110 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.110 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:08.110 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:33:08.110 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 281.83453974499997, 2023-01-15T14:33:08.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:33:08.110 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.110 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:08.110 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:33:08.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:33:08.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:33:08.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:33:08.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:33:08.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:33:08.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:33:08.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.116 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 281.31953118000001, 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.117 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.118 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:33:08.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.118 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:33:08.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:33:08.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:33:08.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:33:08.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:33:08.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:33:08.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:33:08.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:08.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:33:08.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:33:08.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:08.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:33:08.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:33:08.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:33:08.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:33:08.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:33:08.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:33:08.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:08.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.126 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 281.31904882600003, 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:33:08.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:33:08.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:33:08.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:08.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:33:08.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:33:08.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:33:08.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:33:08.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:33:08.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:33:08.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:33:08.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:33:08.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:33:08.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:33:08.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:33:08.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:33:08.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:33:08.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:33:08.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:08.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:33:08.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:33:08.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:08.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:33:08.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:33:08.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:08.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:33:08.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:33:08.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:08.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:33:08.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:33:08.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:08.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:33:08.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:33:08.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:33:08.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:33:08.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:33:08.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:33:08.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:08.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:33:08.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:33:08.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:33:08.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:33:08.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.242 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.243 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:08.243 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:33:08.243 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 281.31899626299997, 2023-01-15T14:33:08.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:33:08.243 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.243 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:08.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.244 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.244 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:33:08.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.244 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:33:08.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:33:08.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:33:08.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:08.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:33:08.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:33:08.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:33:08.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:33:08.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:33:08.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:33:08.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:33:08.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:33:08.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:33:08.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:33:08.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:33:08.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:33:08.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:33:08.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:33:08.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:33:08.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:33:08.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:33:08.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:33:08.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:33:08.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:33:08.252 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.252 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.252 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.253 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:08.253 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:33:08.253 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 267.94779455899999, 2023-01-15T14:33:08.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:33:08.253 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.253 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:08.253 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.254 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.254 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:33:08.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.254 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:33:08.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:33:08.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:33:08.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:08.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:33:08.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:33:08.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:33:08.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:33:08.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:33:08.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:33:08.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:08.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:33:08.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:33:08.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:08.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:33:08.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:33:08.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:08.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:33:08.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:33:08.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:08.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:33:08.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:33:08.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:08.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:33:08.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:33:08.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:33:08.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:33:08.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:33:08.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:33:08.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:08.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.396 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:08.397 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:33:08.397 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 267.94767788500002, 2023-01-15T14:33:08.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:33:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:33:08.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:33:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:33:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:33:08.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:08.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:33:08.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:33:08.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:33:08.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:33:08.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:33:08.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:33:08.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:33:08.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:33:08.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:33:08.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:33:08.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:33:08.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.574 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 260.22154560899997, 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.576 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:33:08.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.576 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:33:08.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:33:08.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:33:08.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:33:08.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:33:08.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:33:08.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:33:08.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:33:08.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:33:08.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:33:08.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:33:08.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:33:08.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:33:08.582 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.582 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.582 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.582 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:33:08.582 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:33:08.582 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 224.00529555, 2023-01-15T14:33:08.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:33:08.583 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.583 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:08.583 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.583 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.583 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.583 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:33:08.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.586 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:33:08.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:33:08.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:33:08.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:08.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:33:08.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:33:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:33:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:33:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:33:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:33:08.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:33:08.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:33:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:33:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:33:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:33:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:33:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:33:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:33:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:33:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:33:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:33:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:33:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:33:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:08.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:33:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:33:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:33:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:33:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:33:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 224.005261983, 2023-01-15T14:33:08.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:33:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.782 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:33:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:08.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:33:08.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:33:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:33:08.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:33:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:33:08.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:33:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:33:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:33:08.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:33:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:33:08.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:33:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:33:08.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:33:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:33:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:33:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:33:08.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:33:08.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:33:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:33:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:33:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:33:08.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:33:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:33:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:33:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:33:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:33:08.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:33:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:33:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:33:08.981 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:33:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 205.23935843699999, 2023-01-15T14:33:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:33:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:33:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:33:08.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:33:08.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:33:08.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:08.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:33:08.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:33:08.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:33:08.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:33:08.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:33:08.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:33:08.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:33:08.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:33:08.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:08.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:33:08.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:33:08.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:33:08.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:33:08.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:33:08.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:33:08.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:08.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:33:08.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:33:08.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:08.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:33:08.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:33:08.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:08.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:33:08.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:33:08.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:08.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:33:08.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:33:08.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:08.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:33:08.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:33:08.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.992 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:33:08.992 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:33:08.992 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 188.696179637, 2023-01-15T14:33:08.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:33:08.992 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:08.992 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:08.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:08.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:08.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:08.992 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:33:08.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:08.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:33:08.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:33:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:33:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:33:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:33:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:08.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:33:08.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:33:08.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:08.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:33:08.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:33:08.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:08.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:33:08.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:33:08.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:08.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:33:08.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:33:08.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:08.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:08.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:33:08.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:33:08.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:08.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:09.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:33:09.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:33:09.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:33:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:33:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:33:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:33:09.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:09.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:33:09.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:33:09.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:09.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:33:09.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:33:09.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.004 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:09.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.004 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:33:09.005 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:33:09.005 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 188.69614046199999, 2023-01-15T14:33:09.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:33:09.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:09.005 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:09.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:09.005 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:09.006 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:09.006 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:33:09.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:09.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:09.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:33:09.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:09.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:33:09.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:33:09.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:09.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:33:09.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:33:09.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:09.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:33:09.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:33:09.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:33:09.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:33:09.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:09.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:33:09.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:33:09.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:33:09.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:33:09.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:09.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:33:09.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:33:09.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:09.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1373 3.12 3:495d4bbc:::smithi09025264-16:head [write 2873523~591733] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 180.30402413600001, 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40949580099999999, 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:09.013 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:09.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:09.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:09.014 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:09.014 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1373 2023-01-15T14:33:09.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.014 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:09.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:09.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:33:09.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693440+0000", 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:09.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693808+0000", 2023-01-15T14:33:09.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000367742 2023-01-15T14:33:09.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:09.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693831+0000", 2023-01-15T14:33:09.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3213e-05 2023-01-15T14:33:09.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:09.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693846+0000", 2023-01-15T14:33:09.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4909000000000001e-05 2023-01-15T14:33:09.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.695833+0000", 2023-01-15T14:33:09.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019869240000000002 2023-01-15T14:33:09.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:09.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696121+0000", 2023-01-15T14:33:09.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028753300000000003 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798577+0000", 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102456808 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798849+0000", 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027138999999999999 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:33:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799155+0000", 2023-01-15T14:33:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030594800000000001 2023-01-15T14:33:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799614+0000", 2023-01-15T14:33:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045960700000000001 2023-01-15T14:33:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:09.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799737+0000", 2023-01-15T14:33:09.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012218900000000001 2023-01-15T14:33:09.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:09.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799998+0000", 2023-01-15T14:33:09.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026129700000000002 2023-01-15T14:33:09.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:09.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800639+0000", 2023-01-15T14:33:09.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064108800000000005 2023-01-15T14:33:09.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:09.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102023+0000", 2023-01-15T14:33:09.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30138417200000001 2023-01-15T14:33:09.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:09.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102198+0000", 2023-01-15T14:33:09.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017453600000000001 2023-01-15T14:33:09.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:09.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102398+0000", 2023-01-15T14:33:09.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020078000000000001 2023-01-15T14:33:09.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:09.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102937+0000", 2023-01-15T14:33:09.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053874499999999998 2023-01-15T14:33:09.022 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 180.303599542, 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:09.023 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:33:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:33:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:33:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:33:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:09.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:33:09.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:33:09.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:09.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:33:09.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:33:09.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:09.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:33:09.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:33:09.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:33:09.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:33:09.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:09.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:33:09.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:33:09.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:33:09.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:33:09.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:09.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:33:09.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:33:09.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:09.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:33:09.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:33:09.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:33:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:33:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:33:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:33:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:33:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:33:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:33:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:33:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:09.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:33:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:33:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:33:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:33:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:33:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:33:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 180.303547213, 2023-01-15T14:33:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:33:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:09.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:33:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:33:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:09.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:33:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:33:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:33:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:33:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:33:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:33:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:33:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:33:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:33:09.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:33:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:33:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:33:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:33:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:33:09.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:33:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:33:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:33:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:33:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.191 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 161.706634014, 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:09.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:09.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:33:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:33:09.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:33:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:33:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:33:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:33:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:33:09.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.198 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:09.198 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:09.198 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:33:09.199 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:09.199 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:09.200 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:09.200 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:09.200 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:09.201 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:33:09.201 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:33:09.201 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:33:09.201 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: filestore_inject_stall = '0' (not observed, change may require restart) osd_enable_op_tracker = 'true' 2023-01-15T14:33:09.201 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:33:09.201 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:33:09.201 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:33:09.202 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:33:09.202 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:33:09.203 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:08.064+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:09.244 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:09.237+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:09.390 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:09.390 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:09.391 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:33:09.391 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:09.391 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:33:09.392 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:09.381+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:09.490 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:09.488+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:09.753 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:09.753 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:09.754 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:33:09.754 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:33:09.754 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:09.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:33:09.755 INFO:tasks.rados.rados.0.smithi090.stdout:1194: finishing write tid 1 to smithi09025264-12 2023-01-15T14:33:09.755 INFO:tasks.rados.rados.0.smithi090.stdout:1194: finishing write tid 2 to smithi09025264-12 2023-01-15T14:33:09.755 INFO:tasks.rados.rados.0.smithi090.stdout:1194: finishing write tid 3 to smithi09025264-12 2023-01-15T14:33:09.755 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:09.745+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:09.761 INFO:tasks.rados.rados.0.smithi090.stdout:1194: finishing write tid 4 to smithi09025264-12 2023-01-15T14:33:09.763 INFO:tasks.rados.rados.0.smithi090.stdout:1194: finishing write tid 5 to smithi09025264-12 2023-01-15T14:33:09.763 INFO:tasks.rados.rados.0.smithi090.stdout:1194: oid 12 updating version 0 to 290 2023-01-15T14:33:09.763 INFO:tasks.rados.rados.0.smithi090.stdout:1194: oid 12 updating version 290 to 293 2023-01-15T14:33:09.763 INFO:tasks.rados.rados.0.smithi090.stdout:1194: oid 12 version 293 is already newer than 291 2023-01-15T14:33:09.763 INFO:tasks.rados.rados.0.smithi090.stdout:1194: oid 12 version 293 is already newer than 292 2023-01-15T14:33:09.763 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 293 (ObjNum 439 snap 121 seq_num 439) dirty exists 2023-01-15T14:33:09.764 INFO:tasks.rados.rados.0.smithi090.stdout:1194: left oid 12 (ObjNum 439 snap 121 seq_num 439) 2023-01-15T14:33:09.764 INFO:tasks.rados.rados.0.smithi090.stdout:1194: done (1 left) 2023-01-15T14:33:09.764 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:09.854 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:09.845+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:09.941 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:09.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.77877643400001, 2023-01-15T14:33:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:33:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:09.942 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:33:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:09.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:33:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:33:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:33:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:33:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:33:09.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:33:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:33:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:33:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:33:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:33:09.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:33:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:33:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:33:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:33:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:33:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:33:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:33:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:33:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:33:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:09.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:33:09.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:33:09.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:09.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:33:09.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:33:09.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:09.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:33:09.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.77847663699998, 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:09.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:33:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:33:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:33:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:33:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:09.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:33:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:33:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:33:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:33:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:33:09.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:33:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:33:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:33:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:33:09.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:33:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:33:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:33:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:09.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:33:09.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:33:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:33:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:33:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:33:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:33:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:33:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:33:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:33:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:09.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:33:09.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:33:09.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:09.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:33:09.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:33:09.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:09.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:33:09.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:33:09.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.965 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:09.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:09.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.965 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:09.965 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:33:09.966 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.77844685299999, 2023-01-15T14:33:09.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:33:09.966 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:09.966 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:09.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:09.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:09.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:09.966 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:33:09.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.966 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:09.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:33:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:33:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:33:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:33:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:33:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:09.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:09.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:33:09.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:33:09.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:33:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:33:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:33:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:33:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:33:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:33:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:33:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:33:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:33:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:33:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:33:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:33:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:33:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:33:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:33:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:33:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:33:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.64863805300001, 2023-01-15T14:33:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:33:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:33:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:33:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:33:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:33:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:33:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:33:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:33:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:33:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:33:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:33:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:33:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:33:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:33:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:33:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:33:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:33:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.64858135600002, 2023-01-15T14:33:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:33:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:33:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:33:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:33:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:33:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:33:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:33:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:33:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:33:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:33:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:33:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:33:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:33:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:33:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:33:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:10.124 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:224 3.0 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e25)", 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 280.03208518299999, 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59671704199999998, 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:10.125 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 224 2023-01-15T14:33:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:33:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831206+0000", 2023-01-15T14:33:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831205+0000", 2023-01-15T14:33:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:10.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831208+0000", 2023-01-15T14:33:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4620000000000001e-06 2023-01-15T14:33:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.831218+0000", 2023-01-15T14:33:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7429999999999997e-06 2023-01-15T14:33:10.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:29.931348+0000", 2023-01-15T14:33:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971599999999 2023-01-15T14:33:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025380+0000", 2023-01-15T14:33:10.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094032260000000006 2023-01-15T14:33:11.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.025493+0000", 2023-01-15T14:33:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001125 2023-01-15T14:33:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:11.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427384+0000", 2023-01-15T14:33:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.401891159 2023-01-15T14:33:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427654+0000", 2023-01-15T14:33:11.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027043400000000001 2023-01-15T14:33:11.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:11.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:11.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:11.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:30.427923+0000", 2023-01-15T14:33:11.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026842000000000001 2023-01-15T14:33:11.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:11.859 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:11.860 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:11.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:11.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:11.860 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:33:11.860 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:33:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 273.00323671199999, 2023-01-15T14:33:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:33:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.003 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:33:12.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:33:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:33:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:33:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:33:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:33:12.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:33:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:33:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:33:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:33:12.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:33:12.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:33:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:12.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:33:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:33:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:33:12.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:33:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:33:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:33:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:33:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:12.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:33:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:33:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:33:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:33:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:33:12.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:33:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:33:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:33:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:33:12.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:33:12.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:12.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:33:12.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:33:12.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:33:12.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:33:12.165 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.165 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.165 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.166 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:33:12.166 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:12.166 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.001949736, 2023-01-15T14:33:12.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:33:12.166 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.166 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:12.166 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.166 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.166 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.166 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:33:12.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.167 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:12.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:12.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:12.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:12.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:33:12.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:33:12.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:33:12.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:33:12.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:33:12.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:33:12.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:12.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:33:12.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:33:12.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:12.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:33:12.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:33:12.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:12.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:33:12.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:33:12.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:33:12.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:33:12.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:33:12.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:33:12.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:12.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:33:12.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:33:12.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:33:12.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:33:12.174 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.174 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:33:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:33:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 195.94137148600001, 2023-01-15T14:33:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:33:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.175 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.182 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.182 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:33:12.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:33:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:33:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:33:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:33:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:33:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:33:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:33:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:33:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:33:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:33:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:33:12.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:33:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:33:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:33:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:33:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:33:12.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:33:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:33:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:33:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:33:12.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:33:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:33:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:33:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:33:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:33:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:33:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:33:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:33:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.328 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:33:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:33:12.330 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 195.94131277599999, 2023-01-15T14:33:12.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:33:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:33:12.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:33:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:33:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:33:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:33:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:33:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:33:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:33:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:33:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:33:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:33:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:33:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:33:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:33:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:12.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:33:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:33:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:33:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:33:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:33:12.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:33:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:33:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:33:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.351 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:33:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 191.98489328299999, 2023-01-15T14:33:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:33:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:33:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.352 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:33:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:33:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:33:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:33:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:33:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:33:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:33:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:33:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:33:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:33:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:33:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:33:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:33:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:33:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:33:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:33:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:33:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:33:12.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:33:12.480 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.480 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.480 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.480 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:12.480 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:33:12.480 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 191.92739560000001, 2023-01-15T14:33:12.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:33:12.481 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.481 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:12.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:33:12.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.482 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:33:12.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:33:12.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:33:12.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:12.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:33:12.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:33:12.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:33:12.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:33:12.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:33:12.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:33:12.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:33:12.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:33:12.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:12.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:33:12.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:33:12.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:33:12.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:33:12.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:33:12.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:12.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:33:12.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:33:12.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:12.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:33:12.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:33:12.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:12.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:33:12.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:33:12.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:33:12.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:33:12.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:33:12.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:33:12.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:12.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:33:12.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:33:12.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:33:12.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:33:12.504 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.504 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.504 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.504 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:33:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 170.55874409500001, 2023-01-15T14:33:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:33:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.505 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:33:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:33:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:33:12.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:33:12.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:33:12.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:33:12.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:33:12.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:33:12.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:33:12.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:33:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:33:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:33:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:33:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:12.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.513 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 170.558641454, 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:33:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:33:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:33:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:33:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:33:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:33:12.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:33:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:33:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:33:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:33:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:33:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:12.521 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.957659459, 2023-01-15T14:33:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:33:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:33:12.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:33:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:33:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:33:12.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:33:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:33:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:33:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:33:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:33:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:33:12.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:12.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:33:12.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:33:12.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:12.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:33:12.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:33:12.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:12.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:33:12.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:33:12.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:12.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:33:12.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:33:12.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:33:12.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:33:12.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:33:12.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:33:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:33:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:33:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:33:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:33:12.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:33:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:33:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 125.957634035, 2023-01-15T14:33:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:33:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.667 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:12.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.668 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.668 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:33:12.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.668 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:33:12.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:33:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:33:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:33:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:33:12.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:33:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:33:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:33:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:33:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:33:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:33:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:33:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:33:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:33:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:33:12.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:33:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:33:12.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:33:12.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:33:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:33:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:33:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:33:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:33:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:33:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:33:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:33:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:33:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:33:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:33:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:33:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:33:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:33:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:33:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:33:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:33:12.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:33:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:33:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:33:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.815 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2301 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e216)", 2023-01-15T14:33:12.817 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:33:12.818 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 86.667759567999994, 2023-01-15T14:33:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61754771200000003, 2023-01-15T14:33:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.819 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.820 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2301 2023-01-15T14:33:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:33:12.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:33:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195530+0000", 2023-01-15T14:33:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:12.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195535+0000", 2023-01-15T14:33:12.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3660000000000002e-06 2023-01-15T14:33:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195546+0000", 2023-01-15T14:33:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1062e-05 2023-01-15T14:33:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195551+0000", 2023-01-15T14:33:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2023-01-15T14:33:12.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195999+0000", 2023-01-15T14:33:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044799800000000002 2023-01-15T14:33:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.196106+0000", 2023-01-15T14:33:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106172 2023-01-15T14:33:12.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510631+0000", 2023-01-15T14:33:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31452575999999999 2023-01-15T14:33:12.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510875+0000", 2023-01-15T14:33:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024415199999999998 2023-01-15T14:33:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812445+0000", 2023-01-15T14:33:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30156923000000002 2023-01-15T14:33:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812727+0000", 2023-01-15T14:33:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028227800000000001 2023-01-15T14:33:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.813079+0000", 2023-01-15T14:33:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199599999999998 2023-01-15T14:33:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.837 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2667 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e259)", 2023-01-15T14:33:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 43.978869672999998, 2023-01-15T14:33:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60917505800000005, 2023-01-15T14:33:12.838 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2667 2023-01-15T14:33:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.839 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967296 2023-01-15T14:33:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884423+0000", 2023-01-15T14:33:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.905e-06 2023-01-15T14:33:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884427+0000", 2023-01-15T14:33:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7689999999999998e-06 2023-01-15T14:33:12.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884429+0000", 2023-01-15T14:33:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5509999999999998e-06 2023-01-15T14:33:12.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.080096+0000", 2023-01-15T14:33:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19566640099999999 2023-01-15T14:33:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.080290+0000", 2023-01-15T14:33:12.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194346 2023-01-15T14:33:12.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.275924+0000", 2023-01-15T14:33:12.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19563417299999999 2023-01-15T14:33:12.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:12.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.276184+0000", 2023-01-15T14:33:12.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025966800000000001 2023-01-15T14:33:12.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493228+0000", 2023-01-15T14:33:12.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21704441599999999 2023-01-15T14:33:12.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493378+0000", 2023-01-15T14:33:12.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014980000000000001 2023-01-15T14:33:12.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:12.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493596+0000", 2023-01-15T14:33:12.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021825300000000001 2023-01-15T14:33:12.962 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.962 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:12.962 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:12.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.963 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2748 3.3d 3:bc1c6b44:::smithi09025264-23:head [setxattr _header (58) in=7b,truncate 2537923] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:33:12.963 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:33:12.963 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.941896595999999, 2023-01-15T14:33:12.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59151328599999997, 2023-01-15T14:33:12.963 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:12.963 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:12.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:12.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:12.963 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2748 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:33:12.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921394+0000", 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921398+0000", 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3139999999999997e-06 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:12.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921407+0000", 2023-01-15T14:33:12.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7050000000000005e-06 2023-01-15T14:33:12.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:12.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921410+0000", 2023-01-15T14:33:12.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7459999999999998e-06 2023-01-15T14:33:12.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.007186+0000", 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.085775343000000004 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.007335+0000", 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014914400000000001 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.209977+0000", 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20264216900000001 2023-01-15T14:33:12.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:12.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.210228+0000", 2023-01-15T14:33:12.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025044800000000001 2023-01-15T14:33:12.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:12.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.210911+0000", 2023-01-15T14:33:12.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00068313300000000005 2023-01-15T14:33:12.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:12.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:12.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:13.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.211407+0000", 2023-01-15T14:33:13.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049586300000000003 2023-01-15T14:33:13.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:13.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.211425+0000", 2023-01-15T14:33:13.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8403e-05 2023-01-15T14:33:13.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:13.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212523+0000", 2023-01-15T14:33:13.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010975710000000001 2023-01-15T14:33:13.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:13.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.213095+0000", 2023-01-15T14:33:13.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057295100000000004 2023-01-15T14:33:13.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:13.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.250158+0000", 2023-01-15T14:33:13.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.037062401000000002 2023-01-15T14:33:13.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:13.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.512701+0000", 2023-01-15T14:33:13.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.26254333400000002 2023-01-15T14:33:13.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:13.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.512801+0000", 2023-01-15T14:33:13.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9333999999999995e-05 2023-01-15T14:33:13.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:13.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.512908+0000", 2023-01-15T14:33:13.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106995 2023-01-15T14:33:13.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:13.149 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:13.283 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:13.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.284 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2749 3.3d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:33:13.284 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:13.284 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 38.941874792, 2023-01-15T14:33:13.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.69206326600000001, 2023-01-15T14:33:13.284 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:13.284 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:13.284 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:13.284 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2749 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:13.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:13.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:13.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:13.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:13.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921418+0000", 2023-01-15T14:33:13.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1670000000000002e-06 2023-01-15T14:33:13.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:13.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921426+0000", 2023-01-15T14:33:13.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0569999999999993e-06 2023-01-15T14:33:13.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921429+0000", 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9629999999999998e-06 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107302+0000", 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.185873327 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107657+0000", 2023-01-15T14:33:13.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035504299999999998 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212694+0000", 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105037095 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212936+0000", 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241492 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:13.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613125+0000", 2023-01-15T14:33:13.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40018927900000001 2023-01-15T14:33:13.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:13.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613294+0000", 2023-01-15T14:33:13.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016838000000000001 2023-01-15T14:33:13.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:13.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613479+0000", 2023-01-15T14:33:13.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018577599999999999 2023-01-15T14:33:13.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:13.293 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:13.293 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:13.293 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:13.293 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:13.293 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:13.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:33:13.294 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:33:13.294 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:33:13.294 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:33:13.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:33:13.295 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:33:13.296 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:13.292+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:13.309 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:33:13.313 INFO:tasks.rados.rados.0.smithi090.stdout:1190: finishing rollback tid 1 to smithi09025264-32 2023-01-15T14:33:13.313 INFO:tasks.rados.rados.0.smithi090.stdout:1190: finishing rollback tid 2 to smithi09025264-32 2023-01-15T14:33:13.314 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 279 (ObjNum 404 snap 108 seq_num 404) dirty exists 2023-01-15T14:33:13.314 INFO:tasks.rados.rados.0.smithi090.stdout:1190: done (0 left) 2023-01-15T14:33:13.314 INFO:tasks.rados.rados.0.smithi090.stdout:1200: done (0 left) 2023-01-15T14:33:13.314 INFO:tasks.rados.rados.0.smithi090.stdout:1201: read oid 37 snap -1 2023-01-15T14:33:13.315 INFO:tasks.rados.rados.0.smithi090.stdout:1201: expect deleted 2023-01-15T14:33:13.315 INFO:tasks.rados.rados.0.smithi090.stdout:1202: read oid 48 snap 119 2023-01-15T14:33:13.315 INFO:tasks.rados.rados.0.smithi090.stdout:1202: expect (ObjNum 436 snap 118 seq_num 436) 2023-01-15T14:33:13.315 INFO:tasks.rados.rados.0.smithi090.stdout:1203: delete oid 44 current snap is 122 2023-01-15T14:33:13.315 INFO:tasks.rados.rados.0.smithi090.stdout:1202: expect (ObjNum 436 snap 118 seq_num 436) 2023-01-15T14:33:13.315 INFO:tasks.rados.rados.0.smithi090.stdout:1201: done (2 left) 2023-01-15T14:33:13.315 INFO:tasks.rados.rados.0.smithi090.stdout:1202: done (1 left) 2023-01-15T14:33:13.315 INFO:tasks.rados.rados.0.smithi090.stdout:1203: done (0 left) 2023-01-15T14:33:13.315 INFO:tasks.rados.rados.0.smithi090.stdout:1204: read oid 49 snap -1 2023-01-15T14:33:13.315 INFO:tasks.rados.rados.0.smithi090.stdout:1204: expect deleted 2023-01-15T14:33:13.316 INFO:tasks.rados.rados.0.smithi090.stdout:1205: write oid 3 current snap is 122 2023-01-15T14:33:13.316 INFO:tasks.rados.rados.0.smithi090.stdout:1205: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2023-01-15T14:33:13.316 INFO:tasks.rados.rados.0.smithi090.stdout:1205: writing smithi09025264-3 from 662308 to 1221338 tid 1 2023-01-15T14:33:13.316 INFO:tasks.rados.rados.0.smithi090.stdout:1205: writing smithi09025264-3 from 1632005 to 2235816 tid 2 2023-01-15T14:33:13.316 INFO:tasks.rados.rados.0.smithi090.stdout:1205: writing smithi09025264-3 from 2860044 to 3462308 tid 3 2023-01-15T14:33:13.316 INFO:tasks.rados.rados.0.smithi090.stdout:1206: copy_from oid 48 from oid 42 current snap is 122 2023-01-15T14:33:13.316 INFO:tasks.rados.rados.0.smithi090.stdout:1207: copy_from oid 24 from oid 13 current snap is 122 2023-01-15T14:33:13.316 INFO:tasks.rados.rados.0.smithi090.stdout:1208: snap_create 2023-01-15T14:33:13.316 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:33:13.317 INFO:tasks.rados.rados.0.smithi090.stdout:1204: done (3 left) 2023-01-15T14:33:13.317 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:13.317 INFO:tasks.rados.rados.0.smithi090.stdout:1206: finishing copy_from to smithi09025264-48 2023-01-15T14:33:13.544 INFO:tasks.rados.rados.0.smithi090.stdout:1206: got expected ENOENT (src dne) 2023-01-15T14:33:13.544 INFO:tasks.rados.rados.0.smithi090.stdout:1206: finishing copy_from racing read to smithi09025264-48 2023-01-15T14:33:13.544 INFO:tasks.rados.rados.0.smithi090.stdout:1206: done (2 left) 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:1205: finishing write tid 1 to smithi09025264-3 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:1205: finishing write tid 2 to smithi09025264-3 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:1205: finishing write tid 3 to smithi09025264-3 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:1205: finishing write tid 4 to smithi09025264-3 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:1205: finishing write tid 5 to smithi09025264-3 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:1205: oid 3 updating version 0 to 328 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:1205: oid 3 version 328 is already newer than 327 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:1205: oid 3 version 328 is already newer than 326 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:1205: oid 3 updating version 328 to 329 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 329 (ObjNum 444 snap 122 seq_num 444) dirty exists 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:1205: left oid 3 (ObjNum 444 snap 122 seq_num 444) 2023-01-15T14:33:13.545 INFO:tasks.rados.rados.0.smithi090.stdout:1205: done (1 left) 2023-01-15T14:33:13.546 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:13.546 INFO:tasks.rados.rados.0.smithi090.stdout:1207: finishing copy_from to smithi09025264-24 2023-01-15T14:33:13.546 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 302 (ObjNum 412 snap 112 seq_num 412) dirty exists 2023-01-15T14:33:13.546 INFO:tasks.rados.rados.0.smithi090.stdout:1207: finishing copy_from racing read to smithi09025264-24 2023-01-15T14:33:13.546 INFO:tasks.rados.rados.0.smithi090.stdout:1207: done (0 left) 2023-01-15T14:33:13.548 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:13.544+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:13.649 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:13.641+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:13.715 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.715 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "replica_scrub(pg: 3.3f,from:0'0,to:297'220,epoch:304/235,start:3:fc000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2023-01-15T14:33:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:13.598879+0000", 2023-01-15T14:33:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.060810708999999998, 2023-01-15T14:33:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.060848988999999999, 2023-01-15T14:33:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:13.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:13.598879+0000", 2023-01-15T14:33:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:13.598879+0000", 2023-01-15T14:33:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:13.598877+0000", 2023-01-15T14:33:13.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:13.598894+0000", 2023-01-15T14:33:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6511000000000002e-05 2023-01-15T14:33:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:13.598936+0000", 2023-01-15T14:33:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2373000000000002e-05 2023-01-15T14:33:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:13.598953+0000", 2023-01-15T14:33:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6790999999999999e-05 2023-01-15T14:33:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:13.719 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:13.720 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:33:13.720 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:13.720 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:33:13.721 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:13.721 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:13.722 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:13.737 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:13.971 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:13.971 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:13.972 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:13.972 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:13.972 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:13.972 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:33:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:33:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:33:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:33:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:33:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:33:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:33:13.973 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:33:13.975 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:33:13.978 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:33:13.980 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:33:13.980 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2023-01-15T14:33:12.931075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":179833488,"num_objects":85,"num_object_clones":40,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":8,"num_read":945,"num_read_kb":889463,"num_write":824,"num_write_kb":254847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":350,"num_bytes_recovered":215397959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12831,"ondisk_log_size":12831,"up":255,"acting":254,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":258,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":5409720,"kb_used_data":556814060,"kb_used_omap":50963,"kb_used_meta":0,"kb_avail":556786560,"statfs":{"total":575688990720,"available":570149437440,"internally_reserved":28160000,"allocated":570177597440,"data_stored":570177597440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":52186462,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6180145,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-38,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":16256,"num_write":14,"num_write_kb":6038,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":23977067,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001190"},"pg_stats":[{"pgid":"3.4b","version":"155'53","reported_seq":639,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.163794+0000","last_change":"2023-01-15T14:33:06.337410+0000","last_active":"2023-01-15T14:33:11.163794+0000","last_peered":"2023-01-15T14:33:11.163794+0000","last_clean":"2023-01-15T14:33:11.163794+0000","last_became_active":"2023-01-15T14:32:25.161408+0000","last_became_peered":"2023-01-15T14:32:25.161408+0000","last_unstale":"2023-01-15T14:33:11.163794+0000","last_undegraded":"2023-01-15T14:33:11.163794+0000","last_fullsized":"2023-01-15T14:33:11.163794+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:40.131","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31388428000000002,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.4a","version":"303'329","reported_seq":1327,"reported_epoch":303,"state":"active+clean","last_fresh":"2023-01-15T14:33:12.669704+0000","last_change":"2023-01-15T14:33:06.089702+0000","last_active":"2023-01-15T14:33:12.669704+0000","last_peered":"2023-01-15T14:33:12.669704+0000","last_clean":"2023-01-15T14:33:12.669704+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:33:12.669704+0000","last_undegraded":"2023-01-15T14:33:12.669704+0000","last_fullsized":"2023-01-15T14:33:12.669704+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:33.185","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.104224267,"stat_sum":{"num_bytes":7395615,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":34133,"num_write":53,"num_write_kb":17429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":1020,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.264207+0000","last_change":"2023-01-15T14:33:10.269527+0000","last_active":"2023-01-15T14:33:11.264207+0000","last_peered":"2023-01-15T14:33:11.264207+0000","last_clean":"2023-01-15T14:33:11.264207+0000","last_became_active":"2023-01-15T14:32:28.259491+0000","last_became_peered":"2023-01-15T14:32:28.259491+0000","last_unstale":"2023-01-15T14:33:11.264207+0000","last_undegraded":"2023-01-15T14:33:11.264207+0000","last_fullsized":"2023-01-15T14:33:11.264207+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:31:44.640345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.640345+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:15.269","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.202384547,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.48","version":"179'204","reported_seq":1224,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.053089+0000","last_change":"2023-01-15T14:33:09.761449+0000","last_active":"2023-01-15T14:33:11.053089+0000","last_peered":"2023-01-15T14:33:11.053089+0000","last_clean":"2023-01-15T14:33:11.053089+0000","last_became_active":"2023-01-15T14:32:28.187902+0000","last_became_peered":"2023-01-15T14:32:28.187902+0000","last_unstale":"2023-01-15T14:33:11.053089+0000","last_undegraded":"2023-01-15T14:33:11.053089+0000","last_fullsized":"2023-01-15T14:33:11.053089+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.83223010200000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1014,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.165391+0000","last_change":"2023-01-15T14:33:06.231856+0000","last_active":"2023-01-15T14:33:11.165391+0000","last_peered":"2023-01-15T14:33:11.165391+0000","last_clean":"2023-01-15T14:33:11.165391+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:33:11.165391+0000","last_undegraded":"2023-01-15T14:33:11.165391+0000","last_fullsized":"2023-01-15T14:33:11.165391+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:03.200","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.10206523200000001,"stat_sum":{"num_bytes":496762,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":5,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":914,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.266438+0000","last_change":"2023-01-15T14:33:06.444430+0000","last_active":"2023-01-15T14:33:11.266438+0000","last_peered":"2023-01-15T14:33:11.266438+0000","last_clean":"2023-01-15T14:33:11.266438+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:33:11.266438+0000","last_undegraded":"2023-01-15T14:33:11.266438+0000","last_fullsized":"2023-01-15T14:33:11.266438+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:31:25.342830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:25.342830+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:32.011","scrub_duration":0.20886623600000001,"objects_trimmed":0,"snaptrim_duration":0.35853839599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":1025,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.955085+0000","last_change":"2023-01-15T14:33:06.298621+0000","last_active":"2023-01-15T14:33:10.955085+0000","last_peered":"2023-01-15T14:33:10.955085+0000","last_clean":"2023-01-15T14:33:10.955085+0000","last_became_active":"2023-01-15T14:32:28.162971+0000","last_became_peered":"2023-01-15T14:32:28.162971+0000","last_unstale":"2023-01-15T14:33:10.955085+0000","last_undegraded":"2023-01-15T14:33:10.955085+0000","last_fullsized":"2023-01-15T14:33:10.955085+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:50.736","scrub_duration":0.206392353,"objects_trimmed":0,"snaptrim_duration":0.31236207999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1125,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.045048+0000","last_change":"2023-01-15T14:33:09.244196+0000","last_active":"2023-01-15T14:33:11.045048+0000","last_peered":"2023-01-15T14:33:11.045048+0000","last_clean":"2023-01-15T14:33:11.045048+0000","last_became_active":"2023-01-15T14:33:09.135859+0000","last_became_peered":"2023-01-15T14:33:09.135859+0000","last_unstale":"2023-01-15T14:33:11.045048+0000","last_undegraded":"2023-01-15T14:33:11.045048+0000","last_fullsized":"2023-01-15T14:33:11.045048+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.107219118,"stat_sum":{"num_bytes":2887217,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":30681,"num_write":15,"num_write_kb":5036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7931054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.43","version":"297'314","reported_seq":1362,"reported_epoch":301,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.040893+0000","last_change":"2023-01-15T14:33:05.922782+0000","last_active":"2023-01-15T14:33:10.040893+0000","last_peered":"2023-01-15T14:33:10.040893+0000","last_clean":"2023-01-15T14:33:10.040893+0000","last_became_active":"2023-01-15T14:32:28.977084+0000","last_became_peered":"2023-01-15T14:32:28.977084+0000","last_unstale":"2023-01-15T14:33:10.040893+0000","last_undegraded":"2023-01-15T14:33:10.040893+0000","last_fullsized":"2023-01-15T14:33:10.040893+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:02.589","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.0066329479999999996,"stat_sum":{"num_bytes":3395080,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.42","version":"155'101","reported_seq":910,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.953810+0000","last_change":"2023-01-15T14:33:09.749856+0000","last_active":"2023-01-15T14:33:10.953810+0000","last_peered":"2023-01-15T14:33:10.953810+0000","last_clean":"2023-01-15T14:33:10.953810+0000","last_became_active":"2023-01-15T14:32:25.163911+0000","last_became_peered":"2023-01-15T14:32:25.163911+0000","last_unstale":"2023-01-15T14:33:10.953810+0000","last_undegraded":"2023-01-15T14:33:10.953810+0000","last_fullsized":"2023-01-15T14:33:10.953810+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.81945557099999999,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":990,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.164422+0000","last_change":"2023-01-15T14:33:06.083978+0000","last_active":"2023-01-15T14:33:11.164422+0000","last_peered":"2023-01-15T14:33:11.164422+0000","last_clean":"2023-01-15T14:33:11.164422+0000","last_became_active":"2023-01-15T14:31:56.927677+0000","last_became_peered":"2023-01-15T14:31:56.927677+0000","last_unstale":"2023-01-15T14:33:11.164422+0000","last_undegraded":"2023-01-15T14:33:11.164422+0000","last_fullsized":"2023-01-15T14:33:11.164422+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:56.100","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.10346578300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.40","version":"300'293","reported_seq":1263,"reported_epoch":303,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:33:12.002828+0000","last_change":"2023-01-15T14:33:12.003428+0000","last_active":"2023-01-15T14:33:12.002828+0000","last_peered":"2023-01-15T14:33:12.002828+0000","last_clean":"2023-01-15T14:33:03.973511+0000","last_became_active":"2023-01-15T14:33:09.136586+0000","last_became_peered":"2023-01-15T14:33:09.136586+0000","last_unstale":"2023-01-15T14:33:12.002828+0000","last_undegraded":"2023-01-15T14:33:08.931835+0000","last_fullsized":"2023-01-15T14:33:08.929002+0000","mapping_epoch":299,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":1.0083311509999999,"stat_sum":{"num_bytes":5896549,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":8741,"num_write":25,"num_write_kb":7603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":16413587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,2],"avail_no_missing":["3","2","4","7"],"object_location_counts":[{"shards":"2,3","objects":3}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"}]},{"pgid":"3.3f","version":"297'220","reported_seq":1196,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.958261+0000","last_change":"2023-01-15T14:33:06.194481+0000","last_active":"2023-01-15T14:33:10.958261+0000","last_peered":"2023-01-15T14:33:10.958261+0000","last_clean":"2023-01-15T14:33:10.958261+0000","last_became_active":"2023-01-15T14:32:02.943089+0000","last_became_peered":"2023-01-15T14:32:02.943089+0000","last_unstale":"2023-01-15T14:33:10.958261+0000","last_undegraded":"2023-01-15T14:33:10.958261+0000","last_fullsized":"2023-01-15T14:33:10.958261+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20837445600000001,"stat_sum":{"num_bytes":3484071,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":108167,"num_write":39,"num_write_kb":10479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":25819820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.3e","version":"294'63","reported_seq":729,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.165526+0000","last_change":"2023-01-15T14:33:06.216481+0000","last_active":"2023-01-15T14:33:11.165526+0000","last_peered":"2023-01-15T14:33:11.165526+0000","last_clean":"2023-01-15T14:33:11.165526+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:33:11.165526+0000","last_undegraded":"2023-01-15T14:33:11.165526+0000","last_fullsized":"2023-01-15T14:33:11.165526+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:42.278","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.233103964,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.3d","version":"283'223","reported_seq":1207,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.164278+0000","last_change":"2023-01-15T14:33:06.228385+0000","last_active":"2023-01-15T14:33:11.164278+0000","last_peered":"2023-01-15T14:33:11.164278+0000","last_clean":"2023-01-15T14:33:11.164278+0000","last_became_active":"2023-01-15T14:31:59.249241+0000","last_became_peered":"2023-01-15T14:31:59.249241+0000","last_unstale":"2023-01-15T14:33:11.164278+0000","last_undegraded":"2023-01-15T14:33:11.164278+0000","last_fullsized":"2023-01-15T14:33:11.164278+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:12.136","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.24440979800000001,"stat_sum":{"num_bytes":5496021,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":10,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.3c","version":"141'46","reported_seq":234,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.954376+0000","last_change":"2023-01-15T14:33:09.751077+0000","last_active":"2023-01-15T14:33:10.954376+0000","last_peered":"2023-01-15T14:33:10.954376+0000","last_clean":"2023-01-15T14:33:10.954376+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:33:10.954376+0000","last_undegraded":"2023-01-15T14:33:10.954376+0000","last_fullsized":"2023-01-15T14:33:10.954376+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.71770830500000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":579834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":889,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.165044+0000","last_change":"2023-01-15T14:33:06.085387+0000","last_active":"2023-01-15T14:33:11.165044+0000","last_peered":"2023-01-15T14:33:11.165044+0000","last_clean":"2023-01-15T14:33:11.165044+0000","last_became_active":"2023-01-15T14:31:54.237985+0000","last_became_peered":"2023-01-15T14:31:54.237985+0000","last_unstale":"2023-01-15T14:33:11.165044+0000","last_undegraded":"2023-01-15T14:33:11.165044+0000","last_fullsized":"2023-01-15T14:33:11.165044+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:52.139","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.10487854200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1233,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.163060+0000","last_change":"2023-01-15T14:33:06.090608+0000","last_active":"2023-01-15T14:33:11.163060+0000","last_peered":"2023-01-15T14:33:11.163060+0000","last_clean":"2023-01-15T14:33:11.163060+0000","last_became_active":"2023-01-15T14:31:57.025706+0000","last_became_peered":"2023-01-15T14:31:57.025706+0000","last_unstale":"2023-01-15T14:33:11.163060+0000","last_undegraded":"2023-01-15T14:33:11.163060+0000","last_fullsized":"2023-01-15T14:33:11.163060+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:30.639","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.10756581699999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.39","version":"294'319","reported_seq":1245,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.953941+0000","last_change":"2023-01-15T14:33:06.088513+0000","last_active":"2023-01-15T14:33:10.953941+0000","last_peered":"2023-01-15T14:33:10.953941+0000","last_clean":"2023-01-15T14:33:10.953941+0000","last_became_active":"2023-01-15T14:32:06.420117+0000","last_became_peered":"2023-01-15T14:32:06.420117+0000","last_unstale":"2023-01-15T14:33:10.953941+0000","last_undegraded":"2023-01-15T14:33:10.953941+0000","last_fullsized":"2023-01-15T14:33:10.953941+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":6,"last_scrub":"135'157","last_scrub_stamp":"2023-01-15T14:30:26.792736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:26.792736+0000","objects_scrubbed":4,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:12.259","scrub_duration":0.61998826900000004,"objects_trimmed":0,"snaptrim_duration":0.10323133,"stat_sum":{"num_bytes":10343584,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":54,"num_read_kb":65220,"num_write":38,"num_write_kb":18960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":31983837,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.38","version":"298'291","reported_seq":1395,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.029951+0000","last_change":"2023-01-15T14:33:09.872906+0000","last_active":"2023-01-15T14:33:11.029951+0000","last_peered":"2023-01-15T14:33:11.029951+0000","last_clean":"2023-01-15T14:33:11.029951+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:33:11.029951+0000","last_undegraded":"2023-01-15T14:33:11.029951+0000","last_fullsized":"2023-01-15T14:33:11.029951+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:55.610","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.93998603999999997,"stat_sum":{"num_bytes":386561,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":23,"num_read_kb":18939,"num_write":23,"num_write_kb":9325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":296,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.058590+0000","last_change":"2023-01-15T14:33:06.299819+0000","last_active":"2023-01-15T14:33:11.058590+0000","last_peered":"2023-01-15T14:33:11.058590+0000","last_clean":"2023-01-15T14:33:11.058590+0000","last_became_active":"2023-01-15T14:31:37.024453+0000","last_became_peered":"2023-01-15T14:31:37.024453+0000","last_unstale":"2023-01-15T14:33:11.058590+0000","last_undegraded":"2023-01-15T14:33:11.058590+0000","last_fullsized":"2023-01-15T14:33:11.058590+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.31340219200000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":457,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.165726+0000","last_change":"2023-01-15T14:33:06.219296+0000","last_active":"2023-01-15T14:33:11.165726+0000","last_peered":"2023-01-15T14:33:11.165726+0000","last_clean":"2023-01-15T14:33:11.165726+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:33:11.165726+0000","last_undegraded":"2023-01-15T14:33:11.165726+0000","last_fullsized":"2023-01-15T14:33:11.165726+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.23574250899999999,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":885,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.954125+0000","last_change":"2023-01-15T14:33:09.863267+0000","last_active":"2023-01-15T14:33:10.954125+0000","last_peered":"2023-01-15T14:33:10.954125+0000","last_clean":"2023-01-15T14:33:10.954125+0000","last_became_active":"2023-01-15T14:31:38.033824+0000","last_became_peered":"2023-01-15T14:31:38.033824+0000","last_unstale":"2023-01-15T14:33:10.954125+0000","last_undegraded":"2023-01-15T14:33:10.954125+0000","last_fullsized":"2023-01-15T14:33:10.954125+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:12.355","scrub_duration":0.282936307,"objects_trimmed":0,"snaptrim_duration":0.88321667500000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.34","version":"169'138","reported_seq":1053,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.264708+0000","last_change":"2023-01-15T14:33:06.185097+0000","last_active":"2023-01-15T14:33:11.264708+0000","last_peered":"2023-01-15T14:33:11.264708+0000","last_clean":"2023-01-15T14:33:11.264708+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:33:11.264708+0000","last_undegraded":"2023-01-15T14:33:11.264708+0000","last_fullsized":"2023-01-15T14:33:11.264708+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:59.872","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.203607066,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1127,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.953318+0000","last_change":"2023-01-15T14:33:09.864000+0000","last_active":"2023-01-15T14:33:10.953318+0000","last_peered":"2023-01-15T14:33:10.953318+0000","last_clean":"2023-01-15T14:33:10.953318+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:33:10.953318+0000","last_undegraded":"2023-01-15T14:33:10.953318+0000","last_fullsized":"2023-01-15T14:33:10.953318+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:54.578","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.933817597,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.32","version":"243'157","reported_seq":729,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.263060+0000","last_change":"2023-01-15T14:33:06.096954+0000","last_active":"2023-01-15T14:33:11.263060+0000","last_peered":"2023-01-15T14:33:11.263060+0000","last_clean":"2023-01-15T14:33:11.263060+0000","last_became_active":"2023-01-15T14:31:36.921952+0000","last_became_peered":"2023-01-15T14:31:36.921952+0000","last_unstale":"2023-01-15T14:33:11.263060+0000","last_undegraded":"2023-01-15T14:33:11.263060+0000","last_fullsized":"2023-01-15T14:33:11.263060+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.113685745,"stat_sum":{"num_bytes":2403401,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.13","version":"295'355","reported_seq":1243,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.054513+0000","last_change":"2023-01-15T14:33:06.296110+0000","last_active":"2023-01-15T14:33:11.054513+0000","last_peered":"2023-01-15T14:33:11.054513+0000","last_clean":"2023-01-15T14:33:11.054513+0000","last_became_active":"2023-01-15T14:31:35.097414+0000","last_became_peered":"2023-01-15T14:31:35.097414+0000","last_unstale":"2023-01-15T14:33:11.054513+0000","last_undegraded":"2023-01-15T14:33:11.054513+0000","last_fullsized":"2023-01-15T14:33:11.054513+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.30996936400000002,"stat_sum":{"num_bytes":12543369,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":34,"num_read_kb":48101,"num_write":23,"num_write_kb":14991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.12","version":"259'164","reported_seq":725,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.954147+0000","last_change":"2023-01-15T14:33:09.871401+0000","last_active":"2023-01-15T14:33:10.954147+0000","last_peered":"2023-01-15T14:33:10.954147+0000","last_clean":"2023-01-15T14:33:10.954147+0000","last_became_active":"2023-01-15T14:31:35.028565+0000","last_became_peered":"2023-01-15T14:31:35.028565+0000","last_unstale":"2023-01-15T14:33:10.954147+0000","last_undegraded":"2023-01-15T14:33:10.954147+0000","last_fullsized":"2023-01-15T14:33:10.954147+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:27.929","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.94051757899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.11","version":"302'279","reported_seq":1325,"reported_epoch":303,"state":"peering","last_fresh":"2023-01-15T14:33:11.868067+0000","last_change":"2023-01-15T14:33:11.868427+0000","last_active":"2023-01-15T14:33:11.868012+0000","last_peered":"2023-01-15T14:33:11.360914+0000","last_clean":"2023-01-15T14:33:02.964282+0000","last_became_active":"2023-01-15T14:33:07.331008+0000","last_became_peered":"2023-01-15T14:33:07.331008+0000","last_unstale":"2023-01-15T14:33:11.868067+0000","last_undegraded":"2023-01-15T14:33:11.868067+0000","last_fullsized":"2023-01-15T14:33:11.868067+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.001290668,"stat_sum":{"num_bytes":10521446,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":45,"num_read_kb":47511,"num_write":35,"num_write_kb":14687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":13311474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,3,4],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"da","length":"1"},{"start":"dd","length":"1"}]},{"pgid":"3.10","version":"298'399","reported_seq":1366,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.130484+0000","last_change":"2023-01-15T14:33:09.876893+0000","last_active":"2023-01-15T14:33:11.130484+0000","last_peered":"2023-01-15T14:33:11.130484+0000","last_clean":"2023-01-15T14:33:11.130484+0000","last_became_active":"2023-01-15T14:31:35.135778+0000","last_became_peered":"2023-01-15T14:31:35.135778+0000","last_unstale":"2023-01-15T14:33:11.130484+0000","last_undegraded":"2023-01-15T14:33:11.130484+0000","last_fullsized":"2023-01-15T14:33:11.130484+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:53.576","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.84251846600000002,"stat_sum":{"num_bytes":12685848,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":73,"num_read_kb":56735,"num_write":88,"num_write_kb":31795,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":13340457,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.f","version":"155'145","reported_seq":1106,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.266113+0000","last_change":"2023-01-15T14:33:06.250500+0000","last_active":"2023-01-15T14:33:11.266113+0000","last_peered":"2023-01-15T14:33:11.266113+0000","last_clean":"2023-01-15T14:33:11.266113+0000","last_became_active":"2023-01-15T14:31:35.032038+0000","last_became_peered":"2023-01-15T14:31:35.032038+0000","last_unstale":"2023-01-15T14:33:11.266113+0000","last_undegraded":"2023-01-15T14:33:11.266113+0000","last_fullsized":"2023-01-15T14:33:11.266113+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:31:22.549180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:22.549180+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:15.089","scrub_duration":0.50470070300000003,"objects_trimmed":0,"snaptrim_duration":0.120188978,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.e","version":"139'38","reported_seq":907,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.265589+0000","last_change":"2023-01-15T14:33:06.188444+0000","last_active":"2023-01-15T14:33:11.265589+0000","last_peered":"2023-01-15T14:33:11.265589+0000","last_clean":"2023-01-15T14:33:11.265589+0000","last_became_active":"2023-01-15T14:31:35.237663+0000","last_became_peered":"2023-01-15T14:31:35.237663+0000","last_unstale":"2023-01-15T14:33:11.265589+0000","last_undegraded":"2023-01-15T14:33:11.265589+0000","last_fullsized":"2023-01-15T14:33:11.265589+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:16.384","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.206363924,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.d","version":"296'232","reported_seq":1181,"reported_epoch":303,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.979972+0000","last_change":"2023-01-15T14:33:08.119752+0000","last_active":"2023-01-15T14:33:11.979972+0000","last_peered":"2023-01-15T14:33:11.979972+0000","last_clean":"2023-01-15T14:33:11.979972+0000","last_became_active":"2023-01-15T14:31:21.132225+0000","last_became_peered":"2023-01-15T14:31:21.132225+0000","last_unstale":"2023-01-15T14:33:11.979972+0000","last_undegraded":"2023-01-15T14:33:11.979972+0000","last_fullsized":"2023-01-15T14:33:11.979972+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.10298273199999999,"stat_sum":{"num_bytes":9226753,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":16259,"num_write":23,"num_write_kb":6561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.c","version":"260'66","reported_seq":922,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.054981+0000","last_change":"2023-01-15T14:33:09.965123+0000","last_active":"2023-01-15T14:33:11.054981+0000","last_peered":"2023-01-15T14:33:11.054981+0000","last_clean":"2023-01-15T14:33:11.054981+0000","last_became_active":"2023-01-15T14:31:21.227945+0000","last_became_peered":"2023-01-15T14:31:21.227945+0000","last_unstale":"2023-01-15T14:33:11.054981+0000","last_undegraded":"2023-01-15T14:33:11.054981+0000","last_fullsized":"2023-01-15T14:33:11.054981+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:02.585","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":1.0338855709999999,"stat_sum":{"num_bytes":2386660,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":20987,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":890,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.954063+0000","last_change":"2023-01-15T14:33:09.871787+0000","last_active":"2023-01-15T14:33:10.954063+0000","last_peered":"2023-01-15T14:33:10.954063+0000","last_clean":"2023-01-15T14:33:10.954063+0000","last_became_active":"2023-01-15T14:32:22.246088+0000","last_became_peered":"2023-01-15T14:32:22.246088+0000","last_unstale":"2023-01-15T14:33:10.954063+0000","last_undegraded":"2023-01-15T14:33:10.954063+0000","last_fullsized":"2023-01-15T14:33:10.954063+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:06.346","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.83800989299999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1211,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.164910+0000","last_change":"2023-01-15T14:33:06.187659+0000","last_active":"2023-01-15T14:33:11.164910+0000","last_peered":"2023-01-15T14:33:11.164910+0000","last_clean":"2023-01-15T14:33:11.164910+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:33:11.164910+0000","last_undegraded":"2023-01-15T14:33:11.164910+0000","last_fullsized":"2023-01-15T14:33:11.164910+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:25.185","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.205904644,"stat_sum":{"num_bytes":2130792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21399,"num_write":27,"num_write_kb":9854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9447726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1008,"reported_epoch":303,"state":"active+clean","last_fresh":"2023-01-15T14:33:12.335761+0000","last_change":"2023-01-15T14:33:12.335604+0000","last_active":"2023-01-15T14:33:12.335761+0000","last_peered":"2023-01-15T14:33:12.335761+0000","last_clean":"2023-01-15T14:33:12.335761+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:33:12.335761+0000","last_undegraded":"2023-01-15T14:33:12.335761+0000","last_fullsized":"2023-01-15T14:33:12.335761+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.10803512999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1204,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.230845+0000","last_change":"2023-01-15T14:33:09.874590+0000","last_active":"2023-01-15T14:33:11.230845+0000","last_peered":"2023-01-15T14:33:11.230845+0000","last_clean":"2023-01-15T14:33:11.230845+0000","last_became_active":"2023-01-15T14:32:22.142296+0000","last_became_peered":"2023-01-15T14:32:22.142296+0000","last_unstale":"2023-01-15T14:33:11.230845+0000","last_undegraded":"2023-01-15T14:33:11.230845+0000","last_fullsized":"2023-01-15T14:33:11.230845+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.94093742599999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"1.0","version":"19'117","reported_seq":449,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.163335+0000","last_change":"2023-01-15T14:32:54.588161+0000","last_active":"2023-01-15T14:33:11.163335+0000","last_peered":"2023-01-15T14:33:11.163335+0000","last_clean":"2023-01-15T14:33:11.163335+0000","last_became_active":"2023-01-15T14:31:03.091071+0000","last_became_peered":"2023-01-15T14:31:03.091071+0000","last_unstale":"2023-01-15T14:33:11.163335+0000","last_undegraded":"2023-01-15T14:33:11.163335+0000","last_fullsized":"2023-01-15T14:33:11.163335+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:16.622","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":274,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.953921+0000","last_change":"2023-01-15T14:31:38.018850+0000","last_active":"2023-01-15T14:33:10.953921+0000","last_peered":"2023-01-15T14:33:10.953921+0000","last_clean":"2023-01-15T14:33:10.953921+0000","last_became_active":"2023-01-15T14:31:38.018145+0000","last_became_peered":"2023-01-15T14:31:38.018145+0000","last_unstale":"2023-01-15T14:33:10.953921+0000","last_undegraded":"2023-01-15T14:33:10.953921+0000","last_fullsized":"2023-01-15T14:33:10.953921+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":895,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.164736+0000","last_change":"2023-01-15T14:33:06.189216+0000","last_active":"2023-01-15T14:33:11.164736+0000","last_peered":"2023-01-15T14:33:11.164736+0000","last_clean":"2023-01-15T14:33:11.164736+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:33:11.164736+0000","last_undegraded":"2023-01-15T14:33:11.164736+0000","last_fullsized":"2023-01-15T14:33:11.164736+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:35.297","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.10801253700000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":348,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.954670+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:33:10.954670+0000","last_peered":"2023-01-15T14:33:10.954670+0000","last_clean":"2023-01-15T14:33:10.954670+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:33:10.954670+0000","last_undegraded":"2023-01-15T14:33:10.954670+0000","last_fullsized":"2023-01-15T14:33:10.954670+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.964","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":768,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.164072+0000","last_change":"2023-01-15T14:33:06.082964+0000","last_active":"2023-01-15T14:33:11.164072+0000","last_peered":"2023-01-15T14:33:11.164072+0000","last_clean":"2023-01-15T14:33:11.164072+0000","last_became_active":"2023-01-15T14:32:40.967729+0000","last_became_peered":"2023-01-15T14:32:40.967729+0000","last_unstale":"2023-01-15T14:33:11.164072+0000","last_undegraded":"2023-01-15T14:33:11.164072+0000","last_fullsized":"2023-01-15T14:33:11.164072+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:43.783","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.102711276,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":103,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.953552+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:33:10.953552+0000","last_peered":"2023-01-15T14:33:10.953552+0000","last_clean":"2023-01-15T14:33:10.953552+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:33:10.953552+0000","last_undegraded":"2023-01-15T14:33:10.953552+0000","last_fullsized":"2023-01-15T14:33:10.953552+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1218,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.263999+0000","last_change":"2023-01-15T14:33:06.237370+0000","last_active":"2023-01-15T14:33:11.263999+0000","last_peered":"2023-01-15T14:33:11.263999+0000","last_clean":"2023-01-15T14:33:11.263999+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:33:11.263999+0000","last_undegraded":"2023-01-15T14:33:11.263999+0000","last_fullsized":"2023-01-15T14:33:11.263999+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:31.185","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.25332313899999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":171,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.165433+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:33:11.165433+0000","last_peered":"2023-01-15T14:33:11.165433+0000","last_clean":"2023-01-15T14:33:11.165433+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:33:11.165433+0000","last_undegraded":"2023-01-15T14:33:11.165433+0000","last_fullsized":"2023-01-15T14:33:11.165433+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:25.583","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"274'309","reported_seq":1275,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.044492+0000","last_change":"2023-01-15T14:33:10.454229+0000","last_active":"2023-01-15T14:33:11.044492+0000","last_peered":"2023-01-15T14:33:11.044492+0000","last_clean":"2023-01-15T14:33:11.044492+0000","last_became_active":"2023-01-15T14:33:10.346559+0000","last_became_peered":"2023-01-15T14:33:10.346559+0000","last_unstale":"2023-01-15T14:33:11.044492+0000","last_undegraded":"2023-01-15T14:33:11.044492+0000","last_fullsized":"2023-01-15T14:33:11.044492+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'251","last_scrub_stamp":"2023-01-15T14:31:15.363400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:15.363400+0000","objects_scrubbed":3,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.208136242,"objects_trimmed":0,"snaptrim_duration":0.006606893,"stat_sum":{"num_bytes":2840449,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":22494,"num_write":15,"num_write_kb":5392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2840449,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.5","version":"299'138","reported_seq":1046,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.954647+0000","last_change":"2023-01-15T14:33:09.875186+0000","last_active":"2023-01-15T14:33:10.954647+0000","last_peered":"2023-01-15T14:33:10.954647+0000","last_clean":"2023-01-15T14:33:10.954647+0000","last_became_active":"2023-01-15T14:32:21.948278+0000","last_became_peered":"2023-01-15T14:32:21.948278+0000","last_unstale":"2023-01-15T14:33:10.954647+0000","last_undegraded":"2023-01-15T14:33:10.954647+0000","last_fullsized":"2023-01-15T14:33:10.954647+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:38.550","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.843355301,"stat_sum":{"num_bytes":4700816,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":13729,"num_write":5,"num_write_kb":1162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":161,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.164197+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:33:11.164197+0000","last_peered":"2023-01-15T14:33:11.164197+0000","last_clean":"2023-01-15T14:33:11.164197+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:33:11.164197+0000","last_undegraded":"2023-01-15T14:33:11.164197+0000","last_fullsized":"2023-01-15T14:33:11.164197+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:37.670","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"294'305","reported_seq":1307,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.164120+0000","last_change":"2023-01-15T14:33:06.231312+0000","last_active":"2023-01-15T14:33:11.164120+0000","last_peered":"2023-01-15T14:33:11.164120+0000","last_clean":"2023-01-15T14:33:11.164120+0000","last_became_active":"2023-01-15T14:32:22.247573+0000","last_became_peered":"2023-01-15T14:32:22.247573+0000","last_unstale":"2023-01-15T14:33:11.164120+0000","last_undegraded":"2023-01-15T14:33:11.164120+0000","last_fullsized":"2023-01-15T14:33:11.164120+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.247327187,"stat_sum":{"num_bytes":904090,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":30683,"num_write":27,"num_write_kb":9142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7931053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":340,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.955442+0000","last_change":"2023-01-15T14:32:34.578760+0000","last_active":"2023-01-15T14:33:10.955442+0000","last_peered":"2023-01-15T14:33:10.955442+0000","last_clean":"2023-01-15T14:33:10.955442+0000","last_became_active":"2023-01-15T14:31:03.088392+0000","last_became_peered":"2023-01-15T14:31:03.088392+0000","last_unstale":"2023-01-15T14:33:10.955442+0000","last_undegraded":"2023-01-15T14:33:10.955442+0000","last_fullsized":"2023-01-15T14:33:10.955442+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:38.541","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"297'114","reported_seq":1011,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.265926+0000","last_change":"2023-01-15T14:33:06.432090+0000","last_active":"2023-01-15T14:33:11.265926+0000","last_peered":"2023-01-15T14:33:11.265926+0000","last_clean":"2023-01-15T14:33:11.265926+0000","last_became_active":"2023-01-15T14:32:22.146367+0000","last_became_peered":"2023-01-15T14:32:22.146367+0000","last_unstale":"2023-01-15T14:33:11.265926+0000","last_undegraded":"2023-01-15T14:33:11.265926+0000","last_fullsized":"2023-01-15T14:33:11.265926+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:03.357","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.20173089699999999,"stat_sum":{"num_bytes":6302724,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":13596,"num_write":1,"num_write_kb":2758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":161,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.266425+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:33:11.266425+0000","last_peered":"2023-01-15T14:33:11.266425+0000","last_clean":"2023-01-15T14:33:11.266425+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:33:11.266425+0000","last_undegraded":"2023-01-15T14:33:11.266425+0000","last_fullsized":"2023-01-15T14:33:11.266425+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:17.360","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":914,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.166025+0000","last_change":"2023-01-15T14:33:06.333184+0000","last_active":"2023-01-15T14:33:11.166025+0000","last_peered":"2023-01-15T14:33:11.166025+0000","last_clean":"2023-01-15T14:33:11.166025+0000","last_became_active":"2023-01-15T14:32:22.247696+0000","last_became_peered":"2023-01-15T14:32:22.247696+0000","last_unstale":"2023-01-15T14:33:11.166025+0000","last_undegraded":"2023-01-15T14:33:11.166025+0000","last_fullsized":"2023-01-15T14:33:11.166025+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:09.533","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.34909034300000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":283,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.054546+0000","last_change":"2023-01-15T14:31:38.112378+0000","last_active":"2023-01-15T14:33:11.054546+0000","last_peered":"2023-01-15T14:33:11.054546+0000","last_clean":"2023-01-15T14:33:11.054546+0000","last_became_active":"2023-01-15T14:31:38.111590+0000","last_became_peered":"2023-01-15T14:31:38.111590+0000","last_unstale":"2023-01-15T14:33:11.054546+0000","last_undegraded":"2023-01-15T14:33:11.054546+0000","last_fullsized":"2023-01-15T14:33:11.054546+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"245'148","reported_seq":1050,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.165325+0000","last_change":"2023-01-15T14:33:06.192324+0000","last_active":"2023-01-15T14:33:11.165325+0000","last_peered":"2023-01-15T14:33:11.165325+0000","last_clean":"2023-01-15T14:33:11.165325+0000","last_became_active":"2023-01-15T14:31:35.134196+0000","last_became_peered":"2023-01-15T14:31:35.134196+0000","last_unstale":"2023-01-15T14:33:11.165325+0000","last_undegraded":"2023-01-15T14:33:11.165325+0000","last_fullsized":"2023-01-15T14:33:11.165325+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:32:19.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:19.303243+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:46.971","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.108999233,"stat_sum":{"num_bytes":2453855,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7195,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":876,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.954433+0000","last_change":"2023-01-15T14:33:09.752557+0000","last_active":"2023-01-15T14:33:10.954433+0000","last_peered":"2023-01-15T14:33:10.954433+0000","last_clean":"2023-01-15T14:33:10.954433+0000","last_became_active":"2023-01-15T14:31:34.951721+0000","last_became_peered":"2023-01-15T14:31:34.951721+0000","last_unstale":"2023-01-15T14:33:10.954433+0000","last_undegraded":"2023-01-15T14:33:10.954433+0000","last_fullsized":"2023-01-15T14:33:10.954433+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:41.128","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.72126160800000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.16","version":"259'113","reported_seq":498,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.162611+0000","last_change":"2023-01-15T14:33:06.335180+0000","last_active":"2023-01-15T14:33:11.162611+0000","last_peered":"2023-01-15T14:33:11.162611+0000","last_clean":"2023-01-15T14:33:11.162611+0000","last_became_active":"2023-01-15T14:31:35.032807+0000","last_became_peered":"2023-01-15T14:31:35.032807+0000","last_unstale":"2023-01-15T14:33:11.162611+0000","last_undegraded":"2023-01-15T14:33:11.162611+0000","last_fullsized":"2023-01-15T14:33:11.162611+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:34:03.444","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.32647762000000002,"stat_sum":{"num_bytes":2353046,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20560,"num_write":10,"num_write_kb":2541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.17","version":"278'93","reported_seq":926,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.054595+0000","last_change":"2023-01-15T14:33:09.869953+0000","last_active":"2023-01-15T14:33:11.054595+0000","last_peered":"2023-01-15T14:33:11.054595+0000","last_clean":"2023-01-15T14:33:11.054595+0000","last_became_active":"2023-01-15T14:31:34.932309+0000","last_became_peered":"2023-01-15T14:31:34.932309+0000","last_unstale":"2023-01-15T14:33:11.054595+0000","last_undegraded":"2023-01-15T14:33:11.054595+0000","last_fullsized":"2023-01-15T14:33:11.054595+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:43.141","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.83860522500000001,"stat_sum":{"num_bytes":3025326,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.18","version":"299'315","reported_seq":1366,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.153383+0000","last_change":"2023-01-15T14:33:09.877964+0000","last_active":"2023-01-15T14:33:11.153383+0000","last_peered":"2023-01-15T14:33:11.153383+0000","last_clean":"2023-01-15T14:33:11.153383+0000","last_became_active":"2023-01-15T14:31:52.330386+0000","last_became_peered":"2023-01-15T14:31:52.330386+0000","last_unstale":"2023-01-15T14:33:11.153383+0000","last_undegraded":"2023-01-15T14:33:11.153383+0000","last_fullsized":"2023-01-15T14:33:11.153383+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:06.616","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.94473271400000003,"stat_sum":{"num_bytes":6551330,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":55265,"num_write":44,"num_write_kb":14485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":11033036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.19","version":"294'256","reported_seq":1192,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.164078+0000","last_change":"2023-01-15T14:33:06.221405+0000","last_active":"2023-01-15T14:33:11.164078+0000","last_peered":"2023-01-15T14:33:11.164078+0000","last_clean":"2023-01-15T14:33:11.164078+0000","last_became_active":"2023-01-15T14:31:51.927440+0000","last_became_peered":"2023-01-15T14:31:51.927440+0000","last_unstale":"2023-01-15T14:33:11.164078+0000","last_undegraded":"2023-01-15T14:33:11.164078+0000","last_fullsized":"2023-01-15T14:33:11.164078+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.23787201899999999,"stat_sum":{"num_bytes":2420274,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":10,"num_read_kb":7098,"num_write":10,"num_write_kb":2362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.1a","version":"303'331","reported_seq":1317,"reported_epoch":303,"state":"active+clean","last_fresh":"2023-01-15T14:33:12.468693+0000","last_change":"2023-01-15T14:33:06.186887+0000","last_active":"2023-01-15T14:33:12.468693+0000","last_peered":"2023-01-15T14:33:12.468693+0000","last_clean":"2023-01-15T14:33:12.468693+0000","last_became_active":"2023-01-15T14:31:52.124026+0000","last_became_peered":"2023-01-15T14:31:52.124026+0000","last_unstale":"2023-01-15T14:33:12.468693+0000","last_undegraded":"2023-01-15T14:33:12.468693+0000","last_fullsized":"2023-01-15T14:33:12.468693+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:49.729","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.20563657699999999,"stat_sum":{"num_bytes":6105307,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":9,"num_read_kb":8573,"num_write":10,"num_write_kb":1196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.1b","version":"283'80","reported_seq":891,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.265047+0000","last_change":"2023-01-15T14:33:06.183734+0000","last_active":"2023-01-15T14:33:11.265047+0000","last_peered":"2023-01-15T14:33:11.265047+0000","last_clean":"2023-01-15T14:33:11.265047+0000","last_became_active":"2023-01-15T14:31:52.321399+0000","last_became_peered":"2023-01-15T14:31:52.321399+0000","last_unstale":"2023-01-15T14:33:11.265047+0000","last_undegraded":"2023-01-15T14:33:11.265047+0000","last_fullsized":"2023-01-15T14:33:11.265047+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:00.204","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.20247839300000001,"stat_sum":{"num_bytes":63320,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":21019,"num_write":6,"num_write_kb":2022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":415,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.165153+0000","last_change":"2023-01-15T14:33:06.190926+0000","last_active":"2023-01-15T14:33:11.165153+0000","last_peered":"2023-01-15T14:33:11.165153+0000","last_clean":"2023-01-15T14:33:11.165153+0000","last_became_active":"2023-01-15T14:31:51.926652+0000","last_became_peered":"2023-01-15T14:31:51.926652+0000","last_unstale":"2023-01-15T14:33:11.165153+0000","last_undegraded":"2023-01-15T14:33:11.165153+0000","last_fullsized":"2023-01-15T14:33:11.165153+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:35.143","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.10811783899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1152,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.265141+0000","last_change":"2023-01-15T14:33:06.186343+0000","last_active":"2023-01-15T14:33:11.265141+0000","last_peered":"2023-01-15T14:33:11.265141+0000","last_clean":"2023-01-15T14:33:11.265141+0000","last_became_active":"2023-01-15T14:31:52.025227+0000","last_became_peered":"2023-01-15T14:31:52.025227+0000","last_unstale":"2023-01-15T14:33:11.265141+0000","last_undegraded":"2023-01-15T14:33:11.265141+0000","last_fullsized":"2023-01-15T14:33:11.265141+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:32.121","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.204758048,"stat_sum":{"num_bytes":5739855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":47622,"num_write":25,"num_write_kb":10630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6928644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":854,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.961854+0000","last_change":"2023-01-15T14:33:09.764888+0000","last_active":"2023-01-15T14:33:10.961854+0000","last_peered":"2023-01-15T14:33:10.961854+0000","last_clean":"2023-01-15T14:33:10.961854+0000","last_became_active":"2023-01-15T14:31:52.329815+0000","last_became_peered":"2023-01-15T14:31:52.329815+0000","last_unstale":"2023-01-15T14:33:10.961854+0000","last_undegraded":"2023-01-15T14:33:10.961854+0000","last_fullsized":"2023-01-15T14:33:10.961854+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:31:44.906253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.906253+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55943661099999997,"objects_trimmed":0,"snaptrim_duration":0.83430569300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.1f","version":"297'254","reported_seq":1265,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.265871+0000","last_change":"2023-01-15T14:33:07.531464+0000","last_active":"2023-01-15T14:33:11.265871+0000","last_peered":"2023-01-15T14:33:11.265871+0000","last_clean":"2023-01-15T14:33:11.265871+0000","last_became_active":"2023-01-15T14:31:52.124821+0000","last_became_peered":"2023-01-15T14:31:52.124821+0000","last_unstale":"2023-01-15T14:33:11.265871+0000","last_undegraded":"2023-01-15T14:33:11.265871+0000","last_fullsized":"2023-01-15T14:33:11.265871+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:55.650","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.59741790800000005,"stat_sum":{"num_bytes":9428859,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":20,"num_read_kb":26683,"num_write":2,"num_write_kb":3398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.20","version":"288'262","reported_seq":1263,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.954448+0000","last_change":"2023-01-15T14:33:06.193356+0000","last_active":"2023-01-15T14:33:10.954448+0000","last_peered":"2023-01-15T14:33:10.954448+0000","last_clean":"2023-01-15T14:33:10.954448+0000","last_became_active":"2023-01-15T14:31:07.906191+0000","last_became_peered":"2023-01-15T14:31:07.906191+0000","last_unstale":"2023-01-15T14:33:10.954448+0000","last_undegraded":"2023-01-15T14:33:10.954448+0000","last_fullsized":"2023-01-15T14:33:10.954448+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.20771506000000001,"stat_sum":{"num_bytes":3294238,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":21727,"num_write":13,"num_write_kb":3695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8877936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.21","version":"244'130","reported_seq":918,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.054340+0000","last_change":"2023-01-15T14:33:06.192460+0000","last_active":"2023-01-15T14:33:11.054340+0000","last_peered":"2023-01-15T14:33:11.054340+0000","last_clean":"2023-01-15T14:33:11.054340+0000","last_became_active":"2023-01-15T14:31:38.233481+0000","last_became_peered":"2023-01-15T14:31:38.233481+0000","last_unstale":"2023-01-15T14:33:11.054340+0000","last_undegraded":"2023-01-15T14:33:11.054340+0000","last_fullsized":"2023-01-15T14:33:11.054340+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:05.612","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.206847896,"stat_sum":{"num_bytes":2067327,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.22","version":"284'157","reported_seq":1190,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.953723+0000","last_change":"2023-01-15T14:33:09.748793+0000","last_active":"2023-01-15T14:33:10.953723+0000","last_peered":"2023-01-15T14:33:10.953723+0000","last_clean":"2023-01-15T14:33:10.953723+0000","last_became_active":"2023-01-15T14:31:08.084483+0000","last_became_peered":"2023-01-15T14:31:08.084483+0000","last_unstale":"2023-01-15T14:33:10.953723+0000","last_undegraded":"2023-01-15T14:33:10.953723+0000","last_fullsized":"2023-01-15T14:33:10.953723+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.81750481799999997,"stat_sum":{"num_bytes":2756414,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":16,"num_write_kb":7631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":1078,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.953503+0000","last_change":"2023-01-15T14:33:09.868676+0000","last_active":"2023-01-15T14:33:10.953503+0000","last_peered":"2023-01-15T14:33:10.953503+0000","last_clean":"2023-01-15T14:33:10.953503+0000","last_became_active":"2023-01-15T14:31:06.029716+0000","last_became_peered":"2023-01-15T14:31:06.029716+0000","last_unstale":"2023-01-15T14:33:10.953503+0000","last_undegraded":"2023-01-15T14:33:10.953503+0000","last_fullsized":"2023-01-15T14:33:10.953503+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:08.017","scrub_duration":0.75838583400000004,"objects_trimmed":0,"snaptrim_duration":0.83566106900000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.24","version":"298'241","reported_seq":1263,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.954630+0000","last_change":"2023-01-15T14:33:09.763836+0000","last_active":"2023-01-15T14:33:10.954630+0000","last_peered":"2023-01-15T14:33:10.954630+0000","last_clean":"2023-01-15T14:33:10.954630+0000","last_became_active":"2023-01-15T14:31:26.168777+0000","last_became_peered":"2023-01-15T14:31:26.168777+0000","last_unstale":"2023-01-15T14:33:10.954630+0000","last_undegraded":"2023-01-15T14:33:10.954630+0000","last_fullsized":"2023-01-15T14:33:10.954630+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:35.100","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.83287826499999995,"stat_sum":{"num_bytes":4122560,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":21546,"num_write":8,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.25","version":"289'94","reported_seq":986,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.164341+0000","last_change":"2023-01-15T14:33:06.291254+0000","last_active":"2023-01-15T14:33:11.164341+0000","last_peered":"2023-01-15T14:33:11.164341+0000","last_clean":"2023-01-15T14:33:11.164341+0000","last_became_active":"2023-01-15T14:31:22.000386+0000","last_became_peered":"2023-01-15T14:31:22.000386+0000","last_unstale":"2023-01-15T14:33:11.164341+0000","last_undegraded":"2023-01-15T14:33:11.164341+0000","last_fullsized":"2023-01-15T14:33:11.164341+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.30710631900000002,"stat_sum":{"num_bytes":2353046,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":6061,"num_write":1,"num_write_kb":2298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":343,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.955549+0000","last_change":"2023-01-15T14:33:06.297145+0000","last_active":"2023-01-15T14:33:10.955549+0000","last_peered":"2023-01-15T14:33:10.955549+0000","last_clean":"2023-01-15T14:33:10.955549+0000","last_became_active":"2023-01-15T14:31:21.901622+0000","last_became_peered":"2023-01-15T14:31:21.901622+0000","last_unstale":"2023-01-15T14:33:10.955549+0000","last_undegraded":"2023-01-15T14:33:10.955549+0000","last_fullsized":"2023-01-15T14:33:10.955549+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:08.119","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.31096602499999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":951,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.954117+0000","last_change":"2023-01-15T14:33:09.766252+0000","last_active":"2023-01-15T14:33:10.954117+0000","last_peered":"2023-01-15T14:33:10.954117+0000","last_clean":"2023-01-15T14:33:10.954117+0000","last_became_active":"2023-01-15T14:31:22.091646+0000","last_became_peered":"2023-01-15T14:31:22.091646+0000","last_unstale":"2023-01-15T14:33:10.954117+0000","last_undegraded":"2023-01-15T14:33:10.954117+0000","last_fullsized":"2023-01-15T14:33:10.954117+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:41.261","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.83565958799999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.28","version":"283'231","reported_seq":1252,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.030104+0000","last_change":"2023-01-15T14:33:09.873825+0000","last_active":"2023-01-15T14:33:11.030104+0000","last_peered":"2023-01-15T14:33:11.030104+0000","last_clean":"2023-01-15T14:33:11.030104+0000","last_became_active":"2023-01-15T14:31:26.054878+0000","last_became_peered":"2023-01-15T14:31:26.054878+0000","last_unstale":"2023-01-15T14:33:11.030104+0000","last_undegraded":"2023-01-15T14:33:11.030104+0000","last_fullsized":"2023-01-15T14:33:11.030104+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":231,"log_dups_size":0,"ondisk_log_size":231,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:01.610","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.94093687800000003,"stat_sum":{"num_bytes":2504309,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":17937,"num_write":9,"num_write_kb":5085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":12193528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":762,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.953419+0000","last_change":"2023-01-15T14:33:09.747566+0000","last_active":"2023-01-15T14:33:10.953419+0000","last_peered":"2023-01-15T14:33:10.953419+0000","last_clean":"2023-01-15T14:33:10.953419+0000","last_became_active":"2023-01-15T14:31:25.136376+0000","last_became_peered":"2023-01-15T14:31:25.136376+0000","last_unstale":"2023-01-15T14:33:10.953419+0000","last_undegraded":"2023-01-15T14:33:10.953419+0000","last_fullsized":"2023-01-15T14:33:10.953419+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:30:08.681869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:08.681869+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:14.346","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81793978499999997,"stat_sum":{"num_bytes":2857174,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":24874,"num_write":16,"num_write_kb":6171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1128,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.954335+0000","last_change":"2023-01-15T14:33:06.191478+0000","last_active":"2023-01-15T14:33:10.954335+0000","last_peered":"2023-01-15T14:33:10.954335+0000","last_clean":"2023-01-15T14:33:10.954335+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:33:10.954335+0000","last_undegraded":"2023-01-15T14:33:10.954335+0000","last_fullsized":"2023-01-15T14:33:10.954335+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.205952154,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":845,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.164666+0000","last_change":"2023-01-15T14:33:06.339498+0000","last_active":"2023-01-15T14:33:11.164666+0000","last_peered":"2023-01-15T14:33:11.164666+0000","last_clean":"2023-01-15T14:33:11.164666+0000","last_became_active":"2023-01-15T14:31:21.896494+0000","last_became_peered":"2023-01-15T14:31:21.896494+0000","last_unstale":"2023-01-15T14:33:11.164666+0000","last_undegraded":"2023-01-15T14:33:11.164666+0000","last_fullsized":"2023-01-15T14:33:11.164666+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:15.975","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.31523454299999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.2c","version":"141'46","reported_seq":332,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.164575+0000","last_change":"2023-01-15T14:33:06.249819+0000","last_active":"2023-01-15T14:33:11.164575+0000","last_peered":"2023-01-15T14:33:11.164575+0000","last_clean":"2023-01-15T14:33:11.164575+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:33:11.164575+0000","last_undegraded":"2023-01-15T14:33:11.164575+0000","last_fullsized":"2023-01-15T14:33:11.164575+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:10.970","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.26806039399999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1101,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.055564+0000","last_change":"2023-01-15T14:33:06.300849+0000","last_active":"2023-01-15T14:33:11.055564+0000","last_peered":"2023-01-15T14:33:11.055564+0000","last_clean":"2023-01-15T14:33:11.055564+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:33:11.055564+0000","last_undegraded":"2023-01-15T14:33:11.055564+0000","last_fullsized":"2023-01-15T14:33:11.055564+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.31430724399999999,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":780,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.165909+0000","last_change":"2023-01-15T14:33:06.326332+0000","last_active":"2023-01-15T14:33:11.165909+0000","last_peered":"2023-01-15T14:33:11.165909+0000","last_clean":"2023-01-15T14:33:11.165909+0000","last_became_active":"2023-01-15T14:31:38.116756+0000","last_became_peered":"2023-01-15T14:31:38.116756+0000","last_unstale":"2023-01-15T14:33:11.165909+0000","last_undegraded":"2023-01-15T14:33:11.165909+0000","last_fullsized":"2023-01-15T14:33:11.165909+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:33:14.595","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.34255647900000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.2f","version":"155'145","reported_seq":976,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.958325+0000","last_change":"2023-01-15T14:33:06.301629+0000","last_active":"2023-01-15T14:33:10.958325+0000","last_peered":"2023-01-15T14:33:10.958325+0000","last_clean":"2023-01-15T14:33:10.958325+0000","last_became_active":"2023-01-15T14:31:40.024531+0000","last_became_peered":"2023-01-15T14:31:40.024531+0000","last_unstale":"2023-01-15T14:33:10.958325+0000","last_undegraded":"2023-01-15T14:33:10.958325+0000","last_fullsized":"2023-01-15T14:33:10.958325+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:34.781","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.215359471,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":1229,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:10.953975+0000","last_change":"2023-01-15T14:33:06.086733+0000","last_active":"2023-01-15T14:33:10.953975+0000","last_peered":"2023-01-15T14:33:10.953975+0000","last_clean":"2023-01-15T14:33:10.953975+0000","last_became_active":"2023-01-15T14:31:39.915702+0000","last_became_peered":"2023-01-15T14:31:39.915702+0000","last_unstale":"2023-01-15T14:33:10.953975+0000","last_undegraded":"2023-01-15T14:33:10.953975+0000","last_fullsized":"2023-01-15T14:33:10.953975+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.1539330249999999,"objects_trimmed":0,"snaptrim_duration":0.101613335,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]},{"pgid":"3.31","version":"204'150","reported_seq":1114,"reported_epoch":302,"state":"active+clean","last_fresh":"2023-01-15T14:33:11.263437+0000","last_change":"2023-01-15T14:33:05.990687+0000","last_active":"2023-01-15T14:33:11.263437+0000","last_peered":"2023-01-15T14:33:11.263437+0000","last_clean":"2023-01-15T14:33:11.263437+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:33:11.263437+0000","last_undegraded":"2023-01-15T14:33:11.263437+0000","last_fullsized":"2023-01-15T14:33:11.263437+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:16.599","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.0076328409999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":76,"stat_sum":{"num_bytes":179243120,"num_objects":83,"num_object_clones":40,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":8,"num_read":829,"num_read_kb":889363,"num_write":665,"num_write_kb":252431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":213036487,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12714,"ondisk_log_size":12714,"up":228,"acting":227,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":29,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":892364,"kb_used_data":92807016,"kb_used_omap":5231,"kb_used_meta":0,"kb_avail":92807016,"statfs":{"total":95948165120,"available":95034384384,"internally_reserved":0,"allocated":95034384384,"data_stored":95034384384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5356754,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":4.774,"15min":4.3040000000000003},"min":{"1min":0.55500000000000005,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.98199999999999998,"5min":148.10900000000001,"15min":148.10900000000001},"last":1.105},{"interface":"front","average":{"1min":2.9169999999999998,"5min":6.2539999999999996,"15min":8.2240000000000002},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":43.125,"5min":73.769000000000005,"15min":73.769000000000005},"last":1.0620000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":2.2909999999999999,"5min":4.4909999999999997,"15min":5.8600000000000003},"min":{"1min":0.56899999999999995,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":31.518999999999998,"5min":88.003,"15min":88.003},"last":0.84699999999999998},{"interface":"front","average":{"1min":9.8260000000000005,"5min":6.79,"15min":6.4500000000000002},"min":{"1min":0.59399999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.581999999999994,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.91200000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":14.355,"5min":7.7380000000000004,"15min":5.7729999999999997},"min":{"1min":0.51400000000000001,"5min":0.436,"15min":0.436},"max":{"1min":151.31800000000001,"5min":151.31800000000001,"15min":151.31800000000001},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":3.2309999999999999,"15min":3.8620000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.633,"5min":91.837999999999994,"15min":91.837999999999994},"last":0.95899999999999996}]},{"osd":3,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":18.995000000000001,"5min":7.3920000000000003,"15min":3.0960000000000001},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":209.05500000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":5.7030000000000003,"5min":4.383,"15min":4.5620000000000003},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":98.825000000000003,"5min":98.825000000000003,"15min":98.825000000000003},"last":1.044}]},{"osd":4,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":2.9430000000000001,"5min":5.5949999999999998,"15min":8.1419999999999995},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":42.893999999999998,"5min":76.658000000000001,"15min":76.658000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":8.5739999999999998,"5min":5.5899999999999999,"15min":2.5059999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":154.999,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.88}]},{"osd":5,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":3.0379999999999998,"5min":4.4029999999999996,"15min":5.6849999999999996},"min":{"1min":0.52100000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":43.029000000000003,"5min":73.561000000000007,"15min":73.561000000000007},"last":0.98599999999999999},{"interface":"front","average":{"1min":3.7930000000000001,"5min":3.9950000000000001,"15min":4.6280000000000001},"min":{"1min":0.48399999999999999,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":58.134999999999998,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.93899999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:32:16 2023","interfaces":[{"interface":"back","average":{"1min":3.109,"5min":5.351,"15min":7.4660000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":42.978999999999999,"5min":73.638999999999996,"15min":73.638999999999996},"last":1.129},{"interface":"front","average":{"1min":3.8410000000000002,"5min":2.5169999999999999,"15min":1.5720000000000001},"min":{"1min":0.60299999999999998,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":58.204999999999998,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.016}]}]},{"osd":6,"up_from":0,"seq":55834574912,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574912,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574911,"num_pgs":58,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":916020,"kb_used_data":92783712,"kb_used_omap":10635,"kb_used_meta":0,"kb_avail":92783360,"statfs":{"total":95948165120,"available":95010160640,"internally_reserved":360448,"allocated":95010521088,"data_stored":95010521088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10890944,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":103.827,"5min":24.414999999999999,"15min":11.414999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":2165.8420000000001,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":5.7409999999999997,"5min":3,"15min":1.52},"min":{"1min":0.56999999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":88.600999999999999,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.71699999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":3.9769999999999999,"5min":6.6109999999999998,"15min":7.9329999999999998},"min":{"1min":0.50700000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":69.415000000000006,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.863,"5min":4.3360000000000003,"15min":1.974},"min":{"1min":0.58999999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":22.411999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.84499999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":103.90300000000001,"5min":25.164999999999999,"15min":12.061999999999999},"min":{"1min":0.59899999999999998,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":2165.3220000000001,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.871,"15min":0.89700000000000002},"min":{"1min":0.623,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0349999999999999,"5min":1.6080000000000001,"15min":1.6080000000000001},"last":0.92400000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":10.015000000000001,"5min":3,"15min":1.8919999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":88.823999999999998,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.78900000000000003},{"interface":"front","average":{"1min":105.232,"5min":25.382000000000001,"15min":12.561},"min":{"1min":0.628,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2165.4690000000001,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":1.117}]},{"osd":5,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.88100000000000001,"15min":0.90700000000000003},"min":{"1min":0.58299999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.302,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.746},{"interface":"front","average":{"1min":5.867,"5min":2.4590000000000001,"15min":2.3519999999999999},"min":{"1min":0.68200000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":88.911000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.871}]},{"osd":6,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":4.8620000000000001,"15min":5.4020000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.341,"5min":99.780000000000001,"15min":99.780000000000001},"last":1.038},{"interface":"front","average":{"1min":104.14,"5min":23.286000000000001,"15min":11.204000000000001},"min":{"1min":0.84599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2165.7330000000002,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":0.99299999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:32:13 2023","interfaces":[{"interface":"back","average":{"1min":9.5570000000000004,"5min":7.5519999999999996,"15min":11.269},"min":{"1min":0.76100000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":89.061000000000007,"5min":179.73500000000001,"15min":179.73500000000001},"last":1.014},{"interface":"front","average":{"1min":104.11,"5min":26.864999999999998,"15min":16.446999999999999},"min":{"1min":0.78600000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2165.5309999999999,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.1699999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":57,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":931528,"kb_used_data":92774320,"kb_used_omap":10813,"kb_used_meta":0,"kb_avail":92767852,"statfs":{"total":95948165120,"available":94994280448,"internally_reserved":6623232,"allocated":95000903680,"data_stored":95000903680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11073073,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":12.667,"5min":8.0429999999999993,"15min":5.9130000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":122.065,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":4.8109999999999999,"15min":6.6980000000000004},"min":{"1min":0.41299999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.0469999999999999,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.71499999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":4.1790000000000003,"5min":9.7569999999999997,"15min":12.273},"min":{"1min":0.45000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":51.162999999999997,"5min":159.256,"15min":159.256},"last":0.88800000000000001},{"interface":"front","average":{"1min":20.414000000000001,"5min":10.041,"15min":3.8410000000000002},"min":{"1min":0.42799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":121.203,"5min":174.578,"15min":174.578},"last":0.78300000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":2.6139999999999999,"15min":3.1040000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3700000000000001,"5min":59.896999999999998,"15min":59.896999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":13.534000000000001,"5min":6.8380000000000001,"15min":3.7410000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":122.13,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.91500000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":122.569,"5min":29.460999999999999,"15min":14.195},"min":{"1min":0.63900000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":2059.6680000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.85099999999999998},{"interface":"front","average":{"1min":122.634,"5min":34.735999999999997,"15min":20.827999999999999},"min":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":2058.4229999999998,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":0.95999999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":2.0019999999999998,"5min":9.0939999999999994,"15min":12.300000000000001},"min":{"1min":0.625,"5min":0.622,"15min":0.622},"max":{"1min":21.789999999999999,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.749},{"interface":"front","average":{"1min":13.038,"5min":5.7619999999999996,"15min":2.6019999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":122.208,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.98599999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":13.013,"5min":5.766,"15min":2.6099999999999999},"min":{"1min":0.65900000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":122.26600000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":1.081},{"interface":"front","average":{"1min":0.98399999999999999,"5min":2.4969999999999999,"15min":3.3399999999999999},"min":{"1min":0.70999999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.49,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.93799999999999994}]},{"osd":7,"last update":"Sun Jan 15 14:32:11 2023","interfaces":[{"interface":"back","average":{"1min":1,"5min":2.5990000000000002,"15min":3.5569999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.522,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.0429999999999999},{"interface":"front","average":{"1min":5.444,"5min":11.496,"15min":15.891999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":55.337000000000003,"5min":159.512,"15min":159.512},"last":1.02}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":46,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":939036,"kb_used_data":92765036,"kb_used_omap":8776,"kb_used_meta":0,"kb_avail":92760344,"statfs":{"total":95948165120,"available":94986592256,"internally_reserved":4804608,"allocated":94991396864,"data_stored":94991396864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8986692,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":3.109,"5min":3.5230000000000001,"15min":1.601},"min":{"1min":0.437,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":51.009999999999998,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.4299999999999999,"15min":0.95299999999999996},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0089999999999999,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.59499999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.1529999999999996,"5min":5.9059999999999997,"15min":2.5009999999999999},"min":{"1min":0.378,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":91.525999999999996,"5min":143.071,"15min":143.071},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.69399999999999995,"5min":2.016,"15min":2.9089999999999998},"min":{"1min":0.42299999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.131,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.79600000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":8.2210000000000001,"15min":9.6750000000000007},"min":{"1min":0.5,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.3609999999999998,"5min":183.53,"15min":183.53},"last":0.70799999999999996},{"interface":"front","average":{"1min":16.692,"5min":7.5170000000000003,"15min":3.0169999999999999},"min":{"1min":0.61199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":155.47900000000001,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.64500000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3650000000000002,"5min":6.5060000000000002,"15min":4.6020000000000003},"min":{"1min":0.67300000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":51.259999999999998,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":2.8809999999999998,"5min":5.2939999999999996,"15min":4.3040000000000003},"min":{"1min":0.48699999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":33.219999999999999,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.84599999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":2.4729999999999999,"15min":1.6080000000000001},"min":{"1min":0.57199999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2130000000000001,"5min":143.262,"15min":143.262},"last":0.92300000000000004},{"interface":"front","average":{"1min":3.4780000000000002,"5min":3.0539999999999998,"15min":1.792},"min":{"1min":0.70999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":51.326000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.82799999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.857,"15min":1.3069999999999999},"min":{"1min":0.60399999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.365,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.995},{"interface":"front","average":{"1min":1.018,"5min":2.6160000000000001,"15min":1.6839999999999999},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.5,"5min":143.215,"15min":143.215},"last":0.87}]},{"osd":7,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":13.092000000000001,"5min":6.3120000000000003,"15min":4.1429999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":120.905,"5min":120.905,"15min":120.905},"last":59.093000000000004},{"interface":"front","average":{"1min":1.0129999999999999,"5min":2.8450000000000002,"15min":2.7989999999999999},"min":{"1min":0.66600000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.552,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.81299999999999994}]}]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":54,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":893776,"kb_used_data":92808740,"kb_used_omap":10326,"kb_used_meta":0,"kb_avail":92805604,"statfs":{"total":95948165120,"available":95032938496,"internally_reserved":3211264,"allocated":95036149760,"data_stored":95036149760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10574480,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":3.633,"15min":3.806},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":3.9140000000000001,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":7.742,"5min":4.6130000000000004,"15min":4.4189999999999996},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":119.39100000000001,"5min":119.39100000000001,"15min":119.39100000000001},"last":0.91200000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":2.6469999999999998,"15min":1.5269999999999999},"min":{"1min":0.29799999999999999,"5min":0.245,"15min":0.245},"max":{"1min":95.879999999999995,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.8620000000000001,"15min":3.3359999999999999},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.9780000000000002,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.497}]},{"osd":3,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.8920000000000003,"5min":9.6750000000000007,"15min":16.449000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":100.137,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":7.782,"5min":6.8899999999999997,"15min":7.1260000000000003},"min":{"1min":0.45600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":119.804,"5min":119.804,"15min":119.804},"last":0.65900000000000003}]},{"osd":4,"last update":"Sun Jan 15 14:32:12 2023","interfaces":[{"interface":"back","average":{"1min":5.6849999999999996,"5min":16.501000000000001,"15min":8.6519999999999992},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":87.128,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":9.8059999999999992,"5min":15.890000000000001,"15min":7.7000000000000002},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":84.629999999999995,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.747}]},{"osd":5,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.5449999999999999,"15min":1.2030000000000001},"min":{"1min":0.59899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.278,"5min":65.120999999999995,"15min":65.120999999999995},"last":1.165},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.6579999999999999,"15min":3.5110000000000001},"min":{"1min":0.60599999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":4.0780000000000003,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.77200000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":8.0069999999999997,"5min":5.0129999999999999,"15min":5.0049999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":119.759,"5min":119.759,"15min":119.759},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.603,"15min":1.2829999999999999},"min":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.349,"5min":65.156000000000006,"15min":65.156000000000006},"last":0.98399999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":3.6030000000000002,"5min":6.1109999999999998,"15min":6.7149999999999999},"min":{"1min":0.56000000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":31.620999999999999,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":8.0779999999999994,"5min":5.0060000000000002,"15min":4.835},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":119.65300000000001,"5min":119.65300000000001,"15min":119.65300000000001},"last":1.0429999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":14,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":836996,"kb_used_data":92875236,"kb_used_omap":5180,"kb_used_meta":0,"kb_avail":92862384,"statfs":{"total":95948165120,"available":95091081216,"internally_reserved":13160448,"allocated":95104241664,"data_stored":95104241664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5304519,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":2.7509999999999999,"5min":3.1230000000000002,"15min":1.456},"min":{"1min":0.29499999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":49.725000000000001,"5min":69.872,"15min":69.872},"last":0.54300000000000004},{"interface":"front","average":{"1min":3.5139999999999998,"5min":3.2349999999999999,"15min":1.5649999999999999},"min":{"1min":0.33700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":61.447000000000003,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.80000000000000004}]},{"osd":2,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":3.4159999999999999,"15min":1.601},"min":{"1min":0.52700000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.2,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.54800000000000004,"5min":2.6499999999999999,"15min":1.3560000000000001},"min":{"1min":0.32400000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.90800000000000003,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.98099999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":2.863,"5min":10.630000000000001,"15min":14.728999999999999},"min":{"1min":0.372,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":49.331000000000003,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":8.0960000000000001,"5min":7.0990000000000002,"15min":2.8999999999999999},"min":{"1min":0.45800000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":162.61600000000001,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.72399999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":68.361999999999995,"5min":16.706,"15min":6.226},"min":{"1min":0.63900000000000001,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1486.309,"5min":1486.309,"15min":1486.309},"last":1.1879999999999999},{"interface":"front","average":{"1min":70.664000000000001,"5min":17.922999999999998,"15min":6.657},"min":{"1min":0.56599999999999995,"5min":0.439,"15min":0.439},"max":{"1min":1486.3789999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":1.002}]},{"osd":5,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":6.4420000000000002,"15min":8.6790000000000003},"min":{"1min":0.64700000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4910000000000001,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.879},{"interface":"front","average":{"1min":0.86499999999999999,"5min":1.7370000000000001,"15min":1.2709999999999999},"min":{"1min":0.60699999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3220000000000001,"5min":71.563999999999993,"15min":71.563999999999993},"last":1.097}]},{"osd":6,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":3.198,"5min":3.609,"15min":1.9830000000000001},"min":{"1min":0.68999999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":49.616,"5min":70.259,"15min":70.259},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.006,"5min":3.0299999999999998,"15min":1.766},"min":{"1min":0.755,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.4419999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":1.1439999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:32:10 2023","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":9.0449999999999999,"15min":13.462},"min":{"1min":0.57199999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.137,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.48},{"interface":"front","average":{"1min":3.21,"5min":3.617,"15min":1.986},"min":{"1min":0.67100000000000004,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":49.670000000000002,"5min":70.322000000000003,"15min":70.322000000000003},"last":1.0649999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:33:14.309 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:33:14.717 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 86 2023-01-15T14:33:14.720 INFO:tasks.rados.rados.0.smithi090.stdout:1208: done (0 left) 2023-01-15T14:33:14.720 INFO:tasks.rados.rados.0.smithi090.stdout:1209: write (excl) oid 21 current snap is 123 2023-01-15T14:33:14.720 INFO:tasks.rados.rados.0.smithi090.stdout:1209: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2023-01-15T14:33:14.720 INFO:tasks.rados.rados.0.smithi090.stdout:1209: writing smithi09025264-21 from 679115 to 1313394 tid 1 2023-01-15T14:33:14.720 INFO:tasks.rados.rados.0.smithi090.stdout:1209: writing smithi09025264-21 from 2090487 to 2554309 tid 2 2023-01-15T14:33:14.721 INFO:tasks.rados.rados.0.smithi090.stdout:1209: writing smithi09025264-21 from 3003820 to 3479115 tid 3 2023-01-15T14:33:14.725 INFO:tasks.rados.rados.0.smithi090.stdout:1210: write (excl) oid 42 current snap is 123 2023-01-15T14:33:14.725 INFO:tasks.rados.rados.0.smithi090.stdout:1210: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2023-01-15T14:33:14.725 INFO:tasks.rados.rados.0.smithi090.stdout:1210: writing smithi09025264-42 from 695922 to 1405450 tid 1 2023-01-15T14:33:14.725 INFO:tasks.rados.rados.0.smithi090.stdout:1210: writing smithi09025264-42 from 2032616 to 2640096 tid 2 2023-01-15T14:33:14.725 INFO:tasks.rados.rados.0.smithi090.stdout:1210: writing smithi09025264-42 from 3198537 to 3495922 tid 3 2023-01-15T14:33:14.725 INFO:tasks.rados.rados.0.smithi090.stdout:1211: write oid 18 current snap is 123 2023-01-15T14:33:14.726 INFO:tasks.rados.rados.0.smithi090.stdout:1211: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2023-01-15T14:33:14.726 INFO:tasks.rados.rados.0.smithi090.stdout:1211: writing smithi09025264-18 from 712729 to 1497506 tid 1 2023-01-15T14:33:14.726 INFO:tasks.rados.rados.0.smithi090.stdout:1211: writing smithi09025264-18 from 2091098 to 2558589 tid 2 2023-01-15T14:33:14.726 INFO:tasks.rados.rados.0.smithi090.stdout:1211: writing smithi09025264-18 from 3342313 to 3512729 tid 3 2023-01-15T14:33:14.726 INFO:tasks.rados.rados.0.smithi090.stdout:1212: read oid 27 snap -1 2023-01-15T14:33:14.726 INFO:tasks.rados.rados.0.smithi090.stdout:1212: expect (ObjNum 421 snap 112 seq_num 421) 2023-01-15T14:33:14.726 INFO:tasks.rados.rados.0.smithi090.stdout:1213: snap_create 2023-01-15T14:33:14.726 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:33:14.727 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:13.965+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:14.729 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:14.716+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:14.729 INFO:tasks.rados.rados.0.smithi090.stdout:1212: expect (ObjNum 421 snap 112 seq_num 421) 2023-01-15T14:33:14.921 INFO:tasks.rados.rados.0.smithi090.stdout:1210: finishing write tid 1 to smithi09025264-42 2023-01-15T14:33:14.921 INFO:tasks.rados.rados.0.smithi090.stdout:1211: finishing write tid 1 to smithi09025264-18 2023-01-15T14:33:14.921 INFO:tasks.rados.rados.0.smithi090.stdout:1212: done (3 left) 2023-01-15T14:33:14.921 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:14.921 INFO:tasks.rados.rados.0.smithi090.stdout:1209: finishing write tid 1 to smithi09025264-21 2023-01-15T14:33:14.921 INFO:tasks.rados.rados.0.smithi090.stdout:1209: finishing write tid 2 to smithi09025264-21 2023-01-15T14:33:14.922 INFO:tasks.rados.rados.0.smithi090.stdout:1210: finishing write tid 2 to smithi09025264-42 2023-01-15T14:33:14.922 INFO:tasks.rados.rados.0.smithi090.stdout:1210: finishing write tid 3 to smithi09025264-42 2023-01-15T14:33:14.922 INFO:tasks.rados.rados.0.smithi090.stdout:1210: finishing write tid 4 to smithi09025264-42 2023-01-15T14:33:14.922 INFO:tasks.rados.rados.0.smithi090.stdout:1210: finishing write tid 5 to smithi09025264-42 2023-01-15T14:33:14.922 INFO:tasks.rados.rados.0.smithi090.stdout:1210: oid 42 updating version 0 to 350 2023-01-15T14:33:14.922 INFO:tasks.rados.rados.0.smithi090.stdout:1210: oid 42 version 350 is already newer than 348 2023-01-15T14:33:14.922 INFO:tasks.rados.rados.0.smithi090.stdout:1210: oid 42 updating version 350 to 351 2023-01-15T14:33:14.922 INFO:tasks.rados.rados.0.smithi090.stdout:1210: oid 42 version 351 is already newer than 349 2023-01-15T14:33:14.922 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 42 v 351 (ObjNum 446 snap 123 seq_num 446) dirty exists 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:1210: left oid 42 (ObjNum 446 snap 123 seq_num 446) 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:1210: done (2 left) 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:1211: finishing write tid 2 to smithi09025264-18 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:1209: finishing write tid 3 to smithi09025264-21 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:1211: finishing write tid 3 to smithi09025264-18 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:1209: finishing write tid 4 to smithi09025264-21 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:1209: finishing write tid 5 to smithi09025264-21 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:1209: oid 21 updating version 0 to 347 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:1209: oid 21 updating version 347 to 350 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:1209: oid 21 version 350 is already newer than 348 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:1209: oid 21 version 350 is already newer than 349 2023-01-15T14:33:14.923 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 350 (ObjNum 445 snap 123 seq_num 445) dirty exists 2023-01-15T14:33:14.924 INFO:tasks.rados.rados.0.smithi090.stdout:1209: left oid 21 (ObjNum 445 snap 123 seq_num 445) 2023-01-15T14:33:14.924 INFO:tasks.rados.rados.0.smithi090.stdout:1209: done (1 left) 2023-01-15T14:33:14.924 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:14.924 INFO:tasks.rados.rados.0.smithi090.stdout:1211: finishing write tid 4 to smithi09025264-18 2023-01-15T14:33:14.924 INFO:tasks.rados.rados.0.smithi090.stdout:1211: finishing write tid 5 to smithi09025264-18 2023-01-15T14:33:14.924 INFO:tasks.rados.rados.0.smithi090.stdout:1211: oid 18 updating version 0 to 225 2023-01-15T14:33:14.924 INFO:tasks.rados.rados.0.smithi090.stdout:1211: oid 18 version 225 is already newer than 224 2023-01-15T14:33:14.924 INFO:tasks.rados.rados.0.smithi090.stdout:1211: oid 18 version 225 is already newer than 223 2023-01-15T14:33:14.924 INFO:tasks.rados.rados.0.smithi090.stdout:1211: oid 18 version 225 is already newer than 222 2023-01-15T14:33:14.924 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 225 (ObjNum 447 snap 123 seq_num 447) dirty exists 2023-01-15T14:33:14.924 INFO:tasks.rados.rados.0.smithi090.stdout:1211: left oid 18 (ObjNum 447 snap 123 seq_num 447) 2023-01-15T14:33:14.925 INFO:tasks.rados.rados.0.smithi090.stdout:1211: done (0 left) 2023-01-15T14:33:14.926 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:14.926 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:14.927 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:33:14.928 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:14.928+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:14.972 INFO:tasks.rados.rados.0.smithi090.stdout:1213: done (0 left) 2023-01-15T14:33:15.122 INFO:tasks.rados.rados.0.smithi090.stdout:1214: write oid 10 current snap is 124 2023-01-15T14:33:15.122 INFO:tasks.rados.rados.0.smithi090.stdout:1214: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2023-01-15T14:33:15.123 INFO:tasks.rados.rados.0.smithi090.stdout:1214: writing smithi09025264-10 from 729536 to 1189562 tid 1 2023-01-15T14:33:15.124 INFO:tasks.rados.rados.0.smithi090.stdout:1214: writing smithi09025264-10 from 1749580 to 2360729 tid 2 2023-01-15T14:33:15.124 INFO:tasks.rados.rados.0.smithi090.stdout:1214: writing smithi09025264-10 from 2853383 to 3305833 tid 3 2023-01-15T14:33:15.124 INFO:tasks.rados.rados.0.smithi090.stdout:1215: read oid 34 snap -1 2023-01-15T14:33:15.124 INFO:tasks.rados.rados.0.smithi090.stdout:1215: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:33:15.124 INFO:tasks.rados.rados.0.smithi090.stdout:1216: copy_from oid 30 from oid 18 current snap is 124 2023-01-15T14:33:15.124 INFO:tasks.rados.rados.0.smithi090.stdout:1217: copy_from oid 2 from oid 17 current snap is 124 2023-01-15T14:33:15.124 INFO:tasks.rados.rados.0.smithi090.stdout:1218: copy_from oid 33 from oid 38 current snap is 124 2023-01-15T14:33:15.124 INFO:tasks.rados.rados.0.smithi090.stdout:1219: snap_create 2023-01-15T14:33:15.124 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:33:15.125 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:15.125 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:15.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:33:15.128 INFO:tasks.rados.rados.0.smithi090.stdout:1215: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:33:15.128 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:15.117+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:15.202 INFO:tasks.rados.rados.0.smithi090.stdout:1215: done (4 left) 2023-01-15T14:33:15.329 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:33:15.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:33:15.334 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:15.334 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:15.335 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:33:15.336 INFO:tasks.rados.rados.0.smithi090.stdout:1218: finishing copy_from to smithi09025264-33 2023-01-15T14:33:15.337 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 197 (ObjNum 397 snap 106 seq_num 397) dirty exists 2023-01-15T14:33:15.337 INFO:tasks.rados.rados.0.smithi090.stdout:1218: finishing copy_from racing read to smithi09025264-33 2023-01-15T14:33:15.337 INFO:tasks.rados.rados.0.smithi090.stdout:1218: done (3 left) 2023-01-15T14:33:15.337 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:15.338 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:15.325+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:15.397 INFO:tasks.rados.rados.0.smithi090.stdout:1216: finishing copy_from to smithi09025264-30 2023-01-15T14:33:15.489 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 225 (ObjNum 447 snap 123 seq_num 447) dirty exists 2023-01-15T14:33:15.489 INFO:tasks.rados.rados.0.smithi090.stdout:1216: finishing copy_from racing read to smithi09025264-30 2023-01-15T14:33:15.489 INFO:tasks.rados.rados.0.smithi090.stdout:1216: done (2 left) 2023-01-15T14:33:15.489 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:15.490 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:15.653 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:15.654 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:33:15.655 INFO:tasks.rados.rados.0.smithi090.stdout:1217: finishing copy_from to smithi09025264-2 2023-01-15T14:33:15.655 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 379 (ObjNum 427 snap 116 seq_num 427) dirty exists 2023-01-15T14:33:15.655 INFO:tasks.rados.rados.0.smithi090.stdout:1217: finishing copy_from racing read to smithi09025264-2 2023-01-15T14:33:15.655 INFO:tasks.rados.rados.0.smithi090.stdout:1217: done (1 left) 2023-01-15T14:33:15.656 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:15.656 INFO:tasks.rados.rados.0.smithi090.stdout:1214: finishing write tid 1 to smithi09025264-10 2023-01-15T14:33:15.657 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:15.488+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:15.658 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:15.653+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:15.698 INFO:tasks.rados.rados.0.smithi090.stdout:1214: finishing write tid 2 to smithi09025264-10 2023-01-15T14:33:15.817 INFO:tasks.rados.rados.0.smithi090.stdout:1214: finishing write tid 3 to smithi09025264-10 2023-01-15T14:33:15.819 INFO:tasks.rados.rados.0.smithi090.stdout:1214: finishing write tid 4 to smithi09025264-10 2023-01-15T14:33:15.819 INFO:tasks.rados.rados.0.smithi090.stdout:1214: finishing write tid 5 to smithi09025264-10 2023-01-15T14:33:15.819 INFO:tasks.rados.rados.0.smithi090.stdout:1214: oid 10 updating version 0 to 401 2023-01-15T14:33:15.819 INFO:tasks.rados.rados.0.smithi090.stdout:1214: oid 10 updating version 401 to 404 2023-01-15T14:33:15.819 INFO:tasks.rados.rados.0.smithi090.stdout:1214: oid 10 version 404 is already newer than 403 2023-01-15T14:33:15.819 INFO:tasks.rados.rados.0.smithi090.stdout:1214: oid 10 version 404 is already newer than 402 2023-01-15T14:33:15.819 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 404 (ObjNum 448 snap 124 seq_num 448) dirty exists 2023-01-15T14:33:15.820 INFO:tasks.rados.rados.0.smithi090.stdout:1214: left oid 10 (ObjNum 448 snap 124 seq_num 448) 2023-01-15T14:33:15.820 INFO:tasks.rados.rados.0.smithi090.stdout:1214: done (0 left) 2023-01-15T14:33:15.820 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:15.820 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:15.821 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:33:15.821 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:15.816+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:15.920 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:15.913+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:15.983 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:33:15.984 INFO:tasks.rados.rados.0.smithi090.stdout:1219: done (0 left) 2023-01-15T14:33:15.984 INFO:tasks.rados.rados.0.smithi090.stdout:1220: snap_remove snap 123 2023-01-15T14:33:15.984 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:33:15.985 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 86 2023-01-15T14:33:15.985 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:15.985 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:15.986 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:33:16.039 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:16.033+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:16.142 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:16.142 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:16.142 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:16.143 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:33:16.144 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:16.140+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:16.243 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:16.240+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:16.627 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:16.627 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:16.627 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:16.627 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:16.627 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:16.627 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:16.628 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:16.628 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:16.628 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:16.629 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:33:16.629 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:16.624+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:33:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:33:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:33:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:33:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:33:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:33:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:33:16.638 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:33:16.727 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:16.724+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:16.784 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:16.785 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:33:16.785 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:33:16.785 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:33:16.785 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:16.795 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:33:16.827 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:16.824+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:17.090 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:17.090 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:33:17.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:33:17.091 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:33:17.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.091 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:5 3.5 3:a1c47f99:::smithi09025264-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:17.091 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:33:17.091 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 291.12629588200002, 2023-01-15T14:33:17.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40548699700000002, 2023-01-15T14:33:17.091 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.091 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.091 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.091 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.092 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.092 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 5 2023-01-15T14:33:17.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.092 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:33:17.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781744+0000", 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782566+0000", 2023-01-15T14:33:17.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000821578 2023-01-15T14:33:17.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782588+0000", 2023-01-15T14:33:17.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.226e-05 2023-01-15T14:33:17.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782601+0000", 2023-01-15T14:33:17.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2998e-05 2023-01-15T14:33:17.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.099 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.099 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882850+0000", 2023-01-15T14:33:17.099 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100248908 2023-01-15T14:33:17.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.099 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:33:17.099 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882871+0000", 2023-01-15T14:33:17.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0873000000000002e-05 2023-01-15T14:33:17.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.100 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.100 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.100 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083679+0000", 2023-01-15T14:33:17.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200808083 2023-01-15T14:33:17.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.100 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.100 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.101 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083948+0000", 2023-01-15T14:33:17.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000269342 2023-01-15T14:33:17.101 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.101 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.101 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:17.101 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084161+0000", 2023-01-15T14:33:17.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021326900000000001 2023-01-15T14:33:17.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084655+0000", 2023-01-15T14:33:17.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049371300000000001 2023-01-15T14:33:17.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084674+0000", 2023-01-15T14:33:17.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9012999999999999e-05 2023-01-15T14:33:17.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084924+0000", 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024989899999999999 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085221+0000", 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029719600000000001 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186953+0000", 2023-01-15T14:33:17.104 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10173174 2023-01-15T14:33:17.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.104 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.104 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187026+0000", 2023-01-15T14:33:17.104 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2957e-05 2023-01-15T14:33:17.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.104 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.104 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187124+0000", 2023-01-15T14:33:17.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8146000000000005e-05 2023-01-15T14:33:17.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.105 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.105 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187233+0000", 2023-01-15T14:33:17.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010931599999999999 2023-01-15T14:33:17.108 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.108 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.108 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.108 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.108 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.108 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:6 3.5 3:a1c47f99:::smithi09025264-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:17.108 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:33:17.108 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 291.12104300999999, 2023-01-15T14:33:17.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50419954199999995, 2023-01-15T14:33:17.108 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 6 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:33:17.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786997+0000", 2023-01-15T14:33:17.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:17.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787486+0000", 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048882999999999999 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787514+0000", 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7591999999999999e-05 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787526+0000", 2023-01-15T14:33:17.112 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1894e-05 2023-01-15T14:33:17.112 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.112 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.112 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.112 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882966+0000", 2023-01-15T14:33:17.112 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095440450999999996 2023-01-15T14:33:17.112 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.112 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.112 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:17.119 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882973+0000", 2023-01-15T14:33:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2570000000000001e-06 2023-01-15T14:33:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084809+0000", 2023-01-15T14:33:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201835763 2023-01-15T14:33:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084928+0000", 2023-01-15T14:33:17.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118946 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187347+0000", 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102419285 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187450+0000", 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010286299999999999 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187735+0000", 2023-01-15T14:33:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028429900000000001 2023-01-15T14:33:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188113+0000", 2023-01-15T14:33:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037883200000000001 2023-01-15T14:33:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188138+0000", 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4110000000000001e-05 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188389+0000", 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025173900000000001 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189251+0000", 2023-01-15T14:33:17.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086129800000000003 2023-01-15T14:33:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290989+0000", 2023-01-15T14:33:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101738524 2023-01-15T14:33:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291075+0000", 2023-01-15T14:33:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5469000000000004e-05 2023-01-15T14:33:17.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291144+0000", 2023-01-15T14:33:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:33:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291199+0000", 2023-01-15T14:33:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4413000000000002e-05 2023-01-15T14:33:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:7 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:33:17.126 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 291.12049655300001, 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50407251399999997, 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 7 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.127 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:33:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:33:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787545+0000", 2023-01-15T14:33:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787551+0000", 2023-01-15T14:33:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8669999999999999e-06 2023-01-15T14:33:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787559+0000", 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9780000000000005e-06 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787564+0000", 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5709999999999997e-06 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883054+0000", 2023-01-15T14:33:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095489722999999999 2023-01-15T14:33:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883061+0000", 2023-01-15T14:33:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2760000000000003e-06 2023-01-15T14:33:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084987+0000", 2023-01-15T14:33:17.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20192545000000001 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085190+0000", 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020288899999999999 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188374+0000", 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103184449 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188543+0000", 2023-01-15T14:33:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016925299999999999 2023-01-15T14:33:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188748+0000", 2023-01-15T14:33:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020473600000000001 2023-01-15T14:33:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188986+0000", 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023780900000000001 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189006+0000", 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0604000000000001e-05 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189084+0000", 2023-01-15T14:33:17.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7526000000000003e-05 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189272+0000", 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018762 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291333+0000", 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102061185 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291466+0000", 2023-01-15T14:33:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013325699999999999 2023-01-15T14:33:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291535+0000", 2023-01-15T14:33:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9309999999999999e-05 2023-01-15T14:33:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291618+0000", 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2773000000000002e-05 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:8 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 291.120470571, 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50442076000000002, 2023-01-15T14:33:17.137 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.138 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:17.138 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.138 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.138 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.138 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 8 2023-01-15T14:33:17.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.138 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:33:17.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:33:17.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787571+0000", 2023-01-15T14:33:17.140 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:17.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.140 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787575+0000", 2023-01-15T14:33:17.140 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7890000000000001e-06 2023-01-15T14:33:17.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.140 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787589+0000", 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4212999999999999e-05 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787594+0000", 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1429999999999997e-06 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983143+0000", 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.195549003 2023-01-15T14:33:17.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983153+0000", 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4369999999999996e-06 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085286+0000", 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102133064 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.142 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085401+0000", 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115176 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189116+0000", 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103715444 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189221+0000", 2023-01-15T14:33:17.143 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010438 2023-01-15T14:33:17.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.144 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.144 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.144 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291724+0000", 2023-01-15T14:33:17.144 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102502998 2023-01-15T14:33:17.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.144 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.144 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.144 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291824+0000", 2023-01-15T14:33:17.144 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010010599999999999 2023-01-15T14:33:17.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291992+0000", 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016856900000000001 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:21 3.5 3:a75958b1:::smithi09025264-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:33:17.145 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 291.07638165600002, 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.45851924500000002, 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 21 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.146 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831657+0000", 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:17.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832814+0000", 2023-01-15T14:33:17.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011564909999999999 2023-01-15T14:33:17.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832864+0000", 2023-01-15T14:33:17.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0243e-05 2023-01-15T14:33:17.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832903+0000", 2023-01-15T14:33:17.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8704000000000001e-05 2023-01-15T14:33:17.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983202+0000", 2023-01-15T14:33:17.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.150299404 2023-01-15T14:33:17.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:17.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983208+0000", 2023-01-15T14:33:17.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0039999999999996e-06 2023-01-15T14:33:17.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185539+0000", 2023-01-15T14:33:17.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202330867 2023-01-15T14:33:17.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185803+0000", 2023-01-15T14:33:17.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026394299999999999 2023-01-15T14:33:17.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:17.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186037+0000", 2023-01-15T14:33:17.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023386300000000001 2023-01-15T14:33:17.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186637+0000", 2023-01-15T14:33:17.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060001800000000001 2023-01-15T14:33:17.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186665+0000", 2023-01-15T14:33:17.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8459999999999999e-05 2023-01-15T14:33:17.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187045+0000", 2023-01-15T14:33:17.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037992899999999999 2023-01-15T14:33:17.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188201+0000", 2023-01-15T14:33:17.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001155309 2023-01-15T14:33:17.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289903+0000", 2023-01-15T14:33:17.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101702601 2023-01-15T14:33:17.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289985+0000", 2023-01-15T14:33:17.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1617000000000001e-05 2023-01-15T14:33:17.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290065+0000", 2023-01-15T14:33:17.159 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0276000000000002e-05 2023-01-15T14:33:17.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290180+0000", 2023-01-15T14:33:17.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011464300000000001 2023-01-15T14:33:17.160 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.160 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.160 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.160 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.160 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:22 3.5 3:a75958b1:::smithi09025264-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:17.161 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:33:17.161 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 291.07191396799999, 2023-01-15T14:33:17.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65790599599999999, 2023-01-15T14:33:17.161 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.161 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.161 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.161 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.162 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.162 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 22 2023-01-15T14:33:17.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.162 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.162 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.162 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.162 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:33:17.162 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.163 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.163 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.163 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.163 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:33:17.163 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.163 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.163 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836126+0000", 2023-01-15T14:33:17.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:17.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836181+0000", 2023-01-15T14:33:17.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4246999999999997e-05 2023-01-15T14:33:17.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836212+0000", 2023-01-15T14:33:17.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1843e-05 2023-01-15T14:33:17.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836226+0000", 2023-01-15T14:33:17.256 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3216e-05 2023-01-15T14:33:17.256 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.256 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186757+0000", 2023-01-15T14:33:17.257 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35053103899999999 2023-01-15T14:33:17.257 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.257 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.257 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.257 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186871+0000", 2023-01-15T14:33:17.257 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114403 2023-01-15T14:33:17.257 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.257 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.257 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.257 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290263+0000", 2023-01-15T14:33:17.257 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103391976 2023-01-15T14:33:17.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.258 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.258 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.258 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290413+0000", 2023-01-15T14:33:17.258 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014991200000000001 2023-01-15T14:33:17.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.258 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.258 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:17.258 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290627+0000", 2023-01-15T14:33:17.258 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021379299999999999 2023-01-15T14:33:17.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290861+0000", 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023479600000000001 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290888+0000", 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6570000000000001e-05 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290990+0000", 2023-01-15T14:33:17.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101723 2023-01-15T14:33:17.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291245+0000", 2023-01-15T14:33:17.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025543899999999999 2023-01-15T14:33:17.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392822+0000", 2023-01-15T14:33:17.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10157725500000001 2023-01-15T14:33:17.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493881+0000", 2023-01-15T14:33:17.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101058262 2023-01-15T14:33:17.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493964+0000", 2023-01-15T14:33:17.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2973000000000007e-05 2023-01-15T14:33:17.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494034+0000", 2023-01-15T14:33:17.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0432999999999995e-05 2023-01-15T14:33:17.262 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.262 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.262 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.263 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:23 3.5 3:a75958b1:::smithi09025264-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:17.263 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:33:17.263 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 291.071798839, 2023-01-15T14:33:17.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65796116900000001, 2023-01-15T14:33:17.263 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.263 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.263 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.263 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.263 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 23 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:33:17.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836242+0000", 2023-01-15T14:33:17.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:17.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836256+0000", 2023-01-15T14:33:17.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3677e-05 2023-01-15T14:33:17.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836265+0000", 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2769999999999994e-06 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936398+0000", 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100132559 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289431+0000", 2023-01-15T14:33:17.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35303322399999998 2023-01-15T14:33:17.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289587+0000", 2023-01-15T14:33:17.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000155873 2023-01-15T14:33:17.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292052+0000", 2023-01-15T14:33:17.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0024647919999999999 2023-01-15T14:33:17.268 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.268 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.268 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292180+0000", 2023-01-15T14:33:17.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012810999999999999 2023-01-15T14:33:17.268 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.268 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.268 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:17.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292373+0000", 2023-01-15T14:33:17.269 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019258599999999999 2023-01-15T14:33:17.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.269 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.269 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292597+0000", 2023-01-15T14:33:17.269 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022465500000000001 2023-01-15T14:33:17.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.269 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292618+0000", 2023-01-15T14:33:17.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0319999999999999e-05 2023-01-15T14:33:17.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.270 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292692+0000", 2023-01-15T14:33:17.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4643999999999999e-05 2023-01-15T14:33:17.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.270 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393103+0000", 2023-01-15T14:33:17.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100411156 2023-01-15T14:33:17.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493827+0000", 2023-01-15T14:33:17.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100724143 2023-01-15T14:33:17.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494085+0000", 2023-01-15T14:33:17.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000257891 2023-01-15T14:33:17.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494141+0000", 2023-01-15T14:33:17.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6132999999999999e-05 2023-01-15T14:33:17.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494204+0000", 2023-01-15T14:33:17.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2966000000000007e-05 2023-01-15T14:33:17.273 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.273 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.273 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.273 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:24 3.5 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:17.274 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:33:17.274 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 290.97161741299999, 2023-01-15T14:33:17.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.55808677600000001, 2023-01-15T14:33:17.274 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.274 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:17.274 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.274 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.274 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.274 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 24 2023-01-15T14:33:17.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.274 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:33:17.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:33:17.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936423+0000", 2023-01-15T14:33:17.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:17.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936433+0000", 2023-01-15T14:33:17.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0443e-05 2023-01-15T14:33:17.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936451+0000", 2023-01-15T14:33:17.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7842999999999998e-05 2023-01-15T14:33:17.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936461+0000", 2023-01-15T14:33:17.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0057e-05 2023-01-15T14:33:17.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289693+0000", 2023-01-15T14:33:17.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.353231617 2023-01-15T14:33:17.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289814+0000", 2023-01-15T14:33:17.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120763 2023-01-15T14:33:17.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392923+0000", 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10310955300000001 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393072+0000", 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000148469 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494242+0000", 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117034499999999 2023-01-15T14:33:17.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494363+0000", 2023-01-15T14:33:17.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120703 2023-01-15T14:33:17.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494511+0000", 2023-01-15T14:33:17.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014873299999999999 2023-01-15T14:33:17.280 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.281 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.281 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.281 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:78 3.c 3:3137f0cb:::smithi09025264-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.281 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:33:17.281 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 290.58165141699999, 2023-01-15T14:33:17.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20331801699999999, 2023-01-15T14:33:17.281 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.281 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.282 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.282 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.282 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.282 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 78 2023-01-15T14:33:17.282 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.282 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.282 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.282 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:33:17.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.282 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326388+0000", 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326672+0000", 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028367900000000001 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326713+0000", 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1539999999999999e-05 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326729+0000", 2023-01-15T14:33:17.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.519e-05 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326879+0000", 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015064899999999999 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327096+0000", 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021682299999999999 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327308+0000", 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000211366 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327757+0000", 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000449009 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327776+0000", 2023-01-15T14:33:17.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9201000000000001e-05 2023-01-15T14:33:17.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328012+0000", 2023-01-15T14:33:17.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236054 2023-01-15T14:33:17.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429534+0000", 2023-01-15T14:33:17.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10152201499999999 2023-01-15T14:33:17.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529340+0000", 2023-01-15T14:33:17.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099806249 2023-01-15T14:33:17.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529437+0000", 2023-01-15T14:33:17.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6949999999999998e-05 2023-01-15T14:33:17.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529587+0000", 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015008600000000001 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529709+0000", 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121636 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.289 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:79 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 290.581281821, 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20348171300000001, 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.290 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 79 2023-01-15T14:33:17.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.291 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:33:17.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:33:17.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326759+0000", 2023-01-15T14:33:17.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:17.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326770+0000", 2023-01-15T14:33:17.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:33:17.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326791+0000", 2023-01-15T14:33:17.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1556000000000002e-05 2023-01-15T14:33:17.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326818+0000", 2023-01-15T14:33:17.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6979999999999999e-05 2023-01-15T14:33:17.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.294 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.294 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.427993+0000", 2023-01-15T14:33:17.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117517199999999 2023-01-15T14:33:17.294 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.294 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.294 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.294 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428266+0000", 2023-01-15T14:33:17.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027233299999999998 2023-01-15T14:33:17.294 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.294 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.294 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428696+0000", 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042996499999999998 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429057+0000", 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000360952 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429076+0000", 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:33:17.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429160+0000", 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429549+0000", 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038884599999999998 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529886+0000", 2023-01-15T14:33:17.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100336261 2023-01-15T14:33:17.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529978+0000", 2023-01-15T14:33:17.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2775999999999994e-05 2023-01-15T14:33:17.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530108+0000", 2023-01-15T14:33:17.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012958300000000001 2023-01-15T14:33:17.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530242+0000", 2023-01-15T14:33:17.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013406 2023-01-15T14:33:17.298 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.298 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.298 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.299 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:80 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.299 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:33:17.299 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 290.581143462, 2023-01-15T14:33:17.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20370874, 2023-01-15T14:33:17.299 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.299 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:17.299 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.299 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.299 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.299 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 80 2023-01-15T14:33:17.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.300 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.300 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:33:17.300 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.300 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:33:17.300 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326897+0000", 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326906+0000", 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6929999999999996e-06 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326931+0000", 2023-01-15T14:33:17.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4757e-05 2023-01-15T14:33:17.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326945+0000", 2023-01-15T14:33:17.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.402e-05 2023-01-15T14:33:17.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429239+0000", 2023-01-15T14:33:17.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102294418 2023-01-15T14:33:17.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429487+0000", 2023-01-15T14:33:17.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024789300000000001 2023-01-15T14:33:17.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530309+0000", 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100821459 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530438+0000", 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012928700000000001 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530607+0000", 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169483 2023-01-15T14:33:17.304 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.305 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.305 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.305 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.305 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:78 3.c e23/20 ondisk, result = 0)", 2023-01-15T14:33:17.305 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:33:17.305 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 290.57925174100001, 2023-01-15T14:33:17.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20094287799999999, 2023-01-15T14:33:17.305 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.305 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:17.306 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:33:17.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:33:17.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328789+0000", 2023-01-15T14:33:17.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:17.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328799+0000", 2023-01-15T14:33:17.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8980000000000008e-06 2023-01-15T14:33:17.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.308 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.308 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328820+0000", 2023-01-15T14:33:17.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1041999999999999e-05 2023-01-15T14:33:17.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.308 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.308 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328832+0000", 2023-01-15T14:33:17.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1616e-05 2023-01-15T14:33:17.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529404+0000", 2023-01-15T14:33:17.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200572631 2023-01-15T14:33:17.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529419+0000", 2023-01-15T14:33:17.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4250000000000001e-05 2023-01-15T14:33:17.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.310 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.310 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529733+0000", 2023-01-15T14:33:17.310 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000314693 2023-01-15T14:33:17.310 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.310 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.310 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.310 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.310 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.310 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:201 3.5 3:a1c47f99:::smithi09025264-2:head [copy-from ver 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.310 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:33:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 289.501645344, 2023-01-15T14:33:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30931411399999997, 2023-01-15T14:33:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 201 2023-01-15T14:33:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:33:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:33:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406394+0000", 2023-01-15T14:33:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406408+0000", 2023-01-15T14:33:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4416999999999999e-05 2023-01-15T14:33:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406446+0000", 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7339999999999998e-05 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406467+0000", 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1137000000000001e-05 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406677+0000", 2023-01-15T14:33:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000210275 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406902+0000", 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022529000000000001 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.513012+0000", 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.106109527 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514754+0000", 2023-01-15T14:33:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0017419779999999999 2023-01-15T14:33:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514773+0000", 2023-01-15T14:33:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8700000000000001e-05 2023-01-15T14:33:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.515531+0000", 2023-01-15T14:33:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075844300000000003 2023-01-15T14:33:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615334+0000", 2023-01-15T14:33:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099803327999999997 2023-01-15T14:33:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615376+0000", 2023-01-15T14:33:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1675999999999999e-05 2023-01-15T14:33:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715546+0000", 2023-01-15T14:33:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016992500000001 2023-01-15T14:33:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715626+0000", 2023-01-15T14:33:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0112999999999997e-05 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715711+0000", 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:202 3.5 3:a1c47f99:::smithi09025264-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 289.50153279800003, 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31002637300000002, 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 202 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:33:17.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:33:17.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406508+0000", 2023-01-15T14:33:17.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:17.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406516+0000", 2023-01-15T14:33:17.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.413e-06 2023-01-15T14:33:17.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406534+0000", 2023-01-15T14:33:17.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8936999999999998e-05 2023-01-15T14:33:17.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406548+0000", 2023-01-15T14:33:17.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3196e-05 2023-01-15T14:33:17.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407271+0000", 2023-01-15T14:33:17.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00072301500000000003 2023-01-15T14:33:17.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:17.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407466+0000", 2023-01-15T14:33:17.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000195009 2023-01-15T14:33:17.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.614996+0000", 2023-01-15T14:33:17.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.207530192 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615227+0000", 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000230979 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716282+0000", 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101055465 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716409+0000", 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012653999999999999 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716536+0000", 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012683 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.327 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:189 3.d e23/20 3:b3f111b5:::smithi09025264-48:head v 23'10, mlcod=23'6)", 2023-01-15T14:33:17.327 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:33:17.327 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 289.45632814999999, 2023-01-15T14:33:17.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20400475100000001, 2023-01-15T14:33:17.327 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.327 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.327 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451712+0000", 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:17.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452272+0000", 2023-01-15T14:33:17.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00056035899999999997 2023-01-15T14:33:17.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452310+0000", 2023-01-15T14:33:17.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8145999999999997e-05 2023-01-15T14:33:17.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452324+0000", 2023-01-15T14:33:17.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4443999999999999e-05 2023-01-15T14:33:17.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652703+0000", 2023-01-15T14:33:17.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20037818199999999 2023-01-15T14:33:17.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652762+0000", 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9302999999999998e-05 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653302+0000", 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054012199999999998 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653333+0000", 2023-01-15T14:33:17.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.112e-05 2023-01-15T14:33:17.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653667+0000", 2023-01-15T14:33:17.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033337900000000002 2023-01-15T14:33:17.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653709+0000", 2023-01-15T14:33:17.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2469999999999998e-05 2023-01-15T14:33:17.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655719+0000", 2023-01-15T14:33:17.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020096699999999999 2023-01-15T14:33:17.333 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.333 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.333 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.333 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:269 3.2 e30/20 3:47dfe935:::smithi09025264-29:head v 30'11, mlcod=23'8)", 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 282.03028270900001, 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23185540399999999, 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878379+0000", 2023-01-15T14:33:17.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061979600000000004 2023-01-15T14:33:17.336 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.336 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.336 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.336 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878406+0000", 2023-01-15T14:33:17.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:33:17.336 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.336 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.336 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.336 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978515+0000", 2023-01-15T14:33:17.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100108461 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078759+0000", 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100244496 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078796+0000", 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6758999999999998e-05 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.337 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079567+0000", 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077137199999999999 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079593+0000", 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079844+0000", 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025075899999999998 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079910+0000", 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6406000000000001e-05 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.109615+0000", 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.029704323000000001 2023-01-15T14:33:17.339 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:281 3.5 3:a75958b1:::smithi09025264-6:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:33:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:33:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 281.92935907999998, 2023-01-15T14:33:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30574310900000001, 2023-01-15T14:33:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 281 2023-01-15T14:33:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:33:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:33:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:33:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978686+0000", 2023-01-15T14:33:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4429999999999999e-06 2023-01-15T14:33:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978690+0000", 2023-01-15T14:33:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0300000000000004e-06 2023-01-15T14:33:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978693+0000", 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.179e-06 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981592+0000", 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0028984140000000002 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981718+0000", 2023-01-15T14:33:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012616600000000001 2023-01-15T14:33:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981982+0000", 2023-01-15T14:33:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026445199999999998 2023-01-15T14:33:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982278+0000", 2023-01-15T14:33:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029514100000000001 2023-01-15T14:33:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982290+0000", 2023-01-15T14:33:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.271e-05 2023-01-15T14:33:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982343+0000", 2023-01-15T14:33:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2914000000000002e-05 2023-01-15T14:33:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982385+0000", 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2114999999999999e-05 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.181937+0000", 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19955213099999999 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284183+0000", 2023-01-15T14:33:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10224538 2023-01-15T14:33:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284318+0000", 2023-01-15T14:33:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000134843 2023-01-15T14:33:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284426+0000", 2023-01-15T14:33:17.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108513 2023-01-15T14:33:17.349 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.349 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.349 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.349 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:373 3.4 e34/20 3:2029fa71:::smithi09025264-39:head v 34'19, mlcod=26'16)", 2023-01-15T14:33:17.349 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:33:17.349 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 276.44837662100002, 2023-01-15T14:33:17.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20212622799999999, 2023-01-15T14:33:17.349 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459664+0000", 2023-01-15T14:33:17.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:17.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460099+0000", 2023-01-15T14:33:17.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043515900000000001 2023-01-15T14:33:17.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460130+0000", 2023-01-15T14:33:17.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1590000000000001e-05 2023-01-15T14:33:17.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560226+0000", 2023-01-15T14:33:17.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100095248 2023-01-15T14:33:17.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560369+0000", 2023-01-15T14:33:17.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014381499999999999 2023-01-15T14:33:17.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560404+0000", 2023-01-15T14:33:17.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4650000000000002e-05 2023-01-15T14:33:17.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560804+0000", 2023-01-15T14:33:17.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000399823 2023-01-15T14:33:17.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560827+0000", 2023-01-15T14:33:17.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3612999999999999e-05 2023-01-15T14:33:17.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.561101+0000", 2023-01-15T14:33:17.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027355499999999998 2023-01-15T14:33:17.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661710+0000", 2023-01-15T14:33:17.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100608716 2023-01-15T14:33:17.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661792+0000", 2023-01-15T14:33:17.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2006000000000001e-05 2023-01-15T14:33:17.356 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.356 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.356 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.356 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:455 3.d e41/20 3:bd982ca3:::smithi09025264-38:head v 41'24, mlcod=38'18)", 2023-01-15T14:33:17.356 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:33:17.356 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 269.39861504800001, 2023-01-15T14:33:17.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20118033699999999, 2023-01-15T14:33:17.356 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.356 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.357 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:33:17.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:33:17.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509425+0000", 2023-01-15T14:33:17.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:17.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509441+0000", 2023-01-15T14:33:17.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5248e-05 2023-01-15T14:33:17.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509474+0000", 2023-01-15T14:33:17.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3531000000000001e-05 2023-01-15T14:33:17.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.609644+0000", 2023-01-15T14:33:17.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016994999999999 2023-01-15T14:33:17.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709872+0000", 2023-01-15T14:33:17.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10022840299999999 2023-01-15T14:33:17.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.416 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709906+0000", 2023-01-15T14:33:17.416 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3160000000000001e-05 2023-01-15T14:33:17.416 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.416 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.416 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710153+0000", 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024735399999999998 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710204+0000", 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1075999999999997e-05 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710280+0000", 2023-01-15T14:33:17.417 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6202e-05 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710317+0000", 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6387000000000002e-05 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710607+0000", 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029079699999999997 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.418 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.419 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.419 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.419 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.419 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:497 3.d e44/20 3:bd982ca3:::smithi09025264-38:head v 44'31, mlcod=42'26)", 2023-01-15T14:33:17.419 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:33:17.419 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 267.13382018599998, 2023-01-15T14:33:17.419 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20082067100000001, 2023-01-15T14:33:17.419 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.419 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.419 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.419 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.420 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774220+0000", 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774238+0000", 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8107000000000001e-05 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774263+0000", 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4122999999999999e-05 2023-01-15T14:33:17.421 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774280+0000", 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7039999999999999e-05 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874060+0000", 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099780698000000001 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.422 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874130+0000", 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9852999999999997e-05 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874351+0000", 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022091400000000001 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874369+0000", 2023-01-15T14:33:17.423 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7876e-05 2023-01-15T14:33:17.424 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.424 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.424 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.424 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874433+0000", 2023-01-15T14:33:17.424 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3793000000000004e-05 2023-01-15T14:33:17.424 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.424 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.424 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.424 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.974927+0000", 2023-01-15T14:33:17.424 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10049393500000001 2023-01-15T14:33:17.424 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.425 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.425 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.425 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.975043+0000", 2023-01-15T14:33:17.425 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115949 2023-01-15T14:33:17.425 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.425 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.425 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.425 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.425 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.425 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:17.426 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:33:17.426 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:17.084+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:17.428 INFO:tasks.rados.rados.0.smithi090.stdout:1220: done (0 left) 2023-01-15T14:33:17.428 INFO:tasks.rados.rados.0.smithi090.stdout:1221: read oid 34 snap -1 2023-01-15T14:33:17.428 INFO:tasks.rados.rados.0.smithi090.stdout:1221: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:33:17.428 INFO:tasks.rados.rados.0.smithi090.stdout:1222: rollback oid 28 current snap is 125 2023-01-15T14:33:17.428 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 28 to 124 2023-01-15T14:33:17.428 INFO:tasks.rados.rados.0.smithi090.stdout:1223: read oid 23 snap -1 2023-01-15T14:33:17.428 INFO:tasks.rados.rados.0.smithi090.stdout:1223: expect (ObjNum 406 snap 108 seq_num 406) 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout:1224: write oid 30 current snap is 125 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout:1224: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout:1224: writing smithi09025264-30 from 746343 to 1397971 tid 1 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout:1224: writing smithi09025264-30 from 1924415 to 2679222 tid 2 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout:1224: writing smithi09025264-30 from 3397159 to 3546343 tid 3 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout:1225: snap_create 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout:1222: finishing rollback tid 0 to smithi09025264-28 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout:1223: expect (ObjNum 406 snap 108 seq_num 406) 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout:1221: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout:1224: finishing write tid 1 to smithi09025264-30 2023-01-15T14:33:17.429 INFO:tasks.rados.rados.0.smithi090.stdout:1222: finishing rollback tid 1 to smithi09025264-28 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout:1224: finishing write tid 2 to smithi09025264-30 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout:1224: finishing write tid 3 to smithi09025264-30 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout:1222: finishing rollback tid 2 to smithi09025264-28 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 359 (ObjNum 426 snap 116 seq_num 426) dirty exists 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout:1221: done (3 left) 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout:1222: done (2 left) 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout:1223: done (1 left) 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout:1224: finishing write tid 4 to smithi09025264-30 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout:1224: finishing write tid 5 to smithi09025264-30 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout:1224: oid 30 updating version 0 to 226 2023-01-15T14:33:17.430 INFO:tasks.rados.rados.0.smithi090.stdout:1224: oid 30 updating version 226 to 228 2023-01-15T14:33:17.431 INFO:tasks.rados.rados.0.smithi090.stdout:1224: oid 30 version 228 is already newer than 227 2023-01-15T14:33:17.431 INFO:tasks.rados.rados.0.smithi090.stdout:1224: oid 30 updating version 228 to 229 2023-01-15T14:33:17.431 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 229 (ObjNum 449 snap 125 seq_num 449) dirty exists 2023-01-15T14:33:17.431 INFO:tasks.rados.rados.0.smithi090.stdout:1224: left oid 30 (ObjNum 449 snap 125 seq_num 449) 2023-01-15T14:33:17.431 INFO:tasks.rados.rados.0.smithi090.stdout:1224: done (0 left) 2023-01-15T14:33:17.431 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:17.425+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:17.529 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:17.521+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:17.539 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:17.539 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:33:17.539 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:33:17.539 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:17.549 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:33:17.630 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:17.628+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:17.656 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:17.656 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:33:17.656 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:33:17.656 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:33:17.656 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:17.665 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:33:17.731 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:17.728+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:17.776 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:17.776 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:33:17.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:33:17.777 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:33:17.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.777 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:97 3.a 3:52df5138:::smithi09025264-25:head [write 403368~555035,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.777 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:33:17.777 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 291.270413555, 2023-01-15T14:33:17.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.340758171, 2023-01-15T14:33:17.777 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.777 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.777 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.778 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.778 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.778 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 97 2023-01-15T14:33:17.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.778 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:33:17.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:33:17.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501140+0000", 2023-01-15T14:33:17.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:17.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501684+0000", 2023-01-15T14:33:17.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054441799999999996 2023-01-15T14:33:17.780 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.780 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.780 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501713+0000", 2023-01-15T14:33:17.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8833e-05 2023-01-15T14:33:17.780 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.780 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.780 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501726+0000", 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3137e-05 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601982+0000", 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100256518 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602426+0000", 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044401500000000002 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:17.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602703+0000", 2023-01-15T14:33:17.782 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000276889 2023-01-15T14:33:17.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603066+0000", 2023-01-15T14:33:17.782 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036278599999999998 2023-01-15T14:33:17.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603098+0000", 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1433000000000002e-05 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603382+0000", 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028402899999999999 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603632+0000", 2023-01-15T14:33:17.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025033 2023-01-15T14:33:17.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805354+0000", 2023-01-15T14:33:17.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20172168700000001 2023-01-15T14:33:17.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841465+0000", 2023-01-15T14:33:17.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.036111852999999999 2023-01-15T14:33:17.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841682+0000", 2023-01-15T14:33:17.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000216949 2023-01-15T14:33:17.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841901+0000", 2023-01-15T14:33:17.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021858700000000001 2023-01-15T14:33:17.785 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.786 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.786 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.786 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.786 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.786 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:98 3.a 3:52df5138:::smithi09025264-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.786 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:33:17.786 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 291.269803884, 2023-01-15T14:33:17.786 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.44410839299999999, 2023-01-15T14:33:17.786 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.786 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.786 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.787 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.787 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.787 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 98 2023-01-15T14:33:17.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.787 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.787 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.787 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:33:17.787 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.788 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.788 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:33:17.788 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.788 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.788 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.788 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501752+0000", 2023-01-15T14:33:17.788 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:17.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502223+0000", 2023-01-15T14:33:17.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000471843 2023-01-15T14:33:17.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502240+0000", 2023-01-15T14:33:17.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6663e-05 2023-01-15T14:33:17.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502248+0000", 2023-01-15T14:33:17.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5669999999999997e-06 2023-01-15T14:33:17.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603236+0000", 2023-01-15T14:33:17.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100988439 2023-01-15T14:33:17.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603384+0000", 2023-01-15T14:33:17.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014844200000000001 2023-01-15T14:33:17.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842016+0000", 2023-01-15T14:33:17.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23863163800000001 2023-01-15T14:33:17.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842267+0000", 2023-01-15T14:33:17.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025101300000000002 2023-01-15T14:33:17.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:17.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842561+0000", 2023-01-15T14:33:17.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029432900000000002 2023-01-15T14:33:17.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843120+0000", 2023-01-15T14:33:17.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055900500000000001 2023-01-15T14:33:17.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843146+0000", 2023-01-15T14:33:17.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5273e-05 2023-01-15T14:33:17.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843512+0000", 2023-01-15T14:33:17.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036621599999999999 2023-01-15T14:33:17.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943324+0000", 2023-01-15T14:33:17.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099812008999999993 2023-01-15T14:33:17.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945437+0000", 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021126069999999999 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945619+0000", 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018280600000000001 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945744+0000", 2023-01-15T14:33:17.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012511299999999999 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945861+0000", 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011646000000000001 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:99 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.796 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:33:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 291.26929362800001, 2023-01-15T14:33:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.54355587900000002, 2023-01-15T14:33:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 99 2023-01-15T14:33:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:33:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:33:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502262+0000", 2023-01-15T14:33:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502270+0000", 2023-01-15T14:33:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1999999999999994e-06 2023-01-15T14:33:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502286+0000", 2023-01-15T14:33:17.800 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5423000000000001e-05 2023-01-15T14:33:17.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.800 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.800 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.800 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502295+0000", 2023-01-15T14:33:17.800 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4339999999999998e-06 2023-01-15T14:33:17.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.800 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.800 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.800 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603467+0000", 2023-01-15T14:33:17.800 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101172225 2023-01-15T14:33:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603585+0000", 2023-01-15T14:33:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117149 2023-01-15T14:33:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843335+0000", 2023-01-15T14:33:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23975083799999999 2023-01-15T14:33:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843573+0000", 2023-01-15T14:33:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023753000000000001 2023-01-15T14:33:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843978+0000", 2023-01-15T14:33:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040533799999999999 2023-01-15T14:33:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844309+0000", 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000330713 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844423+0000", 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114176 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844510+0000", 2023-01-15T14:33:17.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7026000000000003e-05 2023-01-15T14:33:17.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943338+0000", 2023-01-15T14:33:17.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098827398999999996 2023-01-15T14:33:17.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945520+0000", 2023-01-15T14:33:17.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021824169999999999 2023-01-15T14:33:17.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045652+0000", 2023-01-15T14:33:17.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013193200000001 2023-01-15T14:33:17.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045733+0000", 2023-01-15T14:33:17.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1435999999999998e-05 2023-01-15T14:33:17.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045819+0000", 2023-01-15T14:33:17.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5292999999999998e-05 2023-01-15T14:33:17.806 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.806 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.806 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.806 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:100 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.807 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:33:17.807 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 291.269245705, 2023-01-15T14:33:17.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.64425290599999996, 2023-01-15T14:33:17.807 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.807 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:17.807 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.807 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.807 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.807 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 100 2023-01-15T14:33:17.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.807 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.836 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:33:17.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:33:17.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502310+0000", 2023-01-15T14:33:17.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:17.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502316+0000", 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7470000000000002e-06 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502332+0000", 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6866000000000001e-05 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502343+0000", 2023-01-15T14:33:17.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0913e-05 2023-01-15T14:33:17.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.840 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.840 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703594+0000", 2023-01-15T14:33:17.840 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20125087999999999 2023-01-15T14:33:17.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.840 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.840 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703751+0000", 2023-01-15T14:33:17.840 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001573 2023-01-15T14:33:17.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845262+0000", 2023-01-15T14:33:17.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.141510521 2023-01-15T14:33:17.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:17.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845391+0000", 2023-01-15T14:33:17.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012872299999999999 2023-01-15T14:33:17.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146051+0000", 2023-01-15T14:33:17.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.300660023 2023-01-15T14:33:17.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146264+0000", 2023-01-15T14:33:17.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212754 2023-01-15T14:33:17.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146564+0000", 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030010200000000001 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:203 3.a 3:52df5138:::smithi09025264-25:head [write 440350~460568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 290.21844231, 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40523678899999999, 2023-01-15T14:33:17.843 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.844 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.844 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.844 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.844 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.844 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 203 2023-01-15T14:33:17.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.844 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:33:17.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:33:17.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553111+0000", 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553483+0000", 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037153499999999999 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553517+0000", 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4687000000000002e-05 2023-01-15T14:33:17.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553537+0000", 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.967e-05 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553683+0000", 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000146341 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553880+0000", 2023-01-15T14:33:17.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019671900000000001 2023-01-15T14:33:17.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:17.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554219+0000", 2023-01-15T14:33:17.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033893299999999998 2023-01-15T14:33:17.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554536+0000", 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316576 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554562+0000", 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5905999999999999e-05 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554801+0000", 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000239791 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555274+0000", 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047294500000000001 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757267+0000", 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20199229399999999 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958174+0000", 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20090739499999999 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958260+0000", 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6316999999999994e-05 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958351+0000", 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0573000000000002e-05 2023-01-15T14:33:17.851 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.852 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.852 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.852 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.852 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.852 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:204 3.a 3:52df5138:::smithi09025264-25:head [write 1309629~542666] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.852 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:33:17.852 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 290.217979442, 2023-01-15T14:33:17.852 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40500913999999999, 2023-01-15T14:33:17.852 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.852 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.853 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.853 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.853 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.853 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 204 2023-01-15T14:33:17.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.853 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.853 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.853 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553575+0000", 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:17.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554126+0000", 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055090499999999997 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554150+0000", 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3387000000000002e-05 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554165+0000", 2023-01-15T14:33:17.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5116000000000001e-05 2023-01-15T14:33:17.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554626+0000", 2023-01-15T14:33:17.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046093499999999999 2023-01-15T14:33:17.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554738+0000", 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112446 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554898+0000", 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016003100000000001 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555182+0000", 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028382700000000001 2023-01-15T14:33:17.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.858 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.858 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555199+0000", 2023-01-15T14:33:17.858 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6969e-05 2023-01-15T14:33:17.858 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.858 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.858 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555449+0000", 2023-01-15T14:33:17.858 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000250265 2023-01-15T14:33:17.858 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.858 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555574+0000", 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124628 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757316+0000", 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20174214300000001 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958460+0000", 2023-01-15T14:33:17.860 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201144464 2023-01-15T14:33:17.860 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.860 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.860 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958534+0000", 2023-01-15T14:33:17.860 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3380000000000006e-05 2023-01-15T14:33:17.860 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.860 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.860 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958586+0000", 2023-01-15T14:33:17.860 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2370000000000002e-05 2023-01-15T14:33:17.861 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.861 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.861 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.861 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.861 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:205 3.a 3:52df5138:::smithi09025264-25:head [write 2323973~516377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:17.861 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:33:17.861 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 290.21736134700001, 2023-01-15T14:33:17.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404601398, 2023-01-15T14:33:17.861 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.861 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.862 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.862 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.862 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.862 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 205 2023-01-15T14:33:17.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.862 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.862 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.862 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:33:17.862 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:33:17.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554193+0000", 2023-01-15T14:33:17.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:17.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.864 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.864 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554850+0000", 2023-01-15T14:33:17.864 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065674799999999997 2023-01-15T14:33:17.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.864 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.864 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554873+0000", 2023-01-15T14:33:17.864 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2935999999999999e-05 2023-01-15T14:33:17.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.864 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.654962+0000", 2023-01-15T14:33:17.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088949 2023-01-15T14:33:17.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.865 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655107+0000", 2023-01-15T14:33:17.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144759 2023-01-15T14:33:17.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.865 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655373+0000", 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026684899999999999 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655720+0000", 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034601899999999998 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656168+0000", 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044851899999999998 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.866 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.867 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656190+0000", 2023-01-15T14:33:17.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1817000000000001e-05 2023-01-15T14:33:17.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.867 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656574+0000", 2023-01-15T14:33:17.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038404200000000002 2023-01-15T14:33:17.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657649+0000", 2023-01-15T14:33:17.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010746670000000001 2023-01-15T14:33:17.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757539+0000", 2023-01-15T14:33:17.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099890899000000005 2023-01-15T14:33:17.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958683+0000", 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20114332400000001 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958746+0000", 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3509000000000005e-05 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958796+0000", 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0176999999999997e-05 2023-01-15T14:33:17.869 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.870 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.870 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.870 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.870 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.870 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:358 3.a 3:52df5138:::smithi09025264-25:head [write 709262~697258] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:17.870 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:33:17.870 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 277.72077381399998, 2023-01-15T14:33:17.870 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50589529799999999, 2023-01-15T14:33:17.870 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.870 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.871 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.871 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.871 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 358 2023-01-15T14:33:17.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.871 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:33:17.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050780+0000", 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051369+0000", 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058926200000000001 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051390+0000", 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1426000000000001e-05 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051403+0000", 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051520+0000", 2023-01-15T14:33:17.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117318 2023-01-15T14:33:17.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051785+0000", 2023-01-15T14:33:17.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000265018 2023-01-15T14:33:17.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:17.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052173+0000", 2023-01-15T14:33:17.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038734799999999998 2023-01-15T14:33:17.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052705+0000", 2023-01-15T14:33:17.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053209899999999996 2023-01-15T14:33:17.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052727+0000", 2023-01-15T14:33:17.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2362e-05 2023-01-15T14:33:17.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053205+0000", 2023-01-15T14:33:17.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047787200000000001 2023-01-15T14:33:17.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055711+0000", 2023-01-15T14:33:17.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0025054880000000002 2023-01-15T14:33:17.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.877 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.877 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.877 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.255549+0000", 2023-01-15T14:33:17.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199838285 2023-01-15T14:33:17.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.877 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.877 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.877 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556501+0000", 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30095197499999998 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556613+0000", 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111799 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556678+0000", 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5220000000000002e-05 2023-01-15T14:33:17.878 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.879 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.879 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.879 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.879 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:359 3.a 3:52df5138:::smithi09025264-25:head [write 2012635~572018] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:17.879 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:33:17.879 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 277.72012322, 2023-01-15T14:33:17.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50546442499999999, 2023-01-15T14:33:17.879 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.879 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.879 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.880 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.880 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.880 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 359 2023-01-15T14:33:17.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.880 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:33:17.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051432+0000", 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051808+0000", 2023-01-15T14:33:17.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000376074 2023-01-15T14:33:17.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051834+0000", 2023-01-15T14:33:17.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6018999999999999e-05 2023-01-15T14:33:17.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051869+0000", 2023-01-15T14:33:17.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4839999999999998e-05 2023-01-15T14:33:17.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052857+0000", 2023-01-15T14:33:17.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00098761399999999994 2023-01-15T14:33:17.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053071+0000", 2023-01-15T14:33:17.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000214112 2023-01-15T14:33:17.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:17.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053876+0000", 2023-01-15T14:33:17.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000804873 2023-01-15T14:33:17.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054460+0000", 2023-01-15T14:33:17.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058436100000000004 2023-01-15T14:33:17.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054486+0000", 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069999999999999e-05 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054773+0000", 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028716700000000001 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055736+0000", 2023-01-15T14:33:17.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00096295500000000004 2023-01-15T14:33:17.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.355784+0000", 2023-01-15T14:33:17.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30004825899999998 2023-01-15T14:33:17.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556757+0000", 2023-01-15T14:33:17.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20097241199999999 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556830+0000", 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3633000000000005e-05 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556898+0000", 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7212999999999995e-05 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.887 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:17.888 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:17.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.888 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:360 3.a 3:52df5138:::smithi09025264-25:head [write 3046388~62874] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:17.888 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:33:17.888 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 277.71967046399999, 2023-01-15T14:33:17.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50519877000000002, 2023-01-15T14:33:17.888 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:17.888 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:17.888 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:17.888 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:17.889 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:17.889 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 360 2023-01-15T14:33:17.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.889 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:17.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:17.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:33:17.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:17.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:33:17.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:17.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:17.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051885+0000", 2023-01-15T14:33:17.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:17.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:17.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051935+0000", 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9826e-05 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051958+0000", 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3003e-05 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051965+0000", 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6129999999999998e-06 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054655+0000", 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0026896860000000002 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054869+0000", 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021348600000000001 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:17.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055198+0000", 2023-01-15T14:33:17.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032968199999999998 2023-01-15T14:33:17.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:17.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055605+0000", 2023-01-15T14:33:17.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040634200000000002 2023-01-15T14:33:17.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:17.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055622+0000", 2023-01-15T14:33:17.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7020999999999998e-05 2023-01-15T14:33:17.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:17.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055702+0000", 2023-01-15T14:33:17.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0835999999999997e-05 2023-01-15T14:33:17.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:17.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055754+0000", 2023-01-15T14:33:17.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1175999999999999e-05 2023-01-15T14:33:17.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.455996+0000", 2023-01-15T14:33:17.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40024198599999999 2023-01-15T14:33:17.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:17.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556952+0000", 2023-01-15T14:33:17.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10095644400000001 2023-01-15T14:33:17.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:17.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557019+0000", 2023-01-15T14:33:17.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6746000000000001e-05 2023-01-15T14:33:17.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:17.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:17.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:17.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557085+0000", 2023-01-15T14:33:17.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5885999999999999e-05 2023-01-15T14:33:18.000 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.000 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:18.000 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.000 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.000 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.000 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:361 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 3109262] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:18.000 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:33:18.000 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 277.71958055800002, 2023-01-15T14:33:18.000 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60469846000000005, 2023-01-15T14:33:18.000 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:18.000 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:18.001 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:18.001 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:18.001 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:18.001 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 361 2023-01-15T14:33:18.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.001 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:18.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:18.001 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:33:18.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.195 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.196 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.199 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:18.200 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:33:18.201 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.201 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:18.201 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051975+0000", 2023-01-15T14:33:18.201 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:18.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:18.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051980+0000", 2023-01-15T14:33:18.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2170000000000002e-06 2023-01-15T14:33:18.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:18.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051989+0000", 2023-01-15T14:33:18.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5299999999999996e-06 2023-01-15T14:33:18.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051995+0000", 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1369999999999999e-06 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055830+0000", 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0038348549999999999 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056101+0000", 2023-01-15T14:33:18.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027095800000000002 2023-01-15T14:33:18.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:18.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056560+0000", 2023-01-15T14:33:18.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045868599999999998 2023-01-15T14:33:18.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:18.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056954+0000", 2023-01-15T14:33:18.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039487600000000001 2023-01-15T14:33:18.205 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.205 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.205 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:18.208 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057004+0000", 2023-01-15T14:33:18.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9490000000000002e-05 2023-01-15T14:33:18.209 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.209 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:18.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057072+0000", 2023-01-15T14:33:18.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8502000000000002e-05 2023-01-15T14:33:18.209 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.209 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:18.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.158224+0000", 2023-01-15T14:33:18.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101151422 2023-01-15T14:33:18.210 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.210 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.210 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556201+0000", 2023-01-15T14:33:18.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39797714699999998 2023-01-15T14:33:18.210 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.210 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.210 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656421+0000", 2023-01-15T14:33:18.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100219549 2023-01-15T14:33:18.210 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.211 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.211 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:18.211 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656515+0000", 2023-01-15T14:33:18.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4048999999999993e-05 2023-01-15T14:33:18.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.211 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.211 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:18.211 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656674+0000", 2023-01-15T14:33:18.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000159696 2023-01-15T14:33:18.211 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.211 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:362 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 277.71955312799997, 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60517504499999997, 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:18.212 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:18.213 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 362 2023-01-15T14:33:18.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.213 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:18.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:18.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:33:18.213 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:18.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:33:18.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:18.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:33:18.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:18.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:18.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052007+0000", 2023-01-15T14:33:18.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.9759999999999997e-06 2023-01-15T14:33:18.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.215 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.215 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:18.215 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052016+0000", 2023-01-15T14:33:18.215 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4069999999999999e-06 2023-01-15T14:33:18.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.215 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.215 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:18.215 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052022+0000", 2023-01-15T14:33:18.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4670000000000001e-06 2023-01-15T14:33:18.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157159+0000", 2023-01-15T14:33:18.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.105136825 2023-01-15T14:33:18.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:18.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157325+0000", 2023-01-15T14:33:18.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000165262 2023-01-15T14:33:18.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656814+0000", 2023-01-15T14:33:18.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49948963200000002 2023-01-15T14:33:18.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:18.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656993+0000", 2023-01-15T14:33:18.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000178371 2023-01-15T14:33:18.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:18.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.657178+0000", 2023-01-15T14:33:18.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185825 2023-01-15T14:33:18.218 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.218 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:18.218 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.218 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:601 3.a 3:5ba385f1:::smithi09025264-44:head [setxattr _header (57) in=7b,truncate 3663893] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:33:18.218 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:33:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 257.82923049200002, 2023-01-15T14:33:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404742724, 2023-01-15T14:33:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 601 2023-01-15T14:33:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:33:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:33:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942323+0000", 2023-01-15T14:33:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942341+0000", 2023-01-15T14:33:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7659000000000002e-05 2023-01-15T14:33:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942365+0000", 2023-01-15T14:33:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4256999999999998e-05 2023-01-15T14:33:18.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:18.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042495+0000", 2023-01-15T14:33:18.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013043200000001 2023-01-15T14:33:18.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.144809+0000", 2023-01-15T14:33:18.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102314184 2023-01-15T14:33:18.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145098+0000", 2023-01-15T14:33:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000289036 2023-01-15T14:33:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145363+0000", 2023-01-15T14:33:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026449299999999998 2023-01-15T14:33:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:18.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145657+0000", 2023-01-15T14:33:18.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029433600000000002 2023-01-15T14:33:18.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:18.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145678+0000", 2023-01-15T14:33:18.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0939000000000001e-05 2023-01-15T14:33:18.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:18.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145853+0000", 2023-01-15T14:33:18.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017511800000000001 2023-01-15T14:33:18.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:18.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146557+0000", 2023-01-15T14:33:18.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00070337899999999996 2023-01-15T14:33:18.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146845+0000", 2023-01-15T14:33:18.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028790599999999999 2023-01-15T14:33:18.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346819+0000", 2023-01-15T14:33:18.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199974661 2023-01-15T14:33:18.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:18.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346919+0000", 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9234999999999994e-05 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.347069+0000", 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150128 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.228 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1153 3.a 3:5919d83e:::smithi09025264-13:head [write 1566383~595717] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 211.98100543800001, 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.38805106299999997, 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1153 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:18.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:18.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:33:18.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:18.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:33:18.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:18.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790531+0000", 2023-01-15T14:33:18.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999795 2023-01-15T14:33:18.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:18.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791189+0000", 2023-01-15T14:33:18.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065897000000000004 2023-01-15T14:33:18.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:18.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791215+0000", 2023-01-15T14:33:18.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5700000000000001e-05 2023-01-15T14:33:18.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:18.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791232+0000", 2023-01-15T14:33:18.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6546e-05 2023-01-15T14:33:18.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791392+0000", 2023-01-15T14:33:18.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016051699999999999 2023-01-15T14:33:18.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:18.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791408+0000", 2023-01-15T14:33:18.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5566999999999999e-05 2023-01-15T14:33:18.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.974825+0000", 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18341675599999999 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975000+0000", 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017548299999999999 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975255+0000", 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254658 2023-01-15T14:33:18.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:18.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975914+0000", 2023-01-15T14:33:18.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065881800000000003 2023-01-15T14:33:18.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:18.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975931+0000", 2023-01-15T14:33:18.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7523000000000001e-05 2023-01-15T14:33:18.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:18.608 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976267+0000", 2023-01-15T14:33:18.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033545299999999999 2023-01-15T14:33:18.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.609 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.609 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:18.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978320+0000", 2023-01-15T14:33:18.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020537110000000002 2023-01-15T14:33:18.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.610 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978647+0000", 2023-01-15T14:33:18.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032660700000000003 2023-01-15T14:33:18.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178434+0000", 2023-01-15T14:33:18.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19978705699999999 2023-01-15T14:33:18.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:18.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178527+0000", 2023-01-15T14:33:18.612 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3523999999999994e-05 2023-01-15T14:33:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178602+0000", 2023-01-15T14:33:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4622000000000005e-05 2023-01-15T14:33:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1154 3.a 3:5919d83e:::smithi09025264-13:head [write 2888822~102824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:33:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:33:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 211.97566411099999, 2023-01-15T14:33:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48318778800000001, 2023-01-15T14:33:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1154 2023-01-15T14:33:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:33:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:33:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795890+0000", 2023-01-15T14:33:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795967+0000", 2023-01-15T14:33:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6304999999999995e-05 2023-01-15T14:33:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796000+0000", 2023-01-15T14:33:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3593000000000002e-05 2023-01-15T14:33:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796019+0000", 2023-01-15T14:33:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8955999999999999e-05 2023-01-15T14:33:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796189+0000", 2023-01-15T14:33:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017032700000000001 2023-01-15T14:33:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796208+0000", 2023-01-15T14:33:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8389e-05 2023-01-15T14:33:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976335+0000", 2023-01-15T14:33:18.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18012763100000001 2023-01-15T14:33:18.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:18.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976487+0000", 2023-01-15T14:33:18.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015173000000000001 2023-01-15T14:33:18.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:18.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976763+0000", 2023-01-15T14:33:18.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027597400000000001 2023-01-15T14:33:18.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:18.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977085+0000", 2023-01-15T14:33:18.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032233400000000002 2023-01-15T14:33:18.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:18.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977103+0000", 2023-01-15T14:33:18.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7727000000000002e-05 2023-01-15T14:33:18.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:18.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977238+0000", 2023-01-15T14:33:18.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013525399999999999 2023-01-15T14:33:18.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:18.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978347+0000", 2023-01-15T14:33:18.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011084350000000001 2023-01-15T14:33:18.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.078223+0000", 2023-01-15T14:33:18.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099875830999999998 2023-01-15T14:33:18.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.278897+0000", 2023-01-15T14:33:18.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20067421599999999 2023-01-15T14:33:18.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:18.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279021+0000", 2023-01-15T14:33:18.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124354 2023-01-15T14:33:18.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:18.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279080+0000", 2023-01-15T14:33:18.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8811999999999999e-05 2023-01-15T14:33:18.814 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.814 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1155 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 2991646] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:33:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:33:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 211.975491092, 2023-01-15T14:33:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48334445999999998, 2023-01-15T14:33:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1155 2023-01-15T14:33:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:33:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:33:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796064+0000", 2023-01-15T14:33:18.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:18.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:18.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796074+0000", 2023-01-15T14:33:18.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:33:18.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:18.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796096+0000", 2023-01-15T14:33:18.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2422999999999999e-05 2023-01-15T14:33:18.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:18.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796109+0000", 2023-01-15T14:33:18.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:33:18.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796274+0000", 2023-01-15T14:33:18.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000164976 2023-01-15T14:33:18.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:18.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796283+0000", 2023-01-15T14:33:18.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9539999999999993e-06 2023-01-15T14:33:18.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977269+0000", 2023-01-15T14:33:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18098651700000001 2023-01-15T14:33:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977438+0000", 2023-01-15T14:33:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168913 2023-01-15T14:33:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977865+0000", 2023-01-15T14:33:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042707500000000001 2023-01-15T14:33:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:18.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978241+0000", 2023-01-15T14:33:18.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037593000000000001 2023-01-15T14:33:18.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:18.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978253+0000", 2023-01-15T14:33:18.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1817000000000001e-05 2023-01-15T14:33:18.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:18.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978288+0000", 2023-01-15T14:33:18.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4586999999999999e-05 2023-01-15T14:33:18.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:18.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978370+0000", 2023-01-15T14:33:18.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1712999999999995e-05 2023-01-15T14:33:18.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279124+0000", 2023-01-15T14:33:18.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30075418799999998 2023-01-15T14:33:18.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279208+0000", 2023-01-15T14:33:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4500999999999996e-05 2023-01-15T14:33:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279294+0000", 2023-01-15T14:33:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5680999999999995e-05 2023-01-15T14:33:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279410+0000", 2023-01-15T14:33:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115878 2023-01-15T14:33:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1156 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:33:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:33:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 211.975379567, 2023-01-15T14:33:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.58375825000000003, 2023-01-15T14:33:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1156 2023-01-15T14:33:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:33:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:33:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796176+0000", 2023-01-15T14:33:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796182+0000", 2023-01-15T14:33:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6799999999999998e-06 2023-01-15T14:33:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796200+0000", 2023-01-15T14:33:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8136999999999999e-05 2023-01-15T14:33:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796217+0000", 2023-01-15T14:33:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7036000000000001e-05 2023-01-15T14:33:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796337+0000", 2023-01-15T14:33:18.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120674 2023-01-15T14:33:18.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:18.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796345+0000", 2023-01-15T14:33:18.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4540000000000001e-06 2023-01-15T14:33:18.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978423+0000", 2023-01-15T14:33:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18207789599999999 2023-01-15T14:33:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978577+0000", 2023-01-15T14:33:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000153805 2023-01-15T14:33:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379596+0000", 2023-01-15T14:33:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.401019348 2023-01-15T14:33:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379754+0000", 2023-01-15T14:33:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015840400000000001 2023-01-15T14:33:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379935+0000", 2023-01-15T14:33:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018081299999999999 2023-01-15T14:33:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1160 3.a 3:52df5138:::smithi09025264-25:head [write 2632598~375855] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:33:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:33:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 211.01657298000001, 2023-01-15T14:33:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.398337832, 2023-01-15T14:33:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1160 2023-01-15T14:33:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:33:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:33:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754981+0000", 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755360+0000", 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037899899999999998 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755392+0000", 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1977000000000002e-05 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755407+0000", 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4446e-05 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949294+0000", 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.193887483 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949478+0000", 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018400900000000001 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949904+0000", 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042552200000000001 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950531+0000", 2023-01-15T14:33:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062732399999999996 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950552+0000", 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1229999999999998e-05 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951094+0000", 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054169000000000005 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951256+0000", 2023-01-15T14:33:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016176299999999999 2023-01-15T14:33:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052665+0000", 2023-01-15T14:33:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101409257 2023-01-15T14:33:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153132+0000", 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100466783 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153223+0000", 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.1415000000000005e-05 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153321+0000", 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8181000000000001e-05 2023-01-15T14:33:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1161 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (57) in=7b,truncate 3008453] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:33:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:33:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 211.01612939200001, 2023-01-15T14:33:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49840109900000001, 2023-01-15T14:33:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1161 2023-01-15T14:33:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:33:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755426+0000", 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755433+0000", 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.793e-06 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755448+0000", 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5047e-05 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755455+0000", 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3590000000000002e-06 2023-01-15T14:33:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951466+0000", 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.196011038 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951611+0000", 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000145184 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:18.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951921+0000", 2023-01-15T14:33:18.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030989100000000002 2023-01-15T14:33:18.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:18.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952187+0000", 2023-01-15T14:33:18.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026623100000000002 2023-01-15T14:33:18.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.857 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:18.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952207+0000", 2023-01-15T14:33:18.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9510000000000001e-05 2023-01-15T14:33:18.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:18.858 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952271+0000", 2023-01-15T14:33:18.858 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4511999999999995e-05 2023-01-15T14:33:18.858 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.858 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:18.858 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052716+0000", 2023-01-15T14:33:18.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10044497300000001 2023-01-15T14:33:18.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253531+0000", 2023-01-15T14:33:18.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20081490699999999 2023-01-15T14:33:18.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.860 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:18.860 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253635+0000", 2023-01-15T14:33:18.860 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000103838 2023-01-15T14:33:18.860 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.861 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:18.861 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253727+0000", 2023-01-15T14:33:18.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2298000000000003e-05 2023-01-15T14:33:18.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.861 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.861 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:18.862 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253828+0000", 2023-01-15T14:33:18.862 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000100655 2023-01-15T14:33:18.862 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.862 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:18.862 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:18.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.863 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1162 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:33:18.863 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:33:18.863 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 211.01608999699999, 2023-01-15T14:33:18.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.498775839, 2023-01-15T14:33:18.863 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:18.863 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:18.863 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:18.863 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:18.863 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:18.864 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1162 2023-01-15T14:33:18.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.864 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:18.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.864 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:18.864 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:33:18.864 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.864 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:18.864 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:33:18.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:18.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:18.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:18.865 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:18.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:33:18.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:19.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755470+0000", 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0570000000000003e-06 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755484+0000", 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4229999999999999e-05 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755492+0000", 2023-01-15T14:33:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6070000000000002e-06 2023-01-15T14:33:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.152842+0000", 2023-01-15T14:33:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39734988799999998 2023-01-15T14:33:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153014+0000", 2023-01-15T14:33:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017236000000000001 2023-01-15T14:33:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253921+0000", 2023-01-15T14:33:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100907497 2023-01-15T14:33:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254060+0000", 2023-01-15T14:33:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013815400000000001 2023-01-15T14:33:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254242+0000", 2023-01-15T14:33:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000182726 2023-01-15T14:33:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:19.022 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:19.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:33:19.023 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:33:19.026 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:33:19.028 INFO:tasks.rados.rados.0.smithi090.stdout:1225: done (0 left) 2023-01-15T14:33:19.028 INFO:tasks.rados.rados.0.smithi090.stdout:1226: write oid 14 current snap is 126 2023-01-15T14:33:19.028 INFO:tasks.rados.rados.0.smithi090.stdout:1226: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2023-01-15T14:33:19.028 INFO:tasks.rados.rados.0.smithi090.stdout:1226: writing smithi09025264-14 from 763150 to 1490027 tid 1 2023-01-15T14:33:19.029 INFO:tasks.rados.rados.0.smithi090.stdout:1226: writing smithi09025264-14 from 1982897 to 2597715 tid 2 2023-01-15T14:33:19.029 INFO:tasks.rados.rados.0.smithi090.stdout:1226: writing smithi09025264-14 from 3024582 to 3499576 tid 3 2023-01-15T14:33:19.029 INFO:tasks.rados.rados.0.smithi090.stdout:1227: copy_from oid 6 from oid 24 current snap is 126 2023-01-15T14:33:19.219 INFO:tasks.rados.rados.0.smithi090.stdout:1228: read oid 12 snap -1 2023-01-15T14:33:19.219 INFO:tasks.rados.rados.0.smithi090.stdout:1228: expect (ObjNum 439 snap 121 seq_num 439) 2023-01-15T14:33:19.219 INFO:tasks.rados.rados.0.smithi090.stdout:1229: read oid 39 snap -1 2023-01-15T14:33:19.219 INFO:tasks.rados.rados.0.smithi090.stdout:1229: expect deleted 2023-01-15T14:33:19.219 INFO:tasks.rados.rados.0.smithi090.stdout:1230: write oid 20 current snap is 126 2023-01-15T14:33:19.220 INFO:tasks.rados.rados.0.smithi090.stdout:1230: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2023-01-15T14:33:19.220 INFO:tasks.rados.rados.0.smithi090.stdout:1230: writing smithi09025264-20 from 779957 to 1182083 tid 1 2023-01-15T14:33:19.220 INFO:tasks.rados.rados.0.smithi090.stdout:1230: writing smithi09025264-20 from 1925026 to 2683502 tid 2 2023-01-15T14:33:19.220 INFO:tasks.rados.rados.0.smithi090.stdout:1230: writing smithi09025264-20 from 3335652 to 3579957 tid 3 2023-01-15T14:33:19.220 INFO:tasks.rados.rados.0.smithi090.stdout:1231: snap_remove snap 122 2023-01-15T14:33:19.220 INFO:tasks.rados.rados.0.smithi090.stdout:1231: done (5 left) 2023-01-15T14:33:19.220 INFO:tasks.rados.rados.0.smithi090.stdout:1232: delete oid 18 current snap is 126 2023-01-15T14:33:19.220 INFO:tasks.rados.rados.0.smithi090.stdout:1228: expect (ObjNum 439 snap 121 seq_num 439) 2023-01-15T14:33:19.221 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:19.021+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:19.222 INFO:tasks.rados.rados.0.smithi090.stdout:1230: finishing write tid 1 to smithi09025264-20 2023-01-15T14:33:19.222 INFO:tasks.rados.rados.0.smithi090.stdout:1230: finishing write tid 2 to smithi09025264-20 2023-01-15T14:33:19.222 INFO:tasks.rados.rados.0.smithi090.stdout:1230: finishing write tid 3 to smithi09025264-20 2023-01-15T14:33:19.222 INFO:tasks.rados.rados.0.smithi090.stdout:1230: finishing write tid 4 to smithi09025264-20 2023-01-15T14:33:19.222 INFO:tasks.rados.rados.0.smithi090.stdout:1230: finishing write tid 5 to smithi09025264-20 2023-01-15T14:33:19.222 INFO:tasks.rados.rados.0.smithi090.stdout:1230: oid 20 updating version 0 to 250 2023-01-15T14:33:19.222 INFO:tasks.rados.rados.0.smithi090.stdout:1230: oid 20 updating version 250 to 251 2023-01-15T14:33:19.222 INFO:tasks.rados.rados.0.smithi090.stdout:1230: oid 20 updating version 251 to 252 2023-01-15T14:33:19.222 INFO:tasks.rados.rados.0.smithi090.stdout:1230: oid 20 version 252 is already newer than 249 2023-01-15T14:33:19.223 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 252 (ObjNum 451 snap 126 seq_num 451) dirty exists 2023-01-15T14:33:19.223 INFO:tasks.rados.rados.0.smithi090.stdout:1230: left oid 20 (ObjNum 451 snap 126 seq_num 451) 2023-01-15T14:33:19.223 INFO:tasks.rados.rados.0.smithi090.stdout:1227: finishing copy_from to smithi09025264-6 2023-01-15T14:33:19.224 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 302 (ObjNum 412 snap 112 seq_num 412) dirty exists 2023-01-15T14:33:19.224 INFO:tasks.rados.rados.0.smithi090.stdout:1227: finishing copy_from racing read to smithi09025264-6 2023-01-15T14:33:19.224 INFO:tasks.rados.rados.0.smithi090.stdout:1226: finishing write tid 1 to smithi09025264-14 2023-01-15T14:33:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1226: finishing write tid 2 to smithi09025264-14 2023-01-15T14:33:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1226: finishing write tid 3 to smithi09025264-14 2023-01-15T14:33:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1226: finishing write tid 4 to smithi09025264-14 2023-01-15T14:33:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1226: finishing write tid 5 to smithi09025264-14 2023-01-15T14:33:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1226: oid 14 updating version 0 to 287 2023-01-15T14:33:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1226: oid 14 version 287 is already newer than 286 2023-01-15T14:33:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1226: oid 14 updating version 287 to 289 2023-01-15T14:33:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1226: oid 14 version 289 is already newer than 288 2023-01-15T14:33:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 289 (ObjNum 450 snap 126 seq_num 450) dirty exists 2023-01-15T14:33:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1226: left oid 14 (ObjNum 450 snap 126 seq_num 450) 2023-01-15T14:33:19.227 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:19.220+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:19.323 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:19.321+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:19.426 INFO:tasks.rados.rados.0.smithi090.stdout:1226: done (5 left) 2023-01-15T14:33:19.426 INFO:tasks.rados.rados.0.smithi090.stdout:1227: done (4 left) 2023-01-15T14:33:19.426 INFO:tasks.rados.rados.0.smithi090.stdout:1228: done (3 left) 2023-01-15T14:33:19.426 INFO:tasks.rados.rados.0.smithi090.stdout:1229: done (2 left) 2023-01-15T14:33:19.426 INFO:tasks.rados.rados.0.smithi090.stdout:1230: done (1 left) 2023-01-15T14:33:19.440 INFO:tasks.rados.rados.0.smithi090.stdout:1232: done (0 left) 2023-01-15T14:33:19.440 INFO:tasks.rados.rados.0.smithi090.stdout:1233: rollback oid 15 current snap is 126 2023-01-15T14:33:19.441 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 15 to 111 2023-01-15T14:33:19.441 INFO:tasks.rados.rados.0.smithi090.stdout:1234: snap_create 2023-01-15T14:33:19.441 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:19.442 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:19.442 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:33:19.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.443 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3231 3.3f 3:fe04f54c:::smithi09025264-18:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e310)", 2023-01-15T14:33:19.443 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:19.018785+0000", 2023-01-15T14:33:19.443 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.31772916800000001, 2023-01-15T14:33:19.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31777103899999998, 2023-01-15T14:33:19.443 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:19.443 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:33:19.443 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:19.443 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:19.443 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:19.443 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3231 2023-01-15T14:33:19.443 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.444 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:19.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.444 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:19.444 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.018785+0000", 2023-01-15T14:33:19.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:19.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.614 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:19.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.018785+0000", 2023-01-15T14:33:19.614 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:19.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.615 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:19.615 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.018782+0000", 2023-01-15T14:33:19.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:19.616 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.616 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.616 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:19.616 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.018800+0000", 2023-01-15T14:33:19.616 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8726e-05 2023-01-15T14:33:19.616 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:19.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.018854+0000", 2023-01-15T14:33:19.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.3853000000000001e-05 2023-01-15T14:33:19.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:19.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.018878+0000", 2023-01-15T14:33:19.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3472000000000002e-05 2023-01-15T14:33:19.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:19.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.018971+0000", 2023-01-15T14:33:19.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3346999999999999e-05 2023-01-15T14:33:19.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:33:19.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.018981+0000", 2023-01-15T14:33:19.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4120000000000007e-06 2023-01-15T14:33:19.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:19.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.122820+0000", 2023-01-15T14:33:19.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103839138 2023-01-15T14:33:19.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:19.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.122984+0000", 2023-01-15T14:33:19.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000164117 2023-01-15T14:33:19.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:19.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.123425+0000", 2023-01-15T14:33:19.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044121500000000001 2023-01-15T14:33:19.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:19.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.125637+0000", 2023-01-15T14:33:19.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022115870000000001 2023-01-15T14:33:19.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:19.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.125658+0000", 2023-01-15T14:33:19.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1426000000000001e-05 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.125723+0000", 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4743e-05 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.125771+0000", 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:19.624 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.128347+0000", 2023-01-15T14:33:19.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0025758759999999999 2023-01-15T14:33:19.624 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:19.624 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:19.624 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:19.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.624 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:3231 3.3f e310/235)", 2023-01-15T14:33:19.624 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:19.228116+0000", 2023-01-15T14:33:19.625 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.108397724, 2023-01-15T14:33:19.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.108622066, 2023-01-15T14:33:19.625 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:19.625 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:19.625 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:19.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:19.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.228116+0000", 2023-01-15T14:33:19.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:19.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:19.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.228116+0000", 2023-01-15T14:33:19.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:19.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:19.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.228114+0000", 2023-01-15T14:33:19.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:19.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:19.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.228141+0000", 2023-01-15T14:33:19.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6925000000000001e-05 2023-01-15T14:33:19.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:19.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.228175+0000", 2023-01-15T14:33:19.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3708000000000003e-05 2023-01-15T14:33:19.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:19.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.228192+0000", 2023-01-15T14:33:19.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7628000000000001e-05 2023-01-15T14:33:19.628 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:19.628 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:19.628 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:19.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.7.0:683 3.3f e310/235)", 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:19.330438+0000", 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.0060764590000000002, 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0063625419999999997, 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.330438+0000", 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:19.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:19.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.330438+0000", 2023-01-15T14:33:19.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:19.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:19.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.330437+0000", 2023-01-15T14:33:19.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:19.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:19.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.330443+0000", 2023-01-15T14:33:19.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4239999999999998e-06 2023-01-15T14:33:19.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:19.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.330454+0000", 2023-01-15T14:33:19.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1161e-05 2023-01-15T14:33:19.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:19.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:19.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:19.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.330464+0000", 2023-01-15T14:33:19.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5060000000000001e-06 2023-01-15T14:33:19.632 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:19.632 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:19.632 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:19.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:20.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:20.037 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.7.0:683 3.3f e310/235)", 2023-01-15T14:33:20.038 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:19.332779+0000", 2023-01-15T14:33:20.038 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.0037348619999999998, 2023-01-15T14:33:20.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0040809929999999998, 2023-01-15T14:33:20.038 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:20.038 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:20.039 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:20.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:20.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:20.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.332779+0000", 2023-01-15T14:33:20.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:20.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:20.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:20.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:20.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.332779+0000", 2023-01-15T14:33:20.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:20.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:20.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:20.041 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:20.041 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.332779+0000", 2023-01-15T14:33:20.041 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:20.041 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:20.041 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:20.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:20.042 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.332783+0000", 2023-01-15T14:33:20.042 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0740000000000003e-06 2023-01-15T14:33:20.042 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:20.042 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:20.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:20.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.332800+0000", 2023-01-15T14:33:20.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7303e-05 2023-01-15T14:33:20.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:20.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:20.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:20.044 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:19.332807+0000", 2023-01-15T14:33:20.044 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4970000000000004e-06 2023-01-15T14:33:20.044 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:20.044 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:20.044 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:20.045 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:20.045 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2023-01-15T14:33:20.045 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 4 2023-01-15T14:33:20.045 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:20.047 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:20.047 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:20.047 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:20.047 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:20.047 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:20.048 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:33:20.048 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:19.424+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:20.049 INFO:tasks.rados.rados.0.smithi090.stdout:1233: finishing rollback tid 0 to smithi09025264-15 2023-01-15T14:33:20.049 INFO:tasks.rados.rados.0.smithi090.stdout:1233: finishing rollback tid 1 to smithi09025264-15 2023-01-15T14:33:20.050 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:20.050 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:20.050 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:20.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:33:20.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:33:20.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:33:20.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:33:20.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:33:20.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:33:20.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:33:20.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:33:20.052 INFO:tasks.rados.rados.0.smithi090.stdout:1233: finishing rollback tid 2 to smithi09025264-15 2023-01-15T14:33:20.053 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 155 (ObjNum 361 snap 96 seq_num 361) dirty exists 2023-01-15T14:33:20.053 INFO:tasks.rados.rados.0.smithi090.stdout:1233: done (0 left) 2023-01-15T14:33:20.053 INFO:tasks.rados.rados.0.smithi090.stdout:1234: done (0 left) 2023-01-15T14:33:20.053 INFO:tasks.rados.rados.0.smithi090.stdout:1235: read oid 32 snap 125 2023-01-15T14:33:20.053 INFO:tasks.rados.rados.0.smithi090.stdout:1235: expect (ObjNum 404 snap 108 seq_num 404) 2023-01-15T14:33:20.053 INFO:tasks.rados.rados.0.smithi090.stdout:1236: write (excl) oid 16 current snap is 127 2023-01-15T14:33:20.053 INFO:tasks.rados.rados.0.smithi090.stdout:1236: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2023-01-15T14:33:20.053 INFO:tasks.rados.rados.0.smithi090.stdout:1236: writing smithi09025264-16 from 413571 to 966195 tid 1 2023-01-15T14:33:20.053 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:33:20.054 INFO:tasks.rados.rados.0.smithi090.stdout:1236: writing smithi09025264-16 from 1641990 to 2404135 tid 2 2023-01-15T14:33:20.054 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:20.045+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:20.056 INFO:tasks.rados.rados.0.smithi090.stdout:1236: writing smithi09025264-16 from 2990498 to 3415661 tid 3 2023-01-15T14:33:20.056 INFO:tasks.rados.rados.0.smithi090.stdout:1237: read oid 21 snap 119 2023-01-15T14:33:20.056 INFO:tasks.rados.rados.0.smithi090.stdout:1237: expect (ObjNum 339 snap 89 seq_num 339) 2023-01-15T14:33:20.056 INFO:tasks.rados.rados.0.smithi090.stdout:1238: write (excl) oid 5 current snap is 127 2023-01-15T14:33:20.056 INFO:tasks.rados.rados.0.smithi090.stdout:1238: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2023-01-15T14:33:20.060 INFO:tasks.rados.rados.0.smithi090.stdout:1238: writing smithi09025264-5 from 430378 to 1058251 tid 1 2023-01-15T14:33:20.068 INFO:tasks.rados.rados.0.smithi090.stdout:1238: writing smithi09025264-5 from 1700472 to 2322628 tid 2 2023-01-15T14:33:20.075 INFO:tasks.rados.rados.0.smithi090.stdout:1238: writing smithi09025264-5 from 3017921 to 3630378 tid 3 2023-01-15T14:33:20.247 INFO:tasks.rados.rados.0.smithi090.stdout:1239: read oid 29 snap -1 2023-01-15T14:33:20.247 INFO:tasks.rados.rados.0.smithi090.stdout:1239: expect (ObjNum 402 snap 108 seq_num 402) 2023-01-15T14:33:20.247 INFO:tasks.rados.rados.0.smithi090.stdout:1240: read oid 37 snap -1 2023-01-15T14:33:20.247 INFO:tasks.rados.rados.0.smithi090.stdout:1240: expect deleted 2023-01-15T14:33:20.247 INFO:tasks.rados.rados.0.smithi090.stdout:1241: snap_create 2023-01-15T14:33:20.248 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:33:20.250 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:20.250 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:20.251 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:33:20.251 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:20.248+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:20.350 INFO:tasks.rados.rados.0.smithi090.stdout:1238: finishing write tid 1 to smithi09025264-5 2023-01-15T14:33:20.469 INFO:tasks.rados.rados.0.smithi090.stdout:1239: expect (ObjNum 402 snap 108 seq_num 402) 2023-01-15T14:33:20.470 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:20.348+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:20.470 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:20.471 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:20.471 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:20.468+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:20.471 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:33:20.539 INFO:tasks.rados.rados.0.smithi090.stdout:1239: done (5 left) 2023-01-15T14:33:20.683 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:33:20.683 INFO:tasks.rados.rados.0.smithi090.stdout:1237: expect (ObjNum 339 snap 89 seq_num 339) 2023-01-15T14:33:20.685 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:20.685 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:20.685 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:33:20.686 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:20.681+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:20.739 INFO:tasks.rados.rados.0.smithi090.stdout:1235: expect (ObjNum 404 snap 108 seq_num 404) 2023-01-15T14:33:20.897 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:20.898 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:20.898 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:33:20.899 INFO:tasks.rados.rados.0.smithi090.stdout:1236: finishing write tid 1 to smithi09025264-16 2023-01-15T14:33:20.900 INFO:tasks.rados.rados.0.smithi090.stdout:1238: finishing write tid 2 to smithi09025264-5 2023-01-15T14:33:20.900 INFO:tasks.rados.rados.0.smithi090.stdout:1238: finishing write tid 3 to smithi09025264-5 2023-01-15T14:33:20.900 INFO:tasks.rados.rados.0.smithi090.stdout:1238: finishing write tid 4 to smithi09025264-5 2023-01-15T14:33:20.900 INFO:tasks.rados.rados.0.smithi090.stdout:1238: finishing write tid 5 to smithi09025264-5 2023-01-15T14:33:20.900 INFO:tasks.rados.rados.0.smithi090.stdout:1238: oid 5 updating version 0 to 235 2023-01-15T14:33:20.900 INFO:tasks.rados.rados.0.smithi090.stdout:1238: oid 5 version 235 is already newer than 234 2023-01-15T14:33:20.901 INFO:tasks.rados.rados.0.smithi090.stdout:1238: oid 5 version 235 is already newer than 233 2023-01-15T14:33:20.901 INFO:tasks.rados.rados.0.smithi090.stdout:1238: oid 5 updating version 235 to 236 2023-01-15T14:33:20.901 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 236 (ObjNum 454 snap 127 seq_num 454) dirty exists 2023-01-15T14:33:20.901 INFO:tasks.rados.rados.0.smithi090.stdout:1238: left oid 5 (ObjNum 454 snap 127 seq_num 454) 2023-01-15T14:33:20.901 INFO:tasks.rados.rados.0.smithi090.stdout:1236: finishing write tid 2 to smithi09025264-16 2023-01-15T14:33:20.901 INFO:tasks.rados.rados.0.smithi090.stdout:1236: finishing write tid 3 to smithi09025264-16 2023-01-15T14:33:20.901 INFO:tasks.rados.rados.0.smithi090.stdout:1236: finishing write tid 4 to smithi09025264-16 2023-01-15T14:33:20.901 INFO:tasks.rados.rados.0.smithi090.stdout:1236: finishing write tid 5 to smithi09025264-16 2023-01-15T14:33:20.901 INFO:tasks.rados.rados.0.smithi090.stdout:1236: oid 16 updating version 0 to 168 2023-01-15T14:33:20.901 INFO:tasks.rados.rados.0.smithi090.stdout:1236: oid 16 version 168 is already newer than 165 2023-01-15T14:33:20.902 INFO:tasks.rados.rados.0.smithi090.stdout:1236: oid 16 version 168 is already newer than 167 2023-01-15T14:33:20.902 INFO:tasks.rados.rados.0.smithi090.stdout:1236: oid 16 version 168 is already newer than 166 2023-01-15T14:33:20.902 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 168 (ObjNum 453 snap 127 seq_num 453) dirty exists 2023-01-15T14:33:20.902 INFO:tasks.rados.rados.0.smithi090.stdout:1236: left oid 16 (ObjNum 453 snap 127 seq_num 453) 2023-01-15T14:33:20.902 INFO:tasks.rados.rados.0.smithi090.stdout:1235: done (4 left) 2023-01-15T14:33:20.902 INFO:tasks.rados.rados.0.smithi090.stdout:1236: done (3 left) 2023-01-15T14:33:20.902 INFO:tasks.rados.rados.0.smithi090.stdout:1237: done (2 left) 2023-01-15T14:33:20.902 INFO:tasks.rados.rados.0.smithi090.stdout:1238: done (1 left) 2023-01-15T14:33:20.902 INFO:tasks.rados.rados.0.smithi090.stdout:1240: done (0 left) 2023-01-15T14:33:20.903 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:20.896+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:20.997 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:33:21.091 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:33:21.091 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-01-15T14:33:21.091 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-01-15T14:33:21.092 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:20.996+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:21.093 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:21.093 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:21.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:33:21.094 INFO:tasks.rados.rados.0.smithi090.stdout:1241: done (0 left) 2023-01-15T14:33:21.095 INFO:tasks.rados.rados.0.smithi090.stdout:1242: read oid 2 snap -1 2023-01-15T14:33:21.095 INFO:tasks.rados.rados.0.smithi090.stdout:1242: expect (ObjNum 427 snap 116 seq_num 427) 2023-01-15T14:33:21.095 INFO:tasks.rados.rados.0.smithi090.stdout:1243: rollback oid 33 current snap is 128 2023-01-15T14:33:21.095 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 33 to 116 2023-01-15T14:33:21.095 INFO:tasks.rados.rados.0.smithi090.stdout:1244: read oid 10 snap -1 2023-01-15T14:33:21.095 INFO:tasks.rados.rados.0.smithi090.stdout:1244: expect (ObjNum 448 snap 124 seq_num 448) 2023-01-15T14:33:21.095 INFO:tasks.rados.rados.0.smithi090.stdout:1245: snap_create 2023-01-15T14:33:21.095 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:21.098 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:21.093+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:21.199 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:21.197+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:21.292 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:21.293 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:21.293 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:33:21.294 INFO:tasks.rados.rados.0.smithi090.stdout:1243: finishing rollback tid 0 to smithi09025264-33 2023-01-15T14:33:21.294 INFO:tasks.rados.rados.0.smithi090.stdout:1243: finishing rollback tid 1 to smithi09025264-33 2023-01-15T14:33:21.294 INFO:tasks.rados.rados.0.smithi090.stdout:1243: finishing rollback tid 2 to smithi09025264-33 2023-01-15T14:33:21.295 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 200 (ObjNum 378 snap 102 seq_num 378) dirty exists 2023-01-15T14:33:21.295 INFO:tasks.rados.rados.0.smithi090.stdout:1243: done (2 left) 2023-01-15T14:33:21.295 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:21.335 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:21.333+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:21.485 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:33:21.485 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:21.485 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 312, 2023-01-15T14:33:21.486 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "1c77fb2b-d1a7-4884-a24e-3935e16d0b6b", 2023-01-15T14:33:21.486 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2023-01-15T14:27:57.735717+0000", 2023-01-15T14:33:21.486 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2023-01-15T14:33:21.022739+0000", 2023-01-15T14:33:21.486 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2023-01-15T14:28:04.837522+0000", 2023-01-15T14:33:21.486 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2023-01-15T14:30:49.906949+0000", 2023-01-15T14:33:21.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-01-15T14:33:21.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5804032, 2023-01-15T14:33:21.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2023-01-15T14:33:21.487 INFO:teuthology.orchestra.run.smithi090.stdout: "nodeep-scrub", 2023-01-15T14:33:21.487 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2023-01-15T14:33:21.488 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2023-01-15T14:33:21.488 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2023-01-15T14:33:21.488 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2023-01-15T14:33:21.488 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:21.488 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2023-01-15T14:33:21.489 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2023-01-15T14:33:21.489 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-01-15T14:33:21.489 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2023-01-15T14:33:21.489 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2023-01-15T14:33:21.490 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 3, 2023-01-15T14:33:21.490 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2023-01-15T14:33:21.490 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2023-01-15T14:33:21.493 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2023-01-15T14:33:21.493 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "reef", 2023-01-15T14:33:21.493 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2023-01-15T14:33:21.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2023-01-15T14:33:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:04.899282+0000", 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:33:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:33:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2023-01-15T14:33:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2023-01-15T14:33:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2023-01-15T14:33:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2023-01-15T14:33:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2023-01-15T14:33:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:33:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:33:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:33:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:33:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:33:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:33:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:33:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "16", 2023-01-15T14:33:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:33:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:33:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:33:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:33:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:33:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:33:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:33:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:33:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:33:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:33:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:33:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:33:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:33:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:33:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:33:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:33:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:33:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:33:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:33:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:33:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:33:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:33:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:33:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:33:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:33:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:33:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:33:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:33:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:33:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:33:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:33:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:33:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:33:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:33:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:33:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:33:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:33:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2023-01-15T14:33:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2023-01-15T14:33:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2023-01-15T14:33:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:33:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2023-01-15T14:33:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2023-01-15T14:33:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2023-01-15T14:33:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:11.115002+0000", 2023-01-15T14:33:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:33:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:33:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:33:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:33:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:33:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:33:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:33:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:33:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:33:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:33:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:33:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:33:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2023-01-15T14:33:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2023-01-15T14:33:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2023-01-15T14:33:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2023-01-15T14:33:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2023-01-15T14:33:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:33:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:33:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:33:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:33:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:33:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "19", 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:33:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:33:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:33:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:33:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:33:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:33:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:33:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:33:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:33:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:33:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:33:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2023-01-15T14:33:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:33:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "unique_pool_0", 2023-01-15T14:33:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:22.434903+0000", 2023-01-15T14:33:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 16385, 2023-01-15T14:33:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,pool_snaps", 2023-01-15T14:33:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:33:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:33:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:33:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:33:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:33:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:33:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:33:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:33:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:33:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:33:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 86, 2023-01-15T14:33:21.884 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 86, 2023-01-15T14:33:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 86, 2023-01-15T14:33:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 86, 2023-01-15T14:33:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 86, 2023-01-15T14:33:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:33:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:33:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:33:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:33:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:33:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:33:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:33:21.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "312", 2023-01-15T14:33:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:33:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:33:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "310", 2023-01-15T14:33:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:33:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "pool", 2023-01-15T14:33:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 247, 2023-01-15T14:33:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 312, 2023-01-15T14:33:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [ 2023-01-15T14:33:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.886 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 219, 2023-01-15T14:33:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:43.999592+0000", 2023-01-15T14:33:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap112" 2023-01-15T14:33:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 228, 2023-01-15T14:33:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:32:54.385736+0000", 2023-01-15T14:33:21.887 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap117" 2023-01-15T14:33:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.888 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 236, 2023-01-15T14:33:21.889 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:33:03.448556+0000", 2023-01-15T14:33:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap120" 2023-01-15T14:33:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 237, 2023-01-15T14:33:21.890 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:33:03.883049+0000", 2023-01-15T14:33:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap121" 2023-01-15T14:33:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 239, 2023-01-15T14:33:21.891 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:33:11.360450+0000", 2023-01-15T14:33:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap122" 2023-01-15T14:33:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 242, 2023-01-15T14:33:21.892 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:33:15.798243+0000", 2023-01-15T14:33:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap125" 2023-01-15T14:33:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 244, 2023-01-15T14:33:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:33:17.381083+0000", 2023-01-15T14:33:21.893 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap126" 2023-01-15T14:33:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 246, 2023-01-15T14:33:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:33:19.645695+0000", 2023-01-15T14:33:21.894 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap127" 2023-01-15T14:33:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 247, 2023-01-15T14:33:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:33:20.830206+0000", 2023-01-15T14:33:21.895 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap128" 2023-01-15T14:33:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:33:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:33:21.896 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:33:21.897 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:33:21.897 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:33:21.897 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:33:21.897 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:33:21.898 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:33:21.898 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:33:21.899 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:33:21.899 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:33:21.899 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:33:21.899 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:33:21.899 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:33:21.899 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:33:21.900 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:33:21.900 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:33:21.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:33:21.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.900 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:33:21.901 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:33:21.901 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:33:21.901 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:33:21.901 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:33:21.901 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:33:21.902 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:33:21.902 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:33:21.902 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:33:21.902 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:33:21.902 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:33:21.903 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:33:21.903 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:33:21.903 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2023-01-15T14:33:21.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:21.903 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:21.903 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:21.904 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:33:21.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.904 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:33:21.904 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e55c5ac4-3cff-4ee5-8719-373648f83d8d", 2023-01-15T14:33:21.904 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:33:21.905 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:33:21.905 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.3336334228515625, 2023-01-15T14:33:21.906 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:33:21.906 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:33:21.906 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:33:21.906 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:33:21.907 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 300, 2023-01-15T14:33:21.907 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:33:21.907 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:33:21.907 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:33:21.907 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:21.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:21.908 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:21.908 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2023-01-15T14:33:21.908 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:33:21.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:21.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.089 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2023-01-15T14:33:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:33:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:33:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2023-01-15T14:33:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:33:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2023-01-15T14:33:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:33:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:33:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2023-01-15T14:33:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:33:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.093 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2023-01-15T14:33:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:33:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:33:22.094 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2023-01-15T14:33:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:33:22.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2023-01-15T14:33:22.096 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:33:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6811/23473", 2023-01-15T14:33:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6815/23473", 2023-01-15T14:33:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/23473", 2023-01-15T14:33:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6820/23473", 2023-01-15T14:33:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:33:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:33:22.097 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:33:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:33:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "2c1b33c9-0a1e-4734-936d-d51d64fe997a", 2023-01-15T14:33:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:33:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:33:22.098 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:33:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:33:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:33:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:33:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:33:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 299, 2023-01-15T14:33:22.099 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:33:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:33:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:33:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2023-01-15T14:33:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:33:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.100 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2023-01-15T14:33:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:33:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:33:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2023-01-15T14:33:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:33:22.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.102 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:33:22.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.297 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.298 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2023-01-15T14:33:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:33:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.302 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:33:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.303 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2023-01-15T14:33:22.304 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:33:22.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2023-01-15T14:33:22.305 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:33:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/23474", 2023-01-15T14:33:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/23474", 2023-01-15T14:33:22.307 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/23474", 2023-01-15T14:33:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/23474", 2023-01-15T14:33:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:33:22.308 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:33:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:33:22.309 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.311 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:33:22.312 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "b314174a-eca9-4baa-a60c-2589f1fc9a8e", 2023-01-15T14:33:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:33:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:33:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:33:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0, 2023-01-15T14:33:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:33:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:33:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:33:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 298, 2023-01-15T14:33:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:33:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:33:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:33:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.501 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2023-01-15T14:33:22.501 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:33:22.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.501 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.502 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2023-01-15T14:33:22.502 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:33:22.502 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.502 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.502 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:33:22.503 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.503 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.503 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2023-01-15T14:33:22.503 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:33:22.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.504 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.504 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2023-01-15T14:33:22.504 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:33:22.504 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.505 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.505 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:33:22.505 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.505 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.506 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2023-01-15T14:33:22.506 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:33:22.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.506 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.506 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2023-01-15T14:33:22.507 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:33:22.507 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.507 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.507 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:33:22.508 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.508 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.508 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2023-01-15T14:33:22.508 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:33:22.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.509 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.509 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2023-01-15T14:33:22.509 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:33:22.509 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.509 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.510 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6810/23475", 2023-01-15T14:33:22.510 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/23475", 2023-01-15T14:33:22.510 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6821/23475", 2023-01-15T14:33:22.510 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6817/23475", 2023-01-15T14:33:22.510 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:33:22.510 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:33:22.510 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:33:22.511 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.511 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:33:22.511 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "29eb0b17-ed5d-4665-a347-2fb8406d9c62", 2023-01-15T14:33:22.511 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:33:22.512 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:33:22.512 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:33:22.512 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.9044189453125, 2023-01-15T14:33:22.512 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:33:22.512 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:33:22.513 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:33:22.513 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 310, 2023-01-15T14:33:22.513 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:33:22.513 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:33:22.513 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:33:22.513 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.514 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.514 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6824", 2023-01-15T14:33:22.514 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:33:22.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.515 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6825", 2023-01-15T14:33:22.515 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:33:22.516 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.516 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.516 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:33:22.516 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.517 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.517 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6826", 2023-01-15T14:33:22.517 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:33:22.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6827", 2023-01-15T14:33:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:33:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.657 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:33:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6830", 2023-01-15T14:33:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:33:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.658 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6831", 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6828", 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6829", 2023-01-15T14:33:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:33:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6825/23778", 2023-01-15T14:33:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6827/23778", 2023-01-15T14:33:22.660 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6831/23778", 2023-01-15T14:33:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6829/23778", 2023-01-15T14:33:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:33:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:33:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:33:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:33:22.661 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "59362feb-caa9-44e3-b6d0-b79397dacbfc", 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 310, 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6800", 2023-01-15T14:33:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:33:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6801", 2023-01-15T14:33:22.663 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:33:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:33:22.664 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.666 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6802", 2023-01-15T14:33:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:33:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6803", 2023-01-15T14:33:22.667 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:33:22.668 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:33:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6806", 2023-01-15T14:33:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:33:22.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6807", 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6804", 2023-01-15T14:33:22.671 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:33:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6805", 2023-01-15T14:33:22.672 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6801/23523", 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6803/23523", 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6807/23523", 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6805/23523", 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:33:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "fbc8821c-8d78-40f8-a571-b18a70d42000", 2023-01-15T14:33:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:33:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:33:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:33:22.674 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:33:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:33:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:33:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:33:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 20, 2023-01-15T14:33:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:33:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:33:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:33:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.675 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6824", 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6825", 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.677 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6826", 2023-01-15T14:33:22.816 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:33:22.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.816 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6827", 2023-01-15T14:33:22.817 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:33:22.817 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.817 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.817 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:33:22.818 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.818 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.818 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6830", 2023-01-15T14:33:22.818 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:33:22.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.819 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.819 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6831", 2023-01-15T14:33:22.825 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:33:22.825 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.825 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.826 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:33:22.826 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.826 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.826 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6828", 2023-01-15T14:33:22.826 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:33:22.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.827 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6829", 2023-01-15T14:33:22.827 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:33:22.828 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.828 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.828 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6825/23525", 2023-01-15T14:33:22.829 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6827/23525", 2023-01-15T14:33:22.829 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6831/23525", 2023-01-15T14:33:22.829 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6829/23525", 2023-01-15T14:33:22.829 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:33:22.829 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:33:22.829 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:33:22.830 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.830 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:33:22.830 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "34550470-3561-492f-b605-b5f0e8052c1e", 2023-01-15T14:33:22.831 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:33:22.831 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:33:22.831 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:33:22.831 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:33:22.831 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:33:22.832 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:33:22.832 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:33:22.832 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 118, 2023-01-15T14:33:22.832 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:33:22.832 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:33:22.833 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:33:22.833 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.833 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.833 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6808", 2023-01-15T14:33:22.833 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:33:22.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.834 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.834 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6809", 2023-01-15T14:33:22.835 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:33:22.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.835 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.835 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:33:22.836 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.837 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.837 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6810", 2023-01-15T14:33:22.837 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:33:22.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.838 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.838 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6811", 2023-01-15T14:33:22.838 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:33:22.838 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.838 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.838 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:33:22.838 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.839 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6814", 2023-01-15T14:33:22.839 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:33:22.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.839 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.839 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6815", 2023-01-15T14:33:22.839 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:33:22.839 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.839 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.840 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:33:22.840 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.840 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.840 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6812", 2023-01-15T14:33:22.840 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:33:22.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.840 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.840 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6813", 2023-01-15T14:33:22.841 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:33:22.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.841 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.841 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6809/23524", 2023-01-15T14:33:22.841 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6811/23524", 2023-01-15T14:33:22.841 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6815/23524", 2023-01-15T14:33:22.841 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6813/23524", 2023-01-15T14:33:22.841 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:33:22.841 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:33:22.841 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:33:22.842 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.842 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:33:22.842 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1f123d0d-d617-4771-b516-21aa5a29b734", 2023-01-15T14:33:22.842 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:33:22.842 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:33:22.842 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0.5779571533203125, 2023-01-15T14:33:22.842 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:33:22.843 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:33:22.843 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:33:22.843 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:33:22.843 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 311, 2023-01-15T14:33:22.843 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:33:22.843 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:33:22.843 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:33:22.843 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.843 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.844 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6816", 2023-01-15T14:33:22.844 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:33:22.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.844 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6817", 2023-01-15T14:33:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:33:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:33:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6818", 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6819", 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:33:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6822", 2023-01-15T14:33:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:33:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6823", 2023-01-15T14:33:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:33:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:33:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:33:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:33:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6820", 2023-01-15T14:33:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:33:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:33:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6821", 2023-01-15T14:33:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:33:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6817/23526", 2023-01-15T14:33:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6819/23526", 2023-01-15T14:33:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6823/23526", 2023-01-15T14:33:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6821/23526", 2023-01-15T14:33:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:33:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:33:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:33:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2023-01-15T14:33:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:33:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:33:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:33:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:33:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:33:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:33:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:32:49.623376+0000", 2023-01-15T14:33:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:33:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:33:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:33:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:33:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:33:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:33:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:33:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:54.007074+0000", 2023-01-15T14:33:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:33:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:33:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:33:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:33:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:33:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:33:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:33:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:31:53.308067+0000", 2023-01-15T14:33:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:33:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:33:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:33:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:33:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:33:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:33:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:33:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:32:54.262258+0000", 2023-01-15T14:33:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:33:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:33:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:33:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:33:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:33:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:33:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:33:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:33:19.243499+0000", 2023-01-15T14:33:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:33:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:33:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:33:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:33:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:33:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:32:32.921684+0000", 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:33:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:33:09.975420+0000", 2023-01-15T14:33:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:33:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:33:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:33:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:33:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:33:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:33:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:33:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:32:49.967643+0000", 2023-01-15T14:33:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:33:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2023-01-15T14:33:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2023-01-15T14:33:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.3", 2023-01-15T14:33:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:33:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 4, 2023-01-15T14:33:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 0 2023-01-15T14:33:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.11", 2023-01-15T14:33:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:33:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 4, 2023-01-15T14:33:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 0 2023-01-15T14:33:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.39", 2023-01-15T14:33:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:33:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 4, 2023-01-15T14:33:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 2 2023-01-15T14:33:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.40", 2023-01-15T14:33:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:33:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 4, 2023-01-15T14:33:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 7 2023-01-15T14:33:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.44", 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 4, 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 0 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2023-01-15T14:33:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [ 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.51", 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: 7, 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: 3 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2023-01-15T14:33:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2023-01-15T14:33:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 216, 2023-01-15T14:33:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:33:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 218, 2023-01-15T14:33:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:33:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 221, 2023-01-15T14:33:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:33:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 223, 2023-01-15T14:33:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:33:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 229, 2023-01-15T14:33:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2023-01-15T14:33:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 240, 2023-01-15T14:33:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2023-01-15T14:33:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2023-01-15T14:33:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2023-01-15T14:33:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2023-01-15T14:33:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2023-01-15T14:33:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2023-01-15T14:33:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2023-01-15T14:33:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2023-01-15T14:33:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2023-01-15T14:33:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2023-01-15T14:33:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2023-01-15T14:33:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:23.149 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:23.151 INFO:tasks.thrashosds.thrasher:j is {'epoch': 312, 'fsid': '1c77fb2b-d1a7-4884-a24e-3935e16d0b6b', 'created': '2023-01-15T14:27:57.735717+0000', 'modified': '2023-01-15T14:33:21.022739+0000', 'last_up_change': '2023-01-15T14:28:04.837522+0000', 'last_in_change': '2023-01-15T14:30:49.906949+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-01-15T14:28:04.899282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-01-15T14:28:11.115002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-01-15T14:28:22.434903+0000', 'flags': 16385, 'flags_names': 'hashpspool,pool_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 86, 'pg_placement_num': 86, 'pg_placement_num_target': 86, 'pg_num_target': 86, 'pg_num_pending': 86, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '312', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '310', 'auid': 0, 'snap_mode': 'pool', 'snap_seq': 247, 'snap_epoch': 312, 'pool_snaps': [{'snapid': 219, 'stamp': '2023-01-15T14:32:43.999592+0000', 'name': 'smithi09025264-snap112'}, {'snapid': 228, 'stamp': '2023-01-15T14:32:54.385736+0000', 'name': 'smithi09025264-snap117'}, {'snapid': 236, 'stamp': '2023-01-15T14:33:03.448556+0000', 'name': 'smithi09025264-snap120'}, {'snapid': 237, 'stamp': '2023-01-15T14:33:03.883049+0000', 'name': 'smithi09025264-snap121'}, {'snapid': 239, 'stamp': '2023-01-15T14:33:11.360450+0000', 'name': 'smithi09025264-snap122'}, {'snapid': 242, 'stamp': '2023-01-15T14:33:15.798243+0000', 'name': 'smithi09025264-snap125'}, {'snapid': 244, 'stamp': '2023-01-15T14:33:17.381083+0000', 'name': 'smithi09025264-snap126'}, {'snapid': 246, 'stamp': '2023-01-15T14:33:19.645695+0000', 'name': 'smithi09025264-snap127'}, {'snapid': 247, 'stamp': '2023-01-15T14:33:20.830206+0000', 'name': 'smithi09025264-snap128'}], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e55c5ac4-3cff-4ee5-8719-373648f83d8d', 'up': 1, 'in': 1, 'weight': 0.3336334228515625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 300, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 23473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 23473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 23473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6820', 'nonce': 23473}]}, 'public_addr': '172.21.15.90:6811/23473', 'cluster_addr': '172.21.15.90:6815/23473', 'heartbeat_back_addr': '172.21.15.90:6823/23473', 'heartbeat_front_addr': '172.21.15.90:6820/23473', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2c1b33c9-0a1e-4734-936d-d51d64fe997a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 299, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 23474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 23474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 23474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 23474}]}, 'public_addr': '172.21.15.90:6801/23474', 'cluster_addr': '172.21.15.90:6803/23474', 'heartbeat_back_addr': '172.21.15.90:6807/23474', 'heartbeat_front_addr': '172.21.15.90:6805/23474', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b314174a-eca9-4baa-a60c-2589f1fc9a8e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 298, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 23475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 23475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6819', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 23475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 23475}]}, 'public_addr': '172.21.15.90:6810/23475', 'cluster_addr': '172.21.15.90:6814/23475', 'heartbeat_back_addr': '172.21.15.90:6821/23475', 'heartbeat_front_addr': '172.21.15.90:6817/23475', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '29eb0b17-ed5d-4665-a347-2fb8406d9c62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.9044189453125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 310, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6824', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6825', 'nonce': 23778}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6826', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6827', 'nonce': 23778}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6830', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6831', 'nonce': 23778}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6828', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6829', 'nonce': 23778}]}, 'public_addr': '172.21.15.90:6825/23778', 'cluster_addr': '172.21.15.90:6827/23778', 'heartbeat_back_addr': '172.21.15.90:6831/23778', 'heartbeat_front_addr': '172.21.15.90:6829/23778', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '59362feb-caa9-44e3-b6d0-b79397dacbfc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 310, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6800', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6801', 'nonce': 23523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6802', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6803', 'nonce': 23523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6806', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6807', 'nonce': 23523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6804', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6805', 'nonce': 23523}]}, 'public_addr': '172.21.15.141:6801/23523', 'cluster_addr': '172.21.15.141:6803/23523', 'heartbeat_back_addr': '172.21.15.141:6807/23523', 'heartbeat_front_addr': '172.21.15.141:6805/23523', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fbc8821c-8d78-40f8-a571-b18a70d42000', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 20, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6824', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6825', 'nonce': 23525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6826', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6827', 'nonce': 23525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6830', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6831', 'nonce': 23525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6828', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6829', 'nonce': 23525}]}, 'public_addr': '172.21.15.141:6825/23525', 'cluster_addr': '172.21.15.141:6827/23525', 'heartbeat_back_addr': '172.21.15.141:6831/23525', 'heartbeat_front_addr': '172.21.15.141:6829/23525', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '34550470-3561-492f-b605-b5f0e8052c1e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 118, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6808', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6809', 'nonce': 23524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6810', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6811', 'nonce': 23524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6814', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6815', 'nonce': 23524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6812', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6813', 'nonce': 23524}]}, 'public_addr': '172.21.15.141:6809/23524', 'cluster_addr': '172.21.15.141:6811/23524', 'heartbeat_back_addr': '172.21.15.141:6815/23524', 'heartbeat_front_addr': '172.21.15.141:6813/23524', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f123d0d-d617-4771-b516-21aa5a29b734', 'up': 1, 'in': 1, 'weight': 0.5779571533203125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 311, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6816', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6817', 'nonce': 23526}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6818', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6819', 'nonce': 23526}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6822', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6823', 'nonce': 23526}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6820', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6821', 'nonce': 23526}]}, 'public_addr': '172.21.15.141:6817/23526', 'cluster_addr': '172.21.15.141:6819/23526', 'heartbeat_back_addr': '172.21.15.141:6823/23526', 'heartbeat_front_addr': '172.21.15.141:6821/23526', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:32:49.623376+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:54.007074+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:31:53.308067+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:32:54.262258+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:33:19.243499+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:32:32.921684+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:33:09.975420+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:32:49.967643+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '3.11', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '3.39', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.40', 'mappings': [{'from': 4, 'to': 7}]}, {'pgid': '3.44', 'mappings': [{'from': 4, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.51', 'osds': [7, 3]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 216, 'length': 1}, {'begin': 218, 'length': 1}, {'begin': 221, 'length': 1}, {'begin': 223, 'length': 1}, {'begin': 229, 'length': 2}, {'begin': 240, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-01-15T14:33:23.151 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.44 2023-01-15T14:33:23.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap-items 3.44 2023-01-15T14:33:23.152 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:23.152 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:23.153 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:33:23.153 INFO:tasks.rados.rados.0.smithi090.stdout:1242: expect (ObjNum 427 snap 116 seq_num 427) 2023-01-15T14:33:23.153 INFO:tasks.rados.rados.0.smithi090.stdout:1242: done (1 left) 2023-01-15T14:33:23.154 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:23.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:33:23.154 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:33:23.155 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:21.480+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:23.306 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:23.153+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:23.306 INFO:tasks.rados.rados.0.smithi090.stdout:1244: expect (ObjNum 448 snap 124 seq_num 448) 2023-01-15T14:33:23.307 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:23.301+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:23.369 INFO:tasks.rados.rados.0.smithi090.stdout:1244: done (0 left) 2023-01-15T14:33:23.477 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:23.477 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:23.478 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:23.479 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:33:23.479 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:23.480 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:23.475+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:23.497 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:23.632 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:23.632 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:23.632 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:23.633 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:23.633 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:23.634 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:33:23.634 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:23.634 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:33:23.635 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:23.635 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:23.636 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:23.629+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:23.636 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:33:23.636 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:33:23.636 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:33:23.637 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:33:23.637 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:33:23.637 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:33:23.637 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:33:23.637 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:33:23.733 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:23.729+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:23.801 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:23.801 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:23.801 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:33:23.801 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:33:23.801 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:23.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:33:23.834 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:23.831+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:24.006 INFO:tasks.rados.rados.0.smithi090.stdout:1245: done (0 left) 2023-01-15T14:33:24.006 INFO:tasks.rados.rados.0.smithi090.stdout:1246: write (excl) oid 16 current snap is 129 2023-01-15T14:33:24.006 INFO:tasks.rados.rados.0.smithi090.stdout:1246: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2023-01-15T14:33:24.006 INFO:tasks.rados.rados.0.smithi090.stdout:1246: writing smithi09025264-16 from 447185 to 1150307 tid 1 2023-01-15T14:33:24.007 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:33:24.007 INFO:teuthology.orchestra.run.smithi090.stderr:clear 3.44 pg_upmap_items mapping 2023-01-15T14:33:24.008 INFO:tasks.rados.rados.0.smithi090.stdout:1246: writing smithi09025264-16 from 1642601 to 2408415 tid 2 2023-01-15T14:33:24.008 INFO:tasks.rados.rados.0.smithi090.stdout:1246: writing smithi09025264-16 from 2928991 to 3376698 tid 3 2023-01-15T14:33:24.008 INFO:tasks.rados.rados.0.smithi090.stdout:1247: rollback oid 5 current snap is 129 2023-01-15T14:33:24.008 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 5 to 111 2023-01-15T14:33:24.008 INFO:tasks.rados.rados.0.smithi090.stdout:1248: write (excl) oid 25 current snap is 129 2023-01-15T14:33:24.008 INFO:tasks.rados.rados.0.smithi090.stdout:1248: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2023-01-15T14:33:24.010 INFO:tasks.rados.rados.0.smithi090.stdout:1248: writing smithi09025264-25 from 463992 to 1242363 tid 1 2023-01-15T14:33:24.011 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.011 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:33:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 169.02422438400001, 2023-01-15T14:33:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:33:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:24.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:33:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:24.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:33:24.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:33:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:33:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:24.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:33:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:33:24.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:33:24.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:33:24.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:33:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:33:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:33:24.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:24.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:24.418 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 169.02417344099999, 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:24.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:24.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:33:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:33:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:33:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:33:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:33:24.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:33:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:33:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:33:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:33:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:33:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:33:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:33:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:24.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:33:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:33:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:33:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 166.08327420000001, 2023-01-15T14:33:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:33:24.429 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:33:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:24.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:33:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:33:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:33:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:33:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:33:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:33:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:33:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:33:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:33:24.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:33:24.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:33:24.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:33:24.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:33:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:33:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:33:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:33:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:33:24.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:33:24.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:33:24.585 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:24.586 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 164.009500027, 2023-01-15T14:33:24.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:33:24.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:24.586 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:24.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:24.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:24.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:24.586 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:33:24.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.587 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:24.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:24.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:24.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:24.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:24.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:24.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:33:24.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:33:24.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:24.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:33:24.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:33:24.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:33:24.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:24.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:33:24.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:33:24.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:24.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:33:24.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:33:24.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:33:24.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:24.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:33:24.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:33:24.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:24.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:33:24.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:33:24.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:24.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:33:24.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:33:24.595 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:24.595 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:24.595 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:24.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.595 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:33:24.595 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:24.595 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 140.115544607, 2023-01-15T14:33:24.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:33:24.596 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:24.596 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:24.596 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:24.596 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:24.596 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:24.596 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:33:24.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.596 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:24.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:24.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:33:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:33:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:33:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:33:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:33:24.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:33:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:33:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:33:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:33:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:33:24.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:24.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:33:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:33:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 118.23213792599999, 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:24.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:33:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:33:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:33:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:33:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:33:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:33:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:33:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:33:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:33:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:33:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:33:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:33:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:33:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:24.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:33:24.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:33:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:33:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:33:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:33:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:33:24.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:24.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:24.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:24.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:33:24.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:33:24.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:25.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:33:25.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:33:25.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:25.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:33:25.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:33:25.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:33:25.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:33:25.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:33:25.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:33:25.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:25.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:33:25.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:33:25.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:33:25.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:33:25.092 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.092 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.092 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.093 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:25.093 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:25.093 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 118.228730803, 2023-01-15T14:33:25.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:33:25.093 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.093 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:25.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:25.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:25.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:25.094 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:33:25.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.094 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:25.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:33:25.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:33:25.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:25.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:33:25.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:33:25.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:25.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:33:25.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:33:25.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:25.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:33:25.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:33:25.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:25.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:33:25.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:33:25.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:33:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:33:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 118.228515198, 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.262 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:33:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:33:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:33:25.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:33:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:33:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:33:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:33:25.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:33:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:33:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:33:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:33:25.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:33:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:33:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:33:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:33:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:33:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:33:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:33:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:33:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:33:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:33:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:33:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:33:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:33:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:33:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:33:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:33:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:33:25.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:33:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 118.228408124, 2023-01-15T14:33:25.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:33:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:33:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.273 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:33:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:33:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:33:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:33:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:33:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:33:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:33:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:33:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:33:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:33:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:33:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:33:25.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:33:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:33:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:33:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:33:25.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:33:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:33:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:33:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:33:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:33:25.279 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.098169074, 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:33:25.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:33:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:33:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:33:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:33:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:33:25.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:33:25.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.098064537, 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:25.290 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:33:25.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:33:25.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:33:25.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:25.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:33:25.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:33:25.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:25.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:33:25.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:33:25.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:25.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:33:25.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:33:25.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:25.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:33:25.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:33:25.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:25.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:33:25.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:33:25.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:33:25.429 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.429 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.429 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.429 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:25.429 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:25.429 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.097994606, 2023-01-15T14:33:25.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:33:25.430 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.430 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:25.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:25.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:25.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:25.430 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:33:25.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.430 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:25.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:25.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:33:25.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:25.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:33:25.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:33:25.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:33:25.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:33:25.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:33:25.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:33:25.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:33:25.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:33:25.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:25.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:33:25.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:33:25.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:33:25.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:33:25.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:25.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:33:25.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:33:25.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:25.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:33:25.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:33:25.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:25.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:33:25.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:33:25.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:25.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:33:25.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:33:25.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:25.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:33:25.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:33:25.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:25.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:33:25.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:33:25.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:33:25.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:33:25.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:25.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:33:25.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:33:25.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:25.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:33:25.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:33:25.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:33:25.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2780 3.38 e265/232 3:1eaad226:::smithi09025264-49:head v 265'279, mlcod=261'274)", 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 51.926823480000003, 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32005591799999999, 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:25.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:25.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991860+0000", 2023-01-15T14:33:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993927+0000", 2023-01-15T14:33:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020673229999999998 2023-01-15T14:33:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993946+0000", 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9429999999999999e-05 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.094039+0000", 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092898 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285071+0000", 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.191031904 2023-01-15T14:33:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285106+0000", 2023-01-15T14:33:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5227e-05 2023-01-15T14:33:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286323+0000", 2023-01-15T14:33:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012163829999999999 2023-01-15T14:33:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286335+0000", 2023-01-15T14:33:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2224e-05 2023-01-15T14:33:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287218+0000", 2023-01-15T14:33:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088265000000000001 2023-01-15T14:33:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287250+0000", 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2209000000000003e-05 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.311917+0000", 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024667062999999999 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2868 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'278, mlcod=264'275)", 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.924932538999997, 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31268252400000002, 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993751+0000", 2023-01-15T14:33:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993762+0000", 2023-01-15T14:33:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0822000000000001e-05 2023-01-15T14:33:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993777+0000", 2023-01-15T14:33:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5540999999999999e-05 2023-01-15T14:33:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993787+0000", 2023-01-15T14:33:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0173999999999999e-05 2023-01-15T14:33:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290198+0000", 2023-01-15T14:33:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29641022099999997 2023-01-15T14:33:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290212+0000", 2023-01-15T14:33:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4214e-05 2023-01-15T14:33:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290300+0000", 2023-01-15T14:33:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8300000000000005e-05 2023-01-15T14:33:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290312+0000", 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2344000000000001e-05 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290357+0000", 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4367999999999999e-05 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290389+0000", 2023-01-15T14:33:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1983000000000002e-05 2023-01-15T14:33:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.306434+0000", 2023-01-15T14:33:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016045667 2023-01-15T14:33:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:33:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.924883113999996, 2023-01-15T14:33:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:33:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:33:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:33:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:33:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:33:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:33:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:33:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:33:25.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.608 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.921071427000001, 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:33:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:33:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:33:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:33:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:33:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:33:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:33:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:33:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:33:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3120 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'290, mlcod=0'0)", 2023-01-15T14:33:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:33:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.679258150999999, 2023-01-15T14:33:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39853714000000001, 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:33:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239424+0000", 2023-01-15T14:33:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239747+0000", 2023-01-15T14:33:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032243300000000001 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239784+0000", 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7051000000000002e-05 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239802+0000", 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8250999999999999e-05 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536033+0000", 2023-01-15T14:33:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29623121499999999 2023-01-15T14:33:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536081+0000", 2023-01-15T14:33:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2023-01-15T14:33:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536800+0000", 2023-01-15T14:33:25.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071948499999999996 2023-01-15T14:33:25.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:25.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536817+0000", 2023-01-15T14:33:25.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6228000000000001e-05 2023-01-15T14:33:25.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:25.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537048+0000", 2023-01-15T14:33:25.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023136200000000001 2023-01-15T14:33:25.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:25.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637902+0000", 2023-01-15T14:33:25.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10085403900000001 2023-01-15T14:33:25.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:25.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637963+0000", 2023-01-15T14:33:25.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1310999999999996e-05 2023-01-15T14:33:25.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.752 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:25.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:25.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.754 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3121 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'291, mlcod=0'0)", 2023-01-15T14:33:25.754 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:33:25.754 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.677262635, 2023-01-15T14:33:25.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659573799999998, 2023-01-15T14:33:25.754 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:25.754 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:25.755 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:25.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:25.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:33:25.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:25.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:33:25.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:25.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:25.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241420+0000", 2023-01-15T14:33:25.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:25.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:25.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241854+0000", 2023-01-15T14:33:25.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043330099999999998 2023-01-15T14:33:25.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:25.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241878+0000", 2023-01-15T14:33:25.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4559e-05 2023-01-15T14:33:25.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:25.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241892+0000", 2023-01-15T14:33:25.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039999999999999e-05 2023-01-15T14:33:25.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:25.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:25.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:26.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536863+0000", 2023-01-15T14:33:26.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29497073400000001 2023-01-15T14:33:26.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:26.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536887+0000", 2023-01-15T14:33:26.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3683999999999999e-05 2023-01-15T14:33:26.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:26.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537124+0000", 2023-01-15T14:33:26.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023729900000000001 2023-01-15T14:33:26.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:26.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537136+0000", 2023-01-15T14:33:26.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2278e-05 2023-01-15T14:33:26.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:26.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537345+0000", 2023-01-15T14:33:26.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020865200000000001 2023-01-15T14:33:26.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:26.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637981+0000", 2023-01-15T14:33:26.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063575700000001 2023-01-15T14:33:26.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:26.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638018+0000", 2023-01-15T14:33:26.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6826999999999998e-05 2023-01-15T14:33:26.076 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:26.076 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:26.077 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:26.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.077 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3122 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'292, mlcod=0'0)", 2023-01-15T14:33:26.077 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:33:26.077 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.675366795, 2023-01-15T14:33:26.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39623551600000001, 2023-01-15T14:33:26.078 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:26.078 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:26.078 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:26.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:26.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:33:26.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:26.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:26.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:33:26.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:26.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:26.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243316+0000", 2023-01-15T14:33:26.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:26.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:26.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243833+0000", 2023-01-15T14:33:26.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051703600000000001 2023-01-15T14:33:26.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:26.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243863+0000", 2023-01-15T14:33:26.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-01-15T14:33:26.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:26.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243879+0000", 2023-01-15T14:33:26.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6263999999999999e-05 2023-01-15T14:33:26.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:26.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637317+0000", 2023-01-15T14:33:26.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39343783199999999 2023-01-15T14:33:26.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:26.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637356+0000", 2023-01-15T14:33:26.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9579000000000002e-05 2023-01-15T14:33:26.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:26.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637830+0000", 2023-01-15T14:33:26.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047326399999999999 2023-01-15T14:33:26.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637885+0000", 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5074999999999997e-05 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638185+0000", 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029993799999999997 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638337+0000", 2023-01-15T14:33:26.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015208299999999999 2023-01-15T14:33:26.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:26.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.639553+0000", 2023-01-15T14:33:26.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001216389 2023-01-15T14:33:26.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:26.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:26.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:26.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.091 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3123 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'293, mlcod=0'0)", 2023-01-15T14:33:26.092 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:33:26.092 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.673622766999999, 2023-01-15T14:33:26.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.395733102, 2023-01-15T14:33:26.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:26.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:26.092 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:26.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:26.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:33:26.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:26.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:26.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:33:26.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:26.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:26.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245060+0000", 2023-01-15T14:33:26.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:26.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:26.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245075+0000", 2023-01-15T14:33:26.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4459e-05 2023-01-15T14:33:26.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:26.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245107+0000", 2023-01-15T14:33:26.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1795000000000003e-05 2023-01-15T14:33:26.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:26.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245121+0000", 2023-01-15T14:33:26.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4117000000000001e-05 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638063+0000", 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39294204300000002 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638097+0000", 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4385999999999999e-05 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:26.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638299+0000", 2023-01-15T14:33:26.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020133099999999999 2023-01-15T14:33:26.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:26.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638318+0000", 2023-01-15T14:33:26.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514999999999999e-05 2023-01-15T14:33:26.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:26.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638386+0000", 2023-01-15T14:33:26.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8089000000000005e-05 2023-01-15T14:33:26.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:26.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638430+0000", 2023-01-15T14:33:26.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.439e-05 2023-01-15T14:33:26.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:26.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:26.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:26.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.640795+0000", 2023-01-15T14:33:26.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002364274 2023-01-15T14:33:26.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:26.123 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:26.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:26.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:26.123 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:26.123 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:26.124 INFO:tasks.rados.rados.0.smithi090.stdout:1248: writing smithi09025264-25 from 1701083 to 2326908 tid 2 2023-01-15T14:33:26.124 INFO:tasks.rados.rados.0.smithi090.stdout:1248: writing smithi09025264-25 from 2956414 to 3615393 tid 3 2023-01-15T14:33:26.124 INFO:tasks.rados.rados.0.smithi090.stdout:1249: snap_create 2023-01-15T14:33:26.124 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:26.126 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:33:26.127 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:33:26.128 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:33:26.128 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:24.003+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:26.129 INFO:tasks.rados.rados.0.smithi090.stdout:1246: finishing write tid 1 to smithi09025264-16 2023-01-15T14:33:26.130 INFO:tasks.rados.rados.0.smithi090.stdout:1248: finishing write tid 1 to smithi09025264-25 2023-01-15T14:33:26.130 INFO:tasks.rados.rados.0.smithi090.stdout:1247: finishing rollback tid 0 to smithi09025264-5 2023-01-15T14:33:26.130 INFO:tasks.rados.rados.0.smithi090.stdout:1246: finishing write tid 2 to smithi09025264-16 2023-01-15T14:33:26.130 INFO:tasks.rados.rados.0.smithi090.stdout:1246: finishing write tid 3 to smithi09025264-16 2023-01-15T14:33:26.130 INFO:tasks.rados.rados.0.smithi090.stdout:1246: finishing write tid 4 to smithi09025264-16 2023-01-15T14:33:26.130 INFO:tasks.rados.rados.0.smithi090.stdout:1246: finishing write tid 5 to smithi09025264-16 2023-01-15T14:33:26.130 INFO:tasks.rados.rados.0.smithi090.stdout:1246: oid 16 updating version 0 to 173 2023-01-15T14:33:26.130 INFO:tasks.rados.rados.0.smithi090.stdout:1246: oid 16 version 173 is already newer than 171 2023-01-15T14:33:26.130 INFO:tasks.rados.rados.0.smithi090.stdout:1246: oid 16 version 173 is already newer than 170 2023-01-15T14:33:26.131 INFO:tasks.rados.rados.0.smithi090.stdout:1246: oid 16 version 173 is already newer than 172 2023-01-15T14:33:26.131 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 173 (ObjNum 455 snap 129 seq_num 455) dirty exists 2023-01-15T14:33:26.131 INFO:tasks.rados.rados.0.smithi090.stdout:1246: left oid 16 (ObjNum 455 snap 129 seq_num 455) 2023-01-15T14:33:26.131 INFO:tasks.rados.rados.0.smithi090.stdout:1246: done (2 left) 2023-01-15T14:33:26.131 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:26.131 INFO:tasks.rados.rados.0.smithi090.stdout:1247: finishing rollback tid 1 to smithi09025264-5 2023-01-15T14:33:26.131 INFO:tasks.rados.rados.0.smithi090.stdout:1248: finishing write tid 2 to smithi09025264-25 2023-01-15T14:33:26.131 INFO:tasks.rados.rados.0.smithi090.stdout:1248: finishing write tid 3 to smithi09025264-25 2023-01-15T14:33:26.131 INFO:tasks.rados.rados.0.smithi090.stdout:1248: finishing write tid 4 to smithi09025264-25 2023-01-15T14:33:26.131 INFO:tasks.rados.rados.0.smithi090.stdout:1247: finishing rollback tid 2 to smithi09025264-5 2023-01-15T14:33:26.132 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 240 (ObjNum 387 snap 103 seq_num 387) dirty exists 2023-01-15T14:33:26.132 INFO:tasks.rados.rados.0.smithi090.stdout:1247: done (1 left) 2023-01-15T14:33:26.132 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:26.132 INFO:tasks.rados.rados.0.smithi090.stdout:1248: finishing write tid 5 to smithi09025264-25 2023-01-15T14:33:26.132 INFO:tasks.rados.rados.0.smithi090.stdout:1248: oid 25 updating version 0 to 334 2023-01-15T14:33:26.132 INFO:tasks.rados.rados.0.smithi090.stdout:1248: oid 25 version 334 is already newer than 331 2023-01-15T14:33:26.132 INFO:tasks.rados.rados.0.smithi090.stdout:1248: oid 25 version 334 is already newer than 332 2023-01-15T14:33:26.132 INFO:tasks.rados.rados.0.smithi090.stdout:1248: oid 25 version 334 is already newer than 333 2023-01-15T14:33:26.132 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 334 (ObjNum 456 snap 129 seq_num 456) dirty exists 2023-01-15T14:33:26.133 INFO:tasks.rados.rados.0.smithi090.stdout:1248: left oid 25 (ObjNum 456 snap 129 seq_num 456) 2023-01-15T14:33:26.133 INFO:tasks.rados.rados.0.smithi090.stdout:1248: done (0 left) 2023-01-15T14:33:26.133 INFO:tasks.rados.rados.0.smithi090.stdout:1249: done (0 left) 2023-01-15T14:33:26.133 INFO:tasks.rados.rados.0.smithi090.stdout:1250: snap_create 2023-01-15T14:33:26.133 INFO:tasks.rados.rados.0.smithi090.stdout:1250: done (0 left) 2023-01-15T14:33:26.133 INFO:tasks.rados.rados.0.smithi090.stdout:1251: copy_from oid 40 from oid 35 current snap is 131 2023-01-15T14:33:26.133 INFO:tasks.rados.rados.0.smithi090.stdout:1252: rollback oid 13 current snap is 131 2023-01-15T14:33:26.249 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 13 to 124 2023-01-15T14:33:26.249 INFO:tasks.rados.rados.0.smithi090.stdout:1253: copy_from oid 14 from oid 8 current snap is 131 2023-01-15T14:33:26.249 INFO:tasks.rados.rados.0.smithi090.stdout:1254: snap_remove snap 116 2023-01-15T14:33:26.250 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:26.125+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:26.253 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:26.249+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:26.352 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:26.349+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:26.401 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:26.401 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:26.401 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:33:26.401 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:26.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:33:26.402 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:26.415 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:26.440 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:26.589 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:26.589 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:26.589 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:26.590 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:26.590 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:26.591 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:33:26.591 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:26.592 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:26.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:33:26.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:33:26.593 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:33:26.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:33:26.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:33:26.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:33:26.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:33:26.594 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:33:26.595 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:26.587+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:26.690 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:26.687+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:26.798 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:26.799 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:26.799 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:33:26.800 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:26.795+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:26.889 INFO:tasks.rados.rados.0.smithi090.stdout:1254: done (3 left) 2023-01-15T14:33:26.958 INFO:tasks.rados.rados.0.smithi090.stdout:1255: copy_from oid 27 from oid 45 current snap is 131 2023-01-15T14:33:26.958 INFO:tasks.rados.rados.0.smithi090.stdout:1256: copy_from oid 21 from oid 23 current snap is 131 2023-01-15T14:33:26.958 INFO:tasks.rados.rados.0.smithi090.stdout:1252: finishing rollback tid 0 to smithi09025264-13 2023-01-15T14:33:26.958 INFO:tasks.rados.rados.0.smithi090.stdout:1251: finishing copy_from to smithi09025264-40 2023-01-15T14:33:26.958 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 307 (ObjNum 415 snap 112 seq_num 415) dirty exists 2023-01-15T14:33:26.958 INFO:tasks.rados.rados.0.smithi090.stdout:1251: finishing copy_from racing read to smithi09025264-40 2023-01-15T14:33:26.958 INFO:tasks.rados.rados.0.smithi090.stdout:1252: finishing rollback tid 1 to smithi09025264-13 2023-01-15T14:33:26.958 INFO:tasks.rados.rados.0.smithi090.stdout:1252: finishing rollback tid 2 to smithi09025264-13 2023-01-15T14:33:26.958 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 335 (ObjNum 412 snap 112 seq_num 412) dirty exists 2023-01-15T14:33:26.958 INFO:tasks.rados.rados.0.smithi090.stdout:1253: finishing copy_from to smithi09025264-14 2023-01-15T14:33:26.958 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 263 (ObjNum 404 snap 108 seq_num 404) dirty exists 2023-01-15T14:33:26.959 INFO:tasks.rados.rados.0.smithi090.stdout:1253: finishing copy_from racing read to smithi09025264-14 2023-01-15T14:33:26.959 INFO:tasks.rados.rados.0.smithi090.stdout:1251: done (4 left) 2023-01-15T14:33:26.959 INFO:tasks.rados.rados.0.smithi090.stdout:1252: done (3 left) 2023-01-15T14:33:26.960 INFO:tasks.rados.rados.0.smithi090.stdout:1253: done (2 left) 2023-01-15T14:33:26.960 INFO:tasks.rados.rados.0.smithi090.stdout:1257: write (excl) oid 8 current snap is 131 2023-01-15T14:33:26.960 INFO:tasks.rados.rados.0.smithi090.stdout:1257: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2023-01-15T14:33:26.960 INFO:tasks.rados.rados.0.smithi090.stdout:1257: writing smithi09025264-8 from 480799 to 1050772 tid 1 2023-01-15T14:33:26.961 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:33:26.962 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:26.962 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:26.962 INFO:tasks.rados.rados.0.smithi090.stdout:1257: writing smithi09025264-8 from 1475918 to 2245401 tid 2 2023-01-15T14:33:26.963 INFO:tasks.rados.rados.0.smithi090.stdout:1257: writing smithi09025264-8 from 2700190 to 3286794 tid 3 2023-01-15T14:33:26.963 INFO:tasks.rados.rados.0.smithi090.stdout:1258: write (excl) oid 49 current snap is 131 2023-01-15T14:33:26.963 INFO:tasks.rados.rados.0.smithi090.stdout:1257: finishing write tid 1 to smithi09025264-8 2023-01-15T14:33:26.963 INFO:tasks.rados.rados.0.smithi090.stdout:1258: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2023-01-15T14:33:26.963 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:33:26.964 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:26.959+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:26.965 INFO:tasks.rados.rados.0.smithi090.stdout:1258: writing smithi09025264-49 from 497606 to 1142828 tid 1 2023-01-15T14:33:26.965 INFO:tasks.rados.rados.0.smithi090.stdout:1258: writing smithi09025264-49 from 1934400 to 2563894 tid 2 2023-01-15T14:33:26.965 INFO:tasks.rados.rados.0.smithi090.stdout:1258: writing smithi09025264-49 from 3243966 to 3697606 tid 3 2023-01-15T14:33:26.965 INFO:tasks.rados.rados.0.smithi090.stdout:1259: read oid 9 snap 130 2023-01-15T14:33:26.965 INFO:tasks.rados.rados.0.smithi090.stdout:1259: expect deleted 2023-01-15T14:33:26.965 INFO:tasks.rados.rados.0.smithi090.stdout:1260: snap_create 2023-01-15T14:33:26.965 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:33:26.995 INFO:tasks.rados.rados.0.smithi090.stdout:1255: finishing copy_from to smithi09025264-27 2023-01-15T14:33:27.118 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 140 (ObjNum 400 snap 106 seq_num 400) dirty exists 2023-01-15T14:33:27.118 INFO:tasks.rados.rados.0.smithi090.stdout:1255: finishing copy_from racing read to smithi09025264-27 2023-01-15T14:33:27.118 INFO:tasks.rados.rados.0.smithi090.stdout:1255: done (4 left) 2023-01-15T14:33:27.118 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:33:27.121 INFO:tasks.rados.rados.0.smithi090.stdout:1259: done (3 left) 2023-01-15T14:33:27.121 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:27.122 INFO:tasks.rados.rados.0.smithi090.stdout:1258: finishing write tid 1 to smithi09025264-49 2023-01-15T14:33:27.122 INFO:tasks.rados.rados.0.smithi090.stdout:1257: finishing write tid 2 to smithi09025264-8 2023-01-15T14:33:27.122 INFO:tasks.rados.rados.0.smithi090.stdout:1257: finishing write tid 3 to smithi09025264-8 2023-01-15T14:33:27.123 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:27.117+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:27.123 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:27.123 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:27.123 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:33:27.179 INFO:tasks.rados.rados.0.smithi090.stdout:1257: finishing write tid 4 to smithi09025264-8 2023-01-15T14:33:27.279 INFO:tasks.rados.rados.0.smithi090.stdout:1257: finishing write tid 5 to smithi09025264-8 2023-01-15T14:33:27.279 INFO:tasks.rados.rados.0.smithi090.stdout:1257: oid 8 updating version 0 to 294 2023-01-15T14:33:27.279 INFO:tasks.rados.rados.0.smithi090.stdout:1257: oid 8 version 294 is already newer than 292 2023-01-15T14:33:27.280 INFO:tasks.rados.rados.0.smithi090.stdout:1257: oid 8 version 294 is already newer than 291 2023-01-15T14:33:27.280 INFO:tasks.rados.rados.0.smithi090.stdout:1257: oid 8 version 294 is already newer than 293 2023-01-15T14:33:27.280 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 294 (ObjNum 457 snap 131 seq_num 457) dirty exists 2023-01-15T14:33:27.280 INFO:tasks.rados.rados.0.smithi090.stdout:1257: left oid 8 (ObjNum 457 snap 131 seq_num 457) 2023-01-15T14:33:27.280 INFO:tasks.rados.rados.0.smithi090.stdout:1257: done (2 left) 2023-01-15T14:33:27.280 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:27.280 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:27.281 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:27.282 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:33:27.282 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:27.275+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:27.329 INFO:tasks.rados.rados.0.smithi090.stdout:1256: finishing copy_from to smithi09025264-21 2023-01-15T14:33:27.456 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 221 (ObjNum 406 snap 108 seq_num 406) dirty exists 2023-01-15T14:33:27.458 INFO:tasks.rados.rados.0.smithi090.stdout:1256: finishing copy_from racing read to smithi09025264-21 2023-01-15T14:33:27.458 INFO:tasks.rados.rados.0.smithi090.stdout:1256: done (1 left) 2023-01-15T14:33:27.458 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:27.458 INFO:tasks.rados.rados.0.smithi090.stdout:1258: finishing write tid 2 to smithi09025264-49 2023-01-15T14:33:27.459 INFO:tasks.rados.rados.0.smithi090.stdout:1258: finishing write tid 3 to smithi09025264-49 2023-01-15T14:33:27.459 INFO:tasks.rados.rados.0.smithi090.stdout:1258: finishing write tid 4 to smithi09025264-49 2023-01-15T14:33:27.459 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:27.459 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:27.459 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:33:27.460 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:27.455+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:27.489 INFO:tasks.rados.rados.0.smithi090.stdout:1258: finishing write tid 5 to smithi09025264-49 2023-01-15T14:33:27.616 INFO:tasks.rados.rados.0.smithi090.stdout:1258: oid 49 updating version 0 to 352 2023-01-15T14:33:27.616 INFO:tasks.rados.rados.0.smithi090.stdout:1258: oid 49 updating version 352 to 354 2023-01-15T14:33:27.617 INFO:tasks.rados.rados.0.smithi090.stdout:1258: oid 49 version 354 is already newer than 353 2023-01-15T14:33:27.617 INFO:tasks.rados.rados.0.smithi090.stdout:1258: oid 49 updating version 354 to 355 2023-01-15T14:33:27.617 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 355 (ObjNum 458 snap 131 seq_num 458) dirty exists 2023-01-15T14:33:27.617 INFO:tasks.rados.rados.0.smithi090.stdout:1258: left oid 49 (ObjNum 458 snap 131 seq_num 458) 2023-01-15T14:33:27.617 INFO:tasks.rados.rados.0.smithi090.stdout:1258: done (0 left) 2023-01-15T14:33:27.625 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:27.615+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:27.719 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:27.715+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:27.829 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:27.829 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:27.830 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:33:27.831 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:27.825+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:27.882 INFO:tasks.rados.rados.0.smithi090.stdout:1260: done (0 left) 2023-01-15T14:33:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1261: read oid 11 snap 129 2023-01-15T14:33:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1261: expect (ObjNum 407 snap 108 seq_num 407) 2023-01-15T14:33:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1262: snap_remove snap 121 2023-01-15T14:33:28.027 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:28.027 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:28.028 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:33:28.030 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:28.025+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:28.129 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:28.125+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:28.235 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:28.235 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:28.236 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:33:28.236 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:28.231+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:28.336 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:28.333+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:28.435 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:28.436 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:28.436 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:33:28.438 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:28.433+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:28.539 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:28.535+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:28.761 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:33:28.763 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:28.763 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:28.763 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:33:28.764 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:28.761+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:28.864 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:28.859+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:28.942 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:28.942 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:28.942 INFO:tasks.rados.rados.0.smithi090.stdout:1262: done (1 left) 2023-01-15T14:33:28.942 INFO:tasks.rados.rados.0.smithi090.stdout:1263: write (excl) oid 24 current snap is 132 2023-01-15T14:33:28.942 INFO:tasks.rados.rados.0.smithi090.stdout:1263: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2023-01-15T14:33:28.942 INFO:tasks.rados.rados.0.smithi090.stdout:1263: writing smithi09025264-24 from 514413 to 1234884 tid 1 2023-01-15T14:33:28.942 INFO:tasks.rados.rados.0.smithi090.stdout:1263: writing smithi09025264-24 from 1876529 to 2649681 tid 2 2023-01-15T14:33:28.942 INFO:tasks.rados.rados.0.smithi090.stdout:1263: writing smithi09025264-24 from 3438683 to 3714413 tid 3 2023-01-15T14:33:28.942 INFO:tasks.rados.rados.0.smithi090.stdout:1264: delete oid 10 current snap is 132 2023-01-15T14:33:28.942 INFO:tasks.rados.rados.0.smithi090.stdout:1261: expect (ObjNum 407 snap 108 seq_num 407) 2023-01-15T14:33:28.943 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:33:28.943 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:28.944 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:28.965 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:33:29.087 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:33:29.087 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:33:29.087 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:33:29.087 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:33:29.089 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:28.987+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:29.090 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:29.090 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:29.090 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:29.090 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:29.090 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:29.085+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:29.091 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:29.091 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:29.091 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:33:29.107 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:29.144 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:29.252 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:33:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:33:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:33:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:33:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:33:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:33:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:33:29.253 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:33:29.253 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:29.249+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:29.254 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:29.254 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:33:29.254 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:33:29.254 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:33:29.254 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.254 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:75 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:29.254 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:33:29.254 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 302.84036415999998, 2023-01-15T14:33:29.254 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92335155400000002, 2023-01-15T14:33:29.255 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:29.255 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:29.255 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:29.255 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:29.255 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:29.255 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 75 2023-01-15T14:33:29.255 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.256 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:29.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:29.256 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:33:29.256 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.256 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.257 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:29.257 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:33:29.257 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.257 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.257 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.257 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:29.257 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356896+0000", 2023-01-15T14:33:29.258 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:29.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.258 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.258 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:29.258 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356903+0000", 2023-01-15T14:33:29.258 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4619999999999999e-06 2023-01-15T14:33:29.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.258 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.258 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:29.259 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356917+0000", 2023-01-15T14:33:29.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4195999999999999e-05 2023-01-15T14:33:29.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:29.259 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356926+0000", 2023-01-15T14:33:29.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:33:29.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457743+0000", 2023-01-15T14:33:29.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100817668 2023-01-15T14:33:29.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:29.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457841+0000", 2023-01-15T14:33:29.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7079999999999999e-05 2023-01-15T14:33:29.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778474+0000", 2023-01-15T14:33:29.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32063301799999999 2023-01-15T14:33:29.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:29.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778627+0000", 2023-01-15T14:33:29.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015389599999999999 2023-01-15T14:33:29.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778815+0000", 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018726600000000001 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779033+0000", 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021819999999999999 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:29.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779051+0000", 2023-01-15T14:33:29.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7640000000000001e-05 2023-01-15T14:33:29.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.410 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.410 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:29.410 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:33:29.410 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3590999999999995e-05 2023-01-15T14:33:29.410 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779163+0000", 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.081002+0000", 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30183937799999999 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.411 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:29.413 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280043+0000", 2023-01-15T14:33:29.413 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19904084699999999 2023-01-15T14:33:29.413 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280138+0000", 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4915999999999997e-05 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280248+0000", 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110233 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:29.414 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.415 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.415 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.415 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:76 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:29.415 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:33:29.415 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 302.84032255199998, 2023-01-15T14:33:29.415 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92420393899999997, 2023-01-15T14:33:29.415 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:29.415 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:29.415 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:29.415 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:29.415 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:29.416 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 76 2023-01-15T14:33:29.416 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.416 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:29.416 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.416 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:29.416 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:33:29.416 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.416 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.416 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.416 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:29.416 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:33:29.417 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.417 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.417 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.417 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:29.417 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356937+0000", 2023-01-15T14:33:29.417 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:29.417 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.417 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.417 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:29.417 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356948+0000", 2023-01-15T14:33:29.417 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.094e-05 2023-01-15T14:33:29.418 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.418 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.418 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:29.418 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356959+0000", 2023-01-15T14:33:29.418 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1058e-05 2023-01-15T14:33:29.418 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.418 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.418 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:29.418 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356968+0000", 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5339999999999999e-06 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459524+0000", 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102556147 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459662+0000", 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000137756 2023-01-15T14:33:29.419 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.420 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.420 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.420 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978621+0000", 2023-01-15T14:33:29.420 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51895887600000001 2023-01-15T14:33:29.420 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.420 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.420 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:29.420 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978870+0000", 2023-01-15T14:33:29.420 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024928999999999999 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280761+0000", 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30189113400000001 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281023+0000", 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026172800000000003 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.421 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.422 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:29.422 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281142+0000", 2023-01-15T14:33:29.422 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000119296 2023-01-15T14:33:29.422 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.422 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:29.422 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.422 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.422 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.422 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1076 3.f 3:f87c0183:::smithi09025264-50:head [setxattr _header (57) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:33:29.423 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:33:29.423 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 226.43861379699999, 2023-01-15T14:33:29.423 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.88473693200000003, 2023-01-15T14:33:29.575 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:29.575 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1076 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758646+0000", 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758653+0000", 2023-01-15T14:33:29.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8029999999999997e-06 2023-01-15T14:33:29.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:29.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758670+0000", 2023-01-15T14:33:29.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6158999999999999e-05 2023-01-15T14:33:29.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:29.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758677+0000", 2023-01-15T14:33:29.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9840000000000004e-06 2023-01-15T14:33:29.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848049+0000", 2023-01-15T14:33:29.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089372163000000004 2023-01-15T14:33:29.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:29.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848056+0000", 2023-01-15T14:33:29.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0360000000000001e-06 2023-01-15T14:33:29.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951562+0000", 2023-01-15T14:33:29.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10350670100000001 2023-01-15T14:33:29.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:29.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951726+0000", 2023-01-15T14:33:29.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000163311 2023-01-15T14:33:29.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951958+0000", 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000232525 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952168+0000", 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020949799999999999 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952186+0000", 2023-01-15T14:33:29.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7714e-05 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952254+0000", 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7998999999999997e-05 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952278+0000", 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4909000000000002e-05 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:29.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065836+0000", 2023-01-15T14:33:29.732 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.113557081 2023-01-15T14:33:29.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.732 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:29.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643196+0000", 2023-01-15T14:33:29.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.57736012999999997 2023-01-15T14:33:29.733 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:29.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643278+0000", 2023-01-15T14:33:29.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2671999999999997e-05 2023-01-15T14:33:29.733 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:29.734 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643384+0000", 2023-01-15T14:33:29.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010551400000000001 2023-01-15T14:33:29.734 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.734 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:29.734 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.740 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1077 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:33:29.740 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:33:29.744 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 226.43857497799999, 2023-01-15T14:33:29.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.98557123499999999, 2023-01-15T14:33:29.744 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:29.744 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:29.745 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:29.745 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:29.745 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:29.745 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1077 2023-01-15T14:33:29.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.745 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:29.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:29.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:33:29.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:29.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:33:29.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:29.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758685+0000", 2023-01-15T14:33:29.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:29.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:29.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758689+0000", 2023-01-15T14:33:29.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2760000000000002e-06 2023-01-15T14:33:29.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:29.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758704+0000", 2023-01-15T14:33:29.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4187e-05 2023-01-15T14:33:29.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:29.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758710+0000", 2023-01-15T14:33:29.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4330000000000001e-06 2023-01-15T14:33:29.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848086+0000", 2023-01-15T14:33:29.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089376466000000002 2023-01-15T14:33:29.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:29.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848093+0000", 2023-01-15T14:33:29.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4269999999999996e-06 2023-01-15T14:33:29.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952346+0000", 2023-01-15T14:33:29.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104252758 2023-01-15T14:33:29.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:29.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952427+0000", 2023-01-15T14:33:29.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1786000000000006e-05 2023-01-15T14:33:29.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.760 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.760 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743602+0000", 2023-01-15T14:33:29.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.79117428099999998 2023-01-15T14:33:29.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.760 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:29.760 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743892+0000", 2023-01-15T14:33:29.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029067200000000001 2023-01-15T14:33:29.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.744257+0000", 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000364619 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1087 3.f 3:fd983206:::smithi09025264-7:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 226.438540496, 2023-01-15T14:33:29.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0859964360000001, 2023-01-15T14:33:29.762 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:29.762 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:29.762 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:29.762 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:29.762 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:29.762 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1087 2023-01-15T14:33:29.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.763 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:29.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:29.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:33:29.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:29.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:33:29.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:29.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:33:29.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:29.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:29.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758724+0000", 2023-01-15T14:33:29.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7740000000000004e-06 2023-01-15T14:33:29.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:29.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758738+0000", 2023-01-15T14:33:29.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.33e-05 2023-01-15T14:33:29.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758744+0000", 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2859999999999996e-06 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848121+0000", 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089377097000000003 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848127+0000", 2023-01-15T14:33:29.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3500000000000002e-06 2023-01-15T14:33:29.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952454+0000", 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104326167 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952520+0000", 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6166000000000003e-05 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:29.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.063003+0000", 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.110482916 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065265+0000", 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022626370000000001 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065291+0000", 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6095e-05 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.066153+0000", 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086127100000000004 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166533+0000", 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100380177 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:29.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166646+0000", 2023-01-15T14:33:29.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112717 2023-01-15T14:33:29.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:29.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844498+0000", 2023-01-15T14:33:29.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67785237399999998 2023-01-15T14:33:29.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:29.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844605+0000", 2023-01-15T14:33:29.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106691 2023-01-15T14:33:29.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:29.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844717+0000", 2023-01-15T14:33:29.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011203499999999999 2023-01-15T14:33:29.900 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.900 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:29.900 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.901 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1088 3.f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:33:29.901 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:33:29.901 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 226.43848482999999, 2023-01-15T14:33:29.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.086312696, 2023-01-15T14:33:29.901 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:29.901 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:29.901 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:29.901 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:29.902 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:29.902 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1088 2023-01-15T14:33:29.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.902 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:29.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:29.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:33:29.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:29.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:33:29.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:29.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758775+0000", 2023-01-15T14:33:29.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:29.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:29.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758786+0000", 2023-01-15T14:33:29.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.076e-05 2023-01-15T14:33:29.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:29.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758809+0000", 2023-01-15T14:33:29.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3706999999999999e-05 2023-01-15T14:33:29.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758821+0000", 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1976e-05 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878534+0000", 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.119712966 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:29.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878540+0000", 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0680000000000002e-06 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952693+0000", 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.074153067000000003 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952735+0000", 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1792999999999998e-05 2023-01-15T14:33:29.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065518+0000", 2023-01-15T14:33:29.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.11278334499999999 2023-01-15T14:33:29.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:29.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065687+0000", 2023-01-15T14:33:29.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016807700000000001 2023-01-15T14:33:29.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844786+0000", 2023-01-15T14:33:29.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.77909980300000004 2023-01-15T14:33:29.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:29.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844944+0000", 2023-01-15T14:33:29.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157424 2023-01-15T14:33:29.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:29.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.845089+0000", 2023-01-15T14:33:29.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144814 2023-01-15T14:33:29.913 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.913 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:29.913 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.914 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:16 3.13 3:ca393a9e:::smithi09025264-47:head [assert-version v193,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:29.914 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:33:29.914 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 189.891507405, 2023-01-15T14:33:29.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8970339479999998, 2023-01-15T14:33:29.914 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:29.914 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:29.914 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:29.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:29.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:33:29.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:29.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:33:29.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:29.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305751+0000", 2023-01-15T14:33:29.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:29.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:29.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305763+0000", 2023-01-15T14:33:29.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2309e-05 2023-01-15T14:33:29.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:29.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305788+0000", 2023-01-15T14:33:29.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4388000000000001e-05 2023-01-15T14:33:29.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:29.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305799+0000", 2023-01-15T14:33:29.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1304000000000001e-05 2023-01-15T14:33:29.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196425+0000", 2023-01-15T14:33:29.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8906259830000001 2023-01-15T14:33:29.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:29.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196673+0000", 2023-01-15T14:33:29.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024747499999999998 2023-01-15T14:33:29.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:29.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.202787+0000", 2023-01-15T14:33:29.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0061146509999999996 2023-01-15T14:33:29.918 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.918 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:29.919 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.919 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1480 3.13 3:cbfd2d71:::smithi09025264-37:head [write 581575~705262,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:29.920 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:33:29.920 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 189.858239677, 2023-01-15T14:33:29.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.375425221, 2023-01-15T14:33:29.920 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:29.920 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:29.920 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:29.920 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:29.921 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:29.921 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1480 2023-01-15T14:33:29.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.921 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:29.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.921 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:29.921 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:33:29.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:29.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:33:29.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:29.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:29.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339019+0000", 2023-01-15T14:33:29.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:29.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.923 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:29.923 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339659+0000", 2023-01-15T14:33:29.923 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000639742 2023-01-15T14:33:29.923 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.923 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.923 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:29.923 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339690+0000", 2023-01-15T14:33:29.923 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1594000000000003e-05 2023-01-15T14:33:29.923 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.923 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.923 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:29.924 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339703+0000", 2023-01-15T14:33:29.924 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2828e-05 2023-01-15T14:33:29.924 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.924 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.924 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:29.924 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.510837+0000", 2023-01-15T14:33:29.924 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1711338759999999 2023-01-15T14:33:29.924 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.924 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.924 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:29.924 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511048+0000", 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021057899999999999 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511325+0000", 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027690400000000002 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512092+0000", 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00076720800000000002 2023-01-15T14:33:29.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:29.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512113+0000", 2023-01-15T14:33:29.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0917e-05 2023-01-15T14:33:29.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:29.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512439+0000", 2023-01-15T14:33:29.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032636299999999998 2023-01-15T14:33:29.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512871+0000", 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000432087 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.514165+0000", 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001293726 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:29.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714137+0000", 2023-01-15T14:33:29.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19997229499999999 2023-01-15T14:33:29.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:29.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714233+0000", 2023-01-15T14:33:29.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6062000000000003e-05 2023-01-15T14:33:29.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:29.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714446+0000", 2023-01-15T14:33:29.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212952 2023-01-15T14:33:29.929 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.929 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:29.929 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:29.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:29.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:29.929 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1481 3.13 3:cbfd2d71:::smithi09025264-37:head [write 1733272~707409] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:29.929 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:33:29.929 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 189.85139039399999, 2023-01-15T14:33:29.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.572522502, 2023-01-15T14:33:29.929 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:29.929 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:29.930 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:29.930 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:30.064 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:30.064 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1481 2023-01-15T14:33:30.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.064 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.064 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345868+0000", 2023-01-15T14:33:30.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:30.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346316+0000", 2023-01-15T14:33:30.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044813399999999999 2023-01-15T14:33:30.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346348+0000", 2023-01-15T14:33:30.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:33:30.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346362+0000", 2023-01-15T14:33:30.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.375e-05 2023-01-15T14:33:30.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.068 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512276+0000", 2023-01-15T14:33:30.068 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1659136829999999 2023-01-15T14:33:30.068 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.068 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.068 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:30.068 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512427+0000", 2023-01-15T14:33:30.068 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015088699999999999 2023-01-15T14:33:30.068 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.068 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.069 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714565+0000", 2023-01-15T14:33:30.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20213841599999999 2023-01-15T14:33:30.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.069 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.069 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714732+0000", 2023-01-15T14:33:30.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016676700000000001 2023-01-15T14:33:30.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.070 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:30.070 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714964+0000", 2023-01-15T14:33:30.070 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000231678 2023-01-15T14:33:30.070 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.070 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:30.070 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715499+0000", 2023-01-15T14:33:30.070 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053513300000000003 2023-01-15T14:33:30.070 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715523+0000", 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4266000000000001e-05 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715867+0000", 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034419399999999998 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:30.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.816932+0000", 2023-01-15T14:33:30.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10106498999999999 2023-01-15T14:33:30.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:30.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906374+0000", 2023-01-15T14:33:30.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089441442999999995 2023-01-15T14:33:30.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:30.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918050+0000", 2023-01-15T14:33:30.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.011676677999999999 2023-01-15T14:33:30.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.073 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.073 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:30.073 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918218+0000", 2023-01-15T14:33:30.073 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016802000000000001 2023-01-15T14:33:30.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.073 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.073 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.073 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918393+0000", 2023-01-15T14:33:30.073 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017447 2023-01-15T14:33:30.073 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.074 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.074 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.074 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.074 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.074 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1482 3.13 3:cbfd2d71:::smithi09025264-37:head [write 3195938~437497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:30.074 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:33:30.074 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 189.84495453599999, 2023-01-15T14:33:30.074 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5664780289999998, 2023-01-15T14:33:30.075 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:30.075 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:30.075 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:30.075 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:30.075 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:30.075 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1482 2023-01-15T14:33:30.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.075 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:33:30.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:33:30.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352275+0000", 2023-01-15T14:33:30.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999695 2023-01-15T14:33:30.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352644+0000", 2023-01-15T14:33:30.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036870300000000001 2023-01-15T14:33:30.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352676+0000", 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1726000000000001e-05 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352689+0000", 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3617e-05 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512602+0000", 2023-01-15T14:33:30.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1599126690000001 2023-01-15T14:33:30.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.079 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:30.079 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512839+0000", 2023-01-15T14:33:30.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023676900000000001 2023-01-15T14:33:30.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.079 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715649+0000", 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20281044600000001 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715777+0000", 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012738800000000001 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715995+0000", 2023-01-15T14:33:30.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000217858 2023-01-15T14:33:30.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:30.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716694+0000", 2023-01-15T14:33:30.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00069956700000000001 2023-01-15T14:33:30.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:30.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716709+0000", 2023-01-15T14:33:30.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4362000000000001e-05 2023-01-15T14:33:30.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:30.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716936+0000", 2023-01-15T14:33:30.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022744200000000001 2023-01-15T14:33:30.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:30.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.818876+0000", 2023-01-15T14:33:30.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101939635 2023-01-15T14:33:30.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:30.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906390+0000", 2023-01-15T14:33:30.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087513987000000001 2023-01-15T14:33:30.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:30.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918577+0000", 2023-01-15T14:33:30.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012187066 2023-01-15T14:33:30.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:30.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918697+0000", 2023-01-15T14:33:30.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120326 2023-01-15T14:33:30.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918784+0000", 2023-01-15T14:33:30.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7200999999999994e-05 2023-01-15T14:33:30.226 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.226 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.226 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.226 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1483 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3781575] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:30.226 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:33:30.226 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 189.84454638400001, 2023-01-15T14:33:30.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5663522429999999, 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1483 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352713+0000", 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352720+0000", 2023-01-15T14:33:30.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7959999999999998e-06 2023-01-15T14:33:30.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352735+0000", 2023-01-15T14:33:30.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.455e-05 2023-01-15T14:33:30.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.386 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352741+0000", 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8360000000000003e-06 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512901+0000", 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1601594980000001 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.513067+0000", 2023-01-15T14:33:30.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016626 2023-01-15T14:33:30.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716977+0000", 2023-01-15T14:33:30.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20391002799999999 2023-01-15T14:33:30.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717146+0000", 2023-01-15T14:33:30.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168851 2023-01-15T14:33:30.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:30.389 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717467+0000", 2023-01-15T14:33:30.389 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032135600000000002 2023-01-15T14:33:30.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:30.389 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717800+0000", 2023-01-15T14:33:30.389 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033217400000000002 2023-01-15T14:33:30.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.390 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:30.390 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717822+0000", 2023-01-15T14:33:30.390 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2308999999999999e-05 2023-01-15T14:33:30.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.390 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.390 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:30.390 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717970+0000", 2023-01-15T14:33:30.390 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014788099999999999 2023-01-15T14:33:30.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.390 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.390 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.819135+0000", 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116478 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906408+0000", 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087272902999999999 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918887+0000", 2023-01-15T14:33:30.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012479965000000001 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918959+0000", 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1978999999999996e-05 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.919067+0000", 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107083 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.392 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.393 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.393 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.393 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.393 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1485 3.d 3:bc1c6b44:::smithi09025264-23:head [write 598382~780511,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:30.393 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:33:30.393 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 189.83707595000001, 2023-01-15T14:33:30.393 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5474121890000001, 2023-01-15T14:33:30.393 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:30.393 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:30.393 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:30.393 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1485 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.394 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:33:30.395 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.395 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.395 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.395 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.395 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360168+0000", 2023-01-15T14:33:30.395 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999828 2023-01-15T14:33:30.395 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.395 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.395 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.395 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360790+0000", 2023-01-15T14:33:30.395 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062269099999999998 2023-01-15T14:33:30.396 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.396 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.396 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.396 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360822+0000", 2023-01-15T14:33:30.396 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1903000000000003e-05 2023-01-15T14:33:30.396 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.396 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.396 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.396 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360836+0000", 2023-01-15T14:33:30.397 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3380000000000001e-05 2023-01-15T14:33:30.397 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.397 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.397 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.397 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096664+0000", 2023-01-15T14:33:30.397 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7358282320000002 2023-01-15T14:33:30.397 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.397 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.397 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:30.397 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096683+0000", 2023-01-15T14:33:30.397 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9063999999999999e-05 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704561+0000", 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60787808399999999 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704846+0000", 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028527400000000002 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.399 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:30.399 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705135+0000", 2023-01-15T14:33:30.399 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028875400000000001 2023-01-15T14:33:30.399 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.399 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.399 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:30.399 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705735+0000", 2023-01-15T14:33:30.399 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059956899999999999 2023-01-15T14:33:30.399 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.399 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.399 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705761+0000", 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.706129+0000", 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036822299999999999 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906331+0000", 2023-01-15T14:33:30.400 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200202091 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907111+0000", 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077990799999999995 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907379+0000", 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026804800000000001 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.402 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:30.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907529+0000", 2023-01-15T14:33:30.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015022700000000001 2023-01-15T14:33:30.402 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.402 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.402 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907597+0000", 2023-01-15T14:33:30.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7785000000000002e-05 2023-01-15T14:33:30.403 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.403 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.403 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.403 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.403 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.403 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1486 3.d 3:bc1c6b44:::smithi09025264-23:head [write 2075401~451067] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:30.403 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:33:30.403 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 189.83067990199999, 2023-01-15T14:33:30.403 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6461318330000001, 2023-01-15T14:33:30.403 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:30.403 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:30.404 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:30.404 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:30.404 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:30.404 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1486 2023-01-15T14:33:30.404 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.404 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.404 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.404 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.404 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:33:30.404 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.404 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.405 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.405 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.405 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:33:30.405 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.405 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.405 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.406 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366579+0000", 2023-01-15T14:33:30.406 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:30.406 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.406 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.406 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366845+0000", 2023-01-15T14:33:30.406 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026610499999999999 2023-01-15T14:33:30.406 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.406 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.407 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.407 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366878+0000", 2023-01-15T14:33:30.407 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2750000000000003e-05 2023-01-15T14:33:30.407 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.407 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.407 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.467009+0000", 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100131071 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096793+0000", 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6297846869999999 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096806+0000", 2023-01-15T14:33:30.408 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3077e-05 2023-01-15T14:33:30.409 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.409 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.409 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.409 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907957+0000", 2023-01-15T14:33:30.409 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81115055400000002 2023-01-15T14:33:30.409 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.409 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.409 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.409 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908229+0000", 2023-01-15T14:33:30.409 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027187499999999998 2023-01-15T14:33:30.409 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908496+0000", 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026754200000000001 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909025+0000", 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052865200000000001 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.410 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909046+0000", 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0713e-05 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909294+0000", 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024864899999999998 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.911607+0000", 2023-01-15T14:33:30.411 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023128939999999998 2023-01-15T14:33:30.412 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.412 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.412 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:30.412 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012076+0000", 2023-01-15T14:33:30.412 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100468293 2023-01-15T14:33:30.412 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.412 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.412 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:30.412 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012426+0000", 2023-01-15T14:33:30.412 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035044299999999998 2023-01-15T14:33:30.413 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.413 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.413 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:30.413 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012532+0000", 2023-01-15T14:33:30.413 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106142 2023-01-15T14:33:30.413 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.413 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.413 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.413 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012713+0000", 2023-01-15T14:33:30.413 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018048599999999999 2023-01-15T14:33:30.413 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1496 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 189.46103705300001, 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870616037, 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:30.414 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:30.415 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:30.415 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1496 2023-01-15T14:33:30.415 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:33:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:33:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736220+0000", 2023-01-15T14:33:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736238+0000", 2023-01-15T14:33:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7156e-05 2023-01-15T14:33:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736275+0000", 2023-01-15T14:33:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7876000000000001e-05 2023-01-15T14:33:30.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736290+0000", 2023-01-15T14:33:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4449e-05 2023-01-15T14:33:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602217+0000", 2023-01-15T14:33:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8659270299999999 2023-01-15T14:33:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602468+0000", 2023-01-15T14:33:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000251315 2023-01-15T14:33:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.606840+0000", 2023-01-15T14:33:30.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0043715009999999999 2023-01-15T14:33:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1497 3.11 3:8889e450:::smithi09025264-32:head [read 0~2991646,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:33:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 189.460949599, 2023-01-15T14:33:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738359249999998, 2023-01-15T14:33:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1497 2023-01-15T14:33:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:33:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:33:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736310+0000", 2023-01-15T14:33:30.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736317+0000", 2023-01-15T14:33:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7240000000000003e-06 2023-01-15T14:33:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736334+0000", 2023-01-15T14:33:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6878999999999999e-05 2023-01-15T14:33:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736341+0000", 2023-01-15T14:33:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0129999999999996e-06 2023-01-15T14:33:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607070+0000", 2023-01-15T14:33:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870728873 2023-01-15T14:33:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607226+0000", 2023-01-15T14:33:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015595 2023-01-15T14:33:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610147+0000", 2023-01-15T14:33:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.002921416 2023-01-15T14:33:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1498 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:33:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 189.460909493, 2023-01-15T14:33:30.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8752667839999999, 2023-01-15T14:33:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1498 2023-01-15T14:33:30.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:33:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:33:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736350+0000", 2023-01-15T14:33:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736355+0000", 2023-01-15T14:33:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4900000000000002e-06 2023-01-15T14:33:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736363+0000", 2023-01-15T14:33:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2770000000000005e-06 2023-01-15T14:33:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736369+0000", 2023-01-15T14:33:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7400000000000001e-06 2023-01-15T14:33:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610234+0000", 2023-01-15T14:33:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738651439999998 2023-01-15T14:33:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610402+0000", 2023-01-15T14:33:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168283 2023-01-15T14:33:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.611618+0000", 2023-01-15T14:33:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001215647 2023-01-15T14:33:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.7.0:440 3.11 e231/207 ondisk, result = 0)", 2023-01-15T14:33:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:33:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 90.953127835000004, 2023-01-15T14:33:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973962139999998, 2023-01-15T14:33:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:33:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:33:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244132+0000", 2023-01-15T14:33:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244143+0000", 2023-01-15T14:33:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1647e-05 2023-01-15T14:33:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244165+0000", 2023-01-15T14:33:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1501999999999998e-05 2023-01-15T14:33:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244180+0000", 2023-01-15T14:33:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5434000000000002e-05 2023-01-15T14:33:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941489+0000", 2023-01-15T14:33:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973091899999999 2023-01-15T14:33:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941503+0000", 2023-01-15T14:33:30.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.414e-05 2023-01-15T14:33:30.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941529+0000", 2023-01-15T14:33:30.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5622999999999998e-05 2023-01-15T14:33:30.584 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.584 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.584 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.584 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.4.0:562 3.1b e231/224 3:d85f42d9:::smithi09025264-45:a8 v 231'66, mlcod=225'62)", 2023-01-15T14:33:30.584 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:33:30.584 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 90.762776638999995, 2023-01-15T14:33:30.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2578820149999999, 2023-01-15T14:33:30.585 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:30.585 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:30.585 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:33:30.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:33:30.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:33:30.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:30.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434492+0000", 2023-01-15T14:33:30.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4360000000000002e-06 2023-01-15T14:33:30.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434517+0000", 2023-01-15T14:33:30.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.48e-05 2023-01-15T14:33:30.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434530+0000", 2023-01-15T14:33:30.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2982999999999999e-05 2023-01-15T14:33:30.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434663+0000", 2023-01-15T14:33:30.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013307 2023-01-15T14:33:30.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434700+0000", 2023-01-15T14:33:30.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7020000000000001e-05 2023-01-15T14:33:30.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:30.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435438+0000", 2023-01-15T14:33:30.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073808900000000004 2023-01-15T14:33:30.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:30.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435456+0000", 2023-01-15T14:33:30.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8091999999999999e-05 2023-01-15T14:33:30.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:30.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435526+0000", 2023-01-15T14:33:30.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9806000000000002e-05 2023-01-15T14:33:30.594 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:30.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435570+0000", 2023-01-15T14:33:30.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4091999999999997e-05 2023-01-15T14:33:30.594 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.595 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.692366+0000", 2023-01-15T14:33:30.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2567961539999999 2023-01-15T14:33:30.595 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.595 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.596 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.705 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.705 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:24 3.d 3:b3f111b5:::smithi09025264-48:head [assert-version v160,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e232)", 2023-01-15T14:33:30.705 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:33:30.705 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 90.246940422999998, 2023-01-15T14:33:30.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9963031440000001, 2023-01-15T14:33:30.705 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:30.705 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:30.705 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.706 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:33:30.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:33:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950318+0000", 2023-01-15T14:33:30.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:30.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950333+0000", 2023-01-15T14:33:30.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.467e-05 2023-01-15T14:33:30.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950368+0000", 2023-01-15T14:33:30.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4895999999999999e-05 2023-01-15T14:33:30.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050461+0000", 2023-01-15T14:33:30.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10009304400000001 2023-01-15T14:33:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050593+0000", 2023-01-15T14:33:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000132764 2023-01-15T14:33:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:33:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050615+0000", 2023-01-15T14:33:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1483000000000001e-05 2023-01-15T14:33:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945049+0000", 2023-01-15T14:33:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.894434138 2023-01-15T14:33:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945326+0000", 2023-01-15T14:33:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000277368 2023-01-15T14:33:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.946623+0000", 2023-01-15T14:33:30.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012970340000000001 2023-01-15T14:33:30.718 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.718 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.718 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.718 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.3.0:547 3.22 e232/180 3:47dfe935:::smithi09025264-29:81 v 232'124, mlcod=229'121)", 2023-01-15T14:33:30.718 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:33:30.718 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 90.146710674999994, 2023-01-15T14:33:30.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.643153431, 2023-01-15T14:33:30.719 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:30.719 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:30.719 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:30.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:30.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:33:30.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:30.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:33:30.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:30.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:30.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050549+0000", 2023-01-15T14:33:30.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:30.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:30.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050561+0000", 2023-01-15T14:33:30.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.278e-05 2023-01-15T14:33:30.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:30.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050583+0000", 2023-01-15T14:33:30.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1053e-05 2023-01-15T14:33:30.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:30.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050596+0000", 2023-01-15T14:33:30.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3946e-05 2023-01-15T14:33:30.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:30.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050708+0000", 2023-01-15T14:33:30.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011129 2023-01-15T14:33:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050750+0000", 2023-01-15T14:33:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2725000000000002e-05 2023-01-15T14:33:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051324+0000", 2023-01-15T14:33:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00057335700000000003 2023-01-15T14:33:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051340+0000", 2023-01-15T14:33:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5968999999999999e-05 2023-01-15T14:33:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051405+0000", 2023-01-15T14:33:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5419000000000005e-05 2023-01-15T14:33:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051471+0000", 2023-01-15T14:33:30.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5671000000000002e-05 2023-01-15T14:33:30.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:30.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:30.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:30.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.693703+0000", 2023-01-15T14:33:30.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.642232554 2023-01-15T14:33:30.727 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.728 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.728 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.728 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:30.728 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:30.728 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:30.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:33:30.730 INFO:tasks.rados.rados.0.smithi090.stdout:1263: finishing write tid 1 to smithi09025264-24 2023-01-15T14:33:30.734 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:33:31.083 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:33:31.084 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2023-01-15T14:33:28.936348+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":234248501,"num_objects":104,"num_object_clones":56,"num_object_copies":312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":7,"num_read":862,"num_read_kb":758724,"num_write":787,"num_write_kb":257783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":168708470,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14891,"ondisk_log_size":14891,"up":285,"acting":284,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":294,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":5790064,"kb_used_data":556489540,"kb_used_omap":54554,"kb_used_meta":0,"kb_avail":556406216,"statfs":{"total":575688990720,"available":569759965184,"internally_reserved":85323776,"allocated":569845288960,"data_stored":569845288960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":55863320,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4,6,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":26,"commit_latency_ns":0,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24355923,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-15,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":40729,"num_write":42,"num_write_kb":27018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":18740654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001134"},"pg_stats":[{"pgid":"3.55","version":"110'55","reported_seq":923,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.976623+0000","last_change":"2023-01-15T14:33:26.977619+0000","last_active":"2023-01-15T14:33:26.976623+0000","last_peered":"2023-01-15T14:33:26.976623+0000","last_clean":"2023-01-15T14:33:26.976623+0000","last_became_active":"2023-01-15T14:33:22.073078+0000","last_became_peered":"2023-01-15T14:33:22.073078+0000","last_unstale":"2023-01-15T14:33:26.976623+0000","last_undegraded":"2023-01-15T14:33:26.976623+0000","last_fullsized":"2023-01-15T14:33:26.976623+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.102830976,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1111,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.982747+0000","last_change":"2023-01-15T14:33:26.982686+0000","last_active":"2023-01-15T14:33:26.982747+0000","last_peered":"2023-01-15T14:33:26.982747+0000","last_clean":"2023-01-15T14:33:26.982747+0000","last_became_active":"2023-01-15T14:33:25.198058+0000","last_became_peered":"2023-01-15T14:33:25.198058+0000","last_unstale":"2023-01-15T14:33:26.982747+0000","last_undegraded":"2023-01-15T14:33:26.982747+0000","last_fullsized":"2023-01-15T14:33:26.982747+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:32:19.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:19.303243+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.107738234,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.53","version":"308'359","reported_seq":1259,"reported_epoch":318,"state":"peering","last_fresh":"2023-01-15T14:33:26.879909+0000","last_change":"2023-01-15T14:33:26.879909+0000","last_active":"2023-01-15T14:33:21.876367+0000","last_peered":"2023-01-15T14:33:16.979107+0000","last_clean":"2023-01-15T14:33:16.979107+0000","last_became_active":"2023-01-15T14:31:35.097414+0000","last_became_peered":"2023-01-15T14:31:35.097414+0000","last_unstale":"2023-01-15T14:33:26.879909+0000","last_undegraded":"2023-01-15T14:33:26.879909+0000","last_fullsized":"2023-01-15T14:33:26.879909+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":7,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6271685,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":24051,"num_write":13,"num_write_kb":7496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.52","version":"259'164","reported_seq":633,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.077275+0000","last_change":"2023-01-15T14:33:27.077275+0000","last_active":"2023-01-15T14:33:27.077275+0000","last_peered":"2023-01-15T14:33:27.077275+0000","last_clean":"2023-01-15T14:33:27.077275+0000","last_became_active":"2023-01-15T14:33:25.099467+0000","last_became_peered":"2023-01-15T14:33:25.099467+0000","last_unstale":"2023-01-15T14:33:27.077275+0000","last_undegraded":"2023-01-15T14:33:27.077275+0000","last_fullsized":"2023-01-15T14:33:27.077275+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.202065143,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.51","version":"318'294","reported_seq":1410,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.178322+0000","last_change":"2023-01-15T14:33:26.880077+0000","last_active":"2023-01-15T14:33:27.178322+0000","last_peered":"2023-01-15T14:33:27.178322+0000","last_clean":"2023-01-15T14:33:27.178322+0000","last_became_active":"2023-01-15T14:33:26.107940+0000","last_became_peered":"2023-01-15T14:33:26.107940+0000","last_unstale":"2023-01-15T14:33:27.178322+0000","last_undegraded":"2023-01-15T14:33:27.178322+0000","last_fullsized":"2023-01-15T14:33:27.178322+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":294,"log_dups_size":0,"ondisk_log_size":294,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.0010649259999999999,"stat_sum":{"num_bytes":8042981,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":23757,"num_write":25,"num_write_kb":10113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6655737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.50","version":"311'408","reported_seq":1477,"reported_epoch":317,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:33:26.802947+0000","last_change":"2023-01-15T14:33:25.906305+0000","last_active":"2023-01-15T14:33:26.802947+0000","last_peered":"2023-01-15T14:33:26.802947+0000","last_clean":"2023-01-15T14:33:20.551031+0000","last_became_active":"2023-01-15T14:33:23.089487+0000","last_became_peered":"2023-01-15T14:33:23.089487+0000","last_unstale":"2023-01-15T14:33:26.802947+0000","last_undegraded":"2023-01-15T14:33:26.802947+0000","last_fullsized":"2023-01-15T14:33:22.876964+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":42160,"num_write":47,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":13747086,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[1,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2},{"shards":"1,2,3,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f1","length":"1"}]},{"pgid":"3.4f","version":"155'145","reported_seq":746,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.979628+0000","last_change":"2023-01-15T14:33:26.979628+0000","last_active":"2023-01-15T14:33:26.979628+0000","last_peered":"2023-01-15T14:33:26.979628+0000","last_clean":"2023-01-15T14:33:26.979628+0000","last_became_active":"2023-01-15T14:33:25.129936+0000","last_became_peered":"2023-01-15T14:33:25.129936+0000","last_unstale":"2023-01-15T14:33:26.979628+0000","last_undegraded":"2023-01-15T14:33:26.979628+0000","last_fullsized":"2023-01-15T14:33:26.979628+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":15,"reported_epoch":316,"state":"active+clean","last_fresh":"2023-01-15T14:33:24.887989+0000","last_change":"2023-01-15T14:33:21.981323+0000","last_active":"2023-01-15T14:33:24.887989+0000","last_peered":"2023-01-15T14:33:24.887989+0000","last_clean":"2023-01-15T14:33:24.887989+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:33:24.887989+0000","last_undegraded":"2023-01-15T14:33:24.887989+0000","last_fullsized":"2023-01-15T14:33:24.887989+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.4d","version":"318'236","reported_seq":1232,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.729678+0000","last_change":"2023-01-15T14:33:27.729678+0000","last_active":"2023-01-15T14:33:27.729678+0000","last_peered":"2023-01-15T14:33:27.729678+0000","last_clean":"2023-01-15T14:33:27.729678+0000","last_became_active":"2023-01-15T14:33:26.102614+0000","last_became_peered":"2023-01-15T14:33:26.102614+0000","last_unstale":"2023-01-15T14:33:27.729678+0000","last_undegraded":"2023-01-15T14:33:27.729678+0000","last_fullsized":"2023-01-15T14:33:27.729678+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":1,"snaptrim_duration":0.84886855699999997,"stat_sum":{"num_bytes":6258391,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":5940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11291942,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.4c","version":"309'71","reported_seq":20,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.484571+0000","last_change":"2023-01-15T14:33:27.485404+0000","last_active":"2023-01-15T14:33:27.484571+0000","last_peered":"2023-01-15T14:33:27.484571+0000","last_clean":"2023-01-15T14:33:27.484571+0000","last_became_active":"2023-01-15T14:33:22.183235+0000","last_became_peered":"2023-01-15T14:33:22.183235+0000","last_unstale":"2023-01-15T14:33:27.484571+0000","last_undegraded":"2023-01-15T14:33:27.484571+0000","last_fullsized":"2023-01-15T14:33:27.484571+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.60935307800000005,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":666,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.882029+0000","last_change":"2023-01-15T14:33:27.186022+0000","last_active":"2023-01-15T14:33:26.882029+0000","last_peered":"2023-01-15T14:33:26.882029+0000","last_clean":"2023-01-15T14:33:26.882029+0000","last_became_active":"2023-01-15T14:32:25.161408+0000","last_became_peered":"2023-01-15T14:32:25.161408+0000","last_unstale":"2023-01-15T14:33:26.882029+0000","last_undegraded":"2023-01-15T14:33:26.882029+0000","last_fullsized":"2023-01-15T14:33:26.882029+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:40.131","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30420197900000001,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.4a","version":"318'336","reported_seq":1360,"reported_epoch":318,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:27.178460+0000","last_change":"2023-01-15T14:33:27.178460+0000","last_active":"2023-01-15T14:33:27.178460+0000","last_peered":"2023-01-15T14:33:27.178460+0000","last_clean":"2023-01-15T14:33:27.178460+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:33:27.178460+0000","last_undegraded":"2023-01-15T14:33:27.178460+0000","last_fullsized":"2023-01-15T14:33:27.178460+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:33.185","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012949389999999999,"stat_sum":{"num_bytes":9507389,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":34135,"num_write":58,"num_write_kb":19446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":1058,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.084932+0000","last_change":"2023-01-15T14:33:27.084932+0000","last_active":"2023-01-15T14:33:27.084932+0000","last_peered":"2023-01-15T14:33:27.084932+0000","last_clean":"2023-01-15T14:33:27.084932+0000","last_became_active":"2023-01-15T14:32:28.259491+0000","last_became_peered":"2023-01-15T14:32:28.259491+0000","last_unstale":"2023-01-15T14:33:27.084932+0000","last_undegraded":"2023-01-15T14:33:27.084932+0000","last_fullsized":"2023-01-15T14:33:27.084932+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:22.332","scrub_duration":0.47056024400000002,"objects_trimmed":0,"snaptrim_duration":0.20445147399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.48","version":"179'204","reported_seq":1251,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.874040+0000","last_change":"2023-01-15T14:33:26.979890+0000","last_active":"2023-01-15T14:33:26.874040+0000","last_peered":"2023-01-15T14:33:26.874040+0000","last_clean":"2023-01-15T14:33:26.874040+0000","last_became_active":"2023-01-15T14:32:28.187902+0000","last_became_peered":"2023-01-15T14:32:28.187902+0000","last_unstale":"2023-01-15T14:33:26.874040+0000","last_undegraded":"2023-01-15T14:33:26.874040+0000","last_fullsized":"2023-01-15T14:33:26.874040+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.106259668,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1042,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.730617+0000","last_change":"2023-01-15T14:33:27.730617+0000","last_active":"2023-01-15T14:33:27.730617+0000","last_peered":"2023-01-15T14:33:27.730617+0000","last_clean":"2023-01-15T14:33:27.730617+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:33:27.730617+0000","last_undegraded":"2023-01-15T14:33:27.730617+0000","last_fullsized":"2023-01-15T14:33:27.730617+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:03.200","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.84954518300000004,"stat_sum":{"num_bytes":496762,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":5,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":941,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.290548+0000","last_change":"2023-01-15T14:33:27.291400+0000","last_active":"2023-01-15T14:33:27.290548+0000","last_peered":"2023-01-15T14:33:27.290548+0000","last_clean":"2023-01-15T14:33:27.290548+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:33:27.290548+0000","last_undegraded":"2023-01-15T14:33:27.290548+0000","last_fullsized":"2023-01-15T14:33:27.290548+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:31:25.342830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:25.342830+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:32.011","scrub_duration":0.20886623600000001,"objects_trimmed":0,"snaptrim_duration":0.30708341900000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":1052,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.077449+0000","last_change":"2023-01-15T14:33:27.178321+0000","last_active":"2023-01-15T14:33:27.077449+0000","last_peered":"2023-01-15T14:33:27.077449+0000","last_clean":"2023-01-15T14:33:27.077449+0000","last_became_active":"2023-01-15T14:32:28.162971+0000","last_became_peered":"2023-01-15T14:32:28.162971+0000","last_unstale":"2023-01-15T14:33:27.077449+0000","last_undegraded":"2023-01-15T14:33:27.077449+0000","last_fullsized":"2023-01-15T14:33:27.077449+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:50.736","scrub_duration":0.206392353,"objects_trimmed":0,"snaptrim_duration":0.30262687300000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1174,"reported_epoch":319,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:28.081730+0000","last_change":"2023-01-15T14:33:28.081326+0000","last_active":"2023-01-15T14:33:28.081730+0000","last_peered":"2023-01-15T14:33:28.081730+0000","last_clean":"2023-01-15T14:33:28.081730+0000","last_became_active":"2023-01-15T14:33:28.080044+0000","last_became_peered":"2023-01-15T14:33:28.080044+0000","last_unstale":"2023-01-15T14:33:28.081730+0000","last_undegraded":"2023-01-15T14:33:28.081730+0000","last_fullsized":"2023-01-15T14:33:28.081730+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:42.130","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.0079948250000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.43","version":"297'314","reported_seq":1385,"reported_epoch":316,"state":"active+clean","last_fresh":"2023-01-15T14:33:24.979538+0000","last_change":"2023-01-15T14:33:19.130862+0000","last_active":"2023-01-15T14:33:24.979538+0000","last_peered":"2023-01-15T14:33:24.979538+0000","last_clean":"2023-01-15T14:33:24.979538+0000","last_became_active":"2023-01-15T14:32:28.977084+0000","last_became_peered":"2023-01-15T14:32:28.977084+0000","last_unstale":"2023-01-15T14:33:24.979538+0000","last_undegraded":"2023-01-15T14:33:24.979538+0000","last_fullsized":"2023-01-15T14:33:24.979538+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:02.589","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.0068675869999999997,"stat_sum":{"num_bytes":3395080,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.42","version":"155'101","reported_seq":934,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.974710+0000","last_change":"2023-01-15T14:33:26.974710+0000","last_active":"2023-01-15T14:33:26.974710+0000","last_peered":"2023-01-15T14:33:26.974710+0000","last_clean":"2023-01-15T14:33:26.974710+0000","last_became_active":"2023-01-15T14:32:25.163911+0000","last_became_peered":"2023-01-15T14:32:25.163911+0000","last_unstale":"2023-01-15T14:33:26.974710+0000","last_undegraded":"2023-01-15T14:33:26.974710+0000","last_fullsized":"2023-01-15T14:33:26.974710+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.109879875,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1015,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.980036+0000","last_change":"2023-01-15T14:33:26.979973+0000","last_active":"2023-01-15T14:33:26.980036+0000","last_peered":"2023-01-15T14:33:26.980036+0000","last_clean":"2023-01-15T14:33:26.980036+0000","last_became_active":"2023-01-15T14:31:56.927677+0000","last_became_peered":"2023-01-15T14:31:56.927677+0000","last_unstale":"2023-01-15T14:33:26.980036+0000","last_undegraded":"2023-01-15T14:33:26.980036+0000","last_fullsized":"2023-01-15T14:33:26.980036+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:56.100","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.0033798349999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.40","version":"318'295","reported_seq":1323,"reported_epoch":318,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:27.187775+0000","last_change":"2023-01-15T14:33:27.187775+0000","last_active":"2023-01-15T14:33:27.187775+0000","last_peered":"2023-01-15T14:33:27.187775+0000","last_clean":"2023-01-15T14:33:27.187775+0000","last_became_active":"2023-01-15T14:33:14.198194+0000","last_became_peered":"2023-01-15T14:33:14.198194+0000","last_unstale":"2023-01-15T14:33:27.187775+0000","last_undegraded":"2023-01-15T14:33:27.187775+0000","last_fullsized":"2023-01-15T14:33:27.187775+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.101765574,"stat_sum":{"num_bytes":5896549,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":21945,"num_write":25,"num_write_kb":7603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":16413587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.3f","version":"310'229","reported_seq":1239,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.877794+0000","last_change":"2023-01-15T14:33:26.877794+0000","last_active":"2023-01-15T14:33:26.877794+0000","last_peered":"2023-01-15T14:33:26.877794+0000","last_clean":"2023-01-15T14:33:26.877794+0000","last_became_active":"2023-01-15T14:32:02.943089+0000","last_became_peered":"2023-01-15T14:32:02.943089+0000","last_unstale":"2023-01-15T14:33:26.877794+0000","last_undegraded":"2023-01-15T14:33:26.877794+0000","last_fullsized":"2023-01-15T14:33:26.877794+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":229,"log_dups_size":0,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:18.074","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.821245428,"stat_sum":{"num_bytes":4764997,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.3e","version":"294'63","reported_seq":756,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.884348+0000","last_change":"2023-01-15T14:33:26.982238+0000","last_active":"2023-01-15T14:33:26.884348+0000","last_peered":"2023-01-15T14:33:26.884348+0000","last_clean":"2023-01-15T14:33:26.884348+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:33:26.884348+0000","last_undegraded":"2023-01-15T14:33:26.884348+0000","last_fullsized":"2023-01-15T14:33:26.884348+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:42.278","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.098874265000000003,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.3d","version":"283'223","reported_seq":1239,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.983194+0000","last_change":"2023-01-15T14:33:26.983194+0000","last_active":"2023-01-15T14:33:26.983194+0000","last_peered":"2023-01-15T14:33:26.983194+0000","last_clean":"2023-01-15T14:33:26.983194+0000","last_became_active":"2023-01-15T14:31:59.249241+0000","last_became_peered":"2023-01-15T14:31:59.249241+0000","last_unstale":"2023-01-15T14:33:26.983194+0000","last_undegraded":"2023-01-15T14:33:26.983194+0000","last_fullsized":"2023-01-15T14:33:26.983194+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.099840440000000003,"stat_sum":{"num_bytes":5496021,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":13796,"num_write":10,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.3c","version":"141'46","reported_seq":271,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.874893+0000","last_change":"2023-01-15T14:33:26.976383+0000","last_active":"2023-01-15T14:33:26.874893+0000","last_peered":"2023-01-15T14:33:26.874893+0000","last_clean":"2023-01-15T14:33:26.874893+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:33:26.874893+0000","last_undegraded":"2023-01-15T14:33:26.874893+0000","last_fullsized":"2023-01-15T14:33:26.874893+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.470","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.10182548700000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":917,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.086755+0000","last_change":"2023-01-15T14:33:27.086755+0000","last_active":"2023-01-15T14:33:27.086755+0000","last_peered":"2023-01-15T14:33:27.086755+0000","last_clean":"2023-01-15T14:33:27.086755+0000","last_became_active":"2023-01-15T14:31:54.237985+0000","last_became_peered":"2023-01-15T14:31:54.237985+0000","last_unstale":"2023-01-15T14:33:27.086755+0000","last_undegraded":"2023-01-15T14:33:27.086755+0000","last_fullsized":"2023-01-15T14:33:27.086755+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:52.139","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.20621877799999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1260,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.088692+0000","last_change":"2023-01-15T14:33:27.126699+0000","last_active":"2023-01-15T14:33:27.088692+0000","last_peered":"2023-01-15T14:33:27.088692+0000","last_clean":"2023-01-15T14:33:27.088692+0000","last_became_active":"2023-01-15T14:31:57.025706+0000","last_became_peered":"2023-01-15T14:31:57.025706+0000","last_unstale":"2023-01-15T14:33:27.088692+0000","last_undegraded":"2023-01-15T14:33:27.088692+0000","last_fullsized":"2023-01-15T14:33:27.088692+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:30.639","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.24280253099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.39","version":"318'330","reported_seq":1296,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.483324+0000","last_change":"2023-01-15T14:33:27.484436+0000","last_active":"2023-01-15T14:33:27.483324+0000","last_peered":"2023-01-15T14:33:27.483324+0000","last_clean":"2023-01-15T14:33:27.483324+0000","last_became_active":"2023-01-15T14:32:06.420117+0000","last_became_peered":"2023-01-15T14:32:06.420117+0000","last_unstale":"2023-01-15T14:33:27.483324+0000","last_undegraded":"2023-01-15T14:33:27.483324+0000","last_fullsized":"2023-01-15T14:33:27.483324+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":330,"log_dups_size":0,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.413","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.60935607199999997,"stat_sum":{"num_bytes":19058088,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":10,"num_read_kb":18059,"num_write":1,"num_write_kb":2758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.38","version":"318'299","reported_seq":1429,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:27.387846+0000","last_change":"2023-01-15T14:33:26.974370+0000","last_active":"2023-01-15T14:33:27.387846+0000","last_peered":"2023-01-15T14:33:27.387846+0000","last_clean":"2023-01-15T14:33:27.387846+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:33:27.387846+0000","last_undegraded":"2023-01-15T14:33:27.387846+0000","last_fullsized":"2023-01-15T14:33:27.387846+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":299,"log_dups_size":0,"ondisk_log_size":299,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:55.610","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.115015853,"stat_sum":{"num_bytes":7580089,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":18941,"num_write":33,"num_write_kb":12593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":320,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:27.078299+0000","last_change":"2023-01-15T14:33:27.078299+0000","last_active":"2023-01-15T14:33:27.078299+0000","last_peered":"2023-01-15T14:33:27.078299+0000","last_clean":"2023-01-15T14:33:27.078299+0000","last_became_active":"2023-01-15T14:31:37.024453+0000","last_became_peered":"2023-01-15T14:31:37.024453+0000","last_unstale":"2023-01-15T14:33:27.078299+0000","last_undegraded":"2023-01-15T14:33:27.078299+0000","last_fullsized":"2023-01-15T14:33:27.078299+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.11813478500000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":484,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.289202+0000","last_change":"2023-01-15T14:33:27.290342+0000","last_active":"2023-01-15T14:33:27.289202+0000","last_peered":"2023-01-15T14:33:27.289202+0000","last_clean":"2023-01-15T14:33:27.289202+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:33:27.289202+0000","last_undegraded":"2023-01-15T14:33:27.289202+0000","last_fullsized":"2023-01-15T14:33:27.289202+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.30931808599999999,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":912,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.979668+0000","last_change":"2023-01-15T14:33:26.981102+0000","last_active":"2023-01-15T14:33:26.979668+0000","last_peered":"2023-01-15T14:33:26.979668+0000","last_clean":"2023-01-15T14:33:26.979668+0000","last_became_active":"2023-01-15T14:31:38.033824+0000","last_became_peered":"2023-01-15T14:31:38.033824+0000","last_unstale":"2023-01-15T14:33:26.979668+0000","last_undegraded":"2023-01-15T14:33:26.979668+0000","last_fullsized":"2023-01-15T14:33:26.979668+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:12.355","scrub_duration":0.282936307,"objects_trimmed":0,"snaptrim_duration":0.10556074999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.34","version":"169'138","reported_seq":1078,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.983196+0000","last_change":"2023-01-15T14:33:26.983142+0000","last_active":"2023-01-15T14:33:26.983196+0000","last_peered":"2023-01-15T14:33:26.983196+0000","last_clean":"2023-01-15T14:33:26.983196+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:33:26.983196+0000","last_undegraded":"2023-01-15T14:33:26.983196+0000","last_fullsized":"2023-01-15T14:33:26.983196+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:59.872","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.10430138899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1154,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.977814+0000","last_change":"2023-01-15T14:33:26.979354+0000","last_active":"2023-01-15T14:33:26.977814+0000","last_peered":"2023-01-15T14:33:26.977814+0000","last_clean":"2023-01-15T14:33:26.977814+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:33:26.977814+0000","last_undegraded":"2023-01-15T14:33:26.977814+0000","last_fullsized":"2023-01-15T14:33:26.977814+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:54.578","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.104016835,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.32","version":"243'157","reported_seq":756,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.983311+0000","last_change":"2023-01-15T14:33:27.085949+0000","last_active":"2023-01-15T14:33:26.983311+0000","last_peered":"2023-01-15T14:33:26.983311+0000","last_clean":"2023-01-15T14:33:26.983311+0000","last_became_active":"2023-01-15T14:31:36.921952+0000","last_became_peered":"2023-01-15T14:31:36.921952+0000","last_unstale":"2023-01-15T14:33:26.983311+0000","last_undegraded":"2023-01-15T14:33:26.983311+0000","last_fullsized":"2023-01-15T14:33:26.983311+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.20246330300000001,"stat_sum":{"num_bytes":2403401,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.13","version":"311'361","reported_seq":1283,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.878428+0000","last_change":"2023-01-15T14:33:26.878428+0000","last_active":"2023-01-15T14:33:26.878428+0000","last_peered":"2023-01-15T14:33:26.878428+0000","last_clean":"2023-01-15T14:33:26.878428+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:33:26.878428+0000","last_undegraded":"2023-01-15T14:33:26.878428+0000","last_fullsized":"2023-01-15T14:33:26.878428+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":361,"log_dups_size":0,"ondisk_log_size":361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.605536984,"stat_sum":{"num_bytes":6271684,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":24050,"num_write":12,"num_write_kb":7495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.12","version":"315'173","reported_seq":771,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.975215+0000","last_change":"2023-01-15T14:33:26.975186+0000","last_active":"2023-01-15T14:33:26.975215+0000","last_peered":"2023-01-15T14:33:26.975215+0000","last_clean":"2023-01-15T14:33:26.975215+0000","last_became_active":"2023-01-15T14:33:20.335242+0000","last_became_peered":"2023-01-15T14:33:20.335242+0000","last_unstale":"2023-01-15T14:33:26.975215+0000","last_undegraded":"2023-01-15T14:33:26.975215+0000","last_fullsized":"2023-01-15T14:33:26.975215+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.106126703,"stat_sum":{"num_bytes":5563828,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":10,"num_write_kb":3574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.11","version":"317'291","reported_seq":1399,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.877285+0000","last_change":"2023-01-15T14:33:26.877285+0000","last_active":"2023-01-15T14:33:26.877285+0000","last_peered":"2023-01-15T14:33:26.877285+0000","last_clean":"2023-01-15T14:33:26.877285+0000","last_became_active":"2023-01-15T14:33:20.234205+0000","last_became_peered":"2023-01-15T14:33:20.234205+0000","last_unstale":"2023-01-15T14:33:26.877285+0000","last_undegraded":"2023-01-15T14:33:26.877285+0000","last_fullsized":"2023-01-15T14:33:26.877285+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.10209615,"stat_sum":{"num_bytes":8512943,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":31,"num_read_kb":34659,"num_write":21,"num_write_kb":10956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6655737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.10","version":"308'406","reported_seq":1410,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.974958+0000","last_change":"2023-01-15T14:33:26.974958+0000","last_active":"2023-01-15T14:33:26.974958+0000","last_peered":"2023-01-15T14:33:26.974958+0000","last_clean":"2023-01-15T14:33:26.974958+0000","last_became_active":"2023-01-15T14:33:20.335088+0000","last_became_peered":"2023-01-15T14:33:20.335088+0000","last_unstale":"2023-01-15T14:33:26.974958+0000","last_undegraded":"2023-01-15T14:33:26.974958+0000","last_fullsized":"2023-01-15T14:33:26.974958+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:53.576","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.10134734300000001,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":28368,"num_write":46,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6670228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.f","version":"155'145","reported_seq":1151,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.881539+0000","last_change":"2023-01-15T14:33:26.881526+0000","last_active":"2023-01-15T14:33:26.881539+0000","last_peered":"2023-01-15T14:33:26.881539+0000","last_clean":"2023-01-15T14:33:26.881539+0000","last_became_active":"2023-01-15T14:33:20.239855+0000","last_became_peered":"2023-01-15T14:33:20.239855+0000","last_unstale":"2023-01-15T14:33:26.881539+0000","last_undegraded":"2023-01-15T14:33:26.881539+0000","last_fullsized":"2023-01-15T14:33:26.881539+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.201965651,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.e","version":"139'38","reported_seq":942,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.980988+0000","last_change":"2023-01-15T14:33:26.980925+0000","last_active":"2023-01-15T14:33:26.980988+0000","last_peered":"2023-01-15T14:33:26.980988+0000","last_clean":"2023-01-15T14:33:26.980988+0000","last_became_active":"2023-01-15T14:33:20.235974+0000","last_became_peered":"2023-01-15T14:33:20.235974+0000","last_unstale":"2023-01-15T14:33:26.980988+0000","last_undegraded":"2023-01-15T14:33:26.980988+0000","last_fullsized":"2023-01-15T14:33:26.980988+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.001801497,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.d","version":"296'232","reported_seq":1218,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.876829+0000","last_change":"2023-01-15T14:33:26.876829+0000","last_active":"2023-01-15T14:33:26.876829+0000","last_peered":"2023-01-15T14:33:26.876829+0000","last_clean":"2023-01-15T14:33:26.876829+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:33:26.876829+0000","last_undegraded":"2023-01-15T14:33:26.876829+0000","last_fullsized":"2023-01-15T14:33:26.876829+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.002327572,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.c","version":"311'73","reported_seq":965,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.976829+0000","last_change":"2023-01-15T14:33:26.976722+0000","last_active":"2023-01-15T14:33:26.976829+0000","last_peered":"2023-01-15T14:33:26.976829+0000","last_clean":"2023-01-15T14:33:26.976829+0000","last_became_active":"2023-01-15T14:33:20.338147+0000","last_became_peered":"2023-01-15T14:33:20.338147+0000","last_unstale":"2023-01-15T14:33:26.976829+0000","last_undegraded":"2023-01-15T14:33:26.976829+0000","last_fullsized":"2023-01-15T14:33:26.976829+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.007338448,"stat_sum":{"num_bytes":2221858,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":10493,"num_write":2,"num_write_kb":686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":917,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.981196+0000","last_change":"2023-01-15T14:33:27.184339+0000","last_active":"2023-01-15T14:33:26.981196+0000","last_peered":"2023-01-15T14:33:26.981196+0000","last_clean":"2023-01-15T14:33:26.981196+0000","last_became_active":"2023-01-15T14:32:22.246088+0000","last_became_peered":"2023-01-15T14:32:22.246088+0000","last_unstale":"2023-01-15T14:33:26.981196+0000","last_undegraded":"2023-01-15T14:33:26.981196+0000","last_fullsized":"2023-01-15T14:33:26.981196+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:06.346","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.30817184800000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1240,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.087145+0000","last_change":"2023-01-15T14:33:27.087145+0000","last_active":"2023-01-15T14:33:27.087145+0000","last_peered":"2023-01-15T14:33:27.087145+0000","last_clean":"2023-01-15T14:33:27.087145+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:33:27.087145+0000","last_undegraded":"2023-01-15T14:33:27.087145+0000","last_fullsized":"2023-01-15T14:33:27.087145+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.2066405,"stat_sum":{"num_bytes":2130792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21399,"num_write":27,"num_write_kb":9854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9447726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1033,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.981477+0000","last_change":"2023-01-15T14:33:26.981414+0000","last_active":"2023-01-15T14:33:26.981477+0000","last_peered":"2023-01-15T14:33:26.981477+0000","last_clean":"2023-01-15T14:33:26.981477+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:33:26.981477+0000","last_undegraded":"2023-01-15T14:33:26.981477+0000","last_fullsized":"2023-01-15T14:33:26.981477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:38.874","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.0046924829999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1228,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.975891+0000","last_change":"2023-01-15T14:33:26.975891+0000","last_active":"2023-01-15T14:33:26.975891+0000","last_peered":"2023-01-15T14:33:26.975891+0000","last_clean":"2023-01-15T14:33:26.975891+0000","last_became_active":"2023-01-15T14:32:22.142296+0000","last_became_peered":"2023-01-15T14:32:22.142296+0000","last_unstale":"2023-01-15T14:33:26.975891+0000","last_undegraded":"2023-01-15T14:33:26.975891+0000","last_fullsized":"2023-01-15T14:33:26.975891+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.117888563,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"1.0","version":"19'117","reported_seq":464,"reported_epoch":317,"state":"active+clean","last_fresh":"2023-01-15T14:33:25.905796+0000","last_change":"2023-01-15T14:32:54.588161+0000","last_active":"2023-01-15T14:33:25.905796+0000","last_peered":"2023-01-15T14:33:25.905796+0000","last_clean":"2023-01-15T14:33:25.905796+0000","last_became_active":"2023-01-15T14:31:03.091071+0000","last_became_peered":"2023-01-15T14:31:03.091071+0000","last_unstale":"2023-01-15T14:33:25.905796+0000","last_undegraded":"2023-01-15T14:33:25.905796+0000","last_fullsized":"2023-01-15T14:33:25.905796+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:16.622","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":289,"reported_epoch":317,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.198780+0000","last_change":"2023-01-15T14:31:38.018850+0000","last_active":"2023-01-15T14:33:26.198780+0000","last_peered":"2023-01-15T14:33:26.198780+0000","last_clean":"2023-01-15T14:33:26.198780+0000","last_became_active":"2023-01-15T14:31:38.018145+0000","last_became_peered":"2023-01-15T14:31:38.018145+0000","last_unstale":"2023-01-15T14:33:26.198780+0000","last_undegraded":"2023-01-15T14:33:26.198780+0000","last_fullsized":"2023-01-15T14:33:26.198780+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":923,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.087531+0000","last_change":"2023-01-15T14:33:27.087531+0000","last_active":"2023-01-15T14:33:27.087531+0000","last_peered":"2023-01-15T14:33:27.087531+0000","last_clean":"2023-01-15T14:33:27.087531+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:33:27.087531+0000","last_undegraded":"2023-01-15T14:33:27.087531+0000","last_fullsized":"2023-01-15T14:33:27.087531+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:35.297","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.20685313399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":363,"reported_epoch":317,"state":"active+clean","last_fresh":"2023-01-15T14:33:25.910849+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:33:25.910849+0000","last_peered":"2023-01-15T14:33:25.910849+0000","last_clean":"2023-01-15T14:33:25.910849+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:33:25.910849+0000","last_undegraded":"2023-01-15T14:33:25.910849+0000","last_fullsized":"2023-01-15T14:33:25.910849+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.964","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":796,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.983296+0000","last_change":"2023-01-15T14:33:26.983296+0000","last_active":"2023-01-15T14:33:26.983296+0000","last_peered":"2023-01-15T14:33:26.983296+0000","last_clean":"2023-01-15T14:33:26.983296+0000","last_became_active":"2023-01-15T14:32:40.967729+0000","last_became_peered":"2023-01-15T14:32:40.967729+0000","last_unstale":"2023-01-15T14:33:26.983296+0000","last_undegraded":"2023-01-15T14:33:26.983296+0000","last_fullsized":"2023-01-15T14:33:26.983296+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:43.783","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.10388588999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":118,"reported_epoch":317,"state":"active+clean","last_fresh":"2023-01-15T14:33:25.898324+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:33:25.898324+0000","last_peered":"2023-01-15T14:33:25.898324+0000","last_clean":"2023-01-15T14:33:25.898324+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:33:25.898324+0000","last_undegraded":"2023-01-15T14:33:25.898324+0000","last_fullsized":"2023-01-15T14:33:25.898324+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1245,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.291537+0000","last_change":"2023-01-15T14:33:27.292345+0000","last_active":"2023-01-15T14:33:27.291537+0000","last_peered":"2023-01-15T14:33:27.291537+0000","last_clean":"2023-01-15T14:33:27.291537+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:33:27.291537+0000","last_undegraded":"2023-01-15T14:33:27.291537+0000","last_fullsized":"2023-01-15T14:33:27.291537+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:31.185","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.20645706799999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":186,"reported_epoch":317,"state":"active+clean","last_fresh":"2023-01-15T14:33:25.903791+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:33:25.903791+0000","last_peered":"2023-01-15T14:33:25.903791+0000","last_clean":"2023-01-15T14:33:25.903791+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:33:25.903791+0000","last_undegraded":"2023-01-15T14:33:25.903791+0000","last_fullsized":"2023-01-15T14:33:25.903791+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:25.583","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"274'309","reported_seq":1308,"reported_epoch":316,"state":"active+clean","last_fresh":"2023-01-15T14:33:24.887234+0000","last_change":"2023-01-15T14:33:19.128467+0000","last_active":"2023-01-15T14:33:24.887234+0000","last_peered":"2023-01-15T14:33:24.887234+0000","last_clean":"2023-01-15T14:33:24.887234+0000","last_became_active":"2023-01-15T14:33:10.346559+0000","last_became_peered":"2023-01-15T14:33:10.346559+0000","last_unstale":"2023-01-15T14:33:24.887234+0000","last_undegraded":"2023-01-15T14:33:24.887234+0000","last_fullsized":"2023-01-15T14:33:24.887234+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":309,"log_dups_size":0,"ondisk_log_size":309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.652","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.004686801,"stat_sum":{"num_bytes":2840449,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.5","version":"318'146","reported_seq":1082,"reported_epoch":318,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:27.082199+0000","last_change":"2023-01-15T14:33:27.082199+0000","last_active":"2023-01-15T14:33:27.082199+0000","last_peered":"2023-01-15T14:33:27.082199+0000","last_clean":"2023-01-15T14:33:27.082199+0000","last_became_active":"2023-01-15T14:32:21.948278+0000","last_became_peered":"2023-01-15T14:32:21.948278+0000","last_unstale":"2023-01-15T14:33:27.082199+0000","last_undegraded":"2023-01-15T14:33:27.082199+0000","last_fullsized":"2023-01-15T14:33:27.082199+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":146,"log_dups_size":0,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:38.550","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.61394897900000001,"stat_sum":{"num_bytes":7877405,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":26145,"num_write":6,"num_write_kb":4265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":176,"reported_epoch":317,"state":"active+clean","last_fresh":"2023-01-15T14:33:25.904735+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:33:25.904735+0000","last_peered":"2023-01-15T14:33:25.904735+0000","last_clean":"2023-01-15T14:33:25.904735+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:33:25.904735+0000","last_undegraded":"2023-01-15T14:33:25.904735+0000","last_fullsized":"2023-01-15T14:33:25.904735+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:37.670","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"294'305","reported_seq":1334,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.186232+0000","last_change":"2023-01-15T14:33:27.187299+0000","last_active":"2023-01-15T14:33:27.186232+0000","last_peered":"2023-01-15T14:33:27.186232+0000","last_clean":"2023-01-15T14:33:27.186232+0000","last_became_active":"2023-01-15T14:32:22.247573+0000","last_became_peered":"2023-01-15T14:32:22.247573+0000","last_unstale":"2023-01-15T14:33:27.186232+0000","last_undegraded":"2023-01-15T14:33:27.186232+0000","last_fullsized":"2023-01-15T14:33:27.186232+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.235438599,"stat_sum":{"num_bytes":904090,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":30683,"num_write":27,"num_write_kb":9142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7931053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":355,"reported_epoch":317,"state":"active+clean","last_fresh":"2023-01-15T14:33:25.911538+0000","last_change":"2023-01-15T14:32:34.578760+0000","last_active":"2023-01-15T14:33:25.911538+0000","last_peered":"2023-01-15T14:33:25.911538+0000","last_clean":"2023-01-15T14:33:25.911538+0000","last_became_active":"2023-01-15T14:31:03.088392+0000","last_became_peered":"2023-01-15T14:31:03.088392+0000","last_unstale":"2023-01-15T14:33:25.911538+0000","last_undegraded":"2023-01-15T14:33:25.911538+0000","last_fullsized":"2023-01-15T14:33:25.911538+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:38.541","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"318'127","reported_seq":1050,"reported_epoch":319,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:28.079754+0000","last_change":"2023-01-15T14:33:27.832072+0000","last_active":"2023-01-15T14:33:28.079754+0000","last_peered":"2023-01-15T14:33:28.079754+0000","last_clean":"2023-01-15T14:33:28.079754+0000","last_became_active":"2023-01-15T14:32:22.146367+0000","last_became_peered":"2023-01-15T14:32:22.146367+0000","last_unstale":"2023-01-15T14:33:28.079754+0000","last_undegraded":"2023-01-15T14:33:28.079754+0000","last_fullsized":"2023-01-15T14:33:28.079754+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:03.357","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.91273126800000004,"stat_sum":{"num_bytes":11371072,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":13597,"num_write":7,"num_write_kb":7710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":176,"reported_epoch":317,"state":"active+clean","last_fresh":"2023-01-15T14:33:25.905758+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:33:25.905758+0000","last_peered":"2023-01-15T14:33:25.905758+0000","last_clean":"2023-01-15T14:33:25.905758+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:33:25.905758+0000","last_undegraded":"2023-01-15T14:33:25.905758+0000","last_fullsized":"2023-01-15T14:33:25.905758+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:17.360","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":942,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.087339+0000","last_change":"2023-01-15T14:33:27.087818+0000","last_active":"2023-01-15T14:33:27.087339+0000","last_peered":"2023-01-15T14:33:27.087339+0000","last_clean":"2023-01-15T14:33:27.087339+0000","last_became_active":"2023-01-15T14:32:22.247696+0000","last_became_peered":"2023-01-15T14:32:22.247696+0000","last_unstale":"2023-01-15T14:33:27.087339+0000","last_undegraded":"2023-01-15T14:33:27.087339+0000","last_fullsized":"2023-01-15T14:33:27.087339+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:09.533","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.20413878199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":298,"reported_epoch":317,"state":"active+clean","last_fresh":"2023-01-15T14:33:25.999477+0000","last_change":"2023-01-15T14:31:38.112378+0000","last_active":"2023-01-15T14:33:25.999477+0000","last_peered":"2023-01-15T14:33:25.999477+0000","last_clean":"2023-01-15T14:33:25.999477+0000","last_became_active":"2023-01-15T14:31:38.111590+0000","last_became_peered":"2023-01-15T14:31:38.111590+0000","last_unstale":"2023-01-15T14:33:25.999477+0000","last_undegraded":"2023-01-15T14:33:25.999477+0000","last_fullsized":"2023-01-15T14:33:25.999477+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"245'148","reported_seq":1092,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.984640+0000","last_change":"2023-01-15T14:33:26.984540+0000","last_active":"2023-01-15T14:33:26.984640+0000","last_peered":"2023-01-15T14:33:26.984640+0000","last_clean":"2023-01-15T14:33:26.984640+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:33:26.984640+0000","last_undegraded":"2023-01-15T14:33:26.984640+0000","last_fullsized":"2023-01-15T14:33:26.984640+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:32:19.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:19.303243+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.200694598,"stat_sum":{"num_bytes":1226927,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10792,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":913,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.081386+0000","last_change":"2023-01-15T14:33:27.082088+0000","last_active":"2023-01-15T14:33:27.081386+0000","last_peered":"2023-01-15T14:33:27.081386+0000","last_clean":"2023-01-15T14:33:27.081386+0000","last_became_active":"2023-01-15T14:33:20.233417+0000","last_became_peered":"2023-01-15T14:33:20.233417+0000","last_unstale":"2023-01-15T14:33:27.081386+0000","last_undegraded":"2023-01-15T14:33:27.081386+0000","last_fullsized":"2023-01-15T14:33:27.081386+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.20536302200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.16","version":"318'125","reported_seq":535,"reported_epoch":318,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:27.126869+0000","last_change":"2023-01-15T14:33:27.126869+0000","last_active":"2023-01-15T14:33:27.126869+0000","last_peered":"2023-01-15T14:33:27.126869+0000","last_clean":"2023-01-15T14:33:27.126869+0000","last_became_active":"2023-01-15T14:31:35.032807+0000","last_became_peered":"2023-01-15T14:31:35.032807+0000","last_unstale":"2023-01-15T14:33:27.126869+0000","last_undegraded":"2023-01-15T14:33:27.126869+0000","last_fullsized":"2023-01-15T14:33:27.126869+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub will be retried not before 2023-01-15T14:34:03.444","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.009501832,"stat_sum":{"num_bytes":7378471,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":20560,"num_write":13,"num_write_kb":5151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.17","version":"278'93","reported_seq":953,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.186749+0000","last_change":"2023-01-15T14:33:27.187619+0000","last_active":"2023-01-15T14:33:27.186749+0000","last_peered":"2023-01-15T14:33:27.186749+0000","last_clean":"2023-01-15T14:33:27.186749+0000","last_became_active":"2023-01-15T14:31:34.932309+0000","last_became_peered":"2023-01-15T14:31:34.932309+0000","last_unstale":"2023-01-15T14:33:27.186749+0000","last_undegraded":"2023-01-15T14:33:27.186749+0000","last_fullsized":"2023-01-15T14:33:27.186749+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:43.141","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.30908600899999999,"stat_sum":{"num_bytes":3025326,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.18","version":"299'315","reported_seq":1391,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.975440+0000","last_change":"2023-01-15T14:33:26.975440+0000","last_active":"2023-01-15T14:33:26.975440+0000","last_peered":"2023-01-15T14:33:26.975440+0000","last_clean":"2023-01-15T14:33:26.975440+0000","last_became_active":"2023-01-15T14:31:52.330386+0000","last_became_peered":"2023-01-15T14:31:52.330386+0000","last_unstale":"2023-01-15T14:33:26.975440+0000","last_undegraded":"2023-01-15T14:33:26.975440+0000","last_fullsized":"2023-01-15T14:33:26.975440+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:06.616","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.117464982,"stat_sum":{"num_bytes":6551330,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":55265,"num_write":44,"num_write_kb":14485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":11033036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.19","version":"294'256","reported_seq":1220,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.884267+0000","last_change":"2023-01-15T14:33:26.884267+0000","last_active":"2023-01-15T14:33:26.884267+0000","last_peered":"2023-01-15T14:33:26.884267+0000","last_clean":"2023-01-15T14:33:26.884267+0000","last_became_active":"2023-01-15T14:31:51.927440+0000","last_became_peered":"2023-01-15T14:31:51.927440+0000","last_unstale":"2023-01-15T14:33:26.884267+0000","last_undegraded":"2023-01-15T14:33:26.884267+0000","last_fullsized":"2023-01-15T14:33:26.884267+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.001170751,"stat_sum":{"num_bytes":2420274,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":10,"num_read_kb":7098,"num_write":10,"num_write_kb":2362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.1a","version":"318'339","reported_seq":1351,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.785713+0000","last_change":"2023-01-15T14:33:27.785713+0000","last_active":"2023-01-15T14:33:27.785713+0000","last_peered":"2023-01-15T14:33:27.785713+0000","last_clean":"2023-01-15T14:33:27.785713+0000","last_became_active":"2023-01-15T14:31:52.124026+0000","last_became_peered":"2023-01-15T14:31:52.124026+0000","last_unstale":"2023-01-15T14:33:27.785713+0000","last_undegraded":"2023-01-15T14:33:27.785713+0000","last_fullsized":"2023-01-15T14:33:27.785713+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:49.729","scrub_duration":0.60775138699999998,"objects_trimmed":1,"snaptrim_duration":0.90649114900000005,"stat_sum":{"num_bytes":6105307,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":9,"num_read_kb":8573,"num_write":12,"num_write_kb":1196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.1b","version":"283'80","reported_seq":920,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.831929+0000","last_change":"2023-01-15T14:33:27.831929+0000","last_active":"2023-01-15T14:33:27.831929+0000","last_peered":"2023-01-15T14:33:27.831929+0000","last_clean":"2023-01-15T14:33:27.831929+0000","last_became_active":"2023-01-15T14:31:52.321399+0000","last_became_peered":"2023-01-15T14:31:52.321399+0000","last_unstale":"2023-01-15T14:33:27.831929+0000","last_undegraded":"2023-01-15T14:33:27.831929+0000","last_fullsized":"2023-01-15T14:33:27.831929+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:00.204","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.95079322200000005,"stat_sum":{"num_bytes":63320,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":21019,"num_write":6,"num_write_kb":2022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":441,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.984077+0000","last_change":"2023-01-15T14:33:26.984016+0000","last_active":"2023-01-15T14:33:26.984077+0000","last_peered":"2023-01-15T14:33:26.984077+0000","last_clean":"2023-01-15T14:33:26.984077+0000","last_became_active":"2023-01-15T14:31:51.926652+0000","last_became_peered":"2023-01-15T14:31:51.926652+0000","last_unstale":"2023-01-15T14:33:26.984077+0000","last_undegraded":"2023-01-15T14:33:26.984077+0000","last_fullsized":"2023-01-15T14:33:26.984077+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:35.143","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.51838070700000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1178,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:27.082691+0000","last_change":"2023-01-15T14:33:27.082675+0000","last_active":"2023-01-15T14:33:27.082691+0000","last_peered":"2023-01-15T14:33:27.082691+0000","last_clean":"2023-01-15T14:33:27.082691+0000","last_became_active":"2023-01-15T14:31:52.025227+0000","last_became_peered":"2023-01-15T14:31:52.025227+0000","last_unstale":"2023-01-15T14:33:27.082691+0000","last_undegraded":"2023-01-15T14:33:27.082691+0000","last_fullsized":"2023-01-15T14:33:27.082691+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:32.121","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.105089934,"stat_sum":{"num_bytes":5739855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":47622,"num_write":25,"num_write_kb":10630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6928644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":891,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.184551+0000","last_change":"2023-01-15T14:33:27.185821+0000","last_active":"2023-01-15T14:33:27.184551+0000","last_peered":"2023-01-15T14:33:27.184551+0000","last_clean":"2023-01-15T14:33:27.184551+0000","last_became_active":"2023-01-15T14:31:52.329815+0000","last_became_peered":"2023-01-15T14:31:52.329815+0000","last_unstale":"2023-01-15T14:33:27.184551+0000","last_undegraded":"2023-01-15T14:33:27.184551+0000","last_fullsized":"2023-01-15T14:33:27.184551+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:45.579","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.307735539,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.1f","version":"318'256","reported_seq":1293,"reported_epoch":319,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:28.079623+0000","last_change":"2023-01-15T14:33:27.785873+0000","last_active":"2023-01-15T14:33:28.079623+0000","last_peered":"2023-01-15T14:33:28.079623+0000","last_clean":"2023-01-15T14:33:28.079623+0000","last_became_active":"2023-01-15T14:31:52.124821+0000","last_became_peered":"2023-01-15T14:31:52.124821+0000","last_unstale":"2023-01-15T14:33:28.079623+0000","last_undegraded":"2023-01-15T14:33:28.079623+0000","last_fullsized":"2023-01-15T14:33:28.079623+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:55.650","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.30935489599999999,"stat_sum":{"num_bytes":5949777,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":20,"num_read_kb":26683,"num_write":2,"num_write_kb":3398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.20","version":"303'264","reported_seq":1293,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.875360+0000","last_change":"2023-01-15T14:33:27.076550+0000","last_active":"2023-01-15T14:33:26.875360+0000","last_peered":"2023-01-15T14:33:26.875360+0000","last_clean":"2023-01-15T14:33:26.875360+0000","last_became_active":"2023-01-15T14:31:07.906191+0000","last_became_peered":"2023-01-15T14:31:07.906191+0000","last_unstale":"2023-01-15T14:33:26.875360+0000","last_undegraded":"2023-01-15T14:33:26.875360+0000","last_fullsized":"2023-01-15T14:33:26.875360+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.20153467999999999,"stat_sum":{"num_bytes":6218722,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":21727,"num_write":14,"num_write_kb":6551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8877936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.21","version":"310'134","reported_seq":947,"reported_epoch":318,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:27.587003+0000","last_change":"2023-01-15T14:33:27.587121+0000","last_active":"2023-01-15T14:33:27.587003+0000","last_peered":"2023-01-15T14:33:27.587003+0000","last_clean":"2023-01-15T14:33:27.587003+0000","last_became_active":"2023-01-15T14:31:38.233481+0000","last_became_peered":"2023-01-15T14:31:38.233481+0000","last_unstale":"2023-01-15T14:33:27.587003+0000","last_undegraded":"2023-01-15T14:33:27.587003+0000","last_fullsized":"2023-01-15T14:33:27.587003+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:05.612","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.10491219,"stat_sum":{"num_bytes":2067327,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.22","version":"284'157","reported_seq":1217,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:27.074843+0000","last_change":"2023-01-15T14:33:27.074843+0000","last_active":"2023-01-15T14:33:27.074843+0000","last_peered":"2023-01-15T14:33:27.074843+0000","last_clean":"2023-01-15T14:33:27.074843+0000","last_became_active":"2023-01-15T14:31:08.084483+0000","last_became_peered":"2023-01-15T14:31:08.084483+0000","last_unstale":"2023-01-15T14:33:27.074843+0000","last_undegraded":"2023-01-15T14:33:27.074843+0000","last_fullsized":"2023-01-15T14:33:27.074843+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.116542806,"stat_sum":{"num_bytes":2756414,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":14,"num_read_kb":13467,"num_write":16,"num_write_kb":7631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":1105,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.979977+0000","last_change":"2023-01-15T14:33:27.081156+0000","last_active":"2023-01-15T14:33:26.979977+0000","last_peered":"2023-01-15T14:33:26.979977+0000","last_clean":"2023-01-15T14:33:26.979977+0000","last_became_active":"2023-01-15T14:31:06.029716+0000","last_became_peered":"2023-01-15T14:31:06.029716+0000","last_unstale":"2023-01-15T14:33:26.979977+0000","last_undegraded":"2023-01-15T14:33:26.979977+0000","last_fullsized":"2023-01-15T14:33:26.979977+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:08.017","scrub_duration":0.75838583400000004,"objects_trimmed":0,"snaptrim_duration":0.20559251200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.24","version":"317'243","reported_seq":1289,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.975126+0000","last_change":"2023-01-15T14:33:26.975126+0000","last_active":"2023-01-15T14:33:26.975126+0000","last_peered":"2023-01-15T14:33:26.975126+0000","last_clean":"2023-01-15T14:33:26.975126+0000","last_became_active":"2023-01-15T14:31:26.168777+0000","last_became_peered":"2023-01-15T14:31:26.168777+0000","last_unstale":"2023-01-15T14:33:26.975126+0000","last_undegraded":"2023-01-15T14:33:26.975126+0000","last_fullsized":"2023-01-15T14:33:26.975126+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:35.100","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.20534936500000001,"stat_sum":{"num_bytes":7097465,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":21546,"num_write":9,"num_write_kb":5482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.25","version":"310'98","reported_seq":1017,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.087972+0000","last_change":"2023-01-15T14:33:27.088608+0000","last_active":"2023-01-15T14:33:27.087972+0000","last_peered":"2023-01-15T14:33:27.087972+0000","last_clean":"2023-01-15T14:33:27.087972+0000","last_became_active":"2023-01-15T14:31:22.000386+0000","last_became_peered":"2023-01-15T14:31:22.000386+0000","last_unstale":"2023-01-15T14:33:27.087972+0000","last_undegraded":"2023-01-15T14:33:27.087972+0000","last_fullsized":"2023-01-15T14:33:27.087972+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.204925781,"stat_sum":{"num_bytes":5277530,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":6061,"num_write":2,"num_write_kb":5154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":367,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.877187+0000","last_change":"2023-01-15T14:33:26.877187+0000","last_active":"2023-01-15T14:33:26.877187+0000","last_peered":"2023-01-15T14:33:26.877187+0000","last_clean":"2023-01-15T14:33:26.877187+0000","last_became_active":"2023-01-15T14:31:21.901622+0000","last_became_peered":"2023-01-15T14:31:21.901622+0000","last_unstale":"2023-01-15T14:33:26.877187+0000","last_undegraded":"2023-01-15T14:33:26.877187+0000","last_fullsized":"2023-01-15T14:33:26.877187+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:08.119","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.116574172,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":978,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.185905+0000","last_change":"2023-01-15T14:33:27.186632+0000","last_active":"2023-01-15T14:33:27.185905+0000","last_peered":"2023-01-15T14:33:27.185905+0000","last_clean":"2023-01-15T14:33:27.185905+0000","last_became_active":"2023-01-15T14:31:22.091646+0000","last_became_peered":"2023-01-15T14:31:22.091646+0000","last_unstale":"2023-01-15T14:33:27.185905+0000","last_undegraded":"2023-01-15T14:33:27.185905+0000","last_fullsized":"2023-01-15T14:33:27.185905+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:41.261","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.30845329399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.28","version":"315'240","reported_seq":1284,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:27.074828+0000","last_change":"2023-01-15T14:33:27.074828+0000","last_active":"2023-01-15T14:33:27.074828+0000","last_peered":"2023-01-15T14:33:27.074828+0000","last_clean":"2023-01-15T14:33:27.074828+0000","last_became_active":"2023-01-15T14:31:26.054878+0000","last_became_peered":"2023-01-15T14:31:26.054878+0000","last_unstale":"2023-01-15T14:33:27.074828+0000","last_undegraded":"2023-01-15T14:33:27.074828+0000","last_fullsized":"2023-01-15T14:33:27.074828+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:01.610","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.115932206,"stat_sum":{"num_bytes":6130436,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":17938,"num_write":16,"num_write_kb":6906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":12193528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":796,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.974819+0000","last_change":"2023-01-15T14:33:26.974819+0000","last_active":"2023-01-15T14:33:26.974819+0000","last_peered":"2023-01-15T14:33:26.974819+0000","last_clean":"2023-01-15T14:33:26.974819+0000","last_became_active":"2023-01-15T14:31:25.136376+0000","last_became_peered":"2023-01-15T14:31:25.136376+0000","last_unstale":"2023-01-15T14:33:26.974819+0000","last_undegraded":"2023-01-15T14:33:26.974819+0000","last_fullsized":"2023-01-15T14:33:26.974819+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:23.471","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.117531652,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1155,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.485595+0000","last_change":"2023-01-15T14:33:27.586835+0000","last_active":"2023-01-15T14:33:27.485595+0000","last_peered":"2023-01-15T14:33:27.485595+0000","last_clean":"2023-01-15T14:33:27.485595+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:33:27.485595+0000","last_undegraded":"2023-01-15T14:33:27.485595+0000","last_fullsized":"2023-01-15T14:33:27.485595+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.71057234899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":872,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.187421+0000","last_change":"2023-01-15T14:33:27.288925+0000","last_active":"2023-01-15T14:33:27.187421+0000","last_peered":"2023-01-15T14:33:27.187421+0000","last_clean":"2023-01-15T14:33:27.187421+0000","last_became_active":"2023-01-15T14:31:21.896494+0000","last_became_peered":"2023-01-15T14:31:21.896494+0000","last_unstale":"2023-01-15T14:33:27.187421+0000","last_undegraded":"2023-01-15T14:33:27.187421+0000","last_fullsized":"2023-01-15T14:33:27.187421+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:15.975","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.31073314699999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.2c","version":"141'46","reported_seq":357,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.983227+0000","last_change":"2023-01-15T14:33:26.983192+0000","last_active":"2023-01-15T14:33:26.983227+0000","last_peered":"2023-01-15T14:33:26.983227+0000","last_clean":"2023-01-15T14:33:26.983227+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:33:26.983227+0000","last_undegraded":"2023-01-15T14:33:26.983227+0000","last_fullsized":"2023-01-15T14:33:26.983227+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:10.970","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.112292401,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1125,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.877314+0000","last_change":"2023-01-15T14:33:26.877314+0000","last_active":"2023-01-15T14:33:26.877314+0000","last_peered":"2023-01-15T14:33:26.877314+0000","last_clean":"2023-01-15T14:33:26.877314+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:33:26.877314+0000","last_undegraded":"2023-01-15T14:33:26.877314+0000","last_fullsized":"2023-01-15T14:33:26.877314+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.11726210199999999,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":807,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:27.086344+0000","last_change":"2023-01-15T14:33:27.087248+0000","last_active":"2023-01-15T14:33:27.086344+0000","last_peered":"2023-01-15T14:33:27.086344+0000","last_clean":"2023-01-15T14:33:27.086344+0000","last_became_active":"2023-01-15T14:31:38.116756+0000","last_became_peered":"2023-01-15T14:31:38.116756+0000","last_unstale":"2023-01-15T14:33:27.086344+0000","last_undegraded":"2023-01-15T14:33:27.086344+0000","last_fullsized":"2023-01-15T14:33:27.086344+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.203696669,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1000,"reported_epoch":318,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:26.978007+0000","last_change":"2023-01-15T14:33:26.978007+0000","last_active":"2023-01-15T14:33:26.978007+0000","last_peered":"2023-01-15T14:33:26.978007+0000","last_clean":"2023-01-15T14:33:26.978007+0000","last_became_active":"2023-01-15T14:31:40.024531+0000","last_became_peered":"2023-01-15T14:31:40.024531+0000","last_unstale":"2023-01-15T14:33:26.978007+0000","last_undegraded":"2023-01-15T14:33:26.978007+0000","last_fullsized":"2023-01-15T14:33:26.978007+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:34.781","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.117672361,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e5","length":"2"},{"start":"f0","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":1256,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.874795+0000","last_change":"2023-01-15T14:33:26.875781+0000","last_active":"2023-01-15T14:33:26.874795+0000","last_peered":"2023-01-15T14:33:26.874795+0000","last_clean":"2023-01-15T14:33:26.874795+0000","last_became_active":"2023-01-15T14:31:39.915702+0000","last_became_peered":"2023-01-15T14:31:39.915702+0000","last_unstale":"2023-01-15T14:33:26.874795+0000","last_undegraded":"2023-01-15T14:33:26.874795+0000","last_fullsized":"2023-01-15T14:33:26.874795+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.1539330249999999,"objects_trimmed":0,"snaptrim_duration":0.0010309410000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]},{"pgid":"3.31","version":"204'150","reported_seq":1141,"reported_epoch":318,"state":"active+clean","last_fresh":"2023-01-15T14:33:26.879439+0000","last_change":"2023-01-15T14:33:26.881751+0000","last_active":"2023-01-15T14:33:26.879439+0000","last_peered":"2023-01-15T14:33:26.879439+0000","last_clean":"2023-01-15T14:33:26.879439+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:33:26.879439+0000","last_undegraded":"2023-01-15T14:33:26.879439+0000","last_fullsized":"2023-01-15T14:33:26.879439+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.0024409919999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d8","length":"1"},{"start":"da","length":"1"},{"start":"dd","length":"1"},{"start":"df","length":"1"},{"start":"e4","length":"3"},{"start":"f0","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":86,"stat_sum":{"num_bytes":233658133,"num_objects":102,"num_object_clones":56,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":7,"num_read":746,"num_read_kb":758624,"num_write":628,"num_write_kb":255367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":166346998,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14774,"ondisk_log_size":14774,"up":258,"acting":257,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":37,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":947024,"kb_used_data":92769200,"kb_used_omap":5863,"kb_used_meta":0,"kb_avail":92752356,"statfs":{"total":95948165120,"available":94978412544,"internally_reserved":17248256,"allocated":94995660800,"data_stored":94995660800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6003915,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":4.1269999999999998,"15min":4.0890000000000004},"min":{"1min":0.54000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2030000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":9.5380000000000003,"5min":6.3200000000000003,"15min":8.2460000000000004},"min":{"1min":0.58799999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":96.337000000000003,"5min":96.337000000000003,"15min":96.337000000000003},"last":29.997}]},{"osd":1,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":3.6509999999999998,"15min":5.5800000000000001},"min":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":34.585999999999999,"5min":88.003,"15min":88.003},"last":0.82599999999999996},{"interface":"front","average":{"1min":13.302,"5min":8.1950000000000003,"15min":6.9180000000000001},"min":{"1min":0.58799999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.403999999999996,"5min":97.704999999999998,"15min":97.704999999999998},"last":29.940000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":6.9400000000000004,"15min":5.5069999999999997},"min":{"1min":0.57199999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.179,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.91800000000000004,"5min":2.5790000000000002,"15min":3.645},"min":{"1min":0.626,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":91.837999999999994,"15min":91.837999999999994},"last":72.278000000000006}]},{"osd":3,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":7.3799999999999999,"15min":3.0920000000000001},"min":{"1min":0.63600000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2370000000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.91600000000000004,"5min":3.6360000000000001,"15min":4.3129999999999997},"min":{"1min":0.65000000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.2270000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.89500000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5519999999999996,"5min":5.6219999999999999,"15min":8.1509999999999998},"min":{"1min":0.63500000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":96.185000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":29.690000000000001},{"interface":"front","average":{"1min":1.9259999999999999,"5min":5.782,"15min":2.5699999999999998},"min":{"1min":0.73099999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":22.41,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.78400000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.6440000000000001,"5min":5.0659999999999998,"15min":5.9059999999999997},"min":{"1min":0.72799999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":96.263999999999996,"5min":96.263999999999996,"15min":96.263999999999996},"last":29.797999999999998},{"interface":"front","average":{"1min":0.998,"5min":3.206,"15min":4.3650000000000002},"min":{"1min":0.69299999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.355,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.84099999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":5.5949999999999998,"15min":7.5469999999999997},"min":{"1min":0.83599999999999997,"5min":0.496,"15min":0.496},"max":{"1min":96.299999999999997,"5min":96.299999999999997,"15min":96.299999999999997},"last":29.876000000000001},{"interface":"front","average":{"1min":1.089,"5min":2.5150000000000001,"15min":1.571},"min":{"1min":0.84299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.2789999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":0.97899999999999998}]}]},{"osd":6,"up_from":0,"seq":55834574915,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574915,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":62,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":966052,"kb_used_data":92751872,"kb_used_omap":11302,"kb_used_meta":0,"kb_avail":92733328,"statfs":{"total":95948165120,"available":94958927872,"internally_reserved":18989056,"allocated":94977916928,"data_stored":94977916928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11573679,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":23.594000000000001,"15min":11.141},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.133,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.78300000000000003,"5min":3.0009999999999999,"15min":1.5209999999999999},"min":{"1min":0.58299999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.024,"5min":88.600999999999999,"15min":88.600999999999999},"last":1.0189999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":5.0730000000000004,"15min":7.4199999999999999},"min":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6659999999999999,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.343,"15min":1.9770000000000001},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0529999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.77100000000000002}]},{"osd":2,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":24.227,"15min":11.749000000000001},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.157,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":1.101},{"interface":"front","average":{"1min":2.6040000000000001,"5min":1.21,"15min":1.01},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":32.081000000000003,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.63600000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":2.9169999999999998,"15min":1.865},"min":{"1min":0.65300000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.2909999999999999,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.7370000000000001,"5min":24.498999999999999,"15min":12.266999999999999},"min":{"1min":0.64400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":14.961,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":0.92600000000000005}]},{"osd":5,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.88500000000000001,"15min":0.90800000000000003},"min":{"1min":0.55200000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":2.1840000000000002,"15min":2.2599999999999998},"min":{"1min":0.71799999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3200000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.95399999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":3.9329999999999998,"15min":5.093},"min":{"1min":0.69599999999999995,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.365,"5min":99.780000000000001,"15min":99.780000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.0489999999999999,"5min":22.463999999999999,"15min":10.93},"min":{"1min":0.71599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.427,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.1719999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":3.1539999999999999,"5min":5.5570000000000004,"15min":10.603999999999999},"min":{"1min":0.76700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":39.012999999999998,"5min":179.73500000000001,"15min":179.73500000000001},"last":42.551000000000002},{"interface":"front","average":{"1min":1.2370000000000001,"5min":24.864000000000001,"15min":15.779999999999999},"min":{"1min":0.877,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.0680000000000001,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.0429999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574914,"num_pgs":64,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1024884,"kb_used_data":92691748,"kb_used_omap":11609,"kb_used_meta":0,"kb_avail":92674496,"statfs":{"total":95948165120,"available":94898683904,"internally_reserved":17666048,"allocated":94916349952,"data_stored":94916349952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11887641,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,3,6,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":7.4400000000000004,"15min":5.7119999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":17.731000000000002,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":3.4340000000000002,"15min":6.2389999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.9260000000000002,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.56200000000000006}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":7.1790000000000003,"15min":11.414},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96799999999999997,"5min":159.256,"15min":159.256},"last":14.901999999999999},{"interface":"front","average":{"1min":2.6259999999999999,"5min":10.417999999999999,"15min":3.9660000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":17.826000000000001,"5min":174.578,"15min":174.578},"last":0.51700000000000002}]},{"osd":2,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":4.6849999999999996,"5min":2.8809999999999998,"15min":3.1930000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":86.412000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.497},{"interface":"front","average":{"1min":1.95,"5min":6.7889999999999997,"15min":3.7250000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.876000000000001,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.69199999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":7.6879999999999997,"5min":29.686,"15min":14.27},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":84.640000000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.86899999999999999,"5min":32.134999999999998,"15min":19.960999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.615,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":15.07}]},{"osd":5,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":6.4980000000000002,"15min":11.433999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.04,"5min":159.43000000000001,"15min":159.43000000000001},"last":14.977},{"interface":"front","average":{"1min":2.2309999999999999,"5min":6.0039999999999996,"15min":2.6819999999999999},"min":{"1min":0.70099999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":17.917000000000002,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.745}]},{"osd":6,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1949999999999998,"5min":5.9980000000000002,"15min":2.6869999999999998},"min":{"1min":0.66000000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":17.989000000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.964,"15min":3.1619999999999999},"min":{"1min":0.70499999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.23,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.71099999999999997}]},{"osd":7,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":2.0219999999999998,"15min":3.3639999999999999},"min":{"1min":0.69499999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.266,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.753},{"interface":"front","average":{"1min":19.138000000000002,"5min":11.705,"15min":15.961},"min":{"1min":0.66200000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":281.13400000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":15.032}]}]},{"osd":2,"up_from":13,"seq":55834574914,"num_pgs":53,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1028388,"kb_used_data":92677960,"kb_used_omap":9429,"kb_used_meta":0,"kb_avail":92670992,"statfs":{"total":95948165120,"available":94895095808,"internally_reserved":7135232,"allocated":94902231040,"data_stored":94902231040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9656075,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":3.109,"5min":3.5230000000000001,"15min":1.601},"min":{"1min":0.437,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":51.009999999999998,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.4299999999999999,"15min":0.95299999999999996},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0089999999999999,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.40000000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.1529999999999996,"5min":5.9059999999999997,"15min":2.5009999999999999},"min":{"1min":0.378,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":91.525999999999996,"5min":143.071,"15min":143.071},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":2.016,"15min":2.9089999999999998},"min":{"1min":0.42299999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.131,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.436}]},{"osd":3,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":8.2210000000000001,"15min":9.6750000000000007},"min":{"1min":0.5,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.3609999999999998,"5min":183.53,"15min":183.53},"last":0.51800000000000002},{"interface":"front","average":{"1min":16.692,"5min":7.5170000000000003,"15min":3.0169999999999999},"min":{"1min":0.61199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":155.47900000000001,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.67500000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3650000000000002,"5min":6.5060000000000002,"15min":4.6020000000000003},"min":{"1min":0.67300000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":51.259999999999998,"5min":175.34700000000001,"15min":175.34700000000001},"last":1.075},{"interface":"front","average":{"1min":2.8809999999999998,"5min":5.2939999999999996,"15min":4.3040000000000003},"min":{"1min":0.48699999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":33.219999999999999,"5min":175.30500000000001,"15min":175.30500000000001},"last":1.0409999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":2.4729999999999999,"15min":1.6080000000000001},"min":{"1min":0.57199999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2130000000000001,"5min":143.262,"15min":143.262},"last":0.66200000000000003},{"interface":"front","average":{"1min":3.4780000000000002,"5min":3.0539999999999998,"15min":1.792},"min":{"1min":0.70999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":51.326000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.73499999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.857,"15min":1.3069999999999999},"min":{"1min":0.60399999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.365,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.018,"5min":2.6160000000000001,"15min":1.6839999999999999},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.5,"5min":143.215,"15min":143.215},"last":0.753}]},{"osd":7,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":13.092000000000001,"5min":6.3120000000000003,"15min":4.1429999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":120.905,"5min":120.905,"15min":120.905},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.0129999999999999,"5min":2.8450000000000002,"15min":2.7989999999999999},"min":{"1min":0.66600000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.552,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.622}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":971536,"kb_used_data":92740224,"kb_used_omap":10904,"kb_used_meta":0,"kb_avail":92727844,"statfs":{"total":95948165120,"available":94953312256,"internally_reserved":12677120,"allocated":94965989376,"data_stored":94965989376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11165850,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":3.633,"15min":3.806},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":3.9140000000000001,"5min":144.12799999999999,"15min":144.12799999999999},"last":23.68},{"interface":"front","average":{"1min":7.742,"5min":4.6130000000000004,"15min":4.4189999999999996},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":119.39100000000001,"5min":119.39100000000001,"15min":119.39100000000001},"last":0.42299999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":2.6469999999999998,"15min":1.5269999999999999},"min":{"1min":0.29799999999999999,"5min":0.245,"15min":0.245},"max":{"1min":95.879999999999995,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.8620000000000001,"15min":3.3359999999999999},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.9780000000000002,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.28399999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.8920000000000003,"5min":9.6750000000000007,"15min":16.449000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":100.137,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":7.782,"5min":6.8899999999999997,"15min":7.1260000000000003},"min":{"1min":0.45600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":119.804,"5min":119.804,"15min":119.804},"last":0.46500000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:33:17 2023","interfaces":[{"interface":"back","average":{"1min":7.9249999999999998,"5min":17.140999999999998,"15min":8.8650000000000002},"min":{"1min":0.504,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":119.71899999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.7849999999999999,"5min":15.526,"15min":7.5780000000000003},"min":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":16.989999999999998,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.64300000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.5449999999999999,"15min":1.2030000000000001},"min":{"1min":0.59899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.278,"5min":65.120999999999995,"15min":65.120999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.6579999999999999,"15min":3.5110000000000001},"min":{"1min":0.60599999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":4.0780000000000003,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.66000000000000003}]},{"osd":6,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":8.0069999999999997,"5min":5.0129999999999999,"15min":5.0049999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":119.759,"5min":119.759,"15min":119.759},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.603,"15min":1.2829999999999999},"min":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.349,"5min":65.156000000000006,"15min":65.156000000000006},"last":0.751}]},{"osd":7,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":3.6030000000000002,"5min":6.1109999999999998,"15min":6.7149999999999999},"min":{"1min":0.56000000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":31.620999999999999,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":8.0779999999999994,"5min":5.0060000000000002,"15min":4.835},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":119.65300000000001,"5min":119.65300000000001,"15min":119.65300000000001},"last":0.86799999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574915,"num_pgs":18,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":852180,"kb_used_data":92858536,"kb_used_omap":5445,"kb_used_meta":0,"kb_avail":92847200,"statfs":{"total":95948165120,"available":95075532800,"internally_reserved":11608064,"allocated":95087140864,"data_stored":95087140864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5576160,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":3.1219999999999999,"15min":1.456},"min":{"1min":0.40899999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1060000000000001,"5min":69.872,"15min":69.872},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":3.2309999999999999,"15min":1.5629999999999999},"min":{"1min":0.51300000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.2350000000000001,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.93700000000000006}]},{"osd":2,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":3.4220000000000002,"15min":1.603},"min":{"1min":0.46999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.444,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.74099999999999999,"5min":2.6560000000000001,"15min":1.3580000000000001},"min":{"1min":0.52000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.302,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.71599999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":17.571999999999999,"5min":10.789,"15min":14.782},"min":{"1min":0.53300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":199.08000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.835,"5min":7.306,"15min":2.9689999999999999},"min":{"1min":0.56599999999999995,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":15.601000000000001,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.751}]},{"osd":4,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":16.695,"15min":6.2220000000000004},"min":{"1min":0.73199999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1879999999999999,"5min":1486.309,"15min":1486.309},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.038,"5min":17.925000000000001,"15min":6.6580000000000004},"min":{"1min":0.82499999999999996,"5min":0.439,"15min":0.439},"max":{"1min":1.3759999999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":1.0840000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":4.6749999999999998,"15min":8.0899999999999999},"min":{"1min":0.62,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.304,"5min":176.34399999999999,"15min":176.34399999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.008,"5min":1.73,"15min":1.2689999999999999},"min":{"1min":0.69199999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2529999999999999,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.97799999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":3.5910000000000002,"15min":1.9770000000000001},"min":{"1min":0.67800000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5960000000000001,"5min":70.259,"15min":70.259},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":3.0209999999999999,"15min":1.7629999999999999},"min":{"1min":0.83699999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5489999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":1.1160000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":6.1200000000000001,"15min":12.487},"min":{"1min":0.78400000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.48,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.044},{"interface":"front","average":{"1min":1.117,"5min":3.6070000000000002,"15min":1.9830000000000001},"min":{"1min":0.76100000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5229999999999999,"5min":70.322000000000003,"15min":70.322000000000003},"last":1.169}]}]}],"pool_statfs":[]}} 2023-01-15T14:33:31.395 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:30.727+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:31.400 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:33:31.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 96 2023-01-15T14:33:31.401 INFO:tasks.rados.rados.0.smithi090.stdout:1263: finishing write tid 2 to smithi09025264-24 2023-01-15T14:33:31.401 INFO:tasks.rados.rados.0.smithi090.stdout:1263: finishing write tid 3 to smithi09025264-24 2023-01-15T14:33:31.401 INFO:tasks.rados.rados.0.smithi090.stdout:1263: finishing write tid 4 to smithi09025264-24 2023-01-15T14:33:31.402 INFO:tasks.rados.rados.0.smithi090.stdout:1263: finishing write tid 5 to smithi09025264-24 2023-01-15T14:33:31.402 INFO:tasks.rados.rados.0.smithi090.stdout:1263: oid 24 updating version 0 to 304 2023-01-15T14:33:31.402 INFO:tasks.rados.rados.0.smithi090.stdout:1263: oid 24 version 304 is already newer than 303 2023-01-15T14:33:31.402 INFO:tasks.rados.rados.0.smithi090.stdout:1263: oid 24 updating version 304 to 305 2023-01-15T14:33:31.402 INFO:tasks.rados.rados.0.smithi090.stdout:1263: oid 24 updating version 305 to 306 2023-01-15T14:33:31.402 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 306 (ObjNum 459 snap 132 seq_num 459) dirty exists 2023-01-15T14:33:31.402 INFO:tasks.rados.rados.0.smithi090.stdout:1263: left oid 24 (ObjNum 459 snap 132 seq_num 459) 2023-01-15T14:33:31.403 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:33:31.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:33:31.408 INFO:tasks.rados.rados.0.smithi090.stdout:1261: done (2 left) 2023-01-15T14:33:31.408 INFO:tasks.rados.rados.0.smithi090.stdout:1263: done (1 left) 2023-01-15T14:33:31.408 INFO:tasks.rados.rados.0.smithi090.stdout:1264: done (0 left) 2023-01-15T14:33:31.408 INFO:tasks.rados.rados.0.smithi090.stdout:1265: copy_from oid 29 from oid 34 current snap is 132 2023-01-15T14:33:31.408 INFO:tasks.rados.rados.0.smithi090.stdout:1266: snap_create 2023-01-15T14:33:31.408 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:31.408 INFO:tasks.rados.rados.0.smithi090.stdout:1265: finishing copy_from to smithi09025264-29 2023-01-15T14:33:31.409 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 142 (ObjNum 265 snap 65 seq_num 265) dirty exists 2023-01-15T14:33:31.409 INFO:tasks.rados.rados.0.smithi090.stdout:1265: finishing copy_from racing read to smithi09025264-29 2023-01-15T14:33:31.409 INFO:tasks.rados.rados.0.smithi090.stdout:1265: done (0 left) 2023-01-15T14:33:31.409 INFO:tasks.rados.rados.0.smithi090.stdout:1266: done (0 left) 2023-01-15T14:33:31.409 INFO:tasks.rados.rados.0.smithi090.stdout:1267: write (excl) oid 32 current snap is 133 2023-01-15T14:33:31.409 INFO:tasks.rados.rados.0.smithi090.stdout:1267: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2023-01-15T14:33:31.409 INFO:tasks.rados.rados.0.smithi090.stdout:1267: writing smithi09025264-32 from 548027 to 1018996 tid 1 2023-01-15T14:33:31.409 INFO:tasks.rados.rados.0.smithi090.stdout:1267: writing smithi09025264-32 from 1593493 to 2370314 tid 2 2023-01-15T14:33:31.409 INFO:tasks.rados.rados.0.smithi090.stdout:1267: writing smithi09025264-32 from 3093529 to 3725221 tid 3 2023-01-15T14:33:31.410 INFO:tasks.rados.rados.0.smithi090.stdout:1268: read oid 34 snap -1 2023-01-15T14:33:31.410 INFO:tasks.rados.rados.0.smithi090.stdout:1268: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:33:31.410 INFO:tasks.rados.rados.0.smithi090.stdout:1269: write (excl) oid 25 current snap is 133 2023-01-15T14:33:31.410 INFO:tasks.rados.rados.0.smithi090.stdout:1267: finishing write tid 1 to smithi09025264-32 2023-01-15T14:33:31.410 INFO:tasks.rados.rados.0.smithi090.stdout:1269: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2023-01-15T14:33:31.410 INFO:tasks.rados.rados.0.smithi090.stdout:1269: writing smithi09025264-25 from 564834 to 1111052 tid 1 2023-01-15T14:33:31.410 INFO:tasks.rados.rados.0.smithi090.stdout:1269: writing smithi09025264-25 from 1651975 to 2172454 tid 2 2023-01-15T14:33:31.410 INFO:tasks.rados.rados.0.smithi090.stdout:1269: writing smithi09025264-25 from 2720952 to 3280269 tid 3 2023-01-15T14:33:31.410 INFO:tasks.rados.rados.0.smithi090.stdout:1270: read oid 15 snap -1 2023-01-15T14:33:31.410 INFO:tasks.rados.rados.0.smithi090.stdout:1270: expect (ObjNum 361 snap 96 seq_num 361) 2023-01-15T14:33:31.411 INFO:tasks.rados.rados.0.smithi090.stdout:1271: read oid 26 snap -1 2023-01-15T14:33:31.411 INFO:tasks.rados.rados.0.smithi090.stdout:1271: expect (ObjNum 440 snap 121 seq_num 440) 2023-01-15T14:33:31.411 INFO:tasks.rados.rados.0.smithi090.stdout:1272: delete oid 44 current snap is 133 2023-01-15T14:33:31.411 INFO:tasks.rados.rados.0.smithi090.stdout:1272: done (5 left) 2023-01-15T14:33:31.411 INFO:tasks.rados.rados.0.smithi090.stdout:1273: copy_from oid 4 from oid 13 current snap is 133 2023-01-15T14:33:31.411 INFO:tasks.rados.rados.0.smithi090.stdout:1274: write (excl) oid 3 current snap is 133 2023-01-15T14:33:31.411 INFO:tasks.rados.rados.0.smithi090.stdout:1274: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2023-01-15T14:33:31.411 INFO:tasks.rados.rados.0.smithi090.stdout:1274: writing smithi09025264-3 from 598448 to 1011517 tid 1 2023-01-15T14:33:31.504 INFO:tasks.rados.rados.0.smithi090.stdout:1274: writing smithi09025264-3 from 1768939 to 2293087 tid 2 2023-01-15T14:33:31.504 INFO:tasks.rados.rados.0.smithi090.stdout:1274: writing smithi09025264-3 from 2775798 to 3357659 tid 3 2023-01-15T14:33:31.504 INFO:tasks.rados.rados.0.smithi090.stdout:1274: writing smithi09025264-3 from 3781488 to 3798448 tid 4 2023-01-15T14:33:31.504 INFO:tasks.rados.rados.0.smithi090.stdout:1275: read oid 21 snap 119 2023-01-15T14:33:31.504 INFO:tasks.rados.rados.0.smithi090.stdout:1275: expect (ObjNum 339 snap 89 seq_num 339) 2023-01-15T14:33:31.505 INFO:tasks.rados.rados.0.smithi090.stdout:1276: copy_from oid 37 from oid 42 current snap is 133 2023-01-15T14:33:31.505 INFO:tasks.rados.rados.0.smithi090.stdout:1277: copy_from oid 11 from oid 22 current snap is 133 2023-01-15T14:33:31.505 INFO:tasks.rados.rados.0.smithi090.stdout:1278: snap_remove snap 130 2023-01-15T14:33:31.505 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:31.080+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:31.506 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:31.444+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:31.508 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:31.504+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:31.577 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:31.577 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:31.577 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:33:31.577 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:31.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:33:31.609 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:31.607+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:31.643 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:31.668 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:31.691 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:31.704 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:31.704 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:31.708 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:31.709 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:31.707+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:31.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:33:31.732 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:31.758 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:31.780 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:31.805 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:33:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:33:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:33:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:33:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:33:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:33:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:33:31.807 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:33:31.809 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:31.804+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:31.811 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:31.811 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:31.821 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:33:31.883 INFO:tasks.rados.rados.0.smithi090.stdout:1278: done (10 left) 2023-01-15T14:33:31.883 INFO:tasks.rados.rados.0.smithi090.stdout:1279: write oid 35 current snap is 133 2023-01-15T14:33:31.883 INFO:tasks.rados.rados.0.smithi090.stdout:1279: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2023-01-15T14:33:31.891 INFO:tasks.rados.rados.0.smithi090.stdout:1279: writing smithi09025264-35 from 615255 to 1103573 tid 1 2023-01-15T14:33:31.891 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 96 2023-01-15T14:33:31.900 INFO:tasks.rados.rados.0.smithi090.stdout:1279: writing smithi09025264-35 from 1827421 to 2611580 tid 2 2023-01-15T14:33:31.907 INFO:tasks.rados.rados.0.smithi090.stdout:1279: writing smithi09025264-35 from 3203221 to 3815255 tid 3 2023-01-15T14:33:31.907 INFO:tasks.rados.rados.0.smithi090.stdout:1280: snap_create 2023-01-15T14:33:31.907 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 11 2023-01-15T14:33:31.907 INFO:tasks.rados.rados.0.smithi090.stdout:1276: finishing copy_from to smithi09025264-37 2023-01-15T14:33:31.907 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 351 (ObjNum 446 snap 123 seq_num 446) dirty exists 2023-01-15T14:33:31.908 INFO:tasks.rados.rados.0.smithi090.stdout:1276: finishing copy_from racing read to smithi09025264-37 2023-01-15T14:33:31.908 INFO:tasks.rados.rados.0.smithi090.stdout:1267: finishing write tid 2 to smithi09025264-32 2023-01-15T14:33:31.908 INFO:tasks.rados.rados.0.smithi090.stdout:1267: finishing write tid 3 to smithi09025264-32 2023-01-15T14:33:31.908 INFO:tasks.rados.rados.0.smithi090.stdout:1276: done (10 left) 2023-01-15T14:33:31.908 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 10 2023-01-15T14:33:31.908 INFO:tasks.rados.rados.0.smithi090.stdout:1269: finishing write tid 1 to smithi09025264-25 2023-01-15T14:33:31.909 INFO:tasks.rados.rados.0.smithi090.stdout:1269: finishing write tid 2 to smithi09025264-25 2023-01-15T14:33:31.909 INFO:tasks.rados.rados.0.smithi090.stdout:1271: expect (ObjNum 440 snap 121 seq_num 440) 2023-01-15T14:33:31.911 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:31.907+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:31.979 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:31.979 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:31.991 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:33:32.012 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:32.007+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:32.031 INFO:tasks.rados.rados.0.smithi090.stdout:1273: finishing copy_from to smithi09025264-4 2023-01-15T14:33:32.031 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 4 v 335 (ObjNum 412 snap 112 seq_num 412) dirty exists 2023-01-15T14:33:32.031 INFO:tasks.rados.rados.0.smithi090.stdout:1273: finishing copy_from racing read to smithi09025264-4 2023-01-15T14:33:32.031 INFO:tasks.rados.rados.0.smithi090.stdout:1267: finishing write tid 4 to smithi09025264-32 2023-01-15T14:33:32.031 INFO:tasks.rados.rados.0.smithi090.stdout:1268: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:33:32.090 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:32.090 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:32.101 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:33:32.113 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:32.111+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:32.119 INFO:tasks.rados.rados.0.smithi090.stdout:1270: expect (ObjNum 361 snap 96 seq_num 361) 2023-01-15T14:33:32.189 INFO:tasks.rados.rados.0.smithi090.stdout:1275: expect (ObjNum 339 snap 89 seq_num 339) 2023-01-15T14:33:32.202 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:32.203 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:32.213 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:33:32.214 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:32.208+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:32.315 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:32.311+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:32.318 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:32.318 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:32.329 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:33:32.384 INFO:tasks.rados.rados.0.smithi090.stdout:1269: finishing write tid 3 to smithi09025264-25 2023-01-15T14:33:32.385 INFO:tasks.rados.rados.0.smithi090.stdout:1267: finishing write tid 5 to smithi09025264-32 2023-01-15T14:33:32.385 INFO:tasks.rados.rados.0.smithi090.stdout:1267: oid 32 updating version 0 to 295 2023-01-15T14:33:32.385 INFO:tasks.rados.rados.0.smithi090.stdout:1267: oid 32 updating version 295 to 298 2023-01-15T14:33:32.385 INFO:tasks.rados.rados.0.smithi090.stdout:1267: oid 32 version 298 is already newer than 296 2023-01-15T14:33:32.385 INFO:tasks.rados.rados.0.smithi090.stdout:1267: oid 32 version 298 is already newer than 297 2023-01-15T14:33:32.385 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 298 (ObjNum 461 snap 133 seq_num 461) dirty exists 2023-01-15T14:33:32.385 INFO:tasks.rados.rados.0.smithi090.stdout:1267: left oid 32 (ObjNum 461 snap 133 seq_num 461) 2023-01-15T14:33:32.385 INFO:tasks.rados.rados.0.smithi090.stdout:1269: finishing write tid 4 to smithi09025264-25 2023-01-15T14:33:32.385 INFO:tasks.rados.rados.0.smithi090.stdout:1269: finishing write tid 5 to smithi09025264-25 2023-01-15T14:33:32.385 INFO:tasks.rados.rados.0.smithi090.stdout:1269: oid 25 updating version 0 to 344 2023-01-15T14:33:32.385 INFO:tasks.rados.rados.0.smithi090.stdout:1269: oid 25 version 344 is already newer than 342 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:1269: oid 25 version 344 is already newer than 343 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:1269: oid 25 updating version 344 to 345 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 345 (ObjNum 462 snap 133 seq_num 462) dirty exists 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:1269: left oid 25 (ObjNum 462 snap 133 seq_num 462) 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:1277: finishing copy_from to smithi09025264-11 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 346 (ObjNum 429 snap 116 seq_num 429) dirty exists 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:1274: finishing write tid 1 to smithi09025264-3 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:1277: finishing copy_from racing read to smithi09025264-11 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:1274: finishing write tid 2 to smithi09025264-3 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:1274: finishing write tid 3 to smithi09025264-3 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:1274: finishing write tid 4 to smithi09025264-3 2023-01-15T14:33:32.386 INFO:tasks.rados.rados.0.smithi090.stdout:1274: finishing write tid 5 to smithi09025264-3 2023-01-15T14:33:32.387 INFO:tasks.rados.rados.0.smithi090.stdout:1274: finishing write tid 6 to smithi09025264-3 2023-01-15T14:33:32.387 INFO:tasks.rados.rados.0.smithi090.stdout:1274: oid 3 updating version 0 to 347 2023-01-15T14:33:32.387 INFO:tasks.rados.rados.0.smithi090.stdout:1274: oid 3 updating version 347 to 349 2023-01-15T14:33:32.387 INFO:tasks.rados.rados.0.smithi090.stdout:1274: oid 3 updating version 349 to 350 2023-01-15T14:33:32.387 INFO:tasks.rados.rados.0.smithi090.stdout:1274: oid 3 version 350 is already newer than 348 2023-01-15T14:33:32.387 INFO:tasks.rados.rados.0.smithi090.stdout:1274: oid 3 updating version 350 to 351 2023-01-15T14:33:32.387 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 351 (ObjNum 464 snap 133 seq_num 464) dirty exists 2023-01-15T14:33:32.387 INFO:tasks.rados.rados.0.smithi090.stdout:1274: left oid 3 (ObjNum 464 snap 133 seq_num 464) 2023-01-15T14:33:32.387 INFO:tasks.rados.rados.0.smithi090.stdout:1267: done (9 left) 2023-01-15T14:33:32.387 INFO:tasks.rados.rados.0.smithi090.stdout:1268: done (8 left) 2023-01-15T14:33:32.387 INFO:tasks.rados.rados.0.smithi090.stdout:1269: done (7 left) 2023-01-15T14:33:32.388 INFO:tasks.rados.rados.0.smithi090.stdout:1270: done (6 left) 2023-01-15T14:33:32.388 INFO:tasks.rados.rados.0.smithi090.stdout:1271: done (5 left) 2023-01-15T14:33:32.388 INFO:tasks.rados.rados.0.smithi090.stdout:1273: done (4 left) 2023-01-15T14:33:32.388 INFO:tasks.rados.rados.0.smithi090.stdout:1274: done (3 left) 2023-01-15T14:33:32.388 INFO:tasks.rados.rados.0.smithi090.stdout:1275: done (2 left) 2023-01-15T14:33:32.388 INFO:tasks.rados.rados.0.smithi090.stdout:1277: done (1 left) 2023-01-15T14:33:32.388 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:32.416 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:32.411+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:32.430 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:32.430 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:32.440 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:33:32.492 INFO:tasks.rados.rados.0.smithi090.stdout:1279: finishing write tid 1 to smithi09025264-35 2023-01-15T14:33:32.493 INFO:tasks.rados.rados.0.smithi090.stdout:1279: finishing write tid 2 to smithi09025264-35 2023-01-15T14:33:32.493 INFO:tasks.rados.rados.0.smithi090.stdout:1279: finishing write tid 3 to smithi09025264-35 2023-01-15T14:33:32.506 INFO:tasks.rados.rados.0.smithi090.stdout:1279: finishing write tid 4 to smithi09025264-35 2023-01-15T14:33:32.506 INFO:tasks.rados.rados.0.smithi090.stdout:1279: finishing write tid 5 to smithi09025264-35 2023-01-15T14:33:32.506 INFO:tasks.rados.rados.0.smithi090.stdout:1279: oid 35 updating version 0 to 319 2023-01-15T14:33:32.506 INFO:tasks.rados.rados.0.smithi090.stdout:1279: oid 35 version 319 is already newer than 317 2023-01-15T14:33:32.506 INFO:tasks.rados.rados.0.smithi090.stdout:1279: oid 35 updating version 319 to 320 2023-01-15T14:33:32.506 INFO:tasks.rados.rados.0.smithi090.stdout:1279: oid 35 version 320 is already newer than 318 2023-01-15T14:33:32.506 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 320 (ObjNum 465 snap 133 seq_num 465) dirty exists 2023-01-15T14:33:32.506 INFO:tasks.rados.rados.0.smithi090.stdout:1279: left oid 35 (ObjNum 465 snap 133 seq_num 465) 2023-01-15T14:33:32.506 INFO:tasks.rados.rados.0.smithi090.stdout:1279: done (0 left) 2023-01-15T14:33:32.517 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:32.515+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:32.543 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:32.544 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:32.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:33:33.218 INFO:tasks.rados.rados.0.smithi090.stdout:1280: done (0 left) 2023-01-15T14:33:33.219 INFO:tasks.rados.rados.0.smithi090.stdout:1281: write (excl) oid 28 current snap is 134 2023-01-15T14:33:33.219 INFO:tasks.rados.rados.0.smithi090.stdout:1281: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2023-01-15T14:33:33.220 INFO:tasks.rados.rados.0.smithi090.stdout:1281: writing smithi09025264-28 from 632062 to 1195629 tid 1 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:1281: writing smithi09025264-28 from 1885903 to 2413720 tid 2 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:1281: writing smithi09025264-28 from 2830644 to 3551402 tid 3 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:1282: rollback oid 20 current snap is 134 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 20 to 131 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:1283: write (excl) oid 1 current snap is 134 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:1283: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:1283: writing smithi09025264-1 from 648869 to 1287685 tid 1 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:1283: writing smithi09025264-1 from 1828032 to 2615860 tid 2 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:1283: writing smithi09025264-1 from 3141714 to 3673744 tid 3 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:1284: rollback oid 30 current snap is 134 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 30 to 133 2023-01-15T14:33:33.221 INFO:tasks.rados.rados.0.smithi090.stdout:1285: delete oid 47 current snap is 134 2023-01-15T14:33:33.222 INFO:tasks.rados.rados.0.smithi090.stdout:1284: finishing rollback tid 0 to smithi09025264-30 2023-01-15T14:33:33.276 INFO:tasks.rados.rados.0.smithi090.stdout:1284: finishing rollback tid 1 to smithi09025264-30 2023-01-15T14:33:33.277 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:33.211+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:33.278 INFO:tasks.rados.rados.0.smithi090.stdout:1281: finishing write tid 1 to smithi09025264-28 2023-01-15T14:33:33.278 INFO:tasks.rados.rados.0.smithi090.stdout:1284: finishing rollback tid 2 to smithi09025264-30 2023-01-15T14:33:33.278 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 232 (ObjNum 449 snap 125 seq_num 449) dirty exists 2023-01-15T14:33:33.278 INFO:tasks.rados.rados.0.smithi090.stdout:1282: finishing rollback tid 0 to smithi09025264-20 2023-01-15T14:33:33.278 INFO:tasks.rados.rados.0.smithi090.stdout:1282: finishing rollback tid 1 to smithi09025264-20 2023-01-15T14:33:33.278 INFO:tasks.rados.rados.0.smithi090.stdout:1282: finishing rollback tid 2 to smithi09025264-20 2023-01-15T14:33:33.278 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 255 (ObjNum 451 snap 126 seq_num 451) dirty exists 2023-01-15T14:33:33.316 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:33.312+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:33.327 INFO:tasks.rados.rados.0.smithi090.stdout:1283: finishing write tid 1 to smithi09025264-1 2023-01-15T14:33:33.334 INFO:tasks.rados.rados.0.smithi090.stdout:1281: finishing write tid 2 to smithi09025264-28 2023-01-15T14:33:33.334 INFO:tasks.rados.rados.0.smithi090.stdout:1281: finishing write tid 3 to smithi09025264-28 2023-01-15T14:33:33.392 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:33.392 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:33.402 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:33:33.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:33:33.418 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:33.415+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:33.431 INFO:tasks.rados.rados.0.smithi090.stdout:1283: finishing write tid 2 to smithi09025264-1 2023-01-15T14:33:33.434 INFO:tasks.rados.rados.0.smithi090.stdout:1281: finishing write tid 4 to smithi09025264-28 2023-01-15T14:33:33.519 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:33.515+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:33.531 INFO:tasks.rados.rados.0.smithi090.stdout:1283: finishing write tid 3 to smithi09025264-1 2023-01-15T14:33:33.535 INFO:tasks.rados.rados.0.smithi090.stdout:1281: finishing write tid 5 to smithi09025264-28 2023-01-15T14:33:33.535 INFO:tasks.rados.rados.0.smithi090.stdout:1281: oid 28 updating version 0 to 368 2023-01-15T14:33:33.535 INFO:tasks.rados.rados.0.smithi090.stdout:1281: oid 28 version 368 is already newer than 366 2023-01-15T14:33:33.536 INFO:tasks.rados.rados.0.smithi090.stdout:1281: oid 28 version 368 is already newer than 367 2023-01-15T14:33:33.536 INFO:tasks.rados.rados.0.smithi090.stdout:1281: oid 28 version 368 is already newer than 365 2023-01-15T14:33:33.536 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 368 (ObjNum 466 snap 134 seq_num 466) dirty exists 2023-01-15T14:33:33.536 INFO:tasks.rados.rados.0.smithi090.stdout:1281: left oid 28 (ObjNum 466 snap 134 seq_num 466) 2023-01-15T14:33:33.538 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:33.538 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:33.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:33:33.620 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:33.615+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:33.632 INFO:tasks.rados.rados.0.smithi090.stdout:1283: finishing write tid 4 to smithi09025264-1 2023-01-15T14:33:33.632 INFO:tasks.rados.rados.0.smithi090.stdout:1283: finishing write tid 5 to smithi09025264-1 2023-01-15T14:33:33.632 INFO:tasks.rados.rados.0.smithi090.stdout:1283: oid 1 updating version 0 to 363 2023-01-15T14:33:33.632 INFO:tasks.rados.rados.0.smithi090.stdout:1283: oid 1 updating version 363 to 364 2023-01-15T14:33:33.632 INFO:tasks.rados.rados.0.smithi090.stdout:1283: oid 1 updating version 364 to 365 2023-01-15T14:33:33.632 INFO:tasks.rados.rados.0.smithi090.stdout:1283: oid 1 updating version 365 to 366 2023-01-15T14:33:33.632 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 366 (ObjNum 467 snap 134 seq_num 467) dirty exists 2023-01-15T14:33:33.633 INFO:tasks.rados.rados.0.smithi090.stdout:1283: left oid 1 (ObjNum 467 snap 134 seq_num 467) 2023-01-15T14:33:33.649 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:33.649 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:33.659 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:33:33.721 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:33.716+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:33.733 INFO:tasks.rados.rados.0.smithi090.stdout:1281: done (4 left) 2023-01-15T14:33:33.733 INFO:tasks.rados.rados.0.smithi090.stdout:1282: done (3 left) 2023-01-15T14:33:33.733 INFO:tasks.rados.rados.0.smithi090.stdout:1283: done (2 left) 2023-01-15T14:33:33.733 INFO:tasks.rados.rados.0.smithi090.stdout:1284: done (1 left) 2023-01-15T14:33:33.734 INFO:tasks.rados.rados.0.smithi090.stdout:1285: done (0 left) 2023-01-15T14:33:33.734 INFO:tasks.rados.rados.0.smithi090.stdout:1286: snap_create 2023-01-15T14:33:33.776 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:33.776 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:33.786 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:33:33.818 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:33:33.821 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:33.816+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:33.891 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:33.891 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:33.901 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:33:33.922 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:33.916+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:33.949 INFO:tasks.rados.rados.0.smithi090.stdout:1286: done (0 left) 2023-01-15T14:33:33.949 INFO:tasks.rados.rados.0.smithi090.stdout:1287: rollback oid 48 current snap is 135 2023-01-15T14:33:33.949 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 48 to 134 2023-01-15T14:33:33.949 INFO:tasks.rados.rados.0.smithi090.stdout:1288: rollback oid 2 current snap is 135 2023-01-15T14:33:33.949 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 2 to 111 2023-01-15T14:33:33.949 INFO:tasks.rados.rados.0.smithi090.stdout:1289: write (excl) oid 45 current snap is 135 2023-01-15T14:33:33.949 INFO:tasks.rados.rados.0.smithi090.stdout:1289: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2023-01-15T14:33:33.958 INFO:tasks.rados.rados.0.smithi090.stdout:1289: writing smithi09025264-45 from 682483 to 1471797 tid 1 2023-01-15T14:33:33.958 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:33:33.964 INFO:tasks.rados.rados.0.smithi090.stdout:1289: writing smithi09025264-45 from 1944996 to 2736493 tid 2 2023-01-15T14:33:33.971 INFO:tasks.rados.rados.0.smithi090.stdout:1289: writing smithi09025264-45 from 3196560 to 3751134 tid 3 2023-01-15T14:33:33.971 INFO:tasks.rados.rados.0.smithi090.stdout:1290: write oid 16 current snap is 135 2023-01-15T14:33:33.971 INFO:tasks.rados.rados.0.smithi090.stdout:1290: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2023-01-15T14:33:33.978 INFO:tasks.rados.rados.0.smithi090.stdout:1290: writing smithi09025264-16 from 699290 to 1163853 tid 1 2023-01-15T14:33:33.985 INFO:tasks.rados.rados.0.smithi090.stdout:1290: writing smithi09025264-16 from 1603478 to 2138633 tid 2 2023-01-15T14:33:33.992 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:33.994 INFO:tasks.rados.rados.0.smithi090.stdout:1290: writing smithi09025264-16 from 2823983 to 3589829 tid 3 2023-01-15T14:33:33.994 INFO:tasks.rados.rados.0.smithi090.stdout:1291: read oid 3 snap -1 2023-01-15T14:33:33.994 INFO:tasks.rados.rados.0.smithi090.stdout:1291: expect (ObjNum 464 snap 133 seq_num 464) 2023-01-15T14:33:33.994 INFO:tasks.rados.rados.0.smithi090.stdout:1292: rollback oid 42 current snap is 135 2023-01-15T14:33:33.994 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 42 to 134 2023-01-15T14:33:33.995 INFO:tasks.rados.rados.0.smithi090.stdout:1293: snap_create 2023-01-15T14:33:33.995 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:33:34.011 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:34.015 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:34.015 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:34.022 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:34.019+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:34.025 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:33:34.064 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:34.105 INFO:tasks.rados.rados.0.smithi090.stdout:1291: expect (ObjNum 464 snap 133 seq_num 464) 2023-01-15T14:33:34.123 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:34.120+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:34.172 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:34.176 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:34.176 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:34.186 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:33:34.190 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:34.214 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:34.229 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:34.224+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:34.232 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:34.251 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:34.253 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:33:34.253 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:33:34.254 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:33:34.254 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:33:34.254 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:33:34.254 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:33:34.254 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:33:34.254 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:33:34.289 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:33:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:33:34.289 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:33:34.289 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:34.299 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:33:34.371 INFO:tasks.rados.rados.0.smithi090.stdout:1290: finishing write tid 1 to smithi09025264-16 2023-01-15T14:33:34.371 INFO:tasks.rados.rados.0.smithi090.stdout:1290: finishing write tid 2 to smithi09025264-16 2023-01-15T14:33:34.371 INFO:tasks.rados.rados.0.smithi090.stdout:1290: finishing write tid 3 to smithi09025264-16 2023-01-15T14:33:34.372 INFO:tasks.rados.rados.0.smithi090.stdout:1292: finishing rollback tid 0 to smithi09025264-42 2023-01-15T14:33:34.372 INFO:tasks.rados.rados.0.smithi090.stdout:1291: done (5 left) 2023-01-15T14:33:34.372 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:33:34.372 INFO:tasks.rados.rados.0.smithi090.stdout:1290: finishing write tid 4 to smithi09025264-16 2023-01-15T14:33:34.372 INFO:tasks.rados.rados.0.smithi090.stdout:1292: finishing rollback tid 1 to smithi09025264-42 2023-01-15T14:33:34.372 INFO:tasks.rados.rados.0.smithi090.stdout:1292: finishing rollback tid 2 to smithi09025264-42 2023-01-15T14:33:34.372 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 42 v 358 (ObjNum 446 snap 123 seq_num 446) dirty exists 2023-01-15T14:33:34.373 INFO:tasks.rados.rados.0.smithi090.stdout:1292: done (4 left) 2023-01-15T14:33:34.373 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:33:34.374 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:34.368+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:34.457 INFO:tasks.rados.rados.0.smithi090.stdout:1287: finishing rollback tid 0 to smithi09025264-48 2023-01-15T14:33:34.489 INFO:tasks.rados.rados.0.smithi090.stdout:1287: finishing rollback tid 1 to smithi09025264-48 2023-01-15T14:33:34.489 INFO:tasks.rados.rados.0.smithi090.stdout:1290: finishing write tid 5 to smithi09025264-16 2023-01-15T14:33:34.489 INFO:tasks.rados.rados.0.smithi090.stdout:1290: oid 16 updating version 0 to 176 2023-01-15T14:33:34.489 INFO:tasks.rados.rados.0.smithi090.stdout:1290: oid 16 updating version 176 to 177 2023-01-15T14:33:34.489 INFO:tasks.rados.rados.0.smithi090.stdout:1290: oid 16 updating version 177 to 178 2023-01-15T14:33:34.490 INFO:tasks.rados.rados.0.smithi090.stdout:1290: oid 16 version 178 is already newer than 175 2023-01-15T14:33:34.490 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 178 (ObjNum 470 snap 135 seq_num 470) dirty exists 2023-01-15T14:33:34.490 INFO:tasks.rados.rados.0.smithi090.stdout:1290: left oid 16 (ObjNum 470 snap 135 seq_num 470) 2023-01-15T14:33:34.490 INFO:tasks.rados.rados.0.smithi090.stdout:1287: finishing rollback tid 2 to smithi09025264-48 2023-01-15T14:33:34.490 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 244 (ObjNum 436 snap 118 seq_num 436) dirty exists 2023-01-15T14:33:34.490 INFO:tasks.rados.rados.0.smithi090.stdout:1287: done (3 left) 2023-01-15T14:33:34.490 INFO:tasks.rados.rados.0.smithi090.stdout:1290: done (2 left) 2023-01-15T14:33:34.490 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:34.492 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:34.492 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:33:34.492 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:33:34.493 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:33:34.493 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.493 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:5 3.5 3:a1c47f99:::smithi09025264-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:34.493 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:33:34.493 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.70176617200002, 2023-01-15T14:33:34.493 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40548699700000002, 2023-01-15T14:33:34.493 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:34.493 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:34.493 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:34.493 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:34.493 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:34.494 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 5 2023-01-15T14:33:34.494 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.494 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:34.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.494 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:34.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:33:34.494 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.494 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.494 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:34.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:33:34.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.495 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.495 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:34.495 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781744+0000", 2023-01-15T14:33:34.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:34.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.495 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.495 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:34.496 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782566+0000", 2023-01-15T14:33:34.496 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000821578 2023-01-15T14:33:34.497 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.497 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.497 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:34.497 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782588+0000", 2023-01-15T14:33:34.497 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.226e-05 2023-01-15T14:33:34.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.498 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:34.498 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782601+0000", 2023-01-15T14:33:34.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2998e-05 2023-01-15T14:33:34.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882850+0000", 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100248908 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882871+0000", 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0873000000000002e-05 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083679+0000", 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200808083 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:34.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083948+0000", 2023-01-15T14:33:34.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000269342 2023-01-15T14:33:34.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:34.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084161+0000", 2023-01-15T14:33:34.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021326900000000001 2023-01-15T14:33:34.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:34.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084655+0000", 2023-01-15T14:33:34.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049371300000000001 2023-01-15T14:33:34.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:34.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084674+0000", 2023-01-15T14:33:34.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9012999999999999e-05 2023-01-15T14:33:34.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084924+0000", 2023-01-15T14:33:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024989899999999999 2023-01-15T14:33:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085221+0000", 2023-01-15T14:33:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029719600000000001 2023-01-15T14:33:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:34.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186953+0000", 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10173174 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187026+0000", 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2957e-05 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187124+0000", 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8146000000000005e-05 2023-01-15T14:33:34.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187233+0000", 2023-01-15T14:33:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010931599999999999 2023-01-15T14:33:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:34.559 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:6 3.5 3:a1c47f99:::smithi09025264-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:34.560 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:33:34.586 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.69651329999999, 2023-01-15T14:33:34.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50419954199999995, 2023-01-15T14:33:34.587 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:34.587 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:34.587 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:34.587 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:34.587 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:34.587 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 6 2023-01-15T14:33:34.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.587 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:34.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:34.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:33:34.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:34.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:33:34.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:34.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786997+0000", 2023-01-15T14:33:34.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:34.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:34.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787486+0000", 2023-01-15T14:33:34.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048882999999999999 2023-01-15T14:33:34.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:34.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787514+0000", 2023-01-15T14:33:34.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7591999999999999e-05 2023-01-15T14:33:34.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:34.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787526+0000", 2023-01-15T14:33:34.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1894e-05 2023-01-15T14:33:34.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882966+0000", 2023-01-15T14:33:34.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095440450999999996 2023-01-15T14:33:34.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:34.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882973+0000", 2023-01-15T14:33:34.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2570000000000001e-06 2023-01-15T14:33:34.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084809+0000", 2023-01-15T14:33:34.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201835763 2023-01-15T14:33:34.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:34.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084928+0000", 2023-01-15T14:33:34.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118946 2023-01-15T14:33:34.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187347+0000", 2023-01-15T14:33:34.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102419285 2023-01-15T14:33:34.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:34.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187450+0000", 2023-01-15T14:33:34.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010286299999999999 2023-01-15T14:33:34.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.693 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.693 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:34.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187735+0000", 2023-01-15T14:33:34.693 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028429900000000001 2023-01-15T14:33:34.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:34.694 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188113+0000", 2023-01-15T14:33:34.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037883200000000001 2023-01-15T14:33:34.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188138+0000", 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4110000000000001e-05 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188389+0000", 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025173900000000001 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:34.696 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189251+0000", 2023-01-15T14:33:34.696 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086129800000000003 2023-01-15T14:33:34.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.696 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:34.696 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290989+0000", 2023-01-15T14:33:34.696 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101738524 2023-01-15T14:33:34.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.697 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:34.697 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291075+0000", 2023-01-15T14:33:34.697 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5469000000000004e-05 2023-01-15T14:33:34.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.697 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:34.697 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291144+0000", 2023-01-15T14:33:34.698 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:33:34.698 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.698 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.698 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:34.698 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291199+0000", 2023-01-15T14:33:34.698 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4413000000000002e-05 2023-01-15T14:33:34.698 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:34.698 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:34.698 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:34.698 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.699 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.699 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:7 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:34.699 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:33:34.699 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.69596684300001, 2023-01-15T14:33:34.699 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50407251399999997, 2023-01-15T14:33:34.700 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:34.700 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:34.700 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:34.700 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:34.700 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:34.700 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 7 2023-01-15T14:33:34.700 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.700 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:34.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:34.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:33:34.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:34.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:33:34.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:34.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787545+0000", 2023-01-15T14:33:34.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:34.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:34.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787551+0000", 2023-01-15T14:33:34.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8669999999999999e-06 2023-01-15T14:33:34.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:34.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787559+0000", 2023-01-15T14:33:34.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9780000000000005e-06 2023-01-15T14:33:34.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.718 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:34.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787564+0000", 2023-01-15T14:33:34.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5709999999999997e-06 2023-01-15T14:33:34.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.718 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883054+0000", 2023-01-15T14:33:34.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095489722999999999 2023-01-15T14:33:34.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:34.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883061+0000", 2023-01-15T14:33:34.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2760000000000003e-06 2023-01-15T14:33:34.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084987+0000", 2023-01-15T14:33:34.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20192545000000001 2023-01-15T14:33:34.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:34.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085190+0000", 2023-01-15T14:33:34.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020288899999999999 2023-01-15T14:33:34.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188374+0000", 2023-01-15T14:33:34.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103184449 2023-01-15T14:33:34.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:34.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188543+0000", 2023-01-15T14:33:34.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016925299999999999 2023-01-15T14:33:34.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:34.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188748+0000", 2023-01-15T14:33:34.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020473600000000001 2023-01-15T14:33:34.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:34.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188986+0000", 2023-01-15T14:33:34.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023780900000000001 2023-01-15T14:33:34.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:34.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189006+0000", 2023-01-15T14:33:34.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0604000000000001e-05 2023-01-15T14:33:34.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:34.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189084+0000", 2023-01-15T14:33:34.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7526000000000003e-05 2023-01-15T14:33:34.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189272+0000", 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018762 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291333+0000", 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102061185 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:34.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291466+0000", 2023-01-15T14:33:34.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013325699999999999 2023-01-15T14:33:34.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:34.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291535+0000", 2023-01-15T14:33:34.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9309999999999999e-05 2023-01-15T14:33:34.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:34.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291618+0000", 2023-01-15T14:33:34.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2773000000000002e-05 2023-01-15T14:33:34.760 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:34.760 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:34.760 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:34.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.760 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:8 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:34.760 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:33:34.760 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.695940861, 2023-01-15T14:33:34.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50442076000000002, 2023-01-15T14:33:34.760 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:34.760 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:34.761 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:34.761 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:34.761 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:34.761 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 8 2023-01-15T14:33:34.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.761 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:34.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:34.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:33:34.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:34.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:33:34.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:34.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787571+0000", 2023-01-15T14:33:34.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:34.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787575+0000", 2023-01-15T14:33:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7890000000000001e-06 2023-01-15T14:33:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787589+0000", 2023-01-15T14:33:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4212999999999999e-05 2023-01-15T14:33:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787594+0000", 2023-01-15T14:33:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1429999999999997e-06 2023-01-15T14:33:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983143+0000", 2023-01-15T14:33:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.195549003 2023-01-15T14:33:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983153+0000", 2023-01-15T14:33:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4369999999999996e-06 2023-01-15T14:33:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085286+0000", 2023-01-15T14:33:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102133064 2023-01-15T14:33:34.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:34.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085401+0000", 2023-01-15T14:33:34.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115176 2023-01-15T14:33:34.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189116+0000", 2023-01-15T14:33:34.860 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103715444 2023-01-15T14:33:34.860 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.860 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:34.860 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189221+0000", 2023-01-15T14:33:34.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010438 2023-01-15T14:33:34.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.861 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.861 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.861 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291724+0000", 2023-01-15T14:33:34.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102502998 2023-01-15T14:33:34.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.862 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:34.862 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291824+0000", 2023-01-15T14:33:34.862 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010010599999999999 2023-01-15T14:33:34.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.862 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:34.862 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291992+0000", 2023-01-15T14:33:34.862 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016856900000000001 2023-01-15T14:33:34.862 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:34.863 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:34.863 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:34.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.863 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:21 3.5 3:a75958b1:::smithi09025264-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:34.863 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:33:34.863 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.65185194600002, 2023-01-15T14:33:34.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.45851924500000002, 2023-01-15T14:33:34.863 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:34.863 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:34.863 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 21 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.864 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831657+0000", 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832814+0000", 2023-01-15T14:33:34.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011564909999999999 2023-01-15T14:33:34.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:34.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832864+0000", 2023-01-15T14:33:34.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0243e-05 2023-01-15T14:33:34.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832903+0000", 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8704000000000001e-05 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983202+0000", 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.150299404 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983208+0000", 2023-01-15T14:33:34.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0039999999999996e-06 2023-01-15T14:33:34.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185539+0000", 2023-01-15T14:33:34.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202330867 2023-01-15T14:33:34.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:34.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185803+0000", 2023-01-15T14:33:34.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026394299999999999 2023-01-15T14:33:34.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186037+0000", 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023386300000000001 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186637+0000", 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060001800000000001 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:34.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186665+0000", 2023-01-15T14:33:34.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8459999999999999e-05 2023-01-15T14:33:34.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:34.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187045+0000", 2023-01-15T14:33:34.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037992899999999999 2023-01-15T14:33:34.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:34.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188201+0000", 2023-01-15T14:33:34.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001155309 2023-01-15T14:33:34.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.877 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.877 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:34.877 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289903+0000", 2023-01-15T14:33:34.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101702601 2023-01-15T14:33:34.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.877 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.877 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:34.877 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289985+0000", 2023-01-15T14:33:34.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1617000000000001e-05 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290065+0000", 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0276000000000002e-05 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290180+0000", 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011464300000000001 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:34.878 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:22 3.5 3:a75958b1:::smithi09025264-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.64738425799999, 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65790599599999999, 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:34.879 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:34.880 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 22 2023-01-15T14:33:34.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.914 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:34.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:34.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:33:34.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:34.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:33:34.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:34.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:34.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836126+0000", 2023-01-15T14:33:34.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:34.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:34.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836181+0000", 2023-01-15T14:33:34.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4246999999999997e-05 2023-01-15T14:33:34.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:34.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836212+0000", 2023-01-15T14:33:34.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1843e-05 2023-01-15T14:33:34.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:34.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836226+0000", 2023-01-15T14:33:34.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3216e-05 2023-01-15T14:33:34.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186757+0000", 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35053103899999999 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186871+0000", 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114403 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:34.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290263+0000", 2023-01-15T14:33:34.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103391976 2023-01-15T14:33:34.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:34.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290413+0000", 2023-01-15T14:33:34.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014991200000000001 2023-01-15T14:33:34.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:34.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290627+0000", 2023-01-15T14:33:34.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021379299999999999 2023-01-15T14:33:34.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:34.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290861+0000", 2023-01-15T14:33:34.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023479600000000001 2023-01-15T14:33:34.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.921 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:34.921 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290888+0000", 2023-01-15T14:33:34.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6570000000000001e-05 2023-01-15T14:33:34.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.921 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:34.921 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290990+0000", 2023-01-15T14:33:34.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101723 2023-01-15T14:33:34.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.921 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:34.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291245+0000", 2023-01-15T14:33:34.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025543899999999999 2023-01-15T14:33:34.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:34.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392822+0000", 2023-01-15T14:33:34.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10157725500000001 2023-01-15T14:33:34.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:34.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:34.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:34.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493881+0000", 2023-01-15T14:33:34.923 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101058262 2023-01-15T14:33:35.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:35.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493964+0000", 2023-01-15T14:33:35.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2973000000000007e-05 2023-01-15T14:33:35.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494034+0000", 2023-01-15T14:33:35.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0432999999999995e-05 2023-01-15T14:33:35.018 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.019 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.019 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.020 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:23 3.5 3:a75958b1:::smithi09025264-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:35.021 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:33:35.021 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.64726912899999, 2023-01-15T14:33:35.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65796116900000001, 2023-01-15T14:33:35.021 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.021 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:35.022 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:35.022 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:35.022 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:35.022 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 23 2023-01-15T14:33:35.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.022 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.022 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.022 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:33:35.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:33:35.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836242+0000", 2023-01-15T14:33:35.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:35.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836256+0000", 2023-01-15T14:33:35.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3677e-05 2023-01-15T14:33:35.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836265+0000", 2023-01-15T14:33:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2769999999999994e-06 2023-01-15T14:33:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936398+0000", 2023-01-15T14:33:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100132559 2023-01-15T14:33:35.044 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289431+0000", 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35303322399999998 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289587+0000", 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000155873 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.045 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292052+0000", 2023-01-15T14:33:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0024647919999999999 2023-01-15T14:33:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292180+0000", 2023-01-15T14:33:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012810999999999999 2023-01-15T14:33:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:35.046 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292373+0000", 2023-01-15T14:33:35.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019258599999999999 2023-01-15T14:33:35.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:35.047 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292597+0000", 2023-01-15T14:33:35.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022465500000000001 2023-01-15T14:33:35.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:35.047 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292618+0000", 2023-01-15T14:33:35.048 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0319999999999999e-05 2023-01-15T14:33:35.048 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.048 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.048 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:35.049 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292692+0000", 2023-01-15T14:33:35.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4643999999999999e-05 2023-01-15T14:33:35.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.049 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:35.049 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393103+0000", 2023-01-15T14:33:35.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100411156 2023-01-15T14:33:35.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.049 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:35.049 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493827+0000", 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100724143 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494085+0000", 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000257891 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494141+0000", 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6132999999999999e-05 2023-01-15T14:33:35.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.051 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494204+0000", 2023-01-15T14:33:35.051 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2966000000000007e-05 2023-01-15T14:33:35.051 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.051 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.051 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.051 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:24 3.5 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:35.051 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:33:35.052 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.54708770299999, 2023-01-15T14:33:35.052 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.55808677600000001, 2023-01-15T14:33:35.052 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.052 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:35.052 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:35.052 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:35.052 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:35.052 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 24 2023-01-15T14:33:35.052 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.052 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.052 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:33:35.078 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.078 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.078 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.079 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:33:35.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.079 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.079 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936423+0000", 2023-01-15T14:33:35.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:35.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936433+0000", 2023-01-15T14:33:35.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0443e-05 2023-01-15T14:33:35.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936451+0000", 2023-01-15T14:33:35.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7842999999999998e-05 2023-01-15T14:33:35.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936461+0000", 2023-01-15T14:33:35.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0057e-05 2023-01-15T14:33:35.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289693+0000", 2023-01-15T14:33:35.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.353231617 2023-01-15T14:33:35.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:35.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289814+0000", 2023-01-15T14:33:35.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120763 2023-01-15T14:33:35.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392923+0000", 2023-01-15T14:33:35.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10310955300000001 2023-01-15T14:33:35.087 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:35.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393072+0000", 2023-01-15T14:33:35.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000148469 2023-01-15T14:33:35.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494242+0000", 2023-01-15T14:33:35.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117034499999999 2023-01-15T14:33:35.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494363+0000", 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120703 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494511+0000", 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014873299999999999 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.091 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:78 3.c 3:3137f0cb:::smithi09025264-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:35.092 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:33:35.092 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.15712170699999, 2023-01-15T14:33:35.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20331801699999999, 2023-01-15T14:33:35.092 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.092 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:35.092 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:35.092 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:35.092 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:35.092 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 78 2023-01-15T14:33:35.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.093 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:33:35.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:33:35.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326388+0000", 2023-01-15T14:33:35.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:35.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.174 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.174 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.174 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326672+0000", 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028367900000000001 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326713+0000", 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1539999999999999e-05 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326729+0000", 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.519e-05 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.175 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326879+0000", 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015064899999999999 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327096+0000", 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021682299999999999 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:33:35.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327308+0000", 2023-01-15T14:33:35.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000211366 2023-01-15T14:33:35.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:35.177 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327757+0000", 2023-01-15T14:33:35.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000449009 2023-01-15T14:33:35.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:35.177 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327776+0000", 2023-01-15T14:33:35.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9201000000000001e-05 2023-01-15T14:33:35.178 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.178 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.178 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:35.178 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328012+0000", 2023-01-15T14:33:35.178 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236054 2023-01-15T14:33:35.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.179 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.179 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:35.180 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429534+0000", 2023-01-15T14:33:35.180 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10152201499999999 2023-01-15T14:33:35.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.180 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.180 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:35.180 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529340+0000", 2023-01-15T14:33:35.180 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099806249 2023-01-15T14:33:35.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.180 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.180 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:35.180 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529437+0000", 2023-01-15T14:33:35.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6949999999999998e-05 2023-01-15T14:33:35.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:35.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529587+0000", 2023-01-15T14:33:35.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015008600000000001 2023-01-15T14:33:35.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529709+0000", 2023-01-15T14:33:35.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121636 2023-01-15T14:33:35.182 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.182 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.182 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.182 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:79 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:35.182 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:33:35.182 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.156752111, 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20348171300000001, 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 79 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:33:35.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.184 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.184 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.184 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:33:35.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.184 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.184 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.184 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326759+0000", 2023-01-15T14:33:35.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326770+0000", 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326791+0000", 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1556000000000002e-05 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326818+0000", 2023-01-15T14:33:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6979999999999999e-05 2023-01-15T14:33:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.201 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.427993+0000", 2023-01-15T14:33:35.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117517199999999 2023-01-15T14:33:35.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428266+0000", 2023-01-15T14:33:35.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027233299999999998 2023-01-15T14:33:35.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:33:35.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428696+0000", 2023-01-15T14:33:35.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042996499999999998 2023-01-15T14:33:35.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:35.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429057+0000", 2023-01-15T14:33:35.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000360952 2023-01-15T14:33:35.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:35.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429076+0000", 2023-01-15T14:33:35.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:33:35.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:35.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429160+0000", 2023-01-15T14:33:35.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:33:35.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:35.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429549+0000", 2023-01-15T14:33:35.205 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038884599999999998 2023-01-15T14:33:35.205 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.205 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.205 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:35.205 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529886+0000", 2023-01-15T14:33:35.205 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100336261 2023-01-15T14:33:35.205 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.205 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:35.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529978+0000", 2023-01-15T14:33:35.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2775999999999994e-05 2023-01-15T14:33:35.207 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.208 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.208 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:35.208 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530108+0000", 2023-01-15T14:33:35.208 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012958300000000001 2023-01-15T14:33:35.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.208 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530242+0000", 2023-01-15T14:33:35.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013406 2023-01-15T14:33:35.209 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.209 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.209 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.209 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.209 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.209 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:80 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:35.209 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:33:35.210 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.156613752, 2023-01-15T14:33:35.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20370874, 2023-01-15T14:33:35.210 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.210 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:35.210 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:35.210 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:35.210 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:35.211 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 80 2023-01-15T14:33:35.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.212 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.212 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.212 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:33:35.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:33:35.213 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326897+0000", 2023-01-15T14:33:35.213 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:35.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326906+0000", 2023-01-15T14:33:35.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6929999999999996e-06 2023-01-15T14:33:35.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326931+0000", 2023-01-15T14:33:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4757e-05 2023-01-15T14:33:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326945+0000", 2023-01-15T14:33:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.402e-05 2023-01-15T14:33:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429239+0000", 2023-01-15T14:33:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102294418 2023-01-15T14:33:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429487+0000", 2023-01-15T14:33:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024789300000000001 2023-01-15T14:33:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530309+0000", 2023-01-15T14:33:35.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100821459 2023-01-15T14:33:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530438+0000", 2023-01-15T14:33:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012928700000000001 2023-01-15T14:33:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530607+0000", 2023-01-15T14:33:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169483 2023-01-15T14:33:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:78 3.c e23/20 ondisk, result = 0)", 2023-01-15T14:33:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:33:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.15472203100001, 2023-01-15T14:33:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20094287799999999, 2023-01-15T14:33:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328789+0000", 2023-01-15T14:33:35.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:35.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328799+0000", 2023-01-15T14:33:35.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8980000000000008e-06 2023-01-15T14:33:35.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328820+0000", 2023-01-15T14:33:35.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1041999999999999e-05 2023-01-15T14:33:35.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328832+0000", 2023-01-15T14:33:35.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1616e-05 2023-01-15T14:33:35.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529404+0000", 2023-01-15T14:33:35.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200572631 2023-01-15T14:33:35.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529419+0000", 2023-01-15T14:33:35.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4250000000000001e-05 2023-01-15T14:33:35.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529733+0000", 2023-01-15T14:33:35.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000314693 2023-01-15T14:33:35.232 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.232 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.232 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.233 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:201 3.5 3:a1c47f99:::smithi09025264-2:head [copy-from ver 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:35.233 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:33:35.233 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 307.07711563399999, 2023-01-15T14:33:35.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30931411399999997, 2023-01-15T14:33:35.233 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.233 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:35.233 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:35.233 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:35.234 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:35.234 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 201 2023-01-15T14:33:35.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.234 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.234 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.234 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:33:35.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:33:35.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.328 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406394+0000", 2023-01-15T14:33:35.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:35.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406408+0000", 2023-01-15T14:33:35.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4416999999999999e-05 2023-01-15T14:33:35.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406446+0000", 2023-01-15T14:33:35.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7339999999999998e-05 2023-01-15T14:33:35.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406467+0000", 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1137000000000001e-05 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406677+0000", 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000210275 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406902+0000", 2023-01-15T14:33:35.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022529000000000001 2023-01-15T14:33:35.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:35.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.513012+0000", 2023-01-15T14:33:35.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.106109527 2023-01-15T14:33:35.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:35.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514754+0000", 2023-01-15T14:33:35.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0017419779999999999 2023-01-15T14:33:35.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:35.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514773+0000", 2023-01-15T14:33:35.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8700000000000001e-05 2023-01-15T14:33:35.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:35.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.515531+0000", 2023-01-15T14:33:35.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075844300000000003 2023-01-15T14:33:35.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:35.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615334+0000", 2023-01-15T14:33:35.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099803327999999997 2023-01-15T14:33:35.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:35.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615376+0000", 2023-01-15T14:33:35.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1675999999999999e-05 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715546+0000", 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016992500000001 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715626+0000", 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0112999999999997e-05 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715711+0000", 2023-01-15T14:33:35.335 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:33:35.335 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.335 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.335 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.335 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.335 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.335 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:202 3.5 3:a1c47f99:::smithi09025264-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:35.335 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:33:35.335 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 307.07700308800003, 2023-01-15T14:33:35.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31002637300000002, 2023-01-15T14:33:35.336 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.336 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:35.336 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:35.336 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:35.336 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:35.336 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 202 2023-01-15T14:33:35.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.350 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:33:35.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406508+0000", 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406516+0000", 2023-01-15T14:33:35.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.413e-06 2023-01-15T14:33:35.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406534+0000", 2023-01-15T14:33:35.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8936999999999998e-05 2023-01-15T14:33:35.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406548+0000", 2023-01-15T14:33:35.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3196e-05 2023-01-15T14:33:35.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407271+0000", 2023-01-15T14:33:35.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00072301500000000003 2023-01-15T14:33:35.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:35.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407466+0000", 2023-01-15T14:33:35.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000195009 2023-01-15T14:33:35.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.614996+0000", 2023-01-15T14:33:35.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.207530192 2023-01-15T14:33:35.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:35.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615227+0000", 2023-01-15T14:33:35.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000230979 2023-01-15T14:33:35.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716282+0000", 2023-01-15T14:33:35.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101055465 2023-01-15T14:33:35.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716409+0000", 2023-01-15T14:33:35.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012653999999999999 2023-01-15T14:33:35.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716536+0000", 2023-01-15T14:33:35.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012683 2023-01-15T14:33:35.356 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.357 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.357 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.357 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:189 3.d e23/20 3:b3f111b5:::smithi09025264-48:head v 23'10, mlcod=23'6)", 2023-01-15T14:33:35.357 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:33:35.357 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 307.03179843999999, 2023-01-15T14:33:35.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20400475100000001, 2023-01-15T14:33:35.357 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.357 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:35.358 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:33:35.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:33:35.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451712+0000", 2023-01-15T14:33:35.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:35.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452272+0000", 2023-01-15T14:33:35.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00056035899999999997 2023-01-15T14:33:35.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452310+0000", 2023-01-15T14:33:35.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8145999999999997e-05 2023-01-15T14:33:35.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452324+0000", 2023-01-15T14:33:35.368 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4443999999999999e-05 2023-01-15T14:33:35.368 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652703+0000", 2023-01-15T14:33:35.368 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20037818199999999 2023-01-15T14:33:35.369 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652762+0000", 2023-01-15T14:33:35.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9302999999999998e-05 2023-01-15T14:33:35.369 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:35.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653302+0000", 2023-01-15T14:33:35.370 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054012199999999998 2023-01-15T14:33:35.370 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.370 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.370 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:35.370 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653333+0000", 2023-01-15T14:33:35.370 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.112e-05 2023-01-15T14:33:35.370 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.371 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.371 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:35.371 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653667+0000", 2023-01-15T14:33:35.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033337900000000002 2023-01-15T14:33:35.371 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.371 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.371 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:35.371 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653709+0000", 2023-01-15T14:33:35.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2469999999999998e-05 2023-01-15T14:33:35.371 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.372 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.372 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.372 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655719+0000", 2023-01-15T14:33:35.372 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020096699999999999 2023-01-15T14:33:35.372 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.372 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.373 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.373 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:269 3.2 e30/20 3:47dfe935:::smithi09025264-29:head v 30'11, mlcod=23'8)", 2023-01-15T14:33:35.373 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:33:35.373 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 299.605752999, 2023-01-15T14:33:35.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23185540399999999, 2023-01-15T14:33:35.373 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.373 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:35.374 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:33:35.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:33:35.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:33:35.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:35.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878379+0000", 2023-01-15T14:33:35.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061979600000000004 2023-01-15T14:33:35.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878406+0000", 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978515+0000", 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100108461 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078759+0000", 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100244496 2023-01-15T14:33:35.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078796+0000", 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6758999999999998e-05 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079567+0000", 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077137199999999999 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:35.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079593+0000", 2023-01-15T14:33:35.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:33:35.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:35.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079844+0000", 2023-01-15T14:33:35.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025075899999999998 2023-01-15T14:33:35.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:35.608 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079910+0000", 2023-01-15T14:33:35.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6406000000000001e-05 2023-01-15T14:33:35.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.109615+0000", 2023-01-15T14:33:35.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.029704323000000001 2023-01-15T14:33:35.609 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.609 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.609 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.609 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.609 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:281 3.5 3:a75958b1:::smithi09025264-6:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:33:35.609 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:33:35.609 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 299.50482936999998, 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30574310900000001, 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 281 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.610 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:33:35.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:33:35.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:33:35.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:35.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978686+0000", 2023-01-15T14:33:35.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4429999999999999e-06 2023-01-15T14:33:35.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978690+0000", 2023-01-15T14:33:35.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0300000000000004e-06 2023-01-15T14:33:35.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978693+0000", 2023-01-15T14:33:35.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.179e-06 2023-01-15T14:33:35.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981592+0000", 2023-01-15T14:33:35.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0028984140000000002 2023-01-15T14:33:35.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981718+0000", 2023-01-15T14:33:35.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012616600000000001 2023-01-15T14:33:35.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:35.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981982+0000", 2023-01-15T14:33:35.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026445199999999998 2023-01-15T14:33:35.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:35.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982278+0000", 2023-01-15T14:33:35.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029514100000000001 2023-01-15T14:33:35.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:35.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982290+0000", 2023-01-15T14:33:35.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.271e-05 2023-01-15T14:33:35.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:35.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982343+0000", 2023-01-15T14:33:35.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2914000000000002e-05 2023-01-15T14:33:35.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:35.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982385+0000", 2023-01-15T14:33:35.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2114999999999999e-05 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.181937+0000", 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19955213099999999 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284183+0000", 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10224538 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:35.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284318+0000", 2023-01-15T14:33:35.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000134843 2023-01-15T14:33:35.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284426+0000", 2023-01-15T14:33:35.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108513 2023-01-15T14:33:35.813 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.813 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.813 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.814 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:373 3.4 e34/20 3:2029fa71:::smithi09025264-39:head v 34'19, mlcod=26'16)", 2023-01-15T14:33:35.814 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:33:35.814 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 294.02384691100002, 2023-01-15T14:33:35.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20212622799999999, 2023-01-15T14:33:35.814 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.815 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:35.815 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:33:35.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:33:35.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459664+0000", 2023-01-15T14:33:35.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:35.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460099+0000", 2023-01-15T14:33:35.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043515900000000001 2023-01-15T14:33:35.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460130+0000", 2023-01-15T14:33:35.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1590000000000001e-05 2023-01-15T14:33:35.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560226+0000", 2023-01-15T14:33:35.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100095248 2023-01-15T14:33:35.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560369+0000", 2023-01-15T14:33:35.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014381499999999999 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560404+0000", 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4650000000000002e-05 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560804+0000", 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000399823 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560827+0000", 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3612999999999999e-05 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.561101+0000", 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027355499999999998 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661710+0000", 2023-01-15T14:33:35.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100608716 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661792+0000", 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2006000000000001e-05 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:455 3.d e41/20 3:bd982ca3:::smithi09025264-38:head v 41'24, mlcod=38'18)", 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 286.97408533800001, 2023-01-15T14:33:35.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20118033699999999, 2023-01-15T14:33:35.822 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.822 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:35.822 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:33:35.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:33:35.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509425+0000", 2023-01-15T14:33:35.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:35.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509441+0000", 2023-01-15T14:33:35.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5248e-05 2023-01-15T14:33:35.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509474+0000", 2023-01-15T14:33:35.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3531000000000001e-05 2023-01-15T14:33:35.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.609644+0000", 2023-01-15T14:33:35.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016994999999999 2023-01-15T14:33:35.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709872+0000", 2023-01-15T14:33:35.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10022840299999999 2023-01-15T14:33:35.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709906+0000", 2023-01-15T14:33:35.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3160000000000001e-05 2023-01-15T14:33:35.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:35.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710153+0000", 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024735399999999998 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710204+0000", 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1075999999999997e-05 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710280+0000", 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6202e-05 2023-01-15T14:33:35.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:35.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710317+0000", 2023-01-15T14:33:35.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6387000000000002e-05 2023-01-15T14:33:35.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710607+0000", 2023-01-15T14:33:35.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029079699999999997 2023-01-15T14:33:35.828 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.828 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.829 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.829 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:497 3.d e44/20 3:bd982ca3:::smithi09025264-38:head v 44'31, mlcod=42'26)", 2023-01-15T14:33:35.829 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:33:35.829 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 284.70929047599998, 2023-01-15T14:33:35.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20082067100000001, 2023-01-15T14:33:35.829 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:35.829 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:35.829 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.830 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:35.831 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774220+0000", 2023-01-15T14:33:35.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:35.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.831 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:35.831 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774238+0000", 2023-01-15T14:33:35.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8107000000000001e-05 2023-01-15T14:33:35.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.831 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:35.831 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774263+0000", 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4122999999999999e-05 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774280+0000", 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7039999999999999e-05 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874060+0000", 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099780698000000001 2023-01-15T14:33:35.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:35.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874130+0000", 2023-01-15T14:33:35.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9852999999999997e-05 2023-01-15T14:33:35.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:35.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874351+0000", 2023-01-15T14:33:35.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022091400000000001 2023-01-15T14:33:35.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:35.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874369+0000", 2023-01-15T14:33:35.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7876e-05 2023-01-15T14:33:35.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:35.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874433+0000", 2023-01-15T14:33:35.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3793000000000004e-05 2023-01-15T14:33:35.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.972 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:35.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.974927+0000", 2023-01-15T14:33:35.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10049393500000001 2023-01-15T14:33:35.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:35.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:35.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:35.974 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.975043+0000", 2023-01-15T14:33:35.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115949 2023-01-15T14:33:35.974 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.974 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.974 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.974 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:35.975 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:35.975 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:35.976 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:34.487+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:35.977 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:33:35.978 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:33:35.981 INFO:tasks.rados.rados.0.smithi090.stdout:1288: finishing rollback tid 0 to smithi09025264-2 2023-01-15T14:33:35.981 INFO:tasks.rados.rados.0.smithi090.stdout:1288: finishing rollback tid 1 to smithi09025264-2 2023-01-15T14:33:35.985 INFO:tasks.rados.rados.0.smithi090.stdout:1288: finishing rollback tid 2 to smithi09025264-2 2023-01-15T14:33:35.985 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 382 (ObjNum 328 snap 87 seq_num 328) dirty exists 2023-01-15T14:33:35.985 INFO:tasks.rados.rados.0.smithi090.stdout:1288: done (1 left) 2023-01-15T14:33:35.985 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:35.985 INFO:tasks.rados.rados.0.smithi090.stdout:1289: finishing write tid 1 to smithi09025264-45 2023-01-15T14:33:35.985 INFO:tasks.rados.rados.0.smithi090.stdout:1289: finishing write tid 2 to smithi09025264-45 2023-01-15T14:33:35.985 INFO:tasks.rados.rados.0.smithi090.stdout:1289: finishing write tid 3 to smithi09025264-45 2023-01-15T14:33:35.985 INFO:tasks.rados.rados.0.smithi090.stdout:1289: finishing write tid 4 to smithi09025264-45 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:1289: finishing write tid 5 to smithi09025264-45 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:1289: oid 45 updating version 0 to 143 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:1289: oid 45 version 143 is already newer than 141 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:1289: oid 45 updating version 143 to 144 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:1289: oid 45 version 144 is already newer than 142 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 144 (ObjNum 469 snap 135 seq_num 469) dirty exists 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:1289: left oid 45 (ObjNum 469 snap 135 seq_num 469) 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:1289: done (0 left) 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:1293: done (0 left) 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:1294: write oid 22 current snap is 136 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:1294: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2023-01-15T14:33:35.986 INFO:tasks.rados.rados.0.smithi090.stdout:1294: writing smithi09025264-22 from 716097 to 1255909 tid 1 2023-01-15T14:33:35.987 INFO:tasks.rados.rados.0.smithi090.stdout:1294: writing smithi09025264-22 from 1945607 to 2740773 tid 2 2023-01-15T14:33:35.988 INFO:tasks.rados.rados.0.smithi090.stdout:1294: writing smithi09025264-22 from 3535053 to 3916097 tid 3 2023-01-15T14:33:35.988 INFO:tasks.rados.rados.0.smithi090.stdout:1295: read oid 1 snap -1 2023-01-15T14:33:35.988 INFO:tasks.rados.rados.0.smithi090.stdout:1295: expect (ObjNum 467 snap 134 seq_num 467) 2023-01-15T14:33:35.989 INFO:tasks.rados.rados.0.smithi090.stdout:1296: snap_create 2023-01-15T14:33:35.989 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:36.021 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:36.019+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:36.141 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:36.141 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:33:36.141 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:33:36.141 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:36.141 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:36.139+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:36.142 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:33:36.181 INFO:tasks.rados.rados.0.smithi090.stdout:1295: expect (ObjNum 467 snap 134 seq_num 467) 2023-01-15T14:33:36.244 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:36.240+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:36.253 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:36.253 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:33:36.253 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:33:36.253 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:33:36.254 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:36.263 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:33:36.422 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:33:36.423 INFO:tasks.rados.rados.0.smithi090.stdout:1294: finishing write tid 1 to smithi09025264-22 2023-01-15T14:33:36.424 INFO:tasks.rados.rados.0.smithi090.stdout:1295: done (1 left) 2023-01-15T14:33:36.424 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:36.424 INFO:tasks.rados.rados.0.smithi090.stdout:1294: finishing write tid 2 to smithi09025264-22 2023-01-15T14:33:36.425 INFO:tasks.rados.rados.0.smithi090.stdout:1294: finishing write tid 3 to smithi09025264-22 2023-01-15T14:33:36.426 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:36.419+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:36.524 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:36.520+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:36.627 INFO:tasks.rados.rados.0.smithi090.stdout:1294: finishing write tid 4 to smithi09025264-22 2023-01-15T14:33:36.628 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:36.628 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:36.628 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:36.629 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:36.840 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:33:36.840 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:33:36.841 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:33:36.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.841 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:97 3.a 3:52df5138:::smithi09025264-25:head [write 403368~555035,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:36.841 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:33:36.841 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 310.03540796800002, 2023-01-15T14:33:36.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.340758171, 2023-01-15T14:33:36.841 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:36.841 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:36.841 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:36.841 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:36.842 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:36.842 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 97 2023-01-15T14:33:36.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.842 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:36.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:36.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:33:36.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:36.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:36.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:33:36.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:36.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:36.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501140+0000", 2023-01-15T14:33:36.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:36.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:36.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501684+0000", 2023-01-15T14:33:36.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054441799999999996 2023-01-15T14:33:36.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:36.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501713+0000", 2023-01-15T14:33:36.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8833e-05 2023-01-15T14:33:36.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:36.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501726+0000", 2023-01-15T14:33:36.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3137e-05 2023-01-15T14:33:36.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:36.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601982+0000", 2023-01-15T14:33:36.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100256518 2023-01-15T14:33:36.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:36.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602426+0000", 2023-01-15T14:33:36.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044401500000000002 2023-01-15T14:33:36.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:36.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602703+0000", 2023-01-15T14:33:36.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000276889 2023-01-15T14:33:36.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:36.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603066+0000", 2023-01-15T14:33:36.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036278599999999998 2023-01-15T14:33:36.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:36.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603098+0000", 2023-01-15T14:33:36.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1433000000000002e-05 2023-01-15T14:33:36.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:36.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603382+0000", 2023-01-15T14:33:36.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028402899999999999 2023-01-15T14:33:36.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:36.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603632+0000", 2023-01-15T14:33:36.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025033 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805354+0000", 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20172168700000001 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841465+0000", 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.036111852999999999 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:36.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841682+0000", 2023-01-15T14:33:36.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000216949 2023-01-15T14:33:36.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:36.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841901+0000", 2023-01-15T14:33:36.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021858700000000001 2023-01-15T14:33:36.850 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:36.850 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:36.852 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:36.852 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.852 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.853 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:98 3.a 3:52df5138:::smithi09025264-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:36.853 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:33:36.853 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 310.03479829700001, 2023-01-15T14:33:36.853 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.44410839299999999, 2023-01-15T14:33:36.853 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:36.853 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:36.853 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:36.853 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:36.853 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:36.853 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 98 2023-01-15T14:33:36.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:36.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501752+0000", 2023-01-15T14:33:36.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:36.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:36.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502223+0000", 2023-01-15T14:33:36.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000471843 2023-01-15T14:33:36.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:36.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502240+0000", 2023-01-15T14:33:36.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6663e-05 2023-01-15T14:33:36.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:36.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502248+0000", 2023-01-15T14:33:36.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5669999999999997e-06 2023-01-15T14:33:36.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.857 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:36.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603236+0000", 2023-01-15T14:33:36.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100988439 2023-01-15T14:33:36.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.857 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:36.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603384+0000", 2023-01-15T14:33:36.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014844200000000001 2023-01-15T14:33:36.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.858 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:36.858 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842016+0000", 2023-01-15T14:33:36.858 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23863163800000001 2023-01-15T14:33:36.858 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:36.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842267+0000", 2023-01-15T14:33:36.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025101300000000002 2023-01-15T14:33:36.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.860 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:36.860 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842561+0000", 2023-01-15T14:33:36.860 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029432900000000002 2023-01-15T14:33:36.860 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.860 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:36.861 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843120+0000", 2023-01-15T14:33:36.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055900500000000001 2023-01-15T14:33:36.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.861 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.861 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:36.861 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843146+0000", 2023-01-15T14:33:36.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5273e-05 2023-01-15T14:33:36.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.861 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.861 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:36.862 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843512+0000", 2023-01-15T14:33:36.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036621599999999999 2023-01-15T14:33:36.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:36.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943324+0000", 2023-01-15T14:33:36.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099812008999999993 2023-01-15T14:33:36.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:36.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945437+0000", 2023-01-15T14:33:36.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021126069999999999 2023-01-15T14:33:36.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:36.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945619+0000", 2023-01-15T14:33:36.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018280600000000001 2023-01-15T14:33:36.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:36.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945744+0000", 2023-01-15T14:33:36.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012511299999999999 2023-01-15T14:33:36.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.869 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.869 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:36.869 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945861+0000", 2023-01-15T14:33:36.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011646000000000001 2023-01-15T14:33:36.869 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:36.869 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:36.869 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:36.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.869 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.870 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:99 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:36.870 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:33:36.870 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 310.03428804100002, 2023-01-15T14:33:36.870 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.54355587900000002, 2023-01-15T14:33:36.870 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:36.870 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:36.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:36.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:36.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:36.870 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 99 2023-01-15T14:33:36.870 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:36.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502262+0000", 2023-01-15T14:33:36.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:36.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:36.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502270+0000", 2023-01-15T14:33:36.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1999999999999994e-06 2023-01-15T14:33:36.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:36.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502286+0000", 2023-01-15T14:33:36.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5423000000000001e-05 2023-01-15T14:33:36.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:36.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502295+0000", 2023-01-15T14:33:36.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4339999999999998e-06 2023-01-15T14:33:36.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:36.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603467+0000", 2023-01-15T14:33:36.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101172225 2023-01-15T14:33:36.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:36.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603585+0000", 2023-01-15T14:33:36.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117149 2023-01-15T14:33:36.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:36.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:36.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:36.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843335+0000", 2023-01-15T14:33:36.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23975083799999999 2023-01-15T14:33:36.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843573+0000", 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023753000000000001 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843978+0000", 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040533799999999999 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844309+0000", 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000330713 2023-01-15T14:33:37.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844423+0000", 2023-01-15T14:33:37.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114176 2023-01-15T14:33:37.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844510+0000", 2023-01-15T14:33:37.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7026000000000003e-05 2023-01-15T14:33:37.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943338+0000", 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098827398999999996 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945520+0000", 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021824169999999999 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045652+0000", 2023-01-15T14:33:37.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013193200000001 2023-01-15T14:33:37.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045733+0000", 2023-01-15T14:33:37.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1435999999999998e-05 2023-01-15T14:33:37.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045819+0000", 2023-01-15T14:33:37.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5292999999999998e-05 2023-01-15T14:33:37.039 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.039 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.040 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.040 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:100 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:37.040 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:33:37.040 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 310.03424011800001, 2023-01-15T14:33:37.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.64425290599999996, 2023-01-15T14:33:37.040 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.040 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:37.041 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.041 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.041 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.041 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 100 2023-01-15T14:33:37.041 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.041 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.041 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.041 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.041 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:33:37.041 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.041 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.042 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.042 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:33:37.042 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.042 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.042 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.042 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502310+0000", 2023-01-15T14:33:37.042 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:37.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502316+0000", 2023-01-15T14:33:37.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7470000000000002e-06 2023-01-15T14:33:37.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.044 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502332+0000", 2023-01-15T14:33:37.044 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6866000000000001e-05 2023-01-15T14:33:37.045 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.045 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.046 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502343+0000", 2023-01-15T14:33:37.046 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0913e-05 2023-01-15T14:33:37.046 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.046 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.047 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703594+0000", 2023-01-15T14:33:37.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20125087999999999 2023-01-15T14:33:37.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:37.047 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703751+0000", 2023-01-15T14:33:37.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001573 2023-01-15T14:33:37.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.048 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845262+0000", 2023-01-15T14:33:37.048 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.141510521 2023-01-15T14:33:37.048 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.048 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.048 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:37.048 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845391+0000", 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012872299999999999 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146051+0000", 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.300660023 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146264+0000", 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212754 2023-01-15T14:33:37.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.050 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.051 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146564+0000", 2023-01-15T14:33:37.051 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030010200000000001 2023-01-15T14:33:37.051 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.051 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.051 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.051 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:203 3.a 3:52df5138:::smithi09025264-25:head [write 440350~460568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:37.051 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:33:37.051 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.98343672300001, 2023-01-15T14:33:37.052 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40523678899999999, 2023-01-15T14:33:37.052 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.052 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.052 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.052 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.052 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.052 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 203 2023-01-15T14:33:37.052 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.052 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.052 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.052 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:33:37.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:33:37.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553111+0000", 2023-01-15T14:33:37.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:37.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553483+0000", 2023-01-15T14:33:37.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037153499999999999 2023-01-15T14:33:37.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553517+0000", 2023-01-15T14:33:37.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4687000000000002e-05 2023-01-15T14:33:37.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553537+0000", 2023-01-15T14:33:37.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.967e-05 2023-01-15T14:33:37.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553683+0000", 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000146341 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553880+0000", 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019671900000000001 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554219+0000", 2023-01-15T14:33:37.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033893299999999998 2023-01-15T14:33:37.057 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.057 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554536+0000", 2023-01-15T14:33:37.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316576 2023-01-15T14:33:37.057 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.057 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554562+0000", 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5905999999999999e-05 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554801+0000", 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000239791 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555274+0000", 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047294500000000001 2023-01-15T14:33:37.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.059 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.059 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.059 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757267+0000", 2023-01-15T14:33:37.059 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20199229399999999 2023-01-15T14:33:37.059 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.059 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.059 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.059 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958174+0000", 2023-01-15T14:33:37.059 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20090739499999999 2023-01-15T14:33:37.059 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.059 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.060 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.060 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958260+0000", 2023-01-15T14:33:37.060 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6316999999999994e-05 2023-01-15T14:33:37.060 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.061 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.061 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.061 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958351+0000", 2023-01-15T14:33:37.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0573000000000002e-05 2023-01-15T14:33:37.062 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.062 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.062 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.063 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:204 3.a 3:52df5138:::smithi09025264-25:head [write 1309629~542666] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:37.063 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:33:37.063 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.98297385500001, 2023-01-15T14:33:37.063 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40500913999999999, 2023-01-15T14:33:37.063 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.063 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.063 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.064 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.064 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.064 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 204 2023-01-15T14:33:37.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.064 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.064 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.064 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.064 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:33:37.064 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.064 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:33:37.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553575+0000", 2023-01-15T14:33:37.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:37.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554126+0000", 2023-01-15T14:33:37.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055090499999999997 2023-01-15T14:33:37.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554150+0000", 2023-01-15T14:33:37.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3387000000000002e-05 2023-01-15T14:33:37.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554165+0000", 2023-01-15T14:33:37.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5116000000000001e-05 2023-01-15T14:33:37.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554626+0000", 2023-01-15T14:33:37.068 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046093499999999999 2023-01-15T14:33:37.068 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.068 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.068 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.068 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554738+0000", 2023-01-15T14:33:37.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112446 2023-01-15T14:33:37.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.069 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.069 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554898+0000", 2023-01-15T14:33:37.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016003100000000001 2023-01-15T14:33:37.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.070 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.070 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555182+0000", 2023-01-15T14:33:37.070 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028382700000000001 2023-01-15T14:33:37.070 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555199+0000", 2023-01-15T14:33:37.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6969e-05 2023-01-15T14:33:37.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555449+0000", 2023-01-15T14:33:37.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000250265 2023-01-15T14:33:37.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555574+0000", 2023-01-15T14:33:37.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124628 2023-01-15T14:33:37.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757316+0000", 2023-01-15T14:33:37.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20174214300000001 2023-01-15T14:33:37.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.248 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.248 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958460+0000", 2023-01-15T14:33:37.248 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201144464 2023-01-15T14:33:37.248 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.248 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.248 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.248 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958534+0000", 2023-01-15T14:33:37.248 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3380000000000006e-05 2023-01-15T14:33:37.248 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.248 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.248 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958586+0000", 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2370000000000002e-05 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:205 3.a 3:52df5138:::smithi09025264-25:head [write 2323973~516377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 308.98235576000002, 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404601398, 2023-01-15T14:33:37.249 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.250 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.250 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.250 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.250 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.250 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 205 2023-01-15T14:33:37.250 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.250 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.250 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.250 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.250 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:33:37.250 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.251 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.251 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.251 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.251 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:33:37.251 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.251 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.252 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.252 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.252 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554193+0000", 2023-01-15T14:33:37.252 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:37.252 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.252 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.252 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.252 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554850+0000", 2023-01-15T14:33:37.252 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065674799999999997 2023-01-15T14:33:37.252 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.252 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.253 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.253 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554873+0000", 2023-01-15T14:33:37.253 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2935999999999999e-05 2023-01-15T14:33:37.255 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.255 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.255 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.255 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.654962+0000", 2023-01-15T14:33:37.255 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088949 2023-01-15T14:33:37.255 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.255 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.256 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655107+0000", 2023-01-15T14:33:37.256 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144759 2023-01-15T14:33:37.256 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.256 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655373+0000", 2023-01-15T14:33:37.256 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026684899999999999 2023-01-15T14:33:37.256 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.257 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655720+0000", 2023-01-15T14:33:37.257 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034601899999999998 2023-01-15T14:33:37.257 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.257 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.257 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.257 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656168+0000", 2023-01-15T14:33:37.257 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044851899999999998 2023-01-15T14:33:37.257 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.257 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.257 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.258 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656190+0000", 2023-01-15T14:33:37.258 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1817000000000001e-05 2023-01-15T14:33:37.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.258 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.258 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.258 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656574+0000", 2023-01-15T14:33:37.258 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038404200000000002 2023-01-15T14:33:37.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.258 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.258 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.258 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657649+0000", 2023-01-15T14:33:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010746670000000001 2023-01-15T14:33:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757539+0000", 2023-01-15T14:33:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099890899000000005 2023-01-15T14:33:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958683+0000", 2023-01-15T14:33:37.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20114332400000001 2023-01-15T14:33:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958746+0000", 2023-01-15T14:33:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3509000000000005e-05 2023-01-15T14:33:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958796+0000", 2023-01-15T14:33:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0176999999999997e-05 2023-01-15T14:33:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:358 3.a 3:52df5138:::smithi09025264-25:head [write 709262~697258] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:37.261 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:33:37.262 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 296.48576822699999, 2023-01-15T14:33:37.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50589529799999999, 2023-01-15T14:33:37.262 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.262 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.262 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.262 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.262 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.262 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 358 2023-01-15T14:33:37.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.263 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:33:37.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:33:37.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050780+0000", 2023-01-15T14:33:37.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:37.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051369+0000", 2023-01-15T14:33:37.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058926200000000001 2023-01-15T14:33:37.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051390+0000", 2023-01-15T14:33:37.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1426000000000001e-05 2023-01-15T14:33:37.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051403+0000", 2023-01-15T14:33:37.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:33:37.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051520+0000", 2023-01-15T14:33:37.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117318 2023-01-15T14:33:37.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051785+0000", 2023-01-15T14:33:37.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000265018 2023-01-15T14:33:37.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.268 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.268 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052173+0000", 2023-01-15T14:33:37.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038734799999999998 2023-01-15T14:33:37.268 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.268 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.268 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052705+0000", 2023-01-15T14:33:37.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053209899999999996 2023-01-15T14:33:37.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.269 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.269 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052727+0000", 2023-01-15T14:33:37.269 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2362e-05 2023-01-15T14:33:37.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.269 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.269 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053205+0000", 2023-01-15T14:33:37.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047787200000000001 2023-01-15T14:33:37.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.270 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055711+0000", 2023-01-15T14:33:37.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0025054880000000002 2023-01-15T14:33:37.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.270 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.255549+0000", 2023-01-15T14:33:37.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199838285 2023-01-15T14:33:37.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556501+0000", 2023-01-15T14:33:37.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30095197499999998 2023-01-15T14:33:37.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556613+0000", 2023-01-15T14:33:37.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111799 2023-01-15T14:33:37.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556678+0000", 2023-01-15T14:33:37.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5220000000000002e-05 2023-01-15T14:33:37.274 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.274 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.274 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.274 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:359 3.a 3:52df5138:::smithi09025264-25:head [write 2012635~572018] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:37.274 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:33:37.275 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 296.48511763300002, 2023-01-15T14:33:37.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50546442499999999, 2023-01-15T14:33:37.275 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.275 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.275 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.275 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.275 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.275 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 359 2023-01-15T14:33:37.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.276 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:33:37.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:33:37.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051432+0000", 2023-01-15T14:33:37.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:37.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051808+0000", 2023-01-15T14:33:37.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000376074 2023-01-15T14:33:37.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051834+0000", 2023-01-15T14:33:37.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6018999999999999e-05 2023-01-15T14:33:37.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051869+0000", 2023-01-15T14:33:37.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4839999999999998e-05 2023-01-15T14:33:37.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052857+0000", 2023-01-15T14:33:37.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00098761399999999994 2023-01-15T14:33:37.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.281 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.281 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053071+0000", 2023-01-15T14:33:37.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000214112 2023-01-15T14:33:37.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.281 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.281 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053876+0000", 2023-01-15T14:33:37.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000804873 2023-01-15T14:33:37.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.282 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.282 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054460+0000", 2023-01-15T14:33:37.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058436100000000004 2023-01-15T14:33:37.282 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.282 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.282 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054486+0000", 2023-01-15T14:33:37.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069999999999999e-05 2023-01-15T14:33:37.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.283 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.283 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054773+0000", 2023-01-15T14:33:37.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028716700000000001 2023-01-15T14:33:37.283 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055736+0000", 2023-01-15T14:33:37.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00096295500000000004 2023-01-15T14:33:37.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.355784+0000", 2023-01-15T14:33:37.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30004825899999998 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556757+0000", 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20097241199999999 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556830+0000", 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3633000000000005e-05 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556898+0000", 2023-01-15T14:33:37.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7212999999999995e-05 2023-01-15T14:33:37.449 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.449 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.449 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.449 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:360 3.a 3:52df5138:::smithi09025264-25:head [write 3046388~62874] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:37.449 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:33:37.449 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 296.484664877, 2023-01-15T14:33:37.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50519877000000002, 2023-01-15T14:33:37.450 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.450 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.450 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.450 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.450 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.450 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 360 2023-01-15T14:33:37.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.450 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:33:37.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:33:37.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051885+0000", 2023-01-15T14:33:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051935+0000", 2023-01-15T14:33:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9826e-05 2023-01-15T14:33:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051958+0000", 2023-01-15T14:33:37.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3003e-05 2023-01-15T14:33:37.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051965+0000", 2023-01-15T14:33:37.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6129999999999998e-06 2023-01-15T14:33:37.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054655+0000", 2023-01-15T14:33:37.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0026896860000000002 2023-01-15T14:33:37.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.454 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.454 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054869+0000", 2023-01-15T14:33:37.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021348600000000001 2023-01-15T14:33:37.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055198+0000", 2023-01-15T14:33:37.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032968199999999998 2023-01-15T14:33:37.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055605+0000", 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040634200000000002 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055622+0000", 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7020999999999998e-05 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055702+0000", 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0835999999999997e-05 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055754+0000", 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1175999999999999e-05 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.455996+0000", 2023-01-15T14:33:37.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40024198599999999 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556952+0000", 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10095644400000001 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557019+0000", 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6746000000000001e-05 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557085+0000", 2023-01-15T14:33:37.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5885999999999999e-05 2023-01-15T14:33:37.463 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.463 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.463 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.463 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:361 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 3109262] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:37.463 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:33:37.463 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 296.48457497099997, 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60469846000000005, 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 361 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:33:37.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:33:37.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051975+0000", 2023-01-15T14:33:37.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051980+0000", 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2170000000000002e-06 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051989+0000", 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5299999999999996e-06 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051995+0000", 2023-01-15T14:33:37.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1369999999999999e-06 2023-01-15T14:33:37.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055830+0000", 2023-01-15T14:33:37.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0038348549999999999 2023-01-15T14:33:37.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056101+0000", 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027095800000000002 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056560+0000", 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045868599999999998 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056954+0000", 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039487600000000001 2023-01-15T14:33:37.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057004+0000", 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9490000000000002e-05 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057072+0000", 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8502000000000002e-05 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.158224+0000", 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101151422 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556201+0000", 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39797714699999998 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656421+0000", 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100219549 2023-01-15T14:33:37.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656515+0000", 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4048999999999993e-05 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656674+0000", 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000159696 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.471 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:362 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 296.48454754099998, 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60517504499999997, 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.472 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 362 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:33:37.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:37.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052007+0000", 2023-01-15T14:33:37.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.9759999999999997e-06 2023-01-15T14:33:37.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052016+0000", 2023-01-15T14:33:37.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4069999999999999e-06 2023-01-15T14:33:37.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052022+0000", 2023-01-15T14:33:37.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4670000000000001e-06 2023-01-15T14:33:37.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157159+0000", 2023-01-15T14:33:37.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.105136825 2023-01-15T14:33:37.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:37.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157325+0000", 2023-01-15T14:33:37.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000165262 2023-01-15T14:33:37.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656814+0000", 2023-01-15T14:33:37.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49948963200000002 2023-01-15T14:33:37.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656993+0000", 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000178371 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.657178+0000", 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185825 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:601 3.a 3:5ba385f1:::smithi09025264-44:head [setxattr _header (57) in=7b,truncate 3663893] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 276.59422490499998, 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404742724, 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 601 2023-01-15T14:33:37.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942323+0000", 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942341+0000", 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7659000000000002e-05 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942365+0000", 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4256999999999998e-05 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042495+0000", 2023-01-15T14:33:37.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013043200000001 2023-01-15T14:33:37.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.144809+0000", 2023-01-15T14:33:37.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102314184 2023-01-15T14:33:37.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.840 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145098+0000", 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000289036 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145363+0000", 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026449299999999998 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145657+0000", 2023-01-15T14:33:37.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029433600000000002 2023-01-15T14:33:37.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145678+0000", 2023-01-15T14:33:37.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0939000000000001e-05 2023-01-15T14:33:37.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145853+0000", 2023-01-15T14:33:37.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017511800000000001 2023-01-15T14:33:37.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146557+0000", 2023-01-15T14:33:37.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00070337899999999996 2023-01-15T14:33:37.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146845+0000", 2023-01-15T14:33:37.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028790599999999999 2023-01-15T14:33:37.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346819+0000", 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199974661 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346919+0000", 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9234999999999994e-05 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.347069+0000", 2023-01-15T14:33:37.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150128 2023-01-15T14:33:37.845 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.845 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.845 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.845 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.845 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1153 3.a 3:5919d83e:::smithi09025264-13:head [write 1566383~595717] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:33:37.845 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:33:37.845 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 230.74599985099999, 2023-01-15T14:33:37.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.38805106299999997, 2023-01-15T14:33:37.845 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.846 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.846 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.846 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.852 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.852 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1153 2023-01-15T14:33:37.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.853 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.853 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.853 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:33:37.853 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:33:37.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790531+0000", 2023-01-15T14:33:37.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999795 2023-01-15T14:33:37.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791189+0000", 2023-01-15T14:33:37.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065897000000000004 2023-01-15T14:33:37.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791215+0000", 2023-01-15T14:33:37.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5700000000000001e-05 2023-01-15T14:33:37.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.857 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791232+0000", 2023-01-15T14:33:37.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6546e-05 2023-01-15T14:33:37.858 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.858 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.858 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791392+0000", 2023-01-15T14:33:37.858 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016051699999999999 2023-01-15T14:33:37.858 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.858 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:37.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791408+0000", 2023-01-15T14:33:37.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5566999999999999e-05 2023-01-15T14:33:37.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.974825+0000", 2023-01-15T14:33:37.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18341675599999999 2023-01-15T14:33:37.860 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.860 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.860 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975000+0000", 2023-01-15T14:33:37.860 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017548299999999999 2023-01-15T14:33:37.860 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.861 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.861 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975255+0000", 2023-01-15T14:33:37.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254658 2023-01-15T14:33:37.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.861 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.861 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.861 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975914+0000", 2023-01-15T14:33:37.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065881800000000003 2023-01-15T14:33:37.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.862 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.862 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975931+0000", 2023-01-15T14:33:37.862 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7523000000000001e-05 2023-01-15T14:33:37.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976267+0000", 2023-01-15T14:33:37.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033545299999999999 2023-01-15T14:33:37.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978320+0000", 2023-01-15T14:33:37.864 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020537110000000002 2023-01-15T14:33:37.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.864 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.864 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978647+0000", 2023-01-15T14:33:37.864 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032660700000000003 2023-01-15T14:33:37.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.865 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178434+0000", 2023-01-15T14:33:37.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19978705699999999 2023-01-15T14:33:37.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.865 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178527+0000", 2023-01-15T14:33:37.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3523999999999994e-05 2023-01-15T14:33:37.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.866 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.866 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.866 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178602+0000", 2023-01-15T14:33:37.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4622000000000005e-05 2023-01-15T14:33:37.866 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.867 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.867 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.867 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1154 3.a 3:5919d83e:::smithi09025264-13:head [write 2888822~102824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:33:37.867 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:33:37.868 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 230.740658524, 2023-01-15T14:33:37.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48318778800000001, 2023-01-15T14:33:37.869 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.870 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.870 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1154 2023-01-15T14:33:37.870 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.871 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:33:37.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:33:37.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795890+0000", 2023-01-15T14:33:37.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:37.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795967+0000", 2023-01-15T14:33:37.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6304999999999995e-05 2023-01-15T14:33:37.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796000+0000", 2023-01-15T14:33:37.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3593000000000002e-05 2023-01-15T14:33:37.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796019+0000", 2023-01-15T14:33:37.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8955999999999999e-05 2023-01-15T14:33:37.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796189+0000", 2023-01-15T14:33:37.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017032700000000001 2023-01-15T14:33:37.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:37.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796208+0000", 2023-01-15T14:33:37.875 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8389e-05 2023-01-15T14:33:37.875 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.875 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.875 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976335+0000", 2023-01-15T14:33:37.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18012763100000001 2023-01-15T14:33:37.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976487+0000", 2023-01-15T14:33:37.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015173000000000001 2023-01-15T14:33:37.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976763+0000", 2023-01-15T14:33:37.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027597400000000001 2023-01-15T14:33:37.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.877 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.877 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.877 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977085+0000", 2023-01-15T14:33:37.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032233400000000002 2023-01-15T14:33:37.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.877 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.877 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.877 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977103+0000", 2023-01-15T14:33:37.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7727000000000002e-05 2023-01-15T14:33:37.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977238+0000", 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013525399999999999 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978347+0000", 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011084350000000001 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.879 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.078223+0000", 2023-01-15T14:33:37.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099875830999999998 2023-01-15T14:33:37.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.879 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.879 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.879 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.278897+0000", 2023-01-15T14:33:37.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20067421599999999 2023-01-15T14:33:37.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.879 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279021+0000", 2023-01-15T14:33:37.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124354 2023-01-15T14:33:37.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279080+0000", 2023-01-15T14:33:37.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8811999999999999e-05 2023-01-15T14:33:37.881 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.881 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.881 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.882 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1155 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 2991646] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:33:37.882 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:33:37.882 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 230.74048550500001, 2023-01-15T14:33:37.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48334445999999998, 2023-01-15T14:33:37.882 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.882 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.882 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.882 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.882 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.882 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1155 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796064+0000", 2023-01-15T14:33:37.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:37.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796074+0000", 2023-01-15T14:33:37.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:33:37.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796096+0000", 2023-01-15T14:33:37.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2422999999999999e-05 2023-01-15T14:33:37.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796109+0000", 2023-01-15T14:33:37.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:33:37.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796274+0000", 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000164976 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796283+0000", 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9539999999999993e-06 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977269+0000", 2023-01-15T14:33:37.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18098651700000001 2023-01-15T14:33:37.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977438+0000", 2023-01-15T14:33:37.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168913 2023-01-15T14:33:37.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977865+0000", 2023-01-15T14:33:37.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042707500000000001 2023-01-15T14:33:37.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978241+0000", 2023-01-15T14:33:37.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037593000000000001 2023-01-15T14:33:37.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978253+0000", 2023-01-15T14:33:37.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1817000000000001e-05 2023-01-15T14:33:37.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978288+0000", 2023-01-15T14:33:37.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4586999999999999e-05 2023-01-15T14:33:37.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978370+0000", 2023-01-15T14:33:37.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1712999999999995e-05 2023-01-15T14:33:37.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279124+0000", 2023-01-15T14:33:37.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30075418799999998 2023-01-15T14:33:37.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279208+0000", 2023-01-15T14:33:37.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4500999999999996e-05 2023-01-15T14:33:37.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.891 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279294+0000", 2023-01-15T14:33:37.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5680999999999995e-05 2023-01-15T14:33:37.891 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.891 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279410+0000", 2023-01-15T14:33:37.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115878 2023-01-15T14:33:37.891 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.891 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.891 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.892 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1156 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:33:37.892 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:33:37.892 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 230.74037397999999, 2023-01-15T14:33:37.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.58375825000000003, 2023-01-15T14:33:37.892 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.892 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:37.892 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.892 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.892 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.893 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1156 2023-01-15T14:33:37.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.893 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:33:37.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:33:37.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796176+0000", 2023-01-15T14:33:37.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:37.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796182+0000", 2023-01-15T14:33:37.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6799999999999998e-06 2023-01-15T14:33:37.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796200+0000", 2023-01-15T14:33:37.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8136999999999999e-05 2023-01-15T14:33:37.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796217+0000", 2023-01-15T14:33:37.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7036000000000001e-05 2023-01-15T14:33:37.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796337+0000", 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120674 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796345+0000", 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4540000000000001e-06 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978423+0000", 2023-01-15T14:33:37.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18207789599999999 2023-01-15T14:33:37.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:37.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978577+0000", 2023-01-15T14:33:37.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000153805 2023-01-15T14:33:37.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379596+0000", 2023-01-15T14:33:37.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.401019348 2023-01-15T14:33:37.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379754+0000", 2023-01-15T14:33:37.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015840400000000001 2023-01-15T14:33:37.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379935+0000", 2023-01-15T14:33:37.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018081299999999999 2023-01-15T14:33:37.899 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.899 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.899 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.899 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1160 3.a 3:52df5138:::smithi09025264-25:head [write 2632598~375855] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:33:37.899 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:33:37.899 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 229.78156739299999, 2023-01-15T14:33:37.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.398337832, 2023-01-15T14:33:37.899 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.899 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.900 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.900 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.900 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.900 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1160 2023-01-15T14:33:37.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.900 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:33:37.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:33:37.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754981+0000", 2023-01-15T14:33:37.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:37.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755360+0000", 2023-01-15T14:33:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037899899999999998 2023-01-15T14:33:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755392+0000", 2023-01-15T14:33:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1977000000000002e-05 2023-01-15T14:33:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755407+0000", 2023-01-15T14:33:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4446e-05 2023-01-15T14:33:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949294+0000", 2023-01-15T14:33:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.193887483 2023-01-15T14:33:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949478+0000", 2023-01-15T14:33:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018400900000000001 2023-01-15T14:33:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949904+0000", 2023-01-15T14:33:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042552200000000001 2023-01-15T14:33:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950531+0000", 2023-01-15T14:33:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062732399999999996 2023-01-15T14:33:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950552+0000", 2023-01-15T14:33:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1229999999999998e-05 2023-01-15T14:33:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951094+0000", 2023-01-15T14:33:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054169000000000005 2023-01-15T14:33:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951256+0000", 2023-01-15T14:33:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016176299999999999 2023-01-15T14:33:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052665+0000", 2023-01-15T14:33:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101409257 2023-01-15T14:33:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153132+0000", 2023-01-15T14:33:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100466783 2023-01-15T14:33:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153223+0000", 2023-01-15T14:33:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.1415000000000005e-05 2023-01-15T14:33:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153321+0000", 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8181000000000001e-05 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1161 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (57) in=7b,truncate 3008453] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 229.78112380499999, 2023-01-15T14:33:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49840109900000001, 2023-01-15T14:33:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1161 2023-01-15T14:33:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755426+0000", 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755433+0000", 2023-01-15T14:33:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.793e-06 2023-01-15T14:33:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755448+0000", 2023-01-15T14:33:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5047e-05 2023-01-15T14:33:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755455+0000", 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3590000000000002e-06 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951466+0000", 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.196011038 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951611+0000", 2023-01-15T14:33:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000145184 2023-01-15T14:33:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951921+0000", 2023-01-15T14:33:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030989100000000002 2023-01-15T14:33:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952187+0000", 2023-01-15T14:33:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026623100000000002 2023-01-15T14:33:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952207+0000", 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9510000000000001e-05 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952271+0000", 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4511999999999995e-05 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052716+0000", 2023-01-15T14:33:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10044497300000001 2023-01-15T14:33:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253531+0000", 2023-01-15T14:33:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20081490699999999 2023-01-15T14:33:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253635+0000", 2023-01-15T14:33:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000103838 2023-01-15T14:33:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253727+0000", 2023-01-15T14:33:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2298000000000003e-05 2023-01-15T14:33:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253828+0000", 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000100655 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1162 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 229.78108441000001, 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.498775839, 2023-01-15T14:33:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:38.037 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:38.037 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:38.037 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:38.037 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:38.037 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1162 2023-01-15T14:33:38.037 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.037 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:38.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.038 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:33:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:33:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:33:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755470+0000", 2023-01-15T14:33:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0570000000000003e-06 2023-01-15T14:33:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755484+0000", 2023-01-15T14:33:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4229999999999999e-05 2023-01-15T14:33:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755492+0000", 2023-01-15T14:33:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6070000000000002e-06 2023-01-15T14:33:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.152842+0000", 2023-01-15T14:33:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39734988799999998 2023-01-15T14:33:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153014+0000", 2023-01-15T14:33:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017236000000000001 2023-01-15T14:33:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253921+0000", 2023-01-15T14:33:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100907497 2023-01-15T14:33:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254060+0000", 2023-01-15T14:33:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013815400000000001 2023-01-15T14:33:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254242+0000", 2023-01-15T14:33:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000182726 2023-01-15T14:33:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:38.044 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:38.045 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:33:38.045 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:36.623+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:38.046 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:33:38.046 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:33:38.046 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:33:38.046 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:33:38.047 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:38.047 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:38.048 INFO:tasks.rados.rados.0.smithi090.stdout:1294: finishing write tid 5 to smithi09025264-22 2023-01-15T14:33:38.048 INFO:tasks.rados.rados.0.smithi090.stdout:1294: oid 22 updating version 0 to 351 2023-01-15T14:33:38.048 INFO:tasks.rados.rados.0.smithi090.stdout:1294: oid 22 updating version 351 to 354 2023-01-15T14:33:38.048 INFO:tasks.rados.rados.0.smithi090.stdout:1294: oid 22 version 354 is already newer than 353 2023-01-15T14:33:38.048 INFO:tasks.rados.rados.0.smithi090.stdout:1294: oid 22 version 354 is already newer than 352 2023-01-15T14:33:38.048 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 354 (ObjNum 471 snap 136 seq_num 471) dirty exists 2023-01-15T14:33:38.048 INFO:tasks.rados.rados.0.smithi090.stdout:1294: left oid 22 (ObjNum 471 snap 136 seq_num 471) 2023-01-15T14:33:38.048 INFO:tasks.rados.rados.0.smithi090.stdout:1294: done (0 left) 2023-01-15T14:33:38.049 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:38.049 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:38.049 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:33:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:33:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:33:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:33:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:33:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:33:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:33:38.050 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:33:38.051 INFO:tasks.rados.rados.0.smithi090.stdout:1296: done (0 left) 2023-01-15T14:33:38.051 INFO:tasks.rados.rados.0.smithi090.stdout:1297: delete oid 2 current snap is 137 2023-01-15T14:33:38.052 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:33:38.054 INFO:tasks.rados.rados.0.smithi090.stdout:1297: done (0 left) 2023-01-15T14:33:38.054 INFO:tasks.rados.rados.0.smithi090.stdout:1298: copy_from oid 27 from oid 19 current snap is 137 2023-01-15T14:33:38.054 INFO:tasks.rados.rados.0.smithi090.stdout:1299: write (excl) oid 44 current snap is 137 2023-01-15T14:33:38.054 INFO:tasks.rados.rados.0.smithi090.stdout:1299: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2023-01-15T14:33:38.054 INFO:tasks.rados.rados.0.smithi090.stdout:1299: writing smithi09025264-44 from 749711 to 1156374 tid 1 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1299: writing smithi09025264-44 from 1778924 to 2461406 tid 2 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1299: writing smithi09025264-44 from 3189899 to 3905914 tid 3 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1300: read oid 37 snap -1 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1300: expect (ObjNum 446 snap 123 seq_num 446) 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1301: read oid 42 snap 136 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1301: expect (ObjNum 446 snap 123 seq_num 446) 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1302: copy_from oid 12 from oid 31 current snap is 137 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1303: read oid 4 snap 126 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1303: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1304: snap_remove snap 134 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1304: done (6 left) 2023-01-15T14:33:38.055 INFO:tasks.rados.rados.0.smithi090.stdout:1305: rollback oid 6 current snap is 137 2023-01-15T14:33:38.056 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 6 to 128 2023-01-15T14:33:38.056 INFO:tasks.rados.rados.0.smithi090.stdout:1301: expect (ObjNum 446 snap 123 seq_num 446) 2023-01-15T14:33:38.056 INFO:tasks.rados.rados.0.smithi090.stdout:1298: finishing copy_from to smithi09025264-27 2023-01-15T14:33:38.056 INFO:tasks.rados.rados.0.smithi090.stdout:1298: got expected ENOENT (src dne) 2023-01-15T14:33:38.056 INFO:tasks.rados.rados.0.smithi090.stdout:1298: finishing copy_from racing read to smithi09025264-27 2023-01-15T14:33:38.056 INFO:tasks.rados.rados.0.smithi090.stdout:1302: finishing copy_from to smithi09025264-12 2023-01-15T14:33:38.056 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 96 (ObjNum 326 snap 87 seq_num 326) dirty exists 2023-01-15T14:33:38.056 INFO:tasks.rados.rados.0.smithi090.stdout:1302: finishing copy_from racing read to smithi09025264-12 2023-01-15T14:33:38.056 INFO:tasks.rados.rados.0.smithi090.stdout:1300: expect (ObjNum 446 snap 123 seq_num 446) 2023-01-15T14:33:38.057 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:38.044+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:38.129 INFO:tasks.rados.rados.0.smithi090.stdout:1303: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:33:38.203 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:38.204 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:38.204 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:33:38.204 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:38.199+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:38.247 INFO:tasks.rados.rados.0.smithi090.stdout:1298: done (6 left) 2023-01-15T14:33:38.247 INFO:tasks.rados.rados.0.smithi090.stdout:1300: done (5 left) 2023-01-15T14:33:38.248 INFO:tasks.rados.rados.0.smithi090.stdout:1301: done (4 left) 2023-01-15T14:33:38.248 INFO:tasks.rados.rados.0.smithi090.stdout:1302: done (3 left) 2023-01-15T14:33:38.248 INFO:tasks.rados.rados.0.smithi090.stdout:1303: done (2 left) 2023-01-15T14:33:38.248 INFO:tasks.rados.rados.0.smithi090.stdout:1306: write (excl) oid 45 current snap is 137 2023-01-15T14:33:38.248 INFO:tasks.rados.rados.0.smithi090.stdout:1306: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2023-01-15T14:33:38.265 INFO:tasks.rados.rados.0.smithi090.stdout:1306: writing smithi09025264-45 from 766518 to 1248430 tid 1 2023-01-15T14:33:38.273 INFO:tasks.rados.rados.0.smithi090.stdout:1306: writing smithi09025264-45 from 1837406 to 2379899 tid 2 2023-01-15T14:33:38.276 INFO:tasks.rados.rados.0.smithi090.stdout:1306: writing smithi09025264-45 from 2933675 to 3460962 tid 3 2023-01-15T14:33:38.279 INFO:tasks.rados.rados.0.smithi090.stdout:1306: writing smithi09025264-45 from 3876584 to 3966518 tid 4 2023-01-15T14:33:38.367 INFO:tasks.rados.rados.0.smithi090.stdout:1307: snap_remove snap 133 2023-01-15T14:33:38.368 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:38.368 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:38.369 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:33:38.369 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:38.363+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:38.387 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:33:38.389 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:33:38.405 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2023-01-15T14:33:37.881964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":266636759,"num_objects":119,"num_object_clones":69,"num_object_copies":357,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":119,"num_whiteouts":9,"num_read":914,"num_read_kb":830952,"num_write":811,"num_write_kb":266692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":286,"num_bytes_recovered":170648723,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16097,"ondisk_log_size":16097,"up":309,"acting":306,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":303,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":5885212,"kb_used_data":556337368,"kb_used_omap":57541,"kb_used_meta":0,"kb_avail":556311068,"statfs":{"total":575688990720,"available":569662533632,"internally_reserved":26931200,"allocated":569689464832,"data_stored":569689464832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58922147,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":61,"commit_latency_ns":0,"apply_latency_ns":61000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19168942,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":-4,"num_read_kb":5607,"num_write":-8,"num_write_kb":-8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-15,"num_bytes_recovered":-6365820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.943145"},"pg_stats":[{"pgid":"3.5f","version":"0'0","reported_seq":1,"reported_epoch":327,"state":"unknown","last_fresh":"2023-01-15T14:33:35.962727+0000","last_change":"2023-01-15T14:33:36.247349+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:33:35.962727+0000","last_undegraded":"2023-01-15T14:33:35.962727+0000","last_fullsized":"2023-01-15T14:33:35.962727+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":0,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:56.504","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5e","version":"215'39","reported_seq":920,"reported_epoch":328,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.884689+0000","last_change":"2023-01-15T14:33:35.460333+0000","last_active":"2023-01-15T14:33:36.884689+0000","last_peered":"2023-01-15T14:33:36.884689+0000","last_clean":"2023-01-15T14:33:36.884689+0000","last_became_active":"2023-01-15T14:33:35.364143+0000","last_became_peered":"2023-01-15T14:33:35.364143+0000","last_unstale":"2023-01-15T14:33:36.884689+0000","last_undegraded":"2023-01-15T14:33:36.884689+0000","last_fullsized":"2023-01-15T14:33:36.884689+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.5d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:33:33.929478+0000","last_change":"2023-01-15T14:33:33.929478+0000","last_active":"2023-01-15T14:33:33.929478+0000","last_peered":"2023-01-15T14:33:33.929478+0000","last_clean":"2023-01-15T14:33:33.929478+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:33:33.929478+0000","last_undegraded":"2023-01-15T14:33:33.929478+0000","last_fullsized":"2023-01-15T14:33:33.929478+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:33.929478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:33:33.929478+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.929478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5c","version":"141'46","reported_seq":11,"reported_epoch":328,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.992704+0000","last_change":"2023-01-15T14:33:36.991305+0000","last_active":"2023-01-15T14:33:36.992704+0000","last_peered":"2023-01-15T14:33:36.992704+0000","last_clean":"2023-01-15T14:33:36.992704+0000","last_became_active":"2023-01-15T14:33:36.990198+0000","last_became_peered":"2023-01-15T14:33:36.990198+0000","last_unstale":"2023-01-15T14:33:36.992704+0000","last_undegraded":"2023-01-15T14:33:36.992704+0000","last_fullsized":"2023-01-15T14:33:36.992704+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.5b","version":"283'80","reported_seq":843,"reported_epoch":328,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.990058+0000","last_change":"2023-01-15T14:33:36.989671+0000","last_active":"2023-01-15T14:33:36.990058+0000","last_peered":"2023-01-15T14:33:36.990058+0000","last_clean":"2023-01-15T14:33:36.990058+0000","last_became_active":"2023-01-15T14:33:36.988473+0000","last_became_peered":"2023-01-15T14:33:36.988473+0000","last_unstale":"2023-01-15T14:33:36.990058+0000","last_undegraded":"2023-01-15T14:33:36.990058+0000","last_fullsized":"2023-01-15T14:33:36.990058+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.5a","version":"323'344","reported_seq":1374,"reported_epoch":328,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:36.890352+0000","last_change":"2023-01-15T14:33:36.890352+0000","last_active":"2023-01-15T14:33:35.962597+0000","last_peered":"2023-01-15T14:33:32.295271+0000","last_clean":"2023-01-15T14:33:32.295271+0000","last_became_active":"2023-01-15T14:31:52.124026+0000","last_became_peered":"2023-01-15T14:31:52.124026+0000","last_unstale":"2023-01-15T14:33:36.890352+0000","last_undegraded":"2023-01-15T14:33:36.890352+0000","last_fullsized":"2023-01-15T14:33:36.890352+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3052654,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":5,"num_read_kb":4287,"num_write":6,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.59","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:33:33.929478+0000","last_change":"2023-01-15T14:33:33.929478+0000","last_active":"2023-01-15T14:33:33.929478+0000","last_peered":"2023-01-15T14:33:33.929478+0000","last_clean":"2023-01-15T14:33:33.929478+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:33:33.929478+0000","last_undegraded":"2023-01-15T14:33:33.929478+0000","last_fullsized":"2023-01-15T14:33:33.929478+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:33.929478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:33:33.929478+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.929478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.58","version":"323'320","reported_seq":1409,"reported_epoch":328,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:36.887949+0000","last_change":"2023-01-15T14:33:36.887949+0000","last_active":"2023-01-15T14:33:36.887529+0000","last_peered":"2023-01-15T14:33:31.982151+0000","last_clean":"2023-01-15T14:33:31.982151+0000","last_became_active":"2023-01-15T14:31:52.330386+0000","last_became_peered":"2023-01-15T14:31:52.330386+0000","last_unstale":"2023-01-15T14:33:36.887949+0000","last_undegraded":"2023-01-15T14:33:36.887949+0000","last_fullsized":"2023-01-15T14:33:36.887949+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4332079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.57","version":"278'93","reported_seq":2,"reported_epoch":327,"state":"peering","last_fresh":"2023-01-15T14:32:45.801781+0000","last_change":"2023-01-15T14:33:36.247305+0000","last_active":"2023-01-15T14:33:36.247305+0000","last_peered":"2023-01-15T14:32:45.801781+0000","last_clean":"2023-01-15T14:32:45.801781+0000","last_became_active":"2023-01-15T14:31:34.932309+0000","last_became_peered":"2023-01-15T14:31:34.932309+0000","last_unstale":"2023-01-15T14:32:45.801781+0000","last_undegraded":"2023-01-15T14:32:45.801781+0000","last_fullsized":"2023-01-15T14:32:45.801781+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.56","version":"320'127","reported_seq":549,"reported_epoch":328,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:36.885648+0000","last_change":"2023-01-15T14:33:36.885648+0000","last_active":"2023-01-15T14:33:36.068816+0000","last_peered":"2023-01-15T14:33:29.804995+0000","last_clean":"2023-01-15T14:33:29.804995+0000","last_became_active":"2023-01-15T14:31:35.032807+0000","last_became_peered":"2023-01-15T14:31:35.032807+0000","last_unstale":"2023-01-15T14:33:36.885648+0000","last_undegraded":"2023-01-15T14:33:36.885648+0000","last_fullsized":"2023-01-15T14:33:36.885648+0000","mapping_epoch":328,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.55","version":"110'55","reported_seq":942,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.966335+0000","last_change":"2023-01-15T14:33:32.502634+0000","last_active":"2023-01-15T14:33:35.966335+0000","last_peered":"2023-01-15T14:33:35.966335+0000","last_clean":"2023-01-15T14:33:35.966335+0000","last_became_active":"2023-01-15T14:33:22.073078+0000","last_became_peered":"2023-01-15T14:33:22.073078+0000","last_unstale":"2023-01-15T14:33:35.966335+0000","last_undegraded":"2023-01-15T14:33:35.966335+0000","last_fullsized":"2023-01-15T14:33:35.966335+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.52012370699999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.54","version":"245'148","reported_seq":1128,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:31.987078+0000","last_change":"2023-01-15T14:33:31.987078+0000","last_active":"2023-01-15T14:33:31.987078+0000","last_peered":"2023-01-15T14:33:31.987078+0000","last_clean":"2023-01-15T14:33:31.987078+0000","last_became_active":"2023-01-15T14:33:25.198058+0000","last_became_peered":"2023-01-15T14:33:25.198058+0000","last_unstale":"2023-01-15T14:33:31.987078+0000","last_undegraded":"2023-01-15T14:33:31.987078+0000","last_fullsized":"2023-01-15T14:33:31.987078+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:32:19.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:19.303243+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.102672765,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.53","version":"324'368","reported_seq":1316,"reported_epoch":328,"state":"active+clean","last_fresh":"2023-01-15T14:33:37.400828+0000","last_change":"2023-01-15T14:33:37.396766+0000","last_active":"2023-01-15T14:33:37.400828+0000","last_peered":"2023-01-15T14:33:37.400828+0000","last_clean":"2023-01-15T14:33:37.400828+0000","last_became_active":"2023-01-15T14:33:28.184994+0000","last_became_peered":"2023-01-15T14:33:28.184994+0000","last_unstale":"2023-01-15T14:33:37.400828+0000","last_undegraded":"2023-01-15T14:33:37.400828+0000","last_fullsized":"2023-01-15T14:33:37.400828+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:56.683","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.101611487,"stat_sum":{"num_bytes":13429057,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":9,"num_read_kb":17074,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":652,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.963887+0000","last_change":"2023-01-15T14:33:32.830876+0000","last_active":"2023-01-15T14:33:35.963887+0000","last_peered":"2023-01-15T14:33:35.963887+0000","last_clean":"2023-01-15T14:33:35.963887+0000","last_became_active":"2023-01-15T14:33:25.099467+0000","last_became_peered":"2023-01-15T14:33:25.099467+0000","last_unstale":"2023-01-15T14:33:35.963887+0000","last_undegraded":"2023-01-15T14:33:35.963887+0000","last_fullsized":"2023-01-15T14:33:35.963887+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.72602100599999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.51","version":"323'298","reported_seq":1443,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.962718+0000","last_change":"2023-01-15T14:33:33.733676+0000","last_active":"2023-01-15T14:33:35.962718+0000","last_peered":"2023-01-15T14:33:35.962718+0000","last_clean":"2023-01-15T14:33:35.962718+0000","last_became_active":"2023-01-15T14:33:26.107940+0000","last_became_peered":"2023-01-15T14:33:26.107940+0000","last_unstale":"2023-01-15T14:33:35.962718+0000","last_undegraded":"2023-01-15T14:33:35.962718+0000","last_fullsized":"2023-01-15T14:33:35.962718+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:36.458","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.00260677,"stat_sum":{"num_bytes":5110093,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.50","version":"325'416","reported_seq":1449,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.964066+0000","last_change":"2023-01-15T14:33:34.856228+0000","last_active":"2023-01-15T14:33:35.964066+0000","last_peered":"2023-01-15T14:33:35.964066+0000","last_clean":"2023-01-15T14:33:35.964066+0000","last_became_active":"2023-01-15T14:33:30.211595+0000","last_became_peered":"2023-01-15T14:33:30.211595+0000","last_unstale":"2023-01-15T14:33:35.964066+0000","last_undegraded":"2023-01-15T14:33:35.964066+0000","last_fullsized":"2023-01-15T14:33:35.964066+0000","mapping_epoch":320,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.90889536100000001,"stat_sum":{"num_bytes":6151880,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":45,"num_read_kb":42160,"num_write":48,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":22053159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.4f","version":"155'145","reported_seq":768,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.965657+0000","last_change":"2023-01-15T14:33:32.397464+0000","last_active":"2023-01-15T14:33:35.965657+0000","last_peered":"2023-01-15T14:33:35.965657+0000","last_clean":"2023-01-15T14:33:35.965657+0000","last_became_active":"2023-01-15T14:33:25.129936+0000","last_became_peered":"2023-01-15T14:33:25.129936+0000","last_unstale":"2023-01-15T14:33:35.965657+0000","last_undegraded":"2023-01-15T14:33:35.965657+0000","last_fullsized":"2023-01-15T14:33:35.965657+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51181851599999995,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.4e","version":"139'38","reported_seq":38,"reported_epoch":326,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.496679+0000","last_change":"2023-01-15T14:33:31.898003+0000","last_active":"2023-01-15T14:33:35.496679+0000","last_peered":"2023-01-15T14:33:35.496679+0000","last_clean":"2023-01-15T14:33:35.496679+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:33:35.496679+0000","last_undegraded":"2023-01-15T14:33:35.496679+0000","last_fullsized":"2023-01-15T14:33:35.496679+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.002524545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.4d","version":"323'240","reported_seq":1250,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.692142+0000","last_change":"2023-01-15T14:33:32.692142+0000","last_active":"2023-01-15T14:33:32.692142+0000","last_peered":"2023-01-15T14:33:32.692142+0000","last_clean":"2023-01-15T14:33:32.692142+0000","last_became_active":"2023-01-15T14:33:26.102614+0000","last_became_peered":"2023-01-15T14:33:26.102614+0000","last_unstale":"2023-01-15T14:33:32.692142+0000","last_undegraded":"2023-01-15T14:33:32.692142+0000","last_fullsized":"2023-01-15T14:33:32.692142+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":240,"log_dups_size":0,"ondisk_log_size":240,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.80683807299999999,"stat_sum":{"num_bytes":6258391,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":5940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11291942,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.4c","version":"309'71","reported_seq":39,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.965180+0000","last_change":"2023-01-15T14:33:32.829894+0000","last_active":"2023-01-15T14:33:35.965180+0000","last_peered":"2023-01-15T14:33:35.965180+0000","last_clean":"2023-01-15T14:33:35.965180+0000","last_became_active":"2023-01-15T14:33:22.183235+0000","last_became_peered":"2023-01-15T14:33:22.183235+0000","last_unstale":"2023-01-15T14:33:35.965180+0000","last_undegraded":"2023-01-15T14:33:35.965180+0000","last_fullsized":"2023-01-15T14:33:35.965180+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.72504199000000003,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.4b","version":"155'53","reported_seq":685,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.967204+0000","last_change":"2023-01-15T14:33:32.015769+0000","last_active":"2023-01-15T14:33:35.967204+0000","last_peered":"2023-01-15T14:33:35.967204+0000","last_clean":"2023-01-15T14:33:35.967204+0000","last_became_active":"2023-01-15T14:32:25.161408+0000","last_became_peered":"2023-01-15T14:32:25.161408+0000","last_unstale":"2023-01-15T14:33:35.967204+0000","last_undegraded":"2023-01-15T14:33:35.967204+0000","last_fullsized":"2023-01-15T14:33:35.967204+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:40.131","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10165226099999999,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.4a","version":"323'355","reported_seq":1401,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.964246+0000","last_change":"2023-01-15T14:33:33.637936+0000","last_active":"2023-01-15T14:33:35.964246+0000","last_peered":"2023-01-15T14:33:35.964246+0000","last_clean":"2023-01-15T14:33:35.964246+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:33:35.964246+0000","last_undegraded":"2023-01-15T14:33:35.964246+0000","last_fullsized":"2023-01-15T14:33:35.964246+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:33.328","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.69945441200000003,"stat_sum":{"num_bytes":11357972,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":52,"num_read_kb":48981,"num_write":69,"num_write_kb":22538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.49","version":"110'149","reported_seq":1072,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:31.984945+0000","last_change":"2023-01-15T14:33:31.984945+0000","last_active":"2023-01-15T14:33:31.984945+0000","last_peered":"2023-01-15T14:33:31.984945+0000","last_clean":"2023-01-15T14:33:31.984945+0000","last_became_active":"2023-01-15T14:32:28.259491+0000","last_became_peered":"2023-01-15T14:32:28.259491+0000","last_unstale":"2023-01-15T14:33:31.984945+0000","last_undegraded":"2023-01-15T14:33:31.984945+0000","last_fullsized":"2023-01-15T14:33:31.984945+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:22.332","scrub_duration":0.47056024400000002,"objects_trimmed":0,"snaptrim_duration":0.10231617899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.48","version":"179'204","reported_seq":1270,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.957847+0000","last_change":"2023-01-15T14:33:31.880268+0000","last_active":"2023-01-15T14:33:35.957847+0000","last_peered":"2023-01-15T14:33:35.957847+0000","last_clean":"2023-01-15T14:33:35.957847+0000","last_became_active":"2023-01-15T14:32:28.187902+0000","last_became_peered":"2023-01-15T14:32:28.187902+0000","last_unstale":"2023-01-15T14:33:35.957847+0000","last_undegraded":"2023-01-15T14:33:35.957847+0000","last_fullsized":"2023-01-15T14:33:35.957847+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.000452325,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.47","version":"245'112","reported_seq":1056,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.699284+0000","last_change":"2023-01-15T14:33:32.699284+0000","last_active":"2023-01-15T14:33:32.699284+0000","last_peered":"2023-01-15T14:33:32.699284+0000","last_clean":"2023-01-15T14:33:32.699284+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:33:32.699284+0000","last_undegraded":"2023-01-15T14:33:32.699284+0000","last_fullsized":"2023-01-15T14:33:32.699284+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:03.200","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.71293729400000005,"stat_sum":{"num_bytes":496762,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":5,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.46","version":"102'52","reported_seq":970,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.968860+0000","last_change":"2023-01-15T14:33:32.865907+0000","last_active":"2023-01-15T14:33:35.968860+0000","last_peered":"2023-01-15T14:33:35.968860+0000","last_clean":"2023-01-15T14:33:35.968860+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:33:35.968860+0000","last_undegraded":"2023-01-15T14:33:35.968860+0000","last_fullsized":"2023-01-15T14:33:35.968860+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:55.245","scrub_duration":0.30448409700000001,"objects_trimmed":0,"snaptrim_duration":0.306407019,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.45","version":"253'125","reported_seq":1071,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.966350+0000","last_change":"2023-01-15T14:33:32.831844+0000","last_active":"2023-01-15T14:33:35.966350+0000","last_peered":"2023-01-15T14:33:35.966350+0000","last_clean":"2023-01-15T14:33:35.966350+0000","last_became_active":"2023-01-15T14:32:28.162971+0000","last_became_peered":"2023-01-15T14:32:28.162971+0000","last_unstale":"2023-01-15T14:33:35.966350+0000","last_undegraded":"2023-01-15T14:33:35.966350+0000","last_fullsized":"2023-01-15T14:33:35.966350+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:50.736","scrub_duration":0.206392353,"objects_trimmed":0,"snaptrim_duration":0.72653161700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.44","version":"245'268","reported_seq":1191,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:31.986038+0000","last_change":"2023-01-15T14:33:31.986038+0000","last_active":"2023-01-15T14:33:31.986038+0000","last_peered":"2023-01-15T14:33:31.986038+0000","last_clean":"2023-01-15T14:33:31.986038+0000","last_became_active":"2023-01-15T14:33:28.080044+0000","last_became_peered":"2023-01-15T14:33:28.080044+0000","last_unstale":"2023-01-15T14:33:31.986038+0000","last_undegraded":"2023-01-15T14:33:31.986038+0000","last_fullsized":"2023-01-15T14:33:31.986038+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:42.130","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.102234046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.43","version":"297'314","reported_seq":1411,"reported_epoch":326,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.496221+0000","last_change":"2023-01-15T14:33:31.895106+0000","last_active":"2023-01-15T14:33:35.496221+0000","last_peered":"2023-01-15T14:33:35.496221+0000","last_clean":"2023-01-15T14:33:35.496221+0000","last_became_active":"2023-01-15T14:32:28.977084+0000","last_became_peered":"2023-01-15T14:32:28.977084+0000","last_unstale":"2023-01-15T14:33:35.496221+0000","last_undegraded":"2023-01-15T14:33:35.496221+0000","last_fullsized":"2023-01-15T14:33:35.496221+0000","mapping_epoch":261,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:02.589","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.0013156960000000001,"stat_sum":{"num_bytes":3395080,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":16585,"num_write":5,"num_write_kb":1351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":956,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.059669+0000","last_change":"2023-01-15T14:33:32.499305+0000","last_active":"2023-01-15T14:33:36.059669+0000","last_peered":"2023-01-15T14:33:36.059669+0000","last_clean":"2023-01-15T14:33:36.059669+0000","last_became_active":"2023-01-15T14:32:25.163911+0000","last_became_peered":"2023-01-15T14:32:25.163911+0000","last_unstale":"2023-01-15T14:33:36.059669+0000","last_undegraded":"2023-01-15T14:33:36.059669+0000","last_fullsized":"2023-01-15T14:33:36.059669+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.61249039699999996,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.41","version":"155'121","reported_seq":1032,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:31.985129+0000","last_change":"2023-01-15T14:33:31.985129+0000","last_active":"2023-01-15T14:33:31.985129+0000","last_peered":"2023-01-15T14:33:31.985129+0000","last_clean":"2023-01-15T14:33:31.985129+0000","last_became_active":"2023-01-15T14:31:56.927677+0000","last_became_peered":"2023-01-15T14:31:56.927677+0000","last_unstale":"2023-01-15T14:33:31.985129+0000","last_undegraded":"2023-01-15T14:33:31.985129+0000","last_fullsized":"2023-01-15T14:33:31.985129+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:56.100","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.101823631,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.40","version":"328'297","reported_seq":1346,"reported_epoch":328,"state":"active+clean","last_fresh":"2023-01-15T14:33:37.245627+0000","last_change":"2023-01-15T14:33:31.888045+0000","last_active":"2023-01-15T14:33:37.245627+0000","last_peered":"2023-01-15T14:33:37.245627+0000","last_clean":"2023-01-15T14:33:37.245627+0000","last_became_active":"2023-01-15T14:33:14.198194+0000","last_became_peered":"2023-01-15T14:33:14.198194+0000","last_unstale":"2023-01-15T14:33:37.245627+0000","last_undegraded":"2023-01-15T14:33:37.245627+0000","last_fullsized":"2023-01-15T14:33:37.245627+0000","mapping_epoch":304,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.0070985240000000002,"stat_sum":{"num_bytes":9375631,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":21945,"num_write":26,"num_write_kb":11001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":16413587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.3f","version":"321'233","reported_seq":1265,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.066129+0000","last_change":"2023-01-15T14:33:32.930560+0000","last_active":"2023-01-15T14:33:36.066129+0000","last_peered":"2023-01-15T14:33:36.066129+0000","last_clean":"2023-01-15T14:33:36.066129+0000","last_became_active":"2023-01-15T14:32:02.943089+0000","last_became_peered":"2023-01-15T14:32:02.943089+0000","last_unstale":"2023-01-15T14:33:36.066129+0000","last_undegraded":"2023-01-15T14:33:36.066129+0000","last_fullsized":"2023-01-15T14:33:36.066129+0000","mapping_epoch":235,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:18.074","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.82486355600000005,"stat_sum":{"num_bytes":4764997,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.3e","version":"294'63","reported_seq":775,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.067249+0000","last_change":"2023-01-15T14:33:32.117191+0000","last_active":"2023-01-15T14:33:36.067249+0000","last_peered":"2023-01-15T14:33:36.067249+0000","last_clean":"2023-01-15T14:33:36.067249+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:33:36.067249+0000","last_undegraded":"2023-01-15T14:33:36.067249+0000","last_fullsized":"2023-01-15T14:33:36.067249+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:42.278","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.201717385,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.3d","version":"283'223","reported_seq":1258,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.969046+0000","last_change":"2023-01-15T14:33:31.915015+0000","last_active":"2023-01-15T14:33:35.969046+0000","last_peered":"2023-01-15T14:33:35.969046+0000","last_clean":"2023-01-15T14:33:35.969046+0000","last_became_active":"2023-01-15T14:31:59.249241+0000","last_became_peered":"2023-01-15T14:31:59.249241+0000","last_unstale":"2023-01-15T14:33:35.969046+0000","last_undegraded":"2023-01-15T14:33:35.969046+0000","last_fullsized":"2023-01-15T14:33:35.969046+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.0013046830000000001,"stat_sum":{"num_bytes":5496021,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":13796,"num_write":10,"num_write_kb":7056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":290,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.965544+0000","last_change":"2023-01-15T14:33:31.889067+0000","last_active":"2023-01-15T14:33:35.965544+0000","last_peered":"2023-01-15T14:33:35.965544+0000","last_clean":"2023-01-15T14:33:35.965544+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:33:35.965544+0000","last_undegraded":"2023-01-15T14:33:35.965544+0000","last_fullsized":"2023-01-15T14:33:35.965544+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.470","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.0078177119999999992,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.3b","version":"221'60","reported_seq":931,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.087888+0000","last_change":"2023-01-15T14:33:32.087888+0000","last_active":"2023-01-15T14:33:32.087888+0000","last_peered":"2023-01-15T14:33:32.087888+0000","last_clean":"2023-01-15T14:33:32.087888+0000","last_became_active":"2023-01-15T14:31:54.237985+0000","last_became_peered":"2023-01-15T14:31:54.237985+0000","last_unstale":"2023-01-15T14:33:32.087888+0000","last_undegraded":"2023-01-15T14:33:32.087888+0000","last_fullsized":"2023-01-15T14:33:32.087888+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:52.139","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.204058924,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1279,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.066598+0000","last_change":"2023-01-15T14:33:32.219115+0000","last_active":"2023-01-15T14:33:36.066598+0000","last_peered":"2023-01-15T14:33:36.066598+0000","last_clean":"2023-01-15T14:33:36.066598+0000","last_became_active":"2023-01-15T14:31:57.025706+0000","last_became_peered":"2023-01-15T14:31:57.025706+0000","last_unstale":"2023-01-15T14:33:36.066598+0000","last_undegraded":"2023-01-15T14:33:36.066598+0000","last_fullsized":"2023-01-15T14:33:36.066598+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.230245229,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.39","version":"327'339","reported_seq":1328,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.670834+0000","last_change":"2023-01-15T14:33:32.828622+0000","last_active":"2023-01-15T14:33:36.670834+0000","last_peered":"2023-01-15T14:33:36.670834+0000","last_clean":"2023-01-15T14:33:36.670834+0000","last_became_active":"2023-01-15T14:32:06.420117+0000","last_became_peered":"2023-01-15T14:32:06.420117+0000","last_unstale":"2023-01-15T14:33:36.670834+0000","last_undegraded":"2023-01-15T14:33:36.670834+0000","last_fullsized":"2023-01-15T14:33:36.670834+0000","mapping_epoch":238,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.413","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.72426807699999995,"stat_sum":{"num_bytes":21098960,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":19,"num_read_kb":32508,"num_write":6,"num_write_kb":4436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.38","version":"325'303","reported_seq":1458,"reported_epoch":328,"state":"active+clean","last_fresh":"2023-01-15T14:33:37.122293+0000","last_change":"2023-01-15T14:33:31.888866+0000","last_active":"2023-01-15T14:33:37.122293+0000","last_peered":"2023-01-15T14:33:37.122293+0000","last_clean":"2023-01-15T14:33:37.122293+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:33:37.122293+0000","last_undegraded":"2023-01-15T14:33:37.122293+0000","last_fullsized":"2023-01-15T14:33:37.122293+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":303,"log_dups_size":0,"ondisk_log_size":303,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:55.610","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.0077805189999999996,"stat_sum":{"num_bytes":7580089,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":29187,"num_write":35,"num_write_kb":12593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.37","version":"200'78","reported_seq":342,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.967420+0000","last_change":"2023-01-15T14:33:33.034236+0000","last_active":"2023-01-15T14:33:35.967420+0000","last_peered":"2023-01-15T14:33:35.967420+0000","last_clean":"2023-01-15T14:33:35.967420+0000","last_became_active":"2023-01-15T14:31:37.024453+0000","last_became_peered":"2023-01-15T14:31:37.024453+0000","last_unstale":"2023-01-15T14:33:35.967420+0000","last_undegraded":"2023-01-15T14:33:35.967420+0000","last_fullsized":"2023-01-15T14:33:35.967420+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.093515112999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.36","version":"194'81","reported_seq":503,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.968330+0000","last_change":"2023-01-15T14:33:32.321785+0000","last_active":"2023-01-15T14:33:35.968330+0000","last_peered":"2023-01-15T14:33:35.968330+0000","last_clean":"2023-01-15T14:33:35.968330+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:33:35.968330+0000","last_undegraded":"2023-01-15T14:33:35.968330+0000","last_fullsized":"2023-01-15T14:33:35.968330+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.30618451800000002,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.35","version":"110'55","reported_seq":931,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.966644+0000","last_change":"2023-01-15T14:33:32.396909+0000","last_active":"2023-01-15T14:33:35.966644+0000","last_peered":"2023-01-15T14:33:35.966644+0000","last_clean":"2023-01-15T14:33:35.966644+0000","last_became_active":"2023-01-15T14:31:38.033824+0000","last_became_peered":"2023-01-15T14:31:38.033824+0000","last_unstale":"2023-01-15T14:33:35.966644+0000","last_undegraded":"2023-01-15T14:33:35.966644+0000","last_fullsized":"2023-01-15T14:33:35.966644+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:12.355","scrub_duration":0.282936307,"objects_trimmed":0,"snaptrim_duration":0.51155025799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":1095,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.088540+0000","last_change":"2023-01-15T14:33:32.088540+0000","last_active":"2023-01-15T14:33:32.088540+0000","last_peered":"2023-01-15T14:33:32.088540+0000","last_clean":"2023-01-15T14:33:32.088540+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:33:32.088540+0000","last_undegraded":"2023-01-15T14:33:32.088540+0000","last_fullsized":"2023-01-15T14:33:32.088540+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:59.872","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.20411995299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.33","version":"204'275","reported_seq":1173,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.965774+0000","last_change":"2023-01-15T14:33:32.292864+0000","last_active":"2023-01-15T14:33:35.965774+0000","last_peered":"2023-01-15T14:33:35.965774+0000","last_clean":"2023-01-15T14:33:35.965774+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:33:35.965774+0000","last_undegraded":"2023-01-15T14:33:35.965774+0000","last_fullsized":"2023-01-15T14:33:35.965774+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:54.578","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.41023226299999999,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.32","version":"323'161","reported_seq":779,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.066783+0000","last_change":"2023-01-15T14:33:32.924175+0000","last_active":"2023-01-15T14:33:36.066783+0000","last_peered":"2023-01-15T14:33:36.066783+0000","last_clean":"2023-01-15T14:33:36.066783+0000","last_became_active":"2023-01-15T14:31:36.921952+0000","last_became_peered":"2023-01-15T14:31:36.921952+0000","last_unstale":"2023-01-15T14:33:36.066783+0000","last_undegraded":"2023-01-15T14:33:36.066783+0000","last_fullsized":"2023-01-15T14:33:36.066783+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.93590203100000002,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.13","version":"324'368","reported_seq":1312,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.967689+0000","last_change":"2023-01-15T14:33:33.949777+0000","last_active":"2023-01-15T14:33:35.967689+0000","last_peered":"2023-01-15T14:33:35.967689+0000","last_clean":"2023-01-15T14:33:35.967689+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:33:35.967689+0000","last_undegraded":"2023-01-15T14:33:35.967689+0000","last_fullsized":"2023-01-15T14:33:35.967689+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:29.359","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":1.0111862359999999,"stat_sum":{"num_bytes":8364486,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":24051,"num_write":17,"num_write_kb":9266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.12","version":"325'178","reported_seq":798,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.060078+0000","last_change":"2023-01-15T14:33:32.499903+0000","last_active":"2023-01-15T14:33:36.060078+0000","last_peered":"2023-01-15T14:33:36.060078+0000","last_clean":"2023-01-15T14:33:36.060078+0000","last_became_active":"2023-01-15T14:33:20.335242+0000","last_became_peered":"2023-01-15T14:33:20.335242+0000","last_unstale":"2023-01-15T14:33:36.060078+0000","last_undegraded":"2023-01-15T14:33:36.060078+0000","last_fullsized":"2023-01-15T14:33:36.060078+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.61275776100000001,"stat_sum":{"num_bytes":7581497,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":15,"num_write_kb":5299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.11","version":"323'302","reported_seq":1431,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.164031+0000","last_change":"2023-01-15T14:33:32.828298+0000","last_active":"2023-01-15T14:33:36.164031+0000","last_peered":"2023-01-15T14:33:36.164031+0000","last_clean":"2023-01-15T14:33:36.164031+0000","last_became_active":"2023-01-15T14:33:20.234205+0000","last_became_peered":"2023-01-15T14:33:20.234205+0000","last_unstale":"2023-01-15T14:33:36.164031+0000","last_undegraded":"2023-01-15T14:33:36.164031+0000","last_fullsized":"2023-01-15T14:33:36.164031+0000","mapping_epoch":303,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.72359262400000002,"stat_sum":{"num_bytes":10718732,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":34660,"num_write":26,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6655737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.10","version":"308'406","reported_seq":1433,"reported_epoch":328,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.984256+0000","last_change":"2023-01-15T14:33:32.091014+0000","last_active":"2023-01-15T14:33:36.984256+0000","last_peered":"2023-01-15T14:33:36.984256+0000","last_clean":"2023-01-15T14:33:36.984256+0000","last_became_active":"2023-01-15T14:33:20.335088+0000","last_became_peered":"2023-01-15T14:33:20.335088+0000","last_unstale":"2023-01-15T14:33:36.984256+0000","last_undegraded":"2023-01-15T14:33:36.984256+0000","last_fullsized":"2023-01-15T14:33:36.984256+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:53.576","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.20928249700000001,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":28368,"num_write":46,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6670228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1168,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.800199+0000","last_change":"2023-01-15T14:33:32.800199+0000","last_active":"2023-01-15T14:33:32.800199+0000","last_peered":"2023-01-15T14:33:32.800199+0000","last_clean":"2023-01-15T14:33:32.800199+0000","last_became_active":"2023-01-15T14:33:20.239855+0000","last_became_peered":"2023-01-15T14:33:20.239855+0000","last_unstale":"2023-01-15T14:33:32.800199+0000","last_undegraded":"2023-01-15T14:33:32.800199+0000","last_fullsized":"2023-01-15T14:33:32.800199+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.81240387700000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":959,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.696978+0000","last_change":"2023-01-15T14:33:32.696978+0000","last_active":"2023-01-15T14:33:32.696978+0000","last_peered":"2023-01-15T14:33:32.696978+0000","last_clean":"2023-01-15T14:33:32.696978+0000","last_became_active":"2023-01-15T14:33:20.235974+0000","last_became_peered":"2023-01-15T14:33:20.235974+0000","last_unstale":"2023-01-15T14:33:32.696978+0000","last_undegraded":"2023-01-15T14:33:32.696978+0000","last_fullsized":"2023-01-15T14:33:32.696978+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.71073959099999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.d","version":"296'232","reported_seq":1240,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.966604+0000","last_change":"2023-01-15T14:33:32.930986+0000","last_active":"2023-01-15T14:33:35.966604+0000","last_peered":"2023-01-15T14:33:35.966604+0000","last_clean":"2023-01-15T14:33:35.966604+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:33:35.966604+0000","last_undegraded":"2023-01-15T14:33:35.966604+0000","last_fullsized":"2023-01-15T14:33:35.966604+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.82520346600000005,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.c","version":"324'81","reported_seq":994,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.965889+0000","last_change":"2023-01-15T14:33:32.091955+0000","last_active":"2023-01-15T14:33:35.965889+0000","last_peered":"2023-01-15T14:33:35.965889+0000","last_clean":"2023-01-15T14:33:35.965889+0000","last_became_active":"2023-01-15T14:33:20.338147+0000","last_became_peered":"2023-01-15T14:33:20.338147+0000","last_unstale":"2023-01-15T14:33:35.965889+0000","last_undegraded":"2023-01-15T14:33:35.965889+0000","last_fullsized":"2023-01-15T14:33:35.965889+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.21009783000000001,"stat_sum":{"num_bytes":2221858,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":10493,"num_write":4,"num_write_kb":686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":936,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.166053+0000","last_change":"2023-01-15T14:33:32.295696+0000","last_active":"2023-01-15T14:33:36.166053+0000","last_peered":"2023-01-15T14:33:36.166053+0000","last_clean":"2023-01-15T14:33:36.166053+0000","last_became_active":"2023-01-15T14:32:22.246088+0000","last_became_peered":"2023-01-15T14:32:22.246088+0000","last_unstale":"2023-01-15T14:33:36.166053+0000","last_undegraded":"2023-01-15T14:33:36.166053+0000","last_fullsized":"2023-01-15T14:33:36.166053+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:06.346","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.41189098400000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.a","version":"252'279","reported_seq":1254,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.694928+0000","last_change":"2023-01-15T14:33:32.694928+0000","last_active":"2023-01-15T14:33:32.694928+0000","last_peered":"2023-01-15T14:33:32.694928+0000","last_clean":"2023-01-15T14:33:32.694928+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:33:32.694928+0000","last_undegraded":"2023-01-15T14:33:32.694928+0000","last_fullsized":"2023-01-15T14:33:32.694928+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.71109452200000001,"stat_sum":{"num_bytes":2130792,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":21399,"num_write":27,"num_write_kb":9854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9447726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":1050,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.693929+0000","last_change":"2023-01-15T14:33:32.693929+0000","last_active":"2023-01-15T14:33:32.693929+0000","last_peered":"2023-01-15T14:33:32.693929+0000","last_clean":"2023-01-15T14:33:32.693929+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:33:32.693929+0000","last_undegraded":"2023-01-15T14:33:32.693929+0000","last_fullsized":"2023-01-15T14:33:32.693929+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:38.874","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.71015085600000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.8","version":"179'204","reported_seq":1250,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.159682+0000","last_change":"2023-01-15T14:33:32.500509+0000","last_active":"2023-01-15T14:33:36.159682+0000","last_peered":"2023-01-15T14:33:36.159682+0000","last_clean":"2023-01-15T14:33:36.159682+0000","last_became_active":"2023-01-15T14:32:22.142296+0000","last_became_peered":"2023-01-15T14:32:22.142296+0000","last_unstale":"2023-01-15T14:33:36.159682+0000","last_undegraded":"2023-01-15T14:33:36.159682+0000","last_fullsized":"2023-01-15T14:33:36.159682+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.61289480399999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":474,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.965167+0000","last_change":"2023-01-15T14:32:54.588161+0000","last_active":"2023-01-15T14:33:35.965167+0000","last_peered":"2023-01-15T14:33:35.965167+0000","last_clean":"2023-01-15T14:33:35.965167+0000","last_became_active":"2023-01-15T14:31:03.091071+0000","last_became_peered":"2023-01-15T14:31:03.091071+0000","last_unstale":"2023-01-15T14:33:35.965167+0000","last_undegraded":"2023-01-15T14:33:35.965167+0000","last_fullsized":"2023-01-15T14:33:35.965167+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:16.622","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":299,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.065830+0000","last_change":"2023-01-15T14:31:38.018850+0000","last_active":"2023-01-15T14:33:36.065830+0000","last_peered":"2023-01-15T14:33:36.065830+0000","last_clean":"2023-01-15T14:33:36.065830+0000","last_became_active":"2023-01-15T14:31:38.018145+0000","last_became_peered":"2023-01-15T14:31:38.018145+0000","last_unstale":"2023-01-15T14:33:36.065830+0000","last_undegraded":"2023-01-15T14:33:36.065830+0000","last_fullsized":"2023-01-15T14:33:36.065830+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":937,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.693028+0000","last_change":"2023-01-15T14:33:32.693028+0000","last_active":"2023-01-15T14:33:32.693028+0000","last_peered":"2023-01-15T14:33:32.693028+0000","last_clean":"2023-01-15T14:33:32.693028+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:33:32.693028+0000","last_undegraded":"2023-01-15T14:33:32.693028+0000","last_fullsized":"2023-01-15T14:33:32.693028+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:35.297","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.709668725,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":373,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.965183+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:33:35.965183+0000","last_peered":"2023-01-15T14:33:35.965183+0000","last_clean":"2023-01-15T14:33:35.965183+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:33:35.965183+0000","last_undegraded":"2023-01-15T14:33:35.965183+0000","last_fullsized":"2023-01-15T14:33:35.965183+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.964","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":810,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.295127+0000","last_change":"2023-01-15T14:33:32.295127+0000","last_active":"2023-01-15T14:33:32.295127+0000","last_peered":"2023-01-15T14:33:32.295127+0000","last_clean":"2023-01-15T14:33:32.295127+0000","last_became_active":"2023-01-15T14:32:40.967729+0000","last_became_peered":"2023-01-15T14:32:40.967729+0000","last_unstale":"2023-01-15T14:33:32.295127+0000","last_undegraded":"2023-01-15T14:33:32.295127+0000","last_fullsized":"2023-01-15T14:33:32.295127+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:43.783","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.31258807,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":128,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.958570+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:33:35.958570+0000","last_peered":"2023-01-15T14:33:35.958570+0000","last_clean":"2023-01-15T14:33:35.958570+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:33:35.958570+0000","last_undegraded":"2023-01-15T14:33:35.958570+0000","last_fullsized":"2023-01-15T14:33:35.958570+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1264,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.965403+0000","last_change":"2023-01-15T14:33:33.026153+0000","last_active":"2023-01-15T14:33:35.965403+0000","last_peered":"2023-01-15T14:33:35.965403+0000","last_clean":"2023-01-15T14:33:35.965403+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:33:35.965403+0000","last_undegraded":"2023-01-15T14:33:35.965403+0000","last_fullsized":"2023-01-15T14:33:35.965403+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.103373336,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":196,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.167597+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:33:36.167597+0000","last_peered":"2023-01-15T14:33:36.167597+0000","last_clean":"2023-01-15T14:33:36.167597+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:33:36.167597+0000","last_undegraded":"2023-01-15T14:33:36.167597+0000","last_fullsized":"2023-01-15T14:33:36.167597+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:25.583","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"323'313","reported_seq":1338,"reported_epoch":326,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.696352+0000","last_change":"2023-01-15T14:33:32.697707+0000","last_active":"2023-01-15T14:33:35.696352+0000","last_peered":"2023-01-15T14:33:35.696352+0000","last_clean":"2023-01-15T14:33:35.696352+0000","last_became_active":"2023-01-15T14:33:10.346559+0000","last_became_peered":"2023-01-15T14:33:10.346559+0000","last_unstale":"2023-01-15T14:33:35.696352+0000","last_undegraded":"2023-01-15T14:33:35.696352+0000","last_fullsized":"2023-01-15T14:33:35.696352+0000","mapping_epoch":300,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.652","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.80418413300000002,"stat_sum":{"num_bytes":6050652,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":13874,"num_write":1,"num_write_kb":3135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.5","version":"328'154","reported_seq":1109,"reported_epoch":328,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.985655+0000","last_change":"2023-01-15T14:33:32.604608+0000","last_active":"2023-01-15T14:33:36.985655+0000","last_peered":"2023-01-15T14:33:36.985655+0000","last_clean":"2023-01-15T14:33:36.985655+0000","last_became_active":"2023-01-15T14:32:21.948278+0000","last_became_peered":"2023-01-15T14:32:21.948278+0000","last_unstale":"2023-01-15T14:33:36.985655+0000","last_undegraded":"2023-01-15T14:33:36.985655+0000","last_fullsized":"2023-01-15T14:33:36.985655+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:38.550","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.62009058100000003,"stat_sum":{"num_bytes":11390101,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":26145,"num_write":9,"num_write_kb":4265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":186,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.966943+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:33:35.966943+0000","last_peered":"2023-01-15T14:33:35.966943+0000","last_clean":"2023-01-15T14:33:35.966943+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:33:35.966943+0000","last_undegraded":"2023-01-15T14:33:35.966943+0000","last_fullsized":"2023-01-15T14:33:35.966943+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:37.670","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"294'305","reported_seq":1353,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.966813+0000","last_change":"2023-01-15T14:33:32.014348+0000","last_active":"2023-01-15T14:33:35.966813+0000","last_peered":"2023-01-15T14:33:35.966813+0000","last_clean":"2023-01-15T14:33:35.966813+0000","last_became_active":"2023-01-15T14:32:22.247573+0000","last_became_peered":"2023-01-15T14:32:22.247573+0000","last_unstale":"2023-01-15T14:33:35.966813+0000","last_undegraded":"2023-01-15T14:33:35.966813+0000","last_fullsized":"2023-01-15T14:33:35.966813+0000","mapping_epoch":181,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.100686466,"stat_sum":{"num_bytes":904090,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":29,"num_read_kb":30683,"num_write":27,"num_write_kb":9142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7931053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":365,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.967112+0000","last_change":"2023-01-15T14:32:34.578760+0000","last_active":"2023-01-15T14:33:35.967112+0000","last_peered":"2023-01-15T14:33:35.967112+0000","last_clean":"2023-01-15T14:33:35.967112+0000","last_became_active":"2023-01-15T14:31:03.088392+0000","last_became_peered":"2023-01-15T14:31:03.088392+0000","last_unstale":"2023-01-15T14:33:35.967112+0000","last_undegraded":"2023-01-15T14:33:35.967112+0000","last_fullsized":"2023-01-15T14:33:35.967112+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:38.541","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"324'133","reported_seq":1070,"reported_epoch":324,"state":"active+clean","last_fresh":"2023-01-15T14:33:33.154570+0000","last_change":"2023-01-15T14:33:32.192977+0000","last_active":"2023-01-15T14:33:33.154570+0000","last_peered":"2023-01-15T14:33:33.154570+0000","last_clean":"2023-01-15T14:33:33.154570+0000","last_became_active":"2023-01-15T14:32:22.146367+0000","last_became_peered":"2023-01-15T14:32:22.146367+0000","last_unstale":"2023-01-15T14:33:33.154570+0000","last_undegraded":"2023-01-15T14:33:33.154570+0000","last_fullsized":"2023-01-15T14:33:33.154570+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:03.357","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.30560775299999998,"stat_sum":{"num_bytes":8547430,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":13597,"num_write":9,"num_write_kb":7710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":186,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.968924+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:33:35.968924+0000","last_peered":"2023-01-15T14:33:35.968924+0000","last_clean":"2023-01-15T14:33:35.968924+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:33:35.968924+0000","last_undegraded":"2023-01-15T14:33:35.968924+0000","last_fullsized":"2023-01-15T14:33:35.968924+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:17.360","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":961,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.968530+0000","last_change":"2023-01-15T14:33:32.218261+0000","last_active":"2023-01-15T14:33:35.968530+0000","last_peered":"2023-01-15T14:33:35.968530+0000","last_clean":"2023-01-15T14:33:35.968530+0000","last_became_active":"2023-01-15T14:32:22.247696+0000","last_became_peered":"2023-01-15T14:32:22.247696+0000","last_unstale":"2023-01-15T14:33:35.968530+0000","last_undegraded":"2023-01-15T14:33:35.968530+0000","last_fullsized":"2023-01-15T14:33:35.968530+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:09.533","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.30218858300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":308,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.966180+0000","last_change":"2023-01-15T14:31:38.112378+0000","last_active":"2023-01-15T14:33:35.966180+0000","last_peered":"2023-01-15T14:33:35.966180+0000","last_clean":"2023-01-15T14:33:35.966180+0000","last_became_active":"2023-01-15T14:31:38.111590+0000","last_became_peered":"2023-01-15T14:31:38.111590+0000","last_unstale":"2023-01-15T14:33:35.966180+0000","last_undegraded":"2023-01-15T14:33:35.966180+0000","last_fullsized":"2023-01-15T14:33:35.966180+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.14","version":"245'148","reported_seq":1113,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.294036+0000","last_change":"2023-01-15T14:33:32.294036+0000","last_active":"2023-01-15T14:33:32.294036+0000","last_peered":"2023-01-15T14:33:32.294036+0000","last_clean":"2023-01-15T14:33:32.294036+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:33:32.294036+0000","last_undegraded":"2023-01-15T14:33:32.294036+0000","last_fullsized":"2023-01-15T14:33:32.294036+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:32:19.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:19.303243+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.40660142900000001,"stat_sum":{"num_bytes":1226927,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":20384,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":932,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.967451+0000","last_change":"2023-01-15T14:33:32.503069+0000","last_active":"2023-01-15T14:33:35.967451+0000","last_peered":"2023-01-15T14:33:35.967451+0000","last_clean":"2023-01-15T14:33:35.967451+0000","last_became_active":"2023-01-15T14:33:20.233417+0000","last_became_peered":"2023-01-15T14:33:20.233417+0000","last_unstale":"2023-01-15T14:33:35.967451+0000","last_undegraded":"2023-01-15T14:33:35.967451+0000","last_fullsized":"2023-01-15T14:33:35.967451+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.51955222899999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.16","version":"320'127","reported_seq":569,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.968785+0000","last_change":"2023-01-15T14:33:35.161101+0000","last_active":"2023-01-15T14:33:35.968785+0000","last_peered":"2023-01-15T14:33:35.968785+0000","last_clean":"2023-01-15T14:33:35.968785+0000","last_became_active":"2023-01-15T14:33:35.159018+0000","last_became_peered":"2023-01-15T14:33:35.159018+0000","last_unstale":"2023-01-15T14:33:35.968785+0000","last_undegraded":"2023-01-15T14:33:35.968785+0000","last_fullsized":"2023-01-15T14:33:35.968785+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.306292024,"stat_sum":{"num_bytes":3689235,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10280,"num_write":6,"num_write_kb":2575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":983,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.966411+0000","last_change":"2023-01-15T14:33:35.260719+0000","last_active":"2023-01-15T14:33:35.966411+0000","last_peered":"2023-01-15T14:33:35.966411+0000","last_clean":"2023-01-15T14:33:35.966411+0000","last_became_active":"2023-01-15T14:33:35.259364+0000","last_became_peered":"2023-01-15T14:33:35.259364+0000","last_unstale":"2023-01-15T14:33:35.966411+0000","last_undegraded":"2023-01-15T14:33:35.966411+0000","last_fullsized":"2023-01-15T14:33:35.966411+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.61244699199999997,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.18","version":"323'320","reported_seq":1429,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.059282+0000","last_change":"2023-01-15T14:33:35.164823+0000","last_active":"2023-01-15T14:33:36.059282+0000","last_peered":"2023-01-15T14:33:36.059282+0000","last_clean":"2023-01-15T14:33:36.059282+0000","last_became_active":"2023-01-15T14:33:35.163238+0000","last_became_peered":"2023-01-15T14:33:35.163238+0000","last_unstale":"2023-01-15T14:33:36.059282+0000","last_undegraded":"2023-01-15T14:33:36.059282+0000","last_fullsized":"2023-01-15T14:33:36.059282+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:06.616","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.51997143199999996,"stat_sum":{"num_bytes":4332078,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":24,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.19","version":"294'256","reported_seq":1251,"reported_epoch":328,"state":"active+clean+scrubbing","last_fresh":"2023-01-15T14:33:36.989938+0000","last_change":"2023-01-15T14:33:37.452674+0000","last_active":"2023-01-15T14:33:36.989938+0000","last_peered":"2023-01-15T14:33:36.989938+0000","last_clean":"2023-01-15T14:33:36.989938+0000","last_became_active":"2023-01-15T14:33:35.259904+0000","last_became_peered":"2023-01-15T14:33:35.259904+0000","last_unstale":"2023-01-15T14:33:36.989938+0000","last_undegraded":"2023-01-15T14:33:36.989938+0000","last_fullsized":"2023-01-15T14:33:36.989938+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.102550445,"stat_sum":{"num_bytes":1210137,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":3549,"num_write":5,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.1a","version":"323'344","reported_seq":1371,"reported_epoch":324,"state":"active+clean","last_fresh":"2023-01-15T14:33:33.023852+0000","last_change":"2023-01-15T14:33:33.025112+0000","last_active":"2023-01-15T14:33:33.023852+0000","last_peered":"2023-01-15T14:33:33.023852+0000","last_clean":"2023-01-15T14:33:33.023852+0000","last_became_active":"2023-01-15T14:31:52.124026+0000","last_became_peered":"2023-01-15T14:31:52.124026+0000","last_unstale":"2023-01-15T14:33:33.023852+0000","last_undegraded":"2023-01-15T14:33:33.023852+0000","last_fullsized":"2023-01-15T14:33:33.023852+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:49.729","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.099883880999999994,"stat_sum":{"num_bytes":6105307,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":9,"num_read_kb":8573,"num_write":12,"num_write_kb":1196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.1b","version":"283'80","reported_seq":934,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.696080+0000","last_change":"2023-01-15T14:33:32.696080+0000","last_active":"2023-01-15T14:33:32.696080+0000","last_peered":"2023-01-15T14:33:32.696080+0000","last_clean":"2023-01-15T14:33:32.696080+0000","last_became_active":"2023-01-15T14:31:52.321399+0000","last_became_peered":"2023-01-15T14:31:52.321399+0000","last_unstale":"2023-01-15T14:33:32.696080+0000","last_undegraded":"2023-01-15T14:33:32.696080+0000","last_fullsized":"2023-01-15T14:33:32.696080+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:00.204","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.71116357299999999,"stat_sum":{"num_bytes":63320,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":14,"num_read_kb":21019,"num_write":6,"num_write_kb":2022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":458,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.698285+0000","last_change":"2023-01-15T14:33:32.698285+0000","last_active":"2023-01-15T14:33:32.698285+0000","last_peered":"2023-01-15T14:33:32.698285+0000","last_clean":"2023-01-15T14:33:32.698285+0000","last_became_active":"2023-01-15T14:31:51.926652+0000","last_became_peered":"2023-01-15T14:31:51.926652+0000","last_unstale":"2023-01-15T14:33:32.698285+0000","last_undegraded":"2023-01-15T14:33:32.698285+0000","last_fullsized":"2023-01-15T14:33:32.698285+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:33:35.143","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.71196858399999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":579833,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1195,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.090951+0000","last_change":"2023-01-15T14:33:32.090951+0000","last_active":"2023-01-15T14:33:32.090951+0000","last_peered":"2023-01-15T14:33:32.090951+0000","last_clean":"2023-01-15T14:33:32.090951+0000","last_became_active":"2023-01-15T14:31:52.025227+0000","last_became_peered":"2023-01-15T14:31:52.025227+0000","last_unstale":"2023-01-15T14:33:32.090951+0000","last_undegraded":"2023-01-15T14:33:32.090951+0000","last_fullsized":"2023-01-15T14:33:32.090951+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.204838885,"stat_sum":{"num_bytes":5739855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":47622,"num_write":25,"num_write_kb":10630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":6928644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.1e","version":"215'39","reported_seq":920,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.070289+0000","last_change":"2023-01-15T14:33:35.264095+0000","last_active":"2023-01-15T14:33:36.070289+0000","last_peered":"2023-01-15T14:33:36.070289+0000","last_clean":"2023-01-15T14:33:36.070289+0000","last_became_active":"2023-01-15T14:33:35.261561+0000","last_became_peered":"2023-01-15T14:33:35.261561+0000","last_unstale":"2023-01-15T14:33:36.070289+0000","last_undegraded":"2023-01-15T14:33:36.070289+0000","last_fullsized":"2023-01-15T14:33:36.070289+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.41141337700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.1f","version":"318'256","reported_seq":1309,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.092037+0000","last_change":"2023-01-15T14:33:32.092037+0000","last_active":"2023-01-15T14:33:32.092037+0000","last_peered":"2023-01-15T14:33:32.092037+0000","last_clean":"2023-01-15T14:33:32.092037+0000","last_became_active":"2023-01-15T14:31:52.124821+0000","last_became_peered":"2023-01-15T14:31:52.124821+0000","last_unstale":"2023-01-15T14:33:32.092037+0000","last_undegraded":"2023-01-15T14:33:32.092037+0000","last_fullsized":"2023-01-15T14:33:32.092037+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:55.650","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.205157009,"stat_sum":{"num_bytes":5949777,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":20,"num_read_kb":26683,"num_write":2,"num_write_kb":3398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.20","version":"324'273","reported_seq":1321,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.964718+0000","last_change":"2023-01-15T14:33:34.345600+0000","last_active":"2023-01-15T14:33:35.964718+0000","last_peered":"2023-01-15T14:33:35.964718+0000","last_clean":"2023-01-15T14:33:35.964718+0000","last_became_active":"2023-01-15T14:31:07.906191+0000","last_became_peered":"2023-01-15T14:31:07.906191+0000","last_unstale":"2023-01-15T14:33:35.964718+0000","last_undegraded":"2023-01-15T14:33:35.964718+0000","last_fullsized":"2023-01-15T14:33:35.964718+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.39764039699999998,"stat_sum":{"num_bytes":8502274,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":21728,"num_write":19,"num_write_kb":8281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":8877936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.21","version":"321'138","reported_seq":974,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.165736+0000","last_change":"2023-01-15T14:33:32.105747+0000","last_active":"2023-01-15T14:33:36.165736+0000","last_peered":"2023-01-15T14:33:36.165736+0000","last_clean":"2023-01-15T14:33:36.165736+0000","last_became_active":"2023-01-15T14:31:38.233481+0000","last_became_peered":"2023-01-15T14:31:38.233481+0000","last_unstale":"2023-01-15T14:33:36.165736+0000","last_undegraded":"2023-01-15T14:33:36.165736+0000","last_fullsized":"2023-01-15T14:33:36.165736+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:05.612","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.001466766,"stat_sum":{"num_bytes":2067327,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":24236,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.22","version":"323'161","reported_seq":1243,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.059434+0000","last_change":"2023-01-15T14:33:33.176099+0000","last_active":"2023-01-15T14:33:36.059434+0000","last_peered":"2023-01-15T14:33:36.059434+0000","last_clean":"2023-01-15T14:33:36.059434+0000","last_became_active":"2023-01-15T14:31:08.084483+0000","last_became_peered":"2023-01-15T14:31:08.084483+0000","last_unstale":"2023-01-15T14:33:36.059434+0000","last_undegraded":"2023-01-15T14:33:36.059434+0000","last_fullsized":"2023-01-15T14:33:36.059434+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.14031706799999999,"stat_sum":{"num_bytes":5210269,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":13467,"num_write":17,"num_write_kb":10028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.23","version":"149'233","reported_seq":1124,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.065975+0000","last_change":"2023-01-15T14:33:32.503689+0000","last_active":"2023-01-15T14:33:36.065975+0000","last_peered":"2023-01-15T14:33:36.065975+0000","last_clean":"2023-01-15T14:33:36.065975+0000","last_became_active":"2023-01-15T14:31:06.029716+0000","last_became_peered":"2023-01-15T14:31:06.029716+0000","last_unstale":"2023-01-15T14:33:36.065975+0000","last_undegraded":"2023-01-15T14:33:36.065975+0000","last_fullsized":"2023-01-15T14:33:36.065975+0000","mapping_epoch":178,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:08.017","scrub_duration":0.75838583400000004,"objects_trimmed":0,"snaptrim_duration":0.51961349599999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.24","version":"323'249","reported_seq":1317,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.965741+0000","last_change":"2023-01-15T14:33:32.492899+0000","last_active":"2023-01-15T14:33:35.965741+0000","last_peered":"2023-01-15T14:33:35.965741+0000","last_clean":"2023-01-15T14:33:35.965741+0000","last_became_active":"2023-01-15T14:31:26.168777+0000","last_became_peered":"2023-01-15T14:31:26.168777+0000","last_unstale":"2023-01-15T14:33:35.965741+0000","last_undegraded":"2023-01-15T14:33:35.965741+0000","last_fullsized":"2023-01-15T14:33:35.965741+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.61128966100000004,"stat_sum":{"num_bytes":7097465,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":21546,"num_write":9,"num_write_kb":5482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.25","version":"320'100","reported_seq":1039,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.969151+0000","last_change":"2023-01-15T14:33:32.015164+0000","last_active":"2023-01-15T14:33:35.969151+0000","last_peered":"2023-01-15T14:33:35.969151+0000","last_clean":"2023-01-15T14:33:35.969151+0000","last_became_active":"2023-01-15T14:31:22.000386+0000","last_became_peered":"2023-01-15T14:31:22.000386+0000","last_unstale":"2023-01-15T14:33:35.969151+0000","last_undegraded":"2023-01-15T14:33:35.969151+0000","last_fullsized":"2023-01-15T14:33:35.969151+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.101097506,"stat_sum":{"num_bytes":5277530,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":6061,"num_write":2,"num_write_kb":5154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.26","version":"102'52","reported_seq":390,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.065851+0000","last_change":"2023-01-15T14:33:32.929735+0000","last_active":"2023-01-15T14:33:36.065851+0000","last_peered":"2023-01-15T14:33:36.065851+0000","last_clean":"2023-01-15T14:33:36.065851+0000","last_became_active":"2023-01-15T14:31:21.901622+0000","last_became_peered":"2023-01-15T14:31:21.901622+0000","last_unstale":"2023-01-15T14:33:36.065851+0000","last_undegraded":"2023-01-15T14:33:36.065851+0000","last_fullsized":"2023-01-15T14:33:36.065851+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:08.119","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.82485724500000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":997,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.965893+0000","last_change":"2023-01-15T14:33:32.294352+0000","last_active":"2023-01-15T14:33:35.965893+0000","last_peered":"2023-01-15T14:33:35.965893+0000","last_clean":"2023-01-15T14:33:35.965893+0000","last_became_active":"2023-01-15T14:31:22.091646+0000","last_became_peered":"2023-01-15T14:31:22.091646+0000","last_unstale":"2023-01-15T14:33:35.965893+0000","last_undegraded":"2023-01-15T14:33:35.965893+0000","last_fullsized":"2023-01-15T14:33:35.965893+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:41.261","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.41147862499999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.28","version":"321'244","reported_seq":1311,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.058534+0000","last_change":"2023-01-15T14:33:31.889655+0000","last_active":"2023-01-15T14:33:36.058534+0000","last_peered":"2023-01-15T14:33:36.058534+0000","last_clean":"2023-01-15T14:33:36.058534+0000","last_became_active":"2023-01-15T14:31:26.054878+0000","last_became_peered":"2023-01-15T14:31:26.054878+0000","last_unstale":"2023-01-15T14:33:36.058534+0000","last_undegraded":"2023-01-15T14:33:36.058534+0000","last_fullsized":"2023-01-15T14:33:36.058534+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":244,"log_dups_size":0,"ondisk_log_size":244,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:01.610","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.0082439899999999997,"stat_sum":{"num_bytes":6130436,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":17938,"num_write":16,"num_write_kb":6906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":12193528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.29","version":"110'149","reported_seq":819,"reported_epoch":328,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.883314+0000","last_change":"2023-01-15T14:33:32.501333+0000","last_active":"2023-01-15T14:33:36.883314+0000","last_peered":"2023-01-15T14:33:36.883314+0000","last_clean":"2023-01-15T14:33:36.883314+0000","last_became_active":"2023-01-15T14:31:25.136376+0000","last_became_peered":"2023-01-15T14:31:25.136376+0000","last_unstale":"2023-01-15T14:33:36.883314+0000","last_undegraded":"2023-01-15T14:33:36.883314+0000","last_fullsized":"2023-01-15T14:33:36.883314+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:23.471","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.51989621100000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1174,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.964536+0000","last_change":"2023-01-15T14:33:34.048952+0000","last_active":"2023-01-15T14:33:35.964536+0000","last_peered":"2023-01-15T14:33:35.964536+0000","last_clean":"2023-01-15T14:33:35.964536+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:33:35.964536+0000","last_undegraded":"2023-01-15T14:33:35.964536+0000","last_fullsized":"2023-01-15T14:33:35.964536+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.0010047350000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":892,"reported_epoch":328,"state":"active+clean","last_fresh":"2023-01-15T14:33:37.189336+0000","last_change":"2023-01-15T14:33:32.220974+0000","last_active":"2023-01-15T14:33:37.189336+0000","last_peered":"2023-01-15T14:33:37.189336+0000","last_clean":"2023-01-15T14:33:37.189336+0000","last_became_active":"2023-01-15T14:31:21.896494+0000","last_became_peered":"2023-01-15T14:31:21.896494+0000","last_unstale":"2023-01-15T14:33:37.189336+0000","last_undegraded":"2023-01-15T14:33:37.189336+0000","last_fullsized":"2023-01-15T14:33:37.189336+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:15.975","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.206276549,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":374,"reported_epoch":323,"state":"active+clean","last_fresh":"2023-01-15T14:33:32.089718+0000","last_change":"2023-01-15T14:33:32.089718+0000","last_active":"2023-01-15T14:33:32.089718+0000","last_peered":"2023-01-15T14:33:32.089718+0000","last_clean":"2023-01-15T14:33:32.089718+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:33:32.089718+0000","last_undegraded":"2023-01-15T14:33:32.089718+0000","last_fullsized":"2023-01-15T14:33:32.089718+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:10.970","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.20421467400000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1148,"reported_epoch":328,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.892741+0000","last_change":"2023-01-15T14:33:33.032948+0000","last_active":"2023-01-15T14:33:36.892741+0000","last_peered":"2023-01-15T14:33:36.892741+0000","last_clean":"2023-01-15T14:33:36.892741+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:33:36.892741+0000","last_undegraded":"2023-01-15T14:33:36.892741+0000","last_fullsized":"2023-01-15T14:33:36.892741+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.92665474400000003,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.2e","version":"139'38","reported_seq":826,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.968422+0000","last_change":"2023-01-15T14:33:32.218629+0000","last_active":"2023-01-15T14:33:35.968422+0000","last_peered":"2023-01-15T14:33:35.968422+0000","last_clean":"2023-01-15T14:33:35.968422+0000","last_became_active":"2023-01-15T14:31:38.116756+0000","last_became_peered":"2023-01-15T14:31:38.116756+0000","last_unstale":"2023-01-15T14:33:35.968422+0000","last_undegraded":"2023-01-15T14:33:35.968422+0000","last_fullsized":"2023-01-15T14:33:35.968422+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.30277812700000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1022,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.967244+0000","last_change":"2023-01-15T14:33:32.931570+0000","last_active":"2023-01-15T14:33:35.967244+0000","last_peered":"2023-01-15T14:33:35.967244+0000","last_clean":"2023-01-15T14:33:35.967244+0000","last_became_active":"2023-01-15T14:31:40.024531+0000","last_became_peered":"2023-01-15T14:31:40.024531+0000","last_unstale":"2023-01-15T14:33:35.967244+0000","last_undegraded":"2023-01-15T14:33:35.967244+0000","last_fullsized":"2023-01-15T14:33:35.967244+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.82532748300000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.30","version":"206'252","reported_seq":1275,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:36.064688+0000","last_change":"2023-01-15T14:33:33.639145+0000","last_active":"2023-01-15T14:33:36.064688+0000","last_peered":"2023-01-15T14:33:36.064688+0000","last_clean":"2023-01-15T14:33:36.064688+0000","last_became_active":"2023-01-15T14:31:39.915702+0000","last_became_peered":"2023-01-15T14:31:39.915702+0000","last_unstale":"2023-01-15T14:33:36.064688+0000","last_undegraded":"2023-01-15T14:33:36.064688+0000","last_fullsized":"2023-01-15T14:33:36.064688+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.1539330249999999,"objects_trimmed":0,"snaptrim_duration":0.70076556999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]},{"pgid":"3.31","version":"204'150","reported_seq":1160,"reported_epoch":327,"state":"active+clean","last_fresh":"2023-01-15T14:33:35.964660+0000","last_change":"2023-01-15T14:33:32.219673+0000","last_active":"2023-01-15T14:33:35.964660+0000","last_peered":"2023-01-15T14:33:35.964660+0000","last_clean":"2023-01-15T14:33:35.964660+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:33:35.964660+0000","last_undegraded":"2023-01-15T14:33:35.964660+0000","last_fullsized":"2023-01-15T14:33:35.964660+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.20631559799999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"fa","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":96,"stat_sum":{"num_bytes":266046391,"num_objects":117,"num_object_clones":69,"num_object_copies":351,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":117,"num_whiteouts":9,"num_read":798,"num_read_kb":830852,"num_write":652,"num_write_kb":264276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":278,"num_bytes_recovered":168287251,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15980,"ondisk_log_size":15980,"up":282,"acting":279,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574916,"num_pgs":36,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":962276,"kb_used_data":92742984,"kb_used_omap":6174,"kb_used_meta":0,"kb_avail":92737104,"statfs":{"total":95948165120,"available":94962794496,"internally_reserved":6021120,"allocated":94968815616,"data_stored":94968815616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6322845,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":4.1269999999999998,"15min":4.0890000000000004},"min":{"1min":0.54000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2030000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":9.5380000000000003,"5min":6.3200000000000003,"15min":8.2460000000000004},"min":{"1min":0.58799999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":96.337000000000003,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.72799999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":3.6509999999999998,"15min":5.5800000000000001},"min":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":34.585999999999999,"5min":88.003,"15min":88.003},"last":204.74199999999999},{"interface":"front","average":{"1min":13.302,"5min":8.1950000000000003,"15min":6.9180000000000001},"min":{"1min":0.58799999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.403999999999996,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.64600000000000002}]},{"osd":2,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":6.9400000000000004,"15min":5.5069999999999997},"min":{"1min":0.57199999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.179,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":2.5790000000000002,"15min":3.645},"min":{"1min":0.626,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":91.837999999999994,"15min":91.837999999999994},"last":0.76000000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":7.3799999999999999,"15min":3.0920000000000001},"min":{"1min":0.63600000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2370000000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":3.6360000000000001,"15min":4.3129999999999997},"min":{"1min":0.65000000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.2270000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.85199999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5519999999999996,"5min":5.6219999999999999,"15min":8.1509999999999998},"min":{"1min":0.63500000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":96.185000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.9259999999999999,"5min":5.782,"15min":2.5699999999999998},"min":{"1min":0.73099999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":22.41,"5min":164.21199999999999,"15min":164.21199999999999},"last":1.25}]},{"osd":5,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.6440000000000001,"5min":5.0659999999999998,"15min":5.9059999999999997},"min":{"1min":0.72799999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":96.263999999999996,"5min":96.263999999999996,"15min":96.263999999999996},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.998,"5min":3.206,"15min":4.3650000000000002},"min":{"1min":0.69299999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.355,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.1399999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":5.5949999999999998,"15min":7.5469999999999997},"min":{"1min":0.83599999999999997,"5min":0.496,"15min":0.496},"max":{"1min":96.299999999999997,"5min":96.299999999999997,"15min":96.299999999999997},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.089,"5min":2.5150000000000001,"15min":1.571},"min":{"1min":0.84299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.2789999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.198}]}]},{"osd":6,"up_from":0,"seq":55834574917,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574917,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574915,"num_pgs":61,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":946432,"kb_used_data":92759008,"kb_used_omap":11577,"kb_used_meta":0,"kb_avail":92752948,"statfs":{"total":95948165120,"available":94979018752,"internally_reserved":6205440,"allocated":94985224192,"data_stored":94985224192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11855360,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":23.594000000000001,"15min":11.141},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.133,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.78300000000000003,"5min":3.0009999999999999,"15min":1.5209999999999999},"min":{"1min":0.58299999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.024,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.98599999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":5.0730000000000004,"15min":7.4199999999999999},"min":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6659999999999999,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.343,"15min":1.9770000000000001},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0529999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.67100000000000004}]},{"osd":2,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":24.227,"15min":11.749000000000001},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.157,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":2.6040000000000001,"5min":1.21,"15min":1.01},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":32.081000000000003,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.92400000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":2.9169999999999998,"15min":1.865},"min":{"1min":0.65300000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.2909999999999999,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.7370000000000001,"5min":24.498999999999999,"15min":12.266999999999999},"min":{"1min":0.64400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":14.961,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":1.1040000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.88500000000000001,"15min":0.90800000000000003},"min":{"1min":0.55200000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":1.036},{"interface":"front","average":{"1min":0.92300000000000004,"5min":2.1840000000000002,"15min":2.2599999999999998},"min":{"1min":0.71799999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3200000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.85099999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":3.9329999999999998,"15min":5.093},"min":{"1min":0.69599999999999995,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.365,"5min":99.780000000000001,"15min":99.780000000000001},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":22.463999999999999,"15min":10.93},"min":{"1min":0.71599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.427,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.173}]},{"osd":7,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":3.1539999999999999,"5min":5.5570000000000004,"15min":10.603999999999999},"min":{"1min":0.76700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":39.012999999999998,"5min":179.73500000000001,"15min":179.73500000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.2370000000000001,"5min":24.864000000000001,"15min":15.779999999999999},"min":{"1min":0.877,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.0680000000000001,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.276}]}]},{"osd":3,"up_from":13,"seq":55834574916,"num_pgs":69,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1055880,"kb_used_data":92647948,"kb_used_omap":12360,"kb_used_meta":0,"kb_avail":92643500,"statfs":{"total":95948165120,"available":94866944000,"internally_reserved":4554752,"allocated":94871498752,"data_stored":94871498752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12657198,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":20,"commit_latency_ns":0,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":7.4400000000000004,"15min":5.7119999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":17.731000000000002,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":3.4340000000000002,"15min":6.2389999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.9260000000000002,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.81599999999999995}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":7.1790000000000003,"15min":11.414},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96799999999999997,"5min":159.256,"15min":159.256},"last":0.57499999999999996},{"interface":"front","average":{"1min":2.6259999999999999,"5min":10.417999999999999,"15min":3.9660000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":17.826000000000001,"5min":174.578,"15min":174.578},"last":1.006}]},{"osd":2,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":4.6849999999999996,"5min":2.8809999999999998,"15min":3.1930000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":86.412000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.95,"5min":6.7889999999999997,"15min":3.7250000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.876000000000001,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.90200000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":7.6879999999999997,"5min":29.686,"15min":14.27},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":84.640000000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":32.134999999999998,"15min":19.960999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.615,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":0.85199999999999998}]},{"osd":5,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":6.4980000000000002,"15min":11.433999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.04,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.95199999999999996},{"interface":"front","average":{"1min":2.2309999999999999,"5min":6.0039999999999996,"15min":2.6819999999999999},"min":{"1min":0.70099999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":17.917000000000002,"5min":174.61000000000001,"15min":174.61000000000001},"last":1.089}]},{"osd":6,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1949999999999998,"5min":5.9980000000000002,"15min":2.6869999999999998},"min":{"1min":0.66000000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":17.989000000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.964,"15min":3.1619999999999999},"min":{"1min":0.70499999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.23,"5min":60.098999999999997,"15min":60.098999999999997},"last":1.1559999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":2.0219999999999998,"15min":3.3639999999999999},"min":{"1min":0.69499999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.266,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.1080000000000001},{"interface":"front","average":{"1min":19.138000000000002,"5min":11.705,"15min":15.961},"min":{"1min":0.66200000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":281.13400000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":1.1339999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574916,"num_pgs":52,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1064044,"kb_used_data":92645132,"kb_used_omap":10172,"kb_used_meta":0,"kb_avail":92635336,"statfs":{"total":95948165120,"available":94858584064,"internally_reserved":10031104,"allocated":94868615168,"data_stored":94868615168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10417022,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":3.109,"5min":3.5230000000000001,"15min":1.601},"min":{"1min":0.437,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":51.009999999999998,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.4299999999999999,"15min":0.95299999999999996},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0089999999999999,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.56399999999999995}]},{"osd":1,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.1529999999999996,"5min":5.9059999999999997,"15min":2.5009999999999999},"min":{"1min":0.378,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":91.525999999999996,"5min":143.071,"15min":143.071},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":2.016,"15min":2.9089999999999998},"min":{"1min":0.42299999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.131,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.85399999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":8.2210000000000001,"15min":9.6750000000000007},"min":{"1min":0.5,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.3609999999999998,"5min":183.53,"15min":183.53},"last":0.53700000000000003},{"interface":"front","average":{"1min":16.692,"5min":7.5170000000000003,"15min":3.0169999999999999},"min":{"1min":0.61199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":155.47900000000001,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.68000000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3650000000000002,"5min":6.5060000000000002,"15min":4.6020000000000003},"min":{"1min":0.67300000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":51.259999999999998,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":2.8809999999999998,"5min":5.2939999999999996,"15min":4.3040000000000003},"min":{"1min":0.48699999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":33.219999999999999,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.96099999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":2.4729999999999999,"15min":1.6080000000000001},"min":{"1min":0.57199999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2130000000000001,"5min":143.262,"15min":143.262},"last":0.98499999999999999},{"interface":"front","average":{"1min":3.4780000000000002,"5min":3.0539999999999998,"15min":1.792},"min":{"1min":0.70999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":51.326000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":1.181}]},{"osd":6,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.857,"15min":1.3069999999999999},"min":{"1min":0.60399999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.365,"5min":88.542000000000002,"15min":88.542000000000002},"last":1.212},{"interface":"front","average":{"1min":1.018,"5min":2.6160000000000001,"15min":1.6839999999999999},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.5,"5min":143.215,"15min":143.215},"last":1.121}]},{"osd":7,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":13.092000000000001,"5min":6.3120000000000003,"15min":4.1429999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":120.905,"5min":120.905,"15min":120.905},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":2.8450000000000002,"15min":2.7989999999999999},"min":{"1min":0.66600000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.552,"5min":143.17099999999999,"15min":143.17099999999999},"last":1.0660000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":68,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":988820,"kb_used_data":92710572,"kb_used_omap":11671,"kb_used_meta":0,"kb_avail":92710560,"statfs":{"total":95948165120,"available":94935613440,"internally_reserved":12288,"allocated":94935625728,"data_stored":94935625728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11951847,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":12,"commit_latency_ns":0,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":3.633,"15min":3.806},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":3.9140000000000001,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":7.742,"5min":4.6130000000000004,"15min":4.4189999999999996},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":119.39100000000001,"5min":119.39100000000001,"15min":119.39100000000001},"last":1.8120000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":2.6469999999999998,"15min":1.5269999999999999},"min":{"1min":0.29799999999999999,"5min":0.245,"15min":0.245},"max":{"1min":95.879999999999995,"5min":95.879999999999995,"15min":95.879999999999995},"last":1.1100000000000001},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.8620000000000001,"15min":3.3359999999999999},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.9780000000000002,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.46300000000000002}]},{"osd":3,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.8920000000000003,"5min":9.6750000000000007,"15min":16.449000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":100.137,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":7.782,"5min":6.8899999999999997,"15min":7.1260000000000003},"min":{"1min":0.45600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":119.804,"5min":119.804,"15min":119.804},"last":1.8720000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:33:17 2023","interfaces":[{"interface":"back","average":{"1min":7.9249999999999998,"5min":17.140999999999998,"15min":8.8650000000000002},"min":{"1min":0.504,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":119.71899999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":1.9199999999999999},{"interface":"front","average":{"1min":1.7849999999999999,"5min":15.526,"15min":7.5780000000000003},"min":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":16.989999999999998,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.80900000000000005}]},{"osd":5,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.5449999999999999,"15min":1.2030000000000001},"min":{"1min":0.59899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.278,"5min":65.120999999999995,"15min":65.120999999999995},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.6579999999999999,"15min":3.5110000000000001},"min":{"1min":0.60599999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":4.0780000000000003,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.84299999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":8.0069999999999997,"5min":5.0129999999999999,"15min":5.0049999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":119.759,"5min":119.759,"15min":119.759},"last":2.0059999999999998},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.603,"15min":1.2829999999999999},"min":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.349,"5min":65.156000000000006,"15min":65.156000000000006},"last":1.0329999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":3.6030000000000002,"5min":6.1109999999999998,"15min":6.7149999999999999},"min":{"1min":0.56000000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":31.620999999999999,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.75},{"interface":"front","average":{"1min":8.0779999999999994,"5min":5.0060000000000002,"15min":4.835},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":119.65300000000001,"5min":119.65300000000001,"15min":119.65300000000001},"last":1.964}]}]},{"osd":0,"up_from":13,"seq":55834574917,"num_pgs":17,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":867760,"kb_used_data":92831724,"kb_used_omap":5583,"kb_used_meta":0,"kb_avail":92831620,"statfs":{"total":95948165120,"available":95059578880,"internally_reserved":106496,"allocated":95059685376,"data_stored":95059685376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5717875,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":10,"commit_latency_ns":0,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":3.1219999999999999,"15min":1.456},"min":{"1min":0.40899999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1060000000000001,"5min":69.872,"15min":69.872},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":3.2309999999999999,"15min":1.5629999999999999},"min":{"1min":0.51300000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.2350000000000001,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.753}]},{"osd":2,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":3.4220000000000002,"15min":1.603},"min":{"1min":0.46999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.444,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":2.6560000000000001,"15min":1.3580000000000001},"min":{"1min":0.52000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.302,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.60599999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":17.571999999999999,"5min":10.789,"15min":14.782},"min":{"1min":0.53300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":199.08000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":1.835,"5min":7.306,"15min":2.9689999999999999},"min":{"1min":0.56599999999999995,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":15.601000000000001,"5min":174.09399999999999,"15min":174.09399999999999},"last":1.073}]},{"osd":4,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":16.695,"15min":6.2220000000000004},"min":{"1min":0.73199999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1879999999999999,"5min":1486.309,"15min":1486.309},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.038,"5min":17.925000000000001,"15min":6.6580000000000004},"min":{"1min":0.82499999999999996,"5min":0.439,"15min":0.439},"max":{"1min":1.3759999999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":1.0249999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":4.6749999999999998,"15min":8.0899999999999999},"min":{"1min":0.62,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.304,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.008,"5min":1.73,"15min":1.2689999999999999},"min":{"1min":0.69199999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2529999999999999,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.89100000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":3.5910000000000002,"15min":1.9770000000000001},"min":{"1min":0.67800000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5960000000000001,"5min":70.259,"15min":70.259},"last":1.113},{"interface":"front","average":{"1min":1.0880000000000001,"5min":3.0209999999999999,"15min":1.7629999999999999},"min":{"1min":0.83699999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5489999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":1.5409999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":6.1200000000000001,"15min":12.487},"min":{"1min":0.78400000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.48,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.995},{"interface":"front","average":{"1min":1.117,"5min":3.6070000000000002,"15min":1.9830000000000001},"min":{"1min":0.76100000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5229999999999999,"5min":70.322000000000003,"15min":70.322000000000003},"last":1.1779999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:33:38.739 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:33:38.739 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 106 2023-01-15T14:33:38.741 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:38.736+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:38.741 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:38.741 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:38.742 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:33:38.841 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:38.839+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:38.953 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:38.953 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:38.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:33:38.956 INFO:tasks.rados.rados.0.smithi090.stdout:1307: done (3 left) 2023-01-15T14:33:38.956 INFO:tasks.rados.rados.0.smithi090.stdout:1308: write oid 38 current snap is 137 2023-01-15T14:33:38.956 INFO:tasks.rados.rados.0.smithi090.stdout:1308: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2023-01-15T14:33:38.956 INFO:tasks.rados.rados.0.smithi090.stdout:1308: writing smithi09025264-38 from 783325 to 1340486 tid 1 2023-01-15T14:33:38.956 INFO:tasks.rados.rados.0.smithi090.stdout:1308: writing smithi09025264-38 from 1779535 to 2465686 tid 2 2023-01-15T14:33:38.956 INFO:tasks.rados.rados.0.smithi090.stdout:1308: writing smithi09025264-38 from 3244745 to 3699657 tid 3 2023-01-15T14:33:38.956 INFO:tasks.rados.rados.0.smithi090.stdout:1309: copy_from oid 5 from oid 30 current snap is 137 2023-01-15T14:33:38.957 INFO:tasks.rados.rados.0.smithi090.stdout:1306: finishing write tid 1 to smithi09025264-45 2023-01-15T14:33:38.957 INFO:tasks.rados.rados.0.smithi090.stdout:1305: finishing rollback tid 0 to smithi09025264-6 2023-01-15T14:33:38.957 INFO:tasks.rados.rados.0.smithi090.stdout:1305: finishing rollback tid 1 to smithi09025264-6 2023-01-15T14:33:38.957 INFO:tasks.rados.rados.0.smithi090.stdout:1305: finishing rollback tid 2 to smithi09025264-6 2023-01-15T14:33:38.957 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 305 (ObjNum 412 snap 112 seq_num 412) dirty exists 2023-01-15T14:33:38.957 INFO:tasks.rados.rados.0.smithi090.stdout:1306: finishing write tid 2 to smithi09025264-45 2023-01-15T14:33:38.957 INFO:tasks.rados.rados.0.smithi090.stdout:1306: finishing write tid 3 to smithi09025264-45 2023-01-15T14:33:38.957 INFO:tasks.rados.rados.0.smithi090.stdout:1306: finishing write tid 4 to smithi09025264-45 2023-01-15T14:33:38.957 INFO:tasks.rados.rados.0.smithi090.stdout:1299: finishing write tid 1 to smithi09025264-44 2023-01-15T14:33:38.958 INFO:tasks.rados.rados.0.smithi090.stdout:1306: finishing write tid 5 to smithi09025264-45 2023-01-15T14:33:38.958 INFO:tasks.rados.rados.0.smithi090.stdout:1306: finishing write tid 6 to smithi09025264-45 2023-01-15T14:33:38.958 INFO:tasks.rados.rados.0.smithi090.stdout:1306: oid 45 updating version 0 to 146 2023-01-15T14:33:38.958 INFO:tasks.rados.rados.0.smithi090.stdout:1306: oid 45 updating version 146 to 148 2023-01-15T14:33:38.958 INFO:tasks.rados.rados.0.smithi090.stdout:1306: oid 45 version 148 is already newer than 145 2023-01-15T14:33:38.958 INFO:tasks.rados.rados.0.smithi090.stdout:1306: oid 45 version 148 is already newer than 147 2023-01-15T14:33:38.958 INFO:tasks.rados.rados.0.smithi090.stdout:1306: oid 45 updating version 148 to 149 2023-01-15T14:33:38.958 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 149 (ObjNum 474 snap 137 seq_num 474) dirty exists 2023-01-15T14:33:38.958 INFO:tasks.rados.rados.0.smithi090.stdout:1306: left oid 45 (ObjNum 474 snap 137 seq_num 474) 2023-01-15T14:33:38.958 INFO:tasks.rados.rados.0.smithi090.stdout:1299: finishing write tid 2 to smithi09025264-44 2023-01-15T14:33:38.959 INFO:tasks.rados.rados.0.smithi090.stdout:1299: finishing write tid 3 to smithi09025264-44 2023-01-15T14:33:38.959 INFO:tasks.rados.rados.0.smithi090.stdout:1299: finishing write tid 4 to smithi09025264-44 2023-01-15T14:33:38.959 INFO:tasks.rados.rados.0.smithi090.stdout:1299: finishing write tid 5 to smithi09025264-44 2023-01-15T14:33:38.960 INFO:tasks.rados.rados.0.smithi090.stdout:1299: oid 44 updating version 0 to 347 2023-01-15T14:33:38.960 INFO:tasks.rados.rados.0.smithi090.stdout:1299: oid 44 version 347 is already newer than 345 2023-01-15T14:33:38.960 INFO:tasks.rados.rados.0.smithi090.stdout:1299: oid 44 updating version 347 to 348 2023-01-15T14:33:38.960 INFO:tasks.rados.rados.0.smithi090.stdout:1299: oid 44 version 348 is already newer than 346 2023-01-15T14:33:38.961 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 348 (ObjNum 473 snap 137 seq_num 473) dirty exists 2023-01-15T14:33:38.961 INFO:tasks.rados.rados.0.smithi090.stdout:1299: left oid 44 (ObjNum 473 snap 137 seq_num 473) 2023-01-15T14:33:38.961 INFO:tasks.rados.rados.0.smithi090.stdout:1299: done (4 left) 2023-01-15T14:33:38.961 INFO:tasks.rados.rados.0.smithi090.stdout:1305: done (3 left) 2023-01-15T14:33:38.961 INFO:tasks.rados.rados.0.smithi090.stdout:1306: done (2 left) 2023-01-15T14:33:38.961 INFO:tasks.rados.rados.0.smithi090.stdout:1310: read oid 12 snap -1 2023-01-15T14:33:38.961 INFO:tasks.rados.rados.0.smithi090.stdout:1310: expect (ObjNum 326 snap 87 seq_num 326) 2023-01-15T14:33:38.961 INFO:tasks.rados.rados.0.smithi090.stdout:1311: copy_from oid 41 from oid 6 current snap is 137 2023-01-15T14:33:38.961 INFO:tasks.rados.rados.0.smithi090.stdout:1312: snap_create 2023-01-15T14:33:38.961 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:33:38.962 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:38.951+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:39.053 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:39.051+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:39.069 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:39.069 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:39.079 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:33:39.131 INFO:tasks.rados.rados.0.smithi090.stdout:1310: expect (ObjNum 326 snap 87 seq_num 326) 2023-01-15T14:33:39.154 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:39.151+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:39.189 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:39.493 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:39.495 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:33:39.496 INFO:tasks.rados.rados.0.smithi090.stdout:1310: done (3 left) 2023-01-15T14:33:39.497 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:39.497 INFO:tasks.rados.rados.0.smithi090.stdout:1308: finishing write tid 1 to smithi09025264-38 2023-01-15T14:33:39.497 INFO:tasks.rados.rados.0.smithi090.stdout:1309: finishing copy_from to smithi09025264-5 2023-01-15T14:33:39.497 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 232 (ObjNum 449 snap 125 seq_num 449) dirty exists 2023-01-15T14:33:39.497 INFO:tasks.rados.rados.0.smithi090.stdout:1308: finishing write tid 2 to smithi09025264-38 2023-01-15T14:33:39.497 INFO:tasks.rados.rados.0.smithi090.stdout:1308: finishing write tid 3 to smithi09025264-38 2023-01-15T14:33:39.497 INFO:tasks.rados.rados.0.smithi090.stdout:1309: finishing copy_from racing read to smithi09025264-5 2023-01-15T14:33:39.497 INFO:tasks.rados.rados.0.smithi090.stdout:1309: done (2 left) 2023-01-15T14:33:39.497 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:39.498 INFO:tasks.rados.rados.0.smithi090.stdout:1308: finishing write tid 4 to smithi09025264-38 2023-01-15T14:33:39.498 INFO:tasks.rados.rados.0.smithi090.stdout:1308: finishing write tid 5 to smithi09025264-38 2023-01-15T14:33:39.498 INFO:tasks.rados.rados.0.smithi090.stdout:1308: oid 38 updating version 0 to 226 2023-01-15T14:33:39.498 INFO:tasks.rados.rados.0.smithi090.stdout:1308: oid 38 version 226 is already newer than 225 2023-01-15T14:33:39.498 INFO:tasks.rados.rados.0.smithi090.stdout:1308: oid 38 updating version 226 to 228 2023-01-15T14:33:39.498 INFO:tasks.rados.rados.0.smithi090.stdout:1308: oid 38 version 228 is already newer than 227 2023-01-15T14:33:39.498 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 228 (ObjNum 475 snap 137 seq_num 475) dirty exists 2023-01-15T14:33:39.498 INFO:tasks.rados.rados.0.smithi090.stdout:1308: left oid 38 (ObjNum 475 snap 137 seq_num 475) 2023-01-15T14:33:39.498 INFO:tasks.rados.rados.0.smithi090.stdout:1308: done (1 left) 2023-01-15T14:33:39.498 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:39.499 INFO:tasks.rados.rados.0.smithi090.stdout:1311: finishing copy_from to smithi09025264-41 2023-01-15T14:33:39.499 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 305 (ObjNum 412 snap 112 seq_num 412) dirty exists 2023-01-15T14:33:39.499 INFO:tasks.rados.rados.0.smithi090.stdout:1311: finishing copy_from racing read to smithi09025264-41 2023-01-15T14:33:39.499 INFO:tasks.rados.rados.0.smithi090.stdout:1311: done (0 left) 2023-01-15T14:33:39.499 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:39.491+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:39.596 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:39.592+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:39.697 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:39.697 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:39.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:33:39.698 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:39.692+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:39.798 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:39.795+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:39.903 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:39.903 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:39.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:33:39.904 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:39.899+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:39.926 INFO:tasks.rados.rados.0.smithi090.stdout:1312: done (0 left) 2023-01-15T14:33:39.926 INFO:tasks.rados.rados.0.smithi090.stdout:1313: snap_remove snap 137 2023-01-15T14:33:39.931 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 106 2023-01-15T14:33:40.004 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:40.000+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:40.012 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:40.012 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:40.022 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:33:40.051 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:33:40.107 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:33:40.109 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:40.107+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:40.120 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:40.120 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:40.133 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:33:40.209 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:40.207+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:40.240 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:40.268 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:40.286 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:40.288 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:40.288 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:33:40.289 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:33:40.289 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:40.299 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:33:40.305 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:40.309 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:40.304+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:40.339 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:40.372 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:40.404 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:40.675 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:40.675 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:40.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:40.675 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:40.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.675 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 185.53561956600001, 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:40.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:40.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:33:40.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:40.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:40.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:33:40.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:33:40.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:33:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:33:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:33:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:33:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:33:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:33:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:33:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:33:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:33:40.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:33:40.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:40.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:33:40.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:33:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:33:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:33:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:33:40.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:33:40.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:40.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:33:40.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:33:40.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:40.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:33:40.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:33:40.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:40.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:33:40.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:33:40.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:33:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:33:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:33:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:33:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 185.53556862299999, 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.687 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:40.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:40.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:40.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:40.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:40.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:40.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:33:40.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:40.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:40.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:33:40.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:33:40.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:40.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:33:40.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:33:40.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:33:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:33:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:33:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:33:40.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:33:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:33:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:33:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:33:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:40.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:33:40.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:33:40.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:33:40.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:33:40.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.696 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:33:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 182.59466938200001, 2023-01-15T14:33:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:33:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:40.697 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:33:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:40.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:40.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:40.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:33:40.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:40.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:40.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:33:40.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:33:40.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:40.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:33:40.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:33:40.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:40.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:33:40.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:33:40.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:33:40.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:33:40.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:33:40.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:33:40.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:33:40.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:33:40.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:33:40.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:40.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:33:40.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:33:40.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:40.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:33:40.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:33:40.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:40.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:33:40.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:33:40.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:40.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:33:40.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:33:40.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:40.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:33:40.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:40.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:33:40.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:33:40.708 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:40.708 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:40.708 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:40.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.708 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:33:40.708 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 180.520895209, 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:40.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:40.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:40.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:40.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:40.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:33:40.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:40.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:33:40.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:33:40.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:33:40.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:33:40.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:40.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:33:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:33:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:33:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:33:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:33:40.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:40.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:33:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:33:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:33:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:33:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:33:40.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 156.62693978900001, 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:40.718 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:33:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:33:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:33:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:33:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:33:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:33:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:33:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:33:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:33:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:33:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:33:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:33:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:33:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:33:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:33:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:33:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:33:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:33:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.74353310800001, 2023-01-15T14:33:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:33:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:33:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:33:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:33:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:33:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:33:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:33:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:33:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:33:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:33:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:33:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:33:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:33:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:33:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:33:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:33:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:33:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:33:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:33:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:33:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:33:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:33:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:33:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:33:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:33:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:33:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:33:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:41.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.011 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.74012598499999, 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:41.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:41.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:41.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:41.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:41.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:33:41.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:33:41.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:41.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:33:41.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:33:41.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:33:41.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:33:41.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:41.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:33:41.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:33:41.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:33:41.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:33:41.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:41.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:33:41.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:33:41.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:41.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:33:41.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:33:41.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:41.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:33:41.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:33:41.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:41.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:33:41.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:33:41.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:41.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:33:41.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:33:41.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:33:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:33:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:33:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:33:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:33:41.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:33:41.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:41.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:33:41.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:33:41.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:41.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:33:41.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:33:41.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.212 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:41.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.212 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:41.212 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:41.212 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.73991038, 2023-01-15T14:33:41.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:33:41.213 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:41.213 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:41.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:41.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:41.214 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:41.214 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:33:41.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.214 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:41.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:41.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:41.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:41.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:41.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:41.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:33:41.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:41.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:41.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:33:41.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:33:41.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:41.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:33:41.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:33:41.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:41.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:33:41.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:33:41.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:33:41.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:33:41.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:41.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:33:41.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:33:41.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:33:41.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:33:41.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:41.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:33:41.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:33:41.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:41.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:33:41.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:33:41.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:41.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:33:41.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:33:41.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:41.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:33:41.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:33:41.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:41.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:33:41.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:41.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.739803306, 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:41.225 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:33:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:41.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:33:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:41.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:33:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:33:41.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:33:41.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:33:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:33:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:33:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:33:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:33:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:33:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:33:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:33:41.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:33:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:33:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:33:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:33:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.422 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 126.609564256, 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:41.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:33:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:41.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:33:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:33:41.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:33:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:33:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:33:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:33:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:33:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:33:41.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:33:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:33:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:33:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:33:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:33:41.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:33:41.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:33:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:33:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:33:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:41.432 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 126.609459719, 2023-01-15T14:33:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:33:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:41.433 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:41.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:41.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:41.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:41.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:33:41.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.579 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:41.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:41.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:33:41.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:41.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:33:41.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:33:41.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:33:41.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:33:41.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:41.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:33:41.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:33:41.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:33:41.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:33:41.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:41.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:33:41.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:33:41.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:41.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:33:41.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:33:41.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:41.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:33:41.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:33:41.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:41.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:33:41.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:33:41.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:41.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:33:41.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:33:41.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:41.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:33:41.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:33:41.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:33:41.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:33:41.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:33:41.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:33:41.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:41.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:33:41.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:33:41.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:41.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:33:41.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:33:41.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.597 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:41.597 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.597 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:41.598 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:41.598 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 126.609389788, 2023-01-15T14:33:41.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:33:41.599 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:41.599 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:41.599 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:41.599 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:41.600 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:41.600 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:33:41.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.600 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:41.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:41.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:41.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:41.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:41.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:41.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:33:41.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:41.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:41.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:33:41.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:33:41.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:41.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:33:41.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:33:41.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:41.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:33:41.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:33:41.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:33:41.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:33:41.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:41.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:33:41.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:33:41.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:33:41.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:33:41.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:41.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:33:41.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:33:41.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:41.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:33:41.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:33:41.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:41.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:33:41.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:41.743 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:33:41.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:33:41.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:41.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:33:41.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:33:41.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:41.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:33:41.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:33:41.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.745 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:41.745 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.745 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2780 3.38 e265/232 3:1eaad226:::smithi09025264-49:head v 265'279, mlcod=261'274)", 2023-01-15T14:33:41.746 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:41.746 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.438218661999997, 2023-01-15T14:33:41.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32005591799999999, 2023-01-15T14:33:41.746 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:41.746 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:41.746 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:41.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:41.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:41.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:41.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:41.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.747 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:41.747 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991860+0000", 2023-01-15T14:33:41.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:41.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:41.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993927+0000", 2023-01-15T14:33:41.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020673229999999998 2023-01-15T14:33:41.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:41.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993946+0000", 2023-01-15T14:33:41.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9429999999999999e-05 2023-01-15T14:33:41.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:41.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.094039+0000", 2023-01-15T14:33:41.749 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092898 2023-01-15T14:33:41.749 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.749 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285071+0000", 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.191031904 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285106+0000", 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5227e-05 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286323+0000", 2023-01-15T14:33:41.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012163829999999999 2023-01-15T14:33:41.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:41.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286335+0000", 2023-01-15T14:33:41.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2224e-05 2023-01-15T14:33:41.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:41.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287218+0000", 2023-01-15T14:33:41.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088265000000000001 2023-01-15T14:33:41.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:41.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287250+0000", 2023-01-15T14:33:41.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2209000000000003e-05 2023-01-15T14:33:41.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:41.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.311917+0000", 2023-01-15T14:33:41.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024667062999999999 2023-01-15T14:33:41.752 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.752 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:41.753 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.753 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2868 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'278, mlcod=264'275)", 2023-01-15T14:33:41.753 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:41.753 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.436327720999998, 2023-01-15T14:33:41.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31268252400000002, 2023-01-15T14:33:41.753 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:41.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:41.753 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:41.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:41.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993751+0000", 2023-01-15T14:33:41.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:41.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:41.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993762+0000", 2023-01-15T14:33:41.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0822000000000001e-05 2023-01-15T14:33:41.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:41.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993777+0000", 2023-01-15T14:33:41.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5540999999999999e-05 2023-01-15T14:33:41.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:41.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993787+0000", 2023-01-15T14:33:41.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0173999999999999e-05 2023-01-15T14:33:41.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290198+0000", 2023-01-15T14:33:41.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29641022099999997 2023-01-15T14:33:41.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:41.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290212+0000", 2023-01-15T14:33:41.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4214e-05 2023-01-15T14:33:41.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:41.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290300+0000", 2023-01-15T14:33:41.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8300000000000005e-05 2023-01-15T14:33:41.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290312+0000", 2023-01-15T14:33:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2344000000000001e-05 2023-01-15T14:33:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290357+0000", 2023-01-15T14:33:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4367999999999999e-05 2023-01-15T14:33:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:41.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290389+0000", 2023-01-15T14:33:41.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1983000000000002e-05 2023-01-15T14:33:41.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:41.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.306434+0000", 2023-01-15T14:33:41.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016045667 2023-01-15T14:33:41.908 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.909 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:41.909 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.909 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:33:41.909 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:41.910 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.436278295999998, 2023-01-15T14:33:41.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:33:41.910 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:41.910 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:41.910 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:41.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:41.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:41.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:41.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:41.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:41.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:41.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:41.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:41.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:33:41.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:33:41.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:41.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:33:41.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:33:41.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:41.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:33:41.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:33:41.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:41.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:33:41.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:33:41.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:41.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:33:41.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:33:41.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:41.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:33:41.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:33:41.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:41.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:33:41.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:33:41.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:41.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:33:41.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:33:41.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:41.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:33:41.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:33:41.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:41.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:33:41.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:33:41.917 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.917 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:41.917 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:41.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.917 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:33:41.918 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:41.918 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 59.432466609000002, 2023-01-15T14:33:41.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:33:41.918 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:41.918 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:41.918 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:41.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:41.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:41.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:41.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:41.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:41.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:41.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:33:41.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:41.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:41.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:41.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:33:41.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:33:41.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:41.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:42.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:33:42.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:33:42.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:42.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:33:42.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:33:42.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:42.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:33:42.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:33:42.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:42.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:33:42.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:33:42.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:42.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:33:42.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:33:42.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:42.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:33:42.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:33:42.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:42.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:33:42.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:33:42.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:42.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:33:42.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:33:42.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:42.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:33:42.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:33:42.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.079 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:42.079 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.079 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3120 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'290, mlcod=0'0)", 2023-01-15T14:33:42.079 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:33:42.080 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.190653333, 2023-01-15T14:33:42.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39853714000000001, 2023-01-15T14:33:42.080 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:42.080 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:42.080 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:42.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:42.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:33:42.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:33:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239424+0000", 2023-01-15T14:33:42.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239747+0000", 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032243300000000001 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239784+0000", 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7051000000000002e-05 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239802+0000", 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8250999999999999e-05 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536033+0000", 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29623121499999999 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:42.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536081+0000", 2023-01-15T14:33:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2023-01-15T14:33:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536800+0000", 2023-01-15T14:33:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071948499999999996 2023-01-15T14:33:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536817+0000", 2023-01-15T14:33:42.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6228000000000001e-05 2023-01-15T14:33:42.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537048+0000", 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023136200000000001 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637902+0000", 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10085403900000001 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637963+0000", 2023-01-15T14:33:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1310999999999996e-05 2023-01-15T14:33:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3121 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'291, mlcod=0'0)", 2023-01-15T14:33:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:33:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.188657816999999, 2023-01-15T14:33:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659573799999998, 2023-01-15T14:33:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:33:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241420+0000", 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241854+0000", 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043330099999999998 2023-01-15T14:33:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241878+0000", 2023-01-15T14:33:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4559e-05 2023-01-15T14:33:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241892+0000", 2023-01-15T14:33:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039999999999999e-05 2023-01-15T14:33:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:42.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536863+0000", 2023-01-15T14:33:42.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29497073400000001 2023-01-15T14:33:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536887+0000", 2023-01-15T14:33:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3683999999999999e-05 2023-01-15T14:33:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:42.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537124+0000", 2023-01-15T14:33:42.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023729900000000001 2023-01-15T14:33:42.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:42.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537136+0000", 2023-01-15T14:33:42.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2278e-05 2023-01-15T14:33:42.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:42.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537345+0000", 2023-01-15T14:33:42.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020865200000000001 2023-01-15T14:33:42.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:42.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637981+0000", 2023-01-15T14:33:42.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063575700000001 2023-01-15T14:33:42.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:42.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638018+0000", 2023-01-15T14:33:42.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6826999999999998e-05 2023-01-15T14:33:42.240 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.240 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:42.240 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.240 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3122 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'292, mlcod=0'0)", 2023-01-15T14:33:42.240 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:33:42.240 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.186761977, 2023-01-15T14:33:42.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39623551600000001, 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:33:42.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:42.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243316+0000", 2023-01-15T14:33:42.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:42.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:42.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243833+0000", 2023-01-15T14:33:42.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051703600000000001 2023-01-15T14:33:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243863+0000", 2023-01-15T14:33:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-01-15T14:33:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243879+0000", 2023-01-15T14:33:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6263999999999999e-05 2023-01-15T14:33:42.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:42.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637317+0000", 2023-01-15T14:33:42.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39343783199999999 2023-01-15T14:33:42.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:42.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637356+0000", 2023-01-15T14:33:42.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9579000000000002e-05 2023-01-15T14:33:42.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637830+0000", 2023-01-15T14:33:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047326399999999999 2023-01-15T14:33:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637885+0000", 2023-01-15T14:33:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5074999999999997e-05 2023-01-15T14:33:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638185+0000", 2023-01-15T14:33:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029993799999999997 2023-01-15T14:33:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638337+0000", 2023-01-15T14:33:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015208299999999999 2023-01-15T14:33:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.639553+0000", 2023-01-15T14:33:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001216389 2023-01-15T14:33:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3123 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'293, mlcod=0'0)", 2023-01-15T14:33:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:33:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.185017948999999, 2023-01-15T14:33:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.395733102, 2023-01-15T14:33:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:33:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:33:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245060+0000", 2023-01-15T14:33:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245075+0000", 2023-01-15T14:33:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4459e-05 2023-01-15T14:33:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245107+0000", 2023-01-15T14:33:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1795000000000003e-05 2023-01-15T14:33:42.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:42.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245121+0000", 2023-01-15T14:33:42.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4117000000000001e-05 2023-01-15T14:33:42.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638063+0000", 2023-01-15T14:33:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39294204300000002 2023-01-15T14:33:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638097+0000", 2023-01-15T14:33:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4385999999999999e-05 2023-01-15T14:33:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:42.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638299+0000", 2023-01-15T14:33:42.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020133099999999999 2023-01-15T14:33:42.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:42.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638318+0000", 2023-01-15T14:33:42.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514999999999999e-05 2023-01-15T14:33:42.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638386+0000", 2023-01-15T14:33:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8089000000000005e-05 2023-01-15T14:33:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:42.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638430+0000", 2023-01-15T14:33:42.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.439e-05 2023-01-15T14:33:42.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:42.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.640795+0000", 2023-01-15T14:33:42.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002364274 2023-01-15T14:33:42.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.555 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:42.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.555 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:42.555 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:42.556 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:42.556 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:33:42.557 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:33:42.557 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:33:42.557 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:33:42.557 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:33:42.557 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:33:42.557 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:33:42.558 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:33:42.558 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:33:42.559 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:40.672+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:42.560 INFO:tasks.rados.rados.0.smithi090.stdout:1313: done (0 left) 2023-01-15T14:33:42.560 INFO:tasks.rados.rados.0.smithi090.stdout:1314: snap_remove snap 129 2023-01-15T14:33:42.561 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:33:42.562 INFO:tasks.rados.rados.0.smithi090.stdout:1314: done (0 left) 2023-01-15T14:33:42.562 INFO:tasks.rados.rados.0.smithi090.stdout:1315: write oid 1 current snap is 138 2023-01-15T14:33:42.562 INFO:tasks.rados.rados.0.smithi090.stdout:1315: seq_num 476 ranges {400132=632410,1438017=546162} 2023-01-15T14:33:42.563 INFO:tasks.rados.rados.0.smithi090.stdout:1315: writing smithi09025264-1 from 400132 to 1032542 tid 1 2023-01-15T14:33:42.563 INFO:tasks.rados.rados.0.smithi090.stdout:1315: writing smithi09025264-1 from 1438017 to 1984179 tid 2 2023-01-15T14:33:42.563 INFO:tasks.rados.rados.0.smithi090.stdout:1316: snap_create 2023-01-15T14:33:42.563 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:42.563 INFO:tasks.rados.rados.0.smithi090.stdout:1315: finishing write tid 1 to smithi09025264-1 2023-01-15T14:33:42.563 INFO:tasks.rados.rados.0.smithi090.stdout:1315: finishing write tid 2 to smithi09025264-1 2023-01-15T14:33:42.563 INFO:tasks.rados.rados.0.smithi090.stdout:1315: finishing write tid 3 to smithi09025264-1 2023-01-15T14:33:42.563 INFO:tasks.rados.rados.0.smithi090.stdout:1315: finishing write tid 4 to smithi09025264-1 2023-01-15T14:33:42.564 INFO:tasks.rados.rados.0.smithi090.stdout:1315: oid 1 updating version 0 to 374 2023-01-15T14:33:42.564 INFO:tasks.rados.rados.0.smithi090.stdout:1315: oid 1 updating version 374 to 376 2023-01-15T14:33:42.564 INFO:tasks.rados.rados.0.smithi090.stdout:1315: oid 1 version 376 is already newer than 375 2023-01-15T14:33:42.564 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 376 (ObjNum 476 snap 138 seq_num 476) dirty exists 2023-01-15T14:33:42.564 INFO:tasks.rados.rados.0.smithi090.stdout:1315: left oid 1 (ObjNum 476 snap 138 seq_num 476) 2023-01-15T14:33:42.564 INFO:tasks.rados.rados.0.smithi090.stdout:1315: done (0 left) 2023-01-15T14:33:42.603 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:42.600+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:42.711 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:42.707+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:42.719 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:42.720 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:42.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.720 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.3.0:854 3.28 e333/198)", 2023-01-15T14:33:42.721 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:42.638082+0000", 2023-01-15T14:33:42.721 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.079302691999999994, 2023-01-15T14:33:42.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.079320963999999994, 2023-01-15T14:33:42.721 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:42.721 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:42.721 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:42.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:42.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638082+0000", 2023-01-15T14:33:42.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:42.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638082+0000", 2023-01-15T14:33:42.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:42.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638080+0000", 2023-01-15T14:33:42.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:42.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:42.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638094+0000", 2023-01-15T14:33:42.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:33:42.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:42.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638119+0000", 2023-01-15T14:33:42.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4796000000000001e-05 2023-01-15T14:33:42.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:42.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638136+0000", 2023-01-15T14:33:42.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6857000000000001e-05 2023-01-15T14:33:42.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.725 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:42.725 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.725 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.3.0:854 3.28 e333/198)", 2023-01-15T14:33:42.725 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:42.638871+0000", 2023-01-15T14:33:42.725 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.078513587999999995, 2023-01-15T14:33:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.078560546999999994, 2023-01-15T14:33:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638871+0000", 2023-01-15T14:33:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638871+0000", 2023-01-15T14:33:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638869+0000", 2023-01-15T14:33:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638883+0000", 2023-01-15T14:33:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4363e-05 2023-01-15T14:33:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:42.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638910+0000", 2023-01-15T14:33:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6553000000000001e-05 2023-01-15T14:33:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.638946+0000", 2023-01-15T14:33:42.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6634999999999997e-05 2023-01-15T14:33:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.7.0:749 3.4a e333/263)", 2023-01-15T14:33:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:42.639038+0000", 2023-01-15T14:33:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.078346554999999998, 2023-01-15T14:33:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.078417477999999999, 2023-01-15T14:33:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:42.874 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.639038+0000", 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.639038+0000", 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.639037+0000", 2023-01-15T14:33:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.639047+0000", 2023-01-15T14:33:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0477e-05 2023-01-15T14:33:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.639067+0000", 2023-01-15T14:33:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0003000000000001e-05 2023-01-15T14:33:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:42.639080+0000", 2023-01-15T14:33:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2986000000000001e-05 2023-01-15T14:33:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:42.878 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 3 2023-01-15T14:33:42.878 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:42.880 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:33:42.880 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:42.876+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:42.944 INFO:tasks.rados.rados.0.smithi090.stdout:1316: done (0 left) 2023-01-15T14:33:43.036 INFO:tasks.rados.rados.0.smithi090.stdout:1317: read oid 7 snap -1 2023-01-15T14:33:43.036 INFO:tasks.rados.rados.0.smithi090.stdout:1317: expect deleted 2023-01-15T14:33:43.037 INFO:tasks.rados.rados.0.smithi090.stdout:1318: read oid 14 snap -1 2023-01-15T14:33:43.037 INFO:tasks.rados.rados.0.smithi090.stdout:1318: expect (ObjNum 404 snap 108 seq_num 404) 2023-01-15T14:33:43.037 INFO:tasks.rados.rados.0.smithi090.stdout:1319: read oid 3 snap -1 2023-01-15T14:33:43.037 INFO:tasks.rados.rados.0.smithi090.stdout:1319: expect (ObjNum 464 snap 133 seq_num 464) 2023-01-15T14:33:43.038 INFO:tasks.rados.rados.0.smithi090.stdout:1320: rollback oid 8 current snap is 139 2023-01-15T14:33:43.038 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 8 to 119 2023-01-15T14:33:43.038 INFO:tasks.rados.rados.0.smithi090.stdout:1321: snap_create 2023-01-15T14:33:43.038 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:33:43.038 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:43.038 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:43.038 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:33:43.038 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:33:43.039 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:43.039 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:43.032+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:43.049 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:33:43.050 INFO:tasks.rados.rados.0.smithi090.stdout:1320: finishing rollback tid 0 to smithi09025264-8 2023-01-15T14:33:43.200 INFO:tasks.rados.rados.0.smithi090.stdout:1320: finishing rollback tid 1 to smithi09025264-8 2023-01-15T14:33:43.200 INFO:tasks.rados.rados.0.smithi090.stdout:1320: finishing rollback tid 2 to smithi09025264-8 2023-01-15T14:33:43.200 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 304 (ObjNum 404 snap 108 seq_num 404) dirty exists 2023-01-15T14:33:43.200 INFO:tasks.rados.rados.0.smithi090.stdout:1320: done (3 left) 2023-01-15T14:33:43.201 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:43.201 INFO:tasks.rados.rados.0.smithi090.stdout:1319: expect (ObjNum 464 snap 133 seq_num 464) 2023-01-15T14:33:43.201 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:43.196+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:43.274 INFO:tasks.rados.rados.0.smithi090.stdout:1319: done (2 left) 2023-01-15T14:33:43.359 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:43.359 INFO:tasks.rados.rados.0.smithi090.stdout:1318: expect (ObjNum 404 snap 108 seq_num 404) 2023-01-15T14:33:43.361 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:33:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.17875645100003, 2023-01-15T14:33:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:33:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:33:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:33:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:33:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:33:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:33:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:33:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:33:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:33:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:33:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:33:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:33:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:33:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:33:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:33:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:33:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:33:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:33:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:33:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:33:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:33:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:33:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:33:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:33:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:33:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:43.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:33:43.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:33:43.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:43.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:33:43.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:33:43.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:43.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:33:43.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:33:43.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:33:43.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:33:43.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.376 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:33:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.17835311499999, 2023-01-15T14:33:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:33:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:33:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:33:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:33:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:33:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:33:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:33:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:33:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:33:43.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:33:43.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:43.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:33:43.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:33:43.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:33:43.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:33:43.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:33:43.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:33:43.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:43.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:33:43.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:33:43.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:43.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:33:43.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:33:43.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:43.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:33:43.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:33:43.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:43.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:33:43.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:33:43.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:33:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:33:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:33:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:33:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:33:43.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:33:43.523 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.523 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.523 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.523 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:43.523 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:33:43.524 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 316.07812949499998, 2023-01-15T14:33:43.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:33:43.524 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.524 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:43.524 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.524 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.524 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.524 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:33:43.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.524 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:33:43.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:33:43.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:33:43.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:43.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:33:43.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:33:43.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:33:43.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:33:43.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:33:43.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:33:43.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:33:43.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:33:43.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:43.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:33:43.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:33:43.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:33:43.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:33:43.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:43.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:33:43.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:33:43.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:33:43.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:33:43.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:43.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:33:43.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:33:43.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:33:43.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:33:43.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.532 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.532 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:43.532 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:33:43.532 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 225.38481426999999, 2023-01-15T14:33:43.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:33:43.533 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.533 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:33:43.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.533 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.533 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:33:43.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.533 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:33:43.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:33:43.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:33:43.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:33:43.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:33:43.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:33:43.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:33:43.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:33:43.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:43.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:33:43.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:33:43.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:33:43.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:33:43.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 225.37835291600001, 2023-01-15T14:33:43.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:33:43.539 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.539 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:33:43.539 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.540 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.541 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.541 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:33:43.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.541 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:33:43.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:33:43.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:33:43.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:43.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:33:43.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:33:43.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:33:43.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:33:43.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:33:43.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:33:43.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:33:43.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:33:43.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:43.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:33:43.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:33:43.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:33:43.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:33:43.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:33:43.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:33:43.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.680 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.680 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.681 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:43.681 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:33:43.681 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 225.27793099100001, 2023-01-15T14:33:43.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:33:43.681 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.682 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:33:43.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.682 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.682 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:33:43.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.683 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.683 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.683 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:33:43.684 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:33:43.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:33:43.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:43.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:33:43.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:33:43.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:33:43.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:33:43.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:33:43.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:33:43.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:33:43.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:33:43.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:33:43.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:33:43.690 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.690 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.690 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.690 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:43.690 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:33:43.690 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 225.277593505, 2023-01-15T14:33:43.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:33:43.690 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:33:43.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:33:43.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:43.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:33:43.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:33:43.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:33:43.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:33:43.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:43.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:33:43.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:33:43.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:33:43.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:33:43.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:33:43.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:33:43.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.697 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.697 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.697 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:43.697 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:33:43.697 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 225.27753760900001, 2023-01-15T14:33:43.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:33:43.698 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.698 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:33:43.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.698 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.698 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:33:43.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.698 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:33:43.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:43.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:33:43.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:33:43.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:33:43.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:33:43.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:33:43.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:33:43.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:33:43.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:33:43.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:43.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:33:43.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:33:43.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.703 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.704 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:33:43.704 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:33:43.704 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 183.444287191, 2023-01-15T14:33:43.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:33:43.704 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.704 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:43.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.704 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.705 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:33:43.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.705 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:33:43.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.705 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:33:43.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:33:43.706 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:43.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:33:43.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:33:43.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:33:43.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:33:43.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:33:43.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:33:43.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:33:43.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:33:43.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:43.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:33:43.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:33:43.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:33:43.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:43.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:33:43.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:33:43.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:33:43.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:33:43.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.712 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.712 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.712 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2207 3.5 3:a1c47f99:::smithi09025264-2:head [write 2548663~763022] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:43.712 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:33:43.712 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 124.469529463, 2023-01-15T14:33:43.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79657873300000004, 2023-01-15T14:33:43.713 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.713 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:43.713 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.713 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.713 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.713 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2207 2023-01-15T14:33:43.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.713 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837590+0000", 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837960+0000", 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037055900000000001 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837980+0000", 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9953e-05 2023-01-15T14:33:43.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837988+0000", 2023-01-15T14:33:43.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2800000000000003e-06 2023-01-15T14:33:43.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927512+0000", 2023-01-15T14:33:43.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089523563 2023-01-15T14:33:43.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927645+0000", 2023-01-15T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013285799999999999 2023-01-15T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225466+0000", 2023-01-15T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29782133100000002 2023-01-15T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:43.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225553+0000", 2023-01-15T14:33:43.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6866000000000005e-05 2023-01-15T14:33:43.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:33:43.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225708+0000", 2023-01-15T14:33:43.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015455199999999999 2023-01-15T14:33:43.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:43.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225971+0000", 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026317099999999998 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226027+0000", 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6601000000000002e-05 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226470+0000", 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044267500000000001 2023-01-15T14:33:43.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:43.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227372+0000", 2023-01-15T14:33:43.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090215200000000003 2023-01-15T14:33:43.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:43.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429314+0000", 2023-01-15T14:33:43.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20194183499999999 2023-01-15T14:33:43.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:43.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.633999+0000", 2023-01-15T14:33:43.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20468505400000001 2023-01-15T14:33:43.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:43.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634075+0000", 2023-01-15T14:33:43.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:33:43.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634170+0000", 2023-01-15T14:33:43.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4821999999999994e-05 2023-01-15T14:33:43.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.723 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.723 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.724 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2208 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (56) in=7b,truncate 3395047] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:43.724 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:33:43.724 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 124.46910687, 2023-01-15T14:33:43.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79650439900000003, 2023-01-15T14:33:43.724 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.725 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:43.725 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.725 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.725 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.725 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2208 2023-01-15T14:33:43.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.725 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:33:43.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.726 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.726 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:33:43.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:33:43.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:43.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838018+0000", 2023-01-15T14:33:43.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2549999999999997e-06 2023-01-15T14:33:43.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838031+0000", 2023-01-15T14:33:43.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2717999999999999e-05 2023-01-15T14:33:43.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838036+0000", 2023-01-15T14:33:43.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2240000000000003e-06 2023-01-15T14:33:43.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927676+0000", 2023-01-15T14:33:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089639620000000003 2023-01-15T14:33:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927804+0000", 2023-01-15T14:33:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012832300000000001 2023-01-15T14:33:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226096+0000", 2023-01-15T14:33:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29829207400000002 2023-01-15T14:33:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226347+0000", 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025084800000000002 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226750+0000", 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040343200000000001 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227147+0000", 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039682900000000002 2023-01-15T14:33:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227600+0000", 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045321199999999998 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227859+0000", 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259041 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227895+0000", 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5710999999999997e-05 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429405+0000", 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201509415 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634300+0000", 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20489528200000001 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634409+0000", 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109524 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634518+0000", 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108548 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2209 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 124.469078055, 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79724995499999995, 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2209 2023-01-15T14:33:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:33:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:33:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:33:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838044+0000", 2023-01-15T14:33:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6120000000000001e-06 2023-01-15T14:33:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838056+0000", 2023-01-15T14:33:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1236000000000001e-05 2023-01-15T14:33:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838059+0000", 2023-01-15T14:33:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6940000000000001e-06 2023-01-15T14:33:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.027908+0000", 2023-01-15T14:33:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189848287 2023-01-15T14:33:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.028049+0000", 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014132500000000001 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227252+0000", 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19920324 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227341+0000", 2023-01-15T14:33:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8683000000000005e-05 2023-01-15T14:33:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634637+0000", 2023-01-15T14:33:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729567300000002 2023-01-15T14:33:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634980+0000", 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034335000000000002 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.635292+0000", 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003123 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2222 3.c 3:3137f0cb:::smithi09025264-20:head [write 3259733~152121] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 124.44922649599999, 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77651572400000002, 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2222 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:33:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:33:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857891+0000", 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857993+0000", 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010276399999999999 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858021+0000", 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8022e-05 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858034+0000", 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2549e-05 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025371+0000", 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16733653100000001 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025429+0000", 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8922000000000003e-05 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226869+0000", 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20143931100000001 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:43.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227088+0000", 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219328 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227443+0000", 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035481 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227830+0000", 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000386702 2023-01-15T14:33:43.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227861+0000", 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1584000000000001e-05 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227939+0000", 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7354999999999994e-05 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228548+0000", 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060954800000000001 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429817+0000", 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20126904600000001 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634125+0000", 2023-01-15T14:33:43.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20430780300000001 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634259+0000", 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134144 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634410+0000", 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150447 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.863 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2223 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (57) in=7b,truncate 3411854] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 124.449067709, 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87671107599999998, 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:43.864 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2223 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:33:43.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858052+0000", 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858056+0000", 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5619999999999997e-06 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858067+0000", 2023-01-15T14:33:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0328e-05 2023-01-15T14:33:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858071+0000", 2023-01-15T14:33:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6439999999999999e-06 2023-01-15T14:33:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:43.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025446+0000", 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16737407200000001 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025503+0000", 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7952999999999999e-05 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227970+0000", 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202466866 2023-01-15T14:33:44.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:44.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228139+0000", 2023-01-15T14:33:44.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016889100000000001 2023-01-15T14:33:44.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:33:44.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228284+0000", 2023-01-15T14:33:44.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014493700000000001 2023-01-15T14:33:44.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228470+0000", 2023-01-15T14:33:44.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185541 2023-01-15T14:33:44.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228481+0000", 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.167e-05 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228537+0000", 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5912999999999998e-05 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:44.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228563+0000", 2023-01-15T14:33:44.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5613e-05 2023-01-15T14:33:44.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.433181+0000", 2023-01-15T14:33:44.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20461838099999999 2023-01-15T14:33:44.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734620+0000", 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301438386 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734682+0000", 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2373999999999996e-05 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:44.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734764+0000", 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1656999999999994e-05 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2224 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 124.449042659, 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97704279199999999, 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:44.171 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:44.172 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:44.172 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:44.172 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.172 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2224 2023-01-15T14:33:44.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.172 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:44.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:44.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:33:44.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858077+0000", 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:44.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858080+0000", 2023-01-15T14:33:44.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:33:44.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:44.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858089+0000", 2023-01-15T14:33:44.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3449999999999995e-06 2023-01-15T14:33:44.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858093+0000", 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9820000000000002e-06 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025520+0000", 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.167426556 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025594+0000", 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4339999999999996e-05 2023-01-15T14:33:44.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228608+0000", 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20301395699999999 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228752+0000", 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014385 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834878+0000", 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60612591500000002 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834987+0000", 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108733 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.835120+0000", 2023-01-15T14:33:44.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133688 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3120 3.40 3:03b54904:::smithi09025264-12:head [write 578273~466432] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.375090776999997, 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72661194799999995, 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:44.179 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:44.179 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.179 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3120 2023-01-15T14:33:44.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.179 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:44.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:44.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:33:44.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:44.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:33:44.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:44.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932026+0000", 2023-01-15T14:33:44.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:44.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932397+0000", 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037070099999999997 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932419+0000", 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1923999999999999e-05 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932435+0000", 2023-01-15T14:33:44.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6509999999999999e-05 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933751+0000", 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013156050000000001 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933765+0000", 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3952e-05 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238456+0000", 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30469085000000001 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238797+0000", 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034117300000000001 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:33:44.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239214+0000", 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000416845 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240273+0000", 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058811 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240292+0000", 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:33:44.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240555+0000", 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026299500000000003 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242072+0000", 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015175550000000001 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244038+0000", 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001966176 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658099+0000", 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41406103300000002 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658248+0000", 2023-01-15T14:33:44.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148577 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658642+0000", 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039346900000000002 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3121 3.40 3:03b54904:::smithi09025264-12:head [write 1739595~452815] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:33:44.187 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:33:44.188 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.374658830999998, 2023-01-15T14:33:44.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72675957599999996, 2023-01-15T14:33:44.188 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:44.188 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.188 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:44.188 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:44.188 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.188 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3121 2023-01-15T14:33:44.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.188 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:44.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:44.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932460+0000", 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932725+0000", 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026482699999999998 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932741+0000", 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5880999999999999e-05 2023-01-15T14:33:44.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932750+0000", 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2380000000000003e-06 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034479+0000", 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10172906399999999 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034506+0000", 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7336000000000001e-05 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240672+0000", 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206166074 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240894+0000", 2023-01-15T14:33:44.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221189 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241244+0000", 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000350816 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241923+0000", 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067891200000000005 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241943+0000", 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0069999999999999e-05 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243425+0000", 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001481719 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:44.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244087+0000", 2023-01-15T14:33:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066184700000000002 2023-01-15T14:33:44.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244230+0000", 2023-01-15T14:33:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143277 2023-01-15T14:33:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658784+0000", 2023-01-15T14:33:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.414553479 2023-01-15T14:33:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658904+0000", 2023-01-15T14:33:44.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120024 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659221+0000", 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031742700000000001 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3122 3.40 3:03b54904:::smithi09025264-12:head [write 2722929~655344] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:33:44.246 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:33:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.374354732999997, 2023-01-15T14:33:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72706898900000005, 2023-01-15T14:33:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3122 2023-01-15T14:33:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:44.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:33:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:33:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932765+0000", 2023-01-15T14:33:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933097+0000", 2023-01-15T14:33:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220299999999999 2023-01-15T14:33:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:44.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933106+0000", 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5459999999999997e-06 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933113+0000", 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3419999999999996e-06 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034598+0000", 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101485804 2023-01-15T14:33:44.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034625+0000", 2023-01-15T14:33:44.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6859000000000002e-05 2023-01-15T14:33:44.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242147+0000", 2023-01-15T14:33:44.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20752189600000001 2023-01-15T14:33:44.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:44.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242517+0000", 2023-01-15T14:33:44.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037032200000000002 2023-01-15T14:33:44.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:33:44.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243157+0000", 2023-01-15T14:33:44.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063935700000000001 2023-01-15T14:33:44.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243809+0000", 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065222000000000001 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243827+0000", 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8260000000000001e-05 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244122+0000", 2023-01-15T14:33:44.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029469400000000002 2023-01-15T14:33:44.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:44.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245633+0000", 2023-01-15T14:33:44.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001511068 2023-01-15T14:33:44.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.346711+0000", 2023-01-15T14:33:44.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101077812 2023-01-15T14:33:44.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659372+0000", 2023-01-15T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31266154800000001 2023-01-15T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659488+0000", 2023-01-15T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115444 2023-01-15T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:44.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659835+0000", 2023-01-15T14:33:44.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034670300000000002 2023-01-15T14:33:44.257 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.257 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:44.257 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.257 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3123 3.40 3:03b54904:::smithi09025264-12:head [setxattr _header (58) in=7b,truncate 3378273] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:33:44.258 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:33:44.258 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.373985040000001, 2023-01-15T14:33:44.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82751564099999997, 2023-01-15T14:33:44.258 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:44.258 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.258 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:44.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:44.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.259 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3123 2023-01-15T14:33:44.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.259 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:44.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:44.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:33:44.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:44.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:33:44.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:44.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:33:44.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:44.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:44.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933139+0000", 2023-01-15T14:33:44.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7079999999999996e-06 2023-01-15T14:33:44.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:44.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933145+0000", 2023-01-15T14:33:44.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7849999999999997e-06 2023-01-15T14:33:44.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033331+0000", 2023-01-15T14:33:44.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100186076 2023-01-15T14:33:44.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034697+0000", 2023-01-15T14:33:44.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001366138 2023-01-15T14:33:44.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034718+0000", 2023-01-15T14:33:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.065e-05 2023-01-15T14:33:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244319+0000", 2023-01-15T14:33:44.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20960071 2023-01-15T14:33:44.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:44.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244576+0000", 2023-01-15T14:33:44.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025712899999999998 2023-01-15T14:33:44.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:33:44.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244932+0000", 2023-01-15T14:33:44.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035626000000000001 2023-01-15T14:33:44.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245315+0000", 2023-01-15T14:33:44.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038305199999999999 2023-01-15T14:33:44.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245330+0000", 2023-01-15T14:33:44.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5027e-05 2023-01-15T14:33:44.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245384+0000", 2023-01-15T14:33:44.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4267000000000001e-05 2023-01-15T14:33:44.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:44.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245654+0000", 2023-01-15T14:33:44.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026944000000000002 2023-01-15T14:33:44.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.436220+0000", 2023-01-15T14:33:44.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19056593799999999 2023-01-15T14:33:44.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760086+0000", 2023-01-15T14:33:44.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.323865927 2023-01-15T14:33:44.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:44.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760241+0000", 2023-01-15T14:33:44.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015535700000000001 2023-01-15T14:33:44.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:44.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760651+0000", 2023-01-15T14:33:44.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040990699999999999 2023-01-15T14:33:44.276 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.277 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:44.277 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.277 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3124 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:33:44.278 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:33:44.278 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 34.273758561999998, 2023-01-15T14:33:44.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72792673799999996, 2023-01-15T14:33:44.278 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:44.278 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:44.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:44.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:44.279 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.279 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3124 2023-01-15T14:33:44.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.279 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:44.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:44.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:33:44.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:44.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:33:44.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:44.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033360+0000", 2023-01-15T14:33:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:44.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033372+0000", 2023-01-15T14:33:44.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2850000000000001e-05 2023-01-15T14:33:44.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:44.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033397+0000", 2023-01-15T14:33:44.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4349e-05 2023-01-15T14:33:44.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033412+0000", 2023-01-15T14:33:44.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5420000000000001e-05 2023-01-15T14:33:44.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036811+0000", 2023-01-15T14:33:44.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033984169999999999 2023-01-15T14:33:44.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036821+0000", 2023-01-15T14:33:44.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0213e-05 2023-01-15T14:33:44.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245712+0000", 2023-01-15T14:33:44.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20889102800000001 2023-01-15T14:33:44.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245912+0000", 2023-01-15T14:33:44.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020024600000000001 2023-01-15T14:33:44.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:44.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760781+0000", 2023-01-15T14:33:44.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51486881299999998 2023-01-15T14:33:44.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:44.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760970+0000", 2023-01-15T14:33:44.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018893600000000001 2023-01-15T14:33:44.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:44.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:44.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:44.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.761289+0000", 2023-01-15T14:33:44.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031870899999999997 2023-01-15T14:33:44.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.292 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:44.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.292 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:44.293 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:44.293 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:44.293 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:33:44.295 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:43.355+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:44.298 INFO:tasks.rados.rados.0.smithi090.stdout:1317: done (1 left) 2023-01-15T14:33:44.298 INFO:tasks.rados.rados.0.smithi090.stdout:1318: done (0 left) 2023-01-15T14:33:44.298 INFO:tasks.rados.rados.0.smithi090.stdout:1321: done (0 left) 2023-01-15T14:33:44.298 INFO:tasks.rados.rados.0.smithi090.stdout:1322: write (excl) oid 47 current snap is 140 2023-01-15T14:33:44.299 INFO:tasks.rados.rados.0.smithi090.stdout:1322: seq_num 477 ranges {416939=707659,1896499=120440} 2023-01-15T14:33:44.299 INFO:tasks.rados.rados.0.smithi090.stdout:1322: writing smithi09025264-47 from 416939 to 1124598 tid 1 2023-01-15T14:33:44.299 INFO:tasks.rados.rados.0.smithi090.stdout:1322: writing smithi09025264-47 from 1896499 to 2016939 tid 2 2023-01-15T14:33:44.299 INFO:tasks.rados.rados.0.smithi090.stdout:1323: delete oid 38 current snap is 140 2023-01-15T14:33:44.299 INFO:tasks.rados.rados.0.smithi090.stdout:1322: finishing write tid 1 to smithi09025264-47 2023-01-15T14:33:44.300 INFO:tasks.rados.rados.0.smithi090.stdout:1322: finishing write tid 2 to smithi09025264-47 2023-01-15T14:33:44.300 INFO:tasks.rados.rados.0.smithi090.stdout:1322: finishing write tid 3 to smithi09025264-47 2023-01-15T14:33:44.300 INFO:tasks.rados.rados.0.smithi090.stdout:1322: finishing write tid 4 to smithi09025264-47 2023-01-15T14:33:44.300 INFO:tasks.rados.rados.0.smithi090.stdout:1322: oid 47 updating version 0 to 379 2023-01-15T14:33:44.300 INFO:tasks.rados.rados.0.smithi090.stdout:1322: oid 47 version 379 is already newer than 378 2023-01-15T14:33:44.300 INFO:tasks.rados.rados.0.smithi090.stdout:1322: oid 47 version 379 is already newer than 377 2023-01-15T14:33:44.301 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 379 (ObjNum 477 snap 140 seq_num 477) dirty exists 2023-01-15T14:33:44.301 INFO:tasks.rados.rados.0.smithi090.stdout:1322: left oid 47 (ObjNum 477 snap 140 seq_num 477) 2023-01-15T14:33:44.301 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:44.292+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:44.372 INFO:tasks.rados.rados.0.smithi090.stdout:1322: done (1 left) 2023-01-15T14:33:44.373 INFO:tasks.rados.rados.0.smithi090.stdout:1323: done (0 left) 2023-01-15T14:33:44.373 INFO:tasks.rados.rados.0.smithi090.stdout:1324: rollback oid 12 current snap is 140 2023-01-15T14:33:44.373 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 12 to 124 2023-01-15T14:33:44.373 INFO:tasks.rados.rados.0.smithi090.stdout:1325: write (excl) oid 39 current snap is 140 2023-01-15T14:33:44.373 INFO:tasks.rados.rados.0.smithi090.stdout:1325: seq_num 479 ranges {450553=574510,1613463=437090} 2023-01-15T14:33:44.385 INFO:tasks.rados.rados.0.smithi090.stdout:1325: writing smithi09025264-39 from 450553 to 1025063 tid 1 2023-01-15T14:33:44.393 INFO:tasks.rados.rados.0.smithi090.stdout:1325: writing smithi09025264-39 from 1613463 to 2050553 tid 2 2023-01-15T14:33:44.393 INFO:tasks.rados.rados.0.smithi090.stdout:1326: read oid 13 snap 132 2023-01-15T14:33:44.393 INFO:tasks.rados.rados.0.smithi090.stdout:1326: expect (ObjNum 412 snap 112 seq_num 412) 2023-01-15T14:33:44.393 INFO:tasks.rados.rados.0.smithi090.stdout:1327: write (excl) oid 40 current snap is 140 2023-01-15T14:33:44.393 INFO:tasks.rados.rados.0.smithi090.stdout:1325: finishing write tid 1 to smithi09025264-39 2023-01-15T14:33:44.393 INFO:tasks.rados.rados.0.smithi090.stdout:1327: seq_num 480 ranges {467360=649759,1671945=395415} 2023-01-15T14:33:44.396 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:44.395+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:44.399 INFO:tasks.rados.rados.0.smithi090.stdout:1327: writing smithi09025264-40 from 467360 to 1117119 tid 1 2023-01-15T14:33:44.404 INFO:tasks.rados.rados.0.smithi090.stdout:1327: writing smithi09025264-40 from 1671945 to 2067360 tid 2 2023-01-15T14:33:44.405 INFO:tasks.rados.rados.0.smithi090.stdout:1328: snap_create 2023-01-15T14:33:44.405 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:33:44.407 INFO:tasks.rados.rados.0.smithi090.stdout:1326: expect (ObjNum 412 snap 112 seq_num 412) 2023-01-15T14:33:44.413 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:44.413 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:33:44.413 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:33:44.413 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:44.423 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:33:44.497 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:44.492+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:44.522 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:44.523 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:33:44.523 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:33:44.523 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:33:44.523 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:44.532 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:33:44.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:33:44.562 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:33:44.598 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:44.595+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:44.599 INFO:tasks.rados.rados.0.smithi090.stdout:1326: done (3 left) 2023-01-15T14:33:44.599 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:44.599 INFO:tasks.rados.rados.0.smithi090.stdout:1324: finishing rollback tid 0 to smithi09025264-12 2023-01-15T14:33:44.599 INFO:tasks.rados.rados.0.smithi090.stdout:1325: finishing write tid 2 to smithi09025264-39 2023-01-15T14:33:44.599 INFO:tasks.rados.rados.0.smithi090.stdout:1324: finishing rollback tid 1 to smithi09025264-12 2023-01-15T14:33:44.599 INFO:tasks.rados.rados.0.smithi090.stdout:1324: finishing rollback tid 2 to smithi09025264-12 2023-01-15T14:33:44.599 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 307 (ObjNum 439 snap 121 seq_num 439) dirty exists 2023-01-15T14:33:44.599 INFO:tasks.rados.rados.0.smithi090.stdout:1325: finishing write tid 3 to smithi09025264-39 2023-01-15T14:33:44.599 INFO:tasks.rados.rados.0.smithi090.stdout:1325: finishing write tid 4 to smithi09025264-39 2023-01-15T14:33:44.599 INFO:tasks.rados.rados.0.smithi090.stdout:1325: oid 39 updating version 0 to 315 2023-01-15T14:33:44.599 INFO:tasks.rados.rados.0.smithi090.stdout:1325: oid 39 version 315 is already newer than 313 2023-01-15T14:33:44.600 INFO:tasks.rados.rados.0.smithi090.stdout:1325: oid 39 version 315 is already newer than 314 2023-01-15T14:33:44.600 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 315 (ObjNum 479 snap 140 seq_num 479) dirty exists 2023-01-15T14:33:44.600 INFO:tasks.rados.rados.0.smithi090.stdout:1325: left oid 39 (ObjNum 479 snap 140 seq_num 479) 2023-01-15T14:33:44.600 INFO:tasks.rados.rados.0.smithi090.stdout:1324: done (2 left) 2023-01-15T14:33:44.600 INFO:tasks.rados.rados.0.smithi090.stdout:1325: done (1 left) 2023-01-15T14:33:44.600 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:44.641 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:44.642 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:33:44.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:33:44.642 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:33:44.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.642 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3 3.3 3:ca59e2b0:::smithi09025264-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:44.642 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:33:44.642 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 318.86100791899997, 2023-01-15T14:33:44.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62968594499999997, 2023-01-15T14:33:44.642 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.642 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.643 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.643 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.643 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.643 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3 2023-01-15T14:33:44.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.643 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:33:44.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:33:44.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775131+0000", 2023-01-15T14:33:44.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:44.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775138+0000", 2023-01-15T14:33:44.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:33:44.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775148+0000", 2023-01-15T14:33:44.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7969999999999995e-06 2023-01-15T14:33:44.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775161+0000", 2023-01-15T14:33:44.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2469999999999999e-05 2023-01-15T14:33:44.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.857974+0000", 2023-01-15T14:33:44.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.082812887000000002 2023-01-15T14:33:44.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.650 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.650 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.858147+0000", 2023-01-15T14:33:44.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000173883 2023-01-15T14:33:44.650 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.650 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.650 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.651 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106098+0000", 2023-01-15T14:33:44.651 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24795019600000001 2023-01-15T14:33:44.651 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106181+0000", 2023-01-15T14:33:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3248e-05 2023-01-15T14:33:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:33:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106330+0000", 2023-01-15T14:33:44.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014918200000000001 2023-01-15T14:33:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106510+0000", 2023-01-15T14:33:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017979399999999999 2023-01-15T14:33:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106527+0000", 2023-01-15T14:33:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6643999999999999e-05 2023-01-15T14:33:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106605+0000", 2023-01-15T14:33:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8650000000000001e-05 2023-01-15T14:33:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:44.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106644+0000", 2023-01-15T14:33:44.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8686999999999997e-05 2023-01-15T14:33:44.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404687+0000", 2023-01-15T14:33:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.298042745 2023-01-15T14:33:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404735+0000", 2023-01-15T14:33:44.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8306000000000001e-05 2023-01-15T14:33:44.657 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.657 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.657 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:44.657 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404772+0000", 2023-01-15T14:33:44.657 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6977000000000002e-05 2023-01-15T14:33:44.657 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404818+0000", 2023-01-15T14:33:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.6439e-05 2023-01-15T14:33:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.658 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.659 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.659 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:33:44.659 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:33:44.659 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 318.86096985199998, 2023-01-15T14:33:44.659 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62995383500000002, 2023-01-15T14:33:44.659 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.659 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:44.659 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.659 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.660 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.660 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4 2023-01-15T14:33:44.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.660 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.660 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.660 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:33:44.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.661 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.661 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:33:44.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.661 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.661 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:33:44.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:44.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775174+0000", 2023-01-15T14:33:44.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3669999999999999e-06 2023-01-15T14:33:44.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775184+0000", 2023-01-15T14:33:44.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5869999999999999e-06 2023-01-15T14:33:44.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775190+0000", 2023-01-15T14:33:44.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:33:44.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868440+0000", 2023-01-15T14:33:44.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.093250126000000003 2023-01-15T14:33:44.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868555+0000", 2023-01-15T14:33:44.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115003 2023-01-15T14:33:44.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.302962+0000", 2023-01-15T14:33:44.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.43440729700000003 2023-01-15T14:33:44.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303197+0000", 2023-01-15T14:33:44.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023508000000000001 2023-01-15T14:33:44.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404866+0000", 2023-01-15T14:33:44.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101668723 2023-01-15T14:33:44.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404985+0000", 2023-01-15T14:33:44.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118853 2023-01-15T14:33:44.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.405124+0000", 2023-01-15T14:33:44.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013951999999999999 2023-01-15T14:33:44.667 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.667 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.667 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.667 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:144 3.9 3:9988ac9d:::smithi09025264-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:44.668 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:33:44.668 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 317.76963130600001, 2023-01-15T14:33:44.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63423976199999998, 2023-01-15T14:33:44.668 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.668 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:44.668 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.668 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.668 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.668 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 144 2023-01-15T14:33:44.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.669 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:33:44.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:33:44.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866508+0000", 2023-01-15T14:33:44.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:44.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866512+0000", 2023-01-15T14:33:44.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5999999999999998e-06 2023-01-15T14:33:44.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866523+0000", 2023-01-15T14:33:44.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0740000000000001e-05 2023-01-15T14:33:44.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866531+0000", 2023-01-15T14:33:44.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3310000000000004e-06 2023-01-15T14:33:44.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867744+0000", 2023-01-15T14:33:44.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012127399999999999 2023-01-15T14:33:44.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867810+0000", 2023-01-15T14:33:44.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6506000000000003e-05 2023-01-15T14:33:44.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097225+0000", 2023-01-15T14:33:44.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.22941514900000001 2023-01-15T14:33:44.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.674 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097295+0000", 2023-01-15T14:33:44.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:33:44.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500386+0000", 2023-01-15T14:33:44.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40309051699999998 2023-01-15T14:33:44.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500521+0000", 2023-01-15T14:33:44.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013561700000000001 2023-01-15T14:33:44.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500749+0000", 2023-01-15T14:33:44.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022754900000000001 2023-01-15T14:33:44.676 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.676 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.676 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.677 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:509 3.9 3:9f86bdcb:::smithi09025264-22:head [write 2873212~588997] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:33:44.677 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:33:44.677 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 294.82120696300001, 2023-01-15T14:33:44.677 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69623497099999998, 2023-01-15T14:33:44.677 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.677 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.677 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.677 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.677 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.677 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 509 2023-01-15T14:33:44.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.678 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:33:44.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:33:44.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814931+0000", 2023-01-15T14:33:44.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:44.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815512+0000", 2023-01-15T14:33:44.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058082200000000004 2023-01-15T14:33:44.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815553+0000", 2023-01-15T14:33:44.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1081999999999997e-05 2023-01-15T14:33:44.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815571+0000", 2023-01-15T14:33:44.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8028000000000001e-05 2023-01-15T14:33:44.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906452+0000", 2023-01-15T14:33:44.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090880586999999999 2023-01-15T14:33:44.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.682 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906564+0000", 2023-01-15T14:33:44.682 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111976 2023-01-15T14:33:44.682 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.682 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208424+0000", 2023-01-15T14:33:44.682 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.301860199 2023-01-15T14:33:44.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208718+0000", 2023-01-15T14:33:44.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029347299999999999 2023-01-15T14:33:44.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:33:44.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209175+0000", 2023-01-15T14:33:44.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045692399999999999 2023-01-15T14:33:44.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209725+0000", 2023-01-15T14:33:44.684 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055027600000000004 2023-01-15T14:33:44.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209751+0000", 2023-01-15T14:33:44.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5812999999999998e-05 2023-01-15T14:33:44.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.210006+0000", 2023-01-15T14:33:44.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025564099999999997 2023-01-15T14:33:44.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:44.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311247+0000", 2023-01-15T14:33:44.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10124119199999999 2023-01-15T14:33:44.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311454+0000", 2023-01-15T14:33:44.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020695800000000001 2023-01-15T14:33:44.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511022+0000", 2023-01-15T14:33:44.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19956729200000001 2023-01-15T14:33:44.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:44.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511100+0000", 2023-01-15T14:33:44.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8589000000000003e-05 2023-01-15T14:33:44.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511168+0000", 2023-01-15T14:33:44.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8039000000000004e-05 2023-01-15T14:33:44.688 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.688 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.688 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.688 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:510 3.9 3:9f86bdcb:::smithi09025264-22:head [setxattr _header (57) in=7b,truncate 3462209] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:33:44.689 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:33:44.689 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 294.82054349999999, 2023-01-15T14:33:44.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69609656399999997, 2023-01-15T14:33:44.689 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.689 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.689 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.689 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.689 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.689 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 510 2023-01-15T14:33:44.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.690 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:33:44.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:33:44.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815596+0000", 2023-01-15T14:33:44.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:44.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815606+0000", 2023-01-15T14:33:44.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7920000000000004e-06 2023-01-15T14:33:44.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815627+0000", 2023-01-15T14:33:44.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1239000000000001e-05 2023-01-15T14:33:44.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815637+0000", 2023-01-15T14:33:44.693 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0349e-05 2023-01-15T14:33:44.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.693 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.693 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906600+0000", 2023-01-15T14:33:44.693 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090963098000000006 2023-01-15T14:33:44.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.693 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.694 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906717+0000", 2023-01-15T14:33:44.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000116759 2023-01-15T14:33:44.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.694 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310000+0000", 2023-01-15T14:33:44.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40328322599999999 2023-01-15T14:33:44.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310249+0000", 2023-01-15T14:33:44.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024874400000000002 2023-01-15T14:33:44.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:33:44.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310559+0000", 2023-01-15T14:33:44.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310382 2023-01-15T14:33:44.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.696 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.696 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310951+0000", 2023-01-15T14:33:44.696 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039124500000000003 2023-01-15T14:33:44.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.696 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.696 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310975+0000", 2023-01-15T14:33:44.697 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4496000000000001e-05 2023-01-15T14:33:44.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.697 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.697 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311041+0000", 2023-01-15T14:33:44.697 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6309000000000005e-05 2023-01-15T14:33:44.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.697 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:44.698 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311260+0000", 2023-01-15T14:33:44.698 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021863399999999999 2023-01-15T14:33:44.698 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.698 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.698 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.698 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311588+0000", 2023-01-15T14:33:44.698 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032813599999999998 2023-01-15T14:33:44.698 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.698 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.699 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.699 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511351+0000", 2023-01-15T14:33:44.699 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19976275700000001 2023-01-15T14:33:44.699 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.699 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.699 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:44.699 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511507+0000", 2023-01-15T14:33:44.699 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015585100000000001 2023-01-15T14:33:44.699 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.700 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.700 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.700 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511693+0000", 2023-01-15T14:33:44.700 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186627 2023-01-15T14:33:44.700 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.700 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.700 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.700 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.700 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.701 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:511 3.9 3:9f86bdcb:::smithi09025264-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:33:44.701 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:33:44.701 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 294.82048613199998, 2023-01-15T14:33:44.701 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69666647500000001, 2023-01-15T14:33:44.701 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.701 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:44.701 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.701 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.701 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.702 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 511 2023-01-15T14:33:44.702 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.702 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.702 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.702 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.702 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:33:44.702 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.702 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.702 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.703 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:33:44.703 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.703 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.703 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.703 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815653+0000", 2023-01-15T14:33:44.703 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:44.703 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.704 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.704 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815660+0000", 2023-01-15T14:33:44.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0509999999999999e-06 2023-01-15T14:33:44.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.704 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.704 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815683+0000", 2023-01-15T14:33:44.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2742999999999999e-05 2023-01-15T14:33:44.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.705 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.705 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.705 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815698+0000", 2023-01-15T14:33:44.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5133999999999999e-05 2023-01-15T14:33:44.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.705 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.705 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.706 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908029+0000", 2023-01-15T14:33:44.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.092331448999999996 2023-01-15T14:33:44.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.706 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908193+0000", 2023-01-15T14:33:44.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016342799999999999 2023-01-15T14:33:44.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311115+0000", 2023-01-15T14:33:44.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40292222300000002 2023-01-15T14:33:44.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311224+0000", 2023-01-15T14:33:44.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010888899999999999 2023-01-15T14:33:44.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511800+0000", 2023-01-15T14:33:44.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20057596899999999 2023-01-15T14:33:44.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512036+0000", 2023-01-15T14:33:44.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236247 2023-01-15T14:33:44.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512321+0000", 2023-01-15T14:33:44.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028457599999999998 2023-01-15T14:33:44.709 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.709 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.709 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.710 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:693 3.3 3:ca59e2b0:::smithi09025264-1:head [sparse-read 0~2016807] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:33:44.710 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:33:44.710 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 276.737752785, 2023-01-15T14:33:44.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70514199600000005, 2023-01-15T14:33:44.710 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.710 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:44.710 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.711 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.711 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.711 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 693 2023-01-15T14:33:44.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.711 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:33:44.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:33:44.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898387+0000", 2023-01-15T14:33:44.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:44.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898390+0000", 2023-01-15T14:33:44.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.252e-06 2023-01-15T14:33:44.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898401+0000", 2023-01-15T14:33:44.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0977e-05 2023-01-15T14:33:44.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.998532+0000", 2023-01-15T14:33:44.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100130443 2023-01-15T14:33:44.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602663+0000", 2023-01-15T14:33:44.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60413100500000005 2023-01-15T14:33:44.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602817+0000", 2023-01-15T14:33:44.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015440500000000001 2023-01-15T14:33:44.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.603530+0000", 2023-01-15T14:33:44.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071246299999999999 2023-01-15T14:33:44.716 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.716 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.716 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.716 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1421 3.a 3:5919d83e:::smithi09025264-13:head [write 1723287~549075] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:33:44.716 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:33:44.717 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 209.72383827600001, 2023-01-15T14:33:44.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61118175699999999, 2023-01-15T14:33:44.717 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.717 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.717 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.717 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.717 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.717 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1421 2023-01-15T14:33:44.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.717 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.718 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:33:44.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.718 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:33:44.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912300+0000", 2023-01-15T14:33:44.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:44.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913127+0000", 2023-01-15T14:33:44.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00082704000000000002 2023-01-15T14:33:44.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913158+0000", 2023-01-15T14:33:44.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1146999999999998e-05 2023-01-15T14:33:44.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913177+0000", 2023-01-15T14:33:44.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:33:44.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913384+0000", 2023-01-15T14:33:44.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000206589 2023-01-15T14:33:44.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913408+0000", 2023-01-15T14:33:44.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3493e-05 2023-01-15T14:33:44.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318579+0000", 2023-01-15T14:33:44.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40517134399999999 2023-01-15T14:33:44.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318806+0000", 2023-01-15T14:33:44.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022680899999999999 2023-01-15T14:33:44.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:33:44.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319060+0000", 2023-01-15T14:33:44.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254069 2023-01-15T14:33:44.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319527+0000", 2023-01-15T14:33:44.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000467081 2023-01-15T14:33:44.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319544+0000", 2023-01-15T14:33:44.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6949999999999999e-05 2023-01-15T14:33:44.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319778+0000", 2023-01-15T14:33:44.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023391500000000001 2023-01-15T14:33:44.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:44.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421126+0000", 2023-01-15T14:33:44.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101348626 2023-01-15T14:33:44.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522834+0000", 2023-01-15T14:33:44.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10170761 2023-01-15T14:33:44.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523142+0000", 2023-01-15T14:33:44.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854800000000001 2023-01-15T14:33:44.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:44.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523251+0000", 2023-01-15T14:33:44.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108411 2023-01-15T14:33:44.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523484+0000", 2023-01-15T14:33:44.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023292500000000001 2023-01-15T14:33:44.727 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.727 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.727 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.728 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1422 3.a 3:5919d83e:::smithi09025264-13:head [write 2781837~703343] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:33:44.728 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:33:44.728 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 209.72289948, 2023-01-15T14:33:44.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61062001700000001, 2023-01-15T14:33:44.728 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.728 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.728 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.728 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.729 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.729 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1422 2023-01-15T14:33:44.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.729 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:33:44.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:33:44.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913239+0000", 2023-01-15T14:33:44.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:44.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914075+0000", 2023-01-15T14:33:44.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00083594099999999998 2023-01-15T14:33:44.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914104+0000", 2023-01-15T14:33:44.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8938999999999999e-05 2023-01-15T14:33:44.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914121+0000", 2023-01-15T14:33:44.732 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7306999999999999e-05 2023-01-15T14:33:44.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.732 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914234+0000", 2023-01-15T14:33:44.732 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112737 2023-01-15T14:33:44.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914257+0000", 2023-01-15T14:33:44.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3156e-05 2023-01-15T14:33:44.733 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319798+0000", 2023-01-15T14:33:44.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40554153100000001 2023-01-15T14:33:44.733 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.734 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.734 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319991+0000", 2023-01-15T14:33:44.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019217999999999999 2023-01-15T14:33:44.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.734 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:33:44.734 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320237+0000", 2023-01-15T14:33:44.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024621099999999998 2023-01-15T14:33:44.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.734 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.735 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320759+0000", 2023-01-15T14:33:44.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052174700000000003 2023-01-15T14:33:44.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.735 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.735 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.735 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320776+0000", 2023-01-15T14:33:44.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7173e-05 2023-01-15T14:33:44.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.735 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.736 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.736 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.321080+0000", 2023-01-15T14:33:44.736 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030424400000000001 2023-01-15T14:33:44.736 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.736 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.736 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:44.736 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421162+0000", 2023-01-15T14:33:44.736 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100082085 2023-01-15T14:33:44.736 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.736 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522963+0000", 2023-01-15T14:33:44.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10180109399999999 2023-01-15T14:33:44.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523576+0000", 2023-01-15T14:33:44.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061277499999999997 2023-01-15T14:33:44.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:44.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523664+0000", 2023-01-15T14:33:44.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8002000000000002e-05 2023-01-15T14:33:44.738 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523861+0000", 2023-01-15T14:33:44.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019685899999999999 2023-01-15T14:33:44.739 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.739 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.739 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.739 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.739 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1423 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 3630312] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:33:44.739 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:33:44.739 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 209.72197694499999, 2023-01-15T14:33:44.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70862486899999999, 2023-01-15T14:33:44.740 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.740 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.740 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.740 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.740 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.740 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1423 2023-01-15T14:33:44.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.740 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:33:44.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:33:44.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914162+0000", 2023-01-15T14:33:44.742 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:44.742 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914172+0000", 2023-01-15T14:33:44.742 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0267e-05 2023-01-15T14:33:44.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.743 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.743 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914188+0000", 2023-01-15T14:33:44.743 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6212999999999999e-05 2023-01-15T14:33:44.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.743 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.743 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914201+0000", 2023-01-15T14:33:44.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3073e-05 2023-01-15T14:33:44.744 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.744 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.744 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.744 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914315+0000", 2023-01-15T14:33:44.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011357399999999999 2023-01-15T14:33:44.744 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.744 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.744 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.745 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914334+0000", 2023-01-15T14:33:44.745 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9004e-05 2023-01-15T14:33:44.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.745 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.745 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421258+0000", 2023-01-15T14:33:44.745 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.506924606 2023-01-15T14:33:44.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.745 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421518+0000", 2023-01-15T14:33:44.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025950499999999999 2023-01-15T14:33:44.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:33:44.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421843+0000", 2023-01-15T14:33:44.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032498299999999999 2023-01-15T14:33:44.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422222+0000", 2023-01-15T14:33:44.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037867599999999999 2023-01-15T14:33:44.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422245+0000", 2023-01-15T14:33:44.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3096999999999999e-05 2023-01-15T14:33:44.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422343+0000", 2023-01-15T14:33:44.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8487999999999996e-05 2023-01-15T14:33:44.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:44.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522890+0000", 2023-01-15T14:33:44.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10054658900000001 2023-01-15T14:33:44.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523075+0000", 2023-01-15T14:33:44.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185386 2023-01-15T14:33:44.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622151+0000", 2023-01-15T14:33:44.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099075724000000004 2023-01-15T14:33:44.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:44.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622244+0000", 2023-01-15T14:33:44.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3387999999999994e-05 2023-01-15T14:33:44.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622788+0000", 2023-01-15T14:33:44.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000543973 2023-01-15T14:33:44.750 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.751 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.751 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.751 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1424 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:33:44.751 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:33:44.751 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 209.721921573, 2023-01-15T14:33:44.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70984700199999995, 2023-01-15T14:33:44.751 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.751 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:44.751 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.752 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.752 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.752 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1424 2023-01-15T14:33:44.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.752 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:33:44.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.753 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.753 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:33:44.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.753 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.753 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914218+0000", 2023-01-15T14:33:44.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:44.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914224+0000", 2023-01-15T14:33:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6769999999999998e-06 2023-01-15T14:33:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914242+0000", 2023-01-15T14:33:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7759000000000001e-05 2023-01-15T14:33:44.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014331+0000", 2023-01-15T14:33:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088819 2023-01-15T14:33:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114642+0000", 2023-01-15T14:33:44.755 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100311027 2023-01-15T14:33:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114682+0000", 2023-01-15T14:33:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0028999999999999e-05 2023-01-15T14:33:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523918+0000", 2023-01-15T14:33:44.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40923631500000002 2023-01-15T14:33:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524124+0000", 2023-01-15T14:33:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020524900000000001 2023-01-15T14:33:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622921+0000", 2023-01-15T14:33:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098797069000000001 2023-01-15T14:33:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.623328+0000", 2023-01-15T14:33:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000407539 2023-01-15T14:33:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.624066+0000", 2023-01-15T14:33:44.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073762899999999995 2023-01-15T14:33:44.759 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.759 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.759 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.759 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1430 3.a 3:52df5138:::smithi09025264-25:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:33:44.759 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:33:44.759 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 209.62176181199999, 2023-01-15T14:33:44.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81872653100000004, 2023-01-15T14:33:44.759 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.760 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.760 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.760 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.760 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.760 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1430 2023-01-15T14:33:44.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.760 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.760 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:33:44.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:33:44.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014376+0000", 2023-01-15T14:33:44.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:44.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014391+0000", 2023-01-15T14:33:44.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4922999999999999e-05 2023-01-15T14:33:44.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014417+0000", 2023-01-15T14:33:44.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5698999999999999e-05 2023-01-15T14:33:44.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014435+0000", 2023-01-15T14:33:44.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7827000000000001e-05 2023-01-15T14:33:44.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114759+0000", 2023-01-15T14:33:44.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100324859 2023-01-15T14:33:44.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114771+0000", 2023-01-15T14:33:44.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:33:44.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524222+0000", 2023-01-15T14:33:44.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40945100800000001 2023-01-15T14:33:44.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524545+0000", 2023-01-15T14:33:44.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000322367 2023-01-15T14:33:44.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:33:44.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.626981+0000", 2023-01-15T14:33:44.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102436308 2023-01-15T14:33:44.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627874+0000", 2023-01-15T14:33:44.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089274899999999997 2023-01-15T14:33:44.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627930+0000", 2023-01-15T14:33:44.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6122000000000002e-05 2023-01-15T14:33:44.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628609+0000", 2023-01-15T14:33:44.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00067958700000000005 2023-01-15T14:33:44.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628798+0000", 2023-01-15T14:33:44.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018809 2023-01-15T14:33:44.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:44.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628870+0000", 2023-01-15T14:33:44.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2844999999999998e-05 2023-01-15T14:33:44.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832404+0000", 2023-01-15T14:33:44.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20353310299999999 2023-01-15T14:33:44.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:44.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832606+0000", 2023-01-15T14:33:44.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020207999999999999 2023-01-15T14:33:44.770 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833105+0000", 2023-01-15T14:33:44.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049945399999999998 2023-01-15T14:33:44.771 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.771 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.771 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.771 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.771 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.771 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1431 3.a 3:52df5138:::smithi09025264-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:33:44.771 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:33:44.771 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 209.621670524, 2023-01-15T14:33:44.771 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81967516600000001, 2023-01-15T14:33:44.772 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.772 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:44.772 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.772 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.772 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.772 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1431 2023-01-15T14:33:44.772 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.772 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.772 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.773 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.773 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:33:44.773 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.773 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.773 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.773 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.773 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:33:44.773 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.773 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014469+0000", 2023-01-15T14:33:44.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:44.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014477+0000", 2023-01-15T14:33:44.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9300000000000003e-06 2023-01-15T14:33:44.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014495+0000", 2023-01-15T14:33:44.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8363999999999999e-05 2023-01-15T14:33:44.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.776 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114610+0000", 2023-01-15T14:33:44.776 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10011521700000001 2023-01-15T14:33:44.776 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.776 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.776 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.776 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214948+0000", 2023-01-15T14:33:44.776 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100337739 2023-01-15T14:33:44.776 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.776 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214962+0000", 2023-01-15T14:33:44.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4516e-05 2023-01-15T14:33:44.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524874+0000", 2023-01-15T14:33:44.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30991196599999998 2023-01-15T14:33:44.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:44.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.525013+0000", 2023-01-15T14:33:44.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013873999999999999 2023-01-15T14:33:44.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628222+0000", 2023-01-15T14:33:44.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103208721 2023-01-15T14:33:44.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628504+0000", 2023-01-15T14:33:44.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028245700000000001 2023-01-15T14:33:44.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833214+0000", 2023-01-15T14:33:44.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.204709691 2023-01-15T14:33:44.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.780 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.780 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833555+0000", 2023-01-15T14:33:44.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034094000000000003 2023-01-15T14:33:44.780 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.780 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.780 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.834145+0000", 2023-01-15T14:33:44.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059006900000000003 2023-01-15T14:33:44.781 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.781 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.781 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.781 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2439 3.3 3:c11c7d81:::smithi09025264-11:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:33:44.781 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:33:44.781 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 109.552766019, 2023-01-15T14:33:44.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70719632099999996, 2023-01-15T14:33:44.781 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.782 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.782 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.782 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.782 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.782 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2439 2023-01-15T14:33:44.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.782 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:33:44.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:33:44.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083373+0000", 2023-01-15T14:33:44.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:44.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083380+0000", 2023-01-15T14:33:44.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:33:44.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083391+0000", 2023-01-15T14:33:44.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0366e-05 2023-01-15T14:33:44.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083397+0000", 2023-01-15T14:33:44.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2740000000000004e-06 2023-01-15T14:33:44.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.786 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083648+0000", 2023-01-15T14:33:44.786 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025052599999999999 2023-01-15T14:33:44.786 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.786 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.786 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:44.786 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083666+0000", 2023-01-15T14:33:44.786 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8848999999999999e-05 2023-01-15T14:33:44.786 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.786 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.786 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.787 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237725+0000", 2023-01-15T14:33:44.787 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.154058361 2023-01-15T14:33:44.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.787 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.787 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237945+0000", 2023-01-15T14:33:44.787 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021978300000000001 2023-01-15T14:33:44.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.787 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.788 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.386967+0000", 2023-01-15T14:33:44.788 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.149022562 2023-01-15T14:33:44.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.788 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.788 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.788 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387230+0000", 2023-01-15T14:33:44.788 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026256100000000002 2023-01-15T14:33:44.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.788 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:33:44.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387695+0000", 2023-01-15T14:33:44.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046480099999999998 2023-01-15T14:33:44.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388112+0000", 2023-01-15T14:33:44.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041713200000000002 2023-01-15T14:33:44.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388130+0000", 2023-01-15T14:33:44.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8315999999999999e-05 2023-01-15T14:33:44.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388189+0000", 2023-01-15T14:33:44.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8665999999999997e-05 2023-01-15T14:33:44.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:44.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388829+0000", 2023-01-15T14:33:44.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00063992699999999999 2023-01-15T14:33:44.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.489989+0000", 2023-01-15T14:33:44.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116059199999999 2023-01-15T14:33:44.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790300+0000", 2023-01-15T14:33:44.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30031108699999998 2023-01-15T14:33:44.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:44.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790396+0000", 2023-01-15T14:33:44.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5721000000000001e-05 2023-01-15T14:33:44.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790571+0000", 2023-01-15T14:33:44.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017464999999999999 2023-01-15T14:33:44.793 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.793 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.793 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.793 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2440 3.3 3:c11c7d81:::smithi09025264-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:33:44.793 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:33:44.794 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 109.55267428800001, 2023-01-15T14:33:44.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70759506000000005, 2023-01-15T14:33:44.794 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.794 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:44.794 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.794 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.794 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.794 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2440 2023-01-15T14:33:44.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.795 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:33:44.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:33:44.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.796 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.796 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083465+0000", 2023-01-15T14:33:44.796 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:44.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.796 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.796 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083470+0000", 2023-01-15T14:33:44.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2630000000000003e-06 2023-01-15T14:33:44.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.797 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.797 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083481+0000", 2023-01-15T14:33:44.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0699999999999999e-05 2023-01-15T14:33:44.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.797 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083488+0000", 2023-01-15T14:33:44.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8870000000000003e-06 2023-01-15T14:33:44.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180331+0000", 2023-01-15T14:33:44.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.096843296999999995 2023-01-15T14:33:44.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:44.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180342+0000", 2023-01-15T14:33:44.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.031e-05 2023-01-15T14:33:44.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238012+0000", 2023-01-15T14:33:44.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.057670829 2023-01-15T14:33:44.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238257+0000", 2023-01-15T14:33:44.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000244959 2023-01-15T14:33:44.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590126+0000", 2023-01-15T14:33:44.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35186851699999999 2023-01-15T14:33:44.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590389+0000", 2023-01-15T14:33:44.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026276799999999998 2023-01-15T14:33:44.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790692+0000", 2023-01-15T14:33:44.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20030350299999999 2023-01-15T14:33:44.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790891+0000", 2023-01-15T14:33:44.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019893999999999999 2023-01-15T14:33:44.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.791061+0000", 2023-01-15T14:33:44.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169913 2023-01-15T14:33:44.808 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.809 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.809 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.809 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2450 3.a 3:52c3e133:::smithi09025264-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e230)", 2023-01-15T14:33:44.809 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:33:44.809 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 107.788821676, 2023-01-15T14:33:44.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.68251353100000001, 2023-01-15T14:33:44.809 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.809 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:44.809 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.810 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.810 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.810 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2450 2023-01-15T14:33:44.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.810 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:33:44.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:33:44.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847318+0000", 2023-01-15T14:33:44.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:44.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847324+0000", 2023-01-15T14:33:44.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:33:44.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847339+0000", 2023-01-15T14:33:44.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5582000000000001e-05 2023-01-15T14:33:44.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847349+0000", 2023-01-15T14:33:44.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0139999999999999e-05 2023-01-15T14:33:44.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947644+0000", 2023-01-15T14:33:44.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10029455600000001 2023-01-15T14:33:44.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:44.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947657+0000", 2023-01-15T14:33:44.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2796000000000001e-05 2023-01-15T14:33:44.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125638+0000", 2023-01-15T14:33:44.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17798127799999999 2023-01-15T14:33:44.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125780+0000", 2023-01-15T14:33:44.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014173000000000001 2023-01-15T14:33:44.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529035+0000", 2023-01-15T14:33:44.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40325542199999997 2023-01-15T14:33:44.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529366+0000", 2023-01-15T14:33:44.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033111700000000002 2023-01-15T14:33:44.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529832+0000", 2023-01-15T14:33:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046592399999999999 2023-01-15T14:33:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.817 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3390 3.1b 3:d85f42d9:::smithi09025264-45:head [write 1944996~791497] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:33:44.818 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:33:44.818 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 9.5761429669999991, 2023-01-15T14:33:44.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.611364087, 2023-01-15T14:33:44.818 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.818 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.818 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.818 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.818 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3390 2023-01-15T14:33:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:33:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:33:44.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059995+0000", 2023-01-15T14:33:44.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:44.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061164+0000", 2023-01-15T14:33:44.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001168325 2023-01-15T14:33:44.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061194+0000", 2023-01-15T14:33:44.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.0485999999999999e-05 2023-01-15T14:33:44.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061213+0000", 2023-01-15T14:33:44.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.891e-05 2023-01-15T14:33:44.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159748+0000", 2023-01-15T14:33:44.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098535175000000003 2023-01-15T14:33:44.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159766+0000", 2023-01-15T14:33:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:33:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263011+0000", 2023-01-15T14:33:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10324554699999999 2023-01-15T14:33:44.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263253+0000", 2023-01-15T14:33:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000241568 2023-01-15T14:33:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466464+0000", 2023-01-15T14:33:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.203211006 2023-01-15T14:33:44.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466638+0000", 2023-01-15T14:33:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017412100000000001 2023-01-15T14:33:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:33:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466911+0000", 2023-01-15T14:33:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000272971 2023-01-15T14:33:44.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467425+0000", 2023-01-15T14:33:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00051373 2023-01-15T14:33:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467457+0000", 2023-01-15T14:33:44.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2558999999999998e-05 2023-01-15T14:33:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467906+0000", 2023-01-15T14:33:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000448985 2023-01-15T14:33:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469895+0000", 2023-01-15T14:33:44.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0019888810000000001 2023-01-15T14:33:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470611+0000", 2023-01-15T14:33:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071556699999999996 2023-01-15T14:33:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671178+0000", 2023-01-15T14:33:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20056716699999999 2023-01-15T14:33:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671277+0000", 2023-01-15T14:33:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9594999999999998e-05 2023-01-15T14:33:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671361+0000", 2023-01-15T14:33:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4048000000000006e-05 2023-01-15T14:33:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3391 3.1b 3:d85f42d9:::smithi09025264-45:head [write 3196560~554574] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:33:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:33:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 9.5748998440000008, 2023-01-15T14:33:44.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71066412899999998, 2023-01-15T14:33:44.831 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.831 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.831 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.831 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.831 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.831 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3391 2023-01-15T14:33:44.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.831 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:33:44.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:33:44.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061238+0000", 2023-01-15T14:33:44.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:44.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062134+0000", 2023-01-15T14:33:44.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089621299999999998 2023-01-15T14:33:44.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062160+0000", 2023-01-15T14:33:44.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5908999999999999e-05 2023-01-15T14:33:44.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062172+0000", 2023-01-15T14:33:44.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.189e-05 2023-01-15T14:33:44.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159827+0000", 2023-01-15T14:33:44.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097654866000000007 2023-01-15T14:33:44.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159846+0000", 2023-01-15T14:33:44.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8516e-05 2023-01-15T14:33:44.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.836 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263326+0000", 2023-01-15T14:33:44.836 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103480766 2023-01-15T14:33:44.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.836 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263535+0000", 2023-01-15T14:33:44.836 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020861299999999999 2023-01-15T14:33:44.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467576+0000", 2023-01-15T14:33:44.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20404118600000001 2023-01-15T14:33:44.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467736+0000", 2023-01-15T14:33:44.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016000699999999999 2023-01-15T14:33:44.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:33:44.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467987+0000", 2023-01-15T14:33:44.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025042199999999998 2023-01-15T14:33:44.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468369+0000", 2023-01-15T14:33:44.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038238600000000002 2023-01-15T14:33:44.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468387+0000", 2023-01-15T14:33:44.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7730000000000001e-05 2023-01-15T14:33:44.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468719+0000", 2023-01-15T14:33:44.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033272599999999999 2023-01-15T14:33:44.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.840 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:44.840 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470317+0000", 2023-01-15T14:33:44.840 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001597336 2023-01-15T14:33:44.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.840 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.840 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570851+0000", 2023-01-15T14:33:44.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10053426 2023-01-15T14:33:44.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771618+0000", 2023-01-15T14:33:44.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200767152 2023-01-15T14:33:44.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:44.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771768+0000", 2023-01-15T14:33:44.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015015699999999999 2023-01-15T14:33:44.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771905+0000", 2023-01-15T14:33:44.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013621099999999999 2023-01-15T14:33:44.842 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.842 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.842 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.843 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3392 3.1b 3:d85f42d9:::smithi09025264-45:head [setxattr _header (58) in=7b,truncate 3882483] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:33:44.843 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:33:44.897 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 9.5739506090000006, 2023-01-15T14:33:44.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71008243400000004, 2023-01-15T14:33:44.898 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.898 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:44.898 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.898 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.898 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.898 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3392 2023-01-15T14:33:44.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.899 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:33:44.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:33:44.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062188+0000", 2023-01-15T14:33:44.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:44.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062195+0000", 2023-01-15T14:33:44.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2529999999999998e-06 2023-01-15T14:33:44.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062208+0000", 2023-01-15T14:33:44.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3003000000000001e-05 2023-01-15T14:33:44.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062215+0000", 2023-01-15T14:33:44.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5200000000000003e-06 2023-01-15T14:33:44.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159883+0000", 2023-01-15T14:33:44.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097667532000000001 2023-01-15T14:33:44.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159899+0000", 2023-01-15T14:33:44.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6759999999999999e-05 2023-01-15T14:33:44.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263853+0000", 2023-01-15T14:33:44.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10395361 2023-01-15T14:33:44.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264116+0000", 2023-01-15T14:33:44.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026336099999999999 2023-01-15T14:33:44.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468602+0000", 2023-01-15T14:33:44.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20448550900000001 2023-01-15T14:33:44.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:44.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468867+0000", 2023-01-15T14:33:44.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026495700000000002 2023-01-15T14:33:44.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:33:44.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469376+0000", 2023-01-15T14:33:44.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00050893100000000001 2023-01-15T14:33:44.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:44.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469789+0000", 2023-01-15T14:33:44.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000413158 2023-01-15T14:33:44.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:44.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469806+0000", 2023-01-15T14:33:44.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7513e-05 2023-01-15T14:33:44.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:44.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469884+0000", 2023-01-15T14:33:44.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7816000000000002e-05 2023-01-15T14:33:44.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:44.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470332+0000", 2023-01-15T14:33:44.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044807700000000001 2023-01-15T14:33:44.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570976+0000", 2023-01-15T14:33:44.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10064426899999999 2023-01-15T14:33:44.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:44.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772058+0000", 2023-01-15T14:33:44.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20108148200000001 2023-01-15T14:33:44.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:44.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772169+0000", 2023-01-15T14:33:44.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011120099999999999 2023-01-15T14:33:44.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:44.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772272+0000", 2023-01-15T14:33:44.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000102982 2023-01-15T14:33:44.910 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.910 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:44.910 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:44.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.910 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3393 3.1b 3:d85f42d9:::smithi09025264-45:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:33:44.910 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:33:44.910 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 9.5739165330000002, 2023-01-15T14:33:44.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71076842399999995, 2023-01-15T14:33:44.911 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:44.911 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:44.911 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:44.911 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:44.911 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:44.911 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3393 2023-01-15T14:33:44.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.911 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:44.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:44.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:33:44.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:44.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:33:44.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:44.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:44.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062223+0000", 2023-01-15T14:33:44.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:44.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:44.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062227+0000", 2023-01-15T14:33:44.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3900000000000003e-06 2023-01-15T14:33:44.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:44.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062237+0000", 2023-01-15T14:33:44.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4539999999999996e-06 2023-01-15T14:33:44.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.971 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:44.971 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062243+0000", 2023-01-15T14:33:44.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0329999999999997e-06 2023-01-15T14:33:44.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.971 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.972 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161412+0000", 2023-01-15T14:33:44.972 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099168708999999994 2023-01-15T14:33:44.972 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.972 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.972 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:44.972 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161424+0000", 2023-01-15T14:33:44.972 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2415999999999999e-05 2023-01-15T14:33:44.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264185+0000", 2023-01-15T14:33:44.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.1027609 2023-01-15T14:33:44.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264421+0000", 2023-01-15T14:33:44.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023640900000000001 2023-01-15T14:33:44.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.974 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.974 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:44.974 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469985+0000", 2023-01-15T14:33:44.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.205564001 2023-01-15T14:33:44.975 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:44.975 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:44.976 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:44.976 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470264+0000", 2023-01-15T14:33:44.976 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027918700000000002 2023-01-15T14:33:45.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:45.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:45.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:45.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772390+0000", 2023-01-15T14:33:45.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30212558299999998 2023-01-15T14:33:45.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:45.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:45.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:45.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772684+0000", 2023-01-15T14:33:45.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029361799999999999 2023-01-15T14:33:45.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:45.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:45.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:45.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772992+0000", 2023-01-15T14:33:45.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854100000000001 2023-01-15T14:33:45.288 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:45.288 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:45.289 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:45.289 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:45.289 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:45.289 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:45.289 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:33:45.290 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:33:45.290 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2023-01-15T14:33:45.290 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2023-01-15T14:33:45.290 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2023-01-15T14:33:45.290 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2023-01-15T14:33:45.292 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:33:45.293 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:45.293 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:45.293 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:45.293 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:45.293 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:45.294 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:45.294 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:45.294 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:45.295 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:33:45.295 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:33:45.295 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:33:45.295 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:33:45.295 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:33:45.295 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:33:45.295 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:33:45.295 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:33:45.297 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:33:45.313 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:45.284+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:45.381 INFO:tasks.rados.rados.0.smithi090.stdout:1327: finishing write tid 1 to smithi09025264-40 2023-01-15T14:33:45.448 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:45.448 INFO:teuthology.orchestra.run.smithi141.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2023-01-15T14:33:45.448 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:45.449 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:45.449 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:45.450 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:33:45.451 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:45.447+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:45.485 INFO:tasks.rados.rados.0.smithi090.stdout:1327: finishing write tid 2 to smithi09025264-40 2023-01-15T14:33:45.549 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:45.544+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:45.640 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:45.640 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:45.641 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:33:45.650 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:45.644+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:45.684 INFO:tasks.rados.rados.0.smithi090.stdout:1327: finishing write tid 3 to smithi09025264-40 2023-01-15T14:33:45.844 INFO:tasks.rados.rados.0.smithi090.stdout:1327: finishing write tid 4 to smithi09025264-40 2023-01-15T14:33:45.844 INFO:tasks.rados.rados.0.smithi090.stdout:1327: oid 40 updating version 0 to 308 2023-01-15T14:33:45.844 INFO:tasks.rados.rados.0.smithi090.stdout:1327: oid 40 updating version 308 to 310 2023-01-15T14:33:45.844 INFO:tasks.rados.rados.0.smithi090.stdout:1327: oid 40 version 310 is already newer than 309 2023-01-15T14:33:45.844 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 310 (ObjNum 480 snap 140 seq_num 480) dirty exists 2023-01-15T14:33:45.844 INFO:tasks.rados.rados.0.smithi090.stdout:1327: left oid 40 (ObjNum 480 snap 140 seq_num 480) 2023-01-15T14:33:45.844 INFO:tasks.rados.rados.0.smithi090.stdout:1327: done (0 left) 2023-01-15T14:33:45.845 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:45.840+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:45.845 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:45.845 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:45.845 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:33:45.945 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:45.940+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:46.003 INFO:tasks.rados.rados.0.smithi090.stdout:1328: done (0 left) 2023-01-15T14:33:46.003 INFO:tasks.rados.rados.0.smithi090.stdout:1329: rollback oid 10 current snap is 141 2023-01-15T14:33:46.003 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 10 to 127 2023-01-15T14:33:46.003 INFO:tasks.rados.rados.0.smithi090.stdout:1330: snap_create 2023-01-15T14:33:46.003 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:46.004 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:46.004 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:46.004 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:33:46.046 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:46.040+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:46.172 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:46.172 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:46.173 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:33:46.174 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:46.168+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:46.273 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:46.268+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:46.331 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:46.331 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:46.331 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:33:46.377 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:33:46.375+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:33:46.498 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:46.498 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:46.499 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:33:46.499 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:46.495+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:46.603 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:46.596+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:46.671 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:46.671 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:46.672 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:33:46.703 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:46.700+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:46.781 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:47.076 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:47.078 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:33:47.078 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:47.072+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:47.179 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:47.175+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:47.278 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:47.278 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:47.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:33:47.280 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:47.276+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:47.297 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:33:47.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:33:47.438 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:47.438 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:47.438 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:33:47.439 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:47.439 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:47.435+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:47.447 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:47.466 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:47.482 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:47.598 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:47.598 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:47.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:47.599 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:47.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.599 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:47.599 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:33:47.599 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 321.48654150099998, 2023-01-15T14:33:47.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:33:47.599 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:47.599 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:47.599 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:47.599 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:47.599 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:47.600 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:33:47.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.600 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:47.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:47.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:33:47.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:47.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:33:47.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:47.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:33:47.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:33:47.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:47.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:33:47.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:33:47.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:33:47.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:33:47.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:47.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:33:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:33:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:33:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:33:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:33:47.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:47.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:33:47.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:47.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 321.48626125300001, 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:47.609 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:47.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:33:47.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:47.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:47.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:33:47.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:33:47.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:47.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:33:47.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:33:47.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:33:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:33:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:33:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:33:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:33:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:33:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:33:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:33:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:33:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:33:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:33:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:33:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:33:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:33:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:33:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:33:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:33:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:33:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:33:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 321.48622863600002, 2023-01-15T14:33:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:33:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:33:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:33:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:47.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:33:47.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:47.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:33:47.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:47.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:47.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:33:47.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:33:47.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:47.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:33:47.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:33:47.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:47.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:33:47.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:33:47.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:33:47.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:33:47.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:47.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:33:47.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:33:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:33:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:47.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:33:47.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:33:47.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:47.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:33:47.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:33:47.765 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.765 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:47.765 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.766 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:47.766 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:33:47.766 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 321.38491064999999, 2023-01-15T14:33:47.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:33:47.766 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:47.766 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:47.766 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:47.766 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:47.767 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:47.767 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:33:47.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.767 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:47.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:47.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:33:47.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:47.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:33:47.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:47.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:33:47.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:47.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:47.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:33:47.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:33:47.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:47.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:33:47.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:33:47.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:47.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:33:47.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:33:47.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:33:47.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:33:47.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:47.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:33:47.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:33:47.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:33:47.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:33:47.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:47.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:33:47.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:33:47.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:47.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:33:47.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:33:47.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:47.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:33:47.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:33:47.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:47.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:33:47.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:33:47.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:47.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:33:47.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:33:47.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:47.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:33:47.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:33:47.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:47.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:33:47.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:33:47.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:47.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:33:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:33:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:33:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:33:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:33:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:33:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:33:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 321.38488064699999, 2023-01-15T14:33:47.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:33:47.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:47.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:33:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:33:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:33:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:33:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:33:47.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:33:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:33:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:33:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:33:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:33:47.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:33:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:33:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:33:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:33:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:33:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:33:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:33:47.927 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 320.86987208199997, 2023-01-15T14:33:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:33:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:33:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:47.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:33:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:33:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:33:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:47.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:33:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:33:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:33:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:33:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:33:47.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:47.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:33:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:33:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:33:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:33:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:33:47.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:33:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:33:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:33:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:33:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:33:47.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:33:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:33:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:33:47.937 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 320.86938972799999, 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:47.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:33:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:33:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:33:47.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:33:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:33:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:33:47.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:47.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:33:47.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:33:47.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:47.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:33:47.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:33:47.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:47.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:33:47.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:33:47.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:47.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:33:47.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:33:47.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:47.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:33:47.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:33:47.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:47.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:33:47.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:33:47.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:47.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:33:47.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:33:47.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:47.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:33:47.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:33:47.947 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.947 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:47.947 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:47.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.948 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:47.948 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:33:47.948 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 320.86933716499999, 2023-01-15T14:33:47.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:33:47.948 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:47.948 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:47.948 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:47.948 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:47.948 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:47.948 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:33:47.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.949 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:47.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:47.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:33:47.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:47.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:33:47.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:47.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:47.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:33:47.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:47.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:47.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:33:47.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:33:47.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:47.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:33:47.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:33:47.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:47.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:33:47.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:33:47.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:33:47.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:33:47.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:47.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:33:47.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:33:47.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:33:47.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:47.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:33:47.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:33:47.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:47.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:47.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:33:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:33:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:33:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 307.498135461, 2023-01-15T14:33:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:33:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:48.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:33:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.087 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:33:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:33:48.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:33:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:33:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:33:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:33:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:33:48.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:33:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:33:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:33:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:33:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:48.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:33:48.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:33:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:33:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:33:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:33:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:33:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:33:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:33:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:33:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:33:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:33:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 307.49801878699998, 2023-01-15T14:33:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:33:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:48.098 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:48.099 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:33:48.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.241 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:48.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:48.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:33:48.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:48.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:33:48.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:48.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:33:48.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:48.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:48.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:33:48.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:33:48.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:48.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:33:48.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:33:48.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:48.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:33:48.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:33:48.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:33:48.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:33:48.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:48.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:33:48.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:33:48.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:33:48.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:33:48.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:48.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:33:48.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:33:48.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:33:48.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:33:48.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:48.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:33:48.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:33:48.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:33:48.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:33:48.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.254 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.254 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.255 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:33:48.255 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:33:48.255 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 299.77188651099999, 2023-01-15T14:33:48.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:33:48.257 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:48.257 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:48.257 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:48.258 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:48.258 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:48.258 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:33:48.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.258 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:48.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:48.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:33:48.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:48.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:33:48.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:48.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:33:48.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:48.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:48.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:33:48.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:33:48.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:48.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:33:48.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:33:48.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:48.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:33:48.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:33:48.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:33:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:33:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:33:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:33:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:33:48.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:33:48.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:48.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:33:48.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:33:48.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:33:48.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:33:48.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.265 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.265 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.265 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:879 3.1 3:8b0a2647:::smithi09025264-8:head [rollback 2d] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:33:48.265 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:33:48.265 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 263.55563645199999, 2023-01-15T14:33:48.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.405745313, 2023-01-15T14:33:48.265 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:48.265 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:48.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:48.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:48.266 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:48.266 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 879 2023-01-15T14:33:48.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.266 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:48.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:48.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:33:48.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:48.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992170+0000", 2023-01-15T14:33:48.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:48.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992169+0000", 2023-01-15T14:33:48.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:48.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:48.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992174+0000", 2023-01-15T14:33:48.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2739999999999998e-06 2023-01-15T14:33:48.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:48.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992185+0000", 2023-01-15T14:33:48.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0319999999999999e-05 2023-01-15T14:33:48.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:48.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992192+0000", 2023-01-15T14:33:48.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:33:48.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992313+0000", 2023-01-15T14:33:48.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121317 2023-01-15T14:33:48.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:48.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992338+0000", 2023-01-15T14:33:48.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4734000000000001e-05 2023-01-15T14:33:48.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096001+0000", 2023-01-15T14:33:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10366370499999999 2023-01-15T14:33:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096284+0000", 2023-01-15T14:33:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028221799999999999 2023-01-15T14:33:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:48.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.096742+0000", 2023-01-15T14:33:48.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045808900000000001 2023-01-15T14:33:48.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:48.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097185+0000", 2023-01-15T14:33:48.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044343200000000001 2023-01-15T14:33:48.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:48.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097204+0000", 2023-01-15T14:33:48.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9031000000000001e-05 2023-01-15T14:33:48.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:48.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097299+0000", 2023-01-15T14:33:48.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4642999999999995e-05 2023-01-15T14:33:48.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:48.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098218+0000", 2023-01-15T14:33:48.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00091921899999999998 2023-01-15T14:33:48.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.197647+0000", 2023-01-15T14:33:48.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099428699999999995 2023-01-15T14:33:48.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397618+0000", 2023-01-15T14:33:48.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19997094900000001 2023-01-15T14:33:48.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:48.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397773+0000", 2023-01-15T14:33:48.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015553100000000001 2023-01-15T14:33:48.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.397915+0000", 2023-01-15T14:33:48.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014207099999999999 2023-01-15T14:33:48.275 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.275 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.276 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.276 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:33:48.276 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:48.276 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 263.55560288499998, 2023-01-15T14:33:48.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:33:48.276 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:48.276 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:48.276 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:48.277 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:48.277 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:48.277 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:33:48.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.277 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:48.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:33:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:33:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:33:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:33:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:33:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:33:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:33:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:33:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:33:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:33:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:33:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:33:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:33:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:33:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:33:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:48.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:33:48.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:33:48.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:48.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:33:48.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:33:48.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:33:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:33:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:33:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:33:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:33:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:33:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1080 3.1 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:33:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:33:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 244.78969933900001, 2023-01-15T14:33:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729428600000001, 2023-01-15T14:33:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1080 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758107+0000", 2023-01-15T14:33:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758105+0000", 2023-01-15T14:33:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758117+0000", 2023-01-15T14:33:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2403e-05 2023-01-15T14:33:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758144+0000", 2023-01-15T14:33:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:33:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758161+0000", 2023-01-15T14:33:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7170999999999999e-05 2023-01-15T14:33:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758313+0000", 2023-01-15T14:33:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015197100000000001 2023-01-15T14:33:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.758329+0000", 2023-01-15T14:33:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6460000000000002e-05 2023-01-15T14:33:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937158+0000", 2023-01-15T14:33:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.17882843900000001 2023-01-15T14:33:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937442+0000", 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028461399999999998 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.937869+0000", 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000426334 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938236+0000", 2023-01-15T14:33:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036729800000000001 2023-01-15T14:33:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938249+0000", 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2918000000000001e-05 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938283+0000", 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4530999999999999e-05 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:42.938644+0000", 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036028500000000001 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.064737+0000", 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.126092923 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165200+0000", 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10046282500000001 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165321+0000", 2023-01-15T14:33:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012188099999999999 2023-01-15T14:33:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:43.165401+0000", 2023-01-15T14:33:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9893000000000002e-05 2023-01-15T14:33:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:33:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:33:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 228.24652053899999, 2023-01-15T14:33:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:33:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:33:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:33:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:33:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:33:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:33:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:33:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:33:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:33:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:33:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:33:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:33:48.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:33:48.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:48.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:33:48.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:33:48.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:33:48.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:33:48.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:33:48.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:48.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:33:48.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:33:48.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:48.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:33:48.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:33:48.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:48.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:33:48.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:33:48.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:48.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:33:48.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:33:48.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:33:48.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:33:48.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:33:48.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:33:48.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:48.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:33:48.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:33:48.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:33:48.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 228.246481364, 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:48.700 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:48.701 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:48.701 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:33:48.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.701 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:48.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:48.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:33:48.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:48.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:33:48.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:48.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:33:48.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:48.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:48.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:33:48.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:33:48.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:48.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:33:48.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:33:48.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:48.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:33:48.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:33:48.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:33:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:33:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:33:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:33:48.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:33:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:33:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:48.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:33:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:33:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:33:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:33:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.786 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1373 3.12 3:495d4bbc:::smithi09025264-16:head [write 2873523~591733] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:33:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:33:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 219.854365038, 2023-01-15T14:33:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40949580099999999, 2023-01-15T14:33:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:48.787 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1373 2023-01-15T14:33:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:33:48.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693441+0000", 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693440+0000", 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693808+0000", 2023-01-15T14:33:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000367742 2023-01-15T14:33:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693831+0000", 2023-01-15T14:33:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3213e-05 2023-01-15T14:33:48.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693846+0000", 2023-01-15T14:33:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4909000000000001e-05 2023-01-15T14:33:48.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.695833+0000", 2023-01-15T14:33:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0019869240000000002 2023-01-15T14:33:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696121+0000", 2023-01-15T14:33:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028753300000000003 2023-01-15T14:33:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798577+0000", 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102456808 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.798849+0000", 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027138999999999999 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799155+0000", 2023-01-15T14:33:48.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030594800000000001 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799614+0000", 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045960700000000001 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799737+0000", 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012218900000000001 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799998+0000", 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026129700000000002 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800639+0000", 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064108800000000005 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102023+0000", 2023-01-15T14:33:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30138417200000001 2023-01-15T14:33:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102198+0000", 2023-01-15T14:33:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017453600000000001 2023-01-15T14:33:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102398+0000", 2023-01-15T14:33:48.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020078000000000001 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.102937+0000", 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053874499999999998 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:33:48.798 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 219.85394044399999, 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:48.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:33:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:33:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:33:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:33:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:33:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:33:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:33:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:48.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:33:48.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:33:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:33:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:33:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:33:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:33:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:33:48.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:33:48.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:33:48.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:48.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:33:48.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:33:48.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 219.85388811499999, 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.993 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:48.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:48.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:33:48.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:48.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:33:48.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:48.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:48.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:33:48.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:33:48.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:33:48.999 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:49.000 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 201.25697491599999, 2023-01-15T14:33:49.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:33:49.000 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.000 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:49.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:33:49.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.000 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:33:49.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:49.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:33:49.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:33:49.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:33:49.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:33:49.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:33:49.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:33:49.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:33:49.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.005 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.006 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.006 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.006 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:49.006 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:33:49.007 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:47.592+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:49.007 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:33:49.008 INFO:tasks.rados.rados.0.smithi090.stdout:1329: finishing rollback tid 1 to smithi09025264-10 2023-01-15T14:33:49.008 INFO:tasks.rados.rados.0.smithi090.stdout:1329: finishing rollback tid 2 to smithi09025264-10 2023-01-15T14:33:49.008 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 420 (ObjNum 448 snap 124 seq_num 448) dirty exists 2023-01-15T14:33:49.008 INFO:tasks.rados.rados.0.smithi090.stdout:1329: done (0 left) 2023-01-15T14:33:49.009 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:49.012 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:49.007+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:49.126 INFO:tasks.rados.rados.0.smithi090.stdout:1330: done (0 left) 2023-01-15T14:33:49.127 INFO:tasks.rados.rados.0.smithi090.stdout:1331: rollback oid 20 current snap is 142 2023-01-15T14:33:49.127 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 20 to 127 2023-01-15T14:33:49.127 INFO:tasks.rados.rados.0.smithi090.stdout:1332: rollback oid 29 current snap is 142 2023-01-15T14:33:49.127 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 29 to 111 2023-01-15T14:33:49.127 INFO:tasks.rados.rados.0.smithi090.stdout:1333: write (excl) oid 37 current snap is 142 2023-01-15T14:33:49.127 INFO:tasks.rados.rados.0.smithi090.stdout:1333: seq_num 481 ranges {484167=725008,1730427=353740} 2023-01-15T14:33:49.127 INFO:tasks.rados.rados.0.smithi090.stdout:1333: writing smithi09025264-37 from 484167 to 1209175 tid 1 2023-01-15T14:33:49.127 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:49.128 INFO:tasks.rados.rados.0.smithi090.stdout:1333: writing smithi09025264-37 from 1730427 to 2084167 tid 2 2023-01-15T14:33:49.129 INFO:tasks.rados.rados.0.smithi090.stdout:1334: write oid 8 current snap is 142 2023-01-15T14:33:49.129 INFO:tasks.rados.rados.0.smithi090.stdout:1334: seq_num 482 ranges {500974=400257,1388909=557169} 2023-01-15T14:33:49.129 INFO:tasks.rados.rados.0.smithi090.stdout:1334: writing smithi09025264-8 from 500974 to 901231 tid 1 2023-01-15T14:33:49.129 INFO:tasks.rados.rados.0.smithi090.stdout:1334: writing smithi09025264-8 from 1388909 to 1946078 tid 2 2023-01-15T14:33:49.129 INFO:tasks.rados.rados.0.smithi090.stdout:1335: read oid 28 snap -1 2023-01-15T14:33:49.129 INFO:tasks.rados.rados.0.smithi090.stdout:1335: expect (ObjNum 466 snap 134 seq_num 466) 2023-01-15T14:33:49.129 INFO:tasks.rados.rados.0.smithi090.stdout:1336: read oid 50 snap 127 2023-01-15T14:33:49.129 INFO:tasks.rados.rados.0.smithi090.stdout:1336: expect (ObjNum 328 snap 87 seq_num 328) 2023-01-15T14:33:49.129 INFO:tasks.rados.rados.0.smithi090.stdout:1337: delete oid 34 current snap is 142 2023-01-15T14:33:49.130 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:49.130 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:49.131 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:33:49.131 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:33:49.131 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:33:49.131 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:33:49.131 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: filestore_inject_stall = '0' (not observed, change may require restart) osd_enable_op_tracker = 'true' 2023-01-15T14:33:49.131 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:33:49.131 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:33:49.131 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:33:49.131 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3504 3.62 3.ac97fbe2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:33:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.017457+0000", 2023-01-15T14:33:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.094475923000000003, 2023-01-15T14:33:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094514520000000005, 2023-01-15T14:33:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.132 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3504 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.017457+0000", 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.017457+0000", 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.017456+0000", 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.017464+0000", 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3270000000000001e-06 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.017481+0000", 2023-01-15T14:33:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7136999999999999e-05 2023-01-15T14:33:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.017489+0000", 2023-01-15T14:33:49.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2160000000000006e-06 2023-01-15T14:33:49.144 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.144 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.144 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.144 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3505 3.62 3.ac97fbe2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:33:49.144 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.017511+0000", 2023-01-15T14:33:49.144 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.094422735999999993, 2023-01-15T14:33:49.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094584360000000006, 2023-01-15T14:33:49.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.144 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "no flag points reached", 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3505 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.017511+0000", 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.017511+0000", 2023-01-15T14:33:49.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.017510+0000", 2023-01-15T14:33:49.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:49.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.017513+0000", 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1640000000000001e-06 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.017526+0000", 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2516e-05 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.147 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:49.148 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 2 2023-01-15T14:33:49.148 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:49.148 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:49.120+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:49.149 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:33:49.149 INFO:tasks.rados.rados.0.smithi090.stdout:1335: expect (ObjNum 466 snap 134 seq_num 466) 2023-01-15T14:33:49.228 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:49.223+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:49.295 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:49.326 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:49.326 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:33:49.326 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:33:49.327 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:49.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:33:49.329 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:49.323+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:49.428 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:49.424+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:49.441 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:49.442 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 322.35164314799999, 2023-01-15T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.443 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:49.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:33:49.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:49.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 322.35134335100003, 2023-01-15T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.454 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:33:49.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:33:49.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:33:49.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:33:49.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:33:49.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:33:49.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:33:49.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:33:49.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:49.465 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 322.35131356699998, 2023-01-15T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:33:49.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:33:49.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.472 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 322.221504767, 2023-01-15T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.473 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:33:49.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:33:49.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:33:49.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:33:49.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:49.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:33:49.483 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 322.22144807000001, 2023-01-15T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.485 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:33:49.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:33:49.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:334 3.0 3:0a998d04:::smithi09025264-10:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2023-01-15T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:33:49.492 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 312.57610342599997, 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60680952899999996, 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 334 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860054+0000", 2023-01-15T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860053+0000", 2023-01-15T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860060+0000", 2023-01-15T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.737e-06 2023-01-15T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.860085+0000", 2023-01-15T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5406e-05 2023-01-15T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960265+0000", 2023-01-15T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100179693 2023-01-15T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960419+0000", 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015388899999999999 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:36.960436+0000", 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6803000000000002e-05 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164299+0000", 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203863029 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164468+0000", 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169467 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.164707+0000", 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238545 2023-01-15T14:33:49.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165032+0000", 2023-01-15T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032587700000000002 2023-01-15T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165055+0000", 2023-01-15T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2929e-05 2023-01-15T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.165133+0000", 2023-01-15T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7799999999999994e-05 2023-01-15T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.293773+0000", 2023-01-15T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.128640263 2023-01-15T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.366232+0000", 2023-01-15T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.072458645000000002 2023-01-15T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466697+0000", 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100464489 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466799+0000", 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102874 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:37.466864+0000", 2023-01-15T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4206000000000002e-05 2023-01-15T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:680 3.0 3:07a51d33:::smithi09025264-24:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2023-01-15T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.57481645000001, 2023-01-15T14:33:49.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60351502899999998, 2023-01-15T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 680 2023-01-15T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:49.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861341+0000", 2023-01-15T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861343+0000", 2023-01-15T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0930000000000001e-06 2023-01-15T14:33:49.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861347+0000", 2023-01-15T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.0470000000000004e-06 2023-01-15T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:06.861350+0000", 2023-01-15T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.994e-06 2023-01-15T14:33:49.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260715+0000", 2023-01-15T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39936499399999997 2023-01-15T14:33:49.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.260904+0000", 2023-01-15T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018948700000000001 2023-01-15T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261184+0000", 2023-01-15T14:33:49.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279423 2023-01-15T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261653+0000", 2023-01-15T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046940399999999998 2023-01-15T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261676+0000", 2023-01-15T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2376999999999999e-05 2023-01-15T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.261775+0000", 2023-01-15T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9134000000000003e-05 2023-01-15T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264878+0000", 2023-01-15T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.003103218 2023-01-15T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.264991+0000", 2023-01-15T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113239 2023-01-15T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464495+0000", 2023-01-15T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.199503457 2023-01-15T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464686+0000", 2023-01-15T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019126399999999999 2023-01-15T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:07.464856+0000", 2023-01-15T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170211 2023-01-15T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1224 3.4 3:2029fa71:::smithi09025264-39:head [copy-from ver 35] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:33:49.514 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 235.51423819999999, 2023-01-15T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61616421499999996, 2023-01-15T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.515 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.516 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1224 2023-01-15T14:33:49.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.516 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:33:49.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921919+0000", 2023-01-15T14:33:49.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921918+0000", 2023-01-15T14:33:49.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:49.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921929+0000", 2023-01-15T14:33:49.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1609e-05 2023-01-15T14:33:49.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921948+0000", 2023-01-15T14:33:49.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8953999999999998e-05 2023-01-15T14:33:49.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921957+0000", 2023-01-15T14:33:49.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0559999999999994e-06 2023-01-15T14:33:49.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022124+0000", 2023-01-15T14:33:49.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100166568 2023-01-15T14:33:49.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022255+0000", 2023-01-15T14:33:49.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013088600000000001 2023-01-15T14:33:49.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,2", 2023-01-15T14:33:49.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.231637+0000", 2023-01-15T14:33:49.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.209382444 2023-01-15T14:33:49.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:49.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232245+0000", 2023-01-15T14:33:49.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060786499999999997 2023-01-15T14:33:49.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:49.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232261+0000", 2023-01-15T14:33:49.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6050999999999999e-05 2023-01-15T14:33:49.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:49.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235560+0000", 2023-01-15T14:33:49.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0032986309999999998 2023-01-15T14:33:49.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:49.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.235615+0000", 2023-01-15T14:33:49.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5485999999999997e-05 2023-01-15T14:33:49.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.239698+0000", 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0040830240000000002 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537864+0000", 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29816611199999998 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.537967+0000", 2023-01-15T14:33:49.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000103169 2023-01-15T14:33:49.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538084+0000", 2023-01-15T14:33:49.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116165 2023-01-15T14:33:49.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.526 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.527 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:33:49.527 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:33:49.527 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 235.51417949, 2023-01-15T14:33:49.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:33:49.528 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.528 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:49.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.528 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:33:49.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.529 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:33:49.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:33:49.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:33:49.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:49.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:33:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:33:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:33:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:33:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:33:49.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:33:49.535 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.536 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.536 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:49.536 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:33:49.536 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 231.55775999700001, 2023-01-15T14:33:49.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:33:49.536 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.536 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:49.537 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.537 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.537 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.537 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:33:49.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.537 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:33:49.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:33:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:33:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:33:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:33:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:49.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:33:49.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:33:49.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:33:49.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:33:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:33:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:33:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:33:49.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 231.500262314, 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:49.543 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:33:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:33:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:33:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:33:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:33:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:33:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:33:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:33:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:33:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:33:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:33:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:33:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:33:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:33:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:33:49.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:33:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:33:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:33:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:33:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:33:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:33:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:49.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.553 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:33:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 210.13161080899999, 2023-01-15T14:33:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:33:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.554 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:33:49.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:33:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:33:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:33:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:33:49.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:33:49.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:33:49.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:33:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:33:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:33:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:33:49.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:33:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:33:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:49.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:33:49.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:33:49.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:49.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:33:49.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:33:49.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:33:49.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:33:49.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:33:49.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:33:49.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:49.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:33:49.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:33:49.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:33:49.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:33:49.566 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.566 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.566 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.567 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:49.567 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:33:49.567 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 210.13150816800001, 2023-01-15T14:33:49.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:33:49.567 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.567 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:49.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.568 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:33:49.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.568 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:33:49.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:33:49.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:33:49.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:33:49.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:33:49.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:33:49.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:33:49.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:33:49.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:49.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:33:49.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:33:49.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:33:49.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:33:49.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:33:49.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:33:49.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:33:49.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:33:49.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:33:49.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:33:49.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:33:49.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:33:49.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.577 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.577 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:33:49.578 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:49.578 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.530526173, 2023-01-15T14:33:49.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:33:49.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.578 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:49.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:33:49.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.580 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:49.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:49.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:33:49.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:49.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:33:49.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:33:49.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:33:49.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:33:49.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:33:49.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:33:49.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:33:49.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:33:49.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:49.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:33:49.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:33:49.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:33:49.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:33:49.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:33:49.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:33:49.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:33:49.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:33:49.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:33:49.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:33:49.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:33:49.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:33:49.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:49.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:33:49.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:33:49.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:33:49.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:33:49.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:33:49.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:49.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:33:49.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:33:49.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:33:49.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:33:49.593 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.593 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.593 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.593 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:33:49.593 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:33:49.593 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 165.530500749, 2023-01-15T14:33:49.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:33:49.594 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.594 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:49.594 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.594 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.594 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.595 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:33:49.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.595 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:33:49.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:33:49.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:33:49.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:49.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:33:49.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:33:49.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:33:49.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:33:49.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:33:49.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:33:49.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:33:49.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:33:49.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:49.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:33:49.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:33:49.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:33:49.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:33:49.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:33:49.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:33:49.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:33:49.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:33:49.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:33:49.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:33:49.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:49.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:33:49.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:33:49.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:49.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:33:49.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:33:49.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:49.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:33:49.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:33:49.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:49.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:33:49.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:33:49.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:49.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:33:49.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:33:49.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:33:49.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:33:49.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:33:49.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:33:49.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:49.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:33:49.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:33:49.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:33:49.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:33:49.607 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.607 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.607 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.608 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2301 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e216)", 2023-01-15T14:33:49.608 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:33:49.608 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 126.24062628199999, 2023-01-15T14:33:49.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61754771200000003, 2023-01-15T14:33:49.608 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.608 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2301 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:33:49.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195530+0000", 2023-01-15T14:33:49.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195535+0000", 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3660000000000002e-06 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195546+0000", 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1062e-05 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195551+0000", 2023-01-15T14:33:49.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2023-01-15T14:33:49.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195999+0000", 2023-01-15T14:33:49.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044799800000000002 2023-01-15T14:33:49.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.196106+0000", 2023-01-15T14:33:49.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106172 2023-01-15T14:33:49.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510631+0000", 2023-01-15T14:33:49.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31452575999999999 2023-01-15T14:33:49.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510875+0000", 2023-01-15T14:33:49.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024415199999999998 2023-01-15T14:33:49.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812445+0000", 2023-01-15T14:33:49.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30156923000000002 2023-01-15T14:33:49.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812727+0000", 2023-01-15T14:33:49.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028227800000000001 2023-01-15T14:33:49.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.813079+0000", 2023-01-15T14:33:49.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199599999999998 2023-01-15T14:33:49.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.617 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.617 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2667 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e259)", 2023-01-15T14:33:49.618 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:49.618 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 83.551736387000005, 2023-01-15T14:33:49.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60917505800000005, 2023-01-15T14:33:49.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.618 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:49.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.619 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.619 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2667 2023-01-15T14:33:49.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.619 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:49.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:49.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884421+0000", 2023-01-15T14:33:49.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967296 2023-01-15T14:33:49.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884423+0000", 2023-01-15T14:33:49.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.905e-06 2023-01-15T14:33:49.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884427+0000", 2023-01-15T14:33:49.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7689999999999998e-06 2023-01-15T14:33:49.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:25.884429+0000", 2023-01-15T14:33:49.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5509999999999998e-06 2023-01-15T14:33:49.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.080096+0000", 2023-01-15T14:33:49.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19566640099999999 2023-01-15T14:33:49.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.080290+0000", 2023-01-15T14:33:49.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194346 2023-01-15T14:33:49.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.275924+0000", 2023-01-15T14:33:49.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19563417299999999 2023-01-15T14:33:49.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.276184+0000", 2023-01-15T14:33:49.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025966800000000001 2023-01-15T14:33:49.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493228+0000", 2023-01-15T14:33:49.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21704441599999999 2023-01-15T14:33:49.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493378+0000", 2023-01-15T14:33:49.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014980000000000001 2023-01-15T14:33:49.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:26.493596+0000", 2023-01-15T14:33:49.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021825300000000001 2023-01-15T14:33:49.628 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.628 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.628 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2749 3.3d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:33:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.514741506000007, 2023-01-15T14:33:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.69206326600000001, 2023-01-15T14:33:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.629 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2749 2023-01-15T14:33:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:33:49.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921418+0000", 2023-01-15T14:33:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1670000000000002e-06 2023-01-15T14:33:49.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921426+0000", 2023-01-15T14:33:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0569999999999993e-06 2023-01-15T14:33:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921429+0000", 2023-01-15T14:33:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9629999999999998e-06 2023-01-15T14:33:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107302+0000", 2023-01-15T14:33:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.185873327 2023-01-15T14:33:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107657+0000", 2023-01-15T14:33:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035504299999999998 2023-01-15T14:33:49.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212694+0000", 2023-01-15T14:33:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105037095 2023-01-15T14:33:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212936+0000", 2023-01-15T14:33:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241492 2023-01-15T14:33:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613125+0000", 2023-01-15T14:33:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40018927900000001 2023-01-15T14:33:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613294+0000", 2023-01-15T14:33:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016838000000000001 2023-01-15T14:33:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613479+0000", 2023-01-15T14:33:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018577599999999999 2023-01-15T14:33:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3217 3.25 3:a75958b1:::smithi09025264-6:head [copy-from ver 302] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2023-01-15T14:33:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:18.016740+0000", 2023-01-15T14:33:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.419417557999999, 2023-01-15T14:33:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61560662799999999, 2023-01-15T14:33:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.640 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3217 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016740+0000", 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016740+0000", 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016736+0000", 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999962 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016758+0000", 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1503999999999999e-05 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016800+0000", 2023-01-15T14:33:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.2347e-05 2023-01-15T14:33:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016818+0000", 2023-01-15T14:33:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8539e-05 2023-01-15T14:33:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016925+0000", 2023-01-15T14:33:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106272 2023-01-15T14:33:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:33:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016936+0000", 2023-01-15T14:33:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0973e-05 2023-01-15T14:33:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.119787+0000", 2023-01-15T14:33:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102851145 2023-01-15T14:33:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.119952+0000", 2023-01-15T14:33:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016543999999999999 2023-01-15T14:33:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:33:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.425942+0000", 2023-01-15T14:33:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30598952000000001 2023-01-15T14:33:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.427485+0000", 2023-01-15T14:33:49.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015428219999999999 2023-01-15T14:33:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.427511+0000", 2023-01-15T14:33:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6732000000000001e-05 2023-01-15T14:33:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:49.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.428510+0000", 2023-01-15T14:33:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00099899899999999998 2023-01-15T14:33:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.428610+0000", 2023-01-15T14:33:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9681999999999999e-05 2023-01-15T14:33:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.534081+0000", 2023-01-15T14:33:49.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105470913 2023-01-15T14:33:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.632153+0000", 2023-01-15T14:33:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.098072101999999994 2023-01-15T14:33:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.632241+0000", 2023-01-15T14:33:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8015000000000003e-05 2023-01-15T14:33:49.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.632347+0000", 2023-01-15T14:33:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105518 2023-01-15T14:33:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3218 3.25 3:a75958b1:::smithi09025264-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e309)", 2023-01-15T14:33:49.650 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:18.016851+0000", 2023-01-15T14:33:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 31.419306552999998, 2023-01-15T14:33:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61576188899999995, 2023-01-15T14:33:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3218 2023-01-15T14:33:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.651 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016851+0000", 2023-01-15T14:33:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016851+0000", 2023-01-15T14:33:49.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016850+0000", 2023-01-15T14:33:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016856+0000", 2023-01-15T14:33:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5529999999999998e-06 2023-01-15T14:33:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016868+0000", 2023-01-15T14:33:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1629e-05 2023-01-15T14:33:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.016876+0000", 2023-01-15T14:33:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3299999999999999e-06 2023-01-15T14:33:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.017100+0000", 2023-01-15T14:33:49.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022403399999999999 2023-01-15T14:33:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.017115+0000", 2023-01-15T14:33:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4384e-05 2023-01-15T14:33:49.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.120221+0000", 2023-01-15T14:33:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103106718 2023-01-15T14:33:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.120300+0000", 2023-01-15T14:33:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8108000000000005e-05 2023-01-15T14:33:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.427856+0000", 2023-01-15T14:33:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30755611799999999 2023-01-15T14:33:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:49.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.428140+0000", 2023-01-15T14:33:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028438699999999999 2023-01-15T14:33:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.632400+0000", 2023-01-15T14:33:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20425969399999999 2023-01-15T14:33:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.632507+0000", 2023-01-15T14:33:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107686 2023-01-15T14:33:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:49.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:18.632613+0000", 2023-01-15T14:33:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000105488 2023-01-15T14:33:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.663 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.663 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:49.666 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:33:49.667 INFO:tasks.rados.rados.0.smithi090.stdout:1336: expect (ObjNum 328 snap 87 seq_num 328) 2023-01-15T14:33:49.667 INFO:tasks.rados.rados.0.smithi090.stdout:1331: finishing rollback tid 0 to smithi09025264-20 2023-01-15T14:33:49.667 INFO:tasks.rados.rados.0.smithi090.stdout:1331: finishing rollback tid 1 to smithi09025264-20 2023-01-15T14:33:49.668 INFO:tasks.rados.rados.0.smithi090.stdout:1331: finishing rollback tid 2 to smithi09025264-20 2023-01-15T14:33:49.668 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 258 (ObjNum 451 snap 126 seq_num 451) dirty exists 2023-01-15T14:33:49.668 INFO:tasks.rados.rados.0.smithi090.stdout:1333: finishing write tid 1 to smithi09025264-37 2023-01-15T14:33:49.668 INFO:tasks.rados.rados.0.smithi090.stdout:1334: finishing write tid 1 to smithi09025264-8 2023-01-15T14:33:49.668 INFO:tasks.rados.rados.0.smithi090.stdout:1334: finishing write tid 2 to smithi09025264-8 2023-01-15T14:33:49.668 INFO:tasks.rados.rados.0.smithi090.stdout:1333: finishing write tid 2 to smithi09025264-37 2023-01-15T14:33:49.668 INFO:tasks.rados.rados.0.smithi090.stdout:1334: finishing write tid 3 to smithi09025264-8 2023-01-15T14:33:49.669 INFO:tasks.rados.rados.0.smithi090.stdout:1334: finishing write tid 4 to smithi09025264-8 2023-01-15T14:33:49.669 INFO:tasks.rados.rados.0.smithi090.stdout:1334: oid 8 updating version 0 to 308 2023-01-15T14:33:49.669 INFO:tasks.rados.rados.0.smithi090.stdout:1334: oid 8 version 308 is already newer than 306 2023-01-15T14:33:49.669 INFO:tasks.rados.rados.0.smithi090.stdout:1334: oid 8 version 308 is already newer than 307 2023-01-15T14:33:49.669 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 308 (ObjNum 482 snap 142 seq_num 482) dirty exists 2023-01-15T14:33:49.669 INFO:tasks.rados.rados.0.smithi090.stdout:1334: left oid 8 (ObjNum 482 snap 142 seq_num 482) 2023-01-15T14:33:49.670 INFO:tasks.rados.rados.0.smithi090.stdout:1331: done (6 left) 2023-01-15T14:33:49.670 INFO:tasks.rados.rados.0.smithi090.stdout:1334: done (5 left) 2023-01-15T14:33:49.670 INFO:tasks.rados.rados.0.smithi090.stdout:1335: done (4 left) 2023-01-15T14:33:49.670 INFO:tasks.rados.rados.0.smithi090.stdout:1336: done (3 left) 2023-01-15T14:33:49.670 INFO:tasks.rados.rados.0.smithi090.stdout:1337: done (2 left) 2023-01-15T14:33:49.670 INFO:tasks.rados.rados.0.smithi090.stdout:1338: read oid 7 snap -1 2023-01-15T14:33:49.670 INFO:tasks.rados.rados.0.smithi090.stdout:1338: expect deleted 2023-01-15T14:33:49.671 INFO:tasks.rados.rados.0.smithi090.stdout:1339: delete oid 6 current snap is 142 2023-01-15T14:33:49.671 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:49.660+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:49.767 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:49.763+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:49.816 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:49.936 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:49.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.62 340/337 [PushOp(3:47dfe935:::smithi09025264-29:head, version: 323'161, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:47dfe935:::smithi09025264-29:head@323'161, size: 2453855, copy_subset: [0~2453855], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:33:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.617003+0000", 2023-01-15T14:33:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.19730805900000001, 2023-01-15T14:33:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19738103700000001, 2023-01-15T14:33:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.617003+0000", 2023-01-15T14:33:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.617003+0000", 2023-01-15T14:33:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.617001+0000", 2023-01-15T14:33:49.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.617745+0000", 2023-01-15T14:33:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00074394200000000004 2023-01-15T14:33:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.617800+0000", 2023-01-15T14:33:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4882000000000001e-05 2023-01-15T14:33:49.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.717902+0000", 2023-01-15T14:33:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10010227300000001 2023-01-15T14:33:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3525 3.65 e340/337)", 2023-01-15T14:33:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.717948+0000", 2023-01-15T14:33:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.096362616999999998, 2023-01-15T14:33:49.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.09651092, 2023-01-15T14:33:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "no flag points reached", 2023-01-15T14:33:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.717948+0000", 2023-01-15T14:33:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.717948+0000", 2023-01-15T14:33:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.717947+0000", 2023-01-15T14:33:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:49.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.717960+0000", 2023-01-15T14:33:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2386e-05 2023-01-15T14:33:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.717977+0000", 2023-01-15T14:33:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6965000000000001e-05 2023-01-15T14:33:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:49.944 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:49.945 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 2 2023-01-15T14:33:49.945 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:49.946 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:33:49.947 INFO:tasks.rados.rados.0.smithi090.stdout:1333: finishing write tid 3 to smithi09025264-37 2023-01-15T14:33:49.947 INFO:tasks.rados.rados.0.smithi090.stdout:1338: done (3 left) 2023-01-15T14:33:49.947 INFO:tasks.rados.rados.0.smithi090.stdout:1339: done (2 left) 2023-01-15T14:33:49.947 INFO:tasks.rados.rados.0.smithi090.stdout:1340: delete oid 35 current snap is 142 2023-01-15T14:33:49.947 INFO:tasks.rados.rados.0.smithi090.stdout:1333: finishing write tid 4 to smithi09025264-37 2023-01-15T14:33:49.947 INFO:tasks.rados.rados.0.smithi090.stdout:1333: oid 37 updating version 0 to 383 2023-01-15T14:33:49.948 INFO:tasks.rados.rados.0.smithi090.stdout:1333: oid 37 version 383 is already newer than 382 2023-01-15T14:33:49.948 INFO:tasks.rados.rados.0.smithi090.stdout:1333: oid 37 version 383 is already newer than 381 2023-01-15T14:33:49.948 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 383 (ObjNum 481 snap 142 seq_num 481) dirty exists 2023-01-15T14:33:49.948 INFO:tasks.rados.rados.0.smithi090.stdout:1333: left oid 37 (ObjNum 481 snap 142 seq_num 481) 2023-01-15T14:33:49.948 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:49.943+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:50.047 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:50.043+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:50.065 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:33:50.066 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:33:50.066 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:50.075 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:33:50.115 INFO:tasks.rados.rados.0.smithi090.stdout:1333: done (2 left) 2023-01-15T14:33:50.115 INFO:tasks.rados.rados.0.smithi090.stdout:1340: done (1 left) 2023-01-15T14:33:50.115 INFO:tasks.rados.rados.0.smithi090.stdout:1341: write oid 24 current snap is 142 2023-01-15T14:33:50.115 INFO:tasks.rados.rados.0.smithi090.stdout:1341: seq_num 486 ranges {568202=701253,1906484=261718} 2023-01-15T14:33:50.128 INFO:tasks.rados.rados.0.smithi090.stdout:1341: writing smithi09025264-24 from 568202 to 1269455 tid 1 2023-01-15T14:33:50.135 INFO:tasks.rados.rados.0.smithi090.stdout:1341: writing smithi09025264-24 from 1906484 to 2168202 tid 2 2023-01-15T14:33:50.135 INFO:tasks.rados.rados.0.smithi090.stdout:1342: snap_remove snap 138 2023-01-15T14:33:50.148 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:50.144+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:50.179 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.179 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:33:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 195.28058508399999, 2023-01-15T14:33:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:33:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:33:50.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:33:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:33:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:33:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:33:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:33:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:33:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:33:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:33:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:33:50.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:33:50.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:33:50.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:33:50.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:33:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:33:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:33:50.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:33:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:33:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:33:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:33:50.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:33:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:33:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:33:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:33:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:33:50.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:33:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:33:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:33:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:33:50.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:33:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:33:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:33:50.190 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 195.280534141, 2023-01-15T14:33:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:33:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:33:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:33:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:33:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:33:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:33:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:33:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:33:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:33:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:33:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:33:50.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:33:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:33:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:33:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:33:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:33:50.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:50.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:33:50.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:33:50.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:33:50.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:33:50.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:33:50.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:33:50.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:33:50.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:33:50.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.197 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 192.33963489999999, 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.199 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:33:50.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.199 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:50.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:33:50.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:33:50.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:33:50.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:33:50.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:33:50.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:33:50.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:33:50.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:33:50.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:33:50.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:50.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:33:50.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:33:50.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:33:50.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:33:50.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:33:50.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:33:50.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:33:50.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:33:50.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:33:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:33:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:33:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:33:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:33:50.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:33:50.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:33:50.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:33:50.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:33:50.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:33:50.208 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.208 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.208 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.208 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:33:50.208 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:50.208 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 190.26586072699999, 2023-01-15T14:33:50.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:33:50.209 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.209 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.209 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.209 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.209 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.209 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:33:50.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.209 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:33:50.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:50.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:33:50.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:33:50.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:33:50.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:33:50.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:33:50.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:33:50.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:33:50.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:33:50.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:50.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:33:50.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:33:50.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:33:50.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:33:50.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:33:50.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:33:50.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:50.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:33:50.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:33:50.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:33:50.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:33:50.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:33:50.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:33:50.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:33:50.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:33:50.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:50.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:33:50.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:33:50.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:33:50.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:33:50.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:33:50.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:33:50.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:33:50.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:33:50.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:33:50.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:33:50.218 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.218 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.218 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.218 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:33:50.218 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:50.219 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 166.37190530699999, 2023-01-15T14:33:50.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:33:50.219 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.219 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:50.219 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.219 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.219 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.219 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:33:50.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.219 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:50.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:33:50.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:33:50.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:50.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:33:50.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:33:50.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:33:50.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:33:50.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:33:50.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:33:50.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:33:50.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:33:50.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:50.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:33:50.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:33:50.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:33:50.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:33:50.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:33:50.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:33:50.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:33:50.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:33:50.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.224 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.225 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.225 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:50.225 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:50.225 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.48849862599999, 2023-01-15T14:33:50.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:33:50.225 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.225 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.225 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.225 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.226 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:33:50.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.226 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:50.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:33:50.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:33:50.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:50.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:33:50.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:33:50.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:33:50.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:33:50.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:33:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:33:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:33:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:33:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:33:50.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:50.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:33:50.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:33:50.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:33:50.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:33:50.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:33:50.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:33:50.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:33:50.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:33:50.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:50.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:33:50.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:33:50.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:33:50.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:33:50.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:33:50.234 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.234 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.48509150300001, 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.235 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:33:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:33:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:33:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:33:50.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:33:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:33:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:33:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:33:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:33:50.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:33:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:33:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:33:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:33:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:50.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:33:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:33:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:33:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:33:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:33:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:33:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:33:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:33:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:33:50.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:33:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:33:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:33:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:33:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:33:50.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:33:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:33:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:33:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:33:50.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.48487589800001, 2023-01-15T14:33:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:33:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.245 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:33:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:50.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:33:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:33:50.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:33:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:33:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:33:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:33:50.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:33:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:33:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:33:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:33:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:33:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:33:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:33:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:33:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:33:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:33:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:33:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:33:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:33:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:33:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:33:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:33:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:33:50.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:33:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:33:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:33:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:33:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:33:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:33:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:33:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:50.255 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.48476882400001, 2023-01-15T14:33:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:33:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:33:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:33:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:33:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:33:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:33:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:33:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:33:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:33:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:33:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:33:50.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:33:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:33:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:33:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:33:50.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:33:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:33:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:33:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:33:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:33:50.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:33:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:33:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:33:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:33:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.262 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 136.35452977400001, 2023-01-15T14:33:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:33:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.263 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:33:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:33:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:33:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:33:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:33:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:33:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:33:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:33:50.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:33:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:33:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:33:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:33:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:33:50.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:33:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:33:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:33:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:33:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:33:50.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:33:50.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:33:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:33:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:33:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:33:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 136.35442523699999, 2023-01-15T14:33:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:33:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.273 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:33:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:33:50.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:33:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:33:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:33:50.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:33:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:33:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:33:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:33:50.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:50.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:33:50.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:33:50.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:33:50.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:33:50.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.280 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.280 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:33:50.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:33:50.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:33:50.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:33:50.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:50.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:33:50.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:33:50.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:33:50.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:33:50.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:33:50.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:33:50.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:33:50.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:33:50.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:50.284 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 136.354355306, 2023-01-15T14:33:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:33:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:33:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:33:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:33:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:33:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:33:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:33:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:33:50.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:33:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:33:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:33:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:33:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:33:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:33:50.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:33:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:33:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:33:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:33:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:33:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:33:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:33:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:33:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:33:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:33:50.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:33:50.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2780 3.38 e265/232 3:1eaad226:::smithi09025264-49:head v 265'279, mlcod=261'274)", 2023-01-15T14:33:50.296 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 78.183184179999998, 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32005591799999999, 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991860+0000", 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993927+0000", 2023-01-15T14:33:50.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020673229999999998 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993946+0000", 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9429999999999999e-05 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.094039+0000", 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092898 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285071+0000", 2023-01-15T14:33:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.191031904 2023-01-15T14:33:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285106+0000", 2023-01-15T14:33:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5227e-05 2023-01-15T14:33:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286323+0000", 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012163829999999999 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286335+0000", 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2224e-05 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287218+0000", 2023-01-15T14:33:50.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088265000000000001 2023-01-15T14:33:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287250+0000", 2023-01-15T14:33:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2209000000000003e-05 2023-01-15T14:33:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.311917+0000", 2023-01-15T14:33:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024667062999999999 2023-01-15T14:33:50.302 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2868 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'278, mlcod=264'275)", 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.181293238999999, 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31268252400000002, 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993751+0000", 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993762+0000", 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0822000000000001e-05 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993777+0000", 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5540999999999999e-05 2023-01-15T14:33:50.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993787+0000", 2023-01-15T14:33:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0173999999999999e-05 2023-01-15T14:33:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290198+0000", 2023-01-15T14:33:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29641022099999997 2023-01-15T14:33:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290212+0000", 2023-01-15T14:33:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4214e-05 2023-01-15T14:33:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290300+0000", 2023-01-15T14:33:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8300000000000005e-05 2023-01-15T14:33:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290312+0000", 2023-01-15T14:33:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2344000000000001e-05 2023-01-15T14:33:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290357+0000", 2023-01-15T14:33:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4367999999999999e-05 2023-01-15T14:33:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290389+0000", 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1983000000000002e-05 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.306434+0000", 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016045667 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.309 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:33:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.181243813999998, 2023-01-15T14:33:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:33:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:33:50.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:33:50.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.177432127000003, 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:33:50.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:33:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:33:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:33:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:33:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:33:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:33:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:33:50.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:33:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:33:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:33:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:33:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:33:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.555 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3120 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'290, mlcod=0'0)", 2023-01-15T14:33:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:33:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.935618851000001, 2023-01-15T14:33:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39853714000000001, 2023-01-15T14:33:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.556 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:33:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:33:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239424+0000", 2023-01-15T14:33:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239747+0000", 2023-01-15T14:33:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032243300000000001 2023-01-15T14:33:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239784+0000", 2023-01-15T14:33:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7051000000000002e-05 2023-01-15T14:33:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239802+0000", 2023-01-15T14:33:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8250999999999999e-05 2023-01-15T14:33:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536033+0000", 2023-01-15T14:33:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29623121499999999 2023-01-15T14:33:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536081+0000", 2023-01-15T14:33:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2023-01-15T14:33:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536800+0000", 2023-01-15T14:33:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071948499999999996 2023-01-15T14:33:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536817+0000", 2023-01-15T14:33:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6228000000000001e-05 2023-01-15T14:33:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537048+0000", 2023-01-15T14:33:50.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023136200000000001 2023-01-15T14:33:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637902+0000", 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10085403900000001 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637963+0000", 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1310999999999996e-05 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.563 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3121 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'291, mlcod=0'0)", 2023-01-15T14:33:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:33:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.933623335, 2023-01-15T14:33:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659573799999998, 2023-01-15T14:33:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:33:50.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241420+0000", 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241854+0000", 2023-01-15T14:33:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043330099999999998 2023-01-15T14:33:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241878+0000", 2023-01-15T14:33:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4559e-05 2023-01-15T14:33:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241892+0000", 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039999999999999e-05 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536863+0000", 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29497073400000001 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536887+0000", 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3683999999999999e-05 2023-01-15T14:33:50.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537124+0000", 2023-01-15T14:33:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023729900000000001 2023-01-15T14:33:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537136+0000", 2023-01-15T14:33:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2278e-05 2023-01-15T14:33:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537345+0000", 2023-01-15T14:33:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020865200000000001 2023-01-15T14:33:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637981+0000", 2023-01-15T14:33:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063575700000001 2023-01-15T14:33:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638018+0000", 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6826999999999998e-05 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3122 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'292, mlcod=0'0)", 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.931727494999997, 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39623551600000001, 2023-01-15T14:33:50.570 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243316+0000", 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243833+0000", 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051703600000000001 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243863+0000", 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243879+0000", 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6263999999999999e-05 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637317+0000", 2023-01-15T14:33:50.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39343783199999999 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637356+0000", 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9579000000000002e-05 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637830+0000", 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047326399999999999 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637885+0000", 2023-01-15T14:33:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5074999999999997e-05 2023-01-15T14:33:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638185+0000", 2023-01-15T14:33:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029993799999999997 2023-01-15T14:33:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638337+0000", 2023-01-15T14:33:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015208299999999999 2023-01-15T14:33:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.639553+0000", 2023-01-15T14:33:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001216389 2023-01-15T14:33:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3123 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'293, mlcod=0'0)", 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 40.929983467, 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.395733102, 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:33:50.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245060+0000", 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245075+0000", 2023-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4459e-05 2023-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245107+0000", 2023-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1795000000000003e-05 2023-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245121+0000", 2023-01-15T14:33:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4117000000000001e-05 2023-01-15T14:33:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638063+0000", 2023-01-15T14:33:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39294204300000002 2023-01-15T14:33:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638097+0000", 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4385999999999999e-05 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638299+0000", 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020133099999999999 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:50.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638318+0000", 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514999999999999e-05 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638386+0000", 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8089000000000005e-05 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638430+0000", 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.439e-05 2023-01-15T14:33:50.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.640795+0000", 2023-01-15T14:33:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002364274 2023-01-15T14:33:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:50.583 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:50.583 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:50.584 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:33:50.584 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2023-01-15T14:33:50.585 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:33:50.586 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:50.580+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:50.686 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:50.683+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:50.762 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:50.762 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:50.762 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:33:50.762 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:50.763 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:33:50.786 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:50.780+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:50.861 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:50.861 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:33:50.861 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:33:50.861 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:33:50.861 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:50.873 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:33:50.915 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:50.912+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:50.989 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:50.987+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:51.008 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:33:51.019 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:51.020 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:33:51.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:33:51.020 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:51.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.020 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:51.020 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:33:51.020 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 323.887275778, 2023-01-15T14:33:51.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:33:51.020 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.021 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:51.021 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.021 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.021 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.021 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:33:51.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.021 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:33:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:33:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:33:51.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:33:51.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:33:51.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:33:51.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:33:51.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:33:51.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:33:51.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:33:51.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:33:51.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:33:51.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:33:51.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:33:51.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:33:51.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:33:51.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:51.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:33:51.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:51.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:33:51.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:33:51.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:33:51.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:33:51.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:33:51.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:33:51.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:51.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:33:51.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:33:51.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:33:51.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:33:51.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.030 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.030 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:51.030 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:33:51.030 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 323.88687244200003, 2023-01-15T14:33:51.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:33:51.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:33:51.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:33:51.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:51.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:33:51.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:33:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:33:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:33:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:33:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:33:51.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:33:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:33:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:33:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:33:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:33:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:51.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:33:51.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:51.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:33:51.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:33:51.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:33:51.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:33:51.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 323.78664882200002, 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.040 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:33:51.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:33:51.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:51.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:33:51.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:33:51.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:33:51.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:33:51.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:33:51.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:33:51.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:33:51.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:33:51.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:33:51.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:33:51.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:33:51.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.046 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 233.093333597, 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.048 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:33:51.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.048 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:33:51.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:33:51.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:33:51.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:33:51.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:33:51.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:33:51.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.052 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.053 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:51.053 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:33:51.053 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 233.08687224299999, 2023-01-15T14:33:51.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:33:51.053 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.053 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:33:51.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.053 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:33:51.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.054 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:33:51.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:33:51.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:33:51.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:51.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:33:51.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:33:51.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:33:51.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:33:51.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:33:51.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:33:51.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:33:51.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:33:51.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:33:51.059 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.059 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.059 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.059 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:51.059 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:33:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 232.98645031800001, 2023-01-15T14:33:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:33:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:33:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:33:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:33:51.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:33:51.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:33:51.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:33:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:33:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:33:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:33:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:33:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:33:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:33:51.065 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 232.986112832, 2023-01-15T14:33:51.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:33:51.066 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.066 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:33:51.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.066 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:33:51.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.066 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:33:51.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:51.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:33:51.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:33:51.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:33:51.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:33:51.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:51.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:33:51.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:33:51.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:33:51.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:33:51.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 232.98605693600001, 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.071 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:33:51.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.072 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.072 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:33:51.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.072 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:33:51.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:33:51.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:33:51.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:51.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:33:51.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:33:51.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:33:51.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:33:51.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:51.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:33:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:33:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:33:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:33:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:33:51.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:33:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:33:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:33:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 191.15280651800001, 2023-01-15T14:33:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:33:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.077 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:33:51.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:33:51.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:33:51.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:33:51.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:33:51.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:33:51.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:33:51.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:33:51.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:51.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:33:51.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:33:51.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:33:51.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:33:51.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:33:51.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:33:51.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:33:51.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:33:51.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.085 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2207 3.5 3:a1c47f99:::smithi09025264-2:head [write 2548663~763022] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:51.086 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:33:51.086 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.17804878999999, 2023-01-15T14:33:51.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79657873300000004, 2023-01-15T14:33:51.086 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.086 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:51.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.087 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2207 2023-01-15T14:33:51.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.087 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:33:51.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837590+0000", 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837960+0000", 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037055900000000001 2023-01-15T14:33:51.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837980+0000", 2023-01-15T14:33:51.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9953e-05 2023-01-15T14:33:51.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837988+0000", 2023-01-15T14:33:51.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2800000000000003e-06 2023-01-15T14:33:51.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927512+0000", 2023-01-15T14:33:51.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089523563 2023-01-15T14:33:51.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927645+0000", 2023-01-15T14:33:51.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013285799999999999 2023-01-15T14:33:51.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225466+0000", 2023-01-15T14:33:51.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29782133100000002 2023-01-15T14:33:51.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225553+0000", 2023-01-15T14:33:51.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6866000000000005e-05 2023-01-15T14:33:51.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:33:51.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225708+0000", 2023-01-15T14:33:51.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015455199999999999 2023-01-15T14:33:51.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:51.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225971+0000", 2023-01-15T14:33:51.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026317099999999998 2023-01-15T14:33:51.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226027+0000", 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6601000000000002e-05 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226470+0000", 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044267500000000001 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227372+0000", 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090215200000000003 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429314+0000", 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20194183499999999 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.633999+0000", 2023-01-15T14:33:51.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20468505400000001 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634075+0000", 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634170+0000", 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4821999999999994e-05 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.095 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2208 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (56) in=7b,truncate 3395047] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.177626197, 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79650439900000003, 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.097 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2208 2023-01-15T14:33:51.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.097 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:33:51.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:33:51.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:33:51.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:51.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838018+0000", 2023-01-15T14:33:51.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2549999999999997e-06 2023-01-15T14:33:51.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838031+0000", 2023-01-15T14:33:51.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2717999999999999e-05 2023-01-15T14:33:51.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838036+0000", 2023-01-15T14:33:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2240000000000003e-06 2023-01-15T14:33:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927676+0000", 2023-01-15T14:33:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089639620000000003 2023-01-15T14:33:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927804+0000", 2023-01-15T14:33:51.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012832300000000001 2023-01-15T14:33:51.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226096+0000", 2023-01-15T14:33:51.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29829207400000002 2023-01-15T14:33:51.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226347+0000", 2023-01-15T14:33:51.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025084800000000002 2023-01-15T14:33:51.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226750+0000", 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040343200000000001 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227147+0000", 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039682900000000002 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:51.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227600+0000", 2023-01-15T14:33:51.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045321199999999998 2023-01-15T14:33:51.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:51.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227859+0000", 2023-01-15T14:33:51.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259041 2023-01-15T14:33:51.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:51.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227895+0000", 2023-01-15T14:33:51.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5710999999999997e-05 2023-01-15T14:33:51.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429405+0000", 2023-01-15T14:33:51.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201509415 2023-01-15T14:33:51.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634300+0000", 2023-01-15T14:33:51.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20489528200000001 2023-01-15T14:33:51.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:51.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634409+0000", 2023-01-15T14:33:51.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109524 2023-01-15T14:33:51.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634518+0000", 2023-01-15T14:33:51.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108548 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2209 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.17759738199999, 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79724995499999995, 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:51.107 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.108 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2209 2023-01-15T14:33:51.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.108 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:33:51.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:33:51.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:33:51.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:51.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838044+0000", 2023-01-15T14:33:51.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6120000000000001e-06 2023-01-15T14:33:51.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838056+0000", 2023-01-15T14:33:51.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1236000000000001e-05 2023-01-15T14:33:51.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838059+0000", 2023-01-15T14:33:51.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6940000000000001e-06 2023-01-15T14:33:51.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.027908+0000", 2023-01-15T14:33:51.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189848287 2023-01-15T14:33:51.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.028049+0000", 2023-01-15T14:33:51.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014132500000000001 2023-01-15T14:33:51.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227252+0000", 2023-01-15T14:33:51.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19920324 2023-01-15T14:33:51.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227341+0000", 2023-01-15T14:33:51.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8683000000000005e-05 2023-01-15T14:33:51.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634637+0000", 2023-01-15T14:33:51.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729567300000002 2023-01-15T14:33:51.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634980+0000", 2023-01-15T14:33:51.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034335000000000002 2023-01-15T14:33:51.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.635292+0000", 2023-01-15T14:33:51.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003123 2023-01-15T14:33:51.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.173 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.173 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2222 3.c 3:3137f0cb:::smithi09025264-20:head [write 3259733~152121] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:51.173 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:33:51.173 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.157745823, 2023-01-15T14:33:51.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77651572400000002, 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2222 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:33:51.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:33:51.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857891+0000", 2023-01-15T14:33:51.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:51.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857993+0000", 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010276399999999999 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858021+0000", 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8022e-05 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858034+0000", 2023-01-15T14:33:51.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2549e-05 2023-01-15T14:33:51.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025371+0000", 2023-01-15T14:33:51.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16733653100000001 2023-01-15T14:33:51.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025429+0000", 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8922000000000003e-05 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226869+0000", 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20143931100000001 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227088+0000", 2023-01-15T14:33:51.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219328 2023-01-15T14:33:51.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:33:51.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227443+0000", 2023-01-15T14:33:51.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035481 2023-01-15T14:33:51.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:51.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227830+0000", 2023-01-15T14:33:51.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000386702 2023-01-15T14:33:51.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:51.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227861+0000", 2023-01-15T14:33:51.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1584000000000001e-05 2023-01-15T14:33:51.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:51.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227939+0000", 2023-01-15T14:33:51.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7354999999999994e-05 2023-01-15T14:33:51.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:51.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228548+0000", 2023-01-15T14:33:51.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060954800000000001 2023-01-15T14:33:51.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429817+0000", 2023-01-15T14:33:51.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20126904600000001 2023-01-15T14:33:51.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634125+0000", 2023-01-15T14:33:51.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20430780300000001 2023-01-15T14:33:51.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:51.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634259+0000", 2023-01-15T14:33:51.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134144 2023-01-15T14:33:51.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634410+0000", 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150447 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2223 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (57) in=7b,truncate 3411854] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.157587036, 2023-01-15T14:33:51.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87671107599999998, 2023-01-15T14:33:51.184 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.184 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:51.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.184 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.184 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2223 2023-01-15T14:33:51.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.185 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:33:51.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:33:51.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858052+0000", 2023-01-15T14:33:51.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:51.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858056+0000", 2023-01-15T14:33:51.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5619999999999997e-06 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858067+0000", 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0328e-05 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858071+0000", 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6439999999999999e-06 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025446+0000", 2023-01-15T14:33:51.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16737407200000001 2023-01-15T14:33:51.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025503+0000", 2023-01-15T14:33:51.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7952999999999999e-05 2023-01-15T14:33:51.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227970+0000", 2023-01-15T14:33:51.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202466866 2023-01-15T14:33:51.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228139+0000", 2023-01-15T14:33:51.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016889100000000001 2023-01-15T14:33:51.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:33:51.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228284+0000", 2023-01-15T14:33:51.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014493700000000001 2023-01-15T14:33:51.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:51.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228470+0000", 2023-01-15T14:33:51.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185541 2023-01-15T14:33:51.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228481+0000", 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.167e-05 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228537+0000", 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5912999999999998e-05 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:51.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228563+0000", 2023-01-15T14:33:51.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5613e-05 2023-01-15T14:33:51.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.433181+0000", 2023-01-15T14:33:51.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20461838099999999 2023-01-15T14:33:51.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734620+0000", 2023-01-15T14:33:51.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301438386 2023-01-15T14:33:51.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:51.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734682+0000", 2023-01-15T14:33:51.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2373999999999996e-05 2023-01-15T14:33:51.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734764+0000", 2023-01-15T14:33:51.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1656999999999994e-05 2023-01-15T14:33:51.194 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.194 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.194 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.194 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2224 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:33:51.194 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:33:51.194 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 132.15756198599999, 2023-01-15T14:33:51.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97704279199999999, 2023-01-15T14:33:51.194 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.194 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:51.195 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.195 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.195 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.195 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2224 2023-01-15T14:33:51.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.195 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:33:51.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:33:51.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858077+0000", 2023-01-15T14:33:51.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:51.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858080+0000", 2023-01-15T14:33:51.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:33:51.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858089+0000", 2023-01-15T14:33:51.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3449999999999995e-06 2023-01-15T14:33:51.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858093+0000", 2023-01-15T14:33:51.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9820000000000002e-06 2023-01-15T14:33:51.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025520+0000", 2023-01-15T14:33:51.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.167426556 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025594+0000", 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4339999999999996e-05 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228608+0000", 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20301395699999999 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228752+0000", 2023-01-15T14:33:51.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014385 2023-01-15T14:33:51.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834878+0000", 2023-01-15T14:33:51.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60612591500000002 2023-01-15T14:33:51.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834987+0000", 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108733 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.835120+0000", 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133688 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.201 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3120 3.40 3:03b54904:::smithi09025264-12:head [write 578273~466432] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:33:51.202 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:33:51.202 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.083610104000002, 2023-01-15T14:33:51.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72661194799999995, 2023-01-15T14:33:51.202 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.202 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:51.202 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.202 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.202 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.202 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3120 2023-01-15T14:33:51.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.202 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:33:51.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:33:51.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932026+0000", 2023-01-15T14:33:51.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:51.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932397+0000", 2023-01-15T14:33:51.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037070099999999997 2023-01-15T14:33:51.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932419+0000", 2023-01-15T14:33:51.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1923999999999999e-05 2023-01-15T14:33:51.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932435+0000", 2023-01-15T14:33:51.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6509999999999999e-05 2023-01-15T14:33:51.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933751+0000", 2023-01-15T14:33:51.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013156050000000001 2023-01-15T14:33:51.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:51.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933765+0000", 2023-01-15T14:33:51.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3952e-05 2023-01-15T14:33:51.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238456+0000", 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30469085000000001 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238797+0000", 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034117300000000001 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239214+0000", 2023-01-15T14:33:51.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000416845 2023-01-15T14:33:51.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:51.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240273+0000", 2023-01-15T14:33:51.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058811 2023-01-15T14:33:51.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:51.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240292+0000", 2023-01-15T14:33:51.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:33:51.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:51.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240555+0000", 2023-01-15T14:33:51.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026299500000000003 2023-01-15T14:33:51.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:51.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242072+0000", 2023-01-15T14:33:51.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015175550000000001 2023-01-15T14:33:51.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244038+0000", 2023-01-15T14:33:51.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001966176 2023-01-15T14:33:51.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658099+0000", 2023-01-15T14:33:51.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41406103300000002 2023-01-15T14:33:51.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:51.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658248+0000", 2023-01-15T14:33:51.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148577 2023-01-15T14:33:51.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658642+0000", 2023-01-15T14:33:51.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039346900000000002 2023-01-15T14:33:51.211 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.212 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.212 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3121 3.40 3:03b54904:::smithi09025264-12:head [write 1739595~452815] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:33:51.212 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:33:51.212 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.083178158000003, 2023-01-15T14:33:51.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72675957599999996, 2023-01-15T14:33:51.212 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.212 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:51.212 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.213 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.213 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3121 2023-01-15T14:33:51.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.213 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:33:51.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932460+0000", 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932725+0000", 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026482699999999998 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932741+0000", 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5880999999999999e-05 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932750+0000", 2023-01-15T14:33:51.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2380000000000003e-06 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034479+0000", 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10172906399999999 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034506+0000", 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7336000000000001e-05 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240672+0000", 2023-01-15T14:33:51.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206166074 2023-01-15T14:33:51.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240894+0000", 2023-01-15T14:33:51.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221189 2023-01-15T14:33:51.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:33:51.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241244+0000", 2023-01-15T14:33:51.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000350816 2023-01-15T14:33:51.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:51.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241923+0000", 2023-01-15T14:33:51.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067891200000000005 2023-01-15T14:33:51.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:51.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241943+0000", 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0069999999999999e-05 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243425+0000", 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001481719 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244087+0000", 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066184700000000002 2023-01-15T14:33:51.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244230+0000", 2023-01-15T14:33:51.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143277 2023-01-15T14:33:51.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658784+0000", 2023-01-15T14:33:51.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.414553479 2023-01-15T14:33:51.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658904+0000", 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120024 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659221+0000", 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031742700000000001 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.222 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3122 3.40 3:03b54904:::smithi09025264-12:head [write 2722929~655344] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:33:51.222 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:33:51.222 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.082874060000002, 2023-01-15T14:33:51.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72706898900000005, 2023-01-15T14:33:51.222 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.222 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:51.222 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.222 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.222 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.222 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3122 2023-01-15T14:33:51.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932765+0000", 2023-01-15T14:33:51.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:51.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933097+0000", 2023-01-15T14:33:51.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220299999999999 2023-01-15T14:33:51.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933106+0000", 2023-01-15T14:33:51.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5459999999999997e-06 2023-01-15T14:33:51.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933113+0000", 2023-01-15T14:33:51.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3419999999999996e-06 2023-01-15T14:33:51.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034598+0000", 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101485804 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034625+0000", 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6859000000000002e-05 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242147+0000", 2023-01-15T14:33:51.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20752189600000001 2023-01-15T14:33:51.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242517+0000", 2023-01-15T14:33:51.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037032200000000002 2023-01-15T14:33:51.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:33:51.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243157+0000", 2023-01-15T14:33:51.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063935700000000001 2023-01-15T14:33:51.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:51.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243809+0000", 2023-01-15T14:33:51.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065222000000000001 2023-01-15T14:33:51.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:51.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243827+0000", 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8260000000000001e-05 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244122+0000", 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029469400000000002 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245633+0000", 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001511068 2023-01-15T14:33:51.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.346711+0000", 2023-01-15T14:33:51.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101077812 2023-01-15T14:33:51.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659372+0000", 2023-01-15T14:33:51.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31266154800000001 2023-01-15T14:33:51.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659488+0000", 2023-01-15T14:33:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115444 2023-01-15T14:33:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659835+0000", 2023-01-15T14:33:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034670300000000002 2023-01-15T14:33:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.231 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.232 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.232 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3123 3.40 3:03b54904:::smithi09025264-12:head [setxattr _header (58) in=7b,truncate 3378273] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:33:51.232 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:33:51.232 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 42.082504366999999, 2023-01-15T14:33:51.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82751564099999997, 2023-01-15T14:33:51.232 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.232 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:51.232 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.232 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3123 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:33:51.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:33:51.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:51.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933139+0000", 2023-01-15T14:33:51.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7079999999999996e-06 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933145+0000", 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7849999999999997e-06 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033331+0000", 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100186076 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034697+0000", 2023-01-15T14:33:51.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001366138 2023-01-15T14:33:51.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:51.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034718+0000", 2023-01-15T14:33:51.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.065e-05 2023-01-15T14:33:51.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244319+0000", 2023-01-15T14:33:51.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20960071 2023-01-15T14:33:51.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244576+0000", 2023-01-15T14:33:51.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025712899999999998 2023-01-15T14:33:51.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:33:51.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244932+0000", 2023-01-15T14:33:51.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035626000000000001 2023-01-15T14:33:51.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:51.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245315+0000", 2023-01-15T14:33:51.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038305199999999999 2023-01-15T14:33:51.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245330+0000", 2023-01-15T14:33:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5027e-05 2023-01-15T14:33:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245384+0000", 2023-01-15T14:33:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4267000000000001e-05 2023-01-15T14:33:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:33:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245654+0000", 2023-01-15T14:33:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026944000000000002 2023-01-15T14:33:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.436220+0000", 2023-01-15T14:33:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19056593799999999 2023-01-15T14:33:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760086+0000", 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.323865927 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760241+0000", 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015535700000000001 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760651+0000", 2023-01-15T14:33:51.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040990699999999999 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3124 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 41.982277889000002, 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72792673799999996, 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:33:51.242 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3124 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:33:51.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033360+0000", 2023-01-15T14:33:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:51.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033372+0000", 2023-01-15T14:33:51.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2850000000000001e-05 2023-01-15T14:33:51.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:51.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033397+0000", 2023-01-15T14:33:51.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4349e-05 2023-01-15T14:33:51.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033412+0000", 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5420000000000001e-05 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036811+0000", 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033984169999999999 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036821+0000", 2023-01-15T14:33:51.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0213e-05 2023-01-15T14:33:51.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245712+0000", 2023-01-15T14:33:51.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20889102800000001 2023-01-15T14:33:51.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:51.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245912+0000", 2023-01-15T14:33:51.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020024600000000001 2023-01-15T14:33:51.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:33:51.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760781+0000", 2023-01-15T14:33:51.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51486881299999998 2023-01-15T14:33:51.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:33:51.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760970+0000", 2023-01-15T14:33:51.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018893600000000001 2023-01-15T14:33:51.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:51.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:51.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:33:51.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.761289+0000", 2023-01-15T14:33:51.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031870899999999997 2023-01-15T14:33:51.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.381 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:51.381 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:51.382 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:51.382 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:33:51.384 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:33:51.385 INFO:tasks.rados.rados.0.smithi090.stdout:1342: done (2 left) 2023-01-15T14:33:51.385 INFO:tasks.rados.rados.0.smithi090.stdout:1343: rollback oid 6 current snap is 142 2023-01-15T14:33:51.385 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 6 to 135 2023-01-15T14:33:51.385 INFO:tasks.rados.rados.0.smithi090.stdout:1332: finishing rollback tid 0 to smithi09025264-29 2023-01-15T14:33:51.385 INFO:tasks.rados.rados.0.smithi090.stdout:1341: finishing write tid 1 to smithi09025264-24 2023-01-15T14:33:51.385 INFO:tasks.rados.rados.0.smithi090.stdout:1341: finishing write tid 2 to smithi09025264-24 2023-01-15T14:33:51.385 INFO:tasks.rados.rados.0.smithi090.stdout:1341: finishing write tid 3 to smithi09025264-24 2023-01-15T14:33:51.386 INFO:tasks.rados.rados.0.smithi090.stdout:1341: finishing write tid 4 to smithi09025264-24 2023-01-15T14:33:51.386 INFO:tasks.rados.rados.0.smithi090.stdout:1341: oid 24 updating version 0 to 308 2023-01-15T14:33:51.386 INFO:tasks.rados.rados.0.smithi090.stdout:1341: oid 24 version 308 is already newer than 307 2023-01-15T14:33:51.386 INFO:tasks.rados.rados.0.smithi090.stdout:1341: oid 24 updating version 308 to 309 2023-01-15T14:33:51.386 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 309 (ObjNum 486 snap 142 seq_num 486) dirty exists 2023-01-15T14:33:51.386 INFO:tasks.rados.rados.0.smithi090.stdout:1341: left oid 24 (ObjNum 486 snap 142 seq_num 486) 2023-01-15T14:33:51.386 INFO:tasks.rados.rados.0.smithi090.stdout:1332: finishing rollback tid 1 to smithi09025264-29 2023-01-15T14:33:51.386 INFO:tasks.rados.rados.0.smithi090.stdout:1341: done (2 left) 2023-01-15T14:33:51.386 INFO:tasks.rados.rados.0.smithi090.stdout:1344: read oid 18 snap -1 2023-01-15T14:33:51.386 INFO:tasks.rados.rados.0.smithi090.stdout:1344: expect deleted 2023-01-15T14:33:51.386 INFO:tasks.rados.rados.0.smithi090.stdout:1345: read oid 31 snap -1 2023-01-15T14:33:51.387 INFO:tasks.rados.rados.0.smithi090.stdout:1345: expect (ObjNum 326 snap 87 seq_num 326) 2023-01-15T14:33:51.387 INFO:tasks.rados.rados.0.smithi090.stdout:1346: copy_from oid 48 from oid 14 current snap is 142 2023-01-15T14:33:51.387 INFO:tasks.rados.rados.0.smithi090.stdout:1347: copy_from oid 5 from oid 17 current snap is 142 2023-01-15T14:33:51.387 INFO:tasks.rados.rados.0.smithi090.stdout:1348: write (excl) oid 25 current snap is 142 2023-01-15T14:33:51.387 INFO:tasks.rados.rados.0.smithi090.stdout:1348: seq_num 487 ranges {585009=492855,1564966=620043} 2023-01-15T14:33:51.387 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.0 to 1 (10000) 2023-01-15T14:33:51.388 INFO:tasks.rados.rados.0.smithi090.stdout:1348: writing smithi09025264-25 from 585009 to 1077864 tid 1 2023-01-15T14:33:51.388 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2023-01-15T14:33:51.389 INFO:tasks.rados.rados.0.smithi090.stdout:1348: writing smithi09025264-25 from 1564966 to 2185009 tid 2 2023-01-15T14:33:51.390 INFO:tasks.rados.rados.0.smithi090.stdout:1349: read oid 27 snap -1 2023-01-15T14:33:51.390 INFO:tasks.rados.rados.0.smithi090.stdout:1349: expect (ObjNum 400 snap 106 seq_num 400) 2023-01-15T14:33:51.390 INFO:tasks.rados.rados.0.smithi090.stdout:1350: snap_remove snap 140 2023-01-15T14:33:51.390 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:51.380+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:51.484 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:51.480+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:51.790 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:51.790 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:33:51.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.790 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3540 3.4d 3:b3f111b5:::smithi09025264-48:head [copy-from ver 263 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2023-01-15T14:33:51.790 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:51.039129+0000", 2023-01-15T14:33:51.790 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.46479254599999997, 2023-01-15T14:33:51.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.46484064800000002, 2023-01-15T14:33:51.790 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:51.790 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3540 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.039129+0000", 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:51.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.039129+0000", 2023-01-15T14:33:51.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:51.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:51.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.039126+0000", 2023-01-15T14:33:51.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:51.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:51.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.039141+0000", 2023-01-15T14:33:51.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4707e-05 2023-01-15T14:33:51.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:51.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.039177+0000", 2023-01-15T14:33:51.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6032000000000003e-05 2023-01-15T14:33:51.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139276+0000", 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10009870999999999 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.442750+0000", 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30347471999999998 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.443042+0000", 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000291821 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:51.794 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3541 3.4d 3:b3f111b5:::smithi09025264-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e342)", 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:51.139318+0000", 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.36460299200000001, 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.36475892599999998, 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "delayed", 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:51.795 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.796 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3541 2023-01-15T14:33:51.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.796 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:51.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.796 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:51.796 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139318+0000", 2023-01-15T14:33:51.796 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:51.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.797 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:51.797 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139318+0000", 2023-01-15T14:33:51.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:51.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.797 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:51.797 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139315+0000", 2023-01-15T14:33:51.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:33:51.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139330+0000", 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4229999999999999e-05 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139356+0000", 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6333e-05 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139375+0000", 2023-01-15T14:33:51.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9043e-05 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.443435+0000", 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30406046599999997 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.443522+0000", 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7068000000000001e-05 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:51.799 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3550 3.4d 3.9a94c54d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e342)", 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:51.139506+0000", 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.364414986, 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.36466042399999998, 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:51.800 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3550 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139506+0000", 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139506+0000", 2023-01-15T14:33:51.801 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139505+0000", 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139512+0000", 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2559999999999996e-06 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139531+0000", 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8573e-05 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:51.139542+0000", 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0957e-05 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:51.803 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:51.804 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2023-01-15T14:33:51.804 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 3 2023-01-15T14:33:51.804 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:51.804 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:33:51.805 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:51.787+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:51.806 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:51.807 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:51.807 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:51.807 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:51.807 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:51.807 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:51.807 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:51.807 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:51.808 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:33:51.808 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:33:51.808 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:33:51.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:33:51.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:33:51.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:33:51.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:33:51.809 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:33:51.890 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:51.887+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:51.898 INFO:tasks.rados.rados.0.smithi090.stdout:1350: done (8 left) 2023-01-15T14:33:51.899 INFO:tasks.rados.rados.0.smithi090.stdout:1351: copy_from oid 1 from oid 28 current snap is 142 2023-01-15T14:33:51.900 INFO:tasks.rados.rados.0.smithi090.stdout:1352: read oid 22 snap -1 2023-01-15T14:33:51.901 INFO:tasks.rados.rados.0.smithi090.stdout:1352: expect (ObjNum 471 snap 136 seq_num 471) 2023-01-15T14:33:51.901 INFO:tasks.rados.rados.0.smithi090.stdout:1353: copy_from oid 11 from oid 12 current snap is 142 2023-01-15T14:33:51.901 INFO:tasks.rados.rados.0.smithi090.stdout:1354: snap_create 2023-01-15T14:33:51.901 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 11 2023-01-15T14:33:51.902 INFO:tasks.rados.rados.0.smithi090.stdout:1332: finishing rollback tid 2 to smithi09025264-29 2023-01-15T14:33:51.902 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 268 (ObjNum 402 snap 108 seq_num 402) dirty exists 2023-01-15T14:33:51.902 INFO:tasks.rados.rados.0.smithi090.stdout:1348: finishing write tid 1 to smithi09025264-25 2023-01-15T14:33:51.902 INFO:tasks.rados.rados.0.smithi090.stdout:1348: finishing write tid 2 to smithi09025264-25 2023-01-15T14:33:51.902 INFO:tasks.rados.rados.0.smithi090.stdout:1349: expect (ObjNum 400 snap 106 seq_num 400) 2023-01-15T14:33:51.912 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:33:51.930 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.2 to 1 (10000) 2023-01-15T14:33:51.931 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:51.931 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:51.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2023-01-15T14:33:51.941 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:33:51.993 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:51.987+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:52.047 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:52.191 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:52.193 INFO:tasks.rados.rados.0.smithi090.stdout:1348: finishing write tid 3 to smithi09025264-25 2023-01-15T14:33:52.193 INFO:tasks.rados.rados.0.smithi090.stdout:1348: finishing write tid 4 to smithi09025264-25 2023-01-15T14:33:52.193 INFO:tasks.rados.rados.0.smithi090.stdout:1348: oid 25 updating version 0 to 363 2023-01-15T14:33:52.193 INFO:tasks.rados.rados.0.smithi090.stdout:1348: oid 25 version 363 is already newer than 362 2023-01-15T14:33:52.193 INFO:tasks.rados.rados.0.smithi090.stdout:1348: oid 25 version 363 is already newer than 361 2023-01-15T14:33:52.193 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 25 v 363 (ObjNum 487 snap 142 seq_num 487) dirty exists 2023-01-15T14:33:52.193 INFO:tasks.rados.rados.0.smithi090.stdout:1348: left oid 25 (ObjNum 487 snap 142 seq_num 487) 2023-01-15T14:33:52.193 INFO:tasks.rados.rados.0.smithi090.stdout:1332: done (10 left) 2023-01-15T14:33:52.193 INFO:tasks.rados.rados.0.smithi090.stdout:1348: done (9 left) 2023-01-15T14:33:52.193 INFO:tasks.rados.rados.0.smithi090.stdout:1349: done (8 left) 2023-01-15T14:33:52.194 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:33:52.194 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:33:52.196 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:52.188+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:52.294 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:33:52.288+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:33:52.604 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:52.605 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:52.605 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:33:52.605 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:52.603+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:52.705 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:52.700+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:52.814 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:52.814 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:52.815 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:33:52.815 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:52.811+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:52.914 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:52.911+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:53.020 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.3 to 1 (10000) 2023-01-15T14:33:53.020 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:53.020 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:53.021 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2023-01-15T14:33:53.021 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:33:53.025 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:53.016+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:53.429 INFO:tasks.rados.rados.0.smithi090.stdout:1344: done (7 left) 2023-01-15T14:33:53.430 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:33:53.430 INFO:tasks.rados.rados.0.smithi090.stdout:1351: finishing copy_from to smithi09025264-1 2023-01-15T14:33:53.430 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 368 (ObjNum 466 snap 134 seq_num 466) dirty exists 2023-01-15T14:33:53.430 INFO:tasks.rados.rados.0.smithi090.stdout:1351: finishing copy_from racing read to smithi09025264-1 2023-01-15T14:33:53.430 INFO:tasks.rados.rados.0.smithi090.stdout:1351: done (6 left) 2023-01-15T14:33:53.430 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:33:53.432 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:53.433 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:53.433 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:33:53.434 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:53.427+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:53.442 INFO:tasks.rados.rados.0.smithi090.stdout:1353: finishing copy_from to smithi09025264-11 2023-01-15T14:33:53.614 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 307 (ObjNum 439 snap 121 seq_num 439) dirty exists 2023-01-15T14:33:53.615 INFO:tasks.rados.rados.0.smithi090.stdout:1353: finishing copy_from racing read to smithi09025264-11 2023-01-15T14:33:53.616 INFO:tasks.rados.rados.0.smithi090.stdout:1353: done (5 left) 2023-01-15T14:33:53.616 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:33:53.616 INFO:tasks.rados.rados.0.smithi090.stdout:1352: expect (ObjNum 471 snap 136 seq_num 471) 2023-01-15T14:33:53.616 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:53.616 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:53.617 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:33:53.617 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:53.611+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:53.717 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:53.712+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:53.827 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:33:53.829 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:53.829 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:53.833 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:33:53.835 INFO:tasks.rados.rados.0.smithi090.stdout:1343: finishing rollback tid 1 to smithi09025264-6 2023-01-15T14:33:53.835 INFO:tasks.rados.rados.0.smithi090.stdout:1343: finishing rollback tid 2 to smithi09025264-6 2023-01-15T14:33:53.835 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 308 (ObjNum 412 snap 112 seq_num 412) dirty exists 2023-01-15T14:33:53.835 INFO:tasks.rados.rados.0.smithi090.stdout:1343: done (4 left) 2023-01-15T14:33:53.835 INFO:tasks.rados.rados.0.smithi090.stdout:1352: done (3 left) 2023-01-15T14:33:53.836 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:53.836 INFO:tasks.rados.rados.0.smithi090.stdout:1346: finishing copy_from to smithi09025264-48 2023-01-15T14:33:53.836 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 48 v 263 (ObjNum 404 snap 108 seq_num 404) dirty exists 2023-01-15T14:33:53.836 INFO:tasks.rados.rados.0.smithi090.stdout:1346: finishing copy_from racing read to smithi09025264-48 2023-01-15T14:33:53.836 INFO:tasks.rados.rados.0.smithi090.stdout:1346: done (2 left) 2023-01-15T14:33:53.837 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:53.837 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:53.824+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:53.929 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:53.927+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:55.674 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:33:55.676 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:55.676 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:55.676 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:55.676 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:55.677 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:55.677 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:55.678 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:33:55.678 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:55.671+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:55.678 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.4 to 1 (10000) 2023-01-15T14:33:55.679 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2023-01-15T14:33:55.679 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:55.680 INFO:tasks.rados.rados.0.smithi090.stdout:1345: expect (ObjNum 326 snap 87 seq_num 326) 2023-01-15T14:33:55.680 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:55.680 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:55.680 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:55.681 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:33:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:33:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:33:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:33:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:33:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:33:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:33:55.682 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:33:55.683 INFO:tasks.rados.rados.0.smithi090.stdout:1345: done (1 left) 2023-01-15T14:33:55.692 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:55.692 INFO:tasks.rados.rados.0.smithi090.stdout:1347: finishing copy_from to smithi09025264-5 2023-01-15T14:33:55.692 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 379 (ObjNum 427 snap 116 seq_num 427) dirty exists 2023-01-15T14:33:55.692 INFO:tasks.rados.rados.0.smithi090.stdout:1347: finishing copy_from racing read to smithi09025264-5 2023-01-15T14:33:55.692 INFO:tasks.rados.rados.0.smithi090.stdout:1347: done (0 left) 2023-01-15T14:33:55.692 INFO:tasks.rados.rados.0.smithi090.stdout:1354: done (0 left) 2023-01-15T14:33:55.693 INFO:tasks.rados.rados.0.smithi090.stdout:1355: delete oid 13 current snap is 143 2023-01-15T14:33:55.693 INFO:tasks.rados.rados.0.smithi090.stdout:1355: done (0 left) 2023-01-15T14:33:55.693 INFO:tasks.rados.rados.0.smithi090.stdout:1356: snap_remove snap 141 2023-01-15T14:33:55.777 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:55.772+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:55.864 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:55.864 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:33:55.864 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:33:55.864 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:33:55.864 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:55.865 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:33:55.878 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:55.875+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:56.086 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:33:56.086 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:33:56.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:33:56.086 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:33:56.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.087 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:75 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:56.087 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:33:56.087 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 329.62050653300003, 2023-01-15T14:33:56.087 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92335155400000002, 2023-01-15T14:33:56.087 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:56.087 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:56.087 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:56.087 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:56.087 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:56.088 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 75 2023-01-15T14:33:56.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.088 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:56.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:56.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:33:56.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:56.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:33:56.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:56.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356896+0000", 2023-01-15T14:33:56.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:56.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:56.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356903+0000", 2023-01-15T14:33:56.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4619999999999999e-06 2023-01-15T14:33:56.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:56.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356917+0000", 2023-01-15T14:33:56.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4195999999999999e-05 2023-01-15T14:33:56.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:56.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356926+0000", 2023-01-15T14:33:56.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:33:56.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457743+0000", 2023-01-15T14:33:56.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100817668 2023-01-15T14:33:56.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:56.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457841+0000", 2023-01-15T14:33:56.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7079999999999999e-05 2023-01-15T14:33:56.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778474+0000", 2023-01-15T14:33:56.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32063301799999999 2023-01-15T14:33:56.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:56.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778627+0000", 2023-01-15T14:33:56.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015389599999999999 2023-01-15T14:33:56.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:56.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778815+0000", 2023-01-15T14:33:56.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018726600000000001 2023-01-15T14:33:56.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:56.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779033+0000", 2023-01-15T14:33:56.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021819999999999999 2023-01-15T14:33:56.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:56.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779051+0000", 2023-01-15T14:33:56.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7640000000000001e-05 2023-01-15T14:33:56.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:56.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:33:56.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3590999999999995e-05 2023-01-15T14:33:56.097 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.097 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:56.097 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779163+0000", 2023-01-15T14:33:56.097 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:33:56.097 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:56.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.081002+0000", 2023-01-15T14:33:56.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30183937799999999 2023-01-15T14:33:56.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:56.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280043+0000", 2023-01-15T14:33:56.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19904084699999999 2023-01-15T14:33:56.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:56.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280138+0000", 2023-01-15T14:33:56.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4915999999999997e-05 2023-01-15T14:33:56.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:56.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280248+0000", 2023-01-15T14:33:56.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110233 2023-01-15T14:33:56.103 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.103 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:56.103 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:76 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 329.62046492500002, 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92420393899999997, 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 76 2023-01-15T14:33:56.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:33:56.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:33:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.287 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356937+0000", 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356948+0000", 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.094e-05 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:56.288 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356959+0000", 2023-01-15T14:33:56.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1058e-05 2023-01-15T14:33:56.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:56.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356968+0000", 2023-01-15T14:33:56.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5339999999999999e-06 2023-01-15T14:33:56.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459524+0000", 2023-01-15T14:33:56.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102556147 2023-01-15T14:33:56.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:56.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459662+0000", 2023-01-15T14:33:56.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000137756 2023-01-15T14:33:56.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978621+0000", 2023-01-15T14:33:56.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51895887600000001 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978870+0000", 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024928999999999999 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280761+0000", 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30189113400000001 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:56.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281023+0000", 2023-01-15T14:33:56.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026172800000000003 2023-01-15T14:33:56.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:56.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281142+0000", 2023-01-15T14:33:56.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000119296 2023-01-15T14:33:56.292 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.292 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:56.293 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.293 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1076 3.f 3:f87c0183:::smithi09025264-50:head [setxattr _header (57) in=7b,truncate 2857190] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:33:56.293 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:33:56.294 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 253.21875617000001, 2023-01-15T14:33:56.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.88473693200000003, 2023-01-15T14:33:56.294 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:56.294 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:56.294 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:56.294 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:56.294 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:56.294 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1076 2023-01-15T14:33:56.294 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.294 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:56.294 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758647+0000", 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:56.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758646+0000", 2023-01-15T14:33:56.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:56.296 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:56.296 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758653+0000", 2023-01-15T14:33:56.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8029999999999997e-06 2023-01-15T14:33:56.296 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:56.296 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758670+0000", 2023-01-15T14:33:56.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6158999999999999e-05 2023-01-15T14:33:56.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:56.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758677+0000", 2023-01-15T14:33:56.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9840000000000004e-06 2023-01-15T14:33:56.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848049+0000", 2023-01-15T14:33:56.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089372163000000004 2023-01-15T14:33:56.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.491 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.491 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:56.491 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848056+0000", 2023-01-15T14:33:56.491 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0360000000000001e-06 2023-01-15T14:33:56.491 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.491 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.491 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.491 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951562+0000", 2023-01-15T14:33:56.491 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10350670100000001 2023-01-15T14:33:56.492 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.492 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.492 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:56.492 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951726+0000", 2023-01-15T14:33:56.493 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000163311 2023-01-15T14:33:56.493 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.493 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.493 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:56.493 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.951958+0000", 2023-01-15T14:33:56.493 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000232525 2023-01-15T14:33:56.493 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.497 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:56.498 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952168+0000", 2023-01-15T14:33:56.498 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020949799999999999 2023-01-15T14:33:56.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.498 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:56.498 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952186+0000", 2023-01-15T14:33:56.498 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7714e-05 2023-01-15T14:33:56.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.498 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:56.498 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952254+0000", 2023-01-15T14:33:56.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7998999999999997e-05 2023-01-15T14:33:56.499 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.499 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.499 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:56.499 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952278+0000", 2023-01-15T14:33:56.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4909000000000002e-05 2023-01-15T14:33:56.499 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.500 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.500 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:56.500 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065836+0000", 2023-01-15T14:33:56.500 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.113557081 2023-01-15T14:33:56.501 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.501 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.501 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:56.501 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643196+0000", 2023-01-15T14:33:56.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.57736012999999997 2023-01-15T14:33:56.501 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.501 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.501 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:56.501 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643278+0000", 2023-01-15T14:33:56.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2671999999999997e-05 2023-01-15T14:33:56.501 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.502 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:56.502 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.643384+0000", 2023-01-15T14:33:56.502 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010551400000000001 2023-01-15T14:33:56.502 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.502 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:56.502 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.502 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.502 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1077 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:33:56.502 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:33:56.503 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 253.21871735100001, 2023-01-15T14:33:56.503 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.98557123499999999, 2023-01-15T14:33:56.503 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:56.503 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:56.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:56.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:56.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:56.503 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1077 2023-01-15T14:33:56.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.503 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:56.507 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.507 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:56.507 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:33:56.508 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:56.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:33:56.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:56.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758685+0000", 2023-01-15T14:33:56.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:56.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:56.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758689+0000", 2023-01-15T14:33:56.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2760000000000002e-06 2023-01-15T14:33:56.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:56.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758704+0000", 2023-01-15T14:33:56.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4187e-05 2023-01-15T14:33:56.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:56.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758710+0000", 2023-01-15T14:33:56.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4330000000000001e-06 2023-01-15T14:33:56.656 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.656 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.656 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848086+0000", 2023-01-15T14:33:56.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089376466000000002 2023-01-15T14:33:56.658 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:56.658 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848093+0000", 2023-01-15T14:33:56.658 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4269999999999996e-06 2023-01-15T14:33:56.658 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.659 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.659 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.659 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952346+0000", 2023-01-15T14:33:56.659 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104252758 2023-01-15T14:33:56.659 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.659 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.659 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:56.659 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952427+0000", 2023-01-15T14:33:56.659 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1786000000000006e-05 2023-01-15T14:33:56.659 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.660 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.660 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743602+0000", 2023-01-15T14:33:56.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.79117428099999998 2023-01-15T14:33:56.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.660 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:56.660 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743892+0000", 2023-01-15T14:33:56.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029067200000000001 2023-01-15T14:33:56.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:56.661 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.744257+0000", 2023-01-15T14:33:56.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000364619 2023-01-15T14:33:56.661 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.661 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:56.661 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.661 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.661 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1087 3.f 3:fd983206:::smithi09025264-7:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:33:56.661 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:33:56.661 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 253.21868286899999, 2023-01-15T14:33:56.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0859964360000001, 2023-01-15T14:33:56.662 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:56.662 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:56.664 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:56.664 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:56.664 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:56.664 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1087 2023-01-15T14:33:56.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.664 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:56.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:33:56.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:56.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:56.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758724+0000", 2023-01-15T14:33:56.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7740000000000004e-06 2023-01-15T14:33:56.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:56.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758738+0000", 2023-01-15T14:33:56.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.33e-05 2023-01-15T14:33:56.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:56.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758744+0000", 2023-01-15T14:33:56.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2859999999999996e-06 2023-01-15T14:33:56.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848121+0000", 2023-01-15T14:33:56.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089377097000000003 2023-01-15T14:33:56.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848127+0000", 2023-01-15T14:33:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3500000000000002e-06 2023-01-15T14:33:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952454+0000", 2023-01-15T14:33:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104326167 2023-01-15T14:33:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:56.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952520+0000", 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6166000000000003e-05 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.063003+0000", 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.110482916 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065265+0000", 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022626370000000001 2023-01-15T14:33:56.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065291+0000", 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6095e-05 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.066153+0000", 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086127100000000004 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:56.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166533+0000", 2023-01-15T14:33:56.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100380177 2023-01-15T14:33:56.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:56.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166646+0000", 2023-01-15T14:33:56.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112717 2023-01-15T14:33:56.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844498+0000", 2023-01-15T14:33:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67785237399999998 2023-01-15T14:33:56.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844605+0000", 2023-01-15T14:33:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106691 2023-01-15T14:33:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844717+0000", 2023-01-15T14:33:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011203499999999999 2023-01-15T14:33:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.888 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1088 3.f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 253.21862720300001, 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.086312696, 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:56.889 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1088 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:33:56.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.891 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:56.891 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758775+0000", 2023-01-15T14:33:56.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:56.891 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:56.891 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758786+0000", 2023-01-15T14:33:56.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.076e-05 2023-01-15T14:33:56.891 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:56.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758809+0000", 2023-01-15T14:33:56.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3706999999999999e-05 2023-01-15T14:33:56.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:56.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758821+0000", 2023-01-15T14:33:56.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1976e-05 2023-01-15T14:33:56.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878534+0000", 2023-01-15T14:33:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.119712966 2023-01-15T14:33:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:33:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878540+0000", 2023-01-15T14:33:56.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0680000000000002e-06 2023-01-15T14:33:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952693+0000", 2023-01-15T14:33:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.074153067000000003 2023-01-15T14:33:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:33:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952735+0000", 2023-01-15T14:33:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1792999999999998e-05 2023-01-15T14:33:56.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065518+0000", 2023-01-15T14:33:56.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.11278334499999999 2023-01-15T14:33:56.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:56.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065687+0000", 2023-01-15T14:33:56.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016807700000000001 2023-01-15T14:33:56.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844786+0000", 2023-01-15T14:33:56.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.77909980300000004 2023-01-15T14:33:56.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:56.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844944+0000", 2023-01-15T14:33:56.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157424 2023-01-15T14:33:56.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:56.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.845089+0000", 2023-01-15T14:33:56.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144814 2023-01-15T14:33:56.899 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.899 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:56.899 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.899 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:16 3.13 3:ca393a9e:::smithi09025264-47:head [assert-version v193,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:56.900 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:33:56.900 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 216.67164977799999, 2023-01-15T14:33:56.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8970339479999998, 2023-01-15T14:33:56.900 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:56.900 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:56.900 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:56.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:56.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:33:56.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305751+0000", 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:56.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305763+0000", 2023-01-15T14:33:56.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2309e-05 2023-01-15T14:33:56.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:56.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305788+0000", 2023-01-15T14:33:56.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4388000000000001e-05 2023-01-15T14:33:56.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:56.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305799+0000", 2023-01-15T14:33:56.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1304000000000001e-05 2023-01-15T14:33:56.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:56.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196425+0000", 2023-01-15T14:33:56.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8906259830000001 2023-01-15T14:33:56.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:56.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196673+0000", 2023-01-15T14:33:56.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024747499999999998 2023-01-15T14:33:56.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:56.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.202787+0000", 2023-01-15T14:33:56.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0061146509999999996 2023-01-15T14:33:56.904 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.904 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:56.904 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:56.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.905 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1480 3.13 3:cbfd2d71:::smithi09025264-37:head [write 581575~705262,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:56.905 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:33:56.905 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 216.63838204999999, 2023-01-15T14:33:56.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.375425221, 2023-01-15T14:33:56.905 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:56.905 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:56.905 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:56.905 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:56.905 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:56.906 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1480 2023-01-15T14:33:56.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.906 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:56.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:56.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:33:56.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:56.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:56.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:56.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:56.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:33:57.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.050 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:57.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339019+0000", 2023-01-15T14:33:57.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:57.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:57.051 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339659+0000", 2023-01-15T14:33:57.051 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000639742 2023-01-15T14:33:57.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.052 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:57.052 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339690+0000", 2023-01-15T14:33:57.052 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1594000000000003e-05 2023-01-15T14:33:57.052 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:57.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339703+0000", 2023-01-15T14:33:57.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2828e-05 2023-01-15T14:33:57.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.510837+0000", 2023-01-15T14:33:57.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1711338759999999 2023-01-15T14:33:57.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:57.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511048+0000", 2023-01-15T14:33:57.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021057899999999999 2023-01-15T14:33:57.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:57.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511325+0000", 2023-01-15T14:33:57.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027690400000000002 2023-01-15T14:33:57.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:57.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512092+0000", 2023-01-15T14:33:57.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00076720800000000002 2023-01-15T14:33:57.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:57.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512113+0000", 2023-01-15T14:33:57.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0917e-05 2023-01-15T14:33:57.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:57.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512439+0000", 2023-01-15T14:33:57.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032636299999999998 2023-01-15T14:33:57.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:57.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512871+0000", 2023-01-15T14:33:57.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000432087 2023-01-15T14:33:57.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.058 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.058 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.058 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.514165+0000", 2023-01-15T14:33:57.058 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001293726 2023-01-15T14:33:57.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.058 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.058 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.058 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714137+0000", 2023-01-15T14:33:57.058 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19997229499999999 2023-01-15T14:33:57.059 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.060 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.060 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:57.060 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714233+0000", 2023-01-15T14:33:57.060 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6062000000000003e-05 2023-01-15T14:33:57.060 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.060 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.061 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:57.061 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714446+0000", 2023-01-15T14:33:57.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212952 2023-01-15T14:33:57.061 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.061 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:57.061 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.061 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.061 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.061 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1481 3.13 3:cbfd2d71:::smithi09025264-37:head [write 1733272~707409] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:57.061 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:33:57.061 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 216.63153276700001, 2023-01-15T14:33:57.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.572522502, 2023-01-15T14:33:57.062 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:57.062 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:57.062 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:57.062 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:57.062 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:57.062 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1481 2023-01-15T14:33:57.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.080 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:57.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:57.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:33:57.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:57.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:33:57.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:57.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345868+0000", 2023-01-15T14:33:57.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:57.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:57.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346316+0000", 2023-01-15T14:33:57.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044813399999999999 2023-01-15T14:33:57.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:57.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346348+0000", 2023-01-15T14:33:57.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:33:57.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:57.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346362+0000", 2023-01-15T14:33:57.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.375e-05 2023-01-15T14:33:57.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512276+0000", 2023-01-15T14:33:57.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1659136829999999 2023-01-15T14:33:57.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512427+0000", 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015088699999999999 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714565+0000", 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20213841599999999 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714732+0000", 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016676700000000001 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714964+0000", 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000231678 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715499+0000", 2023-01-15T14:33:57.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053513300000000003 2023-01-15T14:33:57.086 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.086 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:57.086 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715523+0000", 2023-01-15T14:33:57.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4266000000000001e-05 2023-01-15T14:33:57.086 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.086 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:57.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715867+0000", 2023-01-15T14:33:57.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034419399999999998 2023-01-15T14:33:57.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.816932+0000", 2023-01-15T14:33:57.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10106498999999999 2023-01-15T14:33:57.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:57.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906374+0000", 2023-01-15T14:33:57.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089441442999999995 2023-01-15T14:33:57.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918050+0000", 2023-01-15T14:33:57.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.011676677999999999 2023-01-15T14:33:57.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:57.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918218+0000", 2023-01-15T14:33:57.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016802000000000001 2023-01-15T14:33:57.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:57.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918393+0000", 2023-01-15T14:33:57.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017447 2023-01-15T14:33:57.092 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.092 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:57.092 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.093 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1482 3.13 3:cbfd2d71:::smithi09025264-37:head [write 3195938~437497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:57.093 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:33:57.093 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 216.62509690900001, 2023-01-15T14:33:57.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5664780289999998, 2023-01-15T14:33:57.093 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:57.093 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:57.093 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1482 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:57.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352275+0000", 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999695 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352644+0000", 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036870300000000001 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:57.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352676+0000", 2023-01-15T14:33:57.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1726000000000001e-05 2023-01-15T14:33:57.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:57.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352689+0000", 2023-01-15T14:33:57.213 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3617e-05 2023-01-15T14:33:57.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512602+0000", 2023-01-15T14:33:57.213 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1599126690000001 2023-01-15T14:33:57.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:57.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512839+0000", 2023-01-15T14:33:57.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023676900000000001 2023-01-15T14:33:57.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715649+0000", 2023-01-15T14:33:57.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20281044600000001 2023-01-15T14:33:57.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715777+0000", 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012738800000000001 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715995+0000", 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000217858 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716694+0000", 2023-01-15T14:33:57.215 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00069956700000000001 2023-01-15T14:33:57.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:57.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716709+0000", 2023-01-15T14:33:57.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4362000000000001e-05 2023-01-15T14:33:57.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:57.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716936+0000", 2023-01-15T14:33:57.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022744200000000001 2023-01-15T14:33:57.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.818876+0000", 2023-01-15T14:33:57.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101939635 2023-01-15T14:33:57.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:57.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906390+0000", 2023-01-15T14:33:57.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087513987000000001 2023-01-15T14:33:57.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918577+0000", 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012187066 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918697+0000", 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120326 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:57.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918784+0000", 2023-01-15T14:33:57.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7200999999999994e-05 2023-01-15T14:33:57.219 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.219 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:57.219 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.219 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1483 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3781575] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:57.219 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:33:57.219 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 216.624688757, 2023-01-15T14:33:57.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5663522429999999, 2023-01-15T14:33:57.219 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:57.220 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:57.220 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:57.220 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:57.220 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:57.220 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1483 2023-01-15T14:33:57.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.220 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:57.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:57.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:33:57.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:57.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:33:57.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:57.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352713+0000", 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352720+0000", 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7959999999999998e-06 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352735+0000", 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.455e-05 2023-01-15T14:33:57.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:57.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352741+0000", 2023-01-15T14:33:57.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8360000000000003e-06 2023-01-15T14:33:57.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512901+0000", 2023-01-15T14:33:57.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1601594980000001 2023-01-15T14:33:57.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:33:57.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.513067+0000", 2023-01-15T14:33:57.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016626 2023-01-15T14:33:57.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716977+0000", 2023-01-15T14:33:57.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20391002799999999 2023-01-15T14:33:57.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:57.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717146+0000", 2023-01-15T14:33:57.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168851 2023-01-15T14:33:57.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:33:57.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717467+0000", 2023-01-15T14:33:57.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032135600000000002 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717800+0000", 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033217400000000002 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717822+0000", 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2308999999999999e-05 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:57.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717970+0000", 2023-01-15T14:33:57.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014788099999999999 2023-01-15T14:33:57.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.819135+0000", 2023-01-15T14:33:57.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116478 2023-01-15T14:33:57.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:57.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906408+0000", 2023-01-15T14:33:57.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087272902999999999 2023-01-15T14:33:57.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918887+0000", 2023-01-15T14:33:57.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012479965000000001 2023-01-15T14:33:57.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:57.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918959+0000", 2023-01-15T14:33:57.368 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1978999999999996e-05 2023-01-15T14:33:57.368 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:57.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.919067+0000", 2023-01-15T14:33:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107083 2023-01-15T14:33:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1485 3.d 3:bc1c6b44:::smithi09025264-23:head [write 598382~780511,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:57.373 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:33:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 216.617218323, 2023-01-15T14:33:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5474121890000001, 2023-01-15T14:33:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1485 2023-01-15T14:33:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:57.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:57.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360168+0000", 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999828 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360790+0000", 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062269099999999998 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360822+0000", 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1903000000000003e-05 2023-01-15T14:33:57.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:57.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360836+0000", 2023-01-15T14:33:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3380000000000001e-05 2023-01-15T14:33:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096664+0000", 2023-01-15T14:33:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7358282320000002 2023-01-15T14:33:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096683+0000", 2023-01-15T14:33:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9063999999999999e-05 2023-01-15T14:33:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704561+0000", 2023-01-15T14:33:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60787808399999999 2023-01-15T14:33:57.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704846+0000", 2023-01-15T14:33:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028527400000000002 2023-01-15T14:33:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705135+0000", 2023-01-15T14:33:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028875400000000001 2023-01-15T14:33:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705735+0000", 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059956899999999999 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705761+0000", 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:57.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.706129+0000", 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036822299999999999 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906331+0000", 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200202091 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907111+0000", 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077990799999999995 2023-01-15T14:33:57.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907379+0000", 2023-01-15T14:33:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026804800000000001 2023-01-15T14:33:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907529+0000", 2023-01-15T14:33:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015022700000000001 2023-01-15T14:33:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907597+0000", 2023-01-15T14:33:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7785000000000002e-05 2023-01-15T14:33:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1486 3.d 3:bc1c6b44:::smithi09025264-23:head [write 2075401~451067] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:33:57.389 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 216.610822275, 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6461318330000001, 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1486 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:57.390 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:33:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:33:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366579+0000", 2023-01-15T14:33:57.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:33:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366845+0000", 2023-01-15T14:33:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026610499999999999 2023-01-15T14:33:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366878+0000", 2023-01-15T14:33:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2750000000000003e-05 2023-01-15T14:33:57.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.467009+0000", 2023-01-15T14:33:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100131071 2023-01-15T14:33:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.393 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096793+0000", 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6297846869999999 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096806+0000", 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3077e-05 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907957+0000", 2023-01-15T14:33:57.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81115055400000002 2023-01-15T14:33:57.395 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.395 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908229+0000", 2023-01-15T14:33:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027187499999999998 2023-01-15T14:33:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:33:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908496+0000", 2023-01-15T14:33:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026754200000000001 2023-01-15T14:33:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909025+0000", 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052865200000000001 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909046+0000", 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0713e-05 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909294+0000", 2023-01-15T14:33:57.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024864899999999998 2023-01-15T14:33:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:33:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.911607+0000", 2023-01-15T14:33:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023128939999999998 2023-01-15T14:33:57.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012076+0000", 2023-01-15T14:33:57.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100468293 2023-01-15T14:33:57.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:33:57.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012426+0000", 2023-01-15T14:33:57.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035044299999999998 2023-01-15T14:33:57.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:57.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012532+0000", 2023-01-15T14:33:57.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106142 2023-01-15T14:33:57.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:57.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012713+0000", 2023-01-15T14:33:57.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018048599999999999 2023-01-15T14:33:57.711 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.711 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:57.711 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.712 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1496 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:57.712 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:33:57.712 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 216.241179426, 2023-01-15T14:33:57.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870616037, 2023-01-15T14:33:57.712 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:57.712 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:57.713 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:57.713 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:57.713 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:57.713 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1496 2023-01-15T14:33:57.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.713 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:57.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:57.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:33:57.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:57.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:33:57.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:57.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736220+0000", 2023-01-15T14:33:57.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:33:57.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:57.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736238+0000", 2023-01-15T14:33:57.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7156e-05 2023-01-15T14:33:57.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:57.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736275+0000", 2023-01-15T14:33:57.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7876000000000001e-05 2023-01-15T14:33:57.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:57.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736290+0000", 2023-01-15T14:33:57.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4449e-05 2023-01-15T14:33:57.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602217+0000", 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8659270299999999 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602468+0000", 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000251315 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:57.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.606840+0000", 2023-01-15T14:33:57.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0043715009999999999 2023-01-15T14:33:57.718 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.718 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:57.718 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.718 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1497 3.11 3:8889e450:::smithi09025264-32:head [read 0~2991646,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:57.718 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:33:57.718 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 216.24109197199999, 2023-01-15T14:33:57.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738359249999998, 2023-01-15T14:33:57.719 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:57.719 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:57.719 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1497 2023-01-15T14:33:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:33:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:33:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736310+0000", 2023-01-15T14:33:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736317+0000", 2023-01-15T14:33:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7240000000000003e-06 2023-01-15T14:33:57.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736334+0000", 2023-01-15T14:33:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6878999999999999e-05 2023-01-15T14:33:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736341+0000", 2023-01-15T14:33:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0129999999999996e-06 2023-01-15T14:33:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607070+0000", 2023-01-15T14:33:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870728873 2023-01-15T14:33:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607226+0000", 2023-01-15T14:33:57.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015595 2023-01-15T14:33:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610147+0000", 2023-01-15T14:33:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.002921416 2023-01-15T14:33:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.913 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1498 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:33:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:33:57.914 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 216.24105186599999, 2023-01-15T14:33:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8752667839999999, 2023-01-15T14:33:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:33:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:33:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:33:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1498 2023-01-15T14:33:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:33:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:33:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736350+0000", 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736355+0000", 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4900000000000002e-06 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736363+0000", 2023-01-15T14:33:57.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2770000000000005e-06 2023-01-15T14:33:57.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:57.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736369+0000", 2023-01-15T14:33:57.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7400000000000001e-06 2023-01-15T14:33:57.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610234+0000", 2023-01-15T14:33:57.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738651439999998 2023-01-15T14:33:57.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:57.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610402+0000", 2023-01-15T14:33:57.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168283 2023-01-15T14:33:57.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:57.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.611618+0000", 2023-01-15T14:33:57.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001215647 2023-01-15T14:33:57.920 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.920 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:57.920 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.921 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.7.0:440 3.11 e231/207 ondisk, result = 0)", 2023-01-15T14:33:57.921 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:33:57.921 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 117.73327020799999, 2023-01-15T14:33:57.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973962139999998, 2023-01-15T14:33:57.921 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:57.921 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:57.921 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:57.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.921 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:57.921 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:33:57.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:57.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:33:57.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:57.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244132+0000", 2023-01-15T14:33:57.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244143+0000", 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1647e-05 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244165+0000", 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1501999999999998e-05 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.923 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.924 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:57.924 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244180+0000", 2023-01-15T14:33:57.924 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5434000000000002e-05 2023-01-15T14:33:57.924 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.924 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.924 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.924 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941489+0000", 2023-01-15T14:33:57.924 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973091899999999 2023-01-15T14:33:57.924 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.924 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.924 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:57.925 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941503+0000", 2023-01-15T14:33:57.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.414e-05 2023-01-15T14:33:57.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.925 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:57.925 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941529+0000", 2023-01-15T14:33:57.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5622999999999998e-05 2023-01-15T14:33:57.925 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.925 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:57.925 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:57.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.926 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.4.0:562 3.1b e231/224 3:d85f42d9:::smithi09025264-45:a8 v 231'66, mlcod=225'62)", 2023-01-15T14:33:57.926 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:33:57.926 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 117.542919012, 2023-01-15T14:33:57.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2578820149999999, 2023-01-15T14:33:57.926 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:57.926 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:57.926 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:57.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:57.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:33:57.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:57.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:33:57.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:57.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:57.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:33:57.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:57.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:57.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434492+0000", 2023-01-15T14:33:57.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4360000000000002e-06 2023-01-15T14:33:57.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:57.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434517+0000", 2023-01-15T14:33:57.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.48e-05 2023-01-15T14:33:57.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:57.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434530+0000", 2023-01-15T14:33:57.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2982999999999999e-05 2023-01-15T14:33:57.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:57.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434663+0000", 2023-01-15T14:33:57.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013307 2023-01-15T14:33:57.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.930 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:57.930 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434700+0000", 2023-01-15T14:33:57.930 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7020000000000001e-05 2023-01-15T14:33:57.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.930 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:57.930 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435438+0000", 2023-01-15T14:33:57.930 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073808900000000004 2023-01-15T14:33:57.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.931 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.931 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:57.931 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435456+0000", 2023-01-15T14:33:57.931 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8091999999999999e-05 2023-01-15T14:33:57.931 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:57.931 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:57.931 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:57.931 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435526+0000", 2023-01-15T14:33:57.931 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9806000000000002e-05 2023-01-15T14:33:58.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.069 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.069 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:58.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435570+0000", 2023-01-15T14:33:58.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4091999999999997e-05 2023-01-15T14:33:58.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.070 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:58.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.692366+0000", 2023-01-15T14:33:58.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2567961539999999 2023-01-15T14:33:58.071 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:58.071 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:58.072 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:58.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.079 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:24 3.d 3:b3f111b5:::smithi09025264-48:head [assert-version v160,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e232)", 2023-01-15T14:33:58.079 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:33:58.080 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 117.027082796, 2023-01-15T14:33:58.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9963031440000001, 2023-01-15T14:33:58.080 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:58.080 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:33:58.080 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:58.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:58.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:33:58.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:58.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950318+0000", 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950333+0000", 2023-01-15T14:33:58.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.467e-05 2023-01-15T14:33:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950368+0000", 2023-01-15T14:33:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4895999999999999e-05 2023-01-15T14:33:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050461+0000", 2023-01-15T14:33:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10009304400000001 2023-01-15T14:33:58.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050593+0000", 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000132764 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050615+0000", 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1483000000000001e-05 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945049+0000", 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.894434138 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945326+0000", 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000277368 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.946623+0000", 2023-01-15T14:33:58.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012970340000000001 2023-01-15T14:33:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.3.0:547 3.22 e232/180 3:47dfe935:::smithi09025264-29:81 v 232'124, mlcod=229'121)", 2023-01-15T14:33:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:33:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 116.926853048, 2023-01-15T14:33:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.643153431, 2023-01-15T14:33:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:33:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:33:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050549+0000", 2023-01-15T14:33:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:33:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:33:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050561+0000", 2023-01-15T14:33:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.278e-05 2023-01-15T14:33:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:33:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050583+0000", 2023-01-15T14:33:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1053e-05 2023-01-15T14:33:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:33:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050596+0000", 2023-01-15T14:33:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3946e-05 2023-01-15T14:33:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:33:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050708+0000", 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011129 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050750+0000", 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2725000000000002e-05 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051324+0000", 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00057335700000000003 2023-01-15T14:33:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051340+0000", 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5968999999999999e-05 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051405+0000", 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5419000000000005e-05 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:33:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051471+0000", 2023-01-15T14:33:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5671000000000002e-05 2023-01-15T14:33:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:33:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:33:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:33:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.693703+0000", 2023-01-15T14:33:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.642232554 2023-01-15T14:33:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:33:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:33:58.093 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:33:58.094 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:56.083+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:58.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:33:58.095 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:33:58.095 INFO:tasks.rados.rados.0.smithi090.stdout:1356: done (0 left) 2023-01-15T14:33:58.095 INFO:tasks.rados.rados.0.smithi090.stdout:1357: delete oid 43 current snap is 143 2023-01-15T14:33:58.096 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:33:58.096 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.7 to 1 (10000) 2023-01-15T14:33:58.097 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2023-01-15T14:33:58.100 INFO:tasks.rados.rados.0.smithi090.stdout:1357: done (0 left) 2023-01-15T14:33:58.100 INFO:tasks.rados.rados.0.smithi090.stdout:1358: read oid 5 snap -1 2023-01-15T14:33:58.100 INFO:tasks.rados.rados.0.smithi090.stdout:1358: expect (ObjNum 427 snap 116 seq_num 427) 2023-01-15T14:33:58.100 INFO:tasks.rados.rados.0.smithi090.stdout:1359: snap_remove snap 132 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1359: done (1 left) 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1360: read oid 42 snap -1 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1360: expect (ObjNum 446 snap 123 seq_num 446) 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1361: write (excl) oid 6 current snap is 143 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1361: seq_num 490 ranges {635430=718602,2140412=95018} 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1361: writing smithi09025264-6 from 635430 to 1354032 tid 1 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1361: writing smithi09025264-6 from 2140412 to 2235430 tid 2 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1358: expect (ObjNum 427 snap 116 seq_num 427) 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1358: done (2 left) 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1362: snap_create 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:58.101 INFO:tasks.rados.rados.0.smithi090.stdout:1360: expect (ObjNum 446 snap 123 seq_num 446) 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout:1360: done (1 left) 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout:1361: finishing write tid 1 to smithi09025264-6 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout:1361: finishing write tid 2 to smithi09025264-6 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout:1361: finishing write tid 3 to smithi09025264-6 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout:1361: finishing write tid 4 to smithi09025264-6 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout:1361: oid 6 updating version 0 to 311 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout:1361: oid 6 version 311 is already newer than 309 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout:1361: oid 6 version 311 is already newer than 310 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 311 (ObjNum 490 snap 143 seq_num 490) dirty exists 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout:1361: left oid 6 (ObjNum 490 snap 143 seq_num 490) 2023-01-15T14:33:58.102 INFO:tasks.rados.rados.0.smithi090.stdout:1361: done (0 left) 2023-01-15T14:33:58.103 INFO:tasks.rados.rados.0.smithi090.stdout:1362: done (0 left) 2023-01-15T14:33:58.103 INFO:tasks.rados.rados.0.smithi090.stdout:1363: read oid 9 snap -1 2023-01-15T14:33:58.103 INFO:tasks.rados.rados.0.smithi090.stdout:1363: expect deleted 2023-01-15T14:33:58.103 INFO:tasks.rados.rados.0.smithi090.stdout:1364: write oid 6 current snap is 144 2023-01-15T14:33:58.103 INFO:tasks.rados.rados.0.smithi090.stdout:1364: seq_num 491 ranges {652237=793851,2198894=53343} 2023-01-15T14:33:58.103 INFO:tasks.rados.rados.0.smithi090.stdout:1364: writing smithi09025264-6 from 652237 to 1446088 tid 1 2023-01-15T14:33:58.103 INFO:tasks.rados.rados.0.smithi090.stdout:1364: writing smithi09025264-6 from 2198894 to 2252237 tid 2 2023-01-15T14:33:58.103 INFO:tasks.rados.rados.0.smithi090.stdout:1365: rollback oid 40 current snap is 144 2023-01-15T14:33:58.103 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 40 to 125 2023-01-15T14:33:58.103 INFO:tasks.rados.rados.0.smithi090.stdout:1366: write (excl) oid 30 current snap is 144 2023-01-15T14:33:58.232 INFO:tasks.rados.rados.0.smithi090.stdout:1366: seq_num 492 ranges {669044=469100,1741023=459161} 2023-01-15T14:33:58.232 INFO:tasks.rados.rados.0.smithi090.stdout:1366: writing smithi09025264-30 from 669044 to 1138144 tid 1 2023-01-15T14:33:58.232 INFO:tasks.rados.rados.0.smithi090.stdout:1366: writing smithi09025264-30 from 1741023 to 2200184 tid 2 2023-01-15T14:33:58.232 INFO:tasks.rados.rados.0.smithi090.stdout:1367: snap_create 2023-01-15T14:33:58.232 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:33:58.232 INFO:tasks.rados.rados.0.smithi090.stdout:1363: done (3 left) 2023-01-15T14:33:58.232 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:33:58.234 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:58.091+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:58.234 INFO:tasks.rados.rados.0.smithi090.stdout:1365: finishing rollback tid 0 to smithi09025264-40 2023-01-15T14:33:58.235 INFO:tasks.rados.rados.0.smithi090.stdout:1364: finishing write tid 1 to smithi09025264-6 2023-01-15T14:33:58.235 INFO:tasks.rados.rados.0.smithi090.stdout:1365: finishing rollback tid 1 to smithi09025264-40 2023-01-15T14:33:58.235 INFO:tasks.rados.rados.0.smithi090.stdout:1365: finishing rollback tid 2 to smithi09025264-40 2023-01-15T14:33:58.235 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 313 (ObjNum 419 snap 112 seq_num 419) dirty exists 2023-01-15T14:33:58.235 INFO:tasks.rados.rados.0.smithi090.stdout:1365: done (2 left) 2023-01-15T14:33:58.235 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:33:58.235 INFO:tasks.rados.rados.0.smithi090.stdout:1364: finishing write tid 2 to smithi09025264-6 2023-01-15T14:33:58.237 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:33:58.232+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:33:58.329 INFO:tasks.rados.rados.0.smithi090.stdout:1364: finishing write tid 3 to smithi09025264-6 2023-01-15T14:33:58.420 INFO:tasks.rados.rados.0.smithi090.stdout:1364: finishing write tid 4 to smithi09025264-6 2023-01-15T14:33:58.420 INFO:tasks.rados.rados.0.smithi090.stdout:1364: oid 6 updating version 0 to 314 2023-01-15T14:33:58.420 INFO:tasks.rados.rados.0.smithi090.stdout:1364: oid 6 version 314 is already newer than 313 2023-01-15T14:33:58.421 INFO:tasks.rados.rados.0.smithi090.stdout:1364: oid 6 version 314 is already newer than 312 2023-01-15T14:33:58.421 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 314 (ObjNum 491 snap 144 seq_num 491) dirty exists 2023-01-15T14:33:58.421 INFO:tasks.rados.rados.0.smithi090.stdout:1364: left oid 6 (ObjNum 491 snap 144 seq_num 491) 2023-01-15T14:33:58.421 INFO:tasks.rados.rados.0.smithi090.stdout:1364: done (1 left) 2023-01-15T14:33:58.421 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:33:58.422 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:33:58.422 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:33:58.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.422 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.1f 349/343 [PushOp(3:f87c0183:::smithi09025264-50:head, version: 247'230, data_included: [3145728~366968], data_size: 366968, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:f87c0183:::smithi09025264-50:head@247'230, size: 3512696, copy_subset: [0~3512696], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3512696, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:33:58.422 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:58.332102+0000", 2023-01-15T14:33:58.422 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0095300960000000001, 2023-01-15T14:33:58.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0095881530000000003, 2023-01-15T14:33:58.422 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:58.423 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:58.423 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:58.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:58.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332102+0000", 2023-01-15T14:33:58.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:58.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:58.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332102+0000", 2023-01-15T14:33:58.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:58.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:58.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332102+0000", 2023-01-15T14:33:58.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:58.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:58.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332269+0000", 2023-01-15T14:33:58.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016771700000000001 2023-01-15T14:33:58.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:58.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332288+0000", 2023-01-15T14:33:58.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8281e-05 2023-01-15T14:33:58.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:58.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332300+0000", 2023-01-15T14:33:58.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2517000000000001e-05 2023-01-15T14:33:58.427 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:58.427 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:58.427 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:58.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.427 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.1f 349/343 [PushOp(3:f8edd34e:::smithi09025264-19:db, version: 294'251, data_included: [2097152~339929], data_size: 339929, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:f8edd34e:::smithi09025264-19:db@294'251, size: 2437081, copy_subset: [0~2437081], clone_subset: {}, snapset: e9=[]:{db=[db]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2437081, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:33:58.427 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:58.332361+0000", 2023-01-15T14:33:58.427 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0092719380000000004, 2023-01-15T14:33:58.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0094188789999999998, 2023-01-15T14:33:58.428 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:58.428 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:58.428 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:58.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:58.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332361+0000", 2023-01-15T14:33:58.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:58.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:58.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332361+0000", 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332360+0000", 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332489+0000", 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012861700000000001 2023-01-15T14:33:58.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:58.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332505+0000", 2023-01-15T14:33:58.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6603e-05 2023-01-15T14:33:58.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:58.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.332518+0000", 2023-01-15T14:33:58.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2476999999999999e-05 2023-01-15T14:33:58.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:58.430 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:58.431 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:58.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.431 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.7 349/348 [PushOp(3:e089c230:::smithi09025264-30:head, version: 333'137, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:e089c230:::smithi09025264-30:head@333'137, size: 3546343, copy_subset: [0~3546343], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3145728, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:33:58.431 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:58.336025+0000", 2023-01-15T14:33:58.431 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.0056072589999999999, 2023-01-15T14:33:58.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0058344790000000001, 2023-01-15T14:33:58.431 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:33:58.431 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:33:58.431 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:33:58.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:33:58.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.336025+0000", 2023-01-15T14:33:58.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:58.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:33:58.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.336025+0000", 2023-01-15T14:33:58.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:33:58.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:33:58.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.336024+0000", 2023-01-15T14:33:58.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:33:58.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:33:58.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.336349+0000", 2023-01-15T14:33:58.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032470300000000003 2023-01-15T14:33:58.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:33:58.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.336373+0000", 2023-01-15T14:33:58.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4267e-05 2023-01-15T14:33:58.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:33:58.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:33:58.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:33:58.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:58.336386+0000", 2023-01-15T14:33:58.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2775999999999999e-05 2023-01-15T14:33:58.435 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:58.435 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:33:58.435 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:58.435 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:33:58.435 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:33:58.435 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 3 2023-01-15T14:33:58.435 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:33:58.436 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:33:58.437 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:33:58.437 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:33:58.437 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:33:58.437 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:33:58.438 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:58.419+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:58.452 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:33:58.576 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:33:58.576 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:33:58.577 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:33:58.577 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:33:58.577 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:33:58.578 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:33:58.578 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:33:58.578 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:33:58.578 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:33:58.578 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:33:58.578 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:33:58.579 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:58.579 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:58.580 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:33:58.580 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:58.575+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:58.664 INFO:tasks.rados.rados.0.smithi090.stdout:1366: finishing write tid 1 to smithi09025264-30 2023-01-15T14:33:58.740 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:58.740 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:58.740 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:33:58.742 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:58.735+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:33:58.840 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:33:58.836+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:33:58.904 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:58.904 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:58.904 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:33:58.905 INFO:tasks.rados.rados.0.smithi090.stdout:1366: finishing write tid 2 to smithi09025264-30 2023-01-15T14:33:58.936 INFO:tasks.rados.rados.0.smithi090.stdout:1366: finishing write tid 3 to smithi09025264-30 2023-01-15T14:33:59.063 INFO:tasks.rados.rados.0.smithi090.stdout:1366: finishing write tid 4 to smithi09025264-30 2023-01-15T14:33:59.063 INFO:tasks.rados.rados.0.smithi090.stdout:1366: oid 30 updating version 0 to 235 2023-01-15T14:33:59.063 INFO:tasks.rados.rados.0.smithi090.stdout:1366: oid 30 version 235 is already newer than 234 2023-01-15T14:33:59.063 INFO:tasks.rados.rados.0.smithi090.stdout:1366: oid 30 version 235 is already newer than 233 2023-01-15T14:33:59.063 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 235 (ObjNum 492 snap 144 seq_num 492) dirty exists 2023-01-15T14:33:59.064 INFO:tasks.rados.rados.0.smithi090.stdout:1366: left oid 30 (ObjNum 492 snap 144 seq_num 492) 2023-01-15T14:33:59.064 INFO:tasks.rados.rados.0.smithi090.stdout:1366: done (0 left) 2023-01-15T14:33:59.065 INFO:teuthology.orchestra.run.smithi090.stderr:reweighted osd.1 to 1 (10000) 2023-01-15T14:33:59.067 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2023-01-15T14:33:59.067 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:33:59.069 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:59.070 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:59.072 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:33:59.076 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:33:59.071+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:33:59.177 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:33:59.172+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:33:59.225 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:59.225 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:59.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:33:59.278 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:33:59.275+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:33:59.351 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:33:59.550 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:33:59.552 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:33:59.556 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:33:59.558 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:33:59.559 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2023-01-15T14:33:58.945288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":338926428,"num_objects":154,"num_object_clones":101,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":162,"num_objects_misplaced":34,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":12,"num_read":1089,"num_read_kb":1027985,"num_write":938,"num_write_kb":322928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":241860355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18749,"ondisk_log_size":18749,"up":345,"acting":325,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":391,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6245992,"kb_used_data":556025280,"kb_used_omap":66010,"kb_used_meta":0,"kb_avail":555950288,"statfs":{"total":575688990720,"available":569293094912,"internally_reserved":76791808,"allocated":569369886720,"data_stored":569369886720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67594499,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":13,"commit_latency_ns":0,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42402958,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":-10,"num_objects_degraded":146,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":151,"num_read_kb":168767,"num_write":96,"num_write_kb":45607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":63588880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.885293"},"pg_stats":[{"pgid":"3.69","version":"110'149","reported_seq":895,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.316235+0000","last_change":"2023-01-15T14:33:57.316235+0000","last_active":"2023-01-15T14:33:57.316235+0000","last_peered":"2023-01-15T14:33:57.316235+0000","last_clean":"2023-01-15T14:33:57.316235+0000","last_became_active":"2023-01-15T14:33:50.216417+0000","last_became_peered":"2023-01-15T14:33:50.216417+0000","last_unstale":"2023-01-15T14:33:57.316235+0000","last_undegraded":"2023-01-15T14:33:57.316235+0000","last_fullsized":"2023-01-15T14:33:57.316235+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.406510072,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.68","version":"345'250","reported_seq":1396,"reported_epoch":347,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2023-01-15T14:33:56.832472+0000","last_change":"2023-01-15T14:33:53.618260+0000","last_active":"2023-01-15T14:33:56.832472+0000","last_peered":"2023-01-15T14:33:56.832472+0000","last_clean":"2023-01-15T14:33:42.635266+0000","last_became_active":"2023-01-15T14:33:53.330670+0000","last_became_peered":"2023-01-15T14:33:53.330670+0000","last_unstale":"2023-01-15T14:33:56.832472+0000","last_undegraded":"2023-01-15T14:33:52.928061+0000","last_fullsized":"2023-01-15T14:33:56.832472+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8014979,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":21385,"num_write":10,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":9643107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[1,7,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.67","version":"191'79","reported_seq":48,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.114675+0000","last_change":"2023-01-15T14:33:57.312535+0000","last_active":"2023-01-15T14:33:57.114675+0000","last_peered":"2023-01-15T14:33:57.114675+0000","last_clean":"2023-01-15T14:33:57.114675+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:33:57.114675+0000","last_undegraded":"2023-01-15T14:33:57.114675+0000","last_fullsized":"2023-01-15T14:33:57.114675+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.40508863699999997,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.66","version":"102'52","reported_seq":358,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.299510+0000","last_change":"2023-01-15T14:33:57.299510+0000","last_active":"2023-01-15T14:33:57.299510+0000","last_peered":"2023-01-15T14:33:57.299510+0000","last_clean":"2023-01-15T14:33:57.299510+0000","last_became_active":"2023-01-15T14:33:48.695009+0000","last_became_peered":"2023-01-15T14:33:48.695009+0000","last_unstale":"2023-01-15T14:33:57.299510+0000","last_undegraded":"2023-01-15T14:33:57.299510+0000","last_fullsized":"2023-01-15T14:33:57.299510+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.390920724,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2142,"num_write":4,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.65","version":"348'116","reported_seq":1147,"reported_epoch":348,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:57.676726+0000","last_change":"2023-01-15T14:33:57.203450+0000","last_active":"2023-01-15T14:33:57.676726+0000","last_peered":"2023-01-15T14:33:57.676726+0000","last_clean":"2023-01-15T14:33:43.349693+0000","last_became_active":"2023-01-15T14:33:57.201006+0000","last_became_peered":"2023-01-15T14:33:57.201006+0000","last_unstale":"2023-01-15T14:33:57.676726+0000","last_undegraded":"2023-01-15T14:33:56.909186+0000","last_fullsized":"2023-01-15T14:33:56.908621+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6376869,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3032,"num_write":8,"num_write_kb":3372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":11126498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,4],"avail_no_missing":["1","2","3","4"],"object_location_counts":[{"shards":"0,1,2,3,4","objects":1},{"shards":"1,2,3,4","objects":1},{"shards":"1,2,4","objects":1},{"shards":"1,4","objects":-1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.64","version":"334'251","reported_seq":1400,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.397056+0000","last_change":"2023-01-15T14:33:56.397607+0000","last_active":"2023-01-15T14:33:56.397056+0000","last_peered":"2023-01-15T14:33:56.397056+0000","last_clean":"2023-01-15T14:33:56.397056+0000","last_became_active":"2023-01-15T14:33:55.675085+0000","last_became_peered":"2023-01-15T14:33:55.675085+0000","last_unstale":"2023-01-15T14:33:56.397056+0000","last_undegraded":"2023-01-15T14:33:56.397056+0000","last_fullsized":"2023-01-15T14:33:56.397056+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.216593806,"stat_sum":{"num_bytes":3548733,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10773,"num_write":5,"num_write_kb":2741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.63","version":"149'233","reported_seq":729,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.394284+0000","last_change":"2023-01-15T14:33:56.394984+0000","last_active":"2023-01-15T14:33:56.394284+0000","last_peered":"2023-01-15T14:33:56.394284+0000","last_clean":"2023-01-15T14:33:56.394284+0000","last_became_active":"2023-01-15T14:33:55.277739+0000","last_became_peered":"2023-01-15T14:33:55.277739+0000","last_unstale":"2023-01-15T14:33:56.394284+0000","last_undegraded":"2023-01-15T14:33:56.394284+0000","last_fullsized":"2023-01-15T14:33:56.394284+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31355392500000001,"stat_sum":{"num_bytes":3802136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":21819,"num_write":19,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.62","version":"347'167","reported_seq":1324,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.625124+0000","last_change":"2023-01-15T14:33:57.625342+0000","last_active":"2023-01-15T14:33:57.625124+0000","last_peered":"2023-01-15T14:33:57.625124+0000","last_clean":"2023-01-15T14:33:57.625124+0000","last_became_active":"2023-01-15T14:33:56.590183+0000","last_became_peered":"2023-01-15T14:33:56.590183+0000","last_unstale":"2023-01-15T14:33:57.625124+0000","last_undegraded":"2023-01-15T14:33:57.625124+0000","last_fullsized":"2023-01-15T14:33:57.625124+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.62383204000000003,"stat_sum":{"num_bytes":5361549,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6734,"num_write":11,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5210269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.61","version":"321'138","reported_seq":990,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.444780+0000","last_change":"2023-01-15T14:33:56.444780+0000","last_active":"2023-01-15T14:33:56.444780+0000","last_peered":"2023-01-15T14:33:56.444780+0000","last_clean":"2023-01-15T14:33:56.444780+0000","last_became_active":"2023-01-15T14:33:53.421056+0000","last_became_peered":"2023-01-15T14:33:53.421056+0000","last_unstale":"2023-01-15T14:33:56.444780+0000","last_undegraded":"2023-01-15T14:33:56.444780+0000","last_fullsized":"2023-01-15T14:33:56.444780+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.36564521799999999,"stat_sum":{"num_bytes":1033664,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6059,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.60","version":"347'279","reported_seq":1387,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.330453+0000","last_change":"2023-01-15T14:33:57.330518+0000","last_active":"2023-01-15T14:33:57.330453+0000","last_peered":"2023-01-15T14:33:57.330453+0000","last_clean":"2023-01-15T14:33:57.330453+0000","last_became_active":"2023-01-15T14:33:56.478613+0000","last_became_peered":"2023-01-15T14:33:56.478613+0000","last_unstale":"2023-01-15T14:33:57.330453+0000","last_undegraded":"2023-01-15T14:33:57.330453+0000","last_fullsized":"2023-01-15T14:33:57.330453+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.232258925,"stat_sum":{"num_bytes":5214108,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":10865,"num_write":14,"num_write_kb":5082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15418818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1363,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.446925+0000","last_change":"2023-01-15T14:33:56.447607+0000","last_active":"2023-01-15T14:33:56.446925+0000","last_peered":"2023-01-15T14:33:56.446925+0000","last_clean":"2023-01-15T14:33:56.446925+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:33:56.446925+0000","last_undegraded":"2023-01-15T14:33:56.446925+0000","last_fullsized":"2023-01-15T14:33:56.446925+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:51.919","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.065866312999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5e","version":"215'39","reported_seq":974,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.316972+0000","last_change":"2023-01-15T14:33:57.316972+0000","last_active":"2023-01-15T14:33:57.316972+0000","last_peered":"2023-01-15T14:33:57.316972+0000","last_clean":"2023-01-15T14:33:57.316972+0000","last_became_active":"2023-01-15T14:33:35.364143+0000","last_became_peered":"2023-01-15T14:33:35.364143+0000","last_unstale":"2023-01-15T14:33:57.316972+0000","last_undegraded":"2023-01-15T14:33:57.316972+0000","last_fullsized":"2023-01-15T14:33:57.316972+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40693008000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1285,"reported_epoch":349,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:33:58.213615+0000","last_change":"2023-01-15T14:33:58.213117+0000","last_active":"2023-01-15T14:33:58.213615+0000","last_peered":"2023-01-15T14:33:58.213615+0000","last_clean":"2023-01-15T14:33:55.169917+0000","last_became_active":"2023-01-15T14:33:58.213117+0000","last_became_peered":"2023-01-15T14:33:58.213117+0000","last_unstale":"2023-01-15T14:33:58.213615+0000","last_undegraded":"2023-01-15T14:33:58.213615+0000","last_fullsized":"2023-01-15T14:33:57.907045+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10421003,"stat_sum":{"num_bytes":2869928,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":23811,"num_write":13,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,0],"avail_no_missing":["4","0","1","7"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.5c","version":"141'46","reported_seq":75,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.009446+0000","last_change":"2023-01-15T14:33:58.026285+0000","last_active":"2023-01-15T14:33:58.009446+0000","last_peered":"2023-01-15T14:33:58.009446+0000","last_clean":"2023-01-15T14:33:58.009446+0000","last_became_active":"2023-01-15T14:33:57.113032+0000","last_became_peered":"2023-01-15T14:33:57.113032+0000","last_unstale":"2023-01-15T14:33:58.009446+0000","last_undegraded":"2023-01-15T14:33:58.009446+0000","last_fullsized":"2023-01-15T14:33:58.009446+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.121922606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5b","version":"283'80","reported_seq":905,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.005338+0000","last_change":"2023-01-15T14:33:57.113633+0000","last_active":"2023-01-15T14:33:57.005338+0000","last_peered":"2023-01-15T14:33:57.005338+0000","last_clean":"2023-01-15T14:33:57.005338+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:33:57.005338+0000","last_undegraded":"2023-01-15T14:33:57.005338+0000","last_fullsized":"2023-01-15T14:33:57.005338+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.20934807,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5a","version":"329'348","reported_seq":1474,"reported_epoch":347,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.488419+0000","last_change":"2023-01-15T14:33:53.433694+0000","last_active":"2023-01-15T14:33:56.488419+0000","last_peered":"2023-01-15T14:33:56.488419+0000","last_clean":"2023-01-15T14:33:50.015328+0000","last_became_active":"2023-01-15T14:33:53.229261+0000","last_became_peered":"2023-01-15T14:33:53.229261+0000","last_unstale":"2023-01-15T14:33:56.488419+0000","last_undegraded":"2023-01-15T14:33:52.914214+0000","last_fullsized":"2023-01-15T14:33:52.913106+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:02.755","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.45505085000000001,"stat_sum":{"num_bytes":5983424,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[1,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.59","version":"294'256","reported_seq":1272,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.406685+0000","last_change":"2023-01-15T14:33:57.406685+0000","last_active":"2023-01-15T14:33:57.406685+0000","last_peered":"2023-01-15T14:33:57.406685+0000","last_clean":"2023-01-15T14:33:57.406685+0000","last_became_active":"2023-01-15T14:33:55.671990+0000","last_became_peered":"2023-01-15T14:33:55.671990+0000","last_unstale":"2023-01-15T14:33:57.406685+0000","last_undegraded":"2023-01-15T14:33:57.406685+0000","last_fullsized":"2023-01-15T14:33:57.406685+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.49887967,"stat_sum":{"num_bytes":1210137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":5,"num_read_kb":3549,"num_write":5,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.58","version":"323'320","reported_seq":1474,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.901143+0000","last_change":"2023-01-15T14:33:57.004059+0000","last_active":"2023-01-15T14:33:56.901143+0000","last_peered":"2023-01-15T14:33:56.901143+0000","last_clean":"2023-01-15T14:33:56.901143+0000","last_became_active":"2023-01-15T14:33:40.127669+0000","last_became_peered":"2023-01-15T14:33:40.127669+0000","last_unstale":"2023-01-15T14:33:56.901143+0000","last_undegraded":"2023-01-15T14:33:56.901143+0000","last_fullsized":"2023-01-15T14:33:56.901143+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.103642679,"stat_sum":{"num_bytes":4332079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.57","version":"347'95","reported_seq":79,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.444988+0000","last_change":"2023-01-15T14:33:56.445824+0000","last_active":"2023-01-15T14:33:56.444988+0000","last_peered":"2023-01-15T14:33:56.444988+0000","last_clean":"2023-01-15T14:33:56.444988+0000","last_became_active":"2023-01-15T14:33:36.994447+0000","last_became_peered":"2023-01-15T14:33:36.994447+0000","last_unstale":"2023-01-15T14:33:56.444988+0000","last_undegraded":"2023-01-15T14:33:56.444988+0000","last_fullsized":"2023-01-15T14:33:56.444988+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.160441256,"stat_sum":{"num_bytes":1512663,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3025326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.56","version":"320'127","reported_seq":602,"reported_epoch":344,"state":"activating+undersized+remapped","last_fresh":"2023-01-15T14:33:52.929175+0000","last_change":"2023-01-15T14:33:52.929424+0000","last_active":"2023-01-15T14:33:51.236802+0000","last_peered":"2023-01-15T14:33:50.013625+0000","last_clean":"2023-01-15T14:33:50.013625+0000","last_became_active":"2023-01-15T14:33:40.152319+0000","last_became_peered":"2023-01-15T14:33:40.152319+0000","last_unstale":"2023-01-15T14:33:52.929175+0000","last_undegraded":"2023-01-15T14:33:52.929175+0000","last_fullsized":"2023-01-15T14:33:52.929175+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.91022528000000003,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4],"avail_no_missing":["3","0","1","4"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.55","version":"110'55","reported_seq":1005,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.315559+0000","last_change":"2023-01-15T14:33:57.315559+0000","last_active":"2023-01-15T14:33:57.315559+0000","last_peered":"2023-01-15T14:33:57.315559+0000","last_clean":"2023-01-15T14:33:57.315559+0000","last_became_active":"2023-01-15T14:33:52.510429+0000","last_became_peered":"2023-01-15T14:33:52.510429+0000","last_unstale":"2023-01-15T14:33:57.315559+0000","last_undegraded":"2023-01-15T14:33:57.315559+0000","last_fullsized":"2023-01-15T14:33:57.315559+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.40629608699999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1119,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.499425+0000","last_change":"2023-01-15T14:33:56.501463+0000","last_active":"2023-01-15T14:33:56.499425+0000","last_peered":"2023-01-15T14:33:56.499425+0000","last_clean":"2023-01-15T14:33:56.499425+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:33:56.499425+0000","last_undegraded":"2023-01-15T14:33:56.499425+0000","last_fullsized":"2023-01-15T14:33:56.499425+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:33:45.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:45.583005+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.321363183,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.53","version":"343'389","reported_seq":1384,"reported_epoch":348,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:56.905391+0000","last_change":"2023-01-15T14:33:56.905760+0000","last_active":"2023-01-15T14:33:56.905347+0000","last_peered":"2023-01-15T14:33:55.165228+0000","last_clean":"2023-01-15T14:33:55.165228+0000","last_became_active":"2023-01-15T14:33:28.184994+0000","last_became_peered":"2023-01-15T14:33:28.184994+0000","last_unstale":"2023-01-15T14:33:56.905391+0000","last_undegraded":"2023-01-15T14:33:56.905391+0000","last_fullsized":"2023-01-15T14:33:56.905391+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:42.205","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.092650028999999995,"stat_sum":{"num_bytes":21535378,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":17077,"num_write":13,"num_write_kb":6760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,7],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":724,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.902208+0000","last_change":"2023-01-15T14:33:56.903307+0000","last_active":"2023-01-15T14:33:56.902208+0000","last_peered":"2023-01-15T14:33:56.902208+0000","last_clean":"2023-01-15T14:33:56.902208+0000","last_became_active":"2023-01-15T14:33:55.472400+0000","last_became_peered":"2023-01-15T14:33:55.472400+0000","last_unstale":"2023-01-15T14:33:56.902208+0000","last_undegraded":"2023-01-15T14:33:56.902208+0000","last_fullsized":"2023-01-15T14:33:56.902208+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.001336634,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.51","version":"348'316","reported_seq":1511,"reported_epoch":348,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:57.332202+0000","last_change":"2023-01-15T14:33:57.332202+0000","last_active":"2023-01-15T14:33:57.332202+0000","last_peered":"2023-01-15T14:33:57.332202+0000","last_clean":"2023-01-15T14:33:57.332202+0000","last_became_active":"2023-01-15T14:33:26.107940+0000","last_became_peered":"2023-01-15T14:33:26.107940+0000","last_unstale":"2023-01-15T14:33:57.332202+0000","last_undegraded":"2023-01-15T14:33:57.332202+0000","last_fullsized":"2023-01-15T14:33:57.332202+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:36.458","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.70457060199999999,"stat_sum":{"num_bytes":7496813,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.50","version":"337'420","reported_seq":1523,"reported_epoch":348,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:33:57.812867+0000","last_change":"2023-01-15T14:33:56.720993+0000","last_active":"2023-01-15T14:33:57.812867+0000","last_peered":"2023-01-15T14:33:57.812867+0000","last_clean":"2023-01-15T14:33:50.116263+0000","last_became_active":"2023-01-15T14:33:53.229903+0000","last_became_peered":"2023-01-15T14:33:53.229903+0000","last_unstale":"2023-01-15T14:33:57.812867+0000","last_undegraded":"2023-01-15T14:33:57.812867+0000","last_fullsized":"2023-01-15T14:33:52.915062+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.70524759299999995,"stat_sum":{"num_bytes":9681416,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":42160,"num_write":49,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":22053159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,3],"avail_no_missing":["7","3","4"],"object_location_counts":[{"shards":"3,4,7","objects":6},{"shards":"3,7","objects":-3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.4f","version":"155'145","reported_seq":822,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.353637+0000","last_change":"2023-01-15T14:33:57.353637+0000","last_active":"2023-01-15T14:33:57.353637+0000","last_peered":"2023-01-15T14:33:57.353637+0000","last_clean":"2023-01-15T14:33:57.353637+0000","last_became_active":"2023-01-15T14:33:25.129936+0000","last_became_peered":"2023-01-15T14:33:25.129936+0000","last_unstale":"2023-01-15T14:33:57.353637+0000","last_undegraded":"2023-01-15T14:33:57.353637+0000","last_fullsized":"2023-01-15T14:33:57.353637+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44322561900000002,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":87,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.387715+0000","last_change":"2023-01-15T14:33:56.393769+0000","last_active":"2023-01-15T14:33:56.387715+0000","last_peered":"2023-01-15T14:33:56.387715+0000","last_clean":"2023-01-15T14:33:56.387715+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:33:56.387715+0000","last_undegraded":"2023-01-15T14:33:56.387715+0000","last_fullsized":"2023-01-15T14:33:56.387715+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.31362827599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4d","version":"344'254","reported_seq":1310,"reported_epoch":344,"state":"active+clean","last_fresh":"2023-01-15T14:33:53.778919+0000","last_change":"2023-01-15T14:33:52.103984+0000","last_active":"2023-01-15T14:33:53.778919+0000","last_peered":"2023-01-15T14:33:53.778919+0000","last_clean":"2023-01-15T14:33:53.778919+0000","last_became_active":"2023-01-15T14:33:26.102614+0000","last_became_peered":"2023-01-15T14:33:26.102614+0000","last_unstale":"2023-01-15T14:33:53.778919+0000","last_undegraded":"2023-01-15T14:33:53.778919+0000","last_fullsized":"2023-01-15T14:33:53.778919+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.104242925,"stat_sum":{"num_bytes":9048419,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":15961,"num_write":16,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11291942,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.4c","version":"309'71","reported_seq":101,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.312666+0000","last_change":"2023-01-15T14:33:57.413785+0000","last_active":"2023-01-15T14:33:57.312666+0000","last_peered":"2023-01-15T14:33:57.312666+0000","last_clean":"2023-01-15T14:33:57.312666+0000","last_became_active":"2023-01-15T14:33:52.311859+0000","last_became_peered":"2023-01-15T14:33:52.311859+0000","last_unstale":"2023-01-15T14:33:57.312666+0000","last_undegraded":"2023-01-15T14:33:57.312666+0000","last_fullsized":"2023-01-15T14:33:57.312666+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.40874315500000002,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":747,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.626691+0000","last_change":"2023-01-15T14:33:57.627070+0000","last_active":"2023-01-15T14:33:57.626691+0000","last_peered":"2023-01-15T14:33:57.626691+0000","last_clean":"2023-01-15T14:33:57.626691+0000","last_became_active":"2023-01-15T14:33:52.406605+0000","last_became_peered":"2023-01-15T14:33:52.406605+0000","last_unstale":"2023-01-15T14:33:57.626691+0000","last_undegraded":"2023-01-15T14:33:57.626691+0000","last_fullsized":"2023-01-15T14:33:57.626691+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:42.594","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.515074951,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4a","version":"348'371","reported_seq":1469,"reported_epoch":348,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:57.415696+0000","last_change":"2023-01-15T14:33:56.904648+0000","last_active":"2023-01-15T14:33:57.415696+0000","last_peered":"2023-01-15T14:33:57.415696+0000","last_clean":"2023-01-15T14:33:57.415696+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:33:57.415696+0000","last_undegraded":"2023-01-15T14:33:57.415696+0000","last_fullsized":"2023-01-15T14:33:57.415696+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:33.328","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81443059699999998,"stat_sum":{"num_bytes":12470870,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":63,"num_read_kb":67536,"num_write":73,"num_write_kb":23626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":881,"reported_epoch":343,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:51.920225+0000","last_change":"2023-01-15T14:33:51.920225+0000","last_active":"2023-01-15T14:32:23.854468+0000","last_peered":"2023-01-15T14:32:22.960105+0000","last_clean":"2023-01-15T14:32:22.960105+0000","last_became_active":"2023-01-15T14:32:22.148707+0000","last_became_peered":"2023-01-15T14:32:22.148707+0000","last_unstale":"2023-01-15T14:33:51.920225+0000","last_undegraded":"2023-01-15T14:33:51.920225+0000","last_fullsized":"2023-01-15T14:33:51.920225+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.006207738,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"f9","length":"2"},{"start":"100","length":"2"},{"start":"106","length":"1"}]},{"pgid":"3.48","version":"179'204","reported_seq":1353,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.356590+0000","last_change":"2023-01-15T14:33:57.356590+0000","last_active":"2023-01-15T14:33:57.356590+0000","last_peered":"2023-01-15T14:33:57.356590+0000","last_clean":"2023-01-15T14:33:57.356590+0000","last_became_active":"2023-01-15T14:33:55.375518+0000","last_became_peered":"2023-01-15T14:33:55.375518+0000","last_unstale":"2023-01-15T14:33:57.356590+0000","last_undegraded":"2023-01-15T14:33:57.356590+0000","last_fullsized":"2023-01-15T14:33:57.356590+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:43.662","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.34933974800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1113,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.300260+0000","last_change":"2023-01-15T14:33:57.300260+0000","last_active":"2023-01-15T14:33:57.300260+0000","last_peered":"2023-01-15T14:33:57.300260+0000","last_clean":"2023-01-15T14:33:57.300260+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:33:57.300260+0000","last_undegraded":"2023-01-15T14:33:57.300260+0000","last_fullsized":"2023-01-15T14:33:57.300260+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:03.200","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.39166528900000003,"stat_sum":{"num_bytes":496762,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":5,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":1023,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.728559+0000","last_change":"2023-01-15T14:33:57.829426+0000","last_active":"2023-01-15T14:33:57.728559+0000","last_peered":"2023-01-15T14:33:57.728559+0000","last_clean":"2023-01-15T14:33:57.728559+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:33:57.728559+0000","last_undegraded":"2023-01-15T14:33:57.728559+0000","last_fullsized":"2023-01-15T14:33:57.728559+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:55.245","scrub_duration":0.30448409700000001,"objects_trimmed":0,"snaptrim_duration":0.714760332,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":904,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.396257+0000","last_change":"2023-01-15T14:33:56.396637+0000","last_active":"2023-01-15T14:33:56.396257+0000","last_peered":"2023-01-15T14:33:56.396257+0000","last_clean":"2023-01-15T14:33:56.396257+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:33:56.396257+0000","last_undegraded":"2023-01-15T14:33:56.396257+0000","last_fullsized":"2023-01-15T14:33:56.396257+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21706618899999999,"stat_sum":{"num_bytes":2635976,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":19927,"num_write":15,"num_write_kb":6659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4745860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1268,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.298701+0000","last_change":"2023-01-15T14:33:57.298701+0000","last_active":"2023-01-15T14:33:57.298701+0000","last_peered":"2023-01-15T14:33:57.298701+0000","last_clean":"2023-01-15T14:33:57.298701+0000","last_became_active":"2023-01-15T14:33:55.678019+0000","last_became_peered":"2023-01-15T14:33:55.678019+0000","last_unstale":"2023-01-15T14:33:57.298701+0000","last_undegraded":"2023-01-15T14:33:57.298701+0000","last_fullsized":"2023-01-15T14:33:57.298701+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:27.120","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.390225449,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.43","version":"297'314","reported_seq":1458,"reported_epoch":347,"state":"peering","last_fresh":"2023-01-15T14:33:56.235641+0000","last_change":"2023-01-15T14:33:56.236004+0000","last_active":"2023-01-15T14:33:56.235585+0000","last_peered":"2023-01-15T14:33:55.176514+0000","last_clean":"2023-01-15T14:33:55.176514+0000","last_became_active":"2023-01-15T14:32:28.977084+0000","last_became_peered":"2023-01-15T14:32:28.977084+0000","last_unstale":"2023-01-15T14:33:56.235641+0000","last_undegraded":"2023-01-15T14:33:56.235641+0000","last_fullsized":"2023-01-15T14:33:56.235641+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.101016546,"stat_sum":{"num_bytes":3395080,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":16585,"num_write":5,"num_write_kb":1351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,7],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":1029,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.621398+0000","last_change":"2023-01-15T14:33:57.621398+0000","last_active":"2023-01-15T14:33:57.621398+0000","last_peered":"2023-01-15T14:33:57.621398+0000","last_clean":"2023-01-15T14:33:57.621398+0000","last_became_active":"2023-01-15T14:33:55.376782+0000","last_became_peered":"2023-01-15T14:33:55.376782+0000","last_unstale":"2023-01-15T14:33:57.621398+0000","last_undegraded":"2023-01-15T14:33:57.621398+0000","last_fullsized":"2023-01-15T14:33:57.621398+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.51101871399999999,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1109,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.908053+0000","last_change":"2023-01-15T14:33:57.302504+0000","last_active":"2023-01-15T14:33:57.908053+0000","last_peered":"2023-01-15T14:33:57.908053+0000","last_clean":"2023-01-15T14:33:57.908053+0000","last_became_active":"2023-01-15T14:33:55.771969+0000","last_became_peered":"2023-01-15T14:33:55.771969+0000","last_unstale":"2023-01-15T14:33:57.908053+0000","last_undegraded":"2023-01-15T14:33:57.908053+0000","last_fullsized":"2023-01-15T14:33:57.908053+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.29405439700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.40","version":"335'307","reported_seq":1383,"reported_epoch":348,"state":"peering","last_fresh":"2023-01-15T14:33:57.104195+0000","last_change":"2023-01-15T14:33:57.002620+0000","last_active":"2023-01-15T14:33:57.002126+0000","last_peered":"2023-01-15T14:33:43.952880+0000","last_clean":"2023-01-15T14:33:43.952880+0000","last_became_active":"2023-01-15T14:33:14.198194+0000","last_became_peered":"2023-01-15T14:33:14.198194+0000","last_unstale":"2023-01-15T14:33:57.104195+0000","last_undegraded":"2023-01-15T14:33:57.104195+0000","last_fullsized":"2023-01-15T14:33:57.104195+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.704289622,"stat_sum":{"num_bytes":12753904,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":35541,"num_write":28,"num_write_kb":11001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":16413587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.3f","version":"321'233","reported_seq":1339,"reported_epoch":348,"state":"peering","last_fresh":"2023-01-15T14:33:57.010251+0000","last_change":"2023-01-15T14:33:57.010838+0000","last_active":"2023-01-15T14:33:57.010178+0000","last_peered":"2023-01-15T14:33:56.616086+0000","last_clean":"2023-01-15T14:33:50.017023+0000","last_became_active":"2023-01-15T14:33:53.022391+0000","last_became_peered":"2023-01-15T14:33:53.022391+0000","last_unstale":"2023-01-15T14:33:57.010251+0000","last_undegraded":"2023-01-15T14:33:57.010251+0000","last_fullsized":"2023-01-15T14:33:57.010251+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:29.095","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.810287749,"stat_sum":{"num_bytes":4764997,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6605283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.3e","version":"294'63","reported_seq":828,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.627138+0000","last_change":"2023-01-15T14:33:57.728361+0000","last_active":"2023-01-15T14:33:57.627138+0000","last_peered":"2023-01-15T14:33:57.627138+0000","last_clean":"2023-01-15T14:33:57.627138+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:33:57.627138+0000","last_undegraded":"2023-01-15T14:33:57.627138+0000","last_fullsized":"2023-01-15T14:33:57.627138+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.61381065800000001,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3d","version":"335'232","reported_seq":1310,"reported_epoch":348,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.998631+0000","last_change":"2023-01-15T14:33:53.127064+0000","last_active":"2023-01-15T14:33:56.998631+0000","last_peered":"2023-01-15T14:33:56.998631+0000","last_clean":"2023-01-15T14:33:50.059397+0000","last_became_active":"2023-01-15T14:33:53.126274+0000","last_became_peered":"2023-01-15T14:33:53.126274+0000","last_unstale":"2023-01-15T14:33:56.998631+0000","last_undegraded":"2023-01-15T14:33:52.928065+0000","last_fullsized":"2023-01-15T14:33:52.926980+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.15835743399999999,"stat_sum":{"num_bytes":8050279,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":13797,"num_write":16,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[1,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":345,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.904495+0000","last_change":"2023-01-15T14:33:56.912541+0000","last_active":"2023-01-15T14:33:57.904495+0000","last_peered":"2023-01-15T14:33:57.904495+0000","last_clean":"2023-01-15T14:33:57.904495+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:33:57.904495+0000","last_undegraded":"2023-01-15T14:33:57.904495+0000","last_fullsized":"2023-01-15T14:33:57.904495+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.470","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.0047236029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":12,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.014266+0000","last_change":"2023-01-15T14:33:57.014068+0000","last_active":"2023-01-15T14:33:57.014266+0000","last_peered":"2023-01-15T14:33:57.014266+0000","last_clean":"2023-01-15T14:33:57.014266+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:33:57.014266+0000","last_undegraded":"2023-01-15T14:33:57.014266+0000","last_fullsized":"2023-01-15T14:33:57.014266+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.17081586700000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1338,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.395588+0000","last_change":"2023-01-15T14:33:56.396153+0000","last_active":"2023-01-15T14:33:56.395588+0000","last_peered":"2023-01-15T14:33:56.395588+0000","last_clean":"2023-01-15T14:33:56.395588+0000","last_became_active":"2023-01-15T14:33:55.474944+0000","last_became_peered":"2023-01-15T14:33:55.474944+0000","last_unstale":"2023-01-15T14:33:56.395588+0000","last_undegraded":"2023-01-15T14:33:56.395588+0000","last_fullsized":"2023-01-15T14:33:56.395588+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.21712495700000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.39","version":"334'351","reported_seq":1388,"reported_epoch":347,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.185894+0000","last_change":"2023-01-15T14:33:53.440119+0000","last_active":"2023-01-15T14:33:56.185894+0000","last_peered":"2023-01-15T14:33:56.185894+0000","last_clean":"2023-01-15T14:33:50.016390+0000","last_became_active":"2023-01-15T14:33:53.436135+0000","last_became_peered":"2023-01-15T14:33:53.436135+0000","last_unstale":"2023-01-15T14:33:56.185894+0000","last_undegraded":"2023-01-15T14:33:53.017325+0000","last_fullsized":"2023-01-15T14:33:53.015555+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:21.301","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.0051603159999999999,"stat_sum":{"num_bytes":24023444,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":43987,"num_write":7,"num_write_kb":7292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,2],"avail_no_missing":["7","2","4"],"object_location_counts":[{"shards":"2,4,7","objects":9}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.38","version":"348'311","reported_seq":1523,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.959969+0000","last_change":"2023-01-15T14:33:58.009343+0000","last_active":"2023-01-15T14:33:57.959969+0000","last_peered":"2023-01-15T14:33:57.959969+0000","last_clean":"2023-01-15T14:33:57.959969+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:33:57.959969+0000","last_undegraded":"2023-01-15T14:33:57.959969+0000","last_fullsized":"2023-01-15T14:33:57.959969+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.10683759900000001,"stat_sum":{"num_bytes":7580089,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":42847,"num_write":35,"num_write_kb":12593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":401,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.109903+0000","last_change":"2023-01-15T14:33:57.109903+0000","last_active":"2023-01-15T14:33:57.109903+0000","last_peered":"2023-01-15T14:33:57.109903+0000","last_clean":"2023-01-15T14:33:57.109903+0000","last_became_active":"2023-01-15T14:33:52.412128+0000","last_became_peered":"2023-01-15T14:33:52.412128+0000","last_unstale":"2023-01-15T14:33:57.109903+0000","last_undegraded":"2023-01-15T14:33:57.109903+0000","last_fullsized":"2023-01-15T14:33:57.109903+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.208440876,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":556,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.331276+0000","last_change":"2023-01-15T14:33:57.332104+0000","last_active":"2023-01-15T14:33:57.331276+0000","last_peered":"2023-01-15T14:33:57.331276+0000","last_clean":"2023-01-15T14:33:57.331276+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:33:57.331276+0000","last_undegraded":"2023-01-15T14:33:57.331276+0000","last_fullsized":"2023-01-15T14:33:57.331276+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.31780763299999998,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":27,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.342717+0000","last_change":"2023-01-15T14:33:56.343772+0000","last_active":"2023-01-15T14:33:56.342717+0000","last_peered":"2023-01-15T14:33:56.342717+0000","last_clean":"2023-01-15T14:33:56.342717+0000","last_became_active":"2023-01-15T14:33:52.612576+0000","last_became_peered":"2023-01-15T14:33:52.612576+0000","last_unstale":"2023-01-15T14:33:56.342717+0000","last_undegraded":"2023-01-15T14:33:56.342717+0000","last_fullsized":"2023-01-15T14:33:56.342717+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.26469577999999999,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.34","version":"169'138","reported_seq":1152,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.301685+0000","last_change":"2023-01-15T14:33:57.301685+0000","last_active":"2023-01-15T14:33:57.301685+0000","last_peered":"2023-01-15T14:33:57.301685+0000","last_clean":"2023-01-15T14:33:57.301685+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:33:57.301685+0000","last_undegraded":"2023-01-15T14:33:57.301685+0000","last_fullsized":"2023-01-15T14:33:57.301685+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:59.872","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.39300976100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1228,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.904458+0000","last_change":"2023-01-15T14:33:57.354727+0000","last_active":"2023-01-15T14:33:57.904458+0000","last_peered":"2023-01-15T14:33:57.904458+0000","last_clean":"2023-01-15T14:33:57.904458+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:33:57.904458+0000","last_undegraded":"2023-01-15T14:33:57.904458+0000","last_fullsized":"2023-01-15T14:33:57.904458+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.442392491,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.32","version":"334'163","reported_seq":828,"reported_epoch":348,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:57.002498+0000","last_change":"2023-01-15T14:33:53.329373+0000","last_active":"2023-01-15T14:33:57.002498+0000","last_peered":"2023-01-15T14:33:57.002498+0000","last_clean":"2023-01-15T14:33:50.154755+0000","last_became_active":"2023-01-15T14:33:53.228307+0000","last_became_peered":"2023-01-15T14:33:53.228307+0000","last_unstale":"2023-01-15T14:33:57.002498+0000","last_undegraded":"2023-01-15T14:33:52.928067+0000","last_fullsized":"2023-01-15T14:33:52.927836+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.80293351999999996,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7048,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[1,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.13","version":"348'374","reported_seq":1373,"reported_epoch":348,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:57.414084+0000","last_change":"2023-01-15T14:33:57.414084+0000","last_active":"2023-01-15T14:33:57.414084+0000","last_peered":"2023-01-15T14:33:57.414084+0000","last_clean":"2023-01-15T14:33:57.414084+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:33:57.414084+0000","last_undegraded":"2023-01-15T14:33:57.414084+0000","last_fullsized":"2023-01-15T14:33:57.414084+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:29.359","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.20500138100000001,"stat_sum":{"num_bytes":8364486,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":42770,"num_write":17,"num_write_kb":9266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.12","version":"335'184","reported_seq":828,"reported_epoch":343,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:51.922895+0000","last_change":"2023-01-15T14:33:51.922895+0000","last_active":"2023-01-15T14:33:51.922608+0000","last_peered":"2023-01-15T14:33:44.161296+0000","last_clean":"2023-01-15T14:33:44.161296+0000","last_became_active":"2023-01-15T14:33:20.335242+0000","last_became_peered":"2023-01-15T14:33:20.335242+0000","last_unstale":"2023-01-15T14:33:51.922895+0000","last_undegraded":"2023-01-15T14:33:51.922895+0000","last_fullsized":"2023-01-15T14:33:51.922895+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.24331072000000001,"stat_sum":{"num_bytes":7581497,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":15,"num_write_kb":5299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"f9","length":"2"},{"start":"100","length":"2"},{"start":"106","length":"1"}]},{"pgid":"3.11","version":"334'306","reported_seq":1484,"reported_epoch":347,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.287045+0000","last_change":"2023-01-15T14:33:53.763768+0000","last_active":"2023-01-15T14:33:56.287045+0000","last_peered":"2023-01-15T14:33:56.287045+0000","last_clean":"2023-01-15T14:33:50.016827+0000","last_became_active":"2023-01-15T14:33:53.659621+0000","last_became_peered":"2023-01-15T14:33:53.659621+0000","last_unstale":"2023-01-15T14:33:56.287045+0000","last_undegraded":"2023-01-15T14:33:53.019571+0000","last_fullsized":"2023-01-15T14:33:53.017412+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.090247347000000006,"stat_sum":{"num_bytes":10718732,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":48289,"num_write":26,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6655737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":6},{"shards":"0,7","objects":-2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.10","version":"308'406","reported_seq":1399,"reported_epoch":347,"state":"active+recovery_wait+remapped","last_fresh":"2023-01-15T14:33:56.284917+0000","last_change":"2023-01-15T14:33:53.820604+0000","last_active":"2023-01-15T14:33:56.284917+0000","last_peered":"2023-01-15T14:33:56.284917+0000","last_clean":"2023-01-15T14:33:17.174184+0000","last_became_active":"2023-01-15T14:33:53.718934+0000","last_became_peered":"2023-01-15T14:33:53.718934+0000","last_unstale":"2023-01-15T14:33:56.284917+0000","last_undegraded":"2023-01-15T14:33:56.284917+0000","last_fullsized":"2023-01-15T14:33:56.284917+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.84251846600000002,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":41964,"num_write":46,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6670228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[1,3,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1213,"reported_epoch":344,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:33:53.140157+0000","last_change":"2023-01-15T14:33:53.139890+0000","last_active":"2023-01-15T14:33:53.140157+0000","last_peered":"2023-01-15T14:33:53.140157+0000","last_clean":"2023-01-15T14:33:50.081207+0000","last_became_active":"2023-01-15T14:33:53.139890+0000","last_became_peered":"2023-01-15T14:33:53.139890+0000","last_unstale":"2023-01-15T14:33:53.140157+0000","last_undegraded":"2023-01-15T14:33:53.140157+0000","last_fullsized":"2023-01-15T14:33:52.914111+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:41.299","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":1.1105973060000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,1],"avail_no_missing":["4","0","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":12,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.412233+0000","last_change":"2023-01-15T14:33:57.411714+0000","last_active":"2023-01-15T14:33:57.412233+0000","last_peered":"2023-01-15T14:33:57.412233+0000","last_clean":"2023-01-15T14:33:57.412233+0000","last_became_active":"2023-01-15T14:33:57.310759+0000","last_became_peered":"2023-01-15T14:33:57.310759+0000","last_unstale":"2023-01-15T14:33:57.412233+0000","last_undegraded":"2023-01-15T14:33:57.412233+0000","last_fullsized":"2023-01-15T14:33:57.412233+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.100760999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.d","version":"296'232","reported_seq":1293,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.113803+0000","last_change":"2023-01-15T14:33:57.114569+0000","last_active":"2023-01-15T14:33:57.113803+0000","last_peered":"2023-01-15T14:33:57.113803+0000","last_clean":"2023-01-15T14:33:57.113803+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:33:57.113803+0000","last_undegraded":"2023-01-15T14:33:57.113803+0000","last_fullsized":"2023-01-15T14:33:57.113803+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.20767458899999999,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.c","version":"348'95","reported_seq":1088,"reported_epoch":348,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:57.826220+0000","last_change":"2023-01-15T14:33:57.622687+0000","last_active":"2023-01-15T14:33:57.826220+0000","last_peered":"2023-01-15T14:33:57.826220+0000","last_clean":"2023-01-15T14:33:57.826220+0000","last_became_active":"2023-01-15T14:33:52.210127+0000","last_became_peered":"2023-01-15T14:33:52.210127+0000","last_unstale":"2023-01-15T14:33:57.826220+0000","last_undegraded":"2023-01-15T14:33:57.826220+0000","last_fullsized":"2023-01-15T14:33:57.826220+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.38959517900000001,"stat_sum":{"num_bytes":2221858,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":10493,"num_write":6,"num_write_kb":686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8023674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":999,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.618325+0000","last_change":"2023-01-15T14:33:57.618325+0000","last_active":"2023-01-15T14:33:57.618325+0000","last_peered":"2023-01-15T14:33:57.618325+0000","last_clean":"2023-01-15T14:33:57.618325+0000","last_became_active":"2023-01-15T14:33:52.425622+0000","last_became_peered":"2023-01-15T14:33:52.425622+0000","last_unstale":"2023-01-15T14:33:57.618325+0000","last_undegraded":"2023-01-15T14:33:57.618325+0000","last_fullsized":"2023-01-15T14:33:57.618325+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.60540339200000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1322,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.297022+0000","last_change":"2023-01-15T14:33:57.297022+0000","last_active":"2023-01-15T14:33:57.297022+0000","last_peered":"2023-01-15T14:33:57.297022+0000","last_clean":"2023-01-15T14:33:57.297022+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:33:57.297022+0000","last_undegraded":"2023-01-15T14:33:57.297022+0000","last_fullsized":"2023-01-15T14:33:57.297022+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:48.664","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.38900274200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1107,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.304492+0000","last_change":"2023-01-15T14:33:57.304492+0000","last_active":"2023-01-15T14:33:57.304492+0000","last_peered":"2023-01-15T14:33:57.304492+0000","last_clean":"2023-01-15T14:33:57.304492+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:33:57.304492+0000","last_undegraded":"2023-01-15T14:33:57.304492+0000","last_fullsized":"2023-01-15T14:33:57.304492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:38.874","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.29516895799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1304,"reported_epoch":349,"state":"activating+undersized+remapped","last_fresh":"2023-01-15T14:33:57.903125+0000","last_change":"2023-01-15T14:33:57.903380+0000","last_active":"2023-01-15T14:33:56.491492+0000","last_peered":"2023-01-15T14:33:55.173516+0000","last_clean":"2023-01-15T14:33:55.173516+0000","last_became_active":"2023-01-15T14:32:22.142296+0000","last_became_peered":"2023-01-15T14:32:22.142296+0000","last_unstale":"2023-01-15T14:33:57.903125+0000","last_undegraded":"2023-01-15T14:33:57.903125+0000","last_fullsized":"2023-01-15T14:33:57.903125+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.20813032000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1],"avail_no_missing":["3","1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":506,"reported_epoch":348,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:57.199812+0000","last_change":"2023-01-15T14:33:57.199812+0000","last_active":"2023-01-15T14:33:56.182917+0000","last_peered":"2023-01-15T14:33:55.072194+0000","last_clean":"2023-01-15T14:33:50.143204+0000","last_became_active":"2023-01-15T14:33:53.718233+0000","last_became_peered":"2023-01-15T14:33:53.718233+0000","last_unstale":"2023-01-15T14:33:57.199812+0000","last_undegraded":"2023-01-15T14:33:57.199812+0000","last_fullsized":"2023-01-15T14:33:57.199812+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.015161+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:33:57.015161+0000","last_peered":"2023-01-15T14:33:57.015161+0000","last_clean":"2023-01-15T14:33:57.015161+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:33:57.015161+0000","last_undegraded":"2023-01-15T14:33:57.015161+0000","last_fullsized":"2023-01-15T14:33:57.015161+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":995,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.297870+0000","last_change":"2023-01-15T14:33:57.297870+0000","last_active":"2023-01-15T14:33:57.297870+0000","last_peered":"2023-01-15T14:33:57.297870+0000","last_clean":"2023-01-15T14:33:57.297870+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:33:57.297870+0000","last_undegraded":"2023-01-15T14:33:57.297870+0000","last_fullsized":"2023-01-15T14:33:57.297870+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.38976201700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":393,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.085787+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:33:56.085787+0000","last_peered":"2023-01-15T14:33:56.085787+0000","last_clean":"2023-01-15T14:33:56.085787+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:33:56.085787+0000","last_undegraded":"2023-01-15T14:33:56.085787+0000","last_fullsized":"2023-01-15T14:33:56.085787+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.964","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":702,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.395060+0000","last_change":"2023-01-15T14:33:56.395495+0000","last_active":"2023-01-15T14:33:56.395060+0000","last_peered":"2023-01-15T14:33:56.395060+0000","last_clean":"2023-01-15T14:33:56.395060+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:33:56.395060+0000","last_undegraded":"2023-01-15T14:33:56.395060+0000","last_fullsized":"2023-01-15T14:33:56.395060+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.31666380900000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":150,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.010096+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:33:57.010096+0000","last_peered":"2023-01-15T14:33:57.010096+0000","last_clean":"2023-01-15T14:33:57.010096+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:33:57.010096+0000","last_undegraded":"2023-01-15T14:33:57.010096+0000","last_fullsized":"2023-01-15T14:33:57.010096+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1314,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.301164+0000","last_change":"2023-01-15T14:33:57.301164+0000","last_active":"2023-01-15T14:33:57.301164+0000","last_peered":"2023-01-15T14:33:57.301164+0000","last_clean":"2023-01-15T14:33:57.301164+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:33:57.301164+0000","last_undegraded":"2023-01-15T14:33:57.301164+0000","last_fullsized":"2023-01-15T14:33:57.301164+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.093344012000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":216,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.488732+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:33:56.488732+0000","last_peered":"2023-01-15T14:33:56.488732+0000","last_clean":"2023-01-15T14:33:56.488732+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:33:56.488732+0000","last_undegraded":"2023-01-15T14:33:56.488732+0000","last_fullsized":"2023-01-15T14:33:56.488732+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:25.583","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"344'317","reported_seq":1390,"reported_epoch":347,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.545414+0000","last_change":"2023-01-15T14:33:56.033295+0000","last_active":"2023-01-15T14:33:56.545414+0000","last_peered":"2023-01-15T14:33:56.545414+0000","last_clean":"2023-01-15T14:33:50.176701+0000","last_became_active":"2023-01-15T14:33:53.226305+0000","last_became_peered":"2023-01-15T14:33:53.226305+0000","last_unstale":"2023-01-15T14:33:56.545414+0000","last_undegraded":"2023-01-15T14:33:52.915843+0000","last_fullsized":"2023-01-15T14:33:52.914697+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:34.328","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.69416308599999998,"stat_sum":{"num_bytes":9428925,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":13874,"num_write":2,"num_write_kb":6435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[0,1],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,1","objects":2},{"shards":"0,1,3","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.5","version":"335'158","reported_seq":1165,"reported_epoch":349,"state":"activating+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:57.908195+0000","last_change":"2023-01-15T14:33:58.109966+0000","last_active":"2023-01-15T14:33:56.088835+0000","last_peered":"2023-01-15T14:33:55.073530+0000","last_clean":"2023-01-15T14:33:55.073530+0000","last_became_active":"2023-01-15T14:32:21.948278+0000","last_became_peered":"2023-01-15T14:32:21.948278+0000","last_unstale":"2023-01-15T14:33:57.908195+0000","last_undegraded":"2023-01-15T14:33:57.908195+0000","last_fullsized":"2023-01-15T14:33:57.905675+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.71299288800000005,"stat_sum":{"num_bytes":11390101,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":26145,"num_write":9,"num_write_kb":4265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":206,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.087555+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:33:56.087555+0000","last_peered":"2023-01-15T14:33:56.087555+0000","last_clean":"2023-01-15T14:33:56.087555+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:33:56.087555+0000","last_undegraded":"2023-01-15T14:33:56.087555+0000","last_fullsized":"2023-01-15T14:33:56.087555+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"335'308","reported_seq":1403,"reported_epoch":347,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.191211+0000","last_change":"2023-01-15T14:33:53.424640+0000","last_active":"2023-01-15T14:33:56.191211+0000","last_peered":"2023-01-15T14:33:56.191211+0000","last_clean":"2023-01-15T14:33:50.017033+0000","last_became_active":"2023-01-15T14:33:53.422817+0000","last_became_peered":"2023-01-15T14:33:53.422817+0000","last_unstale":"2023-01-15T14:33:56.191211+0000","last_undegraded":"2023-01-15T14:33:53.016002+0000","last_fullsized":"2023-01-15T14:33:53.014012+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.70443229200000002,"stat_sum":{"num_bytes":2954643,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":30684,"num_write":31,"num_write_kb":10131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7931053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2","objects":-1},{"shards":"1,2,4","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":395,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.287148+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:33:56.287148+0000","last_peered":"2023-01-15T14:33:56.287148+0000","last_clean":"2023-01-15T14:33:56.287148+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:33:56.287148+0000","last_undegraded":"2023-01-15T14:33:56.287148+0000","last_fullsized":"2023-01-15T14:33:56.287148+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"333'137","reported_seq":1119,"reported_epoch":348,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:56.917815+0000","last_change":"2023-01-15T14:33:56.917815+0000","last_active":"2023-01-15T14:33:56.080946+0000","last_peered":"2023-01-15T14:33:55.068936+0000","last_clean":"2023-01-15T14:33:42.900201+0000","last_became_active":"2023-01-15T14:33:53.137740+0000","last_became_peered":"2023-01-15T14:33:53.137740+0000","last_unstale":"2023-01-15T14:33:56.917815+0000","last_undegraded":"2023-01-15T14:33:56.917815+0000","last_fullsized":"2023-01-15T14:33:56.917815+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.471195633,"stat_sum":{"num_bytes":8547430,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":13597,"num_write":9,"num_write_kb":7710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":206,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.381327+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:33:56.381327+0000","last_peered":"2023-01-15T14:33:56.381327+0000","last_clean":"2023-01-15T14:33:56.381327+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:33:56.381327+0000","last_undegraded":"2023-01-15T14:33:56.381327+0000","last_fullsized":"2023-01-15T14:33:56.381327+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:17.360","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":27,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.386509+0000","last_change":"2023-01-15T14:33:56.387426+0000","last_active":"2023-01-15T14:33:56.386509+0000","last_peered":"2023-01-15T14:33:56.386509+0000","last_clean":"2023-01-15T14:33:56.386509+0000","last_became_active":"2023-01-15T14:33:52.206701+0000","last_became_peered":"2023-01-15T14:33:52.206701+0000","last_unstale":"2023-01-15T14:33:56.386509+0000","last_undegraded":"2023-01-15T14:33:56.386509+0000","last_fullsized":"2023-01-15T14:33:56.386509+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.30826422799999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.413064+0000","last_change":"2023-01-15T14:33:57.412557+0000","last_active":"2023-01-15T14:33:57.413064+0000","last_peered":"2023-01-15T14:33:57.413064+0000","last_clean":"2023-01-15T14:33:57.413064+0000","last_became_active":"2023-01-15T14:33:57.311981+0000","last_became_peered":"2023-01-15T14:33:57.311981+0000","last_unstale":"2023-01-15T14:33:57.413064+0000","last_undegraded":"2023-01-15T14:33:57.413064+0000","last_fullsized":"2023-01-15T14:33:57.413064+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"348'158","reported_seq":1178,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.009873+0000","last_change":"2023-01-15T14:33:58.010967+0000","last_active":"2023-01-15T14:33:58.009873+0000","last_peered":"2023-01-15T14:33:58.009873+0000","last_clean":"2023-01-15T14:33:58.009873+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:33:58.009873+0000","last_undegraded":"2023-01-15T14:33:58.009873+0000","last_fullsized":"2023-01-15T14:33:58.009873+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:32:19.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:19.303243+0000","objects_scrubbed":1,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.0010524499999999999,"stat_sum":{"num_bytes":1226927,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":18,"num_read_kb":20384,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":12,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.579508+0000","last_change":"2023-01-15T14:33:57.579060+0000","last_active":"2023-01-15T14:33:57.579508+0000","last_peered":"2023-01-15T14:33:57.579508+0000","last_clean":"2023-01-15T14:33:57.579508+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:33:57.579508+0000","last_undegraded":"2023-01-15T14:33:57.579508+0000","last_fullsized":"2023-01-15T14:33:57.579508+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.16","version":"320'127","reported_seq":625,"reported_epoch":348,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:57.013072+0000","last_change":"2023-01-15T14:33:57.013400+0000","last_active":"2023-01-15T14:33:57.013035+0000","last_peered":"2023-01-15T14:33:55.270604+0000","last_clean":"2023-01-15T14:33:55.270604+0000","last_became_active":"2023-01-15T14:33:35.159018+0000","last_became_peered":"2023-01-15T14:33:35.159018+0000","last_unstale":"2023-01-15T14:33:57.013072+0000","last_undegraded":"2023-01-15T14:33:57.013072+0000","last_fullsized":"2023-01-15T14:33:57.013072+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:54.467","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.30046122199999997,"stat_sum":{"num_bytes":7378471,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,7],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":27,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.396704+0000","last_change":"2023-01-15T14:33:56.396996+0000","last_active":"2023-01-15T14:33:56.396704+0000","last_peered":"2023-01-15T14:33:56.396704+0000","last_clean":"2023-01-15T14:33:56.396704+0000","last_became_active":"2023-01-15T14:33:52.303197+0000","last_became_peered":"2023-01-15T14:33:52.303197+0000","last_unstale":"2023-01-15T14:33:56.396704+0000","last_undegraded":"2023-01-15T14:33:56.396704+0000","last_fullsized":"2023-01-15T14:33:56.396704+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.21267884200000001,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.18","version":"340'324","reported_seq":1479,"reported_epoch":348,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:57.007858+0000","last_change":"2023-01-15T14:33:53.518579+0000","last_active":"2023-01-15T14:33:57.007858+0000","last_peered":"2023-01-15T14:33:57.007858+0000","last_clean":"2023-01-15T14:33:50.112982+0000","last_became_active":"2023-01-15T14:33:53.517870+0000","last_became_peered":"2023-01-15T14:33:53.517870+0000","last_unstale":"2023-01-15T14:33:57.007858+0000","last_undegraded":"2023-01-15T14:33:52.920700+0000","last_fullsized":"2023-01-15T14:33:52.920090+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:06.616","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.66838752300000004,"stat_sum":{"num_bytes":4332078,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":26,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":4},{"shards":"1,3","objects":-2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.19","version":"294'256","reported_seq":1316,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.626083+0000","last_change":"2023-01-15T14:33:57.626542+0000","last_active":"2023-01-15T14:33:57.626083+0000","last_peered":"2023-01-15T14:33:57.626083+0000","last_clean":"2023-01-15T14:33:57.626083+0000","last_became_active":"2023-01-15T14:33:35.259904+0000","last_became_peered":"2023-01-15T14:33:35.259904+0000","last_unstale":"2023-01-15T14:33:57.626083+0000","last_undegraded":"2023-01-15T14:33:57.626083+0000","last_fullsized":"2023-01-15T14:33:57.626083+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:01.296","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.52734117999999996,"stat_sum":{"num_bytes":2420274,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1a","version":"348'352","reported_seq":1450,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.010827+0000","last_change":"2023-01-15T14:33:58.011960+0000","last_active":"2023-01-15T14:33:58.010827+0000","last_peered":"2023-01-15T14:33:58.010827+0000","last_clean":"2023-01-15T14:33:58.010827+0000","last_became_active":"2023-01-15T14:33:35.063699+0000","last_became_peered":"2023-01-15T14:33:35.063699+0000","last_unstale":"2023-01-15T14:33:58.010827+0000","last_undegraded":"2023-01-15T14:33:58.010827+0000","last_fullsized":"2023-01-15T14:33:58.010827+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.0039007270000000001,"stat_sum":{"num_bytes":3052653,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":15,"num_read_kb":21426,"num_write":7,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1b","version":"333'97","reported_seq":49,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.447812+0000","last_change":"2023-01-15T14:33:56.448600+0000","last_active":"2023-01-15T14:33:56.447812+0000","last_peered":"2023-01-15T14:33:56.447812+0000","last_clean":"2023-01-15T14:33:56.447812+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:33:56.447812+0000","last_undegraded":"2023-01-15T14:33:56.447812+0000","last_fullsized":"2023-01-15T14:33:56.447812+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.36685731199999999,"stat_sum":{"num_bytes":4400087,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10511,"num_write":14,"num_write_kb":4701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9416119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":538,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.011212+0000","last_change":"2023-01-15T14:33:58.012512+0000","last_active":"2023-01-15T14:33:58.011212+0000","last_peered":"2023-01-15T14:33:58.011212+0000","last_clean":"2023-01-15T14:33:58.011212+0000","last_became_active":"2023-01-15T14:33:57.125876+0000","last_became_peered":"2023-01-15T14:33:57.125876+0000","last_unstale":"2023-01-15T14:33:58.011212+0000","last_undegraded":"2023-01-15T14:33:58.011212+0000","last_fullsized":"2023-01-15T14:33:58.011212+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.0028318649999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1263,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.405395+0000","last_change":"2023-01-15T14:33:57.405395+0000","last_active":"2023-01-15T14:33:57.405395+0000","last_peered":"2023-01-15T14:33:57.405395+0000","last_clean":"2023-01-15T14:33:57.405395+0000","last_became_active":"2023-01-15T14:33:35.362912+0000","last_became_peered":"2023-01-15T14:33:35.362912+0000","last_unstale":"2023-01-15T14:33:57.405395+0000","last_undegraded":"2023-01-15T14:33:57.405395+0000","last_fullsized":"2023-01-15T14:33:57.405395+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.394560678,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":974,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.317895+0000","last_change":"2023-01-15T14:33:57.317895+0000","last_active":"2023-01-15T14:33:57.317895+0000","last_peered":"2023-01-15T14:33:57.317895+0000","last_clean":"2023-01-15T14:33:57.317895+0000","last_became_active":"2023-01-15T14:33:35.261561+0000","last_became_peered":"2023-01-15T14:33:35.261561+0000","last_unstale":"2023-01-15T14:33:57.317895+0000","last_undegraded":"2023-01-15T14:33:57.317895+0000","last_fullsized":"2023-01-15T14:33:57.317895+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40752737100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1f","version":"318'256","reported_seq":1306,"reported_epoch":347,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:33:56.715595+0000","last_change":"2023-01-15T14:33:56.748752+0000","last_active":"2023-01-15T14:33:56.715595+0000","last_peered":"2023-01-15T14:33:56.715595+0000","last_clean":"2023-01-15T14:33:27.785873+0000","last_became_active":"2023-01-15T14:33:53.125737+0000","last_became_peered":"2023-01-15T14:33:53.125737+0000","last_unstale":"2023-01-15T14:33:56.715595+0000","last_undegraded":"2023-01-15T14:33:53.115874+0000","last_fullsized":"2023-01-15T14:33:53.114236+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":1,"snaptrim_duration":0.30935489599999999,"stat_sum":{"num_bytes":2974888,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":13341,"num_write":1,"num_write_kb":1699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[2,4],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"2,3,4","objects":3},{"shards":"2,4","objects":-2}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.20","version":"324'273","reported_seq":1345,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.283462+0000","last_change":"2023-01-15T14:33:56.284414+0000","last_active":"2023-01-15T14:33:56.283462+0000","last_peered":"2023-01-15T14:33:56.283462+0000","last_clean":"2023-01-15T14:33:56.283462+0000","last_became_active":"2023-01-15T14:33:55.472883+0000","last_became_peered":"2023-01-15T14:33:55.472883+0000","last_unstale":"2023-01-15T14:33:56.283462+0000","last_undegraded":"2023-01-15T14:33:56.283462+0000","last_fullsized":"2023-01-15T14:33:56.283462+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.20799870300000001,"stat_sum":{"num_bytes":4251137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10864,"num_write":9,"num_write_kb":4140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4438968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.21","version":"321'138","reported_seq":1038,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.004213+0000","last_change":"2023-01-15T14:33:57.005196+0000","last_active":"2023-01-15T14:33:57.004213+0000","last_peered":"2023-01-15T14:33:57.004213+0000","last_clean":"2023-01-15T14:33:57.004213+0000","last_became_active":"2023-01-15T14:33:45.267321+0000","last_became_peered":"2023-01-15T14:33:45.267321+0000","last_unstale":"2023-01-15T14:33:57.004213+0000","last_undegraded":"2023-01-15T14:33:57.004213+0000","last_fullsized":"2023-01-15T14:33:57.004213+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.102357873,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.22","version":"323'161","reported_seq":1329,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.619172+0000","last_change":"2023-01-15T14:33:57.619172+0000","last_active":"2023-01-15T14:33:57.619172+0000","last_peered":"2023-01-15T14:33:57.619172+0000","last_clean":"2023-01-15T14:33:57.619172+0000","last_became_active":"2023-01-15T14:33:55.378253+0000","last_became_peered":"2023-01-15T14:33:55.378253+0000","last_unstale":"2023-01-15T14:33:57.619172+0000","last_undegraded":"2023-01-15T14:33:57.619172+0000","last_fullsized":"2023-01-15T14:33:57.619172+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.50935289699999997,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":729,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.341803+0000","last_change":"2023-01-15T14:33:56.342569+0000","last_active":"2023-01-15T14:33:56.341803+0000","last_peered":"2023-01-15T14:33:56.341803+0000","last_clean":"2023-01-15T14:33:56.341803+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:33:56.341803+0000","last_undegraded":"2023-01-15T14:33:56.341803+0000","last_fullsized":"2023-01-15T14:33:56.341803+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.26366071200000002,"stat_sum":{"num_bytes":3802136,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":21818,"num_write":18,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.24","version":"348'259","reported_seq":1392,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.616353+0000","last_change":"2023-01-15T14:33:57.616353+0000","last_active":"2023-01-15T14:33:57.616353+0000","last_peered":"2023-01-15T14:33:57.616353+0000","last_clean":"2023-01-15T14:33:57.616353+0000","last_became_active":"2023-01-15T14:33:45.075309+0000","last_became_peered":"2023-01-15T14:33:45.075309+0000","last_unstale":"2023-01-15T14:33:57.616353+0000","last_undegraded":"2023-01-15T14:33:57.616353+0000","last_fullsized":"2023-01-15T14:33:57.616353+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.70814871000000001,"stat_sum":{"num_bytes":4593906,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10774,"num_write":8,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.25","version":"333'108","reported_seq":1110,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.330557+0000","last_change":"2023-01-15T14:33:57.331159+0000","last_active":"2023-01-15T14:33:57.330557+0000","last_peered":"2023-01-15T14:33:57.330557+0000","last_clean":"2023-01-15T14:33:57.330557+0000","last_became_active":"2023-01-15T14:33:45.168598+0000","last_became_peered":"2023-01-15T14:33:45.168598+0000","last_unstale":"2023-01-15T14:33:57.330557+0000","last_undegraded":"2023-01-15T14:33:57.330557+0000","last_fullsized":"2023-01-15T14:33:57.330557+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.33264791100000002,"stat_sum":{"num_bytes":2638765,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3030,"num_write":2,"num_write_kb":2577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":459,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.008359+0000","last_change":"2023-01-15T14:33:57.008359+0000","last_active":"2023-01-15T14:33:57.008359+0000","last_peered":"2023-01-15T14:33:57.008359+0000","last_clean":"2023-01-15T14:33:57.008359+0000","last_became_active":"2023-01-15T14:33:52.309110+0000","last_became_peered":"2023-01-15T14:33:52.309110+0000","last_unstale":"2023-01-15T14:33:57.008359+0000","last_undegraded":"2023-01-15T14:33:57.008359+0000","last_fullsized":"2023-01-15T14:33:57.008359+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.208200415,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":1061,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.355572+0000","last_change":"2023-01-15T14:33:57.355572+0000","last_active":"2023-01-15T14:33:57.355572+0000","last_peered":"2023-01-15T14:33:57.355572+0000","last_clean":"2023-01-15T14:33:57.355572+0000","last_became_active":"2023-01-15T14:33:45.171090+0000","last_became_peered":"2023-01-15T14:33:45.171090+0000","last_unstale":"2023-01-15T14:33:57.355572+0000","last_undegraded":"2023-01-15T14:33:57.355572+0000","last_fullsized":"2023-01-15T14:33:57.355572+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.443073948,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.28","version":"333'248","reported_seq":1379,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.617361+0000","last_change":"2023-01-15T14:33:57.617361+0000","last_active":"2023-01-15T14:33:57.617361+0000","last_peered":"2023-01-15T14:33:57.617361+0000","last_clean":"2023-01-15T14:33:57.617361+0000","last_became_active":"2023-01-15T14:33:45.366831+0000","last_became_peered":"2023-01-15T14:33:45.366831+0000","last_unstale":"2023-01-15T14:33:57.617361+0000","last_undegraded":"2023-01-15T14:33:57.617361+0000","last_fullsized":"2023-01-15T14:33:57.617361+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:01.610","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.60974191300000002,"stat_sum":{"num_bytes":4838389,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8969,"num_write":8,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":6096764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":883,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.620313+0000","last_change":"2023-01-15T14:33:57.620313+0000","last_active":"2023-01-15T14:33:57.620313+0000","last_peered":"2023-01-15T14:33:57.620313+0000","last_clean":"2023-01-15T14:33:57.620313+0000","last_became_active":"2023-01-15T14:33:45.276960+0000","last_became_peered":"2023-01-15T14:33:45.276960+0000","last_unstale":"2023-01-15T14:33:57.620313+0000","last_undegraded":"2023-01-15T14:33:57.620313+0000","last_fullsized":"2023-01-15T14:33:57.620313+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.51033217900000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1228,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.903452+0000","last_change":"2023-01-15T14:33:56.904528+0000","last_active":"2023-01-15T14:33:56.903452+0000","last_peered":"2023-01-15T14:33:56.903452+0000","last_clean":"2023-01-15T14:33:56.903452+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:33:56.903452+0000","last_undegraded":"2023-01-15T14:33:56.903452+0000","last_fullsized":"2023-01-15T14:33:56.903452+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.0022419219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":25,"reported_epoch":347,"state":"peering","last_fresh":"2023-01-15T14:33:56.080625+0000","last_change":"2023-01-15T14:33:56.080995+0000","last_active":"2023-01-15T14:33:56.080569+0000","last_peered":"2023-01-15T14:33:55.176119+0000","last_clean":"2023-01-15T14:33:55.176119+0000","last_became_active":"2023-01-15T14:33:52.507094+0000","last_became_peered":"2023-01-15T14:33:52.507094+0000","last_unstale":"2023-01-15T14:33:56.080625+0000","last_undegraded":"2023-01-15T14:33:56.080625+0000","last_fullsized":"2023-01-15T14:33:56.080625+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.101837726,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,7],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":432,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.303687+0000","last_change":"2023-01-15T14:33:57.303687+0000","last_active":"2023-01-15T14:33:57.303687+0000","last_peered":"2023-01-15T14:33:57.303687+0000","last_clean":"2023-01-15T14:33:57.303687+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:33:57.303687+0000","last_undegraded":"2023-01-15T14:33:57.303687+0000","last_fullsized":"2023-01-15T14:33:57.303687+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:10.970","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.29451823399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1198,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.007402+0000","last_change":"2023-01-15T14:33:57.007402+0000","last_active":"2023-01-15T14:33:57.007402+0000","last_peered":"2023-01-15T14:33:57.007402+0000","last_clean":"2023-01-15T14:33:57.007402+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:33:57.007402+0000","last_undegraded":"2023-01-15T14:33:57.007402+0000","last_fullsized":"2023-01-15T14:33:57.007402+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.203775807,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":12,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.211505+0000","last_change":"2023-01-15T14:33:57.211055+0000","last_active":"2023-01-15T14:33:57.211505+0000","last_peered":"2023-01-15T14:33:57.211505+0000","last_clean":"2023-01-15T14:33:57.211505+0000","last_became_active":"2023-01-15T14:33:57.210434+0000","last_became_peered":"2023-01-15T14:33:57.210434+0000","last_unstale":"2023-01-15T14:33:57.211505+0000","last_undegraded":"2023-01-15T14:33:57.211505+0000","last_fullsized":"2023-01-15T14:33:57.211505+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.38796315799999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1091,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.009469+0000","last_change":"2023-01-15T14:33:57.009469+0000","last_active":"2023-01-15T14:33:57.009469+0000","last_peered":"2023-01-15T14:33:57.009469+0000","last_clean":"2023-01-15T14:33:57.009469+0000","last_became_active":"2023-01-15T14:33:55.174128+0000","last_became_peered":"2023-01-15T14:33:55.174128+0000","last_unstale":"2023-01-15T14:33:57.009469+0000","last_undegraded":"2023-01-15T14:33:57.009469+0000","last_fullsized":"2023-01-15T14:33:57.009469+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.16584238700000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":967,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.284720+0000","last_change":"2023-01-15T14:33:56.386256+0000","last_active":"2023-01-15T14:33:56.284720+0000","last_peered":"2023-01-15T14:33:56.284720+0000","last_clean":"2023-01-15T14:33:56.284720+0000","last_became_active":"2023-01-15T14:33:55.373498+0000","last_became_peered":"2023-01-15T14:33:55.373498+0000","last_unstale":"2023-01-15T14:33:56.284720+0000","last_undegraded":"2023-01-15T14:33:56.284720+0000","last_fullsized":"2023-01-15T14:33:56.284720+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":7,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.30982162000000002,"stat_sum":{"num_bytes":5419866,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":25948,"num_write":35,"num_write_kb":13827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":13340458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.31","version":"204'150","reported_seq":1214,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.625989+0000","last_change":"2023-01-15T14:33:57.625989+0000","last_active":"2023-01-15T14:33:57.625989+0000","last_peered":"2023-01-15T14:33:57.625989+0000","last_clean":"2023-01-15T14:33:57.625989+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:33:57.625989+0000","last_undegraded":"2023-01-15T14:33:57.625989+0000","last_fullsized":"2023-01-15T14:33:57.625989+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.52721113600000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":106,"stat_sum":{"num_bytes":338336060,"num_objects":152,"num_object_clones":101,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":162,"num_objects_misplaced":34,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":12,"num_read":973,"num_read_kb":1027885,"num_write":779,"num_write_kb":320512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":239498883,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18632,"ondisk_log_size":18632,"up":318,"acting":299,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574920,"num_pgs":44,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":990632,"kb_used_data":92716184,"kb_used_omap":7065,"kb_used_meta":0,"kb_avail":92708748,"statfs":{"total":95948165120,"available":94933757952,"internally_reserved":7614464,"allocated":94941372416,"data_stored":94941372416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7235562,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":4.1269999999999998,"15min":4.0890000000000004},"min":{"1min":0.54000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2030000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":9.5380000000000003,"5min":6.3200000000000003,"15min":8.2460000000000004},"min":{"1min":0.58799999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":96.337000000000003,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.81799999999999995}]},{"osd":1,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":3.6509999999999998,"15min":5.5800000000000001},"min":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":34.585999999999999,"5min":88.003,"15min":88.003},"last":0.79400000000000004},{"interface":"front","average":{"1min":13.302,"5min":8.1950000000000003,"15min":6.9180000000000001},"min":{"1min":0.58799999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.403999999999996,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.98899999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":6.9400000000000004,"15min":5.5069999999999997},"min":{"1min":0.57199999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.179,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91800000000000004,"5min":2.5790000000000002,"15min":3.645},"min":{"1min":0.626,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":91.837999999999994,"15min":91.837999999999994},"last":0.91500000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":7.3799999999999999,"15min":3.0920000000000001},"min":{"1min":0.63600000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2370000000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.91600000000000004,"5min":3.6360000000000001,"15min":4.3129999999999997},"min":{"1min":0.65000000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.2270000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.96399999999999997}]},{"osd":4,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5519999999999996,"5min":5.6219999999999999,"15min":8.1509999999999998},"min":{"1min":0.63500000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":96.185000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.9259999999999999,"5min":5.782,"15min":2.5699999999999998},"min":{"1min":0.73099999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":22.41,"5min":164.21199999999999,"15min":164.21199999999999},"last":142.44999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.6440000000000001,"5min":5.0659999999999998,"15min":5.9059999999999997},"min":{"1min":0.72799999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":96.263999999999996,"5min":96.263999999999996,"15min":96.263999999999996},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.998,"5min":3.206,"15min":4.3650000000000002},"min":{"1min":0.69299999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.355,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.89600000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":5.5949999999999998,"15min":7.5469999999999997},"min":{"1min":0.83599999999999997,"5min":0.496,"15min":0.496},"max":{"1min":96.299999999999997,"5min":96.299999999999997,"15min":96.299999999999997},"last":1.026},{"interface":"front","average":{"1min":1.089,"5min":2.5150000000000001,"15min":1.571},"min":{"1min":0.84299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.2789999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":0.76000000000000001}]}]},{"osd":6,"up_from":0,"seq":55834574921,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574921,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574920,"num_pgs":69,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1070856,"kb_used_data":92651192,"kb_used_omap":13607,"kb_used_meta":0,"kb_avail":92628524,"statfs":{"total":95948165120,"available":94851608576,"internally_reserved":23212032,"allocated":94874820608,"data_stored":94874820608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13933805,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":23.594000000000001,"15min":11.141},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.133,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":3.0009999999999999,"15min":1.5209999999999999},"min":{"1min":0.58299999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.024,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.74399999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":5.0730000000000004,"15min":7.4199999999999999},"min":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6659999999999999,"5min":171.91499999999999,"15min":171.91499999999999},"last":23.193000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.343,"15min":1.9770000000000001},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0529999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":31.756}]},{"osd":2,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":24.227,"15min":11.749000000000001},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.157,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":23.247},{"interface":"front","average":{"1min":2.6040000000000001,"5min":1.21,"15min":1.01},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":32.081000000000003,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.71199999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":2.9169999999999998,"15min":1.865},"min":{"1min":0.65300000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.2909999999999999,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.7370000000000001,"5min":24.498999999999999,"15min":12.266999999999999},"min":{"1min":0.64400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":14.961,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":23.289000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.88500000000000001,"15min":0.90800000000000003},"min":{"1min":0.55200000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.92300000000000004,"5min":2.1840000000000002,"15min":2.2599999999999998},"min":{"1min":0.71799999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3200000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.81899999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":3.9329999999999998,"15min":5.093},"min":{"1min":0.69599999999999995,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.365,"5min":99.780000000000001,"15min":99.780000000000001},"last":23.329000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":22.463999999999999,"15min":10.93},"min":{"1min":0.71599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.427,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":0.92000000000000004}]},{"osd":7,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":3.1539999999999999,"5min":5.5570000000000004,"15min":10.603999999999999},"min":{"1min":0.76700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":39.012999999999998,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.2370000000000001,"5min":24.864000000000001,"15min":15.779999999999999},"min":{"1min":0.877,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.0680000000000001,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":23.356000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":69,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1084580,"kb_used_data":92614920,"kb_used_omap":13867,"kb_used_meta":0,"kb_avail":92614800,"statfs":{"total":95948165120,"available":94837555200,"internally_reserved":122880,"allocated":94837678080,"data_stored":94837678080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14200138,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":7.4400000000000004,"15min":5.7119999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":17.731000000000002,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":3.4340000000000002,"15min":6.2389999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.9260000000000002,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":7.1790000000000003,"15min":11.414},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96799999999999997,"5min":159.256,"15min":159.256},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.6259999999999999,"5min":10.417999999999999,"15min":3.9660000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":17.826000000000001,"5min":174.578,"15min":174.578},"last":25.515999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":4.6849999999999996,"5min":2.8809999999999998,"15min":3.1930000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":86.412000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.95,"5min":6.7889999999999997,"15min":3.7250000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.876000000000001,"5min":174.45099999999999,"15min":174.45099999999999},"last":15.946999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":7.6879999999999997,"5min":29.686,"15min":14.27},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":84.640000000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.86899999999999999,"5min":32.134999999999998,"15min":19.960999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.615,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":17.036000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":6.4980000000000002,"15min":11.433999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.04,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":2.2309999999999999,"5min":6.0039999999999996,"15min":2.6819999999999999},"min":{"1min":0.70099999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":17.917000000000002,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.71899999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1949999999999998,"5min":5.9980000000000002,"15min":2.6869999999999998},"min":{"1min":0.66000000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":17.989000000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.964,"15min":3.1619999999999999},"min":{"1min":0.70499999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.23,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.75600000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":2.0219999999999998,"15min":3.3639999999999999},"min":{"1min":0.69499999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.266,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.878},{"interface":"front","average":{"1min":19.138000000000002,"5min":11.705,"15min":15.961},"min":{"1min":0.66200000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":281.13400000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":0.98599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574920,"num_pgs":62,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1098488,"kb_used_data":92608588,"kb_used_omap":11369,"kb_used_meta":0,"kb_avail":92600892,"statfs":{"total":95948165120,"available":94823313408,"internally_reserved":7880704,"allocated":94831194112,"data_stored":94831194112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11642021,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":3.109,"5min":3.5230000000000001,"15min":1.601},"min":{"1min":0.437,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":51.009999999999998,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.4299999999999999,"15min":0.95299999999999996},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0089999999999999,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.45300000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.1529999999999996,"5min":5.9059999999999997,"15min":2.5009999999999999},"min":{"1min":0.378,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":91.525999999999996,"5min":143.071,"15min":143.071},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":2.016,"15min":2.9089999999999998},"min":{"1min":0.42299999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.131,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.53900000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":8.2210000000000001,"15min":9.6750000000000007},"min":{"1min":0.5,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.3609999999999998,"5min":183.53,"15min":183.53},"last":0.78100000000000003},{"interface":"front","average":{"1min":16.692,"5min":7.5170000000000003,"15min":3.0169999999999999},"min":{"1min":0.61199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":155.47900000000001,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3650000000000002,"5min":6.5060000000000002,"15min":4.6020000000000003},"min":{"1min":0.67300000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":51.259999999999998,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.751},{"interface":"front","average":{"1min":2.8809999999999998,"5min":5.2939999999999996,"15min":4.3040000000000003},"min":{"1min":0.48699999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":33.219999999999999,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.76900000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":2.4729999999999999,"15min":1.6080000000000001},"min":{"1min":0.57199999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2130000000000001,"5min":143.262,"15min":143.262},"last":1.056},{"interface":"front","average":{"1min":3.4780000000000002,"5min":3.0539999999999998,"15min":1.792},"min":{"1min":0.70999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":51.326000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.85099999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.857,"15min":1.3069999999999999},"min":{"1min":0.60399999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.365,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.018,"5min":2.6160000000000001,"15min":1.6839999999999999},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.5,"5min":143.215,"15min":143.215},"last":1.1319999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":13.092000000000001,"5min":6.3120000000000003,"15min":4.1429999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":120.905,"5min":120.905,"15min":120.905},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.0129999999999999,"5min":2.8450000000000002,"15min":2.7989999999999999},"min":{"1min":0.66600000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.552,"5min":143.17099999999999,"15min":143.17099999999999},"last":1.1919999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":72,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1043108,"kb_used_data":92656484,"kb_used_omap":12948,"kb_used_meta":0,"kb_avail":92656272,"statfs":{"total":95948165120,"available":94880022528,"internally_reserved":217088,"allocated":94880239616,"data_stored":94880239616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13258861,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":3.633,"15min":3.806},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":3.9140000000000001,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":7.742,"5min":4.6130000000000004,"15min":4.4189999999999996},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":119.39100000000001,"5min":119.39100000000001,"15min":119.39100000000001},"last":0.91900000000000004}]},{"osd":2,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":2.6469999999999998,"15min":1.5269999999999999},"min":{"1min":0.29799999999999999,"5min":0.245,"15min":0.245},"max":{"1min":95.879999999999995,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.8620000000000001,"15min":3.3359999999999999},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.9780000000000002,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.72999999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.8920000000000003,"5min":9.6750000000000007,"15min":16.449000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":100.137,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":7.782,"5min":6.8899999999999997,"15min":7.1260000000000003},"min":{"1min":0.45600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":119.804,"5min":119.804,"15min":119.804},"last":0.67400000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:33:17 2023","interfaces":[{"interface":"back","average":{"1min":7.9249999999999998,"5min":17.140999999999998,"15min":8.8650000000000002},"min":{"1min":0.504,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":119.71899999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.7849999999999999,"5min":15.526,"15min":7.5780000000000003},"min":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":16.989999999999998,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":1.2070000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.5449999999999999,"15min":1.2030000000000001},"min":{"1min":0.59899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.278,"5min":65.120999999999995,"15min":65.120999999999995},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.6579999999999999,"15min":3.5110000000000001},"min":{"1min":0.60599999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":4.0780000000000003,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.84999999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":8.0069999999999997,"5min":5.0129999999999999,"15min":5.0049999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":119.759,"5min":119.759,"15min":119.759},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.603,"15min":1.2829999999999999},"min":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.349,"5min":65.156000000000006,"15min":65.156000000000006},"last":1.397}]},{"osd":7,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":3.6030000000000002,"5min":6.1109999999999998,"15min":6.7149999999999999},"min":{"1min":0.56000000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":31.620999999999999,"5min":157.03700000000001,"15min":157.03700000000001},"last":9.2409999999999997},{"interface":"front","average":{"1min":8.0779999999999994,"5min":5.0060000000000002,"15min":4.835},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":119.65300000000001,"5min":119.65300000000001,"15min":119.65300000000001},"last":1.252}]}]},{"osd":0,"up_from":13,"seq":55834574921,"num_pgs":75,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":958328,"kb_used_data":92777912,"kb_used_omap":7152,"kb_used_meta":0,"kb_avail":92741052,"statfs":{"total":95948165120,"available":94966837248,"internally_reserved":37744640,"allocated":95004581888,"data_stored":95004581888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7324112,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":3.1219999999999999,"15min":1.456},"min":{"1min":0.40899999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1060000000000001,"5min":69.872,"15min":69.872},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":3.2309999999999999,"15min":1.5629999999999999},"min":{"1min":0.51300000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.2350000000000001,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.434}]},{"osd":2,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":3.4220000000000002,"15min":1.603},"min":{"1min":0.46999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.444,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":2.6560000000000001,"15min":1.3580000000000001},"min":{"1min":0.52000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.302,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.376}]},{"osd":3,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":17.571999999999999,"5min":10.789,"15min":14.782},"min":{"1min":0.53300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":199.08000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.835,"5min":7.306,"15min":2.9689999999999999},"min":{"1min":0.56599999999999995,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":15.601000000000001,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.36699999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":16.695,"15min":6.2220000000000004},"min":{"1min":0.73199999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1879999999999999,"5min":1486.309,"15min":1486.309},"last":0.443},{"interface":"front","average":{"1min":1.038,"5min":17.925000000000001,"15min":6.6580000000000004},"min":{"1min":0.82499999999999996,"5min":0.439,"15min":0.439},"max":{"1min":1.3759999999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":0.63100000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":4.6749999999999998,"15min":8.0899999999999999},"min":{"1min":0.62,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.304,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.008,"5min":1.73,"15min":1.2689999999999999},"min":{"1min":0.69199999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2529999999999999,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.54000000000000004}]},{"osd":6,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":3.5910000000000002,"15min":1.9770000000000001},"min":{"1min":0.67800000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5960000000000001,"5min":70.259,"15min":70.259},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":3.0209999999999999,"15min":1.7629999999999999},"min":{"1min":0.83699999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5489999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":0.64600000000000002}]},{"osd":7,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":6.1200000000000001,"15min":12.487},"min":{"1min":0.78400000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.48,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.117,"5min":3.6070000000000002,"15min":1.9830000000000001},"min":{"1min":0.76100000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5229999999999999,"5min":70.322000000000003,"15min":70.322000000000003},"last":0.77800000000000002}]}]}],"pool_statfs":[]}} 2023-01-15T14:34:00.047 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:34:00.048 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:33:59.547+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:00.048 INFO:tasks.rados.rados.0.smithi090.stdout:1367: done (0 left) 2023-01-15T14:34:00.048 INFO:tasks.rados.rados.0.smithi090.stdout:1368: delete oid 34 current snap is 145 2023-01-15T14:34:00.049 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:00.044+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:00.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:34:00.204 INFO:tasks.rados.rados.0.smithi090.stdout:1368: done (0 left) 2023-01-15T14:34:00.204 INFO:tasks.rados.rados.0.smithi090.stdout:1369: rollback oid 43 current snap is 145 2023-01-15T14:34:00.204 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 43 to 143 2023-01-15T14:34:00.204 INFO:tasks.rados.rados.0.smithi090.stdout:1370: delete oid 1 current snap is 145 2023-01-15T14:34:00.205 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:00.206 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:00.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:34:00.207 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:00.199+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:00.305 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:00.300+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:00.360 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:00.361 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:00.361 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:34:00.362 INFO:tasks.rados.rados.0.smithi090.stdout:1370: done (1 left) 2023-01-15T14:34:00.362 INFO:tasks.rados.rados.0.smithi090.stdout:1371: write (excl) oid 37 current snap is 145 2023-01-15T14:34:00.362 INFO:tasks.rados.rados.0.smithi090.stdout:1371: seq_num 495 ranges {719465=411200,1632822=686643} 2023-01-15T14:34:00.362 INFO:tasks.rados.rados.0.smithi090.stdout:1371: writing smithi09025264-37 from 719465 to 1130665 tid 1 2023-01-15T14:34:00.366 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:00.368 INFO:tasks.rados.rados.0.smithi090.stdout:1371: writing smithi09025264-37 from 1632822 to 2319465 tid 2 2023-01-15T14:34:00.368 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:34:00.369 INFO:tasks.rados.rados.0.smithi090.stdout:1372: copy_from oid 36 from oid 9 current snap is 145 2023-01-15T14:34:00.369 INFO:tasks.rados.rados.0.smithi090.stdout:1369: finishing rollback tid 1 to smithi09025264-43 2023-01-15T14:34:00.369 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 95 (ObjNum -1095659968 snap 22069 seq_num 304) dirty dne 2023-01-15T14:34:00.369 INFO:tasks.rados.rados.0.smithi090.stdout:1369: done (2 left) 2023-01-15T14:34:00.369 INFO:tasks.rados.rados.0.smithi090.stdout:1373: delete oid 5 current snap is 145 2023-01-15T14:34:00.374 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2023-01-15T14:33:59.928372+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":338926428,"num_objects":154,"num_object_clones":101,"num_object_copies":462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":162,"num_objects_misplaced":34,"num_objects_unfound":0,"num_objects_dirty":154,"num_whiteouts":12,"num_read":1089,"num_read_kb":1027985,"num_write":938,"num_write_kb":322928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":415,"num_bytes_recovered":241860355,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18749,"ondisk_log_size":18749,"up":345,"acting":325,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":391,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6245992,"kb_used_data":556025280,"kb_used_omap":66010,"kb_used_meta":0,"kb_avail":555950288,"statfs":{"total":575688990720,"available":569293094912,"internally_reserved":76791808,"allocated":569369886720,"data_stored":569369886720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":67594499,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":13,"commit_latency_ns":0,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42402958,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":-10,"num_objects_degraded":146,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":151,"num_read_kb":168767,"num_write":96,"num_write_kb":45607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":63588880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.856379"},"pg_stats":[{"pgid":"3.69","version":"110'149","reported_seq":895,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.316235+0000","last_change":"2023-01-15T14:33:57.316235+0000","last_active":"2023-01-15T14:33:57.316235+0000","last_peered":"2023-01-15T14:33:57.316235+0000","last_clean":"2023-01-15T14:33:57.316235+0000","last_became_active":"2023-01-15T14:33:50.216417+0000","last_became_peered":"2023-01-15T14:33:50.216417+0000","last_unstale":"2023-01-15T14:33:57.316235+0000","last_undegraded":"2023-01-15T14:33:57.316235+0000","last_fullsized":"2023-01-15T14:33:57.316235+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.406510072,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.68","version":"345'250","reported_seq":1396,"reported_epoch":347,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2023-01-15T14:33:56.832472+0000","last_change":"2023-01-15T14:33:53.618260+0000","last_active":"2023-01-15T14:33:56.832472+0000","last_peered":"2023-01-15T14:33:56.832472+0000","last_clean":"2023-01-15T14:33:42.635266+0000","last_became_active":"2023-01-15T14:33:53.330670+0000","last_became_peered":"2023-01-15T14:33:53.330670+0000","last_unstale":"2023-01-15T14:33:56.832472+0000","last_undegraded":"2023-01-15T14:33:52.928061+0000","last_fullsized":"2023-01-15T14:33:56.832472+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8014979,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":21385,"num_write":10,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":9643107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[1,7,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.67","version":"191'79","reported_seq":48,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.114675+0000","last_change":"2023-01-15T14:33:57.312535+0000","last_active":"2023-01-15T14:33:57.114675+0000","last_peered":"2023-01-15T14:33:57.114675+0000","last_clean":"2023-01-15T14:33:57.114675+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:33:57.114675+0000","last_undegraded":"2023-01-15T14:33:57.114675+0000","last_fullsized":"2023-01-15T14:33:57.114675+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.40508863699999997,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.66","version":"102'52","reported_seq":358,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.299510+0000","last_change":"2023-01-15T14:33:57.299510+0000","last_active":"2023-01-15T14:33:57.299510+0000","last_peered":"2023-01-15T14:33:57.299510+0000","last_clean":"2023-01-15T14:33:57.299510+0000","last_became_active":"2023-01-15T14:33:48.695009+0000","last_became_peered":"2023-01-15T14:33:48.695009+0000","last_unstale":"2023-01-15T14:33:57.299510+0000","last_undegraded":"2023-01-15T14:33:57.299510+0000","last_fullsized":"2023-01-15T14:33:57.299510+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.390920724,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2142,"num_write":4,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.65","version":"348'116","reported_seq":1147,"reported_epoch":348,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:57.676726+0000","last_change":"2023-01-15T14:33:57.203450+0000","last_active":"2023-01-15T14:33:57.676726+0000","last_peered":"2023-01-15T14:33:57.676726+0000","last_clean":"2023-01-15T14:33:43.349693+0000","last_became_active":"2023-01-15T14:33:57.201006+0000","last_became_peered":"2023-01-15T14:33:57.201006+0000","last_unstale":"2023-01-15T14:33:57.676726+0000","last_undegraded":"2023-01-15T14:33:56.909186+0000","last_fullsized":"2023-01-15T14:33:56.908621+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6376869,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":3032,"num_write":8,"num_write_kb":3372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":11126498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,4],"avail_no_missing":["1","2","3","4"],"object_location_counts":[{"shards":"0,1,2,3,4","objects":1},{"shards":"1,2,3,4","objects":1},{"shards":"1,2,4","objects":1},{"shards":"1,4","objects":-1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.64","version":"334'251","reported_seq":1400,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.397056+0000","last_change":"2023-01-15T14:33:56.397607+0000","last_active":"2023-01-15T14:33:56.397056+0000","last_peered":"2023-01-15T14:33:56.397056+0000","last_clean":"2023-01-15T14:33:56.397056+0000","last_became_active":"2023-01-15T14:33:55.675085+0000","last_became_peered":"2023-01-15T14:33:55.675085+0000","last_unstale":"2023-01-15T14:33:56.397056+0000","last_undegraded":"2023-01-15T14:33:56.397056+0000","last_fullsized":"2023-01-15T14:33:56.397056+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.216593806,"stat_sum":{"num_bytes":3548733,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10773,"num_write":5,"num_write_kb":2741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.63","version":"149'233","reported_seq":729,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.394284+0000","last_change":"2023-01-15T14:33:56.394984+0000","last_active":"2023-01-15T14:33:56.394284+0000","last_peered":"2023-01-15T14:33:56.394284+0000","last_clean":"2023-01-15T14:33:56.394284+0000","last_became_active":"2023-01-15T14:33:55.277739+0000","last_became_peered":"2023-01-15T14:33:55.277739+0000","last_unstale":"2023-01-15T14:33:56.394284+0000","last_undegraded":"2023-01-15T14:33:56.394284+0000","last_fullsized":"2023-01-15T14:33:56.394284+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31355392500000001,"stat_sum":{"num_bytes":3802136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":21819,"num_write":19,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.62","version":"347'167","reported_seq":1324,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.625124+0000","last_change":"2023-01-15T14:33:57.625342+0000","last_active":"2023-01-15T14:33:57.625124+0000","last_peered":"2023-01-15T14:33:57.625124+0000","last_clean":"2023-01-15T14:33:57.625124+0000","last_became_active":"2023-01-15T14:33:56.590183+0000","last_became_peered":"2023-01-15T14:33:56.590183+0000","last_unstale":"2023-01-15T14:33:57.625124+0000","last_undegraded":"2023-01-15T14:33:57.625124+0000","last_fullsized":"2023-01-15T14:33:57.625124+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.62383204000000003,"stat_sum":{"num_bytes":5361549,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6734,"num_write":11,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5210269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.61","version":"321'138","reported_seq":990,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.444780+0000","last_change":"2023-01-15T14:33:56.444780+0000","last_active":"2023-01-15T14:33:56.444780+0000","last_peered":"2023-01-15T14:33:56.444780+0000","last_clean":"2023-01-15T14:33:56.444780+0000","last_became_active":"2023-01-15T14:33:53.421056+0000","last_became_peered":"2023-01-15T14:33:53.421056+0000","last_unstale":"2023-01-15T14:33:56.444780+0000","last_undegraded":"2023-01-15T14:33:56.444780+0000","last_fullsized":"2023-01-15T14:33:56.444780+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.36564521799999999,"stat_sum":{"num_bytes":1033664,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6059,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.60","version":"347'279","reported_seq":1387,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.330453+0000","last_change":"2023-01-15T14:33:57.330518+0000","last_active":"2023-01-15T14:33:57.330453+0000","last_peered":"2023-01-15T14:33:57.330453+0000","last_clean":"2023-01-15T14:33:57.330453+0000","last_became_active":"2023-01-15T14:33:56.478613+0000","last_became_peered":"2023-01-15T14:33:56.478613+0000","last_unstale":"2023-01-15T14:33:57.330453+0000","last_undegraded":"2023-01-15T14:33:57.330453+0000","last_fullsized":"2023-01-15T14:33:57.330453+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.232258925,"stat_sum":{"num_bytes":5214108,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":10865,"num_write":14,"num_write_kb":5082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15418818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1363,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.446925+0000","last_change":"2023-01-15T14:33:56.447607+0000","last_active":"2023-01-15T14:33:56.446925+0000","last_peered":"2023-01-15T14:33:56.446925+0000","last_clean":"2023-01-15T14:33:56.446925+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:33:56.446925+0000","last_undegraded":"2023-01-15T14:33:56.446925+0000","last_fullsized":"2023-01-15T14:33:56.446925+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:51.919","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.065866312999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5e","version":"215'39","reported_seq":974,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.316972+0000","last_change":"2023-01-15T14:33:57.316972+0000","last_active":"2023-01-15T14:33:57.316972+0000","last_peered":"2023-01-15T14:33:57.316972+0000","last_clean":"2023-01-15T14:33:57.316972+0000","last_became_active":"2023-01-15T14:33:35.364143+0000","last_became_peered":"2023-01-15T14:33:35.364143+0000","last_unstale":"2023-01-15T14:33:57.316972+0000","last_undegraded":"2023-01-15T14:33:57.316972+0000","last_fullsized":"2023-01-15T14:33:57.316972+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40693008000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1285,"reported_epoch":349,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:33:58.213615+0000","last_change":"2023-01-15T14:33:58.213117+0000","last_active":"2023-01-15T14:33:58.213615+0000","last_peered":"2023-01-15T14:33:58.213615+0000","last_clean":"2023-01-15T14:33:55.169917+0000","last_became_active":"2023-01-15T14:33:58.213117+0000","last_became_peered":"2023-01-15T14:33:58.213117+0000","last_unstale":"2023-01-15T14:33:58.213615+0000","last_undegraded":"2023-01-15T14:33:58.213615+0000","last_fullsized":"2023-01-15T14:33:57.907045+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10421003,"stat_sum":{"num_bytes":2869928,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":23811,"num_write":13,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,0],"avail_no_missing":["4","0","1","7"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.5c","version":"141'46","reported_seq":75,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.009446+0000","last_change":"2023-01-15T14:33:58.026285+0000","last_active":"2023-01-15T14:33:58.009446+0000","last_peered":"2023-01-15T14:33:58.009446+0000","last_clean":"2023-01-15T14:33:58.009446+0000","last_became_active":"2023-01-15T14:33:57.113032+0000","last_became_peered":"2023-01-15T14:33:57.113032+0000","last_unstale":"2023-01-15T14:33:58.009446+0000","last_undegraded":"2023-01-15T14:33:58.009446+0000","last_fullsized":"2023-01-15T14:33:58.009446+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.121922606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5b","version":"283'80","reported_seq":905,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.005338+0000","last_change":"2023-01-15T14:33:57.113633+0000","last_active":"2023-01-15T14:33:57.005338+0000","last_peered":"2023-01-15T14:33:57.005338+0000","last_clean":"2023-01-15T14:33:57.005338+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:33:57.005338+0000","last_undegraded":"2023-01-15T14:33:57.005338+0000","last_fullsized":"2023-01-15T14:33:57.005338+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.20934807,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5a","version":"329'348","reported_seq":1474,"reported_epoch":347,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.488419+0000","last_change":"2023-01-15T14:33:53.433694+0000","last_active":"2023-01-15T14:33:56.488419+0000","last_peered":"2023-01-15T14:33:56.488419+0000","last_clean":"2023-01-15T14:33:50.015328+0000","last_became_active":"2023-01-15T14:33:53.229261+0000","last_became_peered":"2023-01-15T14:33:53.229261+0000","last_unstale":"2023-01-15T14:33:56.488419+0000","last_undegraded":"2023-01-15T14:33:52.914214+0000","last_fullsized":"2023-01-15T14:33:52.913106+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:02.755","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.45505085000000001,"stat_sum":{"num_bytes":5983424,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[1,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.59","version":"294'256","reported_seq":1272,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.406685+0000","last_change":"2023-01-15T14:33:57.406685+0000","last_active":"2023-01-15T14:33:57.406685+0000","last_peered":"2023-01-15T14:33:57.406685+0000","last_clean":"2023-01-15T14:33:57.406685+0000","last_became_active":"2023-01-15T14:33:55.671990+0000","last_became_peered":"2023-01-15T14:33:55.671990+0000","last_unstale":"2023-01-15T14:33:57.406685+0000","last_undegraded":"2023-01-15T14:33:57.406685+0000","last_fullsized":"2023-01-15T14:33:57.406685+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.49887967,"stat_sum":{"num_bytes":1210137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":5,"num_read_kb":3549,"num_write":5,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.58","version":"323'320","reported_seq":1474,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.901143+0000","last_change":"2023-01-15T14:33:57.004059+0000","last_active":"2023-01-15T14:33:56.901143+0000","last_peered":"2023-01-15T14:33:56.901143+0000","last_clean":"2023-01-15T14:33:56.901143+0000","last_became_active":"2023-01-15T14:33:40.127669+0000","last_became_peered":"2023-01-15T14:33:40.127669+0000","last_unstale":"2023-01-15T14:33:56.901143+0000","last_undegraded":"2023-01-15T14:33:56.901143+0000","last_fullsized":"2023-01-15T14:33:56.901143+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.103642679,"stat_sum":{"num_bytes":4332079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.57","version":"347'95","reported_seq":79,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.444988+0000","last_change":"2023-01-15T14:33:56.445824+0000","last_active":"2023-01-15T14:33:56.444988+0000","last_peered":"2023-01-15T14:33:56.444988+0000","last_clean":"2023-01-15T14:33:56.444988+0000","last_became_active":"2023-01-15T14:33:36.994447+0000","last_became_peered":"2023-01-15T14:33:36.994447+0000","last_unstale":"2023-01-15T14:33:56.444988+0000","last_undegraded":"2023-01-15T14:33:56.444988+0000","last_fullsized":"2023-01-15T14:33:56.444988+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.160441256,"stat_sum":{"num_bytes":1512663,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3025326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.56","version":"320'127","reported_seq":602,"reported_epoch":344,"state":"activating+undersized+remapped","last_fresh":"2023-01-15T14:33:52.929175+0000","last_change":"2023-01-15T14:33:52.929424+0000","last_active":"2023-01-15T14:33:51.236802+0000","last_peered":"2023-01-15T14:33:50.013625+0000","last_clean":"2023-01-15T14:33:50.013625+0000","last_became_active":"2023-01-15T14:33:40.152319+0000","last_became_peered":"2023-01-15T14:33:40.152319+0000","last_unstale":"2023-01-15T14:33:52.929175+0000","last_undegraded":"2023-01-15T14:33:52.929175+0000","last_fullsized":"2023-01-15T14:33:52.929175+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.91022528000000003,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4],"avail_no_missing":["3","0","1","4"],"object_location_counts":[{"shards":"3,4","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.55","version":"110'55","reported_seq":1005,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.315559+0000","last_change":"2023-01-15T14:33:57.315559+0000","last_active":"2023-01-15T14:33:57.315559+0000","last_peered":"2023-01-15T14:33:57.315559+0000","last_clean":"2023-01-15T14:33:57.315559+0000","last_became_active":"2023-01-15T14:33:52.510429+0000","last_became_peered":"2023-01-15T14:33:52.510429+0000","last_unstale":"2023-01-15T14:33:57.315559+0000","last_undegraded":"2023-01-15T14:33:57.315559+0000","last_fullsized":"2023-01-15T14:33:57.315559+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.40629608699999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1119,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.499425+0000","last_change":"2023-01-15T14:33:56.501463+0000","last_active":"2023-01-15T14:33:56.499425+0000","last_peered":"2023-01-15T14:33:56.499425+0000","last_clean":"2023-01-15T14:33:56.499425+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:33:56.499425+0000","last_undegraded":"2023-01-15T14:33:56.499425+0000","last_fullsized":"2023-01-15T14:33:56.499425+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:33:45.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:45.583005+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.321363183,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.53","version":"343'389","reported_seq":1384,"reported_epoch":348,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:56.905391+0000","last_change":"2023-01-15T14:33:56.905760+0000","last_active":"2023-01-15T14:33:56.905347+0000","last_peered":"2023-01-15T14:33:55.165228+0000","last_clean":"2023-01-15T14:33:55.165228+0000","last_became_active":"2023-01-15T14:33:28.184994+0000","last_became_peered":"2023-01-15T14:33:28.184994+0000","last_unstale":"2023-01-15T14:33:56.905391+0000","last_undegraded":"2023-01-15T14:33:56.905391+0000","last_fullsized":"2023-01-15T14:33:56.905391+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:42.205","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.092650028999999995,"stat_sum":{"num_bytes":21535378,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":17077,"num_write":13,"num_write_kb":6760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,7],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":724,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.902208+0000","last_change":"2023-01-15T14:33:56.903307+0000","last_active":"2023-01-15T14:33:56.902208+0000","last_peered":"2023-01-15T14:33:56.902208+0000","last_clean":"2023-01-15T14:33:56.902208+0000","last_became_active":"2023-01-15T14:33:55.472400+0000","last_became_peered":"2023-01-15T14:33:55.472400+0000","last_unstale":"2023-01-15T14:33:56.902208+0000","last_undegraded":"2023-01-15T14:33:56.902208+0000","last_fullsized":"2023-01-15T14:33:56.902208+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.001336634,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.51","version":"348'316","reported_seq":1511,"reported_epoch":348,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:57.332202+0000","last_change":"2023-01-15T14:33:57.332202+0000","last_active":"2023-01-15T14:33:57.332202+0000","last_peered":"2023-01-15T14:33:57.332202+0000","last_clean":"2023-01-15T14:33:57.332202+0000","last_became_active":"2023-01-15T14:33:26.107940+0000","last_became_peered":"2023-01-15T14:33:26.107940+0000","last_unstale":"2023-01-15T14:33:57.332202+0000","last_undegraded":"2023-01-15T14:33:57.332202+0000","last_fullsized":"2023-01-15T14:33:57.332202+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:36.458","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.70457060199999999,"stat_sum":{"num_bytes":7496813,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.50","version":"337'420","reported_seq":1523,"reported_epoch":348,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:33:57.812867+0000","last_change":"2023-01-15T14:33:56.720993+0000","last_active":"2023-01-15T14:33:57.812867+0000","last_peered":"2023-01-15T14:33:57.812867+0000","last_clean":"2023-01-15T14:33:50.116263+0000","last_became_active":"2023-01-15T14:33:53.229903+0000","last_became_peered":"2023-01-15T14:33:53.229903+0000","last_unstale":"2023-01-15T14:33:57.812867+0000","last_undegraded":"2023-01-15T14:33:57.812867+0000","last_fullsized":"2023-01-15T14:33:52.915062+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":321,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.70524759299999995,"stat_sum":{"num_bytes":9681416,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":42160,"num_write":49,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":22053159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,3],"avail_no_missing":["7","3","4"],"object_location_counts":[{"shards":"3,4,7","objects":6},{"shards":"3,7","objects":-3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.4f","version":"155'145","reported_seq":822,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.353637+0000","last_change":"2023-01-15T14:33:57.353637+0000","last_active":"2023-01-15T14:33:57.353637+0000","last_peered":"2023-01-15T14:33:57.353637+0000","last_clean":"2023-01-15T14:33:57.353637+0000","last_became_active":"2023-01-15T14:33:25.129936+0000","last_became_peered":"2023-01-15T14:33:25.129936+0000","last_unstale":"2023-01-15T14:33:57.353637+0000","last_undegraded":"2023-01-15T14:33:57.353637+0000","last_fullsized":"2023-01-15T14:33:57.353637+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44322561900000002,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":87,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.387715+0000","last_change":"2023-01-15T14:33:56.393769+0000","last_active":"2023-01-15T14:33:56.387715+0000","last_peered":"2023-01-15T14:33:56.387715+0000","last_clean":"2023-01-15T14:33:56.387715+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:33:56.387715+0000","last_undegraded":"2023-01-15T14:33:56.387715+0000","last_fullsized":"2023-01-15T14:33:56.387715+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.31362827599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4d","version":"344'254","reported_seq":1310,"reported_epoch":344,"state":"active+clean","last_fresh":"2023-01-15T14:33:53.778919+0000","last_change":"2023-01-15T14:33:52.103984+0000","last_active":"2023-01-15T14:33:53.778919+0000","last_peered":"2023-01-15T14:33:53.778919+0000","last_clean":"2023-01-15T14:33:53.778919+0000","last_became_active":"2023-01-15T14:33:26.102614+0000","last_became_peered":"2023-01-15T14:33:26.102614+0000","last_unstale":"2023-01-15T14:33:53.778919+0000","last_undegraded":"2023-01-15T14:33:53.778919+0000","last_fullsized":"2023-01-15T14:33:53.778919+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.104242925,"stat_sum":{"num_bytes":9048419,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":15961,"num_write":16,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11291942,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.4c","version":"309'71","reported_seq":101,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.312666+0000","last_change":"2023-01-15T14:33:57.413785+0000","last_active":"2023-01-15T14:33:57.312666+0000","last_peered":"2023-01-15T14:33:57.312666+0000","last_clean":"2023-01-15T14:33:57.312666+0000","last_became_active":"2023-01-15T14:33:52.311859+0000","last_became_peered":"2023-01-15T14:33:52.311859+0000","last_unstale":"2023-01-15T14:33:57.312666+0000","last_undegraded":"2023-01-15T14:33:57.312666+0000","last_fullsized":"2023-01-15T14:33:57.312666+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.40874315500000002,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":747,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.626691+0000","last_change":"2023-01-15T14:33:57.627070+0000","last_active":"2023-01-15T14:33:57.626691+0000","last_peered":"2023-01-15T14:33:57.626691+0000","last_clean":"2023-01-15T14:33:57.626691+0000","last_became_active":"2023-01-15T14:33:52.406605+0000","last_became_peered":"2023-01-15T14:33:52.406605+0000","last_unstale":"2023-01-15T14:33:57.626691+0000","last_undegraded":"2023-01-15T14:33:57.626691+0000","last_fullsized":"2023-01-15T14:33:57.626691+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:42.594","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.515074951,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4a","version":"348'371","reported_seq":1469,"reported_epoch":348,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:57.415696+0000","last_change":"2023-01-15T14:33:56.904648+0000","last_active":"2023-01-15T14:33:57.415696+0000","last_peered":"2023-01-15T14:33:57.415696+0000","last_clean":"2023-01-15T14:33:57.415696+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:33:57.415696+0000","last_undegraded":"2023-01-15T14:33:57.415696+0000","last_fullsized":"2023-01-15T14:33:57.415696+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:33.328","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81443059699999998,"stat_sum":{"num_bytes":12470870,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":63,"num_read_kb":67536,"num_write":73,"num_write_kb":23626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":881,"reported_epoch":343,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:51.920225+0000","last_change":"2023-01-15T14:33:51.920225+0000","last_active":"2023-01-15T14:32:23.854468+0000","last_peered":"2023-01-15T14:32:22.960105+0000","last_clean":"2023-01-15T14:32:22.960105+0000","last_became_active":"2023-01-15T14:32:22.148707+0000","last_became_peered":"2023-01-15T14:32:22.148707+0000","last_unstale":"2023-01-15T14:33:51.920225+0000","last_undegraded":"2023-01-15T14:33:51.920225+0000","last_fullsized":"2023-01-15T14:33:51.920225+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.006207738,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"f9","length":"2"},{"start":"100","length":"2"},{"start":"106","length":"1"}]},{"pgid":"3.48","version":"179'204","reported_seq":1353,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.356590+0000","last_change":"2023-01-15T14:33:57.356590+0000","last_active":"2023-01-15T14:33:57.356590+0000","last_peered":"2023-01-15T14:33:57.356590+0000","last_clean":"2023-01-15T14:33:57.356590+0000","last_became_active":"2023-01-15T14:33:55.375518+0000","last_became_peered":"2023-01-15T14:33:55.375518+0000","last_unstale":"2023-01-15T14:33:57.356590+0000","last_undegraded":"2023-01-15T14:33:57.356590+0000","last_fullsized":"2023-01-15T14:33:57.356590+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:43.662","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.34933974800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1113,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.300260+0000","last_change":"2023-01-15T14:33:57.300260+0000","last_active":"2023-01-15T14:33:57.300260+0000","last_peered":"2023-01-15T14:33:57.300260+0000","last_clean":"2023-01-15T14:33:57.300260+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:33:57.300260+0000","last_undegraded":"2023-01-15T14:33:57.300260+0000","last_fullsized":"2023-01-15T14:33:57.300260+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:03.200","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.39166528900000003,"stat_sum":{"num_bytes":496762,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":5,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":1023,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.728559+0000","last_change":"2023-01-15T14:33:57.829426+0000","last_active":"2023-01-15T14:33:57.728559+0000","last_peered":"2023-01-15T14:33:57.728559+0000","last_clean":"2023-01-15T14:33:57.728559+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:33:57.728559+0000","last_undegraded":"2023-01-15T14:33:57.728559+0000","last_fullsized":"2023-01-15T14:33:57.728559+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:55.245","scrub_duration":0.30448409700000001,"objects_trimmed":0,"snaptrim_duration":0.714760332,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":904,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.396257+0000","last_change":"2023-01-15T14:33:56.396637+0000","last_active":"2023-01-15T14:33:56.396257+0000","last_peered":"2023-01-15T14:33:56.396257+0000","last_clean":"2023-01-15T14:33:56.396257+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:33:56.396257+0000","last_undegraded":"2023-01-15T14:33:56.396257+0000","last_fullsized":"2023-01-15T14:33:56.396257+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21706618899999999,"stat_sum":{"num_bytes":2635976,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":19927,"num_write":15,"num_write_kb":6659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4745860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1268,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.298701+0000","last_change":"2023-01-15T14:33:57.298701+0000","last_active":"2023-01-15T14:33:57.298701+0000","last_peered":"2023-01-15T14:33:57.298701+0000","last_clean":"2023-01-15T14:33:57.298701+0000","last_became_active":"2023-01-15T14:33:55.678019+0000","last_became_peered":"2023-01-15T14:33:55.678019+0000","last_unstale":"2023-01-15T14:33:57.298701+0000","last_undegraded":"2023-01-15T14:33:57.298701+0000","last_fullsized":"2023-01-15T14:33:57.298701+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:27.120","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.390225449,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.43","version":"297'314","reported_seq":1458,"reported_epoch":347,"state":"peering","last_fresh":"2023-01-15T14:33:56.235641+0000","last_change":"2023-01-15T14:33:56.236004+0000","last_active":"2023-01-15T14:33:56.235585+0000","last_peered":"2023-01-15T14:33:55.176514+0000","last_clean":"2023-01-15T14:33:55.176514+0000","last_became_active":"2023-01-15T14:32:28.977084+0000","last_became_peered":"2023-01-15T14:32:28.977084+0000","last_unstale":"2023-01-15T14:33:56.235641+0000","last_undegraded":"2023-01-15T14:33:56.235641+0000","last_fullsized":"2023-01-15T14:33:56.235641+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.101016546,"stat_sum":{"num_bytes":3395080,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":16585,"num_write":5,"num_write_kb":1351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,7],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":1029,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.621398+0000","last_change":"2023-01-15T14:33:57.621398+0000","last_active":"2023-01-15T14:33:57.621398+0000","last_peered":"2023-01-15T14:33:57.621398+0000","last_clean":"2023-01-15T14:33:57.621398+0000","last_became_active":"2023-01-15T14:33:55.376782+0000","last_became_peered":"2023-01-15T14:33:55.376782+0000","last_unstale":"2023-01-15T14:33:57.621398+0000","last_undegraded":"2023-01-15T14:33:57.621398+0000","last_fullsized":"2023-01-15T14:33:57.621398+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.51101871399999999,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1109,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.908053+0000","last_change":"2023-01-15T14:33:57.302504+0000","last_active":"2023-01-15T14:33:57.908053+0000","last_peered":"2023-01-15T14:33:57.908053+0000","last_clean":"2023-01-15T14:33:57.908053+0000","last_became_active":"2023-01-15T14:33:55.771969+0000","last_became_peered":"2023-01-15T14:33:55.771969+0000","last_unstale":"2023-01-15T14:33:57.908053+0000","last_undegraded":"2023-01-15T14:33:57.908053+0000","last_fullsized":"2023-01-15T14:33:57.908053+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.29405439700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.40","version":"335'307","reported_seq":1383,"reported_epoch":348,"state":"peering","last_fresh":"2023-01-15T14:33:57.104195+0000","last_change":"2023-01-15T14:33:57.002620+0000","last_active":"2023-01-15T14:33:57.002126+0000","last_peered":"2023-01-15T14:33:43.952880+0000","last_clean":"2023-01-15T14:33:43.952880+0000","last_became_active":"2023-01-15T14:33:14.198194+0000","last_became_peered":"2023-01-15T14:33:14.198194+0000","last_unstale":"2023-01-15T14:33:57.104195+0000","last_undegraded":"2023-01-15T14:33:57.104195+0000","last_fullsized":"2023-01-15T14:33:57.104195+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.704289622,"stat_sum":{"num_bytes":12753904,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":35541,"num_write":28,"num_write_kb":11001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":16413587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.3f","version":"321'233","reported_seq":1339,"reported_epoch":348,"state":"peering","last_fresh":"2023-01-15T14:33:57.010251+0000","last_change":"2023-01-15T14:33:57.010838+0000","last_active":"2023-01-15T14:33:57.010178+0000","last_peered":"2023-01-15T14:33:56.616086+0000","last_clean":"2023-01-15T14:33:50.017023+0000","last_became_active":"2023-01-15T14:33:53.022391+0000","last_became_peered":"2023-01-15T14:33:53.022391+0000","last_unstale":"2023-01-15T14:33:57.010251+0000","last_undegraded":"2023-01-15T14:33:57.010251+0000","last_fullsized":"2023-01-15T14:33:57.010251+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":236,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:29.095","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.810287749,"stat_sum":{"num_bytes":4764997,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6605283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.3e","version":"294'63","reported_seq":828,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.627138+0000","last_change":"2023-01-15T14:33:57.728361+0000","last_active":"2023-01-15T14:33:57.627138+0000","last_peered":"2023-01-15T14:33:57.627138+0000","last_clean":"2023-01-15T14:33:57.627138+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:33:57.627138+0000","last_undegraded":"2023-01-15T14:33:57.627138+0000","last_fullsized":"2023-01-15T14:33:57.627138+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.61381065800000001,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3d","version":"335'232","reported_seq":1310,"reported_epoch":348,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.998631+0000","last_change":"2023-01-15T14:33:53.127064+0000","last_active":"2023-01-15T14:33:56.998631+0000","last_peered":"2023-01-15T14:33:56.998631+0000","last_clean":"2023-01-15T14:33:50.059397+0000","last_became_active":"2023-01-15T14:33:53.126274+0000","last_became_peered":"2023-01-15T14:33:53.126274+0000","last_unstale":"2023-01-15T14:33:56.998631+0000","last_undegraded":"2023-01-15T14:33:52.928065+0000","last_fullsized":"2023-01-15T14:33:52.926980+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.15835743399999999,"stat_sum":{"num_bytes":8050279,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":13797,"num_write":16,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[1,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":345,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.904495+0000","last_change":"2023-01-15T14:33:56.912541+0000","last_active":"2023-01-15T14:33:57.904495+0000","last_peered":"2023-01-15T14:33:57.904495+0000","last_clean":"2023-01-15T14:33:57.904495+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:33:57.904495+0000","last_undegraded":"2023-01-15T14:33:57.904495+0000","last_fullsized":"2023-01-15T14:33:57.904495+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.470","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.0047236029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":12,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.014266+0000","last_change":"2023-01-15T14:33:57.014068+0000","last_active":"2023-01-15T14:33:57.014266+0000","last_peered":"2023-01-15T14:33:57.014266+0000","last_clean":"2023-01-15T14:33:57.014266+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:33:57.014266+0000","last_undegraded":"2023-01-15T14:33:57.014266+0000","last_fullsized":"2023-01-15T14:33:57.014266+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.17081586700000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1338,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.395588+0000","last_change":"2023-01-15T14:33:56.396153+0000","last_active":"2023-01-15T14:33:56.395588+0000","last_peered":"2023-01-15T14:33:56.395588+0000","last_clean":"2023-01-15T14:33:56.395588+0000","last_became_active":"2023-01-15T14:33:55.474944+0000","last_became_peered":"2023-01-15T14:33:55.474944+0000","last_unstale":"2023-01-15T14:33:56.395588+0000","last_undegraded":"2023-01-15T14:33:56.395588+0000","last_fullsized":"2023-01-15T14:33:56.395588+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.21712495700000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.39","version":"334'351","reported_seq":1388,"reported_epoch":347,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.185894+0000","last_change":"2023-01-15T14:33:53.440119+0000","last_active":"2023-01-15T14:33:56.185894+0000","last_peered":"2023-01-15T14:33:56.185894+0000","last_clean":"2023-01-15T14:33:50.016390+0000","last_became_active":"2023-01-15T14:33:53.436135+0000","last_became_peered":"2023-01-15T14:33:53.436135+0000","last_unstale":"2023-01-15T14:33:56.185894+0000","last_undegraded":"2023-01-15T14:33:53.017325+0000","last_fullsized":"2023-01-15T14:33:53.015555+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:21.301","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.0051603159999999999,"stat_sum":{"num_bytes":24023444,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":43987,"num_write":7,"num_write_kb":7292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,2],"avail_no_missing":["7","2","4"],"object_location_counts":[{"shards":"2,4,7","objects":9}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.38","version":"348'311","reported_seq":1523,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.959969+0000","last_change":"2023-01-15T14:33:58.009343+0000","last_active":"2023-01-15T14:33:57.959969+0000","last_peered":"2023-01-15T14:33:57.959969+0000","last_clean":"2023-01-15T14:33:57.959969+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:33:57.959969+0000","last_undegraded":"2023-01-15T14:33:57.959969+0000","last_fullsized":"2023-01-15T14:33:57.959969+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.10683759900000001,"stat_sum":{"num_bytes":7580089,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":42847,"num_write":35,"num_write_kb":12593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":401,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.109903+0000","last_change":"2023-01-15T14:33:57.109903+0000","last_active":"2023-01-15T14:33:57.109903+0000","last_peered":"2023-01-15T14:33:57.109903+0000","last_clean":"2023-01-15T14:33:57.109903+0000","last_became_active":"2023-01-15T14:33:52.412128+0000","last_became_peered":"2023-01-15T14:33:52.412128+0000","last_unstale":"2023-01-15T14:33:57.109903+0000","last_undegraded":"2023-01-15T14:33:57.109903+0000","last_fullsized":"2023-01-15T14:33:57.109903+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.208440876,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":556,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.331276+0000","last_change":"2023-01-15T14:33:57.332104+0000","last_active":"2023-01-15T14:33:57.331276+0000","last_peered":"2023-01-15T14:33:57.331276+0000","last_clean":"2023-01-15T14:33:57.331276+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:33:57.331276+0000","last_undegraded":"2023-01-15T14:33:57.331276+0000","last_fullsized":"2023-01-15T14:33:57.331276+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.31780763299999998,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":27,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.342717+0000","last_change":"2023-01-15T14:33:56.343772+0000","last_active":"2023-01-15T14:33:56.342717+0000","last_peered":"2023-01-15T14:33:56.342717+0000","last_clean":"2023-01-15T14:33:56.342717+0000","last_became_active":"2023-01-15T14:33:52.612576+0000","last_became_peered":"2023-01-15T14:33:52.612576+0000","last_unstale":"2023-01-15T14:33:56.342717+0000","last_undegraded":"2023-01-15T14:33:56.342717+0000","last_fullsized":"2023-01-15T14:33:56.342717+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.26469577999999999,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.34","version":"169'138","reported_seq":1152,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.301685+0000","last_change":"2023-01-15T14:33:57.301685+0000","last_active":"2023-01-15T14:33:57.301685+0000","last_peered":"2023-01-15T14:33:57.301685+0000","last_clean":"2023-01-15T14:33:57.301685+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:33:57.301685+0000","last_undegraded":"2023-01-15T14:33:57.301685+0000","last_fullsized":"2023-01-15T14:33:57.301685+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:59.872","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.39300976100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1228,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.904458+0000","last_change":"2023-01-15T14:33:57.354727+0000","last_active":"2023-01-15T14:33:57.904458+0000","last_peered":"2023-01-15T14:33:57.904458+0000","last_clean":"2023-01-15T14:33:57.904458+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:33:57.904458+0000","last_undegraded":"2023-01-15T14:33:57.904458+0000","last_fullsized":"2023-01-15T14:33:57.904458+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.442392491,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.32","version":"334'163","reported_seq":828,"reported_epoch":348,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:57.002498+0000","last_change":"2023-01-15T14:33:53.329373+0000","last_active":"2023-01-15T14:33:57.002498+0000","last_peered":"2023-01-15T14:33:57.002498+0000","last_clean":"2023-01-15T14:33:50.154755+0000","last_became_active":"2023-01-15T14:33:53.228307+0000","last_became_peered":"2023-01-15T14:33:53.228307+0000","last_unstale":"2023-01-15T14:33:57.002498+0000","last_undegraded":"2023-01-15T14:33:52.928067+0000","last_fullsized":"2023-01-15T14:33:52.927836+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.80293351999999996,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7048,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[1,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.13","version":"348'374","reported_seq":1373,"reported_epoch":348,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:57.414084+0000","last_change":"2023-01-15T14:33:57.414084+0000","last_active":"2023-01-15T14:33:57.414084+0000","last_peered":"2023-01-15T14:33:57.414084+0000","last_clean":"2023-01-15T14:33:57.414084+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:33:57.414084+0000","last_undegraded":"2023-01-15T14:33:57.414084+0000","last_fullsized":"2023-01-15T14:33:57.414084+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:29.359","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.20500138100000001,"stat_sum":{"num_bytes":8364486,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":42770,"num_write":17,"num_write_kb":9266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.12","version":"335'184","reported_seq":828,"reported_epoch":343,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:51.922895+0000","last_change":"2023-01-15T14:33:51.922895+0000","last_active":"2023-01-15T14:33:51.922608+0000","last_peered":"2023-01-15T14:33:44.161296+0000","last_clean":"2023-01-15T14:33:44.161296+0000","last_became_active":"2023-01-15T14:33:20.335242+0000","last_became_peered":"2023-01-15T14:33:20.335242+0000","last_unstale":"2023-01-15T14:33:51.922895+0000","last_undegraded":"2023-01-15T14:33:51.922895+0000","last_fullsized":"2023-01-15T14:33:51.922895+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.24331072000000001,"stat_sum":{"num_bytes":7581497,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":15,"num_write_kb":5299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"e4","length":"1"},{"start":"ef","length":"2"},{"start":"f9","length":"2"},{"start":"100","length":"2"},{"start":"106","length":"1"}]},{"pgid":"3.11","version":"334'306","reported_seq":1484,"reported_epoch":347,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.287045+0000","last_change":"2023-01-15T14:33:53.763768+0000","last_active":"2023-01-15T14:33:56.287045+0000","last_peered":"2023-01-15T14:33:56.287045+0000","last_clean":"2023-01-15T14:33:50.016827+0000","last_became_active":"2023-01-15T14:33:53.659621+0000","last_became_peered":"2023-01-15T14:33:53.659621+0000","last_unstale":"2023-01-15T14:33:56.287045+0000","last_undegraded":"2023-01-15T14:33:53.019571+0000","last_fullsized":"2023-01-15T14:33:53.017412+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.090247347000000006,"stat_sum":{"num_bytes":10718732,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":48289,"num_write":26,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6655737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":6},{"shards":"0,7","objects":-2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.10","version":"308'406","reported_seq":1399,"reported_epoch":347,"state":"active+recovery_wait+remapped","last_fresh":"2023-01-15T14:33:56.284917+0000","last_change":"2023-01-15T14:33:53.820604+0000","last_active":"2023-01-15T14:33:56.284917+0000","last_peered":"2023-01-15T14:33:56.284917+0000","last_clean":"2023-01-15T14:33:17.174184+0000","last_became_active":"2023-01-15T14:33:53.718934+0000","last_became_peered":"2023-01-15T14:33:53.718934+0000","last_unstale":"2023-01-15T14:33:56.284917+0000","last_undegraded":"2023-01-15T14:33:56.284917+0000","last_fullsized":"2023-01-15T14:33:56.284917+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.84251846600000002,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":41964,"num_write":46,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6670228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[1,3,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1213,"reported_epoch":344,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:33:53.140157+0000","last_change":"2023-01-15T14:33:53.139890+0000","last_active":"2023-01-15T14:33:53.140157+0000","last_peered":"2023-01-15T14:33:53.140157+0000","last_clean":"2023-01-15T14:33:50.081207+0000","last_became_active":"2023-01-15T14:33:53.139890+0000","last_became_peered":"2023-01-15T14:33:53.139890+0000","last_unstale":"2023-01-15T14:33:53.140157+0000","last_undegraded":"2023-01-15T14:33:53.140157+0000","last_fullsized":"2023-01-15T14:33:52.914111+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:41.299","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":1.1105973060000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,1],"avail_no_missing":["4","0","1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":12,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.412233+0000","last_change":"2023-01-15T14:33:57.411714+0000","last_active":"2023-01-15T14:33:57.412233+0000","last_peered":"2023-01-15T14:33:57.412233+0000","last_clean":"2023-01-15T14:33:57.412233+0000","last_became_active":"2023-01-15T14:33:57.310759+0000","last_became_peered":"2023-01-15T14:33:57.310759+0000","last_unstale":"2023-01-15T14:33:57.412233+0000","last_undegraded":"2023-01-15T14:33:57.412233+0000","last_fullsized":"2023-01-15T14:33:57.412233+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.100760999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.d","version":"296'232","reported_seq":1293,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.113803+0000","last_change":"2023-01-15T14:33:57.114569+0000","last_active":"2023-01-15T14:33:57.113803+0000","last_peered":"2023-01-15T14:33:57.113803+0000","last_clean":"2023-01-15T14:33:57.113803+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:33:57.113803+0000","last_undegraded":"2023-01-15T14:33:57.113803+0000","last_fullsized":"2023-01-15T14:33:57.113803+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.20767458899999999,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.c","version":"348'95","reported_seq":1088,"reported_epoch":348,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:57.826220+0000","last_change":"2023-01-15T14:33:57.622687+0000","last_active":"2023-01-15T14:33:57.826220+0000","last_peered":"2023-01-15T14:33:57.826220+0000","last_clean":"2023-01-15T14:33:57.826220+0000","last_became_active":"2023-01-15T14:33:52.210127+0000","last_became_peered":"2023-01-15T14:33:52.210127+0000","last_unstale":"2023-01-15T14:33:57.826220+0000","last_undegraded":"2023-01-15T14:33:57.826220+0000","last_fullsized":"2023-01-15T14:33:57.826220+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.38959517900000001,"stat_sum":{"num_bytes":2221858,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":10493,"num_write":6,"num_write_kb":686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8023674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":999,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.618325+0000","last_change":"2023-01-15T14:33:57.618325+0000","last_active":"2023-01-15T14:33:57.618325+0000","last_peered":"2023-01-15T14:33:57.618325+0000","last_clean":"2023-01-15T14:33:57.618325+0000","last_became_active":"2023-01-15T14:33:52.425622+0000","last_became_peered":"2023-01-15T14:33:52.425622+0000","last_unstale":"2023-01-15T14:33:57.618325+0000","last_undegraded":"2023-01-15T14:33:57.618325+0000","last_fullsized":"2023-01-15T14:33:57.618325+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.60540339200000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1322,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.297022+0000","last_change":"2023-01-15T14:33:57.297022+0000","last_active":"2023-01-15T14:33:57.297022+0000","last_peered":"2023-01-15T14:33:57.297022+0000","last_clean":"2023-01-15T14:33:57.297022+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:33:57.297022+0000","last_undegraded":"2023-01-15T14:33:57.297022+0000","last_fullsized":"2023-01-15T14:33:57.297022+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:48.664","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.38900274200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1107,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.304492+0000","last_change":"2023-01-15T14:33:57.304492+0000","last_active":"2023-01-15T14:33:57.304492+0000","last_peered":"2023-01-15T14:33:57.304492+0000","last_clean":"2023-01-15T14:33:57.304492+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:33:57.304492+0000","last_undegraded":"2023-01-15T14:33:57.304492+0000","last_fullsized":"2023-01-15T14:33:57.304492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:38.874","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.29516895799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1304,"reported_epoch":349,"state":"activating+undersized+remapped","last_fresh":"2023-01-15T14:33:57.903125+0000","last_change":"2023-01-15T14:33:57.903380+0000","last_active":"2023-01-15T14:33:56.491492+0000","last_peered":"2023-01-15T14:33:55.173516+0000","last_clean":"2023-01-15T14:33:55.173516+0000","last_became_active":"2023-01-15T14:32:22.142296+0000","last_became_peered":"2023-01-15T14:32:22.142296+0000","last_unstale":"2023-01-15T14:33:57.903125+0000","last_undegraded":"2023-01-15T14:33:57.903125+0000","last_fullsized":"2023-01-15T14:33:57.903125+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.20813032000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1],"avail_no_missing":["3","1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":506,"reported_epoch":348,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:57.199812+0000","last_change":"2023-01-15T14:33:57.199812+0000","last_active":"2023-01-15T14:33:56.182917+0000","last_peered":"2023-01-15T14:33:55.072194+0000","last_clean":"2023-01-15T14:33:50.143204+0000","last_became_active":"2023-01-15T14:33:53.718233+0000","last_became_peered":"2023-01-15T14:33:53.718233+0000","last_unstale":"2023-01-15T14:33:57.199812+0000","last_undegraded":"2023-01-15T14:33:57.199812+0000","last_fullsized":"2023-01-15T14:33:57.199812+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.015161+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:33:57.015161+0000","last_peered":"2023-01-15T14:33:57.015161+0000","last_clean":"2023-01-15T14:33:57.015161+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:33:57.015161+0000","last_undegraded":"2023-01-15T14:33:57.015161+0000","last_fullsized":"2023-01-15T14:33:57.015161+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":995,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.297870+0000","last_change":"2023-01-15T14:33:57.297870+0000","last_active":"2023-01-15T14:33:57.297870+0000","last_peered":"2023-01-15T14:33:57.297870+0000","last_clean":"2023-01-15T14:33:57.297870+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:33:57.297870+0000","last_undegraded":"2023-01-15T14:33:57.297870+0000","last_fullsized":"2023-01-15T14:33:57.297870+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.38976201700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":393,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.085787+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:33:56.085787+0000","last_peered":"2023-01-15T14:33:56.085787+0000","last_clean":"2023-01-15T14:33:56.085787+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:33:56.085787+0000","last_undegraded":"2023-01-15T14:33:56.085787+0000","last_fullsized":"2023-01-15T14:33:56.085787+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.964","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":702,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.395060+0000","last_change":"2023-01-15T14:33:56.395495+0000","last_active":"2023-01-15T14:33:56.395060+0000","last_peered":"2023-01-15T14:33:56.395060+0000","last_clean":"2023-01-15T14:33:56.395060+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:33:56.395060+0000","last_undegraded":"2023-01-15T14:33:56.395060+0000","last_fullsized":"2023-01-15T14:33:56.395060+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.31666380900000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":150,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.010096+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:33:57.010096+0000","last_peered":"2023-01-15T14:33:57.010096+0000","last_clean":"2023-01-15T14:33:57.010096+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:33:57.010096+0000","last_undegraded":"2023-01-15T14:33:57.010096+0000","last_fullsized":"2023-01-15T14:33:57.010096+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1314,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.301164+0000","last_change":"2023-01-15T14:33:57.301164+0000","last_active":"2023-01-15T14:33:57.301164+0000","last_peered":"2023-01-15T14:33:57.301164+0000","last_clean":"2023-01-15T14:33:57.301164+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:33:57.301164+0000","last_undegraded":"2023-01-15T14:33:57.301164+0000","last_fullsized":"2023-01-15T14:33:57.301164+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.093344012000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":216,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.488732+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:33:56.488732+0000","last_peered":"2023-01-15T14:33:56.488732+0000","last_clean":"2023-01-15T14:33:56.488732+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:33:56.488732+0000","last_undegraded":"2023-01-15T14:33:56.488732+0000","last_fullsized":"2023-01-15T14:33:56.488732+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:25.583","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"344'317","reported_seq":1390,"reported_epoch":347,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.545414+0000","last_change":"2023-01-15T14:33:56.033295+0000","last_active":"2023-01-15T14:33:56.545414+0000","last_peered":"2023-01-15T14:33:56.545414+0000","last_clean":"2023-01-15T14:33:50.176701+0000","last_became_active":"2023-01-15T14:33:53.226305+0000","last_became_peered":"2023-01-15T14:33:53.226305+0000","last_unstale":"2023-01-15T14:33:56.545414+0000","last_undegraded":"2023-01-15T14:33:52.915843+0000","last_fullsized":"2023-01-15T14:33:52.914697+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:34.328","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.69416308599999998,"stat_sum":{"num_bytes":9428925,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":13874,"num_write":2,"num_write_kb":6435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[0,1],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,1","objects":2},{"shards":"0,1,3","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.5","version":"335'158","reported_seq":1165,"reported_epoch":349,"state":"activating+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:57.908195+0000","last_change":"2023-01-15T14:33:58.109966+0000","last_active":"2023-01-15T14:33:56.088835+0000","last_peered":"2023-01-15T14:33:55.073530+0000","last_clean":"2023-01-15T14:33:55.073530+0000","last_became_active":"2023-01-15T14:32:21.948278+0000","last_became_peered":"2023-01-15T14:32:21.948278+0000","last_unstale":"2023-01-15T14:33:57.908195+0000","last_undegraded":"2023-01-15T14:33:57.908195+0000","last_fullsized":"2023-01-15T14:33:57.905675+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.71299288800000005,"stat_sum":{"num_bytes":11390101,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":26145,"num_write":9,"num_write_kb":4265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":206,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.087555+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:33:56.087555+0000","last_peered":"2023-01-15T14:33:56.087555+0000","last_clean":"2023-01-15T14:33:56.087555+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:33:56.087555+0000","last_undegraded":"2023-01-15T14:33:56.087555+0000","last_fullsized":"2023-01-15T14:33:56.087555+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"335'308","reported_seq":1403,"reported_epoch":347,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.191211+0000","last_change":"2023-01-15T14:33:53.424640+0000","last_active":"2023-01-15T14:33:56.191211+0000","last_peered":"2023-01-15T14:33:56.191211+0000","last_clean":"2023-01-15T14:33:50.017033+0000","last_became_active":"2023-01-15T14:33:53.422817+0000","last_became_peered":"2023-01-15T14:33:53.422817+0000","last_unstale":"2023-01-15T14:33:56.191211+0000","last_undegraded":"2023-01-15T14:33:53.016002+0000","last_fullsized":"2023-01-15T14:33:53.014012+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.70443229200000002,"stat_sum":{"num_bytes":2954643,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":30684,"num_write":31,"num_write_kb":10131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7931053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2","objects":-1},{"shards":"1,2,4","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":395,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.287148+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:33:56.287148+0000","last_peered":"2023-01-15T14:33:56.287148+0000","last_clean":"2023-01-15T14:33:56.287148+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:33:56.287148+0000","last_undegraded":"2023-01-15T14:33:56.287148+0000","last_fullsized":"2023-01-15T14:33:56.287148+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"333'137","reported_seq":1119,"reported_epoch":348,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:56.917815+0000","last_change":"2023-01-15T14:33:56.917815+0000","last_active":"2023-01-15T14:33:56.080946+0000","last_peered":"2023-01-15T14:33:55.068936+0000","last_clean":"2023-01-15T14:33:42.900201+0000","last_became_active":"2023-01-15T14:33:53.137740+0000","last_became_peered":"2023-01-15T14:33:53.137740+0000","last_unstale":"2023-01-15T14:33:56.917815+0000","last_undegraded":"2023-01-15T14:33:56.917815+0000","last_fullsized":"2023-01-15T14:33:56.917815+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.471195633,"stat_sum":{"num_bytes":8547430,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":13597,"num_write":9,"num_write_kb":7710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":206,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.381327+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:33:56.381327+0000","last_peered":"2023-01-15T14:33:56.381327+0000","last_clean":"2023-01-15T14:33:56.381327+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:33:56.381327+0000","last_undegraded":"2023-01-15T14:33:56.381327+0000","last_fullsized":"2023-01-15T14:33:56.381327+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:17.360","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":27,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.386509+0000","last_change":"2023-01-15T14:33:56.387426+0000","last_active":"2023-01-15T14:33:56.386509+0000","last_peered":"2023-01-15T14:33:56.386509+0000","last_clean":"2023-01-15T14:33:56.386509+0000","last_became_active":"2023-01-15T14:33:52.206701+0000","last_became_peered":"2023-01-15T14:33:52.206701+0000","last_unstale":"2023-01-15T14:33:56.386509+0000","last_undegraded":"2023-01-15T14:33:56.386509+0000","last_fullsized":"2023-01-15T14:33:56.386509+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.30826422799999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.413064+0000","last_change":"2023-01-15T14:33:57.412557+0000","last_active":"2023-01-15T14:33:57.413064+0000","last_peered":"2023-01-15T14:33:57.413064+0000","last_clean":"2023-01-15T14:33:57.413064+0000","last_became_active":"2023-01-15T14:33:57.311981+0000","last_became_peered":"2023-01-15T14:33:57.311981+0000","last_unstale":"2023-01-15T14:33:57.413064+0000","last_undegraded":"2023-01-15T14:33:57.413064+0000","last_fullsized":"2023-01-15T14:33:57.413064+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"348'158","reported_seq":1178,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.009873+0000","last_change":"2023-01-15T14:33:58.010967+0000","last_active":"2023-01-15T14:33:58.009873+0000","last_peered":"2023-01-15T14:33:58.009873+0000","last_clean":"2023-01-15T14:33:58.009873+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:33:58.009873+0000","last_undegraded":"2023-01-15T14:33:58.009873+0000","last_fullsized":"2023-01-15T14:33:58.009873+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:32:19.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:19.303243+0000","objects_scrubbed":1,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.0010524499999999999,"stat_sum":{"num_bytes":1226927,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":18,"num_read_kb":20384,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":12,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.579508+0000","last_change":"2023-01-15T14:33:57.579060+0000","last_active":"2023-01-15T14:33:57.579508+0000","last_peered":"2023-01-15T14:33:57.579508+0000","last_clean":"2023-01-15T14:33:57.579508+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:33:57.579508+0000","last_undegraded":"2023-01-15T14:33:57.579508+0000","last_fullsized":"2023-01-15T14:33:57.579508+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.16","version":"320'127","reported_seq":625,"reported_epoch":348,"state":"remapped+peering","last_fresh":"2023-01-15T14:33:57.013072+0000","last_change":"2023-01-15T14:33:57.013400+0000","last_active":"2023-01-15T14:33:57.013035+0000","last_peered":"2023-01-15T14:33:55.270604+0000","last_clean":"2023-01-15T14:33:55.270604+0000","last_became_active":"2023-01-15T14:33:35.159018+0000","last_became_peered":"2023-01-15T14:33:35.159018+0000","last_unstale":"2023-01-15T14:33:57.013072+0000","last_undegraded":"2023-01-15T14:33:57.013072+0000","last_fullsized":"2023-01-15T14:33:57.013072+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:54.467","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.30046122199999997,"stat_sum":{"num_bytes":7378471,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,7],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":27,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.396704+0000","last_change":"2023-01-15T14:33:56.396996+0000","last_active":"2023-01-15T14:33:56.396704+0000","last_peered":"2023-01-15T14:33:56.396704+0000","last_clean":"2023-01-15T14:33:56.396704+0000","last_became_active":"2023-01-15T14:33:52.303197+0000","last_became_peered":"2023-01-15T14:33:52.303197+0000","last_unstale":"2023-01-15T14:33:56.396704+0000","last_undegraded":"2023-01-15T14:33:56.396704+0000","last_fullsized":"2023-01-15T14:33:56.396704+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.21267884200000001,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.18","version":"340'324","reported_seq":1479,"reported_epoch":348,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:57.007858+0000","last_change":"2023-01-15T14:33:53.518579+0000","last_active":"2023-01-15T14:33:57.007858+0000","last_peered":"2023-01-15T14:33:57.007858+0000","last_clean":"2023-01-15T14:33:50.112982+0000","last_became_active":"2023-01-15T14:33:53.517870+0000","last_became_peered":"2023-01-15T14:33:53.517870+0000","last_unstale":"2023-01-15T14:33:57.007858+0000","last_undegraded":"2023-01-15T14:33:52.920700+0000","last_fullsized":"2023-01-15T14:33:52.920090+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:06.616","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.66838752300000004,"stat_sum":{"num_bytes":4332078,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":26,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":4},{"shards":"1,3","objects":-2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.19","version":"294'256","reported_seq":1316,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.626083+0000","last_change":"2023-01-15T14:33:57.626542+0000","last_active":"2023-01-15T14:33:57.626083+0000","last_peered":"2023-01-15T14:33:57.626083+0000","last_clean":"2023-01-15T14:33:57.626083+0000","last_became_active":"2023-01-15T14:33:35.259904+0000","last_became_peered":"2023-01-15T14:33:35.259904+0000","last_unstale":"2023-01-15T14:33:57.626083+0000","last_undegraded":"2023-01-15T14:33:57.626083+0000","last_fullsized":"2023-01-15T14:33:57.626083+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:01.296","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.52734117999999996,"stat_sum":{"num_bytes":2420274,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1a","version":"348'352","reported_seq":1450,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.010827+0000","last_change":"2023-01-15T14:33:58.011960+0000","last_active":"2023-01-15T14:33:58.010827+0000","last_peered":"2023-01-15T14:33:58.010827+0000","last_clean":"2023-01-15T14:33:58.010827+0000","last_became_active":"2023-01-15T14:33:35.063699+0000","last_became_peered":"2023-01-15T14:33:35.063699+0000","last_unstale":"2023-01-15T14:33:58.010827+0000","last_undegraded":"2023-01-15T14:33:58.010827+0000","last_fullsized":"2023-01-15T14:33:58.010827+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.0039007270000000001,"stat_sum":{"num_bytes":3052653,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":15,"num_read_kb":21426,"num_write":7,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1b","version":"333'97","reported_seq":49,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.447812+0000","last_change":"2023-01-15T14:33:56.448600+0000","last_active":"2023-01-15T14:33:56.447812+0000","last_peered":"2023-01-15T14:33:56.447812+0000","last_clean":"2023-01-15T14:33:56.447812+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:33:56.447812+0000","last_undegraded":"2023-01-15T14:33:56.447812+0000","last_fullsized":"2023-01-15T14:33:56.447812+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.36685731199999999,"stat_sum":{"num_bytes":4400087,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10511,"num_write":14,"num_write_kb":4701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9416119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":538,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.011212+0000","last_change":"2023-01-15T14:33:58.012512+0000","last_active":"2023-01-15T14:33:58.011212+0000","last_peered":"2023-01-15T14:33:58.011212+0000","last_clean":"2023-01-15T14:33:58.011212+0000","last_became_active":"2023-01-15T14:33:57.125876+0000","last_became_peered":"2023-01-15T14:33:57.125876+0000","last_unstale":"2023-01-15T14:33:58.011212+0000","last_undegraded":"2023-01-15T14:33:58.011212+0000","last_fullsized":"2023-01-15T14:33:58.011212+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.0028318649999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1263,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.405395+0000","last_change":"2023-01-15T14:33:57.405395+0000","last_active":"2023-01-15T14:33:57.405395+0000","last_peered":"2023-01-15T14:33:57.405395+0000","last_clean":"2023-01-15T14:33:57.405395+0000","last_became_active":"2023-01-15T14:33:35.362912+0000","last_became_peered":"2023-01-15T14:33:35.362912+0000","last_unstale":"2023-01-15T14:33:57.405395+0000","last_undegraded":"2023-01-15T14:33:57.405395+0000","last_fullsized":"2023-01-15T14:33:57.405395+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.394560678,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":974,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.317895+0000","last_change":"2023-01-15T14:33:57.317895+0000","last_active":"2023-01-15T14:33:57.317895+0000","last_peered":"2023-01-15T14:33:57.317895+0000","last_clean":"2023-01-15T14:33:57.317895+0000","last_became_active":"2023-01-15T14:33:35.261561+0000","last_became_peered":"2023-01-15T14:33:35.261561+0000","last_unstale":"2023-01-15T14:33:57.317895+0000","last_undegraded":"2023-01-15T14:33:57.317895+0000","last_fullsized":"2023-01-15T14:33:57.317895+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40752737100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1f","version":"318'256","reported_seq":1306,"reported_epoch":347,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:33:56.715595+0000","last_change":"2023-01-15T14:33:56.748752+0000","last_active":"2023-01-15T14:33:56.715595+0000","last_peered":"2023-01-15T14:33:56.715595+0000","last_clean":"2023-01-15T14:33:27.785873+0000","last_became_active":"2023-01-15T14:33:53.125737+0000","last_became_peered":"2023-01-15T14:33:53.125737+0000","last_unstale":"2023-01-15T14:33:56.715595+0000","last_undegraded":"2023-01-15T14:33:53.115874+0000","last_fullsized":"2023-01-15T14:33:53.114236+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":1,"snaptrim_duration":0.30935489599999999,"stat_sum":{"num_bytes":2974888,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":13341,"num_write":1,"num_write_kb":1699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[2,4],"avail_no_missing":["2","3","4"],"object_location_counts":[{"shards":"2,3,4","objects":3},{"shards":"2,4","objects":-2}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.20","version":"324'273","reported_seq":1345,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.283462+0000","last_change":"2023-01-15T14:33:56.284414+0000","last_active":"2023-01-15T14:33:56.283462+0000","last_peered":"2023-01-15T14:33:56.283462+0000","last_clean":"2023-01-15T14:33:56.283462+0000","last_became_active":"2023-01-15T14:33:55.472883+0000","last_became_peered":"2023-01-15T14:33:55.472883+0000","last_unstale":"2023-01-15T14:33:56.283462+0000","last_undegraded":"2023-01-15T14:33:56.283462+0000","last_fullsized":"2023-01-15T14:33:56.283462+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.20799870300000001,"stat_sum":{"num_bytes":4251137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10864,"num_write":9,"num_write_kb":4140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4438968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.21","version":"321'138","reported_seq":1038,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.004213+0000","last_change":"2023-01-15T14:33:57.005196+0000","last_active":"2023-01-15T14:33:57.004213+0000","last_peered":"2023-01-15T14:33:57.004213+0000","last_clean":"2023-01-15T14:33:57.004213+0000","last_became_active":"2023-01-15T14:33:45.267321+0000","last_became_peered":"2023-01-15T14:33:45.267321+0000","last_unstale":"2023-01-15T14:33:57.004213+0000","last_undegraded":"2023-01-15T14:33:57.004213+0000","last_fullsized":"2023-01-15T14:33:57.004213+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.102357873,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.22","version":"323'161","reported_seq":1329,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.619172+0000","last_change":"2023-01-15T14:33:57.619172+0000","last_active":"2023-01-15T14:33:57.619172+0000","last_peered":"2023-01-15T14:33:57.619172+0000","last_clean":"2023-01-15T14:33:57.619172+0000","last_became_active":"2023-01-15T14:33:55.378253+0000","last_became_peered":"2023-01-15T14:33:55.378253+0000","last_unstale":"2023-01-15T14:33:57.619172+0000","last_undegraded":"2023-01-15T14:33:57.619172+0000","last_fullsized":"2023-01-15T14:33:57.619172+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.50935289699999997,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":729,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.341803+0000","last_change":"2023-01-15T14:33:56.342569+0000","last_active":"2023-01-15T14:33:56.341803+0000","last_peered":"2023-01-15T14:33:56.341803+0000","last_clean":"2023-01-15T14:33:56.341803+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:33:56.341803+0000","last_undegraded":"2023-01-15T14:33:56.341803+0000","last_fullsized":"2023-01-15T14:33:56.341803+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.26366071200000002,"stat_sum":{"num_bytes":3802136,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":21818,"num_write":18,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.24","version":"348'259","reported_seq":1392,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.616353+0000","last_change":"2023-01-15T14:33:57.616353+0000","last_active":"2023-01-15T14:33:57.616353+0000","last_peered":"2023-01-15T14:33:57.616353+0000","last_clean":"2023-01-15T14:33:57.616353+0000","last_became_active":"2023-01-15T14:33:45.075309+0000","last_became_peered":"2023-01-15T14:33:45.075309+0000","last_unstale":"2023-01-15T14:33:57.616353+0000","last_undegraded":"2023-01-15T14:33:57.616353+0000","last_fullsized":"2023-01-15T14:33:57.616353+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.70814871000000001,"stat_sum":{"num_bytes":4593906,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10774,"num_write":8,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.25","version":"333'108","reported_seq":1110,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.330557+0000","last_change":"2023-01-15T14:33:57.331159+0000","last_active":"2023-01-15T14:33:57.330557+0000","last_peered":"2023-01-15T14:33:57.330557+0000","last_clean":"2023-01-15T14:33:57.330557+0000","last_became_active":"2023-01-15T14:33:45.168598+0000","last_became_peered":"2023-01-15T14:33:45.168598+0000","last_unstale":"2023-01-15T14:33:57.330557+0000","last_undegraded":"2023-01-15T14:33:57.330557+0000","last_fullsized":"2023-01-15T14:33:57.330557+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.33264791100000002,"stat_sum":{"num_bytes":2638765,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3030,"num_write":2,"num_write_kb":2577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":459,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.008359+0000","last_change":"2023-01-15T14:33:57.008359+0000","last_active":"2023-01-15T14:33:57.008359+0000","last_peered":"2023-01-15T14:33:57.008359+0000","last_clean":"2023-01-15T14:33:57.008359+0000","last_became_active":"2023-01-15T14:33:52.309110+0000","last_became_peered":"2023-01-15T14:33:52.309110+0000","last_unstale":"2023-01-15T14:33:57.008359+0000","last_undegraded":"2023-01-15T14:33:57.008359+0000","last_fullsized":"2023-01-15T14:33:57.008359+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.208200415,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":1061,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.355572+0000","last_change":"2023-01-15T14:33:57.355572+0000","last_active":"2023-01-15T14:33:57.355572+0000","last_peered":"2023-01-15T14:33:57.355572+0000","last_clean":"2023-01-15T14:33:57.355572+0000","last_became_active":"2023-01-15T14:33:45.171090+0000","last_became_peered":"2023-01-15T14:33:45.171090+0000","last_unstale":"2023-01-15T14:33:57.355572+0000","last_undegraded":"2023-01-15T14:33:57.355572+0000","last_fullsized":"2023-01-15T14:33:57.355572+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.443073948,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.28","version":"333'248","reported_seq":1379,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.617361+0000","last_change":"2023-01-15T14:33:57.617361+0000","last_active":"2023-01-15T14:33:57.617361+0000","last_peered":"2023-01-15T14:33:57.617361+0000","last_clean":"2023-01-15T14:33:57.617361+0000","last_became_active":"2023-01-15T14:33:45.366831+0000","last_became_peered":"2023-01-15T14:33:45.366831+0000","last_unstale":"2023-01-15T14:33:57.617361+0000","last_undegraded":"2023-01-15T14:33:57.617361+0000","last_fullsized":"2023-01-15T14:33:57.617361+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:01.610","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.60974191300000002,"stat_sum":{"num_bytes":4838389,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8969,"num_write":8,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":6096764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":883,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.620313+0000","last_change":"2023-01-15T14:33:57.620313+0000","last_active":"2023-01-15T14:33:57.620313+0000","last_peered":"2023-01-15T14:33:57.620313+0000","last_clean":"2023-01-15T14:33:57.620313+0000","last_became_active":"2023-01-15T14:33:45.276960+0000","last_became_peered":"2023-01-15T14:33:45.276960+0000","last_unstale":"2023-01-15T14:33:57.620313+0000","last_undegraded":"2023-01-15T14:33:57.620313+0000","last_fullsized":"2023-01-15T14:33:57.620313+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.51033217900000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1228,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.903452+0000","last_change":"2023-01-15T14:33:56.904528+0000","last_active":"2023-01-15T14:33:56.903452+0000","last_peered":"2023-01-15T14:33:56.903452+0000","last_clean":"2023-01-15T14:33:56.903452+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:33:56.903452+0000","last_undegraded":"2023-01-15T14:33:56.903452+0000","last_fullsized":"2023-01-15T14:33:56.903452+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.0022419219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":25,"reported_epoch":347,"state":"peering","last_fresh":"2023-01-15T14:33:56.080625+0000","last_change":"2023-01-15T14:33:56.080995+0000","last_active":"2023-01-15T14:33:56.080569+0000","last_peered":"2023-01-15T14:33:55.176119+0000","last_clean":"2023-01-15T14:33:55.176119+0000","last_became_active":"2023-01-15T14:33:52.507094+0000","last_became_peered":"2023-01-15T14:33:52.507094+0000","last_unstale":"2023-01-15T14:33:56.080625+0000","last_undegraded":"2023-01-15T14:33:56.080625+0000","last_fullsized":"2023-01-15T14:33:56.080625+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.101837726,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,7],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":432,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.303687+0000","last_change":"2023-01-15T14:33:57.303687+0000","last_active":"2023-01-15T14:33:57.303687+0000","last_peered":"2023-01-15T14:33:57.303687+0000","last_clean":"2023-01-15T14:33:57.303687+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:33:57.303687+0000","last_undegraded":"2023-01-15T14:33:57.303687+0000","last_fullsized":"2023-01-15T14:33:57.303687+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:10.970","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.29451823399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1198,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.007402+0000","last_change":"2023-01-15T14:33:57.007402+0000","last_active":"2023-01-15T14:33:57.007402+0000","last_peered":"2023-01-15T14:33:57.007402+0000","last_clean":"2023-01-15T14:33:57.007402+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:33:57.007402+0000","last_undegraded":"2023-01-15T14:33:57.007402+0000","last_fullsized":"2023-01-15T14:33:57.007402+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.203775807,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":12,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.211505+0000","last_change":"2023-01-15T14:33:57.211055+0000","last_active":"2023-01-15T14:33:57.211505+0000","last_peered":"2023-01-15T14:33:57.211505+0000","last_clean":"2023-01-15T14:33:57.211505+0000","last_became_active":"2023-01-15T14:33:57.210434+0000","last_became_peered":"2023-01-15T14:33:57.210434+0000","last_unstale":"2023-01-15T14:33:57.211505+0000","last_undegraded":"2023-01-15T14:33:57.211505+0000","last_fullsized":"2023-01-15T14:33:57.211505+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.38796315799999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1091,"reported_epoch":348,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:33:57.009469+0000","last_change":"2023-01-15T14:33:57.009469+0000","last_active":"2023-01-15T14:33:57.009469+0000","last_peered":"2023-01-15T14:33:57.009469+0000","last_clean":"2023-01-15T14:33:57.009469+0000","last_became_active":"2023-01-15T14:33:55.174128+0000","last_became_peered":"2023-01-15T14:33:55.174128+0000","last_unstale":"2023-01-15T14:33:57.009469+0000","last_undegraded":"2023-01-15T14:33:57.009469+0000","last_fullsized":"2023-01-15T14:33:57.009469+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.16584238700000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":967,"reported_epoch":347,"state":"active+clean","last_fresh":"2023-01-15T14:33:56.284720+0000","last_change":"2023-01-15T14:33:56.386256+0000","last_active":"2023-01-15T14:33:56.284720+0000","last_peered":"2023-01-15T14:33:56.284720+0000","last_clean":"2023-01-15T14:33:56.284720+0000","last_became_active":"2023-01-15T14:33:55.373498+0000","last_became_peered":"2023-01-15T14:33:55.373498+0000","last_unstale":"2023-01-15T14:33:56.284720+0000","last_undegraded":"2023-01-15T14:33:56.284720+0000","last_fullsized":"2023-01-15T14:33:56.284720+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":7,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.30982162000000002,"stat_sum":{"num_bytes":5419866,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":25948,"num_write":35,"num_write_kb":13827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":13340458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.31","version":"204'150","reported_seq":1214,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.625989+0000","last_change":"2023-01-15T14:33:57.625989+0000","last_active":"2023-01-15T14:33:57.625989+0000","last_peered":"2023-01-15T14:33:57.625989+0000","last_clean":"2023-01-15T14:33:57.625989+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:33:57.625989+0000","last_undegraded":"2023-01-15T14:33:57.625989+0000","last_fullsized":"2023-01-15T14:33:57.625989+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.52721113600000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":106,"stat_sum":{"num_bytes":338336060,"num_objects":152,"num_object_clones":101,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":162,"num_objects_misplaced":34,"num_objects_unfound":0,"num_objects_dirty":152,"num_whiteouts":12,"num_read":973,"num_read_kb":1027885,"num_write":779,"num_write_kb":320512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":407,"num_bytes_recovered":239498883,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18632,"ondisk_log_size":18632,"up":318,"acting":299,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574920,"num_pgs":44,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":990632,"kb_used_data":92716184,"kb_used_omap":7065,"kb_used_meta":0,"kb_avail":92708748,"statfs":{"total":95948165120,"available":94933757952,"internally_reserved":7614464,"allocated":94941372416,"data_stored":94941372416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7235562,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":4.1269999999999998,"15min":4.0890000000000004},"min":{"1min":0.54000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2030000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":9.5380000000000003,"5min":6.3200000000000003,"15min":8.2460000000000004},"min":{"1min":0.58799999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":96.337000000000003,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.81799999999999995}]},{"osd":1,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":3.6509999999999998,"15min":5.5800000000000001},"min":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":34.585999999999999,"5min":88.003,"15min":88.003},"last":0.79400000000000004},{"interface":"front","average":{"1min":13.302,"5min":8.1950000000000003,"15min":6.9180000000000001},"min":{"1min":0.58799999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.403999999999996,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.98899999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":6.9400000000000004,"15min":5.5069999999999997},"min":{"1min":0.57199999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.179,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.91800000000000004,"5min":2.5790000000000002,"15min":3.645},"min":{"1min":0.626,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":91.837999999999994,"15min":91.837999999999994},"last":0.91500000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":7.3799999999999999,"15min":3.0920000000000001},"min":{"1min":0.63600000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2370000000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.91600000000000004,"5min":3.6360000000000001,"15min":4.3129999999999997},"min":{"1min":0.65000000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.2270000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.96399999999999997}]},{"osd":4,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5519999999999996,"5min":5.6219999999999999,"15min":8.1509999999999998},"min":{"1min":0.63500000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":96.185000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.9259999999999999,"5min":5.782,"15min":2.5699999999999998},"min":{"1min":0.73099999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":22.41,"5min":164.21199999999999,"15min":164.21199999999999},"last":142.44999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.6440000000000001,"5min":5.0659999999999998,"15min":5.9059999999999997},"min":{"1min":0.72799999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":96.263999999999996,"5min":96.263999999999996,"15min":96.263999999999996},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.998,"5min":3.206,"15min":4.3650000000000002},"min":{"1min":0.69299999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.355,"5min":95.430000000000007,"15min":95.430000000000007},"last":0.89600000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":5.5949999999999998,"15min":7.5469999999999997},"min":{"1min":0.83599999999999997,"5min":0.496,"15min":0.496},"max":{"1min":96.299999999999997,"5min":96.299999999999997,"15min":96.299999999999997},"last":1.026},{"interface":"front","average":{"1min":1.089,"5min":2.5150000000000001,"15min":1.571},"min":{"1min":0.84299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.2789999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":0.76000000000000001}]}]},{"osd":6,"up_from":0,"seq":55834574921,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574921,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574920,"num_pgs":69,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1070856,"kb_used_data":92651192,"kb_used_omap":13607,"kb_used_meta":0,"kb_avail":92628524,"statfs":{"total":95948165120,"available":94851608576,"internally_reserved":23212032,"allocated":94874820608,"data_stored":94874820608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13933805,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":23.594000000000001,"15min":11.141},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.133,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":3.0009999999999999,"15min":1.5209999999999999},"min":{"1min":0.58299999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.024,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.74399999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":5.0730000000000004,"15min":7.4199999999999999},"min":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6659999999999999,"5min":171.91499999999999,"15min":171.91499999999999},"last":23.193000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.343,"15min":1.9770000000000001},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0529999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":31.756}]},{"osd":2,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":24.227,"15min":11.749000000000001},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.157,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":23.247},{"interface":"front","average":{"1min":2.6040000000000001,"5min":1.21,"15min":1.01},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":32.081000000000003,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.71199999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":2.9169999999999998,"15min":1.865},"min":{"1min":0.65300000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.2909999999999999,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.7370000000000001,"5min":24.498999999999999,"15min":12.266999999999999},"min":{"1min":0.64400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":14.961,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":23.289000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.88500000000000001,"15min":0.90800000000000003},"min":{"1min":0.55200000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.92300000000000004,"5min":2.1840000000000002,"15min":2.2599999999999998},"min":{"1min":0.71799999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3200000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.81899999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":3.9329999999999998,"15min":5.093},"min":{"1min":0.69599999999999995,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.365,"5min":99.780000000000001,"15min":99.780000000000001},"last":23.329000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":22.463999999999999,"15min":10.93},"min":{"1min":0.71599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.427,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":0.92000000000000004}]},{"osd":7,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":3.1539999999999999,"5min":5.5570000000000004,"15min":10.603999999999999},"min":{"1min":0.76700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":39.012999999999998,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.2370000000000001,"5min":24.864000000000001,"15min":15.779999999999999},"min":{"1min":0.877,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.0680000000000001,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":23.356000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":69,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1084580,"kb_used_data":92614920,"kb_used_omap":13867,"kb_used_meta":0,"kb_avail":92614800,"statfs":{"total":95948165120,"available":94837555200,"internally_reserved":122880,"allocated":94837678080,"data_stored":94837678080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14200138,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":7.4400000000000004,"15min":5.7119999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":17.731000000000002,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":3.4340000000000002,"15min":6.2389999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.9260000000000002,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":7.1790000000000003,"15min":11.414},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96799999999999997,"5min":159.256,"15min":159.256},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.6259999999999999,"5min":10.417999999999999,"15min":3.9660000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":17.826000000000001,"5min":174.578,"15min":174.578},"last":25.515999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":4.6849999999999996,"5min":2.8809999999999998,"15min":3.1930000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":86.412000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.95,"5min":6.7889999999999997,"15min":3.7250000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.876000000000001,"5min":174.45099999999999,"15min":174.45099999999999},"last":15.946999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":7.6879999999999997,"5min":29.686,"15min":14.27},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":84.640000000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.86899999999999999,"5min":32.134999999999998,"15min":19.960999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.615,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":17.036000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":6.4980000000000002,"15min":11.433999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.04,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":2.2309999999999999,"5min":6.0039999999999996,"15min":2.6819999999999999},"min":{"1min":0.70099999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":17.917000000000002,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.71899999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1949999999999998,"5min":5.9980000000000002,"15min":2.6869999999999998},"min":{"1min":0.66000000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":17.989000000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.964,"15min":3.1619999999999999},"min":{"1min":0.70499999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.23,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.75600000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":2.0219999999999998,"15min":3.3639999999999999},"min":{"1min":0.69499999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.266,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.878},{"interface":"front","average":{"1min":19.138000000000002,"5min":11.705,"15min":15.961},"min":{"1min":0.66200000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":281.13400000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":0.98599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574920,"num_pgs":62,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1098488,"kb_used_data":92608588,"kb_used_omap":11369,"kb_used_meta":0,"kb_avail":92600892,"statfs":{"total":95948165120,"available":94823313408,"internally_reserved":7880704,"allocated":94831194112,"data_stored":94831194112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11642021,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":3.109,"5min":3.5230000000000001,"15min":1.601},"min":{"1min":0.437,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":51.009999999999998,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.4299999999999999,"15min":0.95299999999999996},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0089999999999999,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.45300000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.1529999999999996,"5min":5.9059999999999997,"15min":2.5009999999999999},"min":{"1min":0.378,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":91.525999999999996,"5min":143.071,"15min":143.071},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":2.016,"15min":2.9089999999999998},"min":{"1min":0.42299999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.131,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.53900000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":8.2210000000000001,"15min":9.6750000000000007},"min":{"1min":0.5,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.3609999999999998,"5min":183.53,"15min":183.53},"last":0.78100000000000003},{"interface":"front","average":{"1min":16.692,"5min":7.5170000000000003,"15min":3.0169999999999999},"min":{"1min":0.61199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":155.47900000000001,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.56399999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3650000000000002,"5min":6.5060000000000002,"15min":4.6020000000000003},"min":{"1min":0.67300000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":51.259999999999998,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.751},{"interface":"front","average":{"1min":2.8809999999999998,"5min":5.2939999999999996,"15min":4.3040000000000003},"min":{"1min":0.48699999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":33.219999999999999,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.76900000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":2.4729999999999999,"15min":1.6080000000000001},"min":{"1min":0.57199999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2130000000000001,"5min":143.262,"15min":143.262},"last":1.056},{"interface":"front","average":{"1min":3.4780000000000002,"5min":3.0539999999999998,"15min":1.792},"min":{"1min":0.70999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":51.326000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.85099999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.857,"15min":1.3069999999999999},"min":{"1min":0.60399999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.365,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.018,"5min":2.6160000000000001,"15min":1.6839999999999999},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.5,"5min":143.215,"15min":143.215},"last":1.1319999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":13.092000000000001,"5min":6.3120000000000003,"15min":4.1429999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":120.905,"5min":120.905,"15min":120.905},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.0129999999999999,"5min":2.8450000000000002,"15min":2.7989999999999999},"min":{"1min":0.66600000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.552,"5min":143.17099999999999,"15min":143.17099999999999},"last":1.1919999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574921,"num_pgs":72,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1043108,"kb_used_data":92656484,"kb_used_omap":12948,"kb_used_meta":0,"kb_avail":92656272,"statfs":{"total":95948165120,"available":94880022528,"internally_reserved":217088,"allocated":94880239616,"data_stored":94880239616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13258861,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":3.633,"15min":3.806},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":3.9140000000000001,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":7.742,"5min":4.6130000000000004,"15min":4.4189999999999996},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":119.39100000000001,"5min":119.39100000000001,"15min":119.39100000000001},"last":0.91900000000000004}]},{"osd":2,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":2.6469999999999998,"15min":1.5269999999999999},"min":{"1min":0.29799999999999999,"5min":0.245,"15min":0.245},"max":{"1min":95.879999999999995,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.8620000000000001,"15min":3.3359999999999999},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.9780000000000002,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.72999999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.8920000000000003,"5min":9.6750000000000007,"15min":16.449000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":100.137,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":7.782,"5min":6.8899999999999997,"15min":7.1260000000000003},"min":{"1min":0.45600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":119.804,"5min":119.804,"15min":119.804},"last":0.67400000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:33:17 2023","interfaces":[{"interface":"back","average":{"1min":7.9249999999999998,"5min":17.140999999999998,"15min":8.8650000000000002},"min":{"1min":0.504,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":119.71899999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.7849999999999999,"5min":15.526,"15min":7.5780000000000003},"min":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":16.989999999999998,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":1.2070000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.5449999999999999,"15min":1.2030000000000001},"min":{"1min":0.59899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.278,"5min":65.120999999999995,"15min":65.120999999999995},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.6579999999999999,"15min":3.5110000000000001},"min":{"1min":0.60599999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":4.0780000000000003,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.84999999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":8.0069999999999997,"5min":5.0129999999999999,"15min":5.0049999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":119.759,"5min":119.759,"15min":119.759},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.603,"15min":1.2829999999999999},"min":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.349,"5min":65.156000000000006,"15min":65.156000000000006},"last":1.397}]},{"osd":7,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":3.6030000000000002,"5min":6.1109999999999998,"15min":6.7149999999999999},"min":{"1min":0.56000000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":31.620999999999999,"5min":157.03700000000001,"15min":157.03700000000001},"last":9.2409999999999997},{"interface":"front","average":{"1min":8.0779999999999994,"5min":5.0060000000000002,"15min":4.835},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":119.65300000000001,"5min":119.65300000000001,"15min":119.65300000000001},"last":1.252}]}]},{"osd":0,"up_from":13,"seq":55834574921,"num_pgs":75,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":958328,"kb_used_data":92777912,"kb_used_omap":7152,"kb_used_meta":0,"kb_avail":92741052,"statfs":{"total":95948165120,"available":94966837248,"internally_reserved":37744640,"allocated":95004581888,"data_stored":95004581888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7324112,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":3.1219999999999999,"15min":1.456},"min":{"1min":0.40899999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1060000000000001,"5min":69.872,"15min":69.872},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":3.2309999999999999,"15min":1.5629999999999999},"min":{"1min":0.51300000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.2350000000000001,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.434}]},{"osd":2,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":3.4220000000000002,"15min":1.603},"min":{"1min":0.46999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.444,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":2.6560000000000001,"15min":1.3580000000000001},"min":{"1min":0.52000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.302,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.376}]},{"osd":3,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":17.571999999999999,"5min":10.789,"15min":14.782},"min":{"1min":0.53300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":199.08000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.835,"5min":7.306,"15min":2.9689999999999999},"min":{"1min":0.56599999999999995,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":15.601000000000001,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.36699999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":16.695,"15min":6.2220000000000004},"min":{"1min":0.73199999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1879999999999999,"5min":1486.309,"15min":1486.309},"last":0.443},{"interface":"front","average":{"1min":1.038,"5min":17.925000000000001,"15min":6.6580000000000004},"min":{"1min":0.82499999999999996,"5min":0.439,"15min":0.439},"max":{"1min":1.3759999999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":0.63100000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":4.6749999999999998,"15min":8.0899999999999999},"min":{"1min":0.62,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.304,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.008,"5min":1.73,"15min":1.2689999999999999},"min":{"1min":0.69199999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2529999999999999,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.54000000000000004}]},{"osd":6,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":3.5910000000000002,"15min":1.9770000000000001},"min":{"1min":0.67800000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5960000000000001,"5min":70.259,"15min":70.259},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.0880000000000001,"5min":3.0209999999999999,"15min":1.7629999999999999},"min":{"1min":0.83699999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5489999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":0.64600000000000002}]},{"osd":7,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":6.1200000000000001,"15min":12.487},"min":{"1min":0.78400000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.48,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":1.117,"5min":3.6070000000000002,"15min":1.9830000000000001},"min":{"1min":0.76100000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5229999999999999,"5min":70.322000000000003,"15min":70.322000000000003},"last":0.77800000000000002}]}]}],"pool_statfs":[]}} 2023-01-15T14:34:00.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-01-15T14:34:00.966 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:34:00.967 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:00.968 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:00.968 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:34:00.969 INFO:tasks.rados.rados.0.smithi090.stdout:1372: finishing copy_from to smithi09025264-36 2023-01-15T14:34:00.970 INFO:tasks.rados.rados.0.smithi090.stdout:1372: got expected ENOENT (src dne) 2023-01-15T14:34:00.970 INFO:tasks.rados.rados.0.smithi090.stdout:1372: finishing copy_from racing read to smithi09025264-36 2023-01-15T14:34:00.970 INFO:tasks.rados.rados.0.smithi090.stdout:1372: done (2 left) 2023-01-15T14:34:00.970 INFO:tasks.rados.rados.0.smithi090.stdout:1373: done (1 left) 2023-01-15T14:34:00.970 INFO:tasks.rados.rados.0.smithi090.stdout:1374: snap_create 2023-01-15T14:34:00.970 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:00.970 INFO:tasks.rados.rados.0.smithi090.stdout:1371: finishing write tid 1 to smithi09025264-37 2023-01-15T14:34:00.970 INFO:tasks.rados.rados.0.smithi090.stdout:1371: finishing write tid 2 to smithi09025264-37 2023-01-15T14:34:00.970 INFO:tasks.rados.rados.0.smithi090.stdout:1371: finishing write tid 3 to smithi09025264-37 2023-01-15T14:34:00.970 INFO:tasks.rados.rados.0.smithi090.stdout:1371: finishing write tid 4 to smithi09025264-37 2023-01-15T14:34:00.970 INFO:tasks.rados.rados.0.smithi090.stdout:1371: oid 37 updating version 0 to 395 2023-01-15T14:34:00.971 INFO:tasks.rados.rados.0.smithi090.stdout:1371: oid 37 version 395 is already newer than 394 2023-01-15T14:34:00.971 INFO:tasks.rados.rados.0.smithi090.stdout:1371: oid 37 version 395 is already newer than 393 2023-01-15T14:34:00.971 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 395 (ObjNum 495 snap 145 seq_num 495) dirty exists 2023-01-15T14:34:00.971 INFO:tasks.rados.rados.0.smithi090.stdout:1371: left oid 37 (ObjNum 495 snap 145 seq_num 495) 2023-01-15T14:34:00.971 INFO:tasks.rados.rados.0.smithi090.stdout:1371: done (0 left) 2023-01-15T14:34:01.125 INFO:tasks.rados.rados.0.smithi090.stdout:1374: done (0 left) 2023-01-15T14:34:01.125 INFO:tasks.rados.rados.0.smithi090.stdout:1375: rollback oid 27 current snap is 146 2023-01-15T14:34:01.125 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 27 to 120 2023-01-15T14:34:01.125 INFO:tasks.rados.rados.0.smithi090.stdout:1376: read oid 44 snap -1 2023-01-15T14:34:01.125 INFO:tasks.rados.rados.0.smithi090.stdout:1376: expect (ObjNum 473 snap 137 seq_num 473) 2023-01-15T14:34:01.125 INFO:tasks.rados.rados.0.smithi090.stdout:1377: write (excl) oid 49 current snap is 146 2023-01-15T14:34:01.125 INFO:tasks.rados.rados.0.smithi090.stdout:1377: seq_num 497 ranges {753079=561698,2033433=319646} 2023-01-15T14:34:01.126 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:34:01.127 INFO:tasks.rados.rados.0.smithi090.stdout:1377: writing smithi09025264-49 from 753079 to 1314777 tid 1 2023-01-15T14:34:01.127 INFO:tasks.rados.rados.0.smithi090.stdout:1377: writing smithi09025264-49 from 2033433 to 2353079 tid 2 2023-01-15T14:34:01.127 INFO:tasks.rados.rados.0.smithi090.stdout:1378: read oid 10 snap 145 2023-01-15T14:34:01.127 INFO:tasks.rados.rados.0.smithi090.stdout:1378: expect (ObjNum 448 snap 124 seq_num 448) 2023-01-15T14:34:01.127 INFO:tasks.rados.rados.0.smithi090.stdout:1379: write oid 11 current snap is 146 2023-01-15T14:34:01.127 INFO:tasks.rados.rados.0.smithi090.stdout:1379: seq_num 498 ranges {769886=636947,2091915=277971} 2023-01-15T14:34:01.129 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:00.964+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:01.131 INFO:tasks.rados.rados.0.smithi090.stdout:1379: writing smithi09025264-11 from 769886 to 1406833 tid 1 2023-01-15T14:34:01.131 INFO:tasks.rados.rados.0.smithi090.stdout:1379: writing smithi09025264-11 from 2091915 to 2369886 tid 2 2023-01-15T14:34:01.132 INFO:tasks.rados.rados.0.smithi090.stdout:1380: copy_from oid 26 from oid 4 current snap is 146 2023-01-15T14:34:01.132 INFO:tasks.rados.rados.0.smithi090.stdout:1381: copy_from oid 38 from oid 9 current snap is 146 2023-01-15T14:34:01.132 INFO:tasks.rados.rados.0.smithi090.stdout:1382: read oid 22 snap 145 2023-01-15T14:34:01.132 INFO:tasks.rados.rados.0.smithi090.stdout:1382: expect (ObjNum 471 snap 136 seq_num 471) 2023-01-15T14:34:01.132 INFO:tasks.rados.rados.0.smithi090.stdout:1383: read oid 36 snap -1 2023-01-15T14:34:01.132 INFO:tasks.rados.rados.0.smithi090.stdout:1383: expect deleted 2023-01-15T14:34:01.134 INFO:tasks.rados.rados.0.smithi090.stdout:1384: rollback oid 33 current snap is 146 2023-01-15T14:34:01.134 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 33 to 135 2023-01-15T14:34:01.134 INFO:tasks.rados.rados.0.smithi090.stdout:1385: copy_from oid 37 from oid 3 current snap is 146 2023-01-15T14:34:01.134 INFO:tasks.rados.rados.0.smithi090.stdout:1386: read oid 28 snap -1 2023-01-15T14:34:01.134 INFO:tasks.rados.rados.0.smithi090.stdout:1386: expect (ObjNum 466 snap 134 seq_num 466) 2023-01-15T14:34:01.134 INFO:tasks.rados.rados.0.smithi090.stdout:1387: snap_create 2023-01-15T14:34:01.134 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 12 2023-01-15T14:34:01.134 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:01.124+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:01.152 INFO:tasks.rados.rados.0.smithi090.stdout:1383: done (11 left) 2023-01-15T14:34:01.286 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 11 2023-01-15T14:34:01.288 INFO:tasks.rados.rados.0.smithi090.stdout:1375: finishing rollback tid 0 to smithi09025264-27 2023-01-15T14:34:01.289 INFO:tasks.rados.rados.0.smithi090.stdout:1386: expect (ObjNum 466 snap 134 seq_num 466) 2023-01-15T14:34:01.289 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:01.289 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:01.290 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:34:01.291 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:01.283+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:01.403 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:01.403 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:01.399+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:01.475 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:01.621 INFO:tasks.rados.rados.0.smithi090.stdout:1382: expect (ObjNum 471 snap 136 seq_num 471) 2023-01-15T14:34:01.622 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:01.622 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:01.624 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:01.624 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:34:01.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:34:01.625 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:34:01.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.625 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:01.625 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:01.625 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 334.38722649300001, 2023-01-15T14:34:01.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:34:01.626 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:01.626 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:01.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:01.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:01.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:01.627 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:34:01.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.627 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:01.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:01.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:01.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:01.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:01.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:01.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:01.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:01.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:34:01.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:01.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:01.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:34:01.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:34:01.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:01.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:34:01.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:34:01.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:01.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:34:01.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:34:01.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:01.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:34:01.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:34:01.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:01.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:34:01.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:34:01.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:01.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:34:01.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:34:01.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:01.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:34:01.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:34:01.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:34:01.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:34:01.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:34:01.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:01.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:34:01.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:34:01.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:01.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:34:01.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:34:01.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:01.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:34:01.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:34:01.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:01.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:34:01.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:34:01.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:01.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:34:01.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:34:01.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:01.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:34:01.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:34:01.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:01.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:34:01.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:34:01.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:01.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:34:01.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:34:01.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:01.820 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:01.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:01.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.820 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:01.821 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:01.821 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 334.38682315699998, 2023-01-15T14:34:01.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:34:01.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:01.821 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:01.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:01.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:01.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:01.821 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:34:01.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.821 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:01.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:01.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:01.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:01.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:01.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:01.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:01.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:01.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:34:01.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:01.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:01.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:34:01.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:34:01.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:01.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:34:01.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:34:01.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:01.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:34:01.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:34:01.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:01.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:34:01.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:34:01.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:01.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:34:01.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:34:01.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:01.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:34:01.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:34:01.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:01.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:34:01.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:34:01.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:34:01.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:34:01.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:34:01.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:01.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:34:01.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:34:01.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:01.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:34:01.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:34:01.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:01.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:34:01.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:34:01.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:34:01.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:01.834 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 334.28659953699997, 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:01.835 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:01.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:01.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:01.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:01.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:34:01.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:02.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:02.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:34:02.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:34:02.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:02.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:34:02.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:34:02.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:02.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:34:02.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:34:02.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:34:02.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:34:02.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:02.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:34:02.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:34:02.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:34:02.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:34:02.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:02.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:34:02.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:34:02.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:34:02.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:34:02.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:02.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.031 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:02.032 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:02.032 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.59328431200001, 2023-01-15T14:34:02.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:34:02.032 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:02.032 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:02.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:02.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:02.032 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:02.032 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:34:02.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.032 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:02.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:02.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:02.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:02.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:02.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:02.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:34:02.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:02.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:02.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:34:02.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:34:02.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:02.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:34:02.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:34:02.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:02.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:34:02.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:34:02.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:34:02.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:34:02.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:02.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:34:02.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:34:02.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:34:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:34:02.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:34:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:34:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.212 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:02.213 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:02.214 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.586822958, 2023-01-15T14:34:02.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:34:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:34:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:02.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:34:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:34:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:34:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:34:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:34:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:34:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:34:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:34:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:34:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:34:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:34:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:34:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:34:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:34:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:34:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.48640103299999, 2023-01-15T14:34:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:34:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:02.224 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:34:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:34:02.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:02.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:34:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:34:02.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:02.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:34:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:34:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:34:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:34:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:34:02.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:34:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:34:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:34:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:34:02.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:34:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:34:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:34:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.446 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.48606354699999, 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:02.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:02.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:34:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:34:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:34:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:34:02.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:34:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:34:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:34:02.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:34:02.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:34:02.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:02.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:34:02.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:34:02.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:34:02.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:34:02.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:02.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.48600765099999, 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:02.601 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:34:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:02.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:02.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:02.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:34:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:34:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:34:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:02.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:34:02.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:34:02.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:34:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:34:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:34:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:34:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:02.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:34:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:34:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:34:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:34:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:34:02.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 201.65275723299999, 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:02.617 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:02.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:34:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:34:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:34:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:34:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:34:02.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:34:02.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:34:02.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:34:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:34:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:34:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:34:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2207 3.5 3:a1c47f99:::smithi09025264-2:head [write 2548663~763022] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 142.677999505, 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79657873300000004, 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2207 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:02.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:34:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:34:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837590+0000", 2023-01-15T14:34:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:02.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837960+0000", 2023-01-15T14:34:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037055900000000001 2023-01-15T14:34:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837980+0000", 2023-01-15T14:34:02.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9953e-05 2023-01-15T14:34:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837988+0000", 2023-01-15T14:34:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2800000000000003e-06 2023-01-15T14:34:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927512+0000", 2023-01-15T14:34:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089523563 2023-01-15T14:34:02.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927645+0000", 2023-01-15T14:34:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013285799999999999 2023-01-15T14:34:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225466+0000", 2023-01-15T14:34:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29782133100000002 2023-01-15T14:34:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225553+0000", 2023-01-15T14:34:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6866000000000005e-05 2023-01-15T14:34:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225708+0000", 2023-01-15T14:34:02.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015455199999999999 2023-01-15T14:34:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225971+0000", 2023-01-15T14:34:02.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026317099999999998 2023-01-15T14:34:02.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:02.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226027+0000", 2023-01-15T14:34:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6601000000000002e-05 2023-01-15T14:34:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:02.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226470+0000", 2023-01-15T14:34:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044267500000000001 2023-01-15T14:34:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:02.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227372+0000", 2023-01-15T14:34:02.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090215200000000003 2023-01-15T14:34:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429314+0000", 2023-01-15T14:34:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20194183499999999 2023-01-15T14:34:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.633999+0000", 2023-01-15T14:34:02.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20468505400000001 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634075+0000", 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634170+0000", 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4821999999999994e-05 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.843 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2208 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (56) in=7b,truncate 3395047] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 142.67757691200001, 2023-01-15T14:34:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79650439900000003, 2023-01-15T14:34:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:02.844 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2208 2023-01-15T14:34:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:02.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:02.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838018+0000", 2023-01-15T14:34:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2549999999999997e-06 2023-01-15T14:34:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838031+0000", 2023-01-15T14:34:02.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2717999999999999e-05 2023-01-15T14:34:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838036+0000", 2023-01-15T14:34:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2240000000000003e-06 2023-01-15T14:34:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927676+0000", 2023-01-15T14:34:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089639620000000003 2023-01-15T14:34:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927804+0000", 2023-01-15T14:34:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012832300000000001 2023-01-15T14:34:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226096+0000", 2023-01-15T14:34:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29829207400000002 2023-01-15T14:34:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226347+0000", 2023-01-15T14:34:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025084800000000002 2023-01-15T14:34:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226750+0000", 2023-01-15T14:34:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040343200000000001 2023-01-15T14:34:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227147+0000", 2023-01-15T14:34:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039682900000000002 2023-01-15T14:34:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227600+0000", 2023-01-15T14:34:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045321199999999998 2023-01-15T14:34:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227859+0000", 2023-01-15T14:34:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259041 2023-01-15T14:34:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:02.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227895+0000", 2023-01-15T14:34:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5710999999999997e-05 2023-01-15T14:34:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429405+0000", 2023-01-15T14:34:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201509415 2023-01-15T14:34:02.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634300+0000", 2023-01-15T14:34:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20489528200000001 2023-01-15T14:34:02.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634409+0000", 2023-01-15T14:34:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109524 2023-01-15T14:34:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634518+0000", 2023-01-15T14:34:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108548 2023-01-15T14:34:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2209 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 142.677548097, 2023-01-15T14:34:02.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79724995499999995, 2023-01-15T14:34:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2209 2023-01-15T14:34:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:02.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838044+0000", 2023-01-15T14:34:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6120000000000001e-06 2023-01-15T14:34:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:02.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838056+0000", 2023-01-15T14:34:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1236000000000001e-05 2023-01-15T14:34:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838059+0000", 2023-01-15T14:34:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6940000000000001e-06 2023-01-15T14:34:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.027908+0000", 2023-01-15T14:34:02.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189848287 2023-01-15T14:34:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.028049+0000", 2023-01-15T14:34:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014132500000000001 2023-01-15T14:34:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227252+0000", 2023-01-15T14:34:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19920324 2023-01-15T14:34:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227341+0000", 2023-01-15T14:34:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8683000000000005e-05 2023-01-15T14:34:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634637+0000", 2023-01-15T14:34:03.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729567300000002 2023-01-15T14:34:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634980+0000", 2023-01-15T14:34:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034335000000000002 2023-01-15T14:34:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.635292+0000", 2023-01-15T14:34:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003123 2023-01-15T14:34:03.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2222 3.c 3:3137f0cb:::smithi09025264-20:head [write 3259733~152121] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:34:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 142.65769653800001, 2023-01-15T14:34:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77651572400000002, 2023-01-15T14:34:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:03.003 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2222 2023-01-15T14:34:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:34:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:34:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857891+0000", 2023-01-15T14:34:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857993+0000", 2023-01-15T14:34:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010276399999999999 2023-01-15T14:34:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858021+0000", 2023-01-15T14:34:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8022e-05 2023-01-15T14:34:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858034+0000", 2023-01-15T14:34:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2549e-05 2023-01-15T14:34:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025371+0000", 2023-01-15T14:34:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16733653100000001 2023-01-15T14:34:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025429+0000", 2023-01-15T14:34:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8922000000000003e-05 2023-01-15T14:34:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226869+0000", 2023-01-15T14:34:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20143931100000001 2023-01-15T14:34:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:03.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227088+0000", 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219328 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227443+0000", 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035481 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227830+0000", 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000386702 2023-01-15T14:34:03.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:03.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227861+0000", 2023-01-15T14:34:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1584000000000001e-05 2023-01-15T14:34:03.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227939+0000", 2023-01-15T14:34:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7354999999999994e-05 2023-01-15T14:34:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228548+0000", 2023-01-15T14:34:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060954800000000001 2023-01-15T14:34:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429817+0000", 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20126904600000001 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634125+0000", 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20430780300000001 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:03.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634259+0000", 2023-01-15T14:34:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134144 2023-01-15T14:34:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:03.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634410+0000", 2023-01-15T14:34:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150447 2023-01-15T14:34:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2223 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (57) in=7b,truncate 3411854] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:34:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 142.65753775100001, 2023-01-15T14:34:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87671107599999998, 2023-01-15T14:34:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:03.181 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2223 2023-01-15T14:34:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:34:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:34:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858052+0000", 2023-01-15T14:34:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858056+0000", 2023-01-15T14:34:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5619999999999997e-06 2023-01-15T14:34:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858067+0000", 2023-01-15T14:34:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0328e-05 2023-01-15T14:34:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858071+0000", 2023-01-15T14:34:03.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6439999999999999e-06 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025446+0000", 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16737407200000001 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025503+0000", 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7952999999999999e-05 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227970+0000", 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202466866 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228139+0000", 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016889100000000001 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:03.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228284+0000", 2023-01-15T14:34:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014493700000000001 2023-01-15T14:34:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228470+0000", 2023-01-15T14:34:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185541 2023-01-15T14:34:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228481+0000", 2023-01-15T14:34:03.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.167e-05 2023-01-15T14:34:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228537+0000", 2023-01-15T14:34:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5912999999999998e-05 2023-01-15T14:34:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228563+0000", 2023-01-15T14:34:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5613e-05 2023-01-15T14:34:03.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.433181+0000", 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20461838099999999 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734620+0000", 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301438386 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734682+0000", 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2373999999999996e-05 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734764+0000", 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1656999999999994e-05 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2224 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 142.657512701, 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97704279199999999, 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2224 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.192 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:03.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858077+0000", 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:03.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858080+0000", 2023-01-15T14:34:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:34:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858089+0000", 2023-01-15T14:34:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3449999999999995e-06 2023-01-15T14:34:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858093+0000", 2023-01-15T14:34:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9820000000000002e-06 2023-01-15T14:34:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025520+0000", 2023-01-15T14:34:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.167426556 2023-01-15T14:34:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025594+0000", 2023-01-15T14:34:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4339999999999996e-05 2023-01-15T14:34:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228608+0000", 2023-01-15T14:34:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20301395699999999 2023-01-15T14:34:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228752+0000", 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014385 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834878+0000", 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60612591500000002 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834987+0000", 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108733 2023-01-15T14:34:03.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.835120+0000", 2023-01-15T14:34:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133688 2023-01-15T14:34:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.356 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3120 3.40 3:03b54904:::smithi09025264-12:head [write 578273~466432] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:34:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 52.583560818999999, 2023-01-15T14:34:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72661194799999995, 2023-01-15T14:34:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3120 2023-01-15T14:34:03.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:34:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:34:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932026+0000", 2023-01-15T14:34:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932397+0000", 2023-01-15T14:34:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037070099999999997 2023-01-15T14:34:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932419+0000", 2023-01-15T14:34:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1923999999999999e-05 2023-01-15T14:34:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932435+0000", 2023-01-15T14:34:03.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6509999999999999e-05 2023-01-15T14:34:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933751+0000", 2023-01-15T14:34:03.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013156050000000001 2023-01-15T14:34:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933765+0000", 2023-01-15T14:34:03.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3952e-05 2023-01-15T14:34:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238456+0000", 2023-01-15T14:34:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30469085000000001 2023-01-15T14:34:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238797+0000", 2023-01-15T14:34:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034117300000000001 2023-01-15T14:34:03.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239214+0000", 2023-01-15T14:34:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000416845 2023-01-15T14:34:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240273+0000", 2023-01-15T14:34:03.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058811 2023-01-15T14:34:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240292+0000", 2023-01-15T14:34:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:34:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240555+0000", 2023-01-15T14:34:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026299500000000003 2023-01-15T14:34:03.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242072+0000", 2023-01-15T14:34:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015175550000000001 2023-01-15T14:34:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244038+0000", 2023-01-15T14:34:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001966176 2023-01-15T14:34:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658099+0000", 2023-01-15T14:34:03.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41406103300000002 2023-01-15T14:34:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658248+0000", 2023-01-15T14:34:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148577 2023-01-15T14:34:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658642+0000", 2023-01-15T14:34:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039346900000000002 2023-01-15T14:34:03.369 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3121 3.40 3:03b54904:::smithi09025264-12:head [write 1739595~452815] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:34:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 52.583128873, 2023-01-15T14:34:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72675957599999996, 2023-01-15T14:34:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:03.370 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3121 2023-01-15T14:34:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:34:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:34:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932460+0000", 2023-01-15T14:34:03.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932725+0000", 2023-01-15T14:34:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026482699999999998 2023-01-15T14:34:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932741+0000", 2023-01-15T14:34:03.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5880999999999999e-05 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932750+0000", 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2380000000000003e-06 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034479+0000", 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10172906399999999 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034506+0000", 2023-01-15T14:34:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7336000000000001e-05 2023-01-15T14:34:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240672+0000", 2023-01-15T14:34:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206166074 2023-01-15T14:34:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240894+0000", 2023-01-15T14:34:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221189 2023-01-15T14:34:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241244+0000", 2023-01-15T14:34:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000350816 2023-01-15T14:34:03.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241923+0000", 2023-01-15T14:34:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067891200000000005 2023-01-15T14:34:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241943+0000", 2023-01-15T14:34:03.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0069999999999999e-05 2023-01-15T14:34:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243425+0000", 2023-01-15T14:34:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001481719 2023-01-15T14:34:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244087+0000", 2023-01-15T14:34:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066184700000000002 2023-01-15T14:34:03.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244230+0000", 2023-01-15T14:34:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143277 2023-01-15T14:34:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658784+0000", 2023-01-15T14:34:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.414553479 2023-01-15T14:34:03.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658904+0000", 2023-01-15T14:34:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120024 2023-01-15T14:34:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659221+0000", 2023-01-15T14:34:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031742700000000001 2023-01-15T14:34:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3122 3.40 3:03b54904:::smithi09025264-12:head [write 2722929~655344] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:03.381 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:34:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 52.582824774999999, 2023-01-15T14:34:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72706898900000005, 2023-01-15T14:34:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3122 2023-01-15T14:34:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.382 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:34:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:34:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932765+0000", 2023-01-15T14:34:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933097+0000", 2023-01-15T14:34:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220299999999999 2023-01-15T14:34:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933106+0000", 2023-01-15T14:34:03.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5459999999999997e-06 2023-01-15T14:34:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933113+0000", 2023-01-15T14:34:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3419999999999996e-06 2023-01-15T14:34:03.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034598+0000", 2023-01-15T14:34:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101485804 2023-01-15T14:34:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034625+0000", 2023-01-15T14:34:03.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6859000000000002e-05 2023-01-15T14:34:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242147+0000", 2023-01-15T14:34:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20752189600000001 2023-01-15T14:34:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242517+0000", 2023-01-15T14:34:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037032200000000002 2023-01-15T14:34:03.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243157+0000", 2023-01-15T14:34:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063935700000000001 2023-01-15T14:34:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243809+0000", 2023-01-15T14:34:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065222000000000001 2023-01-15T14:34:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243827+0000", 2023-01-15T14:34:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8260000000000001e-05 2023-01-15T14:34:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244122+0000", 2023-01-15T14:34:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029469400000000002 2023-01-15T14:34:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245633+0000", 2023-01-15T14:34:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001511068 2023-01-15T14:34:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.346711+0000", 2023-01-15T14:34:03.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101077812 2023-01-15T14:34:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659372+0000", 2023-01-15T14:34:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31266154800000001 2023-01-15T14:34:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659488+0000", 2023-01-15T14:34:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115444 2023-01-15T14:34:03.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659835+0000", 2023-01-15T14:34:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034670300000000002 2023-01-15T14:34:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3123 3.40 3:03b54904:::smithi09025264-12:head [setxattr _header (58) in=7b,truncate 3378273] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:03.518 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 52.582455082000003, 2023-01-15T14:34:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82751564099999997, 2023-01-15T14:34:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3123 2023-01-15T14:34:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:03.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933139+0000", 2023-01-15T14:34:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7079999999999996e-06 2023-01-15T14:34:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:03.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933145+0000", 2023-01-15T14:34:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7849999999999997e-06 2023-01-15T14:34:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033331+0000", 2023-01-15T14:34:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100186076 2023-01-15T14:34:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034697+0000", 2023-01-15T14:34:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001366138 2023-01-15T14:34:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034718+0000", 2023-01-15T14:34:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.065e-05 2023-01-15T14:34:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244319+0000", 2023-01-15T14:34:03.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20960071 2023-01-15T14:34:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244576+0000", 2023-01-15T14:34:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025712899999999998 2023-01-15T14:34:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244932+0000", 2023-01-15T14:34:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035626000000000001 2023-01-15T14:34:03.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245315+0000", 2023-01-15T14:34:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038305199999999999 2023-01-15T14:34:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245330+0000", 2023-01-15T14:34:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5027e-05 2023-01-15T14:34:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245384+0000", 2023-01-15T14:34:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4267000000000001e-05 2023-01-15T14:34:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245654+0000", 2023-01-15T14:34:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026944000000000002 2023-01-15T14:34:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.436220+0000", 2023-01-15T14:34:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19056593799999999 2023-01-15T14:34:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760086+0000", 2023-01-15T14:34:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.323865927 2023-01-15T14:34:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:03.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760241+0000", 2023-01-15T14:34:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015535700000000001 2023-01-15T14:34:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760651+0000", 2023-01-15T14:34:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040990699999999999 2023-01-15T14:34:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3124 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:34:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 52.482228603999999, 2023-01-15T14:34:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72792673799999996, 2023-01-15T14:34:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3124 2023-01-15T14:34:03.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:34:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:34:03.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033360+0000", 2023-01-15T14:34:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033372+0000", 2023-01-15T14:34:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2850000000000001e-05 2023-01-15T14:34:03.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033397+0000", 2023-01-15T14:34:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4349e-05 2023-01-15T14:34:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033412+0000", 2023-01-15T14:34:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5420000000000001e-05 2023-01-15T14:34:03.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036811+0000", 2023-01-15T14:34:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033984169999999999 2023-01-15T14:34:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036821+0000", 2023-01-15T14:34:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0213e-05 2023-01-15T14:34:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245712+0000", 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20889102800000001 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245912+0000", 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020024600000000001 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:03.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760781+0000", 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51486881299999998 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760970+0000", 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018893600000000001 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.761289+0000", 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031870899999999997 2023-01-15T14:34:03.535 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:03.536 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:03.536 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:03.536 INFO:teuthology.orchestra.run.smithi090.stdout:{"fsid":"1c77fb2b-d1a7-4884-a24e-3935e16d0b6b","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive, 1 pg peering","count":2},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 162/462 objects degraded (35.065%), 11 pgs degraded","count":11},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":363,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":352,"num_osds":8,"num_up_osds":8,"osd_up_since":1673792884,"num_in_osds":6,"osd_in_since":1673793049,"num_remapped_pgs":16},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":72},{"state_name":"active+clean+snaptrim_wait","count":11},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":8},{"state_name":"remapped+peering","count":6},{"state_name":"active+clean+snaptrim","count":4},{"state_name":"peering","count":4},{"state_name":"activating+undersized+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":2},{"state_name":"active+undersized+remapped","count":2},{"state_name":"active+recovery_wait+remapped","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovery_wait+degraded+remapped","count":1},{"state_name":"activating+undersized+degraded+remapped","count":1}],"num_pgs":115,"num_pools":3,"num_objects":154,"data_bytes":338926428,"bytes_used":6395895808,"bytes_avail":569293094912,"bytes_total":575688990720,"inactive_pgs_ratio":0.11304347962141037,"degraded_objects":162,"degraded_total":462,"degraded_ratio":0.35064935064935066,"misplaced_objects":34,"misplaced_total":462,"misplaced_ratio":0.073593073593073599,"recovering_objects_per_sec":23,"recovering_bytes_per_sec":16366558,"recovering_keys_per_sec":0,"num_objects_recovered":92,"num_bytes_recovered":63588880,"num_keys_recovered":0,"read_bytes_sec":44479886,"write_bytes_sec":12020087,"read_op_per_sec":38,"write_op_per_sec":24},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2023-01-15T14:29:06.857309+0000","services":{}},"progress_events":{}} 2023-01-15T14:34:03.537 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:03.538 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:34:03.539 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2023-01-15T14:34:03.539 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:34:03.540 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:34:03.541 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:01.619+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:03.541 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:03.541 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:03.541 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:03.539+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:03.541 INFO:tasks.rados.rados.0.smithi090.stdout:1376: expect (ObjNum 473 snap 137 seq_num 473) 2023-01-15T14:34:03.542 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:03.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:34:03.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:34:03.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:34:03.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:34:03.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:34:03.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:34:03.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:34:03.543 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:34:03.545 INFO:tasks.rados.rados.0.smithi090.stdout:1385: finishing copy_from to smithi09025264-37 2023-01-15T14:34:03.545 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 351 (ObjNum 464 snap 133 seq_num 464) dirty exists 2023-01-15T14:34:03.545 INFO:tasks.rados.rados.0.smithi090.stdout:1385: finishing copy_from racing read to smithi09025264-37 2023-01-15T14:34:03.545 INFO:tasks.rados.rados.0.smithi090.stdout:1384: finishing rollback tid 0 to smithi09025264-33 2023-01-15T14:34:03.545 INFO:tasks.rados.rados.0.smithi090.stdout:1381: finishing copy_from to smithi09025264-38 2023-01-15T14:34:03.545 INFO:tasks.rados.rados.0.smithi090.stdout:1381: got expected ENOENT (src dne) 2023-01-15T14:34:03.546 INFO:tasks.rados.rados.0.smithi090.stdout:1381: finishing copy_from racing read to smithi09025264-38 2023-01-15T14:34:03.546 INFO:tasks.rados.rados.0.smithi090.stdout:1376: done (10 left) 2023-01-15T14:34:03.546 INFO:tasks.rados.rados.0.smithi090.stdout:1381: done (9 left) 2023-01-15T14:34:03.546 INFO:tasks.rados.rados.0.smithi090.stdout:1382: done (8 left) 2023-01-15T14:34:03.546 INFO:tasks.rados.rados.0.smithi090.stdout:1385: done (7 left) 2023-01-15T14:34:03.546 INFO:tasks.rados.rados.0.smithi090.stdout:1386: done (6 left) 2023-01-15T14:34:03.546 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:34:03.546 INFO:tasks.rados.rados.0.smithi090.stdout:1377: finishing write tid 1 to smithi09025264-49 2023-01-15T14:34:03.546 INFO:tasks.rados.rados.0.smithi090.stdout:1379: finishing write tid 1 to smithi09025264-11 2023-01-15T14:34:03.546 INFO:tasks.rados.rados.0.smithi090.stdout:1377: finishing write tid 2 to smithi09025264-49 2023-01-15T14:34:03.546 INFO:tasks.rados.rados.0.smithi090.stdout:1379: finishing write tid 2 to smithi09025264-11 2023-01-15T14:34:03.547 INFO:tasks.rados.rados.0.smithi090.stdout:1379: finishing write tid 3 to smithi09025264-11 2023-01-15T14:34:03.547 INFO:tasks.rados.rados.0.smithi090.stdout:1377: finishing write tid 3 to smithi09025264-49 2023-01-15T14:34:03.547 INFO:tasks.rados.rados.0.smithi090.stdout:1379: finishing write tid 4 to smithi09025264-11 2023-01-15T14:34:03.547 INFO:tasks.rados.rados.0.smithi090.stdout:1379: oid 11 updating version 0 to 347 2023-01-15T14:34:03.547 INFO:tasks.rados.rados.0.smithi090.stdout:1379: oid 11 updating version 347 to 349 2023-01-15T14:34:03.547 INFO:tasks.rados.rados.0.smithi090.stdout:1379: oid 11 version 349 is already newer than 348 2023-01-15T14:34:03.547 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 11 v 349 (ObjNum 498 snap 146 seq_num 498) dirty exists 2023-01-15T14:34:03.547 INFO:tasks.rados.rados.0.smithi090.stdout:1379: left oid 11 (ObjNum 498 snap 146 seq_num 498) 2023-01-15T14:34:03.547 INFO:tasks.rados.rados.0.smithi090.stdout:1379: done (5 left) 2023-01-15T14:34:03.547 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:34:03.547 INFO:tasks.rados.rados.0.smithi090.stdout:1375: finishing rollback tid 1 to smithi09025264-27 2023-01-15T14:34:03.548 INFO:tasks.rados.rados.0.smithi090.stdout:1375: finishing rollback tid 2 to smithi09025264-27 2023-01-15T14:34:03.548 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 266 (ObjNum 421 snap 112 seq_num 421) dirty exists 2023-01-15T14:34:03.548 INFO:tasks.rados.rados.0.smithi090.stdout:1375: done (4 left) 2023-01-15T14:34:03.548 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:34:03.548 INFO:tasks.rados.rados.0.smithi090.stdout:1377: finishing write tid 4 to smithi09025264-49 2023-01-15T14:34:03.548 INFO:tasks.rados.rados.0.smithi090.stdout:1377: oid 49 updating version 0 to 359 2023-01-15T14:34:03.548 INFO:tasks.rados.rados.0.smithi090.stdout:1377: oid 49 updating version 359 to 361 2023-01-15T14:34:03.548 INFO:tasks.rados.rados.0.smithi090.stdout:1377: oid 49 version 361 is already newer than 360 2023-01-15T14:34:03.548 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 361 (ObjNum 497 snap 146 seq_num 497) dirty exists 2023-01-15T14:34:03.548 INFO:tasks.rados.rados.0.smithi090.stdout:1377: left oid 49 (ObjNum 497 snap 146 seq_num 497) 2023-01-15T14:34:03.548 INFO:tasks.rados.rados.0.smithi090.stdout:1377: done (3 left) 2023-01-15T14:34:03.549 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:34:03.549 INFO:tasks.rados.rados.0.smithi090.stdout:1380: finishing copy_from to smithi09025264-26 2023-01-15T14:34:03.549 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 335 (ObjNum 412 snap 112 seq_num 412) dirty exists 2023-01-15T14:34:03.549 INFO:tasks.rados.rados.0.smithi090.stdout:1380: finishing copy_from racing read to smithi09025264-26 2023-01-15T14:34:03.549 INFO:tasks.rados.rados.0.smithi090.stdout:1380: done (2 left) 2023-01-15T14:34:03.549 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:34:03.549 INFO:tasks.rados.rados.0.smithi090.stdout:1378: expect (ObjNum 448 snap 124 seq_num 448) 2023-01-15T14:34:03.549 INFO:tasks.rados.rados.0.smithi090.stdout:1378: done (1 left) 2023-01-15T14:34:03.549 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:03.642 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:03.636+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:03.675 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:03.676 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:03.676 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:34:03.676 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:03.676 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:34:03.743 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:03.740+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:03.780 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:03.836 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:03.837 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:34:03.837 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:34:03.837 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:03.838 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:34:03.844 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:03.840+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:03.896 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:04.000 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:34:04.001 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":525,"stamp":"2023-01-15T14:34:02.950774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":348673921,"num_objects":162,"num_object_clones":109,"num_object_copies":486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":183,"num_objects_misplaced":54,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":14,"num_read":1129,"num_read_kb":1099300,"num_write":956,"num_write_kb":329448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":498,"num_bytes_recovered":295159559,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18782,"ondisk_log_size":18782,"up":345,"acting":330,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":401,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6398736,"kb_used_data":555906036,"kb_used_omap":67408,"kb_used_meta":0,"kb_avail":555797544,"statfs":{"total":575688990720,"available":569136685056,"internally_reserved":111095808,"allocated":569247780864,"data_stored":569247780864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":69026421,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":14,"commit_latency_ns":0,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9747493,"num_objects":8,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":40,"num_read_kb":71315,"num_write":18,"num_write_kb":6520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":53299204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005486"},"pg_stats":[{"pgid":"3.69","version":"110'149","reported_seq":895,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.316235+0000","last_change":"2023-01-15T14:33:57.316235+0000","last_active":"2023-01-15T14:33:57.316235+0000","last_peered":"2023-01-15T14:33:57.316235+0000","last_clean":"2023-01-15T14:33:57.316235+0000","last_became_active":"2023-01-15T14:33:50.216417+0000","last_became_peered":"2023-01-15T14:33:50.216417+0000","last_unstale":"2023-01-15T14:33:57.316235+0000","last_undegraded":"2023-01-15T14:33:57.316235+0000","last_fullsized":"2023-01-15T14:33:57.316235+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.406510072,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.68","version":"351'252","reported_seq":1403,"reported_epoch":353,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2023-01-15T14:34:01.933758+0000","last_change":"2023-01-15T14:33:53.618260+0000","last_active":"2023-01-15T14:34:01.933758+0000","last_peered":"2023-01-15T14:34:01.933758+0000","last_clean":"2023-01-15T14:33:42.635266+0000","last_became_active":"2023-01-15T14:33:53.330670+0000","last_became_peered":"2023-01-15T14:33:53.330670+0000","last_unstale":"2023-01-15T14:34:01.933758+0000","last_undegraded":"2023-01-15T14:33:52.928061+0000","last_fullsized":"2023-01-15T14:34:01.933758+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8014979,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":9643107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[1,7,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":3},{"shards":"1,2,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.67","version":"191'79","reported_seq":54,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.011883+0000","last_change":"2023-01-15T14:33:57.312535+0000","last_active":"2023-01-15T14:34:02.011883+0000","last_peered":"2023-01-15T14:34:02.011883+0000","last_clean":"2023-01-15T14:34:02.011883+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:34:02.011883+0000","last_undegraded":"2023-01-15T14:34:02.011883+0000","last_fullsized":"2023-01-15T14:34:02.011883+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.40508863699999997,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.66","version":"102'52","reported_seq":358,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.299510+0000","last_change":"2023-01-15T14:33:57.299510+0000","last_active":"2023-01-15T14:33:57.299510+0000","last_peered":"2023-01-15T14:33:57.299510+0000","last_clean":"2023-01-15T14:33:57.299510+0000","last_became_active":"2023-01-15T14:33:48.695009+0000","last_became_peered":"2023-01-15T14:33:48.695009+0000","last_unstale":"2023-01-15T14:33:57.299510+0000","last_undegraded":"2023-01-15T14:33:57.299510+0000","last_fullsized":"2023-01-15T14:33:57.299510+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.390920724,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2142,"num_write":4,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.65","version":"349'120","reported_seq":1157,"reported_epoch":353,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:01.932879+0000","last_change":"2023-01-15T14:33:57.203450+0000","last_active":"2023-01-15T14:34:01.932879+0000","last_peered":"2023-01-15T14:34:01.932879+0000","last_clean":"2023-01-15T14:33:43.349693+0000","last_became_active":"2023-01-15T14:33:57.201006+0000","last_became_peered":"2023-01-15T14:33:57.201006+0000","last_unstale":"2023-01-15T14:34:01.932879+0000","last_undegraded":"2023-01-15T14:33:56.909186+0000","last_fullsized":"2023-01-15T14:33:56.908621+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7224063,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":3033,"num_write":12,"num_write_kb":4201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":11126498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,4],"avail_no_missing":["1","2","3","4"],"object_location_counts":[{"shards":"0,1,2,3,4","objects":1},{"shards":"1,2,3,4","objects":1},{"shards":"1,2,4","objects":1},{"shards":"1,4","objects":-1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.64","version":"334'251","reported_seq":1409,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.039370+0000","last_change":"2023-01-15T14:33:57.417656+0000","last_active":"2023-01-15T14:34:00.039370+0000","last_peered":"2023-01-15T14:34:00.039370+0000","last_clean":"2023-01-15T14:34:00.039370+0000","last_became_active":"2023-01-15T14:33:55.675085+0000","last_became_peered":"2023-01-15T14:33:55.675085+0000","last_unstale":"2023-01-15T14:34:00.039370+0000","last_undegraded":"2023-01-15T14:34:00.039370+0000","last_fullsized":"2023-01-15T14:34:00.039370+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.409133782,"stat_sum":{"num_bytes":3548733,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10773,"num_write":5,"num_write_kb":2741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.63","version":"149'233","reported_seq":738,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.040167+0000","last_change":"2023-01-15T14:33:57.518738+0000","last_active":"2023-01-15T14:34:00.040167+0000","last_peered":"2023-01-15T14:34:00.040167+0000","last_clean":"2023-01-15T14:34:00.040167+0000","last_became_active":"2023-01-15T14:33:55.277739+0000","last_became_peered":"2023-01-15T14:33:55.277739+0000","last_unstale":"2023-01-15T14:34:00.040167+0000","last_undegraded":"2023-01-15T14:34:00.040167+0000","last_fullsized":"2023-01-15T14:34:00.040167+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.41396429299999998,"stat_sum":{"num_bytes":3802136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":21819,"num_write":19,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.62","version":"348'169","reported_seq":1347,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.032467+0000","last_change":"2023-01-15T14:33:58.705842+0000","last_active":"2023-01-15T14:34:02.032467+0000","last_peered":"2023-01-15T14:34:02.032467+0000","last_clean":"2023-01-15T14:34:02.032467+0000","last_became_active":"2023-01-15T14:33:56.590183+0000","last_became_peered":"2023-01-15T14:33:56.590183+0000","last_unstale":"2023-01-15T14:34:02.032467+0000","last_undegraded":"2023-01-15T14:34:02.032467+0000","last_fullsized":"2023-01-15T14:34:02.032467+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.70374445200000002,"stat_sum":{"num_bytes":5361549,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6734,"num_write":11,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5210269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.61","version":"321'138","reported_seq":999,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:33:59.938821+0000","last_change":"2023-01-15T14:33:57.185692+0000","last_active":"2023-01-15T14:33:59.938821+0000","last_peered":"2023-01-15T14:33:59.938821+0000","last_clean":"2023-01-15T14:33:59.938821+0000","last_became_active":"2023-01-15T14:33:53.421056+0000","last_became_peered":"2023-01-15T14:33:53.421056+0000","last_unstale":"2023-01-15T14:33:59.938821+0000","last_undegraded":"2023-01-15T14:33:59.938821+0000","last_fullsized":"2023-01-15T14:33:59.938821+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.27715718700000003,"stat_sum":{"num_bytes":1033664,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6059,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.60","version":"348'281","reported_seq":1410,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:01.931888+0000","last_change":"2023-01-15T14:33:58.810339+0000","last_active":"2023-01-15T14:34:01.931888+0000","last_peered":"2023-01-15T14:34:01.931888+0000","last_clean":"2023-01-15T14:34:01.931888+0000","last_became_active":"2023-01-15T14:33:56.478613+0000","last_became_peered":"2023-01-15T14:33:56.478613+0000","last_unstale":"2023-01-15T14:34:01.931888+0000","last_undegraded":"2023-01-15T14:34:01.931888+0000","last_fullsized":"2023-01-15T14:34:01.931888+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.80913330699999997,"stat_sum":{"num_bytes":5214108,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":10865,"num_write":14,"num_write_kb":5082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15418818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1372,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:33:59.940188+0000","last_change":"2023-01-15T14:33:57.313450+0000","last_active":"2023-01-15T14:33:59.940188+0000","last_peered":"2023-01-15T14:33:59.940188+0000","last_clean":"2023-01-15T14:33:59.940188+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:33:59.940188+0000","last_undegraded":"2023-01-15T14:33:59.940188+0000","last_fullsized":"2023-01-15T14:33:59.940188+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:51.919","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.39161994999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5e","version":"215'39","reported_seq":974,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.316972+0000","last_change":"2023-01-15T14:33:57.316972+0000","last_active":"2023-01-15T14:33:57.316972+0000","last_peered":"2023-01-15T14:33:57.316972+0000","last_clean":"2023-01-15T14:33:57.316972+0000","last_became_active":"2023-01-15T14:33:35.364143+0000","last_became_peered":"2023-01-15T14:33:35.364143+0000","last_unstale":"2023-01-15T14:33:57.316972+0000","last_undegraded":"2023-01-15T14:33:57.316972+0000","last_fullsized":"2023-01-15T14:33:57.316972+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40693008000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1285,"reported_epoch":349,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:33:58.213615+0000","last_change":"2023-01-15T14:33:58.213117+0000","last_active":"2023-01-15T14:33:58.213615+0000","last_peered":"2023-01-15T14:33:58.213615+0000","last_clean":"2023-01-15T14:33:55.169917+0000","last_became_active":"2023-01-15T14:33:58.213117+0000","last_became_peered":"2023-01-15T14:33:58.213117+0000","last_unstale":"2023-01-15T14:33:58.213615+0000","last_undegraded":"2023-01-15T14:33:58.213615+0000","last_fullsized":"2023-01-15T14:33:57.907045+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10421003,"stat_sum":{"num_bytes":2869928,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":23811,"num_write":13,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,0],"avail_no_missing":["4","0","1","7"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.5c","version":"141'46","reported_seq":75,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.009446+0000","last_change":"2023-01-15T14:33:58.026285+0000","last_active":"2023-01-15T14:33:58.009446+0000","last_peered":"2023-01-15T14:33:58.009446+0000","last_clean":"2023-01-15T14:33:58.009446+0000","last_became_active":"2023-01-15T14:33:57.113032+0000","last_became_peered":"2023-01-15T14:33:57.113032+0000","last_unstale":"2023-01-15T14:33:58.009446+0000","last_undegraded":"2023-01-15T14:33:58.009446+0000","last_fullsized":"2023-01-15T14:33:58.009446+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.121922606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5b","version":"283'80","reported_seq":911,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.010948+0000","last_change":"2023-01-15T14:33:57.113633+0000","last_active":"2023-01-15T14:34:02.010948+0000","last_peered":"2023-01-15T14:34:02.010948+0000","last_clean":"2023-01-15T14:34:02.010948+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:34:02.010948+0000","last_undegraded":"2023-01-15T14:34:02.010948+0000","last_fullsized":"2023-01-15T14:34:02.010948+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.20934807,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5a","version":"329'348","reported_seq":1483,"reported_epoch":353,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:01.932219+0000","last_change":"2023-01-15T14:33:53.433694+0000","last_active":"2023-01-15T14:34:01.932219+0000","last_peered":"2023-01-15T14:34:01.932219+0000","last_clean":"2023-01-15T14:33:50.015328+0000","last_became_active":"2023-01-15T14:33:53.229261+0000","last_became_peered":"2023-01-15T14:33:53.229261+0000","last_unstale":"2023-01-15T14:34:01.932219+0000","last_undegraded":"2023-01-15T14:33:52.914214+0000","last_fullsized":"2023-01-15T14:33:52.913106+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:02.755","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.45505085000000001,"stat_sum":{"num_bytes":5983424,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":19294,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[1,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.59","version":"294'256","reported_seq":1272,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.406685+0000","last_change":"2023-01-15T14:33:57.406685+0000","last_active":"2023-01-15T14:33:57.406685+0000","last_peered":"2023-01-15T14:33:57.406685+0000","last_clean":"2023-01-15T14:33:57.406685+0000","last_became_active":"2023-01-15T14:33:55.671990+0000","last_became_peered":"2023-01-15T14:33:55.671990+0000","last_unstale":"2023-01-15T14:33:57.406685+0000","last_undegraded":"2023-01-15T14:33:57.406685+0000","last_fullsized":"2023-01-15T14:33:57.406685+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.49887967,"stat_sum":{"num_bytes":1210137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":5,"num_read_kb":3549,"num_write":5,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.58","version":"323'320","reported_seq":1480,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.037188+0000","last_change":"2023-01-15T14:33:57.004059+0000","last_active":"2023-01-15T14:34:02.037188+0000","last_peered":"2023-01-15T14:34:02.037188+0000","last_clean":"2023-01-15T14:34:02.037188+0000","last_became_active":"2023-01-15T14:33:40.127669+0000","last_became_peered":"2023-01-15T14:33:40.127669+0000","last_unstale":"2023-01-15T14:34:02.037188+0000","last_undegraded":"2023-01-15T14:34:02.037188+0000","last_fullsized":"2023-01-15T14:34:02.037188+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.103642679,"stat_sum":{"num_bytes":4332079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.57","version":"351'98","reported_seq":92,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.346263+0000","last_change":"2023-01-15T14:33:57.890625+0000","last_active":"2023-01-15T14:34:00.346263+0000","last_peered":"2023-01-15T14:34:00.346263+0000","last_clean":"2023-01-15T14:34:00.346263+0000","last_became_active":"2023-01-15T14:33:36.994447+0000","last_became_peered":"2023-01-15T14:33:36.994447+0000","last_unstale":"2023-01-15T14:34:00.346263+0000","last_undegraded":"2023-01-15T14:34:00.346263+0000","last_fullsized":"2023-01-15T14:34:00.346263+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.969000109,"stat_sum":{"num_bytes":1512663,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3025326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.56","version":"320'127","reported_seq":630,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.012441+0000","last_change":"2023-01-15T14:34:00.541392+0000","last_active":"2023-01-15T14:34:02.012441+0000","last_peered":"2023-01-15T14:34:02.012441+0000","last_clean":"2023-01-15T14:34:02.012441+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:34:02.012441+0000","last_undegraded":"2023-01-15T14:34:02.012441+0000","last_fullsized":"2023-01-15T14:34:02.012441+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.30190621200000001,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.55","version":"110'55","reported_seq":1005,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.315559+0000","last_change":"2023-01-15T14:33:57.315559+0000","last_active":"2023-01-15T14:33:57.315559+0000","last_peered":"2023-01-15T14:33:57.315559+0000","last_clean":"2023-01-15T14:33:57.315559+0000","last_became_active":"2023-01-15T14:33:52.510429+0000","last_became_peered":"2023-01-15T14:33:52.510429+0000","last_unstale":"2023-01-15T14:33:57.315559+0000","last_undegraded":"2023-01-15T14:33:57.315559+0000","last_fullsized":"2023-01-15T14:33:57.315559+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.40629608699999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1128,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.138719+0000","last_change":"2023-01-15T14:33:57.109553+0000","last_active":"2023-01-15T14:34:00.138719+0000","last_peered":"2023-01-15T14:34:00.138719+0000","last_clean":"2023-01-15T14:34:00.138719+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:34:00.138719+0000","last_undegraded":"2023-01-15T14:34:00.138719+0000","last_fullsized":"2023-01-15T14:34:00.138719+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:33:45.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:45.583005+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.201302861,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.53","version":"352'397","reported_seq":1403,"reported_epoch":353,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:01.932224+0000","last_change":"2023-01-15T14:33:58.206587+0000","last_active":"2023-01-15T14:34:01.932224+0000","last_peered":"2023-01-15T14:34:01.932224+0000","last_clean":"2023-01-15T14:33:55.165228+0000","last_became_active":"2023-01-15T14:33:58.106461+0000","last_became_peered":"2023-01-15T14:33:58.106461+0000","last_unstale":"2023-01-15T14:34:01.932224+0000","last_undegraded":"2023-01-15T14:33:58.004141+0000","last_fullsized":"2023-01-15T14:33:58.002666+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:44.609","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.092650028999999995,"stat_sum":{"num_bytes":26431669,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":18,"num_read_kb":17078,"num_write":19,"num_write_kb":11543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":6},{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":730,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.010640+0000","last_change":"2023-01-15T14:33:56.903307+0000","last_active":"2023-01-15T14:34:02.010640+0000","last_peered":"2023-01-15T14:34:02.010640+0000","last_clean":"2023-01-15T14:34:02.010640+0000","last_became_active":"2023-01-15T14:33:55.472400+0000","last_became_peered":"2023-01-15T14:33:55.472400+0000","last_unstale":"2023-01-15T14:34:02.010640+0000","last_undegraded":"2023-01-15T14:34:02.010640+0000","last_fullsized":"2023-01-15T14:34:02.010640+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.001336634,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.51","version":"348'316","reported_seq":1519,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:01.933742+0000","last_change":"2023-01-15T14:33:57.893317+0000","last_active":"2023-01-15T14:34:01.933742+0000","last_peered":"2023-01-15T14:34:01.933742+0000","last_clean":"2023-01-15T14:34:01.933742+0000","last_became_active":"2023-01-15T14:33:26.107940+0000","last_became_peered":"2023-01-15T14:33:26.107940+0000","last_unstale":"2023-01-15T14:34:01.933742+0000","last_undegraded":"2023-01-15T14:34:01.933742+0000","last_fullsized":"2023-01-15T14:34:01.933742+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:36.458","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.79495623100000001,"stat_sum":{"num_bytes":7496813,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.50","version":"337'420","reported_seq":1582,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.563859+0000","last_change":"2023-01-15T14:34:02.564430+0000","last_active":"2023-01-15T14:34:02.563859+0000","last_peered":"2023-01-15T14:34:02.563859+0000","last_clean":"2023-01-15T14:34:02.563859+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:34:02.563859+0000","last_undegraded":"2023-01-15T14:34:02.563859+0000","last_fullsized":"2023-01-15T14:34:02.563859+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.0084659680000000008,"stat_sum":{"num_bytes":9681416,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":52505,"num_write":49,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4f","version":"155'145","reported_seq":822,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.353637+0000","last_change":"2023-01-15T14:33:57.353637+0000","last_active":"2023-01-15T14:33:57.353637+0000","last_peered":"2023-01-15T14:33:57.353637+0000","last_clean":"2023-01-15T14:33:57.353637+0000","last_became_active":"2023-01-15T14:33:25.129936+0000","last_became_peered":"2023-01-15T14:33:25.129936+0000","last_unstale":"2023-01-15T14:33:57.353637+0000","last_undegraded":"2023-01-15T14:33:57.353637+0000","last_fullsized":"2023-01-15T14:33:57.353637+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44322561900000002,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":96,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:33:59.939397+0000","last_change":"2023-01-15T14:33:56.915710+0000","last_active":"2023-01-15T14:33:59.939397+0000","last_peered":"2023-01-15T14:33:59.939397+0000","last_clean":"2023-01-15T14:33:59.939397+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:33:59.939397+0000","last_undegraded":"2023-01-15T14:33:59.939397+0000","last_fullsized":"2023-01-15T14:33:59.939397+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.0080109380000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4d","version":"352'258","reported_seq":1341,"reported_epoch":352,"state":"active+recovering+remapped","last_fresh":"2023-01-15T14:34:01.520220+0000","last_change":"2023-01-15T14:34:00.648841+0000","last_active":"2023-01-15T14:34:01.520220+0000","last_peered":"2023-01-15T14:34:01.520220+0000","last_clean":"2023-01-15T14:33:52.916562+0000","last_became_active":"2023-01-15T14:33:58.288904+0000","last_became_peered":"2023-01-15T14:33:58.288904+0000","last_unstale":"2023-01-15T14:34:01.520220+0000","last_undegraded":"2023-01-15T14:34:01.520220+0000","last_fullsized":"2023-01-15T14:34:01.520220+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.104242925,"stat_sum":{"num_bytes":12124166,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":15961,"num_write":18,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":14367689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[0,3,4],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":-1}],"blocked_by":[],"up_primary":7,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.4c","version":"309'71","reported_seq":107,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.011220+0000","last_change":"2023-01-15T14:33:57.413785+0000","last_active":"2023-01-15T14:34:02.011220+0000","last_peered":"2023-01-15T14:34:02.011220+0000","last_clean":"2023-01-15T14:34:02.011220+0000","last_became_active":"2023-01-15T14:33:52.311859+0000","last_became_peered":"2023-01-15T14:33:52.311859+0000","last_unstale":"2023-01-15T14:34:02.011220+0000","last_undegraded":"2023-01-15T14:34:02.011220+0000","last_fullsized":"2023-01-15T14:34:02.011220+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.40874315500000002,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":753,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.033456+0000","last_change":"2023-01-15T14:33:57.627070+0000","last_active":"2023-01-15T14:34:02.033456+0000","last_peered":"2023-01-15T14:34:02.033456+0000","last_clean":"2023-01-15T14:34:02.033456+0000","last_became_active":"2023-01-15T14:33:52.406605+0000","last_became_peered":"2023-01-15T14:33:52.406605+0000","last_unstale":"2023-01-15T14:34:02.033456+0000","last_undegraded":"2023-01-15T14:34:02.033456+0000","last_fullsized":"2023-01-15T14:34:02.033456+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:42.594","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.515074951,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4a","version":"348'371","reported_seq":1478,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.010985+0000","last_change":"2023-01-15T14:33:58.018192+0000","last_active":"2023-01-15T14:34:02.010985+0000","last_peered":"2023-01-15T14:34:02.010985+0000","last_clean":"2023-01-15T14:34:02.010985+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:34:02.010985+0000","last_undegraded":"2023-01-15T14:34:02.010985+0000","last_fullsized":"2023-01-15T14:34:02.010985+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:33.328","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10440110800000001,"stat_sum":{"num_bytes":12470870,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":63,"num_read_kb":67536,"num_write":73,"num_write_kb":23626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":935,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.110382+0000","last_change":"2023-01-15T14:34:00.046526+0000","last_active":"2023-01-15T14:34:02.110382+0000","last_peered":"2023-01-15T14:34:02.110382+0000","last_clean":"2023-01-15T14:34:02.110382+0000","last_became_active":"2023-01-15T14:34:00.043221+0000","last_became_peered":"2023-01-15T14:34:00.043221+0000","last_unstale":"2023-01-15T14:34:02.110382+0000","last_undegraded":"2023-01-15T14:34:02.110382+0000","last_fullsized":"2023-01-15T14:34:02.110382+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.00195963,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.48","version":"179'204","reported_seq":1353,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.356590+0000","last_change":"2023-01-15T14:33:57.356590+0000","last_active":"2023-01-15T14:33:57.356590+0000","last_peered":"2023-01-15T14:33:57.356590+0000","last_clean":"2023-01-15T14:33:57.356590+0000","last_became_active":"2023-01-15T14:33:55.375518+0000","last_became_peered":"2023-01-15T14:33:55.375518+0000","last_unstale":"2023-01-15T14:33:57.356590+0000","last_undegraded":"2023-01-15T14:33:57.356590+0000","last_fullsized":"2023-01-15T14:33:57.356590+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:43.662","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.34933974800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1113,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.300260+0000","last_change":"2023-01-15T14:33:57.300260+0000","last_active":"2023-01-15T14:33:57.300260+0000","last_peered":"2023-01-15T14:33:57.300260+0000","last_clean":"2023-01-15T14:33:57.300260+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:33:57.300260+0000","last_undegraded":"2023-01-15T14:33:57.300260+0000","last_fullsized":"2023-01-15T14:33:57.300260+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:03.200","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.39166528900000003,"stat_sum":{"num_bytes":496762,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":5,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":1029,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.113540+0000","last_change":"2023-01-15T14:33:57.829426+0000","last_active":"2023-01-15T14:34:02.113540+0000","last_peered":"2023-01-15T14:34:02.113540+0000","last_clean":"2023-01-15T14:34:02.113540+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:34:02.113540+0000","last_undegraded":"2023-01-15T14:34:02.113540+0000","last_fullsized":"2023-01-15T14:34:02.113540+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:55.245","scrub_duration":0.30448409700000001,"objects_trimmed":0,"snaptrim_duration":0.714760332,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":913,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:33:59.940479+0000","last_change":"2023-01-15T14:33:56.909121+0000","last_active":"2023-01-15T14:33:59.940479+0000","last_peered":"2023-01-15T14:33:59.940479+0000","last_clean":"2023-01-15T14:33:59.940479+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:33:59.940479+0000","last_undegraded":"2023-01-15T14:33:59.940479+0000","last_fullsized":"2023-01-15T14:33:59.940479+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045307990000000003,"stat_sum":{"num_bytes":2635976,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":19927,"num_write":15,"num_write_kb":6659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4745860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1268,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.298701+0000","last_change":"2023-01-15T14:33:57.298701+0000","last_active":"2023-01-15T14:33:57.298701+0000","last_peered":"2023-01-15T14:33:57.298701+0000","last_clean":"2023-01-15T14:33:57.298701+0000","last_became_active":"2023-01-15T14:33:55.678019+0000","last_became_peered":"2023-01-15T14:33:55.678019+0000","last_unstale":"2023-01-15T14:33:57.298701+0000","last_undegraded":"2023-01-15T14:33:57.298701+0000","last_fullsized":"2023-01-15T14:33:57.298701+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:27.120","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.390225449,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.43","version":"297'314","reported_seq":1497,"reported_epoch":352,"state":"peering","last_fresh":"2023-01-15T14:34:00.946458+0000","last_change":"2023-01-15T14:34:00.946721+0000","last_active":"2023-01-15T14:34:00.946417+0000","last_peered":"2023-01-15T14:34:00.344013+0000","last_clean":"2023-01-15T14:33:55.176514+0000","last_became_active":"2023-01-15T14:33:58.185057+0000","last_became_peered":"2023-01-15T14:33:58.185057+0000","last_unstale":"2023-01-15T14:34:00.946458+0000","last_undegraded":"2023-01-15T14:34:00.946458+0000","last_fullsized":"2023-01-15T14:34:00.946458+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":262,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.101016546,"stat_sum":{"num_bytes":3395080,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":16585,"num_write":5,"num_write_kb":1351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3395080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,7],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":1029,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.621398+0000","last_change":"2023-01-15T14:33:57.621398+0000","last_active":"2023-01-15T14:33:57.621398+0000","last_peered":"2023-01-15T14:33:57.621398+0000","last_clean":"2023-01-15T14:33:57.621398+0000","last_became_active":"2023-01-15T14:33:55.376782+0000","last_became_peered":"2023-01-15T14:33:55.376782+0000","last_unstale":"2023-01-15T14:33:57.621398+0000","last_undegraded":"2023-01-15T14:33:57.621398+0000","last_fullsized":"2023-01-15T14:33:57.621398+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.51101871399999999,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1109,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.908053+0000","last_change":"2023-01-15T14:33:57.302504+0000","last_active":"2023-01-15T14:33:57.908053+0000","last_peered":"2023-01-15T14:33:57.908053+0000","last_clean":"2023-01-15T14:33:57.908053+0000","last_became_active":"2023-01-15T14:33:55.771969+0000","last_became_peered":"2023-01-15T14:33:55.771969+0000","last_unstale":"2023-01-15T14:33:57.908053+0000","last_undegraded":"2023-01-15T14:33:57.908053+0000","last_fullsized":"2023-01-15T14:33:57.908053+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.29405439700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.40","version":"335'307","reported_seq":1395,"reported_epoch":353,"state":"active+recovery_wait+degraded","last_fresh":"2023-01-15T14:34:02.138205+0000","last_change":"2023-01-15T14:33:58.118750+0000","last_active":"2023-01-15T14:34:02.138205+0000","last_peered":"2023-01-15T14:34:02.138205+0000","last_clean":"2023-01-15T14:33:43.952880+0000","last_became_active":"2023-01-15T14:33:58.117085+0000","last_became_peered":"2023-01-15T14:33:58.117085+0000","last_unstale":"2023-01-15T14:34:02.138205+0000","last_undegraded":"2023-01-15T14:33:57.914305+0000","last_fullsized":"2023-01-15T14:34:02.138205+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.704289622,"stat_sum":{"num_bytes":12753904,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":35541,"num_write":28,"num_write_kb":11001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":16413587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"321'233","reported_seq":1369,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.012304+0000","last_change":"2023-01-15T14:33:58.333573+0000","last_active":"2023-01-15T14:34:02.012304+0000","last_peered":"2023-01-15T14:34:02.012304+0000","last_clean":"2023-01-15T14:34:02.012304+0000","last_became_active":"2023-01-15T14:33:58.217783+0000","last_became_peered":"2023-01-15T14:33:58.217783+0000","last_unstale":"2023-01-15T14:34:02.012304+0000","last_undegraded":"2023-01-15T14:34:02.012304+0000","last_fullsized":"2023-01-15T14:34:02.012304+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:28.180","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.014066018,"stat_sum":{"num_bytes":4764997,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6605283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3e","version":"294'63","reported_seq":835,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.033562+0000","last_change":"2023-01-15T14:33:57.728361+0000","last_active":"2023-01-15T14:34:02.033562+0000","last_peered":"2023-01-15T14:34:02.033562+0000","last_clean":"2023-01-15T14:34:02.033562+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:34:02.033562+0000","last_undegraded":"2023-01-15T14:34:02.033562+0000","last_fullsized":"2023-01-15T14:34:02.033562+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.61381065800000001,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3d","version":"335'232","reported_seq":1345,"reported_epoch":353,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:34:02.562625+0000","last_change":"2023-01-15T14:34:00.547509+0000","last_active":"2023-01-15T14:34:02.562625+0000","last_peered":"2023-01-15T14:34:02.562625+0000","last_clean":"2023-01-15T14:33:50.059397+0000","last_became_active":"2023-01-15T14:33:53.126274+0000","last_became_peered":"2023-01-15T14:33:53.126274+0000","last_unstale":"2023-01-15T14:34:02.562625+0000","last_undegraded":"2023-01-15T14:34:02.562625+0000","last_fullsized":"2023-01-15T14:33:52.926980+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.15835743399999999,"stat_sum":{"num_bytes":8050279,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":13797,"num_write":16,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":2823642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[1,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2","objects":1},{"shards":"1,2,4","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3c","version":"141'46","reported_seq":345,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.904495+0000","last_change":"2023-01-15T14:33:56.912541+0000","last_active":"2023-01-15T14:33:57.904495+0000","last_peered":"2023-01-15T14:33:57.904495+0000","last_clean":"2023-01-15T14:33:57.904495+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:33:57.904495+0000","last_undegraded":"2023-01-15T14:33:57.904495+0000","last_fullsized":"2023-01-15T14:33:57.904495+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.470","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.0047236029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":25,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.111044+0000","last_change":"2023-01-15T14:33:58.125276+0000","last_active":"2023-01-15T14:34:02.111044+0000","last_peered":"2023-01-15T14:34:02.111044+0000","last_clean":"2023-01-15T14:34:02.111044+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:34:02.111044+0000","last_undegraded":"2023-01-15T14:34:02.111044+0000","last_fullsized":"2023-01-15T14:34:02.111044+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.21169681200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1347,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.040840+0000","last_change":"2023-01-15T14:33:57.286278+0000","last_active":"2023-01-15T14:34:00.040840+0000","last_peered":"2023-01-15T14:34:00.040840+0000","last_clean":"2023-01-15T14:34:00.040840+0000","last_became_active":"2023-01-15T14:33:55.474944+0000","last_became_peered":"2023-01-15T14:33:55.474944+0000","last_unstale":"2023-01-15T14:34:00.040840+0000","last_undegraded":"2023-01-15T14:34:00.040840+0000","last_fullsized":"2023-01-15T14:34:00.040840+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.37705996200000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.39","version":"334'351","reported_seq":1400,"reported_epoch":353,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:34:02.756481+0000","last_change":"2023-01-15T14:34:02.756481+0000","last_active":"2023-01-15T14:34:02.756481+0000","last_peered":"2023-01-15T14:34:02.756481+0000","last_clean":"2023-01-15T14:33:50.016390+0000","last_became_active":"2023-01-15T14:33:53.436135+0000","last_became_peered":"2023-01-15T14:33:53.436135+0000","last_unstale":"2023-01-15T14:34:02.756481+0000","last_undegraded":"2023-01-15T14:34:02.756481+0000","last_fullsized":"2023-01-15T14:33:53.015555+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:21.301","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.0051603159999999999,"stat_sum":{"num_bytes":24023444,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":66941,"num_write":7,"num_write_kb":7292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,2],"avail_no_missing":["7","2","4"],"object_location_counts":[{"shards":"2,4,7","objects":9}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"348'311","reported_seq":1523,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.959969+0000","last_change":"2023-01-15T14:33:58.009343+0000","last_active":"2023-01-15T14:33:57.959969+0000","last_peered":"2023-01-15T14:33:57.959969+0000","last_clean":"2023-01-15T14:33:57.959969+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:33:57.959969+0000","last_undegraded":"2023-01-15T14:33:57.959969+0000","last_fullsized":"2023-01-15T14:33:57.959969+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.10683759900000001,"stat_sum":{"num_bytes":7580089,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":42847,"num_write":35,"num_write_kb":12593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":411,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.113909+0000","last_change":"2023-01-15T14:33:58.124010+0000","last_active":"2023-01-15T14:34:02.113909+0000","last_peered":"2023-01-15T14:34:02.113909+0000","last_clean":"2023-01-15T14:34:02.113909+0000","last_became_active":"2023-01-15T14:33:52.412128+0000","last_became_peered":"2023-01-15T14:33:52.412128+0000","last_unstale":"2023-01-15T14:34:02.113909+0000","last_undegraded":"2023-01-15T14:34:02.113909+0000","last_fullsized":"2023-01-15T14:34:02.113909+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.207923046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":563,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.034083+0000","last_change":"2023-01-15T14:33:57.332104+0000","last_active":"2023-01-15T14:34:02.034083+0000","last_peered":"2023-01-15T14:34:02.034083+0000","last_clean":"2023-01-15T14:34:02.034083+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:34:02.034083+0000","last_undegraded":"2023-01-15T14:34:02.034083+0000","last_fullsized":"2023-01-15T14:34:02.034083+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.31780763299999998,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":36,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:33:59.940349+0000","last_change":"2023-01-15T14:33:56.908180+0000","last_active":"2023-01-15T14:33:59.940349+0000","last_peered":"2023-01-15T14:33:59.940349+0000","last_clean":"2023-01-15T14:33:59.940349+0000","last_became_active":"2023-01-15T14:33:52.612576+0000","last_became_peered":"2023-01-15T14:33:52.612576+0000","last_unstale":"2023-01-15T14:33:59.940349+0000","last_undegraded":"2023-01-15T14:33:59.940349+0000","last_fullsized":"2023-01-15T14:33:59.940349+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003836173,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.34","version":"169'138","reported_seq":1152,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.301685+0000","last_change":"2023-01-15T14:33:57.301685+0000","last_active":"2023-01-15T14:33:57.301685+0000","last_peered":"2023-01-15T14:33:57.301685+0000","last_clean":"2023-01-15T14:33:57.301685+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:33:57.301685+0000","last_undegraded":"2023-01-15T14:33:57.301685+0000","last_fullsized":"2023-01-15T14:33:57.301685+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:33:59.872","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.39300976100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1228,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.904458+0000","last_change":"2023-01-15T14:33:57.354727+0000","last_active":"2023-01-15T14:33:57.904458+0000","last_peered":"2023-01-15T14:33:57.904458+0000","last_clean":"2023-01-15T14:33:57.904458+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:33:57.904458+0000","last_undegraded":"2023-01-15T14:33:57.904458+0000","last_fullsized":"2023-01-15T14:33:57.904458+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.442392491,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.32","version":"334'163","reported_seq":851,"reported_epoch":353,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:34:02.343298+0000","last_change":"2023-01-15T14:34:02.343908+0000","last_active":"2023-01-15T14:34:02.343298+0000","last_peered":"2023-01-15T14:34:02.343298+0000","last_clean":"2023-01-15T14:33:50.154755+0000","last_became_active":"2023-01-15T14:33:53.228307+0000","last_became_peered":"2023-01-15T14:33:53.228307+0000","last_unstale":"2023-01-15T14:34:02.343298+0000","last_undegraded":"2023-01-15T14:34:02.343298+0000","last_fullsized":"2023-01-15T14:33:52.927836+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.80293351999999996,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7048,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5327885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[1,3],"avail_no_missing":["1","0","2","3"],"object_location_counts":[{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"348'374","reported_seq":1384,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.011506+0000","last_change":"2023-01-15T14:33:58.021945+0000","last_active":"2023-01-15T14:34:02.011506+0000","last_peered":"2023-01-15T14:34:02.011506+0000","last_clean":"2023-01-15T14:34:02.011506+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:34:02.011506+0000","last_undegraded":"2023-01-15T14:34:02.011506+0000","last_fullsized":"2023-01-15T14:34:02.011506+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:29.359","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.0064123280000000001,"stat_sum":{"num_bytes":8364486,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":61489,"num_write":17,"num_write_kb":9266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.12","version":"335'184","reported_seq":874,"reported_epoch":352,"state":"peering","last_fresh":"2023-01-15T14:34:00.942409+0000","last_change":"2023-01-15T14:34:00.942769+0000","last_active":"2023-01-15T14:34:00.942382+0000","last_peered":"2023-01-15T14:34:00.523679+0000","last_clean":"2023-01-15T14:33:44.161296+0000","last_became_active":"2023-01-15T14:33:58.015442+0000","last_became_peered":"2023-01-15T14:33:58.015442+0000","last_unstale":"2023-01-15T14:34:00.942409+0000","last_undegraded":"2023-01-15T14:34:00.942409+0000","last_fullsized":"2023-01-15T14:34:00.942409+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":184,"log_dups_size":0,"ondisk_log_size":184,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.24331072000000001,"stat_sum":{"num_bytes":7581497,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":15,"num_write_kb":5299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3,7],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.11","version":"334'306","reported_seq":1490,"reported_epoch":353,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:02.111627+0000","last_change":"2023-01-15T14:33:53.763768+0000","last_active":"2023-01-15T14:34:02.111627+0000","last_peered":"2023-01-15T14:34:02.111627+0000","last_clean":"2023-01-15T14:33:50.016827+0000","last_became_active":"2023-01-15T14:33:53.659621+0000","last_became_peered":"2023-01-15T14:33:53.659621+0000","last_unstale":"2023-01-15T14:34:02.111627+0000","last_undegraded":"2023-01-15T14:33:53.019571+0000","last_fullsized":"2023-01-15T14:33:53.017412+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.090247347000000006,"stat_sum":{"num_bytes":10718732,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":48289,"num_write":26,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6655737,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":6},{"shards":"0,7","objects":-2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"308'406","reported_seq":1405,"reported_epoch":353,"state":"active+recovery_wait+remapped","last_fresh":"2023-01-15T14:34:02.032550+0000","last_change":"2023-01-15T14:33:53.820604+0000","last_active":"2023-01-15T14:34:02.032550+0000","last_peered":"2023-01-15T14:34:02.032550+0000","last_clean":"2023-01-15T14:33:17.174184+0000","last_became_active":"2023-01-15T14:33:53.718934+0000","last_became_peered":"2023-01-15T14:33:53.718934+0000","last_unstale":"2023-01-15T14:34:02.032550+0000","last_undegraded":"2023-01-15T14:34:02.032550+0000","last_fullsized":"2023-01-15T14:34:02.032550+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.84251846600000002,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":41964,"num_write":46,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6670228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[1,3,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"155'145","reported_seq":1236,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.011534+0000","last_change":"2023-01-15T14:34:00.245548+0000","last_active":"2023-01-15T14:34:02.011534+0000","last_peered":"2023-01-15T14:34:02.011534+0000","last_clean":"2023-01-15T14:34:02.011534+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:34:02.011534+0000","last_undegraded":"2023-01-15T14:34:02.011534+0000","last_fullsized":"2023-01-15T14:34:02.011534+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.0023104509999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.e","version":"139'38","reported_seq":25,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.012012+0000","last_change":"2023-01-15T14:33:58.422638+0000","last_active":"2023-01-15T14:34:02.012012+0000","last_peered":"2023-01-15T14:34:02.012012+0000","last_clean":"2023-01-15T14:34:02.012012+0000","last_became_active":"2023-01-15T14:33:57.310759+0000","last_became_peered":"2023-01-15T14:33:57.310759+0000","last_unstale":"2023-01-15T14:34:02.012012+0000","last_undegraded":"2023-01-15T14:34:02.012012+0000","last_fullsized":"2023-01-15T14:34:02.012012+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.40586427000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.d","version":"296'232","reported_seq":1299,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.178205+0000","last_change":"2023-01-15T14:33:57.114569+0000","last_active":"2023-01-15T14:34:02.178205+0000","last_peered":"2023-01-15T14:34:02.178205+0000","last_clean":"2023-01-15T14:34:02.178205+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:34:02.178205+0000","last_undegraded":"2023-01-15T14:34:02.178205+0000","last_fullsized":"2023-01-15T14:34:02.178205+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.20767458899999999,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.c","version":"348'95","reported_seq":1088,"reported_epoch":348,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:33:57.826220+0000","last_change":"2023-01-15T14:33:57.622687+0000","last_active":"2023-01-15T14:33:57.826220+0000","last_peered":"2023-01-15T14:33:57.826220+0000","last_clean":"2023-01-15T14:33:57.826220+0000","last_became_active":"2023-01-15T14:33:52.210127+0000","last_became_peered":"2023-01-15T14:33:52.210127+0000","last_unstale":"2023-01-15T14:33:57.826220+0000","last_undegraded":"2023-01-15T14:33:57.826220+0000","last_fullsized":"2023-01-15T14:33:57.826220+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.38959517900000001,"stat_sum":{"num_bytes":2221858,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":10493,"num_write":6,"num_write_kb":686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8023674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":999,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.618325+0000","last_change":"2023-01-15T14:33:57.618325+0000","last_active":"2023-01-15T14:33:57.618325+0000","last_peered":"2023-01-15T14:33:57.618325+0000","last_clean":"2023-01-15T14:33:57.618325+0000","last_became_active":"2023-01-15T14:33:52.425622+0000","last_became_peered":"2023-01-15T14:33:52.425622+0000","last_unstale":"2023-01-15T14:33:57.618325+0000","last_undegraded":"2023-01-15T14:33:57.618325+0000","last_fullsized":"2023-01-15T14:33:57.618325+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.60540339200000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1322,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.297022+0000","last_change":"2023-01-15T14:33:57.297022+0000","last_active":"2023-01-15T14:33:57.297022+0000","last_peered":"2023-01-15T14:33:57.297022+0000","last_clean":"2023-01-15T14:33:57.297022+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:33:57.297022+0000","last_undegraded":"2023-01-15T14:33:57.297022+0000","last_fullsized":"2023-01-15T14:33:57.297022+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:48.664","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.38900274200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1107,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.304492+0000","last_change":"2023-01-15T14:33:57.304492+0000","last_active":"2023-01-15T14:33:57.304492+0000","last_peered":"2023-01-15T14:33:57.304492+0000","last_clean":"2023-01-15T14:33:57.304492+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:33:57.304492+0000","last_undegraded":"2023-01-15T14:33:57.304492+0000","last_fullsized":"2023-01-15T14:33:57.304492+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:38.874","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.29516895799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1304,"reported_epoch":349,"state":"activating+undersized+remapped","last_fresh":"2023-01-15T14:33:57.903125+0000","last_change":"2023-01-15T14:33:57.903380+0000","last_active":"2023-01-15T14:33:56.491492+0000","last_peered":"2023-01-15T14:33:55.173516+0000","last_clean":"2023-01-15T14:33:55.173516+0000","last_became_active":"2023-01-15T14:32:22.142296+0000","last_became_peered":"2023-01-15T14:32:22.142296+0000","last_unstale":"2023-01-15T14:33:57.903125+0000","last_undegraded":"2023-01-15T14:33:57.903125+0000","last_fullsized":"2023-01-15T14:33:57.903125+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.20813032000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1],"avail_no_missing":["3","1","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":517,"reported_epoch":352,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:01.146014+0000","last_change":"2023-01-15T14:33:58.506892+0000","last_active":"2023-01-15T14:34:01.146014+0000","last_peered":"2023-01-15T14:34:01.146014+0000","last_clean":"2023-01-15T14:33:50.143204+0000","last_became_active":"2023-01-15T14:33:58.306184+0000","last_became_peered":"2023-01-15T14:33:58.306184+0000","last_unstale":"2023-01-15T14:34:01.146014+0000","last_undegraded":"2023-01-15T14:33:58.003421+0000","last_fullsized":"2023-01-15T14:33:58.001929+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[1,0],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":352,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.946894+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:34:00.946894+0000","last_peered":"2023-01-15T14:34:00.946894+0000","last_clean":"2023-01-15T14:34:00.946894+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:34:00.946894+0000","last_undegraded":"2023-01-15T14:34:00.946894+0000","last_fullsized":"2023-01-15T14:34:00.946894+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":995,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.297870+0000","last_change":"2023-01-15T14:33:57.297870+0000","last_active":"2023-01-15T14:33:57.297870+0000","last_peered":"2023-01-15T14:33:57.297870+0000","last_clean":"2023-01-15T14:33:57.297870+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:33:57.297870+0000","last_undegraded":"2023-01-15T14:33:57.297870+0000","last_fullsized":"2023-01-15T14:33:57.297870+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.38976201700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":352,"state":"active+clean","last_fresh":"2023-01-15T14:34:01.048664+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:34:01.048664+0000","last_peered":"2023-01-15T14:34:01.048664+0000","last_clean":"2023-01-15T14:34:01.048664+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:34:01.048664+0000","last_undegraded":"2023-01-15T14:34:01.048664+0000","last_fullsized":"2023-01-15T14:34:01.048664+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.964","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":711,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:33:59.938743+0000","last_change":"2023-01-15T14:33:57.185185+0000","last_active":"2023-01-15T14:33:59.938743+0000","last_peered":"2023-01-15T14:33:59.938743+0000","last_clean":"2023-01-15T14:33:59.938743+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:33:59.938743+0000","last_undegraded":"2023-01-15T14:33:59.938743+0000","last_fullsized":"2023-01-15T14:33:59.938743+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.280577087,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":150,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.010096+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:33:57.010096+0000","last_peered":"2023-01-15T14:33:57.010096+0000","last_clean":"2023-01-15T14:33:57.010096+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:33:57.010096+0000","last_undegraded":"2023-01-15T14:33:57.010096+0000","last_fullsized":"2023-01-15T14:33:57.010096+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1323,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.034594+0000","last_change":"2023-01-15T14:33:57.830460+0000","last_active":"2023-01-15T14:34:02.034594+0000","last_peered":"2023-01-15T14:34:02.034594+0000","last_clean":"2023-01-15T14:34:02.034594+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:34:02.034594+0000","last_undegraded":"2023-01-15T14:34:02.034594+0000","last_fullsized":"2023-01-15T14:34:02.034594+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.52931870400000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":221,"reported_epoch":352,"state":"active+clean","last_fresh":"2023-01-15T14:34:01.148842+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:34:01.148842+0000","last_peered":"2023-01-15T14:34:01.148842+0000","last_clean":"2023-01-15T14:34:01.148842+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:34:01.148842+0000","last_undegraded":"2023-01-15T14:34:01.148842+0000","last_fullsized":"2023-01-15T14:34:01.148842+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:25.583","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"344'317","reported_seq":1390,"reported_epoch":347,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:56.545414+0000","last_change":"2023-01-15T14:33:56.033295+0000","last_active":"2023-01-15T14:33:56.545414+0000","last_peered":"2023-01-15T14:33:56.545414+0000","last_clean":"2023-01-15T14:33:50.176701+0000","last_became_active":"2023-01-15T14:33:53.226305+0000","last_became_peered":"2023-01-15T14:33:53.226305+0000","last_unstale":"2023-01-15T14:33:56.545414+0000","last_undegraded":"2023-01-15T14:33:52.915843+0000","last_fullsized":"2023-01-15T14:33:52.914697+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":301,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:34.328","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.69416308599999998,"stat_sum":{"num_bytes":9428925,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":13874,"num_write":2,"num_write_kb":6435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[0,1],"avail_no_missing":["0","1","3"],"object_location_counts":[{"shards":"0,1","objects":2},{"shards":"0,1,3","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.5","version":"335'158","reported_seq":1165,"reported_epoch":349,"state":"activating+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:57.908195+0000","last_change":"2023-01-15T14:33:58.109966+0000","last_active":"2023-01-15T14:33:56.088835+0000","last_peered":"2023-01-15T14:33:55.073530+0000","last_clean":"2023-01-15T14:33:55.073530+0000","last_became_active":"2023-01-15T14:32:21.948278+0000","last_became_peered":"2023-01-15T14:32:21.948278+0000","last_unstale":"2023-01-15T14:33:57.908195+0000","last_undegraded":"2023-01-15T14:33:57.908195+0000","last_fullsized":"2023-01-15T14:33:57.905675+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.71299288800000005,"stat_sum":{"num_bytes":11390101,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":26145,"num_write":9,"num_write_kb":4265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":5}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":211,"reported_epoch":352,"state":"active+clean","last_fresh":"2023-01-15T14:34:01.048022+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:34:01.048022+0000","last_peered":"2023-01-15T14:34:01.048022+0000","last_clean":"2023-01-15T14:34:01.048022+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:34:01.048022+0000","last_undegraded":"2023-01-15T14:34:01.048022+0000","last_fullsized":"2023-01-15T14:34:01.048022+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"335'308","reported_seq":1409,"reported_epoch":353,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:02.033049+0000","last_change":"2023-01-15T14:33:53.424640+0000","last_active":"2023-01-15T14:34:02.033049+0000","last_peered":"2023-01-15T14:34:02.033049+0000","last_clean":"2023-01-15T14:33:50.017033+0000","last_became_active":"2023-01-15T14:33:53.422817+0000","last_became_peered":"2023-01-15T14:33:53.422817+0000","last_unstale":"2023-01-15T14:34:02.033049+0000","last_undegraded":"2023-01-15T14:33:53.016002+0000","last_fullsized":"2023-01-15T14:33:53.014012+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.70443229200000002,"stat_sum":{"num_bytes":2954643,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":30684,"num_write":31,"num_write_kb":10131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7931053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2","objects":-1},{"shards":"1,2,4","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":400,"reported_epoch":352,"state":"active+clean","last_fresh":"2023-01-15T14:34:01.048366+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:34:01.048366+0000","last_peered":"2023-01-15T14:34:01.048366+0000","last_clean":"2023-01-15T14:34:01.048366+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:34:01.048366+0000","last_undegraded":"2023-01-15T14:34:01.048366+0000","last_fullsized":"2023-01-15T14:34:01.048366+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"349'141","reported_seq":1139,"reported_epoch":353,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2023-01-15T14:34:01.933247+0000","last_change":"2023-01-15T14:33:58.214638+0000","last_active":"2023-01-15T14:34:01.933247+0000","last_peered":"2023-01-15T14:34:01.933247+0000","last_clean":"2023-01-15T14:33:42.900201+0000","last_became_active":"2023-01-15T14:33:58.115294+0000","last_became_peered":"2023-01-15T14:33:58.115294+0000","last_unstale":"2023-01-15T14:34:01.933247+0000","last_undegraded":"2023-01-15T14:33:57.905179+0000","last_fullsized":"2023-01-15T14:34:01.933247+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.471195633,"stat_sum":{"num_bytes":9475691,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":13598,"num_write":13,"num_write_kb":8618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3546343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[1,3,0],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"0,1,3","objects":2},{"shards":"1,3,4","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":211,"reported_epoch":352,"state":"active+clean","last_fresh":"2023-01-15T14:34:01.050959+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:34:01.050959+0000","last_peered":"2023-01-15T14:34:01.050959+0000","last_clean":"2023-01-15T14:34:01.050959+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:34:01.050959+0000","last_undegraded":"2023-01-15T14:34:01.050959+0000","last_fullsized":"2023-01-15T14:34:01.050959+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:17.360","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":36,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:33:59.939264+0000","last_change":"2023-01-15T14:33:56.909628+0000","last_active":"2023-01-15T14:33:59.939264+0000","last_peered":"2023-01-15T14:33:59.939264+0000","last_clean":"2023-01-15T14:33:59.939264+0000","last_became_active":"2023-01-15T14:33:52.206701+0000","last_became_peered":"2023-01-15T14:33:52.206701+0000","last_unstale":"2023-01-15T14:33:59.939264+0000","last_undegraded":"2023-01-15T14:33:59.939264+0000","last_fullsized":"2023-01-15T14:33:59.939264+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.0022256720000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":16,"reported_epoch":352,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.948352+0000","last_change":"2023-01-15T14:33:57.412557+0000","last_active":"2023-01-15T14:34:00.948352+0000","last_peered":"2023-01-15T14:34:00.948352+0000","last_clean":"2023-01-15T14:34:00.948352+0000","last_became_active":"2023-01-15T14:33:57.311981+0000","last_became_peered":"2023-01-15T14:33:57.311981+0000","last_unstale":"2023-01-15T14:34:00.948352+0000","last_undegraded":"2023-01-15T14:34:00.948352+0000","last_fullsized":"2023-01-15T14:34:00.948352+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"348'158","reported_seq":1178,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.009873+0000","last_change":"2023-01-15T14:33:58.010967+0000","last_active":"2023-01-15T14:33:58.009873+0000","last_peered":"2023-01-15T14:33:58.009873+0000","last_clean":"2023-01-15T14:33:58.009873+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:33:58.009873+0000","last_undegraded":"2023-01-15T14:33:58.009873+0000","last_fullsized":"2023-01-15T14:33:58.009873+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:32:19.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:19.303243+0000","objects_scrubbed":1,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.0010524499999999999,"stat_sum":{"num_bytes":1226927,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":18,"num_read_kb":20384,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":25,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.177523+0000","last_change":"2023-01-15T14:33:58.327664+0000","last_active":"2023-01-15T14:34:02.177523+0000","last_peered":"2023-01-15T14:34:02.177523+0000","last_clean":"2023-01-15T14:34:02.177523+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:34:02.177523+0000","last_undegraded":"2023-01-15T14:34:02.177523+0000","last_fullsized":"2023-01-15T14:34:02.177523+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31327724499999998,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.16","version":"352'129","reported_seq":638,"reported_epoch":353,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:02.113333+0000","last_change":"2023-01-15T14:33:58.219965+0000","last_active":"2023-01-15T14:34:02.113333+0000","last_peered":"2023-01-15T14:34:02.113333+0000","last_clean":"2023-01-15T14:33:55.270604+0000","last_became_active":"2023-01-15T14:33:58.218043+0000","last_became_peered":"2023-01-15T14:33:58.218043+0000","last_unstale":"2023-01-15T14:34:02.113333+0000","last_undegraded":"2023-01-15T14:33:57.906873+0000","last_fullsized":"2023-01-15T14:33:57.906218+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:09.523","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.30046122199999997,"stat_sum":{"num_bytes":7378471,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2},{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":36,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:33:59.940022+0000","last_change":"2023-01-15T14:33:57.186529+0000","last_active":"2023-01-15T14:33:59.940022+0000","last_peered":"2023-01-15T14:33:59.940022+0000","last_clean":"2023-01-15T14:33:59.940022+0000","last_became_active":"2023-01-15T14:33:52.303197+0000","last_became_peered":"2023-01-15T14:33:52.303197+0000","last_unstale":"2023-01-15T14:33:59.940022+0000","last_undegraded":"2023-01-15T14:33:59.940022+0000","last_fullsized":"2023-01-15T14:33:59.940022+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.265135238,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.18","version":"340'324","reported_seq":1479,"reported_epoch":348,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:33:57.007858+0000","last_change":"2023-01-15T14:33:53.518579+0000","last_active":"2023-01-15T14:33:57.007858+0000","last_peered":"2023-01-15T14:33:57.007858+0000","last_clean":"2023-01-15T14:33:50.112982+0000","last_became_active":"2023-01-15T14:33:53.517870+0000","last_became_peered":"2023-01-15T14:33:53.517870+0000","last_unstale":"2023-01-15T14:33:57.007858+0000","last_undegraded":"2023-01-15T14:33:52.920700+0000","last_fullsized":"2023-01-15T14:33:52.920090+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:06.616","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.66838752300000004,"stat_sum":{"num_bytes":4332078,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":26,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":4},{"shards":"1,3","objects":-2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"}]},{"pgid":"3.19","version":"294'256","reported_seq":1322,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:01.933071+0000","last_change":"2023-01-15T14:33:57.626542+0000","last_active":"2023-01-15T14:34:01.933071+0000","last_peered":"2023-01-15T14:34:01.933071+0000","last_clean":"2023-01-15T14:34:01.933071+0000","last_became_active":"2023-01-15T14:33:35.259904+0000","last_became_peered":"2023-01-15T14:33:35.259904+0000","last_unstale":"2023-01-15T14:34:01.933071+0000","last_undegraded":"2023-01-15T14:34:01.933071+0000","last_fullsized":"2023-01-15T14:34:01.933071+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:01.296","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.52734117999999996,"stat_sum":{"num_bytes":2420274,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1a","version":"348'352","reported_seq":1450,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.010827+0000","last_change":"2023-01-15T14:33:58.011960+0000","last_active":"2023-01-15T14:33:58.010827+0000","last_peered":"2023-01-15T14:33:58.010827+0000","last_clean":"2023-01-15T14:33:58.010827+0000","last_became_active":"2023-01-15T14:33:35.063699+0000","last_became_peered":"2023-01-15T14:33:35.063699+0000","last_unstale":"2023-01-15T14:33:58.010827+0000","last_undegraded":"2023-01-15T14:33:58.010827+0000","last_fullsized":"2023-01-15T14:33:58.010827+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.0039007270000000001,"stat_sum":{"num_bytes":3052653,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":15,"num_read_kb":21426,"num_write":7,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1b","version":"348'99","reported_seq":60,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.040315+0000","last_change":"2023-01-15T14:33:58.271017+0000","last_active":"2023-01-15T14:34:00.040315+0000","last_peered":"2023-01-15T14:34:00.040315+0000","last_clean":"2023-01-15T14:34:00.040315+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:34:00.040315+0000","last_undegraded":"2023-01-15T14:34:00.040315+0000","last_fullsized":"2023-01-15T14:34:00.040315+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.26204159500000002,"stat_sum":{"num_bytes":4400087,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10511,"num_write":14,"num_write_kb":4701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9416119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":538,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-01-15T14:33:58.011212+0000","last_change":"2023-01-15T14:33:58.012512+0000","last_active":"2023-01-15T14:33:58.011212+0000","last_peered":"2023-01-15T14:33:58.011212+0000","last_clean":"2023-01-15T14:33:58.011212+0000","last_became_active":"2023-01-15T14:33:57.125876+0000","last_became_peered":"2023-01-15T14:33:57.125876+0000","last_unstale":"2023-01-15T14:33:58.011212+0000","last_undegraded":"2023-01-15T14:33:58.011212+0000","last_fullsized":"2023-01-15T14:33:58.011212+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.0028318649999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1263,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.405395+0000","last_change":"2023-01-15T14:33:57.405395+0000","last_active":"2023-01-15T14:33:57.405395+0000","last_peered":"2023-01-15T14:33:57.405395+0000","last_clean":"2023-01-15T14:33:57.405395+0000","last_became_active":"2023-01-15T14:33:35.362912+0000","last_became_peered":"2023-01-15T14:33:35.362912+0000","last_unstale":"2023-01-15T14:33:57.405395+0000","last_undegraded":"2023-01-15T14:33:57.405395+0000","last_fullsized":"2023-01-15T14:33:57.405395+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.394560678,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":974,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.317895+0000","last_change":"2023-01-15T14:33:57.317895+0000","last_active":"2023-01-15T14:33:57.317895+0000","last_peered":"2023-01-15T14:33:57.317895+0000","last_clean":"2023-01-15T14:33:57.317895+0000","last_became_active":"2023-01-15T14:33:35.261561+0000","last_became_peered":"2023-01-15T14:33:35.261561+0000","last_unstale":"2023-01-15T14:33:57.317895+0000","last_undegraded":"2023-01-15T14:33:57.317895+0000","last_fullsized":"2023-01-15T14:33:57.317895+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40752737100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1f","version":"318'256","reported_seq":1325,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.448154+0000","last_change":"2023-01-15T14:34:00.448154+0000","last_active":"2023-01-15T14:34:00.448154+0000","last_peered":"2023-01-15T14:34:00.448154+0000","last_clean":"2023-01-15T14:34:00.448154+0000","last_became_active":"2023-01-15T14:34:00.240936+0000","last_became_peered":"2023-01-15T14:34:00.240936+0000","last_unstale":"2023-01-15T14:34:00.448154+0000","last_undegraded":"2023-01-15T14:34:00.448154+0000","last_fullsized":"2023-01-15T14:34:00.448154+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.20345100599999999,"stat_sum":{"num_bytes":2974888,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":13341,"num_write":1,"num_write_kb":1699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":5949777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.20","version":"324'273","reported_seq":1354,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:33:59.938072+0000","last_change":"2023-01-15T14:33:57.415738+0000","last_active":"2023-01-15T14:33:59.938072+0000","last_peered":"2023-01-15T14:33:59.938072+0000","last_clean":"2023-01-15T14:33:59.938072+0000","last_became_active":"2023-01-15T14:33:55.472883+0000","last_became_peered":"2023-01-15T14:33:55.472883+0000","last_unstale":"2023-01-15T14:33:59.938072+0000","last_undegraded":"2023-01-15T14:33:59.938072+0000","last_fullsized":"2023-01-15T14:33:59.938072+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.49046371599999999,"stat_sum":{"num_bytes":4251137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10864,"num_write":9,"num_write_kb":4140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4438968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.21","version":"321'138","reported_seq":1044,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.111059+0000","last_change":"2023-01-15T14:33:57.005196+0000","last_active":"2023-01-15T14:34:02.111059+0000","last_peered":"2023-01-15T14:34:02.111059+0000","last_clean":"2023-01-15T14:34:02.111059+0000","last_became_active":"2023-01-15T14:33:45.267321+0000","last_became_peered":"2023-01-15T14:33:45.267321+0000","last_unstale":"2023-01-15T14:34:02.111059+0000","last_undegraded":"2023-01-15T14:34:02.111059+0000","last_fullsized":"2023-01-15T14:34:02.111059+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.102357873,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.22","version":"323'161","reported_seq":1329,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.619172+0000","last_change":"2023-01-15T14:33:57.619172+0000","last_active":"2023-01-15T14:33:57.619172+0000","last_peered":"2023-01-15T14:33:57.619172+0000","last_clean":"2023-01-15T14:33:57.619172+0000","last_became_active":"2023-01-15T14:33:55.378253+0000","last_became_peered":"2023-01-15T14:33:55.378253+0000","last_unstale":"2023-01-15T14:33:57.619172+0000","last_undegraded":"2023-01-15T14:33:57.619172+0000","last_fullsized":"2023-01-15T14:33:57.619172+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.50935289699999997,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":738,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.039871+0000","last_change":"2023-01-15T14:33:57.416771+0000","last_active":"2023-01-15T14:34:00.039871+0000","last_peered":"2023-01-15T14:34:00.039871+0000","last_clean":"2023-01-15T14:34:00.039871+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:34:00.039871+0000","last_undegraded":"2023-01-15T14:34:00.039871+0000","last_fullsized":"2023-01-15T14:34:00.039871+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.45831782799999998,"stat_sum":{"num_bytes":3802136,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":21818,"num_write":18,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.24","version":"348'259","reported_seq":1392,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.616353+0000","last_change":"2023-01-15T14:33:57.616353+0000","last_active":"2023-01-15T14:33:57.616353+0000","last_peered":"2023-01-15T14:33:57.616353+0000","last_clean":"2023-01-15T14:33:57.616353+0000","last_became_active":"2023-01-15T14:33:45.075309+0000","last_became_peered":"2023-01-15T14:33:45.075309+0000","last_unstale":"2023-01-15T14:33:57.616353+0000","last_undegraded":"2023-01-15T14:33:57.616353+0000","last_fullsized":"2023-01-15T14:33:57.616353+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.70814871000000001,"stat_sum":{"num_bytes":4593906,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10774,"num_write":8,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.25","version":"333'108","reported_seq":1116,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:01.957377+0000","last_change":"2023-01-15T14:33:57.331159+0000","last_active":"2023-01-15T14:34:01.957377+0000","last_peered":"2023-01-15T14:34:01.957377+0000","last_clean":"2023-01-15T14:34:01.957377+0000","last_became_active":"2023-01-15T14:33:45.168598+0000","last_became_peered":"2023-01-15T14:33:45.168598+0000","last_unstale":"2023-01-15T14:34:01.957377+0000","last_undegraded":"2023-01-15T14:34:01.957377+0000","last_fullsized":"2023-01-15T14:34:01.957377+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.33264791100000002,"stat_sum":{"num_bytes":2638765,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3030,"num_write":2,"num_write_kb":2577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":468,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.012852+0000","last_change":"2023-01-15T14:33:58.119558+0000","last_active":"2023-01-15T14:34:02.012852+0000","last_peered":"2023-01-15T14:34:02.012852+0000","last_clean":"2023-01-15T14:34:02.012852+0000","last_became_active":"2023-01-15T14:33:52.309110+0000","last_became_peered":"2023-01-15T14:33:52.309110+0000","last_unstale":"2023-01-15T14:34:02.012852+0000","last_undegraded":"2023-01-15T14:34:02.012852+0000","last_fullsized":"2023-01-15T14:34:02.012852+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.0016362760000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":1061,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.355572+0000","last_change":"2023-01-15T14:33:57.355572+0000","last_active":"2023-01-15T14:33:57.355572+0000","last_peered":"2023-01-15T14:33:57.355572+0000","last_clean":"2023-01-15T14:33:57.355572+0000","last_became_active":"2023-01-15T14:33:45.171090+0000","last_became_peered":"2023-01-15T14:33:45.171090+0000","last_unstale":"2023-01-15T14:33:57.355572+0000","last_undegraded":"2023-01-15T14:33:57.355572+0000","last_fullsized":"2023-01-15T14:33:57.355572+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.443073948,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.28","version":"333'248","reported_seq":1379,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.617361+0000","last_change":"2023-01-15T14:33:57.617361+0000","last_active":"2023-01-15T14:33:57.617361+0000","last_peered":"2023-01-15T14:33:57.617361+0000","last_clean":"2023-01-15T14:33:57.617361+0000","last_became_active":"2023-01-15T14:33:45.366831+0000","last_became_peered":"2023-01-15T14:33:45.366831+0000","last_unstale":"2023-01-15T14:33:57.617361+0000","last_undegraded":"2023-01-15T14:33:57.617361+0000","last_fullsized":"2023-01-15T14:33:57.617361+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:01.610","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.60974191300000002,"stat_sum":{"num_bytes":4838389,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8969,"num_write":8,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":6096764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":883,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.620313+0000","last_change":"2023-01-15T14:33:57.620313+0000","last_active":"2023-01-15T14:33:57.620313+0000","last_peered":"2023-01-15T14:33:57.620313+0000","last_clean":"2023-01-15T14:33:57.620313+0000","last_became_active":"2023-01-15T14:33:45.276960+0000","last_became_peered":"2023-01-15T14:33:45.276960+0000","last_unstale":"2023-01-15T14:33:57.620313+0000","last_undegraded":"2023-01-15T14:33:57.620313+0000","last_fullsized":"2023-01-15T14:33:57.620313+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.51033217900000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1234,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.011352+0000","last_change":"2023-01-15T14:33:56.904528+0000","last_active":"2023-01-15T14:34:02.011352+0000","last_peered":"2023-01-15T14:34:02.011352+0000","last_clean":"2023-01-15T14:34:02.011352+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:34:02.011352+0000","last_undegraded":"2023-01-15T14:34:02.011352+0000","last_fullsized":"2023-01-15T14:34:02.011352+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.0022419219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":46,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.039933+0000","last_change":"2023-01-15T14:33:57.893154+0000","last_active":"2023-01-15T14:34:00.039933+0000","last_peered":"2023-01-15T14:34:00.039933+0000","last_clean":"2023-01-15T14:34:00.039933+0000","last_became_active":"2023-01-15T14:33:57.261456+0000","last_became_peered":"2023-01-15T14:33:57.261456+0000","last_unstale":"2023-01-15T14:34:00.039933+0000","last_undegraded":"2023-01-15T14:34:00.039933+0000","last_fullsized":"2023-01-15T14:34:00.039933+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.630651882,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2c","version":"141'46","reported_seq":432,"reported_epoch":348,"state":"active+clean","last_fresh":"2023-01-15T14:33:57.303687+0000","last_change":"2023-01-15T14:33:57.303687+0000","last_active":"2023-01-15T14:33:57.303687+0000","last_peered":"2023-01-15T14:33:57.303687+0000","last_clean":"2023-01-15T14:33:57.303687+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:33:57.303687+0000","last_undegraded":"2023-01-15T14:33:57.303687+0000","last_fullsized":"2023-01-15T14:33:57.303687+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:10.970","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.29451823399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1207,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.178718+0000","last_change":"2023-01-15T14:33:58.122688+0000","last_active":"2023-01-15T14:34:02.178718+0000","last_peered":"2023-01-15T14:34:02.178718+0000","last_clean":"2023-01-15T14:34:02.178718+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:34:02.178718+0000","last_undegraded":"2023-01-15T14:34:02.178718+0000","last_fullsized":"2023-01-15T14:34:02.178718+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.105464771,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":25,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.011795+0000","last_change":"2023-01-15T14:33:58.326712+0000","last_active":"2023-01-15T14:34:02.011795+0000","last_peered":"2023-01-15T14:34:02.011795+0000","last_clean":"2023-01-15T14:34:02.011795+0000","last_became_active":"2023-01-15T14:33:57.210434+0000","last_became_peered":"2023-01-15T14:33:57.210434+0000","last_unstale":"2023-01-15T14:34:02.011795+0000","last_undegraded":"2023-01-15T14:34:02.011795+0000","last_fullsized":"2023-01-15T14:34:02.011795+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.31032643199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1101,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.013255+0000","last_change":"2023-01-15T14:33:58.121127+0000","last_active":"2023-01-15T14:34:02.013255+0000","last_peered":"2023-01-15T14:34:02.013255+0000","last_clean":"2023-01-15T14:34:02.013255+0000","last_became_active":"2023-01-15T14:33:55.174128+0000","last_became_peered":"2023-01-15T14:33:55.174128+0000","last_unstale":"2023-01-15T14:34:02.013255+0000","last_undegraded":"2023-01-15T14:34:02.013255+0000","last_fullsized":"2023-01-15T14:34:02.013255+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.20511622600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":976,"reported_epoch":351,"state":"active+clean","last_fresh":"2023-01-15T14:34:00.038159+0000","last_change":"2023-01-15T14:33:57.314582+0000","last_active":"2023-01-15T14:34:00.038159+0000","last_peered":"2023-01-15T14:34:00.038159+0000","last_clean":"2023-01-15T14:34:00.038159+0000","last_became_active":"2023-01-15T14:33:55.373498+0000","last_became_peered":"2023-01-15T14:33:55.373498+0000","last_unstale":"2023-01-15T14:34:00.038159+0000","last_undegraded":"2023-01-15T14:34:00.038159+0000","last_fullsized":"2023-01-15T14:34:00.038159+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":7,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.389510301,"stat_sum":{"num_bytes":5419866,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":25948,"num_write":35,"num_write_kb":13827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":13340458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f9","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.31","version":"204'150","reported_seq":1220,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:01.932599+0000","last_change":"2023-01-15T14:33:57.625989+0000","last_active":"2023-01-15T14:34:01.932599+0000","last_peered":"2023-01-15T14:34:01.932599+0000","last_clean":"2023-01-15T14:34:01.932599+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:34:01.932599+0000","last_undegraded":"2023-01-15T14:34:01.932599+0000","last_fullsized":"2023-01-15T14:34:01.932599+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.52721113600000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":106,"stat_sum":{"num_bytes":348083553,"num_objects":160,"num_object_clones":109,"num_object_copies":480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":179,"num_objects_misplaced":54,"num_objects_unfound":0,"num_objects_dirty":160,"num_whiteouts":14,"num_read":1013,"num_read_kb":1099200,"num_write":797,"num_write_kb":327032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":490,"num_bytes_recovered":292798087,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18665,"ondisk_log_size":18665,"up":318,"acting":304,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574921,"num_pgs":67,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1063028,"kb_used_data":92676264,"kb_used_omap":7688,"kb_used_meta":0,"kb_avail":92636352,"statfs":{"total":95948165120,"available":94859624448,"internally_reserved":40869888,"allocated":94900494336,"data_stored":94900494336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7873334,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":4.1269999999999998,"15min":4.0890000000000004},"min":{"1min":0.54000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2030000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":9.5380000000000003,"5min":6.3200000000000003,"15min":8.2460000000000004},"min":{"1min":0.58799999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":96.337000000000003,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.67400000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":3.6509999999999998,"15min":5.5800000000000001},"min":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":34.585999999999999,"5min":88.003,"15min":88.003},"last":0.63100000000000001},{"interface":"front","average":{"1min":13.302,"5min":8.1950000000000003,"15min":6.9180000000000001},"min":{"1min":0.58799999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.403999999999996,"5min":97.704999999999998,"15min":97.704999999999998},"last":94.537000000000006}]},{"osd":2,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":6.9400000000000004,"15min":5.5069999999999997},"min":{"1min":0.57199999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.179,"5min":151.31800000000001,"15min":151.31800000000001},"last":25.065999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":2.5790000000000002,"15min":3.645},"min":{"1min":0.626,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":91.837999999999994,"15min":91.837999999999994},"last":0.69799999999999995}]},{"osd":3,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":7.3799999999999999,"15min":3.0920000000000001},"min":{"1min":0.63600000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2370000000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":25.359999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":3.6360000000000001,"15min":4.3129999999999997},"min":{"1min":0.65000000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.2270000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.79600000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5519999999999996,"5min":5.6219999999999999,"15min":8.1509999999999998},"min":{"1min":0.63500000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":96.185000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":1.9259999999999999,"5min":5.782,"15min":2.5699999999999998},"min":{"1min":0.73099999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":22.41,"5min":164.21199999999999,"15min":164.21199999999999},"last":25.138000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.6440000000000001,"5min":5.0659999999999998,"15min":5.9059999999999997},"min":{"1min":0.72799999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":96.263999999999996,"5min":96.263999999999996,"15min":96.263999999999996},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.998,"5min":3.206,"15min":4.3650000000000002},"min":{"1min":0.69299999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.355,"5min":95.430000000000007,"15min":95.430000000000007},"last":25.199000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":5.5949999999999998,"15min":7.5469999999999997},"min":{"1min":0.83599999999999997,"5min":0.496,"15min":0.496},"max":{"1min":96.299999999999997,"5min":96.299999999999997,"15min":96.299999999999997},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.089,"5min":2.5150000000000001,"15min":1.571},"min":{"1min":0.84299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.2789999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":25.236999999999998}]}]},{"osd":6,"up_from":0,"seq":55834574922,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574922,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574920,"num_pgs":69,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1070856,"kb_used_data":92651192,"kb_used_omap":13607,"kb_used_meta":0,"kb_avail":92628524,"statfs":{"total":95948165120,"available":94851608576,"internally_reserved":23212032,"allocated":94874820608,"data_stored":94874820608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13933805,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":23.594000000000001,"15min":11.141},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.133,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":3.0009999999999999,"15min":1.5209999999999999},"min":{"1min":0.58299999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.024,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.74399999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":5.0730000000000004,"15min":7.4199999999999999},"min":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6659999999999999,"5min":171.91499999999999,"15min":171.91499999999999},"last":23.193000000000001},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.343,"15min":1.9770000000000001},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0529999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":31.756}]},{"osd":2,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":24.227,"15min":11.749000000000001},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.157,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":23.247},{"interface":"front","average":{"1min":2.6040000000000001,"5min":1.21,"15min":1.01},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":32.081000000000003,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.71199999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":2.9169999999999998,"15min":1.865},"min":{"1min":0.65300000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.2909999999999999,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.7370000000000001,"5min":24.498999999999999,"15min":12.266999999999999},"min":{"1min":0.64400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":14.961,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":23.289000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.88500000000000001,"15min":0.90800000000000003},"min":{"1min":0.55200000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.92300000000000004,"5min":2.1840000000000002,"15min":2.2599999999999998},"min":{"1min":0.71799999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3200000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.81899999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":3.9329999999999998,"15min":5.093},"min":{"1min":0.69599999999999995,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.365,"5min":99.780000000000001,"15min":99.780000000000001},"last":23.329000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":22.463999999999999,"15min":10.93},"min":{"1min":0.71599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.427,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":0.92000000000000004}]},{"osd":7,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":3.1539999999999999,"5min":5.5570000000000004,"15min":10.603999999999999},"min":{"1min":0.76700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":39.012999999999998,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.2370000000000001,"5min":24.864000000000001,"15min":15.779999999999999},"min":{"1min":0.877,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.0680000000000001,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":23.356000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":69,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1084580,"kb_used_data":92614920,"kb_used_omap":13867,"kb_used_meta":0,"kb_avail":92614800,"statfs":{"total":95948165120,"available":94837555200,"internally_reserved":122880,"allocated":94837678080,"data_stored":94837678080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14200138,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":7.4400000000000004,"15min":5.7119999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":17.731000000000002,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":3.4340000000000002,"15min":6.2389999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.9260000000000002,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.53700000000000003}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":7.1790000000000003,"15min":11.414},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96799999999999997,"5min":159.256,"15min":159.256},"last":0.46800000000000003},{"interface":"front","average":{"1min":2.6259999999999999,"5min":10.417999999999999,"15min":3.9660000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":17.826000000000001,"5min":174.578,"15min":174.578},"last":25.515999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":4.6849999999999996,"5min":2.8809999999999998,"15min":3.1930000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":86.412000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.48999999999999999},{"interface":"front","average":{"1min":1.95,"5min":6.7889999999999997,"15min":3.7250000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.876000000000001,"5min":174.45099999999999,"15min":174.45099999999999},"last":15.946999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":7.6879999999999997,"5min":29.686,"15min":14.27},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":84.640000000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.86899999999999999,"5min":32.134999999999998,"15min":19.960999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.615,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":17.036000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":6.4980000000000002,"15min":11.433999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.04,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":2.2309999999999999,"5min":6.0039999999999996,"15min":2.6819999999999999},"min":{"1min":0.70099999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":17.917000000000002,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.71899999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1949999999999998,"5min":5.9980000000000002,"15min":2.6869999999999998},"min":{"1min":0.66000000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":17.989000000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.964,"15min":3.1619999999999999},"min":{"1min":0.70499999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.23,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.75600000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":2.0219999999999998,"15min":3.3639999999999999},"min":{"1min":0.69499999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.266,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.878},{"interface":"front","average":{"1min":19.138000000000002,"5min":11.705,"15min":15.961},"min":{"1min":0.66200000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":281.13400000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":0.98599999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574921,"num_pgs":53,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1084556,"kb_used_data":92615780,"kb_used_omap":11573,"kb_used_meta":0,"kb_avail":92614824,"statfs":{"total":95948165120,"available":94837579776,"internally_reserved":978944,"allocated":94838558720,"data_stored":94838558720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11851145,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":3.109,"5min":3.5230000000000001,"15min":1.601},"min":{"1min":0.437,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":51.009999999999998,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.4299999999999999,"15min":0.95299999999999996},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0089999999999999,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.40999999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.1529999999999996,"5min":5.9059999999999997,"15min":2.5009999999999999},"min":{"1min":0.378,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":91.525999999999996,"5min":143.071,"15min":143.071},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":2.016,"15min":2.9089999999999998},"min":{"1min":0.42299999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.131,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.51400000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":8.2210000000000001,"15min":9.6750000000000007},"min":{"1min":0.5,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.3609999999999998,"5min":183.53,"15min":183.53},"last":0.63100000000000001},{"interface":"front","average":{"1min":16.692,"5min":7.5170000000000003,"15min":3.0169999999999999},"min":{"1min":0.61199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":155.47900000000001,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.78400000000000003}]},{"osd":4,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3650000000000002,"5min":6.5060000000000002,"15min":4.6020000000000003},"min":{"1min":0.67300000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":51.259999999999998,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":2.8809999999999998,"5min":5.2939999999999996,"15min":4.3040000000000003},"min":{"1min":0.48699999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":33.219999999999999,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.82999999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":2.4729999999999999,"15min":1.6080000000000001},"min":{"1min":0.57199999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2130000000000001,"5min":143.262,"15min":143.262},"last":0.98299999999999998},{"interface":"front","average":{"1min":3.4780000000000002,"5min":3.0539999999999998,"15min":1.792},"min":{"1min":0.70999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":51.326000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.88700000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.857,"15min":1.3069999999999999},"min":{"1min":0.60399999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.365,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.018,"5min":2.6160000000000001,"15min":1.6839999999999999},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.5,"5min":143.215,"15min":143.215},"last":1.0409999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":13.092000000000001,"5min":6.3120000000000003,"15min":4.1429999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":120.905,"5min":120.905,"15min":120.905},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.0129999999999999,"5min":2.8450000000000002,"15min":2.7989999999999999},"min":{"1min":0.66600000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.552,"5min":143.17099999999999,"15min":143.17099999999999},"last":1.228}]}]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":72,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1065444,"kb_used_data":92633940,"kb_used_omap":13136,"kb_used_meta":0,"kb_avail":92633936,"statfs":{"total":95948165120,"available":94857150464,"internally_reserved":4096,"allocated":94857154560,"data_stored":94857154560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13451927,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":3.633,"15min":3.806},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":3.9140000000000001,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":7.742,"5min":4.6130000000000004,"15min":4.4189999999999996},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":119.39100000000001,"5min":119.39100000000001,"15min":119.39100000000001},"last":0.57299999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":2.6469999999999998,"15min":1.5269999999999999},"min":{"1min":0.29799999999999999,"5min":0.245,"15min":0.245},"max":{"1min":95.879999999999995,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.8620000000000001,"15min":3.3359999999999999},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.9780000000000002,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.873}]},{"osd":3,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.8920000000000003,"5min":9.6750000000000007,"15min":16.449000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":100.137,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":7.782,"5min":6.8899999999999997,"15min":7.1260000000000003},"min":{"1min":0.45600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":119.804,"5min":119.804,"15min":119.804},"last":0.53400000000000003}]},{"osd":4,"last update":"Sun Jan 15 14:33:17 2023","interfaces":[{"interface":"back","average":{"1min":7.9249999999999998,"5min":17.140999999999998,"15min":8.8650000000000002},"min":{"1min":0.504,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":119.71899999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.7849999999999999,"5min":15.526,"15min":7.5780000000000003},"min":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":16.989999999999998,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.81399999999999995}]},{"osd":5,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.5449999999999999,"15min":1.2030000000000001},"min":{"1min":0.59899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.278,"5min":65.120999999999995,"15min":65.120999999999995},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.6579999999999999,"15min":3.5110000000000001},"min":{"1min":0.60599999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":4.0780000000000003,"5min":46.905999999999999,"15min":46.905999999999999},"last":0.83599999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":8.0069999999999997,"5min":5.0129999999999999,"15min":5.0049999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":119.759,"5min":119.759,"15min":119.759},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.603,"15min":1.2829999999999999},"min":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.349,"5min":65.156000000000006,"15min":65.156000000000006},"last":0.72699999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":3.6030000000000002,"5min":6.1109999999999998,"15min":6.7149999999999999},"min":{"1min":0.56000000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":31.620999999999999,"5min":157.03700000000001,"15min":157.03700000000001},"last":95.888999999999996},{"interface":"front","average":{"1min":8.0779999999999994,"5min":5.0060000000000002,"15min":4.835},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":119.65300000000001,"5min":119.65300000000001,"15min":119.65300000000001},"last":0.78700000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574922,"num_pgs":71,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1030272,"kb_used_data":92713940,"kb_used_omap":7535,"kb_used_meta":0,"kb_avail":92669108,"statfs":{"total":95948165120,"available":94893166592,"internally_reserved":45907968,"allocated":94939074560,"data_stored":94939074560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7716072,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":3.1219999999999999,"15min":1.456},"min":{"1min":0.40899999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1060000000000001,"5min":69.872,"15min":69.872},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":3.2309999999999999,"15min":1.5629999999999999},"min":{"1min":0.51300000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.2350000000000001,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.499}]},{"osd":2,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":3.4220000000000002,"15min":1.603},"min":{"1min":0.46999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.444,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.74099999999999999,"5min":2.6560000000000001,"15min":1.3580000000000001},"min":{"1min":0.52000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.302,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.37}]},{"osd":3,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":17.571999999999999,"5min":10.789,"15min":14.782},"min":{"1min":0.53300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":199.08000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.835,"5min":7.306,"15min":2.9689999999999999},"min":{"1min":0.56599999999999995,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":15.601000000000001,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.54500000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":16.695,"15min":6.2220000000000004},"min":{"1min":0.73199999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1879999999999999,"5min":1486.309,"15min":1486.309},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.038,"5min":17.925000000000001,"15min":6.6580000000000004},"min":{"1min":0.82499999999999996,"5min":0.439,"15min":0.439},"max":{"1min":1.3759999999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":0.66200000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":4.6749999999999998,"15min":8.0899999999999999},"min":{"1min":0.62,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.304,"5min":176.34399999999999,"15min":176.34399999999999},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.008,"5min":1.73,"15min":1.2689999999999999},"min":{"1min":0.69199999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2529999999999999,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.94999999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":3.5910000000000002,"15min":1.9770000000000001},"min":{"1min":0.67800000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5960000000000001,"5min":70.259,"15min":70.259},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.0880000000000001,"5min":3.0209999999999999,"15min":1.7629999999999999},"min":{"1min":0.83699999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5489999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":0.88800000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":6.1200000000000001,"15min":12.487},"min":{"1min":0.78400000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.48,"5min":292.61099999999999,"15min":292.61099999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.117,"5min":3.6070000000000002,"15min":1.9830000000000001},"min":{"1min":0.76100000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5229999999999999,"5min":70.322000000000003,"15min":70.322000000000003},"last":0.69299999999999995}]}]}],"pool_statfs":[]}} 2023-01-15T14:34:04.344 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:04.543 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:34:04.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:34:04.543 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:34:04.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3 3.3 3:ca59e2b0:::smithi09025264-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.16991361999999, 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62968594499999997, 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.544 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:04.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:04.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:04.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:04.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:04.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775131+0000", 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775138+0000", 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:04.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775148+0000", 2023-01-15T14:34:04.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7969999999999995e-06 2023-01-15T14:34:04.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:04.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775161+0000", 2023-01-15T14:34:04.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2469999999999999e-05 2023-01-15T14:34:04.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.857974+0000", 2023-01-15T14:34:04.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.082812887000000002 2023-01-15T14:34:04.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:04.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.858147+0000", 2023-01-15T14:34:04.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000173883 2023-01-15T14:34:04.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106098+0000", 2023-01-15T14:34:04.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24795019600000001 2023-01-15T14:34:04.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106181+0000", 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3248e-05 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106330+0000", 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014918200000000001 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:04.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106510+0000", 2023-01-15T14:34:04.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017979399999999999 2023-01-15T14:34:04.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:04.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106527+0000", 2023-01-15T14:34:04.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6643999999999999e-05 2023-01-15T14:34:04.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106605+0000", 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8650000000000001e-05 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106644+0000", 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8686999999999997e-05 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404687+0000", 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.298042745 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404735+0000", 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8306000000000001e-05 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404772+0000", 2023-01-15T14:34:04.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6977000000000002e-05 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404818+0000", 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.6439e-05 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:04.554 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.169875553, 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62995383500000002, 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:04.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:04.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:04.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:04.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775174+0000", 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3669999999999999e-06 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775184+0000", 2023-01-15T14:34:04.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5869999999999999e-06 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775190+0000", 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868440+0000", 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.093250126000000003 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:04.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868555+0000", 2023-01-15T14:34:04.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115003 2023-01-15T14:34:04.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.302962+0000", 2023-01-15T14:34:04.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.43440729700000003 2023-01-15T14:34:04.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303197+0000", 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023508000000000001 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404866+0000", 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101668723 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404985+0000", 2023-01-15T14:34:04.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118853 2023-01-15T14:34:04.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:04.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.405124+0000", 2023-01-15T14:34:04.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013951999999999999 2023-01-15T14:34:04.561 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.561 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:04.561 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:144 3.9 3:9988ac9d:::smithi09025264-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 337.07853700700002, 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63423976199999998, 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 144 2023-01-15T14:34:04.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866508+0000", 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866512+0000", 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5999999999999998e-06 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:04.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866523+0000", 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0740000000000001e-05 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866531+0000", 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3310000000000004e-06 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867744+0000", 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012127399999999999 2023-01-15T14:34:04.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:04.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867810+0000", 2023-01-15T14:34:04.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6506000000000003e-05 2023-01-15T14:34:04.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097225+0000", 2023-01-15T14:34:04.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.22941514900000001 2023-01-15T14:34:04.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:04.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097295+0000", 2023-01-15T14:34:04.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:34:04.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500386+0000", 2023-01-15T14:34:04.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40309051699999998 2023-01-15T14:34:04.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500521+0000", 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013561700000000001 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500749+0000", 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022754900000000001 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:509 3.9 3:9f86bdcb:::smithi09025264-22:head [write 2873212~588997] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 314.13011266400002, 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69623497099999998, 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 509 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.569 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:04.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:04.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:04.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.704 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:04.704 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:04.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.704 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:04.704 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814931+0000", 2023-01-15T14:34:04.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:04.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.705 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.705 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:04.705 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815512+0000", 2023-01-15T14:34:04.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058082200000000004 2023-01-15T14:34:04.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.705 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.705 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:04.705 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815553+0000", 2023-01-15T14:34:04.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1081999999999997e-05 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815571+0000", 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8028000000000001e-05 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906452+0000", 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090880586999999999 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906564+0000", 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111976 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208424+0000", 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.301860199 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208718+0000", 2023-01-15T14:34:04.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029347299999999999 2023-01-15T14:34:04.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:04.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209175+0000", 2023-01-15T14:34:04.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045692399999999999 2023-01-15T14:34:04.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209725+0000", 2023-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055027600000000004 2023-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209751+0000", 2023-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5812999999999998e-05 2023-01-15T14:34:04.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.210006+0000", 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025564099999999997 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311247+0000", 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10124119199999999 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:04.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311454+0000", 2023-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020695800000000001 2023-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511022+0000", 2023-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19956729200000001 2023-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:04.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511100+0000", 2023-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8589000000000003e-05 2023-01-15T14:34:04.972 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511168+0000", 2023-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8039000000000004e-05 2023-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:510 3.9 3:9f86bdcb:::smithi09025264-22:head [setxattr _header (57) in=7b,truncate 3462209] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:04.973 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 314.129449201, 2023-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69609656399999997, 2023-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:04.974 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 510 2023-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:04.975 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:04.976 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.977 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815596+0000", 2023-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:04.978 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815606+0000", 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7920000000000004e-06 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815627+0000", 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1239000000000001e-05 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815637+0000", 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0349e-05 2023-01-15T14:34:04.979 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906600+0000", 2023-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090963098000000006 2023-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906717+0000", 2023-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000116759 2023-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.980 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310000+0000", 2023-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40328322599999999 2023-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310249+0000", 2023-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024874400000000002 2023-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.981 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310559+0000", 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310382 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310951+0000", 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039124500000000003 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310975+0000", 2023-01-15T14:34:04.982 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4496000000000001e-05 2023-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311041+0000", 2023-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6309000000000005e-05 2023-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311260+0000", 2023-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021863399999999999 2023-01-15T14:34:04.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311588+0000", 2023-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032813599999999998 2023-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511351+0000", 2023-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19976275700000001 2023-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.984 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511507+0000", 2023-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015585100000000001 2023-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511693+0000", 2023-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186627 2023-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:04.985 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:511 3.9 3:9f86bdcb:::smithi09025264-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 314.129391833, 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69666647500000001, 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:04.986 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 511 2023-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:04.987 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815653+0000", 2023-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:04.988 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815660+0000", 2023-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0509999999999999e-06 2023-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815683+0000", 2023-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2742999999999999e-05 2023-01-15T14:34:04.989 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815698+0000", 2023-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5133999999999999e-05 2023-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908029+0000", 2023-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.092331448999999996 2023-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.990 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908193+0000", 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016342799999999999 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311115+0000", 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40292222300000002 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:04.991 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311224+0000", 2023-01-15T14:34:04.992 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010888899999999999 2023-01-15T14:34:04.992 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.992 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.992 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.992 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511800+0000", 2023-01-15T14:34:04.993 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20057596899999999 2023-01-15T14:34:04.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.993 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.993 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:04.993 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512036+0000", 2023-01-15T14:34:04.993 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236247 2023-01-15T14:34:04.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.993 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.993 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512321+0000", 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028457599999999998 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:693 3.3 3:ca59e2b0:::smithi09025264-1:head [sparse-read 0~2016807] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 296.04665848600001, 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70514199600000005, 2023-01-15T14:34:04.994 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:04.995 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:04.995 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:04.995 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:04.995 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:04.995 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 693 2023-01-15T14:34:04.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.995 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:04.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:04.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:04.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:04.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:04.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:04.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:04.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898387+0000", 2023-01-15T14:34:04.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:04.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:04.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898390+0000", 2023-01-15T14:34:04.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.252e-06 2023-01-15T14:34:04.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:04.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898401+0000", 2023-01-15T14:34:04.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0977e-05 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.998532+0000", 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100130443 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602663+0000", 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60413100500000005 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:04.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602817+0000", 2023-01-15T14:34:04.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015440500000000001 2023-01-15T14:34:04.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:04.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:04.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:04.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.603530+0000", 2023-01-15T14:34:04.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071246299999999999 2023-01-15T14:34:04.999 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:04.999 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:05.000 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.000 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.000 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.000 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1421 3.a 3:5919d83e:::smithi09025264-13:head [write 1723287~549075] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:05.000 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:05.000 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 229.032743977, 2023-01-15T14:34:05.000 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61118175699999999, 2023-01-15T14:34:05.000 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:05.000 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:05.000 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:05.001 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:05.001 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:05.001 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1421 2023-01-15T14:34:05.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.001 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:05.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:05.001 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:05.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:05.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:05.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:05.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912300+0000", 2023-01-15T14:34:05.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:05.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:05.003 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913127+0000", 2023-01-15T14:34:05.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00082704000000000002 2023-01-15T14:34:05.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:05.003 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913158+0000", 2023-01-15T14:34:05.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1146999999999998e-05 2023-01-15T14:34:05.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.004 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:05.004 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913177+0000", 2023-01-15T14:34:05.004 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:34:05.004 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.004 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.004 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913384+0000", 2023-01-15T14:34:05.004 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000206589 2023-01-15T14:34:05.004 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:05.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913408+0000", 2023-01-15T14:34:05.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3493e-05 2023-01-15T14:34:05.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318579+0000", 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40517134399999999 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318806+0000", 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022680899999999999 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319060+0000", 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254069 2023-01-15T14:34:05.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319527+0000", 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000467081 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319544+0000", 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6949999999999999e-05 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.300 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:05.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319778+0000", 2023-01-15T14:34:05.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023391500000000001 2023-01-15T14:34:05.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:05.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421126+0000", 2023-01-15T14:34:05.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101348626 2023-01-15T14:34:05.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522834+0000", 2023-01-15T14:34:05.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10170761 2023-01-15T14:34:05.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523142+0000", 2023-01-15T14:34:05.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854800000000001 2023-01-15T14:34:05.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:05.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523251+0000", 2023-01-15T14:34:05.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108411 2023-01-15T14:34:05.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:05.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523484+0000", 2023-01-15T14:34:05.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023292500000000001 2023-01-15T14:34:05.312 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1422 3.a 3:5919d83e:::smithi09025264-13:head [write 2781837~703343] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 229.03180518100001, 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61062001700000001, 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:05.313 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:05.314 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:05.314 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1422 2023-01-15T14:34:05.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.314 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:05.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:05.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:05.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913239+0000", 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914075+0000", 2023-01-15T14:34:05.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00083594099999999998 2023-01-15T14:34:05.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:05.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914104+0000", 2023-01-15T14:34:05.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8938999999999999e-05 2023-01-15T14:34:05.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:05.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914121+0000", 2023-01-15T14:34:05.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7306999999999999e-05 2023-01-15T14:34:05.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914234+0000", 2023-01-15T14:34:05.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112737 2023-01-15T14:34:05.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:05.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914257+0000", 2023-01-15T14:34:05.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3156e-05 2023-01-15T14:34:05.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319798+0000", 2023-01-15T14:34:05.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40554153100000001 2023-01-15T14:34:05.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:05.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319991+0000", 2023-01-15T14:34:05.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019217999999999999 2023-01-15T14:34:05.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:05.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320237+0000", 2023-01-15T14:34:05.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024621099999999998 2023-01-15T14:34:05.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:05.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320759+0000", 2023-01-15T14:34:05.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052174700000000003 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320776+0000", 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7173e-05 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.321080+0000", 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030424400000000001 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:05.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421162+0000", 2023-01-15T14:34:05.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100082085 2023-01-15T14:34:05.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522963+0000", 2023-01-15T14:34:05.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10180109399999999 2023-01-15T14:34:05.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523576+0000", 2023-01-15T14:34:05.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061277499999999997 2023-01-15T14:34:05.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:05.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523664+0000", 2023-01-15T14:34:05.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8002000000000002e-05 2023-01-15T14:34:05.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:05.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523861+0000", 2023-01-15T14:34:05.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019685899999999999 2023-01-15T14:34:05.326 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.327 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:05.327 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.327 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1423 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 3630312] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:05.327 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:05.327 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 229.03088264600001, 2023-01-15T14:34:05.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70862486899999999, 2023-01-15T14:34:05.328 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:05.328 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:05.328 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:05.328 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:05.328 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:05.328 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1423 2023-01-15T14:34:05.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.328 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:05.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.328 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:05.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:05.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:05.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:05.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:05.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914162+0000", 2023-01-15T14:34:05.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:05.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:05.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914172+0000", 2023-01-15T14:34:05.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0267e-05 2023-01-15T14:34:05.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:05.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914188+0000", 2023-01-15T14:34:05.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6212999999999999e-05 2023-01-15T14:34:05.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:05.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914201+0000", 2023-01-15T14:34:05.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3073e-05 2023-01-15T14:34:05.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914315+0000", 2023-01-15T14:34:05.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011357399999999999 2023-01-15T14:34:05.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:05.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914334+0000", 2023-01-15T14:34:05.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9004e-05 2023-01-15T14:34:05.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421258+0000", 2023-01-15T14:34:05.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.506924606 2023-01-15T14:34:05.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:05.334 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421518+0000", 2023-01-15T14:34:05.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025950499999999999 2023-01-15T14:34:05.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.334 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:05.334 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421843+0000", 2023-01-15T14:34:05.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032498299999999999 2023-01-15T14:34:05.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.335 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:05.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422222+0000", 2023-01-15T14:34:05.335 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037867599999999999 2023-01-15T14:34:05.335 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.335 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:05.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422245+0000", 2023-01-15T14:34:05.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3096999999999999e-05 2023-01-15T14:34:05.336 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.336 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.336 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:05.336 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422343+0000", 2023-01-15T14:34:05.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8487999999999996e-05 2023-01-15T14:34:05.336 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.337 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.337 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:05.337 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522890+0000", 2023-01-15T14:34:05.337 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10054658900000001 2023-01-15T14:34:05.337 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.337 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523075+0000", 2023-01-15T14:34:05.338 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185386 2023-01-15T14:34:05.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.338 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622151+0000", 2023-01-15T14:34:05.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099075724000000004 2023-01-15T14:34:05.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:05.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622244+0000", 2023-01-15T14:34:05.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3387999999999994e-05 2023-01-15T14:34:05.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:05.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622788+0000", 2023-01-15T14:34:05.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000543973 2023-01-15T14:34:05.340 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.341 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:05.341 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.341 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1424 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:05.341 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:05.341 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 229.03082727399999, 2023-01-15T14:34:05.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70984700199999995, 2023-01-15T14:34:05.342 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:05.342 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:05.342 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:05.342 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:05.342 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:05.343 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1424 2023-01-15T14:34:05.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.343 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:05.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:05.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:05.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:05.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:05.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:05.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914218+0000", 2023-01-15T14:34:05.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:05.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:05.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914224+0000", 2023-01-15T14:34:05.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6769999999999998e-06 2023-01-15T14:34:05.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:05.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914242+0000", 2023-01-15T14:34:05.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7759000000000001e-05 2023-01-15T14:34:05.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:05.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014331+0000", 2023-01-15T14:34:05.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088819 2023-01-15T14:34:05.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.349 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.349 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114642+0000", 2023-01-15T14:34:05.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100311027 2023-01-15T14:34:05.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.349 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:05.349 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114682+0000", 2023-01-15T14:34:05.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0028999999999999e-05 2023-01-15T14:34:05.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523918+0000", 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40923631500000002 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524124+0000", 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020524900000000001 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622921+0000", 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098797069000000001 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.623328+0000", 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000407539 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.624066+0000", 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073762899999999995 2023-01-15T14:34:05.351 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.352 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:05.352 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.352 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1430 3.a 3:52df5138:::smithi09025264-25:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:05.352 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:05.352 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 228.930667513, 2023-01-15T14:34:05.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81872653100000004, 2023-01-15T14:34:05.352 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:05.352 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:05.352 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1430 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:05.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:05.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:05.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014376+0000", 2023-01-15T14:34:05.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:05.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:05.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014391+0000", 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4922999999999999e-05 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014417+0000", 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5698999999999999e-05 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014435+0000", 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7827000000000001e-05 2023-01-15T14:34:05.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114759+0000", 2023-01-15T14:34:05.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100324859 2023-01-15T14:34:05.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:05.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114771+0000", 2023-01-15T14:34:05.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:34:05.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524222+0000", 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40945100800000001 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524545+0000", 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000322367 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:05.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.626981+0000", 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102436308 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627874+0000", 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089274899999999997 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627930+0000", 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6122000000000002e-05 2023-01-15T14:34:05.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628609+0000", 2023-01-15T14:34:05.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00067958700000000005 2023-01-15T14:34:05.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:05.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628798+0000", 2023-01-15T14:34:05.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018809 2023-01-15T14:34:05.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:05.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628870+0000", 2023-01-15T14:34:05.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2844999999999998e-05 2023-01-15T14:34:05.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832404+0000", 2023-01-15T14:34:05.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20353310299999999 2023-01-15T14:34:05.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832606+0000", 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020207999999999999 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833105+0000", 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049945399999999998 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1431 3.a 3:52df5138:::smithi09025264-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 228.93057622500001, 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81967516600000001, 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1431 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.362 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:05.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:05.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:05.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:05.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:05.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014469+0000", 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014477+0000", 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9300000000000003e-06 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:05.364 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014495+0000", 2023-01-15T14:34:05.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8363999999999999e-05 2023-01-15T14:34:05.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:05.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114610+0000", 2023-01-15T14:34:05.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10011521700000001 2023-01-15T14:34:05.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.503 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.503 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214948+0000", 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100337739 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214962+0000", 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4516e-05 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524874+0000", 2023-01-15T14:34:05.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30991196599999998 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.525013+0000", 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013873999999999999 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628222+0000", 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103208721 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628504+0000", 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028245700000000001 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833214+0000", 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.204709691 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833555+0000", 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034094000000000003 2023-01-15T14:34:05.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.513 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.513 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:05.513 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.834145+0000", 2023-01-15T14:34:05.513 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059006900000000003 2023-01-15T14:34:05.513 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.513 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2439 3.3 3:c11c7d81:::smithi09025264-11:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 128.86167172, 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70719632099999996, 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:05.514 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2439 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.516 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:05.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083373+0000", 2023-01-15T14:34:05.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:05.516 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.516 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.516 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:05.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083380+0000", 2023-01-15T14:34:05.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:34:05.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.517 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:05.517 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083391+0000", 2023-01-15T14:34:05.517 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0366e-05 2023-01-15T14:34:05.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.517 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:05.517 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083397+0000", 2023-01-15T14:34:05.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2740000000000004e-06 2023-01-15T14:34:05.518 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083648+0000", 2023-01-15T14:34:05.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025052599999999999 2023-01-15T14:34:05.518 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:05.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083666+0000", 2023-01-15T14:34:05.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8848999999999999e-05 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237725+0000", 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.154058361 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237945+0000", 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021978300000000001 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.386967+0000", 2023-01-15T14:34:05.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.149022562 2023-01-15T14:34:05.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:05.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387230+0000", 2023-01-15T14:34:05.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026256100000000002 2023-01-15T14:34:05.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:05.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387695+0000", 2023-01-15T14:34:05.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046480099999999998 2023-01-15T14:34:05.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:05.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388112+0000", 2023-01-15T14:34:05.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041713200000000002 2023-01-15T14:34:05.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:05.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388130+0000", 2023-01-15T14:34:05.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8315999999999999e-05 2023-01-15T14:34:05.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:05.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388189+0000", 2023-01-15T14:34:05.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8665999999999997e-05 2023-01-15T14:34:05.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:05.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388829+0000", 2023-01-15T14:34:05.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00063992699999999999 2023-01-15T14:34:05.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.489989+0000", 2023-01-15T14:34:05.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116059199999999 2023-01-15T14:34:05.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790300+0000", 2023-01-15T14:34:05.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30031108699999998 2023-01-15T14:34:05.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:05.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790396+0000", 2023-01-15T14:34:05.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5721000000000001e-05 2023-01-15T14:34:05.525 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.525 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:05.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790571+0000", 2023-01-15T14:34:05.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017464999999999999 2023-01-15T14:34:05.525 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.525 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:05.525 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2440 3.3 3:c11c7d81:::smithi09025264-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 128.86157998900001, 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70759506000000005, 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:05.526 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2440 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083465+0000", 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083470+0000", 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2630000000000003e-06 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083481+0000", 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0699999999999999e-05 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083488+0000", 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8870000000000003e-06 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180331+0000", 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.096843296999999995 2023-01-15T14:34:05.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.530 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:05.530 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180342+0000", 2023-01-15T14:34:05.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.031e-05 2023-01-15T14:34:05.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.530 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.530 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238012+0000", 2023-01-15T14:34:05.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.057670829 2023-01-15T14:34:05.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:05.531 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238257+0000", 2023-01-15T14:34:05.531 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000244959 2023-01-15T14:34:05.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.531 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590126+0000", 2023-01-15T14:34:05.531 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35186851699999999 2023-01-15T14:34:05.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:05.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590389+0000", 2023-01-15T14:34:05.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026276799999999998 2023-01-15T14:34:05.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790692+0000", 2023-01-15T14:34:05.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20030350299999999 2023-01-15T14:34:05.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790891+0000", 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019893999999999999 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.791061+0000", 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169913 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:05.533 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.708 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2450 3.a 3:52c3e133:::smithi09025264-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e230)", 2023-01-15T14:34:05.709 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:05.709 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 127.097727377, 2023-01-15T14:34:05.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.68251353100000001, 2023-01-15T14:34:05.709 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:05.709 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:05.709 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:05.709 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:05.709 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:05.710 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2450 2023-01-15T14:34:05.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.710 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:05.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:05.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:05.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:05.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:05.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:05.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847318+0000", 2023-01-15T14:34:05.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847324+0000", 2023-01-15T14:34:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:34:05.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847339+0000", 2023-01-15T14:34:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5582000000000001e-05 2023-01-15T14:34:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847349+0000", 2023-01-15T14:34:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0139999999999999e-05 2023-01-15T14:34:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947644+0000", 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10029455600000001 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947657+0000", 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2796000000000001e-05 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125638+0000", 2023-01-15T14:34:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17798127799999999 2023-01-15T14:34:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125780+0000", 2023-01-15T14:34:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014173000000000001 2023-01-15T14:34:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529035+0000", 2023-01-15T14:34:05.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40325542199999997 2023-01-15T14:34:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529366+0000", 2023-01-15T14:34:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033111700000000002 2023-01-15T14:34:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529832+0000", 2023-01-15T14:34:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046592399999999999 2023-01-15T14:34:05.717 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3390 3.1b 3:d85f42d9:::smithi09025264-45:head [write 1944996~791497] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 28.885048668, 2023-01-15T14:34:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.611364087, 2023-01-15T14:34:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:05.718 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3390 2023-01-15T14:34:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:05.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059995+0000", 2023-01-15T14:34:05.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061164+0000", 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001168325 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061194+0000", 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.0485999999999999e-05 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:05.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061213+0000", 2023-01-15T14:34:05.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.891e-05 2023-01-15T14:34:05.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159748+0000", 2023-01-15T14:34:05.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098535175000000003 2023-01-15T14:34:05.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:05.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159766+0000", 2023-01-15T14:34:05.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:34:05.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263011+0000", 2023-01-15T14:34:05.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10324554699999999 2023-01-15T14:34:05.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:05.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263253+0000", 2023-01-15T14:34:05.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000241568 2023-01-15T14:34:05.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:05.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466464+0000", 2023-01-15T14:34:05.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.203211006 2023-01-15T14:34:05.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:05.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466638+0000", 2023-01-15T14:34:05.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017412100000000001 2023-01-15T14:34:05.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:05.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466911+0000", 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000272971 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467425+0000", 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00051373 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467457+0000", 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2558999999999998e-05 2023-01-15T14:34:05.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467906+0000", 2023-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000448985 2023-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469895+0000", 2023-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0019888810000000001 2023-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470611+0000", 2023-01-15T14:34:05.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071556699999999996 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671178+0000", 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20056716699999999 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671277+0000", 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9594999999999998e-05 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671361+0000", 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4048000000000006e-05 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3391 3.1b 3:d85f42d9:::smithi09025264-45:head [write 3196560~554574] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 28.883805545000001, 2023-01-15T14:34:05.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71066412899999998, 2023-01-15T14:34:05.915 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:05.917 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:05.917 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:05.917 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:05.917 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:05.917 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3391 2023-01-15T14:34:05.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.918 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:05.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:05.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:05.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:05.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:05.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:05.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:05.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061238+0000", 2023-01-15T14:34:05.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:05.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:05.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062134+0000", 2023-01-15T14:34:05.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089621299999999998 2023-01-15T14:34:05.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:05.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:05.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:05.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062160+0000", 2023-01-15T14:34:05.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5908999999999999e-05 2023-01-15T14:34:05.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:06.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062172+0000", 2023-01-15T14:34:06.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.189e-05 2023-01-15T14:34:06.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:06.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159827+0000", 2023-01-15T14:34:06.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097654866000000007 2023-01-15T14:34:06.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:06.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159846+0000", 2023-01-15T14:34:06.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8516e-05 2023-01-15T14:34:06.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:06.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263326+0000", 2023-01-15T14:34:06.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103480766 2023-01-15T14:34:06.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:06.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263535+0000", 2023-01-15T14:34:06.124 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020861299999999999 2023-01-15T14:34:06.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:06.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467576+0000", 2023-01-15T14:34:06.124 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20404118600000001 2023-01-15T14:34:06.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.127 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:06.127 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467736+0000", 2023-01-15T14:34:06.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016000699999999999 2023-01-15T14:34:06.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:06.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467987+0000", 2023-01-15T14:34:06.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025042199999999998 2023-01-15T14:34:06.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:06.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468369+0000", 2023-01-15T14:34:06.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038238600000000002 2023-01-15T14:34:06.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:06.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468387+0000", 2023-01-15T14:34:06.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7730000000000001e-05 2023-01-15T14:34:06.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:06.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468719+0000", 2023-01-15T14:34:06.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033272599999999999 2023-01-15T14:34:06.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:06.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470317+0000", 2023-01-15T14:34:06.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001597336 2023-01-15T14:34:06.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:06.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570851+0000", 2023-01-15T14:34:06.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10053426 2023-01-15T14:34:06.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:06.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771618+0000", 2023-01-15T14:34:06.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200767152 2023-01-15T14:34:06.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:06.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771768+0000", 2023-01-15T14:34:06.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015015699999999999 2023-01-15T14:34:06.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:06.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771905+0000", 2023-01-15T14:34:06.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013621099999999999 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3392 3.1b 3:d85f42d9:::smithi09025264-45:head [setxattr _header (58) in=7b,truncate 3882483] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 28.882856310000001, 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71008243400000004, 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:06.133 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3392 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:06.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:06.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:06.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:06.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062188+0000", 2023-01-15T14:34:06.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:06.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:06.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062195+0000", 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2529999999999998e-06 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062208+0000", 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3003000000000001e-05 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062215+0000", 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5200000000000003e-06 2023-01-15T14:34:06.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159883+0000", 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097667532000000001 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159899+0000", 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6759999999999999e-05 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:06.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263853+0000", 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10395361 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264116+0000", 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026336099999999999 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468602+0000", 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20448550900000001 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:06.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468867+0000", 2023-01-15T14:34:06.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026495700000000002 2023-01-15T14:34:06.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:06.142 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469376+0000", 2023-01-15T14:34:06.142 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00050893100000000001 2023-01-15T14:34:06.143 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.143 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.143 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:06.144 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469789+0000", 2023-01-15T14:34:06.144 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000413158 2023-01-15T14:34:06.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.144 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.144 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:06.144 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469806+0000", 2023-01-15T14:34:06.144 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7513e-05 2023-01-15T14:34:06.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.144 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:06.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469884+0000", 2023-01-15T14:34:06.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7816000000000002e-05 2023-01-15T14:34:06.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:06.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470332+0000", 2023-01-15T14:34:06.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044807700000000001 2023-01-15T14:34:06.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:06.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570976+0000", 2023-01-15T14:34:06.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10064426899999999 2023-01-15T14:34:06.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:06.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772058+0000", 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20108148200000001 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772169+0000", 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011120099999999999 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772272+0000", 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000102982 2023-01-15T14:34:06.150 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:06.151 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:06.151 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:06.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.151 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3393 3.1b 3:d85f42d9:::smithi09025264-45:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:06.151 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:06.151 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 28.882822233999999, 2023-01-15T14:34:06.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71076842399999995, 2023-01-15T14:34:06.152 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:06.152 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:06.152 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:06.152 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:06.152 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:06.152 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3393 2023-01-15T14:34:06.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.152 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:06.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:06.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:06.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:06.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:06.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:06.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:06.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:06.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062223+0000", 2023-01-15T14:34:06.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:06.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:06.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062227+0000", 2023-01-15T14:34:06.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3900000000000003e-06 2023-01-15T14:34:06.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:06.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062237+0000", 2023-01-15T14:34:06.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4539999999999996e-06 2023-01-15T14:34:06.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:06.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062243+0000", 2023-01-15T14:34:06.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0329999999999997e-06 2023-01-15T14:34:06.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:06.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161412+0000", 2023-01-15T14:34:06.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099168708999999994 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161424+0000", 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2415999999999999e-05 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264185+0000", 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.1027609 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264421+0000", 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023640900000000001 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469985+0000", 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.205564001 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:06.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470264+0000", 2023-01-15T14:34:06.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027918700000000002 2023-01-15T14:34:06.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:06.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772390+0000", 2023-01-15T14:34:06.335 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30212558299999998 2023-01-15T14:34:06.335 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.335 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:06.335 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772684+0000", 2023-01-15T14:34:06.335 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029361799999999999 2023-01-15T14:34:06.335 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:06.335 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:06.335 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:06.336 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772992+0000", 2023-01-15T14:34:06.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854100000000001 2023-01-15T14:34:06.336 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:06.336 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:06.336 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:06.336 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:06.336 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:06.336 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:06.337 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:34:06.338 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:34:06.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:34:06.341 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:04.036+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:06.343 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:06.339+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:06.442 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:06.439+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:06.535 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:06.535 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:06.535 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:06.536 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:06.536 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:06.536 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:34:06.536 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:06.537 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:34:06.543 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:06.539+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:06.550 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:06.572 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:06.738 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:06.738 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:06.738 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:06.739 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:06.739 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:06.739 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:34:06.742 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:34:06.743 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:34:06.743 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:34:06.743 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:34:06.743 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:34:06.744 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:34:06.744 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:34:06.744 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:34:06.744 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:06.732+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:06.839 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:06.835+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:06.943 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:06.943 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:06.944 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:34:06.944 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:06.939+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:07.044 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:07.039+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:07.108 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:07.108 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:07.108 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:34:07.145 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:07.140+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:07.274 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:07.748 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:07.749 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:34:07.749 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:34:07.768 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:07.272+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:07.769 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:07.763+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:07.868 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:07.864+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:07.919 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:07.919 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:07.920 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:34:07.969 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:07.967+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:08.075 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:08.075 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:08.075 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:34:08.076 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:08.068+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:08.094 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:08.231 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:08.232 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:08.232 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:34:08.232 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":531,"stamp":"2023-01-15T14:34:07.916373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356436800,"num_objects":166,"num_object_clones":113,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":95,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":14,"num_read":1133,"num_read_kb":1099302,"num_write":967,"num_write_kb":334061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":388272772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18813,"ondisk_log_size":18813,"up":345,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":389,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6361420,"kb_used_data":555884476,"kb_used_omap":68245,"kb_used_meta":0,"kb_avail":555834860,"statfs":{"total":575688990720,"available":569174896640,"internally_reserved":50806784,"allocated":569225703424,"data_stored":569225703424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":69882914,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7762879,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-88,"num_objects_misplaced":-14,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":11,"num_write_kb":4613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":93113213,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.965598"},"pg_stats":[{"pgid":"3.69","version":"110'149","reported_seq":901,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.008353+0000","last_change":"2023-01-15T14:33:57.316235+0000","last_active":"2023-01-15T14:34:02.008353+0000","last_peered":"2023-01-15T14:34:02.008353+0000","last_clean":"2023-01-15T14:34:02.008353+0000","last_became_active":"2023-01-15T14:33:50.216417+0000","last_became_peered":"2023-01-15T14:33:50.216417+0000","last_unstale":"2023-01-15T14:34:02.008353+0000","last_undegraded":"2023-01-15T14:34:02.008353+0000","last_fullsized":"2023-01-15T14:34:02.008353+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.406510072,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.68","version":"351'252","reported_seq":1407,"reported_epoch":356,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2023-01-15T14:34:05.084850+0000","last_change":"2023-01-15T14:33:53.618260+0000","last_active":"2023-01-15T14:34:05.084850+0000","last_peered":"2023-01-15T14:34:05.084850+0000","last_clean":"2023-01-15T14:33:42.635266+0000","last_became_active":"2023-01-15T14:33:53.330670+0000","last_became_peered":"2023-01-15T14:33:53.330670+0000","last_unstale":"2023-01-15T14:34:05.084850+0000","last_undegraded":"2023-01-15T14:33:52.928061+0000","last_fullsized":"2023-01-15T14:34:05.084850+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8014979,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":9643107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[1,7,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":3},{"shards":"1,2,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.67","version":"191'79","reported_seq":58,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.079604+0000","last_change":"2023-01-15T14:33:57.312535+0000","last_active":"2023-01-15T14:34:05.079604+0000","last_peered":"2023-01-15T14:34:05.079604+0000","last_clean":"2023-01-15T14:34:05.079604+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:34:05.079604+0000","last_undegraded":"2023-01-15T14:34:05.079604+0000","last_fullsized":"2023-01-15T14:34:05.079604+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.40508863699999997,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.66","version":"102'52","reported_seq":364,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.114800+0000","last_change":"2023-01-15T14:33:57.299510+0000","last_active":"2023-01-15T14:34:02.114800+0000","last_peered":"2023-01-15T14:34:02.114800+0000","last_clean":"2023-01-15T14:34:02.114800+0000","last_became_active":"2023-01-15T14:33:48.695009+0000","last_became_peered":"2023-01-15T14:33:48.695009+0000","last_unstale":"2023-01-15T14:34:02.114800+0000","last_undegraded":"2023-01-15T14:34:02.114800+0000","last_fullsized":"2023-01-15T14:34:02.114800+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.390920724,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2142,"num_write":4,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.65","version":"349'120","reported_seq":1161,"reported_epoch":356,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:04.984721+0000","last_change":"2023-01-15T14:33:57.203450+0000","last_active":"2023-01-15T14:34:04.984721+0000","last_peered":"2023-01-15T14:34:04.984721+0000","last_clean":"2023-01-15T14:33:43.349693+0000","last_became_active":"2023-01-15T14:33:57.201006+0000","last_became_peered":"2023-01-15T14:33:57.201006+0000","last_unstale":"2023-01-15T14:34:04.984721+0000","last_undegraded":"2023-01-15T14:33:56.909186+0000","last_fullsized":"2023-01-15T14:33:56.908621+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7224063,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":3033,"num_write":12,"num_write_kb":4201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":11126498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,4],"avail_no_missing":["1","2","3","4"],"object_location_counts":[{"shards":"0,1,2,3,4","objects":1},{"shards":"1,2,3,4","objects":1},{"shards":"1,2,4","objects":1},{"shards":"1,4","objects":-1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.64","version":"334'251","reported_seq":1414,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.268552+0000","last_change":"2023-01-15T14:33:57.417656+0000","last_active":"2023-01-15T14:34:04.268552+0000","last_peered":"2023-01-15T14:34:04.268552+0000","last_clean":"2023-01-15T14:34:04.268552+0000","last_became_active":"2023-01-15T14:33:55.675085+0000","last_became_peered":"2023-01-15T14:33:55.675085+0000","last_unstale":"2023-01-15T14:34:04.268552+0000","last_undegraded":"2023-01-15T14:34:04.268552+0000","last_fullsized":"2023-01-15T14:34:04.268552+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.409133782,"stat_sum":{"num_bytes":3548733,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10773,"num_write":5,"num_write_kb":2741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.63","version":"149'233","reported_seq":743,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.268579+0000","last_change":"2023-01-15T14:33:57.518738+0000","last_active":"2023-01-15T14:34:04.268579+0000","last_peered":"2023-01-15T14:34:04.268579+0000","last_clean":"2023-01-15T14:34:04.268579+0000","last_became_active":"2023-01-15T14:33:55.277739+0000","last_became_peered":"2023-01-15T14:33:55.277739+0000","last_unstale":"2023-01-15T14:34:04.268579+0000","last_undegraded":"2023-01-15T14:34:04.268579+0000","last_fullsized":"2023-01-15T14:34:04.268579+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.41396429299999998,"stat_sum":{"num_bytes":3802136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":21819,"num_write":19,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.62","version":"348'169","reported_seq":1351,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.984077+0000","last_change":"2023-01-15T14:33:58.705842+0000","last_active":"2023-01-15T14:34:04.984077+0000","last_peered":"2023-01-15T14:34:04.984077+0000","last_clean":"2023-01-15T14:34:04.984077+0000","last_became_active":"2023-01-15T14:33:56.590183+0000","last_became_peered":"2023-01-15T14:33:56.590183+0000","last_unstale":"2023-01-15T14:34:04.984077+0000","last_undegraded":"2023-01-15T14:34:04.984077+0000","last_fullsized":"2023-01-15T14:34:04.984077+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.70374445200000002,"stat_sum":{"num_bytes":5361549,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6734,"num_write":11,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5210269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.61","version":"321'138","reported_seq":1004,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.168154+0000","last_change":"2023-01-15T14:33:57.185692+0000","last_active":"2023-01-15T14:34:04.168154+0000","last_peered":"2023-01-15T14:34:04.168154+0000","last_clean":"2023-01-15T14:34:04.168154+0000","last_became_active":"2023-01-15T14:33:53.421056+0000","last_became_peered":"2023-01-15T14:33:53.421056+0000","last_unstale":"2023-01-15T14:34:04.168154+0000","last_undegraded":"2023-01-15T14:34:04.168154+0000","last_fullsized":"2023-01-15T14:34:04.168154+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.27715718700000003,"stat_sum":{"num_bytes":1033664,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6059,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.60","version":"348'281","reported_seq":1414,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.983550+0000","last_change":"2023-01-15T14:33:58.810339+0000","last_active":"2023-01-15T14:34:04.983550+0000","last_peered":"2023-01-15T14:34:04.983550+0000","last_clean":"2023-01-15T14:34:04.983550+0000","last_became_active":"2023-01-15T14:33:56.478613+0000","last_became_peered":"2023-01-15T14:33:56.478613+0000","last_unstale":"2023-01-15T14:34:04.983550+0000","last_undegraded":"2023-01-15T14:34:04.983550+0000","last_fullsized":"2023-01-15T14:34:04.983550+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.80913330699999997,"stat_sum":{"num_bytes":5214108,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":10865,"num_write":14,"num_write_kb":5082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15418818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1377,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.269898+0000","last_change":"2023-01-15T14:33:57.313450+0000","last_active":"2023-01-15T14:34:04.269898+0000","last_peered":"2023-01-15T14:34:04.269898+0000","last_clean":"2023-01-15T14:34:04.269898+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:34:04.269898+0000","last_undegraded":"2023-01-15T14:34:04.269898+0000","last_fullsized":"2023-01-15T14:34:04.269898+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:51.919","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.39161994999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5e","version":"215'39","reported_seq":980,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.009623+0000","last_change":"2023-01-15T14:33:57.316972+0000","last_active":"2023-01-15T14:34:02.009623+0000","last_peered":"2023-01-15T14:34:02.009623+0000","last_clean":"2023-01-15T14:34:02.009623+0000","last_became_active":"2023-01-15T14:33:35.364143+0000","last_became_peered":"2023-01-15T14:33:35.364143+0000","last_unstale":"2023-01-15T14:34:02.009623+0000","last_undegraded":"2023-01-15T14:34:02.009623+0000","last_fullsized":"2023-01-15T14:34:02.009623+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40693008000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1309,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.033747+0000","last_change":"2023-01-15T14:34:00.343379+0000","last_active":"2023-01-15T14:34:02.033747+0000","last_peered":"2023-01-15T14:34:02.033747+0000","last_clean":"2023-01-15T14:34:02.033747+0000","last_became_active":"2023-01-15T14:34:00.242122+0000","last_became_peered":"2023-01-15T14:34:00.242122+0000","last_unstale":"2023-01-15T14:34:02.033747+0000","last_undegraded":"2023-01-15T14:34:02.033747+0000","last_fullsized":"2023-01-15T14:34:02.033747+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.099745922000000001,"stat_sum":{"num_bytes":2869928,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":23811,"num_write":13,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5c","version":"141'46","reported_seq":81,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.009661+0000","last_change":"2023-01-15T14:33:58.026285+0000","last_active":"2023-01-15T14:34:02.009661+0000","last_peered":"2023-01-15T14:34:02.009661+0000","last_clean":"2023-01-15T14:34:02.009661+0000","last_became_active":"2023-01-15T14:33:57.113032+0000","last_became_peered":"2023-01-15T14:33:57.113032+0000","last_unstale":"2023-01-15T14:34:02.009661+0000","last_undegraded":"2023-01-15T14:34:02.009661+0000","last_fullsized":"2023-01-15T14:34:02.009661+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.121922606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5b","version":"283'80","reported_seq":915,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979550+0000","last_change":"2023-01-15T14:33:57.113633+0000","last_active":"2023-01-15T14:34:04.979550+0000","last_peered":"2023-01-15T14:34:04.979550+0000","last_clean":"2023-01-15T14:34:04.979550+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:34:04.979550+0000","last_undegraded":"2023-01-15T14:34:04.979550+0000","last_fullsized":"2023-01-15T14:34:04.979550+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.20934807,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5a","version":"329'348","reported_seq":1494,"reported_epoch":357,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:34:07.731858+0000","last_change":"2023-01-15T14:34:07.617290+0000","last_active":"2023-01-15T14:34:07.731858+0000","last_peered":"2023-01-15T14:34:07.731858+0000","last_clean":"2023-01-15T14:33:50.015328+0000","last_became_active":"2023-01-15T14:33:53.229261+0000","last_became_peered":"2023-01-15T14:33:53.229261+0000","last_unstale":"2023-01-15T14:34:07.731858+0000","last_undegraded":"2023-01-15T14:34:07.731858+0000","last_fullsized":"2023-01-15T14:33:52.913106+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:02.755","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.45505085000000001,"stat_sum":{"num_bytes":5983424,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":19294,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[1,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.59","version":"294'256","reported_seq":1278,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.036419+0000","last_change":"2023-01-15T14:33:57.406685+0000","last_active":"2023-01-15T14:34:02.036419+0000","last_peered":"2023-01-15T14:34:02.036419+0000","last_clean":"2023-01-15T14:34:02.036419+0000","last_became_active":"2023-01-15T14:33:55.671990+0000","last_became_peered":"2023-01-15T14:33:55.671990+0000","last_unstale":"2023-01-15T14:34:02.036419+0000","last_undegraded":"2023-01-15T14:34:02.036419+0000","last_fullsized":"2023-01-15T14:34:02.036419+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.49887967,"stat_sum":{"num_bytes":1210137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":5,"num_read_kb":3549,"num_write":5,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.58","version":"323'320","reported_seq":1484,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.977996+0000","last_change":"2023-01-15T14:33:57.004059+0000","last_active":"2023-01-15T14:34:04.977996+0000","last_peered":"2023-01-15T14:34:04.977996+0000","last_clean":"2023-01-15T14:34:04.977996+0000","last_became_active":"2023-01-15T14:33:40.127669+0000","last_became_peered":"2023-01-15T14:33:40.127669+0000","last_unstale":"2023-01-15T14:34:04.977996+0000","last_undegraded":"2023-01-15T14:34:04.977996+0000","last_fullsized":"2023-01-15T14:34:04.977996+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.103642679,"stat_sum":{"num_bytes":4332079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.57","version":"351'98","reported_seq":97,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.269756+0000","last_change":"2023-01-15T14:33:57.890625+0000","last_active":"2023-01-15T14:34:04.269756+0000","last_peered":"2023-01-15T14:34:04.269756+0000","last_clean":"2023-01-15T14:34:04.269756+0000","last_became_active":"2023-01-15T14:33:36.994447+0000","last_became_peered":"2023-01-15T14:33:36.994447+0000","last_unstale":"2023-01-15T14:34:04.269756+0000","last_undegraded":"2023-01-15T14:34:04.269756+0000","last_fullsized":"2023-01-15T14:34:04.269756+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.969000109,"stat_sum":{"num_bytes":1512663,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3025326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.56","version":"320'127","reported_seq":634,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979227+0000","last_change":"2023-01-15T14:34:00.541392+0000","last_active":"2023-01-15T14:34:04.979227+0000","last_peered":"2023-01-15T14:34:04.979227+0000","last_clean":"2023-01-15T14:34:04.979227+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:34:04.979227+0000","last_undegraded":"2023-01-15T14:34:04.979227+0000","last_fullsized":"2023-01-15T14:34:04.979227+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.30190621200000001,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.55","version":"110'55","reported_seq":1011,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.158482+0000","last_change":"2023-01-15T14:33:57.315559+0000","last_active":"2023-01-15T14:34:02.158482+0000","last_peered":"2023-01-15T14:34:02.158482+0000","last_clean":"2023-01-15T14:34:02.158482+0000","last_became_active":"2023-01-15T14:33:52.510429+0000","last_became_peered":"2023-01-15T14:33:52.510429+0000","last_unstale":"2023-01-15T14:34:02.158482+0000","last_undegraded":"2023-01-15T14:34:02.158482+0000","last_fullsized":"2023-01-15T14:34:02.158482+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.40629608699999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1133,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:03.976291+0000","last_change":"2023-01-15T14:33:57.109553+0000","last_active":"2023-01-15T14:34:03.976291+0000","last_peered":"2023-01-15T14:34:03.976291+0000","last_clean":"2023-01-15T14:34:03.976291+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:34:03.976291+0000","last_undegraded":"2023-01-15T14:34:03.976291+0000","last_fullsized":"2023-01-15T14:34:03.976291+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:33:45.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:45.583005+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.201302861,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.53","version":"352'397","reported_seq":1407,"reported_epoch":356,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:04.984279+0000","last_change":"2023-01-15T14:33:58.206587+0000","last_active":"2023-01-15T14:34:04.984279+0000","last_peered":"2023-01-15T14:34:04.984279+0000","last_clean":"2023-01-15T14:33:55.165228+0000","last_became_active":"2023-01-15T14:33:58.106461+0000","last_became_peered":"2023-01-15T14:33:58.106461+0000","last_unstale":"2023-01-15T14:34:04.984279+0000","last_undegraded":"2023-01-15T14:33:58.004141+0000","last_fullsized":"2023-01-15T14:33:58.002666+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:44.609","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.092650028999999995,"stat_sum":{"num_bytes":26431669,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":18,"num_read_kb":17078,"num_write":19,"num_write_kb":11543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":6},{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":734,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979524+0000","last_change":"2023-01-15T14:33:56.903307+0000","last_active":"2023-01-15T14:34:04.979524+0000","last_peered":"2023-01-15T14:34:04.979524+0000","last_clean":"2023-01-15T14:34:04.979524+0000","last_became_active":"2023-01-15T14:33:55.472400+0000","last_became_peered":"2023-01-15T14:33:55.472400+0000","last_unstale":"2023-01-15T14:34:04.979524+0000","last_undegraded":"2023-01-15T14:34:04.979524+0000","last_fullsized":"2023-01-15T14:34:04.979524+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.001336634,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.51","version":"348'316","reported_seq":1523,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.983982+0000","last_change":"2023-01-15T14:33:57.893317+0000","last_active":"2023-01-15T14:34:04.983982+0000","last_peered":"2023-01-15T14:34:04.983982+0000","last_clean":"2023-01-15T14:34:04.983982+0000","last_became_active":"2023-01-15T14:33:26.107940+0000","last_became_peered":"2023-01-15T14:33:26.107940+0000","last_unstale":"2023-01-15T14:34:04.983982+0000","last_undegraded":"2023-01-15T14:34:04.983982+0000","last_fullsized":"2023-01-15T14:34:04.983982+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:36.458","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.79495623100000001,"stat_sum":{"num_bytes":7496813,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.50","version":"337'420","reported_seq":1586,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.978292+0000","last_change":"2023-01-15T14:34:02.564430+0000","last_active":"2023-01-15T14:34:04.978292+0000","last_peered":"2023-01-15T14:34:04.978292+0000","last_clean":"2023-01-15T14:34:04.978292+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:34:04.978292+0000","last_undegraded":"2023-01-15T14:34:04.978292+0000","last_fullsized":"2023-01-15T14:34:04.978292+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.0084659680000000008,"stat_sum":{"num_bytes":9681416,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":52505,"num_write":49,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4f","version":"155'145","reported_seq":828,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.110295+0000","last_change":"2023-01-15T14:33:57.353637+0000","last_active":"2023-01-15T14:34:02.110295+0000","last_peered":"2023-01-15T14:34:02.110295+0000","last_clean":"2023-01-15T14:34:02.110295+0000","last_became_active":"2023-01-15T14:33:25.129936+0000","last_became_peered":"2023-01-15T14:33:25.129936+0000","last_unstale":"2023-01-15T14:34:02.110295+0000","last_undegraded":"2023-01-15T14:34:02.110295+0000","last_fullsized":"2023-01-15T14:34:02.110295+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44322561900000002,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":101,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.169360+0000","last_change":"2023-01-15T14:33:56.915710+0000","last_active":"2023-01-15T14:34:04.169360+0000","last_peered":"2023-01-15T14:34:04.169360+0000","last_clean":"2023-01-15T14:34:04.169360+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:34:04.169360+0000","last_undegraded":"2023-01-15T14:34:04.169360+0000","last_fullsized":"2023-01-15T14:34:04.169360+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.0080109380000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4d","version":"356'262","reported_seq":1365,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.911880+0000","last_change":"2023-01-15T14:34:06.586527+0000","last_active":"2023-01-15T14:34:06.911880+0000","last_peered":"2023-01-15T14:34:06.911880+0000","last_clean":"2023-01-15T14:34:06.911880+0000","last_became_active":"2023-01-15T14:34:05.178613+0000","last_became_peered":"2023-01-15T14:34:05.178613+0000","last_unstale":"2023-01-15T14:34:06.911880+0000","last_undegraded":"2023-01-15T14:34:06.911880+0000","last_fullsized":"2023-01-15T14:34:06.911880+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":1.4066433030000001,"stat_sum":{"num_bytes":12124166,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":15961,"num_write":18,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4c","version":"309'71","reported_seq":111,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.980967+0000","last_change":"2023-01-15T14:33:57.413785+0000","last_active":"2023-01-15T14:34:04.980967+0000","last_peered":"2023-01-15T14:34:04.980967+0000","last_clean":"2023-01-15T14:34:04.980967+0000","last_became_active":"2023-01-15T14:33:52.311859+0000","last_became_peered":"2023-01-15T14:33:52.311859+0000","last_unstale":"2023-01-15T14:34:04.980967+0000","last_undegraded":"2023-01-15T14:34:04.980967+0000","last_fullsized":"2023-01-15T14:34:04.980967+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.40874315500000002,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":757,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.984543+0000","last_change":"2023-01-15T14:33:57.627070+0000","last_active":"2023-01-15T14:34:04.984543+0000","last_peered":"2023-01-15T14:34:04.984543+0000","last_clean":"2023-01-15T14:34:04.984543+0000","last_became_active":"2023-01-15T14:33:52.406605+0000","last_became_peered":"2023-01-15T14:33:52.406605+0000","last_unstale":"2023-01-15T14:34:04.984543+0000","last_undegraded":"2023-01-15T14:34:04.984543+0000","last_fullsized":"2023-01-15T14:34:04.984543+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:42.594","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.515074951,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4a","version":"348'371","reported_seq":1482,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.079832+0000","last_change":"2023-01-15T14:33:58.018192+0000","last_active":"2023-01-15T14:34:05.079832+0000","last_peered":"2023-01-15T14:34:05.079832+0000","last_clean":"2023-01-15T14:34:05.079832+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:34:05.079832+0000","last_undegraded":"2023-01-15T14:34:05.079832+0000","last_fullsized":"2023-01-15T14:34:05.079832+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:33.328","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10440110800000001,"stat_sum":{"num_bytes":12470870,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":63,"num_read_kb":67536,"num_write":73,"num_write_kb":23626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":939,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.978037+0000","last_change":"2023-01-15T14:34:00.046526+0000","last_active":"2023-01-15T14:34:04.978037+0000","last_peered":"2023-01-15T14:34:04.978037+0000","last_clean":"2023-01-15T14:34:04.978037+0000","last_became_active":"2023-01-15T14:34:00.043221+0000","last_became_peered":"2023-01-15T14:34:00.043221+0000","last_unstale":"2023-01-15T14:34:04.978037+0000","last_undegraded":"2023-01-15T14:34:04.978037+0000","last_fullsized":"2023-01-15T14:34:04.978037+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.00195963,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.48","version":"179'204","reported_seq":1359,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.137267+0000","last_change":"2023-01-15T14:33:57.356590+0000","last_active":"2023-01-15T14:34:02.137267+0000","last_peered":"2023-01-15T14:34:02.137267+0000","last_clean":"2023-01-15T14:34:02.137267+0000","last_became_active":"2023-01-15T14:33:55.375518+0000","last_became_peered":"2023-01-15T14:33:55.375518+0000","last_unstale":"2023-01-15T14:34:02.137267+0000","last_undegraded":"2023-01-15T14:34:02.137267+0000","last_fullsized":"2023-01-15T14:34:02.137267+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:43.662","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.34933974800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1119,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.034333+0000","last_change":"2023-01-15T14:33:57.300260+0000","last_active":"2023-01-15T14:34:02.034333+0000","last_peered":"2023-01-15T14:34:02.034333+0000","last_clean":"2023-01-15T14:34:02.034333+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:34:02.034333+0000","last_undegraded":"2023-01-15T14:34:02.034333+0000","last_fullsized":"2023-01-15T14:34:02.034333+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.39166528900000003,"stat_sum":{"num_bytes":496762,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":5,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":1033,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.186647+0000","last_change":"2023-01-15T14:33:57.829426+0000","last_active":"2023-01-15T14:34:05.186647+0000","last_peered":"2023-01-15T14:34:05.186647+0000","last_clean":"2023-01-15T14:34:05.186647+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:34:05.186647+0000","last_undegraded":"2023-01-15T14:34:05.186647+0000","last_fullsized":"2023-01-15T14:34:05.186647+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:55.245","scrub_duration":0.30448409700000001,"objects_trimmed":0,"snaptrim_duration":0.714760332,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":918,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.170081+0000","last_change":"2023-01-15T14:33:56.909121+0000","last_active":"2023-01-15T14:34:04.170081+0000","last_peered":"2023-01-15T14:34:04.170081+0000","last_clean":"2023-01-15T14:34:04.170081+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:34:04.170081+0000","last_undegraded":"2023-01-15T14:34:04.170081+0000","last_fullsized":"2023-01-15T14:34:04.170081+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045307990000000003,"stat_sum":{"num_bytes":2635976,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":19927,"num_write":15,"num_write_kb":6659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4745860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1274,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.033732+0000","last_change":"2023-01-15T14:33:57.298701+0000","last_active":"2023-01-15T14:34:02.033732+0000","last_peered":"2023-01-15T14:34:02.033732+0000","last_clean":"2023-01-15T14:34:02.033732+0000","last_became_active":"2023-01-15T14:33:55.678019+0000","last_became_peered":"2023-01-15T14:33:55.678019+0000","last_unstale":"2023-01-15T14:34:02.033732+0000","last_undegraded":"2023-01-15T14:34:02.033732+0000","last_fullsized":"2023-01-15T14:34:02.033732+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:27.120","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.390225449,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.43","version":"353'316","reported_seq":1519,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.268739+0000","last_change":"2023-01-15T14:34:02.217041+0000","last_active":"2023-01-15T14:34:04.268739+0000","last_peered":"2023-01-15T14:34:04.268739+0000","last_clean":"2023-01-15T14:34:04.268739+0000","last_became_active":"2023-01-15T14:34:02.211232+0000","last_became_peered":"2023-01-15T14:34:02.211232+0000","last_unstale":"2023-01-15T14:34:04.268739+0000","last_undegraded":"2023-01-15T14:34:04.268739+0000","last_fullsized":"2023-01-15T14:34:04.268739+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.0028674299999999998,"stat_sum":{"num_bytes":6319564,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":16585,"num_write":6,"num_write_kb":4207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3395080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.42","version":"155'101","reported_seq":1035,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.112021+0000","last_change":"2023-01-15T14:33:57.621398+0000","last_active":"2023-01-15T14:34:02.112021+0000","last_peered":"2023-01-15T14:34:02.112021+0000","last_clean":"2023-01-15T14:34:02.112021+0000","last_became_active":"2023-01-15T14:33:55.376782+0000","last_became_peered":"2023-01-15T14:33:55.376782+0000","last_unstale":"2023-01-15T14:34:02.112021+0000","last_undegraded":"2023-01-15T14:34:02.112021+0000","last_fullsized":"2023-01-15T14:34:02.112021+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.51101871399999999,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1115,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.036887+0000","last_change":"2023-01-15T14:33:57.302504+0000","last_active":"2023-01-15T14:34:02.036887+0000","last_peered":"2023-01-15T14:34:02.036887+0000","last_clean":"2023-01-15T14:34:02.036887+0000","last_became_active":"2023-01-15T14:33:55.771969+0000","last_became_peered":"2023-01-15T14:33:55.771969+0000","last_unstale":"2023-01-15T14:34:02.036887+0000","last_undegraded":"2023-01-15T14:34:02.036887+0000","last_fullsized":"2023-01-15T14:34:02.036887+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.29405439700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.40","version":"335'307","reported_seq":1399,"reported_epoch":356,"state":"active+recovery_wait+degraded","last_fresh":"2023-01-15T14:34:04.979357+0000","last_change":"2023-01-15T14:33:58.118750+0000","last_active":"2023-01-15T14:34:04.979357+0000","last_peered":"2023-01-15T14:34:04.979357+0000","last_clean":"2023-01-15T14:33:43.952880+0000","last_became_active":"2023-01-15T14:33:58.117085+0000","last_became_peered":"2023-01-15T14:33:58.117085+0000","last_unstale":"2023-01-15T14:34:04.979357+0000","last_undegraded":"2023-01-15T14:33:57.914305+0000","last_fullsized":"2023-01-15T14:34:04.979357+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.704289622,"stat_sum":{"num_bytes":12753904,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":35541,"num_write":28,"num_write_kb":11001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":16413587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"321'233","reported_seq":1374,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:07.012116+0000","last_change":"2023-01-15T14:33:58.333573+0000","last_active":"2023-01-15T14:34:07.012116+0000","last_peered":"2023-01-15T14:34:07.012116+0000","last_clean":"2023-01-15T14:34:07.012116+0000","last_became_active":"2023-01-15T14:33:58.217783+0000","last_became_peered":"2023-01-15T14:33:58.217783+0000","last_unstale":"2023-01-15T14:34:07.012116+0000","last_undegraded":"2023-01-15T14:34:07.012116+0000","last_fullsized":"2023-01-15T14:34:07.012116+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:28.180","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.014066018,"stat_sum":{"num_bytes":4764997,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6605283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3e","version":"294'63","reported_seq":839,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.085451+0000","last_change":"2023-01-15T14:33:57.728361+0000","last_active":"2023-01-15T14:34:05.085451+0000","last_peered":"2023-01-15T14:34:05.085451+0000","last_clean":"2023-01-15T14:34:05.085451+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:34:05.085451+0000","last_undegraded":"2023-01-15T14:34:05.085451+0000","last_fullsized":"2023-01-15T14:34:05.085451+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.61381065800000001,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3d","version":"356'236","reported_seq":1332,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.400650+0000","last_change":"2023-01-15T14:34:06.401367+0000","last_active":"2023-01-15T14:34:06.400650+0000","last_peered":"2023-01-15T14:34:06.400650+0000","last_clean":"2023-01-15T14:34:06.400650+0000","last_became_active":"2023-01-15T14:34:05.187121+0000","last_became_peered":"2023-01-15T14:34:05.187121+0000","last_unstale":"2023-01-15T14:34:06.400650+0000","last_undegraded":"2023-01-15T14:34:06.400650+0000","last_fullsized":"2023-01-15T14:34:06.400650+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":1.2126637869999999,"stat_sum":{"num_bytes":8050279,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":13797,"num_write":16,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":8050279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3c","version":"141'46","reported_seq":351,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.010135+0000","last_change":"2023-01-15T14:33:56.912541+0000","last_active":"2023-01-15T14:34:02.010135+0000","last_peered":"2023-01-15T14:34:02.010135+0000","last_clean":"2023-01-15T14:34:02.010135+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:34:02.010135+0000","last_undegraded":"2023-01-15T14:34:02.010135+0000","last_fullsized":"2023-01-15T14:34:02.010135+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.470","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.0047236029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":29,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979267+0000","last_change":"2023-01-15T14:33:58.125276+0000","last_active":"2023-01-15T14:34:04.979267+0000","last_peered":"2023-01-15T14:34:04.979267+0000","last_clean":"2023-01-15T14:34:04.979267+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:34:04.979267+0000","last_undegraded":"2023-01-15T14:34:04.979267+0000","last_fullsized":"2023-01-15T14:34:04.979267+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.21169681200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1352,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.170270+0000","last_change":"2023-01-15T14:33:57.286278+0000","last_active":"2023-01-15T14:34:04.170270+0000","last_peered":"2023-01-15T14:34:04.170270+0000","last_clean":"2023-01-15T14:34:04.170270+0000","last_became_active":"2023-01-15T14:33:55.474944+0000","last_became_peered":"2023-01-15T14:33:55.474944+0000","last_unstale":"2023-01-15T14:34:04.170270+0000","last_undegraded":"2023-01-15T14:34:04.170270+0000","last_fullsized":"2023-01-15T14:34:04.170270+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.37705996200000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.39","version":"334'351","reported_seq":1467,"reported_epoch":357,"state":"peering","last_fresh":"2023-01-15T14:34:06.911835+0000","last_change":"2023-01-15T14:34:06.912209+0000","last_active":"2023-01-15T14:34:06.911783+0000","last_peered":"2023-01-15T14:34:06.173371+0000","last_clean":"2023-01-15T14:33:50.016390+0000","last_became_active":"2023-01-15T14:33:53.436135+0000","last_became_peered":"2023-01-15T14:33:53.436135+0000","last_unstale":"2023-01-15T14:34:06.911835+0000","last_undegraded":"2023-01-15T14:34:06.911835+0000","last_fullsized":"2023-01-15T14:34:06.911835+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:40.787","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.0051603159999999999,"stat_sum":{"num_bytes":24023444,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":66941,"num_write":7,"num_write_kb":7292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":26862613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"352'315","reported_seq":1533,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.037579+0000","last_change":"2023-01-15T14:33:58.009343+0000","last_active":"2023-01-15T14:34:02.037579+0000","last_peered":"2023-01-15T14:34:02.037579+0000","last_clean":"2023-01-15T14:34:02.037579+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:34:02.037579+0000","last_undegraded":"2023-01-15T14:34:02.037579+0000","last_fullsized":"2023-01-15T14:34:02.037579+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.10683759900000001,"stat_sum":{"num_bytes":8461433,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":42848,"num_write":39,"num_write_kb":13455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":415,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.081415+0000","last_change":"2023-01-15T14:33:58.124010+0000","last_active":"2023-01-15T14:34:05.081415+0000","last_peered":"2023-01-15T14:34:05.081415+0000","last_clean":"2023-01-15T14:34:05.081415+0000","last_became_active":"2023-01-15T14:33:52.412128+0000","last_became_peered":"2023-01-15T14:33:52.412128+0000","last_unstale":"2023-01-15T14:34:05.081415+0000","last_undegraded":"2023-01-15T14:34:05.081415+0000","last_fullsized":"2023-01-15T14:34:05.081415+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.207923046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":567,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.085677+0000","last_change":"2023-01-15T14:33:57.332104+0000","last_active":"2023-01-15T14:34:05.085677+0000","last_peered":"2023-01-15T14:34:05.085677+0000","last_clean":"2023-01-15T14:34:05.085677+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:34:05.085677+0000","last_undegraded":"2023-01-15T14:34:05.085677+0000","last_fullsized":"2023-01-15T14:34:05.085677+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.31780763299999998,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":41,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.169813+0000","last_change":"2023-01-15T14:33:56.908180+0000","last_active":"2023-01-15T14:34:04.169813+0000","last_peered":"2023-01-15T14:34:04.169813+0000","last_clean":"2023-01-15T14:34:04.169813+0000","last_became_active":"2023-01-15T14:33:52.612576+0000","last_became_peered":"2023-01-15T14:33:52.612576+0000","last_unstale":"2023-01-15T14:34:04.169813+0000","last_undegraded":"2023-01-15T14:34:04.169813+0000","last_fullsized":"2023-01-15T14:34:04.169813+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003836173,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.34","version":"169'138","reported_seq":1159,"reported_epoch":354,"state":"active+clean","last_fresh":"2023-01-15T14:34:03.345965+0000","last_change":"2023-01-15T14:33:57.301685+0000","last_active":"2023-01-15T14:34:03.345965+0000","last_peered":"2023-01-15T14:34:03.345965+0000","last_clean":"2023-01-15T14:34:03.345965+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:34:03.345965+0000","last_undegraded":"2023-01-15T14:34:03.345965+0000","last_fullsized":"2023-01-15T14:34:03.345965+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.39300976100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1234,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.033541+0000","last_change":"2023-01-15T14:33:57.354727+0000","last_active":"2023-01-15T14:34:02.033541+0000","last_peered":"2023-01-15T14:34:02.033541+0000","last_clean":"2023-01-15T14:34:02.033541+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:34:02.033541+0000","last_undegraded":"2023-01-15T14:34:02.033541+0000","last_fullsized":"2023-01-15T14:34:02.033541+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.442392491,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.32","version":"355'165","reported_seq":849,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.279503+0000","last_change":"2023-01-15T14:34:05.280466+0000","last_active":"2023-01-15T14:34:05.279503+0000","last_peered":"2023-01-15T14:34:05.279503+0000","last_clean":"2023-01-15T14:34:05.279503+0000","last_became_active":"2023-01-15T14:34:04.171493+0000","last_became_peered":"2023-01-15T14:34:04.171493+0000","last_unstale":"2023-01-15T14:34:05.279503+0000","last_undegraded":"2023-01-15T14:34:05.279503+0000","last_fullsized":"2023-01-15T14:34:05.279503+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.19993219000000001,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7048,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5327885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.13","version":"348'374","reported_seq":1388,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979906+0000","last_change":"2023-01-15T14:33:58.021945+0000","last_active":"2023-01-15T14:34:04.979906+0000","last_peered":"2023-01-15T14:34:04.979906+0000","last_clean":"2023-01-15T14:34:04.979906+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:34:04.979906+0000","last_undegraded":"2023-01-15T14:34:04.979906+0000","last_fullsized":"2023-01-15T14:34:04.979906+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:29.359","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.0064123280000000001,"stat_sum":{"num_bytes":8364486,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":61489,"num_write":17,"num_write_kb":9266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.12","version":"353'186","reported_seq":904,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.168258+0000","last_change":"2023-01-15T14:34:02.715457+0000","last_active":"2023-01-15T14:34:04.168258+0000","last_peered":"2023-01-15T14:34:04.168258+0000","last_clean":"2023-01-15T14:34:04.168258+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:34:04.168258+0000","last_undegraded":"2023-01-15T14:34:04.168258+0000","last_fullsized":"2023-01-15T14:34:04.168258+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.57998335899999998,"stat_sum":{"num_bytes":7581497,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":15,"num_write_kb":5299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.11","version":"334'306","reported_seq":1546,"reported_epoch":357,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:34:07.413751+0000","last_change":"2023-01-15T14:34:07.414359+0000","last_active":"2023-01-15T14:34:07.413751+0000","last_peered":"2023-01-15T14:34:07.413751+0000","last_clean":"2023-01-15T14:33:50.016827+0000","last_became_active":"2023-01-15T14:33:53.659621+0000","last_became_peered":"2023-01-15T14:33:53.659621+0000","last_unstale":"2023-01-15T14:34:07.413751+0000","last_undegraded":"2023-01-15T14:34:07.413751+0000","last_fullsized":"2023-01-15T14:33:53.017412+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.090247347000000006,"stat_sum":{"num_bytes":10718732,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":48289,"num_write":26,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":20307357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,0],"avail_no_missing":["7","0","3","4"],"object_location_counts":[{"shards":"0,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"308'406","reported_seq":1409,"reported_epoch":356,"state":"active+recovery_wait+remapped","last_fresh":"2023-01-15T14:34:04.983850+0000","last_change":"2023-01-15T14:33:53.820604+0000","last_active":"2023-01-15T14:34:04.983850+0000","last_peered":"2023-01-15T14:34:04.983850+0000","last_clean":"2023-01-15T14:33:17.174184+0000","last_became_active":"2023-01-15T14:33:53.718934+0000","last_became_peered":"2023-01-15T14:33:53.718934+0000","last_unstale":"2023-01-15T14:34:04.983850+0000","last_undegraded":"2023-01-15T14:34:04.983850+0000","last_fullsized":"2023-01-15T14:34:04.983850+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.84251846600000002,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":41964,"num_write":46,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6670228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[1,3,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"155'145","reported_seq":1240,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979212+0000","last_change":"2023-01-15T14:34:00.245548+0000","last_active":"2023-01-15T14:34:04.979212+0000","last_peered":"2023-01-15T14:34:04.979212+0000","last_clean":"2023-01-15T14:34:04.979212+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:34:04.979212+0000","last_undegraded":"2023-01-15T14:34:04.979212+0000","last_fullsized":"2023-01-15T14:34:04.979212+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.0023104509999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.e","version":"139'38","reported_seq":29,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979082+0000","last_change":"2023-01-15T14:33:58.422638+0000","last_active":"2023-01-15T14:34:04.979082+0000","last_peered":"2023-01-15T14:34:04.979082+0000","last_clean":"2023-01-15T14:34:04.979082+0000","last_became_active":"2023-01-15T14:33:57.310759+0000","last_became_peered":"2023-01-15T14:33:57.310759+0000","last_unstale":"2023-01-15T14:34:04.979082+0000","last_undegraded":"2023-01-15T14:34:04.979082+0000","last_fullsized":"2023-01-15T14:34:04.979082+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.40586427000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.d","version":"296'232","reported_seq":1303,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.077352+0000","last_change":"2023-01-15T14:33:57.114569+0000","last_active":"2023-01-15T14:34:05.077352+0000","last_peered":"2023-01-15T14:34:05.077352+0000","last_clean":"2023-01-15T14:34:05.077352+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:34:05.077352+0000","last_undegraded":"2023-01-15T14:34:05.077352+0000","last_fullsized":"2023-01-15T14:34:05.077352+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.20767458899999999,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.c","version":"348'95","reported_seq":1096,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.011465+0000","last_change":"2023-01-15T14:33:58.326911+0000","last_active":"2023-01-15T14:34:02.011465+0000","last_peered":"2023-01-15T14:34:02.011465+0000","last_clean":"2023-01-15T14:34:02.011465+0000","last_became_active":"2023-01-15T14:33:52.210127+0000","last_became_peered":"2023-01-15T14:33:52.210127+0000","last_unstale":"2023-01-15T14:34:02.011465+0000","last_undegraded":"2023-01-15T14:34:02.011465+0000","last_fullsized":"2023-01-15T14:34:02.011465+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.421830761,"stat_sum":{"num_bytes":2221858,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":10493,"num_write":6,"num_write_kb":686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8023674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":1005,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.111620+0000","last_change":"2023-01-15T14:33:57.618325+0000","last_active":"2023-01-15T14:34:02.111620+0000","last_peered":"2023-01-15T14:34:02.111620+0000","last_clean":"2023-01-15T14:34:02.111620+0000","last_became_active":"2023-01-15T14:33:52.425622+0000","last_became_peered":"2023-01-15T14:33:52.425622+0000","last_unstale":"2023-01-15T14:34:02.111620+0000","last_undegraded":"2023-01-15T14:34:02.111620+0000","last_fullsized":"2023-01-15T14:34:02.111620+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.60540339200000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1328,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.034026+0000","last_change":"2023-01-15T14:33:57.297022+0000","last_active":"2023-01-15T14:34:02.034026+0000","last_peered":"2023-01-15T14:34:02.034026+0000","last_clean":"2023-01-15T14:34:02.034026+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:34:02.034026+0000","last_undegraded":"2023-01-15T14:34:02.034026+0000","last_fullsized":"2023-01-15T14:34:02.034026+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:48.664","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.38900274200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1113,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.136565+0000","last_change":"2023-01-15T14:33:57.304492+0000","last_active":"2023-01-15T14:34:02.136565+0000","last_peered":"2023-01-15T14:34:02.136565+0000","last_clean":"2023-01-15T14:34:02.136565+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:34:02.136565+0000","last_undegraded":"2023-01-15T14:34:02.136565+0000","last_fullsized":"2023-01-15T14:34:02.136565+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:38.874","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.29516895799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1333,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.237885+0000","last_change":"2023-01-15T14:34:00.443594+0000","last_active":"2023-01-15T14:34:02.237885+0000","last_peered":"2023-01-15T14:34:02.237885+0000","last_clean":"2023-01-15T14:34:02.237885+0000","last_became_active":"2023-01-15T14:34:00.140247+0000","last_became_peered":"2023-01-15T14:34:00.140247+0000","last_unstale":"2023-01-15T14:34:02.237885+0000","last_undegraded":"2023-01-15T14:34:02.237885+0000","last_fullsized":"2023-01-15T14:34:02.237885+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.202704782,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"1.0","version":"19'117","reported_seq":521,"reported_epoch":356,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:05.184418+0000","last_change":"2023-01-15T14:33:58.506892+0000","last_active":"2023-01-15T14:34:05.184418+0000","last_peered":"2023-01-15T14:34:05.184418+0000","last_clean":"2023-01-15T14:33:50.143204+0000","last_became_active":"2023-01-15T14:33:58.306184+0000","last_became_peered":"2023-01-15T14:33:58.306184+0000","last_unstale":"2023-01-15T14:34:05.184418+0000","last_undegraded":"2023-01-15T14:33:58.003421+0000","last_fullsized":"2023-01-15T14:33:58.001929+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[1,0],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979004+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:34:04.979004+0000","last_peered":"2023-01-15T14:34:04.979004+0000","last_clean":"2023-01-15T14:34:04.979004+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:34:04.979004+0000","last_undegraded":"2023-01-15T14:34:04.979004+0000","last_fullsized":"2023-01-15T14:34:04.979004+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":1001,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.033685+0000","last_change":"2023-01-15T14:33:57.297870+0000","last_active":"2023-01-15T14:34:02.033685+0000","last_peered":"2023-01-15T14:34:02.033685+0000","last_clean":"2023-01-15T14:34:02.033685+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:34:02.033685+0000","last_undegraded":"2023-01-15T14:34:02.033685+0000","last_fullsized":"2023-01-15T14:34:02.033685+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.38976201700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":402,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979824+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:34:04.979824+0000","last_peered":"2023-01-15T14:34:04.979824+0000","last_clean":"2023-01-15T14:34:04.979824+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:34:04.979824+0000","last_undegraded":"2023-01-15T14:34:04.979824+0000","last_fullsized":"2023-01-15T14:34:04.979824+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.964","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":716,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.167951+0000","last_change":"2023-01-15T14:33:57.185185+0000","last_active":"2023-01-15T14:34:04.167951+0000","last_peered":"2023-01-15T14:34:04.167951+0000","last_clean":"2023-01-15T14:34:04.167951+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:34:04.167951+0000","last_undegraded":"2023-01-15T14:34:04.167951+0000","last_fullsized":"2023-01-15T14:34:04.167951+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.280577087,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":155,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.108711+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:34:02.108711+0000","last_peered":"2023-01-15T14:34:02.108711+0000","last_clean":"2023-01-15T14:34:02.108711+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:34:02.108711+0000","last_undegraded":"2023-01-15T14:34:02.108711+0000","last_fullsized":"2023-01-15T14:34:02.108711+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1327,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.085414+0000","last_change":"2023-01-15T14:33:57.830460+0000","last_active":"2023-01-15T14:34:05.085414+0000","last_peered":"2023-01-15T14:34:05.085414+0000","last_clean":"2023-01-15T14:34:05.085414+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:34:05.085414+0000","last_undegraded":"2023-01-15T14:34:05.085414+0000","last_fullsized":"2023-01-15T14:34:05.085414+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.52931870400000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":225,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.084855+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:34:05.084855+0000","last_peered":"2023-01-15T14:34:05.084855+0000","last_clean":"2023-01-15T14:34:05.084855+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:34:05.084855+0000","last_undegraded":"2023-01-15T14:34:05.084855+0000","last_fullsized":"2023-01-15T14:34:05.084855+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:25.583","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"352'325","reported_seq":1423,"reported_epoch":354,"state":"active+clean","last_fresh":"2023-01-15T14:34:03.345310+0000","last_change":"2023-01-15T14:34:01.953246+0000","last_active":"2023-01-15T14:34:03.345310+0000","last_peered":"2023-01-15T14:34:03.345310+0000","last_clean":"2023-01-15T14:34:03.345310+0000","last_became_active":"2023-01-15T14:34:00.043073+0000","last_became_peered":"2023-01-15T14:34:00.043073+0000","last_unstale":"2023-01-15T14:34:03.345310+0000","last_undegraded":"2023-01-15T14:34:03.345310+0000","last_fullsized":"2023-01-15T14:34:03.345310+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:26.745","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.80204535600000004,"stat_sum":{"num_bytes":10343843,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":13875,"num_write":6,"num_write_kb":7330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9428925,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5","version":"335'158","reported_seq":1201,"reported_epoch":354,"state":"peering","last_fresh":"2023-01-15T14:34:03.058991+0000","last_change":"2023-01-15T14:34:03.059468+0000","last_active":"2023-01-15T14:34:03.058899+0000","last_peered":"2023-01-15T14:34:02.582232+0000","last_clean":"2023-01-15T14:33:55.073530+0000","last_became_active":"2023-01-15T14:33:58.221970+0000","last_became_peered":"2023-01-15T14:33:58.221970+0000","last_unstale":"2023-01-15T14:34:03.058991+0000","last_undegraded":"2023-01-15T14:34:03.058991+0000","last_fullsized":"2023-01-15T14:34:03.058991+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.71299288800000005,"stat_sum":{"num_bytes":11390101,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":26145,"num_write":9,"num_write_kb":4265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":13260921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":215,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.984667+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:34:04.984667+0000","last_peered":"2023-01-15T14:34:04.984667+0000","last_clean":"2023-01-15T14:34:04.984667+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:34:04.984667+0000","last_undegraded":"2023-01-15T14:34:04.984667+0000","last_fullsized":"2023-01-15T14:34:04.984667+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"335'308","reported_seq":1452,"reported_epoch":357,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:34:07.740687+0000","last_change":"2023-01-15T14:34:06.376733+0000","last_active":"2023-01-15T14:34:07.740687+0000","last_peered":"2023-01-15T14:34:07.740687+0000","last_clean":"2023-01-15T14:33:50.017033+0000","last_became_active":"2023-01-15T14:33:53.422817+0000","last_became_peered":"2023-01-15T14:33:53.422817+0000","last_unstale":"2023-01-15T14:34:07.740687+0000","last_undegraded":"2023-01-15T14:34:07.740687+0000","last_fullsized":"2023-01-15T14:33:53.014012+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.70443229200000002,"stat_sum":{"num_bytes":2954643,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":30684,"num_write":31,"num_write_kb":10131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":7931053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2","objects":-1},{"shards":"1,2,4","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":404,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.178007+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:34:05.178007+0000","last_peered":"2023-01-15T14:34:05.178007+0000","last_clean":"2023-01-15T14:34:05.178007+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:34:05.178007+0000","last_undegraded":"2023-01-15T14:34:05.178007+0000","last_fullsized":"2023-01-15T14:34:05.178007+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"349'141","reported_seq":1143,"reported_epoch":356,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2023-01-15T14:34:04.985389+0000","last_change":"2023-01-15T14:33:58.214638+0000","last_active":"2023-01-15T14:34:04.985389+0000","last_peered":"2023-01-15T14:34:04.985389+0000","last_clean":"2023-01-15T14:33:42.900201+0000","last_became_active":"2023-01-15T14:33:58.115294+0000","last_became_peered":"2023-01-15T14:33:58.115294+0000","last_unstale":"2023-01-15T14:34:04.985389+0000","last_undegraded":"2023-01-15T14:33:57.905179+0000","last_fullsized":"2023-01-15T14:34:04.985389+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.471195633,"stat_sum":{"num_bytes":9475691,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":13598,"num_write":13,"num_write_kb":8618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3546343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[1,3,0],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"0,1,3","objects":2},{"shards":"1,3,4","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":215,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.186729+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:34:05.186729+0000","last_peered":"2023-01-15T14:34:05.186729+0000","last_clean":"2023-01-15T14:34:05.186729+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:34:05.186729+0000","last_undegraded":"2023-01-15T14:34:05.186729+0000","last_fullsized":"2023-01-15T14:34:05.186729+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:17.360","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":41,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.169095+0000","last_change":"2023-01-15T14:33:56.909628+0000","last_active":"2023-01-15T14:34:04.169095+0000","last_peered":"2023-01-15T14:34:04.169095+0000","last_clean":"2023-01-15T14:34:04.169095+0000","last_became_active":"2023-01-15T14:33:52.206701+0000","last_became_peered":"2023-01-15T14:33:52.206701+0000","last_unstale":"2023-01-15T14:34:04.169095+0000","last_undegraded":"2023-01-15T14:34:04.169095+0000","last_fullsized":"2023-01-15T14:34:04.169095+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.0022256720000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.180246+0000","last_change":"2023-01-15T14:33:57.412557+0000","last_active":"2023-01-15T14:34:05.180246+0000","last_peered":"2023-01-15T14:34:05.180246+0000","last_clean":"2023-01-15T14:34:05.180246+0000","last_became_active":"2023-01-15T14:33:57.311981+0000","last_became_peered":"2023-01-15T14:33:57.311981+0000","last_unstale":"2023-01-15T14:34:05.180246+0000","last_undegraded":"2023-01-15T14:34:05.180246+0000","last_fullsized":"2023-01-15T14:34:05.180246+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"351'159","reported_seq":1185,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.035338+0000","last_change":"2023-01-15T14:33:58.010967+0000","last_active":"2023-01-15T14:34:02.035338+0000","last_peered":"2023-01-15T14:34:02.035338+0000","last_clean":"2023-01-15T14:34:02.035338+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:34:02.035338+0000","last_undegraded":"2023-01-15T14:34:02.035338+0000","last_fullsized":"2023-01-15T14:34:02.035338+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:32:19.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:19.303243+0000","objects_scrubbed":1,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.0010524499999999999,"stat_sum":{"num_bytes":1226927,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":18,"num_read_kb":20384,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":29,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.980053+0000","last_change":"2023-01-15T14:33:58.327664+0000","last_active":"2023-01-15T14:34:04.980053+0000","last_peered":"2023-01-15T14:34:04.980053+0000","last_clean":"2023-01-15T14:34:04.980053+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:34:04.980053+0000","last_undegraded":"2023-01-15T14:34:04.980053+0000","last_fullsized":"2023-01-15T14:34:04.980053+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31327724499999998,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.16","version":"352'129","reported_seq":642,"reported_epoch":356,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:05.186425+0000","last_change":"2023-01-15T14:33:58.219965+0000","last_active":"2023-01-15T14:34:05.186425+0000","last_peered":"2023-01-15T14:34:05.186425+0000","last_clean":"2023-01-15T14:33:55.270604+0000","last_became_active":"2023-01-15T14:33:58.218043+0000","last_became_peered":"2023-01-15T14:33:58.218043+0000","last_unstale":"2023-01-15T14:34:05.186425+0000","last_undegraded":"2023-01-15T14:33:57.906873+0000","last_fullsized":"2023-01-15T14:33:57.906218+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:09.523","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.30046122199999997,"stat_sum":{"num_bytes":7378471,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2},{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":41,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.270064+0000","last_change":"2023-01-15T14:33:57.186529+0000","last_active":"2023-01-15T14:34:04.270064+0000","last_peered":"2023-01-15T14:34:04.270064+0000","last_clean":"2023-01-15T14:34:04.270064+0000","last_became_active":"2023-01-15T14:33:52.303197+0000","last_became_peered":"2023-01-15T14:33:52.303197+0000","last_unstale":"2023-01-15T14:34:04.270064+0000","last_undegraded":"2023-01-15T14:34:04.270064+0000","last_fullsized":"2023-01-15T14:34:04.270064+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.265135238,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.18","version":"340'324","reported_seq":1514,"reported_epoch":354,"state":"activating","last_fresh":"2023-01-15T14:34:03.058948+0000","last_change":"2023-01-15T14:34:03.058948+0000","last_active":"2023-01-15T14:34:02.035428+0000","last_peered":"2023-01-15T14:34:00.950442+0000","last_clean":"2023-01-15T14:33:50.112982+0000","last_became_active":"2023-01-15T14:33:53.517870+0000","last_became_peered":"2023-01-15T14:33:53.517870+0000","last_unstale":"2023-01-15T14:34:03.058948+0000","last_undegraded":"2023-01-15T14:34:03.058948+0000","last_fullsized":"2023-01-15T14:34:03.058948+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:06.616","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.66838752300000004,"stat_sum":{"num_bytes":4332078,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":26,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":12147901,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":["3","0","1","2"],"object_location_counts":[{"shards":"0,1,3","objects":2}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"294'256","reported_seq":1326,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.184763+0000","last_change":"2023-01-15T14:33:57.626542+0000","last_active":"2023-01-15T14:34:05.184763+0000","last_peered":"2023-01-15T14:34:05.184763+0000","last_clean":"2023-01-15T14:34:05.184763+0000","last_became_active":"2023-01-15T14:33:35.259904+0000","last_became_peered":"2023-01-15T14:33:35.259904+0000","last_unstale":"2023-01-15T14:34:05.184763+0000","last_undegraded":"2023-01-15T14:34:05.184763+0000","last_fullsized":"2023-01-15T14:34:05.184763+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:01.296","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.52734117999999996,"stat_sum":{"num_bytes":2420274,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1a","version":"348'352","reported_seq":1456,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.033296+0000","last_change":"2023-01-15T14:33:58.011960+0000","last_active":"2023-01-15T14:34:02.033296+0000","last_peered":"2023-01-15T14:34:02.033296+0000","last_clean":"2023-01-15T14:34:02.033296+0000","last_became_active":"2023-01-15T14:33:35.063699+0000","last_became_peered":"2023-01-15T14:33:35.063699+0000","last_unstale":"2023-01-15T14:34:02.033296+0000","last_undegraded":"2023-01-15T14:34:02.033296+0000","last_fullsized":"2023-01-15T14:34:02.033296+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.0039007270000000001,"stat_sum":{"num_bytes":3052653,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":15,"num_read_kb":21426,"num_write":7,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1b","version":"348'99","reported_seq":65,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.168643+0000","last_change":"2023-01-15T14:33:58.271017+0000","last_active":"2023-01-15T14:34:04.168643+0000","last_peered":"2023-01-15T14:34:04.168643+0000","last_clean":"2023-01-15T14:34:04.168643+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:34:04.168643+0000","last_undegraded":"2023-01-15T14:34:04.168643+0000","last_fullsized":"2023-01-15T14:34:04.168643+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.26204159500000002,"stat_sum":{"num_bytes":4400087,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10511,"num_write":14,"num_write_kb":4701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9416119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":547,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.035023+0000","last_change":"2023-01-15T14:34:00.229670+0000","last_active":"2023-01-15T14:34:02.035023+0000","last_peered":"2023-01-15T14:34:02.035023+0000","last_clean":"2023-01-15T14:34:02.035023+0000","last_became_active":"2023-01-15T14:33:57.125876+0000","last_became_peered":"2023-01-15T14:33:57.125876+0000","last_unstale":"2023-01-15T14:34:02.035023+0000","last_undegraded":"2023-01-15T14:34:02.035023+0000","last_fullsized":"2023-01-15T14:34:02.035023+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:05.229","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.0028318649999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1269,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.134255+0000","last_change":"2023-01-15T14:33:57.405395+0000","last_active":"2023-01-15T14:34:02.134255+0000","last_peered":"2023-01-15T14:34:02.134255+0000","last_clean":"2023-01-15T14:34:02.134255+0000","last_became_active":"2023-01-15T14:33:35.362912+0000","last_became_peered":"2023-01-15T14:33:35.362912+0000","last_unstale":"2023-01-15T14:34:02.134255+0000","last_undegraded":"2023-01-15T14:34:02.134255+0000","last_fullsized":"2023-01-15T14:34:02.134255+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.394560678,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":980,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.010154+0000","last_change":"2023-01-15T14:33:57.317895+0000","last_active":"2023-01-15T14:34:02.010154+0000","last_peered":"2023-01-15T14:34:02.010154+0000","last_clean":"2023-01-15T14:34:02.010154+0000","last_became_active":"2023-01-15T14:33:35.261561+0000","last_became_peered":"2023-01-15T14:33:35.261561+0000","last_unstale":"2023-01-15T14:34:02.010154+0000","last_undegraded":"2023-01-15T14:34:02.010154+0000","last_fullsized":"2023-01-15T14:34:02.010154+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40752737100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1f","version":"318'256","reported_seq":1330,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.169757+0000","last_change":"2023-01-15T14:34:00.448154+0000","last_active":"2023-01-15T14:34:04.169757+0000","last_peered":"2023-01-15T14:34:04.169757+0000","last_clean":"2023-01-15T14:34:04.169757+0000","last_became_active":"2023-01-15T14:34:00.240936+0000","last_became_peered":"2023-01-15T14:34:00.240936+0000","last_unstale":"2023-01-15T14:34:04.169757+0000","last_undegraded":"2023-01-15T14:34:04.169757+0000","last_fullsized":"2023-01-15T14:34:04.169757+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.20345100599999999,"stat_sum":{"num_bytes":2974888,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":13341,"num_write":1,"num_write_kb":1699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":5949777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.20","version":"324'273","reported_seq":1359,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.167589+0000","last_change":"2023-01-15T14:33:57.415738+0000","last_active":"2023-01-15T14:34:04.167589+0000","last_peered":"2023-01-15T14:34:04.167589+0000","last_clean":"2023-01-15T14:34:04.167589+0000","last_became_active":"2023-01-15T14:33:55.472883+0000","last_became_peered":"2023-01-15T14:33:55.472883+0000","last_unstale":"2023-01-15T14:34:04.167589+0000","last_undegraded":"2023-01-15T14:34:04.167589+0000","last_fullsized":"2023-01-15T14:34:04.167589+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.49046371599999999,"stat_sum":{"num_bytes":4251137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10864,"num_write":9,"num_write_kb":4140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4438968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.21","version":"321'138","reported_seq":1048,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.980964+0000","last_change":"2023-01-15T14:33:57.005196+0000","last_active":"2023-01-15T14:34:04.980964+0000","last_peered":"2023-01-15T14:34:04.980964+0000","last_clean":"2023-01-15T14:34:04.980964+0000","last_became_active":"2023-01-15T14:33:45.267321+0000","last_became_peered":"2023-01-15T14:33:45.267321+0000","last_unstale":"2023-01-15T14:34:04.980964+0000","last_undegraded":"2023-01-15T14:34:04.980964+0000","last_fullsized":"2023-01-15T14:34:04.980964+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.102357873,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.22","version":"323'161","reported_seq":1335,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.209180+0000","last_change":"2023-01-15T14:33:57.619172+0000","last_active":"2023-01-15T14:34:02.209180+0000","last_peered":"2023-01-15T14:34:02.209180+0000","last_clean":"2023-01-15T14:34:02.209180+0000","last_became_active":"2023-01-15T14:33:55.378253+0000","last_became_peered":"2023-01-15T14:33:55.378253+0000","last_unstale":"2023-01-15T14:34:02.209180+0000","last_undegraded":"2023-01-15T14:34:02.209180+0000","last_fullsized":"2023-01-15T14:34:02.209180+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.50935289699999997,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":743,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.168263+0000","last_change":"2023-01-15T14:33:57.416771+0000","last_active":"2023-01-15T14:34:04.168263+0000","last_peered":"2023-01-15T14:34:04.168263+0000","last_clean":"2023-01-15T14:34:04.168263+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:34:04.168263+0000","last_undegraded":"2023-01-15T14:34:04.168263+0000","last_fullsized":"2023-01-15T14:34:04.168263+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.45831782799999998,"stat_sum":{"num_bytes":3802136,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":21818,"num_write":18,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.24","version":"349'263","reported_seq":1401,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.010832+0000","last_change":"2023-01-15T14:33:57.616353+0000","last_active":"2023-01-15T14:34:02.010832+0000","last_peered":"2023-01-15T14:34:02.010832+0000","last_clean":"2023-01-15T14:34:02.010832+0000","last_became_active":"2023-01-15T14:33:45.075309+0000","last_became_peered":"2023-01-15T14:33:45.075309+0000","last_unstale":"2023-01-15T14:34:02.010832+0000","last_undegraded":"2023-01-15T14:34:02.010832+0000","last_fullsized":"2023-01-15T14:34:02.010832+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.70814871000000001,"stat_sum":{"num_bytes":7636039,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":10774,"num_write":10,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.25","version":"333'108","reported_seq":1120,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.085780+0000","last_change":"2023-01-15T14:33:57.331159+0000","last_active":"2023-01-15T14:34:05.085780+0000","last_peered":"2023-01-15T14:34:05.085780+0000","last_clean":"2023-01-15T14:34:05.085780+0000","last_became_active":"2023-01-15T14:33:45.168598+0000","last_became_peered":"2023-01-15T14:33:45.168598+0000","last_unstale":"2023-01-15T14:34:05.085780+0000","last_undegraded":"2023-01-15T14:34:05.085780+0000","last_fullsized":"2023-01-15T14:34:05.085780+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.33264791100000002,"stat_sum":{"num_bytes":2638765,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3030,"num_write":2,"num_write_kb":2577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":472,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979581+0000","last_change":"2023-01-15T14:33:58.119558+0000","last_active":"2023-01-15T14:34:04.979581+0000","last_peered":"2023-01-15T14:34:04.979581+0000","last_clean":"2023-01-15T14:34:04.979581+0000","last_became_active":"2023-01-15T14:33:52.309110+0000","last_became_peered":"2023-01-15T14:33:52.309110+0000","last_unstale":"2023-01-15T14:34:04.979581+0000","last_undegraded":"2023-01-15T14:34:04.979581+0000","last_fullsized":"2023-01-15T14:34:04.979581+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.0016362760000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":1067,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.110635+0000","last_change":"2023-01-15T14:33:57.355572+0000","last_active":"2023-01-15T14:34:02.110635+0000","last_peered":"2023-01-15T14:34:02.110635+0000","last_clean":"2023-01-15T14:34:02.110635+0000","last_became_active":"2023-01-15T14:33:45.171090+0000","last_became_peered":"2023-01-15T14:33:45.171090+0000","last_unstale":"2023-01-15T14:34:02.110635+0000","last_undegraded":"2023-01-15T14:34:02.110635+0000","last_fullsized":"2023-01-15T14:34:02.110635+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.443073948,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.28","version":"333'248","reported_seq":1385,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.037989+0000","last_change":"2023-01-15T14:33:57.617361+0000","last_active":"2023-01-15T14:34:02.037989+0000","last_peered":"2023-01-15T14:34:02.037989+0000","last_clean":"2023-01-15T14:34:02.037989+0000","last_became_active":"2023-01-15T14:33:45.366831+0000","last_became_peered":"2023-01-15T14:33:45.366831+0000","last_unstale":"2023-01-15T14:34:02.037989+0000","last_undegraded":"2023-01-15T14:34:02.037989+0000","last_fullsized":"2023-01-15T14:34:02.037989+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.60974191300000002,"stat_sum":{"num_bytes":4838389,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8969,"num_write":8,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":6096764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":889,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.109340+0000","last_change":"2023-01-15T14:33:57.620313+0000","last_active":"2023-01-15T14:34:02.109340+0000","last_peered":"2023-01-15T14:34:02.109340+0000","last_clean":"2023-01-15T14:34:02.109340+0000","last_became_active":"2023-01-15T14:33:45.276960+0000","last_became_peered":"2023-01-15T14:33:45.276960+0000","last_unstale":"2023-01-15T14:34:02.109340+0000","last_undegraded":"2023-01-15T14:34:02.109340+0000","last_fullsized":"2023-01-15T14:34:02.109340+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.51033217900000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1238,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.078722+0000","last_change":"2023-01-15T14:33:56.904528+0000","last_active":"2023-01-15T14:34:05.078722+0000","last_peered":"2023-01-15T14:34:05.078722+0000","last_clean":"2023-01-15T14:34:05.078722+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:34:05.078722+0000","last_undegraded":"2023-01-15T14:34:05.078722+0000","last_fullsized":"2023-01-15T14:34:05.078722+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.0022419219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":51,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.168392+0000","last_change":"2023-01-15T14:33:57.893154+0000","last_active":"2023-01-15T14:34:04.168392+0000","last_peered":"2023-01-15T14:34:04.168392+0000","last_clean":"2023-01-15T14:34:04.168392+0000","last_became_active":"2023-01-15T14:33:57.261456+0000","last_became_peered":"2023-01-15T14:33:57.261456+0000","last_unstale":"2023-01-15T14:34:04.168392+0000","last_undegraded":"2023-01-15T14:34:04.168392+0000","last_fullsized":"2023-01-15T14:34:04.168392+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.630651882,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2c","version":"141'46","reported_seq":438,"reported_epoch":353,"state":"active+clean","last_fresh":"2023-01-15T14:34:02.034524+0000","last_change":"2023-01-15T14:33:57.303687+0000","last_active":"2023-01-15T14:34:02.034524+0000","last_peered":"2023-01-15T14:34:02.034524+0000","last_clean":"2023-01-15T14:34:02.034524+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:34:02.034524+0000","last_undegraded":"2023-01-15T14:34:02.034524+0000","last_fullsized":"2023-01-15T14:34:02.034524+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:10.970","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.29451823399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1211,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.077646+0000","last_change":"2023-01-15T14:33:58.122688+0000","last_active":"2023-01-15T14:34:05.077646+0000","last_peered":"2023-01-15T14:34:05.077646+0000","last_clean":"2023-01-15T14:34:05.077646+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:34:05.077646+0000","last_undegraded":"2023-01-15T14:34:05.077646+0000","last_fullsized":"2023-01-15T14:34:05.077646+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.105464771,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":29,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.978910+0000","last_change":"2023-01-15T14:33:58.326712+0000","last_active":"2023-01-15T14:34:04.978910+0000","last_peered":"2023-01-15T14:34:04.978910+0000","last_clean":"2023-01-15T14:34:04.978910+0000","last_became_active":"2023-01-15T14:33:57.210434+0000","last_became_peered":"2023-01-15T14:33:57.210434+0000","last_unstale":"2023-01-15T14:34:04.978910+0000","last_undegraded":"2023-01-15T14:34:04.978910+0000","last_fullsized":"2023-01-15T14:34:04.978910+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.31032643199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1105,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.081137+0000","last_change":"2023-01-15T14:33:58.121127+0000","last_active":"2023-01-15T14:34:05.081137+0000","last_peered":"2023-01-15T14:34:05.081137+0000","last_clean":"2023-01-15T14:34:05.081137+0000","last_became_active":"2023-01-15T14:33:55.174128+0000","last_became_peered":"2023-01-15T14:33:55.174128+0000","last_unstale":"2023-01-15T14:34:05.081137+0000","last_undegraded":"2023-01-15T14:34:05.081137+0000","last_fullsized":"2023-01-15T14:34:05.081137+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.20511622600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":981,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.167692+0000","last_change":"2023-01-15T14:33:57.314582+0000","last_active":"2023-01-15T14:34:04.167692+0000","last_peered":"2023-01-15T14:34:04.167692+0000","last_clean":"2023-01-15T14:34:04.167692+0000","last_became_active":"2023-01-15T14:33:55.373498+0000","last_became_peered":"2023-01-15T14:33:55.373498+0000","last_unstale":"2023-01-15T14:34:04.167692+0000","last_undegraded":"2023-01-15T14:34:04.167692+0000","last_fullsized":"2023-01-15T14:34:04.167692+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":7,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.389510301,"stat_sum":{"num_bytes":5419866,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":25948,"num_write":35,"num_write_kb":13827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":13340458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.31","version":"204'150","reported_seq":1224,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.084337+0000","last_change":"2023-01-15T14:33:57.625989+0000","last_active":"2023-01-15T14:34:05.084337+0000","last_peered":"2023-01-15T14:34:05.084337+0000","last_clean":"2023-01-15T14:34:05.084337+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:34:05.084337+0000","last_undegraded":"2023-01-15T14:34:05.084337+0000","last_fullsized":"2023-01-15T14:34:05.084337+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.52721113600000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":106,"stat_sum":{"num_bytes":355846432,"num_objects":164,"num_object_clones":113,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":91,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":14,"num_read":1017,"num_read_kb":1099202,"num_write":808,"num_write_kb":331645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":612,"num_bytes_recovered":385911300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18696,"ondisk_log_size":18696,"up":318,"acting":312,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574922,"num_pgs":63,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1047320,"kb_used_data":92664956,"kb_used_omap":7766,"kb_used_meta":0,"kb_avail":92652060,"statfs":{"total":95948165120,"available":94875709440,"internally_reserved":13205504,"allocated":94888914944,"data_stored":94888914944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7952640,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":4.1269999999999998,"15min":4.0890000000000004},"min":{"1min":0.54000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2030000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":9.5380000000000003,"5min":6.3200000000000003,"15min":8.2460000000000004},"min":{"1min":0.58799999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":96.337000000000003,"5min":96.337000000000003,"15min":96.337000000000003},"last":1.0860000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":3.6509999999999998,"15min":5.5800000000000001},"min":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":34.585999999999999,"5min":88.003,"15min":88.003},"last":1.038},{"interface":"front","average":{"1min":13.302,"5min":8.1950000000000003,"15min":6.9180000000000001},"min":{"1min":0.58799999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.403999999999996,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.91000000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":6.9400000000000004,"15min":5.5069999999999997},"min":{"1min":0.57199999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.179,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":2.5790000000000002,"15min":3.645},"min":{"1min":0.626,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":91.837999999999994,"15min":91.837999999999994},"last":1.143}]},{"osd":3,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":7.3799999999999999,"15min":3.0920000000000001},"min":{"1min":0.63600000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2370000000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":1.173},{"interface":"front","average":{"1min":0.91600000000000004,"5min":3.6360000000000001,"15min":4.3129999999999997},"min":{"1min":0.65000000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.2270000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":1.266}]},{"osd":4,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5519999999999996,"5min":5.6219999999999999,"15min":8.1509999999999998},"min":{"1min":0.63500000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":96.185000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.9259999999999999,"5min":5.782,"15min":2.5699999999999998},"min":{"1min":0.73099999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":22.41,"5min":164.21199999999999,"15min":164.21199999999999},"last":1.3069999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.6440000000000001,"5min":5.0659999999999998,"15min":5.9059999999999997},"min":{"1min":0.72799999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":96.263999999999996,"5min":96.263999999999996,"15min":96.263999999999996},"last":1.339},{"interface":"front","average":{"1min":0.998,"5min":3.206,"15min":4.3650000000000002},"min":{"1min":0.69299999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.355,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.3640000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":5.5949999999999998,"15min":7.5469999999999997},"min":{"1min":0.83599999999999997,"5min":0.496,"15min":0.496},"max":{"1min":96.299999999999997,"5min":96.299999999999997,"15min":96.299999999999997},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.089,"5min":2.5150000000000001,"15min":1.571},"min":{"1min":0.84299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.2789999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.409}]}]},{"osd":6,"up_from":0,"seq":55834574923,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574923,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574921,"num_pgs":69,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1038680,"kb_used_data":92663056,"kb_used_omap":13794,"kb_used_meta":0,"kb_avail":92660700,"statfs":{"total":95948165120,"available":94884556800,"internally_reserved":2412544,"allocated":94886969344,"data_stored":94886969344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14125154,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":23.594000000000001,"15min":11.141},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.133,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.78300000000000003,"5min":3.0009999999999999,"15min":1.5209999999999999},"min":{"1min":0.58299999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.024,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.86099999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":5.0730000000000004,"15min":7.4199999999999999},"min":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6659999999999999,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.343,"15min":1.9770000000000001},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0529999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.96899999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":24.227,"15min":11.749000000000001},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.157,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":1.0580000000000001},{"interface":"front","average":{"1min":2.6040000000000001,"5min":1.21,"15min":1.01},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":32.081000000000003,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.89600000000000002}]},{"osd":3,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":2.9169999999999998,"15min":1.865},"min":{"1min":0.65300000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.2909999999999999,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.7370000000000001,"5min":24.498999999999999,"15min":12.266999999999999},"min":{"1min":0.64400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":14.961,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":58.317}]},{"osd":5,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.88500000000000001,"15min":0.90800000000000003},"min":{"1min":0.55200000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":2.1840000000000002,"15min":2.2599999999999998},"min":{"1min":0.71799999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3200000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":1.024}]},{"osd":6,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":3.9329999999999998,"15min":5.093},"min":{"1min":0.69599999999999995,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.365,"5min":99.780000000000001,"15min":99.780000000000001},"last":1.153},{"interface":"front","average":{"1min":1.0489999999999999,"5min":22.463999999999999,"15min":10.93},"min":{"1min":0.71599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.427,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.095}]},{"osd":7,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":3.1539999999999999,"5min":5.5570000000000004,"15min":10.603999999999999},"min":{"1min":0.76700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":39.012999999999998,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.2370000000000001,"5min":24.864000000000001,"15min":15.779999999999999},"min":{"1min":0.877,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.0680000000000001,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":0.92400000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574921,"num_pgs":65,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1101584,"kb_used_data":92597840,"kb_used_omap":14047,"kb_used_meta":0,"kb_avail":92597796,"statfs":{"total":95948165120,"available":94820143104,"internally_reserved":45056,"allocated":94820188160,"data_stored":94820188160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14384374,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":5,"commit_latency_ns":0,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":7.4400000000000004,"15min":5.7119999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":17.731000000000002,"5min":174.52199999999999,"15min":174.52199999999999},"last":58.351999999999997},{"interface":"front","average":{"1min":0.75800000000000001,"5min":3.4340000000000002,"15min":6.2389999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.9260000000000002,"5min":93.674000000000007,"15min":93.674000000000007},"last":58.323999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":7.1790000000000003,"15min":11.414},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96799999999999997,"5min":159.256,"15min":159.256},"last":0.39400000000000002},{"interface":"front","average":{"1min":2.6259999999999999,"5min":10.417999999999999,"15min":3.9660000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":17.826000000000001,"5min":174.578,"15min":174.578},"last":0.41899999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":4.6849999999999996,"5min":2.8809999999999998,"15min":3.1930000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":86.412000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.46600000000000003},{"interface":"front","average":{"1min":1.95,"5min":6.7889999999999997,"15min":3.7250000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.876000000000001,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.57399999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":7.6879999999999997,"5min":29.686,"15min":14.27},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":84.640000000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.86899999999999999,"5min":32.134999999999998,"15min":19.960999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.615,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":0.63400000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":6.4980000000000002,"15min":11.433999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.04,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":2.2309999999999999,"5min":6.0039999999999996,"15min":2.6819999999999999},"min":{"1min":0.70099999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":17.917000000000002,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.90200000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1949999999999998,"5min":5.9980000000000002,"15min":2.6869999999999998},"min":{"1min":0.66000000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":17.989000000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.964,"15min":3.1619999999999999},"min":{"1min":0.70499999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.23,"5min":60.098999999999997,"15min":60.098999999999997},"last":0.76600000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":2.0219999999999998,"15min":3.3639999999999999},"min":{"1min":0.69499999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.266,"5min":60.165999999999997,"15min":60.165999999999997},"last":0.873},{"interface":"front","average":{"1min":19.138000000000002,"5min":11.705,"15min":15.961},"min":{"1min":0.66200000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":281.13400000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":0.79100000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574922,"num_pgs":50,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1064524,"kb_used_data":92634996,"kb_used_omap":11690,"kb_used_meta":0,"kb_avail":92634856,"statfs":{"total":95948165120,"available":94858092544,"internally_reserved":143360,"allocated":94858235904,"data_stored":94858235904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11970995,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":3.109,"5min":3.5230000000000001,"15min":1.601},"min":{"1min":0.437,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":51.009999999999998,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.621},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.4299999999999999,"15min":0.95299999999999996},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0089999999999999,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.64900000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.1529999999999996,"5min":5.9059999999999997,"15min":2.5009999999999999},"min":{"1min":0.378,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":91.525999999999996,"5min":143.071,"15min":143.071},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":2.016,"15min":2.9089999999999998},"min":{"1min":0.42299999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.131,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.55300000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":8.2210000000000001,"15min":9.6750000000000007},"min":{"1min":0.5,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.3609999999999998,"5min":183.53,"15min":183.53},"last":0.67300000000000004},{"interface":"front","average":{"1min":16.692,"5min":7.5170000000000003,"15min":3.0169999999999999},"min":{"1min":0.61199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":155.47900000000001,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.70799999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3650000000000002,"5min":6.5060000000000002,"15min":4.6020000000000003},"min":{"1min":0.67300000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":51.259999999999998,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":2.8809999999999998,"5min":5.2939999999999996,"15min":4.3040000000000003},"min":{"1min":0.48699999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":33.219999999999999,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.83199999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":2.4729999999999999,"15min":1.6080000000000001},"min":{"1min":0.57199999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2130000000000001,"5min":143.262,"15min":143.262},"last":1.02},{"interface":"front","average":{"1min":3.4780000000000002,"5min":3.0539999999999998,"15min":1.792},"min":{"1min":0.70999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":51.326000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.80600000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.857,"15min":1.3069999999999999},"min":{"1min":0.60399999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.365,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.018,"5min":2.6160000000000001,"15min":1.6839999999999999},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.5,"5min":143.215,"15min":143.215},"last":1.069}]},{"osd":7,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":13.092000000000001,"5min":6.3120000000000003,"15min":4.1429999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":120.905,"5min":120.905,"15min":120.905},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":2.8450000000000002,"15min":2.7989999999999999},"min":{"1min":0.66600000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.552,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.96499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":72,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1063640,"kb_used_data":92635740,"kb_used_omap":13180,"kb_used_meta":0,"kb_avail":92635740,"statfs":{"total":95948165120,"available":94858997760,"internally_reserved":0,"allocated":94858997760,"data_stored":94858997760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13496648,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":3.633,"15min":3.806},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":3.9140000000000001,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":7.742,"5min":4.6130000000000004,"15min":4.4189999999999996},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":119.39100000000001,"5min":119.39100000000001,"15min":119.39100000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":2.6469999999999998,"15min":1.5269999999999999},"min":{"1min":0.29799999999999999,"5min":0.245,"15min":0.245},"max":{"1min":95.879999999999995,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.8620000000000001,"15min":3.3359999999999999},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.9780000000000002,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.67700000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.8920000000000003,"5min":9.6750000000000007,"15min":16.449000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":100.137,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":7.782,"5min":6.8899999999999997,"15min":7.1260000000000003},"min":{"1min":0.45600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":119.804,"5min":119.804,"15min":119.804},"last":0.94299999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:33:17 2023","interfaces":[{"interface":"back","average":{"1min":7.9249999999999998,"5min":17.140999999999998,"15min":8.8650000000000002},"min":{"1min":0.504,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":119.71899999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":1.278},{"interface":"front","average":{"1min":1.7849999999999999,"5min":15.526,"15min":7.5780000000000003},"min":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":16.989999999999998,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":15.141}]},{"osd":5,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.5449999999999999,"15min":1.2030000000000001},"min":{"1min":0.59899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.278,"5min":65.120999999999995,"15min":65.120999999999995},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.6579999999999999,"15min":3.5110000000000001},"min":{"1min":0.60599999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":4.0780000000000003,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.0820000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":8.0069999999999997,"5min":5.0129999999999999,"15min":5.0049999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":119.759,"5min":119.759,"15min":119.759},"last":1.05},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.603,"15min":1.2829999999999999},"min":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.349,"5min":65.156000000000006,"15min":65.156000000000006},"last":1.014}]},{"osd":7,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":3.6030000000000002,"5min":6.1109999999999998,"15min":6.7149999999999999},"min":{"1min":0.56000000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":31.620999999999999,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":8.0779999999999994,"5min":5.0060000000000002,"15min":4.835},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":119.65300000000001,"5min":119.65300000000001,"15min":119.65300000000001},"last":1.2070000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574923,"num_pgs":70,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1045672,"kb_used_data":92687888,"kb_used_omap":7766,"kb_used_meta":0,"kb_avail":92653708,"statfs":{"total":95948165120,"available":94877396992,"internally_reserved":35000320,"allocated":94912397312,"data_stored":94912397312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7953103,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":3.1219999999999999,"15min":1.456},"min":{"1min":0.40899999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1060000000000001,"5min":69.872,"15min":69.872},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":3.2309999999999999,"15min":1.5629999999999999},"min":{"1min":0.51300000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.2350000000000001,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.52800000000000002}]},{"osd":2,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":3.4220000000000002,"15min":1.603},"min":{"1min":0.46999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.444,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":2.6560000000000001,"15min":1.3580000000000001},"min":{"1min":0.52000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.302,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.83799999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":17.571999999999999,"5min":10.789,"15min":14.782},"min":{"1min":0.53300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":199.08000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.835,"5min":7.306,"15min":2.9689999999999999},"min":{"1min":0.56599999999999995,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":15.601000000000001,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.89200000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":16.695,"15min":6.2220000000000004},"min":{"1min":0.73199999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1879999999999999,"5min":1486.309,"15min":1486.309},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.038,"5min":17.925000000000001,"15min":6.6580000000000004},"min":{"1min":0.82499999999999996,"5min":0.439,"15min":0.439},"max":{"1min":1.3759999999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":1.115}]},{"osd":5,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":4.6749999999999998,"15min":8.0899999999999999},"min":{"1min":0.62,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.304,"5min":176.34399999999999,"15min":176.34399999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.008,"5min":1.73,"15min":1.2689999999999999},"min":{"1min":0.69199999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2529999999999999,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.99299999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":3.5910000000000002,"15min":1.9770000000000001},"min":{"1min":0.67800000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5960000000000001,"5min":70.259,"15min":70.259},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.0880000000000001,"5min":3.0209999999999999,"15min":1.7629999999999999},"min":{"1min":0.83699999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5489999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":1.141}]},{"osd":7,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":6.1200000000000001,"15min":12.487},"min":{"1min":0.78400000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.48,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.091},{"interface":"front","average":{"1min":1.117,"5min":3.6070000000000002,"15min":1.9830000000000001},"min":{"1min":0.76100000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5229999999999999,"5min":70.322000000000003,"15min":70.322000000000003},"last":1.0509999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:34:08.715 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:34:08.716 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:08.227+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:08.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-01-15T14:34:08.729 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:34:08.731 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:08.724+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:08.745 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:34:08.831 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:08.828+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:08.886 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:08.886 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:08.886 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:34:08.934 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:08.928+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:09.041 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:09.041 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:09.041 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:09.041 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:09.041 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:09.041 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:34:09.042 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:34:09.042 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:09.042 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:09.038+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:09.046 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:09.050 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:34:09.067 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:09.089 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:09.204 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:09.206 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:34:09.206 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:34:09.206 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:34:09.206 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:34:09.207 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:34:09.207 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:34:09.207 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:34:09.207 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:34:09.208 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:09.693 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:34:09.694 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:34:09.694 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:34:09.694 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:09.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:09.695 INFO:teuthology.orchestra.run.smithi090.stdout:{"fsid":"1c77fb2b-d1a7-4884-a24e-3935e16d0b6b","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 158/498 objects degraded (31.727%), 10 pgs degraded","count":10},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":371,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":359,"num_osds":8,"num_up_osds":8,"osd_up_since":1673792884,"num_in_osds":6,"osd_in_since":1673793049,"num_remapped_pgs":9},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":100},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovery_wait+degraded+remapped","count":2},{"state_name":"active+recovery_wait+remapped","count":1},{"state_name":"active+recovering+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"activating","count":1},{"state_name":"peering","count":1},{"state_name":"active+recovery_wait+degraded","count":1}],"num_pgs":115,"num_pools":3,"num_objects":166,"data_bytes":356436800,"bytes_used":6532026368,"bytes_avail":569156964352,"bytes_total":575688990720,"inactive_pgs_ratio":0.017391303554177284,"degraded_objects":158,"degraded_total":498,"degraded_ratio":0.31726907630522089,"misplaced_objects":50,"misplaced_total":498,"misplaced_ratio":0.10040160642570281,"recovering_objects_per_sec":8,"recovering_bytes_per_sec":8632892,"recovering_keys_per_sec":0,"num_objects_recovered":34,"num_bytes_recovered":34547866,"num_keys_recovered":0,"read_bytes_sec":511,"write_bytes_sec":1180370,"read_op_per_sec":0,"write_op_per_sec":2},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2023-01-15T14:29:06.857309+0000","services":{}},"progress_events":{}} 2023-01-15T14:34:09.696 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2023-01-15T14:34:09.696 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:34:09.697 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:09.202+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:09.701 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:09.694+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:09.798 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:09.792+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:09.860 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:09.860 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:34:09.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:54 3.1 3:883e9a29:::smithi09025264-14:head [write 2011207~207284] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.785740987, 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50036369199999997, 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:09.861 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 54 2023-01-15T14:34:09.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.862 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:09.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:09.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:34:09.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:09.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:09.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061265+0000", 2023-01-15T14:34:09.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:09.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:09.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061264+0000", 2023-01-15T14:34:09.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:09.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:09.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061510+0000", 2023-01-15T14:34:09.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024575399999999999 2023-01-15T14:34:09.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:09.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061527+0000", 2023-01-15T14:34:09.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.721e-05 2023-01-15T14:34:09.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:09.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061535+0000", 2023-01-15T14:34:09.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6939999999999995e-06 2023-01-15T14:34:09.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:09.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.364890+0000", 2023-01-15T14:34:09.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303355243 2023-01-15T14:34:09.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:09.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.365051+0000", 2023-01-15T14:34:09.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016086099999999999 2023-01-15T14:34:09.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457482+0000", 2023-01-15T14:34:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092431252000000005 2023-01-15T14:34:09.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:09.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457594+0000", 2023-01-15T14:34:09.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112086 2023-01-15T14:34:09.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:09.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.457831+0000", 2023-01-15T14:34:09.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023700699999999999 2023-01-15T14:34:09.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:09.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458364+0000", 2023-01-15T14:34:09.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000533052 2023-01-15T14:34:09.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:09.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458379+0000", 2023-01-15T14:34:09.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.452e-05 2023-01-15T14:34:09.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:09.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458521+0000", 2023-01-15T14:34:09.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014200000000000001 2023-01-15T14:34:09.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:09.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.458833+0000", 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312699 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465769+0000", 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0069355529999999997 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561496+0000", 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095727111000000004 2023-01-15T14:34:09.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:09.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561550+0000", 2023-01-15T14:34:09.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3817999999999999e-05 2023-01-15T14:34:09.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:09.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561629+0000", 2023-01-15T14:34:09.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8665000000000007e-05 2023-01-15T14:34:09.878 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:09.878 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:09.879 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:09.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.879 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:09.879 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:34:09.879 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.78546073899997, 2023-01-15T14:34:09.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:34:09.879 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:09.879 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:09.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:09.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:09.880 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:09.880 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:34:09.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.880 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:09.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:09.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:34:09.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:09.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:09.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:34:09.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:34:09.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:09.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:34:09.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:34:09.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:09.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:34:09.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:34:09.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:09.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:34:09.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:34:09.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:09.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:34:09.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:34:09.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:09.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:34:09.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:34:09.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:09.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:34:09.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:34:09.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:09.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:34:09.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:34:09.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:09.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:34:09.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:34:09.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:09.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:34:09.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:34:09.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:09.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:34:09.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:34:09.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:09.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:34:09.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.78542812199998, 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:09.890 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:09.891 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:09.891 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:34:09.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.891 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:09.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:09.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:34:09.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:09.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:09.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:34:09.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:09.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:09.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:34:09.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:09.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:09.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:34:09.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:34:09.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:09.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:34:09.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:34:09.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:09.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:34:09.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:34:09.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:09.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:34:09.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:34:09.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:09.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:34:09.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:34:09.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:09.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:34:09.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:34:09.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:09.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:34:09.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:34:09.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:09.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:34:09.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:34:09.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:09.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:34:09.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:34:09.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:09.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:34:09.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:34:09.898 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:09.898 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:09.898 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:09.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.899 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:59 3.1 3:86bba67f:::smithi09025264-15:head [setxattr _header (56) in=7b,truncate 2235298] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:09.899 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:34:09.899 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.68411013600002, 2023-01-15T14:34:09.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49952147299999999, 2023-01-15T14:34:09.900 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:09.900 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:09.900 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:09.900 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:09.900 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:09.900 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 59 2023-01-15T14:34:09.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.900 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:09.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162896+0000", 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:09.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162895+0000", 2023-01-15T14:34:09.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:09.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:09.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162902+0000", 2023-01-15T14:34:09.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9199999999999998e-06 2023-01-15T14:34:09.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:09.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162910+0000", 2023-01-15T14:34:09.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6370000000000005e-06 2023-01-15T14:34:09.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:09.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162918+0000", 2023-01-15T14:34:09.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1629999999999999e-06 2023-01-15T14:34:09.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:09.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.390987+0000", 2023-01-15T14:34:09.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.2280693 2023-01-15T14:34:09.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:09.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.391202+0000", 2023-01-15T14:34:09.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021479499999999999 2023-01-15T14:34:09.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:09.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459594+0000", 2023-01-15T14:34:09.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.068392174999999999 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459706+0000", 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112649 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.459915+0000", 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020853199999999999 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460189+0000", 2023-01-15T14:34:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027383000000000003 2023-01-15T14:34:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460207+0000", 2023-01-15T14:34:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7940000000000001e-05 2023-01-15T14:34:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:09.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:09.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460271+0000", 2023-01-15T14:34:10.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3969999999999999e-05 2023-01-15T14:34:10.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:10.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460401+0000", 2023-01-15T14:34:10.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012988600000000001 2023-01-15T14:34:10.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.465866+0000", 2023-01-15T14:34:10.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0054650740000000003 2023-01-15T14:34:10.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662091+0000", 2023-01-15T14:34:10.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.196225275 2023-01-15T14:34:10.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:10.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662244+0000", 2023-01-15T14:34:10.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015354400000000001 2023-01-15T14:34:10.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662417+0000", 2023-01-15T14:34:10.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000172683 2023-01-15T14:34:10.015 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.015 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.015 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.016 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:60 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:10.016 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:34:10.016 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.68408013300001, 2023-01-15T14:34:10.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50017562599999998, 2023-01-15T14:34:10.016 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.016 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:10.016 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.016 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.017 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.017 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 60 2023-01-15T14:34:10.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.017 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:34:10.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162926+0000", 2023-01-15T14:34:10.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162925+0000", 2023-01-15T14:34:10.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:10.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162930+0000", 2023-01-15T14:34:10.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6800000000000001e-06 2023-01-15T14:34:10.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162939+0000", 2023-01-15T14:34:10.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:34:10.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.162945+0000", 2023-01-15T14:34:10.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.84e-06 2023-01-15T14:34:10.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392572+0000", 2023-01-15T14:34:10.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22962795499999999 2023-01-15T14:34:10.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.392681+0000", 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108564 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460296+0000", 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067614513000000001 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.460384+0000", 2023-01-15T14:34:10.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8695000000000004e-05 2023-01-15T14:34:10.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662651+0000", 2023-01-15T14:34:10.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20226714600000001 2023-01-15T14:34:10.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.662864+0000", 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021268400000000001 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663101+0000", 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023726500000000001 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.023 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:10.024 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:34:10.024 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.16907156799999, 2023-01-15T14:34:10.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:34:10.024 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.024 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:10.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.024 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.024 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:34:10.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.024 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:34:10.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:34:10.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:34:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:34:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:34:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:34:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:34:10.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:10.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:34:10.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:34:10.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:10.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:34:10.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:34:10.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:10.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:34:10.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:34:10.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:10.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:34:10.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:34:10.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:34:10.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:34:10.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:34:10.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:34:10.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.034 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:10.034 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:34:10.034 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.16858921400001, 2023-01-15T14:34:10.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:34:10.034 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.034 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:10.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.034 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.034 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:34:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:34:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:34:10.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:34:10.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:10.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:34:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:34:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:34:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:34:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:34:10.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:34:10.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:34:10.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:34:10.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:34:10.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:34:10.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:34:10.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:34:10.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:34:10.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:34:10.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:10.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:34:10.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:34:10.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:10.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:34:10.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:34:10.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:10.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:34:10.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:34:10.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:10.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:34:10.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:34:10.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:10.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:34:10.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:34:10.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:34:10.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:34:10.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:34:10.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:34:10.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:10.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:34:10.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:34:10.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:34:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:34:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:34:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.16853665100001, 2023-01-15T14:34:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:34:10.045 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.046 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:10.046 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.046 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.046 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.046 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:34:10.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.046 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:34:10.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:10.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:34:10.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:34:10.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:34:10.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:34:10.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:34:10.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:34:10.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:34:10.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:34:10.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:34:10.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:34:10.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:34:10.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.052 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:364 3.1 3:8889e450:::smithi09025264-32:head [copy-from ver 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:34:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:34:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 329.79733494700002, 2023-01-15T14:34:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41350888200000002, 2023-01-15T14:34:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.053 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 364 2023-01-15T14:34:10.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.054 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:34:10.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049671+0000", 2023-01-15T14:34:10.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049668+0000", 2023-01-15T14:34:10.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:34:10.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049684+0000", 2023-01-15T14:34:10.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5611e-05 2023-01-15T14:34:10.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049719+0000", 2023-01-15T14:34:10.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5777e-05 2023-01-15T14:34:10.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049740+0000", 2023-01-15T14:34:10.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:34:10.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150042+0000", 2023-01-15T14:34:10.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100301155 2023-01-15T14:34:10.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.150200+0000", 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015819200000000001 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.299624+0000", 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.14942408200000001 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:10.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300817+0000", 2023-01-15T14:34:10.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011927489999999999 2023-01-15T14:34:10.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:10.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.300851+0000", 2023-01-15T14:34:10.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4193999999999998e-05 2023-01-15T14:34:10.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301784+0000", 2023-01-15T14:34:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00093318600000000002 2023-01-15T14:34:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301895+0000", 2023-01-15T14:34:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011053299999999999 2023-01-15T14:34:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.405069+0000", 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10317467399999999 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.462926+0000", 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.057856711999999998 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463024+0000", 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8366e-05 2023-01-15T14:34:10.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463180+0000", 2023-01-15T14:34:10.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015547299999999999 2023-01-15T14:34:10.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.062 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.062 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.062 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:365 3.1 3:8889e450:::smithi09025264-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:34:10.062 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 329.797218273, 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.413735776, 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 365 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:34:10.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049788+0000", 2023-01-15T14:34:10.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049786+0000", 2023-01-15T14:34:10.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:10.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049794+0000", 2023-01-15T14:34:10.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6059999999999997e-06 2023-01-15T14:34:10.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049813+0000", 2023-01-15T14:34:10.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.908e-05 2023-01-15T14:34:10.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.049912+0000", 2023-01-15T14:34:10.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8814999999999995e-05 2023-01-15T14:34:10.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250597+0000", 2023-01-15T14:34:10.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20068465199999999 2023-01-15T14:34:10.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:34:10.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.250683+0000", 2023-01-15T14:34:10.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6159e-05 2023-01-15T14:34:10.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301041+0000", 2023-01-15T14:34:10.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.050358170000000001 2023-01-15T14:34:10.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.301193+0000", 2023-01-15T14:34:10.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015212599999999999 2023-01-15T14:34:10.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463245+0000", 2023-01-15T14:34:10.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16205191199999999 2023-01-15T14:34:10.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463371+0000", 2023-01-15T14:34:10.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012614599999999999 2023-01-15T14:34:10.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:40.463523+0000", 2023-01-15T14:34:10.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152299 2023-01-15T14:34:10.073 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.073 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.074 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:469 3.1 3:86bba67f:::smithi09025264-15:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2023-01-15T14:34:10.074 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:34:10.074 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 322.07108599700001, 2023-01-15T14:34:10.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49484308700000001, 2023-01-15T14:34:10.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:10.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 469 2023-01-15T14:34:10.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.075 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:34:10.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775920+0000", 2023-01-15T14:34:10.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775919+0000", 2023-01-15T14:34:10.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:10.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775922+0000", 2023-01-15T14:34:10.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:34:10.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775931+0000", 2023-01-15T14:34:10.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0389999999999993e-06 2023-01-15T14:34:10.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.775936+0000", 2023-01-15T14:34:10.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3499999999999999e-06 2023-01-15T14:34:10.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969000+0000", 2023-01-15T14:34:10.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19306389400000001 2023-01-15T14:34:10.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:47.969189+0000", 2023-01-15T14:34:10.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018922500000000001 2023-01-15T14:34:10.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270401+0000", 2023-01-15T14:34:10.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30121261199999999 2023-01-15T14:34:10.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270564+0000", 2023-01-15T14:34:10.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000163058 2023-01-15T14:34:10.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:48.270763+0000", 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019849900000000001 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:880 3.1 3:8b0a2647:::smithi09025264-8:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 285.85480237100001, 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50627262500000003, 2023-01-15T14:34:10.083 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.084 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:10.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.084 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.084 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 880 2023-01-15T14:34:10.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.084 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:34:10.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992203+0000", 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992210+0000", 2023-01-15T14:34:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6459999999999993e-06 2023-01-15T14:34:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:23.992223+0000", 2023-01-15T14:34:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2656999999999999e-05 2023-01-15T14:34:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092372+0000", 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100148797 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092520+0000", 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014764300000000001 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:10.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.092545+0000", 2023-01-15T14:34:10.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5616999999999999e-05 2023-01-15T14:34:10.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097389+0000", 2023-01-15T14:34:10.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0048441659999999996 2023-01-15T14:34:10.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097551+0000", 2023-01-15T14:34:10.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161788 2023-01-15T14:34:10.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:10.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.097797+0000", 2023-01-15T14:34:10.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024616500000000001 2023-01-15T14:34:10.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:10.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098060+0000", 2023-01-15T14:34:10.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026273399999999998 2023-01-15T14:34:10.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:10.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098078+0000", 2023-01-15T14:34:10.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8090000000000001e-05 2023-01-15T14:34:10.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:10.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098149+0000", 2023-01-15T14:34:10.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0728000000000005e-05 2023-01-15T14:34:10.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:10.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.098247+0000", 2023-01-15T14:34:10.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8412999999999994e-05 2023-01-15T14:34:10.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.398049+0000", 2023-01-15T14:34:10.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29980189200000001 2023-01-15T14:34:10.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498188+0000", 2023-01-15T14:34:10.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100138897 2023-01-15T14:34:10.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498340+0000", 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015200700000000001 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:24.498476+0000", 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013605099999999999 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1312 3.16 3:68fb3263:::smithi09025264-33:head [setxattr _header (57) in=7b,truncate 3361400] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 250.54572002500001, 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47228759199999998, 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.094 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1312 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301286+0000", 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301285+0000", 2023-01-15T14:34:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301290+0000", 2023-01-15T14:34:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2000000000000002e-06 2023-01-15T14:34:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301310+0000", 2023-01-15T14:34:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9930999999999999e-05 2023-01-15T14:34:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301317+0000", 2023-01-15T14:34:10.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2729999999999999e-06 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302075+0000", 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00075879699999999997 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302286+0000", 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021056800000000001 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:34:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302641+0000", 2023-01-15T14:34:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035498300000000001 2023-01-15T14:34:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302948+0000", 2023-01-15T14:34:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030709800000000001 2023-01-15T14:34:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.302961+0000", 2023-01-15T14:34:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3134e-05 2023-01-15T14:34:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303002+0000", 2023-01-15T14:34:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1131000000000003e-05 2023-01-15T14:34:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:10.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303209+0000", 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020668300000000001 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.672781+0000", 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36957224799999999 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773315+0000", 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100534015 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:10.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773411+0000", 2023-01-15T14:34:10.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5575000000000006e-05 2023-01-15T14:34:10.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773573+0000", 2023-01-15T14:34:10.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016266100000000001 2023-01-15T14:34:10.102 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1313 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2023-01-15T14:34:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:34:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 250.54568085, 2023-01-15T14:34:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.472891326, 2023-01-15T14:34:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.103 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1313 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301325+0000", 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301324+0000", 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301328+0000", 2023-01-15T14:34:10.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1879999999999998e-06 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301345+0000", 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7087000000000001e-05 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.301352+0000", 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.233e-06 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303048+0000", 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0016960230000000001 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.303163+0000", 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115096 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.773843+0000", 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.470680406 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774014+0000", 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000170674 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.774216+0000", 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000202191 2023-01-15T14:34:10.108 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1374 3.12 3:495d4bbc:::smithi09025264-16:head [setxattr _header (57) in=7b,truncate 3512663] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 242.15313993000001, 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50900009000000002, 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.109 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1374 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693866+0000", 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693865+0000", 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693874+0000", 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8969999999999997e-06 2023-01-15T14:34:10.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693891+0000", 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693903+0000", 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1440000000000001e-05 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696262+0000", 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023593889999999999 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.696460+0000", 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019847699999999999 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799773+0000", 2023-01-15T14:34:10.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10331261799999999 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.799987+0000", 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021395100000000001 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800324+0000", 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033669499999999997 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800546+0000", 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022260699999999999 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800561+0000", 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4466e-05 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800620+0000", 2023-01-15T14:34:10.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8780999999999998e-05 2023-01-15T14:34:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800651+0000", 2023-01-15T14:34:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1491000000000001e-05 2023-01-15T14:34:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.103068+0000", 2023-01-15T14:34:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.302416505 2023-01-15T14:34:10.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202085+0000", 2023-01-15T14:34:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099017906000000003 2023-01-15T14:34:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202278+0000", 2023-01-15T14:34:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019212 2023-01-15T14:34:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202866+0000", 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058847100000000002 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 242.15308760100001, 2023-01-15T14:34:10.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:34:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:34:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:34:10.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:34:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:34:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:34:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:34:10.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:34:10.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:34:10.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:34:10.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:34:10.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:34:10.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:34:10.386 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.386 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.386 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.387 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:34:10.387 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:34:10.387 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 223.55617440200001, 2023-01-15T14:34:10.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:34:10.501 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:34:10.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:34:10.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:34:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:10.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:34:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:34:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:34:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:10.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:34:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:34:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:34:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:34:10.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:34:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:34:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:34:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:34:10.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:34:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:34:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:34:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:34:10.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:34:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:34:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3562 3.57 3:ea27a813:::smithi09025264-43:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2023-01-15T14:34:10.510 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:34:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 13.761499822999999, 2023-01-15T14:34:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59995680200000001, 2023-01-15T14:34:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3562 2023-01-15T14:34:10.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.117 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:11.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:11.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085504+0000", 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085513+0000", 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7360000000000007e-06 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085528+0000", 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4671999999999999e-05 2023-01-15T14:34:11.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:11.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085539+0000", 2023-01-15T14:34:11.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.113e-05 2023-01-15T14:34:11.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:11.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382469+0000", 2023-01-15T14:34:11.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.296929736 2023-01-15T14:34:11.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382623+0000", 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015408300000000001 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382985+0000", 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036232300000000001 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:11.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383762+0000", 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077680999999999996 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383779+0000", 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7617000000000001e-05 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383838+0000", 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8795000000000002e-05 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383918+0000", 2023-01-15T14:34:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9524999999999995e-05 2023-01-15T14:34:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.385030+0000", 2023-01-15T14:34:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001112664 2023-01-15T14:34:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685250+0000", 2023-01-15T14:34:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30021942800000001 2023-01-15T14:34:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685384+0000", 2023-01-15T14:34:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134055 2023-01-15T14:34:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685463+0000", 2023-01-15T14:34:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8991000000000003e-05 2023-01-15T14:34:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.282 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3618 3.43 3:c3b8cf33:::smithi09025264-26:head [copy-from ver 335] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e352)", 2023-01-15T14:34:11.282 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:34:11.283 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.9397405870000002, 2023-01-15T14:34:11.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73981128600000001, 2023-01-15T14:34:11.283 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:11.283 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:11.283 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:11.283 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:11.283 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:11.283 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3618 2023-01-15T14:34:11.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.284 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:11.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:11.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:34:11.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:11.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:11.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:34:11.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:11.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:11.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907263+0000", 2023-01-15T14:34:11.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:11.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:11.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907273+0000", 2023-01-15T14:34:11.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8109999999999997e-06 2023-01-15T14:34:11.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:11.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907287+0000", 2023-01-15T14:34:11.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4100999999999999e-05 2023-01-15T14:34:11.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:11.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907298+0000", 2023-01-15T14:34:11.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1014e-05 2023-01-15T14:34:11.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:11.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907509+0000", 2023-01-15T14:34:11.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021028299999999999 2023-01-15T14:34:11.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:11.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907519+0000", 2023-01-15T14:34:11.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0686999999999999e-05 2023-01-15T14:34:11.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:11.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.212673+0000", 2023-01-15T14:34:11.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30515353299999998 2023-01-15T14:34:11.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.213072+0000", 2023-01-15T14:34:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039955900000000001 2023-01-15T14:34:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.245152+0000", 2023-01-15T14:34:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032080139000000001 2023-01-15T14:34:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:11.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247179+0000", 2023-01-15T14:34:11.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020266640000000001 2023-01-15T14:34:11.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:11.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247198+0000", 2023-01-15T14:34:11.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:34:11.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.248181+0000", 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0009830310000000001 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.248229+0000", 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7592000000000001e-05 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:11.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.253799+0000", 2023-01-15T14:34:11.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055704659999999996 2023-01-15T14:34:11.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:11.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.646730+0000", 2023-01-15T14:34:11.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39293027400000002 2023-01-15T14:34:11.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:11.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.646885+0000", 2023-01-15T14:34:11.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155243 2023-01-15T14:34:11.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:11.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647077+0000", 2023-01-15T14:34:11.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019156899999999999 2023-01-15T14:34:11.294 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:11.294 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:11.295 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:11.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.296 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3619 3.43 3:c3b8cf33:::smithi09025264-26:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e352)", 2023-01-15T14:34:11.296 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:34:11.296 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.9396891280000004, 2023-01-15T14:34:11.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.74041005699999995, 2023-01-15T14:34:11.296 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:11.296 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:11.296 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:11.296 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:11.296 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:11.297 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3619 2023-01-15T14:34:11.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.297 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:11.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:11.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:34:11.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:11.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:11.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:34:11.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:11.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:11.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907316+0000", 2023-01-15T14:34:11.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:11.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:11.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907320+0000", 2023-01-15T14:34:11.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5820000000000002e-06 2023-01-15T14:34:11.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:11.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907328+0000", 2023-01-15T14:34:11.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.32e-06 2023-01-15T14:34:11.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:11.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907334+0000", 2023-01-15T14:34:11.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5489999999999997e-06 2023-01-15T14:34:11.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:11.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.008282+0000", 2023-01-15T14:34:11.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100948497 2023-01-15T14:34:11.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:11.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.008303+0000", 2023-01-15T14:34:11.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0996e-05 2023-01-15T14:34:11.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:11.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.214382+0000", 2023-01-15T14:34:11.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20607940899999999 2023-01-15T14:34:11.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:34:11.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.214553+0000", 2023-01-15T14:34:11.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017038000000000001 2023-01-15T14:34:11.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:11.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247445+0000", 2023-01-15T14:34:11.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032892271000000001 2023-01-15T14:34:11.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:11.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247684+0000", 2023-01-15T14:34:11.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023870599999999999 2023-01-15T14:34:11.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:11.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647212+0000", 2023-01-15T14:34:11.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39952834599999998 2023-01-15T14:34:11.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:11.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647450+0000", 2023-01-15T14:34:11.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023761799999999999 2023-01-15T14:34:11.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:11.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:11.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:11.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647727+0000", 2023-01-15T14:34:11.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027702100000000002 2023-01-15T14:34:11.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:11.313 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:11.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:11.313 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:11.313 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:11.313 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:11.314 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:34:11.315 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:34:11.316 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:34:11.321 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:11.324 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:34:11.324 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":534,"stamp":"2023-01-15T14:34:09.938505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":356436800,"num_objects":166,"num_object_clones":113,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":95,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":14,"num_read":1133,"num_read_kb":1099302,"num_write":967,"num_write_kb":334061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":388272772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18819,"ondisk_log_size":18819,"up":345,"acting":338,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":385,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6367996,"kb_used_data":555889232,"kb_used_omap":68386,"kb_used_meta":0,"kb_avail":555828284,"statfs":{"total":575688990720,"available":569168162816,"internally_reserved":62410752,"allocated":569230573568,"data_stored":569230573568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70028190,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":6,"commit_latency_ns":0,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2924484,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-63,"num_objects_misplaced":-13,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":91,"num_bytes_recovered":63791984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.967259"},"pg_stats":[{"pgid":"3.69","version":"110'149","reported_seq":905,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.905901+0000","last_change":"2023-01-15T14:33:57.316235+0000","last_active":"2023-01-15T14:34:06.905901+0000","last_peered":"2023-01-15T14:34:06.905901+0000","last_clean":"2023-01-15T14:34:06.905901+0000","last_became_active":"2023-01-15T14:33:50.216417+0000","last_became_peered":"2023-01-15T14:33:50.216417+0000","last_unstale":"2023-01-15T14:34:06.905901+0000","last_undegraded":"2023-01-15T14:34:06.905901+0000","last_fullsized":"2023-01-15T14:34:06.905901+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.406510072,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.68","version":"351'252","reported_seq":1407,"reported_epoch":356,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2023-01-15T14:34:05.084850+0000","last_change":"2023-01-15T14:33:53.618260+0000","last_active":"2023-01-15T14:34:05.084850+0000","last_peered":"2023-01-15T14:34:05.084850+0000","last_clean":"2023-01-15T14:33:42.635266+0000","last_became_active":"2023-01-15T14:33:53.330670+0000","last_became_peered":"2023-01-15T14:33:53.330670+0000","last_unstale":"2023-01-15T14:34:05.084850+0000","last_undegraded":"2023-01-15T14:33:52.928061+0000","last_fullsized":"2023-01-15T14:34:05.084850+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8014979,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":9643107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[1,7,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":3},{"shards":"1,2,7","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.67","version":"191'79","reported_seq":58,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.079604+0000","last_change":"2023-01-15T14:33:57.312535+0000","last_active":"2023-01-15T14:34:05.079604+0000","last_peered":"2023-01-15T14:34:05.079604+0000","last_clean":"2023-01-15T14:34:05.079604+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:34:05.079604+0000","last_undegraded":"2023-01-15T14:34:05.079604+0000","last_fullsized":"2023-01-15T14:34:05.079604+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.40508863699999997,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.66","version":"102'52","reported_seq":368,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.914876+0000","last_change":"2023-01-15T14:33:57.299510+0000","last_active":"2023-01-15T14:34:06.914876+0000","last_peered":"2023-01-15T14:34:06.914876+0000","last_clean":"2023-01-15T14:34:06.914876+0000","last_became_active":"2023-01-15T14:33:48.695009+0000","last_became_peered":"2023-01-15T14:33:48.695009+0000","last_unstale":"2023-01-15T14:34:06.914876+0000","last_undegraded":"2023-01-15T14:34:06.914876+0000","last_fullsized":"2023-01-15T14:34:06.914876+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.390920724,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2142,"num_write":4,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.65","version":"349'120","reported_seq":1161,"reported_epoch":356,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:04.984721+0000","last_change":"2023-01-15T14:33:57.203450+0000","last_active":"2023-01-15T14:34:04.984721+0000","last_peered":"2023-01-15T14:34:04.984721+0000","last_clean":"2023-01-15T14:33:43.349693+0000","last_became_active":"2023-01-15T14:33:57.201006+0000","last_became_peered":"2023-01-15T14:33:57.201006+0000","last_unstale":"2023-01-15T14:34:04.984721+0000","last_undegraded":"2023-01-15T14:33:56.909186+0000","last_fullsized":"2023-01-15T14:33:56.908621+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7224063,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":3033,"num_write":12,"num_write_kb":4201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":11126498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,4],"avail_no_missing":["1","2","3","4"],"object_location_counts":[{"shards":"0,1,2,3,4","objects":1},{"shards":"1,2,3,4","objects":1},{"shards":"1,2,4","objects":1},{"shards":"1,4","objects":-1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.64","version":"334'251","reported_seq":1414,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.268552+0000","last_change":"2023-01-15T14:33:57.417656+0000","last_active":"2023-01-15T14:34:04.268552+0000","last_peered":"2023-01-15T14:34:04.268552+0000","last_clean":"2023-01-15T14:34:04.268552+0000","last_became_active":"2023-01-15T14:33:55.675085+0000","last_became_peered":"2023-01-15T14:33:55.675085+0000","last_unstale":"2023-01-15T14:34:04.268552+0000","last_undegraded":"2023-01-15T14:34:04.268552+0000","last_fullsized":"2023-01-15T14:34:04.268552+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.409133782,"stat_sum":{"num_bytes":3548733,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10773,"num_write":5,"num_write_kb":2741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.63","version":"149'233","reported_seq":743,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.268579+0000","last_change":"2023-01-15T14:33:57.518738+0000","last_active":"2023-01-15T14:34:04.268579+0000","last_peered":"2023-01-15T14:34:04.268579+0000","last_clean":"2023-01-15T14:34:04.268579+0000","last_became_active":"2023-01-15T14:33:55.277739+0000","last_became_peered":"2023-01-15T14:33:55.277739+0000","last_unstale":"2023-01-15T14:34:04.268579+0000","last_undegraded":"2023-01-15T14:34:04.268579+0000","last_fullsized":"2023-01-15T14:34:04.268579+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.41396429299999998,"stat_sum":{"num_bytes":3802136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":21819,"num_write":19,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.62","version":"348'169","reported_seq":1351,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.984077+0000","last_change":"2023-01-15T14:33:58.705842+0000","last_active":"2023-01-15T14:34:04.984077+0000","last_peered":"2023-01-15T14:34:04.984077+0000","last_clean":"2023-01-15T14:34:04.984077+0000","last_became_active":"2023-01-15T14:33:56.590183+0000","last_became_peered":"2023-01-15T14:33:56.590183+0000","last_unstale":"2023-01-15T14:34:04.984077+0000","last_undegraded":"2023-01-15T14:34:04.984077+0000","last_fullsized":"2023-01-15T14:34:04.984077+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.70374445200000002,"stat_sum":{"num_bytes":5361549,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6734,"num_write":11,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5210269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.61","version":"321'138","reported_seq":1004,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.168154+0000","last_change":"2023-01-15T14:33:57.185692+0000","last_active":"2023-01-15T14:34:04.168154+0000","last_peered":"2023-01-15T14:34:04.168154+0000","last_clean":"2023-01-15T14:34:04.168154+0000","last_became_active":"2023-01-15T14:33:53.421056+0000","last_became_peered":"2023-01-15T14:33:53.421056+0000","last_unstale":"2023-01-15T14:34:04.168154+0000","last_undegraded":"2023-01-15T14:34:04.168154+0000","last_fullsized":"2023-01-15T14:34:04.168154+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.27715718700000003,"stat_sum":{"num_bytes":1033664,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6059,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.60","version":"348'281","reported_seq":1414,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.983550+0000","last_change":"2023-01-15T14:33:58.810339+0000","last_active":"2023-01-15T14:34:04.983550+0000","last_peered":"2023-01-15T14:34:04.983550+0000","last_clean":"2023-01-15T14:34:04.983550+0000","last_became_active":"2023-01-15T14:33:56.478613+0000","last_became_peered":"2023-01-15T14:33:56.478613+0000","last_unstale":"2023-01-15T14:34:04.983550+0000","last_undegraded":"2023-01-15T14:34:04.983550+0000","last_fullsized":"2023-01-15T14:34:04.983550+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.80913330699999997,"stat_sum":{"num_bytes":5214108,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":10865,"num_write":14,"num_write_kb":5082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15418818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1377,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.269898+0000","last_change":"2023-01-15T14:33:57.313450+0000","last_active":"2023-01-15T14:34:04.269898+0000","last_peered":"2023-01-15T14:34:04.269898+0000","last_clean":"2023-01-15T14:34:04.269898+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:34:04.269898+0000","last_undegraded":"2023-01-15T14:34:04.269898+0000","last_fullsized":"2023-01-15T14:34:04.269898+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:51.919","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.39161994999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5e","version":"215'39","reported_seq":984,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:07.009126+0000","last_change":"2023-01-15T14:33:57.316972+0000","last_active":"2023-01-15T14:34:07.009126+0000","last_peered":"2023-01-15T14:34:07.009126+0000","last_clean":"2023-01-15T14:34:07.009126+0000","last_became_active":"2023-01-15T14:33:35.364143+0000","last_became_peered":"2023-01-15T14:33:35.364143+0000","last_unstale":"2023-01-15T14:34:07.009126+0000","last_undegraded":"2023-01-15T14:34:07.009126+0000","last_fullsized":"2023-01-15T14:34:07.009126+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40693008000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1313,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.914874+0000","last_change":"2023-01-15T14:34:00.343379+0000","last_active":"2023-01-15T14:34:06.914874+0000","last_peered":"2023-01-15T14:34:06.914874+0000","last_clean":"2023-01-15T14:34:06.914874+0000","last_became_active":"2023-01-15T14:34:00.242122+0000","last_became_peered":"2023-01-15T14:34:00.242122+0000","last_unstale":"2023-01-15T14:34:06.914874+0000","last_undegraded":"2023-01-15T14:34:06.914874+0000","last_fullsized":"2023-01-15T14:34:06.914874+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.099745922000000001,"stat_sum":{"num_bytes":2869928,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":23811,"num_write":13,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5c","version":"141'46","reported_seq":85,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.908864+0000","last_change":"2023-01-15T14:33:58.026285+0000","last_active":"2023-01-15T14:34:06.908864+0000","last_peered":"2023-01-15T14:34:06.908864+0000","last_clean":"2023-01-15T14:34:06.908864+0000","last_became_active":"2023-01-15T14:33:57.113032+0000","last_became_peered":"2023-01-15T14:33:57.113032+0000","last_unstale":"2023-01-15T14:34:06.908864+0000","last_undegraded":"2023-01-15T14:34:06.908864+0000","last_fullsized":"2023-01-15T14:34:06.908864+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.121922606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5b","version":"283'80","reported_seq":915,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979550+0000","last_change":"2023-01-15T14:33:57.113633+0000","last_active":"2023-01-15T14:34:04.979550+0000","last_peered":"2023-01-15T14:34:04.979550+0000","last_clean":"2023-01-15T14:34:04.979550+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:34:04.979550+0000","last_undegraded":"2023-01-15T14:34:04.979550+0000","last_fullsized":"2023-01-15T14:34:04.979550+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.20934807,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5a","version":"329'348","reported_seq":1494,"reported_epoch":357,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:34:07.731858+0000","last_change":"2023-01-15T14:34:07.617290+0000","last_active":"2023-01-15T14:34:07.731858+0000","last_peered":"2023-01-15T14:34:07.731858+0000","last_clean":"2023-01-15T14:33:50.015328+0000","last_became_active":"2023-01-15T14:33:53.229261+0000","last_became_peered":"2023-01-15T14:33:53.229261+0000","last_unstale":"2023-01-15T14:34:07.731858+0000","last_undegraded":"2023-01-15T14:34:07.731858+0000","last_fullsized":"2023-01-15T14:33:52.913106+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":333,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:02.755","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.45505085000000001,"stat_sum":{"num_bytes":5983424,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":19294,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[1,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.59","version":"294'256","reported_seq":1282,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:07.200247+0000","last_change":"2023-01-15T14:33:57.406685+0000","last_active":"2023-01-15T14:34:07.200247+0000","last_peered":"2023-01-15T14:34:07.200247+0000","last_clean":"2023-01-15T14:34:07.200247+0000","last_became_active":"2023-01-15T14:33:55.671990+0000","last_became_peered":"2023-01-15T14:33:55.671990+0000","last_unstale":"2023-01-15T14:34:07.200247+0000","last_undegraded":"2023-01-15T14:34:07.200247+0000","last_fullsized":"2023-01-15T14:34:07.200247+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.49887967,"stat_sum":{"num_bytes":1210137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":5,"num_read_kb":3549,"num_write":5,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.58","version":"323'320","reported_seq":1484,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.977996+0000","last_change":"2023-01-15T14:33:57.004059+0000","last_active":"2023-01-15T14:34:04.977996+0000","last_peered":"2023-01-15T14:34:04.977996+0000","last_clean":"2023-01-15T14:34:04.977996+0000","last_became_active":"2023-01-15T14:33:40.127669+0000","last_became_peered":"2023-01-15T14:33:40.127669+0000","last_unstale":"2023-01-15T14:34:04.977996+0000","last_undegraded":"2023-01-15T14:34:04.977996+0000","last_fullsized":"2023-01-15T14:34:04.977996+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.103642679,"stat_sum":{"num_bytes":4332079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.57","version":"351'98","reported_seq":97,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.269756+0000","last_change":"2023-01-15T14:33:57.890625+0000","last_active":"2023-01-15T14:34:04.269756+0000","last_peered":"2023-01-15T14:34:04.269756+0000","last_clean":"2023-01-15T14:34:04.269756+0000","last_became_active":"2023-01-15T14:33:36.994447+0000","last_became_peered":"2023-01-15T14:33:36.994447+0000","last_unstale":"2023-01-15T14:34:04.269756+0000","last_undegraded":"2023-01-15T14:34:04.269756+0000","last_fullsized":"2023-01-15T14:34:04.269756+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.969000109,"stat_sum":{"num_bytes":1512663,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3025326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.56","version":"320'127","reported_seq":634,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979227+0000","last_change":"2023-01-15T14:34:00.541392+0000","last_active":"2023-01-15T14:34:04.979227+0000","last_peered":"2023-01-15T14:34:04.979227+0000","last_clean":"2023-01-15T14:34:04.979227+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:34:04.979227+0000","last_undegraded":"2023-01-15T14:34:04.979227+0000","last_fullsized":"2023-01-15T14:34:04.979227+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.30190621200000001,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.55","version":"110'55","reported_seq":1015,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.908915+0000","last_change":"2023-01-15T14:33:57.315559+0000","last_active":"2023-01-15T14:34:06.908915+0000","last_peered":"2023-01-15T14:34:06.908915+0000","last_clean":"2023-01-15T14:34:06.908915+0000","last_became_active":"2023-01-15T14:33:52.510429+0000","last_became_peered":"2023-01-15T14:33:52.510429+0000","last_unstale":"2023-01-15T14:34:06.908915+0000","last_undegraded":"2023-01-15T14:34:06.908915+0000","last_fullsized":"2023-01-15T14:34:06.908915+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.40629608699999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1133,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:03.976291+0000","last_change":"2023-01-15T14:33:57.109553+0000","last_active":"2023-01-15T14:34:03.976291+0000","last_peered":"2023-01-15T14:34:03.976291+0000","last_clean":"2023-01-15T14:34:03.976291+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:34:03.976291+0000","last_undegraded":"2023-01-15T14:34:03.976291+0000","last_fullsized":"2023-01-15T14:34:03.976291+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:33:45.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:45.583005+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.201302861,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.53","version":"352'397","reported_seq":1407,"reported_epoch":356,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:04.984279+0000","last_change":"2023-01-15T14:33:58.206587+0000","last_active":"2023-01-15T14:34:04.984279+0000","last_peered":"2023-01-15T14:34:04.984279+0000","last_clean":"2023-01-15T14:33:55.165228+0000","last_became_active":"2023-01-15T14:33:58.106461+0000","last_became_peered":"2023-01-15T14:33:58.106461+0000","last_unstale":"2023-01-15T14:34:04.984279+0000","last_undegraded":"2023-01-15T14:33:58.004141+0000","last_fullsized":"2023-01-15T14:33:58.002666+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":397,"log_dups_size":0,"ondisk_log_size":397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:44.609","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.092650028999999995,"stat_sum":{"num_bytes":26431669,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":18,"num_read_kb":17078,"num_write":19,"num_write_kb":11543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":6},{"shards":"1,3","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":734,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979524+0000","last_change":"2023-01-15T14:33:56.903307+0000","last_active":"2023-01-15T14:34:04.979524+0000","last_peered":"2023-01-15T14:34:04.979524+0000","last_clean":"2023-01-15T14:34:04.979524+0000","last_became_active":"2023-01-15T14:33:55.472400+0000","last_became_peered":"2023-01-15T14:33:55.472400+0000","last_unstale":"2023-01-15T14:34:04.979524+0000","last_undegraded":"2023-01-15T14:34:04.979524+0000","last_fullsized":"2023-01-15T14:34:04.979524+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.001336634,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.51","version":"348'316","reported_seq":1523,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.983982+0000","last_change":"2023-01-15T14:33:57.893317+0000","last_active":"2023-01-15T14:34:04.983982+0000","last_peered":"2023-01-15T14:34:04.983982+0000","last_clean":"2023-01-15T14:34:04.983982+0000","last_became_active":"2023-01-15T14:33:26.107940+0000","last_became_peered":"2023-01-15T14:33:26.107940+0000","last_unstale":"2023-01-15T14:34:04.983982+0000","last_undegraded":"2023-01-15T14:34:04.983982+0000","last_fullsized":"2023-01-15T14:34:04.983982+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:36.458","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.79495623100000001,"stat_sum":{"num_bytes":7496813,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.50","version":"337'420","reported_seq":1586,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.978292+0000","last_change":"2023-01-15T14:34:02.564430+0000","last_active":"2023-01-15T14:34:04.978292+0000","last_peered":"2023-01-15T14:34:04.978292+0000","last_clean":"2023-01-15T14:34:04.978292+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:34:04.978292+0000","last_undegraded":"2023-01-15T14:34:04.978292+0000","last_fullsized":"2023-01-15T14:34:04.978292+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.0084659680000000008,"stat_sum":{"num_bytes":9681416,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":52505,"num_write":49,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4f","version":"155'145","reported_seq":832,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.909215+0000","last_change":"2023-01-15T14:33:57.353637+0000","last_active":"2023-01-15T14:34:06.909215+0000","last_peered":"2023-01-15T14:34:06.909215+0000","last_clean":"2023-01-15T14:34:06.909215+0000","last_became_active":"2023-01-15T14:33:25.129936+0000","last_became_peered":"2023-01-15T14:33:25.129936+0000","last_unstale":"2023-01-15T14:34:06.909215+0000","last_undegraded":"2023-01-15T14:34:06.909215+0000","last_fullsized":"2023-01-15T14:34:06.909215+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44322561900000002,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":101,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.169360+0000","last_change":"2023-01-15T14:33:56.915710+0000","last_active":"2023-01-15T14:34:04.169360+0000","last_peered":"2023-01-15T14:34:04.169360+0000","last_clean":"2023-01-15T14:34:04.169360+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:34:04.169360+0000","last_undegraded":"2023-01-15T14:34:04.169360+0000","last_fullsized":"2023-01-15T14:34:04.169360+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.0080109380000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4d","version":"356'262","reported_seq":1365,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.911880+0000","last_change":"2023-01-15T14:34:06.586527+0000","last_active":"2023-01-15T14:34:06.911880+0000","last_peered":"2023-01-15T14:34:06.911880+0000","last_clean":"2023-01-15T14:34:06.911880+0000","last_became_active":"2023-01-15T14:34:05.178613+0000","last_became_peered":"2023-01-15T14:34:05.178613+0000","last_unstale":"2023-01-15T14:34:06.911880+0000","last_undegraded":"2023-01-15T14:34:06.911880+0000","last_fullsized":"2023-01-15T14:34:06.911880+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":1.4066433030000001,"stat_sum":{"num_bytes":12124166,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":15961,"num_write":18,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4c","version":"309'71","reported_seq":111,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.980967+0000","last_change":"2023-01-15T14:33:57.413785+0000","last_active":"2023-01-15T14:34:04.980967+0000","last_peered":"2023-01-15T14:34:04.980967+0000","last_clean":"2023-01-15T14:34:04.980967+0000","last_became_active":"2023-01-15T14:33:52.311859+0000","last_became_peered":"2023-01-15T14:33:52.311859+0000","last_unstale":"2023-01-15T14:34:04.980967+0000","last_undegraded":"2023-01-15T14:34:04.980967+0000","last_fullsized":"2023-01-15T14:34:04.980967+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.40874315500000002,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":757,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.984543+0000","last_change":"2023-01-15T14:33:57.627070+0000","last_active":"2023-01-15T14:34:04.984543+0000","last_peered":"2023-01-15T14:34:04.984543+0000","last_clean":"2023-01-15T14:34:04.984543+0000","last_became_active":"2023-01-15T14:33:52.406605+0000","last_became_peered":"2023-01-15T14:33:52.406605+0000","last_unstale":"2023-01-15T14:34:04.984543+0000","last_undegraded":"2023-01-15T14:34:04.984543+0000","last_fullsized":"2023-01-15T14:34:04.984543+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:42.594","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.515074951,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4a","version":"348'371","reported_seq":1482,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.079832+0000","last_change":"2023-01-15T14:33:58.018192+0000","last_active":"2023-01-15T14:34:05.079832+0000","last_peered":"2023-01-15T14:34:05.079832+0000","last_clean":"2023-01-15T14:34:05.079832+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:34:05.079832+0000","last_undegraded":"2023-01-15T14:34:05.079832+0000","last_fullsized":"2023-01-15T14:34:05.079832+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:33.328","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10440110800000001,"stat_sum":{"num_bytes":12470870,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":63,"num_read_kb":67536,"num_write":73,"num_write_kb":23626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":939,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.978037+0000","last_change":"2023-01-15T14:34:00.046526+0000","last_active":"2023-01-15T14:34:04.978037+0000","last_peered":"2023-01-15T14:34:04.978037+0000","last_clean":"2023-01-15T14:34:04.978037+0000","last_became_active":"2023-01-15T14:34:00.043221+0000","last_became_peered":"2023-01-15T14:34:00.043221+0000","last_unstale":"2023-01-15T14:34:04.978037+0000","last_undegraded":"2023-01-15T14:34:04.978037+0000","last_fullsized":"2023-01-15T14:34:04.978037+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.00195963,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.48","version":"179'204","reported_seq":1363,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.906113+0000","last_change":"2023-01-15T14:33:57.356590+0000","last_active":"2023-01-15T14:34:06.906113+0000","last_peered":"2023-01-15T14:34:06.906113+0000","last_clean":"2023-01-15T14:34:06.906113+0000","last_became_active":"2023-01-15T14:33:55.375518+0000","last_became_peered":"2023-01-15T14:33:55.375518+0000","last_unstale":"2023-01-15T14:34:06.906113+0000","last_undegraded":"2023-01-15T14:34:06.906113+0000","last_fullsized":"2023-01-15T14:34:06.906113+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:43.662","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.34933974800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1123,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.915477+0000","last_change":"2023-01-15T14:33:57.300260+0000","last_active":"2023-01-15T14:34:06.915477+0000","last_peered":"2023-01-15T14:34:06.915477+0000","last_clean":"2023-01-15T14:34:06.915477+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:34:06.915477+0000","last_undegraded":"2023-01-15T14:34:06.915477+0000","last_fullsized":"2023-01-15T14:34:06.915477+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.39166528900000003,"stat_sum":{"num_bytes":496762,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":5,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":1033,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.186647+0000","last_change":"2023-01-15T14:33:57.829426+0000","last_active":"2023-01-15T14:34:05.186647+0000","last_peered":"2023-01-15T14:34:05.186647+0000","last_clean":"2023-01-15T14:34:05.186647+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:34:05.186647+0000","last_undegraded":"2023-01-15T14:34:05.186647+0000","last_fullsized":"2023-01-15T14:34:05.186647+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:55.245","scrub_duration":0.30448409700000001,"objects_trimmed":0,"snaptrim_duration":0.714760332,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":918,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.170081+0000","last_change":"2023-01-15T14:33:56.909121+0000","last_active":"2023-01-15T14:34:04.170081+0000","last_peered":"2023-01-15T14:34:04.170081+0000","last_clean":"2023-01-15T14:34:04.170081+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:34:04.170081+0000","last_undegraded":"2023-01-15T14:34:04.170081+0000","last_fullsized":"2023-01-15T14:34:04.170081+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045307990000000003,"stat_sum":{"num_bytes":2635976,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":19927,"num_write":15,"num_write_kb":6659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4745860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1278,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:07.014644+0000","last_change":"2023-01-15T14:33:57.298701+0000","last_active":"2023-01-15T14:34:07.014644+0000","last_peered":"2023-01-15T14:34:07.014644+0000","last_clean":"2023-01-15T14:34:07.014644+0000","last_became_active":"2023-01-15T14:33:55.678019+0000","last_became_peered":"2023-01-15T14:33:55.678019+0000","last_unstale":"2023-01-15T14:34:07.014644+0000","last_undegraded":"2023-01-15T14:34:07.014644+0000","last_fullsized":"2023-01-15T14:34:07.014644+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:27.120","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.390225449,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.43","version":"353'316","reported_seq":1519,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.268739+0000","last_change":"2023-01-15T14:34:02.217041+0000","last_active":"2023-01-15T14:34:04.268739+0000","last_peered":"2023-01-15T14:34:04.268739+0000","last_clean":"2023-01-15T14:34:04.268739+0000","last_became_active":"2023-01-15T14:34:02.211232+0000","last_became_peered":"2023-01-15T14:34:02.211232+0000","last_unstale":"2023-01-15T14:34:04.268739+0000","last_undegraded":"2023-01-15T14:34:04.268739+0000","last_fullsized":"2023-01-15T14:34:04.268739+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.0028674299999999998,"stat_sum":{"num_bytes":6319564,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":16585,"num_write":6,"num_write_kb":4207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3395080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.42","version":"155'101","reported_seq":1039,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.908211+0000","last_change":"2023-01-15T14:33:57.621398+0000","last_active":"2023-01-15T14:34:06.908211+0000","last_peered":"2023-01-15T14:34:06.908211+0000","last_clean":"2023-01-15T14:34:06.908211+0000","last_became_active":"2023-01-15T14:33:55.376782+0000","last_became_peered":"2023-01-15T14:33:55.376782+0000","last_unstale":"2023-01-15T14:34:06.908211+0000","last_undegraded":"2023-01-15T14:34:06.908211+0000","last_fullsized":"2023-01-15T14:34:06.908211+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.51101871399999999,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1119,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:07.115485+0000","last_change":"2023-01-15T14:33:57.302504+0000","last_active":"2023-01-15T14:34:07.115485+0000","last_peered":"2023-01-15T14:34:07.115485+0000","last_clean":"2023-01-15T14:34:07.115485+0000","last_became_active":"2023-01-15T14:33:55.771969+0000","last_became_peered":"2023-01-15T14:33:55.771969+0000","last_unstale":"2023-01-15T14:34:07.115485+0000","last_undegraded":"2023-01-15T14:34:07.115485+0000","last_fullsized":"2023-01-15T14:34:07.115485+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.29405439700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.40","version":"335'307","reported_seq":1399,"reported_epoch":356,"state":"active+recovery_wait+degraded","last_fresh":"2023-01-15T14:34:04.979357+0000","last_change":"2023-01-15T14:33:58.118750+0000","last_active":"2023-01-15T14:34:04.979357+0000","last_peered":"2023-01-15T14:34:04.979357+0000","last_clean":"2023-01-15T14:33:43.952880+0000","last_became_active":"2023-01-15T14:33:58.117085+0000","last_became_peered":"2023-01-15T14:33:58.117085+0000","last_unstale":"2023-01-15T14:34:04.979357+0000","last_undegraded":"2023-01-15T14:33:57.914305+0000","last_fullsized":"2023-01-15T14:34:04.979357+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":305,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.704289622,"stat_sum":{"num_bytes":12753904,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":35541,"num_write":28,"num_write_kb":11001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":16413587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3f","version":"321'233","reported_seq":1374,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:07.012116+0000","last_change":"2023-01-15T14:33:58.333573+0000","last_active":"2023-01-15T14:34:07.012116+0000","last_peered":"2023-01-15T14:34:07.012116+0000","last_clean":"2023-01-15T14:34:07.012116+0000","last_became_active":"2023-01-15T14:33:58.217783+0000","last_became_peered":"2023-01-15T14:33:58.217783+0000","last_unstale":"2023-01-15T14:34:07.012116+0000","last_undegraded":"2023-01-15T14:34:07.012116+0000","last_fullsized":"2023-01-15T14:34:07.012116+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:28.180","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.014066018,"stat_sum":{"num_bytes":4764997,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6605283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3e","version":"294'63","reported_seq":839,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.085451+0000","last_change":"2023-01-15T14:33:57.728361+0000","last_active":"2023-01-15T14:34:05.085451+0000","last_peered":"2023-01-15T14:34:05.085451+0000","last_clean":"2023-01-15T14:34:05.085451+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:34:05.085451+0000","last_undegraded":"2023-01-15T14:34:05.085451+0000","last_fullsized":"2023-01-15T14:34:05.085451+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.61381065800000001,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3d","version":"356'236","reported_seq":1332,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.400650+0000","last_change":"2023-01-15T14:34:06.401367+0000","last_active":"2023-01-15T14:34:06.400650+0000","last_peered":"2023-01-15T14:34:06.400650+0000","last_clean":"2023-01-15T14:34:06.400650+0000","last_became_active":"2023-01-15T14:34:05.187121+0000","last_became_peered":"2023-01-15T14:34:05.187121+0000","last_unstale":"2023-01-15T14:34:06.400650+0000","last_undegraded":"2023-01-15T14:34:06.400650+0000","last_fullsized":"2023-01-15T14:34:06.400650+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":1.2126637869999999,"stat_sum":{"num_bytes":8050279,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":13797,"num_write":16,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":8050279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3c","version":"141'46","reported_seq":355,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.909049+0000","last_change":"2023-01-15T14:33:56.912541+0000","last_active":"2023-01-15T14:34:06.909049+0000","last_peered":"2023-01-15T14:34:06.909049+0000","last_clean":"2023-01-15T14:34:06.909049+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:34:06.909049+0000","last_undegraded":"2023-01-15T14:34:06.909049+0000","last_fullsized":"2023-01-15T14:34:06.909049+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.470","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.0047236029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":29,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979267+0000","last_change":"2023-01-15T14:33:58.125276+0000","last_active":"2023-01-15T14:34:04.979267+0000","last_peered":"2023-01-15T14:34:04.979267+0000","last_clean":"2023-01-15T14:34:04.979267+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:34:04.979267+0000","last_undegraded":"2023-01-15T14:34:04.979267+0000","last_fullsized":"2023-01-15T14:34:04.979267+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.21169681200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1352,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.170270+0000","last_change":"2023-01-15T14:33:57.286278+0000","last_active":"2023-01-15T14:34:04.170270+0000","last_peered":"2023-01-15T14:34:04.170270+0000","last_clean":"2023-01-15T14:34:04.170270+0000","last_became_active":"2023-01-15T14:33:55.474944+0000","last_became_peered":"2023-01-15T14:33:55.474944+0000","last_unstale":"2023-01-15T14:34:04.170270+0000","last_undegraded":"2023-01-15T14:34:04.170270+0000","last_fullsized":"2023-01-15T14:34:04.170270+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.37705996200000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.39","version":"334'351","reported_seq":1467,"reported_epoch":357,"state":"peering","last_fresh":"2023-01-15T14:34:06.911835+0000","last_change":"2023-01-15T14:34:06.912209+0000","last_active":"2023-01-15T14:34:06.911783+0000","last_peered":"2023-01-15T14:34:06.173371+0000","last_clean":"2023-01-15T14:33:50.016390+0000","last_became_active":"2023-01-15T14:33:53.436135+0000","last_became_peered":"2023-01-15T14:33:53.436135+0000","last_unstale":"2023-01-15T14:34:06.911835+0000","last_undegraded":"2023-01-15T14:34:06.911835+0000","last_fullsized":"2023-01-15T14:34:06.911835+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":239,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:40.787","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.0051603159999999999,"stat_sum":{"num_bytes":24023444,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":66941,"num_write":7,"num_write_kb":7292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":26862613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"352'315","reported_seq":1537,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.906434+0000","last_change":"2023-01-15T14:33:58.009343+0000","last_active":"2023-01-15T14:34:06.906434+0000","last_peered":"2023-01-15T14:34:06.906434+0000","last_clean":"2023-01-15T14:34:06.906434+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:34:06.906434+0000","last_undegraded":"2023-01-15T14:34:06.906434+0000","last_fullsized":"2023-01-15T14:34:06.906434+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.10683759900000001,"stat_sum":{"num_bytes":8461433,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":42848,"num_write":39,"num_write_kb":13455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":415,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.081415+0000","last_change":"2023-01-15T14:33:58.124010+0000","last_active":"2023-01-15T14:34:05.081415+0000","last_peered":"2023-01-15T14:34:05.081415+0000","last_clean":"2023-01-15T14:34:05.081415+0000","last_became_active":"2023-01-15T14:33:52.412128+0000","last_became_peered":"2023-01-15T14:33:52.412128+0000","last_unstale":"2023-01-15T14:34:05.081415+0000","last_undegraded":"2023-01-15T14:34:05.081415+0000","last_fullsized":"2023-01-15T14:34:05.081415+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.207923046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":567,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.085677+0000","last_change":"2023-01-15T14:33:57.332104+0000","last_active":"2023-01-15T14:34:05.085677+0000","last_peered":"2023-01-15T14:34:05.085677+0000","last_clean":"2023-01-15T14:34:05.085677+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:34:05.085677+0000","last_undegraded":"2023-01-15T14:34:05.085677+0000","last_fullsized":"2023-01-15T14:34:05.085677+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.31780763299999998,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":41,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.169813+0000","last_change":"2023-01-15T14:33:56.908180+0000","last_active":"2023-01-15T14:34:04.169813+0000","last_peered":"2023-01-15T14:34:04.169813+0000","last_clean":"2023-01-15T14:34:04.169813+0000","last_became_active":"2023-01-15T14:33:52.612576+0000","last_became_peered":"2023-01-15T14:33:52.612576+0000","last_unstale":"2023-01-15T14:34:04.169813+0000","last_undegraded":"2023-01-15T14:34:04.169813+0000","last_fullsized":"2023-01-15T14:34:04.169813+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003836173,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.34","version":"169'138","reported_seq":1163,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.914595+0000","last_change":"2023-01-15T14:33:57.301685+0000","last_active":"2023-01-15T14:34:06.914595+0000","last_peered":"2023-01-15T14:34:06.914595+0000","last_clean":"2023-01-15T14:34:06.914595+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:34:06.914595+0000","last_undegraded":"2023-01-15T14:34:06.914595+0000","last_fullsized":"2023-01-15T14:34:06.914595+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.39300976100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1238,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.908726+0000","last_change":"2023-01-15T14:33:57.354727+0000","last_active":"2023-01-15T14:34:06.908726+0000","last_peered":"2023-01-15T14:34:06.908726+0000","last_clean":"2023-01-15T14:34:06.908726+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:34:06.908726+0000","last_undegraded":"2023-01-15T14:34:06.908726+0000","last_fullsized":"2023-01-15T14:34:06.908726+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.442392491,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.32","version":"355'165","reported_seq":849,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.279503+0000","last_change":"2023-01-15T14:34:05.280466+0000","last_active":"2023-01-15T14:34:05.279503+0000","last_peered":"2023-01-15T14:34:05.279503+0000","last_clean":"2023-01-15T14:34:05.279503+0000","last_became_active":"2023-01-15T14:34:04.171493+0000","last_became_peered":"2023-01-15T14:34:04.171493+0000","last_unstale":"2023-01-15T14:34:05.279503+0000","last_undegraded":"2023-01-15T14:34:05.279503+0000","last_fullsized":"2023-01-15T14:34:05.279503+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.19993219000000001,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7048,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5327885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.13","version":"348'374","reported_seq":1388,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979906+0000","last_change":"2023-01-15T14:33:58.021945+0000","last_active":"2023-01-15T14:34:04.979906+0000","last_peered":"2023-01-15T14:34:04.979906+0000","last_clean":"2023-01-15T14:34:04.979906+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:34:04.979906+0000","last_undegraded":"2023-01-15T14:34:04.979906+0000","last_fullsized":"2023-01-15T14:34:04.979906+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:29.359","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.0064123280000000001,"stat_sum":{"num_bytes":8364486,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":61489,"num_write":17,"num_write_kb":9266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.12","version":"353'186","reported_seq":904,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.168258+0000","last_change":"2023-01-15T14:34:02.715457+0000","last_active":"2023-01-15T14:34:04.168258+0000","last_peered":"2023-01-15T14:34:04.168258+0000","last_clean":"2023-01-15T14:34:04.168258+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:34:04.168258+0000","last_undegraded":"2023-01-15T14:34:04.168258+0000","last_fullsized":"2023-01-15T14:34:04.168258+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.57998335899999998,"stat_sum":{"num_bytes":7581497,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":15,"num_write_kb":5299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.11","version":"334'306","reported_seq":1546,"reported_epoch":357,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:34:07.413751+0000","last_change":"2023-01-15T14:34:07.414359+0000","last_active":"2023-01-15T14:34:07.413751+0000","last_peered":"2023-01-15T14:34:07.413751+0000","last_clean":"2023-01-15T14:33:50.016827+0000","last_became_active":"2023-01-15T14:33:53.659621+0000","last_became_peered":"2023-01-15T14:33:53.659621+0000","last_unstale":"2023-01-15T14:34:07.413751+0000","last_undegraded":"2023-01-15T14:34:07.413751+0000","last_fullsized":"2023-01-15T14:33:53.017412+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":306,"log_dups_size":0,"ondisk_log_size":306,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.090247347000000006,"stat_sum":{"num_bytes":10718732,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":48289,"num_write":26,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":20307357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,0],"avail_no_missing":["7","0","3","4"],"object_location_counts":[{"shards":"0,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"308'406","reported_seq":1409,"reported_epoch":356,"state":"active+recovery_wait+remapped","last_fresh":"2023-01-15T14:34:04.983850+0000","last_change":"2023-01-15T14:33:53.820604+0000","last_active":"2023-01-15T14:34:04.983850+0000","last_peered":"2023-01-15T14:34:04.983850+0000","last_clean":"2023-01-15T14:33:17.174184+0000","last_became_active":"2023-01-15T14:33:53.718934+0000","last_became_peered":"2023-01-15T14:33:53.718934+0000","last_unstale":"2023-01-15T14:34:04.983850+0000","last_undegraded":"2023-01-15T14:34:04.983850+0000","last_fullsized":"2023-01-15T14:34:04.983850+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.84251846600000002,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":41964,"num_write":46,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":6670228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[1,3,2],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"155'145","reported_seq":1240,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979212+0000","last_change":"2023-01-15T14:34:00.245548+0000","last_active":"2023-01-15T14:34:04.979212+0000","last_peered":"2023-01-15T14:34:04.979212+0000","last_clean":"2023-01-15T14:34:04.979212+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:34:04.979212+0000","last_undegraded":"2023-01-15T14:34:04.979212+0000","last_fullsized":"2023-01-15T14:34:04.979212+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.0023104509999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.e","version":"139'38","reported_seq":29,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979082+0000","last_change":"2023-01-15T14:33:58.422638+0000","last_active":"2023-01-15T14:34:04.979082+0000","last_peered":"2023-01-15T14:34:04.979082+0000","last_clean":"2023-01-15T14:34:04.979082+0000","last_became_active":"2023-01-15T14:33:57.310759+0000","last_became_peered":"2023-01-15T14:33:57.310759+0000","last_unstale":"2023-01-15T14:34:04.979082+0000","last_undegraded":"2023-01-15T14:34:04.979082+0000","last_fullsized":"2023-01-15T14:34:04.979082+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.40586427000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.d","version":"296'232","reported_seq":1303,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.077352+0000","last_change":"2023-01-15T14:33:57.114569+0000","last_active":"2023-01-15T14:34:05.077352+0000","last_peered":"2023-01-15T14:34:05.077352+0000","last_clean":"2023-01-15T14:34:05.077352+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:34:05.077352+0000","last_undegraded":"2023-01-15T14:34:05.077352+0000","last_fullsized":"2023-01-15T14:34:05.077352+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.20767458899999999,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.c","version":"348'95","reported_seq":1100,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.909492+0000","last_change":"2023-01-15T14:33:58.326911+0000","last_active":"2023-01-15T14:34:06.909492+0000","last_peered":"2023-01-15T14:34:06.909492+0000","last_clean":"2023-01-15T14:34:06.909492+0000","last_became_active":"2023-01-15T14:33:52.210127+0000","last_became_peered":"2023-01-15T14:33:52.210127+0000","last_unstale":"2023-01-15T14:34:06.909492+0000","last_undegraded":"2023-01-15T14:34:06.909492+0000","last_fullsized":"2023-01-15T14:34:06.909492+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.421830761,"stat_sum":{"num_bytes":2221858,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":10493,"num_write":6,"num_write_kb":686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8023674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":1009,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.908859+0000","last_change":"2023-01-15T14:33:57.618325+0000","last_active":"2023-01-15T14:34:06.908859+0000","last_peered":"2023-01-15T14:34:06.908859+0000","last_clean":"2023-01-15T14:34:06.908859+0000","last_became_active":"2023-01-15T14:33:52.425622+0000","last_became_peered":"2023-01-15T14:33:52.425622+0000","last_unstale":"2023-01-15T14:34:06.908859+0000","last_undegraded":"2023-01-15T14:34:06.908859+0000","last_fullsized":"2023-01-15T14:34:06.908859+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.60540339200000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1332,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.914627+0000","last_change":"2023-01-15T14:33:57.297022+0000","last_active":"2023-01-15T14:34:06.914627+0000","last_peered":"2023-01-15T14:34:06.914627+0000","last_clean":"2023-01-15T14:34:06.914627+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:34:06.914627+0000","last_undegraded":"2023-01-15T14:34:06.914627+0000","last_fullsized":"2023-01-15T14:34:06.914627+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:48.664","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.38900274200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1117,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:07.115746+0000","last_change":"2023-01-15T14:33:57.304492+0000","last_active":"2023-01-15T14:34:07.115746+0000","last_peered":"2023-01-15T14:34:07.115746+0000","last_clean":"2023-01-15T14:34:07.115746+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:34:07.115746+0000","last_undegraded":"2023-01-15T14:34:07.115746+0000","last_fullsized":"2023-01-15T14:34:07.115746+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:38.874","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.29516895799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1337,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.907138+0000","last_change":"2023-01-15T14:34:00.443594+0000","last_active":"2023-01-15T14:34:06.907138+0000","last_peered":"2023-01-15T14:34:06.907138+0000","last_clean":"2023-01-15T14:34:06.907138+0000","last_became_active":"2023-01-15T14:34:00.140247+0000","last_became_peered":"2023-01-15T14:34:00.140247+0000","last_unstale":"2023-01-15T14:34:06.907138+0000","last_undegraded":"2023-01-15T14:34:06.907138+0000","last_fullsized":"2023-01-15T14:34:06.907138+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.202704782,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"1.0","version":"19'117","reported_seq":521,"reported_epoch":356,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:05.184418+0000","last_change":"2023-01-15T14:33:58.506892+0000","last_active":"2023-01-15T14:34:05.184418+0000","last_peered":"2023-01-15T14:34:05.184418+0000","last_clean":"2023-01-15T14:33:50.143204+0000","last_became_active":"2023-01-15T14:33:58.306184+0000","last_became_peered":"2023-01-15T14:33:58.306184+0000","last_unstale":"2023-01-15T14:34:05.184418+0000","last_undegraded":"2023-01-15T14:33:58.003421+0000","last_fullsized":"2023-01-15T14:33:58.001929+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[1,0],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979004+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:34:04.979004+0000","last_peered":"2023-01-15T14:34:04.979004+0000","last_clean":"2023-01-15T14:34:04.979004+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:34:04.979004+0000","last_undegraded":"2023-01-15T14:34:04.979004+0000","last_fullsized":"2023-01-15T14:34:04.979004+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":1005,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.914447+0000","last_change":"2023-01-15T14:33:57.297870+0000","last_active":"2023-01-15T14:34:06.914447+0000","last_peered":"2023-01-15T14:34:06.914447+0000","last_clean":"2023-01-15T14:34:06.914447+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:34:06.914447+0000","last_undegraded":"2023-01-15T14:34:06.914447+0000","last_fullsized":"2023-01-15T14:34:06.914447+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.38976201700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":402,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979824+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:34:04.979824+0000","last_peered":"2023-01-15T14:34:04.979824+0000","last_clean":"2023-01-15T14:34:04.979824+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:34:04.979824+0000","last_undegraded":"2023-01-15T14:34:04.979824+0000","last_fullsized":"2023-01-15T14:34:04.979824+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.964","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":716,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.167951+0000","last_change":"2023-01-15T14:33:57.185185+0000","last_active":"2023-01-15T14:34:04.167951+0000","last_peered":"2023-01-15T14:34:04.167951+0000","last_clean":"2023-01-15T14:34:04.167951+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:34:04.167951+0000","last_undegraded":"2023-01-15T14:34:04.167951+0000","last_fullsized":"2023-01-15T14:34:04.167951+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.280577087,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":159,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.908505+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:34:06.908505+0000","last_peered":"2023-01-15T14:34:06.908505+0000","last_clean":"2023-01-15T14:34:06.908505+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:34:06.908505+0000","last_undegraded":"2023-01-15T14:34:06.908505+0000","last_fullsized":"2023-01-15T14:34:06.908505+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1327,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.085414+0000","last_change":"2023-01-15T14:33:57.830460+0000","last_active":"2023-01-15T14:34:05.085414+0000","last_peered":"2023-01-15T14:34:05.085414+0000","last_clean":"2023-01-15T14:34:05.085414+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:34:05.085414+0000","last_undegraded":"2023-01-15T14:34:05.085414+0000","last_fullsized":"2023-01-15T14:34:05.085414+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.52931870400000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":225,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.084855+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:34:05.084855+0000","last_peered":"2023-01-15T14:34:05.084855+0000","last_clean":"2023-01-15T14:34:05.084855+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:34:05.084855+0000","last_undegraded":"2023-01-15T14:34:05.084855+0000","last_fullsized":"2023-01-15T14:34:05.084855+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:25.583","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"352'325","reported_seq":1427,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.914119+0000","last_change":"2023-01-15T14:34:01.953246+0000","last_active":"2023-01-15T14:34:06.914119+0000","last_peered":"2023-01-15T14:34:06.914119+0000","last_clean":"2023-01-15T14:34:06.914119+0000","last_became_active":"2023-01-15T14:34:00.043073+0000","last_became_peered":"2023-01-15T14:34:00.043073+0000","last_unstale":"2023-01-15T14:34:06.914119+0000","last_undegraded":"2023-01-15T14:34:06.914119+0000","last_fullsized":"2023-01-15T14:34:06.914119+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:26.745","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.80204535600000004,"stat_sum":{"num_bytes":10343843,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":13875,"num_write":6,"num_write_kb":7330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9428925,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5","version":"355'160","reported_seq":1223,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:07.009268+0000","last_change":"2023-01-15T14:34:04.889983+0000","last_active":"2023-01-15T14:34:07.009268+0000","last_peered":"2023-01-15T14:34:07.009268+0000","last_clean":"2023-01-15T14:34:07.009268+0000","last_became_active":"2023-01-15T14:34:04.175473+0000","last_became_peered":"2023-01-15T14:34:04.175473+0000","last_unstale":"2023-01-15T14:34:07.009268+0000","last_undegraded":"2023-01-15T14:34:07.009268+0000","last_fullsized":"2023-01-15T14:34:07.009268+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.61306555500000004,"stat_sum":{"num_bytes":11390101,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":26145,"num_write":9,"num_write_kb":4265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":13260921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":215,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.984667+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:34:04.984667+0000","last_peered":"2023-01-15T14:34:04.984667+0000","last_clean":"2023-01-15T14:34:04.984667+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:34:04.984667+0000","last_undegraded":"2023-01-15T14:34:04.984667+0000","last_fullsized":"2023-01-15T14:34:04.984667+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"335'308","reported_seq":1452,"reported_epoch":357,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:34:07.740687+0000","last_change":"2023-01-15T14:34:06.376733+0000","last_active":"2023-01-15T14:34:07.740687+0000","last_peered":"2023-01-15T14:34:07.740687+0000","last_clean":"2023-01-15T14:33:50.017033+0000","last_became_active":"2023-01-15T14:33:53.422817+0000","last_became_peered":"2023-01-15T14:33:53.422817+0000","last_unstale":"2023-01-15T14:34:07.740687+0000","last_undegraded":"2023-01-15T14:34:07.740687+0000","last_fullsized":"2023-01-15T14:33:53.014012+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.70443229200000002,"stat_sum":{"num_bytes":2954643,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":30684,"num_write":31,"num_write_kb":10131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":7931053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2","objects":-1},{"shards":"1,2,4","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":404,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.178007+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:34:05.178007+0000","last_peered":"2023-01-15T14:34:05.178007+0000","last_clean":"2023-01-15T14:34:05.178007+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:34:05.178007+0000","last_undegraded":"2023-01-15T14:34:05.178007+0000","last_fullsized":"2023-01-15T14:34:05.178007+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"349'141","reported_seq":1143,"reported_epoch":356,"state":"active+recovery_wait+degraded+remapped","last_fresh":"2023-01-15T14:34:04.985389+0000","last_change":"2023-01-15T14:33:58.214638+0000","last_active":"2023-01-15T14:34:04.985389+0000","last_peered":"2023-01-15T14:34:04.985389+0000","last_clean":"2023-01-15T14:33:42.900201+0000","last_became_active":"2023-01-15T14:33:58.115294+0000","last_became_peered":"2023-01-15T14:33:58.115294+0000","last_unstale":"2023-01-15T14:34:04.985389+0000","last_undegraded":"2023-01-15T14:33:57.905179+0000","last_fullsized":"2023-01-15T14:34:04.985389+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.471195633,"stat_sum":{"num_bytes":9475691,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":13,"num_read_kb":13598,"num_write":13,"num_write_kb":8618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3546343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[1,3,0],"avail_no_missing":["1","3","4"],"object_location_counts":[{"shards":"0,1,3","objects":2},{"shards":"1,3,4","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":215,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.186729+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:34:05.186729+0000","last_peered":"2023-01-15T14:34:05.186729+0000","last_clean":"2023-01-15T14:34:05.186729+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:34:05.186729+0000","last_undegraded":"2023-01-15T14:34:05.186729+0000","last_fullsized":"2023-01-15T14:34:05.186729+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:17.360","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":41,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.169095+0000","last_change":"2023-01-15T14:33:56.909628+0000","last_active":"2023-01-15T14:34:04.169095+0000","last_peered":"2023-01-15T14:34:04.169095+0000","last_clean":"2023-01-15T14:34:04.169095+0000","last_became_active":"2023-01-15T14:33:52.206701+0000","last_became_peered":"2023-01-15T14:33:52.206701+0000","last_unstale":"2023-01-15T14:34:04.169095+0000","last_undegraded":"2023-01-15T14:34:04.169095+0000","last_fullsized":"2023-01-15T14:34:04.169095+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.0022256720000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.180246+0000","last_change":"2023-01-15T14:33:57.412557+0000","last_active":"2023-01-15T14:34:05.180246+0000","last_peered":"2023-01-15T14:34:05.180246+0000","last_clean":"2023-01-15T14:34:05.180246+0000","last_became_active":"2023-01-15T14:33:57.311981+0000","last_became_peered":"2023-01-15T14:33:57.311981+0000","last_unstale":"2023-01-15T14:34:05.180246+0000","last_undegraded":"2023-01-15T14:34:05.180246+0000","last_fullsized":"2023-01-15T14:34:05.180246+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"351'159","reported_seq":1189,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:07.015254+0000","last_change":"2023-01-15T14:33:58.010967+0000","last_active":"2023-01-15T14:34:07.015254+0000","last_peered":"2023-01-15T14:34:07.015254+0000","last_clean":"2023-01-15T14:34:07.015254+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:34:07.015254+0000","last_undegraded":"2023-01-15T14:34:07.015254+0000","last_fullsized":"2023-01-15T14:34:07.015254+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:32:19.303243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:19.303243+0000","objects_scrubbed":1,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.0010524499999999999,"stat_sum":{"num_bytes":1226927,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":18,"num_read_kb":20384,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":29,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.980053+0000","last_change":"2023-01-15T14:33:58.327664+0000","last_active":"2023-01-15T14:34:04.980053+0000","last_peered":"2023-01-15T14:34:04.980053+0000","last_clean":"2023-01-15T14:34:04.980053+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:34:04.980053+0000","last_undegraded":"2023-01-15T14:34:04.980053+0000","last_fullsized":"2023-01-15T14:34:04.980053+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31327724499999998,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.16","version":"352'129","reported_seq":642,"reported_epoch":356,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:05.186425+0000","last_change":"2023-01-15T14:33:58.219965+0000","last_active":"2023-01-15T14:34:05.186425+0000","last_peered":"2023-01-15T14:34:05.186425+0000","last_clean":"2023-01-15T14:33:55.270604+0000","last_became_active":"2023-01-15T14:33:58.218043+0000","last_became_peered":"2023-01-15T14:33:58.218043+0000","last_unstale":"2023-01-15T14:34:05.186425+0000","last_undegraded":"2023-01-15T14:33:57.906873+0000","last_fullsized":"2023-01-15T14:33:57.906218+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:09.523","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.30046122199999997,"stat_sum":{"num_bytes":7378471,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,2,3","objects":2},{"shards":"1,3","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":41,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.270064+0000","last_change":"2023-01-15T14:33:57.186529+0000","last_active":"2023-01-15T14:34:04.270064+0000","last_peered":"2023-01-15T14:34:04.270064+0000","last_clean":"2023-01-15T14:34:04.270064+0000","last_became_active":"2023-01-15T14:33:52.303197+0000","last_became_peered":"2023-01-15T14:33:52.303197+0000","last_unstale":"2023-01-15T14:34:04.270064+0000","last_undegraded":"2023-01-15T14:34:04.270064+0000","last_fullsized":"2023-01-15T14:34:04.270064+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.265135238,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.18","version":"355'328","reported_seq":1544,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.906949+0000","last_change":"2023-01-15T14:34:04.674366+0000","last_active":"2023-01-15T14:34:06.906949+0000","last_peered":"2023-01-15T14:34:06.906949+0000","last_clean":"2023-01-15T14:34:06.906949+0000","last_became_active":"2023-01-15T14:34:03.265912+0000","last_became_peered":"2023-01-15T14:34:03.265912+0000","last_unstale":"2023-01-15T14:34:06.906949+0000","last_undegraded":"2023-01-15T14:34:06.906949+0000","last_fullsized":"2023-01-15T14:34:06.906949+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.70539038499999995,"stat_sum":{"num_bytes":4332078,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":26,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":12147901,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.19","version":"294'256","reported_seq":1326,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.184763+0000","last_change":"2023-01-15T14:33:57.626542+0000","last_active":"2023-01-15T14:34:05.184763+0000","last_peered":"2023-01-15T14:34:05.184763+0000","last_clean":"2023-01-15T14:34:05.184763+0000","last_became_active":"2023-01-15T14:33:35.259904+0000","last_became_peered":"2023-01-15T14:33:35.259904+0000","last_unstale":"2023-01-15T14:34:05.184763+0000","last_undegraded":"2023-01-15T14:34:05.184763+0000","last_fullsized":"2023-01-15T14:34:05.184763+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:01.296","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.52734117999999996,"stat_sum":{"num_bytes":2420274,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1a","version":"348'352","reported_seq":1460,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.914272+0000","last_change":"2023-01-15T14:33:58.011960+0000","last_active":"2023-01-15T14:34:06.914272+0000","last_peered":"2023-01-15T14:34:06.914272+0000","last_clean":"2023-01-15T14:34:06.914272+0000","last_became_active":"2023-01-15T14:33:35.063699+0000","last_became_peered":"2023-01-15T14:33:35.063699+0000","last_unstale":"2023-01-15T14:34:06.914272+0000","last_undegraded":"2023-01-15T14:34:06.914272+0000","last_fullsized":"2023-01-15T14:34:06.914272+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.0039007270000000001,"stat_sum":{"num_bytes":3052653,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":15,"num_read_kb":21426,"num_write":7,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1b","version":"348'99","reported_seq":65,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.168643+0000","last_change":"2023-01-15T14:33:58.271017+0000","last_active":"2023-01-15T14:34:04.168643+0000","last_peered":"2023-01-15T14:34:04.168643+0000","last_clean":"2023-01-15T14:34:04.168643+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:34:04.168643+0000","last_undegraded":"2023-01-15T14:34:04.168643+0000","last_fullsized":"2023-01-15T14:34:04.168643+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.26204159500000002,"stat_sum":{"num_bytes":4400087,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10511,"num_write":14,"num_write_kb":4701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9416119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":551,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:07.014950+0000","last_change":"2023-01-15T14:34:00.229670+0000","last_active":"2023-01-15T14:34:07.014950+0000","last_peered":"2023-01-15T14:34:07.014950+0000","last_clean":"2023-01-15T14:34:07.014950+0000","last_became_active":"2023-01-15T14:33:57.125876+0000","last_became_peered":"2023-01-15T14:33:57.125876+0000","last_unstale":"2023-01-15T14:34:07.014950+0000","last_undegraded":"2023-01-15T14:34:07.014950+0000","last_fullsized":"2023-01-15T14:34:07.014950+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.0028318649999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1273,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:07.015179+0000","last_change":"2023-01-15T14:33:57.405395+0000","last_active":"2023-01-15T14:34:07.015179+0000","last_peered":"2023-01-15T14:34:07.015179+0000","last_clean":"2023-01-15T14:34:07.015179+0000","last_became_active":"2023-01-15T14:33:35.362912+0000","last_became_peered":"2023-01-15T14:33:35.362912+0000","last_unstale":"2023-01-15T14:34:07.015179+0000","last_undegraded":"2023-01-15T14:34:07.015179+0000","last_fullsized":"2023-01-15T14:34:07.015179+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.394560678,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":984,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.909267+0000","last_change":"2023-01-15T14:33:57.317895+0000","last_active":"2023-01-15T14:34:06.909267+0000","last_peered":"2023-01-15T14:34:06.909267+0000","last_clean":"2023-01-15T14:34:06.909267+0000","last_became_active":"2023-01-15T14:33:35.261561+0000","last_became_peered":"2023-01-15T14:33:35.261561+0000","last_unstale":"2023-01-15T14:34:06.909267+0000","last_undegraded":"2023-01-15T14:34:06.909267+0000","last_fullsized":"2023-01-15T14:34:06.909267+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40752737100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1f","version":"318'256","reported_seq":1330,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.169757+0000","last_change":"2023-01-15T14:34:00.448154+0000","last_active":"2023-01-15T14:34:04.169757+0000","last_peered":"2023-01-15T14:34:04.169757+0000","last_clean":"2023-01-15T14:34:04.169757+0000","last_became_active":"2023-01-15T14:34:00.240936+0000","last_became_peered":"2023-01-15T14:34:00.240936+0000","last_unstale":"2023-01-15T14:34:04.169757+0000","last_undegraded":"2023-01-15T14:34:04.169757+0000","last_fullsized":"2023-01-15T14:34:04.169757+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.20345100599999999,"stat_sum":{"num_bytes":2974888,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":13341,"num_write":1,"num_write_kb":1699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":5949777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.20","version":"324'273","reported_seq":1359,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.167589+0000","last_change":"2023-01-15T14:33:57.415738+0000","last_active":"2023-01-15T14:34:04.167589+0000","last_peered":"2023-01-15T14:34:04.167589+0000","last_clean":"2023-01-15T14:34:04.167589+0000","last_became_active":"2023-01-15T14:33:55.472883+0000","last_became_peered":"2023-01-15T14:33:55.472883+0000","last_unstale":"2023-01-15T14:34:04.167589+0000","last_undegraded":"2023-01-15T14:34:04.167589+0000","last_fullsized":"2023-01-15T14:34:04.167589+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.49046371599999999,"stat_sum":{"num_bytes":4251137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10864,"num_write":9,"num_write_kb":4140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4438968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.21","version":"321'138","reported_seq":1048,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.980964+0000","last_change":"2023-01-15T14:33:57.005196+0000","last_active":"2023-01-15T14:34:04.980964+0000","last_peered":"2023-01-15T14:34:04.980964+0000","last_clean":"2023-01-15T14:34:04.980964+0000","last_became_active":"2023-01-15T14:33:45.267321+0000","last_became_peered":"2023-01-15T14:33:45.267321+0000","last_unstale":"2023-01-15T14:34:04.980964+0000","last_undegraded":"2023-01-15T14:34:04.980964+0000","last_fullsized":"2023-01-15T14:34:04.980964+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.102357873,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.22","version":"323'161","reported_seq":1339,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.908116+0000","last_change":"2023-01-15T14:33:57.619172+0000","last_active":"2023-01-15T14:34:06.908116+0000","last_peered":"2023-01-15T14:34:06.908116+0000","last_clean":"2023-01-15T14:34:06.908116+0000","last_became_active":"2023-01-15T14:33:55.378253+0000","last_became_peered":"2023-01-15T14:33:55.378253+0000","last_unstale":"2023-01-15T14:34:06.908116+0000","last_undegraded":"2023-01-15T14:34:06.908116+0000","last_fullsized":"2023-01-15T14:34:06.908116+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.50935289699999997,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":743,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.168263+0000","last_change":"2023-01-15T14:33:57.416771+0000","last_active":"2023-01-15T14:34:04.168263+0000","last_peered":"2023-01-15T14:34:04.168263+0000","last_clean":"2023-01-15T14:34:04.168263+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:34:04.168263+0000","last_undegraded":"2023-01-15T14:34:04.168263+0000","last_fullsized":"2023-01-15T14:34:04.168263+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.45831782799999998,"stat_sum":{"num_bytes":3802136,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":21818,"num_write":18,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.24","version":"349'263","reported_seq":1405,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.909375+0000","last_change":"2023-01-15T14:33:57.616353+0000","last_active":"2023-01-15T14:34:06.909375+0000","last_peered":"2023-01-15T14:34:06.909375+0000","last_clean":"2023-01-15T14:34:06.909375+0000","last_became_active":"2023-01-15T14:33:45.075309+0000","last_became_peered":"2023-01-15T14:33:45.075309+0000","last_unstale":"2023-01-15T14:34:06.909375+0000","last_undegraded":"2023-01-15T14:34:06.909375+0000","last_fullsized":"2023-01-15T14:34:06.909375+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.70814871000000001,"stat_sum":{"num_bytes":7636039,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":10774,"num_write":10,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.25","version":"333'108","reported_seq":1120,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.085780+0000","last_change":"2023-01-15T14:33:57.331159+0000","last_active":"2023-01-15T14:34:05.085780+0000","last_peered":"2023-01-15T14:34:05.085780+0000","last_clean":"2023-01-15T14:34:05.085780+0000","last_became_active":"2023-01-15T14:33:45.168598+0000","last_became_peered":"2023-01-15T14:33:45.168598+0000","last_unstale":"2023-01-15T14:34:05.085780+0000","last_undegraded":"2023-01-15T14:34:05.085780+0000","last_fullsized":"2023-01-15T14:34:05.085780+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.33264791100000002,"stat_sum":{"num_bytes":2638765,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3030,"num_write":2,"num_write_kb":2577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":472,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.979581+0000","last_change":"2023-01-15T14:33:58.119558+0000","last_active":"2023-01-15T14:34:04.979581+0000","last_peered":"2023-01-15T14:34:04.979581+0000","last_clean":"2023-01-15T14:34:04.979581+0000","last_became_active":"2023-01-15T14:33:52.309110+0000","last_became_peered":"2023-01-15T14:33:52.309110+0000","last_unstale":"2023-01-15T14:34:04.979581+0000","last_undegraded":"2023-01-15T14:34:04.979581+0000","last_fullsized":"2023-01-15T14:34:04.979581+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.0016362760000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":1071,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.909402+0000","last_change":"2023-01-15T14:33:57.355572+0000","last_active":"2023-01-15T14:34:06.909402+0000","last_peered":"2023-01-15T14:34:06.909402+0000","last_clean":"2023-01-15T14:34:06.909402+0000","last_became_active":"2023-01-15T14:33:45.171090+0000","last_became_peered":"2023-01-15T14:33:45.171090+0000","last_unstale":"2023-01-15T14:34:06.909402+0000","last_undegraded":"2023-01-15T14:34:06.909402+0000","last_fullsized":"2023-01-15T14:34:06.909402+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.443073948,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.28","version":"333'248","reported_seq":1389,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.906626+0000","last_change":"2023-01-15T14:33:57.617361+0000","last_active":"2023-01-15T14:34:06.906626+0000","last_peered":"2023-01-15T14:34:06.906626+0000","last_clean":"2023-01-15T14:34:06.906626+0000","last_became_active":"2023-01-15T14:33:45.366831+0000","last_became_peered":"2023-01-15T14:33:45.366831+0000","last_unstale":"2023-01-15T14:34:06.906626+0000","last_undegraded":"2023-01-15T14:34:06.906626+0000","last_fullsized":"2023-01-15T14:34:06.906626+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.60974191300000002,"stat_sum":{"num_bytes":4838389,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8969,"num_write":8,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":6096764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":893,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.908349+0000","last_change":"2023-01-15T14:33:57.620313+0000","last_active":"2023-01-15T14:34:06.908349+0000","last_peered":"2023-01-15T14:34:06.908349+0000","last_clean":"2023-01-15T14:34:06.908349+0000","last_became_active":"2023-01-15T14:33:45.276960+0000","last_became_peered":"2023-01-15T14:33:45.276960+0000","last_unstale":"2023-01-15T14:34:06.908349+0000","last_undegraded":"2023-01-15T14:34:06.908349+0000","last_fullsized":"2023-01-15T14:34:06.908349+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.51033217900000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1238,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.078722+0000","last_change":"2023-01-15T14:33:56.904528+0000","last_active":"2023-01-15T14:34:05.078722+0000","last_peered":"2023-01-15T14:34:05.078722+0000","last_clean":"2023-01-15T14:34:05.078722+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:34:05.078722+0000","last_undegraded":"2023-01-15T14:34:05.078722+0000","last_fullsized":"2023-01-15T14:34:05.078722+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.0022419219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":51,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.168392+0000","last_change":"2023-01-15T14:33:57.893154+0000","last_active":"2023-01-15T14:34:04.168392+0000","last_peered":"2023-01-15T14:34:04.168392+0000","last_clean":"2023-01-15T14:34:04.168392+0000","last_became_active":"2023-01-15T14:33:57.261456+0000","last_became_peered":"2023-01-15T14:33:57.261456+0000","last_unstale":"2023-01-15T14:34:04.168392+0000","last_undegraded":"2023-01-15T14:34:04.168392+0000","last_fullsized":"2023-01-15T14:34:04.168392+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.630651882,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2c","version":"141'46","reported_seq":442,"reported_epoch":357,"state":"active+clean","last_fresh":"2023-01-15T14:34:06.914735+0000","last_change":"2023-01-15T14:33:57.303687+0000","last_active":"2023-01-15T14:34:06.914735+0000","last_peered":"2023-01-15T14:34:06.914735+0000","last_clean":"2023-01-15T14:34:06.914735+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:34:06.914735+0000","last_undegraded":"2023-01-15T14:34:06.914735+0000","last_fullsized":"2023-01-15T14:34:06.914735+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:10.970","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.29451823399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1211,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.077646+0000","last_change":"2023-01-15T14:33:58.122688+0000","last_active":"2023-01-15T14:34:05.077646+0000","last_peered":"2023-01-15T14:34:05.077646+0000","last_clean":"2023-01-15T14:34:05.077646+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:34:05.077646+0000","last_undegraded":"2023-01-15T14:34:05.077646+0000","last_fullsized":"2023-01-15T14:34:05.077646+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.105464771,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":29,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.978910+0000","last_change":"2023-01-15T14:33:58.326712+0000","last_active":"2023-01-15T14:34:04.978910+0000","last_peered":"2023-01-15T14:34:04.978910+0000","last_clean":"2023-01-15T14:34:04.978910+0000","last_became_active":"2023-01-15T14:33:57.210434+0000","last_became_peered":"2023-01-15T14:33:57.210434+0000","last_unstale":"2023-01-15T14:34:04.978910+0000","last_undegraded":"2023-01-15T14:34:04.978910+0000","last_fullsized":"2023-01-15T14:34:04.978910+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.31032643199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1105,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.081137+0000","last_change":"2023-01-15T14:33:58.121127+0000","last_active":"2023-01-15T14:34:05.081137+0000","last_peered":"2023-01-15T14:34:05.081137+0000","last_clean":"2023-01-15T14:34:05.081137+0000","last_became_active":"2023-01-15T14:33:55.174128+0000","last_became_peered":"2023-01-15T14:33:55.174128+0000","last_unstale":"2023-01-15T14:34:05.081137+0000","last_undegraded":"2023-01-15T14:34:05.081137+0000","last_fullsized":"2023-01-15T14:34:05.081137+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.20511622600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":981,"reported_epoch":355,"state":"active+clean","last_fresh":"2023-01-15T14:34:04.167692+0000","last_change":"2023-01-15T14:33:57.314582+0000","last_active":"2023-01-15T14:34:04.167692+0000","last_peered":"2023-01-15T14:34:04.167692+0000","last_clean":"2023-01-15T14:34:04.167692+0000","last_became_active":"2023-01-15T14:33:55.373498+0000","last_became_peered":"2023-01-15T14:33:55.373498+0000","last_unstale":"2023-01-15T14:34:04.167692+0000","last_undegraded":"2023-01-15T14:34:04.167692+0000","last_fullsized":"2023-01-15T14:34:04.167692+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":7,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.389510301,"stat_sum":{"num_bytes":5419866,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":25948,"num_write":35,"num_write_kb":13827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":13340458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.31","version":"204'150","reported_seq":1224,"reported_epoch":356,"state":"active+clean","last_fresh":"2023-01-15T14:34:05.084337+0000","last_change":"2023-01-15T14:33:57.625989+0000","last_active":"2023-01-15T14:34:05.084337+0000","last_peered":"2023-01-15T14:34:05.084337+0000","last_clean":"2023-01-15T14:34:05.084337+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:34:05.084337+0000","last_undegraded":"2023-01-15T14:34:05.084337+0000","last_fullsized":"2023-01-15T14:34:05.084337+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.52721113600000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":106,"stat_sum":{"num_bytes":355846432,"num_objects":164,"num_object_clones":113,"num_object_copies":492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":91,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":164,"num_whiteouts":14,"num_read":1017,"num_read_kb":1099202,"num_write":808,"num_write_kb":331645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":612,"num_bytes_recovered":385911300,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18702,"ondisk_log_size":18702,"up":318,"acting":312,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574922,"num_pgs":63,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1047320,"kb_used_data":92664956,"kb_used_omap":7766,"kb_used_meta":0,"kb_avail":92652060,"statfs":{"total":95948165120,"available":94875709440,"internally_reserved":13205504,"allocated":94888914944,"data_stored":94888914944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7952640,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":4.1269999999999998,"15min":4.0890000000000004},"min":{"1min":0.54000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2030000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":9.5380000000000003,"5min":6.3200000000000003,"15min":8.2460000000000004},"min":{"1min":0.58799999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":96.337000000000003,"5min":96.337000000000003,"15min":96.337000000000003},"last":1.0860000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":3.6509999999999998,"15min":5.5800000000000001},"min":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":34.585999999999999,"5min":88.003,"15min":88.003},"last":1.038},{"interface":"front","average":{"1min":13.302,"5min":8.1950000000000003,"15min":6.9180000000000001},"min":{"1min":0.58799999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.403999999999996,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.91000000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":6.9400000000000004,"15min":5.5069999999999997},"min":{"1min":0.57199999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.179,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":2.5790000000000002,"15min":3.645},"min":{"1min":0.626,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":91.837999999999994,"15min":91.837999999999994},"last":1.143}]},{"osd":3,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":7.3799999999999999,"15min":3.0920000000000001},"min":{"1min":0.63600000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2370000000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":1.173},{"interface":"front","average":{"1min":0.91600000000000004,"5min":3.6360000000000001,"15min":4.3129999999999997},"min":{"1min":0.65000000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.2270000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":1.266}]},{"osd":4,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5519999999999996,"5min":5.6219999999999999,"15min":8.1509999999999998},"min":{"1min":0.63500000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":96.185000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.9259999999999999,"5min":5.782,"15min":2.5699999999999998},"min":{"1min":0.73099999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":22.41,"5min":164.21199999999999,"15min":164.21199999999999},"last":1.3069999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.6440000000000001,"5min":5.0659999999999998,"15min":5.9059999999999997},"min":{"1min":0.72799999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":96.263999999999996,"5min":96.263999999999996,"15min":96.263999999999996},"last":1.339},{"interface":"front","average":{"1min":0.998,"5min":3.206,"15min":4.3650000000000002},"min":{"1min":0.69299999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.355,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.3640000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":5.5949999999999998,"15min":7.5469999999999997},"min":{"1min":0.83599999999999997,"5min":0.496,"15min":0.496},"max":{"1min":96.299999999999997,"5min":96.299999999999997,"15min":96.299999999999997},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.089,"5min":2.5150000000000001,"15min":1.571},"min":{"1min":0.84299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.2789999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.409}]}]},{"osd":6,"up_from":0,"seq":55834574923,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574923,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574922,"num_pgs":66,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1045252,"kb_used_data":92667860,"kb_used_omap":13875,"kb_used_meta":0,"kb_avail":92654128,"statfs":{"total":95948165120,"available":94877827072,"internally_reserved":14061568,"allocated":94891888640,"data_stored":94891888640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14208207,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":23.594000000000001,"15min":11.141},"min":{"1min":0.50700000000000001,"5min":0.44,"15min":0.44},"max":{"1min":1.133,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.78300000000000003,"5min":3.0009999999999999,"15min":1.5209999999999999},"min":{"1min":0.58299999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.024,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.72799999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":5.0730000000000004,"15min":7.4199999999999999},"min":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6659999999999999,"5min":171.91499999999999,"15min":171.91499999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.343,"15min":1.9770000000000001},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0529999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":1.036}]},{"osd":2,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":24.227,"15min":11.749000000000001},"min":{"1min":0.5,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.157,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":2.6040000000000001,"5min":1.21,"15min":1.01},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":32.081000000000003,"5min":32.081000000000003,"15min":32.081000000000003},"last":1.1160000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":2.9169999999999998,"15min":1.865},"min":{"1min":0.65300000000000002,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.2909999999999999,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.7370000000000001,"5min":24.498999999999999,"15min":12.266999999999999},"min":{"1min":0.64400000000000002,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":14.961,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":0.68600000000000005}]},{"osd":5,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.88500000000000001,"15min":0.90800000000000003},"min":{"1min":0.55200000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":2.1840000000000002,"15min":2.2599999999999998},"min":{"1min":0.71799999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.3200000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.79200000000000004}]},{"osd":6,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":3.9329999999999998,"15min":5.093},"min":{"1min":0.69599999999999995,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.365,"5min":99.780000000000001,"15min":99.780000000000001},"last":1.097},{"interface":"front","average":{"1min":1.0489999999999999,"5min":22.463999999999999,"15min":10.93},"min":{"1min":0.71599999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.427,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.0740000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:33:13 2023","interfaces":[{"interface":"back","average":{"1min":3.1539999999999999,"5min":5.5570000000000004,"15min":10.603999999999999},"min":{"1min":0.76700000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":39.012999999999998,"5min":179.73500000000001,"15min":179.73500000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.2370000000000001,"5min":24.864000000000001,"15min":15.779999999999999},"min":{"1min":0.877,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":3.0680000000000001,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.004}]}]},{"osd":3,"up_from":13,"seq":55834574922,"num_pgs":64,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1101588,"kb_used_data":92597792,"kb_used_omap":14108,"kb_used_meta":0,"kb_avail":92597792,"statfs":{"total":95948165120,"available":94820139008,"internally_reserved":0,"allocated":94820139008,"data_stored":94820139008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14446597,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":7.4400000000000004,"15min":5.7119999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":17.731000000000002,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.75800000000000001,"5min":3.4340000000000002,"15min":6.2389999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.9260000000000002,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.84999999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":7.1790000000000003,"15min":11.414},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96799999999999997,"5min":159.256,"15min":159.256},"last":1.4750000000000001},{"interface":"front","average":{"1min":2.6259999999999999,"5min":10.417999999999999,"15min":3.9660000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":17.826000000000001,"5min":174.578,"15min":174.578},"last":1.381}]},{"osd":2,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":4.6849999999999996,"5min":2.8809999999999998,"15min":3.1930000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":86.412000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.95,"5min":6.7889999999999997,"15min":3.7250000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.876000000000001,"5min":174.45099999999999,"15min":174.45099999999999},"last":1.147}]},{"osd":4,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":7.6879999999999997,"5min":29.686,"15min":14.27},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":84.640000000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.86899999999999999,"5min":32.134999999999998,"15min":19.960999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.615,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":0.996}]},{"osd":5,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":6.4980000000000002,"15min":11.433999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.04,"5min":159.43000000000001,"15min":159.43000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":2.2309999999999999,"5min":6.0039999999999996,"15min":2.6819999999999999},"min":{"1min":0.70099999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":17.917000000000002,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.80700000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1949999999999998,"5min":5.9980000000000002,"15min":2.6869999999999998},"min":{"1min":0.66000000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":17.989000000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.964,"15min":3.1619999999999999},"min":{"1min":0.70499999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.23,"5min":60.098999999999997,"15min":60.098999999999997},"last":1.0289999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":2.0219999999999998,"15min":3.3639999999999999},"min":{"1min":0.69499999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.266,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.103},{"interface":"front","average":{"1min":19.138000000000002,"5min":11.705,"15min":15.961},"min":{"1min":0.66200000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":281.13400000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":1.3440000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574922,"num_pgs":50,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1064524,"kb_used_data":92634996,"kb_used_omap":11690,"kb_used_meta":0,"kb_avail":92634856,"statfs":{"total":95948165120,"available":94858092544,"internally_reserved":143360,"allocated":94858235904,"data_stored":94858235904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11970995,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":3.109,"5min":3.5230000000000001,"15min":1.601},"min":{"1min":0.437,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":51.009999999999998,"5min":66.277000000000001,"15min":66.277000000000001},"last":0.621},{"interface":"front","average":{"1min":0.66100000000000003,"5min":1.4299999999999999,"15min":0.95299999999999996},"min":{"1min":0.36099999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.0089999999999999,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.64900000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.1529999999999996,"5min":5.9059999999999997,"15min":2.5009999999999999},"min":{"1min":0.378,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":91.525999999999996,"5min":143.071,"15min":143.071},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":2.016,"15min":2.9089999999999998},"min":{"1min":0.42299999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.131,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.55300000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":8.2210000000000001,"15min":9.6750000000000007},"min":{"1min":0.5,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":5.3609999999999998,"5min":183.53,"15min":183.53},"last":0.67300000000000004},{"interface":"front","average":{"1min":16.692,"5min":7.5170000000000003,"15min":3.0169999999999999},"min":{"1min":0.61199999999999999,"5min":0.316,"15min":0.316},"max":{"1min":155.47900000000001,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.70799999999999996}]},{"osd":4,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":5.3650000000000002,"5min":6.5060000000000002,"15min":4.6020000000000003},"min":{"1min":0.67300000000000004,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":51.259999999999998,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":2.8809999999999998,"5min":5.2939999999999996,"15min":4.3040000000000003},"min":{"1min":0.48699999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":33.219999999999999,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.83199999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":2.4729999999999999,"15min":1.6080000000000001},"min":{"1min":0.57199999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.2130000000000001,"5min":143.262,"15min":143.262},"last":1.02},{"interface":"front","average":{"1min":3.4780000000000002,"5min":3.0539999999999998,"15min":1.792},"min":{"1min":0.70999999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":51.326000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.80600000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.857,"15min":1.3069999999999999},"min":{"1min":0.60399999999999998,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.365,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.018,"5min":2.6160000000000001,"15min":1.6839999999999999},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.5,"5min":143.215,"15min":143.215},"last":1.069}]},{"osd":7,"last update":"Sun Jan 15 14:33:10 2023","interfaces":[{"interface":"back","average":{"1min":13.092000000000001,"5min":6.3120000000000003,"15min":4.1429999999999998},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":120.905,"5min":120.905,"15min":120.905},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.0129999999999999,"5min":2.8450000000000002,"15min":2.7989999999999999},"min":{"1min":0.66600000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.552,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.96499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":72,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1063640,"kb_used_data":92635740,"kb_used_omap":13180,"kb_used_meta":0,"kb_avail":92635740,"statfs":{"total":95948165120,"available":94858997760,"internally_reserved":0,"allocated":94858997760,"data_stored":94858997760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13496648,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":3.633,"15min":3.806},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":3.9140000000000001,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":7.742,"5min":4.6130000000000004,"15min":4.4189999999999996},"min":{"1min":0.374,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":119.39100000000001,"5min":119.39100000000001,"15min":119.39100000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.3849999999999998,"5min":2.6469999999999998,"15min":1.5269999999999999},"min":{"1min":0.29799999999999999,"5min":0.245,"15min":0.245},"max":{"1min":95.879999999999995,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.85999999999999999,"5min":1.8620000000000001,"15min":3.3359999999999999},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.9780000000000002,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.67700000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":5.8920000000000003,"5min":9.6750000000000007,"15min":16.449000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":100.137,"5min":282.01400000000001,"15min":282.01400000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":7.782,"5min":6.8899999999999997,"15min":7.1260000000000003},"min":{"1min":0.45600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":119.804,"5min":119.804,"15min":119.804},"last":0.94299999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:33:17 2023","interfaces":[{"interface":"back","average":{"1min":7.9249999999999998,"5min":17.140999999999998,"15min":8.8650000000000002},"min":{"1min":0.504,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":119.71899999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":1.278},{"interface":"front","average":{"1min":1.7849999999999999,"5min":15.526,"15min":7.5780000000000003},"min":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":16.989999999999998,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":15.141}]},{"osd":5,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":1.5449999999999999,"15min":1.2030000000000001},"min":{"1min":0.59899999999999998,"5min":0.433,"15min":0.433},"max":{"1min":1.278,"5min":65.120999999999995,"15min":65.120999999999995},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.6579999999999999,"15min":3.5110000000000001},"min":{"1min":0.60599999999999998,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":4.0780000000000003,"5min":46.905999999999999,"15min":46.905999999999999},"last":1.0820000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":8.0069999999999997,"5min":5.0129999999999999,"15min":5.0049999999999999},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":119.759,"5min":119.759,"15min":119.759},"last":1.05},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.603,"15min":1.2829999999999999},"min":{"1min":0.61499999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.349,"5min":65.156000000000006,"15min":65.156000000000006},"last":1.014}]},{"osd":7,"last update":"Sun Jan 15 14:33:12 2023","interfaces":[{"interface":"back","average":{"1min":3.6030000000000002,"5min":6.1109999999999998,"15min":6.7149999999999999},"min":{"1min":0.56000000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":31.620999999999999,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.98099999999999998},{"interface":"front","average":{"1min":8.0779999999999994,"5min":5.0060000000000002,"15min":4.835},"min":{"1min":0.68999999999999995,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":119.65300000000001,"5min":119.65300000000001,"15min":119.65300000000001},"last":1.2070000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574923,"num_pgs":70,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1045672,"kb_used_data":92687888,"kb_used_omap":7766,"kb_used_meta":0,"kb_avail":92653708,"statfs":{"total":95948165120,"available":94877396992,"internally_reserved":35000320,"allocated":94912397312,"data_stored":94912397312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7953103,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":3.1219999999999999,"15min":1.456},"min":{"1min":0.40899999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.1060000000000001,"5min":69.872,"15min":69.872},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.70899999999999996,"5min":3.2309999999999999,"15min":1.5629999999999999},"min":{"1min":0.51300000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.2350000000000001,"5min":98.203999999999994,"15min":98.203999999999994},"last":0.52800000000000002}]},{"osd":2,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":3.4220000000000002,"15min":1.603},"min":{"1min":0.46999999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.444,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.74099999999999999,"5min":2.6560000000000001,"15min":1.3580000000000001},"min":{"1min":0.52000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.302,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.83799999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":17.571999999999999,"5min":10.789,"15min":14.782},"min":{"1min":0.53300000000000003,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":199.08000000000001,"5min":284.62799999999999,"15min":284.62799999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.835,"5min":7.306,"15min":2.9689999999999999},"min":{"1min":0.56599999999999995,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":15.601000000000001,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.89200000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.92900000000000005,"5min":16.695,"15min":6.2220000000000004},"min":{"1min":0.73199999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1879999999999999,"5min":1486.309,"15min":1486.309},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.038,"5min":17.925000000000001,"15min":6.6580000000000004},"min":{"1min":0.82499999999999996,"5min":0.439,"15min":0.439},"max":{"1min":1.3759999999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":1.115}]},{"osd":5,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":4.6749999999999998,"15min":8.0899999999999999},"min":{"1min":0.62,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.304,"5min":176.34399999999999,"15min":176.34399999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.008,"5min":1.73,"15min":1.2689999999999999},"min":{"1min":0.69199999999999995,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.2529999999999999,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.99299999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":3.5910000000000002,"15min":1.9770000000000001},"min":{"1min":0.67800000000000005,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.5960000000000001,"5min":70.259,"15min":70.259},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.0880000000000001,"5min":3.0209999999999999,"15min":1.7629999999999999},"min":{"1min":0.83699999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.5489999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":1.141}]},{"osd":7,"last update":"Sun Jan 15 14:33:11 2023","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":6.1200000000000001,"15min":12.487},"min":{"1min":0.78400000000000003,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.48,"5min":292.61099999999999,"15min":292.61099999999999},"last":1.091},{"interface":"front","average":{"1min":1.117,"5min":3.6070000000000002,"15min":1.9830000000000001},"min":{"1min":0.76100000000000001,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.5229999999999999,"5min":70.322000000000003,"15min":70.322000000000003},"last":1.0509999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:34:11.867 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:11.308+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:11.869 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:11.866+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:11.970 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:11.964+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:12.269 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:12.269 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:34:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGRecoveryDeleteReply(3.10 e361,343 3:094fbf13:::smithi09025264-31:7,34'48)", 2023-01-15T14:34:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:11.890787+0000", 2023-01-15T14:34:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.093515529999999999, 2023-01-15T14:34:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.093553608999999996, 2023-01-15T14:34:12.270 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "no flag points reached", 2023-01-15T14:34:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:12.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:11.890787+0000", 2023-01-15T14:34:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:12.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:11.890787+0000", 2023-01-15T14:34:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:12.273 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:11.890785+0000", 2023-01-15T14:34:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:11.890797+0000", 2023-01-15T14:34:12.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2031999999999999e-05 2023-01-15T14:34:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:11.890826+0000", 2023-01-15T14:34:12.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8425e-05 2023-01-15T14:34:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:12.276 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:34:12.277 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:34:12.277 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:12.278 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:34:12.280 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:12.469 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:12.469 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:12.470 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:12.470 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:12.470 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:12.470 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:12.470 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:12.472 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:34:12.472 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:34:12.473 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:34:12.473 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:34:12.473 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:34:12.473 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:34:12.473 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:34:12.474 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:34:12.475 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:12.262+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:12.516 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:12.512+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:12.671 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:12.671 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:12.672 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:34:12.672 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:12.666+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:12.772 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:12.770+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:12.877 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:12.878 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:12.878 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:34:12.879 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:12.874+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:12.978 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:12.972+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:13.040 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:13.040 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:13.040 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:34:13.079 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:13.074+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:13.203 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:13.203 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:13.203 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:34:13.204 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:13.198+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:13.304 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:13.300+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:13.363 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:34:13.365 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:13.366 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:13.366 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:34:13.405 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:13.402+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:13.532 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:13.526+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:13.632 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:13.628+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:13.707 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:13.707 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:13.708 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:34:13.751 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:13.748+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:13.834 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:13.830+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:13.860 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:13.860 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:13.870 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:34:13.935 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:13.932+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:13.950 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:34:13.981 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:13.981 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:13.991 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:34:14.036 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:14.032+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:14.109 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:14.109 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:14.119 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:34:14.424 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:14.420+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:14.475 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:34:14.536 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:14.530+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:14.547 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:14.547 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:14.556 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:34:14.687 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:14.684+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:14.689 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:14.709 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:14.727 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:14.852 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:14.852 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:14.852 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:14.853 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:14.853 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:14.854 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:14.854 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:34:14.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:34:14.854 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:34:14.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.855 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3 3.3 3:ca59e2b0:::smithi09025264-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:14.855 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:14.855 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 349.024503917, 2023-01-15T14:34:14.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62968594499999997, 2023-01-15T14:34:14.855 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:14.855 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:14.855 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:14.855 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:14.856 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:14.856 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3 2023-01-15T14:34:14.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.856 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:14.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:14.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:14.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:14.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.857 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:14.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:14.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:14.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:14.858 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775131+0000", 2023-01-15T14:34:14.858 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:14.858 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.858 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:14.858 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775138+0000", 2023-01-15T14:34:14.858 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:34:14.858 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:14.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775148+0000", 2023-01-15T14:34:14.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7969999999999995e-06 2023-01-15T14:34:14.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:14.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775161+0000", 2023-01-15T14:34:14.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2469999999999999e-05 2023-01-15T14:34:14.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.860 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:14.860 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.857974+0000", 2023-01-15T14:34:14.860 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.082812887000000002 2023-01-15T14:34:14.860 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.860 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:14.860 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.858147+0000", 2023-01-15T14:34:14.860 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000173883 2023-01-15T14:34:14.860 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.861 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.861 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:14.861 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106098+0000", 2023-01-15T14:34:14.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24795019600000001 2023-01-15T14:34:14.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.861 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.861 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:14.861 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106181+0000", 2023-01-15T14:34:14.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3248e-05 2023-01-15T14:34:14.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.862 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:34:14.862 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106330+0000", 2023-01-15T14:34:14.862 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014918200000000001 2023-01-15T14:34:14.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:14.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106510+0000", 2023-01-15T14:34:14.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017979399999999999 2023-01-15T14:34:14.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:14.864 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106527+0000", 2023-01-15T14:34:14.864 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6643999999999999e-05 2023-01-15T14:34:14.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.864 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:14.864 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106605+0000", 2023-01-15T14:34:14.864 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8650000000000001e-05 2023-01-15T14:34:14.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.865 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:14.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106644+0000", 2023-01-15T14:34:14.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8686999999999997e-05 2023-01-15T14:34:14.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.865 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:14.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404687+0000", 2023-01-15T14:34:14.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.298042745 2023-01-15T14:34:14.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.866 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:14.866 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404735+0000", 2023-01-15T14:34:14.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8306000000000001e-05 2023-01-15T14:34:14.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.866 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.866 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:14.866 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404772+0000", 2023-01-15T14:34:14.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6977000000000002e-05 2023-01-15T14:34:14.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:14.867 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404818+0000", 2023-01-15T14:34:14.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.6439e-05 2023-01-15T14:34:14.867 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:14.867 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:14.867 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:14.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.868 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:14.868 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:14.868 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 349.02446585000001, 2023-01-15T14:34:14.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62995383500000002, 2023-01-15T14:34:14.868 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:14.868 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:14.868 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:14.868 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:14.869 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:14.869 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4 2023-01-15T14:34:14.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.869 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:14.869 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.869 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:14.869 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:14.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:14.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.870 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.870 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:14.870 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:14.870 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:14.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:14.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:14.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:14.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:14.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775174+0000", 2023-01-15T14:34:14.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3669999999999999e-06 2023-01-15T14:34:14.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:14.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775184+0000", 2023-01-15T14:34:14.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5869999999999999e-06 2023-01-15T14:34:14.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:14.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775190+0000", 2023-01-15T14:34:14.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:34:14.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:14.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868440+0000", 2023-01-15T14:34:14.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.093250126000000003 2023-01-15T14:34:14.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:14.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868555+0000", 2023-01-15T14:34:14.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115003 2023-01-15T14:34:14.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.874 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.874 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:14.874 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.302962+0000", 2023-01-15T14:34:14.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.43440729700000003 2023-01-15T14:34:14.874 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:14.875 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:14.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:14.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303197+0000", 2023-01-15T14:34:15.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023508000000000001 2023-01-15T14:34:15.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404866+0000", 2023-01-15T14:34:15.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101668723 2023-01-15T14:34:15.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404985+0000", 2023-01-15T14:34:15.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118853 2023-01-15T14:34:15.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.405124+0000", 2023-01-15T14:34:15.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013951999999999999 2023-01-15T14:34:15.013 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.013 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.013 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.014 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:144 3.9 3:9988ac9d:::smithi09025264-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:15.014 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:15.014 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 347.93312730399998, 2023-01-15T14:34:15.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63423976199999998, 2023-01-15T14:34:15.014 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.014 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:15.014 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.014 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.014 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.015 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 144 2023-01-15T14:34:15.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.015 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:15.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.019 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.019 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:15.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.019 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.019 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866508+0000", 2023-01-15T14:34:15.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:15.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866512+0000", 2023-01-15T14:34:15.020 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5999999999999998e-06 2023-01-15T14:34:15.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866523+0000", 2023-01-15T14:34:15.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0740000000000001e-05 2023-01-15T14:34:15.021 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866531+0000", 2023-01-15T14:34:15.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3310000000000004e-06 2023-01-15T14:34:15.021 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867744+0000", 2023-01-15T14:34:15.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012127399999999999 2023-01-15T14:34:15.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.022 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:15.022 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867810+0000", 2023-01-15T14:34:15.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6506000000000003e-05 2023-01-15T14:34:15.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.022 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.022 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097225+0000", 2023-01-15T14:34:15.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.22941514900000001 2023-01-15T14:34:15.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:15.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097295+0000", 2023-01-15T14:34:15.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:34:15.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500386+0000", 2023-01-15T14:34:15.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40309051699999998 2023-01-15T14:34:15.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500521+0000", 2023-01-15T14:34:15.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013561700000000001 2023-01-15T14:34:15.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500749+0000", 2023-01-15T14:34:15.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022754900000000001 2023-01-15T14:34:15.025 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.025 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.025 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.026 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:509 3.9 3:9f86bdcb:::smithi09025264-22:head [write 2873212~588997] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:15.026 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:15.026 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 324.98470296099998, 2023-01-15T14:34:15.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69623497099999998, 2023-01-15T14:34:15.026 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.026 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:15.026 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.027 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.027 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.027 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 509 2023-01-15T14:34:15.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.027 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:15.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:15.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814931+0000", 2023-01-15T14:34:15.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:15.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.142 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.143 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815512+0000", 2023-01-15T14:34:15.143 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058082200000000004 2023-01-15T14:34:15.144 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.145 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.145 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815553+0000", 2023-01-15T14:34:15.145 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1081999999999997e-05 2023-01-15T14:34:15.146 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.146 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.146 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815571+0000", 2023-01-15T14:34:15.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8028000000000001e-05 2023-01-15T14:34:15.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906452+0000", 2023-01-15T14:34:15.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090880586999999999 2023-01-15T14:34:15.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:15.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906564+0000", 2023-01-15T14:34:15.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111976 2023-01-15T14:34:15.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208424+0000", 2023-01-15T14:34:15.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.301860199 2023-01-15T14:34:15.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208718+0000", 2023-01-15T14:34:15.187 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029347299999999999 2023-01-15T14:34:15.188 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.189 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.189 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:15.189 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209175+0000", 2023-01-15T14:34:15.189 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045692399999999999 2023-01-15T14:34:15.189 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209725+0000", 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055027600000000004 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209751+0000", 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5812999999999998e-05 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.210006+0000", 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025564099999999997 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311247+0000", 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10124119199999999 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311454+0000", 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020695800000000001 2023-01-15T14:34:15.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511022+0000", 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19956729200000001 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511100+0000", 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8589000000000003e-05 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511168+0000", 2023-01-15T14:34:15.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8039000000000004e-05 2023-01-15T14:34:15.193 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.193 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.193 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.193 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:510 3.9 3:9f86bdcb:::smithi09025264-22:head [setxattr _header (57) in=7b,truncate 3462209] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:15.193 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 324.98403949800002, 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69609656399999997, 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 510 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.196 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:15.338 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.338 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815596+0000", 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815606+0000", 2023-01-15T14:34:15.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7920000000000004e-06 2023-01-15T14:34:15.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815627+0000", 2023-01-15T14:34:15.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1239000000000001e-05 2023-01-15T14:34:15.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815637+0000", 2023-01-15T14:34:15.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0349e-05 2023-01-15T14:34:15.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.341 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906600+0000", 2023-01-15T14:34:15.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090963098000000006 2023-01-15T14:34:15.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.341 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:15.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906717+0000", 2023-01-15T14:34:15.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000116759 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310000+0000", 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40328322599999999 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310249+0000", 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024874400000000002 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:15.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310559+0000", 2023-01-15T14:34:15.343 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310382 2023-01-15T14:34:15.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:15.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310951+0000", 2023-01-15T14:34:15.343 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039124500000000003 2023-01-15T14:34:15.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:15.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310975+0000", 2023-01-15T14:34:15.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4496000000000001e-05 2023-01-15T14:34:15.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311041+0000", 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6309000000000005e-05 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311260+0000", 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021863399999999999 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311588+0000", 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032813599999999998 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511351+0000", 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19976275700000001 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511507+0000", 2023-01-15T14:34:15.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015585100000000001 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511693+0000", 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186627 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:511 3.9 3:9f86bdcb:::smithi09025264-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:15.348 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:15.349 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 324.98398213000002, 2023-01-15T14:34:15.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69666647500000001, 2023-01-15T14:34:15.349 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.349 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:15.350 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.350 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.350 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.350 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 511 2023-01-15T14:34:15.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.350 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:15.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815653+0000", 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815660+0000", 2023-01-15T14:34:15.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0509999999999999e-06 2023-01-15T14:34:15.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815683+0000", 2023-01-15T14:34:15.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2742999999999999e-05 2023-01-15T14:34:15.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815698+0000", 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5133999999999999e-05 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908029+0000", 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.092331448999999996 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908193+0000", 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016342799999999999 2023-01-15T14:34:15.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311115+0000", 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40292222300000002 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311224+0000", 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010888899999999999 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511800+0000", 2023-01-15T14:34:15.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20057596899999999 2023-01-15T14:34:15.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512036+0000", 2023-01-15T14:34:15.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236247 2023-01-15T14:34:15.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512321+0000", 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028457599999999998 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:693 3.3 3:ca59e2b0:::smithi09025264-1:head [sparse-read 0~2016807] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 306.90124878300003, 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70514199600000005, 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.356 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 693 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:15.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898387+0000", 2023-01-15T14:34:15.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:15.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898390+0000", 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.252e-06 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898401+0000", 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0977e-05 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.998532+0000", 2023-01-15T14:34:15.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100130443 2023-01-15T14:34:15.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602663+0000", 2023-01-15T14:34:15.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60413100500000005 2023-01-15T14:34:15.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.538 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602817+0000", 2023-01-15T14:34:15.539 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015440500000000001 2023-01-15T14:34:15.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.603530+0000", 2023-01-15T14:34:15.539 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071246299999999999 2023-01-15T14:34:15.539 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.539 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1421 3.a 3:5919d83e:::smithi09025264-13:head [write 1723287~549075] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 239.88733427400001, 2023-01-15T14:34:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61118175699999999, 2023-01-15T14:34:15.540 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1421 2023-01-15T14:34:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912300+0000", 2023-01-15T14:34:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913127+0000", 2023-01-15T14:34:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00082704000000000002 2023-01-15T14:34:15.543 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.544 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.544 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913158+0000", 2023-01-15T14:34:15.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1146999999999998e-05 2023-01-15T14:34:15.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.544 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.544 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913177+0000", 2023-01-15T14:34:15.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:34:15.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913384+0000", 2023-01-15T14:34:15.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000206589 2023-01-15T14:34:15.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:15.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913408+0000", 2023-01-15T14:34:15.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3493e-05 2023-01-15T14:34:15.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318579+0000", 2023-01-15T14:34:15.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40517134399999999 2023-01-15T14:34:15.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318806+0000", 2023-01-15T14:34:15.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022680899999999999 2023-01-15T14:34:15.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:15.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319060+0000", 2023-01-15T14:34:15.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254069 2023-01-15T14:34:15.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:15.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319527+0000", 2023-01-15T14:34:15.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000467081 2023-01-15T14:34:15.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:15.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319544+0000", 2023-01-15T14:34:15.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6949999999999999e-05 2023-01-15T14:34:15.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:15.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319778+0000", 2023-01-15T14:34:15.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023391500000000001 2023-01-15T14:34:15.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:15.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421126+0000", 2023-01-15T14:34:15.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101348626 2023-01-15T14:34:15.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522834+0000", 2023-01-15T14:34:15.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10170761 2023-01-15T14:34:15.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523142+0000", 2023-01-15T14:34:15.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854800000000001 2023-01-15T14:34:15.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:15.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523251+0000", 2023-01-15T14:34:15.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108411 2023-01-15T14:34:15.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523484+0000", 2023-01-15T14:34:15.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023292500000000001 2023-01-15T14:34:15.551 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.551 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.551 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.552 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1422 3.a 3:5919d83e:::smithi09025264-13:head [write 2781837~703343] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:15.552 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:15.552 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 239.886395478, 2023-01-15T14:34:15.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61062001700000001, 2023-01-15T14:34:15.552 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.552 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:15.552 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.553 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.553 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.553 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1422 2023-01-15T14:34:15.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.553 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:15.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:15.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913239+0000", 2023-01-15T14:34:15.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:15.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914075+0000", 2023-01-15T14:34:15.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00083594099999999998 2023-01-15T14:34:15.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914104+0000", 2023-01-15T14:34:15.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8938999999999999e-05 2023-01-15T14:34:15.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914121+0000", 2023-01-15T14:34:15.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7306999999999999e-05 2023-01-15T14:34:15.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914234+0000", 2023-01-15T14:34:15.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112737 2023-01-15T14:34:15.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:15.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914257+0000", 2023-01-15T14:34:15.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3156e-05 2023-01-15T14:34:15.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319798+0000", 2023-01-15T14:34:15.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40554153100000001 2023-01-15T14:34:15.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319991+0000", 2023-01-15T14:34:15.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019217999999999999 2023-01-15T14:34:15.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:15.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320237+0000", 2023-01-15T14:34:15.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024621099999999998 2023-01-15T14:34:15.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:15.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320759+0000", 2023-01-15T14:34:15.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052174700000000003 2023-01-15T14:34:15.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:15.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320776+0000", 2023-01-15T14:34:15.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7173e-05 2023-01-15T14:34:15.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:15.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.321080+0000", 2023-01-15T14:34:15.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030424400000000001 2023-01-15T14:34:15.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:15.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421162+0000", 2023-01-15T14:34:15.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100082085 2023-01-15T14:34:15.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522963+0000", 2023-01-15T14:34:15.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10180109399999999 2023-01-15T14:34:15.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523576+0000", 2023-01-15T14:34:15.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061277499999999997 2023-01-15T14:34:15.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:15.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523664+0000", 2023-01-15T14:34:15.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8002000000000002e-05 2023-01-15T14:34:15.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523861+0000", 2023-01-15T14:34:15.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019685899999999999 2023-01-15T14:34:15.564 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.564 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.564 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.564 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1423 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 3630312] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:15.564 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:15.564 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 239.885472943, 2023-01-15T14:34:15.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70862486899999999, 2023-01-15T14:34:15.565 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.565 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:15.565 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.565 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.565 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.565 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1423 2023-01-15T14:34:15.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.565 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:15.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:15.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914162+0000", 2023-01-15T14:34:15.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:15.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914172+0000", 2023-01-15T14:34:15.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0267e-05 2023-01-15T14:34:15.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914188+0000", 2023-01-15T14:34:15.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6212999999999999e-05 2023-01-15T14:34:15.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914201+0000", 2023-01-15T14:34:15.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3073e-05 2023-01-15T14:34:15.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914315+0000", 2023-01-15T14:34:15.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011357399999999999 2023-01-15T14:34:15.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:15.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914334+0000", 2023-01-15T14:34:15.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9004e-05 2023-01-15T14:34:15.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421258+0000", 2023-01-15T14:34:15.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.506924606 2023-01-15T14:34:15.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421518+0000", 2023-01-15T14:34:15.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025950499999999999 2023-01-15T14:34:15.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:15.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421843+0000", 2023-01-15T14:34:15.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032498299999999999 2023-01-15T14:34:15.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:15.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422222+0000", 2023-01-15T14:34:15.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037867599999999999 2023-01-15T14:34:15.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:15.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422245+0000", 2023-01-15T14:34:15.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3096999999999999e-05 2023-01-15T14:34:15.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:15.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422343+0000", 2023-01-15T14:34:15.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8487999999999996e-05 2023-01-15T14:34:15.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.753 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:15.753 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522890+0000", 2023-01-15T14:34:15.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10054658900000001 2023-01-15T14:34:15.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.753 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.753 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523075+0000", 2023-01-15T14:34:15.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185386 2023-01-15T14:34:15.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622151+0000", 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099075724000000004 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622244+0000", 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3387999999999994e-05 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622788+0000", 2023-01-15T14:34:15.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000543973 2023-01-15T14:34:15.755 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.755 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.755 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.755 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1424 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:15.755 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:15.755 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 239.885417571, 2023-01-15T14:34:15.755 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70984700199999995, 2023-01-15T14:34:15.755 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.755 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:15.756 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.756 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.756 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.756 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1424 2023-01-15T14:34:15.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.756 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:15.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914218+0000", 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914224+0000", 2023-01-15T14:34:15.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6769999999999998e-06 2023-01-15T14:34:15.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914242+0000", 2023-01-15T14:34:15.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7759000000000001e-05 2023-01-15T14:34:15.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014331+0000", 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088819 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114642+0000", 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100311027 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114682+0000", 2023-01-15T14:34:15.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0028999999999999e-05 2023-01-15T14:34:15.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.760 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.760 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523918+0000", 2023-01-15T14:34:15.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40923631500000002 2023-01-15T14:34:15.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.760 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:15.760 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524124+0000", 2023-01-15T14:34:15.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020524900000000001 2023-01-15T14:34:15.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622921+0000", 2023-01-15T14:34:15.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098797069000000001 2023-01-15T14:34:15.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.623328+0000", 2023-01-15T14:34:15.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000407539 2023-01-15T14:34:15.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.624066+0000", 2023-01-15T14:34:15.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073762899999999995 2023-01-15T14:34:15.762 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.762 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.762 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.762 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1430 3.a 3:52df5138:::smithi09025264-25:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:15.763 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:15.763 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 239.78525780999999, 2023-01-15T14:34:15.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81872653100000004, 2023-01-15T14:34:15.763 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.763 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:15.763 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.763 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.763 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.763 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1430 2023-01-15T14:34:15.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.763 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:15.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:15.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014376+0000", 2023-01-15T14:34:15.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:15.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014391+0000", 2023-01-15T14:34:15.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4922999999999999e-05 2023-01-15T14:34:15.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014417+0000", 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5698999999999999e-05 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014435+0000", 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7827000000000001e-05 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114759+0000", 2023-01-15T14:34:15.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100324859 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114771+0000", 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524222+0000", 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40945100800000001 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524545+0000", 2023-01-15T14:34:15.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000322367 2023-01-15T14:34:15.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:15.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.626981+0000", 2023-01-15T14:34:15.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102436308 2023-01-15T14:34:15.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:15.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627874+0000", 2023-01-15T14:34:15.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089274899999999997 2023-01-15T14:34:15.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:15.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627930+0000", 2023-01-15T14:34:15.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6122000000000002e-05 2023-01-15T14:34:15.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.769 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.769 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628609+0000", 2023-01-15T14:34:15.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00067958700000000005 2023-01-15T14:34:15.770 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:15.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628798+0000", 2023-01-15T14:34:15.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018809 2023-01-15T14:34:15.770 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:15.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628870+0000", 2023-01-15T14:34:15.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2844999999999998e-05 2023-01-15T14:34:15.771 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.771 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.771 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.771 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832404+0000", 2023-01-15T14:34:15.771 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20353310299999999 2023-01-15T14:34:15.771 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.771 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.771 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:15.771 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832606+0000", 2023-01-15T14:34:15.772 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020207999999999999 2023-01-15T14:34:15.772 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.772 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.772 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.772 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833105+0000", 2023-01-15T14:34:15.772 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049945399999999998 2023-01-15T14:34:15.772 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.772 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.772 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.772 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.772 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.773 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1431 3.a 3:52df5138:::smithi09025264-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:15.773 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:15.773 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 239.785166522, 2023-01-15T14:34:15.773 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81967516600000001, 2023-01-15T14:34:15.773 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.773 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:15.773 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.773 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.773 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.773 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1431 2023-01-15T14:34:15.773 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.774 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:15.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:15.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014469+0000", 2023-01-15T14:34:15.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:15.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014477+0000", 2023-01-15T14:34:15.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9300000000000003e-06 2023-01-15T14:34:15.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.776 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.776 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014495+0000", 2023-01-15T14:34:15.776 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8363999999999999e-05 2023-01-15T14:34:15.776 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.776 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.776 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.776 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114610+0000", 2023-01-15T14:34:15.776 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10011521700000001 2023-01-15T14:34:15.776 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.776 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.776 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214948+0000", 2023-01-15T14:34:15.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100337739 2023-01-15T14:34:15.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:15.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214962+0000", 2023-01-15T14:34:15.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4516e-05 2023-01-15T14:34:15.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524874+0000", 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30991196599999998 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.525013+0000", 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013873999999999999 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628222+0000", 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103208721 2023-01-15T14:34:15.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:15.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628504+0000", 2023-01-15T14:34:15.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028245700000000001 2023-01-15T14:34:15.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833214+0000", 2023-01-15T14:34:15.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.204709691 2023-01-15T14:34:15.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833555+0000", 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034094000000000003 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.834145+0000", 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059006900000000003 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.780 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.781 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2439 3.3 3:c11c7d81:::smithi09025264-11:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:34:15.781 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:15.781 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 139.71626201699999, 2023-01-15T14:34:15.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70719632099999996, 2023-01-15T14:34:15.781 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.781 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:15.781 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.781 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.781 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2439 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:15.782 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083373+0000", 2023-01-15T14:34:15.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:15.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083380+0000", 2023-01-15T14:34:15.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:34:15.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083391+0000", 2023-01-15T14:34:15.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0366e-05 2023-01-15T14:34:15.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083397+0000", 2023-01-15T14:34:15.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2740000000000004e-06 2023-01-15T14:34:15.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083648+0000", 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025052599999999999 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083666+0000", 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8848999999999999e-05 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237725+0000", 2023-01-15T14:34:15.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.154058361 2023-01-15T14:34:15.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.947 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.947 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:15.947 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237945+0000", 2023-01-15T14:34:15.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021978300000000001 2023-01-15T14:34:15.948 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.948 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.948 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.948 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.386967+0000", 2023-01-15T14:34:15.948 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.149022562 2023-01-15T14:34:15.948 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.948 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.948 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.948 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387230+0000", 2023-01-15T14:34:15.949 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026256100000000002 2023-01-15T14:34:15.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387695+0000", 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046480099999999998 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388112+0000", 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041713200000000002 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:15.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388130+0000", 2023-01-15T14:34:15.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8315999999999999e-05 2023-01-15T14:34:15.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:15.952 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388189+0000", 2023-01-15T14:34:15.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8665999999999997e-05 2023-01-15T14:34:15.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.952 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:15.952 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388829+0000", 2023-01-15T14:34:15.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00063992699999999999 2023-01-15T14:34:15.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.952 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.953 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.489989+0000", 2023-01-15T14:34:15.953 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116059199999999 2023-01-15T14:34:15.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.953 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:15.953 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790300+0000", 2023-01-15T14:34:15.953 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30031108699999998 2023-01-15T14:34:15.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.953 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790396+0000", 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5721000000000001e-05 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790571+0000", 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017464999999999999 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.954 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2440 3.3 3:c11c7d81:::smithi09025264-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 139.71617028599999, 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70759506000000005, 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2440 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.955 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083465+0000", 2023-01-15T14:34:15.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:15.960 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.960 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.960 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:15.961 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083470+0000", 2023-01-15T14:34:15.961 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2630000000000003e-06 2023-01-15T14:34:15.961 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.961 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.961 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:15.961 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083481+0000", 2023-01-15T14:34:15.961 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0699999999999999e-05 2023-01-15T14:34:15.961 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.961 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.961 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:15.961 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083488+0000", 2023-01-15T14:34:15.962 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8870000000000003e-06 2023-01-15T14:34:15.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.962 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.962 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.962 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180331+0000", 2023-01-15T14:34:15.962 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.096843296999999995 2023-01-15T14:34:15.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.962 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.962 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:15.962 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180342+0000", 2023-01-15T14:34:15.962 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.031e-05 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238012+0000", 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.057670829 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238257+0000", 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000244959 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590126+0000", 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35186851699999999 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590389+0000", 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026276799999999998 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:15.964 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790692+0000", 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20030350299999999 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790891+0000", 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019893999999999999 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.791061+0000", 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169913 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.965 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:15.966 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:15.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:15.966 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:15.966 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2450 3.a 3:52c3e133:::smithi09025264-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e230)", 2023-01-15T14:34:15.966 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:15.966 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 137.952317674, 2023-01-15T14:34:15.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.68251353100000001, 2023-01-15T14:34:15.966 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:16.563 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:16.563 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:16.563 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:16.563 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:16.564 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2450 2023-01-15T14:34:16.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:16.564 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:16.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:16.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:16.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:16.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:16.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:16.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:16.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:16.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:16.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:16.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:16.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:16.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:16.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847318+0000", 2023-01-15T14:34:16.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:16.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:16.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:16.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:16.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847324+0000", 2023-01-15T14:34:16.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:34:16.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:16.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:16.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:16.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847339+0000", 2023-01-15T14:34:16.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5582000000000001e-05 2023-01-15T14:34:16.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:16.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:16.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:16.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847349+0000", 2023-01-15T14:34:16.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0139999999999999e-05 2023-01-15T14:34:16.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:16.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:16.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:16.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947644+0000", 2023-01-15T14:34:16.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10029455600000001 2023-01-15T14:34:16.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:16.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:16.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:16.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947657+0000", 2023-01-15T14:34:16.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2796000000000001e-05 2023-01-15T14:34:16.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:16.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:16.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:16.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125638+0000", 2023-01-15T14:34:16.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17798127799999999 2023-01-15T14:34:16.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:16.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:18.406 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125780+0000", 2023-01-15T14:34:18.407 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014173000000000001 2023-01-15T14:34:18.407 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.407 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.407 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:18.407 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529035+0000", 2023-01-15T14:34:18.407 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40325542199999997 2023-01-15T14:34:18.407 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.407 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.407 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:18.407 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529366+0000", 2023-01-15T14:34:18.408 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033111700000000002 2023-01-15T14:34:18.408 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.408 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.408 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:18.408 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529832+0000", 2023-01-15T14:34:18.408 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046592399999999999 2023-01-15T14:34:18.408 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:18.408 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:18.408 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:18.408 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.408 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.409 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3390 3.1b 3:d85f42d9:::smithi09025264-45:head [write 1944996~791497] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:18.409 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:18.409 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 39.739638964999997, 2023-01-15T14:34:18.409 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.611364087, 2023-01-15T14:34:18.409 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:18.612 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:18.612 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:18.612 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:18.612 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:18.612 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3390 2023-01-15T14:34:18.612 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.612 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:18.612 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.612 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059995+0000", 2023-01-15T14:34:18.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061164+0000", 2023-01-15T14:34:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001168325 2023-01-15T14:34:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061194+0000", 2023-01-15T14:34:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.0485999999999999e-05 2023-01-15T14:34:18.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061213+0000", 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.891e-05 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159748+0000", 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098535175000000003 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.615 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159766+0000", 2023-01-15T14:34:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:34:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263011+0000", 2023-01-15T14:34:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10324554699999999 2023-01-15T14:34:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263253+0000", 2023-01-15T14:34:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000241568 2023-01-15T14:34:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466464+0000", 2023-01-15T14:34:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.203211006 2023-01-15T14:34:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466638+0000", 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017412100000000001 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466911+0000", 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000272971 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467425+0000", 2023-01-15T14:34:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00051373 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467457+0000", 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2558999999999998e-05 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467906+0000", 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000448985 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469895+0000", 2023-01-15T14:34:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0019888810000000001 2023-01-15T14:34:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470611+0000", 2023-01-15T14:34:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071556699999999996 2023-01-15T14:34:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671178+0000", 2023-01-15T14:34:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20056716699999999 2023-01-15T14:34:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671277+0000", 2023-01-15T14:34:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9594999999999998e-05 2023-01-15T14:34:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671361+0000", 2023-01-15T14:34:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4048000000000006e-05 2023-01-15T14:34:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3391 3.1b 3:d85f42d9:::smithi09025264-45:head [write 3196560~554574] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 39.738395842000003, 2023-01-15T14:34:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71066412899999998, 2023-01-15T14:34:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:18.625 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:18.625 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:18.625 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:18.625 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:18.625 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3391 2023-01-15T14:34:18.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.625 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:18.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:18.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:18.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:18.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:18.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:18.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:18.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:18.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061238+0000", 2023-01-15T14:34:18.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:18.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:18.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062134+0000", 2023-01-15T14:34:18.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089621299999999998 2023-01-15T14:34:18.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:18.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062160+0000", 2023-01-15T14:34:18.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5908999999999999e-05 2023-01-15T14:34:18.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:18.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062172+0000", 2023-01-15T14:34:18.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.189e-05 2023-01-15T14:34:18.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:18.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159827+0000", 2023-01-15T14:34:18.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097654866000000007 2023-01-15T14:34:18.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159846+0000", 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8516e-05 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263326+0000", 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103480766 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263535+0000", 2023-01-15T14:34:18.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020861299999999999 2023-01-15T14:34:18.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:18.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467576+0000", 2023-01-15T14:34:18.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20404118600000001 2023-01-15T14:34:18.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:18.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467736+0000", 2023-01-15T14:34:18.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016000699999999999 2023-01-15T14:34:18.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:18.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467987+0000", 2023-01-15T14:34:18.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025042199999999998 2023-01-15T14:34:18.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:18.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468369+0000", 2023-01-15T14:34:18.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038238600000000002 2023-01-15T14:34:18.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:18.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468387+0000", 2023-01-15T14:34:18.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7730000000000001e-05 2023-01-15T14:34:18.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:18.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468719+0000", 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033272599999999999 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470317+0000", 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001597336 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570851+0000", 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10053426 2023-01-15T14:34:18.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:18.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771618+0000", 2023-01-15T14:34:18.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200767152 2023-01-15T14:34:18.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:18.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771768+0000", 2023-01-15T14:34:18.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015015699999999999 2023-01-15T14:34:18.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771905+0000", 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013621099999999999 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3392 3.1b 3:d85f42d9:::smithi09025264-45:head [setxattr _header (58) in=7b,truncate 3882483] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 39.737446607000003, 2023-01-15T14:34:18.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71008243400000004, 2023-01-15T14:34:18.636 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:18.636 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:18.636 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:18.636 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:18.636 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:18.636 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3392 2023-01-15T14:34:18.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.636 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:18.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:18.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:18.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:18.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062188+0000", 2023-01-15T14:34:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062195+0000", 2023-01-15T14:34:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2529999999999998e-06 2023-01-15T14:34:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062208+0000", 2023-01-15T14:34:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3003000000000001e-05 2023-01-15T14:34:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:18.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062215+0000", 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5200000000000003e-06 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159883+0000", 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097667532000000001 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159899+0000", 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6759999999999999e-05 2023-01-15T14:34:18.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263853+0000", 2023-01-15T14:34:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10395361 2023-01-15T14:34:18.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264116+0000", 2023-01-15T14:34:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026336099999999999 2023-01-15T14:34:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468602+0000", 2023-01-15T14:34:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20448550900000001 2023-01-15T14:34:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468867+0000", 2023-01-15T14:34:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026495700000000002 2023-01-15T14:34:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469376+0000", 2023-01-15T14:34:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00050893100000000001 2023-01-15T14:34:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469789+0000", 2023-01-15T14:34:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000413158 2023-01-15T14:34:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469806+0000", 2023-01-15T14:34:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7513e-05 2023-01-15T14:34:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469884+0000", 2023-01-15T14:34:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7816000000000002e-05 2023-01-15T14:34:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470332+0000", 2023-01-15T14:34:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044807700000000001 2023-01-15T14:34:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570976+0000", 2023-01-15T14:34:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10064426899999999 2023-01-15T14:34:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772058+0000", 2023-01-15T14:34:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20108148200000001 2023-01-15T14:34:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772169+0000", 2023-01-15T14:34:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011120099999999999 2023-01-15T14:34:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772272+0000", 2023-01-15T14:34:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000102982 2023-01-15T14:34:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3393 3.1b 3:d85f42d9:::smithi09025264-45:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 39.737412530999997, 2023-01-15T14:34:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71076842399999995, 2023-01-15T14:34:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3393 2023-01-15T14:34:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062223+0000", 2023-01-15T14:34:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062227+0000", 2023-01-15T14:34:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3900000000000003e-06 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062237+0000", 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4539999999999996e-06 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062243+0000", 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0329999999999997e-06 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161412+0000", 2023-01-15T14:34:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099168708999999994 2023-01-15T14:34:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161424+0000", 2023-01-15T14:34:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2415999999999999e-05 2023-01-15T14:34:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264185+0000", 2023-01-15T14:34:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.1027609 2023-01-15T14:34:19.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:19.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264421+0000", 2023-01-15T14:34:19.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023640900000000001 2023-01-15T14:34:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469985+0000", 2023-01-15T14:34:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.205564001 2023-01-15T14:34:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470264+0000", 2023-01-15T14:34:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027918700000000002 2023-01-15T14:34:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772390+0000", 2023-01-15T14:34:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30212558299999998 2023-01-15T14:34:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772684+0000", 2023-01-15T14:34:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029361799999999999 2023-01-15T14:34:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772992+0000", 2023-01-15T14:34:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854100000000001 2023-01-15T14:34:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:19.035 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:19.035 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:34:19.036 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:14.846+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:34:19.036 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:34:19.037 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:34:19.037 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:34:19.037 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:34:19.037 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:34:19.038 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:34:19.038 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:34:19.038 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:34:19.039 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:34:19.045 INFO:tasks.rados.rados.0.smithi090.stdout:1384: finishing rollback tid 1 to smithi09025264-33 2023-01-15T14:34:19.045 INFO:tasks.rados.rados.0.smithi090.stdout:1384: finishing rollback tid 2 to smithi09025264-33 2023-01-15T14:34:19.045 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 33 v 203 (ObjNum 378 snap 102 seq_num 378) dirty exists 2023-01-15T14:34:19.046 INFO:tasks.rados.rados.0.smithi090.stdout:1384: done (0 left) 2023-01-15T14:34:19.046 INFO:tasks.rados.rados.0.smithi090.stdout:1387: done (0 left) 2023-01-15T14:34:19.047 INFO:tasks.rados.rados.0.smithi090.stdout:1388: read oid 40 snap -1 2023-01-15T14:34:19.047 INFO:tasks.rados.rados.0.smithi090.stdout:1388: expect (ObjNum 419 snap 112 seq_num 419) 2023-01-15T14:34:19.047 INFO:tasks.rados.rados.0.smithi090.stdout:1389: copy_from oid 1 from oid 49 current snap is 147 2023-01-15T14:34:19.047 INFO:tasks.rados.rados.0.smithi090.stdout:1390: read oid 37 snap -1 2023-01-15T14:34:19.047 INFO:tasks.rados.rados.0.smithi090.stdout:1390: expect (ObjNum 464 snap 133 seq_num 464) 2023-01-15T14:34:19.047 INFO:tasks.rados.rados.0.smithi090.stdout:1391: rollback oid 39 current snap is 147 2023-01-15T14:34:19.048 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 39 to 128 2023-01-15T14:34:19.048 INFO:tasks.rados.rados.0.smithi090.stdout:1392: snap_create 2023-01-15T14:34:19.048 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:34:19.048 INFO:tasks.rados.rados.0.smithi090.stdout:1388: expect (ObjNum 419 snap 112 seq_num 419) 2023-01-15T14:34:19.048 INFO:tasks.rados.rados.0.smithi090.stdout:1388: done (3 left) 2023-01-15T14:34:19.223 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:34:19.223 INFO:tasks.rados.rados.0.smithi090.stdout:1390: expect (ObjNum 464 snap 133 seq_num 464) 2023-01-15T14:34:19.223 INFO:tasks.rados.rados.0.smithi090.stdout:1390: done (2 left) 2023-01-15T14:34:19.224 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:34:19.224 INFO:tasks.rados.rados.0.smithi090.stdout:1389: finishing copy_from to smithi09025264-1 2023-01-15T14:34:19.224 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 361 (ObjNum 497 snap 146 seq_num 497) dirty exists 2023-01-15T14:34:19.224 INFO:tasks.rados.rados.0.smithi090.stdout:1389: finishing copy_from racing read to smithi09025264-1 2023-01-15T14:34:19.224 INFO:tasks.rados.rados.0.smithi090.stdout:1391: finishing rollback tid 0 to smithi09025264-39 2023-01-15T14:34:19.224 INFO:tasks.rados.rados.0.smithi090.stdout:1389: done (1 left) 2023-01-15T14:34:19.224 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:19.224 INFO:tasks.rados.rados.0.smithi090.stdout:1391: finishing rollback tid 1 to smithi09025264-39 2023-01-15T14:34:19.224 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 317 (ObjNum 33 snap 0 seq_num 3193457584) dirty dne 2023-01-15T14:34:19.224 INFO:tasks.rados.rados.0.smithi090.stdout:1391: done (0 left) 2023-01-15T14:34:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1392: done (0 left) 2023-01-15T14:34:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1393: read oid 24 snap 128 2023-01-15T14:34:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1393: expect (ObjNum 412 snap 112 seq_num 412) 2023-01-15T14:34:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1394: snap_create 2023-01-15T14:34:19.225 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1393: expect (ObjNum 412 snap 112 seq_num 412) 2023-01-15T14:34:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1393: done (0 left) 2023-01-15T14:34:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1394: done (0 left) 2023-01-15T14:34:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1395: write (excl) oid 19 current snap is 149 2023-01-15T14:34:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1395: seq_num 499 ranges {786693=712196,2150397=236296} 2023-01-15T14:34:19.225 INFO:tasks.rados.rados.0.smithi090.stdout:1395: writing smithi09025264-19 from 786693 to 1498889 tid 1 2023-01-15T14:34:19.226 INFO:tasks.rados.rados.0.smithi090.stdout:1395: writing smithi09025264-19 from 2150397 to 2386693 tid 2 2023-01-15T14:34:19.226 INFO:tasks.rados.rados.0.smithi090.stdout:1396: snap_remove snap 125 2023-01-15T14:34:19.226 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:19.035+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:19.228 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:19.228 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:19.228 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:34:19.228 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:19.229 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:34:19.230 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:19.223+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:19.329 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:19.323+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:19.436 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:19.436 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:19.436 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:34:19.436 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:34:19.436 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:19.437 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:34:19.437 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:19.431+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:19.538 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:19.543 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:34:19.543 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:19.539+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:19.554 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":548,"stamp":"2023-01-15T14:34:18.998446+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":360016807,"num_objects":168,"num_object_clones":114,"num_object_copies":504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":168,"num_whiteouts":14,"num_read":1132,"num_read_kb":1109359,"num_write":970,"num_write_kb":336359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":732,"num_bytes_recovered":468860199,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18845,"ondisk_log_size":18845,"up":345,"acting":342,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":373,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6333856,"kb_used_data":555912356,"kb_used_omap":69112,"kb_used_meta":0,"kb_avail":555862424,"statfs":{"total":575688990720,"available":569203122176,"internally_reserved":51130368,"allocated":569254252544,"data_stored":569254252544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70770963,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":16,"commit_latency_ns":0,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3580007,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-66,"num_objects_misplaced":-27,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":-1,"num_read_kb":10057,"num_write":3,"num_write_kb":2298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":52622759,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.042151"},"pg_stats":[{"pgid":"3.69","version":"110'149","reported_seq":915,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077255+0000","last_change":"2023-01-15T14:33:57.316235+0000","last_active":"2023-01-15T14:34:17.077255+0000","last_peered":"2023-01-15T14:34:17.077255+0000","last_clean":"2023-01-15T14:34:17.077255+0000","last_became_active":"2023-01-15T14:33:50.216417+0000","last_became_peered":"2023-01-15T14:33:50.216417+0000","last_unstale":"2023-01-15T14:34:17.077255+0000","last_undegraded":"2023-01-15T14:34:17.077255+0000","last_fullsized":"2023-01-15T14:34:17.077255+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.406510072,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.68","version":"363'256","reported_seq":1436,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.971126+0000","last_change":"2023-01-15T14:34:15.154245+0000","last_active":"2023-01-15T14:34:15.971126+0000","last_peered":"2023-01-15T14:34:15.971126+0000","last_clean":"2023-01-15T14:34:15.971126+0000","last_became_active":"2023-01-15T14:34:13.134258+0000","last_became_peered":"2023-01-15T14:34:13.134258+0000","last_unstale":"2023-01-15T14:34:15.971126+0000","last_undegraded":"2023-01-15T14:34:15.971126+0000","last_fullsized":"2023-01-15T14:34:15.971126+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.18941453,"stat_sum":{"num_bytes":8014979,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17027823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.67","version":"191'79","reported_seq":68,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070541+0000","last_change":"2023-01-15T14:33:57.312535+0000","last_active":"2023-01-15T14:34:16.070541+0000","last_peered":"2023-01-15T14:34:16.070541+0000","last_clean":"2023-01-15T14:34:16.070541+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:34:16.070541+0000","last_undegraded":"2023-01-15T14:34:16.070541+0000","last_fullsized":"2023-01-15T14:34:16.070541+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.40508863699999997,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.66","version":"102'52","reported_seq":378,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.078285+0000","last_change":"2023-01-15T14:33:57.299510+0000","last_active":"2023-01-15T14:34:17.078285+0000","last_peered":"2023-01-15T14:34:17.078285+0000","last_clean":"2023-01-15T14:34:17.078285+0000","last_became_active":"2023-01-15T14:33:48.695009+0000","last_became_peered":"2023-01-15T14:33:48.695009+0000","last_unstale":"2023-01-15T14:34:17.078285+0000","last_undegraded":"2023-01-15T14:34:17.078285+0000","last_fullsized":"2023-01-15T14:34:17.078285+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.390920724,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2142,"num_write":4,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.65","version":"367'124","reported_seq":1229,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.801494+0000","last_change":"2023-01-15T14:34:17.802215+0000","last_active":"2023-01-15T14:34:17.801494+0000","last_peered":"2023-01-15T14:34:17.801494+0000","last_clean":"2023-01-15T14:34:17.801494+0000","last_became_active":"2023-01-15T14:34:16.276292+0000","last_became_peered":"2023-01-15T14:34:16.276292+0000","last_unstale":"2023-01-15T14:34:17.801494+0000","last_undegraded":"2023-01-15T14:34:17.801494+0000","last_fullsized":"2023-01-15T14:34:17.801494+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.71301477300000005,"stat_sum":{"num_bytes":7224063,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":3033,"num_write":12,"num_write_kb":4201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":20989326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.64","version":"334'251","reported_seq":1424,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.063314+0000","last_change":"2023-01-15T14:33:57.417656+0000","last_active":"2023-01-15T14:34:15.063314+0000","last_peered":"2023-01-15T14:34:15.063314+0000","last_clean":"2023-01-15T14:34:15.063314+0000","last_became_active":"2023-01-15T14:33:55.675085+0000","last_became_peered":"2023-01-15T14:33:55.675085+0000","last_unstale":"2023-01-15T14:34:15.063314+0000","last_undegraded":"2023-01-15T14:34:15.063314+0000","last_fullsized":"2023-01-15T14:34:15.063314+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.409133782,"stat_sum":{"num_bytes":3548733,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10773,"num_write":5,"num_write_kb":2741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.63","version":"149'233","reported_seq":753,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.062800+0000","last_change":"2023-01-15T14:33:57.518738+0000","last_active":"2023-01-15T14:34:15.062800+0000","last_peered":"2023-01-15T14:34:15.062800+0000","last_clean":"2023-01-15T14:34:15.062800+0000","last_became_active":"2023-01-15T14:33:55.277739+0000","last_became_peered":"2023-01-15T14:33:55.277739+0000","last_unstale":"2023-01-15T14:34:15.062800+0000","last_undegraded":"2023-01-15T14:34:15.062800+0000","last_fullsized":"2023-01-15T14:34:15.062800+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.41396429299999998,"stat_sum":{"num_bytes":3802136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":21819,"num_write":19,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.62","version":"348'169","reported_seq":1361,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.975292+0000","last_change":"2023-01-15T14:33:58.705842+0000","last_active":"2023-01-15T14:34:15.975292+0000","last_peered":"2023-01-15T14:34:15.975292+0000","last_clean":"2023-01-15T14:34:15.975292+0000","last_became_active":"2023-01-15T14:33:56.590183+0000","last_became_peered":"2023-01-15T14:33:56.590183+0000","last_unstale":"2023-01-15T14:34:15.975292+0000","last_undegraded":"2023-01-15T14:34:15.975292+0000","last_fullsized":"2023-01-15T14:34:15.975292+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.70374445200000002,"stat_sum":{"num_bytes":5361549,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6734,"num_write":11,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5210269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.61","version":"321'138","reported_seq":1014,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.963020+0000","last_change":"2023-01-15T14:33:57.185692+0000","last_active":"2023-01-15T14:34:14.963020+0000","last_peered":"2023-01-15T14:34:14.963020+0000","last_clean":"2023-01-15T14:34:14.963020+0000","last_became_active":"2023-01-15T14:33:53.421056+0000","last_became_peered":"2023-01-15T14:33:53.421056+0000","last_unstale":"2023-01-15T14:34:14.963020+0000","last_undegraded":"2023-01-15T14:34:14.963020+0000","last_fullsized":"2023-01-15T14:34:14.963020+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.27715718700000003,"stat_sum":{"num_bytes":1033664,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6059,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.60","version":"348'281","reported_seq":1424,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.974641+0000","last_change":"2023-01-15T14:33:58.810339+0000","last_active":"2023-01-15T14:34:15.974641+0000","last_peered":"2023-01-15T14:34:15.974641+0000","last_clean":"2023-01-15T14:34:15.974641+0000","last_became_active":"2023-01-15T14:33:56.478613+0000","last_became_peered":"2023-01-15T14:33:56.478613+0000","last_unstale":"2023-01-15T14:34:15.974641+0000","last_undegraded":"2023-01-15T14:34:15.974641+0000","last_fullsized":"2023-01-15T14:34:15.974641+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.80913330699999997,"stat_sum":{"num_bytes":5214108,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":10865,"num_write":14,"num_write_kb":5082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15418818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1387,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.964478+0000","last_change":"2023-01-15T14:33:57.313450+0000","last_active":"2023-01-15T14:34:14.964478+0000","last_peered":"2023-01-15T14:34:14.964478+0000","last_clean":"2023-01-15T14:34:14.964478+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:34:14.964478+0000","last_undegraded":"2023-01-15T14:34:14.964478+0000","last_fullsized":"2023-01-15T14:34:14.964478+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:51.919","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.39161994999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5e","version":"215'39","reported_seq":994,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.979959+0000","last_change":"2023-01-15T14:33:57.316972+0000","last_active":"2023-01-15T14:34:16.979959+0000","last_peered":"2023-01-15T14:34:16.979959+0000","last_clean":"2023-01-15T14:34:16.979959+0000","last_became_active":"2023-01-15T14:33:35.364143+0000","last_became_peered":"2023-01-15T14:33:35.364143+0000","last_unstale":"2023-01-15T14:34:16.979959+0000","last_undegraded":"2023-01-15T14:34:16.979959+0000","last_fullsized":"2023-01-15T14:34:16.979959+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40693008000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1323,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978460+0000","last_change":"2023-01-15T14:34:00.343379+0000","last_active":"2023-01-15T14:34:16.978460+0000","last_peered":"2023-01-15T14:34:16.978460+0000","last_clean":"2023-01-15T14:34:16.978460+0000","last_became_active":"2023-01-15T14:34:00.242122+0000","last_became_peered":"2023-01-15T14:34:00.242122+0000","last_unstale":"2023-01-15T14:34:16.978460+0000","last_undegraded":"2023-01-15T14:34:16.978460+0000","last_fullsized":"2023-01-15T14:34:16.978460+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.099745922000000001,"stat_sum":{"num_bytes":2869928,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":23811,"num_write":13,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5c","version":"141'46","reported_seq":95,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.977998+0000","last_change":"2023-01-15T14:33:58.026285+0000","last_active":"2023-01-15T14:34:16.977998+0000","last_peered":"2023-01-15T14:34:16.977998+0000","last_clean":"2023-01-15T14:34:16.977998+0000","last_became_active":"2023-01-15T14:33:57.113032+0000","last_became_peered":"2023-01-15T14:33:57.113032+0000","last_unstale":"2023-01-15T14:34:16.977998+0000","last_undegraded":"2023-01-15T14:34:16.977998+0000","last_fullsized":"2023-01-15T14:34:16.977998+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.121922606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5b","version":"283'80","reported_seq":925,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.169713+0000","last_change":"2023-01-15T14:33:57.113633+0000","last_active":"2023-01-15T14:34:16.169713+0000","last_peered":"2023-01-15T14:34:16.169713+0000","last_clean":"2023-01-15T14:34:16.169713+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:34:16.169713+0000","last_undegraded":"2023-01-15T14:34:16.169713+0000","last_fullsized":"2023-01-15T14:34:16.169713+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.20934807,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5a","version":"329'348","reported_seq":1498,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.962963+0000","last_change":"2023-01-15T14:34:11.256946+0000","last_active":"2023-01-15T14:34:14.962963+0000","last_peered":"2023-01-15T14:34:14.962963+0000","last_clean":"2023-01-15T14:34:14.962963+0000","last_became_active":"2023-01-15T14:34:11.050316+0000","last_became_peered":"2023-01-15T14:34:11.050316+0000","last_unstale":"2023-01-15T14:34:14.962963+0000","last_undegraded":"2023-01-15T14:34:14.962963+0000","last_fullsized":"2023-01-15T14:34:14.962963+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:05.853","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.104528501,"stat_sum":{"num_bytes":5983424,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":19294,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.59","version":"294'256","reported_seq":1292,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077795+0000","last_change":"2023-01-15T14:33:57.406685+0000","last_active":"2023-01-15T14:34:17.077795+0000","last_peered":"2023-01-15T14:34:17.077795+0000","last_clean":"2023-01-15T14:34:17.077795+0000","last_became_active":"2023-01-15T14:33:55.671990+0000","last_became_peered":"2023-01-15T14:33:55.671990+0000","last_unstale":"2023-01-15T14:34:17.077795+0000","last_undegraded":"2023-01-15T14:34:17.077795+0000","last_fullsized":"2023-01-15T14:34:17.077795+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.49887967,"stat_sum":{"num_bytes":1210137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":5,"num_read_kb":3549,"num_write":5,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.58","version":"323'320","reported_seq":1494,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069047+0000","last_change":"2023-01-15T14:33:57.004059+0000","last_active":"2023-01-15T14:34:16.069047+0000","last_peered":"2023-01-15T14:34:16.069047+0000","last_clean":"2023-01-15T14:34:16.069047+0000","last_became_active":"2023-01-15T14:33:40.127669+0000","last_became_peered":"2023-01-15T14:33:40.127669+0000","last_unstale":"2023-01-15T14:34:16.069047+0000","last_undegraded":"2023-01-15T14:34:16.069047+0000","last_fullsized":"2023-01-15T14:34:16.069047+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.103642679,"stat_sum":{"num_bytes":4332079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.57","version":"351'98","reported_seq":107,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.964415+0000","last_change":"2023-01-15T14:33:57.890625+0000","last_active":"2023-01-15T14:34:14.964415+0000","last_peered":"2023-01-15T14:34:14.964415+0000","last_clean":"2023-01-15T14:34:14.964415+0000","last_became_active":"2023-01-15T14:33:36.994447+0000","last_became_peered":"2023-01-15T14:33:36.994447+0000","last_unstale":"2023-01-15T14:34:14.964415+0000","last_undegraded":"2023-01-15T14:34:14.964415+0000","last_fullsized":"2023-01-15T14:34:14.964415+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.969000109,"stat_sum":{"num_bytes":1512663,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3025326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.56","version":"320'127","reported_seq":644,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070546+0000","last_change":"2023-01-15T14:34:00.541392+0000","last_active":"2023-01-15T14:34:16.070546+0000","last_peered":"2023-01-15T14:34:16.070546+0000","last_clean":"2023-01-15T14:34:16.070546+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:34:16.070546+0000","last_undegraded":"2023-01-15T14:34:16.070546+0000","last_fullsized":"2023-01-15T14:34:16.070546+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.30190621200000001,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.55","version":"110'55","reported_seq":1025,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.078394+0000","last_change":"2023-01-15T14:33:57.315559+0000","last_active":"2023-01-15T14:34:17.078394+0000","last_peered":"2023-01-15T14:34:17.078394+0000","last_clean":"2023-01-15T14:34:17.078394+0000","last_became_active":"2023-01-15T14:33:52.510429+0000","last_became_peered":"2023-01-15T14:33:52.510429+0000","last_unstale":"2023-01-15T14:34:17.078394+0000","last_undegraded":"2023-01-15T14:34:17.078394+0000","last_fullsized":"2023-01-15T14:34:17.078394+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.40629608699999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1143,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.063157+0000","last_change":"2023-01-15T14:33:57.109553+0000","last_active":"2023-01-15T14:34:15.063157+0000","last_peered":"2023-01-15T14:34:15.063157+0000","last_clean":"2023-01-15T14:34:15.063157+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:34:15.063157+0000","last_undegraded":"2023-01-15T14:34:15.063157+0000","last_fullsized":"2023-01-15T14:34:15.063157+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:33:45.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:45.583005+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.201302861,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.53","version":"367'398","reported_seq":1475,"reported_epoch":367,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:17.817983+0000","last_change":"2023-01-15T14:34:14.611119+0000","last_active":"2023-01-15T14:34:17.817983+0000","last_peered":"2023-01-15T14:34:17.817983+0000","last_clean":"2023-01-15T14:33:55.165228+0000","last_became_active":"2023-01-15T14:33:58.106461+0000","last_became_peered":"2023-01-15T14:33:58.106461+0000","last_unstale":"2023-01-15T14:34:17.817983+0000","last_undegraded":"2023-01-15T14:33:58.004141+0000","last_fullsized":"2023-01-15T14:33:58.002666+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:44.609","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.092650028999999995,"stat_sum":{"num_bytes":28784748,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":28,"num_read_kb":35632,"num_write":20,"num_write_kb":13841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":23952448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,3","objects":9}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":744,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.169320+0000","last_change":"2023-01-15T14:33:56.903307+0000","last_active":"2023-01-15T14:34:16.169320+0000","last_peered":"2023-01-15T14:34:16.169320+0000","last_clean":"2023-01-15T14:34:16.169320+0000","last_became_active":"2023-01-15T14:33:55.472400+0000","last_became_peered":"2023-01-15T14:33:55.472400+0000","last_unstale":"2023-01-15T14:34:16.169320+0000","last_undegraded":"2023-01-15T14:34:16.169320+0000","last_fullsized":"2023-01-15T14:34:16.169320+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.001336634,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.51","version":"348'316","reported_seq":1533,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.975292+0000","last_change":"2023-01-15T14:33:57.893317+0000","last_active":"2023-01-15T14:34:15.975292+0000","last_peered":"2023-01-15T14:34:15.975292+0000","last_clean":"2023-01-15T14:34:15.975292+0000","last_became_active":"2023-01-15T14:33:26.107940+0000","last_became_peered":"2023-01-15T14:33:26.107940+0000","last_unstale":"2023-01-15T14:34:15.975292+0000","last_undegraded":"2023-01-15T14:34:15.975292+0000","last_fullsized":"2023-01-15T14:34:15.975292+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:36.458","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.79495623100000001,"stat_sum":{"num_bytes":7496813,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.50","version":"337'420","reported_seq":1596,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069335+0000","last_change":"2023-01-15T14:34:02.564430+0000","last_active":"2023-01-15T14:34:16.069335+0000","last_peered":"2023-01-15T14:34:16.069335+0000","last_clean":"2023-01-15T14:34:16.069335+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:34:16.069335+0000","last_undegraded":"2023-01-15T14:34:16.069335+0000","last_fullsized":"2023-01-15T14:34:16.069335+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.0084659680000000008,"stat_sum":{"num_bytes":9681416,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":52505,"num_write":49,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4f","version":"155'145","reported_seq":842,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978803+0000","last_change":"2023-01-15T14:33:57.353637+0000","last_active":"2023-01-15T14:34:16.978803+0000","last_peered":"2023-01-15T14:34:16.978803+0000","last_clean":"2023-01-15T14:34:16.978803+0000","last_became_active":"2023-01-15T14:33:25.129936+0000","last_became_peered":"2023-01-15T14:33:25.129936+0000","last_unstale":"2023-01-15T14:34:16.978803+0000","last_undegraded":"2023-01-15T14:34:16.978803+0000","last_fullsized":"2023-01-15T14:34:16.978803+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44322561900000002,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":111,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.963033+0000","last_change":"2023-01-15T14:33:56.915710+0000","last_active":"2023-01-15T14:34:14.963033+0000","last_peered":"2023-01-15T14:34:14.963033+0000","last_clean":"2023-01-15T14:34:14.963033+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:34:14.963033+0000","last_undegraded":"2023-01-15T14:34:14.963033+0000","last_fullsized":"2023-01-15T14:34:14.963033+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.0080109380000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4d","version":"356'262","reported_seq":1375,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.170264+0000","last_change":"2023-01-15T14:34:06.586527+0000","last_active":"2023-01-15T14:34:16.170264+0000","last_peered":"2023-01-15T14:34:16.170264+0000","last_clean":"2023-01-15T14:34:16.170264+0000","last_became_active":"2023-01-15T14:34:05.178613+0000","last_became_peered":"2023-01-15T14:34:05.178613+0000","last_unstale":"2023-01-15T14:34:16.170264+0000","last_undegraded":"2023-01-15T14:34:16.170264+0000","last_fullsized":"2023-01-15T14:34:16.170264+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":1.4066433030000001,"stat_sum":{"num_bytes":12124166,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":15961,"num_write":18,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4c","version":"309'71","reported_seq":121,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069680+0000","last_change":"2023-01-15T14:33:57.413785+0000","last_active":"2023-01-15T14:34:16.069680+0000","last_peered":"2023-01-15T14:34:16.069680+0000","last_clean":"2023-01-15T14:34:16.069680+0000","last_became_active":"2023-01-15T14:33:52.311859+0000","last_became_peered":"2023-01-15T14:33:52.311859+0000","last_unstale":"2023-01-15T14:34:16.069680+0000","last_undegraded":"2023-01-15T14:34:16.069680+0000","last_fullsized":"2023-01-15T14:34:16.069680+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.40874315500000002,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":767,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.975593+0000","last_change":"2023-01-15T14:33:57.627070+0000","last_active":"2023-01-15T14:34:15.975593+0000","last_peered":"2023-01-15T14:34:15.975593+0000","last_clean":"2023-01-15T14:34:15.975593+0000","last_became_active":"2023-01-15T14:33:52.406605+0000","last_became_peered":"2023-01-15T14:33:52.406605+0000","last_unstale":"2023-01-15T14:34:15.975593+0000","last_undegraded":"2023-01-15T14:34:15.975593+0000","last_fullsized":"2023-01-15T14:34:15.975593+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:42.594","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.515074951,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4a","version":"348'371","reported_seq":1492,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069382+0000","last_change":"2023-01-15T14:33:58.018192+0000","last_active":"2023-01-15T14:34:16.069382+0000","last_peered":"2023-01-15T14:34:16.069382+0000","last_clean":"2023-01-15T14:34:16.069382+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:34:16.069382+0000","last_undegraded":"2023-01-15T14:34:16.069382+0000","last_fullsized":"2023-01-15T14:34:16.069382+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:33.328","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10440110800000001,"stat_sum":{"num_bytes":12470870,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":63,"num_read_kb":67536,"num_write":73,"num_write_kb":23626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":949,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.043422+0000","last_change":"2023-01-15T14:34:00.046526+0000","last_active":"2023-01-15T14:34:16.043422+0000","last_peered":"2023-01-15T14:34:16.043422+0000","last_clean":"2023-01-15T14:34:16.043422+0000","last_became_active":"2023-01-15T14:34:00.043221+0000","last_became_peered":"2023-01-15T14:34:00.043221+0000","last_unstale":"2023-01-15T14:34:16.043422+0000","last_undegraded":"2023-01-15T14:34:16.043422+0000","last_fullsized":"2023-01-15T14:34:16.043422+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.00195963,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.48","version":"179'204","reported_seq":1373,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.976967+0000","last_change":"2023-01-15T14:33:57.356590+0000","last_active":"2023-01-15T14:34:16.976967+0000","last_peered":"2023-01-15T14:34:16.976967+0000","last_clean":"2023-01-15T14:34:16.976967+0000","last_became_active":"2023-01-15T14:33:55.375518+0000","last_became_peered":"2023-01-15T14:33:55.375518+0000","last_unstale":"2023-01-15T14:34:16.976967+0000","last_undegraded":"2023-01-15T14:34:16.976967+0000","last_fullsized":"2023-01-15T14:34:16.976967+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:43.662","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.34933974800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1136,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978419+0000","last_change":"2023-01-15T14:34:14.145798+0000","last_active":"2023-01-15T14:34:16.978419+0000","last_peered":"2023-01-15T14:34:16.978419+0000","last_clean":"2023-01-15T14:34:16.978419+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:34:16.978419+0000","last_undegraded":"2023-01-15T14:34:16.978419+0000","last_fullsized":"2023-01-15T14:34:16.978419+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:19.145","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.39166528900000003,"stat_sum":{"num_bytes":496762,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":5,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":1043,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.176717+0000","last_change":"2023-01-15T14:33:57.829426+0000","last_active":"2023-01-15T14:34:16.176717+0000","last_peered":"2023-01-15T14:34:16.176717+0000","last_clean":"2023-01-15T14:34:16.176717+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:34:16.176717+0000","last_undegraded":"2023-01-15T14:34:16.176717+0000","last_fullsized":"2023-01-15T14:34:16.176717+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:55.245","scrub_duration":0.30448409700000001,"objects_trimmed":0,"snaptrim_duration":0.714760332,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":928,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.965613+0000","last_change":"2023-01-15T14:33:56.909121+0000","last_active":"2023-01-15T14:34:14.965613+0000","last_peered":"2023-01-15T14:34:14.965613+0000","last_clean":"2023-01-15T14:34:14.965613+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:34:14.965613+0000","last_undegraded":"2023-01-15T14:34:14.965613+0000","last_fullsized":"2023-01-15T14:34:14.965613+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0045307990000000003,"stat_sum":{"num_bytes":2635976,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":19927,"num_write":15,"num_write_kb":6659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4745860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1288,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.979369+0000","last_change":"2023-01-15T14:33:57.298701+0000","last_active":"2023-01-15T14:34:16.979369+0000","last_peered":"2023-01-15T14:34:16.979369+0000","last_clean":"2023-01-15T14:34:16.979369+0000","last_became_active":"2023-01-15T14:33:55.678019+0000","last_became_peered":"2023-01-15T14:33:55.678019+0000","last_unstale":"2023-01-15T14:34:16.979369+0000","last_undegraded":"2023-01-15T14:34:16.979369+0000","last_fullsized":"2023-01-15T14:34:16.979369+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:27.120","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.390225449,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.43","version":"353'316","reported_seq":1529,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.063262+0000","last_change":"2023-01-15T14:34:02.217041+0000","last_active":"2023-01-15T14:34:15.063262+0000","last_peered":"2023-01-15T14:34:15.063262+0000","last_clean":"2023-01-15T14:34:15.063262+0000","last_became_active":"2023-01-15T14:34:02.211232+0000","last_became_peered":"2023-01-15T14:34:02.211232+0000","last_unstale":"2023-01-15T14:34:15.063262+0000","last_undegraded":"2023-01-15T14:34:15.063262+0000","last_fullsized":"2023-01-15T14:34:15.063262+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.0028674299999999998,"stat_sum":{"num_bytes":6319564,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":16585,"num_write":6,"num_write_kb":4207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3395080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.42","version":"155'101","reported_seq":1049,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077958+0000","last_change":"2023-01-15T14:33:57.621398+0000","last_active":"2023-01-15T14:34:17.077958+0000","last_peered":"2023-01-15T14:34:17.077958+0000","last_clean":"2023-01-15T14:34:17.077958+0000","last_became_active":"2023-01-15T14:33:55.376782+0000","last_became_peered":"2023-01-15T14:33:55.376782+0000","last_unstale":"2023-01-15T14:34:17.077958+0000","last_undegraded":"2023-01-15T14:34:17.077958+0000","last_fullsized":"2023-01-15T14:34:17.077958+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.51101871399999999,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1129,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077913+0000","last_change":"2023-01-15T14:33:57.302504+0000","last_active":"2023-01-15T14:34:17.077913+0000","last_peered":"2023-01-15T14:34:17.077913+0000","last_clean":"2023-01-15T14:34:17.077913+0000","last_became_active":"2023-01-15T14:33:55.771969+0000","last_became_peered":"2023-01-15T14:33:55.771969+0000","last_unstale":"2023-01-15T14:34:17.077913+0000","last_undegraded":"2023-01-15T14:34:17.077913+0000","last_fullsized":"2023-01-15T14:34:17.077913+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.29405439700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.40","version":"363'311","reported_seq":1467,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070021+0000","last_change":"2023-01-15T14:34:14.041682+0000","last_active":"2023-01-15T14:34:16.070021+0000","last_peered":"2023-01-15T14:34:16.070021+0000","last_clean":"2023-01-15T14:34:16.070021+0000","last_became_active":"2023-01-15T14:33:58.117085+0000","last_became_peered":"2023-01-15T14:33:58.117085+0000","last_unstale":"2023-01-15T14:34:16.070021+0000","last_undegraded":"2023-01-15T14:34:16.070021+0000","last_fullsized":"2023-01-15T14:34:16.070021+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":1.0057765489999999,"stat_sum":{"num_bytes":12753904,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":35541,"num_write":28,"num_write_kb":11001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":29167491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3f","version":"321'233","reported_seq":1384,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070697+0000","last_change":"2023-01-15T14:33:58.333573+0000","last_active":"2023-01-15T14:34:16.070697+0000","last_peered":"2023-01-15T14:34:16.070697+0000","last_clean":"2023-01-15T14:34:16.070697+0000","last_became_active":"2023-01-15T14:33:58.217783+0000","last_became_peered":"2023-01-15T14:33:58.217783+0000","last_unstale":"2023-01-15T14:34:16.070697+0000","last_undegraded":"2023-01-15T14:34:16.070697+0000","last_fullsized":"2023-01-15T14:34:16.070697+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:28.180","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.014066018,"stat_sum":{"num_bytes":4764997,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6605283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3e","version":"294'63","reported_seq":849,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.975662+0000","last_change":"2023-01-15T14:33:57.728361+0000","last_active":"2023-01-15T14:34:15.975662+0000","last_peered":"2023-01-15T14:34:15.975662+0000","last_clean":"2023-01-15T14:34:15.975662+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:34:15.975662+0000","last_undegraded":"2023-01-15T14:34:15.975662+0000","last_fullsized":"2023-01-15T14:34:15.975662+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.61381065800000001,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3d","version":"356'236","reported_seq":1342,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.965366+0000","last_change":"2023-01-15T14:34:06.401367+0000","last_active":"2023-01-15T14:34:14.965366+0000","last_peered":"2023-01-15T14:34:14.965366+0000","last_clean":"2023-01-15T14:34:14.965366+0000","last_became_active":"2023-01-15T14:34:05.187121+0000","last_became_peered":"2023-01-15T14:34:05.187121+0000","last_unstale":"2023-01-15T14:34:14.965366+0000","last_undegraded":"2023-01-15T14:34:14.965366+0000","last_fullsized":"2023-01-15T14:34:14.965366+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":1.2126637869999999,"stat_sum":{"num_bytes":8050279,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":13797,"num_write":16,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":8050279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3c","version":"141'46","reported_seq":365,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978107+0000","last_change":"2023-01-15T14:33:56.912541+0000","last_active":"2023-01-15T14:34:16.978107+0000","last_peered":"2023-01-15T14:34:16.978107+0000","last_clean":"2023-01-15T14:34:16.978107+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:34:16.978107+0000","last_undegraded":"2023-01-15T14:34:16.978107+0000","last_fullsized":"2023-01-15T14:34:16.978107+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.470","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.0047236029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":39,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.169621+0000","last_change":"2023-01-15T14:33:58.125276+0000","last_active":"2023-01-15T14:34:16.169621+0000","last_peered":"2023-01-15T14:34:16.169621+0000","last_clean":"2023-01-15T14:34:16.169621+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:34:16.169621+0000","last_undegraded":"2023-01-15T14:34:16.169621+0000","last_fullsized":"2023-01-15T14:34:16.169621+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.21169681200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1362,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.962871+0000","last_change":"2023-01-15T14:33:57.286278+0000","last_active":"2023-01-15T14:34:14.962871+0000","last_peered":"2023-01-15T14:34:14.962871+0000","last_clean":"2023-01-15T14:34:14.962871+0000","last_became_active":"2023-01-15T14:33:55.474944+0000","last_became_peered":"2023-01-15T14:33:55.474944+0000","last_unstale":"2023-01-15T14:34:14.962871+0000","last_undegraded":"2023-01-15T14:34:14.962871+0000","last_fullsized":"2023-01-15T14:34:14.962871+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.37705996200000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.39","version":"358'355","reported_seq":1504,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.023581+0000","last_change":"2023-01-15T14:34:08.632379+0000","last_active":"2023-01-15T14:34:16.023581+0000","last_peered":"2023-01-15T14:34:16.023581+0000","last_clean":"2023-01-15T14:34:16.023581+0000","last_became_active":"2023-01-15T14:34:08.023332+0000","last_became_peered":"2023-01-15T14:34:08.023332+0000","last_unstale":"2023-01-15T14:34:16.023581+0000","last_undegraded":"2023-01-15T14:34:16.023581+0000","last_fullsized":"2023-01-15T14:34:16.023581+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:28.106","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.60799269700000003,"stat_sum":{"num_bytes":24023444,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":66941,"num_write":7,"num_write_kb":7292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":26862613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.38","version":"352'315","reported_seq":1549,"reported_epoch":368,"state":"active+clean","last_fresh":"2023-01-15T14:34:18.084841+0000","last_change":"2023-01-15T14:33:58.009343+0000","last_active":"2023-01-15T14:34:18.084841+0000","last_peered":"2023-01-15T14:34:18.084841+0000","last_clean":"2023-01-15T14:34:18.084841+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:34:18.084841+0000","last_undegraded":"2023-01-15T14:34:18.084841+0000","last_fullsized":"2023-01-15T14:34:18.084841+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.10683759900000001,"stat_sum":{"num_bytes":8461433,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":42848,"num_write":39,"num_write_kb":13455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":425,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070951+0000","last_change":"2023-01-15T14:33:58.124010+0000","last_active":"2023-01-15T14:34:16.070951+0000","last_peered":"2023-01-15T14:34:16.070951+0000","last_clean":"2023-01-15T14:34:16.070951+0000","last_became_active":"2023-01-15T14:33:52.412128+0000","last_became_peered":"2023-01-15T14:33:52.412128+0000","last_unstale":"2023-01-15T14:34:16.070951+0000","last_undegraded":"2023-01-15T14:34:16.070951+0000","last_fullsized":"2023-01-15T14:34:16.070951+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.207923046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":577,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.975829+0000","last_change":"2023-01-15T14:33:57.332104+0000","last_active":"2023-01-15T14:34:15.975829+0000","last_peered":"2023-01-15T14:34:15.975829+0000","last_clean":"2023-01-15T14:34:15.975829+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:34:15.975829+0000","last_undegraded":"2023-01-15T14:34:15.975829+0000","last_fullsized":"2023-01-15T14:34:15.975829+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.31780763299999998,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":51,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.965489+0000","last_change":"2023-01-15T14:33:56.908180+0000","last_active":"2023-01-15T14:34:14.965489+0000","last_peered":"2023-01-15T14:34:14.965489+0000","last_clean":"2023-01-15T14:34:14.965489+0000","last_became_active":"2023-01-15T14:33:52.612576+0000","last_became_peered":"2023-01-15T14:33:52.612576+0000","last_unstale":"2023-01-15T14:34:14.965489+0000","last_undegraded":"2023-01-15T14:34:14.965489+0000","last_fullsized":"2023-01-15T14:34:14.965489+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003836173,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.34","version":"169'138","reported_seq":1173,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.979539+0000","last_change":"2023-01-15T14:33:57.301685+0000","last_active":"2023-01-15T14:34:16.979539+0000","last_peered":"2023-01-15T14:34:16.979539+0000","last_clean":"2023-01-15T14:34:16.979539+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:34:16.979539+0000","last_undegraded":"2023-01-15T14:34:16.979539+0000","last_fullsized":"2023-01-15T14:34:16.979539+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.39300976100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1248,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.977908+0000","last_change":"2023-01-15T14:33:57.354727+0000","last_active":"2023-01-15T14:34:16.977908+0000","last_peered":"2023-01-15T14:34:16.977908+0000","last_clean":"2023-01-15T14:34:16.977908+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:34:16.977908+0000","last_undegraded":"2023-01-15T14:34:16.977908+0000","last_fullsized":"2023-01-15T14:34:16.977908+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.442392491,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.32","version":"355'165","reported_seq":859,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.962735+0000","last_change":"2023-01-15T14:34:05.280466+0000","last_active":"2023-01-15T14:34:14.962735+0000","last_peered":"2023-01-15T14:34:14.962735+0000","last_clean":"2023-01-15T14:34:14.962735+0000","last_became_active":"2023-01-15T14:34:04.171493+0000","last_became_peered":"2023-01-15T14:34:04.171493+0000","last_unstale":"2023-01-15T14:34:14.962735+0000","last_undegraded":"2023-01-15T14:34:14.962735+0000","last_fullsized":"2023-01-15T14:34:14.962735+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.19993219000000001,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7048,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5327885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.13","version":"348'374","reported_seq":1398,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.270042+0000","last_change":"2023-01-15T14:33:58.021945+0000","last_active":"2023-01-15T14:34:16.270042+0000","last_peered":"2023-01-15T14:34:16.270042+0000","last_clean":"2023-01-15T14:34:16.270042+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:34:16.270042+0000","last_undegraded":"2023-01-15T14:34:16.270042+0000","last_fullsized":"2023-01-15T14:34:16.270042+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:29.359","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.0064123280000000001,"stat_sum":{"num_bytes":8364486,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":61489,"num_write":17,"num_write_kb":9266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.12","version":"353'186","reported_seq":914,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.962731+0000","last_change":"2023-01-15T14:34:02.715457+0000","last_active":"2023-01-15T14:34:14.962731+0000","last_peered":"2023-01-15T14:34:14.962731+0000","last_clean":"2023-01-15T14:34:14.962731+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:34:14.962731+0000","last_undegraded":"2023-01-15T14:34:14.962731+0000","last_fullsized":"2023-01-15T14:34:14.962731+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.57998335899999998,"stat_sum":{"num_bytes":7581497,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":15,"num_write_kb":5299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.11","version":"359'308","reported_seq":1584,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069400+0000","last_change":"2023-01-15T14:34:10.264293+0000","last_active":"2023-01-15T14:34:16.069400+0000","last_peered":"2023-01-15T14:34:16.069400+0000","last_clean":"2023-01-15T14:34:16.069400+0000","last_became_active":"2023-01-15T14:34:09.541429+0000","last_became_peered":"2023-01-15T14:34:09.541429+0000","last_unstale":"2023-01-15T14:34:16.069400+0000","last_undegraded":"2023-01-15T14:34:16.069400+0000","last_fullsized":"2023-01-15T14:34:16.069400+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.20050804999999999,"stat_sum":{"num_bytes":10718732,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":48289,"num_write":26,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":20307357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.10","version":"308'406","reported_seq":1414,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.377563+0000","last_change":"2023-01-15T14:34:16.478474+0000","last_active":"2023-01-15T14:34:16.377563+0000","last_peered":"2023-01-15T14:34:16.377563+0000","last_clean":"2023-01-15T14:34:16.377563+0000","last_became_active":"2023-01-15T14:34:16.272883+0000","last_became_peered":"2023-01-15T14:34:16.272883+0000","last_unstale":"2023-01-15T14:34:16.377563+0000","last_undegraded":"2023-01-15T14:34:16.377563+0000","last_fullsized":"2023-01-15T14:34:16.377563+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.204053383,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":41964,"num_write":46,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12804882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.f","version":"155'145","reported_seq":1250,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070343+0000","last_change":"2023-01-15T14:34:00.245548+0000","last_active":"2023-01-15T14:34:16.070343+0000","last_peered":"2023-01-15T14:34:16.070343+0000","last_clean":"2023-01-15T14:34:16.070343+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:34:16.070343+0000","last_undegraded":"2023-01-15T14:34:16.070343+0000","last_fullsized":"2023-01-15T14:34:16.070343+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.0023104509999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.e","version":"139'38","reported_seq":39,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070296+0000","last_change":"2023-01-15T14:33:58.422638+0000","last_active":"2023-01-15T14:34:16.070296+0000","last_peered":"2023-01-15T14:34:16.070296+0000","last_clean":"2023-01-15T14:34:16.070296+0000","last_became_active":"2023-01-15T14:33:57.310759+0000","last_became_peered":"2023-01-15T14:33:57.310759+0000","last_unstale":"2023-01-15T14:34:16.070296+0000","last_undegraded":"2023-01-15T14:34:16.070296+0000","last_fullsized":"2023-01-15T14:34:16.070296+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.40586427000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.d","version":"296'232","reported_seq":1313,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.170440+0000","last_change":"2023-01-15T14:33:57.114569+0000","last_active":"2023-01-15T14:34:16.170440+0000","last_peered":"2023-01-15T14:34:16.170440+0000","last_clean":"2023-01-15T14:34:16.170440+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:34:16.170440+0000","last_undegraded":"2023-01-15T14:34:16.170440+0000","last_fullsized":"2023-01-15T14:34:16.170440+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.20767458899999999,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.c","version":"348'95","reported_seq":1110,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978331+0000","last_change":"2023-01-15T14:33:58.326911+0000","last_active":"2023-01-15T14:34:16.978331+0000","last_peered":"2023-01-15T14:34:16.978331+0000","last_clean":"2023-01-15T14:34:16.978331+0000","last_became_active":"2023-01-15T14:33:52.210127+0000","last_became_peered":"2023-01-15T14:33:52.210127+0000","last_unstale":"2023-01-15T14:34:16.978331+0000","last_undegraded":"2023-01-15T14:34:16.978331+0000","last_fullsized":"2023-01-15T14:34:16.978331+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.421830761,"stat_sum":{"num_bytes":2221858,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":10493,"num_write":6,"num_write_kb":686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8023674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":1019,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978294+0000","last_change":"2023-01-15T14:33:57.618325+0000","last_active":"2023-01-15T14:34:16.978294+0000","last_peered":"2023-01-15T14:34:16.978294+0000","last_clean":"2023-01-15T14:34:16.978294+0000","last_became_active":"2023-01-15T14:33:52.425622+0000","last_became_peered":"2023-01-15T14:33:52.425622+0000","last_unstale":"2023-01-15T14:34:16.978294+0000","last_undegraded":"2023-01-15T14:34:16.978294+0000","last_fullsized":"2023-01-15T14:34:16.978294+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.60540339200000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1342,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.078306+0000","last_change":"2023-01-15T14:33:57.297022+0000","last_active":"2023-01-15T14:34:17.078306+0000","last_peered":"2023-01-15T14:34:17.078306+0000","last_clean":"2023-01-15T14:34:17.078306+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:34:17.078306+0000","last_undegraded":"2023-01-15T14:34:17.078306+0000","last_fullsized":"2023-01-15T14:34:17.078306+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:48.664","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.38900274200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1127,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.178406+0000","last_change":"2023-01-15T14:33:57.304492+0000","last_active":"2023-01-15T14:34:17.178406+0000","last_peered":"2023-01-15T14:34:17.178406+0000","last_clean":"2023-01-15T14:34:17.178406+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:34:17.178406+0000","last_undegraded":"2023-01-15T14:34:17.178406+0000","last_fullsized":"2023-01-15T14:34:17.178406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:38.874","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.29516895799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1347,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077954+0000","last_change":"2023-01-15T14:34:00.443594+0000","last_active":"2023-01-15T14:34:17.077954+0000","last_peered":"2023-01-15T14:34:17.077954+0000","last_clean":"2023-01-15T14:34:17.077954+0000","last_became_active":"2023-01-15T14:34:00.140247+0000","last_became_peered":"2023-01-15T14:34:00.140247+0000","last_unstale":"2023-01-15T14:34:17.077954+0000","last_undegraded":"2023-01-15T14:34:17.077954+0000","last_fullsized":"2023-01-15T14:34:17.077954+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.202704782,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"1.0","version":"19'117","reported_seq":535,"reported_epoch":367,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:34:17.810327+0000","last_change":"2023-01-15T14:34:17.809839+0000","last_active":"2023-01-15T14:34:17.810327+0000","last_peered":"2023-01-15T14:34:17.810327+0000","last_clean":"2023-01-15T14:33:50.143204+0000","last_became_active":"2023-01-15T14:33:58.306184+0000","last_became_peered":"2023-01-15T14:33:58.306184+0000","last_unstale":"2023-01-15T14:34:17.810327+0000","last_undegraded":"2023-01-15T14:34:17.810327+0000","last_fullsized":"2023-01-15T14:33:58.001929+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2624160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[1,0],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"1,2,4","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":30,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.169562+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:34:16.169562+0000","last_peered":"2023-01-15T14:34:16.169562+0000","last_clean":"2023-01-15T14:34:16.169562+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:34:16.169562+0000","last_undegraded":"2023-01-15T14:34:16.169562+0000","last_fullsized":"2023-01-15T14:34:16.169562+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":1015,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.078175+0000","last_change":"2023-01-15T14:33:57.297870+0000","last_active":"2023-01-15T14:34:17.078175+0000","last_peered":"2023-01-15T14:34:17.078175+0000","last_clean":"2023-01-15T14:34:17.078175+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:34:17.078175+0000","last_undegraded":"2023-01-15T14:34:17.078175+0000","last_fullsized":"2023-01-15T14:34:17.078175+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.38976201700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":413,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.083544+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:34:17.083544+0000","last_peered":"2023-01-15T14:34:17.083544+0000","last_clean":"2023-01-15T14:34:17.083544+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:34:17.083544+0000","last_undegraded":"2023-01-15T14:34:17.083544+0000","last_fullsized":"2023-01-15T14:34:17.083544+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.964","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":726,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.962566+0000","last_change":"2023-01-15T14:33:57.185185+0000","last_active":"2023-01-15T14:34:14.962566+0000","last_peered":"2023-01-15T14:34:14.962566+0000","last_clean":"2023-01-15T14:34:14.962566+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:34:14.962566+0000","last_undegraded":"2023-01-15T14:34:14.962566+0000","last_fullsized":"2023-01-15T14:34:14.962566+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.280577087,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":169,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077721+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:34:17.077721+0000","last_peered":"2023-01-15T14:34:17.077721+0000","last_clean":"2023-01-15T14:34:17.077721+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:34:17.077721+0000","last_undegraded":"2023-01-15T14:34:17.077721+0000","last_fullsized":"2023-01-15T14:34:17.077721+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1337,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.076095+0000","last_change":"2023-01-15T14:33:57.830460+0000","last_active":"2023-01-15T14:34:16.076095+0000","last_peered":"2023-01-15T14:34:16.076095+0000","last_clean":"2023-01-15T14:34:16.076095+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:34:16.076095+0000","last_undegraded":"2023-01-15T14:34:16.076095+0000","last_fullsized":"2023-01-15T14:34:16.076095+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.52931870400000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":235,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.976628+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:34:15.976628+0000","last_peered":"2023-01-15T14:34:15.976628+0000","last_clean":"2023-01-15T14:34:15.976628+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:34:15.976628+0000","last_undegraded":"2023-01-15T14:34:15.976628+0000","last_fullsized":"2023-01-15T14:34:15.976628+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:25.583","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"352'325","reported_seq":1437,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077942+0000","last_change":"2023-01-15T14:34:01.953246+0000","last_active":"2023-01-15T14:34:17.077942+0000","last_peered":"2023-01-15T14:34:17.077942+0000","last_clean":"2023-01-15T14:34:17.077942+0000","last_became_active":"2023-01-15T14:34:00.043073+0000","last_became_peered":"2023-01-15T14:34:00.043073+0000","last_unstale":"2023-01-15T14:34:17.077942+0000","last_undegraded":"2023-01-15T14:34:17.077942+0000","last_fullsized":"2023-01-15T14:34:17.077942+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:26.745","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.80204535600000004,"stat_sum":{"num_bytes":10343843,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":13875,"num_write":6,"num_write_kb":7330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9428925,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5","version":"355'160","reported_seq":1233,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.178680+0000","last_change":"2023-01-15T14:34:04.889983+0000","last_active":"2023-01-15T14:34:17.178680+0000","last_peered":"2023-01-15T14:34:17.178680+0000","last_clean":"2023-01-15T14:34:17.178680+0000","last_became_active":"2023-01-15T14:34:04.175473+0000","last_became_peered":"2023-01-15T14:34:04.175473+0000","last_unstale":"2023-01-15T14:34:17.178680+0000","last_undegraded":"2023-01-15T14:34:17.178680+0000","last_fullsized":"2023-01-15T14:34:17.178680+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.61306555500000004,"stat_sum":{"num_bytes":11390101,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":26145,"num_write":9,"num_write_kb":4265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":13260921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":225,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.976058+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:34:15.976058+0000","last_peered":"2023-01-15T14:34:15.976058+0000","last_clean":"2023-01-15T14:34:15.976058+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:34:15.976058+0000","last_undegraded":"2023-01-15T14:34:15.976058+0000","last_fullsized":"2023-01-15T14:34:15.976058+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"367'311","reported_seq":1505,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.385128+0000","last_change":"2023-01-15T14:34:11.259757+0000","last_active":"2023-01-15T14:34:17.385128+0000","last_peered":"2023-01-15T14:34:17.385128+0000","last_clean":"2023-01-15T14:34:17.385128+0000","last_became_active":"2023-01-15T14:34:11.152893+0000","last_became_peered":"2023-01-15T14:34:11.152893+0000","last_unstale":"2023-01-15T14:34:17.385128+0000","last_undegraded":"2023-01-15T14:34:17.385128+0000","last_fullsized":"2023-01-15T14:34:17.385128+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.104600024,"stat_sum":{"num_bytes":2954643,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":31,"num_read_kb":30684,"num_write":33,"num_write_kb":10131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":14805347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":414,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.170714+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:34:16.170714+0000","last_peered":"2023-01-15T14:34:16.170714+0000","last_clean":"2023-01-15T14:34:16.170714+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:34:16.170714+0000","last_undegraded":"2023-01-15T14:34:16.170714+0000","last_fullsized":"2023-01-15T14:34:16.170714+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"367'143","reported_seq":1149,"reported_epoch":367,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:17.487872+0000","last_change":"2023-01-15T14:34:17.229236+0000","last_active":"2023-01-15T14:34:17.487872+0000","last_peered":"2023-01-15T14:34:17.487872+0000","last_clean":"2023-01-15T14:34:17.487872+0000","last_became_active":"2023-01-15T14:34:17.186013+0000","last_became_peered":"2023-01-15T14:34:17.186013+0000","last_unstale":"2023-01-15T14:34:17.487872+0000","last_undegraded":"2023-01-15T14:34:17.487872+0000","last_fullsized":"2023-01-15T14:34:17.487872+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.471195633,"stat_sum":{"num_bytes":9475691,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":13597,"num_write":13,"num_write_kb":8618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3546343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":225,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.076417+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:34:16.076417+0000","last_peered":"2023-01-15T14:34:16.076417+0000","last_clean":"2023-01-15T14:34:16.076417+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:34:16.076417+0000","last_undegraded":"2023-01-15T14:34:16.076417+0000","last_fullsized":"2023-01-15T14:34:16.076417+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:17.360","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":51,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.963520+0000","last_change":"2023-01-15T14:33:56.909628+0000","last_active":"2023-01-15T14:34:14.963520+0000","last_peered":"2023-01-15T14:34:14.963520+0000","last_clean":"2023-01-15T14:34:14.963520+0000","last_became_active":"2023-01-15T14:33:52.206701+0000","last_became_peered":"2023-01-15T14:33:52.206701+0000","last_unstale":"2023-01-15T14:34:14.963520+0000","last_undegraded":"2023-01-15T14:34:14.963520+0000","last_fullsized":"2023-01-15T14:34:14.963520+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.0022256720000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070771+0000","last_change":"2023-01-15T14:33:57.412557+0000","last_active":"2023-01-15T14:34:16.070771+0000","last_peered":"2023-01-15T14:34:16.070771+0000","last_clean":"2023-01-15T14:34:16.070771+0000","last_became_active":"2023-01-15T14:33:57.311981+0000","last_became_peered":"2023-01-15T14:33:57.311981+0000","last_unstale":"2023-01-15T14:34:16.070771+0000","last_undegraded":"2023-01-15T14:34:16.070771+0000","last_fullsized":"2023-01-15T14:34:16.070771+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"351'159","reported_seq":1209,"reported_epoch":368,"state":"active+clean","last_fresh":"2023-01-15T14:34:18.607921+0000","last_change":"2023-01-15T14:34:18.607843+0000","last_active":"2023-01-15T14:34:18.607921+0000","last_peered":"2023-01-15T14:34:18.607921+0000","last_clean":"2023-01-15T14:34:18.607921+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:34:18.607921+0000","last_undegraded":"2023-01-15T14:34:18.607921+0000","last_fullsized":"2023-01-15T14:34:18.607921+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"351'159","last_scrub_stamp":"2023-01-15T14:34:18.607751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:18.607751+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.40666956999999998,"objects_trimmed":0,"snaptrim_duration":0.0010524499999999999,"stat_sum":{"num_bytes":2453855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":39,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070097+0000","last_change":"2023-01-15T14:33:58.327664+0000","last_active":"2023-01-15T14:34:16.070097+0000","last_peered":"2023-01-15T14:34:16.070097+0000","last_clean":"2023-01-15T14:34:16.070097+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:34:16.070097+0000","last_undegraded":"2023-01-15T14:34:16.070097+0000","last_fullsized":"2023-01-15T14:34:16.070097+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31327724499999998,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.16","version":"366'131","reported_seq":686,"reported_epoch":367,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:34:17.386323+0000","last_change":"2023-01-15T14:34:17.387118+0000","last_active":"2023-01-15T14:34:17.386323+0000","last_peered":"2023-01-15T14:34:17.386323+0000","last_clean":"2023-01-15T14:33:55.270604+0000","last_became_active":"2023-01-15T14:33:58.218043+0000","last_became_peered":"2023-01-15T14:33:58.218043+0000","last_unstale":"2023-01-15T14:34:17.386323+0000","last_undegraded":"2023-01-15T14:33:57.906873+0000","last_fullsized":"2023-01-15T14:33:57.906218+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:09.523","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.30046122199999997,"stat_sum":{"num_bytes":7378471,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7378471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":["1","2","3","7"],"object_location_counts":[{"shards":"1,3","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":51,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.964340+0000","last_change":"2023-01-15T14:33:57.186529+0000","last_active":"2023-01-15T14:34:14.964340+0000","last_peered":"2023-01-15T14:34:14.964340+0000","last_clean":"2023-01-15T14:34:14.964340+0000","last_became_active":"2023-01-15T14:33:52.303197+0000","last_became_peered":"2023-01-15T14:33:52.303197+0000","last_unstale":"2023-01-15T14:34:14.964340+0000","last_undegraded":"2023-01-15T14:34:14.964340+0000","last_fullsized":"2023-01-15T14:34:14.964340+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.265135238,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.18","version":"355'328","reported_seq":1554,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077790+0000","last_change":"2023-01-15T14:34:04.674366+0000","last_active":"2023-01-15T14:34:17.077790+0000","last_peered":"2023-01-15T14:34:17.077790+0000","last_clean":"2023-01-15T14:34:17.077790+0000","last_became_active":"2023-01-15T14:34:03.265912+0000","last_became_peered":"2023-01-15T14:34:03.265912+0000","last_unstale":"2023-01-15T14:34:17.077790+0000","last_undegraded":"2023-01-15T14:34:17.077790+0000","last_fullsized":"2023-01-15T14:34:17.077790+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.70539038499999995,"stat_sum":{"num_bytes":4332078,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":26,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":12147901,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.19","version":"294'256","reported_seq":1336,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.976382+0000","last_change":"2023-01-15T14:33:57.626542+0000","last_active":"2023-01-15T14:34:15.976382+0000","last_peered":"2023-01-15T14:34:15.976382+0000","last_clean":"2023-01-15T14:34:15.976382+0000","last_became_active":"2023-01-15T14:33:35.259904+0000","last_became_peered":"2023-01-15T14:33:35.259904+0000","last_unstale":"2023-01-15T14:34:15.976382+0000","last_undegraded":"2023-01-15T14:34:15.976382+0000","last_fullsized":"2023-01-15T14:34:15.976382+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:01.296","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.52734117999999996,"stat_sum":{"num_bytes":2420274,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1a","version":"348'352","reported_seq":1470,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.977959+0000","last_change":"2023-01-15T14:33:58.011960+0000","last_active":"2023-01-15T14:34:16.977959+0000","last_peered":"2023-01-15T14:34:16.977959+0000","last_clean":"2023-01-15T14:34:16.977959+0000","last_became_active":"2023-01-15T14:33:35.063699+0000","last_became_peered":"2023-01-15T14:33:35.063699+0000","last_unstale":"2023-01-15T14:34:16.977959+0000","last_undegraded":"2023-01-15T14:34:16.977959+0000","last_fullsized":"2023-01-15T14:34:16.977959+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.0039007270000000001,"stat_sum":{"num_bytes":3052653,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":15,"num_read_kb":21426,"num_write":7,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1b","version":"348'99","reported_seq":75,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.063026+0000","last_change":"2023-01-15T14:33:58.271017+0000","last_active":"2023-01-15T14:34:15.063026+0000","last_peered":"2023-01-15T14:34:15.063026+0000","last_clean":"2023-01-15T14:34:15.063026+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:34:15.063026+0000","last_undegraded":"2023-01-15T14:34:15.063026+0000","last_fullsized":"2023-01-15T14:34:15.063026+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.26204159500000002,"stat_sum":{"num_bytes":4400087,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10511,"num_write":14,"num_write_kb":4701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9416119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":567,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.079973+0000","last_change":"2023-01-15T14:34:16.278357+0000","last_active":"2023-01-15T14:34:17.079973+0000","last_peered":"2023-01-15T14:34:17.079973+0000","last_clean":"2023-01-15T14:34:17.079973+0000","last_became_active":"2023-01-15T14:33:57.125876+0000","last_became_peered":"2023-01-15T14:33:57.125876+0000","last_unstale":"2023-01-15T14:34:17.079973+0000","last_undegraded":"2023-01-15T14:34:17.079973+0000","last_fullsized":"2023-01-15T14:34:17.079973+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:21.278","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.0028318649999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1283,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978994+0000","last_change":"2023-01-15T14:33:57.405395+0000","last_active":"2023-01-15T14:34:16.978994+0000","last_peered":"2023-01-15T14:34:16.978994+0000","last_clean":"2023-01-15T14:34:16.978994+0000","last_became_active":"2023-01-15T14:33:35.362912+0000","last_became_peered":"2023-01-15T14:33:35.362912+0000","last_unstale":"2023-01-15T14:34:16.978994+0000","last_undegraded":"2023-01-15T14:34:16.978994+0000","last_fullsized":"2023-01-15T14:34:16.978994+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.394560678,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":994,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978393+0000","last_change":"2023-01-15T14:33:57.317895+0000","last_active":"2023-01-15T14:34:16.978393+0000","last_peered":"2023-01-15T14:34:16.978393+0000","last_clean":"2023-01-15T14:34:16.978393+0000","last_became_active":"2023-01-15T14:33:35.261561+0000","last_became_peered":"2023-01-15T14:33:35.261561+0000","last_unstale":"2023-01-15T14:34:16.978393+0000","last_undegraded":"2023-01-15T14:34:16.978393+0000","last_fullsized":"2023-01-15T14:34:16.978393+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40752737100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1f","version":"318'256","reported_seq":1340,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.964229+0000","last_change":"2023-01-15T14:34:00.448154+0000","last_active":"2023-01-15T14:34:14.964229+0000","last_peered":"2023-01-15T14:34:14.964229+0000","last_clean":"2023-01-15T14:34:14.964229+0000","last_became_active":"2023-01-15T14:34:00.240936+0000","last_became_peered":"2023-01-15T14:34:00.240936+0000","last_unstale":"2023-01-15T14:34:14.964229+0000","last_undegraded":"2023-01-15T14:34:14.964229+0000","last_fullsized":"2023-01-15T14:34:14.964229+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.20345100599999999,"stat_sum":{"num_bytes":2974888,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":13341,"num_write":1,"num_write_kb":1699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":5949777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.20","version":"324'273","reported_seq":1369,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.964387+0000","last_change":"2023-01-15T14:33:57.415738+0000","last_active":"2023-01-15T14:34:14.964387+0000","last_peered":"2023-01-15T14:34:14.964387+0000","last_clean":"2023-01-15T14:34:14.964387+0000","last_became_active":"2023-01-15T14:33:55.472883+0000","last_became_peered":"2023-01-15T14:33:55.472883+0000","last_unstale":"2023-01-15T14:34:14.964387+0000","last_undegraded":"2023-01-15T14:34:14.964387+0000","last_fullsized":"2023-01-15T14:34:14.964387+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.49046371599999999,"stat_sum":{"num_bytes":4251137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10864,"num_write":9,"num_write_kb":4140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4438968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.21","version":"321'138","reported_seq":1058,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069125+0000","last_change":"2023-01-15T14:33:57.005196+0000","last_active":"2023-01-15T14:34:16.069125+0000","last_peered":"2023-01-15T14:34:16.069125+0000","last_clean":"2023-01-15T14:34:16.069125+0000","last_became_active":"2023-01-15T14:33:45.267321+0000","last_became_peered":"2023-01-15T14:33:45.267321+0000","last_unstale":"2023-01-15T14:34:16.069125+0000","last_undegraded":"2023-01-15T14:34:16.069125+0000","last_fullsized":"2023-01-15T14:34:16.069125+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.102357873,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.22","version":"323'161","reported_seq":1349,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.977591+0000","last_change":"2023-01-15T14:33:57.619172+0000","last_active":"2023-01-15T14:34:16.977591+0000","last_peered":"2023-01-15T14:34:16.977591+0000","last_clean":"2023-01-15T14:34:16.977591+0000","last_became_active":"2023-01-15T14:33:55.378253+0000","last_became_peered":"2023-01-15T14:33:55.378253+0000","last_unstale":"2023-01-15T14:34:16.977591+0000","last_undegraded":"2023-01-15T14:34:16.977591+0000","last_fullsized":"2023-01-15T14:34:16.977591+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.50935289699999997,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":753,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.965541+0000","last_change":"2023-01-15T14:33:57.416771+0000","last_active":"2023-01-15T14:34:14.965541+0000","last_peered":"2023-01-15T14:34:14.965541+0000","last_clean":"2023-01-15T14:34:14.965541+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:34:14.965541+0000","last_undegraded":"2023-01-15T14:34:14.965541+0000","last_fullsized":"2023-01-15T14:34:14.965541+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.45831782799999998,"stat_sum":{"num_bytes":3802136,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":21818,"num_write":18,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.24","version":"349'263","reported_seq":1418,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.987406+0000","last_change":"2023-01-15T14:33:57.616353+0000","last_active":"2023-01-15T14:34:16.987406+0000","last_peered":"2023-01-15T14:34:16.987406+0000","last_clean":"2023-01-15T14:34:16.987406+0000","last_became_active":"2023-01-15T14:33:45.075309+0000","last_became_peered":"2023-01-15T14:33:45.075309+0000","last_unstale":"2023-01-15T14:34:16.987406+0000","last_undegraded":"2023-01-15T14:34:16.987406+0000","last_fullsized":"2023-01-15T14:34:16.987406+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.70814871000000001,"stat_sum":{"num_bytes":7636039,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":22662,"num_write":10,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.25","version":"333'108","reported_seq":1130,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.076583+0000","last_change":"2023-01-15T14:33:57.331159+0000","last_active":"2023-01-15T14:34:16.076583+0000","last_peered":"2023-01-15T14:34:16.076583+0000","last_clean":"2023-01-15T14:34:16.076583+0000","last_became_active":"2023-01-15T14:33:45.168598+0000","last_became_peered":"2023-01-15T14:33:45.168598+0000","last_unstale":"2023-01-15T14:34:16.076583+0000","last_undegraded":"2023-01-15T14:34:16.076583+0000","last_fullsized":"2023-01-15T14:34:16.076583+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.33264791100000002,"stat_sum":{"num_bytes":2638765,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3030,"num_write":2,"num_write_kb":2577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":482,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070812+0000","last_change":"2023-01-15T14:33:58.119558+0000","last_active":"2023-01-15T14:34:16.070812+0000","last_peered":"2023-01-15T14:34:16.070812+0000","last_clean":"2023-01-15T14:34:16.070812+0000","last_became_active":"2023-01-15T14:33:52.309110+0000","last_became_peered":"2023-01-15T14:33:52.309110+0000","last_unstale":"2023-01-15T14:34:16.070812+0000","last_undegraded":"2023-01-15T14:34:16.070812+0000","last_fullsized":"2023-01-15T14:34:16.070812+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.0016362760000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":1081,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.979186+0000","last_change":"2023-01-15T14:33:57.355572+0000","last_active":"2023-01-15T14:34:16.979186+0000","last_peered":"2023-01-15T14:34:16.979186+0000","last_clean":"2023-01-15T14:34:16.979186+0000","last_became_active":"2023-01-15T14:33:45.171090+0000","last_became_peered":"2023-01-15T14:33:45.171090+0000","last_unstale":"2023-01-15T14:34:16.979186+0000","last_undegraded":"2023-01-15T14:34:16.979186+0000","last_fullsized":"2023-01-15T14:34:16.979186+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.443073948,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.28","version":"333'248","reported_seq":1399,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.977267+0000","last_change":"2023-01-15T14:33:57.617361+0000","last_active":"2023-01-15T14:34:16.977267+0000","last_peered":"2023-01-15T14:34:16.977267+0000","last_clean":"2023-01-15T14:34:16.977267+0000","last_became_active":"2023-01-15T14:33:45.366831+0000","last_became_peered":"2023-01-15T14:33:45.366831+0000","last_unstale":"2023-01-15T14:34:16.977267+0000","last_undegraded":"2023-01-15T14:34:16.977267+0000","last_fullsized":"2023-01-15T14:34:16.977267+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.60974191300000002,"stat_sum":{"num_bytes":4838389,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8969,"num_write":8,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":6096764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":903,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077611+0000","last_change":"2023-01-15T14:33:57.620313+0000","last_active":"2023-01-15T14:34:17.077611+0000","last_peered":"2023-01-15T14:34:17.077611+0000","last_clean":"2023-01-15T14:34:17.077611+0000","last_became_active":"2023-01-15T14:33:45.276960+0000","last_became_peered":"2023-01-15T14:33:45.276960+0000","last_unstale":"2023-01-15T14:34:17.077611+0000","last_undegraded":"2023-01-15T14:34:17.077611+0000","last_fullsized":"2023-01-15T14:34:17.077611+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.51033217900000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1248,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069645+0000","last_change":"2023-01-15T14:33:56.904528+0000","last_active":"2023-01-15T14:34:16.069645+0000","last_peered":"2023-01-15T14:34:16.069645+0000","last_clean":"2023-01-15T14:34:16.069645+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:34:16.069645+0000","last_undegraded":"2023-01-15T14:34:16.069645+0000","last_fullsized":"2023-01-15T14:34:16.069645+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.0022419219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":61,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.065904+0000","last_change":"2023-01-15T14:33:57.893154+0000","last_active":"2023-01-15T14:34:15.065904+0000","last_peered":"2023-01-15T14:34:15.065904+0000","last_clean":"2023-01-15T14:34:15.065904+0000","last_became_active":"2023-01-15T14:33:57.261456+0000","last_became_peered":"2023-01-15T14:33:57.261456+0000","last_unstale":"2023-01-15T14:34:15.065904+0000","last_undegraded":"2023-01-15T14:34:15.065904+0000","last_fullsized":"2023-01-15T14:34:15.065904+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.630651882,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2c","version":"141'46","reported_seq":452,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.979605+0000","last_change":"2023-01-15T14:33:57.303687+0000","last_active":"2023-01-15T14:34:16.979605+0000","last_peered":"2023-01-15T14:34:16.979605+0000","last_clean":"2023-01-15T14:34:16.979605+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:34:16.979605+0000","last_undegraded":"2023-01-15T14:34:16.979605+0000","last_fullsized":"2023-01-15T14:34:16.979605+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.29451823399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1221,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.170580+0000","last_change":"2023-01-15T14:33:58.122688+0000","last_active":"2023-01-15T14:34:16.170580+0000","last_peered":"2023-01-15T14:34:16.170580+0000","last_clean":"2023-01-15T14:34:16.170580+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:34:16.170580+0000","last_undegraded":"2023-01-15T14:34:16.170580+0000","last_fullsized":"2023-01-15T14:34:16.170580+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.105464771,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":39,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070235+0000","last_change":"2023-01-15T14:33:58.326712+0000","last_active":"2023-01-15T14:34:16.070235+0000","last_peered":"2023-01-15T14:34:16.070235+0000","last_clean":"2023-01-15T14:34:16.070235+0000","last_became_active":"2023-01-15T14:33:57.210434+0000","last_became_peered":"2023-01-15T14:33:57.210434+0000","last_unstale":"2023-01-15T14:34:16.070235+0000","last_undegraded":"2023-01-15T14:34:16.070235+0000","last_fullsized":"2023-01-15T14:34:16.070235+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.31032643199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1115,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070820+0000","last_change":"2023-01-15T14:33:58.121127+0000","last_active":"2023-01-15T14:34:16.070820+0000","last_peered":"2023-01-15T14:34:16.070820+0000","last_clean":"2023-01-15T14:34:16.070820+0000","last_became_active":"2023-01-15T14:33:55.174128+0000","last_became_peered":"2023-01-15T14:33:55.174128+0000","last_unstale":"2023-01-15T14:34:16.070820+0000","last_undegraded":"2023-01-15T14:34:16.070820+0000","last_fullsized":"2023-01-15T14:34:16.070820+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.20511622600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":991,"reported_epoch":365,"state":"active+clean","last_fresh":"2023-01-15T14:34:14.964193+0000","last_change":"2023-01-15T14:33:57.314582+0000","last_active":"2023-01-15T14:34:14.964193+0000","last_peered":"2023-01-15T14:34:14.964193+0000","last_clean":"2023-01-15T14:34:14.964193+0000","last_became_active":"2023-01-15T14:33:55.373498+0000","last_became_peered":"2023-01-15T14:33:55.373498+0000","last_unstale":"2023-01-15T14:34:14.964193+0000","last_undegraded":"2023-01-15T14:34:14.964193+0000","last_fullsized":"2023-01-15T14:34:14.964193+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":7,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.389510301,"stat_sum":{"num_bytes":5419866,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":25948,"num_write":35,"num_write_kb":13827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":13340458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.31","version":"204'150","reported_seq":1234,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.076397+0000","last_change":"2023-01-15T14:33:57.625989+0000","last_active":"2023-01-15T14:34:16.076397+0000","last_peered":"2023-01-15T14:34:16.076397+0000","last_clean":"2023-01-15T14:34:16.076397+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:34:16.076397+0000","last_undegraded":"2023-01-15T14:34:16.076397+0000","last_fullsized":"2023-01-15T14:34:16.076397+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.52721113600000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":106,"stat_sum":{"num_bytes":359426439,"num_objects":166,"num_object_clones":114,"num_object_copies":498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":166,"num_whiteouts":14,"num_read":1016,"num_read_kb":1109259,"num_write":811,"num_write_kb":333943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":723,"num_bytes_recovered":466236039,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18728,"ondisk_log_size":18728,"up":318,"acting":316,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2624160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574924,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1107636,"kb_used_data":92630184,"kb_used_omap":7967,"kb_used_meta":0,"kb_avail":92591744,"statfs":{"total":95948165120,"available":94813945856,"internally_reserved":39362560,"allocated":94853308416,"data_stored":94853308416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8158317,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":4.1269999999999998,"15min":4.0890000000000004},"min":{"1min":0.54000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2030000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":9.5380000000000003,"5min":6.3200000000000003,"15min":8.2460000000000004},"min":{"1min":0.58799999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":96.337000000000003,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.80900000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":3.6509999999999998,"15min":5.5800000000000001},"min":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":34.585999999999999,"5min":88.003,"15min":88.003},"last":0.90200000000000002},{"interface":"front","average":{"1min":13.302,"5min":8.1950000000000003,"15min":6.9180000000000001},"min":{"1min":0.58799999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.403999999999996,"5min":97.704999999999998,"15min":97.704999999999998},"last":1.3460000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":6.9400000000000004,"15min":5.5069999999999997},"min":{"1min":0.57199999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.179,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":2.5790000000000002,"15min":3.645},"min":{"1min":0.626,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":91.837999999999994,"15min":91.837999999999994},"last":0.99399999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":7.3799999999999999,"15min":3.0920000000000001},"min":{"1min":0.63600000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2370000000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.91600000000000004,"5min":3.6360000000000001,"15min":4.3129999999999997},"min":{"1min":0.65000000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.2270000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":1.1220000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5519999999999996,"5min":5.6219999999999999,"15min":8.1509999999999998},"min":{"1min":0.63500000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":96.185000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.9259999999999999,"5min":5.782,"15min":2.5699999999999998},"min":{"1min":0.73099999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":22.41,"5min":164.21199999999999,"15min":164.21199999999999},"last":1.042}]},{"osd":5,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.6440000000000001,"5min":5.0659999999999998,"15min":5.9059999999999997},"min":{"1min":0.72799999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":96.263999999999996,"5min":96.263999999999996,"15min":96.263999999999996},"last":1.071},{"interface":"front","average":{"1min":0.998,"5min":3.206,"15min":4.3650000000000002},"min":{"1min":0.69299999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.355,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.141}]},{"osd":6,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":5.5949999999999998,"15min":7.5469999999999997},"min":{"1min":0.83599999999999997,"5min":0.496,"15min":0.496},"max":{"1min":96.299999999999997,"5min":96.299999999999997,"15min":96.299999999999997},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.089,"5min":2.5150000000000001,"15min":1.571},"min":{"1min":0.84299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.2789999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.2050000000000001}]}]},{"osd":6,"up_from":0,"seq":55834574925,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574925,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574924,"num_pgs":62,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":989272,"kb_used_data":92710224,"kb_used_omap":13956,"kb_used_meta":0,"kb_avail":92710108,"statfs":{"total":95948165120,"available":94935150592,"internally_reserved":118784,"allocated":94935269376,"data_stored":94935269376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14290945,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":22.818000000000001,"15min":10.882},"min":{"1min":0.505,"5min":0.44,"15min":0.44},"max":{"1min":5.0049999999999999,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.997},{"interface":"front","average":{"1min":0.77400000000000002,"5min":2.9990000000000001,"15min":1.52},"min":{"1min":0.51200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.127,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.73799999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":5.0730000000000004,"15min":7.4199999999999999},"min":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6659999999999999,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.343,"15min":1.9770000000000001},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0529999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.93300000000000005}]},{"osd":2,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.0950000000000002,"5min":23.544,"15min":11.522},"min":{"1min":0.497,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":23.247,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.26,"5min":1.28,"15min":1.0329999999999999},"min":{"1min":0.61899999999999999,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.5789999999999997,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.68500000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":2.802,"15min":1.8260000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.006,"5min":88.823999999999998,"15min":88.823999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":5.3380000000000001,"5min":24.337,"15min":12.212999999999999},"min":{"1min":0.68600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":58.317,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":0.89900000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88100000000000001,"15min":0.90700000000000003},"min":{"1min":0.61499999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.175,"5min":2.577,"15min":2.577},"last":1.1499999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.903,"15min":2.1669999999999998},"min":{"1min":0.69099999999999995,"5min":0.65200000000000002,"15min":0.55300000000000005},"max":{"1min":1.0700000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.82299999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":3.2389999999999999,"15min":4.8609999999999998},"min":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":23.329000000000001,"5min":85.721999999999994,"15min":99.780000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.999,"5min":21.631,"15min":10.651999999999999},"min":{"1min":0.79500000000000004,"5min":0.66700000000000004,"15min":0.58699999999999997},"max":{"1min":1.173,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.095}]},{"osd":7,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":3.238,"5min":3.5790000000000002,"15min":9.9450000000000003},"min":{"1min":0.73999999999999999,"5min":0.63400000000000001,"15min":0.59499999999999997},"max":{"1min":42.551000000000002,"5min":89.061000000000007,"15min":179.73500000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":2.274,"5min":23.071000000000002,"15min":15.183},"min":{"1min":0.78700000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":23.356000000000002,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.0660000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":61,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1057144,"kb_used_data":92642248,"kb_used_omap":14201,"kb_used_meta":0,"kb_avail":92642236,"statfs":{"total":95948165120,"available":94865649664,"internally_reserved":12288,"allocated":94865661952,"data_stored":94865661952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14541852,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":7.4400000000000004,"15min":5.7119999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":17.731000000000002,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":3.4340000000000002,"15min":6.2389999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.9260000000000002,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.77300000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":7.1790000000000003,"15min":11.414},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96799999999999997,"5min":159.256,"15min":159.256},"last":0.65300000000000002},{"interface":"front","average":{"1min":2.6259999999999999,"5min":10.417999999999999,"15min":3.9660000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":17.826000000000001,"5min":174.578,"15min":174.578},"last":0.58999999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":4.6849999999999996,"5min":2.8809999999999998,"15min":3.1930000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":86.412000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.628},{"interface":"front","average":{"1min":1.95,"5min":6.7889999999999997,"15min":3.7250000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.876000000000001,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.747}]},{"osd":4,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":7.6879999999999997,"5min":29.686,"15min":14.27},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":84.640000000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.86899999999999999,"5min":32.134999999999998,"15min":19.960999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.615,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":1.125}]},{"osd":5,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":6.4980000000000002,"15min":11.433999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.04,"5min":159.43000000000001,"15min":159.43000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":2.2309999999999999,"5min":6.0039999999999996,"15min":2.6819999999999999},"min":{"1min":0.70099999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":17.917000000000002,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.86199999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1949999999999998,"5min":5.9980000000000002,"15min":2.6869999999999998},"min":{"1min":0.66000000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":17.989000000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.964,"15min":3.1619999999999999},"min":{"1min":0.70499999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.23,"5min":60.098999999999997,"15min":60.098999999999997},"last":1.006}]},{"osd":7,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":2.0219999999999998,"15min":3.3639999999999999},"min":{"1min":0.69499999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.266,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.0589999999999999},{"interface":"front","average":{"1min":19.138000000000002,"5min":11.705,"15min":15.961},"min":{"1min":0.66200000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":281.13400000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":1.1579999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574924,"num_pgs":49,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1038832,"kb_used_data":92660684,"kb_used_omap":11768,"kb_used_meta":0,"kb_avail":92660548,"statfs":{"total":95948165120,"available":94884401152,"internally_reserved":139264,"allocated":94884540416,"data_stored":94884540416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12050976,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":4.2469999999999999,"5min":4.2450000000000001,"15min":1.8420000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":78.456000000000003,"5min":78.456000000000003,"15min":78.456000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.4039999999999999,"15min":0.94399999999999995},"min":{"1min":0.36799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.68600000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":10.247,"5min":7.7949999999999999,"15min":3.1309999999999998},"min":{"1min":0.35099999999999998,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":128.749,"5min":143.071,"15min":143.071},"last":87.209999999999994},{"interface":"front","average":{"1min":3.105,"5min":1.966,"15min":2.8919999999999999},"min":{"1min":0.35999999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":52.927999999999997,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.752}]},{"osd":3,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":3.8879999999999999,"5min":6.9180000000000001,"15min":9.2400000000000002},"min":{"1min":0.33700000000000002,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":34.244,"5min":143.13300000000001,"15min":183.53},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":7.5039999999999996,"15min":3.012},"min":{"1min":0.41399999999999998,"5min":0.316,"15min":0.316},"max":{"1min":1.3959999999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.76700000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":5.9480000000000004,"15min":4.4160000000000004},"min":{"1min":0.67200000000000004,"5min":0.624,"15min":0.58399999999999996},"max":{"1min":1.2450000000000001,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.88300000000000001,"5min":4.7080000000000002,"15min":4.1079999999999997},"min":{"1min":0.57799999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.052,"5min":175.30500000000001,"15min":175.30500000000001},"last":1.1359999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":2.5590000000000002,"5min":2.7490000000000001,"15min":1.7},"min":{"1min":0.63500000000000001,"5min":0.57199999999999995,"15min":0.54800000000000004},"max":{"1min":34.256999999999998,"5min":143.262,"15min":143.262},"last":1.014},{"interface":"front","average":{"1min":0.95099999999999996,"5min":3.012,"15min":1.778},"min":{"1min":0.70199999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3160000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.89700000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.839,"15min":1.3009999999999999},"min":{"1min":0.61199999999999999,"5min":0.59599999999999997,"15min":0.55000000000000004},"max":{"1min":1.276,"5min":88.542000000000002,"15min":88.542000000000002},"last":1.0580000000000001},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8980000000000001,"15min":1.778},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":34.305999999999997,"5min":143.215,"15min":143.215},"last":0.94199999999999995}]},{"osd":7,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":11.396000000000001,"5min":7.9800000000000004,"15min":4.6989999999999998},"min":{"1min":0.60599999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":176.44499999999999,"5min":176.44499999999999,"15min":176.44499999999999},"last":1.1879999999999999},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8159999999999998,"15min":2.79},"min":{"1min":0.622,"5min":0.622,"15min":0.54500000000000004},"max":{"1min":34.207999999999998,"5min":143.17099999999999,"15min":143.17099999999999},"last":1.117}]}]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":72,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1061392,"kb_used_data":92637988,"kb_used_omap":13231,"kb_used_meta":0,"kb_avail":92637988,"statfs":{"total":95948165120,"available":94861299712,"internally_reserved":0,"allocated":94861299712,"data_stored":94861299712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13549314,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":7,"commit_latency_ns":0,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":2.0619999999999998,"5min":3.2669999999999999,"15min":3.6840000000000002},"min":{"1min":0.37,"5min":0.314,"15min":0.314},"max":{"1min":23.68,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":14.157,"5min":6.5789999999999997,"15min":5.0750000000000002},"min":{"1min":0.42299999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":176.01599999999999,"5min":176.01599999999999,"15min":176.01599999999999},"last":101.319}]},{"osd":2,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":4.8959999999999999,"5min":3.4329999999999998,"15min":1.7889999999999999},"min":{"1min":0.44400000000000001,"5min":0.245,"15min":0.245},"max":{"1min":69.277000000000001,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.629,"5min":1.173,"15min":3.1059999999999999},"min":{"1min":0.28399999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.873,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":5.8440000000000003,"15min":15.172000000000001},"min":{"1min":0.38400000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.92500000000000004,"5min":245.637,"15min":282.01400000000001},"last":0.88},{"interface":"front","average":{"1min":14.228999999999999,"5min":8.2870000000000008,"15min":7.5919999999999996},"min":{"1min":0.46500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":176.13900000000001,"5min":176.13900000000001,"15min":176.13900000000001},"last":101.401}]},{"osd":4,"last update":"Sun Jan 15 14:33:17 2023","interfaces":[{"interface":"back","average":{"1min":7.9249999999999998,"5min":17.140999999999998,"15min":8.8650000000000002},"min":{"1min":0.504,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":119.71899999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":101.575},{"interface":"front","average":{"1min":1.7849999999999999,"5min":15.526,"15min":7.5780000000000003},"min":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":16.989999999999998,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":1.724}]},{"osd":5,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":5.2060000000000004,"5min":2.3799999999999999,"15min":1.482},"min":{"1min":0.56899999999999995,"5min":0.433,"15min":0.433},"max":{"1min":69.478999999999999,"5min":69.478999999999999,"15min":69.478999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":3.2770000000000001},"min":{"1min":0.66000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3089999999999999,"5min":4.0780000000000003,"15min":46.905999999999999},"last":0.82999999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":14.491,"5min":6.9109999999999996,"15min":5.6379999999999999},"min":{"1min":0.73199999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":176.249,"5min":176.249,"15min":176.249},"last":101.45999999999999},{"interface":"front","average":{"1min":5.2629999999999999,"5min":2.431,"15min":1.5589999999999999},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":69.549999999999997,"5min":69.549999999999997,"15min":69.549999999999997},"last":1.02}]},{"osd":7,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":7.617,"5min":6.2309999999999999,"15min":6.7549999999999999},"min":{"1min":0.63900000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":95.888999999999996,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":14.544,"5min":6.9649999999999999,"15min":5.4880000000000004},"min":{"1min":0.78700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":176.22,"5min":176.22,"15min":176.22},"last":101.515}]}]},{"osd":0,"up_from":13,"seq":55834574925,"num_pgs":69,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1079580,"kb_used_data":92631028,"kb_used_omap":7987,"kb_used_meta":0,"kb_avail":92619800,"statfs":{"total":95948165120,"available":94842675200,"internally_reserved":11497472,"allocated":94854172672,"data_stored":94854172672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8179559,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":3.3460000000000001,"15min":1.53},"min":{"1min":0.34899999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":25.960999999999999,"5min":69.872,"15min":69.872},"last":0.52500000000000002},{"interface":"front","average":{"1min":11.249000000000001,"5min":5.335,"15min":2.2650000000000001},"min":{"1min":0.434,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":162.91200000000001,"5min":162.91200000000001,"15min":162.91200000000001},"last":0.79700000000000004}]},{"osd":2,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":3.415,"15min":1.601},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0109999999999999,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":2.6320000000000001,"15min":1.3500000000000001},"min":{"1min":0.37,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86899999999999999,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.73099999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":7.8079999999999998,"15min":13.788},"min":{"1min":0.48199999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":26.027000000000001,"5min":199.08000000000001,"15min":284.62799999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":3.6070000000000002,"5min":7.867,"15min":3.1560000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33600000000000002},"max":{"1min":61.119,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.93100000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":16.664999999999999,"15min":6.2119999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1719999999999999,"5min":1486.309,"15min":1486.309},"last":1.016},{"interface":"front","average":{"1min":2.1030000000000002,"5min":18.140999999999998,"15min":6.7300000000000004},"min":{"1min":0.63100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":26.137,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":0.71699999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":2.8980000000000001,"15min":7.4980000000000002},"min":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1930000000000001,"5min":98.411000000000001,"15min":176.34399999999999},"last":1.046},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.7070000000000001,"15min":1.2609999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.101,"5min":71.563999999999993,"15min":71.563999999999993},"last":1.101}]},{"osd":6,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":2.1600000000000001,"5min":3.7890000000000001,"15min":2.0430000000000001},"min":{"1min":0.71899999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":26.082000000000001,"5min":70.259,"15min":70.259},"last":0.97099999999999997},{"interface":"front","average":{"1min":1.026,"5min":3,"15min":1.756},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.57499999999999996},"max":{"1min":1.5409999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":1.2729999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":3.2770000000000001,"15min":11.539},"min":{"1min":0.70099999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":11.906000000000001,"5min":73.864999999999995,"15min":292.61099999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":2.1819999999999999,"5min":3.8090000000000002,"15min":2.0499999999999998},"min":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.58699999999999997},"max":{"1min":26.187000000000001,"5min":70.322000000000003,"15min":70.322000000000003},"last":1.0660000000000001}]}]}],"pool_statfs":[]}} 2023-01-15T14:34:19.799 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-01-15T14:34:20.285 INFO:tasks.rados.rados.0.smithi090.stdout:1396: done (1 left) 2023-01-15T14:34:20.285 INFO:tasks.rados.rados.0.smithi090.stdout:1397: write oid 1 current snap is 149 2023-01-15T14:34:20.285 INFO:tasks.rados.rados.0.smithi090.stdout:1397: seq_num 500 ranges {403500=787445,1692526=473837} 2023-01-15T14:34:20.286 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:34:20.288 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:20.288 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:34:20.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:34:20.289 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:34:20.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.289 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:5 3.5 3:a1c47f99:::smithi09025264-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:20.289 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:20.290 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.76777907399998, 2023-01-15T14:34:20.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40548699700000002, 2023-01-15T14:34:20.290 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:20.290 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:20.290 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:20.290 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:20.291 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:20.291 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 5 2023-01-15T14:34:20.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.291 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:20.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:20.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:20.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:20.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:20.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:20.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:20.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:20.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781744+0000", 2023-01-15T14:34:20.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:20.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.609 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.609 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:20.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782566+0000", 2023-01-15T14:34:20.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000821578 2023-01-15T14:34:20.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.609 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.609 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:20.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782588+0000", 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.226e-05 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782601+0000", 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2998e-05 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882850+0000", 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100248908 2023-01-15T14:34:20.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882871+0000", 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0873000000000002e-05 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083679+0000", 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200808083 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:20.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083948+0000", 2023-01-15T14:34:20.612 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000269342 2023-01-15T14:34:20.612 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.612 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.612 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:20.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084161+0000", 2023-01-15T14:34:20.612 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021326900000000001 2023-01-15T14:34:20.612 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.612 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.612 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:20.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084655+0000", 2023-01-15T14:34:20.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049371300000000001 2023-01-15T14:34:20.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.615 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:20.615 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084674+0000", 2023-01-15T14:34:20.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9012999999999999e-05 2023-01-15T14:34:20.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.615 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:20.615 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084924+0000", 2023-01-15T14:34:20.616 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024989899999999999 2023-01-15T14:34:20.616 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:20.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085221+0000", 2023-01-15T14:34:20.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029719600000000001 2023-01-15T14:34:20.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:20.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186953+0000", 2023-01-15T14:34:20.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10173174 2023-01-15T14:34:20.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.762 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:20.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187026+0000", 2023-01-15T14:34:20.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2957e-05 2023-01-15T14:34:20.762 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.762 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187124+0000", 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8146000000000005e-05 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187233+0000", 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010931599999999999 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:20.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.764 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:6 3.5 3:a1c47f99:::smithi09025264-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:20.764 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:20.764 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.762526202, 2023-01-15T14:34:20.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50419954199999995, 2023-01-15T14:34:20.764 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:20.764 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:20.764 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:20.765 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:20.765 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:20.765 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 6 2023-01-15T14:34:20.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.765 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:20.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:20.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:20.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:20.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:20.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:20.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:20.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:20.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786997+0000", 2023-01-15T14:34:20.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:20.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:20.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787486+0000", 2023-01-15T14:34:20.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048882999999999999 2023-01-15T14:34:20.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:20.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787514+0000", 2023-01-15T14:34:20.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7591999999999999e-05 2023-01-15T14:34:20.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:20.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787526+0000", 2023-01-15T14:34:20.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1894e-05 2023-01-15T14:34:20.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882966+0000", 2023-01-15T14:34:20.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095440450999999996 2023-01-15T14:34:20.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:20.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882973+0000", 2023-01-15T14:34:20.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2570000000000001e-06 2023-01-15T14:34:20.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084809+0000", 2023-01-15T14:34:20.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201835763 2023-01-15T14:34:20.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.776 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:20.776 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084928+0000", 2023-01-15T14:34:20.776 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118946 2023-01-15T14:34:20.776 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.776 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.781 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.781 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187347+0000", 2023-01-15T14:34:20.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102419285 2023-01-15T14:34:20.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.781 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:20.781 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187450+0000", 2023-01-15T14:34:20.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010286299999999999 2023-01-15T14:34:20.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.781 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:20.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187735+0000", 2023-01-15T14:34:20.782 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028429900000000001 2023-01-15T14:34:20.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:20.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188113+0000", 2023-01-15T14:34:20.782 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037883200000000001 2023-01-15T14:34:20.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:20.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188138+0000", 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4110000000000001e-05 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188389+0000", 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025173900000000001 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189251+0000", 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086129800000000003 2023-01-15T14:34:20.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290989+0000", 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101738524 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291075+0000", 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5469000000000004e-05 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291144+0000", 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291199+0000", 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4413000000000002e-05 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.786 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:7 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:20.786 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:20.786 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.76197974500002, 2023-01-15T14:34:20.786 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50407251399999997, 2023-01-15T14:34:20.786 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:20.786 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:20.786 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:20.786 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:20.786 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:20.786 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 7 2023-01-15T14:34:20.786 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.788 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:20.788 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787545+0000", 2023-01-15T14:34:20.788 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:20.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:20.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787551+0000", 2023-01-15T14:34:20.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8669999999999999e-06 2023-01-15T14:34:20.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.789 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.789 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:20.789 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787559+0000", 2023-01-15T14:34:20.789 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9780000000000005e-06 2023-01-15T14:34:20.789 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:20.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787564+0000", 2023-01-15T14:34:20.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5709999999999997e-06 2023-01-15T14:34:20.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883054+0000", 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095489722999999999 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883061+0000", 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2760000000000003e-06 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084987+0000", 2023-01-15T14:34:20.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20192545000000001 2023-01-15T14:34:20.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:20.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085190+0000", 2023-01-15T14:34:20.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020288899999999999 2023-01-15T14:34:20.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188374+0000", 2023-01-15T14:34:20.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103184449 2023-01-15T14:34:20.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:20.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188543+0000", 2023-01-15T14:34:20.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016925299999999999 2023-01-15T14:34:20.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:20.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188748+0000", 2023-01-15T14:34:20.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020473600000000001 2023-01-15T14:34:20.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:20.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188986+0000", 2023-01-15T14:34:20.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023780900000000001 2023-01-15T14:34:20.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:20.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189006+0000", 2023-01-15T14:34:20.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0604000000000001e-05 2023-01-15T14:34:20.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.925 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:20.925 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189084+0000", 2023-01-15T14:34:20.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7526000000000003e-05 2023-01-15T14:34:20.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.925 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189272+0000", 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018762 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291333+0000", 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102061185 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291466+0000", 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013325699999999999 2023-01-15T14:34:20.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:20.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291535+0000", 2023-01-15T14:34:20.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9309999999999999e-05 2023-01-15T14:34:20.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:20.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291618+0000", 2023-01-15T14:34:20.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2773000000000002e-05 2023-01-15T14:34:20.928 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:20.928 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:20.928 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:20.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.928 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:8 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:20.928 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:20.928 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.76195376300001, 2023-01-15T14:34:20.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50442076000000002, 2023-01-15T14:34:20.929 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:20.929 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:20.930 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:20.930 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:20.930 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:20.930 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 8 2023-01-15T14:34:20.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.930 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:20.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.931 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:20.931 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:20.931 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:20.931 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.931 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.931 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:20.931 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:20.931 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:20.931 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.932 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.932 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:20.932 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787571+0000", 2023-01-15T14:34:20.932 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:20.932 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.932 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.932 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:20.932 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787575+0000", 2023-01-15T14:34:20.932 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7890000000000001e-06 2023-01-15T14:34:20.932 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.932 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.933 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:20.933 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787589+0000", 2023-01-15T14:34:20.933 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4212999999999999e-05 2023-01-15T14:34:20.933 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.933 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.933 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:20.933 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787594+0000", 2023-01-15T14:34:20.933 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1429999999999997e-06 2023-01-15T14:34:20.933 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.933 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.933 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.934 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983143+0000", 2023-01-15T14:34:20.934 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.195549003 2023-01-15T14:34:20.934 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.934 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.934 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:20.934 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983153+0000", 2023-01-15T14:34:20.934 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4369999999999996e-06 2023-01-15T14:34:20.934 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.934 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.934 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.934 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085286+0000", 2023-01-15T14:34:20.935 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102133064 2023-01-15T14:34:20.937 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.937 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.938 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:20.939 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085401+0000", 2023-01-15T14:34:20.939 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115176 2023-01-15T14:34:20.939 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.939 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.939 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.940 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189116+0000", 2023-01-15T14:34:20.940 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103715444 2023-01-15T14:34:20.941 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.941 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.941 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:20.941 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189221+0000", 2023-01-15T14:34:20.941 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010438 2023-01-15T14:34:20.942 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.942 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.942 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.942 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291724+0000", 2023-01-15T14:34:20.942 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102502998 2023-01-15T14:34:20.942 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.943 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.943 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:20.943 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291824+0000", 2023-01-15T14:34:20.943 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010010599999999999 2023-01-15T14:34:20.943 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.943 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.943 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:20.944 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291992+0000", 2023-01-15T14:34:20.944 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016856900000000001 2023-01-15T14:34:20.944 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:20.944 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:20.944 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:20.944 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.944 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.945 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:21 3.5 3:a75958b1:::smithi09025264-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:20.945 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:20.945 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.71786484799998, 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.45851924500000002, 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 21 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:20.946 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831657+0000", 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:20.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.948 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.948 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:20.948 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832814+0000", 2023-01-15T14:34:20.948 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011564909999999999 2023-01-15T14:34:20.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.949 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:20.949 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832864+0000", 2023-01-15T14:34:20.949 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0243e-05 2023-01-15T14:34:20.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.949 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832903+0000", 2023-01-15T14:34:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8704000000000001e-05 2023-01-15T14:34:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983202+0000", 2023-01-15T14:34:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.150299404 2023-01-15T14:34:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983208+0000", 2023-01-15T14:34:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0039999999999996e-06 2023-01-15T14:34:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185539+0000", 2023-01-15T14:34:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202330867 2023-01-15T14:34:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185803+0000", 2023-01-15T14:34:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026394299999999999 2023-01-15T14:34:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186037+0000", 2023-01-15T14:34:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023386300000000001 2023-01-15T14:34:20.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186637+0000", 2023-01-15T14:34:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060001800000000001 2023-01-15T14:34:20.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186665+0000", 2023-01-15T14:34:21.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8459999999999999e-05 2023-01-15T14:34:21.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187045+0000", 2023-01-15T14:34:21.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037992899999999999 2023-01-15T14:34:21.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:21.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188201+0000", 2023-01-15T14:34:21.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001155309 2023-01-15T14:34:21.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289903+0000", 2023-01-15T14:34:21.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101702601 2023-01-15T14:34:21.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.086 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.086 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289985+0000", 2023-01-15T14:34:21.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1617000000000001e-05 2023-01-15T14:34:21.086 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.086 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.086 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290065+0000", 2023-01-15T14:34:21.087 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0276000000000002e-05 2023-01-15T14:34:21.087 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.087 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.087 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290180+0000", 2023-01-15T14:34:21.087 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011464300000000001 2023-01-15T14:34:21.087 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.087 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.087 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.087 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.088 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:22 3.5 3:a75958b1:::smithi09025264-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:21.088 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:21.088 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.71339716, 2023-01-15T14:34:21.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65790599599999999, 2023-01-15T14:34:21.088 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.088 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:21.088 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:21.088 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:21.088 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:21.088 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 22 2023-01-15T14:34:21.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.089 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:21.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:21.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836126+0000", 2023-01-15T14:34:21.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:21.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836181+0000", 2023-01-15T14:34:21.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4246999999999997e-05 2023-01-15T14:34:21.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836212+0000", 2023-01-15T14:34:21.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1843e-05 2023-01-15T14:34:21.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836226+0000", 2023-01-15T14:34:21.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3216e-05 2023-01-15T14:34:21.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186757+0000", 2023-01-15T14:34:21.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35053103899999999 2023-01-15T14:34:21.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:21.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186871+0000", 2023-01-15T14:34:21.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114403 2023-01-15T14:34:21.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290263+0000", 2023-01-15T14:34:21.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103391976 2023-01-15T14:34:21.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290413+0000", 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014991200000000001 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290627+0000", 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021379299999999999 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290861+0000", 2023-01-15T14:34:21.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023479600000000001 2023-01-15T14:34:21.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290888+0000", 2023-01-15T14:34:21.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6570000000000001e-05 2023-01-15T14:34:21.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290990+0000", 2023-01-15T14:34:21.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101723 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291245+0000", 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025543899999999999 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392822+0000", 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10157725500000001 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.097 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.097 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493881+0000", 2023-01-15T14:34:21.097 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101058262 2023-01-15T14:34:21.097 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.097 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.097 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493964+0000", 2023-01-15T14:34:21.097 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2973000000000007e-05 2023-01-15T14:34:21.097 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.097 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494034+0000", 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0432999999999995e-05 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:23 3.5 3:a75958b1:::smithi09025264-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.71328203100001, 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65796116900000001, 2023-01-15T14:34:21.098 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.241 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:21.241 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:21.241 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:21.242 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:21.242 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 23 2023-01-15T14:34:21.242 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.242 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.242 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.242 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.243 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:21.243 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.243 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.243 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:21.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.244 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.245 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836242+0000", 2023-01-15T14:34:21.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:21.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.245 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.245 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836256+0000", 2023-01-15T14:34:21.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3677e-05 2023-01-15T14:34:21.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836265+0000", 2023-01-15T14:34:21.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2769999999999994e-06 2023-01-15T14:34:21.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.249 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936398+0000", 2023-01-15T14:34:21.250 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100132559 2023-01-15T14:34:21.250 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.250 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.253 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.253 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289431+0000", 2023-01-15T14:34:21.253 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35303322399999998 2023-01-15T14:34:21.253 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.253 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.253 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:21.253 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289587+0000", 2023-01-15T14:34:21.253 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000155873 2023-01-15T14:34:21.253 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.253 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.253 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.254 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292052+0000", 2023-01-15T14:34:21.254 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0024647919999999999 2023-01-15T14:34:21.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292180+0000", 2023-01-15T14:34:21.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012810999999999999 2023-01-15T14:34:21.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:21.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292373+0000", 2023-01-15T14:34:21.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019258599999999999 2023-01-15T14:34:21.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:21.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292597+0000", 2023-01-15T14:34:21.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022465500000000001 2023-01-15T14:34:21.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292618+0000", 2023-01-15T14:34:21.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0319999999999999e-05 2023-01-15T14:34:21.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292692+0000", 2023-01-15T14:34:21.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4643999999999999e-05 2023-01-15T14:34:21.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:21.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393103+0000", 2023-01-15T14:34:21.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100411156 2023-01-15T14:34:21.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493827+0000", 2023-01-15T14:34:21.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100724143 2023-01-15T14:34:21.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494085+0000", 2023-01-15T14:34:21.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000257891 2023-01-15T14:34:21.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494141+0000", 2023-01-15T14:34:21.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6132999999999999e-05 2023-01-15T14:34:21.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494204+0000", 2023-01-15T14:34:21.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2966000000000007e-05 2023-01-15T14:34:21.267 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.267 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.268 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.268 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.268 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.268 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:24 3.5 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:21.268 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:21.268 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.613100605, 2023-01-15T14:34:21.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.55808677600000001, 2023-01-15T14:34:21.268 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.268 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:21.268 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:21.268 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:21.269 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:21.269 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 24 2023-01-15T14:34:21.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.269 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.269 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.269 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:21.269 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.269 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936423+0000", 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936433+0000", 2023-01-15T14:34:21.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0443e-05 2023-01-15T14:34:21.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936451+0000", 2023-01-15T14:34:21.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7842999999999998e-05 2023-01-15T14:34:21.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936461+0000", 2023-01-15T14:34:21.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0057e-05 2023-01-15T14:34:21.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289693+0000", 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.353231617 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289814+0000", 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120763 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392923+0000", 2023-01-15T14:34:21.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10310955300000001 2023-01-15T14:34:21.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:21.413 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393072+0000", 2023-01-15T14:34:21.413 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000148469 2023-01-15T14:34:21.413 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494242+0000", 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117034499999999 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494363+0000", 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120703 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.414 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.415 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494511+0000", 2023-01-15T14:34:21.416 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014873299999999999 2023-01-15T14:34:21.416 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.416 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.416 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.416 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.416 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.416 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:78 3.c 3:3137f0cb:::smithi09025264-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:21.416 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:21.416 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.223134609, 2023-01-15T14:34:21.416 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20331801699999999, 2023-01-15T14:34:21.416 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.417 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:21.417 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:21.418 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:21.418 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:21.418 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 78 2023-01-15T14:34:21.418 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.418 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.419 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.419 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.419 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:21.420 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.420 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.420 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.420 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.420 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:21.421 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.421 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.422 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.422 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.422 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326388+0000", 2023-01-15T14:34:21.423 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:21.423 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.424 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.424 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.425 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326672+0000", 2023-01-15T14:34:21.425 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028367900000000001 2023-01-15T14:34:21.425 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.426 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.426 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.426 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326713+0000", 2023-01-15T14:34:21.426 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1539999999999999e-05 2023-01-15T14:34:21.426 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.426 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.427 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.427 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326729+0000", 2023-01-15T14:34:21.427 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.519e-05 2023-01-15T14:34:21.427 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.427 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.427 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.427 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326879+0000", 2023-01-15T14:34:21.427 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015064899999999999 2023-01-15T14:34:21.427 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.427 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.428 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.428 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327096+0000", 2023-01-15T14:34:21.428 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021682299999999999 2023-01-15T14:34:21.428 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.428 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.428 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:34:21.428 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327308+0000", 2023-01-15T14:34:21.428 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000211366 2023-01-15T14:34:21.428 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.428 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.429 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:21.429 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327757+0000", 2023-01-15T14:34:21.429 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000449009 2023-01-15T14:34:21.429 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.429 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.429 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.429 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327776+0000", 2023-01-15T14:34:21.429 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9201000000000001e-05 2023-01-15T14:34:21.429 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.429 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.430 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328012+0000", 2023-01-15T14:34:21.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236054 2023-01-15T14:34:21.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.430 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:21.430 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429534+0000", 2023-01-15T14:34:21.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10152201499999999 2023-01-15T14:34:21.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.430 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.431 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529340+0000", 2023-01-15T14:34:21.431 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099806249 2023-01-15T14:34:21.431 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.431 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.431 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.431 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529437+0000", 2023-01-15T14:34:21.431 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6949999999999998e-05 2023-01-15T14:34:21.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529587+0000", 2023-01-15T14:34:21.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015008600000000001 2023-01-15T14:34:21.432 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.432 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.432 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.432 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529709+0000", 2023-01-15T14:34:21.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121636 2023-01-15T14:34:21.432 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.433 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.433 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.433 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.433 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:79 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:21.433 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:21.433 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.22276501300001, 2023-01-15T14:34:21.434 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20348171300000001, 2023-01-15T14:34:21.434 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.434 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:21.434 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:21.434 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:21.434 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:21.434 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 79 2023-01-15T14:34:21.434 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.434 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.434 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.434 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326759+0000", 2023-01-15T14:34:21.435 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:21.436 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.436 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.436 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.436 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326770+0000", 2023-01-15T14:34:21.437 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:34:21.437 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.437 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.437 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.437 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326791+0000", 2023-01-15T14:34:21.437 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1556000000000002e-05 2023-01-15T14:34:21.437 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.437 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.437 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.437 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326818+0000", 2023-01-15T14:34:21.438 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6979999999999999e-05 2023-01-15T14:34:21.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.438 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.438 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.438 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.427993+0000", 2023-01-15T14:34:21.438 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117517199999999 2023-01-15T14:34:21.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.438 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.438 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.438 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428266+0000", 2023-01-15T14:34:21.438 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027233299999999998 2023-01-15T14:34:21.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:34:21.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428696+0000", 2023-01-15T14:34:21.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042996499999999998 2023-01-15T14:34:21.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:21.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429057+0000", 2023-01-15T14:34:21.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000360952 2023-01-15T14:34:21.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.440 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429076+0000", 2023-01-15T14:34:21.440 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:34:21.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429160+0000", 2023-01-15T14:34:21.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:34:21.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:21.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429549+0000", 2023-01-15T14:34:21.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038884599999999998 2023-01-15T14:34:21.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529886+0000", 2023-01-15T14:34:21.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100336261 2023-01-15T14:34:21.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529978+0000", 2023-01-15T14:34:21.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2775999999999994e-05 2023-01-15T14:34:21.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530108+0000", 2023-01-15T14:34:21.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012958300000000001 2023-01-15T14:34:21.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530242+0000", 2023-01-15T14:34:21.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013406 2023-01-15T14:34:21.587 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.587 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.587 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.587 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:80 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:21.587 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:21.587 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.22262665400001, 2023-01-15T14:34:21.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20370874, 2023-01-15T14:34:21.588 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.588 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:21.588 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:21.588 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:21.588 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 80 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:21.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326897+0000", 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326906+0000", 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6929999999999996e-06 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326931+0000", 2023-01-15T14:34:21.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4757e-05 2023-01-15T14:34:21.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326945+0000", 2023-01-15T14:34:21.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.402e-05 2023-01-15T14:34:21.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429239+0000", 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102294418 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429487+0000", 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024789300000000001 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530309+0000", 2023-01-15T14:34:21.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100821459 2023-01-15T14:34:21.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530438+0000", 2023-01-15T14:34:21.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012928700000000001 2023-01-15T14:34:21.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530607+0000", 2023-01-15T14:34:21.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169483 2023-01-15T14:34:21.594 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.594 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.595 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.595 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.595 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:78 3.c e23/20 ondisk, result = 0)", 2023-01-15T14:34:21.595 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:21.595 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 353.22073493300002, 2023-01-15T14:34:21.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20094287799999999, 2023-01-15T14:34:21.596 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.596 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:21.596 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.596 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:21.596 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:21.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328789+0000", 2023-01-15T14:34:21.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:21.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328799+0000", 2023-01-15T14:34:21.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8980000000000008e-06 2023-01-15T14:34:21.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328820+0000", 2023-01-15T14:34:21.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1041999999999999e-05 2023-01-15T14:34:21.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328832+0000", 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1616e-05 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529404+0000", 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200572631 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529419+0000", 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4250000000000001e-05 2023-01-15T14:34:21.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529733+0000", 2023-01-15T14:34:21.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000314693 2023-01-15T14:34:21.739 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.739 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.739 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.739 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.739 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:201 3.5 3:a1c47f99:::smithi09025264-2:head [copy-from ver 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:21.739 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:21.739 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 352.14312853600001, 2023-01-15T14:34:21.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30931411399999997, 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 201 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:21.740 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:21.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.741 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.741 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.741 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.741 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406394+0000", 2023-01-15T14:34:21.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:34:21.742 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.742 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.742 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.742 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406408+0000", 2023-01-15T14:34:21.745 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4416999999999999e-05 2023-01-15T14:34:21.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.745 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.745 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406446+0000", 2023-01-15T14:34:21.745 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7339999999999998e-05 2023-01-15T14:34:21.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.745 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406467+0000", 2023-01-15T14:34:21.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1137000000000001e-05 2023-01-15T14:34:21.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406677+0000", 2023-01-15T14:34:21.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000210275 2023-01-15T14:34:21.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406902+0000", 2023-01-15T14:34:21.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022529000000000001 2023-01-15T14:34:21.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:21.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.513012+0000", 2023-01-15T14:34:21.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.106109527 2023-01-15T14:34:21.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:21.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514754+0000", 2023-01-15T14:34:21.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0017419779999999999 2023-01-15T14:34:21.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514773+0000", 2023-01-15T14:34:21.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8700000000000001e-05 2023-01-15T14:34:21.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.515531+0000", 2023-01-15T14:34:21.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075844300000000003 2023-01-15T14:34:21.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615334+0000", 2023-01-15T14:34:21.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099803327999999997 2023-01-15T14:34:21.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615376+0000", 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1675999999999999e-05 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715546+0000", 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016992500000001 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715626+0000", 2023-01-15T14:34:21.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0112999999999997e-05 2023-01-15T14:34:21.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715711+0000", 2023-01-15T14:34:21.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:34:21.751 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.751 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.751 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.752 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:202 3.5 3:a1c47f99:::smithi09025264-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:21.753 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:21.753 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 352.14301598999998, 2023-01-15T14:34:21.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31002637300000002, 2023-01-15T14:34:21.753 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.753 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:21.753 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:21.753 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:21.753 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:21.753 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 202 2023-01-15T14:34:21.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.754 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:21.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:21.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.755 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406508+0000", 2023-01-15T14:34:21.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:21.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406516+0000", 2023-01-15T14:34:21.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.413e-06 2023-01-15T14:34:21.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406534+0000", 2023-01-15T14:34:21.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8936999999999998e-05 2023-01-15T14:34:21.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406548+0000", 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3196e-05 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407271+0000", 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00072301500000000003 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:34:21.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407466+0000", 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000195009 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.614996+0000", 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.207530192 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615227+0000", 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000230979 2023-01-15T14:34:21.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.760 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.760 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716282+0000", 2023-01-15T14:34:21.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101055465 2023-01-15T14:34:21.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.760 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.760 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716409+0000", 2023-01-15T14:34:21.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012653999999999999 2023-01-15T14:34:21.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716536+0000", 2023-01-15T14:34:21.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012683 2023-01-15T14:34:21.761 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.761 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.761 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.761 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:189 3.d e23/20 3:b3f111b5:::smithi09025264-48:head v 23'10, mlcod=23'6)", 2023-01-15T14:34:21.761 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:21.762 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 352.097811342, 2023-01-15T14:34:21.762 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20400475100000001, 2023-01-15T14:34:21.762 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.762 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:21.763 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:21.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.763 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.763 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.763 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:21.763 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.763 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451712+0000", 2023-01-15T14:34:21.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:21.764 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.764 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.764 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.764 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452272+0000", 2023-01-15T14:34:21.764 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00056035899999999997 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452310+0000", 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8145999999999997e-05 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452324+0000", 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4443999999999999e-05 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.765 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652703+0000", 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20037818199999999 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652762+0000", 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9302999999999998e-05 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653302+0000", 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054012199999999998 2023-01-15T14:34:21.766 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653333+0000", 2023-01-15T14:34:21.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.112e-05 2023-01-15T14:34:21.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.767 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.767 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.767 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653667+0000", 2023-01-15T14:34:21.767 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033337900000000002 2023-01-15T14:34:21.767 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653709+0000", 2023-01-15T14:34:21.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2469999999999998e-05 2023-01-15T14:34:21.768 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.768 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.768 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.768 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655719+0000", 2023-01-15T14:34:21.768 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020096699999999999 2023-01-15T14:34:21.768 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.768 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.769 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.769 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.769 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:269 3.2 e30/20 3:47dfe935:::smithi09025264-29:head v 30'11, mlcod=23'8)", 2023-01-15T14:34:21.769 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:21.769 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 344.67176590100001, 2023-01-15T14:34:21.769 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23185540399999999, 2023-01-15T14:34:21.769 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.769 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:21.769 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.769 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:21.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.770 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.770 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:21.770 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.770 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.770 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.770 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.771 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:21.771 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:21.771 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.771 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.771 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.771 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878379+0000", 2023-01-15T14:34:21.771 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061979600000000004 2023-01-15T14:34:21.771 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.771 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.771 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.772 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878406+0000", 2023-01-15T14:34:21.772 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:34:21.772 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.772 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.772 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.772 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978515+0000", 2023-01-15T14:34:21.772 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100108461 2023-01-15T14:34:21.772 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.772 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.772 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.773 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078759+0000", 2023-01-15T14:34:21.773 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100244496 2023-01-15T14:34:21.773 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.773 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.773 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.773 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078796+0000", 2023-01-15T14:34:21.773 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6758999999999998e-05 2023-01-15T14:34:21.773 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.773 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.773 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079567+0000", 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077137199999999999 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079593+0000", 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079844+0000", 2023-01-15T14:34:21.774 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025075899999999998 2023-01-15T14:34:21.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079910+0000", 2023-01-15T14:34:21.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6406000000000001e-05 2023-01-15T14:34:21.775 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.775 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.775 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.775 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.109615+0000", 2023-01-15T14:34:21.775 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.029704323000000001 2023-01-15T14:34:21.775 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.776 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.776 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.776 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.776 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.776 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:281 3.5 3:a75958b1:::smithi09025264-6:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:34:21.776 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:21.776 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 344.57084227199999, 2023-01-15T14:34:21.776 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30574310900000001, 2023-01-15T14:34:21.776 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.776 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:21.777 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:21.777 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:21.777 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:21.777 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 281 2023-01-15T14:34:21.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.777 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:21.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978686+0000", 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4429999999999999e-06 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978690+0000", 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0300000000000004e-06 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978693+0000", 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.179e-06 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981592+0000", 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0028984140000000002 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981718+0000", 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012616600000000001 2023-01-15T14:34:21.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:21.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981982+0000", 2023-01-15T14:34:21.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026445199999999998 2023-01-15T14:34:21.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:21.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982278+0000", 2023-01-15T14:34:21.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029514100000000001 2023-01-15T14:34:21.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982290+0000", 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.271e-05 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982343+0000", 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2914000000000002e-05 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:21.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982385+0000", 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2114999999999999e-05 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.181937+0000", 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19955213099999999 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284183+0000", 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10224538 2023-01-15T14:34:21.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284318+0000", 2023-01-15T14:34:21.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000134843 2023-01-15T14:34:21.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284426+0000", 2023-01-15T14:34:21.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108513 2023-01-15T14:34:21.907 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.907 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:373 3.4 e34/20 3:2029fa71:::smithi09025264-39:head v 34'19, mlcod=26'16)", 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 339.08985981299998, 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20212622799999999, 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:21.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:21.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459664+0000", 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460099+0000", 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043515900000000001 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460130+0000", 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1590000000000001e-05 2023-01-15T14:34:21.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560226+0000", 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100095248 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560369+0000", 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014381499999999999 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560404+0000", 2023-01-15T14:34:21.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4650000000000002e-05 2023-01-15T14:34:21.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:21.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560804+0000", 2023-01-15T14:34:21.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000399823 2023-01-15T14:34:21.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560827+0000", 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3612999999999999e-05 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.561101+0000", 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027355499999999998 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661710+0000", 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100608716 2023-01-15T14:34:21.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661792+0000", 2023-01-15T14:34:21.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2006000000000001e-05 2023-01-15T14:34:21.914 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.914 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.914 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.914 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:455 3.d e41/20 3:bd982ca3:::smithi09025264-38:head v 41'24, mlcod=38'18)", 2023-01-15T14:34:21.914 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 332.04009824000002, 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20118033699999999, 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:21.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509425+0000", 2023-01-15T14:34:21.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:21.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509441+0000", 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5248e-05 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509474+0000", 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3531000000000001e-05 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.609644+0000", 2023-01-15T14:34:21.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016994999999999 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709872+0000", 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10022840299999999 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709906+0000", 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3160000000000001e-05 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710153+0000", 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024735399999999998 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710204+0000", 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1075999999999997e-05 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710280+0000", 2023-01-15T14:34:21.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6202e-05 2023-01-15T14:34:21.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710317+0000", 2023-01-15T14:34:21.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6387000000000002e-05 2023-01-15T14:34:21.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710607+0000", 2023-01-15T14:34:21.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029079699999999997 2023-01-15T14:34:21.921 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.921 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.921 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.921 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:497 3.d e44/20 3:bd982ca3:::smithi09025264-38:head v 44'31, mlcod=42'26)", 2023-01-15T14:34:21.921 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:21.921 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 329.77530337799999, 2023-01-15T14:34:21.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20082067100000001, 2023-01-15T14:34:21.921 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:21.921 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:21.922 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:21.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:21.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:21.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.981 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.981 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.981 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:21.981 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:21.981 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:21.981 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.981 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.981 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:21.981 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774220+0000", 2023-01-15T14:34:21.981 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:21.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.982 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.982 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:21.982 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774238+0000", 2023-01-15T14:34:21.982 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8107000000000001e-05 2023-01-15T14:34:21.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.982 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.982 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:21.982 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774263+0000", 2023-01-15T14:34:21.982 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4122999999999999e-05 2023-01-15T14:34:21.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.983 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:21.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774280+0000", 2023-01-15T14:34:21.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7039999999999999e-05 2023-01-15T14:34:21.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.983 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:21.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874060+0000", 2023-01-15T14:34:21.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099780698000000001 2023-01-15T14:34:21.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.984 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.984 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:21.984 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874130+0000", 2023-01-15T14:34:21.984 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9852999999999997e-05 2023-01-15T14:34:21.984 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.984 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.984 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:21.984 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874351+0000", 2023-01-15T14:34:21.984 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022091400000000001 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874369+0000", 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7876e-05 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874433+0000", 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3793000000000004e-05 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.986 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:21.986 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.974927+0000", 2023-01-15T14:34:21.986 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10049393500000001 2023-01-15T14:34:21.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:21.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:21.986 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:21.986 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.975043+0000", 2023-01-15T14:34:21.986 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115949 2023-01-15T14:34:21.986 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.986 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.986 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.987 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:21.987 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:21.987 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:21.988 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:34:21.989 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:20.278+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:21.989 INFO:tasks.rados.rados.0.smithi090.stdout:1397: writing smithi09025264-1 from 403500 to 1190945 tid 1 2023-01-15T14:34:21.991 INFO:tasks.rados.rados.0.smithi090.stdout:1397: writing smithi09025264-1 from 1692526 to 2166363 tid 2 2023-01-15T14:34:21.992 INFO:tasks.rados.rados.0.smithi090.stdout:1398: snap_create 2023-01-15T14:34:21.992 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:34:21.992 INFO:tasks.rados.rados.0.smithi090.stdout:1395: finishing write tid 1 to smithi09025264-19 2023-01-15T14:34:21.992 INFO:tasks.rados.rados.0.smithi090.stdout:1395: finishing write tid 2 to smithi09025264-19 2023-01-15T14:34:21.992 INFO:tasks.rados.rados.0.smithi090.stdout:1395: finishing write tid 3 to smithi09025264-19 2023-01-15T14:34:21.992 INFO:tasks.rados.rados.0.smithi090.stdout:1395: finishing write tid 4 to smithi09025264-19 2023-01-15T14:34:21.992 INFO:tasks.rados.rados.0.smithi090.stdout:1395: oid 19 updating version 0 to 257 2023-01-15T14:34:21.992 INFO:tasks.rados.rados.0.smithi090.stdout:1395: oid 19 updating version 257 to 258 2023-01-15T14:34:21.993 INFO:tasks.rados.rados.0.smithi090.stdout:1395: oid 19 updating version 258 to 259 2023-01-15T14:34:21.993 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 259 (ObjNum 499 snap 149 seq_num 499) dirty exists 2023-01-15T14:34:21.993 INFO:tasks.rados.rados.0.smithi090.stdout:1395: left oid 19 (ObjNum 499 snap 149 seq_num 499) 2023-01-15T14:34:21.993 INFO:tasks.rados.rados.0.smithi090.stdout:1395: done (1 left) 2023-01-15T14:34:21.993 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:21.994 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:34:21.999 INFO:tasks.rados.rados.0.smithi090.stdout:1397: finishing write tid 1 to smithi09025264-1 2023-01-15T14:34:22.071 INFO:tasks.rados.rados.0.smithi090.stdout:1397: finishing write tid 2 to smithi09025264-1 2023-01-15T14:34:22.071 INFO:tasks.rados.rados.0.smithi090.stdout:1397: finishing write tid 3 to smithi09025264-1 2023-01-15T14:34:22.071 INFO:tasks.rados.rados.0.smithi090.stdout:1397: finishing write tid 4 to smithi09025264-1 2023-01-15T14:34:22.071 INFO:tasks.rados.rados.0.smithi090.stdout:1397: oid 1 updating version 0 to 401 2023-01-15T14:34:22.071 INFO:tasks.rados.rados.0.smithi090.stdout:1397: oid 1 version 401 is already newer than 400 2023-01-15T14:34:22.071 INFO:tasks.rados.rados.0.smithi090.stdout:1397: oid 1 updating version 401 to 402 2023-01-15T14:34:22.072 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 402 (ObjNum 500 snap 149 seq_num 500) dirty exists 2023-01-15T14:34:22.072 INFO:tasks.rados.rados.0.smithi090.stdout:1397: left oid 1 (ObjNum 500 snap 149 seq_num 500) 2023-01-15T14:34:22.072 INFO:tasks.rados.rados.0.smithi090.stdout:1397: done (0 left) 2023-01-15T14:34:22.072 INFO:tasks.rados.rados.0.smithi090.stdout:1398: done (0 left) 2023-01-15T14:34:22.072 INFO:tasks.rados.rados.0.smithi090.stdout:1399: delete oid 26 current snap is 150 2023-01-15T14:34:22.073 INFO:tasks.rados.rados.0.smithi090.stdout:1399: done (0 left) 2023-01-15T14:34:22.073 INFO:tasks.rados.rados.0.smithi090.stdout:1400: delete oid 5 current snap is 150 2023-01-15T14:34:22.073 INFO:tasks.rados.rados.0.smithi090.stdout:1400: done (0 left) 2023-01-15T14:34:22.073 INFO:tasks.rados.rados.0.smithi090.stdout:1401: rollback oid 35 current snap is 150 2023-01-15T14:34:22.073 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 35 to 148 2023-01-15T14:34:22.074 INFO:tasks.rados.rados.0.smithi090.stdout:1402: write (excl) oid 21 current snap is 150 2023-01-15T14:34:22.074 INFO:tasks.rados.rados.0.smithi090.stdout:1402: seq_num 503 ranges {453921=729545,1584325=621164} 2023-01-15T14:34:22.074 INFO:tasks.rados.rados.0.smithi090.stdout:1402: writing smithi09025264-21 from 453921 to 1183466 tid 1 2023-01-15T14:34:22.074 INFO:tasks.rados.rados.0.smithi090.stdout:1402: writing smithi09025264-21 from 1584325 to 2205489 tid 2 2023-01-15T14:34:22.074 INFO:tasks.rados.rados.0.smithi090.stdout:1403: copy_from oid 28 from oid 9 current snap is 150 2023-01-15T14:34:22.075 INFO:tasks.rados.rados.0.smithi090.stdout:1404: write oid 15 current snap is 150 2023-01-15T14:34:22.075 INFO:tasks.rados.rados.0.smithi090.stdout:1404: seq_num 504 ranges {470728=404794,1526454=481175} 2023-01-15T14:34:22.075 INFO:tasks.rados.rados.0.smithi090.stdout:1404: writing smithi09025264-15 from 470728 to 875522 tid 1 2023-01-15T14:34:22.075 INFO:tasks.rados.rados.0.smithi090.stdout:1404: writing smithi09025264-15 from 1526454 to 2007629 tid 2 2023-01-15T14:34:22.075 INFO:tasks.rados.rados.0.smithi090.stdout:1405: delete oid 44 current snap is 150 2023-01-15T14:34:22.075 INFO:tasks.rados.rados.0.smithi090.stdout:1403: finishing copy_from to smithi09025264-28 2023-01-15T14:34:22.076 INFO:tasks.rados.rados.0.smithi090.stdout:1403: got expected ENOENT (src dne) 2023-01-15T14:34:22.076 INFO:tasks.rados.rados.0.smithi090.stdout:1403: finishing copy_from racing read to smithi09025264-28 2023-01-15T14:34:22.076 INFO:tasks.rados.rados.0.smithi090.stdout:1401: finishing rollback tid 1 to smithi09025264-35 2023-01-15T14:34:22.076 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 324 (ObjNum 33 snap 0 seq_num 3193456704) dirty dne 2023-01-15T14:34:22.076 INFO:tasks.rados.rados.0.smithi090.stdout:1402: finishing write tid 1 to smithi09025264-21 2023-01-15T14:34:22.077 INFO:tasks.rados.rados.0.smithi090.stdout:1404: finishing write tid 1 to smithi09025264-15 2023-01-15T14:34:22.077 INFO:tasks.rados.rados.0.smithi090.stdout:1404: finishing write tid 2 to smithi09025264-15 2023-01-15T14:34:22.077 INFO:tasks.rados.rados.0.smithi090.stdout:1402: finishing write tid 2 to smithi09025264-21 2023-01-15T14:34:22.077 INFO:tasks.rados.rados.0.smithi090.stdout:1401: done (4 left) 2023-01-15T14:34:22.077 INFO:tasks.rados.rados.0.smithi090.stdout:1403: done (3 left) 2023-01-15T14:34:22.078 INFO:tasks.rados.rados.0.smithi090.stdout:1405: done (2 left) 2023-01-15T14:34:22.078 INFO:tasks.rados.rados.0.smithi090.stdout:1406: read oid 20 snap -1 2023-01-15T14:34:22.078 INFO:tasks.rados.rados.0.smithi090.stdout:1406: expect (ObjNum 451 snap 126 seq_num 451) 2023-01-15T14:34:22.078 INFO:tasks.rados.rados.0.smithi090.stdout:1407: write (excl) oid 46 current snap is 150 2023-01-15T14:34:22.078 INFO:tasks.rados.rados.0.smithi090.stdout:1407: seq_num 506 ranges {504342=555292,1643418=484844} 2023-01-15T14:34:22.079 INFO:tasks.rados.rados.0.smithi090.stdout:1407: writing smithi09025264-46 from 504342 to 1059634 tid 1 2023-01-15T14:34:22.079 INFO:tasks.rados.rados.0.smithi090.stdout:1407: writing smithi09025264-46 from 1643418 to 2128262 tid 2 2023-01-15T14:34:22.079 INFO:tasks.rados.rados.0.smithi090.stdout:1408: read oid 33 snap -1 2023-01-15T14:34:22.079 INFO:tasks.rados.rados.0.smithi090.stdout:1408: expect (ObjNum 378 snap 102 seq_num 378) 2023-01-15T14:34:22.079 INFO:tasks.rados.rados.0.smithi090.stdout:1409: write (excl) oid 36 current snap is 150 2023-01-15T14:34:22.080 INFO:tasks.rados.rados.0.smithi090.stdout:1404: finishing write tid 3 to smithi09025264-15 2023-01-15T14:34:22.080 INFO:tasks.rados.rados.0.smithi090.stdout:1409: seq_num 507 ranges {521149=630541,1701900=628502} 2023-01-15T14:34:22.080 INFO:tasks.rados.rados.0.smithi090.stdout:1409: writing smithi09025264-36 from 521149 to 1151690 tid 1 2023-01-15T14:34:22.080 INFO:tasks.rados.rados.0.smithi090.stdout:1409: writing smithi09025264-36 from 1701900 to 2330402 tid 2 2023-01-15T14:34:22.080 INFO:tasks.rados.rados.0.smithi090.stdout:1410: rollback oid 27 current snap is 150 2023-01-15T14:34:22.080 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 27 to 136 2023-01-15T14:34:22.081 INFO:tasks.rados.rados.0.smithi090.stdout:1402: finishing write tid 3 to smithi09025264-21 2023-01-15T14:34:22.081 INFO:tasks.rados.rados.0.smithi090.stdout:1402: finishing write tid 4 to smithi09025264-21 2023-01-15T14:34:22.081 INFO:tasks.rados.rados.0.smithi090.stdout:1402: oid 21 updating version 0 to 365 2023-01-15T14:34:22.081 INFO:tasks.rados.rados.0.smithi090.stdout:1402: oid 21 version 365 is already newer than 364 2023-01-15T14:34:22.081 INFO:tasks.rados.rados.0.smithi090.stdout:1402: oid 21 version 365 is already newer than 363 2023-01-15T14:34:22.082 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 365 (ObjNum 503 snap 150 seq_num 503) dirty exists 2023-01-15T14:34:22.082 INFO:tasks.rados.rados.0.smithi090.stdout:1402: left oid 21 (ObjNum 503 snap 150 seq_num 503) 2023-01-15T14:34:22.082 INFO:tasks.rados.rados.0.smithi090.stdout:1402: done (6 left) 2023-01-15T14:34:22.082 INFO:tasks.rados.rados.0.smithi090.stdout:1411: write (excl) oid 16 current snap is 150 2023-01-15T14:34:22.082 INFO:tasks.rados.rados.0.smithi090.stdout:1411: seq_num 508 ranges {537956=705790,1644029=772160} 2023-01-15T14:34:22.082 INFO:tasks.rados.rados.0.smithi090.stdout:1411: writing smithi09025264-16 from 537956 to 1243746 tid 1 2023-01-15T14:34:22.083 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:21.987+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:22.084 INFO:tasks.rados.rados.0.smithi090.stdout:1411: writing smithi09025264-16 from 1644029 to 2416189 tid 2 2023-01-15T14:34:22.084 INFO:tasks.rados.rados.0.smithi090.stdout:1412: read oid 47 snap -1 2023-01-15T14:34:22.085 INFO:tasks.rados.rados.0.smithi090.stdout:1412: expect (ObjNum 477 snap 140 seq_num 477) 2023-01-15T14:34:22.085 INFO:tasks.rados.rados.0.smithi090.stdout:1413: snap_create 2023-01-15T14:34:22.085 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 8 2023-01-15T14:34:22.087 INFO:tasks.rados.rados.0.smithi090.stdout:1406: expect (ObjNum 451 snap 126 seq_num 451) 2023-01-15T14:34:22.108 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:22.108 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:22.109 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:34:22.109 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:22.119 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:34:22.135 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:22.131+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:22.196 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:22.191+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:22.237 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:22.244 INFO:tasks.rados.rados.0.smithi090.stdout:1404: finishing write tid 4 to smithi09025264-15 2023-01-15T14:34:22.244 INFO:tasks.rados.rados.0.smithi090.stdout:1404: oid 15 updating version 0 to 158 2023-01-15T14:34:22.244 INFO:tasks.rados.rados.0.smithi090.stdout:1404: oid 15 version 158 is already newer than 156 2023-01-15T14:34:22.244 INFO:tasks.rados.rados.0.smithi090.stdout:1404: oid 15 version 158 is already newer than 157 2023-01-15T14:34:22.244 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 158 (ObjNum 504 snap 150 seq_num 504) dirty exists 2023-01-15T14:34:22.244 INFO:tasks.rados.rados.0.smithi090.stdout:1404: left oid 15 (ObjNum 504 snap 150 seq_num 504) 2023-01-15T14:34:22.244 INFO:tasks.rados.rados.0.smithi090.stdout:1407: finishing write tid 1 to smithi09025264-46 2023-01-15T14:34:22.244 INFO:tasks.rados.rados.0.smithi090.stdout:1404: done (7 left) 2023-01-15T14:34:22.244 INFO:tasks.rados.rados.0.smithi090.stdout:1406: done (6 left) 2023-01-15T14:34:22.245 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:34:22.266 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:22.268 INFO:tasks.rados.rados.0.smithi090.stdout:1409: finishing write tid 1 to smithi09025264-36 2023-01-15T14:34:22.272 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:22.272 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:22.272 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:34:22.272 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:34:22.273 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:22.282 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:34:22.288 INFO:tasks.rados.rados.0.smithi090.stdout:1411: finishing write tid 1 to smithi09025264-16 2023-01-15T14:34:22.291 INFO:tasks.rados.rados.0.smithi090.stdout:1408: expect (ObjNum 378 snap 102 seq_num 378) 2023-01-15T14:34:22.295 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:22.296 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:22.291+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:22.313 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:22.334 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:22.353 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:22.372 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:22.378 INFO:tasks.rados.rados.0.smithi090.stdout:1412: expect (ObjNum 477 snap 140 seq_num 477) 2023-01-15T14:34:22.385 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:22.385 INFO:teuthology.orchestra.run.smithi090.stdout:{"fsid":"1c77fb2b-d1a7-4884-a24e-3935e16d0b6b","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/504 objects degraded (0.397%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":384,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":372,"num_osds":8,"num_up_osds":8,"osd_up_since":1673792884,"num_in_osds":6,"osd_in_since":1673793049,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":111},{"state_name":"active+clean+snaptrim","count":1},{"state_name":"active+recovering+undersized+degraded+remapped","count":1},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":115,"num_pools":3,"num_objects":168,"data_bytes":360016807,"bytes_used":6485868544,"bytes_avail":569203122176,"bytes_total":575688990720,"degraded_objects":2,"degraded_total":504,"degraded_ratio":0.003968253968253968,"misplaced_objects":2,"misplaced_total":504,"misplaced_ratio":0.003968253968253968,"recovering_objects_per_sec":11,"recovering_bytes_per_sec":9771877,"recovering_keys_per_sec":0,"num_objects_recovered":47,"num_bytes_recovered":39103389,"num_keys_recovered":0,"write_bytes_sec":588049,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2023-01-15T14:29:06.857309+0000","services":{}},"progress_events":{}} 2023-01-15T14:34:22.388 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:22.388 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:22.392 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:22.394 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:34:22.394 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:34:22.394 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:34:22.394 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:34:22.395 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:34:22.395 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:34:22.395 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:34:22.395 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:34:22.397 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:22.394+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:22.398 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:34:22.405 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2023-01-15T14:34:22.406 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:34:22.447 INFO:tasks.rados.rados.0.smithi090.stdout:1407: finishing write tid 2 to smithi09025264-46 2023-01-15T14:34:22.447 INFO:tasks.rados.rados.0.smithi090.stdout:1407: finishing write tid 3 to smithi09025264-46 2023-01-15T14:34:22.447 INFO:tasks.rados.rados.0.smithi090.stdout:1407: finishing write tid 4 to smithi09025264-46 2023-01-15T14:34:22.448 INFO:tasks.rados.rados.0.smithi090.stdout:1407: oid 46 updating version 0 to 319 2023-01-15T14:34:22.448 INFO:tasks.rados.rados.0.smithi090.stdout:1407: oid 46 updating version 319 to 320 2023-01-15T14:34:22.448 INFO:tasks.rados.rados.0.smithi090.stdout:1407: oid 46 version 320 is already newer than 318 2023-01-15T14:34:22.448 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 320 (ObjNum 506 snap 150 seq_num 506) dirty exists 2023-01-15T14:34:22.448 INFO:tasks.rados.rados.0.smithi090.stdout:1407: left oid 46 (ObjNum 506 snap 150 seq_num 506) 2023-01-15T14:34:22.448 INFO:tasks.rados.rados.0.smithi090.stdout:1410: finishing rollback tid 0 to smithi09025264-27 2023-01-15T14:34:22.448 INFO:tasks.rados.rados.0.smithi090.stdout:1410: finishing rollback tid 1 to smithi09025264-27 2023-01-15T14:34:22.449 INFO:tasks.rados.rados.0.smithi090.stdout:1407: done (5 left) 2023-01-15T14:34:22.449 INFO:tasks.rados.rados.0.smithi090.stdout:1408: done (4 left) 2023-01-15T14:34:22.449 INFO:tasks.rados.rados.0.smithi090.stdout:1412: done (3 left) 2023-01-15T14:34:22.449 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:34:22.486 INFO:tasks.rados.rados.0.smithi090.stdout:1410: finishing rollback tid 2 to smithi09025264-27 2023-01-15T14:34:22.538 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 269 (ObjNum 400 snap 106 seq_num 400) dirty exists 2023-01-15T14:34:22.538 INFO:tasks.rados.rados.0.smithi090.stdout:1410: done (2 left) 2023-01-15T14:34:22.539 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:34:22.541 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:22.541 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:22.542 INFO:tasks.rados.rados.0.smithi090.stdout:1411: finishing write tid 2 to smithi09025264-16 2023-01-15T14:34:22.542 INFO:tasks.rados.rados.0.smithi090.stdout:1411: finishing write tid 3 to smithi09025264-16 2023-01-15T14:34:22.542 INFO:tasks.rados.rados.0.smithi090.stdout:1411: finishing write tid 4 to smithi09025264-16 2023-01-15T14:34:22.542 INFO:tasks.rados.rados.0.smithi090.stdout:1411: oid 16 updating version 0 to 190 2023-01-15T14:34:22.542 INFO:tasks.rados.rados.0.smithi090.stdout:1411: oid 16 version 190 is already newer than 189 2023-01-15T14:34:22.542 INFO:tasks.rados.rados.0.smithi090.stdout:1411: oid 16 version 190 is already newer than 188 2023-01-15T14:34:22.542 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 190 (ObjNum 508 snap 150 seq_num 508) dirty exists 2023-01-15T14:34:22.542 INFO:tasks.rados.rados.0.smithi090.stdout:1411: left oid 16 (ObjNum 508 snap 150 seq_num 508) 2023-01-15T14:34:22.542 INFO:tasks.rados.rados.0.smithi090.stdout:1411: done (1 left) 2023-01-15T14:34:22.543 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:22.543 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:34:22.543 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:22.535+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:22.571 INFO:tasks.rados.rados.0.smithi090.stdout:1409: finishing write tid 2 to smithi09025264-36 2023-01-15T14:34:22.572 INFO:tasks.rados.rados.0.smithi090.stdout:1409: finishing write tid 3 to smithi09025264-36 2023-01-15T14:34:22.640 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:22.638+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:22.652 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:22.652 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:22.661 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:34:22.671 INFO:tasks.rados.rados.0.smithi090.stdout:1409: finishing write tid 4 to smithi09025264-36 2023-01-15T14:34:22.672 INFO:tasks.rados.rados.0.smithi090.stdout:1409: oid 36 updating version 0 to 259 2023-01-15T14:34:22.672 INFO:tasks.rados.rados.0.smithi090.stdout:1409: oid 36 version 259 is already newer than 257 2023-01-15T14:34:22.672 INFO:tasks.rados.rados.0.smithi090.stdout:1409: oid 36 version 259 is already newer than 258 2023-01-15T14:34:22.672 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 259 (ObjNum 507 snap 150 seq_num 507) dirty exists 2023-01-15T14:34:22.672 INFO:tasks.rados.rados.0.smithi090.stdout:1409: left oid 36 (ObjNum 507 snap 150 seq_num 507) 2023-01-15T14:34:22.672 INFO:tasks.rados.rados.0.smithi090.stdout:1409: done (0 left) 2023-01-15T14:34:22.705 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:22.707 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:34:22.720 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":552,"stamp":"2023-01-15T14:34:21.910181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":362403500,"num_objects":169,"num_object_clones":115,"num_object_copies":507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":169,"num_whiteouts":15,"num_read":1135,"num_read_kb":1109360,"num_write":975,"num_write_kb":337286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":732,"num_bytes_recovered":468860199,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18861,"ondisk_log_size":18861,"up":345,"acting":342,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":370,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6296048,"kb_used_data":555941756,"kb_used_omap":69298,"kb_used_meta":0,"kb_avail":555900232,"statfs":{"total":575688990720,"available":569241837568,"internally_reserved":42520576,"allocated":569284358144,"data_stored":569284358144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":70961370,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":20,"commit_latency_ns":0,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3613621,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":-7,"num_read_kb":-8495,"num_write":4,"num_write_kb":927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.922211"},"pg_stats":[{"pgid":"3.69","version":"110'149","reported_seq":915,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077255+0000","last_change":"2023-01-15T14:33:57.316235+0000","last_active":"2023-01-15T14:34:17.077255+0000","last_peered":"2023-01-15T14:34:17.077255+0000","last_clean":"2023-01-15T14:34:17.077255+0000","last_became_active":"2023-01-15T14:33:50.216417+0000","last_became_peered":"2023-01-15T14:33:50.216417+0000","last_unstale":"2023-01-15T14:34:17.077255+0000","last_undegraded":"2023-01-15T14:34:17.077255+0000","last_fullsized":"2023-01-15T14:34:17.077255+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.406510072,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.68","version":"371'257","reported_seq":1443,"reported_epoch":371,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:21.026244+0000","last_change":"2023-01-15T14:34:20.120598+0000","last_active":"2023-01-15T14:34:21.026244+0000","last_peered":"2023-01-15T14:34:21.026244+0000","last_clean":"2023-01-15T14:34:21.026244+0000","last_became_active":"2023-01-15T14:34:13.134258+0000","last_became_peered":"2023-01-15T14:34:13.134258+0000","last_unstale":"2023-01-15T14:34:21.026244+0000","last_undegraded":"2023-01-15T14:34:21.026244+0000","last_fullsized":"2023-01-15T14:34:21.026244+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.18941453,"stat_sum":{"num_bytes":8014979,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17027823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.67","version":"191'79","reported_seq":68,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070541+0000","last_change":"2023-01-15T14:33:57.312535+0000","last_active":"2023-01-15T14:34:16.070541+0000","last_peered":"2023-01-15T14:34:16.070541+0000","last_clean":"2023-01-15T14:34:16.070541+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:34:16.070541+0000","last_undegraded":"2023-01-15T14:34:16.070541+0000","last_fullsized":"2023-01-15T14:34:16.070541+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.40508863699999997,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.66","version":"102'52","reported_seq":378,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.078285+0000","last_change":"2023-01-15T14:33:57.299510+0000","last_active":"2023-01-15T14:34:17.078285+0000","last_peered":"2023-01-15T14:34:17.078285+0000","last_clean":"2023-01-15T14:34:17.078285+0000","last_became_active":"2023-01-15T14:33:48.695009+0000","last_became_peered":"2023-01-15T14:33:48.695009+0000","last_unstale":"2023-01-15T14:34:17.078285+0000","last_undegraded":"2023-01-15T14:34:17.078285+0000","last_fullsized":"2023-01-15T14:34:17.078285+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.390920724,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2142,"num_write":4,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.65","version":"367'124","reported_seq":1229,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.801494+0000","last_change":"2023-01-15T14:34:17.802215+0000","last_active":"2023-01-15T14:34:17.801494+0000","last_peered":"2023-01-15T14:34:17.801494+0000","last_clean":"2023-01-15T14:34:17.801494+0000","last_became_active":"2023-01-15T14:34:16.276292+0000","last_became_peered":"2023-01-15T14:34:16.276292+0000","last_unstale":"2023-01-15T14:34:17.801494+0000","last_undegraded":"2023-01-15T14:34:17.801494+0000","last_fullsized":"2023-01-15T14:34:17.801494+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.71301477300000005,"stat_sum":{"num_bytes":7224063,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":3033,"num_write":12,"num_write_kb":4201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":20989326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.64","version":"334'251","reported_seq":1433,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.225131+0000","last_change":"2023-01-15T14:34:20.225131+0000","last_active":"2023-01-15T14:34:20.225131+0000","last_peered":"2023-01-15T14:34:20.225131+0000","last_clean":"2023-01-15T14:34:20.225131+0000","last_became_active":"2023-01-15T14:33:55.675085+0000","last_became_peered":"2023-01-15T14:33:55.675085+0000","last_unstale":"2023-01-15T14:34:20.225131+0000","last_undegraded":"2023-01-15T14:34:20.225131+0000","last_fullsized":"2023-01-15T14:34:20.225131+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.2053372,"stat_sum":{"num_bytes":3548733,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10773,"num_write":5,"num_write_kb":2741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.63","version":"149'233","reported_seq":762,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.734855+0000","last_change":"2023-01-15T14:34:20.734855+0000","last_active":"2023-01-15T14:34:20.734855+0000","last_peered":"2023-01-15T14:34:20.734855+0000","last_clean":"2023-01-15T14:34:20.734855+0000","last_became_active":"2023-01-15T14:33:55.277739+0000","last_became_peered":"2023-01-15T14:33:55.277739+0000","last_unstale":"2023-01-15T14:34:20.734855+0000","last_undegraded":"2023-01-15T14:34:20.734855+0000","last_fullsized":"2023-01-15T14:34:20.734855+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.71404028600000002,"stat_sum":{"num_bytes":3802136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":21819,"num_write":19,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.62","version":"348'169","reported_seq":1361,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.975292+0000","last_change":"2023-01-15T14:33:58.705842+0000","last_active":"2023-01-15T14:34:15.975292+0000","last_peered":"2023-01-15T14:34:15.975292+0000","last_clean":"2023-01-15T14:34:15.975292+0000","last_became_active":"2023-01-15T14:33:56.590183+0000","last_became_peered":"2023-01-15T14:33:56.590183+0000","last_unstale":"2023-01-15T14:34:15.975292+0000","last_undegraded":"2023-01-15T14:34:15.975292+0000","last_fullsized":"2023-01-15T14:34:15.975292+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.70374445200000002,"stat_sum":{"num_bytes":5361549,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6734,"num_write":11,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5210269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.61","version":"370'140","reported_seq":1025,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.729203+0000","last_change":"2023-01-15T14:34:20.729203+0000","last_active":"2023-01-15T14:34:20.729203+0000","last_peered":"2023-01-15T14:34:20.729203+0000","last_clean":"2023-01-15T14:34:20.729203+0000","last_became_active":"2023-01-15T14:33:53.421056+0000","last_became_peered":"2023-01-15T14:33:53.421056+0000","last_unstale":"2023-01-15T14:34:20.729203+0000","last_undegraded":"2023-01-15T14:34:20.729203+0000","last_fullsized":"2023-01-15T14:34:20.729203+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.71097001800000004,"stat_sum":{"num_bytes":1033664,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6059,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.60","version":"348'281","reported_seq":1424,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.974641+0000","last_change":"2023-01-15T14:33:58.810339+0000","last_active":"2023-01-15T14:34:15.974641+0000","last_peered":"2023-01-15T14:34:15.974641+0000","last_clean":"2023-01-15T14:34:15.974641+0000","last_became_active":"2023-01-15T14:33:56.478613+0000","last_became_peered":"2023-01-15T14:33:56.478613+0000","last_unstale":"2023-01-15T14:34:15.974641+0000","last_undegraded":"2023-01-15T14:34:15.974641+0000","last_fullsized":"2023-01-15T14:34:15.974641+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.80913330699999997,"stat_sum":{"num_bytes":5214108,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":10865,"num_write":14,"num_write_kb":5082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15418818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1393,"reported_epoch":370,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:20.120281+0000","last_change":"2023-01-15T14:34:20.120227+0000","last_active":"2023-01-15T14:34:20.120281+0000","last_peered":"2023-01-15T14:34:20.120281+0000","last_clean":"2023-01-15T14:34:20.120281+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:34:20.120281+0000","last_undegraded":"2023-01-15T14:34:20.120281+0000","last_fullsized":"2023-01-15T14:34:20.120281+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:51.919","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.39161994999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5e","version":"215'39","reported_seq":994,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.979959+0000","last_change":"2023-01-15T14:33:57.316972+0000","last_active":"2023-01-15T14:34:16.979959+0000","last_peered":"2023-01-15T14:34:16.979959+0000","last_clean":"2023-01-15T14:34:16.979959+0000","last_became_active":"2023-01-15T14:33:35.364143+0000","last_became_peered":"2023-01-15T14:33:35.364143+0000","last_unstale":"2023-01-15T14:34:16.979959+0000","last_undegraded":"2023-01-15T14:34:16.979959+0000","last_fullsized":"2023-01-15T14:34:16.979959+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40693008000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1323,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978460+0000","last_change":"2023-01-15T14:34:00.343379+0000","last_active":"2023-01-15T14:34:16.978460+0000","last_peered":"2023-01-15T14:34:16.978460+0000","last_clean":"2023-01-15T14:34:16.978460+0000","last_became_active":"2023-01-15T14:34:00.242122+0000","last_became_peered":"2023-01-15T14:34:00.242122+0000","last_unstale":"2023-01-15T14:34:16.978460+0000","last_undegraded":"2023-01-15T14:34:16.978460+0000","last_fullsized":"2023-01-15T14:34:16.978460+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.099745922000000001,"stat_sum":{"num_bytes":2869928,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":23811,"num_write":13,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5c","version":"141'46","reported_seq":95,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.977998+0000","last_change":"2023-01-15T14:33:58.026285+0000","last_active":"2023-01-15T14:34:16.977998+0000","last_peered":"2023-01-15T14:34:16.977998+0000","last_clean":"2023-01-15T14:34:16.977998+0000","last_became_active":"2023-01-15T14:33:57.113032+0000","last_became_peered":"2023-01-15T14:33:57.113032+0000","last_unstale":"2023-01-15T14:34:16.977998+0000","last_undegraded":"2023-01-15T14:34:16.977998+0000","last_fullsized":"2023-01-15T14:34:16.977998+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.121922606,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5b","version":"283'80","reported_seq":925,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.169713+0000","last_change":"2023-01-15T14:33:57.113633+0000","last_active":"2023-01-15T14:34:16.169713+0000","last_peered":"2023-01-15T14:34:16.169713+0000","last_clean":"2023-01-15T14:34:16.169713+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:34:16.169713+0000","last_undegraded":"2023-01-15T14:34:16.169713+0000","last_fullsized":"2023-01-15T14:34:16.169713+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.20934807,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5a","version":"329'348","reported_seq":1507,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.730138+0000","last_change":"2023-01-15T14:34:20.730138+0000","last_active":"2023-01-15T14:34:20.730138+0000","last_peered":"2023-01-15T14:34:20.730138+0000","last_clean":"2023-01-15T14:34:20.730138+0000","last_became_active":"2023-01-15T14:34:11.050316+0000","last_became_peered":"2023-01-15T14:34:11.050316+0000","last_unstale":"2023-01-15T14:34:20.730138+0000","last_undegraded":"2023-01-15T14:34:20.730138+0000","last_fullsized":"2023-01-15T14:34:20.730138+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:05.853","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.71004301000000003,"stat_sum":{"num_bytes":5983424,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":19294,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.59","version":"294'256","reported_seq":1292,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077795+0000","last_change":"2023-01-15T14:33:57.406685+0000","last_active":"2023-01-15T14:34:17.077795+0000","last_peered":"2023-01-15T14:34:17.077795+0000","last_clean":"2023-01-15T14:34:17.077795+0000","last_became_active":"2023-01-15T14:33:55.671990+0000","last_became_peered":"2023-01-15T14:33:55.671990+0000","last_unstale":"2023-01-15T14:34:17.077795+0000","last_undegraded":"2023-01-15T14:34:17.077795+0000","last_fullsized":"2023-01-15T14:34:17.077795+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.49887967,"stat_sum":{"num_bytes":1210137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":5,"num_read_kb":3549,"num_write":5,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.58","version":"323'320","reported_seq":1494,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069047+0000","last_change":"2023-01-15T14:33:57.004059+0000","last_active":"2023-01-15T14:34:16.069047+0000","last_peered":"2023-01-15T14:34:16.069047+0000","last_clean":"2023-01-15T14:34:16.069047+0000","last_became_active":"2023-01-15T14:33:40.127669+0000","last_became_peered":"2023-01-15T14:33:40.127669+0000","last_unstale":"2023-01-15T14:34:16.069047+0000","last_undegraded":"2023-01-15T14:34:16.069047+0000","last_fullsized":"2023-01-15T14:34:16.069047+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.103642679,"stat_sum":{"num_bytes":4332079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.57","version":"351'98","reported_seq":116,"reported_epoch":370,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:20.120316+0000","last_change":"2023-01-15T14:34:20.120286+0000","last_active":"2023-01-15T14:34:20.120316+0000","last_peered":"2023-01-15T14:34:20.120316+0000","last_clean":"2023-01-15T14:34:20.120316+0000","last_became_active":"2023-01-15T14:33:36.994447+0000","last_became_peered":"2023-01-15T14:33:36.994447+0000","last_unstale":"2023-01-15T14:34:20.120316+0000","last_undegraded":"2023-01-15T14:34:20.120316+0000","last_fullsized":"2023-01-15T14:34:20.120316+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:22.009","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.969000109,"stat_sum":{"num_bytes":1512663,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":3025326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.56","version":"320'127","reported_seq":644,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070546+0000","last_change":"2023-01-15T14:34:00.541392+0000","last_active":"2023-01-15T14:34:16.070546+0000","last_peered":"2023-01-15T14:34:16.070546+0000","last_clean":"2023-01-15T14:34:16.070546+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:34:16.070546+0000","last_undegraded":"2023-01-15T14:34:16.070546+0000","last_fullsized":"2023-01-15T14:34:16.070546+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.30190621200000001,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.55","version":"110'55","reported_seq":1025,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.078394+0000","last_change":"2023-01-15T14:33:57.315559+0000","last_active":"2023-01-15T14:34:17.078394+0000","last_peered":"2023-01-15T14:34:17.078394+0000","last_clean":"2023-01-15T14:34:17.078394+0000","last_became_active":"2023-01-15T14:33:52.510429+0000","last_became_peered":"2023-01-15T14:33:52.510429+0000","last_unstale":"2023-01-15T14:34:17.078394+0000","last_undegraded":"2023-01-15T14:34:17.078394+0000","last_fullsized":"2023-01-15T14:34:17.078394+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.40629608699999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1152,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.222818+0000","last_change":"2023-01-15T14:34:20.222818+0000","last_active":"2023-01-15T14:34:20.222818+0000","last_peered":"2023-01-15T14:34:20.222818+0000","last_clean":"2023-01-15T14:34:20.222818+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:34:20.222818+0000","last_undegraded":"2023-01-15T14:34:20.222818+0000","last_fullsized":"2023-01-15T14:34:20.222818+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:33:45.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:45.583005+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.203180269,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.53","version":"367'398","reported_seq":1475,"reported_epoch":367,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:17.817983+0000","last_change":"2023-01-15T14:34:14.611119+0000","last_active":"2023-01-15T14:34:17.817983+0000","last_peered":"2023-01-15T14:34:17.817983+0000","last_clean":"2023-01-15T14:33:55.165228+0000","last_became_active":"2023-01-15T14:33:58.106461+0000","last_became_peered":"2023-01-15T14:33:58.106461+0000","last_unstale":"2023-01-15T14:34:17.817983+0000","last_undegraded":"2023-01-15T14:33:58.004141+0000","last_fullsized":"2023-01-15T14:33:58.002666+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:44.609","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.092650028999999995,"stat_sum":{"num_bytes":28784748,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":28,"num_read_kb":35632,"num_write":20,"num_write_kb":13841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":23952448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":["1","2","3"],"object_location_counts":[{"shards":"1,3","objects":9}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":744,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.169320+0000","last_change":"2023-01-15T14:33:56.903307+0000","last_active":"2023-01-15T14:34:16.169320+0000","last_peered":"2023-01-15T14:34:16.169320+0000","last_clean":"2023-01-15T14:34:16.169320+0000","last_became_active":"2023-01-15T14:33:55.472400+0000","last_became_peered":"2023-01-15T14:33:55.472400+0000","last_unstale":"2023-01-15T14:34:16.169320+0000","last_undegraded":"2023-01-15T14:34:16.169320+0000","last_fullsized":"2023-01-15T14:34:16.169320+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.001336634,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.51","version":"348'316","reported_seq":1533,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.975292+0000","last_change":"2023-01-15T14:33:57.893317+0000","last_active":"2023-01-15T14:34:15.975292+0000","last_peered":"2023-01-15T14:34:15.975292+0000","last_clean":"2023-01-15T14:34:15.975292+0000","last_became_active":"2023-01-15T14:33:26.107940+0000","last_became_peered":"2023-01-15T14:33:26.107940+0000","last_unstale":"2023-01-15T14:34:15.975292+0000","last_undegraded":"2023-01-15T14:34:15.975292+0000","last_fullsized":"2023-01-15T14:34:15.975292+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:36.458","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.79495623100000001,"stat_sum":{"num_bytes":7496813,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.50","version":"337'420","reported_seq":1596,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069335+0000","last_change":"2023-01-15T14:34:02.564430+0000","last_active":"2023-01-15T14:34:16.069335+0000","last_peered":"2023-01-15T14:34:16.069335+0000","last_clean":"2023-01-15T14:34:16.069335+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:34:16.069335+0000","last_undegraded":"2023-01-15T14:34:16.069335+0000","last_fullsized":"2023-01-15T14:34:16.069335+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.0084659680000000008,"stat_sum":{"num_bytes":9681416,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":52505,"num_write":49,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4f","version":"155'145","reported_seq":842,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978803+0000","last_change":"2023-01-15T14:33:57.353637+0000","last_active":"2023-01-15T14:34:16.978803+0000","last_peered":"2023-01-15T14:34:16.978803+0000","last_clean":"2023-01-15T14:34:16.978803+0000","last_became_active":"2023-01-15T14:33:25.129936+0000","last_became_peered":"2023-01-15T14:33:25.129936+0000","last_unstale":"2023-01-15T14:34:16.978803+0000","last_undegraded":"2023-01-15T14:34:16.978803+0000","last_fullsized":"2023-01-15T14:34:16.978803+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44322561900000002,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":117,"reported_epoch":370,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:20.119558+0000","last_change":"2023-01-15T14:34:20.119499+0000","last_active":"2023-01-15T14:34:20.119558+0000","last_peered":"2023-01-15T14:34:20.119558+0000","last_clean":"2023-01-15T14:34:20.119558+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:34:20.119558+0000","last_undegraded":"2023-01-15T14:34:20.119558+0000","last_fullsized":"2023-01-15T14:34:20.119558+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.0080109380000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4d","version":"356'262","reported_seq":1375,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.170264+0000","last_change":"2023-01-15T14:34:06.586527+0000","last_active":"2023-01-15T14:34:16.170264+0000","last_peered":"2023-01-15T14:34:16.170264+0000","last_clean":"2023-01-15T14:34:16.170264+0000","last_became_active":"2023-01-15T14:34:05.178613+0000","last_became_peered":"2023-01-15T14:34:05.178613+0000","last_unstale":"2023-01-15T14:34:16.170264+0000","last_undegraded":"2023-01-15T14:34:16.170264+0000","last_fullsized":"2023-01-15T14:34:16.170264+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":262,"log_dups_size":0,"ondisk_log_size":262,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":1.4066433030000001,"stat_sum":{"num_bytes":12124166,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":15961,"num_write":18,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4c","version":"309'71","reported_seq":121,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069680+0000","last_change":"2023-01-15T14:33:57.413785+0000","last_active":"2023-01-15T14:34:16.069680+0000","last_peered":"2023-01-15T14:34:16.069680+0000","last_clean":"2023-01-15T14:34:16.069680+0000","last_became_active":"2023-01-15T14:33:52.311859+0000","last_became_peered":"2023-01-15T14:33:52.311859+0000","last_unstale":"2023-01-15T14:34:16.069680+0000","last_undegraded":"2023-01-15T14:34:16.069680+0000","last_fullsized":"2023-01-15T14:34:16.069680+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.40874315500000002,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":767,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.975593+0000","last_change":"2023-01-15T14:33:57.627070+0000","last_active":"2023-01-15T14:34:15.975593+0000","last_peered":"2023-01-15T14:34:15.975593+0000","last_clean":"2023-01-15T14:34:15.975593+0000","last_became_active":"2023-01-15T14:33:52.406605+0000","last_became_peered":"2023-01-15T14:33:52.406605+0000","last_unstale":"2023-01-15T14:34:15.975593+0000","last_undegraded":"2023-01-15T14:34:15.975593+0000","last_fullsized":"2023-01-15T14:34:15.975593+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:42.594","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.515074951,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4a","version":"348'371","reported_seq":1492,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069382+0000","last_change":"2023-01-15T14:33:58.018192+0000","last_active":"2023-01-15T14:34:16.069382+0000","last_peered":"2023-01-15T14:34:16.069382+0000","last_clean":"2023-01-15T14:34:16.069382+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:34:16.069382+0000","last_undegraded":"2023-01-15T14:34:16.069382+0000","last_fullsized":"2023-01-15T14:34:16.069382+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:33.328","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10440110800000001,"stat_sum":{"num_bytes":12470870,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":63,"num_read_kb":67536,"num_write":73,"num_write_kb":23626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":949,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.043422+0000","last_change":"2023-01-15T14:34:00.046526+0000","last_active":"2023-01-15T14:34:16.043422+0000","last_peered":"2023-01-15T14:34:16.043422+0000","last_clean":"2023-01-15T14:34:16.043422+0000","last_became_active":"2023-01-15T14:34:00.043221+0000","last_became_peered":"2023-01-15T14:34:00.043221+0000","last_unstale":"2023-01-15T14:34:16.043422+0000","last_undegraded":"2023-01-15T14:34:16.043422+0000","last_fullsized":"2023-01-15T14:34:16.043422+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.00195963,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.48","version":"179'204","reported_seq":1373,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.976967+0000","last_change":"2023-01-15T14:33:57.356590+0000","last_active":"2023-01-15T14:34:16.976967+0000","last_peered":"2023-01-15T14:34:16.976967+0000","last_clean":"2023-01-15T14:34:16.976967+0000","last_became_active":"2023-01-15T14:33:55.375518+0000","last_became_peered":"2023-01-15T14:33:55.375518+0000","last_unstale":"2023-01-15T14:34:16.976967+0000","last_undegraded":"2023-01-15T14:34:16.976967+0000","last_fullsized":"2023-01-15T14:34:16.976967+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:43.662","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.34933974800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1136,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978419+0000","last_change":"2023-01-15T14:34:14.145798+0000","last_active":"2023-01-15T14:34:16.978419+0000","last_peered":"2023-01-15T14:34:16.978419+0000","last_clean":"2023-01-15T14:34:16.978419+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:34:16.978419+0000","last_undegraded":"2023-01-15T14:34:16.978419+0000","last_fullsized":"2023-01-15T14:34:16.978419+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"116'73","last_scrub_stamp":"2023-01-15T14:30:11.958290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.958290+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:19.145","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.39166528900000003,"stat_sum":{"num_bytes":496762,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":5,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":1043,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.176717+0000","last_change":"2023-01-15T14:33:57.829426+0000","last_active":"2023-01-15T14:34:16.176717+0000","last_peered":"2023-01-15T14:34:16.176717+0000","last_clean":"2023-01-15T14:34:16.176717+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:34:16.176717+0000","last_undegraded":"2023-01-15T14:34:16.176717+0000","last_fullsized":"2023-01-15T14:34:16.176717+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:55.245","scrub_duration":0.30448409700000001,"objects_trimmed":0,"snaptrim_duration":0.714760332,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":937,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.028367+0000","last_change":"2023-01-15T14:34:20.028367+0000","last_active":"2023-01-15T14:34:20.028367+0000","last_peered":"2023-01-15T14:34:20.028367+0000","last_clean":"2023-01-15T14:34:20.028367+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:34:20.028367+0000","last_undegraded":"2023-01-15T14:34:20.028367+0000","last_fullsized":"2023-01-15T14:34:20.028367+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0089002609999999996,"stat_sum":{"num_bytes":2635976,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":19927,"num_write":15,"num_write_kb":6659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4745860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1288,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.979369+0000","last_change":"2023-01-15T14:33:57.298701+0000","last_active":"2023-01-15T14:34:16.979369+0000","last_peered":"2023-01-15T14:34:16.979369+0000","last_clean":"2023-01-15T14:34:16.979369+0000","last_became_active":"2023-01-15T14:33:55.678019+0000","last_became_peered":"2023-01-15T14:33:55.678019+0000","last_unstale":"2023-01-15T14:34:16.979369+0000","last_undegraded":"2023-01-15T14:34:16.979369+0000","last_fullsized":"2023-01-15T14:34:16.979369+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:27.120","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.390225449,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.43","version":"371'320","reported_seq":1540,"reported_epoch":371,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:21.329924+0000","last_change":"2023-01-15T14:34:21.222090+0000","last_active":"2023-01-15T14:34:21.329924+0000","last_peered":"2023-01-15T14:34:21.329924+0000","last_clean":"2023-01-15T14:34:21.329924+0000","last_became_active":"2023-01-15T14:34:02.211232+0000","last_became_peered":"2023-01-15T14:34:02.211232+0000","last_unstale":"2023-01-15T14:34:21.329924+0000","last_undegraded":"2023-01-15T14:34:21.329924+0000","last_fullsized":"2023-01-15T14:34:21.329924+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.0028674299999999998,"stat_sum":{"num_bytes":6319564,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":12,"num_read_kb":16585,"num_write":7,"num_write_kb":4207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3395080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.42","version":"155'101","reported_seq":1049,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077958+0000","last_change":"2023-01-15T14:33:57.621398+0000","last_active":"2023-01-15T14:34:17.077958+0000","last_peered":"2023-01-15T14:34:17.077958+0000","last_clean":"2023-01-15T14:34:17.077958+0000","last_became_active":"2023-01-15T14:33:55.376782+0000","last_became_peered":"2023-01-15T14:33:55.376782+0000","last_unstale":"2023-01-15T14:34:17.077958+0000","last_undegraded":"2023-01-15T14:34:17.077958+0000","last_fullsized":"2023-01-15T14:34:17.077958+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.51101871399999999,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1129,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077913+0000","last_change":"2023-01-15T14:33:57.302504+0000","last_active":"2023-01-15T14:34:17.077913+0000","last_peered":"2023-01-15T14:34:17.077913+0000","last_clean":"2023-01-15T14:34:17.077913+0000","last_became_active":"2023-01-15T14:33:55.771969+0000","last_became_peered":"2023-01-15T14:33:55.771969+0000","last_unstale":"2023-01-15T14:34:17.077913+0000","last_undegraded":"2023-01-15T14:34:17.077913+0000","last_fullsized":"2023-01-15T14:34:17.077913+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.29405439700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.40","version":"363'311","reported_seq":1467,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070021+0000","last_change":"2023-01-15T14:34:14.041682+0000","last_active":"2023-01-15T14:34:16.070021+0000","last_peered":"2023-01-15T14:34:16.070021+0000","last_clean":"2023-01-15T14:34:16.070021+0000","last_became_active":"2023-01-15T14:33:58.117085+0000","last_became_peered":"2023-01-15T14:33:58.117085+0000","last_unstale":"2023-01-15T14:34:16.070021+0000","last_undegraded":"2023-01-15T14:34:16.070021+0000","last_fullsized":"2023-01-15T14:34:16.070021+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":1.0057765489999999,"stat_sum":{"num_bytes":12753904,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":35541,"num_write":28,"num_write_kb":11001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":29167491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3f","version":"321'233","reported_seq":1384,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070697+0000","last_change":"2023-01-15T14:33:58.333573+0000","last_active":"2023-01-15T14:34:16.070697+0000","last_peered":"2023-01-15T14:34:16.070697+0000","last_clean":"2023-01-15T14:34:16.070697+0000","last_became_active":"2023-01-15T14:33:58.217783+0000","last_became_peered":"2023-01-15T14:33:58.217783+0000","last_unstale":"2023-01-15T14:34:16.070697+0000","last_undegraded":"2023-01-15T14:34:16.070697+0000","last_fullsized":"2023-01-15T14:34:16.070697+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:28.180","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.014066018,"stat_sum":{"num_bytes":4764997,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6605283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3e","version":"294'63","reported_seq":849,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.975662+0000","last_change":"2023-01-15T14:33:57.728361+0000","last_active":"2023-01-15T14:34:15.975662+0000","last_peered":"2023-01-15T14:34:15.975662+0000","last_clean":"2023-01-15T14:34:15.975662+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:34:15.975662+0000","last_undegraded":"2023-01-15T14:34:15.975662+0000","last_fullsized":"2023-01-15T14:34:15.975662+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.61381065800000001,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3d","version":"371'238","reported_seq":1351,"reported_epoch":371,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:21.339374+0000","last_change":"2023-01-15T14:34:21.339374+0000","last_active":"2023-01-15T14:34:21.339374+0000","last_peered":"2023-01-15T14:34:21.339374+0000","last_clean":"2023-01-15T14:34:21.339374+0000","last_became_active":"2023-01-15T14:34:05.187121+0000","last_became_peered":"2023-01-15T14:34:05.187121+0000","last_unstale":"2023-01-15T14:34:21.339374+0000","last_undegraded":"2023-01-15T14:34:21.339374+0000","last_fullsized":"2023-01-15T14:34:21.339374+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":1.2126637869999999,"stat_sum":{"num_bytes":8050279,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":13797,"num_write":16,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":8050279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3c","version":"141'46","reported_seq":365,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978107+0000","last_change":"2023-01-15T14:33:56.912541+0000","last_active":"2023-01-15T14:34:16.978107+0000","last_peered":"2023-01-15T14:34:16.978107+0000","last_clean":"2023-01-15T14:34:16.978107+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:34:16.978107+0000","last_undegraded":"2023-01-15T14:34:16.978107+0000","last_fullsized":"2023-01-15T14:34:16.978107+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.470","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.0047236029999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":39,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.169621+0000","last_change":"2023-01-15T14:33:58.125276+0000","last_active":"2023-01-15T14:34:16.169621+0000","last_peered":"2023-01-15T14:34:16.169621+0000","last_clean":"2023-01-15T14:34:16.169621+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:34:16.169621+0000","last_undegraded":"2023-01-15T14:34:16.169621+0000","last_fullsized":"2023-01-15T14:34:16.169621+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.21169681200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1368,"reported_epoch":370,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:20.119860+0000","last_change":"2023-01-15T14:34:20.119810+0000","last_active":"2023-01-15T14:34:20.119860+0000","last_peered":"2023-01-15T14:34:20.119860+0000","last_clean":"2023-01-15T14:34:20.119860+0000","last_became_active":"2023-01-15T14:33:55.474944+0000","last_became_peered":"2023-01-15T14:33:55.474944+0000","last_unstale":"2023-01-15T14:34:20.119860+0000","last_undegraded":"2023-01-15T14:34:20.119860+0000","last_fullsized":"2023-01-15T14:34:20.119860+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.37705996200000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.39","version":"358'355","reported_seq":1504,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.023581+0000","last_change":"2023-01-15T14:34:08.632379+0000","last_active":"2023-01-15T14:34:16.023581+0000","last_peered":"2023-01-15T14:34:16.023581+0000","last_clean":"2023-01-15T14:34:16.023581+0000","last_became_active":"2023-01-15T14:34:08.023332+0000","last_became_peered":"2023-01-15T14:34:08.023332+0000","last_unstale":"2023-01-15T14:34:16.023581+0000","last_undegraded":"2023-01-15T14:34:16.023581+0000","last_fullsized":"2023-01-15T14:34:16.023581+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:28.106","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.60799269700000003,"stat_sum":{"num_bytes":24023444,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":66941,"num_write":7,"num_write_kb":7292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":26862613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.38","version":"352'315","reported_seq":1549,"reported_epoch":368,"state":"active+clean","last_fresh":"2023-01-15T14:34:18.084841+0000","last_change":"2023-01-15T14:33:58.009343+0000","last_active":"2023-01-15T14:34:18.084841+0000","last_peered":"2023-01-15T14:34:18.084841+0000","last_clean":"2023-01-15T14:34:18.084841+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:34:18.084841+0000","last_undegraded":"2023-01-15T14:34:18.084841+0000","last_fullsized":"2023-01-15T14:34:18.084841+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.10683759900000001,"stat_sum":{"num_bytes":8461433,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":42848,"num_write":39,"num_write_kb":13455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":425,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070951+0000","last_change":"2023-01-15T14:33:58.124010+0000","last_active":"2023-01-15T14:34:16.070951+0000","last_peered":"2023-01-15T14:34:16.070951+0000","last_clean":"2023-01-15T14:34:16.070951+0000","last_became_active":"2023-01-15T14:33:52.412128+0000","last_became_peered":"2023-01-15T14:33:52.412128+0000","last_unstale":"2023-01-15T14:34:16.070951+0000","last_undegraded":"2023-01-15T14:34:16.070951+0000","last_fullsized":"2023-01-15T14:34:16.070951+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.207923046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":577,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.975829+0000","last_change":"2023-01-15T14:33:57.332104+0000","last_active":"2023-01-15T14:34:15.975829+0000","last_peered":"2023-01-15T14:34:15.975829+0000","last_clean":"2023-01-15T14:34:15.975829+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:34:15.975829+0000","last_undegraded":"2023-01-15T14:34:15.975829+0000","last_fullsized":"2023-01-15T14:34:15.975829+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.31780763299999998,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":60,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.026757+0000","last_change":"2023-01-15T14:34:20.026757+0000","last_active":"2023-01-15T14:34:20.026757+0000","last_peered":"2023-01-15T14:34:20.026757+0000","last_clean":"2023-01-15T14:34:20.026757+0000","last_became_active":"2023-01-15T14:33:52.612576+0000","last_became_peered":"2023-01-15T14:33:52.612576+0000","last_unstale":"2023-01-15T14:34:20.026757+0000","last_undegraded":"2023-01-15T14:34:20.026757+0000","last_fullsized":"2023-01-15T14:34:20.026757+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075332710000000002,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.34","version":"169'138","reported_seq":1173,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.979539+0000","last_change":"2023-01-15T14:33:57.301685+0000","last_active":"2023-01-15T14:34:16.979539+0000","last_peered":"2023-01-15T14:34:16.979539+0000","last_clean":"2023-01-15T14:34:16.979539+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:34:16.979539+0000","last_undegraded":"2023-01-15T14:34:16.979539+0000","last_fullsized":"2023-01-15T14:34:16.979539+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.39300976100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1248,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.977908+0000","last_change":"2023-01-15T14:33:57.354727+0000","last_active":"2023-01-15T14:34:16.977908+0000","last_peered":"2023-01-15T14:34:16.977908+0000","last_clean":"2023-01-15T14:34:16.977908+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:34:16.977908+0000","last_undegraded":"2023-01-15T14:34:16.977908+0000","last_fullsized":"2023-01-15T14:34:16.977908+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.442392491,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.32","version":"370'167","reported_seq":870,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.220637+0000","last_change":"2023-01-15T14:34:21.221888+0000","last_active":"2023-01-15T14:34:21.220637+0000","last_peered":"2023-01-15T14:34:21.220637+0000","last_clean":"2023-01-15T14:34:21.220637+0000","last_became_active":"2023-01-15T14:34:04.171493+0000","last_became_peered":"2023-01-15T14:34:04.171493+0000","last_unstale":"2023-01-15T14:34:21.220637+0000","last_undegraded":"2023-01-15T14:34:21.220637+0000","last_fullsized":"2023-01-15T14:34:21.220637+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.19558203800000001,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7048,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5327885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.13","version":"348'374","reported_seq":1398,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.270042+0000","last_change":"2023-01-15T14:33:58.021945+0000","last_active":"2023-01-15T14:34:16.270042+0000","last_peered":"2023-01-15T14:34:16.270042+0000","last_clean":"2023-01-15T14:34:16.270042+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:34:16.270042+0000","last_undegraded":"2023-01-15T14:34:16.270042+0000","last_fullsized":"2023-01-15T14:34:16.270042+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:29.359","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.0064123280000000001,"stat_sum":{"num_bytes":8364486,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":37,"num_read_kb":61489,"num_write":17,"num_write_kb":9266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.12","version":"353'186","reported_seq":923,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.027874+0000","last_change":"2023-01-15T14:34:20.027874+0000","last_active":"2023-01-15T14:34:20.027874+0000","last_peered":"2023-01-15T14:34:20.027874+0000","last_clean":"2023-01-15T14:34:20.027874+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:34:20.027874+0000","last_undegraded":"2023-01-15T14:34:20.027874+0000","last_fullsized":"2023-01-15T14:34:20.027874+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.0085978389999999995,"stat_sum":{"num_bytes":7581497,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":15,"num_write_kb":5299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.11","version":"359'308","reported_seq":1584,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069400+0000","last_change":"2023-01-15T14:34:10.264293+0000","last_active":"2023-01-15T14:34:16.069400+0000","last_peered":"2023-01-15T14:34:16.069400+0000","last_clean":"2023-01-15T14:34:16.069400+0000","last_became_active":"2023-01-15T14:34:09.541429+0000","last_became_peered":"2023-01-15T14:34:09.541429+0000","last_unstale":"2023-01-15T14:34:16.069400+0000","last_undegraded":"2023-01-15T14:34:16.069400+0000","last_fullsized":"2023-01-15T14:34:16.069400+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.20050804999999999,"stat_sum":{"num_bytes":10718732,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":48289,"num_write":26,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":20307357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.10","version":"308'406","reported_seq":1423,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.732144+0000","last_change":"2023-01-15T14:34:20.732144+0000","last_active":"2023-01-15T14:34:20.732144+0000","last_peered":"2023-01-15T14:34:20.732144+0000","last_clean":"2023-01-15T14:34:20.732144+0000","last_became_active":"2023-01-15T14:34:16.272883+0000","last_became_peered":"2023-01-15T14:34:16.272883+0000","last_unstale":"2023-01-15T14:34:20.732144+0000","last_undegraded":"2023-01-15T14:34:20.732144+0000","last_fullsized":"2023-01-15T14:34:20.732144+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.71194310800000005,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":41964,"num_write":46,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12804882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.f","version":"155'145","reported_seq":1250,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070343+0000","last_change":"2023-01-15T14:34:00.245548+0000","last_active":"2023-01-15T14:34:16.070343+0000","last_peered":"2023-01-15T14:34:16.070343+0000","last_clean":"2023-01-15T14:34:16.070343+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:34:16.070343+0000","last_undegraded":"2023-01-15T14:34:16.070343+0000","last_fullsized":"2023-01-15T14:34:16.070343+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.0023104509999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.e","version":"139'38","reported_seq":39,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070296+0000","last_change":"2023-01-15T14:33:58.422638+0000","last_active":"2023-01-15T14:34:16.070296+0000","last_peered":"2023-01-15T14:34:16.070296+0000","last_clean":"2023-01-15T14:34:16.070296+0000","last_became_active":"2023-01-15T14:33:57.310759+0000","last_became_peered":"2023-01-15T14:33:57.310759+0000","last_unstale":"2023-01-15T14:34:16.070296+0000","last_undegraded":"2023-01-15T14:34:16.070296+0000","last_fullsized":"2023-01-15T14:34:16.070296+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.40586427000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.d","version":"296'232","reported_seq":1313,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.170440+0000","last_change":"2023-01-15T14:33:57.114569+0000","last_active":"2023-01-15T14:34:16.170440+0000","last_peered":"2023-01-15T14:34:16.170440+0000","last_clean":"2023-01-15T14:34:16.170440+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:34:16.170440+0000","last_undegraded":"2023-01-15T14:34:16.170440+0000","last_fullsized":"2023-01-15T14:34:16.170440+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.20767458899999999,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.c","version":"348'95","reported_seq":1110,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978331+0000","last_change":"2023-01-15T14:33:58.326911+0000","last_active":"2023-01-15T14:34:16.978331+0000","last_peered":"2023-01-15T14:34:16.978331+0000","last_clean":"2023-01-15T14:34:16.978331+0000","last_became_active":"2023-01-15T14:33:52.210127+0000","last_became_peered":"2023-01-15T14:33:52.210127+0000","last_unstale":"2023-01-15T14:34:16.978331+0000","last_undegraded":"2023-01-15T14:34:16.978331+0000","last_fullsized":"2023-01-15T14:34:16.978331+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.421830761,"stat_sum":{"num_bytes":2221858,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":10493,"num_write":6,"num_write_kb":686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8023674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":1019,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978294+0000","last_change":"2023-01-15T14:33:57.618325+0000","last_active":"2023-01-15T14:34:16.978294+0000","last_peered":"2023-01-15T14:34:16.978294+0000","last_clean":"2023-01-15T14:34:16.978294+0000","last_became_active":"2023-01-15T14:33:52.425622+0000","last_became_peered":"2023-01-15T14:33:52.425622+0000","last_unstale":"2023-01-15T14:34:16.978294+0000","last_undegraded":"2023-01-15T14:34:16.978294+0000","last_fullsized":"2023-01-15T14:34:16.978294+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.60540339200000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1342,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.078306+0000","last_change":"2023-01-15T14:33:57.297022+0000","last_active":"2023-01-15T14:34:17.078306+0000","last_peered":"2023-01-15T14:34:17.078306+0000","last_clean":"2023-01-15T14:34:17.078306+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:34:17.078306+0000","last_undegraded":"2023-01-15T14:34:17.078306+0000","last_fullsized":"2023-01-15T14:34:17.078306+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:48.664","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.38900274200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1127,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.178406+0000","last_change":"2023-01-15T14:33:57.304492+0000","last_active":"2023-01-15T14:34:17.178406+0000","last_peered":"2023-01-15T14:34:17.178406+0000","last_clean":"2023-01-15T14:34:17.178406+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:34:17.178406+0000","last_undegraded":"2023-01-15T14:34:17.178406+0000","last_fullsized":"2023-01-15T14:34:17.178406+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:38.874","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.29516895799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1347,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077954+0000","last_change":"2023-01-15T14:34:00.443594+0000","last_active":"2023-01-15T14:34:17.077954+0000","last_peered":"2023-01-15T14:34:17.077954+0000","last_clean":"2023-01-15T14:34:17.077954+0000","last_became_active":"2023-01-15T14:34:00.140247+0000","last_became_peered":"2023-01-15T14:34:00.140247+0000","last_unstale":"2023-01-15T14:34:17.077954+0000","last_undegraded":"2023-01-15T14:34:17.077954+0000","last_fullsized":"2023-01-15T14:34:17.077954+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.202704782,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"1.0","version":"19'117","reported_seq":535,"reported_epoch":367,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:34:17.810327+0000","last_change":"2023-01-15T14:34:17.809839+0000","last_active":"2023-01-15T14:34:17.810327+0000","last_peered":"2023-01-15T14:34:17.810327+0000","last_clean":"2023-01-15T14:33:50.143204+0000","last_became_active":"2023-01-15T14:33:58.306184+0000","last_became_peered":"2023-01-15T14:33:58.306184+0000","last_unstale":"2023-01-15T14:34:17.810327+0000","last_undegraded":"2023-01-15T14:34:17.810327+0000","last_fullsized":"2023-01-15T14:33:58.001929+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2624160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[1,0],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"1,2,4","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":30,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.169562+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:34:16.169562+0000","last_peered":"2023-01-15T14:34:16.169562+0000","last_clean":"2023-01-15T14:34:16.169562+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:34:16.169562+0000","last_undegraded":"2023-01-15T14:34:16.169562+0000","last_fullsized":"2023-01-15T14:34:16.169562+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":1015,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.078175+0000","last_change":"2023-01-15T14:33:57.297870+0000","last_active":"2023-01-15T14:34:17.078175+0000","last_peered":"2023-01-15T14:34:17.078175+0000","last_clean":"2023-01-15T14:34:17.078175+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:34:17.078175+0000","last_undegraded":"2023-01-15T14:34:17.078175+0000","last_fullsized":"2023-01-15T14:34:17.078175+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.38976201700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":413,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.083544+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:34:17.083544+0000","last_peered":"2023-01-15T14:34:17.083544+0000","last_clean":"2023-01-15T14:34:17.083544+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:34:17.083544+0000","last_undegraded":"2023-01-15T14:34:17.083544+0000","last_fullsized":"2023-01-15T14:34:17.083544+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:20.964","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":735,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.020705+0000","last_change":"2023-01-15T14:34:20.020705+0000","last_active":"2023-01-15T14:34:20.020705+0000","last_peered":"2023-01-15T14:34:20.020705+0000","last_clean":"2023-01-15T14:34:20.020705+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:34:20.020705+0000","last_undegraded":"2023-01-15T14:34:20.020705+0000","last_fullsized":"2023-01-15T14:34:20.020705+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.0027174830000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":169,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077721+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:34:17.077721+0000","last_peered":"2023-01-15T14:34:17.077721+0000","last_clean":"2023-01-15T14:34:17.077721+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:34:17.077721+0000","last_undegraded":"2023-01-15T14:34:17.077721+0000","last_fullsized":"2023-01-15T14:34:17.077721+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1337,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.076095+0000","last_change":"2023-01-15T14:33:57.830460+0000","last_active":"2023-01-15T14:34:16.076095+0000","last_peered":"2023-01-15T14:34:16.076095+0000","last_clean":"2023-01-15T14:34:16.076095+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:34:16.076095+0000","last_undegraded":"2023-01-15T14:34:16.076095+0000","last_fullsized":"2023-01-15T14:34:16.076095+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.52931870400000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":235,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.976628+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:34:15.976628+0000","last_peered":"2023-01-15T14:34:15.976628+0000","last_clean":"2023-01-15T14:34:15.976628+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:34:15.976628+0000","last_undegraded":"2023-01-15T14:34:15.976628+0000","last_fullsized":"2023-01-15T14:34:15.976628+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:25.583","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"352'325","reported_seq":1437,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077942+0000","last_change":"2023-01-15T14:34:01.953246+0000","last_active":"2023-01-15T14:34:17.077942+0000","last_peered":"2023-01-15T14:34:17.077942+0000","last_clean":"2023-01-15T14:34:17.077942+0000","last_became_active":"2023-01-15T14:34:00.043073+0000","last_became_peered":"2023-01-15T14:34:00.043073+0000","last_unstale":"2023-01-15T14:34:17.077942+0000","last_undegraded":"2023-01-15T14:34:17.077942+0000","last_fullsized":"2023-01-15T14:34:17.077942+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:26.745","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.80204535600000004,"stat_sum":{"num_bytes":10343843,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":13875,"num_write":6,"num_write_kb":7330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9428925,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5","version":"355'160","reported_seq":1233,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.178680+0000","last_change":"2023-01-15T14:34:04.889983+0000","last_active":"2023-01-15T14:34:17.178680+0000","last_peered":"2023-01-15T14:34:17.178680+0000","last_clean":"2023-01-15T14:34:17.178680+0000","last_became_active":"2023-01-15T14:34:04.175473+0000","last_became_peered":"2023-01-15T14:34:04.175473+0000","last_unstale":"2023-01-15T14:34:17.178680+0000","last_undegraded":"2023-01-15T14:34:17.178680+0000","last_fullsized":"2023-01-15T14:34:17.178680+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.61306555500000004,"stat_sum":{"num_bytes":11390101,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":26145,"num_write":9,"num_write_kb":4265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":13260921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":225,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.976058+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:34:15.976058+0000","last_peered":"2023-01-15T14:34:15.976058+0000","last_clean":"2023-01-15T14:34:15.976058+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:34:15.976058+0000","last_undegraded":"2023-01-15T14:34:15.976058+0000","last_fullsized":"2023-01-15T14:34:15.976058+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"367'311","reported_seq":1505,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.385128+0000","last_change":"2023-01-15T14:34:11.259757+0000","last_active":"2023-01-15T14:34:17.385128+0000","last_peered":"2023-01-15T14:34:17.385128+0000","last_clean":"2023-01-15T14:34:17.385128+0000","last_became_active":"2023-01-15T14:34:11.152893+0000","last_became_peered":"2023-01-15T14:34:11.152893+0000","last_unstale":"2023-01-15T14:34:17.385128+0000","last_undegraded":"2023-01-15T14:34:17.385128+0000","last_fullsized":"2023-01-15T14:34:17.385128+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"125'143","last_scrub_stamp":"2023-01-15T14:30:22.558931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:22.558931+0000","objects_scrubbed":3,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50739213100000002,"objects_trimmed":0,"snaptrim_duration":0.104600024,"stat_sum":{"num_bytes":2954643,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":31,"num_read_kb":30684,"num_write":33,"num_write_kb":10131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":41,"num_bytes_recovered":14805347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":414,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.170714+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:34:16.170714+0000","last_peered":"2023-01-15T14:34:16.170714+0000","last_clean":"2023-01-15T14:34:16.170714+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:34:16.170714+0000","last_undegraded":"2023-01-15T14:34:16.170714+0000","last_fullsized":"2023-01-15T14:34:16.170714+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"367'143","reported_seq":1149,"reported_epoch":367,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:17.487872+0000","last_change":"2023-01-15T14:34:17.229236+0000","last_active":"2023-01-15T14:34:17.487872+0000","last_peered":"2023-01-15T14:34:17.487872+0000","last_clean":"2023-01-15T14:34:17.487872+0000","last_became_active":"2023-01-15T14:34:17.186013+0000","last_became_peered":"2023-01-15T14:34:17.186013+0000","last_unstale":"2023-01-15T14:34:17.487872+0000","last_undegraded":"2023-01-15T14:34:17.487872+0000","last_fullsized":"2023-01-15T14:34:17.487872+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.471195633,"stat_sum":{"num_bytes":9475691,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":13597,"num_write":13,"num_write_kb":8618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3546343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":225,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.076417+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:34:16.076417+0000","last_peered":"2023-01-15T14:34:16.076417+0000","last_clean":"2023-01-15T14:34:16.076417+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:34:16.076417+0000","last_undegraded":"2023-01-15T14:34:16.076417+0000","last_fullsized":"2023-01-15T14:34:16.076417+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:17.360","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":60,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.027314+0000","last_change":"2023-01-15T14:34:20.027314+0000","last_active":"2023-01-15T14:34:20.027314+0000","last_peered":"2023-01-15T14:34:20.027314+0000","last_clean":"2023-01-15T14:34:20.027314+0000","last_became_active":"2023-01-15T14:33:52.206701+0000","last_became_peered":"2023-01-15T14:33:52.206701+0000","last_unstale":"2023-01-15T14:34:20.027314+0000","last_undegraded":"2023-01-15T14:34:20.027314+0000","last_fullsized":"2023-01-15T14:34:20.027314+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.0080812169999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070771+0000","last_change":"2023-01-15T14:33:57.412557+0000","last_active":"2023-01-15T14:34:16.070771+0000","last_peered":"2023-01-15T14:34:16.070771+0000","last_clean":"2023-01-15T14:34:16.070771+0000","last_became_active":"2023-01-15T14:33:57.311981+0000","last_became_peered":"2023-01-15T14:33:57.311981+0000","last_unstale":"2023-01-15T14:34:16.070771+0000","last_undegraded":"2023-01-15T14:34:16.070771+0000","last_fullsized":"2023-01-15T14:34:16.070771+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"351'159","reported_seq":1209,"reported_epoch":368,"state":"active+clean","last_fresh":"2023-01-15T14:34:18.607921+0000","last_change":"2023-01-15T14:34:18.607843+0000","last_active":"2023-01-15T14:34:18.607921+0000","last_peered":"2023-01-15T14:34:18.607921+0000","last_clean":"2023-01-15T14:34:18.607921+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:34:18.607921+0000","last_undegraded":"2023-01-15T14:34:18.607921+0000","last_fullsized":"2023-01-15T14:34:18.607921+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"351'159","last_scrub_stamp":"2023-01-15T14:34:18.607751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:18.607751+0000","objects_scrubbed":2,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":0.40666956999999998,"objects_trimmed":0,"snaptrim_duration":0.0010524499999999999,"stat_sum":{"num_bytes":2453855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":39,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070097+0000","last_change":"2023-01-15T14:33:58.327664+0000","last_active":"2023-01-15T14:34:16.070097+0000","last_peered":"2023-01-15T14:34:16.070097+0000","last_clean":"2023-01-15T14:34:16.070097+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:34:16.070097+0000","last_undegraded":"2023-01-15T14:34:16.070097+0000","last_fullsized":"2023-01-15T14:34:16.070097+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31327724499999998,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.16","version":"366'131","reported_seq":686,"reported_epoch":367,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:34:17.386323+0000","last_change":"2023-01-15T14:34:17.387118+0000","last_active":"2023-01-15T14:34:17.386323+0000","last_peered":"2023-01-15T14:34:17.386323+0000","last_clean":"2023-01-15T14:33:55.270604+0000","last_became_active":"2023-01-15T14:33:58.218043+0000","last_became_peered":"2023-01-15T14:33:58.218043+0000","last_unstale":"2023-01-15T14:34:17.386323+0000","last_undegraded":"2023-01-15T14:33:57.906873+0000","last_fullsized":"2023-01-15T14:33:57.906218+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:09.523","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.30046122199999997,"stat_sum":{"num_bytes":7378471,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7378471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[1,3],"avail_no_missing":["1","2","3","7"],"object_location_counts":[{"shards":"1,3","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":1,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":57,"reported_epoch":370,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:20.119910+0000","last_change":"2023-01-15T14:34:20.119861+0000","last_active":"2023-01-15T14:34:20.119910+0000","last_peered":"2023-01-15T14:34:20.119910+0000","last_clean":"2023-01-15T14:34:20.119910+0000","last_became_active":"2023-01-15T14:33:52.303197+0000","last_became_peered":"2023-01-15T14:33:52.303197+0000","last_unstale":"2023-01-15T14:34:20.119910+0000","last_undegraded":"2023-01-15T14:34:20.119910+0000","last_fullsized":"2023-01-15T14:34:20.119910+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.265135238,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.18","version":"355'328","reported_seq":1554,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077790+0000","last_change":"2023-01-15T14:34:04.674366+0000","last_active":"2023-01-15T14:34:17.077790+0000","last_peered":"2023-01-15T14:34:17.077790+0000","last_clean":"2023-01-15T14:34:17.077790+0000","last_became_active":"2023-01-15T14:34:03.265912+0000","last_became_peered":"2023-01-15T14:34:03.265912+0000","last_unstale":"2023-01-15T14:34:17.077790+0000","last_undegraded":"2023-01-15T14:34:17.077790+0000","last_fullsized":"2023-01-15T14:34:17.077790+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.70539038499999995,"stat_sum":{"num_bytes":4332078,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":26,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":12147901,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.19","version":"294'256","reported_seq":1336,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:15.976382+0000","last_change":"2023-01-15T14:33:57.626542+0000","last_active":"2023-01-15T14:34:15.976382+0000","last_peered":"2023-01-15T14:34:15.976382+0000","last_clean":"2023-01-15T14:34:15.976382+0000","last_became_active":"2023-01-15T14:33:35.259904+0000","last_became_peered":"2023-01-15T14:33:35.259904+0000","last_unstale":"2023-01-15T14:34:15.976382+0000","last_undegraded":"2023-01-15T14:34:15.976382+0000","last_fullsized":"2023-01-15T14:34:15.976382+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:01.296","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.52734117999999996,"stat_sum":{"num_bytes":2420274,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1a","version":"348'352","reported_seq":1470,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.977959+0000","last_change":"2023-01-15T14:33:58.011960+0000","last_active":"2023-01-15T14:34:16.977959+0000","last_peered":"2023-01-15T14:34:16.977959+0000","last_clean":"2023-01-15T14:34:16.977959+0000","last_became_active":"2023-01-15T14:33:35.063699+0000","last_became_peered":"2023-01-15T14:33:35.063699+0000","last_unstale":"2023-01-15T14:34:16.977959+0000","last_undegraded":"2023-01-15T14:34:16.977959+0000","last_fullsized":"2023-01-15T14:34:16.977959+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.0039007270000000001,"stat_sum":{"num_bytes":3052653,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":15,"num_read_kb":21426,"num_write":7,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1b","version":"370'101","reported_seq":86,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.338457+0000","last_change":"2023-01-15T14:34:21.339277+0000","last_active":"2023-01-15T14:34:21.338457+0000","last_peered":"2023-01-15T14:34:21.338457+0000","last_clean":"2023-01-15T14:34:21.338457+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:34:21.338457+0000","last_undegraded":"2023-01-15T14:34:21.338457+0000","last_fullsized":"2023-01-15T14:34:21.338457+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.31244948099999997,"stat_sum":{"num_bytes":4400087,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10511,"num_write":14,"num_write_kb":4701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9416119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":567,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.079973+0000","last_change":"2023-01-15T14:34:16.278357+0000","last_active":"2023-01-15T14:34:17.079973+0000","last_peered":"2023-01-15T14:34:17.079973+0000","last_clean":"2023-01-15T14:34:17.079973+0000","last_became_active":"2023-01-15T14:33:57.125876+0000","last_became_peered":"2023-01-15T14:33:57.125876+0000","last_unstale":"2023-01-15T14:34:17.079973+0000","last_undegraded":"2023-01-15T14:34:17.079973+0000","last_fullsized":"2023-01-15T14:34:17.079973+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:21.278","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.0028318649999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1283,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978994+0000","last_change":"2023-01-15T14:33:57.405395+0000","last_active":"2023-01-15T14:34:16.978994+0000","last_peered":"2023-01-15T14:34:16.978994+0000","last_clean":"2023-01-15T14:34:16.978994+0000","last_became_active":"2023-01-15T14:33:35.362912+0000","last_became_peered":"2023-01-15T14:33:35.362912+0000","last_unstale":"2023-01-15T14:34:16.978994+0000","last_undegraded":"2023-01-15T14:34:16.978994+0000","last_fullsized":"2023-01-15T14:34:16.978994+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.394560678,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":994,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.978393+0000","last_change":"2023-01-15T14:33:57.317895+0000","last_active":"2023-01-15T14:34:16.978393+0000","last_peered":"2023-01-15T14:34:16.978393+0000","last_clean":"2023-01-15T14:34:16.978393+0000","last_became_active":"2023-01-15T14:33:35.261561+0000","last_became_peered":"2023-01-15T14:33:35.261561+0000","last_unstale":"2023-01-15T14:34:16.978393+0000","last_undegraded":"2023-01-15T14:34:16.978393+0000","last_fullsized":"2023-01-15T14:34:16.978393+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.40752737100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1f","version":"369'259","reported_seq":1350,"reported_epoch":370,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:20.119452+0000","last_change":"2023-01-15T14:34:20.119419+0000","last_active":"2023-01-15T14:34:20.119452+0000","last_peered":"2023-01-15T14:34:20.119452+0000","last_clean":"2023-01-15T14:34:20.119452+0000","last_became_active":"2023-01-15T14:34:00.240936+0000","last_became_peered":"2023-01-15T14:34:00.240936+0000","last_unstale":"2023-01-15T14:34:20.119452+0000","last_undegraded":"2023-01-15T14:34:20.119452+0000","last_fullsized":"2023-01-15T14:34:20.119452+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.20345100599999999,"stat_sum":{"num_bytes":5361581,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":13342,"num_write":5,"num_write_kb":2626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":5949777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.20","version":"324'273","reported_seq":1378,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.223902+0000","last_change":"2023-01-15T14:34:20.223902+0000","last_active":"2023-01-15T14:34:20.223902+0000","last_peered":"2023-01-15T14:34:20.223902+0000","last_clean":"2023-01-15T14:34:20.223902+0000","last_became_active":"2023-01-15T14:33:55.472883+0000","last_became_peered":"2023-01-15T14:33:55.472883+0000","last_unstale":"2023-01-15T14:34:20.223902+0000","last_undegraded":"2023-01-15T14:34:20.223902+0000","last_fullsized":"2023-01-15T14:34:20.223902+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.20407808399999999,"stat_sum":{"num_bytes":4251137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10864,"num_write":9,"num_write_kb":4140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4438968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.21","version":"321'138","reported_seq":1058,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069125+0000","last_change":"2023-01-15T14:33:57.005196+0000","last_active":"2023-01-15T14:34:16.069125+0000","last_peered":"2023-01-15T14:34:16.069125+0000","last_clean":"2023-01-15T14:34:16.069125+0000","last_became_active":"2023-01-15T14:33:45.267321+0000","last_became_peered":"2023-01-15T14:33:45.267321+0000","last_unstale":"2023-01-15T14:34:16.069125+0000","last_undegraded":"2023-01-15T14:34:16.069125+0000","last_fullsized":"2023-01-15T14:34:16.069125+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.102357873,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.22","version":"323'161","reported_seq":1349,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.977591+0000","last_change":"2023-01-15T14:33:57.619172+0000","last_active":"2023-01-15T14:34:16.977591+0000","last_peered":"2023-01-15T14:34:16.977591+0000","last_clean":"2023-01-15T14:34:16.977591+0000","last_became_active":"2023-01-15T14:33:55.378253+0000","last_became_peered":"2023-01-15T14:33:55.378253+0000","last_unstale":"2023-01-15T14:34:16.977591+0000","last_undegraded":"2023-01-15T14:34:16.977591+0000","last_fullsized":"2023-01-15T14:34:16.977591+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.50935289699999997,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":762,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.731183+0000","last_change":"2023-01-15T14:34:20.731183+0000","last_active":"2023-01-15T14:34:20.731183+0000","last_peered":"2023-01-15T14:34:20.731183+0000","last_clean":"2023-01-15T14:34:20.731183+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:34:20.731183+0000","last_undegraded":"2023-01-15T14:34:20.731183+0000","last_fullsized":"2023-01-15T14:34:20.731183+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.71105042500000004,"stat_sum":{"num_bytes":3802136,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":21818,"num_write":18,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.24","version":"349'263","reported_seq":1418,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.987406+0000","last_change":"2023-01-15T14:33:57.616353+0000","last_active":"2023-01-15T14:34:16.987406+0000","last_peered":"2023-01-15T14:34:16.987406+0000","last_clean":"2023-01-15T14:34:16.987406+0000","last_became_active":"2023-01-15T14:33:45.075309+0000","last_became_peered":"2023-01-15T14:33:45.075309+0000","last_unstale":"2023-01-15T14:34:16.987406+0000","last_undegraded":"2023-01-15T14:34:16.987406+0000","last_fullsized":"2023-01-15T14:34:16.987406+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":263,"log_dups_size":0,"ondisk_log_size":263,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.70814871000000001,"stat_sum":{"num_bytes":7636039,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":22662,"num_write":10,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.25","version":"333'108","reported_seq":1130,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.076583+0000","last_change":"2023-01-15T14:33:57.331159+0000","last_active":"2023-01-15T14:34:16.076583+0000","last_peered":"2023-01-15T14:34:16.076583+0000","last_clean":"2023-01-15T14:34:16.076583+0000","last_became_active":"2023-01-15T14:33:45.168598+0000","last_became_peered":"2023-01-15T14:33:45.168598+0000","last_unstale":"2023-01-15T14:34:16.076583+0000","last_undegraded":"2023-01-15T14:34:16.076583+0000","last_fullsized":"2023-01-15T14:34:16.076583+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.33264791100000002,"stat_sum":{"num_bytes":2638765,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3030,"num_write":2,"num_write_kb":2577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":482,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070812+0000","last_change":"2023-01-15T14:33:58.119558+0000","last_active":"2023-01-15T14:34:16.070812+0000","last_peered":"2023-01-15T14:34:16.070812+0000","last_clean":"2023-01-15T14:34:16.070812+0000","last_became_active":"2023-01-15T14:33:52.309110+0000","last_became_peered":"2023-01-15T14:33:52.309110+0000","last_unstale":"2023-01-15T14:34:16.070812+0000","last_undegraded":"2023-01-15T14:34:16.070812+0000","last_fullsized":"2023-01-15T14:34:16.070812+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.0016362760000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":1081,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.979186+0000","last_change":"2023-01-15T14:33:57.355572+0000","last_active":"2023-01-15T14:34:16.979186+0000","last_peered":"2023-01-15T14:34:16.979186+0000","last_clean":"2023-01-15T14:34:16.979186+0000","last_became_active":"2023-01-15T14:33:45.171090+0000","last_became_peered":"2023-01-15T14:33:45.171090+0000","last_unstale":"2023-01-15T14:34:16.979186+0000","last_undegraded":"2023-01-15T14:34:16.979186+0000","last_fullsized":"2023-01-15T14:34:16.979186+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.443073948,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.28","version":"333'248","reported_seq":1399,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.977267+0000","last_change":"2023-01-15T14:33:57.617361+0000","last_active":"2023-01-15T14:34:16.977267+0000","last_peered":"2023-01-15T14:34:16.977267+0000","last_clean":"2023-01-15T14:34:16.977267+0000","last_became_active":"2023-01-15T14:33:45.366831+0000","last_became_peered":"2023-01-15T14:33:45.366831+0000","last_unstale":"2023-01-15T14:34:16.977267+0000","last_undegraded":"2023-01-15T14:34:16.977267+0000","last_fullsized":"2023-01-15T14:34:16.977267+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.60974191300000002,"stat_sum":{"num_bytes":4838389,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8969,"num_write":8,"num_write_kb":5185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":6096764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":903,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:17.077611+0000","last_change":"2023-01-15T14:33:57.620313+0000","last_active":"2023-01-15T14:34:17.077611+0000","last_peered":"2023-01-15T14:34:17.077611+0000","last_clean":"2023-01-15T14:34:17.077611+0000","last_became_active":"2023-01-15T14:33:45.276960+0000","last_became_peered":"2023-01-15T14:33:45.276960+0000","last_unstale":"2023-01-15T14:34:17.077611+0000","last_undegraded":"2023-01-15T14:34:17.077611+0000","last_fullsized":"2023-01-15T14:34:17.077611+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.51033217900000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1248,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.069645+0000","last_change":"2023-01-15T14:33:56.904528+0000","last_active":"2023-01-15T14:34:16.069645+0000","last_peered":"2023-01-15T14:34:16.069645+0000","last_clean":"2023-01-15T14:34:16.069645+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:34:16.069645+0000","last_undegraded":"2023-01-15T14:34:16.069645+0000","last_fullsized":"2023-01-15T14:34:16.069645+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.0022419219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":70,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.734017+0000","last_change":"2023-01-15T14:34:20.734017+0000","last_active":"2023-01-15T14:34:20.734017+0000","last_peered":"2023-01-15T14:34:20.734017+0000","last_clean":"2023-01-15T14:34:20.734017+0000","last_became_active":"2023-01-15T14:33:57.261456+0000","last_became_peered":"2023-01-15T14:33:57.261456+0000","last_unstale":"2023-01-15T14:34:20.734017+0000","last_undegraded":"2023-01-15T14:34:20.734017+0000","last_fullsized":"2023-01-15T14:34:20.734017+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.71353561799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2c","version":"141'46","reported_seq":452,"reported_epoch":367,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.979605+0000","last_change":"2023-01-15T14:33:57.303687+0000","last_active":"2023-01-15T14:34:16.979605+0000","last_peered":"2023-01-15T14:34:16.979605+0000","last_clean":"2023-01-15T14:34:16.979605+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:34:16.979605+0000","last_undegraded":"2023-01-15T14:34:16.979605+0000","last_fullsized":"2023-01-15T14:34:16.979605+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.29451823399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1221,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.170580+0000","last_change":"2023-01-15T14:33:58.122688+0000","last_active":"2023-01-15T14:34:16.170580+0000","last_peered":"2023-01-15T14:34:16.170580+0000","last_clean":"2023-01-15T14:34:16.170580+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:34:16.170580+0000","last_undegraded":"2023-01-15T14:34:16.170580+0000","last_fullsized":"2023-01-15T14:34:16.170580+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.105464771,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":39,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070235+0000","last_change":"2023-01-15T14:33:58.326712+0000","last_active":"2023-01-15T14:34:16.070235+0000","last_peered":"2023-01-15T14:34:16.070235+0000","last_clean":"2023-01-15T14:34:16.070235+0000","last_became_active":"2023-01-15T14:33:57.210434+0000","last_became_peered":"2023-01-15T14:33:57.210434+0000","last_unstale":"2023-01-15T14:34:16.070235+0000","last_undegraded":"2023-01-15T14:34:16.070235+0000","last_fullsized":"2023-01-15T14:34:16.070235+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.31032643199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1115,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.070820+0000","last_change":"2023-01-15T14:33:58.121127+0000","last_active":"2023-01-15T14:34:16.070820+0000","last_peered":"2023-01-15T14:34:16.070820+0000","last_clean":"2023-01-15T14:34:16.070820+0000","last_became_active":"2023-01-15T14:33:55.174128+0000","last_became_peered":"2023-01-15T14:33:55.174128+0000","last_unstale":"2023-01-15T14:34:16.070820+0000","last_undegraded":"2023-01-15T14:34:16.070820+0000","last_fullsized":"2023-01-15T14:34:16.070820+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.20511622600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":1000,"reported_epoch":370,"state":"active+clean","last_fresh":"2023-01-15T14:34:20.121669+0000","last_change":"2023-01-15T14:34:20.121669+0000","last_active":"2023-01-15T14:34:20.121669+0000","last_peered":"2023-01-15T14:34:20.121669+0000","last_clean":"2023-01-15T14:34:20.121669+0000","last_became_active":"2023-01-15T14:33:55.373498+0000","last_became_peered":"2023-01-15T14:33:55.373498+0000","last_unstale":"2023-01-15T14:34:20.121669+0000","last_undegraded":"2023-01-15T14:34:20.121669+0000","last_fullsized":"2023-01-15T14:34:20.121669+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":7,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.102098561,"stat_sum":{"num_bytes":5419866,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":25948,"num_write":35,"num_write_kb":13827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":13340458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.31","version":"204'150","reported_seq":1234,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-01-15T14:34:16.076397+0000","last_change":"2023-01-15T14:33:57.625989+0000","last_active":"2023-01-15T14:34:16.076397+0000","last_peered":"2023-01-15T14:34:16.076397+0000","last_clean":"2023-01-15T14:34:16.076397+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:34:16.076397+0000","last_undegraded":"2023-01-15T14:34:16.076397+0000","last_fullsized":"2023-01-15T14:34:16.076397+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.52721113600000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":106,"stat_sum":{"num_bytes":361813132,"num_objects":167,"num_object_clones":115,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":15,"num_read":1019,"num_read_kb":1109260,"num_write":816,"num_write_kb":334870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":723,"num_bytes_recovered":466236039,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18744,"ondisk_log_size":18744,"up":318,"acting":316,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2624160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574924,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1107636,"kb_used_data":92630184,"kb_used_omap":7967,"kb_used_meta":0,"kb_avail":92591744,"statfs":{"total":95948165120,"available":94813945856,"internally_reserved":39362560,"allocated":94853308416,"data_stored":94853308416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8158317,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":4.1269999999999998,"15min":4.0890000000000004},"min":{"1min":0.54000000000000004,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2030000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":9.5380000000000003,"5min":6.3200000000000003,"15min":8.2460000000000004},"min":{"1min":0.58799999999999997,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":96.337000000000003,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.80900000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":2.343,"5min":3.6509999999999998,"15min":5.5800000000000001},"min":{"1min":0.48399999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":34.585999999999999,"5min":88.003,"15min":88.003},"last":0.90200000000000002},{"interface":"front","average":{"1min":13.302,"5min":8.1950000000000003,"15min":6.9180000000000001},"min":{"1min":0.58799999999999997,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":96.403999999999996,"5min":97.704999999999998,"15min":97.704999999999998},"last":1.3460000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":6.9400000000000004,"15min":5.5069999999999997},"min":{"1min":0.57199999999999995,"5min":0.436,"15min":0.436},"max":{"1min":1.179,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.91800000000000004,"5min":2.5790000000000002,"15min":3.645},"min":{"1min":0.626,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.2050000000000001,"5min":91.837999999999994,"15min":91.837999999999994},"last":0.99399999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":7.3799999999999999,"15min":3.0920000000000001},"min":{"1min":0.63600000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.2370000000000001,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.91600000000000004,"5min":3.6360000000000001,"15min":4.3129999999999997},"min":{"1min":0.65000000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.2270000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":1.1220000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.5519999999999996,"5min":5.6219999999999999,"15min":8.1509999999999998},"min":{"1min":0.63500000000000001,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":96.185000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.9259999999999999,"5min":5.782,"15min":2.5699999999999998},"min":{"1min":0.73099999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":22.41,"5min":164.21199999999999,"15min":164.21199999999999},"last":1.042}]},{"osd":5,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.6440000000000001,"5min":5.0659999999999998,"15min":5.9059999999999997},"min":{"1min":0.72799999999999998,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":96.263999999999996,"5min":96.263999999999996,"15min":96.263999999999996},"last":1.071},{"interface":"front","average":{"1min":0.998,"5min":3.206,"15min":4.3650000000000002},"min":{"1min":0.69299999999999995,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.355,"5min":95.430000000000007,"15min":95.430000000000007},"last":1.141}]},{"osd":6,"last update":"Sun Jan 15 14:33:18 2023","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":5.5949999999999998,"15min":7.5469999999999997},"min":{"1min":0.83599999999999997,"5min":0.496,"15min":0.496},"max":{"1min":96.299999999999997,"5min":96.299999999999997,"15min":96.299999999999997},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.089,"5min":2.5150000000000001,"15min":1.571},"min":{"1min":0.84299999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.2789999999999999,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.2050000000000001}]}]},{"osd":6,"up_from":0,"seq":55834574926,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574926,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574924,"num_pgs":62,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":989272,"kb_used_data":92710224,"kb_used_omap":13956,"kb_used_meta":0,"kb_avail":92710108,"statfs":{"total":95948165120,"available":94935150592,"internally_reserved":118784,"allocated":94935269376,"data_stored":94935269376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14290945,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":22.818000000000001,"15min":10.882},"min":{"1min":0.505,"5min":0.44,"15min":0.44},"max":{"1min":5.0049999999999999,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.997},{"interface":"front","average":{"1min":0.77400000000000002,"5min":2.9990000000000001,"15min":1.52},"min":{"1min":0.51200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.127,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.73799999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":5.0730000000000004,"15min":7.4199999999999999},"min":{"1min":0.49299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6659999999999999,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.343,"15min":1.9770000000000001},"min":{"1min":0.58399999999999996,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0529999999999999,"5min":110.06999999999999,"15min":110.06999999999999},"last":0.93300000000000005}]},{"osd":2,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.0950000000000002,"5min":23.544,"15min":11.522},"min":{"1min":0.497,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":23.247,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.26,"5min":1.28,"15min":1.0329999999999999},"min":{"1min":0.61899999999999999,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.5789999999999997,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.68500000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":2.802,"15min":1.8260000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.006,"5min":88.823999999999998,"15min":88.823999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":5.3380000000000001,"5min":24.337,"15min":12.212999999999999},"min":{"1min":0.68600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":58.317,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":0.89900000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88100000000000001,"15min":0.90700000000000003},"min":{"1min":0.61499999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.175,"5min":2.577,"15min":2.577},"last":1.1499999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.903,"15min":2.1669999999999998},"min":{"1min":0.69099999999999995,"5min":0.65200000000000002,"15min":0.55300000000000005},"max":{"1min":1.0700000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.82299999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":3.2389999999999999,"15min":4.8609999999999998},"min":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":23.329000000000001,"5min":85.721999999999994,"15min":99.780000000000001},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.999,"5min":21.631,"15min":10.651999999999999},"min":{"1min":0.79500000000000004,"5min":0.66700000000000004,"15min":0.58699999999999997},"max":{"1min":1.173,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.095}]},{"osd":7,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":3.238,"5min":3.5790000000000002,"15min":9.9450000000000003},"min":{"1min":0.73999999999999999,"5min":0.63400000000000001,"15min":0.59499999999999997},"max":{"1min":42.551000000000002,"5min":89.061000000000007,"15min":179.73500000000001},"last":0.88800000000000001},{"interface":"front","average":{"1min":2.274,"5min":23.071000000000002,"15min":15.183},"min":{"1min":0.78700000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":23.356000000000002,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.0660000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574924,"num_pgs":61,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1057144,"kb_used_data":92642248,"kb_used_omap":14201,"kb_used_meta":0,"kb_avail":92642236,"statfs":{"total":95948165120,"available":94865649664,"internally_reserved":12288,"allocated":94865661952,"data_stored":94865661952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14541852,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.8360000000000001,"5min":7.4400000000000004,"15min":5.7119999999999997},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":17.731000000000002,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":3.4340000000000002,"15min":6.2389999999999999},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":2.9260000000000002,"5min":93.674000000000007,"15min":93.674000000000007},"last":0.77300000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":7.1790000000000003,"15min":11.414},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96799999999999997,"5min":159.256,"15min":159.256},"last":0.65300000000000002},{"interface":"front","average":{"1min":2.6259999999999999,"5min":10.417999999999999,"15min":3.9660000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":17.826000000000001,"5min":174.578,"15min":174.578},"last":0.58999999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":4.6849999999999996,"5min":2.8809999999999998,"15min":3.1930000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":86.412000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.628},{"interface":"front","average":{"1min":1.95,"5min":6.7889999999999997,"15min":3.7250000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":17.876000000000001,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.747}]},{"osd":4,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":7.6879999999999997,"5min":29.686,"15min":14.27},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":84.640000000000001,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.86899999999999999,"5min":32.134999999999998,"15min":19.960999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.615,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":1.125}]},{"osd":5,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":6.4980000000000002,"15min":11.433999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":2.04,"5min":159.43000000000001,"15min":159.43000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":2.2309999999999999,"5min":6.0039999999999996,"15min":2.6819999999999999},"min":{"1min":0.70099999999999996,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":17.917000000000002,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.86199999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1949999999999998,"5min":5.9980000000000002,"15min":2.6869999999999998},"min":{"1min":0.66000000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":17.989000000000001,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.964,"15min":3.1619999999999999},"min":{"1min":0.70499999999999996,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":3.23,"5min":60.098999999999997,"15min":60.098999999999997},"last":1.006}]},{"osd":7,"last update":"Sun Jan 15 14:33:16 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":2.0219999999999998,"15min":3.3639999999999999},"min":{"1min":0.69499999999999995,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":3.266,"5min":60.165999999999997,"15min":60.165999999999997},"last":1.0589999999999999},{"interface":"front","average":{"1min":19.138000000000002,"5min":11.705,"15min":15.961},"min":{"1min":0.66200000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":281.13400000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":1.1579999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574925,"num_pgs":47,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1009728,"kb_used_data":92690828,"kb_used_omap":11856,"kb_used_meta":0,"kb_avail":92689652,"statfs":{"total":95948165120,"available":94914203648,"internally_reserved":1204224,"allocated":94915407872,"data_stored":94915407872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12140583,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":4.2469999999999999,"5min":4.2450000000000001,"15min":1.8420000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":78.456000000000003,"5min":78.456000000000003,"15min":78.456000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.4039999999999999,"15min":0.94399999999999995},"min":{"1min":0.36799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.57599999999999996}]},{"osd":1,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":10.247,"5min":7.7949999999999999,"15min":3.1309999999999998},"min":{"1min":0.35099999999999998,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":128.749,"5min":143.071,"15min":143.071},"last":0.54800000000000004},{"interface":"front","average":{"1min":3.105,"5min":1.966,"15min":2.8919999999999999},"min":{"1min":0.35999999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":52.927999999999997,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.65500000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":3.8879999999999999,"5min":6.9180000000000001,"15min":9.2400000000000002},"min":{"1min":0.33700000000000002,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":34.244,"5min":143.13300000000001,"15min":183.53},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.70399999999999996,"5min":7.5039999999999996,"15min":3.012},"min":{"1min":0.41399999999999998,"5min":0.316,"15min":0.316},"max":{"1min":1.3959999999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.73499999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":5.9480000000000004,"15min":4.4160000000000004},"min":{"1min":0.67200000000000004,"5min":0.624,"15min":0.58399999999999996},"max":{"1min":1.2450000000000001,"5min":175.34700000000001,"15min":175.34700000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.88300000000000001,"5min":4.7080000000000002,"15min":4.1079999999999997},"min":{"1min":0.57799999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.052,"5min":175.30500000000001,"15min":175.30500000000001},"last":1.036}]},{"osd":5,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":2.5590000000000002,"5min":2.7490000000000001,"15min":1.7},"min":{"1min":0.63500000000000001,"5min":0.57199999999999995,"15min":0.54800000000000004},"max":{"1min":34.256999999999998,"5min":143.262,"15min":143.262},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.95099999999999996,"5min":3.012,"15min":1.778},"min":{"1min":0.70199999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3160000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":1.0980000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.839,"15min":1.3009999999999999},"min":{"1min":0.61199999999999999,"5min":0.59599999999999997,"15min":0.55000000000000004},"max":{"1min":1.276,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.88100000000000001},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8980000000000001,"15min":1.778},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":34.305999999999997,"5min":143.215,"15min":143.215},"last":1.1299999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":11.396000000000001,"5min":7.9800000000000004,"15min":4.6989999999999998},"min":{"1min":0.60599999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":176.44499999999999,"5min":176.44499999999999,"15min":176.44499999999999},"last":49.232999999999997},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8159999999999998,"15min":2.79},"min":{"1min":0.622,"5min":0.622,"15min":0.54500000000000004},"max":{"1min":34.207999999999998,"5min":143.17099999999999,"15min":143.17099999999999},"last":1.0780000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":72,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1061392,"kb_used_data":92637988,"kb_used_omap":13231,"kb_used_meta":0,"kb_avail":92637988,"statfs":{"total":95948165120,"available":94861299712,"internally_reserved":0,"allocated":94861299712,"data_stored":94861299712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13549314,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":7,"commit_latency_ns":0,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":2.0619999999999998,"5min":3.2669999999999999,"15min":3.6840000000000002},"min":{"1min":0.37,"5min":0.314,"15min":0.314},"max":{"1min":23.68,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":14.157,"5min":6.5789999999999997,"15min":5.0750000000000002},"min":{"1min":0.42299999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":176.01599999999999,"5min":176.01599999999999,"15min":176.01599999999999},"last":101.319}]},{"osd":2,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":4.8959999999999999,"5min":3.4329999999999998,"15min":1.7889999999999999},"min":{"1min":0.44400000000000001,"5min":0.245,"15min":0.245},"max":{"1min":69.277000000000001,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.629,"5min":1.173,"15min":3.1059999999999999},"min":{"1min":0.28399999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.873,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.55900000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":5.8440000000000003,"15min":15.172000000000001},"min":{"1min":0.38400000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.92500000000000004,"5min":245.637,"15min":282.01400000000001},"last":0.88},{"interface":"front","average":{"1min":14.228999999999999,"5min":8.2870000000000008,"15min":7.5919999999999996},"min":{"1min":0.46500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":176.13900000000001,"5min":176.13900000000001,"15min":176.13900000000001},"last":101.401}]},{"osd":4,"last update":"Sun Jan 15 14:33:17 2023","interfaces":[{"interface":"back","average":{"1min":7.9249999999999998,"5min":17.140999999999998,"15min":8.8650000000000002},"min":{"1min":0.504,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":119.71899999999999,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":101.575},{"interface":"front","average":{"1min":1.7849999999999999,"5min":15.526,"15min":7.5780000000000003},"min":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":16.989999999999998,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":1.724}]},{"osd":5,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":5.2060000000000004,"5min":2.3799999999999999,"15min":1.482},"min":{"1min":0.56899999999999995,"5min":0.433,"15min":0.433},"max":{"1min":69.478999999999999,"5min":69.478999999999999,"15min":69.478999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":3.2770000000000001},"min":{"1min":0.66000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3089999999999999,"5min":4.0780000000000003,"15min":46.905999999999999},"last":0.82999999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":14.491,"5min":6.9109999999999996,"15min":5.6379999999999999},"min":{"1min":0.73199999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":176.249,"5min":176.249,"15min":176.249},"last":101.45999999999999},{"interface":"front","average":{"1min":5.2629999999999999,"5min":2.431,"15min":1.5589999999999999},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":69.549999999999997,"5min":69.549999999999997,"15min":69.549999999999997},"last":1.02}]},{"osd":7,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":7.617,"5min":6.2309999999999999,"15min":6.7549999999999999},"min":{"1min":0.63900000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":95.888999999999996,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":14.544,"5min":6.9649999999999999,"15min":5.4880000000000004},"min":{"1min":0.78700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":176.22,"5min":176.22,"15min":176.22},"last":101.515}]}]},{"osd":0,"up_from":13,"seq":55834574926,"num_pgs":68,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1070876,"kb_used_data":92630284,"kb_used_omap":8086,"kb_used_meta":0,"kb_avail":92628504,"statfs":{"total":95948165120,"available":94851588096,"internally_reserved":1822720,"allocated":94853410816,"data_stored":94853410816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8280359,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":3.3460000000000001,"15min":1.53},"min":{"1min":0.34899999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":25.960999999999999,"5min":69.872,"15min":69.872},"last":0.73099999999999998},{"interface":"front","average":{"1min":11.249000000000001,"5min":5.335,"15min":2.2650000000000001},"min":{"1min":0.434,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":162.91200000000001,"5min":162.91200000000001,"15min":162.91200000000001},"last":0.51400000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":3.415,"15min":1.601},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0109999999999999,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":2.6320000000000001,"15min":1.3500000000000001},"min":{"1min":0.37,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86899999999999999,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.48699999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":7.8079999999999998,"15min":13.788},"min":{"1min":0.48199999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":26.027000000000001,"5min":199.08000000000001,"15min":284.62799999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":3.6070000000000002,"5min":7.867,"15min":3.1560000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33600000000000002},"max":{"1min":61.119,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.752}]},{"osd":4,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":16.664999999999999,"15min":6.2119999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1719999999999999,"5min":1486.309,"15min":1486.309},"last":0.81000000000000005},{"interface":"front","average":{"1min":2.1030000000000002,"5min":18.140999999999998,"15min":6.7300000000000004},"min":{"1min":0.63100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":26.137,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":0.77200000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":2.8980000000000001,"15min":7.4980000000000002},"min":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1930000000000001,"5min":98.411000000000001,"15min":176.34399999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.7070000000000001,"15min":1.2609999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.101,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.91100000000000003}]},{"osd":6,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":2.1600000000000001,"5min":3.7890000000000001,"15min":2.0430000000000001},"min":{"1min":0.71899999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":26.082000000000001,"5min":70.259,"15min":70.259},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.026,"5min":3,"15min":1.756},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.57499999999999996},"max":{"1min":1.5409999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":1.0089999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":3.2770000000000001,"15min":11.539},"min":{"1min":0.70099999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":11.906000000000001,"5min":73.864999999999995,"15min":292.61099999999999},"last":0.96499999999999997},{"interface":"front","average":{"1min":2.1819999999999999,"5min":3.8090000000000002,"15min":2.0499999999999998},"min":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.58699999999999997},"max":{"1min":26.187000000000001,"5min":70.322000000000003,"15min":70.322000000000003},"last":0.83799999999999997}]}]}],"pool_statfs":[]}} 2023-01-15T14:34:23.037 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:23.205 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:23.205 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:23.034+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:23.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:34:23.213 INFO:tasks.rados.rados.0.smithi090.stdout:1413: done (0 left) 2023-01-15T14:34:23.213 INFO:tasks.rados.rados.0.smithi090.stdout:1414: read oid 6 snap 149 2023-01-15T14:34:23.214 INFO:tasks.rados.rados.0.smithi090.stdout:1414: expect (ObjNum 491 snap 144 seq_num 491) 2023-01-15T14:34:23.214 INFO:tasks.rados.rados.0.smithi090.stdout:1415: snap_remove snap 147 2023-01-15T14:34:23.214 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:23.210+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:23.314 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:23.307+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:23.352 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:23.352 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:23.352 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:34:23.416 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:23.410+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:23.464 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:23.547 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:23.549 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:34:23.549 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:23.543+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:23.649 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:23.643+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:23.739 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:23.739 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:23.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:34:23.750 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:23.746+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:24.174 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:34:24.175 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:24.170+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:24.175 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:24.175 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:24.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:34:24.179 INFO:tasks.rados.rados.0.smithi090.stdout:1415: done (1 left) 2023-01-15T14:34:24.179 INFO:tasks.rados.rados.0.smithi090.stdout:1416: write (excl) oid 2 current snap is 151 2023-01-15T14:34:24.179 INFO:tasks.rados.rados.0.smithi090.stdout:1416: seq_num 509 ranges {554763=781039,2102511=452252} 2023-01-15T14:34:24.179 INFO:tasks.rados.rados.0.smithi090.stdout:1416: writing smithi09025264-2 from 554763 to 1335802 tid 1 2023-01-15T14:34:24.179 INFO:tasks.rados.rados.0.smithi090.stdout:1416: writing smithi09025264-2 from 2102511 to 2554763 tid 2 2023-01-15T14:34:24.180 INFO:tasks.rados.rados.0.smithi090.stdout:1417: read oid 47 snap -1 2023-01-15T14:34:24.180 INFO:tasks.rados.rados.0.smithi090.stdout:1417: expect (ObjNum 477 snap 140 seq_num 477) 2023-01-15T14:34:24.180 INFO:tasks.rados.rados.0.smithi090.stdout:1418: write (excl) oid 35 current snap is 151 2023-01-15T14:34:24.180 INFO:tasks.rados.rados.0.smithi090.stdout:1418: seq_num 510 ranges {571570=572641,1877346=694224} 2023-01-15T14:34:24.180 INFO:tasks.rados.rados.0.smithi090.stdout:1418: writing smithi09025264-35 from 571570 to 1144211 tid 1 2023-01-15T14:34:24.180 INFO:tasks.rados.rados.0.smithi090.stdout:1418: writing smithi09025264-35 from 1877346 to 2571570 tid 2 2023-01-15T14:34:24.180 INFO:tasks.rados.rados.0.smithi090.stdout:1419: read oid 29 snap -1 2023-01-15T14:34:24.180 INFO:tasks.rados.rados.0.smithi090.stdout:1419: expect (ObjNum 402 snap 108 seq_num 402) 2023-01-15T14:34:24.181 INFO:tasks.rados.rados.0.smithi090.stdout:1420: snap_remove snap 136 2023-01-15T14:34:24.276 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:24.271+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:24.345 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:24.346 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:24.346 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:34:24.377 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:24.374+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:24.407 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:34:24.550 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:24.550 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:24.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:34:24.551 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:24.546+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:24.650 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:24.646+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:24.757 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:24.757 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:24.758 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:34:24.759 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:24.759 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:24.759 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:24.759 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:24.760 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:24.750+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:24.775 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:24.811 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:25.163 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:25.164 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:34:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:34:25.164 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:34:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:34:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:34:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:34:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:34:25.165 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:34:25.165 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:25.165 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:34:25.166 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:34:25.166 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:34:25.166 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:25.169 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:34:25.170 INFO:tasks.rados.rados.0.smithi090.stdout:1420: done (5 left) 2023-01-15T14:34:25.170 INFO:tasks.rados.rados.0.smithi090.stdout:1421: copy_from oid 15 from oid 48 current snap is 151 2023-01-15T14:34:25.170 INFO:tasks.rados.rados.0.smithi090.stdout:1414: expect (ObjNum 491 snap 144 seq_num 491) 2023-01-15T14:34:25.171 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:34:25.171 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:25.159+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:25.260 INFO:tasks.rados.rados.0.smithi090.stdout:1417: expect (ObjNum 477 snap 140 seq_num 477) 2023-01-15T14:34:25.375 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:25.572 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:34:25.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:34:25.572 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:34:25.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.572 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:34:25.573 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:34:25.573 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 230.425720106, 2023-01-15T14:34:25.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:34:25.573 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:25.573 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:25.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:25.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:25.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:25.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:34:25.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.573 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:25.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:25.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:34:25.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:25.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:34:25.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:25.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:34:25.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:25.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:25.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:34:25.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:34:25.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:25.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:34:25.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:34:25.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:25.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:34:25.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:34:25.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:34:25.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:34:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:34:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:34:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:34:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:34:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:34:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:34:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:34:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:34:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:34:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:34:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:34:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:34:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:34:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:34:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:34:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:34:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:34:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:34:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:34:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 230.42566916300001, 2023-01-15T14:34:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:34:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:34:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:34:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:34:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:34:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:34:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:34:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:34:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:34:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:34:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:34:25.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:34:25.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:34:25.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.780 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:25.781 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.781 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:34:25.781 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:34:25.781 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 227.484769922, 2023-01-15T14:34:25.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:34:25.781 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:25.781 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:25.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:25.781 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:25.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:34:25.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:25.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:34:25.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:25.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:25.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:34:25.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:34:25.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:25.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:34:25.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:34:25.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:25.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:34:25.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:34:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:34:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:34:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:34:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:34:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:34:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:34:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:34:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:34:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:34:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:34:25.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:34:25.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:34:25.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:25.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:34:25.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:34:25.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:34:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:34:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:34:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:34:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:34:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:34:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:34:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:34:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:34:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:34:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:34:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:34:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:25.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:34:25.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:34:25.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.791 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:25.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.791 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:34:25.791 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:34:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 225.41099574899999, 2023-01-15T14:34:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:34:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:34:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:34:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:34:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:34:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:34:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:34:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:34:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:34:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:34:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:34:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:34:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:34:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:34:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:34:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:34:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:34:25.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:25.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:34:25.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:34:25.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:34:25.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:34:25.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:34:25.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:34:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:34:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:34:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:34:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:34:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:34:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:34:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:34:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:34:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:34:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:34:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:34:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:34:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:34:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:34:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:34:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:34:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:34:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 201.517040329, 2023-01-15T14:34:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:34:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:34:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:34:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:34:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:34:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:34:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:34:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:34:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:34:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:34:25.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:34:25.962 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 179.633633648, 2023-01-15T14:34:25.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:25.964 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:34:25.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:34:25.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:34:25.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:25.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:25.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:34:25.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:34:25.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:34:25.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 179.63022652500001, 2023-01-15T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:34:25.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:25.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:34:25.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:25.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:25.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:34:26.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:34:26.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:26.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:34:26.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:34:26.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:34:26.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:26.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:34:26.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:34:26.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:26.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:34:26.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:34:26.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:26.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:34:26.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:34:26.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:26.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:34:26.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:34:26.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:26.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:34:26.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:34:26.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:26.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:34:26.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:34:26.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.185 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:26.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.185 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:34:26.186 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:34:26.186 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 179.63001091999999, 2023-01-15T14:34:26.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:34:26.186 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:26.186 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:26.187 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:26.187 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:26.187 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:26.187 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:34:26.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.187 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:26.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:26.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:34:26.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:26.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:34:26.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:26.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:34:26.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:26.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:26.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:34:26.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:34:26.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:26.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:34:26.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:34:26.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:26.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:34:26.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:34:26.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:34:26.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:34:26.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:26.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:34:26.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:34:26.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:34:26.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:26.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:34:26.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:34:26.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:26.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:34:26.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:34:26.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:26.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:34:26.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:34:26.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:26.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:34:26.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:34:26.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:26.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:34:26.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:34:26.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:26.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:34:26.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:34:26.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:26.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:34:26.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:34:26.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:26.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:34:26.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:34:26.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.345 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:26.347 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.347 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:34:26.353 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:34:26.353 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 179.62990384599999, 2023-01-15T14:34:26.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:34:26.353 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:26.353 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:26.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:26.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:26.353 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:26.353 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:34:26.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.354 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:26.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:26.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:34:26.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:26.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:34:26.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:34:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:34:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:34:26.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:34:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:34:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:34:26.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:34:26.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:34:26.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:34:26.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:26.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:34:26.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:26.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:34:26.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:26.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:34:26.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:34:26.498 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:34:26.500 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.49966479599999, 2023-01-15T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:34:26.501 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:26.502 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:34:26.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:34:26.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:34:26.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:34:26.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:26.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:26.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:34:26.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.516 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.49956025899999, 2023-01-15T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:26.517 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:26.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:34:26.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:34:26.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:34:26.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:34:26.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:26.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:34:26.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:34:26.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:34:26.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:34:26.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:26.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:34:26.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:34:26.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:26.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:34:26.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:34:26.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:26.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:34:26.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:34:26.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:26.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:34:26.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:34:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:34:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:34:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:26.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:34:26.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:34:26.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:26.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:34:26.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:34:26.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:26.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:34:26.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:34:26.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:26.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:34:26.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:34:26.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:26.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:34:26.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:34:26.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.654 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:26.654 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.655 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:34:26.655 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:34:26.655 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 171.49949032800001, 2023-01-15T14:34:26.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:34:26.655 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:26.655 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:26.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:26.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:26.655 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:26.655 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:34:26.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.656 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:26.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:26.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:34:26.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:26.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:34:26.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:26.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:34:26.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:26.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:26.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:34:26.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:34:26.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:26.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:34:26.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:34:26.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:26.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:34:26.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:34:26.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:34:26.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:34:26.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:26.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:34:26.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:34:26.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:34:26.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:34:26.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:26.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:34:26.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:34:26.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:26.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:34:26.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:34:26.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:26.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:34:26.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:34:26.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:26.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:34:26.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:34:26.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:34:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:34:26.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:34:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:34:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:34:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:34:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:34:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:34:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:34:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:34:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:34:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:34:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:26.694 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2780 3.38 e265/232 3:1eaad226:::smithi09025264-49:head v 265'279, mlcod=261'274)", 2023-01-15T14:34:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:34:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 113.328319202, 2023-01-15T14:34:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32005591799999999, 2023-01-15T14:34:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:34:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:34:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991860+0000", 2023-01-15T14:34:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993927+0000", 2023-01-15T14:34:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020673229999999998 2023-01-15T14:34:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993946+0000", 2023-01-15T14:34:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9429999999999999e-05 2023-01-15T14:34:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.094039+0000", 2023-01-15T14:34:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092898 2023-01-15T14:34:26.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285071+0000", 2023-01-15T14:34:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.191031904 2023-01-15T14:34:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285106+0000", 2023-01-15T14:34:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5227e-05 2023-01-15T14:34:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286323+0000", 2023-01-15T14:34:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012163829999999999 2023-01-15T14:34:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286335+0000", 2023-01-15T14:34:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2224e-05 2023-01-15T14:34:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287218+0000", 2023-01-15T14:34:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088265000000000001 2023-01-15T14:34:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287250+0000", 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2209000000000003e-05 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.311917+0000", 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024667062999999999 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.827 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2868 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'278, mlcod=264'275)", 2023-01-15T14:34:26.827 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:34:26.827 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 104.326428261, 2023-01-15T14:34:26.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31268252400000002, 2023-01-15T14:34:26.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:26.827 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:26.827 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:26.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:26.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:34:26.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:26.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:34:26.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:26.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993751+0000", 2023-01-15T14:34:26.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:26.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:26.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993762+0000", 2023-01-15T14:34:26.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0822000000000001e-05 2023-01-15T14:34:26.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:26.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993777+0000", 2023-01-15T14:34:26.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5540999999999999e-05 2023-01-15T14:34:26.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:26.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993787+0000", 2023-01-15T14:34:26.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0173999999999999e-05 2023-01-15T14:34:26.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290198+0000", 2023-01-15T14:34:26.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29641022099999997 2023-01-15T14:34:26.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:26.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290212+0000", 2023-01-15T14:34:26.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4214e-05 2023-01-15T14:34:26.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:26.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290300+0000", 2023-01-15T14:34:26.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8300000000000005e-05 2023-01-15T14:34:26.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:26.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290312+0000", 2023-01-15T14:34:26.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2344000000000001e-05 2023-01-15T14:34:26.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:26.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290357+0000", 2023-01-15T14:34:26.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4367999999999999e-05 2023-01-15T14:34:26.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:26.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290389+0000", 2023-01-15T14:34:26.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1983000000000002e-05 2023-01-15T14:34:26.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:26.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.306434+0000", 2023-01-15T14:34:26.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016045667 2023-01-15T14:34:26.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.852 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:26.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.853 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:34:26.853 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:34:26.853 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 104.326378836, 2023-01-15T14:34:26.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:34:26.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:26.854 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:26.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:26.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:26.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:34:26.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:26.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:34:26.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:26.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:26.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:34:26.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:26.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:26.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:34:26.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:34:26.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:26.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:26.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:34:26.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:26.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:34:26.990 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 104.32256714899999, 2023-01-15T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:27.005 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:34:27.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:34:27.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:34:27.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:27.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:34:27.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:34:27.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3120 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'290, mlcod=0'0)", 2023-01-15T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.080753873000006, 2023-01-15T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39853714000000001, 2023-01-15T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:34:27.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:27.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239424+0000", 2023-01-15T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:27.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239747+0000", 2023-01-15T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032243300000000001 2023-01-15T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239784+0000", 2023-01-15T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7051000000000002e-05 2023-01-15T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239802+0000", 2023-01-15T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8250999999999999e-05 2023-01-15T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536033+0000", 2023-01-15T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29623121499999999 2023-01-15T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:27.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536081+0000", 2023-01-15T14:34:27.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2023-01-15T14:34:27.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:27.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536800+0000", 2023-01-15T14:34:27.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071948499999999996 2023-01-15T14:34:27.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:27.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536817+0000", 2023-01-15T14:34:27.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6228000000000001e-05 2023-01-15T14:34:27.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:27.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537048+0000", 2023-01-15T14:34:27.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023136200000000001 2023-01-15T14:34:27.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:27.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637902+0000", 2023-01-15T14:34:27.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10085403900000001 2023-01-15T14:34:27.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:27.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637963+0000", 2023-01-15T14:34:27.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1310999999999996e-05 2023-01-15T14:34:27.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.144 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:27.144 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.144 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3121 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'291, mlcod=0'0)", 2023-01-15T14:34:27.145 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:34:27.145 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.078758356999998, 2023-01-15T14:34:27.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659573799999998, 2023-01-15T14:34:27.145 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:27.145 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:27.145 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:27.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:27.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:34:27.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:27.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:34:27.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241420+0000", 2023-01-15T14:34:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:27.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241854+0000", 2023-01-15T14:34:27.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043330099999999998 2023-01-15T14:34:27.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:27.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241878+0000", 2023-01-15T14:34:27.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4559e-05 2023-01-15T14:34:27.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:27.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241892+0000", 2023-01-15T14:34:27.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039999999999999e-05 2023-01-15T14:34:27.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:27.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536863+0000", 2023-01-15T14:34:27.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29497073400000001 2023-01-15T14:34:27.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:27.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536887+0000", 2023-01-15T14:34:27.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3683999999999999e-05 2023-01-15T14:34:27.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:27.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537124+0000", 2023-01-15T14:34:27.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023729900000000001 2023-01-15T14:34:27.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:27.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537136+0000", 2023-01-15T14:34:27.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2278e-05 2023-01-15T14:34:27.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:27.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537345+0000", 2023-01-15T14:34:27.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020865200000000001 2023-01-15T14:34:27.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:27.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637981+0000", 2023-01-15T14:34:27.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063575700000001 2023-01-15T14:34:27.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:27.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638018+0000", 2023-01-15T14:34:27.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6826999999999998e-05 2023-01-15T14:34:27.183 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.183 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:27.184 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.184 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3122 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'292, mlcod=0'0)", 2023-01-15T14:34:27.304 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:34:27.304 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.076862516999995, 2023-01-15T14:34:27.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39623551600000001, 2023-01-15T14:34:27.304 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:27.304 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:27.304 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:27.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:27.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:34:27.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:27.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:34:27.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:27.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243316+0000", 2023-01-15T14:34:27.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:27.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:27.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243833+0000", 2023-01-15T14:34:27.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051703600000000001 2023-01-15T14:34:27.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243863+0000", 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243879+0000", 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6263999999999999e-05 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637317+0000", 2023-01-15T14:34:27.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39343783199999999 2023-01-15T14:34:27.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:27.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637356+0000", 2023-01-15T14:34:27.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9579000000000002e-05 2023-01-15T14:34:27.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637830+0000", 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047326399999999999 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637885+0000", 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5074999999999997e-05 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:27.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638185+0000", 2023-01-15T14:34:27.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029993799999999997 2023-01-15T14:34:27.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:27.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638337+0000", 2023-01-15T14:34:27.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015208299999999999 2023-01-15T14:34:27.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:27.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.639553+0000", 2023-01-15T14:34:27.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001216389 2023-01-15T14:34:27.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.328 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:27.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.328 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3123 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'293, mlcod=0'0)", 2023-01-15T14:34:27.329 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:34:27.329 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.075118489000005, 2023-01-15T14:34:27.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.395733102, 2023-01-15T14:34:27.329 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:27.329 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:27.329 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:27.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:27.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:34:27.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:27.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:34:27.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:27.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245060+0000", 2023-01-15T14:34:27.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:27.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:27.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245075+0000", 2023-01-15T14:34:27.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4459e-05 2023-01-15T14:34:27.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:27.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245107+0000", 2023-01-15T14:34:27.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1795000000000003e-05 2023-01-15T14:34:27.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:27.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245121+0000", 2023-01-15T14:34:27.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4117000000000001e-05 2023-01-15T14:34:27.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638063+0000", 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39294204300000002 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638097+0000", 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4385999999999999e-05 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638299+0000", 2023-01-15T14:34:27.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020133099999999999 2023-01-15T14:34:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638318+0000", 2023-01-15T14:34:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514999999999999e-05 2023-01-15T14:34:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:27.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638386+0000", 2023-01-15T14:34:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8089000000000005e-05 2023-01-15T14:34:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638430+0000", 2023-01-15T14:34:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.439e-05 2023-01-15T14:34:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:27.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.640795+0000", 2023-01-15T14:34:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002364274 2023-01-15T14:34:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.336 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:27.336 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:27.337 INFO:tasks.rados.rados.0.smithi090.stdout:1419: expect (ObjNum 402 snap 108 seq_num 402) 2023-01-15T14:34:27.338 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:34:27.338 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:25.367+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:27.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:34:27.340 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:34:27.341 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:34:27.343 INFO:tasks.rados.rados.0.smithi090.stdout:1416: finishing write tid 1 to smithi09025264-2 2023-01-15T14:34:27.343 INFO:tasks.rados.rados.0.smithi090.stdout:1418: finishing write tid 1 to smithi09025264-35 2023-01-15T14:34:27.343 INFO:tasks.rados.rados.0.smithi090.stdout:1418: finishing write tid 2 to smithi09025264-35 2023-01-15T14:34:27.343 INFO:tasks.rados.rados.0.smithi090.stdout:1418: finishing write tid 3 to smithi09025264-35 2023-01-15T14:34:27.343 INFO:tasks.rados.rados.0.smithi090.stdout:1418: finishing write tid 4 to smithi09025264-35 2023-01-15T14:34:27.344 INFO:tasks.rados.rados.0.smithi090.stdout:1418: oid 35 updating version 0 to 334 2023-01-15T14:34:27.344 INFO:tasks.rados.rados.0.smithi090.stdout:1418: oid 35 version 334 is already newer than 332 2023-01-15T14:34:27.344 INFO:tasks.rados.rados.0.smithi090.stdout:1418: oid 35 version 334 is already newer than 333 2023-01-15T14:34:27.344 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 334 (ObjNum 510 snap 151 seq_num 510) dirty exists 2023-01-15T14:34:27.344 INFO:tasks.rados.rados.0.smithi090.stdout:1418: left oid 35 (ObjNum 510 snap 151 seq_num 510) 2023-01-15T14:34:27.344 INFO:tasks.rados.rados.0.smithi090.stdout:1416: finishing write tid 2 to smithi09025264-2 2023-01-15T14:34:27.344 INFO:tasks.rados.rados.0.smithi090.stdout:1416: finishing write tid 3 to smithi09025264-2 2023-01-15T14:34:27.344 INFO:tasks.rados.rados.0.smithi090.stdout:1414: done (5 left) 2023-01-15T14:34:27.344 INFO:tasks.rados.rados.0.smithi090.stdout:1417: done (4 left) 2023-01-15T14:34:27.344 INFO:tasks.rados.rados.0.smithi090.stdout:1418: done (3 left) 2023-01-15T14:34:27.345 INFO:tasks.rados.rados.0.smithi090.stdout:1419: done (2 left) 2023-01-15T14:34:27.345 INFO:tasks.rados.rados.0.smithi090.stdout:1422: delete oid 6 current snap is 151 2023-01-15T14:34:27.345 INFO:tasks.rados.rados.0.smithi090.stdout:1416: finishing write tid 4 to smithi09025264-2 2023-01-15T14:34:27.345 INFO:tasks.rados.rados.0.smithi090.stdout:1416: oid 2 updating version 0 to 384 2023-01-15T14:34:27.345 INFO:tasks.rados.rados.0.smithi090.stdout:1416: oid 2 updating version 384 to 385 2023-01-15T14:34:27.345 INFO:tasks.rados.rados.0.smithi090.stdout:1416: oid 2 updating version 385 to 386 2023-01-15T14:34:27.345 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 386 (ObjNum 509 snap 151 seq_num 509) dirty exists 2023-01-15T14:34:27.345 INFO:tasks.rados.rados.0.smithi090.stdout:1416: left oid 2 (ObjNum 509 snap 151 seq_num 509) 2023-01-15T14:34:27.345 INFO:tasks.rados.rados.0.smithi090.stdout:1421: finishing copy_from to smithi09025264-15 2023-01-15T14:34:27.345 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 15 v 263 (ObjNum 404 snap 108 seq_num 404) dirty exists 2023-01-15T14:34:27.345 INFO:tasks.rados.rados.0.smithi090.stdout:1421: finishing copy_from racing read to smithi09025264-15 2023-01-15T14:34:27.346 INFO:tasks.rados.rados.0.smithi090.stdout:1416: done (2 left) 2023-01-15T14:34:27.346 INFO:tasks.rados.rados.0.smithi090.stdout:1421: done (1 left) 2023-01-15T14:34:27.346 INFO:tasks.rados.rados.0.smithi090.stdout:1422: done (0 left) 2023-01-15T14:34:27.346 INFO:tasks.rados.rados.0.smithi090.stdout:1423: rollback oid 35 current snap is 151 2023-01-15T14:34:27.346 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 35 to 126 2023-01-15T14:34:27.346 INFO:tasks.rados.rados.0.smithi090.stdout:1424: snap_remove snap 150 2023-01-15T14:34:27.346 INFO:tasks.rados.rados.0.smithi090.stdout:1424: done (1 left) 2023-01-15T14:34:27.346 INFO:tasks.rados.rados.0.smithi090.stdout:1425: read oid 28 snap -1 2023-01-15T14:34:27.346 INFO:tasks.rados.rados.0.smithi090.stdout:1425: expect (ObjNum 466 snap 134 seq_num 466) 2023-01-15T14:34:27.346 INFO:tasks.rados.rados.0.smithi090.stdout:1426: delete oid 8 current snap is 151 2023-01-15T14:34:27.347 INFO:tasks.rados.rados.0.smithi090.stdout:1423: finishing rollback tid 0 to smithi09025264-35 2023-01-15T14:34:27.347 INFO:tasks.rados.rados.0.smithi090.stdout:1423: finishing rollback tid 1 to smithi09025264-35 2023-01-15T14:34:27.347 INFO:tasks.rados.rados.0.smithi090.stdout:1423: finishing rollback tid 2 to smithi09025264-35 2023-01-15T14:34:27.347 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 35 v 337 (ObjNum 415 snap 112 seq_num 415) dirty exists 2023-01-15T14:34:27.347 INFO:tasks.rados.rados.0.smithi090.stdout:1423: done (2 left) 2023-01-15T14:34:27.347 INFO:tasks.rados.rados.0.smithi090.stdout:1426: done (1 left) 2023-01-15T14:34:27.347 INFO:tasks.rados.rados.0.smithi090.stdout:1427: rollback oid 45 current snap is 151 2023-01-15T14:34:27.347 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 45 to 119 2023-01-15T14:34:27.347 INFO:tasks.rados.rados.0.smithi090.stdout:1428: copy_from oid 16 from oid 27 current snap is 151 2023-01-15T14:34:27.347 INFO:tasks.rados.rados.0.smithi090.stdout:1429: snap_remove snap 144 2023-01-15T14:34:27.347 INFO:tasks.rados.rados.0.smithi090.stdout:1429: done (3 left) 2023-01-15T14:34:27.348 INFO:tasks.rados.rados.0.smithi090.stdout:1430: read oid 22 snap -1 2023-01-15T14:34:27.348 INFO:tasks.rados.rados.0.smithi090.stdout:1430: expect (ObjNum 471 snap 136 seq_num 471) 2023-01-15T14:34:27.348 INFO:tasks.rados.rados.0.smithi090.stdout:1431: snap_create 2023-01-15T14:34:27.348 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:34:27.348 INFO:tasks.rados.rados.0.smithi090.stdout:1425: expect (ObjNum 466 snap 134 seq_num 466) 2023-01-15T14:34:27.348 INFO:tasks.rados.rados.0.smithi090.stdout:1428: finishing copy_from to smithi09025264-16 2023-01-15T14:34:27.348 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 269 (ObjNum 400 snap 106 seq_num 400) dirty exists 2023-01-15T14:34:27.349 INFO:tasks.rados.rados.0.smithi090.stdout:1428: finishing copy_from racing read to smithi09025264-16 2023-01-15T14:34:27.349 INFO:tasks.rados.rados.0.smithi090.stdout:1427: finishing rollback tid 0 to smithi09025264-45 2023-01-15T14:34:27.349 INFO:tasks.rados.rados.0.smithi090.stdout:1427: finishing rollback tid 1 to smithi09025264-45 2023-01-15T14:34:27.349 INFO:tasks.rados.rados.0.smithi090.stdout:1427: finishing rollback tid 2 to smithi09025264-45 2023-01-15T14:34:27.349 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 152 (ObjNum 400 snap 106 seq_num 400) dirty exists 2023-01-15T14:34:27.349 INFO:tasks.rados.rados.0.smithi090.stdout:1430: expect (ObjNum 471 snap 136 seq_num 471) 2023-01-15T14:34:27.350 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:27.335+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:27.440 INFO:tasks.rados.rados.0.smithi090.stdout:1425: done (3 left) 2023-01-15T14:34:27.689 INFO:tasks.rados.rados.0.smithi090.stdout:1427: done (2 left) 2023-01-15T14:34:27.690 INFO:tasks.rados.rados.0.smithi090.stdout:1428: done (1 left) 2023-01-15T14:34:27.690 INFO:tasks.rados.rados.0.smithi090.stdout:1430: done (0 left) 2023-01-15T14:34:27.691 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:27.691 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:34:27.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.691 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(osd.3.0:905 3.38 e377/232)", 2023-01-15T14:34:27.691 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:27.415083+0000", 2023-01-15T14:34:27.691 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.076032321, 2023-01-15T14:34:27.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.076051705999999997, 2023-01-15T14:34:27.691 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:27.692 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:34:27.692 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:27.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:27.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:27.415083+0000", 2023-01-15T14:34:27.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:27.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:27.415083+0000", 2023-01-15T14:34:27.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:27.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:27.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:27.415081+0000", 2023-01-15T14:34:27.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:27.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:27.415093+0000", 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1178999999999999e-05 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:27.415107+0000", 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3986e-05 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:27.415116+0000", 2023-01-15T14:34:27.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7200000000000001e-06 2023-01-15T14:34:27.695 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.695 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:27.695 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.695 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:27.695 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:34:27.695 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:34:27.695 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:27.695 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:27.697 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:34:27.705 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:27.705 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:27.705 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:27.705 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:27.706 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:27.706 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:27.706 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:34:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:34:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:34:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:34:27.707 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:34:27.912 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:34:27.912 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:34:27.912 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:34:27.917 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:27.919 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:34:27.919 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":560,"stamp":"2023-01-15T14:34:26.955643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":380121358,"num_objects":177,"num_object_clones":122,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":15,"num_read":1098,"num_read_kb":1070478,"num_write":928,"num_write_kb":321972,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":436785683,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18958,"ondisk_log_size":18958,"up":345,"acting":345,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":365,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6306156,"kb_used_data":555914600,"kb_used_omap":70257,"kb_used_meta":0,"kb_avail":555890124,"statfs":{"total":575688990720,"available":569231486976,"internally_reserved":25063424,"allocated":569256550400,"data_stored":569256550400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":71943389,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":15,"commit_latency_ns":0,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4475630,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":-37,"num_read_kb":-33526,"num_write":-32,"num_write_kb":-9982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-32,"num_bytes_recovered":-23158008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.916050"},"pg_stats":[{"pgid":"3.69","version":"110'149","reported_seq":924,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.914372+0000","last_change":"2023-01-15T14:34:20.016599+0000","last_active":"2023-01-15T14:34:21.914372+0000","last_peered":"2023-01-15T14:34:21.914372+0000","last_clean":"2023-01-15T14:34:21.914372+0000","last_became_active":"2023-01-15T14:33:50.216417+0000","last_became_peered":"2023-01-15T14:33:50.216417+0000","last_unstale":"2023-01-15T14:34:21.914372+0000","last_undegraded":"2023-01-15T14:34:21.914372+0000","last_fullsized":"2023-01-15T14:34:21.914372+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.002032421,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.68","version":"372'259","reported_seq":1460,"reported_epoch":376,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:26.577601+0000","last_change":"2023-01-15T14:34:26.577668+0000","last_active":"2023-01-15T14:34:26.577601+0000","last_peered":"2023-01-15T14:34:26.577601+0000","last_clean":"2023-01-15T14:34:26.577601+0000","last_became_active":"2023-01-15T14:34:13.134258+0000","last_became_peered":"2023-01-15T14:34:13.134258+0000","last_unstale":"2023-01-15T14:34:26.577601+0000","last_undegraded":"2023-01-15T14:34:26.577601+0000","last_fullsized":"2023-01-15T14:34:26.577601+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.102953804,"stat_sum":{"num_bytes":8014979,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17027823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.67","version":"191'79","reported_seq":74,"reported_epoch":371,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:21.232692+0000","last_change":"2023-01-15T14:34:20.218584+0000","last_active":"2023-01-15T14:34:21.232692+0000","last_peered":"2023-01-15T14:34:21.232692+0000","last_clean":"2023-01-15T14:34:21.232692+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:34:21.232692+0000","last_undegraded":"2023-01-15T14:34:21.232692+0000","last_fullsized":"2023-01-15T14:34:21.232692+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.40508863699999997,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.66","version":"102'52","reported_seq":387,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985234+0000","last_change":"2023-01-15T14:34:20.334895+0000","last_active":"2023-01-15T14:34:21.985234+0000","last_peered":"2023-01-15T14:34:21.985234+0000","last_clean":"2023-01-15T14:34:21.985234+0000","last_became_active":"2023-01-15T14:33:48.695009+0000","last_became_peered":"2023-01-15T14:33:48.695009+0000","last_unstale":"2023-01-15T14:34:21.985234+0000","last_undegraded":"2023-01-15T14:34:21.985234+0000","last_fullsized":"2023-01-15T14:34:21.985234+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.30362193900000001,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2142,"num_write":4,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.65","version":"371'126","reported_seq":1240,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.081008+0000","last_change":"2023-01-15T14:34:22.081796+0000","last_active":"2023-01-15T14:34:22.081008+0000","last_peered":"2023-01-15T14:34:22.081008+0000","last_clean":"2023-01-15T14:34:22.081008+0000","last_became_active":"2023-01-15T14:34:16.276292+0000","last_became_peered":"2023-01-15T14:34:16.276292+0000","last_unstale":"2023-01-15T14:34:22.081008+0000","last_undegraded":"2023-01-15T14:34:22.081008+0000","last_fullsized":"2023-01-15T14:34:22.081008+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.019369396000000001,"stat_sum":{"num_bytes":7224063,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":8,"num_read_kb":3033,"num_write":12,"num_write_kb":4201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":20989326,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.64","version":"334'251","reported_seq":1443,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162784+0000","last_change":"2023-01-15T14:34:25.162761+0000","last_active":"2023-01-15T14:34:25.162784+0000","last_peered":"2023-01-15T14:34:25.162784+0000","last_clean":"2023-01-15T14:34:25.162784+0000","last_became_active":"2023-01-15T14:33:55.675085+0000","last_became_peered":"2023-01-15T14:33:55.675085+0000","last_unstale":"2023-01-15T14:34:25.162784+0000","last_undegraded":"2023-01-15T14:34:25.162784+0000","last_fullsized":"2023-01-15T14:34:25.162784+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.91927382199999996,"stat_sum":{"num_bytes":3548733,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10773,"num_write":5,"num_write_kb":2741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.63","version":"149'233","reported_seq":772,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.063759+0000","last_change":"2023-01-15T14:34:25.063740+0000","last_active":"2023-01-15T14:34:25.063759+0000","last_peered":"2023-01-15T14:34:25.063759+0000","last_clean":"2023-01-15T14:34:25.063759+0000","last_became_active":"2023-01-15T14:33:55.277739+0000","last_became_peered":"2023-01-15T14:33:55.277739+0000","last_unstale":"2023-01-15T14:34:25.063759+0000","last_undegraded":"2023-01-15T14:34:25.063759+0000","last_fullsized":"2023-01-15T14:34:25.063759+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.01371225,"stat_sum":{"num_bytes":3802136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":21819,"num_write":19,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.62","version":"370'171","reported_seq":1372,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.021100+0000","last_change":"2023-01-15T14:34:20.630780+0000","last_active":"2023-01-15T14:34:21.021100+0000","last_peered":"2023-01-15T14:34:21.021100+0000","last_clean":"2023-01-15T14:34:21.021100+0000","last_became_active":"2023-01-15T14:33:56.590183+0000","last_became_peered":"2023-01-15T14:33:56.590183+0000","last_unstale":"2023-01-15T14:34:21.021100+0000","last_undegraded":"2023-01-15T14:34:21.021100+0000","last_fullsized":"2023-01-15T14:34:21.021100+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.61771345899999996,"stat_sum":{"num_bytes":5361549,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":6734,"num_write":11,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5210269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.61","version":"375'148","reported_seq":1043,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.473779+0000","last_change":"2023-01-15T14:34:26.473873+0000","last_active":"2023-01-15T14:34:26.473779+0000","last_peered":"2023-01-15T14:34:26.473779+0000","last_clean":"2023-01-15T14:34:26.473779+0000","last_became_active":"2023-01-15T14:33:53.421056+0000","last_became_peered":"2023-01-15T14:33:53.421056+0000","last_unstale":"2023-01-15T14:34:26.473779+0000","last_undegraded":"2023-01-15T14:34:26.473779+0000","last_fullsized":"2023-01-15T14:34:26.473779+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.301418146,"stat_sum":{"num_bytes":5113062,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":6060,"num_write":6,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.60","version":"370'283","reported_seq":1438,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.223355+0000","last_change":"2023-01-15T14:34:21.223355+0000","last_active":"2023-01-15T14:34:21.223355+0000","last_peered":"2023-01-15T14:34:21.223355+0000","last_clean":"2023-01-15T14:34:21.223355+0000","last_became_active":"2023-01-15T14:33:56.478613+0000","last_became_peered":"2023-01-15T14:33:56.478613+0000","last_unstale":"2023-01-15T14:34:21.223355+0000","last_undegraded":"2023-01-15T14:34:21.223355+0000","last_fullsized":"2023-01-15T14:34:21.223355+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.102275617,"stat_sum":{"num_bytes":5214108,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":19437,"num_write":14,"num_write_kb":5082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15418818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1406,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.474713+0000","last_change":"2023-01-15T14:34:26.474781+0000","last_active":"2023-01-15T14:34:26.474713+0000","last_peered":"2023-01-15T14:34:26.474713+0000","last_clean":"2023-01-15T14:34:26.474713+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:34:26.474713+0000","last_undegraded":"2023-01-15T14:34:26.474713+0000","last_fullsized":"2023-01-15T14:34:26.474713+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:51.919","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.100394097,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5e","version":"215'39","reported_seq":1003,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.916320+0000","last_change":"2023-01-15T14:34:20.119902+0000","last_active":"2023-01-15T14:34:21.916320+0000","last_peered":"2023-01-15T14:34:21.916320+0000","last_clean":"2023-01-15T14:34:21.916320+0000","last_became_active":"2023-01-15T14:33:35.364143+0000","last_became_peered":"2023-01-15T14:33:35.364143+0000","last_unstale":"2023-01-15T14:34:21.916320+0000","last_undegraded":"2023-01-15T14:34:21.916320+0000","last_fullsized":"2023-01-15T14:34:21.916320+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.103481502,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1342,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984976+0000","last_change":"2023-01-15T14:34:20.336111+0000","last_active":"2023-01-15T14:34:21.984976+0000","last_peered":"2023-01-15T14:34:21.984976+0000","last_clean":"2023-01-15T14:34:21.984976+0000","last_became_active":"2023-01-15T14:34:00.242122+0000","last_became_peered":"2023-01-15T14:34:00.242122+0000","last_unstale":"2023-01-15T14:34:21.984976+0000","last_undegraded":"2023-01-15T14:34:21.984976+0000","last_fullsized":"2023-01-15T14:34:21.984976+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:19.729021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:19.729021+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:29.740","scrub_duration":0.50499948100000003,"objects_trimmed":0,"snaptrim_duration":0.30454418900000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5c","version":"141'46","reported_seq":104,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.915404+0000","last_change":"2023-01-15T14:34:20.120026+0000","last_active":"2023-01-15T14:34:21.915404+0000","last_peered":"2023-01-15T14:34:21.915404+0000","last_clean":"2023-01-15T14:34:21.915404+0000","last_became_active":"2023-01-15T14:33:57.113032+0000","last_became_peered":"2023-01-15T14:33:57.113032+0000","last_unstale":"2023-01-15T14:34:21.915404+0000","last_undegraded":"2023-01-15T14:34:21.915404+0000","last_fullsized":"2023-01-15T14:34:21.915404+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.103747292,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5b","version":"283'80","reported_seq":934,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.785391+0000","last_change":"2023-01-15T14:34:22.785757+0000","last_active":"2023-01-15T14:34:22.785391+0000","last_peered":"2023-01-15T14:34:22.785391+0000","last_clean":"2023-01-15T14:34:22.785391+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:34:22.785391+0000","last_undegraded":"2023-01-15T14:34:22.785391+0000","last_fullsized":"2023-01-15T14:34:22.785391+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.61437087700000004,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5a","version":"374'352","reported_seq":1520,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162591+0000","last_change":"2023-01-15T14:34:25.162531+0000","last_active":"2023-01-15T14:34:25.162591+0000","last_peered":"2023-01-15T14:34:25.162591+0000","last_clean":"2023-01-15T14:34:25.162591+0000","last_became_active":"2023-01-15T14:34:11.050316+0000","last_became_peered":"2023-01-15T14:34:11.050316+0000","last_unstale":"2023-01-15T14:34:25.162591+0000","last_undegraded":"2023-01-15T14:34:25.162591+0000","last_fullsized":"2023-01-15T14:34:25.162591+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:05.853","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.71046558199999998,"stat_sum":{"num_bytes":5983424,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":19294,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.59","version":"294'256","reported_seq":1301,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984171+0000","last_change":"2023-01-15T14:34:20.333913+0000","last_active":"2023-01-15T14:34:21.984171+0000","last_peered":"2023-01-15T14:34:21.984171+0000","last_clean":"2023-01-15T14:34:21.984171+0000","last_became_active":"2023-01-15T14:33:55.671990+0000","last_became_peered":"2023-01-15T14:33:55.671990+0000","last_unstale":"2023-01-15T14:34:21.984171+0000","last_undegraded":"2023-01-15T14:34:21.984171+0000","last_fullsized":"2023-01-15T14:34:21.984171+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.30268093299999999,"stat_sum":{"num_bytes":1210137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":5,"num_read_kb":3549,"num_write":5,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.58","version":"323'320","reported_seq":1503,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.231223+0000","last_change":"2023-01-15T14:34:21.042215+0000","last_active":"2023-01-15T14:34:21.231223+0000","last_peered":"2023-01-15T14:34:21.231223+0000","last_clean":"2023-01-15T14:34:21.231223+0000","last_became_active":"2023-01-15T14:33:40.127669+0000","last_became_peered":"2023-01-15T14:33:40.127669+0000","last_unstale":"2023-01-15T14:34:21.231223+0000","last_undegraded":"2023-01-15T14:34:21.231223+0000","last_fullsized":"2023-01-15T14:34:21.231223+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.92497239099999995,"stat_sum":{"num_bytes":4332079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.57","version":"375'102","reported_seq":143,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.547916+0000","last_change":"2023-01-15T14:34:26.548062+0000","last_active":"2023-01-15T14:34:26.547916+0000","last_peered":"2023-01-15T14:34:26.547916+0000","last_clean":"2023-01-15T14:34:26.547916+0000","last_became_active":"2023-01-15T14:33:36.994447+0000","last_became_peered":"2023-01-15T14:33:36.994447+0000","last_unstale":"2023-01-15T14:34:26.547916+0000","last_undegraded":"2023-01-15T14:34:26.547916+0000","last_fullsized":"2023-01-15T14:34:26.547916+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'100","last_scrub_stamp":"2023-01-15T14:34:23.447852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.447852+0000","objects_scrubbed":2,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:30.799","scrub_duration":0.52101476099999999,"objects_trimmed":0,"snaptrim_duration":0.17314665600000001,"stat_sum":{"num_bytes":3025326,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.56","version":"320'127","reported_seq":653,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.783640+0000","last_change":"2023-01-15T14:34:22.783941+0000","last_active":"2023-01-15T14:34:22.783640+0000","last_peered":"2023-01-15T14:34:22.783640+0000","last_clean":"2023-01-15T14:34:22.783640+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:34:22.783640+0000","last_undegraded":"2023-01-15T14:34:22.783640+0000","last_fullsized":"2023-01-15T14:34:22.783640+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.81242844000000003,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.55","version":"110'55","reported_seq":1034,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.015630+0000","last_change":"2023-01-15T14:34:20.226765+0000","last_active":"2023-01-15T14:34:22.015630+0000","last_peered":"2023-01-15T14:34:22.015630+0000","last_clean":"2023-01-15T14:34:22.015630+0000","last_became_active":"2023-01-15T14:33:52.510429+0000","last_became_peered":"2023-01-15T14:33:52.510429+0000","last_unstale":"2023-01-15T14:34:22.015630+0000","last_undegraded":"2023-01-15T14:34:22.015630+0000","last_fullsized":"2023-01-15T14:34:22.015630+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.20908486400000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1162,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162579+0000","last_change":"2023-01-15T14:34:25.162553+0000","last_active":"2023-01-15T14:34:25.162579+0000","last_peered":"2023-01-15T14:34:25.162579+0000","last_clean":"2023-01-15T14:34:25.162579+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:34:25.162579+0000","last_undegraded":"2023-01-15T14:34:25.162579+0000","last_fullsized":"2023-01-15T14:34:25.162579+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:33:45.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:45.583005+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.91300059600000005,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.53","version":"370'402","reported_seq":1489,"reported_epoch":372,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:22.189075+0000","last_change":"2023-01-15T14:34:20.326582+0000","last_active":"2023-01-15T14:34:22.189075+0000","last_peered":"2023-01-15T14:34:22.189075+0000","last_clean":"2023-01-15T14:34:22.189075+0000","last_became_active":"2023-01-15T14:34:20.324151+0000","last_became_peered":"2023-01-15T14:34:20.324151+0000","last_unstale":"2023-01-15T14:34:22.189075+0000","last_undegraded":"2023-01-15T14:34:22.189075+0000","last_fullsized":"2023-01-15T14:34:22.189075+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:44.432","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.092650028999999995,"stat_sum":{"num_bytes":30096451,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":38,"num_read_kb":45487,"num_write":24,"num_write_kb":15073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":30103975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"109","length":"1"},{"start":"10b","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":753,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.886872+0000","last_change":"2023-01-15T14:34:22.887757+0000","last_active":"2023-01-15T14:34:22.886872+0000","last_peered":"2023-01-15T14:34:22.886872+0000","last_clean":"2023-01-15T14:34:22.886872+0000","last_became_active":"2023-01-15T14:33:55.472400+0000","last_became_peered":"2023-01-15T14:33:55.472400+0000","last_unstale":"2023-01-15T14:34:22.886872+0000","last_undegraded":"2023-01-15T14:34:22.886872+0000","last_fullsized":"2023-01-15T14:34:22.886872+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.91726039999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.51","version":"370'318","reported_seq":1544,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.319910+0000","last_change":"2023-01-15T14:34:21.319910+0000","last_active":"2023-01-15T14:34:21.319910+0000","last_peered":"2023-01-15T14:34:21.319910+0000","last_clean":"2023-01-15T14:34:21.319910+0000","last_became_active":"2023-01-15T14:33:26.107940+0000","last_became_peered":"2023-01-15T14:33:26.107940+0000","last_unstale":"2023-01-15T14:34:21.319910+0000","last_undegraded":"2023-01-15T14:34:21.319910+0000","last_fullsized":"2023-01-15T14:34:21.319910+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:36.458","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.29649700499999998,"stat_sum":{"num_bytes":7496813,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.50","version":"371'422","reported_seq":1607,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.249554+0000","last_change":"2023-01-15T14:34:22.350581+0000","last_active":"2023-01-15T14:34:22.249554+0000","last_peered":"2023-01-15T14:34:22.249554+0000","last_clean":"2023-01-15T14:34:22.249554+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:34:22.249554+0000","last_undegraded":"2023-01-15T14:34:22.249554+0000","last_fullsized":"2023-01-15T14:34:22.249554+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.38030728499999999,"stat_sum":{"num_bytes":9681416,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":52505,"num_write":49,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4f","version":"155'145","reported_seq":851,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.916914+0000","last_change":"2023-01-15T14:34:20.222818+0000","last_active":"2023-01-15T14:34:21.916914+0000","last_peered":"2023-01-15T14:34:21.916914+0000","last_clean":"2023-01-15T14:34:21.916914+0000","last_became_active":"2023-01-15T14:33:25.129936+0000","last_became_peered":"2023-01-15T14:33:25.129936+0000","last_unstale":"2023-01-15T14:34:21.916914+0000","last_undegraded":"2023-01-15T14:34:21.916914+0000","last_fullsized":"2023-01-15T14:34:21.916914+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20562384,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":130,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.062983+0000","last_change":"2023-01-15T14:34:25.062951+0000","last_active":"2023-01-15T14:34:25.062983+0000","last_peered":"2023-01-15T14:34:25.062983+0000","last_clean":"2023-01-15T14:34:25.062983+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:34:25.062983+0000","last_undegraded":"2023-01-15T14:34:25.062983+0000","last_fullsized":"2023-01-15T14:34:25.062983+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.92170574500000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.4d","version":"372'266","reported_seq":1384,"reported_epoch":372,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:22.385106+0000","last_change":"2023-01-15T14:34:20.217750+0000","last_active":"2023-01-15T14:34:22.385106+0000","last_peered":"2023-01-15T14:34:22.385106+0000","last_clean":"2023-01-15T14:34:22.385106+0000","last_became_active":"2023-01-15T14:34:05.178613+0000","last_became_peered":"2023-01-15T14:34:05.178613+0000","last_unstale":"2023-01-15T14:34:22.385106+0000","last_undegraded":"2023-01-15T14:34:22.385106+0000","last_fullsized":"2023-01-15T14:34:22.385106+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":1.4066433030000001,"stat_sum":{"num_bytes":14846966,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":15961,"num_write":20,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4c","version":"309'71","reported_seq":130,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.231837+0000","last_change":"2023-01-15T14:34:21.146480+0000","last_active":"2023-01-15T14:34:21.231837+0000","last_peered":"2023-01-15T14:34:21.231837+0000","last_clean":"2023-01-15T14:34:21.231837+0000","last_became_active":"2023-01-15T14:33:52.311859+0000","last_became_peered":"2023-01-15T14:33:52.311859+0000","last_unstale":"2023-01-15T14:34:21.231837+0000","last_undegraded":"2023-01-15T14:34:21.231837+0000","last_fullsized":"2023-01-15T14:34:21.231837+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":1.0291399059999999,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":776,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.320820+0000","last_change":"2023-01-15T14:34:21.320820+0000","last_active":"2023-01-15T14:34:21.320820+0000","last_peered":"2023-01-15T14:34:21.320820+0000","last_clean":"2023-01-15T14:34:21.320820+0000","last_became_active":"2023-01-15T14:33:52.406605+0000","last_became_peered":"2023-01-15T14:33:52.406605+0000","last_unstale":"2023-01-15T14:34:21.320820+0000","last_undegraded":"2023-01-15T14:34:21.320820+0000","last_fullsized":"2023-01-15T14:34:21.320820+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:42.594","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.19631784299999999,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4a","version":"370'375","reported_seq":1504,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.431870+0000","last_change":"2023-01-15T14:34:21.041621+0000","last_active":"2023-01-15T14:34:21.431870+0000","last_peered":"2023-01-15T14:34:21.431870+0000","last_clean":"2023-01-15T14:34:21.431870+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:34:21.431870+0000","last_undegraded":"2023-01-15T14:34:21.431870+0000","last_fullsized":"2023-01-15T14:34:21.431870+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:33.328","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.92452315299999999,"stat_sum":{"num_bytes":12470870,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":63,"num_read_kb":67536,"num_write":73,"num_write_kb":23626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":958,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.785869+0000","last_change":"2023-01-15T14:34:22.886695+0000","last_active":"2023-01-15T14:34:22.785869+0000","last_peered":"2023-01-15T14:34:22.785869+0000","last_clean":"2023-01-15T14:34:22.785869+0000","last_became_active":"2023-01-15T14:34:00.043221+0000","last_became_peered":"2023-01-15T14:34:00.043221+0000","last_unstale":"2023-01-15T14:34:22.785869+0000","last_undegraded":"2023-01-15T14:34:22.785869+0000","last_fullsized":"2023-01-15T14:34:22.785869+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.91642319400000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.48","version":"179'204","reported_seq":1382,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.912930+0000","last_change":"2023-01-15T14:34:20.227341+0000","last_active":"2023-01-15T14:34:21.912930+0000","last_peered":"2023-01-15T14:34:21.912930+0000","last_clean":"2023-01-15T14:34:21.912930+0000","last_became_active":"2023-01-15T14:33:55.375518+0000","last_became_peered":"2023-01-15T14:33:55.375518+0000","last_unstale":"2023-01-15T14:34:21.912930+0000","last_undegraded":"2023-01-15T14:34:21.912930+0000","last_fullsized":"2023-01-15T14:34:21.912930+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:43.662","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.11151976,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1155,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985389+0000","last_change":"2023-01-15T14:34:21.450418+0000","last_active":"2023-01-15T14:34:21.985389+0000","last_peered":"2023-01-15T14:34:21.985389+0000","last_clean":"2023-01-15T14:34:21.985389+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:34:21.985389+0000","last_undegraded":"2023-01-15T14:34:21.985389+0000","last_fullsized":"2023-01-15T14:34:21.985389+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:34:21.450320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:21.450320+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:50.200","scrub_duration":0.30497759099999999,"objects_trimmed":0,"snaptrim_duration":0.30380391600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":1052,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.326403+0000","last_change":"2023-01-15T14:34:21.326403+0000","last_active":"2023-01-15T14:34:21.326403+0000","last_peered":"2023-01-15T14:34:21.326403+0000","last_clean":"2023-01-15T14:34:21.326403+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:34:21.326403+0000","last_undegraded":"2023-01-15T14:34:21.326403+0000","last_fullsized":"2023-01-15T14:34:21.326403+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:55.245","scrub_duration":0.30448409700000001,"objects_trimmed":0,"snaptrim_duration":0.203275821,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":947,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.064135+0000","last_change":"2023-01-15T14:34:25.064135+0000","last_active":"2023-01-15T14:34:25.064135+0000","last_peered":"2023-01-15T14:34:25.064135+0000","last_clean":"2023-01-15T14:34:25.064135+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:34:25.064135+0000","last_undegraded":"2023-01-15T14:34:25.064135+0000","last_fullsized":"2023-01-15T14:34:25.064135+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090126099999999999,"stat_sum":{"num_bytes":2635976,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":19927,"num_write":15,"num_write_kb":6659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4745860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1297,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984753+0000","last_change":"2023-01-15T14:34:20.337679+0000","last_active":"2023-01-15T14:34:21.984753+0000","last_peered":"2023-01-15T14:34:21.984753+0000","last_clean":"2023-01-15T14:34:21.984753+0000","last_became_active":"2023-01-15T14:33:55.678019+0000","last_became_peered":"2023-01-15T14:33:55.678019+0000","last_unstale":"2023-01-15T14:34:21.984753+0000","last_undegraded":"2023-01-15T14:34:21.984753+0000","last_fullsized":"2023-01-15T14:34:21.984753+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:27.120","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.30552496200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.43","version":"374'322","reported_seq":1554,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.664339+0000","last_change":"2023-01-15T14:34:25.664339+0000","last_active":"2023-01-15T14:34:25.664339+0000","last_peered":"2023-01-15T14:34:25.664339+0000","last_clean":"2023-01-15T14:34:25.664339+0000","last_became_active":"2023-01-15T14:34:02.211232+0000","last_became_peered":"2023-01-15T14:34:02.211232+0000","last_unstale":"2023-01-15T14:34:25.664339+0000","last_undegraded":"2023-01-15T14:34:25.664339+0000","last_fullsized":"2023-01-15T14:34:25.664339+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.60018934099999999,"stat_sum":{"num_bytes":6319564,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":12,"num_read_kb":16585,"num_write":7,"num_write_kb":4207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3395080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":1058,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.115137+0000","last_change":"2023-01-15T14:34:20.118682+0000","last_active":"2023-01-15T14:34:22.115137+0000","last_peered":"2023-01-15T14:34:22.115137+0000","last_clean":"2023-01-15T14:34:22.115137+0000","last_became_active":"2023-01-15T14:33:55.376782+0000","last_became_peered":"2023-01-15T14:33:55.376782+0000","last_unstale":"2023-01-15T14:34:22.115137+0000","last_undegraded":"2023-01-15T14:34:22.115137+0000","last_fullsized":"2023-01-15T14:34:22.115137+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.103556779,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1138,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984413+0000","last_change":"2023-01-15T14:34:20.438567+0000","last_active":"2023-01-15T14:34:21.984413+0000","last_peered":"2023-01-15T14:34:21.984413+0000","last_clean":"2023-01-15T14:34:21.984413+0000","last_became_active":"2023-01-15T14:33:55.771969+0000","last_became_peered":"2023-01-15T14:33:55.771969+0000","last_unstale":"2023-01-15T14:34:21.984413+0000","last_undegraded":"2023-01-15T14:34:21.984413+0000","last_fullsized":"2023-01-15T14:34:21.984413+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.30972029499999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.40","version":"363'311","reported_seq":1475,"reported_epoch":372,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:22.887808+0000","last_change":"2023-01-15T14:34:22.887868+0000","last_active":"2023-01-15T14:34:22.887808+0000","last_peered":"2023-01-15T14:34:22.887808+0000","last_clean":"2023-01-15T14:34:22.887808+0000","last_became_active":"2023-01-15T14:33:58.117085+0000","last_became_peered":"2023-01-15T14:33:58.117085+0000","last_unstale":"2023-01-15T14:34:22.887808+0000","last_undegraded":"2023-01-15T14:34:22.887808+0000","last_fullsized":"2023-01-15T14:34:22.887808+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":1.0057765489999999,"stat_sum":{"num_bytes":12753904,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":35541,"num_write":28,"num_write_kb":11001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":29167491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3f","version":"321'233","reported_seq":1390,"reported_epoch":371,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:21.332941+0000","last_change":"2023-01-15T14:34:20.219334+0000","last_active":"2023-01-15T14:34:21.332941+0000","last_peered":"2023-01-15T14:34:21.332941+0000","last_clean":"2023-01-15T14:34:21.332941+0000","last_became_active":"2023-01-15T14:33:58.217783+0000","last_became_peered":"2023-01-15T14:33:58.217783+0000","last_unstale":"2023-01-15T14:34:21.332941+0000","last_undegraded":"2023-01-15T14:34:21.332941+0000","last_fullsized":"2023-01-15T14:34:21.332941+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:28.180","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.014066018,"stat_sum":{"num_bytes":4764997,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6605283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3e","version":"294'63","reported_seq":858,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.422162+0000","last_change":"2023-01-15T14:34:21.422162+0000","last_active":"2023-01-15T14:34:21.422162+0000","last_peered":"2023-01-15T14:34:21.422162+0000","last_clean":"2023-01-15T14:34:21.422162+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:34:21.422162+0000","last_undegraded":"2023-01-15T14:34:21.422162+0000","last_fullsized":"2023-01-15T14:34:21.422162+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.399996725,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3d","version":"371'238","reported_seq":1363,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162883+0000","last_change":"2023-01-15T14:34:25.162856+0000","last_active":"2023-01-15T14:34:25.162883+0000","last_peered":"2023-01-15T14:34:25.162883+0000","last_clean":"2023-01-15T14:34:25.162883+0000","last_became_active":"2023-01-15T14:34:05.187121+0000","last_became_peered":"2023-01-15T14:34:05.187121+0000","last_unstale":"2023-01-15T14:34:25.162883+0000","last_undegraded":"2023-01-15T14:34:25.162883+0000","last_fullsized":"2023-01-15T14:34:25.162883+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.92019914000000003,"stat_sum":{"num_bytes":8050279,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":13797,"num_write":16,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":8050279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":374,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.915658+0000","last_change":"2023-01-15T14:34:20.221556+0000","last_active":"2023-01-15T14:34:21.915658+0000","last_peered":"2023-01-15T14:34:21.915658+0000","last_clean":"2023-01-15T14:34:21.915658+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:34:21.915658+0000","last_undegraded":"2023-01-15T14:34:21.915658+0000","last_fullsized":"2023-01-15T14:34:21.915658+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.470","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.205049071,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":48,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.784726+0000","last_change":"2023-01-15T14:34:22.785231+0000","last_active":"2023-01-15T14:34:22.784726+0000","last_peered":"2023-01-15T14:34:22.784726+0000","last_clean":"2023-01-15T14:34:22.784726+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:34:22.784726+0000","last_undegraded":"2023-01-15T14:34:22.784726+0000","last_fullsized":"2023-01-15T14:34:22.784726+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.61382743699999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1381,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.062202+0000","last_change":"2023-01-15T14:34:25.062182+0000","last_active":"2023-01-15T14:34:25.062202+0000","last_peered":"2023-01-15T14:34:25.062202+0000","last_clean":"2023-01-15T14:34:25.062202+0000","last_became_active":"2023-01-15T14:33:55.474944+0000","last_became_peered":"2023-01-15T14:33:55.474944+0000","last_unstale":"2023-01-15T14:34:25.062202+0000","last_undegraded":"2023-01-15T14:34:25.062202+0000","last_fullsized":"2023-01-15T14:34:25.062202+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.104032045,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.39","version":"371'365","reported_seq":1522,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.961648+0000","last_change":"2023-01-15T14:34:21.545978+0000","last_active":"2023-01-15T14:34:21.961648+0000","last_peered":"2023-01-15T14:34:21.961648+0000","last_clean":"2023-01-15T14:34:21.961648+0000","last_became_active":"2023-01-15T14:34:08.023332+0000","last_became_peered":"2023-01-15T14:34:08.023332+0000","last_unstale":"2023-01-15T14:34:21.961648+0000","last_undegraded":"2023-01-15T14:34:21.961648+0000","last_fullsized":"2023-01-15T14:34:21.961648+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:28.106","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.31467561799999999,"stat_sum":{"num_bytes":25374153,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":39,"num_read_kb":66942,"num_write":11,"num_write_kb":8612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":26862613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.38","version":"370'317","reported_seq":1560,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.913299+0000","last_change":"2023-01-15T14:34:21.021163+0000","last_active":"2023-01-15T14:34:21.913299+0000","last_peered":"2023-01-15T14:34:21.913299+0000","last_clean":"2023-01-15T14:34:21.913299+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:34:21.913299+0000","last_undegraded":"2023-01-15T14:34:21.913299+0000","last_fullsized":"2023-01-15T14:34:21.913299+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.90503079399999997,"stat_sum":{"num_bytes":8461433,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":42848,"num_write":39,"num_write_kb":13455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":431,"reported_epoch":371,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:21.233231+0000","last_change":"2023-01-15T14:34:20.219159+0000","last_active":"2023-01-15T14:34:21.233231+0000","last_peered":"2023-01-15T14:34:21.233231+0000","last_clean":"2023-01-15T14:34:21.233231+0000","last_became_active":"2023-01-15T14:33:52.412128+0000","last_became_peered":"2023-01-15T14:33:52.412128+0000","last_unstale":"2023-01-15T14:34:21.233231+0000","last_undegraded":"2023-01-15T14:34:21.233231+0000","last_fullsized":"2023-01-15T14:34:21.233231+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.207923046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":586,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.325513+0000","last_change":"2023-01-15T14:34:21.325513+0000","last_active":"2023-01-15T14:34:21.325513+0000","last_peered":"2023-01-15T14:34:21.325513+0000","last_clean":"2023-01-15T14:34:21.325513+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:34:21.325513+0000","last_undegraded":"2023-01-15T14:34:21.325513+0000","last_fullsized":"2023-01-15T14:34:21.325513+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.30313143199999998,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":70,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.062912+0000","last_change":"2023-01-15T14:34:25.062878+0000","last_active":"2023-01-15T14:34:25.062912+0000","last_peered":"2023-01-15T14:34:25.062912+0000","last_clean":"2023-01-15T14:34:25.062912+0000","last_became_active":"2023-01-15T14:33:52.612576+0000","last_became_peered":"2023-01-15T14:33:52.612576+0000","last_unstale":"2023-01-15T14:34:25.062912+0000","last_undegraded":"2023-01-15T14:34:25.062912+0000","last_fullsized":"2023-01-15T14:34:25.062912+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.92285126200000001,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":1182,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985051+0000","last_change":"2023-01-15T14:34:20.836036+0000","last_active":"2023-01-15T14:34:21.985051+0000","last_peered":"2023-01-15T14:34:21.985051+0000","last_clean":"2023-01-15T14:34:21.985051+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:34:21.985051+0000","last_undegraded":"2023-01-15T14:34:21.985051+0000","last_fullsized":"2023-01-15T14:34:21.985051+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.70463370000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1257,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.915226+0000","last_change":"2023-01-15T14:34:20.018447+0000","last_active":"2023-01-15T14:34:21.915226+0000","last_peered":"2023-01-15T14:34:21.915226+0000","last_clean":"2023-01-15T14:34:21.915226+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:34:21.915226+0000","last_undegraded":"2023-01-15T14:34:21.915226+0000","last_fullsized":"2023-01-15T14:34:21.915226+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.0027284480000000001,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.32","version":"370'167","reported_seq":884,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.577340+0000","last_change":"2023-01-15T14:34:26.577470+0000","last_active":"2023-01-15T14:34:26.577340+0000","last_peered":"2023-01-15T14:34:26.577340+0000","last_clean":"2023-01-15T14:34:26.577340+0000","last_became_active":"2023-01-15T14:34:04.171493+0000","last_became_peered":"2023-01-15T14:34:04.171493+0000","last_unstale":"2023-01-15T14:34:26.577340+0000","last_undegraded":"2023-01-15T14:34:26.577340+0000","last_fullsized":"2023-01-15T14:34:26.577340+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.40477696400000002,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7048,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5327885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.13","version":"348'374","reported_seq":1405,"reported_epoch":371,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:21.558249+0000","last_change":"2023-01-15T14:34:20.521701+0000","last_active":"2023-01-15T14:34:21.558249+0000","last_peered":"2023-01-15T14:34:21.558249+0000","last_clean":"2023-01-15T14:34:21.558249+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:34:21.558249+0000","last_undegraded":"2023-01-15T14:34:21.558249+0000","last_fullsized":"2023-01-15T14:34:21.558249+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:29.359","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.0064123280000000001,"stat_sum":{"num_bytes":8364486,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":61489,"num_write":17,"num_write_kb":9266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.12","version":"376'195","reported_seq":944,"reported_epoch":376,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:26.549449+0000","last_change":"2023-01-15T14:34:26.549449+0000","last_active":"2023-01-15T14:34:26.549449+0000","last_peered":"2023-01-15T14:34:26.549449+0000","last_clean":"2023-01-15T14:34:26.549449+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:34:26.549449+0000","last_undegraded":"2023-01-15T14:34:26.549449+0000","last_fullsized":"2023-01-15T14:34:26.549449+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.92260187500000002,"stat_sum":{"num_bytes":10304297,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":20,"num_write_kb":9403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.11","version":"370'310","reported_seq":1595,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.933364+0000","last_change":"2023-01-15T14:34:21.933364+0000","last_active":"2023-01-15T14:34:21.933364+0000","last_peered":"2023-01-15T14:34:21.933364+0000","last_clean":"2023-01-15T14:34:21.933364+0000","last_became_active":"2023-01-15T14:34:09.541429+0000","last_became_peered":"2023-01-15T14:34:09.541429+0000","last_unstale":"2023-01-15T14:34:21.933364+0000","last_undegraded":"2023-01-15T14:34:21.933364+0000","last_fullsized":"2023-01-15T14:34:21.933364+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:21.075","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.70148824099999996,"stat_sum":{"num_bytes":10718732,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":48289,"num_write":26,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":20307357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.10","version":"308'406","reported_seq":1433,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.061906+0000","last_change":"2023-01-15T14:34:25.061875+0000","last_active":"2023-01-15T14:34:25.061906+0000","last_peered":"2023-01-15T14:34:25.061906+0000","last_clean":"2023-01-15T14:34:25.061906+0000","last_became_active":"2023-01-15T14:34:16.272883+0000","last_became_peered":"2023-01-15T14:34:16.272883+0000","last_unstale":"2023-01-15T14:34:25.061906+0000","last_undegraded":"2023-01-15T14:34:25.061906+0000","last_fullsized":"2023-01-15T14:34:25.061906+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.103703527,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":41964,"num_write":46,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12804882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1259,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.350695+0000","last_change":"2023-01-15T14:34:22.351402+0000","last_active":"2023-01-15T14:34:22.350695+0000","last_peered":"2023-01-15T14:34:22.350695+0000","last_clean":"2023-01-15T14:34:22.350695+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:34:22.350695+0000","last_undegraded":"2023-01-15T14:34:22.350695+0000","last_fullsized":"2023-01-15T14:34:22.350695+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.37932328300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.e","version":"139'38","reported_seq":48,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.352727+0000","last_change":"2023-01-15T14:34:22.453636+0000","last_active":"2023-01-15T14:34:22.352727+0000","last_peered":"2023-01-15T14:34:22.352727+0000","last_clean":"2023-01-15T14:34:22.352727+0000","last_became_active":"2023-01-15T14:33:57.310759+0000","last_became_peered":"2023-01-15T14:33:57.310759+0000","last_unstale":"2023-01-15T14:34:22.352727+0000","last_undegraded":"2023-01-15T14:34:22.352727+0000","last_fullsized":"2023-01-15T14:34:22.352727+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.48202801699999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.d","version":"296'232","reported_seq":1319,"reported_epoch":371,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:21.533074+0000","last_change":"2023-01-15T14:34:20.218166+0000","last_active":"2023-01-15T14:34:21.533074+0000","last_peered":"2023-01-15T14:34:21.533074+0000","last_clean":"2023-01-15T14:34:21.533074+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:34:21.533074+0000","last_undegraded":"2023-01-15T14:34:21.533074+0000","last_fullsized":"2023-01-15T14:34:21.533074+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.20767458899999999,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.c","version":"370'97","reported_seq":1124,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.054164+0000","last_change":"2023-01-15T14:34:20.727478+0000","last_active":"2023-01-15T14:34:22.054164+0000","last_peered":"2023-01-15T14:34:22.054164+0000","last_clean":"2023-01-15T14:34:22.054164+0000","last_became_active":"2023-01-15T14:33:52.210127+0000","last_became_peered":"2023-01-15T14:33:52.210127+0000","last_unstale":"2023-01-15T14:34:22.054164+0000","last_undegraded":"2023-01-15T14:34:22.054164+0000","last_fullsized":"2023-01-15T14:34:22.054164+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.71048677699999996,"stat_sum":{"num_bytes":2221858,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":24485,"num_write":6,"num_write_kb":686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8023674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":1028,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.915560+0000","last_change":"2023-01-15T14:34:20.225456+0000","last_active":"2023-01-15T14:34:21.915560+0000","last_peered":"2023-01-15T14:34:21.915560+0000","last_clean":"2023-01-15T14:34:21.915560+0000","last_became_active":"2023-01-15T14:33:52.425622+0000","last_became_peered":"2023-01-15T14:33:52.425622+0000","last_unstale":"2023-01-15T14:34:21.915560+0000","last_undegraded":"2023-01-15T14:34:21.915560+0000","last_fullsized":"2023-01-15T14:34:21.915560+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.20781496899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1351,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.085005+0000","last_change":"2023-01-15T14:34:20.232842+0000","last_active":"2023-01-15T14:34:22.085005+0000","last_peered":"2023-01-15T14:34:22.085005+0000","last_clean":"2023-01-15T14:34:22.085005+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:34:22.085005+0000","last_undegraded":"2023-01-15T14:34:22.085005+0000","last_fullsized":"2023-01-15T14:34:22.085005+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:48.664","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.20203781800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1136,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984939+0000","last_change":"2023-01-15T14:34:20.943666+0000","last_active":"2023-01-15T14:34:21.984939+0000","last_peered":"2023-01-15T14:34:21.984939+0000","last_clean":"2023-01-15T14:34:21.984939+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:34:21.984939+0000","last_undegraded":"2023-01-15T14:34:21.984939+0000","last_fullsized":"2023-01-15T14:34:21.984939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:38.874","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.71155139000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1356,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.914366+0000","last_change":"2023-01-15T14:34:21.433204+0000","last_active":"2023-01-15T14:34:21.914366+0000","last_peered":"2023-01-15T14:34:21.914366+0000","last_clean":"2023-01-15T14:34:21.914366+0000","last_became_active":"2023-01-15T14:34:00.140247+0000","last_became_peered":"2023-01-15T14:34:00.140247+0000","last_unstale":"2023-01-15T14:34:21.914366+0000","last_undegraded":"2023-01-15T14:34:21.914366+0000","last_fullsized":"2023-01-15T14:34:21.914366+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.20730312000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"1.0","version":"19'117","reported_seq":523,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.969936+0000","last_change":"2023-01-15T14:34:20.340358+0000","last_active":"2023-01-15T14:34:21.969936+0000","last_peered":"2023-01-15T14:34:21.969936+0000","last_clean":"2023-01-15T14:34:21.969936+0000","last_became_active":"2023-01-15T14:34:20.239089+0000","last_became_peered":"2023-01-15T14:34:20.239089+0000","last_unstale":"2023-01-15T14:34:21.969936+0000","last_undegraded":"2023-01-15T14:34:21.969936+0000","last_fullsized":"2023-01-15T14:34:21.969936+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":35,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.236849+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:34:21.236849+0000","last_peered":"2023-01-15T14:34:21.236849+0000","last_clean":"2023-01-15T14:34:21.236849+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:34:21.236849+0000","last_undegraded":"2023-01-15T14:34:21.236849+0000","last_fullsized":"2023-01-15T14:34:21.236849+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":1024,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.084734+0000","last_change":"2023-01-15T14:34:20.131724+0000","last_active":"2023-01-15T14:34:22.084734+0000","last_peered":"2023-01-15T14:34:22.084734+0000","last_clean":"2023-01-15T14:34:22.084734+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:34:22.084734+0000","last_undegraded":"2023-01-15T14:34:22.084734+0000","last_fullsized":"2023-01-15T14:34:22.084734+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.101156255,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":418,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.231851+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:34:21.231851+0000","last_peered":"2023-01-15T14:34:21.231851+0000","last_clean":"2023-01-15T14:34:21.231851+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:34:21.231851+0000","last_undegraded":"2023-01-15T14:34:21.231851+0000","last_fullsized":"2023-01-15T14:34:21.231851+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":745,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162180+0000","last_change":"2023-01-15T14:34:25.162149+0000","last_active":"2023-01-15T14:34:25.162180+0000","last_peered":"2023-01-15T14:34:25.162180+0000","last_clean":"2023-01-15T14:34:25.162180+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:34:25.162180+0000","last_undegraded":"2023-01-15T14:34:25.162180+0000","last_fullsized":"2023-01-15T14:34:25.162180+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.10135596099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":174,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.014687+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:34:22.014687+0000","last_peered":"2023-01-15T14:34:22.014687+0000","last_clean":"2023-01-15T14:34:22.014687+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:34:22.014687+0000","last_undegraded":"2023-01-15T14:34:22.014687+0000","last_fullsized":"2023-01-15T14:34:22.014687+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1346,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.423142+0000","last_change":"2023-01-15T14:34:21.423142+0000","last_active":"2023-01-15T14:34:21.423142+0000","last_peered":"2023-01-15T14:34:21.423142+0000","last_clean":"2023-01-15T14:34:21.423142+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:34:21.423142+0000","last_undegraded":"2023-01-15T14:34:21.423142+0000","last_fullsized":"2023-01-15T14:34:21.423142+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.20144286,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":240,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.021973+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:34:21.021973+0000","last_peered":"2023-01-15T14:34:21.021973+0000","last_clean":"2023-01-15T14:34:21.021973+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:34:21.021973+0000","last_undegraded":"2023-01-15T14:34:21.021973+0000","last_fullsized":"2023-01-15T14:34:21.021973+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:25.583","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"370'327","reported_seq":1448,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984426+0000","last_change":"2023-01-15T14:34:20.942782+0000","last_active":"2023-01-15T14:34:21.984426+0000","last_peered":"2023-01-15T14:34:21.984426+0000","last_clean":"2023-01-15T14:34:21.984426+0000","last_became_active":"2023-01-15T14:34:00.043073+0000","last_became_peered":"2023-01-15T14:34:00.043073+0000","last_unstale":"2023-01-15T14:34:21.984426+0000","last_undegraded":"2023-01-15T14:34:21.984426+0000","last_fullsized":"2023-01-15T14:34:21.984426+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:26.745","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.81220491699999997,"stat_sum":{"num_bytes":10343843,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":13875,"num_write":6,"num_write_kb":7330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9428925,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5","version":"370'162","reported_seq":1244,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.115852+0000","last_change":"2023-01-15T14:34:21.332251+0000","last_active":"2023-01-15T14:34:22.115852+0000","last_peered":"2023-01-15T14:34:22.115852+0000","last_clean":"2023-01-15T14:34:22.115852+0000","last_became_active":"2023-01-15T14:34:04.175473+0000","last_became_peered":"2023-01-15T14:34:04.175473+0000","last_unstale":"2023-01-15T14:34:22.115852+0000","last_undegraded":"2023-01-15T14:34:22.115852+0000","last_fullsized":"2023-01-15T14:34:22.115852+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.104735222,"stat_sum":{"num_bytes":11390101,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":26145,"num_write":9,"num_write_kb":4265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":13260921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":230,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.023193+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:34:21.023193+0000","last_peered":"2023-01-15T14:34:21.023193+0000","last_clean":"2023-01-15T14:34:21.023193+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:34:21.023193+0000","last_undegraded":"2023-01-15T14:34:21.023193+0000","last_fullsized":"2023-01-15T14:34:21.023193+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"372'315","reported_seq":1528,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.375241+0000","last_change":"2023-01-15T14:34:21.328677+0000","last_active":"2023-01-15T14:34:22.375241+0000","last_peered":"2023-01-15T14:34:22.375241+0000","last_clean":"2023-01-15T14:34:22.375241+0000","last_became_active":"2023-01-15T14:34:11.152893+0000","last_became_peered":"2023-01-15T14:34:11.152893+0000","last_unstale":"2023-01-15T14:34:22.375241+0000","last_undegraded":"2023-01-15T14:34:22.375241+0000","last_fullsized":"2023-01-15T14:34:22.375241+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:26.118","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0.20533874699999999,"stat_sum":{"num_bytes":8290510,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":419,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.580668+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:34:21.580668+0000","last_peered":"2023-01-15T14:34:21.580668+0000","last_clean":"2023-01-15T14:34:21.580668+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:34:21.580668+0000","last_undegraded":"2023-01-15T14:34:21.580668+0000","last_fullsized":"2023-01-15T14:34:21.580668+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"372'145","reported_seq":1172,"reported_epoch":372,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:22.453884+0000","last_change":"2023-01-15T14:34:22.453884+0000","last_active":"2023-01-15T14:34:22.453884+0000","last_peered":"2023-01-15T14:34:22.453884+0000","last_clean":"2023-01-15T14:34:22.453884+0000","last_became_active":"2023-01-15T14:34:17.186013+0000","last_became_peered":"2023-01-15T14:34:17.186013+0000","last_unstale":"2023-01-15T14:34:22.453884+0000","last_undegraded":"2023-01-15T14:34:22.453884+0000","last_fullsized":"2023-01-15T14:34:22.453884+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.76402682499999997,"stat_sum":{"num_bytes":9475691,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":13597,"num_write":13,"num_write_kb":8618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3546343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":230,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.123068+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:34:21.123068+0000","last_peered":"2023-01-15T14:34:21.123068+0000","last_clean":"2023-01-15T14:34:21.123068+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:34:21.123068+0000","last_undegraded":"2023-01-15T14:34:21.123068+0000","last_fullsized":"2023-01-15T14:34:21.123068+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":70,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.062807+0000","last_change":"2023-01-15T14:34:25.062784+0000","last_active":"2023-01-15T14:34:25.062807+0000","last_peered":"2023-01-15T14:34:25.062807+0000","last_clean":"2023-01-15T14:34:25.062807+0000","last_became_active":"2023-01-15T14:33:52.206701+0000","last_became_peered":"2023-01-15T14:33:52.206701+0000","last_unstale":"2023-01-15T14:34:25.062807+0000","last_undegraded":"2023-01-15T14:34:25.062807+0000","last_fullsized":"2023-01-15T14:34:25.062807+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.92108564599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":35,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.232904+0000","last_change":"2023-01-15T14:33:57.412557+0000","last_active":"2023-01-15T14:34:21.232904+0000","last_peered":"2023-01-15T14:34:21.232904+0000","last_clean":"2023-01-15T14:34:21.232904+0000","last_became_active":"2023-01-15T14:33:57.311981+0000","last_became_peered":"2023-01-15T14:33:57.311981+0000","last_unstale":"2023-01-15T14:34:21.232904+0000","last_undegraded":"2023-01-15T14:34:21.232904+0000","last_fullsized":"2023-01-15T14:34:21.232904+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"370'161","reported_seq":1220,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985701+0000","last_change":"2023-01-15T14:34:21.542422+0000","last_active":"2023-01-15T14:34:21.985701+0000","last_peered":"2023-01-15T14:34:21.985701+0000","last_clean":"2023-01-15T14:34:21.985701+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:34:21.985701+0000","last_undegraded":"2023-01-15T14:34:21.985701+0000","last_fullsized":"2023-01-15T14:34:21.985701+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"351'159","last_scrub_stamp":"2023-01-15T14:34:18.607751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:18.607751+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:36.520","scrub_duration":0.40666956999999998,"objects_trimmed":0,"snaptrim_duration":0.193730597,"stat_sum":{"num_bytes":2453855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":45,"reported_epoch":371,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:21.432392+0000","last_change":"2023-01-15T14:34:20.217871+0000","last_active":"2023-01-15T14:34:21.432392+0000","last_peered":"2023-01-15T14:34:21.432392+0000","last_clean":"2023-01-15T14:34:21.432392+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:34:21.432392+0000","last_undegraded":"2023-01-15T14:34:21.432392+0000","last_fullsized":"2023-01-15T14:34:21.432392+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31327724499999998,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.16","version":"371'133","reported_seq":715,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.783457+0000","last_change":"2023-01-15T14:34:22.783457+0000","last_active":"2023-01-15T14:34:22.783457+0000","last_peered":"2023-01-15T14:34:22.783457+0000","last_clean":"2023-01-15T14:34:22.783457+0000","last_became_active":"2023-01-15T14:34:19.412068+0000","last_became_peered":"2023-01-15T14:34:19.412068+0000","last_unstale":"2023-01-15T14:34:22.783457+0000","last_undegraded":"2023-01-15T14:34:22.783457+0000","last_fullsized":"2023-01-15T14:34:22.783457+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:00.515","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.81222085300000002,"stat_sum":{"num_bytes":7378471,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":6898,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7378471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.17","version":"278'93","reported_seq":70,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.843256+0000","last_change":"2023-01-15T14:34:25.843256+0000","last_active":"2023-01-15T14:34:25.843256+0000","last_peered":"2023-01-15T14:34:25.843256+0000","last_clean":"2023-01-15T14:34:25.843256+0000","last_became_active":"2023-01-15T14:33:52.303197+0000","last_became_peered":"2023-01-15T14:33:52.303197+0000","last_unstale":"2023-01-15T14:34:25.843256+0000","last_undegraded":"2023-01-15T14:34:25.843256+0000","last_fullsized":"2023-01-15T14:34:25.843256+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.67956067499999995,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.18","version":"371'331","reported_seq":1566,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.914081+0000","last_change":"2023-01-15T14:34:21.725480+0000","last_active":"2023-01-15T14:34:21.914081+0000","last_peered":"2023-01-15T14:34:21.914081+0000","last_clean":"2023-01-15T14:34:21.914081+0000","last_became_active":"2023-01-15T14:34:03.265912+0000","last_became_peered":"2023-01-15T14:34:03.265912+0000","last_unstale":"2023-01-15T14:34:21.914081+0000","last_undegraded":"2023-01-15T14:34:21.914081+0000","last_fullsized":"2023-01-15T14:34:21.914081+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.49970922699999998,"stat_sum":{"num_bytes":4332078,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":26,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":12147901,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.19","version":"372'259","reported_seq":1349,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.568955+0000","last_change":"2023-01-15T14:34:21.324339+0000","last_active":"2023-01-15T14:34:22.568955+0000","last_peered":"2023-01-15T14:34:22.568955+0000","last_clean":"2023-01-15T14:34:22.568955+0000","last_became_active":"2023-01-15T14:33:35.259904+0000","last_became_peered":"2023-01-15T14:33:35.259904+0000","last_unstale":"2023-01-15T14:34:22.568955+0000","last_undegraded":"2023-01-15T14:34:22.568955+0000","last_fullsized":"2023-01-15T14:34:22.568955+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:01.296","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.30031391499999999,"stat_sum":{"num_bytes":4941423,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1a","version":"370'354","reported_seq":1481,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.184862+0000","last_change":"2023-01-15T14:34:20.834956+0000","last_active":"2023-01-15T14:34:22.184862+0000","last_peered":"2023-01-15T14:34:22.184862+0000","last_clean":"2023-01-15T14:34:22.184862+0000","last_became_active":"2023-01-15T14:33:35.063699+0000","last_became_peered":"2023-01-15T14:33:35.063699+0000","last_unstale":"2023-01-15T14:34:22.184862+0000","last_undegraded":"2023-01-15T14:34:22.184862+0000","last_fullsized":"2023-01-15T14:34:22.184862+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.80461396600000001,"stat_sum":{"num_bytes":3052653,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":15,"num_read_kb":21426,"num_write":7,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1b","version":"376'107","reported_seq":111,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.577271+0000","last_change":"2023-01-15T14:34:25.667646+0000","last_active":"2023-01-15T14:34:26.577271+0000","last_peered":"2023-01-15T14:34:26.577271+0000","last_clean":"2023-01-15T14:34:26.577271+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:34:26.577271+0000","last_undegraded":"2023-01-15T14:34:26.577271+0000","last_fullsized":"2023-01-15T14:34:26.577271+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"370'101","last_scrub_stamp":"2023-01-15T14:34:24.137270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:24.137270+0000","objects_scrubbed":3,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:44.214","scrub_duration":0.106791863,"objects_trimmed":0,"snaptrim_duration":0.60357927199999994,"stat_sum":{"num_bytes":8765935,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":586,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.441047+0000","last_change":"2023-01-15T14:34:22.440913+0000","last_active":"2023-01-15T14:34:22.441047+0000","last_peered":"2023-01-15T14:34:22.441047+0000","last_clean":"2023-01-15T14:34:22.441047+0000","last_became_active":"2023-01-15T14:33:57.125876+0000","last_became_peered":"2023-01-15T14:33:57.125876+0000","last_unstale":"2023-01-15T14:34:22.441047+0000","last_undegraded":"2023-01-15T14:34:22.441047+0000","last_fullsized":"2023-01-15T14:34:22.441047+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:22.440826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:22.440826+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:44.562","scrub_duration":0.30343327599999997,"objects_trimmed":0,"snaptrim_duration":0.80641668200000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1292,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985419+0000","last_change":"2023-01-15T14:34:20.336911+0000","last_active":"2023-01-15T14:34:21.985419+0000","last_peered":"2023-01-15T14:34:21.985419+0000","last_clean":"2023-01-15T14:34:21.985419+0000","last_became_active":"2023-01-15T14:33:35.362912+0000","last_became_peered":"2023-01-15T14:33:35.362912+0000","last_unstale":"2023-01-15T14:34:21.985419+0000","last_undegraded":"2023-01-15T14:34:21.985419+0000","last_fullsized":"2023-01-15T14:34:21.985419+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30490724499999999,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":1003,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.916682+0000","last_change":"2023-01-15T14:34:20.221227+0000","last_active":"2023-01-15T14:34:21.916682+0000","last_peered":"2023-01-15T14:34:21.916682+0000","last_clean":"2023-01-15T14:34:21.916682+0000","last_became_active":"2023-01-15T14:33:35.261561+0000","last_became_peered":"2023-01-15T14:33:35.261561+0000","last_unstale":"2023-01-15T14:34:21.916682+0000","last_undegraded":"2023-01-15T14:34:21.916682+0000","last_fullsized":"2023-01-15T14:34:21.916682+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.20440873000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1f","version":"369'259","reported_seq":1363,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.842273+0000","last_change":"2023-01-15T14:34:25.842273+0000","last_active":"2023-01-15T14:34:25.842273+0000","last_peered":"2023-01-15T14:34:25.842273+0000","last_clean":"2023-01-15T14:34:25.842273+0000","last_became_active":"2023-01-15T14:34:00.240936+0000","last_became_peered":"2023-01-15T14:34:00.240936+0000","last_unstale":"2023-01-15T14:34:25.842273+0000","last_undegraded":"2023-01-15T14:34:25.842273+0000","last_fullsized":"2023-01-15T14:34:25.842273+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.57876590299999997,"stat_sum":{"num_bytes":5361581,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":13342,"num_write":5,"num_write_kb":2626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":5949777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.20","version":"324'273","reported_seq":1392,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.549250+0000","last_change":"2023-01-15T14:34:26.549351+0000","last_active":"2023-01-15T14:34:26.549250+0000","last_peered":"2023-01-15T14:34:26.549250+0000","last_clean":"2023-01-15T14:34:26.549250+0000","last_became_active":"2023-01-15T14:33:55.472883+0000","last_became_peered":"2023-01-15T14:33:55.472883+0000","last_unstale":"2023-01-15T14:34:26.549250+0000","last_undegraded":"2023-01-15T14:34:26.549250+0000","last_fullsized":"2023-01-15T14:34:26.549250+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.377629667,"stat_sum":{"num_bytes":4251137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10864,"num_write":9,"num_write_kb":4140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4438968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.21","version":"321'138","reported_seq":1067,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.331744+0000","last_change":"2023-01-15T14:34:21.045292+0000","last_active":"2023-01-15T14:34:21.331744+0000","last_peered":"2023-01-15T14:34:21.331744+0000","last_clean":"2023-01-15T14:34:21.331744+0000","last_became_active":"2023-01-15T14:33:45.267321+0000","last_became_peered":"2023-01-15T14:33:45.267321+0000","last_unstale":"2023-01-15T14:34:21.331744+0000","last_undegraded":"2023-01-15T14:34:21.331744+0000","last_fullsized":"2023-01-15T14:34:21.331744+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.92801007899999999,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.22","version":"323'161","reported_seq":1358,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.914804+0000","last_change":"2023-01-15T14:34:20.015929+0000","last_active":"2023-01-15T14:34:21.914804+0000","last_peered":"2023-01-15T14:34:21.914804+0000","last_clean":"2023-01-15T14:34:21.914804+0000","last_became_active":"2023-01-15T14:33:55.378253+0000","last_became_peered":"2023-01-15T14:33:55.378253+0000","last_unstale":"2023-01-15T14:34:21.914804+0000","last_undegraded":"2023-01-15T14:34:21.914804+0000","last_fullsized":"2023-01-15T14:34:21.914804+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.001164259,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":772,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162879+0000","last_change":"2023-01-15T14:34:25.162827+0000","last_active":"2023-01-15T14:34:25.162879+0000","last_peered":"2023-01-15T14:34:25.162879+0000","last_clean":"2023-01-15T14:34:25.162879+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:34:25.162879+0000","last_undegraded":"2023-01-15T14:34:25.162879+0000","last_fullsized":"2023-01-15T14:34:25.162879+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.811229019,"stat_sum":{"num_bytes":3802136,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":21818,"num_write":18,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.24","version":"371'265","reported_seq":1430,"reported_epoch":373,"state":"active+clean","last_fresh":"2023-01-15T14:34:23.142787+0000","last_change":"2023-01-15T14:34:22.134264+0000","last_active":"2023-01-15T14:34:23.142787+0000","last_peered":"2023-01-15T14:34:23.142787+0000","last_clean":"2023-01-15T14:34:23.142787+0000","last_became_active":"2023-01-15T14:33:45.075309+0000","last_became_peered":"2023-01-15T14:33:45.075309+0000","last_unstale":"2023-01-15T14:34:23.142787+0000","last_undegraded":"2023-01-15T14:34:23.142787+0000","last_fullsized":"2023-01-15T14:34:23.142787+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.218163993,"stat_sum":{"num_bytes":7636039,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":22662,"num_write":10,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.25","version":"333'108","reported_seq":1139,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.327424+0000","last_change":"2023-01-15T14:34:21.327424+0000","last_active":"2023-01-15T14:34:21.327424+0000","last_peered":"2023-01-15T14:34:21.327424+0000","last_clean":"2023-01-15T14:34:21.327424+0000","last_became_active":"2023-01-15T14:33:45.168598+0000","last_became_peered":"2023-01-15T14:33:45.168598+0000","last_unstale":"2023-01-15T14:34:21.327424+0000","last_undegraded":"2023-01-15T14:34:21.327424+0000","last_fullsized":"2023-01-15T14:34:21.327424+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.205454889,"stat_sum":{"num_bytes":2638765,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3030,"num_write":2,"num_write_kb":2577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":491,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.784143+0000","last_change":"2023-01-15T14:34:22.784486+0000","last_active":"2023-01-15T14:34:22.784143+0000","last_peered":"2023-01-15T14:34:22.784143+0000","last_clean":"2023-01-15T14:34:22.784143+0000","last_became_active":"2023-01-15T14:33:52.309110+0000","last_became_peered":"2023-01-15T14:33:52.309110+0000","last_unstale":"2023-01-15T14:34:22.784143+0000","last_undegraded":"2023-01-15T14:34:22.784143+0000","last_fullsized":"2023-01-15T14:34:22.784143+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.81247055599999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":1091,"reported_epoch":373,"state":"active+clean","last_fresh":"2023-01-15T14:34:23.243100+0000","last_change":"2023-01-15T14:34:20.224116+0000","last_active":"2023-01-15T14:34:23.243100+0000","last_peered":"2023-01-15T14:34:23.243100+0000","last_clean":"2023-01-15T14:34:23.243100+0000","last_became_active":"2023-01-15T14:33:45.171090+0000","last_became_peered":"2023-01-15T14:33:45.171090+0000","last_unstale":"2023-01-15T14:34:23.243100+0000","last_undegraded":"2023-01-15T14:34:23.243100+0000","last_fullsized":"2023-01-15T14:34:23.243100+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.20673013800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.28","version":"333'248","reported_seq":1418,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.913587+0000","last_change":"2023-01-15T14:34:21.726679+0000","last_active":"2023-01-15T14:34:21.913587+0000","last_peered":"2023-01-15T14:34:21.913587+0000","last_clean":"2023-01-15T14:34:21.913587+0000","last_became_active":"2023-01-15T14:33:45.366831+0000","last_became_peered":"2023-01-15T14:33:45.366831+0000","last_unstale":"2023-01-15T14:34:21.913587+0000","last_undegraded":"2023-01-15T14:34:21.913587+0000","last_fullsized":"2023-01-15T14:34:21.913587+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'248","last_scrub_stamp":"2023-01-15T14:34:20.620068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.620068+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:34.011","scrub_duration":0.40448800800000001,"objects_trimmed":0,"snaptrim_duration":0.50127574100000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":912,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.015075+0000","last_change":"2023-01-15T14:34:20.017312+0000","last_active":"2023-01-15T14:34:22.015075+0000","last_peered":"2023-01-15T14:34:22.015075+0000","last_clean":"2023-01-15T14:34:22.015075+0000","last_became_active":"2023-01-15T14:33:45.276960+0000","last_became_peered":"2023-01-15T14:33:45.276960+0000","last_unstale":"2023-01-15T14:34:22.015075+0000","last_undegraded":"2023-01-15T14:34:22.015075+0000","last_fullsized":"2023-01-15T14:34:22.015075+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.0021954629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1257,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.331801+0000","last_change":"2023-01-15T14:34:21.147869+0000","last_active":"2023-01-15T14:34:21.331801+0000","last_peered":"2023-01-15T14:34:21.331801+0000","last_clean":"2023-01-15T14:34:21.331801+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:34:21.331801+0000","last_undegraded":"2023-01-15T14:34:21.331801+0000","last_fullsized":"2023-01-15T14:34:21.331801+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":1.0302976619999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":80,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.063561+0000","last_change":"2023-01-15T14:34:25.063541+0000","last_active":"2023-01-15T14:34:25.063561+0000","last_peered":"2023-01-15T14:34:25.063561+0000","last_clean":"2023-01-15T14:34:25.063561+0000","last_became_active":"2023-01-15T14:33:57.261456+0000","last_became_peered":"2023-01-15T14:33:57.261456+0000","last_unstale":"2023-01-15T14:34:25.063561+0000","last_undegraded":"2023-01-15T14:34:25.063561+0000","last_fullsized":"2023-01-15T14:34:25.063561+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.812261444,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":461,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985234+0000","last_change":"2023-01-15T14:34:20.937488+0000","last_active":"2023-01-15T14:34:21.985234+0000","last_peered":"2023-01-15T14:34:21.985234+0000","last_clean":"2023-01-15T14:34:21.985234+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:34:21.985234+0000","last_undegraded":"2023-01-15T14:34:21.985234+0000","last_fullsized":"2023-01-15T14:34:21.985234+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.80575186899999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1227,"reported_epoch":371,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:21.480362+0000","last_change":"2023-01-15T14:34:20.218268+0000","last_active":"2023-01-15T14:34:21.480362+0000","last_peered":"2023-01-15T14:34:21.480362+0000","last_clean":"2023-01-15T14:34:21.480362+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:34:21.480362+0000","last_undegraded":"2023-01-15T14:34:21.480362+0000","last_fullsized":"2023-01-15T14:34:21.480362+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.105464771,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":48,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.351487+0000","last_change":"2023-01-15T14:34:22.352553+0000","last_active":"2023-01-15T14:34:22.351487+0000","last_peered":"2023-01-15T14:34:22.351487+0000","last_clean":"2023-01-15T14:34:22.351487+0000","last_became_active":"2023-01-15T14:33:57.210434+0000","last_became_peered":"2023-01-15T14:33:57.210434+0000","last_unstale":"2023-01-15T14:34:22.351487+0000","last_undegraded":"2023-01-15T14:34:22.351487+0000","last_fullsized":"2023-01-15T14:34:22.351487+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.381230034,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1121,"reported_epoch":371,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:21.233072+0000","last_change":"2023-01-15T14:34:20.218937+0000","last_active":"2023-01-15T14:34:21.233072+0000","last_peered":"2023-01-15T14:34:21.233072+0000","last_clean":"2023-01-15T14:34:21.233072+0000","last_became_active":"2023-01-15T14:33:55.174128+0000","last_became_peered":"2023-01-15T14:33:55.174128+0000","last_unstale":"2023-01-15T14:34:21.233072+0000","last_undegraded":"2023-01-15T14:34:21.233072+0000","last_fullsized":"2023-01-15T14:34:21.233072+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.20511622600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":1014,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.575991+0000","last_change":"2023-01-15T14:34:26.576215+0000","last_active":"2023-01-15T14:34:26.575991+0000","last_peered":"2023-01-15T14:34:26.575991+0000","last_clean":"2023-01-15T14:34:26.575991+0000","last_became_active":"2023-01-15T14:33:55.373498+0000","last_became_peered":"2023-01-15T14:33:55.373498+0000","last_unstale":"2023-01-15T14:34:26.575991+0000","last_undegraded":"2023-01-15T14:34:26.575991+0000","last_fullsized":"2023-01-15T14:34:26.575991+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":7,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.40446849800000001,"stat_sum":{"num_bytes":5419866,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":25948,"num_write":35,"num_write_kb":13827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":13340458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.31","version":"204'150","reported_seq":1243,"reported_epoch":371,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.430303+0000","last_change":"2023-01-15T14:34:21.430303+0000","last_active":"2023-01-15T14:34:21.430303+0000","last_peered":"2023-01-15T14:34:21.430303+0000","last_clean":"2023-01-15T14:34:21.430303+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:34:21.430303+0000","last_undegraded":"2023-01-15T14:34:21.430303+0000","last_fullsized":"2023-01-15T14:34:21.430303+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.406629089,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":106,"stat_sum":{"num_bytes":379530990,"num_objects":175,"num_object_clones":122,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":15,"num_read":982,"num_read_kb":1070378,"num_write":769,"num_write_kb":319556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":652,"num_bytes_recovered":434424211,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18841,"ondisk_log_size":18841,"up":318,"acting":318,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574925,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1112468,"kb_used_data":92599616,"kb_used_omap":8103,"kb_used_meta":0,"kb_avail":92586912,"statfs":{"total":95948165120,"available":94808997888,"internally_reserved":13008896,"allocated":94822006784,"data_stored":94822006784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8298388,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":3.452,"15min":3.8639999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1830000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":6.0389999999999997,"5min":5.6859999999999999,"15min":8.0340000000000007},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":36.622,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.94699999999999995}]},{"osd":1,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":12.064,"5min":4.7549999999999999,"15min":5.9480000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":204.74199999999999,"5min":204.74199999999999,"15min":204.74199999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":11.260999999999999,"5min":9.1910000000000007,"15min":7.25},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":94.537000000000006,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.84099999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":6.4119999999999999,"15min":5.3310000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":25.065999999999999,"5min":151.31800000000001,"15min":151.31800000000001},"last":1.105},{"interface":"front","average":{"1min":15.106,"5min":4.7649999999999997,"15min":4.3730000000000002},"min":{"1min":0.59599999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":162.96799999999999,"5min":162.96799999999999,"15min":162.96799999999999},"last":0.89000000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.198,"5min":7.6299999999999999,"15min":3.1749999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":25.359999999999999,"5min":209.05500000000001,"15min":209.05500000000001},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":2.879,"15min":4.0609999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.266,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.97999999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.4430000000000001,"5min":4.2279999999999998,"15min":7.6859999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":29.690000000000001,"5min":96.185000000000002,"15min":96.185000000000002},"last":1.0649999999999999},{"interface":"front","average":{"1min":10.949999999999999,"5min":7.7789999999999999,"15min":3.2360000000000002},"min":{"1min":0.46200000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":142.44999999999999,"5min":164.21199999999999,"15min":164.21199999999999},"last":1.151}]},{"osd":5,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.5489999999999999,"5min":4.3099999999999996,"15min":5.6539999999999999},"min":{"1min":0.63800000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":29.797999999999998,"5min":96.263999999999996,"15min":96.263999999999996},"last":1.004},{"interface":"front","average":{"1min":2.3519999999999999,"5min":2.6869999999999998,"15min":4.1920000000000002},"min":{"1min":0.55300000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":25.199000000000002,"5min":70.387,"15min":95.430000000000007},"last":1.2}]},{"osd":6,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.645,"5min":4.4189999999999996,"15min":7.1550000000000002},"min":{"1min":0.60999999999999999,"5min":0.496,"15min":0.496},"max":{"1min":29.876000000000001,"5min":96.299999999999997,"15min":96.299999999999997},"last":1.127},{"interface":"front","average":{"1min":2.399,"5min":2.7749999999999999,"15min":1.6579999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":25.236999999999998,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.4570000000000001}]}]},{"osd":6,"up_from":0,"seq":55834574927,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574927,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1002352,"kb_used_data":92700484,"kb_used_omap":14083,"kb_used_meta":0,"kb_avail":92697028,"statfs":{"total":95948165120,"available":94921756672,"internally_reserved":3538944,"allocated":94925295616,"data_stored":94925295616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14421733,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":22.818000000000001,"15min":10.882},"min":{"1min":0.505,"5min":0.44,"15min":0.44},"max":{"1min":5.0049999999999999,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":2.9990000000000001,"15min":1.52},"min":{"1min":0.51200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.127,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.92800000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.9390000000000001,"5min":3.742,"15min":6.976},"min":{"1min":0.499,"5min":0.49299999999999999,"15min":0.47399999999999998},"max":{"1min":23.193000000000001,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":10.99,"5min":6.3819999999999997,"15min":2.6560000000000001},"min":{"1min":0.67100000000000004,"5min":0.52900000000000003,"15min":0.50800000000000001},"max":{"1min":162.976,"5min":162.976,"15min":162.976},"last":0.93999999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.0950000000000002,"5min":23.544,"15min":11.522},"min":{"1min":0.497,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":23.247,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.26,"5min":1.28,"15min":1.0329999999999999},"min":{"1min":0.61899999999999999,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.5789999999999997,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.76200000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":2.802,"15min":1.8260000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.006,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.995},{"interface":"front","average":{"1min":5.3380000000000001,"5min":24.337,"15min":12.212999999999999},"min":{"1min":0.68600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":58.317,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":0.94999999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88100000000000001,"15min":0.90700000000000003},"min":{"1min":0.61499999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.175,"5min":2.577,"15min":2.577},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.903,"15min":2.1669999999999998},"min":{"1min":0.69099999999999995,"5min":0.65200000000000002,"15min":0.55300000000000005},"max":{"1min":1.0700000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.81699999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":3.2389999999999999,"15min":4.8609999999999998},"min":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":23.329000000000001,"5min":85.721999999999994,"15min":99.780000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.999,"5min":21.631,"15min":10.651999999999999},"min":{"1min":0.79500000000000004,"5min":0.66700000000000004,"15min":0.58699999999999997},"max":{"1min":1.173,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.016}]},{"osd":7,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":3.238,"5min":3.5790000000000002,"15min":9.9450000000000003},"min":{"1min":0.73999999999999999,"5min":0.63400000000000001,"15min":0.59499999999999997},"max":{"1min":42.551000000000002,"5min":89.061000000000007,"15min":179.73500000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":2.274,"5min":23.071000000000002,"15min":15.183},"min":{"1min":0.78700000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":23.356000000000002,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.034}]}]},{"osd":3,"up_from":13,"seq":55834574925,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1068616,"kb_used_data":92630792,"kb_used_omap":14319,"kb_used_meta":0,"kb_avail":92630764,"statfs":{"total":95948165120,"available":94853902336,"internally_reserved":28672,"allocated":94853931008,"data_stored":94853931008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14662681,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":7.8860000000000001,"5min":8.048,"15min":5.9139999999999997},"min":{"1min":0.39500000000000002,"5min":0.255,"15min":0.255},"max":{"1min":74.245999999999995,"5min":174.52199999999999,"15min":174.52199999999999},"last":74.245999999999995},{"interface":"front","average":{"1min":7.9649999999999999,"5min":3.4990000000000001,"15min":6.2610000000000001},"min":{"1min":0.46600000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":74.280000000000001,"5min":74.280000000000001,"15min":93.674000000000007},"last":74.280000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":10.762,"5min":6.625,"15min":11.228999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":117.664,"5min":117.664,"15min":159.256},"last":117.664},{"interface":"front","average":{"1min":4.6909999999999998,"5min":11.208,"15min":4.2300000000000004},"min":{"1min":0.41899999999999998,"5min":0.317,"15min":0.317},"max":{"1min":47.963000000000001,"5min":174.578,"15min":174.578},"last":0.51400000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":2.4820000000000002,"15min":3.0600000000000001},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":13.113,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.3079999999999998,"5min":6.8120000000000003,"15min":3.7320000000000002},"min":{"1min":0.44,"5min":0.375,"15min":0.375},"max":{"1min":15.946999999999999,"5min":174.45099999999999,"15min":174.45099999999999},"last":13.837}]},{"osd":4,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":28.689,"15min":13.936999999999999},"min":{"1min":0.56299999999999994,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":14.234999999999999,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":14.234999999999999},{"interface":"front","average":{"1min":11.906000000000001,"5min":31.741,"15min":19.829000000000001},"min":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":117.59,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":117.59}]},{"osd":5,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":11.022,"5min":5.9219999999999997,"15min":11.242000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":117.744,"5min":117.744,"15min":159.43000000000001},"last":117.744},{"interface":"front","average":{"1min":0.91700000000000004,"5min":5.9829999999999997,"15min":2.6749999999999998},"min":{"1min":0.66200000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2210000000000001,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.72799999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":5.984,"15min":2.6819999999999999},"min":{"1min":0.68300000000000005,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.286,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.3979999999999999,"15min":2.9740000000000002},"min":{"1min":0.71099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6799999999999999,"5min":43.661000000000001,"15min":60.098999999999997},"last":0.82999999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.4119999999999999,"15min":3.161},"min":{"1min":0.753,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.758,"5min":43.712000000000003,"15min":60.165999999999997},"last":0.92900000000000005},{"interface":"front","average":{"1min":19.324999999999999,"5min":11.952,"15min":16.044},"min":{"1min":0.65400000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":142.02000000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":117.822}]}]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":45,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1023264,"kb_used_data":92677672,"kb_used_omap":12025,"kb_used_meta":0,"kb_avail":92676116,"statfs":{"total":95948165120,"available":94900342784,"internally_reserved":1593344,"allocated":94901936128,"data_stored":94901936128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12314193,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":4.2469999999999999,"5min":4.2450000000000001,"15min":1.8420000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":78.456000000000003,"5min":78.456000000000003,"15min":78.456000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.4039999999999999,"15min":0.94399999999999995},"min":{"1min":0.36799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.35099999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":10.247,"5min":7.7949999999999999,"15min":3.1309999999999998},"min":{"1min":0.35099999999999998,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":128.749,"5min":143.071,"15min":143.071},"last":0.31900000000000001},{"interface":"front","average":{"1min":3.105,"5min":1.966,"15min":2.8919999999999999},"min":{"1min":0.35999999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":52.927999999999997,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.55700000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":3.8879999999999999,"5min":6.9180000000000001,"15min":9.2400000000000002},"min":{"1min":0.33700000000000002,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":34.244,"5min":143.13300000000001,"15min":183.53},"last":60.747999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":7.5039999999999996,"15min":3.012},"min":{"1min":0.41399999999999998,"5min":0.316,"15min":0.316},"max":{"1min":1.3959999999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.54500000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":5.9480000000000004,"15min":4.4160000000000004},"min":{"1min":0.67200000000000004,"5min":0.624,"15min":0.58399999999999996},"max":{"1min":1.2450000000000001,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.88300000000000001,"5min":4.7080000000000002,"15min":4.1079999999999997},"min":{"1min":0.57799999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.052,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.64200000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":2.5590000000000002,"5min":2.7490000000000001,"15min":1.7},"min":{"1min":0.63500000000000001,"5min":0.57199999999999995,"15min":0.54800000000000004},"max":{"1min":34.256999999999998,"5min":143.262,"15min":143.262},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.95099999999999996,"5min":3.012,"15min":1.778},"min":{"1min":0.70199999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3160000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.78900000000000003}]},{"osd":6,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.839,"15min":1.3009999999999999},"min":{"1min":0.61199999999999999,"5min":0.59599999999999997,"15min":0.55000000000000004},"max":{"1min":1.276,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8980000000000001,"15min":1.778},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":34.305999999999997,"5min":143.215,"15min":143.215},"last":0.73899999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":11.396000000000001,"5min":7.9800000000000004,"15min":4.6989999999999998},"min":{"1min":0.60599999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":176.44499999999999,"5min":176.44499999999999,"15min":176.44499999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8159999999999998,"15min":2.79},"min":{"1min":0.622,"5min":0.622,"15min":0.54500000000000004},"max":{"1min":34.207999999999998,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.78000000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":72,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1004276,"kb_used_data":92696224,"kb_used_omap":13440,"kb_used_meta":0,"kb_avail":92695104,"statfs":{"total":95948165120,"available":94919786496,"internally_reserved":1146880,"allocated":94920933376,"data_stored":94920933376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13762843,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":2.0619999999999998,"5min":3.2669999999999999,"15min":3.6840000000000002},"min":{"1min":0.37,"5min":0.314,"15min":0.314},"max":{"1min":23.68,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":14.157,"5min":6.5789999999999997,"15min":5.0750000000000002},"min":{"1min":0.42299999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":176.01599999999999,"5min":176.01599999999999,"15min":176.01599999999999},"last":13.068}]},{"osd":2,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":4.8959999999999999,"5min":3.4329999999999998,"15min":1.7889999999999999},"min":{"1min":0.44400000000000001,"5min":0.245,"15min":0.245},"max":{"1min":69.277000000000001,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.753},{"interface":"front","average":{"1min":0.629,"5min":1.173,"15min":3.1059999999999999},"min":{"1min":0.28399999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.873,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.63800000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":5.8440000000000003,"15min":15.172000000000001},"min":{"1min":0.38400000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.92500000000000004,"5min":245.637,"15min":282.01400000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":14.228999999999999,"5min":8.2870000000000008,"15min":7.5919999999999996},"min":{"1min":0.46500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":176.13900000000001,"5min":176.13900000000001,"15min":176.13900000000001},"last":13.005000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":19.170000000000002,"5min":20.029,"15min":9.8279999999999994},"min":{"1min":0.56299999999999994,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":176.18600000000001,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":13.361000000000001},{"interface":"front","average":{"1min":5.5510000000000002,"5min":15.916,"15min":7.7080000000000002},"min":{"1min":0.64300000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":69.611999999999995,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.83299999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":5.2060000000000004,"5min":2.3799999999999999,"15min":1.482},"min":{"1min":0.56899999999999995,"5min":0.433,"15min":0.433},"max":{"1min":69.478999999999999,"5min":69.478999999999999,"15min":69.478999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":3.2770000000000001},"min":{"1min":0.66000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3089999999999999,"5min":4.0780000000000003,"15min":46.905999999999999},"last":0.80700000000000005}]},{"osd":6,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":14.491,"5min":6.9109999999999996,"15min":5.6379999999999999},"min":{"1min":0.73199999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":176.249,"5min":176.249,"15min":176.249},"last":13.334},{"interface":"front","average":{"1min":5.2629999999999999,"5min":2.431,"15min":1.5589999999999999},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":69.549999999999997,"5min":69.549999999999997,"15min":69.549999999999997},"last":0.97199999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":7.617,"5min":6.2309999999999999,"15min":6.7549999999999999},"min":{"1min":0.63900000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":95.888999999999996,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.88},{"interface":"front","average":{"1min":14.544,"5min":6.9649999999999999,"15min":5.4880000000000004},"min":{"1min":0.78700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":176.22,"5min":176.22,"15min":176.22},"last":13.305}]}]},{"osd":0,"up_from":13,"seq":55834574927,"num_pgs":68,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1095180,"kb_used_data":92609812,"kb_used_omap":8284,"kb_used_meta":0,"kb_avail":92604200,"statfs":{"total":95948165120,"available":94826700800,"internally_reserved":5746688,"allocated":94832447488,"data_stored":94832447488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8483551,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":3.3460000000000001,"15min":1.53},"min":{"1min":0.34899999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":25.960999999999999,"5min":69.872,"15min":69.872},"last":0.39900000000000002},{"interface":"front","average":{"1min":11.249000000000001,"5min":5.335,"15min":2.2650000000000001},"min":{"1min":0.434,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":162.91200000000001,"5min":162.91200000000001,"15min":162.91200000000001},"last":0.438}]},{"osd":2,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":3.415,"15min":1.601},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0109999999999999,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":2.6320000000000001,"15min":1.3500000000000001},"min":{"1min":0.37,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86899999999999999,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.76200000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":7.8079999999999998,"15min":13.788},"min":{"1min":0.48199999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":26.027000000000001,"5min":199.08000000000001,"15min":284.62799999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":3.6070000000000002,"5min":7.867,"15min":3.1560000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33600000000000002},"max":{"1min":61.119,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.625}]},{"osd":4,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":16.664999999999999,"15min":6.2119999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1719999999999999,"5min":1486.309,"15min":1486.309},"last":0.96899999999999997},{"interface":"front","average":{"1min":2.1030000000000002,"5min":18.140999999999998,"15min":6.7300000000000004},"min":{"1min":0.63100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":26.137,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":0.879}]},{"osd":5,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":2.8980000000000001,"15min":7.4980000000000002},"min":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1930000000000001,"5min":98.411000000000001,"15min":176.34399999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.7070000000000001,"15min":1.2609999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.101,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.86299999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":2.1600000000000001,"5min":3.7890000000000001,"15min":2.0430000000000001},"min":{"1min":0.71899999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":26.082000000000001,"5min":70.259,"15min":70.259},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.026,"5min":3,"15min":1.756},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.57499999999999996},"max":{"1min":1.5409999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":0.98299999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":3.2770000000000001,"15min":11.539},"min":{"1min":0.70099999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":11.906000000000001,"5min":73.864999999999995,"15min":292.61099999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":2.1819999999999999,"5min":3.8090000000000002,"15min":2.0499999999999998},"min":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.58699999999999997},"max":{"1min":26.187000000000001,"5min":70.322000000000003,"15min":70.322000000000003},"last":0.80900000000000005}]}]}],"pool_statfs":[]}} 2023-01-15T14:34:28.324 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2023-01-15T14:34:28.325 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:34:28.326 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2023-01-15T14:34:28.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:34:28.327 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:27.687+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:28.327 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:27.907+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:28.327 INFO:tasks.rados.rados.0.smithi090.stdout:1431: done (0 left) 2023-01-15T14:34:28.327 INFO:tasks.rados.rados.0.smithi090.stdout:1432: write oid 41 current snap is 152 2023-01-15T14:34:28.328 INFO:tasks.rados.rados.0.smithi090.stdout:1432: seq_num 513 ranges {621991=798388,1936439=639509} 2023-01-15T14:34:28.328 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:34:28.329 INFO:tasks.rados.rados.0.smithi090.stdout:1432: writing smithi09025264-41 from 621991 to 1420379 tid 1 2023-01-15T14:34:28.329 INFO:tasks.rados.rados.0.smithi090.stdout:1432: writing smithi09025264-41 from 1936439 to 2575948 tid 2 2023-01-15T14:34:28.329 INFO:tasks.rados.rados.0.smithi090.stdout:1433: snap_create 2023-01-15T14:34:28.329 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:28.332 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:28.326+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:28.417 INFO:tasks.rados.rados.0.smithi090.stdout:1432: finishing write tid 1 to smithi09025264-41 2023-01-15T14:34:28.478 INFO:tasks.rados.rados.0.smithi090.stdout:1432: finishing write tid 2 to smithi09025264-41 2023-01-15T14:34:28.479 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:28.479 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:28.480 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:34:28.480 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:28.474+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:28.516 INFO:tasks.rados.rados.0.smithi090.stdout:1432: finishing write tid 3 to smithi09025264-41 2023-01-15T14:34:28.646 INFO:tasks.rados.rados.0.smithi090.stdout:1432: finishing write tid 4 to smithi09025264-41 2023-01-15T14:34:28.646 INFO:tasks.rados.rados.0.smithi090.stdout:1432: oid 41 updating version 0 to 374 2023-01-15T14:34:28.647 INFO:tasks.rados.rados.0.smithi090.stdout:1432: oid 41 updating version 374 to 375 2023-01-15T14:34:28.648 INFO:tasks.rados.rados.0.smithi090.stdout:1432: oid 41 version 375 is already newer than 373 2023-01-15T14:34:28.649 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 41 v 375 (ObjNum 513 snap 152 seq_num 513) dirty exists 2023-01-15T14:34:28.649 INFO:tasks.rados.rados.0.smithi090.stdout:1432: left oid 41 (ObjNum 513 snap 152 seq_num 513) 2023-01-15T14:34:28.649 INFO:tasks.rados.rados.0.smithi090.stdout:1432: done (0 left) 2023-01-15T14:34:28.650 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:28.651 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:28.652 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:34:28.658 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:28.660 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:34:28.660 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:28.639+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:28.661 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":561,"stamp":"2023-01-15T14:34:28.079958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":382760123,"num_objects":180,"num_object_clones":125,"num_object_copies":540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":180,"num_whiteouts":17,"num_read":1134,"num_read_kb":1134576,"num_write":918,"num_write_kb":317771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":415796357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19012,"ondisk_log_size":19012,"up":345,"acting":345,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":345,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6316800,"kb_used_data":555894816,"kb_used_omap":70758,"kb_used_meta":0,"kb_avail":555879480,"statfs":{"total":575688990720,"available":569220587520,"internally_reserved":15704064,"allocated":569236291584,"data_stored":569236291584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":72456752,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":11,"commit_latency_ns":0,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15319474,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":3,"num_read":28,"num_read_kb":53589,"num_write":-14,"num_write_kb":-1798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-41,"num_bytes_recovered":-33430771,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.037017"},"pg_stats":[{"pgid":"3.69","version":"110'149","reported_seq":924,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.914372+0000","last_change":"2023-01-15T14:34:20.016599+0000","last_active":"2023-01-15T14:34:21.914372+0000","last_peered":"2023-01-15T14:34:21.914372+0000","last_clean":"2023-01-15T14:34:21.914372+0000","last_became_active":"2023-01-15T14:33:50.216417+0000","last_became_peered":"2023-01-15T14:33:50.216417+0000","last_unstale":"2023-01-15T14:34:21.914372+0000","last_undegraded":"2023-01-15T14:34:21.914372+0000","last_fullsized":"2023-01-15T14:34:21.914372+0000","mapping_epoch":340,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":341,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.002032421,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.68","version":"372'259","reported_seq":1460,"reported_epoch":376,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:26.577601+0000","last_change":"2023-01-15T14:34:26.577668+0000","last_active":"2023-01-15T14:34:26.577601+0000","last_peered":"2023-01-15T14:34:26.577601+0000","last_clean":"2023-01-15T14:34:26.577601+0000","last_became_active":"2023-01-15T14:34:13.134258+0000","last_became_peered":"2023-01-15T14:34:13.134258+0000","last_unstale":"2023-01-15T14:34:26.577601+0000","last_undegraded":"2023-01-15T14:34:26.577601+0000","last_fullsized":"2023-01-15T14:34:26.577601+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.102953804,"stat_sum":{"num_bytes":8014979,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17027823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.67","version":"191'79","reported_seq":92,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.477283+0000","last_change":"2023-01-15T14:34:27.477360+0000","last_active":"2023-01-15T14:34:27.477283+0000","last_peered":"2023-01-15T14:34:27.477283+0000","last_clean":"2023-01-15T14:34:27.477283+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:34:27.477283+0000","last_undegraded":"2023-01-15T14:34:27.477283+0000","last_fullsized":"2023-01-15T14:34:27.477283+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.50550153399999997,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"1"}]},{"pgid":"3.66","version":"102'52","reported_seq":387,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985234+0000","last_change":"2023-01-15T14:34:20.334895+0000","last_active":"2023-01-15T14:34:21.985234+0000","last_peered":"2023-01-15T14:34:21.985234+0000","last_clean":"2023-01-15T14:34:21.985234+0000","last_became_active":"2023-01-15T14:33:48.695009+0000","last_became_peered":"2023-01-15T14:33:48.695009+0000","last_unstale":"2023-01-15T14:34:21.985234+0000","last_undegraded":"2023-01-15T14:34:21.985234+0000","last_fullsized":"2023-01-15T14:34:21.985234+0000","mapping_epoch":336,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":339,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.30362193900000001,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":2142,"num_write":4,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.65","version":"376'132","reported_seq":1277,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.577171+0000","last_change":"2023-01-15T14:34:27.577223+0000","last_active":"2023-01-15T14:34:27.577171+0000","last_peered":"2023-01-15T14:34:27.577171+0000","last_clean":"2023-01-15T14:34:27.577171+0000","last_became_active":"2023-01-15T14:34:16.276292+0000","last_became_peered":"2023-01-15T14:34:16.276292+0000","last_unstale":"2023-01-15T14:34:27.577171+0000","last_undegraded":"2023-01-15T14:34:27.577171+0000","last_fullsized":"2023-01-15T14:34:27.577171+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'126","last_scrub_stamp":"2023-01-15T14:34:23.903906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.903906+0000","objects_scrubbed":6,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:42.899","scrub_duration":0.30981859099999998,"objects_trimmed":0,"snaptrim_duration":0.29338232800000003,"stat_sum":{"num_bytes":9862828,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":10,"num_read_kb":13204,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"3.64","version":"334'251","reported_seq":1443,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162784+0000","last_change":"2023-01-15T14:34:25.162761+0000","last_active":"2023-01-15T14:34:25.162784+0000","last_peered":"2023-01-15T14:34:25.162784+0000","last_clean":"2023-01-15T14:34:25.162784+0000","last_became_active":"2023-01-15T14:33:55.675085+0000","last_became_peered":"2023-01-15T14:33:55.675085+0000","last_unstale":"2023-01-15T14:34:25.162784+0000","last_undegraded":"2023-01-15T14:34:25.162784+0000","last_fullsized":"2023-01-15T14:34:25.162784+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.91927382199999996,"stat_sum":{"num_bytes":3548733,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10773,"num_write":5,"num_write_kb":2741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.63","version":"149'233","reported_seq":772,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.063759+0000","last_change":"2023-01-15T14:34:25.063740+0000","last_active":"2023-01-15T14:34:25.063759+0000","last_peered":"2023-01-15T14:34:25.063759+0000","last_clean":"2023-01-15T14:34:25.063759+0000","last_became_active":"2023-01-15T14:33:55.277739+0000","last_became_peered":"2023-01-15T14:33:55.277739+0000","last_unstale":"2023-01-15T14:34:25.063759+0000","last_undegraded":"2023-01-15T14:34:25.063759+0000","last_fullsized":"2023-01-15T14:34:25.063759+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.01371225,"stat_sum":{"num_bytes":3802136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":21819,"num_write":19,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.62","version":"375'173","reported_seq":1396,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.183370+0000","last_change":"2023-01-15T14:34:27.183370+0000","last_active":"2023-01-15T14:34:27.183370+0000","last_peered":"2023-01-15T14:34:27.183370+0000","last_clean":"2023-01-15T14:34:27.183370+0000","last_became_active":"2023-01-15T14:33:56.590183+0000","last_became_peered":"2023-01-15T14:33:56.590183+0000","last_unstale":"2023-01-15T14:34:27.183370+0000","last_undegraded":"2023-01-15T14:34:27.183370+0000","last_fullsized":"2023-01-15T14:34:27.183370+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.925888936,"stat_sum":{"num_bytes":5361549,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":14814,"num_write":11,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5210269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"},{"start":"11a","length":"1"}]},{"pgid":"3.61","version":"375'148","reported_seq":1043,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.473779+0000","last_change":"2023-01-15T14:34:26.473873+0000","last_active":"2023-01-15T14:34:26.473779+0000","last_peered":"2023-01-15T14:34:26.473779+0000","last_clean":"2023-01-15T14:34:26.473779+0000","last_became_active":"2023-01-15T14:33:53.421056+0000","last_became_peered":"2023-01-15T14:33:53.421056+0000","last_unstale":"2023-01-15T14:34:26.473779+0000","last_undegraded":"2023-01-15T14:34:26.473779+0000","last_fullsized":"2023-01-15T14:34:26.473779+0000","mapping_epoch":343,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.301418146,"stat_sum":{"num_bytes":5113062,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":6060,"num_write":6,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.60","version":"376'285","reported_seq":1455,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.383912+0000","last_change":"2023-01-15T14:34:27.383958+0000","last_active":"2023-01-15T14:34:27.383912+0000","last_peered":"2023-01-15T14:34:27.383912+0000","last_clean":"2023-01-15T14:34:27.383912+0000","last_became_active":"2023-01-15T14:33:56.478613+0000","last_became_peered":"2023-01-15T14:33:56.478613+0000","last_unstale":"2023-01-15T14:34:27.383912+0000","last_undegraded":"2023-01-15T14:34:27.383912+0000","last_fullsized":"2023-01-15T14:34:27.383912+0000","mapping_epoch":346,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.201261894,"stat_sum":{"num_bytes":5214108,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":19437,"num_write":14,"num_write_kb":5082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15418818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1406,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.474713+0000","last_change":"2023-01-15T14:34:26.474781+0000","last_active":"2023-01-15T14:34:26.474713+0000","last_peered":"2023-01-15T14:34:26.474713+0000","last_clean":"2023-01-15T14:34:26.474713+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:34:26.474713+0000","last_undegraded":"2023-01-15T14:34:26.474713+0000","last_fullsized":"2023-01-15T14:34:26.474713+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:51.919","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.100394097,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5e","version":"215'39","reported_seq":1003,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.916320+0000","last_change":"2023-01-15T14:34:20.119902+0000","last_active":"2023-01-15T14:34:21.916320+0000","last_peered":"2023-01-15T14:34:21.916320+0000","last_clean":"2023-01-15T14:34:21.916320+0000","last_became_active":"2023-01-15T14:33:35.364143+0000","last_became_peered":"2023-01-15T14:33:35.364143+0000","last_unstale":"2023-01-15T14:34:21.916320+0000","last_undegraded":"2023-01-15T14:34:21.916320+0000","last_fullsized":"2023-01-15T14:34:21.916320+0000","mapping_epoch":325,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.103481502,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1342,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984976+0000","last_change":"2023-01-15T14:34:20.336111+0000","last_active":"2023-01-15T14:34:21.984976+0000","last_peered":"2023-01-15T14:34:21.984976+0000","last_clean":"2023-01-15T14:34:21.984976+0000","last_became_active":"2023-01-15T14:34:00.242122+0000","last_became_peered":"2023-01-15T14:34:00.242122+0000","last_unstale":"2023-01-15T14:34:21.984976+0000","last_undegraded":"2023-01-15T14:34:21.984976+0000","last_fullsized":"2023-01-15T14:34:21.984976+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:19.729021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:19.729021+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:29.740","scrub_duration":0.50499948100000003,"objects_trimmed":0,"snaptrim_duration":0.30454418900000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5c","version":"141'46","reported_seq":104,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.915404+0000","last_change":"2023-01-15T14:34:20.120026+0000","last_active":"2023-01-15T14:34:21.915404+0000","last_peered":"2023-01-15T14:34:21.915404+0000","last_clean":"2023-01-15T14:34:21.915404+0000","last_became_active":"2023-01-15T14:33:57.113032+0000","last_became_peered":"2023-01-15T14:33:57.113032+0000","last_unstale":"2023-01-15T14:34:21.915404+0000","last_undegraded":"2023-01-15T14:34:21.915404+0000","last_fullsized":"2023-01-15T14:34:21.915404+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.103747292,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5b","version":"283'80","reported_seq":945,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.969926+0000","last_change":"2023-01-15T14:34:26.757992+0000","last_active":"2023-01-15T14:34:26.969926+0000","last_peered":"2023-01-15T14:34:26.969926+0000","last_clean":"2023-01-15T14:34:26.969926+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:34:26.969926+0000","last_undegraded":"2023-01-15T14:34:26.969926+0000","last_fullsized":"2023-01-15T14:34:26.969926+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.48382510899999998,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.5a","version":"374'352","reported_seq":1520,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162591+0000","last_change":"2023-01-15T14:34:25.162531+0000","last_active":"2023-01-15T14:34:25.162591+0000","last_peered":"2023-01-15T14:34:25.162591+0000","last_clean":"2023-01-15T14:34:25.162591+0000","last_became_active":"2023-01-15T14:34:11.050316+0000","last_became_peered":"2023-01-15T14:34:11.050316+0000","last_unstale":"2023-01-15T14:34:25.162591+0000","last_undegraded":"2023-01-15T14:34:25.162591+0000","last_fullsized":"2023-01-15T14:34:25.162591+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:05.853","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.71046558199999998,"stat_sum":{"num_bytes":5983424,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":19294,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.59","version":"294'256","reported_seq":1301,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984171+0000","last_change":"2023-01-15T14:34:20.333913+0000","last_active":"2023-01-15T14:34:21.984171+0000","last_peered":"2023-01-15T14:34:21.984171+0000","last_clean":"2023-01-15T14:34:21.984171+0000","last_became_active":"2023-01-15T14:33:55.671990+0000","last_became_peered":"2023-01-15T14:33:55.671990+0000","last_unstale":"2023-01-15T14:34:21.984171+0000","last_undegraded":"2023-01-15T14:34:21.984171+0000","last_fullsized":"2023-01-15T14:34:21.984171+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"230'225","last_scrub_stamp":"2023-01-15T14:32:06.416338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:06.416338+0000","objects_scrubbed":2,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.59030486699999996,"objects_trimmed":0,"snaptrim_duration":0.30268093299999999,"stat_sum":{"num_bytes":1210137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":5,"num_read_kb":3549,"num_write":5,"num_write_kb":1181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.58","version":"323'320","reported_seq":1514,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.968859+0000","last_change":"2023-01-15T14:34:26.759869+0000","last_active":"2023-01-15T14:34:26.968859+0000","last_peered":"2023-01-15T14:34:26.968859+0000","last_clean":"2023-01-15T14:34:26.968859+0000","last_became_active":"2023-01-15T14:33:40.127669+0000","last_became_peered":"2023-01-15T14:33:40.127669+0000","last_unstale":"2023-01-15T14:34:26.968859+0000","last_undegraded":"2023-01-15T14:34:26.968859+0000","last_fullsized":"2023-01-15T14:34:26.968859+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.68782092900000003,"stat_sum":{"num_bytes":4332079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.57","version":"375'102","reported_seq":143,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.547916+0000","last_change":"2023-01-15T14:34:26.548062+0000","last_active":"2023-01-15T14:34:26.547916+0000","last_peered":"2023-01-15T14:34:26.547916+0000","last_clean":"2023-01-15T14:34:26.547916+0000","last_became_active":"2023-01-15T14:33:36.994447+0000","last_became_peered":"2023-01-15T14:33:36.994447+0000","last_unstale":"2023-01-15T14:34:26.547916+0000","last_undegraded":"2023-01-15T14:34:26.547916+0000","last_fullsized":"2023-01-15T14:34:26.547916+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'100","last_scrub_stamp":"2023-01-15T14:34:23.447852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.447852+0000","objects_scrubbed":2,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:30.799","scrub_duration":0.52101476099999999,"objects_trimmed":0,"snaptrim_duration":0.17314665600000001,"stat_sum":{"num_bytes":3025326,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.56","version":"320'127","reported_seq":664,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.971697+0000","last_change":"2023-01-15T14:34:26.550720+0000","last_active":"2023-01-15T14:34:26.971697+0000","last_peered":"2023-01-15T14:34:26.971697+0000","last_clean":"2023-01-15T14:34:26.971697+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:34:26.971697+0000","last_undegraded":"2023-01-15T14:34:26.971697+0000","last_fullsized":"2023-01-15T14:34:26.971697+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.47742669199999999,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.55","version":"110'55","reported_seq":1034,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.015630+0000","last_change":"2023-01-15T14:34:20.226765+0000","last_active":"2023-01-15T14:34:22.015630+0000","last_peered":"2023-01-15T14:34:22.015630+0000","last_clean":"2023-01-15T14:34:22.015630+0000","last_became_active":"2023-01-15T14:33:52.510429+0000","last_became_peered":"2023-01-15T14:33:52.510429+0000","last_unstale":"2023-01-15T14:34:22.015630+0000","last_undegraded":"2023-01-15T14:34:22.015630+0000","last_fullsized":"2023-01-15T14:34:22.015630+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.20908486400000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1162,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162579+0000","last_change":"2023-01-15T14:34:25.162553+0000","last_active":"2023-01-15T14:34:25.162579+0000","last_peered":"2023-01-15T14:34:25.162579+0000","last_clean":"2023-01-15T14:34:25.162579+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:34:25.162579+0000","last_undegraded":"2023-01-15T14:34:25.162579+0000","last_fullsized":"2023-01-15T14:34:25.162579+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:33:45.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:45.583005+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.91300059600000005,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.53","version":"373'404","reported_seq":1508,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.968653+0000","last_change":"2023-01-15T14:34:26.861727+0000","last_active":"2023-01-15T14:34:26.968653+0000","last_peered":"2023-01-15T14:34:26.968653+0000","last_clean":"2023-01-15T14:34:26.968653+0000","last_became_active":"2023-01-15T14:34:20.324151+0000","last_became_peered":"2023-01-15T14:34:20.324151+0000","last_unstale":"2023-01-15T14:34:26.968653+0000","last_undegraded":"2023-01-15T14:34:26.968653+0000","last_fullsized":"2023-01-15T14:34:26.968653+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:44.432","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.78888702099999997,"stat_sum":{"num_bytes":30096451,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":48,"num_read_kb":57311,"num_write":24,"num_write_kb":15073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":30103975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":768,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.068446+0000","last_change":"2023-01-15T14:34:26.963936+0000","last_active":"2023-01-15T14:34:27.068446+0000","last_peered":"2023-01-15T14:34:27.068446+0000","last_clean":"2023-01-15T14:34:27.068446+0000","last_became_active":"2023-01-15T14:33:55.472400+0000","last_became_peered":"2023-01-15T14:33:55.472400+0000","last_unstale":"2023-01-15T14:34:27.068446+0000","last_undegraded":"2023-01-15T14:34:27.068446+0000","last_fullsized":"2023-01-15T14:34:27.068446+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.89182815299999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"3.51","version":"377'324","reported_seq":1565,"reported_epoch":377,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:27.577346+0000","last_change":"2023-01-15T14:34:27.577346+0000","last_active":"2023-01-15T14:34:27.577346+0000","last_peered":"2023-01-15T14:34:27.577346+0000","last_clean":"2023-01-15T14:34:27.577346+0000","last_became_active":"2023-01-15T14:33:26.107940+0000","last_became_peered":"2023-01-15T14:33:26.107940+0000","last_unstale":"2023-01-15T14:34:27.577346+0000","last_undegraded":"2023-01-15T14:34:27.577346+0000","last_fullsized":"2023-01-15T14:34:27.577346+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:36.458","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.80165744800000005,"stat_sum":{"num_bytes":7496813,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"3.50","version":"371'422","reported_seq":1618,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.969468+0000","last_change":"2023-01-15T14:34:26.754846+0000","last_active":"2023-01-15T14:34:26.969468+0000","last_peered":"2023-01-15T14:34:26.969468+0000","last_clean":"2023-01-15T14:34:26.969468+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:34:26.969468+0000","last_undegraded":"2023-01-15T14:34:26.969468+0000","last_fullsized":"2023-01-15T14:34:26.969468+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":422,"log_dups_size":0,"ondisk_log_size":422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.582673302,"stat_sum":{"num_bytes":9681416,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":52505,"num_write":49,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.4f","version":"155'145","reported_seq":851,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.916914+0000","last_change":"2023-01-15T14:34:20.222818+0000","last_active":"2023-01-15T14:34:21.916914+0000","last_peered":"2023-01-15T14:34:21.916914+0000","last_clean":"2023-01-15T14:34:21.916914+0000","last_became_active":"2023-01-15T14:33:25.129936+0000","last_became_peered":"2023-01-15T14:33:25.129936+0000","last_unstale":"2023-01-15T14:34:21.916914+0000","last_undegraded":"2023-01-15T14:34:21.916914+0000","last_fullsized":"2023-01-15T14:34:21.916914+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20562384,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":130,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.062983+0000","last_change":"2023-01-15T14:34:25.062951+0000","last_active":"2023-01-15T14:34:25.062983+0000","last_peered":"2023-01-15T14:34:25.062983+0000","last_clean":"2023-01-15T14:34:25.062983+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:34:25.062983+0000","last_undegraded":"2023-01-15T14:34:25.062983+0000","last_fullsized":"2023-01-15T14:34:25.062983+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.92170574500000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.4d","version":"375'274","reported_seq":1406,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.971724+0000","last_change":"2023-01-15T14:34:26.652047+0000","last_active":"2023-01-15T14:34:26.971724+0000","last_peered":"2023-01-15T14:34:26.971724+0000","last_clean":"2023-01-15T14:34:26.971724+0000","last_became_active":"2023-01-15T14:34:05.178613+0000","last_became_peered":"2023-01-15T14:34:05.178613+0000","last_unstale":"2023-01-15T14:34:26.971724+0000","last_undegraded":"2023-01-15T14:34:26.971724+0000","last_fullsized":"2023-01-15T14:34:26.971724+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.50070625199999996,"stat_sum":{"num_bytes":14846966,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":15961,"num_write":20,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.4c","version":"309'71","reported_seq":145,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.476124+0000","last_change":"2023-01-15T14:34:27.476281+0000","last_active":"2023-01-15T14:34:27.476124+0000","last_peered":"2023-01-15T14:34:27.476124+0000","last_clean":"2023-01-15T14:34:27.476124+0000","last_became_active":"2023-01-15T14:33:52.311859+0000","last_became_peered":"2023-01-15T14:33:52.311859+0000","last_unstale":"2023-01-15T14:34:27.476124+0000","last_undegraded":"2023-01-15T14:34:27.476124+0000","last_fullsized":"2023-01-15T14:34:27.476124+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:27.027","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.50639235400000004,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"1"}]},{"pgid":"3.4b","version":"155'53","reported_seq":791,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.182933+0000","last_change":"2023-01-15T14:34:27.021418+0000","last_active":"2023-01-15T14:34:27.182933+0000","last_peered":"2023-01-15T14:34:27.182933+0000","last_clean":"2023-01-15T14:34:27.182933+0000","last_became_active":"2023-01-15T14:33:52.406605+0000","last_became_peered":"2023-01-15T14:33:52.406605+0000","last_unstale":"2023-01-15T14:34:27.182933+0000","last_undegraded":"2023-01-15T14:34:27.182933+0000","last_fullsized":"2023-01-15T14:34:27.182933+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:42.594","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.92073573900000005,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"3.4a","version":"375'377","reported_seq":1521,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.474696+0000","last_change":"2023-01-15T14:34:27.474877+0000","last_active":"2023-01-15T14:34:27.474696+0000","last_peered":"2023-01-15T14:34:27.474696+0000","last_clean":"2023-01-15T14:34:27.474696+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:34:27.474696+0000","last_undegraded":"2023-01-15T14:34:27.474696+0000","last_fullsized":"2023-01-15T14:34:27.474696+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":377,"log_dups_size":0,"ondisk_log_size":377,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:33.328","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50577030599999995,"stat_sum":{"num_bytes":12470870,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":63,"num_read_kb":67536,"num_write":73,"num_write_kb":23626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"1"}]},{"pgid":"3.49","version":"110'149","reported_seq":973,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.064362+0000","last_change":"2023-01-15T14:34:26.962843+0000","last_active":"2023-01-15T14:34:27.064362+0000","last_peered":"2023-01-15T14:34:27.064362+0000","last_clean":"2023-01-15T14:34:27.064362+0000","last_became_active":"2023-01-15T14:34:00.043221+0000","last_became_peered":"2023-01-15T14:34:00.043221+0000","last_unstale":"2023-01-15T14:34:27.064362+0000","last_undegraded":"2023-01-15T14:34:27.064362+0000","last_fullsized":"2023-01-15T14:34:27.064362+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.89088023000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"3.48","version":"179'204","reported_seq":1382,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.912930+0000","last_change":"2023-01-15T14:34:20.227341+0000","last_active":"2023-01-15T14:34:21.912930+0000","last_peered":"2023-01-15T14:34:21.912930+0000","last_clean":"2023-01-15T14:34:21.912930+0000","last_became_active":"2023-01-15T14:33:55.375518+0000","last_became_peered":"2023-01-15T14:33:55.375518+0000","last_unstale":"2023-01-15T14:34:21.912930+0000","last_undegraded":"2023-01-15T14:34:21.912930+0000","last_fullsized":"2023-01-15T14:34:21.912930+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:43.662","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.11151976,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1155,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985389+0000","last_change":"2023-01-15T14:34:21.450418+0000","last_active":"2023-01-15T14:34:21.985389+0000","last_peered":"2023-01-15T14:34:21.985389+0000","last_clean":"2023-01-15T14:34:21.985389+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:34:21.985389+0000","last_undegraded":"2023-01-15T14:34:21.985389+0000","last_fullsized":"2023-01-15T14:34:21.985389+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:34:21.450320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:21.450320+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:50.200","scrub_duration":0.30497759099999999,"objects_trimmed":0,"snaptrim_duration":0.30380391600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":1067,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.284983+0000","last_change":"2023-01-15T14:34:27.025106+0000","last_active":"2023-01-15T14:34:27.284983+0000","last_peered":"2023-01-15T14:34:27.284983+0000","last_clean":"2023-01-15T14:34:27.284983+0000","last_became_active":"2023-01-15T14:32:25.065196+0000","last_became_peered":"2023-01-15T14:32:25.065196+0000","last_unstale":"2023-01-15T14:34:27.284983+0000","last_undegraded":"2023-01-15T14:34:27.284983+0000","last_fullsized":"2023-01-15T14:34:27.284983+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:55.245","scrub_duration":0.30448409700000001,"objects_trimmed":0,"snaptrim_duration":0.72314178399999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"3.45","version":"253'125","reported_seq":947,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.064135+0000","last_change":"2023-01-15T14:34:25.064135+0000","last_active":"2023-01-15T14:34:25.064135+0000","last_peered":"2023-01-15T14:34:25.064135+0000","last_clean":"2023-01-15T14:34:25.064135+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:34:25.064135+0000","last_undegraded":"2023-01-15T14:34:25.064135+0000","last_fullsized":"2023-01-15T14:34:25.064135+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090126099999999999,"stat_sum":{"num_bytes":2635976,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":19927,"num_write":15,"num_write_kb":6659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4745860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1297,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984753+0000","last_change":"2023-01-15T14:34:20.337679+0000","last_active":"2023-01-15T14:34:21.984753+0000","last_peered":"2023-01-15T14:34:21.984753+0000","last_clean":"2023-01-15T14:34:21.984753+0000","last_became_active":"2023-01-15T14:33:55.678019+0000","last_became_peered":"2023-01-15T14:33:55.678019+0000","last_unstale":"2023-01-15T14:34:21.984753+0000","last_undegraded":"2023-01-15T14:34:21.984753+0000","last_fullsized":"2023-01-15T14:34:21.984753+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:27.120","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.30552496200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.43","version":"374'322","reported_seq":1554,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.664339+0000","last_change":"2023-01-15T14:34:25.664339+0000","last_active":"2023-01-15T14:34:25.664339+0000","last_peered":"2023-01-15T14:34:25.664339+0000","last_clean":"2023-01-15T14:34:25.664339+0000","last_became_active":"2023-01-15T14:34:02.211232+0000","last_became_peered":"2023-01-15T14:34:02.211232+0000","last_unstale":"2023-01-15T14:34:25.664339+0000","last_undegraded":"2023-01-15T14:34:25.664339+0000","last_fullsized":"2023-01-15T14:34:25.664339+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.60018934099999999,"stat_sum":{"num_bytes":6319564,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":12,"num_read_kb":16585,"num_write":7,"num_write_kb":4207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3395080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":1058,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.115137+0000","last_change":"2023-01-15T14:34:20.118682+0000","last_active":"2023-01-15T14:34:22.115137+0000","last_peered":"2023-01-15T14:34:22.115137+0000","last_clean":"2023-01-15T14:34:22.115137+0000","last_became_active":"2023-01-15T14:33:55.376782+0000","last_became_peered":"2023-01-15T14:33:55.376782+0000","last_unstale":"2023-01-15T14:34:22.115137+0000","last_undegraded":"2023-01-15T14:34:22.115137+0000","last_fullsized":"2023-01-15T14:34:22.115137+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.103556779,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1138,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984413+0000","last_change":"2023-01-15T14:34:20.438567+0000","last_active":"2023-01-15T14:34:21.984413+0000","last_peered":"2023-01-15T14:34:21.984413+0000","last_clean":"2023-01-15T14:34:21.984413+0000","last_became_active":"2023-01-15T14:33:55.771969+0000","last_became_peered":"2023-01-15T14:33:55.771969+0000","last_unstale":"2023-01-15T14:34:21.984413+0000","last_undegraded":"2023-01-15T14:34:21.984413+0000","last_fullsized":"2023-01-15T14:34:21.984413+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.30972029499999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.40","version":"376'315","reported_seq":1491,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.471786+0000","last_change":"2023-01-15T14:34:27.471928+0000","last_active":"2023-01-15T14:34:27.471786+0000","last_peered":"2023-01-15T14:34:27.471786+0000","last_clean":"2023-01-15T14:34:27.471786+0000","last_became_active":"2023-01-15T14:33:58.117085+0000","last_became_peered":"2023-01-15T14:33:58.117085+0000","last_unstale":"2023-01-15T14:34:27.471786+0000","last_undegraded":"2023-01-15T14:34:27.471786+0000","last_fullsized":"2023-01-15T14:34:27.471786+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.40830263300000003,"stat_sum":{"num_bytes":12753904,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":35541,"num_write":28,"num_write_kb":11001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":29167491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.3f","version":"373'235","reported_seq":1406,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.972169+0000","last_change":"2023-01-15T14:34:26.758404+0000","last_active":"2023-01-15T14:34:26.972169+0000","last_peered":"2023-01-15T14:34:26.972169+0000","last_clean":"2023-01-15T14:34:26.972169+0000","last_became_active":"2023-01-15T14:33:58.217783+0000","last_became_peered":"2023-01-15T14:33:58.217783+0000","last_unstale":"2023-01-15T14:34:26.972169+0000","last_undegraded":"2023-01-15T14:34:26.972169+0000","last_fullsized":"2023-01-15T14:34:26.972169+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:28.180","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.58449199200000002,"stat_sum":{"num_bytes":4764997,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":1391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6605283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.3e","version":"294'63","reported_seq":873,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.184490+0000","last_change":"2023-01-15T14:34:27.005447+0000","last_active":"2023-01-15T14:34:27.184490+0000","last_peered":"2023-01-15T14:34:27.184490+0000","last_clean":"2023-01-15T14:34:27.184490+0000","last_became_active":"2023-01-15T14:31:54.320788+0000","last_became_peered":"2023-01-15T14:31:54.320788+0000","last_unstale":"2023-01-15T14:34:27.184490+0000","last_undegraded":"2023-01-15T14:34:27.184490+0000","last_fullsized":"2023-01-15T14:34:27.184490+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.63983210400000001,"stat_sum":{"num_bytes":2621958,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"3.3d","version":"371'238","reported_seq":1363,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162883+0000","last_change":"2023-01-15T14:34:25.162856+0000","last_active":"2023-01-15T14:34:25.162883+0000","last_peered":"2023-01-15T14:34:25.162883+0000","last_clean":"2023-01-15T14:34:25.162883+0000","last_became_active":"2023-01-15T14:34:05.187121+0000","last_became_peered":"2023-01-15T14:34:05.187121+0000","last_unstale":"2023-01-15T14:34:25.162883+0000","last_undegraded":"2023-01-15T14:34:25.162883+0000","last_fullsized":"2023-01-15T14:34:25.162883+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.92019914000000003,"stat_sum":{"num_bytes":8050279,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":17,"num_read_kb":13797,"num_write":16,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":8050279,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":374,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.915658+0000","last_change":"2023-01-15T14:34:20.221556+0000","last_active":"2023-01-15T14:34:21.915658+0000","last_peered":"2023-01-15T14:34:21.915658+0000","last_clean":"2023-01-15T14:34:21.915658+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:34:21.915658+0000","last_undegraded":"2023-01-15T14:34:21.915658+0000","last_fullsized":"2023-01-15T14:34:21.915658+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:32.470","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.205049071,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":59,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.969469+0000","last_change":"2023-01-15T14:34:26.759529+0000","last_active":"2023-01-15T14:34:26.969469+0000","last_peered":"2023-01-15T14:34:26.969469+0000","last_clean":"2023-01-15T14:34:26.969469+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:34:26.969469+0000","last_undegraded":"2023-01-15T14:34:26.969469+0000","last_fullsized":"2023-01-15T14:34:26.969469+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.58592214600000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1381,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.062202+0000","last_change":"2023-01-15T14:34:25.062182+0000","last_active":"2023-01-15T14:34:25.062202+0000","last_peered":"2023-01-15T14:34:25.062202+0000","last_clean":"2023-01-15T14:34:25.062202+0000","last_became_active":"2023-01-15T14:33:55.474944+0000","last_became_peered":"2023-01-15T14:33:55.474944+0000","last_unstale":"2023-01-15T14:34:25.062202+0000","last_undegraded":"2023-01-15T14:34:25.062202+0000","last_fullsized":"2023-01-15T14:34:25.062202+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.104032045,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.39","version":"377'371","reported_seq":1542,"reported_epoch":377,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:27.482601+0000","last_change":"2023-01-15T14:34:27.064321+0000","last_active":"2023-01-15T14:34:27.482601+0000","last_peered":"2023-01-15T14:34:27.482601+0000","last_clean":"2023-01-15T14:34:27.482601+0000","last_became_active":"2023-01-15T14:34:08.023332+0000","last_became_peered":"2023-01-15T14:34:08.023332+0000","last_unstale":"2023-01-15T14:34:27.482601+0000","last_undegraded":"2023-01-15T14:34:27.482601+0000","last_fullsized":"2023-01-15T14:34:27.482601+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:28.106","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.22151658799999999,"stat_sum":{"num_bytes":25374153,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":82246,"num_write":11,"num_write_kb":8612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":26862613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"3.38","version":"370'317","reported_seq":1560,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.913299+0000","last_change":"2023-01-15T14:34:21.021163+0000","last_active":"2023-01-15T14:34:21.913299+0000","last_peered":"2023-01-15T14:34:21.913299+0000","last_clean":"2023-01-15T14:34:21.913299+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:34:21.913299+0000","last_undegraded":"2023-01-15T14:34:21.913299+0000","last_fullsized":"2023-01-15T14:34:21.913299+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.90503079399999997,"stat_sum":{"num_bytes":8461433,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":42848,"num_write":39,"num_write_kb":13455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":445,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.072691+0000","last_change":"2023-01-15T14:34:26.756851+0000","last_active":"2023-01-15T14:34:27.072691+0000","last_peered":"2023-01-15T14:34:27.072691+0000","last_clean":"2023-01-15T14:34:27.072691+0000","last_became_active":"2023-01-15T14:33:52.412128+0000","last_became_peered":"2023-01-15T14:33:52.412128+0000","last_unstale":"2023-01-15T14:34:27.072691+0000","last_undegraded":"2023-01-15T14:34:27.072691+0000","last_fullsized":"2023-01-15T14:34:27.072691+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.58261089300000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.36","version":"194'81","reported_seq":601,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.184717+0000","last_change":"2023-01-15T14:34:27.023039+0000","last_active":"2023-01-15T14:34:27.184717+0000","last_peered":"2023-01-15T14:34:27.184717+0000","last_clean":"2023-01-15T14:34:27.184717+0000","last_became_active":"2023-01-15T14:31:36.916461+0000","last_became_peered":"2023-01-15T14:31:36.916461+0000","last_unstale":"2023-01-15T14:34:27.184717+0000","last_undegraded":"2023-01-15T14:34:27.184717+0000","last_fullsized":"2023-01-15T14:34:27.184717+0000","mapping_epoch":209,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":210,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.72163543399999996,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"3.35","version":"110'55","reported_seq":70,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.062912+0000","last_change":"2023-01-15T14:34:25.062878+0000","last_active":"2023-01-15T14:34:25.062912+0000","last_peered":"2023-01-15T14:34:25.062912+0000","last_clean":"2023-01-15T14:34:25.062912+0000","last_became_active":"2023-01-15T14:33:52.612576+0000","last_became_peered":"2023-01-15T14:33:52.612576+0000","last_unstale":"2023-01-15T14:34:25.062912+0000","last_undegraded":"2023-01-15T14:34:25.062912+0000","last_fullsized":"2023-01-15T14:34:25.062912+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.92285126200000001,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":1182,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985051+0000","last_change":"2023-01-15T14:34:20.836036+0000","last_active":"2023-01-15T14:34:21.985051+0000","last_peered":"2023-01-15T14:34:21.985051+0000","last_clean":"2023-01-15T14:34:21.985051+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:34:21.985051+0000","last_undegraded":"2023-01-15T14:34:21.985051+0000","last_fullsized":"2023-01-15T14:34:21.985051+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.70463370000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1257,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.915226+0000","last_change":"2023-01-15T14:34:20.018447+0000","last_active":"2023-01-15T14:34:21.915226+0000","last_peered":"2023-01-15T14:34:21.915226+0000","last_clean":"2023-01-15T14:34:21.915226+0000","last_became_active":"2023-01-15T14:31:39.924505+0000","last_became_peered":"2023-01-15T14:31:39.924505+0000","last_unstale":"2023-01-15T14:34:21.915226+0000","last_undegraded":"2023-01-15T14:34:21.915226+0000","last_fullsized":"2023-01-15T14:34:21.915226+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.0027284480000000001,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.32","version":"370'167","reported_seq":884,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.577340+0000","last_change":"2023-01-15T14:34:26.577470+0000","last_active":"2023-01-15T14:34:26.577340+0000","last_peered":"2023-01-15T14:34:26.577340+0000","last_clean":"2023-01-15T14:34:26.577340+0000","last_became_active":"2023-01-15T14:34:04.171493+0000","last_became_peered":"2023-01-15T14:34:04.171493+0000","last_unstale":"2023-01-15T14:34:26.577340+0000","last_undegraded":"2023-01-15T14:34:26.577340+0000","last_fullsized":"2023-01-15T14:34:26.577340+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.40477696400000002,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7048,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5327885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.13","version":"374'376","reported_seq":1424,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.970553+0000","last_change":"2023-01-15T14:34:26.964825+0000","last_active":"2023-01-15T14:34:26.970553+0000","last_peered":"2023-01-15T14:34:26.970553+0000","last_clean":"2023-01-15T14:34:26.970553+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:34:26.970553+0000","last_undegraded":"2023-01-15T14:34:26.970553+0000","last_fullsized":"2023-01-15T14:34:26.970553+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'284","last_scrub_stamp":"2023-01-15T14:31:41.671409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:41.671409+0000","objects_scrubbed":7,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:29.359","scrub_duration":0.20976766199999999,"objects_trimmed":0,"snaptrim_duration":0.69403022299999995,"stat_sum":{"num_bytes":8364486,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":47,"num_read_kb":80208,"num_write":17,"num_write_kb":9266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.12","version":"376'195","reported_seq":944,"reported_epoch":376,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:26.549449+0000","last_change":"2023-01-15T14:34:26.549449+0000","last_active":"2023-01-15T14:34:26.549449+0000","last_peered":"2023-01-15T14:34:26.549449+0000","last_clean":"2023-01-15T14:34:26.549449+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:34:26.549449+0000","last_undegraded":"2023-01-15T14:34:26.549449+0000","last_fullsized":"2023-01-15T14:34:26.549449+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.92260187500000002,"stat_sum":{"num_bytes":10304297,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":20,"num_write_kb":9403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.11","version":"370'310","reported_seq":1610,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.473832+0000","last_change":"2023-01-15T14:34:27.473931+0000","last_active":"2023-01-15T14:34:27.473832+0000","last_peered":"2023-01-15T14:34:27.473832+0000","last_clean":"2023-01-15T14:34:27.473832+0000","last_became_active":"2023-01-15T14:34:09.541429+0000","last_became_peered":"2023-01-15T14:34:09.541429+0000","last_unstale":"2023-01-15T14:34:27.473832+0000","last_undegraded":"2023-01-15T14:34:27.473832+0000","last_fullsized":"2023-01-15T14:34:27.473832+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:21.075","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.40763146,"stat_sum":{"num_bytes":10718732,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":48289,"num_write":26,"num_write_kb":12792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":20307357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"1"}]},{"pgid":"3.10","version":"308'406","reported_seq":1433,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.061906+0000","last_change":"2023-01-15T14:34:25.061875+0000","last_active":"2023-01-15T14:34:25.061906+0000","last_peered":"2023-01-15T14:34:25.061906+0000","last_clean":"2023-01-15T14:34:25.061906+0000","last_became_active":"2023-01-15T14:34:16.272883+0000","last_became_peered":"2023-01-15T14:34:16.272883+0000","last_unstale":"2023-01-15T14:34:25.061906+0000","last_undegraded":"2023-01-15T14:34:25.061906+0000","last_fullsized":"2023-01-15T14:34:25.061906+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.103703527,"stat_sum":{"num_bytes":7223055,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":41964,"num_write":46,"num_write_kb":16642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12804882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1274,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.476664+0000","last_change":"2023-01-15T14:34:27.476744+0000","last_active":"2023-01-15T14:34:27.476664+0000","last_peered":"2023-01-15T14:34:27.476664+0000","last_clean":"2023-01-15T14:34:27.476664+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:34:27.476664+0000","last_undegraded":"2023-01-15T14:34:27.476664+0000","last_fullsized":"2023-01-15T14:34:27.476664+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.50523771200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":63,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.478892+0000","last_change":"2023-01-15T14:34:27.478940+0000","last_active":"2023-01-15T14:34:27.478892+0000","last_peered":"2023-01-15T14:34:27.478892+0000","last_clean":"2023-01-15T14:34:27.478892+0000","last_became_active":"2023-01-15T14:33:57.310759+0000","last_became_peered":"2023-01-15T14:33:57.310759+0000","last_unstale":"2023-01-15T14:34:27.478892+0000","last_undegraded":"2023-01-15T14:34:27.478892+0000","last_fullsized":"2023-01-15T14:34:27.478892+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.40718660499999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"1"}]},{"pgid":"3.d","version":"296'232","reported_seq":1333,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.972166+0000","last_change":"2023-01-15T14:34:26.653499+0000","last_active":"2023-01-15T14:34:26.972166+0000","last_peered":"2023-01-15T14:34:26.972166+0000","last_clean":"2023-01-15T14:34:26.972166+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:34:26.972166+0000","last_undegraded":"2023-01-15T14:34:26.972166+0000","last_fullsized":"2023-01-15T14:34:26.972166+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:22.089","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.50174801499999999,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.c","version":"370'97","reported_seq":1124,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.054164+0000","last_change":"2023-01-15T14:34:20.727478+0000","last_active":"2023-01-15T14:34:22.054164+0000","last_peered":"2023-01-15T14:34:22.054164+0000","last_clean":"2023-01-15T14:34:22.054164+0000","last_became_active":"2023-01-15T14:33:52.210127+0000","last_became_peered":"2023-01-15T14:33:52.210127+0000","last_unstale":"2023-01-15T14:34:22.054164+0000","last_undegraded":"2023-01-15T14:34:22.054164+0000","last_fullsized":"2023-01-15T14:34:22.054164+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":344,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.71048677699999996,"stat_sum":{"num_bytes":2221858,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":24485,"num_write":6,"num_write_kb":686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":8023674,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":1028,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.915560+0000","last_change":"2023-01-15T14:34:20.225456+0000","last_active":"2023-01-15T14:34:21.915560+0000","last_peered":"2023-01-15T14:34:21.915560+0000","last_clean":"2023-01-15T14:34:21.915560+0000","last_became_active":"2023-01-15T14:33:52.425622+0000","last_became_peered":"2023-01-15T14:33:52.425622+0000","last_unstale":"2023-01-15T14:34:21.915560+0000","last_undegraded":"2023-01-15T14:34:21.915560+0000","last_fullsized":"2023-01-15T14:34:21.915560+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.20781496899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1351,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.085005+0000","last_change":"2023-01-15T14:34:20.232842+0000","last_active":"2023-01-15T14:34:22.085005+0000","last_peered":"2023-01-15T14:34:22.085005+0000","last_clean":"2023-01-15T14:34:22.085005+0000","last_became_active":"2023-01-15T14:32:21.952226+0000","last_became_peered":"2023-01-15T14:32:21.952226+0000","last_unstale":"2023-01-15T14:34:22.085005+0000","last_undegraded":"2023-01-15T14:34:22.085005+0000","last_fullsized":"2023-01-15T14:34:22.085005+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:48.664","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.20203781800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1136,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984939+0000","last_change":"2023-01-15T14:34:20.943666+0000","last_active":"2023-01-15T14:34:21.984939+0000","last_peered":"2023-01-15T14:34:21.984939+0000","last_clean":"2023-01-15T14:34:21.984939+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:34:21.984939+0000","last_undegraded":"2023-01-15T14:34:21.984939+0000","last_fullsized":"2023-01-15T14:34:21.984939+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:38.874","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.71155139000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1356,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.914366+0000","last_change":"2023-01-15T14:34:21.433204+0000","last_active":"2023-01-15T14:34:21.914366+0000","last_peered":"2023-01-15T14:34:21.914366+0000","last_clean":"2023-01-15T14:34:21.914366+0000","last_became_active":"2023-01-15T14:34:00.140247+0000","last_became_peered":"2023-01-15T14:34:00.140247+0000","last_unstale":"2023-01-15T14:34:21.914366+0000","last_undegraded":"2023-01-15T14:34:21.914366+0000","last_fullsized":"2023-01-15T14:34:21.914366+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.20730312000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"1.0","version":"19'117","reported_seq":528,"reported_epoch":376,"state":"active+clean","last_fresh":"2023-01-15T14:34:26.071441+0000","last_change":"2023-01-15T14:34:20.340358+0000","last_active":"2023-01-15T14:34:26.071441+0000","last_peered":"2023-01-15T14:34:26.071441+0000","last_clean":"2023-01-15T14:34:26.071441+0000","last_became_active":"2023-01-15T14:34:20.239089+0000","last_became_peered":"2023-01-15T14:34:20.239089+0000","last_unstale":"2023-01-15T14:34:26.071441+0000","last_undegraded":"2023-01-15T14:34:26.071441+0000","last_fullsized":"2023-01-15T14:34:26.071441+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":376,"state":"active+clean","last_fresh":"2023-01-15T14:34:26.173169+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:34:26.173169+0000","last_peered":"2023-01-15T14:34:26.173169+0000","last_clean":"2023-01-15T14:34:26.173169+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:34:26.173169+0000","last_undegraded":"2023-01-15T14:34:26.173169+0000","last_fullsized":"2023-01-15T14:34:26.173169+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":1024,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.084734+0000","last_change":"2023-01-15T14:34:20.131724+0000","last_active":"2023-01-15T14:34:22.084734+0000","last_peered":"2023-01-15T14:34:22.084734+0000","last_clean":"2023-01-15T14:34:22.084734+0000","last_became_active":"2023-01-15T14:32:22.152859+0000","last_became_peered":"2023-01-15T14:32:22.152859+0000","last_unstale":"2023-01-15T14:34:22.084734+0000","last_undegraded":"2023-01-15T14:34:22.084734+0000","last_fullsized":"2023-01-15T14:34:22.084734+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.101156255,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":423,"reported_epoch":376,"state":"active+clean","last_fresh":"2023-01-15T14:34:26.272310+0000","last_change":"2023-01-15T14:33:06.912551+0000","last_active":"2023-01-15T14:34:26.272310+0000","last_peered":"2023-01-15T14:34:26.272310+0000","last_clean":"2023-01-15T14:34:26.272310+0000","last_became_active":"2023-01-15T14:30:51.033153+0000","last_became_peered":"2023-01-15T14:30:51.033153+0000","last_unstale":"2023-01-15T14:34:26.272310+0000","last_undegraded":"2023-01-15T14:34:26.272310+0000","last_fullsized":"2023-01-15T14:34:26.272310+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":745,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162180+0000","last_change":"2023-01-15T14:34:25.162149+0000","last_active":"2023-01-15T14:34:25.162180+0000","last_peered":"2023-01-15T14:34:25.162180+0000","last_clean":"2023-01-15T14:34:25.162180+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:34:25.162180+0000","last_undegraded":"2023-01-15T14:34:25.162180+0000","last_fullsized":"2023-01-15T14:34:25.162180+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.10135596099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":174,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.014687+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:34:22.014687+0000","last_peered":"2023-01-15T14:34:22.014687+0000","last_clean":"2023-01-15T14:34:22.014687+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:34:22.014687+0000","last_undegraded":"2023-01-15T14:34:22.014687+0000","last_fullsized":"2023-01-15T14:34:22.014687+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1361,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.383080+0000","last_change":"2023-01-15T14:34:27.023442+0000","last_active":"2023-01-15T14:34:27.383080+0000","last_peered":"2023-01-15T14:34:27.383080+0000","last_clean":"2023-01-15T14:34:27.383080+0000","last_became_active":"2023-01-15T14:31:52.229079+0000","last_became_peered":"2023-01-15T14:31:52.229079+0000","last_unstale":"2023-01-15T14:34:27.383080+0000","last_undegraded":"2023-01-15T14:34:27.383080+0000","last_fullsized":"2023-01-15T14:34:27.383080+0000","mapping_epoch":168,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30406088999999997,"objects_trimmed":0,"snaptrim_duration":0.92165383499999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":245,"reported_epoch":376,"state":"active+clean","last_fresh":"2023-01-15T14:34:26.201350+0000","last_change":"2023-01-15T14:32:57.480689+0000","last_active":"2023-01-15T14:34:26.201350+0000","last_peered":"2023-01-15T14:34:26.201350+0000","last_clean":"2023-01-15T14:34:26.201350+0000","last_became_active":"2023-01-15T14:30:51.236701+0000","last_became_peered":"2023-01-15T14:30:51.236701+0000","last_unstale":"2023-01-15T14:34:26.201350+0000","last_undegraded":"2023-01-15T14:34:26.201350+0000","last_fullsized":"2023-01-15T14:34:26.201350+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20359888700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"370'327","reported_seq":1448,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.984426+0000","last_change":"2023-01-15T14:34:20.942782+0000","last_active":"2023-01-15T14:34:21.984426+0000","last_peered":"2023-01-15T14:34:21.984426+0000","last_clean":"2023-01-15T14:34:21.984426+0000","last_became_active":"2023-01-15T14:34:00.043073+0000","last_became_peered":"2023-01-15T14:34:00.043073+0000","last_unstale":"2023-01-15T14:34:21.984426+0000","last_undegraded":"2023-01-15T14:34:21.984426+0000","last_fullsized":"2023-01-15T14:34:21.984426+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:26.745","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.81220491699999997,"stat_sum":{"num_bytes":10343843,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":13875,"num_write":6,"num_write_kb":7330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":9428925,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.5","version":"370'162","reported_seq":1244,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.115852+0000","last_change":"2023-01-15T14:34:21.332251+0000","last_active":"2023-01-15T14:34:22.115852+0000","last_peered":"2023-01-15T14:34:22.115852+0000","last_clean":"2023-01-15T14:34:22.115852+0000","last_became_active":"2023-01-15T14:34:04.175473+0000","last_became_peered":"2023-01-15T14:34:04.175473+0000","last_unstale":"2023-01-15T14:34:22.115852+0000","last_undegraded":"2023-01-15T14:34:22.115852+0000","last_fullsized":"2023-01-15T14:34:22.115852+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.104735222,"stat_sum":{"num_bytes":11390101,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":26145,"num_write":9,"num_write_kb":4265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":13260921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":235,"reported_epoch":376,"state":"active+clean","last_fresh":"2023-01-15T14:34:26.100567+0000","last_change":"2023-01-15T14:32:21.422286+0000","last_active":"2023-01-15T14:34:26.100567+0000","last_peered":"2023-01-15T14:34:26.100567+0000","last_clean":"2023-01-15T14:34:26.100567+0000","last_became_active":"2023-01-15T14:30:51.237309+0000","last_became_peered":"2023-01-15T14:30:51.237309+0000","last_unstale":"2023-01-15T14:34:26.100567+0000","last_undegraded":"2023-01-15T14:34:26.100567+0000","last_fullsized":"2023-01-15T14:34:26.100567+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50891173499999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"377'323","reported_seq":1551,"reported_epoch":377,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:27.666841+0000","last_change":"2023-01-15T14:34:27.384661+0000","last_active":"2023-01-15T14:34:27.666841+0000","last_peered":"2023-01-15T14:34:27.666841+0000","last_clean":"2023-01-15T14:34:27.666841+0000","last_became_active":"2023-01-15T14:34:11.152893+0000","last_became_peered":"2023-01-15T14:34:11.152893+0000","last_unstale":"2023-01-15T14:34:27.666841+0000","last_undegraded":"2023-01-15T14:34:27.666841+0000","last_fullsized":"2023-01-15T14:34:27.666841+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:26.118","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0.50119190999999996,"stat_sum":{"num_bytes":8290510,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":424,"reported_epoch":376,"state":"active+clean","last_fresh":"2023-01-15T14:34:26.151910+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:34:26.151910+0000","last_peered":"2023-01-15T14:34:26.151910+0000","last_clean":"2023-01-15T14:34:26.151910+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:34:26.151910+0000","last_undegraded":"2023-01-15T14:34:26.151910+0000","last_fullsized":"2023-01-15T14:34:26.151910+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"375'147","reported_seq":1191,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.475307+0000","last_change":"2023-01-15T14:34:27.475351+0000","last_active":"2023-01-15T14:34:27.475307+0000","last_peered":"2023-01-15T14:34:27.475307+0000","last_clean":"2023-01-15T14:34:27.475307+0000","last_became_active":"2023-01-15T14:34:17.186013+0000","last_became_peered":"2023-01-15T14:34:17.186013+0000","last_unstale":"2023-01-15T14:34:27.475307+0000","last_undegraded":"2023-01-15T14:34:27.475307+0000","last_fullsized":"2023-01-15T14:34:27.475307+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.50394517900000002,"stat_sum":{"num_bytes":9475691,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":12,"num_read_kb":13597,"num_write":13,"num_write_kb":8618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3546343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":235,"reported_epoch":376,"state":"active+clean","last_fresh":"2023-01-15T14:34:26.301999+0000","last_change":"2023-01-15T14:32:51.551175+0000","last_active":"2023-01-15T14:34:26.301999+0000","last_peered":"2023-01-15T14:34:26.301999+0000","last_clean":"2023-01-15T14:34:26.301999+0000","last_became_active":"2023-01-15T14:30:51.035348+0000","last_became_peered":"2023-01-15T14:30:51.035348+0000","last_unstale":"2023-01-15T14:34:26.301999+0000","last_undegraded":"2023-01-15T14:34:26.301999+0000","last_fullsized":"2023-01-15T14:34:26.301999+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30439642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":70,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.062807+0000","last_change":"2023-01-15T14:34:25.062784+0000","last_active":"2023-01-15T14:34:25.062807+0000","last_peered":"2023-01-15T14:34:25.062807+0000","last_clean":"2023-01-15T14:34:25.062807+0000","last_became_active":"2023-01-15T14:33:52.206701+0000","last_became_peered":"2023-01-15T14:33:52.206701+0000","last_unstale":"2023-01-15T14:34:25.062807+0000","last_undegraded":"2023-01-15T14:34:25.062807+0000","last_fullsized":"2023-01-15T14:34:25.062807+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.92108564599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":40,"reported_epoch":376,"state":"active+clean","last_fresh":"2023-01-15T14:34:26.273979+0000","last_change":"2023-01-15T14:33:57.412557+0000","last_active":"2023-01-15T14:34:26.273979+0000","last_peered":"2023-01-15T14:34:26.273979+0000","last_clean":"2023-01-15T14:34:26.273979+0000","last_became_active":"2023-01-15T14:33:57.311981+0000","last_became_peered":"2023-01-15T14:33:57.311981+0000","last_unstale":"2023-01-15T14:34:26.273979+0000","last_undegraded":"2023-01-15T14:34:26.273979+0000","last_fullsized":"2023-01-15T14:34:26.273979+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"370'161","reported_seq":1220,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985701+0000","last_change":"2023-01-15T14:34:21.542422+0000","last_active":"2023-01-15T14:34:21.985701+0000","last_peered":"2023-01-15T14:34:21.985701+0000","last_clean":"2023-01-15T14:34:21.985701+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:34:21.985701+0000","last_undegraded":"2023-01-15T14:34:21.985701+0000","last_fullsized":"2023-01-15T14:34:21.985701+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"351'159","last_scrub_stamp":"2023-01-15T14:34:18.607751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:18.607751+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:36.520","scrub_duration":0.40666956999999998,"objects_trimmed":0,"snaptrim_duration":0.193730597,"stat_sum":{"num_bytes":2453855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":63,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.478113+0000","last_change":"2023-01-15T14:34:27.478243+0000","last_active":"2023-01-15T14:34:27.478113+0000","last_peered":"2023-01-15T14:34:27.478113+0000","last_clean":"2023-01-15T14:34:27.478113+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:34:27.478113+0000","last_undegraded":"2023-01-15T14:34:27.478113+0000","last_fullsized":"2023-01-15T14:34:27.478113+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50678318700000002,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"1"}]},{"pgid":"3.16","version":"375'135","reported_seq":730,"reported_epoch":377,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:27.680078+0000","last_change":"2023-01-15T14:34:27.680145+0000","last_active":"2023-01-15T14:34:27.680078+0000","last_peered":"2023-01-15T14:34:27.680078+0000","last_clean":"2023-01-15T14:34:27.680078+0000","last_became_active":"2023-01-15T14:34:19.412068+0000","last_became_peered":"2023-01-15T14:34:19.412068+0000","last_unstale":"2023-01-15T14:34:27.680078+0000","last_undegraded":"2023-01-15T14:34:27.680078+0000","last_fullsized":"2023-01-15T14:34:27.680078+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:00.515","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.47668729900000001,"stat_sum":{"num_bytes":7378471,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":6898,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":7378471,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":70,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.843256+0000","last_change":"2023-01-15T14:34:25.843256+0000","last_active":"2023-01-15T14:34:25.843256+0000","last_peered":"2023-01-15T14:34:25.843256+0000","last_clean":"2023-01-15T14:34:25.843256+0000","last_became_active":"2023-01-15T14:33:52.303197+0000","last_became_peered":"2023-01-15T14:33:52.303197+0000","last_unstale":"2023-01-15T14:34:25.843256+0000","last_undegraded":"2023-01-15T14:34:25.843256+0000","last_fullsized":"2023-01-15T14:34:25.843256+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.67956067499999995,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.18","version":"371'331","reported_seq":1566,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.914081+0000","last_change":"2023-01-15T14:34:21.725480+0000","last_active":"2023-01-15T14:34:21.914081+0000","last_peered":"2023-01-15T14:34:21.914081+0000","last_clean":"2023-01-15T14:34:21.914081+0000","last_became_active":"2023-01-15T14:34:03.265912+0000","last_became_peered":"2023-01-15T14:34:03.265912+0000","last_unstale":"2023-01-15T14:34:21.914081+0000","last_undegraded":"2023-01-15T14:34:21.914081+0000","last_fullsized":"2023-01-15T14:34:21.914081+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.49970922699999998,"stat_sum":{"num_bytes":4332078,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":26,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":12147901,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.19","version":"372'259","reported_seq":1364,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.284112+0000","last_change":"2023-01-15T14:34:27.024540+0000","last_active":"2023-01-15T14:34:27.284112+0000","last_peered":"2023-01-15T14:34:27.284112+0000","last_clean":"2023-01-15T14:34:27.284112+0000","last_became_active":"2023-01-15T14:33:35.259904+0000","last_became_peered":"2023-01-15T14:33:35.259904+0000","last_unstale":"2023-01-15T14:34:27.284112+0000","last_undegraded":"2023-01-15T14:34:27.284112+0000","last_fullsized":"2023-01-15T14:34:27.284112+0000","mapping_epoch":171,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:01.296","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.9227244,"stat_sum":{"num_bytes":4941423,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]},{"pgid":"3.1a","version":"370'354","reported_seq":1481,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.184862+0000","last_change":"2023-01-15T14:34:20.834956+0000","last_active":"2023-01-15T14:34:22.184862+0000","last_peered":"2023-01-15T14:34:22.184862+0000","last_clean":"2023-01-15T14:34:22.184862+0000","last_became_active":"2023-01-15T14:33:35.063699+0000","last_became_peered":"2023-01-15T14:33:35.063699+0000","last_unstale":"2023-01-15T14:34:22.184862+0000","last_undegraded":"2023-01-15T14:34:22.184862+0000","last_fullsized":"2023-01-15T14:34:22.184862+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'293","last_scrub_stamp":"2023-01-15T14:32:39.836023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.836023+0000","objects_scrubbed":3,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60775138699999998,"objects_trimmed":0,"snaptrim_duration":0.80461396600000001,"stat_sum":{"num_bytes":3052653,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":15,"num_read_kb":21426,"num_write":7,"num_write_kb":598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1b","version":"376'107","reported_seq":111,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.577271+0000","last_change":"2023-01-15T14:34:25.667646+0000","last_active":"2023-01-15T14:34:26.577271+0000","last_peered":"2023-01-15T14:34:26.577271+0000","last_clean":"2023-01-15T14:34:26.577271+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:34:26.577271+0000","last_undegraded":"2023-01-15T14:34:26.577271+0000","last_fullsized":"2023-01-15T14:34:26.577271+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"370'101","last_scrub_stamp":"2023-01-15T14:34:24.137270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:24.137270+0000","objects_scrubbed":3,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:44.214","scrub_duration":0.106791863,"objects_trimmed":0,"snaptrim_duration":0.60357927199999994,"stat_sum":{"num_bytes":8765935,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":586,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.441047+0000","last_change":"2023-01-15T14:34:22.440913+0000","last_active":"2023-01-15T14:34:22.441047+0000","last_peered":"2023-01-15T14:34:22.441047+0000","last_clean":"2023-01-15T14:34:22.441047+0000","last_became_active":"2023-01-15T14:33:57.125876+0000","last_became_peered":"2023-01-15T14:33:57.125876+0000","last_unstale":"2023-01-15T14:34:22.441047+0000","last_undegraded":"2023-01-15T14:34:22.441047+0000","last_fullsized":"2023-01-15T14:34:22.441047+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:22.440826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:22.440826+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:44.562","scrub_duration":0.30343327599999997,"objects_trimmed":0,"snaptrim_duration":0.80641668200000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1292,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985419+0000","last_change":"2023-01-15T14:34:20.336911+0000","last_active":"2023-01-15T14:34:21.985419+0000","last_peered":"2023-01-15T14:34:21.985419+0000","last_clean":"2023-01-15T14:34:21.985419+0000","last_became_active":"2023-01-15T14:33:35.362912+0000","last_became_peered":"2023-01-15T14:33:35.362912+0000","last_unstale":"2023-01-15T14:34:21.985419+0000","last_undegraded":"2023-01-15T14:34:21.985419+0000","last_fullsized":"2023-01-15T14:34:21.985419+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30490724499999999,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":1003,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.916682+0000","last_change":"2023-01-15T14:34:20.221227+0000","last_active":"2023-01-15T14:34:21.916682+0000","last_peered":"2023-01-15T14:34:21.916682+0000","last_clean":"2023-01-15T14:34:21.916682+0000","last_became_active":"2023-01-15T14:33:35.261561+0000","last_became_peered":"2023-01-15T14:33:35.261561+0000","last_unstale":"2023-01-15T14:34:21.916682+0000","last_undegraded":"2023-01-15T14:34:21.916682+0000","last_fullsized":"2023-01-15T14:34:21.916682+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.20440873000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.1f","version":"369'259","reported_seq":1363,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.842273+0000","last_change":"2023-01-15T14:34:25.842273+0000","last_active":"2023-01-15T14:34:25.842273+0000","last_peered":"2023-01-15T14:34:25.842273+0000","last_clean":"2023-01-15T14:34:25.842273+0000","last_became_active":"2023-01-15T14:34:00.240936+0000","last_became_peered":"2023-01-15T14:34:00.240936+0000","last_unstale":"2023-01-15T14:34:25.842273+0000","last_undegraded":"2023-01-15T14:34:25.842273+0000","last_fullsized":"2023-01-15T14:34:25.842273+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.57876590299999997,"stat_sum":{"num_bytes":5361581,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":13342,"num_write":5,"num_write_kb":2626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":5949777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.20","version":"324'273","reported_seq":1392,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.549250+0000","last_change":"2023-01-15T14:34:26.549351+0000","last_active":"2023-01-15T14:34:26.549250+0000","last_peered":"2023-01-15T14:34:26.549250+0000","last_clean":"2023-01-15T14:34:26.549250+0000","last_became_active":"2023-01-15T14:33:55.472883+0000","last_became_peered":"2023-01-15T14:33:55.472883+0000","last_unstale":"2023-01-15T14:34:26.549250+0000","last_undegraded":"2023-01-15T14:34:26.549250+0000","last_fullsized":"2023-01-15T14:34:26.549250+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.377629667,"stat_sum":{"num_bytes":4251137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10864,"num_write":9,"num_write_kb":4140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4438968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.21","version":"321'138","reported_seq":1082,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.680020+0000","last_change":"2023-01-15T14:34:27.680117+0000","last_active":"2023-01-15T14:34:27.680020+0000","last_peered":"2023-01-15T14:34:27.680020+0000","last_clean":"2023-01-15T14:34:27.680020+0000","last_became_active":"2023-01-15T14:33:45.267321+0000","last_became_peered":"2023-01-15T14:33:45.267321+0000","last_unstale":"2023-01-15T14:34:27.680020+0000","last_undegraded":"2023-01-15T14:34:27.680020+0000","last_fullsized":"2023-01-15T14:34:27.680020+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.61457987700000005,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"1"}]},{"pgid":"3.22","version":"323'161","reported_seq":1358,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.914804+0000","last_change":"2023-01-15T14:34:20.015929+0000","last_active":"2023-01-15T14:34:21.914804+0000","last_peered":"2023-01-15T14:34:21.914804+0000","last_clean":"2023-01-15T14:34:21.914804+0000","last_became_active":"2023-01-15T14:33:55.378253+0000","last_became_peered":"2023-01-15T14:33:55.378253+0000","last_unstale":"2023-01-15T14:34:21.914804+0000","last_undegraded":"2023-01-15T14:34:21.914804+0000","last_fullsized":"2023-01-15T14:34:21.914804+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.001164259,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":772,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.162879+0000","last_change":"2023-01-15T14:34:25.162827+0000","last_active":"2023-01-15T14:34:25.162879+0000","last_peered":"2023-01-15T14:34:25.162879+0000","last_clean":"2023-01-15T14:34:25.162879+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:34:25.162879+0000","last_undegraded":"2023-01-15T14:34:25.162879+0000","last_fullsized":"2023-01-15T14:34:25.162879+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.811229019,"stat_sum":{"num_bytes":3802136,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":21818,"num_write":18,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.24","version":"371'265","reported_seq":1430,"reported_epoch":373,"state":"active+clean","last_fresh":"2023-01-15T14:34:23.142787+0000","last_change":"2023-01-15T14:34:22.134264+0000","last_active":"2023-01-15T14:34:23.142787+0000","last_peered":"2023-01-15T14:34:23.142787+0000","last_clean":"2023-01-15T14:34:23.142787+0000","last_became_active":"2023-01-15T14:33:45.075309+0000","last_became_peered":"2023-01-15T14:33:45.075309+0000","last_unstale":"2023-01-15T14:34:23.142787+0000","last_undegraded":"2023-01-15T14:34:23.142787+0000","last_fullsized":"2023-01-15T14:34:23.142787+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.218163993,"stat_sum":{"num_bytes":7636039,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":22662,"num_write":10,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.25","version":"333'108","reported_seq":1158,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.384589+0000","last_change":"2023-01-15T14:34:27.384648+0000","last_active":"2023-01-15T14:34:27.384589+0000","last_peered":"2023-01-15T14:34:27.384589+0000","last_clean":"2023-01-15T14:34:27.384589+0000","last_became_active":"2023-01-15T14:33:45.168598+0000","last_became_peered":"2023-01-15T14:33:45.168598+0000","last_unstale":"2023-01-15T14:34:27.384589+0000","last_undegraded":"2023-01-15T14:34:27.384589+0000","last_fullsized":"2023-01-15T14:34:27.384589+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.167404157,"stat_sum":{"num_bytes":2638765,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3030,"num_write":2,"num_write_kb":2577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"1"}]},{"pgid":"3.26","version":"102'52","reported_seq":502,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.072485+0000","last_change":"2023-01-15T14:34:26.654666+0000","last_active":"2023-01-15T14:34:27.072485+0000","last_peered":"2023-01-15T14:34:27.072485+0000","last_clean":"2023-01-15T14:34:27.072485+0000","last_became_active":"2023-01-15T14:33:52.309110+0000","last_became_peered":"2023-01-15T14:33:52.309110+0000","last_unstale":"2023-01-15T14:34:27.072485+0000","last_undegraded":"2023-01-15T14:34:27.072485+0000","last_fullsized":"2023-01-15T14:34:27.072485+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.58107779299999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":1091,"reported_epoch":373,"state":"active+clean","last_fresh":"2023-01-15T14:34:23.243100+0000","last_change":"2023-01-15T14:34:20.224116+0000","last_active":"2023-01-15T14:34:23.243100+0000","last_peered":"2023-01-15T14:34:23.243100+0000","last_clean":"2023-01-15T14:34:23.243100+0000","last_became_active":"2023-01-15T14:33:45.171090+0000","last_became_peered":"2023-01-15T14:33:45.171090+0000","last_unstale":"2023-01-15T14:34:23.243100+0000","last_undegraded":"2023-01-15T14:34:23.243100+0000","last_fullsized":"2023-01-15T14:34:23.243100+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.20673013800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.28","version":"333'248","reported_seq":1418,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.913587+0000","last_change":"2023-01-15T14:34:21.726679+0000","last_active":"2023-01-15T14:34:21.913587+0000","last_peered":"2023-01-15T14:34:21.913587+0000","last_clean":"2023-01-15T14:34:21.913587+0000","last_became_active":"2023-01-15T14:33:45.366831+0000","last_became_peered":"2023-01-15T14:33:45.366831+0000","last_unstale":"2023-01-15T14:34:21.913587+0000","last_undegraded":"2023-01-15T14:34:21.913587+0000","last_fullsized":"2023-01-15T14:34:21.913587+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'248","last_scrub_stamp":"2023-01-15T14:34:20.620068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.620068+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:34.011","scrub_duration":0.40448800800000001,"objects_trimmed":0,"snaptrim_duration":0.50127574100000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":912,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:22.015075+0000","last_change":"2023-01-15T14:34:20.017312+0000","last_active":"2023-01-15T14:34:22.015075+0000","last_peered":"2023-01-15T14:34:22.015075+0000","last_clean":"2023-01-15T14:34:22.015075+0000","last_became_active":"2023-01-15T14:33:45.276960+0000","last_became_peered":"2023-01-15T14:33:45.276960+0000","last_unstale":"2023-01-15T14:34:22.015075+0000","last_undegraded":"2023-01-15T14:34:22.015075+0000","last_fullsized":"2023-01-15T14:34:22.015075+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.0021954629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1272,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.472987+0000","last_change":"2023-01-15T14:34:27.473080+0000","last_active":"2023-01-15T14:34:27.472987+0000","last_peered":"2023-01-15T14:34:27.472987+0000","last_clean":"2023-01-15T14:34:27.472987+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:34:27.472987+0000","last_undegraded":"2023-01-15T14:34:27.472987+0000","last_fullsized":"2023-01-15T14:34:27.472987+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.50344784499999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":80,"reported_epoch":375,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:25.063561+0000","last_change":"2023-01-15T14:34:25.063541+0000","last_active":"2023-01-15T14:34:25.063561+0000","last_peered":"2023-01-15T14:34:25.063561+0000","last_clean":"2023-01-15T14:34:25.063561+0000","last_became_active":"2023-01-15T14:33:57.261456+0000","last_became_peered":"2023-01-15T14:33:57.261456+0000","last_unstale":"2023-01-15T14:34:25.063561+0000","last_undegraded":"2023-01-15T14:34:25.063561+0000","last_fullsized":"2023-01-15T14:34:25.063561+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.812261444,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":461,"reported_epoch":372,"state":"active+clean","last_fresh":"2023-01-15T14:34:21.985234+0000","last_change":"2023-01-15T14:34:20.937488+0000","last_active":"2023-01-15T14:34:21.985234+0000","last_peered":"2023-01-15T14:34:21.985234+0000","last_clean":"2023-01-15T14:34:21.985234+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:34:21.985234+0000","last_undegraded":"2023-01-15T14:34:21.985234+0000","last_fullsized":"2023-01-15T14:34:21.985234+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.80575186899999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1241,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.972495+0000","last_change":"2023-01-15T14:34:26.653519+0000","last_active":"2023-01-15T14:34:26.972495+0000","last_peered":"2023-01-15T14:34:26.972495+0000","last_clean":"2023-01-15T14:34:26.972495+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:34:26.972495+0000","last_undegraded":"2023-01-15T14:34:26.972495+0000","last_fullsized":"2023-01-15T14:34:26.972495+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.50164169700000005,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.2e","version":"139'38","reported_seq":59,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.971142+0000","last_change":"2023-01-15T14:34:26.756748+0000","last_active":"2023-01-15T14:34:26.971142+0000","last_peered":"2023-01-15T14:34:26.971142+0000","last_clean":"2023-01-15T14:34:26.971142+0000","last_became_active":"2023-01-15T14:33:57.210434+0000","last_became_peered":"2023-01-15T14:33:57.210434+0000","last_unstale":"2023-01-15T14:34:26.971142+0000","last_undegraded":"2023-01-15T14:34:26.971142+0000","last_fullsized":"2023-01-15T14:34:26.971142+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.68353835600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1135,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.972158+0000","last_change":"2023-01-15T14:34:26.552248+0000","last_active":"2023-01-15T14:34:26.972158+0000","last_peered":"2023-01-15T14:34:26.972158+0000","last_clean":"2023-01-15T14:34:26.972158+0000","last_became_active":"2023-01-15T14:33:55.174128+0000","last_became_peered":"2023-01-15T14:33:55.174128+0000","last_unstale":"2023-01-15T14:34:26.972158+0000","last_undegraded":"2023-01-15T14:34:26.972158+0000","last_fullsized":"2023-01-15T14:34:26.972158+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.378192256,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"}]},{"pgid":"3.30","version":"206'252","reported_seq":1014,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:26.575991+0000","last_change":"2023-01-15T14:34:26.576215+0000","last_active":"2023-01-15T14:34:26.575991+0000","last_peered":"2023-01-15T14:34:26.575991+0000","last_clean":"2023-01-15T14:34:26.575991+0000","last_became_active":"2023-01-15T14:33:55.373498+0000","last_became_peered":"2023-01-15T14:33:55.373498+0000","last_unstale":"2023-01-15T14:34:26.575991+0000","last_undegraded":"2023-01-15T14:34:26.575991+0000","last_fullsized":"2023-01-15T14:34:26.575991+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:31:52.633422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:52.633422+0000","objects_scrubbed":7,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.40446849800000001,"stat_sum":{"num_bytes":5419866,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":25948,"num_write":35,"num_write_kb":13827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":13340458,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"109","length":"1"},{"start":"10b","length":"2"},{"start":"116","length":"1"}]},{"pgid":"3.31","version":"204'150","reported_seq":1258,"reported_epoch":377,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:27.283180+0000","last_change":"2023-01-15T14:34:27.003806+0000","last_active":"2023-01-15T14:34:27.283180+0000","last_peered":"2023-01-15T14:34:27.283180+0000","last_clean":"2023-01-15T14:34:27.283180+0000","last_became_active":"2023-01-15T14:31:39.938966+0000","last_became_peered":"2023-01-15T14:31:39.938966+0000","last_unstale":"2023-01-15T14:34:27.283180+0000","last_undegraded":"2023-01-15T14:34:27.283180+0000","last_fullsized":"2023-01-15T14:34:27.283180+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.90254259100000001,"objects_trimmed":0,"snaptrim_duration":0.90247673100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f4","length":"1"},{"start":"fe","length":"1"},{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"116","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":106,"stat_sum":{"num_bytes":382169755,"num_objects":178,"num_object_clones":125,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":17,"num_read":1018,"num_read_kb":1134476,"num_write":759,"num_write_kb":315355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":413434885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18895,"ondisk_log_size":18895,"up":318,"acting":318,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574926,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1112800,"kb_used_data":92591172,"kb_used_omap":8341,"kb_used_meta":0,"kb_avail":92586580,"statfs":{"total":95948165120,"available":94808657920,"internally_reserved":4702208,"allocated":94813360128,"data_stored":94813360128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8542185,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":3.452,"15min":3.8639999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1830000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":6.0389999999999997,"5min":5.6859999999999999,"15min":8.0340000000000007},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":36.622,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.66500000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":12.064,"5min":4.7549999999999999,"15min":5.9480000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":204.74199999999999,"5min":204.74199999999999,"15min":204.74199999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":11.260999999999999,"5min":9.1910000000000007,"15min":7.25},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":94.537000000000006,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.86799999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":6.4119999999999999,"15min":5.3310000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":25.065999999999999,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":15.106,"5min":4.7649999999999997,"15min":4.3730000000000002},"min":{"1min":0.59599999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":162.96799999999999,"5min":162.96799999999999,"15min":162.96799999999999},"last":0.72199999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.198,"5min":7.6299999999999999,"15min":3.1749999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":25.359999999999999,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.748},{"interface":"front","average":{"1min":0.86499999999999999,"5min":2.879,"15min":4.0609999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.266,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.80700000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.4430000000000001,"5min":4.2279999999999998,"15min":7.6859999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":29.690000000000001,"5min":96.185000000000002,"15min":96.185000000000002},"last":0.76500000000000001},{"interface":"front","average":{"1min":10.949999999999999,"5min":7.7789999999999999,"15min":3.2360000000000002},"min":{"1min":0.46200000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":142.44999999999999,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.82999999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.5489999999999999,"5min":4.3099999999999996,"15min":5.6539999999999999},"min":{"1min":0.63800000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":29.797999999999998,"5min":96.263999999999996,"15min":96.263999999999996},"last":1.026},{"interface":"front","average":{"1min":2.3519999999999999,"5min":2.6869999999999998,"15min":4.1920000000000002},"min":{"1min":0.55300000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":25.199000000000002,"5min":70.387,"15min":95.430000000000007},"last":0.89100000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.645,"5min":4.4189999999999996,"15min":7.1550000000000002},"min":{"1min":0.60999999999999999,"5min":0.496,"15min":0.496},"max":{"1min":29.876000000000001,"5min":96.299999999999997,"15min":96.299999999999997},"last":0.94699999999999995},{"interface":"front","average":{"1min":2.399,"5min":2.7749999999999999,"15min":1.6579999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":25.236999999999998,"5min":70.530000000000001,"15min":70.530000000000001},"last":0.97399999999999998}]}]},{"osd":6,"up_from":0,"seq":55834574927,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574927,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574925,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1002352,"kb_used_data":92700484,"kb_used_omap":14083,"kb_used_meta":0,"kb_avail":92697028,"statfs":{"total":95948165120,"available":94921756672,"internally_reserved":3538944,"allocated":94925295616,"data_stored":94925295616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14421733,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":22.818000000000001,"15min":10.882},"min":{"1min":0.505,"5min":0.44,"15min":0.44},"max":{"1min":5.0049999999999999,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.77400000000000002,"5min":2.9990000000000001,"15min":1.52},"min":{"1min":0.51200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.127,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.92800000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.9390000000000001,"5min":3.742,"15min":6.976},"min":{"1min":0.499,"5min":0.49299999999999999,"15min":0.47399999999999998},"max":{"1min":23.193000000000001,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":10.99,"5min":6.3819999999999997,"15min":2.6560000000000001},"min":{"1min":0.67100000000000004,"5min":0.52900000000000003,"15min":0.50800000000000001},"max":{"1min":162.976,"5min":162.976,"15min":162.976},"last":0.93999999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.0950000000000002,"5min":23.544,"15min":11.522},"min":{"1min":0.497,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":23.247,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.26,"5min":1.28,"15min":1.0329999999999999},"min":{"1min":0.61899999999999999,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.5789999999999997,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.76200000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":2.802,"15min":1.8260000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.006,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.995},{"interface":"front","average":{"1min":5.3380000000000001,"5min":24.337,"15min":12.212999999999999},"min":{"1min":0.68600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":58.317,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":0.94999999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88100000000000001,"15min":0.90700000000000003},"min":{"1min":0.61499999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.175,"5min":2.577,"15min":2.577},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.903,"15min":2.1669999999999998},"min":{"1min":0.69099999999999995,"5min":0.65200000000000002,"15min":0.55300000000000005},"max":{"1min":1.0700000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.81699999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":3.2389999999999999,"15min":4.8609999999999998},"min":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":23.329000000000001,"5min":85.721999999999994,"15min":99.780000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.999,"5min":21.631,"15min":10.651999999999999},"min":{"1min":0.79500000000000004,"5min":0.66700000000000004,"15min":0.58699999999999997},"max":{"1min":1.173,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.016}]},{"osd":7,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":3.238,"5min":3.5790000000000002,"15min":9.9450000000000003},"min":{"1min":0.73999999999999999,"5min":0.63400000000000001,"15min":0.59499999999999997},"max":{"1min":42.551000000000002,"5min":89.061000000000007,"15min":179.73500000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":2.274,"5min":23.071000000000002,"15min":15.183},"min":{"1min":0.78700000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":23.356000000000002,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.034}]}]},{"osd":3,"up_from":13,"seq":55834574925,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1068616,"kb_used_data":92630792,"kb_used_omap":14319,"kb_used_meta":0,"kb_avail":92630764,"statfs":{"total":95948165120,"available":94853902336,"internally_reserved":28672,"allocated":94853931008,"data_stored":94853931008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14662681,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":7.8860000000000001,"5min":8.048,"15min":5.9139999999999997},"min":{"1min":0.39500000000000002,"5min":0.255,"15min":0.255},"max":{"1min":74.245999999999995,"5min":174.52199999999999,"15min":174.52199999999999},"last":74.245999999999995},{"interface":"front","average":{"1min":7.9649999999999999,"5min":3.4990000000000001,"15min":6.2610000000000001},"min":{"1min":0.46600000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":74.280000000000001,"5min":74.280000000000001,"15min":93.674000000000007},"last":74.280000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":10.762,"5min":6.625,"15min":11.228999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":117.664,"5min":117.664,"15min":159.256},"last":117.664},{"interface":"front","average":{"1min":4.6909999999999998,"5min":11.208,"15min":4.2300000000000004},"min":{"1min":0.41899999999999998,"5min":0.317,"15min":0.317},"max":{"1min":47.963000000000001,"5min":174.578,"15min":174.578},"last":0.51400000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":2.4820000000000002,"15min":3.0600000000000001},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":13.113,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.54100000000000004},{"interface":"front","average":{"1min":2.3079999999999998,"5min":6.8120000000000003,"15min":3.7320000000000002},"min":{"1min":0.44,"5min":0.375,"15min":0.375},"max":{"1min":15.946999999999999,"5min":174.45099999999999,"15min":174.45099999999999},"last":13.837}]},{"osd":4,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":28.689,"15min":13.936999999999999},"min":{"1min":0.56299999999999994,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":14.234999999999999,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":14.234999999999999},{"interface":"front","average":{"1min":11.906000000000001,"5min":31.741,"15min":19.829000000000001},"min":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":117.59,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":117.59}]},{"osd":5,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":11.022,"5min":5.9219999999999997,"15min":11.242000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":117.744,"5min":117.744,"15min":159.43000000000001},"last":117.744},{"interface":"front","average":{"1min":0.91700000000000004,"5min":5.9829999999999997,"15min":2.6749999999999998},"min":{"1min":0.66200000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2210000000000001,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.72799999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":5.984,"15min":2.6819999999999999},"min":{"1min":0.68300000000000005,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.286,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.3979999999999999,"15min":2.9740000000000002},"min":{"1min":0.71099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6799999999999999,"5min":43.661000000000001,"15min":60.098999999999997},"last":0.82999999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.4119999999999999,"15min":3.161},"min":{"1min":0.753,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.758,"5min":43.712000000000003,"15min":60.165999999999997},"last":0.92900000000000005},{"interface":"front","average":{"1min":19.324999999999999,"5min":11.952,"15min":16.044},"min":{"1min":0.65400000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":142.02000000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":117.822}]}]},{"osd":2,"up_from":13,"seq":55834574926,"num_pgs":45,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1023264,"kb_used_data":92677672,"kb_used_omap":12025,"kb_used_meta":0,"kb_avail":92676116,"statfs":{"total":95948165120,"available":94900342784,"internally_reserved":1593344,"allocated":94901936128,"data_stored":94901936128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12314193,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":4.2469999999999999,"5min":4.2450000000000001,"15min":1.8420000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":78.456000000000003,"5min":78.456000000000003,"15min":78.456000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.4039999999999999,"15min":0.94399999999999995},"min":{"1min":0.36799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.35099999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":10.247,"5min":7.7949999999999999,"15min":3.1309999999999998},"min":{"1min":0.35099999999999998,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":128.749,"5min":143.071,"15min":143.071},"last":0.31900000000000001},{"interface":"front","average":{"1min":3.105,"5min":1.966,"15min":2.8919999999999999},"min":{"1min":0.35999999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":52.927999999999997,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.55700000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":3.8879999999999999,"5min":6.9180000000000001,"15min":9.2400000000000002},"min":{"1min":0.33700000000000002,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":34.244,"5min":143.13300000000001,"15min":183.53},"last":60.747999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":7.5039999999999996,"15min":3.012},"min":{"1min":0.41399999999999998,"5min":0.316,"15min":0.316},"max":{"1min":1.3959999999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.54500000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":5.9480000000000004,"15min":4.4160000000000004},"min":{"1min":0.67200000000000004,"5min":0.624,"15min":0.58399999999999996},"max":{"1min":1.2450000000000001,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.88300000000000001,"5min":4.7080000000000002,"15min":4.1079999999999997},"min":{"1min":0.57799999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.052,"5min":175.30500000000001,"15min":175.30500000000001},"last":0.64200000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":2.5590000000000002,"5min":2.7490000000000001,"15min":1.7},"min":{"1min":0.63500000000000001,"5min":0.57199999999999995,"15min":0.54800000000000004},"max":{"1min":34.256999999999998,"5min":143.262,"15min":143.262},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.95099999999999996,"5min":3.012,"15min":1.778},"min":{"1min":0.70199999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3160000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.78900000000000003}]},{"osd":6,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.839,"15min":1.3009999999999999},"min":{"1min":0.61199999999999999,"5min":0.59599999999999997,"15min":0.55000000000000004},"max":{"1min":1.276,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.86599999999999999},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8980000000000001,"15min":1.778},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":34.305999999999997,"5min":143.215,"15min":143.215},"last":0.73899999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":11.396000000000001,"5min":7.9800000000000004,"15min":4.6989999999999998},"min":{"1min":0.60599999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":176.44499999999999,"5min":176.44499999999999,"15min":176.44499999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8159999999999998,"15min":2.79},"min":{"1min":0.622,"5min":0.622,"15min":0.54500000000000004},"max":{"1min":34.207999999999998,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.78000000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":52,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1014588,"kb_used_data":92684884,"kb_used_omap":13703,"kb_used_meta":0,"kb_avail":92684792,"statfs":{"total":95948165120,"available":94909227008,"internally_reserved":94208,"allocated":94909321216,"data_stored":94909321216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14032409,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":2.0619999999999998,"5min":3.2669999999999999,"15min":3.6840000000000002},"min":{"1min":0.37,"5min":0.314,"15min":0.314},"max":{"1min":23.68,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":14.157,"5min":6.5789999999999997,"15min":5.0750000000000002},"min":{"1min":0.42299999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":176.01599999999999,"5min":176.01599999999999,"15min":176.01599999999999},"last":0.57199999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":4.8959999999999999,"5min":3.4329999999999998,"15min":1.7889999999999999},"min":{"1min":0.44400000000000001,"5min":0.245,"15min":0.245},"max":{"1min":69.277000000000001,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.629,"5min":1.173,"15min":3.1059999999999999},"min":{"1min":0.28399999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.873,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.40400000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":5.8440000000000003,"15min":15.172000000000001},"min":{"1min":0.38400000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.92500000000000004,"5min":245.637,"15min":282.01400000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":14.228999999999999,"5min":8.2870000000000008,"15min":7.5919999999999996},"min":{"1min":0.46500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":176.13900000000001,"5min":176.13900000000001,"15min":176.13900000000001},"last":0.74199999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":19.170000000000002,"5min":20.029,"15min":9.8279999999999994},"min":{"1min":0.56299999999999994,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":176.18600000000001,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":1.0860000000000001},{"interface":"front","average":{"1min":5.5510000000000002,"5min":15.916,"15min":7.7080000000000002},"min":{"1min":0.64300000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":69.611999999999995,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.69199999999999995}]},{"osd":5,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":5.2060000000000004,"5min":2.3799999999999999,"15min":1.482},"min":{"1min":0.56899999999999995,"5min":0.433,"15min":0.433},"max":{"1min":69.478999999999999,"5min":69.478999999999999,"15min":69.478999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":3.2770000000000001},"min":{"1min":0.66000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3089999999999999,"5min":4.0780000000000003,"15min":46.905999999999999},"last":0.88600000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":14.491,"5min":6.9109999999999996,"15min":5.6379999999999999},"min":{"1min":0.73199999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":176.249,"5min":176.249,"15min":176.249},"last":1.1679999999999999},{"interface":"front","average":{"1min":5.2629999999999999,"5min":2.431,"15min":1.5589999999999999},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":69.549999999999997,"5min":69.549999999999997,"15min":69.549999999999997},"last":0.72699999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":7.617,"5min":6.2309999999999999,"15min":6.7549999999999999},"min":{"1min":0.63900000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":95.888999999999996,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":14.544,"5min":6.9649999999999999,"15min":5.4880000000000004},"min":{"1min":0.78700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":176.22,"5min":176.22,"15min":176.22},"last":1.133}]}]},{"osd":0,"up_from":13,"seq":55834574927,"num_pgs":68,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1095180,"kb_used_data":92609812,"kb_used_omap":8284,"kb_used_meta":0,"kb_avail":92604200,"statfs":{"total":95948165120,"available":94826700800,"internally_reserved":5746688,"allocated":94832447488,"data_stored":94832447488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8483551,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":3.3460000000000001,"15min":1.53},"min":{"1min":0.34899999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":25.960999999999999,"5min":69.872,"15min":69.872},"last":0.39900000000000002},{"interface":"front","average":{"1min":11.249000000000001,"5min":5.335,"15min":2.2650000000000001},"min":{"1min":0.434,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":162.91200000000001,"5min":162.91200000000001,"15min":162.91200000000001},"last":0.438}]},{"osd":2,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":3.415,"15min":1.601},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0109999999999999,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":2.6320000000000001,"15min":1.3500000000000001},"min":{"1min":0.37,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86899999999999999,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.76200000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":7.8079999999999998,"15min":13.788},"min":{"1min":0.48199999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":26.027000000000001,"5min":199.08000000000001,"15min":284.62799999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":3.6070000000000002,"5min":7.867,"15min":3.1560000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33600000000000002},"max":{"1min":61.119,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.625}]},{"osd":4,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":16.664999999999999,"15min":6.2119999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1719999999999999,"5min":1486.309,"15min":1486.309},"last":0.96899999999999997},{"interface":"front","average":{"1min":2.1030000000000002,"5min":18.140999999999998,"15min":6.7300000000000004},"min":{"1min":0.63100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":26.137,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":0.879}]},{"osd":5,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":2.8980000000000001,"15min":7.4980000000000002},"min":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1930000000000001,"5min":98.411000000000001,"15min":176.34399999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.7070000000000001,"15min":1.2609999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.101,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.86299999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":2.1600000000000001,"5min":3.7890000000000001,"15min":2.0430000000000001},"min":{"1min":0.71899999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":26.082000000000001,"5min":70.259,"15min":70.259},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.026,"5min":3,"15min":1.756},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.57499999999999996},"max":{"1min":1.5409999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":0.98299999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":3.2770000000000001,"15min":11.539},"min":{"1min":0.70099999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":11.906000000000001,"5min":73.864999999999995,"15min":292.61099999999999},"last":0.90600000000000003},{"interface":"front","average":{"1min":2.1819999999999999,"5min":3.8090000000000002,"15min":2.0499999999999998},"min":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.58699999999999997},"max":{"1min":26.187000000000001,"5min":70.322000000000003,"15min":70.322000000000003},"last":0.80900000000000005}]}]}],"pool_statfs":[]}} 2023-01-15T14:34:29.186 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pgp_num 106 2023-01-15T14:34:29.190 INFO:tasks.rados.rados.0.smithi090.stdout:1433: done (0 left) 2023-01-15T14:34:29.190 INFO:tasks.rados.rados.0.smithi090.stdout:1434: read oid 23 snap -1 2023-01-15T14:34:29.190 INFO:tasks.rados.rados.0.smithi090.stdout:1434: expect (ObjNum 406 snap 108 seq_num 406) 2023-01-15T14:34:29.190 INFO:tasks.rados.rados.0.smithi090.stdout:1435: read oid 8 snap -1 2023-01-15T14:34:29.190 INFO:tasks.rados.rados.0.smithi090.stdout:1435: expect deleted 2023-01-15T14:34:29.191 INFO:tasks.rados.rados.0.smithi090.stdout:1436: snap_create 2023-01-15T14:34:29.191 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:34:29.191 INFO:tasks.rados.rados.0.smithi090.stdout:1435: done (1 left) 2023-01-15T14:34:29.191 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:29.192 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:29.335 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:29.335 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:34:29.341 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:29.182+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:29.342 INFO:tasks.rados.rados.0.smithi090.stdout:1434: expect (ObjNum 406 snap 108 seq_num 406) 2023-01-15T14:34:29.343 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:29.335+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:29.443 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:29.438+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:29.883 INFO:tasks.rados.rados.0.smithi090.stdout:1434: done (0 left) 2023-01-15T14:34:29.884 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:29.884 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:29.885 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:34:29.886 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:29.878+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:30.090 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:34:30.092 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:30.087+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:30.111 INFO:tasks.rados.rados.0.smithi090.stdout:1436: done (0 left) 2023-01-15T14:34:30.296 INFO:tasks.rados.rados.0.smithi090.stdout:1437: snap_remove snap 153 2023-01-15T14:34:30.298 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:30.298 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:30.298 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pgp_num to 106 2023-01-15T14:34:30.299 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:34:30.299 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:30.305 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:30.305 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:30.305 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:30.291+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:30.310 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:30.331 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:34:30.495 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:30.495 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:30.496 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:30.496 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:30.496 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:30.496 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:30.496 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:34:30.496 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:30.497 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:34:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:34:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:34:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:34:30.498 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:34:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:34:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:34:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:34:30.499 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:34:30.500 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:30.491+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:30.596 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:30.590+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:30.697 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:30.698 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:30.699 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:34:30.699 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:34:30.700 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:30.701 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:34:30.701 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:30.694+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:30.799 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:30.794+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:30.900 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:30.895+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:30.902 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:30.902 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:34:30.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:34:30.903 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:34:30.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:30.903 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:5 3.5 3:a1c47f99:::smithi09025264-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:30.903 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:30.903 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 365.07241097299999, 2023-01-15T14:34:30.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40548699700000002, 2023-01-15T14:34:30.904 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:30.904 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:30.904 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:30.904 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:30.904 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:30.905 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 5 2023-01-15T14:34:30.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.104 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:31.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.105 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:31.105 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:31.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.105 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:31.105 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:31.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.106 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.106 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.106 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:31.106 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781744+0000", 2023-01-15T14:34:31.107 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:31.107 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.107 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.107 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:31.107 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782566+0000", 2023-01-15T14:34:31.107 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000821578 2023-01-15T14:34:31.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:31.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782588+0000", 2023-01-15T14:34:31.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.226e-05 2023-01-15T14:34:31.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:31.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782601+0000", 2023-01-15T14:34:31.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2998e-05 2023-01-15T14:34:31.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882850+0000", 2023-01-15T14:34:31.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100248908 2023-01-15T14:34:31.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.112 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.112 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:34:31.112 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882871+0000", 2023-01-15T14:34:31.112 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0873000000000002e-05 2023-01-15T14:34:31.112 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.112 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.112 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.112 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083679+0000", 2023-01-15T14:34:31.112 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200808083 2023-01-15T14:34:31.113 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.114 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.114 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:31.114 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083948+0000", 2023-01-15T14:34:31.115 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000269342 2023-01-15T14:34:31.115 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.115 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.115 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:31.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084161+0000", 2023-01-15T14:34:31.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021326900000000001 2023-01-15T14:34:31.116 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.116 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:31.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084655+0000", 2023-01-15T14:34:31.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049371300000000001 2023-01-15T14:34:31.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.118 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:31.118 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084674+0000", 2023-01-15T14:34:31.118 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9012999999999999e-05 2023-01-15T14:34:31.118 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.119 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.119 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:31.119 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084924+0000", 2023-01-15T14:34:31.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024989899999999999 2023-01-15T14:34:31.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:31.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085221+0000", 2023-01-15T14:34:31.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029719600000000001 2023-01-15T14:34:31.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186953+0000", 2023-01-15T14:34:31.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10173174 2023-01-15T14:34:31.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187026+0000", 2023-01-15T14:34:31.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2957e-05 2023-01-15T14:34:31.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:31.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187124+0000", 2023-01-15T14:34:31.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8146000000000005e-05 2023-01-15T14:34:31.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.127 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:31.127 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187233+0000", 2023-01-15T14:34:31.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010931599999999999 2023-01-15T14:34:31.128 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.128 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:31.128 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.128 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:6 3.5 3:a1c47f99:::smithi09025264-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:31.129 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:31.129 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 365.06715810100002, 2023-01-15T14:34:31.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50419954199999995, 2023-01-15T14:34:31.129 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:31.129 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:31.129 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:31.129 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:31.129 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:31.129 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 6 2023-01-15T14:34:31.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.130 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:31.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:31.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:31.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:31.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:31.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:31.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786997+0000", 2023-01-15T14:34:31.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:31.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:31.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787486+0000", 2023-01-15T14:34:31.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048882999999999999 2023-01-15T14:34:31.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:31.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787514+0000", 2023-01-15T14:34:31.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7591999999999999e-05 2023-01-15T14:34:31.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:31.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787526+0000", 2023-01-15T14:34:31.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1894e-05 2023-01-15T14:34:31.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882966+0000", 2023-01-15T14:34:31.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095440450999999996 2023-01-15T14:34:31.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:31.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882973+0000", 2023-01-15T14:34:31.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2570000000000001e-06 2023-01-15T14:34:31.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084809+0000", 2023-01-15T14:34:31.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201835763 2023-01-15T14:34:31.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:31.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084928+0000", 2023-01-15T14:34:31.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118946 2023-01-15T14:34:31.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187347+0000", 2023-01-15T14:34:31.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102419285 2023-01-15T14:34:31.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:31.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187450+0000", 2023-01-15T14:34:31.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010286299999999999 2023-01-15T14:34:31.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:31.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187735+0000", 2023-01-15T14:34:31.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028429900000000001 2023-01-15T14:34:31.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:31.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188113+0000", 2023-01-15T14:34:31.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037883200000000001 2023-01-15T14:34:31.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:31.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188138+0000", 2023-01-15T14:34:31.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4110000000000001e-05 2023-01-15T14:34:31.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:31.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188389+0000", 2023-01-15T14:34:31.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025173900000000001 2023-01-15T14:34:31.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:31.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189251+0000", 2023-01-15T14:34:31.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086129800000000003 2023-01-15T14:34:31.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290989+0000", 2023-01-15T14:34:31.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101738524 2023-01-15T14:34:31.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291075+0000", 2023-01-15T14:34:31.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5469000000000004e-05 2023-01-15T14:34:31.268 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.268 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.268 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:31.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291144+0000", 2023-01-15T14:34:31.269 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:34:31.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.269 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:31.269 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291199+0000", 2023-01-15T14:34:31.269 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4413000000000002e-05 2023-01-15T14:34:31.269 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.269 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:31.270 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.270 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:7 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:31.270 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:31.270 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 365.06661164399998, 2023-01-15T14:34:31.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50407251399999997, 2023-01-15T14:34:31.270 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:31.270 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:31.270 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:31.270 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:31.271 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:31.271 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 7 2023-01-15T14:34:31.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.271 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:31.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:31.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:31.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787545+0000", 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787551+0000", 2023-01-15T14:34:31.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8669999999999999e-06 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787559+0000", 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9780000000000005e-06 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787564+0000", 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5709999999999997e-06 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883054+0000", 2023-01-15T14:34:31.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095489722999999999 2023-01-15T14:34:31.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:31.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883061+0000", 2023-01-15T14:34:31.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2760000000000003e-06 2023-01-15T14:34:31.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084987+0000", 2023-01-15T14:34:31.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20192545000000001 2023-01-15T14:34:31.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:31.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085190+0000", 2023-01-15T14:34:31.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020288899999999999 2023-01-15T14:34:31.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188374+0000", 2023-01-15T14:34:31.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103184449 2023-01-15T14:34:31.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:31.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188543+0000", 2023-01-15T14:34:31.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016925299999999999 2023-01-15T14:34:31.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:31.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188748+0000", 2023-01-15T14:34:31.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020473600000000001 2023-01-15T14:34:31.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:31.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188986+0000", 2023-01-15T14:34:31.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023780900000000001 2023-01-15T14:34:31.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:31.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189006+0000", 2023-01-15T14:34:31.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0604000000000001e-05 2023-01-15T14:34:31.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:31.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189084+0000", 2023-01-15T14:34:31.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7526000000000003e-05 2023-01-15T14:34:31.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:31.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189272+0000", 2023-01-15T14:34:31.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018762 2023-01-15T14:34:31.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291333+0000", 2023-01-15T14:34:31.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102061185 2023-01-15T14:34:31.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.281 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291466+0000", 2023-01-15T14:34:31.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013325699999999999 2023-01-15T14:34:31.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.281 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:31.281 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291535+0000", 2023-01-15T14:34:31.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9309999999999999e-05 2023-01-15T14:34:31.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.282 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:31.282 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291618+0000", 2023-01-15T14:34:31.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2773000000000002e-05 2023-01-15T14:34:31.282 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.282 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:31.282 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.282 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.282 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.283 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:8 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:31.283 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:31.283 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 365.06658566200002, 2023-01-15T14:34:31.283 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50442076000000002, 2023-01-15T14:34:31.287 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:31.287 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:31.288 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:31.288 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:31.288 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 8 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:31.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787571+0000", 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787575+0000", 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7890000000000001e-06 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:31.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787589+0000", 2023-01-15T14:34:31.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4212999999999999e-05 2023-01-15T14:34:31.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:31.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787594+0000", 2023-01-15T14:34:31.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1429999999999997e-06 2023-01-15T14:34:31.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983143+0000", 2023-01-15T14:34:31.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.195549003 2023-01-15T14:34:31.296 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:31.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983153+0000", 2023-01-15T14:34:31.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4369999999999996e-06 2023-01-15T14:34:31.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085286+0000", 2023-01-15T14:34:31.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102133064 2023-01-15T14:34:31.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:31.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085401+0000", 2023-01-15T14:34:31.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115176 2023-01-15T14:34:31.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189116+0000", 2023-01-15T14:34:31.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103715444 2023-01-15T14:34:31.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.429 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.429 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:31.429 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189221+0000", 2023-01-15T14:34:31.429 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010438 2023-01-15T14:34:31.429 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.429 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.429 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.429 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291724+0000", 2023-01-15T14:34:31.430 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102502998 2023-01-15T14:34:31.430 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.430 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.430 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:31.430 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291824+0000", 2023-01-15T14:34:31.432 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010010599999999999 2023-01-15T14:34:31.433 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.433 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:31.433 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291992+0000", 2023-01-15T14:34:31.433 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016856900000000001 2023-01-15T14:34:31.433 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.433 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:31.433 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.433 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.433 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.434 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:21 3.5 3:a75958b1:::smithi09025264-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:31.437 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:31.438 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 365.02249674699999, 2023-01-15T14:34:31.438 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.45851924500000002, 2023-01-15T14:34:31.438 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:31.438 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:31.438 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:31.438 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:31.438 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:31.438 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 21 2023-01-15T14:34:31.438 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.439 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:31.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:31.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:31.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:31.439 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:31.440 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.440 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:31.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831657+0000", 2023-01-15T14:34:31.440 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:31.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:31.441 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832814+0000", 2023-01-15T14:34:31.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011564909999999999 2023-01-15T14:34:31.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.442 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:31.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832864+0000", 2023-01-15T14:34:31.442 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0243e-05 2023-01-15T14:34:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832903+0000", 2023-01-15T14:34:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8704000000000001e-05 2023-01-15T14:34:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983202+0000", 2023-01-15T14:34:31.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.150299404 2023-01-15T14:34:31.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.444 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:31.444 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983208+0000", 2023-01-15T14:34:31.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0039999999999996e-06 2023-01-15T14:34:31.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.444 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.444 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185539+0000", 2023-01-15T14:34:31.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202330867 2023-01-15T14:34:31.445 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.445 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:31.445 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185803+0000", 2023-01-15T14:34:31.445 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026394299999999999 2023-01-15T14:34:31.445 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.445 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:31.445 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186037+0000", 2023-01-15T14:34:31.445 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023386300000000001 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186637+0000", 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060001800000000001 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186665+0000", 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8459999999999999e-05 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:31.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187045+0000", 2023-01-15T14:34:31.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037992899999999999 2023-01-15T14:34:31.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:31.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188201+0000", 2023-01-15T14:34:31.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001155309 2023-01-15T14:34:31.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289903+0000", 2023-01-15T14:34:31.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101702601 2023-01-15T14:34:31.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289985+0000", 2023-01-15T14:34:31.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1617000000000001e-05 2023-01-15T14:34:31.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:31.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290065+0000", 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0276000000000002e-05 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290180+0000", 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011464300000000001 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.450 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:22 3.5 3:a75958b1:::smithi09025264-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:31.451 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:31.451 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 365.01802905900001, 2023-01-15T14:34:31.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65790599599999999, 2023-01-15T14:34:31.451 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:31.451 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:31.451 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:31.451 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:31.451 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:31.451 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 22 2023-01-15T14:34:31.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.451 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836126+0000", 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836181+0000", 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4246999999999997e-05 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836212+0000", 2023-01-15T14:34:31.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1843e-05 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836226+0000", 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3216e-05 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186757+0000", 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35053103899999999 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.454 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186871+0000", 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114403 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290263+0000", 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103391976 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:31.455 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290413+0000", 2023-01-15T14:34:31.456 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014991200000000001 2023-01-15T14:34:31.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:31.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290627+0000", 2023-01-15T14:34:31.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021379299999999999 2023-01-15T14:34:31.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:31.845 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290861+0000", 2023-01-15T14:34:31.845 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023479600000000001 2023-01-15T14:34:31.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:31.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290888+0000", 2023-01-15T14:34:31.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6570000000000001e-05 2023-01-15T14:34:31.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:31.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290990+0000", 2023-01-15T14:34:31.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101723 2023-01-15T14:34:31.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:31.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291245+0000", 2023-01-15T14:34:31.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025543899999999999 2023-01-15T14:34:31.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392822+0000", 2023-01-15T14:34:31.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10157725500000001 2023-01-15T14:34:31.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493881+0000", 2023-01-15T14:34:31.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101058262 2023-01-15T14:34:31.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:31.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493964+0000", 2023-01-15T14:34:31.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2973000000000007e-05 2023-01-15T14:34:31.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:31.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494034+0000", 2023-01-15T14:34:31.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0432999999999995e-05 2023-01-15T14:34:31.850 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.850 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:31.850 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.850 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:23 3.5 3:a75958b1:::smithi09025264-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:31.850 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:31.851 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 365.01791393000002, 2023-01-15T14:34:31.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65796116900000001, 2023-01-15T14:34:31.851 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:31.851 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:31.851 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:31.896 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:31.896 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:31.896 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 23 2023-01-15T14:34:31.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:31.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836242+0000", 2023-01-15T14:34:31.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:31.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:31.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836256+0000", 2023-01-15T14:34:31.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3677e-05 2023-01-15T14:34:31.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:31.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836265+0000", 2023-01-15T14:34:31.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2769999999999994e-06 2023-01-15T14:34:31.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:31.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936398+0000", 2023-01-15T14:34:31.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100132559 2023-01-15T14:34:31.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289431+0000", 2023-01-15T14:34:31.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35303322399999998 2023-01-15T14:34:31.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:31.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289587+0000", 2023-01-15T14:34:31.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000155873 2023-01-15T14:34:31.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:31.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292052+0000", 2023-01-15T14:34:31.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0024647919999999999 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292180+0000", 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012810999999999999 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292373+0000", 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019258599999999999 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:31.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292597+0000", 2023-01-15T14:34:31.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022465500000000001 2023-01-15T14:34:31.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:31.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292618+0000", 2023-01-15T14:34:31.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0319999999999999e-05 2023-01-15T14:34:31.902 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.902 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:31.902 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292692+0000", 2023-01-15T14:34:31.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4643999999999999e-05 2023-01-15T14:34:31.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:31.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393103+0000", 2023-01-15T14:34:31.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100411156 2023-01-15T14:34:31.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493827+0000", 2023-01-15T14:34:31.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100724143 2023-01-15T14:34:31.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:31.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494085+0000", 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000257891 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494141+0000", 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6132999999999999e-05 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494204+0000", 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2966000000000007e-05 2023-01-15T14:34:31.905 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.906 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:31.906 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:31.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.906 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:24 3.5 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:31.906 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:31.906 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 364.91773250400001, 2023-01-15T14:34:31.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.55808677600000001, 2023-01-15T14:34:31.906 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:31.906 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:31.907 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:31.907 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:31.907 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:31.907 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 24 2023-01-15T14:34:31.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.907 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:31.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:31.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:31.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:31.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:31.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:31.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:31.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936423+0000", 2023-01-15T14:34:31.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:31.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:31.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936433+0000", 2023-01-15T14:34:31.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0443e-05 2023-01-15T14:34:31.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:31.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936451+0000", 2023-01-15T14:34:31.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7842999999999998e-05 2023-01-15T14:34:31.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:31.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:31.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:31.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936461+0000", 2023-01-15T14:34:32.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0057e-05 2023-01-15T14:34:32.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289693+0000", 2023-01-15T14:34:32.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.353231617 2023-01-15T14:34:32.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:32.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289814+0000", 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120763 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392923+0000", 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10310955300000001 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393072+0000", 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000148469 2023-01-15T14:34:32.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494242+0000", 2023-01-15T14:34:32.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117034499999999 2023-01-15T14:34:32.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:32.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494363+0000", 2023-01-15T14:34:32.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120703 2023-01-15T14:34:32.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:32.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494511+0000", 2023-01-15T14:34:32.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014873299999999999 2023-01-15T14:34:32.016 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.016 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:32.016 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.016 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:78 3.c 3:3137f0cb:::smithi09025264-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:32.016 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 364.52776650800001, 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20331801699999999, 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 78 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:32.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:32.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:32.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:32.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:32.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326388+0000", 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326672+0000", 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028367900000000001 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326713+0000", 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1539999999999999e-05 2023-01-15T14:34:32.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:32.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326729+0000", 2023-01-15T14:34:32.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.519e-05 2023-01-15T14:34:32.061 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.061 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.061 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.061 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326879+0000", 2023-01-15T14:34:32.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015064899999999999 2023-01-15T14:34:32.061 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.061 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:32.062 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327096+0000", 2023-01-15T14:34:32.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021682299999999999 2023-01-15T14:34:32.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.062 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:34:32.062 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327308+0000", 2023-01-15T14:34:32.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000211366 2023-01-15T14:34:32.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.063 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:32.063 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327757+0000", 2023-01-15T14:34:32.063 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000449009 2023-01-15T14:34:32.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.063 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:32.063 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327776+0000", 2023-01-15T14:34:32.063 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9201000000000001e-05 2023-01-15T14:34:32.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.064 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:32.064 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328012+0000", 2023-01-15T14:34:32.064 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236054 2023-01-15T14:34:32.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.064 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.064 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:32.064 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429534+0000", 2023-01-15T14:34:32.064 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10152201499999999 2023-01-15T14:34:32.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:32.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529340+0000", 2023-01-15T14:34:32.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099806249 2023-01-15T14:34:32.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:32.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529437+0000", 2023-01-15T14:34:32.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6949999999999998e-05 2023-01-15T14:34:32.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:32.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529587+0000", 2023-01-15T14:34:32.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015008600000000001 2023-01-15T14:34:32.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.182 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:32.182 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529709+0000", 2023-01-15T14:34:32.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121636 2023-01-15T14:34:32.182 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.182 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:32.182 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.183 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:79 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:32.183 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:32.184 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 364.52739691199997, 2023-01-15T14:34:32.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20348171300000001, 2023-01-15T14:34:32.184 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:32.185 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:32.185 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:32.219 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:32.219 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:32.219 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 79 2023-01-15T14:34:32.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.220 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:32.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:32.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:32.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:32.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:32.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:32.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326759+0000", 2023-01-15T14:34:32.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:32.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:32.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326770+0000", 2023-01-15T14:34:32.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:34:32.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:32.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326791+0000", 2023-01-15T14:34:32.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1556000000000002e-05 2023-01-15T14:34:32.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:32.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326818+0000", 2023-01-15T14:34:32.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6979999999999999e-05 2023-01-15T14:34:32.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.427993+0000", 2023-01-15T14:34:32.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117517199999999 2023-01-15T14:34:32.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:32.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428266+0000", 2023-01-15T14:34:32.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027233299999999998 2023-01-15T14:34:32.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:34:32.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428696+0000", 2023-01-15T14:34:32.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042996499999999998 2023-01-15T14:34:32.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:32.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429057+0000", 2023-01-15T14:34:32.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000360952 2023-01-15T14:34:32.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:32.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429076+0000", 2023-01-15T14:34:32.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:34:32.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:32.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429160+0000", 2023-01-15T14:34:32.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:34:32.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:32.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429549+0000", 2023-01-15T14:34:32.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038884599999999998 2023-01-15T14:34:32.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:32.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529886+0000", 2023-01-15T14:34:32.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100336261 2023-01-15T14:34:32.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:32.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529978+0000", 2023-01-15T14:34:32.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2775999999999994e-05 2023-01-15T14:34:32.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:32.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530108+0000", 2023-01-15T14:34:32.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012958300000000001 2023-01-15T14:34:32.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:32.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530242+0000", 2023-01-15T14:34:32.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013406 2023-01-15T14:34:32.228 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.228 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:32.228 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.229 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:80 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:32.229 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:32.229 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 364.52725855300002, 2023-01-15T14:34:32.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20370874, 2023-01-15T14:34:32.229 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:32.229 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:32.229 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:32.229 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:32.230 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:32.230 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 80 2023-01-15T14:34:32.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.230 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:32.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:32.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:32.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:32.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326897+0000", 2023-01-15T14:34:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326906+0000", 2023-01-15T14:34:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6929999999999996e-06 2023-01-15T14:34:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326931+0000", 2023-01-15T14:34:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4757e-05 2023-01-15T14:34:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326945+0000", 2023-01-15T14:34:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.402e-05 2023-01-15T14:34:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429239+0000", 2023-01-15T14:34:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102294418 2023-01-15T14:34:32.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429487+0000", 2023-01-15T14:34:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024789300000000001 2023-01-15T14:34:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530309+0000", 2023-01-15T14:34:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100821459 2023-01-15T14:34:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530438+0000", 2023-01-15T14:34:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012928700000000001 2023-01-15T14:34:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530607+0000", 2023-01-15T14:34:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169483 2023-01-15T14:34:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:32.348 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.383 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:78 3.c e23/20 ondisk, result = 0)", 2023-01-15T14:34:32.383 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:32.383 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 364.52536683199997, 2023-01-15T14:34:32.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20094287799999999, 2023-01-15T14:34:32.383 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:32.384 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:32.384 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:32.384 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:32.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:32.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.384 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:32.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:32.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:32.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328789+0000", 2023-01-15T14:34:32.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:32.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:32.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328799+0000", 2023-01-15T14:34:32.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8980000000000008e-06 2023-01-15T14:34:32.386 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.386 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:32.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328820+0000", 2023-01-15T14:34:32.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1041999999999999e-05 2023-01-15T14:34:32.386 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.386 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:32.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328832+0000", 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1616e-05 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529404+0000", 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200572631 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529419+0000", 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4250000000000001e-05 2023-01-15T14:34:32.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529733+0000", 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000314693 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:201 3.5 3:a1c47f99:::smithi09025264-2:head [copy-from ver 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:32.388 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 363.44776043500002, 2023-01-15T14:34:32.389 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30931411399999997, 2023-01-15T14:34:32.389 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:32.389 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:32.389 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:32.389 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:32.389 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:32.389 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 201 2023-01-15T14:34:32.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.389 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:32.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:32.390 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:32.390 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.390 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.512 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406394+0000", 2023-01-15T14:34:32.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:34:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406408+0000", 2023-01-15T14:34:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4416999999999999e-05 2023-01-15T14:34:32.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:32.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406446+0000", 2023-01-15T14:34:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7339999999999998e-05 2023-01-15T14:34:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406467+0000", 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1137000000000001e-05 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406677+0000", 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000210275 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:32.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406902+0000", 2023-01-15T14:34:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022529000000000001 2023-01-15T14:34:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.513012+0000", 2023-01-15T14:34:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.106109527 2023-01-15T14:34:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514754+0000", 2023-01-15T14:34:32.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0017419779999999999 2023-01-15T14:34:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514773+0000", 2023-01-15T14:34:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8700000000000001e-05 2023-01-15T14:34:32.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.515531+0000", 2023-01-15T14:34:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075844300000000003 2023-01-15T14:34:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615334+0000", 2023-01-15T14:34:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099803327999999997 2023-01-15T14:34:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615376+0000", 2023-01-15T14:34:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1675999999999999e-05 2023-01-15T14:34:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715546+0000", 2023-01-15T14:34:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016992500000001 2023-01-15T14:34:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715626+0000", 2023-01-15T14:34:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0112999999999997e-05 2023-01-15T14:34:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715711+0000", 2023-01-15T14:34:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:34:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:32.667 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:202 3.5 3:a1c47f99:::smithi09025264-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 363.447647889, 2023-01-15T14:34:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31002637300000002, 2023-01-15T14:34:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:32.668 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:32.669 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 202 2023-01-15T14:34:32.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.669 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:32.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:32.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:32.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:32.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:32.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:32.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406508+0000", 2023-01-15T14:34:32.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:32.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:32.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406516+0000", 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.413e-06 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406534+0000", 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8936999999999998e-05 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406548+0000", 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3196e-05 2023-01-15T14:34:32.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407271+0000", 2023-01-15T14:34:32.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00072301500000000003 2023-01-15T14:34:32.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:34:32.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407466+0000", 2023-01-15T14:34:32.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000195009 2023-01-15T14:34:32.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.614996+0000", 2023-01-15T14:34:32.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.207530192 2023-01-15T14:34:32.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:32.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615227+0000", 2023-01-15T14:34:32.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000230979 2023-01-15T14:34:32.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716282+0000", 2023-01-15T14:34:32.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101055465 2023-01-15T14:34:32.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.674 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:32.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716409+0000", 2023-01-15T14:34:32.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012653999999999999 2023-01-15T14:34:32.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.674 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:32.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716536+0000", 2023-01-15T14:34:32.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012683 2023-01-15T14:34:32.675 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.675 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:32.675 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.675 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:189 3.d e23/20 3:b3f111b5:::smithi09025264-48:head v 23'10, mlcod=23'6)", 2023-01-15T14:34:32.675 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:32.675 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 363.40244324100001, 2023-01-15T14:34:32.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20400475100000001, 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:32.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.677 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:32.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451712+0000", 2023-01-15T14:34:32.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:32.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:32.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452272+0000", 2023-01-15T14:34:32.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00056035899999999997 2023-01-15T14:34:32.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:32.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452310+0000", 2023-01-15T14:34:32.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8145999999999997e-05 2023-01-15T14:34:32.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:32.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452324+0000", 2023-01-15T14:34:32.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4443999999999999e-05 2023-01-15T14:34:32.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652703+0000", 2023-01-15T14:34:32.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20037818199999999 2023-01-15T14:34:32.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652762+0000", 2023-01-15T14:34:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9302999999999998e-05 2023-01-15T14:34:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653302+0000", 2023-01-15T14:34:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054012199999999998 2023-01-15T14:34:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653333+0000", 2023-01-15T14:34:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.112e-05 2023-01-15T14:34:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653667+0000", 2023-01-15T14:34:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033337900000000002 2023-01-15T14:34:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653709+0000", 2023-01-15T14:34:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2469999999999998e-05 2023-01-15T14:34:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:32.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655719+0000", 2023-01-15T14:34:32.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020096699999999999 2023-01-15T14:34:32.825 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.825 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:269 3.2 e30/20 3:47dfe935:::smithi09025264-29:head v 30'11, mlcod=23'8)", 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 355.97639779999997, 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23185540399999999, 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:32.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:32.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:32.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:32.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:32.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:32.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:32.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:32.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878379+0000", 2023-01-15T14:34:32.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061979600000000004 2023-01-15T14:34:32.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:32.830 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878406+0000", 2023-01-15T14:34:32.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:34:32.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.830 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:32.830 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978515+0000", 2023-01-15T14:34:32.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100108461 2023-01-15T14:34:32.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.830 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.830 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078759+0000", 2023-01-15T14:34:32.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100244496 2023-01-15T14:34:32.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.831 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:32.831 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078796+0000", 2023-01-15T14:34:32.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6758999999999998e-05 2023-01-15T14:34:32.831 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.831 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.831 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:32.831 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079567+0000", 2023-01-15T14:34:32.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077137199999999999 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079593+0000", 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079844+0000", 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025075899999999998 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:32.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079910+0000", 2023-01-15T14:34:32.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6406000000000001e-05 2023-01-15T14:34:32.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.109615+0000", 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.029704323000000001 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:281 3.5 3:a75958b1:::smithi09025264-6:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 355.87547417100001, 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30574310900000001, 2023-01-15T14:34:32.869 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:32.870 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:32.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:32.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:32.870 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:32.870 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 281 2023-01-15T14:34:32.870 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.870 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:32.870 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.870 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:32.870 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:32.870 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:32.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978686+0000", 2023-01-15T14:34:32.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4429999999999999e-06 2023-01-15T14:34:32.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:32.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978690+0000", 2023-01-15T14:34:32.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0300000000000004e-06 2023-01-15T14:34:32.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978693+0000", 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.179e-06 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981592+0000", 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0028984140000000002 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:32.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981718+0000", 2023-01-15T14:34:32.874 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012616600000000001 2023-01-15T14:34:32.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:32.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981982+0000", 2023-01-15T14:34:32.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026445199999999998 2023-01-15T14:34:32.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:32.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982278+0000", 2023-01-15T14:34:32.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029514100000000001 2023-01-15T14:34:32.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982290+0000", 2023-01-15T14:34:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.271e-05 2023-01-15T14:34:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.990 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982343+0000", 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2914000000000002e-05 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982385+0000", 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2114999999999999e-05 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.181937+0000", 2023-01-15T14:34:32.991 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19955213099999999 2023-01-15T14:34:32.992 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.992 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.992 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:32.992 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284183+0000", 2023-01-15T14:34:32.992 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10224538 2023-01-15T14:34:32.992 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.992 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.992 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:32.992 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284318+0000", 2023-01-15T14:34:32.992 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000134843 2023-01-15T14:34:32.992 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.993 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.993 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:32.993 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284426+0000", 2023-01-15T14:34:32.993 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108513 2023-01-15T14:34:32.993 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.993 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:32.993 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:32.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.993 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.993 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:373 3.4 e34/20 3:2029fa71:::smithi09025264-39:head v 34'19, mlcod=26'16)", 2023-01-15T14:34:32.993 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:32.994 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 350.39449171199999, 2023-01-15T14:34:32.994 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20212622799999999, 2023-01-15T14:34:32.994 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:32.994 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:32.994 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:32.994 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.994 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:32.994 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:32.994 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.994 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.994 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459664+0000", 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:32.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460099+0000", 2023-01-15T14:34:32.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043515900000000001 2023-01-15T14:34:32.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:32.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460130+0000", 2023-01-15T14:34:32.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1590000000000001e-05 2023-01-15T14:34:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560226+0000", 2023-01-15T14:34:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100095248 2023-01-15T14:34:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560369+0000", 2023-01-15T14:34:32.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014381499999999999 2023-01-15T14:34:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560404+0000", 2023-01-15T14:34:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4650000000000002e-05 2023-01-15T14:34:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:32.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560804+0000", 2023-01-15T14:34:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000399823 2023-01-15T14:34:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560827+0000", 2023-01-15T14:34:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3612999999999999e-05 2023-01-15T14:34:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:32.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.000 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:33.000 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.561101+0000", 2023-01-15T14:34:33.000 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027355499999999998 2023-01-15T14:34:33.000 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661710+0000", 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100608716 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661792+0000", 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2006000000000001e-05 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:33.001 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:33.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.028 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:455 3.d e41/20 3:bd982ca3:::smithi09025264-38:head v 41'24, mlcod=38'18)", 2023-01-15T14:34:33.028 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:33.028 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 343.34473013899998, 2023-01-15T14:34:33.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20118033699999999, 2023-01-15T14:34:33.028 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:33.028 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:33.028 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:33.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:33.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:33.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:33.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:33.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:33.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:33.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:33.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509425+0000", 2023-01-15T14:34:33.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:33.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:33.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509441+0000", 2023-01-15T14:34:33.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5248e-05 2023-01-15T14:34:33.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:33.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509474+0000", 2023-01-15T14:34:33.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3531000000000001e-05 2023-01-15T14:34:33.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:33.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.609644+0000", 2023-01-15T14:34:33.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016994999999999 2023-01-15T14:34:33.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:33.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709872+0000", 2023-01-15T14:34:33.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10022840299999999 2023-01-15T14:34:33.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:33.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709906+0000", 2023-01-15T14:34:33.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3160000000000001e-05 2023-01-15T14:34:33.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:33.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710153+0000", 2023-01-15T14:34:33.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024735399999999998 2023-01-15T14:34:33.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:33.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710204+0000", 2023-01-15T14:34:33.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1075999999999997e-05 2023-01-15T14:34:33.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:33.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710280+0000", 2023-01-15T14:34:33.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6202e-05 2023-01-15T14:34:33.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:33.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710317+0000", 2023-01-15T14:34:33.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6387000000000002e-05 2023-01-15T14:34:33.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:33.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710607+0000", 2023-01-15T14:34:33.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029079699999999997 2023-01-15T14:34:33.036 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:33.036 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:33.036 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:33.145 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.145 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.145 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:497 3.d e44/20 3:bd982ca3:::smithi09025264-38:head v 44'31, mlcod=42'26)", 2023-01-15T14:34:33.145 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:33.145 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 341.079935277, 2023-01-15T14:34:33.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20082067100000001, 2023-01-15T14:34:33.146 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:33.146 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:33.146 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:33.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.152 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:33.152 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:33.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:33.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:33.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:33.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:33.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:33.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774220+0000", 2023-01-15T14:34:33.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:33.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:33.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774238+0000", 2023-01-15T14:34:33.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8107000000000001e-05 2023-01-15T14:34:33.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:33.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774263+0000", 2023-01-15T14:34:33.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4122999999999999e-05 2023-01-15T14:34:33.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:33.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774280+0000", 2023-01-15T14:34:33.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7039999999999999e-05 2023-01-15T14:34:33.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:33.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874060+0000", 2023-01-15T14:34:33.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099780698000000001 2023-01-15T14:34:33.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:33.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874130+0000", 2023-01-15T14:34:33.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9852999999999997e-05 2023-01-15T14:34:33.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:33.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874351+0000", 2023-01-15T14:34:33.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022091400000000001 2023-01-15T14:34:33.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:33.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874369+0000", 2023-01-15T14:34:33.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7876e-05 2023-01-15T14:34:33.194 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:33.194 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874433+0000", 2023-01-15T14:34:33.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3793000000000004e-05 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.974927+0000", 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10049393500000001 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.975043+0000", 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115949 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:33.195 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:33.196 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:33.196 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:33.196 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:33.196 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:33.196 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:34:33.197 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:34:33.200 INFO:tasks.rados.rados.0.smithi090.stdout:1437: done (0 left) 2023-01-15T14:34:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1438: read oid 26 snap -1 2023-01-15T14:34:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1438: expect deleted 2023-01-15T14:34:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1439: read oid 13 snap -1 2023-01-15T14:34:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1439: expect deleted 2023-01-15T14:34:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1440: copy_from oid 17 from oid 46 current snap is 154 2023-01-15T14:34:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1441: delete oid 14 current snap is 154 2023-01-15T14:34:33.202 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:34:33.205 INFO:tasks.rados.rados.0.smithi090.stdout:1438: done (3 left) 2023-01-15T14:34:33.205 INFO:tasks.rados.rados.0.smithi090.stdout:1439: done (2 left) 2023-01-15T14:34:33.206 INFO:tasks.rados.rados.0.smithi090.stdout:1441: done (1 left) 2023-01-15T14:34:33.206 INFO:tasks.rados.rados.0.smithi090.stdout:1442: copy_from oid 9 from oid 40 current snap is 154 2023-01-15T14:34:33.206 INFO:tasks.rados.rados.0.smithi090.stdout:1443: read oid 20 snap -1 2023-01-15T14:34:33.206 INFO:tasks.rados.rados.0.smithi090.stdout:1443: expect (ObjNum 451 snap 126 seq_num 451) 2023-01-15T14:34:33.207 INFO:tasks.rados.rados.0.smithi090.stdout:1444: delete oid 37 current snap is 154 2023-01-15T14:34:33.207 INFO:tasks.rados.rados.0.smithi090.stdout:1440: finishing copy_from to smithi09025264-17 2023-01-15T14:34:33.207 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 320 (ObjNum 506 snap 150 seq_num 506) dirty exists 2023-01-15T14:34:33.207 INFO:tasks.rados.rados.0.smithi090.stdout:1440: finishing copy_from racing read to smithi09025264-17 2023-01-15T14:34:33.207 INFO:tasks.rados.rados.0.smithi090.stdout:1440: done (3 left) 2023-01-15T14:34:33.208 INFO:tasks.rados.rados.0.smithi090.stdout:1444: done (2 left) 2023-01-15T14:34:33.208 INFO:tasks.rados.rados.0.smithi090.stdout:1445: rollback oid 3 current snap is 154 2023-01-15T14:34:33.208 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 3 to 139 2023-01-15T14:34:33.208 INFO:tasks.rados.rados.0.smithi090.stdout:1446: rollback oid 23 current snap is 154 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 23 to 149 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout:1447: snap_remove snap 152 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout:1447: done (4 left) 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout:1448: snap_create 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout:1443: expect (ObjNum 451 snap 126 seq_num 451) 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout:1446: finishing rollback tid 0 to smithi09025264-23 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout:1446: finishing rollback tid 1 to smithi09025264-23 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout:1446: finishing rollback tid 2 to smithi09025264-23 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 244 (ObjNum 406 snap 108 seq_num 406) dirty exists 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout:1442: finishing copy_from to smithi09025264-9 2023-01-15T14:34:33.209 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 313 (ObjNum 419 snap 112 seq_num 419) dirty exists 2023-01-15T14:34:33.304 INFO:tasks.rados.rados.0.smithi090.stdout:1442: finishing copy_from racing read to smithi09025264-9 2023-01-15T14:34:33.304 INFO:tasks.rados.rados.0.smithi090.stdout:1442: done (3 left) 2023-01-15T14:34:33.304 INFO:tasks.rados.rados.0.smithi090.stdout:1443: done (2 left) 2023-01-15T14:34:33.305 INFO:tasks.rados.rados.0.smithi090.stdout:1446: done (1 left) 2023-01-15T14:34:33.305 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:33.305 INFO:tasks.rados.rados.0.smithi090.stdout:1445: finishing rollback tid 0 to smithi09025264-3 2023-01-15T14:34:33.305 INFO:tasks.rados.rados.0.smithi090.stdout:1445: finishing rollback tid 1 to smithi09025264-3 2023-01-15T14:34:33.305 INFO:tasks.rados.rados.0.smithi090.stdout:1445: finishing rollback tid 2 to smithi09025264-3 2023-01-15T14:34:33.305 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 381 (ObjNum 464 snap 133 seq_num 464) dirty exists 2023-01-15T14:34:33.305 INFO:tasks.rados.rados.0.smithi090.stdout:1445: done (0 left) 2023-01-15T14:34:33.305 INFO:tasks.rados.rados.0.smithi090.stdout:1448: done (0 left) 2023-01-15T14:34:33.305 INFO:tasks.rados.rados.0.smithi090.stdout:1449: snap_remove snap 145 2023-01-15T14:34:33.306 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:33.191+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:33.310 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:33.303+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:33.316 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:33.316 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:33.317 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:34:33.317 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:33.326 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:34:33.408 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:33.403+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:33.467 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:33.474 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:33.477 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:33.478 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:33.478 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:34:33.478 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:34:33.478 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:33.487 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:34:33.492 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:33.523 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:33.524 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:33.519+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:33.526 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:33.546 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:33.629 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:33.629 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:33.630 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:33.630 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:33.630 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:34:33.631 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:34:33.631 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:34:33.631 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:34:33.631 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:34:33.631 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:34:33.631 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:34:33.631 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:34:33.632 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:34:33.632 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:33.623+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:33.730 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:33.727+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:33.806 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:33.806 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:33.807 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:34:33.831 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:33.823+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:33.917 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:34.084 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:34.085 INFO:tasks.rados.rados.0.smithi090.stdout:1449: done (0 left) 2023-01-15T14:34:34.085 INFO:tasks.rados.rados.0.smithi090.stdout:1450: delete oid 35 current snap is 155 2023-01-15T14:34:34.085 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:34.079+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:34.086 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:34:34.187 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:34.182+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:34.204 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:34.204 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:34.214 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:34:34.287 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:34.283+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:34.370 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:34.370 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:34.371 INFO:tasks.rados.rados.0.smithi090.stdout:1450: done (0 left) 2023-01-15T14:34:34.371 INFO:tasks.rados.rados.0.smithi090.stdout:1451: rollback oid 30 current snap is 155 2023-01-15T14:34:34.371 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 30 to 126 2023-01-15T14:34:34.372 INFO:tasks.rados.rados.0.smithi090.stdout:1452: write oid 50 current snap is 155 2023-01-15T14:34:34.372 INFO:tasks.rados.rados.0.smithi090.stdout:1452: seq_num 517 ranges {689219=415737,1770367=646847} 2023-01-15T14:34:34.373 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:34:34.373 INFO:tasks.rados.rados.0.smithi090.stdout:1452: writing smithi09025264-50 from 689219 to 1104956 tid 1 2023-01-15T14:34:34.377 INFO:tasks.rados.rados.0.smithi090.stdout:1452: writing smithi09025264-50 from 1770367 to 2417214 tid 2 2023-01-15T14:34:34.377 INFO:tasks.rados.rados.0.smithi090.stdout:1453: write oid 10 current snap is 155 2023-01-15T14:34:34.377 INFO:tasks.rados.rados.0.smithi090.stdout:1453: seq_num 518 ranges {706026=490986,1828849=790505} 2023-01-15T14:34:34.384 INFO:tasks.rados.rados.0.smithi090.stdout:1453: writing smithi09025264-10 from 706026 to 1197012 tid 1 2023-01-15T14:34:34.388 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:34.383+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:34.423 INFO:tasks.rados.rados.0.smithi090.stdout:1453: writing smithi09025264-10 from 1828849 to 2619354 tid 2 2023-01-15T14:34:34.424 INFO:tasks.rados.rados.0.smithi090.stdout:1454: snap_create 2023-01-15T14:34:34.424 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:34:34.475 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:34.546 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:34.547 INFO:tasks.rados.rados.0.smithi090.stdout:1452: finishing write tid 1 to smithi09025264-50 2023-01-15T14:34:34.547 INFO:tasks.rados.rados.0.smithi090.stdout:1452: finishing write tid 2 to smithi09025264-50 2023-01-15T14:34:34.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:34:34.548 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:34.543+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:34.648 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:34.642+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:34.710 INFO:tasks.rados.rados.0.smithi090.stdout:1453: finishing write tid 1 to smithi09025264-10 2023-01-15T14:34:34.710 INFO:tasks.rados.rados.0.smithi090.stdout:1452: finishing write tid 3 to smithi09025264-50 2023-01-15T14:34:34.710 INFO:tasks.rados.rados.0.smithi090.stdout:1453: finishing write tid 2 to smithi09025264-10 2023-01-15T14:34:34.710 INFO:tasks.rados.rados.0.smithi090.stdout:1452: finishing write tid 4 to smithi09025264-50 2023-01-15T14:34:34.710 INFO:tasks.rados.rados.0.smithi090.stdout:1452: oid 50 updating version 0 to 261 2023-01-15T14:34:34.711 INFO:tasks.rados.rados.0.smithi090.stdout:1452: oid 50 updating version 261 to 262 2023-01-15T14:34:34.711 INFO:tasks.rados.rados.0.smithi090.stdout:1452: oid 50 updating version 262 to 263 2023-01-15T14:34:34.711 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 50 v 263 (ObjNum 517 snap 155 seq_num 517) dirty exists 2023-01-15T14:34:34.711 INFO:tasks.rados.rados.0.smithi090.stdout:1452: left oid 50 (ObjNum 517 snap 155 seq_num 517) 2023-01-15T14:34:34.711 INFO:tasks.rados.rados.0.smithi090.stdout:1452: done (2 left) 2023-01-15T14:34:34.711 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:34:34.712 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:34.712 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:34.712 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:34:34.749 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:34.743+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:34.773 INFO:tasks.rados.rados.0.smithi090.stdout:1453: finishing write tid 3 to smithi09025264-10 2023-01-15T14:34:34.773 INFO:tasks.rados.rados.0.smithi090.stdout:1453: finishing write tid 4 to smithi09025264-10 2023-01-15T14:34:34.773 INFO:tasks.rados.rados.0.smithi090.stdout:1453: oid 10 updating version 0 to 431 2023-01-15T14:34:34.773 INFO:tasks.rados.rados.0.smithi090.stdout:1453: oid 10 updating version 431 to 432 2023-01-15T14:34:34.773 INFO:tasks.rados.rados.0.smithi090.stdout:1453: oid 10 version 432 is already newer than 430 2023-01-15T14:34:34.774 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 10 v 432 (ObjNum 518 snap 155 seq_num 518) dirty exists 2023-01-15T14:34:34.774 INFO:tasks.rados.rados.0.smithi090.stdout:1453: left oid 10 (ObjNum 518 snap 155 seq_num 518) 2023-01-15T14:34:34.774 INFO:tasks.rados.rados.0.smithi090.stdout:1453: done (1 left) 2023-01-15T14:34:34.774 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:34.816 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:34.874 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:34.876 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:34:34.876 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:34.871+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:34.948 INFO:tasks.rados.rados.0.smithi090.stdout:1451: finishing rollback tid 0 to smithi09025264-30 2023-01-15T14:34:34.989 INFO:tasks.rados.rados.0.smithi090.stdout:1451: finishing rollback tid 1 to smithi09025264-30 2023-01-15T14:34:34.990 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:34.986+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:35.034 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:35.034 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:35.044 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:34:35.075 INFO:tasks.rados.rados.0.smithi090.stdout:1451: finishing rollback tid 2 to smithi09025264-30 2023-01-15T14:34:35.076 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 238 (ObjNum 449 snap 125 seq_num 449) dirty exists 2023-01-15T14:34:35.076 INFO:tasks.rados.rados.0.smithi090.stdout:1451: done (0 left) 2023-01-15T14:34:35.094 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:35.087+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:35.186 INFO:tasks.rados.rados.0.smithi090.stdout:1454: done (0 left) 2023-01-15T14:34:35.186 INFO:tasks.rados.rados.0.smithi090.stdout:1455: snap_create 2023-01-15T14:34:35.195 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:35.195 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:35.196 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:35.190+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:35.201 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:34:35.204 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:34:35.297 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:35.291+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:35.327 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:34:35.327 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:34:35.327 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 2, 3, 4, 7, 1] 2023-01-15T14:34:35.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2023-01-15T14:34:35.357 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:35.357 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:35.369 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:34:35.411 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:35.407+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:35.516 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:35.516 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:35.507+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:35.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:34:35.616 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:35.610+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:35.675 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:34:35.676 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:35.676 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:35.677 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:34:35.677 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-01-15T14:34:35.627+0000 7fb90e2b4700 -1 mon.a@0(leader).osd e385 definitely_dead 0 2023-01-15T14:34:35.717 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:35.711+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:35.786 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:35.866 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:35.868 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:34:35.869 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:35.869 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:35.869 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:35.869 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:35.869 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:35.866+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:35.889 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:35.913 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:35.932 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:35.964 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:36.049 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:36.046+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:34:36.050 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:34:36.051 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:34:36.053 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:36.061 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:34:36.145 INFO:tasks.rados.rados.0.smithi090.stdout:1455: done (0 left) 2023-01-15T14:34:36.220 INFO:tasks.rados.rados.0.smithi090.stdout:1456: delete oid 40 current snap is 157 2023-01-15T14:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:34:36.221 INFO:teuthology.orchestra.run.smithi090.stderr:marked out osd.1. 2023-01-15T14:34:36.223 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:36.215+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:36.225 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:34:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:34:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:34:36.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.08897509399998, 2023-01-15T14:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.226 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:34:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:34:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:34:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:34:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:34:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:34:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:34:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:34:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:34:36.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:34:36.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:34:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:34:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:34:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:34:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:34:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:34:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:34:36.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:34:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:34:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:34:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:34:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:34:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:34:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:34:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:34:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:34:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:34:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:34:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:34:36.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:34:36.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.287 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 369.088571758, 2023-01-15T14:34:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:34:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.288 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:34:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:36.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:34:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:36.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:34:36.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:34:36.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:34:36.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:34:36.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:34:36.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:34:36.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:34:36.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:34:36.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:34:36.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:34:36.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:34:36.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:34:36.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:36.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:34:36.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:34:36.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:34:36.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:34:36.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:34:36.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:36.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:34:36.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:34:36.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:36.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:34:36.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:34:36.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:36.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:34:36.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:34:36.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:36.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:34:36.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:34:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:34:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:34:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:34:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:34:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:34:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:34:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:34:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:34:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 368.98834813799999, 2023-01-15T14:34:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:34:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:34:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:34:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:34:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:34:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:34:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:34:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:34:36.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:34:36.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:34:36.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:36.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:34:36.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:34:36.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:34:36.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:34:36.447 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.447 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.448 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:36.448 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:36.448 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 278.295032913, 2023-01-15T14:34:36.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:34:36.448 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.448 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:36.448 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.448 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.449 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:34:36.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.449 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:36.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:36.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:34:36.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:36.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:34:36.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:34:36.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:34:36.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:34:36.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:34:36.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:34:36.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:34:36.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:34:36.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:36.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:34:36.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:34:36.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:34:36.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:34:36.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 278.28857155899999, 2023-01-15T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:34:36.513 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.514 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:36.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.516 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:36.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:34:36.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:34:36.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:34:36.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:34:36.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:36.541 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 278.18814963400001, 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:36.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:34:36.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.546 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 278.18781214799998, 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.547 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:34:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:34:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:34:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:34:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:34:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:34:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:34:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:34:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 278.18775625199999, 2023-01-15T14:34:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:34:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:34:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:34:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:34:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:34:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:34:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:34:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:34:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:34:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:34:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:34:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:34:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:34:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:34:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 236.35450583400001, 2023-01-15T14:34:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:34:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:34:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:34:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:34:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:34:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:34:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:34:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:34:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:34:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:34:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:34:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:34:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:34:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:34:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:34:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:34:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:34:36.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:34:36.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:36.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:34:36.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:34:36.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:34:36.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:34:36.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.704 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.704 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.705 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.705 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2207 3.5 3:a1c47f99:::smithi09025264-2:head [write 2548663~763022] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:36.705 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:34:36.705 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.37974810599999, 2023-01-15T14:34:36.705 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79657873300000004, 2023-01-15T14:34:36.705 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.705 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:36.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.705 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.706 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.706 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2207 2023-01-15T14:34:36.706 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.706 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.706 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.706 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.706 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:34:36.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.707 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.707 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:34:36.707 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.707 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.707 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837590+0000", 2023-01-15T14:34:36.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:36.708 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.708 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.708 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.708 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837960+0000", 2023-01-15T14:34:36.708 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037055900000000001 2023-01-15T14:34:36.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837980+0000", 2023-01-15T14:34:36.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9953e-05 2023-01-15T14:34:36.709 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.709 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.709 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.709 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837988+0000", 2023-01-15T14:34:36.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2800000000000003e-06 2023-01-15T14:34:36.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927512+0000", 2023-01-15T14:34:36.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089523563 2023-01-15T14:34:36.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927645+0000", 2023-01-15T14:34:36.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013285799999999999 2023-01-15T14:34:36.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.711 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.711 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225466+0000", 2023-01-15T14:34:36.711 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29782133100000002 2023-01-15T14:34:36.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:36.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225553+0000", 2023-01-15T14:34:36.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6866000000000005e-05 2023-01-15T14:34:36.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:36.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225708+0000", 2023-01-15T14:34:36.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015455199999999999 2023-01-15T14:34:36.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:36.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225971+0000", 2023-01-15T14:34:36.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026317099999999998 2023-01-15T14:34:36.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226027+0000", 2023-01-15T14:34:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6601000000000002e-05 2023-01-15T14:34:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226470+0000", 2023-01-15T14:34:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044267500000000001 2023-01-15T14:34:36.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227372+0000", 2023-01-15T14:34:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090215200000000003 2023-01-15T14:34:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429314+0000", 2023-01-15T14:34:36.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20194183499999999 2023-01-15T14:34:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.633999+0000", 2023-01-15T14:34:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20468505400000001 2023-01-15T14:34:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:36.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634075+0000", 2023-01-15T14:34:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:34:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634170+0000", 2023-01-15T14:34:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4821999999999994e-05 2023-01-15T14:34:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2208 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (56) in=7b,truncate 3395047] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.379325513, 2023-01-15T14:34:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79650439900000003, 2023-01-15T14:34:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.718 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2208 2023-01-15T14:34:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:36.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838018+0000", 2023-01-15T14:34:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2549999999999997e-06 2023-01-15T14:34:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838031+0000", 2023-01-15T14:34:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2717999999999999e-05 2023-01-15T14:34:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838036+0000", 2023-01-15T14:34:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2240000000000003e-06 2023-01-15T14:34:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927676+0000", 2023-01-15T14:34:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089639620000000003 2023-01-15T14:34:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927804+0000", 2023-01-15T14:34:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012832300000000001 2023-01-15T14:34:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226096+0000", 2023-01-15T14:34:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29829207400000002 2023-01-15T14:34:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226347+0000", 2023-01-15T14:34:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025084800000000002 2023-01-15T14:34:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226750+0000", 2023-01-15T14:34:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040343200000000001 2023-01-15T14:34:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227147+0000", 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039682900000000002 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227600+0000", 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045321199999999998 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227859+0000", 2023-01-15T14:34:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259041 2023-01-15T14:34:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227895+0000", 2023-01-15T14:34:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5710999999999997e-05 2023-01-15T14:34:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429405+0000", 2023-01-15T14:34:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201509415 2023-01-15T14:34:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634300+0000", 2023-01-15T14:34:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20489528200000001 2023-01-15T14:34:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634409+0000", 2023-01-15T14:34:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109524 2023-01-15T14:34:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634518+0000", 2023-01-15T14:34:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108548 2023-01-15T14:34:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2209 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.37929669799999, 2023-01-15T14:34:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79724995499999995, 2023-01-15T14:34:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2209 2023-01-15T14:34:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838044+0000", 2023-01-15T14:34:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6120000000000001e-06 2023-01-15T14:34:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838056+0000", 2023-01-15T14:34:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1236000000000001e-05 2023-01-15T14:34:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838059+0000", 2023-01-15T14:34:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6940000000000001e-06 2023-01-15T14:34:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.027908+0000", 2023-01-15T14:34:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189848287 2023-01-15T14:34:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.028049+0000", 2023-01-15T14:34:36.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014132500000000001 2023-01-15T14:34:36.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227252+0000", 2023-01-15T14:34:36.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19920324 2023-01-15T14:34:36.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227341+0000", 2023-01-15T14:34:36.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8683000000000005e-05 2023-01-15T14:34:36.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634637+0000", 2023-01-15T14:34:36.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729567300000002 2023-01-15T14:34:36.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:36.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634980+0000", 2023-01-15T14:34:36.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034335000000000002 2023-01-15T14:34:36.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.635292+0000", 2023-01-15T14:34:36.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003123 2023-01-15T14:34:36.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.880 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.881 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2222 3.c 3:3137f0cb:::smithi09025264-20:head [write 3259733~152121] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:36.881 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:34:36.881 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.359445139, 2023-01-15T14:34:36.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77651572400000002, 2023-01-15T14:34:36.881 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.881 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:36.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.881 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.881 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2222 2023-01-15T14:34:36.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.882 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:34:36.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:34:36.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857891+0000", 2023-01-15T14:34:36.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:36.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857993+0000", 2023-01-15T14:34:36.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010276399999999999 2023-01-15T14:34:36.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858021+0000", 2023-01-15T14:34:36.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8022e-05 2023-01-15T14:34:36.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858034+0000", 2023-01-15T14:34:36.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2549e-05 2023-01-15T14:34:36.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025371+0000", 2023-01-15T14:34:36.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16733653100000001 2023-01-15T14:34:36.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025429+0000", 2023-01-15T14:34:36.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8922000000000003e-05 2023-01-15T14:34:36.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226869+0000", 2023-01-15T14:34:36.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20143931100000001 2023-01-15T14:34:36.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:36.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227088+0000", 2023-01-15T14:34:36.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219328 2023-01-15T14:34:36.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:36.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227443+0000", 2023-01-15T14:34:36.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035481 2023-01-15T14:34:36.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:36.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227830+0000", 2023-01-15T14:34:36.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000386702 2023-01-15T14:34:36.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:36.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227861+0000", 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1584000000000001e-05 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227939+0000", 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7354999999999994e-05 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228548+0000", 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060954800000000001 2023-01-15T14:34:36.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429817+0000", 2023-01-15T14:34:36.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20126904600000001 2023-01-15T14:34:36.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634125+0000", 2023-01-15T14:34:36.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20430780300000001 2023-01-15T14:34:36.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:36.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634259+0000", 2023-01-15T14:34:36.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134144 2023-01-15T14:34:36.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634410+0000", 2023-01-15T14:34:36.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150447 2023-01-15T14:34:36.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.918 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.918 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.920 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2223 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (57) in=7b,truncate 3411854] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:36.920 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:34:36.920 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.359286352, 2023-01-15T14:34:36.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87671107599999998, 2023-01-15T14:34:36.920 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.921 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:36.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2223 2023-01-15T14:34:36.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.922 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:34:36.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:34:36.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858052+0000", 2023-01-15T14:34:36.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:36.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:36.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858056+0000", 2023-01-15T14:34:36.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5619999999999997e-06 2023-01-15T14:34:36.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:36.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858067+0000", 2023-01-15T14:34:36.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0328e-05 2023-01-15T14:34:36.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:36.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858071+0000", 2023-01-15T14:34:36.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6439999999999999e-06 2023-01-15T14:34:36.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025446+0000", 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16737407200000001 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025503+0000", 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7952999999999999e-05 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227970+0000", 2023-01-15T14:34:36.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202466866 2023-01-15T14:34:36.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:36.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228139+0000", 2023-01-15T14:34:36.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016889100000000001 2023-01-15T14:34:36.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:36.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228284+0000", 2023-01-15T14:34:36.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014493700000000001 2023-01-15T14:34:36.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228470+0000", 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185541 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228481+0000", 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.167e-05 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228537+0000", 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5912999999999998e-05 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228563+0000", 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5613e-05 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.433181+0000", 2023-01-15T14:34:36.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20461838099999999 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734620+0000", 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301438386 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734682+0000", 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2373999999999996e-05 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734764+0000", 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1656999999999994e-05 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2224 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.35926130199999, 2023-01-15T14:34:36.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97704279199999999, 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2224 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:34:36.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858077+0000", 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:36.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:36.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858080+0000", 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858089+0000", 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3449999999999995e-06 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858093+0000", 2023-01-15T14:34:37.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9820000000000002e-06 2023-01-15T14:34:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025520+0000", 2023-01-15T14:34:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.167426556 2023-01-15T14:34:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025594+0000", 2023-01-15T14:34:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4339999999999996e-05 2023-01-15T14:34:37.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228608+0000", 2023-01-15T14:34:37.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20301395699999999 2023-01-15T14:34:37.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:37.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228752+0000", 2023-01-15T14:34:37.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014385 2023-01-15T14:34:37.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834878+0000", 2023-01-15T14:34:37.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60612591500000002 2023-01-15T14:34:37.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:37.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834987+0000", 2023-01-15T14:34:37.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108733 2023-01-15T14:34:37.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:37.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.835120+0000", 2023-01-15T14:34:37.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133688 2023-01-15T14:34:37.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.095 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:37.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.095 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3120 3.40 3:03b54904:::smithi09025264-12:head [write 578273~466432] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:37.095 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:34:37.095 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.285309420000004, 2023-01-15T14:34:37.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72661194799999995, 2023-01-15T14:34:37.095 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:37.096 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:37.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:37.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:37.096 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:37.096 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3120 2023-01-15T14:34:37.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.096 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:37.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:37.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932026+0000", 2023-01-15T14:34:37.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:37.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:37.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932397+0000", 2023-01-15T14:34:37.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037070099999999997 2023-01-15T14:34:37.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:37.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932419+0000", 2023-01-15T14:34:37.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1923999999999999e-05 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932435+0000", 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6509999999999999e-05 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933751+0000", 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013156050000000001 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:37.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933765+0000", 2023-01-15T14:34:37.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3952e-05 2023-01-15T14:34:37.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238456+0000", 2023-01-15T14:34:37.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30469085000000001 2023-01-15T14:34:37.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:37.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238797+0000", 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034117300000000001 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239214+0000", 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000416845 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240273+0000", 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058811 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:37.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240292+0000", 2023-01-15T14:34:37.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:34:37.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:37.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240555+0000", 2023-01-15T14:34:37.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026299500000000003 2023-01-15T14:34:37.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:37.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242072+0000", 2023-01-15T14:34:37.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015175550000000001 2023-01-15T14:34:37.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:37.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244038+0000", 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001966176 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658099+0000", 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41406103300000002 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658248+0000", 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148577 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:37.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658642+0000", 2023-01-15T14:34:37.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039346900000000002 2023-01-15T14:34:37.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.190 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:37.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.190 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3121 3.40 3:03b54904:::smithi09025264-12:head [write 1739595~452815] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:37.190 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:34:37.191 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.284877473999998, 2023-01-15T14:34:37.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72675957599999996, 2023-01-15T14:34:37.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:37.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:37.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:37.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:37.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:37.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3121 2023-01-15T14:34:37.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.192 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:37.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:37.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:34:37.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:37.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:37.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:34:37.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:37.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:37.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932460+0000", 2023-01-15T14:34:37.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:37.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:37.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932725+0000", 2023-01-15T14:34:37.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026482699999999998 2023-01-15T14:34:37.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932741+0000", 2023-01-15T14:34:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5880999999999999e-05 2023-01-15T14:34:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932750+0000", 2023-01-15T14:34:37.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2380000000000003e-06 2023-01-15T14:34:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034479+0000", 2023-01-15T14:34:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10172906399999999 2023-01-15T14:34:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034506+0000", 2023-01-15T14:34:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7336000000000001e-05 2023-01-15T14:34:37.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240672+0000", 2023-01-15T14:34:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206166074 2023-01-15T14:34:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:37.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240894+0000", 2023-01-15T14:34:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221189 2023-01-15T14:34:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241244+0000", 2023-01-15T14:34:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000350816 2023-01-15T14:34:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241923+0000", 2023-01-15T14:34:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067891200000000005 2023-01-15T14:34:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241943+0000", 2023-01-15T14:34:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0069999999999999e-05 2023-01-15T14:34:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243425+0000", 2023-01-15T14:34:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001481719 2023-01-15T14:34:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244087+0000", 2023-01-15T14:34:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066184700000000002 2023-01-15T14:34:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:37.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244230+0000", 2023-01-15T14:34:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143277 2023-01-15T14:34:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658784+0000", 2023-01-15T14:34:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.414553479 2023-01-15T14:34:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:37.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658904+0000", 2023-01-15T14:34:37.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120024 2023-01-15T14:34:37.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:37.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659221+0000", 2023-01-15T14:34:37.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031742700000000001 2023-01-15T14:34:37.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.249 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:37.249 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3122 3.40 3:03b54904:::smithi09025264-12:head [write 2722929~655344] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:34:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.284573375999997, 2023-01-15T14:34:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72706898900000005, 2023-01-15T14:34:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:37.250 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3122 2023-01-15T14:34:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:34:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932765+0000", 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933097+0000", 2023-01-15T14:34:37.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220299999999999 2023-01-15T14:34:37.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:37.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933106+0000", 2023-01-15T14:34:37.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5459999999999997e-06 2023-01-15T14:34:37.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:37.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933113+0000", 2023-01-15T14:34:37.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3419999999999996e-06 2023-01-15T14:34:37.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034598+0000", 2023-01-15T14:34:37.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101485804 2023-01-15T14:34:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034625+0000", 2023-01-15T14:34:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6859000000000002e-05 2023-01-15T14:34:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242147+0000", 2023-01-15T14:34:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20752189600000001 2023-01-15T14:34:37.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242517+0000", 2023-01-15T14:34:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037032200000000002 2023-01-15T14:34:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243157+0000", 2023-01-15T14:34:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063935700000000001 2023-01-15T14:34:37.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243809+0000", 2023-01-15T14:34:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065222000000000001 2023-01-15T14:34:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243827+0000", 2023-01-15T14:34:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8260000000000001e-05 2023-01-15T14:34:37.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244122+0000", 2023-01-15T14:34:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029469400000000002 2023-01-15T14:34:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:37.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245633+0000", 2023-01-15T14:34:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001511068 2023-01-15T14:34:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.346711+0000", 2023-01-15T14:34:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101077812 2023-01-15T14:34:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659372+0000", 2023-01-15T14:34:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31266154800000001 2023-01-15T14:34:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659488+0000", 2023-01-15T14:34:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115444 2023-01-15T14:34:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659835+0000", 2023-01-15T14:34:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034670300000000002 2023-01-15T14:34:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3123 3.40 3:03b54904:::smithi09025264-12:head [setxattr _header (58) in=7b,truncate 3378273] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.284203683000001, 2023-01-15T14:34:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82751564099999997, 2023-01-15T14:34:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3123 2023-01-15T14:34:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933139+0000", 2023-01-15T14:34:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7079999999999996e-06 2023-01-15T14:34:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933145+0000", 2023-01-15T14:34:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7849999999999997e-06 2023-01-15T14:34:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033331+0000", 2023-01-15T14:34:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100186076 2023-01-15T14:34:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034697+0000", 2023-01-15T14:34:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001366138 2023-01-15T14:34:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034718+0000", 2023-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.065e-05 2023-01-15T14:34:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244319+0000", 2023-01-15T14:34:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20960071 2023-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244576+0000", 2023-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025712899999999998 2023-01-15T14:34:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244932+0000", 2023-01-15T14:34:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035626000000000001 2023-01-15T14:34:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245315+0000", 2023-01-15T14:34:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038305199999999999 2023-01-15T14:34:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245330+0000", 2023-01-15T14:34:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5027e-05 2023-01-15T14:34:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245384+0000", 2023-01-15T14:34:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4267000000000001e-05 2023-01-15T14:34:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245654+0000", 2023-01-15T14:34:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026944000000000002 2023-01-15T14:34:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.436220+0000", 2023-01-15T14:34:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19056593799999999 2023-01-15T14:34:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760086+0000", 2023-01-15T14:34:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.323865927 2023-01-15T14:34:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760241+0000", 2023-01-15T14:34:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015535700000000001 2023-01-15T14:34:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760651+0000", 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040990699999999999 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3124 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 87.183977205000005, 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72792673799999996, 2023-01-15T14:34:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3124 2023-01-15T14:34:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:34:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:34:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033360+0000", 2023-01-15T14:34:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033372+0000", 2023-01-15T14:34:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2850000000000001e-05 2023-01-15T14:34:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033397+0000", 2023-01-15T14:34:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4349e-05 2023-01-15T14:34:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033412+0000", 2023-01-15T14:34:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5420000000000001e-05 2023-01-15T14:34:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036811+0000", 2023-01-15T14:34:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033984169999999999 2023-01-15T14:34:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036821+0000", 2023-01-15T14:34:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0213e-05 2023-01-15T14:34:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245712+0000", 2023-01-15T14:34:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20889102800000001 2023-01-15T14:34:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245912+0000", 2023-01-15T14:34:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020024600000000001 2023-01-15T14:34:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760781+0000", 2023-01-15T14:34:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51486881299999998 2023-01-15T14:34:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760970+0000", 2023-01-15T14:34:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018893600000000001 2023-01-15T14:34:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.761289+0000", 2023-01-15T14:34:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031870899999999997 2023-01-15T14:34:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:37.425 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:37.427 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:34:37.428 INFO:tasks.rados.rados.0.smithi090.stdout:1456: done (0 left) 2023-01-15T14:34:37.428 INFO:tasks.rados.rados.0.smithi090.stdout:1457: snap_remove snap 120 2023-01-15T14:34:37.428 INFO:tasks.rados.rados.0.smithi090.stdout:1457: done (0 left) 2023-01-15T14:34:37.428 INFO:tasks.rados.rados.0.smithi090.stdout:1458: snap_create 2023-01-15T14:34:37.428 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:37.422+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:37.528 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:37.522+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:37.666 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:37.666 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:37.666 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:34:37.667 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:37.667 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:34:37.667 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:37.659+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:37.767 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:37.759+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:37.781 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:37.781 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:37.781 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:34:37.781 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:34:37.781 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:37.791 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:34:37.868 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:37.862+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:37.902 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:34:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:34:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:34:37.902 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3 3.3 3:ca59e2b0:::smithi09025264-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 372.122118753, 2023-01-15T14:34:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62968594499999997, 2023-01-15T14:34:37.903 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:37.904 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3 2023-01-15T14:34:37.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:37.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775131+0000", 2023-01-15T14:34:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775138+0000", 2023-01-15T14:34:37.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775148+0000", 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7969999999999995e-06 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775161+0000", 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2469999999999999e-05 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.857974+0000", 2023-01-15T14:34:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.082812887000000002 2023-01-15T14:34:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.858147+0000", 2023-01-15T14:34:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000173883 2023-01-15T14:34:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106098+0000", 2023-01-15T14:34:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24795019600000001 2023-01-15T14:34:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106181+0000", 2023-01-15T14:34:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3248e-05 2023-01-15T14:34:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:34:37.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106330+0000", 2023-01-15T14:34:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014918200000000001 2023-01-15T14:34:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106510+0000", 2023-01-15T14:34:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017979399999999999 2023-01-15T14:34:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106527+0000", 2023-01-15T14:34:37.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6643999999999999e-05 2023-01-15T14:34:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106605+0000", 2023-01-15T14:34:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8650000000000001e-05 2023-01-15T14:34:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106644+0000", 2023-01-15T14:34:37.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8686999999999997e-05 2023-01-15T14:34:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404687+0000", 2023-01-15T14:34:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.298042745 2023-01-15T14:34:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404735+0000", 2023-01-15T14:34:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8306000000000001e-05 2023-01-15T14:34:37.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404772+0000", 2023-01-15T14:34:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6977000000000002e-05 2023-01-15T14:34:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404818+0000", 2023-01-15T14:34:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.6439e-05 2023-01-15T14:34:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:37.915 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:37.916 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:37.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.916 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:37.916 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:37.916 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 372.122080686, 2023-01-15T14:34:37.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62995383500000002, 2023-01-15T14:34:37.916 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:37.916 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:37.916 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:37.916 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:37.917 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:37.917 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4 2023-01-15T14:34:37.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.917 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:37.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:37.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:37.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:37.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775174+0000", 2023-01-15T14:34:37.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3669999999999999e-06 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775184+0000", 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5869999999999999e-06 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775190+0000", 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:37.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868440+0000", 2023-01-15T14:34:37.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.093250126000000003 2023-01-15T14:34:37.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:37.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868555+0000", 2023-01-15T14:34:37.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115003 2023-01-15T14:34:37.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.921 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:37.921 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.302962+0000", 2023-01-15T14:34:37.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.43440729700000003 2023-01-15T14:34:37.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.921 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:37.921 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303197+0000", 2023-01-15T14:34:37.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023508000000000001 2023-01-15T14:34:37.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404866+0000", 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101668723 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404985+0000", 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118853 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:37.922 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.405124+0000", 2023-01-15T14:34:37.923 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013951999999999999 2023-01-15T14:34:37.923 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:37.923 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:37.923 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:37.923 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.923 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.923 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:144 3.9 3:9988ac9d:::smithi09025264-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:37.924 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:37.924 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 371.03074213999997, 2023-01-15T14:34:37.924 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63423976199999998, 2023-01-15T14:34:37.924 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:37.924 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:37.924 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:37.924 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:37.924 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:37.924 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 144 2023-01-15T14:34:37.924 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.924 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:37.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.925 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:37.925 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:37.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:37.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.925 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:37.925 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:37.925 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:37.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866508+0000", 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866512+0000", 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5999999999999998e-06 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:37.926 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866523+0000", 2023-01-15T14:34:37.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0740000000000001e-05 2023-01-15T14:34:37.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:37.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866531+0000", 2023-01-15T14:34:37.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3310000000000004e-06 2023-01-15T14:34:37.927 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.927 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.927 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:37.927 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867744+0000", 2023-01-15T14:34:37.927 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012127399999999999 2023-01-15T14:34:37.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:37.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867810+0000", 2023-01-15T14:34:37.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6506000000000003e-05 2023-01-15T14:34:37.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.928 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.928 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:37.928 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097225+0000", 2023-01-15T14:34:37.928 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.22941514900000001 2023-01-15T14:34:37.928 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097295+0000", 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500386+0000", 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40309051699999998 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:37.929 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500521+0000", 2023-01-15T14:34:37.930 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013561700000000001 2023-01-15T14:34:37.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:37.930 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:37.930 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500749+0000", 2023-01-15T14:34:37.930 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022754900000000001 2023-01-15T14:34:37.930 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:37.930 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:37.930 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:37.930 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:37.930 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.024 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:509 3.9 3:9f86bdcb:::smithi09025264-22:head [write 2873212~588997] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:38.024 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:38.024 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 348.08231779699997, 2023-01-15T14:34:38.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69623497099999998, 2023-01-15T14:34:38.024 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.024 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:38.025 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.025 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.025 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.025 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 509 2023-01-15T14:34:38.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.025 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:38.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:38.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814931+0000", 2023-01-15T14:34:38.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:38.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815512+0000", 2023-01-15T14:34:38.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058082200000000004 2023-01-15T14:34:38.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815553+0000", 2023-01-15T14:34:38.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1081999999999997e-05 2023-01-15T14:34:38.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815571+0000", 2023-01-15T14:34:38.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8028000000000001e-05 2023-01-15T14:34:38.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906452+0000", 2023-01-15T14:34:38.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090880586999999999 2023-01-15T14:34:38.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906564+0000", 2023-01-15T14:34:38.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111976 2023-01-15T14:34:38.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208424+0000", 2023-01-15T14:34:38.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.301860199 2023-01-15T14:34:38.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208718+0000", 2023-01-15T14:34:38.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029347299999999999 2023-01-15T14:34:38.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:38.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209175+0000", 2023-01-15T14:34:38.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045692399999999999 2023-01-15T14:34:38.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:38.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209725+0000", 2023-01-15T14:34:38.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055027600000000004 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209751+0000", 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5812999999999998e-05 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.210006+0000", 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025564099999999997 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311247+0000", 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10124119199999999 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311454+0000", 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020695800000000001 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511022+0000", 2023-01-15T14:34:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19956729200000001 2023-01-15T14:34:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511100+0000", 2023-01-15T14:34:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8589000000000003e-05 2023-01-15T14:34:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511168+0000", 2023-01-15T14:34:38.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8039000000000004e-05 2023-01-15T14:34:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:510 3.9 3:9f86bdcb:::smithi09025264-22:head [setxattr _header (57) in=7b,truncate 3462209] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 348.08165433400001, 2023-01-15T14:34:38.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69609656399999997, 2023-01-15T14:34:38.037 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.037 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:38.037 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.038 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.038 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.038 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 510 2023-01-15T14:34:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:38.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815596+0000", 2023-01-15T14:34:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815606+0000", 2023-01-15T14:34:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7920000000000004e-06 2023-01-15T14:34:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815627+0000", 2023-01-15T14:34:38.041 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1239000000000001e-05 2023-01-15T14:34:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815637+0000", 2023-01-15T14:34:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0349e-05 2023-01-15T14:34:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906600+0000", 2023-01-15T14:34:38.042 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090963098000000006 2023-01-15T14:34:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906717+0000", 2023-01-15T14:34:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000116759 2023-01-15T14:34:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310000+0000", 2023-01-15T14:34:38.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40328322599999999 2023-01-15T14:34:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310249+0000", 2023-01-15T14:34:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024874400000000002 2023-01-15T14:34:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.044 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:38.045 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310559+0000", 2023-01-15T14:34:38.045 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310382 2023-01-15T14:34:38.045 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.045 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:38.045 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310951+0000", 2023-01-15T14:34:38.045 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039124500000000003 2023-01-15T14:34:38.045 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.045 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:38.045 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310975+0000", 2023-01-15T14:34:38.046 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4496000000000001e-05 2023-01-15T14:34:38.046 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.046 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.046 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:38.046 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311041+0000", 2023-01-15T14:34:38.046 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6309000000000005e-05 2023-01-15T14:34:38.046 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.046 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.046 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:38.046 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311260+0000", 2023-01-15T14:34:38.046 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021863399999999999 2023-01-15T14:34:38.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.047 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311588+0000", 2023-01-15T14:34:38.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032813599999999998 2023-01-15T14:34:38.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.047 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511351+0000", 2023-01-15T14:34:38.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19976275700000001 2023-01-15T14:34:38.048 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.048 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.048 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:38.048 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511507+0000", 2023-01-15T14:34:38.048 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015585100000000001 2023-01-15T14:34:38.048 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.048 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.048 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.048 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511693+0000", 2023-01-15T14:34:38.048 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186627 2023-01-15T14:34:38.049 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.049 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.049 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.049 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:511 3.9 3:9f86bdcb:::smithi09025264-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:38.049 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:38.049 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 348.08159696600001, 2023-01-15T14:34:38.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69666647500000001, 2023-01-15T14:34:38.049 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.049 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:38.050 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.050 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.050 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.050 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 511 2023-01-15T14:34:38.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.050 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.050 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:38.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.051 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:38.051 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.052 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815653+0000", 2023-01-15T14:34:38.052 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:38.052 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.052 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.052 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.052 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815660+0000", 2023-01-15T14:34:38.052 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0509999999999999e-06 2023-01-15T14:34:38.052 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.052 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.052 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.052 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815683+0000", 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2742999999999999e-05 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815698+0000", 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5133999999999999e-05 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908029+0000", 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.092331448999999996 2023-01-15T14:34:38.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908193+0000", 2023-01-15T14:34:38.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016342799999999999 2023-01-15T14:34:38.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311115+0000", 2023-01-15T14:34:38.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40292222300000002 2023-01-15T14:34:38.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.180 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311224+0000", 2023-01-15T14:34:38.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010888899999999999 2023-01-15T14:34:38.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.182 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.182 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511800+0000", 2023-01-15T14:34:38.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20057596899999999 2023-01-15T14:34:38.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512036+0000", 2023-01-15T14:34:38.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236247 2023-01-15T14:34:38.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512321+0000", 2023-01-15T14:34:38.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028457599999999998 2023-01-15T14:34:38.183 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.183 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.184 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.184 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.184 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:693 3.3 3:ca59e2b0:::smithi09025264-1:head [sparse-read 0~2016807] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:34:38.184 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:38.184 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 329.99886361900002, 2023-01-15T14:34:38.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70514199600000005, 2023-01-15T14:34:38.184 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.184 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:38.185 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.185 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.185 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.185 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 693 2023-01-15T14:34:38.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.185 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:38.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.186 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:38.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.186 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898387+0000", 2023-01-15T14:34:38.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:38.187 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.187 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.187 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.187 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898390+0000", 2023-01-15T14:34:38.187 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.252e-06 2023-01-15T14:34:38.187 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.187 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.187 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.188 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898401+0000", 2023-01-15T14:34:38.188 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0977e-05 2023-01-15T14:34:38.188 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.188 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.188 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.188 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.998532+0000", 2023-01-15T14:34:38.188 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100130443 2023-01-15T14:34:38.188 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.188 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.189 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.189 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602663+0000", 2023-01-15T14:34:38.189 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60413100500000005 2023-01-15T14:34:38.189 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.189 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.189 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.189 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602817+0000", 2023-01-15T14:34:38.189 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015440500000000001 2023-01-15T14:34:38.189 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.189 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.189 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.603530+0000", 2023-01-15T14:34:38.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071246299999999999 2023-01-15T14:34:38.190 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.190 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.190 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.190 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1421 3.a 3:5919d83e:::smithi09025264-13:head [write 1723287~549075] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:38.190 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:38.190 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 262.98494911, 2023-01-15T14:34:38.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61118175699999999, 2023-01-15T14:34:38.194 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.195 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:38.195 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.195 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.195 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.195 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1421 2023-01-15T14:34:38.195 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.195 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.195 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.195 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:38.195 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.196 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.196 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.196 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.196 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:38.196 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.196 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.196 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.196 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.197 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912300+0000", 2023-01-15T14:34:38.197 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:38.197 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.197 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.197 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.197 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913127+0000", 2023-01-15T14:34:38.197 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00082704000000000002 2023-01-15T14:34:38.197 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.197 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.198 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.198 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913158+0000", 2023-01-15T14:34:38.198 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1146999999999998e-05 2023-01-15T14:34:38.198 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.198 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.198 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.198 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913177+0000", 2023-01-15T14:34:38.198 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:34:38.198 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.198 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.200 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.200 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913384+0000", 2023-01-15T14:34:38.200 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000206589 2023-01-15T14:34:38.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:38.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913408+0000", 2023-01-15T14:34:38.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3493e-05 2023-01-15T14:34:38.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318579+0000", 2023-01-15T14:34:38.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40517134399999999 2023-01-15T14:34:38.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.341 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318806+0000", 2023-01-15T14:34:38.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022680899999999999 2023-01-15T14:34:38.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.341 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:38.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319060+0000", 2023-01-15T14:34:38.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254069 2023-01-15T14:34:38.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.342 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:38.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319527+0000", 2023-01-15T14:34:38.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000467081 2023-01-15T14:34:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319544+0000", 2023-01-15T14:34:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6949999999999999e-05 2023-01-15T14:34:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319778+0000", 2023-01-15T14:34:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023391500000000001 2023-01-15T14:34:38.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421126+0000", 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101348626 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522834+0000", 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10170761 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523142+0000", 2023-01-15T14:34:38.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854800000000001 2023-01-15T14:34:38.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:38.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523251+0000", 2023-01-15T14:34:38.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108411 2023-01-15T14:34:38.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523484+0000", 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023292500000000001 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1422 3.a 3:5919d83e:::smithi09025264-13:head [write 2781837~703343] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 262.98401031399999, 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61062001700000001, 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.346 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:38.347 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.347 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.348 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.348 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1422 2023-01-15T14:34:38.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.348 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:38.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.349 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.349 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:38.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.349 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.349 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913239+0000", 2023-01-15T14:34:38.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:38.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914075+0000", 2023-01-15T14:34:38.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00083594099999999998 2023-01-15T14:34:38.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.350 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.350 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914104+0000", 2023-01-15T14:34:38.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8938999999999999e-05 2023-01-15T14:34:38.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914121+0000", 2023-01-15T14:34:38.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7306999999999999e-05 2023-01-15T14:34:38.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914234+0000", 2023-01-15T14:34:38.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112737 2023-01-15T14:34:38.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:38.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914257+0000", 2023-01-15T14:34:38.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3156e-05 2023-01-15T14:34:38.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319798+0000", 2023-01-15T14:34:38.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40554153100000001 2023-01-15T14:34:38.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319991+0000", 2023-01-15T14:34:38.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019217999999999999 2023-01-15T14:34:38.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:38.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320237+0000", 2023-01-15T14:34:38.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024621099999999998 2023-01-15T14:34:38.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:38.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320759+0000", 2023-01-15T14:34:38.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052174700000000003 2023-01-15T14:34:38.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:38.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320776+0000", 2023-01-15T14:34:38.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7173e-05 2023-01-15T14:34:38.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.321080+0000", 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030424400000000001 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421162+0000", 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100082085 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522963+0000", 2023-01-15T14:34:38.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10180109399999999 2023-01-15T14:34:38.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523576+0000", 2023-01-15T14:34:38.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061277499999999997 2023-01-15T14:34:38.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:38.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523664+0000", 2023-01-15T14:34:38.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8002000000000002e-05 2023-01-15T14:34:38.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523861+0000", 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019685899999999999 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1423 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 3630312] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:38.357 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 262.98308777900002, 2023-01-15T14:34:38.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70862486899999999, 2023-01-15T14:34:38.358 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.358 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:38.358 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.358 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.358 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.358 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1423 2023-01-15T14:34:38.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.358 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:38.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:38.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914162+0000", 2023-01-15T14:34:38.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:38.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914172+0000", 2023-01-15T14:34:38.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0267e-05 2023-01-15T14:34:38.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914188+0000", 2023-01-15T14:34:38.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6212999999999999e-05 2023-01-15T14:34:38.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914201+0000", 2023-01-15T14:34:38.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3073e-05 2023-01-15T14:34:38.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914315+0000", 2023-01-15T14:34:38.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011357399999999999 2023-01-15T14:34:38.362 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.362 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.362 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:38.362 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914334+0000", 2023-01-15T14:34:38.362 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9004e-05 2023-01-15T14:34:38.362 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.362 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.362 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.362 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421258+0000", 2023-01-15T14:34:38.362 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.506924606 2023-01-15T14:34:38.362 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421518+0000", 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025950499999999999 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421843+0000", 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032498299999999999 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:38.364 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422222+0000", 2023-01-15T14:34:38.364 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037867599999999999 2023-01-15T14:34:38.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.364 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:38.364 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422245+0000", 2023-01-15T14:34:38.364 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3096999999999999e-05 2023-01-15T14:34:38.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.364 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:38.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422343+0000", 2023-01-15T14:34:38.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8487999999999996e-05 2023-01-15T14:34:38.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:38.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522890+0000", 2023-01-15T14:34:38.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10054658900000001 2023-01-15T14:34:38.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523075+0000", 2023-01-15T14:34:38.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185386 2023-01-15T14:34:38.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622151+0000", 2023-01-15T14:34:38.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099075724000000004 2023-01-15T14:34:38.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:38.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622244+0000", 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3387999999999994e-05 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622788+0000", 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000543973 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.367 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1424 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:38.368 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:38.368 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 262.983032407, 2023-01-15T14:34:38.368 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70984700199999995, 2023-01-15T14:34:38.368 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.368 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:38.368 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.368 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.368 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.368 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1424 2023-01-15T14:34:38.368 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.369 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:38.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.370 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.370 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.370 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.370 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:38.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.371 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.371 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.371 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.372 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914218+0000", 2023-01-15T14:34:38.372 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:38.372 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.372 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.372 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.372 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914224+0000", 2023-01-15T14:34:38.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6769999999999998e-06 2023-01-15T14:34:38.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.373 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914242+0000", 2023-01-15T14:34:38.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7759000000000001e-05 2023-01-15T14:34:38.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014331+0000", 2023-01-15T14:34:38.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088819 2023-01-15T14:34:38.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114642+0000", 2023-01-15T14:34:38.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100311027 2023-01-15T14:34:38.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:38.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114682+0000", 2023-01-15T14:34:38.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0028999999999999e-05 2023-01-15T14:34:38.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523918+0000", 2023-01-15T14:34:38.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40923631500000002 2023-01-15T14:34:38.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524124+0000", 2023-01-15T14:34:38.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020524900000000001 2023-01-15T14:34:38.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.378 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.378 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622921+0000", 2023-01-15T14:34:38.378 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098797069000000001 2023-01-15T14:34:38.378 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.378 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.378 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.379 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.623328+0000", 2023-01-15T14:34:38.379 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000407539 2023-01-15T14:34:38.379 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.379 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.379 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.624066+0000", 2023-01-15T14:34:38.380 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073762899999999995 2023-01-15T14:34:38.380 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.380 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.380 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.380 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.380 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.381 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1430 3.a 3:52df5138:::smithi09025264-25:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:38.381 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:38.381 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 262.88287264600001, 2023-01-15T14:34:38.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81872653100000004, 2023-01-15T14:34:38.381 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.381 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:38.381 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.382 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.382 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.382 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1430 2023-01-15T14:34:38.382 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.382 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.382 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.382 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.383 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:38.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.383 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.383 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:38.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.384 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014376+0000", 2023-01-15T14:34:38.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:38.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.384 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014391+0000", 2023-01-15T14:34:38.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4922999999999999e-05 2023-01-15T14:34:38.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014417+0000", 2023-01-15T14:34:38.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5698999999999999e-05 2023-01-15T14:34:38.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014435+0000", 2023-01-15T14:34:38.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7827000000000001e-05 2023-01-15T14:34:38.386 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.386 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114759+0000", 2023-01-15T14:34:38.386 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100324859 2023-01-15T14:34:38.386 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.386 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.386 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:38.386 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114771+0000", 2023-01-15T14:34:38.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:34:38.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.387 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.387 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.387 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524222+0000", 2023-01-15T14:34:38.387 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40945100800000001 2023-01-15T14:34:38.387 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524545+0000", 2023-01-15T14:34:38.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000322367 2023-01-15T14:34:38.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:38.389 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.626981+0000", 2023-01-15T14:34:38.389 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102436308 2023-01-15T14:34:38.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.389 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:38.390 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627874+0000", 2023-01-15T14:34:38.390 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089274899999999997 2023-01-15T14:34:38.390 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.390 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.390 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:38.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627930+0000", 2023-01-15T14:34:38.391 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6122000000000002e-05 2023-01-15T14:34:38.391 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.391 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.391 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.391 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628609+0000", 2023-01-15T14:34:38.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00067958700000000005 2023-01-15T14:34:38.392 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.392 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.392 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:38.392 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628798+0000", 2023-01-15T14:34:38.392 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018809 2023-01-15T14:34:38.393 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.393 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.393 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:38.393 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628870+0000", 2023-01-15T14:34:38.393 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2844999999999998e-05 2023-01-15T14:34:38.393 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.394 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.394 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.394 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832404+0000", 2023-01-15T14:34:38.394 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20353310299999999 2023-01-15T14:34:38.394 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.395 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.395 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:38.395 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832606+0000", 2023-01-15T14:34:38.395 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020207999999999999 2023-01-15T14:34:38.395 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.395 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.395 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.396 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833105+0000", 2023-01-15T14:34:38.396 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049945399999999998 2023-01-15T14:34:38.396 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.396 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.396 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.396 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.396 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.397 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1431 3.a 3:52df5138:::smithi09025264-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:38.397 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:38.397 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 262.88278135799999, 2023-01-15T14:34:38.397 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81967516600000001, 2023-01-15T14:34:38.397 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.398 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:38.398 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.398 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.398 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.398 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1431 2023-01-15T14:34:38.398 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.398 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:38.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.530 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:38.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014469+0000", 2023-01-15T14:34:38.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:38.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014477+0000", 2023-01-15T14:34:38.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9300000000000003e-06 2023-01-15T14:34:38.536 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.536 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014495+0000", 2023-01-15T14:34:38.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8363999999999999e-05 2023-01-15T14:34:38.537 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.537 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.537 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.537 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114610+0000", 2023-01-15T14:34:38.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10011521700000001 2023-01-15T14:34:38.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.538 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.538 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.538 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214948+0000", 2023-01-15T14:34:38.538 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100337739 2023-01-15T14:34:38.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.538 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.538 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:38.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214962+0000", 2023-01-15T14:34:38.539 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4516e-05 2023-01-15T14:34:38.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524874+0000", 2023-01-15T14:34:38.539 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30991196599999998 2023-01-15T14:34:38.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:34:38.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.525013+0000", 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013873999999999999 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628222+0000", 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103208721 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628504+0000", 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028245700000000001 2023-01-15T14:34:38.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833214+0000", 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.204709691 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833555+0000", 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034094000000000003 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.542 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.834145+0000", 2023-01-15T14:34:38.542 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059006900000000003 2023-01-15T14:34:38.542 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.542 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.542 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.543 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2439 3.3 3:c11c7d81:::smithi09025264-11:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:34:38.543 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:38.543 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 162.81387685300001, 2023-01-15T14:34:38.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70719632099999996, 2023-01-15T14:34:38.543 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.543 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:38.543 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.543 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.543 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.543 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2439 2023-01-15T14:34:38.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.544 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.544 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.544 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:38.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.544 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.544 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:38.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083373+0000", 2023-01-15T14:34:38.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:38.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083380+0000", 2023-01-15T14:34:38.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:34:38.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083391+0000", 2023-01-15T14:34:38.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0366e-05 2023-01-15T14:34:38.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083397+0000", 2023-01-15T14:34:38.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2740000000000004e-06 2023-01-15T14:34:38.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083648+0000", 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025052599999999999 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083666+0000", 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8848999999999999e-05 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237725+0000", 2023-01-15T14:34:38.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.154058361 2023-01-15T14:34:38.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237945+0000", 2023-01-15T14:34:38.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021978300000000001 2023-01-15T14:34:38.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.386967+0000", 2023-01-15T14:34:38.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.149022562 2023-01-15T14:34:38.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387230+0000", 2023-01-15T14:34:38.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026256100000000002 2023-01-15T14:34:38.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:38.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387695+0000", 2023-01-15T14:34:38.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046480099999999998 2023-01-15T14:34:38.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:38.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388112+0000", 2023-01-15T14:34:38.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041713200000000002 2023-01-15T14:34:38.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:38.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388130+0000", 2023-01-15T14:34:38.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8315999999999999e-05 2023-01-15T14:34:38.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:38.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388189+0000", 2023-01-15T14:34:38.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8665999999999997e-05 2023-01-15T14:34:38.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:38.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388829+0000", 2023-01-15T14:34:38.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00063992699999999999 2023-01-15T14:34:38.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.489989+0000", 2023-01-15T14:34:38.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116059199999999 2023-01-15T14:34:38.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790300+0000", 2023-01-15T14:34:38.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30031108699999998 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790396+0000", 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5721000000000001e-05 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790571+0000", 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017464999999999999 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.553 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.554 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.554 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2440 3.3 3:c11c7d81:::smithi09025264-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:34:38.554 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:38.554 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 162.81378512200001, 2023-01-15T14:34:38.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70759506000000005, 2023-01-15T14:34:38.554 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.554 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:38.554 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.554 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2440 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:38.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083465+0000", 2023-01-15T14:34:38.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:38.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083470+0000", 2023-01-15T14:34:38.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2630000000000003e-06 2023-01-15T14:34:38.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083481+0000", 2023-01-15T14:34:38.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0699999999999999e-05 2023-01-15T14:34:38.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083488+0000", 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8870000000000003e-06 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180331+0000", 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.096843296999999995 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180342+0000", 2023-01-15T14:34:38.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.031e-05 2023-01-15T14:34:38.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238012+0000", 2023-01-15T14:34:38.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.057670829 2023-01-15T14:34:38.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238257+0000", 2023-01-15T14:34:38.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000244959 2023-01-15T14:34:38.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590126+0000", 2023-01-15T14:34:38.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35186851699999999 2023-01-15T14:34:38.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590389+0000", 2023-01-15T14:34:38.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026276799999999998 2023-01-15T14:34:38.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790692+0000", 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20030350299999999 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790891+0000", 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019893999999999999 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.791061+0000", 2023-01-15T14:34:38.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169913 2023-01-15T14:34:38.562 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.562 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.562 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.563 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2450 3.a 3:52c3e133:::smithi09025264-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e230)", 2023-01-15T14:34:38.563 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:38.563 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 161.04993250999999, 2023-01-15T14:34:38.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.68251353100000001, 2023-01-15T14:34:38.563 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.563 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:38.563 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.563 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.563 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.564 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2450 2023-01-15T14:34:38.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.564 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:38.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847318+0000", 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847324+0000", 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:34:38.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847339+0000", 2023-01-15T14:34:38.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5582000000000001e-05 2023-01-15T14:34:38.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847349+0000", 2023-01-15T14:34:38.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0139999999999999e-05 2023-01-15T14:34:38.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947644+0000", 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10029455600000001 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947657+0000", 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2796000000000001e-05 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125638+0000", 2023-01-15T14:34:38.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17798127799999999 2023-01-15T14:34:38.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125780+0000", 2023-01-15T14:34:38.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014173000000000001 2023-01-15T14:34:38.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529035+0000", 2023-01-15T14:34:38.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40325542199999997 2023-01-15T14:34:38.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529366+0000", 2023-01-15T14:34:38.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033111700000000002 2023-01-15T14:34:38.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529832+0000", 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046592399999999999 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3390 3.1b 3:d85f42d9:::smithi09025264-45:head [write 1944996~791497] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:38.570 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 62.837253801000003, 2023-01-15T14:34:38.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.611364087, 2023-01-15T14:34:38.571 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.571 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:38.571 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.571 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.571 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.571 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3390 2023-01-15T14:34:38.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.571 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:38.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:38.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059995+0000", 2023-01-15T14:34:38.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:38.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061164+0000", 2023-01-15T14:34:38.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001168325 2023-01-15T14:34:38.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061194+0000", 2023-01-15T14:34:38.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.0485999999999999e-05 2023-01-15T14:34:38.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061213+0000", 2023-01-15T14:34:38.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.891e-05 2023-01-15T14:34:38.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159748+0000", 2023-01-15T14:34:38.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098535175000000003 2023-01-15T14:34:38.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:38.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159766+0000", 2023-01-15T14:34:38.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:34:38.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263011+0000", 2023-01-15T14:34:38.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10324554699999999 2023-01-15T14:34:38.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263253+0000", 2023-01-15T14:34:38.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000241568 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466464+0000", 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.203211006 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466638+0000", 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017412100000000001 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466911+0000", 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000272971 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467425+0000", 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00051373 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:38.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467457+0000", 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2558999999999998e-05 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467906+0000", 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000448985 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469895+0000", 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0019888810000000001 2023-01-15T14:34:38.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470611+0000", 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071556699999999996 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671178+0000", 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20056716699999999 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:38.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671277+0000", 2023-01-15T14:34:38.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9594999999999998e-05 2023-01-15T14:34:38.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671361+0000", 2023-01-15T14:34:38.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4048000000000006e-05 2023-01-15T14:34:38.581 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.581 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.582 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.582 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3391 3.1b 3:d85f42d9:::smithi09025264-45:head [write 3196560~554574] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:38.582 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:38.582 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 62.836010678000001, 2023-01-15T14:34:38.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71066412899999998, 2023-01-15T14:34:38.582 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.582 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:38.582 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3391 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061238+0000", 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062134+0000", 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089621299999999998 2023-01-15T14:34:38.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062160+0000", 2023-01-15T14:34:38.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5908999999999999e-05 2023-01-15T14:34:38.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062172+0000", 2023-01-15T14:34:38.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.189e-05 2023-01-15T14:34:38.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159827+0000", 2023-01-15T14:34:38.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097654866000000007 2023-01-15T14:34:38.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159846+0000", 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8516e-05 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263326+0000", 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103480766 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.587 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263535+0000", 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020861299999999999 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467576+0000", 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20404118600000001 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467736+0000", 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016000699999999999 2023-01-15T14:34:38.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467987+0000", 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025042199999999998 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468369+0000", 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038238600000000002 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:38.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468387+0000", 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7730000000000001e-05 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468719+0000", 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033272599999999999 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470317+0000", 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001597336 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570851+0000", 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10053426 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771618+0000", 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200767152 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771768+0000", 2023-01-15T14:34:38.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015015699999999999 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771905+0000", 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013621099999999999 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3392 3.1b 3:d85f42d9:::smithi09025264-45:head [setxattr _header (58) in=7b,truncate 3882483] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:38.592 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:38.593 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 62.835061443000001, 2023-01-15T14:34:38.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71008243400000004, 2023-01-15T14:34:38.593 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.593 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:38.593 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.593 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.593 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.593 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3392 2023-01-15T14:34:38.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.593 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:38.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.594 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062188+0000", 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062195+0000", 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2529999999999998e-06 2023-01-15T14:34:38.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062208+0000", 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3003000000000001e-05 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062215+0000", 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5200000000000003e-06 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159883+0000", 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097667532000000001 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159899+0000", 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6759999999999999e-05 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263853+0000", 2023-01-15T14:34:38.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10395361 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264116+0000", 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026336099999999999 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468602+0000", 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20448550900000001 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468867+0000", 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026495700000000002 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469376+0000", 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00050893100000000001 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:38.599 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469789+0000", 2023-01-15T14:34:38.600 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000413158 2023-01-15T14:34:38.600 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.600 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.600 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:38.600 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469806+0000", 2023-01-15T14:34:38.600 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7513e-05 2023-01-15T14:34:38.600 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.600 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.600 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:38.600 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469884+0000", 2023-01-15T14:34:38.600 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7816000000000002e-05 2023-01-15T14:34:38.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.601 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.601 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:38.601 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470332+0000", 2023-01-15T14:34:38.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044807700000000001 2023-01-15T14:34:38.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.601 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.601 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.601 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570976+0000", 2023-01-15T14:34:38.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10064426899999999 2023-01-15T14:34:38.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772058+0000", 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20108148200000001 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772169+0000", 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011120099999999999 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.602 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772272+0000", 2023-01-15T14:34:38.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000102982 2023-01-15T14:34:38.603 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.603 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.603 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.603 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3393 3.1b 3:d85f42d9:::smithi09025264-45:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:38.603 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:38.603 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 62.835027367000002, 2023-01-15T14:34:38.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71076842399999995, 2023-01-15T14:34:38.604 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:38.604 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:38.604 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:38.604 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:38.604 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:38.604 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3393 2023-01-15T14:34:38.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.604 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:38.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:38.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:38.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:38.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:38.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:38.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:38.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062223+0000", 2023-01-15T14:34:38.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:38.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:38.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062227+0000", 2023-01-15T14:34:38.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3900000000000003e-06 2023-01-15T14:34:38.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:38.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062237+0000", 2023-01-15T14:34:38.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4539999999999996e-06 2023-01-15T14:34:38.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062243+0000", 2023-01-15T14:34:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0329999999999997e-06 2023-01-15T14:34:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161412+0000", 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099168708999999994 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161424+0000", 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2415999999999999e-05 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.689 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264185+0000", 2023-01-15T14:34:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.1027609 2023-01-15T14:34:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264421+0000", 2023-01-15T14:34:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023640900000000001 2023-01-15T14:34:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469985+0000", 2023-01-15T14:34:38.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.205564001 2023-01-15T14:34:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470264+0000", 2023-01-15T14:34:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027918700000000002 2023-01-15T14:34:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772390+0000", 2023-01-15T14:34:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30212558299999998 2023-01-15T14:34:38.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772684+0000", 2023-01-15T14:34:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029361799999999999 2023-01-15T14:34:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772992+0000", 2023-01-15T14:34:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854100000000001 2023-01-15T14:34:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.692 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.693 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.693 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:38.693 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:38.693 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:38.693 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:34:38.694 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:34:38.696 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:34:38.697 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:38.690+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:38.697 INFO:tasks.rados.rados.0.smithi090.stdout:1458: done (0 left) 2023-01-15T14:34:38.698 INFO:tasks.rados.rados.0.smithi090.stdout:1459: rollback oid 39 current snap is 158 2023-01-15T14:34:38.698 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 39 to 155 2023-01-15T14:34:38.698 INFO:tasks.rados.rados.0.smithi090.stdout:1460: copy_from oid 28 from oid 41 current snap is 158 2023-01-15T14:34:38.698 INFO:tasks.rados.rados.0.smithi090.stdout:1461: write oid 31 current snap is 158 2023-01-15T14:34:38.698 INFO:tasks.rados.rados.0.smithi090.stdout:1461: seq_num 520 ranges {739640=641484,1829460=794174} 2023-01-15T14:34:38.698 INFO:tasks.rados.rados.0.smithi090.stdout:1461: writing smithi09025264-31 from 739640 to 1381124 tid 1 2023-01-15T14:34:38.698 INFO:tasks.rados.rados.0.smithi090.stdout:1461: writing smithi09025264-31 from 1829460 to 2623634 tid 2 2023-01-15T14:34:38.698 INFO:tasks.rados.rados.0.smithi090.stdout:1462: read oid 38 snap -1 2023-01-15T14:34:38.698 INFO:tasks.rados.rados.0.smithi090.stdout:1462: expect deleted 2023-01-15T14:34:38.698 INFO:tasks.rados.rados.0.smithi090.stdout:1463: write oid 29 current snap is 158 2023-01-15T14:34:38.698 INFO:tasks.rados.rados.0.smithi090.stdout:1463: seq_num 521 ranges {756447=716733,1887942=537832} 2023-01-15T14:34:38.699 INFO:tasks.rados.rados.0.smithi090.stdout:1463: writing smithi09025264-29 from 756447 to 1473180 tid 1 2023-01-15T14:34:38.699 INFO:tasks.rados.rados.0.smithi090.stdout:1463: writing smithi09025264-29 from 1887942 to 2425774 tid 2 2023-01-15T14:34:38.699 INFO:tasks.rados.rados.0.smithi090.stdout:1464: copy_from oid 32 from oid 47 current snap is 158 2023-01-15T14:34:38.699 INFO:tasks.rados.rados.0.smithi090.stdout:1465: write oid 2 current snap is 158 2023-01-15T14:34:38.699 INFO:tasks.rados.rados.0.smithi090.stdout:1465: seq_num 522 ranges {773254=791982,2346424=426830} 2023-01-15T14:34:38.699 INFO:tasks.rados.rados.0.smithi090.stdout:1465: writing smithi09025264-2 from 773254 to 1565236 tid 1 2023-01-15T14:34:38.699 INFO:tasks.rados.rados.0.smithi090.stdout:1465: writing smithi09025264-2 from 2346424 to 2773254 tid 2 2023-01-15T14:34:38.699 INFO:tasks.rados.rados.0.smithi090.stdout:1466: snap_remove snap 128 2023-01-15T14:34:38.795 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:38.791+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:38.854 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:38.854 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:38.854 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:34:38.854 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:38.865 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:34:38.896 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:38.890+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:38.897 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:38.915 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:38.942 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:38.960 INFO:tasks.rados.rados.0.smithi090.stdout:1466: done (7 left) 2023-01-15T14:34:38.961 INFO:tasks.rados.rados.0.smithi090.stdout:1467: write oid 26 current snap is 158 2023-01-15T14:34:38.961 INFO:tasks.rados.rados.0.smithi090.stdout:1467: seq_num 523 ranges {790061=467231,2004906=541501} 2023-01-15T14:34:38.961 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:38.979 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:38.979 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:38.979 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:34:38.979 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:34:38.979 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:38.981 INFO:tasks.rados.rados.0.smithi090.stdout:1467: writing smithi09025264-26 from 790061 to 1257292 tid 1 2023-01-15T14:34:38.989 INFO:tasks.rados.rados.0.smithi090.stdout:1467: writing smithi09025264-26 from 2004906 to 2546407 tid 2 2023-01-15T14:34:38.989 INFO:tasks.rados.rados.0.smithi090.stdout:1468: delete oid 40 current snap is 158 2023-01-15T14:34:38.989 INFO:tasks.rados.rados.0.smithi090.stdout:1459: finishing rollback tid 1 to smithi09025264-39 2023-01-15T14:34:38.989 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 39 v 317 (ObjNum 33 snap 0 seq_num 3196677104) dirty dne 2023-01-15T14:34:38.989 INFO:tasks.rados.rados.0.smithi090.stdout:1461: finishing write tid 1 to smithi09025264-31 2023-01-15T14:34:38.989 INFO:tasks.rados.rados.0.smithi090.stdout:1461: finishing write tid 2 to smithi09025264-31 2023-01-15T14:34:38.990 INFO:tasks.rados.rados.0.smithi090.stdout:1461: finishing write tid 3 to smithi09025264-31 2023-01-15T14:34:38.990 INFO:tasks.rados.rados.0.smithi090.stdout:1461: finishing write tid 4 to smithi09025264-31 2023-01-15T14:34:38.990 INFO:tasks.rados.rados.0.smithi090.stdout:1461: oid 31 updating version 0 to 408 2023-01-15T14:34:38.990 INFO:tasks.rados.rados.0.smithi090.stdout:1461: oid 31 updating version 408 to 409 2023-01-15T14:34:38.990 INFO:tasks.rados.rados.0.smithi090.stdout:1461: oid 31 updating version 409 to 410 2023-01-15T14:34:38.990 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 410 (ObjNum 520 snap 158 seq_num 520) dirty exists 2023-01-15T14:34:38.990 INFO:tasks.rados.rados.0.smithi090.stdout:1461: left oid 31 (ObjNum 520 snap 158 seq_num 520) 2023-01-15T14:34:38.990 INFO:tasks.rados.rados.0.smithi090.stdout:1463: finishing write tid 1 to smithi09025264-29 2023-01-15T14:34:38.990 INFO:tasks.rados.rados.0.smithi090.stdout:1463: finishing write tid 2 to smithi09025264-29 2023-01-15T14:34:38.990 INFO:tasks.rados.rados.0.smithi090.stdout:1463: finishing write tid 3 to smithi09025264-29 2023-01-15T14:34:38.991 INFO:tasks.rados.rados.0.smithi090.stdout:1463: finishing write tid 4 to smithi09025264-29 2023-01-15T14:34:38.991 INFO:tasks.rados.rados.0.smithi090.stdout:1463: oid 29 updating version 0 to 270 2023-01-15T14:34:38.991 INFO:tasks.rados.rados.0.smithi090.stdout:1463: oid 29 version 270 is already newer than 269 2023-01-15T14:34:38.991 INFO:tasks.rados.rados.0.smithi090.stdout:1463: oid 29 updating version 270 to 271 2023-01-15T14:34:38.991 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 271 (ObjNum 521 snap 158 seq_num 521) dirty exists 2023-01-15T14:34:38.991 INFO:tasks.rados.rados.0.smithi090.stdout:1463: left oid 29 (ObjNum 521 snap 158 seq_num 521) 2023-01-15T14:34:38.991 INFO:tasks.rados.rados.0.smithi090.stdout:1465: finishing write tid 1 to smithi09025264-2 2023-01-15T14:34:38.991 INFO:tasks.rados.rados.0.smithi090.stdout:1465: finishing write tid 2 to smithi09025264-2 2023-01-15T14:34:38.991 INFO:tasks.rados.rados.0.smithi090.stdout:1465: finishing write tid 3 to smithi09025264-2 2023-01-15T14:34:38.991 INFO:tasks.rados.rados.0.smithi090.stdout:1465: finishing write tid 4 to smithi09025264-2 2023-01-15T14:34:38.992 INFO:tasks.rados.rados.0.smithi090.stdout:1465: oid 2 updating version 0 to 387 2023-01-15T14:34:38.992 INFO:tasks.rados.rados.0.smithi090.stdout:1465: oid 2 updating version 387 to 389 2023-01-15T14:34:38.992 INFO:tasks.rados.rados.0.smithi090.stdout:1465: oid 2 version 389 is already newer than 388 2023-01-15T14:34:38.992 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 389 (ObjNum 522 snap 158 seq_num 522) dirty exists 2023-01-15T14:34:38.992 INFO:tasks.rados.rados.0.smithi090.stdout:1465: left oid 2 (ObjNum 522 snap 158 seq_num 522) 2023-01-15T14:34:38.992 INFO:tasks.rados.rados.0.smithi090.stdout:1460: finishing copy_from to smithi09025264-28 2023-01-15T14:34:38.992 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 375 (ObjNum 513 snap 152 seq_num 513) dirty exists 2023-01-15T14:34:38.992 INFO:tasks.rados.rados.0.smithi090.stdout:1460: finishing copy_from racing read to smithi09025264-28 2023-01-15T14:34:38.992 INFO:tasks.rados.rados.0.smithi090.stdout:1464: finishing copy_from to smithi09025264-32 2023-01-15T14:34:38.993 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 379 (ObjNum 477 snap 140 seq_num 477) dirty exists 2023-01-15T14:34:38.993 INFO:tasks.rados.rados.0.smithi090.stdout:1464: finishing copy_from racing read to smithi09025264-32 2023-01-15T14:34:38.994 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:34:39.037 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:39.034+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:39.097 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:39.094+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:39.153 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:39.154 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:34:39.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:34:39.154 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:34:39.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.154 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:5 3.5 3:a1c47f99:::smithi09025264-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:39.154 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:39.154 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 373.36619530199999, 2023-01-15T14:34:39.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40548699700000002, 2023-01-15T14:34:39.155 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.155 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.155 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.155 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.155 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.155 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 5 2023-01-15T14:34:39.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.156 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:39.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:39.159 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781744+0000", 2023-01-15T14:34:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782566+0000", 2023-01-15T14:34:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000821578 2023-01-15T14:34:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.160 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782588+0000", 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.226e-05 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782601+0000", 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2998e-05 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882850+0000", 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100248908 2023-01-15T14:34:39.161 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882871+0000", 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0873000000000002e-05 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083679+0000", 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200808083 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.162 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083948+0000", 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000269342 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084161+0000", 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021326900000000001 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084655+0000", 2023-01-15T14:34:39.163 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049371300000000001 2023-01-15T14:34:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084674+0000", 2023-01-15T14:34:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9012999999999999e-05 2023-01-15T14:34:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084924+0000", 2023-01-15T14:34:39.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024989899999999999 2023-01-15T14:34:39.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:39.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085221+0000", 2023-01-15T14:34:39.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029719600000000001 2023-01-15T14:34:39.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186953+0000", 2023-01-15T14:34:39.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10173174 2023-01-15T14:34:39.169 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.169 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.169 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.169 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187026+0000", 2023-01-15T14:34:39.170 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2957e-05 2023-01-15T14:34:39.170 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.170 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.170 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.170 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187124+0000", 2023-01-15T14:34:39.170 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8146000000000005e-05 2023-01-15T14:34:39.171 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.171 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.171 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.171 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187233+0000", 2023-01-15T14:34:39.171 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010931599999999999 2023-01-15T14:34:39.171 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.172 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.172 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.172 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.172 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.172 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:6 3.5 3:a1c47f99:::smithi09025264-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:39.172 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:39.173 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 373.36094243000002, 2023-01-15T14:34:39.173 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50419954199999995, 2023-01-15T14:34:39.173 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.173 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.173 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.173 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.173 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.174 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 6 2023-01-15T14:34:39.174 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.174 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.174 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.174 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.174 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:39.175 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.175 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.175 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.175 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.175 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:39.175 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.176 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.176 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786997+0000", 2023-01-15T14:34:39.176 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:39.176 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.176 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787486+0000", 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048882999999999999 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787514+0000", 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7591999999999999e-05 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787526+0000", 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1894e-05 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.178 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.178 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882966+0000", 2023-01-15T14:34:39.178 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095440450999999996 2023-01-15T14:34:39.178 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.178 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.178 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:39.178 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882973+0000", 2023-01-15T14:34:39.178 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2570000000000001e-06 2023-01-15T14:34:39.178 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.179 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.179 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.179 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084809+0000", 2023-01-15T14:34:39.179 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201835763 2023-01-15T14:34:39.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.179 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.179 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:39.179 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084928+0000", 2023-01-15T14:34:39.179 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118946 2023-01-15T14:34:39.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.179 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.180 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.180 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187347+0000", 2023-01-15T14:34:39.180 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102419285 2023-01-15T14:34:39.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.180 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.180 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.180 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187450+0000", 2023-01-15T14:34:39.180 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010286299999999999 2023-01-15T14:34:39.180 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.180 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.180 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:39.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187735+0000", 2023-01-15T14:34:39.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028429900000000001 2023-01-15T14:34:39.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.181 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188113+0000", 2023-01-15T14:34:39.181 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037883200000000001 2023-01-15T14:34:39.181 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.181 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.181 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.182 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188138+0000", 2023-01-15T14:34:39.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4110000000000001e-05 2023-01-15T14:34:39.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.182 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.182 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188389+0000", 2023-01-15T14:34:39.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025173900000000001 2023-01-15T14:34:39.182 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:39.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189251+0000", 2023-01-15T14:34:39.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086129800000000003 2023-01-15T14:34:39.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290989+0000", 2023-01-15T14:34:39.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101738524 2023-01-15T14:34:39.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.184 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.184 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291075+0000", 2023-01-15T14:34:39.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5469000000000004e-05 2023-01-15T14:34:39.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.184 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.184 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.184 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291144+0000", 2023-01-15T14:34:39.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:34:39.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.184 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.184 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291199+0000", 2023-01-15T14:34:39.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4413000000000002e-05 2023-01-15T14:34:39.185 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.185 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.185 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.185 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:7 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:39.185 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:39.185 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 373.36039597299998, 2023-01-15T14:34:39.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50407251399999997, 2023-01-15T14:34:39.186 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.186 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.186 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.186 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.186 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.186 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 7 2023-01-15T14:34:39.187 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.187 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.187 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.187 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.187 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:39.187 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.187 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.188 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.188 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.188 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:39.188 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.188 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.188 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.188 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.189 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787545+0000", 2023-01-15T14:34:39.189 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:39.189 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.189 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.189 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.189 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787551+0000", 2023-01-15T14:34:39.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8669999999999999e-06 2023-01-15T14:34:39.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787559+0000", 2023-01-15T14:34:39.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9780000000000005e-06 2023-01-15T14:34:39.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787564+0000", 2023-01-15T14:34:39.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5709999999999997e-06 2023-01-15T14:34:39.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883054+0000", 2023-01-15T14:34:39.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095489722999999999 2023-01-15T14:34:39.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:39.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883061+0000", 2023-01-15T14:34:39.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2760000000000003e-06 2023-01-15T14:34:39.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084987+0000", 2023-01-15T14:34:39.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20192545000000001 2023-01-15T14:34:39.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:39.194 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085190+0000", 2023-01-15T14:34:39.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020288899999999999 2023-01-15T14:34:39.194 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188374+0000", 2023-01-15T14:34:39.195 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103184449 2023-01-15T14:34:39.195 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.195 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.195 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188543+0000", 2023-01-15T14:34:39.196 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016925299999999999 2023-01-15T14:34:39.196 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.196 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.196 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:39.196 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188748+0000", 2023-01-15T14:34:39.196 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020473600000000001 2023-01-15T14:34:39.196 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.197 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.197 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.197 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188986+0000", 2023-01-15T14:34:39.197 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023780900000000001 2023-01-15T14:34:39.197 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.197 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.197 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.198 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189006+0000", 2023-01-15T14:34:39.198 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0604000000000001e-05 2023-01-15T14:34:39.198 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.198 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.198 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.198 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189084+0000", 2023-01-15T14:34:39.199 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7526000000000003e-05 2023-01-15T14:34:39.199 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.199 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.199 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:39.199 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189272+0000", 2023-01-15T14:34:39.199 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018762 2023-01-15T14:34:39.200 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.200 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.200 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.200 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291333+0000", 2023-01-15T14:34:39.200 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102061185 2023-01-15T14:34:39.200 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.201 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.201 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291466+0000", 2023-01-15T14:34:39.201 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013325699999999999 2023-01-15T14:34:39.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291535+0000", 2023-01-15T14:34:39.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9309999999999999e-05 2023-01-15T14:34:39.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291618+0000", 2023-01-15T14:34:39.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2773000000000002e-05 2023-01-15T14:34:39.203 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.203 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.203 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.204 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:8 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:39.204 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:39.204 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 373.36036999100003, 2023-01-15T14:34:39.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50442076000000002, 2023-01-15T14:34:39.204 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.204 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:39.205 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.205 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.205 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.205 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 8 2023-01-15T14:34:39.206 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.206 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.206 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.206 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.206 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:39.206 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.207 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.207 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:39.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.208 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.208 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.208 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787571+0000", 2023-01-15T14:34:39.208 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:39.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.209 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787575+0000", 2023-01-15T14:34:39.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7890000000000001e-06 2023-01-15T14:34:39.209 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.210 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.210 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787589+0000", 2023-01-15T14:34:39.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4212999999999999e-05 2023-01-15T14:34:39.210 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.211 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.211 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787594+0000", 2023-01-15T14:34:39.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1429999999999997e-06 2023-01-15T14:34:39.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.211 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.211 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.211 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983143+0000", 2023-01-15T14:34:39.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.195549003 2023-01-15T14:34:39.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.212 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:39.212 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983153+0000", 2023-01-15T14:34:39.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4369999999999996e-06 2023-01-15T14:34:39.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.212 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.212 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085286+0000", 2023-01-15T14:34:39.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102133064 2023-01-15T14:34:39.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085401+0000", 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115176 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189116+0000", 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103715444 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:39.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189221+0000", 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010438 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291724+0000", 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102502998 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291824+0000", 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010010599999999999 2023-01-15T14:34:39.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291992+0000", 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016856900000000001 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:21 3.5 3:a75958b1:::smithi09025264-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:39.215 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 373.316281076, 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.45851924500000002, 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 21 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:39.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:39.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831657+0000", 2023-01-15T14:34:39.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832814+0000", 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011564909999999999 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832864+0000", 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0243e-05 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832903+0000", 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8704000000000001e-05 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983202+0000", 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.150299404 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983208+0000", 2023-01-15T14:34:39.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0039999999999996e-06 2023-01-15T14:34:39.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185539+0000", 2023-01-15T14:34:39.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202330867 2023-01-15T14:34:39.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185803+0000", 2023-01-15T14:34:39.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026394299999999999 2023-01-15T14:34:39.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186037+0000", 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023386300000000001 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186637+0000", 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060001800000000001 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186665+0000", 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8459999999999999e-05 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187045+0000", 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037992899999999999 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188201+0000", 2023-01-15T14:34:39.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001155309 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289903+0000", 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101702601 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289985+0000", 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1617000000000001e-05 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290065+0000", 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0276000000000002e-05 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290180+0000", 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011464300000000001 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.224 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:22 3.5 3:a75958b1:::smithi09025264-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 373.31181338800002, 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65790599599999999, 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 22 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836126+0000", 2023-01-15T14:34:39.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836181+0000", 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4246999999999997e-05 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836212+0000", 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1843e-05 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.227 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836226+0000", 2023-01-15T14:34:39.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3216e-05 2023-01-15T14:34:39.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186757+0000", 2023-01-15T14:34:39.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35053103899999999 2023-01-15T14:34:39.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:39.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186871+0000", 2023-01-15T14:34:39.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114403 2023-01-15T14:34:39.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290263+0000", 2023-01-15T14:34:39.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103391976 2023-01-15T14:34:39.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290413+0000", 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014991200000000001 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290627+0000", 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021379299999999999 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290861+0000", 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023479600000000001 2023-01-15T14:34:39.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290888+0000", 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6570000000000001e-05 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290990+0000", 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101723 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:39.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291245+0000", 2023-01-15T14:34:39.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025543899999999999 2023-01-15T14:34:39.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392822+0000", 2023-01-15T14:34:39.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10157725500000001 2023-01-15T14:34:39.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493881+0000", 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101058262 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493964+0000", 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2973000000000007e-05 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494034+0000", 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0432999999999995e-05 2023-01-15T14:34:39.233 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:23 3.5 3:a75958b1:::smithi09025264-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 373.31169825900002, 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65796116900000001, 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.234 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 23 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.235 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836242+0000", 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836256+0000", 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3677e-05 2023-01-15T14:34:39.236 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836265+0000", 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2769999999999994e-06 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936398+0000", 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100132559 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289431+0000", 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35303322399999998 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289587+0000", 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000155873 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292052+0000", 2023-01-15T14:34:39.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0024647919999999999 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292180+0000", 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012810999999999999 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292373+0000", 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019258599999999999 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292597+0000", 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022465500000000001 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292618+0000", 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0319999999999999e-05 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292692+0000", 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4643999999999999e-05 2023-01-15T14:34:39.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393103+0000", 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100411156 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493827+0000", 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100724143 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.241 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494085+0000", 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000257891 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494141+0000", 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6132999999999999e-05 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494204+0000", 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2966000000000007e-05 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.242 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:24 3.5 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 373.21151683300002, 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.55808677600000001, 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.243 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 24 2023-01-15T14:34:39.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.244 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.244 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:39.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.244 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:39.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936423+0000", 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936433+0000", 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0443e-05 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936451+0000", 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7842999999999998e-05 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936461+0000", 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0057e-05 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289693+0000", 2023-01-15T14:34:39.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.353231617 2023-01-15T14:34:39.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:39.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289814+0000", 2023-01-15T14:34:39.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120763 2023-01-15T14:34:39.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392923+0000", 2023-01-15T14:34:39.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10310955300000001 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393072+0000", 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000148469 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494242+0000", 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117034499999999 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.248 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.249 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.249 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494363+0000", 2023-01-15T14:34:39.249 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120703 2023-01-15T14:34:39.249 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.249 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.249 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.249 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494511+0000", 2023-01-15T14:34:39.249 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014873299999999999 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:78 3.c 3:3137f0cb:::smithi09025264-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 372.82155083700002, 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20331801699999999, 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.250 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 78 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.251 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326388+0000", 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326672+0000", 2023-01-15T14:34:39.252 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028367900000000001 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326713+0000", 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1539999999999999e-05 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326729+0000", 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.519e-05 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.253 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326879+0000", 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015064899999999999 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327096+0000", 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021682299999999999 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:34:39.254 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327308+0000", 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000211366 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327757+0000", 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000449009 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327776+0000", 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9201000000000001e-05 2023-01-15T14:34:39.255 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328012+0000", 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236054 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429534+0000", 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10152201499999999 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.256 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529340+0000", 2023-01-15T14:34:39.257 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099806249 2023-01-15T14:34:39.257 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.257 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.257 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.257 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529437+0000", 2023-01-15T14:34:39.257 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6949999999999998e-05 2023-01-15T14:34:39.257 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.257 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.257 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.257 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529587+0000", 2023-01-15T14:34:39.257 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015008600000000001 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529709+0000", 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121636 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:79 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:39.258 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:39.259 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 372.82118124099998, 2023-01-15T14:34:39.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20348171300000001, 2023-01-15T14:34:39.259 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.259 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.259 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.259 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.259 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.259 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 79 2023-01-15T14:34:39.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.259 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:39.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:39.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326759+0000", 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326770+0000", 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326791+0000", 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1556000000000002e-05 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326818+0000", 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6979999999999999e-05 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.427993+0000", 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117517199999999 2023-01-15T14:34:39.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428266+0000", 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027233299999999998 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428696+0000", 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042996499999999998 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429057+0000", 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000360952 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429076+0000", 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429160+0000", 2023-01-15T14:34:39.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429549+0000", 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038884599999999998 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529886+0000", 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100336261 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529978+0000", 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2775999999999994e-05 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530108+0000", 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012958300000000001 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530242+0000", 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013406 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:80 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 372.82104288199997, 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20370874, 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.267 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:39.268 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.268 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.268 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.268 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 80 2023-01-15T14:34:39.268 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.268 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.294 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.294 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.294 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:39.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:39.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.296 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326897+0000", 2023-01-15T14:34:39.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:39.296 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.296 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326906+0000", 2023-01-15T14:34:39.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6929999999999996e-06 2023-01-15T14:34:39.296 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.296 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326931+0000", 2023-01-15T14:34:39.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4757e-05 2023-01-15T14:34:39.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326945+0000", 2023-01-15T14:34:39.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.402e-05 2023-01-15T14:34:39.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429239+0000", 2023-01-15T14:34:39.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102294418 2023-01-15T14:34:39.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:39.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429487+0000", 2023-01-15T14:34:39.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024789300000000001 2023-01-15T14:34:39.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530309+0000", 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100821459 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530438+0000", 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012928700000000001 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530607+0000", 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169483 2023-01-15T14:34:39.299 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.300 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.300 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.300 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:78 3.c e23/20 ondisk, result = 0)", 2023-01-15T14:34:39.300 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:39.300 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 372.81915116099998, 2023-01-15T14:34:39.300 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20094287799999999, 2023-01-15T14:34:39.300 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.300 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:39.301 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:39.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:39.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328789+0000", 2023-01-15T14:34:39.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:39.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328799+0000", 2023-01-15T14:34:39.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8980000000000008e-06 2023-01-15T14:34:39.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328820+0000", 2023-01-15T14:34:39.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1041999999999999e-05 2023-01-15T14:34:39.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328832+0000", 2023-01-15T14:34:39.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1616e-05 2023-01-15T14:34:39.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.304 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529404+0000", 2023-01-15T14:34:39.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200572631 2023-01-15T14:34:39.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.304 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529419+0000", 2023-01-15T14:34:39.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4250000000000001e-05 2023-01-15T14:34:39.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.305 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.305 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529733+0000", 2023-01-15T14:34:39.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000314693 2023-01-15T14:34:39.305 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.305 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.305 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.305 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.305 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:201 3.5 3:a1c47f99:::smithi09025264-2:head [copy-from ver 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:39.305 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 371.74154476400003, 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30931411399999997, 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 201 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:39.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:39.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406394+0000", 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406408+0000", 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4416999999999999e-05 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406446+0000", 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7339999999999998e-05 2023-01-15T14:34:39.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406467+0000", 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1137000000000001e-05 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406677+0000", 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000210275 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.310 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406902+0000", 2023-01-15T14:34:39.310 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022529000000000001 2023-01-15T14:34:39.310 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.310 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.310 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:39.310 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.513012+0000", 2023-01-15T14:34:39.310 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.106109527 2023-01-15T14:34:39.310 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.310 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.310 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.311 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514754+0000", 2023-01-15T14:34:39.311 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0017419779999999999 2023-01-15T14:34:39.311 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.311 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.311 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.311 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514773+0000", 2023-01-15T14:34:39.311 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8700000000000001e-05 2023-01-15T14:34:39.311 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.311 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.311 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.515531+0000", 2023-01-15T14:34:39.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075844300000000003 2023-01-15T14:34:39.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615334+0000", 2023-01-15T14:34:39.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099803327999999997 2023-01-15T14:34:39.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:39.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615376+0000", 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1675999999999999e-05 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715546+0000", 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016992500000001 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715626+0000", 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0112999999999997e-05 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715711+0000", 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:202 3.5 3:a1c47f99:::smithi09025264-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 371.741432218, 2023-01-15T14:34:39.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31002637300000002, 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 202 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406508+0000", 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406516+0000", 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.413e-06 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406534+0000", 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8936999999999998e-05 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406548+0000", 2023-01-15T14:34:39.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3196e-05 2023-01-15T14:34:39.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407271+0000", 2023-01-15T14:34:39.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00072301500000000003 2023-01-15T14:34:39.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:34:39.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407466+0000", 2023-01-15T14:34:39.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000195009 2023-01-15T14:34:39.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.614996+0000", 2023-01-15T14:34:39.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.207530192 2023-01-15T14:34:39.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:39.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615227+0000", 2023-01-15T14:34:39.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000230979 2023-01-15T14:34:39.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716282+0000", 2023-01-15T14:34:39.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101055465 2023-01-15T14:34:39.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716409+0000", 2023-01-15T14:34:39.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012653999999999999 2023-01-15T14:34:39.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716536+0000", 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012683 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:189 3.d e23/20 3:b3f111b5:::smithi09025264-48:head v 23'10, mlcod=23'6)", 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:39.321 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 371.69622757000002, 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20400475100000001, 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:39.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451712+0000", 2023-01-15T14:34:39.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:39.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452272+0000", 2023-01-15T14:34:39.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00056035899999999997 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452310+0000", 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8145999999999997e-05 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452324+0000", 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4443999999999999e-05 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652703+0000", 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20037818199999999 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652762+0000", 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9302999999999998e-05 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653302+0000", 2023-01-15T14:34:39.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054012199999999998 2023-01-15T14:34:39.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653333+0000", 2023-01-15T14:34:39.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.112e-05 2023-01-15T14:34:39.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653667+0000", 2023-01-15T14:34:39.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033337900000000002 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653709+0000", 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2469999999999998e-05 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655719+0000", 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020096699999999999 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.327 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.328 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.328 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.328 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.328 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:269 3.2 e30/20 3:47dfe935:::smithi09025264-29:head v 30'11, mlcod=23'8)", 2023-01-15T14:34:39.328 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:39.328 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 364.27018212899998, 2023-01-15T14:34:39.328 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23185540399999999, 2023-01-15T14:34:39.328 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.328 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.329 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:39.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.329 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.329 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.329 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:39.329 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.329 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878379+0000", 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061979600000000004 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.330 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878406+0000", 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978515+0000", 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100108461 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078759+0000", 2023-01-15T14:34:39.331 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100244496 2023-01-15T14:34:39.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078796+0000", 2023-01-15T14:34:39.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6758999999999998e-05 2023-01-15T14:34:39.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.332 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.332 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.332 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079567+0000", 2023-01-15T14:34:39.332 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077137199999999999 2023-01-15T14:34:39.332 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079593+0000", 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079844+0000", 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025075899999999998 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.333 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079910+0000", 2023-01-15T14:34:39.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6406000000000001e-05 2023-01-15T14:34:39.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.334 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.334 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.109615+0000", 2023-01-15T14:34:39.334 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.029704323000000001 2023-01-15T14:34:39.334 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.334 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.334 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.334 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.334 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:281 3.5 3:a75958b1:::smithi09025264-6:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 364.16925850000001, 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30574310900000001, 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 281 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.335 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.336 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.336 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.336 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:39.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.336 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.336 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.336 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.336 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:39.336 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.336 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.336 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.337 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.337 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:39.337 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:39.337 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.337 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.337 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.337 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978686+0000", 2023-01-15T14:34:39.337 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4429999999999999e-06 2023-01-15T14:34:39.337 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.337 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978690+0000", 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0300000000000004e-06 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978693+0000", 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.179e-06 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981592+0000", 2023-01-15T14:34:39.338 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0028984140000000002 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981718+0000", 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012616600000000001 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981982+0000", 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026445199999999998 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.339 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982278+0000", 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029514100000000001 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982290+0000", 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.271e-05 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982343+0000", 2023-01-15T14:34:39.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2914000000000002e-05 2023-01-15T14:34:39.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.341 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:39.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982385+0000", 2023-01-15T14:34:39.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2114999999999999e-05 2023-01-15T14:34:39.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.341 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.181937+0000", 2023-01-15T14:34:39.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19955213099999999 2023-01-15T14:34:39.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.342 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:39.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284183+0000", 2023-01-15T14:34:39.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10224538 2023-01-15T14:34:39.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.342 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284318+0000", 2023-01-15T14:34:39.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000134843 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284426+0000", 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108513 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:373 3.4 e34/20 3:2029fa71:::smithi09025264-39:head v 34'19, mlcod=26'16)", 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:39.343 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 358.68827604099999, 2023-01-15T14:34:39.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20212622799999999, 2023-01-15T14:34:39.344 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.344 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.344 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:39.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:39.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459664+0000", 2023-01-15T14:34:39.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:39.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460099+0000", 2023-01-15T14:34:39.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043515900000000001 2023-01-15T14:34:39.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460130+0000", 2023-01-15T14:34:39.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1590000000000001e-05 2023-01-15T14:34:39.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560226+0000", 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100095248 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560369+0000", 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014381499999999999 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560404+0000", 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4650000000000002e-05 2023-01-15T14:34:39.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560804+0000", 2023-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000399823 2023-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560827+0000", 2023-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3612999999999999e-05 2023-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.561101+0000", 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027355499999999998 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661710+0000", 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100608716 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.349 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661792+0000", 2023-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2006000000000001e-05 2023-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:455 3.d e41/20 3:bd982ca3:::smithi09025264-38:head v 41'24, mlcod=38'18)", 2023-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 351.63851446799998, 2023-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20118033699999999, 2023-01-15T14:34:39.350 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:39.351 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509425+0000", 2023-01-15T14:34:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509441+0000", 2023-01-15T14:34:39.352 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5248e-05 2023-01-15T14:34:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509474+0000", 2023-01-15T14:34:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3531000000000001e-05 2023-01-15T14:34:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.609644+0000", 2023-01-15T14:34:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016994999999999 2023-01-15T14:34:39.353 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709872+0000", 2023-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10022840299999999 2023-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709906+0000", 2023-01-15T14:34:39.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3160000000000001e-05 2023-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710153+0000", 2023-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024735399999999998 2023-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710204+0000", 2023-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1075999999999997e-05 2023-01-15T14:34:39.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710280+0000", 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6202e-05 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710317+0000", 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6387000000000002e-05 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.356 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710607+0000", 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029079699999999997 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:497 3.d e44/20 3:bd982ca3:::smithi09025264-38:head v 44'31, mlcod=42'26)", 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 349.37371960600001, 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20082067100000001, 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:39.357 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774220+0000", 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774238+0000", 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8107000000000001e-05 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:39.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774263+0000", 2023-01-15T14:34:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4122999999999999e-05 2023-01-15T14:34:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774280+0000", 2023-01-15T14:34:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7039999999999999e-05 2023-01-15T14:34:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874060+0000", 2023-01-15T14:34:39.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099780698000000001 2023-01-15T14:34:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874130+0000", 2023-01-15T14:34:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9852999999999997e-05 2023-01-15T14:34:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874351+0000", 2023-01-15T14:34:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022091400000000001 2023-01-15T14:34:39.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.362 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.362 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:39.362 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874369+0000", 2023-01-15T14:34:39.362 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7876e-05 2023-01-15T14:34:39.362 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.362 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.362 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:39.362 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874433+0000", 2023-01-15T14:34:39.362 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3793000000000004e-05 2023-01-15T14:34:39.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:39.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.974927+0000", 2023-01-15T14:34:39.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10049393500000001 2023-01-15T14:34:39.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:39.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:39.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:39.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.975043+0000", 2023-01-15T14:34:39.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115949 2023-01-15T14:34:39.364 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.364 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.364 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.364 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:39.364 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:39.364 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:39.366 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:34:39.367 INFO:tasks.rados.rados.0.smithi090.stdout:1459: done (8 left) 2023-01-15T14:34:39.367 INFO:tasks.rados.rados.0.smithi090.stdout:1460: done (7 left) 2023-01-15T14:34:39.367 INFO:tasks.rados.rados.0.smithi090.stdout:1461: done (6 left) 2023-01-15T14:34:39.367 INFO:tasks.rados.rados.0.smithi090.stdout:1462: done (5 left) 2023-01-15T14:34:39.367 INFO:tasks.rados.rados.0.smithi090.stdout:1463: done (4 left) 2023-01-15T14:34:39.367 INFO:tasks.rados.rados.0.smithi090.stdout:1464: done (3 left) 2023-01-15T14:34:39.367 INFO:tasks.rados.rados.0.smithi090.stdout:1465: done (2 left) 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1468: done (1 left) 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1469: write oid 1 current snap is 158 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1469: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1469: writing smithi09025264-1 from 423675 to 1157757 tid 1 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1469: writing smithi09025264-1 from 1721870 to 2267040 tid 2 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1469: writing smithi09025264-1 from 2717777 to 2823675 tid 3 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1470: read oid 30 snap 157 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1470: expect (ObjNum 449 snap 125 seq_num 449) 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1471: delete oid 40 current snap is 158 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1467: finishing write tid 1 to smithi09025264-26 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1467: finishing write tid 2 to smithi09025264-26 2023-01-15T14:34:39.368 INFO:tasks.rados.rados.0.smithi090.stdout:1467: finishing write tid 3 to smithi09025264-26 2023-01-15T14:34:39.369 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:39.369 INFO:tasks.rados.rados.0.smithi090.stdout:1471: done (3 left) 2023-01-15T14:34:39.369 INFO:tasks.rados.rados.0.smithi090.stdout:1472: write oid 43 current snap is 158 2023-01-15T14:34:39.369 INFO:tasks.rados.rados.0.smithi090.stdout:1472: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2023-01-15T14:34:39.370 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:39.371 INFO:tasks.rados.rados.0.smithi090.stdout:1472: writing smithi09025264-43 from 457289 to 941869 tid 1 2023-01-15T14:34:39.371 INFO:tasks.rados.rados.0.smithi090.stdout:1472: writing smithi09025264-43 from 1438834 to 1987673 tid 2 2023-01-15T14:34:39.371 INFO:tasks.rados.rados.0.smithi090.stdout:1472: writing smithi09025264-43 from 2772623 to 2857289 tid 3 2023-01-15T14:34:39.371 INFO:tasks.rados.rados.0.smithi090.stdout:1473: read oid 37 snap -1 2023-01-15T14:34:39.371 INFO:tasks.rados.rados.0.smithi090.stdout:1473: expect deleted 2023-01-15T14:34:39.371 INFO:tasks.rados.rados.0.smithi090.stdout:1474: write oid 20 current snap is 158 2023-01-15T14:34:39.372 INFO:tasks.rados.rados.0.smithi090.stdout:1470: expect (ObjNum 449 snap 125 seq_num 449) 2023-01-15T14:34:39.372 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:39.372 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:39.359+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:39.372 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:39.374 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:34:39.374 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:34:39.374 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:34:39.374 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:34:39.375 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:34:39.375 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:34:39.375 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:34:39.375 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:34:39.467 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:39.462+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:39.503 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:39.503 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:39.504 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:34:39.564 INFO:tasks.rados.rados.0.smithi090.stdout:1474: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2023-01-15T14:34:39.567 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:39.562+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:39.570 INFO:tasks.rados.rados.0.smithi090.stdout:1474: writing smithi09025264-20 from 474096 to 1033925 tid 1 2023-01-15T14:34:39.578 INFO:tasks.rados.rados.0.smithi090.stdout:1474: writing smithi09025264-20 from 1780963 to 2189813 tid 2 2023-01-15T14:34:39.583 INFO:tasks.rados.rados.0.smithi090.stdout:1474: writing smithi09025264-20 from 2800046 to 2874096 tid 3 2023-01-15T14:34:39.583 INFO:tasks.rados.rados.0.smithi090.stdout:1470: done (5 left) 2023-01-15T14:34:39.583 INFO:tasks.rados.rados.0.smithi090.stdout:1475: delete oid 42 current snap is 158 2023-01-15T14:34:39.583 INFO:tasks.rados.rados.0.smithi090.stdout:1467: finishing write tid 4 to smithi09025264-26 2023-01-15T14:34:39.583 INFO:tasks.rados.rados.0.smithi090.stdout:1467: oid 26 updating version 0 to 337 2023-01-15T14:34:39.583 INFO:tasks.rados.rados.0.smithi090.stdout:1467: oid 26 updating version 337 to 339 2023-01-15T14:34:39.583 INFO:tasks.rados.rados.0.smithi090.stdout:1467: oid 26 version 339 is already newer than 338 2023-01-15T14:34:39.583 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 339 (ObjNum 523 snap 158 seq_num 523) dirty exists 2023-01-15T14:34:39.584 INFO:tasks.rados.rados.0.smithi090.stdout:1467: left oid 26 (ObjNum 523 snap 158 seq_num 523) 2023-01-15T14:34:39.584 INFO:tasks.rados.rados.0.smithi090.stdout:1469: finishing write tid 1 to smithi09025264-1 2023-01-15T14:34:39.584 INFO:tasks.rados.rados.0.smithi090.stdout:1469: finishing write tid 2 to smithi09025264-1 2023-01-15T14:34:39.584 INFO:tasks.rados.rados.0.smithi090.stdout:1472: finishing write tid 1 to smithi09025264-43 2023-01-15T14:34:39.584 INFO:tasks.rados.rados.0.smithi090.stdout:1472: finishing write tid 2 to smithi09025264-43 2023-01-15T14:34:39.584 INFO:tasks.rados.rados.0.smithi090.stdout:1474: finishing write tid 1 to smithi09025264-20 2023-01-15T14:34:39.584 INFO:tasks.rados.rados.0.smithi090.stdout:1474: finishing write tid 2 to smithi09025264-20 2023-01-15T14:34:39.608 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:39.608 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:39.617 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:34:39.639 INFO:tasks.rados.rados.0.smithi090.stdout:1472: finishing write tid 3 to smithi09025264-43 2023-01-15T14:34:39.916 INFO:tasks.rados.rados.0.smithi090.stdout:1472: finishing write tid 4 to smithi09025264-43 2023-01-15T14:34:39.923 INFO:tasks.rados.rados.0.smithi090.stdout:1472: finishing write tid 5 to smithi09025264-43 2023-01-15T14:34:39.923 INFO:tasks.rados.rados.0.smithi090.stdout:1472: oid 43 updating version 0 to 108 2023-01-15T14:34:39.923 INFO:tasks.rados.rados.0.smithi090.stdout:1472: oid 43 version 108 is already newer than 107 2023-01-15T14:34:39.924 INFO:tasks.rados.rados.0.smithi090.stdout:1472: oid 43 version 108 is already newer than 105 2023-01-15T14:34:39.924 INFO:tasks.rados.rados.0.smithi090.stdout:1472: oid 43 version 108 is already newer than 106 2023-01-15T14:34:39.924 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 108 (ObjNum 527 snap 158 seq_num 527) dirty exists 2023-01-15T14:34:39.924 INFO:tasks.rados.rados.0.smithi090.stdout:1472: left oid 43 (ObjNum 527 snap 158 seq_num 527) 2023-01-15T14:34:39.924 INFO:tasks.rados.rados.0.smithi090.stdout:1467: done (5 left) 2023-01-15T14:34:39.925 INFO:tasks.rados.rados.0.smithi090.stdout:1472: done (4 left) 2023-01-15T14:34:39.925 INFO:tasks.rados.rados.0.smithi090.stdout:1473: done (3 left) 2023-01-15T14:34:39.925 INFO:tasks.rados.rados.0.smithi090.stdout:1475: done (2 left) 2023-01-15T14:34:39.925 INFO:tasks.rados.rados.0.smithi090.stdout:1476: snap_create 2023-01-15T14:34:39.925 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:34:39.926 INFO:tasks.rados.rados.0.smithi090.stdout:1469: finishing write tid 3 to smithi09025264-1 2023-01-15T14:34:39.926 INFO:tasks.rados.rados.0.smithi090.stdout:1469: finishing write tid 4 to smithi09025264-1 2023-01-15T14:34:39.926 INFO:tasks.rados.rados.0.smithi090.stdout:1469: finishing write tid 5 to smithi09025264-1 2023-01-15T14:34:39.926 INFO:tasks.rados.rados.0.smithi090.stdout:1469: oid 1 updating version 0 to 423 2023-01-15T14:34:39.926 INFO:tasks.rados.rados.0.smithi090.stdout:1469: oid 1 updating version 423 to 424 2023-01-15T14:34:39.926 INFO:tasks.rados.rados.0.smithi090.stdout:1469: oid 1 updating version 424 to 425 2023-01-15T14:34:39.926 INFO:tasks.rados.rados.0.smithi090.stdout:1469: oid 1 version 425 is already newer than 422 2023-01-15T14:34:39.927 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 425 (ObjNum 525 snap 158 seq_num 525) dirty exists 2023-01-15T14:34:39.927 INFO:tasks.rados.rados.0.smithi090.stdout:1469: left oid 1 (ObjNum 525 snap 158 seq_num 525) 2023-01-15T14:34:39.927 INFO:tasks.rados.rados.0.smithi090.stdout:1469: done (1 left) 2023-01-15T14:34:39.927 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:39.927 INFO:tasks.rados.rados.0.smithi090.stdout:1474: finishing write tid 3 to smithi09025264-20 2023-01-15T14:34:39.927 INFO:tasks.rados.rados.0.smithi090.stdout:1474: finishing write tid 4 to smithi09025264-20 2023-01-15T14:34:39.928 INFO:tasks.rados.rados.0.smithi090.stdout:1474: finishing write tid 5 to smithi09025264-20 2023-01-15T14:34:39.928 INFO:tasks.rados.rados.0.smithi090.stdout:1474: oid 20 updating version 0 to 262 2023-01-15T14:34:39.928 INFO:tasks.rados.rados.0.smithi090.stdout:1474: oid 20 version 262 is already newer than 259 2023-01-15T14:34:39.931 INFO:tasks.rados.rados.0.smithi090.stdout:1474: oid 20 version 262 is already newer than 261 2023-01-15T14:34:39.931 INFO:tasks.rados.rados.0.smithi090.stdout:1474: oid 20 version 262 is already newer than 260 2023-01-15T14:34:39.931 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 262 (ObjNum 528 snap 158 seq_num 528) dirty exists 2023-01-15T14:34:39.931 INFO:tasks.rados.rados.0.smithi090.stdout:1474: left oid 20 (ObjNum 528 snap 158 seq_num 528) 2023-01-15T14:34:39.931 INFO:tasks.rados.rados.0.smithi090.stdout:1474: done (0 left) 2023-01-15T14:34:39.932 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:39.911+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:39.970 INFO:tasks.rados.rados.0.smithi090.stdout:1476: done (0 left) 2023-01-15T14:34:39.970 INFO:tasks.rados.rados.0.smithi090.stdout:1477: read oid 18 snap 127 2023-01-15T14:34:39.971 INFO:tasks.rados.rados.0.smithi090.stdout:1477: expect deleted 2023-01-15T14:34:39.971 INFO:tasks.rados.rados.0.smithi090.stdout:1478: write oid 7 current snap is 159 2023-01-15T14:34:39.971 INFO:tasks.rados.rados.0.smithi090.stdout:1478: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2023-01-15T14:34:39.975 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:34:39.979 INFO:tasks.rados.rados.0.smithi090.stdout:1478: writing smithi09025264-7 from 507710 to 1218037 tid 1 2023-01-15T14:34:39.987 INFO:tasks.rados.rados.0.smithi090.stdout:1478: writing smithi09025264-7 from 1897927 to 2310446 tid 2 2023-01-15T14:34:39.990 INFO:tasks.rados.rados.0.smithi090.stdout:1478: writing smithi09025264-7 from 2854892 to 2907710 tid 3 2023-01-15T14:34:39.990 INFO:tasks.rados.rados.0.smithi090.stdout:1479: copy_from oid 6 from oid 26 current snap is 159 2023-01-15T14:34:39.990 INFO:tasks.rados.rados.0.smithi090.stdout:1480: read oid 21 snap -1 2023-01-15T14:34:39.990 INFO:tasks.rados.rados.0.smithi090.stdout:1480: expect (ObjNum 503 snap 150 seq_num 503) 2023-01-15T14:34:39.990 INFO:tasks.rados.rados.0.smithi090.stdout:1481: read oid 34 snap -1 2023-01-15T14:34:39.991 INFO:tasks.rados.rados.0.smithi090.stdout:1481: expect deleted 2023-01-15T14:34:39.991 INFO:tasks.rados.rados.0.smithi090.stdout:1482: rollback oid 14 current snap is 159 2023-01-15T14:34:39.991 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 14 to 156 2023-01-15T14:34:39.991 INFO:tasks.rados.rados.0.smithi090.stdout:1483: write oid 17 current snap is 159 2023-01-15T14:34:39.991 INFO:tasks.rados.rados.0.smithi090.stdout:1483: seq_num 531 ranges {524517=785576,1956409=556177} 2023-01-15T14:34:39.997 INFO:tasks.rados.rados.0.smithi090.stdout:1483: writing smithi09025264-17 from 524517 to 1310093 tid 1 2023-01-15T14:34:40.005 INFO:tasks.rados.rados.0.smithi090.stdout:1483: writing smithi09025264-17 from 1956409 to 2512586 tid 2 2023-01-15T14:34:40.005 INFO:tasks.rados.rados.0.smithi090.stdout:1484: copy_from oid 32 from oid 23 current snap is 159 2023-01-15T14:34:40.006 INFO:tasks.rados.rados.0.smithi090.stdout:1485: delete oid 35 current snap is 159 2023-01-15T14:34:40.019 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:40.015+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:40.038 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:40.039 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:40.049 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:34:40.120 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:40.115+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:40.155 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:40.155 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:40.165 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:34:40.221 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:40.218+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:40.265 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:40.266 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:40.276 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:34:40.322 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:40.318+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:40.389 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:40.389 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:40.400 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:34:40.401 INFO:tasks.rados.rados.0.smithi090.stdout:1480: expect (ObjNum 503 snap 150 seq_num 503) 2023-01-15T14:34:40.441 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:40.438+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:40.519 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:40.519 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:40.524 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:40.518+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:40.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:34:40.554 INFO:tasks.rados.rados.0.smithi090.stdout:1478: finishing write tid 1 to smithi09025264-7 2023-01-15T14:34:40.554 INFO:tasks.rados.rados.0.smithi090.stdout:1483: finishing write tid 1 to smithi09025264-17 2023-01-15T14:34:40.554 INFO:tasks.rados.rados.0.smithi090.stdout:1478: finishing write tid 2 to smithi09025264-7 2023-01-15T14:34:40.554 INFO:tasks.rados.rados.0.smithi090.stdout:1483: finishing write tid 2 to smithi09025264-17 2023-01-15T14:34:40.554 INFO:tasks.rados.rados.0.smithi090.stdout:1483: finishing write tid 3 to smithi09025264-17 2023-01-15T14:34:40.555 INFO:tasks.rados.rados.0.smithi090.stdout:1478: finishing write tid 3 to smithi09025264-7 2023-01-15T14:34:40.555 INFO:tasks.rados.rados.0.smithi090.stdout:1478: finishing write tid 4 to smithi09025264-7 2023-01-15T14:34:40.555 INFO:tasks.rados.rados.0.smithi090.stdout:1483: finishing write tid 4 to smithi09025264-17 2023-01-15T14:34:40.555 INFO:tasks.rados.rados.0.smithi090.stdout:1483: oid 17 updating version 0 to 440 2023-01-15T14:34:40.555 INFO:tasks.rados.rados.0.smithi090.stdout:1483: oid 17 version 440 is already newer than 438 2023-01-15T14:34:40.555 INFO:tasks.rados.rados.0.smithi090.stdout:1483: oid 17 version 440 is already newer than 439 2023-01-15T14:34:40.555 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 440 (ObjNum 531 snap 159 seq_num 531) dirty exists 2023-01-15T14:34:40.555 INFO:tasks.rados.rados.0.smithi090.stdout:1483: left oid 17 (ObjNum 531 snap 159 seq_num 531) 2023-01-15T14:34:40.555 INFO:tasks.rados.rados.0.smithi090.stdout:1478: finishing write tid 5 to smithi09025264-7 2023-01-15T14:34:40.555 INFO:tasks.rados.rados.0.smithi090.stdout:1478: oid 7 updating version 0 to 241 2023-01-15T14:34:40.555 INFO:tasks.rados.rados.0.smithi090.stdout:1478: oid 7 version 241 is already newer than 240 2023-01-15T14:34:40.556 INFO:tasks.rados.rados.0.smithi090.stdout:1478: oid 7 version 241 is already newer than 239 2023-01-15T14:34:40.556 INFO:tasks.rados.rados.0.smithi090.stdout:1478: oid 7 version 241 is already newer than 238 2023-01-15T14:34:40.556 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 241 (ObjNum 530 snap 159 seq_num 530) dirty exists 2023-01-15T14:34:40.556 INFO:tasks.rados.rados.0.smithi090.stdout:1478: left oid 7 (ObjNum 530 snap 159 seq_num 530) 2023-01-15T14:34:40.556 INFO:tasks.rados.rados.0.smithi090.stdout:1479: finishing copy_from to smithi09025264-6 2023-01-15T14:34:40.556 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 339 (ObjNum 523 snap 158 seq_num 523) dirty exists 2023-01-15T14:34:40.556 INFO:tasks.rados.rados.0.smithi090.stdout:1479: finishing copy_from racing read to smithi09025264-6 2023-01-15T14:34:40.577 INFO:tasks.rados.rados.0.smithi090.stdout:1477: done (8 left) 2023-01-15T14:34:40.578 INFO:tasks.rados.rados.0.smithi090.stdout:1478: done (7 left) 2023-01-15T14:34:40.578 INFO:tasks.rados.rados.0.smithi090.stdout:1479: done (6 left) 2023-01-15T14:34:40.578 INFO:tasks.rados.rados.0.smithi090.stdout:1480: done (5 left) 2023-01-15T14:34:40.578 INFO:tasks.rados.rados.0.smithi090.stdout:1481: done (4 left) 2023-01-15T14:34:40.578 INFO:tasks.rados.rados.0.smithi090.stdout:1483: done (3 left) 2023-01-15T14:34:40.578 INFO:tasks.rados.rados.0.smithi090.stdout:1485: done (2 left) 2023-01-15T14:34:40.578 INFO:tasks.rados.rados.0.smithi090.stdout:1486: snap_create 2023-01-15T14:34:40.578 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:34:40.625 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:40.619+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:40.630 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:40.630 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:40.640 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:34:40.675 INFO:tasks.rados.rados.0.smithi090.stdout:1482: finishing rollback tid 1 to smithi09025264-14 2023-01-15T14:34:40.675 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 312 (ObjNum 33 snap 0 seq_num 3182657928) dirty dne 2023-01-15T14:34:40.675 INFO:tasks.rados.rados.0.smithi090.stdout:1482: done (1 left) 2023-01-15T14:34:40.676 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:40.676 INFO:tasks.rados.rados.0.smithi090.stdout:1484: finishing copy_from to smithi09025264-32 2023-01-15T14:34:40.676 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 244 (ObjNum 406 snap 108 seq_num 406) dirty exists 2023-01-15T14:34:40.676 INFO:tasks.rados.rados.0.smithi090.stdout:1484: finishing copy_from racing read to smithi09025264-32 2023-01-15T14:34:40.677 INFO:tasks.rados.rados.0.smithi090.stdout:1484: done (0 left) 2023-01-15T14:34:40.726 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:40.722+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:40.745 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:40.755 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:34:40.827 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:40.823+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:40.908 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:40.908 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:40.919 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:34:40.968 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:40.963+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:40.979 INFO:tasks.rados.rados.0.smithi090.stdout:1486: done (0 left) 2023-01-15T14:34:40.979 INFO:tasks.rados.rados.0.smithi090.stdout:1487: rollback oid 30 current snap is 160 2023-01-15T14:34:40.979 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 30 to 154 2023-01-15T14:34:40.979 INFO:tasks.rados.rados.0.smithi090.stdout:1488: snap_create 2023-01-15T14:34:40.980 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:41.029 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:41.023+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:41.121 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:41.122 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:41.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:34:41.130 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:41.126+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:41.224 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7] out_osds: [5, 6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:34:41.527 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:34:41.528 INFO:tasks.thrashosds.thrasher:Adding osd 1 2023-01-15T14:34:41.528 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2023-01-15T14:34:41.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:34:41.530 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:41.530 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:41.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:34:41.531 INFO:tasks.rados.rados.0.smithi090.stdout:1487: finishing rollback tid 0 to smithi09025264-30 2023-01-15T14:34:41.532 INFO:tasks.rados.rados.0.smithi090.stdout:1487: finishing rollback tid 1 to smithi09025264-30 2023-01-15T14:34:41.532 INFO:tasks.rados.rados.0.smithi090.stdout:1487: finishing rollback tid 2 to smithi09025264-30 2023-01-15T14:34:41.532 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 241 (ObjNum 492 snap 144 seq_num 492) dirty exists 2023-01-15T14:34:41.532 INFO:tasks.rados.rados.0.smithi090.stdout:1487: done (0 left) 2023-01-15T14:34:41.533 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:41.523+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:41.631 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:41.626+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:41.732 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:41.732 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:41.732 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:34:41.733 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:41.733 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:41.733 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:41.733 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:41.733 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:41.727+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:41.742 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:41.768 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:41.895 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:41.896 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-01-15T14:34:41.831+0000 7fb90e2b4700 -1 mon.a@0(leader).osd e391 definitely_dead 0 2023-01-15T14:34:41.896 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:41.896 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:34:41.897 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:34:41.897 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:34:41.897 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:41.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:34:41.898 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:41.898 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:34:41.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:34:41.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:34:41.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:34:41.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:34:41.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:34:41.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:34:41.899 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:34:41.900 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:41.890+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:41.937 INFO:tasks.rados.rados.0.smithi090.stdout:1488: done (0 left) 2023-01-15T14:34:42.057 INFO:tasks.rados.rados.0.smithi090.stdout:1489: read oid 34 snap -1 2023-01-15T14:34:42.057 INFO:tasks.rados.rados.0.smithi090.stdout:1489: expect deleted 2023-01-15T14:34:42.057 INFO:tasks.rados.rados.0.smithi090.stdout:1490: write (excl) oid 12 current snap is 161 2023-01-15T14:34:42.057 INFO:tasks.rados.rados.0.smithi090.stdout:1490: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2023-01-15T14:34:42.058 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:34:42.059 INFO:tasks.rados.rados.0.smithi090.stdout:1490: writing smithi09025264-12 from 558131 to 1210558 tid 1 2023-01-15T14:34:42.059 INFO:tasks.rados.rados.0.smithi090.stdout:1490: writing smithi09025264-12 from 1673373 to 2233219 tid 2 2023-01-15T14:34:42.059 INFO:teuthology.orchestra.run.smithi090.stderr:marked in osd.1. 2023-01-15T14:34:42.059 INFO:tasks.rados.rados.0.smithi090.stdout:1490: writing smithi09025264-12 from 2820808 to 2958131 tid 3 2023-01-15T14:34:42.060 INFO:tasks.rados.rados.0.smithi090.stdout:1491: delete oid 17 current snap is 161 2023-01-15T14:34:42.061 INFO:tasks.thrashosds.thrasher:Added osd 1 2023-01-15T14:34:42.061 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:42.051+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:42.159 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:42.154+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:42.230 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:42.230 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:34:42.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:34:42.230 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:34:42.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.230 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:34:42.230 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:34:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 247.27251769099999, 2023-01-15T14:34:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:34:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:34:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.231 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:34:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:34:42.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:34:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:34:42.234 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:34:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:34:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:34:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:34:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:34:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:42.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:34:42.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:42.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:34:42.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:34:42.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:34:42.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:34:42.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:34:42.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:34:42.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:42.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:34:42.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:34:42.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:42.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:34:42.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:34:42.318 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.318 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:42.318 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.318 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:34:42.318 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:34:42.318 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 247.272466748, 2023-01-15T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:42.319 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:42.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:34:42.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:42.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:34:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:34:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:34:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:34:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:34:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:34:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:34:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:34:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:34:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:34:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:34:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:34:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:34:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:34:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 244.33156750699999, 2023-01-15T14:34:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:34:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:34:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:34:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:34:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:34:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:34:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:34:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:34:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:34:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:34:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:34:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:34:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:34:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:34:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:34:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:34:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:34:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:34:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:34:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:34:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:34:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:34:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:34:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:34:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:34:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:34:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:34:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:34:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:34:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:34:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:34:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:34:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:34:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:34:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:34:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:34:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:34:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 242.25779333400001, 2023-01-15T14:34:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:34:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:34:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:34:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:34:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:34:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:34:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:34:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:34:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:34:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:34:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:34:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:34:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:34:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:34:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:34:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:34:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:34:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:34:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:34:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:34:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:34:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:34:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:34:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:34:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 218.36383791399999, 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:34:42.410 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:34:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:42.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:34:42.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:34:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:42.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:34:42.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:34:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:34:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:42.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:34:42.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:34:42.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:42.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:34:42.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:34:42.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:34:42.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:34:42.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:42.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:34:42.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:34:42.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:34:42.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:34:42.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:42.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:34:42.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:34:42.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:42.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:34:42.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:34:42.558 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.558 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:42.558 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.558 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:34:42.559 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:34:42.559 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.48043123299999, 2023-01-15T14:34:42.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:34:42.559 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:42.559 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:42.559 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:42.559 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:42.559 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:42.559 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:34:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:34:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:34:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:34:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:34:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:34:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:34:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:34:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:34:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:34:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:34:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:34:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.47702411, 2023-01-15T14:34:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:34:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:34:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:34:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:34:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:34:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:34:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:34:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:34:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:34:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:34:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:34:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:34:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:34:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:34:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:34:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:34:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:34:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:34:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:34:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:34:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.47680850500001, 2023-01-15T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:34:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:34:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:34:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:34:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:34:42.711 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.711 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:42.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:34:42.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:34:42.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:34:42.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:42.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:34:42.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:34:42.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:42.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:34:42.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:34:42.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:42.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:42.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:42.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:34:42.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:34:42.717 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:42.717 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.034 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.035 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:34:43.039 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:34:43.047 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 196.47670143100001, 2023-01-15T14:34:43.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:34:43.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.047 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:43.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:43.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:43.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:43.047 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:34:43.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.048 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:34:43.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:34:43.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:34:43.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:43.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:34:43.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:34:43.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:34:43.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:34:43.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:43.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:34:43.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:34:43.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:34:43.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:34:43.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:34:43.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:34:43.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:34:43.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:34:43.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.054 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.054 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:34:43.055 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:34:43.055 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 188.34646238100001, 2023-01-15T14:34:43.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:34:43.055 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.055 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:43.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:43.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:43.055 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:34:43.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.056 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:34:43.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:34:43.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:34:43.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:34:43.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:34:43.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:34:43.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:34:43.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:34:43.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:34:43.064 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 188.34635784400001, 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:34:43.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:34:43.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:34:43.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:43.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:34:43.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:34:43.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:34:43.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:34:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:34:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:34:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:34:43.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:34:43.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:34:43.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:34:43.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:43.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:34:43.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:34:43.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:34:43.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:34:43.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:34:43.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:34:43.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:34:43.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:34:43.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:34:43.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:34:43.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:34:43.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:34:43.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:34:43.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:34:43.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.075 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.075 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.075 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:34:43.075 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:34:43.075 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 188.346287913, 2023-01-15T14:34:43.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:34:43.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:43.076 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:43.076 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:43.076 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:34:43.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.076 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:34:43.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:34:43.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:34:43.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:43.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:34:43.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:34:43.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:34:43.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:34:43.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:34:43.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:34:43.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:34:43.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:34:43.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:34:43.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:34:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:34:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:34:43.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2780 3.38 e265/232 3:1eaad226:::smithi09025264-49:head v 265'279, mlcod=261'274)", 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 130.17511678700001, 2023-01-15T14:34:43.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32005591799999999, 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:34:43.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991860+0000", 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993927+0000", 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020673229999999998 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993946+0000", 2023-01-15T14:34:43.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9429999999999999e-05 2023-01-15T14:34:43.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.094039+0000", 2023-01-15T14:34:43.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092898 2023-01-15T14:34:43.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285071+0000", 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.191031904 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285106+0000", 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5227e-05 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286323+0000", 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012163829999999999 2023-01-15T14:34:43.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286335+0000", 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2224e-05 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287218+0000", 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088265000000000001 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287250+0000", 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2209000000000003e-05 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.311917+0000", 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024667062999999999 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2868 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'278, mlcod=264'275)", 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 121.17322584599999, 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31268252400000002, 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:34:43.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993751+0000", 2023-01-15T14:34:43.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:43.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993762+0000", 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0822000000000001e-05 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993777+0000", 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5540999999999999e-05 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993787+0000", 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0173999999999999e-05 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290198+0000", 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29641022099999997 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290212+0000", 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4214e-05 2023-01-15T14:34:43.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290300+0000", 2023-01-15T14:34:43.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8300000000000005e-05 2023-01-15T14:34:43.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290312+0000", 2023-01-15T14:34:43.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2344000000000001e-05 2023-01-15T14:34:43.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290357+0000", 2023-01-15T14:34:43.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4367999999999999e-05 2023-01-15T14:34:43.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290389+0000", 2023-01-15T14:34:43.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1983000000000002e-05 2023-01-15T14:34:43.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.306434+0000", 2023-01-15T14:34:43.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016045667 2023-01-15T14:34:43.210 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.210 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.210 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.210 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:34:43.210 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:34:43.210 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 121.17317642099999, 2023-01-15T14:34:43.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:34:43.211 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.211 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.211 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:34:43.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:34:43.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:34:43.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:43.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:34:43.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:34:43.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:34:43.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:34:43.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:34:43.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:34:43.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:34:43.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:34:43.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:34:43.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:34:43.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:34:43.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.217 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.217 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:34:43.217 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:34:43.218 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 121.169364734, 2023-01-15T14:34:43.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:34:43.218 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.218 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.218 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:34:43.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:34:43.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:34:43.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:43.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:34:43.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:34:43.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:34:43.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:34:43.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:34:43.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:34:43.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:34:43.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:34:43.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:34:43.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:34:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:34:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3120 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'290, mlcod=0'0)", 2023-01-15T14:34:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:34:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.927551457999996, 2023-01-15T14:34:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39853714000000001, 2023-01-15T14:34:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:34:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239424+0000", 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239747+0000", 2023-01-15T14:34:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032243300000000001 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239784+0000", 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7051000000000002e-05 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239802+0000", 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8250999999999999e-05 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536033+0000", 2023-01-15T14:34:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29623121499999999 2023-01-15T14:34:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536081+0000", 2023-01-15T14:34:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2023-01-15T14:34:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536800+0000", 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071948499999999996 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536817+0000", 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6228000000000001e-05 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537048+0000", 2023-01-15T14:34:43.229 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023136200000000001 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637902+0000", 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10085403900000001 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637963+0000", 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1310999999999996e-05 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.230 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.231 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.231 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3121 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'291, mlcod=0'0)", 2023-01-15T14:34:43.231 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:34:43.231 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.925555942000003, 2023-01-15T14:34:43.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659573799999998, 2023-01-15T14:34:43.231 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.231 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.357 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:34:43.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:34:43.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241420+0000", 2023-01-15T14:34:43.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:43.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241854+0000", 2023-01-15T14:34:43.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043330099999999998 2023-01-15T14:34:43.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241878+0000", 2023-01-15T14:34:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4559e-05 2023-01-15T14:34:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241892+0000", 2023-01-15T14:34:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039999999999999e-05 2023-01-15T14:34:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536863+0000", 2023-01-15T14:34:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29497073400000001 2023-01-15T14:34:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536887+0000", 2023-01-15T14:34:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3683999999999999e-05 2023-01-15T14:34:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537124+0000", 2023-01-15T14:34:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023729900000000001 2023-01-15T14:34:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537136+0000", 2023-01-15T14:34:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2278e-05 2023-01-15T14:34:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537345+0000", 2023-01-15T14:34:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020865200000000001 2023-01-15T14:34:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637981+0000", 2023-01-15T14:34:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063575700000001 2023-01-15T14:34:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638018+0000", 2023-01-15T14:34:43.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6826999999999998e-05 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3122 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'292, mlcod=0'0)", 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.923660101999999, 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39623551600000001, 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.365 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:34:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:34:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243316+0000", 2023-01-15T14:34:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243833+0000", 2023-01-15T14:34:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051703600000000001 2023-01-15T14:34:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243863+0000", 2023-01-15T14:34:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-01-15T14:34:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243879+0000", 2023-01-15T14:34:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6263999999999999e-05 2023-01-15T14:34:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637317+0000", 2023-01-15T14:34:43.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39343783199999999 2023-01-15T14:34:43.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637356+0000", 2023-01-15T14:34:43.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9579000000000002e-05 2023-01-15T14:34:43.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637830+0000", 2023-01-15T14:34:43.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047326399999999999 2023-01-15T14:34:43.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637885+0000", 2023-01-15T14:34:43.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5074999999999997e-05 2023-01-15T14:34:43.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638185+0000", 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029993799999999997 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638337+0000", 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015208299999999999 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.639553+0000", 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001216389 2023-01-15T14:34:43.371 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.372 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.372 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.372 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3123 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'293, mlcod=0'0)", 2023-01-15T14:34:43.372 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:34:43.372 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.921916073999995, 2023-01-15T14:34:43.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.395733102, 2023-01-15T14:34:43.372 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.372 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.372 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:34:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:34:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245060+0000", 2023-01-15T14:34:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245075+0000", 2023-01-15T14:34:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4459e-05 2023-01-15T14:34:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245107+0000", 2023-01-15T14:34:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1795000000000003e-05 2023-01-15T14:34:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245121+0000", 2023-01-15T14:34:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4117000000000001e-05 2023-01-15T14:34:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638063+0000", 2023-01-15T14:34:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39294204300000002 2023-01-15T14:34:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638097+0000", 2023-01-15T14:34:43.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4385999999999999e-05 2023-01-15T14:34:43.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638299+0000", 2023-01-15T14:34:43.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020133099999999999 2023-01-15T14:34:43.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638318+0000", 2023-01-15T14:34:43.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514999999999999e-05 2023-01-15T14:34:43.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638386+0000", 2023-01-15T14:34:43.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8089000000000005e-05 2023-01-15T14:34:43.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638430+0000", 2023-01-15T14:34:43.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.439e-05 2023-01-15T14:34:43.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.640795+0000", 2023-01-15T14:34:43.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002364274 2023-01-15T14:34:43.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.385 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.386 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.386 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.386 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:43.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:34:43.389 INFO:tasks.rados.rados.0.smithi090.stdout:1490: finishing write tid 1 to smithi09025264-12 2023-01-15T14:34:43.389 INFO:tasks.rados.rados.0.smithi090.stdout:1490: finishing write tid 2 to smithi09025264-12 2023-01-15T14:34:43.389 INFO:tasks.rados.rados.0.smithi090.stdout:1490: finishing write tid 3 to smithi09025264-12 2023-01-15T14:34:43.389 INFO:tasks.rados.rados.0.smithi090.stdout:1490: finishing write tid 4 to smithi09025264-12 2023-01-15T14:34:43.390 INFO:tasks.rados.rados.0.smithi090.stdout:1489: done (2 left) 2023-01-15T14:34:43.390 INFO:tasks.rados.rados.0.smithi090.stdout:1491: done (1 left) 2023-01-15T14:34:43.390 INFO:tasks.rados.rados.0.smithi090.stdout:1492: read oid 37 snap 143 2023-01-15T14:34:43.390 INFO:tasks.rados.rados.0.smithi090.stdout:1492: expect (ObjNum 481 snap 142 seq_num 481) 2023-01-15T14:34:43.390 INFO:tasks.rados.rados.0.smithi090.stdout:1493: delete oid 26 current snap is 161 2023-01-15T14:34:43.392 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:34:43.392 INFO:tasks.rados.rados.0.smithi090.stdout:1490: finishing write tid 5 to smithi09025264-12 2023-01-15T14:34:43.393 INFO:tasks.rados.rados.0.smithi090.stdout:1490: oid 12 updating version 0 to 321 2023-01-15T14:34:43.393 INFO:tasks.rados.rados.0.smithi090.stdout:1490: oid 12 version 321 is already newer than 319 2023-01-15T14:34:43.393 INFO:tasks.rados.rados.0.smithi090.stdout:1490: oid 12 version 321 is already newer than 320 2023-01-15T14:34:43.393 INFO:tasks.rados.rados.0.smithi090.stdout:1490: oid 12 updating version 321 to 322 2023-01-15T14:34:43.393 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 322 (ObjNum 533 snap 161 seq_num 533) dirty exists 2023-01-15T14:34:43.394 INFO:tasks.rados.rados.0.smithi090.stdout:1490: left oid 12 (ObjNum 533 snap 161 seq_num 533) 2023-01-15T14:34:43.394 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:43.386+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:43.464 INFO:tasks.rados.rados.0.smithi090.stdout:1490: done (2 left) 2023-01-15T14:34:43.521 INFO:tasks.rados.rados.0.smithi090.stdout:1493: done (1 left) 2023-01-15T14:34:43.521 INFO:tasks.rados.rados.0.smithi090.stdout:1494: snap_create 2023-01-15T14:34:43.521 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:43.522 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:34:43.522 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:34:43.522 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:43.523 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:34:43.523 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:43.515+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:43.623 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:43.619+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:43.634 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:43.634 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:34:43.634 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:34:43.634 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:34:43.634 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:43.644 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:34:43.724 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:43.719+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:43.801 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:43.801 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:34:43.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:34:43.802 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:34:43.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.802 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:43.802 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:43.802 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 376.66819830600002, 2023-01-15T14:34:43.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:34:43.802 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.802 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:43.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:43.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:43.803 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:34:43.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.803 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:43.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:43.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:34:43.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:43.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:34:43.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:34:43.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:34:43.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:34:43.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:34:43.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:34:43.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:34:43.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:34:43.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:43.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:34:43.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:34:43.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:34:43.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:34:43.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:34:43.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:34:43.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:34:43.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:34:43.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:34:43.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:34:43.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:34:43.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:34:43.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:34:43.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:34:43.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:34:43.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:43.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:34:43.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:34:43.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:43.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:34:43.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:34:43.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:43.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:34:43.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:34:43.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:34:43.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:34:43.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:34:43.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:34:43.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.814 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.815 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:43.815 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:43.815 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 376.66779496999999, 2023-01-15T14:34:43.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:34:43.815 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.815 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:43.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:43.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:43.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:43.816 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:34:43.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.816 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:43.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:43.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:34:43.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:43.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:34:43.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:34:43.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:34:43.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:34:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:34:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:34:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:34:43.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:34:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:34:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:34:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:34:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:34:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:34:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:34:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:34:43.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:34:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:34:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:34:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:34:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:34:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:34:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:34:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:34:43.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:34:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:34:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:34:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:34:43.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:34:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:34:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:34:43.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:34:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:34:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:34:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 376.56757134999998, 2023-01-15T14:34:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:34:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:43.827 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:34:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:34:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:43.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:34:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:34:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:34:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:34:43.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:34:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:34:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:34:43.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:34:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:34:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:34:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:34:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:34:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:34:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:34:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:34:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:34:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:34:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:34:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:34:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:34:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:43.835 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 285.87425612499999, 2023-01-15T14:34:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:34:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:34:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.836 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:43.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:34:43.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:34:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:34:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:34:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:34:43.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:34:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:34:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:34:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:34:43.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:34:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:34:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:34:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:34:43.841 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 285.86779477099998, 2023-01-15T14:34:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:34:43.842 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:34:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:34:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:34:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:34:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:34:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:34:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:34:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:34:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:34:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:34:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:34:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:34:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.848 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 285.767372846, 2023-01-15T14:34:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:34:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:43.849 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:34:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:34:43.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:34:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:34:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:34:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:34:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:34:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:34:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:34:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:34:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:34:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:34:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:34:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:34:43.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:34:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:34:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.856 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 285.76703536000002, 2023-01-15T14:34:43.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:34:43.997 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:34:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:43.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:34:43.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:34:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:34:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:34:44.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:34:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:34:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:34:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:34:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:34:44.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:34:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:34:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:34:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:34:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:34:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:34:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:44.003 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 285.76697946399997, 2023-01-15T14:34:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:34:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:34:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:34:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:44.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:34:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:34:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:34:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:34:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:44.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 243.933729046, 2023-01-15T14:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:34:44.325 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:44.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:44.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:34:44.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:34:44.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:34:44.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:34:44.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:34:44.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:34:44.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:44.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:34:44.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:34:44.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:34:44.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:34:44.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:44.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:34:44.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:34:44.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:44.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:34:44.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:34:44.487 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.487 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:44.488 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.488 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2207 3.5 3:a1c47f99:::smithi09025264-2:head [write 2548663~763022] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:44.488 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:34:44.488 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 184.95897131800001, 2023-01-15T14:34:44.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79657873300000004, 2023-01-15T14:34:44.488 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:44.488 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:44.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:44.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:44.489 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:44.489 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2207 2023-01-15T14:34:44.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.489 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:44.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:44.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:34:44.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:44.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:34:44.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:44.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837590+0000", 2023-01-15T14:34:44.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:44.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:44.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837960+0000", 2023-01-15T14:34:44.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037055900000000001 2023-01-15T14:34:44.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:44.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837980+0000", 2023-01-15T14:34:44.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9953e-05 2023-01-15T14:34:44.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:44.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837988+0000", 2023-01-15T14:34:44.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2800000000000003e-06 2023-01-15T14:34:44.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927512+0000", 2023-01-15T14:34:44.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089523563 2023-01-15T14:34:44.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:44.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927645+0000", 2023-01-15T14:34:44.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013285799999999999 2023-01-15T14:34:44.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225466+0000", 2023-01-15T14:34:44.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29782133100000002 2023-01-15T14:34:44.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:44.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225553+0000", 2023-01-15T14:34:44.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6866000000000005e-05 2023-01-15T14:34:44.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:44.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225708+0000", 2023-01-15T14:34:44.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015455199999999999 2023-01-15T14:34:44.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:44.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225971+0000", 2023-01-15T14:34:44.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026317099999999998 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226027+0000", 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6601000000000002e-05 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226470+0000", 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044267500000000001 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227372+0000", 2023-01-15T14:34:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090215200000000003 2023-01-15T14:34:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429314+0000", 2023-01-15T14:34:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20194183499999999 2023-01-15T14:34:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:44.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.633999+0000", 2023-01-15T14:34:44.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20468505400000001 2023-01-15T14:34:44.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:44.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634075+0000", 2023-01-15T14:34:44.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:34:44.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:44.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634170+0000", 2023-01-15T14:34:44.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4821999999999994e-05 2023-01-15T14:34:44.498 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.498 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:44.498 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.498 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2208 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (56) in=7b,truncate 3395047] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:44.498 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:44.498 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 184.95854872500001, 2023-01-15T14:34:44.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79650439900000003, 2023-01-15T14:34:44.498 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2208 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:44.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838018+0000", 2023-01-15T14:34:44.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2549999999999997e-06 2023-01-15T14:34:44.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:44.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838031+0000", 2023-01-15T14:34:44.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2717999999999999e-05 2023-01-15T14:34:44.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:44.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838036+0000", 2023-01-15T14:34:44.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2240000000000003e-06 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927676+0000", 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089639620000000003 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927804+0000", 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012832300000000001 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226096+0000", 2023-01-15T14:34:44.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29829207400000002 2023-01-15T14:34:44.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:44.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226347+0000", 2023-01-15T14:34:44.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025084800000000002 2023-01-15T14:34:44.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:44.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226750+0000", 2023-01-15T14:34:44.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040343200000000001 2023-01-15T14:34:44.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:44.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227147+0000", 2023-01-15T14:34:44.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039682900000000002 2023-01-15T14:34:44.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:44.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227600+0000", 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045321199999999998 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227859+0000", 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259041 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227895+0000", 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5710999999999997e-05 2023-01-15T14:34:44.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:44.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429405+0000", 2023-01-15T14:34:44.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201509415 2023-01-15T14:34:44.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:44.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634300+0000", 2023-01-15T14:34:44.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20489528200000001 2023-01-15T14:34:44.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:44.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634409+0000", 2023-01-15T14:34:44.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109524 2023-01-15T14:34:44.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:44.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634518+0000", 2023-01-15T14:34:44.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108548 2023-01-15T14:34:44.634 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.634 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:44.634 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.634 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2209 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 184.95851991000001, 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79724995499999995, 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2209 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:44.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:44.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:44.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:44.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:44.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:44.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:34:44.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:44.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:44.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838044+0000", 2023-01-15T14:34:44.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6120000000000001e-06 2023-01-15T14:34:44.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:44.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838056+0000", 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1236000000000001e-05 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838059+0000", 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6940000000000001e-06 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.027908+0000", 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189848287 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:44.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.028049+0000", 2023-01-15T14:34:44.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014132500000000001 2023-01-15T14:34:44.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227252+0000", 2023-01-15T14:34:44.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19920324 2023-01-15T14:34:44.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227341+0000", 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8683000000000005e-05 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634637+0000", 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729567300000002 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634980+0000", 2023-01-15T14:34:44.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034335000000000002 2023-01-15T14:34:44.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:44.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.635292+0000", 2023-01-15T14:34:44.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003123 2023-01-15T14:34:44.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.644 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:44.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.645 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2222 3.c 3:3137f0cb:::smithi09025264-20:head [write 3259733~152121] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:44.645 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:34:44.645 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 184.93866835099999, 2023-01-15T14:34:44.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77651572400000002, 2023-01-15T14:34:44.645 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:44.645 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:44.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:44.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:44.645 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:44.645 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2222 2023-01-15T14:34:44.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.646 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:44.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:44.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:34:44.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:44.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:34:44.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:44.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857891+0000", 2023-01-15T14:34:44.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:44.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:44.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857993+0000", 2023-01-15T14:34:44.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010276399999999999 2023-01-15T14:34:44.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:44.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858021+0000", 2023-01-15T14:34:44.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8022e-05 2023-01-15T14:34:44.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:44.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858034+0000", 2023-01-15T14:34:44.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2549e-05 2023-01-15T14:34:44.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025371+0000", 2023-01-15T14:34:44.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16733653100000001 2023-01-15T14:34:44.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:44.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025429+0000", 2023-01-15T14:34:44.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8922000000000003e-05 2023-01-15T14:34:44.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226869+0000", 2023-01-15T14:34:44.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20143931100000001 2023-01-15T14:34:44.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:44.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227088+0000", 2023-01-15T14:34:44.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219328 2023-01-15T14:34:44.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:44.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227443+0000", 2023-01-15T14:34:44.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035481 2023-01-15T14:34:44.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:44.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227830+0000", 2023-01-15T14:34:44.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000386702 2023-01-15T14:34:44.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:44.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227861+0000", 2023-01-15T14:34:44.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1584000000000001e-05 2023-01-15T14:34:44.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:44.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227939+0000", 2023-01-15T14:34:44.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7354999999999994e-05 2023-01-15T14:34:44.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:44.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228548+0000", 2023-01-15T14:34:44.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060954800000000001 2023-01-15T14:34:44.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:44.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429817+0000", 2023-01-15T14:34:44.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20126904600000001 2023-01-15T14:34:44.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:44.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634125+0000", 2023-01-15T14:34:44.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20430780300000001 2023-01-15T14:34:44.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:44.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634259+0000", 2023-01-15T14:34:44.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134144 2023-01-15T14:34:44.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:44.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634410+0000", 2023-01-15T14:34:44.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150447 2023-01-15T14:34:44.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.656 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:44.656 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.657 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2223 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (57) in=7b,truncate 3411854] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:44.657 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:34:44.699 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 184.93850956399999, 2023-01-15T14:34:44.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87671107599999998, 2023-01-15T14:34:44.699 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:44.699 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:44.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:44.699 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:44.700 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:44.700 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2223 2023-01-15T14:34:44.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.700 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:44.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:44.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:34:44.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:44.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:34:44.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858052+0000", 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858056+0000", 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5619999999999997e-06 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:44.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858067+0000", 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0328e-05 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858071+0000", 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6439999999999999e-06 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025446+0000", 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16737407200000001 2023-01-15T14:34:44.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:44.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025503+0000", 2023-01-15T14:34:44.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7952999999999999e-05 2023-01-15T14:34:44.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.704 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.704 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227970+0000", 2023-01-15T14:34:44.704 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202466866 2023-01-15T14:34:44.704 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.704 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.705 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:44.705 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228139+0000", 2023-01-15T14:34:44.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016889100000000001 2023-01-15T14:34:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:34:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228284+0000", 2023-01-15T14:34:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014493700000000001 2023-01-15T14:34:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228470+0000", 2023-01-15T14:34:44.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185541 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228481+0000", 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.167e-05 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228537+0000", 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5912999999999998e-05 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228563+0000", 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5613e-05 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.433181+0000", 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20461838099999999 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:44.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734620+0000", 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301438386 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734682+0000", 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2373999999999996e-05 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734764+0000", 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1656999999999994e-05 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.815 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2224 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 184.93848451400001, 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97704279199999999, 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:44.816 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2224 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:34:44.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:44.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:44.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858077+0000", 2023-01-15T14:34:44.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:44.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858080+0000", 2023-01-15T14:34:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:34:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858089+0000", 2023-01-15T14:34:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3449999999999995e-06 2023-01-15T14:34:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:44.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858093+0000", 2023-01-15T14:34:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9820000000000002e-06 2023-01-15T14:34:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025520+0000", 2023-01-15T14:34:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.167426556 2023-01-15T14:34:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025594+0000", 2023-01-15T14:34:44.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4339999999999996e-05 2023-01-15T14:34:44.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228608+0000", 2023-01-15T14:34:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20301395699999999 2023-01-15T14:34:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228752+0000", 2023-01-15T14:34:44.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014385 2023-01-15T14:34:44.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834878+0000", 2023-01-15T14:34:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60612591500000002 2023-01-15T14:34:44.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834987+0000", 2023-01-15T14:34:45.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108733 2023-01-15T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.835120+0000", 2023-01-15T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133688 2023-01-15T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3120 3.40 3:03b54904:::smithi09025264-12:head [write 578273~466432] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.864532632000007, 2023-01-15T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72661194799999995, 2023-01-15T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:45.287 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3120 2023-01-15T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:34:45.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932026+0000", 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932397+0000", 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037070099999999997 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932419+0000", 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1923999999999999e-05 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:45.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932435+0000", 2023-01-15T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6509999999999999e-05 2023-01-15T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933751+0000", 2023-01-15T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013156050000000001 2023-01-15T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933765+0000", 2023-01-15T14:34:45.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3952e-05 2023-01-15T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238456+0000", 2023-01-15T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30469085000000001 2023-01-15T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238797+0000", 2023-01-15T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034117300000000001 2023-01-15T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239214+0000", 2023-01-15T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000416845 2023-01-15T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240273+0000", 2023-01-15T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058811 2023-01-15T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240292+0000", 2023-01-15T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240555+0000", 2023-01-15T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026299500000000003 2023-01-15T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242072+0000", 2023-01-15T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015175550000000001 2023-01-15T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244038+0000", 2023-01-15T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001966176 2023-01-15T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658099+0000", 2023-01-15T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41406103300000002 2023-01-15T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658248+0000", 2023-01-15T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148577 2023-01-15T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:45.300 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658642+0000", 2023-01-15T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039346900000000002 2023-01-15T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3121 3.40 3:03b54904:::smithi09025264-12:head [write 1739595~452815] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.864100686, 2023-01-15T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72675957599999996, 2023-01-15T14:34:45.301 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3121 2023-01-15T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:34:45.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932460+0000", 2023-01-15T14:34:45.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932725+0000", 2023-01-15T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026482699999999998 2023-01-15T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:45.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932741+0000", 2023-01-15T14:34:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5880999999999999e-05 2023-01-15T14:34:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932750+0000", 2023-01-15T14:34:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2380000000000003e-06 2023-01-15T14:34:45.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034479+0000", 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10172906399999999 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034506+0000", 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7336000000000001e-05 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:45.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240672+0000", 2023-01-15T14:34:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206166074 2023-01-15T14:34:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240894+0000", 2023-01-15T14:34:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221189 2023-01-15T14:34:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241244+0000", 2023-01-15T14:34:45.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000350816 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241923+0000", 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067891200000000005 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241943+0000", 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0069999999999999e-05 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243425+0000", 2023-01-15T14:34:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001481719 2023-01-15T14:34:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244087+0000", 2023-01-15T14:34:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066184700000000002 2023-01-15T14:34:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244230+0000", 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143277 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658784+0000", 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.414553479 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:45.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658904+0000", 2023-01-15T14:34:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120024 2023-01-15T14:34:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659221+0000", 2023-01-15T14:34:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031742700000000001 2023-01-15T14:34:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3122 3.40 3:03b54904:::smithi09025264-12:head [write 2722929~655344] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:34:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.863796588, 2023-01-15T14:34:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72706898900000005, 2023-01-15T14:34:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:45.437 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3122 2023-01-15T14:34:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:34:45.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:45.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:45.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:34:45.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932765+0000", 2023-01-15T14:34:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933097+0000", 2023-01-15T14:34:45.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220299999999999 2023-01-15T14:34:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933106+0000", 2023-01-15T14:34:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5459999999999997e-06 2023-01-15T14:34:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933113+0000", 2023-01-15T14:34:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3419999999999996e-06 2023-01-15T14:34:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034598+0000", 2023-01-15T14:34:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101485804 2023-01-15T14:34:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:45.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034625+0000", 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6859000000000002e-05 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242147+0000", 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20752189600000001 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242517+0000", 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037032200000000002 2023-01-15T14:34:45.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243157+0000", 2023-01-15T14:34:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063935700000000001 2023-01-15T14:34:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243809+0000", 2023-01-15T14:34:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065222000000000001 2023-01-15T14:34:45.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243827+0000", 2023-01-15T14:34:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8260000000000001e-05 2023-01-15T14:34:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244122+0000", 2023-01-15T14:34:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029469400000000002 2023-01-15T14:34:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245633+0000", 2023-01-15T14:34:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001511068 2023-01-15T14:34:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.346711+0000", 2023-01-15T14:34:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101077812 2023-01-15T14:34:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659372+0000", 2023-01-15T14:34:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31266154800000001 2023-01-15T14:34:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659488+0000", 2023-01-15T14:34:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115444 2023-01-15T14:34:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659835+0000", 2023-01-15T14:34:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034670300000000002 2023-01-15T14:34:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3123 3.40 3:03b54904:::smithi09025264-12:head [setxattr _header (58) in=7b,truncate 3378273] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:45.474 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.863426895000003, 2023-01-15T14:34:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82751564099999997, 2023-01-15T14:34:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3123 2023-01-15T14:34:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:45.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:45.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:45.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:45.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:34:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:45.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933139+0000", 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7079999999999996e-06 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933145+0000", 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7849999999999997e-06 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033331+0000", 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100186076 2023-01-15T14:34:45.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034697+0000", 2023-01-15T14:34:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001366138 2023-01-15T14:34:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034718+0000", 2023-01-15T14:34:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.065e-05 2023-01-15T14:34:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244319+0000", 2023-01-15T14:34:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20960071 2023-01-15T14:34:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:45.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244576+0000", 2023-01-15T14:34:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025712899999999998 2023-01-15T14:34:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244932+0000", 2023-01-15T14:34:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035626000000000001 2023-01-15T14:34:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245315+0000", 2023-01-15T14:34:45.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038305199999999999 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245330+0000", 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5027e-05 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245384+0000", 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4267000000000001e-05 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:45.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245654+0000", 2023-01-15T14:34:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026944000000000002 2023-01-15T14:34:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.436220+0000", 2023-01-15T14:34:45.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19056593799999999 2023-01-15T14:34:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760086+0000", 2023-01-15T14:34:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.323865927 2023-01-15T14:34:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:45.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760241+0000", 2023-01-15T14:34:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015535700000000001 2023-01-15T14:34:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760651+0000", 2023-01-15T14:34:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040990699999999999 2023-01-15T14:34:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3124 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:34:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:34:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.763200416999993, 2023-01-15T14:34:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72792673799999996, 2023-01-15T14:34:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:45.488 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3124 2023-01-15T14:34:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:34:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:34:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033360+0000", 2023-01-15T14:34:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:45.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033372+0000", 2023-01-15T14:34:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2850000000000001e-05 2023-01-15T14:34:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033397+0000", 2023-01-15T14:34:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4349e-05 2023-01-15T14:34:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.492 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:45.492 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033412+0000", 2023-01-15T14:34:45.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5420000000000001e-05 2023-01-15T14:34:45.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:45.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036811+0000", 2023-01-15T14:34:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033984169999999999 2023-01-15T14:34:45.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036821+0000", 2023-01-15T14:34:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0213e-05 2023-01-15T14:34:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245712+0000", 2023-01-15T14:34:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20889102800000001 2023-01-15T14:34:45.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245912+0000", 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020024600000000001 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760781+0000", 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51486881299999998 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760970+0000", 2023-01-15T14:34:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018893600000000001 2023-01-15T14:34:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.761289+0000", 2023-01-15T14:34:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031870899999999997 2023-01-15T14:34:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.608 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:45.609 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:45.609 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:45.609 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:34:45.610 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:34:45.611 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:45.606+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:45.612 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:34:45.613 INFO:tasks.rados.rados.0.smithi090.stdout:1492: expect (ObjNum 481 snap 142 seq_num 481) 2023-01-15T14:34:45.614 INFO:tasks.rados.rados.0.smithi090.stdout:1492: done (0 left) 2023-01-15T14:34:45.614 INFO:tasks.rados.rados.0.smithi090.stdout:1494: done (0 left) 2023-01-15T14:34:45.614 INFO:tasks.rados.rados.0.smithi090.stdout:1495: snap_create 2023-01-15T14:34:45.712 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:45.707+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:45.751 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:45.752 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:34:45.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:45.752 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.1.0:625 3.65 e395/393)", 2023-01-15T14:34:45.752 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:34:45.723069+0000", 2023-01-15T14:34:45.752 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.0078119540000000003, 2023-01-15T14:34:45.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0078411330000000001, 2023-01-15T14:34:45.792 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:45.793 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "no flag points reached", 2023-01-15T14:34:45.793 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:45.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:45.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:45.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:45.723069+0000", 2023-01-15T14:34:45.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:45.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:45.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:45.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:45.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:45.723069+0000", 2023-01-15T14:34:45.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:45.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:45.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:45.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:45.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:45.723067+0000", 2023-01-15T14:34:45.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:45.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:45.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:45.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:45.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:45.723081+0000", 2023-01-15T14:34:45.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:34:45.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:45.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:45.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:45.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:45.723117+0000", 2023-01-15T14:34:45.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6295999999999999e-05 2023-01-15T14:34:45.795 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:45.795 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:45.795 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:45.795 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:45.795 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2023-01-15T14:34:45.796 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 1 2023-01-15T14:34:45.796 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:45.796 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:34:45.797 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:45.932 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:45.932 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:45.932 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:45.932 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:45.932 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:45.932 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:45.933 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:45.926+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:45.937 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:45.939 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:34:45.939 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:34:45.939 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:34:45.939 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:34:45.939 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:34:45.939 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:34:45.952 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:34:46.106 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:34:46.108 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:46.109 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:46.110 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:34:46.110 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:46.102+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:46.111 INFO:tasks.rados.rados.0.smithi090.stdout:1495: done (0 left) 2023-01-15T14:34:46.111 INFO:tasks.rados.rados.0.smithi090.stdout:1496: read oid 31 snap -1 2023-01-15T14:34:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:1496: expect (ObjNum 520 snap 158 seq_num 520) 2023-01-15T14:34:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:1497: copy_from oid 2 from oid 26 current snap is 163 2023-01-15T14:34:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:1498: read oid 3 snap -1 2023-01-15T14:34:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:1498: expect (ObjNum 464 snap 133 seq_num 464) 2023-01-15T14:34:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:1499: read oid 46 snap -1 2023-01-15T14:34:46.112 INFO:tasks.rados.rados.0.smithi090.stdout:1499: expect (ObjNum 506 snap 150 seq_num 506) 2023-01-15T14:34:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:1500: rollback oid 29 current snap is 163 2023-01-15T14:34:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 29 to 139 2023-01-15T14:34:46.113 INFO:tasks.rados.rados.0.smithi090.stdout:1501: snap_remove snap 156 2023-01-15T14:34:46.114 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:34:46.209 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:46.203+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:46.264 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:46.269 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:46.274 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:34:46.721 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:46.715+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:46.822 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:46.818+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:46.886 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:46.886 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:46.887 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:34:46.923 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:46.919+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:46.947 INFO:tasks.rados.rados.0.smithi090.stdout:1501: done (5 left) 2023-01-15T14:34:46.947 INFO:tasks.rados.rados.0.smithi090.stdout:1502: snap_create 2023-01-15T14:34:46.947 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:34:46.947 INFO:tasks.rados.rados.0.smithi090.stdout:1496: expect (ObjNum 520 snap 158 seq_num 520) 2023-01-15T14:34:47.024 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:47.019+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:47.047 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:47.047 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:47.048 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:34:47.062 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:34:47.062 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:34:47.062 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:34:47.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:34:47.089 INFO:tasks.rados.rados.0.smithi090.stdout:1498: expect (ObjNum 464 snap 133 seq_num 464) 2023-01-15T14:34:47.125 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:47.119+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:47.153 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:47.229 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:47.231 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:34:47.231 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:47.223+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:47.300 INFO:tasks.rados.rados.0.smithi090.stdout:1499: expect (ObjNum 506 snap 150 seq_num 506) 2023-01-15T14:34:48.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:34:48.151 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:34:48.158 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:48.158 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:48.159 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:34:48.159 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:48.138+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:48.166 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:48.171 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:34:48.171 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":590,"stamp":"2023-01-15T14:34:46.962625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":417880554,"num_objects":204,"num_object_clones":148,"num_object_copies":612,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":204,"num_whiteouts":18,"num_read":1073,"num_read_kb":1044461,"num_write":906,"num_write_kb":311819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":424803324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18808,"ondisk_log_size":18808,"up":345,"acting":326,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":352,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6542076,"kb_used_data":555669520,"kb_used_omap":78925,"kb_used_meta":0,"kb_avail":555654204,"statfs":{"total":575688990720,"available":568989904896,"internally_reserved":15683584,"allocated":569005588480,"data_stored":569005588480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80819307,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,0,2,3,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":13,"commit_latency_ns":0,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17143979,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-102,"num_objects_misplaced":-10,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":37,"num_read_kb":37630,"num_write":32,"num_write_kb":13444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":22175531,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001845"},"pg_stats":[{"pgid":"3.69","version":"110'149","reported_seq":1006,"reported_epoch":393,"state":"remapped+peering","last_fresh":"2023-01-15T14:34:42.958267+0000","last_change":"2023-01-15T14:34:42.958501+0000","last_active":"2023-01-15T14:34:41.943256+0000","last_peered":"2023-01-15T14:34:41.006187+0000","last_clean":"2023-01-15T14:34:41.006187+0000","last_became_active":"2023-01-15T14:34:40.496753+0000","last_became_peered":"2023-01-15T14:34:40.496753+0000","last_unstale":"2023-01-15T14:34:42.958267+0000","last_undegraded":"2023-01-15T14:34:42.958267+0000","last_fullsized":"2023-01-15T14:34:42.958267+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.50810617599999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.68","version":"391'267","reported_seq":1520,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.167129+0000","last_change":"2023-01-15T14:34:42.216961+0000","last_active":"2023-01-15T14:34:45.167129+0000","last_peered":"2023-01-15T14:34:45.167129+0000","last_clean":"2023-01-15T14:34:45.167129+0000","last_became_active":"2023-01-15T14:34:13.134258+0000","last_became_peered":"2023-01-15T14:34:13.134258+0000","last_unstale":"2023-01-15T14:34:45.167129+0000","last_undegraded":"2023-01-15T14:34:45.167129+0000","last_fullsized":"2023-01-15T14:34:45.167129+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.27587281400000002,"stat_sum":{"num_bytes":8014979,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17027823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.67","version":"191'79","reported_seq":137,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.317061+0000","last_change":"2023-01-15T14:34:42.317061+0000","last_active":"2023-01-15T14:34:42.317061+0000","last_peered":"2023-01-15T14:34:42.317061+0000","last_clean":"2023-01-15T14:34:42.317061+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:34:42.317061+0000","last_undegraded":"2023-01-15T14:34:42.317061+0000","last_fullsized":"2023-01-15T14:34:42.317061+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.37334164600000003,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.66","version":"102'52","reported_seq":476,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.362845+0000","last_change":"2023-01-15T14:34:43.362488+0000","last_active":"2023-01-15T14:34:43.362845+0000","last_peered":"2023-01-15T14:34:43.362845+0000","last_clean":"2023-01-15T14:34:43.362845+0000","last_became_active":"2023-01-15T14:34:43.359681+0000","last_became_peered":"2023-01-15T14:34:43.359681+0000","last_unstale":"2023-01-15T14:34:43.362845+0000","last_undegraded":"2023-01-15T14:34:43.362845+0000","last_fullsized":"2023-01-15T14:34:43.362845+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:34:32.591909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.591909+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:36.991","scrub_duration":0.39200112999999998,"objects_trimmed":0,"snaptrim_duration":0.20166806600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.65","version":"384'136","reported_seq":1314,"reported_epoch":388,"state":"active+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:38.247630+0000","last_change":"2023-01-15T14:34:38.246778+0000","last_active":"2023-01-15T14:34:38.247630+0000","last_peered":"2023-01-15T14:34:38.247630+0000","last_clean":"2023-01-15T14:34:34.854778+0000","last_became_active":"2023-01-15T14:34:38.246778+0000","last_became_peered":"2023-01-15T14:34:38.246778+0000","last_unstale":"2023-01-15T14:34:38.247630+0000","last_undegraded":"2023-01-15T14:34:37.943614+0000","last_fullsized":"2023-01-15T14:34:37.942823+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'126","last_scrub_stamp":"2023-01-15T14:34:23.903906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.903906+0000","objects_scrubbed":6,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:25.294","scrub_duration":0.30981859099999998,"objects_trimmed":0,"snaptrim_duration":0.102466811,"stat_sum":{"num_bytes":9862828,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":10,"num_read_kb":13204,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4],"avail_no_missing":["7","1","4"],"object_location_counts":[{"shards":"1,4,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.64","version":"334'251","reported_seq":1497,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.167495+0000","last_change":"2023-01-15T14:34:39.269378+0000","last_active":"2023-01-15T14:34:45.167495+0000","last_peered":"2023-01-15T14:34:45.167495+0000","last_clean":"2023-01-15T14:34:45.167495+0000","last_became_active":"2023-01-15T14:33:55.675085+0000","last_became_peered":"2023-01-15T14:33:55.675085+0000","last_unstale":"2023-01-15T14:34:45.167495+0000","last_undegraded":"2023-01-15T14:34:45.167495+0000","last_fullsized":"2023-01-15T14:34:45.167495+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.305529465,"stat_sum":{"num_bytes":3548733,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10773,"num_write":5,"num_write_kb":2741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.63","version":"149'233","reported_seq":826,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.067586+0000","last_change":"2023-01-15T14:34:39.272890+0000","last_active":"2023-01-15T14:34:45.067586+0000","last_peered":"2023-01-15T14:34:45.067586+0000","last_clean":"2023-01-15T14:34:45.067586+0000","last_became_active":"2023-01-15T14:33:55.277739+0000","last_became_peered":"2023-01-15T14:33:55.277739+0000","last_unstale":"2023-01-15T14:34:45.067586+0000","last_undegraded":"2023-01-15T14:34:45.067586+0000","last_fullsized":"2023-01-15T14:34:45.067586+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30305257699999999,"stat_sum":{"num_bytes":3802136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":21819,"num_write":19,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.62","version":"388'177","reported_seq":1404,"reported_epoch":389,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:38.966644+0000","last_change":"2023-01-15T14:34:38.347807+0000","last_active":"2023-01-15T14:34:38.966644+0000","last_peered":"2023-01-15T14:34:38.966644+0000","last_clean":"2023-01-15T14:34:25.278266+0000","last_became_active":"2023-01-15T14:34:38.345967+0000","last_became_peered":"2023-01-15T14:34:38.345967+0000","last_unstale":"2023-01-15T14:34:38.966644+0000","last_undegraded":"2023-01-15T14:34:38.035502+0000","last_fullsized":"2023-01-15T14:34:38.034611+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"375'173","last_scrub_stamp":"2023-01-15T14:34:34.093838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:34.093838+0000","objects_scrubbed":2,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.12526480100000001,"stat_sum":{"num_bytes":6616147,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":14815,"num_write":15,"num_write_kb":6240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5210269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":2},{"shards":"2,4","objects":-1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.61","version":"395'158","reported_seq":1128,"reported_epoch":396,"state":"active+clean","last_fresh":"2023-01-15T14:34:46.576220+0000","last_change":"2023-01-15T14:34:46.677230+0000","last_active":"2023-01-15T14:34:46.576220+0000","last_peered":"2023-01-15T14:34:46.576220+0000","last_clean":"2023-01-15T14:34:46.576220+0000","last_became_active":"2023-01-15T14:34:44.473866+0000","last_became_peered":"2023-01-15T14:34:44.473866+0000","last_unstale":"2023-01-15T14:34:46.576220+0000","last_undegraded":"2023-01-15T14:34:46.576220+0000","last_fullsized":"2023-01-15T14:34:46.576220+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:32.734","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.70442659299999999,"stat_sum":{"num_bytes":3823692,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6060,"num_write":6,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.60","version":"376'285","reported_seq":1464,"reported_epoch":389,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:38.963954+0000","last_change":"2023-01-15T14:34:38.349090+0000","last_active":"2023-01-15T14:34:38.963954+0000","last_peered":"2023-01-15T14:34:38.963954+0000","last_clean":"2023-01-15T14:34:26.602032+0000","last_became_active":"2023-01-15T14:34:38.247298+0000","last_became_peered":"2023-01-15T14:34:38.247298+0000","last_unstale":"2023-01-15T14:34:38.963954+0000","last_undegraded":"2023-01-15T14:34:37.938765+0000","last_fullsized":"2023-01-15T14:34:37.937286+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":347,"parent":"0.0","parent_split_bits":7,"last_scrub":"376'285","last_scrub_stamp":"2023-01-15T14:34:32.444552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.444552+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.026168291999999999,"stat_sum":{"num_bytes":5214108,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":19437,"num_write":14,"num_write_kb":5082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15418818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":4},{"shards":"2,4","objects":-1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1460,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.069825+0000","last_change":"2023-01-15T14:34:39.947186+0000","last_active":"2023-01-15T14:34:45.069825+0000","last_peered":"2023-01-15T14:34:45.069825+0000","last_clean":"2023-01-15T14:34:45.069825+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:34:45.069825+0000","last_undegraded":"2023-01-15T14:34:45.069825+0000","last_fullsized":"2023-01-15T14:34:45.069825+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:51.919","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.97439868299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.5e","version":"215'39","reported_seq":1081,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.164850+0000","last_change":"2023-01-15T14:34:43.164850+0000","last_active":"2023-01-15T14:34:43.164850+0000","last_peered":"2023-01-15T14:34:43.164850+0000","last_clean":"2023-01-15T14:34:43.164850+0000","last_became_active":"2023-01-15T14:34:43.164436+0000","last_became_peered":"2023-01-15T14:34:43.164436+0000","last_unstale":"2023-01-15T14:34:43.164850+0000","last_undegraded":"2023-01-15T14:34:43.164850+0000","last_fullsized":"2023-01-15T14:34:43.164850+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.71418314000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1402,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.053810+0000","last_change":"2023-01-15T14:34:38.968387+0000","last_active":"2023-01-15T14:34:43.053810+0000","last_peered":"2023-01-15T14:34:43.053810+0000","last_clean":"2023-01-15T14:34:43.053810+0000","last_became_active":"2023-01-15T14:34:00.242122+0000","last_became_peered":"2023-01-15T14:34:00.242122+0000","last_unstale":"2023-01-15T14:34:43.053810+0000","last_undegraded":"2023-01-15T14:34:43.053810+0000","last_fullsized":"2023-01-15T14:34:43.053810+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:19.729021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:19.729021+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:29.740","scrub_duration":0.50499948100000003,"objects_trimmed":0,"snaptrim_duration":0.0023203350000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.5c","version":"141'46","reported_seq":163,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.037476+0000","last_change":"2023-01-15T14:34:38.974958+0000","last_active":"2023-01-15T14:34:42.037476+0000","last_peered":"2023-01-15T14:34:42.037476+0000","last_clean":"2023-01-15T14:34:42.037476+0000","last_became_active":"2023-01-15T14:33:57.113032+0000","last_became_peered":"2023-01-15T14:33:57.113032+0000","last_unstale":"2023-01-15T14:34:42.037476+0000","last_undegraded":"2023-01-15T14:34:42.037476+0000","last_fullsized":"2023-01-15T14:34:42.037476+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.0093463320000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.5b","version":"283'80","reported_seq":991,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:42.045705+0000","last_change":"2023-01-15T14:34:40.423552+0000","last_active":"2023-01-15T14:34:42.045705+0000","last_peered":"2023-01-15T14:34:42.045705+0000","last_clean":"2023-01-15T14:34:42.045705+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:34:42.045705+0000","last_undegraded":"2023-01-15T14:34:42.045705+0000","last_fullsized":"2023-01-15T14:34:42.045705+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:42.155","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.45357660399999999,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.5a","version":"384'356","reported_seq":1599,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.379157+0000","last_change":"2023-01-15T14:34:45.379157+0000","last_active":"2023-01-15T14:34:45.379157+0000","last_peered":"2023-01-15T14:34:45.379157+0000","last_clean":"2023-01-15T14:34:45.379157+0000","last_became_active":"2023-01-15T14:34:44.469148+0000","last_became_peered":"2023-01-15T14:34:44.469148+0000","last_unstale":"2023-01-15T14:34:45.379157+0000","last_undegraded":"2023-01-15T14:34:45.379157+0000","last_fullsized":"2023-01-15T14:34:45.379157+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:03.828","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.30549605499999999,"stat_sum":{"num_bytes":5983424,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":19294,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.59","version":"294'256","reported_seq":1371,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.052201+0000","last_change":"2023-01-15T14:34:38.967978+0000","last_active":"2023-01-15T14:34:43.052201+0000","last_peered":"2023-01-15T14:34:43.052201+0000","last_clean":"2023-01-15T14:34:43.052201+0000","last_became_active":"2023-01-15T14:33:55.671990+0000","last_became_peered":"2023-01-15T14:33:55.671990+0000","last_unstale":"2023-01-15T14:34:43.052201+0000","last_undegraded":"2023-01-15T14:34:43.052201+0000","last_fullsized":"2023-01-15T14:34:43.052201+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:34:30.207952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:30.207952+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:47.670","scrub_duration":0.024191634,"objects_trimmed":0,"snaptrim_duration":0.0034370849999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.58","version":"323'320","reported_seq":1581,"reported_epoch":393,"state":"remapped+peering","last_fresh":"2023-01-15T14:34:43.063936+0000","last_change":"2023-01-15T14:34:43.064234+0000","last_active":"2023-01-15T14:34:41.944099+0000","last_peered":"2023-01-15T14:34:41.007716+0000","last_clean":"2023-01-15T14:34:41.007716+0000","last_became_active":"2023-01-15T14:34:40.296819+0000","last_became_peered":"2023-01-15T14:34:40.296819+0000","last_unstale":"2023-01-15T14:34:43.063936+0000","last_undegraded":"2023-01-15T14:34:43.063936+0000","last_fullsized":"2023-01-15T14:34:43.063936+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.0035816229999999999,"stat_sum":{"num_bytes":4332079,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":27633,"num_write":25,"num_write_kb":8163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":5516518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.57","version":"390'110","reported_seq":206,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.069527+0000","last_change":"2023-01-15T14:34:41.578122+0000","last_active":"2023-01-15T14:34:45.069527+0000","last_peered":"2023-01-15T14:34:45.069527+0000","last_clean":"2023-01-15T14:34:45.069527+0000","last_became_active":"2023-01-15T14:33:36.994447+0000","last_became_peered":"2023-01-15T14:33:36.994447+0000","last_unstale":"2023-01-15T14:34:45.069527+0000","last_undegraded":"2023-01-15T14:34:45.069527+0000","last_fullsized":"2023-01-15T14:34:45.069527+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'100","last_scrub_stamp":"2023-01-15T14:34:23.447852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.447852+0000","objects_scrubbed":2,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:30.799","scrub_duration":0.52101476099999999,"objects_trimmed":0,"snaptrim_duration":0.095173800000000003,"stat_sum":{"num_bytes":5882615,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.56","version":"320'127","reported_seq":710,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:42.047355+0000","last_change":"2023-01-15T14:34:40.500232+0000","last_active":"2023-01-15T14:34:42.047355+0000","last_peered":"2023-01-15T14:34:42.047355+0000","last_clean":"2023-01-15T14:34:42.047355+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:34:42.047355+0000","last_undegraded":"2023-01-15T14:34:42.047355+0000","last_fullsized":"2023-01-15T14:34:42.047355+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.38385509499999998,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.55","version":"110'55","reported_seq":1093,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.037212+0000","last_change":"2023-01-15T14:34:38.975399+0000","last_active":"2023-01-15T14:34:42.037212+0000","last_peered":"2023-01-15T14:34:42.037212+0000","last_clean":"2023-01-15T14:34:42.037212+0000","last_became_active":"2023-01-15T14:33:52.510429+0000","last_became_peered":"2023-01-15T14:33:52.510429+0000","last_unstale":"2023-01-15T14:34:42.037212+0000","last_undegraded":"2023-01-15T14:34:42.037212+0000","last_fullsized":"2023-01-15T14:34:42.037212+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.0094724019999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1216,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.167779+0000","last_change":"2023-01-15T14:34:39.168206+0000","last_active":"2023-01-15T14:34:45.167779+0000","last_peered":"2023-01-15T14:34:45.167779+0000","last_clean":"2023-01-15T14:34:45.167779+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:34:45.167779+0000","last_undegraded":"2023-01-15T14:34:45.167779+0000","last_fullsized":"2023-01-15T14:34:45.167779+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:33:45.583005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:45.583005+0000","objects_scrubbed":1,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103664637,"objects_trimmed":0,"snaptrim_duration":0.204543998,"stat_sum":{"num_bytes":1226928,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10793,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.53","version":"389'425","reported_seq":1585,"reported_epoch":393,"state":"peering","last_fresh":"2023-01-15T14:34:42.962577+0000","last_change":"2023-01-15T14:34:42.962577+0000","last_active":"2023-01-15T14:34:42.042819+0000","last_peered":"2023-01-15T14:34:40.977304+0000","last_clean":"2023-01-15T14:34:35.379440+0000","last_became_active":"2023-01-15T14:34:38.442628+0000","last_became_peered":"2023-01-15T14:34:38.442628+0000","last_unstale":"2023-01-15T14:34:42.962577+0000","last_undegraded":"2023-01-15T14:34:42.962577+0000","last_fullsized":"2023-01-15T14:34:42.962577+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.176740491,"stat_sum":{"num_bytes":31795878,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":50,"num_read_kb":57312,"num_write":30,"num_write_kb":16427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":30103975,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"120","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":831,"reported_epoch":393,"state":"remapped+peering","last_fresh":"2023-01-15T14:34:42.962013+0000","last_change":"2023-01-15T14:34:42.962276+0000","last_active":"2023-01-15T14:34:42.042784+0000","last_peered":"2023-01-15T14:34:41.077907+0000","last_clean":"2023-01-15T14:34:41.077907+0000","last_became_active":"2023-01-15T14:34:40.296355+0000","last_became_peered":"2023-01-15T14:34:40.296355+0000","last_unstale":"2023-01-15T14:34:42.962013+0000","last_undegraded":"2023-01-15T14:34:42.962013+0000","last_fullsized":"2023-01-15T14:34:42.962013+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:28.012","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.17764701599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.51","version":"384'326","reported_seq":1607,"reported_epoch":390,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:40.069391+0000","last_change":"2023-01-15T14:34:38.162052+0000","last_active":"2023-01-15T14:34:40.069391+0000","last_peered":"2023-01-15T14:34:40.069391+0000","last_clean":"2023-01-15T14:34:34.053960+0000","last_became_active":"2023-01-15T14:34:38.161164+0000","last_became_peered":"2023-01-15T14:34:38.161164+0000","last_unstale":"2023-01-15T14:34:40.069391+0000","last_undegraded":"2023-01-15T14:34:37.945903+0000","last_fullsized":"2023-01-15T14:34:37.936471+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.0023765840000000002,"stat_sum":{"num_bytes":7496813,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.50","version":"392'444","reported_seq":1684,"reported_epoch":392,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:42.963199+0000","last_change":"2023-01-15T14:34:42.858448+0000","last_active":"2023-01-15T14:34:42.963199+0000","last_peered":"2023-01-15T14:34:42.963199+0000","last_clean":"2023-01-15T14:34:42.963199+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:34:42.963199+0000","last_undegraded":"2023-01-15T14:34:42.963199+0000","last_fullsized":"2023-01-15T14:34:42.963199+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:37.144","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.49987923699999998,"stat_sum":{"num_bytes":15220933,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":56,"num_read_kb":52507,"num_write":59,"num_write_kb":21652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.4f","version":"155'145","reported_seq":933,"reported_epoch":393,"state":"remapped+peering","last_fresh":"2023-01-15T14:34:43.056752+0000","last_change":"2023-01-15T14:34:43.057149+0000","last_active":"2023-01-15T14:34:41.947275+0000","last_peered":"2023-01-15T14:34:41.182498+0000","last_clean":"2023-01-15T14:34:41.182498+0000","last_became_active":"2023-01-15T14:34:40.074049+0000","last_became_peered":"2023-01-15T14:34:40.074049+0000","last_unstale":"2023-01-15T14:34:43.056752+0000","last_undegraded":"2023-01-15T14:34:43.056752+0000","last_fullsized":"2023-01-15T14:34:43.056752+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.82554090199999997,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":184,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.068084+0000","last_change":"2023-01-15T14:34:39.270663+0000","last_active":"2023-01-15T14:34:45.068084+0000","last_peered":"2023-01-15T14:34:45.068084+0000","last_clean":"2023-01-15T14:34:45.068084+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:34:45.068084+0000","last_undegraded":"2023-01-15T14:34:45.068084+0000","last_fullsized":"2023-01-15T14:34:45.068084+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.303656386,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.4d","version":"392'288","reported_seq":1466,"reported_epoch":392,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:43.060637+0000","last_change":"2023-01-15T14:34:42.317174+0000","last_active":"2023-01-15T14:34:43.060637+0000","last_peered":"2023-01-15T14:34:43.060637+0000","last_clean":"2023-01-15T14:34:43.060637+0000","last_became_active":"2023-01-15T14:34:05.178613+0000","last_became_peered":"2023-01-15T14:34:05.178613+0000","last_unstale":"2023-01-15T14:34:43.060637+0000","last_undegraded":"2023-01-15T14:34:43.060637+0000","last_fullsized":"2023-01-15T14:34:43.060637+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.41470547499999999,"stat_sum":{"num_bytes":14846966,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":15961,"num_write":20,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.4c","version":"309'71","reported_seq":190,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:41.942009+0000","last_change":"2023-01-15T14:34:40.710078+0000","last_active":"2023-01-15T14:34:41.942009+0000","last_peered":"2023-01-15T14:34:41.942009+0000","last_clean":"2023-01-15T14:34:41.942009+0000","last_became_active":"2023-01-15T14:33:52.311859+0000","last_became_peered":"2023-01-15T14:33:52.311859+0000","last_unstale":"2023-01-15T14:34:41.942009+0000","last_undegraded":"2023-01-15T14:34:41.942009+0000","last_fullsized":"2023-01-15T14:34:41.942009+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:27.027","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.74045655399999999,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":2,"reported_epoch":392,"state":"peering","last_fresh":"2023-01-15T14:33:52.103417+0000","last_change":"2023-01-15T14:33:51.140761+0000","last_active":"2023-01-15T14:33:51.140537+0000","last_peered":"2023-01-15T14:33:50.015575+0000","last_clean":"2023-01-15T14:33:50.015575+0000","last_became_active":"2023-01-15T14:32:25.161408+0000","last_became_peered":"2023-01-15T14:32:25.161408+0000","last_unstale":"2023-01-15T14:33:52.103417+0000","last_undegraded":"2023-01-15T14:33:52.103417+0000","last_fullsized":"2023-01-15T14:33:52.103417+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70372832699999999,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.4a","version":"390'391","reported_seq":1578,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.143633+0000","last_change":"2023-01-15T14:34:41.393734+0000","last_active":"2023-01-15T14:34:42.143633+0000","last_peered":"2023-01-15T14:34:42.143633+0000","last_clean":"2023-01-15T14:34:42.143633+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:34:42.143633+0000","last_undegraded":"2023-01-15T14:34:42.143633+0000","last_fullsized":"2023-01-15T14:34:42.143633+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:34.117","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.415856377,"stat_sum":{"num_bytes":12470870,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":63,"num_read_kb":67536,"num_write":75,"num_write_kb":23626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":1015,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:41.943299+0000","last_change":"2023-01-15T14:34:39.877252+0000","last_active":"2023-01-15T14:34:41.943299+0000","last_peered":"2023-01-15T14:34:41.943299+0000","last_clean":"2023-01-15T14:34:41.943299+0000","last_became_active":"2023-01-15T14:34:00.043221+0000","last_became_peered":"2023-01-15T14:34:00.043221+0000","last_unstale":"2023-01-15T14:34:41.943299+0000","last_undegraded":"2023-01-15T14:34:41.943299+0000","last_fullsized":"2023-01-15T14:34:41.943299+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.82149750899999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.48","version":"179'204","reported_seq":1442,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.035848+0000","last_change":"2023-01-15T14:34:38.974466+0000","last_active":"2023-01-15T14:34:42.035848+0000","last_peered":"2023-01-15T14:34:42.035848+0000","last_clean":"2023-01-15T14:34:42.035848+0000","last_became_active":"2023-01-15T14:33:55.375518+0000","last_became_peered":"2023-01-15T14:33:55.375518+0000","last_unstale":"2023-01-15T14:34:42.035848+0000","last_undegraded":"2023-01-15T14:34:42.035848+0000","last_fullsized":"2023-01-15T14:34:42.035848+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:43.662","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.0088956069999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1216,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.955158+0000","last_change":"2023-01-15T14:34:39.271908+0000","last_active":"2023-01-15T14:34:42.955158+0000","last_peered":"2023-01-15T14:34:42.955158+0000","last_clean":"2023-01-15T14:34:42.955158+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:34:42.955158+0000","last_undegraded":"2023-01-15T14:34:42.955158+0000","last_fullsized":"2023-01-15T14:34:42.955158+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:34:21.450320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:21.450320+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:50.200","scrub_duration":0.30497759099999999,"objects_trimmed":0,"snaptrim_duration":0.20552836599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":2,"reported_epoch":392,"state":"peering","last_fresh":"2023-01-15T14:34:36.937520+0000","last_change":"2023-01-15T14:34:36.147207+0000","last_active":"2023-01-15T14:34:36.146349+0000","last_peered":"2023-01-15T14:29:48.848534+0000","last_clean":"2023-01-15T14:29:48.848534+0000","last_became_active":"2023-01-15T14:28:23.841189+0000","last_became_peered":"2023-01-15T14:28:23.841189+0000","last_unstale":"2023-01-15T14:34:36.937520+0000","last_undegraded":"2023-01-15T14:34:36.937520+0000","last_fullsized":"2023-01-15T14:34:36.937520+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2142,"num_write":3,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":1001,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.068264+0000","last_change":"2023-01-15T14:34:41.493064+0000","last_active":"2023-01-15T14:34:45.068264+0000","last_peered":"2023-01-15T14:34:45.068264+0000","last_clean":"2023-01-15T14:34:45.068264+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:34:45.068264+0000","last_undegraded":"2023-01-15T14:34:45.068264+0000","last_fullsized":"2023-01-15T14:34:45.068264+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:32:32.384694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:32.384694+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30865616200000001,"stat_sum":{"num_bytes":2635976,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":19927,"num_write":15,"num_write_kb":6659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4745860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1357,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.953827+0000","last_change":"2023-01-15T14:34:38.966252+0000","last_active":"2023-01-15T14:34:42.953827+0000","last_peered":"2023-01-15T14:34:42.953827+0000","last_clean":"2023-01-15T14:34:42.953827+0000","last_became_active":"2023-01-15T14:33:55.678019+0000","last_became_peered":"2023-01-15T14:33:55.678019+0000","last_unstale":"2023-01-15T14:34:42.953827+0000","last_undegraded":"2023-01-15T14:34:42.953827+0000","last_fullsized":"2023-01-15T14:34:42.953827+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.001123892,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.43","version":"393'333","reported_seq":1621,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.168041+0000","last_change":"2023-01-15T14:34:41.490750+0000","last_active":"2023-01-15T14:34:45.168041+0000","last_peered":"2023-01-15T14:34:45.168041+0000","last_clean":"2023-01-15T14:34:45.168041+0000","last_became_active":"2023-01-15T14:34:02.211232+0000","last_became_peered":"2023-01-15T14:34:02.211232+0000","last_unstale":"2023-01-15T14:34:45.168041+0000","last_undegraded":"2023-01-15T14:34:45.168041+0000","last_fullsized":"2023-01-15T14:34:45.168041+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.308597554,"stat_sum":{"num_bytes":9109625,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":13,"num_read_kb":16586,"num_write":12,"num_write_kb":5193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3395080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.42","version":"155'101","reported_seq":1118,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.236402+0000","last_change":"2023-01-15T14:34:40.376566+0000","last_active":"2023-01-15T14:34:42.236402+0000","last_peered":"2023-01-15T14:34:42.236402+0000","last_clean":"2023-01-15T14:34:42.236402+0000","last_became_active":"2023-01-15T14:33:55.376782+0000","last_became_peered":"2023-01-15T14:33:55.376782+0000","last_unstale":"2023-01-15T14:34:42.236402+0000","last_undegraded":"2023-01-15T14:34:42.236402+0000","last_fullsized":"2023-01-15T14:34:42.236402+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.40965996599999999,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1227,"reported_epoch":394,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:34:44.377099+0000","last_change":"2023-01-15T14:34:44.376782+0000","last_active":"2023-01-15T14:34:44.377099+0000","last_peered":"2023-01-15T14:34:44.377099+0000","last_clean":"2023-01-15T14:34:41.080239+0000","last_became_active":"2023-01-15T14:34:44.376782+0000","last_became_peered":"2023-01-15T14:34:44.376782+0000","last_unstale":"2023-01-15T14:34:44.377099+0000","last_undegraded":"2023-01-15T14:34:44.377099+0000","last_fullsized":"2023-01-15T14:34:43.965033+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.30697919099999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0],"avail_no_missing":["4","0","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.40","version":"392'322","reported_seq":1544,"reported_epoch":393,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:43.165422+0000","last_change":"2023-01-15T14:34:39.876385+0000","last_active":"2023-01-15T14:34:43.165422+0000","last_peered":"2023-01-15T14:34:43.165422+0000","last_clean":"2023-01-15T14:34:43.165422+0000","last_became_active":"2023-01-15T14:33:58.117085+0000","last_became_peered":"2023-01-15T14:33:58.117085+0000","last_unstale":"2023-01-15T14:34:43.165422+0000","last_undegraded":"2023-01-15T14:34:43.165422+0000","last_fullsized":"2023-01-15T14:34:43.165422+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.80041662199999997,"stat_sum":{"num_bytes":14103500,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":35542,"num_write":33,"num_write_kb":12321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":29167491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.3f","version":"390'241","reported_seq":1459,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:41.943937+0000","last_change":"2023-01-15T14:34:40.312627+0000","last_active":"2023-01-15T14:34:41.943937+0000","last_peered":"2023-01-15T14:34:41.943937+0000","last_clean":"2023-01-15T14:34:41.943937+0000","last_became_active":"2023-01-15T14:33:58.217783+0000","last_became_peered":"2023-01-15T14:33:58.217783+0000","last_unstale":"2023-01-15T14:34:41.943937+0000","last_undegraded":"2023-01-15T14:34:41.943937+0000","last_fullsized":"2023-01-15T14:34:41.943937+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.34186497999999998,"stat_sum":{"num_bytes":7672707,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":2,"num_read_kb":2,"num_write":11,"num_write_kb":2540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6605283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.3e","version":"381'64","reported_seq":2,"reported_epoch":392,"state":"peering","last_fresh":"2023-01-15T14:34:41.681685+0000","last_change":"2023-01-15T14:34:42.952031+0000","last_active":"2023-01-15T14:34:42.952031+0000","last_peered":"2023-01-15T14:34:41.681685+0000","last_clean":"2023-01-15T14:34:41.681685+0000","last_became_active":"2023-01-15T14:34:37.247238+0000","last_became_peered":"2023-01-15T14:34:37.247238+0000","last_unstale":"2023-01-15T14:34:41.681685+0000","last_undegraded":"2023-01-15T14:34:41.681685+0000","last_fullsized":"2023-01-15T14:34:41.681685+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.39891121200000002,"stat_sum":{"num_bytes":5664091,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":4255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":5664091,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.3d","version":"394'252","reported_seq":1487,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.378007+0000","last_change":"2023-01-15T14:34:45.378007+0000","last_active":"2023-01-15T14:34:45.378007+0000","last_peered":"2023-01-15T14:34:45.378007+0000","last_clean":"2023-01-15T14:34:45.378007+0000","last_became_active":"2023-01-15T14:34:44.167696+0000","last_became_peered":"2023-01-15T14:34:44.167696+0000","last_unstale":"2023-01-15T14:34:45.378007+0000","last_undegraded":"2023-01-15T14:34:45.378007+0000","last_fullsized":"2023-01-15T14:34:45.378007+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.31006661600000002,"stat_sum":{"num_bytes":8050279,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":27,"num_read_kb":30349,"num_write":18,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":14705983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.3c","version":"141'46","reported_seq":433,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.037813+0000","last_change":"2023-01-15T14:34:39.883787+0000","last_active":"2023-01-15T14:34:42.037813+0000","last_peered":"2023-01-15T14:34:42.037813+0000","last_clean":"2023-01-15T14:34:42.037813+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:34:42.037813+0000","last_undegraded":"2023-01-15T14:34:42.037813+0000","last_fullsized":"2023-01-15T14:34:42.037813+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.91472550200000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":105,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:42.144644+0000","last_change":"2023-01-15T14:34:40.424695+0000","last_active":"2023-01-15T14:34:42.144644+0000","last_peered":"2023-01-15T14:34:42.144644+0000","last_clean":"2023-01-15T14:34:42.144644+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:34:42.144644+0000","last_undegraded":"2023-01-15T14:34:42.144644+0000","last_fullsized":"2023-01-15T14:34:42.144644+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.35517283500000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1474,"reported_epoch":396,"state":"active+clean","last_fresh":"2023-01-15T14:34:46.573867+0000","last_change":"2023-01-15T14:34:46.573867+0000","last_active":"2023-01-15T14:34:46.573867+0000","last_peered":"2023-01-15T14:34:46.573867+0000","last_clean":"2023-01-15T14:34:46.573867+0000","last_became_active":"2023-01-15T14:34:46.572815+0000","last_became_peered":"2023-01-15T14:34:46.572815+0000","last_unstale":"2023-01-15T14:34:46.573867+0000","last_undegraded":"2023-01-15T14:34:46.573867+0000","last_fullsized":"2023-01-15T14:34:46.573867+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.502316608,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.39","version":"391'391","reported_seq":1612,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.858199+0000","last_change":"2023-01-15T14:34:42.858199+0000","last_active":"2023-01-15T14:34:42.858199+0000","last_peered":"2023-01-15T14:34:42.858199+0000","last_clean":"2023-01-15T14:34:42.858199+0000","last_became_active":"2023-01-15T14:34:08.023332+0000","last_became_peered":"2023-01-15T14:34:08.023332+0000","last_unstale":"2023-01-15T14:34:42.858199+0000","last_undegraded":"2023-01-15T14:34:42.858199+0000","last_fullsized":"2023-01-15T14:34:42.858199+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.91468942799999997,"stat_sum":{"num_bytes":26812050,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":56,"num_read_kb":91839,"num_write":15,"num_write_kb":10017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":26862613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.38","version":"389'327","reported_seq":1628,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.038081+0000","last_change":"2023-01-15T14:34:39.775455+0000","last_active":"2023-01-15T14:34:42.038081+0000","last_peered":"2023-01-15T14:34:42.038081+0000","last_clean":"2023-01-15T14:34:42.038081+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:34:42.038081+0000","last_undegraded":"2023-01-15T14:34:42.038081+0000","last_fullsized":"2023-01-15T14:34:42.038081+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:32.007","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.80938584199999997,"stat_sum":{"num_bytes":8461433,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":45,"num_read_kb":42848,"num_write":40,"num_write_kb":13455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":513,"reported_epoch":393,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:43.157690+0000","last_change":"2023-01-15T14:34:43.157690+0000","last_active":"2023-01-15T14:34:43.157690+0000","last_peered":"2023-01-15T14:34:43.157690+0000","last_clean":"2023-01-15T14:34:43.157690+0000","last_became_active":"2023-01-15T14:34:43.156532+0000","last_became_peered":"2023-01-15T14:34:43.156532+0000","last_unstale":"2023-01-15T14:34:43.157690+0000","last_undegraded":"2023-01-15T14:34:43.157690+0000","last_fullsized":"2023-01-15T14:34:43.157690+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.538903523,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.36","version":"194'81","reported_seq":2,"reported_epoch":392,"state":"peering","last_fresh":"2023-01-15T14:34:37.047017+0000","last_change":"2023-01-15T14:34:36.247762+0000","last_active":"2023-01-15T14:31:35.853937+0000","last_peered":"2023-01-15T14:31:35.728549+0000","last_clean":"2023-01-15T14:31:35.728549+0000","last_became_active":"2023-01-15T14:31:35.228767+0000","last_became_peered":"2023-01-15T14:31:35.228767+0000","last_unstale":"2023-01-15T14:34:37.047017+0000","last_undegraded":"2023-01-15T14:34:37.047017+0000","last_fullsized":"2023-01-15T14:34:37.047017+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":6,"last_scrub":"180'79","last_scrub_stamp":"2023-01-15T14:31:11.311298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:11.311298+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205307658,"objects_trimmed":0,"snaptrim_duration":0.49890758600000001,"stat_sum":{"num_bytes":1117682,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":123,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.168217+0000","last_change":"2023-01-15T14:34:41.491993+0000","last_active":"2023-01-15T14:34:45.168217+0000","last_peered":"2023-01-15T14:34:45.168217+0000","last_clean":"2023-01-15T14:34:45.168217+0000","last_became_active":"2023-01-15T14:33:52.612576+0000","last_became_peered":"2023-01-15T14:33:52.612576+0000","last_unstale":"2023-01-15T14:34:45.168217+0000","last_undegraded":"2023-01-15T14:34:45.168217+0000","last_fullsized":"2023-01-15T14:34:45.168217+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:27.799","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30788173200000002,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.34","version":"169'138","reported_seq":1242,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.054501+0000","last_change":"2023-01-15T14:34:38.966977+0000","last_active":"2023-01-15T14:34:43.054501+0000","last_peered":"2023-01-15T14:34:43.054501+0000","last_clean":"2023-01-15T14:34:43.054501+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:34:43.054501+0000","last_undegraded":"2023-01-15T14:34:43.054501+0000","last_fullsized":"2023-01-15T14:34:43.054501+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.00166909,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1339,"reported_epoch":393,"state":"remapped+peering","last_fresh":"2023-01-15T14:34:43.062213+0000","last_change":"2023-01-15T14:34:43.063034+0000","last_active":"2023-01-15T14:34:42.038738+0000","last_peered":"2023-01-15T14:34:41.081621+0000","last_clean":"2023-01-15T14:34:41.081621+0000","last_became_active":"2023-01-15T14:34:40.189416+0000","last_became_peered":"2023-01-15T14:34:40.189416+0000","last_unstale":"2023-01-15T14:34:43.062213+0000","last_undegraded":"2023-01-15T14:34:43.062213+0000","last_fullsized":"2023-01-15T14:34:43.062213+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:26.972","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.712528469,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.32","version":"394'169","reported_seq":957,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.381099+0000","last_change":"2023-01-15T14:34:45.381099+0000","last_active":"2023-01-15T14:34:45.381099+0000","last_peered":"2023-01-15T14:34:45.381099+0000","last_clean":"2023-01-15T14:34:45.381099+0000","last_became_active":"2023-01-15T14:34:44.468157+0000","last_became_peered":"2023-01-15T14:34:44.468157+0000","last_unstale":"2023-01-15T14:34:45.381099+0000","last_undegraded":"2023-01-15T14:34:45.381099+0000","last_fullsized":"2023-01-15T14:34:45.381099+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.30768722799999998,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7048,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5327885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.13","version":"390'380","reported_seq":1485,"reported_epoch":393,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:42.962711+0000","last_change":"2023-01-15T14:34:41.403887+0000","last_active":"2023-01-15T14:34:42.962711+0000","last_peered":"2023-01-15T14:34:42.962711+0000","last_clean":"2023-01-15T14:34:42.962711+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:34:42.962711+0000","last_undegraded":"2023-01-15T14:34:42.962711+0000","last_fullsized":"2023-01-15T14:34:42.962711+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"374'376","last_scrub_stamp":"2023-01-15T14:34:32.489483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.489483+0000","objects_scrubbed":4,"log_size":380,"log_dups_size":0,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:32.530","scrub_duration":0.381937625,"objects_trimmed":0,"snaptrim_duration":0.42547845099999998,"stat_sum":{"num_bytes":9889842,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":2561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.12","version":"389'201","reported_seq":1003,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.073036+0000","last_change":"2023-01-15T14:34:39.916577+0000","last_active":"2023-01-15T14:34:45.073036+0000","last_peered":"2023-01-15T14:34:45.073036+0000","last_clean":"2023-01-15T14:34:45.073036+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:34:45.073036+0000","last_undegraded":"2023-01-15T14:34:45.073036+0000","last_fullsized":"2023-01-15T14:34:45.073036+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.94582292999999995,"stat_sum":{"num_bytes":10304297,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":20,"num_write_kb":9403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.11","version":"390'323","reported_seq":1665,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:41.944457+0000","last_change":"2023-01-15T14:34:40.422641+0000","last_active":"2023-01-15T14:34:41.944457+0000","last_peered":"2023-01-15T14:34:41.944457+0000","last_clean":"2023-01-15T14:34:41.944457+0000","last_became_active":"2023-01-15T14:34:09.541429+0000","last_became_peered":"2023-01-15T14:34:09.541429+0000","last_unstale":"2023-01-15T14:34:41.944457+0000","last_undegraded":"2023-01-15T14:34:41.944457+0000","last_fullsized":"2023-01-15T14:34:41.944457+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":323,"log_dups_size":0,"ondisk_log_size":323,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:21.075","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.40706913900000002,"stat_sum":{"num_bytes":15559313,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":45,"num_read_kb":48289,"num_write":29,"num_write_kb":17520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":20307357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.10","version":"389'412","reported_seq":1496,"reported_epoch":396,"state":"active+clean","last_fresh":"2023-01-15T14:34:46.076948+0000","last_change":"2023-01-15T14:34:40.874543+0000","last_active":"2023-01-15T14:34:46.076948+0000","last_peered":"2023-01-15T14:34:46.076948+0000","last_clean":"2023-01-15T14:34:46.076948+0000","last_became_active":"2023-01-15T14:34:16.272883+0000","last_became_peered":"2023-01-15T14:34:16.272883+0000","last_unstale":"2023-01-15T14:34:46.076948+0000","last_undegraded":"2023-01-15T14:34:46.076948+0000","last_fullsized":"2023-01-15T14:34:46.076948+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.80543094999999998,"stat_sum":{"num_bytes":8658713,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":54,"num_read_kb":52673,"num_write":50,"num_write_kb":18045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12804882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.f","version":"155'145","reported_seq":1319,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.316069+0000","last_change":"2023-01-15T14:34:42.316069+0000","last_active":"2023-01-15T14:34:42.316069+0000","last_peered":"2023-01-15T14:34:42.316069+0000","last_clean":"2023-01-15T14:34:42.316069+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:34:42.316069+0000","last_undegraded":"2023-01-15T14:34:42.316069+0000","last_fullsized":"2023-01-15T14:34:42.316069+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.37268368800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.e","version":"139'38","reported_seq":127,"reported_epoch":393,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:43.266934+0000","last_change":"2023-01-15T14:34:43.266934+0000","last_active":"2023-01-15T14:34:43.266934+0000","last_peered":"2023-01-15T14:34:43.266934+0000","last_clean":"2023-01-15T14:34:43.266934+0000","last_became_active":"2023-01-15T14:34:43.265641+0000","last_became_peered":"2023-01-15T14:34:43.265641+0000","last_unstale":"2023-01-15T14:34:43.266934+0000","last_undegraded":"2023-01-15T14:34:43.266934+0000","last_fullsized":"2023-01-15T14:34:43.266934+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.59169766000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.d","version":"296'232","reported_seq":1379,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:42.049191+0000","last_change":"2023-01-15T14:34:39.878172+0000","last_active":"2023-01-15T14:34:42.049191+0000","last_peered":"2023-01-15T14:34:42.049191+0000","last_clean":"2023-01-15T14:34:42.049191+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:34:42.049191+0000","last_undegraded":"2023-01-15T14:34:42.049191+0000","last_fullsized":"2023-01-15T14:34:42.049191+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:22.089","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.81988239399999996,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.c","version":"390'108","reported_seq":1230,"reported_epoch":393,"state":"remapped+peering","last_fresh":"2023-01-15T14:34:43.057177+0000","last_change":"2023-01-15T14:34:43.057598+0000","last_active":"2023-01-15T14:34:42.047348+0000","last_peered":"2023-01-15T14:34:41.283332+0000","last_clean":"2023-01-15T14:34:41.283332+0000","last_became_active":"2023-01-15T14:34:37.443618+0000","last_became_peered":"2023-01-15T14:34:37.443618+0000","last_unstale":"2023-01-15T14:34:43.057177+0000","last_undegraded":"2023-01-15T14:34:43.057177+0000","last_fullsized":"2023-01-15T14:34:43.057177+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.100921735,"stat_sum":{"num_bytes":3264587,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":41975,"num_write":11,"num_write_kb":1706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":16047348,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":1087,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.241176+0000","last_change":"2023-01-15T14:34:40.377626+0000","last_active":"2023-01-15T14:34:42.241176+0000","last_peered":"2023-01-15T14:34:42.241176+0000","last_clean":"2023-01-15T14:34:42.241176+0000","last_became_active":"2023-01-15T14:33:52.425622+0000","last_became_peered":"2023-01-15T14:33:52.425622+0000","last_unstale":"2023-01-15T14:34:42.241176+0000","last_undegraded":"2023-01-15T14:34:42.241176+0000","last_fullsized":"2023-01-15T14:34:42.241176+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.29399699299999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1440,"reported_epoch":394,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:34:44.274734+0000","last_change":"2023-01-15T14:34:44.274569+0000","last_active":"2023-01-15T14:34:44.274734+0000","last_peered":"2023-01-15T14:34:44.274734+0000","last_clean":"2023-01-15T14:34:40.980630+0000","last_became_active":"2023-01-15T14:34:44.274569+0000","last_became_peered":"2023-01-15T14:34:44.274569+0000","last_unstale":"2023-01-15T14:34:44.274734+0000","last_undegraded":"2023-01-15T14:34:44.274734+0000","last_fullsized":"2023-01-15T14:34:43.966929+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.40388617799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,3],"avail_no_missing":["4","1","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1196,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.056006+0000","last_change":"2023-01-15T14:34:39.268770+0000","last_active":"2023-01-15T14:34:43.056006+0000","last_peered":"2023-01-15T14:34:43.056006+0000","last_clean":"2023-01-15T14:34:43.056006+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:34:43.056006+0000","last_undegraded":"2023-01-15T14:34:43.056006+0000","last_fullsized":"2023-01-15T14:34:43.056006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.30102435900000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1438,"reported_epoch":393,"state":"remapped+peering","last_fresh":"2023-01-15T14:34:43.054081+0000","last_change":"2023-01-15T14:34:43.059514+0000","last_active":"2023-01-15T14:34:41.942795+0000","last_peered":"2023-01-15T14:34:41.081628+0000","last_clean":"2023-01-15T14:34:41.081628+0000","last_became_active":"2023-01-15T14:34:40.473822+0000","last_became_peered":"2023-01-15T14:34:40.473822+0000","last_unstale":"2023-01-15T14:34:43.054081+0000","last_undegraded":"2023-01-15T14:34:43.054081+0000","last_fullsized":"2023-01-15T14:34:43.054081+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:32:07.606363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.606363+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.43004361000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,7],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"1.0","version":"19'117","reported_seq":558,"reported_epoch":393,"state":"peering","last_fresh":"2023-01-15T14:34:43.063991+0000","last_change":"2023-01-15T14:34:43.064086+0000","last_active":"2023-01-15T14:34:41.942340+0000","last_peered":"2023-01-15T14:34:41.076954+0000","last_clean":"2023-01-15T14:34:35.579022+0000","last_became_active":"2023-01-15T14:34:38.334850+0000","last_became_peered":"2023-01-15T14:34:38.334850+0000","last_unstale":"2023-01-15T14:34:43.063991+0000","last_undegraded":"2023-01-15T14:34:43.063991+0000","last_fullsized":"2023-01-15T14:34:43.063991+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,2],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":56,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.044305+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:34:42.044305+0000","last_peered":"2023-01-15T14:34:42.044305+0000","last_clean":"2023-01-15T14:34:42.044305+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:34:42.044305+0000","last_undegraded":"2023-01-15T14:34:42.044305+0000","last_fullsized":"2023-01-15T14:34:42.044305+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":1113,"reported_epoch":394,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:34:44.273870+0000","last_change":"2023-01-15T14:34:44.273590+0000","last_active":"2023-01-15T14:34:44.273870+0000","last_peered":"2023-01-15T14:34:44.273870+0000","last_clean":"2023-01-15T14:34:40.980646+0000","last_became_active":"2023-01-15T14:34:44.273590+0000","last_became_peered":"2023-01-15T14:34:44.273590+0000","last_unstale":"2023-01-15T14:34:44.273870+0000","last_undegraded":"2023-01-15T14:34:44.273870+0000","last_fullsized":"2023-01-15T14:34:43.966165+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.30358357400000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3],"avail_no_missing":["4","1","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":459,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.269231+0000","last_change":"2023-01-15T14:34:43.269231+0000","last_active":"2023-01-15T14:34:43.269231+0000","last_peered":"2023-01-15T14:34:43.269231+0000","last_clean":"2023-01-15T14:34:43.269231+0000","last_became_active":"2023-01-15T14:34:43.268688+0000","last_became_peered":"2023-01-15T14:34:43.268688+0000","last_unstale":"2023-01-15T14:34:43.269231+0000","last_undegraded":"2023-01-15T14:34:43.269231+0000","last_fullsized":"2023-01-15T14:34:43.269231+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":799,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.167133+0000","last_change":"2023-01-15T14:34:40.583784+0000","last_active":"2023-01-15T14:34:45.167133+0000","last_peered":"2023-01-15T14:34:45.167133+0000","last_clean":"2023-01-15T14:34:45.167133+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:34:45.167133+0000","last_undegraded":"2023-01-15T14:34:45.167133+0000","last_fullsized":"2023-01-15T14:34:45.167133+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.51481025700000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":194,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.148543+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:34:42.148543+0000","last_peered":"2023-01-15T14:34:42.148543+0000","last_clean":"2023-01-15T14:34:42.148543+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:34:42.148543+0000","last_undegraded":"2023-01-15T14:34:42.148543+0000","last_fullsized":"2023-01-15T14:34:42.148543+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1,"reported_epoch":392,"state":"unknown","last_fresh":"2023-01-15T14:34:41.944004+0000","last_change":"2023-01-15T14:34:42.951712+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:41.944004+0000","last_undegraded":"2023-01-15T14:34:41.944004+0000","last_fullsized":"2023-01-15T14:34:41.944004+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":2,"reported_epoch":392,"state":"peering","last_fresh":"2023-01-15T14:34:41.181931+0000","last_change":"2023-01-15T14:34:42.951802+0000","last_active":"2023-01-15T14:34:42.951802+0000","last_peered":"2023-01-15T14:34:41.181931+0000","last_clean":"2023-01-15T14:34:41.181931+0000","last_became_active":"2023-01-15T14:34:37.454643+0000","last_became_peered":"2023-01-15T14:34:37.454643+0000","last_unstale":"2023-01-15T14:34:41.181931+0000","last_undegraded":"2023-01-15T14:34:41.181931+0000","last_fullsized":"2023-01-15T14:34:41.181931+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"393'335","reported_seq":1561,"reported_epoch":394,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.967703+0000","last_change":"2023-01-15T14:34:44.069931+0000","last_active":"2023-01-15T14:34:43.967703+0000","last_peered":"2023-01-15T14:34:43.967703+0000","last_clean":"2023-01-15T14:34:43.967703+0000","last_became_active":"2023-01-15T14:34:43.258728+0000","last_became_peered":"2023-01-15T14:34:43.258728+0000","last_unstale":"2023-01-15T14:34:43.967703+0000","last_undegraded":"2023-01-15T14:34:43.967703+0000","last_fullsized":"2023-01-15T14:34:43.967703+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.104548853,"stat_sum":{"num_bytes":10343843,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":13875,"num_write":6,"num_write_kb":7330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":21227736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.5","version":"390'175","reported_seq":1317,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.039545+0000","last_change":"2023-01-15T14:34:40.791733+0000","last_active":"2023-01-15T14:34:42.039545+0000","last_peered":"2023-01-15T14:34:42.039545+0000","last_clean":"2023-01-15T14:34:42.039545+0000","last_became_active":"2023-01-15T14:34:04.175473+0000","last_became_peered":"2023-01-15T14:34:04.175473+0000","last_unstale":"2023-01-15T14:34:42.039545+0000","last_undegraded":"2023-01-15T14:34:42.039545+0000","last_fullsized":"2023-01-15T14:34:42.039545+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.82412401300000004,"stat_sum":{"num_bytes":15163676,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":22,"num_read_kb":26147,"num_write":17,"num_write_kb":6661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":13260921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":2,"reported_epoch":392,"state":"peering","last_fresh":"2023-01-15T14:34:41.283888+0000","last_change":"2023-01-15T14:34:42.951878+0000","last_active":"2023-01-15T14:34:42.951878+0000","last_peered":"2023-01-15T14:34:41.283888+0000","last_clean":"2023-01-15T14:34:41.283888+0000","last_became_active":"2023-01-15T14:34:37.349184+0000","last_became_peered":"2023-01-15T14:34:37.349184+0000","last_unstale":"2023-01-15T14:34:41.283888+0000","last_undegraded":"2023-01-15T14:34:41.283888+0000","last_fullsized":"2023-01-15T14:34:41.283888+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"388'328","reported_seq":1595,"reported_epoch":390,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:34:40.069364+0000","last_change":"2023-01-15T14:34:38.144926+0000","last_active":"2023-01-15T14:34:40.069364+0000","last_peered":"2023-01-15T14:34:40.069364+0000","last_clean":"2023-01-15T14:34:34.356733+0000","last_became_active":"2023-01-15T14:34:38.143496+0000","last_became_peered":"2023-01-15T14:34:38.143496+0000","last_unstale":"2023-01-15T14:34:40.069364+0000","last_undegraded":"2023-01-15T14:34:38.034357+0000","last_fullsized":"2023-01-15T14:34:38.032547+0000","mapping_epoch":387,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:37.513","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0.20118056200000001,"stat_sum":{"num_bytes":8290510,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":441,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.141051+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:34:42.141051+0000","last_peered":"2023-01-15T14:34:42.141051+0000","last_clean":"2023-01-15T14:34:42.141051+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:34:42.141051+0000","last_undegraded":"2023-01-15T14:34:42.141051+0000","last_fullsized":"2023-01-15T14:34:42.141051+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"391'159","reported_seq":1249,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.314152+0000","last_change":"2023-01-15T14:34:42.314152+0000","last_active":"2023-01-15T14:34:42.314152+0000","last_peered":"2023-01-15T14:34:42.314152+0000","last_clean":"2023-01-15T14:34:42.314152+0000","last_became_active":"2023-01-15T14:34:17.186013+0000","last_became_peered":"2023-01-15T14:34:17.186013+0000","last_unstale":"2023-01-15T14:34:42.314152+0000","last_undegraded":"2023-01-15T14:34:42.314152+0000","last_fullsized":"2023-01-15T14:34:42.314152+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":159,"log_dups_size":0,"ondisk_log_size":159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.37090368800000001,"stat_sum":{"num_bytes":12609512,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":30921,"num_write":17,"num_write_kb":8618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3546343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":2,"reported_epoch":392,"state":"peering","last_fresh":"2023-01-15T14:34:36.941127+0000","last_change":"2023-01-15T14:34:36.148756+0000","last_active":"2023-01-15T14:30:12.791129+0000","last_peered":"2023-01-15T14:30:11.853284+0000","last_clean":"2023-01-15T14:30:11.853284+0000","last_became_active":"2023-01-15T14:29:36.099234+0000","last_became_peered":"2023-01-15T14:29:36.099234+0000","last_unstale":"2023-01-15T14:34:36.941127+0000","last_undegraded":"2023-01-15T14:34:36.941127+0000","last_fullsized":"2023-01-15T14:34:36.941127+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":143,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.067738+0000","last_change":"2023-01-15T14:34:43.462659+0000","last_active":"2023-01-15T14:34:45.067738+0000","last_peered":"2023-01-15T14:34:45.067738+0000","last_clean":"2023-01-15T14:34:45.067738+0000","last_became_active":"2023-01-15T14:34:43.461395+0000","last_became_peered":"2023-01-15T14:34:43.461395+0000","last_unstale":"2023-01-15T14:34:45.067738+0000","last_undegraded":"2023-01-15T14:34:45.067738+0000","last_fullsized":"2023-01-15T14:34:45.067738+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.30278768299999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":56,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:41.944033+0000","last_change":"2023-01-15T14:33:57.412557+0000","last_active":"2023-01-15T14:34:41.944033+0000","last_peered":"2023-01-15T14:34:41.944033+0000","last_clean":"2023-01-15T14:34:41.944033+0000","last_became_active":"2023-01-15T14:33:57.311981+0000","last_became_peered":"2023-01-15T14:33:57.311981+0000","last_unstale":"2023-01-15T14:34:41.944033+0000","last_undegraded":"2023-01-15T14:34:41.944033+0000","last_fullsized":"2023-01-15T14:34:41.944033+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"389'167","reported_seq":1286,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.056051+0000","last_change":"2023-01-15T14:34:39.672793+0000","last_active":"2023-01-15T14:34:43.056051+0000","last_peered":"2023-01-15T14:34:43.056051+0000","last_clean":"2023-01-15T14:34:43.056051+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:34:43.056051+0000","last_undegraded":"2023-01-15T14:34:43.056051+0000","last_fullsized":"2023-01-15T14:34:43.056051+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"351'159","last_scrub_stamp":"2023-01-15T14:34:18.607751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:18.607751+0000","objects_scrubbed":2,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:36.520","scrub_duration":0.40666956999999998,"objects_trimmed":0,"snaptrim_duration":0.707003194,"stat_sum":{"num_bytes":2453855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":108,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.315021+0000","last_change":"2023-01-15T14:34:42.315021+0000","last_active":"2023-01-15T14:34:42.315021+0000","last_peered":"2023-01-15T14:34:42.315021+0000","last_clean":"2023-01-15T14:34:42.315021+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:34:42.315021+0000","last_undegraded":"2023-01-15T14:34:42.315021+0000","last_fullsized":"2023-01-15T14:34:42.315021+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.27454862000000002,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.16","version":"384'139","reported_seq":811,"reported_epoch":393,"state":"peering","last_fresh":"2023-01-15T14:34:42.960220+0000","last_change":"2023-01-15T14:34:42.961065+0000","last_active":"2023-01-15T14:34:42.045797+0000","last_peered":"2023-01-15T14:34:41.630330+0000","last_clean":"2023-01-15T14:34:35.482132+0000","last_became_active":"2023-01-15T14:34:38.146029+0000","last_became_peered":"2023-01-15T14:34:38.146029+0000","last_unstale":"2023-01-15T14:34:42.960220+0000","last_undegraded":"2023-01-15T14:34:42.960220+0000","last_fullsized":"2023-01-15T14:34:42.960220+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:45.365","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.0016029320000000001,"stat_sum":{"num_bytes":7378471,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":6898,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9731517,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.17","version":"278'93","reported_seq":143,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.069185+0000","last_change":"2023-01-15T14:34:43.365782+0000","last_active":"2023-01-15T14:34:45.069185+0000","last_peered":"2023-01-15T14:34:45.069185+0000","last_clean":"2023-01-15T14:34:45.069185+0000","last_became_active":"2023-01-15T14:34:43.264755+0000","last_became_peered":"2023-01-15T14:34:43.264755+0000","last_unstale":"2023-01-15T14:34:45.069185+0000","last_undegraded":"2023-01-15T14:34:45.069185+0000","last_fullsized":"2023-01-15T14:34:45.069185+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:29.773","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.97342173300000001,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.18","version":"390'342","reported_seq":1652,"reported_epoch":393,"state":"peering","last_fresh":"2023-01-15T14:34:43.262251+0000","last_change":"2023-01-15T14:34:43.262251+0000","last_active":"2023-01-15T14:34:42.039908+0000","last_peered":"2023-01-15T14:34:41.181489+0000","last_clean":"2023-01-15T14:34:35.477634+0000","last_became_active":"2023-01-15T14:34:38.346256+0000","last_became_peered":"2023-01-15T14:34:38.346256+0000","last_unstale":"2023-01-15T14:34:43.262251+0000","last_undegraded":"2023-01-15T14:34:43.262251+0000","last_fullsized":"2023-01-15T14:34:43.262251+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":5,"last_scrub":"384'341","last_scrub_stamp":"2023-01-15T14:34:35.477211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:35.477211+0000","objects_scrubbed":5,"log_size":342,"log_dups_size":0,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:59.843","scrub_duration":0.54215560600000001,"objects_trimmed":0,"snaptrim_duration":0.20784000499999999,"stat_sum":{"num_bytes":9299174,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.19","version":"372'259","reported_seq":1397,"reported_epoch":392,"state":"peering","last_fresh":"2023-01-15T14:34:41.945207+0000","last_change":"2023-01-15T14:34:41.945019+0000","last_active":"2023-01-15T14:34:41.944749+0000","last_peered":"2023-01-15T14:34:35.490782+0000","last_clean":"2023-01-15T14:34:35.490782+0000","last_became_active":"2023-01-15T14:33:35.259904+0000","last_became_peered":"2023-01-15T14:33:35.259904+0000","last_unstale":"2023-01-15T14:34:41.945207+0000","last_undegraded":"2023-01-15T14:34:41.945207+0000","last_fullsized":"2023-01-15T14:34:41.945207+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":326,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:57.574","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.204275173,"stat_sum":{"num_bytes":4941423,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"103","length":"1"},{"start":"10c","length":"1"},{"start":"113","length":"2"},{"start":"116","length":"1"},{"start":"11a","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.1a","version":"394'358","reported_seq":1570,"reported_epoch":394,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:44.276088+0000","last_change":"2023-01-15T14:34:44.275924+0000","last_active":"2023-01-15T14:34:44.276088+0000","last_peered":"2023-01-15T14:34:44.276088+0000","last_clean":"2023-01-15T14:34:44.276088+0000","last_became_active":"2023-01-15T14:34:44.273541+0000","last_became_peered":"2023-01-15T14:34:44.273541+0000","last_unstale":"2023-01-15T14:34:44.276088+0000","last_undegraded":"2023-01-15T14:34:44.276088+0000","last_fullsized":"2023-01-15T14:34:44.276088+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"375'356","last_scrub_stamp":"2023-01-15T14:34:29.460843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:29.460843+0000","objects_scrubbed":4,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:56.905","scrub_duration":0.30741503599999997,"objects_trimmed":0,"snaptrim_duration":0.0041200940000000004,"stat_sum":{"num_bytes":4071594,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.1b","version":"389'115","reported_seq":173,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.067607+0000","last_change":"2023-01-15T14:34:39.945190+0000","last_active":"2023-01-15T14:34:45.067607+0000","last_peered":"2023-01-15T14:34:45.067607+0000","last_clean":"2023-01-15T14:34:45.067607+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:34:45.067607+0000","last_undegraded":"2023-01-15T14:34:45.067607+0000","last_fullsized":"2023-01-15T14:34:45.067607+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"370'101","last_scrub_stamp":"2023-01-15T14:34:24.137270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:24.137270+0000","objects_scrubbed":3,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:44.214","scrub_duration":0.106791863,"objects_trimmed":0,"snaptrim_duration":0.97518794399999997,"stat_sum":{"num_bytes":8765935,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":665,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.165268+0000","last_change":"2023-01-15T14:34:43.164897+0000","last_active":"2023-01-15T14:34:43.165268+0000","last_peered":"2023-01-15T14:34:43.165268+0000","last_clean":"2023-01-15T14:34:43.165268+0000","last_became_active":"2023-01-15T14:34:43.163678+0000","last_became_peered":"2023-01-15T14:34:43.163678+0000","last_unstale":"2023-01-15T14:34:43.165268+0000","last_undegraded":"2023-01-15T14:34:43.165268+0000","last_fullsized":"2023-01-15T14:34:43.165268+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:22.440826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:22.440826+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:37.391","scrub_duration":0.30343327599999997,"objects_trimmed":0,"snaptrim_duration":0.0018140840000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1381,"reported_epoch":394,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:34:44.473028+0000","last_change":"2023-01-15T14:34:44.472829+0000","last_active":"2023-01-15T14:34:44.473028+0000","last_peered":"2023-01-15T14:34:44.473028+0000","last_clean":"2023-01-15T14:34:40.981057+0000","last_became_active":"2023-01-15T14:34:44.472829+0000","last_became_peered":"2023-01-15T14:34:44.472829+0000","last_unstale":"2023-01-15T14:34:44.473028+0000","last_undegraded":"2023-01-15T14:34:44.473028+0000","last_fullsized":"2023-01-15T14:34:44.069891+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.069055095999999996,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,3],"avail_no_missing":["4","1","3","7"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":1081,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.164725+0000","last_change":"2023-01-15T14:34:43.164725+0000","last_active":"2023-01-15T14:34:43.164725+0000","last_peered":"2023-01-15T14:34:43.164725+0000","last_clean":"2023-01-15T14:34:43.164725+0000","last_became_active":"2023-01-15T14:34:43.163816+0000","last_became_peered":"2023-01-15T14:34:43.163816+0000","last_unstale":"2023-01-15T14:34:43.164725+0000","last_undegraded":"2023-01-15T14:34:43.164725+0000","last_fullsized":"2023-01-15T14:34:43.164725+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.41142812200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.1f","version":"389'267","reported_seq":1425,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.169242+0000","last_change":"2023-01-15T14:34:40.521359+0000","last_active":"2023-01-15T14:34:45.169242+0000","last_peered":"2023-01-15T14:34:45.169242+0000","last_clean":"2023-01-15T14:34:45.169242+0000","last_became_active":"2023-01-15T14:34:00.240936+0000","last_became_peered":"2023-01-15T14:34:00.240936+0000","last_unstale":"2023-01-15T14:34:45.169242+0000","last_undegraded":"2023-01-15T14:34:45.169242+0000","last_fullsized":"2023-01-15T14:34:45.169242+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.44753137999999998,"stat_sum":{"num_bytes":6424165,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":13343,"num_write":9,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":5949777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.20","version":"324'273","reported_seq":1445,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.166600+0000","last_change":"2023-01-15T14:34:43.364416+0000","last_active":"2023-01-15T14:34:45.166600+0000","last_peered":"2023-01-15T14:34:45.166600+0000","last_clean":"2023-01-15T14:34:45.166600+0000","last_became_active":"2023-01-15T14:33:55.472883+0000","last_became_peered":"2023-01-15T14:33:55.472883+0000","last_unstale":"2023-01-15T14:34:45.166600+0000","last_undegraded":"2023-01-15T14:34:45.166600+0000","last_fullsized":"2023-01-15T14:34:45.166600+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'224","last_scrub_stamp":"2023-01-15T14:30:23.749054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.749054+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:34:48.364","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.88508231199999998,"stat_sum":{"num_bytes":4251137,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10864,"num_write":9,"num_write_kb":4140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4438968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.21","version":"321'138","reported_seq":1124,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:41.944302+0000","last_change":"2023-01-15T14:34:40.397726+0000","last_active":"2023-01-15T14:34:41.944302+0000","last_peered":"2023-01-15T14:34:41.944302+0000","last_clean":"2023-01-15T14:34:41.944302+0000","last_became_active":"2023-01-15T14:33:45.267321+0000","last_became_peered":"2023-01-15T14:33:45.267321+0000","last_unstale":"2023-01-15T14:34:41.944302+0000","last_undegraded":"2023-01-15T14:34:41.944302+0000","last_fullsized":"2023-01-15T14:34:41.944302+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.38195272400000002,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12118,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.22","version":"323'161","reported_seq":1440,"reported_epoch":393,"state":"remapped+peering","last_fresh":"2023-01-15T14:34:42.959375+0000","last_change":"2023-01-15T14:34:42.959509+0000","last_active":"2023-01-15T14:34:41.943624+0000","last_peered":"2023-01-15T14:34:41.081441+0000","last_clean":"2023-01-15T14:34:41.081441+0000","last_became_active":"2023-01-15T14:34:40.072878+0000","last_became_peered":"2023-01-15T14:34:40.072878+0000","last_unstale":"2023-01-15T14:34:42.959375+0000","last_undegraded":"2023-01-15T14:34:42.959375+0000","last_fullsized":"2023-01-15T14:34:42.959375+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.727074989,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1,7],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":826,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.067265+0000","last_change":"2023-01-15T14:34:39.270753+0000","last_active":"2023-01-15T14:34:45.067265+0000","last_peered":"2023-01-15T14:34:45.067265+0000","last_clean":"2023-01-15T14:34:45.067265+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:34:45.067265+0000","last_undegraded":"2023-01-15T14:34:45.067265+0000","last_fullsized":"2023-01-15T14:34:45.067265+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30128483900000003,"stat_sum":{"num_bytes":3802136,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":21818,"num_write":18,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.24","version":"389'275","reported_seq":1499,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.038403+0000","last_change":"2023-01-15T14:34:40.375473+0000","last_active":"2023-01-15T14:34:42.038403+0000","last_peered":"2023-01-15T14:34:42.038403+0000","last_clean":"2023-01-15T14:34:42.038403+0000","last_became_active":"2023-01-15T14:33:45.075309+0000","last_became_peered":"2023-01-15T14:33:45.075309+0000","last_unstale":"2023-01-15T14:34:42.038403+0000","last_undegraded":"2023-01-15T14:34:42.038403+0000","last_fullsized":"2023-01-15T14:34:42.038403+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.40916475699999999,"stat_sum":{"num_bytes":7636039,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":20,"num_read_kb":22662,"num_write":11,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.25","version":"0'0","reported_seq":1,"reported_epoch":392,"state":"unknown","last_fresh":"2023-01-15T14:34:41.957854+0000","last_change":"2023-01-15T14:34:42.951995+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:41.957854+0000","last_undegraded":"2023-01-15T14:34:41.957854+0000","last_fullsized":"2023-01-15T14:34:41.957854+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"102'52","reported_seq":548,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:42.147981+0000","last_change":"2023-01-15T14:34:40.500846+0000","last_active":"2023-01-15T14:34:42.147981+0000","last_peered":"2023-01-15T14:34:42.147981+0000","last_clean":"2023-01-15T14:34:42.147981+0000","last_became_active":"2023-01-15T14:33:52.309110+0000","last_became_peered":"2023-01-15T14:33:52.309110+0000","last_unstale":"2023-01-15T14:34:42.147981+0000","last_undegraded":"2023-01-15T14:34:42.147981+0000","last_fullsized":"2023-01-15T14:34:42.147981+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.42986139600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":1150,"reported_epoch":392,"state":"active+clean","last_fresh":"2023-01-15T14:34:42.352054+0000","last_change":"2023-01-15T14:34:39.984673+0000","last_active":"2023-01-15T14:34:42.352054+0000","last_peered":"2023-01-15T14:34:42.352054+0000","last_clean":"2023-01-15T14:34:42.352054+0000","last_became_active":"2023-01-15T14:33:45.171090+0000","last_became_peered":"2023-01-15T14:33:45.171090+0000","last_unstale":"2023-01-15T14:34:42.352054+0000","last_undegraded":"2023-01-15T14:34:42.352054+0000","last_fullsized":"2023-01-15T14:34:42.352054+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":1.015148183,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.28","version":"333'248","reported_seq":1500,"reported_epoch":393,"state":"remapped+peering","last_fresh":"2023-01-15T14:34:43.163285+0000","last_change":"2023-01-15T14:34:43.163536+0000","last_active":"2023-01-15T14:34:42.036222+0000","last_peered":"2023-01-15T14:34:40.980983+0000","last_clean":"2023-01-15T14:34:40.980983+0000","last_became_active":"2023-01-15T14:34:40.371901+0000","last_became_peered":"2023-01-15T14:34:40.371901+0000","last_unstale":"2023-01-15T14:34:43.163285+0000","last_undegraded":"2023-01-15T14:34:43.163285+0000","last_fullsized":"2023-01-15T14:34:43.163285+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'248","last_scrub_stamp":"2023-01-15T14:34:20.620068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.620068+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:40.951","scrub_duration":0.40448800800000001,"objects_trimmed":0,"snaptrim_duration":0.43092023899999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":994,"reported_epoch":393,"state":"remapped+peering","last_fresh":"2023-01-15T14:34:42.959574+0000","last_change":"2023-01-15T14:34:42.959765+0000","last_active":"2023-01-15T14:34:41.948654+0000","last_peered":"2023-01-15T14:34:41.107638+0000","last_clean":"2023-01-15T14:34:41.107638+0000","last_became_active":"2023-01-15T14:34:40.497565+0000","last_became_peered":"2023-01-15T14:34:40.497565+0000","last_unstale":"2023-01-15T14:34:42.959574+0000","last_undegraded":"2023-01-15T14:34:42.959574+0000","last_fullsized":"2023-01-15T14:34:42.959574+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.015893349000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1314,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:42.045505+0000","last_change":"2023-01-15T14:34:40.499541+0000","last_active":"2023-01-15T14:34:42.045505+0000","last_peered":"2023-01-15T14:34:42.045505+0000","last_clean":"2023-01-15T14:34:42.045505+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:34:42.045505+0000","last_undegraded":"2023-01-15T14:34:42.045505+0000","last_fullsized":"2023-01-15T14:34:42.045505+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.53033388100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":153,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.067239+0000","last_change":"2023-01-15T14:34:43.358345+0000","last_active":"2023-01-15T14:34:45.067239+0000","last_peered":"2023-01-15T14:34:45.067239+0000","last_clean":"2023-01-15T14:34:45.067239+0000","last_became_active":"2023-01-15T14:34:43.357195+0000","last_became_peered":"2023-01-15T14:34:43.357195+0000","last_unstale":"2023-01-15T14:34:45.067239+0000","last_undegraded":"2023-01-15T14:34:45.067239+0000","last_fullsized":"2023-01-15T14:34:45.067239+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.30229355499999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.2c","version":"141'46","reported_seq":521,"reported_epoch":393,"state":"active+clean","last_fresh":"2023-01-15T14:34:43.053029+0000","last_change":"2023-01-15T14:34:39.270892+0000","last_active":"2023-01-15T14:34:43.053029+0000","last_peered":"2023-01-15T14:34:43.053029+0000","last_clean":"2023-01-15T14:34:43.053029+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:34:43.053029+0000","last_undegraded":"2023-01-15T14:34:43.053029+0000","last_fullsized":"2023-01-15T14:34:43.053029+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.20522305299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1287,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:42.149546+0000","last_change":"2023-01-15T14:34:40.313864+0000","last_active":"2023-01-15T14:34:42.149546+0000","last_peered":"2023-01-15T14:34:42.149546+0000","last_clean":"2023-01-15T14:34:42.149546+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:34:42.149546+0000","last_undegraded":"2023-01-15T14:34:42.149546+0000","last_fullsized":"2023-01-15T14:34:42.149546+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.292820048,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":123,"reported_epoch":393,"state":"active","last_fresh":"2023-01-15T14:34:43.265888+0000","last_change":"2023-01-15T14:34:43.265888+0000","last_active":"2023-01-15T14:34:43.265888+0000","last_peered":"2023-01-15T14:34:43.265888+0000","last_clean":"2023-01-15T14:34:41.079073+0000","last_became_active":"2023-01-15T14:34:43.265888+0000","last_became_peered":"2023-01-15T14:34:43.265888+0000","last_unstale":"2023-01-15T14:34:43.265888+0000","last_undegraded":"2023-01-15T14:34:43.265888+0000","last_fullsized":"2023-01-15T14:34:43.265888+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":387,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.58930503300000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":["7","1","2","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1181,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:41.944170+0000","last_change":"2023-01-15T14:34:40.603128+0000","last_active":"2023-01-15T14:34:41.944170+0000","last_peered":"2023-01-15T14:34:41.944170+0000","last_clean":"2023-01-15T14:34:41.944170+0000","last_became_active":"2023-01-15T14:33:55.174128+0000","last_became_peered":"2023-01-15T14:33:55.174128+0000","last_unstale":"2023-01-15T14:34:41.944170+0000","last_undegraded":"2023-01-15T14:34:41.944170+0000","last_fullsized":"2023-01-15T14:34:41.944170+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.53294120099999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.30","version":"206'252","reported_seq":1074,"reported_epoch":395,"state":"active+clean","last_fresh":"2023-01-15T14:34:45.066108+0000","last_change":"2023-01-15T14:34:39.268767+0000","last_active":"2023-01-15T14:34:45.066108+0000","last_peered":"2023-01-15T14:34:45.066108+0000","last_clean":"2023-01-15T14:34:45.066108+0000","last_became_active":"2023-01-15T14:33:55.373498+0000","last_became_peered":"2023-01-15T14:33:55.373498+0000","last_unstale":"2023-01-15T14:34:45.066108+0000","last_undegraded":"2023-01-15T14:34:45.066108+0000","last_fullsized":"2023-01-15T14:34:45.066108+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:34:36.078678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:36.078678+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:50.459","scrub_duration":1.158842567,"objects_trimmed":0,"snaptrim_duration":0.30616606099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"}]},{"pgid":"3.31","version":"0'0","reported_seq":1,"reported_epoch":392,"state":"unknown","last_fresh":"2023-01-15T14:34:41.955571+0000","last_change":"2023-01-15T14:34:42.951751+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:41.955571+0000","last_undegraded":"2023-01-15T14:34:41.955571+0000","last_fullsized":"2023-01-15T14:34:41.955571+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":106,"stat_sum":{"num_bytes":417290186,"num_objects":202,"num_object_clones":148,"num_object_copies":606,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":202,"num_whiteouts":18,"num_read":957,"num_read_kb":1044361,"num_write":747,"num_write_kb":309403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":631,"num_bytes_recovered":422441852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18691,"ondisk_log_size":18691,"up":318,"acting":299,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574929,"num_pgs":72,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1167596,"kb_used_data":92533252,"kb_used_omap":9636,"kb_used_meta":0,"kb_avail":92531784,"statfs":{"total":95948165120,"available":94752546816,"internally_reserved":1503232,"allocated":94754050048,"data_stored":94754050048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9867272,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2,3,0,5],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":3.452,"15min":3.8639999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1830000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":6.0389999999999997,"5min":5.6859999999999999,"15min":8.0340000000000007},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":36.622,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.91700000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":12.064,"5min":4.7549999999999999,"15min":5.9480000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":204.74199999999999,"5min":204.74199999999999,"15min":204.74199999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":11.260999999999999,"5min":9.1910000000000007,"15min":7.25},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":94.537000000000006,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.96399999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":6.4119999999999999,"15min":5.3310000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":25.065999999999999,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.89700000000000002},{"interface":"front","average":{"1min":15.106,"5min":4.7649999999999997,"15min":4.3730000000000002},"min":{"1min":0.59599999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":162.96799999999999,"5min":162.96799999999999,"15min":162.96799999999999},"last":1.0029999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.198,"5min":7.6299999999999999,"15min":3.1749999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":25.359999999999999,"5min":209.05500000000001,"15min":209.05500000000001},"last":1.103},{"interface":"front","average":{"1min":0.86499999999999999,"5min":2.879,"15min":4.0609999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.266,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.85499999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.4430000000000001,"5min":4.2279999999999998,"15min":7.6859999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":29.690000000000001,"5min":96.185000000000002,"15min":96.185000000000002},"last":0.65000000000000002},{"interface":"front","average":{"1min":10.949999999999999,"5min":7.7789999999999999,"15min":3.2360000000000002},"min":{"1min":0.46200000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":142.44999999999999,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.76000000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.5489999999999999,"5min":4.3099999999999996,"15min":5.6539999999999999},"min":{"1min":0.63800000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":29.797999999999998,"5min":96.263999999999996,"15min":96.263999999999996},"last":0.872},{"interface":"front","average":{"1min":2.3519999999999999,"5min":2.6869999999999998,"15min":4.1920000000000002},"min":{"1min":0.55300000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":25.199000000000002,"5min":70.387,"15min":95.430000000000007},"last":1.0429999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.645,"5min":4.4189999999999996,"15min":7.1550000000000002},"min":{"1min":0.60999999999999999,"5min":0.496,"15min":0.496},"max":{"1min":29.876000000000001,"5min":96.299999999999997,"15min":96.299999999999997},"last":0.80900000000000005},{"interface":"front","average":{"1min":2.399,"5min":2.7749999999999999,"15min":1.6579999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":25.236999999999998,"5min":70.530000000000001,"15min":70.530000000000001},"last":0.97599999999999998}]}]},{"osd":6,"up_from":0,"seq":55834574931,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574931,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574929,"num_pgs":65,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1034824,"kb_used_data":92674212,"kb_used_omap":15501,"kb_used_meta":0,"kb_avail":92664556,"statfs":{"total":95948165120,"available":94888505344,"internally_reserved":9887744,"allocated":94898393088,"data_stored":94898393088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15873364,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":22.818000000000001,"15min":10.882},"min":{"1min":0.505,"5min":0.44,"15min":0.44},"max":{"1min":5.0049999999999999,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.77400000000000002,"5min":2.9990000000000001,"15min":1.52},"min":{"1min":0.51200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.127,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.96799999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.9390000000000001,"5min":3.742,"15min":6.976},"min":{"1min":0.499,"5min":0.49299999999999999,"15min":0.47399999999999998},"max":{"1min":23.193000000000001,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":10.99,"5min":6.3819999999999997,"15min":2.6560000000000001},"min":{"1min":0.67100000000000004,"5min":0.52900000000000003,"15min":0.50800000000000001},"max":{"1min":162.976,"5min":162.976,"15min":162.976},"last":0.64400000000000002}]},{"osd":2,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.0950000000000002,"5min":23.544,"15min":11.522},"min":{"1min":0.497,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":23.247,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.26,"5min":1.28,"15min":1.0329999999999999},"min":{"1min":0.61899999999999999,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.5789999999999997,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.754}]},{"osd":3,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":2.802,"15min":1.8260000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.006,"5min":88.823999999999998,"15min":88.823999999999998},"last":1.052},{"interface":"front","average":{"1min":5.3380000000000001,"5min":24.337,"15min":12.212999999999999},"min":{"1min":0.68600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":58.317,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":1.0029999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88100000000000001,"15min":0.90700000000000003},"min":{"1min":0.61499999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.175,"5min":2.577,"15min":2.577},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.903,"15min":2.1669999999999998},"min":{"1min":0.69099999999999995,"5min":0.65200000000000002,"15min":0.55300000000000005},"max":{"1min":1.0700000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":1.105}]},{"osd":6,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":3.2389999999999999,"15min":4.8609999999999998},"min":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":23.329000000000001,"5min":85.721999999999994,"15min":99.780000000000001},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.999,"5min":21.631,"15min":10.651999999999999},"min":{"1min":0.79500000000000004,"5min":0.66700000000000004,"15min":0.58699999999999997},"max":{"1min":1.173,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":0.95099999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":3.238,"5min":3.5790000000000002,"15min":9.9450000000000003},"min":{"1min":0.73999999999999999,"5min":0.63400000000000001,"15min":0.59499999999999997},"max":{"1min":42.551000000000002,"5min":89.061000000000007,"15min":179.73500000000001},"last":1.1599999999999999},{"interface":"front","average":{"1min":2.274,"5min":23.071000000000002,"15min":15.183},"min":{"1min":0.78700000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":23.356000000000002,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.0880000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574929,"num_pgs":70,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1127316,"kb_used_data":92573488,"kb_used_omap":15811,"kb_used_meta":0,"kb_avail":92572064,"statfs":{"total":95948165120,"available":94793793536,"internally_reserved":1458176,"allocated":94795251712,"data_stored":94795251712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16190974,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":7.8860000000000001,"5min":8.048,"15min":5.9139999999999997},"min":{"1min":0.39500000000000002,"5min":0.255,"15min":0.255},"max":{"1min":74.245999999999995,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":7.9649999999999999,"5min":3.4990000000000001,"15min":6.2610000000000001},"min":{"1min":0.46600000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":74.280000000000001,"5min":74.280000000000001,"15min":93.674000000000007},"last":68.239000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":10.762,"5min":6.625,"15min":11.228999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":117.664,"5min":117.664,"15min":159.256},"last":0.71199999999999997},{"interface":"front","average":{"1min":4.6909999999999998,"5min":11.208,"15min":4.2300000000000004},"min":{"1min":0.41899999999999998,"5min":0.317,"15min":0.317},"max":{"1min":47.963000000000001,"5min":174.578,"15min":174.578},"last":0.66100000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":2.4820000000000002,"15min":3.0600000000000001},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":13.113,"5min":86.412000000000006,"15min":86.412000000000006},"last":68.156000000000006},{"interface":"front","average":{"1min":2.3079999999999998,"5min":6.8120000000000003,"15min":3.7320000000000002},"min":{"1min":0.44,"5min":0.375,"15min":0.375},"max":{"1min":15.946999999999999,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.62}]},{"osd":4,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":28.689,"15min":13.936999999999999},"min":{"1min":0.56299999999999994,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":14.234999999999999,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":68.296000000000006},{"interface":"front","average":{"1min":11.906000000000001,"5min":31.741,"15min":19.829000000000001},"min":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":117.59,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":0.81799999999999995}]},{"osd":5,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":11.022,"5min":5.9219999999999997,"15min":11.242000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":117.744,"5min":117.744,"15min":159.43000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":5.9829999999999997,"15min":2.6749999999999998},"min":{"1min":0.66200000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2210000000000001,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.90000000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":5.984,"15min":2.6819999999999999},"min":{"1min":0.68300000000000005,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.286,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.3979999999999999,"15min":2.9740000000000002},"min":{"1min":0.71099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6799999999999999,"5min":43.661000000000001,"15min":60.098999999999997},"last":68.394999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.4119999999999999,"15min":3.161},"min":{"1min":0.753,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.758,"5min":43.712000000000003,"15min":60.165999999999997},"last":68.469999999999999},{"interface":"front","average":{"1min":19.324999999999999,"5min":11.952,"15min":16.044},"min":{"1min":0.65400000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":142.02000000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":1}]}]},{"osd":2,"up_from":13,"seq":55834574930,"num_pgs":57,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1068652,"kb_used_data":92631404,"kb_used_omap":13518,"kb_used_meta":0,"kb_avail":92630728,"statfs":{"total":95948165120,"available":94853865472,"internally_reserved":692224,"allocated":94854557696,"data_stored":94854557696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13843271,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":4.2469999999999999,"5min":4.2450000000000001,"15min":1.8420000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":78.456000000000003,"5min":78.456000000000003,"15min":78.456000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.4039999999999999,"15min":0.94399999999999995},"min":{"1min":0.36799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.55000000000000004}]},{"osd":1,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":10.247,"5min":7.7949999999999999,"15min":3.1309999999999998},"min":{"1min":0.35099999999999998,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":128.749,"5min":143.071,"15min":143.071},"last":0.60499999999999998},{"interface":"front","average":{"1min":3.105,"5min":1.966,"15min":2.8919999999999999},"min":{"1min":0.35999999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":52.927999999999997,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.65700000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":3.8879999999999999,"5min":6.9180000000000001,"15min":9.2400000000000002},"min":{"1min":0.33700000000000002,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":34.244,"5min":143.13300000000001,"15min":183.53},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":7.5039999999999996,"15min":3.012},"min":{"1min":0.41399999999999998,"5min":0.316,"15min":0.316},"max":{"1min":1.3959999999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.51900000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":5.9480000000000004,"15min":4.4160000000000004},"min":{"1min":0.67200000000000004,"5min":0.624,"15min":0.58399999999999996},"max":{"1min":1.2450000000000001,"5min":175.34700000000001,"15min":175.34700000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.88300000000000001,"5min":4.7080000000000002,"15min":4.1079999999999997},"min":{"1min":0.57799999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.052,"5min":175.30500000000001,"15min":175.30500000000001},"last":1.0620000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":2.5590000000000002,"5min":2.7490000000000001,"15min":1.7},"min":{"1min":0.63500000000000001,"5min":0.57199999999999995,"15min":0.54800000000000004},"max":{"1min":34.256999999999998,"5min":143.262,"15min":143.262},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.95099999999999996,"5min":3.012,"15min":1.778},"min":{"1min":0.70199999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3160000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.95999999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.839,"15min":1.3009999999999999},"min":{"1min":0.61199999999999999,"5min":0.59599999999999997,"15min":0.55000000000000004},"max":{"1min":1.276,"5min":88.542000000000002,"15min":88.542000000000002},"last":1.1499999999999999},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8980000000000001,"15min":1.778},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":34.305999999999997,"5min":143.215,"15min":143.215},"last":0.879}]},{"osd":7,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":11.396000000000001,"5min":7.9800000000000004,"15min":4.6989999999999998},"min":{"1min":0.60599999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":176.44499999999999,"5min":176.44499999999999,"15min":176.44499999999999},"last":1.115},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8159999999999998,"15min":2.79},"min":{"1min":0.622,"5min":0.622,"15min":0.54500000000000004},"max":{"1min":34.207999999999998,"5min":143.17099999999999,"15min":143.17099999999999},"last":0.93799999999999994}]}]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":16,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":994028,"kb_used_data":92705368,"kb_used_omap":14475,"kb_used_meta":0,"kb_avail":92705352,"statfs":{"total":95948165120,"available":94930280448,"internally_reserved":16384,"allocated":94930296832,"data_stored":94930296832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14823194,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":2.0619999999999998,"5min":3.2669999999999999,"15min":3.6840000000000002},"min":{"1min":0.37,"5min":0.314,"15min":0.314},"max":{"1min":23.68,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":14.157,"5min":6.5789999999999997,"15min":5.0750000000000002},"min":{"1min":0.42299999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":176.01599999999999,"5min":176.01599999999999,"15min":176.01599999999999},"last":0.41299999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":4.8959999999999999,"5min":3.4329999999999998,"15min":1.7889999999999999},"min":{"1min":0.44400000000000001,"5min":0.245,"15min":0.245},"max":{"1min":69.277000000000001,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.624},{"interface":"front","average":{"1min":0.629,"5min":1.173,"15min":3.1059999999999999},"min":{"1min":0.28399999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.873,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.66500000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":5.8440000000000003,"15min":15.172000000000001},"min":{"1min":0.38400000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.92500000000000004,"5min":245.637,"15min":282.01400000000001},"last":125.736},{"interface":"front","average":{"1min":14.228999999999999,"5min":8.2870000000000008,"15min":7.5919999999999996},"min":{"1min":0.46500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":176.13900000000001,"5min":176.13900000000001,"15min":176.13900000000001},"last":0.53600000000000003}]},{"osd":4,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":19.170000000000002,"5min":20.029,"15min":9.8279999999999994},"min":{"1min":0.56299999999999994,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":176.18600000000001,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":5.5510000000000002,"5min":15.916,"15min":7.7080000000000002},"min":{"1min":0.64300000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":69.611999999999995,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.999}]},{"osd":5,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":5.2060000000000004,"5min":2.3799999999999999,"15min":1.482},"min":{"1min":0.56899999999999995,"5min":0.433,"15min":0.433},"max":{"1min":69.478999999999999,"5min":69.478999999999999,"15min":69.478999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":3.2770000000000001},"min":{"1min":0.66000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3089999999999999,"5min":4.0780000000000003,"15min":46.905999999999999},"last":0.75}]},{"osd":6,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":14.491,"5min":6.9109999999999996,"15min":5.6379999999999999},"min":{"1min":0.73199999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":176.249,"5min":176.249,"15min":176.249},"last":0.80500000000000005},{"interface":"front","average":{"1min":5.2629999999999999,"5min":2.431,"15min":1.5589999999999999},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":69.549999999999997,"5min":69.549999999999997,"15min":69.549999999999997},"last":1.087}]},{"osd":7,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":7.617,"5min":6.2309999999999999,"15min":6.7549999999999999},"min":{"1min":0.63900000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":95.888999999999996,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":14.544,"5min":6.9649999999999999,"15min":5.4880000000000004},"min":{"1min":0.78700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":176.22,"5min":176.22,"15min":176.22},"last":0.75600000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574931,"num_pgs":72,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1149660,"kb_used_data":92551796,"kb_used_omap":9981,"kb_used_meta":0,"kb_avail":92549720,"statfs":{"total":95948165120,"available":94770913280,"internally_reserved":2125824,"allocated":94773039104,"data_stored":94773039104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10221232,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":3.3460000000000001,"15min":1.53},"min":{"1min":0.34899999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":25.960999999999999,"5min":69.872,"15min":69.872},"last":0.378},{"interface":"front","average":{"1min":11.249000000000001,"5min":5.335,"15min":2.2650000000000001},"min":{"1min":0.434,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":162.91200000000001,"5min":162.91200000000001,"15min":162.91200000000001},"last":0.29899999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":3.415,"15min":1.601},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0109999999999999,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.58599999999999997,"5min":2.6320000000000001,"15min":1.3500000000000001},"min":{"1min":0.37,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86899999999999999,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.34300000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":7.8079999999999998,"15min":13.788},"min":{"1min":0.48199999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":26.027000000000001,"5min":199.08000000000001,"15min":284.62799999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":3.6070000000000002,"5min":7.867,"15min":3.1560000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33600000000000002},"max":{"1min":61.119,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.47999999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":16.664999999999999,"15min":6.2119999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1719999999999999,"5min":1486.309,"15min":1486.309},"last":0.59099999999999997},{"interface":"front","average":{"1min":2.1030000000000002,"5min":18.140999999999998,"15min":6.7300000000000004},"min":{"1min":0.63100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":26.137,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":0.64500000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":2.8980000000000001,"15min":7.4980000000000002},"min":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1930000000000001,"5min":98.411000000000001,"15min":176.34399999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.7070000000000001,"15min":1.2609999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.101,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.63100000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":2.1600000000000001,"5min":3.7890000000000001,"15min":2.0430000000000001},"min":{"1min":0.71899999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":26.082000000000001,"5min":70.259,"15min":70.259},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.026,"5min":3,"15min":1.756},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.57499999999999996},"max":{"1min":1.5409999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":0.751}]},{"osd":7,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":3.2770000000000001,"15min":11.539},"min":{"1min":0.70099999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":11.906000000000001,"5min":73.864999999999995,"15min":292.61099999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":2.1819999999999999,"5min":3.8090000000000002,"15min":2.0499999999999998},"min":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.58699999999999997},"max":{"1min":26.187000000000001,"5min":70.322000000000003,"15min":70.322000000000003},"last":0.78300000000000003}]}]}],"pool_statfs":[]}} 2023-01-15T14:34:48.737 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:34:48.737 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 116 2023-01-15T14:34:48.738 INFO:tasks.rados.rados.0.smithi090.stdout:1497: finishing copy_from to smithi09025264-2 2023-01-15T14:34:48.738 INFO:tasks.rados.rados.0.smithi090.stdout:1497: got expected ENOENT (src dne) 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:1497: finishing copy_from racing read to smithi09025264-2 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:1500: finishing rollback tid 0 to smithi09025264-29 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:1500: finishing rollback tid 1 to smithi09025264-29 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:1500: finishing rollback tid 2 to smithi09025264-29 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 274 (ObjNum 265 snap 65 seq_num 265) dirty exists 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:1496: done (4 left) 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:1497: done (3 left) 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:1498: done (2 left) 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:1499: done (1 left) 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:1500: done (0 left) 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:1502: done (0 left) 2023-01-15T14:34:48.739 INFO:tasks.rados.rados.0.smithi090.stdout:1503: copy_from oid 21 from oid 46 current snap is 164 2023-01-15T14:34:48.740 INFO:tasks.rados.rados.0.smithi090.stdout:1504: snap_create 2023-01-15T14:34:48.740 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:48.740 INFO:tasks.rados.rados.0.smithi090.stdout:1503: finishing copy_from to smithi09025264-21 2023-01-15T14:34:48.740 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 21 v 320 (ObjNum 506 snap 150 seq_num 506) dirty exists 2023-01-15T14:34:48.740 INFO:tasks.rados.rados.0.smithi090.stdout:1503: finishing copy_from racing read to smithi09025264-21 2023-01-15T14:34:48.740 INFO:tasks.rados.rados.0.smithi090.stdout:1503: done (0 left) 2023-01-15T14:34:48.743 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:48.735+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:48.743 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:48.743 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:48.744 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:34:48.842 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:48.835+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:48.922 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:48.922 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:48.922 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:48.922 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:48.922 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:48.923 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:34:48.932 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:48.953 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:48.965 INFO:tasks.rados.rados.0.smithi090.stdout:1504: done (0 left) 2023-01-15T14:34:48.965 INFO:tasks.rados.rados.0.smithi090.stdout:1505: read oid 24 snap -1 2023-01-15T14:34:48.965 INFO:tasks.rados.rados.0.smithi090.stdout:1505: expect (ObjNum 486 snap 142 seq_num 486) 2023-01-15T14:34:48.965 INFO:tasks.rados.rados.0.smithi090.stdout:1506: read oid 50 snap -1 2023-01-15T14:34:48.965 INFO:tasks.rados.rados.0.smithi090.stdout:1506: expect (ObjNum 517 snap 155 seq_num 517) 2023-01-15T14:34:48.966 INFO:tasks.rados.rados.0.smithi090.stdout:1507: delete oid 13 current snap is 165 2023-01-15T14:34:48.970 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:48.966+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:48.977 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:49.117 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:49.117 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:49.118 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:49.118 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:49.118 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:34:49.119 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:49.120 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:34:49.121 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:49.110+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:49.121 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:34:49.121 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:34:49.122 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:34:49.122 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:34:49.122 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:34:49.122 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:34:49.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:34:49.123 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:34:49.167 INFO:tasks.rados.rados.0.smithi090.stdout:1507: done (2 left) 2023-01-15T14:34:49.944 INFO:tasks.rados.rados.0.smithi090.stdout:1508: snap_create 2023-01-15T14:34:49.945 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:34:49.947 INFO:tasks.rados.rados.0.smithi090.stdout:1506: expect (ObjNum 517 snap 155 seq_num 517) 2023-01-15T14:34:49.948 INFO:tasks.rados.rados.0.smithi090.stdout:1505: expect (ObjNum 486 snap 142 seq_num 486) 2023-01-15T14:34:49.948 INFO:tasks.rados.rados.0.smithi090.stdout:1505: done (1 left) 2023-01-15T14:34:49.948 INFO:tasks.rados.rados.0.smithi090.stdout:1506: done (0 left) 2023-01-15T14:34:49.949 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:49.939+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:49.949 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:49.950 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:49.950 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:34:49.950 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:34:49.950 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:49.951 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:34:50.156 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:50.150+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:50.157 INFO:tasks.rados.rados.0.smithi090.stdout:1508: done (0 left) 2023-01-15T14:34:50.158 INFO:tasks.rados.rados.0.smithi090.stdout:1509: read oid 22 snap -1 2023-01-15T14:34:50.158 INFO:tasks.rados.rados.0.smithi090.stdout:1509: expect (ObjNum 471 snap 136 seq_num 471) 2023-01-15T14:34:50.158 INFO:tasks.rados.rados.0.smithi090.stdout:1510: read oid 23 snap -1 2023-01-15T14:34:50.158 INFO:tasks.rados.rados.0.smithi090.stdout:1510: expect (ObjNum 406 snap 108 seq_num 406) 2023-01-15T14:34:50.158 INFO:tasks.rados.rados.0.smithi090.stdout:1511: read oid 30 snap 149 2023-01-15T14:34:50.353 INFO:tasks.rados.rados.0.smithi090.stdout:1511: expect (ObjNum 492 snap 144 seq_num 492) 2023-01-15T14:34:50.354 INFO:tasks.rados.rados.0.smithi090.stdout:1512: write (excl) oid 18 current snap is 166 2023-01-15T14:34:50.354 INFO:tasks.rados.rados.0.smithi090.stdout:1512: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2023-01-15T14:34:50.354 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 116 2023-01-15T14:34:50.355 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:34:50.356 INFO:tasks.rados.rados.0.smithi090.stdout:1512: writing smithi09025264-18 from 625359 to 1178782 tid 1 2023-01-15T14:34:50.356 INFO:tasks.rados.rados.0.smithi090.stdout:1512: writing smithi09025264-18 from 1790948 to 2358132 tid 2 2023-01-15T14:34:50.356 INFO:tasks.rados.rados.0.smithi090.stdout:1512: writing smithi09025264-18 from 2930500 to 3025359 tid 3 2023-01-15T14:34:50.357 INFO:tasks.rados.rados.0.smithi090.stdout:1513: delete oid 39 current snap is 166 2023-01-15T14:34:50.361 INFO:tasks.rados.rados.0.smithi090.stdout:1513: done (4 left) 2023-01-15T14:34:50.361 INFO:tasks.rados.rados.0.smithi090.stdout:1514: snap_remove snap 155 2023-01-15T14:34:50.362 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:50.362 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:34:50.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:34:50.363 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:34:50.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.363 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:75 3.f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (56) in=7b,truncate 2302526] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:50.363 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:34:50.363 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 383.90654398200002, 2023-01-15T14:34:50.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92335155400000002, 2023-01-15T14:34:50.363 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.363 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:50.363 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.363 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.364 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.364 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 75 2023-01-15T14:34:50.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.364 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.364 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.364 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:34:50.364 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356897+0000", 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356896+0000", 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356903+0000", 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4619999999999999e-06 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356917+0000", 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4195999999999999e-05 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356926+0000", 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6039999999999993e-06 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457743+0000", 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100817668 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.457841+0000", 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7079999999999999e-05 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778474+0000", 2023-01-15T14:34:50.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.32063301799999999 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778627+0000", 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015389599999999999 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.778815+0000", 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018726600000000001 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:50.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779033+0000", 2023-01-15T14:34:50.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021819999999999999 2023-01-15T14:34:50.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:50.373 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779051+0000", 2023-01-15T14:34:50.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7640000000000001e-05 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779114+0000", 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3590999999999995e-05 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.779163+0000", 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8551000000000003e-05 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.081002+0000", 2023-01-15T14:34:50.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30183937799999999 2023-01-15T14:34:50.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280043+0000", 2023-01-15T14:34:50.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19904084699999999 2023-01-15T14:34:50.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:50.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280138+0000", 2023-01-15T14:34:50.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4915999999999997e-05 2023-01-15T14:34:50.378 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.378 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.378 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.378 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280248+0000", 2023-01-15T14:34:50.378 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000110233 2023-01-15T14:34:50.378 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.378 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.379 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.379 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.379 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:76 3.f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:50.379 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:34:50.379 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 383.90650237400001, 2023-01-15T14:34:50.379 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.92420393899999997, 2023-01-15T14:34:50.380 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.380 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:50.380 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.380 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.380 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.381 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 76 2023-01-15T14:34:50.381 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.381 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.381 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.381 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.381 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:34:50.382 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.382 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.382 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.382 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.382 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356938+0000", 2023-01-15T14:34:50.382 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.383 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.383 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356937+0000", 2023-01-15T14:34:50.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:50.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356948+0000", 2023-01-15T14:34:50.384 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.094e-05 2023-01-15T14:34:50.384 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.384 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.384 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.384 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356959+0000", 2023-01-15T14:34:50.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1058e-05 2023-01-15T14:34:50.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.356968+0000", 2023-01-15T14:34:50.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5339999999999999e-06 2023-01-15T14:34:50.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.385 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459524+0000", 2023-01-15T14:34:50.514 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102556147 2023-01-15T14:34:50.514 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.514 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:50.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.459662+0000", 2023-01-15T14:34:50.515 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000137756 2023-01-15T14:34:50.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978621+0000", 2023-01-15T14:34:50.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.51895887600000001 2023-01-15T14:34:50.516 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.516 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.516 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:50.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.978870+0000", 2023-01-15T14:34:50.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024928999999999999 2023-01-15T14:34:50.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.517 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.517 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.280761+0000", 2023-01-15T14:34:50.517 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30189113400000001 2023-01-15T14:34:50.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.517 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281023+0000", 2023-01-15T14:34:50.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026172800000000003 2023-01-15T14:34:50.518 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.281142+0000", 2023-01-15T14:34:50.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000119296 2023-01-15T14:34:50.519 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.519 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.519 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.519 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1077 3.f 3:f87c0183:::smithi09025264-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:34:50.519 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:34:50.519 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 307.5047548, 2023-01-15T14:34:50.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.98557123499999999, 2023-01-15T14:34:50.520 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.520 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:50.520 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.520 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.520 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.520 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1077 2023-01-15T14:34:50.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.521 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:34:50.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758686+0000", 2023-01-15T14:34:50.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758685+0000", 2023-01-15T14:34:50.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:50.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758689+0000", 2023-01-15T14:34:50.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2760000000000002e-06 2023-01-15T14:34:50.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758704+0000", 2023-01-15T14:34:50.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4187e-05 2023-01-15T14:34:50.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758710+0000", 2023-01-15T14:34:50.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4330000000000001e-06 2023-01-15T14:34:50.525 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.525 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848086+0000", 2023-01-15T14:34:50.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089376466000000002 2023-01-15T14:34:50.525 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.525 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:50.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848093+0000", 2023-01-15T14:34:50.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4269999999999996e-06 2023-01-15T14:34:50.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.526 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.526 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952346+0000", 2023-01-15T14:34:50.526 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104252758 2023-01-15T14:34:50.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:50.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952427+0000", 2023-01-15T14:34:50.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1786000000000006e-05 2023-01-15T14:34:50.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743602+0000", 2023-01-15T14:34:50.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.79117428099999998 2023-01-15T14:34:50.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.743892+0000", 2023-01-15T14:34:50.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029067200000000001 2023-01-15T14:34:50.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.744257+0000", 2023-01-15T14:34:50.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000364619 2023-01-15T14:34:50.529 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.529 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.529 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.530 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1087 3.f 3:fd983206:::smithi09025264-7:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:34:50.530 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:34:50.530 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 307.50472031800001, 2023-01-15T14:34:50.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0859964360000001, 2023-01-15T14:34:50.530 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.530 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:50.531 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.531 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.531 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.531 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1087 2023-01-15T14:34:50.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.531 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.531 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:34:50.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:34:50.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:34:50.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:50.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758724+0000", 2023-01-15T14:34:50.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7740000000000004e-06 2023-01-15T14:34:50.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758738+0000", 2023-01-15T14:34:50.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.33e-05 2023-01-15T14:34:50.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758744+0000", 2023-01-15T14:34:50.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2859999999999996e-06 2023-01-15T14:34:50.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.536 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848121+0000", 2023-01-15T14:34:50.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089377097000000003 2023-01-15T14:34:50.536 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.536 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:50.537 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848127+0000", 2023-01-15T14:34:50.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3500000000000002e-06 2023-01-15T14:34:50.537 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.537 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.537 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.537 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952454+0000", 2023-01-15T14:34:50.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104326167 2023-01-15T14:34:50.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.538 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.538 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.538 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952520+0000", 2023-01-15T14:34:50.538 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6166000000000003e-05 2023-01-15T14:34:50.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.538 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:34:50.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.063003+0000", 2023-01-15T14:34:50.539 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.110482916 2023-01-15T14:34:50.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:50.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065265+0000", 2023-01-15T14:34:50.540 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022626370000000001 2023-01-15T14:34:50.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.540 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.540 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:50.540 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065291+0000", 2023-01-15T14:34:50.540 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6095e-05 2023-01-15T14:34:50.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.540 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:50.541 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.066153+0000", 2023-01-15T14:34:50.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086127100000000004 2023-01-15T14:34:50.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.541 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:50.541 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166533+0000", 2023-01-15T14:34:50.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100380177 2023-01-15T14:34:50.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.542 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.542 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166646+0000", 2023-01-15T14:34:50.542 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112717 2023-01-15T14:34:50.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.542 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.542 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844498+0000", 2023-01-15T14:34:50.542 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67785237399999998 2023-01-15T14:34:50.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.543 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:50.543 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844605+0000", 2023-01-15T14:34:50.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106691 2023-01-15T14:34:50.543 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.543 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.543 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844717+0000", 2023-01-15T14:34:50.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011203499999999999 2023-01-15T14:34:50.543 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.544 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.544 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.544 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.544 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1088 3.f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:34:50.545 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:34:50.545 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 307.50466465199997, 2023-01-15T14:34:50.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.086312696, 2023-01-15T14:34:50.545 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.545 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:50.545 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.545 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.545 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.545 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1088 2023-01-15T14:34:50.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.546 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:34:50.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:34:50.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758775+0000", 2023-01-15T14:34:50.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:50.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758786+0000", 2023-01-15T14:34:50.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.076e-05 2023-01-15T14:34:50.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758809+0000", 2023-01-15T14:34:50.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3706999999999999e-05 2023-01-15T14:34:50.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758821+0000", 2023-01-15T14:34:50.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1976e-05 2023-01-15T14:34:50.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878534+0000", 2023-01-15T14:34:50.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.119712966 2023-01-15T14:34:50.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:50.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878540+0000", 2023-01-15T14:34:50.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0680000000000002e-06 2023-01-15T14:34:50.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952693+0000", 2023-01-15T14:34:50.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.074153067000000003 2023-01-15T14:34:50.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:34:50.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952735+0000", 2023-01-15T14:34:50.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1792999999999998e-05 2023-01-15T14:34:50.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065518+0000", 2023-01-15T14:34:50.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.11278334499999999 2023-01-15T14:34:50.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:50.551 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065687+0000", 2023-01-15T14:34:50.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016807700000000001 2023-01-15T14:34:50.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.551 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.551 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844786+0000", 2023-01-15T14:34:50.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.77909980300000004 2023-01-15T14:34:50.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844944+0000", 2023-01-15T14:34:50.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157424 2023-01-15T14:34:50.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.845089+0000", 2023-01-15T14:34:50.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144814 2023-01-15T14:34:50.553 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.553 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.553 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.553 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:16 3.13 3:ca393a9e:::smithi09025264-47:head [assert-version v193,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:34:50.553 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:34:50.553 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 270.95768722700001, 2023-01-15T14:34:50.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8970339479999998, 2023-01-15T14:34:50.554 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.554 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:50.554 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.674 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:34:50.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:34:50.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305751+0000", 2023-01-15T14:34:50.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:50.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305763+0000", 2023-01-15T14:34:50.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2309e-05 2023-01-15T14:34:50.677 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.677 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.677 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.677 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305788+0000", 2023-01-15T14:34:50.677 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4388000000000001e-05 2023-01-15T14:34:50.677 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.677 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.677 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.677 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305799+0000", 2023-01-15T14:34:50.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1304000000000001e-05 2023-01-15T14:34:50.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196425+0000", 2023-01-15T14:34:50.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8906259830000001 2023-01-15T14:34:50.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196673+0000", 2023-01-15T14:34:50.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024747499999999998 2023-01-15T14:34:50.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.202787+0000", 2023-01-15T14:34:50.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0061146509999999996 2023-01-15T14:34:50.679 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.679 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.679 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.679 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1480 3.13 3:cbfd2d71:::smithi09025264-37:head [write 581575~705262,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 270.92441949900001, 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.375425221, 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1480 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:34:50.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:34:50.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339019+0000", 2023-01-15T14:34:50.682 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:50.682 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.682 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339659+0000", 2023-01-15T14:34:50.682 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000639742 2023-01-15T14:34:50.682 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.682 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339690+0000", 2023-01-15T14:34:50.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1594000000000003e-05 2023-01-15T14:34:50.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339703+0000", 2023-01-15T14:34:50.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2828e-05 2023-01-15T14:34:50.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.510837+0000", 2023-01-15T14:34:50.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1711338759999999 2023-01-15T14:34:50.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511048+0000", 2023-01-15T14:34:50.684 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021057899999999999 2023-01-15T14:34:50.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:34:50.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511325+0000", 2023-01-15T14:34:50.684 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027690400000000002 2023-01-15T14:34:50.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:50.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512092+0000", 2023-01-15T14:34:50.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00076720800000000002 2023-01-15T14:34:50.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:50.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512113+0000", 2023-01-15T14:34:50.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0917e-05 2023-01-15T14:34:50.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512439+0000", 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032636299999999998 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512871+0000", 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000432087 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.514165+0000", 2023-01-15T14:34:50.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001293726 2023-01-15T14:34:50.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714137+0000", 2023-01-15T14:34:50.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19997229499999999 2023-01-15T14:34:50.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:50.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714233+0000", 2023-01-15T14:34:50.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6062000000000003e-05 2023-01-15T14:34:50.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714446+0000", 2023-01-15T14:34:50.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212952 2023-01-15T14:34:50.688 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.688 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.688 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.689 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1481 3.13 3:cbfd2d71:::smithi09025264-37:head [write 1733272~707409] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:34:50.689 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:34:50.689 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 270.917570216, 2023-01-15T14:34:50.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.572522502, 2023-01-15T14:34:50.689 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.689 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:50.689 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.689 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.689 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.689 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1481 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345868+0000", 2023-01-15T14:34:50.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:50.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346316+0000", 2023-01-15T14:34:50.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044813399999999999 2023-01-15T14:34:50.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346348+0000", 2023-01-15T14:34:50.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:34:50.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346362+0000", 2023-01-15T14:34:50.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.375e-05 2023-01-15T14:34:50.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512276+0000", 2023-01-15T14:34:50.693 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1659136829999999 2023-01-15T14:34:50.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.693 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.693 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:50.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512427+0000", 2023-01-15T14:34:50.693 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015088699999999999 2023-01-15T14:34:50.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.693 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.693 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714565+0000", 2023-01-15T14:34:50.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20213841599999999 2023-01-15T14:34:50.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.694 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714732+0000", 2023-01-15T14:34:50.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016676700000000001 2023-01-15T14:34:50.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:34:50.694 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714964+0000", 2023-01-15T14:34:50.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000231678 2023-01-15T14:34:50.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:50.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715499+0000", 2023-01-15T14:34:50.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053513300000000003 2023-01-15T14:34:50.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:50.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715523+0000", 2023-01-15T14:34:50.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4266000000000001e-05 2023-01-15T14:34:50.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.696 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:50.696 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715867+0000", 2023-01-15T14:34:50.696 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034419399999999998 2023-01-15T14:34:50.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.696 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.696 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.816932+0000", 2023-01-15T14:34:50.696 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10106498999999999 2023-01-15T14:34:50.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.697 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:50.697 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906374+0000", 2023-01-15T14:34:50.697 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089441442999999995 2023-01-15T14:34:50.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.697 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.697 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918050+0000", 2023-01-15T14:34:50.697 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.011676677999999999 2023-01-15T14:34:50.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918218+0000", 2023-01-15T14:34:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016802000000000001 2023-01-15T14:34:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918393+0000", 2023-01-15T14:34:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017447 2023-01-15T14:34:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.698 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.699 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.699 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.699 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1482 3.13 3:cbfd2d71:::smithi09025264-37:head [write 3195938~437497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:34:50.699 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:34:50.699 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 270.91113435800003, 2023-01-15T14:34:50.699 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5664780289999998, 2023-01-15T14:34:50.699 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.699 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:50.699 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.699 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.699 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.700 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1482 2023-01-15T14:34:50.700 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.700 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.700 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.700 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.700 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:34:50.700 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.700 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.700 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.700 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.701 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:34:50.701 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.701 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.701 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.701 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.701 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352275+0000", 2023-01-15T14:34:50.701 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999695 2023-01-15T14:34:50.701 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.701 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.701 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352644+0000", 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036870300000000001 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352676+0000", 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1726000000000001e-05 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352689+0000", 2023-01-15T14:34:50.702 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3617e-05 2023-01-15T14:34:50.703 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.703 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.703 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512602+0000", 2023-01-15T14:34:50.703 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1599126690000001 2023-01-15T14:34:50.703 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.703 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.703 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:50.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512839+0000", 2023-01-15T14:34:50.703 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023676900000000001 2023-01-15T14:34:50.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.704 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.704 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715649+0000", 2023-01-15T14:34:50.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20281044600000001 2023-01-15T14:34:50.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.704 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.704 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715777+0000", 2023-01-15T14:34:50.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012738800000000001 2023-01-15T14:34:50.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.705 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.705 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:34:50.705 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715995+0000", 2023-01-15T14:34:50.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000217858 2023-01-15T14:34:50.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.705 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.705 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:50.705 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716694+0000", 2023-01-15T14:34:50.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00069956700000000001 2023-01-15T14:34:50.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:50.706 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716709+0000", 2023-01-15T14:34:50.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4362000000000001e-05 2023-01-15T14:34:50.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:50.706 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716936+0000", 2023-01-15T14:34:50.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022744200000000001 2023-01-15T14:34:50.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.818876+0000", 2023-01-15T14:34:50.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101939635 2023-01-15T14:34:50.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:50.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906390+0000", 2023-01-15T14:34:50.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087513987000000001 2023-01-15T14:34:50.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918577+0000", 2023-01-15T14:34:50.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012187066 2023-01-15T14:34:50.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:50.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918697+0000", 2023-01-15T14:34:50.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120326 2023-01-15T14:34:50.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918784+0000", 2023-01-15T14:34:50.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7200999999999994e-05 2023-01-15T14:34:50.709 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.709 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.709 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.709 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1483 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3781575] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:34:50.709 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:34:50.709 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 270.91072620599999, 2023-01-15T14:34:50.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5663522429999999, 2023-01-15T14:34:50.709 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.710 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:50.710 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.710 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.710 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.710 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1483 2023-01-15T14:34:50.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.710 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:34:50.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:34:50.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352713+0000", 2023-01-15T14:34:50.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:50.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352720+0000", 2023-01-15T14:34:50.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7959999999999998e-06 2023-01-15T14:34:50.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352735+0000", 2023-01-15T14:34:50.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.455e-05 2023-01-15T14:34:50.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352741+0000", 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8360000000000003e-06 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512901+0000", 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1601594980000001 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:50.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.513067+0000", 2023-01-15T14:34:50.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016626 2023-01-15T14:34:50.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716977+0000", 2023-01-15T14:34:50.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20391002799999999 2023-01-15T14:34:50.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717146+0000", 2023-01-15T14:34:50.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168851 2023-01-15T14:34:50.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:34:50.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717467+0000", 2023-01-15T14:34:50.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032135600000000002 2023-01-15T14:34:50.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:50.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717800+0000", 2023-01-15T14:34:50.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033217400000000002 2023-01-15T14:34:50.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:50.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717822+0000", 2023-01-15T14:34:50.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2308999999999999e-05 2023-01-15T14:34:50.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:50.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717970+0000", 2023-01-15T14:34:50.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014788099999999999 2023-01-15T14:34:50.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.819135+0000", 2023-01-15T14:34:50.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116478 2023-01-15T14:34:50.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:50.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906408+0000", 2023-01-15T14:34:50.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087272902999999999 2023-01-15T14:34:50.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.718 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918887+0000", 2023-01-15T14:34:50.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012479965000000001 2023-01-15T14:34:50.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.718 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:50.718 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918959+0000", 2023-01-15T14:34:50.718 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1978999999999996e-05 2023-01-15T14:34:50.718 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.718 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.919067+0000", 2023-01-15T14:34:50.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107083 2023-01-15T14:34:50.719 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.719 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.719 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.719 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1485 3.d 3:bc1c6b44:::smithi09025264-23:head [write 598382~780511,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:34:50.719 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:34:50.719 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 270.90325577200002, 2023-01-15T14:34:50.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5474121890000001, 2023-01-15T14:34:50.720 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.720 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:50.720 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.720 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.720 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.720 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1485 2023-01-15T14:34:50.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.720 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.721 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360168+0000", 2023-01-15T14:34:50.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999828 2023-01-15T14:34:50.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360790+0000", 2023-01-15T14:34:50.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062269099999999998 2023-01-15T14:34:50.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360822+0000", 2023-01-15T14:34:50.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1903000000000003e-05 2023-01-15T14:34:50.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360836+0000", 2023-01-15T14:34:50.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3380000000000001e-05 2023-01-15T14:34:50.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096664+0000", 2023-01-15T14:34:50.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7358282320000002 2023-01-15T14:34:50.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:50.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096683+0000", 2023-01-15T14:34:50.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9063999999999999e-05 2023-01-15T14:34:50.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704561+0000", 2023-01-15T14:34:50.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60787808399999999 2023-01-15T14:34:50.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704846+0000", 2023-01-15T14:34:50.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028527400000000002 2023-01-15T14:34:50.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:50.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705135+0000", 2023-01-15T14:34:50.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028875400000000001 2023-01-15T14:34:50.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:50.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705735+0000", 2023-01-15T14:34:50.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059956899999999999 2023-01-15T14:34:50.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:50.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705761+0000", 2023-01-15T14:34:50.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:34:50.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:50.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.706129+0000", 2023-01-15T14:34:50.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036822299999999999 2023-01-15T14:34:50.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:50.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906331+0000", 2023-01-15T14:34:50.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200202091 2023-01-15T14:34:50.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907111+0000", 2023-01-15T14:34:50.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077990799999999995 2023-01-15T14:34:50.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907379+0000", 2023-01-15T14:34:50.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026804800000000001 2023-01-15T14:34:50.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:50.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907529+0000", 2023-01-15T14:34:50.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015022700000000001 2023-01-15T14:34:50.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907597+0000", 2023-01-15T14:34:50.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7785000000000002e-05 2023-01-15T14:34:50.729 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.729 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.729 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.730 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1486 3.d 3:bc1c6b44:::smithi09025264-23:head [write 2075401~451067] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:34:50.730 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:34:50.730 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 270.89685972400002, 2023-01-15T14:34:50.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6461318330000001, 2023-01-15T14:34:50.730 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.730 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:50.730 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.730 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.730 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.731 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1486 2023-01-15T14:34:50.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.731 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:34:50.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:34:50.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366579+0000", 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366845+0000", 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026610499999999999 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366878+0000", 2023-01-15T14:34:50.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2750000000000003e-05 2023-01-15T14:34:50.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.844 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.844 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.467009+0000", 2023-01-15T14:34:50.844 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100131071 2023-01-15T14:34:50.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.845 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.845 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096793+0000", 2023-01-15T14:34:50.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6297846869999999 2023-01-15T14:34:50.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:50.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096806+0000", 2023-01-15T14:34:50.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3077e-05 2023-01-15T14:34:50.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.846 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.846 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907957+0000", 2023-01-15T14:34:50.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81115055400000002 2023-01-15T14:34:50.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908229+0000", 2023-01-15T14:34:50.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027187499999999998 2023-01-15T14:34:50.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:34:50.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908496+0000", 2023-01-15T14:34:50.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026754200000000001 2023-01-15T14:34:50.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:50.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909025+0000", 2023-01-15T14:34:50.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052865200000000001 2023-01-15T14:34:50.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:50.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909046+0000", 2023-01-15T14:34:50.848 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0713e-05 2023-01-15T14:34:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909294+0000", 2023-01-15T14:34:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024864899999999998 2023-01-15T14:34:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.911607+0000", 2023-01-15T14:34:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023128939999999998 2023-01-15T14:34:50.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012076+0000", 2023-01-15T14:34:50.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100468293 2023-01-15T14:34:50.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:50.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012426+0000", 2023-01-15T14:34:50.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035044299999999998 2023-01-15T14:34:50.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.851 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:50.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012532+0000", 2023-01-15T14:34:50.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106142 2023-01-15T14:34:50.851 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.851 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.851 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012713+0000", 2023-01-15T14:34:50.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018048599999999999 2023-01-15T14:34:50.851 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.851 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.851 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.852 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.852 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.852 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1496 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:34:50.852 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:34:50.852 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 270.52721687500002, 2023-01-15T14:34:50.852 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870616037, 2023-01-15T14:34:50.852 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.852 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:50.852 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.852 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.852 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:50.853 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1496 2023-01-15T14:34:50.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.853 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:50.853 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:50.853 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:34:50.853 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.853 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:50.853 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:34:50.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:50.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:50.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736220+0000", 2023-01-15T14:34:50.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:50.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:50.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736238+0000", 2023-01-15T14:34:50.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7156e-05 2023-01-15T14:34:50.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:50.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736275+0000", 2023-01-15T14:34:50.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7876000000000001e-05 2023-01-15T14:34:50.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:50.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736290+0000", 2023-01-15T14:34:50.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4449e-05 2023-01-15T14:34:50.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:50.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602217+0000", 2023-01-15T14:34:50.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8659270299999999 2023-01-15T14:34:50.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:50.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602468+0000", 2023-01-15T14:34:50.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000251315 2023-01-15T14:34:50.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.857 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:50.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.606840+0000", 2023-01-15T14:34:50.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0043715009999999999 2023-01-15T14:34:50.857 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.857 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:50.857 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:50.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:50.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:50.857 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1497 3.11 3:8889e450:::smithi09025264-32:head [read 0~2991646,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:34:50.858 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:34:50.858 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 270.52712942099998, 2023-01-15T14:34:50.858 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738359249999998, 2023-01-15T14:34:50.858 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:50.858 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:50.858 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:50.858 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:50.858 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:51.002 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1497 2023-01-15T14:34:51.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.002 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:51.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:51.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:34:51.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:51.003 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:34:51.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:51.003 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736310+0000", 2023-01-15T14:34:51.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:51.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736317+0000", 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7240000000000003e-06 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736334+0000", 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6878999999999999e-05 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.004 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:51.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736341+0000", 2023-01-15T14:34:51.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0129999999999996e-06 2023-01-15T14:34:51.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:51.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607070+0000", 2023-01-15T14:34:51.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870728873 2023-01-15T14:34:51.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:51.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607226+0000", 2023-01-15T14:34:51.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015595 2023-01-15T14:34:51.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.006 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:51.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610147+0000", 2023-01-15T14:34:51.006 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.002921416 2023-01-15T14:34:51.006 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.006 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:51.006 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.007 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1498 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:34:51.007 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:34:51.007 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 270.52708931500001, 2023-01-15T14:34:51.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8752667839999999, 2023-01-15T14:34:51.007 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:51.007 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:51.007 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:51.007 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:51.007 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:51.007 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1498 2023-01-15T14:34:51.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.008 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:51.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.008 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:51.008 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:34:51.008 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.008 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.008 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.008 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:51.008 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:34:51.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:51.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736350+0000", 2023-01-15T14:34:51.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:51.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:51.009 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736355+0000", 2023-01-15T14:34:51.009 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4900000000000002e-06 2023-01-15T14:34:51.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:51.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736363+0000", 2023-01-15T14:34:51.010 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2770000000000005e-06 2023-01-15T14:34:51.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:51.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736369+0000", 2023-01-15T14:34:51.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7400000000000001e-06 2023-01-15T14:34:51.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:51.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610234+0000", 2023-01-15T14:34:51.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738651439999998 2023-01-15T14:34:51.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:51.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610402+0000", 2023-01-15T14:34:51.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168283 2023-01-15T14:34:51.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:51.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.611618+0000", 2023-01-15T14:34:51.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001215647 2023-01-15T14:34:51.012 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.012 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:51.013 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.013 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.7.0:440 3.11 e231/207 ondisk, result = 0)", 2023-01-15T14:34:51.013 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:34:51.013 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 172.01930765700001, 2023-01-15T14:34:51.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973962139999998, 2023-01-15T14:34:51.013 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:51.013 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:51.014 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:51.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:51.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:34:51.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:51.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:34:51.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:51.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244132+0000", 2023-01-15T14:34:51.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:51.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:51.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244143+0000", 2023-01-15T14:34:51.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1647e-05 2023-01-15T14:34:51.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:51.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244165+0000", 2023-01-15T14:34:51.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1501999999999998e-05 2023-01-15T14:34:51.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:51.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244180+0000", 2023-01-15T14:34:51.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5434000000000002e-05 2023-01-15T14:34:51.017 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.017 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.017 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:51.017 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941489+0000", 2023-01-15T14:34:51.017 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973091899999999 2023-01-15T14:34:51.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:51.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941503+0000", 2023-01-15T14:34:51.018 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.414e-05 2023-01-15T14:34:51.018 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.018 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.018 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:51.018 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941529+0000", 2023-01-15T14:34:51.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5622999999999998e-05 2023-01-15T14:34:51.019 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.019 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:51.019 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.019 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.019 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.4.0:562 3.1b e231/224 3:d85f42d9:::smithi09025264-45:a8 v 231'66, mlcod=225'62)", 2023-01-15T14:34:51.019 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:34:51.019 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 171.82895646099999, 2023-01-15T14:34:51.019 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2578820149999999, 2023-01-15T14:34:51.019 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:51.020 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:51.020 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:51.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:51.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:34:51.020 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.020 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:51.020 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:34:51.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.021 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:51.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:34:51.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:51.021 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:51.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434492+0000", 2023-01-15T14:34:51.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4360000000000002e-06 2023-01-15T14:34:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434517+0000", 2023-01-15T14:34:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.48e-05 2023-01-15T14:34:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434530+0000", 2023-01-15T14:34:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2982999999999999e-05 2023-01-15T14:34:51.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:51.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434663+0000", 2023-01-15T14:34:51.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013307 2023-01-15T14:34:51.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:51.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434700+0000", 2023-01-15T14:34:51.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7020000000000001e-05 2023-01-15T14:34:51.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:51.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435438+0000", 2023-01-15T14:34:51.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073808900000000004 2023-01-15T14:34:51.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:51.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435456+0000", 2023-01-15T14:34:51.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8091999999999999e-05 2023-01-15T14:34:51.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435526+0000", 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9806000000000002e-05 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435570+0000", 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4091999999999997e-05 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:51.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.692366+0000", 2023-01-15T14:34:51.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2567961539999999 2023-01-15T14:34:51.026 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.026 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:51.026 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.026 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:24 3.d 3:b3f111b5:::smithi09025264-48:head [assert-version v160,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e232)", 2023-01-15T14:34:51.026 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:34:51.026 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 171.31312024499999, 2023-01-15T14:34:51.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9963031440000001, 2023-01-15T14:34:51.027 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:51.027 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:51.027 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:51.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:51.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:34:51.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:51.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:34:51.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:51.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950318+0000", 2023-01-15T14:34:51.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:51.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:51.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950333+0000", 2023-01-15T14:34:51.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.467e-05 2023-01-15T14:34:51.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:51.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950368+0000", 2023-01-15T14:34:51.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4895999999999999e-05 2023-01-15T14:34:51.029 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:51.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050461+0000", 2023-01-15T14:34:51.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10009304400000001 2023-01-15T14:34:51.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:51.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050593+0000", 2023-01-15T14:34:51.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000132764 2023-01-15T14:34:51.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.030 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:34:51.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050615+0000", 2023-01-15T14:34:51.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1483000000000001e-05 2023-01-15T14:34:51.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:51.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945049+0000", 2023-01-15T14:34:51.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.894434138 2023-01-15T14:34:51.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:51.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945326+0000", 2023-01-15T14:34:51.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000277368 2023-01-15T14:34:51.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:51.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.946623+0000", 2023-01-15T14:34:51.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012970340000000001 2023-01-15T14:34:51.032 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.032 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:51.032 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.032 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.3.0:547 3.22 e232/180 3:47dfe935:::smithi09025264-29:81 v 232'124, mlcod=229'121)", 2023-01-15T14:34:51.032 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:34:51.032 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 171.21289049699999, 2023-01-15T14:34:51.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.643153431, 2023-01-15T14:34:51.033 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:51.033 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:51.033 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:51.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:51.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:34:51.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:51.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:34:51.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:51.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050549+0000", 2023-01-15T14:34:51.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:51.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:51.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050561+0000", 2023-01-15T14:34:51.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.278e-05 2023-01-15T14:34:51.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:51.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050583+0000", 2023-01-15T14:34:51.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1053e-05 2023-01-15T14:34:51.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:51.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050596+0000", 2023-01-15T14:34:51.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3946e-05 2023-01-15T14:34:51.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.036 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:51.036 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050708+0000", 2023-01-15T14:34:51.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011129 2023-01-15T14:34:51.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.036 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:51.036 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050750+0000", 2023-01-15T14:34:51.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2725000000000002e-05 2023-01-15T14:34:51.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.037 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:51.037 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051324+0000", 2023-01-15T14:34:51.037 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00057335700000000003 2023-01-15T14:34:51.037 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.037 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:51.037 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051340+0000", 2023-01-15T14:34:51.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5968999999999999e-05 2023-01-15T14:34:51.038 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.038 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:51.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051405+0000", 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5419000000000005e-05 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051471+0000", 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5671000000000002e-05 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.693703+0000", 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.642232554 2023-01-15T14:34:51.369 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3881 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e392)", 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:34:42.155799+0000", 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 8.1076416760000001, 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1132586929999999, 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:51.370 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3881 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155799+0000", 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155799+0000", 2023-01-15T14:34:51.371 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:51.372 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.372 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.372 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:51.372 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155798+0000", 2023-01-15T14:34:51.372 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:51.372 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.372 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.372 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:51.372 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155804+0000", 2023-01-15T14:34:51.372 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4969999999999998e-06 2023-01-15T14:34:51.372 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:51.373 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155820+0000", 2023-01-15T14:34:51.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5387999999999999e-05 2023-01-15T14:34:51.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.373 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:51.373 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155827+0000", 2023-01-15T14:34:51.373 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9759999999999998e-06 2023-01-15T14:34:51.373 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.373 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.651043+0000", 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49521586499999998 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.651233+0000", 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000190259 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:51.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:43.268845+0000", 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61761231400000005 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:43.268939+0000", 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4122000000000004e-05 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:43.269058+0000", 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011818400000000001 2023-01-15T14:34:51.375 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.376 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:51.376 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.376 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:51.376 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:51.376 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:51.376 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:50.358+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:51.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:34:51.377 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:34:51.378 INFO:tasks.rados.rados.0.smithi090.stdout:1510: expect (ObjNum 406 snap 108 seq_num 406) 2023-01-15T14:34:51.378 INFO:tasks.rados.rados.0.smithi090.stdout:1510: done (4 left) 2023-01-15T14:34:51.379 INFO:tasks.rados.rados.0.smithi090.stdout:1514: done (3 left) 2023-01-15T14:34:51.379 INFO:tasks.rados.rados.0.smithi090.stdout:1515: rollback oid 44 current snap is 166 2023-01-15T14:34:51.379 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 44 to 126 2023-01-15T14:34:51.379 INFO:tasks.rados.rados.0.smithi090.stdout:1511: expect (ObjNum 492 snap 144 seq_num 492) 2023-01-15T14:34:51.379 INFO:tasks.rados.rados.0.smithi090.stdout:1512: finishing write tid 1 to smithi09025264-18 2023-01-15T14:34:51.379 INFO:tasks.rados.rados.0.smithi090.stdout:1509: expect (ObjNum 471 snap 136 seq_num 471) 2023-01-15T14:34:51.380 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:51.374+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:51.472 INFO:tasks.rados.rados.0.smithi090.stdout:1512: finishing write tid 2 to smithi09025264-18 2023-01-15T14:34:51.532 INFO:tasks.rados.rados.0.smithi090.stdout:1512: finishing write tid 3 to smithi09025264-18 2023-01-15T14:34:51.532 INFO:tasks.rados.rados.0.smithi090.stdout:1512: finishing write tid 4 to smithi09025264-18 2023-01-15T14:34:51.532 INFO:tasks.rados.rados.0.smithi090.stdout:1509: done (3 left) 2023-01-15T14:34:51.533 INFO:tasks.rados.rados.0.smithi090.stdout:1511: done (2 left) 2023-01-15T14:34:51.533 INFO:tasks.rados.rados.0.smithi090.stdout:1516: snap_remove snap 139 2023-01-15T14:34:51.534 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:34:51.534 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:34:51.534 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:51.535 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:34:51.535 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:51.535 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:51.536 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:51.527+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:51.545 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:51.563 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:51.599 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:51.684 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:51.684 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:51.685 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:51.685 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:51.686 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:34:51.686 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:34:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:34:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:34:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:34:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:34:51.687 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:34:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:34:51.688 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:34:51.689 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:51.682+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:51.787 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:51.783+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:51.873 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:51.873 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:51.874 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:34:51.919 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:51.915+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:52.197 INFO:tasks.rados.rados.0.smithi090.stdout:1516: done (2 left) 2023-01-15T14:34:52.198 INFO:tasks.rados.rados.0.smithi090.stdout:1517: rollback oid 22 current snap is 166 2023-01-15T14:34:52.198 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 22 to 119 2023-01-15T14:34:52.198 INFO:tasks.rados.rados.0.smithi090.stdout:1512: finishing write tid 5 to smithi09025264-18 2023-01-15T14:34:52.198 INFO:tasks.rados.rados.0.smithi090.stdout:1512: oid 18 updating version 0 to 242 2023-01-15T14:34:52.198 INFO:tasks.rados.rados.0.smithi090.stdout:1512: oid 18 updating version 242 to 243 2023-01-15T14:34:52.198 INFO:tasks.rados.rados.0.smithi090.stdout:1512: oid 18 updating version 243 to 245 2023-01-15T14:34:52.198 INFO:tasks.rados.rados.0.smithi090.stdout:1512: oid 18 version 245 is already newer than 244 2023-01-15T14:34:52.198 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 245 (ObjNum 537 snap 166 seq_num 537) dirty exists 2023-01-15T14:34:52.198 INFO:tasks.rados.rados.0.smithi090.stdout:1512: left oid 18 (ObjNum 537 snap 166 seq_num 537) 2023-01-15T14:34:52.198 INFO:tasks.rados.rados.0.smithi090.stdout:1515: finishing rollback tid 1 to smithi09025264-44 2023-01-15T14:34:52.198 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 350 (ObjNum 305 snap 0 seq_num 0) dirty dne 2023-01-15T14:34:52.199 INFO:tasks.rados.rados.0.smithi090.stdout:1512: done (2 left) 2023-01-15T14:34:52.199 INFO:tasks.rados.rados.0.smithi090.stdout:1515: done (1 left) 2023-01-15T14:34:52.199 INFO:tasks.rados.rados.0.smithi090.stdout:1518: snap_create 2023-01-15T14:34:52.199 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:52.200 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:52.200 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:52.200 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:34:52.201 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:52.194+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:52.299 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:52.294+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:52.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:34:52.380 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:52.380 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:52.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:34:52.381 INFO:tasks.rados.rados.0.smithi090.stdout:1517: finishing rollback tid 0 to smithi09025264-22 2023-01-15T14:34:52.429 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:52.423+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:52.542 INFO:tasks.rados.rados.0.smithi090.stdout:1517: finishing rollback tid 1 to smithi09025264-22 2023-01-15T14:34:52.542 INFO:tasks.rados.rados.0.smithi090.stdout:1517: finishing rollback tid 2 to smithi09025264-22 2023-01-15T14:34:52.542 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 399 (ObjNum 429 snap 116 seq_num 429) dirty exists 2023-01-15T14:34:52.542 INFO:tasks.rados.rados.0.smithi090.stdout:1517: done (0 left) 2023-01-15T14:34:52.543 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:52.543 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:52.543 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:52.538+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:52.551 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:34:52.642 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:52.638+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:52.707 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:52.707 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:52.707 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:34:52.743 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:52.738+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:52.896 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:52.897 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:52.897 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:34:52.897 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:52.890+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:52.996 INFO:tasks.rados.rados.0.smithi090.stdout:1518: done (0 left) 2023-01-15T14:34:53.266 INFO:tasks.rados.rados.0.smithi090.stdout:1519: snap_create 2023-01-15T14:34:53.267 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:53.268 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:53.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:34:53.269 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:53.262+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:53.368 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:53.363+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:53.379 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:53.424 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:53.425 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:34:53.470 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:53.466+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:53.579 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:53.579 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:53.579 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:53.574+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:53.587 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:34:53.680 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:53.675+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:53.741 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:34:53.742 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:53.742 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:53.743 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:34:53.789 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:53.786+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:54.096 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:54.096 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:54.096 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:54.096 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:54.096 INFO:tasks.rados.rados.0.smithi090.stdout:1519: done (0 left) 2023-01-15T14:34:54.097 INFO:tasks.rados.rados.0.smithi090.stdout:1520: delete oid 5 current snap is 168 2023-01-15T14:34:54.097 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:54.097 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:54.098 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:34:54.099 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:54.099 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:54.099 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:54.099 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:54.100 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:34:54.237 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:34:54.237 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:34:54.237 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:34:54.237 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:34:54.237 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:34:54.237 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:34:54.237 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:34:54.238 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:54.091+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:54.239 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:54.235+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:54.296 INFO:tasks.rados.rados.0.smithi090.stdout:1520: done (0 left) 2023-01-15T14:34:54.390 INFO:tasks.rados.rados.0.smithi090.stdout:1521: snap_create 2023-01-15T14:34:54.391 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:54.391 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:54.391 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:34:54.391 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:34:54.391 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:54.392 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:34:54.392 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:54.387+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:54.492 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:54.485+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:54.508 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:54.509 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:34:54.509 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:34:54.509 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:34:54.509 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.509 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3 3.3 3:ca59e2b0:::smithi09025264-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:54.509 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:54.509 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 388.72808188499999, 2023-01-15T14:34:54.509 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62968594499999997, 2023-01-15T14:34:54.509 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.509 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:54.510 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.510 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.510 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.510 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3 2023-01-15T14:34:54.510 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.510 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.510 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.510 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.510 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:54.510 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.511 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.511 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.511 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.511 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:34:54.511 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.511 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.512 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.512 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.512 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775131+0000", 2023-01-15T14:34:54.512 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:54.512 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.512 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.512 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.512 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775138+0000", 2023-01-15T14:34:54.513 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:34:54.513 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.513 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.513 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.513 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775148+0000", 2023-01-15T14:34:54.513 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7969999999999995e-06 2023-01-15T14:34:54.513 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.513 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.513 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.513 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775161+0000", 2023-01-15T14:34:54.513 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2469999999999999e-05 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.857974+0000", 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.082812887000000002 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.858147+0000", 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000173883 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.514 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106098+0000", 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24795019600000001 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106181+0000", 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3248e-05 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:34:54.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106330+0000", 2023-01-15T14:34:54.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014918200000000001 2023-01-15T14:34:54.516 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.516 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.516 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:54.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106510+0000", 2023-01-15T14:34:54.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017979399999999999 2023-01-15T14:34:54.516 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.516 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.516 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:54.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106527+0000", 2023-01-15T14:34:54.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6643999999999999e-05 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106605+0000", 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8650000000000001e-05 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106644+0000", 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8686999999999997e-05 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404687+0000", 2023-01-15T14:34:54.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.298042745 2023-01-15T14:34:54.518 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404735+0000", 2023-01-15T14:34:54.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8306000000000001e-05 2023-01-15T14:34:54.518 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404772+0000", 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6977000000000002e-05 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404818+0000", 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.6439e-05 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.520 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:54.520 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:54.520 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 388.728043818, 2023-01-15T14:34:54.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62995383500000002, 2023-01-15T14:34:54.555 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.556 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:54.556 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.556 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.556 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.556 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4 2023-01-15T14:34:54.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.557 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:54.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.558 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.558 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.558 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:54.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:34:54.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:54.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775174+0000", 2023-01-15T14:34:54.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3669999999999999e-06 2023-01-15T14:34:54.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775184+0000", 2023-01-15T14:34:54.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5869999999999999e-06 2023-01-15T14:34:54.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775190+0000", 2023-01-15T14:34:54.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:34:54.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868440+0000", 2023-01-15T14:34:54.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.093250126000000003 2023-01-15T14:34:54.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868555+0000", 2023-01-15T14:34:54.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115003 2023-01-15T14:34:54.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.302962+0000", 2023-01-15T14:34:54.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.43440729700000003 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303197+0000", 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023508000000000001 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404866+0000", 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101668723 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404985+0000", 2023-01-15T14:34:54.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118853 2023-01-15T14:34:54.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.405124+0000", 2023-01-15T14:34:54.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013951999999999999 2023-01-15T14:34:54.564 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.564 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.565 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.565 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:144 3.9 3:9988ac9d:::smithi09025264-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:54.565 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:54.565 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 387.63670527199997, 2023-01-15T14:34:54.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63423976199999998, 2023-01-15T14:34:54.565 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.565 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:54.565 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.565 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.566 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.566 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 144 2023-01-15T14:34:54.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.566 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:54.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:34:54.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866508+0000", 2023-01-15T14:34:54.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:54.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866512+0000", 2023-01-15T14:34:54.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5999999999999998e-06 2023-01-15T14:34:54.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866523+0000", 2023-01-15T14:34:54.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0740000000000001e-05 2023-01-15T14:34:54.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866531+0000", 2023-01-15T14:34:54.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3310000000000004e-06 2023-01-15T14:34:54.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867744+0000", 2023-01-15T14:34:54.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012127399999999999 2023-01-15T14:34:54.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867810+0000", 2023-01-15T14:34:54.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6506000000000003e-05 2023-01-15T14:34:54.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097225+0000", 2023-01-15T14:34:54.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.22941514900000001 2023-01-15T14:34:54.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097295+0000", 2023-01-15T14:34:54.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:34:54.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500386+0000", 2023-01-15T14:34:54.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40309051699999998 2023-01-15T14:34:54.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500521+0000", 2023-01-15T14:34:54.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013561700000000001 2023-01-15T14:34:54.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500749+0000", 2023-01-15T14:34:54.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022754900000000001 2023-01-15T14:34:54.575 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.575 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.576 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.576 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:509 3.9 3:9f86bdcb:::smithi09025264-22:head [write 2873212~588997] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:54.576 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:54.576 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 364.68828092899997, 2023-01-15T14:34:54.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69623497099999998, 2023-01-15T14:34:54.576 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.577 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:54.577 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.577 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.577 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.577 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 509 2023-01-15T14:34:54.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.577 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:54.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:34:54.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814931+0000", 2023-01-15T14:34:54.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:54.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815512+0000", 2023-01-15T14:34:54.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058082200000000004 2023-01-15T14:34:54.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815553+0000", 2023-01-15T14:34:54.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1081999999999997e-05 2023-01-15T14:34:54.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815571+0000", 2023-01-15T14:34:54.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8028000000000001e-05 2023-01-15T14:34:54.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906452+0000", 2023-01-15T14:34:54.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090880586999999999 2023-01-15T14:34:54.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906564+0000", 2023-01-15T14:34:54.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111976 2023-01-15T14:34:54.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208424+0000", 2023-01-15T14:34:54.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.301860199 2023-01-15T14:34:54.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208718+0000", 2023-01-15T14:34:54.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029347299999999999 2023-01-15T14:34:54.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:54.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209175+0000", 2023-01-15T14:34:54.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045692399999999999 2023-01-15T14:34:54.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:54.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209725+0000", 2023-01-15T14:34:54.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055027600000000004 2023-01-15T14:34:54.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:54.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209751+0000", 2023-01-15T14:34:54.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5812999999999998e-05 2023-01-15T14:34:54.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:54.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.210006+0000", 2023-01-15T14:34:54.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025564099999999997 2023-01-15T14:34:54.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:54.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311247+0000", 2023-01-15T14:34:54.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10124119199999999 2023-01-15T14:34:54.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311454+0000", 2023-01-15T14:34:54.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020695800000000001 2023-01-15T14:34:54.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511022+0000", 2023-01-15T14:34:54.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19956729200000001 2023-01-15T14:34:54.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:54.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511100+0000", 2023-01-15T14:34:54.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8589000000000003e-05 2023-01-15T14:34:54.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511168+0000", 2023-01-15T14:34:54.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8039000000000004e-05 2023-01-15T14:34:54.592 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.592 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.592 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.592 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:510 3.9 3:9f86bdcb:::smithi09025264-22:head [setxattr _header (57) in=7b,truncate 3462209] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:54.593 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:54.593 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 364.68761746600001, 2023-01-15T14:34:54.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69609656399999997, 2023-01-15T14:34:54.593 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.593 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:54.593 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.594 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.594 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.594 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 510 2023-01-15T14:34:54.594 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.594 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.595 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.595 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:54.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.595 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.596 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:34:54.596 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815596+0000", 2023-01-15T14:34:54.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:54.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815606+0000", 2023-01-15T14:34:54.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7920000000000004e-06 2023-01-15T14:34:54.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.599 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.599 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815627+0000", 2023-01-15T14:34:54.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1239000000000001e-05 2023-01-15T14:34:54.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.600 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.600 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815637+0000", 2023-01-15T14:34:54.600 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0349e-05 2023-01-15T14:34:54.600 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.600 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.601 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.601 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906600+0000", 2023-01-15T14:34:54.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090963098000000006 2023-01-15T14:34:54.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.601 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.601 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.602 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906717+0000", 2023-01-15T14:34:54.602 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000116759 2023-01-15T14:34:54.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.602 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310000+0000", 2023-01-15T14:34:54.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40328322599999999 2023-01-15T14:34:54.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310249+0000", 2023-01-15T14:34:54.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024874400000000002 2023-01-15T14:34:54.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:34:54.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310559+0000", 2023-01-15T14:34:54.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310382 2023-01-15T14:34:54.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:54.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310951+0000", 2023-01-15T14:34:54.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039124500000000003 2023-01-15T14:34:54.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:54.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310975+0000", 2023-01-15T14:34:54.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4496000000000001e-05 2023-01-15T14:34:54.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:54.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311041+0000", 2023-01-15T14:34:54.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6309000000000005e-05 2023-01-15T14:34:54.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:54.608 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311260+0000", 2023-01-15T14:34:54.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021863399999999999 2023-01-15T14:34:54.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.609 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.609 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311588+0000", 2023-01-15T14:34:54.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032813599999999998 2023-01-15T14:34:54.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.610 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511351+0000", 2023-01-15T14:34:54.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19976275700000001 2023-01-15T14:34:54.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:54.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511507+0000", 2023-01-15T14:34:54.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015585100000000001 2023-01-15T14:34:54.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.612 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.612 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511693+0000", 2023-01-15T14:34:54.612 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186627 2023-01-15T14:34:54.612 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.613 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.613 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.613 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.613 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:511 3.9 3:9f86bdcb:::smithi09025264-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:34:54.613 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:54.614 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 364.68756009800001, 2023-01-15T14:34:54.614 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69666647500000001, 2023-01-15T14:34:54.614 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.614 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:54.614 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.614 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.615 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.615 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 511 2023-01-15T14:34:54.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.615 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.616 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.616 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:54.616 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.616 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.616 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.616 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.616 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:34:54.616 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.616 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815653+0000", 2023-01-15T14:34:54.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:54.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815660+0000", 2023-01-15T14:34:54.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0509999999999999e-06 2023-01-15T14:34:54.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815683+0000", 2023-01-15T14:34:54.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2742999999999999e-05 2023-01-15T14:34:54.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815698+0000", 2023-01-15T14:34:54.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5133999999999999e-05 2023-01-15T14:34:54.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908029+0000", 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.092331448999999996 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908193+0000", 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016342799999999999 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311115+0000", 2023-01-15T14:34:54.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40292222300000002 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311224+0000", 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010888899999999999 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511800+0000", 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20057596899999999 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512036+0000", 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236247 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512321+0000", 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028457599999999998 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.622 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:693 3.3 3:ca59e2b0:::smithi09025264-1:head [sparse-read 0~2016807] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:34:54.622 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:54.622 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 346.60482675100002, 2023-01-15T14:34:54.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70514199600000005, 2023-01-15T14:34:54.622 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.622 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:54.622 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.622 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.622 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.622 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 693 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.624 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898387+0000", 2023-01-15T14:34:54.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:54.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.624 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898390+0000", 2023-01-15T14:34:54.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.252e-06 2023-01-15T14:34:54.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898401+0000", 2023-01-15T14:34:54.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0977e-05 2023-01-15T14:34:54.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.998532+0000", 2023-01-15T14:34:54.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100130443 2023-01-15T14:34:54.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602663+0000", 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60413100500000005 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602817+0000", 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015440500000000001 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.603530+0000", 2023-01-15T14:34:54.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071246299999999999 2023-01-15T14:34:54.627 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.627 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.627 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.627 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1421 3.a 3:5919d83e:::smithi09025264-13:head [write 1723287~549075] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:54.627 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:54.627 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 279.590912242, 2023-01-15T14:34:54.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61118175699999999, 2023-01-15T14:34:54.627 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.627 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1421 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:34:54.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912300+0000", 2023-01-15T14:34:54.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:54.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913127+0000", 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00082704000000000002 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913158+0000", 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1146999999999998e-05 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913177+0000", 2023-01-15T14:34:54.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:34:54.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913384+0000", 2023-01-15T14:34:54.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000206589 2023-01-15T14:34:54.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:54.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913408+0000", 2023-01-15T14:34:54.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3493e-05 2023-01-15T14:34:54.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318579+0000", 2023-01-15T14:34:54.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40517134399999999 2023-01-15T14:34:54.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318806+0000", 2023-01-15T14:34:54.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022680899999999999 2023-01-15T14:34:54.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319060+0000", 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254069 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319527+0000", 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000467081 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:54.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319544+0000", 2023-01-15T14:34:54.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6949999999999999e-05 2023-01-15T14:34:54.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:54.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319778+0000", 2023-01-15T14:34:54.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023391500000000001 2023-01-15T14:34:54.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:54.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421126+0000", 2023-01-15T14:34:54.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101348626 2023-01-15T14:34:54.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.635 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522834+0000", 2023-01-15T14:34:54.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10170761 2023-01-15T14:34:54.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.635 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523142+0000", 2023-01-15T14:34:54.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854800000000001 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523251+0000", 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108411 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523484+0000", 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023292500000000001 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.636 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.637 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.637 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.637 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1422 3.a 3:5919d83e:::smithi09025264-13:head [write 2781837~703343] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:54.637 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:54.637 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 279.58997344599999, 2023-01-15T14:34:54.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61062001700000001, 2023-01-15T14:34:54.637 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.637 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:54.637 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.638 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.638 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.638 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1422 2023-01-15T14:34:54.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.638 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:54.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913239+0000", 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914075+0000", 2023-01-15T14:34:54.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00083594099999999998 2023-01-15T14:34:54.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914104+0000", 2023-01-15T14:34:54.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8938999999999999e-05 2023-01-15T14:34:54.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914121+0000", 2023-01-15T14:34:54.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7306999999999999e-05 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914234+0000", 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112737 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914257+0000", 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3156e-05 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.642 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319798+0000", 2023-01-15T14:34:54.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40554153100000001 2023-01-15T14:34:54.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.642 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319991+0000", 2023-01-15T14:34:54.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019217999999999999 2023-01-15T14:34:54.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.642 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:54.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320237+0000", 2023-01-15T14:34:54.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024621099999999998 2023-01-15T14:34:54.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:54.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320759+0000", 2023-01-15T14:34:54.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052174700000000003 2023-01-15T14:34:54.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:54.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320776+0000", 2023-01-15T14:34:54.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7173e-05 2023-01-15T14:34:54.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:54.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.321080+0000", 2023-01-15T14:34:54.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030424400000000001 2023-01-15T14:34:54.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:54.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421162+0000", 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100082085 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522963+0000", 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10180109399999999 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523576+0000", 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061277499999999997 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:54.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523664+0000", 2023-01-15T14:34:54.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8002000000000002e-05 2023-01-15T14:34:54.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523861+0000", 2023-01-15T14:34:54.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019685899999999999 2023-01-15T14:34:54.646 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.646 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.646 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1423 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 3630312] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 279.58905091100002, 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70862486899999999, 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.647 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1423 2023-01-15T14:34:54.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.648 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:54.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914162+0000", 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914172+0000", 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0267e-05 2023-01-15T14:34:54.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.650 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.650 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.650 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914188+0000", 2023-01-15T14:34:54.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6212999999999999e-05 2023-01-15T14:34:54.650 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.650 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.650 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.650 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914201+0000", 2023-01-15T14:34:54.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3073e-05 2023-01-15T14:34:54.650 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.651 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.651 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.651 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914315+0000", 2023-01-15T14:34:54.651 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011357399999999999 2023-01-15T14:34:54.651 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.651 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.651 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:54.651 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914334+0000", 2023-01-15T14:34:54.651 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9004e-05 2023-01-15T14:34:54.651 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.651 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.652 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.652 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421258+0000", 2023-01-15T14:34:54.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.506924606 2023-01-15T14:34:54.652 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.652 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.652 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.652 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421518+0000", 2023-01-15T14:34:54.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025950499999999999 2023-01-15T14:34:54.652 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.652 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.652 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:54.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421843+0000", 2023-01-15T14:34:54.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032498299999999999 2023-01-15T14:34:54.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:54.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422222+0000", 2023-01-15T14:34:54.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037867599999999999 2023-01-15T14:34:54.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:54.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422245+0000", 2023-01-15T14:34:54.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3096999999999999e-05 2023-01-15T14:34:54.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:54.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422343+0000", 2023-01-15T14:34:54.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8487999999999996e-05 2023-01-15T14:34:54.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:54.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522890+0000", 2023-01-15T14:34:54.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10054658900000001 2023-01-15T14:34:54.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.655 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523075+0000", 2023-01-15T14:34:54.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185386 2023-01-15T14:34:54.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.655 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.655 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622151+0000", 2023-01-15T14:34:54.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099075724000000004 2023-01-15T14:34:54.656 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.656 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.656 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:54.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622244+0000", 2023-01-15T14:34:54.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3387999999999994e-05 2023-01-15T14:34:54.656 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.656 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.656 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.656 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622788+0000", 2023-01-15T14:34:54.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000543973 2023-01-15T14:34:54.656 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.657 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.657 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.657 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.657 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.657 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1424 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:54.657 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:54.657 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 279.588995539, 2023-01-15T14:34:54.657 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70984700199999995, 2023-01-15T14:34:54.657 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.657 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:54.657 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1424 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.658 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.659 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:34:54.659 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.659 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.659 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.659 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.659 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914218+0000", 2023-01-15T14:34:54.659 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:54.659 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.659 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.659 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.659 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914224+0000", 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6769999999999998e-06 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914242+0000", 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7759000000000001e-05 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014331+0000", 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088819 2023-01-15T14:34:54.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114642+0000", 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100311027 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114682+0000", 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0028999999999999e-05 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523918+0000", 2023-01-15T14:34:54.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40923631500000002 2023-01-15T14:34:54.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524124+0000", 2023-01-15T14:34:54.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020524900000000001 2023-01-15T14:34:54.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622921+0000", 2023-01-15T14:34:54.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098797069000000001 2023-01-15T14:34:54.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.623328+0000", 2023-01-15T14:34:54.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000407539 2023-01-15T14:34:54.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.624066+0000", 2023-01-15T14:34:54.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073762899999999995 2023-01-15T14:34:54.664 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.664 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.664 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.664 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1430 3.a 3:52df5138:::smithi09025264-25:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:54.664 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:54.664 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 279.48883577800001, 2023-01-15T14:34:54.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81872653100000004, 2023-01-15T14:34:54.664 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.664 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1430 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:34:54.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014376+0000", 2023-01-15T14:34:54.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:54.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014391+0000", 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4922999999999999e-05 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014417+0000", 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5698999999999999e-05 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014435+0000", 2023-01-15T14:34:54.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7827000000000001e-05 2023-01-15T14:34:54.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114759+0000", 2023-01-15T14:34:54.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100324859 2023-01-15T14:34:54.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:54.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114771+0000", 2023-01-15T14:34:54.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:34:54.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524222+0000", 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40945100800000001 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524545+0000", 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000322367 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:54.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.626981+0000", 2023-01-15T14:34:54.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102436308 2023-01-15T14:34:54.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:54.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627874+0000", 2023-01-15T14:34:54.670 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089274899999999997 2023-01-15T14:34:54.670 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.670 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.670 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:54.670 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627930+0000", 2023-01-15T14:34:54.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6122000000000002e-05 2023-01-15T14:34:54.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628609+0000", 2023-01-15T14:34:54.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00067958700000000005 2023-01-15T14:34:54.671 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.671 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.671 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:54.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628798+0000", 2023-01-15T14:34:54.671 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018809 2023-01-15T14:34:54.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:54.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628870+0000", 2023-01-15T14:34:54.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2844999999999998e-05 2023-01-15T14:34:54.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.672 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.672 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.672 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832404+0000", 2023-01-15T14:34:54.672 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20353310299999999 2023-01-15T14:34:54.672 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832606+0000", 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020207999999999999 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833105+0000", 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049945399999999998 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.673 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.674 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.674 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1431 3.a 3:52df5138:::smithi09025264-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:34:54.674 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:54.674 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 279.48874448999999, 2023-01-15T14:34:54.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81967516600000001, 2023-01-15T14:34:54.674 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.674 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:54.674 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.674 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.674 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1431 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:34:54.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014469+0000", 2023-01-15T14:34:54.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:54.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014477+0000", 2023-01-15T14:34:54.677 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9300000000000003e-06 2023-01-15T14:34:54.677 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.677 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.677 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.677 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014495+0000", 2023-01-15T14:34:54.677 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8363999999999999e-05 2023-01-15T14:34:54.677 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.677 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.677 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.677 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114610+0000", 2023-01-15T14:34:54.677 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10011521700000001 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214948+0000", 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100337739 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214962+0000", 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4516e-05 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524874+0000", 2023-01-15T14:34:54.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30991196599999998 2023-01-15T14:34:54.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:34:54.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.525013+0000", 2023-01-15T14:34:54.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013873999999999999 2023-01-15T14:34:54.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628222+0000", 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103208721 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628504+0000", 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028245700000000001 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833214+0000", 2023-01-15T14:34:54.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.204709691 2023-01-15T14:34:54.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833555+0000", 2023-01-15T14:34:54.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034094000000000003 2023-01-15T14:34:54.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.834145+0000", 2023-01-15T14:34:54.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059006900000000003 2023-01-15T14:34:54.681 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2439 3.3 3:c11c7d81:::smithi09025264-11:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 179.41983998500001, 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70719632099999996, 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.682 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.683 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.683 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2439 2023-01-15T14:34:54.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.683 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:54.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:34:54.684 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083373+0000", 2023-01-15T14:34:54.684 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:54.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083380+0000", 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083391+0000", 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0366e-05 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083397+0000", 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2740000000000004e-06 2023-01-15T14:34:54.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083648+0000", 2023-01-15T14:34:54.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025052599999999999 2023-01-15T14:34:54.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:54.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083666+0000", 2023-01-15T14:34:54.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8848999999999999e-05 2023-01-15T14:34:54.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237725+0000", 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.154058361 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237945+0000", 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021978300000000001 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.386967+0000", 2023-01-15T14:34:54.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.149022562 2023-01-15T14:34:54.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387230+0000", 2023-01-15T14:34:54.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026256100000000002 2023-01-15T14:34:54.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.688 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:34:54.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387695+0000", 2023-01-15T14:34:54.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046480099999999998 2023-01-15T14:34:54.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.689 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:54.689 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388112+0000", 2023-01-15T14:34:54.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041713200000000002 2023-01-15T14:34:54.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.689 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:54.689 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388130+0000", 2023-01-15T14:34:54.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8315999999999999e-05 2023-01-15T14:34:54.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:54.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388189+0000", 2023-01-15T14:34:54.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8665999999999997e-05 2023-01-15T14:34:54.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:54.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388829+0000", 2023-01-15T14:34:54.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00063992699999999999 2023-01-15T14:34:54.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.489989+0000", 2023-01-15T14:34:54.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116059199999999 2023-01-15T14:34:54.719 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.719 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.719 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.719 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790300+0000", 2023-01-15T14:34:54.719 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30031108699999998 2023-01-15T14:34:54.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:54.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790396+0000", 2023-01-15T14:34:54.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5721000000000001e-05 2023-01-15T14:34:54.720 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.720 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.720 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.720 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790571+0000", 2023-01-15T14:34:54.720 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017464999999999999 2023-01-15T14:34:54.720 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2440 3.3 3:c11c7d81:::smithi09025264-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 179.41974825400001, 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70759506000000005, 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.721 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.722 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.722 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2440 2023-01-15T14:34:54.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.722 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.722 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:54.722 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.722 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.722 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.722 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:34:54.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083465+0000", 2023-01-15T14:34:54.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:54.723 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.723 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.723 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.723 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083470+0000", 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2630000000000003e-06 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083481+0000", 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0699999999999999e-05 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083488+0000", 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8870000000000003e-06 2023-01-15T14:34:54.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180331+0000", 2023-01-15T14:34:54.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.096843296999999995 2023-01-15T14:34:54.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:54.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180342+0000", 2023-01-15T14:34:54.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.031e-05 2023-01-15T14:34:54.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238012+0000", 2023-01-15T14:34:54.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.057670829 2023-01-15T14:34:54.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238257+0000", 2023-01-15T14:34:54.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000244959 2023-01-15T14:34:54.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590126+0000", 2023-01-15T14:34:54.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35186851699999999 2023-01-15T14:34:54.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590389+0000", 2023-01-15T14:34:54.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026276799999999998 2023-01-15T14:34:54.727 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.727 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790692+0000", 2023-01-15T14:34:54.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20030350299999999 2023-01-15T14:34:54.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790891+0000", 2023-01-15T14:34:54.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019893999999999999 2023-01-15T14:34:54.728 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.728 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.728 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.728 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.791061+0000", 2023-01-15T14:34:54.728 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169913 2023-01-15T14:34:54.729 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.729 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.729 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.729 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2450 3.a 3:52c3e133:::smithi09025264-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e230)", 2023-01-15T14:34:54.729 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:54.729 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 177.65589564199999, 2023-01-15T14:34:54.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.68251353100000001, 2023-01-15T14:34:54.729 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2450 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.730 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:34:54.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847318+0000", 2023-01-15T14:34:54.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:54.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847324+0000", 2023-01-15T14:34:54.732 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:34:54.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.732 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847339+0000", 2023-01-15T14:34:54.732 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5582000000000001e-05 2023-01-15T14:34:54.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.732 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847349+0000", 2023-01-15T14:34:54.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0139999999999999e-05 2023-01-15T14:34:54.733 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.733 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947644+0000", 2023-01-15T14:34:54.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10029455600000001 2023-01-15T14:34:54.733 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.733 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:54.734 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947657+0000", 2023-01-15T14:34:54.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2796000000000001e-05 2023-01-15T14:34:54.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.734 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.734 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125638+0000", 2023-01-15T14:34:54.734 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17798127799999999 2023-01-15T14:34:54.734 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.734 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.735 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125780+0000", 2023-01-15T14:34:54.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014173000000000001 2023-01-15T14:34:54.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.735 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.735 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.735 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529035+0000", 2023-01-15T14:34:54.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40325542199999997 2023-01-15T14:34:54.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.735 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.735 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.735 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529366+0000", 2023-01-15T14:34:54.736 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033111700000000002 2023-01-15T14:34:54.736 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.736 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.736 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529832+0000", 2023-01-15T14:34:54.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046592399999999999 2023-01-15T14:34:54.737 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.737 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.737 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.741 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3390 3.1b 3:d85f42d9:::smithi09025264-45:head [write 1944996~791497] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:54.741 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:54.741 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 79.443216933000002, 2023-01-15T14:34:54.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.611364087, 2023-01-15T14:34:54.742 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.742 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:54.742 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.742 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.743 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.743 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3390 2023-01-15T14:34:54.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.743 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.743 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.743 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:54.743 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.743 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.743 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.744 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.744 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:34:54.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.744 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.744 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.744 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.744 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059995+0000", 2023-01-15T14:34:54.744 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:54.744 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.744 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.744 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061164+0000", 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001168325 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061194+0000", 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.0485999999999999e-05 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061213+0000", 2023-01-15T14:34:54.745 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.891e-05 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159748+0000", 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098535175000000003 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159766+0000", 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263011+0000", 2023-01-15T14:34:54.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10324554699999999 2023-01-15T14:34:54.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263253+0000", 2023-01-15T14:34:54.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000241568 2023-01-15T14:34:54.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466464+0000", 2023-01-15T14:34:54.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.203211006 2023-01-15T14:34:54.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466638+0000", 2023-01-15T14:34:54.748 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017412100000000001 2023-01-15T14:34:54.748 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.748 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.748 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:54.748 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466911+0000", 2023-01-15T14:34:54.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000272971 2023-01-15T14:34:54.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:54.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467425+0000", 2023-01-15T14:34:54.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00051373 2023-01-15T14:34:54.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:54.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467457+0000", 2023-01-15T14:34:54.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2558999999999998e-05 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467906+0000", 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000448985 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469895+0000", 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0019888810000000001 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470611+0000", 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071556699999999996 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671178+0000", 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20056716699999999 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:54.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671277+0000", 2023-01-15T14:34:54.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9594999999999998e-05 2023-01-15T14:34:54.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671361+0000", 2023-01-15T14:34:54.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4048000000000006e-05 2023-01-15T14:34:54.752 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.752 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.753 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.753 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.753 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.753 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3391 3.1b 3:d85f42d9:::smithi09025264-45:head [write 3196560~554574] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:54.753 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:54.753 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 79.441973809999993, 2023-01-15T14:34:54.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71066412899999998, 2023-01-15T14:34:54.753 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.753 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:54.753 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3391 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.754 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:34:54.755 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.755 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061238+0000", 2023-01-15T14:34:54.755 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:54.755 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.755 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.755 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.755 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062134+0000", 2023-01-15T14:34:54.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089621299999999998 2023-01-15T14:34:54.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062160+0000", 2023-01-15T14:34:54.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5908999999999999e-05 2023-01-15T14:34:54.756 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.756 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.756 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.756 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062172+0000", 2023-01-15T14:34:54.756 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.189e-05 2023-01-15T14:34:54.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159827+0000", 2023-01-15T14:34:54.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097654866000000007 2023-01-15T14:34:54.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.757 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.757 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:54.757 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159846+0000", 2023-01-15T14:34:54.757 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8516e-05 2023-01-15T14:34:54.757 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263326+0000", 2023-01-15T14:34:54.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103480766 2023-01-15T14:34:54.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.758 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.758 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.758 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263535+0000", 2023-01-15T14:34:54.758 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020861299999999999 2023-01-15T14:34:54.758 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467576+0000", 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20404118600000001 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467736+0000", 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016000699999999999 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.759 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:54.760 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467987+0000", 2023-01-15T14:34:54.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025042199999999998 2023-01-15T14:34:54.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.760 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:54.760 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468369+0000", 2023-01-15T14:34:54.760 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038238600000000002 2023-01-15T14:34:54.760 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.760 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:54.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468387+0000", 2023-01-15T14:34:54.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7730000000000001e-05 2023-01-15T14:34:54.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:54.761 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468719+0000", 2023-01-15T14:34:54.761 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033272599999999999 2023-01-15T14:34:54.761 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.761 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.761 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:54.762 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470317+0000", 2023-01-15T14:34:54.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001597336 2023-01-15T14:34:54.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570851+0000", 2023-01-15T14:34:54.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10053426 2023-01-15T14:34:54.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771618+0000", 2023-01-15T14:34:54.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200767152 2023-01-15T14:34:54.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:54.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771768+0000", 2023-01-15T14:34:54.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015015699999999999 2023-01-15T14:34:54.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771905+0000", 2023-01-15T14:34:54.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013621099999999999 2023-01-15T14:34:54.884 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.884 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.884 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.885 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3392 3.1b 3:d85f42d9:::smithi09025264-45:head [setxattr _header (58) in=7b,truncate 3882483] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:54.885 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:54.885 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 79.441024575, 2023-01-15T14:34:54.885 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71008243400000004, 2023-01-15T14:34:54.885 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.885 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:54.885 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.885 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.885 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.886 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3392 2023-01-15T14:34:54.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.886 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:54.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:34:54.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062188+0000", 2023-01-15T14:34:54.887 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:54.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062195+0000", 2023-01-15T14:34:54.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2529999999999998e-06 2023-01-15T14:34:54.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062208+0000", 2023-01-15T14:34:54.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3003000000000001e-05 2023-01-15T14:34:54.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062215+0000", 2023-01-15T14:34:54.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5200000000000003e-06 2023-01-15T14:34:54.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159883+0000", 2023-01-15T14:34:54.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097667532000000001 2023-01-15T14:34:54.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:54.891 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159899+0000", 2023-01-15T14:34:54.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6759999999999999e-05 2023-01-15T14:34:54.891 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263853+0000", 2023-01-15T14:34:54.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10395361 2023-01-15T14:34:54.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264116+0000", 2023-01-15T14:34:54.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026336099999999999 2023-01-15T14:34:54.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468602+0000", 2023-01-15T14:34:54.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20448550900000001 2023-01-15T14:34:54.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468867+0000", 2023-01-15T14:34:54.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026495700000000002 2023-01-15T14:34:54.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:34:54.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469376+0000", 2023-01-15T14:34:54.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00050893100000000001 2023-01-15T14:34:54.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:54.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469789+0000", 2023-01-15T14:34:54.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000413158 2023-01-15T14:34:54.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:54.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469806+0000", 2023-01-15T14:34:54.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7513e-05 2023-01-15T14:34:54.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:54.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469884+0000", 2023-01-15T14:34:54.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7816000000000002e-05 2023-01-15T14:34:54.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:54.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470332+0000", 2023-01-15T14:34:54.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044807700000000001 2023-01-15T14:34:54.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.898 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570976+0000", 2023-01-15T14:34:54.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10064426899999999 2023-01-15T14:34:54.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:54.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772058+0000", 2023-01-15T14:34:54.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20108148200000001 2023-01-15T14:34:54.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:54.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772169+0000", 2023-01-15T14:34:54.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011120099999999999 2023-01-15T14:34:54.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772272+0000", 2023-01-15T14:34:54.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000102982 2023-01-15T14:34:54.901 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.901 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.901 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.901 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.901 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3393 3.1b 3:d85f42d9:::smithi09025264-45:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:34:54.901 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:54.901 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 79.440990498999994, 2023-01-15T14:34:54.902 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71076842399999995, 2023-01-15T14:34:54.902 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:54.902 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:54.902 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:54.902 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:54.902 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:54.903 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3393 2023-01-15T14:34:54.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.903 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:54.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:54.903 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:54.903 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.903 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.903 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:54.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:34:54.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:54.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.904 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:54.904 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062223+0000", 2023-01-15T14:34:54.904 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:54.904 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.905 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:54.905 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062227+0000", 2023-01-15T14:34:54.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3900000000000003e-06 2023-01-15T14:34:54.905 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:54.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062237+0000", 2023-01-15T14:34:54.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4539999999999996e-06 2023-01-15T14:34:54.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:54.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062243+0000", 2023-01-15T14:34:54.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0329999999999997e-06 2023-01-15T14:34:54.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161412+0000", 2023-01-15T14:34:54.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099168708999999994 2023-01-15T14:34:54.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:34:54.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161424+0000", 2023-01-15T14:34:54.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2415999999999999e-05 2023-01-15T14:34:54.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264185+0000", 2023-01-15T14:34:54.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.1027609 2023-01-15T14:34:54.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.908 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.908 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264421+0000", 2023-01-15T14:34:54.908 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023640900000000001 2023-01-15T14:34:54.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469985+0000", 2023-01-15T14:34:54.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.205564001 2023-01-15T14:34:54.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:54.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470264+0000", 2023-01-15T14:34:54.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027918700000000002 2023-01-15T14:34:54.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:54.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772390+0000", 2023-01-15T14:34:54.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30212558299999998 2023-01-15T14:34:54.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:54.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772684+0000", 2023-01-15T14:34:54.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029361799999999999 2023-01-15T14:34:54.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:54.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:54.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:54.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772992+0000", 2023-01-15T14:34:54.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854100000000001 2023-01-15T14:34:54.912 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.912 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.912 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.912 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:54.912 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:54.912 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:54.914 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:34:54.914 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:54.909+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:55.015 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:55.011+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:55.041 INFO:tasks.rados.rados.0.smithi090.stdout:1521: done (0 left) 2023-01-15T14:34:55.041 INFO:tasks.rados.rados.0.smithi090.stdout:1522: read oid 18 snap -1 2023-01-15T14:34:55.042 INFO:tasks.rados.rados.0.smithi090.stdout:1522: expect (ObjNum 537 snap 166 seq_num 537) 2023-01-15T14:34:55.042 INFO:tasks.rados.rados.0.smithi090.stdout:1523: snap_create 2023-01-15T14:34:55.042 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:34:55.075 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:55.075 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:55.075 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:34:55.075 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:55.085 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:34:55.206 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:55.199+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:55.268 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:34:55.305 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:55.301+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:55.319 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:55.320 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:55.320 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:34:55.320 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:34:55.320 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:55.330 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:34:55.364 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:34:55.364 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:34:55.364 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:34:55.364 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:34:55.407 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:55.401+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:55.436 INFO:tasks.rados.rados.0.smithi090.stdout:1522: expect (ObjNum 537 snap 166 seq_num 537) 2023-01-15T14:34:55.480 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:55.480 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:34:55.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:34:55.480 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:34:55.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.481 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:5 3.5 3:a1c47f99:::smithi09025264-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:55.481 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:55.481 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.69365904199998, 2023-01-15T14:34:55.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40548699700000002, 2023-01-15T14:34:55.481 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.481 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.481 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.481 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.481 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.481 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 5 2023-01-15T14:34:55.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.482 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:55.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:34:55.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781744+0000", 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782566+0000", 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000821578 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.484 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.484 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782588+0000", 2023-01-15T14:34:55.484 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.226e-05 2023-01-15T14:34:55.484 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.484 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.484 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.484 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782601+0000", 2023-01-15T14:34:55.484 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2998e-05 2023-01-15T14:34:55.484 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.484 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.484 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.485 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882850+0000", 2023-01-15T14:34:55.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100248908 2023-01-15T14:34:55.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:34:55.485 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882871+0000", 2023-01-15T14:34:55.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0873000000000002e-05 2023-01-15T14:34:55.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083679+0000", 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200808083 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083948+0000", 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000269342 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084161+0000", 2023-01-15T14:34:55.486 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021326900000000001 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084655+0000", 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049371300000000001 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084674+0000", 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9012999999999999e-05 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.487 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.488 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084924+0000", 2023-01-15T14:34:55.488 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024989899999999999 2023-01-15T14:34:55.488 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:55.488 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085221+0000", 2023-01-15T14:34:55.488 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029719600000000001 2023-01-15T14:34:55.488 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186953+0000", 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10173174 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187026+0000", 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2957e-05 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187124+0000", 2023-01-15T14:34:55.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8146000000000005e-05 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187233+0000", 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010931599999999999 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:6 3.5 3:a1c47f99:::smithi09025264-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:55.490 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.68840617000001, 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50419954199999995, 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 6 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.491 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.492 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:55.492 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.492 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.492 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.492 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.492 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:34:55.492 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.492 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.492 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.492 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.492 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786997+0000", 2023-01-15T14:34:55.493 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:55.493 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.493 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.493 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.493 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787486+0000", 2023-01-15T14:34:55.493 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048882999999999999 2023-01-15T14:34:55.493 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.493 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.493 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.493 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787514+0000", 2023-01-15T14:34:55.494 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7591999999999999e-05 2023-01-15T14:34:55.494 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.494 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787526+0000", 2023-01-15T14:34:55.494 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1894e-05 2023-01-15T14:34:55.494 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.494 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882966+0000", 2023-01-15T14:34:55.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095440450999999996 2023-01-15T14:34:55.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.495 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.495 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:55.495 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882973+0000", 2023-01-15T14:34:55.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2570000000000001e-06 2023-01-15T14:34:55.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.495 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.496 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.496 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084809+0000", 2023-01-15T14:34:55.496 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201835763 2023-01-15T14:34:55.496 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.496 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.496 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:55.496 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084928+0000", 2023-01-15T14:34:55.496 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118946 2023-01-15T14:34:55.496 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.496 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.497 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.497 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187347+0000", 2023-01-15T14:34:55.497 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102419285 2023-01-15T14:34:55.497 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.497 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.497 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.497 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187450+0000", 2023-01-15T14:34:55.497 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010286299999999999 2023-01-15T14:34:55.497 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.497 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.497 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:55.498 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187735+0000", 2023-01-15T14:34:55.498 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028429900000000001 2023-01-15T14:34:55.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.498 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.498 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188113+0000", 2023-01-15T14:34:55.498 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037883200000000001 2023-01-15T14:34:55.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.498 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.499 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188138+0000", 2023-01-15T14:34:55.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4110000000000001e-05 2023-01-15T14:34:55.499 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.499 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.499 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.499 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188389+0000", 2023-01-15T14:34:55.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025173900000000001 2023-01-15T14:34:55.499 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.500 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.500 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:55.500 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189251+0000", 2023-01-15T14:34:55.500 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086129800000000003 2023-01-15T14:34:55.500 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.500 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.500 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.500 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290989+0000", 2023-01-15T14:34:55.500 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101738524 2023-01-15T14:34:55.500 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.501 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.501 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.501 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291075+0000", 2023-01-15T14:34:55.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5469000000000004e-05 2023-01-15T14:34:55.501 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.501 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.501 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.501 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291144+0000", 2023-01-15T14:34:55.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:34:55.501 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.501 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.502 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.502 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291199+0000", 2023-01-15T14:34:55.502 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4413000000000002e-05 2023-01-15T14:34:55.502 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.502 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.502 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.502 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:7 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.68785971300002, 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50407251399999997, 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 7 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.503 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:55.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:34:55.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787545+0000", 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787551+0000", 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8669999999999999e-06 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787559+0000", 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9780000000000005e-06 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787564+0000", 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5709999999999997e-06 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883054+0000", 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095489722999999999 2023-01-15T14:34:55.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883061+0000", 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2760000000000003e-06 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084987+0000", 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20192545000000001 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.507 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:55.508 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085190+0000", 2023-01-15T14:34:55.508 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020288899999999999 2023-01-15T14:34:55.508 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.508 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.508 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.508 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188374+0000", 2023-01-15T14:34:55.508 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103184449 2023-01-15T14:34:55.508 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.508 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.508 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.508 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188543+0000", 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016925299999999999 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188748+0000", 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020473600000000001 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188986+0000", 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023780900000000001 2023-01-15T14:34:55.509 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.510 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.510 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.510 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189006+0000", 2023-01-15T14:34:55.510 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0604000000000001e-05 2023-01-15T14:34:55.510 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.510 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.510 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.510 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189084+0000", 2023-01-15T14:34:55.510 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7526000000000003e-05 2023-01-15T14:34:55.510 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.510 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189272+0000", 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018762 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291333+0000", 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102061185 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.511 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291466+0000", 2023-01-15T14:34:55.512 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013325699999999999 2023-01-15T14:34:55.512 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.512 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.512 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.512 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291535+0000", 2023-01-15T14:34:55.512 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9309999999999999e-05 2023-01-15T14:34:55.512 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.512 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.512 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.512 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291618+0000", 2023-01-15T14:34:55.512 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2773000000000002e-05 2023-01-15T14:34:55.513 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.513 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.513 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.513 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.513 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.513 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:8 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:55.513 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:55.513 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.68783373100001, 2023-01-15T14:34:55.513 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50442076000000002, 2023-01-15T14:34:55.513 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.513 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 8 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.514 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:34:55.515 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.515 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787571+0000", 2023-01-15T14:34:55.515 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:55.515 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.515 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.515 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787575+0000", 2023-01-15T14:34:55.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7890000000000001e-06 2023-01-15T14:34:55.516 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.516 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.516 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787589+0000", 2023-01-15T14:34:55.516 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4212999999999999e-05 2023-01-15T14:34:55.516 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.516 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.516 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.516 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787594+0000", 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1429999999999997e-06 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983143+0000", 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.195549003 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983153+0000", 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4369999999999996e-06 2023-01-15T14:34:55.517 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085286+0000", 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102133064 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085401+0000", 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115176 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.518 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189116+0000", 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103715444 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189221+0000", 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010438 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291724+0000", 2023-01-15T14:34:55.519 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102502998 2023-01-15T14:34:55.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291824+0000", 2023-01-15T14:34:55.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010010599999999999 2023-01-15T14:34:55.520 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.520 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.520 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.520 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291992+0000", 2023-01-15T14:34:55.520 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016856900000000001 2023-01-15T14:34:55.520 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.521 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.521 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.521 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:21 3.5 3:a75958b1:::smithi09025264-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:55.521 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:55.521 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.64374481599998, 2023-01-15T14:34:55.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.45851924500000002, 2023-01-15T14:34:55.521 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.521 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.521 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 21 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:34:55.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831657+0000", 2023-01-15T14:34:55.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:34:55.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832814+0000", 2023-01-15T14:34:55.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011564909999999999 2023-01-15T14:34:55.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832864+0000", 2023-01-15T14:34:55.524 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0243e-05 2023-01-15T14:34:55.524 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.524 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832903+0000", 2023-01-15T14:34:55.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8704000000000001e-05 2023-01-15T14:34:55.525 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.525 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983202+0000", 2023-01-15T14:34:55.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.150299404 2023-01-15T14:34:55.525 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.525 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.525 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:55.525 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983208+0000", 2023-01-15T14:34:55.525 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0039999999999996e-06 2023-01-15T14:34:55.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.526 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.526 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185539+0000", 2023-01-15T14:34:55.526 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202330867 2023-01-15T14:34:55.526 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.526 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.526 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.526 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185803+0000", 2023-01-15T14:34:55.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026394299999999999 2023-01-15T14:34:55.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:55.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186037+0000", 2023-01-15T14:34:55.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023386300000000001 2023-01-15T14:34:55.527 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.527 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.527 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.527 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186637+0000", 2023-01-15T14:34:55.527 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060001800000000001 2023-01-15T14:34:55.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186665+0000", 2023-01-15T14:34:55.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8459999999999999e-05 2023-01-15T14:34:55.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.528 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.528 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.528 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187045+0000", 2023-01-15T14:34:55.528 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037992899999999999 2023-01-15T14:34:55.528 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188201+0000", 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001155309 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289903+0000", 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101702601 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.529 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289985+0000", 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1617000000000001e-05 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290065+0000", 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0276000000000002e-05 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290180+0000", 2023-01-15T14:34:55.530 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011464300000000001 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:22 3.5 3:a75958b1:::smithi09025264-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.639277128, 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65790599599999999, 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.531 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 22 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.532 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:34:55.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836126+0000", 2023-01-15T14:34:55.533 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:55.533 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.533 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.533 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.533 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836181+0000", 2023-01-15T14:34:55.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4246999999999997e-05 2023-01-15T14:34:55.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836212+0000", 2023-01-15T14:34:55.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1843e-05 2023-01-15T14:34:55.534 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.534 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.534 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.534 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836226+0000", 2023-01-15T14:34:55.534 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3216e-05 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186757+0000", 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35053103899999999 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186871+0000", 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114403 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.535 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290263+0000", 2023-01-15T14:34:55.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103391976 2023-01-15T14:34:55.536 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.536 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.536 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290413+0000", 2023-01-15T14:34:55.536 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014991200000000001 2023-01-15T14:34:55.536 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.536 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.536 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290627+0000", 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021379299999999999 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290861+0000", 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023479600000000001 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290888+0000", 2023-01-15T14:34:55.537 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6570000000000001e-05 2023-01-15T14:34:55.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.538 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.538 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.538 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290990+0000", 2023-01-15T14:34:55.538 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101723 2023-01-15T14:34:55.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.538 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.538 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:55.538 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291245+0000", 2023-01-15T14:34:55.538 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025543899999999999 2023-01-15T14:34:55.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392822+0000", 2023-01-15T14:34:55.539 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10157725500000001 2023-01-15T14:34:55.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493881+0000", 2023-01-15T14:34:55.539 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101058262 2023-01-15T14:34:55.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.540 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.540 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.540 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493964+0000", 2023-01-15T14:34:55.540 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2973000000000007e-05 2023-01-15T14:34:55.540 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.540 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.540 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.540 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494034+0000", 2023-01-15T14:34:55.540 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0432999999999995e-05 2023-01-15T14:34:55.540 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.541 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.541 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.541 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.541 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.541 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:23 3.5 3:a75958b1:::smithi09025264-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:55.541 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:55.541 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.63916199900001, 2023-01-15T14:34:55.541 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65796116900000001, 2023-01-15T14:34:55.541 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.541 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.542 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.542 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.542 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.542 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 23 2023-01-15T14:34:55.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.542 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.542 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.542 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.542 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:55.542 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.542 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836242+0000", 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.543 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836256+0000", 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3677e-05 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836265+0000", 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2769999999999994e-06 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936398+0000", 2023-01-15T14:34:55.544 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100132559 2023-01-15T14:34:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289431+0000", 2023-01-15T14:34:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35303322399999998 2023-01-15T14:34:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289587+0000", 2023-01-15T14:34:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000155873 2023-01-15T14:34:55.545 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292052+0000", 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0024647919999999999 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292180+0000", 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012810999999999999 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.546 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292373+0000", 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019258599999999999 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292597+0000", 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022465500000000001 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292618+0000", 2023-01-15T14:34:55.547 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0319999999999999e-05 2023-01-15T14:34:55.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292692+0000", 2023-01-15T14:34:55.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4643999999999999e-05 2023-01-15T14:34:55.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.548 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.548 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:55.548 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393103+0000", 2023-01-15T14:34:55.548 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100411156 2023-01-15T14:34:55.548 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493827+0000", 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100724143 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494085+0000", 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000257891 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.549 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494141+0000", 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6132999999999999e-05 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494204+0000", 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2966000000000007e-05 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.550 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:24 3.5 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.538980573, 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.55808677600000001, 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 24 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.551 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:55.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:34:55.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936423+0000", 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936433+0000", 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0443e-05 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936451+0000", 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7842999999999998e-05 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936461+0000", 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0057e-05 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289693+0000", 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.353231617 2023-01-15T14:34:55.554 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289814+0000", 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120763 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392923+0000", 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10310955300000001 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.555 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:55.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393072+0000", 2023-01-15T14:34:55.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000148469 2023-01-15T14:34:55.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494242+0000", 2023-01-15T14:34:55.556 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117034499999999 2023-01-15T14:34:55.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.556 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.556 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494363+0000", 2023-01-15T14:34:55.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120703 2023-01-15T14:34:55.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494511+0000", 2023-01-15T14:34:55.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014873299999999999 2023-01-15T14:34:55.557 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.557 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.557 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:78 3.c 3:3137f0cb:::smithi09025264-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.149014577, 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20331801699999999, 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 78 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.558 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.559 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326388+0000", 2023-01-15T14:34:55.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:55.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326672+0000", 2023-01-15T14:34:55.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028367900000000001 2023-01-15T14:34:55.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326713+0000", 2023-01-15T14:34:55.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1539999999999999e-05 2023-01-15T14:34:55.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326729+0000", 2023-01-15T14:34:55.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.519e-05 2023-01-15T14:34:55.561 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.561 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326879+0000", 2023-01-15T14:34:55.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015064899999999999 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327096+0000", 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021682299999999999 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327308+0000", 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000211366 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.562 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327757+0000", 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000449009 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327776+0000", 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9201000000000001e-05 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.563 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328012+0000", 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236054 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429534+0000", 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10152201499999999 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529340+0000", 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099806249 2023-01-15T14:34:55.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529437+0000", 2023-01-15T14:34:55.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6949999999999998e-05 2023-01-15T14:34:55.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529587+0000", 2023-01-15T14:34:55.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015008600000000001 2023-01-15T14:34:55.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529709+0000", 2023-01-15T14:34:55.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121636 2023-01-15T14:34:55.566 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.566 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.566 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.566 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:79 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:55.566 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.14864498100002, 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20348171300000001, 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 79 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:55.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:34:55.568 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.568 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326759+0000", 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326770+0000", 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326791+0000", 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1556000000000002e-05 2023-01-15T14:34:55.569 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326818+0000", 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6979999999999999e-05 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.427993+0000", 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117517199999999 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.570 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428266+0000", 2023-01-15T14:34:55.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027233299999999998 2023-01-15T14:34:55.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:34:55.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428696+0000", 2023-01-15T14:34:55.571 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042996499999999998 2023-01-15T14:34:55.571 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.571 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.571 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.571 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429057+0000", 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000360952 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429076+0000", 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429160+0000", 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:34:55.572 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429549+0000", 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038884599999999998 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529886+0000", 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100336261 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529978+0000", 2023-01-15T14:34:55.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2775999999999994e-05 2023-01-15T14:34:55.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530108+0000", 2023-01-15T14:34:55.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012958300000000001 2023-01-15T14:34:55.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530242+0000", 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013406 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:80 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.14850662200001, 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20370874, 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.575 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:55.576 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.576 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.576 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.576 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 80 2023-01-15T14:34:55.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.576 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:55.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326897+0000", 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.577 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326906+0000", 2023-01-15T14:34:55.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6929999999999996e-06 2023-01-15T14:34:55.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326931+0000", 2023-01-15T14:34:55.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4757e-05 2023-01-15T14:34:55.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326945+0000", 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.402e-05 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429239+0000", 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102294418 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429487+0000", 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024789300000000001 2023-01-15T14:34:55.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530309+0000", 2023-01-15T14:34:55.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100821459 2023-01-15T14:34:55.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530438+0000", 2023-01-15T14:34:55.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012928700000000001 2023-01-15T14:34:55.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530607+0000", 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169483 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:78 3.c e23/20 ondisk, result = 0)", 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 389.14661490100002, 2023-01-15T14:34:55.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20094287799999999, 2023-01-15T14:34:55.582 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.582 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:55.582 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:55.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:34:55.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328789+0000", 2023-01-15T14:34:55.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:55.583 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.583 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.583 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.583 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328799+0000", 2023-01-15T14:34:55.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8980000000000008e-06 2023-01-15T14:34:55.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328820+0000", 2023-01-15T14:34:55.584 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1041999999999999e-05 2023-01-15T14:34:55.584 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.584 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.584 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.584 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328832+0000", 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1616e-05 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529404+0000", 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200572631 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529419+0000", 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4250000000000001e-05 2023-01-15T14:34:55.585 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529733+0000", 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000314693 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:201 3.5 3:a1c47f99:::smithi09025264-2:head [copy-from ver 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:55.586 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 388.06900850400001, 2023-01-15T14:34:55.587 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30931411399999997, 2023-01-15T14:34:55.587 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.587 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.587 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.587 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.587 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.587 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 201 2023-01-15T14:34:55.587 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.587 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.587 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.587 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:55.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:34:55.588 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.588 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.588 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.588 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406394+0000", 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406408+0000", 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4416999999999999e-05 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406446+0000", 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7339999999999998e-05 2023-01-15T14:34:55.589 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406467+0000", 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1137000000000001e-05 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406677+0000", 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000210275 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.590 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406902+0000", 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022529000000000001 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.513012+0000", 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.106109527 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514754+0000", 2023-01-15T14:34:55.591 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0017419779999999999 2023-01-15T14:34:55.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514773+0000", 2023-01-15T14:34:55.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8700000000000001e-05 2023-01-15T14:34:55.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.592 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.592 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.592 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.515531+0000", 2023-01-15T14:34:55.592 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075844300000000003 2023-01-15T14:34:55.592 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615334+0000", 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099803327999999997 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615376+0000", 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1675999999999999e-05 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.593 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715546+0000", 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016992500000001 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715626+0000", 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0112999999999997e-05 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715711+0000", 2023-01-15T14:34:55.594 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:202 3.5 3:a1c47f99:::smithi09025264-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 388.06889595799998, 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31002637300000002, 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:34:55.595 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 202 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.596 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406508+0000", 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406516+0000", 2023-01-15T14:34:55.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.413e-06 2023-01-15T14:34:55.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406534+0000", 2023-01-15T14:34:55.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8936999999999998e-05 2023-01-15T14:34:55.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406548+0000", 2023-01-15T14:34:55.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3196e-05 2023-01-15T14:34:55.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.599 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.599 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407271+0000", 2023-01-15T14:34:55.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00072301500000000003 2023-01-15T14:34:55.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.599 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:34:55.599 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407466+0000", 2023-01-15T14:34:55.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000195009 2023-01-15T14:34:55.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.614996+0000", 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.207530192 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615227+0000", 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000230979 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.600 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716282+0000", 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101055465 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716409+0000", 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012653999999999999 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716536+0000", 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012683 2023-01-15T14:34:55.601 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:189 3.d e23/20 3:b3f111b5:::smithi09025264-48:head v 23'10, mlcod=23'6)", 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 388.02369131, 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20400475100000001, 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.602 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:55.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.603 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:34:55.603 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.603 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.603 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.603 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451712+0000", 2023-01-15T14:34:55.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:55.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452272+0000", 2023-01-15T14:34:55.604 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00056035899999999997 2023-01-15T14:34:55.604 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.604 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.604 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.604 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452310+0000", 2023-01-15T14:34:55.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8145999999999997e-05 2023-01-15T14:34:55.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452324+0000", 2023-01-15T14:34:55.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4443999999999999e-05 2023-01-15T14:34:55.605 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.605 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.605 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652703+0000", 2023-01-15T14:34:55.605 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20037818199999999 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652762+0000", 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9302999999999998e-05 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653302+0000", 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054012199999999998 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653333+0000", 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.112e-05 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653667+0000", 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033337900000000002 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653709+0000", 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2469999999999998e-05 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655719+0000", 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020096699999999999 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.608 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:269 3.2 e30/20 3:47dfe935:::smithi09025264-29:head v 30'11, mlcod=23'8)", 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 380.59764586900002, 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23185540399999999, 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.609 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.610 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:55.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.610 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:34:55.610 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:55.610 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.610 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.610 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878379+0000", 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061979600000000004 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878406+0000", 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978515+0000", 2023-01-15T14:34:55.611 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100108461 2023-01-15T14:34:55.612 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.612 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.612 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078759+0000", 2023-01-15T14:34:55.612 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100244496 2023-01-15T14:34:55.612 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.612 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.612 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.612 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078796+0000", 2023-01-15T14:34:55.612 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6758999999999998e-05 2023-01-15T14:34:55.612 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079567+0000", 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077137199999999999 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079593+0000", 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.613 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079844+0000", 2023-01-15T14:34:55.614 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025075899999999998 2023-01-15T14:34:55.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.614 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.614 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079910+0000", 2023-01-15T14:34:55.614 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6406000000000001e-05 2023-01-15T14:34:55.614 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.614 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.614 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.615 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.109615+0000", 2023-01-15T14:34:55.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.029704323000000001 2023-01-15T14:34:55.615 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.615 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.615 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.615 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.615 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.615 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:281 3.5 3:a75958b1:::smithi09025264-6:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:34:55.615 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:55.615 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 380.49672224, 2023-01-15T14:34:55.615 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30574310900000001, 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 281 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:55.616 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:55.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.617 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:34:55.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:55.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978686+0000", 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4429999999999999e-06 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978690+0000", 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0300000000000004e-06 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978693+0000", 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.179e-06 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981592+0000", 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0028984140000000002 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981718+0000", 2023-01-15T14:34:55.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012616600000000001 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981982+0000", 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026445199999999998 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982278+0000", 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029514100000000001 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982290+0000", 2023-01-15T14:34:55.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.271e-05 2023-01-15T14:34:55.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982343+0000", 2023-01-15T14:34:55.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2914000000000002e-05 2023-01-15T14:34:55.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982385+0000", 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2114999999999999e-05 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.181937+0000", 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19955213099999999 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284183+0000", 2023-01-15T14:34:55.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10224538 2023-01-15T14:34:55.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284318+0000", 2023-01-15T14:34:55.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000134843 2023-01-15T14:34:55.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284426+0000", 2023-01-15T14:34:55.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108513 2023-01-15T14:34:55.623 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:373 3.4 e34/20 3:2029fa71:::smithi09025264-39:head v 34'19, mlcod=26'16)", 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 375.01573978099998, 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20212622799999999, 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:55.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.625 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:34:55.625 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.625 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.625 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459664+0000", 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460099+0000", 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043515900000000001 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460130+0000", 2023-01-15T14:34:55.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1590000000000001e-05 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560226+0000", 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100095248 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560369+0000", 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014381499999999999 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560404+0000", 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4650000000000002e-05 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560804+0000", 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000399823 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.628 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560827+0000", 2023-01-15T14:34:55.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3612999999999999e-05 2023-01-15T14:34:55.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.561101+0000", 2023-01-15T14:34:55.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027355499999999998 2023-01-15T14:34:55.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.629 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.629 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661710+0000", 2023-01-15T14:34:55.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100608716 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661792+0000", 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2006000000000001e-05 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:455 3.d e41/20 3:bd982ca3:::smithi09025264-38:head v 41'24, mlcod=38'18)", 2023-01-15T14:34:55.630 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 367.96597820800002, 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20118033699999999, 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:34:55.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509425+0000", 2023-01-15T14:34:55.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:55.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509441+0000", 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5248e-05 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509474+0000", 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3531000000000001e-05 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.609644+0000", 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016994999999999 2023-01-15T14:34:55.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709872+0000", 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10022840299999999 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709906+0000", 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3160000000000001e-05 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710153+0000", 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024735399999999998 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710204+0000", 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1075999999999997e-05 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710280+0000", 2023-01-15T14:34:55.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6202e-05 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710317+0000", 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6387000000000002e-05 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710607+0000", 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029079699999999997 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.636 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:497 3.d e44/20 3:bd982ca3:::smithi09025264-38:head v 44'31, mlcod=42'26)", 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 365.70118334599999, 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20082067100000001, 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.637 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774220+0000", 2023-01-15T14:34:55.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:34:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774238+0000", 2023-01-15T14:34:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8107000000000001e-05 2023-01-15T14:34:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:34:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774263+0000", 2023-01-15T14:34:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4122999999999999e-05 2023-01-15T14:34:55.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:34:55.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774280+0000", 2023-01-15T14:34:55.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7039999999999999e-05 2023-01-15T14:34:55.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:34:55.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874060+0000", 2023-01-15T14:34:55.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099780698000000001 2023-01-15T14:34:55.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:34:55.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874130+0000", 2023-01-15T14:34:55.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9852999999999997e-05 2023-01-15T14:34:55.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:55.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874351+0000", 2023-01-15T14:34:55.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022091400000000001 2023-01-15T14:34:55.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:55.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874369+0000", 2023-01-15T14:34:55.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7876e-05 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874433+0000", 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3793000000000004e-05 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.974927+0000", 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10049393500000001 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:34:55.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:34:55.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:34:55.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.975043+0000", 2023-01-15T14:34:55.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115949 2023-01-15T14:34:55.716 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.716 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.716 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.716 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:34:55.716 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:34:55.716 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:55.717 INFO:tasks.rados.rados.0.smithi090.stdout:1522: done (0 left) 2023-01-15T14:34:55.719 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:34:55.722 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:34:55.725 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:34:55.725 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":602,"stamp":"2023-01-15T14:34:55.008200+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":420314629,"num_objects":199,"num_object_clones":146,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":16,"num_read":1059,"num_read_kb":1055478,"num_write":859,"num_write_kb":300445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":661,"num_bytes_recovered":439437762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19447,"ondisk_log_size":19447,"up":345,"acting":345,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":346,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6500536,"kb_used_data":555702920,"kb_used_omap":81494,"kb_used_meta":0,"kb_avail":555695744,"statfs":{"total":575688990720,"available":569032441856,"internally_reserved":7348224,"allocated":569039790080,"data_stored":569039790080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":83449895,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13502281,"num_objects":-9,"num_object_clones":-5,"num_object_copies":-27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":-1,"num_read":-35,"num_read_kb":-18969,"num_write":-35,"num_write_kb":-17667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-18,"num_bytes_recovered":-12649150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.026521"},"pg_stats":[{"pgid":"3.73","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:34:54.997645+0000","last_change":"2023-01-15T14:34:54.997645+0000","last_active":"2023-01-15T14:34:54.997645+0000","last_peered":"2023-01-15T14:34:54.997645+0000","last_clean":"2023-01-15T14:34:54.997645+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:54.997645+0000","last_undegraded":"2023-01-15T14:34:54.997645+0000","last_fullsized":"2023-01-15T14:34:54.997645+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_clean_scrub_stamp":"2023-01-15T14:34:54.997645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.72","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:34:54.997645+0000","last_change":"2023-01-15T14:34:54.997645+0000","last_active":"2023-01-15T14:34:54.997645+0000","last_peered":"2023-01-15T14:34:54.997645+0000","last_clean":"2023-01-15T14:34:54.997645+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:54.997645+0000","last_undegraded":"2023-01-15T14:34:54.997645+0000","last_fullsized":"2023-01-15T14:34:54.997645+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_clean_scrub_stamp":"2023-01-15T14:34:54.997645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.71","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:34:54.997645+0000","last_change":"2023-01-15T14:34:54.997645+0000","last_active":"2023-01-15T14:34:54.997645+0000","last_peered":"2023-01-15T14:34:54.997645+0000","last_clean":"2023-01-15T14:34:54.997645+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:54.997645+0000","last_undegraded":"2023-01-15T14:34:54.997645+0000","last_fullsized":"2023-01-15T14:34:54.997645+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_clean_scrub_stamp":"2023-01-15T14:34:54.997645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.70","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:34:54.997645+0000","last_change":"2023-01-15T14:34:54.997645+0000","last_active":"2023-01-15T14:34:54.997645+0000","last_peered":"2023-01-15T14:34:54.997645+0000","last_clean":"2023-01-15T14:34:54.997645+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:54.997645+0000","last_undegraded":"2023-01-15T14:34:54.997645+0000","last_fullsized":"2023-01-15T14:34:54.997645+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_clean_scrub_stamp":"2023-01-15T14:34:54.997645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:34:54.997645+0000","last_change":"2023-01-15T14:34:54.997645+0000","last_active":"2023-01-15T14:34:54.997645+0000","last_peered":"2023-01-15T14:34:54.997645+0000","last_clean":"2023-01-15T14:34:54.997645+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:54.997645+0000","last_undegraded":"2023-01-15T14:34:54.997645+0000","last_fullsized":"2023-01-15T14:34:54.997645+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_clean_scrub_stamp":"2023-01-15T14:34:54.997645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:34:54.997645+0000","last_change":"2023-01-15T14:34:54.997645+0000","last_active":"2023-01-15T14:34:54.997645+0000","last_peered":"2023-01-15T14:34:54.997645+0000","last_clean":"2023-01-15T14:34:54.997645+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:54.997645+0000","last_undegraded":"2023-01-15T14:34:54.997645+0000","last_fullsized":"2023-01-15T14:34:54.997645+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_clean_scrub_stamp":"2023-01-15T14:34:54.997645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:34:54.997645+0000","last_change":"2023-01-15T14:34:54.997645+0000","last_active":"2023-01-15T14:34:54.997645+0000","last_peered":"2023-01-15T14:34:54.997645+0000","last_clean":"2023-01-15T14:34:54.997645+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:54.997645+0000","last_undegraded":"2023-01-15T14:34:54.997645+0000","last_fullsized":"2023-01-15T14:34:54.997645+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_clean_scrub_stamp":"2023-01-15T14:34:54.997645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:34:54.997645+0000","last_change":"2023-01-15T14:34:54.997645+0000","last_active":"2023-01-15T14:34:54.997645+0000","last_peered":"2023-01-15T14:34:54.997645+0000","last_clean":"2023-01-15T14:34:54.997645+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:54.997645+0000","last_undegraded":"2023-01-15T14:34:54.997645+0000","last_fullsized":"2023-01-15T14:34:54.997645+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_clean_scrub_stamp":"2023-01-15T14:34:54.997645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:34:54.997645+0000","last_change":"2023-01-15T14:34:54.997645+0000","last_active":"2023-01-15T14:34:54.997645+0000","last_peered":"2023-01-15T14:34:54.997645+0000","last_clean":"2023-01-15T14:34:54.997645+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:54.997645+0000","last_undegraded":"2023-01-15T14:34:54.997645+0000","last_fullsized":"2023-01-15T14:34:54.997645+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_clean_scrub_stamp":"2023-01-15T14:34:54.997645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:34:54.997645+0000","last_change":"2023-01-15T14:34:54.997645+0000","last_active":"2023-01-15T14:34:54.997645+0000","last_peered":"2023-01-15T14:34:54.997645+0000","last_clean":"2023-01-15T14:34:54.997645+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:34:54.997645+0000","last_undegraded":"2023-01-15T14:34:54.997645+0000","last_fullsized":"2023-01-15T14:34:54.997645+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:34:54.997645+0000","last_clean_scrub_stamp":"2023-01-15T14:34:54.997645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.69","version":"110'149","reported_seq":1045,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.608334+0000","last_change":"2023-01-15T14:34:52.608334+0000","last_active":"2023-01-15T14:34:52.608334+0000","last_peered":"2023-01-15T14:34:52.608334+0000","last_clean":"2023-01-15T14:34:52.608334+0000","last_became_active":"2023-01-15T14:34:46.074868+0000","last_became_peered":"2023-01-15T14:34:46.074868+0000","last_unstale":"2023-01-15T14:34:52.608334+0000","last_undegraded":"2023-01-15T14:34:52.608334+0000","last_fullsized":"2023-01-15T14:34:52.608334+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.56595390499999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.68","version":"391'267","reported_seq":1533,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.459507+0000","last_change":"2023-01-15T14:34:51.559930+0000","last_active":"2023-01-15T14:34:51.459507+0000","last_peered":"2023-01-15T14:34:51.459507+0000","last_clean":"2023-01-15T14:34:51.459507+0000","last_became_active":"2023-01-15T14:34:13.134258+0000","last_became_peered":"2023-01-15T14:34:13.134258+0000","last_unstale":"2023-01-15T14:34:51.459507+0000","last_undegraded":"2023-01-15T14:34:51.459507+0000","last_fullsized":"2023-01-15T14:34:51.459507+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.34227123599999998,"stat_sum":{"num_bytes":8014979,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17027823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.67","version":"191'79","reported_seq":157,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.044318+0000","last_change":"2023-01-15T14:34:52.044267+0000","last_active":"2023-01-15T14:34:52.044318+0000","last_peered":"2023-01-15T14:34:52.044318+0000","last_clean":"2023-01-15T14:34:52.044318+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:34:52.044318+0000","last_undegraded":"2023-01-15T14:34:52.044318+0000","last_fullsized":"2023-01-15T14:34:52.044318+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.778876919,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.66","version":"102'52","reported_seq":499,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.092256+0000","last_change":"2023-01-15T14:34:52.163936+0000","last_active":"2023-01-15T14:34:53.092256+0000","last_peered":"2023-01-15T14:34:53.092256+0000","last_clean":"2023-01-15T14:34:53.092256+0000","last_became_active":"2023-01-15T14:34:43.359681+0000","last_became_peered":"2023-01-15T14:34:43.359681+0000","last_unstale":"2023-01-15T14:34:53.092256+0000","last_undegraded":"2023-01-15T14:34:53.092256+0000","last_fullsized":"2023-01-15T14:34:53.092256+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:34:32.591909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.591909+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:36.991","scrub_duration":0.39200112999999998,"objects_trimmed":0,"snaptrim_duration":0.101434654,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.65","version":"402'143","reported_seq":1357,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.060774+0000","last_change":"2023-01-15T14:34:53.061410+0000","last_active":"2023-01-15T14:34:53.060774+0000","last_peered":"2023-01-15T14:34:53.060774+0000","last_clean":"2023-01-15T14:34:53.060774+0000","last_became_active":"2023-01-15T14:34:44.471358+0000","last_became_peered":"2023-01-15T14:34:44.471358+0000","last_unstale":"2023-01-15T14:34:53.060774+0000","last_undegraded":"2023-01-15T14:34:53.060774+0000","last_fullsized":"2023-01-15T14:34:53.060774+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'126","last_scrub_stamp":"2023-01-15T14:34:23.903906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.903906+0000","objects_scrubbed":6,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:35.127","scrub_duration":0.30981859099999998,"objects_trimmed":1,"snaptrim_duration":1.005018642,"stat_sum":{"num_bytes":12652889,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":13204,"num_write":2,"num_write_kb":2725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2790061,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.64","version":"334'251","reported_seq":1520,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.458457+0000","last_change":"2023-01-15T14:34:51.459443+0000","last_active":"2023-01-15T14:34:51.458457+0000","last_peered":"2023-01-15T14:34:51.458457+0000","last_clean":"2023-01-15T14:34:51.458457+0000","last_became_active":"2023-01-15T14:33:55.675085+0000","last_became_peered":"2023-01-15T14:33:55.675085+0000","last_unstale":"2023-01-15T14:34:51.458457+0000","last_undegraded":"2023-01-15T14:34:51.458457+0000","last_fullsized":"2023-01-15T14:34:51.458457+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"334'251","last_scrub_stamp":"2023-01-15T14:34:48.644529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.644529+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:52.706","scrub_duration":0.70872698300000003,"objects_trimmed":0,"snaptrim_duration":0.242507893,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.63","version":"149'233","reported_seq":839,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.321664+0000","last_change":"2023-01-15T14:34:51.322269+0000","last_active":"2023-01-15T14:34:51.321664+0000","last_peered":"2023-01-15T14:34:51.321664+0000","last_clean":"2023-01-15T14:34:51.321664+0000","last_became_active":"2023-01-15T14:33:55.277739+0000","last_became_peered":"2023-01-15T14:33:55.277739+0000","last_unstale":"2023-01-15T14:34:51.321664+0000","last_undegraded":"2023-01-15T14:34:51.321664+0000","last_fullsized":"2023-01-15T14:34:51.321664+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20494165,"stat_sum":{"num_bytes":3802136,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":24,"num_read_kb":21819,"num_write":19,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.62","version":"401'187","reported_seq":1484,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.356856+0000","last_change":"2023-01-15T14:34:52.356957+0000","last_active":"2023-01-15T14:34:52.356856+0000","last_peered":"2023-01-15T14:34:52.356856+0000","last_clean":"2023-01-15T14:34:52.356856+0000","last_became_active":"2023-01-15T14:34:44.164909+0000","last_became_peered":"2023-01-15T14:34:44.164909+0000","last_unstale":"2023-01-15T14:34:52.356856+0000","last_undegraded":"2023-01-15T14:34:52.356856+0000","last_fullsized":"2023-01-15T14:34:52.356856+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":7,"last_scrub":"375'173","last_scrub_stamp":"2023-01-15T14:34:34.093838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:34.093838+0000","objects_scrubbed":2,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.199880059,"stat_sum":{"num_bytes":9070002,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":14814,"num_write":17,"num_write_kb":6240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":9221281,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.61","version":"395'158","reported_seq":1141,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.116789+0000","last_change":"2023-01-15T14:34:51.318285+0000","last_active":"2023-01-15T14:34:51.116789+0000","last_peered":"2023-01-15T14:34:51.116789+0000","last_clean":"2023-01-15T14:34:51.116789+0000","last_became_active":"2023-01-15T14:34:44.473866+0000","last_became_peered":"2023-01-15T14:34:44.473866+0000","last_unstale":"2023-01-15T14:34:51.116789+0000","last_undegraded":"2023-01-15T14:34:51.116789+0000","last_fullsized":"2023-01-15T14:34:51.116789+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:32.734","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.20174270999999999,"stat_sum":{"num_bytes":3823692,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6060,"num_write":6,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.60","version":"402'291","reported_seq":1544,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.860260+0000","last_change":"2023-01-15T14:34:52.861052+0000","last_active":"2023-01-15T14:34:52.860260+0000","last_peered":"2023-01-15T14:34:52.860260+0000","last_clean":"2023-01-15T14:34:52.860260+0000","last_became_active":"2023-01-15T14:34:44.073670+0000","last_became_peered":"2023-01-15T14:34:44.073670+0000","last_unstale":"2023-01-15T14:34:52.860260+0000","last_undegraded":"2023-01-15T14:34:52.860260+0000","last_fullsized":"2023-01-15T14:34:52.860260+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"376'285","last_scrub_stamp":"2023-01-15T14:34:32.444552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.444552+0000","objects_scrubbed":1,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70464121800000001,"objects_trimmed":0,"snaptrim_duration":0.80549143300000003,"stat_sum":{"num_bytes":5214108,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":27913,"num_write":14,"num_write_kb":5082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":15418818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1473,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.766149+0000","last_change":"2023-01-15T14:34:51.767398+0000","last_active":"2023-01-15T14:34:51.766149+0000","last_peered":"2023-01-15T14:34:51.766149+0000","last_clean":"2023-01-15T14:34:51.766149+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:34:51.766149+0000","last_undegraded":"2023-01-15T14:34:51.766149+0000","last_fullsized":"2023-01-15T14:34:51.766149+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:51.919","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.44785891100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.5e","version":"215'39","reported_seq":1104,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.648331+0000","last_change":"2023-01-15T14:34:52.648331+0000","last_active":"2023-01-15T14:34:52.648331+0000","last_peered":"2023-01-15T14:34:52.648331+0000","last_clean":"2023-01-15T14:34:52.648331+0000","last_became_active":"2023-01-15T14:34:43.164436+0000","last_became_peered":"2023-01-15T14:34:43.164436+0000","last_unstale":"2023-01-15T14:34:52.648331+0000","last_undegraded":"2023-01-15T14:34:52.648331+0000","last_fullsized":"2023-01-15T14:34:52.648331+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.50622271200000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1425,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.192644+0000","last_change":"2023-01-15T14:34:52.265082+0000","last_active":"2023-01-15T14:34:53.192644+0000","last_peered":"2023-01-15T14:34:53.192644+0000","last_clean":"2023-01-15T14:34:53.192644+0000","last_became_active":"2023-01-15T14:34:00.242122+0000","last_became_peered":"2023-01-15T14:34:00.242122+0000","last_unstale":"2023-01-15T14:34:53.192644+0000","last_undegraded":"2023-01-15T14:34:53.192644+0000","last_fullsized":"2023-01-15T14:34:53.192644+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:19.729021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:19.729021+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:29.740","scrub_duration":0.50499948100000003,"objects_trimmed":0,"snaptrim_duration":0.20240625200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.5c","version":"141'46","reported_seq":186,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.646813+0000","last_change":"2023-01-15T14:34:52.646813+0000","last_active":"2023-01-15T14:34:52.646813+0000","last_peered":"2023-01-15T14:34:52.646813+0000","last_clean":"2023-01-15T14:34:52.646813+0000","last_became_active":"2023-01-15T14:33:57.113032+0000","last_became_peered":"2023-01-15T14:33:57.113032+0000","last_unstale":"2023-01-15T14:34:52.646813+0000","last_undegraded":"2023-01-15T14:34:52.646813+0000","last_fullsized":"2023-01-15T14:34:52.646813+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.60468838599999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.5b","version":"283'80","reported_seq":1012,"reported_epoch":403,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:53.459616+0000","last_change":"2023-01-15T14:34:53.459676+0000","last_active":"2023-01-15T14:34:53.459616+0000","last_peered":"2023-01-15T14:34:53.459616+0000","last_clean":"2023-01-15T14:34:53.459616+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:34:53.459616+0000","last_undegraded":"2023-01-15T14:34:53.459616+0000","last_fullsized":"2023-01-15T14:34:53.459616+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:42.155","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.91008361800000004,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"3.5a","version":"401'357","reported_seq":1613,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.662495+0000","last_change":"2023-01-15T14:34:51.662823+0000","last_active":"2023-01-15T14:34:51.662495+0000","last_peered":"2023-01-15T14:34:51.662495+0000","last_clean":"2023-01-15T14:34:51.662495+0000","last_became_active":"2023-01-15T14:34:44.469148+0000","last_became_peered":"2023-01-15T14:34:44.469148+0000","last_unstale":"2023-01-15T14:34:51.662495+0000","last_undegraded":"2023-01-15T14:34:51.662495+0000","last_fullsized":"2023-01-15T14:34:51.662495+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:03.828","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.41772785200000001,"stat_sum":{"num_bytes":5983424,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":19294,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.59","version":"294'256","reported_seq":1394,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.091486+0000","last_change":"2023-01-15T14:34:52.190239+0000","last_active":"2023-01-15T14:34:53.091486+0000","last_peered":"2023-01-15T14:34:53.091486+0000","last_clean":"2023-01-15T14:34:53.091486+0000","last_became_active":"2023-01-15T14:33:55.671990+0000","last_became_peered":"2023-01-15T14:33:55.671990+0000","last_unstale":"2023-01-15T14:34:53.091486+0000","last_undegraded":"2023-01-15T14:34:53.091486+0000","last_fullsized":"2023-01-15T14:34:53.091486+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:34:30.207952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:30.207952+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:47.670","scrub_duration":0.024191634,"objects_trimmed":0,"snaptrim_duration":0.12766946900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.58","version":"323'320","reported_seq":1635,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.089143+0000","last_change":"2023-01-15T14:34:52.089075+0000","last_active":"2023-01-15T14:34:52.089143+0000","last_peered":"2023-01-15T14:34:52.089143+0000","last_clean":"2023-01-15T14:34:52.089143+0000","last_became_active":"2023-01-15T14:34:46.366277+0000","last_became_peered":"2023-01-15T14:34:46.366277+0000","last_unstale":"2023-01-15T14:34:52.089143+0000","last_undegraded":"2023-01-15T14:34:52.089143+0000","last_fullsized":"2023-01-15T14:34:52.089143+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'320","last_scrub_stamp":"2023-01-15T14:34:47.541655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.541655+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:12.461","scrub_duration":0.40398073200000001,"objects_trimmed":0,"snaptrim_duration":0.147930108,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.57","version":"390'110","reported_seq":219,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.765565+0000","last_change":"2023-01-15T14:34:51.765953+0000","last_active":"2023-01-15T14:34:51.765565+0000","last_peered":"2023-01-15T14:34:51.765565+0000","last_clean":"2023-01-15T14:34:51.765565+0000","last_became_active":"2023-01-15T14:33:36.994447+0000","last_became_peered":"2023-01-15T14:33:36.994447+0000","last_unstale":"2023-01-15T14:34:51.765565+0000","last_undegraded":"2023-01-15T14:34:51.765565+0000","last_fullsized":"2023-01-15T14:34:51.765565+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'100","last_scrub_stamp":"2023-01-15T14:34:23.447852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.447852+0000","objects_scrubbed":2,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:30.799","scrub_duration":0.52101476099999999,"objects_trimmed":0,"snaptrim_duration":0.44676323400000001,"stat_sum":{"num_bytes":5882615,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":1093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.56","version":"320'127","reported_seq":733,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.146228+0000","last_change":"2023-01-15T14:34:52.146178+0000","last_active":"2023-01-15T14:34:52.146228+0000","last_peered":"2023-01-15T14:34:52.146228+0000","last_clean":"2023-01-15T14:34:52.146228+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:34:52.146228+0000","last_undegraded":"2023-01-15T14:34:52.146228+0000","last_fullsized":"2023-01-15T14:34:52.146228+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.25505350700000001,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.55","version":"110'55","reported_seq":1116,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.252360+0000","last_change":"2023-01-15T14:34:53.353345+0000","last_active":"2023-01-15T14:34:53.252360+0000","last_peered":"2023-01-15T14:34:53.252360+0000","last_clean":"2023-01-15T14:34:53.252360+0000","last_became_active":"2023-01-15T14:33:52.510429+0000","last_became_peered":"2023-01-15T14:33:52.510429+0000","last_unstale":"2023-01-15T14:34:53.252360+0000","last_undegraded":"2023-01-15T14:34:53.252360+0000","last_fullsized":"2023-01-15T14:34:53.252360+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.36394697799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.54","version":"245'148","reported_seq":1239,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.320326+0000","last_change":"2023-01-15T14:34:51.397456+0000","last_active":"2023-01-15T14:34:51.320326+0000","last_peered":"2023-01-15T14:34:51.320326+0000","last_clean":"2023-01-15T14:34:51.320326+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:34:51.320326+0000","last_undegraded":"2023-01-15T14:34:51.320326+0000","last_fullsized":"2023-01-15T14:34:51.320326+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:34:47.392699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.392699+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:14.435","scrub_duration":0.50581767899999996,"objects_trimmed":0,"snaptrim_duration":0.28060572099999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.53","version":"402'435","reported_seq":1644,"reported_epoch":402,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:53.014116+0000","last_change":"2023-01-15T14:34:52.550656+0000","last_active":"2023-01-15T14:34:53.014116+0000","last_peered":"2023-01-15T14:34:53.014116+0000","last_clean":"2023-01-15T14:34:53.014116+0000","last_became_active":"2023-01-15T14:34:44.173172+0000","last_became_peered":"2023-01-15T14:34:44.173172+0000","last_unstale":"2023-01-15T14:34:53.014116+0000","last_undegraded":"2023-01-15T14:34:53.014116+0000","last_fullsized":"2023-01-15T14:34:53.014116+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.70333423699999997,"stat_sum":{"num_bytes":29519463,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":59,"num_read_kb":67496,"num_write":30,"num_write_kb":16427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":32800828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":875,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.042771+0000","last_change":"2023-01-15T14:34:52.042719+0000","last_active":"2023-01-15T14:34:52.042771+0000","last_peered":"2023-01-15T14:34:52.042771+0000","last_clean":"2023-01-15T14:34:52.042771+0000","last_became_active":"2023-01-15T14:34:46.374191+0000","last_became_peered":"2023-01-15T14:34:46.374191+0000","last_unstale":"2023-01-15T14:34:52.042771+0000","last_undegraded":"2023-01-15T14:34:52.042771+0000","last_fullsized":"2023-01-15T14:34:52.042771+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:28.012","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.001697437,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.51","version":"394'328","reported_seq":1635,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.156362+0000","last_change":"2023-01-15T14:34:52.156362+0000","last_active":"2023-01-15T14:34:52.156362+0000","last_peered":"2023-01-15T14:34:52.156362+0000","last_clean":"2023-01-15T14:34:52.156362+0000","last_became_active":"2023-01-15T14:34:44.372829+0000","last_became_peered":"2023-01-15T14:34:44.372829+0000","last_unstale":"2023-01-15T14:34:52.156362+0000","last_undegraded":"2023-01-15T14:34:52.156362+0000","last_fullsized":"2023-01-15T14:34:52.156362+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.0031133350000000001,"stat_sum":{"num_bytes":7496813,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.50","version":"401'450","reported_seq":1713,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.089713+0000","last_change":"2023-01-15T14:34:52.089685+0000","last_active":"2023-01-15T14:34:52.089713+0000","last_peered":"2023-01-15T14:34:52.089713+0000","last_clean":"2023-01-15T14:34:52.089713+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:34:52.089713+0000","last_undegraded":"2023-01-15T14:34:52.089713+0000","last_fullsized":"2023-01-15T14:34:52.089713+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":450,"log_dups_size":0,"ondisk_log_size":450,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:37.144","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.77742515400000001,"stat_sum":{"num_bytes":15220933,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":56,"num_read_kb":52507,"num_write":59,"num_write_kb":21652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.4f","version":"155'145","reported_seq":972,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.609250+0000","last_change":"2023-01-15T14:34:52.609250+0000","last_active":"2023-01-15T14:34:52.609250+0000","last_peered":"2023-01-15T14:34:52.609250+0000","last_clean":"2023-01-15T14:34:52.609250+0000","last_became_active":"2023-01-15T14:34:46.178391+0000","last_became_peered":"2023-01-15T14:34:46.178391+0000","last_unstale":"2023-01-15T14:34:52.609250+0000","last_undegraded":"2023-01-15T14:34:52.609250+0000","last_fullsized":"2023-01-15T14:34:52.609250+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.56638766799999996,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.4e","version":"139'38","reported_seq":197,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.325638+0000","last_change":"2023-01-15T14:34:51.426416+0000","last_active":"2023-01-15T14:34:51.325638+0000","last_peered":"2023-01-15T14:34:51.325638+0000","last_clean":"2023-01-15T14:34:51.325638+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:34:51.325638+0000","last_undegraded":"2023-01-15T14:34:51.325638+0000","last_fullsized":"2023-01-15T14:34:51.325638+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.30839007400000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.4d","version":"392'288","reported_seq":1487,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.144867+0000","last_change":"2023-01-15T14:34:52.144843+0000","last_active":"2023-01-15T14:34:52.144867+0000","last_peered":"2023-01-15T14:34:52.144867+0000","last_clean":"2023-01-15T14:34:52.144867+0000","last_became_active":"2023-01-15T14:34:05.178613+0000","last_became_peered":"2023-01-15T14:34:05.178613+0000","last_unstale":"2023-01-15T14:34:52.144867+0000","last_undegraded":"2023-01-15T14:34:52.144867+0000","last_fullsized":"2023-01-15T14:34:52.144867+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":288,"log_dups_size":0,"ondisk_log_size":288,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.25171616499999999,"stat_sum":{"num_bytes":14846966,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":16,"num_read_kb":15961,"num_write":20,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.4c","version":"309'71","reported_seq":210,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.143079+0000","last_change":"2023-01-15T14:34:52.143030+0000","last_active":"2023-01-15T14:34:52.143079+0000","last_peered":"2023-01-15T14:34:52.143079+0000","last_clean":"2023-01-15T14:34:52.143079+0000","last_became_active":"2023-01-15T14:33:52.311859+0000","last_became_peered":"2023-01-15T14:33:52.311859+0000","last_unstale":"2023-01-15T14:34:52.143079+0000","last_undegraded":"2023-01-15T14:34:52.143079+0000","last_fullsized":"2023-01-15T14:34:52.143079+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:27.027","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.25295688500000002,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":43,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.061699+0000","last_change":"2023-01-15T14:34:53.062254+0000","last_active":"2023-01-15T14:34:53.061699+0000","last_peered":"2023-01-15T14:34:53.061699+0000","last_clean":"2023-01-15T14:34:53.061699+0000","last_became_active":"2023-01-15T14:34:43.357288+0000","last_became_peered":"2023-01-15T14:34:43.357288+0000","last_unstale":"2023-01-15T14:34:53.061699+0000","last_undegraded":"2023-01-15T14:34:53.061699+0000","last_fullsized":"2023-01-15T14:34:53.061699+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:34:44.116779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:44.116779+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:08.328","scrub_duration":0.50673716899999999,"objects_trimmed":0,"snaptrim_duration":0.96515980800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.4a","version":"390'391","reported_seq":1602,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.042548+0000","last_change":"2023-01-15T14:34:52.042504+0000","last_active":"2023-01-15T14:34:52.042548+0000","last_peered":"2023-01-15T14:34:52.042548+0000","last_clean":"2023-01-15T14:34:52.042548+0000","last_became_active":"2023-01-15T14:32:31.207733+0000","last_became_peered":"2023-01-15T14:32:31.207733+0000","last_unstale":"2023-01-15T14:34:52.042548+0000","last_undegraded":"2023-01-15T14:34:52.042548+0000","last_fullsized":"2023-01-15T14:34:52.042548+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:34.117","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0025649380000000001,"stat_sum":{"num_bytes":12470870,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":71,"num_read_kb":82380,"num_write":75,"num_write_kb":23626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.49","version":"110'149","reported_seq":1038,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.042485+0000","last_change":"2023-01-15T14:34:52.042434+0000","last_active":"2023-01-15T14:34:52.042485+0000","last_peered":"2023-01-15T14:34:52.042485+0000","last_clean":"2023-01-15T14:34:52.042485+0000","last_became_active":"2023-01-15T14:34:00.043221+0000","last_became_peered":"2023-01-15T14:34:00.043221+0000","last_unstale":"2023-01-15T14:34:52.042485+0000","last_undegraded":"2023-01-15T14:34:52.042485+0000","last_fullsized":"2023-01-15T14:34:52.042485+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.101880014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.48","version":"179'204","reported_seq":1465,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.311321+0000","last_change":"2023-01-15T14:34:53.312164+0000","last_active":"2023-01-15T14:34:53.311321+0000","last_peered":"2023-01-15T14:34:53.311321+0000","last_clean":"2023-01-15T14:34:53.311321+0000","last_became_active":"2023-01-15T14:33:55.375518+0000","last_became_peered":"2023-01-15T14:33:55.375518+0000","last_unstale":"2023-01-15T14:34:53.311321+0000","last_undegraded":"2023-01-15T14:34:53.311321+0000","last_fullsized":"2023-01-15T14:34:53.311321+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.223401234,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.47","version":"245'112","reported_seq":1239,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.995546+0000","last_change":"2023-01-15T14:34:52.191267+0000","last_active":"2023-01-15T14:34:52.995546+0000","last_peered":"2023-01-15T14:34:52.995546+0000","last_clean":"2023-01-15T14:34:52.995546+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:34:52.995546+0000","last_undegraded":"2023-01-15T14:34:52.995546+0000","last_fullsized":"2023-01-15T14:34:52.995546+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:34:21.450320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:21.450320+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:50.200","scrub_duration":0.30497759099999999,"objects_trimmed":0,"snaptrim_duration":0.12839761699999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.46","version":"102'52","reported_seq":30,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.257328+0000","last_change":"2023-01-15T14:34:52.257328+0000","last_active":"2023-01-15T14:34:52.257328+0000","last_peered":"2023-01-15T14:34:52.257328+0000","last_clean":"2023-01-15T14:34:52.257328+0000","last_became_active":"2023-01-15T14:34:43.367276+0000","last_became_peered":"2023-01-15T14:34:43.367276+0000","last_unstale":"2023-01-15T14:34:52.257328+0000","last_undegraded":"2023-01-15T14:34:52.257328+0000","last_fullsized":"2023-01-15T14:34:52.257328+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.20296336800000001,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2142,"num_write":3,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":1024,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.325038+0000","last_change":"2023-01-15T14:34:51.325583+0000","last_active":"2023-01-15T14:34:51.325038+0000","last_peered":"2023-01-15T14:34:51.325038+0000","last_clean":"2023-01-15T14:34:51.325038+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:34:51.325038+0000","last_undegraded":"2023-01-15T14:34:51.325038+0000","last_fullsized":"2023-01-15T14:34:51.325038+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:34:50.381073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:50.381073+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:56.850","scrub_duration":0.52191215300000005,"objects_trimmed":0,"snaptrim_duration":0.20763790300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.44","version":"245'268","reported_seq":1380,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.091988+0000","last_change":"2023-01-15T14:34:52.966562+0000","last_active":"2023-01-15T14:34:53.091988+0000","last_peered":"2023-01-15T14:34:53.091988+0000","last_clean":"2023-01-15T14:34:53.091988+0000","last_became_active":"2023-01-15T14:33:55.678019+0000","last_became_peered":"2023-01-15T14:33:55.678019+0000","last_unstale":"2023-01-15T14:34:53.091988+0000","last_undegraded":"2023-01-15T14:34:53.091988+0000","last_fullsized":"2023-01-15T14:34:53.091988+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.80280411900000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.43","version":"393'333","reported_seq":1634,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.661466+0000","last_change":"2023-01-15T14:34:51.662212+0000","last_active":"2023-01-15T14:34:51.661466+0000","last_peered":"2023-01-15T14:34:51.661466+0000","last_clean":"2023-01-15T14:34:51.661466+0000","last_became_active":"2023-01-15T14:34:02.211232+0000","last_became_peered":"2023-01-15T14:34:02.211232+0000","last_unstale":"2023-01-15T14:34:51.661466+0000","last_undegraded":"2023-01-15T14:34:51.661466+0000","last_fullsized":"2023-01-15T14:34:51.661466+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.44349906,"stat_sum":{"num_bytes":9109625,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":13,"num_read_kb":16586,"num_write":12,"num_write_kb":5193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3395080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.42","version":"155'101","reported_seq":1141,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.414554+0000","last_change":"2023-01-15T14:34:53.415348+0000","last_active":"2023-01-15T14:34:53.414554+0000","last_peered":"2023-01-15T14:34:53.414554+0000","last_clean":"2023-01-15T14:34:53.414554+0000","last_became_active":"2023-01-15T14:33:55.376782+0000","last_became_peered":"2023-01-15T14:33:55.376782+0000","last_unstale":"2023-01-15T14:34:53.414554+0000","last_undegraded":"2023-01-15T14:34:53.414554+0000","last_fullsized":"2023-01-15T14:34:53.414554+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"103'86","last_scrub_stamp":"2023-01-15T14:30:09.901725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:09.901725+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.42553845400000001,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.41","version":"155'121","reported_seq":1260,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.191874+0000","last_change":"2023-01-15T14:34:52.965674+0000","last_active":"2023-01-15T14:34:53.191874+0000","last_peered":"2023-01-15T14:34:53.191874+0000","last_clean":"2023-01-15T14:34:53.191874+0000","last_became_active":"2023-01-15T14:34:46.172046+0000","last_became_peered":"2023-01-15T14:34:46.172046+0000","last_unstale":"2023-01-15T14:34:53.191874+0000","last_undegraded":"2023-01-15T14:34:53.191874+0000","last_fullsized":"2023-01-15T14:34:53.191874+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.80193903499999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.40","version":"401'328","reported_seq":1573,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.090346+0000","last_change":"2023-01-15T14:34:52.090317+0000","last_active":"2023-01-15T14:34:52.090346+0000","last_peered":"2023-01-15T14:34:52.090346+0000","last_clean":"2023-01-15T14:34:52.090346+0000","last_became_active":"2023-01-15T14:33:58.117085+0000","last_became_peered":"2023-01-15T14:33:58.117085+0000","last_unstale":"2023-01-15T14:34:52.090346+0000","last_undegraded":"2023-01-15T14:34:52.090346+0000","last_fullsized":"2023-01-15T14:34:52.090346+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.85843142800000005,"stat_sum":{"num_bytes":14103500,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":35542,"num_write":33,"num_write_kb":12321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":29167491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.3f","version":"400'245","reported_seq":1483,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.044687+0000","last_change":"2023-01-15T14:34:51.145975+0000","last_active":"2023-01-15T14:34:52.044687+0000","last_peered":"2023-01-15T14:34:52.044687+0000","last_clean":"2023-01-15T14:34:52.044687+0000","last_became_active":"2023-01-15T14:33:58.217783+0000","last_became_peered":"2023-01-15T14:33:58.217783+0000","last_unstale":"2023-01-15T14:34:52.044687+0000","last_undegraded":"2023-01-15T14:34:52.044687+0000","last_fullsized":"2023-01-15T14:34:52.044687+0000","mapping_epoch":348,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.090074175000000006,"stat_sum":{"num_bytes":10698066,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":16,"num_write_kb":3728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6605283,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"3.3e","version":"381'64","reported_seq":48,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.244605+0000","last_change":"2023-01-15T14:34:52.244605+0000","last_active":"2023-01-15T14:34:52.244605+0000","last_peered":"2023-01-15T14:34:52.244605+0000","last_clean":"2023-01-15T14:34:52.244605+0000","last_became_active":"2023-01-15T14:34:43.468893+0000","last_became_peered":"2023-01-15T14:34:43.468893+0000","last_unstale":"2023-01-15T14:34:52.244605+0000","last_undegraded":"2023-01-15T14:34:52.244605+0000","last_fullsized":"2023-01-15T14:34:52.244605+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.270279983,"stat_sum":{"num_bytes":5664091,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":4255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":11328182,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.3d","version":"394'252","reported_seq":1503,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.322456+0000","last_change":"2023-01-15T14:34:51.323372+0000","last_active":"2023-01-15T14:34:51.322456+0000","last_peered":"2023-01-15T14:34:51.322456+0000","last_clean":"2023-01-15T14:34:51.322456+0000","last_became_active":"2023-01-15T14:34:44.167696+0000","last_became_peered":"2023-01-15T14:34:44.167696+0000","last_unstale":"2023-01-15T14:34:51.322456+0000","last_undegraded":"2023-01-15T14:34:51.322456+0000","last_fullsized":"2023-01-15T14:34:51.322456+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.20584090299999999,"stat_sum":{"num_bytes":8050279,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":35,"num_read_kb":41385,"num_write":18,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":14705983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.3c","version":"141'46","reported_seq":456,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.607643+0000","last_change":"2023-01-15T14:34:52.607643+0000","last_active":"2023-01-15T14:34:52.607643+0000","last_peered":"2023-01-15T14:34:52.607643+0000","last_clean":"2023-01-15T14:34:52.607643+0000","last_became_active":"2023-01-15T14:31:54.140607+0000","last_became_peered":"2023-01-15T14:31:54.140607+0000","last_unstale":"2023-01-15T14:34:52.607643+0000","last_undegraded":"2023-01-15T14:34:52.607643+0000","last_fullsized":"2023-01-15T14:34:52.607643+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.56543953800000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":128,"reported_epoch":403,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:53.459548+0000","last_change":"2023-01-15T14:34:53.459604+0000","last_active":"2023-01-15T14:34:53.459548+0000","last_peered":"2023-01-15T14:34:53.459548+0000","last_clean":"2023-01-15T14:34:53.459548+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:34:53.459548+0000","last_undegraded":"2023-01-15T14:34:53.459548+0000","last_fullsized":"2023-01-15T14:34:53.459548+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.139138545,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1487,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.662893+0000","last_change":"2023-01-15T14:34:51.764051+0000","last_active":"2023-01-15T14:34:51.662893+0000","last_peered":"2023-01-15T14:34:51.662893+0000","last_clean":"2023-01-15T14:34:51.662893+0000","last_became_active":"2023-01-15T14:34:46.572815+0000","last_became_peered":"2023-01-15T14:34:46.572815+0000","last_unstale":"2023-01-15T14:34:51.662893+0000","last_undegraded":"2023-01-15T14:34:51.662893+0000","last_fullsized":"2023-01-15T14:34:51.662893+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.51877381300000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.39","version":"402'399","reported_seq":1642,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.550653+0000","last_change":"2023-01-15T14:34:52.550653+0000","last_active":"2023-01-15T14:34:52.550653+0000","last_peered":"2023-01-15T14:34:52.550653+0000","last_clean":"2023-01-15T14:34:52.550653+0000","last_became_active":"2023-01-15T14:34:08.023332+0000","last_became_peered":"2023-01-15T14:34:08.023332+0000","last_unstale":"2023-01-15T14:34:52.550653+0000","last_undegraded":"2023-01-15T14:34:52.550653+0000","last_fullsized":"2023-01-15T14:34:52.550653+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.50806326999999996,"stat_sum":{"num_bytes":30651370,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":67,"num_read_kb":114793,"num_write":18,"num_write_kb":12463,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":26862613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.38","version":"403'335","reported_seq":1656,"reported_epoch":403,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:53.419322+0000","last_change":"2023-01-15T14:34:53.415486+0000","last_active":"2023-01-15T14:34:53.419322+0000","last_peered":"2023-01-15T14:34:53.419322+0000","last_clean":"2023-01-15T14:34:53.419322+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:34:53.419322+0000","last_undegraded":"2023-01-15T14:34:53.419322+0000","last_fullsized":"2023-01-15T14:34:53.419322+0000","mapping_epoch":232,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:32.007","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.42436911500000002,"stat_sum":{"num_bytes":8461433,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":45,"num_read_kb":42848,"num_write":40,"num_write_kb":13455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.37","version":"200'78","reported_seq":532,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.045195+0000","last_change":"2023-01-15T14:34:51.146779+0000","last_active":"2023-01-15T14:34:52.045195+0000","last_peered":"2023-01-15T14:34:52.045195+0000","last_clean":"2023-01-15T14:34:52.045195+0000","last_became_active":"2023-01-15T14:34:43.156532+0000","last_became_peered":"2023-01-15T14:34:43.156532+0000","last_unstale":"2023-01-15T14:34:52.045195+0000","last_undegraded":"2023-01-15T14:34:52.045195+0000","last_fullsized":"2023-01-15T14:34:52.045195+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.79557248199999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"3.36","version":"194'81","reported_seq":44,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.445177+0000","last_change":"2023-01-15T14:34:52.445177+0000","last_active":"2023-01-15T14:34:52.445177+0000","last_peered":"2023-01-15T14:34:52.445177+0000","last_clean":"2023-01-15T14:34:52.445177+0000","last_became_active":"2023-01-15T14:34:43.367667+0000","last_became_peered":"2023-01-15T14:34:43.367667+0000","last_unstale":"2023-01-15T14:34:52.445177+0000","last_undegraded":"2023-01-15T14:34:52.445177+0000","last_fullsized":"2023-01-15T14:34:52.445177+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:34:51.296526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.296526+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:12.007","scrub_duration":0.66934546500000003,"objects_trimmed":0,"snaptrim_duration":0.20144030199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.35","version":"110'55","reported_seq":136,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.560022+0000","last_change":"2023-01-15T14:34:51.661372+0000","last_active":"2023-01-15T14:34:51.560022+0000","last_peered":"2023-01-15T14:34:51.560022+0000","last_clean":"2023-01-15T14:34:51.560022+0000","last_became_active":"2023-01-15T14:33:52.612576+0000","last_became_peered":"2023-01-15T14:33:52.612576+0000","last_unstale":"2023-01-15T14:34:51.560022+0000","last_undegraded":"2023-01-15T14:34:51.560022+0000","last_fullsized":"2023-01-15T14:34:51.560022+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:27.799","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44338971300000002,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.34","version":"169'138","reported_seq":1265,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.092537+0000","last_change":"2023-01-15T14:34:52.293944+0000","last_active":"2023-01-15T14:34:53.092537+0000","last_peered":"2023-01-15T14:34:53.092537+0000","last_clean":"2023-01-15T14:34:53.092537+0000","last_became_active":"2023-01-15T14:31:39.925178+0000","last_became_peered":"2023-01-15T14:31:39.925178+0000","last_unstale":"2023-01-15T14:34:53.092537+0000","last_undegraded":"2023-01-15T14:34:53.092537+0000","last_fullsized":"2023-01-15T14:34:53.092537+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.19423288499999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.33","version":"204'275","reported_seq":1378,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.606527+0000","last_change":"2023-01-15T14:34:52.606527+0000","last_active":"2023-01-15T14:34:52.606527+0000","last_peered":"2023-01-15T14:34:52.606527+0000","last_clean":"2023-01-15T14:34:52.606527+0000","last_became_active":"2023-01-15T14:34:46.268250+0000","last_became_peered":"2023-01-15T14:34:46.268250+0000","last_unstale":"2023-01-15T14:34:52.606527+0000","last_undegraded":"2023-01-15T14:34:52.606527+0000","last_fullsized":"2023-01-15T14:34:52.606527+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:26.972","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.56460682100000004,"stat_sum":{"num_bytes":6970954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":22378,"num_write":14,"num_write_kb":6571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.32","version":"394'169","reported_seq":970,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.398227+0000","last_change":"2023-01-15T14:34:51.398687+0000","last_active":"2023-01-15T14:34:51.398227+0000","last_peered":"2023-01-15T14:34:51.398227+0000","last_clean":"2023-01-15T14:34:51.398227+0000","last_became_active":"2023-01-15T14:34:44.468157+0000","last_became_peered":"2023-01-15T14:34:44.468157+0000","last_unstale":"2023-01-15T14:34:51.398227+0000","last_undegraded":"2023-01-15T14:34:51.398227+0000","last_fullsized":"2023-01-15T14:34:51.398227+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.25366604100000001,"stat_sum":{"num_bytes":5327885,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":7048,"num_write":1,"num_write_kb":2856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5327885,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.13","version":"398'384","reported_seq":1508,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.045329+0000","last_change":"2023-01-15T14:34:51.146271+0000","last_active":"2023-01-15T14:34:52.045329+0000","last_peered":"2023-01-15T14:34:52.045329+0000","last_clean":"2023-01-15T14:34:52.045329+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:34:52.045329+0000","last_undegraded":"2023-01-15T14:34:52.045329+0000","last_fullsized":"2023-01-15T14:34:52.045329+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"374'376","last_scrub_stamp":"2023-01-15T14:34:32.489483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.489483+0000","objects_scrubbed":4,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:32.530","scrub_duration":0.381937625,"objects_trimmed":0,"snaptrim_duration":0.45234028999999998,"stat_sum":{"num_bytes":9889842,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":2561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"3.12","version":"389'201","reported_seq":1016,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.397513+0000","last_change":"2023-01-15T14:34:51.398158+0000","last_active":"2023-01-15T14:34:51.397513+0000","last_peered":"2023-01-15T14:34:51.397513+0000","last_clean":"2023-01-15T14:34:51.397513+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:34:51.397513+0000","last_undegraded":"2023-01-15T14:34:51.397513+0000","last_fullsized":"2023-01-15T14:34:51.397513+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.25328467599999999,"stat_sum":{"num_bytes":10304297,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":20,"num_write_kb":9403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.11","version":"401'331","reported_seq":1695,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.608495+0000","last_change":"2023-01-15T14:34:52.608495+0000","last_active":"2023-01-15T14:34:52.608495+0000","last_peered":"2023-01-15T14:34:52.608495+0000","last_clean":"2023-01-15T14:34:52.608495+0000","last_became_active":"2023-01-15T14:34:09.541429+0000","last_became_peered":"2023-01-15T14:34:09.541429+0000","last_unstale":"2023-01-15T14:34:52.608495+0000","last_undegraded":"2023-01-15T14:34:52.608495+0000","last_fullsized":"2023-01-15T14:34:52.608495+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":331,"log_dups_size":0,"ondisk_log_size":331,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:21.075","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.36458336899999999,"stat_sum":{"num_bytes":15559313,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":45,"num_read_kb":48289,"num_write":29,"num_write_kb":17520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":20307357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.10","version":"401'416","reported_seq":1511,"reported_epoch":401,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:51.499659+0000","last_change":"2023-01-15T14:34:51.499659+0000","last_active":"2023-01-15T14:34:51.499659+0000","last_peered":"2023-01-15T14:34:51.499659+0000","last_clean":"2023-01-15T14:34:51.499659+0000","last_became_active":"2023-01-15T14:34:16.272883+0000","last_became_peered":"2023-01-15T14:34:16.272883+0000","last_unstale":"2023-01-15T14:34:51.499659+0000","last_undegraded":"2023-01-15T14:34:51.499659+0000","last_fullsized":"2023-01-15T14:34:51.499659+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.89805710800000005,"stat_sum":{"num_bytes":8658713,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":54,"num_read_kb":52673,"num_write":50,"num_write_kb":18045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12804882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"127","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1339,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.043820+0000","last_change":"2023-01-15T14:34:52.043771+0000","last_active":"2023-01-15T14:34:52.043820+0000","last_peered":"2023-01-15T14:34:52.043820+0000","last_clean":"2023-01-15T14:34:52.043820+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:34:52.043820+0000","last_undegraded":"2023-01-15T14:34:52.043820+0000","last_fullsized":"2023-01-15T14:34:52.043820+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.77813042600000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.e","version":"139'38","reported_seq":150,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.145134+0000","last_change":"2023-01-15T14:34:52.145107+0000","last_active":"2023-01-15T14:34:52.145134+0000","last_peered":"2023-01-15T14:34:52.145134+0000","last_clean":"2023-01-15T14:34:52.145134+0000","last_became_active":"2023-01-15T14:34:43.265641+0000","last_became_peered":"2023-01-15T14:34:43.265641+0000","last_unstale":"2023-01-15T14:34:52.145134+0000","last_undegraded":"2023-01-15T14:34:52.145134+0000","last_fullsized":"2023-01-15T14:34:52.145134+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.25399587000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.d","version":"296'232","reported_seq":1398,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.090005+0000","last_change":"2023-01-15T14:34:51.146545+0000","last_active":"2023-01-15T14:34:52.090005+0000","last_peered":"2023-01-15T14:34:52.090005+0000","last_clean":"2023-01-15T14:34:52.090005+0000","last_became_active":"2023-01-15T14:33:20.440734+0000","last_became_peered":"2023-01-15T14:33:20.440734+0000","last_unstale":"2023-01-15T14:34:52.090005+0000","last_undegraded":"2023-01-15T14:34:52.090005+0000","last_fullsized":"2023-01-15T14:34:52.090005+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:22.089","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":1.213506465,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"3.c","version":"403'114","reported_seq":1295,"reported_epoch":403,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:53.456598+0000","last_change":"2023-01-15T14:34:53.456598+0000","last_active":"2023-01-15T14:34:53.456598+0000","last_peered":"2023-01-15T14:34:53.456598+0000","last_clean":"2023-01-15T14:34:53.456598+0000","last_became_active":"2023-01-15T14:34:48.641120+0000","last_became_peered":"2023-01-15T14:34:48.641120+0000","last_unstale":"2023-01-15T14:34:53.456598+0000","last_undegraded":"2023-01-15T14:34:53.456598+0000","last_fullsized":"2023-01-15T14:34:53.456598+0000","mapping_epoch":397,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.50306588699999999,"stat_sum":{"num_bytes":3264587,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":41975,"num_write":11,"num_write_kb":1706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":21533794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.b","version":"155'53","reported_seq":1110,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.455564+0000","last_change":"2023-01-15T14:34:53.456457+0000","last_active":"2023-01-15T14:34:53.455564+0000","last_peered":"2023-01-15T14:34:53.455564+0000","last_clean":"2023-01-15T14:34:53.455564+0000","last_became_active":"2023-01-15T14:33:52.425622+0000","last_became_peered":"2023-01-15T14:33:52.425622+0000","last_unstale":"2023-01-15T14:34:53.455564+0000","last_undegraded":"2023-01-15T14:34:53.455564+0000","last_fullsized":"2023-01-15T14:34:53.455564+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.36705569799999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.a","version":"252'279","reported_seq":1473,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.092240+0000","last_change":"2023-01-15T14:34:52.873040+0000","last_active":"2023-01-15T14:34:53.092240+0000","last_peered":"2023-01-15T14:34:53.092240+0000","last_clean":"2023-01-15T14:34:53.092240+0000","last_became_active":"2023-01-15T14:34:46.374272+0000","last_became_peered":"2023-01-15T14:34:46.374272+0000","last_unstale":"2023-01-15T14:34:53.092240+0000","last_undegraded":"2023-01-15T14:34:53.092240+0000","last_fullsized":"2023-01-15T14:34:53.092240+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.71030521000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.9","version":"110'149","reported_seq":1219,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.192159+0000","last_change":"2023-01-15T14:34:53.067516+0000","last_active":"2023-01-15T14:34:53.192159+0000","last_peered":"2023-01-15T14:34:53.192159+0000","last_clean":"2023-01-15T14:34:53.192159+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:34:53.192159+0000","last_undegraded":"2023-01-15T14:34:53.192159+0000","last_fullsized":"2023-01-15T14:34:53.192159+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.903084573,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.8","version":"179'204","reported_seq":1487,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.647667+0000","last_change":"2023-01-15T14:34:52.647667+0000","last_active":"2023-01-15T14:34:52.647667+0000","last_peered":"2023-01-15T14:34:52.647667+0000","last_clean":"2023-01-15T14:34:52.647667+0000","last_became_active":"2023-01-15T14:34:46.365412+0000","last_became_peered":"2023-01-15T14:34:46.365412+0000","last_unstale":"2023-01-15T14:34:52.647667+0000","last_undegraded":"2023-01-15T14:34:52.647667+0000","last_fullsized":"2023-01-15T14:34:52.647667+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:34:48.269667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.269667+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:13.159","scrub_duration":0.20349077600000001,"objects_trimmed":0,"snaptrim_duration":0.56629040500000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"1.0","version":"19'117","reported_seq":576,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.088453+0000","last_change":"2023-01-15T14:34:44.272736+0000","last_active":"2023-01-15T14:34:52.088453+0000","last_peered":"2023-01-15T14:34:52.088453+0000","last_clean":"2023-01-15T14:34:52.088453+0000","last_became_active":"2023-01-15T14:34:44.171419+0000","last_became_peered":"2023-01-15T14:34:44.171419+0000","last_unstale":"2023-01-15T14:34:52.088453+0000","last_undegraded":"2023-01-15T14:34:52.088453+0000","last_fullsized":"2023-01-15T14:34:52.088453+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":66,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.043180+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:34:52.043180+0000","last_peered":"2023-01-15T14:34:52.043180+0000","last_clean":"2023-01-15T14:34:52.043180+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:34:52.043180+0000","last_undegraded":"2023-01-15T14:34:52.043180+0000","last_fullsized":"2023-01-15T14:34:52.043180+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":1146,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.192061+0000","last_change":"2023-01-15T14:34:52.872149+0000","last_active":"2023-01-15T14:34:53.192061+0000","last_peered":"2023-01-15T14:34:53.192061+0000","last_clean":"2023-01-15T14:34:53.192061+0000","last_became_active":"2023-01-15T14:34:46.376300+0000","last_became_peered":"2023-01-15T14:34:46.376300+0000","last_unstale":"2023-01-15T14:34:53.192061+0000","last_undegraded":"2023-01-15T14:34:53.192061+0000","last_fullsized":"2023-01-15T14:34:53.192061+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.70970878800000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"2.0","version":"0'0","reported_seq":469,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.090631+0000","last_change":"2023-01-15T14:34:43.269231+0000","last_active":"2023-01-15T14:34:52.090631+0000","last_peered":"2023-01-15T14:34:52.090631+0000","last_clean":"2023-01-15T14:34:52.090631+0000","last_became_active":"2023-01-15T14:34:43.268688+0000","last_became_peered":"2023-01-15T14:34:43.268688+0000","last_unstale":"2023-01-15T14:34:52.090631+0000","last_undegraded":"2023-01-15T14:34:52.090631+0000","last_fullsized":"2023-01-15T14:34:52.090631+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":812,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.398970+0000","last_change":"2023-01-15T14:34:51.499572+0000","last_active":"2023-01-15T14:34:51.398970+0000","last_peered":"2023-01-15T14:34:51.398970+0000","last_clean":"2023-01-15T14:34:51.398970+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:34:51.398970+0000","last_undegraded":"2023-01-15T14:34:51.398970+0000","last_fullsized":"2023-01-15T14:34:51.398970+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.283272986,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"2.1","version":"0'0","reported_seq":204,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.043866+0000","last_change":"2023-01-15T14:31:38.232879+0000","last_active":"2023-01-15T14:34:52.043866+0000","last_peered":"2023-01-15T14:34:52.043866+0000","last_clean":"2023-01-15T14:34:52.043866+0000","last_became_active":"2023-01-15T14:31:38.232119+0000","last_became_peered":"2023-01-15T14:31:38.232119+0000","last_unstale":"2023-01-15T14:34:52.043866+0000","last_undegraded":"2023-01-15T14:34:52.043866+0000","last_fullsized":"2023-01-15T14:34:52.043866+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1036,"reported_epoch":402,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:53.062369+0000","last_change":"2023-01-15T14:34:53.062488+0000","last_active":"2023-01-15T14:34:53.062369+0000","last_peered":"2023-01-15T14:34:53.062369+0000","last_clean":"2023-01-15T14:34:53.062369+0000","last_became_active":"2023-01-15T14:34:46.370892+0000","last_became_peered":"2023-01-15T14:34:46.370892+0000","last_unstale":"2023-01-15T14:34:53.062369+0000","last_undegraded":"2023-01-15T14:34:53.062369+0000","last_fullsized":"2023-01-15T14:34:53.062369+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:32:07.111399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:07.111399+0000","objects_scrubbed":2,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.00104685,"stat_sum":{"num_bytes":4294221,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":1938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.249649+0000","last_change":"2023-01-15T14:34:43.272032+0000","last_active":"2023-01-15T14:34:51.249649+0000","last_peered":"2023-01-15T14:34:51.249649+0000","last_clean":"2023-01-15T14:34:51.249649+0000","last_became_active":"2023-01-15T14:34:43.271645+0000","last_became_peered":"2023-01-15T14:34:43.271645+0000","last_unstale":"2023-01-15T14:34:51.249649+0000","last_undegraded":"2023-01-15T14:34:51.249649+0000","last_fullsized":"2023-01-15T14:34:51.249649+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"402'337","reported_seq":1586,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.577160+0000","last_change":"2023-01-15T14:34:53.577160+0000","last_active":"2023-01-15T14:34:53.577160+0000","last_peered":"2023-01-15T14:34:53.577160+0000","last_clean":"2023-01-15T14:34:53.577160+0000","last_became_active":"2023-01-15T14:34:43.258728+0000","last_became_peered":"2023-01-15T14:34:43.258728+0000","last_unstale":"2023-01-15T14:34:53.577160+0000","last_undegraded":"2023-01-15T14:34:53.577160+0000","last_fullsized":"2023-01-15T14:34:53.577160+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":337,"log_dups_size":0,"ondisk_log_size":337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.485319206,"stat_sum":{"num_bytes":10343843,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":13875,"num_write":6,"num_write_kb":7330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":21227736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.5","version":"401'179","reported_seq":1345,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.312284+0000","last_change":"2023-01-15T14:34:53.413134+0000","last_active":"2023-01-15T14:34:53.312284+0000","last_peered":"2023-01-15T14:34:53.312284+0000","last_clean":"2023-01-15T14:34:53.312284+0000","last_became_active":"2023-01-15T14:34:04.175473+0000","last_became_peered":"2023-01-15T14:34:04.175473+0000","last_unstale":"2023-01-15T14:34:53.312284+0000","last_undegraded":"2023-01-15T14:34:53.312284+0000","last_fullsized":"2023-01-15T14:34:53.312284+0000","mapping_epoch":354,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.22256463500000001,"stat_sum":{"num_bytes":15163676,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":26147,"num_write":17,"num_write_kb":6661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":13260921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.249987+0000","last_change":"2023-01-15T14:34:43.371445+0000","last_active":"2023-01-15T14:34:51.249987+0000","last_peered":"2023-01-15T14:34:51.249987+0000","last_clean":"2023-01-15T14:34:51.249987+0000","last_became_active":"2023-01-15T14:34:43.253700+0000","last_became_peered":"2023-01-15T14:34:43.253700+0000","last_unstale":"2023-01-15T14:34:51.249987+0000","last_undegraded":"2023-01-15T14:34:51.249987+0000","last_fullsized":"2023-01-15T14:34:51.249987+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"402'333","reported_seq":1631,"reported_epoch":402,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:34:52.861260+0000","last_change":"2023-01-15T14:34:52.861260+0000","last_active":"2023-01-15T14:34:52.861260+0000","last_peered":"2023-01-15T14:34:52.861260+0000","last_clean":"2023-01-15T14:34:52.861260+0000","last_became_active":"2023-01-15T14:34:44.164446+0000","last_became_peered":"2023-01-15T14:34:44.164446+0000","last_unstale":"2023-01-15T14:34:52.861260+0000","last_undegraded":"2023-01-15T14:34:52.861260+0000","last_fullsized":"2023-01-15T14:34:52.861260+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:32.407","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0.0020006809999999998,"stat_sum":{"num_bytes":8290510,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":10,"num_read_kb":9789,"num_write":4,"num_write_kb":1017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"2.5","version":"0'0","reported_seq":451,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.090566+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:34:52.090566+0000","last_peered":"2023-01-15T14:34:52.090566+0000","last_clean":"2023-01-15T14:34:52.090566+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:34:52.090566+0000","last_undegraded":"2023-01-15T14:34:52.090566+0000","last_fullsized":"2023-01-15T14:34:52.090566+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"402'163","reported_seq":1276,"reported_epoch":403,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:53.458342+0000","last_change":"2023-01-15T14:34:53.458428+0000","last_active":"2023-01-15T14:34:53.458342+0000","last_peered":"2023-01-15T14:34:53.458342+0000","last_clean":"2023-01-15T14:34:53.458342+0000","last_became_active":"2023-01-15T14:34:17.186013+0000","last_became_peered":"2023-01-15T14:34:17.186013+0000","last_unstale":"2023-01-15T14:34:53.458342+0000","last_undegraded":"2023-01-15T14:34:53.458342+0000","last_fullsized":"2023-01-15T14:34:53.458342+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.201544111,"stat_sum":{"num_bytes":12609512,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":30,"num_read_kb":42005,"num_write":17,"num_write_kb":8618,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3546343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.365721+0000","last_change":"2023-01-15T14:34:43.566133+0000","last_active":"2023-01-15T14:34:51.365721+0000","last_peered":"2023-01-15T14:34:51.365721+0000","last_clean":"2023-01-15T14:34:51.365721+0000","last_became_active":"2023-01-15T14:34:43.265211+0000","last_became_peered":"2023-01-15T14:34:43.265211+0000","last_unstale":"2023-01-15T14:34:51.365721+0000","last_undegraded":"2023-01-15T14:34:51.365721+0000","last_fullsized":"2023-01-15T14:34:51.365721+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":156,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.323656+0000","last_change":"2023-01-15T14:34:51.324359+0000","last_active":"2023-01-15T14:34:51.323656+0000","last_peered":"2023-01-15T14:34:51.323656+0000","last_clean":"2023-01-15T14:34:51.323656+0000","last_became_active":"2023-01-15T14:34:43.461395+0000","last_became_peered":"2023-01-15T14:34:43.461395+0000","last_unstale":"2023-01-15T14:34:51.323656+0000","last_undegraded":"2023-01-15T14:34:51.323656+0000","last_fullsized":"2023-01-15T14:34:51.323656+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.20653734500000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"2.7","version":"0'0","reported_seq":66,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.044626+0000","last_change":"2023-01-15T14:33:57.412557+0000","last_active":"2023-01-15T14:34:52.044626+0000","last_peered":"2023-01-15T14:34:52.044626+0000","last_clean":"2023-01-15T14:34:52.044626+0000","last_became_active":"2023-01-15T14:33:57.311981+0000","last_became_peered":"2023-01-15T14:33:57.311981+0000","last_unstale":"2023-01-15T14:34:52.044626+0000","last_undegraded":"2023-01-15T14:34:52.044626+0000","last_fullsized":"2023-01-15T14:34:52.044626+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"402'169","reported_seq":1312,"reported_epoch":404,"state":"active+clean","last_fresh":"2023-01-15T14:34:54.386456+0000","last_change":"2023-01-15T14:34:52.871112+0000","last_active":"2023-01-15T14:34:54.386456+0000","last_peered":"2023-01-15T14:34:54.386456+0000","last_clean":"2023-01-15T14:34:54.386456+0000","last_became_active":"2023-01-15T14:33:20.242697+0000","last_became_peered":"2023-01-15T14:33:20.242697+0000","last_unstale":"2023-01-15T14:34:54.386456+0000","last_undegraded":"2023-01-15T14:34:54.386456+0000","last_fullsized":"2023-01-15T14:34:54.386456+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"351'159","last_scrub_stamp":"2023-01-15T14:34:18.607751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:18.607751+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:36.520","scrub_duration":0.40666956999999998,"objects_trimmed":0,"snaptrim_duration":0.77011770300000004,"stat_sum":{"num_bytes":2453855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.15","version":"110'55","reported_seq":128,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.044080+0000","last_change":"2023-01-15T14:34:52.044032+0000","last_active":"2023-01-15T14:34:52.044080+0000","last_peered":"2023-01-15T14:34:52.044080+0000","last_clean":"2023-01-15T14:34:52.044080+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:34:52.044080+0000","last_undegraded":"2023-01-15T14:34:52.044080+0000","last_fullsized":"2023-01-15T14:34:52.044080+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.24840472599999999,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.16","version":"395'143","reported_seq":851,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.043789+0000","last_change":"2023-01-15T14:34:52.043744+0000","last_active":"2023-01-15T14:34:52.043789+0000","last_peered":"2023-01-15T14:34:52.043789+0000","last_clean":"2023-01-15T14:34:52.043789+0000","last_became_active":"2023-01-15T14:34:44.491303+0000","last_became_peered":"2023-01-15T14:34:44.491303+0000","last_unstale":"2023-01-15T14:34:52.043789+0000","last_undegraded":"2023-01-15T14:34:52.043789+0000","last_fullsized":"2023-01-15T14:34:52.043789+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:01.517","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.10343862700000001,"stat_sum":{"num_bytes":7378471,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":6898,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9731517,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.17","version":"278'93","reported_seq":156,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.764707+0000","last_change":"2023-01-15T14:34:51.765421+0000","last_active":"2023-01-15T14:34:51.764707+0000","last_peered":"2023-01-15T14:34:51.764707+0000","last_clean":"2023-01-15T14:34:51.764707+0000","last_became_active":"2023-01-15T14:34:43.264755+0000","last_became_peered":"2023-01-15T14:34:43.264755+0000","last_unstale":"2023-01-15T14:34:51.764707+0000","last_undegraded":"2023-01-15T14:34:51.764707+0000","last_fullsized":"2023-01-15T14:34:51.764707+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:29.773","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.44665146700000002,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.18","version":"402'346","reported_seq":1696,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.311153+0000","last_change":"2023-01-15T14:34:53.311153+0000","last_active":"2023-01-15T14:34:53.311153+0000","last_peered":"2023-01-15T14:34:53.311153+0000","last_clean":"2023-01-15T14:34:53.311153+0000","last_became_active":"2023-01-15T14:34:44.375124+0000","last_became_peered":"2023-01-15T14:34:44.375124+0000","last_unstale":"2023-01-15T14:34:53.311153+0000","last_undegraded":"2023-01-15T14:34:53.311153+0000","last_fullsized":"2023-01-15T14:34:53.311153+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"384'341","last_scrub_stamp":"2023-01-15T14:34:35.477211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:35.477211+0000","objects_scrubbed":5,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:59.843","scrub_duration":0.54215560600000001,"objects_trimmed":0,"snaptrim_duration":0.221069722,"stat_sum":{"num_bytes":9299174,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.19","version":"372'259","reported_seq":1433,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.156960+0000","last_change":"2023-01-15T14:34:52.156960+0000","last_active":"2023-01-15T14:34:52.156960+0000","last_peered":"2023-01-15T14:34:52.156960+0000","last_clean":"2023-01-15T14:34:52.156960+0000","last_became_active":"2023-01-15T14:34:43.263173+0000","last_became_peered":"2023-01-15T14:34:43.263173+0000","last_unstale":"2023-01-15T14:34:52.156960+0000","last_undegraded":"2023-01-15T14:34:52.156960+0000","last_fullsized":"2023-01-15T14:34:52.156960+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:34:55.595","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.0028969909999999998,"stat_sum":{"num_bytes":4941423,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":4,"num_write_kb":1230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4941423,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.1a","version":"402'361","reported_seq":1602,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.091901+0000","last_change":"2023-01-15T14:34:52.964787+0000","last_active":"2023-01-15T14:34:53.091901+0000","last_peered":"2023-01-15T14:34:53.091901+0000","last_clean":"2023-01-15T14:34:53.091901+0000","last_became_active":"2023-01-15T14:34:44.273541+0000","last_became_peered":"2023-01-15T14:34:44.273541+0000","last_unstale":"2023-01-15T14:34:53.091901+0000","last_undegraded":"2023-01-15T14:34:53.091901+0000","last_fullsized":"2023-01-15T14:34:53.091901+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"375'356","last_scrub_stamp":"2023-01-15T14:34:29.460843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:29.460843+0000","objects_scrubbed":4,"log_size":361,"log_dups_size":0,"ondisk_log_size":361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:56.905","scrub_duration":0.30741503599999997,"objects_trimmed":0,"snaptrim_duration":0.80273172699999995,"stat_sum":{"num_bytes":4071594,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.1b","version":"389'115","reported_seq":186,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.324421+0000","last_change":"2023-01-15T14:34:51.324989+0000","last_active":"2023-01-15T14:34:51.324421+0000","last_peered":"2023-01-15T14:34:51.324421+0000","last_clean":"2023-01-15T14:34:51.324421+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:34:51.324421+0000","last_undegraded":"2023-01-15T14:34:51.324421+0000","last_fullsized":"2023-01-15T14:34:51.324421+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"370'101","last_scrub_stamp":"2023-01-15T14:34:24.137270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:24.137270+0000","objects_scrubbed":3,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:44.214","scrub_duration":0.106791863,"objects_trimmed":0,"snaptrim_duration":0.207168344,"stat_sum":{"num_bytes":8765935,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.1c","version":"141'46","reported_seq":688,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.192487+0000","last_change":"2023-01-15T14:34:52.267126+0000","last_active":"2023-01-15T14:34:53.192487+0000","last_peered":"2023-01-15T14:34:53.192487+0000","last_clean":"2023-01-15T14:34:53.192487+0000","last_became_active":"2023-01-15T14:34:43.163678+0000","last_became_peered":"2023-01-15T14:34:43.163678+0000","last_unstale":"2023-01-15T14:34:53.192487+0000","last_undegraded":"2023-01-15T14:34:53.192487+0000","last_fullsized":"2023-01-15T14:34:53.192487+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:22.440826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:22.440826+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:37.391","scrub_duration":0.30343327599999997,"objects_trimmed":0,"snaptrim_duration":0.166523535,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1424,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.193006+0000","last_change":"2023-01-15T14:34:52.192686+0000","last_active":"2023-01-15T14:34:53.193006+0000","last_peered":"2023-01-15T14:34:53.193006+0000","last_clean":"2023-01-15T14:34:53.193006+0000","last_became_active":"2023-01-15T14:34:46.648894+0000","last_became_peered":"2023-01-15T14:34:46.648894+0000","last_unstale":"2023-01-15T14:34:53.193006+0000","last_undegraded":"2023-01-15T14:34:53.193006+0000","last_fullsized":"2023-01-15T14:34:53.193006+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":5,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:51.945865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.945865+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:08.396","scrub_duration":0.30386408399999998,"objects_trimmed":0,"snaptrim_duration":0.12911487199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.1e","version":"215'39","reported_seq":1105,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.414346+0000","last_change":"2023-01-15T14:34:53.414346+0000","last_active":"2023-01-15T14:34:53.414346+0000","last_peered":"2023-01-15T14:34:53.414346+0000","last_clean":"2023-01-15T14:34:53.414346+0000","last_became_active":"2023-01-15T14:34:43.163816+0000","last_became_peered":"2023-01-15T14:34:43.163816+0000","last_unstale":"2023-01-15T14:34:53.414346+0000","last_undegraded":"2023-01-15T14:34:53.414346+0000","last_fullsized":"2023-01-15T14:34:53.414346+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.42472569900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.1f","version":"389'267","reported_seq":1441,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.764159+0000","last_change":"2023-01-15T14:34:51.764598+0000","last_active":"2023-01-15T14:34:51.764159+0000","last_peered":"2023-01-15T14:34:51.764159+0000","last_clean":"2023-01-15T14:34:51.764159+0000","last_became_active":"2023-01-15T14:34:00.240936+0000","last_became_peered":"2023-01-15T14:34:00.240936+0000","last_unstale":"2023-01-15T14:34:51.764159+0000","last_undegraded":"2023-01-15T14:34:51.764159+0000","last_fullsized":"2023-01-15T14:34:51.764159+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":267,"log_dups_size":0,"ondisk_log_size":267,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.44627647199999998,"stat_sum":{"num_bytes":6424165,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":29109,"num_write":9,"num_write_kb":3664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":5949777,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.20","version":"324'273","reported_seq":1468,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.320597+0000","last_change":"2023-01-15T14:34:51.321442+0000","last_active":"2023-01-15T14:34:51.320597+0000","last_peered":"2023-01-15T14:34:51.320597+0000","last_clean":"2023-01-15T14:34:51.320597+0000","last_became_active":"2023-01-15T14:33:55.472883+0000","last_became_peered":"2023-01-15T14:33:55.472883+0000","last_unstale":"2023-01-15T14:34:51.320597+0000","last_undegraded":"2023-01-15T14:34:51.320597+0000","last_fullsized":"2023-01-15T14:34:51.320597+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"324'273","last_scrub_stamp":"2023-01-15T14:34:49.100502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.100502+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:02.308","scrub_duration":0.204192292,"objects_trimmed":0,"snaptrim_duration":0.20435626900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.21","version":"321'138","reported_seq":1153,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.143248+0000","last_change":"2023-01-15T14:34:51.244423+0000","last_active":"2023-01-15T14:34:52.143248+0000","last_peered":"2023-01-15T14:34:52.143248+0000","last_clean":"2023-01-15T14:34:52.143248+0000","last_became_active":"2023-01-15T14:33:45.267321+0000","last_became_peered":"2023-01-15T14:33:45.267321+0000","last_unstale":"2023-01-15T14:34:52.143248+0000","last_undegraded":"2023-01-15T14:34:52.143248+0000","last_fullsized":"2023-01-15T14:34:52.143248+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"321'138","last_scrub_stamp":"2023-01-15T14:34:49.495975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.495975+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:17.375","scrub_duration":0.40443773500000002,"objects_trimmed":0,"snaptrim_duration":0.193094087,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"3.22","version":"323'161","reported_seq":1479,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.353614+0000","last_change":"2023-01-15T14:34:53.454441+0000","last_active":"2023-01-15T14:34:53.353614+0000","last_peered":"2023-01-15T14:34:53.353614+0000","last_clean":"2023-01-15T14:34:53.353614+0000","last_became_active":"2023-01-15T14:34:46.463340+0000","last_became_peered":"2023-01-15T14:34:46.463340+0000","last_unstale":"2023-01-15T14:34:53.353614+0000","last_undegraded":"2023-01-15T14:34:53.353614+0000","last_fullsized":"2023-01-15T14:34:53.353614+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.46517983400000001,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.23","version":"149'233","reported_seq":839,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.318581+0000","last_change":"2023-01-15T14:34:51.319656+0000","last_active":"2023-01-15T14:34:51.318581+0000","last_peered":"2023-01-15T14:34:51.318581+0000","last_clean":"2023-01-15T14:34:51.318581+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:34:51.318581+0000","last_undegraded":"2023-01-15T14:34:51.318581+0000","last_fullsized":"2023-01-15T14:34:51.318581+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:32:52.316916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:52.316916+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20278886800000001,"stat_sum":{"num_bytes":3802136,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":23,"num_read_kb":21818,"num_write":18,"num_write_kb":6580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1234747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.24","version":"402'281","reported_seq":1528,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.251106+0000","last_change":"2023-01-15T14:34:53.252158+0000","last_active":"2023-01-15T14:34:53.251106+0000","last_peered":"2023-01-15T14:34:53.251106+0000","last_clean":"2023-01-15T14:34:53.251106+0000","last_became_active":"2023-01-15T14:33:45.075309+0000","last_became_peered":"2023-01-15T14:33:45.075309+0000","last_unstale":"2023-01-15T14:34:53.251106+0000","last_undegraded":"2023-01-15T14:34:53.251106+0000","last_fullsized":"2023-01-15T14:34:53.251106+0000","mapping_epoch":198,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.26275459299999998,"stat_sum":{"num_bytes":7636039,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":20,"num_read_kb":22662,"num_write":11,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.25","version":"333'108","reported_seq":1101,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.356970+0000","last_change":"2023-01-15T14:34:52.357674+0000","last_active":"2023-01-15T14:34:52.356970+0000","last_peered":"2023-01-15T14:34:52.356970+0000","last_clean":"2023-01-15T14:34:52.356970+0000","last_became_active":"2023-01-15T14:34:46.346453+0000","last_became_peered":"2023-01-15T14:34:46.346453+0000","last_unstale":"2023-01-15T14:34:52.356970+0000","last_undegraded":"2023-01-15T14:34:52.356970+0000","last_fullsized":"2023-01-15T14:34:52.356970+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"198'79","last_scrub_stamp":"2023-01-15T14:31:30.478561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.478561+0000","objects_scrubbed":4,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51208220299999996,"objects_trimmed":0,"snaptrim_duration":0.30144994800000002,"stat_sum":{"num_bytes":2638765,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3030,"num_write":2,"num_write_kb":2577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.26","version":"102'52","reported_seq":571,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.146219+0000","last_change":"2023-01-15T14:34:52.146169+0000","last_active":"2023-01-15T14:34:52.146219+0000","last_peered":"2023-01-15T14:34:52.146219+0000","last_clean":"2023-01-15T14:34:52.146219+0000","last_became_active":"2023-01-15T14:33:52.309110+0000","last_became_peered":"2023-01-15T14:33:52.309110+0000","last_unstale":"2023-01-15T14:34:52.146219+0000","last_undegraded":"2023-01-15T14:34:52.146219+0000","last_fullsized":"2023-01-15T14:34:52.146219+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:48.121","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.77708315800000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.27","version":"191'79","reported_seq":1173,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.610104+0000","last_change":"2023-01-15T14:34:52.610104+0000","last_active":"2023-01-15T14:34:52.610104+0000","last_peered":"2023-01-15T14:34:52.610104+0000","last_clean":"2023-01-15T14:34:52.610104+0000","last_became_active":"2023-01-15T14:33:45.171090+0000","last_became_peered":"2023-01-15T14:33:45.171090+0000","last_unstale":"2023-01-15T14:34:52.610104+0000","last_undegraded":"2023-01-15T14:34:52.610104+0000","last_fullsized":"2023-01-15T14:34:52.610104+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":336,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30352980400000001,"objects_trimmed":0,"snaptrim_duration":0.566099562,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.28","version":"333'248","reported_seq":1539,"reported_epoch":402,"state":"active+clean","last_fresh":"2023-01-15T14:34:52.610951+0000","last_change":"2023-01-15T14:34:52.610951+0000","last_active":"2023-01-15T14:34:52.610951+0000","last_peered":"2023-01-15T14:34:52.610951+0000","last_clean":"2023-01-15T14:34:52.610951+0000","last_became_active":"2023-01-15T14:34:46.465249+0000","last_became_peered":"2023-01-15T14:34:46.465249+0000","last_unstale":"2023-01-15T14:34:52.610951+0000","last_undegraded":"2023-01-15T14:34:52.610951+0000","last_fullsized":"2023-01-15T14:34:52.610951+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'248","last_scrub_stamp":"2023-01-15T14:34:20.620068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.620068+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:20.827","scrub_duration":0.40448800800000001,"objects_trimmed":0,"snaptrim_duration":0.53069864300000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.29","version":"110'149","reported_seq":1033,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.454599+0000","last_change":"2023-01-15T14:34:53.455460+0000","last_active":"2023-01-15T14:34:53.454599+0000","last_peered":"2023-01-15T14:34:53.454599+0000","last_clean":"2023-01-15T14:34:53.454599+0000","last_became_active":"2023-01-15T14:34:46.177223+0000","last_became_peered":"2023-01-15T14:34:46.177223+0000","last_unstale":"2023-01-15T14:34:53.454599+0000","last_undegraded":"2023-01-15T14:34:53.454599+0000","last_fullsized":"2023-01-15T14:34:53.454599+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.46624356500000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1337,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.043199+0000","last_change":"2023-01-15T14:34:52.043158+0000","last_active":"2023-01-15T14:34:52.043199+0000","last_peered":"2023-01-15T14:34:52.043199+0000","last_clean":"2023-01-15T14:34:52.043199+0000","last_became_active":"2023-01-15T14:31:25.135856+0000","last_became_peered":"2023-01-15T14:31:25.135856+0000","last_unstale":"2023-01-15T14:34:52.043199+0000","last_undegraded":"2023-01-15T14:34:52.043199+0000","last_fullsized":"2023-01-15T14:34:52.043199+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.102535697,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.2b","version":"155'53","reported_seq":166,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.426571+0000","last_change":"2023-01-15T14:34:51.458328+0000","last_active":"2023-01-15T14:34:51.426571+0000","last_peered":"2023-01-15T14:34:51.426571+0000","last_clean":"2023-01-15T14:34:51.426571+0000","last_became_active":"2023-01-15T14:34:43.357195+0000","last_became_peered":"2023-01-15T14:34:43.357195+0000","last_unstale":"2023-01-15T14:34:51.426571+0000","last_undegraded":"2023-01-15T14:34:51.426571+0000","last_fullsized":"2023-01-15T14:34:51.426571+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.24152158000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.2c","version":"141'46","reported_seq":544,"reported_epoch":403,"state":"active+clean","last_fresh":"2023-01-15T14:34:53.192971+0000","last_change":"2023-01-15T14:34:52.266236+0000","last_active":"2023-01-15T14:34:53.192971+0000","last_peered":"2023-01-15T14:34:53.192971+0000","last_clean":"2023-01-15T14:34:53.192971+0000","last_became_active":"2023-01-15T14:31:21.900040+0000","last_became_peered":"2023-01-15T14:31:21.900040+0000","last_unstale":"2023-01-15T14:34:53.192971+0000","last_undegraded":"2023-01-15T14:34:53.192971+0000","last_fullsized":"2023-01-15T14:34:53.192971+0000","mapping_epoch":194,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":195,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:32:50.271402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:50.271402+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.107635167,"objects_trimmed":0,"snaptrim_duration":0.16621101999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10a","length":"1"},{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1310,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.090380+0000","last_change":"2023-01-15T14:34:52.090361+0000","last_active":"2023-01-15T14:34:52.090380+0000","last_peered":"2023-01-15T14:34:52.090380+0000","last_clean":"2023-01-15T14:34:52.090380+0000","last_became_active":"2023-01-15T14:31:25.035203+0000","last_became_peered":"2023-01-15T14:31:25.035203+0000","last_unstale":"2023-01-15T14:34:52.090380+0000","last_undegraded":"2023-01-15T14:34:52.090380+0000","last_fullsized":"2023-01-15T14:34:52.090380+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.68358103000000003,"stat_sum":{"num_bytes":3033664,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":5419,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.2e","version":"139'38","reported_seq":150,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.044533+0000","last_change":"2023-01-15T14:34:52.044490+0000","last_active":"2023-01-15T14:34:52.044533+0000","last_peered":"2023-01-15T14:34:52.044533+0000","last_clean":"2023-01-15T14:34:52.044533+0000","last_became_active":"2023-01-15T14:34:43.265888+0000","last_became_peered":"2023-01-15T14:34:43.265888+0000","last_unstale":"2023-01-15T14:34:52.044533+0000","last_undegraded":"2023-01-15T14:34:52.044533+0000","last_fullsized":"2023-01-15T14:34:52.044533+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.25000760999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1200,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.045105+0000","last_change":"2023-01-15T14:34:51.146551+0000","last_active":"2023-01-15T14:34:52.045105+0000","last_peered":"2023-01-15T14:34:52.045105+0000","last_clean":"2023-01-15T14:34:52.045105+0000","last_became_active":"2023-01-15T14:33:55.174128+0000","last_became_peered":"2023-01-15T14:33:55.174128+0000","last_unstale":"2023-01-15T14:34:52.045105+0000","last_undegraded":"2023-01-15T14:34:52.045105+0000","last_fullsized":"2023-01-15T14:34:52.045105+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.79687536999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"121","length":"1"},{"start":"127","length":"1"}]},{"pgid":"3.30","version":"206'252","reported_seq":1087,"reported_epoch":401,"state":"active+clean","last_fresh":"2023-01-15T14:34:51.116847+0000","last_change":"2023-01-15T14:34:51.320511+0000","last_active":"2023-01-15T14:34:51.116847+0000","last_peered":"2023-01-15T14:34:51.116847+0000","last_clean":"2023-01-15T14:34:51.116847+0000","last_became_active":"2023-01-15T14:33:55.373498+0000","last_became_peered":"2023-01-15T14:33:55.373498+0000","last_unstale":"2023-01-15T14:34:51.116847+0000","last_undegraded":"2023-01-15T14:34:51.116847+0000","last_fullsized":"2023-01-15T14:34:51.116847+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:34:36.078678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:36.078678+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:50.459","scrub_duration":1.158842567,"objects_trimmed":0,"snaptrim_duration":0.20388425800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"f8","length":"1"},{"start":"114","length":"1"},{"start":"120","length":"2"},{"start":"126","length":"2"}]},{"pgid":"3.31","version":"204'150","reported_seq":855,"reported_epoch":402,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:34:52.244473+0000","last_change":"2023-01-15T14:34:52.244473+0000","last_active":"2023-01-15T14:34:52.244473+0000","last_peered":"2023-01-15T14:34:52.244473+0000","last_clean":"2023-01-15T14:34:52.244473+0000","last_became_active":"2023-01-15T14:34:46.275656+0000","last_became_peered":"2023-01-15T14:34:46.275656+0000","last_unstale":"2023-01-15T14:34:52.244473+0000","last_undegraded":"2023-01-15T14:34:52.244473+0000","last_fullsized":"2023-01-15T14:34:52.244473+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.0028007359999999999,"stat_sum":{"num_bytes":7117814,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":21449,"num_write":7,"num_write_kb":6179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":7210220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"121","length":"1"},{"start":"126","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":116,"stat_sum":{"num_bytes":419724261,"num_objects":197,"num_object_clones":146,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":16,"num_read":943,"num_read_kb":1055378,"num_write":700,"num_write_kb":298029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":653,"num_bytes_recovered":437076290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19330,"ondisk_log_size":19330,"up":318,"acting":318,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574931,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1152956,"kb_used_data":92546476,"kb_used_omap":10099,"kb_used_meta":0,"kb_avail":92546424,"statfs":{"total":95948165120,"available":94767538176,"internally_reserved":53248,"allocated":94767591424,"data_stored":94767591424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10341830,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":3.452,"15min":3.8639999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1830000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":6.0389999999999997,"5min":5.6859999999999999,"15min":8.0340000000000007},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":36.622,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.60099999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":12.064,"5min":4.7549999999999999,"15min":5.9480000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":204.74199999999999,"5min":204.74199999999999,"15min":204.74199999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":11.260999999999999,"5min":9.1910000000000007,"15min":7.25},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":94.537000000000006,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.74099999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":6.4119999999999999,"15min":5.3310000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":25.065999999999999,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.627},{"interface":"front","average":{"1min":15.106,"5min":4.7649999999999997,"15min":4.3730000000000002},"min":{"1min":0.59599999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":162.96799999999999,"5min":162.96799999999999,"15min":162.96799999999999},"last":0.76800000000000002}]},{"osd":3,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.198,"5min":7.6299999999999999,"15min":3.1749999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":25.359999999999999,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.86499999999999999,"5min":2.879,"15min":4.0609999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.266,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.85799999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.4430000000000001,"5min":4.2279999999999998,"15min":7.6859999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":29.690000000000001,"5min":96.185000000000002,"15min":96.185000000000002},"last":1.054},{"interface":"front","average":{"1min":10.949999999999999,"5min":7.7789999999999999,"15min":3.2360000000000002},"min":{"1min":0.46200000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":142.44999999999999,"5min":164.21199999999999,"15min":164.21199999999999},"last":1.0940000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.5489999999999999,"5min":4.3099999999999996,"15min":5.6539999999999999},"min":{"1min":0.63800000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":29.797999999999998,"5min":96.263999999999996,"15min":96.263999999999996},"last":0.80200000000000005},{"interface":"front","average":{"1min":2.3519999999999999,"5min":2.6869999999999998,"15min":4.1920000000000002},"min":{"1min":0.55300000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":25.199000000000002,"5min":70.387,"15min":95.430000000000007},"last":0.88}]},{"osd":6,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.645,"5min":4.4189999999999996,"15min":7.1550000000000002},"min":{"1min":0.60999999999999999,"5min":0.496,"15min":0.496},"max":{"1min":29.876000000000001,"5min":96.299999999999997,"15min":96.299999999999997},"last":0.84099999999999997},{"interface":"front","average":{"1min":2.399,"5min":2.7749999999999999,"15min":1.6579999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":25.236999999999998,"5min":70.530000000000001,"15min":70.530000000000001},"last":0.92900000000000005}]}]},{"osd":6,"up_from":0,"seq":55834574932,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":5,"up_from":0,"seq":55834574932,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574931,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1022780,"kb_used_data":92676624,"kb_used_omap":15984,"kb_used_meta":0,"kb_avail":92676600,"statfs":{"total":95948165120,"available":94900838400,"internally_reserved":24576,"allocated":94900862976,"data_stored":94900862976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16367765,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":22.818000000000001,"15min":10.882},"min":{"1min":0.505,"5min":0.44,"15min":0.44},"max":{"1min":5.0049999999999999,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":2.9990000000000001,"15min":1.52},"min":{"1min":0.51200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.127,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.53300000000000003}]},{"osd":1,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.9390000000000001,"5min":3.742,"15min":6.976},"min":{"1min":0.499,"5min":0.49299999999999999,"15min":0.47399999999999998},"max":{"1min":23.193000000000001,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":10.99,"5min":6.3819999999999997,"15min":2.6560000000000001},"min":{"1min":0.67100000000000004,"5min":0.52900000000000003,"15min":0.50800000000000001},"max":{"1min":162.976,"5min":162.976,"15min":162.976},"last":0.77600000000000002}]},{"osd":2,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.0950000000000002,"5min":23.544,"15min":11.522},"min":{"1min":0.497,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":23.247,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.26,"5min":1.28,"15min":1.0329999999999999},"min":{"1min":0.61899999999999999,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.5789999999999997,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.88700000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":2.802,"15min":1.8260000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.006,"5min":88.823999999999998,"15min":88.823999999999998},"last":45.584000000000003},{"interface":"front","average":{"1min":5.3380000000000001,"5min":24.337,"15min":12.212999999999999},"min":{"1min":0.68600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":58.317,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":3.5339999999999998}]},{"osd":5,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88100000000000001,"15min":0.90700000000000003},"min":{"1min":0.61499999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.175,"5min":2.577,"15min":2.577},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.903,"15min":2.1669999999999998},"min":{"1min":0.69099999999999995,"5min":0.65200000000000002,"15min":0.55300000000000005},"max":{"1min":1.0700000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":0.98899999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":3.2389999999999999,"15min":4.8609999999999998},"min":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":23.329000000000001,"5min":85.721999999999994,"15min":99.780000000000001},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.999,"5min":21.631,"15min":10.651999999999999},"min":{"1min":0.79500000000000004,"5min":0.66700000000000004,"15min":0.58699999999999997},"max":{"1min":1.173,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.0700000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":3.238,"5min":3.5790000000000002,"15min":9.9450000000000003},"min":{"1min":0.73999999999999999,"5min":0.63400000000000001,"15min":0.59499999999999997},"max":{"1min":42.551000000000002,"5min":89.061000000000007,"15min":179.73500000000001},"last":1.099},{"interface":"front","average":{"1min":2.274,"5min":23.071000000000002,"15min":15.183},"min":{"1min":0.78700000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":23.356000000000002,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":0.93100000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574931,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1100700,"kb_used_data":92598776,"kb_used_omap":16338,"kb_used_meta":0,"kb_avail":92598680,"statfs":{"total":95948165120,"available":94821048320,"internally_reserved":98304,"allocated":94821146624,"data_stored":94821146624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16730461,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":7.8860000000000001,"5min":8.048,"15min":5.9139999999999997},"min":{"1min":0.39500000000000002,"5min":0.255,"15min":0.255},"max":{"1min":74.245999999999995,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.32400000000000001},{"interface":"front","average":{"1min":7.9649999999999999,"5min":3.4990000000000001,"15min":6.2610000000000001},"min":{"1min":0.46600000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":74.280000000000001,"5min":74.280000000000001,"15min":93.674000000000007},"last":0.26100000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":10.762,"5min":6.625,"15min":11.228999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":117.664,"5min":117.664,"15min":159.256},"last":0.54400000000000004},{"interface":"front","average":{"1min":4.6909999999999998,"5min":11.208,"15min":4.2300000000000004},"min":{"1min":0.41899999999999998,"5min":0.317,"15min":0.317},"max":{"1min":47.963000000000001,"5min":174.578,"15min":174.578},"last":0.621}]},{"osd":2,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":2.4820000000000002,"15min":3.0600000000000001},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":13.113,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.436},{"interface":"front","average":{"1min":2.3079999999999998,"5min":6.8120000000000003,"15min":3.7320000000000002},"min":{"1min":0.44,"5min":0.375,"15min":0.375},"max":{"1min":15.946999999999999,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.64800000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":28.689,"15min":13.936999999999999},"min":{"1min":0.56299999999999994,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":14.234999999999999,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":1.016},{"interface":"front","average":{"1min":11.906000000000001,"5min":31.741,"15min":19.829000000000001},"min":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":117.59,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":1.073}]},{"osd":5,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":11.022,"5min":5.9219999999999997,"15min":11.242000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":117.744,"5min":117.744,"15min":159.43000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":5.9829999999999997,"15min":2.6749999999999998},"min":{"1min":0.66200000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2210000000000001,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.90400000000000003}]},{"osd":6,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":5.984,"15min":2.6819999999999999},"min":{"1min":0.68300000000000005,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.286,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.3979999999999999,"15min":2.9740000000000002},"min":{"1min":0.71099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6799999999999999,"5min":43.661000000000001,"15min":60.098999999999997},"last":0.58799999999999997}]},{"osd":7,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.4119999999999999,"15min":3.161},"min":{"1min":0.753,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.758,"5min":43.712000000000003,"15min":60.165999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":19.324999999999999,"5min":11.952,"15min":16.044},"min":{"1min":0.65400000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":142.02000000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":0.77600000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574931,"num_pgs":46,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1062656,"kb_used_data":92639400,"kb_used_omap":13641,"kb_used_meta":0,"kb_avail":92636724,"statfs":{"total":95948165120,"available":94860005376,"internally_reserved":2740224,"allocated":94862745600,"data_stored":94862745600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968572,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":4.2469999999999999,"5min":4.2450000000000001,"15min":1.8420000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":78.456000000000003,"5min":78.456000000000003,"15min":78.456000000000003},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.58499999999999996,"5min":1.4039999999999999,"15min":0.94399999999999995},"min":{"1min":0.36799999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.59599999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":10.247,"5min":7.7949999999999999,"15min":3.1309999999999998},"min":{"1min":0.35099999999999998,"5min":0.27700000000000002,"15min":0.27000000000000002},"max":{"1min":128.749,"5min":143.071,"15min":143.071},"last":0.57299999999999995},{"interface":"front","average":{"1min":3.105,"5min":1.966,"15min":2.8919999999999999},"min":{"1min":0.35999999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":52.927999999999997,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.88}]},{"osd":3,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":3.8879999999999999,"5min":6.9180000000000001,"15min":9.2400000000000002},"min":{"1min":0.33700000000000002,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":34.244,"5min":143.13300000000001,"15min":183.53},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":7.5039999999999996,"15min":3.012},"min":{"1min":0.41399999999999998,"5min":0.316,"15min":0.316},"max":{"1min":1.3959999999999999,"5min":195.16399999999999,"15min":195.16399999999999},"last":0.63500000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":5.9480000000000004,"15min":4.4160000000000004},"min":{"1min":0.67200000000000004,"5min":0.624,"15min":0.58399999999999996},"max":{"1min":1.2450000000000001,"5min":175.34700000000001,"15min":175.34700000000001},"last":1.002},{"interface":"front","average":{"1min":0.88300000000000001,"5min":4.7080000000000002,"15min":4.1079999999999997},"min":{"1min":0.57799999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.052,"5min":175.30500000000001,"15min":175.30500000000001},"last":1.02}]},{"osd":5,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":2.5590000000000002,"5min":2.7490000000000001,"15min":1.7},"min":{"1min":0.63500000000000001,"5min":0.57199999999999995,"15min":0.54800000000000004},"max":{"1min":34.256999999999998,"5min":143.262,"15min":143.262},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.95099999999999996,"5min":3.012,"15min":1.778},"min":{"1min":0.70199999999999996,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.3160000000000001,"5min":66.456999999999994,"15min":66.456999999999994},"last":0.93799999999999994}]},{"osd":6,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":1.839,"15min":1.3009999999999999},"min":{"1min":0.61199999999999999,"5min":0.59599999999999997,"15min":0.55000000000000004},"max":{"1min":1.276,"5min":88.542000000000002,"15min":88.542000000000002},"last":1.079},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8980000000000001,"15min":1.778},"min":{"1min":0.69199999999999995,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":34.305999999999997,"5min":143.215,"15min":143.215},"last":0.98199999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:34:11 2023","interfaces":[{"interface":"back","average":{"1min":11.396000000000001,"5min":7.9800000000000004,"15min":4.6989999999999998},"min":{"1min":0.60599999999999998,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":176.44499999999999,"5min":176.44499999999999,"15min":176.44499999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":2.6299999999999999,"5min":2.8159999999999998,"15min":2.79},"min":{"1min":0.622,"5min":0.622,"15min":0.54500000000000004},"max":{"1min":34.207999999999998,"5min":143.17099999999999,"15min":143.17099999999999},"last":1.046}]}]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":52,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1014328,"kb_used_data":92685176,"kb_used_omap":15222,"kb_used_meta":0,"kb_avail":92685052,"statfs":{"total":95948165120,"available":94909493248,"internally_reserved":126976,"allocated":94909620224,"data_stored":94909620224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15588232,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":2.0619999999999998,"5min":3.2669999999999999,"15min":3.6840000000000002},"min":{"1min":0.37,"5min":0.314,"15min":0.314},"max":{"1min":23.68,"5min":144.12799999999999,"15min":144.12799999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":14.157,"5min":6.5789999999999997,"15min":5.0750000000000002},"min":{"1min":0.42299999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":176.01599999999999,"5min":176.01599999999999,"15min":176.01599999999999},"last":0.78800000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":4.8959999999999999,"5min":3.4329999999999998,"15min":1.7889999999999999},"min":{"1min":0.44400000000000001,"5min":0.245,"15min":0.245},"max":{"1min":69.277000000000001,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.629,"5min":1.173,"15min":3.1059999999999999},"min":{"1min":0.28399999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.873,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.753}]},{"osd":3,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":5.8440000000000003,"15min":15.172000000000001},"min":{"1min":0.38400000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.92500000000000004,"5min":245.637,"15min":282.01400000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":14.228999999999999,"5min":8.2870000000000008,"15min":7.5919999999999996},"min":{"1min":0.46500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":176.13900000000001,"5min":176.13900000000001,"15min":176.13900000000001},"last":0.94799999999999995}]},{"osd":4,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":19.170000000000002,"5min":20.029,"15min":9.8279999999999994},"min":{"1min":0.56299999999999994,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":176.18600000000001,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":1.02},{"interface":"front","average":{"1min":5.5510000000000002,"5min":15.916,"15min":7.7080000000000002},"min":{"1min":0.64300000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":69.611999999999995,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.73399999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":5.2060000000000004,"5min":2.3799999999999999,"15min":1.482},"min":{"1min":0.56899999999999995,"5min":0.433,"15min":0.433},"max":{"1min":69.478999999999999,"5min":69.478999999999999,"15min":69.478999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":3.2770000000000001},"min":{"1min":0.66000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3089999999999999,"5min":4.0780000000000003,"15min":46.905999999999999},"last":0.995}]},{"osd":6,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":14.491,"5min":6.9109999999999996,"15min":5.6379999999999999},"min":{"1min":0.73199999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":176.249,"5min":176.249,"15min":176.249},"last":1.0629999999999999},{"interface":"front","average":{"1min":5.2629999999999999,"5min":2.431,"15min":1.5589999999999999},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":69.549999999999997,"5min":69.549999999999997,"15min":69.549999999999997},"last":1.04}]},{"osd":7,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":7.617,"5min":6.2309999999999999,"15min":6.7549999999999999},"min":{"1min":0.63900000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":95.888999999999996,"5min":157.03700000000001,"15min":157.03700000000001},"last":1.0720000000000001},{"interface":"front","average":{"1min":14.544,"5min":6.9649999999999999,"15min":5.4880000000000004},"min":{"1min":0.78700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":176.22,"5min":176.22,"15min":176.22},"last":1.095}]}]},{"osd":0,"up_from":13,"seq":55834574932,"num_pgs":68,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1147116,"kb_used_data":92556468,"kb_used_omap":10208,"kb_used_meta":0,"kb_avail":92552264,"statfs":{"total":95948165120,"available":94773518336,"internally_reserved":4304896,"allocated":94777823232,"data_stored":94777823232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10453035,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":3.3460000000000001,"15min":1.53},"min":{"1min":0.34899999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":25.960999999999999,"5min":69.872,"15min":69.872},"last":0.59699999999999998},{"interface":"front","average":{"1min":11.249000000000001,"5min":5.335,"15min":2.2650000000000001},"min":{"1min":0.434,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":162.91200000000001,"5min":162.91200000000001,"15min":162.91200000000001},"last":0.36299999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":3.415,"15min":1.601},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0109999999999999,"5min":98.286000000000001,"15min":98.286000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":2.6320000000000001,"15min":1.3500000000000001},"min":{"1min":0.37,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86899999999999999,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.48399999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":7.8079999999999998,"15min":13.788},"min":{"1min":0.48199999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":26.027000000000001,"5min":199.08000000000001,"15min":284.62799999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":3.6070000000000002,"5min":7.867,"15min":3.1560000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33600000000000002},"max":{"1min":61.119,"5min":174.09399999999999,"15min":174.09399999999999},"last":0.433}]},{"osd":4,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":16.664999999999999,"15min":6.2119999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1719999999999999,"5min":1486.309,"15min":1486.309},"last":0.79900000000000004},{"interface":"front","average":{"1min":2.1030000000000002,"5min":18.140999999999998,"15min":6.7300000000000004},"min":{"1min":0.63100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":26.137,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":0.98999999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":2.8980000000000001,"15min":7.4980000000000002},"min":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1930000000000001,"5min":98.411000000000001,"15min":176.34399999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.7070000000000001,"15min":1.2609999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.101,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.81200000000000006}]},{"osd":6,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":2.1600000000000001,"5min":3.7890000000000001,"15min":2.0430000000000001},"min":{"1min":0.71899999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":26.082000000000001,"5min":70.259,"15min":70.259},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.026,"5min":3,"15min":1.756},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.57499999999999996},"max":{"1min":1.5409999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":0.93300000000000005}]},{"osd":7,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":3.2770000000000001,"15min":11.539},"min":{"1min":0.70099999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":11.906000000000001,"5min":73.864999999999995,"15min":292.61099999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":2.1819999999999999,"5min":3.8090000000000002,"15min":2.0499999999999998},"min":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.58699999999999997},"max":{"1min":26.187000000000001,"5min":70.322000000000003,"15min":70.322000000000003},"last":0.78200000000000003}]}]}],"pool_statfs":[]}} 2023-01-15T14:34:56.305 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:34:56.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 126 2023-01-15T14:34:56.498 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:55.711+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:56.499 INFO:tasks.rados.rados.0.smithi090.stdout:1523: done (0 left) 2023-01-15T14:34:56.499 INFO:tasks.rados.rados.0.smithi090.stdout:1524: snap_remove snap 135 2023-01-15T14:34:56.500 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:34:56.500 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:34:56.545 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:56.539+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:56.603 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:56.595+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:56.705 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:56.705 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:56.705 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:56.699+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:56.706 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:34:56.706 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:34:56.706 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:34:56.706 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:34:56.706 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:34:56.717 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:56.736 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:56.759 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:56.778 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:56.798 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:56.805 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:56.799+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:56.818 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:56.820 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:34:56.820 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:34:56.820 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:34:56.820 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:34:56.821 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:34:56.821 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:34:56.821 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:34:56.821 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:34:56.822 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:56.822 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:56.832 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:34:56.906 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:56.901+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:56.939 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:56.939 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:56.949 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:34:56.950 INFO:tasks.rados.rados.0.smithi090.stdout:1524: done (0 left) 2023-01-15T14:34:56.950 INFO:tasks.rados.rados.0.smithi090.stdout:1525: read oid 20 snap -1 2023-01-15T14:34:56.950 INFO:tasks.rados.rados.0.smithi090.stdout:1525: expect (ObjNum 528 snap 158 seq_num 528) 2023-01-15T14:34:56.950 INFO:tasks.rados.rados.0.smithi090.stdout:1526: write (excl) oid 37 current snap is 170 2023-01-15T14:34:56.950 INFO:tasks.rados.rados.0.smithi090.stdout:1526: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2023-01-15T14:34:56.957 INFO:tasks.rados.rados.0.smithi090.stdout:1526: writing smithi09025264-37 from 675780 to 1171303 tid 1 2023-01-15T14:34:56.960 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 126 2023-01-15T14:34:56.965 INFO:tasks.rados.rados.0.smithi090.stdout:1526: writing smithi09025264-37 from 1682747 to 2397258 tid 2 2023-01-15T14:34:56.969 INFO:tasks.rados.rados.0.smithi090.stdout:1526: writing smithi09025264-37 from 3012769 to 3075780 tid 3 2023-01-15T14:34:56.976 INFO:tasks.rados.rados.0.smithi090.stdout:1525: expect (ObjNum 528 snap 158 seq_num 528) 2023-01-15T14:34:57.008 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:57.003+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:57.115 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:57.116 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:57.117 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:34:57.117 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:57.111+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:57.152 INFO:tasks.rados.rados.0.smithi090.stdout:1526: finishing write tid 1 to smithi09025264-37 2023-01-15T14:34:57.153 INFO:tasks.rados.rados.0.smithi090.stdout:1525: done (1 left) 2023-01-15T14:34:57.153 INFO:tasks.rados.rados.0.smithi090.stdout:1527: rollback oid 28 current snap is 170 2023-01-15T14:34:57.153 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 28 to 146 2023-01-15T14:34:57.154 INFO:tasks.rados.rados.0.smithi090.stdout:1528: delete oid 9 current snap is 170 2023-01-15T14:34:57.178 INFO:tasks.rados.rados.0.smithi090.stdout:1526: finishing write tid 2 to smithi09025264-37 2023-01-15T14:34:57.179 INFO:tasks.rados.rados.0.smithi090.stdout:1526: finishing write tid 3 to smithi09025264-37 2023-01-15T14:34:57.216 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:57.209+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:57.324 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:57.326 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:57.327 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:34:57.327 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:57.321+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:57.367 INFO:tasks.rados.rados.0.smithi090.stdout:1526: finishing write tid 4 to smithi09025264-37 2023-01-15T14:34:57.367 INFO:tasks.rados.rados.0.smithi090.stdout:1527: finishing rollback tid 0 to smithi09025264-28 2023-01-15T14:34:57.367 INFO:tasks.rados.rados.0.smithi090.stdout:1526: finishing write tid 5 to smithi09025264-37 2023-01-15T14:34:57.367 INFO:tasks.rados.rados.0.smithi090.stdout:1526: oid 37 updating version 0 to 437 2023-01-15T14:34:57.367 INFO:tasks.rados.rados.0.smithi090.stdout:1526: oid 37 updating version 437 to 439 2023-01-15T14:34:57.367 INFO:tasks.rados.rados.0.smithi090.stdout:1526: oid 37 version 439 is already newer than 436 2023-01-15T14:34:57.367 INFO:tasks.rados.rados.0.smithi090.stdout:1526: oid 37 version 439 is already newer than 438 2023-01-15T14:34:57.368 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 439 (ObjNum 540 snap 170 seq_num 540) dirty exists 2023-01-15T14:34:57.368 INFO:tasks.rados.rados.0.smithi090.stdout:1526: left oid 37 (ObjNum 540 snap 170 seq_num 540) 2023-01-15T14:34:57.368 INFO:tasks.rados.rados.0.smithi090.stdout:1527: finishing rollback tid 1 to smithi09025264-28 2023-01-15T14:34:57.368 INFO:tasks.rados.rados.0.smithi090.stdout:1527: finishing rollback tid 2 to smithi09025264-28 2023-01-15T14:34:57.368 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 390 (ObjNum 466 snap 134 seq_num 466) dirty exists 2023-01-15T14:34:57.425 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:57.419+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:57.530 INFO:tasks.rados.rados.0.smithi090.stdout:1526: done (2 left) 2023-01-15T14:34:57.530 INFO:tasks.rados.rados.0.smithi090.stdout:1527: done (1 left) 2023-01-15T14:34:57.530 INFO:tasks.rados.rados.0.smithi090.stdout:1528: done (0 left) 2023-01-15T14:34:57.530 INFO:tasks.rados.rados.0.smithi090.stdout:1529: snap_remove snap 142 2023-01-15T14:34:57.530 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:57.531 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:57.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:34:57.532 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:57.523+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:57.631 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:57.625+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:57.728 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:57.728 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:57.728 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:34:57.732 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:57.727+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:57.832 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:57.827+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:57.843 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:57.843 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:57.853 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:34:57.934 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:57.927+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:57.968 INFO:tasks.rados.rados.0.smithi090.stdout:1529: done (0 left) 2023-01-15T14:34:57.969 INFO:tasks.rados.rados.0.smithi090.stdout:1530: snap_create 2023-01-15T14:34:58.012 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:58.012 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:58.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:34:58.067 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:34:58.063+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:34:58.132 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:58.141 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:58.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:34:58.143 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:34:58.137+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:34:58.243 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:58.237+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:58.375 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:58.375 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:58.376 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:34:58.377 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:58.367+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:58.476 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:58.471+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:58.556 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:34:58.558 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:58.558 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:58.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:34:58.576 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:58.573+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:34:58.677 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:58.673+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:58.751 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:58.751 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:58.751 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:34:58.779 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:58.773+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:58.831 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:34:58.959 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:58.959 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:58.960 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:34:58.960 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:34:58.951+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:34:58.980 INFO:tasks.rados.rados.0.smithi090.stdout:1530: done (0 left) 2023-01-15T14:34:58.980 INFO:tasks.rados.rados.0.smithi090.stdout:1531: write (excl) oid 6 current snap is 171 2023-01-15T14:34:58.980 INFO:tasks.rados.rados.0.smithi090.stdout:1531: seq_num 542 ranges {709394=646021,2083358=718180} 2023-01-15T14:34:58.983 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:34:58.986 INFO:tasks.rados.rados.0.smithi090.stdout:1531: writing smithi09025264-6 from 709394 to 1355415 tid 1 2023-01-15T14:34:58.995 INFO:tasks.rados.rados.0.smithi090.stdout:1531: writing smithi09025264-6 from 2083358 to 2801538 tid 2 2023-01-15T14:34:58.995 INFO:tasks.rados.rados.0.smithi090.stdout:1532: write (excl) oid 2 current snap is 171 2023-01-15T14:34:58.995 INFO:tasks.rados.rados.0.smithi090.stdout:1532: seq_num 543 ranges {726201=721270,2141840=578191} 2023-01-15T14:34:59.003 INFO:tasks.rados.rados.0.smithi090.stdout:1532: writing smithi09025264-2 from 726201 to 1447471 tid 1 2023-01-15T14:34:59.011 INFO:tasks.rados.rados.0.smithi090.stdout:1532: writing smithi09025264-2 from 2141840 to 2720031 tid 2 2023-01-15T14:34:59.011 INFO:tasks.rados.rados.0.smithi090.stdout:1533: delete oid 22 current snap is 171 2023-01-15T14:34:59.060 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:59.055+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:59.083 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:34:59.084 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:34:59.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:34:59.107 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:34:59.125 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:34:59.143 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:34:59.161 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:34:59.161 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:34:59.157+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:34:59.184 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:34:59.205 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:34:59.217 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:34:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDScrubReserve(3.55 GRANT e409)", 2023-01-15T14:34:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:59.171155+0000", 2023-01-15T14:34:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.043579093999999999, 2023-01-15T14:34:59.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.043608278, 2023-01-15T14:34:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:34:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171155+0000", 2023-01-15T14:34:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171155+0000", 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171154+0000", 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171158+0000", 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8990000000000003e-06 2023-01-15T14:34:59.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171173+0000", 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4929999999999999e-05 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171180+0000", 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3749999999999997e-06 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.4530.0:3964 3.5 e409/354)", 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:59.171915+0000", 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.042819389999999999, 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042910678000000001, 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171915+0000", 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171915+0000", 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171913+0000", 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:59.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171926+0000", 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3377000000000001e-05 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171955+0000", 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8949999999999999e-05 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.171971+0000", 2023-01-15T14:34:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.552e-05 2023-01-15T14:34:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop_reply(client.4530.0:3964 3.5 e409/354)", 2023-01-15T14:34:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:59.172613+0000", 2023-01-15T14:34:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.042120405999999999, 2023-01-15T14:34:59.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042264019999999999, 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.172613+0000", 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.172613+0000", 2023-01-15T14:34:59.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.172613+0000", 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.172620+0000", 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1640000000000004e-06 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.172636+0000", 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6138999999999999e-05 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.172646+0000", 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8779999999999993e-06 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.228 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:34:59.228 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 3 2023-01-15T14:34:59.228 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:59.228 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:34:59.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:34:59.247 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:34:59.249 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:34:59.249 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:34:59.249 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:34:59.249 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:34:59.250 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:34:59.250 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:34:59.250 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:34:59.250 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:34:59.262 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:34:59.257+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:34:59.274 INFO:tasks.rados.rados.0.smithi090.stdout:1532: finishing write tid 1 to smithi09025264-2 2023-01-15T14:34:59.277 INFO:tasks.rados.rados.0.smithi090.stdout:1532: finishing write tid 2 to smithi09025264-2 2023-01-15T14:34:59.279 INFO:tasks.rados.rados.0.smithi090.stdout:1533: done (2 left) 2023-01-15T14:34:59.279 INFO:tasks.rados.rados.0.smithi090.stdout:1534: read oid 20 snap -1 2023-01-15T14:34:59.279 INFO:tasks.rados.rados.0.smithi090.stdout:1534: expect (ObjNum 528 snap 158 seq_num 528) 2023-01-15T14:34:59.279 INFO:tasks.rados.rados.0.smithi090.stdout:1535: write oid 7 current snap is 171 2023-01-15T14:34:59.279 INFO:tasks.rados.rados.0.smithi090.stdout:1535: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2023-01-15T14:34:59.287 INFO:tasks.rados.rados.0.smithi090.stdout:1535: writing smithi09025264-7 from 759815 to 1231583 tid 1 2023-01-15T14:34:59.293 INFO:tasks.rados.rados.0.smithi090.stdout:1535: writing smithi09025264-7 from 1742451 to 2207958 tid 2 2023-01-15T14:34:59.300 INFO:tasks.rados.rados.0.smithi090.stdout:1535: writing smithi09025264-7 from 2917178 to 3159815 tid 3 2023-01-15T14:34:59.300 INFO:tasks.rados.rados.0.smithi090.stdout:1536: rollback oid 27 current snap is 171 2023-01-15T14:34:59.300 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 27 to 148 2023-01-15T14:34:59.300 INFO:tasks.rados.rados.0.smithi090.stdout:1537: read oid 30 snap -1 2023-01-15T14:34:59.300 INFO:tasks.rados.rados.0.smithi090.stdout:1537: expect (ObjNum 492 snap 144 seq_num 492) 2023-01-15T14:34:59.300 INFO:tasks.rados.rados.0.smithi090.stdout:1538: write oid 3 current snap is 171 2023-01-15T14:34:59.300 INFO:tasks.rados.rados.0.smithi090.stdout:1538: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2023-01-15T14:34:59.306 INFO:tasks.rados.rados.0.smithi090.stdout:1538: writing smithi09025264-3 from 776622 to 1323639 tid 1 2023-01-15T14:34:59.313 INFO:tasks.rados.rados.0.smithi090.stdout:1538: writing smithi09025264-3 from 1800933 to 2526451 tid 2 2023-01-15T14:34:59.318 INFO:tasks.rados.rados.0.smithi090.stdout:1538: writing smithi09025264-3 from 2944601 to 3176622 tid 3 2023-01-15T14:34:59.318 INFO:tasks.rados.rados.0.smithi090.stdout:1539: read oid 36 snap -1 2023-01-15T14:34:59.318 INFO:tasks.rados.rados.0.smithi090.stdout:1539: expect (ObjNum 507 snap 150 seq_num 507) 2023-01-15T14:34:59.319 INFO:tasks.rados.rados.0.smithi090.stdout:1540: read oid 15 snap -1 2023-01-15T14:34:59.319 INFO:tasks.rados.rados.0.smithi090.stdout:1540: expect (ObjNum 404 snap 108 seq_num 404) 2023-01-15T14:34:59.319 INFO:tasks.rados.rados.0.smithi090.stdout:1541: rollback oid 34 current snap is 171 2023-01-15T14:34:59.319 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 34 to 170 2023-01-15T14:34:59.319 INFO:tasks.rados.rados.0.smithi090.stdout:1542: snap_create 2023-01-15T14:34:59.319 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 10 2023-01-15T14:34:59.326 INFO:tasks.rados.rados.0.smithi090.stdout:1539: expect (ObjNum 507 snap 150 seq_num 507) 2023-01-15T14:34:59.356 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:59.378 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:34:59.379 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:34:59.379 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:34:59.379 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:34:59.381 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:34:59.381 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:34:59.375+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:34:59.414 INFO:tasks.rados.rados.0.smithi090.stdout:1532: finishing write tid 3 to smithi09025264-2 2023-01-15T14:34:59.414 INFO:tasks.rados.rados.0.smithi090.stdout:1532: finishing write tid 4 to smithi09025264-2 2023-01-15T14:34:59.414 INFO:tasks.rados.rados.0.smithi090.stdout:1532: oid 2 updating version 0 to 390 2023-01-15T14:34:59.414 INFO:tasks.rados.rados.0.smithi090.stdout:1532: oid 2 updating version 390 to 391 2023-01-15T14:34:59.415 INFO:tasks.rados.rados.0.smithi090.stdout:1532: oid 2 updating version 391 to 392 2023-01-15T14:34:59.415 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 392 (ObjNum 543 snap 171 seq_num 543) dirty exists 2023-01-15T14:34:59.415 INFO:tasks.rados.rados.0.smithi090.stdout:1532: left oid 2 (ObjNum 543 snap 171 seq_num 543) 2023-01-15T14:34:59.415 INFO:tasks.rados.rados.0.smithi090.stdout:1534: expect (ObjNum 528 snap 158 seq_num 528) 2023-01-15T14:34:59.482 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:34:59.475+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:34:59.779 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:34:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:34:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:34:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:34:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:59.779 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 392.40359402000001, 2023-01-15T14:34:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:34:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:34:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.780 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:59.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:34:59.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:34:59.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:59.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:34:59.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:34:59.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:34:59.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:34:59.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:59.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:34:59.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:34:59.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:34:59.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:34:59.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:34:59.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:59.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:34:59.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:34:59.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:59.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:34:59.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:34:59.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:59.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:34:59.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:34:59.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:59.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:34:59.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:34:59.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:59.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:34:59.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:34:59.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:59.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:34:59.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:34:59.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:59.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:34:59.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:34:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:34:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:34:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.789 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:59.790 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:59.790 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 392.40319068399998, 2023-01-15T14:34:59.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:34:59.790 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.790 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:34:59.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:59.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:59.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:59.790 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:34:59.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.790 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:59.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:34:59.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:34:59.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:59.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:34:59.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:34:59.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:34:59.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:34:59.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:34:59.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:34:59.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:34:59.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:34:59.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:59.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:34:59.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:34:59.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:34:59.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:34:59.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:59.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:34:59.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:34:59.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:34:59.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:34:59.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:34:59.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:59.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:34:59.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:34:59.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:34:59.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:34:59.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:34:59.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:34:59.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:34:59.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:34:59.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:59.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:34:59.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:34:59.800 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.800 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.800 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.800 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:34:59.800 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:59.800 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 392.30296706399997, 2023-01-15T14:34:59.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:34:59.800 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.800 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:34:59.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:34:59.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:59.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:34:59.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:34:59.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:34:59.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:34:59.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:34:59.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:34:59.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:34:59.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:34:59.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:59.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:34:59.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:34:59.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:34:59.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:34:59.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:59.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:34:59.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:34:59.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:34:59.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:34:59.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:59.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:34:59.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:34:59.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:59.807 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 301.60965183899998, 2023-01-15T14:34:59.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:34:59.808 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.808 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:59.808 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:59.808 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:59.808 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:59.808 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:34:59.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.808 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:59.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:34:59.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:34:59.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:59.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:34:59.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:34:59.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:34:59.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:59.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:34:59.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:34:59.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:34:59.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:34:59.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:59.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:34:59.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:34:59.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.813 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.813 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.813 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:59.813 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:59.814 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 301.60319048500003, 2023-01-15T14:34:59.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:34:59.814 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.814 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:59.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:59.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:59.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:59.814 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:34:59.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.814 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:59.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:34:59.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:34:59.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:59.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:34:59.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:34:59.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:34:59.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:34:59.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:34:59.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:34:59.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:34:59.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:34:59.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:59.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:34:59.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:34:59.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.823 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.824 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:59.824 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:59.824 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 301.50276855999999, 2023-01-15T14:34:59.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:34:59.824 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.824 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:59.824 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:59.824 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:59.824 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:34:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:34:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:34:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:34:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:34:59.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:34:59.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:34:59.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:59.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:34:59.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:34:59.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.829 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.830 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:59.830 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:59.830 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 301.50243107400001, 2023-01-15T14:34:59.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:34:59.830 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.830 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:59.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:59.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:59.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:59.830 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:34:59.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.831 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:34:59.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:34:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:34:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:34:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:34:59.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:34:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:34:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:34:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:34:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.985 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:34:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 301.50237517800002, 2023-01-15T14:34:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:34:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:34:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:34:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.986 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:34:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:34:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:34:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:34:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:34:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.989 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:34:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:34:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:34:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:34:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:34:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:34:59.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:59.991 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 259.66912475999999, 2023-01-15T14:34:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:34:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:34:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:34:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:34:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:34:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:34:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:34:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:34:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:34:59.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:34:59.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:34:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:34:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:34:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:34:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:34:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:34:59.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:34:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:34:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:34:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:34:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:34:59.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:34:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:34:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:34:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:34:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:34:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:34:59.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:34:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:34:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:34:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:34:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:34:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:34:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2207 3.5 3:a1c47f99:::smithi09025264-2:head [write 2548663~763022] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:34:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:34:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 200.694367032, 2023-01-15T14:34:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79657873300000004, 2023-01-15T14:34:59.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2207 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:35:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837590+0000", 2023-01-15T14:35:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837960+0000", 2023-01-15T14:35:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037055900000000001 2023-01-15T14:35:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837980+0000", 2023-01-15T14:35:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9953e-05 2023-01-15T14:35:00.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837988+0000", 2023-01-15T14:35:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2800000000000003e-06 2023-01-15T14:35:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927512+0000", 2023-01-15T14:35:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089523563 2023-01-15T14:35:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927645+0000", 2023-01-15T14:35:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013285799999999999 2023-01-15T14:35:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225466+0000", 2023-01-15T14:35:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29782133100000002 2023-01-15T14:35:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225553+0000", 2023-01-15T14:35:00.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6866000000000005e-05 2023-01-15T14:35:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:35:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225708+0000", 2023-01-15T14:35:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015455199999999999 2023-01-15T14:35:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225971+0000", 2023-01-15T14:35:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026317099999999998 2023-01-15T14:35:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226027+0000", 2023-01-15T14:35:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6601000000000002e-05 2023-01-15T14:35:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226470+0000", 2023-01-15T14:35:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044267500000000001 2023-01-15T14:35:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227372+0000", 2023-01-15T14:35:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090215200000000003 2023-01-15T14:35:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429314+0000", 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20194183499999999 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.633999+0000", 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20468505400000001 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634075+0000", 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:35:00.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634170+0000", 2023-01-15T14:35:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4821999999999994e-05 2023-01-15T14:35:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2208 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (56) in=7b,truncate 3395047] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:35:00.192 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:35:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 200.69394443900001, 2023-01-15T14:35:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79650439900000003, 2023-01-15T14:35:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2208 2023-01-15T14:35:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.193 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:35:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838018+0000", 2023-01-15T14:35:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2549999999999997e-06 2023-01-15T14:35:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:00.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838031+0000", 2023-01-15T14:35:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2717999999999999e-05 2023-01-15T14:35:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838036+0000", 2023-01-15T14:35:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2240000000000003e-06 2023-01-15T14:35:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927676+0000", 2023-01-15T14:35:00.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089639620000000003 2023-01-15T14:35:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927804+0000", 2023-01-15T14:35:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012832300000000001 2023-01-15T14:35:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226096+0000", 2023-01-15T14:35:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29829207400000002 2023-01-15T14:35:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226347+0000", 2023-01-15T14:35:00.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025084800000000002 2023-01-15T14:35:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:35:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226750+0000", 2023-01-15T14:35:00.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040343200000000001 2023-01-15T14:35:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227147+0000", 2023-01-15T14:35:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039682900000000002 2023-01-15T14:35:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227600+0000", 2023-01-15T14:35:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045321199999999998 2023-01-15T14:35:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227859+0000", 2023-01-15T14:35:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259041 2023-01-15T14:35:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227895+0000", 2023-01-15T14:35:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5710999999999997e-05 2023-01-15T14:35:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429405+0000", 2023-01-15T14:35:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201509415 2023-01-15T14:35:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634300+0000", 2023-01-15T14:35:00.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20489528200000001 2023-01-15T14:35:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634409+0000", 2023-01-15T14:35:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109524 2023-01-15T14:35:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634518+0000", 2023-01-15T14:35:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108548 2023-01-15T14:35:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.208 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2209 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:35:00.208 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:35:00.208 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 200.693915624, 2023-01-15T14:35:00.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79724995499999995, 2023-01-15T14:35:00.209 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:00.209 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:00.209 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:00.209 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:00.209 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:00.209 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2209 2023-01-15T14:35:00.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.209 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:00.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:00.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:35:00.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:00.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:35:00.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:00.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:35:00.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:00.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838044+0000", 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6120000000000001e-06 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838056+0000", 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1236000000000001e-05 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838059+0000", 2023-01-15T14:35:00.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6940000000000001e-06 2023-01-15T14:35:00.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.027908+0000", 2023-01-15T14:35:00.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189848287 2023-01-15T14:35:00.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:00.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.028049+0000", 2023-01-15T14:35:00.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014132500000000001 2023-01-15T14:35:00.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227252+0000", 2023-01-15T14:35:00.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19920324 2023-01-15T14:35:00.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:00.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227341+0000", 2023-01-15T14:35:00.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8683000000000005e-05 2023-01-15T14:35:00.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634637+0000", 2023-01-15T14:35:00.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729567300000002 2023-01-15T14:35:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:00.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634980+0000", 2023-01-15T14:35:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034335000000000002 2023-01-15T14:35:00.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.635292+0000", 2023-01-15T14:35:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003123 2023-01-15T14:35:00.601 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2222 3.c 3:3137f0cb:::smithi09025264-20:head [write 3259733~152121] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:35:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:35:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 200.67406406500001, 2023-01-15T14:35:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77651572400000002, 2023-01-15T14:35:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:00.602 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2222 2023-01-15T14:35:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:35:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:35:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857891+0000", 2023-01-15T14:35:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:35:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857993+0000", 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010276399999999999 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858021+0000", 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8022e-05 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:00.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858034+0000", 2023-01-15T14:35:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2549e-05 2023-01-15T14:35:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025371+0000", 2023-01-15T14:35:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16733653100000001 2023-01-15T14:35:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025429+0000", 2023-01-15T14:35:00.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8922000000000003e-05 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226869+0000", 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20143931100000001 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227088+0000", 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219328 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:35:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227443+0000", 2023-01-15T14:35:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035481 2023-01-15T14:35:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227830+0000", 2023-01-15T14:35:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000386702 2023-01-15T14:35:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227861+0000", 2023-01-15T14:35:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1584000000000001e-05 2023-01-15T14:35:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227939+0000", 2023-01-15T14:35:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7354999999999994e-05 2023-01-15T14:35:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:00.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228548+0000", 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060954800000000001 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429817+0000", 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20126904600000001 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634125+0000", 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20430780300000001 2023-01-15T14:35:00.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634259+0000", 2023-01-15T14:35:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134144 2023-01-15T14:35:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634410+0000", 2023-01-15T14:35:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150447 2023-01-15T14:35:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.611 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2223 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (57) in=7b,truncate 3411854] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:35:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:35:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 200.67390527800001, 2023-01-15T14:35:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87671107599999998, 2023-01-15T14:35:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:00.612 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2223 2023-01-15T14:35:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:35:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:00.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:35:00.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:00.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858052+0000", 2023-01-15T14:35:00.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:00.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:00.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858056+0000", 2023-01-15T14:35:00.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5619999999999997e-06 2023-01-15T14:35:00.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:00.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858067+0000", 2023-01-15T14:35:00.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0328e-05 2023-01-15T14:35:00.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:00.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858071+0000", 2023-01-15T14:35:00.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6439999999999999e-06 2023-01-15T14:35:00.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025446+0000", 2023-01-15T14:35:00.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16737407200000001 2023-01-15T14:35:00.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:00.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025503+0000", 2023-01-15T14:35:00.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7952999999999999e-05 2023-01-15T14:35:00.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227970+0000", 2023-01-15T14:35:00.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202466866 2023-01-15T14:35:00.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:00.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228139+0000", 2023-01-15T14:35:00.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016889100000000001 2023-01-15T14:35:00.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:35:00.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228284+0000", 2023-01-15T14:35:00.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014493700000000001 2023-01-15T14:35:00.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:00.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228470+0000", 2023-01-15T14:35:00.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185541 2023-01-15T14:35:00.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:00.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228481+0000", 2023-01-15T14:35:00.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.167e-05 2023-01-15T14:35:00.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:00.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228537+0000", 2023-01-15T14:35:00.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5912999999999998e-05 2023-01-15T14:35:00.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:00.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228563+0000", 2023-01-15T14:35:00.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5613e-05 2023-01-15T14:35:00.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.433181+0000", 2023-01-15T14:35:00.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20461838099999999 2023-01-15T14:35:00.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734620+0000", 2023-01-15T14:35:00.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301438386 2023-01-15T14:35:00.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:00.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734682+0000", 2023-01-15T14:35:00.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2373999999999996e-05 2023-01-15T14:35:00.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:00.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734764+0000", 2023-01-15T14:35:00.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1656999999999994e-05 2023-01-15T14:35:00.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.769 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:00.769 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.769 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2224 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:35:00.769 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:35:00.769 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 200.673880228, 2023-01-15T14:35:00.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97704279199999999, 2023-01-15T14:35:00.770 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:00.770 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:00.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:00.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:00.770 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:00.770 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2224 2023-01-15T14:35:00.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.770 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:00.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:00.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:35:00.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:00.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:35:00.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:00.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858077+0000", 2023-01-15T14:35:00.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:00.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:00.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858080+0000", 2023-01-15T14:35:00.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:35:00.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:00.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858089+0000", 2023-01-15T14:35:00.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3449999999999995e-06 2023-01-15T14:35:00.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:00.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858093+0000", 2023-01-15T14:35:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9820000000000002e-06 2023-01-15T14:35:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025520+0000", 2023-01-15T14:35:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.167426556 2023-01-15T14:35:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025594+0000", 2023-01-15T14:35:00.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4339999999999996e-05 2023-01-15T14:35:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228608+0000", 2023-01-15T14:35:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20301395699999999 2023-01-15T14:35:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228752+0000", 2023-01-15T14:35:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014385 2023-01-15T14:35:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834878+0000", 2023-01-15T14:35:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60612591500000002 2023-01-15T14:35:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834987+0000", 2023-01-15T14:35:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108733 2023-01-15T14:35:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.835120+0000", 2023-01-15T14:35:00.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133688 2023-01-15T14:35:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3120 3.40 3:03b54904:::smithi09025264-12:head [write 578273~466432] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:35:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:35:00.778 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.599928346, 2023-01-15T14:35:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72661194799999995, 2023-01-15T14:35:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3120 2023-01-15T14:35:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.779 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:35:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:35:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932026+0000", 2023-01-15T14:35:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:35:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932397+0000", 2023-01-15T14:35:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037070099999999997 2023-01-15T14:35:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932419+0000", 2023-01-15T14:35:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1923999999999999e-05 2023-01-15T14:35:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932435+0000", 2023-01-15T14:35:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6509999999999999e-05 2023-01-15T14:35:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933751+0000", 2023-01-15T14:35:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013156050000000001 2023-01-15T14:35:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933765+0000", 2023-01-15T14:35:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3952e-05 2023-01-15T14:35:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238456+0000", 2023-01-15T14:35:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30469085000000001 2023-01-15T14:35:00.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238797+0000", 2023-01-15T14:35:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034117300000000001 2023-01-15T14:35:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:35:00.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239214+0000", 2023-01-15T14:35:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000416845 2023-01-15T14:35:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:00.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240273+0000", 2023-01-15T14:35:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058811 2023-01-15T14:35:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240292+0000", 2023-01-15T14:35:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:35:00.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240555+0000", 2023-01-15T14:35:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026299500000000003 2023-01-15T14:35:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:00.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242072+0000", 2023-01-15T14:35:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015175550000000001 2023-01-15T14:35:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244038+0000", 2023-01-15T14:35:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001966176 2023-01-15T14:35:00.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658099+0000", 2023-01-15T14:35:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41406103300000002 2023-01-15T14:35:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658248+0000", 2023-01-15T14:35:00.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148577 2023-01-15T14:35:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658642+0000", 2023-01-15T14:35:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039346900000000002 2023-01-15T14:35:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:00.791 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3121 3.40 3:03b54904:::smithi09025264-12:head [write 1739595~452815] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:35:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:35:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.59949640000001, 2023-01-15T14:35:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72675957599999996, 2023-01-15T14:35:00.792 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3121 2023-01-15T14:35:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:00.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:35:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:00.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:35:00.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932460+0000", 2023-01-15T14:35:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932725+0000", 2023-01-15T14:35:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026482699999999998 2023-01-15T14:35:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:00.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932741+0000", 2023-01-15T14:35:00.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5880999999999999e-05 2023-01-15T14:35:00.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:00.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932750+0000", 2023-01-15T14:35:00.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2380000000000003e-06 2023-01-15T14:35:00.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034479+0000", 2023-01-15T14:35:00.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10172906399999999 2023-01-15T14:35:00.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:00.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034506+0000", 2023-01-15T14:35:00.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7336000000000001e-05 2023-01-15T14:35:00.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240672+0000", 2023-01-15T14:35:00.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206166074 2023-01-15T14:35:00.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:00.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240894+0000", 2023-01-15T14:35:00.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221189 2023-01-15T14:35:00.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:35:00.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241244+0000", 2023-01-15T14:35:00.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000350816 2023-01-15T14:35:00.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:00.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241923+0000", 2023-01-15T14:35:00.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067891200000000005 2023-01-15T14:35:00.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:00.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241943+0000", 2023-01-15T14:35:00.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0069999999999999e-05 2023-01-15T14:35:00.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:00.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243425+0000", 2023-01-15T14:35:00.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001481719 2023-01-15T14:35:00.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:00.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244087+0000", 2023-01-15T14:35:00.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066184700000000002 2023-01-15T14:35:00.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244230+0000", 2023-01-15T14:35:00.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143277 2023-01-15T14:35:00.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658784+0000", 2023-01-15T14:35:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.414553479 2023-01-15T14:35:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658904+0000", 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120024 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659221+0000", 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031742700000000001 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3122 3.40 3:03b54904:::smithi09025264-12:head [write 2722929~655344] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:35:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:35:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.59919230200001, 2023-01-15T14:35:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72706898900000005, 2023-01-15T14:35:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:00.941 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3122 2023-01-15T14:35:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:35:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:35:00.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932765+0000", 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933097+0000", 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220299999999999 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933106+0000", 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5459999999999997e-06 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933113+0000", 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3419999999999996e-06 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034598+0000", 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101485804 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034625+0000", 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6859000000000002e-05 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242147+0000", 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20752189600000001 2023-01-15T14:35:00.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242517+0000", 2023-01-15T14:35:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037032200000000002 2023-01-15T14:35:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:35:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243157+0000", 2023-01-15T14:35:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063935700000000001 2023-01-15T14:35:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243809+0000", 2023-01-15T14:35:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065222000000000001 2023-01-15T14:35:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:00.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243827+0000", 2023-01-15T14:35:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8260000000000001e-05 2023-01-15T14:35:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244122+0000", 2023-01-15T14:35:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029469400000000002 2023-01-15T14:35:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245633+0000", 2023-01-15T14:35:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001511068 2023-01-15T14:35:01.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.346711+0000", 2023-01-15T14:35:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101077812 2023-01-15T14:35:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:01.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659372+0000", 2023-01-15T14:35:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31266154800000001 2023-01-15T14:35:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659488+0000", 2023-01-15T14:35:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115444 2023-01-15T14:35:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659835+0000", 2023-01-15T14:35:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034670300000000002 2023-01-15T14:35:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.259 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3123 3.40 3:03b54904:::smithi09025264-12:head [setxattr _header (58) in=7b,truncate 3378273] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:35:01.259 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:35:01.259 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.598822609, 2023-01-15T14:35:01.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82751564099999997, 2023-01-15T14:35:01.259 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:01.259 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:01.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:01.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:01.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:01.259 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3123 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:01.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933139+0000", 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7079999999999996e-06 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933145+0000", 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7849999999999997e-06 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033331+0000", 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100186076 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034697+0000", 2023-01-15T14:35:01.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001366138 2023-01-15T14:35:01.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:01.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034718+0000", 2023-01-15T14:35:01.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.065e-05 2023-01-15T14:35:01.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:01.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244319+0000", 2023-01-15T14:35:01.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20960071 2023-01-15T14:35:01.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:01.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244576+0000", 2023-01-15T14:35:01.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025712899999999998 2023-01-15T14:35:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:35:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244932+0000", 2023-01-15T14:35:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035626000000000001 2023-01-15T14:35:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:01.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245315+0000", 2023-01-15T14:35:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038305199999999999 2023-01-15T14:35:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245330+0000", 2023-01-15T14:35:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5027e-05 2023-01-15T14:35:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245384+0000", 2023-01-15T14:35:01.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4267000000000001e-05 2023-01-15T14:35:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245654+0000", 2023-01-15T14:35:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026944000000000002 2023-01-15T14:35:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.436220+0000", 2023-01-15T14:35:01.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19056593799999999 2023-01-15T14:35:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760086+0000", 2023-01-15T14:35:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.323865927 2023-01-15T14:35:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760241+0000", 2023-01-15T14:35:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015535700000000001 2023-01-15T14:35:01.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760651+0000", 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040990699999999999 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3124 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:35:01.418 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.498596131, 2023-01-15T14:35:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72792673799999996, 2023-01-15T14:35:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3124 2023-01-15T14:35:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:01.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:35:01.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:01.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:01.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:35:01.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:01.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:01.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033360+0000", 2023-01-15T14:35:01.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:01.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:01.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033372+0000", 2023-01-15T14:35:01.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2850000000000001e-05 2023-01-15T14:35:01.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:01.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033397+0000", 2023-01-15T14:35:01.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4349e-05 2023-01-15T14:35:01.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:01.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033412+0000", 2023-01-15T14:35:01.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5420000000000001e-05 2023-01-15T14:35:01.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:01.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036811+0000", 2023-01-15T14:35:01.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033984169999999999 2023-01-15T14:35:01.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:01.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036821+0000", 2023-01-15T14:35:01.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0213e-05 2023-01-15T14:35:01.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:01.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245712+0000", 2023-01-15T14:35:01.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20889102800000001 2023-01-15T14:35:01.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:01.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245912+0000", 2023-01-15T14:35:01.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020024600000000001 2023-01-15T14:35:01.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:01.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760781+0000", 2023-01-15T14:35:01.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51486881299999998 2023-01-15T14:35:01.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:01.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760970+0000", 2023-01-15T14:35:01.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018893600000000001 2023-01-15T14:35:01.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:01.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:01.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:01.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.761289+0000", 2023-01-15T14:35:01.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031870899999999997 2023-01-15T14:35:01.425 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:01.426 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:01.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:01.426 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:01.426 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:01.426 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:01.427 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:35:01.428 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:35:01.429 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:35:01.430 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:34:59.773+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:01.432 INFO:tasks.rados.rados.0.smithi090.stdout:1531: finishing write tid 1 to smithi09025264-6 2023-01-15T14:35:01.433 INFO:tasks.rados.rados.0.smithi090.stdout:1541: finishing rollback tid 1 to smithi09025264-34 2023-01-15T14:35:01.433 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 34 v 150 (ObjNum 49 snap 0 seq_num 3197216688) dirty dne 2023-01-15T14:35:01.433 INFO:tasks.rados.rados.0.smithi090.stdout:1531: finishing write tid 2 to smithi09025264-6 2023-01-15T14:35:01.433 INFO:tasks.rados.rados.0.smithi090.stdout:1531: finishing write tid 3 to smithi09025264-6 2023-01-15T14:35:01.433 INFO:tasks.rados.rados.0.smithi090.stdout:1531: finishing write tid 4 to smithi09025264-6 2023-01-15T14:35:01.433 INFO:tasks.rados.rados.0.smithi090.stdout:1531: oid 6 updating version 0 to 342 2023-01-15T14:35:01.433 INFO:tasks.rados.rados.0.smithi090.stdout:1531: oid 6 version 342 is already newer than 340 2023-01-15T14:35:01.433 INFO:tasks.rados.rados.0.smithi090.stdout:1531: oid 6 version 342 is already newer than 341 2023-01-15T14:35:01.433 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 342 (ObjNum 542 snap 171 seq_num 542) dirty exists 2023-01-15T14:35:01.433 INFO:tasks.rados.rados.0.smithi090.stdout:1531: left oid 6 (ObjNum 542 snap 171 seq_num 542) 2023-01-15T14:35:01.433 INFO:tasks.rados.rados.0.smithi090.stdout:1535: finishing write tid 1 to smithi09025264-7 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:1535: finishing write tid 2 to smithi09025264-7 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:1537: expect (ObjNum 492 snap 144 seq_num 492) 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:1540: expect (ObjNum 404 snap 108 seq_num 404) 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:1536: finishing rollback tid 0 to smithi09025264-27 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:1535: finishing write tid 3 to smithi09025264-7 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:1535: finishing write tid 4 to smithi09025264-7 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:1535: finishing write tid 5 to smithi09025264-7 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:1535: oid 7 updating version 0 to 248 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:1535: oid 7 updating version 248 to 250 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:1535: oid 7 version 250 is already newer than 247 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:1535: oid 7 version 250 is already newer than 249 2023-01-15T14:35:01.434 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 250 (ObjNum 545 snap 171 seq_num 545) dirty exists 2023-01-15T14:35:01.435 INFO:tasks.rados.rados.0.smithi090.stdout:1535: left oid 7 (ObjNum 545 snap 171 seq_num 545) 2023-01-15T14:35:01.435 INFO:tasks.rados.rados.0.smithi090.stdout:1536: finishing rollback tid 1 to smithi09025264-27 2023-01-15T14:35:01.435 INFO:tasks.rados.rados.0.smithi090.stdout:1531: done (9 left) 2023-01-15T14:35:01.435 INFO:tasks.rados.rados.0.smithi090.stdout:1532: done (8 left) 2023-01-15T14:35:01.435 INFO:tasks.rados.rados.0.smithi090.stdout:1534: done (7 left) 2023-01-15T14:35:01.435 INFO:tasks.rados.rados.0.smithi090.stdout:1535: done (6 left) 2023-01-15T14:35:01.435 INFO:tasks.rados.rados.0.smithi090.stdout:1537: done (5 left) 2023-01-15T14:35:01.435 INFO:tasks.rados.rados.0.smithi090.stdout:1539: done (4 left) 2023-01-15T14:35:01.435 INFO:tasks.rados.rados.0.smithi090.stdout:1540: done (3 left) 2023-01-15T14:35:01.435 INFO:tasks.rados.rados.0.smithi090.stdout:1541: done (2 left) 2023-01-15T14:35:01.435 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:1538: finishing write tid 1 to smithi09025264-3 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:1538: finishing write tid 2 to smithi09025264-3 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:1538: finishing write tid 3 to smithi09025264-3 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:1536: finishing rollback tid 2 to smithi09025264-27 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 296 (ObjNum 421 snap 112 seq_num 421) dirty exists 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:1538: finishing write tid 4 to smithi09025264-3 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:1538: finishing write tid 5 to smithi09025264-3 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:1538: oid 3 updating version 0 to 404 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:1538: oid 3 version 404 is already newer than 403 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:1538: oid 3 version 404 is already newer than 401 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:1538: oid 3 version 404 is already newer than 402 2023-01-15T14:35:01.436 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 404 (ObjNum 546 snap 171 seq_num 546) dirty exists 2023-01-15T14:35:01.437 INFO:tasks.rados.rados.0.smithi090.stdout:1538: left oid 3 (ObjNum 546 snap 171 seq_num 546) 2023-01-15T14:35:01.437 INFO:tasks.rados.rados.0.smithi090.stdout:1536: done (1 left) 2023-01-15T14:35:01.437 INFO:tasks.rados.rados.0.smithi090.stdout:1538: done (0 left) 2023-01-15T14:35:01.437 INFO:tasks.rados.rados.0.smithi090.stdout:1542: done (0 left) 2023-01-15T14:35:01.437 INFO:tasks.rados.rados.0.smithi090.stdout:1543: copy_from oid 31 from oid 9 current snap is 172 2023-01-15T14:35:01.437 INFO:tasks.rados.rados.0.smithi090.stdout:1544: write (excl) oid 4 current snap is 172 2023-01-15T14:35:01.437 INFO:tasks.rados.rados.0.smithi090.stdout:1544: seq_num 547 ranges {793429=622266,1859415=469176,2972024=221405} 2023-01-15T14:35:01.437 INFO:tasks.rados.rados.0.smithi090.stdout:1544: writing smithi09025264-4 from 793429 to 1415695 tid 1 2023-01-15T14:35:01.437 INFO:tasks.rados.rados.0.smithi090.stdout:1544: writing smithi09025264-4 from 1859415 to 2328591 tid 2 2023-01-15T14:35:01.437 INFO:tasks.rados.rados.0.smithi090.stdout:1544: writing smithi09025264-4 from 2972024 to 3193429 tid 3 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout:1545: read oid 43 snap -1 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout:1545: expect (ObjNum 527 snap 158 seq_num 527) 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout:1546: snap_remove snap 161 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout:1546: done (3 left) 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout:1547: write (excl) oid 47 current snap is 172 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout:1547: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout:1547: writing smithi09025264-47 from 410236 to 824104 tid 1 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout:1547: writing smithi09025264-47 from 1234250 to 1963437 tid 2 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout:1547: writing smithi09025264-47 from 2715800 to 3138863 tid 3 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout:1548: snap_create 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:35:01.438 INFO:tasks.rados.rados.0.smithi090.stdout:1545: expect (ObjNum 527 snap 158 seq_num 527) 2023-01-15T14:35:01.439 INFO:tasks.rados.rados.0.smithi090.stdout:1543: finishing copy_from to smithi09025264-31 2023-01-15T14:35:01.439 INFO:tasks.rados.rados.0.smithi090.stdout:1543: got expected ENOENT (src dne) 2023-01-15T14:35:01.439 INFO:tasks.rados.rados.0.smithi090.stdout:1543: finishing copy_from racing read to smithi09025264-31 2023-01-15T14:35:01.439 INFO:tasks.rados.rados.0.smithi090.stdout:1544: finishing write tid 1 to smithi09025264-4 2023-01-15T14:35:01.439 INFO:tasks.rados.rados.0.smithi090.stdout:1544: finishing write tid 2 to smithi09025264-4 2023-01-15T14:35:01.439 INFO:tasks.rados.rados.0.smithi090.stdout:1544: finishing write tid 3 to smithi09025264-4 2023-01-15T14:35:01.439 INFO:tasks.rados.rados.0.smithi090.stdout:1543: done (3 left) 2023-01-15T14:35:01.439 INFO:tasks.rados.rados.0.smithi090.stdout:1545: done (2 left) 2023-01-15T14:35:01.439 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:35:01.439 INFO:tasks.rados.rados.0.smithi090.stdout:1544: finishing write tid 4 to smithi09025264-4 2023-01-15T14:35:01.439 INFO:tasks.rados.rados.0.smithi090.stdout:1544: finishing write tid 5 to smithi09025264-4 2023-01-15T14:35:01.440 INFO:tasks.rados.rados.0.smithi090.stdout:1544: oid 4 updating version 0 to 338 2023-01-15T14:35:01.440 INFO:tasks.rados.rados.0.smithi090.stdout:1544: oid 4 version 338 is already newer than 336 2023-01-15T14:35:01.440 INFO:tasks.rados.rados.0.smithi090.stdout:1544: oid 4 version 338 is already newer than 337 2023-01-15T14:35:01.609 INFO:tasks.rados.rados.0.smithi090.stdout:1544: oid 4 updating version 338 to 339 2023-01-15T14:35:01.609 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 4 v 339 (ObjNum 547 snap 172 seq_num 547) dirty exists 2023-01-15T14:35:01.609 INFO:tasks.rados.rados.0.smithi090.stdout:1544: left oid 4 (ObjNum 547 snap 172 seq_num 547) 2023-01-15T14:35:01.609 INFO:tasks.rados.rados.0.smithi090.stdout:1544: done (1 left) 2023-01-15T14:35:01.609 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:01.610 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:01.422+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:01.614 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:35:01.614 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:35:01.615 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:35:01.615 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:35:01.616 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:35:01.617 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:01.606+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:01.621 INFO:tasks.rados.rados.0.smithi090.stdout:1547: finishing write tid 1 to smithi09025264-47 2023-01-15T14:35:01.828 INFO:tasks.rados.rados.0.smithi090.stdout:1547: finishing write tid 2 to smithi09025264-47 2023-01-15T14:35:01.828 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:01.821+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:01.836 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:01.850 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:01.989 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1] out_osds: [5, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:35:01.990 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:35:01.990 INFO:tasks.thrashosds.thrasher:Adding osd 6 2023-01-15T14:35:01.990 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2023-01-15T14:35:01.992 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:01.998 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:01.998 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:01.999 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:01.999 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:35:02.000 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:01.985+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:02.000 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:35:02.010 INFO:tasks.rados.rados.0.smithi090.stdout:1547: finishing write tid 3 to smithi09025264-47 2023-01-15T14:35:02.010 INFO:tasks.rados.rados.0.smithi090.stdout:1547: finishing write tid 4 to smithi09025264-47 2023-01-15T14:35:02.014 INFO:tasks.rados.rados.0.smithi090.stdout:1547: finishing write tid 5 to smithi09025264-47 2023-01-15T14:35:02.014 INFO:tasks.rados.rados.0.smithi090.stdout:1547: oid 47 updating version 0 to 448 2023-01-15T14:35:02.014 INFO:tasks.rados.rados.0.smithi090.stdout:1547: oid 47 version 448 is already newer than 446 2023-01-15T14:35:02.014 INFO:tasks.rados.rados.0.smithi090.stdout:1547: oid 47 version 448 is already newer than 447 2023-01-15T14:35:02.015 INFO:tasks.rados.rados.0.smithi090.stdout:1547: oid 47 version 448 is already newer than 445 2023-01-15T14:35:02.015 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 448 (ObjNum 548 snap 172 seq_num 548) dirty exists 2023-01-15T14:35:02.015 INFO:tasks.rados.rados.0.smithi090.stdout:1547: left oid 47 (ObjNum 548 snap 172 seq_num 548) 2023-01-15T14:35:02.015 INFO:tasks.rados.rados.0.smithi090.stdout:1547: done (0 left) 2023-01-15T14:35:02.034 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:02.154 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:02.154 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:02.154 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:35:02.154 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:35:02.154 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:35:02.155 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:35:02.155 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:35:02.155 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:35:02.155 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:35:02.155 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:35:02.155 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:35:02.156 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:02.149+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:02.256 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:02.250+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:02.317 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-01-15T14:35:02.306+0000 7fb90e2b4700 -1 mon.a@0(leader).osd e412 definitely_dead 0 2023-01-15T14:35:02.317 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:02.317 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:02.318 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:35:02.354 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:02.349+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:02.433 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:02.433 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:02.439 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:35:02.455 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:02.446+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:02.541 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:02.644 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:02.647 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:35:02.647 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:02.642+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:02.746 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:02.742+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:02.851 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:02.851 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:02.851 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:35:02.852 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:02.845+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:02.952 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:02.946+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:03.056 INFO:tasks.rados.rados.0.smithi090.stdout:1548: done (0 left) 2023-01-15T14:35:03.056 INFO:tasks.rados.rados.0.smithi090.stdout:1549: snap_create 2023-01-15T14:35:03.057 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:03.057 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:03.058 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:35:03.058 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:03.049+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:03.059 INFO:teuthology.orchestra.run.smithi090.stderr:marked in osd.6. 2023-01-15T14:35:03.059 INFO:tasks.thrashosds.thrasher:Added osd 6 2023-01-15T14:35:03.157 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:03.153+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:03.624 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:03.625 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:03.625 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:35:03.625 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:03.618+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:03.726 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:03.721+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:03.785 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:03.786 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:03.786 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:35:03.827 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:03.818+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:03.948 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:03.942+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:03.981 INFO:tasks.rados.rados.0.smithi090.stdout:1549: done (0 left) 2023-01-15T14:35:04.107 INFO:tasks.rados.rados.0.smithi090.stdout:1550: copy_from oid 7 from oid 29 current snap is 174 2023-01-15T14:35:04.107 INFO:tasks.rados.rados.0.smithi090.stdout:1551: copy_from oid 34 from oid 9 current snap is 174 2023-01-15T14:35:04.107 INFO:tasks.rados.rados.0.smithi090.stdout:1552: delete oid 20 current snap is 174 2023-01-15T14:35:04.108 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:35:04.109 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:04.109 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:04.110 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:35:04.110 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:04.105+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:04.156 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:35:04.269 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:04.269 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:04.269 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:04.262+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:04.276 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:35:04.370 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:04.362+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:04.430 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:04.430 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:04.431 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:35:04.431 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:04.431 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:04.442 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:04.459 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:04.470 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:04.462+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:04.486 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:04.591 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:04.591 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:04.592 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:04.593 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:04.593 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:04.593 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:04.593 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:04.594 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:35:04.594 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:04.595 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:35:04.595 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:35:04.595 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:35:04.595 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:35:04.595 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:35:04.595 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:35:04.595 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:35:04.595 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:35:04.596 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:04.586+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:04.692 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:04.686+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:04.752 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:35:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 398.68401444699998, 2023-01-15T14:35:04.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:35:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:35:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:04.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:35:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:35:04.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:35:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:04.755 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:35:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:35:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:35:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:35:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:35:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:35:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:35:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:35:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.757 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:35:04.758 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.759 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:35:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:35:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:35:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:35:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.760 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:35:04.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:04.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 398.68398182999999, 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:35:04.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.764 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:04.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:04.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:35:04.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:04.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:04.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:35:04.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:04.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:04.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:35:04.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:04.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:04.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:35:04.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:35:04.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:04.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:35:04.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:35:04.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.768 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:04.768 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:35:04.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:35:04.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.768 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:35:04.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:35:04.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:04.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:35:04.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:35:04.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:35:04.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:35:04.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:04.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:35:04.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:35:04.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:35:04.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:35:04.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:04.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:35:04.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:35:04.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:04.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:35:04.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:35:04.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:04.773 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:04.773 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:04.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.773 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:04.773 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:35:04.773 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 398.067625276, 2023-01-15T14:35:04.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:35:04.774 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:04.774 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:04.774 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:04.774 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:04.774 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:04.774 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:35:04.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.774 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:04.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:04.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:35:04.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:04.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:04.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:35:04.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:04.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:04.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:35:04.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:35:04.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:04.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:35:04.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:35:04.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:04.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:35:04.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:35:04.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:04.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:35:04.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:35:04.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:35:04.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:35:04.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:04.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:35:04.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:35:04.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:35:04.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:35:04.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:04.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:35:04.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:35:04.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:04.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:35:04.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:35:04.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:04.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:35:04.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:35:04.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:04.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:35:04.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:35:04.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:04.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:35:04.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:35:04.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:35:04.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:35:04.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:04.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:35:04.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:35:04.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:04.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:35:04.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:35:04.784 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:04.785 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:04.785 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:04.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.785 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:04.785 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:35:04.785 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 398.06714292200002, 2023-01-15T14:35:04.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:35:04.785 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:04.785 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:04.785 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:04.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:35:04.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:04.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:04.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:35:04.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:04.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:04.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:35:04.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:35:04.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:04.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:35:04.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:35:04.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:04.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:35:04.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:35:04.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:35:04.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:35:04.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:04.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:35:04.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:04.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:35:04.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:35:04.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:04.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:35:04.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:35:04.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:04.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:35:04.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:35:04.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:04.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:35:04.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:35:04.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:04.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:35:04.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:35:04.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:04.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:35:04.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:35:04.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:04.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:35:04.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:35:04.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:04.926 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:04.926 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:04.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.926 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:04.926 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:35:04.926 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 398.06709035900002, 2023-01-15T14:35:04.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:35:04.927 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:04.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:04.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:04.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:04.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:04.928 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:35:04.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.929 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:04.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:04.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:35:04.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:04.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:04.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:35:04.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:04.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:35:04.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:35:04.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:04.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:35:04.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:35:04.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:35:04.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:35:04.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:04.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:35:04.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:35:04.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:35:04.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:04.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:35:04.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:35:04.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:04.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:35:04.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:35:04.934 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:04.934 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:04.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:04.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:04.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:04.935 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1375 3.12 3:495d4bbc:::smithi09025264-16:head [read 0~1] snapc 0=[] RETRY=2 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e120)", 2023-01-15T14:35:04.935 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:35:04.935 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 297.05164130899999, 2023-01-15T14:35:04.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51009843700000002, 2023-01-15T14:35:04.935 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:04.935 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:04.935 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:04.935 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1375 2023-01-15T14:35:05.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:35:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693918+0000", 2023-01-15T14:35:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693917+0000", 2023-01-15T14:35:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693924+0000", 2023-01-15T14:35:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9160000000000004e-06 2023-01-15T14:35:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693942+0000", 2023-01-15T14:35:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.787e-05 2023-01-15T14:35:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.693956+0000", 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3913e-05 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793092+0000", 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099136625000000006 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.793203+0000", 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011078699999999999 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800697+0000", 2023-01-15T14:35:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0074932940000000002 2023-01-15T14:35:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:05.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:07.800794+0000", 2023-01-15T14:35:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7411000000000006e-05 2023-01-15T14:35:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.202978+0000", 2023-01-15T14:35:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40218354899999997 2023-01-15T14:35:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.203290+0000", 2023-01-15T14:35:05.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031226499999999999 2023-01-15T14:35:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:08.204017+0000", 2023-01-15T14:35:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072687299999999995 2023-01-15T14:35:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1568 3.17 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e139)", 2023-01-15T14:35:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:35:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 278.45472811000002, 2023-01-15T14:35:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58983439500000001, 2023-01-15T14:35:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1568 2023-01-15T14:35:05.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:35:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:35:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290831+0000", 2023-01-15T14:35:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290834+0000", 2023-01-15T14:35:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6569999999999999e-06 2023-01-15T14:35:05.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.290843+0000", 2023-01-15T14:35:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0359999999999995e-06 2023-01-15T14:35:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.390943+0000", 2023-01-15T14:35:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099438 2023-01-15T14:35:05.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581152+0000", 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19020936399999999 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.581477+0000", 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032506700000000001 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880302+0000", 2023-01-15T14:35:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29882509299999999 2023-01-15T14:35:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880461+0000", 2023-01-15T14:35:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159128 2023-01-15T14:35:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:26.880666+0000", 2023-01-15T14:35:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000204339 2023-01-15T14:35:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3562 3.57 3:ea27a813:::smithi09025264-43:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2023-01-15T14:35:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:35:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 68.660053531000003, 2023-01-15T14:35:05.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59995680200000001, 2023-01-15T14:35:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3562 2023-01-15T14:35:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:35:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:35:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085504+0000", 2023-01-15T14:35:05.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085513+0000", 2023-01-15T14:35:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7360000000000007e-06 2023-01-15T14:35:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085528+0000", 2023-01-15T14:35:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4671999999999999e-05 2023-01-15T14:35:05.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085539+0000", 2023-01-15T14:35:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.113e-05 2023-01-15T14:35:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382469+0000", 2023-01-15T14:35:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.296929736 2023-01-15T14:35:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382623+0000", 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015408300000000001 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382985+0000", 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036232300000000001 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:05.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383762+0000", 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077680999999999996 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383779+0000", 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7617000000000001e-05 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383838+0000", 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8795000000000002e-05 2023-01-15T14:35:05.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383918+0000", 2023-01-15T14:35:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9524999999999995e-05 2023-01-15T14:35:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.385030+0000", 2023-01-15T14:35:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001112664 2023-01-15T14:35:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685250+0000", 2023-01-15T14:35:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30021942800000001 2023-01-15T14:35:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:05.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685384+0000", 2023-01-15T14:35:05.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134055 2023-01-15T14:35:05.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685463+0000", 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8991000000000003e-05 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3618 3.43 3:c3b8cf33:::smithi09025264-26:head [copy-from ver 335] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e352)", 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 62.838294294999997, 2023-01-15T14:35:05.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73981128600000001, 2023-01-15T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3618 2023-01-15T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907263+0000", 2023-01-15T14:35:05.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:05.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907273+0000", 2023-01-15T14:35:05.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8109999999999997e-06 2023-01-15T14:35:05.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907287+0000", 2023-01-15T14:35:05.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4100999999999999e-05 2023-01-15T14:35:05.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907298+0000", 2023-01-15T14:35:05.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1014e-05 2023-01-15T14:35:05.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907509+0000", 2023-01-15T14:35:05.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021028299999999999 2023-01-15T14:35:05.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:05.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907519+0000", 2023-01-15T14:35:05.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0686999999999999e-05 2023-01-15T14:35:05.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.212673+0000", 2023-01-15T14:35:05.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30515353299999998 2023-01-15T14:35:05.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.213072+0000", 2023-01-15T14:35:05.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039955900000000001 2023-01-15T14:35:05.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:35:05.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.245152+0000", 2023-01-15T14:35:05.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032080139000000001 2023-01-15T14:35:05.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:05.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247179+0000", 2023-01-15T14:35:05.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020266640000000001 2023-01-15T14:35:05.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:05.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247198+0000", 2023-01-15T14:35:05.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:35:05.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:05.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.248181+0000", 2023-01-15T14:35:05.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0009830310000000001 2023-01-15T14:35:05.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:05.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.248229+0000", 2023-01-15T14:35:05.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7592000000000001e-05 2023-01-15T14:35:05.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.256 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.253799+0000", 2023-01-15T14:35:05.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055704659999999996 2023-01-15T14:35:05.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.646730+0000", 2023-01-15T14:35:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39293027400000002 2023-01-15T14:35:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.646885+0000", 2023-01-15T14:35:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155243 2023-01-15T14:35:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647077+0000", 2023-01-15T14:35:05.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019156899999999999 2023-01-15T14:35:05.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.258 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.258 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.258 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3619 3.43 3:c3b8cf33:::smithi09025264-26:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e352)", 2023-01-15T14:35:05.258 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:35:05.258 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 62.838242835999999, 2023-01-15T14:35:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.74041005699999995, 2023-01-15T14:35:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3619 2023-01-15T14:35:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:35:05.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907316+0000", 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907320+0000", 2023-01-15T14:35:05.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5820000000000002e-06 2023-01-15T14:35:05.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907328+0000", 2023-01-15T14:35:05.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.32e-06 2023-01-15T14:35:05.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907334+0000", 2023-01-15T14:35:05.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5489999999999997e-06 2023-01-15T14:35:05.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.008282+0000", 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100948497 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.008303+0000", 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0996e-05 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.214382+0000", 2023-01-15T14:35:05.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20607940899999999 2023-01-15T14:35:05.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:05.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.214553+0000", 2023-01-15T14:35:05.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017038000000000001 2023-01-15T14:35:05.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247445+0000", 2023-01-15T14:35:05.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032892271000000001 2023-01-15T14:35:05.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247684+0000", 2023-01-15T14:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023870599999999999 2023-01-15T14:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647212+0000", 2023-01-15T14:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39952834599999998 2023-01-15T14:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647450+0000", 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023761799999999999 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647727+0000", 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027702100000000002 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.268 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3654 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (58) in=7b,truncate 2386693] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2023-01-15T14:35:05.268 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:19.020570+0000", 2023-01-15T14:35:05.268 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.724989346999998, 2023-01-15T14:35:05.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60711035099999999, 2023-01-15T14:35:05.407 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.407 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:05.408 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.408 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.408 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.408 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3654 2023-01-15T14:35:05.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.408 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020570+0000", 2023-01-15T14:35:05.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020570+0000", 2023-01-15T14:35:05.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020569+0000", 2023-01-15T14:35:05.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020575+0000", 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2220000000000001e-06 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020585+0000", 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0664e-05 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020592+0000", 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4500000000000001e-06 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120721+0000", 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971699999999 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:05.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120877+0000", 2023-01-15T14:35:05.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015582200000000001 2023-01-15T14:35:05.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.425053+0000", 2023-01-15T14:35:05.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30417533000000002 2023-01-15T14:35:05.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.425328+0000", 2023-01-15T14:35:05.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027568299999999998 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.425793+0000", 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046481000000000002 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426265+0000", 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047148999999999997 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426283+0000", 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8697000000000001e-05 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426366+0000", 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3008000000000002e-05 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:05.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426599+0000", 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023217999999999999 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.527135+0000", 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10053680299999999 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627472+0000", 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10033634299999999 2023-01-15T14:35:05.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:05.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627569+0000", 2023-01-15T14:35:05.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7781000000000005e-05 2023-01-15T14:35:05.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627681+0000", 2023-01-15T14:35:05.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111117 2023-01-15T14:35:05.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.416 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3655 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e369)", 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:19.020607+0000", 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 45.724952399000003, 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60748634800000001, 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3655 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020607+0000", 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020607+0000", 2023-01-15T14:35:05.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020606+0000", 2023-01-15T14:35:05.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:05.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020611+0000", 2023-01-15T14:35:05.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1860000000000002e-06 2023-01-15T14:35:05.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020627+0000", 2023-01-15T14:35:05.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6294999999999999e-05 2023-01-15T14:35:05.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120742+0000", 2023-01-15T14:35:05.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100114781 2023-01-15T14:35:05.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120971+0000", 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022973700000000001 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.121239+0000", 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026751500000000002 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426424+0000", 2023-01-15T14:35:05.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30518495299999998 2023-01-15T14:35:05.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:05.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426525+0000", 2023-01-15T14:35:05.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101542 2023-01-15T14:35:05.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627751+0000", 2023-01-15T14:35:05.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20122512000000001 2023-01-15T14:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627909+0000", 2023-01-15T14:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015884699999999999 2023-01-15T14:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.628093+0000", 2023-01-15T14:35:05.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018405200000000001 2023-01-15T14:35:05.424 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.424 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.424 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.424 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3798 3.10 3:094fbf13:::smithi09025264-31:head [write 739640~641484] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e388)", 2023-01-15T14:35:05.424 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:37.949180+0000", 2023-01-15T14:35:05.424 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.796379229999999, 2023-01-15T14:35:05.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51653281500000003, 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3798 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.949180+0000", 2023-01-15T14:35:05.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.949180+0000", 2023-01-15T14:35:05.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.949179+0000", 2023-01-15T14:35:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.949426+0000", 2023-01-15T14:35:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024744099999999999 2023-01-15T14:35:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.949445+0000", 2023-01-15T14:35:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9301e-05 2023-01-15T14:35:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.949455+0000", 2023-01-15T14:35:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7289999999999995e-06 2023-01-15T14:35:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.135713+0000", 2023-01-15T14:35:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18625755999999999 2023-01-15T14:35:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.136076+0000", 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036331100000000001 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.136578+0000", 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050157200000000004 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:05.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.138472+0000", 2023-01-15T14:35:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0018939689999999999 2023-01-15T14:35:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.140234+0000", 2023-01-15T14:35:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001762448 2023-01-15T14:35:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:05.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.140878+0000", 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064421699999999999 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.140969+0000", 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0544999999999995e-05 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.233493+0000", 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.092523870999999994 2023-01-15T14:35:05.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.465469+0000", 2023-01-15T14:35:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23197684199999999 2023-01-15T14:35:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.465622+0000", 2023-01-15T14:35:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000152631 2023-01-15T14:35:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.465713+0000", 2023-01-15T14:35:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1032000000000005e-05 2023-01-15T14:35:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3801 3.10 3:094fbf13:::smithi09025264-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e388)", 2023-01-15T14:35:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:37.956852+0000", 2023-01-15T14:35:05.573 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.788707059, 2023-01-15T14:35:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.510330493, 2023-01-15T14:35:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3801 2023-01-15T14:35:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.956852+0000", 2023-01-15T14:35:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.956852+0000", 2023-01-15T14:35:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.956852+0000", 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.956854+0000", 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8679999999999999e-06 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.956861+0000", 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7170000000000002e-06 2023-01-15T14:35:05.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:37.956864+0000", 2023-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9510000000000002e-06 2023-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.239656+0000", 2023-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28279220500000002 2023-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.239839+0000", 2023-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018325000000000001 2023-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.466716+0000", 2023-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22687718600000001 2023-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.466986+0000", 2023-01-15T14:35:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026978899999999999 2023-01-15T14:35:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:38.467183+0000", 2023-01-15T14:35:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019677799999999999 2023-01-15T14:35:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.579 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.65 391/387 [PushOp(3:a75958b1:::smithi09025264-6:f4, version: 371'125, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a75958b1:::smithi09025264-6:f4@371'125, size: 2353046, copy_subset: [0~2353046], clone_subset: {3:a75958b1:::smithi09025264-6:111=[]}, snapset: 12b=[]:{f4=[f2,ed,ec],104=[102,fc,f8,f7,f6],10c=[10a],111=[10f],112=[112],11c=[119,117,115]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:41.938596+0000", 2023-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.806963883999998, 2023-01-15T14:35:05.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62354823699999995, 2023-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.938596+0000", 2023-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.938596+0000", 2023-01-15T14:35:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.938595+0000", 2023-01-15T14:35:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.939286+0000", 2023-01-15T14:35:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069103999999999997 2023-01-15T14:35:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.939310+0000", 2023-01-15T14:35:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4451e-05 2023-01-15T14:35:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.939328+0000", 2023-01-15T14:35:05.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.785e-05 2023-01-15T14:35:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:42.562117+0000", 2023-01-15T14:35:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62278947399999995 2023-01-15T14:35:05.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:42.562144+0000", 2023-01-15T14:35:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6526e-05 2023-01-15T14:35:05.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3998 3.32 3:4d8f159b:::smithi09025264-4:head [write 793429~622266,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2023-01-15T14:35:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:59.985797+0000", 2023-01-15T14:35:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.759762888, 2023-01-15T14:35:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.58454122200000003, 2023-01-15T14:35:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:05.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3998 2023-01-15T14:35:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.985797+0000", 2023-01-15T14:35:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.985797+0000", 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.985795+0000", 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.985949+0000", 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000154213 2023-01-15T14:35:05.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.985964+0000", 2023-01-15T14:35:05.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4876e-05 2023-01-15T14:35:05.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.985974+0000", 2023-01-15T14:35:05.737 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0407999999999999e-05 2023-01-15T14:35:05.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.737 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.074607+0000", 2023-01-15T14:35:05.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.088632889000000006 2023-01-15T14:35:05.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.075017+0000", 2023-01-15T14:35:05.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041004099999999999 2023-01-15T14:35:05.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:35:05.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.075384+0000", 2023-01-15T14:35:05.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036647300000000003 2023-01-15T14:35:05.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:05.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.076469+0000", 2023-01-15T14:35:05.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010851739999999999 2023-01-15T14:35:05.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:05.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.076494+0000", 2023-01-15T14:35:05.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4536e-05 2023-01-15T14:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.076813+0000", 2023-01-15T14:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031990000000000002 2023-01-15T14:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.077464+0000", 2023-01-15T14:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065036300000000005 2023-01-15T14:35:05.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.280327+0000", 2023-01-15T14:35:05.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20286269200000001 2023-01-15T14:35:05.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.741 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.741 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.570025+0000", 2023-01-15T14:35:05.741 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.289698385 2023-01-15T14:35:05.741 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.741 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:05.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.570146+0000", 2023-01-15T14:35:05.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012080500000000001 2023-01-15T14:35:05.742 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.742 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.742 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.742 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.570338+0000", 2023-01-15T14:35:05.742 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019215899999999999 2023-01-15T14:35:05.742 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.742 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.742 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.743 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.743 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.743 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3999 3.32 3:4d8f159b:::smithi09025264-4:head [write 1859415~469176] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2023-01-15T14:35:05.743 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:59.991169+0000", 2023-01-15T14:35:05.743 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.7543902999999998, 2023-01-15T14:35:05.743 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59728521800000001, 2023-01-15T14:35:05.743 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.743 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:05.743 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.743 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3999 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.991169+0000", 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.744 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.991169+0000", 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.991168+0000", 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.991304+0000", 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013587199999999999 2023-01-15T14:35:05.745 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:59.991320+0000", 2023-01-15T14:35:05.746 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6180000000000001e-05 2023-01-15T14:35:05.746 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.746 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.746 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.746 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.091426+0000", 2023-01-15T14:35:05.747 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100105681 2023-01-15T14:35:05.747 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.747 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.748 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.748 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.091542+0000", 2023-01-15T14:35:05.748 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000116037 2023-01-15T14:35:05.748 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.748 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.749 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:05.749 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.091744+0000", 2023-01-15T14:35:05.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020185600000000001 2023-01-15T14:35:05.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.570483+0000", 2023-01-15T14:35:05.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.478739419 2023-01-15T14:35:05.750 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.750 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.750 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.750 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.570658+0000", 2023-01-15T14:35:05.750 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017535900000000001 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.570946+0000", 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028777099999999998 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.571350+0000", 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000403655 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.751 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.571400+0000", 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0475e-05 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.571661+0000", 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026084200000000001 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:05.752 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.572866+0000", 2023-01-15T14:35:05.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012046509999999999 2023-01-15T14:35:05.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.588244+0000", 2023-01-15T14:35:05.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015378549 2023-01-15T14:35:05.753 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.753 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.753 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.753 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.588318+0000", 2023-01-15T14:35:05.753 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3986999999999995e-05 2023-01-15T14:35:05.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:05.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.588373+0000", 2023-01-15T14:35:05.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5038999999999999e-05 2023-01-15T14:35:05.754 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.754 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.754 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.754 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.588454+0000", 2023-01-15T14:35:05.754 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1131999999999995e-05 2023-01-15T14:35:05.754 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4000 3.32 3:4d8f159b:::smithi09025264-4:head [write 2972024~221405] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:00.091474+0000", 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.6540850950000001, 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59829139600000003, 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.755 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.756 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.756 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4000 2023-01-15T14:35:05.756 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.756 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.756 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.756 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.756 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.091474+0000", 2023-01-15T14:35:05.756 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.091474+0000", 2023-01-15T14:35:05.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.091472+0000", 2023-01-15T14:35:05.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:05.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.091607+0000", 2023-01-15T14:35:05.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013529699999999999 2023-01-15T14:35:05.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.091634+0000", 2023-01-15T14:35:05.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7002e-05 2023-01-15T14:35:05.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191743+0000", 2023-01-15T14:35:05.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10010881000000001 2023-01-15T14:35:05.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191793+0000", 2023-01-15T14:35:05.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0151999999999997e-05 2023-01-15T14:35:05.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:05.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191913+0000", 2023-01-15T14:35:05.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119851 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.571438+0000", 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37952533999999999 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.571574+0000", 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013553400000000001 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:35:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.572060+0000", 2023-01-15T14:35:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00048638300000000001 2023-01-15T14:35:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.572734+0000", 2023-01-15T14:35:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067380599999999999 2023-01-15T14:35:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:05.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.572767+0000", 2023-01-15T14:35:05.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2950000000000001e-05 2023-01-15T14:35:05.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:05.912 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.573271+0000", 2023-01-15T14:35:05.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050385899999999995 2023-01-15T14:35:05.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.912 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:05.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.573373+0000", 2023-01-15T14:35:05.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010242399999999999 2023-01-15T14:35:05.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.588516+0000", 2023-01-15T14:35:05.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.015143301 2023-01-15T14:35:05.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.689531+0000", 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101014862 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.689623+0000", 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1887999999999998e-05 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.689766+0000", 2023-01-15T14:35:05.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014260999999999999 2023-01-15T14:35:05.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.915 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:05.915 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:05.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.915 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4001 3.32 3:4d8f159b:::smithi09025264-4:head [setxattr _header (57) in=7b,truncate 3193429] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2023-01-15T14:35:05.915 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:00.191763+0000", 2023-01-15T14:35:05.915 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.5537970139999997, 2023-01-15T14:35:05.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59790821100000002, 2023-01-15T14:35:05.916 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:05.916 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:05.916 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:05.916 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:05.916 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:05.916 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4001 2023-01-15T14:35:05.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.916 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:05.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191763+0000", 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191763+0000", 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191761+0000", 2023-01-15T14:35:05.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191770+0000", 2023-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5469999999999997e-06 2023-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191780+0000", 2023-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.004e-05 2023-01-15T14:35:05.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191786+0000", 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0170000000000002e-06 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191941+0000", 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015572700000000001 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:05.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.192043+0000", 2023-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010119300000000001 2023-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.587472+0000", 2023-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39542982500000001 2023-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.587671+0000", 2023-01-15T14:35:05.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019892 2023-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.587885+0000", 2023-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021382000000000001 2023-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:05.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.588107+0000", 2023-01-15T14:35:06.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221721 2023-01-15T14:35:06.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:06.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.588125+0000", 2023-01-15T14:35:06.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8444000000000002e-05 2023-01-15T14:35:06.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:06.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.588184+0000", 2023-01-15T14:35:06.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9135000000000002e-05 2023-01-15T14:35:06.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:06.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.588280+0000", 2023-01-15T14:35:06.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5866999999999995e-05 2023-01-15T14:35:06.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:06.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.689448+0000", 2023-01-15T14:35:06.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10116784700000001 2023-01-15T14:35:06.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:06.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.789413+0000", 2023-01-15T14:35:06.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099965008999999994 2023-01-15T14:35:06.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:06.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.789521+0000", 2023-01-15T14:35:06.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010771400000000001 2023-01-15T14:35:06.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:06.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.789671+0000", 2023-01-15T14:35:06.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014988999999999999 2023-01-15T14:35:06.088 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.088 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:06.088 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.088 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4002 3.32 3:4d8f159b:::smithi09025264-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e410)", 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:00.191793+0000", 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 4.5537666669999997, 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59839242599999998, 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4002 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:06.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191793+0000", 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191793+0000", 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:06.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191793+0000", 2023-01-15T14:35:06.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:06.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:06.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191796+0000", 2023-01-15T14:35:06.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2569999999999999e-06 2023-01-15T14:35:06.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:06.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191805+0000", 2023-01-15T14:35:06.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8829999999999995e-06 2023-01-15T14:35:06.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:06.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.191810+0000", 2023-01-15T14:35:06.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0599999999999998e-06 2023-01-15T14:35:06.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.192078+0000", 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026869799999999999 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.192191+0000", 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000112938 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.689079+0000", 2023-01-15T14:35:06.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49688738199999999 2023-01-15T14:35:06.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.689258+0000", 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017956299999999999 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.789756+0000", 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10049802300000001 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.789969+0000", 2023-01-15T14:35:06.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021269299999999999 2023-01-15T14:35:06.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:06.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:00.790185+0000", 2023-01-15T14:35:06.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021631100000000001 2023-01-15T14:35:06.097 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.097 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:06.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.098 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:06.098 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:06.098 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:35:06.099 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:04.917+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:06.100 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:35:06.102 INFO:tasks.rados.rados.0.smithi090.stdout:1552: done (2 left) 2023-01-15T14:35:06.102 INFO:tasks.rados.rados.0.smithi090.stdout:1553: delete oid 2 current snap is 174 2023-01-15T14:35:06.103 INFO:tasks.rados.rados.0.smithi090.stdout:1551: finishing copy_from to smithi09025264-34 2023-01-15T14:35:06.103 INFO:tasks.rados.rados.0.smithi090.stdout:1551: got expected ENOENT (src dne) 2023-01-15T14:35:06.103 INFO:tasks.rados.rados.0.smithi090.stdout:1551: finishing copy_from racing read to smithi09025264-34 2023-01-15T14:35:06.103 INFO:tasks.rados.rados.0.smithi090.stdout:1551: done (2 left) 2023-01-15T14:35:06.103 INFO:tasks.rados.rados.0.smithi090.stdout:1553: done (1 left) 2023-01-15T14:35:06.103 INFO:tasks.rados.rados.0.smithi090.stdout:1554: write (excl) oid 6 current snap is 174 2023-01-15T14:35:06.103 INFO:tasks.rados.rados.0.smithi090.stdout:1554: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2023-01-15T14:35:06.104 INFO:tasks.rados.rados.0.smithi090.stdout:1554: writing smithi09025264-6 from 460657 to 1100272 tid 1 2023-01-15T14:35:06.104 INFO:tasks.rados.rados.0.smithi090.stdout:1554: writing smithi09025264-6 from 1693343 to 2169857 tid 2 2023-01-15T14:35:06.104 INFO:tasks.rados.rados.0.smithi090.stdout:1554: writing smithi09025264-6 from 2681716 to 3260657 tid 3 2023-01-15T14:35:06.104 INFO:tasks.rados.rados.0.smithi090.stdout:1555: snap_create 2023-01-15T14:35:06.104 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:35:06.104 INFO:tasks.rados.rados.0.smithi090.stdout:1550: finishing copy_from to smithi09025264-7 2023-01-15T14:35:06.104 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 274 (ObjNum 265 snap 65 seq_num 265) dirty exists 2023-01-15T14:35:06.104 INFO:tasks.rados.rados.0.smithi090.stdout:1554: finishing write tid 1 to smithi09025264-6 2023-01-15T14:35:06.104 INFO:tasks.rados.rados.0.smithi090.stdout:1550: finishing copy_from racing read to smithi09025264-7 2023-01-15T14:35:06.104 INFO:tasks.rados.rados.0.smithi090.stdout:1550: done (1 left) 2023-01-15T14:35:06.104 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:06.105 INFO:tasks.rados.rados.0.smithi090.stdout:1554: finishing write tid 2 to smithi09025264-6 2023-01-15T14:35:06.105 INFO:tasks.rados.rados.0.smithi090.stdout:1554: finishing write tid 3 to smithi09025264-6 2023-01-15T14:35:06.105 INFO:tasks.rados.rados.0.smithi090.stdout:1554: finishing write tid 4 to smithi09025264-6 2023-01-15T14:35:06.105 INFO:tasks.rados.rados.0.smithi090.stdout:1554: finishing write tid 5 to smithi09025264-6 2023-01-15T14:35:06.105 INFO:tasks.rados.rados.0.smithi090.stdout:1554: oid 6 updating version 0 to 344 2023-01-15T14:35:06.105 INFO:tasks.rados.rados.0.smithi090.stdout:1554: oid 6 version 344 is already newer than 343 2023-01-15T14:35:06.105 INFO:tasks.rados.rados.0.smithi090.stdout:1554: oid 6 updating version 344 to 345 2023-01-15T14:35:06.106 INFO:tasks.rados.rados.0.smithi090.stdout:1554: oid 6 updating version 345 to 346 2023-01-15T14:35:06.106 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 346 (ObjNum 551 snap 174 seq_num 551) dirty exists 2023-01-15T14:35:06.106 INFO:tasks.rados.rados.0.smithi090.stdout:1554: left oid 6 (ObjNum 551 snap 174 seq_num 551) 2023-01-15T14:35:06.106 INFO:tasks.rados.rados.0.smithi090.stdout:1554: done (0 left) 2023-01-15T14:35:06.106 INFO:tasks.rados.rados.0.smithi090.stdout:1555: done (0 left) 2023-01-15T14:35:06.106 INFO:tasks.rados.rados.0.smithi090.stdout:1556: copy_from oid 46 from oid 1 current snap is 175 2023-01-15T14:35:06.106 INFO:tasks.rados.rados.0.smithi090.stdout:1557: delete oid 8 current snap is 175 2023-01-15T14:35:06.107 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:06.093+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:06.201 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:06.194+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:06.255 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4026 3.4 3.5fbf4404 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e416)", 2023-01-15T14:35:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:06.064231+0000", 2023-01-15T14:35:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.188740875, 2023-01-15T14:35:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.188781954, 2023-01-15T14:35:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:35:06.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4026 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064231+0000", 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064231+0000", 2023-01-15T14:35:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:06.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064229+0000", 2023-01-15T14:35:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064240+0000", 2023-01-15T14:35:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1678999999999999e-05 2023-01-15T14:35:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:06.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064255+0000", 2023-01-15T14:35:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4917e-05 2023-01-15T14:35:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064267+0000", 2023-01-15T14:35:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1953999999999999e-05 2023-01-15T14:35:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4027 3.4 3.5fbf4404 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e416)", 2023-01-15T14:35:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:06.064279+0000", 2023-01-15T14:35:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.18869270899999999, 2023-01-15T14:35:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18882795299999999, 2023-01-15T14:35:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:35:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:06.261 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4027 2023-01-15T14:35:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064279+0000", 2023-01-15T14:35:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064279+0000", 2023-01-15T14:35:06.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:06.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:06.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064278+0000", 2023-01-15T14:35:06.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:06.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:06.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064282+0000", 2023-01-15T14:35:06.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7060000000000002e-06 2023-01-15T14:35:06.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064290+0000", 2023-01-15T14:35:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9240000000000007e-06 2023-01-15T14:35:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064295+0000", 2023-01-15T14:35:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7899999999999999e-06 2023-01-15T14:35:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.397 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4028 3.51 3:8b0a2647:::smithi09025264-8:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e416)", 2023-01-15T14:35:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:06.064301+0000", 2023-01-15T14:35:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.18867034199999999, 2023-01-15T14:35:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18889236700000001, 2023-01-15T14:35:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:06.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4028 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064301+0000", 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064301+0000", 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064301+0000", 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064304+0000", 2023-01-15T14:35:06.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1970000000000001e-06 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064310+0000", 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0100000000000001e-06 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.064315+0000", 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4020000000000004e-06 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.159072+0000", 2023-01-15T14:35:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.094756926000000005 2023-01-15T14:35:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:06.159241+0000", 2023-01-15T14:35:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000169677 2023-01-15T14:35:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:06.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:35:06.403 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 3 2023-01-15T14:35:06.403 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:06.405 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:35:06.405 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:06.398+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:06.462 INFO:tasks.rados.rados.0.smithi090.stdout:1557: done (1 left) 2023-01-15T14:35:06.562 INFO:tasks.rados.rados.0.smithi090.stdout:1558: read oid 45 snap -1 2023-01-15T14:35:06.562 INFO:tasks.rados.rados.0.smithi090.stdout:1558: expect (ObjNum 400 snap 106 seq_num 400) 2023-01-15T14:35:06.564 INFO:tasks.rados.rados.0.smithi090.stdout:1559: write (excl) oid 17 current snap is 175 2023-01-15T14:35:06.717 INFO:tasks.rados.rados.0.smithi090.stdout:1559: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2023-01-15T14:35:06.717 INFO:tasks.rados.rados.0.smithi090.stdout:1559: writing smithi09025264-17 from 494271 to 1284384 tid 1 2023-01-15T14:35:06.718 INFO:tasks.rados.rados.0.smithi090.stdout:1559: writing smithi09025264-17 from 1693954 to 2174137 tid 2 2023-01-15T14:35:06.718 INFO:tasks.rados.rados.0.smithi090.stdout:1559: writing smithi09025264-17 from 2620209 to 3132338 tid 3 2023-01-15T14:35:06.718 INFO:tasks.rados.rados.0.smithi090.stdout:1560: copy_from oid 14 from oid 43 current snap is 175 2023-01-15T14:35:06.718 INFO:tasks.rados.rados.0.smithi090.stdout:1556: finishing copy_from to smithi09025264-46 2023-01-15T14:35:06.718 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 425 (ObjNum 525 snap 158 seq_num 525) dirty exists 2023-01-15T14:35:06.718 INFO:tasks.rados.rados.0.smithi090.stdout:1556: finishing copy_from racing read to smithi09025264-46 2023-01-15T14:35:06.718 INFO:tasks.rados.rados.0.smithi090.stdout:1556: done (3 left) 2023-01-15T14:35:06.718 INFO:tasks.rados.rados.0.smithi090.stdout:1561: copy_from oid 32 from oid 47 current snap is 175 2023-01-15T14:35:06.718 INFO:tasks.rados.rados.0.smithi090.stdout:1562: read oid 38 snap -1 2023-01-15T14:35:06.718 INFO:tasks.rados.rados.0.smithi090.stdout:1562: expect deleted 2023-01-15T14:35:06.719 INFO:tasks.rados.rados.0.smithi090.stdout:1563: snap_remove snap 143 2023-01-15T14:35:06.719 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:06.720 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:06.720 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:06.720 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:06.720 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:06.720 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:06.558+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:06.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:35:06.722 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:35:06.723 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:06.714+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:06.824 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:06.817+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:06.884 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:06.884 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:06.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:06.884 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:06.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.885 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:06.885 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:06.885 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 399.738896754, 2023-01-15T14:35:06.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:35:06.885 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:06.886 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:06.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:06.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:06.886 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:06.886 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:35:06.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.886 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:06.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:06.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:06.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:06.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:06.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:06.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:06.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:06.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:06.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:06.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:06.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:35:06.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:35:06.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:06.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:35:06.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:35:06.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:06.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:35:06.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:35:06.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:35:06.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:35:06.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:06.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:06.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:35:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:35:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:35:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:35:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:35:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:35:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:07.030 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 399.73859695700003, 2023-01-15T14:35:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:35:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:35:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:35:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:35:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:35:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:35:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:35:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:35:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:35:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:35:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:35:07.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:35:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:35:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:35:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:35:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:35:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:35:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:35:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:35:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:35:07.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:35:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:35:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:35:07.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:35:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:35:07.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:35:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:35:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:35:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:35:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:35:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:35:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:07.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:35:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:35:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:35:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:35:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.056 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 399.73856717299998, 2023-01-15T14:35:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:35:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.057 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:35:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:35:07.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:35:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:35:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:35:07.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:35:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:35:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:35:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:35:07.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:35:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:35:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:35:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:35:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:35:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:35:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:35:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:35:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:35:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:35:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:35:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:35:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:35:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 399.608758373, 2023-01-15T14:35:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:35:07.065 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:35:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:35:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:35:07.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:35:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:35:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:35:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:35:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:35:07.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:35:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:35:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:35:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:35:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:35:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:35:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:35:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:35:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:35:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:35:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:35:07.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:35:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:07.256 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:35:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:35:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:35:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:35:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:35:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:35:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:35:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:35:07.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:35:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:35:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:35:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:35:07.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:35:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:35:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.261 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:07.261 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:07.261 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 399.60870167600001, 2023-01-15T14:35:07.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:35:07.261 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.261 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:07.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.261 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.261 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:35:07.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.262 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:07.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:07.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:35:07.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:07.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.263 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:35:07.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:35:07.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:35:07.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:35:07.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 312.90143309600001, 2023-01-15T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.377 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.394 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:35:07.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:35:07.400 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 308.94501360300001, 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.401 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:35:07.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:07.407 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 308.88751592, 2023-01-15T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:35:07.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:35:07.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:35:07.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:35:07.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:35:07.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:07.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:35:07.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:35:07.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:35:07.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:35:07.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:07.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:35:07.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:35:07.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:07.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:35:07.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:35:07.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:07.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:35:07.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:35:07.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.539 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:07.539 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:35:07.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:35:07.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:35:07.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:35:07.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:35:07.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:35:07.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:07.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:35:07.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:35:07.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:35:07.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:35:07.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.543 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.543 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.543 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:07.554 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:07.554 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 287.518864415, 2023-01-15T14:35:07.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:35:07.555 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.555 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:07.555 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.555 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.555 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.555 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:35:07.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.555 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:07.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:07.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:35:07.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:07.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:35:07.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:35:07.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:35:07.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:35:07.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:35:07.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:07.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:35:07.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:35:07.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:35:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:35:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:35:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:35:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:07.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:35:07.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:35:07.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:07.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:35:07.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:35:07.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:35:07.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:35:07.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:35:07.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:35:07.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:07.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:35:07.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:35:07.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:35:07.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:35:07.563 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.581 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.581 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.581 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 287.51876177399998, 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:35:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:35:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:35:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:35:07.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:35:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:35:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:35:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:35:07.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:35:07.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:07.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:35:07.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:35:07.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:35:07.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:35:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:35:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:35:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:35:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:35:07.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.588 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 242.917779779, 2023-01-15T14:35:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:35:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.589 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:07.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:35:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:35:07.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:35:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:35:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:35:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:35:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:35:07.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:35:07.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.694 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:07.694 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:35:07.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:07.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:35:07.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:35:07.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:35:07.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:35:07.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:35:07.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:35:07.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:35:07.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:35:07.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:35:07.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:35:07.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:07.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:35:07.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:35:07.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:07.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:35:07.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:35:07.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:07.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:35:07.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:35:07.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:07.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:35:07.709 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:35:07.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.710 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:07.710 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:35:07.710 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:35:07.710 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.710 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.712 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.712 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:35:07.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:35:07.713 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.713 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.713 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.713 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:35:07.713 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:35:07.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.714 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.714 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:07.714 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:35:07.714 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:35:07.714 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.715 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.715 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:35:07.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:35:07.715 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.715 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.716 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.716 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:07.716 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:07.716 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 242.917754355, 2023-01-15T14:35:07.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:35:07.717 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.717 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:07.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.717 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.717 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:35:07.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.718 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:07.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:07.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:35:07.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:07.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:35:07.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:35:07.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.720 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.720 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:35:07.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:35:07.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.720 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:35:07.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:35:07.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:35:07.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:35:07.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:07.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:35:07.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:35:07.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.722 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:35:07.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:35:07.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:07.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:35:07.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:35:07.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:35:07.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:35:07.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.727 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.727 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.727 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:35:07.727 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:35:07.727 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.728 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:35:07.728 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:35:07.728 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.728 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.728 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:35:07.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:35:07.729 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.729 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.729 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:07.729 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:35:07.729 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:35:07.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:07.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:35:07.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:35:07.730 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:07.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:35:07.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:35:07.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:07.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:35:07.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:35:07.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:07.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:35:07.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:35:07.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:35:07.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:35:07.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:35:07.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:35:07.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:07.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:35:07.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:35:07.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:35:07.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:35:07.735 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.735 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.735 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.736 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2301 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e216)", 2023-01-15T14:35:07.736 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:07.736 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 203.627879888, 2023-01-15T14:35:07.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61754771200000003, 2023-01-15T14:35:07.736 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.736 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:07.736 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.737 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.737 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.737 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2301 2023-01-15T14:35:07.737 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.737 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.737 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.737 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:07.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.738 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.738 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.738 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:07.738 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.738 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.739 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195530+0000", 2023-01-15T14:35:07.739 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:07.739 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.739 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.739 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.740 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195535+0000", 2023-01-15T14:35:07.740 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3660000000000002e-06 2023-01-15T14:35:07.740 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.740 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.740 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195546+0000", 2023-01-15T14:35:07.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1062e-05 2023-01-15T14:35:07.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195551+0000", 2023-01-15T14:35:07.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2023-01-15T14:35:07.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195999+0000", 2023-01-15T14:35:07.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044799800000000002 2023-01-15T14:35:07.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.196106+0000", 2023-01-15T14:35:07.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106172 2023-01-15T14:35:07.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510631+0000", 2023-01-15T14:35:07.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31452575999999999 2023-01-15T14:35:07.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510875+0000", 2023-01-15T14:35:07.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024415199999999998 2023-01-15T14:35:07.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812445+0000", 2023-01-15T14:35:07.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30156923000000002 2023-01-15T14:35:07.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812727+0000", 2023-01-15T14:35:07.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028227800000000001 2023-01-15T14:35:07.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.813079+0000", 2023-01-15T14:35:07.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199599999999998 2023-01-15T14:35:07.861 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.862 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.862 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.862 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2749 3.3d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:35:07.863 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:07.863 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 155.90199511200001, 2023-01-15T14:35:07.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.69206326600000001, 2023-01-15T14:35:07.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:07.863 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.864 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.864 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.864 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2749 2023-01-15T14:35:07.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.864 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:07.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:07.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:07.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:07.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921418+0000", 2023-01-15T14:35:07.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1670000000000002e-06 2023-01-15T14:35:07.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921426+0000", 2023-01-15T14:35:07.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0569999999999993e-06 2023-01-15T14:35:07.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921429+0000", 2023-01-15T14:35:07.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9629999999999998e-06 2023-01-15T14:35:07.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107302+0000", 2023-01-15T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.185873327 2023-01-15T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107657+0000", 2023-01-15T14:35:07.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035504299999999998 2023-01-15T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212694+0000", 2023-01-15T14:35:07.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105037095 2023-01-15T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212936+0000", 2023-01-15T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241492 2023-01-15T14:35:07.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613125+0000", 2023-01-15T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40018927900000001 2023-01-15T14:35:07.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613294+0000", 2023-01-15T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016838000000000001 2023-01-15T14:35:07.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613479+0000", 2023-01-15T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018577599999999999 2023-01-15T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.876 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3509 3.53 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (58) in=7b,truncate 2084167] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:07.877 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.705351977000007, 2023-01-15T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70658069899999998, 2023-01-15T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.878 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.879 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3509 2023-01-15T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:07.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:07.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118062+0000", 2023-01-15T14:35:07.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2260000000000001e-06 2023-01-15T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118067+0000", 2023-01-15T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3700000000000003e-06 2023-01-15T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118072+0000", 2023-01-15T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4880000000000001e-06 2023-01-15T14:35:07.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218253+0000", 2023-01-15T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10018108100000001 2023-01-15T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218258+0000", 2023-01-15T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.831e-06 2023-01-15T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321058+0000", 2023-01-15T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10280054499999999 2023-01-15T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321223+0000", 2023-01-15T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016474 2023-01-15T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323014+0000", 2023-01-15T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017909550000000001 2023-01-15T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323124+0000", 2023-01-15T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110326 2023-01-15T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323363+0000", 2023-01-15T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238385 2023-01-15T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323663+0000", 2023-01-15T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029987699999999999 2023-01-15T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:07.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323677+0000", 2023-01-15T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4346e-05 2023-01-15T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323729+0000", 2023-01-15T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2049999999999998e-05 2023-01-15T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323775+0000", 2023-01-15T14:35:07.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5872999999999999e-05 2023-01-15T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.616791+0000", 2023-01-15T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29301572199999998 2023-01-15T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824267+0000", 2023-01-15T14:35:07.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20747585199999999 2023-01-15T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824425+0000", 2023-01-15T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015861900000000001 2023-01-15T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824640+0000", 2023-01-15T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021475899999999999 2023-01-15T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.897 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3510 3.53 3:cbfd2d71:::smithi09025264-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.705274137999993, 2023-01-15T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.80704639, 2023-01-15T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.898 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3510 2023-01-15T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118135+0000", 2023-01-15T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:07.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118143+0000", 2023-01-15T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0579999999999998e-06 2023-01-15T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118162+0000", 2023-01-15T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8702000000000002e-05 2023-01-15T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118178+0000", 2023-01-15T14:35:07.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5645e-05 2023-01-15T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321268+0000", 2023-01-15T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20308996000000001 2023-01-15T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321370+0000", 2023-01-15T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010268399999999999 2023-01-15T14:35:07.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323825+0000", 2023-01-15T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024546450000000001 2023-01-15T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323934+0000", 2023-01-15T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010940199999999999 2023-01-15T14:35:07.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.924815+0000", 2023-01-15T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60088053500000005 2023-01-15T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925009+0000", 2023-01-15T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019363599999999999 2023-01-15T14:35:07.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925183+0000", 2023-01-15T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000174823 2023-01-15T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3505 3.62 3:47dfe935:::smithi09025264-29:head [rollback db] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:07.905 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.807671428000006, 2023-01-15T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95152285400000003, 2023-01-15T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3505 2023-01-15T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.906 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:07.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:07.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:08.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015739+0000", 2023-01-15T14:35:08.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:08.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:08.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015742+0000", 2023-01-15T14:35:08.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.869e-06 2023-01-15T14:35:08.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:08.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015748+0000", 2023-01-15T14:35:08.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0360000000000003e-06 2023-01-15T14:35:08.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:08.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015754+0000", 2023-01-15T14:35:08.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.733e-06 2023-01-15T14:35:08.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:08.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318179+0000", 2023-01-15T14:35:08.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30242492300000001 2023-01-15T14:35:08.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:08.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318502+0000", 2023-01-15T14:35:08.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032355699999999998 2023-01-15T14:35:08.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:08.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365019+0000", 2023-01-15T14:35:08.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046516950000000001 2023-01-15T14:35:08.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:08.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862003+0000", 2023-01-15T14:35:08.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49698320800000001 2023-01-15T14:35:08.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:08.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862314+0000", 2023-01-15T14:35:08.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000311236 2023-01-15T14:35:08.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:08.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863083+0000", 2023-01-15T14:35:08.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076928399999999996 2023-01-15T14:35:08.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:08.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863709+0000", 2023-01-15T14:35:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062558700000000004 2023-01-15T14:35:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863724+0000", 2023-01-15T14:35:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5288999999999999e-05 2023-01-15T14:35:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863799+0000", 2023-01-15T14:35:08.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4552e-05 2023-01-15T14:35:08.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:08.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863851+0000", 2023-01-15T14:35:08.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2018999999999998e-05 2023-01-15T14:35:08.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:08.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966087+0000", 2023-01-15T14:35:08.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102235952 2023-01-15T14:35:08.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:08.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966811+0000", 2023-01-15T14:35:08.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072484800000000005 2023-01-15T14:35:08.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:08.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966950+0000", 2023-01-15T14:35:08.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013907399999999999 2023-01-15T14:35:08.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:08.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.967263+0000", 2023-01-15T14:35:08.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312107 2023-01-15T14:35:08.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:08.029 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:08.029 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:08.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.030 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3506 3.62 3:47dfe935:::smithi09025264-29:head [append 0~0] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:08.030 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:08.030 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 76.807651605999993, 2023-01-15T14:35:08.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.234729387, 2023-01-15T14:35:08.030 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:08.030 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:08.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:08.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:08.031 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:08.031 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3506 2023-01-15T14:35:08.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.031 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:08.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:08.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:08.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:08.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:08.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:08.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:08.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:08.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015759+0000", 2023-01-15T14:35:08.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:08.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:08.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015763+0000", 2023-01-15T14:35:08.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.517e-06 2023-01-15T14:35:08.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:08.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015768+0000", 2023-01-15T14:35:08.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5620000000000003e-06 2023-01-15T14:35:08.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:08.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015773+0000", 2023-01-15T14:35:08.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3730000000000003e-06 2023-01-15T14:35:08.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:08.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365160+0000", 2023-01-15T14:35:08.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34938724300000001 2023-01-15T14:35:08.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:08.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365370+0000", 2023-01-15T14:35:08.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021001599999999999 2023-01-15T14:35:08.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:08.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863926+0000", 2023-01-15T14:35:08.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49855574600000002 2023-01-15T14:35:08.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:08.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.864284+0000", 2023-01-15T14:35:08.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035790600000000001 2023-01-15T14:35:08.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:08.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865099+0000", 2023-01-15T14:35:08.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081499999999999997 2023-01-15T14:35:08.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:08.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865640+0000", 2023-01-15T14:35:08.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054118899999999997 2023-01-15T14:35:08.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:08.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865653+0000", 2023-01-15T14:35:08.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3614e-05 2023-01-15T14:35:08.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:08.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865699+0000", 2023-01-15T14:35:08.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5751000000000003e-05 2023-01-15T14:35:08.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:08.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966206+0000", 2023-01-15T14:35:08.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100506741 2023-01-15T14:35:08.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:08.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966665+0000", 2023-01-15T14:35:08.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000459345 2023-01-15T14:35:08.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:08.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.249842+0000", 2023-01-15T14:35:08.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28317711200000001 2023-01-15T14:35:08.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:08.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250054+0000", 2023-01-15T14:35:08.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021161899999999999 2023-01-15T14:35:08.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:08.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250489+0000", 2023-01-15T14:35:08.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043499700000000001 2023-01-15T14:35:08.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:08.047 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:08.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:08.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.047 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3572 3.65 3:a75958b1:::smithi09025264-6:head [setxattr _header (57) in=7b,truncate 2235430] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:08.048 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:08.048 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.801021781000003, 2023-01-15T14:35:08.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65482807700000001, 2023-01-15T14:35:08.048 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:08.048 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:08.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:08.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:08.049 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:08.049 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3572 2023-01-15T14:35:08.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.049 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:08.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:08.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:08.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:08.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:08.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:08.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:08.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:08.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022388+0000", 2023-01-15T14:35:08.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:08.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:08.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022397+0000", 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.704e-06 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022412+0000", 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5554e-05 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022422+0000", 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2839999999999995e-06 2023-01-15T14:35:08.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:08.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114230+0000", 2023-01-15T14:35:08.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.091808129000000002 2023-01-15T14:35:08.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:08.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114234+0000", 2023-01-15T14:35:08.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4329999999999997e-06 2023-01-15T14:35:08.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:08.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204200+0000", 2023-01-15T14:35:08.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089966321000000002 2023-01-15T14:35:08.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:08.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204439+0000", 2023-01-15T14:35:08.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238595 2023-01-15T14:35:08.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:08.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.305883+0000", 2023-01-15T14:35:08.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101443713 2023-01-15T14:35:08.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:08.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306066+0000", 2023-01-15T14:35:08.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182962 2023-01-15T14:35:08.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2023-01-15T14:35:08.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306621+0000", 2023-01-15T14:35:08.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055552800000000001 2023-01-15T14:35:08.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:08.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307013+0000", 2023-01-15T14:35:08.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039210599999999998 2023-01-15T14:35:08.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:08.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307033+0000", 2023-01-15T14:35:08.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9423e-05 2023-01-15T14:35:08.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:08.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307093+0000", 2023-01-15T14:35:08.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0402000000000002e-05 2023-01-15T14:35:08.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:08.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307445+0000", 2023-01-15T14:35:08.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000351677 2023-01-15T14:35:08.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:08.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307662+0000", 2023-01-15T14:35:08.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021765099999999999 2023-01-15T14:35:08.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:08.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676523+0000", 2023-01-15T14:35:08.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36886056699999997 2023-01-15T14:35:08.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:08.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676723+0000", 2023-01-15T14:35:08.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019947899999999999 2023-01-15T14:35:08.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:08.065 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677217+0000", 2023-01-15T14:35:08.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049493900000000001 2023-01-15T14:35:08.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:08.065 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:08.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:08.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.066 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3573 3.65 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:08.066 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:08.066 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 69.800978654000005, 2023-01-15T14:35:08.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65551266799999997, 2023-01-15T14:35:08.066 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:08.066 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:08.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:08.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:08.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:08.066 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3573 2023-01-15T14:35:08.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.067 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:08.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:08.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:08.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:08.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:08.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:08.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:08.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022432+0000", 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022436+0000", 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.693e-06 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:08.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022447+0000", 2023-01-15T14:35:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1639e-05 2023-01-15T14:35:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122543+0000", 2023-01-15T14:35:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100095369 2023-01-15T14:35:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:08.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122863+0000", 2023-01-15T14:35:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032020799999999999 2023-01-15T14:35:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122869+0000", 2023-01-15T14:35:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7640000000000002e-06 2023-01-15T14:35:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:08.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206080+0000", 2023-01-15T14:35:08.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.083211794000000006 2023-01-15T14:35:08.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:08.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206289+0000", 2023-01-15T14:35:08.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020866600000000001 2023-01-15T14:35:08.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:08.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307193+0000", 2023-01-15T14:35:08.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10090391999999999 2023-01-15T14:35:08.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:08.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307374+0000", 2023-01-15T14:35:08.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018061299999999999 2023-01-15T14:35:08.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:08.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677342+0000", 2023-01-15T14:35:08.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.369968034 2023-01-15T14:35:08.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:08.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677535+0000", 2023-01-15T14:35:08.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019311299999999999 2023-01-15T14:35:08.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:08.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:08.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:08.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677945+0000", 2023-01-15T14:35:08.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041040299999999999 2023-01-15T14:35:08.596 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:08.596 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:08.596 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:08.596 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:08.596 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:08.596 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:08.597 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:08.598 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:08.599 INFO:tasks.rados.rados.0.smithi090.stdout:1563: done (5 left) 2023-01-15T14:35:08.599 INFO:tasks.rados.rados.0.smithi090.stdout:1564: copy_from oid 37 from oid 46 current snap is 175 2023-01-15T14:35:08.599 INFO:tasks.rados.rados.0.smithi090.stdout:1558: expect (ObjNum 400 snap 106 seq_num 400) 2023-01-15T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:08.599 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:08.600 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:08.600 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1, 6] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:35:08.789 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:35:08.789 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-01-15T14:35:08.789 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.000000 2023-01-15T14:35:08.789 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0 2023-01-15T14:35:08.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:35:08.791 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:35:08.791 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:35:08.791 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:35:08.791 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:35:08.791 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:35:08.792 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:35:08.792 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:35:08.792 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:35:08.795 INFO:tasks.rados.rados.0.smithi090.stdout:1558: done (5 left) 2023-01-15T14:35:08.795 INFO:tasks.rados.rados.0.smithi090.stdout:1565: read oid 25 snap -1 2023-01-15T14:35:08.795 INFO:tasks.rados.rados.0.smithi090.stdout:1565: expect (ObjNum 487 snap 142 seq_num 487) 2023-01-15T14:35:08.795 INFO:tasks.rados.rados.0.smithi090.stdout:1566: delete oid 23 current snap is 175 2023-01-15T14:35:08.795 INFO:tasks.rados.rados.0.smithi090.stdout:1560: finishing copy_from to smithi09025264-14 2023-01-15T14:35:08.795 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 108 (ObjNum 527 snap 158 seq_num 527) dirty exists 2023-01-15T14:35:08.795 INFO:tasks.rados.rados.0.smithi090.stdout:1560: finishing copy_from racing read to smithi09025264-14 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout:1561: finishing copy_from to smithi09025264-32 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 448 (ObjNum 548 snap 172 seq_num 548) dirty exists 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout:1561: finishing copy_from racing read to smithi09025264-32 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout:1559: finishing write tid 1 to smithi09025264-17 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout:1560: done (6 left) 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout:1561: done (5 left) 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout:1562: done (4 left) 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout:1566: done (3 left) 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout:1567: snap_create 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout:1565: expect (ObjNum 487 snap 142 seq_num 487) 2023-01-15T14:35:08.796 INFO:tasks.rados.rados.0.smithi090.stdout:1559: finishing write tid 2 to smithi09025264-17 2023-01-15T14:35:08.797 INFO:tasks.rados.rados.0.smithi090.stdout:1559: finishing write tid 3 to smithi09025264-17 2023-01-15T14:35:08.797 INFO:tasks.rados.rados.0.smithi090.stdout:1559: finishing write tid 4 to smithi09025264-17 2023-01-15T14:35:08.797 INFO:tasks.rados.rados.0.smithi090.stdout:1559: finishing write tid 5 to smithi09025264-17 2023-01-15T14:35:08.797 INFO:tasks.rados.rados.0.smithi090.stdout:1559: oid 17 updating version 0 to 458 2023-01-15T14:35:08.797 INFO:tasks.rados.rados.0.smithi090.stdout:1559: oid 17 updating version 458 to 459 2023-01-15T14:35:08.797 INFO:tasks.rados.rados.0.smithi090.stdout:1559: oid 17 updating version 459 to 460 2023-01-15T14:35:08.797 INFO:tasks.rados.rados.0.smithi090.stdout:1559: oid 17 version 460 is already newer than 457 2023-01-15T14:35:08.797 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 460 (ObjNum 553 snap 175 seq_num 553) dirty exists 2023-01-15T14:35:08.797 INFO:tasks.rados.rados.0.smithi090.stdout:1559: left oid 17 (ObjNum 553 snap 175 seq_num 553) 2023-01-15T14:35:08.798 INFO:tasks.rados.rados.0.smithi090.stdout:1559: done (2 left) 2023-01-15T14:35:08.798 INFO:tasks.rados.rados.0.smithi090.stdout:1565: done (1 left) 2023-01-15T14:35:08.798 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:08.798 INFO:tasks.rados.rados.0.smithi090.stdout:1564: finishing copy_from to smithi09025264-37 2023-01-15T14:35:08.798 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 425 (ObjNum 525 snap 158 seq_num 525) dirty exists 2023-01-15T14:35:08.798 INFO:tasks.rados.rados.0.smithi090.stdout:1564: finishing copy_from racing read to smithi09025264-37 2023-01-15T14:35:08.798 INFO:tasks.rados.rados.0.smithi090.stdout:1564: done (0 left) 2023-01-15T14:35:08.798 INFO:tasks.rados.rados.0.smithi090.stdout:1567: done (0 left) 2023-01-15T14:35:08.798 INFO:tasks.rados.rados.0.smithi090.stdout:1568: read oid 7 snap 167 2023-01-15T14:35:08.798 INFO:tasks.rados.rados.0.smithi090.stdout:1568: expect (ObjNum 530 snap 159 seq_num 530) 2023-01-15T14:35:08.798 INFO:tasks.rados.rados.0.smithi090.stdout:1569: rollback oid 1 current snap is 176 2023-01-15T14:35:08.799 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 1 to 165 2023-01-15T14:35:08.799 INFO:tasks.rados.rados.0.smithi090.stdout:1570: read oid 23 snap -1 2023-01-15T14:35:08.799 INFO:tasks.rados.rados.0.smithi090.stdout:1570: expect deleted 2023-01-15T14:35:08.799 INFO:tasks.rados.rados.0.smithi090.stdout:1571: copy_from oid 45 from oid 43 current snap is 176 2023-01-15T14:35:08.799 INFO:tasks.rados.rados.0.smithi090.stdout:1572: read oid 42 snap -1 2023-01-15T14:35:08.799 INFO:tasks.rados.rados.0.smithi090.stdout:1572: expect deleted 2023-01-15T14:35:08.799 INFO:tasks.rados.rados.0.smithi090.stdout:1573: delete oid 3 current snap is 176 2023-01-15T14:35:08.799 INFO:tasks.rados.rados.0.smithi090.stdout:1570: done (5 left) 2023-01-15T14:35:08.799 INFO:tasks.rados.rados.0.smithi090.stdout:1572: done (4 left) 2023-01-15T14:35:08.799 INFO:tasks.rados.rados.0.smithi090.stdout:1573: done (3 left) 2023-01-15T14:35:08.799 INFO:tasks.rados.rados.0.smithi090.stdout:1574: snap_remove snap 119 2023-01-15T14:35:08.800 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:08.594+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:08.839 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:08.830+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:08.954 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:08.946+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:08.958 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:08.959 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:08.969 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:35:09.233 INFO:tasks.rados.rados.0.smithi090.stdout:1574: done (3 left) 2023-01-15T14:35:09.233 INFO:tasks.rados.rados.0.smithi090.stdout:1575: write oid 50 current snap is 176 2023-01-15T14:35:09.233 INFO:tasks.rados.rados.0.smithi090.stdout:1569: finishing rollback tid 0 to smithi09025264-1 2023-01-15T14:35:09.233 INFO:tasks.rados.rados.0.smithi090.stdout:1575: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2023-01-15T14:35:09.233 INFO:tasks.rados.rados.0.smithi090.stdout:1575: writing smithi09025264-50 from 544692 to 1276905 tid 1 2023-01-15T14:35:09.234 INFO:tasks.rados.rados.0.smithi090.stdout:1575: writing smithi09025264-50 from 1985753 to 2729616 tid 2 2023-01-15T14:35:09.234 INFO:tasks.rados.rados.0.smithi090.stdout:1575: writing smithi09025264-50 from 3335184 to 3344692 tid 3 2023-01-15T14:35:09.234 INFO:tasks.rados.rados.0.smithi090.stdout:1576: read oid 10 snap -1 2023-01-15T14:35:09.234 INFO:tasks.rados.rados.0.smithi090.stdout:1576: expect (ObjNum 518 snap 155 seq_num 518) 2023-01-15T14:35:09.234 INFO:tasks.rados.rados.0.smithi090.stdout:1577: copy_from oid 20 from oid 38 current snap is 176 2023-01-15T14:35:09.234 INFO:tasks.rados.rados.0.smithi090.stdout:1578: read oid 9 snap -1 2023-01-15T14:35:09.235 INFO:tasks.rados.rados.0.smithi090.stdout:1578: expect deleted 2023-01-15T14:35:09.235 INFO:tasks.rados.rados.0.smithi090.stdout:1579: read oid 34 snap -1 2023-01-15T14:35:09.235 INFO:tasks.rados.rados.0.smithi090.stdout:1579: expect deleted 2023-01-15T14:35:09.235 INFO:tasks.rados.rados.0.smithi090.stdout:1580: copy_from oid 37 from oid 33 current snap is 176 2023-01-15T14:35:09.235 INFO:tasks.rados.rados.0.smithi090.stdout:1568: expect (ObjNum 530 snap 159 seq_num 530) 2023-01-15T14:35:09.235 INFO:tasks.rados.rados.0.smithi090.stdout:1571: finishing copy_from to smithi09025264-45 2023-01-15T14:35:09.235 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 108 (ObjNum 527 snap 158 seq_num 527) dirty exists 2023-01-15T14:35:09.236 INFO:tasks.rados.rados.0.smithi090.stdout:1571: finishing copy_from racing read to smithi09025264-45 2023-01-15T14:35:09.236 INFO:tasks.rados.rados.0.smithi090.stdout:1568: done (8 left) 2023-01-15T14:35:09.236 INFO:tasks.rados.rados.0.smithi090.stdout:1571: done (7 left) 2023-01-15T14:35:09.404 INFO:tasks.rados.rados.0.smithi090.stdout:1581: delete oid 32 current snap is 176 2023-01-15T14:35:09.404 INFO:tasks.rados.rados.0.smithi090.stdout:1577: finishing copy_from to smithi09025264-20 2023-01-15T14:35:09.404 INFO:tasks.rados.rados.0.smithi090.stdout:1577: got expected ENOENT (src dne) 2023-01-15T14:35:09.404 INFO:tasks.rados.rados.0.smithi090.stdout:1577: finishing copy_from racing read to smithi09025264-20 2023-01-15T14:35:09.405 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:09.225+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:09.405 INFO:tasks.rados.rados.0.smithi090.stdout:1576: expect (ObjNum 518 snap 155 seq_num 518) 2023-01-15T14:35:09.406 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:09.401+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:09.485 INFO:tasks.rados.rados.0.smithi090.stdout:1575: finishing write tid 1 to smithi09025264-50 2023-01-15T14:35:09.616 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:09.616 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:09.616 INFO:tasks.rados.rados.0.smithi090.stdout:1580: finishing copy_from to smithi09025264-37 2023-01-15T14:35:09.617 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 203 (ObjNum 378 snap 102 seq_num 378) dirty exists 2023-01-15T14:35:09.617 INFO:tasks.rados.rados.0.smithi090.stdout:1580: finishing copy_from racing read to smithi09025264-37 2023-01-15T14:35:09.618 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:35:09.619 INFO:tasks.rados.rados.0.smithi090.stdout:1575: finishing write tid 2 to smithi09025264-50 2023-01-15T14:35:09.824 INFO:tasks.rados.rados.0.smithi090.stdout:1575: finishing write tid 3 to smithi09025264-50 2023-01-15T14:35:09.824 INFO:tasks.rados.rados.0.smithi090.stdout:1575: finishing write tid 4 to smithi09025264-50 2023-01-15T14:35:09.824 INFO:tasks.rados.rados.0.smithi090.stdout:1575: finishing write tid 5 to smithi09025264-50 2023-01-15T14:35:09.824 INFO:tasks.rados.rados.0.smithi090.stdout:1575: oid 50 updating version 0 to 278 2023-01-15T14:35:09.824 INFO:tasks.rados.rados.0.smithi090.stdout:1575: oid 50 version 278 is already newer than 275 2023-01-15T14:35:09.824 INFO:tasks.rados.rados.0.smithi090.stdout:1575: oid 50 version 278 is already newer than 276 2023-01-15T14:35:09.824 INFO:tasks.rados.rados.0.smithi090.stdout:1575: oid 50 version 278 is already newer than 277 2023-01-15T14:35:09.825 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 50 v 278 (ObjNum 556 snap 176 seq_num 556) dirty exists 2023-01-15T14:35:09.825 INFO:tasks.rados.rados.0.smithi090.stdout:1575: left oid 50 (ObjNum 556 snap 176 seq_num 556) 2023-01-15T14:35:09.825 INFO:tasks.rados.rados.0.smithi090.stdout:1575: done (7 left) 2023-01-15T14:35:09.825 INFO:tasks.rados.rados.0.smithi090.stdout:1576: done (6 left) 2023-01-15T14:35:09.827 INFO:tasks.rados.rados.0.smithi090.stdout:1577: done (5 left) 2023-01-15T14:35:09.827 INFO:tasks.rados.rados.0.smithi090.stdout:1578: done (4 left) 2023-01-15T14:35:09.827 INFO:tasks.rados.rados.0.smithi090.stdout:1579: done (3 left) 2023-01-15T14:35:09.827 INFO:tasks.rados.rados.0.smithi090.stdout:1580: done (2 left) 2023-01-15T14:35:09.827 INFO:tasks.rados.rados.0.smithi090.stdout:1581: done (1 left) 2023-01-15T14:35:09.828 INFO:tasks.rados.rados.0.smithi090.stdout:1582: snap_remove snap 167 2023-01-15T14:35:09.828 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:09.610+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:09.830 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:09.825+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:09.931 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:09.925+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:10.016 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:10.017 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:10.017 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:35:10.032 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:10.025+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:10.128 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:10.221 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:10.222 INFO:tasks.rados.rados.0.smithi090.stdout:1582: done (1 left) 2023-01-15T14:35:10.222 INFO:tasks.rados.rados.0.smithi090.stdout:1583: write oid 6 current snap is 176 2023-01-15T14:35:10.223 INFO:tasks.rados.rados.0.smithi090.stdout:1583: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2023-01-15T14:35:10.223 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:35:10.223 INFO:tasks.rados.rados.0.smithi090.stdout:1583: writing smithi09025264-6 from 578306 to 1061017 tid 1 2023-01-15T14:35:10.223 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.3 primary-affinity to 0 (802) 2023-01-15T14:35:10.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:35:10.225 INFO:tasks.rados.rados.0.smithi090.stdout:1583: writing smithi09025264-6 from 1586364 to 2217543 tid 2 2023-01-15T14:35:10.227 INFO:tasks.rados.rados.0.smithi090.stdout:1583: writing smithi09025264-6 from 2757324 to 3242166 tid 3 2023-01-15T14:35:10.228 INFO:tasks.rados.rados.0.smithi090.stdout:1584: rollback oid 18 current snap is 176 2023-01-15T14:35:10.228 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 18 to 124 2023-01-15T14:35:10.228 INFO:tasks.rados.rados.0.smithi090.stdout:1585: write (excl) oid 36 current snap is 176 2023-01-15T14:35:10.228 INFO:tasks.rados.rados.0.smithi090.stdout:1585: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2023-01-15T14:35:10.228 INFO:tasks.rados.rados.0.smithi090.stdout:1585: writing smithi09025264-36 from 595113 to 1153073 tid 1 2023-01-15T14:35:10.228 INFO:tasks.rados.rados.0.smithi090.stdout:1585: writing smithi09025264-36 from 1644846 to 2136036 tid 2 2023-01-15T14:35:10.228 INFO:tasks.rados.rados.0.smithi090.stdout:1585: writing smithi09025264-36 from 2784747 to 3395113 tid 3 2023-01-15T14:35:10.228 INFO:tasks.rados.rados.0.smithi090.stdout:1586: read oid 28 snap -1 2023-01-15T14:35:10.228 INFO:tasks.rados.rados.0.smithi090.stdout:1586: expect (ObjNum 466 snap 134 seq_num 466) 2023-01-15T14:35:10.228 INFO:tasks.rados.rados.0.smithi090.stdout:1587: delete oid 38 current snap is 176 2023-01-15T14:35:10.229 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:10.217+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:10.273 INFO:tasks.rados.rados.0.smithi090.stdout:1586: expect (ObjNum 466 snap 134 seq_num 466) 2023-01-15T14:35:10.432 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:10.432 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:10.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:35:10.433 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:10.425+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:10.533 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:10.526+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:10.599 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:10.600 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:35:10.601 INFO:tasks.rados.rados.0.smithi090.stdout:1584: finishing rollback tid 0 to smithi09025264-18 2023-01-15T14:35:10.601 INFO:tasks.rados.rados.0.smithi090.stdout:1586: done (5 left) 2023-01-15T14:35:10.601 INFO:tasks.rados.rados.0.smithi090.stdout:1587: done (4 left) 2023-01-15T14:35:10.601 INFO:tasks.rados.rados.0.smithi090.stdout:1588: rollback oid 13 current snap is 176 2023-01-15T14:35:10.602 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 13 to 160 2023-01-15T14:35:10.602 INFO:tasks.rados.rados.0.smithi090.stdout:1589: write oid 37 current snap is 176 2023-01-15T14:35:10.602 INFO:tasks.rados.rados.0.smithi090.stdout:1589: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2023-01-15T14:35:10.602 INFO:tasks.rados.rados.0.smithi090.stdout:1589: writing smithi09025264-37 from 628727 to 1337185 tid 1 2023-01-15T14:35:10.607 INFO:tasks.rados.rados.0.smithi090.stdout:1589: writing smithi09025264-37 from 2045457 to 2540316 tid 2 2023-01-15T14:35:10.613 INFO:tasks.rados.rados.0.smithi090.stdout:1589: writing smithi09025264-37 from 3123240 to 3428727 tid 3 2023-01-15T14:35:10.613 INFO:tasks.rados.rados.0.smithi090.stdout:1590: write oid 47 current snap is 176 2023-01-15T14:35:10.613 INFO:tasks.rados.rados.0.smithi090.stdout:1590: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2023-01-15T14:35:10.620 INFO:tasks.rados.rados.0.smithi090.stdout:1590: writing smithi09025264-47 from 645534 to 1429241 tid 1 2023-01-15T14:35:10.754 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:10.756 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:10.757 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:35:10.757 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:10.749+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:10.758 INFO:tasks.rados.rados.0.smithi090.stdout:1590: writing smithi09025264-47 from 2103939 to 2742456 tid 2 2023-01-15T14:35:10.766 INFO:tasks.rados.rados.0.smithi090.stdout:1590: writing smithi09025264-47 from 3150663 to 3445534 tid 3 2023-01-15T14:35:10.767 INFO:tasks.rados.rados.0.smithi090.stdout:1591: read oid 40 snap -1 2023-01-15T14:35:10.767 INFO:tasks.rados.rados.0.smithi090.stdout:1591: expect deleted 2023-01-15T14:35:10.767 INFO:tasks.rados.rados.0.smithi090.stdout:1592: write (excl) oid 45 current snap is 176 2023-01-15T14:35:10.767 INFO:tasks.rados.rados.0.smithi090.stdout:1592: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2023-01-15T14:35:10.767 INFO:tasks.rados.rados.0.smithi090.stdout:1592: writing smithi09025264-45 from 662341 to 1237650 tid 1 2023-01-15T14:35:10.767 INFO:tasks.rados.rados.0.smithi090.stdout:1592: writing smithi09025264-45 from 1878774 to 2377302 tid 2 2023-01-15T14:35:10.767 INFO:tasks.rados.rados.0.smithi090.stdout:1592: writing smithi09025264-45 from 2894439 to 3351994 tid 3 2023-01-15T14:35:10.767 INFO:tasks.rados.rados.0.smithi090.stdout:1593: delete oid 27 current snap is 176 2023-01-15T14:35:10.767 INFO:tasks.rados.rados.0.smithi090.stdout:1583: finishing write tid 1 to smithi09025264-6 2023-01-15T14:35:10.767 INFO:tasks.rados.rados.0.smithi090.stdout:1583: finishing write tid 2 to smithi09025264-6 2023-01-15T14:35:10.767 INFO:tasks.rados.rados.0.smithi090.stdout:1583: finishing write tid 3 to smithi09025264-6 2023-01-15T14:35:10.776 INFO:tasks.rados.rados.0.smithi090.stdout:1588: finishing rollback tid 1 to smithi09025264-13 2023-01-15T14:35:10.776 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 348 (ObjNum 273 snap 0 seq_num 0) dirty dne 2023-01-15T14:35:10.777 INFO:tasks.rados.rados.0.smithi090.stdout:1592: finishing write tid 1 to smithi09025264-45 2023-01-15T14:35:10.788 INFO:tasks.rados.rados.0.smithi090.stdout:1585: finishing write tid 1 to smithi09025264-36 2023-01-15T14:35:10.912 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:35:10.914 INFO:tasks.rados.rados.0.smithi090.stdout:1589: finishing write tid 1 to smithi09025264-37 2023-01-15T14:35:10.914 INFO:tasks.rados.rados.0.smithi090.stdout:1589: finishing write tid 2 to smithi09025264-37 2023-01-15T14:35:10.914 INFO:tasks.rados.rados.0.smithi090.stdout:1583: finishing write tid 4 to smithi09025264-6 2023-01-15T14:35:10.914 INFO:tasks.rados.rados.0.smithi090.stdout:1583: finishing write tid 5 to smithi09025264-6 2023-01-15T14:35:10.914 INFO:tasks.rados.rados.0.smithi090.stdout:1583: oid 6 updating version 0 to 350 2023-01-15T14:35:10.914 INFO:tasks.rados.rados.0.smithi090.stdout:1583: oid 6 version 350 is already newer than 348 2023-01-15T14:35:10.914 INFO:tasks.rados.rados.0.smithi090.stdout:1583: oid 6 version 350 is already newer than 347 2023-01-15T14:35:10.915 INFO:tasks.rados.rados.0.smithi090.stdout:1583: oid 6 version 350 is already newer than 349 2023-01-15T14:35:10.915 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 350 (ObjNum 558 snap 176 seq_num 558) dirty exists 2023-01-15T14:35:10.915 INFO:tasks.rados.rados.0.smithi090.stdout:1583: left oid 6 (ObjNum 558 snap 176 seq_num 558) 2023-01-15T14:35:10.915 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:10.915 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:10.915 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:35:10.917 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:10.906+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:10.982 INFO:tasks.rados.rados.0.smithi090.stdout:1592: finishing write tid 2 to smithi09025264-45 2023-01-15T14:35:11.091 INFO:tasks.rados.rados.0.smithi090.stdout:1585: finishing write tid 2 to smithi09025264-36 2023-01-15T14:35:11.091 INFO:tasks.rados.rados.0.smithi090.stdout:1584: finishing rollback tid 1 to smithi09025264-18 2023-01-15T14:35:11.091 INFO:tasks.rados.rados.0.smithi090.stdout:1589: finishing write tid 3 to smithi09025264-37 2023-01-15T14:35:11.091 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:11.092 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:11.092 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:11.094 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:35:11.094 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:11.085+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:11.095 INFO:tasks.rados.rados.0.smithi090.stdout:1592: finishing write tid 3 to smithi09025264-45 2023-01-15T14:35:11.095 INFO:tasks.rados.rados.0.smithi090.stdout:1592: finishing write tid 4 to smithi09025264-45 2023-01-15T14:35:11.095 INFO:tasks.rados.rados.0.smithi090.stdout:1592: finishing write tid 5 to smithi09025264-45 2023-01-15T14:35:11.095 INFO:tasks.rados.rados.0.smithi090.stdout:1592: oid 45 updating version 0 to 156 2023-01-15T14:35:11.095 INFO:tasks.rados.rados.0.smithi090.stdout:1592: oid 45 version 156 is already newer than 155 2023-01-15T14:35:11.095 INFO:tasks.rados.rados.0.smithi090.stdout:1592: oid 45 version 156 is already newer than 154 2023-01-15T14:35:11.095 INFO:tasks.rados.rados.0.smithi090.stdout:1592: oid 45 version 156 is already newer than 153 2023-01-15T14:35:11.095 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 156 (ObjNum 563 snap 176 seq_num 563) dirty exists 2023-01-15T14:35:11.095 INFO:tasks.rados.rados.0.smithi090.stdout:1592: left oid 45 (ObjNum 563 snap 176 seq_num 563) 2023-01-15T14:35:11.099 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:11.136 INFO:tasks.rados.rados.0.smithi090.stdout:1584: finishing rollback tid 2 to smithi09025264-18 2023-01-15T14:35:11.136 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 277 (ObjNum 447 snap 123 seq_num 447) dirty exists 2023-01-15T14:35:11.136 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:11.142 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:11.255 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:11.264 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:11.265 INFO:tasks.rados.rados.0.smithi090.stdout:1585: finishing write tid 3 to smithi09025264-36 2023-01-15T14:35:11.265 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:35:11.265 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:35:11.266 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:35:11.266 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:35:11.266 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:35:11.289 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:11.249+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:11.289 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:11.289 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:11.290 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:35:11.290 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:35:11.291 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:35:11.291 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:35:11.291 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:35:11.291 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:35:11.291 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:35:11.291 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:35:11.292 INFO:tasks.rados.rados.0.smithi090.stdout:1589: finishing write tid 4 to smithi09025264-37 2023-01-15T14:35:11.294 INFO:tasks.rados.rados.0.smithi090.stdout:1585: finishing write tid 4 to smithi09025264-36 2023-01-15T14:35:11.412 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:35:11.413 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:35:11.425 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:35:11.425 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:35:11.425 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:35:11.426 INFO:tasks.rados.rados.0.smithi090.stdout:1589: finishing write tid 5 to smithi09025264-37 2023-01-15T14:35:11.426 INFO:tasks.rados.rados.0.smithi090.stdout:1589: oid 37 updating version 0 to 461 2023-01-15T14:35:11.426 INFO:tasks.rados.rados.0.smithi090.stdout:1589: oid 37 version 461 is already newer than 460 2023-01-15T14:35:11.426 INFO:tasks.rados.rados.0.smithi090.stdout:1589: oid 37 version 461 is already newer than 459 2023-01-15T14:35:11.426 INFO:tasks.rados.rados.0.smithi090.stdout:1589: oid 37 updating version 461 to 462 2023-01-15T14:35:11.426 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 462 (ObjNum 561 snap 176 seq_num 561) dirty exists 2023-01-15T14:35:11.426 INFO:tasks.rados.rados.0.smithi090.stdout:1589: left oid 37 (ObjNum 561 snap 176 seq_num 561) 2023-01-15T14:35:11.426 INFO:tasks.rados.rados.0.smithi090.stdout:1590: finishing write tid 1 to smithi09025264-47 2023-01-15T14:35:11.426 INFO:tasks.rados.rados.0.smithi090.stdout:1590: finishing write tid 2 to smithi09025264-47 2023-01-15T14:35:11.426 INFO:tasks.rados.rados.0.smithi090.stdout:1585: finishing write tid 5 to smithi09025264-36 2023-01-15T14:35:11.427 INFO:tasks.rados.rados.0.smithi090.stdout:1585: oid 36 updating version 0 to 264 2023-01-15T14:35:11.427 INFO:tasks.rados.rados.0.smithi090.stdout:1585: oid 36 version 264 is already newer than 261 2023-01-15T14:35:11.427 INFO:tasks.rados.rados.0.smithi090.stdout:1585: oid 36 version 264 is already newer than 263 2023-01-15T14:35:11.427 INFO:tasks.rados.rados.0.smithi090.stdout:1585: oid 36 version 264 is already newer than 262 2023-01-15T14:35:11.427 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 264 (ObjNum 559 snap 176 seq_num 559) dirty exists 2023-01-15T14:35:11.427 INFO:tasks.rados.rados.0.smithi090.stdout:1585: left oid 36 (ObjNum 559 snap 176 seq_num 559) 2023-01-15T14:35:11.427 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:11.406+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:11.428 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:35:11.445 INFO:tasks.rados.rados.0.smithi090.stdout:1590: finishing write tid 3 to smithi09025264-47 2023-01-15T14:35:11.458 INFO:tasks.rados.rados.0.smithi090.stdout:1590: finishing write tid 4 to smithi09025264-47 2023-01-15T14:35:11.459 INFO:tasks.rados.rados.0.smithi090.stdout:1590: finishing write tid 5 to smithi09025264-47 2023-01-15T14:35:11.459 INFO:tasks.rados.rados.0.smithi090.stdout:1590: oid 47 updating version 0 to 466 2023-01-15T14:35:11.459 INFO:tasks.rados.rados.0.smithi090.stdout:1590: oid 47 version 466 is already newer than 465 2023-01-15T14:35:11.460 INFO:tasks.rados.rados.0.smithi090.stdout:1590: oid 47 updating version 466 to 467 2023-01-15T14:35:11.460 INFO:tasks.rados.rados.0.smithi090.stdout:1590: oid 47 version 467 is already newer than 464 2023-01-15T14:35:11.460 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 47 v 467 (ObjNum 562 snap 176 seq_num 562) dirty exists 2023-01-15T14:35:11.460 INFO:tasks.rados.rados.0.smithi090.stdout:1590: left oid 47 (ObjNum 562 snap 176 seq_num 562) 2023-01-15T14:35:11.461 INFO:tasks.rados.rados.0.smithi090.stdout:1583: done (9 left) 2023-01-15T14:35:11.461 INFO:tasks.rados.rados.0.smithi090.stdout:1584: done (8 left) 2023-01-15T14:35:11.461 INFO:tasks.rados.rados.0.smithi090.stdout:1585: done (7 left) 2023-01-15T14:35:11.461 INFO:tasks.rados.rados.0.smithi090.stdout:1588: done (6 left) 2023-01-15T14:35:11.461 INFO:tasks.rados.rados.0.smithi090.stdout:1589: done (5 left) 2023-01-15T14:35:11.461 INFO:tasks.rados.rados.0.smithi090.stdout:1590: done (4 left) 2023-01-15T14:35:11.461 INFO:tasks.rados.rados.0.smithi090.stdout:1592: done (3 left) 2023-01-15T14:35:11.461 INFO:tasks.rados.rados.0.smithi090.stdout:1593: done (2 left) 2023-01-15T14:35:11.461 INFO:tasks.rados.rados.0.smithi090.stdout:1594: write oid 3 current snap is 176 2023-01-15T14:35:11.461 INFO:tasks.rados.rados.0.smithi090.stdout:1594: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2023-01-15T14:35:11.462 INFO:tasks.rados.rados.0.smithi090.stdout:1594: writing smithi09025264-3 from 695955 to 1421762 tid 1 2023-01-15T14:35:11.467 INFO:tasks.rados.rados.0.smithi090.stdout:1594: writing smithi09025264-3 from 1879385 to 2381582 tid 2 2023-01-15T14:35:11.659 INFO:tasks.rados.rados.0.smithi090.stdout:1594: writing smithi09025264-3 from 2949285 to 3429384 tid 3 2023-01-15T14:35:11.659 INFO:tasks.rados.rados.0.smithi090.stdout:1595: snap_remove snap 169 2023-01-15T14:35:11.659 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:11.650+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:11.660 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:35:11.660 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:35:11.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:35:11.661 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:35:11.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.661 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:5 3.5 3:a1c47f99:::smithi09025264-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:35:11.661 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:35:11.661 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.75814700400002, 2023-01-15T14:35:11.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40548699700000002, 2023-01-15T14:35:11.661 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:11.661 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:11.661 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:11.661 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:11.661 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 5 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:35:11.662 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781744+0000", 2023-01-15T14:35:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782566+0000", 2023-01-15T14:35:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000821578 2023-01-15T14:35:11.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782588+0000", 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.226e-05 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782601+0000", 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2998e-05 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882850+0000", 2023-01-15T14:35:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100248908 2023-01-15T14:35:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:35:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882871+0000", 2023-01-15T14:35:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0873000000000002e-05 2023-01-15T14:35:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:11.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083679+0000", 2023-01-15T14:35:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200808083 2023-01-15T14:35:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083948+0000", 2023-01-15T14:35:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000269342 2023-01-15T14:35:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084161+0000", 2023-01-15T14:35:11.666 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021326900000000001 2023-01-15T14:35:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084655+0000", 2023-01-15T14:35:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049371300000000001 2023-01-15T14:35:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:11.668 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084674+0000", 2023-01-15T14:35:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9012999999999999e-05 2023-01-15T14:35:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:11.669 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084924+0000", 2023-01-15T14:35:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024989899999999999 2023-01-15T14:35:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.673 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085221+0000", 2023-01-15T14:35:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029719600000000001 2023-01-15T14:35:11.674 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:11.864 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186953+0000", 2023-01-15T14:35:11.864 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10173174 2023-01-15T14:35:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187026+0000", 2023-01-15T14:35:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2957e-05 2023-01-15T14:35:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187124+0000", 2023-01-15T14:35:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8146000000000005e-05 2023-01-15T14:35:11.865 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187233+0000", 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010931599999999999 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:6 3.5 3:a1c47f99:::smithi09025264-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.75289413199999, 2023-01-15T14:35:11.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50419954199999995, 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 6 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:35:11.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:11.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:35:11.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:11.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:11.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786997+0000", 2023-01-15T14:35:11.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:11.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:11.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787486+0000", 2023-01-15T14:35:11.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048882999999999999 2023-01-15T14:35:11.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.884 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:11.884 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787514+0000", 2023-01-15T14:35:11.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7591999999999999e-05 2023-01-15T14:35:11.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:11.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787526+0000", 2023-01-15T14:35:11.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1894e-05 2023-01-15T14:35:11.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:11.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882966+0000", 2023-01-15T14:35:11.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095440450999999996 2023-01-15T14:35:11.887 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.887 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.887 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:11.887 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882973+0000", 2023-01-15T14:35:11.888 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2570000000000001e-06 2023-01-15T14:35:11.888 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.888 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.888 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:11.888 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084809+0000", 2023-01-15T14:35:11.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201835763 2023-01-15T14:35:11.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:11.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084928+0000", 2023-01-15T14:35:11.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118946 2023-01-15T14:35:11.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.890 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:11.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187347+0000", 2023-01-15T14:35:11.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102419285 2023-01-15T14:35:11.891 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:11.891 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187450+0000", 2023-01-15T14:35:11.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010286299999999999 2023-01-15T14:35:11.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:11.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187735+0000", 2023-01-15T14:35:11.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028429900000000001 2023-01-15T14:35:11.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:11.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:11.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:11.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188113+0000", 2023-01-15T14:35:11.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037883200000000001 2023-01-15T14:35:11.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:12.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188138+0000", 2023-01-15T14:35:12.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4110000000000001e-05 2023-01-15T14:35:12.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:12.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188389+0000", 2023-01-15T14:35:12.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025173900000000001 2023-01-15T14:35:12.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189251+0000", 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086129800000000003 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290989+0000", 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101738524 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:12.186 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291075+0000", 2023-01-15T14:35:12.187 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5469000000000004e-05 2023-01-15T14:35:12.187 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.187 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.187 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:12.187 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291144+0000", 2023-01-15T14:35:12.187 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:35:12.188 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.188 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.188 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:12.188 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291199+0000", 2023-01-15T14:35:12.188 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4413000000000002e-05 2023-01-15T14:35:12.188 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:12.188 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:12.188 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:12.188 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.188 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.189 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:7 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:35:12.189 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:35:12.189 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.75234767500001, 2023-01-15T14:35:12.189 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50407251399999997, 2023-01-15T14:35:12.189 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:12.189 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:12.189 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:12.189 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:12.189 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:12.189 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 7 2023-01-15T14:35:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:35:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:35:12.190 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:12.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:12.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787545+0000", 2023-01-15T14:35:12.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:12.191 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.191 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.191 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:12.191 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787551+0000", 2023-01-15T14:35:12.191 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8669999999999999e-06 2023-01-15T14:35:12.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:12.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787559+0000", 2023-01-15T14:35:12.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9780000000000005e-06 2023-01-15T14:35:12.192 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.192 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.192 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:12.192 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787564+0000", 2023-01-15T14:35:12.192 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5709999999999997e-06 2023-01-15T14:35:12.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883054+0000", 2023-01-15T14:35:12.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095489722999999999 2023-01-15T14:35:12.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.193 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:12.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883061+0000", 2023-01-15T14:35:12.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2760000000000003e-06 2023-01-15T14:35:12.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.194 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084987+0000", 2023-01-15T14:35:12.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20192545000000001 2023-01-15T14:35:12.194 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.194 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:12.194 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085190+0000", 2023-01-15T14:35:12.194 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020288899999999999 2023-01-15T14:35:12.194 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.194 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188374+0000", 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103184449 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188543+0000", 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016925299999999999 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:12.195 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188748+0000", 2023-01-15T14:35:12.196 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020473600000000001 2023-01-15T14:35:12.196 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.196 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.196 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:12.196 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188986+0000", 2023-01-15T14:35:12.196 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023780900000000001 2023-01-15T14:35:12.196 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.196 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.196 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:12.197 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189006+0000", 2023-01-15T14:35:12.197 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0604000000000001e-05 2023-01-15T14:35:12.197 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.198 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.198 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:12.198 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189084+0000", 2023-01-15T14:35:12.198 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7526000000000003e-05 2023-01-15T14:35:12.198 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.199 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.199 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:12.199 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189272+0000", 2023-01-15T14:35:12.199 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018762 2023-01-15T14:35:12.199 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.199 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.200 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:12.200 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291333+0000", 2023-01-15T14:35:12.200 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102061185 2023-01-15T14:35:12.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:12.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291466+0000", 2023-01-15T14:35:12.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013325699999999999 2023-01-15T14:35:12.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:12.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291535+0000", 2023-01-15T14:35:12.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9309999999999999e-05 2023-01-15T14:35:12.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:12.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291618+0000", 2023-01-15T14:35:12.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2773000000000002e-05 2023-01-15T14:35:12.204 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:12.204 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:12.204 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:12.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.204 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:8 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:35:12.205 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:35:12.205 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.752321693, 2023-01-15T14:35:12.206 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50442076000000002, 2023-01-15T14:35:12.206 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:12.206 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:12.207 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:12.207 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:12.207 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:12.207 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 8 2023-01-15T14:35:12.207 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.207 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:12.207 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.208 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:12.208 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:35:12.208 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:12.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.208 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.208 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:12.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:35:12.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:12.209 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.209 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:12.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787571+0000", 2023-01-15T14:35:12.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:12.209 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.210 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:12.210 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787575+0000", 2023-01-15T14:35:12.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7890000000000001e-06 2023-01-15T14:35:12.210 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.210 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:12.210 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787589+0000", 2023-01-15T14:35:12.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4212999999999999e-05 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787594+0000", 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1429999999999997e-06 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983143+0000", 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.195549003 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.211 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.212 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:12.212 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983153+0000", 2023-01-15T14:35:12.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4369999999999996e-06 2023-01-15T14:35:12.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.212 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.212 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085286+0000", 2023-01-15T14:35:12.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102133064 2023-01-15T14:35:12.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:12.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085401+0000", 2023-01-15T14:35:12.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115176 2023-01-15T14:35:12.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189116+0000", 2023-01-15T14:35:12.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103715444 2023-01-15T14:35:12.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:12.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189221+0000", 2023-01-15T14:35:12.354 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010438 2023-01-15T14:35:12.354 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.354 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.354 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.354 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291724+0000", 2023-01-15T14:35:12.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102502998 2023-01-15T14:35:12.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:12.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291824+0000", 2023-01-15T14:35:12.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010010599999999999 2023-01-15T14:35:12.355 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.355 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.355 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:12.355 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291992+0000", 2023-01-15T14:35:12.355 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016856900000000001 2023-01-15T14:35:12.356 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:12.356 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:12.356 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:12.356 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.356 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.356 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:21 3.5 3:a75958b1:::smithi09025264-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:35:12.356 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:35:12.356 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.70823277800002, 2023-01-15T14:35:12.356 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.45851924500000002, 2023-01-15T14:35:12.356 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 21 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:12.357 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:12.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:35:12.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:12.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.358 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:12.358 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831657+0000", 2023-01-15T14:35:12.358 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:35:12.358 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.358 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.359 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:12.359 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832814+0000", 2023-01-15T14:35:12.359 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011564909999999999 2023-01-15T14:35:12.359 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:12.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832864+0000", 2023-01-15T14:35:12.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0243e-05 2023-01-15T14:35:12.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:12.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832903+0000", 2023-01-15T14:35:12.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8704000000000001e-05 2023-01-15T14:35:12.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983202+0000", 2023-01-15T14:35:12.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.150299404 2023-01-15T14:35:12.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.484 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.484 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:12.484 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983208+0000", 2023-01-15T14:35:12.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0039999999999996e-06 2023-01-15T14:35:12.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.485 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185539+0000", 2023-01-15T14:35:12.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202330867 2023-01-15T14:35:12.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:12.486 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185803+0000", 2023-01-15T14:35:12.486 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026394299999999999 2023-01-15T14:35:12.486 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.486 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.521 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:12.521 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186037+0000", 2023-01-15T14:35:12.521 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023386300000000001 2023-01-15T14:35:12.521 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.521 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:12.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186637+0000", 2023-01-15T14:35:12.522 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060001800000000001 2023-01-15T14:35:12.522 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.522 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.522 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:12.522 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186665+0000", 2023-01-15T14:35:12.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8459999999999999e-05 2023-01-15T14:35:12.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.523 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:12.523 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187045+0000", 2023-01-15T14:35:12.523 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037992899999999999 2023-01-15T14:35:12.523 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.523 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.524 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:12.524 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188201+0000", 2023-01-15T14:35:12.685 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001155309 2023-01-15T14:35:12.685 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.685 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.685 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:12.685 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289903+0000", 2023-01-15T14:35:12.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101702601 2023-01-15T14:35:12.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:12.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289985+0000", 2023-01-15T14:35:12.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1617000000000001e-05 2023-01-15T14:35:12.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.686 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.686 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:12.686 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290065+0000", 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0276000000000002e-05 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290180+0000", 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011464300000000001 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.687 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:22 3.5 3:a75958b1:::smithi09025264-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:35:12.688 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:35:12.688 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.70376508999999, 2023-01-15T14:35:12.688 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65790599599999999, 2023-01-15T14:35:12.688 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:12.688 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:12.688 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:12.688 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:12.688 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:12.688 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 22 2023-01-15T14:35:12.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.689 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:12.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.689 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:12.689 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:35:12.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:12.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.689 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:12.689 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:35:12.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:12.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836126+0000", 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836181+0000", 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4246999999999997e-05 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:12.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836212+0000", 2023-01-15T14:35:12.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1843e-05 2023-01-15T14:35:12.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:12.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836226+0000", 2023-01-15T14:35:12.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3216e-05 2023-01-15T14:35:12.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186757+0000", 2023-01-15T14:35:12.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35053103899999999 2023-01-15T14:35:12.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:12.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186871+0000", 2023-01-15T14:35:12.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114403 2023-01-15T14:35:12.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290263+0000", 2023-01-15T14:35:12.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103391976 2023-01-15T14:35:12.885 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.885 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.885 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:12.885 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290413+0000", 2023-01-15T14:35:12.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014991200000000001 2023-01-15T14:35:12.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:12.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290627+0000", 2023-01-15T14:35:12.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021379299999999999 2023-01-15T14:35:12.886 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.886 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.886 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:12.886 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290861+0000", 2023-01-15T14:35:12.886 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023479600000000001 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290888+0000", 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6570000000000001e-05 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290990+0000", 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101723 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291245+0000", 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025543899999999999 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392822+0000", 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10157725500000001 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493881+0000", 2023-01-15T14:35:12.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101058262 2023-01-15T14:35:12.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:12.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493964+0000", 2023-01-15T14:35:12.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2973000000000007e-05 2023-01-15T14:35:12.907 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:12.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494034+0000", 2023-01-15T14:35:12.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0432999999999995e-05 2023-01-15T14:35:12.911 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:12.911 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:12.911 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:12.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.912 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:23 3.5 3:a75958b1:::smithi09025264-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:35:12.912 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:35:12.912 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.703649961, 2023-01-15T14:35:12.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65796116900000001, 2023-01-15T14:35:12.912 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:12.913 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:12.913 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:12.913 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:12.913 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:12.913 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 23 2023-01-15T14:35:12.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.914 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:12.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:12.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:35:12.914 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:12.914 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:12.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:35:12.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:12.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.916 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:12.916 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836242+0000", 2023-01-15T14:35:12.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:12.916 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.916 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:12.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836256+0000", 2023-01-15T14:35:12.917 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3677e-05 2023-01-15T14:35:12.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:12.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836265+0000", 2023-01-15T14:35:12.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2769999999999994e-06 2023-01-15T14:35:12.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:12.918 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936398+0000", 2023-01-15T14:35:12.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100132559 2023-01-15T14:35:12.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:12.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:12.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:12.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289431+0000", 2023-01-15T14:35:13.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35303322399999998 2023-01-15T14:35:13.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:13.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289587+0000", 2023-01-15T14:35:13.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000155873 2023-01-15T14:35:13.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:13.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292052+0000", 2023-01-15T14:35:13.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0024647919999999999 2023-01-15T14:35:13.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:13.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292180+0000", 2023-01-15T14:35:13.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012810999999999999 2023-01-15T14:35:13.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:13.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292373+0000", 2023-01-15T14:35:13.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019258599999999999 2023-01-15T14:35:13.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:13.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292597+0000", 2023-01-15T14:35:13.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022465500000000001 2023-01-15T14:35:13.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:13.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292618+0000", 2023-01-15T14:35:13.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0319999999999999e-05 2023-01-15T14:35:13.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:13.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292692+0000", 2023-01-15T14:35:13.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4643999999999999e-05 2023-01-15T14:35:13.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:13.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393103+0000", 2023-01-15T14:35:13.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100411156 2023-01-15T14:35:13.294 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.294 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.294 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:13.294 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493827+0000", 2023-01-15T14:35:13.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100724143 2023-01-15T14:35:13.294 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.294 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.294 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:13.294 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494085+0000", 2023-01-15T14:35:13.294 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000257891 2023-01-15T14:35:13.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:13.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494141+0000", 2023-01-15T14:35:13.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6132999999999999e-05 2023-01-15T14:35:13.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.295 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:13.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494204+0000", 2023-01-15T14:35:13.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2966000000000007e-05 2023-01-15T14:35:13.295 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:13.296 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:13.296 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:13.296 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.296 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.296 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:24 3.5 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:35:13.296 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:35:13.296 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.60346853499999, 2023-01-15T14:35:13.296 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.55808677600000001, 2023-01-15T14:35:13.296 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:13.296 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:13.297 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:13.297 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:13.297 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:13.297 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 24 2023-01-15T14:35:13.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.297 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:13.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:13.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:35:13.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:13.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:13.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:35:13.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:13.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:13.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936423+0000", 2023-01-15T14:35:13.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:13.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936433+0000", 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0443e-05 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936451+0000", 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7842999999999998e-05 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:13.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936461+0000", 2023-01-15T14:35:13.300 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0057e-05 2023-01-15T14:35:13.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.308 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:13.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289693+0000", 2023-01-15T14:35:13.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.353231617 2023-01-15T14:35:13.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:13.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289814+0000", 2023-01-15T14:35:13.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120763 2023-01-15T14:35:13.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:13.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392923+0000", 2023-01-15T14:35:13.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10310955300000001 2023-01-15T14:35:13.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:13.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393072+0000", 2023-01-15T14:35:13.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000148469 2023-01-15T14:35:13.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:13.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494242+0000", 2023-01-15T14:35:13.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117034499999999 2023-01-15T14:35:13.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:13.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494363+0000", 2023-01-15T14:35:13.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120703 2023-01-15T14:35:13.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:13.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494511+0000", 2023-01-15T14:35:13.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014873299999999999 2023-01-15T14:35:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:78 3.c 3:3137f0cb:::smithi09025264-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:35:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.21350253899999, 2023-01-15T14:35:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20331801699999999, 2023-01-15T14:35:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:13.461 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 78 2023-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:13.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326388+0000", 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326672+0000", 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028367900000000001 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326713+0000", 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1539999999999999e-05 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326729+0000", 2023-01-15T14:35:13.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.519e-05 2023-01-15T14:35:13.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:13.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326879+0000", 2023-01-15T14:35:13.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015064899999999999 2023-01-15T14:35:13.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327096+0000", 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021682299999999999 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327308+0000", 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000211366 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327757+0000", 2023-01-15T14:35:13.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000449009 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327776+0000", 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9201000000000001e-05 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328012+0000", 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236054 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:13.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429534+0000", 2023-01-15T14:35:13.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10152201499999999 2023-01-15T14:35:13.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:13.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529340+0000", 2023-01-15T14:35:13.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099806249 2023-01-15T14:35:13.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:13.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529437+0000", 2023-01-15T14:35:13.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6949999999999998e-05 2023-01-15T14:35:13.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:13.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529587+0000", 2023-01-15T14:35:13.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015008600000000001 2023-01-15T14:35:13.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.937 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.938 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:13.938 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529709+0000", 2023-01-15T14:35:13.938 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121636 2023-01-15T14:35:13.938 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:13.938 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:13.938 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:13.938 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.939 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.939 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:79 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:13.939 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:35:13.939 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.21313294300001, 2023-01-15T14:35:13.939 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20348171300000001, 2023-01-15T14:35:13.939 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:13.939 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:13.940 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:13.940 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:13.940 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:13.942 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 79 2023-01-15T14:35:13.943 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.946 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:13.946 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.946 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:13.946 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:35:13.946 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:13.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.947 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.947 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:13.947 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:35:13.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:13.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.947 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.947 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:13.947 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326759+0000", 2023-01-15T14:35:13.947 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:13.947 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.948 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.948 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:13.948 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326770+0000", 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326791+0000", 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1556000000000002e-05 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326818+0000", 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6979999999999999e-05 2023-01-15T14:35:13.949 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:13.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.427993+0000", 2023-01-15T14:35:13.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117517199999999 2023-01-15T14:35:13.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.950 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:13.950 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428266+0000", 2023-01-15T14:35:13.950 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027233299999999998 2023-01-15T14:35:13.950 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.950 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.951 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:35:13.951 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428696+0000", 2023-01-15T14:35:13.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042996499999999998 2023-01-15T14:35:13.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.951 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:13.951 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429057+0000", 2023-01-15T14:35:13.951 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000360952 2023-01-15T14:35:13.951 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.951 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.951 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:13.952 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429076+0000", 2023-01-15T14:35:13.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:35:13.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.952 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:13.952 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429160+0000", 2023-01-15T14:35:13.952 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:35:13.952 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.952 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.952 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:13.953 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429549+0000", 2023-01-15T14:35:13.953 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038884599999999998 2023-01-15T14:35:13.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.953 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:13.953 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529886+0000", 2023-01-15T14:35:13.953 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100336261 2023-01-15T14:35:13.953 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:13.953 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:13.953 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:13.953 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529978+0000", 2023-01-15T14:35:13.954 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2775999999999994e-05 2023-01-15T14:35:14.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.099 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:14.099 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530108+0000", 2023-01-15T14:35:14.099 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012958300000000001 2023-01-15T14:35:14.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.099 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:14.099 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530242+0000", 2023-01-15T14:35:14.099 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013406 2023-01-15T14:35:14.099 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.099 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:14.100 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.100 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.100 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:80 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:14.100 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:35:14.100 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.212994584, 2023-01-15T14:35:14.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20370874, 2023-01-15T14:35:14.100 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:14.100 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:14.100 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:14.101 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:14.101 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:14.101 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 80 2023-01-15T14:35:14.101 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.101 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:14.101 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.101 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:14.101 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:35:14.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:14.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:35:14.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:14.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326897+0000", 2023-01-15T14:35:14.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:14.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326906+0000", 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6929999999999996e-06 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326931+0000", 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4757e-05 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:14.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326945+0000", 2023-01-15T14:35:14.104 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.402e-05 2023-01-15T14:35:14.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.104 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.104 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:14.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429239+0000", 2023-01-15T14:35:14.104 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102294418 2023-01-15T14:35:14.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.105 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:14.105 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429487+0000", 2023-01-15T14:35:14.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024789300000000001 2023-01-15T14:35:14.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.105 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:14.105 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530309+0000", 2023-01-15T14:35:14.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100821459 2023-01-15T14:35:14.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.106 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.106 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:14.106 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530438+0000", 2023-01-15T14:35:14.106 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012928700000000001 2023-01-15T14:35:14.106 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.106 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.106 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:14.106 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530607+0000", 2023-01-15T14:35:14.106 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169483 2023-01-15T14:35:14.106 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.106 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:14.107 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.107 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.107 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.107 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(client.4530.0:78 3.c e23/20 ondisk, result = 0)", 2023-01-15T14:35:14.107 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:35:14.107 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 405.21110286300001, 2023-01-15T14:35:14.107 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20094287799999999, 2023-01-15T14:35:14.107 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:14.107 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:14.107 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:14.108 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.108 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:14.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:35:14.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.108 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.108 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.108 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:14.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328790+0000", 2023-01-15T14:35:14.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:14.109 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328789+0000", 2023-01-15T14:35:14.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:14.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:14.112 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328799+0000", 2023-01-15T14:35:14.112 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8980000000000008e-06 2023-01-15T14:35:14.115 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.116 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:14.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328820+0000", 2023-01-15T14:35:14.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1041999999999999e-05 2023-01-15T14:35:14.116 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.116 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:14.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328832+0000", 2023-01-15T14:35:14.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1616e-05 2023-01-15T14:35:14.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:14.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529404+0000", 2023-01-15T14:35:14.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200572631 2023-01-15T14:35:14.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:14.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529419+0000", 2023-01-15T14:35:14.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4250000000000001e-05 2023-01-15T14:35:14.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.118 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.118 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:14.118 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529733+0000", 2023-01-15T14:35:14.118 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000314693 2023-01-15T14:35:14.118 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.118 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:14.118 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.118 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.118 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.118 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:201 3.5 3:a1c47f99:::smithi09025264-2:head [copy-from ver 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:14.119 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:35:14.119 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 404.133496466, 2023-01-15T14:35:14.119 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30931411399999997, 2023-01-15T14:35:14.119 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:14.119 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:14.119 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:14.119 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:14.119 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:14.119 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 201 2023-01-15T14:35:14.119 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.120 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:14.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:14.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:35:14.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:14.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:35:14.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:14.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406394+0000", 2023-01-15T14:35:14.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:14.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:14.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406408+0000", 2023-01-15T14:35:14.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4416999999999999e-05 2023-01-15T14:35:14.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:14.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406446+0000", 2023-01-15T14:35:14.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7339999999999998e-05 2023-01-15T14:35:14.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:14.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406467+0000", 2023-01-15T14:35:14.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1137000000000001e-05 2023-01-15T14:35:14.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:14.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406677+0000", 2023-01-15T14:35:14.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000210275 2023-01-15T14:35:14.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:14.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406902+0000", 2023-01-15T14:35:14.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022529000000000001 2023-01-15T14:35:14.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.513012+0000", 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.106109527 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514754+0000", 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0017419779999999999 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:14.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514773+0000", 2023-01-15T14:35:14.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8700000000000001e-05 2023-01-15T14:35:14.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:14.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.515531+0000", 2023-01-15T14:35:14.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075844300000000003 2023-01-15T14:35:14.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:14.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615334+0000", 2023-01-15T14:35:14.265 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099803327999999997 2023-01-15T14:35:14.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:14.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615376+0000", 2023-01-15T14:35:14.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1675999999999999e-05 2023-01-15T14:35:14.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:14.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715546+0000", 2023-01-15T14:35:14.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016992500000001 2023-01-15T14:35:14.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:14.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715626+0000", 2023-01-15T14:35:14.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0112999999999997e-05 2023-01-15T14:35:14.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.268 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:14.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715711+0000", 2023-01-15T14:35:14.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:35:14.268 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.268 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:14.268 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.268 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.269 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:202 3.5 3:a1c47f99:::smithi09025264-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:14.269 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:35:14.269 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 404.13338392000003, 2023-01-15T14:35:14.269 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31002637300000002, 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 202 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:35:14.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:14.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:35:14.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406508+0000", 2023-01-15T14:35:14.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:14.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:14.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406516+0000", 2023-01-15T14:35:14.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.413e-06 2023-01-15T14:35:14.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:14.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406534+0000", 2023-01-15T14:35:14.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8936999999999998e-05 2023-01-15T14:35:14.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:14.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406548+0000", 2023-01-15T14:35:14.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3196e-05 2023-01-15T14:35:14.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:14.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407271+0000", 2023-01-15T14:35:14.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00072301500000000003 2023-01-15T14:35:14.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:14.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407466+0000", 2023-01-15T14:35:14.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000195009 2023-01-15T14:35:14.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:14.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.614996+0000", 2023-01-15T14:35:14.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.207530192 2023-01-15T14:35:14.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:14.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615227+0000", 2023-01-15T14:35:14.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000230979 2023-01-15T14:35:14.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.279 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.279 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:14.279 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716282+0000", 2023-01-15T14:35:14.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101055465 2023-01-15T14:35:14.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:14.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716409+0000", 2023-01-15T14:35:14.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012653999999999999 2023-01-15T14:35:14.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:14.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716536+0000", 2023-01-15T14:35:14.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012683 2023-01-15T14:35:14.281 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.281 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:14.281 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.281 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:189 3.d e23/20 3:b3f111b5:::smithi09025264-48:head v 23'10, mlcod=23'6)", 2023-01-15T14:35:14.281 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:35:14.282 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 404.08817927199999, 2023-01-15T14:35:14.282 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20400475100000001, 2023-01-15T14:35:14.282 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:14.282 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:14.439 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:14.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:14.441 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:35:14.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:14.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:35:14.442 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.442 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.442 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:14.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451712+0000", 2023-01-15T14:35:14.445 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:14.445 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.445 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:14.445 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452272+0000", 2023-01-15T14:35:14.445 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00056035899999999997 2023-01-15T14:35:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452310+0000", 2023-01-15T14:35:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8145999999999997e-05 2023-01-15T14:35:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452324+0000", 2023-01-15T14:35:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4443999999999999e-05 2023-01-15T14:35:14.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652703+0000", 2023-01-15T14:35:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20037818199999999 2023-01-15T14:35:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652762+0000", 2023-01-15T14:35:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9302999999999998e-05 2023-01-15T14:35:14.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:14.588 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653302+0000", 2023-01-15T14:35:14.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054012199999999998 2023-01-15T14:35:14.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.597 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.597 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:14.597 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653333+0000", 2023-01-15T14:35:14.597 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.112e-05 2023-01-15T14:35:14.597 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653667+0000", 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033337900000000002 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653709+0000", 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2469999999999998e-05 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.598 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:14.599 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655719+0000", 2023-01-15T14:35:14.599 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020096699999999999 2023-01-15T14:35:14.599 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.599 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:14.599 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.599 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.599 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.599 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:269 3.2 e30/20 3:47dfe935:::smithi09025264-29:head v 30'11, mlcod=23'8)", 2023-01-15T14:35:14.599 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:35:14.745 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 396.66213383100001, 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23185540399999999, 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:14.746 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:35:14.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:14.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:14.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:35:14.747 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:14.747 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.747 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.747 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:14.747 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878379+0000", 2023-01-15T14:35:14.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061979600000000004 2023-01-15T14:35:14.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.749 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:14.749 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878406+0000", 2023-01-15T14:35:14.749 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:35:14.749 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.749 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978515+0000", 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100108461 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078759+0000", 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100244496 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:14.750 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078796+0000", 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6758999999999998e-05 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079567+0000", 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077137199999999999 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079593+0000", 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:35:14.751 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079844+0000", 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025075899999999998 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079910+0000", 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6406000000000001e-05 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:14.752 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:14.753 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.109615+0000", 2023-01-15T14:35:14.753 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.029704323000000001 2023-01-15T14:35:14.753 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:14.753 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:14.753 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:15.060 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.060 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.061 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:281 3.5 3:a75958b1:::smithi09025264-6:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:35:15.061 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:35:15.061 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 396.56121020199998, 2023-01-15T14:35:15.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30574310900000001, 2023-01-15T14:35:15.061 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:15.061 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:15.061 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:15.061 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:15.061 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:15.061 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 281 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:15.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.063 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:15.063 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:35:15.063 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:15.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.063 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:15.063 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978686+0000", 2023-01-15T14:35:15.063 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4429999999999999e-06 2023-01-15T14:35:15.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.064 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:15.064 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978690+0000", 2023-01-15T14:35:15.064 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0300000000000004e-06 2023-01-15T14:35:15.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.064 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.064 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:15.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978693+0000", 2023-01-15T14:35:15.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.179e-06 2023-01-15T14:35:15.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:15.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981592+0000", 2023-01-15T14:35:15.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0028984140000000002 2023-01-15T14:35:15.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:15.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981718+0000", 2023-01-15T14:35:15.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012616600000000001 2023-01-15T14:35:15.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:15.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981982+0000", 2023-01-15T14:35:15.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026445199999999998 2023-01-15T14:35:15.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:15.068 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982278+0000", 2023-01-15T14:35:15.068 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029514100000000001 2023-01-15T14:35:15.068 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.068 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.068 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:15.068 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982290+0000", 2023-01-15T14:35:15.068 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.271e-05 2023-01-15T14:35:15.068 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.068 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.069 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:15.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982343+0000", 2023-01-15T14:35:15.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2914000000000002e-05 2023-01-15T14:35:15.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.069 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.070 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:15.070 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982385+0000", 2023-01-15T14:35:15.070 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2114999999999999e-05 2023-01-15T14:35:15.070 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.070 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:15.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.181937+0000", 2023-01-15T14:35:15.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19955213099999999 2023-01-15T14:35:15.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.071 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:15.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284183+0000", 2023-01-15T14:35:15.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10224538 2023-01-15T14:35:15.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:15.073 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284318+0000", 2023-01-15T14:35:15.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000134843 2023-01-15T14:35:15.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:15.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284426+0000", 2023-01-15T14:35:15.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108513 2023-01-15T14:35:15.222 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:15.222 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:15.222 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:15.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.223 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:373 3.4 e34/20 3:2029fa71:::smithi09025264-39:head v 34'19, mlcod=26'16)", 2023-01-15T14:35:15.223 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:35:15.223 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 391.08022774300002, 2023-01-15T14:35:15.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20212622799999999, 2023-01-15T14:35:15.223 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:15.223 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:15.223 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:15.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:15.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:35:15.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:15.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:15.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459665+0000", 2023-01-15T14:35:15.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:15.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:15.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.459664+0000", 2023-01-15T14:35:15.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:15.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:15.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460099+0000", 2023-01-15T14:35:15.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00043515900000000001 2023-01-15T14:35:15.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:15.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.460130+0000", 2023-01-15T14:35:15.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1590000000000001e-05 2023-01-15T14:35:15.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:15.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560226+0000", 2023-01-15T14:35:15.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100095248 2023-01-15T14:35:15.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.241 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.241 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:15.241 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560369+0000", 2023-01-15T14:35:15.241 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014381499999999999 2023-01-15T14:35:15.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.241 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.241 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:15.241 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560404+0000", 2023-01-15T14:35:15.241 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4650000000000002e-05 2023-01-15T14:35:15.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.242 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.242 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:15.242 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560804+0000", 2023-01-15T14:35:15.242 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000399823 2023-01-15T14:35:15.242 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.242 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.243 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:15.243 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.560827+0000", 2023-01-15T14:35:15.243 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3612999999999999e-05 2023-01-15T14:35:15.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.243 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.243 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.561101+0000", 2023-01-15T14:35:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027355499999999998 2023-01-15T14:35:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661710+0000", 2023-01-15T14:35:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100608716 2023-01-15T14:35:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:15.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.661792+0000", 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2006000000000001e-05 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:455 3.d e41/20 3:bd982ca3:::smithi09025264-38:head v 41'24, mlcod=38'18)", 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 384.03046617000001, 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20118033699999999, 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:15.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509427+0000", 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:15.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509425+0000", 2023-01-15T14:35:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509441+0000", 2023-01-15T14:35:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5248e-05 2023-01-15T14:35:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:15.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.509474+0000", 2023-01-15T14:35:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3531000000000001e-05 2023-01-15T14:35:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.609644+0000", 2023-01-15T14:35:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016994999999999 2023-01-15T14:35:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:15.248 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709872+0000", 2023-01-15T14:35:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10022840299999999 2023-01-15T14:35:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:15.249 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.709906+0000", 2023-01-15T14:35:15.379 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3160000000000001e-05 2023-01-15T14:35:15.379 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.379 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:15.380 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710153+0000", 2023-01-15T14:35:15.380 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024735399999999998 2023-01-15T14:35:15.380 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.380 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.380 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:15.381 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710204+0000", 2023-01-15T14:35:15.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1075999999999997e-05 2023-01-15T14:35:15.381 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.381 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.381 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:15.382 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710280+0000", 2023-01-15T14:35:15.382 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6202e-05 2023-01-15T14:35:15.382 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.382 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.382 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:15.382 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710317+0000", 2023-01-15T14:35:15.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6387000000000002e-05 2023-01-15T14:35:15.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.383 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:15.383 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:47.710607+0000", 2023-01-15T14:35:15.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029079699999999997 2023-01-15T14:35:15.390 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:15.397 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:497 3.d e44/20 3:bd982ca3:::smithi09025264-38:head v 44'31, mlcod=42'26)", 2023-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 381.76567130799998, 2023-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20082067100000001, 2023-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:15.398 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.399 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:15.399 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:35:15.399 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:15.399 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.400 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.400 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:15.400 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774222+0000", 2023-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:15.401 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774220+0000", 2023-01-15T14:35:15.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:15.402 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.402 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.402 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:15.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774238+0000", 2023-01-15T14:35:15.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8107000000000001e-05 2023-01-15T14:35:15.402 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.402 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.402 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:15.402 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774263+0000", 2023-01-15T14:35:15.402 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4122999999999999e-05 2023-01-15T14:35:15.403 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.403 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.403 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:15.403 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.774280+0000", 2023-01-15T14:35:15.403 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7039999999999999e-05 2023-01-15T14:35:15.403 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.403 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.403 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:15.403 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874060+0000", 2023-01-15T14:35:15.403 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099780698000000001 2023-01-15T14:35:15.404 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.404 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.404 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:15.404 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874130+0000", 2023-01-15T14:35:15.404 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9852999999999997e-05 2023-01-15T14:35:15.404 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.404 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.404 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:15.404 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874351+0000", 2023-01-15T14:35:15.404 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022091400000000001 2023-01-15T14:35:15.405 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.405 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.405 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:15.405 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874369+0000", 2023-01-15T14:35:15.405 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7876e-05 2023-01-15T14:35:15.405 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.405 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.405 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:15.405 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.874433+0000", 2023-01-15T14:35:15.405 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3793000000000004e-05 2023-01-15T14:35:15.405 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.406 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:15.406 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.974927+0000", 2023-01-15T14:35:15.406 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10049393500000001 2023-01-15T14:35:15.406 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:15.406 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:15.406 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:15.406 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.975043+0000", 2023-01-15T14:35:15.406 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115949 2023-01-15T14:35:15.406 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:15.406 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:15.407 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:15.407 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:15.407 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:15.407 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:35:15.407 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:35:15.408 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:35:15.409 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:35:15.409 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1, 6] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:35:15.409 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:35:15.409 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:35:15.410 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:35:15.411 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:15.405+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:15.418 INFO:tasks.rados.rados.0.smithi090.stdout:1595: done (3 left) 2023-01-15T14:35:15.538 INFO:tasks.rados.rados.0.smithi090.stdout:1596: snap_remove snap 168 2023-01-15T14:35:15.538 INFO:tasks.rados.rados.0.smithi090.stdout:1594: finishing write tid 1 to smithi09025264-3 2023-01-15T14:35:15.538 INFO:tasks.rados.rados.0.smithi090.stdout:1594: finishing write tid 2 to smithi09025264-3 2023-01-15T14:35:15.538 INFO:tasks.rados.rados.0.smithi090.stdout:1594: finishing write tid 3 to smithi09025264-3 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:1594: finishing write tid 4 to smithi09025264-3 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:1594: finishing write tid 5 to smithi09025264-3 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:1594: oid 3 updating version 0 to 409 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:1594: oid 3 version 409 is already newer than 407 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:1594: oid 3 version 409 is already newer than 408 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:1594: oid 3 updating version 409 to 410 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 410 (ObjNum 565 snap 176 seq_num 565) dirty exists 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:1594: left oid 3 (ObjNum 565 snap 176 seq_num 565) 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:1591: done (3 left) 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:1594: done (2 left) 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:1596: done (1 left) 2023-01-15T14:35:15.539 INFO:tasks.rados.rados.0.smithi090.stdout:1597: write oid 30 current snap is 176 2023-01-15T14:35:15.540 INFO:tasks.rados.rados.0.smithi090.stdout:1597: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2023-01-15T14:35:15.540 INFO:tasks.rados.rados.0.smithi090.stdout:1597: writing smithi09025264-30 from 712762 to 1113818 tid 1 2023-01-15T14:35:15.540 INFO:tasks.rados.rados.0.smithi090.stdout:1597: writing smithi09025264-30 from 1537867 to 2183722 tid 2 2023-01-15T14:35:15.540 INFO:tasks.rados.rados.0.smithi090.stdout:1597: writing smithi09025264-30 from 2860355 to 3512762 tid 3 2023-01-15T14:35:15.540 INFO:tasks.rados.rados.0.smithi090.stdout:1598: rollback oid 4 current snap is 176 2023-01-15T14:35:15.540 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 4 to 131 2023-01-15T14:35:15.541 INFO:tasks.rados.rados.0.smithi090.stdout:1599: read oid 29 snap -1 2023-01-15T14:35:15.541 INFO:tasks.rados.rados.0.smithi090.stdout:1599: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:35:15.541 INFO:tasks.rados.rados.0.smithi090.stdout:1600: write oid 14 current snap is 176 2023-01-15T14:35:15.541 INFO:tasks.rados.rados.0.smithi090.stdout:1600: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2023-01-15T14:35:15.541 INFO:tasks.rados.rados.0.smithi090.stdout:1600: writing smithi09025264-14 from 729569 to 1205874 tid 1 2023-01-15T14:35:15.542 INFO:tasks.rados.rados.0.smithi090.stdout:1600: writing smithi09025264-14 from 1996349 to 2502215 tid 2 2023-01-15T14:35:15.542 INFO:tasks.rados.rados.0.smithi090.stdout:1600: writing smithi09025264-14 from 3004131 to 3529569 tid 3 2023-01-15T14:35:15.542 INFO:tasks.rados.rados.0.smithi090.stdout:1601: read oid 10 snap -1 2023-01-15T14:35:15.542 INFO:tasks.rados.rados.0.smithi090.stdout:1601: expect (ObjNum 518 snap 155 seq_num 518) 2023-01-15T14:35:15.542 INFO:tasks.rados.rados.0.smithi090.stdout:1602: write oid 22 current snap is 176 2023-01-15T14:35:15.543 INFO:tasks.rados.rados.0.smithi090.stdout:1602: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2023-01-15T14:35:15.543 INFO:tasks.rados.rados.0.smithi090.stdout:1602: writing smithi09025264-22 from 746376 to 1297930 tid 1 2023-01-15T14:35:15.699 INFO:tasks.rados.rados.0.smithi090.stdout:1602: writing smithi09025264-22 from 2054831 to 2704355 tid 2 2023-01-15T14:35:15.860 INFO:tasks.rados.rados.0.smithi090.stdout:1602: writing smithi09025264-22 from 3315201 to 3546376 tid 3 2023-01-15T14:35:15.860 INFO:tasks.rados.rados.0.smithi090.stdout:1603: read oid 43 snap -1 2023-01-15T14:35:15.860 INFO:tasks.rados.rados.0.smithi090.stdout:1603: expect (ObjNum 527 snap 158 seq_num 527) 2023-01-15T14:35:15.860 INFO:tasks.rados.rados.0.smithi090.stdout:1604: write oid 12 current snap is 176 2023-01-15T14:35:15.860 INFO:tasks.rados.rados.0.smithi090.stdout:1604: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2023-01-15T14:35:15.860 INFO:tasks.rados.rados.0.smithi090.stdout:1604: writing smithi09025264-12 from 763183 to 1389986 tid 1 2023-01-15T14:35:15.860 INFO:tasks.rados.rados.0.smithi090.stdout:1604: writing smithi09025264-12 from 1996960 to 2790142 tid 2 2023-01-15T14:35:15.861 INFO:tasks.rados.rados.0.smithi090.stdout:1604: writing smithi09025264-12 from 3226271 to 3563183 tid 3 2023-01-15T14:35:15.861 INFO:tasks.rados.rados.0.smithi090.stdout:1605: snap_create 2023-01-15T14:35:15.861 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 9 2023-01-15T14:35:15.861 INFO:tasks.rados.rados.0.smithi090.stdout:1599: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:35:15.861 INFO:tasks.rados.rados.0.smithi090.stdout:1601: expect (ObjNum 518 snap 155 seq_num 518) 2023-01-15T14:35:15.861 INFO:tasks.rados.rados.0.smithi090.stdout:1599: done (8 left) 2023-01-15T14:35:15.861 INFO:tasks.rados.rados.0.smithi090.stdout:1601: done (7 left) 2023-01-15T14:35:15.861 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 7 2023-01-15T14:35:15.861 INFO:tasks.rados.rados.0.smithi090.stdout:1598: finishing rollback tid 0 to smithi09025264-4 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout:1603: expect (ObjNum 527 snap 158 seq_num 527) 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout:1598: finishing rollback tid 1 to smithi09025264-4 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout:1603: done (6 left) 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout:1604: finishing write tid 1 to smithi09025264-12 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout:1598: finishing rollback tid 2 to smithi09025264-4 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 4 v 342 (ObjNum 143 snap 27 seq_num 143) dirty exists 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout:1598: done (5 left) 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout:1597: finishing write tid 1 to smithi09025264-30 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout:1597: finishing write tid 2 to smithi09025264-30 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout:1597: finishing write tid 3 to smithi09025264-30 2023-01-15T14:35:15.862 INFO:tasks.rados.rados.0.smithi090.stdout:1597: finishing write tid 4 to smithi09025264-30 2023-01-15T14:35:15.863 INFO:tasks.rados.rados.0.smithi090.stdout:1597: finishing write tid 5 to smithi09025264-30 2023-01-15T14:35:15.863 INFO:tasks.rados.rados.0.smithi090.stdout:1597: oid 30 updating version 0 to 242 2023-01-15T14:35:15.863 INFO:tasks.rados.rados.0.smithi090.stdout:1597: oid 30 updating version 242 to 244 2023-01-15T14:35:15.863 INFO:tasks.rados.rados.0.smithi090.stdout:1597: oid 30 version 244 is already newer than 243 2023-01-15T14:35:15.863 INFO:tasks.rados.rados.0.smithi090.stdout:1597: oid 30 updating version 244 to 245 2023-01-15T14:35:15.863 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 245 (ObjNum 566 snap 176 seq_num 566) dirty exists 2023-01-15T14:35:15.863 INFO:tasks.rados.rados.0.smithi090.stdout:1597: left oid 30 (ObjNum 566 snap 176 seq_num 566) 2023-01-15T14:35:15.864 INFO:tasks.rados.rados.0.smithi090.stdout:1597: done (4 left) 2023-01-15T14:35:15.864 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:35:15.865 INFO:tasks.rados.rados.0.smithi090.stdout:1602: finishing write tid 1 to smithi09025264-22 2023-01-15T14:35:15.865 INFO:tasks.rados.rados.0.smithi090.stdout:1602: finishing write tid 2 to smithi09025264-22 2023-01-15T14:35:15.865 INFO:tasks.rados.rados.0.smithi090.stdout:1602: finishing write tid 3 to smithi09025264-22 2023-01-15T14:35:15.865 INFO:tasks.rados.rados.0.smithi090.stdout:1602: finishing write tid 4 to smithi09025264-22 2023-01-15T14:35:15.865 INFO:tasks.rados.rados.0.smithi090.stdout:1602: finishing write tid 5 to smithi09025264-22 2023-01-15T14:35:15.865 INFO:tasks.rados.rados.0.smithi090.stdout:1602: oid 22 updating version 0 to 430 2023-01-15T14:35:15.865 INFO:tasks.rados.rados.0.smithi090.stdout:1602: oid 22 updating version 430 to 432 2023-01-15T14:35:15.865 INFO:tasks.rados.rados.0.smithi090.stdout:1602: oid 22 updating version 432 to 433 2023-01-15T14:35:15.865 INFO:tasks.rados.rados.0.smithi090.stdout:1602: oid 22 version 433 is already newer than 431 2023-01-15T14:35:15.865 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 22 v 433 (ObjNum 568 snap 176 seq_num 568) dirty exists 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout:1602: left oid 22 (ObjNum 568 snap 176 seq_num 568) 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout:1602: done (3 left) 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout:1604: finishing write tid 2 to smithi09025264-12 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout:1604: finishing write tid 3 to smithi09025264-12 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout:1604: finishing write tid 4 to smithi09025264-12 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout:1600: finishing write tid 1 to smithi09025264-14 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout:1604: finishing write tid 5 to smithi09025264-12 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout:1604: oid 12 updating version 0 to 336 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout:1604: oid 12 updating version 336 to 338 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout:1604: oid 12 version 338 is already newer than 337 2023-01-15T14:35:15.866 INFO:tasks.rados.rados.0.smithi090.stdout:1604: oid 12 updating version 338 to 339 2023-01-15T14:35:15.867 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 339 (ObjNum 569 snap 176 seq_num 569) dirty exists 2023-01-15T14:35:15.867 INFO:tasks.rados.rados.0.smithi090.stdout:1604: left oid 12 (ObjNum 569 snap 176 seq_num 569) 2023-01-15T14:35:15.867 INFO:tasks.rados.rados.0.smithi090.stdout:1604: done (2 left) 2023-01-15T14:35:15.867 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:35:15.867 INFO:tasks.rados.rados.0.smithi090.stdout:1600: finishing write tid 2 to smithi09025264-14 2023-01-15T14:35:15.867 INFO:tasks.rados.rados.0.smithi090.stdout:1600: finishing write tid 3 to smithi09025264-14 2023-01-15T14:35:15.867 INFO:tasks.rados.rados.0.smithi090.stdout:1600: finishing write tid 4 to smithi09025264-14 2023-01-15T14:35:15.867 INFO:tasks.rados.rados.0.smithi090.stdout:1600: finishing write tid 5 to smithi09025264-14 2023-01-15T14:35:15.867 INFO:tasks.rados.rados.0.smithi090.stdout:1600: oid 14 updating version 0 to 453 2023-01-15T14:35:15.867 INFO:tasks.rados.rados.0.smithi090.stdout:1600: oid 14 version 453 is already newer than 450 2023-01-15T14:35:15.867 INFO:tasks.rados.rados.0.smithi090.stdout:1600: oid 14 version 453 is already newer than 452 2023-01-15T14:35:15.868 INFO:tasks.rados.rados.0.smithi090.stdout:1600: oid 14 version 453 is already newer than 451 2023-01-15T14:35:15.868 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 453 (ObjNum 567 snap 176 seq_num 567) dirty exists 2023-01-15T14:35:15.868 INFO:tasks.rados.rados.0.smithi090.stdout:1600: left oid 14 (ObjNum 567 snap 176 seq_num 567) 2023-01-15T14:35:15.868 INFO:tasks.rados.rados.0.smithi090.stdout:1600: done (1 left) 2023-01-15T14:35:15.868 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:15.870 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:35:15.876 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:35:15.876 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:35:15.878 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:35:15.879 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:35:15.881 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:15.866+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:15.971 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:15.965+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:16.020 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:16.020 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:35:16.021 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:35:16.021 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:35:16.021 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:35:16.021 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:35:16.022 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:35:16.038 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:16.060 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:16.178 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:16.178 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:16.178 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:16.178 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:16.181 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:35:16.181 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:35:16.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:35:16.182 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:35:16.182 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.182 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:97 3.a 3:52df5138:::smithi09025264-25:head [write 403368~555035,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:16.182 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:35:16.182 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 409.63295923800001, 2023-01-15T14:35:16.182 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.340758171, 2023-01-15T14:35:16.182 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:16.182 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:16.182 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:16.183 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:16.183 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:16.183 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 97 2023-01-15T14:35:16.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.183 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:16.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.183 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:16.183 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:35:16.183 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.183 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.183 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.184 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:16.184 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501143+0000", 2023-01-15T14:35:16.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.184 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.184 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:16.184 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501140+0000", 2023-01-15T14:35:16.184 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:35:16.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.184 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501684+0000", 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054441799999999996 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501713+0000", 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8833e-05 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:16.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501726+0000", 2023-01-15T14:35:16.186 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3137e-05 2023-01-15T14:35:16.186 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:16.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.601982+0000", 2023-01-15T14:35:16.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100256518 2023-01-15T14:35:16.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.202 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.202 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602426+0000", 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044401500000000002 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.602703+0000", 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000276889 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603066+0000", 2023-01-15T14:35:16.203 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036278599999999998 2023-01-15T14:35:16.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:16.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603098+0000", 2023-01-15T14:35:16.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1433000000000002e-05 2023-01-15T14:35:16.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:16.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603382+0000", 2023-01-15T14:35:16.205 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028402899999999999 2023-01-15T14:35:16.205 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.205 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.205 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:16.205 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603632+0000", 2023-01-15T14:35:16.205 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025033 2023-01-15T14:35:16.205 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.205 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.205 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:16.205 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.805354+0000", 2023-01-15T14:35:16.206 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20172168700000001 2023-01-15T14:35:16.206 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.206 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.206 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:16.206 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841465+0000", 2023-01-15T14:35:16.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.036111852999999999 2023-01-15T14:35:16.207 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.207 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:16.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841682+0000", 2023-01-15T14:35:16.234 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000216949 2023-01-15T14:35:16.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.234 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.234 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:16.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.841901+0000", 2023-01-15T14:35:16.234 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021858700000000001 2023-01-15T14:35:16.234 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:16.234 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:16.234 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:16.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.235 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.235 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:98 3.a 3:52df5138:::smithi09025264-25:head [write 1454509~727675] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:16.235 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:35:16.235 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 409.63234956700001, 2023-01-15T14:35:16.235 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.44410839299999999, 2023-01-15T14:35:16.235 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:16.235 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:16.235 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:16.235 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:16.235 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:16.235 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 98 2023-01-15T14:35:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:35:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501753+0000", 2023-01-15T14:35:16.236 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.501752+0000", 2023-01-15T14:35:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502223+0000", 2023-01-15T14:35:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000471843 2023-01-15T14:35:16.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502240+0000", 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6663e-05 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502248+0000", 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.5669999999999997e-06 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603236+0000", 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100988439 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603384+0000", 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014844200000000001 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842016+0000", 2023-01-15T14:35:16.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23863163800000001 2023-01-15T14:35:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842267+0000", 2023-01-15T14:35:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025101300000000002 2023-01-15T14:35:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:16.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.842561+0000", 2023-01-15T14:35:16.241 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029432900000000002 2023-01-15T14:35:16.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.359 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.360 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:16.360 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843120+0000", 2023-01-15T14:35:16.360 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055900500000000001 2023-01-15T14:35:16.360 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.360 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:16.361 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843146+0000", 2023-01-15T14:35:16.361 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5273e-05 2023-01-15T14:35:16.361 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.361 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.361 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:16.362 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843512+0000", 2023-01-15T14:35:16.362 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036621599999999999 2023-01-15T14:35:16.362 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.362 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.362 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:16.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943324+0000", 2023-01-15T14:35:16.363 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099812008999999993 2023-01-15T14:35:16.363 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.363 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.363 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:16.363 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945437+0000", 2023-01-15T14:35:16.364 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021126069999999999 2023-01-15T14:35:16.364 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.364 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:16.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945619+0000", 2023-01-15T14:35:16.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018280600000000001 2023-01-15T14:35:16.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.365 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:16.365 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945744+0000", 2023-01-15T14:35:16.365 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012511299999999999 2023-01-15T14:35:16.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.365 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:16.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945861+0000", 2023-01-15T14:35:16.366 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011646000000000001 2023-01-15T14:35:16.366 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:16.366 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:16.366 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:16.366 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.367 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:99 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:16.367 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:35:16.367 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 409.63183931100002, 2023-01-15T14:35:16.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.54355587900000002, 2023-01-15T14:35:16.367 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:16.367 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:16.367 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:16.367 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:16.367 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:16.367 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 99 2023-01-15T14:35:16.368 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.368 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:16.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:16.368 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:35:16.368 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.368 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.368 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.368 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502263+0000", 2023-01-15T14:35:16.381 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.382 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502262+0000", 2023-01-15T14:35:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.383 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502270+0000", 2023-01-15T14:35:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1999999999999994e-06 2023-01-15T14:35:16.385 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:16.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502286+0000", 2023-01-15T14:35:16.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5423000000000001e-05 2023-01-15T14:35:16.388 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.388 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.388 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:16.388 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502295+0000", 2023-01-15T14:35:16.388 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4339999999999998e-06 2023-01-15T14:35:16.389 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.389 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.538 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:16.538 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603467+0000", 2023-01-15T14:35:16.538 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101172225 2023-01-15T14:35:16.538 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.539 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:16.539 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.603585+0000", 2023-01-15T14:35:16.539 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117149 2023-01-15T14:35:16.539 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.539 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.688 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:16.688 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843335+0000", 2023-01-15T14:35:16.689 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23975083799999999 2023-01-15T14:35:16.689 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.689 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:16.689 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843573+0000", 2023-01-15T14:35:16.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023753000000000001 2023-01-15T14:35:16.690 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.690 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:16.690 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.843978+0000", 2023-01-15T14:35:16.690 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040533799999999999 2023-01-15T14:35:16.691 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.691 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.691 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:16.691 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844309+0000", 2023-01-15T14:35:16.691 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000330713 2023-01-15T14:35:16.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.692 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.692 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:16.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844423+0000", 2023-01-15T14:35:16.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114176 2023-01-15T14:35:16.692 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.693 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.693 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:16.693 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.844510+0000", 2023-01-15T14:35:16.693 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7026000000000003e-05 2023-01-15T14:35:16.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.693 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.694 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:16.694 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.943338+0000", 2023-01-15T14:35:16.694 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098827398999999996 2023-01-15T14:35:16.694 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:16.695 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.945520+0000", 2023-01-15T14:35:16.695 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0021824169999999999 2023-01-15T14:35:16.695 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.695 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.696 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:16.696 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045652+0000", 2023-01-15T14:35:16.696 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013193200000001 2023-01-15T14:35:16.696 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.696 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.697 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:16.697 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045733+0000", 2023-01-15T14:35:16.697 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1435999999999998e-05 2023-01-15T14:35:16.697 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.697 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.698 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:16.698 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.045819+0000", 2023-01-15T14:35:16.698 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5292999999999998e-05 2023-01-15T14:35:16.703 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:16.703 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:16.703 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:16.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.704 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:100 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:16.704 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:35:16.704 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 409.63179138800001, 2023-01-15T14:35:16.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.64425290599999996, 2023-01-15T14:35:16.705 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:16.705 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:16.705 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:16.705 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:16.705 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:16.705 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 100 2023-01-15T14:35:16.705 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.706 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:16.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:16.706 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:35:16.706 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:16.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502311+0000", 2023-01-15T14:35:16.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502310+0000", 2023-01-15T14:35:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:16.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:16.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502316+0000", 2023-01-15T14:35:16.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7470000000000002e-06 2023-01-15T14:35:16.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:16.711 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502332+0000", 2023-01-15T14:35:16.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6866000000000001e-05 2023-01-15T14:35:16.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.711 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.502343+0000", 2023-01-15T14:35:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0913e-05 2023-01-15T14:35:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:16.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703594+0000", 2023-01-15T14:35:16.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20125087999999999 2023-01-15T14:35:16.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:16.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.703751+0000", 2023-01-15T14:35:16.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0001573 2023-01-15T14:35:16.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.850 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:16.850 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845262+0000", 2023-01-15T14:35:16.850 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.141510521 2023-01-15T14:35:16.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.845391+0000", 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012872299999999999 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146051+0000", 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.300660023 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:16.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146264+0000", 2023-01-15T14:35:16.852 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212754 2023-01-15T14:35:16.852 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.852 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:16.853 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.146564+0000", 2023-01-15T14:35:16.853 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030010200000000001 2023-01-15T14:35:16.853 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:16.853 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:16.853 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:16.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.854 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:203 3.a 3:52df5138:::smithi09025264-25:head [write 440350~460568] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:16.854 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:35:16.854 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 408.58098799300001, 2023-01-15T14:35:16.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40523678899999999, 2023-01-15T14:35:16.854 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:16.855 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:16.855 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:16.855 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:16.855 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:16.855 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 203 2023-01-15T14:35:16.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.855 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:16.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:16.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:35:16.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:16.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553114+0000", 2023-01-15T14:35:16.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.857 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:16.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553111+0000", 2023-01-15T14:35:16.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:16.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.857 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:16.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553483+0000", 2023-01-15T14:35:16.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037153499999999999 2023-01-15T14:35:16.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.858 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:16.858 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553517+0000", 2023-01-15T14:35:16.858 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4687000000000002e-05 2023-01-15T14:35:16.858 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:16.861 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553537+0000", 2023-01-15T14:35:16.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.967e-05 2023-01-15T14:35:16.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.861 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:16.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553683+0000", 2023-01-15T14:35:16.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000146341 2023-01-15T14:35:16.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:16.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553880+0000", 2023-01-15T14:35:16.864 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019671900000000001 2023-01-15T14:35:16.864 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.866 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:16.866 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554219+0000", 2023-01-15T14:35:16.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033893299999999998 2023-01-15T14:35:16.866 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.866 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.866 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:16.866 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554536+0000", 2023-01-15T14:35:16.866 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000316576 2023-01-15T14:35:16.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:16.867 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554562+0000", 2023-01-15T14:35:16.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5905999999999999e-05 2023-01-15T14:35:16.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.867 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.867 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:16.867 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554801+0000", 2023-01-15T14:35:16.867 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000239791 2023-01-15T14:35:16.867 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:16.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555274+0000", 2023-01-15T14:35:16.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047294500000000001 2023-01-15T14:35:16.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.868 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:16.868 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757267+0000", 2023-01-15T14:35:16.868 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20199229399999999 2023-01-15T14:35:16.868 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.868 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.869 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:16.869 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958174+0000", 2023-01-15T14:35:16.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20090739499999999 2023-01-15T14:35:16.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.869 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.869 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:16.869 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958260+0000", 2023-01-15T14:35:16.869 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6316999999999994e-05 2023-01-15T14:35:16.869 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.869 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.869 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958351+0000", 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.0573000000000002e-05 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:204 3.a 3:52df5138:::smithi09025264-25:head [write 1309629~542666] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 408.58052512500001, 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40500913999999999, 2023-01-15T14:35:16.870 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:16.871 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:16.871 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:16.871 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:16.871 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:16.871 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 204 2023-01-15T14:35:16.871 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.871 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:16.871 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.871 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:16.871 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:35:16.871 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:16.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553577+0000", 2023-01-15T14:35:16.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:16.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.872 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.872 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:16.872 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.553575+0000", 2023-01-15T14:35:16.872 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:16.872 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.873 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.873 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:16.873 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554126+0000", 2023-01-15T14:35:16.873 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055090499999999997 2023-01-15T14:35:16.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:16.876 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554150+0000", 2023-01-15T14:35:16.876 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3387000000000002e-05 2023-01-15T14:35:16.876 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:16.876 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:16.876 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:17.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554165+0000", 2023-01-15T14:35:17.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5116000000000001e-05 2023-01-15T14:35:17.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:17.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554626+0000", 2023-01-15T14:35:17.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046093499999999999 2023-01-15T14:35:17.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.014 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:17.014 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554738+0000", 2023-01-15T14:35:17.014 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112446 2023-01-15T14:35:17.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:17.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554898+0000", 2023-01-15T14:35:17.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016003100000000001 2023-01-15T14:35:17.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.015 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.015 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:17.015 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555182+0000", 2023-01-15T14:35:17.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028382700000000001 2023-01-15T14:35:17.015 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:17.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555199+0000", 2023-01-15T14:35:17.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6969e-05 2023-01-15T14:35:17.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.016 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.016 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:17.016 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555449+0000", 2023-01-15T14:35:17.016 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000250265 2023-01-15T14:35:17.016 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.172 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.172 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:17.172 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.555574+0000", 2023-01-15T14:35:17.172 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124628 2023-01-15T14:35:17.172 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.172 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:17.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757316+0000", 2023-01-15T14:35:17.176 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20174214300000001 2023-01-15T14:35:17.176 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.176 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.176 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:17.176 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958460+0000", 2023-01-15T14:35:17.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201144464 2023-01-15T14:35:17.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:17.177 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958534+0000", 2023-01-15T14:35:17.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3380000000000006e-05 2023-01-15T14:35:17.177 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.177 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.177 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:17.177 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958586+0000", 2023-01-15T14:35:17.177 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2370000000000002e-05 2023-01-15T14:35:17.178 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:17.178 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:17.178 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:17.178 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.178 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.178 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:205 3.a 3:52df5138:::smithi09025264-25:head [write 2323973~516377] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:17.178 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:35:17.178 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 408.57990703000002, 2023-01-15T14:35:17.178 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404601398, 2023-01-15T14:35:17.178 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:17.179 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:17.179 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:17.179 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:17.179 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:17.179 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 205 2023-01-15T14:35:17.179 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.179 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:17.179 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.179 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:17.179 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:35:17.179 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:17.184 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:17.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554195+0000", 2023-01-15T14:35:17.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:17.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.185 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.185 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:17.185 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554193+0000", 2023-01-15T14:35:17.185 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:17.185 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.186 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.186 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554850+0000", 2023-01-15T14:35:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065674799999999997 2023-01-15T14:35:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.193 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.196 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:17.197 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.554873+0000", 2023-01-15T14:35:17.197 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2935999999999999e-05 2023-01-15T14:35:17.197 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.197 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.197 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:17.197 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.654962+0000", 2023-01-15T14:35:17.198 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088949 2023-01-15T14:35:17.198 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.198 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.198 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:17.198 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655107+0000", 2023-01-15T14:35:17.198 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144759 2023-01-15T14:35:17.199 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.199 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.199 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:17.199 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655373+0000", 2023-01-15T14:35:17.199 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026684899999999999 2023-01-15T14:35:17.199 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.199 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655720+0000", 2023-01-15T14:35:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034601899999999998 2023-01-15T14:35:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.200 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656168+0000", 2023-01-15T14:35:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044851899999999998 2023-01-15T14:35:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.201 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656190+0000", 2023-01-15T14:35:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1817000000000001e-05 2023-01-15T14:35:17.202 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.205 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.205 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:17.205 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.656574+0000", 2023-01-15T14:35:17.206 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038404200000000002 2023-01-15T14:35:17.206 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.206 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.311 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.657649+0000", 2023-01-15T14:35:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010746670000000001 2023-01-15T14:35:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.757539+0000", 2023-01-15T14:35:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099890899000000005 2023-01-15T14:35:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958683+0000", 2023-01-15T14:35:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20114332400000001 2023-01-15T14:35:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958746+0000", 2023-01-15T14:35:17.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3509000000000005e-05 2023-01-15T14:35:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.958796+0000", 2023-01-15T14:35:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0176999999999997e-05 2023-01-15T14:35:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.314 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:358 3.a 3:52df5138:::smithi09025264-25:head [write 709262~697258] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:35:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:35:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 396.08331949699999, 2023-01-15T14:35:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50589529799999999, 2023-01-15T14:35:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:17.316 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 358 2023-01-15T14:35:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:17.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:35:17.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050783+0000", 2023-01-15T14:35:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:17.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.050780+0000", 2023-01-15T14:35:17.340 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051369+0000", 2023-01-15T14:35:17.341 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058926200000000001 2023-01-15T14:35:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051390+0000", 2023-01-15T14:35:17.342 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1426000000000001e-05 2023-01-15T14:35:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.343 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051403+0000", 2023-01-15T14:35:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:35:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:17.344 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051520+0000", 2023-01-15T14:35:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000117318 2023-01-15T14:35:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051785+0000", 2023-01-15T14:35:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000265018 2023-01-15T14:35:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052173+0000", 2023-01-15T14:35:17.345 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038734799999999998 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052705+0000", 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053209899999999996 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052727+0000", 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2362e-05 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.346 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053205+0000", 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00047787200000000001 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055711+0000", 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0025054880000000002 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:17.347 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.255549+0000", 2023-01-15T14:35:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199838285 2023-01-15T14:35:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556501+0000", 2023-01-15T14:35:17.348 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30095197499999998 2023-01-15T14:35:17.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:17.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556613+0000", 2023-01-15T14:35:17.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111799 2023-01-15T14:35:17.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:17.472 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556678+0000", 2023-01-15T14:35:17.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5220000000000002e-05 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:359 3.a 3:52df5138:::smithi09025264-25:head [write 2012635~572018] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 396.08266890300001, 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50546442499999999, 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:17.473 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:17.474 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:17.474 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:17.474 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 359 2023-01-15T14:35:17.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.474 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:17.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:17.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:35:17.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:17.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:17.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051433+0000", 2023-01-15T14:35:17.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:17.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:17.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051432+0000", 2023-01-15T14:35:17.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:17.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:17.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051808+0000", 2023-01-15T14:35:17.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000376074 2023-01-15T14:35:17.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.501 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.501 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:17.501 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051834+0000", 2023-01-15T14:35:17.501 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6018999999999999e-05 2023-01-15T14:35:17.502 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.502 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:17.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051869+0000", 2023-01-15T14:35:17.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4839999999999998e-05 2023-01-15T14:35:17.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.507 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:17.508 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052857+0000", 2023-01-15T14:35:17.508 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00098761399999999994 2023-01-15T14:35:17.508 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.508 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.508 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:17.508 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053071+0000", 2023-01-15T14:35:17.508 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000214112 2023-01-15T14:35:17.509 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.666 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.666 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:17.666 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.053876+0000", 2023-01-15T14:35:17.667 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000804873 2023-01-15T14:35:17.667 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.667 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.667 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:17.671 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054460+0000", 2023-01-15T14:35:17.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058436100000000004 2023-01-15T14:35:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054486+0000", 2023-01-15T14:35:17.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069999999999999e-05 2023-01-15T14:35:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.799 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:17.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054773+0000", 2023-01-15T14:35:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028716700000000001 2023-01-15T14:35:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055736+0000", 2023-01-15T14:35:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00096295500000000004 2023-01-15T14:35:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.355784+0000", 2023-01-15T14:35:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30004825899999998 2023-01-15T14:35:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:17.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556757+0000", 2023-01-15T14:35:17.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20097241199999999 2023-01-15T14:35:17.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:17.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556830+0000", 2023-01-15T14:35:17.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3633000000000005e-05 2023-01-15T14:35:17.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:17.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556898+0000", 2023-01-15T14:35:17.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7212999999999995e-05 2023-01-15T14:35:17.824 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:17.824 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:17.824 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:17.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.825 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:360 3.a 3:52df5138:::smithi09025264-25:head [write 3046388~62874] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:35:17.825 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:35:17.825 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 396.082216147, 2023-01-15T14:35:17.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50519877000000002, 2023-01-15T14:35:17.826 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:17.826 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:17.826 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:17.826 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:17.826 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:17.826 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 360 2023-01-15T14:35:17.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.827 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:17.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:17.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:35:17.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:17.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.955 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:17.955 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051886+0000", 2023-01-15T14:35:17.955 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:17.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.956 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:17.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051885+0000", 2023-01-15T14:35:17.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:17.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.956 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:17.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051935+0000", 2023-01-15T14:35:17.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9826e-05 2023-01-15T14:35:17.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:17.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051958+0000", 2023-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3003e-05 2023-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051965+0000", 2023-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6129999999999998e-06 2023-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054655+0000", 2023-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0026896860000000002 2023-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.054869+0000", 2023-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021348600000000001 2023-01-15T14:35:17.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055198+0000", 2023-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032968199999999998 2023-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.972 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055605+0000", 2023-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00040634200000000002 2023-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055622+0000", 2023-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7020999999999998e-05 2023-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:17.974 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055702+0000", 2023-01-15T14:35:17.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0835999999999997e-05 2023-01-15T14:35:17.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.974 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.974 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:17.975 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055754+0000", 2023-01-15T14:35:17.975 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1175999999999999e-05 2023-01-15T14:35:17.976 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.455996+0000", 2023-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40024198599999999 2023-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556952+0000", 2023-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10095644400000001 2023-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.977 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.978 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:17.978 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557019+0000", 2023-01-15T14:35:17.987 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6746000000000001e-05 2023-01-15T14:35:17.987 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.988 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.988 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:17.988 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.557085+0000", 2023-01-15T14:35:17.988 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5885999999999999e-05 2023-01-15T14:35:17.988 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:17.988 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:17.988 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:17.988 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.988 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.989 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:361 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (56) in=7b,truncate 3109262] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:35:17.989 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:35:17.989 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 396.08212624100003, 2023-01-15T14:35:17.989 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60469846000000005, 2023-01-15T14:35:17.989 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:17.989 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:17.989 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:17.990 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:17.990 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:17.991 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 361 2023-01-15T14:35:17.991 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.992 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:17.992 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.992 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:17.992 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:35:17.993 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:17.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.993 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:17.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051976+0000", 2023-01-15T14:35:17.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:17.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:17.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:17.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:17.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051975+0000", 2023-01-15T14:35:17.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:17.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:18.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051980+0000", 2023-01-15T14:35:18.124 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2170000000000002e-06 2023-01-15T14:35:18.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051989+0000", 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5299999999999996e-06 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.051995+0000", 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.1369999999999999e-06 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.055830+0000", 2023-01-15T14:35:18.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0038348549999999999 2023-01-15T14:35:18.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.127 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:18.127 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056101+0000", 2023-01-15T14:35:18.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027095800000000002 2023-01-15T14:35:18.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.127 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:18.127 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056560+0000", 2023-01-15T14:35:18.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045868599999999998 2023-01-15T14:35:18.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:18.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.056954+0000", 2023-01-15T14:35:18.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039487600000000001 2023-01-15T14:35:18.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:18.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057004+0000", 2023-01-15T14:35:18.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.9490000000000002e-05 2023-01-15T14:35:18.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:18.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.057072+0000", 2023-01-15T14:35:18.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8502000000000002e-05 2023-01-15T14:35:18.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:18.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.158224+0000", 2023-01-15T14:35:18.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101151422 2023-01-15T14:35:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.556201+0000", 2023-01-15T14:35:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39797714699999998 2023-01-15T14:35:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:18.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656421+0000", 2023-01-15T14:35:18.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100219549 2023-01-15T14:35:18.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:18.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656515+0000", 2023-01-15T14:35:18.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4048999999999993e-05 2023-01-15T14:35:18.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:18.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656674+0000", 2023-01-15T14:35:18.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000159696 2023-01-15T14:35:18.134 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.155 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:18.155 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.156 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:362 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e34)", 2023-01-15T14:35:18.156 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:35:18.156 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 396.08209881099998, 2023-01-15T14:35:18.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60517504499999997, 2023-01-15T14:35:18.156 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:18.156 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 362 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:18.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:35:18.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:18.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052003+0000", 2023-01-15T14:35:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052007+0000", 2023-01-15T14:35:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.9759999999999997e-06 2023-01-15T14:35:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052016+0000", 2023-01-15T14:35:18.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4069999999999999e-06 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.052022+0000", 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4670000000000001e-06 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157159+0000", 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.105136825 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.162 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:18.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.157325+0000", 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000165262 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656814+0000", 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49948963200000002 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.656993+0000", 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000178371 2023-01-15T14:35:18.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:40.657178+0000", 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185825 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:601 3.a 3:5ba385f1:::smithi09025264-44:head [setxattr _header (57) in=7b,truncate 3663893] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54)", 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:35:18.221 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 376.19177617499997, 2023-01-15T14:35:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.404742724, 2023-01-15T14:35:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:18.222 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 601 2023-01-15T14:35:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:18.224 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.312 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:18.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:35:18.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:18.313 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942326+0000", 2023-01-15T14:35:18.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.313 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.313 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.313 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:18.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942323+0000", 2023-01-15T14:35:18.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:18.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:18.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942341+0000", 2023-01-15T14:35:18.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7659000000000002e-05 2023-01-15T14:35:18.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:59.942365+0000", 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4256999999999998e-05 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.042495+0000", 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10013043200000001 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.144809+0000", 2023-01-15T14:35:18.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102314184 2023-01-15T14:35:18.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:18.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145098+0000", 2023-01-15T14:35:18.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000289036 2023-01-15T14:35:18.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:18.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145363+0000", 2023-01-15T14:35:18.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026449299999999998 2023-01-15T14:35:18.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:18.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145657+0000", 2023-01-15T14:35:18.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029433600000000002 2023-01-15T14:35:18.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:18.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145678+0000", 2023-01-15T14:35:18.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0939000000000001e-05 2023-01-15T14:35:18.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:18.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.145853+0000", 2023-01-15T14:35:18.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017511800000000001 2023-01-15T14:35:18.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:18.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146557+0000", 2023-01-15T14:35:18.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00070337899999999996 2023-01-15T14:35:18.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:18.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.146845+0000", 2023-01-15T14:35:18.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028790599999999999 2023-01-15T14:35:18.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:18.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346819+0000", 2023-01-15T14:35:18.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.199974661 2023-01-15T14:35:18.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:18.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.346919+0000", 2023-01-15T14:35:18.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9234999999999994e-05 2023-01-15T14:35:18.365 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.366 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.366 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:18.366 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:00.347069+0000", 2023-01-15T14:35:18.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000150128 2023-01-15T14:35:18.367 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.367 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:18.367 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.367 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.367 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.367 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1153 3.a 3:5919d83e:::smithi09025264-13:head [write 1566383~595717] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:35:18.367 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:35:18.367 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 330.34355112100002, 2023-01-15T14:35:18.367 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.38805106299999997, 2023-01-15T14:35:18.367 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:18.368 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:18.368 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:18.368 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:18.368 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:18.368 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1153 2023-01-15T14:35:18.369 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.369 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:18.369 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.369 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:18.369 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:35:18.369 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.374 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.374 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:18.374 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790551+0000", 2023-01-15T14:35:18.374 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.374 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:18.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.790531+0000", 2023-01-15T14:35:18.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999795 2023-01-15T14:35:18.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.375 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:18.375 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791189+0000", 2023-01-15T14:35:18.375 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065897000000000004 2023-01-15T14:35:18.375 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.375 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:18.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791215+0000", 2023-01-15T14:35:18.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5700000000000001e-05 2023-01-15T14:35:18.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.376 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:18.376 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791232+0000", 2023-01-15T14:35:18.376 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6546e-05 2023-01-15T14:35:18.376 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.376 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791392+0000", 2023-01-15T14:35:18.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016051699999999999 2023-01-15T14:35:18.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.377 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.377 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:18.377 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.791408+0000", 2023-01-15T14:35:18.377 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5566999999999999e-05 2023-01-15T14:35:18.377 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.974825+0000", 2023-01-15T14:35:18.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18341675599999999 2023-01-15T14:35:18.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:18.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975000+0000", 2023-01-15T14:35:18.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017548299999999999 2023-01-15T14:35:18.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:18.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975255+0000", 2023-01-15T14:35:18.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254658 2023-01-15T14:35:18.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:18.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975914+0000", 2023-01-15T14:35:18.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00065881800000000003 2023-01-15T14:35:18.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:18.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.975931+0000", 2023-01-15T14:35:18.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7523000000000001e-05 2023-01-15T14:35:18.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:18.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976267+0000", 2023-01-15T14:35:18.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033545299999999999 2023-01-15T14:35:18.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:18.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978320+0000", 2023-01-15T14:35:18.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020537110000000002 2023-01-15T14:35:18.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.477 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:18.477 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978647+0000", 2023-01-15T14:35:18.478 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032660700000000003 2023-01-15T14:35:18.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:18.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178434+0000", 2023-01-15T14:35:18.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19978705699999999 2023-01-15T14:35:18.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:18.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178527+0000", 2023-01-15T14:35:18.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3523999999999994e-05 2023-01-15T14:35:18.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.178602+0000", 2023-01-15T14:35:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4622000000000005e-05 2023-01-15T14:35:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1154 3.a 3:5919d83e:::smithi09025264-13:head [write 2888822~102824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:35:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:35:18.616 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 330.33820979400002, 2023-01-15T14:35:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48318778800000001, 2023-01-15T14:35:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1154 2023-01-15T14:35:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.617 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795892+0000", 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795890+0000", 2023-01-15T14:35:18.618 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.795967+0000", 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6304999999999995e-05 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796000+0000", 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.3593000000000002e-05 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.619 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796019+0000", 2023-01-15T14:35:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8955999999999999e-05 2023-01-15T14:35:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796189+0000", 2023-01-15T14:35:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017032700000000001 2023-01-15T14:35:18.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796208+0000", 2023-01-15T14:35:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8389e-05 2023-01-15T14:35:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976335+0000", 2023-01-15T14:35:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18012763100000001 2023-01-15T14:35:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976487+0000", 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015173000000000001 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.976763+0000", 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027597400000000001 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:18.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977085+0000", 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032233400000000002 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977103+0000", 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7727000000000002e-05 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977238+0000", 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013525399999999999 2023-01-15T14:35:18.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978347+0000", 2023-01-15T14:35:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011084350000000001 2023-01-15T14:35:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.624 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.642 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:18.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.078223+0000", 2023-01-15T14:35:18.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099875830999999998 2023-01-15T14:35:18.642 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.642 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.642 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:18.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.278897+0000", 2023-01-15T14:35:18.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20067421599999999 2023-01-15T14:35:18.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:18.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279021+0000", 2023-01-15T14:35:18.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000124354 2023-01-15T14:35:18.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:18.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279080+0000", 2023-01-15T14:35:18.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8811999999999999e-05 2023-01-15T14:35:18.646 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.646 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:18.646 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.647 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1155 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 2991646] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:35:18.647 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:35:18.647 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 330.33803677499998, 2023-01-15T14:35:18.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.48334445999999998, 2023-01-15T14:35:18.647 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:18.648 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:18.648 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:18.648 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:18.648 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:18.648 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1155 2023-01-15T14:35:18.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.649 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:18.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:18.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:35:18.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.650 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.650 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.650 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:18.650 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796065+0000", 2023-01-15T14:35:18.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.651 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.651 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.651 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:18.651 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796064+0000", 2023-01-15T14:35:18.651 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:18.651 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.652 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.652 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:18.652 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796074+0000", 2023-01-15T14:35:18.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:35:18.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:18.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796096+0000", 2023-01-15T14:35:18.654 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2422999999999999e-05 2023-01-15T14:35:18.654 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.655 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:18.655 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796109+0000", 2023-01-15T14:35:18.655 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2792999999999999e-05 2023-01-15T14:35:18.655 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.656 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.657 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.659 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796274+0000", 2023-01-15T14:35:18.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000164976 2023-01-15T14:35:18.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.661 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:18.661 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796283+0000", 2023-01-15T14:35:18.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.9539999999999993e-06 2023-01-15T14:35:18.662 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.662 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.662 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.663 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977269+0000", 2023-01-15T14:35:18.663 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18098651700000001 2023-01-15T14:35:18.663 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.663 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.663 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:18.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977438+0000", 2023-01-15T14:35:18.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168913 2023-01-15T14:35:18.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:18.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.977865+0000", 2023-01-15T14:35:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042707500000000001 2023-01-15T14:35:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978241+0000", 2023-01-15T14:35:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037593000000000001 2023-01-15T14:35:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:18.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978253+0000", 2023-01-15T14:35:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1817000000000001e-05 2023-01-15T14:35:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978288+0000", 2023-01-15T14:35:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4586999999999999e-05 2023-01-15T14:35:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:18.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978370+0000", 2023-01-15T14:35:18.826 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1712999999999995e-05 2023-01-15T14:35:18.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.826 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.826 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:18.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279124+0000", 2023-01-15T14:35:18.827 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30075418799999998 2023-01-15T14:35:18.827 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.827 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.827 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:18.827 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279208+0000", 2023-01-15T14:35:18.828 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4500999999999996e-05 2023-01-15T14:35:18.828 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.828 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.828 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:18.828 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279294+0000", 2023-01-15T14:35:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5680999999999995e-05 2023-01-15T14:35:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:18.829 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.279410+0000", 2023-01-15T14:35:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115878 2023-01-15T14:35:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.830 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1156 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e99)", 2023-01-15T14:35:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:35:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 330.33792525000001, 2023-01-15T14:35:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.58375825000000003, 2023-01-15T14:35:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:18.831 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1156 2023-01-15T14:35:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:18.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:35:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796177+0000", 2023-01-15T14:35:18.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796176+0000", 2023-01-15T14:35:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:18.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796182+0000", 2023-01-15T14:35:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6799999999999998e-06 2023-01-15T14:35:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796200+0000", 2023-01-15T14:35:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8136999999999999e-05 2023-01-15T14:35:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796217+0000", 2023-01-15T14:35:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7036000000000001e-05 2023-01-15T14:35:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796337+0000", 2023-01-15T14:35:18.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120674 2023-01-15T14:35:18.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.796345+0000", 2023-01-15T14:35:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4540000000000001e-06 2023-01-15T14:35:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978423+0000", 2023-01-15T14:35:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18207789599999999 2023-01-15T14:35:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.840 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:45.978577+0000", 2023-01-15T14:35:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000153805 2023-01-15T14:35:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.841 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379596+0000", 2023-01-15T14:35:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.401019348 2023-01-15T14:35:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:18.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379754+0000", 2023-01-15T14:35:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015840400000000001 2023-01-15T14:35:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.379935+0000", 2023-01-15T14:35:18.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018081299999999999 2023-01-15T14:35:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.844 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1160 3.a 3:52df5138:::smithi09025264-25:head [write 2632598~375855] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:35:18.845 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:35:18.846 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 329.37911866299999, 2023-01-15T14:35:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.398337832, 2023-01-15T14:35:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:18.847 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:18.848 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:18.849 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1160 2023-01-15T14:35:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:18.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:35:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.851 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754983+0000", 2023-01-15T14:35:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:18.852 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.754981+0000", 2023-01-15T14:35:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755360+0000", 2023-01-15T14:35:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037899899999999998 2023-01-15T14:35:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755392+0000", 2023-01-15T14:35:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1977000000000002e-05 2023-01-15T14:35:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755407+0000", 2023-01-15T14:35:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4446e-05 2023-01-15T14:35:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:18.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949294+0000", 2023-01-15T14:35:18.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.193887483 2023-01-15T14:35:18.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:18.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949478+0000", 2023-01-15T14:35:18.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018400900000000001 2023-01-15T14:35:18.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:18.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.949904+0000", 2023-01-15T14:35:18.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042552200000000001 2023-01-15T14:35:18.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:18.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950531+0000", 2023-01-15T14:35:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062732399999999996 2023-01-15T14:35:18.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.000 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:19.006 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.950552+0000", 2023-01-15T14:35:19.007 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1229999999999998e-05 2023-01-15T14:35:19.007 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.007 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.008 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:19.008 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951094+0000", 2023-01-15T14:35:19.008 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054169000000000005 2023-01-15T14:35:19.009 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.009 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.009 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:19.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951256+0000", 2023-01-15T14:35:19.010 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016176299999999999 2023-01-15T14:35:19.010 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.010 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.010 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:19.010 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052665+0000", 2023-01-15T14:35:19.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101409257 2023-01-15T14:35:19.011 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.011 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.011 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:19.011 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153132+0000", 2023-01-15T14:35:19.011 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100466783 2023-01-15T14:35:19.012 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.012 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.012 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:19.012 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153223+0000", 2023-01-15T14:35:19.012 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.1415000000000005e-05 2023-01-15T14:35:19.013 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.013 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.013 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:19.013 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153321+0000", 2023-01-15T14:35:19.013 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8181000000000001e-05 2023-01-15T14:35:19.013 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:19.014 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:19.014 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:19.014 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.014 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.014 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1161 3.a 3:52df5138:::smithi09025264-25:head [setxattr _header (57) in=7b,truncate 3008453] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:35:19.014 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:35:19.015 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 329.37867507499999, 2023-01-15T14:35:19.015 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49840109900000001, 2023-01-15T14:35:19.016 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:19.018 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1161 2023-01-15T14:35:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.019 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.020 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:35:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:19.021 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755427+0000", 2023-01-15T14:35:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755426+0000", 2023-01-15T14:35:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755433+0000", 2023-01-15T14:35:19.022 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.793e-06 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755448+0000", 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5047e-05 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755455+0000", 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.3590000000000002e-06 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.023 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951466+0000", 2023-01-15T14:35:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.196011038 2023-01-15T14:35:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.024 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951611+0000", 2023-01-15T14:35:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000145184 2023-01-15T14:35:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,4", 2023-01-15T14:35:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.951921+0000", 2023-01-15T14:35:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030989100000000002 2023-01-15T14:35:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.025 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952187+0000", 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026623100000000002 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952207+0000", 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9510000000000001e-05 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.952271+0000", 2023-01-15T14:35:19.026 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4511999999999995e-05 2023-01-15T14:35:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.052716+0000", 2023-01-15T14:35:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10044497300000001 2023-01-15T14:35:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253531+0000", 2023-01-15T14:35:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20081490699999999 2023-01-15T14:35:19.027 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:19.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253635+0000", 2023-01-15T14:35:19.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000103838 2023-01-15T14:35:19.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.028 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.028 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:19.028 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253727+0000", 2023-01-15T14:35:19.028 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2298000000000003e-05 2023-01-15T14:35:19.028 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.029 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.029 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:19.029 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253828+0000", 2023-01-15T14:35:19.029 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000100655 2023-01-15T14:35:19.029 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:19.029 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:19.029 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1162 3.a 3:52df5138:::smithi09025264-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e100)", 2023-01-15T14:35:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:35:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 329.37863568, 2023-01-15T14:35:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.498775839, 2023-01-15T14:35:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:19.030 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1162 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:35:19.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755466+0000", 2023-01-15T14:35:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755470+0000", 2023-01-15T14:35:19.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0570000000000003e-06 2023-01-15T14:35:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755484+0000", 2023-01-15T14:35:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4229999999999999e-05 2023-01-15T14:35:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:46.755492+0000", 2023-01-15T14:35:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6070000000000002e-06 2023-01-15T14:35:19.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.152842+0000", 2023-01-15T14:35:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.39734988799999998 2023-01-15T14:35:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.153014+0000", 2023-01-15T14:35:19.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017236000000000001 2023-01-15T14:35:19.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:19.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.253921+0000", 2023-01-15T14:35:19.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100907497 2023-01-15T14:35:19.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:19.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254060+0000", 2023-01-15T14:35:19.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013815400000000001 2023-01-15T14:35:19.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:19.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:19.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:19.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:47.254242+0000", 2023-01-15T14:35:19.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000182726 2023-01-15T14:35:19.156 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:19.156 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:19.156 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:19.156 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:19.156 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:19.156 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:35:19.157 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:35:19.158 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:16.173+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:19.158 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:19.159 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:35:19.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:35:19.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:35:19.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:35:19.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:35:19.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:35:19.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:35:19.160 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:35:19.165 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:19.157+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:19.166 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:35:19.169 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:35:19.169 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":633,"stamp":"2023-01-15T14:35:16.159827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":437415051,"num_objects":207,"num_object_clones":153,"num_object_copies":621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":120,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":207,"num_whiteouts":18,"num_read":1144,"num_read_kb":1158161,"num_write":924,"num_write_kb":327809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":746,"num_bytes_recovered":491068220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21366,"ondisk_log_size":21366,"up":375,"acting":364,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":406,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":7386796,"kb_used_data":648562604,"kb_used_omap":92764,"kb_used_meta":0,"kb_avail":648508864,"statfs":{"total":671637155840,"available":664073076736,"internally_reserved":55029760,"allocated":664128106496,"data_stored":664128106496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":94991354,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":13,"commit_latency_ns":0,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19666281,"num_objects":8,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-70,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":53,"num_read_kb":58986,"num_write":59,"num_write_kb":21201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":48779232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.210140"},"pg_stats":[{"pgid":"3.73","version":"204'275","reported_seq":1407,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.956897+0000","last_change":"2023-01-15T14:35:12.956897+0000","last_active":"2023-01-15T14:35:12.956897+0000","last_peered":"2023-01-15T14:35:12.956897+0000","last_clean":"2023-01-15T14:35:12.956897+0000","last_became_active":"2023-01-15T14:35:02.250324+0000","last_became_peered":"2023-01-15T14:35:02.250324+0000","last_unstale":"2023-01-15T14:35:12.956897+0000","last_undegraded":"2023-01-15T14:35:12.956897+0000","last_fullsized":"2023-01-15T14:35:12.956897+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.076880229999999994,"stat_sum":{"num_bytes":3485477,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":11189,"num_write":7,"num_write_kb":3286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"},{"start":"139","length":"1"}]},{"pgid":"3.72","version":"394'169","reported_seq":1056,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.065604+0000","last_change":"2023-01-15T14:35:12.065604+0000","last_active":"2023-01-15T14:35:12.065604+0000","last_peered":"2023-01-15T14:35:12.065604+0000","last_clean":"2023-01-15T14:35:12.065604+0000","last_became_active":"2023-01-15T14:35:02.148756+0000","last_became_peered":"2023-01-15T14:35:02.148756+0000","last_unstale":"2023-01-15T14:35:12.065604+0000","last_undegraded":"2023-01-15T14:35:12.065604+0000","last_fullsized":"2023-01-15T14:35:12.065604+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.97458847699999995,"stat_sum":{"num_bytes":2663943,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3524,"num_write":1,"num_write_kb":1428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2663943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.71","version":"204'150","reported_seq":923,"reported_epoch":423,"state":"active+clean","last_fresh":"2023-01-15T14:35:12.968880+0000","last_change":"2023-01-15T14:35:12.968880+0000","last_active":"2023-01-15T14:35:12.968880+0000","last_peered":"2023-01-15T14:35:12.968880+0000","last_clean":"2023-01-15T14:35:12.968880+0000","last_became_active":"2023-01-15T14:35:02.151543+0000","last_became_peered":"2023-01-15T14:35:02.151543+0000","last_unstale":"2023-01-15T14:35:12.968880+0000","last_undegraded":"2023-01-15T14:35:12.968880+0000","last_fullsized":"2023-01-15T14:35:12.968880+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.0033666540000000002,"stat_sum":{"num_bytes":3558907,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10725,"num_write":4,"num_write_kb":3090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3605110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.70","version":"206'252","reported_seq":1188,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.056446+0000","last_change":"2023-01-15T14:35:13.056446+0000","last_active":"2023-01-15T14:35:13.056446+0000","last_peered":"2023-01-15T14:35:13.056446+0000","last_clean":"2023-01-15T14:35:13.056446+0000","last_became_active":"2023-01-15T14:35:07.457483+0000","last_became_peered":"2023-01-15T14:35:07.457483+0000","last_unstale":"2023-01-15T14:35:13.056446+0000","last_undegraded":"2023-01-15T14:35:13.056446+0000","last_fullsized":"2023-01-15T14:35:13.056446+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:34:36.078678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:36.078678+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.158842567,"objects_trimmed":0,"snaptrim_duration":0.076250391000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"},{"start":"139","length":"1"}]},{"pgid":"3.6f","version":"155'145","reported_seq":1124,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:11.967495+0000","last_change":"2023-01-15T14:35:11.967495+0000","last_active":"2023-01-15T14:35:11.967495+0000","last_peered":"2023-01-15T14:35:11.967495+0000","last_clean":"2023-01-15T14:35:11.967495+0000","last_became_active":"2023-01-15T14:35:02.254791+0000","last_became_peered":"2023-01-15T14:35:02.254791+0000","last_unstale":"2023-01-15T14:35:11.967495+0000","last_undegraded":"2023-01-15T14:35:11.967495+0000","last_fullsized":"2023-01-15T14:35:11.967495+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.60462007600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.6e","version":"139'38","reported_seq":59,"reported_epoch":423,"state":"active+clean","last_fresh":"2023-01-15T14:35:13.175353+0000","last_change":"2023-01-15T14:35:13.175353+0000","last_active":"2023-01-15T14:35:13.175353+0000","last_peered":"2023-01-15T14:35:13.175353+0000","last_clean":"2023-01-15T14:35:13.175353+0000","last_became_active":"2023-01-15T14:34:59.097277+0000","last_became_peered":"2023-01-15T14:34:59.097277+0000","last_unstale":"2023-01-15T14:35:13.175353+0000","last_undegraded":"2023-01-15T14:35:13.175353+0000","last_fullsized":"2023-01-15T14:35:13.175353+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.107590932,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.6d","version":"168'156","reported_seq":812,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.441720+0000","last_change":"2023-01-15T14:35:10.441720+0000","last_active":"2023-01-15T14:35:10.441720+0000","last_peered":"2023-01-15T14:35:10.441720+0000","last_clean":"2023-01-15T14:35:10.441720+0000","last_became_active":"2023-01-15T14:35:07.163676+0000","last_became_peered":"2023-01-15T14:35:07.163676+0000","last_unstale":"2023-01-15T14:35:10.441720+0000","last_undegraded":"2023-01-15T14:35:10.441720+0000","last_fullsized":"2023-01-15T14:35:10.441720+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.30494026899999999,"stat_sum":{"num_bytes":1516832,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2710,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.6c","version":"141'46","reported_seq":64,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.861837+0000","last_change":"2023-01-15T14:35:12.861837+0000","last_active":"2023-01-15T14:35:12.861837+0000","last_peered":"2023-01-15T14:35:12.861837+0000","last_clean":"2023-01-15T14:35:12.861837+0000","last_became_active":"2023-01-15T14:35:04.085770+0000","last_became_peered":"2023-01-15T14:35:04.085770+0000","last_unstale":"2023-01-15T14:35:12.861837+0000","last_undegraded":"2023-01-15T14:35:12.861837+0000","last_fullsized":"2023-01-15T14:35:12.861837+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:57.577405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:57.577405+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.80598328100000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.6b","version":"155'53","reported_seq":79,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.493024+0000","last_change":"2023-01-15T14:35:13.493119+0000","last_active":"2023-01-15T14:35:13.493024+0000","last_peered":"2023-01-15T14:35:13.493024+0000","last_clean":"2023-01-15T14:35:13.493024+0000","last_became_active":"2023-01-15T14:34:59.176998+0000","last_became_peered":"2023-01-15T14:34:59.176998+0000","last_unstale":"2023-01-15T14:35:13.493024+0000","last_undegraded":"2023-01-15T14:35:13.493024+0000","last_fullsized":"2023-01-15T14:35:13.493024+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.53444265800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.6a","version":"191'191","reported_seq":854,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.447350+0000","last_change":"2023-01-15T14:35:10.447350+0000","last_active":"2023-01-15T14:35:10.447350+0000","last_peered":"2023-01-15T14:35:10.447350+0000","last_clean":"2023-01-15T14:35:10.447350+0000","last_became_active":"2023-01-15T14:35:07.360156+0000","last_became_peered":"2023-01-15T14:35:07.360156+0000","last_unstale":"2023-01-15T14:35:10.447350+0000","last_undegraded":"2023-01-15T14:35:10.447350+0000","last_fullsized":"2023-01-15T14:35:10.447350+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21606388600000001,"stat_sum":{"num_bytes":1518387,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":5931,"num_write":14,"num_write_kb":6502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9447727,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.69","version":"110'149","reported_seq":1041,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.396336+0000","last_change":"2023-01-15T14:35:10.396336+0000","last_active":"2023-01-15T14:35:10.396336+0000","last_peered":"2023-01-15T14:35:10.396336+0000","last_clean":"2023-01-15T14:35:10.396336+0000","last_became_active":"2023-01-15T14:35:07.344216+0000","last_became_peered":"2023-01-15T14:35:07.344216+0000","last_unstale":"2023-01-15T14:35:10.396336+0000","last_undegraded":"2023-01-15T14:35:10.396336+0000","last_fullsized":"2023-01-15T14:35:10.396336+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.26435677299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.68","version":"420'278","reported_seq":1591,"reported_epoch":422,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:12.266999+0000","last_change":"2023-01-15T14:35:11.011179+0000","last_active":"2023-01-15T14:35:12.266999+0000","last_peered":"2023-01-15T14:35:12.266999+0000","last_clean":"2023-01-15T14:35:12.266999+0000","last_became_active":"2023-01-15T14:34:13.134258+0000","last_became_peered":"2023-01-15T14:34:13.134258+0000","last_unstale":"2023-01-15T14:35:12.266999+0000","last_undegraded":"2023-01-15T14:35:12.266999+0000","last_fullsized":"2023-01-15T14:35:12.266999+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:00.032","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.29294053399999997,"stat_sum":{"num_bytes":1292047,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17027823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.67","version":"191'79","reported_seq":211,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.616351+0000","last_change":"2023-01-15T14:35:13.616512+0000","last_active":"2023-01-15T14:35:13.616351+0000","last_peered":"2023-01-15T14:35:13.616351+0000","last_clean":"2023-01-15T14:35:13.616351+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:35:13.616351+0000","last_undegraded":"2023-01-15T14:35:13.616351+0000","last_fullsized":"2023-01-15T14:35:13.616351+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:02.115","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.55731053799999997,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.66","version":"102'52","reported_seq":30,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.440763+0000","last_change":"2023-01-15T14:35:10.440763+0000","last_active":"2023-01-15T14:35:10.440763+0000","last_peered":"2023-01-15T14:35:10.440763+0000","last_clean":"2023-01-15T14:35:10.440763+0000","last_became_active":"2023-01-15T14:35:04.489876+0000","last_became_peered":"2023-01-15T14:35:04.489876+0000","last_unstale":"2023-01-15T14:35:10.440763+0000","last_undegraded":"2023-01-15T14:35:10.440763+0000","last_fullsized":"2023-01-15T14:35:10.440763+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:34:32.591909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.591909+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:56.258","scrub_duration":0.39200112999999998,"objects_trimmed":0,"snaptrim_duration":0.306077983,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.65","version":"423'169","reported_seq":1483,"reported_epoch":423,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:13.098130+0000","last_change":"2023-01-15T14:35:13.098130+0000","last_active":"2023-01-15T14:35:13.098130+0000","last_peered":"2023-01-15T14:35:13.098130+0000","last_clean":"2023-01-15T14:35:13.098130+0000","last_became_active":"2023-01-15T14:35:10.233584+0000","last_became_peered":"2023-01-15T14:35:10.233584+0000","last_unstale":"2023-01-15T14:35:13.098130+0000","last_undegraded":"2023-01-15T14:35:13.098130+0000","last_fullsized":"2023-01-15T14:35:13.098130+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'126","last_scrub_stamp":"2023-01-15T14:34:23.903906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.903906+0000","objects_scrubbed":6,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:42.475","scrub_duration":0.30981859099999998,"objects_trimmed":0,"snaptrim_duration":0.26040120999999999,"stat_sum":{"num_bytes":15403336,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":15,"num_read_kb":13207,"num_write":16,"num_write_kb":7278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20310327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.64","version":"334'251","reported_seq":1589,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:11.967222+0000","last_change":"2023-01-15T14:35:10.344075+0000","last_active":"2023-01-15T14:35:11.967222+0000","last_peered":"2023-01-15T14:35:11.967222+0000","last_clean":"2023-01-15T14:35:11.967222+0000","last_became_active":"2023-01-15T14:35:07.057736+0000","last_became_peered":"2023-01-15T14:35:07.057736+0000","last_unstale":"2023-01-15T14:35:11.967222+0000","last_undegraded":"2023-01-15T14:35:11.967222+0000","last_fullsized":"2023-01-15T14:35:11.967222+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"334'251","last_scrub_stamp":"2023-01-15T14:34:48.644529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.644529+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:49.145","scrub_duration":0.70872698300000003,"objects_trimmed":0,"snaptrim_duration":0.20271830399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.63","version":"149'233","reported_seq":922,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.067745+0000","last_change":"2023-01-15T14:35:12.067745+0000","last_active":"2023-01-15T14:35:12.067745+0000","last_peered":"2023-01-15T14:35:12.067745+0000","last_clean":"2023-01-15T14:35:12.067745+0000","last_became_active":"2023-01-15T14:35:07.161207+0000","last_became_peered":"2023-01-15T14:35:07.161207+0000","last_unstale":"2023-01-15T14:35:12.067745+0000","last_undegraded":"2023-01-15T14:35:12.067745+0000","last_fullsized":"2023-01-15T14:35:12.067745+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:34:56.637790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:56.637790+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:16.461","scrub_duration":0.63257800399999997,"objects_trimmed":0,"snaptrim_duration":0.00089178700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.62","version":"419'199","reported_seq":1561,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.161782+0000","last_change":"2023-01-15T14:35:12.359381+0000","last_active":"2023-01-15T14:35:13.161782+0000","last_peered":"2023-01-15T14:35:13.161782+0000","last_clean":"2023-01-15T14:35:13.161782+0000","last_became_active":"2023-01-15T14:34:44.164909+0000","last_became_peered":"2023-01-15T14:34:44.164909+0000","last_unstale":"2023-01-15T14:35:13.161782+0000","last_undegraded":"2023-01-15T14:35:13.161782+0000","last_fullsized":"2023-01-15T14:35:13.161782+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":7,"last_scrub":"403'189","last_scrub_stamp":"2023-01-15T14:34:56.851447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:56.851447+0000","objects_scrubbed":5,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:20.283","scrub_duration":0.30960584400000002,"objects_trimmed":0,"snaptrim_duration":0.20318757100000001,"stat_sum":{"num_bytes":9221281,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":9592,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.61","version":"420'166","reported_seq":1201,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.066328+0000","last_change":"2023-01-15T14:35:11.112761+0000","last_active":"2023-01-15T14:35:12.066328+0000","last_peered":"2023-01-15T14:35:12.066328+0000","last_clean":"2023-01-15T14:35:12.066328+0000","last_became_active":"2023-01-15T14:34:44.473866+0000","last_became_peered":"2023-01-15T14:34:44.473866+0000","last_unstale":"2023-01-15T14:35:12.066328+0000","last_undegraded":"2023-01-15T14:35:12.066328+0000","last_fullsized":"2023-01-15T14:35:12.066328+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:32.734","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.97313557900000003,"stat_sum":{"num_bytes":3823692,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":16964,"num_write":6,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.60","version":"419'295","reported_seq":1610,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.057455+0000","last_change":"2023-01-15T14:35:13.057455+0000","last_active":"2023-01-15T14:35:13.057455+0000","last_peered":"2023-01-15T14:35:13.057455+0000","last_clean":"2023-01-15T14:35:13.057455+0000","last_became_active":"2023-01-15T14:34:44.073670+0000","last_became_peered":"2023-01-15T14:34:44.073670+0000","last_unstale":"2023-01-15T14:35:13.057455+0000","last_undegraded":"2023-01-15T14:35:13.057455+0000","last_fullsized":"2023-01-15T14:35:13.057455+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"402'291","last_scrub_stamp":"2023-01-15T14:34:55.535113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:55.535113+0000","objects_scrubbed":4,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:04.248","scrub_duration":0.814267513,"objects_trimmed":0,"snaptrim_duration":0.26406278500000002,"stat_sum":{"num_bytes":3971199,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"},{"start":"139","length":"1"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1522,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:11.968192+0000","last_change":"2023-01-15T14:35:10.408692+0000","last_active":"2023-01-15T14:35:11.968192+0000","last_peered":"2023-01-15T14:35:11.968192+0000","last_clean":"2023-01-15T14:35:11.968192+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:35:11.968192+0000","last_undegraded":"2023-01-15T14:35:11.968192+0000","last_fullsized":"2023-01-15T14:35:11.968192+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.16550610399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.5e","version":"215'39","reported_seq":1098,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.960798+0000","last_change":"2023-01-15T14:35:12.960798+0000","last_active":"2023-01-15T14:35:12.960798+0000","last_peered":"2023-01-15T14:35:12.960798+0000","last_clean":"2023-01-15T14:35:12.960798+0000","last_became_active":"2023-01-15T14:35:11.445183+0000","last_became_peered":"2023-01-15T14:35:11.445183+0000","last_unstale":"2023-01-15T14:35:12.960798+0000","last_undegraded":"2023-01-15T14:35:12.960798+0000","last_fullsized":"2023-01-15T14:35:12.960798+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.0028244810000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1479,"reported_epoch":423,"state":"active+clean","last_fresh":"2023-01-15T14:35:13.176061+0000","last_change":"2023-01-15T14:35:13.176061+0000","last_active":"2023-01-15T14:35:13.176061+0000","last_peered":"2023-01-15T14:35:13.176061+0000","last_clean":"2023-01-15T14:35:13.176061+0000","last_became_active":"2023-01-15T14:34:00.242122+0000","last_became_peered":"2023-01-15T14:34:00.242122+0000","last_unstale":"2023-01-15T14:35:13.176061+0000","last_undegraded":"2023-01-15T14:35:13.176061+0000","last_fullsized":"2023-01-15T14:35:13.176061+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:19.729021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:19.729021+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:29.740","scrub_duration":0.50499948100000003,"objects_trimmed":0,"snaptrim_duration":0.108291174,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.5c","version":"141'46","reported_seq":229,"reported_epoch":421,"state":"active+clean","last_fresh":"2023-01-15T14:35:11.351965+0000","last_change":"2023-01-15T14:35:11.351756+0000","last_active":"2023-01-15T14:35:11.351965+0000","last_peered":"2023-01-15T14:35:11.351965+0000","last_clean":"2023-01-15T14:35:11.351965+0000","last_became_active":"2023-01-15T14:35:11.349587+0000","last_became_peered":"2023-01-15T14:35:11.349587+0000","last_unstale":"2023-01-15T14:35:11.351965+0000","last_undegraded":"2023-01-15T14:35:11.351965+0000","last_fullsized":"2023-01-15T14:35:11.351965+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.001191302,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.5b","version":"283'80","reported_seq":1068,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.717651+0000","last_change":"2023-01-15T14:35:13.717876+0000","last_active":"2023-01-15T14:35:13.717651+0000","last_peered":"2023-01-15T14:35:13.717651+0000","last_clean":"2023-01-15T14:35:13.717651+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:35:13.717651+0000","last_undegraded":"2023-01-15T14:35:13.717651+0000","last_fullsized":"2023-01-15T14:35:13.717651+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:42.155","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.65803331399999998,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.5a","version":"411'361","reported_seq":1666,"reported_epoch":422,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:12.067011+0000","last_change":"2023-01-15T14:35:05.482085+0000","last_active":"2023-01-15T14:35:12.067011+0000","last_peered":"2023-01-15T14:35:12.067011+0000","last_clean":"2023-01-15T14:35:02.396095+0000","last_became_active":"2023-01-15T14:35:05.480728+0000","last_became_peered":"2023-01-15T14:35:05.480728+0000","last_unstale":"2023-01-15T14:35:12.067011+0000","last_undegraded":"2023-01-15T14:35:05.171683+0000","last_fullsized":"2023-01-15T14:35:05.170835+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":361,"log_dups_size":0,"ondisk_log_size":361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.448168648,"stat_sum":{"num_bytes":5983424,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":9,"num_read_kb":19294,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":3}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.59","version":"294'256","reported_seq":1468,"reported_epoch":423,"state":"active+clean","last_fresh":"2023-01-15T14:35:12.968126+0000","last_change":"2023-01-15T14:35:12.968126+0000","last_active":"2023-01-15T14:35:12.968126+0000","last_peered":"2023-01-15T14:35:12.968126+0000","last_clean":"2023-01-15T14:35:12.968126+0000","last_became_active":"2023-01-15T14:35:07.462526+0000","last_became_peered":"2023-01-15T14:35:07.462526+0000","last_unstale":"2023-01-15T14:35:12.968126+0000","last_undegraded":"2023-01-15T14:35:12.968126+0000","last_fullsized":"2023-01-15T14:35:12.968126+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:34:30.207952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:30.207952+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:30.349","scrub_duration":0.024191634,"objects_trimmed":0,"snaptrim_duration":0.0038271999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.58","version":"323'320","reported_seq":1704,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.391482+0000","last_change":"2023-01-15T14:35:13.391555+0000","last_active":"2023-01-15T14:35:13.391482+0000","last_peered":"2023-01-15T14:35:13.391482+0000","last_clean":"2023-01-15T14:35:13.391482+0000","last_became_active":"2023-01-15T14:35:07.359588+0000","last_became_peered":"2023-01-15T14:35:07.359588+0000","last_unstale":"2023-01-15T14:35:13.391482+0000","last_undegraded":"2023-01-15T14:35:13.391482+0000","last_fullsized":"2023-01-15T14:35:13.391482+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'320","last_scrub_stamp":"2023-01-15T14:34:47.541655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.541655+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:52.418","scrub_duration":0.40398073200000001,"objects_trimmed":0,"snaptrim_duration":0.43416753699999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.57","version":"408'114","reported_seq":278,"reported_epoch":422,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:11.967926+0000","last_change":"2023-01-15T14:35:05.286321+0000","last_active":"2023-01-15T14:35:11.967926+0000","last_peered":"2023-01-15T14:35:11.967926+0000","last_clean":"2023-01-15T14:35:02.556894+0000","last_became_active":"2023-01-15T14:35:05.185559+0000","last_became_peered":"2023-01-15T14:35:05.185559+0000","last_unstale":"2023-01-15T14:35:11.967926+0000","last_undegraded":"2023-01-15T14:35:04.976030+0000","last_fullsized":"2023-01-15T14:35:04.975477+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'100","last_scrub_stamp":"2023-01-15T14:34:23.447852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.447852+0000","objects_scrubbed":2,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:38.971","scrub_duration":0.52101476099999999,"objects_trimmed":0,"snaptrim_duration":0.50438350200000004,"stat_sum":{"num_bytes":5882615,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":11169,"num_write":5,"num_write_kb":1093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,2],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2,3","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.56","version":"320'127","reported_seq":783,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.962935+0000","last_change":"2023-01-15T14:35:12.962935+0000","last_active":"2023-01-15T14:35:12.962935+0000","last_peered":"2023-01-15T14:35:12.962935+0000","last_clean":"2023-01-15T14:35:12.962935+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:35:12.962935+0000","last_undegraded":"2023-01-15T14:35:12.962935+0000","last_fullsized":"2023-01-15T14:35:12.962935+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.90406710800000001,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.55","version":"110'55","reported_seq":991,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:11.966102+0000","last_change":"2023-01-15T14:35:11.448346+0000","last_active":"2023-01-15T14:35:11.966102+0000","last_peered":"2023-01-15T14:35:11.966102+0000","last_clean":"2023-01-15T14:35:11.966102+0000","last_became_active":"2023-01-15T14:35:11.347080+0000","last_became_peered":"2023-01-15T14:35:11.347080+0000","last_unstale":"2023-01-15T14:35:11.966102+0000","last_undegraded":"2023-01-15T14:35:11.966102+0000","last_fullsized":"2023-01-15T14:35:11.966102+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:34:59.815570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:59.815570+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.93584606000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.54","version":"245'148","reported_seq":1288,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.066699+0000","last_change":"2023-01-15T14:35:10.343443+0000","last_active":"2023-01-15T14:35:12.066699+0000","last_peered":"2023-01-15T14:35:12.066699+0000","last_clean":"2023-01-15T14:35:12.066699+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:35:12.066699+0000","last_undegraded":"2023-01-15T14:35:12.066699+0000","last_fullsized":"2023-01-15T14:35:12.066699+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:34:47.392699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.392699+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:14.435","scrub_duration":0.50581767899999996,"objects_trimmed":0,"snaptrim_duration":0.102879821,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.53","version":"420'467","reported_seq":1738,"reported_epoch":423,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:13.516169+0000","last_change":"2023-01-15T14:35:12.652138+0000","last_active":"2023-01-15T14:35:13.516169+0000","last_peered":"2023-01-15T14:35:13.516169+0000","last_clean":"2023-01-15T14:35:02.014101+0000","last_became_active":"2023-01-15T14:35:05.283610+0000","last_became_peered":"2023-01-15T14:35:05.283610+0000","last_unstale":"2023-01-15T14:35:13.516169+0000","last_undegraded":"2023-01-15T14:35:04.977181+0000","last_fullsized":"2023-01-15T14:35:04.975844+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":467,"log_dups_size":0,"ondisk_log_size":467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.40805575199999999,"stat_sum":{"num_bytes":36561720,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":67,"num_read_kb":67500,"num_write":53,"num_write_kb":27412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":32800828,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[1,7],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":8},{"shards":"1,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":1,"purged_snaps":[{"start":"10f","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":726,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.450104+0000","last_change":"2023-01-15T14:35:10.450104+0000","last_active":"2023-01-15T14:35:10.450104+0000","last_peered":"2023-01-15T14:35:10.450104+0000","last_clean":"2023-01-15T14:35:10.450104+0000","last_became_active":"2023-01-15T14:35:07.357440+0000","last_became_peered":"2023-01-15T14:35:07.357440+0000","last_unstale":"2023-01-15T14:35:10.450104+0000","last_undegraded":"2023-01-15T14:35:10.450104+0000","last_fullsized":"2023-01-15T14:35:10.450104+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.11836247599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.51","version":"419'339","reported_seq":1701,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.057886+0000","last_change":"2023-01-15T14:35:13.057886+0000","last_active":"2023-01-15T14:35:13.057886+0000","last_peered":"2023-01-15T14:35:13.057886+0000","last_clean":"2023-01-15T14:35:13.057886+0000","last_became_active":"2023-01-15T14:34:44.372829+0000","last_became_peered":"2023-01-15T14:34:44.372829+0000","last_unstale":"2023-01-15T14:35:13.057886+0000","last_undegraded":"2023-01-15T14:35:13.057886+0000","last_fullsized":"2023-01-15T14:35:13.057886+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.19755230400000001,"stat_sum":{"num_bytes":6392131,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"},{"start":"139","length":"1"}]},{"pgid":"3.50","version":"419'468","reported_seq":1790,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.594044+0000","last_change":"2023-01-15T14:35:13.594156+0000","last_active":"2023-01-15T14:35:13.594044+0000","last_peered":"2023-01-15T14:35:13.594044+0000","last_clean":"2023-01-15T14:35:13.594044+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:35:13.594044+0000","last_undegraded":"2023-01-15T14:35:13.594044+0000","last_fullsized":"2023-01-15T14:35:13.594044+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:37.144","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.63628666099999998,"stat_sum":{"num_bytes":16579116,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":76,"num_read_kb":78946,"num_write":64,"num_write_kb":23394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.4f","version":"155'145","reported_seq":968,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.448089+0000","last_change":"2023-01-15T14:35:10.448089+0000","last_active":"2023-01-15T14:35:10.448089+0000","last_peered":"2023-01-15T14:35:10.448089+0000","last_clean":"2023-01-15T14:35:10.448089+0000","last_became_active":"2023-01-15T14:35:07.159725+0000","last_became_peered":"2023-01-15T14:35:07.159725+0000","last_unstale":"2023-01-15T14:35:10.448089+0000","last_undegraded":"2023-01-15T14:35:10.448089+0000","last_fullsized":"2023-01-15T14:35:10.448089+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.21517017299999999,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.4e","version":"139'38","reported_seq":250,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:11.969014+0000","last_change":"2023-01-15T14:35:11.969047+0000","last_active":"2023-01-15T14:35:11.969014+0000","last_peered":"2023-01-15T14:35:11.969014+0000","last_clean":"2023-01-15T14:35:11.969014+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:35:11.969014+0000","last_undegraded":"2023-01-15T14:35:11.969014+0000","last_fullsized":"2023-01-15T14:35:11.969014+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.0026819130000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.4d","version":"422'312","reported_seq":1556,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.261155+0000","last_change":"2023-01-15T14:35:13.261299+0000","last_active":"2023-01-15T14:35:13.261155+0000","last_peered":"2023-01-15T14:35:13.261155+0000","last_clean":"2023-01-15T14:35:13.261155+0000","last_became_active":"2023-01-15T14:34:05.178613+0000","last_became_peered":"2023-01-15T14:34:05.178613+0000","last_unstale":"2023-01-15T14:35:13.261155+0000","last_undegraded":"2023-01-15T14:35:13.261155+0000","last_fullsized":"2023-01-15T14:35:13.261155+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.30235490300000001,"stat_sum":{"num_bytes":17922713,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":16,"num_read_kb":15961,"num_write":23,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.4c","version":"309'71","reported_seq":277,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.718936+0000","last_change":"2023-01-15T14:35:13.719081+0000","last_active":"2023-01-15T14:35:13.718936+0000","last_peered":"2023-01-15T14:35:13.718936+0000","last_clean":"2023-01-15T14:35:13.718936+0000","last_became_active":"2023-01-15T14:35:04.379289+0000","last_became_peered":"2023-01-15T14:35:04.379289+0000","last_unstale":"2023-01-15T14:35:13.718936+0000","last_undegraded":"2023-01-15T14:35:13.718936+0000","last_fullsized":"2023-01-15T14:35:13.718936+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:27.027","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.66126780699999999,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.4b","version":"155'53","reported_seq":30,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.449035+0000","last_change":"2023-01-15T14:35:10.449035+0000","last_active":"2023-01-15T14:35:10.449035+0000","last_peered":"2023-01-15T14:35:10.449035+0000","last_clean":"2023-01-15T14:35:10.449035+0000","last_became_active":"2023-01-15T14:35:04.179749+0000","last_became_peered":"2023-01-15T14:35:04.179749+0000","last_unstale":"2023-01-15T14:35:10.449035+0000","last_undegraded":"2023-01-15T14:35:10.449035+0000","last_fullsized":"2023-01-15T14:35:10.449035+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:34:44.116779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:44.116779+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.117826915,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.4a","version":"421'410","reported_seq":1668,"reported_epoch":422,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:12.156117+0000","last_change":"2023-01-15T14:35:05.287631+0000","last_active":"2023-01-15T14:35:12.156117+0000","last_peered":"2023-01-15T14:35:12.156117+0000","last_clean":"2023-01-15T14:35:02.815674+0000","last_became_active":"2023-01-15T14:35:05.286607+0000","last_became_peered":"2023-01-15T14:35:05.286607+0000","last_unstale":"2023-01-15T14:35:12.156117+0000","last_undegraded":"2023-01-15T14:35:05.084004+0000","last_fullsized":"2023-01-15T14:35:05.082815+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:34.117","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70426192200000004,"stat_sum":{"num_bytes":15797888,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":32,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":84,"num_read_kb":95190,"num_write":86,"num_write_kb":26766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20473217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,3],"avail_no_missing":["7","2","3"],"object_location_counts":[{"shards":"2,3,7","objects":7},{"shards":"3,7","objects":-2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.49","version":"110'149","reported_seq":1092,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.392063+0000","last_change":"2023-01-15T14:35:13.392107+0000","last_active":"2023-01-15T14:35:13.392063+0000","last_peered":"2023-01-15T14:35:13.392063+0000","last_clean":"2023-01-15T14:35:13.392063+0000","last_became_active":"2023-01-15T14:34:00.043221+0000","last_became_peered":"2023-01-15T14:34:00.043221+0000","last_unstale":"2023-01-15T14:35:13.392063+0000","last_undegraded":"2023-01-15T14:35:13.392063+0000","last_fullsized":"2023-01-15T14:35:13.392063+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.43500733200000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.48","version":"179'204","reported_seq":1332,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:11.965250+0000","last_change":"2023-01-15T14:35:11.544320+0000","last_active":"2023-01-15T14:35:11.965250+0000","last_peered":"2023-01-15T14:35:11.965250+0000","last_clean":"2023-01-15T14:35:11.965250+0000","last_became_active":"2023-01-15T14:35:11.517680+0000","last_became_peered":"2023-01-15T14:35:11.517680+0000","last_unstale":"2023-01-15T14:35:11.965250+0000","last_undegraded":"2023-01-15T14:35:11.965250+0000","last_fullsized":"2023-01-15T14:35:11.965250+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.0046281250000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.47","version":"245'112","reported_seq":1293,"reported_epoch":423,"state":"active+clean","last_fresh":"2023-01-15T14:35:13.170604+0000","last_change":"2023-01-15T14:35:13.170604+0000","last_active":"2023-01-15T14:35:13.170604+0000","last_peered":"2023-01-15T14:35:13.170604+0000","last_clean":"2023-01-15T14:35:13.170604+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:35:13.170604+0000","last_undegraded":"2023-01-15T14:35:13.170604+0000","last_fullsized":"2023-01-15T14:35:13.170604+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:34:21.450320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:21.450320+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:50.200","scrub_duration":0.30497759099999999,"objects_trimmed":0,"snaptrim_duration":0.20356896199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.46","version":"102'52","reported_seq":95,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.098054+0000","last_change":"2023-01-15T14:35:13.098106+0000","last_active":"2023-01-15T14:35:13.098054+0000","last_peered":"2023-01-15T14:35:13.098054+0000","last_clean":"2023-01-15T14:35:13.098054+0000","last_became_active":"2023-01-15T14:35:04.373495+0000","last_became_peered":"2023-01-15T14:35:04.373495+0000","last_unstale":"2023-01-15T14:35:13.098054+0000","last_undegraded":"2023-01-15T14:35:13.098054+0000","last_fullsized":"2023-01-15T14:35:13.098054+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.13881173599999999,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2142,"num_write":3,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.45","version":"253'125","reported_seq":1073,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:11.966823+0000","last_change":"2023-01-15T14:35:10.405999+0000","last_active":"2023-01-15T14:35:11.966823+0000","last_peered":"2023-01-15T14:35:11.966823+0000","last_clean":"2023-01-15T14:35:11.966823+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:35:11.966823+0000","last_undegraded":"2023-01-15T14:35:11.966823+0000","last_fullsized":"2023-01-15T14:35:11.966823+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:34:50.381073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:50.381073+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:56.850","scrub_duration":0.52191215300000005,"objects_trimmed":0,"snaptrim_duration":0.000471454,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.44","version":"245'268","reported_seq":1265,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.442639+0000","last_change":"2023-01-15T14:35:10.442639+0000","last_active":"2023-01-15T14:35:10.442639+0000","last_peered":"2023-01-15T14:35:10.442639+0000","last_clean":"2023-01-15T14:35:10.442639+0000","last_became_active":"2023-01-15T14:35:07.262970+0000","last_became_peered":"2023-01-15T14:35:07.262970+0000","last_unstale":"2023-01-15T14:35:10.442639+0000","last_undegraded":"2023-01-15T14:35:10.442639+0000","last_fullsized":"2023-01-15T14:35:10.442639+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.30485241699999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.43","version":"411'339","reported_seq":1684,"reported_epoch":422,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:12.068186+0000","last_change":"2023-01-15T14:35:05.180473+0000","last_active":"2023-01-15T14:35:12.068186+0000","last_peered":"2023-01-15T14:35:12.068186+0000","last_clean":"2023-01-15T14:35:02.305445+0000","last_became_active":"2023-01-15T14:35:05.179499+0000","last_became_peered":"2023-01-15T14:35:05.179499+0000","last_unstale":"2023-01-15T14:35:12.068186+0000","last_undegraded":"2023-01-15T14:35:04.973763+0000","last_fullsized":"2023-01-15T14:35:04.972474+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.25741995200000001,"stat_sum":{"num_bytes":9109625,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":13,"num_read_kb":16586,"num_write":12,"num_write_kb":5193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3395080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[0,2],"avail_no_missing":["0","2","7"],"object_location_counts":[{"shards":"0,2,7","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":1196,"reported_epoch":421,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:11.451833+0000","last_change":"2023-01-15T14:35:11.451750+0000","last_active":"2023-01-15T14:35:11.451833+0000","last_peered":"2023-01-15T14:35:11.451833+0000","last_clean":"2023-01-15T14:35:11.451833+0000","last_became_active":"2023-01-15T14:35:11.450430+0000","last_became_peered":"2023-01-15T14:35:11.450430+0000","last_unstale":"2023-01-15T14:35:11.451833+0000","last_undegraded":"2023-01-15T14:35:11.451833+0000","last_fullsized":"2023-01-15T14:35:11.451833+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:34:57.046809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:57.046809+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:16.212","scrub_duration":0.71122905999999997,"objects_trimmed":0,"snaptrim_duration":0.106276949,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.41","version":"155'121","reported_seq":1105,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.295401+0000","last_change":"2023-01-15T14:35:10.295401+0000","last_active":"2023-01-15T14:35:10.295401+0000","last_peered":"2023-01-15T14:35:10.295401+0000","last_clean":"2023-01-15T14:35:10.295401+0000","last_became_active":"2023-01-15T14:35:07.565877+0000","last_became_peered":"2023-01-15T14:35:07.565877+0000","last_unstale":"2023-01-15T14:35:10.295401+0000","last_undegraded":"2023-01-15T14:35:10.295401+0000","last_fullsized":"2023-01-15T14:35:10.295401+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.16394845799999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.40","version":"412'334","reported_seq":1615,"reported_epoch":422,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:12.267466+0000","last_change":"2023-01-15T14:35:05.478832+0000","last_active":"2023-01-15T14:35:12.267466+0000","last_peered":"2023-01-15T14:35:12.267466+0000","last_clean":"2023-01-15T14:35:02.920679+0000","last_became_active":"2023-01-15T14:35:05.372244+0000","last_became_peered":"2023-01-15T14:35:05.372244+0000","last_unstale":"2023-01-15T14:35:12.267466+0000","last_undegraded":"2023-01-15T14:35:05.073987+0000","last_fullsized":"2023-01-15T14:35:05.072453+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.00105211,"stat_sum":{"num_bytes":10624418,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":35542,"num_write":33,"num_write_kb":12321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":29167491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[0,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[{"start":"126","length":"1"}]},{"pgid":"3.3f","version":"420'256","reported_seq":1547,"reported_epoch":422,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:12.057800+0000","last_change":"2023-01-15T14:35:05.174749+0000","last_active":"2023-01-15T14:35:12.057800+0000","last_peered":"2023-01-15T14:35:12.057800+0000","last_clean":"2023-01-15T14:35:02.011791+0000","last_became_active":"2023-01-15T14:35:05.173356+0000","last_became_peered":"2023-01-15T14:35:05.173356+0000","last_unstale":"2023-01-15T14:35:12.057800+0000","last_undegraded":"2023-01-15T14:35:04.979135+0000","last_fullsized":"2023-01-15T14:35:04.978012+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.60129771799999998,"stat_sum":{"num_bytes":17854030,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":20352,"num_write":24,"num_write_kb":7278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":10118012,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,0],"avail_no_missing":["7","0","4"],"object_location_counts":[{"shards":"0,4,7","objects":2},{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10f","length":"1"}]},{"pgid":"3.3e","version":"423'72","reported_seq":137,"reported_epoch":423,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:12.960845+0000","last_change":"2023-01-15T14:35:12.960845+0000","last_active":"2023-01-15T14:35:12.960845+0000","last_peered":"2023-01-15T14:35:12.960845+0000","last_clean":"2023-01-15T14:35:12.960845+0000","last_became_active":"2023-01-15T14:35:04.273985+0000","last_became_peered":"2023-01-15T14:35:04.273985+0000","last_unstale":"2023-01-15T14:35:12.960845+0000","last_undegraded":"2023-01-15T14:35:12.960845+0000","last_fullsized":"2023-01-15T14:35:12.960845+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.89673946000000004,"stat_sum":{"num_bytes":5664091,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":7,"num_write_kb":4255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":16992273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.3d","version":"421'273","reported_seq":1573,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:11.966525+0000","last_change":"2023-01-15T14:35:11.966525+0000","last_active":"2023-01-15T14:35:11.966525+0000","last_peered":"2023-01-15T14:35:11.966525+0000","last_clean":"2023-01-15T14:35:11.966525+0000","last_became_active":"2023-01-15T14:34:44.167696+0000","last_became_peered":"2023-01-15T14:34:44.167696+0000","last_unstale":"2023-01-15T14:35:11.966525+0000","last_undegraded":"2023-01-15T14:35:11.966525+0000","last_fullsized":"2023-01-15T14:35:11.966525+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.803581463,"stat_sum":{"num_bytes":5496021,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":36,"num_read_kb":41385,"num_write":19,"num_write_kb":8717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":14705983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":27,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.721878+0000","last_change":"2023-01-15T14:35:13.721878+0000","last_active":"2023-01-15T14:35:13.721878+0000","last_peered":"2023-01-15T14:35:13.721878+0000","last_clean":"2023-01-15T14:35:13.721878+0000","last_became_active":"2023-01-15T14:35:11.341616+0000","last_became_peered":"2023-01-15T14:35:11.341616+0000","last_unstale":"2023-01-15T14:35:13.721878+0000","last_undegraded":"2023-01-15T14:35:13.721878+0000","last_fullsized":"2023-01-15T14:35:13.721878+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.76236820699999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":579834,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.3b","version":"221'60","reported_seq":182,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.614529+0000","last_change":"2023-01-15T14:35:13.614685+0000","last_active":"2023-01-15T14:35:13.614529+0000","last_peered":"2023-01-15T14:35:13.614529+0000","last_clean":"2023-01-15T14:35:13.614529+0000","last_became_active":"2023-01-15T14:33:57.013585+0000","last_became_peered":"2023-01-15T14:33:57.013585+0000","last_unstale":"2023-01-15T14:35:13.614529+0000","last_undegraded":"2023-01-15T14:35:13.614529+0000","last_fullsized":"2023-01-15T14:35:13.614529+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.45502314199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1536,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.066562+0000","last_change":"2023-01-15T14:35:10.509625+0000","last_active":"2023-01-15T14:35:12.066562+0000","last_peered":"2023-01-15T14:35:12.066562+0000","last_clean":"2023-01-15T14:35:12.066562+0000","last_became_active":"2023-01-15T14:34:46.572815+0000","last_became_peered":"2023-01-15T14:34:46.572815+0000","last_unstale":"2023-01-15T14:35:12.066562+0000","last_undegraded":"2023-01-15T14:35:12.066562+0000","last_fullsized":"2023-01-15T14:35:12.066562+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.268609971,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.39","version":"423'433","reported_seq":1719,"reported_epoch":423,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:13.718765+0000","last_change":"2023-01-15T14:35:13.261274+0000","last_active":"2023-01-15T14:35:13.718765+0000","last_peered":"2023-01-15T14:35:13.718765+0000","last_clean":"2023-01-15T14:35:13.718765+0000","last_became_active":"2023-01-15T14:34:08.023332+0000","last_became_peered":"2023-01-15T14:34:08.023332+0000","last_unstale":"2023-01-15T14:35:13.718765+0000","last_undegraded":"2023-01-15T14:35:13.718765+0000","last_fullsized":"2023-01-15T14:35:13.718765+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.80413571900000003,"stat_sum":{"num_bytes":34197746,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":68,"num_read_kb":114793,"num_write":24,"num_write_kb":13863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":26862613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.38","version":"408'343","reported_seq":1698,"reported_epoch":421,"state":"peering","last_fresh":"2023-01-15T14:34:58.898642+0000","last_change":"2023-01-15T14:35:11.546446+0000","last_active":"2023-01-15T14:35:11.546446+0000","last_peered":"2023-01-15T14:34:58.898642+0000","last_clean":"2023-01-15T14:34:58.898642+0000","last_became_active":"2023-01-15T14:32:00.062947+0000","last_became_peered":"2023-01-15T14:32:00.062947+0000","last_unstale":"2023-01-15T14:34:58.898642+0000","last_undegraded":"2023-01-15T14:34:58.898642+0000","last_fullsized":"2023-01-15T14:34:58.898642+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":1.00781638,"stat_sum":{"num_bytes":8461433,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":45,"num_read_kb":42848,"num_write":40,"num_write_kb":13455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":20850952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.37","version":"200'78","reported_seq":586,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.268783+0000","last_change":"2023-01-15T14:35:13.268859+0000","last_active":"2023-01-15T14:35:13.268783+0000","last_peered":"2023-01-15T14:35:13.268783+0000","last_clean":"2023-01-15T14:35:13.268783+0000","last_became_active":"2023-01-15T14:34:43.156532+0000","last_became_peered":"2023-01-15T14:34:43.156532+0000","last_unstale":"2023-01-15T14:35:13.268783+0000","last_undegraded":"2023-01-15T14:35:13.268783+0000","last_fullsized":"2023-01-15T14:35:13.268783+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.20895788000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.36","version":"194'81","reported_seq":30,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.296569+0000","last_change":"2023-01-15T14:35:10.296569+0000","last_active":"2023-01-15T14:35:10.296569+0000","last_peered":"2023-01-15T14:35:10.296569+0000","last_clean":"2023-01-15T14:35:10.296569+0000","last_became_active":"2023-01-15T14:35:04.286361+0000","last_became_peered":"2023-01-15T14:35:04.286361+0000","last_unstale":"2023-01-15T14:35:10.296569+0000","last_undegraded":"2023-01-15T14:35:10.296569+0000","last_fullsized":"2023-01-15T14:35:10.296569+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:34:51.296526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.296526+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:56.486","scrub_duration":0.66934546500000003,"objects_trimmed":0,"snaptrim_duration":0.16267029699999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.35","version":"110'55","reported_seq":30,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.338328+0000","last_change":"2023-01-15T14:35:10.338328+0000","last_active":"2023-01-15T14:35:10.338328+0000","last_peered":"2023-01-15T14:35:10.338328+0000","last_clean":"2023-01-15T14:35:10.338328+0000","last_became_active":"2023-01-15T14:35:04.186590+0000","last_became_peered":"2023-01-15T14:35:04.186590+0000","last_unstale":"2023-01-15T14:35:10.338328+0000","last_undegraded":"2023-01-15T14:35:10.338328+0000","last_fullsized":"2023-01-15T14:35:10.338328+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20441562099999999,"stat_sum":{"num_bytes":1592455,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10339,"num_write":6,"num_write_kb":3114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1063027,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":1339,"reported_epoch":423,"state":"active+clean","last_fresh":"2023-01-15T14:35:13.172309+0000","last_change":"2023-01-15T14:35:13.172309+0000","last_active":"2023-01-15T14:35:13.172309+0000","last_peered":"2023-01-15T14:35:13.172309+0000","last_clean":"2023-01-15T14:35:13.172309+0000","last_became_active":"2023-01-15T14:35:07.264058+0000","last_became_peered":"2023-01-15T14:35:07.264058+0000","last_unstale":"2023-01-15T14:35:13.172309+0000","last_undegraded":"2023-01-15T14:35:13.172309+0000","last_fullsized":"2023-01-15T14:35:13.172309+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.107909011,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.33","version":"204'275","reported_seq":1434,"reported_epoch":421,"state":"active+clean","last_fresh":"2023-01-15T14:35:11.353935+0000","last_change":"2023-01-15T14:35:11.353935+0000","last_active":"2023-01-15T14:35:11.353935+0000","last_peered":"2023-01-15T14:35:11.353935+0000","last_clean":"2023-01-15T14:35:11.353935+0000","last_became_active":"2023-01-15T14:35:11.352099+0000","last_became_peered":"2023-01-15T14:35:11.352099+0000","last_unstale":"2023-01-15T14:35:11.353935+0000","last_undegraded":"2023-01-15T14:35:11.353935+0000","last_fullsized":"2023-01-15T14:35:11.353935+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.00095217100000000003,"stat_sum":{"num_bytes":3485477,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11189,"num_write":7,"num_write_kb":3285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.32","version":"420'180","reported_seq":1041,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.066047+0000","last_change":"2023-01-15T14:35:11.011225+0000","last_active":"2023-01-15T14:35:12.066047+0000","last_peered":"2023-01-15T14:35:12.066047+0000","last_clean":"2023-01-15T14:35:12.066047+0000","last_became_active":"2023-01-15T14:34:56.393418+0000","last_became_peered":"2023-01-15T14:34:56.393418+0000","last_unstale":"2023-01-15T14:35:12.066047+0000","last_undegraded":"2023-01-15T14:35:12.066047+0000","last_fullsized":"2023-01-15T14:35:12.066047+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.770135193,"stat_sum":{"num_bytes":4024329,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3525,"num_write":5,"num_write_kb":2712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2663942,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.13","version":"423'402","reported_seq":1579,"reported_epoch":423,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:13.721856+0000","last_change":"2023-01-15T14:35:13.721856+0000","last_active":"2023-01-15T14:35:13.721856+0000","last_peered":"2023-01-15T14:35:13.721856+0000","last_clean":"2023-01-15T14:35:13.721856+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:35:13.721856+0000","last_undegraded":"2023-01-15T14:35:13.721856+0000","last_fullsized":"2023-01-15T14:35:13.721856+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"374'376","last_scrub_stamp":"2023-01-15T14:34:32.489483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.489483+0000","objects_scrubbed":4,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:32.530","scrub_duration":0.381937625,"objects_trimmed":0,"snaptrim_duration":0.60274494300000003,"stat_sum":{"num_bytes":13721904,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":10,"num_read_kb":18719,"num_write":3,"num_write_kb":2561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.12","version":"417'207","reported_seq":1071,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.067040+0000","last_change":"2023-01-15T14:35:10.406434+0000","last_active":"2023-01-15T14:35:12.067040+0000","last_peered":"2023-01-15T14:35:12.067040+0000","last_clean":"2023-01-15T14:35:12.067040+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:35:12.067040+0000","last_undegraded":"2023-01-15T14:35:12.067040+0000","last_fullsized":"2023-01-15T14:35:12.067040+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:58.076","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.16554843799999999,"stat_sum":{"num_bytes":10304297,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":4,"num_write":20,"num_write_kb":9403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.11","version":"423'346","reported_seq":1820,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.616796+0000","last_change":"2023-01-15T14:35:13.616796+0000","last_active":"2023-01-15T14:35:13.616796+0000","last_peered":"2023-01-15T14:35:13.616796+0000","last_clean":"2023-01-15T14:35:13.616796+0000","last_became_active":"2023-01-15T14:35:13.513113+0000","last_became_peered":"2023-01-15T14:35:13.513113+0000","last_unstale":"2023-01-15T14:35:13.616796+0000","last_undegraded":"2023-01-15T14:35:13.616796+0000","last_fullsized":"2023-01-15T14:35:13.616796+0000","mapping_epoch":422,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:21.075","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.67565938599999997,"stat_sum":{"num_bytes":22103143,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":48,"num_read_kb":48289,"num_write":33,"num_write_kb":23912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":44867083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"126","length":"1"}]},{"pgid":"3.10","version":"409'420","reported_seq":1562,"reported_epoch":422,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:12.165741+0000","last_change":"2023-01-15T14:35:05.479074+0000","last_active":"2023-01-15T14:35:12.165741+0000","last_peered":"2023-01-15T14:35:12.165741+0000","last_clean":"2023-01-15T14:35:02.307833+0000","last_became_active":"2023-01-15T14:35:05.377956+0000","last_became_peered":"2023-01-15T14:35:05.377956+0000","last_unstale":"2023-01-15T14:35:12.165741+0000","last_undegraded":"2023-01-15T14:35:05.071354+0000","last_fullsized":"2023-01-15T14:35:05.070163+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.361291529,"stat_sum":{"num_bytes":8658713,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":55,"num_read_kb":52673,"num_write":50,"num_write_kb":18045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12804882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[0,3],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2,3","objects":3},{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1389,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.166438+0000","last_change":"2023-01-15T14:35:13.166580+0000","last_active":"2023-01-15T14:35:13.166438+0000","last_peered":"2023-01-15T14:35:13.166438+0000","last_clean":"2023-01-15T14:35:13.166438+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:35:13.166438+0000","last_undegraded":"2023-01-15T14:35:13.166438+0000","last_fullsized":"2023-01-15T14:35:13.166438+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.20782473900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":203,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.167403+0000","last_change":"2023-01-15T14:35:13.167512+0000","last_active":"2023-01-15T14:35:13.167403+0000","last_peered":"2023-01-15T14:35:13.167403+0000","last_clean":"2023-01-15T14:35:13.167403+0000","last_became_active":"2023-01-15T14:34:43.265641+0000","last_became_peered":"2023-01-15T14:34:43.265641+0000","last_unstale":"2023-01-15T14:35:13.167403+0000","last_undegraded":"2023-01-15T14:35:13.167403+0000","last_fullsized":"2023-01-15T14:35:13.167403+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.20843127,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.d","version":"296'232","reported_seq":1468,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.390936+0000","last_change":"2023-01-15T14:35:13.390936+0000","last_active":"2023-01-15T14:35:13.390936+0000","last_peered":"2023-01-15T14:35:13.390936+0000","last_clean":"2023-01-15T14:35:13.390936+0000","last_became_active":"2023-01-15T14:35:07.267234+0000","last_became_peered":"2023-01-15T14:35:07.267234+0000","last_unstale":"2023-01-15T14:35:13.390936+0000","last_undegraded":"2023-01-15T14:35:13.390936+0000","last_fullsized":"2023-01-15T14:35:13.390936+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:22.089","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.43105289499999999,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.c","version":"421'126","reported_seq":1406,"reported_epoch":421,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:11.450690+0000","last_change":"2023-01-15T14:35:11.448564+0000","last_active":"2023-01-15T14:35:11.450690+0000","last_peered":"2023-01-15T14:35:11.450690+0000","last_clean":"2023-01-15T14:35:11.450690+0000","last_became_active":"2023-01-15T14:35:11.346741+0000","last_became_peered":"2023-01-15T14:35:11.346741+0000","last_unstale":"2023-01-15T14:35:11.450690+0000","last_undegraded":"2023-01-15T14:35:11.450690+0000","last_fullsized":"2023-01-15T14:35:11.450690+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.69268427700000001,"stat_sum":{"num_bytes":3264587,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":43,"num_read_kb":67246,"num_write":12,"num_write_kb":1706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":29077297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":985,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:11.965585+0000","last_change":"2023-01-15T14:35:11.448234+0000","last_active":"2023-01-15T14:35:11.965585+0000","last_peered":"2023-01-15T14:35:11.965585+0000","last_clean":"2023-01-15T14:35:11.965585+0000","last_became_active":"2023-01-15T14:35:11.447676+0000","last_became_peered":"2023-01-15T14:35:11.447676+0000","last_unstale":"2023-01-15T14:35:11.965585+0000","last_undegraded":"2023-01-15T14:35:11.965585+0000","last_fullsized":"2023-01-15T14:35:11.965585+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.200457632,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.a","version":"252'279","reported_seq":1468,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.445453+0000","last_change":"2023-01-15T14:35:10.445453+0000","last_active":"2023-01-15T14:35:10.445453+0000","last_peered":"2023-01-15T14:35:10.445453+0000","last_clean":"2023-01-15T14:35:10.445453+0000","last_became_active":"2023-01-15T14:35:07.461643+0000","last_became_peered":"2023-01-15T14:35:07.461643+0000","last_unstale":"2023-01-15T14:35:10.445453+0000","last_undegraded":"2023-01-15T14:35:10.445453+0000","last_fullsized":"2023-01-15T14:35:10.445453+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.21468090000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":1274,"reported_epoch":423,"state":"active+clean","last_fresh":"2023-01-15T14:35:12.969387+0000","last_change":"2023-01-15T14:35:12.969387+0000","last_active":"2023-01-15T14:35:12.969387+0000","last_peered":"2023-01-15T14:35:12.969387+0000","last_clean":"2023-01-15T14:35:12.969387+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:35:12.969387+0000","last_undegraded":"2023-01-15T14:35:12.969387+0000","last_fullsized":"2023-01-15T14:35:12.969387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.003568763,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.8","version":"179'204","reported_seq":1464,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.056185+0000","last_change":"2023-01-15T14:35:13.056160+0000","last_active":"2023-01-15T14:35:13.056185+0000","last_peered":"2023-01-15T14:35:13.056185+0000","last_clean":"2023-01-15T14:35:13.056185+0000","last_became_active":"2023-01-15T14:35:11.437153+0000","last_became_peered":"2023-01-15T14:35:11.437153+0000","last_unstale":"2023-01-15T14:35:13.056185+0000","last_undegraded":"2023-01-15T14:35:13.056185+0000","last_fullsized":"2023-01-15T14:35:13.056185+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:34:48.269667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.269667+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.26306564999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"},{"start":"139","length":"1"}]},{"pgid":"1.0","version":"19'117","reported_seq":606,"reported_epoch":422,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:12.054339+0000","last_change":"2023-01-15T14:35:05.375412+0000","last_active":"2023-01-15T14:35:12.054339+0000","last_peered":"2023-01-15T14:35:12.054339+0000","last_clean":"2023-01-15T14:35:02.009600+0000","last_became_active":"2023-01-15T14:35:05.374546+0000","last_became_peered":"2023-01-15T14:35:05.374546+0000","last_unstale":"2023-01-15T14:35:12.054339+0000","last_undegraded":"2023-01-15T14:35:04.979005+0000","last_fullsized":"2023-01-15T14:35:04.978006+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":86,"reported_epoch":422,"state":"active+clean","last_fresh":"2023-01-15T14:35:12.157230+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:35:12.157230+0000","last_peered":"2023-01-15T14:35:12.157230+0000","last_clean":"2023-01-15T14:35:12.157230+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:35:12.157230+0000","last_undegraded":"2023-01-15T14:35:12.157230+0000","last_fullsized":"2023-01-15T14:35:12.157230+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":1141,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.446599+0000","last_change":"2023-01-15T14:35:10.446599+0000","last_active":"2023-01-15T14:35:10.446599+0000","last_peered":"2023-01-15T14:35:10.446599+0000","last_clean":"2023-01-15T14:35:10.446599+0000","last_became_active":"2023-01-15T14:35:07.458021+0000","last_became_peered":"2023-01-15T14:35:07.458021+0000","last_unstale":"2023-01-15T14:35:10.446599+0000","last_undegraded":"2023-01-15T14:35:10.446599+0000","last_fullsized":"2023-01-15T14:35:10.446599+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.21556881899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":489,"reported_epoch":422,"state":"active+clean","last_fresh":"2023-01-15T14:35:12.156248+0000","last_change":"2023-01-15T14:34:43.269231+0000","last_active":"2023-01-15T14:35:12.156248+0000","last_peered":"2023-01-15T14:35:12.156248+0000","last_clean":"2023-01-15T14:35:12.156248+0000","last_became_active":"2023-01-15T14:34:43.268688+0000","last_became_peered":"2023-01-15T14:34:43.268688+0000","last_unstale":"2023-01-15T14:35:12.156248+0000","last_undegraded":"2023-01-15T14:35:12.156248+0000","last_fullsized":"2023-01-15T14:35:12.156248+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":865,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.065940+0000","last_change":"2023-01-15T14:35:12.065940+0000","last_active":"2023-01-15T14:35:12.065940+0000","last_peered":"2023-01-15T14:35:12.065940+0000","last_clean":"2023-01-15T14:35:12.065940+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:35:12.065940+0000","last_undegraded":"2023-01-15T14:35:12.065940+0000","last_fullsized":"2023-01-15T14:35:12.065940+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.97440208699999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":422,"state":"active+clean","last_fresh":"2023-01-15T14:35:12.066975+0000","last_change":"2023-01-15T14:35:11.444424+0000","last_active":"2023-01-15T14:35:12.066975+0000","last_peered":"2023-01-15T14:35:12.066975+0000","last_clean":"2023-01-15T14:35:12.066975+0000","last_became_active":"2023-01-15T14:35:11.443876+0000","last_became_peered":"2023-01-15T14:35:11.443876+0000","last_unstale":"2023-01-15T14:35:12.066975+0000","last_undegraded":"2023-01-15T14:35:12.066975+0000","last_fullsized":"2023-01-15T14:35:12.066975+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1119,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.056914+0000","last_change":"2023-01-15T14:35:13.056914+0000","last_active":"2023-01-15T14:35:13.056914+0000","last_peered":"2023-01-15T14:35:13.056914+0000","last_clean":"2023-01-15T14:35:13.056914+0000","last_became_active":"2023-01-15T14:35:07.358411+0000","last_became_peered":"2023-01-15T14:35:07.358411+0000","last_unstale":"2023-01-15T14:35:13.056914+0000","last_undegraded":"2023-01-15T14:35:13.056914+0000","last_fullsized":"2023-01-15T14:35:13.056914+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:35:01.091016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:01.091016+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:22.866","scrub_duration":0.55784813600000005,"objects_trimmed":0,"snaptrim_duration":0.26454138700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"},{"start":"139","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":422,"state":"active+clean","last_fresh":"2023-01-15T14:35:12.158032+0000","last_change":"2023-01-15T14:35:04.086793+0000","last_active":"2023-01-15T14:35:12.158032+0000","last_peered":"2023-01-15T14:35:12.158032+0000","last_clean":"2023-01-15T14:35:12.158032+0000","last_became_active":"2023-01-15T14:35:04.086427+0000","last_became_peered":"2023-01-15T14:35:04.086427+0000","last_unstale":"2023-01-15T14:35:12.158032+0000","last_undegraded":"2023-01-15T14:35:12.158032+0000","last_fullsized":"2023-01-15T14:35:12.158032+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"408'341","reported_seq":1655,"reported_epoch":424,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:35:14.454803+0000","last_change":"2023-01-15T14:35:12.783113+0000","last_active":"2023-01-15T14:35:14.454803+0000","last_peered":"2023-01-15T14:35:14.454803+0000","last_clean":"2023-01-15T14:35:02.150417+0000","last_became_active":"2023-01-15T14:35:05.274424+0000","last_became_peered":"2023-01-15T14:35:05.274424+0000","last_unstale":"2023-01-15T14:35:14.454803+0000","last_undegraded":"2023-01-15T14:35:14.454803+0000","last_fullsized":"2023-01-15T14:35:04.975432+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":341,"log_dups_size":0,"ondisk_log_size":341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.10529269500000001,"stat_sum":{"num_bytes":7133640,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":13875,"num_write":6,"num_write_kb":7330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":26438071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0],"avail_no_missing":["4","0","1"],"object_location_counts":[{"shards":"0,1,4","objects":1},{"shards":"0,4","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.5","version":"415'189","reported_seq":1393,"reported_epoch":421,"state":"peering","last_fresh":"2023-01-15T14:35:05.185783+0000","last_change":"2023-01-15T14:35:11.546848+0000","last_active":"2023-01-15T14:35:11.546848+0000","last_peered":"2023-01-15T14:35:05.185783+0000","last_clean":"2023-01-15T14:35:02.103725+0000","last_became_active":"2023-01-15T14:35:05.184664+0000","last_became_peered":"2023-01-15T14:35:05.184664+0000","last_unstale":"2023-01-15T14:35:05.185783+0000","last_undegraded":"2023-01-15T14:35:04.981909+0000","last_fullsized":"2023-01-15T14:35:04.981371+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":355,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":189,"log_dups_size":0,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.151584846,"stat_sum":{"num_bytes":13303388,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":25,"num_read_kb":26148,"num_write":22,"num_write_kb":7931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":13260921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":41,"reported_epoch":422,"state":"active+clean","last_fresh":"2023-01-15T14:35:12.194860+0000","last_change":"2023-01-15T14:34:43.371445+0000","last_active":"2023-01-15T14:35:12.194860+0000","last_peered":"2023-01-15T14:35:12.194860+0000","last_clean":"2023-01-15T14:35:12.194860+0000","last_became_active":"2023-01-15T14:34:43.253700+0000","last_became_peered":"2023-01-15T14:34:43.253700+0000","last_unstale":"2023-01-15T14:35:12.194860+0000","last_undegraded":"2023-01-15T14:35:12.194860+0000","last_fullsized":"2023-01-15T14:35:12.194860+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"422'351","reported_seq":1703,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.999090+0000","last_change":"2023-01-15T14:35:12.999090+0000","last_active":"2023-01-15T14:35:12.999090+0000","last_peered":"2023-01-15T14:35:12.999090+0000","last_clean":"2023-01-15T14:35:12.999090+0000","last_became_active":"2023-01-15T14:34:44.164446+0000","last_became_peered":"2023-01-15T14:34:44.164446+0000","last_unstale":"2023-01-15T14:35:12.999090+0000","last_undegraded":"2023-01-15T14:35:12.999090+0000","last_fullsized":"2023-01-15T14:35:12.999090+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:32.407","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0.86415964199999995,"stat_sum":{"num_bytes":11114185,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":11,"num_read_kb":9789,"num_write":5,"num_write_kb":3775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"},{"start":"139","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":471,"reported_epoch":422,"state":"active+clean","last_fresh":"2023-01-15T14:35:12.263411+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:35:12.263411+0000","last_peered":"2023-01-15T14:35:12.263411+0000","last_clean":"2023-01-15T14:35:12.263411+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:35:12.263411+0000","last_undegraded":"2023-01-15T14:35:12.263411+0000","last_fullsized":"2023-01-15T14:35:12.263411+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"423'176","reported_seq":1346,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.694668+0000","last_change":"2023-01-15T14:35:13.612987+0000","last_active":"2023-01-15T14:35:13.694668+0000","last_peered":"2023-01-15T14:35:13.694668+0000","last_clean":"2023-01-15T14:35:13.694668+0000","last_became_active":"2023-01-15T14:34:17.186013+0000","last_became_peered":"2023-01-15T14:34:17.186013+0000","last_unstale":"2023-01-15T14:35:13.694668+0000","last_undegraded":"2023-01-15T14:35:13.694668+0000","last_fullsized":"2023-01-15T14:35:13.694668+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.65466190599999996,"stat_sum":{"num_bytes":14900141,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":55306,"num_write":22,"num_write_kb":10279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3546343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"2.6","version":"0'0","reported_seq":51,"reported_epoch":422,"state":"active+clean","last_fresh":"2023-01-15T14:35:12.295387+0000","last_change":"2023-01-15T14:35:04.384333+0000","last_active":"2023-01-15T14:35:12.295387+0000","last_peered":"2023-01-15T14:35:12.295387+0000","last_clean":"2023-01-15T14:35:12.295387+0000","last_became_active":"2023-01-15T14:35:04.383307+0000","last_became_peered":"2023-01-15T14:35:04.383307+0000","last_unstale":"2023-01-15T14:35:12.295387+0000","last_undegraded":"2023-01-15T14:35:12.295387+0000","last_fullsized":"2023-01-15T14:35:12.295387+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":209,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.050044+0000","last_change":"2023-01-15T14:35:12.050044+0000","last_active":"2023-01-15T14:35:12.050044+0000","last_peered":"2023-01-15T14:35:12.050044+0000","last_clean":"2023-01-15T14:35:12.050044+0000","last_became_active":"2023-01-15T14:34:43.461395+0000","last_became_peered":"2023-01-15T14:34:43.461395+0000","last_unstale":"2023-01-15T14:35:12.050044+0000","last_undegraded":"2023-01-15T14:35:12.050044+0000","last_fullsized":"2023-01-15T14:35:12.050044+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.822281119,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.233031+0000","last_change":"2023-01-15T14:35:04.185970+0000","last_active":"2023-01-15T14:35:10.233031+0000","last_peered":"2023-01-15T14:35:10.233031+0000","last_clean":"2023-01-15T14:35:10.233031+0000","last_became_active":"2023-01-15T14:35:04.185021+0000","last_became_peered":"2023-01-15T14:35:04.185021+0000","last_unstale":"2023-01-15T14:35:10.233031+0000","last_undegraded":"2023-01-15T14:35:10.233031+0000","last_fullsized":"2023-01-15T14:35:10.233031+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"424'174","reported_seq":1391,"reported_epoch":424,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:14.152117+0000","last_change":"2023-01-15T14:35:14.151961+0000","last_active":"2023-01-15T14:35:14.152117+0000","last_peered":"2023-01-15T14:35:14.152117+0000","last_clean":"2023-01-15T14:35:14.152117+0000","last_became_active":"2023-01-15T14:35:14.050115+0000","last_became_peered":"2023-01-15T14:35:14.050115+0000","last_unstale":"2023-01-15T14:35:14.152117+0000","last_undegraded":"2023-01-15T14:35:14.152117+0000","last_fullsized":"2023-01-15T14:35:14.152117+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"351'159","last_scrub_stamp":"2023-01-15T14:34:18.607751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:18.607751+0000","objects_scrubbed":2,"log_size":174,"log_dups_size":0,"ondisk_log_size":174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:42.578","scrub_duration":0.40666956999999998,"objects_trimmed":0,"snaptrim_duration":0.106557613,"stat_sum":{"num_bytes":2453855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2453855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":182,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.719950+0000","last_change":"2023-01-15T14:35:13.720055+0000","last_active":"2023-01-15T14:35:13.719950+0000","last_peered":"2023-01-15T14:35:13.719950+0000","last_clean":"2023-01-15T14:35:13.719950+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:35:13.719950+0000","last_undegraded":"2023-01-15T14:35:13.719950+0000","last_fullsized":"2023-01-15T14:35:13.719950+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.76154624199999998,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.16","version":"421'151","reported_seq":910,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.389570+0000","last_change":"2023-01-15T14:35:13.389725+0000","last_active":"2023-01-15T14:35:13.389570+0000","last_peered":"2023-01-15T14:35:13.389570+0000","last_clean":"2023-01-15T14:35:13.389570+0000","last_became_active":"2023-01-15T14:34:44.491303+0000","last_became_peered":"2023-01-15T14:34:44.491303+0000","last_unstale":"2023-01-15T14:35:13.389570+0000","last_undegraded":"2023-01-15T14:35:13.389570+0000","last_fullsized":"2023-01-15T14:35:13.389570+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.431269034,"stat_sum":{"num_bytes":7378471,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":6898,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9731517,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":222,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:11.968558+0000","last_change":"2023-01-15T14:35:11.968607+0000","last_active":"2023-01-15T14:35:11.968558+0000","last_peered":"2023-01-15T14:35:11.968558+0000","last_clean":"2023-01-15T14:35:11.968558+0000","last_became_active":"2023-01-15T14:35:04.289453+0000","last_became_peered":"2023-01-15T14:35:04.289453+0000","last_unstale":"2023-01-15T14:35:11.968558+0000","last_undegraded":"2023-01-15T14:35:11.968558+0000","last_fullsized":"2023-01-15T14:35:11.968558+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:29.773","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.00075902599999999999,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.18","version":"419'350","reported_seq":1750,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.066392+0000","last_change":"2023-01-15T14:35:11.545959+0000","last_active":"2023-01-15T14:35:12.066392+0000","last_peered":"2023-01-15T14:35:12.066392+0000","last_clean":"2023-01-15T14:35:12.066392+0000","last_became_active":"2023-01-15T14:35:11.544123+0000","last_became_peered":"2023-01-15T14:35:11.544123+0000","last_unstale":"2023-01-15T14:35:12.066392+0000","last_undegraded":"2023-01-15T14:35:12.066392+0000","last_fullsized":"2023-01-15T14:35:12.066392+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":5,"last_scrub":"384'341","last_scrub_stamp":"2023-01-15T14:34:35.477211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:35.477211+0000","objects_scrubbed":5,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:54.983","scrub_duration":0.54215560600000001,"objects_trimmed":0,"snaptrim_duration":0.0026856829999999999,"stat_sum":{"num_bytes":7186347,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.19","version":"422'266","reported_seq":1498,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.097188+0000","last_change":"2023-01-15T14:35:13.097287+0000","last_active":"2023-01-15T14:35:13.097188+0000","last_peered":"2023-01-15T14:35:13.097188+0000","last_clean":"2023-01-15T14:35:13.097188+0000","last_became_active":"2023-01-15T14:34:43.263173+0000","last_became_peered":"2023-01-15T14:34:43.263173+0000","last_unstale":"2023-01-15T14:35:13.097188+0000","last_undegraded":"2023-01-15T14:35:13.097188+0000","last_fullsized":"2023-01-15T14:35:13.097188+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":266,"log_dups_size":0,"ondisk_log_size":266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.137192852,"stat_sum":{"num_bytes":6864537,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":12320,"num_write":9,"num_write_kb":2852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4941423,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"},{"start":"139","length":"1"}]},{"pgid":"3.1a","version":"420'368","reported_seq":1663,"reported_epoch":423,"state":"active+clean","last_fresh":"2023-01-15T14:35:13.173040+0000","last_change":"2023-01-15T14:35:13.173040+0000","last_active":"2023-01-15T14:35:13.173040+0000","last_peered":"2023-01-15T14:35:13.173040+0000","last_clean":"2023-01-15T14:35:13.173040+0000","last_became_active":"2023-01-15T14:34:44.273541+0000","last_became_peered":"2023-01-15T14:34:44.273541+0000","last_unstale":"2023-01-15T14:35:13.173040+0000","last_undegraded":"2023-01-15T14:35:13.173040+0000","last_fullsized":"2023-01-15T14:35:13.173040+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"375'356","last_scrub_stamp":"2023-01-15T14:34:29.460843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:29.460843+0000","objects_scrubbed":4,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:56.905","scrub_duration":0.30741503599999997,"objects_trimmed":0,"snaptrim_duration":0.108068317,"stat_sum":{"num_bytes":4071594,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.1b","version":"420'131","reported_seq":258,"reported_epoch":422,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:12.067977+0000","last_change":"2023-01-15T14:35:11.969100+0000","last_active":"2023-01-15T14:35:12.067977+0000","last_peered":"2023-01-15T14:35:12.067977+0000","last_clean":"2023-01-15T14:35:12.067977+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:35:12.067977+0000","last_undegraded":"2023-01-15T14:35:12.067977+0000","last_fullsized":"2023-01-15T14:35:12.067977+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"370'101","last_scrub_stamp":"2023-01-15T14:34:24.137270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:24.137270+0000","objects_scrubbed":3,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:44.214","scrub_duration":0.106791863,"objects_trimmed":0,"snaptrim_duration":0.20104394,"stat_sum":{"num_bytes":10778275,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":13300,"num_write":8,"num_write_kb":4287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":752,"reported_epoch":423,"state":"active+clean","last_fresh":"2023-01-15T14:35:13.174436+0000","last_change":"2023-01-15T14:35:13.174436+0000","last_active":"2023-01-15T14:35:13.174436+0000","last_peered":"2023-01-15T14:35:13.174436+0000","last_clean":"2023-01-15T14:35:13.174436+0000","last_became_active":"2023-01-15T14:35:04.384767+0000","last_became_peered":"2023-01-15T14:35:04.384767+0000","last_unstale":"2023-01-15T14:35:13.174436+0000","last_undegraded":"2023-01-15T14:35:13.174436+0000","last_fullsized":"2023-01-15T14:35:13.174436+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":5,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:22.440826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:22.440826+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:25.536","scrub_duration":0.30343327599999997,"objects_trimmed":0,"snaptrim_duration":0.108573741,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1498,"reported_epoch":423,"state":"active+clean","last_fresh":"2023-01-15T14:35:13.176623+0000","last_change":"2023-01-15T14:35:13.176623+0000","last_active":"2023-01-15T14:35:13.176623+0000","last_peered":"2023-01-15T14:35:13.176623+0000","last_clean":"2023-01-15T14:35:13.176623+0000","last_became_active":"2023-01-15T14:35:07.268393+0000","last_became_peered":"2023-01-15T14:35:07.268393+0000","last_unstale":"2023-01-15T14:35:13.176623+0000","last_undegraded":"2023-01-15T14:35:13.176623+0000","last_fullsized":"2023-01-15T14:35:13.176623+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":5,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:51.945865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.945865+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:04.107","scrub_duration":0.30386408399999998,"objects_trimmed":0,"snaptrim_duration":0.108211218,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.1e","version":"215'39","reported_seq":1148,"reported_epoch":421,"state":"active+clean","last_fresh":"2023-01-15T14:35:11.246261+0000","last_change":"2023-01-15T14:35:11.246261+0000","last_active":"2023-01-15T14:35:11.246261+0000","last_peered":"2023-01-15T14:35:11.246261+0000","last_clean":"2023-01-15T14:35:11.246261+0000","last_became_active":"2023-01-15T14:35:11.243676+0000","last_became_peered":"2023-01-15T14:35:11.243676+0000","last_unstale":"2023-01-15T14:35:11.246261+0000","last_undegraded":"2023-01-15T14:35:11.246261+0000","last_fullsized":"2023-01-15T14:35:11.246261+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.001455848,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.1f","version":"419'278","reported_seq":1539,"reported_epoch":422,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:12.166520+0000","last_change":"2023-01-15T14:35:10.243135+0000","last_active":"2023-01-15T14:35:12.166520+0000","last_peered":"2023-01-15T14:35:12.166520+0000","last_clean":"2023-01-15T14:35:02.505628+0000","last_became_active":"2023-01-15T14:35:05.184287+0000","last_became_peered":"2023-01-15T14:35:05.184287+0000","last_unstale":"2023-01-15T14:35:12.166520+0000","last_undegraded":"2023-01-15T14:35:04.974202+0000","last_fullsized":"2023-01-15T14:35:04.972901+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:59.076","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.45332477399999999,"stat_sum":{"num_bytes":8515317,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":29110,"num_write":14,"num_write_kb":5117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":8386858,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2],"avail_no_missing":["0","2","4"],"object_location_counts":[{"shards":"0,2","objects":-1},{"shards":"0,2,4","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.20","version":"324'273","reported_seq":1531,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.132979+0000","last_change":"2023-01-15T14:35:10.132979+0000","last_active":"2023-01-15T14:35:10.132979+0000","last_peered":"2023-01-15T14:35:10.132979+0000","last_clean":"2023-01-15T14:35:10.132979+0000","last_became_active":"2023-01-15T14:35:07.459590+0000","last_became_peered":"2023-01-15T14:35:07.459590+0000","last_unstale":"2023-01-15T14:35:10.132979+0000","last_undegraded":"2023-01-15T14:35:10.132979+0000","last_fullsized":"2023-01-15T14:35:10.132979+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"324'273","last_scrub_stamp":"2023-01-15T14:34:49.100502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.100502+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:51.047","scrub_duration":0.204192292,"objects_trimmed":0,"snaptrim_duration":0.002205463,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.21","version":"321'138","reported_seq":998,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.233082+0000","last_change":"2023-01-15T14:35:10.233082+0000","last_active":"2023-01-15T14:35:10.233082+0000","last_peered":"2023-01-15T14:35:10.233082+0000","last_clean":"2023-01-15T14:35:10.233082+0000","last_became_active":"2023-01-15T14:35:07.146982+0000","last_became_peered":"2023-01-15T14:35:07.146982+0000","last_unstale":"2023-01-15T14:35:10.233082+0000","last_undegraded":"2023-01-15T14:35:10.233082+0000","last_fullsized":"2023-01-15T14:35:10.233082+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"321'138","last_scrub_stamp":"2023-01-15T14:34:49.495975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.495975+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.10215977,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6059,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.22","version":"323'161","reported_seq":1308,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:11.965424+0000","last_change":"2023-01-15T14:35:11.448065+0000","last_active":"2023-01-15T14:35:11.965424+0000","last_peered":"2023-01-15T14:35:11.965424+0000","last_clean":"2023-01-15T14:35:11.965424+0000","last_became_active":"2023-01-15T14:35:11.447340+0000","last_became_peered":"2023-01-15T14:35:11.447340+0000","last_unstale":"2023-01-15T14:35:11.965424+0000","last_undegraded":"2023-01-15T14:35:11.965424+0000","last_fullsized":"2023-01-15T14:35:11.965424+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.0011832189999999999,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.23","version":"149'233","reported_seq":902,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.067579+0000","last_change":"2023-01-15T14:35:12.067579+0000","last_active":"2023-01-15T14:35:12.067579+0000","last_peered":"2023-01-15T14:35:12.067579+0000","last_clean":"2023-01-15T14:35:12.067579+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:35:12.067579+0000","last_undegraded":"2023-01-15T14:35:12.067579+0000","last_fullsized":"2023-01-15T14:35:12.067579+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:34:55.674244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:55.674244+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:18.674","scrub_duration":0.50374311999999999,"objects_trimmed":0,"snaptrim_duration":0.001750536,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.24","version":"419'289","reported_seq":1599,"reported_epoch":424,"state":"active+clean","last_fresh":"2023-01-15T14:35:14.251975+0000","last_change":"2023-01-15T14:35:13.173819+0000","last_active":"2023-01-15T14:35:14.251975+0000","last_peered":"2023-01-15T14:35:14.251975+0000","last_clean":"2023-01-15T14:35:14.251975+0000","last_became_active":"2023-01-15T14:35:11.349162+0000","last_became_peered":"2023-01-15T14:35:11.349162+0000","last_unstale":"2023-01-15T14:35:14.251975+0000","last_undegraded":"2023-01-15T14:35:14.251975+0000","last_fullsized":"2023-01-15T14:35:14.251975+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":289,"log_dups_size":0,"ondisk_log_size":289,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.108537855,"stat_sum":{"num_bytes":6590865,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":20,"num_read_kb":22662,"num_write":11,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.25","version":"333'108","reported_seq":1163,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.957463+0000","last_change":"2023-01-15T14:35:12.957463+0000","last_active":"2023-01-15T14:35:12.957463+0000","last_peered":"2023-01-15T14:35:12.957463+0000","last_clean":"2023-01-15T14:35:12.957463+0000","last_became_active":"2023-01-15T14:34:46.346453+0000","last_became_peered":"2023-01-15T14:34:46.346453+0000","last_unstale":"2023-01-15T14:35:12.957463+0000","last_undegraded":"2023-01-15T14:35:12.957463+0000","last_fullsized":"2023-01-15T14:35:12.957463+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'108","last_scrub_stamp":"2023-01-15T14:35:02.975751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:02.975751+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:28.471","scrub_duration":0.30430094200000002,"objects_trimmed":0,"snaptrim_duration":0.26415460699999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"},{"start":"139","length":"1"}]},{"pgid":"3.26","version":"102'52","reported_seq":38,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.339496+0000","last_change":"2023-01-15T14:35:10.339496+0000","last_active":"2023-01-15T14:35:10.339496+0000","last_peered":"2023-01-15T14:35:10.339496+0000","last_clean":"2023-01-15T14:35:10.339496+0000","last_became_active":"2023-01-15T14:35:04.285753+0000","last_became_peered":"2023-01-15T14:35:04.285753+0000","last_unstale":"2023-01-15T14:35:10.339496+0000","last_undegraded":"2023-01-15T14:35:10.339496+0000","last_fullsized":"2023-01-15T14:35:10.339496+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.20527930999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":641,"reported_epoch":424,"state":"active+clean","last_fresh":"2023-01-15T14:35:14.074671+0000","last_change":"2023-01-15T14:35:13.271426+0000","last_active":"2023-01-15T14:35:14.074671+0000","last_peered":"2023-01-15T14:35:14.074671+0000","last_clean":"2023-01-15T14:35:14.074671+0000","last_became_active":"2023-01-15T14:35:11.451119+0000","last_became_peered":"2023-01-15T14:35:11.451119+0000","last_unstale":"2023-01-15T14:35:14.074671+0000","last_undegraded":"2023-01-15T14:35:14.074671+0000","last_fullsized":"2023-01-15T14:35:14.074671+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.30402811800000001,"stat_sum":{"num_bytes":1242779,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":4228,"num_write":1,"num_write_kb":368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.28","version":"333'248","reported_seq":1526,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.056373+0000","last_change":"2023-01-15T14:35:13.056350+0000","last_active":"2023-01-15T14:35:13.056373+0000","last_peered":"2023-01-15T14:35:13.056373+0000","last_clean":"2023-01-15T14:35:13.056373+0000","last_became_active":"2023-01-15T14:35:11.436994+0000","last_became_peered":"2023-01-15T14:35:11.436994+0000","last_unstale":"2023-01-15T14:35:13.056373+0000","last_undegraded":"2023-01-15T14:35:13.056373+0000","last_fullsized":"2023-01-15T14:35:13.056373+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'248","last_scrub_stamp":"2023-01-15T14:34:20.620068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.620068+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:32.103","scrub_duration":0.40448800800000001,"objects_trimmed":0,"snaptrim_duration":0.263432736,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"},{"start":"139","length":"1"}]},{"pgid":"3.29","version":"110'149","reported_seq":685,"reported_epoch":424,"state":"active+clean","last_fresh":"2023-01-15T14:35:14.173891+0000","last_change":"2023-01-15T14:35:12.968059+0000","last_active":"2023-01-15T14:35:14.173891+0000","last_peered":"2023-01-15T14:35:14.173891+0000","last_clean":"2023-01-15T14:35:14.173891+0000","last_became_active":"2023-01-15T14:35:11.345959+0000","last_became_peered":"2023-01-15T14:35:11.345959+0000","last_unstale":"2023-01-15T14:35:14.173891+0000","last_undegraded":"2023-01-15T14:35:14.173891+0000","last_fullsized":"2023-01-15T14:35:14.173891+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40398525099999999,"objects_trimmed":0,"snaptrim_duration":0.0029891319999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1416,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.388546+0000","last_change":"2023-01-15T14:35:13.388663+0000","last_active":"2023-01-15T14:35:13.388546+0000","last_peered":"2023-01-15T14:35:13.388546+0000","last_clean":"2023-01-15T14:35:13.388546+0000","last_became_active":"2023-01-15T14:35:07.358010+0000","last_became_peered":"2023-01-15T14:35:07.358010+0000","last_unstale":"2023-01-15T14:35:13.388546+0000","last_undegraded":"2023-01-15T14:35:13.388546+0000","last_fullsized":"2023-01-15T14:35:13.388546+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.43077436499999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":229,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.065875+0000","last_change":"2023-01-15T14:35:12.065875+0000","last_active":"2023-01-15T14:35:12.065875+0000","last_peered":"2023-01-15T14:35:12.065875+0000","last_clean":"2023-01-15T14:35:12.065875+0000","last_became_active":"2023-01-15T14:34:56.436012+0000","last_became_peered":"2023-01-15T14:34:56.436012+0000","last_unstale":"2023-01-15T14:35:12.065875+0000","last_undegraded":"2023-01-15T14:35:12.065875+0000","last_fullsized":"2023-01-15T14:35:12.065875+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":1.0744738599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":30,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.443676+0000","last_change":"2023-01-15T14:35:10.443676+0000","last_active":"2023-01-15T14:35:10.443676+0000","last_peered":"2023-01-15T14:35:10.443676+0000","last_clean":"2023-01-15T14:35:10.443676+0000","last_became_active":"2023-01-15T14:35:04.386746+0000","last_became_peered":"2023-01-15T14:35:04.386746+0000","last_unstale":"2023-01-15T14:35:10.443676+0000","last_undegraded":"2023-01-15T14:35:10.443676+0000","last_fullsized":"2023-01-15T14:35:10.443676+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:58.535965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:58.535965+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.305370849,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1388,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.387464+0000","last_change":"2023-01-15T14:35:13.387606+0000","last_active":"2023-01-15T14:35:13.387464+0000","last_peered":"2023-01-15T14:35:13.387464+0000","last_clean":"2023-01-15T14:35:13.387464+0000","last_became_active":"2023-01-15T14:35:07.164990+0000","last_became_peered":"2023-01-15T14:35:07.164990+0000","last_unstale":"2023-01-15T14:35:13.387464+0000","last_undegraded":"2023-01-15T14:35:13.387464+0000","last_fullsized":"2023-01-15T14:35:13.387464+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:00.115","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.32751643400000002,"stat_sum":{"num_bytes":1516832,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2709,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.2e","version":"139'38","reported_seq":38,"reported_epoch":420,"state":"active+clean","last_fresh":"2023-01-15T14:35:10.497442+0000","last_change":"2023-01-15T14:35:10.497442+0000","last_active":"2023-01-15T14:35:10.497442+0000","last_peered":"2023-01-15T14:35:10.497442+0000","last_clean":"2023-01-15T14:35:10.497442+0000","last_became_active":"2023-01-15T14:35:04.286907+0000","last_became_peered":"2023-01-15T14:35:04.286907+0000","last_unstale":"2023-01-15T14:35:10.497442+0000","last_undegraded":"2023-01-15T14:35:10.497442+0000","last_fullsized":"2023-01-15T14:35:10.497442+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.361981569,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"102","length":"1"},{"start":"10a","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1282,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:13.269713+0000","last_change":"2023-01-15T14:35:13.269798+0000","last_active":"2023-01-15T14:35:13.269713+0000","last_peered":"2023-01-15T14:35:13.269713+0000","last_clean":"2023-01-15T14:35:13.269713+0000","last_became_active":"2023-01-15T14:35:07.161199+0000","last_became_peered":"2023-01-15T14:35:07.161199+0000","last_unstale":"2023-01-15T14:35:13.269713+0000","last_undegraded":"2023-01-15T14:35:13.269713+0000","last_fullsized":"2023-01-15T14:35:13.269713+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.21017754299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.30","version":"206'252","reported_seq":1160,"reported_epoch":422,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.065852+0000","last_change":"2023-01-15T14:35:12.065852+0000","last_active":"2023-01-15T14:35:12.065852+0000","last_peered":"2023-01-15T14:35:12.065852+0000","last_clean":"2023-01-15T14:35:12.065852+0000","last_became_active":"2023-01-15T14:34:56.539037+0000","last_became_peered":"2023-01-15T14:34:56.539037+0000","last_unstale":"2023-01-15T14:35:12.065852+0000","last_undegraded":"2023-01-15T14:35:12.065852+0000","last_fullsized":"2023-01-15T14:35:12.065852+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:35:03.572774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:03.572774+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:18.492","scrub_duration":0.36120358800000002,"objects_trimmed":0,"snaptrim_duration":0.974840225,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.31","version":"204'150","reported_seq":920,"reported_epoch":423,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:12.956993+0000","last_change":"2023-01-15T14:35:12.956993+0000","last_active":"2023-01-15T14:35:12.956993+0000","last_peered":"2023-01-15T14:35:12.956993+0000","last_clean":"2023-01-15T14:35:12.956993+0000","last_became_active":"2023-01-15T14:34:56.391718+0000","last_became_peered":"2023-01-15T14:34:56.391718+0000","last_unstale":"2023-01-15T14:35:12.956993+0000","last_undegraded":"2023-01-15T14:35:12.956993+0000","last_fullsized":"2023-01-15T14:35:12.956993+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.19766803799999999,"stat_sum":{"num_bytes":3558907,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10724,"num_write":3,"num_write_kb":3089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3605110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"1"},{"start":"139","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":116,"stat_sum":{"num_bytes":436824683,"num_objects":205,"num_object_clones":153,"num_object_copies":615,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":118,"num_objects_misplaced":11,"num_objects_unfound":0,"num_objects_dirty":205,"num_whiteouts":18,"num_read":1028,"num_read_kb":1158061,"num_write":765,"num_write_kb":325393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":738,"num_bytes_recovered":488706748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21249,"ondisk_log_size":21249,"up":348,"acting":338,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574935,"num_pgs":60,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1184336,"kb_used_data":92516092,"kb_used_omap":11321,"kb_used_meta":0,"kb_avail":92515044,"statfs":{"total":95948165120,"available":94735405056,"internally_reserved":1073152,"allocated":94736478208,"data_stored":94736478208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11593415,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":3.452,"15min":3.8639999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.1830000000000001,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":6.0389999999999997,"5min":5.6859999999999999,"15min":8.0340000000000007},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":36.622,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.68100000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":12.064,"5min":4.7549999999999999,"15min":5.9480000000000004},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":204.74199999999999,"5min":204.74199999999999,"15min":204.74199999999999},"last":0.98899999999999999},{"interface":"front","average":{"1min":11.260999999999999,"5min":9.1910000000000007,"15min":7.25},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":94.537000000000006,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.90500000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":6.4119999999999999,"15min":5.3310000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":25.065999999999999,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":15.106,"5min":4.7649999999999997,"15min":4.3730000000000002},"min":{"1min":0.59599999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":162.96799999999999,"5min":162.96799999999999,"15min":162.96799999999999},"last":0.93300000000000005}]},{"osd":3,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.198,"5min":7.6299999999999999,"15min":3.1749999999999998},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":25.359999999999999,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.872},{"interface":"front","average":{"1min":0.86499999999999999,"5min":2.879,"15min":4.0609999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.266,"5min":98.825000000000003,"15min":98.825000000000003},"last":1.0349999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.4430000000000001,"5min":4.2279999999999998,"15min":7.6859999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":29.690000000000001,"5min":96.185000000000002,"15min":96.185000000000002},"last":40.563000000000002},{"interface":"front","average":{"1min":10.949999999999999,"5min":7.7789999999999999,"15min":3.2360000000000002},"min":{"1min":0.46200000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":142.44999999999999,"5min":164.21199999999999,"15min":164.21199999999999},"last":0.74199999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.5489999999999999,"5min":4.3099999999999996,"15min":5.6539999999999999},"min":{"1min":0.63800000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":29.797999999999998,"5min":96.263999999999996,"15min":96.263999999999996},"last":0.83999999999999997},{"interface":"front","average":{"1min":2.3519999999999999,"5min":2.6869999999999998,"15min":4.1920000000000002},"min":{"1min":0.55300000000000005,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":25.199000000000002,"5min":70.387,"15min":95.430000000000007},"last":1.01}]},{"osd":6,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":2.645,"5min":4.4189999999999996,"15min":7.1550000000000002},"min":{"1min":0.60999999999999999,"5min":0.496,"15min":0.496},"max":{"1min":29.876000000000001,"5min":96.299999999999997,"15min":96.299999999999997},"last":0.97399999999999998},{"interface":"front","average":{"1min":2.399,"5min":2.7749999999999999,"15min":1.6579999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":25.236999999999998,"5min":70.530000000000001,"15min":70.530000000000001},"last":1.073}]}]},{"osd":6,"up_from":13,"seq":55834574936,"num_pgs":61,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":916724,"kb_used_data":92825100,"kb_used_omap":3817,"kb_used_meta":0,"kb_avail":92782656,"statfs":{"total":95948165120,"available":95009439744,"internally_reserved":43462656,"allocated":95052902400,"data_stored":95052902400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3909529,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":2.0710000000000002,"15min":1.175},"min":{"1min":0.44600000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.1919999999999999,"5min":122.012,"15min":122.012},"last":166.19900000000001},{"interface":"front","average":{"1min":0.82099999999999995,"5min":1.0189999999999999,"15min":0.85699999999999998},"min":{"1min":0.50900000000000001,"5min":0.48199999999999998,"15min":0.47699999999999998},"max":{"1min":1.0469999999999999,"5min":19.413,"15min":19.413},"last":0.42299999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":4.7969999999999997,"15min":2.1579999999999999},"min":{"1min":0.58499999999999996,"5min":0.47499999999999998,"15min":0.436},"max":{"1min":1.2210000000000001,"5min":99.203000000000003,"15min":99.203000000000003},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.876,"5min":3.9860000000000002,"15min":1.909},"min":{"1min":0.61799999999999999,"5min":0.48499999999999999,"15min":0.439},"max":{"1min":1.357,"5min":98.882000000000005,"15min":98.882000000000005},"last":0.76400000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":2.8839999999999999,"15min":1.5129999999999999},"min":{"1min":0.58399999999999996,"5min":0.45600000000000002,"15min":0.41399999999999998},"max":{"1min":1.125,"5min":128.173,"15min":128.173},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.91700000000000004,"5min":3.3879999999999999,"15min":1.748},"min":{"1min":0.59799999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.2829999999999999,"5min":114.724,"15min":114.724},"last":0.58799999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":2.7309999999999999,"15min":4.9489999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1779999999999999,"5min":93.710999999999999,"15min":93.710999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.93899999999999995,"5min":3.625,"15min":8.8450000000000006},"min":{"1min":0.64000000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.175,"5min":167.178,"15min":255.24299999999999},"last":0.67300000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":4.5010000000000003,"5min":19.122,"15min":6.9530000000000003},"min":{"1min":0.60399999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":83.216999999999999,"5min":1256.867,"15min":1256.867},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.98099999999999998,"5min":15.724,"15min":9.6859999999999999},"min":{"1min":0.67100000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.385,"5min":1255.77,"15min":1255.77},"last":0.49399999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.786,"15min":1.222},"min":{"1min":0.67400000000000004,"5min":0.59899999999999998,"15min":0.44400000000000001},"max":{"1min":1.349,"5min":67.469999999999999,"15min":67.469999999999999},"last":0.629},{"interface":"front","average":{"1min":1.012,"5min":0.94599999999999995,"15min":0.95399999999999996},"min":{"1min":0.68600000000000005,"5min":0.58299999999999996,"15min":0.46100000000000002},"max":{"1min":1.609,"5min":1.8819999999999999,"15min":1.8819999999999999},"last":0.70599999999999996}]},{"osd":7,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.2569999999999999,"15min":3.2719999999999998},"min":{"1min":0.75,"5min":0.628,"15min":0.51300000000000001},"max":{"1min":1.3200000000000001,"5min":19.571000000000002,"15min":78.888000000000005},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.3089999999999999,"5min":3.5739999999999998,"15min":3.2080000000000002},"min":{"1min":0.90300000000000002,"5min":0.70899999999999996,"15min":0.499},"max":{"1min":5.6100000000000003,"5min":121.297,"15min":121.297},"last":0.878}]}]},{"osd":5,"up_from":0,"seq":55834574937,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574935,"num_pgs":67,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":975440,"kb_used_data":92724060,"kb_used_omap":17350,"kb_used_meta":0,"kb_avail":92723940,"statfs":{"total":95948165120,"available":94949314560,"internally_reserved":122880,"allocated":94949437440,"data_stored":94949437440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17766766,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":22.818000000000001,"15min":10.882},"min":{"1min":0.505,"5min":0.44,"15min":0.44},"max":{"1min":5.0049999999999999,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":1.107},{"interface":"front","average":{"1min":0.77400000000000002,"5min":2.9990000000000001,"15min":1.52},"min":{"1min":0.51200000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.127,"5min":88.600999999999999,"15min":88.600999999999999},"last":0.98699999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.9390000000000001,"5min":3.742,"15min":6.976},"min":{"1min":0.499,"5min":0.49299999999999999,"15min":0.47399999999999998},"max":{"1min":23.193000000000001,"5min":171.91499999999999,"15min":171.91499999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":10.99,"5min":6.3819999999999997,"15min":2.6560000000000001},"min":{"1min":0.67100000000000004,"5min":0.52900000000000003,"15min":0.50800000000000001},"max":{"1min":162.976,"5min":162.976,"15min":162.976},"last":0.72699999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.0950000000000002,"5min":23.544,"15min":11.522},"min":{"1min":0.497,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":23.247,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":1.26,"5min":1.28,"15min":1.0329999999999999},"min":{"1min":0.61899999999999999,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":7.5789999999999997,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.86399999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":2.802,"15min":1.8260000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.006,"5min":88.823999999999998,"15min":88.823999999999998},"last":0.91100000000000003},{"interface":"front","average":{"1min":5.3380000000000001,"5min":24.337,"15min":12.212999999999999},"min":{"1min":0.68600000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":58.317,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":1.0780000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":0.90100000000000002,"5min":0.88100000000000001,"15min":0.90700000000000003},"min":{"1min":0.61499999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.175,"5min":2.577,"15min":2.577},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":1.903,"15min":2.1669999999999998},"min":{"1min":0.69099999999999995,"5min":0.65200000000000002,"15min":0.55300000000000005},"max":{"1min":1.0700000000000001,"5min":88.911000000000001,"15min":88.911000000000001},"last":1.129}]},{"osd":6,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":3.2389999999999999,"15min":4.8609999999999998},"min":{"1min":0.59899999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":23.329000000000001,"5min":85.721999999999994,"15min":99.780000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.999,"5min":21.631,"15min":10.651999999999999},"min":{"1min":0.79500000000000004,"5min":0.66700000000000004,"15min":0.58699999999999997},"max":{"1min":1.173,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.1739999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:34:14 2023","interfaces":[{"interface":"back","average":{"1min":3.238,"5min":3.5790000000000002,"15min":9.9450000000000003},"min":{"1min":0.73999999999999999,"5min":0.63400000000000001,"15min":0.59499999999999997},"max":{"1min":42.551000000000002,"5min":89.061000000000007,"15min":179.73500000000001},"last":1.0509999999999999},{"interface":"front","average":{"1min":2.274,"5min":23.071000000000002,"15min":15.183},"min":{"1min":0.78700000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":23.356000000000002,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.0029999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574935,"num_pgs":54,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1090756,"kb_used_data":92608764,"kb_used_omap":17584,"kb_used_meta":0,"kb_avail":92608624,"statfs":{"total":95948165120,"available":94831230976,"internally_reserved":143360,"allocated":94831374336,"data_stored":94831374336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18006308,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":7.8860000000000001,"5min":8.048,"15min":5.9139999999999997},"min":{"1min":0.39500000000000002,"5min":0.255,"15min":0.255},"max":{"1min":74.245999999999995,"5min":174.52199999999999,"15min":174.52199999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":7.9649999999999999,"5min":3.4990000000000001,"15min":6.2610000000000001},"min":{"1min":0.46600000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":74.280000000000001,"5min":74.280000000000001,"15min":93.674000000000007},"last":0.53600000000000003}]},{"osd":1,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":10.762,"5min":6.625,"15min":11.228999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":117.664,"5min":117.664,"15min":159.256},"last":0.54900000000000004},{"interface":"front","average":{"1min":4.6909999999999998,"5min":11.208,"15min":4.2300000000000004},"min":{"1min":0.41899999999999998,"5min":0.317,"15min":0.317},"max":{"1min":47.963000000000001,"5min":174.578,"15min":174.578},"last":35.75}]},{"osd":2,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":2.4820000000000002,"15min":3.0600000000000001},"min":{"1min":0.34200000000000003,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":13.113,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.63500000000000001},{"interface":"front","average":{"1min":2.3079999999999998,"5min":6.8120000000000003,"15min":3.7320000000000002},"min":{"1min":0.44,"5min":0.375,"15min":0.375},"max":{"1min":15.946999999999999,"5min":174.45099999999999,"15min":174.45099999999999},"last":0.84299999999999997}]},{"osd":4,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":28.689,"15min":13.936999999999999},"min":{"1min":0.56299999999999994,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":14.234999999999999,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":11.906000000000001,"5min":31.741,"15min":19.829000000000001},"min":{"1min":0.56000000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":117.59,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":0.70999999999999996}]},{"osd":5,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":11.022,"5min":5.9219999999999997,"15min":11.242000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":117.744,"5min":117.744,"15min":159.43000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.91700000000000004,"5min":5.9829999999999997,"15min":2.6749999999999998},"min":{"1min":0.66200000000000003,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.2210000000000001,"5min":174.61000000000001,"15min":174.61000000000001},"last":0.88100000000000001}]},{"osd":6,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":5.984,"15min":2.6819999999999999},"min":{"1min":0.68300000000000005,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.286,"5min":174.63999999999999,"15min":174.63999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.93400000000000005,"5min":1.3979999999999999,"15min":2.9740000000000002},"min":{"1min":0.71099999999999997,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6799999999999999,"5min":43.661000000000001,"15min":60.098999999999997},"last":0.94399999999999995}]},{"osd":7,"last update":"Sun Jan 15 14:34:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.4119999999999999,"15min":3.161},"min":{"1min":0.753,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.758,"5min":43.712000000000003,"15min":60.165999999999997},"last":96.635000000000005},{"interface":"front","average":{"1min":19.324999999999999,"5min":11.952,"15min":16.044},"min":{"1min":0.65400000000000003,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":142.02000000000001,"5min":281.13400000000001,"15min":281.13400000000001},"last":0.94899999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574935,"num_pgs":44,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1048364,"kb_used_data":92653256,"kb_used_omap":14682,"kb_used_meta":0,"kb_avail":92651016,"statfs":{"total":95948165120,"available":94874640384,"internally_reserved":2293760,"allocated":94876934144,"data_stored":94876934144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15035153,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":5.6779999999999999,"5min":4.7789999999999999,"15min":2.1779999999999999},"min":{"1min":0.32200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":98.227000000000004,"5min":98.227000000000004,"15min":98.227000000000004},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.393,"15min":0.93200000000000005},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.82999999999999996,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.46700000000000003}]},{"osd":1,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":10.779999999999999,"5min":7.5599999999999996,"15min":3.7959999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":87.209999999999994,"5min":128.749,"15min":143.071},"last":0.502},{"interface":"front","average":{"1min":3.5470000000000002,"5min":2.5289999999999999,"15min":2.9049999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":56.457999999999998,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.62}]},{"osd":3,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":7.8970000000000002,"5min":4.4180000000000001,"15min":9.0730000000000004},"min":{"1min":0.35099999999999998,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":78.359999999999999,"5min":86.191999999999993,"15min":183.53},"last":0.59799999999999998},{"interface":"front","average":{"1min":5.7469999999999999,"5min":6.4530000000000003,"15min":3.3439999999999999},"min":{"1min":0.36599999999999999,"5min":0.316,"15min":0.316},"max":{"1min":98.295000000000002,"5min":155.47900000000001,"15min":195.16399999999999},"last":0.41099999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":5.9390000000000001,"5min":4.5730000000000004,"15min":4.5679999999999996},"min":{"1min":0.64300000000000002,"5min":0.624,"15min":0.58399999999999996},"max":{"1min":98.375,"5min":98.375,"15min":175.34700000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":2.323,"15min":3.911},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.1359999999999999,"5min":88.459999999999994,"15min":175.30500000000001},"last":0.96399999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":1.23,"15min":1.677},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1100000000000001,"5min":34.256999999999998,"15min":143.262},"last":0.65800000000000003},{"interface":"front","average":{"1min":5.9939999999999998,"5min":4.0090000000000003,"15min":2.1000000000000001},"min":{"1min":0.58599999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":98.427000000000007,"5min":98.427000000000007,"15min":98.427000000000007},"last":0.877}]},{"osd":6,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.75,"15min":1.292},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.175,"5min":88.542000000000002,"15min":88.542000000000002},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.3129999999999999,"15min":1.76},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.228,"5min":34.305999999999997,"15min":143.215},"last":0.90800000000000003}]},{"osd":7,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":14.125,"5min":10.615,"15min":5.4370000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":114.056,"5min":176.44499999999999,"15min":176.44499999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.3149999999999999,"15min":2.6669999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2490000000000001,"5min":34.207999999999998,"15min":143.17099999999999},"last":0.76200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":49,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1025520,"kb_used_data":92674108,"kb_used_omap":16458,"kb_used_meta":0,"kb_avail":92673860,"statfs":{"total":95948165120,"available":94898032640,"internally_reserved":253952,"allocated":94898286592,"data_stored":94898286592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16853383,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,4,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":2.0619999999999998,"5min":3.2669999999999999,"15min":3.6840000000000002},"min":{"1min":0.37,"5min":0.314,"15min":0.314},"max":{"1min":23.68,"5min":144.12799999999999,"15min":144.12799999999999},"last":57.098999999999997},{"interface":"front","average":{"1min":14.157,"5min":6.5789999999999997,"15min":5.0750000000000002},"min":{"1min":0.42299999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":176.01599999999999,"5min":176.01599999999999,"15min":176.01599999999999},"last":0.67100000000000004}]},{"osd":2,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":4.8959999999999999,"5min":3.4329999999999998,"15min":1.7889999999999999},"min":{"1min":0.44400000000000001,"5min":0.245,"15min":0.245},"max":{"1min":69.277000000000001,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.629,"5min":1.173,"15min":3.1059999999999999},"min":{"1min":0.28399999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.873,"5min":48.119999999999997,"15min":48.119999999999997},"last":0.58999999999999997}]},{"osd":3,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":5.8440000000000003,"15min":15.172000000000001},"min":{"1min":0.38400000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.92500000000000004,"5min":245.637,"15min":282.01400000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":14.228999999999999,"5min":8.2870000000000008,"15min":7.5919999999999996},"min":{"1min":0.46500000000000002,"5min":0.253,"15min":0.253},"max":{"1min":176.13900000000001,"5min":176.13900000000001,"15min":176.13900000000001},"last":0.61399999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:34:18 2023","interfaces":[{"interface":"back","average":{"1min":19.170000000000002,"5min":20.029,"15min":9.8279999999999994},"min":{"1min":0.56299999999999994,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":176.18600000000001,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":5.5510000000000002,"5min":15.916,"15min":7.7080000000000002},"min":{"1min":0.64300000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":69.611999999999995,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.78800000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":5.2060000000000004,"5min":2.3799999999999999,"15min":1.482},"min":{"1min":0.56899999999999995,"5min":0.433,"15min":0.433},"max":{"1min":69.478999999999999,"5min":69.478999999999999,"15min":69.478999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":3.2770000000000001},"min":{"1min":0.66000000000000003,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3089999999999999,"5min":4.0780000000000003,"15min":46.905999999999999},"last":1.036}]},{"osd":6,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":14.491,"5min":6.9109999999999996,"15min":5.6379999999999999},"min":{"1min":0.73199999999999998,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":176.249,"5min":176.249,"15min":176.249},"last":0.84299999999999997},{"interface":"front","average":{"1min":5.2629999999999999,"5min":2.431,"15min":1.5589999999999999},"min":{"1min":0.60399999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":69.549999999999997,"5min":69.549999999999997,"15min":69.549999999999997},"last":1.169}]},{"osd":7,"last update":"Sun Jan 15 14:34:12 2023","interfaces":[{"interface":"back","average":{"1min":7.617,"5min":6.2309999999999999,"15min":6.7549999999999999},"min":{"1min":0.63900000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":95.888999999999996,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":14.544,"5min":6.9649999999999999,"15min":5.4880000000000004},"min":{"1min":0.78700000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":176.22,"5min":176.22,"15min":176.22},"last":1.012}]}]},{"osd":0,"up_from":13,"seq":55834574936,"num_pgs":71,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1145656,"kb_used_data":92561224,"kb_used_omap":11549,"kb_used_meta":0,"kb_avail":92553724,"statfs":{"total":95948165120,"available":94775013376,"internally_reserved":7680000,"allocated":94782693376,"data_stored":94782693376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11826800,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.7410000000000001,"5min":3.3460000000000001,"15min":1.53},"min":{"1min":0.34899999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":25.960999999999999,"5min":69.872,"15min":69.872},"last":0.41299999999999998},{"interface":"front","average":{"1min":11.249000000000001,"5min":5.335,"15min":2.2650000000000001},"min":{"1min":0.434,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":162.91200000000001,"5min":162.91200000000001,"15min":162.91200000000001},"last":100.407}]},{"osd":2,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":3.415,"15min":1.601},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.0109999999999999,"5min":98.286000000000001,"15min":98.286000000000001},"last":100.58799999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":2.6320000000000001,"15min":1.3500000000000001},"min":{"1min":0.37,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86899999999999999,"5min":71.406999999999996,"15min":71.406999999999996},"last":0.46600000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.8740000000000001,"5min":7.8079999999999998,"15min":13.788},"min":{"1min":0.48199999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":26.027000000000001,"5min":199.08000000000001,"15min":284.62799999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":3.6070000000000002,"5min":7.867,"15min":3.1560000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.33600000000000002},"max":{"1min":61.119,"5min":174.09399999999999,"15min":174.09399999999999},"last":100.371}]},{"osd":4,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":16.664999999999999,"15min":6.2119999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.1719999999999999,"5min":1486.309,"15min":1486.309},"last":0.78700000000000003},{"interface":"front","average":{"1min":2.1030000000000002,"5min":18.140999999999998,"15min":6.7300000000000004},"min":{"1min":0.63100000000000001,"5min":0.439,"15min":0.439},"max":{"1min":26.137,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":0.88200000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":2.8980000000000001,"15min":7.4980000000000002},"min":{"1min":0.60799999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1930000000000001,"5min":98.411000000000001,"15min":176.34399999999999},"last":100.46299999999999},{"interface":"front","average":{"1min":0.92000000000000004,"5min":1.7070000000000001,"15min":1.2609999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.101,"5min":71.563999999999993,"15min":71.563999999999993},"last":0.70499999999999996}]},{"osd":6,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":2.1600000000000001,"5min":3.7890000000000001,"15min":2.0430000000000001},"min":{"1min":0.71899999999999997,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":26.082000000000001,"5min":70.259,"15min":70.259},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.026,"5min":3,"15min":1.756},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.57499999999999996},"max":{"1min":1.5409999999999999,"5min":98.484999999999999,"15min":98.484999999999999},"last":100.544}]},{"osd":7,"last update":"Sun Jan 15 14:34:15 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":3.2770000000000001,"15min":11.539},"min":{"1min":0.70099999999999996,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":11.906000000000001,"5min":73.864999999999995,"15min":292.61099999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":2.1819999999999999,"5min":3.8090000000000002,"15min":2.0499999999999998},"min":{"1min":0.69299999999999995,"5min":0.67100000000000004,"15min":0.58699999999999997},"max":{"1min":26.187000000000001,"5min":70.322000000000003,"15min":70.322000000000003},"last":0.94999999999999996}]}]}],"pool_statfs":[]}} 2023-01-15T14:35:19.896 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:35:20.053 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 136 2023-01-15T14:35:20.054 INFO:tasks.rados.rados.0.smithi090.stdout:1569: finishing rollback tid 1 to smithi09025264-1 2023-01-15T14:35:20.054 INFO:tasks.rados.rados.0.smithi090.stdout:1569: finishing rollback tid 2 to smithi09025264-1 2023-01-15T14:35:20.055 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 469 (ObjNum 525 snap 158 seq_num 525) dirty exists 2023-01-15T14:35:20.055 INFO:tasks.rados.rados.0.smithi090.stdout:1569: done (0 left) 2023-01-15T14:35:20.055 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:35:20.056 INFO:tasks.rados.rados.0.smithi090.stdout:1605: done (0 left) 2023-01-15T14:35:20.056 INFO:tasks.rados.rados.0.smithi090.stdout:1606: write (excl) oid 19 current snap is 177 2023-01-15T14:35:20.056 INFO:tasks.rados.rados.0.smithi090.stdout:1606: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2023-01-15T14:35:20.056 INFO:tasks.rados.rados.0.smithi090.stdout:1606: writing smithi09025264-19 from 779990 to 1482042 tid 1 2023-01-15T14:35:20.056 INFO:tasks.rados.rados.0.smithi090.stdout:1606: writing smithi09025264-19 from 2055442 to 2708635 tid 2 2023-01-15T14:35:20.059 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:20.059 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:20.060 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:35:20.061 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:20.053+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:20.064 INFO:tasks.rados.rados.0.smithi090.stdout:1606: writing smithi09025264-19 from 3253694 to 3579990 tid 3 2023-01-15T14:35:20.064 INFO:tasks.rados.rados.0.smithi090.stdout:1607: snap_remove snap 149 2023-01-15T14:35:20.064 INFO:tasks.rados.rados.0.smithi090.stdout:1606: finishing write tid 1 to smithi09025264-19 2023-01-15T14:35:20.064 INFO:tasks.rados.rados.0.smithi090.stdout:1606: finishing write tid 2 to smithi09025264-19 2023-01-15T14:35:20.064 INFO:tasks.rados.rados.0.smithi090.stdout:1606: finishing write tid 3 to smithi09025264-19 2023-01-15T14:35:20.064 INFO:tasks.rados.rados.0.smithi090.stdout:1606: finishing write tid 4 to smithi09025264-19 2023-01-15T14:35:20.064 INFO:tasks.rados.rados.0.smithi090.stdout:1606: finishing write tid 5 to smithi09025264-19 2023-01-15T14:35:20.065 INFO:tasks.rados.rados.0.smithi090.stdout:1606: oid 19 updating version 0 to 286 2023-01-15T14:35:20.065 INFO:tasks.rados.rados.0.smithi090.stdout:1606: oid 19 version 286 is already newer than 285 2023-01-15T14:35:20.065 INFO:tasks.rados.rados.0.smithi090.stdout:1606: oid 19 version 286 is already newer than 284 2023-01-15T14:35:20.065 INFO:tasks.rados.rados.0.smithi090.stdout:1606: oid 19 updating version 286 to 287 2023-01-15T14:35:20.065 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 287 (ObjNum 570 snap 177 seq_num 570) dirty exists 2023-01-15T14:35:20.065 INFO:tasks.rados.rados.0.smithi090.stdout:1606: left oid 19 (ObjNum 570 snap 177 seq_num 570) 2023-01-15T14:35:20.065 INFO:tasks.rados.rados.0.smithi090.stdout:1606: done (1 left) 2023-01-15T14:35:20.065 INFO:tasks.rados.rados.0.smithi090.stdout:1607: done (0 left) 2023-01-15T14:35:20.065 INFO:tasks.rados.rados.0.smithi090.stdout:1608: rollback oid 44 current snap is 177 2023-01-15T14:35:20.264 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 44 to 151 2023-01-15T14:35:20.265 INFO:tasks.rados.rados.0.smithi090.stdout:1609: write (excl) oid 8 current snap is 177 2023-01-15T14:35:20.265 INFO:tasks.rados.rados.0.smithi090.stdout:1609: seq_num 571 ranges {796797=493654,1830277=796851,3397470=199327} 2023-01-15T14:35:20.265 INFO:tasks.rados.rados.0.smithi090.stdout:1609: writing smithi09025264-8 from 796797 to 1290451 tid 1 2023-01-15T14:35:20.265 INFO:tasks.rados.rados.0.smithi090.stdout:1609: writing smithi09025264-8 from 1830277 to 2627128 tid 2 2023-01-15T14:35:20.265 INFO:tasks.rados.rados.0.smithi090.stdout:1609: writing smithi09025264-8 from 3397470 to 3596797 tid 3 2023-01-15T14:35:20.265 INFO:tasks.rados.rados.0.smithi090.stdout:1610: delete oid 18 current snap is 177 2023-01-15T14:35:20.266 INFO:tasks.rados.rados.0.smithi090.stdout:1610: done (2 left) 2023-01-15T14:35:20.266 INFO:tasks.rados.rados.0.smithi090.stdout:1611: copy_from oid 16 from oid 11 current snap is 177 2023-01-15T14:35:20.266 INFO:tasks.rados.rados.0.smithi090.stdout:1612: delete oid 34 current snap is 177 2023-01-15T14:35:20.266 INFO:tasks.rados.rados.0.smithi090.stdout:1608: finishing rollback tid 1 to smithi09025264-44 2023-01-15T14:35:20.266 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 350 (ObjNum 273 snap 0 seq_num 0) dirty dne 2023-01-15T14:35:20.266 INFO:tasks.rados.rados.0.smithi090.stdout:1609: finishing write tid 1 to smithi09025264-8 2023-01-15T14:35:20.266 INFO:tasks.rados.rados.0.smithi090.stdout:1608: done (3 left) 2023-01-15T14:35:20.267 INFO:tasks.rados.rados.0.smithi090.stdout:1612: done (2 left) 2023-01-15T14:35:20.267 INFO:tasks.rados.rados.0.smithi090.stdout:1613: delete oid 44 current snap is 177 2023-01-15T14:35:20.267 INFO:tasks.rados.rados.0.smithi090.stdout:1613: done (2 left) 2023-01-15T14:35:20.267 INFO:tasks.rados.rados.0.smithi090.stdout:1614: read oid 32 snap -1 2023-01-15T14:35:20.267 INFO:tasks.rados.rados.0.smithi090.stdout:1614: expect deleted 2023-01-15T14:35:20.267 INFO:tasks.rados.rados.0.smithi090.stdout:1615: read oid 19 snap -1 2023-01-15T14:35:20.267 INFO:tasks.rados.rados.0.smithi090.stdout:1615: expect (ObjNum 570 snap 177 seq_num 570) 2023-01-15T14:35:20.268 INFO:tasks.rados.rados.0.smithi090.stdout:1616: write (excl) oid 44 current snap is 177 2023-01-15T14:35:20.268 INFO:tasks.rados.rados.0.smithi090.stdout:1616: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2023-01-15T14:35:20.268 INFO:tasks.rados.rados.0.smithi090.stdout:1616: writing smithi09025264-44 from 464025 to 1258675 tid 1 2023-01-15T14:35:20.268 INFO:tasks.rados.rados.0.smithi090.stdout:1616: writing smithi09025264-44 from 1947852 to 2352041 tid 2 2023-01-15T14:35:20.268 INFO:tasks.rados.rados.0.smithi090.stdout:1616: writing smithi09025264-44 from 2990809 to 3416334 tid 3 2023-01-15T14:35:20.271 INFO:tasks.rados.rados.0.smithi090.stdout:1617: snap_create 2023-01-15T14:35:20.272 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:35:20.272 INFO:tasks.rados.rados.0.smithi090.stdout:1615: expect (ObjNum 570 snap 177 seq_num 570) 2023-01-15T14:35:20.272 INFO:tasks.rados.rados.0.smithi090.stdout:1609: finishing write tid 2 to smithi09025264-8 2023-01-15T14:35:20.272 INFO:tasks.rados.rados.0.smithi090.stdout:1609: finishing write tid 3 to smithi09025264-8 2023-01-15T14:35:20.272 INFO:tasks.rados.rados.0.smithi090.stdout:1609: finishing write tid 4 to smithi09025264-8 2023-01-15T14:35:20.272 INFO:tasks.rados.rados.0.smithi090.stdout:1609: finishing write tid 5 to smithi09025264-8 2023-01-15T14:35:20.272 INFO:tasks.rados.rados.0.smithi090.stdout:1609: oid 8 updating version 0 to 345 2023-01-15T14:35:20.273 INFO:tasks.rados.rados.0.smithi090.stdout:1609: oid 8 version 345 is already newer than 342 2023-01-15T14:35:20.273 INFO:tasks.rados.rados.0.smithi090.stdout:1609: oid 8 version 345 is already newer than 343 2023-01-15T14:35:20.273 INFO:tasks.rados.rados.0.smithi090.stdout:1609: oid 8 version 345 is already newer than 344 2023-01-15T14:35:20.273 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 8 v 345 (ObjNum 571 snap 177 seq_num 571) dirty exists 2023-01-15T14:35:20.273 INFO:tasks.rados.rados.0.smithi090.stdout:1609: left oid 8 (ObjNum 571 snap 177 seq_num 571) 2023-01-15T14:35:20.273 INFO:tasks.rados.rados.0.smithi090.stdout:1609: done (4 left) 2023-01-15T14:35:20.274 INFO:tasks.rados.rados.0.smithi090.stdout:1615: done (3 left) 2023-01-15T14:35:20.274 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:35:20.274 INFO:tasks.rados.rados.0.smithi090.stdout:1611: finishing copy_from to smithi09025264-16 2023-01-15T14:35:20.274 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 349 (ObjNum 498 snap 146 seq_num 498) dirty exists 2023-01-15T14:35:20.274 INFO:tasks.rados.rados.0.smithi090.stdout:1611: finishing copy_from racing read to smithi09025264-16 2023-01-15T14:35:20.275 INFO:tasks.rados.rados.0.smithi090.stdout:1616: finishing write tid 1 to smithi09025264-44 2023-01-15T14:35:20.275 INFO:tasks.rados.rados.0.smithi090.stdout:1611: done (2 left) 2023-01-15T14:35:20.275 INFO:tasks.rados.rados.0.smithi090.stdout:1614: done (1 left) 2023-01-15T14:35:20.275 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:20.275 INFO:tasks.rados.rados.0.smithi090.stdout:1616: finishing write tid 2 to smithi09025264-44 2023-01-15T14:35:20.276 INFO:tasks.rados.rados.0.smithi090.stdout:1616: finishing write tid 3 to smithi09025264-44 2023-01-15T14:35:20.276 INFO:tasks.rados.rados.0.smithi090.stdout:1616: finishing write tid 4 to smithi09025264-44 2023-01-15T14:35:20.276 INFO:tasks.rados.rados.0.smithi090.stdout:1616: finishing write tid 5 to smithi09025264-44 2023-01-15T14:35:20.276 INFO:tasks.rados.rados.0.smithi090.stdout:1616: oid 44 updating version 0 to 364 2023-01-15T14:35:20.276 INFO:tasks.rados.rados.0.smithi090.stdout:1616: oid 44 updating version 364 to 367 2023-01-15T14:35:20.276 INFO:tasks.rados.rados.0.smithi090.stdout:1616: oid 44 version 367 is already newer than 365 2023-01-15T14:35:20.276 INFO:tasks.rados.rados.0.smithi090.stdout:1616: oid 44 version 367 is already newer than 366 2023-01-15T14:35:20.276 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 367 (ObjNum 575 snap 177 seq_num 575) dirty exists 2023-01-15T14:35:20.276 INFO:tasks.rados.rados.0.smithi090.stdout:1616: left oid 44 (ObjNum 575 snap 177 seq_num 575) 2023-01-15T14:35:20.277 INFO:tasks.rados.rados.0.smithi090.stdout:1616: done (0 left) 2023-01-15T14:35:20.277 INFO:tasks.rados.rados.0.smithi090.stdout:1617: done (0 left) 2023-01-15T14:35:20.278 INFO:tasks.rados.rados.0.smithi090.stdout:1618: snap_remove snap 174 2023-01-15T14:35:20.281 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:20.273+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:20.380 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:20.373+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:20.428 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:20.429 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:20.429 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:35:20.481 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:20.477+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:20.585 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:20.586 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:20.586 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:20.578+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:20.598 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:35:20.687 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:20.681+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:20.747 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:20.748 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:20.748 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:35:20.788 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:20.781+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:20.855 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:20.953 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:20.955 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:35:20.958 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:20.949+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:21.002 INFO:tasks.rados.rados.0.smithi090.stdout:1618: done (0 left) 2023-01-15T14:35:21.002 INFO:tasks.rados.rados.0.smithi090.stdout:1619: read oid 28 snap -1 2023-01-15T14:35:21.002 INFO:tasks.rados.rados.0.smithi090.stdout:1619: expect (ObjNum 466 snap 134 seq_num 466) 2023-01-15T14:35:21.003 INFO:tasks.rados.rados.0.smithi090.stdout:1620: write (excl) oid 13 current snap is 178 2023-01-15T14:35:21.003 INFO:tasks.rados.rados.0.smithi090.stdout:1620: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2023-01-15T14:35:21.003 INFO:tasks.rados.rados.0.smithi090.stdout:1620: writing smithi09025264-13 from 480832 to 950731 tid 1 2023-01-15T14:35:21.003 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 136 2023-01-15T14:35:21.009 INFO:tasks.rados.rados.0.smithi090.stdout:1620: writing smithi09025264-13 from 1606334 to 2270534 tid 2 2023-01-15T14:35:21.018 INFO:tasks.rados.rados.0.smithi090.stdout:1620: writing smithi09025264-13 from 3018232 to 3680832 tid 3 2023-01-15T14:35:21.019 INFO:tasks.rados.rados.0.smithi090.stdout:1621: read oid 7 snap -1 2023-01-15T14:35:21.019 INFO:tasks.rados.rados.0.smithi090.stdout:1621: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:35:21.019 INFO:tasks.rados.rados.0.smithi090.stdout:1622: read oid 31 snap -1 2023-01-15T14:35:21.019 INFO:tasks.rados.rados.0.smithi090.stdout:1622: expect (ObjNum 520 snap 158 seq_num 520) 2023-01-15T14:35:21.019 INFO:tasks.rados.rados.0.smithi090.stdout:1623: write (excl) oid 43 current snap is 178 2023-01-15T14:35:21.019 INFO:tasks.rados.rados.0.smithi090.stdout:1623: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2023-01-15T14:35:21.024 INFO:tasks.rados.rados.0.smithi090.stdout:1623: writing smithi09025264-43 from 497639 to 1042787 tid 1 2023-01-15T14:35:21.029 INFO:tasks.rados.rados.0.smithi090.stdout:1623: writing smithi09025264-43 from 1548463 to 1956321 tid 2 2023-01-15T14:35:21.035 INFO:tasks.rados.rados.0.smithi090.stdout:1623: writing smithi09025264-43 from 2529302 to 3093724 tid 3 2023-01-15T14:35:21.039 INFO:tasks.rados.rados.0.smithi090.stdout:1623: writing smithi09025264-43 from 3528143 to 3697639 tid 4 2023-01-15T14:35:21.039 INFO:tasks.rados.rados.0.smithi090.stdout:1624: snap_remove snap 151 2023-01-15T14:35:21.055 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:21.049+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:21.076 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:21.076 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:21.085 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:35:21.156 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:21.149+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:21.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:35:21.327 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:21.328 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:21.328 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:35:21.329 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:21.321+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:21.427 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:21.418+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:21.486 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:21.486 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:21.498 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:35:21.521 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:21.528 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:21.522+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:21.539 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:21.558 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:21.575 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:21.594 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:21.609 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:21.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:21.610 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 414.51432270100003, 2023-01-15T14:35:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:35:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:21.611 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:35:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:21.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:35:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:35:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:35:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:35:21.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:35:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:35:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:35:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:35:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:21.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:35:21.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:35:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:35:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:35:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:35:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:21.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:35:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:35:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:35:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:35:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:35:21.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:35:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 414.514022904, 2023-01-15T14:35:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:35:21.622 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:35:21.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:35:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:21.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:35:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:35:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:35:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:35:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:35:21.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:35:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:35:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:35:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:35:21.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:35:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:35:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:35:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:35:21.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:35:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:35:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:35:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:35:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:35:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:35:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:35:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:35:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:21.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:35:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:35:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:35:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:35:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:21.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:35:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:35:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 414.51399312000001, 2023-01-15T14:35:21.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:21.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:35:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:35:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:35:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:35:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:35:21.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:35:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:35:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:35:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:35:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:35:21.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:35:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:35:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:35:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:35:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:35:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:35:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.640 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 414.38418431999997, 2023-01-15T14:35:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:35:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:21.641 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:35:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:35:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:35:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:35:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:35:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:35:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:35:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:35:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:21.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:35:21.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:21.651 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 414.38412762299998, 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:21.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:35:21.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:35:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:35:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:35:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:35:21.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:35:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:35:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:35:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:35:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:35:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:35:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:35:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:35:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:35:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:35:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:35:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:35:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:35:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:35:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:35:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:35:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:21.658 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:35:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 327.67685904299998, 2023-01-15T14:35:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:35:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:21.659 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:35:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:21.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:35:21.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:35:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:35:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:35:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:35:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:21.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 323.72043954999998, 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:21.666 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:35:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:35:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:35:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:35:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:35:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:35:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:35:21.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:35:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:35:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:35:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:35:21.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:35:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:35:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:35:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.672 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 323.66294186699997, 2023-01-15T14:35:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:35:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:35:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.673 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:21.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:35:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:35:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:35:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:35:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:35:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:35:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:35:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:35:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:35:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:35:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:35:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:35:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:35:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:35:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:35:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:35:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:35:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:35:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:35:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:35:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:35:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:35:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:35:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:35:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:35:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:21.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:35:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:35:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:35:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:35:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:35:21.682 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:35:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 302.29429036200003, 2023-01-15T14:35:21.683 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:35:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:35:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:21.684 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:21.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:35:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:35:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:35:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:21.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:35:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:35:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:35:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:35:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:35:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:35:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:35:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:35:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.688 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:35:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:35:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:35:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:35:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:35:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:35:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:35:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:35:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:35:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:35:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:35:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:35:21.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:35:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:35:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:35:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:35:21.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:21.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:35:21.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:35:21.973 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.973 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:21.974 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.974 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:21.974 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:21.974 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 302.29418772100001, 2023-01-15T14:35:21.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:35:21.974 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:21.975 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:21.975 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:21.975 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:21.975 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:21.975 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:35:21.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.975 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:21.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:21.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:21.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:21.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:21.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:21.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:35:21.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:21.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:21.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:35:21.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:35:21.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:21.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:35:21.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:35:21.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:21.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:35:21.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:35:21.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:35:21.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:35:21.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:21.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:35:21.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:35:21.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:35:21.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:35:21.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:21.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:35:21.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:35:21.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:35:21.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:35:21.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:21.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:35:21.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:35:21.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:21.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 257.69320572599997, 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:21.983 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:21.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:21.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:21.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:21.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:21.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:21.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:21.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:35:21.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:35:21.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:35:21.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:35:21.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:21.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:35:21.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:35:21.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:35:21.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:35:21.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:21.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:35:21.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:35:21.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:21.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:35:21.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:35:21.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:35:21.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:35:21.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:21.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:35:21.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:35:21.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:21.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:21.999 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:21.999 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:35:21.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:35:21.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:35:22.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:35:22.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:22.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:35:22.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:35:22.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:22.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:35:22.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:35:22.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:22.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:35:22.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:35:22.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:22.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:35:22.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:35:22.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:22.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:35:22.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:35:22.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:35:22.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:35:22.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:35:22.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:35:22.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:22.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:35:22.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:35:22.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:35:22.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:35:22.006 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.006 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.006 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.007 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:22.007 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:22.007 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 257.69318030199997, 2023-01-15T14:35:22.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:35:22.008 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.008 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:22.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.008 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.008 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:35:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:22.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:35:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:35:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:35:22.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:35:22.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:35:22.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:35:22.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:35:22.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:35:22.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:35:22.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:22.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:35:22.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:35:22.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:35:22.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:35:22.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:22.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:35:22.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:35:22.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:35:22.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:35:22.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:35:22.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:35:22.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:35:22.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:35:22.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:35:22.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:35:22.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.016 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:22.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:35:22.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:35:22.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:22.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:35:22.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:35:22.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:22.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:35:22.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:35:22.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:35:22.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:35:22.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:35:22.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:35:22.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:22.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:35:22.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:35:22.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2301 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e216)", 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 218.403305835, 2023-01-15T14:35:22.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61754771200000003, 2023-01-15T14:35:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2301 2023-01-15T14:35:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195530+0000", 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195535+0000", 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3660000000000002e-06 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195546+0000", 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1062e-05 2023-01-15T14:35:22.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195551+0000", 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195999+0000", 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044799800000000002 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.196106+0000", 2023-01-15T14:35:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106172 2023-01-15T14:35:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510631+0000", 2023-01-15T14:35:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31452575999999999 2023-01-15T14:35:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510875+0000", 2023-01-15T14:35:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024415199999999998 2023-01-15T14:35:22.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812445+0000", 2023-01-15T14:35:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30156923000000002 2023-01-15T14:35:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812727+0000", 2023-01-15T14:35:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028227800000000001 2023-01-15T14:35:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.813079+0000", 2023-01-15T14:35:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199599999999998 2023-01-15T14:35:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.144 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2749 3.3d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:35:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 170.67742105900001, 2023-01-15T14:35:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.69206326600000001, 2023-01-15T14:35:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.145 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2749 2023-01-15T14:35:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921418+0000", 2023-01-15T14:35:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1670000000000002e-06 2023-01-15T14:35:22.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921426+0000", 2023-01-15T14:35:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0569999999999993e-06 2023-01-15T14:35:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921429+0000", 2023-01-15T14:35:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9629999999999998e-06 2023-01-15T14:35:22.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107302+0000", 2023-01-15T14:35:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.185873327 2023-01-15T14:35:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107657+0000", 2023-01-15T14:35:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035504299999999998 2023-01-15T14:35:22.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212694+0000", 2023-01-15T14:35:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105037095 2023-01-15T14:35:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212936+0000", 2023-01-15T14:35:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241492 2023-01-15T14:35:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613125+0000", 2023-01-15T14:35:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40018927900000001 2023-01-15T14:35:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613294+0000", 2023-01-15T14:35:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016838000000000001 2023-01-15T14:35:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613479+0000", 2023-01-15T14:35:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018577599999999999 2023-01-15T14:35:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3509 3.53 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (58) in=7b,truncate 2084167] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:22.153 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.480777923999995, 2023-01-15T14:35:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70658069899999998, 2023-01-15T14:35:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3509 2023-01-15T14:35:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118062+0000", 2023-01-15T14:35:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2260000000000001e-06 2023-01-15T14:35:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118067+0000", 2023-01-15T14:35:22.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3700000000000003e-06 2023-01-15T14:35:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118072+0000", 2023-01-15T14:35:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4880000000000001e-06 2023-01-15T14:35:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218253+0000", 2023-01-15T14:35:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10018108100000001 2023-01-15T14:35:22.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218258+0000", 2023-01-15T14:35:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.831e-06 2023-01-15T14:35:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321058+0000", 2023-01-15T14:35:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10280054499999999 2023-01-15T14:35:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321223+0000", 2023-01-15T14:35:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016474 2023-01-15T14:35:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323014+0000", 2023-01-15T14:35:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017909550000000001 2023-01-15T14:35:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323124+0000", 2023-01-15T14:35:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110326 2023-01-15T14:35:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323363+0000", 2023-01-15T14:35:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238385 2023-01-15T14:35:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323663+0000", 2023-01-15T14:35:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029987699999999999 2023-01-15T14:35:22.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323677+0000", 2023-01-15T14:35:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4346e-05 2023-01-15T14:35:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323729+0000", 2023-01-15T14:35:22.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2049999999999998e-05 2023-01-15T14:35:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323775+0000", 2023-01-15T14:35:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5872999999999999e-05 2023-01-15T14:35:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.616791+0000", 2023-01-15T14:35:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29301572199999998 2023-01-15T14:35:22.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824267+0000", 2023-01-15T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20747585199999999 2023-01-15T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:22.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824425+0000", 2023-01-15T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015861900000000001 2023-01-15T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824640+0000", 2023-01-15T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021475899999999999 2023-01-15T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.165 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3510 3.53 3:cbfd2d71:::smithi09025264-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 92.480700084999995, 2023-01-15T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.80704639, 2023-01-15T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.166 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3510 2023-01-15T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118135+0000", 2023-01-15T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118143+0000", 2023-01-15T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0579999999999998e-06 2023-01-15T14:35:22.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118162+0000", 2023-01-15T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8702000000000002e-05 2023-01-15T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118178+0000", 2023-01-15T14:35:22.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5645e-05 2023-01-15T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321268+0000", 2023-01-15T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20308996000000001 2023-01-15T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321370+0000", 2023-01-15T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010268399999999999 2023-01-15T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323825+0000", 2023-01-15T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024546450000000001 2023-01-15T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323934+0000", 2023-01-15T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010940199999999999 2023-01-15T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.924815+0000", 2023-01-15T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60088053500000005 2023-01-15T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925009+0000", 2023-01-15T14:35:22.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019363599999999999 2023-01-15T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925183+0000", 2023-01-15T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000174823 2023-01-15T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3505 3.62 3:47dfe935:::smithi09025264-29:head [rollback db] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.583097374999994, 2023-01-15T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95152285400000003, 2023-01-15T14:35:22.177 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3505 2023-01-15T14:35:22.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015739+0000", 2023-01-15T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015742+0000", 2023-01-15T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.869e-06 2023-01-15T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015748+0000", 2023-01-15T14:35:22.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0360000000000003e-06 2023-01-15T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015754+0000", 2023-01-15T14:35:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.733e-06 2023-01-15T14:35:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318179+0000", 2023-01-15T14:35:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30242492300000001 2023-01-15T14:35:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318502+0000", 2023-01-15T14:35:22.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032355699999999998 2023-01-15T14:35:22.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:22.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365019+0000", 2023-01-15T14:35:22.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046516950000000001 2023-01-15T14:35:22.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862003+0000", 2023-01-15T14:35:22.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49698320800000001 2023-01-15T14:35:22.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862314+0000", 2023-01-15T14:35:22.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000311236 2023-01-15T14:35:22.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:22.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863083+0000", 2023-01-15T14:35:22.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076928399999999996 2023-01-15T14:35:22.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:22.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863709+0000", 2023-01-15T14:35:22.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062558700000000004 2023-01-15T14:35:22.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:22.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863724+0000", 2023-01-15T14:35:22.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5288999999999999e-05 2023-01-15T14:35:22.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:22.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863799+0000", 2023-01-15T14:35:22.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4552e-05 2023-01-15T14:35:22.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:22.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863851+0000", 2023-01-15T14:35:22.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2018999999999998e-05 2023-01-15T14:35:22.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966087+0000", 2023-01-15T14:35:22.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102235952 2023-01-15T14:35:22.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966811+0000", 2023-01-15T14:35:22.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072484800000000005 2023-01-15T14:35:22.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:22.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966950+0000", 2023-01-15T14:35:22.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013907399999999999 2023-01-15T14:35:22.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.967263+0000", 2023-01-15T14:35:22.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312107 2023-01-15T14:35:22.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.328 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.328 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.329 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3506 3.62 3:47dfe935:::smithi09025264-29:head [append 0~0] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:22.329 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:22.329 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 91.583077552999995, 2023-01-15T14:35:22.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.234729387, 2023-01-15T14:35:22.329 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.330 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:22.330 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.330 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.330 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.330 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3506 2023-01-15T14:35:22.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.330 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:22.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:22.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015759+0000", 2023-01-15T14:35:22.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:22.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015763+0000", 2023-01-15T14:35:22.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.517e-06 2023-01-15T14:35:22.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015768+0000", 2023-01-15T14:35:22.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5620000000000003e-06 2023-01-15T14:35:22.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015773+0000", 2023-01-15T14:35:22.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3730000000000003e-06 2023-01-15T14:35:22.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365160+0000", 2023-01-15T14:35:22.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34938724300000001 2023-01-15T14:35:22.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:22.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365370+0000", 2023-01-15T14:35:22.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021001599999999999 2023-01-15T14:35:22.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863926+0000", 2023-01-15T14:35:22.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49855574600000002 2023-01-15T14:35:22.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.864284+0000", 2023-01-15T14:35:22.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035790600000000001 2023-01-15T14:35:22.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:22.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865099+0000", 2023-01-15T14:35:22.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081499999999999997 2023-01-15T14:35:22.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:22.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865640+0000", 2023-01-15T14:35:22.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054118899999999997 2023-01-15T14:35:22.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:22.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865653+0000", 2023-01-15T14:35:22.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3614e-05 2023-01-15T14:35:22.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:22.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865699+0000", 2023-01-15T14:35:22.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5751000000000003e-05 2023-01-15T14:35:22.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966206+0000", 2023-01-15T14:35:22.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100506741 2023-01-15T14:35:22.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:22.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966665+0000", 2023-01-15T14:35:22.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000459345 2023-01-15T14:35:22.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.249842+0000", 2023-01-15T14:35:22.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28317711200000001 2023-01-15T14:35:22.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:22.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250054+0000", 2023-01-15T14:35:22.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021161899999999999 2023-01-15T14:35:22.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250489+0000", 2023-01-15T14:35:22.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043499700000000001 2023-01-15T14:35:22.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.345 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.345 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3572 3.65 3:a75958b1:::smithi09025264-6:head [setxattr _header (57) in=7b,truncate 2235430] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:22.345 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:22.345 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.576447728000005, 2023-01-15T14:35:22.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65482807700000001, 2023-01-15T14:35:22.346 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.346 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:22.346 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.346 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.346 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.346 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3572 2023-01-15T14:35:22.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.346 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022388+0000", 2023-01-15T14:35:22.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022397+0000", 2023-01-15T14:35:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.704e-06 2023-01-15T14:35:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022412+0000", 2023-01-15T14:35:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5554e-05 2023-01-15T14:35:22.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022422+0000", 2023-01-15T14:35:22.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2839999999999995e-06 2023-01-15T14:35:22.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114230+0000", 2023-01-15T14:35:22.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.091808129000000002 2023-01-15T14:35:22.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:22.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114234+0000", 2023-01-15T14:35:22.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4329999999999997e-06 2023-01-15T14:35:22.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204200+0000", 2023-01-15T14:35:22.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089966321000000002 2023-01-15T14:35:22.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204439+0000", 2023-01-15T14:35:22.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238595 2023-01-15T14:35:22.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.305883+0000", 2023-01-15T14:35:22.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101443713 2023-01-15T14:35:22.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306066+0000", 2023-01-15T14:35:22.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182962 2023-01-15T14:35:22.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2023-01-15T14:35:22.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306621+0000", 2023-01-15T14:35:22.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055552800000000001 2023-01-15T14:35:22.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:22.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307013+0000", 2023-01-15T14:35:22.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039210599999999998 2023-01-15T14:35:22.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:22.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307033+0000", 2023-01-15T14:35:22.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9423e-05 2023-01-15T14:35:22.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:22.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307093+0000", 2023-01-15T14:35:22.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0402000000000002e-05 2023-01-15T14:35:22.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:22.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307445+0000", 2023-01-15T14:35:22.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000351677 2023-01-15T14:35:22.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307662+0000", 2023-01-15T14:35:22.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021765099999999999 2023-01-15T14:35:22.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676523+0000", 2023-01-15T14:35:22.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36886056699999997 2023-01-15T14:35:22.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:22.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676723+0000", 2023-01-15T14:35:22.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019947899999999999 2023-01-15T14:35:22.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677217+0000", 2023-01-15T14:35:22.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049493900000000001 2023-01-15T14:35:22.357 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.357 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.357 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.357 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3573 3.65 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:22.357 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:22.357 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 84.576404600999993, 2023-01-15T14:35:22.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65551266799999997, 2023-01-15T14:35:22.357 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.357 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:22.358 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.358 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.358 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.358 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3573 2023-01-15T14:35:22.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.358 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:22.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:22.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022432+0000", 2023-01-15T14:35:22.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:22.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022436+0000", 2023-01-15T14:35:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.693e-06 2023-01-15T14:35:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022447+0000", 2023-01-15T14:35:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1639e-05 2023-01-15T14:35:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122543+0000", 2023-01-15T14:35:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100095369 2023-01-15T14:35:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122863+0000", 2023-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032020799999999999 2023-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122869+0000", 2023-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7640000000000002e-06 2023-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206080+0000", 2023-01-15T14:35:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.083211794000000006 2023-01-15T14:35:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206289+0000", 2023-01-15T14:35:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020866600000000001 2023-01-15T14:35:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307193+0000", 2023-01-15T14:35:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10090391999999999 2023-01-15T14:35:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307374+0000", 2023-01-15T14:35:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018061299999999999 2023-01-15T14:35:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677342+0000", 2023-01-15T14:35:22.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.369968034 2023-01-15T14:35:22.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677535+0000", 2023-01-15T14:35:22.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019311299999999999 2023-01-15T14:35:22.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677945+0000", 2023-01-15T14:35:22.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041040299999999999 2023-01-15T14:35:22.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.526 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.526 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.528 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.528 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.528 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:22.529 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:35:22.530 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:22.530 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:22.525+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:22.531 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:35:22.531 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:22.532 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:22.532 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:35:22.532 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:35:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:35:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:35:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:35:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:35:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:35:22.533 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:35:22.534 INFO:tasks.rados.rados.0.smithi090.stdout:1624: done (5 left) 2023-01-15T14:35:22.534 INFO:tasks.rados.rados.0.smithi090.stdout:1625: copy_from oid 49 from oid 26 current snap is 178 2023-01-15T14:35:22.534 INFO:tasks.rados.rados.0.smithi090.stdout:1620: finishing write tid 1 to smithi09025264-13 2023-01-15T14:35:22.534 INFO:tasks.rados.rados.0.smithi090.stdout:1619: expect (ObjNum 466 snap 134 seq_num 466) 2023-01-15T14:35:22.534 INFO:tasks.rados.rados.0.smithi090.stdout:1622: expect (ObjNum 520 snap 158 seq_num 520) 2023-01-15T14:35:22.534 INFO:tasks.rados.rados.0.smithi090.stdout:1623: finishing write tid 1 to smithi09025264-43 2023-01-15T14:35:22.534 INFO:tasks.rados.rados.0.smithi090.stdout:1623: finishing write tid 2 to smithi09025264-43 2023-01-15T14:35:22.534 INFO:tasks.rados.rados.0.smithi090.stdout:1620: finishing write tid 2 to smithi09025264-13 2023-01-15T14:35:22.534 INFO:tasks.rados.rados.0.smithi090.stdout:1623: finishing write tid 3 to smithi09025264-43 2023-01-15T14:35:22.535 INFO:tasks.rados.rados.0.smithi090.stdout:1620: finishing write tid 3 to smithi09025264-13 2023-01-15T14:35:22.535 INFO:tasks.rados.rados.0.smithi090.stdout:1623: finishing write tid 4 to smithi09025264-43 2023-01-15T14:35:22.535 INFO:tasks.rados.rados.0.smithi090.stdout:1623: finishing write tid 5 to smithi09025264-43 2023-01-15T14:35:22.535 INFO:tasks.rados.rados.0.smithi090.stdout:1619: done (5 left) 2023-01-15T14:35:22.535 INFO:tasks.rados.rados.0.smithi090.stdout:1622: done (4 left) 2023-01-15T14:35:22.535 INFO:tasks.rados.rados.0.smithi090.stdout:1626: snap_remove snap 166 2023-01-15T14:35:22.630 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:22.622+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4217 3.78 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.72008138499999996, 2023-01-15T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72010099000000005, 2023-01-15T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.715 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "delayed", 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4217 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.716 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963851+0000", 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.717 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963857+0000", 2023-01-15T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8699999999999997e-06 2023-01-15T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963867+0000", 2023-01-15T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2023-01-15T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.718 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963876+0000", 2023-01-15T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1120000000000002e-06 2023-01-15T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.719 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064133+0000", 2023-01-15T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100256427 2023-01-15T14:35:22.720 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064164+0000", 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1649000000000001e-05 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4216 3.78 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:22.721 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.72017693999999999, 2023-01-15T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72023755599999995, 2023-01-15T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "delayed", 2023-01-15T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4216 2023-01-15T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.722 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963753+0000", 2023-01-15T14:35:22.723 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999962 2023-01-15T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963772+0000", 2023-01-15T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9293999999999999e-05 2023-01-15T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963806+0000", 2023-01-15T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3946999999999999e-05 2023-01-15T14:35:22.724 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963822+0000", 2023-01-15T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5744e-05 2023-01-15T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963930+0000", 2023-01-15T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010870300000000001 2023-01-15T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.725 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963946+0000", 2023-01-15T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5919000000000001e-05 2023-01-15T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:35:22.726 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 2 2023-01-15T14:35:22.727 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:22.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:35:22.772 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:22.766+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:22.844 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:22.844 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:22.844 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:22.845 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:22.845 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:22.861 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:35:22.924 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:22.826+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:22.938 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:22.933+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:22.968 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:22.968 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:22.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:22.969 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:22.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.969 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:22.969 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:22.969 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 288.06726950500001, 2023-01-15T14:35:22.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:35:22.969 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.969 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:22.969 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.970 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.970 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.970 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:35:22.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.970 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:22.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:22.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:35:22.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:22.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:35:22.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:35:22.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:35:22.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:35:22.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:35:22.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:35:22.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:35:22.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:35:22.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:22.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:35:22.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:35:22.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:35:22.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:35:22.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:35:22.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:35:22.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:35:22.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:35:22.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:35:22.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:22.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:35:22.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:35:22.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:22.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:35:22.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:35:22.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:22.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:35:22.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:35:22.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:22.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:35:22.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:35:22.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:35:22.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:35:22.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:35:22.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:35:22.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:22.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:35:22.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:35:22.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:35:22.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:35:22.980 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.980 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.980 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.981 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:22.981 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:22.981 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 288.06721856199999, 2023-01-15T14:35:22.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:35:22.981 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.981 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:22.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.981 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.981 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:35:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:22.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:35:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:35:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:35:22.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:35:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:35:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:35:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:35:22.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:35:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:35:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:35:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:35:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:35:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:35:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:35:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:35:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:35:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:35:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:35:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:35:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:35:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:35:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1676 3.4 3:2029fa71:::smithi09025264-39:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e151)", 2023-01-15T14:35:22.988 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:35:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 285.12631932099998, 2023-01-15T14:35:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35228993400000003, 2023-01-15T14:35:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1676 2023-01-15T14:35:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.989 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:35:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835410+0000", 2023-01-15T14:35:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835409+0000", 2023-01-15T14:35:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835417+0000", 2023-01-15T14:35:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5690000000000005e-06 2023-01-15T14:35:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835436+0000", 2023-01-15T14:35:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8817999999999998e-05 2023-01-15T14:35:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.835446+0000", 2023-01-15T14:35:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6339999999999996e-06 2023-01-15T14:35:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862710+0000", 2023-01-15T14:35:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.027264206999999999 2023-01-15T14:35:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:35:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.862730+0000", 2023-01-15T14:35:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9715e-05 2023-01-15T14:35:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969332+0000", 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10660212299999999 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969502+0000", 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0001705 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.969844+0000", 2023-01-15T14:35:22.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034215100000000001 2023-01-15T14:35:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970422+0000", 2023-01-15T14:35:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057731799999999995 2023-01-15T14:35:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:22.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970433+0000", 2023-01-15T14:35:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1456e-05 2023-01-15T14:35:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.970494+0000", 2023-01-15T14:35:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0823999999999999e-05 2023-01-15T14:35:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:37.987226+0000", 2023-01-15T14:35:22.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016732461000000001 2023-01-15T14:35:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.032730+0000", 2023-01-15T14:35:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.045503707999999997 2023-01-15T14:35:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187483+0000", 2023-01-15T14:35:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15475328699999999 2023-01-15T14:35:22.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187538+0000", 2023-01-15T14:35:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4324999999999999e-05 2023-01-15T14:35:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:38.187700+0000", 2023-01-15T14:35:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016237000000000001 2023-01-15T14:35:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1711 3.4 3:2022fdfa:::smithi09025264-46:head [write 702592~703393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:35:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:35:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 283.05254514799998, 2023-01-15T14:35:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30893886999999998, 2023-01-15T14:35:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:22.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:23.000 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:23.000 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1711 2023-01-15T14:35:23.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.000 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:35:23.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909184+0000", 2023-01-15T14:35:23.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909182+0000", 2023-01-15T14:35:23.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:23.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909482+0000", 2023-01-15T14:35:23.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029977699999999999 2023-01-15T14:35:23.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909504+0000", 2023-01-15T14:35:23.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.2110999999999999e-05 2023-01-15T14:35:23.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.909514+0000", 2023-01-15T14:35:23.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6870000000000006e-06 2023-01-15T14:35:23.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014794+0000", 2023-01-15T14:35:23.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105280242 2023-01-15T14:35:23.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:23.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.014810+0000", 2023-01-15T14:35:23.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5707000000000001e-05 2023-01-15T14:35:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016574+0000", 2023-01-15T14:35:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017645720000000001 2023-01-15T14:35:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.004 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.016856+0000", 2023-01-15T14:35:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028175100000000001 2023-01-15T14:35:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:35:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017147+0000", 2023-01-15T14:35:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029076699999999999 2023-01-15T14:35:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.005 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017672+0000", 2023-01-15T14:35:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052472399999999996 2023-01-15T14:35:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017686+0000", 2023-01-15T14:35:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4202000000000001e-05 2023-01-15T14:35:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.006 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.017953+0000", 2023-01-15T14:35:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026752500000000001 2023-01-15T14:35:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.018980+0000", 2023-01-15T14:35:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0010267799999999999 2023-01-15T14:35:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.150377+0000", 2023-01-15T14:35:23.007 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13139646199999999 2023-01-15T14:35:23.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218006+0000", 2023-01-15T14:35:23.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067629927000000006 2023-01-15T14:35:23.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.008 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.008 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218067+0000", 2023-01-15T14:35:23.008 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0090999999999998e-05 2023-01-15T14:35:23.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.009 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.009 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.218123+0000", 2023-01-15T14:35:23.009 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6731000000000003e-05 2023-01-15T14:35:23.009 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.009 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.009 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.009 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 259.15858972799998, 2023-01-15T14:35:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:35:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:35:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.010 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:23.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:23.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:35:23.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:23.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.012 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.012 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.012 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:35:23.012 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:35:23.012 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:35:23.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:35:23.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.013 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.013 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.013 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:35:23.013 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:35:23.013 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:35:23.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:35:23.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.014 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:23.014 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:35:23.014 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:35:23.014 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.014 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.015 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.016 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:35:23.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:35:23.016 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.016 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.016 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.016 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2047 3.1f 3:f8edd34e:::smithi09025264-19:head [write 1765266~792426] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:23.016 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:35:23.016 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 237.27518304700001, 2023-01-15T14:35:23.016 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30952270199999998, 2023-01-15T14:35:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2047 2023-01-15T14:35:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:35:23.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686547+0000", 2023-01-15T14:35:23.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.018 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.018 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686544+0000", 2023-01-15T14:35:23.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:23.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686885+0000", 2023-01-15T14:35:23.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034084600000000001 2023-01-15T14:35:23.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.019 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.019 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686916+0000", 2023-01-15T14:35:23.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0905000000000003e-05 2023-01-15T14:35:23.019 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.686929+0000", 2023-01-15T14:35:23.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2345e-05 2023-01-15T14:35:23.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.020 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.020 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687014+0000", 2023-01-15T14:35:23.020 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5876000000000002e-05 2023-01-15T14:35:23.020 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.020 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.687174+0000", 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000159572 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784677+0000", 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097502801 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.021 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.784856+0000", 2023-01-15T14:35:23.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017946499999999999 2023-01-15T14:35:23.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.022 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:23.022 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785165+0000", 2023-01-15T14:35:23.022 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030839700000000002 2023-01-15T14:35:23.022 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.022 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785637+0000", 2023-01-15T14:35:23.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047224300000000001 2023-01-15T14:35:23.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.023 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.785653+0000", 2023-01-15T14:35:23.023 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:35:23.023 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.023 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.023 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787401+0000", 2023-01-15T14:35:23.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017480040000000001 2023-01-15T14:35:23.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:23.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787470+0000", 2023-01-15T14:35:23.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8961999999999995e-05 2023-01-15T14:35:23.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.024 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.024 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892003+0000", 2023-01-15T14:35:23.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104532823 2023-01-15T14:35:23.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995742+0000", 2023-01-15T14:35:23.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103739583 2023-01-15T14:35:23.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.995923+0000", 2023-01-15T14:35:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018039400000000001 2023-01-15T14:35:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996069+0000", 2023-01-15T14:35:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014664600000000001 2023-01-15T14:35:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.027 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2048 3.1f 3:f8edd34e:::smithi09025264-19:head [write 2964741~195008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:23.027 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:35:23.027 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 237.271775924, 2023-01-15T14:35:23.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306523883, 2023-01-15T14:35:23.027 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.027 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:23.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:23.027 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:23.027 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2048 2023-01-15T14:35:23.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.028 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:35:23.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689954+0000", 2023-01-15T14:35:23.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.689952+0000", 2023-01-15T14:35:23.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:23.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690099+0000", 2023-01-15T14:35:23.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014766099999999999 2023-01-15T14:35:23.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690126+0000", 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6738999999999999e-05 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690138+0000", 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2309e-05 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690350+0000", 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021151899999999999 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690577+0000", 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022691599999999999 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786239+0000", 2023-01-15T14:35:23.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.095661754000000002 2023-01-15T14:35:23.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786445+0000", 2023-01-15T14:35:23.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020641999999999999 2023-01-15T14:35:23.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786686+0000", 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241318 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.786941+0000", 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025421200000000003 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.787405+0000", 2023-01-15T14:35:23.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046497800000000001 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789807+0000", 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024014700000000002 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.789894+0000", 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6960999999999996e-05 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.892181+0000", 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102286716 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996254+0000", 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.104073446 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996385+0000", 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013121399999999999 2023-01-15T14:35:23.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.996478+0000", 2023-01-15T14:35:23.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2225000000000006e-05 2023-01-15T14:35:23.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.036 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.036 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.036 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:23.036 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:23.037 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 237.271560319, 2023-01-15T14:35:23.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:35:23.037 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.037 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:23.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:23.037 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:23.037 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:35:23.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.037 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:23.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:23.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:35:23.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:23.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:35:23.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:35:23.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:35:23.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:35:23.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:35:23.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:35:23.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:35:23.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:35:23.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:23.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:35:23.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:35:23.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:35:23.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:35:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:35:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:35:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:35:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:35:23.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:35:23.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:35:23.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:35:23.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:35:23.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:35:23.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:35:23.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:23.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:35:23.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:35:23.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.044 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:35:23.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:35:23.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:35:23.045 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:35:23.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.045 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.045 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:35:23.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:35:23.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.046 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.046 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:35:23.046 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:35:23.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.046 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.047 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.047 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:23.047 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:23.047 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 237.271453245, 2023-01-15T14:35:23.047 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:35:23.047 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.047 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:23.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:23.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:23.047 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:23.047 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:35:23.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.048 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:23.048 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.048 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.048 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:35:23.049 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:35:23.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:35:23.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.050 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.050 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:35:23.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:35:23.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:35:23.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:35:23.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:23.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:35:23.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:35:23.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:35:23.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:35:23.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:23.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:35:23.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:35:23.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:35:23.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:35:23.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:35:23.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:35:23.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:35:23.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2152 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 229.141214195, 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31387535500000002, 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.054 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:23.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:23.055 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:23.055 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2152 2023-01-15T14:35:23.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.055 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.055 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.055 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.055 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:35:23.055 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820515+0000", 2023-01-15T14:35:23.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.056 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.056 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820513+0000", 2023-01-15T14:35:23.056 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:23.056 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820529+0000", 2023-01-15T14:35:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6362999999999999e-05 2023-01-15T14:35:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820563+0000", 2023-01-15T14:35:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3766000000000002e-05 2023-01-15T14:35:23.057 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820581+0000", 2023-01-15T14:35:23.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8224e-05 2023-01-15T14:35:23.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.058 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.058 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.058 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820725+0000", 2023-01-15T14:35:23.058 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014394700000000001 2023-01-15T14:35:23.058 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820736+0000", 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031125+0000", 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21038817000000001 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.059 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031389+0000", 2023-01-15T14:35:23.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264622 2023-01-15T14:35:23.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:23.060 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.031734+0000", 2023-01-15T14:35:23.060 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034470300000000002 2023-01-15T14:35:23.060 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.060 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.060 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032623+0000", 2023-01-15T14:35:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088898799999999995 2023-01-15T14:35:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032641+0000", 2023-01-15T14:35:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8216e-05 2023-01-15T14:35:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.061 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032702+0000", 2023-01-15T14:35:23.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1196000000000002e-05 2023-01-15T14:35:23.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:23.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033788+0000", 2023-01-15T14:35:23.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001086098 2023-01-15T14:35:23.062 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.062 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.062 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.062 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134061+0000", 2023-01-15T14:35:23.062 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100272915 2023-01-15T14:35:23.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134183+0000", 2023-01-15T14:35:23.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121848 2023-01-15T14:35:23.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134305+0000", 2023-01-15T14:35:23.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121783 2023-01-15T14:35:23.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134391+0000", 2023-01-15T14:35:23.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.5791999999999994e-05 2023-01-15T14:35:23.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.064 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.064 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2153 3.1f 3:f8edd34e:::smithi09025264-19:head [rollback 8c] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:35:23.064 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 229.141109658, 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31416199700000003, 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2153 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.065 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:35:23.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820620+0000", 2023-01-15T14:35:23.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820619+0000", 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820624+0000", 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3930000000000001e-06 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820634+0000", 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:35:23.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820640+0000", 2023-01-15T14:35:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5940000000000001e-06 2023-01-15T14:35:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820834+0000", 2023-01-15T14:35:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000194537 2023-01-15T14:35:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820842+0000", 2023-01-15T14:35:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1339999999999998e-06 2023-01-15T14:35:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032760+0000", 2023-01-15T14:35:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21191849099999999 2023-01-15T14:35:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.032941+0000", 2023-01-15T14:35:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018087999999999999 2023-01-15T14:35:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033331+0000", 2023-01-15T14:35:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038987800000000002 2023-01-15T14:35:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033680+0000", 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034908999999999998 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033743+0000", 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2909000000000004e-05 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033801+0000", 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8681999999999998e-05 2023-01-15T14:35:23.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.033858+0000", 2023-01-15T14:35:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6635000000000002e-05 2023-01-15T14:35:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134506+0000", 2023-01-15T14:35:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100647548 2023-01-15T14:35:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134577+0000", 2023-01-15T14:35:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1477999999999996e-05 2023-01-15T14:35:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134649+0000", 2023-01-15T14:35:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1722000000000002e-05 2023-01-15T14:35:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134782+0000", 2023-01-15T14:35:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133051 2023-01-15T14:35:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:35:23.074 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 229.14103972699999, 2023-01-15T14:35:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:35:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:35:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:35:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:35:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:35:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:35:23.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:35:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:35:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:35:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:35:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:35:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:35:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:35:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:35:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:35:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:35:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:35:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:35:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:35:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:35:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:35:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:35:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:35:23.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:35:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:35:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2780 3.38 e265/232 3:1eaad226:::smithi09025264-49:head v 265'279, mlcod=261'274)", 2023-01-15T14:35:23.084 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:35:23.115 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 170.969868601, 2023-01-15T14:35:23.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32005591799999999, 2023-01-15T14:35:23.115 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.115 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.115 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:35:23.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991861+0000", 2023-01-15T14:35:23.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.991860+0000", 2023-01-15T14:35:23.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:23.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993927+0000", 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020673229999999998 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.993946+0000", 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9429999999999999e-05 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.094039+0000", 2023-01-15T14:35:23.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100092898 2023-01-15T14:35:23.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285071+0000", 2023-01-15T14:35:23.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.191031904 2023-01-15T14:35:23.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.285106+0000", 2023-01-15T14:35:23.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5227e-05 2023-01-15T14:35:23.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286323+0000", 2023-01-15T14:35:23.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012163829999999999 2023-01-15T14:35:23.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.286335+0000", 2023-01-15T14:35:23.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2224e-05 2023-01-15T14:35:23.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287218+0000", 2023-01-15T14:35:23.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00088265000000000001 2023-01-15T14:35:23.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.287250+0000", 2023-01-15T14:35:23.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2209000000000003e-05 2023-01-15T14:35:23.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:32.311917+0000", 2023-01-15T14:35:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.024667062999999999 2023-01-15T14:35:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2868 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'278, mlcod=264'275)", 2023-01-15T14:35:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:35:23.122 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 161.96797766, 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31268252400000002, 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993752+0000", 2023-01-15T14:35:23.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993751+0000", 2023-01-15T14:35:23.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:23.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993762+0000", 2023-01-15T14:35:23.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0822000000000001e-05 2023-01-15T14:35:23.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993777+0000", 2023-01-15T14:35:23.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5540999999999999e-05 2023-01-15T14:35:23.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993787+0000", 2023-01-15T14:35:23.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0173999999999999e-05 2023-01-15T14:35:23.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290198+0000", 2023-01-15T14:35:23.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29641022099999997 2023-01-15T14:35:23.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290212+0000", 2023-01-15T14:35:23.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4214e-05 2023-01-15T14:35:23.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290300+0000", 2023-01-15T14:35:23.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8300000000000005e-05 2023-01-15T14:35:23.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290312+0000", 2023-01-15T14:35:23.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2344000000000001e-05 2023-01-15T14:35:23.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290357+0000", 2023-01-15T14:35:23.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4367999999999999e-05 2023-01-15T14:35:23.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.290389+0000", 2023-01-15T14:35:23.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1983000000000002e-05 2023-01-15T14:35:23.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.306434+0000", 2023-01-15T14:35:23.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.016045667 2023-01-15T14:35:23.129 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.129 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.129 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.130 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:35:23.130 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:23.130 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 161.96792823499999, 2023-01-15T14:35:23.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:35:23.130 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.130 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.130 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.130 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:23.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:23.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:23.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:23.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:35:23.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:35:23.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:35:23.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:35:23.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:35:23.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.136 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:35:23.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:35:23.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.136 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.136 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:35:23.137 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:23.137 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 161.96411654799999, 2023-01-15T14:35:23.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:35:23.137 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.137 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.137 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:23.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:23.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:35:23.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:35:23.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:35:23.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:35:23.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:35:23.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:35:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:35:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:35:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:35:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:35:23.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:35:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:35:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:35:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.143 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3120 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'290, mlcod=0'0)", 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 133.722303272, 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39853714000000001, 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239424+0000", 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239747+0000", 2023-01-15T14:35:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032243300000000001 2023-01-15T14:35:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239784+0000", 2023-01-15T14:35:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7051000000000002e-05 2023-01-15T14:35:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239802+0000", 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8250999999999999e-05 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536033+0000", 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29623121499999999 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536081+0000", 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2023-01-15T14:35:23.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536800+0000", 2023-01-15T14:35:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071948499999999996 2023-01-15T14:35:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536817+0000", 2023-01-15T14:35:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6228000000000001e-05 2023-01-15T14:35:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537048+0000", 2023-01-15T14:35:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023136200000000001 2023-01-15T14:35:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637902+0000", 2023-01-15T14:35:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10085403900000001 2023-01-15T14:35:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637963+0000", 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1310999999999996e-05 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3121 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'291, mlcod=0'0)", 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 133.72030775600001, 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659573799999998, 2023-01-15T14:35:23.150 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.151 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.151 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:23.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:23.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241420+0000", 2023-01-15T14:35:23.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:23.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241854+0000", 2023-01-15T14:35:23.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043330099999999998 2023-01-15T14:35:23.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241878+0000", 2023-01-15T14:35:23.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4559e-05 2023-01-15T14:35:23.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241892+0000", 2023-01-15T14:35:23.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039999999999999e-05 2023-01-15T14:35:23.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536863+0000", 2023-01-15T14:35:23.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29497073400000001 2023-01-15T14:35:23.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536887+0000", 2023-01-15T14:35:23.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3683999999999999e-05 2023-01-15T14:35:23.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537124+0000", 2023-01-15T14:35:23.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023729900000000001 2023-01-15T14:35:23.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537136+0000", 2023-01-15T14:35:23.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2278e-05 2023-01-15T14:35:23.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537345+0000", 2023-01-15T14:35:23.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020865200000000001 2023-01-15T14:35:23.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637981+0000", 2023-01-15T14:35:23.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063575700000001 2023-01-15T14:35:23.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.156 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.156 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638018+0000", 2023-01-15T14:35:23.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6826999999999998e-05 2023-01-15T14:35:23.157 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.157 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.157 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.157 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3122 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'292, mlcod=0'0)", 2023-01-15T14:35:23.157 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:23.157 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 133.71841191600001, 2023-01-15T14:35:23.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39623551600000001, 2023-01-15T14:35:23.157 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.158 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.158 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:23.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:23.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243316+0000", 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243833+0000", 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051703600000000001 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243863+0000", 2023-01-15T14:35:23.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-01-15T14:35:23.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243879+0000", 2023-01-15T14:35:23.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6263999999999999e-05 2023-01-15T14:35:23.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637317+0000", 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39343783199999999 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637356+0000", 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9579000000000002e-05 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637830+0000", 2023-01-15T14:35:23.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047326399999999999 2023-01-15T14:35:23.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637885+0000", 2023-01-15T14:35:23.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5074999999999997e-05 2023-01-15T14:35:23.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638185+0000", 2023-01-15T14:35:23.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029993799999999997 2023-01-15T14:35:23.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638337+0000", 2023-01-15T14:35:23.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015208299999999999 2023-01-15T14:35:23.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.163 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.639553+0000", 2023-01-15T14:35:23.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001216389 2023-01-15T14:35:23.163 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.164 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.164 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.164 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3123 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'293, mlcod=0'0)", 2023-01-15T14:35:23.164 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:23.164 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 133.71666788799999, 2023-01-15T14:35:23.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.395733102, 2023-01-15T14:35:23.164 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.164 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.164 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:23.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:23.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245060+0000", 2023-01-15T14:35:23.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:23.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245075+0000", 2023-01-15T14:35:23.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4459e-05 2023-01-15T14:35:23.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245107+0000", 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1795000000000003e-05 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245121+0000", 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4117000000000001e-05 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638063+0000", 2023-01-15T14:35:23.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39294204300000002 2023-01-15T14:35:23.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638097+0000", 2023-01-15T14:35:23.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4385999999999999e-05 2023-01-15T14:35:23.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.168 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.168 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638299+0000", 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020133099999999999 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638318+0000", 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514999999999999e-05 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638386+0000", 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8089000000000005e-05 2023-01-15T14:35:23.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638430+0000", 2023-01-15T14:35:23.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.439e-05 2023-01-15T14:35:23.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:23.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.640795+0000", 2023-01-15T14:35:23.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002364274 2023-01-15T14:35:23.170 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.171 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.171 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.171 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:23.171 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:23.171 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:23.173 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:35:23.173 INFO:tasks.rados.rados.0.smithi090.stdout:1626: done (4 left) 2023-01-15T14:35:23.173 INFO:tasks.rados.rados.0.smithi090.stdout:1627: write oid 20 current snap is 178 2023-01-15T14:35:23.173 INFO:tasks.rados.rados.0.smithi090.stdout:1627: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2023-01-15T14:35:23.173 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:35:23.174 INFO:tasks.rados.rados.0.smithi090.stdout:1627: writing smithi09025264-20 from 514446 to 1251196 tid 1 2023-01-15T14:35:23.176 INFO:tasks.rados.rados.0.smithi090.stdout:1627: writing smithi09025264-20 from 1723298 to 2391167 tid 2 2023-01-15T14:35:23.176 INFO:tasks.rados.rados.0.smithi090.stdout:1627: writing smithi09025264-20 from 3073078 to 3714446 tid 3 2023-01-15T14:35:23.176 INFO:tasks.rados.rados.0.smithi090.stdout:1628: write (excl) oid 42 current snap is 178 2023-01-15T14:35:23.176 INFO:tasks.rados.rados.0.smithi090.stdout:1620: finishing write tid 4 to smithi09025264-13 2023-01-15T14:35:23.176 INFO:tasks.rados.rados.0.smithi090.stdout:1628: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2023-01-15T14:35:23.177 INFO:tasks.rados.rados.0.smithi090.stdout:1628: writing smithi09025264-42 from 531253 to 943252 tid 1 2023-01-15T14:35:23.177 INFO:tasks.rados.rados.0.smithi090.stdout:1628: writing smithi09025264-42 from 1381780 to 1793307 tid 2 2023-01-15T14:35:23.177 INFO:tasks.rados.rados.0.smithi090.stdout:1628: writing smithi09025264-42 from 2300501 to 2887467 tid 3 2023-01-15T14:35:23.177 INFO:tasks.rados.rados.0.smithi090.stdout:1628: writing smithi09025264-42 from 3376974 to 3731253 tid 4 2023-01-15T14:35:23.177 INFO:tasks.rados.rados.0.smithi090.stdout:1629: snap_create 2023-01-15T14:35:23.177 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:35:23.177 INFO:tasks.rados.rados.0.smithi090.stdout:1623: finishing write tid 6 to smithi09025264-43 2023-01-15T14:35:23.177 INFO:tasks.rados.rados.0.smithi090.stdout:1623: oid 43 updating version 0 to 116 2023-01-15T14:35:23.177 INFO:tasks.rados.rados.0.smithi090.stdout:1623: oid 43 updating version 116 to 118 2023-01-15T14:35:23.177 INFO:tasks.rados.rados.0.smithi090.stdout:1623: oid 43 version 118 is already newer than 117 2023-01-15T14:35:23.177 INFO:tasks.rados.rados.0.smithi090.stdout:1623: oid 43 updating version 118 to 120 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout:1623: oid 43 version 120 is already newer than 119 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 120 (ObjNum 577 snap 178 seq_num 577) dirty exists 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout:1623: left oid 43 (ObjNum 577 snap 178 seq_num 577) 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout:1623: done (5 left) 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout:1620: finishing write tid 5 to smithi09025264-13 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout:1620: oid 13 updating version 0 to 371 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout:1620: oid 13 updating version 371 to 372 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout:1620: oid 13 version 372 is already newer than 370 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout:1620: oid 13 version 372 is already newer than 369 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 372 (ObjNum 576 snap 178 seq_num 576) dirty exists 2023-01-15T14:35:23.178 INFO:tasks.rados.rados.0.smithi090.stdout:1620: left oid 13 (ObjNum 576 snap 178 seq_num 576) 2023-01-15T14:35:23.179 INFO:tasks.rados.rados.0.smithi090.stdout:1620: done (4 left) 2023-01-15T14:35:23.179 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 4 2023-01-15T14:35:23.179 INFO:tasks.rados.rados.0.smithi090.stdout:1621: expect (ObjNum 265 snap 65 seq_num 265) 2023-01-15T14:35:23.179 INFO:tasks.rados.rados.0.smithi090.stdout:1627: finishing write tid 1 to smithi09025264-20 2023-01-15T14:35:23.179 INFO:tasks.rados.rados.0.smithi090.stdout:1621: done (3 left) 2023-01-15T14:35:23.179 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:35:23.179 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:23.166+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:23.188 INFO:tasks.rados.rados.0.smithi090.stdout:1627: finishing write tid 2 to smithi09025264-20 2023-01-15T14:35:23.274 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:23.266+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:23.535 INFO:tasks.rados.rados.0.smithi090.stdout:1627: finishing write tid 3 to smithi09025264-20 2023-01-15T14:35:23.535 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:23.536 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:35:23.536 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:23.538 INFO:tasks.rados.rados.0.smithi090.stdout:1627: finishing write tid 4 to smithi09025264-20 2023-01-15T14:35:23.538 INFO:tasks.rados.rados.0.smithi090.stdout:1627: finishing write tid 5 to smithi09025264-20 2023-01-15T14:35:23.538 INFO:tasks.rados.rados.0.smithi090.stdout:1627: oid 20 updating version 0 to 266 2023-01-15T14:35:23.538 INFO:tasks.rados.rados.0.smithi090.stdout:1627: oid 20 version 266 is already newer than 265 2023-01-15T14:35:23.539 INFO:tasks.rados.rados.0.smithi090.stdout:1627: oid 20 updating version 266 to 267 2023-01-15T14:35:23.539 INFO:tasks.rados.rados.0.smithi090.stdout:1627: oid 20 version 267 is already newer than 264 2023-01-15T14:35:23.539 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 267 (ObjNum 578 snap 178 seq_num 578) dirty exists 2023-01-15T14:35:23.539 INFO:tasks.rados.rados.0.smithi090.stdout:1627: left oid 20 (ObjNum 578 snap 178 seq_num 578) 2023-01-15T14:35:23.539 INFO:tasks.rados.rados.0.smithi090.stdout:1627: done (2 left) 2023-01-15T14:35:23.539 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:35:23.541 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:35:23.542 INFO:tasks.rados.rados.0.smithi090.stdout:1625: finishing copy_from to smithi09025264-49 2023-01-15T14:35:23.542 INFO:tasks.rados.rados.0.smithi090.stdout:1625: got expected ENOENT (src dne) 2023-01-15T14:35:23.542 INFO:tasks.rados.rados.0.smithi090.stdout:1625: finishing copy_from racing read to smithi09025264-49 2023-01-15T14:35:23.542 INFO:tasks.rados.rados.0.smithi090.stdout:1625: done (1 left) 2023-01-15T14:35:23.543 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:23.543 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:23.526+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:23.636 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:23.629+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:23.742 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:23.742 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:23.742 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:23.742 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:23.742 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:23.743 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:35:23.743 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:23.734+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:23.817 INFO:tasks.rados.rados.0.smithi090.stdout:1628: finishing write tid 1 to smithi09025264-42 2023-01-15T14:35:23.947 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:23.947 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:23.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:23.948 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:23.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.948 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:170 3.7 3:ea27a813:::smithi09025264-43:head [write 1758126~644343] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:23.948 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:35:23.948 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 416.72943275900002, 2023-01-15T14:35:23.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76504344199999996, 2023-01-15T14:35:23.949 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:23.949 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:23.949 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:23.949 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:23.950 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:23.950 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 170 2023-01-15T14:35:23.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.950 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:23.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:23.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:35:23.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:23.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128364+0000", 2023-01-15T14:35:23.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:23.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:23.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128362+0000", 2023-01-15T14:35:23.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:23.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:23.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128670+0000", 2023-01-15T14:35:23.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030817799999999999 2023-01-15T14:35:23.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:23.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128701+0000", 2023-01-15T14:35:23.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1151e-05 2023-01-15T14:35:23.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:23.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128719+0000", 2023-01-15T14:35:23.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7723e-05 2023-01-15T14:35:23.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131056+0000", 2023-01-15T14:35:23.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0023369839999999998 2023-01-15T14:35:23.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:23.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.131304+0000", 2023-01-15T14:35:23.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247741 2023-01-15T14:35:23.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:23.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330230+0000", 2023-01-15T14:35:23.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.198926191 2023-01-15T14:35:23.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:23.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330360+0000", 2023-01-15T14:35:23.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013008400000000001 2023-01-15T14:35:23.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:35:23.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.330583+0000", 2023-01-15T14:35:23.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022332999999999999 2023-01-15T14:35:23.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:23.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331460+0000", 2023-01-15T14:35:23.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00087655100000000004 2023-01-15T14:35:23.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:23.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331488+0000", 2023-01-15T14:35:23.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8007000000000001e-05 2023-01-15T14:35:23.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:23.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331868+0000", 2023-01-15T14:35:23.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037965200000000002 2023-01-15T14:35:23.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:23.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332591+0000", 2023-01-15T14:35:23.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072314600000000003 2023-01-15T14:35:23.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.437029+0000", 2023-01-15T14:35:23.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10443859899999999 2023-01-15T14:35:23.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:23.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893236+0000", 2023-01-15T14:35:23.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.45620644999999999 2023-01-15T14:35:23.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:23.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893341+0000", 2023-01-15T14:35:23.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104774 2023-01-15T14:35:23.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:23.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:23.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:24.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893407+0000", 2023-01-15T14:35:24.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6821000000000003e-05 2023-01-15T14:35:24.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:24.338 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:24.338 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:24.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.339 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:171 3.7 3:ea27a813:::smithi09025264-43:head [setxattr _header (56) in=7b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:24.339 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:35:24.339 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 416.72902942299999, 2023-01-15T14:35:24.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76490074699999999, 2023-01-15T14:35:24.339 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:24.339 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:24.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:24.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:24.339 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:24.339 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 171 2023-01-15T14:35:24.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.340 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:24.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:24.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:35:24.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:24.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:24.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128767+0000", 2023-01-15T14:35:24.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:24.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:24.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128766+0000", 2023-01-15T14:35:24.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:24.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:24.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128773+0000", 2023-01-15T14:35:24.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3000000000000004e-06 2023-01-15T14:35:24.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:24.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.128788+0000", 2023-01-15T14:35:24.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5064e-05 2023-01-15T14:35:24.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:24.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228941+0000", 2023-01-15T14:35:24.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100152673 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229086+0000", 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014478099999999999 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229421+0000", 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033531499999999998 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:24.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331642+0000", 2023-01-15T14:35:24.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10222099900000001 2023-01-15T14:35:24.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:24.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.331819+0000", 2023-01-15T14:35:24.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017660399999999999 2023-01-15T14:35:24.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,7", 2023-01-15T14:35:24.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332050+0000", 2023-01-15T14:35:24.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000231635 2023-01-15T14:35:24.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:24.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332240+0000", 2023-01-15T14:35:24.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019003999999999999 2023-01-15T14:35:24.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:24.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332257+0000", 2023-01-15T14:35:24.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6583999999999999e-05 2023-01-15T14:35:24.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:24.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332337+0000", 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9713000000000001e-05 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332616+0000", 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000279051 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.537201+0000", 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20458557999999999 2023-01-15T14:35:24.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:24.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893508+0000", 2023-01-15T14:35:24.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35630658100000001 2023-01-15T14:35:24.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:24.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893585+0000", 2023-01-15T14:35:24.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6647e-05 2023-01-15T14:35:24.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:24.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.893668+0000", 2023-01-15T14:35:24.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3354000000000002e-05 2023-01-15T14:35:24.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:24.349 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:24.349 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:24.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.349 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:172 3.7 3:ea27a813:::smithi09025264-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:24.349 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:35:24.350 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 416.62880580299998, 2023-01-15T14:35:24.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.76516487499999997, 2023-01-15T14:35:24.350 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:24.350 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:24.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:24.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:24.350 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:24.350 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 172 2023-01-15T14:35:24.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.350 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:24.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:24.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:35:24.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:24.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228991+0000", 2023-01-15T14:35:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.228989+0000", 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229002+0000", 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.288e-05 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229027+0000", 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4967e-05 2023-01-15T14:35:24.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.229044+0000", 2023-01-15T14:35:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7217000000000001e-05 2023-01-15T14:35:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329577+0000", 2023-01-15T14:35:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100533335 2023-01-15T14:35:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.329716+0000", 2023-01-15T14:35:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138894 2023-01-15T14:35:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332412+0000", 2023-01-15T14:35:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0026964549999999999 2023-01-15T14:35:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.332541+0000", 2023-01-15T14:35:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000128121 2023-01-15T14:35:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:24.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993867+0000", 2023-01-15T14:35:24.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.66132623099999999 2023-01-15T14:35:24.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:24.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.993991+0000", 2023-01-15T14:35:24.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000124091 2023-01-15T14:35:24.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:24.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.994156+0000", 2023-01-15T14:35:24.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016484400000000001 2023-01-15T14:35:24.567 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:24.567 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:24.567 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:24.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.567 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1285 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:24.567 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:35:24.567 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 325.93549057799999, 2023-01-15T14:35:24.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0605659869999999, 2023-01-15T14:35:24.568 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:24.568 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:24.568 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:24.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:24.573 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:24.573 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1285 2023-01-15T14:35:24.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.574 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:24.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:24.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:35:24.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:24.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:24.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922306+0000", 2023-01-15T14:35:24.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:24.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922305+0000", 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922467+0000", 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000162504 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:24.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922488+0000", 2023-01-15T14:35:24.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0599999999999999e-05 2023-01-15T14:35:24.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.922497+0000", 2023-01-15T14:35:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.73e-06 2023-01-15T14:35:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159278+0000", 2023-01-15T14:35:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.23678047399999999 2023-01-15T14:35:24.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159293+0000", 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4645e-05 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982771+0000", 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8234786160000001 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.982872+0000", 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000100895 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1286 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 325.92902922399998, 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0565264540000001, 2023-01-15T14:35:24.765 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:24.766 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:24.969 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:24.969 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1286 2023-01-15T14:35:24.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.970 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:24.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:24.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:35:24.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:24.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:24.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928767+0000", 2023-01-15T14:35:24.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:24.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:24.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928766+0000", 2023-01-15T14:35:24.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:24.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:24.988 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:24.988 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:24.988 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928955+0000", 2023-01-15T14:35:25.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018916399999999999 2023-01-15T14:35:25.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:25.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.928976+0000", 2023-01-15T14:35:25.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0519000000000001e-05 2023-01-15T14:35:25.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:25.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029144+0000", 2023-01-15T14:35:25.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100168271 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159343+0000", 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.13019868700000001 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159354+0000", 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1694e-05 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985260+0000", 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259052310000001 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985294+0000", 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4295999999999998e-05 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1287 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:35:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 325.828607299, 2023-01-15T14:35:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.956155608, 2023-01-15T14:35:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:25.176 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1287 2023-01-15T14:35:25.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.177 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:25.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:25.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:35:25.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:25.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029189+0000", 2023-01-15T14:35:25.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:25.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029187+0000", 2023-01-15T14:35:25.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:25.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:25.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029461+0000", 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027461400000000001 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029487+0000", 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5989000000000001e-05 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029504+0000", 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.641e-05 2023-01-15T14:35:25.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159400+0000", 2023-01-15T14:35:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12989614199999999 2023-01-15T14:35:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159408+0000", 2023-01-15T14:35:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.354e-06 2023-01-15T14:35:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985328+0000", 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8259196129999999 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985345+0000", 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7033000000000001e-05 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.182 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1288 3.18 3.644b5578 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:25.182 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:35:25.182 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 325.82826981300002, 2023-01-15T14:35:25.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.955872625, 2023-01-15T14:35:25.182 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:25.182 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:25.182 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:25.182 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:25.182 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:25.182 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1288 2023-01-15T14:35:25.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.183 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:25.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:25.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:35:25.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:25.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029527+0000", 2023-01-15T14:35:25.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:25.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029526+0000", 2023-01-15T14:35:25.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:25.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:25.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029536+0000", 2023-01-15T14:35:25.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0149999999999999e-05 2023-01-15T14:35:25.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029555+0000", 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9570000000000001e-05 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029567+0000", 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1739000000000001e-05 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159489+0000", 2023-01-15T14:35:25.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12992189500000001 2023-01-15T14:35:25.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159502+0000", 2023-01-15T14:35:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3097e-05 2023-01-15T14:35:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985388+0000", 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8258855710000002 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985399+0000", 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.184e-05 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1289 3.18 3.644b5578 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:35:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 325.82821391700003, 2023-01-15T14:35:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9558554849999998, 2023-01-15T14:35:25.375 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:25.378 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:25.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:25.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:25.379 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:25.379 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1289 2023-01-15T14:35:25.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.380 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:25.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:25.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:35:25.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:25.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029583+0000", 2023-01-15T14:35:25.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:25.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029582+0000", 2023-01-15T14:35:25.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:25.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:25.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029589+0000", 2023-01-15T14:35:25.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:35:25.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:25.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029610+0000", 2023-01-15T14:35:25.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0636999999999999e-05 2023-01-15T14:35:25.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:25.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.029621+0000", 2023-01-15T14:35:25.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1600000000000001e-05 2023-01-15T14:35:25.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159568+0000", 2023-01-15T14:35:25.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.12994715200000001 2023-01-15T14:35:25.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:25.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.159577+0000", 2023-01-15T14:35:25.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3599999999999996e-06 2023-01-15T14:35:25.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985427+0000", 2023-01-15T14:35:25.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.82585051 2023-01-15T14:35:25.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:25.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.985438+0000", 2023-01-15T14:35:25.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1e-05 2023-01-15T14:35:25.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.385 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:25.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.385 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1692 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e153)", 2023-01-15T14:35:25.385 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:35:25.385 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 283.99496349899999, 2023-01-15T14:35:25.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72663678300000001, 2023-01-15T14:35:25.386 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:25.386 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:25.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:25.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:25.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:25.386 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1692 2023-01-15T14:35:25.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.386 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:25.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862833+0000", 2023-01-15T14:35:25.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:25.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:25.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862836+0000", 2023-01-15T14:35:25.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.942e-06 2023-01-15T14:35:25.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:25.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862842+0000", 2023-01-15T14:35:25.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3799999999999999e-06 2023-01-15T14:35:25.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:25.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.862846+0000", 2023-01-15T14:35:25.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1500000000000001e-06 2023-01-15T14:35:25.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963065+0000", 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100218523 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:39.963073+0000", 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3680000000000002e-06 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152184+0000", 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18911077500000001 2023-01-15T14:35:25.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:25.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.152321+0000", 2023-01-15T14:35:25.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013738300000000001 2023-01-15T14:35:25.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.370975+0000", 2023-01-15T14:35:25.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21865378899999999 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.371190+0000", 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215217 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.588877+0000", 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.21768679499999999 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589141+0000", 2023-01-15T14:35:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000264312 2023-01-15T14:35:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:40.589470+0000", 2023-01-15T14:35:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032859300000000002 2023-01-15T14:35:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.580 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2207 3.5 3:a1c47f99:::smithi09025264-2:head [write 2548663~763022] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:35:25.581 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:35:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 225.02020577100001, 2023-01-15T14:35:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79657873300000004, 2023-01-15T14:35:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:25.582 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2207 2023-01-15T14:35:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:35:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:25.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837591+0000", 2023-01-15T14:35:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:25.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837590+0000", 2023-01-15T14:35:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837960+0000", 2023-01-15T14:35:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037055900000000001 2023-01-15T14:35:25.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837980+0000", 2023-01-15T14:35:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9953e-05 2023-01-15T14:35:25.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.837988+0000", 2023-01-15T14:35:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2800000000000003e-06 2023-01-15T14:35:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927512+0000", 2023-01-15T14:35:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089523563 2023-01-15T14:35:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927645+0000", 2023-01-15T14:35:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013285799999999999 2023-01-15T14:35:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225466+0000", 2023-01-15T14:35:25.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29782133100000002 2023-01-15T14:35:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225553+0000", 2023-01-15T14:35:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6866000000000005e-05 2023-01-15T14:35:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.593 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:35:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225708+0000", 2023-01-15T14:35:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015455199999999999 2023-01-15T14:35:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.225971+0000", 2023-01-15T14:35:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026317099999999998 2023-01-15T14:35:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226027+0000", 2023-01-15T14:35:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.6601000000000002e-05 2023-01-15T14:35:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226470+0000", 2023-01-15T14:35:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044267500000000001 2023-01-15T14:35:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227372+0000", 2023-01-15T14:35:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00090215200000000003 2023-01-15T14:35:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429314+0000", 2023-01-15T14:35:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20194183499999999 2023-01-15T14:35:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.633999+0000", 2023-01-15T14:35:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20468505400000001 2023-01-15T14:35:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634075+0000", 2023-01-15T14:35:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.5729000000000006e-05 2023-01-15T14:35:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634170+0000", 2023-01-15T14:35:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4821999999999994e-05 2023-01-15T14:35:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2208 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (56) in=7b,truncate 3395047] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:35:25.598 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:35:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 225.01978317800001, 2023-01-15T14:35:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79650439900000003, 2023-01-15T14:35:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2208 2023-01-15T14:35:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.599 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:25.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:25.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:35:25.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:25.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:35:25.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:25.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838013+0000", 2023-01-15T14:35:25.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:25.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:25.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838018+0000", 2023-01-15T14:35:25.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2549999999999997e-06 2023-01-15T14:35:25.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:25.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838031+0000", 2023-01-15T14:35:25.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2717999999999999e-05 2023-01-15T14:35:25.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:25.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838036+0000", 2023-01-15T14:35:25.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2240000000000003e-06 2023-01-15T14:35:25.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927676+0000", 2023-01-15T14:35:25.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089639620000000003 2023-01-15T14:35:25.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:25.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.927804+0000", 2023-01-15T14:35:25.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012832300000000001 2023-01-15T14:35:25.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226096+0000", 2023-01-15T14:35:25.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29829207400000002 2023-01-15T14:35:25.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:25.604 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226347+0000", 2023-01-15T14:35:25.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025084800000000002 2023-01-15T14:35:25.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:35:25.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226750+0000", 2023-01-15T14:35:25.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040343200000000001 2023-01-15T14:35:25.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227147+0000", 2023-01-15T14:35:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039682900000000002 2023-01-15T14:35:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227600+0000", 2023-01-15T14:35:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00045321199999999998 2023-01-15T14:35:25.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227859+0000", 2023-01-15T14:35:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000259041 2023-01-15T14:35:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:25.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227895+0000", 2023-01-15T14:35:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5710999999999997e-05 2023-01-15T14:35:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429405+0000", 2023-01-15T14:35:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201509415 2023-01-15T14:35:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634300+0000", 2023-01-15T14:35:25.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20489528200000001 2023-01-15T14:35:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634409+0000", 2023-01-15T14:35:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000109524 2023-01-15T14:35:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:25.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634518+0000", 2023-01-15T14:35:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108548 2023-01-15T14:35:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2209 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:35:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:35:25.613 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 225.019754363, 2023-01-15T14:35:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.79724995499999995, 2023-01-15T14:35:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:25.614 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2209 2023-01-15T14:35:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:35:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:35:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838042+0000", 2023-01-15T14:35:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838044+0000", 2023-01-15T14:35:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6120000000000001e-06 2023-01-15T14:35:25.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838056+0000", 2023-01-15T14:35:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1236000000000001e-05 2023-01-15T14:35:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.838059+0000", 2023-01-15T14:35:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6940000000000001e-06 2023-01-15T14:35:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.027908+0000", 2023-01-15T14:35:25.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.189848287 2023-01-15T14:35:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.028049+0000", 2023-01-15T14:35:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014132500000000001 2023-01-15T14:35:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227252+0000", 2023-01-15T14:35:25.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19920324 2023-01-15T14:35:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227341+0000", 2023-01-15T14:35:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8683000000000005e-05 2023-01-15T14:35:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634637+0000", 2023-01-15T14:35:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40729567300000002 2023-01-15T14:35:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634980+0000", 2023-01-15T14:35:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034335000000000002 2023-01-15T14:35:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.635292+0000", 2023-01-15T14:35:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0003123 2023-01-15T14:35:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:25.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2222 3.c 3:3137f0cb:::smithi09025264-20:head [write 3259733~152121] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:35:25.792 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:35:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 224.99990280399999, 2023-01-15T14:35:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.77651572400000002, 2023-01-15T14:35:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2222 2023-01-15T14:35:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:25.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857894+0000", 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:25.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857891+0000", 2023-01-15T14:35:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:35:25.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.857993+0000", 2023-01-15T14:35:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010276399999999999 2023-01-15T14:35:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858021+0000", 2023-01-15T14:35:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8022e-05 2023-01-15T14:35:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858034+0000", 2023-01-15T14:35:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2549e-05 2023-01-15T14:35:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025371+0000", 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16733653100000001 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025429+0000", 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8922000000000003e-05 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.226869+0000", 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20143931100000001 2023-01-15T14:35:25.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227088+0000", 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000219328 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227443+0000", 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035481 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227830+0000", 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000386702 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227861+0000", 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1584000000000001e-05 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227939+0000", 2023-01-15T14:35:25.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7354999999999994e-05 2023-01-15T14:35:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228548+0000", 2023-01-15T14:35:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060954800000000001 2023-01-15T14:35:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.429817+0000", 2023-01-15T14:35:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20126904600000001 2023-01-15T14:35:25.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634125+0000", 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20430780300000001 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634259+0000", 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134144 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.634410+0000", 2023-01-15T14:35:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000150447 2023-01-15T14:35:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.805 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2223 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (57) in=7b,truncate 3411854] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:35:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:35:25.988 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 224.99974401700001, 2023-01-15T14:35:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.87671107599999998, 2023-01-15T14:35:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2223 2023-01-15T14:35:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:25.989 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858053+0000", 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:25.990 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858052+0000", 2023-01-15T14:35:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858056+0000", 2023-01-15T14:35:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5619999999999997e-06 2023-01-15T14:35:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:25.991 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858067+0000", 2023-01-15T14:35:25.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0328e-05 2023-01-15T14:35:25.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:25.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858071+0000", 2023-01-15T14:35:25.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6439999999999999e-06 2023-01-15T14:35:25.992 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.992 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.992 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.992 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025446+0000", 2023-01-15T14:35:25.992 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.16737407200000001 2023-01-15T14:35:25.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:25.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025503+0000", 2023-01-15T14:35:25.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7952999999999999e-05 2023-01-15T14:35:25.993 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.993 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.993 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:25.993 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.227970+0000", 2023-01-15T14:35:25.993 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.202466866 2023-01-15T14:35:25.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:25.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228139+0000", 2023-01-15T14:35:25.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016889100000000001 2023-01-15T14:35:25.994 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.994 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.994 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 1,2", 2023-01-15T14:35:25.994 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228284+0000", 2023-01-15T14:35:25.994 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014493700000000001 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228470+0000", 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000185541 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228481+0000", 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.167e-05 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.995 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:25.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228537+0000", 2023-01-15T14:35:25.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5912999999999998e-05 2023-01-15T14:35:25.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:25.996 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228563+0000", 2023-01-15T14:35:25.996 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5613e-05 2023-01-15T14:35:25.996 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.996 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.996 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:25.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.433181+0000", 2023-01-15T14:35:25.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20461838099999999 2023-01-15T14:35:25.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:25.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734620+0000", 2023-01-15T14:35:25.997 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301438386 2023-01-15T14:35:25.997 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.997 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.997 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:25.997 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734682+0000", 2023-01-15T14:35:25.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2373999999999996e-05 2023-01-15T14:35:25.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.998 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:25.998 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.734764+0000", 2023-01-15T14:35:25.998 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1656999999999994e-05 2023-01-15T14:35:25.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.998 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:25.998 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:25.998 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:25.998 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:25.999 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2224 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e212)", 2023-01-15T14:35:25.999 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:35:25.999 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 224.99971896700001, 2023-01-15T14:35:25.999 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.97704279199999999, 2023-01-15T14:35:25.999 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:25.999 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:25.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:25.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:25.999 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:25.999 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2224 2023-01-15T14:35:26.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.000 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:26.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:26.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:35:26.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.000 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.000 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.000 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:26.000 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858078+0000", 2023-01-15T14:35:26.000 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858077+0000", 2023-01-15T14:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858080+0000", 2023-01-15T14:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9280000000000002e-06 2023-01-15T14:35:26.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858089+0000", 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3449999999999995e-06 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:38.858093+0000", 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9820000000000002e-06 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.002 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025520+0000", 2023-01-15T14:35:26.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.167426556 2023-01-15T14:35:26.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.003 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:26.003 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.025594+0000", 2023-01-15T14:35:26.003 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4339999999999996e-05 2023-01-15T14:35:26.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.004 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228608+0000", 2023-01-15T14:35:26.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20301395699999999 2023-01-15T14:35:26.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.010 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:26.010 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.228752+0000", 2023-01-15T14:35:26.010 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014385 2023-01-15T14:35:26.010 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.010 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834878+0000", 2023-01-15T14:35:26.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60612591500000002 2023-01-15T14:35:26.011 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.011 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.011 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:26.011 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.834987+0000", 2023-01-15T14:35:26.011 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000108733 2023-01-15T14:35:26.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:26.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:39.835120+0000", 2023-01-15T14:35:26.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133688 2023-01-15T14:35:26.195 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.196 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3120 3.40 3:03b54904:::smithi09025264-12:head [write 578273~466432] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:35:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:35:26.197 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.92576708499999, 2023-01-15T14:35:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72661194799999995, 2023-01-15T14:35:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3120 2023-01-15T14:35:26.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932030+0000", 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932026+0000", 2023-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:26.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932397+0000", 2023-01-15T14:35:26.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037070099999999997 2023-01-15T14:35:26.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:26.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932419+0000", 2023-01-15T14:35:26.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1923999999999999e-05 2023-01-15T14:35:26.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932435+0000", 2023-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6509999999999999e-05 2023-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933751+0000", 2023-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0013156050000000001 2023-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933765+0000", 2023-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3952e-05 2023-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238456+0000", 2023-01-15T14:35:26.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30469085000000001 2023-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.238797+0000", 2023-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034117300000000001 2023-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239214+0000", 2023-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000416845 2023-01-15T14:35:26.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240273+0000", 2023-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058811 2023-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240292+0000", 2023-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240555+0000", 2023-01-15T14:35:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026299500000000003 2023-01-15T14:35:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242072+0000", 2023-01-15T14:35:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0015175550000000001 2023-01-15T14:35:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244038+0000", 2023-01-15T14:35:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001966176 2023-01-15T14:35:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658099+0000", 2023-01-15T14:35:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41406103300000002 2023-01-15T14:35:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658248+0000", 2023-01-15T14:35:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000148577 2023-01-15T14:35:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658642+0000", 2023-01-15T14:35:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039346900000000002 2023-01-15T14:35:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3121 3.40 3:03b54904:::smithi09025264-12:head [write 1739595~452815] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:35:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:35:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.925335139, 2023-01-15T14:35:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72675957599999996, 2023-01-15T14:35:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:26.210 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3121 2023-01-15T14:35:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:35:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932462+0000", 2023-01-15T14:35:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932460+0000", 2023-01-15T14:35:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:26.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932725+0000", 2023-01-15T14:35:26.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026482699999999998 2023-01-15T14:35:26.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932741+0000", 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5880999999999999e-05 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932750+0000", 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2380000000000003e-06 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034479+0000", 2023-01-15T14:35:26.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10172906399999999 2023-01-15T14:35:26.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:26.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034506+0000", 2023-01-15T14:35:26.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7336000000000001e-05 2023-01-15T14:35:26.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240672+0000", 2023-01-15T14:35:26.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206166074 2023-01-15T14:35:26.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:26.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.240894+0000", 2023-01-15T14:35:26.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000221189 2023-01-15T14:35:26.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:35:26.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241244+0000", 2023-01-15T14:35:26.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000350816 2023-01-15T14:35:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241923+0000", 2023-01-15T14:35:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00067891200000000005 2023-01-15T14:35:26.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241943+0000", 2023-01-15T14:35:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0069999999999999e-05 2023-01-15T14:35:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243425+0000", 2023-01-15T14:35:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001481719 2023-01-15T14:35:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:26.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244087+0000", 2023-01-15T14:35:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066184700000000002 2023-01-15T14:35:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244230+0000", 2023-01-15T14:35:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000143277 2023-01-15T14:35:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658784+0000", 2023-01-15T14:35:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.414553479 2023-01-15T14:35:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.658904+0000", 2023-01-15T14:35:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000120024 2023-01-15T14:35:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659221+0000", 2023-01-15T14:35:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031742700000000001 2023-01-15T14:35:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3122 3.40 3:03b54904:::smithi09025264-12:head [write 2722929~655344] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:35:26.415 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:35:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.92503104100001, 2023-01-15T14:35:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72706898900000005, 2023-01-15T14:35:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3122 2023-01-15T14:35:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.416 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:35:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932766+0000", 2023-01-15T14:35:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.932765+0000", 2023-01-15T14:35:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933097+0000", 2023-01-15T14:35:26.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00033220299999999999 2023-01-15T14:35:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933106+0000", 2023-01-15T14:35:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5459999999999997e-06 2023-01-15T14:35:26.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933113+0000", 2023-01-15T14:35:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3419999999999996e-06 2023-01-15T14:35:26.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034598+0000", 2023-01-15T14:35:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101485804 2023-01-15T14:35:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034625+0000", 2023-01-15T14:35:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6859000000000002e-05 2023-01-15T14:35:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242147+0000", 2023-01-15T14:35:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20752189600000001 2023-01-15T14:35:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:26.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.242517+0000", 2023-01-15T14:35:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037032200000000002 2023-01-15T14:35:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:35:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243157+0000", 2023-01-15T14:35:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00063935700000000001 2023-01-15T14:35:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243809+0000", 2023-01-15T14:35:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065222000000000001 2023-01-15T14:35:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243827+0000", 2023-01-15T14:35:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8260000000000001e-05 2023-01-15T14:35:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244122+0000", 2023-01-15T14:35:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029469400000000002 2023-01-15T14:35:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:26.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245633+0000", 2023-01-15T14:35:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001511068 2023-01-15T14:35:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.346711+0000", 2023-01-15T14:35:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101077812 2023-01-15T14:35:26.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659372+0000", 2023-01-15T14:35:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31266154800000001 2023-01-15T14:35:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659488+0000", 2023-01-15T14:35:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115444 2023-01-15T14:35:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:26.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.659835+0000", 2023-01-15T14:35:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034670300000000002 2023-01-15T14:35:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3123 3.40 3:03b54904:::smithi09025264-12:head [setxattr _header (58) in=7b,truncate 3378273] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:35:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:35:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.924661348, 2023-01-15T14:35:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82751564099999997, 2023-01-15T14:35:26.432 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:26.433 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3123 2023-01-15T14:35:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:35:26.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:35:26.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933135+0000", 2023-01-15T14:35:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:26.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933139+0000", 2023-01-15T14:35:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7079999999999996e-06 2023-01-15T14:35:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:26.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:08.933145+0000", 2023-01-15T14:35:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7849999999999997e-06 2023-01-15T14:35:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033331+0000", 2023-01-15T14:35:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100186076 2023-01-15T14:35:26.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034697+0000", 2023-01-15T14:35:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001366138 2023-01-15T14:35:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.034718+0000", 2023-01-15T14:35:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.065e-05 2023-01-15T14:35:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244319+0000", 2023-01-15T14:35:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20960071 2023-01-15T14:35:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244576+0000", 2023-01-15T14:35:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025712899999999998 2023-01-15T14:35:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:35:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.244932+0000", 2023-01-15T14:35:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035626000000000001 2023-01-15T14:35:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:26.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245315+0000", 2023-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00038305199999999999 2023-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:26.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245330+0000", 2023-01-15T14:35:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5027e-05 2023-01-15T14:35:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245384+0000", 2023-01-15T14:35:26.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4267000000000001e-05 2023-01-15T14:35:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245654+0000", 2023-01-15T14:35:26.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026944000000000002 2023-01-15T14:35:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.436220+0000", 2023-01-15T14:35:26.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19056593799999999 2023-01-15T14:35:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760086+0000", 2023-01-15T14:35:26.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.323865927 2023-01-15T14:35:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760241+0000", 2023-01-15T14:35:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015535700000000001 2023-01-15T14:35:26.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760651+0000", 2023-01-15T14:35:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040990699999999999 2023-01-15T14:35:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:26.451 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3124 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e299)", 2023-01-15T14:35:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:35:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 134.82443487, 2023-01-15T14:35:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72792673799999996, 2023-01-15T14:35:26.452 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3124 2023-01-15T14:35:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.453 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:35:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:26.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033362+0000", 2023-01-15T14:35:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033360+0000", 2023-01-15T14:35:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:26.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033372+0000", 2023-01-15T14:35:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2850000000000001e-05 2023-01-15T14:35:26.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033397+0000", 2023-01-15T14:35:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4349e-05 2023-01-15T14:35:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.033412+0000", 2023-01-15T14:35:26.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5420000000000001e-05 2023-01-15T14:35:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036811+0000", 2023-01-15T14:35:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0033984169999999999 2023-01-15T14:35:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.036821+0000", 2023-01-15T14:35:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0213e-05 2023-01-15T14:35:26.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245712+0000", 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20889102800000001 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245912+0000", 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020024600000000001 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760781+0000", 2023-01-15T14:35:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51486881299999998 2023-01-15T14:35:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.760970+0000", 2023-01-15T14:35:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018893600000000001 2023-01-15T14:35:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:26.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.761289+0000", 2023-01-15T14:35:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00031870899999999997 2023-01-15T14:35:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:26.461 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:26.462 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:26.462 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:26.569 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:35:26.569 INFO:tasks.rados.rados.0.smithi090.stdout:1628: finishing write tid 2 to smithi09025264-42 2023-01-15T14:35:26.571 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:23.938+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:26.573 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:35:26.574 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:35:26.574 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1, 6] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:35:26.575 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:35:26.575 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:35:26.580 INFO:tasks.rados.rados.0.smithi090.stdout:1628: finishing write tid 3 to smithi09025264-42 2023-01-15T14:35:26.580 INFO:tasks.rados.rados.0.smithi090.stdout:1628: finishing write tid 4 to smithi09025264-42 2023-01-15T14:35:26.580 INFO:tasks.rados.rados.0.smithi090.stdout:1628: finishing write tid 5 to smithi09025264-42 2023-01-15T14:35:26.580 INFO:tasks.rados.rados.0.smithi090.stdout:1628: finishing write tid 6 to smithi09025264-42 2023-01-15T14:35:26.581 INFO:tasks.rados.rados.0.smithi090.stdout:1628: oid 42 updating version 0 to 366 2023-01-15T14:35:26.581 INFO:tasks.rados.rados.0.smithi090.stdout:1628: oid 42 version 366 is already newer than 363 2023-01-15T14:35:26.581 INFO:tasks.rados.rados.0.smithi090.stdout:1628: oid 42 updating version 366 to 367 2023-01-15T14:35:26.581 INFO:tasks.rados.rados.0.smithi090.stdout:1628: oid 42 version 367 is already newer than 364 2023-01-15T14:35:26.581 INFO:tasks.rados.rados.0.smithi090.stdout:1628: oid 42 version 367 is already newer than 365 2023-01-15T14:35:26.581 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 42 v 367 (ObjNum 579 snap 178 seq_num 579) dirty exists 2023-01-15T14:35:26.581 INFO:tasks.rados.rados.0.smithi090.stdout:1628: left oid 42 (ObjNum 579 snap 178 seq_num 579) 2023-01-15T14:35:26.581 INFO:tasks.rados.rados.0.smithi090.stdout:1628: done (0 left) 2023-01-15T14:35:26.581 INFO:tasks.rados.rados.0.smithi090.stdout:1629: done (0 left) 2023-01-15T14:35:26.581 INFO:tasks.rados.rados.0.smithi090.stdout:1630: write (excl) oid 6 current snap is 179 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1630: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1630: writing smithi09025264-6 from 548060 to 1035308 tid 1 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1630: writing smithi09025264-6 from 1440262 to 2111800 tid 2 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1630: writing smithi09025264-6 from 2844277 to 3642515 tid 3 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1631: write oid 31 current snap is 179 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1631: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1631: writing smithi09025264-31 from 564867 to 1127364 tid 1 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1631: writing smithi09025264-31 from 1898744 to 2313940 tid 2 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1631: writing smithi09025264-31 from 2755347 to 3481210 tid 3 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1632: delete oid 42 current snap is 179 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1630: finishing write tid 1 to smithi09025264-6 2023-01-15T14:35:26.582 INFO:tasks.rados.rados.0.smithi090.stdout:1632: done (2 left) 2023-01-15T14:35:26.583 INFO:tasks.rados.rados.0.smithi090.stdout:1633: rollback oid 30 current snap is 179 2023-01-15T14:35:26.583 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 30 to 177 2023-01-15T14:35:26.583 INFO:tasks.rados.rados.0.smithi090.stdout:1634: write (excl) oid 9 current snap is 179 2023-01-15T14:35:26.583 INFO:tasks.rados.rados.0.smithi090.stdout:1634: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2023-01-15T14:35:26.583 INFO:tasks.rados.rados.0.smithi090.stdout:1634: writing smithi09025264-9 from 598481 to 1311476 tid 1 2023-01-15T14:35:26.583 INFO:tasks.rados.rados.0.smithi090.stdout:1634: writing smithi09025264-9 from 1899355 to 2318220 tid 2 2023-01-15T14:35:26.583 INFO:tasks.rados.rados.0.smithi090.stdout:1634: writing smithi09025264-9 from 3093840 to 3798481 tid 3 2023-01-15T14:35:26.583 INFO:tasks.rados.rados.0.smithi090.stdout:1635: write oid 28 current snap is 179 2023-01-15T14:35:26.583 INFO:tasks.rados.rados.0.smithi090.stdout:1634: finishing write tid 1 to smithi09025264-9 2023-01-15T14:35:26.583 INFO:tasks.rados.rados.0.smithi090.stdout:1635: seq_num 584 ranges {615288=788244,1957837=562523,3121263=559679} 2023-01-15T14:35:26.583 INFO:tasks.rados.rados.0.smithi090.stdout:1635: writing smithi09025264-28 from 615288 to 1403532 tid 1 2023-01-15T14:35:26.584 INFO:tasks.rados.rados.0.smithi090.stdout:1635: writing smithi09025264-28 from 1957837 to 2520360 tid 2 2023-01-15T14:35:26.584 INFO:tasks.rados.rados.0.smithi090.stdout:1635: writing smithi09025264-28 from 3121263 to 3680942 tid 3 2023-01-15T14:35:26.584 INFO:tasks.rados.rados.0.smithi090.stdout:1636: snap_remove snap 131 2023-01-15T14:35:26.584 INFO:tasks.rados.rados.0.smithi090.stdout:1636: done (5 left) 2023-01-15T14:35:26.584 INFO:tasks.rados.rados.0.smithi090.stdout:1637: read oid 40 snap -1 2023-01-15T14:35:26.584 INFO:tasks.rados.rados.0.smithi090.stdout:1637: expect deleted 2023-01-15T14:35:26.584 INFO:tasks.rados.rados.0.smithi090.stdout:1630: finishing write tid 2 to smithi09025264-6 2023-01-15T14:35:26.584 INFO:tasks.rados.rados.0.smithi090.stdout:1630: finishing write tid 3 to smithi09025264-6 2023-01-15T14:35:26.584 INFO:tasks.rados.rados.0.smithi090.stdout:1634: finishing write tid 2 to smithi09025264-9 2023-01-15T14:35:26.584 INFO:tasks.rados.rados.0.smithi090.stdout:1630: finishing write tid 4 to smithi09025264-6 2023-01-15T14:35:26.584 INFO:tasks.rados.rados.0.smithi090.stdout:1630: finishing write tid 5 to smithi09025264-6 2023-01-15T14:35:26.585 INFO:tasks.rados.rados.0.smithi090.stdout:1630: oid 6 updating version 0 to 354 2023-01-15T14:35:26.585 INFO:tasks.rados.rados.0.smithi090.stdout:1630: oid 6 version 354 is already newer than 352 2023-01-15T14:35:26.585 INFO:tasks.rados.rados.0.smithi090.stdout:1630: oid 6 version 354 is already newer than 351 2023-01-15T14:35:26.585 INFO:tasks.rados.rados.0.smithi090.stdout:1630: oid 6 version 354 is already newer than 353 2023-01-15T14:35:26.585 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 354 (ObjNum 580 snap 179 seq_num 580) dirty exists 2023-01-15T14:35:26.585 INFO:tasks.rados.rados.0.smithi090.stdout:1630: left oid 6 (ObjNum 580 snap 179 seq_num 580) 2023-01-15T14:35:26.585 INFO:tasks.rados.rados.0.smithi090.stdout:1634: finishing write tid 3 to smithi09025264-9 2023-01-15T14:35:26.585 INFO:tasks.rados.rados.0.smithi090.stdout:1635: finishing write tid 1 to smithi09025264-28 2023-01-15T14:35:26.585 INFO:tasks.rados.rados.0.smithi090.stdout:1634: finishing write tid 4 to smithi09025264-9 2023-01-15T14:35:26.585 INFO:tasks.rados.rados.0.smithi090.stdout:1634: finishing write tid 5 to smithi09025264-9 2023-01-15T14:35:26.585 INFO:tasks.rados.rados.0.smithi090.stdout:1634: oid 9 updating version 0 to 317 2023-01-15T14:35:26.586 INFO:tasks.rados.rados.0.smithi090.stdout:1634: oid 9 version 317 is already newer than 316 2023-01-15T14:35:26.586 INFO:tasks.rados.rados.0.smithi090.stdout:1634: oid 9 updating version 317 to 318 2023-01-15T14:35:26.586 INFO:tasks.rados.rados.0.smithi090.stdout:1634: oid 9 version 318 is already newer than 315 2023-01-15T14:35:26.586 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 9 v 318 (ObjNum 583 snap 179 seq_num 583) dirty exists 2023-01-15T14:35:26.586 INFO:tasks.rados.rados.0.smithi090.stdout:1634: left oid 9 (ObjNum 583 snap 179 seq_num 583) 2023-01-15T14:35:26.586 INFO:tasks.rados.rados.0.smithi090.stdout:1635: finishing write tid 2 to smithi09025264-28 2023-01-15T14:35:26.586 INFO:tasks.rados.rados.0.smithi090.stdout:1633: finishing rollback tid 0 to smithi09025264-30 2023-01-15T14:35:26.586 INFO:tasks.rados.rados.0.smithi090.stdout:1633: finishing rollback tid 1 to smithi09025264-30 2023-01-15T14:35:26.587 INFO:tasks.rados.rados.0.smithi090.stdout:1635: finishing write tid 3 to smithi09025264-28 2023-01-15T14:35:26.587 INFO:tasks.rados.rados.0.smithi090.stdout:1633: finishing rollback tid 2 to smithi09025264-30 2023-01-15T14:35:26.587 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 248 (ObjNum 566 snap 176 seq_num 566) dirty exists 2023-01-15T14:35:26.587 INFO:tasks.rados.rados.0.smithi090.stdout:1635: finishing write tid 4 to smithi09025264-28 2023-01-15T14:35:26.587 INFO:tasks.rados.rados.0.smithi090.stdout:1635: finishing write tid 5 to smithi09025264-28 2023-01-15T14:35:26.587 INFO:tasks.rados.rados.0.smithi090.stdout:1635: oid 28 updating version 0 to 409 2023-01-15T14:35:26.587 INFO:tasks.rados.rados.0.smithi090.stdout:1635: oid 28 updating version 409 to 411 2023-01-15T14:35:26.587 INFO:tasks.rados.rados.0.smithi090.stdout:1635: oid 28 version 411 is already newer than 410 2023-01-15T14:35:26.587 INFO:tasks.rados.rados.0.smithi090.stdout:1635: oid 28 version 411 is already newer than 408 2023-01-15T14:35:26.587 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 28 v 411 (ObjNum 584 snap 179 seq_num 584) dirty exists 2023-01-15T14:35:26.587 INFO:tasks.rados.rados.0.smithi090.stdout:1635: left oid 28 (ObjNum 584 snap 179 seq_num 584) 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:1631: finishing write tid 1 to smithi09025264-31 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:1630: done (5 left) 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:1633: done (4 left) 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:1634: done (3 left) 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:1635: done (2 left) 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:1638: delete oid 39 current snap is 179 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:1638: done (2 left) 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:1639: rollback oid 26 current snap is 179 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 26 to 154 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:1640: write oid 7 current snap is 179 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:1640: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2023-01-15T14:35:26.588 INFO:tasks.rados.rados.0.smithi090.stdout:1640: writing smithi09025264-7 from 648902 to 1303997 tid 1 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1640: writing smithi09025264-7 from 2074801 to 2640993 tid 2 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1640: writing smithi09025264-7 from 3176109 to 3848902 tid 3 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1641: read oid 21 snap -1 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1641: expect (ObjNum 506 snap 150 seq_num 506) 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1642: read oid 6 snap -1 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1642: expect (ObjNum 580 snap 179 seq_num 580) 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1643: read oid 24 snap 157 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1643: expect (ObjNum 486 snap 142 seq_num 486) 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1644: delete oid 38 current snap is 179 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1637: done (7 left) 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1644: done (6 left) 2023-01-15T14:35:26.589 INFO:tasks.rados.rados.0.smithi090.stdout:1645: delete oid 43 current snap is 179 2023-01-15T14:35:26.590 INFO:tasks.rados.rados.0.smithi090.stdout:1639: finishing rollback tid 1 to smithi09025264-26 2023-01-15T14:35:26.590 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 340 (ObjNum 305 snap 0 seq_num 3192853472) dirty dne 2023-01-15T14:35:26.590 INFO:tasks.rados.rados.0.smithi090.stdout:1643: expect (ObjNum 486 snap 142 seq_num 486) 2023-01-15T14:35:26.590 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:26.566+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:26.591 INFO:tasks.rados.rados.0.smithi090.stdout:1642: expect (ObjNum 580 snap 179 seq_num 580) 2023-01-15T14:35:26.675 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:26.666+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:26.730 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:26.730 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:35:26.731 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:35:26.731 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:35:26.731 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:35:26.731 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:35:26.733 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:26.766 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:26.776 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:26.769+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:26.892 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:26.892 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:26.892 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:26.893 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:26.893 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:26.893 INFO:tasks.rados.rados.0.smithi090.stdout:1639: done (6 left) 2023-01-15T14:35:26.893 INFO:tasks.rados.rados.0.smithi090.stdout:1642: done (5 left) 2023-01-15T14:35:26.893 INFO:tasks.rados.rados.0.smithi090.stdout:1643: done (4 left) 2023-01-15T14:35:27.052 INFO:tasks.rados.rados.0.smithi090.stdout:1645: done (3 left) 2023-01-15T14:35:27.052 INFO:tasks.rados.rados.0.smithi090.stdout:1646: read oid 19 snap 173 2023-01-15T14:35:27.052 INFO:tasks.rados.rados.0.smithi090.stdout:1646: expect (ObjNum 499 snap 149 seq_num 499) 2023-01-15T14:35:27.052 INFO:tasks.rados.rados.0.smithi090.stdout:1647: copy_from oid 41 from oid 35 current snap is 179 2023-01-15T14:35:27.052 INFO:tasks.rados.rados.0.smithi090.stdout:1641: expect (ObjNum 506 snap 150 seq_num 506) 2023-01-15T14:35:27.053 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:35:27.054 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:27.054 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:26.886+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:27.054 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:27.056 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:35:27.056 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:35:27.056 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:35:27.056 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:35:27.056 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:35:27.056 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:35:27.056 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:35:27.056 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:35:27.057 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:35:27.057 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:35:27.061 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:27.049+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:27.065 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:35:27.214 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":648,"stamp":"2023-01-15T14:35:25.989844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":452266928,"num_objects":209,"num_object_clones":154,"num_object_copies":627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":59,"num_objects_unfound":0,"num_objects_dirty":209,"num_whiteouts":16,"num_read":1201,"num_read_kb":1220462,"num_write":966,"num_write_kb":341250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":897,"num_bytes_recovered":613802925,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23311,"ondisk_log_size":23311,"up":405,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":7445308,"kb_used_data":648493060,"kb_used_omap":97922,"kb_used_meta":0,"kb_avail":648450352,"statfs":{"total":671637155840,"available":664013160448,"internally_reserved":43732992,"allocated":664056893440,"data_stored":664056893440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":100272819,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":10,"commit_latency_ns":0,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19782841,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":51,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":-1,"num_read":71,"num_read_kb":74933,"num_write":46,"num_write_kb":16873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":50699231,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.034001"},"pg_stats":[{"pgid":"3.7d","version":"424'277","reported_seq":1600,"reported_epoch":433,"state":"peering","last_fresh":"2023-01-15T14:35:22.982714+0000","last_change":"2023-01-15T14:35:22.982714+0000","last_active":"2023-01-15T14:35:22.982442+0000","last_peered":"2023-01-15T14:35:22.171815+0000","last_clean":"2023-01-15T14:35:14.696293+0000","last_became_active":"2023-01-15T14:35:22.171383+0000","last_became_peered":"2023-01-15T14:35:22.171383+0000","last_unstale":"2023-01-15T14:35:22.982714+0000","last_undegraded":"2023-01-15T14:35:22.982714+0000","last_fullsized":"2023-01-15T14:35:22.982714+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":7,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2748011,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":18,"num_read_kb":20693,"num_write":10,"num_write_kb":4359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":7352992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.7c","version":"141'46","reported_seq":25,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.321315+0000","last_change":"2023-01-15T14:35:23.321315+0000","last_active":"2023-01-15T14:35:23.321315+0000","last_peered":"2023-01-15T14:35:23.321315+0000","last_clean":"2023-01-15T14:35:23.321315+0000","last_became_active":"2023-01-15T14:35:21.099039+0000","last_became_peered":"2023-01-15T14:35:21.099039+0000","last_unstale":"2023-01-15T14:35:23.321315+0000","last_undegraded":"2023-01-15T14:35:23.321315+0000","last_fullsized":"2023-01-15T14:35:23.321315+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.33955803299999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.7b","version":"221'60","reported_seq":224,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.317061+0000","last_change":"2023-01-15T14:35:22.317061+0000","last_active":"2023-01-15T14:35:22.317061+0000","last_peered":"2023-01-15T14:35:22.317061+0000","last_clean":"2023-01-15T14:35:22.317061+0000","last_became_active":"2023-01-15T14:35:21.099347+0000","last_became_peered":"2023-01-15T14:35:21.099347+0000","last_unstale":"2023-01-15T14:35:22.317061+0000","last_undegraded":"2023-01-15T14:35:22.317061+0000","last_fullsized":"2023-01-15T14:35:22.317061+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.144868356,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.7a","version":"223'238","reported_seq":1580,"reported_epoch":432,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:35:22.369973+0000","last_change":"2023-01-15T14:35:22.368797+0000","last_active":"2023-01-15T14:35:22.369973+0000","last_peered":"2023-01-15T14:35:22.369973+0000","last_clean":"2023-01-15T14:35:19.408806+0000","last_became_active":"2023-01-15T14:35:22.368797+0000","last_became_peered":"2023-01-15T14:35:22.368797+0000","last_unstale":"2023-01-15T14:35:22.369973+0000","last_undegraded":"2023-01-15T14:35:22.369973+0000","last_fullsized":"2023-01-15T14:35:22.065285+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":7,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,3],"avail_no_missing":["0","1","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.79","version":"423'435","reported_seq":1782,"reported_epoch":433,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:35:23.801546+0000","last_change":"2023-01-15T14:35:23.804780+0000","last_active":"2023-01-15T14:35:23.801546+0000","last_peered":"2023-01-15T14:35:23.801546+0000","last_clean":"2023-01-15T14:35:17.168373+0000","last_became_active":"2023-01-15T14:35:22.177129+0000","last_became_peered":"2023-01-15T14:35:22.177129+0000","last_unstale":"2023-01-15T14:35:23.801546+0000","last_undegraded":"2023-01-15T14:35:23.801264+0000","last_fullsized":"2023-01-15T14:35:21.970153+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":358,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17098873,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":35,"num_read_kb":57397,"num_write":12,"num_write_kb":6932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":13431307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[7,3],"avail_no_missing":["7","0","2"],"object_location_counts":[{"shards":"0,2,7","objects":14},{"shards":"3,7","objects":-7}],"blocked_by":[],"up_primary":4,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.78","version":"429'349","reported_seq":1796,"reported_epoch":432,"state":"activating+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:22.070866+0000","last_change":"2023-01-15T14:35:22.473580+0000","last_active":"2023-01-15T14:35:20.995053+0000","last_peered":"2023-01-15T14:35:19.691093+0000","last_clean":"2023-01-15T14:35:19.691093+0000","last_became_active":"2023-01-15T14:35:19.589996+0000","last_became_peered":"2023-01-15T14:35:19.589996+0000","last_unstale":"2023-01-15T14:35:22.070866+0000","last_undegraded":"2023-01-15T14:35:22.070866+0000","last_fullsized":"2023-01-15T14:35:22.065440+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":349,"log_dups_size":0,"ondisk_log_size":349,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4230717,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":52,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":23,"num_read_kb":21424,"num_write":20,"num_write_kb":6728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":16442531,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[2,0],"avail_no_missing":["2","3","6"],"object_location_counts":[{"shards":"0,2","objects":-3},{"shards":"2,3,6","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":2,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.77","version":"200'78","reported_seq":528,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.318247+0000","last_change":"2023-01-15T14:35:23.318247+0000","last_active":"2023-01-15T14:35:23.318247+0000","last_peered":"2023-01-15T14:35:23.318247+0000","last_clean":"2023-01-15T14:35:23.318247+0000","last_became_active":"2023-01-15T14:35:21.197411+0000","last_became_peered":"2023-01-15T14:35:21.197411+0000","last_unstale":"2023-01-15T14:35:23.318247+0000","last_undegraded":"2023-01-15T14:35:23.318247+0000","last_fullsized":"2023-01-15T14:35:23.318247+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.33512720600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.76","version":"194'81","reported_seq":13,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.964406+0000","last_change":"2023-01-15T14:35:21.403852+0000","last_active":"2023-01-15T14:35:21.964406+0000","last_peered":"2023-01-15T14:35:21.964406+0000","last_clean":"2023-01-15T14:35:21.964406+0000","last_became_active":"2023-01-15T14:35:21.302673+0000","last_became_peered":"2023-01-15T14:35:21.302673+0000","last_unstale":"2023-01-15T14:35:21.964406+0000","last_undegraded":"2023-01-15T14:35:21.964406+0000","last_fullsized":"2023-01-15T14:35:21.964406+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:34:51.296526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.296526+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.66934546500000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.31","version":"204'150","reported_seq":949,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.324028+0000","last_change":"2023-01-15T14:35:23.324810+0000","last_active":"2023-01-15T14:35:23.324028+0000","last_peered":"2023-01-15T14:35:23.324028+0000","last_clean":"2023-01-15T14:35:23.324028+0000","last_became_active":"2023-01-15T14:34:56.391718+0000","last_became_peered":"2023-01-15T14:34:56.391718+0000","last_unstale":"2023-01-15T14:35:23.324028+0000","last_undegraded":"2023-01-15T14:35:23.324028+0000","last_fullsized":"2023-01-15T14:35:23.324028+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:22.439","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.24377805599999999,"stat_sum":{"num_bytes":3558907,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10724,"num_write":3,"num_write_kb":3089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3605110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.30","version":"206'252","reported_seq":1186,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.964142+0000","last_change":"2023-01-15T14:35:21.964142+0000","last_active":"2023-01-15T14:35:21.964142+0000","last_peered":"2023-01-15T14:35:21.964142+0000","last_clean":"2023-01-15T14:35:21.964142+0000","last_became_active":"2023-01-15T14:34:56.539037+0000","last_became_peered":"2023-01-15T14:34:56.539037+0000","last_unstale":"2023-01-15T14:35:21.964142+0000","last_undegraded":"2023-01-15T14:35:21.964142+0000","last_fullsized":"2023-01-15T14:35:21.964142+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:35:03.572774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:03.572774+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:18.492","scrub_duration":0.36120358800000002,"objects_trimmed":0,"snaptrim_duration":0.10683906999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1316,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.319101+0000","last_change":"2023-01-15T14:35:22.319101+0000","last_active":"2023-01-15T14:35:22.319101+0000","last_peered":"2023-01-15T14:35:22.319101+0000","last_clean":"2023-01-15T14:35:22.319101+0000","last_became_active":"2023-01-15T14:35:07.161199+0000","last_became_peered":"2023-01-15T14:35:07.161199+0000","last_unstale":"2023-01-15T14:35:22.319101+0000","last_undegraded":"2023-01-15T14:35:22.319101+0000","last_fullsized":"2023-01-15T14:35:22.319101+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.346697861,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.2e","version":"139'38","reported_seq":65,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.525051+0000","last_change":"2023-01-15T14:35:21.525603+0000","last_active":"2023-01-15T14:35:21.525051+0000","last_peered":"2023-01-15T14:35:21.525051+0000","last_clean":"2023-01-15T14:35:21.525051+0000","last_became_active":"2023-01-15T14:35:04.286907+0000","last_became_peered":"2023-01-15T14:35:04.286907+0000","last_unstale":"2023-01-15T14:35:21.525051+0000","last_undegraded":"2023-01-15T14:35:21.525051+0000","last_fullsized":"2023-01-15T14:35:21.525051+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.322173074,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1422,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.315053+0000","last_change":"2023-01-15T14:35:22.315053+0000","last_active":"2023-01-15T14:35:22.315053+0000","last_peered":"2023-01-15T14:35:22.315053+0000","last_clean":"2023-01-15T14:35:22.315053+0000","last_became_active":"2023-01-15T14:35:07.164990+0000","last_became_peered":"2023-01-15T14:35:07.164990+0000","last_unstale":"2023-01-15T14:35:22.315053+0000","last_undegraded":"2023-01-15T14:35:22.315053+0000","last_fullsized":"2023-01-15T14:35:22.315053+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:00.115","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.20120526899999999,"stat_sum":{"num_bytes":1516832,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2709,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.2c","version":"141'46","reported_seq":57,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.523349+0000","last_change":"2023-01-15T14:35:21.524128+0000","last_active":"2023-01-15T14:35:21.523349+0000","last_peered":"2023-01-15T14:35:21.523349+0000","last_clean":"2023-01-15T14:35:21.523349+0000","last_became_active":"2023-01-15T14:35:04.386746+0000","last_became_peered":"2023-01-15T14:35:04.386746+0000","last_unstale":"2023-01-15T14:35:21.523349+0000","last_undegraded":"2023-01-15T14:35:21.523349+0000","last_fullsized":"2023-01-15T14:35:21.523349+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:58.535965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:58.535965+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.420846522,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289916,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":255,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.165914+0000","last_change":"2023-01-15T14:35:22.165914+0000","last_active":"2023-01-15T14:35:22.165914+0000","last_peered":"2023-01-15T14:35:22.165914+0000","last_clean":"2023-01-15T14:35:22.165914+0000","last_became_active":"2023-01-15T14:34:56.436012+0000","last_became_peered":"2023-01-15T14:34:56.436012+0000","last_unstale":"2023-01-15T14:35:22.165914+0000","last_undegraded":"2023-01-15T14:35:22.165914+0000","last_fullsized":"2023-01-15T14:35:22.165914+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.91144277900000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1450,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.699848+0000","last_change":"2023-01-15T14:35:23.700082+0000","last_active":"2023-01-15T14:35:23.699848+0000","last_peered":"2023-01-15T14:35:23.699848+0000","last_clean":"2023-01-15T14:35:23.699848+0000","last_became_active":"2023-01-15T14:35:07.358010+0000","last_became_peered":"2023-01-15T14:35:07.358010+0000","last_unstale":"2023-01-15T14:35:23.699848+0000","last_undegraded":"2023-01-15T14:35:23.699848+0000","last_fullsized":"2023-01-15T14:35:23.699848+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.71208893900000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.29","version":"110'149","reported_seq":714,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.089491+0000","last_change":"2023-01-15T14:35:23.089491+0000","last_active":"2023-01-15T14:35:23.089491+0000","last_peered":"2023-01-15T14:35:23.089491+0000","last_clean":"2023-01-15T14:35:23.089491+0000","last_became_active":"2023-01-15T14:35:11.345959+0000","last_became_peered":"2023-01-15T14:35:11.345959+0000","last_unstale":"2023-01-15T14:35:23.089491+0000","last_undegraded":"2023-01-15T14:35:23.089491+0000","last_fullsized":"2023-01-15T14:35:23.089491+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40398525099999999,"objects_trimmed":0,"snaptrim_duration":0.002897115,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.28","version":"333'248","reported_seq":1555,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.322227+0000","last_change":"2023-01-15T14:35:23.323025+0000","last_active":"2023-01-15T14:35:23.322227+0000","last_peered":"2023-01-15T14:35:23.322227+0000","last_clean":"2023-01-15T14:35:23.322227+0000","last_became_active":"2023-01-15T14:35:11.436994+0000","last_became_peered":"2023-01-15T14:35:11.436994+0000","last_unstale":"2023-01-15T14:35:23.322227+0000","last_undegraded":"2023-01-15T14:35:23.322227+0000","last_fullsized":"2023-01-15T14:35:23.322227+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'248","last_scrub_stamp":"2023-01-15T14:34:20.620068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.620068+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:32.103","scrub_duration":0.40448800800000001,"objects_trimmed":0,"snaptrim_duration":0.24299388,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":667,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.091583+0000","last_change":"2023-01-15T14:35:23.091583+0000","last_active":"2023-01-15T14:35:23.091583+0000","last_peered":"2023-01-15T14:35:23.091583+0000","last_clean":"2023-01-15T14:35:23.091583+0000","last_became_active":"2023-01-15T14:35:11.451119+0000","last_became_peered":"2023-01-15T14:35:11.451119+0000","last_unstale":"2023-01-15T14:35:23.091583+0000","last_undegraded":"2023-01-15T14:35:23.091583+0000","last_fullsized":"2023-01-15T14:35:23.091583+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.0020738430000000001,"stat_sum":{"num_bytes":1242779,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":4228,"num_write":1,"num_write_kb":368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.26","version":"102'52","reported_seq":65,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.519056+0000","last_change":"2023-01-15T14:35:21.519791+0000","last_active":"2023-01-15T14:35:21.519056+0000","last_peered":"2023-01-15T14:35:21.519056+0000","last_clean":"2023-01-15T14:35:21.519056+0000","last_became_active":"2023-01-15T14:35:04.285753+0000","last_became_peered":"2023-01-15T14:35:04.285753+0000","last_unstale":"2023-01-15T14:35:21.519056+0000","last_undegraded":"2023-01-15T14:35:21.519056+0000","last_fullsized":"2023-01-15T14:35:21.519056+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.41704533599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.25","version":"333'108","reported_seq":1192,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.324898+0000","last_change":"2023-01-15T14:35:23.325374+0000","last_active":"2023-01-15T14:35:23.324898+0000","last_peered":"2023-01-15T14:35:23.324898+0000","last_clean":"2023-01-15T14:35:23.324898+0000","last_became_active":"2023-01-15T14:34:46.346453+0000","last_became_peered":"2023-01-15T14:34:46.346453+0000","last_unstale":"2023-01-15T14:35:23.324898+0000","last_undegraded":"2023-01-15T14:35:23.324898+0000","last_fullsized":"2023-01-15T14:35:23.324898+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":396,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'108","last_scrub_stamp":"2023-01-15T14:35:02.975751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:02.975751+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:28.471","scrub_duration":0.30430094200000002,"objects_trimmed":0,"snaptrim_duration":0.24224642199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.24","version":"432'293","reported_seq":1632,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.191313+0000","last_change":"2023-01-15T14:35:23.191313+0000","last_active":"2023-01-15T14:35:23.191313+0000","last_peered":"2023-01-15T14:35:23.191313+0000","last_clean":"2023-01-15T14:35:23.191313+0000","last_became_active":"2023-01-15T14:35:11.349162+0000","last_became_peered":"2023-01-15T14:35:11.349162+0000","last_unstale":"2023-01-15T14:35:23.191313+0000","last_undegraded":"2023-01-15T14:35:23.191313+0000","last_fullsized":"2023-01-15T14:35:23.191313+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:24.556","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.003152859,"stat_sum":{"num_bytes":6590865,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":20,"num_read_kb":22662,"num_write":11,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.23","version":"149'233","reported_seq":928,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.165439+0000","last_change":"2023-01-15T14:35:22.165439+0000","last_active":"2023-01-15T14:35:22.165439+0000","last_peered":"2023-01-15T14:35:22.165439+0000","last_clean":"2023-01-15T14:35:22.165439+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:35:22.165439+0000","last_undegraded":"2023-01-15T14:35:22.165439+0000","last_fullsized":"2023-01-15T14:35:22.165439+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:34:55.674244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:55.674244+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:18.674","scrub_duration":0.50374311999999999,"objects_trimmed":0,"snaptrim_duration":0.908397024,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.22","version":"323'161","reported_seq":1338,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.964743+0000","last_change":"2023-01-15T14:35:21.964743+0000","last_active":"2023-01-15T14:35:21.964743+0000","last_peered":"2023-01-15T14:35:21.964743+0000","last_clean":"2023-01-15T14:35:21.964743+0000","last_became_active":"2023-01-15T14:35:11.447340+0000","last_became_peered":"2023-01-15T14:35:11.447340+0000","last_unstale":"2023-01-15T14:35:21.964743+0000","last_undegraded":"2023-01-15T14:35:21.964743+0000","last_fullsized":"2023-01-15T14:35:21.964743+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.20220659099999999,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.21","version":"321'138","reported_seq":1025,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.407261+0000","last_change":"2023-01-15T14:35:21.408099+0000","last_active":"2023-01-15T14:35:21.407261+0000","last_peered":"2023-01-15T14:35:21.407261+0000","last_clean":"2023-01-15T14:35:21.407261+0000","last_became_active":"2023-01-15T14:35:07.146982+0000","last_became_peered":"2023-01-15T14:35:07.146982+0000","last_unstale":"2023-01-15T14:35:21.407261+0000","last_undegraded":"2023-01-15T14:35:21.407261+0000","last_fullsized":"2023-01-15T14:35:21.407261+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"321'138","last_scrub_stamp":"2023-01-15T14:34:49.495975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.495975+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.30700908500000001,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6059,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.20","version":"324'273","reported_seq":1558,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.304241+0000","last_change":"2023-01-15T14:35:21.305069+0000","last_active":"2023-01-15T14:35:21.304241+0000","last_peered":"2023-01-15T14:35:21.304241+0000","last_clean":"2023-01-15T14:35:21.304241+0000","last_became_active":"2023-01-15T14:35:07.459590+0000","last_became_peered":"2023-01-15T14:35:07.459590+0000","last_unstale":"2023-01-15T14:35:21.304241+0000","last_undegraded":"2023-01-15T14:35:21.304241+0000","last_fullsized":"2023-01-15T14:35:21.304241+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"324'273","last_scrub_stamp":"2023-01-15T14:34:49.100502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.100502+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:51.047","scrub_duration":0.204192292,"objects_trimmed":0,"snaptrim_duration":0.214046453,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.1f","version":"429'291","reported_seq":1608,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.191658+0000","last_change":"2023-01-15T14:35:21.191644+0000","last_active":"2023-01-15T14:35:21.191658+0000","last_peered":"2023-01-15T14:35:21.191658+0000","last_clean":"2023-01-15T14:35:21.191658+0000","last_became_active":"2023-01-15T14:35:14.276949+0000","last_became_peered":"2023-01-15T14:35:14.276949+0000","last_unstale":"2023-01-15T14:35:21.191658+0000","last_undegraded":"2023-01-15T14:35:21.191658+0000","last_fullsized":"2023-01-15T14:35:21.191658+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:59.076","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.91883148800000003,"stat_sum":{"num_bytes":10741917,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":43103,"num_write":19,"num_write_kb":6760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":17439983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.1e","version":"215'39","reported_seq":1175,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.412975+0000","last_change":"2023-01-15T14:35:21.413483+0000","last_active":"2023-01-15T14:35:21.412975+0000","last_peered":"2023-01-15T14:35:21.412975+0000","last_clean":"2023-01-15T14:35:21.412975+0000","last_became_active":"2023-01-15T14:35:11.243676+0000","last_became_peered":"2023-01-15T14:35:11.243676+0000","last_unstale":"2023-01-15T14:35:21.412975+0000","last_undegraded":"2023-01-15T14:35:21.412975+0000","last_fullsized":"2023-01-15T14:35:21.412975+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.31138659800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1524,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.192351+0000","last_change":"2023-01-15T14:35:23.192351+0000","last_active":"2023-01-15T14:35:23.192351+0000","last_peered":"2023-01-15T14:35:23.192351+0000","last_clean":"2023-01-15T14:35:23.192351+0000","last_became_active":"2023-01-15T14:35:07.268393+0000","last_became_peered":"2023-01-15T14:35:07.268393+0000","last_unstale":"2023-01-15T14:35:23.192351+0000","last_undegraded":"2023-01-15T14:35:23.192351+0000","last_fullsized":"2023-01-15T14:35:23.192351+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":5,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:51.945865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.945865+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:04.107","scrub_duration":0.30386408399999998,"objects_trimmed":0,"snaptrim_duration":0.102279061,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":778,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.289579+0000","last_change":"2023-01-15T14:35:23.289579+0000","last_active":"2023-01-15T14:35:23.289579+0000","last_peered":"2023-01-15T14:35:23.289579+0000","last_clean":"2023-01-15T14:35:23.289579+0000","last_became_active":"2023-01-15T14:35:04.384767+0000","last_became_peered":"2023-01-15T14:35:04.384767+0000","last_unstale":"2023-01-15T14:35:23.289579+0000","last_undegraded":"2023-01-15T14:35:23.289579+0000","last_fullsized":"2023-01-15T14:35:23.289579+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":5,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:22.440826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:22.440826+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:25.536","scrub_duration":0.30343327599999997,"objects_trimmed":0,"snaptrim_duration":0.001202217,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.1b","version":"431'139","reported_seq":291,"reported_epoch":432,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:22.166654+0000","last_change":"2023-01-15T14:35:21.906868+0000","last_active":"2023-01-15T14:35:22.166654+0000","last_peered":"2023-01-15T14:35:22.166654+0000","last_clean":"2023-01-15T14:35:22.166654+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:35:22.166654+0000","last_undegraded":"2023-01-15T14:35:22.166654+0000","last_fullsized":"2023-01-15T14:35:22.166654+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"370'101","last_scrub_stamp":"2023-01-15T14:34:24.137270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:24.137270+0000","objects_scrubbed":3,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:44.214","scrub_duration":0.106791863,"objects_trimmed":0,"snaptrim_duration":0.31658353900000002,"stat_sum":{"num_bytes":10778275,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":13300,"num_write":8,"num_write_kb":4287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.1a","version":"431'372","reported_seq":1694,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.088836+0000","last_change":"2023-01-15T14:35:23.088836+0000","last_active":"2023-01-15T14:35:23.088836+0000","last_peered":"2023-01-15T14:35:23.088836+0000","last_clean":"2023-01-15T14:35:23.088836+0000","last_became_active":"2023-01-15T14:34:44.273541+0000","last_became_peered":"2023-01-15T14:34:44.273541+0000","last_unstale":"2023-01-15T14:35:23.088836+0000","last_undegraded":"2023-01-15T14:35:23.088836+0000","last_fullsized":"2023-01-15T14:35:23.088836+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"375'356","last_scrub_stamp":"2023-01-15T14:34:29.460843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:29.460843+0000","objects_scrubbed":4,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:56.905","scrub_duration":0.30741503599999997,"objects_trimmed":0,"snaptrim_duration":0.0032257079999999999,"stat_sum":{"num_bytes":7752426,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.19","version":"432'272","reported_seq":1527,"reported_epoch":433,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:23.181457+0000","last_change":"2023-01-15T14:35:22.975342+0000","last_active":"2023-01-15T14:35:23.181457+0000","last_peered":"2023-01-15T14:35:23.181457+0000","last_clean":"2023-01-15T14:35:23.181457+0000","last_became_active":"2023-01-15T14:34:43.263173+0000","last_became_peered":"2023-01-15T14:34:43.263173+0000","last_unstale":"2023-01-15T14:35:23.181457+0000","last_undegraded":"2023-01-15T14:35:23.181457+0000","last_fullsized":"2023-01-15T14:35:23.181457+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":272,"log_dups_size":0,"ondisk_log_size":272,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.617038313,"stat_sum":{"num_bytes":6864537,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":12320,"num_write":9,"num_write_kb":2852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4941423,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.18","version":"432'352","reported_seq":1783,"reported_epoch":432,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:22.403982+0000","last_change":"2023-01-15T14:35:22.403982+0000","last_active":"2023-01-15T14:35:22.403982+0000","last_peered":"2023-01-15T14:35:22.403982+0000","last_clean":"2023-01-15T14:35:22.403982+0000","last_became_active":"2023-01-15T14:35:11.544123+0000","last_became_peered":"2023-01-15T14:35:11.544123+0000","last_unstale":"2023-01-15T14:35:22.403982+0000","last_undegraded":"2023-01-15T14:35:22.403982+0000","last_fullsized":"2023-01-15T14:35:22.403982+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":5,"last_scrub":"384'341","last_scrub_stamp":"2023-01-15T14:34:35.477211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:35.477211+0000","objects_scrubbed":5,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:54.983","scrub_duration":0.54215560600000001,"objects_trimmed":0,"snaptrim_duration":0.40728094100000001,"stat_sum":{"num_bytes":7186347,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.17","version":"278'93","reported_seq":244,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.093568+0000","last_change":"2023-01-15T14:35:21.093528+0000","last_active":"2023-01-15T14:35:21.093568+0000","last_peered":"2023-01-15T14:35:21.093568+0000","last_clean":"2023-01-15T14:35:21.093568+0000","last_became_active":"2023-01-15T14:35:04.289453+0000","last_became_peered":"2023-01-15T14:35:04.289453+0000","last_unstale":"2023-01-15T14:35:21.093568+0000","last_undegraded":"2023-01-15T14:35:21.093568+0000","last_fullsized":"2023-01-15T14:35:21.093568+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:29.773","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.31436613899999999,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.16","version":"421'151","reported_seq":940,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.987965+0000","last_change":"2023-01-15T14:35:22.987965+0000","last_active":"2023-01-15T14:35:22.987965+0000","last_peered":"2023-01-15T14:35:22.987965+0000","last_clean":"2023-01-15T14:35:22.987965+0000","last_became_active":"2023-01-15T14:34:44.491303+0000","last_became_peered":"2023-01-15T14:34:44.491303+0000","last_unstale":"2023-01-15T14:35:22.987965+0000","last_undegraded":"2023-01-15T14:35:22.987965+0000","last_fullsized":"2023-01-15T14:35:22.987965+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.909596603,"stat_sum":{"num_bytes":7378471,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7,"num_read_kb":6898,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9731517,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":208,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.698855+0000","last_change":"2023-01-15T14:35:23.699067+0000","last_active":"2023-01-15T14:35:23.698855+0000","last_peered":"2023-01-15T14:35:23.698855+0000","last_clean":"2023-01-15T14:35:23.698855+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:35:23.698855+0000","last_undegraded":"2023-01-15T14:35:23.698855+0000","last_fullsized":"2023-01-15T14:35:23.698855+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.67652493700000005,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.14","version":"428'175","reported_seq":1436,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.193392+0000","last_change":"2023-01-15T14:35:23.193392+0000","last_active":"2023-01-15T14:35:23.193392+0000","last_peered":"2023-01-15T14:35:23.193392+0000","last_clean":"2023-01-15T14:35:23.193392+0000","last_became_active":"2023-01-15T14:35:14.050115+0000","last_became_peered":"2023-01-15T14:35:14.050115+0000","last_unstale":"2023-01-15T14:35:23.193392+0000","last_undegraded":"2023-01-15T14:35:23.193392+0000","last_fullsized":"2023-01-15T14:35:23.193392+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"351'159","last_scrub_stamp":"2023-01-15T14:34:18.607751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:18.607751+0000","objects_scrubbed":2,"log_size":175,"log_dups_size":0,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:42.578","scrub_duration":0.40666956999999998,"objects_trimmed":0,"snaptrim_duration":0.0047093949999999999,"stat_sum":{"num_bytes":2453855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2453855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":28,"reported_epoch":430,"state":"active+clean","last_fresh":"2023-01-15T14:35:19.982780+0000","last_change":"2023-01-15T14:35:04.185970+0000","last_active":"2023-01-15T14:35:19.982780+0000","last_peered":"2023-01-15T14:35:19.982780+0000","last_clean":"2023-01-15T14:35:19.982780+0000","last_became_active":"2023-01-15T14:35:04.185021+0000","last_became_peered":"2023-01-15T14:35:04.185021+0000","last_unstale":"2023-01-15T14:35:19.982780+0000","last_undegraded":"2023-01-15T14:35:19.982780+0000","last_fullsized":"2023-01-15T14:35:19.982780+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":231,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.090005+0000","last_change":"2023-01-15T14:35:21.089948+0000","last_active":"2023-01-15T14:35:21.090005+0000","last_peered":"2023-01-15T14:35:21.090005+0000","last_clean":"2023-01-15T14:35:21.090005+0000","last_became_active":"2023-01-15T14:34:43.461395+0000","last_became_peered":"2023-01-15T14:34:43.461395+0000","last_unstale":"2023-01-15T14:35:21.090005+0000","last_undegraded":"2023-01-15T14:35:21.090005+0000","last_fullsized":"2023-01-15T14:35:21.090005+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.103411955,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"2.6","version":"0'0","reported_seq":61,"reported_epoch":432,"state":"active+clean","last_fresh":"2023-01-15T14:35:22.169369+0000","last_change":"2023-01-15T14:35:04.384333+0000","last_active":"2023-01-15T14:35:22.169369+0000","last_peered":"2023-01-15T14:35:22.169369+0000","last_clean":"2023-01-15T14:35:22.169369+0000","last_became_active":"2023-01-15T14:35:04.383307+0000","last_became_peered":"2023-01-15T14:35:04.383307+0000","last_unstale":"2023-01-15T14:35:22.169369+0000","last_undegraded":"2023-01-15T14:35:22.169369+0000","last_fullsized":"2023-01-15T14:35:22.169369+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"431'180","reported_seq":1372,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.310770+0000","last_change":"2023-01-15T14:35:22.310770+0000","last_active":"2023-01-15T14:35:22.310770+0000","last_peered":"2023-01-15T14:35:22.310770+0000","last_clean":"2023-01-15T14:35:22.310770+0000","last_became_active":"2023-01-15T14:34:17.186013+0000","last_became_peered":"2023-01-15T14:34:17.186013+0000","last_unstale":"2023-01-15T14:35:22.310770+0000","last_undegraded":"2023-01-15T14:35:22.310770+0000","last_fullsized":"2023-01-15T14:35:22.310770+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.33909664,"stat_sum":{"num_bytes":14900141,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":55306,"num_write":22,"num_write_kb":10279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":3546343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":481,"reported_epoch":432,"state":"active+clean","last_fresh":"2023-01-15T14:35:22.314132+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:35:22.314132+0000","last_peered":"2023-01-15T14:35:22.314132+0000","last_clean":"2023-01-15T14:35:22.314132+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:35:22.314132+0000","last_undegraded":"2023-01-15T14:35:22.314132+0000","last_fullsized":"2023-01-15T14:35:22.314132+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"433'361","reported_seq":1740,"reported_epoch":433,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:23.325467+0000","last_change":"2023-01-15T14:35:23.325467+0000","last_active":"2023-01-15T14:35:23.325467+0000","last_peered":"2023-01-15T14:35:23.325467+0000","last_clean":"2023-01-15T14:35:23.325467+0000","last_became_active":"2023-01-15T14:34:44.164446+0000","last_became_peered":"2023-01-15T14:34:44.164446+0000","last_unstale":"2023-01-15T14:35:23.325467+0000","last_undegraded":"2023-01-15T14:35:23.325467+0000","last_fullsized":"2023-01-15T14:35:23.325467+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":361,"log_dups_size":0,"ondisk_log_size":361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:32.407","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0.23003705799999999,"stat_sum":{"num_bytes":11114185,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":11,"num_read_kb":9789,"num_write":5,"num_write_kb":3775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":51,"reported_epoch":432,"state":"active+clean","last_fresh":"2023-01-15T14:35:22.005596+0000","last_change":"2023-01-15T14:34:43.371445+0000","last_active":"2023-01-15T14:35:22.005596+0000","last_peered":"2023-01-15T14:35:22.005596+0000","last_clean":"2023-01-15T14:35:22.005596+0000","last_became_active":"2023-01-15T14:34:43.253700+0000","last_became_peered":"2023-01-15T14:34:43.253700+0000","last_unstale":"2023-01-15T14:35:22.005596+0000","last_undegraded":"2023-01-15T14:35:22.005596+0000","last_fullsized":"2023-01-15T14:35:22.005596+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"425'193","reported_seq":1473,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.413589+0000","last_change":"2023-01-15T14:35:21.514412+0000","last_active":"2023-01-15T14:35:21.413589+0000","last_peered":"2023-01-15T14:35:21.413589+0000","last_clean":"2023-01-15T14:35:21.413589+0000","last_became_active":"2023-01-15T14:35:12.083710+0000","last_became_peered":"2023-01-15T14:35:12.083710+0000","last_unstale":"2023-01-15T14:35:21.413589+0000","last_undegraded":"2023-01-15T14:35:21.413589+0000","last_fullsized":"2023-01-15T14:35:21.413589+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.41218985000000002,"stat_sum":{"num_bytes":12115268,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":25,"num_read_kb":26148,"num_write":22,"num_write_kb":7931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":26564309,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.3","version":"426'345","reported_seq":1719,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.090878+0000","last_change":"2023-01-15T14:35:23.090878+0000","last_active":"2023-01-15T14:35:23.090878+0000","last_peered":"2023-01-15T14:35:23.090878+0000","last_clean":"2023-01-15T14:35:23.090878+0000","last_became_active":"2023-01-15T14:35:16.368672+0000","last_became_peered":"2023-01-15T14:35:16.368672+0000","last_unstale":"2023-01-15T14:35:23.090878+0000","last_undegraded":"2023-01-15T14:35:23.090878+0000","last_fullsized":"2023-01-15T14:35:23.090878+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.0034801210000000001,"stat_sum":{"num_bytes":5210335,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":13875,"num_write":6,"num_write_kb":7330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":29816344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":61,"reported_epoch":432,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.967880+0000","last_change":"2023-01-15T14:35:04.086793+0000","last_active":"2023-01-15T14:35:21.967880+0000","last_peered":"2023-01-15T14:35:21.967880+0000","last_clean":"2023-01-15T14:35:21.967880+0000","last_became_active":"2023-01-15T14:35:04.086427+0000","last_became_peered":"2023-01-15T14:35:04.086427+0000","last_unstale":"2023-01-15T14:35:21.967880+0000","last_undegraded":"2023-01-15T14:35:21.967880+0000","last_fullsized":"2023-01-15T14:35:21.967880+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1145,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.319097+0000","last_change":"2023-01-15T14:35:23.319182+0000","last_active":"2023-01-15T14:35:23.319097+0000","last_peered":"2023-01-15T14:35:23.319097+0000","last_clean":"2023-01-15T14:35:23.319097+0000","last_became_active":"2023-01-15T14:35:07.358411+0000","last_became_peered":"2023-01-15T14:35:07.358411+0000","last_unstale":"2023-01-15T14:35:23.319097+0000","last_undegraded":"2023-01-15T14:35:23.319097+0000","last_fullsized":"2023-01-15T14:35:23.319097+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:35:01.091016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:01.091016+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:22.866","scrub_duration":0.55784813600000005,"objects_trimmed":0,"snaptrim_duration":0.13850547199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":29,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.098964+0000","last_change":"2023-01-15T14:35:11.444424+0000","last_active":"2023-01-15T14:35:21.098964+0000","last_peered":"2023-01-15T14:35:21.098964+0000","last_clean":"2023-01-15T14:35:21.098964+0000","last_became_active":"2023-01-15T14:35:11.443876+0000","last_became_peered":"2023-01-15T14:35:11.443876+0000","last_unstale":"2023-01-15T14:35:21.098964+0000","last_undegraded":"2023-01-15T14:35:21.098964+0000","last_fullsized":"2023-01-15T14:35:21.098964+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":891,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.064672+0000","last_change":"2023-01-15T14:35:22.064672+0000","last_active":"2023-01-15T14:35:22.064672+0000","last_peered":"2023-01-15T14:35:22.064672+0000","last_clean":"2023-01-15T14:35:22.064672+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:35:22.064672+0000","last_undegraded":"2023-01-15T14:35:22.064672+0000","last_fullsized":"2023-01-15T14:35:22.064672+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.909641586,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":499,"reported_epoch":432,"state":"active+clean","last_fresh":"2023-01-15T14:35:22.170262+0000","last_change":"2023-01-15T14:34:43.269231+0000","last_active":"2023-01-15T14:35:22.170262+0000","last_peered":"2023-01-15T14:35:22.170262+0000","last_clean":"2023-01-15T14:35:22.170262+0000","last_became_active":"2023-01-15T14:34:43.268688+0000","last_became_peered":"2023-01-15T14:34:43.268688+0000","last_unstale":"2023-01-15T14:35:22.170262+0000","last_undegraded":"2023-01-15T14:35:22.170262+0000","last_fullsized":"2023-01-15T14:35:22.170262+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":1168,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.518240+0000","last_change":"2023-01-15T14:35:21.518881+0000","last_active":"2023-01-15T14:35:21.518240+0000","last_peered":"2023-01-15T14:35:21.518240+0000","last_clean":"2023-01-15T14:35:21.518240+0000","last_became_active":"2023-01-15T14:35:07.458021+0000","last_became_peered":"2023-01-15T14:35:07.458021+0000","last_unstale":"2023-01-15T14:35:21.518240+0000","last_undegraded":"2023-01-15T14:35:21.518240+0000","last_fullsized":"2023-01-15T14:35:21.518240+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.41618196800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":96,"reported_epoch":432,"state":"active+clean","last_fresh":"2023-01-15T14:35:22.071483+0000","last_change":"2023-01-15T14:33:57.015034+0000","last_active":"2023-01-15T14:35:22.071483+0000","last_peered":"2023-01-15T14:35:22.071483+0000","last_clean":"2023-01-15T14:35:22.071483+0000","last_became_active":"2023-01-15T14:33:57.014857+0000","last_became_peered":"2023-01-15T14:33:57.014857+0000","last_unstale":"2023-01-15T14:35:22.071483+0000","last_undegraded":"2023-01-15T14:35:22.071483+0000","last_fullsized":"2023-01-15T14:35:22.071483+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":621,"reported_epoch":433,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:35:23.821678+0000","last_change":"2023-01-15T14:35:23.716539+0000","last_active":"2023-01-15T14:35:23.821678+0000","last_peered":"2023-01-15T14:35:23.821678+0000","last_clean":"2023-01-15T14:35:02.009600+0000","last_became_active":"2023-01-15T14:35:05.374546+0000","last_became_peered":"2023-01-15T14:35:05.374546+0000","last_unstale":"2023-01-15T14:35:23.821678+0000","last_undegraded":"2023-01-15T14:35:23.821678+0000","last_fullsized":"2023-01-15T14:35:04.978006+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0],"avail_no_missing":["7","0","1","6"],"object_location_counts":[{"shards":"0,7","objects":2}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"179'204","reported_seq":1490,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.212888+0000","last_change":"2023-01-15T14:35:23.213014+0000","last_active":"2023-01-15T14:35:23.212888+0000","last_peered":"2023-01-15T14:35:23.212888+0000","last_clean":"2023-01-15T14:35:23.212888+0000","last_became_active":"2023-01-15T14:35:11.437153+0000","last_became_peered":"2023-01-15T14:35:11.437153+0000","last_unstale":"2023-01-15T14:35:23.212888+0000","last_undegraded":"2023-01-15T14:35:23.212888+0000","last_fullsized":"2023-01-15T14:35:23.212888+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:34:48.269667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.269667+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.13301798000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":1300,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.090887+0000","last_change":"2023-01-15T14:35:23.090887+0000","last_active":"2023-01-15T14:35:23.090887+0000","last_peered":"2023-01-15T14:35:23.090887+0000","last_clean":"2023-01-15T14:35:23.090887+0000","last_became_active":"2023-01-15T14:32:22.140909+0000","last_became_peered":"2023-01-15T14:32:22.140909+0000","last_unstale":"2023-01-15T14:35:23.090887+0000","last_undegraded":"2023-01-15T14:35:23.090887+0000","last_fullsized":"2023-01-15T14:35:23.090887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.003424996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.a","version":"252'279","reported_seq":1495,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.516325+0000","last_change":"2023-01-15T14:35:21.517104+0000","last_active":"2023-01-15T14:35:21.516325+0000","last_peered":"2023-01-15T14:35:21.516325+0000","last_clean":"2023-01-15T14:35:21.516325+0000","last_became_active":"2023-01-15T14:35:07.461643+0000","last_became_peered":"2023-01-15T14:35:07.461643+0000","last_unstale":"2023-01-15T14:35:21.516325+0000","last_undegraded":"2023-01-15T14:35:21.516325+0000","last_fullsized":"2023-01-15T14:35:21.516325+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.41465976500000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":1015,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.210101+0000","last_change":"2023-01-15T14:35:22.210101+0000","last_active":"2023-01-15T14:35:22.210101+0000","last_peered":"2023-01-15T14:35:22.210101+0000","last_clean":"2023-01-15T14:35:22.210101+0000","last_became_active":"2023-01-15T14:35:11.447676+0000","last_became_peered":"2023-01-15T14:35:11.447676+0000","last_unstale":"2023-01-15T14:35:22.210101+0000","last_undegraded":"2023-01-15T14:35:22.210101+0000","last_fullsized":"2023-01-15T14:35:22.210101+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.90710911599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.c","version":"428'132","reported_seq":1442,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.102944+0000","last_change":"2023-01-15T14:35:21.304114+0000","last_active":"2023-01-15T14:35:21.102944+0000","last_peered":"2023-01-15T14:35:21.102944+0000","last_clean":"2023-01-15T14:35:21.102944+0000","last_became_active":"2023-01-15T14:35:11.346741+0000","last_became_peered":"2023-01-15T14:35:11.346741+0000","last_unstale":"2023-01-15T14:35:21.102944+0000","last_undegraded":"2023-01-15T14:35:21.102944+0000","last_fullsized":"2023-01-15T14:35:21.102944+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.30177256800000002,"stat_sum":{"num_bytes":3264587,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":43,"num_read_kb":67246,"num_write":12,"num_write_kb":1706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":29077297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.d","version":"296'232","reported_seq":1502,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.315967+0000","last_change":"2023-01-15T14:35:22.315967+0000","last_active":"2023-01-15T14:35:22.315967+0000","last_peered":"2023-01-15T14:35:22.315967+0000","last_clean":"2023-01-15T14:35:22.315967+0000","last_became_active":"2023-01-15T14:35:07.267234+0000","last_became_peered":"2023-01-15T14:35:07.267234+0000","last_unstale":"2023-01-15T14:35:22.315967+0000","last_undegraded":"2023-01-15T14:35:22.315967+0000","last_fullsized":"2023-01-15T14:35:22.315967+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.202583139,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.e","version":"139'38","reported_seq":233,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.988159+0000","last_change":"2023-01-15T14:35:22.988159+0000","last_active":"2023-01-15T14:35:22.988159+0000","last_peered":"2023-01-15T14:35:22.988159+0000","last_clean":"2023-01-15T14:35:22.988159+0000","last_became_active":"2023-01-15T14:34:43.265641+0000","last_became_peered":"2023-01-15T14:34:43.265641+0000","last_unstale":"2023-01-15T14:35:22.988159+0000","last_undegraded":"2023-01-15T14:35:22.988159+0000","last_fullsized":"2023-01-15T14:35:22.988159+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.871062173,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1419,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.988154+0000","last_change":"2023-01-15T14:35:22.988154+0000","last_active":"2023-01-15T14:35:22.988154+0000","last_peered":"2023-01-15T14:35:22.988154+0000","last_clean":"2023-01-15T14:35:22.988154+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:35:22.988154+0000","last_undegraded":"2023-01-15T14:35:22.988154+0000","last_fullsized":"2023-01-15T14:35:22.988154+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":1.014484594,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.10","version":"409'420","reported_seq":1575,"reported_epoch":431,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:21.098501+0000","last_change":"2023-01-15T14:35:05.479074+0000","last_active":"2023-01-15T14:35:21.098501+0000","last_peered":"2023-01-15T14:35:21.098501+0000","last_clean":"2023-01-15T14:35:02.307833+0000","last_became_active":"2023-01-15T14:35:05.377956+0000","last_became_peered":"2023-01-15T14:35:05.377956+0000","last_unstale":"2023-01-15T14:35:21.098501+0000","last_undegraded":"2023-01-15T14:35:05.071354+0000","last_fullsized":"2023-01-15T14:35:05.070163+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.361291529,"stat_sum":{"num_bytes":8658713,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":63,"num_read_kb":63381,"num_write":50,"num_write_kb":18045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12804882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[0,3],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2,3","objects":3},{"shards":"0,3","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.11","version":"432'367","reported_seq":1890,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.025672+0000","last_change":"2023-01-15T14:35:23.025744+0000","last_active":"2023-01-15T14:35:23.025672+0000","last_peered":"2023-01-15T14:35:23.025672+0000","last_clean":"2023-01-15T14:35:23.025672+0000","last_became_active":"2023-01-15T14:35:13.513113+0000","last_became_peered":"2023-01-15T14:35:13.513113+0000","last_unstale":"2023-01-15T14:35:23.025672+0000","last_undegraded":"2023-01-15T14:35:23.025672+0000","last_fullsized":"2023-01-15T14:35:23.025672+0000","mapping_epoch":422,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.038749324000000002,"stat_sum":{"num_bytes":23281289,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":49,"num_read_kb":48290,"num_write":37,"num_write_kb":24921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":44867083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.12","version":"431'213","reported_seq":1107,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.165961+0000","last_change":"2023-01-15T14:35:22.165961+0000","last_active":"2023-01-15T14:35:22.165961+0000","last_peered":"2023-01-15T14:35:22.165961+0000","last_clean":"2023-01-15T14:35:22.165961+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:35:22.165961+0000","last_undegraded":"2023-01-15T14:35:22.165961+0000","last_fullsized":"2023-01-15T14:35:22.165961+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:58.076","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.908901717,"stat_sum":{"num_bytes":12674183,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":21,"num_write_kb":11718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.13","version":"433'406","reported_seq":1619,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.697684+0000","last_change":"2023-01-15T14:35:23.697792+0000","last_active":"2023-01-15T14:35:23.697684+0000","last_peered":"2023-01-15T14:35:23.697684+0000","last_clean":"2023-01-15T14:35:23.697684+0000","last_became_active":"2023-01-15T14:33:20.230538+0000","last_became_peered":"2023-01-15T14:33:20.230538+0000","last_unstale":"2023-01-15T14:35:23.697684+0000","last_undegraded":"2023-01-15T14:35:23.697684+0000","last_fullsized":"2023-01-15T14:35:23.697684+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":311,"parent":"0.0","parent_split_bits":5,"last_scrub":"374'376","last_scrub_stamp":"2023-01-15T14:34:32.489483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.489483+0000","objects_scrubbed":4,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:32.530","scrub_duration":0.381937625,"objects_trimmed":0,"snaptrim_duration":0.609234477,"stat_sum":{"num_bytes":13721904,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":29952,"num_write":3,"num_write_kb":2561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.32","version":"431'192","reported_seq":1082,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.066157+0000","last_change":"2023-01-15T14:35:22.066157+0000","last_active":"2023-01-15T14:35:22.066157+0000","last_peered":"2023-01-15T14:35:22.066157+0000","last_clean":"2023-01-15T14:35:22.066157+0000","last_became_active":"2023-01-15T14:34:56.393418+0000","last_became_peered":"2023-01-15T14:34:56.393418+0000","last_unstale":"2023-01-15T14:35:22.066157+0000","last_undegraded":"2023-01-15T14:35:22.066157+0000","last_fullsized":"2023-01-15T14:35:22.066157+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.80638554500000004,"stat_sum":{"num_bytes":6427730,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3525,"num_write":7,"num_write_kb":2712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2663942,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.33","version":"204'275","reported_seq":1461,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.410112+0000","last_change":"2023-01-15T14:35:21.411082+0000","last_active":"2023-01-15T14:35:21.410112+0000","last_peered":"2023-01-15T14:35:21.410112+0000","last_clean":"2023-01-15T14:35:21.410112+0000","last_became_active":"2023-01-15T14:35:11.352099+0000","last_became_peered":"2023-01-15T14:35:11.352099+0000","last_unstale":"2023-01-15T14:35:21.410112+0000","last_undegraded":"2023-01-15T14:35:21.410112+0000","last_fullsized":"2023-01-15T14:35:21.410112+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.30934613100000002,"stat_sum":{"num_bytes":3485477,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11189,"num_write":7,"num_write_kb":3285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":1375,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.189773+0000","last_change":"2023-01-15T14:35:23.189773+0000","last_active":"2023-01-15T14:35:23.189773+0000","last_peered":"2023-01-15T14:35:23.189773+0000","last_clean":"2023-01-15T14:35:23.189773+0000","last_became_active":"2023-01-15T14:35:19.392381+0000","last_became_peered":"2023-01-15T14:35:19.392381+0000","last_unstale":"2023-01-15T14:35:23.189773+0000","last_undegraded":"2023-01-15T14:35:23.189773+0000","last_fullsized":"2023-01-15T14:35:23.189773+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.0024531750000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.35","version":"110'55","reported_seq":67,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.514541+0000","last_change":"2023-01-15T14:35:21.515320+0000","last_active":"2023-01-15T14:35:21.514541+0000","last_peered":"2023-01-15T14:35:21.514541+0000","last_clean":"2023-01-15T14:35:21.514541+0000","last_became_active":"2023-01-15T14:35:19.387459+0000","last_became_peered":"2023-01-15T14:35:19.387459+0000","last_unstale":"2023-01-15T14:35:21.514541+0000","last_undegraded":"2023-01-15T14:35:21.514541+0000","last_fullsized":"2023-01-15T14:35:21.514541+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.41305103399999998,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.36","version":"194'81","reported_seq":67,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.517309+0000","last_change":"2023-01-15T14:35:21.518117+0000","last_active":"2023-01-15T14:35:21.517309+0000","last_peered":"2023-01-15T14:35:21.517309+0000","last_clean":"2023-01-15T14:35:21.517309+0000","last_became_active":"2023-01-15T14:35:19.288888+0000","last_became_peered":"2023-01-15T14:35:19.288888+0000","last_unstale":"2023-01-15T14:35:21.517309+0000","last_undegraded":"2023-01-15T14:35:21.517309+0000","last_fullsized":"2023-01-15T14:35:21.517309+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:34:51.296526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.296526+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.66934546500000003,"objects_trimmed":0,"snaptrim_duration":0.41558602100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.37","version":"200'78","reported_seq":622,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.318047+0000","last_change":"2023-01-15T14:35:22.318047+0000","last_active":"2023-01-15T14:35:22.318047+0000","last_peered":"2023-01-15T14:35:22.318047+0000","last_clean":"2023-01-15T14:35:22.318047+0000","last_became_active":"2023-01-15T14:35:19.152223+0000","last_became_peered":"2023-01-15T14:35:19.152223+0000","last_unstale":"2023-01-15T14:35:22.318047+0000","last_undegraded":"2023-01-15T14:35:22.318047+0000","last_fullsized":"2023-01-15T14:35:22.318047+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.34564812099999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.38","version":"426'347","reported_seq":1797,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.305263+0000","last_change":"2023-01-15T14:35:21.406965+0000","last_active":"2023-01-15T14:35:21.305263+0000","last_peered":"2023-01-15T14:35:21.305263+0000","last_clean":"2023-01-15T14:35:21.305263+0000","last_became_active":"2023-01-15T14:35:19.385249+0000","last_became_peered":"2023-01-15T14:35:19.385249+0000","last_unstale":"2023-01-15T14:35:21.305263+0000","last_undegraded":"2023-01-15T14:35:21.305263+0000","last_fullsized":"2023-01-15T14:35:21.305263+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.30586007100000001,"stat_sum":{"num_bytes":4230716,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":21424,"num_write":20,"num_write_kb":6727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":16442530,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.39","version":"423'435","reported_seq":1764,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.986420+0000","last_change":"2023-01-15T14:35:22.986420+0000","last_active":"2023-01-15T14:35:22.986420+0000","last_peered":"2023-01-15T14:35:22.986420+0000","last_clean":"2023-01-15T14:35:22.986420+0000","last_became_active":"2023-01-15T14:35:19.092523+0000","last_became_peered":"2023-01-15T14:35:19.092523+0000","last_unstale":"2023-01-15T14:35:22.986420+0000","last_undegraded":"2023-01-15T14:35:22.986420+0000","last_fullsized":"2023-01-15T14:35:22.986420+0000","mapping_epoch":357,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.35011291300000003,"stat_sum":{"num_bytes":17098873,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":34,"num_read_kb":57397,"num_write":12,"num_write_kb":6931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":13431306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1576,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.167006+0000","last_change":"2023-01-15T14:35:22.167006+0000","last_active":"2023-01-15T14:35:22.167006+0000","last_peered":"2023-01-15T14:35:22.167006+0000","last_clean":"2023-01-15T14:35:22.167006+0000","last_became_active":"2023-01-15T14:35:19.407553+0000","last_became_peered":"2023-01-15T14:35:19.407553+0000","last_unstale":"2023-01-15T14:35:22.167006+0000","last_undegraded":"2023-01-15T14:35:22.167006+0000","last_fullsized":"2023-01-15T14:35:22.167006+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.80696214099999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.3b","version":"221'60","reported_seq":218,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.190388+0000","last_change":"2023-01-15T14:35:23.190534+0000","last_active":"2023-01-15T14:35:23.190388+0000","last_peered":"2023-01-15T14:35:23.190388+0000","last_clean":"2023-01-15T14:35:23.190388+0000","last_became_active":"2023-01-15T14:35:19.190669+0000","last_became_peered":"2023-01-15T14:35:19.190669+0000","last_unstale":"2023-01-15T14:35:23.190388+0000","last_undegraded":"2023-01-15T14:35:23.190388+0000","last_fullsized":"2023-01-15T14:35:23.190388+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.10256452000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":63,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.803392+0000","last_change":"2023-01-15T14:35:23.803535+0000","last_active":"2023-01-15T14:35:23.803392+0000","last_peered":"2023-01-15T14:35:23.803392+0000","last_clean":"2023-01-15T14:35:23.803392+0000","last_became_active":"2023-01-15T14:35:19.092969+0000","last_became_peered":"2023-01-15T14:35:19.092969+0000","last_unstale":"2023-01-15T14:35:23.803392+0000","last_undegraded":"2023-01-15T14:35:23.803392+0000","last_fullsized":"2023-01-15T14:35:23.803392+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.81563057500000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.3d","version":"431'281","reported_seq":1618,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.403911+0000","last_change":"2023-01-15T14:35:22.404038+0000","last_active":"2023-01-15T14:35:22.403911+0000","last_peered":"2023-01-15T14:35:22.403911+0000","last_clean":"2023-01-15T14:35:22.403911+0000","last_became_active":"2023-01-15T14:35:19.286241+0000","last_became_peered":"2023-01-15T14:35:19.286241+0000","last_unstale":"2023-01-15T14:35:22.403911+0000","last_undegraded":"2023-01-15T14:35:22.403911+0000","last_fullsized":"2023-01-15T14:35:22.403911+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.43865708199999998,"stat_sum":{"num_bytes":2748010,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":18,"num_read_kb":20692,"num_write":9,"num_write_kb":4358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":7352991,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.3e","version":"423'72","reported_seq":166,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.315549+0000","last_change":"2023-01-15T14:35:23.315651+0000","last_active":"2023-01-15T14:35:23.315549+0000","last_peered":"2023-01-15T14:35:23.315549+0000","last_clean":"2023-01-15T14:35:23.315549+0000","last_became_active":"2023-01-15T14:35:04.273985+0000","last_became_peered":"2023-01-15T14:35:04.273985+0000","last_unstale":"2023-01-15T14:35:23.315549+0000","last_undegraded":"2023-01-15T14:35:23.315549+0000","last_fullsized":"2023-01-15T14:35:23.315549+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.231662382,"stat_sum":{"num_bytes":5664091,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":7,"num_write_kb":4255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":16992273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.3f","version":"428'258","reported_seq":1604,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.207588+0000","last_change":"2023-01-15T14:35:23.207588+0000","last_active":"2023-01-15T14:35:23.207588+0000","last_peered":"2023-01-15T14:35:23.207588+0000","last_clean":"2023-01-15T14:35:23.207588+0000","last_became_active":"2023-01-15T14:35:23.206778+0000","last_became_peered":"2023-01-15T14:35:23.206778+0000","last_unstale":"2023-01-15T14:35:23.207588+0000","last_undegraded":"2023-01-15T14:35:23.207588+0000","last_fullsized":"2023-01-15T14:35:23.207588+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.60129771799999998,"stat_sum":{"num_bytes":17854030,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":29,"num_read_kb":29944,"num_write":25,"num_write_kb":7278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":26429748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10f","length":"1"}]},{"pgid":"3.40","version":"423'339","reported_seq":1655,"reported_epoch":432,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:22.228986+0000","last_change":"2023-01-15T14:35:20.923722+0000","last_active":"2023-01-15T14:35:22.228986+0000","last_peered":"2023-01-15T14:35:22.228986+0000","last_clean":"2023-01-15T14:35:02.920679+0000","last_became_active":"2023-01-15T14:35:05.372244+0000","last_became_peered":"2023-01-15T14:35:05.372244+0000","last_unstale":"2023-01-15T14:35:22.228986+0000","last_undegraded":"2023-01-15T14:35:05.073987+0000","last_fullsized":"2023-01-15T14:35:05.072453+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":362,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.00105211,"stat_sum":{"num_bytes":12649455,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":37,"num_read_kb":35543,"num_write":38,"num_write_kb":14039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":29167491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[0,7],"avail_no_missing":["0","3","7"],"object_location_counts":[{"shards":"0,3,7","objects":4},{"shards":"0,7","objects":2}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[{"start":"126","length":"1"}]},{"pgid":"3.41","version":"155'121","reported_seq":1132,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.001054+0000","last_change":"2023-01-15T14:35:21.101798+0000","last_active":"2023-01-15T14:35:21.001054+0000","last_peered":"2023-01-15T14:35:21.001054+0000","last_clean":"2023-01-15T14:35:21.001054+0000","last_became_active":"2023-01-15T14:35:07.565877+0000","last_became_peered":"2023-01-15T14:35:07.565877+0000","last_unstale":"2023-01-15T14:35:21.001054+0000","last_undegraded":"2023-01-15T14:35:21.001054+0000","last_fullsized":"2023-01-15T14:35:21.001054+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.100877728,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":1224,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.408322+0000","last_change":"2023-01-15T14:35:21.409130+0000","last_active":"2023-01-15T14:35:21.408322+0000","last_peered":"2023-01-15T14:35:21.408322+0000","last_clean":"2023-01-15T14:35:21.408322+0000","last_became_active":"2023-01-15T14:35:11.450430+0000","last_became_peered":"2023-01-15T14:35:11.450430+0000","last_unstale":"2023-01-15T14:35:21.408322+0000","last_undegraded":"2023-01-15T14:35:21.408322+0000","last_fullsized":"2023-01-15T14:35:21.408322+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:34:57.046809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:57.046809+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:16.212","scrub_duration":0.71122905999999997,"objects_trimmed":0,"snaptrim_duration":0.30770751400000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.43","version":"429'343","reported_seq":1723,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.412863+0000","last_change":"2023-01-15T14:35:21.412863+0000","last_active":"2023-01-15T14:35:21.412863+0000","last_peered":"2023-01-15T14:35:21.412863+0000","last_clean":"2023-01-15T14:35:21.412863+0000","last_became_active":"2023-01-15T14:35:18.088548+0000","last_became_peered":"2023-01-15T14:35:18.088548+0000","last_unstale":"2023-01-15T14:35:21.412863+0000","last_undegraded":"2023-01-15T14:35:21.412863+0000","last_fullsized":"2023-01-15T14:35:21.412863+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.31093401199999998,"stat_sum":{"num_bytes":9109625,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":13,"num_read_kb":16586,"num_write":12,"num_write_kb":5193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12504705,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.44","version":"245'268","reported_seq":1292,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.521787+0000","last_change":"2023-01-15T14:35:21.522431+0000","last_active":"2023-01-15T14:35:21.521787+0000","last_peered":"2023-01-15T14:35:21.521787+0000","last_clean":"2023-01-15T14:35:21.521787+0000","last_became_active":"2023-01-15T14:35:07.262970+0000","last_became_peered":"2023-01-15T14:35:07.262970+0000","last_unstale":"2023-01-15T14:35:21.521787+0000","last_undegraded":"2023-01-15T14:35:21.521787+0000","last_fullsized":"2023-01-15T14:35:21.521787+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.41936194599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,4],"acting":[6,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.45","version":"253'125","reported_seq":1103,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.965520+0000","last_change":"2023-01-15T14:35:21.965520+0000","last_active":"2023-01-15T14:35:21.965520+0000","last_peered":"2023-01-15T14:35:21.965520+0000","last_clean":"2023-01-15T14:35:21.965520+0000","last_became_active":"2023-01-15T14:33:55.277495+0000","last_became_peered":"2023-01-15T14:33:55.277495+0000","last_unstale":"2023-01-15T14:35:21.965520+0000","last_undegraded":"2023-01-15T14:35:21.965520+0000","last_fullsized":"2023-01-15T14:35:21.965520+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:34:50.381073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:50.381073+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:56.850","scrub_duration":0.52191215300000005,"objects_trimmed":0,"snaptrim_duration":0.90770510199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.46","version":"102'52","reported_seq":121,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.316122+0000","last_change":"2023-01-15T14:35:23.316167+0000","last_active":"2023-01-15T14:35:23.316122+0000","last_peered":"2023-01-15T14:35:23.316122+0000","last_clean":"2023-01-15T14:35:23.316122+0000","last_became_active":"2023-01-15T14:35:04.373495+0000","last_became_peered":"2023-01-15T14:35:04.373495+0000","last_unstale":"2023-01-15T14:35:23.316122+0000","last_undegraded":"2023-01-15T14:35:23.316122+0000","last_fullsized":"2023-01-15T14:35:23.316122+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:21.428","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.234998872,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2142,"num_write":3,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.47","version":"245'112","reported_seq":1319,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.091387+0000","last_change":"2023-01-15T14:35:23.091387+0000","last_active":"2023-01-15T14:35:23.091387+0000","last_peered":"2023-01-15T14:35:23.091387+0000","last_clean":"2023-01-15T14:35:23.091387+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:35:23.091387+0000","last_undegraded":"2023-01-15T14:35:23.091387+0000","last_fullsized":"2023-01-15T14:35:23.091387+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:34:21.450320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:21.450320+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:50.200","scrub_duration":0.30497759099999999,"objects_trimmed":0,"snaptrim_duration":0.0024277299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.48","version":"179'204","reported_seq":1362,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.963775+0000","last_change":"2023-01-15T14:35:21.963775+0000","last_active":"2023-01-15T14:35:21.963775+0000","last_peered":"2023-01-15T14:35:21.963775+0000","last_clean":"2023-01-15T14:35:21.963775+0000","last_became_active":"2023-01-15T14:35:11.517680+0000","last_became_peered":"2023-01-15T14:35:11.517680+0000","last_unstale":"2023-01-15T14:35:21.963775+0000","last_undegraded":"2023-01-15T14:35:21.963775+0000","last_fullsized":"2023-01-15T14:35:21.963775+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.102959697,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.49","version":"110'149","reported_seq":1118,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.986404+0000","last_change":"2023-01-15T14:35:22.986404+0000","last_active":"2023-01-15T14:35:22.986404+0000","last_peered":"2023-01-15T14:35:22.986404+0000","last_clean":"2023-01-15T14:35:22.986404+0000","last_became_active":"2023-01-15T14:34:00.043221+0000","last_became_peered":"2023-01-15T14:34:00.043221+0000","last_unstale":"2023-01-15T14:35:22.986404+0000","last_undegraded":"2023-01-15T14:35:22.986404+0000","last_fullsized":"2023-01-15T14:35:22.986404+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.35008524400000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.4a","version":"433'412","reported_seq":1758,"reported_epoch":433,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:23.803522+0000","last_change":"2023-01-15T14:35:23.803522+0000","last_active":"2023-01-15T14:35:23.803522+0000","last_peered":"2023-01-15T14:35:23.803522+0000","last_clean":"2023-01-15T14:35:23.803522+0000","last_became_active":"2023-01-15T14:35:22.264683+0000","last_became_peered":"2023-01-15T14:35:22.264683+0000","last_unstale":"2023-01-15T14:35:23.803522+0000","last_undegraded":"2023-01-15T14:35:23.803522+0000","last_fullsized":"2023-01-15T14:35:23.803522+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:34.117","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.70426192200000004,"stat_sum":{"num_bytes":15797888,"num_objects":7,"num_object_clones":7,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":84,"num_read_kb":95190,"num_write":86,"num_write_kb":26766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":39901450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.4b","version":"155'53","reported_seq":57,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.409263+0000","last_change":"2023-01-15T14:35:21.410001+0000","last_active":"2023-01-15T14:35:21.409263+0000","last_peered":"2023-01-15T14:35:21.409263+0000","last_clean":"2023-01-15T14:35:21.409263+0000","last_became_active":"2023-01-15T14:35:04.179749+0000","last_became_peered":"2023-01-15T14:35:04.179749+0000","last_unstale":"2023-01-15T14:35:21.409263+0000","last_undegraded":"2023-01-15T14:35:21.409263+0000","last_fullsized":"2023-01-15T14:35:21.409263+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:34:44.116779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:44.116779+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.308301194,"stat_sum":{"num_bytes":432780,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":1674,"num_write":3,"num_write_kb":1409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.4c","version":"309'71","reported_seq":303,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.702172+0000","last_change":"2023-01-15T14:35:23.702354+0000","last_active":"2023-01-15T14:35:23.702172+0000","last_peered":"2023-01-15T14:35:23.702172+0000","last_clean":"2023-01-15T14:35:23.702172+0000","last_became_active":"2023-01-15T14:35:04.379289+0000","last_became_peered":"2023-01-15T14:35:04.379289+0000","last_unstale":"2023-01-15T14:35:23.702172+0000","last_undegraded":"2023-01-15T14:35:23.702172+0000","last_fullsized":"2023-01-15T14:35:23.702172+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:27.027","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.71468080499999997,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.4d","version":"432'318","reported_seq":1588,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.880115+0000","last_change":"2023-01-15T14:35:22.880115+0000","last_active":"2023-01-15T14:35:22.880115+0000","last_peered":"2023-01-15T14:35:22.880115+0000","last_clean":"2023-01-15T14:35:22.880115+0000","last_became_active":"2023-01-15T14:34:05.178613+0000","last_became_peered":"2023-01-15T14:34:05.178613+0000","last_unstale":"2023-01-15T14:35:22.880115+0000","last_undegraded":"2023-01-15T14:35:22.880115+0000","last_fullsized":"2023-01-15T14:35:22.880115+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.76770838799999996,"stat_sum":{"num_bytes":17922713,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":16,"num_read_kb":15961,"num_write":23,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.4e","version":"139'38","reported_seq":276,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.065333+0000","last_change":"2023-01-15T14:35:22.065333+0000","last_active":"2023-01-15T14:35:22.065333+0000","last_peered":"2023-01-15T14:35:22.065333+0000","last_clean":"2023-01-15T14:35:22.065333+0000","last_became_active":"2023-01-15T14:33:21.980461+0000","last_became_peered":"2023-01-15T14:33:21.980461+0000","last_unstale":"2023-01-15T14:35:22.065333+0000","last_undegraded":"2023-01-15T14:35:22.065333+0000","last_fullsized":"2023-01-15T14:35:22.065333+0000","mapping_epoch":312,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":313,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.20235814599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.4f","version":"155'145","reported_seq":995,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.102267+0000","last_change":"2023-01-15T14:35:21.102677+0000","last_active":"2023-01-15T14:35:21.102267+0000","last_peered":"2023-01-15T14:35:21.102267+0000","last_clean":"2023-01-15T14:35:21.102267+0000","last_became_active":"2023-01-15T14:35:07.159725+0000","last_became_peered":"2023-01-15T14:35:07.159725+0000","last_unstale":"2023-01-15T14:35:21.102267+0000","last_undegraded":"2023-01-15T14:35:21.102267+0000","last_fullsized":"2023-01-15T14:35:21.102267+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10047453100000001,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.50","version":"433'476","reported_seq":1819,"reported_epoch":433,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:23.386705+0000","last_change":"2023-01-15T14:35:22.987676+0000","last_active":"2023-01-15T14:35:23.386705+0000","last_peered":"2023-01-15T14:35:23.386705+0000","last_clean":"2023-01-15T14:35:23.386705+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:35:23.386705+0000","last_undegraded":"2023-01-15T14:35:23.386705+0000","last_fullsized":"2023-01-15T14:35:23.386705+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:37.144","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.91713547399999995,"stat_sum":{"num_bytes":16579116,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":76,"num_read_kb":78946,"num_write":64,"num_write_kb":23394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.51","version":"428'345","reported_seq":1737,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.319828+0000","last_change":"2023-01-15T14:35:23.320507+0000","last_active":"2023-01-15T14:35:23.319828+0000","last_peered":"2023-01-15T14:35:23.319828+0000","last_clean":"2023-01-15T14:35:23.319828+0000","last_became_active":"2023-01-15T14:34:44.372829+0000","last_became_peered":"2023-01-15T14:34:44.372829+0000","last_unstale":"2023-01-15T14:35:23.319828+0000","last_undegraded":"2023-01-15T14:35:23.319828+0000","last_fullsized":"2023-01-15T14:35:23.319828+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":345,"log_dups_size":0,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.340278372,"stat_sum":{"num_bytes":9988928,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":2393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":753,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.522585+0000","last_change":"2023-01-15T14:35:21.523270+0000","last_active":"2023-01-15T14:35:21.522585+0000","last_peered":"2023-01-15T14:35:21.522585+0000","last_clean":"2023-01-15T14:35:21.522585+0000","last_became_active":"2023-01-15T14:35:07.357440+0000","last_became_peered":"2023-01-15T14:35:07.357440+0000","last_unstale":"2023-01-15T14:35:21.522585+0000","last_undegraded":"2023-01-15T14:35:21.522585+0000","last_fullsized":"2023-01-15T14:35:21.522585+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.42000558599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.53","version":"430'477","reported_seq":1810,"reported_epoch":431,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:21.197184+0000","last_change":"2023-01-15T14:35:20.295732+0000","last_active":"2023-01-15T14:35:21.197184+0000","last_peered":"2023-01-15T14:35:21.197184+0000","last_clean":"2023-01-15T14:35:21.197184+0000","last_became_active":"2023-01-15T14:35:19.383102+0000","last_became_peered":"2023-01-15T14:35:19.383102+0000","last_unstale":"2023-01-15T14:35:21.197184+0000","last_undegraded":"2023-01-15T14:35:21.197184+0000","last_fullsized":"2023-01-15T14:35:21.197184+0000","mapping_epoch":428,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.21322300299999999,"stat_sum":{"num_bytes":30645491,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":67,"num_read_kb":67500,"num_write":54,"num_write_kb":27412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":72326676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"}]},{"pgid":"3.54","version":"245'148","reported_seq":1314,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.099368+0000","last_change":"2023-01-15T14:35:21.099337+0000","last_active":"2023-01-15T14:35:21.099368+0000","last_peered":"2023-01-15T14:35:21.099368+0000","last_clean":"2023-01-15T14:35:21.099368+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:35:21.099368+0000","last_undegraded":"2023-01-15T14:35:21.099368+0000","last_fullsized":"2023-01-15T14:35:21.099368+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:34:47.392699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.392699+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:14.435","scrub_duration":0.50581767899999996,"objects_trimmed":0,"snaptrim_duration":0.40512653500000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.55","version":"110'55","reported_seq":1021,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.964887+0000","last_change":"2023-01-15T14:35:21.964887+0000","last_active":"2023-01-15T14:35:21.964887+0000","last_peered":"2023-01-15T14:35:21.964887+0000","last_clean":"2023-01-15T14:35:21.964887+0000","last_became_active":"2023-01-15T14:35:11.347080+0000","last_became_peered":"2023-01-15T14:35:11.347080+0000","last_unstale":"2023-01-15T14:35:21.964887+0000","last_undegraded":"2023-01-15T14:35:21.964887+0000","last_fullsized":"2023-01-15T14:35:21.964887+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:34:59.815570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:59.815570+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.90788102599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.56","version":"320'127","reported_seq":813,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.988348+0000","last_change":"2023-01-15T14:35:22.988348+0000","last_active":"2023-01-15T14:35:22.988348+0000","last_peered":"2023-01-15T14:35:22.988348+0000","last_clean":"2023-01-15T14:35:22.988348+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:35:22.988348+0000","last_undegraded":"2023-01-15T14:35:22.988348+0000","last_fullsized":"2023-01-15T14:35:22.988348+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.91027767000000004,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.57","version":"431'120","reported_seq":308,"reported_epoch":432,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:22.476225+0000","last_change":"2023-01-15T14:35:22.009355+0000","last_active":"2023-01-15T14:35:22.476225+0000","last_peered":"2023-01-15T14:35:22.476225+0000","last_clean":"2023-01-15T14:35:02.556894+0000","last_became_active":"2023-01-15T14:35:05.185559+0000","last_became_peered":"2023-01-15T14:35:05.185559+0000","last_unstale":"2023-01-15T14:35:22.476225+0000","last_undegraded":"2023-01-15T14:35:04.976030+0000","last_fullsized":"2023-01-15T14:35:04.975477+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'100","last_scrub_stamp":"2023-01-15T14:34:23.447852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.447852+0000","objects_scrubbed":2,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:38.971","scrub_duration":0.52101476099999999,"objects_trimmed":0,"snaptrim_duration":0.50438350200000004,"stat_sum":{"num_bytes":8003958,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":25129,"num_write":11,"num_write_kb":2743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,2],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2","objects":2},{"shards":"0,2,3","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.58","version":"323'320","reported_seq":1738,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.701023+0000","last_change":"2023-01-15T14:35:23.701185+0000","last_active":"2023-01-15T14:35:23.701023+0000","last_peered":"2023-01-15T14:35:23.701023+0000","last_clean":"2023-01-15T14:35:23.701023+0000","last_became_active":"2023-01-15T14:35:07.359588+0000","last_became_peered":"2023-01-15T14:35:07.359588+0000","last_unstale":"2023-01-15T14:35:23.701023+0000","last_undegraded":"2023-01-15T14:35:23.701023+0000","last_fullsized":"2023-01-15T14:35:23.701023+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'320","last_scrub_stamp":"2023-01-15T14:34:47.541655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.541655+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:52.418","scrub_duration":0.40398073200000001,"objects_trimmed":0,"snaptrim_duration":0.71400382600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.59","version":"294'256","reported_seq":1494,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.088269+0000","last_change":"2023-01-15T14:35:23.088269+0000","last_active":"2023-01-15T14:35:23.088269+0000","last_peered":"2023-01-15T14:35:23.088269+0000","last_clean":"2023-01-15T14:35:23.088269+0000","last_became_active":"2023-01-15T14:35:07.462526+0000","last_became_peered":"2023-01-15T14:35:07.462526+0000","last_unstale":"2023-01-15T14:35:23.088269+0000","last_undegraded":"2023-01-15T14:35:23.088269+0000","last_fullsized":"2023-01-15T14:35:23.088269+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:34:30.207952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:30.207952+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:30.349","scrub_duration":0.024191634,"objects_trimmed":0,"snaptrim_duration":0.0026843140000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.5a","version":"428'367","reported_seq":1683,"reported_epoch":431,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:20.990618+0000","last_change":"2023-01-15T14:35:05.482085+0000","last_active":"2023-01-15T14:35:20.990618+0000","last_peered":"2023-01-15T14:35:20.990618+0000","last_clean":"2023-01-15T14:35:02.396095+0000","last_became_active":"2023-01-15T14:35:05.480728+0000","last_became_peered":"2023-01-15T14:35:05.480728+0000","last_unstale":"2023-01-15T14:35:20.990618+0000","last_undegraded":"2023-01-15T14:35:05.171683+0000","last_fullsized":"2023-01-15T14:35:05.170835+0000","mapping_epoch":414,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.448168648,"stat_sum":{"num_bytes":9647449,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":19295,"num_write":6,"num_write_kb":1588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":5983424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1","objects":1},{"shards":"0,1,2","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10f","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.5b","version":"283'80","reported_seq":1094,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.089163+0000","last_change":"2023-01-15T14:35:23.089223+0000","last_active":"2023-01-15T14:35:23.089163+0000","last_peered":"2023-01-15T14:35:23.089163+0000","last_clean":"2023-01-15T14:35:23.089163+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:35:23.089163+0000","last_undegraded":"2023-01-15T14:35:23.089163+0000","last_fullsized":"2023-01-15T14:35:23.089163+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:42.155","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.0011420919999999999,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.5c","version":"141'46","reported_seq":256,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.524281+0000","last_change":"2023-01-15T14:35:21.524875+0000","last_active":"2023-01-15T14:35:21.524281+0000","last_peered":"2023-01-15T14:35:21.524281+0000","last_clean":"2023-01-15T14:35:21.524281+0000","last_became_active":"2023-01-15T14:35:11.349587+0000","last_became_peered":"2023-01-15T14:35:11.349587+0000","last_unstale":"2023-01-15T14:35:21.524281+0000","last_undegraded":"2023-01-15T14:35:21.524281+0000","last_fullsized":"2023-01-15T14:35:21.524281+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.32242468699999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1505,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.089668+0000","last_change":"2023-01-15T14:35:23.089668+0000","last_active":"2023-01-15T14:35:23.089668+0000","last_peered":"2023-01-15T14:35:23.089668+0000","last_clean":"2023-01-15T14:35:23.089668+0000","last_became_active":"2023-01-15T14:34:00.242122+0000","last_became_peered":"2023-01-15T14:34:00.242122+0000","last_unstale":"2023-01-15T14:35:23.089668+0000","last_undegraded":"2023-01-15T14:35:23.089668+0000","last_fullsized":"2023-01-15T14:35:23.089668+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:19.729021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:19.729021+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:29.740","scrub_duration":0.50499948100000003,"objects_trimmed":0,"snaptrim_duration":0.0040018229999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.5e","version":"215'39","reported_seq":1124,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.314447+0000","last_change":"2023-01-15T14:35:23.314498+0000","last_active":"2023-01-15T14:35:23.314447+0000","last_peered":"2023-01-15T14:35:23.314447+0000","last_clean":"2023-01-15T14:35:23.314447+0000","last_became_active":"2023-01-15T14:35:11.445183+0000","last_became_peered":"2023-01-15T14:35:11.445183+0000","last_unstale":"2023-01-15T14:35:23.314447+0000","last_undegraded":"2023-01-15T14:35:23.314447+0000","last_fullsized":"2023-01-15T14:35:23.314447+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.33098932800000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1548,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.094328+0000","last_change":"2023-01-15T14:35:21.094287+0000","last_active":"2023-01-15T14:35:21.094328+0000","last_peered":"2023-01-15T14:35:21.094328+0000","last_clean":"2023-01-15T14:35:21.094328+0000","last_became_active":"2023-01-15T14:33:40.125703+0000","last_became_peered":"2023-01-15T14:33:40.125703+0000","last_unstale":"2023-01-15T14:35:21.094328+0000","last_undegraded":"2023-01-15T14:35:21.094328+0000","last_fullsized":"2023-01-15T14:35:21.094328+0000","mapping_epoch":330,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":331,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.1901728,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.60","version":"419'295","reported_seq":1636,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.316714+0000","last_change":"2023-01-15T14:35:23.316758+0000","last_active":"2023-01-15T14:35:23.316714+0000","last_peered":"2023-01-15T14:35:23.316714+0000","last_clean":"2023-01-15T14:35:23.316714+0000","last_became_active":"2023-01-15T14:34:44.073670+0000","last_became_peered":"2023-01-15T14:34:44.073670+0000","last_unstale":"2023-01-15T14:35:23.316714+0000","last_undegraded":"2023-01-15T14:35:23.316714+0000","last_fullsized":"2023-01-15T14:35:23.316714+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"402'291","last_scrub_stamp":"2023-01-15T14:34:55.535113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:55.535113+0000","objects_scrubbed":4,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:04.248","scrub_duration":0.814267513,"objects_trimmed":0,"snaptrim_duration":0.235876219,"stat_sum":{"num_bytes":3971199,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.61","version":"428'168","reported_seq":1233,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.065227+0000","last_change":"2023-01-15T14:35:22.065227+0000","last_active":"2023-01-15T14:35:22.065227+0000","last_peered":"2023-01-15T14:35:22.065227+0000","last_clean":"2023-01-15T14:35:22.065227+0000","last_became_active":"2023-01-15T14:34:44.473866+0000","last_became_peered":"2023-01-15T14:34:44.473866+0000","last_unstale":"2023-01-15T14:35:22.065227+0000","last_undegraded":"2023-01-15T14:35:22.065227+0000","last_fullsized":"2023-01-15T14:35:22.065227+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:32.734","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.91217811000000004,"stat_sum":{"num_bytes":3823692,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":16964,"num_write":6,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.62","version":"432'203","reported_seq":1598,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.321445+0000","last_change":"2023-01-15T14:35:23.322149+0000","last_active":"2023-01-15T14:35:23.321445+0000","last_peered":"2023-01-15T14:35:23.321445+0000","last_clean":"2023-01-15T14:35:23.321445+0000","last_became_active":"2023-01-15T14:34:44.164909+0000","last_became_peered":"2023-01-15T14:34:44.164909+0000","last_unstale":"2023-01-15T14:35:23.321445+0000","last_undegraded":"2023-01-15T14:35:23.321445+0000","last_fullsized":"2023-01-15T14:35:23.321445+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":7,"last_scrub":"403'189","last_scrub_stamp":"2023-01-15T14:34:56.851447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:56.851447+0000","objects_scrubbed":5,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:20.283","scrub_duration":0.30960584400000002,"objects_trimmed":0,"snaptrim_duration":0.34025424599999998,"stat_sum":{"num_bytes":9221281,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":9592,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.63","version":"149'233","reported_seq":944,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.093046+0000","last_change":"2023-01-15T14:35:21.093019+0000","last_active":"2023-01-15T14:35:21.093046+0000","last_peered":"2023-01-15T14:35:21.093046+0000","last_clean":"2023-01-15T14:35:21.093046+0000","last_became_active":"2023-01-15T14:35:07.161207+0000","last_became_peered":"2023-01-15T14:35:07.161207+0000","last_unstale":"2023-01-15T14:35:21.093046+0000","last_undegraded":"2023-01-15T14:35:21.093046+0000","last_fullsized":"2023-01-15T14:35:21.093046+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:34:56.637790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:56.637790+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:16.461","scrub_duration":0.63257800399999997,"objects_trimmed":0,"snaptrim_duration":0.41056638699999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.64","version":"334'251","reported_seq":1615,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.189148+0000","last_change":"2023-01-15T14:35:21.189113+0000","last_active":"2023-01-15T14:35:21.189148+0000","last_peered":"2023-01-15T14:35:21.189148+0000","last_clean":"2023-01-15T14:35:21.189148+0000","last_became_active":"2023-01-15T14:35:07.057736+0000","last_became_peered":"2023-01-15T14:35:07.057736+0000","last_unstale":"2023-01-15T14:35:21.189148+0000","last_undegraded":"2023-01-15T14:35:21.189148+0000","last_fullsized":"2023-01-15T14:35:21.189148+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"334'251","last_scrub_stamp":"2023-01-15T14:34:48.644529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.644529+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:49.145","scrub_duration":0.70872698300000003,"objects_trimmed":0,"snaptrim_duration":0.318712053,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.65","version":"431'173","reported_seq":1520,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.319690+0000","last_change":"2023-01-15T14:35:23.319802+0000","last_active":"2023-01-15T14:35:23.319690+0000","last_peered":"2023-01-15T14:35:23.319690+0000","last_clean":"2023-01-15T14:35:23.319690+0000","last_became_active":"2023-01-15T14:35:10.233584+0000","last_became_peered":"2023-01-15T14:35:10.233584+0000","last_unstale":"2023-01-15T14:35:23.319690+0000","last_undegraded":"2023-01-15T14:35:23.319690+0000","last_fullsized":"2023-01-15T14:35:23.319690+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'126","last_scrub_stamp":"2023-01-15T14:34:23.903906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.903906+0000","objects_scrubbed":6,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:42.475","scrub_duration":0.30981859099999998,"objects_trimmed":0,"snaptrim_duration":0.33594651199999997,"stat_sum":{"num_bytes":15403336,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":15,"num_read_kb":13207,"num_write":16,"num_write_kb":7278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20310327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.66","version":"102'52","reported_seq":57,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.519941+0000","last_change":"2023-01-15T14:35:21.520778+0000","last_active":"2023-01-15T14:35:21.519941+0000","last_peered":"2023-01-15T14:35:21.519941+0000","last_clean":"2023-01-15T14:35:21.519941+0000","last_became_active":"2023-01-15T14:35:04.489876+0000","last_became_peered":"2023-01-15T14:35:04.489876+0000","last_unstale":"2023-01-15T14:35:21.519941+0000","last_undegraded":"2023-01-15T14:35:21.519941+0000","last_fullsized":"2023-01-15T14:35:21.519941+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:34:32.591909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.591909+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:56.258","scrub_duration":0.39200112999999998,"objects_trimmed":0,"snaptrim_duration":0.417862752,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.67","version":"191'79","reported_seq":237,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.988457+0000","last_change":"2023-01-15T14:35:22.988457+0000","last_active":"2023-01-15T14:35:22.988457+0000","last_peered":"2023-01-15T14:35:22.988457+0000","last_clean":"2023-01-15T14:35:22.988457+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:35:22.988457+0000","last_undegraded":"2023-01-15T14:35:22.988457+0000","last_fullsized":"2023-01-15T14:35:22.988457+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:02.115","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":1.015014726,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.68","version":"420'278","reported_seq":1619,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.088141+0000","last_change":"2023-01-15T14:35:21.088109+0000","last_active":"2023-01-15T14:35:21.088141+0000","last_peered":"2023-01-15T14:35:21.088141+0000","last_clean":"2023-01-15T14:35:21.088141+0000","last_became_active":"2023-01-15T14:34:13.134258+0000","last_became_peered":"2023-01-15T14:34:13.134258+0000","last_unstale":"2023-01-15T14:35:21.088141+0000","last_undegraded":"2023-01-15T14:35:21.088141+0000","last_fullsized":"2023-01-15T14:35:21.088141+0000","mapping_epoch":362,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":363,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":278,"log_dups_size":0,"ondisk_log_size":278,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:00.032","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.40841129199999998,"stat_sum":{"num_bytes":1292047,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17027823,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.69","version":"110'149","reported_seq":1068,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.411206+0000","last_change":"2023-01-15T14:35:21.411978+0000","last_active":"2023-01-15T14:35:21.411206+0000","last_peered":"2023-01-15T14:35:21.411206+0000","last_clean":"2023-01-15T14:35:21.411206+0000","last_became_active":"2023-01-15T14:35:07.344216+0000","last_became_peered":"2023-01-15T14:35:07.344216+0000","last_unstale":"2023-01-15T14:35:21.411206+0000","last_undegraded":"2023-01-15T14:35:21.411206+0000","last_fullsized":"2023-01-15T14:35:21.411206+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.31022905299999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.6a","version":"191'191","reported_seq":881,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.520874+0000","last_change":"2023-01-15T14:35:21.521685+0000","last_active":"2023-01-15T14:35:21.520874+0000","last_peered":"2023-01-15T14:35:21.520874+0000","last_clean":"2023-01-15T14:35:21.520874+0000","last_became_active":"2023-01-15T14:35:07.360156+0000","last_became_peered":"2023-01-15T14:35:07.360156+0000","last_unstale":"2023-01-15T14:35:21.520874+0000","last_undegraded":"2023-01-15T14:35:21.520874+0000","last_fullsized":"2023-01-15T14:35:21.520874+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.418710109,"stat_sum":{"num_bytes":1518387,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":5931,"num_write":14,"num_write_kb":6502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9447727,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.6b","version":"155'53","reported_seq":105,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.987150+0000","last_change":"2023-01-15T14:35:22.987150+0000","last_active":"2023-01-15T14:35:22.987150+0000","last_peered":"2023-01-15T14:35:22.987150+0000","last_clean":"2023-01-15T14:35:22.987150+0000","last_became_active":"2023-01-15T14:34:59.176998+0000","last_became_peered":"2023-01-15T14:34:59.176998+0000","last_unstale":"2023-01-15T14:35:22.987150+0000","last_undegraded":"2023-01-15T14:35:22.987150+0000","last_fullsized":"2023-01-15T14:35:22.987150+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.81209180700000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.6c","version":"141'46","reported_seq":94,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.986974+0000","last_change":"2023-01-15T14:35:22.986974+0000","last_active":"2023-01-15T14:35:22.986974+0000","last_peered":"2023-01-15T14:35:22.986974+0000","last_clean":"2023-01-15T14:35:22.986974+0000","last_became_active":"2023-01-15T14:35:04.085770+0000","last_became_peered":"2023-01-15T14:35:04.085770+0000","last_unstale":"2023-01-15T14:35:22.986974+0000","last_undegraded":"2023-01-15T14:35:22.986974+0000","last_fullsized":"2023-01-15T14:35:22.986974+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:57.577405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:57.577405+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.813191844,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.6d","version":"168'156","reported_seq":839,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-01-15T14:35:21.515449+0000","last_change":"2023-01-15T14:35:21.516158+0000","last_active":"2023-01-15T14:35:21.515449+0000","last_peered":"2023-01-15T14:35:21.515449+0000","last_clean":"2023-01-15T14:35:21.515449+0000","last_became_active":"2023-01-15T14:35:07.163676+0000","last_became_peered":"2023-01-15T14:35:07.163676+0000","last_unstale":"2023-01-15T14:35:21.515449+0000","last_undegraded":"2023-01-15T14:35:21.515449+0000","last_fullsized":"2023-01-15T14:35:21.515449+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.41371426300000003,"stat_sum":{"num_bytes":1516832,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2710,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.6e","version":"139'38","reported_seq":88,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.388660+0000","last_change":"2023-01-15T14:35:23.388660+0000","last_active":"2023-01-15T14:35:23.388660+0000","last_peered":"2023-01-15T14:35:23.388660+0000","last_clean":"2023-01-15T14:35:23.388660+0000","last_became_active":"2023-01-15T14:34:59.097277+0000","last_became_peered":"2023-01-15T14:34:59.097277+0000","last_unstale":"2023-01-15T14:35:23.388660+0000","last_undegraded":"2023-01-15T14:35:23.388660+0000","last_fullsized":"2023-01-15T14:35:23.388660+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.19787689899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.6f","version":"155'145","reported_seq":1146,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:21.091115+0000","last_change":"2023-01-15T14:35:21.091049+0000","last_active":"2023-01-15T14:35:21.091115+0000","last_peered":"2023-01-15T14:35:21.091115+0000","last_clean":"2023-01-15T14:35:21.091115+0000","last_became_active":"2023-01-15T14:35:02.254791+0000","last_became_peered":"2023-01-15T14:35:02.254791+0000","last_unstale":"2023-01-15T14:35:21.091115+0000","last_undegraded":"2023-01-15T14:35:21.091115+0000","last_fullsized":"2023-01-15T14:35:21.091115+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.213960076,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.70","version":"206'252","reported_seq":1217,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.323431+0000","last_change":"2023-01-15T14:35:23.323907+0000","last_active":"2023-01-15T14:35:23.323431+0000","last_peered":"2023-01-15T14:35:23.323431+0000","last_clean":"2023-01-15T14:35:23.323431+0000","last_became_active":"2023-01-15T14:35:07.457483+0000","last_became_peered":"2023-01-15T14:35:07.457483+0000","last_unstale":"2023-01-15T14:35:23.323431+0000","last_undegraded":"2023-01-15T14:35:23.323431+0000","last_fullsized":"2023-01-15T14:35:23.323431+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:34:36.078678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:36.078678+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.158842567,"objects_trimmed":0,"snaptrim_duration":0.24376605300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.71","version":"204'150","reported_seq":952,"reported_epoch":433,"state":"active+clean","last_fresh":"2023-01-15T14:35:23.090862+0000","last_change":"2023-01-15T14:35:23.090862+0000","last_active":"2023-01-15T14:35:23.090862+0000","last_peered":"2023-01-15T14:35:23.090862+0000","last_clean":"2023-01-15T14:35:23.090862+0000","last_became_active":"2023-01-15T14:35:02.151543+0000","last_became_peered":"2023-01-15T14:35:02.151543+0000","last_unstale":"2023-01-15T14:35:23.090862+0000","last_undegraded":"2023-01-15T14:35:23.090862+0000","last_fullsized":"2023-01-15T14:35:23.090862+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.003373694,"stat_sum":{"num_bytes":3558907,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10725,"num_write":4,"num_write_kb":3090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3605110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.72","version":"394'169","reported_seq":1082,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:22.065092+0000","last_change":"2023-01-15T14:35:22.065092+0000","last_active":"2023-01-15T14:35:22.065092+0000","last_peered":"2023-01-15T14:35:22.065092+0000","last_clean":"2023-01-15T14:35:22.065092+0000","last_became_active":"2023-01-15T14:35:02.148756+0000","last_became_peered":"2023-01-15T14:35:02.148756+0000","last_unstale":"2023-01-15T14:35:22.065092+0000","last_undegraded":"2023-01-15T14:35:22.065092+0000","last_fullsized":"2023-01-15T14:35:22.065092+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.90671223400000001,"stat_sum":{"num_bytes":2663943,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3524,"num_write":1,"num_write_kb":1428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2663943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.73","version":"204'275","reported_seq":1433,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.213572+0000","last_change":"2023-01-15T14:35:23.213660+0000","last_active":"2023-01-15T14:35:23.213572+0000","last_peered":"2023-01-15T14:35:23.213572+0000","last_clean":"2023-01-15T14:35:23.213572+0000","last_became_active":"2023-01-15T14:35:02.250324+0000","last_became_peered":"2023-01-15T14:35:02.250324+0000","last_unstale":"2023-01-15T14:35:23.213572+0000","last_undegraded":"2023-01-15T14:35:23.213572+0000","last_fullsized":"2023-01-15T14:35:23.213572+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.23356668799999999,"stat_sum":{"num_bytes":3485477,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":11189,"num_write":7,"num_write_kb":3286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.74","version":"169'138","reported_seq":771,"reported_epoch":432,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:35:22.170456+0000","last_change":"2023-01-15T14:35:22.067256+0000","last_active":"2023-01-15T14:35:22.170456+0000","last_peered":"2023-01-15T14:35:22.170456+0000","last_clean":"2023-01-15T14:31:35.942380+0000","last_became_active":"2023-01-15T14:35:22.067256+0000","last_became_peered":"2023-01-15T14:35:22.067256+0000","last_unstale":"2023-01-15T14:35:22.170456+0000","last_undegraded":"2023-01-15T14:35:22.170456+0000","last_fullsized":"2023-01-15T14:35:21.963852+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":7,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":613464,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2809,"num_write":2,"num_write_kb":305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2],"avail_no_missing":["0","2","3"],"object_location_counts":[{"shards":"0,2","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.75","version":"110'55","reported_seq":22,"reported_epoch":433,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:23.317666+0000","last_change":"2023-01-15T14:35:23.317666+0000","last_active":"2023-01-15T14:35:23.317666+0000","last_peered":"2023-01-15T14:35:23.317666+0000","last_clean":"2023-01-15T14:35:23.317666+0000","last_became_active":"2023-01-15T14:35:21.196393+0000","last_became_peered":"2023-01-15T14:35:21.196393+0000","last_unstale":"2023-01-15T14:35:23.317666+0000","last_undegraded":"2023-01-15T14:35:23.317666+0000","last_fullsized":"2023-01-15T14:35:23.317666+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.33765785700000001,"stat_sum":{"num_bytes":796228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5170,"num_write":3,"num_write_kb":1557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":531514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ec","length":"1"},{"start":"10f","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"}]}],"pool_stats":[{"poolid":3,"num_pg":126,"stat_sum":{"num_bytes":451676560,"num_objects":207,"num_object_clones":154,"num_object_copies":621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":59,"num_objects_unfound":0,"num_objects_dirty":207,"num_whiteouts":16,"num_read":1085,"num_read_kb":1220362,"num_write":807,"num_write_kb":338834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":610851085,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23194,"ondisk_log_size":23194,"up":378,"acting":370,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574937,"num_pgs":64,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1164156,"kb_used_data":92538632,"kb_used_omap":12102,"kb_used_meta":0,"kb_avail":92535224,"statfs":{"total":95948165120,"available":94756069376,"internally_reserved":3489792,"allocated":94759559168,"data_stored":94759559168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12392508,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":6.6619999999999999,"5min":4.6289999999999996,"15min":4.0369999999999999},"min":{"1min":0.53400000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":83.435000000000002,"5min":148.10900000000001,"15min":148.10900000000001},"last":83.435000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":4.0030000000000001,"15min":7.4720000000000004},"min":{"1min":0.59599999999999997,"5min":0.314,"15min":0.314},"max":{"1min":1.1599999999999999,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.59599999999999997}]},{"osd":1,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":3.6589999999999998,"15min":5.5670000000000002},"min":{"1min":0.64300000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.03,"5min":204.74199999999999,"15min":204.74199999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":1.022,"5min":8.3290000000000006,"15min":6.899},"min":{"1min":0.67500000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":3.4089999999999998,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.85299999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":9.9890000000000008,"5min":5.6239999999999997,"15min":5.6769999999999996},"min":{"1min":0.54400000000000004,"5min":0.377,"15min":0.377},"max":{"1min":76.272999999999996,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.752},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.75,"15min":4.1500000000000004},"min":{"1min":0.64900000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.105,"5min":162.96799999999999,"15min":162.96799999999999},"last":1.105}]},{"osd":3,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":8.1660000000000004,"5min":6.375,"15min":3.6560000000000001},"min":{"1min":0.69499999999999995,"5min":0.434,"15min":0.434},"max":{"1min":76.114999999999995,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.97299999999999998,"5min":2.2599999999999998,"15min":3.8149999999999999},"min":{"1min":0.70199999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7170000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":1.018}]},{"osd":4,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":3.6850000000000001,"5min":3.8860000000000001,"15min":7.3040000000000003},"min":{"1min":0.48099999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":40.563000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":1.05},{"interface":"front","average":{"1min":24.846,"5min":9.4670000000000005,"15min":4.8280000000000003},"min":{"1min":0.74199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":184.37200000000001,"5min":184.37200000000001,"15min":184.37200000000001},"last":1.073}]},{"osd":5,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":3.4340000000000002,"15min":5.3040000000000003},"min":{"1min":0.58799999999999997,"5min":0.51200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1400000000000001,"5min":96.263999999999996,"15min":96.263999999999996},"last":1.1319999999999999},{"interface":"front","average":{"1min":8.2710000000000008,"5min":3.2669999999999999,"15min":4.4139999999999997},"min":{"1min":0.871,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":76.159000000000006,"5min":76.159000000000006,"15min":95.430000000000007},"last":0.90100000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":3.4929999999999999,"15min":6.6550000000000002},"min":{"1min":0.64200000000000002,"5min":0.496,"15min":0.496},"max":{"1min":1.401,"5min":96.299999999999997,"15min":96.299999999999997},"last":1.1819999999999999},{"interface":"front","average":{"1min":8.3369999999999997,"5min":3.3340000000000001,"15min":2.1400000000000001},"min":{"1min":0.91500000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":76.209000000000003,"5min":76.209000000000003,"15min":76.209000000000003},"last":1.1539999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574938,"num_pgs":66,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1048644,"kb_used_data":92678124,"kb_used_omap":4748,"kb_used_meta":0,"kb_avail":92650736,"statfs":{"total":95948165120,"available":94874353664,"internally_reserved":28045312,"allocated":94902398976,"data_stored":94902398976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4862302,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":16.390000000000001,"5min":5.2050000000000001,"15min":2.2189999999999999},"min":{"1min":0.56000000000000005,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":166.19900000000001,"5min":166.19900000000001,"15min":166.19900000000001},"last":8.9979999999999993},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.76000000000000001,"15min":0.85099999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92900000000000005,"5min":1.474,"15min":19.413},"last":222.06100000000001}]},{"osd":1,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.8109999999999999,"5min":2.194,"15min":2.3559999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":45.875999999999998,"5min":56.844999999999999,"15min":99.203000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":5.5110000000000001,"5min":2.5609999999999999,"15min":2.218},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.439},"max":{"1min":69.733000000000004,"5min":69.733000000000004,"15min":98.882000000000005},"last":0.46999999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.069,"15min":1.518},"min":{"1min":0.55600000000000005,"5min":0.45600000000000002,"15min":0.41399999999999998},"max":{"1min":2.081,"5min":23.135999999999999,"15min":128.173},"last":27.526},{"interface":"front","average":{"1min":3.9500000000000002,"5min":1.7430000000000001,"15min":1.95},"min":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":45.773000000000003,"5min":45.773000000000003,"15min":114.724},"last":0.375}]},{"osd":3,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.8319999999999999,"5min":2.4910000000000001,"15min":4.8010000000000002},"min":{"1min":0.56200000000000006,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":45.813000000000002,"5min":93.710999999999999,"15min":93.710999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":11.069000000000001,"5min":5.4180000000000001,"15min":8.8190000000000008},"min":{"1min":0.625,"5min":0.625,"15min":0.60599999999999998},"max":{"1min":152.66499999999999,"5min":167.178,"15min":255.24299999999999},"last":27.744}]},{"osd":4,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":12.531000000000001,"5min":21.152000000000001,"15min":7.7300000000000004},"min":{"1min":0.46300000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":107.03,"5min":1256.867,"15min":1256.867},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.90400000000000003,"5min":14.872,"15min":9.3019999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.4350000000000001,"5min":1255.77,"15min":1255.77},"last":27.603000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.8370000000000002,"5min":1.532,"15min":1.415},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":45.673999999999999,"5min":45.673999999999999,"15min":67.469999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":5.5330000000000004,"5min":1.8799999999999999,"15min":1.2589999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.46100000000000002},"max":{"1min":69.554000000000002,"5min":69.554000000000002,"15min":69.554000000000002},"last":0.51100000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.0289999999999999,"15min":3.056},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51300000000000001},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":78.888000000000005},"last":27.824000000000002},{"interface":"front","average":{"1min":9.6920000000000002,"5min":3.6299999999999999,"15min":3.6520000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.499},"max":{"1min":85.358000000000004,"5min":85.358000000000004,"15min":121.297},"last":0.59299999999999997}]}]},{"osd":5,"up_from":0,"seq":55834574938,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574937,"num_pgs":59,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":939324,"kb_used_data":92761928,"kb_used_omap":17989,"kb_used_meta":0,"kb_avail":92760056,"statfs":{"total":95948165120,"available":94986297344,"internally_reserved":1916928,"allocated":94988214272,"data_stored":94988214272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18421270,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":21.449999999999999,"15min":10.609},"min":{"1min":0.505,"5min":0.44,"15min":0.44},"max":{"1min":1.1200000000000001,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":83.123999999999995},{"interface":"front","average":{"1min":14.188000000000001,"5min":5.1139999999999999,"15min":2.4140000000000001},"min":{"1min":0.53300000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":180.33600000000001,"5min":180.33600000000001,"15min":180.33600000000001},"last":0.65200000000000002}]},{"osd":1,"last update":"Sun Jan 15 14:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.954,"15min":6.4569999999999999},"min":{"1min":0.5,"5min":0.49299999999999999,"15min":0.47399999999999998},"max":{"1min":1.9039999999999999,"5min":69.415000000000006,"15min":171.91499999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":4.3010000000000002,"5min":6.0970000000000004,"15min":2.8900000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":70.757000000000005,"5min":162.976,"15min":162.976},"last":0.72599999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":5.4720000000000004,"5min":23.584,"15min":11.519},"min":{"1min":0.58499999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":98.298000000000002,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.276,"15min":1.0289999999999999},"min":{"1min":0.66200000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.089,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.88}]},{"osd":3,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":15.048,"5min":5.5209999999999999,"15min":2.7400000000000002},"min":{"1min":0.67600000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":180.37200000000001,"5min":180.37200000000001,"15min":180.37200000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":6.3010000000000002,"5min":24.248000000000001,"15min":12.223000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":79.888000000000005,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":0.90800000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.88200000000000001,"15min":0.90500000000000003},"min":{"1min":0.60299999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.214,"5min":2.577,"15min":2.577},"last":0.79200000000000004},{"interface":"front","average":{"1min":12.973000000000001,"5min":4.3120000000000003,"15min":2.8780000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":180.255,"5min":180.255,"15min":180.255},"last":0.84899999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":3.6509999999999998,"5min":2.0070000000000001,"15min":4.7270000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":58.231999999999999,"5min":58.231999999999999,"15min":99.780000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.024,"5min":21.626999999999999,"15min":10.375999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.3109999999999999,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":0.94699999999999995}]},{"osd":7,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":24.492000000000001,"5min":8.2859999999999996,"15min":10.702},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":338.72699999999998,"5min":338.72699999999998,"15min":338.72699999999998},"last":179.13800000000001},{"interface":"front","average":{"1min":1.028,"5min":22.184999999999999,"15min":14.502000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.4530000000000001,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.0269999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574937,"num_pgs":58,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1062712,"kb_used_data":92637532,"kb_used_omap":18211,"kb_used_meta":0,"kb_avail":92636668,"statfs":{"total":95948165120,"available":94859948032,"internally_reserved":884736,"allocated":94860832768,"data_stored":94860832768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18648890,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":6.3230000000000004,"15min":5.6289999999999996},"min":{"1min":0.32400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.98199999999999998,"5min":122.065,"15min":174.52199999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":4.3959999999999999,"5min":3.7160000000000002,"15min":6.0439999999999996},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":68.239000000000004,"5min":74.280000000000001,"15min":93.674000000000007},"last":0.501}]},{"osd":1,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.0190000000000001,"5min":5.391,"15min":10.595000000000001},"min":{"1min":0.41499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":62.805999999999997,"5min":117.664,"15min":159.256},"last":62.805999999999997},{"interface":"front","average":{"1min":2.5659999999999998,"5min":10.003,"15min":4.351},"min":{"1min":0.32900000000000001,"5min":0.317,"15min":0.317},"max":{"1min":35.75,"5min":155.44,"15min":174.578},"last":0.58299999999999996}]},{"osd":2,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.3869999999999996,"5min":2.843,"15min":3.129},"min":{"1min":0.433,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":68.156000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":5.1600000000000001,"15min":3.633},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.004,"5min":122.13,"15min":174.45099999999999},"last":0.65000000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.5679999999999996,"5min":29.02,"15min":13.804},"min":{"1min":0.58699999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":68.296000000000006,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":4.242,"5min":29.707000000000001,"15min":19.187000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":62.914999999999999,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":62.914999999999999}]},{"osd":5,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.2949999999999999,"5min":4.6929999999999996,"15min":10.602},"min":{"1min":0.58399999999999996,"5min":0.504,"15min":0.504},"max":{"1min":63.055,"5min":117.744,"15min":159.43000000000001},"last":63.055},{"interface":"front","average":{"1min":0.93500000000000005,"5min":4.3940000000000001,"15min":2.6699999999999999},"min":{"1min":0.73399999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.306,"5min":122.208,"15min":174.61000000000001},"last":0.90100000000000002}]},{"osd":6,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":6.6639999999999997,"5min":5.5419999999999998,"15min":3.0579999999999998},"min":{"1min":0.67500000000000004,"5min":0.65900000000000003,"15min":0.63500000000000001},"max":{"1min":54.311999999999998,"5min":122.26600000000001,"15min":174.63999999999999},"last":54.311999999999998},{"interface":"front","average":{"1min":8.7650000000000006,"5min":2.5550000000000002,"15min":3.3069999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":75.340999999999994,"5min":75.340999999999994,"15min":75.340999999999994},"last":0.93400000000000005}]},{"osd":7,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":10.919,"5min":3.0070000000000001,"15min":3.6200000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":96.635000000000005,"5min":96.635000000000005,"15min":96.635000000000005},"last":0.871},{"interface":"front","average":{"1min":4.351,"5min":10.726000000000001,"15min":15.128},"min":{"1min":0.69499999999999995,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":62.984000000000002,"5min":281.13400000000001,"15min":281.13400000000001},"last":62.984000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574937,"num_pgs":47,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1059440,"kb_used_data":92646808,"kb_used_omap":15371,"kb_used_meta":0,"kb_avail":92639940,"statfs":{"total":95948165120,"available":94863298560,"internally_reserved":7032832,"allocated":94870331392,"data_stored":94870331392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15740848,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":3,"commit_latency_ns":0,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":5.6779999999999999,"5min":4.7789999999999999,"15min":2.1779999999999999},"min":{"1min":0.32200000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":98.227000000000004,"5min":98.227000000000004,"15min":98.227000000000004},"last":10.975},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.393,"15min":0.93200000000000005},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.82999999999999996,"5min":88.299999999999997,"15min":88.299999999999997},"last":0.56499999999999995}]},{"osd":1,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":10.779999999999999,"5min":7.5599999999999996,"15min":3.7959999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":87.209999999999994,"5min":128.749,"15min":143.071},"last":0.52800000000000002},{"interface":"front","average":{"1min":3.5470000000000002,"5min":2.5289999999999999,"15min":2.9049999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":56.457999999999998,"5min":88.186000000000007,"15min":88.186000000000007},"last":0.67400000000000004}]},{"osd":3,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":7.8970000000000002,"5min":4.4180000000000001,"15min":9.0730000000000004},"min":{"1min":0.35099999999999998,"5min":0.308,"15min":0.29899999999999999},"max":{"1min":78.359999999999999,"5min":86.191999999999993,"15min":183.53},"last":0.77000000000000002},{"interface":"front","average":{"1min":5.7469999999999999,"5min":6.4530000000000003,"15min":3.3439999999999999},"min":{"1min":0.36599999999999999,"5min":0.316,"15min":0.316},"max":{"1min":98.295000000000002,"5min":155.47900000000001,"15min":195.16399999999999},"last":0.621}]},{"osd":4,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":5.9390000000000001,"5min":4.5730000000000004,"15min":4.5679999999999996},"min":{"1min":0.64300000000000002,"5min":0.624,"15min":0.58399999999999996},"max":{"1min":98.375,"5min":98.375,"15min":175.34700000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.84899999999999998,"5min":2.323,"15min":3.911},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.1359999999999999,"5min":88.459999999999994,"15min":175.30500000000001},"last":0.89900000000000002}]},{"osd":5,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":1.23,"15min":1.677},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1100000000000001,"5min":34.256999999999998,"15min":143.262},"last":0.88700000000000001},{"interface":"front","average":{"1min":5.9939999999999998,"5min":4.0090000000000003,"15min":2.1000000000000001},"min":{"1min":0.58599999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":98.427000000000007,"5min":98.427000000000007,"15min":98.427000000000007},"last":0.86599999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.75,"15min":1.292},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.175,"5min":88.542000000000002,"15min":88.542000000000002},"last":0.995},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.3129999999999999,"15min":1.76},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.228,"5min":34.305999999999997,"15min":143.215},"last":0.69499999999999995}]},{"osd":7,"last update":"Sun Jan 15 14:35:12 2023","interfaces":[{"interface":"back","average":{"1min":14.125,"5min":10.615,"15min":5.4370000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":114.056,"5min":176.44499999999999,"15min":176.44499999999999},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.3149999999999999,"15min":2.6669999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.2490000000000001,"5min":34.207999999999998,"15min":143.17099999999999},"last":0.85199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":54,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1014148,"kb_used_data":92687476,"kb_used_omap":16952,"kb_used_meta":0,"kb_avail":92685232,"statfs":{"total":95948165120,"available":94909677568,"internally_reserved":2297856,"allocated":94911975424,"data_stored":94911975424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17358939,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.6840000000000002,"5min":2.4870000000000001,"15min":3.6699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":57.098999999999997,"5min":100.533,"15min":144.12799999999999},"last":0.28499999999999998},{"interface":"front","average":{"1min":6.8630000000000004,"5min":7.8300000000000001,"15min":5.2439999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":101.319,"5min":176.01599999999999,"15min":176.01599999999999},"last":0.61199999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":3.4319999999999999,"15min":1.762},"min":{"1min":0.36299999999999999,"5min":0.25700000000000001,"15min":0.245},"max":{"1min":0.80900000000000005,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68600000000000005,"15min":2.8769999999999998},"min":{"1min":0.372,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.98099999999999998,"5min":3.9780000000000002,"15min":48.119999999999997},"last":0.52900000000000003}]},{"osd":3,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":9.3140000000000001,"5min":7.5700000000000003,"15min":14.470000000000001},"min":{"1min":0.33700000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":125.736,"5min":245.637,"15min":282.01400000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":6.9790000000000001,"5min":9.157,"15min":7.5739999999999998},"min":{"1min":0.40899999999999997,"5min":0.253,"15min":0.253},"max":{"1min":101.401,"5min":176.13900000000001,"15min":176.13900000000001},"last":0.55200000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":19.507000000000001,"15min":9.5679999999999996},"min":{"1min":0.44900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.155,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.81499999999999995,"5min":15.012,"15min":7.5220000000000002},"min":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.28,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.85299999999999998}]},{"osd":5,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":2.367,"15min":1.47},"min":{"1min":0.61299999999999999,"5min":0.56899999999999995,"15min":0.433},"max":{"1min":1.3999999999999999,"5min":69.478999999999999,"15min":69.478999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.95499999999999996,"15min":3.044},"min":{"1min":0.64300000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8049999999999999,"5min":4.0780000000000003,"15min":46.905999999999999},"last":0.66900000000000004}]},{"osd":6,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":7.2590000000000003,"5min":8.1709999999999994,"15min":5.7880000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":101.45999999999999,"5min":176.249,"15min":176.249},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.98099999999999998,"5min":2.4409999999999998,"15min":1.55},"min":{"1min":0.66600000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.6759999999999999,"5min":69.549999999999997,"15min":69.549999999999997},"last":0.78200000000000003}]},{"osd":7,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":7.4199999999999999,"5min":6.4710000000000001,"15min":6.7809999999999997},"min":{"1min":0.58699999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":97.539000000000001,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.75},{"interface":"front","average":{"1min":17.943000000000001,"5min":10.349,"15min":6.367},"min":{"1min":0.55400000000000005,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":193.74600000000001,"5min":193.74600000000001,"15min":193.74600000000001},"last":0.76800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574938,"num_pgs":78,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1156884,"kb_used_data":92542560,"kb_used_omap":12546,"kb_used_meta":0,"kb_avail":92542496,"statfs":{"total":95948165120,"available":94763515904,"internally_reserved":65536,"allocated":94763581440,"data_stored":94763581440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12848062,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":10.833,"5min":4.6390000000000002,"15min":2.2109999999999999},"min":{"1min":0.27100000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":127.86,"5min":127.86,"15min":127.86},"last":0.66600000000000004},{"interface":"front","average":{"1min":7.3220000000000001,"5min":5.681,"15min":2.7040000000000002},"min":{"1min":0.29899999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":100.407,"5min":162.91200000000001,"15min":162.91200000000001},"last":0.69399999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":7.4370000000000003,"5min":3.052,"15min":2.0499999999999998},"min":{"1min":0.38900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":100.58799999999999,"5min":100.58799999999999,"15min":100.58799999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55200000000000005,"5min":1.0900000000000001,"15min":1.339},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28499999999999998},"max":{"1min":0.80600000000000005,"5min":48.893000000000001,"15min":71.406999999999996},"last":155.27500000000001}]},{"osd":3,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":11.68,"5min":9.2279999999999998,"15min":13.448},"min":{"1min":0.35999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":127.803,"5min":199.08000000000001,"15min":284.62799999999999},"last":0.71399999999999997},{"interface":"front","average":{"1min":7.452,"5min":6.2939999999999996,"15min":3.6000000000000001},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.33600000000000002},"max":{"1min":100.371,"5min":162.61600000000001,"15min":174.09399999999999},"last":0.73399999999999999}]},{"osd":4,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":15.289999999999999,"15min":6.1970000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96899999999999997,"5min":1486.309,"15min":1486.309},"last":155.33099999999999},{"interface":"front","average":{"1min":11.106,"5min":18.841999999999999,"15min":7.4020000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":127.61499999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":1.1850000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":7.6619999999999999,"5min":3.2709999999999999,"15min":7.3550000000000004},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":100.46299999999999,"5min":100.46299999999999,"15min":176.34399999999999},"last":1.03},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.92100000000000004,"15min":1.2490000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.391,"5min":2.2959999999999998,"15min":71.563999999999993},"last":155.36600000000001}]},{"osd":6,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":11.205,"5min":5.0679999999999996,"15min":2.7120000000000002},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":127.68000000000001,"5min":127.68000000000001,"15min":127.68000000000001},"last":1.1339999999999999},{"interface":"front","average":{"1min":7.7210000000000001,"5min":3.3650000000000002,"15min":2.1949999999999998},"min":{"1min":0.65200000000000002,"5min":0.64600000000000002,"15min":0.57499999999999996},"max":{"1min":100.544,"5min":100.544,"15min":100.544},"last":0.96799999999999997}]},{"osd":7,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.7789999999999999,"15min":10.553000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4690000000000001,"5min":73.864999999999995,"15min":292.61099999999999},"last":210.99299999999999},{"interface":"front","average":{"1min":17.018999999999998,"5min":6.2489999999999997,"15min":3.1059999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":127.46599999999999,"5min":127.46599999999999,"15min":127.46599999999999},"last":1.224}]}]}],"pool_statfs":[]}} 2023-01-15T14:35:27.721 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg force-backfill 3.10 3.57 3.5a 2023-01-15T14:35:27.884 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:28.022 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:28.023 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:35:28.023 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:27.877+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:28.024 INFO:tasks.rados.rados.0.smithi090.stdout:1640: finishing write tid 1 to smithi09025264-7 2023-01-15T14:35:28.024 INFO:tasks.rados.rados.0.smithi090.stdout:1640: finishing write tid 2 to smithi09025264-7 2023-01-15T14:35:28.024 INFO:tasks.rados.rados.0.smithi090.stdout:1640: finishing write tid 3 to smithi09025264-7 2023-01-15T14:35:28.024 INFO:tasks.rados.rados.0.smithi090.stdout:1640: finishing write tid 4 to smithi09025264-7 2023-01-15T14:35:28.024 INFO:tasks.rados.rados.0.smithi090.stdout:1646: expect (ObjNum 499 snap 149 seq_num 499) 2023-01-15T14:35:28.024 INFO:tasks.rados.rados.0.smithi090.stdout:1640: finishing write tid 5 to smithi09025264-7 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout:1640: oid 7 updating version 0 to 281 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout:1640: oid 7 version 281 is already newer than 280 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout:1640: oid 7 updating version 281 to 282 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout:1640: oid 7 version 282 is already newer than 279 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 282 (ObjNum 586 snap 179 seq_num 586) dirty exists 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout:1640: left oid 7 (ObjNum 586 snap 179 seq_num 586) 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout:1640: done (4 left) 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout:1641: done (3 left) 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout:1646: done (2 left) 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout:1648: snap_create 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:35:28.025 INFO:tasks.rados.rados.0.smithi090.stdout:1647: finishing copy_from to smithi09025264-41 2023-01-15T14:35:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1647: got expected ENOENT (src dne) 2023-01-15T14:35:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1647: finishing copy_from racing read to smithi09025264-41 2023-01-15T14:35:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1647: done (1 left) 2023-01-15T14:35:28.026 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1631: finishing write tid 2 to smithi09025264-31 2023-01-15T14:35:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1631: finishing write tid 3 to smithi09025264-31 2023-01-15T14:35:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1631: finishing write tid 4 to smithi09025264-31 2023-01-15T14:35:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1631: finishing write tid 5 to smithi09025264-31 2023-01-15T14:35:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1631: oid 31 updating version 0 to 425 2023-01-15T14:35:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1631: oid 31 version 425 is already newer than 423 2023-01-15T14:35:28.026 INFO:tasks.rados.rados.0.smithi090.stdout:1631: oid 31 version 425 is already newer than 422 2023-01-15T14:35:28.027 INFO:tasks.rados.rados.0.smithi090.stdout:1631: oid 31 version 425 is already newer than 424 2023-01-15T14:35:28.027 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 31 v 425 (ObjNum 581 snap 179 seq_num 581) dirty exists 2023-01-15T14:35:28.027 INFO:tasks.rados.rados.0.smithi090.stdout:1631: left oid 31 (ObjNum 581 snap 179 seq_num 581) 2023-01-15T14:35:28.027 INFO:tasks.rados.rados.0.smithi090.stdout:1631: done (0 left) 2023-01-15T14:35:28.027 INFO:tasks.rados.rados.0.smithi090.stdout:1648: done (0 left) 2023-01-15T14:35:28.027 INFO:tasks.rados.rados.0.smithi090.stdout:1649: delete oid 24 current snap is 180 2023-01-15T14:35:28.029 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:28.021+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:28.130 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:28.125+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:28.145 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:28.145 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:28.155 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:35:28.221 INFO:tasks.rados.rados.0.smithi090.stdout:1649: done (0 left) 2023-01-15T14:35:28.221 INFO:tasks.rados.rados.0.smithi090.stdout:1650: snap_remove snap 154 2023-01-15T14:35:28.231 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:28.225+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:28.352 INFO:teuthology.orchestra.run.smithi090.stderr:pg 3.10 doesn't require backfilling; pg 3.57 doesn't require backfilling; pg 3.5a doesn't require backfilling; 2023-01-15T14:35:28.352 INFO:teuthology.orchestra.run.smithi090.stderr: 2023-01-15T14:35:28.352 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:28.352 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:28.353 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:35:28.353 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:28.345+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:28.453 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:28.446+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:28.676 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:28.676 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:28.677 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:35:28.677 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:28.670+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:28.778 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:28.773+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:28.836 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:28.836 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:28.836 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:35:28.879 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:28.874+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:28.990 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:28.991 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:28.992 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:28.982+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:28.998 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:35:29.035 INFO:tasks.rados.rados.0.smithi090.stdout:1650: done (0 left) 2023-01-15T14:35:29.175 INFO:tasks.rados.rados.0.smithi090.stdout:1651: read oid 44 snap -1 2023-01-15T14:35:29.175 INFO:tasks.rados.rados.0.smithi090.stdout:1651: expect (ObjNum 575 snap 177 seq_num 575) 2023-01-15T14:35:29.175 INFO:tasks.rados.rados.0.smithi090.stdout:1652: read oid 8 snap 159 2023-01-15T14:35:29.175 INFO:tasks.rados.rados.0.smithi090.stdout:1652: expect deleted 2023-01-15T14:35:29.175 INFO:tasks.rados.rados.0.smithi090.stdout:1653: rollback oid 30 current snap is 180 2023-01-15T14:35:29.175 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 30 to 158 2023-01-15T14:35:29.176 INFO:tasks.rados.rados.0.smithi090.stdout:1654: snap_remove snap 176 2023-01-15T14:35:29.176 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:35:29.178 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:29.180 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:29.181 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:35:29.181 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:29.170+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:29.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:35:29.322 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:29.314+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:29.323 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:29.323 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:29.323 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:29.324 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:35:29.369 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:29.479 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:29.479 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:29.479 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:29.479 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:29.479 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:29.480 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:35:29.480 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:29.473+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:29.484 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:29.518 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:29.640 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:29.642 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:35:29.642 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:35:29.642 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:35:29.642 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:35:29.642 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:35:29.643 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:35:29.643 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:35:29.643 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:35:29.643 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:35:29.643 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:35:29.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:35:29.643 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:35:29.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.643 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1087 3.f 3:fd983206:::smithi09025264-7:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:35:29.656 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:35:29.656 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 346.829291614, 2023-01-15T14:35:29.656 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0859964360000001, 2023-01-15T14:35:29.657 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:29.657 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:29.657 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:29.659 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:29.659 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:29.659 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1087 2023-01-15T14:35:29.659 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.659 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:29.659 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.659 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:29.659 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:35:29.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:29.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.660 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:29.660 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:35:29.660 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:29.660 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.660 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.660 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:29.660 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:35:29.661 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:29.661 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.661 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:29.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758724+0000", 2023-01-15T14:35:29.664 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7740000000000004e-06 2023-01-15T14:35:29.664 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.664 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.664 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:29.664 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758738+0000", 2023-01-15T14:35:29.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.33e-05 2023-01-15T14:35:29.665 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.665 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.665 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:29.665 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758744+0000", 2023-01-15T14:35:29.665 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2859999999999996e-06 2023-01-15T14:35:29.675 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.675 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.675 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:29.675 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848121+0000", 2023-01-15T14:35:29.675 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089377097000000003 2023-01-15T14:35:29.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.676 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:29.676 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848127+0000", 2023-01-15T14:35:29.676 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3500000000000002e-06 2023-01-15T14:35:29.676 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.676 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.677 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:29.677 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952454+0000", 2023-01-15T14:35:29.677 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104326167 2023-01-15T14:35:29.677 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.677 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.677 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:29.677 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952520+0000", 2023-01-15T14:35:29.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6166000000000003e-05 2023-01-15T14:35:29.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:35:29.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.063003+0000", 2023-01-15T14:35:29.678 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.110482916 2023-01-15T14:35:29.678 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:29.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065265+0000", 2023-01-15T14:35:29.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022626370000000001 2023-01-15T14:35:29.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:29.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065291+0000", 2023-01-15T14:35:29.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6095e-05 2023-01-15T14:35:29.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.681 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:29.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.066153+0000", 2023-01-15T14:35:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086127100000000004 2023-01-15T14:35:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166533+0000", 2023-01-15T14:35:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100380177 2023-01-15T14:35:29.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166646+0000", 2023-01-15T14:35:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112717 2023-01-15T14:35:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844498+0000", 2023-01-15T14:35:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67785237399999998 2023-01-15T14:35:29.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844605+0000", 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106691 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844717+0000", 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011203499999999999 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:29.805 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1088 3.f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:35:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:35:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 346.82923594800002, 2023-01-15T14:35:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.086312696, 2023-01-15T14:35:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:29.806 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1088 2023-01-15T14:35:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:35:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:29.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758775+0000", 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758786+0000", 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.076e-05 2023-01-15T14:35:29.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:29.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758809+0000", 2023-01-15T14:35:29.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3706999999999999e-05 2023-01-15T14:35:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758821+0000", 2023-01-15T14:35:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1976e-05 2023-01-15T14:35:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878534+0000", 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.119712966 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878540+0000", 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0680000000000002e-06 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952693+0000", 2023-01-15T14:35:29.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.074153067000000003 2023-01-15T14:35:29.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:29.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952735+0000", 2023-01-15T14:35:29.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1792999999999998e-05 2023-01-15T14:35:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065518+0000", 2023-01-15T14:35:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.11278334499999999 2023-01-15T14:35:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:29.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065687+0000", 2023-01-15T14:35:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016807700000000001 2023-01-15T14:35:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844786+0000", 2023-01-15T14:35:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.77909980300000004 2023-01-15T14:35:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844944+0000", 2023-01-15T14:35:29.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157424 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.845089+0000", 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144814 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:16 3.13 3:ca393a9e:::smithi09025264-47:head [assert-version v193,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:29.822 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 310.282258523, 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8970339479999998, 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:29.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:35:29.831 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305751+0000", 2023-01-15T14:35:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305763+0000", 2023-01-15T14:35:29.832 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2309e-05 2023-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305788+0000", 2023-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4388000000000001e-05 2023-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305799+0000", 2023-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1304000000000001e-05 2023-01-15T14:35:29.833 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196425+0000", 2023-01-15T14:35:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8906259830000001 2023-01-15T14:35:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196673+0000", 2023-01-15T14:35:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024747499999999998 2023-01-15T14:35:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.834 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.202787+0000", 2023-01-15T14:35:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0061146509999999996 2023-01-15T14:35:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1480 3.13 3:cbfd2d71:::smithi09025264-37:head [write 581575~705262,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:35:29.835 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 310.248990795, 2023-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.375425221, 2023-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1480 2023-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.836 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.837 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339019+0000", 2023-01-15T14:35:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339659+0000", 2023-01-15T14:35:29.838 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000639742 2023-01-15T14:35:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339690+0000", 2023-01-15T14:35:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1594000000000003e-05 2023-01-15T14:35:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339703+0000", 2023-01-15T14:35:29.839 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2828e-05 2023-01-15T14:35:29.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.962 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.962 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:29.962 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.510837+0000", 2023-01-15T14:35:29.962 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1711338759999999 2023-01-15T14:35:29.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:29.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511048+0000", 2023-01-15T14:35:29.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021057899999999999 2023-01-15T14:35:29.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.963 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:29.963 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511325+0000", 2023-01-15T14:35:29.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027690400000000002 2023-01-15T14:35:29.964 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.964 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.964 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:29.964 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512092+0000", 2023-01-15T14:35:29.964 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00076720800000000002 2023-01-15T14:35:29.964 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.964 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.964 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:29.964 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512113+0000", 2023-01-15T14:35:29.964 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0917e-05 2023-01-15T14:35:29.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:29.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512439+0000", 2023-01-15T14:35:29.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032636299999999998 2023-01-15T14:35:29.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:29.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512871+0000", 2023-01-15T14:35:29.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000432087 2023-01-15T14:35:29.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:29.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.514165+0000", 2023-01-15T14:35:29.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001293726 2023-01-15T14:35:29.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:29.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714137+0000", 2023-01-15T14:35:29.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19997229499999999 2023-01-15T14:35:29.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.970 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:29.970 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714233+0000", 2023-01-15T14:35:29.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6062000000000003e-05 2023-01-15T14:35:29.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.971 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:29.986 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714446+0000", 2023-01-15T14:35:29.986 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212952 2023-01-15T14:35:29.986 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:29.986 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:29.986 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:29.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.986 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1481 3.13 3:cbfd2d71:::smithi09025264-37:head [write 1733272~707409] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:29.986 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:35:29.986 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 310.24214151199999, 2023-01-15T14:35:29.987 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.572522502, 2023-01-15T14:35:29.987 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:29.987 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:29.987 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:29.987 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:29.987 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:29.987 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1481 2023-01-15T14:35:29.987 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.987 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:29.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:29.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:35:29.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:35:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:29.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345868+0000", 2023-01-15T14:35:29.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:29.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:29.999 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:29.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:30.118 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346316+0000", 2023-01-15T14:35:30.118 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044813399999999999 2023-01-15T14:35:30.118 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.118 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.119 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:30.119 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346348+0000", 2023-01-15T14:35:30.119 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:35:30.119 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.119 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.119 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:30.119 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346362+0000", 2023-01-15T14:35:30.119 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.375e-05 2023-01-15T14:35:30.119 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.119 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512276+0000", 2023-01-15T14:35:30.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1659136829999999 2023-01-15T14:35:30.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:30.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512427+0000", 2023-01-15T14:35:30.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015088699999999999 2023-01-15T14:35:30.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714565+0000", 2023-01-15T14:35:30.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20213841599999999 2023-01-15T14:35:30.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:30.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714732+0000", 2023-01-15T14:35:30.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016676700000000001 2023-01-15T14:35:30.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:30.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714964+0000", 2023-01-15T14:35:30.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000231678 2023-01-15T14:35:30.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:30.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715499+0000", 2023-01-15T14:35:30.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053513300000000003 2023-01-15T14:35:30.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:30.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715523+0000", 2023-01-15T14:35:30.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4266000000000001e-05 2023-01-15T14:35:30.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:30.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715867+0000", 2023-01-15T14:35:30.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034419399999999998 2023-01-15T14:35:30.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:30.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.816932+0000", 2023-01-15T14:35:30.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10106498999999999 2023-01-15T14:35:30.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:30.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906374+0000", 2023-01-15T14:35:30.124 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089441442999999995 2023-01-15T14:35:30.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:30.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918050+0000", 2023-01-15T14:35:30.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.011676677999999999 2023-01-15T14:35:30.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:30.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918218+0000", 2023-01-15T14:35:30.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016802000000000001 2023-01-15T14:35:30.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.126 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:30.126 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918393+0000", 2023-01-15T14:35:30.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017447 2023-01-15T14:35:30.126 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.127 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:30.127 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.127 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1482 3.13 3:cbfd2d71:::smithi09025264-37:head [write 3195938~437497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:30.127 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:35:30.127 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 310.23570565400001, 2023-01-15T14:35:30.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5664780289999998, 2023-01-15T14:35:30.127 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:30.127 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:30.127 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:30.128 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:30.128 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:30.128 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1482 2023-01-15T14:35:30.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.128 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:30.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:30.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:35:30.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352275+0000", 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999695 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:30.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352644+0000", 2023-01-15T14:35:30.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036870300000000001 2023-01-15T14:35:30.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:30.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352676+0000", 2023-01-15T14:35:30.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1726000000000001e-05 2023-01-15T14:35:30.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:30.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352689+0000", 2023-01-15T14:35:30.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3617e-05 2023-01-15T14:35:30.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.131 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512602+0000", 2023-01-15T14:35:30.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1599126690000001 2023-01-15T14:35:30.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:30.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512839+0000", 2023-01-15T14:35:30.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023676900000000001 2023-01-15T14:35:30.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715649+0000", 2023-01-15T14:35:30.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20281044600000001 2023-01-15T14:35:30.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:30.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715777+0000", 2023-01-15T14:35:30.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012738800000000001 2023-01-15T14:35:30.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:30.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715995+0000", 2023-01-15T14:35:30.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000217858 2023-01-15T14:35:30.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:30.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716694+0000", 2023-01-15T14:35:30.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00069956700000000001 2023-01-15T14:35:30.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716709+0000", 2023-01-15T14:35:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4362000000000001e-05 2023-01-15T14:35:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716936+0000", 2023-01-15T14:35:30.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022744200000000001 2023-01-15T14:35:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.818876+0000", 2023-01-15T14:35:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101939635 2023-01-15T14:35:30.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906390+0000", 2023-01-15T14:35:30.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087513987000000001 2023-01-15T14:35:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918577+0000", 2023-01-15T14:35:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012187066 2023-01-15T14:35:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918697+0000", 2023-01-15T14:35:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120326 2023-01-15T14:35:30.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918784+0000", 2023-01-15T14:35:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7200999999999994e-05 2023-01-15T14:35:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1483 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3781575] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:30.160 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:35:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 310.23529750199998, 2023-01-15T14:35:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5663522429999999, 2023-01-15T14:35:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1483 2023-01-15T14:35:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:30.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.162 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:30.162 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:35:30.162 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.283 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.284 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:30.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:35:30.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.284 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.284 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:30.284 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352713+0000", 2023-01-15T14:35:30.284 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:30.284 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:30.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352720+0000", 2023-01-15T14:35:30.285 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7959999999999998e-06 2023-01-15T14:35:30.285 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.285 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.285 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:30.285 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352735+0000", 2023-01-15T14:35:30.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.455e-05 2023-01-15T14:35:30.286 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.286 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.286 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:30.286 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352741+0000", 2023-01-15T14:35:30.286 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8360000000000003e-06 2023-01-15T14:35:30.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512901+0000", 2023-01-15T14:35:30.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1601594980000001 2023-01-15T14:35:30.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:30.293 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.513067+0000", 2023-01-15T14:35:30.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016626 2023-01-15T14:35:30.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.294 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.294 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.295 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716977+0000", 2023-01-15T14:35:30.295 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20391002799999999 2023-01-15T14:35:30.295 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.295 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:30.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717146+0000", 2023-01-15T14:35:30.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168851 2023-01-15T14:35:30.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:30.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717467+0000", 2023-01-15T14:35:30.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032135600000000002 2023-01-15T14:35:30.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:30.299 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717800+0000", 2023-01-15T14:35:30.299 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033217400000000002 2023-01-15T14:35:30.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.300 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:30.300 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717822+0000", 2023-01-15T14:35:30.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2308999999999999e-05 2023-01-15T14:35:30.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:30.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717970+0000", 2023-01-15T14:35:30.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014788099999999999 2023-01-15T14:35:30.301 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.301 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.301 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:30.301 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.819135+0000", 2023-01-15T14:35:30.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116478 2023-01-15T14:35:30.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:30.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906408+0000", 2023-01-15T14:35:30.302 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087272902999999999 2023-01-15T14:35:30.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.302 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.302 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:30.302 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918887+0000", 2023-01-15T14:35:30.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012479965000000001 2023-01-15T14:35:30.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:30.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918959+0000", 2023-01-15T14:35:30.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1978999999999996e-05 2023-01-15T14:35:30.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:30.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.919067+0000", 2023-01-15T14:35:30.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107083 2023-01-15T14:35:30.304 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.304 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:30.304 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.318 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1485 3.d 3:bc1c6b44:::smithi09025264-23:head [write 598382~780511,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:35:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 310.22782706800001, 2023-01-15T14:35:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5474121890000001, 2023-01-15T14:35:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1485 2023-01-15T14:35:30.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:35:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:35:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360168+0000", 2023-01-15T14:35:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999828 2023-01-15T14:35:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360790+0000", 2023-01-15T14:35:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062269099999999998 2023-01-15T14:35:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360822+0000", 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1903000000000003e-05 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360836+0000", 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3380000000000001e-05 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096664+0000", 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7358282320000002 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096683+0000", 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9063999999999999e-05 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704561+0000", 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60787808399999999 2023-01-15T14:35:30.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704846+0000", 2023-01-15T14:35:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028527400000000002 2023-01-15T14:35:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705135+0000", 2023-01-15T14:35:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028875400000000001 2023-01-15T14:35:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705735+0000", 2023-01-15T14:35:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059956899999999999 2023-01-15T14:35:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705761+0000", 2023-01-15T14:35:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:35:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.325 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.706129+0000", 2023-01-15T14:35:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036822299999999999 2023-01-15T14:35:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906331+0000", 2023-01-15T14:35:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200202091 2023-01-15T14:35:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:30.326 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907111+0000", 2023-01-15T14:35:30.327 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077990799999999995 2023-01-15T14:35:30.327 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.327 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907379+0000", 2023-01-15T14:35:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026804800000000001 2023-01-15T14:35:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.552 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907529+0000", 2023-01-15T14:35:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015022700000000001 2023-01-15T14:35:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907597+0000", 2023-01-15T14:35:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7785000000000002e-05 2023-01-15T14:35:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.553 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1486 3.d 3:bc1c6b44:::smithi09025264-23:head [write 2075401~451067] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:35:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 310.22143102000001, 2023-01-15T14:35:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6461318330000001, 2023-01-15T14:35:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:30.554 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1486 2023-01-15T14:35:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:30.556 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:35:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:35:30.557 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.559 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366579+0000", 2023-01-15T14:35:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366845+0000", 2023-01-15T14:35:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026610499999999999 2023-01-15T14:35:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.560 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366878+0000", 2023-01-15T14:35:30.561 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2750000000000003e-05 2023-01-15T14:35:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.564 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.467009+0000", 2023-01-15T14:35:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100131071 2023-01-15T14:35:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096793+0000", 2023-01-15T14:35:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6297846869999999 2023-01-15T14:35:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.565 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:35:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096806+0000", 2023-01-15T14:35:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3077e-05 2023-01-15T14:35:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907957+0000", 2023-01-15T14:35:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81115055400000002 2023-01-15T14:35:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.566 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908229+0000", 2023-01-15T14:35:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027187499999999998 2023-01-15T14:35:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908496+0000", 2023-01-15T14:35:30.567 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026754200000000001 2023-01-15T14:35:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.568 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909025+0000", 2023-01-15T14:35:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052865200000000001 2023-01-15T14:35:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909046+0000", 2023-01-15T14:35:30.573 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0713e-05 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909294+0000", 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024864899999999998 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.911607+0000", 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023128939999999998 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.574 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012076+0000", 2023-01-15T14:35:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100468293 2023-01-15T14:35:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012426+0000", 2023-01-15T14:35:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035044299999999998 2023-01-15T14:35:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.575 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012532+0000", 2023-01-15T14:35:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106142 2023-01-15T14:35:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012713+0000", 2023-01-15T14:35:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018048599999999999 2023-01-15T14:35:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.576 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1496 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 309.85178817100001, 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870616037, 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1496 2023-01-15T14:35:30.577 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:35:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.578 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736220+0000", 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.579 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736238+0000", 2023-01-15T14:35:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7156e-05 2023-01-15T14:35:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736275+0000", 2023-01-15T14:35:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7876000000000001e-05 2023-01-15T14:35:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.580 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736290+0000", 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4449e-05 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602217+0000", 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8659270299999999 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602468+0000", 2023-01-15T14:35:30.581 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000251315 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.606840+0000", 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0043715009999999999 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1497 3.11 3:8889e450:::smithi09025264-32:head [read 0~2991646,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:30.582 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:35:30.583 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 309.85170071700003, 2023-01-15T14:35:30.583 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738359249999998, 2023-01-15T14:35:30.583 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:30.583 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:30.712 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:30.712 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:30.712 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:30.712 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1497 2023-01-15T14:35:30.712 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.712 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:30.712 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.713 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:30.713 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:35:30.713 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.713 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.713 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:35:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736310+0000", 2023-01-15T14:35:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.714 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736317+0000", 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7240000000000003e-06 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736334+0000", 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6878999999999999e-05 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:30.715 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736341+0000", 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0129999999999996e-06 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607070+0000", 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870728873 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607226+0000", 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015595 2023-01-15T14:35:30.716 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610147+0000", 2023-01-15T14:35:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.002921416 2023-01-15T14:35:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.717 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1498 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 309.851660611, 2023-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8752667839999999, 2023-01-15T14:35:30.723 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1498 2023-01-15T14:35:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.724 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:35:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:35:30.725 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736350+0000", 2023-01-15T14:35:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.726 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.727 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:30.727 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736355+0000", 2023-01-15T14:35:30.727 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4900000000000002e-06 2023-01-15T14:35:30.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.729 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.729 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:30.729 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736363+0000", 2023-01-15T14:35:30.729 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2770000000000005e-06 2023-01-15T14:35:30.729 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.731 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:30.731 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736369+0000", 2023-01-15T14:35:30.731 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7400000000000001e-06 2023-01-15T14:35:30.731 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.731 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610234+0000", 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738651439999998 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610402+0000", 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168283 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:30.732 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.611618+0000", 2023-01-15T14:35:30.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001215647 2023-01-15T14:35:30.733 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.733 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:30.733 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.733 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.733 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.733 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.7.0:440 3.11 e231/207 ondisk, result = 0)", 2023-01-15T14:35:30.733 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:35:30.733 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 211.343878953, 2023-01-15T14:35:30.733 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973962139999998, 2023-01-15T14:35:30.734 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:30.734 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:30.734 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:30.734 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.734 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:30.734 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:35:30.735 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.735 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:30.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:35:30.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.737 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:30.737 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244132+0000", 2023-01-15T14:35:30.737 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:30.737 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.737 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:30.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244143+0000", 2023-01-15T14:35:30.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1647e-05 2023-01-15T14:35:30.738 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:30.738 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244165+0000", 2023-01-15T14:35:30.738 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1501999999999998e-05 2023-01-15T14:35:30.738 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.738 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.738 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244180+0000", 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5434000000000002e-05 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941489+0000", 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973091899999999 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941503+0000", 2023-01-15T14:35:30.739 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.414e-05 2023-01-15T14:35:30.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.740 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:30.740 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941529+0000", 2023-01-15T14:35:30.740 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5622999999999998e-05 2023-01-15T14:35:30.740 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.740 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:30.740 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.740 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.740 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.740 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.4.0:562 3.1b e231/224 3:d85f42d9:::smithi09025264-45:a8 v 231'66, mlcod=225'62)", 2023-01-15T14:35:30.741 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:35:30.741 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 211.15352775700001, 2023-01-15T14:35:30.741 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2578820149999999, 2023-01-15T14:35:30.741 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:30.741 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:30.741 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:30.982 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.982 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:30.982 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:35:30.982 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.982 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.982 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:30.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:35:30.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.983 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:30.983 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:35:30.983 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:30.983 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.983 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.983 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:30.984 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434492+0000", 2023-01-15T14:35:30.984 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4360000000000002e-06 2023-01-15T14:35:30.984 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.984 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.984 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:30.984 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434517+0000", 2023-01-15T14:35:30.984 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.48e-05 2023-01-15T14:35:30.984 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.984 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.984 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:30.984 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434530+0000", 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2982999999999999e-05 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434663+0000", 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013307 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434700+0000", 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7020000000000001e-05 2023-01-15T14:35:30.985 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435438+0000", 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073808900000000004 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435456+0000", 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8091999999999999e-05 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:30.986 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435526+0000", 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9806000000000002e-05 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435570+0000", 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4091999999999997e-05 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.692366+0000", 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2567961539999999 2023-01-15T14:35:30.987 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.988 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:30.988 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.988 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.988 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.988 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:24 3.d 3:b3f111b5:::smithi09025264-48:head [assert-version v160,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e232)", 2023-01-15T14:35:30.988 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:35:30.988 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 210.63769154100001, 2023-01-15T14:35:30.988 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9963031440000001, 2023-01-15T14:35:30.988 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:30.988 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:30.988 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.989 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:30.990 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950318+0000", 2023-01-15T14:35:30.990 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:30.993 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.993 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.993 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:30.994 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950333+0000", 2023-01-15T14:35:30.994 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.467e-05 2023-01-15T14:35:30.994 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.994 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.994 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:30.994 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950368+0000", 2023-01-15T14:35:30.994 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4895999999999999e-05 2023-01-15T14:35:30.994 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050461+0000", 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10009304400000001 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050593+0000", 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000132764 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.995 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:35:30.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050615+0000", 2023-01-15T14:35:30.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1483000000000001e-05 2023-01-15T14:35:30.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:30.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945049+0000", 2023-01-15T14:35:30.996 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.894434138 2023-01-15T14:35:30.996 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.996 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.996 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:30.996 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945326+0000", 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000277368 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.946623+0000", 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012970340000000001 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.997 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.3.0:547 3.22 e232/180 3:47dfe935:::smithi09025264-29:81 v 232'124, mlcod=229'121)", 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 210.53746179300001, 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.643153431, 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:30.998 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:30.999 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:30.999 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:35:30.999 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:30.999 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:31.001 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050549+0000", 2023-01-15T14:35:31.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:31.001 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.001 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.001 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:31.001 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050561+0000", 2023-01-15T14:35:31.001 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.278e-05 2023-01-15T14:35:31.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:31.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050583+0000", 2023-01-15T14:35:31.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1053e-05 2023-01-15T14:35:31.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.002 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.002 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:31.002 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050596+0000", 2023-01-15T14:35:31.002 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3946e-05 2023-01-15T14:35:31.002 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:31.003 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050708+0000", 2023-01-15T14:35:31.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011129 2023-01-15T14:35:31.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.003 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:31.003 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050750+0000", 2023-01-15T14:35:31.003 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2725000000000002e-05 2023-01-15T14:35:31.003 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.003 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.004 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:31.004 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051324+0000", 2023-01-15T14:35:31.004 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00057335700000000003 2023-01-15T14:35:31.004 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.004 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:31.004 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051340+0000", 2023-01-15T14:35:31.004 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5968999999999999e-05 2023-01-15T14:35:31.004 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.004 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.004 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051405+0000", 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5419000000000005e-05 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051471+0000", 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5671000000000002e-05 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.693703+0000", 2023-01-15T14:35:31.005 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.642232554 2023-01-15T14:35:31.006 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:31.006 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:31.006 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:31.006 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.006 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.006 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3881 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e392)", 2023-01-15T14:35:31.006 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:34:42.155799+0000", 2023-01-15T14:35:31.006 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 47.432212972000002, 2023-01-15T14:35:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1132586929999999, 2023-01-15T14:35:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:31.146 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3881 2023-01-15T14:35:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155799+0000", 2023-01-15T14:35:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.147 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155799+0000", 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155798+0000", 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155804+0000", 2023-01-15T14:35:31.148 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4969999999999998e-06 2023-01-15T14:35:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155820+0000", 2023-01-15T14:35:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5387999999999999e-05 2023-01-15T14:35:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155827+0000", 2023-01-15T14:35:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9759999999999998e-06 2023-01-15T14:35:31.149 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.651043+0000", 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49521586499999998 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.651233+0000", 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000190259 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.150 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:43.268845+0000", 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61761231400000005 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:43.268939+0000", 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4122000000000004e-05 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:43.269058+0000", 2023-01-15T14:35:31.151 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011818400000000001 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4009 3.53 3:ca393a9e:::smithi09025264-47:head [write 2715800~423063] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:35:01.016400+0000", 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 28.571612135999999, 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.99314385299999997, 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:31.152 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4009 2023-01-15T14:35:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016400+0000", 2023-01-15T14:35:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:31.153 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016400+0000", 2023-01-15T14:35:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016399+0000", 2023-01-15T14:35:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.154 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016791+0000", 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039251500000000002 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016804+0000", 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2826000000000001e-05 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:31.155 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016812+0000", 2023-01-15T14:35:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.6820000000000003e-06 2023-01-15T14:35:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.202478+0000", 2023-01-15T14:35:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18566589 2023-01-15T14:35:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.202736+0000", 2023-01-15T14:35:31.156 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025787800000000001 2023-01-15T14:35:31.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:31.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.805145+0000", 2023-01-15T14:35:31.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60240954099999999 2023-01-15T14:35:31.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.157 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.157 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:31.157 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.805308+0000", 2023-01-15T14:35:31.157 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016293699999999999 2023-01-15T14:35:31.157 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:35:31.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.805592+0000", 2023-01-15T14:35:31.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028417399999999999 2023-01-15T14:35:31.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.158 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:31.158 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.806260+0000", 2023-01-15T14:35:31.158 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00066786899999999997 2023-01-15T14:35:31.158 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.158 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:31.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.806281+0000", 2023-01-15T14:35:31.159 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1345999999999999e-05 2023-01-15T14:35:31.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.159 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:31.159 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.806523+0000", 2023-01-15T14:35:31.159 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024123800000000001 2023-01-15T14:35:31.159 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.159 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.160 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:31.160 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.807381+0000", 2023-01-15T14:35:31.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00085809500000000002 2023-01-15T14:35:31.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.160 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.160 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:31.160 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:02.008831+0000", 2023-01-15T14:35:31.160 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201450504 2023-01-15T14:35:31.160 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.161 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:31.161 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:02.009215+0000", 2023-01-15T14:35:31.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038404800000000001 2023-01-15T14:35:31.161 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.161 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:31.161 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:02.009416+0000", 2023-01-15T14:35:31.161 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020065299999999999 2023-01-15T14:35:31.161 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.161 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.162 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:31.162 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:02.009544+0000", 2023-01-15T14:35:31.162 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012751700000000001 2023-01-15T14:35:31.162 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:31.162 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:31.162 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:31.162 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.162 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.162 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4011 3.53 3:ca393a9e:::smithi09025264-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e411)", 2023-01-15T14:35:31.162 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:35:01.016838+0000", 2023-01-15T14:35:31.162 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 28.571173466000001, 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.997305671, 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4011 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:31.163 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016838+0000", 2023-01-15T14:35:31.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:31.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:31.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016838+0000", 2023-01-15T14:35:31.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:31.164 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.164 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.164 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:31.164 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016838+0000", 2023-01-15T14:35:31.164 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016840+0000", 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4949999999999998e-06 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016850+0000", 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5189999999999998e-06 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.165 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.016854+0000", 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3189999999999996e-06 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.203039+0000", 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.18618500900000001 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:31.166 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.203208+0000", 2023-01-15T14:35:31.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168763 2023-01-15T14:35:31.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:31.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.808007+0000", 2023-01-15T14:35:31.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60479915399999995 2023-01-15T14:35:31.167 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.167 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.167 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:31.167 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:01.808170+0000", 2023-01-15T14:35:31.167 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000162466 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:02.013874+0000", 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20570471300000001 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:02.014004+0000", 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012986399999999999 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.168 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:02.014144+0000", 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013977999999999999 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4147 3.11 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e423)", 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:35:13.102895+0000", 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 16.485116503, 2023-01-15T14:35:31.169 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.018866324, 2023-01-15T14:35:31.318 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:31.318 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:31.318 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:31.318 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:31.318 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:31.318 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4147 2023-01-15T14:35:31.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.318 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102895+0000", 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102895+0000", 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:31.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102895+0000", 2023-01-15T14:35:31.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:31.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:31.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102899+0000", 2023-01-15T14:35:31.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1300000000000003e-06 2023-01-15T14:35:31.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:31.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102911+0000", 2023-01-15T14:35:31.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2096000000000001e-05 2023-01-15T14:35:31.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:31.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102917+0000", 2023-01-15T14:35:31.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8470000000000001e-06 2023-01-15T14:35:31.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:31.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102983+0000", 2023-01-15T14:35:31.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6594000000000006e-05 2023-01-15T14:35:31.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:35:31.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102996+0000", 2023-01-15T14:35:31.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2459999999999999e-05 2023-01-15T14:35:31.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:31.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.823282+0000", 2023-01-15T14:35:31.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.72028614400000002 2023-01-15T14:35:31.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:31.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.823457+0000", 2023-01-15T14:35:31.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017553299999999999 2023-01-15T14:35:31.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:31.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:14.121348+0000", 2023-01-15T14:35:31.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.297891085 2023-01-15T14:35:31.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:31.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:14.121531+0000", 2023-01-15T14:35:31.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000182086 2023-01-15T14:35:31.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:31.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:31.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:31.325 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:14.121762+0000", 2023-01-15T14:35:31.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023108199999999999 2023-01-15T14:35:31.325 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:31.325 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:31.325 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:31.325 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:31.325 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:31.325 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:35:31.326 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:29.633+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:31.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:35:31.327 INFO:tasks.rados.rados.0.smithi090.stdout:1654: done (3 left) 2023-01-15T14:35:31.327 INFO:tasks.rados.rados.0.smithi090.stdout:1655: write (excl) oid 13 current snap is 180 2023-01-15T14:35:31.327 INFO:tasks.rados.rados.0.smithi090.stdout:1655: seq_num 590 ranges {716130=556091,1792376=573530,2769448=460017,3821956=94174} 2023-01-15T14:35:31.327 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:35:31.328 INFO:tasks.rados.rados.0.smithi090.stdout:1655: writing smithi09025264-13 from 716130 to 1272221 tid 1 2023-01-15T14:35:31.328 INFO:tasks.rados.rados.0.smithi090.stdout:1655: writing smithi09025264-13 from 1792376 to 2365906 tid 2 2023-01-15T14:35:31.332 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:31.321+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:31.333 INFO:tasks.rados.rados.0.smithi090.stdout:1655: writing smithi09025264-13 from 2769448 to 3229465 tid 3 2023-01-15T14:35:31.333 INFO:tasks.rados.rados.0.smithi090.stdout:1655: writing smithi09025264-13 from 3821956 to 3916130 tid 4 2023-01-15T14:35:31.333 INFO:tasks.rados.rados.0.smithi090.stdout:1656: delete oid 3 current snap is 180 2023-01-15T14:35:31.333 INFO:tasks.rados.rados.0.smithi090.stdout:1653: finishing rollback tid 0 to smithi09025264-30 2023-01-15T14:35:31.333 INFO:tasks.rados.rados.0.smithi090.stdout:1653: finishing rollback tid 1 to smithi09025264-30 2023-01-15T14:35:31.333 INFO:tasks.rados.rados.0.smithi090.stdout:1653: finishing rollback tid 2 to smithi09025264-30 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 251 (ObjNum 449 snap 125 seq_num 449) dirty exists 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:1651: expect (ObjNum 575 snap 177 seq_num 575) 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:1651: done (4 left) 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:1652: done (3 left) 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:1653: done (2 left) 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:1656: done (1 left) 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:1657: rollback oid 3 current snap is 180 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 3 to 146 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:1658: write oid 18 current snap is 180 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:1658: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:1658: writing smithi09025264-18 from 749744 to 1456333 tid 1 2023-01-15T14:35:31.334 INFO:tasks.rados.rados.0.smithi090.stdout:1658: writing smithi09025264-18 from 1909340 to 2486539 tid 2 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:1658: writing smithi09025264-18 from 3224294 to 3706855 tid 3 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:1659: rollback oid 2 current snap is 180 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 2 to 159 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:1655: finishing write tid 1 to smithi09025264-13 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:1658: finishing write tid 1 to smithi09025264-18 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:1660: rollback oid 40 current snap is 180 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 40 to 163 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:1661: delete oid 32 current snap is 180 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:1660: finishing rollback tid 1 to smithi09025264-40 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 314 (ObjNum 97 snap 0 seq_num 3190872288) dirty dne 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:1655: finishing write tid 2 to smithi09025264-13 2023-01-15T14:35:31.335 INFO:tasks.rados.rados.0.smithi090.stdout:1655: finishing write tid 3 to smithi09025264-13 2023-01-15T14:35:31.336 INFO:tasks.rados.rados.0.smithi090.stdout:1655: finishing write tid 4 to smithi09025264-13 2023-01-15T14:35:31.728 INFO:tasks.rados.rados.0.smithi090.stdout:1659: finishing rollback tid 1 to smithi09025264-2 2023-01-15T14:35:31.728 INFO:tasks.rados.rados.0.smithi090.stdout:1657: finishing rollback tid 1 to smithi09025264-3 2023-01-15T14:35:31.728 INFO:tasks.rados.rados.0.smithi090.stdout:1657: finishing rollback tid 2 to smithi09025264-3 2023-01-15T14:35:31.728 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 3 v 422 (ObjNum 464 snap 133 seq_num 464) dirty exists 2023-01-15T14:35:31.728 INFO:tasks.rados.rados.0.smithi090.stdout:1657: done (5 left) 2023-01-15T14:35:31.728 INFO:tasks.rados.rados.0.smithi090.stdout:1660: done (4 left) 2023-01-15T14:35:31.728 INFO:tasks.rados.rados.0.smithi090.stdout:1661: done (3 left) 2023-01-15T14:35:31.728 INFO:tasks.rados.rados.0.smithi090.stdout:1662: snap_remove snap 111 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:1662: done (3 left) 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:1663: rollback oid 37 current snap is 180 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 37 to 126 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:1655: finishing write tid 5 to smithi09025264-13 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:1655: finishing write tid 6 to smithi09025264-13 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:1655: oid 13 updating version 0 to 377 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:1655: oid 13 version 377 is already newer than 376 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:1655: oid 13 updating version 377 to 380 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:1655: oid 13 version 380 is already newer than 379 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:1655: oid 13 version 380 is already newer than 378 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 380 (ObjNum 590 snap 180 seq_num 590) dirty exists 2023-01-15T14:35:31.729 INFO:tasks.rados.rados.0.smithi090.stdout:1655: left oid 13 (ObjNum 590 snap 180 seq_num 590) 2023-01-15T14:35:31.730 INFO:tasks.rados.rados.0.smithi090.stdout:1659: finishing rollback tid 2 to smithi09025264-2 2023-01-15T14:35:31.730 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 395 (ObjNum 522 snap 158 seq_num 522) dirty exists 2023-01-15T14:35:31.730 INFO:tasks.rados.rados.0.smithi090.stdout:1658: finishing write tid 2 to smithi09025264-18 2023-01-15T14:35:31.730 INFO:tasks.rados.rados.0.smithi090.stdout:1658: finishing write tid 3 to smithi09025264-18 2023-01-15T14:35:31.730 INFO:tasks.rados.rados.0.smithi090.stdout:1658: finishing write tid 4 to smithi09025264-18 2023-01-15T14:35:31.730 INFO:tasks.rados.rados.0.smithi090.stdout:1658: finishing write tid 5 to smithi09025264-18 2023-01-15T14:35:31.730 INFO:tasks.rados.rados.0.smithi090.stdout:1658: oid 18 updating version 0 to 283 2023-01-15T14:35:31.730 INFO:tasks.rados.rados.0.smithi090.stdout:1658: oid 18 updating version 283 to 286 2023-01-15T14:35:31.730 INFO:tasks.rados.rados.0.smithi090.stdout:1658: oid 18 version 286 is already newer than 284 2023-01-15T14:35:31.730 INFO:tasks.rados.rados.0.smithi090.stdout:1658: oid 18 version 286 is already newer than 285 2023-01-15T14:35:31.730 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 18 v 286 (ObjNum 592 snap 180 seq_num 592) dirty exists 2023-01-15T14:35:31.731 INFO:tasks.rados.rados.0.smithi090.stdout:1658: left oid 18 (ObjNum 592 snap 180 seq_num 592) 2023-01-15T14:35:31.731 INFO:tasks.rados.rados.0.smithi090.stdout:1655: done (3 left) 2023-01-15T14:35:31.731 INFO:tasks.rados.rados.0.smithi090.stdout:1658: done (2 left) 2023-01-15T14:35:31.731 INFO:tasks.rados.rados.0.smithi090.stdout:1659: done (1 left) 2023-01-15T14:35:31.731 INFO:tasks.rados.rados.0.smithi090.stdout:1664: read oid 41 snap -1 2023-01-15T14:35:31.731 INFO:tasks.rados.rados.0.smithi090.stdout:1664: expect (ObjNum 513 snap 152 seq_num 513) 2023-01-15T14:35:31.731 INFO:tasks.rados.rados.0.smithi090.stdout:1665: write (excl) oid 13 current snap is 180 2023-01-15T14:35:31.731 INFO:tasks.rados.rados.0.smithi090.stdout:1665: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2023-01-15T14:35:31.731 INFO:tasks.rados.rados.0.smithi090.stdout:1665: writing smithi09025264-13 from 783358 to 1356798 tid 1 2023-01-15T14:35:31.731 INFO:tasks.rados.rados.0.smithi090.stdout:1665: writing smithi09025264-13 from 2026304 to 2607172 tid 2 2023-01-15T14:35:31.731 INFO:tasks.rados.rados.0.smithi090.stdout:1665: writing smithi09025264-13 from 3279140 to 3900598 tid 3 2023-01-15T14:35:31.732 INFO:tasks.rados.rados.0.smithi090.stdout:1666: write (excl) oid 6 current snap is 180 2023-01-15T14:35:31.732 INFO:tasks.rados.rados.0.smithi090.stdout:1666: seq_num 595 ranges {400165=648689,1684786=315379} 2023-01-15T14:35:31.732 INFO:tasks.rados.rados.0.smithi090.stdout:1666: writing smithi09025264-6 from 400165 to 1048854 tid 1 2023-01-15T14:35:31.732 INFO:tasks.rados.rados.0.smithi090.stdout:1666: writing smithi09025264-6 from 1684786 to 2000165 tid 2 2023-01-15T14:35:31.732 INFO:tasks.rados.rados.0.smithi090.stdout:1667: write (excl) oid 27 current snap is 180 2023-01-15T14:35:31.732 INFO:tasks.rados.rados.0.smithi090.stdout:1667: seq_num 596 ranges {416972=723938,1743268=273704} 2023-01-15T14:35:31.732 INFO:tasks.rados.rados.0.smithi090.stdout:1667: writing smithi09025264-27 from 416972 to 1140910 tid 1 2023-01-15T14:35:31.732 INFO:tasks.rados.rados.0.smithi090.stdout:1667: writing smithi09025264-27 from 1743268 to 2016972 tid 2 2023-01-15T14:35:31.732 INFO:tasks.rados.rados.0.smithi090.stdout:1668: rollback oid 26 current snap is 180 2023-01-15T14:35:31.732 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 26 to 165 2023-01-15T14:35:31.732 INFO:tasks.rados.rados.0.smithi090.stdout:1669: read oid 30 snap -1 2023-01-15T14:35:31.733 INFO:tasks.rados.rados.0.smithi090.stdout:1669: expect (ObjNum 449 snap 125 seq_num 449) 2023-01-15T14:35:31.733 INFO:tasks.rados.rados.0.smithi090.stdout:1670: read oid 5 snap -1 2023-01-15T14:35:31.733 INFO:tasks.rados.rados.0.smithi090.stdout:1670: expect deleted 2023-01-15T14:35:31.733 INFO:tasks.rados.rados.0.smithi090.stdout:1671: snap_remove snap 146 2023-01-15T14:35:31.735 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:31.726+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:31.836 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:31.833+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:31.932 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:31.932 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:35:31.932 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:31.932 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:35:31.936 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:31.933+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:31.969 INFO:tasks.rados.rados.0.smithi090.stdout:1671: done (8 left) 2023-01-15T14:35:32.096 INFO:tasks.rados.rados.0.smithi090.stdout:1672: read oid 11 snap -1 2023-01-15T14:35:32.096 INFO:tasks.rados.rados.0.smithi090.stdout:1672: expect (ObjNum 498 snap 146 seq_num 498) 2023-01-15T14:35:32.097 INFO:tasks.rados.rados.0.smithi090.stdout:1673: snap_remove snap 124 2023-01-15T14:35:32.098 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:32.098 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:32.098 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:32.098 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:32.098 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:32.099 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:35:32.099 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:32.090+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:32.199 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:32.190+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:32.262 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:32.262 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:32.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:32.262 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:32.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.262 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:32.262 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 426.19293559599998, 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.263 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:35:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:35:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:35:32.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:35:32.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:35:32.265 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.265 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:35:32.265 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:35:32.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:35:32.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:35:32.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.266 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.266 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.266 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:35:32.266 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:35:32.266 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:35:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:35:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:35:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:35:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.267 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:35:32.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:35:32.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:32.268 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:35:32.268 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:35:32.268 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.268 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.268 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:35:32.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:35:32.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:35:32.269 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:35:32.269 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.269 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.269 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.269 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:35:32.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:35:32.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.270 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:35:32.270 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:35:32.270 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.270 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.270 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:35:32.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:35:32.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:35:32.271 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:35:32.271 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.271 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.271 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.271 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:35:32.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:35:32.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.272 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.272 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.272 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:35:32.272 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:35:32.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.272 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.272 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.272 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.273 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.273 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:32.273 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:35:32.273 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 426.192902979, 2023-01-15T14:35:32.273 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:35:32.273 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.273 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.273 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.273 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:35:32.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.274 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:35:32.274 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.274 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.274 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.274 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.274 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:35:32.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:35:32.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:32.275 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.275 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.275 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.275 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:35:32.275 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:35:32.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:35:32.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:35:32.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.276 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.276 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.276 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:35:32.276 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:35:32.276 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:35:32.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:35:32.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.277 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.277 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:35:32.277 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:35:32.277 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.277 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.278 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:35:32.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:35:32.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:35:32.279 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:35:32.279 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.279 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.279 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.279 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:35:32.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:35:32.280 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.280 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.280 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.280 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.280 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.280 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:32.280 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:35:32.280 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 425.576546425, 2023-01-15T14:35:32.280 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:35:32.280 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.281 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.281 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.281 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:35:32.281 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.281 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.281 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.281 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.281 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:35:32.281 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:35:32.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.282 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.282 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.282 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:35:32.282 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:35:32.282 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:35:32.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:35:32.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.283 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.283 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:35:32.283 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:35:32.283 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.283 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:35:32.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:35:32.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.284 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.284 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:35:32.284 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:35:32.284 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.284 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.285 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:35:32.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:35:32.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:32.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:35:32.286 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:35:32.286 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.286 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.286 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.286 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:35:32.287 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:35:32.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:35:32.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.288 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.288 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:35:32.288 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:35:32.288 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.288 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:35:32.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:35:32.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.289 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:35:32.289 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:35:32.289 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.289 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.289 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 425.57606407100002, 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:35:32.290 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.291 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:35:32.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.292 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.292 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:35:32.292 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:32.292 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.292 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.293 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:35:32.294 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.295 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:32.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:35:32.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:35:32.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.296 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:35:32.296 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:35:32.296 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.296 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.296 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:35:32.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:35:32.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:35:32.297 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:35:32.297 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.297 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.297 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.297 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:35:32.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:35:32.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:35:32.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:35:32.298 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.298 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.298 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.298 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:35:32.298 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.299 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.300 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.300 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.300 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:32.300 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:35:32.300 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 425.57601150800002, 2023-01-15T14:35:32.300 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:35:32.300 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.300 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.300 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.300 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.300 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:35:32.301 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:35:32.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:35:32.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:35:32.303 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:35:32.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:35:32.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:35:32.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:35:32.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:35:32.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:35:32.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:35:32.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:35:32.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:35:32.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:35:32.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:35:32.306 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.306 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.307 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.307 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3562 3.57 3:ea27a813:::smithi09025264-43:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2023-01-15T14:35:32.307 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:35:32.307 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 96.168974680000005, 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59995680200000001, 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3562 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:35:32.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:35:32.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085504+0000", 2023-01-15T14:35:32.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:32.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085513+0000", 2023-01-15T14:35:32.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7360000000000007e-06 2023-01-15T14:35:32.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085528+0000", 2023-01-15T14:35:32.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4671999999999999e-05 2023-01-15T14:35:32.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085539+0000", 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.113e-05 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382469+0000", 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.296929736 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382623+0000", 2023-01-15T14:35:32.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015408300000000001 2023-01-15T14:35:32.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:32.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382985+0000", 2023-01-15T14:35:32.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036232300000000001 2023-01-15T14:35:32.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383762+0000", 2023-01-15T14:35:32.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077680999999999996 2023-01-15T14:35:32.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383779+0000", 2023-01-15T14:35:32.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7617000000000001e-05 2023-01-15T14:35:32.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383838+0000", 2023-01-15T14:35:32.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8795000000000002e-05 2023-01-15T14:35:32.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383918+0000", 2023-01-15T14:35:32.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9524999999999995e-05 2023-01-15T14:35:32.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.385030+0000", 2023-01-15T14:35:32.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001112664 2023-01-15T14:35:32.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685250+0000", 2023-01-15T14:35:32.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30021942800000001 2023-01-15T14:35:32.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685384+0000", 2023-01-15T14:35:32.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134055 2023-01-15T14:35:32.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685463+0000", 2023-01-15T14:35:32.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8991000000000003e-05 2023-01-15T14:35:32.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.317 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.317 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.317 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3618 3.43 3:c3b8cf33:::smithi09025264-26:head [copy-from ver 335] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e352)", 2023-01-15T14:35:32.317 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:35:32.317 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.347215444, 2023-01-15T14:35:32.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73981128600000001, 2023-01-15T14:35:32.317 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.317 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3618 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907263+0000", 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907273+0000", 2023-01-15T14:35:32.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8109999999999997e-06 2023-01-15T14:35:32.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907287+0000", 2023-01-15T14:35:32.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4100999999999999e-05 2023-01-15T14:35:32.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907298+0000", 2023-01-15T14:35:32.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1014e-05 2023-01-15T14:35:32.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907509+0000", 2023-01-15T14:35:32.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021028299999999999 2023-01-15T14:35:32.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:32.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907519+0000", 2023-01-15T14:35:32.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0686999999999999e-05 2023-01-15T14:35:32.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.212673+0000", 2023-01-15T14:35:32.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30515353299999998 2023-01-15T14:35:32.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.213072+0000", 2023-01-15T14:35:32.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039955900000000001 2023-01-15T14:35:32.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:35:32.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.245152+0000", 2023-01-15T14:35:32.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032080139000000001 2023-01-15T14:35:32.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247179+0000", 2023-01-15T14:35:32.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020266640000000001 2023-01-15T14:35:32.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247198+0000", 2023-01-15T14:35:32.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:35:32.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.248181+0000", 2023-01-15T14:35:32.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0009830310000000001 2023-01-15T14:35:32.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.248229+0000", 2023-01-15T14:35:32.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7592000000000001e-05 2023-01-15T14:35:32.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.253799+0000", 2023-01-15T14:35:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055704659999999996 2023-01-15T14:35:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.646730+0000", 2023-01-15T14:35:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39293027400000002 2023-01-15T14:35:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.646885+0000", 2023-01-15T14:35:32.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155243 2023-01-15T14:35:32.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647077+0000", 2023-01-15T14:35:32.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019156899999999999 2023-01-15T14:35:32.327 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.327 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.327 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.327 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3619 3.43 3:c3b8cf33:::smithi09025264-26:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e352)", 2023-01-15T14:35:32.328 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:35:32.328 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 90.347163984999995, 2023-01-15T14:35:32.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.74041005699999995, 2023-01-15T14:35:32.328 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.328 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.328 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.328 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3619 2023-01-15T14:35:32.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.329 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:35:32.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:35:32.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907316+0000", 2023-01-15T14:35:32.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:32.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907320+0000", 2023-01-15T14:35:32.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5820000000000002e-06 2023-01-15T14:35:32.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907328+0000", 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.32e-06 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907334+0000", 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5489999999999997e-06 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.008282+0000", 2023-01-15T14:35:32.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100948497 2023-01-15T14:35:32.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:32.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.008303+0000", 2023-01-15T14:35:32.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0996e-05 2023-01-15T14:35:32.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.214382+0000", 2023-01-15T14:35:32.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20607940899999999 2023-01-15T14:35:32.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:32.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.214553+0000", 2023-01-15T14:35:32.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017038000000000001 2023-01-15T14:35:32.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247445+0000", 2023-01-15T14:35:32.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032892271000000001 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247684+0000", 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023870599999999999 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647212+0000", 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39952834599999998 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647450+0000", 2023-01-15T14:35:32.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023761799999999999 2023-01-15T14:35:32.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647727+0000", 2023-01-15T14:35:32.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027702100000000002 2023-01-15T14:35:32.335 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.335 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.335 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.336 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3654 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (58) in=7b,truncate 2386693] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2023-01-15T14:35:32.336 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:19.020570+0000", 2023-01-15T14:35:32.336 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.233910495999993, 2023-01-15T14:35:32.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60711035099999999, 2023-01-15T14:35:32.336 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.336 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.336 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.336 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.336 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.337 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3654 2023-01-15T14:35:32.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.337 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020570+0000", 2023-01-15T14:35:32.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020570+0000", 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020569+0000", 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020575+0000", 2023-01-15T14:35:32.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2220000000000001e-06 2023-01-15T14:35:32.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020585+0000", 2023-01-15T14:35:32.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0664e-05 2023-01-15T14:35:32.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020592+0000", 2023-01-15T14:35:32.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4500000000000001e-06 2023-01-15T14:35:32.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120721+0000", 2023-01-15T14:35:32.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971699999999 2023-01-15T14:35:32.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120877+0000", 2023-01-15T14:35:32.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015582200000000001 2023-01-15T14:35:32.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.425053+0000", 2023-01-15T14:35:32.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30417533000000002 2023-01-15T14:35:32.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.341 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.341 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.425328+0000", 2023-01-15T14:35:32.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027568299999999998 2023-01-15T14:35:32.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.341 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:32.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.425793+0000", 2023-01-15T14:35:32.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046481000000000002 2023-01-15T14:35:32.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426265+0000", 2023-01-15T14:35:32.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047148999999999997 2023-01-15T14:35:32.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426283+0000", 2023-01-15T14:35:32.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8697000000000001e-05 2023-01-15T14:35:32.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426366+0000", 2023-01-15T14:35:32.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3008000000000002e-05 2023-01-15T14:35:32.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426599+0000", 2023-01-15T14:35:32.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023217999999999999 2023-01-15T14:35:32.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.527135+0000", 2023-01-15T14:35:32.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10053680299999999 2023-01-15T14:35:32.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627472+0000", 2023-01-15T14:35:32.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10033634299999999 2023-01-15T14:35:32.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627569+0000", 2023-01-15T14:35:32.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7781000000000005e-05 2023-01-15T14:35:32.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627681+0000", 2023-01-15T14:35:32.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111117 2023-01-15T14:35:32.345 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.346 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.346 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.346 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3655 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e369)", 2023-01-15T14:35:32.346 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:19.020607+0000", 2023-01-15T14:35:32.346 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 73.233873548000005, 2023-01-15T14:35:32.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60748634800000001, 2023-01-15T14:35:32.346 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.346 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.346 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.347 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.347 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.347 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3655 2023-01-15T14:35:32.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.347 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020607+0000", 2023-01-15T14:35:32.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020607+0000", 2023-01-15T14:35:32.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.348 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020606+0000", 2023-01-15T14:35:32.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:32.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.348 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020611+0000", 2023-01-15T14:35:32.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1860000000000002e-06 2023-01-15T14:35:32.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020627+0000", 2023-01-15T14:35:32.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6294999999999999e-05 2023-01-15T14:35:32.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120742+0000", 2023-01-15T14:35:32.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100114781 2023-01-15T14:35:32.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120971+0000", 2023-01-15T14:35:32.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022973700000000001 2023-01-15T14:35:32.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.121239+0000", 2023-01-15T14:35:32.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026751500000000002 2023-01-15T14:35:32.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426424+0000", 2023-01-15T14:35:32.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30518495299999998 2023-01-15T14:35:32.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426525+0000", 2023-01-15T14:35:32.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101542 2023-01-15T14:35:32.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627751+0000", 2023-01-15T14:35:32.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20122512000000001 2023-01-15T14:35:32.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627909+0000", 2023-01-15T14:35:32.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015884699999999999 2023-01-15T14:35:32.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.628093+0000", 2023-01-15T14:35:32.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018405200000000001 2023-01-15T14:35:32.353 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.353 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.353 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.354 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.65 391/387 [PushOp(3:a75958b1:::smithi09025264-6:f4, version: 371'125, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a75958b1:::smithi09025264-6:f4@371'125, size: 2353046, copy_subset: [0~2353046], clone_subset: {3:a75958b1:::smithi09025264-6:111=[]}, snapset: 12b=[]:{f4=[f2,ed,ec],104=[102,fc,f8,f7,f6],10c=[10a],111=[10f],112=[112],11c=[119,117,115]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-01-15T14:35:32.354 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:41.938596+0000", 2023-01-15T14:35:32.354 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 50.315885033000001, 2023-01-15T14:35:32.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62354823699999995, 2023-01-15T14:35:32.354 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.354 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:32.354 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.938596+0000", 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.938596+0000", 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.938595+0000", 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.939286+0000", 2023-01-15T14:35:32.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069103999999999997 2023-01-15T14:35:32.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.356 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.356 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.939310+0000", 2023-01-15T14:35:32.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4451e-05 2023-01-15T14:35:32.356 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.356 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.939328+0000", 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.785e-05 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:42.562117+0000", 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62278947399999995 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:42.562144+0000", 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6526e-05 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4160 3.40 3:03b54904:::smithi09025264-12:head [write 1996960~793182] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:13.186652+0000", 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.067829195000002, 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72775391700000003, 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.358 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.359 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.359 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.359 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.359 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4160 2023-01-15T14:35:32.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.359 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.186652+0000", 2023-01-15T14:35:32.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.186652+0000", 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.186650+0000", 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.186927+0000", 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000276944 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.186939+0000", 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1936000000000001e-05 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287039+0000", 2023-01-15T14:35:32.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099677 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.289904+0000", 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002865392 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.290150+0000", 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024561500000000003 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 6,7", 2023-01-15T14:35:32.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.290499+0000", 2023-01-15T14:35:32.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000349081 2023-01-15T14:35:32.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.291028+0000", 2023-01-15T14:35:32.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052859200000000004 2023-01-15T14:35:32.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.291054+0000", 2023-01-15T14:35:32.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6899000000000001e-05 2023-01-15T14:35:32.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.291425+0000", 2023-01-15T14:35:32.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000370944 2023-01-15T14:35:32.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392151+0000", 2023-01-15T14:35:32.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10072581799999999 2023-01-15T14:35:32.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392411+0000", 2023-01-15T14:35:32.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025941199999999999 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.913967+0000", 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52155602300000004 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.914081+0000", 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114181 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.914405+0000", 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032457699999999999 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4161 3.40 3:03b54904:::smithi09025264-12:head [write 3226271~336912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:13.287063+0000", 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.967418000999999, 2023-01-15T14:35:32.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62799013199999998, 2023-01-15T14:35:32.368 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.368 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.368 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.368 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4161 2023-01-15T14:35:32.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.368 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287063+0000", 2023-01-15T14:35:32.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287063+0000", 2023-01-15T14:35:32.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287061+0000", 2023-01-15T14:35:32.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:32.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287183+0000", 2023-01-15T14:35:32.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121996 2023-01-15T14:35:32.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287199+0000", 2023-01-15T14:35:32.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5553000000000002e-05 2023-01-15T14:35:32.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287220+0000", 2023-01-15T14:35:32.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:35:32.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.391251+0000", 2023-01-15T14:35:32.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10403082900000001 2023-01-15T14:35:32.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.391397+0000", 2023-01-15T14:35:32.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014677900000000001 2023-01-15T14:35:32.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 6,7", 2023-01-15T14:35:32.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.391606+0000", 2023-01-15T14:35:32.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000208889 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392070+0000", 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000464179 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392083+0000", 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2265e-05 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392255+0000", 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017268700000000001 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392318+0000", 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2959000000000005e-05 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393772+0000", 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014537899999999999 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.914584+0000", 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52081158500000002 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.914703+0000", 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118968 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.915053+0000", 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035010899999999998 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4162 3.40 3:03b54904:::smithi09025264-12:head [setxattr _header (58) in=7b,truncate 3563183] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:13.287234+0000", 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.967246933999999, 2023-01-15T14:35:32.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62852534299999996, 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4162 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287234+0000", 2023-01-15T14:35:32.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287234+0000", 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287233+0000", 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287240+0000", 2023-01-15T14:35:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2359999999999998e-06 2023-01-15T14:35:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287251+0000", 2023-01-15T14:35:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0604999999999999e-05 2023-01-15T14:35:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287259+0000", 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2539999999999992e-06 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392489+0000", 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10523047000000001 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392777+0000", 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000287869 2023-01-15T14:35:32.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 6,7", 2023-01-15T14:35:32.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393227+0000", 2023-01-15T14:35:32.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044994000000000002 2023-01-15T14:35:32.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393605+0000", 2023-01-15T14:35:32.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037751499999999998 2023-01-15T14:35:32.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393617+0000", 2023-01-15T14:35:32.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2401999999999999e-05 2023-01-15T14:35:32.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393666+0000", 2023-01-15T14:35:32.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8542999999999999e-05 2023-01-15T14:35:32.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393945+0000", 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027894500000000001 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393984+0000", 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8924000000000002e-05 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.915291+0000", 2023-01-15T14:35:32.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52130743999999996 2023-01-15T14:35:32.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.915442+0000", 2023-01-15T14:35:32.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000151371 2023-01-15T14:35:32.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.915759+0000", 2023-01-15T14:35:32.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000316769 2023-01-15T14:35:32.384 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.385 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.385 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.385 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4163 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e423)", 2023-01-15T14:35:32.385 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:13.287269+0000", 2023-01-15T14:35:32.385 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.967211464999998, 2023-01-15T14:35:32.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62932847400000003, 2023-01-15T14:35:32.385 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.386 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.386 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.386 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4163 2023-01-15T14:35:32.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.386 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287269+0000", 2023-01-15T14:35:32.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287269+0000", 2023-01-15T14:35:32.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287268+0000", 2023-01-15T14:35:32.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287274+0000", 2023-01-15T14:35:32.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1059999999999999e-06 2023-01-15T14:35:32.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287289+0000", 2023-01-15T14:35:32.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5591999999999999e-05 2023-01-15T14:35:32.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287298+0000", 2023-01-15T14:35:32.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.439e-06 2023-01-15T14:35:32.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393819+0000", 2023-01-15T14:35:32.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.106521479 2023-01-15T14:35:32.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393925+0000", 2023-01-15T14:35:32.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010621899999999999 2023-01-15T14:35:32.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.915815+0000", 2023-01-15T14:35:32.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.521889406 2023-01-15T14:35:32.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.916124+0000", 2023-01-15T14:35:32.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030919299999999998 2023-01-15T14:35:32.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.916598+0000", 2023-01-15T14:35:32.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047383400000000002 2023-01-15T14:35:32.396 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.396 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.396 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.397 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4236 3.10 3:094fbf13:::smithi09025264-31:head [write 564867~562497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2023-01-15T14:35:32.397 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:25.021367+0000", 2023-01-15T14:35:32.397 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.2331140869999997, 2023-01-15T14:35:32.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93630742600000005, 2023-01-15T14:35:32.397 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.397 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.398 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.398 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4236 2023-01-15T14:35:32.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.399 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021367+0000", 2023-01-15T14:35:32.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021367+0000", 2023-01-15T14:35:32.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021364+0000", 2023-01-15T14:35:32.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:32.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021711+0000", 2023-01-15T14:35:32.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034746799999999997 2023-01-15T14:35:32.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021730+0000", 2023-01-15T14:35:32.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8861000000000001e-05 2023-01-15T14:35:32.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021740+0000", 2023-01-15T14:35:32.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6269999999999995e-06 2023-01-15T14:35:32.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.240176+0000", 2023-01-15T14:35:32.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218435822 2023-01-15T14:35:32.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.240589+0000", 2023-01-15T14:35:32.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000412944 2023-01-15T14:35:32.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.750275+0000", 2023-01-15T14:35:32.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.509686097 2023-01-15T14:35:32.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.750458+0000", 2023-01-15T14:35:32.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018323600000000001 2023-01-15T14:35:32.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:35:32.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.750880+0000", 2023-01-15T14:35:32.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042178799999999998 2023-01-15T14:35:32.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.752668+0000", 2023-01-15T14:35:32.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017886689999999999 2023-01-15T14:35:32.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.752722+0000", 2023-01-15T14:35:32.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3548999999999998e-05 2023-01-15T14:35:32.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.752996+0000", 2023-01-15T14:35:32.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027365100000000003 2023-01-15T14:35:32.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.754266+0000", 2023-01-15T14:35:32.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012705629999999999 2023-01-15T14:35:32.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.956860+0000", 2023-01-15T14:35:32.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20259402100000001 2023-01-15T14:35:32.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.956998+0000", 2023-01-15T14:35:32.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138071 2023-01-15T14:35:32.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.957159+0000", 2023-01-15T14:35:32.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016042300000000001 2023-01-15T14:35:32.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.957674+0000", 2023-01-15T14:35:32.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000515286 2023-01-15T14:35:32.409 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.409 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.409 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.410 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4237 3.10 3:094fbf13:::smithi09025264-31:head [write 1898744~415196] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2023-01-15T14:35:32.410 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:25.028069+0000", 2023-01-15T14:35:32.410 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.2264121130000003, 2023-01-15T14:35:32.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.033009273, 2023-01-15T14:35:32.410 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.410 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:35:32.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.411 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.411 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4237 2023-01-15T14:35:32.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.411 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028069+0000", 2023-01-15T14:35:32.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028069+0000", 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028066+0000", 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028356+0000", 2023-01-15T14:35:32.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000289552 2023-01-15T14:35:32.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028381+0000", 2023-01-15T14:35:32.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5344999999999999e-05 2023-01-15T14:35:32.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028392+0000", 2023-01-15T14:35:32.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:35:32.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.340825+0000", 2023-01-15T14:35:32.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31243343600000001 2023-01-15T14:35:32.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341080+0000", 2023-01-15T14:35:32.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025433500000000001 2023-01-15T14:35:32.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.753065+0000", 2023-01-15T14:35:32.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.411985137 2023-01-15T14:35:32.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.753235+0000", 2023-01-15T14:35:32.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017001599999999999 2023-01-15T14:35:32.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:35:32.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.753588+0000", 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352899 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.754120+0000", 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053270200000000002 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.754143+0000", 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3023e-05 2023-01-15T14:35:32.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.754383+0000", 2023-01-15T14:35:32.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000239125 2023-01-15T14:35:32.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.855893+0000", 2023-01-15T14:35:32.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101510855 2023-01-15T14:35:32.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.957885+0000", 2023-01-15T14:35:32.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101991164 2023-01-15T14:35:32.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.061078+0000", 2023-01-15T14:35:32.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103193295 2023-01-15T14:35:32.440 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.441 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.441 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.441 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4238 3.10 3:094fbf13:::smithi09025264-31:head [write 2755347~725863] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2023-01-15T14:35:32.441 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:25.034638+0000", 2023-01-15T14:35:32.441 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.219842227, 2023-01-15T14:35:32.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0264306940000001, 2023-01-15T14:35:32.441 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.441 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:35:32.441 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.442 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.442 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.442 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4238 2023-01-15T14:35:32.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.442 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.034638+0000", 2023-01-15T14:35:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.034638+0000", 2023-01-15T14:35:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.034636+0000", 2023-01-15T14:35:32.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:32.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035141+0000", 2023-01-15T14:35:32.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050502999999999995 2023-01-15T14:35:32.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035167+0000", 2023-01-15T14:35:32.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5602999999999999e-05 2023-01-15T14:35:32.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035179+0000", 2023-01-15T14:35:32.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1493e-05 2023-01-15T14:35:32.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341224+0000", 2023-01-15T14:35:32.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30604500099999998 2023-01-15T14:35:32.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341519+0000", 2023-01-15T14:35:32.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029507399999999999 2023-01-15T14:35:32.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.854433+0000", 2023-01-15T14:35:32.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51291424299999999 2023-01-15T14:35:32.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.854645+0000", 2023-01-15T14:35:32.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021194200000000001 2023-01-15T14:35:32.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:35:32.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.855002+0000", 2023-01-15T14:35:32.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000357306 2023-01-15T14:35:32.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.855747+0000", 2023-01-15T14:35:32.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00074505399999999999 2023-01-15T14:35:32.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.855768+0000", 2023-01-15T14:35:32.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1248999999999999e-05 2023-01-15T14:35:32.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.856119+0000", 2023-01-15T14:35:32.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035049000000000001 2023-01-15T14:35:32.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.856193+0000", 2023-01-15T14:35:32.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3658000000000005e-05 2023-01-15T14:35:32.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.058153+0000", 2023-01-15T14:35:32.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201960634 2023-01-15T14:35:32.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.061069+0000", 2023-01-15T14:35:32.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002915969 2023-01-15T14:35:32.456 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.456 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.456 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.457 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4239 3.10 3:094fbf13:::smithi09025264-31:head [setxattr _header (58) in=7b,truncate 3764867] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2023-01-15T14:35:32.457 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:25.035235+0000", 2023-01-15T14:35:32.457 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.2192456319999998, 2023-01-15T14:35:32.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.026252191, 2023-01-15T14:35:32.458 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.458 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:35:32.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.458 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4239 2023-01-15T14:35:32.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.459 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035235+0000", 2023-01-15T14:35:32.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035235+0000", 2023-01-15T14:35:32.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035234+0000", 2023-01-15T14:35:32.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035241+0000", 2023-01-15T14:35:32.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:35:32.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035257+0000", 2023-01-15T14:35:32.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5588999999999999e-05 2023-01-15T14:35:32.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035263+0000", 2023-01-15T14:35:32.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0769999999999996e-06 2023-01-15T14:35:32.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341588+0000", 2023-01-15T14:35:32.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30632460299999997 2023-01-15T14:35:32.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341770+0000", 2023-01-15T14:35:32.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182759 2023-01-15T14:35:32.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.058251+0000", 2023-01-15T14:35:32.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71648015099999995 2023-01-15T14:35:32.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.058660+0000", 2023-01-15T14:35:32.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040896800000000001 2023-01-15T14:35:32.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:35:32.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.059357+0000", 2023-01-15T14:35:32.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069784899999999995 2023-01-15T14:35:32.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.060114+0000", 2023-01-15T14:35:32.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000756224 2023-01-15T14:35:32.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.060130+0000", 2023-01-15T14:35:32.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6373000000000001e-05 2023-01-15T14:35:32.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.060192+0000", 2023-01-15T14:35:32.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1700999999999998e-05 2023-01-15T14:35:32.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.061487+0000", 2023-01-15T14:35:32.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012955639999999999 2023-01-15T14:35:32.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.469 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.469 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.469 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4240 3.10 3:094fbf13:::smithi09025264-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e435)", 2023-01-15T14:35:32.469 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:25.035274+0000", 2023-01-15T14:35:32.469 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 7.2192064550000001, 2023-01-15T14:35:32.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2388804330000001, 2023-01-15T14:35:32.469 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.469 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:32.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4240 2023-01-15T14:35:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035274+0000", 2023-01-15T14:35:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035274+0000", 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035274+0000", 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035282+0000", 2023-01-15T14:35:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6640000000000004e-06 2023-01-15T14:35:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035294+0000", 2023-01-15T14:35:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1797000000000001e-05 2023-01-15T14:35:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035301+0000", 2023-01-15T14:35:32.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5939999999999999e-06 2023-01-15T14:35:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341812+0000", 2023-01-15T14:35:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306511757 2023-01-15T14:35:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.342005+0000", 2023-01-15T14:35:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192039 2023-01-15T14:35:32.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.274131+0000", 2023-01-15T14:35:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93212631000000001 2023-01-15T14:35:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.274155+0000", 2023-01-15T14:35:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3847000000000001e-05 2023-01-15T14:35:32.474 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.475 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.475 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:32.476 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:35:32.477 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:32.469+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:32.578 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:32.573+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:32.598 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:32.598 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:32.599 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:35:32.599 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:32.603 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:35:32.679 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:32.677+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:32.703 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:32.703 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:32.703 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:32.704 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:32.704 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:32.714 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:35:32.780 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:32.774+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:32.821 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:32.821 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 425.73143872100002, 2023-01-15T14:35:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:35:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.822 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:35:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:32.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:35:32.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:35:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:35:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:35:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:35:32.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:35:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:35:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:35:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:35:32.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:35:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:35:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:35:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:35:32.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:35:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:35:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:35:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:35:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:35:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:35:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:35:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:35:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:35:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:35:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:35:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:35:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:35:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:35:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:35:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:35:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:35:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:35:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:35:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.834 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:32.835 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 425.73113892399999, 2023-01-15T14:35:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:35:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:35:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:35:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:35:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:35:32.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:35:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:35:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:35:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:35:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:35:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:35:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:35:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:35:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:35:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:35:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:35:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:35:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:35:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:35:32.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:35:32.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:35:32.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:35:32.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:35:32.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:35:32.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:35:32.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:35:32.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:35:32.851 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 425.73110914, 2023-01-15T14:35:32.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:35:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.853 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:35:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:35:32.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:35:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:35:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:35:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:35:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:35:32.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:35:32.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:35:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:35:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:35:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:32.862 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 425.60130034000002, 2023-01-15T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.863 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:35:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.864 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:35:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.867 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.869 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:35:32.870 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:35:32.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.873 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:35:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:35:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:35:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:35:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.874 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:35:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:35:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:35:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:35:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.875 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 425.60124364299998, 2023-01-15T14:35:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:35:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.876 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.877 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.878 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:35:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:35:32.879 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:35:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:35:32.880 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:35:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:35:32.881 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:35:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:35:32.882 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:35:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:35:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:35:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:35:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:35:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:35:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:35:32.884 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:35:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:35:32.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:35:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:35:32.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:35:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:35:32.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:35:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 338.89397506300003, 2023-01-15T14:35:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:35:32.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:35:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.889 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:32.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.891 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:35:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:35:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:35:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:35:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:32.892 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:35:32.893 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:35:32.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:35:32.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.894 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.894 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.894 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:35:32.894 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:35:32.894 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:35:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:35:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:32.895 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:32.896 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 334.93755556999997, 2023-01-15T14:35:32.896 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:35:32.896 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.896 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.896 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.896 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:35:32.896 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.896 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.896 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.897 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:35:32.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:32.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:35:32.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:35:32.898 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.898 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.898 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.898 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:35:32.898 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:35:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:35:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:35:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:35:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:35:32.899 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:32.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:35:32.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:35:32.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.900 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.900 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:35:32.900 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:35:32.900 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.900 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.901 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 334.88005788700002, 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.902 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:32.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:35:32.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:35:32.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:35:32.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:35:32.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:35:32.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:35:32.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:35:32.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.912 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:32.912 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:32.912 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 313.51140638200002, 2023-01-15T14:35:32.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:35:32.912 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.912 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:35:32.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:35:32.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:35:32.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:35:32.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:35:32.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:32.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:35:32.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:35:32.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:35:32.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:35:32.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:35:32.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:35:32.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:35:32.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:32.920 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 313.51130374100001, 2023-01-15T14:35:32.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:35:32.921 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.921 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.921 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.921 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:35:32.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.921 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:35:32.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:32.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:35:32.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 268.91032174600002, 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:35:32.927 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.928 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.928 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.928 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:35:32.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.928 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:32.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:35:32.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:35:32.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:35:32.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:35:32.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:35:32.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:35:32.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:35:32.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:35:32.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:35:32.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:35:32.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:35:32.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:32.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:35:32.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:35:32.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:35:32.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:35:32.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:35:32.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:35:32.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:35:32.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:35:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:35:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 268.91029632200002, 2023-01-15T14:35:32.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:32.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:35:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:32.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:35:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:35:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:35:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:35:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:32.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:35:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:35:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:35:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:35:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:32.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:35:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:35:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:35:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:35:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:35:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:35:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:35:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:35:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:35:32.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:35:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:35:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:35:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:35:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:35:32.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:35:32.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:35:32.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:35:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:35:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:35:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:35:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:35:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:35:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:35:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:35:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:35:32.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:35:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:35:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:35:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.951 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2301 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e216)", 2023-01-15T14:35:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 229.62042185499999, 2023-01-15T14:35:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61754771200000003, 2023-01-15T14:35:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2301 2023-01-15T14:35:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.952 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195530+0000", 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195535+0000", 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3660000000000002e-06 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195546+0000", 2023-01-15T14:35:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1062e-05 2023-01-15T14:35:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195551+0000", 2023-01-15T14:35:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2023-01-15T14:35:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195999+0000", 2023-01-15T14:35:32.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044799800000000002 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.196106+0000", 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106172 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510631+0000", 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31452575999999999 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510875+0000", 2023-01-15T14:35:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024415199999999998 2023-01-15T14:35:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812445+0000", 2023-01-15T14:35:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30156923000000002 2023-01-15T14:35:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812727+0000", 2023-01-15T14:35:32.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028227800000000001 2023-01-15T14:35:32.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.813079+0000", 2023-01-15T14:35:32.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199599999999998 2023-01-15T14:35:32.958 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.958 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.958 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2749 3.3d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 181.894537079, 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.69206326600000001, 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.959 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2749 2023-01-15T14:35:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:32.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921418+0000", 2023-01-15T14:35:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1670000000000002e-06 2023-01-15T14:35:32.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921426+0000", 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0569999999999993e-06 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921429+0000", 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9629999999999998e-06 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107302+0000", 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.185873327 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107657+0000", 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035504299999999998 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212694+0000", 2023-01-15T14:35:32.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105037095 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212936+0000", 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241492 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613125+0000", 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40018927900000001 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613294+0000", 2023-01-15T14:35:32.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016838000000000001 2023-01-15T14:35:32.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613479+0000", 2023-01-15T14:35:32.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018577599999999999 2023-01-15T14:35:32.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.965 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.966 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3509 3.53 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (58) in=7b,truncate 2084167] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:32.966 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:32.966 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 103.697893944, 2023-01-15T14:35:32.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70658069899999998, 2023-01-15T14:35:32.966 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.966 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.967 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.967 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3509 2023-01-15T14:35:32.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.967 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:32.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.968 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.968 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.968 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:32.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:32.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:32.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118062+0000", 2023-01-15T14:35:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2260000000000001e-06 2023-01-15T14:35:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118067+0000", 2023-01-15T14:35:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3700000000000003e-06 2023-01-15T14:35:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118072+0000", 2023-01-15T14:35:32.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4880000000000001e-06 2023-01-15T14:35:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218253+0000", 2023-01-15T14:35:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10018108100000001 2023-01-15T14:35:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218258+0000", 2023-01-15T14:35:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.831e-06 2023-01-15T14:35:32.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321058+0000", 2023-01-15T14:35:32.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10280054499999999 2023-01-15T14:35:32.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321223+0000", 2023-01-15T14:35:32.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016474 2023-01-15T14:35:32.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323014+0000", 2023-01-15T14:35:32.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017909550000000001 2023-01-15T14:35:32.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323124+0000", 2023-01-15T14:35:32.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110326 2023-01-15T14:35:32.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:32.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323363+0000", 2023-01-15T14:35:32.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238385 2023-01-15T14:35:32.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:32.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323663+0000", 2023-01-15T14:35:32.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029987699999999999 2023-01-15T14:35:32.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.974 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323677+0000", 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4346e-05 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323729+0000", 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2049999999999998e-05 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323775+0000", 2023-01-15T14:35:32.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5872999999999999e-05 2023-01-15T14:35:32.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.616791+0000", 2023-01-15T14:35:32.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29301572199999998 2023-01-15T14:35:32.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:32.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824267+0000", 2023-01-15T14:35:32.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20747585199999999 2023-01-15T14:35:32.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:32.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824425+0000", 2023-01-15T14:35:32.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015861900000000001 2023-01-15T14:35:32.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824640+0000", 2023-01-15T14:35:32.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021475899999999999 2023-01-15T14:35:32.977 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.977 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3510 3.53 3:cbfd2d71:::smithi09025264-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 103.697816105, 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.80704639, 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.978 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3510 2023-01-15T14:35:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:32.979 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:32.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:32.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:32.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118135+0000", 2023-01-15T14:35:32.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:32.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:32.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118143+0000", 2023-01-15T14:35:32.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0579999999999998e-06 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118162+0000", 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8702000000000002e-05 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118178+0000", 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5645e-05 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321268+0000", 2023-01-15T14:35:32.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20308996000000001 2023-01-15T14:35:32.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321370+0000", 2023-01-15T14:35:32.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010268399999999999 2023-01-15T14:35:32.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323825+0000", 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024546450000000001 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323934+0000", 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010940199999999999 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.924815+0000", 2023-01-15T14:35:32.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60088053500000005 2023-01-15T14:35:32.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:32.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925009+0000", 2023-01-15T14:35:32.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019363599999999999 2023-01-15T14:35:32.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:32.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925183+0000", 2023-01-15T14:35:32.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000174823 2023-01-15T14:35:32.984 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3505 3.62 3:47dfe935:::smithi09025264-29:head [rollback db] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 102.800213395, 2023-01-15T14:35:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95152285400000003, 2023-01-15T14:35:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:32.985 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:32.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:32.986 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:32.986 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3505 2023-01-15T14:35:32.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:32.986 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:33.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015739+0000", 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015742+0000", 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.869e-06 2023-01-15T14:35:33.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015748+0000", 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0360000000000003e-06 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015754+0000", 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.733e-06 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318179+0000", 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30242492300000001 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318502+0000", 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032355699999999998 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365019+0000", 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046516950000000001 2023-01-15T14:35:33.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862003+0000", 2023-01-15T14:35:33.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49698320800000001 2023-01-15T14:35:33.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862314+0000", 2023-01-15T14:35:33.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000311236 2023-01-15T14:35:33.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:33.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863083+0000", 2023-01-15T14:35:33.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076928399999999996 2023-01-15T14:35:33.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863709+0000", 2023-01-15T14:35:33.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062558700000000004 2023-01-15T14:35:33.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863724+0000", 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5288999999999999e-05 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863799+0000", 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4552e-05 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863851+0000", 2023-01-15T14:35:33.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2018999999999998e-05 2023-01-15T14:35:33.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966087+0000", 2023-01-15T14:35:33.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102235952 2023-01-15T14:35:33.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966811+0000", 2023-01-15T14:35:33.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072484800000000005 2023-01-15T14:35:33.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966950+0000", 2023-01-15T14:35:33.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013907399999999999 2023-01-15T14:35:33.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.967263+0000", 2023-01-15T14:35:33.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312107 2023-01-15T14:35:33.081 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.081 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.081 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.081 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3506 3.62 3:47dfe935:::smithi09025264-29:head [append 0~0] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:33.081 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:33.081 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 102.800193573, 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.234729387, 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3506 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:33.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:33.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015759+0000", 2023-01-15T14:35:33.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:33.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015763+0000", 2023-01-15T14:35:33.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.517e-06 2023-01-15T14:35:33.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015768+0000", 2023-01-15T14:35:33.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5620000000000003e-06 2023-01-15T14:35:33.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015773+0000", 2023-01-15T14:35:33.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3730000000000003e-06 2023-01-15T14:35:33.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365160+0000", 2023-01-15T14:35:33.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34938724300000001 2023-01-15T14:35:33.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:33.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365370+0000", 2023-01-15T14:35:33.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021001599999999999 2023-01-15T14:35:33.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863926+0000", 2023-01-15T14:35:33.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49855574600000002 2023-01-15T14:35:33.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.864284+0000", 2023-01-15T14:35:33.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035790600000000001 2023-01-15T14:35:33.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:33.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865099+0000", 2023-01-15T14:35:33.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081499999999999997 2023-01-15T14:35:33.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865640+0000", 2023-01-15T14:35:33.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054118899999999997 2023-01-15T14:35:33.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865653+0000", 2023-01-15T14:35:33.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3614e-05 2023-01-15T14:35:33.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865699+0000", 2023-01-15T14:35:33.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5751000000000003e-05 2023-01-15T14:35:33.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966206+0000", 2023-01-15T14:35:33.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100506741 2023-01-15T14:35:33.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:33.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966665+0000", 2023-01-15T14:35:33.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000459345 2023-01-15T14:35:33.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.249842+0000", 2023-01-15T14:35:33.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28317711200000001 2023-01-15T14:35:33.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250054+0000", 2023-01-15T14:35:33.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021161899999999999 2023-01-15T14:35:33.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250489+0000", 2023-01-15T14:35:33.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043499700000000001 2023-01-15T14:35:33.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.091 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.091 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3572 3.65 3:a75958b1:::smithi09025264-6:head [setxattr _header (57) in=7b,truncate 2235430] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:33.092 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:33.092 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.793563747999997, 2023-01-15T14:35:33.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65482807700000001, 2023-01-15T14:35:33.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.092 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3572 2023-01-15T14:35:33.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.092 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022388+0000", 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022397+0000", 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.704e-06 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022412+0000", 2023-01-15T14:35:33.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5554e-05 2023-01-15T14:35:33.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022422+0000", 2023-01-15T14:35:33.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2839999999999995e-06 2023-01-15T14:35:33.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114230+0000", 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.091808129000000002 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114234+0000", 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4329999999999997e-06 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204200+0000", 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089966321000000002 2023-01-15T14:35:33.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204439+0000", 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238595 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.305883+0000", 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101443713 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306066+0000", 2023-01-15T14:35:33.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182962 2023-01-15T14:35:33.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2023-01-15T14:35:33.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306621+0000", 2023-01-15T14:35:33.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055552800000000001 2023-01-15T14:35:33.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307013+0000", 2023-01-15T14:35:33.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039210599999999998 2023-01-15T14:35:33.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307033+0000", 2023-01-15T14:35:33.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9423e-05 2023-01-15T14:35:33.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307093+0000", 2023-01-15T14:35:33.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0402000000000002e-05 2023-01-15T14:35:33.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:33.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307445+0000", 2023-01-15T14:35:33.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000351677 2023-01-15T14:35:33.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307662+0000", 2023-01-15T14:35:33.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021765099999999999 2023-01-15T14:35:33.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676523+0000", 2023-01-15T14:35:33.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36886056699999997 2023-01-15T14:35:33.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676723+0000", 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019947899999999999 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677217+0000", 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049493900000000001 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.103 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3573 3.65 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:33.103 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:33.103 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 95.793520620999999, 2023-01-15T14:35:33.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65551266799999997, 2023-01-15T14:35:33.103 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.103 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:33.103 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.103 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.103 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.103 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3573 2023-01-15T14:35:33.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022432+0000", 2023-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022436+0000", 2023-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.693e-06 2023-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022447+0000", 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1639e-05 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122543+0000", 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100095369 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122863+0000", 2023-01-15T14:35:33.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032020799999999999 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122869+0000", 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7640000000000002e-06 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206080+0000", 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.083211794000000006 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206289+0000", 2023-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020866600000000001 2023-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307193+0000", 2023-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10090391999999999 2023-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307374+0000", 2023-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018061299999999999 2023-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677342+0000", 2023-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.369968034 2023-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677535+0000", 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019311299999999999 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677945+0000", 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041040299999999999 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.110 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.111 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:33.112 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:35:33.113 INFO:tasks.rados.rados.0.smithi090.stdout:1673: done (9 left) 2023-01-15T14:35:33.113 INFO:tasks.rados.rados.0.smithi090.stdout:1674: read oid 10 snap 170 2023-01-15T14:35:33.113 INFO:tasks.rados.rados.0.smithi090.stdout:1674: expect (ObjNum 518 snap 155 seq_num 518) 2023-01-15T14:35:33.113 INFO:tasks.rados.rados.0.smithi090.stdout:1675: rollback oid 40 current snap is 180 2023-01-15T14:35:33.113 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 40 to 157 2023-01-15T14:35:33.113 INFO:tasks.rados.rados.0.smithi090.stdout:1665: finishing write tid 1 to smithi09025264-13 2023-01-15T14:35:33.113 INFO:tasks.rados.rados.0.smithi090.stdout:1670: done (10 left) 2023-01-15T14:35:33.113 INFO:tasks.rados.rados.0.smithi090.stdout:1676: write oid 24 current snap is 180 2023-01-15T14:35:33.114 INFO:tasks.rados.rados.0.smithi090.stdout:1666: finishing write tid 1 to smithi09025264-6 2023-01-15T14:35:33.114 INFO:tasks.rados.rados.0.smithi090.stdout:1669: expect (ObjNum 449 snap 125 seq_num 449) 2023-01-15T14:35:33.114 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:33.105+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:33.198 INFO:tasks.rados.rados.0.smithi090.stdout:1665: finishing write tid 2 to smithi09025264-13 2023-01-15T14:35:33.199 INFO:tasks.rados.rados.0.smithi090.stdout:1663: finishing rollback tid 0 to smithi09025264-37 2023-01-15T14:35:33.199 INFO:tasks.rados.rados.0.smithi090.stdout:1666: finishing write tid 2 to smithi09025264-6 2023-01-15T14:35:33.199 INFO:tasks.rados.rados.0.smithi090.stdout:1663: finishing rollback tid 1 to smithi09025264-37 2023-01-15T14:35:33.199 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 37 v 502 (ObjNum 129 snap 0 seq_num 3199195400) dirty dne 2023-01-15T14:35:33.199 INFO:tasks.rados.rados.0.smithi090.stdout:1668: finishing rollback tid 1 to smithi09025264-26 2023-01-15T14:35:33.199 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 340 (ObjNum 289 snap 0 seq_num 1) dirty dne 2023-01-15T14:35:33.199 INFO:tasks.rados.rados.0.smithi090.stdout:1665: finishing write tid 3 to smithi09025264-13 2023-01-15T14:35:33.199 INFO:tasks.rados.rados.0.smithi090.stdout:1666: finishing write tid 3 to smithi09025264-6 2023-01-15T14:35:33.199 INFO:tasks.rados.rados.0.smithi090.stdout:1666: finishing write tid 4 to smithi09025264-6 2023-01-15T14:35:33.199 INFO:tasks.rados.rados.0.smithi090.stdout:1666: oid 6 updating version 0 to 356 2023-01-15T14:35:33.199 INFO:tasks.rados.rados.0.smithi090.stdout:1666: oid 6 updating version 356 to 357 2023-01-15T14:35:33.200 INFO:tasks.rados.rados.0.smithi090.stdout:1666: oid 6 version 357 is already newer than 355 2023-01-15T14:35:33.200 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 6 v 357 (ObjNum 595 snap 180 seq_num 595) dirty exists 2023-01-15T14:35:33.200 INFO:tasks.rados.rados.0.smithi090.stdout:1666: left oid 6 (ObjNum 595 snap 180 seq_num 595) 2023-01-15T14:35:33.200 INFO:tasks.rados.rados.0.smithi090.stdout:1667: finishing write tid 1 to smithi09025264-27 2023-01-15T14:35:33.200 INFO:tasks.rados.rados.0.smithi090.stdout:1667: finishing write tid 2 to smithi09025264-27 2023-01-15T14:35:33.200 INFO:tasks.rados.rados.0.smithi090.stdout:1667: finishing write tid 3 to smithi09025264-27 2023-01-15T14:35:33.200 INFO:tasks.rados.rados.0.smithi090.stdout:1665: finishing write tid 4 to smithi09025264-13 2023-01-15T14:35:33.200 INFO:tasks.rados.rados.0.smithi090.stdout:1665: finishing write tid 5 to smithi09025264-13 2023-01-15T14:35:33.200 INFO:tasks.rados.rados.0.smithi090.stdout:1665: oid 13 updating version 0 to 384 2023-01-15T14:35:33.200 INFO:tasks.rados.rados.0.smithi090.stdout:1665: oid 13 version 384 is already newer than 383 2023-01-15T14:35:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1665: oid 13 updating version 384 to 385 2023-01-15T14:35:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1665: oid 13 updating version 385 to 386 2023-01-15T14:35:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 13 v 386 (ObjNum 594 snap 180 seq_num 594) dirty exists 2023-01-15T14:35:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1665: left oid 13 (ObjNum 594 snap 180 seq_num 594) 2023-01-15T14:35:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1667: finishing write tid 4 to smithi09025264-27 2023-01-15T14:35:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1667: oid 27 updating version 0 to 327 2023-01-15T14:35:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1667: oid 27 updating version 327 to 328 2023-01-15T14:35:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1667: oid 27 updating version 328 to 329 2023-01-15T14:35:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 27 v 329 (ObjNum 596 snap 180 seq_num 596) dirty exists 2023-01-15T14:35:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1667: left oid 27 (ObjNum 596 snap 180 seq_num 596) 2023-01-15T14:35:33.201 INFO:tasks.rados.rados.0.smithi090.stdout:1672: expect (ObjNum 498 snap 146 seq_num 498) 2023-01-15T14:35:33.213 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:33.209+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:33.215 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:33.215 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.1.0:749 3.4 e443/393)", 2023-01-15T14:35:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:33.173650+0000", 2023-01-15T14:35:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.039267108000000002, 2023-01-15T14:35:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.039284711999999999, 2023-01-15T14:35:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:35:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:33.173650+0000", 2023-01-15T14:35:33.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:33.173650+0000", 2023-01-15T14:35:33.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:33.173649+0000", 2023-01-15T14:35:33.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:33.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.218 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.218 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:33.173665+0000", 2023-01-15T14:35:33.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5903e-05 2023-01-15T14:35:33.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:33.173701+0000", 2023-01-15T14:35:33.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6335999999999999e-05 2023-01-15T14:35:33.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:33.173719+0000", 2023-01-15T14:35:33.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7476999999999999e-05 2023-01-15T14:35:33.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.220 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.220 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.220 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:35:33.221 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:35:33.221 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:33.225 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:35:33.314 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:33.309+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:33.326 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:35:33.329 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:33.329 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:33.329 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:33.329 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:33.330 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:33.330 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:35:33.333 INFO:tasks.rados.rados.0.smithi090.stdout:1664: expect (ObjNum 513 snap 152 seq_num 513) 2023-01-15T14:35:33.339 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:35:33.353 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 7, 1, 6] out_osds: [5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:35:33.353 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:35:33.353 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 2, 3, 4, 7, 1, 6] 2023-01-15T14:35:33.354 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2023-01-15T14:35:33.415 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:33.406+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:33.474 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:33.474 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:33.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:33.474 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:33.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.475 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:33.475 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:33.475 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 298.57495178400001, 2023-01-15T14:35:33.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:35:33.475 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.475 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.475 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.475 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:35:33.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.476 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:33.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:33.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:35:33.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:33.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:35:33.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:35:33.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:35:33.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:35:33.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:35:33.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:35:33.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:35:33.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:35:33.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:33.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:35:33.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:35:33.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:35:33.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:35:33.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:35:33.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:35:33.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:35:33.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:35:33.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:35:33.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:35:33.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:35:33.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:35:33.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:35:33.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:35:33.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:35:33.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 298.57490084099999, 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:33.485 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:33.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:35:33.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:33.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.487 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.487 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:35:33.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:35:33.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:35:33.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:35:33.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:35:33.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:35:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:35:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:35:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:35:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:35:33.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:35:33.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.492 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.492 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.492 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:33.492 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:33.492 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 269.66627200699998, 2023-01-15T14:35:33.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:35:33.492 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.492 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:33.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.493 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.493 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:35:33.493 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.493 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.493 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.493 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.493 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:33.493 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:35:33.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:35:33.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:35:33.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:35:33.496 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.496 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:33.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:35:33.497 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:33.498 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 247.779242598, 2023-01-15T14:35:33.499 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:35:33.499 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.499 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.499 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.499 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.499 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.499 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:35:33.499 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.499 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.499 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.499 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:33.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:33.500 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.500 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.500 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.500 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.500 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:35:33.501 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:35:33.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:35:33.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.502 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.502 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:35:33.502 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:35:33.502 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.502 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.503 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:35:33.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:35:33.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:33.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:35:33.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:35:33.504 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.504 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.504 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.504 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:35:33.504 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.505 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.506 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:35:33.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:35:33.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:35:33.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:35:33.507 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.507 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.507 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.507 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:35:33.507 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:35:33.508 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.508 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.508 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.508 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.508 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.508 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:33.508 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:33.508 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 247.779135524, 2023-01-15T14:35:33.508 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:35:33.508 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.508 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:33.509 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.509 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.509 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.509 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:35:33.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.509 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.509 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.509 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.509 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:33.509 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.509 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:35:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.510 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:35:33.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:35:33.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.511 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:35:33.511 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:35:33.511 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.511 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.511 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:35:33.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:35:33.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.512 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:35:33.512 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:35:33.512 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.512 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.512 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:35:33.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:35:33.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.513 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:35:33.513 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:35:33.513 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.513 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.513 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:35:33.514 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:35:33.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.515 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.515 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:35:33.515 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:35:33.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.515 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.515 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.515 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.515 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.515 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 239.64872200600001, 2023-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.516 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.517 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:35:33.518 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.519 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:33.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:35:33.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:35:33.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.520 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:35:33.520 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:35:33.520 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.520 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.520 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:35:33.521 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:35:33.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:35:33.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.522 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.522 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:35:33.522 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:35:33.522 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.522 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:33.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:35:33.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:35:33.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.523 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:35:33.523 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:35:33.523 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.523 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.523 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:35:33.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:35:33.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:35:33.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:35:33.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:35:33.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:35:33.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.525 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.525 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.525 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:35:33.525 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:33.525 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.47561051400001, 2023-01-15T14:35:33.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:35:33.525 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.526 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.526 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:33.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:33.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:33.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:33.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.527 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.527 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:35:33.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:35:33.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:35:33.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:35:33.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:35:33.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:35:33.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:35:33.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:35:33.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:35:33.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:35:33.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:35:33.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:35:33.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:35:33.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:35:33.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:35:33.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:35:33.531 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.532 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.532 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.532 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:35:33.532 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:33.532 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 172.47179882699999, 2023-01-15T14:35:33.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:35:33.532 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.532 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.532 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:35:33.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:35:33.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:35:33.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:35:33.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:35:33.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:35:33.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:35:33.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:35:33.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:35:33.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:35:33.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:35:33.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:35:33.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.537 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.537 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:35:33.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:35:33.537 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:35:33.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:35:33.538 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.538 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.538 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.538 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:35:33.538 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:35:33.538 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.538 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.538 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.539 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.539 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3120 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'290, mlcod=0'0)", 2023-01-15T14:35:33.539 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:33.539 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.229985551, 2023-01-15T14:35:33.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39853714000000001, 2023-01-15T14:35:33.539 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.539 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.539 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:33.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:33.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239424+0000", 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239747+0000", 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032243300000000001 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239784+0000", 2023-01-15T14:35:33.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7051000000000002e-05 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239802+0000", 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8250999999999999e-05 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536033+0000", 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29623121499999999 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536081+0000", 2023-01-15T14:35:33.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2023-01-15T14:35:33.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536800+0000", 2023-01-15T14:35:33.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071948499999999996 2023-01-15T14:35:33.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536817+0000", 2023-01-15T14:35:33.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6228000000000001e-05 2023-01-15T14:35:33.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.544 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537048+0000", 2023-01-15T14:35:33.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023136200000000001 2023-01-15T14:35:33.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637902+0000", 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10085403900000001 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637963+0000", 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1310999999999996e-05 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.546 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3121 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'291, mlcod=0'0)", 2023-01-15T14:35:33.546 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:33.546 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.227990035, 2023-01-15T14:35:33.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659573799999998, 2023-01-15T14:35:33.546 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.546 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.546 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:33.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:33.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241420+0000", 2023-01-15T14:35:33.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:33.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241854+0000", 2023-01-15T14:35:33.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043330099999999998 2023-01-15T14:35:33.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241878+0000", 2023-01-15T14:35:33.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4559e-05 2023-01-15T14:35:33.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241892+0000", 2023-01-15T14:35:33.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039999999999999e-05 2023-01-15T14:35:33.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536863+0000", 2023-01-15T14:35:33.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29497073400000001 2023-01-15T14:35:33.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536887+0000", 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3683999999999999e-05 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537124+0000", 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023729900000000001 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537136+0000", 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2278e-05 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537345+0000", 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020865200000000001 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637981+0000", 2023-01-15T14:35:33.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063575700000001 2023-01-15T14:35:33.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638018+0000", 2023-01-15T14:35:33.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6826999999999998e-05 2023-01-15T14:35:33.552 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.552 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.552 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.552 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3122 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'292, mlcod=0'0)", 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.226094195, 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39623551600000001, 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243316+0000", 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243833+0000", 2023-01-15T14:35:33.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051703600000000001 2023-01-15T14:35:33.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243863+0000", 2023-01-15T14:35:33.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-01-15T14:35:33.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243879+0000", 2023-01-15T14:35:33.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6263999999999999e-05 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637317+0000", 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39343783199999999 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637356+0000", 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9579000000000002e-05 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637830+0000", 2023-01-15T14:35:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047326399999999999 2023-01-15T14:35:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637885+0000", 2023-01-15T14:35:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5074999999999997e-05 2023-01-15T14:35:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638185+0000", 2023-01-15T14:35:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029993799999999997 2023-01-15T14:35:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638337+0000", 2023-01-15T14:35:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015208299999999999 2023-01-15T14:35:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.639553+0000", 2023-01-15T14:35:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001216389 2023-01-15T14:35:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3123 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'293, mlcod=0'0)", 2023-01-15T14:35:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 144.22435016700001, 2023-01-15T14:35:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.395733102, 2023-01-15T14:35:33.559 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245060+0000", 2023-01-15T14:35:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245075+0000", 2023-01-15T14:35:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4459e-05 2023-01-15T14:35:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245107+0000", 2023-01-15T14:35:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1795000000000003e-05 2023-01-15T14:35:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245121+0000", 2023-01-15T14:35:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4117000000000001e-05 2023-01-15T14:35:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638063+0000", 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39294204300000002 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638097+0000", 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4385999999999999e-05 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638299+0000", 2023-01-15T14:35:33.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020133099999999999 2023-01-15T14:35:33.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638318+0000", 2023-01-15T14:35:33.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514999999999999e-05 2023-01-15T14:35:33.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638386+0000", 2023-01-15T14:35:33.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8089000000000005e-05 2023-01-15T14:35:33.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638430+0000", 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.439e-05 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.640795+0000", 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002364274 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.565 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4216 3.78 3:1eaad226:::smithi09025264-49:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.50565542, 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82042120600000001, 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.566 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4216 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963753+0000", 2023-01-15T14:35:33.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999962 2023-01-15T14:35:33.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963772+0000", 2023-01-15T14:35:33.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9293999999999999e-05 2023-01-15T14:35:33.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963806+0000", 2023-01-15T14:35:33.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3946999999999999e-05 2023-01-15T14:35:33.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963822+0000", 2023-01-15T14:35:33.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5744e-05 2023-01-15T14:35:33.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963930+0000", 2023-01-15T14:35:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010870300000000001 2023-01-15T14:35:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:35:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963946+0000", 2023-01-15T14:35:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5919000000000001e-05 2023-01-15T14:35:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.780699+0000", 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.81675266400000002 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784178+0000", 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034785699999999998 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.571 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4217 3.78 3:1eaad226:::smithi09025264-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:33.572 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:33.572 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.505559865, 2023-01-15T14:35:33.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.820584859, 2023-01-15T14:35:33.572 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.572 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:33.572 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.572 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.572 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.572 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4217 2023-01-15T14:35:33.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.572 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963851+0000", 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963857+0000", 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8699999999999997e-06 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963867+0000", 2023-01-15T14:35:33.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2023-01-15T14:35:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963876+0000", 2023-01-15T14:35:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1120000000000002e-06 2023-01-15T14:35:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064133+0000", 2023-01-15T14:35:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100256427 2023-01-15T14:35:33.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064164+0000", 2023-01-15T14:35:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1649000000000001e-05 2023-01-15T14:35:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784340+0000", 2023-01-15T14:35:33.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72017558699999995 2023-01-15T14:35:33.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784437+0000", 2023-01-15T14:35:33.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6831999999999997e-05 2023-01-15T14:35:33.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.577 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.577 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.577 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4216 3.78 3:1eaad226:::smithi09025264-49:head [copy-from ver 0] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:33.578 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:33.578 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 10.472549973, 2023-01-15T14:35:33.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39069526799999998, 2023-01-15T14:35:33.578 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.578 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:33.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.578 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.578 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4216 2023-01-15T14:35:33.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.578 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:33.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:33.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996860+0000", 2023-01-15T14:35:33.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:33.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996869+0000", 2023-01-15T14:35:33.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2069999999999995e-06 2023-01-15T14:35:33.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996881+0000", 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2611e-05 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.096971+0000", 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100089364 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.284700+0000", 2023-01-15T14:35:33.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18772960999999999 2023-01-15T14:35:33.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.284814+0000", 2023-01-15T14:35:33.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113427 2023-01-15T14:35:33.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.387557+0000", 2023-01-15T14:35:33.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102743529 2023-01-15T14:35:33.582 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.583 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.583 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.583 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4225 3.78 3:1e54a7f5:::smithi09025264-42:head [write 1381780~411527] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:33.583 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:33.583 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.9759889909999995, 2023-01-15T14:35:33.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42766908999999997, 2023-01-15T14:35:33.583 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.583 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.583 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4225 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:33.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493422+0000", 2023-01-15T14:35:33.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:33.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493846+0000", 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042452100000000002 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493861+0000", 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5136e-05 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493872+0000", 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0958000000000001e-05 2023-01-15T14:35:33.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494867+0000", 2023-01-15T14:35:33.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00099453600000000007 2023-01-15T14:35:33.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495083+0000", 2023-01-15T14:35:33.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021589499999999999 2023-01-15T14:35:33.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.815757+0000", 2023-01-15T14:35:33.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32067462499999999 2023-01-15T14:35:33.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.815988+0000", 2023-01-15T14:35:33.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000230762 2023-01-15T14:35:33.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.816381+0000", 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039289799999999998 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817159+0000", 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077780299999999998 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817179+0000", 2023-01-15T14:35:33.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0489e-05 2023-01-15T14:35:33.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817418+0000", 2023-01-15T14:35:33.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023831500000000001 2023-01-15T14:35:33.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:33.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819234+0000", 2023-01-15T14:35:33.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001816553 2023-01-15T14:35:33.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820529+0000", 2023-01-15T14:35:33.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012949719999999999 2023-01-15T14:35:33.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920621+0000", 2023-01-15T14:35:33.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100091965 2023-01-15T14:35:33.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920679+0000", 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8319000000000002e-05 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920795+0000", 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115964 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.921092+0000", 2023-01-15T14:35:33.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000296531 2023-01-15T14:35:33.593 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.593 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.593 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.593 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.593 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.593 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4226 3.78 3:1e54a7f5:::smithi09025264-42:head [write 2300501~586966] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:33.593 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:33.593 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.9754976210000006, 2023-01-15T14:35:33.593 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.730699722, 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4226 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:33.594 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:33.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.595 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.595 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.595 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493913+0000", 2023-01-15T14:35:33.595 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:33.595 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494499+0000", 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058589500000000001 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494549+0000", 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0296000000000001e-05 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.596 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494577+0000", 2023-01-15T14:35:33.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8056999999999999e-05 2023-01-15T14:35:33.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495162+0000", 2023-01-15T14:35:33.597 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058508500000000005 2023-01-15T14:35:33.597 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.597 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.597 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.597 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495377+0000", 2023-01-15T14:35:33.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021484400000000001 2023-01-15T14:35:33.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817336+0000", 2023-01-15T14:35:33.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32195888499999997 2023-01-15T14:35:33.598 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.598 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.598 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.598 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817638+0000", 2023-01-15T14:35:33.598 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030184599999999998 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.818285+0000", 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064671399999999995 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819118+0000", 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083369499999999997 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.599 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819137+0000", 2023-01-15T14:35:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8326999999999999e-05 2023-01-15T14:35:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819385+0000", 2023-01-15T14:35:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024822599999999999 2023-01-15T14:35:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.600 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820443+0000", 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058219 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.923538+0000", 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103094642 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.020572+0000", 2023-01-15T14:35:33.601 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097034387999999999 2023-01-15T14:35:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224051+0000", 2023-01-15T14:35:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20347895399999999 2023-01-15T14:35:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224219+0000", 2023-01-15T14:35:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016772 2023-01-15T14:35:33.602 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224614+0000", 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039523500000000002 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4227 3.78 3:1e54a7f5:::smithi09025264-42:head [write 3376974~354279] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:33.603 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.9747733180000004, 2023-01-15T14:35:33.604 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73066059400000005, 2023-01-15T14:35:33.604 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.604 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.604 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.604 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4227 2023-01-15T14:35:33.604 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.604 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.604 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.604 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:33.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:33.605 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.605 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.605 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.605 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.605 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494637+0000", 2023-01-15T14:35:33.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:33.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494805+0000", 2023-01-15T14:35:33.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016839000000000001 2023-01-15T14:35:33.606 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.606 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.606 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.606 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494819+0000", 2023-01-15T14:35:33.606 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4287000000000001e-05 2023-01-15T14:35:33.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494828+0000", 2023-01-15T14:35:33.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8119999999999997e-06 2023-01-15T14:35:33.607 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.607 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.607 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.607 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495478+0000", 2023-01-15T14:35:33.607 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064994399999999996 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495668+0000", 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018991900000000001 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819291+0000", 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32362327600000002 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.608 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819503+0000", 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021184900000000001 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819880+0000", 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037692500000000001 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.609 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820352+0000", 2023-01-15T14:35:33.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047233000000000003 2023-01-15T14:35:33.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820368+0000", 2023-01-15T14:35:33.610 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5877999999999999e-05 2023-01-15T14:35:33.610 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.610 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.610 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.610 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820533+0000", 2023-01-15T14:35:33.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164508 2023-01-15T14:35:33.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:33.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821690+0000", 2023-01-15T14:35:33.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011576290000000001 2023-01-15T14:35:33.611 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.611 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.611 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.611 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.020668+0000", 2023-01-15T14:35:33.611 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19897753800000001 2023-01-15T14:35:33.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.223918+0000", 2023-01-15T14:35:33.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20324972499999999 2023-01-15T14:35:33.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.612 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.612 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.612 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224796+0000", 2023-01-15T14:35:33.612 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000878655 2023-01-15T14:35:33.612 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224907+0000", 2023-01-15T14:35:33.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011042799999999999 2023-01-15T14:35:33.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225299+0000", 2023-01-15T14:35:33.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000392431 2023-01-15T14:35:33.613 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.614 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.614 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4228 3.78 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (58) in=7b,truncate 3731253] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:33.614 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:33.614 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.9745702089999995, 2023-01-15T14:35:33.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73116634000000003, 2023-01-15T14:35:33.614 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.614 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:33.614 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4228 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494841+0000", 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494846+0000", 2023-01-15T14:35:33.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4960000000000004e-06 2023-01-15T14:35:33.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494879+0000", 2023-01-15T14:35:33.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2376000000000003e-05 2023-01-15T14:35:33.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494887+0000", 2023-01-15T14:35:33.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9100000000000005e-06 2023-01-15T14:35:33.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.595858+0000", 2023-01-15T14:35:33.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100971879 2023-01-15T14:35:33.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596092+0000", 2023-01-15T14:35:33.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023309200000000001 2023-01-15T14:35:33.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820587+0000", 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22449530500000001 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820789+0000", 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000201802 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:33.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821144+0000", 2023-01-15T14:35:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035586199999999998 2023-01-15T14:35:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821591+0000", 2023-01-15T14:35:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000446458 2023-01-15T14:35:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821605+0000", 2023-01-15T14:35:33.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4287000000000001e-05 2023-01-15T14:35:33.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:33.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821653+0000", 2023-01-15T14:35:33.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7781000000000002e-05 2023-01-15T14:35:33.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:33.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821717+0000", 2023-01-15T14:35:33.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3911999999999994e-05 2023-01-15T14:35:33.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.121593+0000", 2023-01-15T14:35:33.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29987650900000001 2023-01-15T14:35:33.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224737+0000", 2023-01-15T14:35:33.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103143918 2023-01-15T14:35:33.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225390+0000", 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065281900000000001 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225496+0000", 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106038 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.226008+0000", 2023-01-15T14:35:33.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051175199999999998 2023-01-15T14:35:33.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.624 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.624 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.624 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4229 3.78 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:33.624 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:33.624 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.9745146630000008, 2023-01-15T14:35:33.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82811652800000002, 2023-01-15T14:35:33.625 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:33.625 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:33.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:33.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:33.625 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:33.625 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4229 2023-01-15T14:35:33.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.625 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:33.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:33.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:33.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:33.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:33.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:33.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:33.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494896+0000", 2023-01-15T14:35:33.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:33.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:33.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494901+0000", 2023-01-15T14:35:33.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8269999999999997e-06 2023-01-15T14:35:33.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:33.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494924+0000", 2023-01-15T14:35:33.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3281999999999998e-05 2023-01-15T14:35:33.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:33.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494936+0000", 2023-01-15T14:35:33.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1457000000000001e-05 2023-01-15T14:35:33.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596198+0000", 2023-01-15T14:35:33.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10126203 2023-01-15T14:35:33.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596400+0000", 2023-01-15T14:35:33.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020234499999999999 2023-01-15T14:35:33.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821768+0000", 2023-01-15T14:35:33.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.225367977 2023-01-15T14:35:33.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.822124+0000", 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035540600000000001 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.322208+0000", 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50008402100000005 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.322487+0000", 2023-01-15T14:35:33.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027940400000000002 2023-01-15T14:35:33.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:33.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:33.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:33.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.323014+0000", 2023-01-15T14:35:33.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052676600000000002 2023-01-15T14:35:33.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.631 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:33.631 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:33.631 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:33.633 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:35:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:33.634 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:33.634 INFO:tasks.rados.rados.0.smithi090.stdout:1676: seq_num 597 ranges {433779=799187,1801750=232029} 2023-01-15T14:35:33.634 INFO:tasks.rados.rados.0.smithi090.stdout:1676: writing smithi09025264-24 from 433779 to 1232966 tid 1 2023-01-15T14:35:33.635 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:33.635 INFO:tasks.rados.rados.0.smithi090.stdout:1676: writing smithi09025264-24 from 1801750 to 2033779 tid 2 2023-01-15T14:35:33.635 INFO:tasks.rados.rados.0.smithi090.stdout:1663: done (10 left) 2023-01-15T14:35:33.636 INFO:tasks.rados.rados.0.smithi090.stdout:1664: done (9 left) 2023-01-15T14:35:33.636 INFO:tasks.rados.rados.0.smithi090.stdout:1665: done (8 left) 2023-01-15T14:35:33.636 INFO:tasks.rados.rados.0.smithi090.stdout:1666: done (7 left) 2023-01-15T14:35:33.636 INFO:tasks.rados.rados.0.smithi090.stdout:1667: done (6 left) 2023-01-15T14:35:33.636 INFO:tasks.rados.rados.0.smithi090.stdout:1668: done (5 left) 2023-01-15T14:35:33.636 INFO:tasks.rados.rados.0.smithi090.stdout:1669: done (4 left) 2023-01-15T14:35:33.636 INFO:tasks.rados.rados.0.smithi090.stdout:1672: done (3 left) 2023-01-15T14:35:33.636 INFO:tasks.rados.rados.0.smithi090.stdout:1677: snap_remove snap 160 2023-01-15T14:35:33.637 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:33.637 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:33.637 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:33.637 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:33.629+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:33.643 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:33.645 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:35:33.645 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:35:33.645 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:35:33.645 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:35:33.645 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:35:33.645 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:35:33.645 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:35:33.645 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:35:33.692 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-01-15T14:35:33.686+0000 7fb90e2b4700 -1 mon.a@0(leader).osd e443 definitely_dead 0 2023-01-15T14:35:33.734 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:33.733+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:33.752 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:33.753 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:33.762 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:35:33.835 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:33.830+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:33.914 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:33.914 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:33.923 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:35:33.972 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:33.966+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:33.985 INFO:tasks.rados.rados.0.smithi090.stdout:1677: done (3 left) 2023-01-15T14:35:33.985 INFO:tasks.rados.rados.0.smithi090.stdout:1678: snap_create 2023-01-15T14:35:33.986 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:35:33.986 INFO:tasks.rados.rados.0.smithi090.stdout:1674: expect (ObjNum 518 snap 155 seq_num 518) 2023-01-15T14:35:33.996 INFO:teuthology.orchestra.run.smithi090.stderr:marked out osd.2. 2023-01-15T14:35:33.999 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:35:34.037 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:34.030+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:34.039 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:34.039 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:34.049 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:35:34.127 INFO:tasks.rados.rados.0.smithi090.stdout:1675: finishing rollback tid 1 to smithi09025264-40 2023-01-15T14:35:34.171 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 314 (ObjNum 97 snap 0 seq_num 3860882904) dirty dne 2023-01-15T14:35:34.171 INFO:tasks.rados.rados.0.smithi090.stdout:1676: finishing write tid 1 to smithi09025264-24 2023-01-15T14:35:34.172 INFO:tasks.rados.rados.0.smithi090.stdout:1676: finishing write tid 2 to smithi09025264-24 2023-01-15T14:35:34.172 INFO:tasks.rados.rados.0.smithi090.stdout:1676: finishing write tid 3 to smithi09025264-24 2023-01-15T14:35:34.172 INFO:tasks.rados.rados.0.smithi090.stdout:1676: finishing write tid 4 to smithi09025264-24 2023-01-15T14:35:34.172 INFO:tasks.rados.rados.0.smithi090.stdout:1676: oid 24 updating version 0 to 311 2023-01-15T14:35:34.172 INFO:tasks.rados.rados.0.smithi090.stdout:1676: oid 24 updating version 311 to 312 2023-01-15T14:35:34.172 INFO:tasks.rados.rados.0.smithi090.stdout:1676: oid 24 updating version 312 to 313 2023-01-15T14:35:34.173 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 24 v 313 (ObjNum 597 snap 180 seq_num 597) dirty exists 2023-01-15T14:35:34.173 INFO:tasks.rados.rados.0.smithi090.stdout:1676: left oid 24 (ObjNum 597 snap 180 seq_num 597) 2023-01-15T14:35:34.173 INFO:tasks.rados.rados.0.smithi090.stdout:1674: done (2 left) 2023-01-15T14:35:34.173 INFO:tasks.rados.rados.0.smithi090.stdout:1675: done (1 left) 2023-01-15T14:35:34.173 INFO:tasks.rados.rados.0.smithi090.stdout:1676: done (0 left) 2023-01-15T14:35:34.174 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:34.174 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:34.175 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:34.170+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:34.180 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:35:34.276 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:34.270+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:34.348 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:34.348 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:34.349 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:35:34.376 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:34.370+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:34.462 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:34.513 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:34.515 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:35:34.515 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:34.509+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:34.615 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:34.609+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:34.664 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:34.678 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:34.679 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:35:34.729 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:34.725+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:34.791 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:34.810 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:34.812 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:35:34.817 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:34.810+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:34.917 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:34.917 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:34.918 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:34.913+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:34.927 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:35:34.997 INFO:tasks.rados.rados.0.smithi090.stdout:1678: done (0 left) 2023-01-15T14:35:34.997 INFO:tasks.rados.rados.0.smithi090.stdout:1679: rollback oid 38 current snap is 181 2023-01-15T14:35:34.997 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 38 to 173 2023-01-15T14:35:34.998 INFO:tasks.rados.rados.0.smithi090.stdout:1680: snap_create 2023-01-15T14:35:34.998 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:35.018 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:35.013+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:35.039 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:35.040 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:35.049 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:35:35.119 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:35.113+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:35.187 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:35.190 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:35.197 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:35:35.221 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:35.217+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:35.303 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:35.303 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:35.310 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:35:35.321 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:35.314+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:35.422 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:35.414+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:35.468 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:35.469 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:35.476 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:35:35.523 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:35.517+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:35.580 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:35.636 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:35.638 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:35:35.639 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:35.633+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:35.656 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:35:35.738 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:35.730+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:35.797 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:35.798 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:35.802 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:35:35.823 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:35.833 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:35.839 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:35.830+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:35.866 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:35.961 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:35.961 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:35.962 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:35.962 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:35.962 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:35:35.962 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:35.962 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:35.963 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:35:35.964 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:35.954+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:35.969 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:36.007 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:36.008 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:35:36.009 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:35:36.009 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:35:36.009 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:35:36.009 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:35:36.009 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:35:36.009 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:35:36.009 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:35:36.015 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:35:36.062 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:36.057+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:36.141 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:36.143 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:36.143 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:36.143 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:36.143 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:36.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:35:36.163 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:36.157+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:36.251 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:55 3.1 3:883e9a29:::smithi09025264-14:head [setxattr _header (56) in=7b,truncate 2218491] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:35:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 430.18378776899999, 2023-01-15T14:35:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.601921861, 2023-01-15T14:35:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.252 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.306 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 55 2023-01-15T14:35:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:35:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061545+0000", 2023-01-15T14:35:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061544+0000", 2023-01-15T14:35:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061550+0000", 2023-01-15T14:35:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8000000000000004e-06 2023-01-15T14:35:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061563+0000", 2023-01-15T14:35:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3135999999999999e-05 2023-01-15T14:35:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061570+0000", 2023-01-15T14:35:36.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6000000000000003e-06 2023-01-15T14:35:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366420+0000", 2023-01-15T14:35:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30484977499999999 2023-01-15T14:35:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366643+0000", 2023-01-15T14:35:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022308500000000001 2023-01-15T14:35:36.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560556+0000", 2023-01-15T14:35:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19391356600000001 2023-01-15T14:35:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560736+0000", 2023-01-15T14:35:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179858 2023-01-15T14:35:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.560960+0000", 2023-01-15T14:35:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000223527 2023-01-15T14:35:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561209+0000", 2023-01-15T14:35:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024938800000000002 2023-01-15T14:35:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561234+0000", 2023-01-15T14:35:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4660000000000001e-05 2023-01-15T14:35:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561347+0000", 2023-01-15T14:35:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113551 2023-01-15T14:35:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.313 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561449+0000", 2023-01-15T14:35:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010186 2023-01-15T14:35:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663147+0000", 2023-01-15T14:35:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101698475 2023-01-15T14:35:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663218+0000", 2023-01-15T14:35:36.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.0541000000000001e-05 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663332+0000", 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114221 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663467+0000", 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013473099999999999 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.315 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:56 3.1 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:35:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 430.183755152, 2023-01-15T14:35:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.602118072, 2023-01-15T14:35:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:36.316 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 56 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061578+0000", 2023-01-15T14:35:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061577+0000", 2023-01-15T14:35:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:36.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061581+0000", 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.061589+0000", 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.687e-06 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.161692+0000", 2023-01-15T14:35:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100102463 2023-01-15T14:35:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366710+0000", 2023-01-15T14:35:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20501829299999999 2023-01-15T14:35:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.366927+0000", 2023-01-15T14:35:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021698099999999999 2023-01-15T14:35:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561317+0000", 2023-01-15T14:35:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19438993199999999 2023-01-15T14:35:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.561395+0000", 2023-01-15T14:35:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7433000000000002e-05 2023-01-15T14:35:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663511+0000", 2023-01-15T14:35:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10211632 2023-01-15T14:35:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663591+0000", 2023-01-15T14:35:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0156999999999998e-05 2023-01-15T14:35:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.663696+0000", 2023-01-15T14:35:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000104796 2023-01-15T14:35:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.323 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:126 3.1 3:8889e450:::smithi09025264-32:head [write 1747530~482340] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:35:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 429.56739859800001, 2023-01-15T14:35:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65725732199999998, 2023-01-15T14:35:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.324 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 126 2023-01-15T14:35:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:35:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677934+0000", 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.677931+0000", 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999967 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678323+0000", 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039191399999999999 2023-01-15T14:35:36.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678357+0000", 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3708999999999999e-05 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678368+0000", 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1745e-05 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874449+0000", 2023-01-15T14:35:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19608108299999999 2023-01-15T14:35:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874579+0000", 2023-01-15T14:35:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129259 2023-01-15T14:35:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176494+0000", 2023-01-15T14:35:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.301915726 2023-01-15T14:35:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.176734+0000", 2023-01-15T14:35:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023981100000000001 2023-01-15T14:35:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177128+0000", 2023-01-15T14:35:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039334500000000002 2023-01-15T14:35:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177605+0000", 2023-01-15T14:35:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047764500000000001 2023-01-15T14:35:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177624+0000", 2023-01-15T14:35:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8637000000000001e-05 2023-01-15T14:35:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.363 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177874+0000", 2023-01-15T14:35:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024995099999999999 2023-01-15T14:35:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179109+0000", 2023-01-15T14:35:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001235489 2023-01-15T14:35:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.279555+0000", 2023-01-15T14:35:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100445461 2023-01-15T14:35:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.334965+0000", 2023-01-15T14:35:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.055410102000000003 2023-01-15T14:35:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335098+0000", 2023-01-15T14:35:36.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013353 2023-01-15T14:35:36.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335192+0000", 2023-01-15T14:35:36.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.3240999999999997e-05 2023-01-15T14:35:36.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.366 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.366 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.366 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:127 3.1 3:8889e450:::smithi09025264-32:head [setxattr _header (56) in=7b,truncate 2521017] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:36.367 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:35:36.367 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 429.56691624400003, 2023-01-15T14:35:36.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82865254600000005, 2023-01-15T14:35:36.367 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.367 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:36.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.367 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.367 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 127 2023-01-15T14:35:36.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.367 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:35:36.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678417+0000", 2023-01-15T14:35:36.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678415+0000", 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678426+0000", 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1263e-05 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678446+0000", 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9185999999999999e-05 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678454+0000", 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7080000000000003e-06 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874669+0000", 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19621412799999999 2023-01-15T14:35:36.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874844+0000", 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000175358 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177792+0000", 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30294852900000002 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.177987+0000", 2023-01-15T14:35:36.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019478799999999999 2023-01-15T14:35:36.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:36.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178392+0000", 2023-01-15T14:35:36.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000405012 2023-01-15T14:35:36.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178739+0000", 2023-01-15T14:35:36.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034668699999999998 2023-01-15T14:35:36.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178758+0000", 2023-01-15T14:35:36.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9213000000000001e-05 2023-01-15T14:35:36.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178798+0000", 2023-01-15T14:35:36.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9548999999999997e-05 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179127+0000", 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032966000000000001 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.335335+0000", 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.15620754100000001 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506898+0000", 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.171563361 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.506960+0000", 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1284000000000005e-05 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507069+0000", 2023-01-15T14:35:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010967300000000001 2023-01-15T14:35:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:128 3.1 3:8889e450:::smithi09025264-32:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:36.376 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:35:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 429.56686368099997, 2023-01-15T14:35:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82890738500000005, 2023-01-15T14:35:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:36.377 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.460 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.460 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.460 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 128 2023-01-15T14:35:36.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.461 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:35:36.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678469+0000", 2023-01-15T14:35:36.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678468+0000", 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678474+0000", 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.9259999999999997e-06 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678494+0000", 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9809e-05 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.678505+0000", 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0874e-05 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.874939+0000", 2023-01-15T14:35:36.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19643435400000001 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:26.875075+0000", 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013578099999999999 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.178906+0000", 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30383073300000002 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.179053+0000", 2023-01-15T14:35:36.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014731100000000001 2023-01-15T14:35:36.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507141+0000", 2023-01-15T14:35:36.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32808760599999998 2023-01-15T14:35:36.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507243+0000", 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000102694 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.507377+0000", 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133277 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3562 3.57 3:ea27a813:::smithi09025264-43:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e347)", 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.159826853, 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.59995680200000001, 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3562 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.467 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:35:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085506+0000", 2023-01-15T14:35:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085504+0000", 2023-01-15T14:35:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085513+0000", 2023-01-15T14:35:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7360000000000007e-06 2023-01-15T14:35:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085528+0000", 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4671999999999999e-05 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.085539+0000", 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.113e-05 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382469+0000", 2023-01-15T14:35:36.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.296929736 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382623+0000", 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015408300000000001 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.382985+0000", 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00036232300000000001 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383762+0000", 2023-01-15T14:35:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077680999999999996 2023-01-15T14:35:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383779+0000", 2023-01-15T14:35:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7617000000000001e-05 2023-01-15T14:35:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383838+0000", 2023-01-15T14:35:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8795000000000002e-05 2023-01-15T14:35:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.383918+0000", 2023-01-15T14:35:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9524999999999995e-05 2023-01-15T14:35:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.385030+0000", 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001112664 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685250+0000", 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30021942800000001 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685384+0000", 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000134055 2023-01-15T14:35:36.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:56.685463+0000", 2023-01-15T14:35:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8991000000000003e-05 2023-01-15T14:35:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.477 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3618 3.43 3:c3b8cf33:::smithi09025264-26:head [copy-from ver 335] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e352)", 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.338067616999993, 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73981128600000001, 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3618 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907265+0000", 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907263+0000", 2023-01-15T14:35:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907273+0000", 2023-01-15T14:35:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8109999999999997e-06 2023-01-15T14:35:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907287+0000", 2023-01-15T14:35:36.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4100999999999999e-05 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907298+0000", 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1014e-05 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907509+0000", 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021028299999999999 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907519+0000", 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0686999999999999e-05 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.212673+0000", 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30515353299999998 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.213072+0000", 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039955900000000001 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.245152+0000", 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032080139000000001 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247179+0000", 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020266640000000001 2023-01-15T14:35:36.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247198+0000", 2023-01-15T14:35:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9225999999999999e-05 2023-01-15T14:35:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.248181+0000", 2023-01-15T14:35:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0009830310000000001 2023-01-15T14:35:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.248229+0000", 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7592000000000001e-05 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.253799+0000", 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0055704659999999996 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.646730+0000", 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39293027400000002 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.646885+0000", 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000155243 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647077+0000", 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019156899999999999 2023-01-15T14:35:36.486 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3619 3.43 3:c3b8cf33:::smithi09025264-26:head [stat] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e352)", 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 94.338016158000002, 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.74041005699999995, 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.487 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3619 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.488 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907317+0000", 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907316+0000", 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907320+0000", 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.5820000000000002e-06 2023-01-15T14:35:36.489 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907328+0000", 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.32e-06 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:01.907334+0000", 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5489999999999997e-06 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.490 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.008282+0000", 2023-01-15T14:35:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100948497 2023-01-15T14:35:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.008303+0000", 2023-01-15T14:35:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0996e-05 2023-01-15T14:35:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.491 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.214382+0000", 2023-01-15T14:35:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20607940899999999 2023-01-15T14:35:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.492 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.214553+0000", 2023-01-15T14:35:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017038000000000001 2023-01-15T14:35:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.524 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247445+0000", 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.032892271000000001 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.247684+0000", 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023870599999999999 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.525 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647212+0000", 2023-01-15T14:35:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39952834599999998 2023-01-15T14:35:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647450+0000", 2023-01-15T14:35:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023761799999999999 2023-01-15T14:35:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:02.647727+0000", 2023-01-15T14:35:36.526 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027702100000000002 2023-01-15T14:35:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3654 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (58) in=7b,truncate 2386693] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2023-01-15T14:35:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:19.020570+0000", 2023-01-15T14:35:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.224762669, 2023-01-15T14:35:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60711035099999999, 2023-01-15T14:35:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.527 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3654 2023-01-15T14:35:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020570+0000", 2023-01-15T14:35:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.528 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020570+0000", 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020569+0000", 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.529 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020575+0000", 2023-01-15T14:35:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2220000000000001e-06 2023-01-15T14:35:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020585+0000", 2023-01-15T14:35:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0664e-05 2023-01-15T14:35:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.530 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020592+0000", 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4500000000000001e-06 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120721+0000", 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10012971699999999 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120877+0000", 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015582200000000001 2023-01-15T14:35:36.531 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.425053+0000", 2023-01-15T14:35:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30417533000000002 2023-01-15T14:35:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.425328+0000", 2023-01-15T14:35:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027568299999999998 2023-01-15T14:35:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.532 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.425793+0000", 2023-01-15T14:35:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00046481000000000002 2023-01-15T14:35:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426265+0000", 2023-01-15T14:35:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047148999999999997 2023-01-15T14:35:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.533 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426283+0000", 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8697000000000001e-05 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426366+0000", 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3008000000000002e-05 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426599+0000", 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023217999999999999 2023-01-15T14:35:36.534 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.527135+0000", 2023-01-15T14:35:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10053680299999999 2023-01-15T14:35:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627472+0000", 2023-01-15T14:35:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10033634299999999 2023-01-15T14:35:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.535 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627569+0000", 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7781000000000005e-05 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627681+0000", 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111117 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.536 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3655 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e369)", 2023-01-15T14:35:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:19.020607+0000", 2023-01-15T14:35:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 77.224725720999999, 2023-01-15T14:35:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60748634800000001, 2023-01-15T14:35:36.537 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.619 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:36.619 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3655 2023-01-15T14:35:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020607+0000", 2023-01-15T14:35:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020607+0000", 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020606+0000", 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020611+0000", 2023-01-15T14:35:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1860000000000002e-06 2023-01-15T14:35:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.020627+0000", 2023-01-15T14:35:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6294999999999999e-05 2023-01-15T14:35:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120742+0000", 2023-01-15T14:35:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100114781 2023-01-15T14:35:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.120971+0000", 2023-01-15T14:35:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022973700000000001 2023-01-15T14:35:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.121239+0000", 2023-01-15T14:35:36.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026751500000000002 2023-01-15T14:35:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426424+0000", 2023-01-15T14:35:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30518495299999998 2023-01-15T14:35:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.426525+0000", 2023-01-15T14:35:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101542 2023-01-15T14:35:36.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627751+0000", 2023-01-15T14:35:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20122512000000001 2023-01-15T14:35:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.627909+0000", 2023-01-15T14:35:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015884699999999999 2023-01-15T14:35:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:19.628093+0000", 2023-01-15T14:35:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018405200000000001 2023-01-15T14:35:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.65 391/387 [PushOp(3:a75958b1:::smithi09025264-6:f4, version: 371'125, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:a75958b1:::smithi09025264-6:f4@371'125, size: 2353046, copy_subset: [0~2353046], clone_subset: {3:a75958b1:::smithi09025264-6:111=[]}, snapset: 12b=[]:{f4=[f2,ed,ec],104=[102,fc,f8,f7,f6],10c=[10a],111=[10f],112=[112],11c=[119,117,115]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2023-01-15T14:35:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:34:41.938596+0000", 2023-01-15T14:35:36.626 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 54.306737206000001, 2023-01-15T14:35:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62354823699999995, 2023-01-15T14:35:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.938596+0000", 2023-01-15T14:35:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.938596+0000", 2023-01-15T14:35:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.938595+0000", 2023-01-15T14:35:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.939286+0000", 2023-01-15T14:35:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069103999999999997 2023-01-15T14:35:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.939310+0000", 2023-01-15T14:35:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4451e-05 2023-01-15T14:35:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:41.939328+0000", 2023-01-15T14:35:36.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.785e-05 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:42.562117+0000", 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62278947399999995 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:34:42.562144+0000", 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6526e-05 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.630 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4160 3.40 3:03b54904:::smithi09025264-12:head [write 1996960~793182] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2023-01-15T14:35:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:13.186652+0000", 2023-01-15T14:35:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 23.058681367999998, 2023-01-15T14:35:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72775391700000003, 2023-01-15T14:35:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4160 2023-01-15T14:35:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.186652+0000", 2023-01-15T14:35:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.186652+0000", 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.186650+0000", 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.186927+0000", 2023-01-15T14:35:36.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000276944 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.186939+0000", 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1936000000000001e-05 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287039+0000", 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100099677 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.289904+0000", 2023-01-15T14:35:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002865392 2023-01-15T14:35:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.290150+0000", 2023-01-15T14:35:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024561500000000003 2023-01-15T14:35:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 6,7", 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.290499+0000", 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000349081 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.291028+0000", 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052859200000000004 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.291054+0000", 2023-01-15T14:35:36.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6899000000000001e-05 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.291425+0000", 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000370944 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392151+0000", 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10072581799999999 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392411+0000", 2023-01-15T14:35:36.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025941199999999999 2023-01-15T14:35:36.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.913967+0000", 2023-01-15T14:35:36.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52155602300000004 2023-01-15T14:35:36.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.914081+0000", 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000114181 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.914405+0000", 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032457699999999999 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.639 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.640 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4161 3.40 3:03b54904:::smithi09025264-12:head [write 3226271~336912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2023-01-15T14:35:36.640 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:13.287063+0000", 2023-01-15T14:35:36.640 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.958270173999999, 2023-01-15T14:35:36.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62799013199999998, 2023-01-15T14:35:36.640 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.640 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:36.640 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.640 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.640 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.640 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4161 2023-01-15T14:35:36.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287063+0000", 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287063+0000", 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287061+0000", 2023-01-15T14:35:36.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:36.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287183+0000", 2023-01-15T14:35:36.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000121996 2023-01-15T14:35:36.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287199+0000", 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5553000000000002e-05 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287220+0000", 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1166e-05 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.391251+0000", 2023-01-15T14:35:36.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10403082900000001 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.391397+0000", 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014677900000000001 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 6,7", 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.391606+0000", 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000208889 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392070+0000", 2023-01-15T14:35:36.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000464179 2023-01-15T14:35:36.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.645 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392083+0000", 2023-01-15T14:35:36.645 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2265e-05 2023-01-15T14:35:36.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.645 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392255+0000", 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017268700000000001 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392318+0000", 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2959000000000005e-05 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393772+0000", 2023-01-15T14:35:36.646 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014537899999999999 2023-01-15T14:35:36.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.647 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.914584+0000", 2023-01-15T14:35:36.647 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52081158500000002 2023-01-15T14:35:36.647 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.647 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.647 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:36.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.914703+0000", 2023-01-15T14:35:36.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000118968 2023-01-15T14:35:36.648 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.648 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.648 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.648 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.915053+0000", 2023-01-15T14:35:36.648 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035010899999999998 2023-01-15T14:35:36.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.648 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.648 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.649 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.649 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.649 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4162 3.40 3:03b54904:::smithi09025264-12:head [setxattr _header (58) in=7b,truncate 3563183] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2023-01-15T14:35:36.649 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:13.287234+0000", 2023-01-15T14:35:36.649 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.958099106999999, 2023-01-15T14:35:36.649 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62852534299999996, 2023-01-15T14:35:36.649 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.649 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:36.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.649 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4162 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287234+0000", 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287234+0000", 2023-01-15T14:35:36.650 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287233+0000", 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287240+0000", 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2359999999999998e-06 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.651 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287251+0000", 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0604999999999999e-05 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287259+0000", 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2539999999999992e-06 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.652 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392489+0000", 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10523047000000001 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.392777+0000", 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000287869 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 6,7", 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393227+0000", 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044994000000000002 2023-01-15T14:35:36.653 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.654 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.654 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.654 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393605+0000", 2023-01-15T14:35:36.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037751499999999998 2023-01-15T14:35:36.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.684 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393617+0000", 2023-01-15T14:35:36.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2401999999999999e-05 2023-01-15T14:35:36.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.685 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.685 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393666+0000", 2023-01-15T14:35:36.685 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8542999999999999e-05 2023-01-15T14:35:36.685 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.685 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393945+0000", 2023-01-15T14:35:36.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027894500000000001 2023-01-15T14:35:36.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.686 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393984+0000", 2023-01-15T14:35:36.686 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8924000000000002e-05 2023-01-15T14:35:36.686 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.686 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.686 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.915291+0000", 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.52130743999999996 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.915442+0000", 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000151371 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.915759+0000", 2023-01-15T14:35:36.687 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000316769 2023-01-15T14:35:36.688 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.688 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.688 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.688 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.688 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.688 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4163 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e423)", 2023-01-15T14:35:36.688 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:13.287269+0000", 2023-01-15T14:35:36.688 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 22.958063637999999, 2023-01-15T14:35:36.688 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.62932847400000003, 2023-01-15T14:35:36.688 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.688 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4163 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287269+0000", 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.689 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287269+0000", 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287268+0000", 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.690 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287274+0000", 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.1059999999999999e-06 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287289+0000", 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5591999999999999e-05 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.287298+0000", 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.439e-06 2023-01-15T14:35:36.691 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393819+0000", 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.106521479 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.393925+0000", 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010621899999999999 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.692 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.915815+0000", 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.521889406 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.916124+0000", 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030919299999999998 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:13.916598+0000", 2023-01-15T14:35:36.693 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047383400000000002 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4236 3.10 3:094fbf13:::smithi09025264-31:head [write 564867~562497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:25.021367+0000", 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.223966259999999, 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93630742600000005, 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:36.694 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4236 2023-01-15T14:35:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021367+0000", 2023-01-15T14:35:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.695 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021367+0000", 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021364+0000", 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.696 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021711+0000", 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00034746799999999997 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021730+0000", 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8861000000000001e-05 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.021740+0000", 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6269999999999995e-06 2023-01-15T14:35:36.697 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.240176+0000", 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.218435822 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.240589+0000", 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000412944 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.698 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.750275+0000", 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.509686097 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.750458+0000", 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018323600000000001 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.750880+0000", 2023-01-15T14:35:36.699 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042178799999999998 2023-01-15T14:35:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.752668+0000", 2023-01-15T14:35:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017886689999999999 2023-01-15T14:35:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.752722+0000", 2023-01-15T14:35:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3548999999999998e-05 2023-01-15T14:35:36.700 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.752996+0000", 2023-01-15T14:35:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027365100000000003 2023-01-15T14:35:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.754266+0000", 2023-01-15T14:35:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012705629999999999 2023-01-15T14:35:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.701 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.956860+0000", 2023-01-15T14:35:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20259402100000001 2023-01-15T14:35:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.702 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.956998+0000", 2023-01-15T14:35:36.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138071 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.957159+0000", 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016042300000000001 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.957674+0000", 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000515286 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.785 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4237 3.10 3:094fbf13:::smithi09025264-31:head [write 1898744~415196] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:25.028069+0000", 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.217264286000001, 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.033009273, 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.786 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4237 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028069+0000", 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028069+0000", 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028066+0000", 2023-01-15T14:35:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028356+0000", 2023-01-15T14:35:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000289552 2023-01-15T14:35:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028381+0000", 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5344999999999999e-05 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.028392+0000", 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0587e-05 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.340825+0000", 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31243343600000001 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341080+0000", 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025433500000000001 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.753065+0000", 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.411985137 2023-01-15T14:35:36.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.753235+0000", 2023-01-15T14:35:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017001599999999999 2023-01-15T14:35:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:35:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.753588+0000", 2023-01-15T14:35:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000352899 2023-01-15T14:35:36.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.754120+0000", 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053270200000000002 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.754143+0000", 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3023e-05 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.754383+0000", 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000239125 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.855893+0000", 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101510855 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.957885+0000", 2023-01-15T14:35:36.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101991164 2023-01-15T14:35:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.061078+0000", 2023-01-15T14:35:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103193295 2023-01-15T14:35:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.794 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4238 3.10 3:094fbf13:::smithi09025264-31:head [write 2755347~725863] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:25.034638+0000", 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.2106944, 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0264306940000001, 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4238 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.034638+0000", 2023-01-15T14:35:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.034638+0000", 2023-01-15T14:35:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.034636+0000", 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035141+0000", 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050502999999999995 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035167+0000", 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.5602999999999999e-05 2023-01-15T14:35:36.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035179+0000", 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1493e-05 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341224+0000", 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30604500099999998 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341519+0000", 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029507399999999999 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.854433+0000", 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.51291424299999999 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.854645+0000", 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021194200000000001 2023-01-15T14:35:36.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:35:36.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.855002+0000", 2023-01-15T14:35:36.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000357306 2023-01-15T14:35:36.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.855747+0000", 2023-01-15T14:35:36.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00074505399999999999 2023-01-15T14:35:36.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.855768+0000", 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1248999999999999e-05 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.856119+0000", 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035049000000000001 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:36.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.856193+0000", 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3658000000000005e-05 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.058153+0000", 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.201960634 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.061069+0000", 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002915969 2023-01-15T14:35:36.802 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4239 3.10 3:094fbf13:::smithi09025264-31:head [setxattr _header (58) in=7b,truncate 3764867] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:25.035235+0000", 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.210097805, 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.026252191, 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "waiting for sub ops", 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.803 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4239 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035235+0000", 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035235+0000", 2023-01-15T14:35:36.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035234+0000", 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035241+0000", 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4100000000000002e-06 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035257+0000", 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5588999999999999e-05 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035263+0000", 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0769999999999996e-06 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341588+0000", 2023-01-15T14:35:36.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30632460299999997 2023-01-15T14:35:36.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341770+0000", 2023-01-15T14:35:36.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182759 2023-01-15T14:35:36.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.807 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.807 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.058251+0000", 2023-01-15T14:35:36.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71648015099999995 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.058660+0000", 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00040896800000000001 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.059357+0000", 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00069784899999999995 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.060114+0000", 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000756224 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.060130+0000", 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6373000000000001e-05 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:36.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.060192+0000", 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1700999999999998e-05 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.061487+0000", 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012955639999999999 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.810 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4240 3.10 3:094fbf13:::smithi09025264-31:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e435)", 2023-01-15T14:35:36.811 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:25.035274+0000", 2023-01-15T14:35:36.811 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 11.210058628000001, 2023-01-15T14:35:36.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2388804330000001, 2023-01-15T14:35:36.811 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:36.811 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:36.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:36.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:36.811 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:36.811 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4240 2023-01-15T14:35:36.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.811 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035274+0000", 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035274+0000", 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:36.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035274+0000", 2023-01-15T14:35:36.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:36.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:36.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035282+0000", 2023-01-15T14:35:36.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6640000000000004e-06 2023-01-15T14:35:36.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:36.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035294+0000", 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1797000000000001e-05 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.035301+0000", 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5939999999999999e-06 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.341812+0000", 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.306511757 2023-01-15T14:35:36.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:25.342005+0000", 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000192039 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.274131+0000", 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.93212631000000001 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:36.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:36.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:26.274155+0000", 2023-01-15T14:35:36.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3847000000000001e-05 2023-01-15T14:35:36.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.816 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.816 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:36.847 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:36.847 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:36.849 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:35:36.849 INFO:tasks.rados.rados.0.smithi090.stdout:1679: finishing rollback tid 1 to smithi09025264-38 2023-01-15T14:35:36.849 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 38 v 232 (ObjNum 129 snap 0 seq_num 3199175544) dirty dne 2023-01-15T14:35:36.849 INFO:tasks.rados.rados.0.smithi090.stdout:1679: done (0 left) 2023-01-15T14:35:36.850 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:36.845+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:36.950 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:36.942+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:36.969 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:36.969 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:36.970 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:35:36.970 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:36.988 INFO:tasks.rados.rados.0.smithi090.stdout:1680: done (0 left) 2023-01-15T14:35:36.988 INFO:tasks.rados.rados.0.smithi090.stdout:1681: rollback oid 12 current snap is 182 2023-01-15T14:35:36.988 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 12 to 158 2023-01-15T14:35:36.988 INFO:tasks.rados.rados.0.smithi090.stdout:1682: read oid 4 snap -1 2023-01-15T14:35:36.988 INFO:tasks.rados.rados.0.smithi090.stdout:1682: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:35:36.988 INFO:tasks.rados.rados.0.smithi090.stdout:1683: rollback oid 46 current snap is 182 2023-01-15T14:35:36.988 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 46 to 178 2023-01-15T14:35:36.988 INFO:tasks.rados.rados.0.smithi090.stdout:1684: copy_from oid 7 from oid 18 current snap is 182 2023-01-15T14:35:36.988 INFO:tasks.rados.rados.0.smithi090.stdout:1685: write oid 14 current snap is 182 2023-01-15T14:35:36.988 INFO:tasks.rados.rados.0.smithi090.stdout:1685: seq_num 598 ranges {450586=474436,1343879=588206} 2023-01-15T14:35:36.989 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:35:36.989 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:35:36.990 INFO:tasks.rados.rados.0.smithi090.stdout:1685: writing smithi09025264-14 from 450586 to 925022 tid 1 2023-01-15T14:35:37.082 INFO:tasks.rados.rados.0.smithi090.stdout:1685: writing smithi09025264-14 from 1343879 to 1932085 tid 2 2023-01-15T14:35:37.083 INFO:tasks.rados.rados.0.smithi090.stdout:1686: write oid 1 current snap is 182 2023-01-15T14:35:37.083 INFO:tasks.rados.rados.0.smithi090.stdout:1686: seq_num 599 ranges {467393=549685,1802361=265032} 2023-01-15T14:35:37.084 INFO:tasks.rados.rados.0.smithi090.stdout:1686: writing smithi09025264-1 from 467393 to 1017078 tid 1 2023-01-15T14:35:37.084 INFO:tasks.rados.rados.0.smithi090.stdout:1686: writing smithi09025264-1 from 1802361 to 2067393 tid 2 2023-01-15T14:35:37.084 INFO:tasks.rados.rados.0.smithi090.stdout:1687: snap_remove snap 163 2023-01-15T14:35:37.085 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:37.077+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:37.180 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:37.180 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:37.180 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:37.180 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:37.180 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:37.183 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:37.176+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:37.192 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:35:37.284 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:37.276+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:37.301 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:37.302 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:37.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:37.302 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:37.302 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.302 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:37.302 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:37.302 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 430.21139787800001, 2023-01-15T14:35:37.302 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:35:37.302 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:35:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.303 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.304 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:35:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:35:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:35:37.305 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:35:37.306 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:35:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:35:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:35:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:35:37.307 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:35:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:35:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:35:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:35:37.308 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.309 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.310 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:37.311 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:35:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:35:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:35:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:35:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.312 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 430.21109808099999, 2023-01-15T14:35:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:35:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.313 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.314 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:35:37.315 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:35:37.316 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.317 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:35:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:35:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:35:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:35:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.318 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:35:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:35:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:35:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:35:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:37.319 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:35:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:35:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:35:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:35:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:35:37.320 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:35:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:35:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:35:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:35:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:35:37.321 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:35:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:35:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:35:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:35:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.322 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 430.211068297, 2023-01-15T14:35:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:35:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.323 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:35:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.324 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:35:37.325 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:35:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:35:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:35:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:35:37.326 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:35:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:35:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:35:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:35:37.327 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:35:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:35:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:35:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:35:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:35:37.328 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:35:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:35:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:35:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:35:37.329 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:35:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:35:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:37.330 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:37.331 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 430.08125949700002, 2023-01-15T14:35:37.331 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:35:37.331 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.331 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:37.331 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.331 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.331 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.331 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:35:37.331 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.331 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.331 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:37.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.332 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.332 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:37.332 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.332 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.332 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.333 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:35:37.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:35:37.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:35:37.334 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:35:37.334 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.334 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.334 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.334 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:35:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:35:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:35:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:35:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:35:37.335 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.336 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:37.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:35:37.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:35:37.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:37.337 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:35:37.337 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:35:37.337 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.337 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.337 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:35:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:35:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:35:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:35:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.338 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:35:37.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:35:37.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:35:37.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:35:37.339 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.339 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.339 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:37.339 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:35:37.339 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:35:37.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.340 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.340 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:35:37.340 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:35:37.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.340 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.340 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.340 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.340 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.340 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:37.341 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:37.341 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 430.08120280000003, 2023-01-15T14:35:37.341 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:35:37.341 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.341 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:37.341 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.341 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.341 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.341 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:35:37.341 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.341 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:37.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.342 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.342 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.342 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:37.342 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.342 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:35:37.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:37.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.343 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.343 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.343 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:35:37.343 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:35:37.343 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:35:37.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:35:37.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.344 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.344 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:35:37.344 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:35:37.344 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.344 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:35:37.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:35:37.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.345 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:35:37.345 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:35:37.345 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.345 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.345 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:35:37.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:35:37.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.346 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:35:37.346 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:35:37.346 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.346 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.346 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:35:37.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:35:37.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:35:37.347 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:35:37.347 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.347 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.347 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.347 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:35:37.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:35:37.348 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.348 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.348 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.348 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.348 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.348 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1225 3.4 3:2029fa71:::smithi09025264-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2023-01-15T14:35:37.348 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:37.348 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 343.37393422000002, 2023-01-15T14:35:37.348 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61660987899999997, 2023-01-15T14:35:37.348 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.349 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:37.349 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.349 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.349 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.349 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1225 2023-01-15T14:35:37.349 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.349 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.349 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.349 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.349 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:37.349 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921978+0000", 2023-01-15T14:35:37.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.350 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.350 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.350 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.350 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921977+0000", 2023-01-15T14:35:37.350 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921981+0000", 2023-01-15T14:35:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7100000000000001e-06 2023-01-15T14:35:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:53.921989+0000", 2023-01-15T14:35:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0630000000000006e-06 2023-01-15T14:35:37.351 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.022091+0000", 2023-01-15T14:35:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100101594 2023-01-15T14:35:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122510+0000", 2023-01-15T14:35:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100418858 2023-01-15T14:35:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.352 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:37.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.122622+0000", 2023-01-15T14:35:37.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011207800000000001 2023-01-15T14:35:37.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.353 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.353 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232338+0000", 2023-01-15T14:35:37.353 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10971595100000001 2023-01-15T14:35:37.353 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.353 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.232426+0000", 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8239000000000007e-05 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538214+0000", 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30578765600000002 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.354 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538417+0000", 2023-01-15T14:35:37.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020340000000000001 2023-01-15T14:35:37.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.355 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.355 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:54.538588+0000", 2023-01-15T14:35:37.355 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000171008 2023-01-15T14:35:37.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.355 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.355 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.355 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.355 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.356 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:37.356 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:37.356 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 339.41751472700003, 2023-01-15T14:35:37.356 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:35:37.356 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.356 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:37.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.356 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.356 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:35:37.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.357 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.357 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:37.357 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.357 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.357 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.357 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:37.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.358 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:35:37.358 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:37.358 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.358 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.358 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:35:37.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:35:37.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.359 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:35:37.359 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:35:37.359 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.359 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.359 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:35:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:35:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:35:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:35:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:37.360 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:35:37.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:35:37.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:35:37.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:35:37.361 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.361 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.361 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.361 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:35:37.361 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:37.362 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 339.36001704400002, 2023-01-15T14:35:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:35:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:35:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.363 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:37.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.364 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.364 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:37.364 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.364 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.364 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:35:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:35:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:35:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.365 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:35:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:35:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:35:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:35:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:35:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:35:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:35:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:35:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:35:37.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:35:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:35:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:35:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:35:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:35:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:35:37.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:35:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:35:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:35:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:35:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:35:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:35:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:35:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:35:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:35:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:35:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:35:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:35:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:37.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:35:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:35:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:35:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:35:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 317.99136553900001, 2023-01-15T14:35:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:35:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.373 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:35:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:35:37.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:35:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:35:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:35:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:35:37.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:35:37.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:35:37.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:35:37.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:35:37.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:37.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:35:37.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:35:37.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:37.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:35:37.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:35:37.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:37.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:35:37.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:35:37.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:37.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:35:37.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:35:37.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:37.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:35:37.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:35:37.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:35:37.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:35:37.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:35:37.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:35:37.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:37.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:35:37.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:35:37.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.381 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.381 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.381 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:35:37.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:35:37.382 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.382 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.382 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.382 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:37.382 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:37.382 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 317.991262898, 2023-01-15T14:35:37.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:35:37.382 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.383 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:37.383 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.383 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.383 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.383 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:35:37.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.383 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:37.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:35:37.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:35:37.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:35:37.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:35:37.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:37.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:35:37.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:35:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:35:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:35:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:35:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:35:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:35:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:37.389 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 273.39028090300002, 2023-01-15T14:35:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:35:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:35:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:37.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:37.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:35:37.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:35:37.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:35:37.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:35:37.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:35:37.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:37.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:35:37.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:35:37.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:35:37.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:35:37.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:35:37.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:35:37.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:37.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:35:37.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:35:37.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:35:37.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:35:37.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:35:37.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:35:37.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:35:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:35:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:35:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:35:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:35:37.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:35:37.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:37.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:35:37.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:35:37.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:37.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:35:37.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:35:37.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:37.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:35:37.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:35:37.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:37.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:35:37.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:35:37.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:35:37.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:35:37.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:35:37.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:35:37.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:37.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:35:37.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:35:37.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:35:37.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:35:37.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.401 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.401 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 273.39025547900002, 2023-01-15T14:35:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:35:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.402 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.403 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:35:37.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.403 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:37.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:37.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:35:37.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:35:37.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:35:37.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:35:37.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:35:37.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:37.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:35:37.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:35:37.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:35:37.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:35:37.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:35:37.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:35:37.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:37.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:35:37.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:35:37.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:37.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:35:37.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:35:37.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:37.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:35:37.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:35:37.413 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.413 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.413 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:37.413 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:35:37.413 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:35:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:35:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:35:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:35:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:35:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:35:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2301 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e216)", 2023-01-15T14:35:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 234.10038101200001, 2023-01-15T14:35:37.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61754771200000003, 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2301 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:37.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195530+0000", 2023-01-15T14:35:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:37.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195535+0000", 2023-01-15T14:35:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3660000000000002e-06 2023-01-15T14:35:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195546+0000", 2023-01-15T14:35:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1062e-05 2023-01-15T14:35:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195551+0000", 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195999+0000", 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044799800000000002 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.196106+0000", 2023-01-15T14:35:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106172 2023-01-15T14:35:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510631+0000", 2023-01-15T14:35:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31452575999999999 2023-01-15T14:35:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510875+0000", 2023-01-15T14:35:37.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024415199999999998 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812445+0000", 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30156923000000002 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812727+0000", 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028227800000000001 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.813079+0000", 2023-01-15T14:35:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199599999999998 2023-01-15T14:35:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2749 3.3d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:35:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:37.423 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 186.374496236, 2023-01-15T14:35:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.69206326600000001, 2023-01-15T14:35:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2749 2023-01-15T14:35:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921418+0000", 2023-01-15T14:35:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1670000000000002e-06 2023-01-15T14:35:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921426+0000", 2023-01-15T14:35:37.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0569999999999993e-06 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921429+0000", 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9629999999999998e-06 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107302+0000", 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.185873327 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107657+0000", 2023-01-15T14:35:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035504299999999998 2023-01-15T14:35:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212694+0000", 2023-01-15T14:35:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105037095 2023-01-15T14:35:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212936+0000", 2023-01-15T14:35:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241492 2023-01-15T14:35:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613125+0000", 2023-01-15T14:35:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40018927900000001 2023-01-15T14:35:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613294+0000", 2023-01-15T14:35:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016838000000000001 2023-01-15T14:35:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613479+0000", 2023-01-15T14:35:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018577599999999999 2023-01-15T14:35:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3509 3.53 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (58) in=7b,truncate 2084167] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.177853101, 2023-01-15T14:35:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70658069899999998, 2023-01-15T14:35:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.431 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3509 2023-01-15T14:35:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:37.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:37.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:37.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118062+0000", 2023-01-15T14:35:37.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2260000000000001e-06 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118067+0000", 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3700000000000003e-06 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118072+0000", 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4880000000000001e-06 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218253+0000", 2023-01-15T14:35:37.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10018108100000001 2023-01-15T14:35:37.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:37.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218258+0000", 2023-01-15T14:35:37.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.831e-06 2023-01-15T14:35:37.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321058+0000", 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10280054499999999 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321223+0000", 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016474 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323014+0000", 2023-01-15T14:35:37.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017909550000000001 2023-01-15T14:35:37.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323124+0000", 2023-01-15T14:35:37.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110326 2023-01-15T14:35:37.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:37.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323363+0000", 2023-01-15T14:35:37.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238385 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323663+0000", 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029987699999999999 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323677+0000", 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4346e-05 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:37.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323729+0000", 2023-01-15T14:35:37.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2049999999999998e-05 2023-01-15T14:35:37.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:37.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323775+0000", 2023-01-15T14:35:37.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5872999999999999e-05 2023-01-15T14:35:37.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.616791+0000", 2023-01-15T14:35:37.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29301572199999998 2023-01-15T14:35:37.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824267+0000", 2023-01-15T14:35:37.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20747585199999999 2023-01-15T14:35:37.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:37.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824425+0000", 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015861900000000001 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824640+0000", 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021475899999999999 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.441 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3510 3.53 3:cbfd2d71:::smithi09025264-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 108.177775262, 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.80704639, 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3510 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:37.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:37.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118135+0000", 2023-01-15T14:35:37.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:37.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118143+0000", 2023-01-15T14:35:37.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0579999999999998e-06 2023-01-15T14:35:37.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118162+0000", 2023-01-15T14:35:37.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8702000000000002e-05 2023-01-15T14:35:37.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118178+0000", 2023-01-15T14:35:37.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5645e-05 2023-01-15T14:35:37.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321268+0000", 2023-01-15T14:35:37.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20308996000000001 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321370+0000", 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010268399999999999 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323825+0000", 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024546450000000001 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323934+0000", 2023-01-15T14:35:37.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010940199999999999 2023-01-15T14:35:37.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.924815+0000", 2023-01-15T14:35:37.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60088053500000005 2023-01-15T14:35:37.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925009+0000", 2023-01-15T14:35:37.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019363599999999999 2023-01-15T14:35:37.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925183+0000", 2023-01-15T14:35:37.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000174823 2023-01-15T14:35:37.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.448 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.448 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3505 3.62 3:47dfe935:::smithi09025264-29:head [rollback db] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.280172552, 2023-01-15T14:35:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95152285400000003, 2023-01-15T14:35:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.449 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3505 2023-01-15T14:35:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:37.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015739+0000", 2023-01-15T14:35:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015742+0000", 2023-01-15T14:35:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.869e-06 2023-01-15T14:35:37.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015748+0000", 2023-01-15T14:35:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0360000000000003e-06 2023-01-15T14:35:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015754+0000", 2023-01-15T14:35:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.733e-06 2023-01-15T14:35:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318179+0000", 2023-01-15T14:35:37.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30242492300000001 2023-01-15T14:35:37.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318502+0000", 2023-01-15T14:35:37.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032355699999999998 2023-01-15T14:35:37.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:37.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365019+0000", 2023-01-15T14:35:37.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046516950000000001 2023-01-15T14:35:37.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862003+0000", 2023-01-15T14:35:37.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49698320800000001 2023-01-15T14:35:37.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862314+0000", 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000311236 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863083+0000", 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076928399999999996 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863709+0000", 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062558700000000004 2023-01-15T14:35:37.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:37.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863724+0000", 2023-01-15T14:35:37.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5288999999999999e-05 2023-01-15T14:35:37.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:37.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863799+0000", 2023-01-15T14:35:37.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4552e-05 2023-01-15T14:35:37.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863851+0000", 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2018999999999998e-05 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966087+0000", 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102235952 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966811+0000", 2023-01-15T14:35:37.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072484800000000005 2023-01-15T14:35:37.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:37.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966950+0000", 2023-01-15T14:35:37.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013907399999999999 2023-01-15T14:35:37.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.967263+0000", 2023-01-15T14:35:37.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312107 2023-01-15T14:35:37.459 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.459 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.459 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.459 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3506 3.62 3:47dfe935:::smithi09025264-29:head [append 0~0] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:37.459 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:37.459 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 107.28015273, 2023-01-15T14:35:37.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.234729387, 2023-01-15T14:35:37.459 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.460 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:37.460 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.460 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.460 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.460 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3506 2023-01-15T14:35:37.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.460 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015759+0000", 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:37.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015763+0000", 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.517e-06 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015768+0000", 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5620000000000003e-06 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015773+0000", 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3730000000000003e-06 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365160+0000", 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34938724300000001 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365370+0000", 2023-01-15T14:35:37.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021001599999999999 2023-01-15T14:35:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863926+0000", 2023-01-15T14:35:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49855574600000002 2023-01-15T14:35:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.864284+0000", 2023-01-15T14:35:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035790600000000001 2023-01-15T14:35:37.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865099+0000", 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081499999999999997 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865640+0000", 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054118899999999997 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865653+0000", 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3614e-05 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865699+0000", 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5751000000000003e-05 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966206+0000", 2023-01-15T14:35:37.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100506741 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966665+0000", 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000459345 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.249842+0000", 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28317711200000001 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250054+0000", 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021161899999999999 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250489+0000", 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043499700000000001 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3572 3.65 3:a75958b1:::smithi09025264-6:head [setxattr _header (57) in=7b,truncate 2235430] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.27352290499999, 2023-01-15T14:35:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65482807700000001, 2023-01-15T14:35:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.469 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3572 2023-01-15T14:35:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022388+0000", 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022397+0000", 2023-01-15T14:35:37.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.704e-06 2023-01-15T14:35:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022412+0000", 2023-01-15T14:35:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5554e-05 2023-01-15T14:35:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022422+0000", 2023-01-15T14:35:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2839999999999995e-06 2023-01-15T14:35:37.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114230+0000", 2023-01-15T14:35:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.091808129000000002 2023-01-15T14:35:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114234+0000", 2023-01-15T14:35:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4329999999999997e-06 2023-01-15T14:35:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204200+0000", 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089966321000000002 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204439+0000", 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238595 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.305883+0000", 2023-01-15T14:35:37.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101443713 2023-01-15T14:35:37.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306066+0000", 2023-01-15T14:35:37.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182962 2023-01-15T14:35:37.494 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.494 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.494 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2023-01-15T14:35:37.494 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306621+0000", 2023-01-15T14:35:37.494 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055552800000000001 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307013+0000", 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039210599999999998 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307033+0000", 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9423e-05 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.495 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.496 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:37.496 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307093+0000", 2023-01-15T14:35:37.496 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0402000000000002e-05 2023-01-15T14:35:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307445+0000", 2023-01-15T14:35:37.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000351677 2023-01-15T14:35:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307662+0000", 2023-01-15T14:35:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021765099999999999 2023-01-15T14:35:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676523+0000", 2023-01-15T14:35:37.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36886056699999997 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676723+0000", 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019947899999999999 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677217+0000", 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049493900000000001 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.666 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3573 3.65 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 100.273479778, 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65551266799999997, 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:37.667 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3573 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:37.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022432+0000", 2023-01-15T14:35:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022436+0000", 2023-01-15T14:35:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.693e-06 2023-01-15T14:35:37.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022447+0000", 2023-01-15T14:35:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1639e-05 2023-01-15T14:35:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122543+0000", 2023-01-15T14:35:37.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100095369 2023-01-15T14:35:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122863+0000", 2023-01-15T14:35:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032020799999999999 2023-01-15T14:35:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122869+0000", 2023-01-15T14:35:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7640000000000002e-06 2023-01-15T14:35:37.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206080+0000", 2023-01-15T14:35:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.083211794000000006 2023-01-15T14:35:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206289+0000", 2023-01-15T14:35:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020866600000000001 2023-01-15T14:35:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307193+0000", 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10090391999999999 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307374+0000", 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018061299999999999 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:37.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677342+0000", 2023-01-15T14:35:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.369968034 2023-01-15T14:35:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677535+0000", 2023-01-15T14:35:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019311299999999999 2023-01-15T14:35:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677945+0000", 2023-01-15T14:35:37.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041040299999999999 2023-01-15T14:35:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:37.675 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:37.675 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:37.676 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:35:37.677 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:37.670+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:37.778 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:37.770+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:37.793 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:37.793 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:37.793 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:35:37.793 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:37.804 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:35:37.879 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:37.870+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:37.960 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:37.960 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:37.960 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:37.970 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:35:37.979 INFO:tasks.rados.rados.0.smithi090.stdout:1687: done (6 left) 2023-01-15T14:35:37.980 INFO:tasks.rados.rados.0.smithi090.stdout:1688: delete oid 19 current snap is 182 2023-01-15T14:35:37.980 INFO:tasks.rados.rados.0.smithi090.stdout:1681: finishing rollback tid 0 to smithi09025264-12 2023-01-15T14:35:37.980 INFO:tasks.rados.rados.0.smithi090.stdout:1685: finishing write tid 1 to smithi09025264-14 2023-01-15T14:35:37.980 INFO:tasks.rados.rados.0.smithi090.stdout:1682: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:35:37.981 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:37.974+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:38.024 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:35:38.074 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:38.074 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:38.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:38.074 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:38.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.074 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:38.074 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 303.17445486700001, 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:38.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:38.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:35:38.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:35:38.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:35:38.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:35:38.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:35:38.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:35:38.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:35:38.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:35:38.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:35:38.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:35:38.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:35:38.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:35:38.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:35:38.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:35:38.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:35:38.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:35:38.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:35:38.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:35:38.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:35:38.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:35:38.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:35:38.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:35:38.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.084 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.084 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.085 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:38.085 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:38.085 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 303.17440392399999, 2023-01-15T14:35:38.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:35:38.085 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.085 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:38.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.085 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.085 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:35:38.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:35:38.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:38.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:35:38.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:35:38.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:35:38.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:35:38.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:35:38.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:35:38.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:35:38.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:35:38.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:38.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:35:38.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:35:38.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:35:38.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:35:38.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:38.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:35:38.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:35:38.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:35:38.092 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.092 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.092 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.092 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:38.092 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:38.093 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 274.26577508999998, 2023-01-15T14:35:38.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:35:38.093 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.093 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:38.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.093 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:35:38.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.093 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:38.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:35:38.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:38.098 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 252.378745681, 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:38.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:35:38.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:35:38.104 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.105 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:35:38.106 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:38.107 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:38.108 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 252.378638607, 2023-01-15T14:35:38.108 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:35:38.108 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.108 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:38.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.108 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.108 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:35:38.108 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.108 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.108 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:35:38.109 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.110 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:35:38.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:35:38.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.111 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.111 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:35:38.111 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:35:38.111 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.111 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:38.112 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.113 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 244.24822508899999, 2023-01-15T14:35:38.114 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:38.115 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:38.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.116 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.116 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.116 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:35:38.116 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:38.116 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.117 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:35:38.118 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.119 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:35:38.120 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.121 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.122 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:35:38.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:35:38.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.123 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.123 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:35:38.123 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:35:38.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.123 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.123 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.07511359700001, 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.125 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:35:38.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:35:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:35:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:35:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:35:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.128 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:35:38.129 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 177.07130190999999, 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.130 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.131 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:35:38.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.133 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:35:38.134 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.135 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.136 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:35:38.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:35:38.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.136 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.136 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.136 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.136 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3120 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'290, mlcod=0'0)", 2023-01-15T14:35:38.136 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:38.136 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 148.829488634, 2023-01-15T14:35:38.136 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39853714000000001, 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:38.137 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239424+0000", 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239747+0000", 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032243300000000001 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.138 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239784+0000", 2023-01-15T14:35:38.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7051000000000002e-05 2023-01-15T14:35:38.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239802+0000", 2023-01-15T14:35:38.139 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8250999999999999e-05 2023-01-15T14:35:38.139 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.139 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.139 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.139 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536033+0000", 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29623121499999999 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536081+0000", 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536800+0000", 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071948499999999996 2023-01-15T14:35:38.140 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536817+0000", 2023-01-15T14:35:38.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6228000000000001e-05 2023-01-15T14:35:38.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.141 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.141 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537048+0000", 2023-01-15T14:35:38.141 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023136200000000001 2023-01-15T14:35:38.141 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.141 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637902+0000", 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10085403900000001 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637963+0000", 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1310999999999996e-05 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.142 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3121 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'291, mlcod=0'0)", 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 148.82749311800001, 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659573799999998, 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:38.143 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241420+0000", 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.144 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241854+0000", 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043330099999999998 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241878+0000", 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4559e-05 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.145 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241892+0000", 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039999999999999e-05 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536863+0000", 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29497073400000001 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536887+0000", 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3683999999999999e-05 2023-01-15T14:35:38.146 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537124+0000", 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023729900000000001 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537136+0000", 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2278e-05 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537345+0000", 2023-01-15T14:35:38.147 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020865200000000001 2023-01-15T14:35:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637981+0000", 2023-01-15T14:35:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063575700000001 2023-01-15T14:35:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.148 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638018+0000", 2023-01-15T14:35:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6826999999999998e-05 2023-01-15T14:35:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.149 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3122 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'292, mlcod=0'0)", 2023-01-15T14:35:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 148.825597278, 2023-01-15T14:35:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39623551600000001, 2023-01-15T14:35:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.150 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243316+0000", 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:38.151 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243833+0000", 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051703600000000001 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243863+0000", 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.152 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243879+0000", 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6263999999999999e-05 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637317+0000", 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39343783199999999 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637356+0000", 2023-01-15T14:35:38.153 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9579000000000002e-05 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637830+0000", 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047326399999999999 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637885+0000", 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5074999999999997e-05 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.154 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638185+0000", 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029993799999999997 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638337+0000", 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015208299999999999 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.639553+0000", 2023-01-15T14:35:38.155 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001216389 2023-01-15T14:35:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3123 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'293, mlcod=0'0)", 2023-01-15T14:35:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 148.82385325000001, 2023-01-15T14:35:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.395733102, 2023-01-15T14:35:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.156 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.157 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245060+0000", 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245075+0000", 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4459e-05 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.158 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245107+0000", 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1795000000000003e-05 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245121+0000", 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4117000000000001e-05 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638063+0000", 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39294204300000002 2023-01-15T14:35:38.159 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638097+0000", 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4385999999999999e-05 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638299+0000", 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020133099999999999 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.160 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638318+0000", 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514999999999999e-05 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638386+0000", 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8089000000000005e-05 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638430+0000", 2023-01-15T14:35:38.161 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.439e-05 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.640795+0000", 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002364274 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4216 3.78 3:1eaad226:::smithi09025264-49:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:38.162 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.105158502999998, 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82042120600000001, 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4216 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.163 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963753+0000", 2023-01-15T14:35:38.164 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999962 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963772+0000", 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9293999999999999e-05 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963806+0000", 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3946999999999999e-05 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.165 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963822+0000", 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5744e-05 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963930+0000", 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010870300000000001 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963946+0000", 2023-01-15T14:35:38.166 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5919000000000001e-05 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.780699+0000", 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.81675266400000002 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784178+0000", 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034785699999999998 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.167 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4217 3.78 3:1eaad226:::smithi09025264-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 16.105062948, 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.820584859, 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.168 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4217 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:38.169 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963851+0000", 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963857+0000", 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8699999999999997e-06 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.170 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963867+0000", 2023-01-15T14:35:38.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2023-01-15T14:35:38.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963876+0000", 2023-01-15T14:35:38.171 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1120000000000002e-06 2023-01-15T14:35:38.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064133+0000", 2023-01-15T14:35:38.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100256427 2023-01-15T14:35:38.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:38.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064164+0000", 2023-01-15T14:35:38.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1649000000000001e-05 2023-01-15T14:35:38.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784340+0000", 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72017558699999995 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784437+0000", 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6831999999999997e-05 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4216 3.78 3:1eaad226:::smithi09025264-49:head [copy-from ver 0] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:38.173 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 15.072053056, 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39069526799999998, 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4216 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996860+0000", 2023-01-15T14:35:38.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996869+0000", 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2069999999999995e-06 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996881+0000", 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2611e-05 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.096971+0000", 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100089364 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.284700+0000", 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18772960999999999 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.177 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.284814+0000", 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113427 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.387557+0000", 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102743529 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4225 3.78 3:1e54a7f5:::smithi09025264-42:head [write 1381780~411527] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:38.178 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.575492074, 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42766908999999997, 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4225 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.179 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493422+0000", 2023-01-15T14:35:38.180 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493846+0000", 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042452100000000002 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493861+0000", 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5136e-05 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493872+0000", 2023-01-15T14:35:38.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0958000000000001e-05 2023-01-15T14:35:38.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494867+0000", 2023-01-15T14:35:38.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00099453600000000007 2023-01-15T14:35:38.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495083+0000", 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021589499999999999 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.815757+0000", 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32067462499999999 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.815988+0000", 2023-01-15T14:35:38.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000230762 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.816381+0000", 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039289799999999998 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817159+0000", 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077780299999999998 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817179+0000", 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0489e-05 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817418+0000", 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023831500000000001 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:38.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819234+0000", 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001816553 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820529+0000", 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012949719999999999 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920621+0000", 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100091965 2023-01-15T14:35:38.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920679+0000", 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8319000000000002e-05 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920795+0000", 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115964 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.921092+0000", 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000296531 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4226 3.78 3:1e54a7f5:::smithi09025264-42:head [write 2300501~586966] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.575000704000001, 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.730699722, 2023-01-15T14:35:38.188 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4226 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:38.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493913+0000", 2023-01-15T14:35:38.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:38.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494499+0000", 2023-01-15T14:35:38.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058589500000000001 2023-01-15T14:35:38.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494549+0000", 2023-01-15T14:35:38.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0296000000000001e-05 2023-01-15T14:35:38.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494577+0000", 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8056999999999999e-05 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495162+0000", 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058508500000000005 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495377+0000", 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021484400000000001 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817336+0000", 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32195888499999997 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817638+0000", 2023-01-15T14:35:38.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030184599999999998 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.818285+0000", 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064671399999999995 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819118+0000", 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083369499999999997 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819137+0000", 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8326999999999999e-05 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819385+0000", 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024822599999999999 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:38.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820443+0000", 2023-01-15T14:35:38.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058219 2023-01-15T14:35:38.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.923538+0000", 2023-01-15T14:35:38.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103094642 2023-01-15T14:35:38.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.020572+0000", 2023-01-15T14:35:38.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097034387999999999 2023-01-15T14:35:38.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224051+0000", 2023-01-15T14:35:38.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20347895399999999 2023-01-15T14:35:38.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224219+0000", 2023-01-15T14:35:38.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016772 2023-01-15T14:35:38.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224614+0000", 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039523500000000002 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4227 3.78 3:1e54a7f5:::smithi09025264-42:head [write 3376974~354279] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:38.198 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.574276401000001, 2023-01-15T14:35:38.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73066059400000005, 2023-01-15T14:35:38.199 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.199 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.199 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.199 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4227 2023-01-15T14:35:38.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.199 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494637+0000", 2023-01-15T14:35:38.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:38.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494805+0000", 2023-01-15T14:35:38.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016839000000000001 2023-01-15T14:35:38.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494819+0000", 2023-01-15T14:35:38.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4287000000000001e-05 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494828+0000", 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8119999999999997e-06 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495478+0000", 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064994399999999996 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495668+0000", 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018991900000000001 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819291+0000", 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32362327600000002 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819503+0000", 2023-01-15T14:35:38.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021184900000000001 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819880+0000", 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037692500000000001 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820352+0000", 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047233000000000003 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820368+0000", 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5877999999999999e-05 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820533+0000", 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164508 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:38.205 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821690+0000", 2023-01-15T14:35:38.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011576290000000001 2023-01-15T14:35:38.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.020668+0000", 2023-01-15T14:35:38.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19897753800000001 2023-01-15T14:35:38.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.206 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.206 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.223918+0000", 2023-01-15T14:35:38.206 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20324972499999999 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224796+0000", 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000878655 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224907+0000", 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011042799999999999 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.207 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225299+0000", 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000392431 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4228 3.78 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (58) in=7b,truncate 3731253] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.574073292, 2023-01-15T14:35:38.208 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73116634000000003, 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4228 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:38.209 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494841+0000", 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494846+0000", 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4960000000000004e-06 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494879+0000", 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2376000000000003e-05 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.211 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494887+0000", 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9100000000000005e-06 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.595858+0000", 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100971879 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596092+0000", 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023309200000000001 2023-01-15T14:35:38.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820587+0000", 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22449530500000001 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820789+0000", 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000201802 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.213 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821144+0000", 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035586199999999998 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821591+0000", 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000446458 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821605+0000", 2023-01-15T14:35:38.214 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4287000000000001e-05 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821653+0000", 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7781000000000002e-05 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821717+0000", 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3911999999999994e-05 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.121593+0000", 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29987650900000001 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224737+0000", 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103143918 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:38.216 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225390+0000", 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065281900000000001 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225496+0000", 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106038 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.226008+0000", 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051175199999999998 2023-01-15T14:35:38.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4229 3.78 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 14.574017745999999, 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82811652800000002, 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:38.218 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4229 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:38.219 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:38.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:38.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:38.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494896+0000", 2023-01-15T14:35:38.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:38.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.220 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:38.220 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494901+0000", 2023-01-15T14:35:38.220 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8269999999999997e-06 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494924+0000", 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3281999999999998e-05 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494936+0000", 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1457000000000001e-05 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.221 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596198+0000", 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10126203 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596400+0000", 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020234499999999999 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821768+0000", 2023-01-15T14:35:38.222 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.225367977 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.822124+0000", 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035540600000000001 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.322208+0000", 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50008402100000005 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.322487+0000", 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027940400000000002 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.323014+0000", 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052676600000000002 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.224 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:38.225 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:38.225 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:38.226 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:35:38.227 INFO:tasks.rados.rados.0.smithi090.stdout:1685: finishing write tid 2 to smithi09025264-14 2023-01-15T14:35:38.227 INFO:tasks.rados.rados.0.smithi090.stdout:1681: finishing rollback tid 1 to smithi09025264-12 2023-01-15T14:35:38.227 INFO:tasks.rados.rados.0.smithi090.stdout:1686: finishing write tid 1 to smithi09025264-1 2023-01-15T14:35:38.227 INFO:tasks.rados.rados.0.smithi090.stdout:1686: finishing write tid 2 to smithi09025264-1 2023-01-15T14:35:38.227 INFO:tasks.rados.rados.0.smithi090.stdout:1686: finishing write tid 3 to smithi09025264-1 2023-01-15T14:35:38.227 INFO:tasks.rados.rados.0.smithi090.stdout:1686: finishing write tid 4 to smithi09025264-1 2023-01-15T14:35:38.227 INFO:tasks.rados.rados.0.smithi090.stdout:1686: oid 1 updating version 0 to 512 2023-01-15T14:35:38.227 INFO:tasks.rados.rados.0.smithi090.stdout:1686: oid 1 updating version 512 to 513 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:1686: oid 1 updating version 513 to 514 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 514 (ObjNum 599 snap 182 seq_num 599) dirty exists 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:1686: left oid 1 (ObjNum 599 snap 182 seq_num 599) 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:1681: finishing rollback tid 2 to smithi09025264-12 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 353 (ObjNum 439 snap 121 seq_num 439) dirty exists 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:1685: finishing write tid 3 to smithi09025264-14 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:1685: finishing write tid 4 to smithi09025264-14 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:1685: oid 14 updating version 0 to 455 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:1685: oid 14 version 455 is already newer than 454 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:1685: oid 14 updating version 455 to 456 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 456 (ObjNum 598 snap 182 seq_num 598) dirty exists 2023-01-15T14:35:38.228 INFO:tasks.rados.rados.0.smithi090.stdout:1685: left oid 14 (ObjNum 598 snap 182 seq_num 598) 2023-01-15T14:35:38.229 INFO:tasks.rados.rados.0.smithi090.stdout:1684: finishing copy_from to smithi09025264-7 2023-01-15T14:35:38.229 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 7 v 286 (ObjNum 592 snap 180 seq_num 592) dirty exists 2023-01-15T14:35:38.229 INFO:tasks.rados.rados.0.smithi090.stdout:1683: finishing rollback tid 0 to smithi09025264-46 2023-01-15T14:35:38.229 INFO:tasks.rados.rados.0.smithi090.stdout:1684: finishing copy_from racing read to smithi09025264-7 2023-01-15T14:35:38.229 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:38.230 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:38.220+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:38.295 INFO:tasks.rados.rados.0.smithi090.stdout:1681: done (6 left) 2023-01-15T14:35:38.296 INFO:tasks.rados.rados.0.smithi090.stdout:1682: done (5 left) 2023-01-15T14:35:38.296 INFO:tasks.rados.rados.0.smithi090.stdout:1684: done (4 left) 2023-01-15T14:35:38.296 INFO:tasks.rados.rados.0.smithi090.stdout:1685: done (3 left) 2023-01-15T14:35:38.296 INFO:tasks.rados.rados.0.smithi090.stdout:1686: done (2 left) 2023-01-15T14:35:38.296 INFO:tasks.rados.rados.0.smithi090.stdout:1688: done (1 left) 2023-01-15T14:35:38.296 INFO:tasks.rados.rados.0.smithi090.stdout:1689: write oid 23 current snap is 182 2023-01-15T14:35:38.296 INFO:tasks.rados.rados.0.smithi090.stdout:1689: seq_num 601 ranges {501007=416536,1635678=465329} 2023-01-15T14:35:38.297 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:38.297 INFO:tasks.rados.rados.0.smithi090.stdout:1689: writing smithi09025264-23 from 501007 to 917543 tid 1 2023-01-15T14:35:38.304 INFO:tasks.rados.rados.0.smithi090.stdout:1689: writing smithi09025264-23 from 1635678 to 2101007 tid 2 2023-01-15T14:35:38.304 INFO:tasks.rados.rados.0.smithi090.stdout:1690: delete oid 24 current snap is 182 2023-01-15T14:35:38.315 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:38.327 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:38.318+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:38.331 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:38.341 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:38.342 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:38.349 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:38.351 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:35:38.370 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:38.411 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:38.428 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:38.420+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:38.430 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:35:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:35:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:35:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:35:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:35:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:35:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:35:38.432 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:35:38.455 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:38.455 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:38.465 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:35:38.529 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:38.524+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:38.610 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:38.611 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:38.620 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:35:38.665 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:38.660+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:38.703 INFO:tasks.rados.rados.0.smithi090.stdout:1689: finishing write tid 1 to smithi09025264-23 2023-01-15T14:35:38.703 INFO:tasks.rados.rados.0.smithi090.stdout:1689: finishing write tid 2 to smithi09025264-23 2023-01-15T14:35:38.704 INFO:tasks.rados.rados.0.smithi090.stdout:1689: finishing write tid 3 to smithi09025264-23 2023-01-15T14:35:38.732 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:38.724+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:38.736 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:38.736 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:38.748 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:35:38.806 INFO:tasks.rados.rados.0.smithi090.stdout:1689: finishing write tid 4 to smithi09025264-23 2023-01-15T14:35:38.806 INFO:tasks.rados.rados.0.smithi090.stdout:1689: oid 23 updating version 0 to 298 2023-01-15T14:35:38.806 INFO:tasks.rados.rados.0.smithi090.stdout:1689: oid 23 version 298 is already newer than 296 2023-01-15T14:35:38.806 INFO:tasks.rados.rados.0.smithi090.stdout:1689: oid 23 version 298 is already newer than 297 2023-01-15T14:35:38.806 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 298 (ObjNum 601 snap 182 seq_num 601) dirty exists 2023-01-15T14:35:38.806 INFO:tasks.rados.rados.0.smithi090.stdout:1689: left oid 23 (ObjNum 601 snap 182 seq_num 601) 2023-01-15T14:35:38.813 INFO:tasks.rados.rados.0.smithi090.stdout:1689: done (2 left) 2023-01-15T14:35:38.814 INFO:tasks.rados.rados.0.smithi090.stdout:1690: done (1 left) 2023-01-15T14:35:38.814 INFO:tasks.rados.rados.0.smithi090.stdout:1691: snap_remove snap 148 2023-01-15T14:35:38.832 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:38.826+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:38.851 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:38.852 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:38.862 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:35:38.933 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:38.926+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:38.972 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:38.973 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:38.982 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:35:38.997 INFO:tasks.rados.rados.0.smithi090.stdout:1691: done (1 left) 2023-01-15T14:35:38.997 INFO:tasks.rados.rados.0.smithi090.stdout:1692: read oid 16 snap -1 2023-01-15T14:35:38.997 INFO:tasks.rados.rados.0.smithi090.stdout:1692: expect (ObjNum 498 snap 146 seq_num 498) 2023-01-15T14:35:38.998 INFO:tasks.rados.rados.0.smithi090.stdout:1693: write oid 50 current snap is 182 2023-01-15T14:35:38.998 INFO:tasks.rados.rados.0.smithi090.stdout:1693: seq_num 603 ranges {534621=567034,1636289=498332} 2023-01-15T14:35:39.006 INFO:tasks.rados.rados.0.smithi090.stdout:1693: writing smithi09025264-50 from 534621 to 1101655 tid 1 2023-01-15T14:35:39.013 INFO:tasks.rados.rados.0.smithi090.stdout:1693: writing smithi09025264-50 from 1636289 to 2134621 tid 2 2023-01-15T14:35:39.013 INFO:tasks.rados.rados.0.smithi090.stdout:1694: snap_remove snap 177 2023-01-15T14:35:39.019 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 7, 1, 6] out_osds: [5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:35:39.019 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:35:39.019 INFO:tasks.thrashosds.thrasher:primary_affinity 2023-01-15T14:35:39.019 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.024112 2023-01-15T14:35:39.019 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0.02411161595332545 2023-01-15T14:35:39.034 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:39.028+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:39.082 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:39.082 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:39.084 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:35:39.093 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:35:39.135 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:39.128+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:39.189 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:39.190 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:39.202 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:35:39.245 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:39.240+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:39.337 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:39.332+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:39.357 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:39.357 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:39.366 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:35:39.438 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:39.432+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:39.473 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:39.473 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:39.484 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:35:39.539 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:39.530+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:39.633 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:39.634 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:39.639 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:39.630+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:39.644 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:35:39.740 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:39.734+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:39.748 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:39.749 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:39.759 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:35:39.841 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:39.834+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:39.860 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:39.860 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:39.870 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:35:39.942 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:39.936+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:39.979 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:39.979 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:39.989 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:35:40.007 INFO:tasks.rados.rados.0.smithi090.stdout:1694: done (3 left) 2023-01-15T14:35:40.007 INFO:tasks.rados.rados.0.smithi090.stdout:1695: snap_create 2023-01-15T14:35:40.007 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:35:40.007 INFO:tasks.rados.rados.0.smithi090.stdout:1692: expect (ObjNum 498 snap 146 seq_num 498) 2023-01-15T14:35:40.013 INFO:teuthology.orchestra.run.smithi090.stderr:set osd.6 primary-affinity to 0.0241116 (815802) 2023-01-15T14:35:40.016 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:35:40.043 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:40.036+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:40.091 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:40.091 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:40.102 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:35:40.144 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:40.136+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:40.146 INFO:tasks.rados.rados.0.smithi090.stdout:1692: done (2 left) 2023-01-15T14:35:40.146 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:35:40.146 INFO:tasks.rados.rados.0.smithi090.stdout:1693: finishing write tid 1 to smithi09025264-50 2023-01-15T14:35:40.146 INFO:tasks.rados.rados.0.smithi090.stdout:1693: finishing write tid 2 to smithi09025264-50 2023-01-15T14:35:40.146 INFO:tasks.rados.rados.0.smithi090.stdout:1693: finishing write tid 3 to smithi09025264-50 2023-01-15T14:35:40.146 INFO:tasks.rados.rados.0.smithi090.stdout:1693: finishing write tid 4 to smithi09025264-50 2023-01-15T14:35:40.146 INFO:tasks.rados.rados.0.smithi090.stdout:1693: oid 50 updating version 0 to 309 2023-01-15T14:35:40.146 INFO:tasks.rados.rados.0.smithi090.stdout:1693: oid 50 updating version 309 to 311 2023-01-15T14:35:40.146 INFO:tasks.rados.rados.0.smithi090.stdout:1693: oid 50 version 311 is already newer than 310 2023-01-15T14:35:40.146 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 50 v 311 (ObjNum 603 snap 182 seq_num 603) dirty exists 2023-01-15T14:35:40.146 INFO:tasks.rados.rados.0.smithi090.stdout:1693: left oid 50 (ObjNum 603 snap 182 seq_num 603) 2023-01-15T14:35:40.147 INFO:tasks.rados.rados.0.smithi090.stdout:1693: done (1 left) 2023-01-15T14:35:40.147 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:40.204 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:40.204 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:40.215 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:35:40.245 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:40.240+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:40.314 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:40.315 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:40.325 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:35:40.346 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:40.338+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:40.427 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:40.427 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:40.437 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:35:40.444 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:35:40.446 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:40.440+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:40.539 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:40.539 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:40.546 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:40.540+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:40.548 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:35:40.582 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:40.598 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:40.615 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:40.633 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:40.645 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:40.646 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:40.646 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:40.646 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:40.647 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:40.648 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:40.640+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:40.655 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:40.657 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:35:40.679 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:40.701 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:40.724 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:40.726 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:35:40.726 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:35:40.726 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:35:40.726 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:35:40.726 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:35:40.726 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:35:40.727 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:35:40.727 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:35:40.767 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.767 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 433.67495488600002, 2023-01-15T14:35:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:35:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:35:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.768 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.769 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.770 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:35:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:35:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:35:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:35:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.771 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:35:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:35:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:35:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:35:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.772 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:35:40.773 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:35:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:35:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:35:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:35:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:35:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:35:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:35:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:35:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:35:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:35:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.776 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:35:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:35:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:35:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:35:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 433.674655089, 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:35:40.779 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:35:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.780 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:40.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:35:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:35:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:35:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:35:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:35:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:35:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:35:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:35:40.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:35:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:35:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:35:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:35:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:35:40.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:35:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:35:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:35:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:35:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:35:40.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.787 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:35:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:35:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:35:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:35:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:35:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:35:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 433.67462530500001, 2023-01-15T14:35:40.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:40.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:35:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:40.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:35:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:35:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:35:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:35:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:35:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:35:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:35:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:35:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:35:40.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:35:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:35:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:40.796 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 433.54481650499997, 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.797 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:35:40.798 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:35:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:35:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:35:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:35:40.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:35:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:35:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:35:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:35:40.800 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:35:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:35:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:35:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:35:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.801 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:35:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:35:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:35:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:35:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.802 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:35:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:35:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:35:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:35:40.803 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:35:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:35:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:35:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:35:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.804 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:35:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:35:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:35:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:35:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.805 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:35:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:35:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:35:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:35:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.806 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 433.54475980799998, 2023-01-15T14:35:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:35:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.807 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:40.808 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:35:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.809 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:35:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:35:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:35:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:35:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.810 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:35:40.811 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.812 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.813 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 342.88107173499998, 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:40.814 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:35:40.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:35:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:35:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:35:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:35:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:35:40.819 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 342.82357405200003, 2023-01-15T14:35:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:35:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:40.820 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:35:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:35:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:35:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:35:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:35:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:35:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:35:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:35:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:35:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:35:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:35:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:35:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:35:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:35:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:35:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:35:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:35:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:35:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:40.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:35:40.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:40.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:35:40.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:35:40.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:35:40.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:35:40.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:35:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:35:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 321.45492254700002, 2023-01-15T14:35:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:35:40.830 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:35:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:40.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:35:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:35:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:35:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:35:40.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:35:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:35:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:35:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:35:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:35:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:35:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:35:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:35:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:35:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:35:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:35:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:35:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:35:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:35:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:35:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:35:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:35:40.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:35:40.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:35:40.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:35:40.839 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.839 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.839 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.839 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 321.45481990600001, 2023-01-15T14:35:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:35:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:35:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.840 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:35:40.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:40.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:35:40.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:35:40.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:35:40.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:40.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:35:40.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:35:40.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:35:40.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:35:40.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:35:40.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:35:40.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:35:40.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:35:40.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.847 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:40.847 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:40.847 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 276.85383791100003, 2023-01-15T14:35:40.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:35:40.847 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.847 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:40.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.847 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.847 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:35:40.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:40.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:35:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:35:40.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:35:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:35:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:35:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:35:40.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:35:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:35:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:35:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:35:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:40.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:35:40.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:35:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:35:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:35:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:35:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.854 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:35:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:35:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:35:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:35:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.855 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:35:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:35:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:35:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:35:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:40.856 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:35:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:35:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:35:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:35:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:35:40.857 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:35:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:35:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:35:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:35:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:35:40.858 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 276.85381248700003, 2023-01-15T14:35:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:35:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:40.859 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:35:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.860 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.861 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:35:40.862 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:35:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:35:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:35:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:35:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.863 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.864 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.865 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:35:40.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:35:40.866 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.866 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.866 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:40.866 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:35:40.866 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:35:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:35:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:35:40.901 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.902 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:35:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:35:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:35:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:35:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.903 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:35:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:35:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:35:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:35:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:35:40.904 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2301 3.16 3:68fb3263:::smithi09025264-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e216)", 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 237.56393801999999, 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.61754771200000003, 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:40.905 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2301 2023-01-15T14:35:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.906 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195531+0000", 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195530+0000", 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.907 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195535+0000", 2023-01-15T14:35:40.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3660000000000002e-06 2023-01-15T14:35:40.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195546+0000", 2023-01-15T14:35:40.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1062e-05 2023-01-15T14:35:40.908 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.908 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.908 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.908 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195551+0000", 2023-01-15T14:35:40.908 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5559999999999998e-06 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.195999+0000", 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044799800000000002 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.196106+0000", 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106172 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.909 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510631+0000", 2023-01-15T14:35:40.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31452575999999999 2023-01-15T14:35:40.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.910 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.510875+0000", 2023-01-15T14:35:40.910 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024415199999999998 2023-01-15T14:35:40.910 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.910 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.910 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812445+0000", 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30156923000000002 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.812727+0000", 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00028227800000000001 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:43.813079+0000", 2023-01-15T14:35:40.911 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199599999999998 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2749 3.3d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 189.83805324400001, 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.69206326600000001, 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:40.912 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2749 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.913 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:40.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:40.914 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:40.914 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.914 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.914 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.914 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921418+0000", 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1670000000000002e-06 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921426+0000", 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0569999999999993e-06 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921429+0000", 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9629999999999998e-06 2023-01-15T14:35:40.915 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107302+0000", 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.185873327 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107657+0000", 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035504299999999998 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.916 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212694+0000", 2023-01-15T14:35:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105037095 2023-01-15T14:35:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212936+0000", 2023-01-15T14:35:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241492 2023-01-15T14:35:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.917 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613125+0000", 2023-01-15T14:35:40.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40018927900000001 2023-01-15T14:35:40.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613294+0000", 2023-01-15T14:35:40.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016838000000000001 2023-01-15T14:35:40.918 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.918 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.918 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.918 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613479+0000", 2023-01-15T14:35:40.918 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018577599999999999 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3509 3.53 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (58) in=7b,truncate 2084167] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 111.64141010900001, 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70658069899999998, 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:40.919 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.920 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.920 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.920 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3509 2023-01-15T14:35:40.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.920 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.920 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.920 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:40.920 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.920 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.920 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.921 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118062+0000", 2023-01-15T14:35:40.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2260000000000001e-06 2023-01-15T14:35:40.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118067+0000", 2023-01-15T14:35:40.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3700000000000003e-06 2023-01-15T14:35:40.922 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.922 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.922 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.922 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118072+0000", 2023-01-15T14:35:40.922 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4880000000000001e-06 2023-01-15T14:35:40.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218253+0000", 2023-01-15T14:35:40.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10018108100000001 2023-01-15T14:35:40.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.923 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.923 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:40.923 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218258+0000", 2023-01-15T14:35:40.923 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.831e-06 2023-01-15T14:35:40.923 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321058+0000", 2023-01-15T14:35:40.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10280054499999999 2023-01-15T14:35:40.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.924 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321223+0000", 2023-01-15T14:35:40.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016474 2023-01-15T14:35:40.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323014+0000", 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017909550000000001 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323124+0000", 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110326 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:40.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323363+0000", 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238385 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323663+0000", 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029987699999999999 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323677+0000", 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4346e-05 2023-01-15T14:35:40.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:40.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323729+0000", 2023-01-15T14:35:40.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2049999999999998e-05 2023-01-15T14:35:40.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:40.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323775+0000", 2023-01-15T14:35:40.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5872999999999999e-05 2023-01-15T14:35:40.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.616791+0000", 2023-01-15T14:35:40.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29301572199999998 2023-01-15T14:35:40.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824267+0000", 2023-01-15T14:35:40.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20747585199999999 2023-01-15T14:35:40.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:40.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824425+0000", 2023-01-15T14:35:40.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015861900000000001 2023-01-15T14:35:40.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.929 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824640+0000", 2023-01-15T14:35:40.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021475899999999999 2023-01-15T14:35:40.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.929 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.929 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.930 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3510 3.53 3:cbfd2d71:::smithi09025264-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:40.930 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:40.930 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 111.64133227000001, 2023-01-15T14:35:40.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.80704639, 2023-01-15T14:35:40.930 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.930 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:40.930 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.930 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.930 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3510 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:40.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118135+0000", 2023-01-15T14:35:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118143+0000", 2023-01-15T14:35:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0579999999999998e-06 2023-01-15T14:35:40.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118162+0000", 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8702000000000002e-05 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118178+0000", 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5645e-05 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321268+0000", 2023-01-15T14:35:40.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20308996000000001 2023-01-15T14:35:40.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321370+0000", 2023-01-15T14:35:40.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010268399999999999 2023-01-15T14:35:40.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323825+0000", 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024546450000000001 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323934+0000", 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010940199999999999 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.924815+0000", 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60088053500000005 2023-01-15T14:35:40.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925009+0000", 2023-01-15T14:35:40.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019363599999999999 2023-01-15T14:35:40.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925183+0000", 2023-01-15T14:35:40.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000174823 2023-01-15T14:35:40.936 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3505 3.62 3:47dfe935:::smithi09025264-29:head [rollback db] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.74372956000001, 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95152285400000003, 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.937 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.938 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.938 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3505 2023-01-15T14:35:40.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.938 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:40.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:40.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015739+0000", 2023-01-15T14:35:40.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:40.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015742+0000", 2023-01-15T14:35:40.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.869e-06 2023-01-15T14:35:40.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015748+0000", 2023-01-15T14:35:40.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0360000000000003e-06 2023-01-15T14:35:40.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015754+0000", 2023-01-15T14:35:40.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.733e-06 2023-01-15T14:35:40.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318179+0000", 2023-01-15T14:35:40.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30242492300000001 2023-01-15T14:35:40.941 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.941 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.941 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318502+0000", 2023-01-15T14:35:40.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032355699999999998 2023-01-15T14:35:40.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:40.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365019+0000", 2023-01-15T14:35:40.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046516950000000001 2023-01-15T14:35:40.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862003+0000", 2023-01-15T14:35:40.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49698320800000001 2023-01-15T14:35:40.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862314+0000", 2023-01-15T14:35:40.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000311236 2023-01-15T14:35:40.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:40.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863083+0000", 2023-01-15T14:35:40.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076928399999999996 2023-01-15T14:35:40.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:40.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863709+0000", 2023-01-15T14:35:40.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062558700000000004 2023-01-15T14:35:40.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:40.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863724+0000", 2023-01-15T14:35:40.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5288999999999999e-05 2023-01-15T14:35:40.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863799+0000", 2023-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4552e-05 2023-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863851+0000", 2023-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2018999999999998e-05 2023-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966087+0000", 2023-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102235952 2023-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966811+0000", 2023-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072484800000000005 2023-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.946 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966950+0000", 2023-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013907399999999999 2023-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.967263+0000", 2023-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312107 2023-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:40.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3506 3.62 3:47dfe935:::smithi09025264-29:head [append 0~0] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 110.74370973800001, 2023-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.234729387, 2023-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:40.948 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3506 2023-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:40.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015759+0000", 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015763+0000", 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.517e-06 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015768+0000", 2023-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5620000000000003e-06 2023-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015773+0000", 2023-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3730000000000003e-06 2023-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365160+0000", 2023-01-15T14:35:40.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34938724300000001 2023-01-15T14:35:40.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:40.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365370+0000", 2023-01-15T14:35:40.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021001599999999999 2023-01-15T14:35:40.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:40.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863926+0000", 2023-01-15T14:35:40.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49855574600000002 2023-01-15T14:35:40.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:40.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.864284+0000", 2023-01-15T14:35:40.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035790600000000001 2023-01-15T14:35:40.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:40.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865099+0000", 2023-01-15T14:35:40.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081499999999999997 2023-01-15T14:35:40.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:40.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865640+0000", 2023-01-15T14:35:40.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054118899999999997 2023-01-15T14:35:40.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:40.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865653+0000", 2023-01-15T14:35:40.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3614e-05 2023-01-15T14:35:40.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:40.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865699+0000", 2023-01-15T14:35:40.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5751000000000003e-05 2023-01-15T14:35:40.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:40.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:40.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966206+0000", 2023-01-15T14:35:40.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100506741 2023-01-15T14:35:40.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:40.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:41.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966665+0000", 2023-01-15T14:35:41.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000459345 2023-01-15T14:35:41.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.063 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.063 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:41.063 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.249842+0000", 2023-01-15T14:35:41.063 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28317711200000001 2023-01-15T14:35:41.063 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:41.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250054+0000", 2023-01-15T14:35:41.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021161899999999999 2023-01-15T14:35:41.064 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.064 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.064 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:41.064 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250489+0000", 2023-01-15T14:35:41.064 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043499700000000001 2023-01-15T14:35:41.064 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:41.065 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:41.065 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:41.065 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.065 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.065 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3572 3.65 3:a75958b1:::smithi09025264-6:head [setxattr _header (57) in=7b,truncate 2235430] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:41.065 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:41.065 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 103.737079913, 2023-01-15T14:35:41.065 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65482807700000001, 2023-01-15T14:35:41.065 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:41.065 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:41.065 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:41.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:41.066 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:41.066 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3572 2023-01-15T14:35:41.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.066 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:41.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.066 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:41.066 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:41.066 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:41.066 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.066 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022388+0000", 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:41.067 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022397+0000", 2023-01-15T14:35:41.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.704e-06 2023-01-15T14:35:41.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:41.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022412+0000", 2023-01-15T14:35:41.068 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5554e-05 2023-01-15T14:35:41.068 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.068 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.068 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:41.068 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022422+0000", 2023-01-15T14:35:41.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2839999999999995e-06 2023-01-15T14:35:41.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:41.069 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114230+0000", 2023-01-15T14:35:41.069 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.091808129000000002 2023-01-15T14:35:41.069 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.069 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.069 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:41.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114234+0000", 2023-01-15T14:35:41.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4329999999999997e-06 2023-01-15T14:35:41.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:41.070 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204200+0000", 2023-01-15T14:35:41.070 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089966321000000002 2023-01-15T14:35:41.070 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.070 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.070 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:41.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204439+0000", 2023-01-15T14:35:41.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238595 2023-01-15T14:35:41.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:41.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.305883+0000", 2023-01-15T14:35:41.071 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101443713 2023-01-15T14:35:41.071 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.071 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.071 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:41.071 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306066+0000", 2023-01-15T14:35:41.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182962 2023-01-15T14:35:41.072 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.072 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.072 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2023-01-15T14:35:41.072 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306621+0000", 2023-01-15T14:35:41.072 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055552800000000001 2023-01-15T14:35:41.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:41.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307013+0000", 2023-01-15T14:35:41.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039210599999999998 2023-01-15T14:35:41.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.073 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.073 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:41.073 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307033+0000", 2023-01-15T14:35:41.073 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9423e-05 2023-01-15T14:35:41.073 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.074 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:41.074 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307093+0000", 2023-01-15T14:35:41.074 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0402000000000002e-05 2023-01-15T14:35:41.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307445+0000", 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000351677 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307662+0000", 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021765099999999999 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:41.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676523+0000", 2023-01-15T14:35:41.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36886056699999997 2023-01-15T14:35:41.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:41.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676723+0000", 2023-01-15T14:35:41.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019947899999999999 2023-01-15T14:35:41.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:41.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677217+0000", 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049493900000000001 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3573 3.65 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 103.737036786, 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65551266799999997, 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:41.077 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:41.078 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:41.078 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:41.078 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:41.078 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3573 2023-01-15T14:35:41.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.078 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:41.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:41.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:41.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:41.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:41.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:41.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:41.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:41.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022432+0000", 2023-01-15T14:35:41.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:41.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:41.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022436+0000", 2023-01-15T14:35:41.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.693e-06 2023-01-15T14:35:41.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.080 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:41.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022447+0000", 2023-01-15T14:35:41.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1639e-05 2023-01-15T14:35:41.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:41.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122543+0000", 2023-01-15T14:35:41.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100095369 2023-01-15T14:35:41.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:41.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122863+0000", 2023-01-15T14:35:41.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032020799999999999 2023-01-15T14:35:41.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122869+0000", 2023-01-15T14:35:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7640000000000002e-06 2023-01-15T14:35:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206080+0000", 2023-01-15T14:35:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.083211794000000006 2023-01-15T14:35:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:41.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206289+0000", 2023-01-15T14:35:41.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020866600000000001 2023-01-15T14:35:41.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:41.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307193+0000", 2023-01-15T14:35:41.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10090391999999999 2023-01-15T14:35:41.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:41.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307374+0000", 2023-01-15T14:35:41.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018061299999999999 2023-01-15T14:35:41.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:41.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677342+0000", 2023-01-15T14:35:41.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.369968034 2023-01-15T14:35:41.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:41.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677535+0000", 2023-01-15T14:35:41.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019311299999999999 2023-01-15T14:35:41.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:41.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677945+0000", 2023-01-15T14:35:41.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041040299999999999 2023-01-15T14:35:41.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:41.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:41.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:41.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.085 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4362 3.4 3:2022fdfa:::smithi09025264-46:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2023-01-15T14:35:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:36.985449+0000", 2023-01-15T14:35:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 3.7740200960000001, 2023-01-15T14:35:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71198219100000004, 2023-01-15T14:35:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4362 2023-01-15T14:35:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.086 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:41.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:41.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985449+0000", 2023-01-15T14:35:41.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:41.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:41.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985449+0000", 2023-01-15T14:35:41.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:41.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:41.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985447+0000", 2023-01-15T14:35:41.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:41.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:41.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985463+0000", 2023-01-15T14:35:41.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5761e-05 2023-01-15T14:35:41.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985492+0000", 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9677000000000001e-05 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985507+0000", 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4312e-05 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985646+0000", 2023-01-15T14:35:41.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013939599999999999 2023-01-15T14:35:41.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:35:41.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985662+0000", 2023-01-15T14:35:41.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5974999999999999e-05 2023-01-15T14:35:41.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:41.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.272912+0000", 2023-01-15T14:35:41.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28724966400000002 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.273577+0000", 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066540200000000005 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.274338+0000", 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076103800000000001 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:41.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.275507+0000", 2023-01-15T14:35:41.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011694380000000001 2023-01-15T14:35:41.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:41.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.275529+0000", 2023-01-15T14:35:41.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.177e-05 2023-01-15T14:35:41.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:41.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.275613+0000", 2023-01-15T14:35:41.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4050999999999999e-05 2023-01-15T14:35:41.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:41.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.275679+0000", 2023-01-15T14:35:41.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5425000000000005e-05 2023-01-15T14:35:41.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:41.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.490519+0000", 2023-01-15T14:35:41.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.214840436 2023-01-15T14:35:41.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:41.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.696746+0000", 2023-01-15T14:35:41.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20622658799999999 2023-01-15T14:35:41.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:41.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.696860+0000", 2023-01-15T14:35:41.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113955 2023-01-15T14:35:41.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:41.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.697431+0000", 2023-01-15T14:35:41.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057170300000000001 2023-01-15T14:35:41.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:41.095 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:41.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:41.095 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:41.095 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:41.345 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:41.346 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:40.758+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:41.347 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:35:41.348 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:41.340+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:41.450 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:41.444+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:41.558 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:41.558 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:41.558 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:35:41.558 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:41.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:35:41.559 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:41.552+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:41.659 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:41.652+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:41.758 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:41.758 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:41.758 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:41.758 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:41.758 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:41.759 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:35:41.760 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:41.752+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:41.862 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:41.856+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:41.952 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:41.953 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:41.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:41.953 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:41.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.953 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:41.953 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:41.953 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 306.98859679600002, 2023-01-15T14:35:41.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:35:41.953 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:41.953 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:41.953 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:35:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:41.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:41.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:41.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:41.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:35:41.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:41.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:41.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:35:41.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:35:41.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:41.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:35:41.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:35:41.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:41.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:35:41.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:35:41.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:41.958 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:35:41.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:35:41.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:41.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:35:41.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:35:41.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:41.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:35:41.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:35:41.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:41.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:35:41.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:35:41.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:35:41.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:35:41.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:35:41.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:41.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:35:41.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:35:41.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:41.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:35:41.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:35:41.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:41.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:35:41.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:35:41.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:41.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:35:41.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:35:41.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:41.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:35:41.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:35:41.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:41.965 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 306.98854585300001, 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:41.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:41.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:41.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:41.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:41.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:41.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:41.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.171 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.171 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.171 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.171 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:35:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:35:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:35:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:35:42.172 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.173 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:42.174 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:35:42.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:35:42.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:35:42.175 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:35:42.175 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.175 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.175 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.175 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:35:42.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:35:42.176 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.176 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.176 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.176 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:35:42.176 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:35:42.176 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.177 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.177 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.177 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.178 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.179 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:42.179 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:42.179 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 278.07991701899999, 2023-01-15T14:35:42.179 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:35:42.179 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.180 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:42.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:42.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:42.180 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:42.180 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:35:42.180 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.181 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.181 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.181 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.181 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:42.181 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.181 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:42.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:35:42.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:42.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:35:42.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:35:42.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:35:42.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:35:42.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:35:42.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:35:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:35:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:35:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.186 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:42.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:35:42.187 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:35:42.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:35:42.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:35:42.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:35:42.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:35:42.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:35:42.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:35:42.189 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.190 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.190 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.190 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:42.190 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:42.191 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 256.19288761000001, 2023-01-15T14:35:42.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:35:42.191 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.191 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:42.191 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:42.192 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:42.192 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:35:42.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.192 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:42.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:42.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:35:42.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:42.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:35:42.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:35:42.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:35:42.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:35:42.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:35:42.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:35:42.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.198 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.198 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.198 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:35:42.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:35:42.198 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:42.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:35:42.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:35:42.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:35:42.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:35:42.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:35:42.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:35:42.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:42.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:35:42.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:35:42.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:35:42.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:35:42.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:35:42.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:35:42.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:35:42.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:35:42.377 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:42.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:35:42.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:35:42.378 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.378 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.378 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.378 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:35:42.378 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:35:42.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:35:42.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:35:42.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.379 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.379 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:42.379 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:35:42.379 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:35:42.379 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.380 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.380 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:35:42.380 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:35:42.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.380 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.380 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.380 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.380 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.380 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:42.381 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:42.381 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 256.19278053599999, 2023-01-15T14:35:42.381 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:35:42.381 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.381 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:42.381 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:42.381 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:42.381 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:42.381 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:35:42.381 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.381 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:42.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.382 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.382 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:42.382 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.382 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.382 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.383 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:35:42.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:35:42.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.384 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:35:42.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:35:42.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:35:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:35:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:42.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:35:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:35:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:35:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:35:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:35:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:35:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:35:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:35:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:35:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:35:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:35:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:35:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:35:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 248.062367018, 2023-01-15T14:35:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:35:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:42.390 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:35:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:35:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:42.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:35:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:35:42.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:35:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:35:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:35:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:35:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:35:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:35:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:42.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:35:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:35:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:35:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:35:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:35:42.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:35:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:35:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:35:42.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:35:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:35:42.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:35:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:35:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:35:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:35:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:42.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:35:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:35:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:35:42.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:35:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:35:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:35:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:42.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:35:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:35:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:35:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:35:42.405 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:35:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 180.889255526, 2023-01-15T14:35:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:35:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.406 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:35:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:35:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:35:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:35:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:35:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:35:42.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:35:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:35:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:35:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:35:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:35:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:35:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:35:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:35:42.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:35:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:35:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:35:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:35:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:35:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:35:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:35:42.565 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 180.885443839, 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:35:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:35:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:35:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:35:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:35:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:35:42.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:35:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:35:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:35:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:35:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:35:42.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:35:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3120 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'290, mlcod=0'0)", 2023-01-15T14:35:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.64363056299999, 2023-01-15T14:35:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39853714000000001, 2023-01-15T14:35:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.572 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:42.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239424+0000", 2023-01-15T14:35:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239747+0000", 2023-01-15T14:35:42.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032243300000000001 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239784+0000", 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7051000000000002e-05 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239802+0000", 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8250999999999999e-05 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536033+0000", 2023-01-15T14:35:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29623121499999999 2023-01-15T14:35:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536081+0000", 2023-01-15T14:35:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2023-01-15T14:35:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536800+0000", 2023-01-15T14:35:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071948499999999996 2023-01-15T14:35:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536817+0000", 2023-01-15T14:35:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6228000000000001e-05 2023-01-15T14:35:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537048+0000", 2023-01-15T14:35:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023136200000000001 2023-01-15T14:35:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637902+0000", 2023-01-15T14:35:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10085403900000001 2023-01-15T14:35:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637963+0000", 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1310999999999996e-05 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3121 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'291, mlcod=0'0)", 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.64163504699999, 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659573799999998, 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.579 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241420+0000", 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241854+0000", 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043330099999999998 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241878+0000", 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4559e-05 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241892+0000", 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039999999999999e-05 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536863+0000", 2023-01-15T14:35:42.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29497073400000001 2023-01-15T14:35:42.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536887+0000", 2023-01-15T14:35:42.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3683999999999999e-05 2023-01-15T14:35:42.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537124+0000", 2023-01-15T14:35:42.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023729900000000001 2023-01-15T14:35:42.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537136+0000", 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2278e-05 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537345+0000", 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020865200000000001 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:42.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637981+0000", 2023-01-15T14:35:42.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063575700000001 2023-01-15T14:35:42.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.585 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638018+0000", 2023-01-15T14:35:42.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6826999999999998e-05 2023-01-15T14:35:42.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.585 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3122 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'292, mlcod=0'0)", 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.63973920699999, 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39623551600000001, 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:42.586 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243316+0000", 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243833+0000", 2023-01-15T14:35:42.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051703600000000001 2023-01-15T14:35:42.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243863+0000", 2023-01-15T14:35:42.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-01-15T14:35:42.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243879+0000", 2023-01-15T14:35:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6263999999999999e-05 2023-01-15T14:35:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637317+0000", 2023-01-15T14:35:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39343783199999999 2023-01-15T14:35:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.760 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637356+0000", 2023-01-15T14:35:42.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9579000000000002e-05 2023-01-15T14:35:42.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637830+0000", 2023-01-15T14:35:42.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047326399999999999 2023-01-15T14:35:42.761 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.761 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.761 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.761 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637885+0000", 2023-01-15T14:35:42.761 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5074999999999997e-05 2023-01-15T14:35:42.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638185+0000", 2023-01-15T14:35:42.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029993799999999997 2023-01-15T14:35:42.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.762 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.762 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:42.762 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638337+0000", 2023-01-15T14:35:42.762 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015208299999999999 2023-01-15T14:35:42.762 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.763 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.763 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.639553+0000", 2023-01-15T14:35:42.763 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001216389 2023-01-15T14:35:42.763 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.763 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.763 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.763 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.763 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.763 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3123 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'293, mlcod=0'0)", 2023-01-15T14:35:42.763 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:42.764 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 152.637995179, 2023-01-15T14:35:42.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.395733102, 2023-01-15T14:35:42.764 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.764 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.764 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.764 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.764 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:42.764 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.764 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.764 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:42.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.765 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245060+0000", 2023-01-15T14:35:42.765 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:42.765 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.765 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.765 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245075+0000", 2023-01-15T14:35:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4459e-05 2023-01-15T14:35:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245107+0000", 2023-01-15T14:35:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1795000000000003e-05 2023-01-15T14:35:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.766 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245121+0000", 2023-01-15T14:35:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4117000000000001e-05 2023-01-15T14:35:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.767 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638063+0000", 2023-01-15T14:35:42.924 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39294204300000002 2023-01-15T14:35:42.924 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.924 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.924 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638097+0000", 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4385999999999999e-05 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638299+0000", 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020133099999999999 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638318+0000", 2023-01-15T14:35:42.925 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514999999999999e-05 2023-01-15T14:35:42.926 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.926 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.926 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.926 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638386+0000", 2023-01-15T14:35:42.926 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8089000000000005e-05 2023-01-15T14:35:42.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.927 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.927 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:42.927 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638430+0000", 2023-01-15T14:35:42.927 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.439e-05 2023-01-15T14:35:42.927 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.640795+0000", 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002364274 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4216 3.78 3:1eaad226:::smithi09025264-49:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:42.928 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.919300432, 2023-01-15T14:35:42.929 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82042120600000001, 2023-01-15T14:35:42.929 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.929 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:42.929 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:42.929 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:42.929 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:42.929 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4216 2023-01-15T14:35:42.929 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.929 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.929 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.929 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:42.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:42.930 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.930 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.930 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.930 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.930 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963753+0000", 2023-01-15T14:35:42.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999962 2023-01-15T14:35:42.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.931 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.931 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.931 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963772+0000", 2023-01-15T14:35:42.931 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9293999999999999e-05 2023-01-15T14:35:42.931 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963806+0000", 2023-01-15T14:35:42.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3946999999999999e-05 2023-01-15T14:35:42.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.932 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.932 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963822+0000", 2023-01-15T14:35:42.932 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5744e-05 2023-01-15T14:35:42.932 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.932 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963930+0000", 2023-01-15T14:35:42.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010870300000000001 2023-01-15T14:35:42.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:35:42.933 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963946+0000", 2023-01-15T14:35:42.933 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5919000000000001e-05 2023-01-15T14:35:42.933 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.933 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.933 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.780699+0000", 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.81675266400000002 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784178+0000", 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034785699999999998 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.935 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4217 3.78 3:1eaad226:::smithi09025264-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:42.935 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:42.935 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 19.919204876999999, 2023-01-15T14:35:42.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.820584859, 2023-01-15T14:35:42.935 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.935 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:42.935 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:42.935 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:42.935 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:42.935 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4217 2023-01-15T14:35:42.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.936 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:42.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.936 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.936 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.936 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:42.936 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.936 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963851+0000", 2023-01-15T14:35:42.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:42.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.937 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.937 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.937 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963857+0000", 2023-01-15T14:35:42.937 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8699999999999997e-06 2023-01-15T14:35:42.937 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963867+0000", 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963876+0000", 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1120000000000002e-06 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.938 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064133+0000", 2023-01-15T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100256427 2023-01-15T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064164+0000", 2023-01-15T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1649000000000001e-05 2023-01-15T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.939 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784340+0000", 2023-01-15T14:35:42.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72017558699999995 2023-01-15T14:35:42.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.940 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.940 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.940 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784437+0000", 2023-01-15T14:35:42.940 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6831999999999997e-05 2023-01-15T14:35:42.940 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.940 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.940 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.940 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.941 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.941 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4216 3.78 3:1eaad226:::smithi09025264-49:head [copy-from ver 0] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:42.941 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:42.941 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.886194984999999, 2023-01-15T14:35:42.941 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39069526799999998, 2023-01-15T14:35:42.941 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.941 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:42.941 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:42.941 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:42.941 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:42.941 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4216 2023-01-15T14:35:42.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.942 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:42.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.942 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.942 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.942 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.942 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:42.942 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996860+0000", 2023-01-15T14:35:42.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:42.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.943 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.943 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.943 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996869+0000", 2023-01-15T14:35:42.943 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2069999999999995e-06 2023-01-15T14:35:42.943 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996881+0000", 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2611e-05 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.096971+0000", 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100089364 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.944 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.284700+0000", 2023-01-15T14:35:42.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18772960999999999 2023-01-15T14:35:42.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.284814+0000", 2023-01-15T14:35:42.945 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113427 2023-01-15T14:35:42.945 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.945 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.945 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.945 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.387557+0000", 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102743529 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4225 3.78 3:1e54a7f5:::smithi09025264-42:head [write 1381780~411527] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.389634003000001, 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42766908999999997, 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.946 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:42.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:42.947 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:42.947 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4225 2023-01-15T14:35:42.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.947 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.947 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.947 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.947 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:42.947 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.947 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:42.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.948 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.948 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493422+0000", 2023-01-15T14:35:42.948 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:42.948 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.948 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493846+0000", 2023-01-15T14:35:42.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042452100000000002 2023-01-15T14:35:42.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.949 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493861+0000", 2023-01-15T14:35:42.949 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5136e-05 2023-01-15T14:35:42.949 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.949 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.949 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493872+0000", 2023-01-15T14:35:42.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0958000000000001e-05 2023-01-15T14:35:42.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.950 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.950 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494867+0000", 2023-01-15T14:35:42.950 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00099453600000000007 2023-01-15T14:35:42.950 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.950 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:42.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495083+0000", 2023-01-15T14:35:42.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021589499999999999 2023-01-15T14:35:42.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.951 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.951 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.815757+0000", 2023-01-15T14:35:42.951 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32067462499999999 2023-01-15T14:35:42.951 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.951 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.815988+0000", 2023-01-15T14:35:42.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000230762 2023-01-15T14:35:42.952 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.952 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.952 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:42.952 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.816381+0000", 2023-01-15T14:35:42.952 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039289799999999998 2023-01-15T14:35:42.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817159+0000", 2023-01-15T14:35:42.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077780299999999998 2023-01-15T14:35:42.953 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.953 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.953 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.953 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817179+0000", 2023-01-15T14:35:42.953 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0489e-05 2023-01-15T14:35:42.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817418+0000", 2023-01-15T14:35:42.954 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023831500000000001 2023-01-15T14:35:42.954 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.954 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.954 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:42.954 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819234+0000", 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001816553 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820529+0000", 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012949719999999999 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920621+0000", 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100091965 2023-01-15T14:35:42.955 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920679+0000", 2023-01-15T14:35:42.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8319000000000002e-05 2023-01-15T14:35:42.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.956 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:42.956 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920795+0000", 2023-01-15T14:35:42.956 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115964 2023-01-15T14:35:42.956 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.956 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.957 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.957 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.921092+0000", 2023-01-15T14:35:42.957 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000296531 2023-01-15T14:35:42.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.957 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.957 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.957 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.957 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.957 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4226 3.78 3:1e54a7f5:::smithi09025264-42:head [write 2300501~586966] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:42.957 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:42.957 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.389142632999999, 2023-01-15T14:35:42.958 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.730699722, 2023-01-15T14:35:42.958 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.958 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:42.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:42.958 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:42.958 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4226 2023-01-15T14:35:42.958 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.958 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.958 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.958 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493913+0000", 2023-01-15T14:35:42.959 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:42.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494499+0000", 2023-01-15T14:35:42.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058589500000000001 2023-01-15T14:35:42.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.960 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.960 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.960 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494549+0000", 2023-01-15T14:35:42.960 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0296000000000001e-05 2023-01-15T14:35:42.960 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494577+0000", 2023-01-15T14:35:42.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8056999999999999e-05 2023-01-15T14:35:42.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.961 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.961 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495162+0000", 2023-01-15T14:35:42.961 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058508500000000005 2023-01-15T14:35:42.961 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.961 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495377+0000", 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021484400000000001 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817336+0000", 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32195888499999997 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.962 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817638+0000", 2023-01-15T14:35:42.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030184599999999998 2023-01-15T14:35:42.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:42.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.818285+0000", 2023-01-15T14:35:42.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064671399999999995 2023-01-15T14:35:42.963 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.963 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.963 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.963 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819118+0000", 2023-01-15T14:35:42.963 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083369499999999997 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819137+0000", 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8326999999999999e-05 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819385+0000", 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024822599999999999 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.964 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:42.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820443+0000", 2023-01-15T14:35:42.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058219 2023-01-15T14:35:42.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.965 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.923538+0000", 2023-01-15T14:35:42.965 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103094642 2023-01-15T14:35:42.965 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.965 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.965 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.020572+0000", 2023-01-15T14:35:42.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097034387999999999 2023-01-15T14:35:42.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224051+0000", 2023-01-15T14:35:42.966 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20347895399999999 2023-01-15T14:35:42.966 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.966 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.966 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:42.966 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224219+0000", 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016772 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224614+0000", 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039523500000000002 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.967 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4227 3.78 3:1e54a7f5:::smithi09025264-42:head [write 3376974~354279] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:42.968 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:42.968 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.38841833, 2023-01-15T14:35:42.968 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73066059400000005, 2023-01-15T14:35:42.968 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.968 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:42.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:42.968 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:42.968 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4227 2023-01-15T14:35:42.968 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.968 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.969 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494637+0000", 2023-01-15T14:35:42.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:42.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.970 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494805+0000", 2023-01-15T14:35:42.970 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016839000000000001 2023-01-15T14:35:42.970 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.970 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.970 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494819+0000", 2023-01-15T14:35:42.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4287000000000001e-05 2023-01-15T14:35:42.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494828+0000", 2023-01-15T14:35:42.971 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8119999999999997e-06 2023-01-15T14:35:42.971 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.971 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.971 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.971 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495478+0000", 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064994399999999996 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495668+0000", 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018991900000000001 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819291+0000", 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32362327600000002 2023-01-15T14:35:42.972 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819503+0000", 2023-01-15T14:35:42.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021184900000000001 2023-01-15T14:35:42.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.973 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:42.973 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819880+0000", 2023-01-15T14:35:42.973 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037692500000000001 2023-01-15T14:35:42.973 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.973 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.974 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.974 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820352+0000", 2023-01-15T14:35:42.974 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047233000000000003 2023-01-15T14:35:42.974 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820368+0000", 2023-01-15T14:35:42.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5877999999999999e-05 2023-01-15T14:35:42.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.975 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.975 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.975 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820533+0000", 2023-01-15T14:35:42.975 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164508 2023-01-15T14:35:42.975 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:42.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821690+0000", 2023-01-15T14:35:42.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011576290000000001 2023-01-15T14:35:42.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.976 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.976 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.020668+0000", 2023-01-15T14:35:42.976 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19897753800000001 2023-01-15T14:35:42.976 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.976 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.223918+0000", 2023-01-15T14:35:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20324972499999999 2023-01-15T14:35:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224796+0000", 2023-01-15T14:35:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000878655 2023-01-15T14:35:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.977 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:42.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224907+0000", 2023-01-15T14:35:42.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011042799999999999 2023-01-15T14:35:42.978 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.978 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.978 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:42.978 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225299+0000", 2023-01-15T14:35:42.978 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000392431 2023-01-15T14:35:42.978 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.978 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:42.978 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:42.979 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.979 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.979 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4228 3.78 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (58) in=7b,truncate 3731253] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:42.979 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:42.979 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.388215220999999, 2023-01-15T14:35:42.979 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73116634000000003, 2023-01-15T14:35:42.979 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:42.979 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:42.979 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:42.980 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:42.980 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:42.980 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4228 2023-01-15T14:35:42.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.980 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:42.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.980 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:42.980 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:42.980 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.980 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.980 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494841+0000", 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:42.981 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494846+0000", 2023-01-15T14:35:42.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4960000000000004e-06 2023-01-15T14:35:42.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:42.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494879+0000", 2023-01-15T14:35:42.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2376000000000003e-05 2023-01-15T14:35:42.982 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.982 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.982 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:42.982 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494887+0000", 2023-01-15T14:35:42.982 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9100000000000005e-06 2023-01-15T14:35:42.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.595858+0000", 2023-01-15T14:35:42.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100971879 2023-01-15T14:35:42.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.983 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.983 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:42.983 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596092+0000", 2023-01-15T14:35:42.983 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023309200000000001 2023-01-15T14:35:42.983 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820587+0000", 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22449530500000001 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820789+0000", 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000201802 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.984 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:42.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821144+0000", 2023-01-15T14:35:42.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035586199999999998 2023-01-15T14:35:42.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:42.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821591+0000", 2023-01-15T14:35:42.985 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000446458 2023-01-15T14:35:42.985 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.985 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.985 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:42.985 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821605+0000", 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4287000000000001e-05 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821653+0000", 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7781000000000002e-05 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821717+0000", 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3911999999999994e-05 2023-01-15T14:35:42.986 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:42.987 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:42.987 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.121593+0000", 2023-01-15T14:35:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29987650900000001 2023-01-15T14:35:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224737+0000", 2023-01-15T14:35:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103143918 2023-01-15T14:35:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225390+0000", 2023-01-15T14:35:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065281900000000001 2023-01-15T14:35:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225496+0000", 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106038 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.226008+0000", 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051175199999999998 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.086 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4229 3.78 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:43.086 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:43.086 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 18.388159675000001, 2023-01-15T14:35:43.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82811652800000002, 2023-01-15T14:35:43.086 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:43.086 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:43.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:43.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:43.086 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:43.086 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4229 2023-01-15T14:35:43.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:43.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494896+0000", 2023-01-15T14:35:43.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:43.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:43.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494901+0000", 2023-01-15T14:35:43.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8269999999999997e-06 2023-01-15T14:35:43.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494924+0000", 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3281999999999998e-05 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494936+0000", 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1457000000000001e-05 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:43.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596198+0000", 2023-01-15T14:35:43.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10126203 2023-01-15T14:35:43.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:43.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596400+0000", 2023-01-15T14:35:43.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020234499999999999 2023-01-15T14:35:43.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:43.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821768+0000", 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.225367977 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.822124+0000", 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035540600000000001 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.322208+0000", 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50008402100000005 2023-01-15T14:35:43.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:43.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.322487+0000", 2023-01-15T14:35:43.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027940400000000002 2023-01-15T14:35:43.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:43.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.323014+0000", 2023-01-15T14:35:43.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052676600000000002 2023-01-15T14:35:43.092 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:43.093 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:43.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:43.093 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:43.093 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:43.093 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:43.093 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:35:43.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:35:43.094 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:35:43.096 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:43.086+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:43.196 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:43.190+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:43.224 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:43.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPush(3.51 453/445 [PushOp(3:8b0a2647:::smithi09025264-8:11d, version: 428'340, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:8b0a2647:::smithi09025264-8:11d@428'340, size: 2100974, copy_subset: [0~2100974], clone_subset: {}, snapset: 149=[]:{fb=[f7,f6],11d=[117,115]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2097152, data_complete:false, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:35:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:43.154542+0000", 2023-01-15T14:35:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.066985817000000003, 2023-01-15T14:35:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.067046380000000003, 2023-01-15T14:35:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "no flag points reached", 2023-01-15T14:35:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:43.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:43.154542+0000", 2023-01-15T14:35:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:43.154542+0000", 2023-01-15T14:35:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.226 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:43.154542+0000", 2023-01-15T14:35:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:43.155116+0000", 2023-01-15T14:35:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057459399999999999 2023-01-15T14:35:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:43.227 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:43.155134+0000", 2023-01-15T14:35:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7652e-05 2023-01-15T14:35:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:35:43.228 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:35:43.228 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:43.237 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:35:43.249 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:43.272 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:43.291 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:43.297 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:43.292+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:43.308 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:43.328 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:43.348 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:43.348 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:43.351 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:43.358 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:35:43.373 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:43.397 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:43.397 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:43.390+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:35:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:35:43.398 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:35:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:35:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:35:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:35:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:35:43.399 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:35:43.489 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:43.490 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:43.498 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:43.492+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:43.500 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:35:43.599 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:43.594+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:43.622 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:43.622 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:43.632 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:35:43.700 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:43.692+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:43.740 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:43.740 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:43.750 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:35:43.801 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:43.794+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:43.849 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:43.849 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:43.858 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:35:43.902 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:43.896+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:43.965 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:43.965 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:43.974 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:35:44.003 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:43.994+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:44.059 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:35:44.112 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:44.106+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:44.127 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:44.127 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:44.137 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:35:44.205 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:44.198+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:44.236 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:44.236 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:44.246 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:35:44.305 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:44.300+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:44.345 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:44.350 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:44.356 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:35:44.408 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:44.398+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:44.507 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:44.498+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:44.509 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:44.509 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:44.519 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:35:44.608 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:44.602+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:44.626 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:44.627 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:44.636 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:35:44.709 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:44.702+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:44.744 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:44.744 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:44.754 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:35:44.810 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:44.804+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:44.857 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:44.857 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:44.867 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:35:44.911 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:44.904+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:44.971 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:44.981 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:35:45.012 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:45.004+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:45.029 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 7, 1, 6] out_osds: [5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:35:45.030 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:35:45.030 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:35:45.030 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:35:45.113 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:45.106+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:45.113 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:45.114 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:45.124 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:35:45.213 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:45.208+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:45.284 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:45.294 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:35:45.315 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:45.308+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:45.379 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:35:45.386 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:35:45.398 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2023-01-15T14:35:45.043379+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":389273542,"num_objects":173,"num_object_clones":119,"num_object_copies":519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":173,"num_whiteouts":14,"num_read":1306,"num_read_kb":1333027,"num_write":1060,"num_write_kb":368194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1111,"num_bytes_recovered":746655390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25715,"ondisk_log_size":25715,"up":429,"acting":425,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":412,"num_osds":6,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":562196280,"kb_used":6244388,"kb_used_data":555969672,"kb_used_omap":93000,"kb_used_meta":0,"kb_avail":555951892,"statfs":{"total":575688990720,"available":569294737408,"internally_reserved":18206720,"allocated":569312944128,"data_stored":569312944128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":95232823,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":8,"commit_latency_ns":0,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21829931,"num_objects":-8,"num_object_clones":-7,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-110,"num_objects_misplaced":-17,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":1,"num_read":2,"num_read_kb":-781,"num_write":12,"num_write_kb":1197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":69054396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.033718"},"pg_stats":[{"pgid":"3.87","version":"450'200","reported_seq":1457,"reported_epoch":454,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:44.765894+0000","last_change":"2023-01-15T14:35:44.765504+0000","last_active":"2023-01-15T14:35:44.765894+0000","last_peered":"2023-01-15T14:35:44.765894+0000","last_clean":"2023-01-15T14:35:44.765894+0000","last_became_active":"2023-01-15T14:35:44.663461+0000","last_became_peered":"2023-01-15T14:35:44.663461+0000","last_unstale":"2023-01-15T14:35:44.765894+0000","last_undegraded":"2023-01-15T14:35:44.765894+0000","last_fullsized":"2023-01-15T14:35:44.765894+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":8,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5689755,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":34583,"num_write":13,"num_write_kb":5140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1773172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.86","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:35:43.032771+0000","last_change":"2023-01-15T14:35:43.032771+0000","last_active":"2023-01-15T14:35:43.032771+0000","last_peered":"2023-01-15T14:35:43.032771+0000","last_clean":"2023-01-15T14:35:43.032771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:35:43.032771+0000","last_undegraded":"2023-01-15T14:35:43.032771+0000","last_fullsized":"2023-01-15T14:35:43.032771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:35:43.032771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:35:43.032771+0000","last_clean_scrub_stamp":"2023-01-15T14:35:43.032771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.85","version":"443'207","reported_seq":1565,"reported_epoch":454,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:44.450188+0000","last_change":"2023-01-15T14:35:44.449822+0000","last_active":"2023-01-15T14:35:44.450188+0000","last_peered":"2023-01-15T14:35:44.450188+0000","last_clean":"2023-01-15T14:35:44.450188+0000","last_became_active":"2023-01-15T14:35:44.354936+0000","last_became_peered":"2023-01-15T14:35:44.354936+0000","last_unstale":"2023-01-15T14:35:44.450188+0000","last_undegraded":"2023-01-15T14:35:44.450188+0000","last_fullsized":"2023-01-15T14:35:44.450188+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":8,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5687913,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":13074,"num_write":12,"num_write_kb":3966,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":13282155,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.84","version":"447'372","reported_seq":1858,"reported_epoch":453,"state":"unknown","last_fresh":"2023-01-15T14:35:43.147602+0000","last_change":"2023-01-15T14:35:43.706981+0000","last_active":"2023-01-15T14:35:43.147561+0000","last_peered":"2023-01-15T14:35:43.033016+0000","last_clean":"2023-01-15T14:35:33.746958+0000","last_became_active":"2023-01-15T14:35:36.497035+0000","last_became_peered":"2023-01-15T14:35:36.497035+0000","last_unstale":"2023-01-15T14:35:43.147602+0000","last_undegraded":"2023-01-15T14:35:43.147602+0000","last_fullsized":"2023-01-15T14:35:43.147602+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":8,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4170482,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":6,"num_read_kb":4895,"num_write":3,"num_write_kb":1888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2277448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.83","version":"444'351","reported_seq":1801,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.458280+0000","last_change":"2023-01-15T14:35:44.457746+0000","last_active":"2023-01-15T14:35:44.458280+0000","last_peered":"2023-01-15T14:35:44.458280+0000","last_clean":"2023-01-15T14:35:44.458280+0000","last_became_active":"2023-01-15T14:35:44.455508+0000","last_became_peered":"2023-01-15T14:35:44.455508+0000","last_unstale":"2023-01-15T14:35:44.458280+0000","last_undegraded":"2023-01-15T14:35:44.458280+0000","last_fullsized":"2023-01-15T14:35:44.458280+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":8,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2605168,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":12727,"num_write":3,"num_write_kb":3665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":14908172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.82","version":"155'101","reported_seq":1201,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.355866+0000","last_change":"2023-01-15T14:35:44.354465+0000","last_active":"2023-01-15T14:35:44.355866+0000","last_peered":"2023-01-15T14:35:44.355866+0000","last_clean":"2023-01-15T14:35:44.355866+0000","last_became_active":"2023-01-15T14:35:44.352733+0000","last_became_peered":"2023-01-15T14:35:44.352733+0000","last_unstale":"2023-01-15T14:35:44.355866+0000","last_undegraded":"2023-01-15T14:35:44.355866+0000","last_fullsized":"2023-01-15T14:35:44.355866+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":8,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.81","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:35:43.032771+0000","last_change":"2023-01-15T14:35:43.032771+0000","last_active":"2023-01-15T14:35:43.032771+0000","last_peered":"2023-01-15T14:35:43.032771+0000","last_clean":"2023-01-15T14:35:43.032771+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:35:43.032771+0000","last_undegraded":"2023-01-15T14:35:43.032771+0000","last_fullsized":"2023-01-15T14:35:43.032771+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:35:43.032771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:35:43.032771+0000","last_clean_scrub_stamp":"2023-01-15T14:35:43.032771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.80","version":"223'257","reported_seq":1234,"reported_epoch":453,"state":"unknown","last_fresh":"2023-01-15T14:35:43.146128+0000","last_change":"2023-01-15T14:35:43.146040+0000","last_active":"2023-01-15T14:35:43.146040+0000","last_peered":"2023-01-15T14:35:42.076582+0000","last_clean":"2023-01-15T14:35:42.076582+0000","last_became_active":"2023-01-15T14:35:38.174774+0000","last_became_peered":"2023-01-15T14:35:38.174774+0000","last_unstale":"2023-01-15T14:35:43.146128+0000","last_undegraded":"2023-01-15T14:35:43.146128+0000","last_fullsized":"2023-01-15T14:35:43.146128+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":8,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:35:01.091016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:01.091016+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55784813600000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.7f","version":"447'273","reported_seq":1679,"reported_epoch":454,"state":"activating","last_fresh":"2023-01-15T14:35:44.765966+0000","last_change":"2023-01-15T14:35:44.662585+0000","last_active":"2023-01-15T14:35:43.051511+0000","last_peered":"2023-01-15T14:35:42.288771+0000","last_clean":"2023-01-15T14:35:42.288771+0000","last_became_active":"2023-01-15T14:35:23.206778+0000","last_became_peered":"2023-01-15T14:35:23.206778+0000","last_unstale":"2023-01-15T14:35:44.765966+0000","last_undegraded":"2023-01-15T14:35:44.765966+0000","last_fullsized":"2023-01-15T14:35:44.765966+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":7,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11708859,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":14973,"num_write":18,"num_write_kb":7357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":13214874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":["7","0","6"],"object_location_counts":[{"shards":"0,6,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"}]},{"pgid":"3.7e","version":"448'86","reported_seq":248,"reported_epoch":453,"state":"unknown","last_fresh":"2023-01-15T14:35:43.047817+0000","last_change":"2023-01-15T14:35:43.047769+0000","last_active":"2023-01-15T14:35:43.047769+0000","last_peered":"2023-01-15T14:35:42.079476+0000","last_clean":"2023-01-15T14:35:42.079476+0000","last_became_active":"2023-01-15T14:35:04.273985+0000","last_became_peered":"2023-01-15T14:35:04.273985+0000","last_unstale":"2023-01-15T14:35:43.047817+0000","last_undegraded":"2023-01-15T14:35:43.047817+0000","last_fullsized":"2023-01-15T14:35:43.047817+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":7,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3420307,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":3026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":8496137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.7d","version":"424'277","reported_seq":1686,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.978556+0000","last_change":"2023-01-15T14:35:40.212462+0000","last_active":"2023-01-15T14:35:41.978556+0000","last_peered":"2023-01-15T14:35:41.978556+0000","last_clean":"2023-01-15T14:35:41.978556+0000","last_became_active":"2023-01-15T14:35:24.190356+0000","last_became_peered":"2023-01-15T14:35:24.190356+0000","last_unstale":"2023-01-15T14:35:41.978556+0000","last_undegraded":"2023-01-15T14:35:41.978556+0000","last_fullsized":"2023-01-15T14:35:41.978556+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":7,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.20682565,"stat_sum":{"num_bytes":2748011,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":18,"num_read_kb":20693,"num_write":10,"num_write_kb":4359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":7352992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.7c","version":"141'46","reported_seq":87,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.977772+0000","last_change":"2023-01-15T14:35:40.345194+0000","last_active":"2023-01-15T14:35:41.977772+0000","last_peered":"2023-01-15T14:35:41.977772+0000","last_clean":"2023-01-15T14:35:41.977772+0000","last_became_active":"2023-01-15T14:35:21.099039+0000","last_became_peered":"2023-01-15T14:35:21.099039+0000","last_unstale":"2023-01-15T14:35:41.977772+0000","last_undegraded":"2023-01-15T14:35:41.977772+0000","last_fullsized":"2023-01-15T14:35:41.977772+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.34847078100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.7b","version":"221'60","reported_seq":283,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.709735+0000","last_change":"2023-01-15T14:35:43.709735+0000","last_active":"2023-01-15T14:35:43.709735+0000","last_peered":"2023-01-15T14:35:43.709735+0000","last_clean":"2023-01-15T14:35:43.709735+0000","last_became_active":"2023-01-15T14:35:21.099347+0000","last_became_peered":"2023-01-15T14:35:21.099347+0000","last_unstale":"2023-01-15T14:35:43.709735+0000","last_undegraded":"2023-01-15T14:35:43.709735+0000","last_fullsized":"2023-01-15T14:35:43.709735+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.55538002900000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.7a","version":"223'238","reported_seq":1653,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.329438+0000","last_change":"2023-01-15T14:35:41.228580+0000","last_active":"2023-01-15T14:35:42.329438+0000","last_peered":"2023-01-15T14:35:42.329438+0000","last_clean":"2023-01-15T14:35:42.329438+0000","last_became_active":"2023-01-15T14:35:24.294312+0000","last_became_peered":"2023-01-15T14:35:24.294312+0000","last_unstale":"2023-01-15T14:35:42.329438+0000","last_undegraded":"2023-01-15T14:35:42.329438+0000","last_fullsized":"2023-01-15T14:35:42.329438+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":7,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.10487716,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.79","version":"451'463","reported_seq":1857,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.044233+0000","last_change":"2023-01-15T14:35:42.542610+0000","last_active":"2023-01-15T14:35:43.044233+0000","last_peered":"2023-01-15T14:35:43.044233+0000","last_clean":"2023-01-15T14:35:43.044233+0000","last_became_active":"2023-01-15T14:35:32.375313+0000","last_became_peered":"2023-01-15T14:35:32.375313+0000","last_unstale":"2023-01-15T14:35:43.044233+0000","last_undegraded":"2023-01-15T14:35:43.044233+0000","last_fullsized":"2023-01-15T14:35:43.044233+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":463,"log_dups_size":0,"ondisk_log_size":463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.46769129300000001,"stat_sum":{"num_bytes":6815725,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":70206,"num_write":12,"num_write_kb":6932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":13431307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.78","version":"451'364","reported_seq":1929,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.600515+0000","last_change":"2023-01-15T14:35:43.600515+0000","last_active":"2023-01-15T14:35:43.600515+0000","last_peered":"2023-01-15T14:35:43.600515+0000","last_clean":"2023-01-15T14:35:43.600515+0000","last_became_active":"2023-01-15T14:35:30.253853+0000","last_became_peered":"2023-01-15T14:35:30.253853+0000","last_unstale":"2023-01-15T14:35:43.600515+0000","last_undegraded":"2023-01-15T14:35:43.600515+0000","last_fullsized":"2023-01-15T14:35:43.600515+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.44885802299999999,"stat_sum":{"num_bytes":3248597,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":27,"num_read_kb":21425,"num_write":27,"num_write_kb":8453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":18795610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.77","version":"200'78","reported_seq":606,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.979709+0000","last_change":"2023-01-15T14:35:40.498076+0000","last_active":"2023-01-15T14:35:41.979709+0000","last_peered":"2023-01-15T14:35:41.979709+0000","last_clean":"2023-01-15T14:35:41.979709+0000","last_became_active":"2023-01-15T14:35:35.101268+0000","last_became_peered":"2023-01-15T14:35:35.101268+0000","last_unstale":"2023-01-15T14:35:41.979709+0000","last_undegraded":"2023-01-15T14:35:41.979709+0000","last_fullsized":"2023-01-15T14:35:41.979709+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":7,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.49080717699999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.76","version":"194'81","reported_seq":79,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.129428+0000","last_change":"2023-01-15T14:35:41.221035+0000","last_active":"2023-01-15T14:35:42.129428+0000","last_peered":"2023-01-15T14:35:42.129428+0000","last_clean":"2023-01-15T14:35:42.129428+0000","last_became_active":"2023-01-15T14:35:21.302673+0000","last_became_peered":"2023-01-15T14:35:21.302673+0000","last_unstale":"2023-01-15T14:35:42.129428+0000","last_undegraded":"2023-01-15T14:35:42.129428+0000","last_fullsized":"2023-01-15T14:35:42.129428+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:34:51.296526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.296526+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.66934546500000003,"objects_trimmed":0,"snaptrim_duration":0.10143555,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.31","version":"204'150","reported_seq":1011,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.977034+0000","last_change":"2023-01-15T14:35:40.498734+0000","last_active":"2023-01-15T14:35:41.977034+0000","last_peered":"2023-01-15T14:35:41.977034+0000","last_clean":"2023-01-15T14:35:41.977034+0000","last_became_active":"2023-01-15T14:34:56.391718+0000","last_became_peered":"2023-01-15T14:34:56.391718+0000","last_unstale":"2023-01-15T14:35:41.977034+0000","last_undegraded":"2023-01-15T14:35:41.977034+0000","last_fullsized":"2023-01-15T14:35:41.977034+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:22.439","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.40179741499999999,"stat_sum":{"num_bytes":3558907,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10724,"num_write":3,"num_write_kb":3089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3605110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.30","version":"206'252","reported_seq":1252,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.626394+0000","last_change":"2023-01-15T14:35:42.626549+0000","last_active":"2023-01-15T14:35:42.626394+0000","last_peered":"2023-01-15T14:35:42.626394+0000","last_clean":"2023-01-15T14:35:42.626394+0000","last_became_active":"2023-01-15T14:34:56.539037+0000","last_became_peered":"2023-01-15T14:34:56.539037+0000","last_unstale":"2023-01-15T14:35:42.626394+0000","last_undegraded":"2023-01-15T14:35:42.626394+0000","last_fullsized":"2023-01-15T14:35:42.626394+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:35:03.572774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:03.572774+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:18.492","scrub_duration":0.36120358800000002,"objects_trimmed":0,"snaptrim_duration":0.49752586399999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":1375,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.760368+0000","last_change":"2023-01-15T14:35:43.760368+0000","last_active":"2023-01-15T14:35:43.760368+0000","last_peered":"2023-01-15T14:35:43.760368+0000","last_clean":"2023-01-15T14:35:43.760368+0000","last_became_active":"2023-01-15T14:35:07.161199+0000","last_became_peered":"2023-01-15T14:35:07.161199+0000","last_unstale":"2023-01-15T14:35:43.760368+0000","last_undegraded":"2023-01-15T14:35:43.760368+0000","last_fullsized":"2023-01-15T14:35:43.760368+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.50760002500000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.2e","version":"139'38","reported_seq":202,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.652799+0000","last_change":"2023-01-15T14:35:44.653249+0000","last_active":"2023-01-15T14:35:44.652799+0000","last_peered":"2023-01-15T14:35:44.652799+0000","last_clean":"2023-01-15T14:35:44.652799+0000","last_became_active":"2023-01-15T14:35:41.428076+0000","last_became_peered":"2023-01-15T14:35:41.428076+0000","last_unstale":"2023-01-15T14:35:44.652799+0000","last_undegraded":"2023-01-15T14:35:44.652799+0000","last_fullsized":"2023-01-15T14:35:44.652799+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:35:30.876307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:30.876307+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.50737191699999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1495,"reported_epoch":454,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:44.551406+0000","last_change":"2023-01-15T14:35:44.551620+0000","last_active":"2023-01-15T14:35:44.551406+0000","last_peered":"2023-01-15T14:35:44.551406+0000","last_clean":"2023-01-15T14:35:44.551406+0000","last_became_active":"2023-01-15T14:35:38.280808+0000","last_became_peered":"2023-01-15T14:35:38.280808+0000","last_unstale":"2023-01-15T14:35:44.551406+0000","last_undegraded":"2023-01-15T14:35:44.551406+0000","last_fullsized":"2023-01-15T14:35:44.551406+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:00.115","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.30365924999999999,"stat_sum":{"num_bytes":1516832,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2709,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.2c","version":"141'46","reported_seq":609,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.045976+0000","last_change":"2023-01-15T14:35:41.542023+0000","last_active":"2023-01-15T14:35:43.045976+0000","last_peered":"2023-01-15T14:35:43.045976+0000","last_clean":"2023-01-15T14:35:43.045976+0000","last_became_active":"2023-01-15T14:35:41.525322+0000","last_became_peered":"2023-01-15T14:35:41.525322+0000","last_unstale":"2023-01-15T14:35:43.045976+0000","last_undegraded":"2023-01-15T14:35:43.045976+0000","last_fullsized":"2023-01-15T14:35:43.045976+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:58.535965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:58.535965+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:00.841","scrub_duration":0.104214843,"objects_trimmed":0,"snaptrim_duration":0.015410431,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":321,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.333694+0000","last_change":"2023-01-15T14:35:42.333791+0000","last_active":"2023-01-15T14:35:42.333694+0000","last_peered":"2023-01-15T14:35:42.333694+0000","last_clean":"2023-01-15T14:35:42.333694+0000","last_became_active":"2023-01-15T14:34:56.436012+0000","last_became_peered":"2023-01-15T14:34:56.436012+0000","last_unstale":"2023-01-15T14:35:42.333694+0000","last_undegraded":"2023-01-15T14:35:42.333694+0000","last_fullsized":"2023-01-15T14:35:42.333694+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:32.542","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.0016384909999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.2a","version":"191'191","reported_seq":1505,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.602285+0000","last_change":"2023-01-15T14:35:43.602285+0000","last_active":"2023-01-15T14:35:43.602285+0000","last_peered":"2023-01-15T14:35:43.602285+0000","last_clean":"2023-01-15T14:35:43.602285+0000","last_became_active":"2023-01-15T14:35:07.358010+0000","last_became_peered":"2023-01-15T14:35:07.358010+0000","last_unstale":"2023-01-15T14:35:43.602285+0000","last_undegraded":"2023-01-15T14:35:43.602285+0000","last_fullsized":"2023-01-15T14:35:43.602285+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.10401284099999999,"objects_trimmed":0,"snaptrim_duration":0.45002706999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.29","version":"110'149","reported_seq":776,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.146805+0000","last_change":"2023-01-15T14:35:40.209831+0000","last_active":"2023-01-15T14:35:43.146805+0000","last_peered":"2023-01-15T14:35:43.146805+0000","last_clean":"2023-01-15T14:35:43.146805+0000","last_became_active":"2023-01-15T14:35:11.345959+0000","last_became_peered":"2023-01-15T14:35:11.345959+0000","last_unstale":"2023-01-15T14:35:43.146805+0000","last_undegraded":"2023-01-15T14:35:43.146805+0000","last_fullsized":"2023-01-15T14:35:43.146805+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:34.348","scrub_duration":0.40398525099999999,"objects_trimmed":0,"snaptrim_duration":0.206213487,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.28","version":"333'248","reported_seq":1639,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:42.075978+0000","last_change":"2023-01-15T14:35:40.446049+0000","last_active":"2023-01-15T14:35:42.075978+0000","last_peered":"2023-01-15T14:35:42.075978+0000","last_clean":"2023-01-15T14:35:42.075978+0000","last_became_active":"2023-01-15T14:35:38.080781+0000","last_became_peered":"2023-01-15T14:35:38.080781+0000","last_unstale":"2023-01-15T14:35:42.075978+0000","last_undegraded":"2023-01-15T14:35:42.075978+0000","last_fullsized":"2023-01-15T14:35:42.075978+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'248","last_scrub_stamp":"2023-01-15T14:34:20.620068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.620068+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40448800800000001,"objects_trimmed":0,"snaptrim_duration":0.42151634599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":742,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.051436+0000","last_change":"2023-01-15T14:35:40.209153+0000","last_active":"2023-01-15T14:35:43.051436+0000","last_peered":"2023-01-15T14:35:43.051436+0000","last_clean":"2023-01-15T14:35:43.051436+0000","last_became_active":"2023-01-15T14:35:34.996509+0000","last_became_peered":"2023-01-15T14:35:34.996509+0000","last_unstale":"2023-01-15T14:35:43.051436+0000","last_undegraded":"2023-01-15T14:35:43.051436+0000","last_fullsized":"2023-01-15T14:35:43.051436+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.20642961600000001,"stat_sum":{"num_bytes":1242779,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":4228,"num_write":1,"num_write_kb":368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.26","version":"102'52","reported_seq":134,"reported_epoch":450,"state":"active+clean","last_fresh":"2023-01-15T14:35:40.302675+0000","last_change":"2023-01-15T14:35:40.302675+0000","last_active":"2023-01-15T14:35:40.302675+0000","last_peered":"2023-01-15T14:35:40.302675+0000","last_clean":"2023-01-15T14:35:40.302675+0000","last_became_active":"2023-01-15T14:35:04.285753+0000","last_became_peered":"2023-01-15T14:35:04.285753+0000","last_unstale":"2023-01-15T14:35:40.302675+0000","last_undegraded":"2023-01-15T14:35:40.302675+0000","last_fullsized":"2023-01-15T14:35:40.302675+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:29.161","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.30375046,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"db","length":"1"},{"start":"f2","length":"1"},{"start":"fc","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.25","version":"333'108","reported_seq":1276,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:42.076569+0000","last_change":"2023-01-15T14:35:40.853763+0000","last_active":"2023-01-15T14:35:42.076569+0000","last_peered":"2023-01-15T14:35:42.076569+0000","last_clean":"2023-01-15T14:35:42.076569+0000","last_became_active":"2023-01-15T14:35:38.285989+0000","last_became_peered":"2023-01-15T14:35:38.285989+0000","last_unstale":"2023-01-15T14:35:42.076569+0000","last_undegraded":"2023-01-15T14:35:42.076569+0000","last_fullsized":"2023-01-15T14:35:42.076569+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'108","last_scrub_stamp":"2023-01-15T14:35:02.975751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:02.975751+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:08.244","scrub_duration":0.30430094200000002,"objects_trimmed":0,"snaptrim_duration":0.754968101,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.24","version":"449'307","reported_seq":1708,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.046206+0000","last_change":"2023-01-15T14:35:40.216887+0000","last_active":"2023-01-15T14:35:43.046206+0000","last_peered":"2023-01-15T14:35:43.046206+0000","last_clean":"2023-01-15T14:35:43.046206+0000","last_became_active":"2023-01-15T14:35:11.349162+0000","last_became_peered":"2023-01-15T14:35:11.349162+0000","last_unstale":"2023-01-15T14:35:43.046206+0000","last_undegraded":"2023-01-15T14:35:43.046206+0000","last_fullsized":"2023-01-15T14:35:43.046206+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:35.350","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.21294928900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":20,"num_read_kb":22662,"num_write":11,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.23","version":"149'233","reported_seq":994,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.231176+0000","last_change":"2023-01-15T14:35:41.533012+0000","last_active":"2023-01-15T14:35:42.231176+0000","last_peered":"2023-01-15T14:35:42.231176+0000","last_clean":"2023-01-15T14:35:42.231176+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:35:42.231176+0000","last_undegraded":"2023-01-15T14:35:42.231176+0000","last_fullsized":"2023-01-15T14:35:42.231176+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:34:55.674244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:55.674244+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:18.674","scrub_duration":0.50374311999999999,"objects_trimmed":0,"snaptrim_duration":0.41179454399999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.22","version":"323'161","reported_seq":1404,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.432553+0000","last_change":"2023-01-15T14:35:42.432742+0000","last_active":"2023-01-15T14:35:42.432553+0000","last_peered":"2023-01-15T14:35:42.432553+0000","last_clean":"2023-01-15T14:35:42.432553+0000","last_became_active":"2023-01-15T14:35:11.447340+0000","last_became_peered":"2023-01-15T14:35:11.447340+0000","last_unstale":"2023-01-15T14:35:42.432553+0000","last_undegraded":"2023-01-15T14:35:42.432553+0000","last_fullsized":"2023-01-15T14:35:42.432553+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.30361324899999997,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.21","version":"321'138","reported_seq":1200,"reported_epoch":454,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:44.368133+0000","last_change":"2023-01-15T14:35:44.368171+0000","last_active":"2023-01-15T14:35:44.368133+0000","last_peered":"2023-01-15T14:35:44.368133+0000","last_clean":"2023-01-15T14:35:44.368133+0000","last_became_active":"2023-01-15T14:35:41.235242+0000","last_became_peered":"2023-01-15T14:35:41.235242+0000","last_unstale":"2023-01-15T14:35:44.368133+0000","last_undegraded":"2023-01-15T14:35:44.368133+0000","last_fullsized":"2023-01-15T14:35:44.368133+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"321'138","last_scrub_stamp":"2023-01-15T14:34:49.495975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.495975+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:55.426","scrub_duration":0.40443773500000002,"objects_trimmed":0,"snaptrim_duration":0.121954167,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.20","version":"324'273","reported_seq":1522,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.129070+0000","last_change":"2023-01-15T14:35:41.623075+0000","last_active":"2023-01-15T14:35:42.129070+0000","last_peered":"2023-01-15T14:35:42.129070+0000","last_clean":"2023-01-15T14:35:42.129070+0000","last_became_active":"2023-01-15T14:35:41.521596+0000","last_became_peered":"2023-01-15T14:35:41.521596+0000","last_unstale":"2023-01-15T14:35:42.129070+0000","last_undegraded":"2023-01-15T14:35:42.129070+0000","last_fullsized":"2023-01-15T14:35:42.129070+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"324'273","last_scrub_stamp":"2023-01-15T14:34:49.100502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.100502+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:05.063","scrub_duration":0.204192292,"objects_trimmed":0,"snaptrim_duration":0.60875908499999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.1f","version":"449'311","reported_seq":1729,"reported_epoch":452,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:35:42.529663+0000","last_change":"2023-01-15T14:35:42.630615+0000","last_active":"2023-01-15T14:35:42.529663+0000","last_peered":"2023-01-15T14:35:42.529663+0000","last_clean":"2023-01-15T14:35:33.019907+0000","last_became_active":"2023-01-15T14:35:36.301492+0000","last_became_peered":"2023-01-15T14:35:36.301492+0000","last_unstale":"2023-01-15T14:35:42.529663+0000","last_undegraded":"2023-01-15T14:35:35.996707+0000","last_fullsized":"2023-01-15T14:35:35.995663+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:59.076","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.77562913200000005,"stat_sum":{"num_bytes":9370202,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":43,"num_read_kb":52432,"num_write":24,"num_write_kb":7801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":29785074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6],"avail_no_missing":["0","2","4","6"],"object_location_counts":[{"shards":"0,6","objects":5}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.1e","version":"215'39","reported_seq":1242,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.148681+0000","last_change":"2023-01-15T14:35:41.629509+0000","last_active":"2023-01-15T14:35:43.148681+0000","last_peered":"2023-01-15T14:35:43.148681+0000","last_clean":"2023-01-15T14:35:43.148681+0000","last_became_active":"2023-01-15T14:35:41.422705+0000","last_became_peered":"2023-01-15T14:35:41.422705+0000","last_unstale":"2023-01-15T14:35:43.148681+0000","last_undegraded":"2023-01-15T14:35:43.148681+0000","last_fullsized":"2023-01-15T14:35:43.148681+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.20544089800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1586,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.146333+0000","last_change":"2023-01-15T14:35:40.212540+0000","last_active":"2023-01-15T14:35:43.146333+0000","last_peered":"2023-01-15T14:35:43.146333+0000","last_clean":"2023-01-15T14:35:43.146333+0000","last_became_active":"2023-01-15T14:35:07.268393+0000","last_became_peered":"2023-01-15T14:35:07.268393+0000","last_unstale":"2023-01-15T14:35:43.146333+0000","last_undegraded":"2023-01-15T14:35:43.146333+0000","last_fullsized":"2023-01-15T14:35:43.146333+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":5,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:51.945865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.945865+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:04.107","scrub_duration":0.30386408399999998,"objects_trimmed":0,"snaptrim_duration":0.208565941,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":840,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.046427+0000","last_change":"2023-01-15T14:35:40.213543+0000","last_active":"2023-01-15T14:35:43.046427+0000","last_peered":"2023-01-15T14:35:43.046427+0000","last_clean":"2023-01-15T14:35:43.046427+0000","last_became_active":"2023-01-15T14:35:04.384767+0000","last_became_peered":"2023-01-15T14:35:04.384767+0000","last_unstale":"2023-01-15T14:35:43.046427+0000","last_undegraded":"2023-01-15T14:35:43.046427+0000","last_fullsized":"2023-01-15T14:35:43.046427+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":5,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:22.440826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:22.440826+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30343327599999997,"objects_trimmed":0,"snaptrim_duration":0.20932579200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.1b","version":"450'153","reported_seq":366,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.331869+0000","last_change":"2023-01-15T14:35:41.464471+0000","last_active":"2023-01-15T14:35:42.331869+0000","last_peered":"2023-01-15T14:35:42.331869+0000","last_clean":"2023-01-15T14:35:42.331869+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:35:42.331869+0000","last_undegraded":"2023-01-15T14:35:42.331869+0000","last_fullsized":"2023-01-15T14:35:42.331869+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"370'101","last_scrub_stamp":"2023-01-15T14:34:24.137270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:24.137270+0000","objects_scrubbed":3,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:44.214","scrub_duration":0.106791863,"objects_trimmed":0,"snaptrim_duration":0.24636550199999999,"stat_sum":{"num_bytes":7859849,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":13300,"num_write":8,"num_write_kb":4287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.1a","version":"443'388","reported_seq":1829,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.460225+0000","last_change":"2023-01-15T14:35:44.460601+0000","last_active":"2023-01-15T14:35:44.460225+0000","last_peered":"2023-01-15T14:35:44.460225+0000","last_clean":"2023-01-15T14:35:44.460225+0000","last_became_active":"2023-01-15T14:35:44.351248+0000","last_became_peered":"2023-01-15T14:35:44.351248+0000","last_unstale":"2023-01-15T14:35:44.460225+0000","last_undegraded":"2023-01-15T14:35:44.460225+0000","last_fullsized":"2023-01-15T14:35:44.460225+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":5,"last_scrub":"375'356","last_scrub_stamp":"2023-01-15T14:34:29.460843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:29.460843+0000","objects_scrubbed":4,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30741503599999997,"objects_trimmed":0,"snaptrim_duration":0.0066213590000000003,"stat_sum":{"num_bytes":9225015,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":16,"num_write_kb":5138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":9225015,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.19","version":"448'282","reported_seq":1605,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.977310+0000","last_change":"2023-01-15T14:35:40.499298+0000","last_active":"2023-01-15T14:35:41.977310+0000","last_peered":"2023-01-15T14:35:41.977310+0000","last_clean":"2023-01-15T14:35:41.977310+0000","last_became_active":"2023-01-15T14:34:43.263173+0000","last_became_peered":"2023-01-15T14:34:43.263173+0000","last_unstale":"2023-01-15T14:35:41.977310+0000","last_undegraded":"2023-01-15T14:35:41.977310+0000","last_fullsized":"2023-01-15T14:35:41.977310+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.40173102999999999,"stat_sum":{"num_bytes":4444263,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":12320,"num_write":9,"num_write_kb":2852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4941423,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.18","version":"444'360","reported_seq":1856,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.727596+0000","last_change":"2023-01-15T14:35:42.727745+0000","last_active":"2023-01-15T14:35:42.727596+0000","last_peered":"2023-01-15T14:35:42.727596+0000","last_clean":"2023-01-15T14:35:42.727596+0000","last_became_active":"2023-01-15T14:35:11.544123+0000","last_became_peered":"2023-01-15T14:35:11.544123+0000","last_unstale":"2023-01-15T14:35:42.727596+0000","last_undegraded":"2023-01-15T14:35:42.727596+0000","last_fullsized":"2023-01-15T14:35:42.727596+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":5,"last_scrub":"384'341","last_scrub_stamp":"2023-01-15T14:34:35.477211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:35.477211+0000","objects_scrubbed":5,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:54.983","scrub_duration":0.54215560600000001,"objects_trimmed":0,"snaptrim_duration":0.59814132900000005,"stat_sum":{"num_bytes":2974905,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":310,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.226007+0000","last_change":"2023-01-15T14:35:41.330539+0000","last_active":"2023-01-15T14:35:42.226007+0000","last_peered":"2023-01-15T14:35:42.226007+0000","last_clean":"2023-01-15T14:35:42.226007+0000","last_became_active":"2023-01-15T14:35:04.289453+0000","last_became_peered":"2023-01-15T14:35:04.289453+0000","last_unstale":"2023-01-15T14:35:42.226007+0000","last_undegraded":"2023-01-15T14:35:42.226007+0000","last_fullsized":"2023-01-15T14:35:42.226007+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:30.569","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.20826753200000001,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.16","version":"444'157","reported_seq":1001,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.607582+0000","last_change":"2023-01-15T14:35:43.607582+0000","last_active":"2023-01-15T14:35:43.607582+0000","last_peered":"2023-01-15T14:35:43.607582+0000","last_clean":"2023-01-15T14:35:43.607582+0000","last_became_active":"2023-01-15T14:34:44.491303+0000","last_became_peered":"2023-01-15T14:34:44.491303+0000","last_unstale":"2023-01-15T14:35:43.607582+0000","last_undegraded":"2023-01-15T14:35:43.607582+0000","last_fullsized":"2023-01-15T14:35:43.607582+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.55594299000000003,"stat_sum":{"num_bytes":5025425,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6898,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9731517,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":263,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.605899+0000","last_change":"2023-01-15T14:35:43.605899+0000","last_active":"2023-01-15T14:35:43.605899+0000","last_peered":"2023-01-15T14:35:43.605899+0000","last_clean":"2023-01-15T14:35:43.605899+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:35:43.605899+0000","last_undegraded":"2023-01-15T14:35:43.605899+0000","last_fullsized":"2023-01-15T14:35:43.605899+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.552403278,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.14","version":"445'181","reported_seq":1504,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.151621+0000","last_change":"2023-01-15T14:35:40.214338+0000","last_active":"2023-01-15T14:35:43.151621+0000","last_peered":"2023-01-15T14:35:43.151621+0000","last_clean":"2023-01-15T14:35:43.151621+0000","last_became_active":"2023-01-15T14:35:14.050115+0000","last_became_peered":"2023-01-15T14:35:14.050115+0000","last_unstale":"2023-01-15T14:35:43.151621+0000","last_undegraded":"2023-01-15T14:35:43.151621+0000","last_fullsized":"2023-01-15T14:35:43.151621+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"351'159","last_scrub_stamp":"2023-01-15T14:34:18.607751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:18.607751+0000","objects_scrubbed":2,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40666956999999998,"objects_trimmed":0,"snaptrim_duration":0.20971284800000001,"stat_sum":{"num_bytes":2453855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2453855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":90,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.152258+0000","last_change":"2023-01-15T14:35:41.423046+0000","last_active":"2023-01-15T14:35:43.152258+0000","last_peered":"2023-01-15T14:35:43.152258+0000","last_clean":"2023-01-15T14:35:43.152258+0000","last_became_active":"2023-01-15T14:35:41.421509+0000","last_became_peered":"2023-01-15T14:35:41.421509+0000","last_unstale":"2023-01-15T14:35:43.152258+0000","last_undegraded":"2023-01-15T14:35:43.152258+0000","last_fullsized":"2023-01-15T14:35:43.152258+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":301,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.830097+0000","last_change":"2023-01-15T14:35:42.830288+0000","last_active":"2023-01-15T14:35:42.830097+0000","last_peered":"2023-01-15T14:35:42.830097+0000","last_clean":"2023-01-15T14:35:42.830097+0000","last_became_active":"2023-01-15T14:34:43.461395+0000","last_became_peered":"2023-01-15T14:34:43.461395+0000","last_unstale":"2023-01-15T14:35:42.830097+0000","last_undegraded":"2023-01-15T14:35:42.830097+0000","last_fullsized":"2023-01-15T14:35:42.830097+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.69957312000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":81,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:42.180684+0000","last_change":"2023-01-15T14:35:04.384333+0000","last_active":"2023-01-15T14:35:42.180684+0000","last_peered":"2023-01-15T14:35:42.180684+0000","last_clean":"2023-01-15T14:35:42.180684+0000","last_became_active":"2023-01-15T14:35:04.383307+0000","last_became_peered":"2023-01-15T14:35:04.383307+0000","last_unstale":"2023-01-15T14:35:42.180684+0000","last_undegraded":"2023-01-15T14:35:42.180684+0000","last_fullsized":"2023-01-15T14:35:42.180684+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"450'200","reported_seq":1454,"reported_epoch":454,"state":"active","last_fresh":"2023-01-15T14:35:44.762050+0000","last_change":"2023-01-15T14:35:44.762050+0000","last_active":"2023-01-15T14:35:44.762050+0000","last_peered":"2023-01-15T14:35:44.762050+0000","last_clean":"2023-01-15T14:35:43.000163+0000","last_became_active":"2023-01-15T14:35:44.762050+0000","last_became_peered":"2023-01-15T14:35:44.762050+0000","last_unstale":"2023-01-15T14:35:44.762050+0000","last_undegraded":"2023-01-15T14:35:44.762050+0000","last_fullsized":"2023-01-15T14:35:44.762050+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":367,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.91326063099999999,"stat_sum":{"num_bytes":5689754,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":34583,"num_write":13,"num_write_kb":5139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":1773171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":502,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.252897+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:35:43.252897+0000","last_peered":"2023-01-15T14:35:43.252897+0000","last_clean":"2023-01-15T14:35:43.252897+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:35:43.252897+0000","last_undegraded":"2023-01-15T14:35:43.252897+0000","last_fullsized":"2023-01-15T14:35:43.252897+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"447'372","reported_seq":1858,"reported_epoch":453,"state":"remapped+peering","last_fresh":"2023-01-15T14:35:43.147602+0000","last_change":"2023-01-15T14:35:43.149598+0000","last_active":"2023-01-15T14:35:43.147561+0000","last_peered":"2023-01-15T14:35:43.033016+0000","last_clean":"2023-01-15T14:35:33.746958+0000","last_became_active":"2023-01-15T14:35:36.497035+0000","last_became_peered":"2023-01-15T14:35:36.497035+0000","last_unstale":"2023-01-15T14:35:43.147602+0000","last_undegraded":"2023-01-15T14:35:43.147602+0000","last_fullsized":"2023-01-15T14:35:43.147602+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0.77882038499999995,"stat_sum":{"num_bytes":4170482,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":4894,"num_write":3,"num_write_kb":1887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2277447,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2,6],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":71,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.978353+0000","last_change":"2023-01-15T14:34:43.371445+0000","last_active":"2023-01-15T14:35:41.978353+0000","last_peered":"2023-01-15T14:35:41.978353+0000","last_clean":"2023-01-15T14:35:41.978353+0000","last_became_active":"2023-01-15T14:34:43.253700+0000","last_became_peered":"2023-01-15T14:34:43.253700+0000","last_unstale":"2023-01-15T14:35:41.978353+0000","last_undegraded":"2023-01-15T14:35:41.978353+0000","last_fullsized":"2023-01-15T14:35:41.978353+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"443'207","reported_seq":1565,"reported_epoch":454,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:44.456531+0000","last_change":"2023-01-15T14:35:44.356218+0000","last_active":"2023-01-15T14:35:44.456531+0000","last_peered":"2023-01-15T14:35:44.456531+0000","last_clean":"2023-01-15T14:35:44.456531+0000","last_became_active":"2023-01-15T14:35:44.352091+0000","last_became_peered":"2023-01-15T14:35:44.352091+0000","last_unstale":"2023-01-15T14:35:44.456531+0000","last_undegraded":"2023-01-15T14:35:44.456531+0000","last_fullsized":"2023-01-15T14:35:44.456531+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.91761946299999997,"stat_sum":{"num_bytes":5687913,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":13074,"num_write":11,"num_write_kb":3965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":13282154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.3","version":"444'351","reported_seq":1800,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.253328+0000","last_change":"2023-01-15T14:35:44.253328+0000","last_active":"2023-01-15T14:35:44.253328+0000","last_peered":"2023-01-15T14:35:44.253328+0000","last_clean":"2023-01-15T14:35:44.253328+0000","last_became_active":"2023-01-15T14:35:44.252095+0000","last_became_peered":"2023-01-15T14:35:44.252095+0000","last_unstale":"2023-01-15T14:35:44.253328+0000","last_undegraded":"2023-01-15T14:35:44.253328+0000","last_fullsized":"2023-01-15T14:35:44.253328+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.20452356799999999,"stat_sum":{"num_bytes":2605167,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12727,"num_write":3,"num_write_kb":3665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":14908172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":81,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.976668+0000","last_change":"2023-01-15T14:35:04.086793+0000","last_active":"2023-01-15T14:35:41.976668+0000","last_peered":"2023-01-15T14:35:41.976668+0000","last_clean":"2023-01-15T14:35:41.976668+0000","last_became_active":"2023-01-15T14:35:04.086427+0000","last_became_peered":"2023-01-15T14:35:04.086427+0000","last_unstale":"2023-01-15T14:35:41.976668+0000","last_undegraded":"2023-01-15T14:35:41.976668+0000","last_fullsized":"2023-01-15T14:35:41.976668+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1234,"reported_epoch":453,"state":"peering","last_fresh":"2023-01-15T14:35:43.146128+0000","last_change":"2023-01-15T14:35:43.147235+0000","last_active":"2023-01-15T14:35:43.146040+0000","last_peered":"2023-01-15T14:35:42.076582+0000","last_clean":"2023-01-15T14:35:42.076582+0000","last_became_active":"2023-01-15T14:35:38.174774+0000","last_became_peered":"2023-01-15T14:35:38.174774+0000","last_unstale":"2023-01-15T14:35:43.146128+0000","last_undegraded":"2023-01-15T14:35:43.146128+0000","last_fullsized":"2023-01-15T14:35:43.146128+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:35:01.091016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:01.091016+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:17.313","scrub_duration":0.55784813600000005,"objects_trimmed":0,"snaptrim_duration":0.52207260899999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":59,"reported_epoch":451,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.220042+0000","last_change":"2023-01-15T14:35:35.298341+0000","last_active":"2023-01-15T14:35:41.220042+0000","last_peered":"2023-01-15T14:35:41.220042+0000","last_clean":"2023-01-15T14:35:41.220042+0000","last_became_active":"2023-01-15T14:35:35.296662+0000","last_became_peered":"2023-01-15T14:35:35.296662+0000","last_unstale":"2023-01-15T14:35:41.220042+0000","last_undegraded":"2023-01-15T14:35:41.220042+0000","last_fullsized":"2023-01-15T14:35:41.220042+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":957,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.334606+0000","last_change":"2023-01-15T14:35:42.334745+0000","last_active":"2023-01-15T14:35:42.334606+0000","last_peered":"2023-01-15T14:35:42.334606+0000","last_clean":"2023-01-15T14:35:42.334606+0000","last_became_active":"2023-01-15T14:33:55.672203+0000","last_became_peered":"2023-01-15T14:33:55.672203+0000","last_unstale":"2023-01-15T14:35:42.334606+0000","last_undegraded":"2023-01-15T14:35:42.334606+0000","last_fullsized":"2023-01-15T14:35:42.334606+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.204381956,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":520,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.253365+0000","last_change":"2023-01-15T14:34:43.269231+0000","last_active":"2023-01-15T14:35:43.253365+0000","last_peered":"2023-01-15T14:35:43.253365+0000","last_clean":"2023-01-15T14:35:43.253365+0000","last_became_active":"2023-01-15T14:34:43.268688+0000","last_became_peered":"2023-01-15T14:34:43.268688+0000","last_unstale":"2023-01-15T14:35:43.253365+0000","last_undegraded":"2023-01-15T14:35:43.253365+0000","last_fullsized":"2023-01-15T14:35:43.253365+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":1201,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.656732+0000","last_change":"2023-01-15T14:35:44.656732+0000","last_active":"2023-01-15T14:35:44.656732+0000","last_peered":"2023-01-15T14:35:44.656732+0000","last_clean":"2023-01-15T14:35:44.656732+0000","last_became_active":"2023-01-15T14:35:44.555399+0000","last_became_peered":"2023-01-15T14:35:44.555399+0000","last_unstale":"2023-01-15T14:35:44.656732+0000","last_undegraded":"2023-01-15T14:35:44.656732+0000","last_fullsized":"2023-01-15T14:35:44.656732+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50469969100000001,"objects_trimmed":0,"snaptrim_duration":0.016650939,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":127,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.153668+0000","last_change":"2023-01-15T14:35:35.097620+0000","last_active":"2023-01-15T14:35:43.153668+0000","last_peered":"2023-01-15T14:35:43.153668+0000","last_clean":"2023-01-15T14:35:43.153668+0000","last_became_active":"2023-01-15T14:35:35.097317+0000","last_became_peered":"2023-01-15T14:35:35.097317+0000","last_unstale":"2023-01-15T14:35:43.153668+0000","last_undegraded":"2023-01-15T14:35:43.153668+0000","last_fullsized":"2023-01-15T14:35:43.153668+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":653,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.151719+0000","last_change":"2023-01-15T14:35:26.786467+0000","last_active":"2023-01-15T14:35:43.151719+0000","last_peered":"2023-01-15T14:35:43.151719+0000","last_clean":"2023-01-15T14:35:43.151719+0000","last_became_active":"2023-01-15T14:35:26.685181+0000","last_became_peered":"2023-01-15T14:35:26.685181+0000","last_unstale":"2023-01-15T14:35:43.151719+0000","last_undegraded":"2023-01-15T14:35:43.151719+0000","last_fullsized":"2023-01-15T14:35:43.151719+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"179'204","reported_seq":1555,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:42.077734+0000","last_change":"2023-01-15T14:35:40.851383+0000","last_active":"2023-01-15T14:35:42.077734+0000","last_peered":"2023-01-15T14:35:42.077734+0000","last_clean":"2023-01-15T14:35:42.077734+0000","last_became_active":"2023-01-15T14:35:11.437153+0000","last_became_peered":"2023-01-15T14:35:11.437153+0000","last_unstale":"2023-01-15T14:35:42.077734+0000","last_undegraded":"2023-01-15T14:35:42.077734+0000","last_fullsized":"2023-01-15T14:35:42.077734+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:34:48.269667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.269667+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.740582501,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":1384,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.147647+0000","last_change":"2023-01-15T14:35:40.217578+0000","last_active":"2023-01-15T14:35:43.147647+0000","last_peered":"2023-01-15T14:35:43.147647+0000","last_clean":"2023-01-15T14:35:43.147647+0000","last_became_active":"2023-01-15T14:35:38.176913+0000","last_became_peered":"2023-01-15T14:35:38.176913+0000","last_unstale":"2023-01-15T14:35:43.147647+0000","last_undegraded":"2023-01-15T14:35:43.147647+0000","last_fullsized":"2023-01-15T14:35:43.147647+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.115124012,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.a","version":"252'279","reported_seq":1519,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.248415+0000","last_change":"2023-01-15T14:35:41.332189+0000","last_active":"2023-01-15T14:35:44.248415+0000","last_peered":"2023-01-15T14:35:44.248415+0000","last_clean":"2023-01-15T14:35:44.248415+0000","last_became_active":"2023-01-15T14:35:41.228986+0000","last_became_peered":"2023-01-15T14:35:41.228986+0000","last_unstale":"2023-01-15T14:35:44.248415+0000","last_undegraded":"2023-01-15T14:35:44.248415+0000","last_fullsized":"2023-01-15T14:35:44.248415+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.102537815,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":1081,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.230745+0000","last_change":"2023-01-15T14:35:41.331196+0000","last_active":"2023-01-15T14:35:42.230745+0000","last_peered":"2023-01-15T14:35:42.230745+0000","last_clean":"2023-01-15T14:35:42.230745+0000","last_became_active":"2023-01-15T14:35:11.447676+0000","last_became_peered":"2023-01-15T14:35:11.447676+0000","last_unstale":"2023-01-15T14:35:42.230745+0000","last_undegraded":"2023-01-15T14:35:42.230745+0000","last_fullsized":"2023-01-15T14:35:42.230745+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.113519025,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.c","version":"451'156","reported_seq":1546,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.046382+0000","last_change":"2023-01-15T14:35:42.245805+0000","last_active":"2023-01-15T14:35:43.046382+0000","last_peered":"2023-01-15T14:35:43.046382+0000","last_clean":"2023-01-15T14:35:43.046382+0000","last_became_active":"2023-01-15T14:35:41.319042+0000","last_became_peered":"2023-01-15T14:35:41.319042+0000","last_unstale":"2023-01-15T14:35:43.046382+0000","last_undegraded":"2023-01-15T14:35:43.046382+0000","last_fullsized":"2023-01-15T14:35:43.046382+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.26762328800000001,"stat_sum":{"num_bytes":6115273,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":67247,"num_write":17,"num_write_kb":3706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":29077297,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.d","version":"296'232","reported_seq":1561,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.708732+0000","last_change":"2023-01-15T14:35:43.708732+0000","last_active":"2023-01-15T14:35:43.708732+0000","last_peered":"2023-01-15T14:35:43.708732+0000","last_clean":"2023-01-15T14:35:43.708732+0000","last_became_active":"2023-01-15T14:35:07.267234+0000","last_became_peered":"2023-01-15T14:35:07.267234+0000","last_unstale":"2023-01-15T14:35:43.708732+0000","last_undegraded":"2023-01-15T14:35:43.708732+0000","last_fullsized":"2023-01-15T14:35:43.708732+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.65610589100000005,"stat_sum":{"num_bytes":4613376,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":15641,"num_write":11,"num_write_kb":3280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.e","version":"139'38","reported_seq":288,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.599380+0000","last_change":"2023-01-15T14:35:43.599380+0000","last_active":"2023-01-15T14:35:43.599380+0000","last_peered":"2023-01-15T14:35:43.599380+0000","last_clean":"2023-01-15T14:35:43.599380+0000","last_became_active":"2023-01-15T14:34:43.265641+0000","last_became_peered":"2023-01-15T14:34:43.265641+0000","last_unstale":"2023-01-15T14:35:43.599380+0000","last_undegraded":"2023-01-15T14:35:43.599380+0000","last_fullsized":"2023-01-15T14:35:43.599380+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.44511017800000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1474,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.603957+0000","last_change":"2023-01-15T14:35:43.603957+0000","last_active":"2023-01-15T14:35:43.603957+0000","last_peered":"2023-01-15T14:35:43.603957+0000","last_clean":"2023-01-15T14:35:43.603957+0000","last_became_active":"2023-01-15T14:34:00.241951+0000","last_became_peered":"2023-01-15T14:34:00.241951+0000","last_unstale":"2023-01-15T14:35:43.603957+0000","last_undegraded":"2023-01-15T14:35:43.603957+0000","last_fullsized":"2023-01-15T14:35:43.603957+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.54761984200000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.10","version":"449'441","reported_seq":1729,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.129802+0000","last_change":"2023-01-15T14:35:41.623364+0000","last_active":"2023-01-15T14:35:42.129802+0000","last_peered":"2023-01-15T14:35:42.129802+0000","last_clean":"2023-01-15T14:35:42.129802+0000","last_became_active":"2023-01-15T14:35:41.521224+0000","last_became_peered":"2023-01-15T14:35:41.521224+0000","last_unstale":"2023-01-15T14:35:42.129802+0000","last_undegraded":"2023-01-15T14:35:42.129802+0000","last_fullsized":"2023-01-15T14:35:42.129802+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.80607083599999996,"stat_sum":{"num_bytes":8006061,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":64,"num_read_kb":63382,"num_write":55,"num_write_kb":19710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":20658851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"3"}]},{"pgid":"3.11","version":"451'390","reported_seq":1961,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.495672+0000","last_change":"2023-01-15T14:35:43.495672+0000","last_active":"2023-01-15T14:35:43.495672+0000","last_peered":"2023-01-15T14:35:43.495672+0000","last_clean":"2023-01-15T14:35:43.495672+0000","last_became_active":"2023-01-15T14:35:13.513113+0000","last_became_peered":"2023-01-15T14:35:13.513113+0000","last_unstale":"2023-01-15T14:35:43.495672+0000","last_undegraded":"2023-01-15T14:35:43.495672+0000","last_fullsized":"2023-01-15T14:35:43.495672+0000","mapping_epoch":422,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":423,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:43.295","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.44238419499999998,"stat_sum":{"num_bytes":19251278,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":50,"num_read_kb":48291,"num_write":41,"num_write_kb":25960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":44867083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.12","version":"451'225","reported_seq":1184,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.622988+0000","last_change":"2023-01-15T14:35:42.623092+0000","last_active":"2023-01-15T14:35:42.622988+0000","last_peered":"2023-01-15T14:35:42.622988+0000","last_clean":"2023-01-15T14:35:42.622988+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:35:42.622988+0000","last_undegraded":"2023-01-15T14:35:42.622988+0000","last_fullsized":"2023-01-15T14:35:42.622988+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":225,"log_dups_size":0,"ondisk_log_size":225,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:58.076","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.39340773600000001,"stat_sum":{"num_bytes":8706257,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":6953,"num_write":21,"num_write_kb":11718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.13","version":"454'421","reported_seq":1735,"reported_epoch":454,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:44.653966+0000","last_change":"2023-01-15T14:35:44.653966+0000","last_active":"2023-01-15T14:35:44.653966+0000","last_peered":"2023-01-15T14:35:44.653966+0000","last_clean":"2023-01-15T14:35:44.653966+0000","last_became_active":"2023-01-15T14:35:41.221503+0000","last_became_peered":"2023-01-15T14:35:41.221503+0000","last_unstale":"2023-01-15T14:35:44.653966+0000","last_undegraded":"2023-01-15T14:35:44.653966+0000","last_fullsized":"2023-01-15T14:35:44.653966+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"374'376","last_scrub_stamp":"2023-01-15T14:34:32.489483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.489483+0000","objects_scrubbed":4,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.381937625,"objects_trimmed":0,"snaptrim_duration":0.13945940200000001,"stat_sum":{"num_bytes":13617083,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":29953,"num_write":8,"num_write_kb":4428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":13617083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.32","version":"449'204","reported_seq":1159,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.728751+0000","last_change":"2023-01-15T14:35:42.728928+0000","last_active":"2023-01-15T14:35:42.728751+0000","last_peered":"2023-01-15T14:35:42.728751+0000","last_clean":"2023-01-15T14:35:42.728751+0000","last_became_active":"2023-01-15T14:34:56.393418+0000","last_became_peered":"2023-01-15T14:34:56.393418+0000","last_unstale":"2023-01-15T14:35:42.728751+0000","last_undegraded":"2023-01-15T14:35:42.728751+0000","last_fullsized":"2023-01-15T14:35:42.728751+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:29.581","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.497895755,"stat_sum":{"num_bytes":6427730,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":12921,"num_write":7,"num_write_kb":2712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2663942,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.33","version":"204'275","reported_seq":1364,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.046484+0000","last_change":"2023-01-15T14:35:41.528169+0000","last_active":"2023-01-15T14:35:43.046484+0000","last_peered":"2023-01-15T14:35:43.046484+0000","last_clean":"2023-01-15T14:35:43.046484+0000","last_became_active":"2023-01-15T14:35:41.223122+0000","last_became_peered":"2023-01-15T14:35:41.223122+0000","last_unstale":"2023-01-15T14:35:43.046484+0000","last_undegraded":"2023-01-15T14:35:43.046484+0000","last_fullsized":"2023-01-15T14:35:43.046484+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.30405338799999998,"stat_sum":{"num_bytes":3485477,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11189,"num_write":7,"num_write_kb":3285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":1437,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.045768+0000","last_change":"2023-01-15T14:35:40.211572+0000","last_active":"2023-01-15T14:35:43.045768+0000","last_peered":"2023-01-15T14:35:43.045768+0000","last_clean":"2023-01-15T14:35:43.045768+0000","last_became_active":"2023-01-15T14:35:19.392381+0000","last_became_peered":"2023-01-15T14:35:19.392381+0000","last_unstale":"2023-01-15T14:35:43.045768+0000","last_undegraded":"2023-01-15T14:35:43.045768+0000","last_fullsized":"2023-01-15T14:35:43.045768+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.604857588,"objects_trimmed":0,"snaptrim_duration":0.207913822,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.35","version":"110'55","reported_seq":180,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.229582+0000","last_change":"2023-01-15T14:35:41.624741+0000","last_active":"2023-01-15T14:35:42.229582+0000","last_peered":"2023-01-15T14:35:42.229582+0000","last_clean":"2023-01-15T14:35:42.229582+0000","last_became_active":"2023-01-15T14:35:41.624187+0000","last_became_peered":"2023-01-15T14:35:41.624187+0000","last_unstale":"2023-01-15T14:35:42.229582+0000","last_undegraded":"2023-01-15T14:35:42.229582+0000","last_fullsized":"2023-01-15T14:35:42.229582+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44800488300000002,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.36","version":"194'81","reported_seq":93,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:42.180299+0000","last_change":"2023-01-15T14:35:41.627166+0000","last_active":"2023-01-15T14:35:42.180299+0000","last_peered":"2023-01-15T14:35:42.180299+0000","last_clean":"2023-01-15T14:35:42.180299+0000","last_became_active":"2023-01-15T14:35:41.323111+0000","last_became_peered":"2023-01-15T14:35:41.323111+0000","last_unstale":"2023-01-15T14:35:42.180299+0000","last_undegraded":"2023-01-15T14:35:42.180299+0000","last_fullsized":"2023-01-15T14:35:42.180299+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:34:51.296526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.296526+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.66934546500000003,"objects_trimmed":0,"snaptrim_duration":0.30248265400000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.37","version":"200'78","reported_seq":681,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.710334+0000","last_change":"2023-01-15T14:35:43.710334+0000","last_active":"2023-01-15T14:35:43.710334+0000","last_peered":"2023-01-15T14:35:43.710334+0000","last_clean":"2023-01-15T14:35:43.710334+0000","last_became_active":"2023-01-15T14:35:19.152223+0000","last_became_peered":"2023-01-15T14:35:19.152223+0000","last_unstale":"2023-01-15T14:35:43.710334+0000","last_undegraded":"2023-01-15T14:35:43.710334+0000","last_fullsized":"2023-01-15T14:35:43.710334+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.55366588699999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.38","version":"426'347","reported_seq":1880,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.047742+0000","last_change":"2023-01-15T14:35:41.542873+0000","last_active":"2023-01-15T14:35:44.047742+0000","last_peered":"2023-01-15T14:35:44.047742+0000","last_clean":"2023-01-15T14:35:44.047742+0000","last_became_active":"2023-01-15T14:35:41.220388+0000","last_became_peered":"2023-01-15T14:35:41.220388+0000","last_unstale":"2023-01-15T14:35:44.047742+0000","last_undegraded":"2023-01-15T14:35:44.047742+0000","last_fullsized":"2023-01-15T14:35:44.047742+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.220702496,"stat_sum":{"num_bytes":4230716,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":21424,"num_write":20,"num_write_kb":6727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":16442530,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.39","version":"423'435","reported_seq":1837,"reported_epoch":454,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:44.367557+0000","last_change":"2023-01-15T14:35:44.367646+0000","last_active":"2023-01-15T14:35:44.367557+0000","last_peered":"2023-01-15T14:35:44.367557+0000","last_clean":"2023-01-15T14:35:44.367557+0000","last_became_active":"2023-01-15T14:35:38.280557+0000","last_became_peered":"2023-01-15T14:35:38.280557+0000","last_unstale":"2023-01-15T14:35:44.367557+0000","last_undegraded":"2023-01-15T14:35:44.367557+0000","last_fullsized":"2023-01-15T14:35:44.367557+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.22175452300000001,"stat_sum":{"num_bytes":17098873,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":34,"num_read_kb":57397,"num_write":12,"num_write_kb":6931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":13431306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1642,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.625292+0000","last_change":"2023-01-15T14:35:42.625391+0000","last_active":"2023-01-15T14:35:42.625292+0000","last_peered":"2023-01-15T14:35:42.625292+0000","last_clean":"2023-01-15T14:35:42.625292+0000","last_became_active":"2023-01-15T14:35:19.407553+0000","last_became_peered":"2023-01-15T14:35:19.407553+0000","last_unstale":"2023-01-15T14:35:42.625292+0000","last_undegraded":"2023-01-15T14:35:42.625292+0000","last_fullsized":"2023-01-15T14:35:42.625292+0000","mapping_epoch":395,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.39384855600000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.3b","version":"221'60","reported_seq":282,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.711549+0000","last_change":"2023-01-15T14:35:43.711549+0000","last_active":"2023-01-15T14:35:43.711549+0000","last_peered":"2023-01-15T14:35:43.711549+0000","last_clean":"2023-01-15T14:35:43.711549+0000","last_became_active":"2023-01-15T14:35:35.100811+0000","last_became_peered":"2023-01-15T14:35:35.100811+0000","last_unstale":"2023-01-15T14:35:43.711549+0000","last_undegraded":"2023-01-15T14:35:43.711549+0000","last_fullsized":"2023-01-15T14:35:43.711549+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.55758860899999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":127,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.710932+0000","last_change":"2023-01-15T14:35:43.710932+0000","last_active":"2023-01-15T14:35:43.710932+0000","last_peered":"2023-01-15T14:35:43.710932+0000","last_clean":"2023-01-15T14:35:43.710932+0000","last_became_active":"2023-01-15T14:35:35.098817+0000","last_became_peered":"2023-01-15T14:35:35.098817+0000","last_unstale":"2023-01-15T14:35:43.710932+0000","last_undegraded":"2023-01-15T14:35:43.710932+0000","last_fullsized":"2023-01-15T14:35:43.710932+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.65820551000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"}]},{"pgid":"3.3d","version":"448'298","reported_seq":1724,"reported_epoch":452,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:35:42.582936+0000","last_change":"2023-01-15T14:35:42.783888+0000","last_active":"2023-01-15T14:35:42.582936+0000","last_peered":"2023-01-15T14:35:42.582936+0000","last_clean":"2023-01-15T14:35:33.117506+0000","last_became_active":"2023-01-15T14:35:36.606769+0000","last_became_peered":"2023-01-15T14:35:36.606769+0000","last_unstale":"2023-01-15T14:35:42.582936+0000","last_undegraded":"2023-01-15T14:35:35.997821+0000","last_fullsized":"2023-01-15T14:35:35.996854+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.37436582200000001,"stat_sum":{"num_bytes":4849017,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":20693,"num_write":13,"num_write_kb":5220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":14950019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1],"avail_no_missing":["0","1","2","7"],"object_location_counts":[{"shards":"0,1","objects":2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.3e","version":"448'86","reported_seq":248,"reported_epoch":453,"state":"peering","last_fresh":"2023-01-15T14:35:43.047817+0000","last_change":"2023-01-15T14:35:43.048999+0000","last_active":"2023-01-15T14:35:43.047769+0000","last_peered":"2023-01-15T14:35:42.079476+0000","last_clean":"2023-01-15T14:35:42.079476+0000","last_became_active":"2023-01-15T14:35:04.273985+0000","last_became_peered":"2023-01-15T14:35:04.273985+0000","last_unstale":"2023-01-15T14:35:43.047817+0000","last_undegraded":"2023-01-15T14:35:43.047817+0000","last_fullsized":"2023-01-15T14:35:43.047817+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.64084343300000002,"stat_sum":{"num_bytes":3420307,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":3025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":8496136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,6],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.3f","version":"447'273","reported_seq":1686,"reported_epoch":454,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:44.764463+0000","last_change":"2023-01-15T14:35:44.764463+0000","last_active":"2023-01-15T14:35:44.764463+0000","last_peered":"2023-01-15T14:35:44.764463+0000","last_clean":"2023-01-15T14:35:44.764463+0000","last_became_active":"2023-01-15T14:35:44.661557+0000","last_became_peered":"2023-01-15T14:35:44.661557+0000","last_unstale":"2023-01-15T14:35:44.764463+0000","last_undegraded":"2023-01-15T14:35:44.764463+0000","last_fullsized":"2023-01-15T14:35:44.764463+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.20100641699999999,"stat_sum":{"num_bytes":11708858,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":14973,"num_write":18,"num_write_kb":7357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":13214874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"}]},{"pgid":"3.40","version":"452'355","reported_seq":1701,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.759432+0000","last_change":"2023-01-15T14:35:43.759432+0000","last_active":"2023-01-15T14:35:43.759432+0000","last_peered":"2023-01-15T14:35:43.759432+0000","last_clean":"2023-01-15T14:35:43.759432+0000","last_became_active":"2023-01-15T14:35:41.632354+0000","last_became_peered":"2023-01-15T14:35:41.632354+0000","last_unstale":"2023-01-15T14:35:43.759432+0000","last_undegraded":"2023-01-15T14:35:43.759432+0000","last_fullsized":"2023-01-15T14:35:43.759432+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.60611443899999995,"stat_sum":{"num_bytes":12637268,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":35543,"num_write":40,"num_write_kb":14039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":43425481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.41","version":"155'121","reported_seq":1305,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.246147+0000","last_change":"2023-01-15T14:35:41.229958+0000","last_active":"2023-01-15T14:35:43.246147+0000","last_peered":"2023-01-15T14:35:43.246147+0000","last_clean":"2023-01-15T14:35:43.246147+0000","last_became_active":"2023-01-15T14:35:41.128041+0000","last_became_peered":"2023-01-15T14:35:41.128041+0000","last_unstale":"2023-01-15T14:35:43.246147+0000","last_undegraded":"2023-01-15T14:35:43.246147+0000","last_fullsized":"2023-01-15T14:35:43.246147+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.101464981,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":1008,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.330328+0000","last_change":"2023-01-15T14:35:41.517322+0000","last_active":"2023-01-15T14:35:42.330328+0000","last_peered":"2023-01-15T14:35:42.330328+0000","last_clean":"2023-01-15T14:35:42.330328+0000","last_became_active":"2023-01-15T14:35:41.516587+0000","last_became_peered":"2023-01-15T14:35:41.516587+0000","last_unstale":"2023-01-15T14:35:42.330328+0000","last_undegraded":"2023-01-15T14:35:42.330328+0000","last_fullsized":"2023-01-15T14:35:42.330328+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:34:57.046809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:57.046809+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.20550260100000001,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.43","version":"441'347","reported_seq":1754,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.332461+0000","last_change":"2023-01-15T14:35:41.426609+0000","last_active":"2023-01-15T14:35:42.332461+0000","last_peered":"2023-01-15T14:35:42.332461+0000","last_clean":"2023-01-15T14:35:42.332461+0000","last_became_active":"2023-01-15T14:35:41.425468+0000","last_became_peered":"2023-01-15T14:35:41.425468+0000","last_unstale":"2023-01-15T14:35:42.332461+0000","last_undegraded":"2023-01-15T14:35:42.332461+0000","last_fullsized":"2023-01-15T14:35:42.332461+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.58727570299999998,"stat_sum":{"num_bytes":9109625,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":13,"num_read_kb":16586,"num_write":12,"num_write_kb":5193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12504705,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.44","version":"245'268","reported_seq":1438,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.050710+0000","last_change":"2023-01-15T14:35:41.545287+0000","last_active":"2023-01-15T14:35:44.050710+0000","last_peered":"2023-01-15T14:35:44.050710+0000","last_clean":"2023-01-15T14:35:44.050710+0000","last_became_active":"2023-01-15T14:35:41.325427+0000","last_became_peered":"2023-01-15T14:35:41.325427+0000","last_unstale":"2023-01-15T14:35:44.050710+0000","last_undegraded":"2023-01-15T14:35:44.050710+0000","last_fullsized":"2023-01-15T14:35:44.050710+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.120501842,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.45","version":"253'125","reported_seq":1175,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.436162+0000","last_change":"2023-01-15T14:35:42.436288+0000","last_active":"2023-01-15T14:35:42.436162+0000","last_peered":"2023-01-15T14:35:42.436162+0000","last_clean":"2023-01-15T14:35:42.436162+0000","last_became_active":"2023-01-15T14:35:38.500236+0000","last_became_peered":"2023-01-15T14:35:38.500236+0000","last_unstale":"2023-01-15T14:35:42.436162+0000","last_undegraded":"2023-01-15T14:35:42.436162+0000","last_fullsized":"2023-01-15T14:35:42.436162+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:34:50.381073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:50.381073+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:05.295","scrub_duration":0.52191215300000005,"objects_trimmed":0,"snaptrim_duration":0.20568805300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.46","version":"102'52","reported_seq":186,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:42.280010+0000","last_change":"2023-01-15T14:35:40.750204+0000","last_active":"2023-01-15T14:35:42.280010+0000","last_peered":"2023-01-15T14:35:42.280010+0000","last_clean":"2023-01-15T14:35:42.280010+0000","last_became_active":"2023-01-15T14:35:04.373495+0000","last_became_peered":"2023-01-15T14:35:04.373495+0000","last_unstale":"2023-01-15T14:35:42.280010+0000","last_undegraded":"2023-01-15T14:35:42.280010+0000","last_fullsized":"2023-01-15T14:35:42.280010+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:21.428","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.641311984,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2142,"num_write":3,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.47","version":"245'112","reported_seq":1381,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.151595+0000","last_change":"2023-01-15T14:35:40.208342+0000","last_active":"2023-01-15T14:35:43.151595+0000","last_peered":"2023-01-15T14:35:43.151595+0000","last_clean":"2023-01-15T14:35:43.151595+0000","last_became_active":"2023-01-15T14:32:28.160193+0000","last_became_peered":"2023-01-15T14:32:28.160193+0000","last_unstale":"2023-01-15T14:35:43.151595+0000","last_undegraded":"2023-01-15T14:35:43.151595+0000","last_fullsized":"2023-01-15T14:35:43.151595+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:34:21.450320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:21.450320+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:50.200","scrub_duration":0.30497759099999999,"objects_trimmed":0,"snaptrim_duration":0.20599663900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.48","version":"179'204","reported_seq":1428,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.233658+0000","last_change":"2023-01-15T14:35:42.233764+0000","last_active":"2023-01-15T14:35:42.233658+0000","last_peered":"2023-01-15T14:35:42.233658+0000","last_clean":"2023-01-15T14:35:42.233658+0000","last_became_active":"2023-01-15T14:35:11.517680+0000","last_became_peered":"2023-01-15T14:35:11.517680+0000","last_unstale":"2023-01-15T14:35:42.233658+0000","last_undegraded":"2023-01-15T14:35:42.233658+0000","last_fullsized":"2023-01-15T14:35:42.233658+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.105037641,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.49","version":"110'149","reported_seq":1191,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.762128+0000","last_change":"2023-01-15T14:35:43.762128+0000","last_active":"2023-01-15T14:35:43.762128+0000","last_peered":"2023-01-15T14:35:43.762128+0000","last_clean":"2023-01-15T14:35:43.762128+0000","last_became_active":"2023-01-15T14:35:38.285855+0000","last_became_peered":"2023-01-15T14:35:38.285855+0000","last_unstale":"2023-01-15T14:35:43.762128+0000","last_undegraded":"2023-01-15T14:35:43.762128+0000","last_fullsized":"2023-01-15T14:35:43.762128+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.710450146,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.4a","version":"449'430","reported_seq":1827,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.497144+0000","last_change":"2023-01-15T14:35:43.497144+0000","last_active":"2023-01-15T14:35:43.497144+0000","last_peered":"2023-01-15T14:35:43.497144+0000","last_clean":"2023-01-15T14:35:43.497144+0000","last_became_active":"2023-01-15T14:35:22.264683+0000","last_became_peered":"2023-01-15T14:35:22.264683+0000","last_unstale":"2023-01-15T14:35:43.497144+0000","last_undegraded":"2023-01-15T14:35:43.497144+0000","last_fullsized":"2023-01-15T14:35:43.497144+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":430,"log_dups_size":0,"ondisk_log_size":430,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:34.189","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.34410400899999999,"stat_sum":{"num_bytes":15111564,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":84,"num_read_kb":95190,"num_write":88,"num_write_kb":26766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":39901450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.4b","version":"155'53","reported_seq":89,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.978123+0000","last_change":"2023-01-15T14:35:41.628078+0000","last_active":"2023-01-15T14:35:41.978123+0000","last_peered":"2023-01-15T14:35:41.978123+0000","last_clean":"2023-01-15T14:35:41.978123+0000","last_became_active":"2023-01-15T14:35:41.324684+0000","last_became_peered":"2023-01-15T14:35:41.324684+0000","last_unstale":"2023-01-15T14:35:41.978123+0000","last_undegraded":"2023-01-15T14:35:41.978123+0000","last_fullsized":"2023-01-15T14:35:41.978123+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:34:44.116779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:44.116779+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:59.161","scrub_duration":0.50673716899999999,"objects_trimmed":0,"snaptrim_duration":0.30173127500000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.4c","version":"309'71","reported_seq":359,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.607077+0000","last_change":"2023-01-15T14:35:43.607077+0000","last_active":"2023-01-15T14:35:43.607077+0000","last_peered":"2023-01-15T14:35:43.607077+0000","last_clean":"2023-01-15T14:35:43.607077+0000","last_became_active":"2023-01-15T14:35:04.379289+0000","last_became_peered":"2023-01-15T14:35:04.379289+0000","last_unstale":"2023-01-15T14:35:43.607077+0000","last_undegraded":"2023-01-15T14:35:43.607077+0000","last_fullsized":"2023-01-15T14:35:43.607077+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.45435679099999998,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.4d","version":"453'339","reported_seq":1663,"reported_epoch":454,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:44.652672+0000","last_change":"2023-01-15T14:35:44.652772+0000","last_active":"2023-01-15T14:35:44.652672+0000","last_peered":"2023-01-15T14:35:44.652672+0000","last_clean":"2023-01-15T14:35:44.652672+0000","last_became_active":"2023-01-15T14:34:05.178613+0000","last_became_peered":"2023-01-15T14:34:05.178613+0000","last_unstale":"2023-01-15T14:35:44.652672+0000","last_undegraded":"2023-01-15T14:35:44.652672+0000","last_fullsized":"2023-01-15T14:35:44.652672+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.404490075,"stat_sum":{"num_bytes":12395770,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":15962,"num_write":27,"num_write_kb":9640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"}]},{"pgid":"3.4e","version":"139'38","reported_seq":348,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.130251+0000","last_change":"2023-01-15T14:35:41.432068+0000","last_active":"2023-01-15T14:35:42.130251+0000","last_peered":"2023-01-15T14:35:42.130251+0000","last_clean":"2023-01-15T14:35:42.130251+0000","last_became_active":"2023-01-15T14:35:35.391600+0000","last_became_peered":"2023-01-15T14:35:35.391600+0000","last_unstale":"2023-01-15T14:35:42.130251+0000","last_undegraded":"2023-01-15T14:35:42.130251+0000","last_fullsized":"2023-01-15T14:35:42.130251+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.31038190700000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.4f","version":"155'145","reported_seq":1077,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.047210+0000","last_change":"2023-01-15T14:35:41.128094+0000","last_active":"2023-01-15T14:35:43.047210+0000","last_peered":"2023-01-15T14:35:43.047210+0000","last_clean":"2023-01-15T14:35:43.047210+0000","last_became_active":"2023-01-15T14:35:41.119087+0000","last_became_peered":"2023-01-15T14:35:41.119087+0000","last_unstale":"2023-01-15T14:35:43.047210+0000","last_undegraded":"2023-01-15T14:35:43.047210+0000","last_fullsized":"2023-01-15T14:35:43.047210+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075457170000000004,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.50","version":"453'498","reported_seq":1894,"reported_epoch":454,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:44.366717+0000","last_change":"2023-01-15T14:35:44.366793+0000","last_active":"2023-01-15T14:35:44.366717+0000","last_peered":"2023-01-15T14:35:44.366717+0000","last_clean":"2023-01-15T14:35:44.366717+0000","last_became_active":"2023-01-15T14:34:02.440915+0000","last_became_peered":"2023-01-15T14:34:02.440915+0000","last_unstale":"2023-01-15T14:35:44.366717+0000","last_undegraded":"2023-01-15T14:35:44.366717+0000","last_fullsized":"2023-01-15T14:35:44.366717+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.220775253,"stat_sum":{"num_bytes":10573847,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":92165,"num_write":64,"num_write_kb":23394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.51","version":"443'351","reported_seq":1814,"reported_epoch":453,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:35:43.682363+0000","last_change":"2023-01-15T14:35:43.683027+0000","last_active":"2023-01-15T14:35:43.682363+0000","last_peered":"2023-01-15T14:35:43.682363+0000","last_clean":"2023-01-15T14:35:33.924915+0000","last_became_active":"2023-01-15T14:35:36.397266+0000","last_became_peered":"2023-01-15T14:35:36.397266+0000","last_unstale":"2023-01-15T14:35:43.682363+0000","last_undegraded":"2023-01-15T14:35:43.682363+0000","last_fullsized":"2023-01-15T14:35:36.091114+0000","mapping_epoch":445,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.74045614599999998,"stat_sum":{"num_bytes":7668863,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":2393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":8487799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0],"avail_no_missing":["1","0","2","3"],"object_location_counts":[{"shards":"0,1","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":921,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.653368+0000","last_change":"2023-01-15T14:35:44.653850+0000","last_active":"2023-01-15T14:35:44.653368+0000","last_peered":"2023-01-15T14:35:44.653368+0000","last_clean":"2023-01-15T14:35:44.653368+0000","last_became_active":"2023-01-15T14:35:41.519122+0000","last_became_peered":"2023-01-15T14:35:41.519122+0000","last_unstale":"2023-01-15T14:35:44.653368+0000","last_undegraded":"2023-01-15T14:35:44.653368+0000","last_fullsized":"2023-01-15T14:35:44.653368+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.50813712600000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.53","version":"454'524","reported_seq":1958,"reported_epoch":454,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:44.551669+0000","last_change":"2023-01-15T14:35:44.551669+0000","last_active":"2023-01-15T14:35:44.551669+0000","last_peered":"2023-01-15T14:35:44.551669+0000","last_clean":"2023-01-15T14:35:44.551669+0000","last_became_active":"2023-01-15T14:35:41.426783+0000","last_became_peered":"2023-01-15T14:35:41.426783+0000","last_unstale":"2023-01-15T14:35:44.551669+0000","last_undegraded":"2023-01-15T14:35:44.551669+0000","last_fullsized":"2023-01-15T14:35:44.551669+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":524,"log_dups_size":0,"ondisk_log_size":524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.91597736900000004,"stat_sum":{"num_bytes":23660871,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":68,"num_read_kb":67501,"num_write":60,"num_write_kb":28208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":72326676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.54","version":"245'148","reported_seq":1380,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.230430+0000","last_change":"2023-01-15T14:35:41.222252+0000","last_active":"2023-01-15T14:35:42.230430+0000","last_peered":"2023-01-15T14:35:42.230430+0000","last_clean":"2023-01-15T14:35:42.230430+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:35:42.230430+0000","last_undegraded":"2023-01-15T14:35:42.230430+0000","last_fullsized":"2023-01-15T14:35:42.230430+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:34:47.392699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.392699+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:14.435","scrub_duration":0.50581767899999996,"objects_trimmed":0,"snaptrim_duration":0.203409859,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.55","version":"110'55","reported_seq":1087,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.229170+0000","last_change":"2023-01-15T14:35:41.528306+0000","last_active":"2023-01-15T14:35:42.229170+0000","last_peered":"2023-01-15T14:35:42.229170+0000","last_clean":"2023-01-15T14:35:42.229170+0000","last_became_active":"2023-01-15T14:35:11.347080+0000","last_became_peered":"2023-01-15T14:35:11.347080+0000","last_unstale":"2023-01-15T14:35:42.229170+0000","last_undegraded":"2023-01-15T14:35:42.229170+0000","last_fullsized":"2023-01-15T14:35:42.229170+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:34:59.815570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:59.815570+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.50879290099999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.56","version":"320'127","reported_seq":868,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.601670+0000","last_change":"2023-01-15T14:35:43.601670+0000","last_active":"2023-01-15T14:35:43.601670+0000","last_peered":"2023-01-15T14:35:43.601670+0000","last_clean":"2023-01-15T14:35:43.601670+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:35:43.601670+0000","last_undegraded":"2023-01-15T14:35:43.601670+0000","last_fullsized":"2023-01-15T14:35:43.601670+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.44739377499999999,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.57","version":"452'126","reported_seq":410,"reported_epoch":452,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:42.436329+0000","last_change":"2023-01-15T14:35:42.436329+0000","last_active":"2023-01-15T14:35:42.436329+0000","last_peered":"2023-01-15T14:35:42.436329+0000","last_clean":"2023-01-15T14:35:42.436329+0000","last_became_active":"2023-01-15T14:35:40.183681+0000","last_became_peered":"2023-01-15T14:35:40.183681+0000","last_unstale":"2023-01-15T14:35:42.436329+0000","last_undegraded":"2023-01-15T14:35:42.436329+0000","last_fullsized":"2023-01-15T14:35:42.436329+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'100","last_scrub_stamp":"2023-01-15T14:34:23.447852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.447852+0000","objects_scrubbed":2,"log_size":126,"log_dups_size":0,"ondisk_log_size":126,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.52101476099999999,"objects_trimmed":0,"snaptrim_duration":0.27390267699999998,"stat_sum":{"num_bytes":8003958,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":21,"num_read_kb":25129,"num_write":12,"num_write_kb":2743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":17584212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"126","length":"1"},{"start":"12e","length":"1"}]},{"pgid":"3.58","version":"323'320","reported_seq":1807,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.761036+0000","last_change":"2023-01-15T14:35:43.761036+0000","last_active":"2023-01-15T14:35:43.761036+0000","last_peered":"2023-01-15T14:35:43.761036+0000","last_clean":"2023-01-15T14:35:43.761036+0000","last_became_active":"2023-01-15T14:35:38.384100+0000","last_became_peered":"2023-01-15T14:35:38.384100+0000","last_unstale":"2023-01-15T14:35:43.761036+0000","last_undegraded":"2023-01-15T14:35:43.761036+0000","last_fullsized":"2023-01-15T14:35:43.761036+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'320","last_scrub_stamp":"2023-01-15T14:34:47.541655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.541655+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:06.787","scrub_duration":0.40398073200000001,"objects_trimmed":0,"snaptrim_duration":0.60838642799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.59","version":"294'256","reported_seq":1556,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.146087+0000","last_change":"2023-01-15T14:35:40.207249+0000","last_active":"2023-01-15T14:35:43.146087+0000","last_peered":"2023-01-15T14:35:43.146087+0000","last_clean":"2023-01-15T14:35:43.146087+0000","last_became_active":"2023-01-15T14:35:07.462526+0000","last_became_peered":"2023-01-15T14:35:07.462526+0000","last_unstale":"2023-01-15T14:35:43.146087+0000","last_undegraded":"2023-01-15T14:35:43.146087+0000","last_fullsized":"2023-01-15T14:35:43.146087+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:34:30.207952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:30.207952+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024191634,"objects_trimmed":0,"snaptrim_duration":0.20534904400000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.5a","version":"449'373","reported_seq":1785,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.624151+0000","last_change":"2023-01-15T14:35:42.624258+0000","last_active":"2023-01-15T14:35:42.624151+0000","last_peered":"2023-01-15T14:35:42.624151+0000","last_clean":"2023-01-15T14:35:42.624151+0000","last_became_active":"2023-01-15T14:35:28.411845+0000","last_became_peered":"2023-01-15T14:35:28.411845+0000","last_unstale":"2023-01-15T14:35:42.624151+0000","last_undegraded":"2023-01-15T14:35:42.624151+0000","last_fullsized":"2023-01-15T14:35:42.624151+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.29180472000000002,"stat_sum":{"num_bytes":3664025,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":37194,"num_write":6,"num_write_kb":1588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":15630873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"126","length":"1"}]},{"pgid":"3.5b","version":"283'80","reported_seq":1150,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.605331+0000","last_change":"2023-01-15T14:35:43.605331+0000","last_active":"2023-01-15T14:35:43.605331+0000","last_peered":"2023-01-15T14:35:43.605331+0000","last_clean":"2023-01-15T14:35:43.605331+0000","last_became_active":"2023-01-15T14:33:52.109863+0000","last_became_peered":"2023-01-15T14:33:52.109863+0000","last_unstale":"2023-01-15T14:35:43.605331+0000","last_undegraded":"2023-01-15T14:35:43.605331+0000","last_fullsized":"2023-01-15T14:35:43.605331+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.451060509,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.5c","version":"141'46","reported_seq":324,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.145668+0000","last_change":"2023-01-15T14:35:43.763341+0000","last_active":"2023-01-15T14:35:44.145668+0000","last_peered":"2023-01-15T14:35:44.145668+0000","last_clean":"2023-01-15T14:35:44.145668+0000","last_became_active":"2023-01-15T14:35:41.024937+0000","last_became_peered":"2023-01-15T14:35:41.024937+0000","last_unstale":"2023-01-15T14:35:44.145668+0000","last_undegraded":"2023-01-15T14:35:44.145668+0000","last_fullsized":"2023-01-15T14:35:44.145668+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.71100123100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1568,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.051020+0000","last_change":"2023-01-15T14:35:40.210600+0000","last_active":"2023-01-15T14:35:44.051020+0000","last_peered":"2023-01-15T14:35:44.051020+0000","last_clean":"2023-01-15T14:35:44.051020+0000","last_became_active":"2023-01-15T14:34:00.242122+0000","last_became_peered":"2023-01-15T14:34:00.242122+0000","last_unstale":"2023-01-15T14:35:44.051020+0000","last_undegraded":"2023-01-15T14:35:44.051020+0000","last_fullsized":"2023-01-15T14:35:44.051020+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:19.729021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:19.729021+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50499948100000003,"objects_trimmed":0,"snaptrim_duration":0.206976142,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.5e","version":"215'39","reported_seq":1202,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:42.179465+0000","last_change":"2023-01-15T14:35:40.648141+0000","last_active":"2023-01-15T14:35:42.179465+0000","last_peered":"2023-01-15T14:35:42.179465+0000","last_clean":"2023-01-15T14:35:42.179465+0000","last_became_active":"2023-01-15T14:35:35.187326+0000","last_became_peered":"2023-01-15T14:35:35.187326+0000","last_unstale":"2023-01-15T14:35:42.179465+0000","last_undegraded":"2023-01-15T14:35:42.179465+0000","last_fullsized":"2023-01-15T14:35:42.179465+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.53999073900000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1621,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.226998+0000","last_change":"2023-01-15T14:35:41.229397+0000","last_active":"2023-01-15T14:35:42.226998+0000","last_peered":"2023-01-15T14:35:42.226998+0000","last_clean":"2023-01-15T14:35:42.226998+0000","last_became_active":"2023-01-15T14:35:38.282477+0000","last_became_peered":"2023-01-15T14:35:38.282477+0000","last_unstale":"2023-01-15T14:35:42.226998+0000","last_undegraded":"2023-01-15T14:35:42.226998+0000","last_fullsized":"2023-01-15T14:35:42.226998+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.043093477999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.60","version":"453'312","reported_seq":1754,"reported_epoch":453,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:43.356040+0000","last_change":"2023-01-15T14:35:43.356040+0000","last_active":"2023-01-15T14:35:43.356040+0000","last_peered":"2023-01-15T14:35:43.356040+0000","last_clean":"2023-01-15T14:35:43.356040+0000","last_became_active":"2023-01-15T14:35:43.153862+0000","last_became_peered":"2023-01-15T14:35:43.153862+0000","last_unstale":"2023-01-15T14:35:43.356040+0000","last_undegraded":"2023-01-15T14:35:43.356040+0000","last_fullsized":"2023-01-15T14:35:43.356040+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":7,"last_scrub":"402'291","last_scrub_stamp":"2023-01-15T14:34:55.535113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:55.535113+0000","objects_scrubbed":4,"log_size":312,"log_dups_size":0,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:56.538","scrub_duration":0.814267513,"objects_trimmed":0,"snaptrim_duration":0.100668826,"stat_sum":{"num_bytes":6004978,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":11,"num_read_kb":10595,"num_write":6,"num_write_kb":1008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":7126465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.61","version":"449'178","reported_seq":1305,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.434884+0000","last_change":"2023-01-15T14:35:42.435055+0000","last_active":"2023-01-15T14:35:42.434884+0000","last_peered":"2023-01-15T14:35:42.434884+0000","last_clean":"2023-01-15T14:35:42.434884+0000","last_became_active":"2023-01-15T14:34:44.473866+0000","last_became_peered":"2023-01-15T14:34:44.473866+0000","last_unstale":"2023-01-15T14:35:42.434884+0000","last_undegraded":"2023-01-15T14:35:42.434884+0000","last_fullsized":"2023-01-15T14:35:42.434884+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.30404993299999999,"stat_sum":{"num_bytes":3823692,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":15,"num_read_kb":16964,"num_write":6,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.62","version":"451'213","reported_seq":1729,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:42.322568+0000","last_change":"2023-01-15T14:35:42.322568+0000","last_active":"2023-01-15T14:35:42.322568+0000","last_peered":"2023-01-15T14:35:42.322568+0000","last_clean":"2023-01-15T14:35:42.322568+0000","last_became_active":"2023-01-15T14:35:40.393956+0000","last_became_peered":"2023-01-15T14:35:40.393956+0000","last_unstale":"2023-01-15T14:35:42.322568+0000","last_undegraded":"2023-01-15T14:35:42.322568+0000","last_fullsized":"2023-01-15T14:35:42.322568+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":7,"last_scrub":"403'189","last_scrub_stamp":"2023-01-15T14:34:56.851447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:56.851447+0000","objects_scrubbed":5,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:58.143","scrub_duration":0.30960584400000002,"objects_trimmed":0,"snaptrim_duration":0.34535628299999999,"stat_sum":{"num_bytes":9221281,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":8,"num_read_kb":9592,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10723130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.63","version":"149'233","reported_seq":1016,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.331045+0000","last_change":"2023-01-15T14:35:41.465340+0000","last_active":"2023-01-15T14:35:42.331045+0000","last_peered":"2023-01-15T14:35:42.331045+0000","last_clean":"2023-01-15T14:35:42.331045+0000","last_became_active":"2023-01-15T14:35:38.386493+0000","last_became_peered":"2023-01-15T14:35:38.386493+0000","last_unstale":"2023-01-15T14:35:42.331045+0000","last_undegraded":"2023-01-15T14:35:42.331045+0000","last_fullsized":"2023-01-15T14:35:42.331045+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:34:56.637790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:56.637790+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:20.521","scrub_duration":0.63257800399999997,"objects_trimmed":0,"snaptrim_duration":0.24265574300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"}]},{"pgid":"3.64","version":"334'251","reported_seq":1681,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.323681+0000","last_change":"2023-01-15T14:35:41.022454+0000","last_active":"2023-01-15T14:35:42.323681+0000","last_peered":"2023-01-15T14:35:42.323681+0000","last_clean":"2023-01-15T14:35:42.323681+0000","last_became_active":"2023-01-15T14:35:07.057736+0000","last_became_peered":"2023-01-15T14:35:07.057736+0000","last_unstale":"2023-01-15T14:35:42.323681+0000","last_undegraded":"2023-01-15T14:35:42.323681+0000","last_fullsized":"2023-01-15T14:35:42.323681+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"334'251","last_scrub_stamp":"2023-01-15T14:34:48.644529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.644529+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:35:49.145","scrub_duration":0.70872698300000003,"objects_trimmed":0,"snaptrim_duration":0.0035732120000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.65","version":"450'200","reported_seq":1615,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:42.076704+0000","last_change":"2023-01-15T14:35:41.019866+0000","last_active":"2023-01-15T14:35:42.076704+0000","last_peered":"2023-01-15T14:35:42.076704+0000","last_clean":"2023-01-15T14:35:42.076704+0000","last_became_active":"2023-01-15T14:35:10.233584+0000","last_became_peered":"2023-01-15T14:35:10.233584+0000","last_unstale":"2023-01-15T14:35:42.076704+0000","last_undegraded":"2023-01-15T14:35:42.076704+0000","last_fullsized":"2023-01-15T14:35:42.076704+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'126","last_scrub_stamp":"2023-01-15T14:34:23.903906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.903906+0000","objects_scrubbed":6,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30981859099999998,"objects_trimmed":0,"snaptrim_duration":0.92065251199999998,"stat_sum":{"num_bytes":13824690,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":27,"num_read_kb":27857,"num_write":25,"num_write_kb":10132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20310327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.66","version":"102'52","reported_seq":544,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.145038+0000","last_change":"2023-01-15T14:35:41.540398+0000","last_active":"2023-01-15T14:35:43.145038+0000","last_peered":"2023-01-15T14:35:43.145038+0000","last_clean":"2023-01-15T14:35:43.145038+0000","last_became_active":"2023-01-15T14:35:41.337828+0000","last_became_peered":"2023-01-15T14:35:41.337828+0000","last_unstale":"2023-01-15T14:35:43.145038+0000","last_undegraded":"2023-01-15T14:35:43.145038+0000","last_fullsized":"2023-01-15T14:35:43.145038+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:34:32.591909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.591909+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.39200112999999998,"objects_trimmed":0,"snaptrim_duration":0.20144126800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.67","version":"191'79","reported_seq":292,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.606552+0000","last_change":"2023-01-15T14:35:43.606552+0000","last_active":"2023-01-15T14:35:43.606552+0000","last_peered":"2023-01-15T14:35:43.606552+0000","last_clean":"2023-01-15T14:35:43.606552+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:35:43.606552+0000","last_undegraded":"2023-01-15T14:35:43.606552+0000","last_fullsized":"2023-01-15T14:35:43.606552+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:02.115","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.45002882100000002,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.68","version":"451'284","reported_seq":1730,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.232710+0000","last_change":"2023-01-15T14:35:42.232862+0000","last_active":"2023-01-15T14:35:42.232710+0000","last_peered":"2023-01-15T14:35:42.232710+0000","last_clean":"2023-01-15T14:35:42.232710+0000","last_became_active":"2023-01-15T14:35:39.400045+0000","last_became_peered":"2023-01-15T14:35:39.400045+0000","last_unstale":"2023-01-15T14:35:42.232710+0000","last_undegraded":"2023-01-15T14:35:42.232710+0000","last_fullsized":"2023-01-15T14:35:42.232710+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":284,"log_dups_size":0,"ondisk_log_size":284,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:00.032","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.101987466,"stat_sum":{"num_bytes":42018,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":23162510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.69","version":"110'149","reported_seq":1041,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.045890+0000","last_change":"2023-01-15T14:35:41.624802+0000","last_active":"2023-01-15T14:35:43.045890+0000","last_peered":"2023-01-15T14:35:43.045890+0000","last_clean":"2023-01-15T14:35:43.045890+0000","last_became_active":"2023-01-15T14:35:41.124289+0000","last_became_peered":"2023-01-15T14:35:41.124289+0000","last_unstale":"2023-01-15T14:35:43.045890+0000","last_undegraded":"2023-01-15T14:35:43.045890+0000","last_fullsized":"2023-01-15T14:35:43.045890+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.39983555599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.6a","version":"191'191","reported_seq":853,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.976714+0000","last_change":"2023-01-15T14:35:41.628826+0000","last_active":"2023-01-15T14:35:41.976714+0000","last_peered":"2023-01-15T14:35:41.976714+0000","last_clean":"2023-01-15T14:35:41.976714+0000","last_became_active":"2023-01-15T14:35:41.324516+0000","last_became_peered":"2023-01-15T14:35:41.324516+0000","last_unstale":"2023-01-15T14:35:41.976714+0000","last_undegraded":"2023-01-15T14:35:41.976714+0000","last_fullsized":"2023-01-15T14:35:41.976714+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30227180599999998,"stat_sum":{"num_bytes":1518387,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":5931,"num_write":14,"num_write_kb":6502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9447727,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.6b","version":"155'53","reported_seq":160,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.602855+0000","last_change":"2023-01-15T14:35:43.602855+0000","last_active":"2023-01-15T14:35:43.602855+0000","last_peered":"2023-01-15T14:35:43.602855+0000","last_clean":"2023-01-15T14:35:43.602855+0000","last_became_active":"2023-01-15T14:34:59.176998+0000","last_became_peered":"2023-01-15T14:34:59.176998+0000","last_unstale":"2023-01-15T14:35:43.602855+0000","last_undegraded":"2023-01-15T14:35:43.602855+0000","last_fullsized":"2023-01-15T14:35:43.602855+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.450548051,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.6c","version":"141'46","reported_seq":149,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:43.498194+0000","last_change":"2023-01-15T14:35:43.498194+0000","last_active":"2023-01-15T14:35:43.498194+0000","last_peered":"2023-01-15T14:35:43.498194+0000","last_clean":"2023-01-15T14:35:43.498194+0000","last_became_active":"2023-01-15T14:35:04.085770+0000","last_became_peered":"2023-01-15T14:35:04.085770+0000","last_unstale":"2023-01-15T14:35:43.498194+0000","last_undegraded":"2023-01-15T14:35:43.498194+0000","last_fullsized":"2023-01-15T14:35:43.498194+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:57.577405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:57.577405+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.44556594599999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.6d","version":"168'156","reported_seq":832,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-01-15T14:35:44.050952+0000","last_change":"2023-01-15T14:35:41.231476+0000","last_active":"2023-01-15T14:35:44.050952+0000","last_peered":"2023-01-15T14:35:44.050952+0000","last_clean":"2023-01-15T14:35:44.050952+0000","last_became_active":"2023-01-15T14:35:41.216059+0000","last_became_peered":"2023-01-15T14:35:41.216059+0000","last_unstale":"2023-01-15T14:35:44.050952+0000","last_undegraded":"2023-01-15T14:35:44.050952+0000","last_fullsized":"2023-01-15T14:35:44.050952+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.014276735,"stat_sum":{"num_bytes":1516832,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2710,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.6e","version":"139'38","reported_seq":150,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.247848+0000","last_change":"2023-01-15T14:35:40.405314+0000","last_active":"2023-01-15T14:35:43.247848+0000","last_peered":"2023-01-15T14:35:43.247848+0000","last_clean":"2023-01-15T14:35:43.247848+0000","last_became_active":"2023-01-15T14:34:59.097277+0000","last_became_peered":"2023-01-15T14:34:59.097277+0000","last_unstale":"2023-01-15T14:35:43.247848+0000","last_undegraded":"2023-01-15T14:35:43.247848+0000","last_fullsized":"2023-01-15T14:35:43.247848+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:32.337","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.20242064600000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.6f","version":"155'145","reported_seq":1214,"reported_epoch":452,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:42.831583+0000","last_change":"2023-01-15T14:35:42.831731+0000","last_active":"2023-01-15T14:35:42.831583+0000","last_peered":"2023-01-15T14:35:42.831583+0000","last_clean":"2023-01-15T14:35:42.831583+0000","last_became_active":"2023-01-15T14:35:02.254791+0000","last_became_peered":"2023-01-15T14:35:02.254791+0000","last_unstale":"2023-01-15T14:35:42.831583+0000","last_undegraded":"2023-01-15T14:35:42.831583+0000","last_fullsized":"2023-01-15T14:35:42.831583+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":1.007059565,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"}]},{"pgid":"3.70","version":"206'252","reported_seq":1301,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.976856+0000","last_change":"2023-01-15T14:35:40.852597+0000","last_active":"2023-01-15T14:35:41.976856+0000","last_peered":"2023-01-15T14:35:41.976856+0000","last_clean":"2023-01-15T14:35:41.976856+0000","last_became_active":"2023-01-15T14:35:38.181940+0000","last_became_peered":"2023-01-15T14:35:38.181940+0000","last_unstale":"2023-01-15T14:35:41.976856+0000","last_undegraded":"2023-01-15T14:35:41.976856+0000","last_fullsized":"2023-01-15T14:35:41.976856+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:34:36.078678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:36.078678+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.158842567,"objects_trimmed":0,"snaptrim_duration":0.84201322499999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.71","version":"204'150","reported_seq":1014,"reported_epoch":453,"state":"active+clean","last_fresh":"2023-01-15T14:35:43.147322+0000","last_change":"2023-01-15T14:35:40.216242+0000","last_active":"2023-01-15T14:35:43.147322+0000","last_peered":"2023-01-15T14:35:43.147322+0000","last_clean":"2023-01-15T14:35:43.147322+0000","last_became_active":"2023-01-15T14:35:02.151543+0000","last_became_peered":"2023-01-15T14:35:02.151543+0000","last_unstale":"2023-01-15T14:35:43.147322+0000","last_undegraded":"2023-01-15T14:35:43.147322+0000","last_fullsized":"2023-01-15T14:35:43.147322+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:31.314","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.112184091,"stat_sum":{"num_bytes":3558907,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10725,"num_write":4,"num_write_kb":3090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3605110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.72","version":"394'169","reported_seq":1151,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.433697+0000","last_change":"2023-01-15T14:35:42.433863+0000","last_active":"2023-01-15T14:35:42.433697+0000","last_peered":"2023-01-15T14:35:42.433697+0000","last_clean":"2023-01-15T14:35:42.433697+0000","last_became_active":"2023-01-15T14:35:02.148756+0000","last_became_peered":"2023-01-15T14:35:02.148756+0000","last_unstale":"2023-01-15T14:35:42.433697+0000","last_undegraded":"2023-01-15T14:35:42.433697+0000","last_fullsized":"2023-01-15T14:35:42.433697+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.304809624,"stat_sum":{"num_bytes":2663943,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3524,"num_write":1,"num_write_kb":1428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2663943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.73","version":"204'275","reported_seq":1498,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.976950+0000","last_change":"2023-01-15T14:35:40.211413+0000","last_active":"2023-01-15T14:35:41.976950+0000","last_peered":"2023-01-15T14:35:41.976950+0000","last_clean":"2023-01-15T14:35:41.976950+0000","last_became_active":"2023-01-15T14:35:02.250324+0000","last_became_peered":"2023-01-15T14:35:02.250324+0000","last_unstale":"2023-01-15T14:35:41.976950+0000","last_undegraded":"2023-01-15T14:35:41.976950+0000","last_fullsized":"2023-01-15T14:35:41.976950+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:34.588","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.2146052,"stat_sum":{"num_bytes":3485477,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":11189,"num_write":7,"num_write_kb":3286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.74","version":"169'138","reported_seq":854,"reported_epoch":452,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:42.831469+0000","last_change":"2023-01-15T14:35:42.831661+0000","last_active":"2023-01-15T14:35:42.831469+0000","last_peered":"2023-01-15T14:35:42.831469+0000","last_clean":"2023-01-15T14:35:42.831469+0000","last_became_active":"2023-01-15T14:35:38.078356+0000","last_became_peered":"2023-01-15T14:35:38.078356+0000","last_unstale":"2023-01-15T14:35:42.831469+0000","last_undegraded":"2023-01-15T14:35:42.831469+0000","last_fullsized":"2023-01-15T14:35:42.831469+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.70175273199999999,"stat_sum":{"num_bytes":613464,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2809,"num_write":2,"num_write_kb":305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12e","length":"1"},{"start":"137","length":"3"}]},{"pgid":"3.75","version":"110'55","reported_seq":87,"reported_epoch":452,"state":"active+clean","last_fresh":"2023-01-15T14:35:41.978365+0000","last_change":"2023-01-15T14:35:40.499843+0000","last_active":"2023-01-15T14:35:41.978365+0000","last_peered":"2023-01-15T14:35:41.978365+0000","last_clean":"2023-01-15T14:35:41.978365+0000","last_became_active":"2023-01-15T14:35:21.196393+0000","last_became_peered":"2023-01-15T14:35:21.196393+0000","last_unstale":"2023-01-15T14:35:41.978365+0000","last_undegraded":"2023-01-15T14:35:41.978365+0000","last_fullsized":"2023-01-15T14:35:41.978365+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40170904400000002,"stat_sum":{"num_bytes":796228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5170,"num_write":3,"num_write_kb":1557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":531514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":136,"stat_sum":{"num_bytes":388683174,"num_objects":171,"num_object_clones":119,"num_object_copies":513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":14,"num_read":1190,"num_read_kb":1332927,"num_write":901,"num_write_kb":365778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1101,"num_bytes_recovered":743703550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25598,"ondisk_log_size":25598,"up":402,"acting":398,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574941,"num_pgs":69,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1133076,"kb_used_data":92566876,"kb_used_omap":14028,"kb_used_meta":0,"kb_avail":92566304,"statfs":{"total":95948165120,"available":94787895296,"internally_reserved":585728,"allocated":94788481024,"data_stored":94788481024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14364824,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":6.6619999999999999,"5min":4.6289999999999996,"15min":4.0369999999999999},"min":{"1min":0.53400000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":83.435000000000002,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.77100000000000002,"5min":4.0030000000000001,"15min":7.4720000000000004},"min":{"1min":0.59599999999999997,"5min":0.314,"15min":0.314},"max":{"1min":1.1599999999999999,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.88900000000000001}]},{"osd":1,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":3.6589999999999998,"15min":5.5670000000000002},"min":{"1min":0.64300000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.03,"5min":204.74199999999999,"15min":204.74199999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.022,"5min":8.3290000000000006,"15min":6.899},"min":{"1min":0.67500000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":3.4089999999999998,"5min":97.704999999999998,"15min":97.704999999999998},"last":1.002}]},{"osd":2,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":9.9890000000000008,"5min":5.6239999999999997,"15min":5.6769999999999996},"min":{"1min":0.54400000000000004,"5min":0.377,"15min":0.377},"max":{"1min":76.272999999999996,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.75,"15min":4.1500000000000004},"min":{"1min":0.64900000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.105,"5min":162.96799999999999,"15min":162.96799999999999},"last":0.84799999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":8.1660000000000004,"5min":6.375,"15min":3.6560000000000001},"min":{"1min":0.69499999999999995,"5min":0.434,"15min":0.434},"max":{"1min":76.114999999999995,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.97299999999999998,"5min":2.2599999999999998,"15min":3.8149999999999999},"min":{"1min":0.70199999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7170000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.96899999999999997}]},{"osd":4,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":3.6850000000000001,"5min":3.8860000000000001,"15min":7.3040000000000003},"min":{"1min":0.48099999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":40.563000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":0.94699999999999995},{"interface":"front","average":{"1min":24.846,"5min":9.4670000000000005,"15min":4.8280000000000003},"min":{"1min":0.74199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":184.37200000000001,"5min":184.37200000000001,"15min":184.37200000000001},"last":0.90700000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":3.4340000000000002,"15min":5.3040000000000003},"min":{"1min":0.58799999999999997,"5min":0.51200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1400000000000001,"5min":96.263999999999996,"15min":96.263999999999996},"last":0.746},{"interface":"front","average":{"1min":8.2710000000000008,"5min":3.2669999999999999,"15min":4.4139999999999997},"min":{"1min":0.871,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":76.159000000000006,"5min":76.159000000000006,"15min":95.430000000000007},"last":0.871}]},{"osd":6,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":3.4929999999999999,"15min":6.6550000000000002},"min":{"1min":0.64200000000000002,"5min":0.496,"15min":0.496},"max":{"1min":1.401,"5min":96.299999999999997,"15min":96.299999999999997},"last":1.0409999999999999},{"interface":"front","average":{"1min":8.3369999999999997,"5min":3.3340000000000001,"15min":2.1400000000000001},"min":{"1min":0.91500000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":76.209000000000003,"5min":76.209000000000003,"15min":76.209000000000003},"last":1.113}]}]},{"osd":6,"up_from":13,"seq":55834574942,"num_pgs":72,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1082544,"kb_used_data":92632064,"kb_used_omap":6630,"kb_used_meta":0,"kb_avail":92616836,"statfs":{"total":95948165120,"available":94839640064,"internally_reserved":15593472,"allocated":94855233536,"data_stored":94855233536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6789899,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":4,"commit_latency_ns":0,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":16.390000000000001,"5min":5.2050000000000001,"15min":2.2189999999999999},"min":{"1min":0.56000000000000005,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":166.19900000000001,"5min":166.19900000000001,"15min":166.19900000000001},"last":0.438},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.76000000000000001,"15min":0.85099999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92900000000000005,"5min":1.474,"15min":19.413},"last":74.150000000000006}]},{"osd":1,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.8109999999999999,"5min":2.194,"15min":2.3559999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":45.875999999999998,"5min":56.844999999999999,"15min":99.203000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":5.5110000000000001,"5min":2.5609999999999999,"15min":2.218},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.439},"max":{"1min":69.733000000000004,"5min":69.733000000000004,"15min":98.882000000000005},"last":0.57499999999999996}]},{"osd":2,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.069,"15min":1.518},"min":{"1min":0.55600000000000005,"5min":0.45600000000000002,"15min":0.41399999999999998},"max":{"1min":2.081,"5min":23.135999999999999,"15min":128.173},"last":0.41499999999999998},{"interface":"front","average":{"1min":3.9500000000000002,"5min":1.7430000000000001,"15min":1.95},"min":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":45.773000000000003,"5min":45.773000000000003,"15min":114.724},"last":0.60399999999999998}]},{"osd":3,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.8319999999999999,"5min":2.4910000000000001,"15min":4.8010000000000002},"min":{"1min":0.56200000000000006,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":45.813000000000002,"5min":93.710999999999999,"15min":93.710999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":11.069000000000001,"5min":5.4180000000000001,"15min":8.8190000000000008},"min":{"1min":0.625,"5min":0.625,"15min":0.60599999999999998},"max":{"1min":152.66499999999999,"5min":167.178,"15min":255.24299999999999},"last":0.55400000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":12.531000000000001,"5min":21.152000000000001,"15min":7.7300000000000004},"min":{"1min":0.46300000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":107.03,"5min":1256.867,"15min":1256.867},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.90400000000000003,"5min":14.872,"15min":9.3019999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.4350000000000001,"5min":1255.77,"15min":1255.77},"last":0.51500000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.8370000000000002,"5min":1.532,"15min":1.415},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":45.673999999999999,"5min":45.673999999999999,"15min":67.469999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":5.5330000000000004,"5min":1.8799999999999999,"15min":1.2589999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.46100000000000002},"max":{"1min":69.554000000000002,"5min":69.554000000000002,"15min":69.554000000000002},"last":0.73899999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.0289999999999999,"15min":3.056},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51300000000000001},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":78.888000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":9.6920000000000002,"5min":3.6299999999999999,"15min":3.6520000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.499},"max":{"1min":85.358000000000004,"5min":85.358000000000004,"15min":121.297},"last":0.86199999999999999}]}]},{"osd":5,"up_from":0,"seq":55834574942,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":13,"seq":55834574941,"num_pgs":65,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":962256,"kb_used_data":92737324,"kb_used_omap":19627,"kb_used_meta":0,"kb_avail":92737124,"statfs":{"total":95948165120,"available":94962814976,"internally_reserved":204800,"allocated":94963019776,"data_stored":94963019776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20098382,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":21.449999999999999,"15min":10.609},"min":{"1min":0.505,"5min":0.44,"15min":0.44},"max":{"1min":1.1200000000000001,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":14.188000000000001,"5min":5.1139999999999999,"15min":2.4140000000000001},"min":{"1min":0.53300000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":180.33600000000001,"5min":180.33600000000001,"15min":180.33600000000001},"last":42.341999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.954,"15min":6.4569999999999999},"min":{"1min":0.5,"5min":0.49299999999999999,"15min":0.47399999999999998},"max":{"1min":1.9039999999999999,"5min":69.415000000000006,"15min":171.91499999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":4.3010000000000002,"5min":6.0970000000000004,"15min":2.8900000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":70.757000000000005,"5min":162.976,"15min":162.976},"last":0.54700000000000004}]},{"osd":2,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":5.4720000000000004,"5min":23.584,"15min":11.519},"min":{"1min":0.58499999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":98.298000000000002,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.276,"15min":1.0289999999999999},"min":{"1min":0.66200000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.089,"5min":32.081000000000003,"15min":32.081000000000003},"last":0.64800000000000002}]},{"osd":3,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":15.048,"5min":5.5209999999999999,"15min":2.7400000000000002},"min":{"1min":0.67600000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":180.37200000000001,"5min":180.37200000000001,"15min":180.37200000000001},"last":42.405999999999999},{"interface":"front","average":{"1min":6.3010000000000002,"5min":24.248000000000001,"15min":12.223000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":79.888000000000005,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":0.749}]},{"osd":5,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.88200000000000001,"15min":0.90500000000000003},"min":{"1min":0.60299999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.214,"5min":2.577,"15min":2.577},"last":0.72399999999999998},{"interface":"front","average":{"1min":12.973000000000001,"5min":4.3120000000000003,"15min":2.8780000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":180.255,"5min":180.255,"15min":180.255},"last":42.457999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":3.6509999999999998,"5min":2.0070000000000001,"15min":4.7270000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":58.231999999999999,"5min":58.231999999999999,"15min":99.780000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":1.024,"5min":21.626999999999999,"15min":10.375999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.3109999999999999,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":0.77300000000000002}]},{"osd":7,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":24.492000000000001,"5min":8.2859999999999996,"15min":10.702},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":338.72699999999998,"5min":338.72699999999998,"15min":338.72699999999998},"last":42.503},{"interface":"front","average":{"1min":1.028,"5min":22.184999999999999,"15min":14.502000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.4530000000000001,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":0.88500000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574941,"num_pgs":67,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":999444,"kb_used_data":92701252,"kb_used_omap":19862,"kb_used_meta":0,"kb_avail":92699936,"statfs":{"total":95948165120,"available":94924734464,"internally_reserved":1347584,"allocated":94926082048,"data_stored":94926082048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20338824,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":6.3230000000000004,"15min":5.6289999999999996},"min":{"1min":0.32400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.98199999999999998,"5min":122.065,"15min":174.52199999999999},"last":0.495},{"interface":"front","average":{"1min":4.3959999999999999,"5min":3.7160000000000002,"15min":6.0439999999999996},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":68.239000000000004,"5min":74.280000000000001,"15min":93.674000000000007},"last":0.72499999999999998}]},{"osd":1,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.0190000000000001,"5min":5.391,"15min":10.595000000000001},"min":{"1min":0.41499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":62.805999999999997,"5min":117.664,"15min":159.256},"last":0.66400000000000003},{"interface":"front","average":{"1min":2.5659999999999998,"5min":10.003,"15min":4.351},"min":{"1min":0.32900000000000001,"5min":0.317,"15min":0.317},"max":{"1min":35.75,"5min":155.44,"15min":174.578},"last":0.68999999999999995}]},{"osd":2,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.3869999999999996,"5min":2.843,"15min":3.129},"min":{"1min":0.433,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":68.156000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.626},{"interface":"front","average":{"1min":0.70699999999999996,"5min":5.1600000000000001,"15min":3.633},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.004,"5min":122.13,"15min":174.45099999999999},"last":0.77700000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.5679999999999996,"5min":29.02,"15min":13.804},"min":{"1min":0.58699999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":68.296000000000006,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":4.242,"5min":29.707000000000001,"15min":19.187000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":62.914999999999999,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":1.042}]},{"osd":5,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.2949999999999999,"5min":4.6929999999999996,"15min":10.602},"min":{"1min":0.58399999999999996,"5min":0.504,"15min":0.504},"max":{"1min":63.055,"5min":117.744,"15min":159.43000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":4.3940000000000001,"15min":2.6699999999999999},"min":{"1min":0.73399999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.306,"5min":122.208,"15min":174.61000000000001},"last":0.97599999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":6.6639999999999997,"5min":5.5419999999999998,"15min":3.0579999999999998},"min":{"1min":0.67500000000000004,"5min":0.65900000000000003,"15min":0.63500000000000001},"max":{"1min":54.311999999999998,"5min":122.26600000000001,"15min":174.63999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":8.7650000000000006,"5min":2.5550000000000002,"15min":3.3069999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":75.340999999999994,"5min":75.340999999999994,"15min":75.340999999999994},"last":0.90900000000000003}]},{"osd":7,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":10.919,"5min":3.0070000000000001,"15min":3.6200000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":96.635000000000005,"5min":96.635000000000005,"15min":96.635000000000005},"last":0.996},{"interface":"front","average":{"1min":4.351,"5min":10.726000000000001,"15min":15.128},"min":{"1min":0.69499999999999995,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":62.984000000000002,"5min":281.13400000000001,"15min":281.13400000000001},"last":1.1479999999999999}]}]},{"osd":2,"up_from":0,"seq":55834574941,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":63,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":983308,"kb_used_data":92716336,"kb_used_omap":18434,"kb_used_meta":0,"kb_avail":92716072,"statfs":{"total":95948165120,"available":94941257728,"internally_reserved":270336,"allocated":94941528064,"data_stored":94941528064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18877120,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,0,0,0,0,0,0,1],"upper_bound":16384},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.6840000000000002,"5min":2.4870000000000001,"15min":3.6699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":57.098999999999997,"5min":100.533,"15min":144.12799999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":6.8630000000000004,"5min":7.8300000000000001,"15min":5.2439999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":101.319,"5min":176.01599999999999,"15min":176.01599999999999},"last":0.34699999999999998}]},{"osd":2,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":3.4319999999999999,"15min":1.762},"min":{"1min":0.36299999999999999,"5min":0.25700000000000001,"15min":0.245},"max":{"1min":0.80900000000000005,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68600000000000005,"15min":2.8769999999999998},"min":{"1min":0.372,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.98099999999999998,"5min":3.9780000000000002,"15min":48.119999999999997},"last":0.61899999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":9.3140000000000001,"5min":7.5700000000000003,"15min":14.470000000000001},"min":{"1min":0.33700000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":125.736,"5min":245.637,"15min":282.01400000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":6.9790000000000001,"5min":9.157,"15min":7.5739999999999998},"min":{"1min":0.40899999999999997,"5min":0.253,"15min":0.253},"max":{"1min":101.401,"5min":176.13900000000001,"15min":176.13900000000001},"last":0.40899999999999997}]},{"osd":4,"last update":"Sun Jan 15 14:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":19.507000000000001,"15min":9.5679999999999996},"min":{"1min":0.44900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.155,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":15.012,"15min":7.5220000000000002},"min":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.28,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":0.83699999999999997}]},{"osd":5,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":2.367,"15min":1.47},"min":{"1min":0.61299999999999999,"5min":0.56899999999999995,"15min":0.433},"max":{"1min":1.3999999999999999,"5min":69.478999999999999,"15min":69.478999999999999},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.95499999999999996,"15min":3.044},"min":{"1min":0.64300000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8049999999999999,"5min":4.0780000000000003,"15min":46.905999999999999},"last":0.69799999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":7.2590000000000003,"5min":8.1709999999999994,"15min":5.7880000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":101.45999999999999,"5min":176.249,"15min":176.249},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.98099999999999998,"5min":2.4409999999999998,"15min":1.55},"min":{"1min":0.66600000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.6759999999999999,"5min":69.549999999999997,"15min":69.549999999999997},"last":0.879}]},{"osd":7,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":7.4199999999999999,"5min":6.4710000000000001,"15min":6.7809999999999997},"min":{"1min":0.58699999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":97.539000000000001,"5min":157.03700000000001,"15min":157.03700000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":17.943000000000001,"5min":10.349,"15min":6.367},"min":{"1min":0.55400000000000005,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":193.74600000000001,"5min":193.74600000000001,"15min":193.74600000000001},"last":49.335000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574942,"num_pgs":76,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1083760,"kb_used_data":92615820,"kb_used_omap":14417,"kb_used_meta":0,"kb_avail":92615620,"statfs":{"total":95948165120,"available":94838394880,"internally_reserved":204800,"allocated":94838599680,"data_stored":94838599680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14763774,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":10.833,"5min":4.6390000000000002,"15min":2.2109999999999999},"min":{"1min":0.27100000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":127.86,"5min":127.86,"15min":127.86},"last":0.53800000000000003},{"interface":"front","average":{"1min":7.3220000000000001,"5min":5.681,"15min":2.7040000000000002},"min":{"1min":0.29899999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":100.407,"5min":162.91200000000001,"15min":162.91200000000001},"last":0.93600000000000005}]},{"osd":2,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":7.4370000000000003,"5min":3.052,"15min":2.0499999999999998},"min":{"1min":0.38900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":100.58799999999999,"5min":100.58799999999999,"15min":100.58799999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.55200000000000005,"5min":1.0900000000000001,"15min":1.339},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28499999999999998},"max":{"1min":0.80600000000000005,"5min":48.893000000000001,"15min":71.406999999999996},"last":0.82199999999999995}]},{"osd":3,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":11.68,"5min":9.2279999999999998,"15min":13.448},"min":{"1min":0.35999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":127.803,"5min":199.08000000000001,"15min":284.62799999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":7.452,"5min":6.2939999999999996,"15min":3.6000000000000001},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.33600000000000002},"max":{"1min":100.371,"5min":162.61600000000001,"15min":174.09399999999999},"last":0.878}]},{"osd":4,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":15.289999999999999,"15min":6.1970000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96899999999999997,"5min":1486.309,"15min":1486.309},"last":1.0389999999999999},{"interface":"front","average":{"1min":11.106,"5min":18.841999999999999,"15min":7.4020000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":127.61499999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":1.1200000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":7.6619999999999999,"5min":3.2709999999999999,"15min":7.3550000000000004},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":100.46299999999999,"5min":100.46299999999999,"15min":176.34399999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.92100000000000004,"15min":1.2490000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.391,"5min":2.2959999999999998,"15min":71.563999999999993},"last":1.1479999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":11.205,"5min":5.0679999999999996,"15min":2.7120000000000002},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":127.68000000000001,"5min":127.68000000000001,"15min":127.68000000000001},"last":0.91300000000000003},{"interface":"front","average":{"1min":7.7210000000000001,"5min":3.3650000000000002,"15min":2.1949999999999998},"min":{"1min":0.65200000000000002,"5min":0.64600000000000002,"15min":0.57499999999999996},"max":{"1min":100.544,"5min":100.544,"15min":100.544},"last":1.224}]},{"osd":7,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.7789999999999999,"15min":10.553000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4690000000000001,"5min":73.864999999999995,"15min":292.61099999999999},"last":0.95299999999999996},{"interface":"front","average":{"1min":17.018999999999998,"5min":6.2489999999999997,"15min":3.1059999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":127.46599999999999,"5min":127.46599999999999,"15min":127.46599999999999},"last":0.99099999999999999}]}]}],"pool_statfs":[]}} 2023-01-15T14:35:45.947 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:35:45.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 146 2023-01-15T14:35:45.952 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:35:45.954 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:45.954 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:45.954 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:35:45.955 INFO:tasks.rados.rados.0.smithi090.stdout:1683: finishing rollback tid 1 to smithi09025264-46 2023-01-15T14:35:45.955 INFO:tasks.rados.rados.0.smithi090.stdout:1683: finishing rollback tid 2 to smithi09025264-46 2023-01-15T14:35:45.955 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 428 (ObjNum 525 snap 158 seq_num 525) dirty exists 2023-01-15T14:35:45.955 INFO:tasks.rados.rados.0.smithi090.stdout:1683: done (0 left) 2023-01-15T14:35:45.956 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:45.952+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:46.055 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:46.045+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:46.100 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:35:46.101 INFO:tasks.rados.rados.0.smithi090.stdout:1695: done (0 left) 2023-01-15T14:35:46.101 INFO:tasks.rados.rados.0.smithi090.stdout:1696: delete oid 49 current snap is 183 2023-01-15T14:35:46.111 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:46.111 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:46.122 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:35:46.147 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:46.253 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:46.253 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:46.253 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:46.253 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:46.254 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:46.245+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:46.259 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:46.281 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:46.302 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:46.430 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:35:46.430 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:35:46.430 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:35:46.431 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:35:46.431 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:35:46.431 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:35:46.431 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:35:46.431 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:35:46.431 INFO:tasks.rados.rados.0.smithi090.stdout:1696: done (0 left) 2023-01-15T14:35:46.432 INFO:tasks.rados.rados.0.smithi090.stdout:1697: write (excl) oid 30 current snap is 183 2023-01-15T14:35:46.432 INFO:tasks.rados.rados.0.smithi090.stdout:1697: seq_num 605 ranges {568235=717532,1753253=414982} 2023-01-15T14:35:46.433 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:46.433 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:46.433 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:46.433 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:46.433 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:46.433 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:35:46.434 INFO:tasks.rados.rados.0.smithi090.stdout:1697: writing smithi09025264-30 from 568235 to 1285767 tid 1 2023-01-15T14:35:46.434 INFO:tasks.rados.rados.0.smithi090.stdout:1697: writing smithi09025264-30 from 1753253 to 2168235 tid 2 2023-01-15T14:35:46.434 INFO:tasks.rados.rados.0.smithi090.stdout:1698: rollback oid 23 current snap is 183 2023-01-15T14:35:46.435 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 23 to 159 2023-01-15T14:35:46.435 INFO:tasks.rados.rados.0.smithi090.stdout:1699: snap_create 2023-01-15T14:35:46.435 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:35:46.435 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:46.424+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:46.487 INFO:tasks.rados.rados.0.smithi090.stdout:1698: finishing rollback tid 0 to smithi09025264-23 2023-01-15T14:35:46.653 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.653 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:158 3.4 3:25618ef5:::smithi09025264-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 439.48798301599999, 2023-01-15T14:35:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70885734099999997, 2023-01-15T14:35:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 158 2023-01-15T14:35:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.654 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.655 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084514+0000", 2023-01-15T14:35:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084771+0000", 2023-01-15T14:35:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025696099999999999 2023-01-15T14:35:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:46.656 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084797+0000", 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6322999999999999e-05 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084804+0000", 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.9999999999999999e-06 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288012+0000", 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203207894 2023-01-15T14:35:46.657 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288169+0000", 2023-01-15T14:35:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156813 2023-01-15T14:35:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592129+0000", 2023-01-15T14:35:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30395998600000002 2023-01-15T14:35:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.658 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592275+0000", 2023-01-15T14:35:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00014683400000000001 2023-01-15T14:35:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.592490+0000", 2023-01-15T14:35:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000215121 2023-01-15T14:35:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.659 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593021+0000", 2023-01-15T14:35:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00053074899999999996 2023-01-15T14:35:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593043+0000", 2023-01-15T14:35:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1841e-05 2023-01-15T14:35:46.660 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593315+0000", 2023-01-15T14:35:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027153000000000002 2023-01-15T14:35:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:46.661 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655597+0000", 2023-01-15T14:35:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.062282671999999997 2023-01-15T14:35:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695719+0000", 2023-01-15T14:35:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.040121308000000001 2023-01-15T14:35:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793143+0000", 2023-01-15T14:35:46.662 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097424483000000006 2023-01-15T14:35:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793263+0000", 2023-01-15T14:35:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000119703 2023-01-15T14:35:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793372+0000", 2023-01-15T14:35:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010896299999999999 2023-01-15T14:35:46.663 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:159 3.4 3:25618ef5:::smithi09025264-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 439.48768321900002, 2023-01-15T14:35:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71062588999999998, 2023-01-15T14:35:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:46.664 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 159 2023-01-15T14:35:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.665 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084814+0000", 2023-01-15T14:35:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084813+0000", 2023-01-15T14:35:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.666 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084819+0000", 2023-01-15T14:35:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3859999999999998e-06 2023-01-15T14:35:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084829+0000", 2023-01-15T14:35:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9899999999999992e-06 2023-01-15T14:35:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:46.667 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084836+0000", 2023-01-15T14:35:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4610000000000002e-06 2023-01-15T14:35:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288230+0000", 2023-01-15T14:35:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20339406700000001 2023-01-15T14:35:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.288349+0000", 2023-01-15T14:35:46.668 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011872 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593318+0000", 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30496938099999998 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593400+0000", 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1450000000000001e-05 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.669 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593535+0000", 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000135069 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593759+0000", 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00022448300000000001 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:46.670 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593774+0000", 2023-01-15T14:35:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4479e-05 2023-01-15T14:35:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593841+0000", 2023-01-15T14:35:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.6978999999999997e-05 2023-01-15T14:35:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.655617+0000", 2023-01-15T14:35:46.671 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.061776385000000003 2023-01-15T14:35:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.793451+0000", 2023-01-15T14:35:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.137833912 2023-01-15T14:35:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795289+0000", 2023-01-15T14:35:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001837792 2023-01-15T14:35:46.672 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795351+0000", 2023-01-15T14:35:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1712999999999997e-05 2023-01-15T14:35:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795440+0000", 2023-01-15T14:35:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9419000000000006e-05 2023-01-15T14:35:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.673 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:160 3.4 3:25618ef5:::smithi09025264-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 439.48765343500003, 2023-01-15T14:35:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71100275599999996, 2023-01-15T14:35:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:46.674 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 160 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:46.675 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084844+0000", 2023-01-15T14:35:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084843+0000", 2023-01-15T14:35:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084847+0000", 2023-01-15T14:35:46.676 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8060000000000001e-06 2023-01-15T14:35:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084859+0000", 2023-01-15T14:35:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1579999999999999e-05 2023-01-15T14:35:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.084864+0000", 2023-01-15T14:35:46.677 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5670000000000002e-06 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.388796+0000", 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.303932225 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.389007+0000", 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000210712 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.678 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.593862+0000", 2023-01-15T14:35:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20485476799999999 2023-01-15T14:35:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594013+0000", 2023-01-15T14:35:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015155100000000001 2023-01-15T14:35:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.679 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795503+0000", 2023-01-15T14:35:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20148999100000001 2023-01-15T14:35:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795641+0000", 2023-01-15T14:35:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013715900000000001 2023-01-15T14:35:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.680 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.795847+0000", 2023-01-15T14:35:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020613000000000001 2023-01-15T14:35:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:183 3.4 3:2022fdfa:::smithi09025264-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:46.681 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 439.35784463499999, 2023-01-15T14:35:46.813 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68102720800000005, 2023-01-15T14:35:46.813 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:46.813 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:46.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:46.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:46.813 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:46.813 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 183 2023-01-15T14:35:46.813 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.814 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:46.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:46.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:46.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.814 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.814 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:46.814 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214653+0000", 2023-01-15T14:35:46.814 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.814 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214651+0000", 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214663+0000", 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.163e-05 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.815 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214680+0000", 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7030000000000001e-05 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214692+0000", 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1875999999999999e-05 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594067+0000", 2023-01-15T14:35:46.816 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.37937513899999997 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.594199+0000", 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000131576 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695030+0000", 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100831585 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695114+0000", 2023-01-15T14:35:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.3378e-05 2023-01-15T14:35:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,5", 2023-01-15T14:35:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695250+0000", 2023-01-15T14:35:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000136549 2023-01-15T14:35:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:46.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695378+0000", 2023-01-15T14:35:46.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000127572 2023-01-15T14:35:46.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:46.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695957+0000", 2023-01-15T14:35:46.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057925000000000001 2023-01-15T14:35:46.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:46.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696090+0000", 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000133168 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.696117+0000", 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.7124e-05 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.796421+0000", 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100304038 2023-01-15T14:35:46.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:46.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895511+0000", 2023-01-15T14:35:46.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099089941000000001 2023-01-15T14:35:46.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.821 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:46.821 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895591+0000", 2023-01-15T14:35:46.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9801000000000004e-05 2023-01-15T14:35:46.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.821 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895680+0000", 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8883999999999998e-05 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:184 3.4 3:2022fdfa:::smithi09025264-46:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 439.357787938, 2023-01-15T14:35:46.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.68130706200000002, 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 184 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:46.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:46.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214709+0000", 2023-01-15T14:35:46.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:46.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214708+0000", 2023-01-15T14:35:46.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:46.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214715+0000", 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2729999999999997e-06 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214739+0000", 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3663999999999999e-05 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.214752+0000", 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.312e-05 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.690900+0000", 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.47614748000000001 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.691107+0000", 2023-01-15T14:35:46.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020686700000000001 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695455+0000", 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0043485520000000003 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.695666+0000", 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021124499999999999 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895742+0000", 2023-01-15T14:35:46.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200075739 2023-01-15T14:35:46.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:46.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.895855+0000", 2023-01-15T14:35:46.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011246100000000001 2023-01-15T14:35:46.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:28:27.896017+0000", 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000161924 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1268 3.14 3:2911de9d:::smithi09025264-34:head [rollback 55] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 348.694099865, 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5424722239999999, 2023-01-15T14:35:46.829 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1268 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878398+0000", 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878396+0000", 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.831 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:46.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878404+0000", 2023-01-15T14:35:46.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2570000000000007e-06 2023-01-15T14:35:46.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.832 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:46.832 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878420+0000", 2023-01-15T14:35:46.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5852e-05 2023-01-15T14:35:46.832 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.832 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.878429+0000", 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.9260000000000006e-06 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064077+0000", 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18564828799999999 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:58.064091+0000", 2023-01-15T14:35:46.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3611999999999999e-05 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069128+0000", 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.005037424 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.069390+0000", 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000261981 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:46.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.420870+0000", 2023-01-15T14:35:46.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.35147932199999998 2023-01-15T14:35:46.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.835 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:46.835 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.835 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1290 3.10 3:0b54190a:::smithi09025264-17:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e111)", 2023-01-15T14:35:46.835 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:46.836 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 348.63660218199999, 2023-01-15T14:35:46.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2766481860000001, 2023-01-15T14:35:46.836 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:46.836 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:46.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:46.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:46.836 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:46.836 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1290 2023-01-15T14:35:46.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.836 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935895+0000", 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:46.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935893+0000", 2023-01-15T14:35:46.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:46.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:46.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935906+0000", 2023-01-15T14:35:46.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2634e-05 2023-01-15T14:35:46.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:46.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935937+0000", 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1056000000000002e-05 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.935951+0000", 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4352000000000001e-05 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965684+0000", 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029733005999999999 2023-01-15T14:35:46.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:57.965695+0000", 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0942e-05 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166759+0000", 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.201064211 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.840 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.166924+0000", 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000165234 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,6", 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.167129+0000", 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000205126 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168271+0000", 2023-01-15T14:35:46.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011411310000000001 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168279+0000", 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0190000000000006e-06 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.168309+0000", 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0114999999999999e-05 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169081+0000", 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077234700000000005 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169160+0000", 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.8899000000000005e-05 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169659+0000", 2023-01-15T14:35:46.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049929900000000003 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.169696+0000", 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6906999999999997e-05 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:29:59.212543+0000", 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042847221999999997 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:46.844 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.845 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1473 3.19 3:9e7635b9:::smithi09025264-21:head [copy-from ver 193] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:46.845 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:46.845 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 327.26795067699999, 2023-01-15T14:35:46.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2232503459999999, 2023-01-15T14:35:46.845 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:46.845 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:46.845 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:46.845 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:46.845 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1473 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304547+0000", 2023-01-15T14:35:46.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304544+0000", 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999971 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304560+0000", 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6305999999999999e-05 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304599+0000", 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.8670000000000001e-05 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304616+0000", 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6711e-05 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:46.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304715+0000", 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9703000000000004e-05 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304963+0000", 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000247373 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.209564+0000", 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9046015270000001 2023-01-15T14:35:46.849 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210954+0000", 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001389901 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.210985+0000", 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0623000000000001e-05 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.850 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:46.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213056+0000", 2023-01-15T14:35:46.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0020716789999999999 2023-01-15T14:35:46.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:46.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.213118+0000", 2023-01-15T14:35:46.851 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1657999999999998e-05 2023-01-15T14:35:46.851 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.851 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.851 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:46.851 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.215716+0000", 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025973939999999998 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527360+0000", 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.311644808 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527563+0000", 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020299200000000001 2023-01-15T14:35:46.852 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.527797+0000", 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023375499999999999 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1474 3.19 3:9e7635b9:::smithi09025264-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:46.853 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 327.26784803599998, 2023-01-15T14:35:46.854 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.323602068, 2023-01-15T14:35:46.854 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:46.854 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:46.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:46.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:46.854 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:46.854 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1474 2023-01-15T14:35:47.074 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.074 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.074 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304649+0000", 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.075 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304648+0000", 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304659+0000", 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0403e-05 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304675+0000", 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6178999999999999e-05 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.076 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.304685+0000", 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.9320000000000008e-06 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305187+0000", 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00050203400000000005 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:47.077 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:19.305267+0000", 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0372999999999998e-05 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211374+0000", 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.906106549 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.211577+0000", 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020278900000000001 2023-01-15T14:35:47.078 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628006+0000", 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.416429622 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628121+0000", 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011530400000000001 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.079 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:22.628251+0000", 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000129954 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1835 3.4 3:25618ef5:::smithi09025264-40:head [rollback 7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.66686604099999, 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1299815579999999, 2023-01-15T14:35:47.080 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:47.081 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:47.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:47.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:47.081 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:47.081 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1835 2023-01-15T14:35:47.081 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.081 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.081 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.081 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.081 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:47.081 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905631+0000", 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.082 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905635+0000", 2023-01-15T14:35:47.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6440000000000002e-06 2023-01-15T14:35:47.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.083 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905642+0000", 2023-01-15T14:35:47.083 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9070000000000007e-06 2023-01-15T14:35:47.083 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.083 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.083 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905650+0000", 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3370000000000002e-06 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005857+0000", 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100206876 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005878+0000", 2023-01-15T14:35:47.084 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1222000000000001e-05 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008172+0000", 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022938020000000002 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008294+0000", 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00012268000000000001 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.085 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008451+0000", 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000156331 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.996947+0000", 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884964549999999 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:47.086 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997086+0000", 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000139307 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003414+0000", 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0063272529999999997 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.003600+0000", 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018604599999999999 2023-01-15T14:35:47.087 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:47.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004047+0000", 2023-01-15T14:35:47.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00044764799999999998 2023-01-15T14:35:47.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.088 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.088 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:47.088 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004375+0000", 2023-01-15T14:35:47.088 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032739299999999999 2023-01-15T14:35:47.088 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004384+0000", 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6029999999999994e-06 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004426+0000", 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.1384000000000002e-05 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.089 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005384+0000", 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00095796200000000003 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005464+0000", 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0432000000000005e-05 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005536+0000", 2023-01-15T14:35:47.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.1831999999999999e-05 2023-01-15T14:35:47.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:47.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005647+0000", 2023-01-15T14:35:47.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110758 2023-01-15T14:35:47.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035613+0000", 2023-01-15T14:35:47.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029966129000000001 2023-01-15T14:35:47.091 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1836 3.4 3:25618ef5:::smithi09025264-40:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 282.66684061699999, 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1302056359999999, 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:47.092 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:47.093 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:47.093 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1836 2023-01-15T14:35:47.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.093 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:47.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905657+0000", 2023-01-15T14:35:47.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905656+0000", 2023-01-15T14:35:47.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:47.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905659+0000", 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.0000000000000001e-06 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905667+0000", 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.7049999999999999e-06 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.905672+0000", 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0730000000000004e-06 2023-01-15T14:35:47.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005912+0000", 2023-01-15T14:35:47.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100239413 2023-01-15T14:35:47.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:47.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.005928+0000", 2023-01-15T14:35:47.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5979000000000001e-05 2023-01-15T14:35:47.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.096 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008523+0000", 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025958980000000001 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.008639+0000", 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011558 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997115+0000", 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9884756330000002 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:06.997313+0000", 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000198316 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004467+0000", 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0071538260000000003 2023-01-15T14:35:47.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004647+0000", 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00017985200000000001 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.004999+0000", 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035199199999999997 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:47.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005304+0000", 2023-01-15T14:35:47.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030565200000000001 2023-01-15T14:35:47.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:47.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005314+0000", 2023-01-15T14:35:47.100 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.8749999999999995e-06 2023-01-15T14:35:47.100 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.100 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:47.100 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005549+0000", 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023497899999999999 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005901+0000", 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035138900000000002 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.005953+0000", 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2355999999999998e-05 2023-01-15T14:35:47.101 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006015+0000", 2023-01-15T14:35:47.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.2288999999999999e-05 2023-01-15T14:35:47.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.102 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:47.102 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.006122+0000", 2023-01-15T14:35:47.102 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000107132 2023-01-15T14:35:47.102 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.102 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:07.035862+0000", 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.029740141000000001 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2749 3.3d 3:bc1c6b44:::smithi09025264-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 195.651081374, 2023-01-15T14:35:47.103 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.69206326600000001, 2023-01-15T14:35:47.104 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:47.104 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:47.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:47.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:47.104 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:47.104 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2749 2023-01-15T14:35:47.104 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.104 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:47.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:47.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921416+0000", 2023-01-15T14:35:47.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:47.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921418+0000", 2023-01-15T14:35:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1670000000000002e-06 2023-01-15T14:35:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921426+0000", 2023-01-15T14:35:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0569999999999993e-06 2023-01-15T14:35:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:30.921429+0000", 2023-01-15T14:35:47.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9629999999999998e-06 2023-01-15T14:35:47.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107302+0000", 2023-01-15T14:35:47.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.185873327 2023-01-15T14:35:47.241 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:47.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.107657+0000", 2023-01-15T14:35:47.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035504299999999998 2023-01-15T14:35:47.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.242 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.242 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212694+0000", 2023-01-15T14:35:47.242 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.105037095 2023-01-15T14:35:47.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:47.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.212936+0000", 2023-01-15T14:35:47.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000241492 2023-01-15T14:35:47.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613125+0000", 2023-01-15T14:35:47.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40018927900000001 2023-01-15T14:35:47.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613294+0000", 2023-01-15T14:35:47.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016838000000000001 2023-01-15T14:35:47.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:31.613479+0000", 2023-01-15T14:35:47.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018577599999999999 2023-01-15T14:35:47.244 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.244 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3509 3.53 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (58) in=7b,truncate 2084167] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 117.454438239, 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.70658069899999998, 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:47.245 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:47.246 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3509 2023-01-15T14:35:47.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.246 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:47.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:47.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118059+0000", 2023-01-15T14:35:47.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:47.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118062+0000", 2023-01-15T14:35:47.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2260000000000001e-06 2023-01-15T14:35:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118067+0000", 2023-01-15T14:35:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3700000000000003e-06 2023-01-15T14:35:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118072+0000", 2023-01-15T14:35:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4880000000000001e-06 2023-01-15T14:35:47.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218253+0000", 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10018108100000001 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.218258+0000", 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.831e-06 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321058+0000", 2023-01-15T14:35:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10280054499999999 2023-01-15T14:35:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321223+0000", 2023-01-15T14:35:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016474 2023-01-15T14:35:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323014+0000", 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0017909550000000001 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323124+0000", 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000110326 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323363+0000", 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238385 2023-01-15T14:35:47.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:47.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323663+0000", 2023-01-15T14:35:47.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029987699999999999 2023-01-15T14:35:47.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:47.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323677+0000", 2023-01-15T14:35:47.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4346e-05 2023-01-15T14:35:47.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:47.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323729+0000", 2023-01-15T14:35:47.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2049999999999998e-05 2023-01-15T14:35:47.253 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.253 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.253 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:47.253 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323775+0000", 2023-01-15T14:35:47.253 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5872999999999999e-05 2023-01-15T14:35:47.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.616791+0000", 2023-01-15T14:35:47.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29301572199999998 2023-01-15T14:35:47.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.254 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.254 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.254 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824267+0000", 2023-01-15T14:35:47.254 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20747585199999999 2023-01-15T14:35:47.254 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824425+0000", 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015861900000000001 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.824640+0000", 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021475899999999999 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.255 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.256 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.256 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.256 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3510 3.53 3:cbfd2d71:::smithi09025264-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:47.256 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:47.256 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 117.4543604, 2023-01-15T14:35:47.256 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.80704639, 2023-01-15T14:35:47.256 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:47.256 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:47.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:47.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:47.256 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:47.257 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3510 2023-01-15T14:35:47.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.257 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:47.257 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.257 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.257 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.257 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.257 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118137+0000", 2023-01-15T14:35:47.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118135+0000", 2023-01-15T14:35:47.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:47.258 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.258 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.258 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.258 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118143+0000", 2023-01-15T14:35:47.258 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.0579999999999998e-06 2023-01-15T14:35:47.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118162+0000", 2023-01-15T14:35:47.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8702000000000002e-05 2023-01-15T14:35:47.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.259 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.259 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.259 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.118178+0000", 2023-01-15T14:35:47.259 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5645e-05 2023-01-15T14:35:47.259 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321268+0000", 2023-01-15T14:35:47.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20308996000000001 2023-01-15T14:35:47.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.260 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:47.260 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.321370+0000", 2023-01-15T14:35:47.260 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010268399999999999 2023-01-15T14:35:47.260 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.260 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323825+0000", 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0024546450000000001 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.323934+0000", 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010940199999999999 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.261 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.924815+0000", 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.60088053500000005 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925009+0000", 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019363599999999999 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:49.925183+0000", 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000174823 2023-01-15T14:35:47.262 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.263 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.263 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.263 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.263 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.263 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3505 3.62 3:47dfe935:::smithi09025264-29:head [rollback db] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:47.263 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:47.263 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 116.55675769, 2023-01-15T14:35:47.263 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.95152285400000003, 2023-01-15T14:35:47.263 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:47.263 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:47.263 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:47.264 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:47.264 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:47.264 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3505 2023-01-15T14:35:47.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.264 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.264 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.264 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:47.264 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.264 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.264 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.265 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.265 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015740+0000", 2023-01-15T14:35:47.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015739+0000", 2023-01-15T14:35:47.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:47.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.556 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015742+0000", 2023-01-15T14:35:47.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.869e-06 2023-01-15T14:35:47.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015748+0000", 2023-01-15T14:35:47.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0360000000000003e-06 2023-01-15T14:35:47.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015754+0000", 2023-01-15T14:35:47.557 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.733e-06 2023-01-15T14:35:47.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318179+0000", 2023-01-15T14:35:47.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30242492300000001 2023-01-15T14:35:47.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.318502+0000", 2023-01-15T14:35:47.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032355699999999998 2023-01-15T14:35:47.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:47.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365019+0000", 2023-01-15T14:35:47.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.046516950000000001 2023-01-15T14:35:47.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862003+0000", 2023-01-15T14:35:47.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49698320800000001 2023-01-15T14:35:47.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.862314+0000", 2023-01-15T14:35:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000311236 2023-01-15T14:35:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863083+0000", 2023-01-15T14:35:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076928399999999996 2023-01-15T14:35:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863709+0000", 2023-01-15T14:35:47.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00062558700000000004 2023-01-15T14:35:47.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:47.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863724+0000", 2023-01-15T14:35:47.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5288999999999999e-05 2023-01-15T14:35:47.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:47.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863799+0000", 2023-01-15T14:35:47.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.4552e-05 2023-01-15T14:35:47.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863851+0000", 2023-01-15T14:35:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.2018999999999998e-05 2023-01-15T14:35:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966087+0000", 2023-01-15T14:35:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102235952 2023-01-15T14:35:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966811+0000", 2023-01-15T14:35:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00072484800000000005 2023-01-15T14:35:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966950+0000", 2023-01-15T14:35:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013907399999999999 2023-01-15T14:35:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.967263+0000", 2023-01-15T14:35:47.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000312107 2023-01-15T14:35:47.564 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.564 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.564 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.564 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3506 3.62 3:47dfe935:::smithi09025264-29:head [append 0~0] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e340)", 2023-01-15T14:35:47.564 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:47.564 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 116.556737868, 2023-01-15T14:35:47.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.234729387, 2023-01-15T14:35:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3506 2023-01-15T14:35:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:47.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015760+0000", 2023-01-15T14:35:47.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015759+0000", 2023-01-15T14:35:47.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:47.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015763+0000", 2023-01-15T14:35:47.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.517e-06 2023-01-15T14:35:47.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015768+0000", 2023-01-15T14:35:47.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5620000000000003e-06 2023-01-15T14:35:47.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.015773+0000", 2023-01-15T14:35:47.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.3730000000000003e-06 2023-01-15T14:35:47.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.568 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365160+0000", 2023-01-15T14:35:47.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.34938724300000001 2023-01-15T14:35:47.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:47.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.365370+0000", 2023-01-15T14:35:47.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021001599999999999 2023-01-15T14:35:47.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.863926+0000", 2023-01-15T14:35:47.569 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49855574600000002 2023-01-15T14:35:47.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.864284+0000", 2023-01-15T14:35:47.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035790600000000001 2023-01-15T14:35:47.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:47.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865099+0000", 2023-01-15T14:35:47.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00081499999999999997 2023-01-15T14:35:47.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:47.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865640+0000", 2023-01-15T14:35:47.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054118899999999997 2023-01-15T14:35:47.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:47.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865653+0000", 2023-01-15T14:35:47.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3614e-05 2023-01-15T14:35:47.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:47.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.865699+0000", 2023-01-15T14:35:47.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.5751000000000003e-05 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966206+0000", 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100506741 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:50.966665+0000", 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000459345 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.249842+0000", 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28317711200000001 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250054+0000", 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021161899999999999 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:51.250489+0000", 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043499700000000001 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3572 3.65 3:a75958b1:::smithi09025264-6:head [setxattr _header (57) in=7b,truncate 2235430] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.55010804299999, 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65482807700000001, 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:47.574 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:47.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:47.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:47.575 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:47.575 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3572 2023-01-15T14:35:47.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.575 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:47.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022389+0000", 2023-01-15T14:35:47.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022388+0000", 2023-01-15T14:35:47.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:47.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022397+0000", 2023-01-15T14:35:47.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.704e-06 2023-01-15T14:35:47.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022412+0000", 2023-01-15T14:35:47.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5554e-05 2023-01-15T14:35:47.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022422+0000", 2023-01-15T14:35:47.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2839999999999995e-06 2023-01-15T14:35:47.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114230+0000", 2023-01-15T14:35:47.578 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.091808129000000002 2023-01-15T14:35:47.578 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.578 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.578 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:47.578 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.114234+0000", 2023-01-15T14:35:47.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.4329999999999997e-06 2023-01-15T14:35:47.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204200+0000", 2023-01-15T14:35:47.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.089966321000000002 2023-01-15T14:35:47.579 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.579 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.579 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:47.579 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.204439+0000", 2023-01-15T14:35:47.579 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000238595 2023-01-15T14:35:47.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.305883+0000", 2023-01-15T14:35:47.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101443713 2023-01-15T14:35:47.580 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.580 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.580 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.580 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306066+0000", 2023-01-15T14:35:47.580 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000182962 2023-01-15T14:35:47.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 4,7", 2023-01-15T14:35:47.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.306621+0000", 2023-01-15T14:35:47.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00055552800000000001 2023-01-15T14:35:47.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.581 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.581 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:47.581 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307013+0000", 2023-01-15T14:35:47.581 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039210599999999998 2023-01-15T14:35:47.581 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:47.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307033+0000", 2023-01-15T14:35:47.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9423e-05 2023-01-15T14:35:47.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.582 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.582 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:47.582 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307093+0000", 2023-01-15T14:35:47.582 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0402000000000002e-05 2023-01-15T14:35:47.582 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:47.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307445+0000", 2023-01-15T14:35:47.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000351677 2023-01-15T14:35:47.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.583 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.583 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307662+0000", 2023-01-15T14:35:47.583 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021765099999999999 2023-01-15T14:35:47.583 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.583 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676523+0000", 2023-01-15T14:35:47.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.36886056699999997 2023-01-15T14:35:47.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:47.584 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.676723+0000", 2023-01-15T14:35:47.584 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019947899999999999 2023-01-15T14:35:47.584 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.584 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.584 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.585 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677217+0000", 2023-01-15T14:35:47.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00049493900000000001 2023-01-15T14:35:47.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.585 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.585 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.585 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.585 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.585 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:3573 3.65 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e348)", 2023-01-15T14:35:47.585 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:47.585 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 109.550064916, 2023-01-15T14:35:47.585 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.65551266799999997, 2023-01-15T14:35:47.586 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:47.586 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:47.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:47.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:47.586 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:47.586 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 3573 2023-01-15T14:35:47.586 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.586 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.586 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.586 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.586 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022433+0000", 2023-01-15T14:35:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022432+0000", 2023-01-15T14:35:47.587 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:47.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022436+0000", 2023-01-15T14:35:47.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.693e-06 2023-01-15T14:35:47.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.588 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.588 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.588 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.022447+0000", 2023-01-15T14:35:47.588 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1639e-05 2023-01-15T14:35:47.588 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122543+0000", 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100095369 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122863+0000", 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032020799999999999 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.589 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:47.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.122869+0000", 2023-01-15T14:35:47.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.7640000000000002e-06 2023-01-15T14:35:47.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206080+0000", 2023-01-15T14:35:47.590 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.083211794000000006 2023-01-15T14:35:47.590 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.590 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.590 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:47.590 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.206289+0000", 2023-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020866600000000001 2023-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307193+0000", 2023-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10090391999999999 2023-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.307374+0000", 2023-01-15T14:35:47.591 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018061299999999999 2023-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677342+0000", 2023-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.369968034 2023-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.592 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.613 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.613 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677535+0000", 2023-01-15T14:35:47.613 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00019311299999999999 2023-01-15T14:35:47.613 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.613 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:57.677945+0000", 2023-01-15T14:35:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00041040299999999999 2023-01-15T14:35:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.614 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4362 3.4 3:2022fdfa:::smithi09025264-46:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2023-01-15T14:35:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:36.985449+0000", 2023-01-15T14:35:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.5870482260000003, 2023-01-15T14:35:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.71198219100000004, 2023-01-15T14:35:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:47.615 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4362 2023-01-15T14:35:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985449+0000", 2023-01-15T14:35:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985449+0000", 2023-01-15T14:35:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985447+0000", 2023-01-15T14:35:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985463+0000", 2023-01-15T14:35:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5761e-05 2023-01-15T14:35:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985492+0000", 2023-01-15T14:35:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9677000000000001e-05 2023-01-15T14:35:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985507+0000", 2023-01-15T14:35:47.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4312e-05 2023-01-15T14:35:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985646+0000", 2023-01-15T14:35:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013939599999999999 2023-01-15T14:35:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:35:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985662+0000", 2023-01-15T14:35:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5974999999999999e-05 2023-01-15T14:35:47.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.272912+0000", 2023-01-15T14:35:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.28724966400000002 2023-01-15T14:35:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.273577+0000", 2023-01-15T14:35:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00066540200000000005 2023-01-15T14:35:47.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:35:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.274338+0000", 2023-01-15T14:35:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076103800000000001 2023-01-15T14:35:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.275507+0000", 2023-01-15T14:35:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011694380000000001 2023-01-15T14:35:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.275529+0000", 2023-01-15T14:35:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.177e-05 2023-01-15T14:35:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.275613+0000", 2023-01-15T14:35:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.4050999999999999e-05 2023-01-15T14:35:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:47.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.275679+0000", 2023-01-15T14:35:47.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5425000000000005e-05 2023-01-15T14:35:47.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.490519+0000", 2023-01-15T14:35:47.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.214840436 2023-01-15T14:35:47.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:47.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.696746+0000", 2023-01-15T14:35:47.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20622658799999999 2023-01-15T14:35:47.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:47.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.696860+0000", 2023-01-15T14:35:47.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113955 2023-01-15T14:35:47.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.697431+0000", 2023-01-15T14:35:47.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00057170300000000001 2023-01-15T14:35:47.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.625 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.625 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.625 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4363 3.4 3:2022fdfa:::smithi09025264-46:head [rollback 14e] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2023-01-15T14:35:47.625 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:36.985527+0000", 2023-01-15T14:35:47.625 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 9.5869707109999993, 2023-01-15T14:35:47.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1646122490000002, 2023-01-15T14:35:47.626 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:47.626 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:47.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:47.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:47.626 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:47.626 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4363 2023-01-15T14:35:47.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.626 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:47.730 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.730 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:47.730 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985527+0000", 2023-01-15T14:35:47.730 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:47.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985527+0000", 2023-01-15T14:35:47.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:47.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.731 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.731 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:47.731 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985526+0000", 2023-01-15T14:35:47.731 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:47.731 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:47.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985530+0000", 2023-01-15T14:35:47.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7050000000000001e-06 2023-01-15T14:35:47.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.732 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:47.732 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985541+0000", 2023-01-15T14:35:47.732 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1402e-05 2023-01-15T14:35:47.732 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.732 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985547+0000", 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3979999999999998e-06 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985720+0000", 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000173172 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:47.733 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:36.985729+0000", 2023-01-15T14:35:47.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.7960000000000001e-06 2023-01-15T14:35:47.734 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.734 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.734 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.734 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.275820+0000", 2023-01-15T14:35:47.734 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29009138400000001 2023-01-15T14:35:47.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:47.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.276118+0000", 2023-01-15T14:35:47.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029761300000000001 2023-01-15T14:35:47.735 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.735 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.735 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for degraded object", 2023-01-15T14:35:47.735 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:37.276371+0000", 2023-01-15T14:35:47.735 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00025362200000000001 2023-01-15T14:35:47.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:47.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:43.150122+0000", 2023-01-15T14:35:47.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.873750963 2023-01-15T14:35:47.736 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:47.736 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:47.736 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:47.736 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:43.150139+0000", 2023-01-15T14:35:47.736 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6665999999999999e-05 2023-01-15T14:35:47.736 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.737 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.737 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.737 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:47.737 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:47.737 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:47.737 INFO:tasks.rados.rados.0.smithi090.stdout:1698: finishing rollback tid 1 to smithi09025264-23 2023-01-15T14:35:47.738 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:35:47.739 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:46.649+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:47.740 INFO:tasks.rados.rados.0.smithi090.stdout:1698: finishing rollback tid 2 to smithi09025264-23 2023-01-15T14:35:47.740 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 23 v 306 (ObjNum 406 snap 108 seq_num 406) dirty exists 2023-01-15T14:35:47.741 INFO:tasks.rados.rados.0.smithi090.stdout:1698: done (1 left) 2023-01-15T14:35:47.741 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:47.741 INFO:tasks.rados.rados.0.smithi090.stdout:1697: finishing write tid 1 to smithi09025264-30 2023-01-15T14:35:47.741 INFO:tasks.rados.rados.0.smithi090.stdout:1697: finishing write tid 2 to smithi09025264-30 2023-01-15T14:35:47.741 INFO:tasks.rados.rados.0.smithi090.stdout:1697: finishing write tid 3 to smithi09025264-30 2023-01-15T14:35:47.741 INFO:tasks.rados.rados.0.smithi090.stdout:1697: finishing write tid 4 to smithi09025264-30 2023-01-15T14:35:47.741 INFO:tasks.rados.rados.0.smithi090.stdout:1697: oid 30 updating version 0 to 254 2023-01-15T14:35:47.741 INFO:tasks.rados.rados.0.smithi090.stdout:1697: oid 30 version 254 is already newer than 252 2023-01-15T14:35:47.741 INFO:tasks.rados.rados.0.smithi090.stdout:1697: oid 30 version 254 is already newer than 253 2023-01-15T14:35:47.741 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 30 v 254 (ObjNum 605 snap 183 seq_num 605) dirty exists 2023-01-15T14:35:47.742 INFO:tasks.rados.rados.0.smithi090.stdout:1697: left oid 30 (ObjNum 605 snap 183 seq_num 605) 2023-01-15T14:35:47.742 INFO:tasks.rados.rados.0.smithi090.stdout:1697: done (0 left) 2023-01-15T14:35:47.742 INFO:tasks.rados.rados.0.smithi090.stdout:1699: done (0 left) 2023-01-15T14:35:47.742 INFO:tasks.rados.rados.0.smithi090.stdout:1700: rollback oid 44 current snap is 184 2023-01-15T14:35:47.742 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 44 to 178 2023-01-15T14:35:47.742 INFO:tasks.rados.rados.0.smithi090.stdout:1701: snap_create 2023-01-15T14:35:47.742 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:47.742 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 146 2023-01-15T14:35:47.787 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:35:47.789 INFO:tasks.rados.rados.0.smithi090.stdout:1700: finishing rollback tid 0 to smithi09025264-44 2023-01-15T14:35:47.789 INFO:tasks.rados.rados.0.smithi090.stdout:1700: finishing rollback tid 1 to smithi09025264-44 2023-01-15T14:35:47.790 INFO:tasks.rados.rados.0.smithi090.stdout:1700: finishing rollback tid 2 to smithi09025264-44 2023-01-15T14:35:47.790 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 44 v 377 (ObjNum 575 snap 177 seq_num 575) dirty exists 2023-01-15T14:35:47.790 INFO:tasks.rados.rados.0.smithi090.stdout:1700: done (0 left) 2023-01-15T14:35:47.790 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:47.736+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:47.842 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:47.833+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:48.061 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:48.062 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:48.062 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:35:48.062 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:48.062 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:35:48.063 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:48.057+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:48.087 INFO:tasks.rados.rados.0.smithi090.stdout:1701: done (0 left) 2023-01-15T14:35:48.226 INFO:tasks.rados.rados.0.smithi090.stdout:1702: rollback oid 49 current snap is 185 2023-01-15T14:35:48.226 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 49 to 179 2023-01-15T14:35:48.226 INFO:tasks.rados.rados.0.smithi090.stdout:1703: read oid 21 snap 183 2023-01-15T14:35:48.226 INFO:tasks.rados.rados.0.smithi090.stdout:1703: expect (ObjNum 506 snap 150 seq_num 506) 2023-01-15T14:35:48.227 INFO:tasks.rados.rados.0.smithi090.stdout:1704: delete oid 6 current snap is 185 2023-01-15T14:35:48.228 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:48.228 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:48.229 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:48.229 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:48.229 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:48.229 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:35:48.230 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:48.220+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:48.300 INFO:tasks.rados.rados.0.smithi090.stdout:1703: expect (ObjNum 506 snap 150 seq_num 506) 2023-01-15T14:35:48.328 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:48.321+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:48.383 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:48.383 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:48.383 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:48.383 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:48.383 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.383 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:48.383 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:48.384 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 313.48190915100002, 2023-01-15T14:35:48.384 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:35:48.384 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.384 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:48.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:48.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:48.384 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:48.384 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:35:48.384 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.384 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:48.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.385 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.385 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:48.385 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.385 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.385 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:35:48.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:48.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.386 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:35:48.386 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:35:48.386 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.386 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.386 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:35:48.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:35:48.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:35:48.387 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:35:48.387 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.387 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.387 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.387 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:35:48.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:35:48.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:48.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:35:48.388 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:35:48.388 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.388 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.388 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.388 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:35:48.389 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:35:48.389 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.389 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.389 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:35:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:35:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:35:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:35:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:35:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.390 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:35:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:35:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:35:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:35:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:48.391 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:35:48.392 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:35:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:35:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:35:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:35:48.393 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:48.394 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 313.48185820800001, 2023-01-15T14:35:48.395 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:35:48.395 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.395 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:48.395 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:48.395 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:48.395 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:48.395 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:35:48.395 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.395 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.395 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.395 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:48.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.396 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:48.396 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.396 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.396 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.396 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:35:48.397 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:48.397 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.397 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.397 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.397 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:35:48.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:35:48.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.398 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:35:48.398 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:35:48.398 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.398 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.398 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:35:48.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:35:48.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.399 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.399 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:35:48.399 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:35:48.399 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.399 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:48.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:35:48.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:35:48.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.400 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.400 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:35:48.400 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:35:48.400 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.400 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:48.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:35:48.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:35:48.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.401 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.401 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:35:48.401 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:35:48.401 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.401 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:35:48.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:35:48.402 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.402 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.402 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.402 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:35:48.402 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:35:48.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.402 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.402 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 284.57322937399999, 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:48.403 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:35:48.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.404 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:48.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.404 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.404 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.404 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.404 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:48.404 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:35:48.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:48.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.405 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.405 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.405 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:35:48.405 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:35:48.405 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.406 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:35:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:35:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:35:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:35:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.407 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:35:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:35:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:35:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:35:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:35:48.408 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:35:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 262.68619996500001, 2023-01-15T14:35:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:35:48.409 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.410 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:35:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.411 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:35:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:35:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:35:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:35:48.412 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.414 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:35:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:35:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:35:48.415 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:35:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:35:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:35:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.416 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:35:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:35:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:35:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:35:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:48.417 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:35:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:35:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:35:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:35:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.418 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:35:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:35:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:35:48.419 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:35:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:35:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:35:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:35:48.420 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.421 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:35:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:35:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:48.422 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 262.68609289099999, 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.423 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:35:48.424 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.425 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:48.426 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:35:48.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:35:48.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:35:48.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:35:48.427 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.427 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.427 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:48.427 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:35:48.427 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.428 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 254.555679373, 2023-01-15T14:35:48.429 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:48.430 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:48.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.431 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.431 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.431 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:35:48.431 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:48.431 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.432 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:35:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:35:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:35:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:35:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:35:48.433 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:48.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:35:48.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:35:48.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:48.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:35:48.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:35:48.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:35:48.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:48.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:35:48.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:35:48.438 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.438 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.438 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.438 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:35:48.438 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:35:48.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.438 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.438 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.439 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:35:48.439 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:48.439 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 187.382567881, 2023-01-15T14:35:48.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:35:48.439 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.439 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:48.439 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:35:48.441 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:35:48.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:35:48.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.442 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.442 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:35:48.442 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:35:48.442 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.442 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:35:48.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:35:48.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.443 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:48.443 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:35:48.443 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:35:48.443 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.443 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:48.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:35:48.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:35:48.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:48.444 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:35:48.444 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:35:48.444 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.444 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.444 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:48.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:35:48.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:35:48.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.445 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.445 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.445 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:35:48.445 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:35:48.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.445 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.445 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.445 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 187.378756194, 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:48.446 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.447 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.448 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:35:48.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:35:48.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:35:48.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:35:48.449 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.449 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.449 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.449 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:35:48.449 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:35:48.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:48.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:35:48.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:35:48.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.450 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.450 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:48.450 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:35:48.450 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:35:48.450 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:35:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:35:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:35:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:35:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.451 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3120 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'290, mlcod=0'0)", 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 159.13694291799999, 2023-01-15T14:35:48.452 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39853714000000001, 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:48.453 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239424+0000", 2023-01-15T14:35:48.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:48.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.454 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.454 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.454 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239747+0000", 2023-01-15T14:35:48.454 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032243300000000001 2023-01-15T14:35:48.454 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239784+0000", 2023-01-15T14:35:48.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7051000000000002e-05 2023-01-15T14:35:48.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.455 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.455 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.455 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239802+0000", 2023-01-15T14:35:48.455 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8250999999999999e-05 2023-01-15T14:35:48.455 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536033+0000", 2023-01-15T14:35:48.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29623121499999999 2023-01-15T14:35:48.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.456 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.456 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536081+0000", 2023-01-15T14:35:48.456 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2023-01-15T14:35:48.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:48.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536800+0000", 2023-01-15T14:35:48.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071948499999999996 2023-01-15T14:35:48.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:48.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536817+0000", 2023-01-15T14:35:48.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6228000000000001e-05 2023-01-15T14:35:48.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537048+0000", 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023136200000000001 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637902+0000", 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10085403900000001 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.458 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637963+0000", 2023-01-15T14:35:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1310999999999996e-05 2023-01-15T14:35:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3121 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'291, mlcod=0'0)", 2023-01-15T14:35:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 159.13494740199999, 2023-01-15T14:35:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659573799999998, 2023-01-15T14:35:48.459 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.460 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:48.460 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:48.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:48.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241420+0000", 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241854+0000", 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043330099999999998 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241878+0000", 2023-01-15T14:35:48.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4559e-05 2023-01-15T14:35:48.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241892+0000", 2023-01-15T14:35:48.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039999999999999e-05 2023-01-15T14:35:48.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.462 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536863+0000", 2023-01-15T14:35:48.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29497073400000001 2023-01-15T14:35:48.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536887+0000", 2023-01-15T14:35:48.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3683999999999999e-05 2023-01-15T14:35:48.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:48.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537124+0000", 2023-01-15T14:35:48.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023729900000000001 2023-01-15T14:35:48.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:48.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537136+0000", 2023-01-15T14:35:48.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2278e-05 2023-01-15T14:35:48.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:48.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537345+0000", 2023-01-15T14:35:48.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020865200000000001 2023-01-15T14:35:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637981+0000", 2023-01-15T14:35:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063575700000001 2023-01-15T14:35:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638018+0000", 2023-01-15T14:35:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6826999999999998e-05 2023-01-15T14:35:48.465 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3122 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'292, mlcod=0'0)", 2023-01-15T14:35:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 159.13305156199999, 2023-01-15T14:35:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39623551600000001, 2023-01-15T14:35:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:48.466 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243316+0000", 2023-01-15T14:35:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243833+0000", 2023-01-15T14:35:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051703600000000001 2023-01-15T14:35:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243863+0000", 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243879+0000", 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6263999999999999e-05 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637317+0000", 2023-01-15T14:35:48.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39343783199999999 2023-01-15T14:35:48.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637356+0000", 2023-01-15T14:35:48.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9579000000000002e-05 2023-01-15T14:35:48.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:48.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637830+0000", 2023-01-15T14:35:48.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047326399999999999 2023-01-15T14:35:48.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:48.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637885+0000", 2023-01-15T14:35:48.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5074999999999997e-05 2023-01-15T14:35:48.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:48.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638185+0000", 2023-01-15T14:35:48.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029993799999999997 2023-01-15T14:35:48.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:48.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638337+0000", 2023-01-15T14:35:48.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015208299999999999 2023-01-15T14:35:48.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.639553+0000", 2023-01-15T14:35:48.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001216389 2023-01-15T14:35:48.472 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.472 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.473 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.473 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3123 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'293, mlcod=0'0)", 2023-01-15T14:35:48.473 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:48.473 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 159.131307534, 2023-01-15T14:35:48.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.395733102, 2023-01-15T14:35:48.473 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.473 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:48.473 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245060+0000", 2023-01-15T14:35:48.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:48.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245075+0000", 2023-01-15T14:35:48.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4459e-05 2023-01-15T14:35:48.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245107+0000", 2023-01-15T14:35:48.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1795000000000003e-05 2023-01-15T14:35:48.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245121+0000", 2023-01-15T14:35:48.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4117000000000001e-05 2023-01-15T14:35:48.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638063+0000", 2023-01-15T14:35:48.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39294204300000002 2023-01-15T14:35:48.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638097+0000", 2023-01-15T14:35:48.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4385999999999999e-05 2023-01-15T14:35:48.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.477 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:48.477 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638299+0000", 2023-01-15T14:35:48.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020133099999999999 2023-01-15T14:35:48.477 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:48.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638318+0000", 2023-01-15T14:35:48.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514999999999999e-05 2023-01-15T14:35:48.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:48.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638386+0000", 2023-01-15T14:35:48.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8089000000000005e-05 2023-01-15T14:35:48.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:48.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638430+0000", 2023-01-15T14:35:48.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.439e-05 2023-01-15T14:35:48.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.640795+0000", 2023-01-15T14:35:48.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002364274 2023-01-15T14:35:48.479 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.479 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.479 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.480 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4216 3.78 3:1eaad226:::smithi09025264-49:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:48.480 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:48.480 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.412612787, 2023-01-15T14:35:48.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82042120600000001, 2023-01-15T14:35:48.480 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.480 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:48.480 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:48.480 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:48.481 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:48.481 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4216 2023-01-15T14:35:48.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.481 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:48.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:48.539 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.539 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.540 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.540 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.540 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963753+0000", 2023-01-15T14:35:48.540 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999962 2023-01-15T14:35:48.540 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963772+0000", 2023-01-15T14:35:48.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9293999999999999e-05 2023-01-15T14:35:48.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.541 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.541 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963806+0000", 2023-01-15T14:35:48.541 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3946999999999999e-05 2023-01-15T14:35:48.541 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.541 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963822+0000", 2023-01-15T14:35:48.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5744e-05 2023-01-15T14:35:48.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.542 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.542 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963930+0000", 2023-01-15T14:35:48.542 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010870300000000001 2023-01-15T14:35:48.542 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.542 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:35:48.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963946+0000", 2023-01-15T14:35:48.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5919000000000001e-05 2023-01-15T14:35:48.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.543 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.780699+0000", 2023-01-15T14:35:48.543 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.81675266400000002 2023-01-15T14:35:48.543 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.543 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.543 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.544 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784178+0000", 2023-01-15T14:35:48.544 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034785699999999998 2023-01-15T14:35:48.544 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.544 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.544 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.544 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.544 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.544 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4217 3.78 3:1eaad226:::smithi09025264-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:48.544 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:48.544 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 26.412517231999999, 2023-01-15T14:35:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.820584859, 2023-01-15T14:35:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4217 2023-01-15T14:35:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.545 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963851+0000", 2023-01-15T14:35:48.546 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963857+0000", 2023-01-15T14:35:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8699999999999997e-06 2023-01-15T14:35:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963867+0000", 2023-01-15T14:35:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2023-01-15T14:35:48.547 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963876+0000", 2023-01-15T14:35:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1120000000000002e-06 2023-01-15T14:35:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064133+0000", 2023-01-15T14:35:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100256427 2023-01-15T14:35:48.548 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064164+0000", 2023-01-15T14:35:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1649000000000001e-05 2023-01-15T14:35:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.549 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784340+0000", 2023-01-15T14:35:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72017558699999995 2023-01-15T14:35:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784437+0000", 2023-01-15T14:35:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6831999999999997e-05 2023-01-15T14:35:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.550 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4216 3.78 3:1eaad226:::smithi09025264-49:head [copy-from ver 0] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 25.37950734, 2023-01-15T14:35:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39069526799999998, 2023-01-15T14:35:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4216 2023-01-15T14:35:48.551 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:48.552 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996860+0000", 2023-01-15T14:35:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996869+0000", 2023-01-15T14:35:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2069999999999995e-06 2023-01-15T14:35:48.553 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996881+0000", 2023-01-15T14:35:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2611e-05 2023-01-15T14:35:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.096971+0000", 2023-01-15T14:35:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100089364 2023-01-15T14:35:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.554 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.284700+0000", 2023-01-15T14:35:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18772960999999999 2023-01-15T14:35:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.284814+0000", 2023-01-15T14:35:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113427 2023-01-15T14:35:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.555 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.387557+0000", 2023-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102743529 2023-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4225 3.78 3:1e54a7f5:::smithi09025264-42:head [write 1381780~411527] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.882946358000002, 2023-01-15T14:35:48.556 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42766908999999997, 2023-01-15T14:35:48.557 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.557 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:48.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:48.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:48.557 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:48.557 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4225 2023-01-15T14:35:48.557 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.557 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.557 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.557 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.557 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493422+0000", 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:48.558 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493846+0000", 2023-01-15T14:35:48.559 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042452100000000002 2023-01-15T14:35:48.559 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.559 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.559 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.559 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493861+0000", 2023-01-15T14:35:48.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5136e-05 2023-01-15T14:35:48.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.560 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.560 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493872+0000", 2023-01-15T14:35:48.560 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0958000000000001e-05 2023-01-15T14:35:48.560 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.560 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494867+0000", 2023-01-15T14:35:48.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00099453600000000007 2023-01-15T14:35:48.561 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.561 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.561 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:48.561 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495083+0000", 2023-01-15T14:35:48.561 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021589499999999999 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.815757+0000", 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32067462499999999 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.815988+0000", 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000230762 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.562 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.816381+0000", 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039289799999999998 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817159+0000", 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077780299999999998 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:48.563 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817179+0000", 2023-01-15T14:35:48.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0489e-05 2023-01-15T14:35:48.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:48.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817418+0000", 2023-01-15T14:35:48.564 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023831500000000001 2023-01-15T14:35:48.564 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.564 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.564 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:48.564 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819234+0000", 2023-01-15T14:35:48.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001816553 2023-01-15T14:35:48.565 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.565 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.565 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.565 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820529+0000", 2023-01-15T14:35:48.565 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012949719999999999 2023-01-15T14:35:48.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.566 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920621+0000", 2023-01-15T14:35:48.566 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100091965 2023-01-15T14:35:48.566 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.566 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.566 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920679+0000", 2023-01-15T14:35:48.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8319000000000002e-05 2023-01-15T14:35:48.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:48.567 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920795+0000", 2023-01-15T14:35:48.567 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115964 2023-01-15T14:35:48.567 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.567 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.567 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:48.568 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.921092+0000", 2023-01-15T14:35:48.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000296531 2023-01-15T14:35:48.568 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.568 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:48.568 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:48.568 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.568 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.568 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4226 3.78 3:1e54a7f5:::smithi09025264-42:head [write 2300501~586966] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:48.568 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:48.568 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.882454987999999, 2023-01-15T14:35:48.568 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.730699722, 2023-01-15T14:35:48.569 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:48.569 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:48.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:48.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:48.569 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:48.569 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4226 2023-01-15T14:35:48.569 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.569 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:48.569 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.569 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:48.569 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:48.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:48.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:48.570 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:48.570 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.570 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.570 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:48.570 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493913+0000", 2023-01-15T14:35:48.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:48.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:48.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494499+0000", 2023-01-15T14:35:48.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058589500000000001 2023-01-15T14:35:48.571 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.571 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.571 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:48.571 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494549+0000", 2023-01-15T14:35:48.571 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0296000000000001e-05 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494577+0000", 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8056999999999999e-05 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495162+0000", 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058508500000000005 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.572 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:48.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495377+0000", 2023-01-15T14:35:48.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021484400000000001 2023-01-15T14:35:48.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:48.573 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817336+0000", 2023-01-15T14:35:48.573 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32195888499999997 2023-01-15T14:35:48.573 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.573 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.573 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817638+0000", 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030184599999999998 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.818285+0000", 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064671399999999995 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819118+0000", 2023-01-15T14:35:48.574 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083369499999999997 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819137+0000", 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8326999999999999e-05 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819385+0000", 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024822599999999999 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.575 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820443+0000", 2023-01-15T14:35:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058219 2023-01-15T14:35:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.923538+0000", 2023-01-15T14:35:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103094642 2023-01-15T14:35:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.576 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.020572+0000", 2023-01-15T14:35:48.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097034387999999999 2023-01-15T14:35:48.577 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:48.577 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:48.577 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:48.577 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224051+0000", 2023-01-15T14:35:48.577 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20347895399999999 2023-01-15T14:35:49.016 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.016 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:49.017 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224219+0000", 2023-01-15T14:35:49.017 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016772 2023-01-15T14:35:49.017 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.017 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.017 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:49.018 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224614+0000", 2023-01-15T14:35:49.018 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039523500000000002 2023-01-15T14:35:49.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.018 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:49.018 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.018 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.019 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.019 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4227 3.78 3:1e54a7f5:::smithi09025264-42:head [write 3376974~354279] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:49.019 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:49.019 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.881730685000001, 2023-01-15T14:35:49.019 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73066059400000005, 2023-01-15T14:35:49.019 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:49.020 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:49.089 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4227 2023-01-15T14:35:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:49.090 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:49.091 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:49.092 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494637+0000", 2023-01-15T14:35:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494805+0000", 2023-01-15T14:35:49.093 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016839000000000001 2023-01-15T14:35:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494819+0000", 2023-01-15T14:35:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4287000000000001e-05 2023-01-15T14:35:49.094 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494828+0000", 2023-01-15T14:35:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8119999999999997e-06 2023-01-15T14:35:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.095 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495478+0000", 2023-01-15T14:35:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064994399999999996 2023-01-15T14:35:49.096 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495668+0000", 2023-01-15T14:35:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018991900000000001 2023-01-15T14:35:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.097 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819291+0000", 2023-01-15T14:35:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32362327600000002 2023-01-15T14:35:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819503+0000", 2023-01-15T14:35:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021184900000000001 2023-01-15T14:35:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:49.098 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819880+0000", 2023-01-15T14:35:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037692500000000001 2023-01-15T14:35:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820352+0000", 2023-01-15T14:35:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047233000000000003 2023-01-15T14:35:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.099 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.100 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:49.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820368+0000", 2023-01-15T14:35:49.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5877999999999999e-05 2023-01-15T14:35:49.182 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.182 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.182 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:49.182 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820533+0000", 2023-01-15T14:35:49.182 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164508 2023-01-15T14:35:49.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:49.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821690+0000", 2023-01-15T14:35:49.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011576290000000001 2023-01-15T14:35:49.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.183 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.183 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:49.183 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.020668+0000", 2023-01-15T14:35:49.183 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19897753800000001 2023-01-15T14:35:49.183 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.223918+0000", 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20324972499999999 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224796+0000", 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000878655 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.184 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224907+0000", 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011042799999999999 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225299+0000", 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000392431 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.185 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.186 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4228 3.78 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (58) in=7b,truncate 3731253] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:49.186 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:49.186 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.881527576, 2023-01-15T14:35:49.186 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73116634000000003, 2023-01-15T14:35:49.186 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:49.186 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:49.187 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:49.187 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:49.187 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:49.187 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4228 2023-01-15T14:35:49.187 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.187 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:49.187 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.187 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:49.187 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:49.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:49.188 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:49.188 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.188 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.188 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.188 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:49.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494841+0000", 2023-01-15T14:35:49.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:49.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:49.189 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494846+0000", 2023-01-15T14:35:49.189 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4960000000000004e-06 2023-01-15T14:35:49.189 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.189 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.189 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494879+0000", 2023-01-15T14:35:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2376000000000003e-05 2023-01-15T14:35:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494887+0000", 2023-01-15T14:35:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9100000000000005e-06 2023-01-15T14:35:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:49.190 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.595858+0000", 2023-01-15T14:35:49.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100971879 2023-01-15T14:35:49.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:49.191 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596092+0000", 2023-01-15T14:35:49.191 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023309200000000001 2023-01-15T14:35:49.191 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.191 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.191 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:49.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820587+0000", 2023-01-15T14:35:49.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22449530500000001 2023-01-15T14:35:49.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.192 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.192 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:49.192 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820789+0000", 2023-01-15T14:35:49.192 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000201802 2023-01-15T14:35:49.192 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821144+0000", 2023-01-15T14:35:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035586199999999998 2023-01-15T14:35:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821591+0000", 2023-01-15T14:35:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000446458 2023-01-15T14:35:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.193 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821605+0000", 2023-01-15T14:35:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4287000000000001e-05 2023-01-15T14:35:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821653+0000", 2023-01-15T14:35:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7781000000000002e-05 2023-01-15T14:35:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.194 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821717+0000", 2023-01-15T14:35:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3911999999999994e-05 2023-01-15T14:35:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.121593+0000", 2023-01-15T14:35:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29987650900000001 2023-01-15T14:35:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:49.195 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224737+0000", 2023-01-15T14:35:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103143918 2023-01-15T14:35:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225390+0000", 2023-01-15T14:35:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065281900000000001 2023-01-15T14:35:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225496+0000", 2023-01-15T14:35:49.196 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106038 2023-01-15T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.226008+0000", 2023-01-15T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051175199999999998 2023-01-15T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.197 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4229 3.78 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 24.881472030000001, 2023-01-15T14:35:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82811652800000002, 2023-01-15T14:35:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:49.198 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4229 2023-01-15T14:35:49.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.199 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:49.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:49.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:49.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.199 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.199 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.199 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:49.199 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:49.199 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:49.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494896+0000", 2023-01-15T14:35:49.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:49.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.200 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:49.200 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494901+0000", 2023-01-15T14:35:49.200 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8269999999999997e-06 2023-01-15T14:35:49.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:49.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494924+0000", 2023-01-15T14:35:49.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3281999999999998e-05 2023-01-15T14:35:49.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.201 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.201 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:49.201 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494936+0000", 2023-01-15T14:35:49.201 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1457000000000001e-05 2023-01-15T14:35:49.201 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596198+0000", 2023-01-15T14:35:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10126203 2023-01-15T14:35:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596400+0000", 2023-01-15T14:35:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020234499999999999 2023-01-15T14:35:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:49.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821768+0000", 2023-01-15T14:35:49.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.225367977 2023-01-15T14:35:49.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:49.203 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.822124+0000", 2023-01-15T14:35:49.203 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035540600000000001 2023-01-15T14:35:49.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.203 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:49.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.322208+0000", 2023-01-15T14:35:49.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50008402100000005 2023-01-15T14:35:49.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:49.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.322487+0000", 2023-01-15T14:35:49.204 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027940400000000002 2023-01-15T14:35:49.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.204 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:49.204 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.323014+0000", 2023-01-15T14:35:49.205 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052676600000000002 2023-01-15T14:35:49.205 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.205 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:49.205 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.205 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.205 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:49.205 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:49.206 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:35:49.207 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:35:49.209 INFO:tasks.rados.rados.0.smithi090.stdout:1703: done (2 left) 2023-01-15T14:35:49.209 INFO:tasks.rados.rados.0.smithi090.stdout:1704: done (1 left) 2023-01-15T14:35:49.209 INFO:tasks.rados.rados.0.smithi090.stdout:1705: copy_from oid 1 from oid 41 current snap is 185 2023-01-15T14:35:49.209 INFO:tasks.rados.rados.0.smithi090.stdout:1706: read oid 18 snap -1 2023-01-15T14:35:49.209 INFO:tasks.rados.rados.0.smithi090.stdout:1706: expect (ObjNum 592 snap 180 seq_num 592) 2023-01-15T14:35:49.210 INFO:tasks.rados.rados.0.smithi090.stdout:1707: copy_from oid 5 from oid 24 current snap is 185 2023-01-15T14:35:49.210 INFO:tasks.rados.rados.0.smithi090.stdout:1708: copy_from oid 14 from oid 11 current snap is 185 2023-01-15T14:35:49.210 INFO:tasks.rados.rados.0.smithi090.stdout:1709: snap_create 2023-01-15T14:35:49.210 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:35:49.210 INFO:tasks.rados.rados.0.smithi090.stdout:1706: expect (ObjNum 592 snap 180 seq_num 592) 2023-01-15T14:35:49.210 INFO:tasks.rados.rados.0.smithi090.stdout:1702: finishing rollback tid 1 to smithi09025264-49 2023-01-15T14:35:49.210 INFO:tasks.rados.rados.0.smithi090.stdout:1707: finishing copy_from to smithi09025264-5 2023-01-15T14:35:49.210 INFO:tasks.rados.rados.0.smithi090.stdout:1707: got expected ENOENT (src dne) 2023-01-15T14:35:49.210 INFO:tasks.rados.rados.0.smithi090.stdout:1707: finishing copy_from racing read to smithi09025264-5 2023-01-15T14:35:49.210 INFO:tasks.rados.rados.0.smithi090.stdout:1706: done (4 left) 2023-01-15T14:35:49.210 INFO:tasks.rados.rados.0.smithi090.stdout:1707: done (3 left) 2023-01-15T14:35:49.211 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:35:49.211 INFO:tasks.rados.rados.0.smithi090.stdout:1702: finishing rollback tid 2 to smithi09025264-49 2023-01-15T14:35:49.211 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 49 v 371 (ObjNum 497 snap 146 seq_num 497) dirty exists 2023-01-15T14:35:49.211 INFO:tasks.rados.rados.0.smithi090.stdout:1702: done (2 left) 2023-01-15T14:35:49.211 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:35:49.211 INFO:tasks.rados.rados.0.smithi090.stdout:1708: finishing copy_from to smithi09025264-14 2023-01-15T14:35:49.211 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 349 (ObjNum 498 snap 146 seq_num 498) dirty exists 2023-01-15T14:35:49.211 INFO:tasks.rados.rados.0.smithi090.stdout:1708: finishing copy_from racing read to smithi09025264-14 2023-01-15T14:35:49.211 INFO:tasks.rados.rados.0.smithi090.stdout:1708: done (1 left) 2023-01-15T14:35:49.211 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:49.211 INFO:tasks.rados.rados.0.smithi090.stdout:1705: finishing copy_from to smithi09025264-1 2023-01-15T14:35:49.212 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 1 v 375 (ObjNum 513 snap 152 seq_num 513) dirty exists 2023-01-15T14:35:49.212 INFO:tasks.rados.rados.0.smithi090.stdout:1705: finishing copy_from racing read to smithi09025264-1 2023-01-15T14:35:49.212 INFO:tasks.rados.rados.0.smithi090.stdout:1705: done (0 left) 2023-01-15T14:35:49.212 INFO:tasks.rados.rados.0.smithi090.stdout:1709: done (0 left) 2023-01-15T14:35:49.212 INFO:tasks.rados.rados.0.smithi090.stdout:1710: rollback oid 26 current snap is 186 2023-01-15T14:35:49.212 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 26 to 179 2023-01-15T14:35:49.212 INFO:tasks.rados.rados.0.smithi090.stdout:1711: delete oid 2 current snap is 186 2023-01-15T14:35:49.212 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:49.197+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:49.304 INFO:tasks.rados.rados.0.smithi090.stdout:1710: finishing rollback tid 1 to smithi09025264-26 2023-01-15T14:35:49.304 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 340 (ObjNum 273 snap 0 seq_num 0) dirty dne 2023-01-15T14:35:49.307 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:49.301+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:49.365 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:49.365 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:49.365 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.365 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_backoff(3.85 ack-block id 20 [3:a1c47f99:::smithi09025264-2:head,3:a1c47f99:::smithi09025264-2:head) e459)", 2023-01-15T14:35:49.365 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:49.290123+0000", 2023-01-15T14:35:49.365 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.071842733000000006, 2023-01-15T14:35:49.365 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.071869825999999998, 2023-01-15T14:35:49.365 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:49.366 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:35:49.366 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:49.366 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:49.366 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:49.366 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 0 2023-01-15T14:35:49.366 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.366 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:49.366 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.366 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:49.366 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.290123+0000", 2023-01-15T14:35:49.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:49.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.290123+0000", 2023-01-15T14:35:49.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.367 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.367 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.367 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:49.367 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.290121+0000", 2023-01-15T14:35:49.367 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.290134+0000", 2023-01-15T14:35:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3458999999999999e-05 2023-01-15T14:35:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.290154+0000", 2023-01-15T14:35:49.368 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0134999999999999e-05 2023-01-15T14:35:49.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.369 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.369 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:49.369 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.290168+0000", 2023-01-15T14:35:49.369 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.3725e-05 2023-01-15T14:35:49.369 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.369 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:49.369 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.369 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.370 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(3.85 459/456 [PushReplyOp(3:a1c47f99:::smithi09025264-2:head)])", 2023-01-15T14:35:49.370 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:49.298498+0000", 2023-01-15T14:35:49.370 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.063467709999999997, 2023-01-15T14:35:49.370 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063544870000000003, 2023-01-15T14:35:49.370 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:49.370 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:35:49.370 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:49.370 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.370 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:49.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298498+0000", 2023-01-15T14:35:49.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:49.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298498+0000", 2023-01-15T14:35:49.371 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.371 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.371 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.371 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:49.371 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298497+0000", 2023-01-15T14:35:49.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:49.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:49.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298508+0000", 2023-01-15T14:35:49.372 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1080000000000001e-05 2023-01-15T14:35:49.372 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.372 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.372 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:49.372 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298536+0000", 2023-01-15T14:35:49.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8090999999999999e-05 2023-01-15T14:35:49.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.373 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:49.373 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298552+0000", 2023-01-15T14:35:49.373 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5869999999999999e-05 2023-01-15T14:35:49.373 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.373 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:49.373 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.373 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.373 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.374 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "MOSDPGPushReply(3.85 459/456 [PushReplyOp(3:a1c47f99:::smithi09025264-2:head)])", 2023-01-15T14:35:49.374 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:49.298550+0000", 2023-01-15T14:35:49.374 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.063415965000000005, 2023-01-15T14:35:49.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.063528625000000005, 2023-01-15T14:35:49.374 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:49.374 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "queued for pg", 2023-01-15T14:35:49.374 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:49.374 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.374 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:49.374 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298550+0000", 2023-01-15T14:35:49.374 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298550+0000", 2023-01-15T14:35:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298549+0000", 2023-01-15T14:35:49.375 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298558+0000", 2023-01-15T14:35:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.0569999999999999e-06 2023-01-15T14:35:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298582+0000", 2023-01-15T14:35:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4841000000000002e-05 2023-01-15T14:35:49.376 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:49.377 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:49.377 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:49.377 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:49.298594+0000", 2023-01-15T14:35:49.377 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1809e-05 2023-01-15T14:35:49.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.377 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:49.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.377 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:49.377 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:35:49.377 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 3 2023-01-15T14:35:49.378 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:49.379 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:35:49.396 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:49.409 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:49.404+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:49.413 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:49.431 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:49.452 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:49.473 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:49.487 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:49.488 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:49.501 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:35:49.510 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:49.504+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:49.601 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:49.601 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:49.611 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:35:49.612 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:49.604+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:49.700 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:49.712 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:49.705+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:49.725 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:49.726 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:49.726 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:49.736 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:35:49.747 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:35:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:35:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:35:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:35:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:35:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:35:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:35:49.750 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:35:49.790 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:35:49.812 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:49.804+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:49.839 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:49.839 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:49.849 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:35:49.913 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:49.908+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:49.952 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:49.952 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:49.962 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:35:50.014 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:50.005+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:50.023 INFO:tasks.rados.rados.0.smithi090.stdout:1710: done (1 left) 2023-01-15T14:35:50.023 INFO:tasks.rados.rados.0.smithi090.stdout:1711: done (0 left) 2023-01-15T14:35:50.023 INFO:tasks.rados.rados.0.smithi090.stdout:1712: snap_create 2023-01-15T14:35:50.063 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:50.063 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:50.073 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:35:50.107 INFO:tasks.rados.rados.0.smithi090.stdout:1712: done (0 left) 2023-01-15T14:35:50.108 INFO:tasks.rados.rados.0.smithi090.stdout:1713: write (excl) oid 36 current snap is 187 2023-01-15T14:35:50.108 INFO:tasks.rados.rados.0.smithi090.stdout:1713: seq_num 608 ranges {618656=543279,1812346=406310} 2023-01-15T14:35:50.112 INFO:tasks.rados.rados.0.smithi090.stdout:1713: writing smithi09025264-36 from 618656 to 1161935 tid 1 2023-01-15T14:35:50.114 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:50.105+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:50.119 INFO:tasks.rados.rados.0.smithi090.stdout:1713: writing smithi09025264-36 from 1812346 to 2218656 tid 2 2023-01-15T14:35:50.119 INFO:tasks.rados.rados.0.smithi090.stdout:1714: rollback oid 5 current snap is 187 2023-01-15T14:35:50.119 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 5 to 164 2023-01-15T14:35:50.119 INFO:tasks.rados.rados.0.smithi090.stdout:1715: write oid 16 current snap is 187 2023-01-15T14:35:50.119 INFO:tasks.rados.rados.0.smithi090.stdout:1715: seq_num 609 ranges {635463=734881,1987181=248282} 2023-01-15T14:35:50.127 INFO:tasks.rados.rados.0.smithi090.stdout:1715: writing smithi09025264-16 from 635463 to 1370344 tid 1 2023-01-15T14:35:50.132 INFO:tasks.rados.rados.0.smithi090.stdout:1715: writing smithi09025264-16 from 1987181 to 2235463 tid 2 2023-01-15T14:35:50.132 INFO:tasks.rados.rados.0.smithi090.stdout:1716: snap_remove snap 175 2023-01-15T14:35:50.182 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:50.182 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:50.191 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:35:50.216 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:50.209+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:50.291 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:50.291 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:50.302 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:35:50.353 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:50.348+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:50.407 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:50.407 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:50.418 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:50.409+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:50.418 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:35:50.519 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:50.509+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:50.523 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:50.523 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:50.532 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:35:50.620 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:50.612+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:50.633 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:50.633 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:50.643 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:35:50.721 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:50.713+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:50.796 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:50.796 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:50.806 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:35:50.821 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:50.813+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:50.907 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:50.907 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:50.916 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:35:50.921 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:50.916+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:51.023 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:51.013+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:51.024 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:51.024 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:51.035 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:35:51.125 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:51.116+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:51.135 INFO:tasks.rados.rados.0.smithi090.stdout:1716: done (3 left) 2023-01-15T14:35:51.135 INFO:tasks.rados.rados.0.smithi090.stdout:1717: snap_remove snap 165 2023-01-15T14:35:51.139 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:35:51.152 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:51.152 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:51.162 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:35:51.225 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:51.220+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:51.264 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:51.274 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:35:51.326 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:51.320+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:51.385 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:51.385 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:51.387 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:35:51.427 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:51.420+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:51.494 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:51.590 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:51.592 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:35:51.592 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:51.585+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:51.692 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:51.685+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:51.801 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:35:51.802 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:51.803 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:51.803 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:35:51.803 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:51.793+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:51.904 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:51.897+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:51.957 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:51.958 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:51.967 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:35:52.007 INFO:tasks.rados.rados.0.smithi090.stdout:1717: done (3 left) 2023-01-15T14:35:52.007 INFO:tasks.rados.rados.0.smithi090.stdout:1718: write (excl) oid 2 current snap is 187 2023-01-15T14:35:52.007 INFO:tasks.rados.rados.0.smithi090.stdout:1718: seq_num 610 ranges {652270=410130,1529310=493867} 2023-01-15T14:35:52.007 INFO:tasks.rados.rados.0.smithi090.stdout:1718: writing smithi09025264-2 from 652270 to 1062400 tid 1 2023-01-15T14:35:52.007 INFO:tasks.rados.rados.0.smithi090.stdout:1718: writing smithi09025264-2 from 1529310 to 2023177 tid 2 2023-01-15T14:35:52.007 INFO:tasks.rados.rados.0.smithi090.stdout:1719: read oid 4 snap -1 2023-01-15T14:35:52.007 INFO:tasks.rados.rados.0.smithi090.stdout:1719: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:35:52.007 INFO:tasks.rados.rados.0.smithi090.stdout:1720: snap_create 2023-01-15T14:35:52.007 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 5 2023-01-15T14:35:52.008 INFO:tasks.rados.rados.0.smithi090.stdout:1713: finishing write tid 1 to smithi09025264-36 2023-01-15T14:35:52.008 INFO:tasks.rados.rados.0.smithi090.stdout:1713: finishing write tid 2 to smithi09025264-36 2023-01-15T14:35:52.008 INFO:tasks.rados.rados.0.smithi090.stdout:1714: finishing rollback tid 1 to smithi09025264-5 2023-01-15T14:35:52.008 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 5 v 380 (ObjNum 97 snap 0 seq_num 3199167552) dirty dne 2023-01-15T14:35:52.008 INFO:tasks.rados.rados.0.smithi090.stdout:1713: finishing write tid 3 to smithi09025264-36 2023-01-15T14:35:52.008 INFO:tasks.rados.rados.0.smithi090.stdout:1713: finishing write tid 4 to smithi09025264-36 2023-01-15T14:35:52.008 INFO:tasks.rados.rados.0.smithi090.stdout:1713: oid 36 updating version 0 to 285 2023-01-15T14:35:52.008 INFO:tasks.rados.rados.0.smithi090.stdout:1713: oid 36 updating version 285 to 286 2023-01-15T14:35:52.008 INFO:tasks.rados.rados.0.smithi090.stdout:1713: oid 36 version 286 is already newer than 284 2023-01-15T14:35:52.008 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 36 v 286 (ObjNum 608 snap 187 seq_num 608) dirty exists 2023-01-15T14:35:52.009 INFO:tasks.rados.rados.0.smithi090.stdout:1713: left oid 36 (ObjNum 608 snap 187 seq_num 608) 2023-01-15T14:35:52.009 INFO:tasks.rados.rados.0.smithi090.stdout:1715: finishing write tid 1 to smithi09025264-16 2023-01-15T14:35:52.009 INFO:tasks.rados.rados.0.smithi090.stdout:1715: finishing write tid 2 to smithi09025264-16 2023-01-15T14:35:52.009 INFO:tasks.rados.rados.0.smithi090.stdout:1715: finishing write tid 3 to smithi09025264-16 2023-01-15T14:35:52.009 INFO:tasks.rados.rados.0.smithi090.stdout:1715: finishing write tid 4 to smithi09025264-16 2023-01-15T14:35:52.009 INFO:tasks.rados.rados.0.smithi090.stdout:1715: oid 16 updating version 0 to 352 2023-01-15T14:35:52.009 INFO:tasks.rados.rados.0.smithi090.stdout:1715: oid 16 version 352 is already newer than 350 2023-01-15T14:35:52.009 INFO:tasks.rados.rados.0.smithi090.stdout:1715: oid 16 version 352 is already newer than 351 2023-01-15T14:35:52.009 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 16 v 352 (ObjNum 609 snap 187 seq_num 609) dirty exists 2023-01-15T14:35:52.009 INFO:tasks.rados.rados.0.smithi090.stdout:1715: left oid 16 (ObjNum 609 snap 187 seq_num 609) 2023-01-15T14:35:52.009 INFO:tasks.rados.rados.0.smithi090.stdout:1713: done (4 left) 2023-01-15T14:35:52.010 INFO:tasks.rados.rados.0.smithi090.stdout:1714: done (3 left) 2023-01-15T14:35:52.010 INFO:tasks.rados.rados.0.smithi090.stdout:1715: done (2 left) 2023-01-15T14:35:52.010 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:35:52.010 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:52.000+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:52.107 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:52.097+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:52.204 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:52.204 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:52.205 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:35:52.205 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:52.208 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:52.200+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:52.218 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:52.236 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:52.237 INFO:tasks.rados.rados.0.smithi090.stdout:1719: expect (ObjNum 143 snap 27 seq_num 143) 2023-01-15T14:35:52.255 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:52.308 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:35:52.300+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:35:52.421 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:52.422 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "complaint_time": 30, 2023-01-15T14:35:52.422 INFO:teuthology.orchestra.run.smithi090.stdout: "num_blocked_ops": 0 2023-01-15T14:35:52.422 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:52.423 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:35:52.423 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:52.423 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:52.424 INFO:tasks.rados.rados.0.smithi090.stdout:1718: finishing write tid 1 to smithi09025264-2 2023-01-15T14:35:52.424 INFO:tasks.rados.rados.0.smithi090.stdout:1719: done (1 left) 2023-01-15T14:35:52.424 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:35:52.424 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:52.425 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:35:52.425 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:35:52.425 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:35:52.425 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:35:52.425 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:35:52.425 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:35:52.426 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:35:52.426 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:35:52.426 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:52.416+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:52.496 INFO:tasks.rados.rados.0.smithi090.stdout:1718: finishing write tid 2 to smithi09025264-2 2023-01-15T14:35:52.615 INFO:tasks.rados.rados.0.smithi090.stdout:1718: finishing write tid 3 to smithi09025264-2 2023-01-15T14:35:52.615 INFO:tasks.rados.rados.0.smithi090.stdout:1718: finishing write tid 4 to smithi09025264-2 2023-01-15T14:35:52.615 INFO:tasks.rados.rados.0.smithi090.stdout:1718: oid 2 updating version 0 to 398 2023-01-15T14:35:52.615 INFO:tasks.rados.rados.0.smithi090.stdout:1718: oid 2 version 398 is already newer than 397 2023-01-15T14:35:52.615 INFO:tasks.rados.rados.0.smithi090.stdout:1718: oid 2 updating version 398 to 399 2023-01-15T14:35:52.615 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 2 v 399 (ObjNum 610 snap 187 seq_num 610) dirty exists 2023-01-15T14:35:52.615 INFO:tasks.rados.rados.0.smithi090.stdout:1718: left oid 2 (ObjNum 610 snap 187 seq_num 610) 2023-01-15T14:35:52.616 INFO:tasks.rados.rados.0.smithi090.stdout:1718: done (0 left) 2023-01-15T14:35:52.617 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 20, 2023-01-15T14:35:52.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 600, 2023-01-15T14:35:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:35:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1651 3.4 3:2029fa71:::smithi09025264-39:head [setxattr _header (57) in=7b,truncate 2134522] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 317.68029046100003, 2023-01-15T14:35:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41077204299999998, 2023-01-15T14:35:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:52.618 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1651 2023-01-15T14:35:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:52.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894460+0000", 2023-01-15T14:35:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894459+0000", 2023-01-15T14:35:52.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894469+0000", 2023-01-15T14:35:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0448999999999999e-05 2023-01-15T14:35:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894484+0000", 2023-01-15T14:35:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5296999999999999e-05 2023-01-15T14:35:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894495+0000", 2023-01-15T14:35:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.116e-05 2023-01-15T14:35:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895081+0000", 2023-01-15T14:35:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058554600000000005 2023-01-15T14:35:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.895089+0000", 2023-01-15T14:35:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.1300000000000001e-06 2023-01-15T14:35:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.100781+0000", 2023-01-15T14:35:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.205692233 2023-01-15T14:35:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101174+0000", 2023-01-15T14:35:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039264600000000002 2023-01-15T14:35:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:35:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.101936+0000", 2023-01-15T14:35:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00076228799999999996 2023-01-15T14:35:52.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103321+0000", 2023-01-15T14:35:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00138418 2023-01-15T14:35:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103342+0000", 2023-01-15T14:35:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.1665999999999999e-05 2023-01-15T14:35:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103392+0000", 2023-01-15T14:35:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0120999999999997e-05 2023-01-15T14:35:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103935+0000", 2023-01-15T14:35:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00054310999999999997 2023-01-15T14:35:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304903+0000", 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.200968024 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.304971+0000", 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.7204000000000002e-05 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305047+0000", 2023-01-15T14:35:52.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.6329999999999996e-05 2023-01-15T14:35:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305232+0000", 2023-01-15T14:35:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018512499999999999 2023-01-15T14:35:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:52.630 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1652 3.4 3:2029fa71:::smithi09025264-39:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e148)", 2023-01-15T14:35:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 317.68023951800001, 2023-01-15T14:35:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41103427599999998, 2023-01-15T14:35:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:52.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1652 2023-01-15T14:35:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894511+0000", 2023-01-15T14:35:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894510+0000", 2023-01-15T14:35:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894517+0000", 2023-01-15T14:35:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.72e-06 2023-01-15T14:35:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894537+0000", 2023-01-15T14:35:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0273000000000001e-05 2023-01-15T14:35:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.634 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.894549+0000", 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1863e-05 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896737+0000", 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0021881330000000001 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:34.896749+0000", 2023-01-15T14:35:52.635 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2121e-05 2023-01-15T14:35:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103492+0000", 2023-01-15T14:35:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.206743185 2023-01-15T14:35:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:52.636 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.103872+0000", 2023-01-15T14:35:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037938499999999999 2023-01-15T14:35:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305298+0000", 2023-01-15T14:35:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20142636999999999 2023-01-15T14:35:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305395+0000", 2023-01-15T14:35:52.637 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.7298000000000003e-05 2023-01-15T14:35:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:30:35.305545+0000", 2023-01-15T14:35:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015008099999999999 2023-01-15T14:35:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.638 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:1819 3.1f 3:fd983206:::smithi09025264-7:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2023-01-15T14:35:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 288.771610684, 2023-01-15T14:35:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.40623445499999999, 2023-01-15T14:35:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 1819 2023-01-15T14:35:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.639 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803140+0000", 2023-01-15T14:35:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.640 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803138+0000", 2023-01-15T14:35:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803145+0000", 2023-01-15T14:35:52.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.4899999999999997e-06 2023-01-15T14:35:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.803156+0000", 2023-01-15T14:35:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0597e-05 2023-01-15T14:35:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:52.817 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903246+0000", 2023-01-15T14:35:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10009052 2023-01-15T14:35:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903380+0000", 2023-01-15T14:35:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00013379700000000001 2023-01-15T14:35:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for peered", 2023-01-15T14:35:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:03.903404+0000", 2023-01-15T14:35:52.818 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4491e-05 2023-01-15T14:35:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107342+0000", 2023-01-15T14:35:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.203938007 2023-01-15T14:35:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.107765+0000", 2023-01-15T14:35:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042294300000000001 2023-01-15T14:35:52.819 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:04.209374+0000", 2023-01-15T14:35:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.101608963 2023-01-15T14:35:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.820 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2049 3.1f 3:f8edd34e:::smithi09025264-19:head [setxattr _header (57) in=7b,truncate 3159749] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 266.88458127500002, 2023-01-15T14:35:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.4074391, 2023-01-15T14:35:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2049 2023-01-15T14:35:52.821 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690169+0000", 2023-01-15T14:35:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.822 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690168+0000", 2023-01-15T14:35:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690177+0000", 2023-01-15T14:35:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.6999999999999997e-06 2023-01-15T14:35:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.823 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690192+0000", 2023-01-15T14:35:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5102999999999999e-05 2023-01-15T14:35:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690212+0000", 2023-01-15T14:35:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0370999999999999e-05 2023-01-15T14:35:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.824 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690640+0000", 2023-01-15T14:35:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000427339 2023-01-15T14:35:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690851+0000", 2023-01-15T14:35:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021090599999999999 2023-01-15T14:35:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.825 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790326+0000", 2023-01-15T14:35:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099475615000000003 2023-01-15T14:35:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790506+0000", 2023-01-15T14:35:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000179727 2023-01-15T14:35:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:52.826 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.790859+0000", 2023-01-15T14:35:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035298399999999998 2023-01-15T14:35:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791166+0000", 2023-01-15T14:35:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030756000000000002 2023-01-15T14:35:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791186+0000", 2023-01-15T14:35:52.827 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9894e-05 2023-01-15T14:35:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.791247+0000", 2023-01-15T14:35:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.0902000000000001e-05 2023-01-15T14:35:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.891882+0000", 2023-01-15T14:35:52.828 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063493799999999 2023-01-15T14:35:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.096533+0000", 2023-01-15T14:35:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.204651218 2023-01-15T14:35:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097358+0000", 2023-01-15T14:35:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00082478599999999999 2023-01-15T14:35:52.829 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097470+0000", 2023-01-15T14:35:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000111935 2023-01-15T14:35:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.097608+0000", 2023-01-15T14:35:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000138274 2023-01-15T14:35:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:52.830 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.831 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2050 3.1f 3:f8edd34e:::smithi09025264-19:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2023-01-15T14:35:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 266.88447420099999, 2023-01-15T14:35:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50811537100000004, 2023-01-15T14:35:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:52.832 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2050 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.833 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690276+0000", 2023-01-15T14:35:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690275+0000", 2023-01-15T14:35:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.834 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690285+0000", 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.064e-05 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690309+0000", 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3730000000000001e-05 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690323+0000", 2023-01-15T14:35:52.835 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4209999999999999e-05 2023-01-15T14:35:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.690931+0000", 2023-01-15T14:35:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00060764100000000002 2023-01-15T14:35:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.691164+0000", 2023-01-15T14:35:52.836 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000233279 2023-01-15T14:35:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895154+0000", 2023-01-15T14:35:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20398972900000001 2023-01-15T14:35:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:25.895476+0000", 2023-01-15T14:35:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032187399999999999 2023-01-15T14:35:52.837 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.197912+0000", 2023-01-15T14:35:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.30243634400000002 2023-01-15T14:35:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198127+0000", 2023-01-15T14:35:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021495299999999999 2023-01-15T14:35:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.838 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:26.198392+0000", 2023-01-15T14:35:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00026461099999999999 2023-01-15T14:35:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:2154 3.1f 3:f8edd34e:::smithi09025264-19:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e207)", 2023-01-15T14:35:52.839 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 258.75406068299998, 2023-01-15T14:35:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.41384504900000002, 2023-01-15T14:35:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 2154 2023-01-15T14:35:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.840 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820690+0000", 2023-01-15T14:35:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.841 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820689+0000", 2023-01-15T14:35:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820694+0000", 2023-01-15T14:35:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4129999999999997e-06 2023-01-15T14:35:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.842 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820705+0000", 2023-01-15T14:35:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0536999999999999e-05 2023-01-15T14:35:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820712+0000", 2023-01-15T14:35:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.2869999999999999e-06 2023-01-15T14:35:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.843 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820873+0000", 2023-01-15T14:35:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000160674 2023-01-15T14:35:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:33.820877+0000", 2023-01-15T14:35:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6600000000000001e-06 2023-01-15T14:35:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.844 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:52.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131280+0000", 2023-01-15T14:35:52.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.31040351900000002 2023-01-15T14:35:52.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:52.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131470+0000", 2023-01-15T14:35:52.845 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018936300000000001 2023-01-15T14:35:52.845 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.845 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.845 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:52.845 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.131772+0000", 2023-01-15T14:35:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030212000000000001 2023-01-15T14:35:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132142+0000", 2023-01-15T14:35:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037018899999999999 2023-01-15T14:35:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132169+0000", 2023-01-15T14:35:52.846 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.6767e-05 2023-01-15T14:35:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132263+0000", 2023-01-15T14:35:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.4288000000000003e-05 2023-01-15T14:35:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.132358+0000", 2023-01-15T14:35:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.5113999999999998e-05 2023-01-15T14:35:52.847 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.134897+0000", 2023-01-15T14:35:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0025389620000000001 2023-01-15T14:35:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:52.848 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234351+0000", 2023-01-15T14:35:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.099453913000000005 2023-01-15T14:35:53.024 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234452+0000", 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000101308 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:31:34.234535+0000", 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2707999999999995e-05 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:53.025 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2869 3.40 e274/234 3:03b54904:::smithi09025264-12:head v 274'279, mlcod=264'275)", 2023-01-15T14:35:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 191.580949191, 2023-01-15T14:35:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.497521993, 2023-01-15T14:35:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.026 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.027 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993801+0000", 2023-01-15T14:35:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993807+0000", 2023-01-15T14:35:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3539999999999996e-06 2023-01-15T14:35:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.028 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993816+0000", 2023-01-15T14:35:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1239999999999994e-06 2023-01-15T14:35:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.993823+0000", 2023-01-15T14:35:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.5010000000000001e-06 2023-01-15T14:35:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.029 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490505+0000", 2023-01-15T14:35:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.496681873 2023-01-15T14:35:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490542+0000", 2023-01-15T14:35:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7682000000000003e-05 2023-01-15T14:35:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.030 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490770+0000", 2023-01-15T14:35:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000227579 2023-01-15T14:35:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490808+0000", 2023-01-15T14:35:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7625e-05 2023-01-15T14:35:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.031 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490854+0000", 2023-01-15T14:35:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.6124000000000001e-05 2023-01-15T14:35:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490927+0000", 2023-01-15T14:35:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.3405000000000006e-05 2023-01-15T14:35:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.032 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491323+0000", 2023-01-15T14:35:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039612799999999999 2023-01-15T14:35:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:2870 3.18 e274/224 3:18f3951e:::smithi09025264-35:head v 274'296, mlcod=269'293)", 2023-01-15T14:35:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 191.57713750400001, 2023-01-15T14:35:53.033 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50157772499999997, 2023-01-15T14:35:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:53.034 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997613+0000", 2023-01-15T14:35:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.997611+0000", 2023-01-15T14:35:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:53.035 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999082+0000", 2023-01-15T14:35:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0014705059999999999 2023-01-15T14:35:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999103+0000", 2023-01-15T14:35:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0588e-05 2023-01-15T14:35:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:53.036 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:40.999112+0000", 2023-01-15T14:35:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1279999999999997e-06 2023-01-15T14:35:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.490988+0000", 2023-01-15T14:35:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.49187608399999999 2023-01-15T14:35:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:53.037 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.491042+0000", 2023-01-15T14:35:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.3786000000000003e-05 2023-01-15T14:35:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493340+0000", 2023-01-15T14:35:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0022985729999999999 2023-01-15T14:35:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.493365+0000", 2023-01-15T14:35:53.038 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4848999999999999e-05 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494223+0000", 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00085791199999999995 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.494240+0000", 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.7221999999999999e-05 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:32:41.499191+0000", 2023-01-15T14:35:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0049506539999999996 2023-01-15T14:35:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3120 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'290, mlcod=0'0)", 2023-01-15T14:35:53.040 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 163.33532422799999, 2023-01-15T14:35:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39853714000000001, 2023-01-15T14:35:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239426+0000", 2023-01-15T14:35:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239424+0000", 2023-01-15T14:35:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239747+0000", 2023-01-15T14:35:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00032243300000000001 2023-01-15T14:35:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239784+0000", 2023-01-15T14:35:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.7051000000000002e-05 2023-01-15T14:35:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.043 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.239802+0000", 2023-01-15T14:35:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8250999999999999e-05 2023-01-15T14:35:53.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536033+0000", 2023-01-15T14:35:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29623121499999999 2023-01-15T14:35:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.230 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536081+0000", 2023-01-15T14:35:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7633000000000003e-05 2023-01-15T14:35:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536800+0000", 2023-01-15T14:35:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00071948499999999996 2023-01-15T14:35:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.231 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536817+0000", 2023-01-15T14:35:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6228000000000001e-05 2023-01-15T14:35:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537048+0000", 2023-01-15T14:35:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023136200000000001 2023-01-15T14:35:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.232 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637902+0000", 2023-01-15T14:35:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10085403900000001 2023-01-15T14:35:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637963+0000", 2023-01-15T14:35:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.1310999999999996e-05 2023-01-15T14:35:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.233 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3121 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'291, mlcod=0'0)", 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 163.333328712, 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39659573799999998, 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:53.234 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241422+0000", 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241420+0000", 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.235 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241854+0000", 2023-01-15T14:35:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00043330099999999998 2023-01-15T14:35:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241878+0000", 2023-01-15T14:35:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.4559e-05 2023-01-15T14:35:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.236 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.241892+0000", 2023-01-15T14:35:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4039999999999999e-05 2023-01-15T14:35:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536863+0000", 2023-01-15T14:35:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29497073400000001 2023-01-15T14:35:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.237 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.536887+0000", 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3683999999999999e-05 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537124+0000", 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023729900000000001 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537136+0000", 2023-01-15T14:35:53.238 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2278e-05 2023-01-15T14:35:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.537345+0000", 2023-01-15T14:35:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020865200000000001 2023-01-15T14:35:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637981+0000", 2023-01-15T14:35:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10063575700000001 2023-01-15T14:35:53.239 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638018+0000", 2023-01-15T14:35:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.6826999999999998e-05 2023-01-15T14:35:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3122 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'292, mlcod=0'0)", 2023-01-15T14:35:53.240 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 163.33143287199999, 2023-01-15T14:35:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39623551600000001, 2023-01-15T14:35:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:53.241 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.242 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243318+0000", 2023-01-15T14:35:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243316+0000", 2023-01-15T14:35:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.243 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243833+0000", 2023-01-15T14:35:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051703600000000001 2023-01-15T14:35:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243863+0000", 2023-01-15T14:35:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.9379000000000001e-05 2023-01-15T14:35:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.244 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.243879+0000", 2023-01-15T14:35:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.6263999999999999e-05 2023-01-15T14:35:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637317+0000", 2023-01-15T14:35:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39343783199999999 2023-01-15T14:35:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:53.245 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637356+0000", 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.9579000000000002e-05 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637830+0000", 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047326399999999999 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.637885+0000", 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.5074999999999997e-05 2023-01-15T14:35:53.246 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638185+0000", 2023-01-15T14:35:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00029993799999999997 2023-01-15T14:35:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638337+0000", 2023-01-15T14:35:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00015208299999999999 2023-01-15T14:35:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.247 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.639553+0000", 2023-01-15T14:35:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001216389 2023-01-15T14:35:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(client.4530.0:3123 3.40 e300/299 3:03b54904:::smithi09025264-12:head v 300'293, mlcod=0'0)", 2023-01-15T14:35:53.248 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 163.329688844, 2023-01-15T14:35:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.395733102, 2023-01-15T14:35:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245062+0000", 2023-01-15T14:35:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245060+0000", 2023-01-15T14:35:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245075+0000", 2023-01-15T14:35:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4459e-05 2023-01-15T14:35:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245107+0000", 2023-01-15T14:35:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1795000000000003e-05 2023-01-15T14:35:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:53.251 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.245121+0000", 2023-01-15T14:35:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4117000000000001e-05 2023-01-15T14:35:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638063+0000", 2023-01-15T14:35:53.252 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39294204300000002 2023-01-15T14:35:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638097+0000", 2023-01-15T14:35:53.434 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.4385999999999999e-05 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638299+0000", 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020133099999999999 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638318+0000", 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9514999999999999e-05 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.435 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638386+0000", 2023-01-15T14:35:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.8089000000000005e-05 2023-01-15T14:35:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.638430+0000", 2023-01-15T14:35:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.439e-05 2023-01-15T14:35:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.436 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:33:09.640795+0000", 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.002364274 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4216 3.78 3:1eaad226:::smithi09025264-49:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.610994096999999, 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82042120600000001, 2023-01-15T14:35:53.437 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:53.438 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4216 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963756+0000", 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:53.439 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:53.440 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963753+0000", 2023-01-15T14:35:53.641 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999962 2023-01-15T14:35:53.641 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963772+0000", 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.9293999999999999e-05 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963806+0000", 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.3946999999999999e-05 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.642 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963822+0000", 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5744e-05 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963930+0000", 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00010870300000000001 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op must wait for map", 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963946+0000", 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5919000000000001e-05 2023-01-15T14:35:53.643 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:53.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.780699+0000", 2023-01-15T14:35:53.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.81675266400000002 2023-01-15T14:35:53.644 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.644 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.644 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:53.644 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784178+0000", 2023-01-15T14:35:53.644 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0034785699999999998 2023-01-15T14:35:53.644 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.644 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:53.645 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:53.645 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:53.645 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:53.645 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4217 3.78 3:1eaad226:::smithi09025264-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e432)", 2023-01-15T14:35:53.645 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:53.645 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 30.610898542000001, 2023-01-15T14:35:54.050 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.820584859, 2023-01-15T14:35:54.050 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:54.050 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "reached pg", 2023-01-15T14:35:54.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:54.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:54.050 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:54.050 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4217 2023-01-15T14:35:54.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.050 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963852+0000", 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.051 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963851+0000", 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963857+0000", 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8699999999999997e-06 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963867+0000", 2023-01-15T14:35:54.052 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0257e-05 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:21.963876+0000", 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.1120000000000002e-06 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064133+0000", 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100256427 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.053 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:54.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.064164+0000", 2023-01-15T14:35:54.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.1649000000000001e-05 2023-01-15T14:35:54.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.054 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.054 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784340+0000", 2023-01-15T14:35:54.054 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.72017558699999995 2023-01-15T14:35:54.456 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.456 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.784437+0000", 2023-01-15T14:35:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.6831999999999997e-05 2023-01-15T14:35:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.457 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4216 3.78 3:1eaad226:::smithi09025264-49:head [copy-from ver 0] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.577888649999998, 2023-01-15T14:35:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.39069526799999998, 2023-01-15T14:35:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:54.458 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4216 2023-01-15T14:35:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:54.459 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996862+0000", 2023-01-15T14:35:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.460 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996860+0000", 2023-01-15T14:35:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.461 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996869+0000", 2023-01-15T14:35:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.2069999999999995e-06 2023-01-15T14:35:54.462 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:22.996881+0000", 2023-01-15T14:35:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.2611e-05 2023-01-15T14:35:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:54.463 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.096971+0000", 2023-01-15T14:35:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100089364 2023-01-15T14:35:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.284700+0000", 2023-01-15T14:35:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.18772960999999999 2023-01-15T14:35:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:54.464 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.284814+0000", 2023-01-15T14:35:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000113427 2023-01-15T14:35:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.387557+0000", 2023-01-15T14:35:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.102743529 2023-01-15T14:35:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.465 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4225 3.78 3:1e54a7f5:::smithi09025264-42:head [write 1381780~411527] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.081327668, 2023-01-15T14:35:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.42766908999999997, 2023-01-15T14:35:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4225 2023-01-15T14:35:54.466 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493423+0000", 2023-01-15T14:35:54.467 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493422+0000", 2023-01-15T14:35:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493846+0000", 2023-01-15T14:35:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00042452100000000002 2023-01-15T14:35:54.468 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493861+0000", 2023-01-15T14:35:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5136e-05 2023-01-15T14:35:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493872+0000", 2023-01-15T14:35:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.0958000000000001e-05 2023-01-15T14:35:54.469 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494867+0000", 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00099453600000000007 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495083+0000", 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021589499999999999 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.470 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.815757+0000", 2023-01-15T14:35:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32067462499999999 2023-01-15T14:35:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.815988+0000", 2023-01-15T14:35:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000230762 2023-01-15T14:35:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:54.471 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.816381+0000", 2023-01-15T14:35:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039289799999999998 2023-01-15T14:35:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817159+0000", 2023-01-15T14:35:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00077780299999999998 2023-01-15T14:35:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:54.472 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817179+0000", 2023-01-15T14:35:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.0489e-05 2023-01-15T14:35:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817418+0000", 2023-01-15T14:35:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023831500000000001 2023-01-15T14:35:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:54.473 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819234+0000", 2023-01-15T14:35:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001816553 2023-01-15T14:35:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820529+0000", 2023-01-15T14:35:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0012949719999999999 2023-01-15T14:35:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920621+0000", 2023-01-15T14:35:54.474 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100091965 2023-01-15T14:35:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920679+0000", 2023-01-15T14:35:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.8319000000000002e-05 2023-01-15T14:35:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.920795+0000", 2023-01-15T14:35:54.475 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000115964 2023-01-15T14:35:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.921092+0000", 2023-01-15T14:35:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000296531 2023-01-15T14:35:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.476 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4226 3.78 3:1e54a7f5:::smithi09025264-42:head [write 2300501~586966] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.080836298000001, 2023-01-15T14:35:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.730699722, 2023-01-15T14:35:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:54.477 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4226 2023-01-15T14:35:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.478 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493914+0000", 2023-01-15T14:35:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.493913+0000", 2023-01-15T14:35:54.479 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:35:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494499+0000", 2023-01-15T14:35:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058589500000000001 2023-01-15T14:35:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494549+0000", 2023-01-15T14:35:54.480 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.0296000000000001e-05 2023-01-15T14:35:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494577+0000", 2023-01-15T14:35:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.8056999999999999e-05 2023-01-15T14:35:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495162+0000", 2023-01-15T14:35:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00058508500000000005 2023-01-15T14:35:54.481 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495377+0000", 2023-01-15T14:35:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021484400000000001 2023-01-15T14:35:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.482 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817336+0000", 2023-01-15T14:35:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32195888499999997 2023-01-15T14:35:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:54.483 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.817638+0000", 2023-01-15T14:35:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00030184599999999998 2023-01-15T14:35:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.818285+0000", 2023-01-15T14:35:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064671399999999995 2023-01-15T14:35:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.484 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819118+0000", 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00083369499999999997 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819137+0000", 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.8326999999999999e-05 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819385+0000", 2023-01-15T14:35:54.485 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00024822599999999999 2023-01-15T14:35:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.486 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.614 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:54.614 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820443+0000", 2023-01-15T14:35:54.614 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.001058219 2023-01-15T14:35:54.614 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.923538+0000", 2023-01-15T14:35:54.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103094642 2023-01-15T14:35:54.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.615 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.615 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.615 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.020572+0000", 2023-01-15T14:35:54.615 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.097034387999999999 2023-01-15T14:35:54.615 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224051+0000", 2023-01-15T14:35:54.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20347895399999999 2023-01-15T14:35:54.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.616 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:54.616 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224219+0000", 2023-01-15T14:35:54.616 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016772 2023-01-15T14:35:54.616 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.616 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.617 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:54.617 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224614+0000", 2023-01-15T14:35:54.617 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00039523500000000002 2023-01-15T14:35:54.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.617 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:54.617 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.617 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.617 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.617 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4227 3.78 3:1e54a7f5:::smithi09025264-42:head [write 3376974~354279] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:54.617 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:54.617 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.080111994999999, 2023-01-15T14:35:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73066059400000005, 2023-01-15T14:35:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4227 2023-01-15T14:35:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.618 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:54.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:54.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.619 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.619 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.619 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:54.619 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494639+0000", 2023-01-15T14:35:54.619 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.620 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.620 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.620 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:54.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494637+0000", 2023-01-15T14:35:54.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:54.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:54.621 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494805+0000", 2023-01-15T14:35:54.621 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00016839000000000001 2023-01-15T14:35:54.621 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.621 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.621 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:54.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494819+0000", 2023-01-15T14:35:54.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4287000000000001e-05 2023-01-15T14:35:54.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:54.622 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494828+0000", 2023-01-15T14:35:54.622 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 8.8119999999999997e-06 2023-01-15T14:35:54.622 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.622 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.622 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495478+0000", 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00064994399999999996 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.495668+0000", 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00018991900000000001 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819291+0000", 2023-01-15T14:35:54.623 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.32362327600000002 2023-01-15T14:35:54.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:54.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819503+0000", 2023-01-15T14:35:54.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00021184900000000001 2023-01-15T14:35:54.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.624 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.624 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:54.624 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.819880+0000", 2023-01-15T14:35:54.624 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00037692500000000001 2023-01-15T14:35:54.624 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:54.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820352+0000", 2023-01-15T14:35:54.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00047233000000000003 2023-01-15T14:35:54.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.625 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.625 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:54.625 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820368+0000", 2023-01-15T14:35:54.625 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.5877999999999999e-05 2023-01-15T14:35:54.625 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:54.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820533+0000", 2023-01-15T14:35:54.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000164508 2023-01-15T14:35:54.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.626 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.626 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:54.626 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821690+0000", 2023-01-15T14:35:54.626 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.0011576290000000001 2023-01-15T14:35:54.626 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.020668+0000", 2023-01-15T14:35:54.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.19897753800000001 2023-01-15T14:35:54.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.627 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.627 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.627 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.223918+0000", 2023-01-15T14:35:54.627 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.20324972499999999 2023-01-15T14:35:54.627 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224796+0000", 2023-01-15T14:35:54.628 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000878655 2023-01-15T14:35:54.628 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.628 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.628 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:54.628 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224907+0000", 2023-01-15T14:35:54.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00011042799999999999 2023-01-15T14:35:54.629 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.629 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.629 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:54.629 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225299+0000", 2023-01-15T14:35:54.629 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000392431 2023-01-15T14:35:54.629 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.630 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:54.630 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.630 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.630 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.630 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4228 3.78 3:1e54a7f5:::smithi09025264-42:head [setxattr _header (58) in=7b,truncate 3731253] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:54.630 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:54.630 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.079908885999998, 2023-01-15T14:35:54.630 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.73116634000000003, 2023-01-15T14:35:54.630 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:54.630 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:54.630 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4228 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.631 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:54.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494842+0000", 2023-01-15T14:35:54.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:54.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494841+0000", 2023-01-15T14:35:54.632 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:54.632 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.632 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.632 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:54.632 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494846+0000", 2023-01-15T14:35:54.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 5.4960000000000004e-06 2023-01-15T14:35:54.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:54.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494879+0000", 2023-01-15T14:35:54.633 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 3.2376000000000003e-05 2023-01-15T14:35:54.633 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.633 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:54.633 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494887+0000", 2023-01-15T14:35:54.634 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 7.9100000000000005e-06 2023-01-15T14:35:54.634 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.773 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.773 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.773 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.595858+0000", 2023-01-15T14:35:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.100971879 2023-01-15T14:35:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596092+0000", 2023-01-15T14:35:54.774 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00023309200000000001 2023-01-15T14:35:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820587+0000", 2023-01-15T14:35:54.775 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.22449530500000001 2023-01-15T14:35:54.776 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.777 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:54.777 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.820789+0000", 2023-01-15T14:35:54.777 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000201802 2023-01-15T14:35:54.777 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.777 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for subops from 0,4,7", 2023-01-15T14:35:54.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821144+0000", 2023-01-15T14:35:54.778 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035586199999999998 2023-01-15T14:35:54.778 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.778 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.778 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:54.778 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821591+0000", 2023-01-15T14:35:54.779 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000446458 2023-01-15T14:35:54.779 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.779 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.779 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:54.779 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821605+0000", 2023-01-15T14:35:54.780 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.4287000000000001e-05 2023-01-15T14:35:54.780 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.780 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.780 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:54.781 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821653+0000", 2023-01-15T14:35:54.781 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.7781000000000002e-05 2023-01-15T14:35:54.781 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.781 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.781 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "op_commit", 2023-01-15T14:35:54.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821717+0000", 2023-01-15T14:35:54.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 6.3911999999999994e-05 2023-01-15T14:35:54.782 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.782 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.782 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.782 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.121593+0000", 2023-01-15T14:35:54.782 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.29987650900000001 2023-01-15T14:35:54.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.783 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.783 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.783 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.224737+0000", 2023-01-15T14:35:54.783 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.103143918 2023-01-15T14:35:54.783 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:54.784 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225390+0000", 2023-01-15T14:35:54.784 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00065281900000000001 2023-01-15T14:35:54.784 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.784 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.784 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "commit_sent", 2023-01-15T14:35:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.225496+0000", 2023-01-15T14:35:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.000106038 2023-01-15T14:35:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.226008+0000", 2023-01-15T14:35:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00051175199999999998 2023-01-15T14:35:54.785 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_op(client.4530.0:4229 3.78 3:1e54a7f5:::smithi09025264-42:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e433)", 2023-01-15T14:35:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 29.07985334, 2023-01-15T14:35:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.82811652800000002, 2023-01-15T14:35:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:35:54.786 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "started", 2023-01-15T14:35:54.787 INFO:teuthology.orchestra.run.smithi090.stdout: "client_info": { 2023-01-15T14:35:54.787 INFO:teuthology.orchestra.run.smithi090.stdout: "client": "client.4530", 2023-01-15T14:35:54.787 INFO:teuthology.orchestra.run.smithi090.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:54.787 INFO:teuthology.orchestra.run.smithi090.stdout: "tid": 4229 2023-01-15T14:35:54.787 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.788 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:35:54.788 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.788 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:35:54.788 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:54.788 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.788 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.789 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:35:54.789 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494897+0000", 2023-01-15T14:35:54.789 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:35:54.789 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.789 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:35:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494896+0000", 2023-01-15T14:35:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.999999 2023-01-15T14:35:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.790 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:35:54.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494901+0000", 2023-01-15T14:35:54.791 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4.8269999999999997e-06 2023-01-15T14:35:54.791 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.791 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.791 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:35:54.791 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494924+0000", 2023-01-15T14:35:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 2.3281999999999998e-05 2023-01-15T14:35:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "queued_for_pg", 2023-01-15T14:35:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.494936+0000", 2023-01-15T14:35:54.792 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.1457000000000001e-05 2023-01-15T14:35:54.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.793 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.793 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.793 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596198+0000", 2023-01-15T14:35:54.793 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.10126203 2023-01-15T14:35:54.793 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.794 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.794 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:54.794 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.596400+0000", 2023-01-15T14:35:54.794 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00020234499999999999 2023-01-15T14:35:54.794 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.795 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.795 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.821768+0000", 2023-01-15T14:35:54.795 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.225367977 2023-01-15T14:35:54.795 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.795 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.796 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:54.796 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:23.822124+0000", 2023-01-15T14:35:54.796 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00035540600000000001 2023-01-15T14:35:54.796 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "reached_pg", 2023-01-15T14:35:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.322208+0000", 2023-01-15T14:35:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.50008402100000005 2023-01-15T14:35:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "started", 2023-01-15T14:35:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.322487+0000", 2023-01-15T14:35:54.934 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00027940400000000002 2023-01-15T14:35:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:35:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:35:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "done", 2023-01-15T14:35:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:35:24.323014+0000", 2023-01-15T14:35:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.00052676600000000002 2023-01-15T14:35:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:35:54.935 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:35:54.936 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:54.936 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:35:54.937 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 7, 1, 6] out_osds: [5, 2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:35:54.937 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:35:54.937 INFO:tasks.thrashosds.thrasher:Adding osd 5 2023-01-15T14:35:54.937 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2023-01-15T14:35:54.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2023-01-15T14:35:54.938 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:35:54.939 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:52.608+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:54.939 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:54.932+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:54.943 INFO:tasks.rados.rados.0.smithi090.stdout:1720: done (0 left) 2023-01-15T14:35:54.943 INFO:tasks.rados.rados.0.smithi090.stdout:1721: rollback oid 32 current snap is 188 2023-01-15T14:35:54.943 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 32 to 127 2023-01-15T14:35:54.943 INFO:tasks.rados.rados.0.smithi090.stdout:1722: delete oid 36 current snap is 188 2023-01-15T14:35:54.944 INFO:tasks.rados.rados.0.smithi090.stdout:1722: done (1 left) 2023-01-15T14:35:54.944 INFO:tasks.rados.rados.0.smithi090.stdout:1723: read oid 50 snap 180 2023-01-15T14:35:54.944 INFO:tasks.rados.rados.0.smithi090.stdout:1723: expect (ObjNum 556 snap 176 seq_num 556) 2023-01-15T14:35:54.944 INFO:tasks.rados.rados.0.smithi090.stdout:1724: write (excl) oid 45 current snap is 188 2023-01-15T14:35:54.944 INFO:tasks.rados.rados.0.smithi090.stdout:1724: seq_num 612 ranges {685884=560628,2046274=239610} 2023-01-15T14:35:54.944 INFO:tasks.rados.rados.0.smithi090.stdout:1724: writing smithi09025264-45 from 685884 to 1246512 tid 1 2023-01-15T14:35:54.944 INFO:tasks.rados.rados.0.smithi090.stdout:1724: writing smithi09025264-45 from 2046274 to 2285884 tid 2 2023-01-15T14:35:54.944 INFO:tasks.rados.rados.0.smithi090.stdout:1725: delete oid 6 current snap is 188 2023-01-15T14:35:54.944 INFO:tasks.rados.rados.0.smithi090.stdout:1723: expect (ObjNum 556 snap 176 seq_num 556) 2023-01-15T14:35:54.944 INFO:tasks.rados.rados.0.smithi090.stdout:1723: done (3 left) 2023-01-15T14:35:54.944 INFO:tasks.rados.rados.0.smithi090.stdout:1725: done (2 left) 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1726: write (excl) oid 29 current snap is 188 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1724: finishing write tid 1 to smithi09025264-45 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1726: seq_num 614 ranges {719498=711126,2046885=272613} 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1726: writing smithi09025264-29 from 719498 to 1430624 tid 1 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1726: writing smithi09025264-29 from 2046885 to 2319498 tid 2 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1727: copy_from oid 6 from oid 19 current snap is 188 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1728: write oid 4 current snap is 188 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1728: seq_num 615 ranges {736305=786375,2105367=230938} 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1728: writing smithi09025264-4 from 736305 to 1522680 tid 1 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1728: writing smithi09025264-4 from 2105367 to 2336305 tid 2 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1729: delete oid 15 current snap is 188 2023-01-15T14:35:54.945 INFO:tasks.rados.rados.0.smithi090.stdout:1724: finishing write tid 2 to smithi09025264-45 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:1724: finishing write tid 3 to smithi09025264-45 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:1724: finishing write tid 4 to smithi09025264-45 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:1724: oid 45 updating version 0 to 159 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:1724: oid 45 version 159 is already newer than 158 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:1724: oid 45 version 159 is already newer than 157 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 45 v 159 (ObjNum 612 snap 188 seq_num 612) dirty exists 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:1724: left oid 45 (ObjNum 612 snap 188 seq_num 612) 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:1726: finishing write tid 1 to smithi09025264-29 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:1728: finishing write tid 1 to smithi09025264-4 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:1727: finishing copy_from to smithi09025264-6 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:1727: got expected ENOENT (src dne) 2023-01-15T14:35:54.946 INFO:tasks.rados.rados.0.smithi090.stdout:1728: finishing write tid 2 to smithi09025264-4 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:1727: finishing copy_from racing read to smithi09025264-6 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:1726: finishing write tid 2 to smithi09025264-29 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:1726: finishing write tid 3 to smithi09025264-29 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:1726: finishing write tid 4 to smithi09025264-29 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:1726: oid 29 updating version 0 to 275 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:1726: oid 29 updating version 275 to 276 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:1726: oid 29 updating version 276 to 277 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 29 v 277 (ObjNum 614 snap 188 seq_num 614) dirty exists 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:1726: left oid 29 (ObjNum 614 snap 188 seq_num 614) 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:1728: finishing write tid 3 to smithi09025264-4 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:1728: finishing write tid 4 to smithi09025264-4 2023-01-15T14:35:54.947 INFO:tasks.rados.rados.0.smithi090.stdout:1728: oid 4 updating version 0 to 345 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:1728: oid 4 version 345 is already newer than 343 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:1728: oid 4 version 345 is already newer than 344 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 4 v 345 (ObjNum 615 snap 188 seq_num 615) dirty exists 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:1728: left oid 4 (ObjNum 615 snap 188 seq_num 615) 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:1721: finishing rollback tid 1 to smithi09025264-32 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:1724: done (5 left) 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:1726: done (4 left) 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:1727: done (3 left) 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:1728: done (2 left) 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:1729: done (1 left) 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:1730: write oid 20 current snap is 188 2023-01-15T14:35:54.948 INFO:tasks.rados.rados.0.smithi090.stdout:1730: seq_num 617 ranges {769919=653226,1938684=431235} 2023-01-15T14:35:54.949 INFO:tasks.rados.rados.0.smithi090.stdout:1730: writing smithi09025264-20 from 769919 to 1423145 tid 1 2023-01-15T14:35:54.949 INFO:tasks.rados.rados.0.smithi090.stdout:1730: writing smithi09025264-20 from 1938684 to 2369919 tid 2 2023-01-15T14:35:54.949 INFO:tasks.rados.rados.0.smithi090.stdout:1731: read oid 43 snap 186 2023-01-15T14:35:54.949 INFO:tasks.rados.rados.0.smithi090.stdout:1731: expect deleted 2023-01-15T14:35:55.153 INFO:tasks.rados.rados.0.smithi090.stdout:1732: snap_remove snap 179 2023-01-15T14:35:55.155 INFO:tasks.rados.rados.0.smithi090.stdout:1732: done (3 left) 2023-01-15T14:35:55.158 INFO:tasks.rados.rados.0.smithi090.stdout:1733: snap_create 2023-01-15T14:35:55.158 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:35:55.158 INFO:tasks.rados.rados.0.smithi090.stdout:1721: finishing rollback tid 2 to smithi09025264-32 2023-01-15T14:35:55.158 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 32 v 458 (ObjNum 404 snap 108 seq_num 404) dirty exists 2023-01-15T14:35:55.158 INFO:tasks.rados.rados.0.smithi090.stdout:1730: finishing write tid 1 to smithi09025264-20 2023-01-15T14:35:55.158 INFO:tasks.rados.rados.0.smithi090.stdout:1730: finishing write tid 2 to smithi09025264-20 2023-01-15T14:35:55.158 INFO:tasks.rados.rados.0.smithi090.stdout:1721: done (2 left) 2023-01-15T14:35:55.159 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:35:55.159 INFO:tasks.rados.rados.0.smithi090.stdout:1730: finishing write tid 3 to smithi09025264-20 2023-01-15T14:35:55.159 INFO:tasks.rados.rados.0.smithi090.stdout:1730: finishing write tid 4 to smithi09025264-20 2023-01-15T14:35:55.159 INFO:tasks.rados.rados.0.smithi090.stdout:1730: oid 20 updating version 0 to 268 2023-01-15T14:35:55.159 INFO:tasks.rados.rados.0.smithi090.stdout:1730: oid 20 updating version 268 to 269 2023-01-15T14:35:55.159 INFO:tasks.rados.rados.0.smithi090.stdout:1730: oid 20 updating version 269 to 270 2023-01-15T14:35:55.159 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 20 v 270 (ObjNum 617 snap 188 seq_num 617) dirty exists 2023-01-15T14:35:55.159 INFO:tasks.rados.rados.0.smithi090.stdout:1730: left oid 20 (ObjNum 617 snap 188 seq_num 617) 2023-01-15T14:35:55.159 INFO:tasks.rados.rados.0.smithi090.stdout:1730: done (1 left) 2023-01-15T14:35:55.159 INFO:tasks.rados.rados.0.smithi090.stdout:1731: done (0 left) 2023-01-15T14:35:55.164 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:55.149+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:55.256 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:55.249+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:55.360 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:35:55.360 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [], 2023-01-15T14:35:55.360 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 0 2023-01-15T14:35:55.360 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:35:55.360 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:35:55.361 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:55.361 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:55.361 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:55.362 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:55.356+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:55.378 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:55.398 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:55.545 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:55.545 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:55.546 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:55.546 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:55.546 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:55.547 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:35:55.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:35:55.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:35:55.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:35:55.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:35:55.548 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:35:55.549 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:35:55.549 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:35:55.549 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:35:55.549 INFO:tasks.ceph.mon.a.smithi090.stderr:2023-01-15T14:35:55.493+0000 7fb90e2b4700 -1 mon.a@0(leader).osd e465 definitely_dead 0 2023-01-15T14:35:55.550 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:55.544+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:55.646 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:35:55.637+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:35:55.707 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:55.707 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:55.718 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:35:55.748 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:35:55.741+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:35:55.831 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:55.963 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:55.965 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:35:55.966 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:55.957+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:56.014 INFO:tasks.rados.rados.0.smithi090.stdout:1733: done (0 left) 2023-01-15T14:35:56.015 INFO:tasks.rados.rados.0.smithi090.stdout:1734: snap_remove snap 159 2023-01-15T14:35:56.030 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is unset 2023-01-15T14:35:56.125 INFO:teuthology.orchestra.run.smithi090.stderr:marked in osd.5. 2023-01-15T14:35:56.126 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:56.126 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:56.127 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:35:56.127 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:56.124+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:56.128 INFO:tasks.thrashosds.thrasher:Added osd 5 2023-01-15T14:35:56.225 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:56.302 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:56.303 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:56.224+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:56.304 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:35:56.327 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:56.324+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:56.411 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:56.712 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:56.713 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:35:56.714 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:56.712+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:56.814 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:35:56.812+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:35:56.867 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:56.868 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:56.878 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:35:56.916 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:35:56.912+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:35:56.985 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:56.985 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:56.991 INFO:tasks.rados.rados.0.smithi090.stdout:1734: done (0 left) 2023-01-15T14:35:56.991 INFO:tasks.rados.rados.0.smithi090.stdout:1735: read oid 1 snap -1 2023-01-15T14:35:56.991 INFO:tasks.rados.rados.0.smithi090.stdout:1735: expect (ObjNum 513 snap 152 seq_num 513) 2023-01-15T14:35:56.996 INFO:tasks.rados.rados.0.smithi090.stdout:1736: rollback oid 14 current snap is 189 2023-01-15T14:35:56.996 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 14 to 182 2023-01-15T14:35:56.996 INFO:tasks.rados.rados.0.smithi090.stdout:1737: rollback oid 40 current snap is 189 2023-01-15T14:35:56.996 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 40 to 180 2023-01-15T14:35:56.997 INFO:tasks.rados.rados.0.smithi090.stdout:1738: read oid 23 snap -1 2023-01-15T14:35:56.997 INFO:tasks.rados.rados.0.smithi090.stdout:1738: expect (ObjNum 406 snap 108 seq_num 406) 2023-01-15T14:35:56.997 INFO:tasks.rados.rados.0.smithi090.stdout:1739: rollback oid 46 current snap is 189 2023-01-15T14:35:56.997 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 46 to 188 2023-01-15T14:35:56.997 INFO:tasks.rados.rados.0.smithi090.stdout:1740: copy_from oid 45 from oid 34 current snap is 189 2023-01-15T14:35:56.997 INFO:tasks.rados.rados.0.smithi090.stdout:1741: write (excl) oid 43 current snap is 189 2023-01-15T14:35:56.997 INFO:tasks.rados.rados.0.smithi090.stdout:1741: seq_num 618 ranges {786726=728475,2280813=105913} 2023-01-15T14:35:56.998 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:35:57.013 INFO:tasks.rados.rados.0.smithi090.stdout:1741: writing smithi09025264-43 from 786726 to 1515201 tid 1 2023-01-15T14:35:57.019 INFO:tasks.rados.rados.0.smithi090.stdout:1741: writing smithi09025264-43 from 2280813 to 2386726 tid 2 2023-01-15T14:35:57.019 INFO:tasks.rados.rados.0.smithi090.stdout:1742: read oid 24 snap 171 2023-01-15T14:35:57.020 INFO:tasks.rados.rados.0.smithi090.stdout:1742: expect (ObjNum 486 snap 142 seq_num 486) 2023-01-15T14:35:57.020 INFO:tasks.rados.rados.0.smithi090.stdout:1743: snap_remove snap 178 2023-01-15T14:35:57.021 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:57.013+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:57.117 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:57.109+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:57.157 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:57.157 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:57.169 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:35:57.217 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:57.209+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:57.285 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:57.286 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:57.286 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:35:57.318 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:57.309+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:57.389 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:57.389 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:57.519 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:35:57.520 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:57.509+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:57.550 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:35:57.620 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:35:57.616+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:35:57.639 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:57.639 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:57.649 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2023-01-15T14:35:57.703 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:35:57.922 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:35:57.923 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:35:57.923 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2023-01-15T14:35:57.923 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:35:57.924 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:35:57.924 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:35:57.924 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:35:57.924 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:35:57.924 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:35:57.924 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:35:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:35:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:35:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:35:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:35:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:35:57.925 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:35:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:35:57.926 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:35:57.926 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:35:57.913+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:35:58.009 INFO:tasks.rados.rados.0.smithi090.stdout:1743: done (8 left) 2023-01-15T14:35:58.009 INFO:tasks.rados.rados.0.smithi090.stdout:1744: snap_remove snap 173 2023-01-15T14:35:58.022 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:35:58.013+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:35:58.086 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:35:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:35:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:35:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:35:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1087 3.f 3:fd983206:::smithi09025264-7:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:35:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:35:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 375.29113698399999, 2023-01-15T14:35:58.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0859964360000001, 2023-01-15T14:35:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1087 2023-01-15T14:35:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.087 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:35:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:35:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758720+0000", 2023-01-15T14:35:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:35:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758724+0000", 2023-01-15T14:35:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.7740000000000004e-06 2023-01-15T14:35:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:58.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758738+0000", 2023-01-15T14:35:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.33e-05 2023-01-15T14:35:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758744+0000", 2023-01-15T14:35:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2859999999999996e-06 2023-01-15T14:35:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848121+0000", 2023-01-15T14:35:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089377097000000003 2023-01-15T14:35:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.848127+0000", 2023-01-15T14:35:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.3500000000000002e-06 2023-01-15T14:35:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952454+0000", 2023-01-15T14:35:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.104326167 2023-01-15T14:35:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952520+0000", 2023-01-15T14:35:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6166000000000003e-05 2023-01-15T14:35:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,4", 2023-01-15T14:35:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.063003+0000", 2023-01-15T14:35:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.110482916 2023-01-15T14:35:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065265+0000", 2023-01-15T14:35:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0022626370000000001 2023-01-15T14:35:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065291+0000", 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6095e-05 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.066153+0000", 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086127100000000004 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166533+0000", 2023-01-15T14:35:58.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100380177 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.166646+0000", 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112717 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844498+0000", 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.67785237399999998 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:58.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844605+0000", 2023-01-15T14:35:58.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106691 2023-01-15T14:35:58.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.096 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.096 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:58.096 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844717+0000", 2023-01-15T14:35:58.096 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011203499999999999 2023-01-15T14:35:58.096 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:58.096 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:58.096 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:58.096 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.097 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1088 3.f 3:fd983206:::smithi09025264-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e97)", 2023-01-15T14:35:58.097 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:35:58.097 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 375.29108131800001, 2023-01-15T14:35:58.097 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.086312696, 2023-01-15T14:35:58.097 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:58.097 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:58.097 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:58.097 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1088 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:58.098 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758776+0000", 2023-01-15T14:35:58.099 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:58.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.099 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:58.099 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758775+0000", 2023-01-15T14:35:58.099 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:58.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.099 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:58.099 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758786+0000", 2023-01-15T14:35:58.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.076e-05 2023-01-15T14:35:58.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.100 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.100 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:58.100 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758809+0000", 2023-01-15T14:35:58.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3706999999999999e-05 2023-01-15T14:35:58.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.100 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.100 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:58.100 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.758821+0000", 2023-01-15T14:35:58.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1976e-05 2023-01-15T14:35:58.101 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.101 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.101 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:58.101 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878534+0000", 2023-01-15T14:35:58.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.119712966 2023-01-15T14:35:58.101 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.101 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.101 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:35:58.101 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.878540+0000", 2023-01-15T14:35:58.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0680000000000002e-06 2023-01-15T14:35:58.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:58.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952693+0000", 2023-01-15T14:35:58.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.074153067000000003 2023-01-15T14:35:58.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.102 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:35:58.102 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:42.952735+0000", 2023-01-15T14:35:58.102 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1792999999999998e-05 2023-01-15T14:35:58.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:58.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065518+0000", 2023-01-15T14:35:58.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.11278334499999999 2023-01-15T14:35:58.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.103 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:58.103 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.065687+0000", 2023-01-15T14:35:58.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016807700000000001 2023-01-15T14:35:58.103 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.103 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844786+0000", 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.77909980300000004 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.844944+0000", 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000157424 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:58.104 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:43.845089+0000", 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000144814 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:16 3.13 3:ca393a9e:::smithi09025264-47:head [assert-version v193,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.74410389299999, 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8970339479999998, 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:58.105 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305753+0000", 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.106 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305751+0000", 2023-01-15T14:35:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305763+0000", 2023-01-15T14:35:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2309e-05 2023-01-15T14:35:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.107 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305788+0000", 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4388000000000001e-05 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.305799+0000", 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1304000000000001e-05 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196425+0000", 2023-01-15T14:35:58.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8906259830000001 2023-01-15T14:35:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.196673+0000", 2023-01-15T14:35:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024747499999999998 2023-01-15T14:35:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.202787+0000", 2023-01-15T14:35:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0061146509999999996 2023-01-15T14:35:58.109 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1480 3.13 3:cbfd2d71:::smithi09025264-37:head [write 581575~705262,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:35:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.71083616499999, 2023-01-15T14:35:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.375425221, 2023-01-15T14:35:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:58.110 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1480 2023-01-15T14:35:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:35:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:58.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339021+0000", 2023-01-15T14:35:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339019+0000", 2023-01-15T14:35:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:58.112 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339659+0000", 2023-01-15T14:35:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000639742 2023-01-15T14:35:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339690+0000", 2023-01-15T14:35:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1594000000000003e-05 2023-01-15T14:35:58.113 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.339703+0000", 2023-01-15T14:35:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2828e-05 2023-01-15T14:35:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.510837+0000", 2023-01-15T14:35:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1711338759999999 2023-01-15T14:35:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.114 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511048+0000", 2023-01-15T14:35:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021057899999999999 2023-01-15T14:35:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.511325+0000", 2023-01-15T14:35:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027690400000000002 2023-01-15T14:35:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.115 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512092+0000", 2023-01-15T14:35:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00076720800000000002 2023-01-15T14:35:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512113+0000", 2023-01-15T14:35:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0917e-05 2023-01-15T14:35:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.116 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512439+0000", 2023-01-15T14:35:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032636299999999998 2023-01-15T14:35:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512871+0000", 2023-01-15T14:35:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000432087 2023-01-15T14:35:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.514165+0000", 2023-01-15T14:35:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001293726 2023-01-15T14:35:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:58.118 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714137+0000", 2023-01-15T14:35:59.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19997229499999999 2023-01-15T14:35:59.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:59.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714233+0000", 2023-01-15T14:35:59.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6062000000000003e-05 2023-01-15T14:35:59.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.268 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:59.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714446+0000", 2023-01-15T14:35:59.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000212952 2023-01-15T14:35:59.268 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.268 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:59.268 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.268 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.268 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.268 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1481 3.13 3:cbfd2d71:::smithi09025264-37:head [write 1733272~707409] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:59.268 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:35:59.268 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.70398688199998, 2023-01-15T14:35:59.269 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.572522502, 2023-01-15T14:35:59.269 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:59.269 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:59.269 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:59.269 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:59.269 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:59.269 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1481 2023-01-15T14:35:59.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.269 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:59.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.269 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:59.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:35:59.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.270 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:59.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345870+0000", 2023-01-15T14:35:59.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:59.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.345868+0000", 2023-01-15T14:35:59.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:35:59.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:59.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346316+0000", 2023-01-15T14:35:59.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044813399999999999 2023-01-15T14:35:59.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:59.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346348+0000", 2023-01-15T14:35:59.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2406000000000001e-05 2023-01-15T14:35:59.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:59.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.346362+0000", 2023-01-15T14:35:59.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.375e-05 2023-01-15T14:35:59.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512276+0000", 2023-01-15T14:35:59.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1659136829999999 2023-01-15T14:35:59.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:59.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512427+0000", 2023-01-15T14:35:59.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015088699999999999 2023-01-15T14:35:59.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714565+0000", 2023-01-15T14:35:59.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20213841599999999 2023-01-15T14:35:59.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:59.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714732+0000", 2023-01-15T14:35:59.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016676700000000001 2023-01-15T14:35:59.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:59.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.714964+0000", 2023-01-15T14:35:59.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000231678 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715499+0000", 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00053513300000000003 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715523+0000", 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4266000000000001e-05 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:59.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715867+0000", 2023-01-15T14:35:59.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034419399999999998 2023-01-15T14:35:59.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:59.276 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.816932+0000", 2023-01-15T14:35:59.276 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10106498999999999 2023-01-15T14:35:59.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.276 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:59.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906374+0000", 2023-01-15T14:35:59.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.089441442999999995 2023-01-15T14:35:59.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:59.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918050+0000", 2023-01-15T14:35:59.277 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.011676677999999999 2023-01-15T14:35:59.277 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.277 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.277 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:59.277 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918218+0000", 2023-01-15T14:35:59.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016802000000000001 2023-01-15T14:35:59.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.278 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:59.278 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918393+0000", 2023-01-15T14:35:59.278 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017447 2023-01-15T14:35:59.278 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.278 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:59.278 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.278 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.278 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.279 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1482 3.13 3:cbfd2d71:::smithi09025264-37:head [write 3195938~437497] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:59.279 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:35:59.279 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.69755102400001, 2023-01-15T14:35:59.279 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5664780289999998, 2023-01-15T14:35:59.279 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:59.279 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:59.279 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:59.279 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:59.279 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:59.279 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1482 2023-01-15T14:35:59.279 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.280 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:59.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:59.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:35:59.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.280 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.280 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:59.280 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352306+0000", 2023-01-15T14:35:59.280 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.280 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.281 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:59.281 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352275+0000", 2023-01-15T14:35:59.281 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999695 2023-01-15T14:35:59.281 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.281 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:59.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352644+0000", 2023-01-15T14:35:59.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036870300000000001 2023-01-15T14:35:59.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:59.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352676+0000", 2023-01-15T14:35:59.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1726000000000001e-05 2023-01-15T14:35:59.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:59.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352689+0000", 2023-01-15T14:35:59.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3617e-05 2023-01-15T14:35:59.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512602+0000", 2023-01-15T14:35:59.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1599126690000001 2023-01-15T14:35:59.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:59.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512839+0000", 2023-01-15T14:35:59.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023676900000000001 2023-01-15T14:35:59.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715649+0000", 2023-01-15T14:35:59.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20281044600000001 2023-01-15T14:35:59.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:59.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715777+0000", 2023-01-15T14:35:59.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012738800000000001 2023-01-15T14:35:59.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:59.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.715995+0000", 2023-01-15T14:35:59.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000217858 2023-01-15T14:35:59.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.795 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:59.795 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716694+0000", 2023-01-15T14:35:59.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00069956700000000001 2023-01-15T14:35:59.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716709+0000", 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4362000000000001e-05 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716936+0000", 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022744200000000001 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.796 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:59.797 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.818876+0000", 2023-01-15T14:35:59.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101939635 2023-01-15T14:35:59.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.797 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:59.797 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906390+0000", 2023-01-15T14:35:59.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087513987000000001 2023-01-15T14:35:59.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.797 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:59.797 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918577+0000", 2023-01-15T14:35:59.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012187066 2023-01-15T14:35:59.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:59.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918697+0000", 2023-01-15T14:35:59.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120326 2023-01-15T14:35:59.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.799 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:59.799 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918784+0000", 2023-01-15T14:35:59.799 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.7200999999999994e-05 2023-01-15T14:35:59.799 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.799 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:59.799 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.800 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1483 3.13 3:cbfd2d71:::smithi09025264-37:head [setxattr _header (57) in=7b,truncate 3781575] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:59.800 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:35:59.800 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.69714287199997, 2023-01-15T14:35:59.800 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5663522429999999, 2023-01-15T14:35:59.800 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:59.800 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:59.800 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:59.800 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:59.800 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:59.800 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1483 2023-01-15T14:35:59.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.801 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:59.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:59.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:35:59.801 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:59.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352714+0000", 2023-01-15T14:35:59.801 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352713+0000", 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352720+0000", 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7959999999999998e-06 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352735+0000", 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.455e-05 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.352741+0000", 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8360000000000003e-06 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.512901+0000", 2023-01-15T14:35:59.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1601594980000001 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.513067+0000", 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016626 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.716977+0000", 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20391002799999999 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717146+0000", 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168851 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,4", 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717467+0000", 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032135600000000002 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:59.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717800+0000", 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033217400000000002 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717822+0000", 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2308999999999999e-05 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.717970+0000", 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014788099999999999 2023-01-15T14:35:59.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:59.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.819135+0000", 2023-01-15T14:35:59.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116478 2023-01-15T14:35:59.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:59.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906408+0000", 2023-01-15T14:35:59.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.087272902999999999 2023-01-15T14:35:59.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918887+0000", 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.012479965000000001 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.918959+0000", 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1978999999999996e-05 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:59.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.919067+0000", 2023-01-15T14:35:59.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107083 2023-01-15T14:35:59.809 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.809 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:59.809 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.809 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1485 3.d 3:bc1c6b44:::smithi09025264-23:head [write 598382~780511,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:59.809 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:35:59.809 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.689672438, 2023-01-15T14:35:59.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5474121890000001, 2023-01-15T14:35:59.809 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1485 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360185+0000", 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360168+0000", 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999828 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360790+0000", 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062269099999999998 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360822+0000", 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1903000000000003e-05 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.360836+0000", 2023-01-15T14:35:59.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3380000000000001e-05 2023-01-15T14:35:59.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096664+0000", 2023-01-15T14:35:59.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7358282320000002 2023-01-15T14:35:59.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:35:59.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096683+0000", 2023-01-15T14:35:59.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9063999999999999e-05 2023-01-15T14:35:59.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704561+0000", 2023-01-15T14:35:59.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60787808399999999 2023-01-15T14:35:59.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:59.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.704846+0000", 2023-01-15T14:35:59.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028527400000000002 2023-01-15T14:35:59.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705135+0000", 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028875400000000001 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705735+0000", 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059956899999999999 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:59.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.705761+0000", 2023-01-15T14:35:59.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:35:59.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:59.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.706129+0000", 2023-01-15T14:35:59.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00036822299999999999 2023-01-15T14:35:59.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:59.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.906331+0000", 2023-01-15T14:35:59.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200202091 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907111+0000", 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077990799999999995 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907379+0000", 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026804800000000001 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907529+0000", 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015022700000000001 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907597+0000", 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7785000000000002e-05 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1486 3.d 3:bc1c6b44:::smithi09025264-23:head [write 2075401~451067] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.68327639, 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6461318330000001, 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1486 2023-01-15T14:35:59.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.820 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:59.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:59.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:35:59.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:59.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366581+0000", 2023-01-15T14:35:59.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366579+0000", 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366845+0000", 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026610499999999999 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:59.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.366878+0000", 2023-01-15T14:35:59.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2750000000000003e-05 2023-01-15T14:35:59.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:59.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.467009+0000", 2023-01-15T14:35:59.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100131071 2023-01-15T14:35:59.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096793+0000", 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6297846869999999 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.096806+0000", 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3077e-05 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.907957+0000", 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81115055400000002 2023-01-15T14:35:59.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908229+0000", 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027187499999999998 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,4", 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.908496+0000", 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026754200000000001 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909025+0000", 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052865200000000001 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909046+0000", 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0713e-05 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.909294+0000", 2023-01-15T14:35:59.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024864899999999998 2023-01-15T14:35:59.826 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:35:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.911607+0000", 2023-01-15T14:35:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0023128939999999998 2023-01-15T14:35:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012076+0000", 2023-01-15T14:35:59.955 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100468293 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012426+0000", 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00035044299999999998 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012532+0000", 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000106142 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.956 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.957 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:59.957 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:23.012713+0000", 2023-01-15T14:35:59.957 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018048599999999999 2023-01-15T14:35:59.957 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.957 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:59.957 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.957 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.957 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.957 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1496 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:59.957 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:35:59.957 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.313633541, 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870616037, 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1496 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:59.958 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736224+0000", 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736220+0000", 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:35:59.959 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.960 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.960 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:59.960 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736238+0000", 2023-01-15T14:35:59.960 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7156e-05 2023-01-15T14:35:59.960 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.960 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.960 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:59.960 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736275+0000", 2023-01-15T14:35:59.960 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7876000000000001e-05 2023-01-15T14:35:59.961 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.961 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.961 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:59.961 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736290+0000", 2023-01-15T14:35:59.961 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4449e-05 2023-01-15T14:35:59.961 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.961 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.961 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.961 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602217+0000", 2023-01-15T14:35:59.961 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8659270299999999 2023-01-15T14:35:59.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.962 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.962 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:59.962 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.602468+0000", 2023-01-15T14:35:59.962 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000251315 2023-01-15T14:35:59.962 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.962 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.962 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:59.962 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.606840+0000", 2023-01-15T14:35:59.962 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0043715009999999999 2023-01-15T14:35:59.962 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.963 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:59.963 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.963 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.963 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.963 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1497 3.11 3:8889e450:::smithi09025264-32:head [read 0~2991646,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:35:59.963 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:35:59.963 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.31354608700002, 2023-01-15T14:35:59.963 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738359249999998, 2023-01-15T14:35:59.963 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:35:59.963 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:35:59.963 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:35:59.964 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:35:59.964 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:35:59.964 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1497 2023-01-15T14:35:59.964 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.964 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:35:59.964 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.964 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:35:59.964 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:35:59.964 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.964 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.964 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736311+0000", 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736310+0000", 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:35:59.965 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736317+0000", 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.7240000000000003e-06 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736334+0000", 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6878999999999999e-05 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736341+0000", 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0129999999999996e-06 2023-01-15T14:35:59.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.967 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:35:59.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607070+0000", 2023-01-15T14:35:59.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.870728873 2023-01-15T14:35:59.967 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.967 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:35:59.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.607226+0000", 2023-01-15T14:35:59.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015595 2023-01-15T14:35:59.967 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:35:59.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:35:59.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:35:59.968 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610147+0000", 2023-01-15T14:35:59.968 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.002921416 2023-01-15T14:35:59.968 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.968 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:35:59.968 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:35:59.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.114 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.114 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1498 3.11 3:8889e450:::smithi09025264-32:head [sparse-read 0~2991646] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e132)", 2023-01-15T14:36:00.114 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:36:00.114 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 338.31350598099999, 2023-01-15T14:36:00.114 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8752667839999999, 2023-01-15T14:36:00.114 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:00.114 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:00.114 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:00.114 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:00.114 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:00.115 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1498 2023-01-15T14:36:00.115 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.115 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:00.115 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.115 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:00.115 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:36:00.115 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.115 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.115 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.115 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:00.115 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736351+0000", 2023-01-15T14:36:00.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.116 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.116 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:00.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736350+0000", 2023-01-15T14:36:00.116 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:00.116 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.116 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.116 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:00.116 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736355+0000", 2023-01-15T14:36:00.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4900000000000002e-06 2023-01-15T14:36:00.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:00.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736363+0000", 2023-01-15T14:36:00.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2770000000000005e-06 2023-01-15T14:36:00.117 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.117 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.117 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:00.117 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:19.736369+0000", 2023-01-15T14:36:00.117 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.7400000000000001e-06 2023-01-15T14:36:00.118 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.118 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.118 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.118 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610234+0000", 2023-01-15T14:36:00.118 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8738651439999998 2023-01-15T14:36:00.118 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.118 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.118 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:00.118 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.610402+0000", 2023-01-15T14:36:00.118 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000168283 2023-01-15T14:36:00.118 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.119 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.119 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:00.119 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:22.611618+0000", 2023-01-15T14:36:00.119 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001215647 2023-01-15T14:36:00.119 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.119 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:00.119 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.119 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.119 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.119 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop_reply(osd.7.0:440 3.11 e231/207 ondisk, result = 0)", 2023-01-15T14:36:00.119 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:36:00.120 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 239.80572432299999, 2023-01-15T14:36:00.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973962139999998, 2023-01-15T14:36:00.120 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:00.120 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:00.120 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:00.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.120 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:00.120 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:36:00.120 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.120 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.120 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244133+0000", 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244132+0000", 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:00.121 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244143+0000", 2023-01-15T14:36:00.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1647e-05 2023-01-15T14:36:00.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:00.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244165+0000", 2023-01-15T14:36:00.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1501999999999998e-05 2023-01-15T14:36:00.122 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.122 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.122 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:00.122 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.244180+0000", 2023-01-15T14:36:00.122 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5434000000000002e-05 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941489+0000", 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6973091899999999 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941503+0000", 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.414e-05 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.123 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.124 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:00.124 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.941529+0000", 2023-01-15T14:36:00.124 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5622999999999998e-05 2023-01-15T14:36:00.124 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.124 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:00.124 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.124 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.124 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.124 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.4.0:562 3.1b e231/224 3:d85f42d9:::smithi09025264-45:a8 v 231'66, mlcod=225'62)", 2023-01-15T14:36:00.124 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:36:00.124 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 239.615373127, 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2578820149999999, 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:00.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:36:00.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.126 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:00.126 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434484+0000", 2023-01-15T14:36:00.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:36:00.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.126 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:00.126 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434492+0000", 2023-01-15T14:36:00.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4360000000000002e-06 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434517+0000", 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.48e-05 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434530+0000", 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2982999999999999e-05 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434663+0000", 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013307 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.434700+0000", 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7020000000000001e-05 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:00.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435438+0000", 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073808900000000004 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435456+0000", 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8091999999999999e-05 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435526+0000", 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9806000000000002e-05 2023-01-15T14:36:00.129 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:00.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.435570+0000", 2023-01-15T14:36:00.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.4091999999999997e-05 2023-01-15T14:36:00.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:00.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.692366+0000", 2023-01-15T14:36:00.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2567961539999999 2023-01-15T14:36:00.130 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.130 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(osd.1.13:24 3.d 3:b3f111b5:::smithi09025264-48:head [assert-version v160,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e232)", 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 239.099536911, 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9963031440000001, 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.131 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:00.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:36:00.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:00.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950320+0000", 2023-01-15T14:36:00.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.132 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.132 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.132 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:00.132 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950318+0000", 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950333+0000", 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.467e-05 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:58.950368+0000", 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4895999999999999e-05 2023-01-15T14:36:00.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050461+0000", 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10009304400000001 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050593+0000", 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000132764 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:36:00.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050615+0000", 2023-01-15T14:36:00.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1483000000000001e-05 2023-01-15T14:36:00.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945049+0000", 2023-01-15T14:36:00.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.894434138 2023-01-15T14:36:00.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.945326+0000", 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000277368 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.946623+0000", 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012970340000000001 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:00.136 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.137 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(osd.3.0:547 3.22 e232/180 3:47dfe935:::smithi09025264-29:81 v 232'124, mlcod=229'121)", 2023-01-15T14:36:00.137 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:36:00.137 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 238.999307163, 2023-01-15T14:36:00.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.643153431, 2023-01-15T14:36:00.137 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:00.137 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:00.137 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:00.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:00.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:36:00.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:00.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050550+0000", 2023-01-15T14:36:00.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:00.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050549+0000", 2023-01-15T14:36:00.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:36:00.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:00.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050561+0000", 2023-01-15T14:36:00.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.278e-05 2023-01-15T14:36:00.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:00.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050583+0000", 2023-01-15T14:36:00.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1053e-05 2023-01-15T14:36:00.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:00.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050596+0000", 2023-01-15T14:36:00.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3946e-05 2023-01-15T14:36:00.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050708+0000", 2023-01-15T14:36:00.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011129 2023-01-15T14:36:00.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:00.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.050750+0000", 2023-01-15T14:36:00.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2725000000000002e-05 2023-01-15T14:36:00.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:00.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051324+0000", 2023-01-15T14:36:00.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00057335700000000003 2023-01-15T14:36:00.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:00.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051340+0000", 2023-01-15T14:36:00.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5968999999999999e-05 2023-01-15T14:36:00.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:00.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051405+0000", 2023-01-15T14:36:00.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5419000000000005e-05 2023-01-15T14:36:00.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.815 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:00.815 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:59.051471+0000", 2023-01-15T14:36:00.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5671000000000002e-05 2023-01-15T14:36:00.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:00.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:32:00.693703+0000", 2023-01-15T14:36:00.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.642232554 2023-01-15T14:36:00.816 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.816 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:00.816 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.816 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3881 3.40 3:03b54904:::smithi09025264-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e392)", 2023-01-15T14:36:00.816 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:34:42.155799+0000", 2023-01-15T14:36:00.817 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 75.894058341999994, 2023-01-15T14:36:00.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1132586929999999, 2023-01-15T14:36:00.817 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:00.817 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:00.817 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:00.817 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:00.817 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:00.817 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3881 2023-01-15T14:36:00.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.817 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:00.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:00.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155799+0000", 2023-01-15T14:36:00.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:00.818 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155799+0000", 2023-01-15T14:36:00.818 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.818 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.818 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.818 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:00.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155798+0000", 2023-01-15T14:36:00.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:00.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:00.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155804+0000", 2023-01-15T14:36:00.819 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4969999999999998e-06 2023-01-15T14:36:00.819 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.819 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.819 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:00.819 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155820+0000", 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5387999999999999e-05 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.155827+0000", 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9759999999999998e-06 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.651043+0000", 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.49521586499999998 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.820 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:00.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:42.651233+0000", 2023-01-15T14:36:00.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000190259 2023-01-15T14:36:00.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.821 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.821 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.821 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:43.268845+0000", 2023-01-15T14:36:00.821 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61761231400000005 2023-01-15T14:36:00.821 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:00.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:43.268939+0000", 2023-01-15T14:36:00.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4122000000000004e-05 2023-01-15T14:36:00.822 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.822 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.822 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:00.822 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:34:43.269058+0000", 2023-01-15T14:36:00.822 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011818400000000001 2023-01-15T14:36:00.822 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.822 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4147 3.11 3:883e9a29:::smithi09025264-14:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e423)", 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:35:13.102895+0000", 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 44.946961872999999, 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.018866324, 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:00.823 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:00.824 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4147 2023-01-15T14:36:00.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.824 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:00.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:00.824 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102895+0000", 2023-01-15T14:36:00.824 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.824 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.824 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.824 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:00.825 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102895+0000", 2023-01-15T14:36:00.825 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.965 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:00.965 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102895+0000", 2023-01-15T14:36:00.965 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:00.965 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.965 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.965 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:00.966 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102899+0000", 2023-01-15T14:36:00.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1300000000000003e-06 2023-01-15T14:36:00.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.966 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.966 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:00.966 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102911+0000", 2023-01-15T14:36:00.966 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2096000000000001e-05 2023-01-15T14:36:00.966 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.966 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.966 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:00.966 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102917+0000", 2023-01-15T14:36:00.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8470000000000001e-06 2023-01-15T14:36:00.967 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.967 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102983+0000", 2023-01-15T14:36:00.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6594000000000006e-05 2023-01-15T14:36:00.967 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.967 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.967 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:36:00.967 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.102996+0000", 2023-01-15T14:36:00.967 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2459999999999999e-05 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.823282+0000", 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.72028614400000002 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:13.823457+0000", 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017553299999999999 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.968 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:14.121348+0000", 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.297891085 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:14.121531+0000", 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000182086 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:00.969 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:14.121762+0000", 2023-01-15T14:36:00.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023108199999999999 2023-01-15T14:36:00.970 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.970 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:00.970 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:00.970 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.970 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.970 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4443 3.11 3:8889e450:::smithi09025264-32:head [rollback f7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2023-01-15T14:36:00.970 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:35:52.999947+0000", 2023-01-15T14:36:00.970 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 5.0499097620000004, 2023-01-15T14:36:00.970 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2087994360000001, 2023-01-15T14:36:00.970 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4443 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:52.999947+0000", 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.971 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.972 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.972 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:00.972 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:52.999947+0000", 2023-01-15T14:36:00.972 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:00.972 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.972 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.972 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:00.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:52.999945+0000", 2023-01-15T14:36:00.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:36:00.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:00.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:52.999955+0000", 2023-01-15T14:36:00.973 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9059999999999997e-06 2023-01-15T14:36:00.973 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.973 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.973 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:00.973 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:52.999969+0000", 2023-01-15T14:36:00.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4107999999999999e-05 2023-01-15T14:36:00.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.974 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.974 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:00.974 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:52.999980+0000", 2023-01-15T14:36:00.974 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0628999999999999e-05 2023-01-15T14:36:00.974 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.974 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.975 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:00.975 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.070765+0000", 2023-01-15T14:36:00.975 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.070784861000000004 2023-01-15T14:36:00.975 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.975 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.975 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:00.975 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.070987+0000", 2023-01-15T14:36:00.975 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022242900000000001 2023-01-15T14:36:00.975 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.975 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.975 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:36:00.976 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.071610+0000", 2023-01-15T14:36:00.976 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00062306299999999998 2023-01-15T14:36:00.976 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:00.976 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:00.976 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:00.976 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.072035+0000", 2023-01-15T14:36:00.976 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042529800000000001 2023-01-15T14:36:01.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:01.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.072050+0000", 2023-01-15T14:36:01.125 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4683e-05 2023-01-15T14:36:01.125 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.125 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.125 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:01.125 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.072101+0000", 2023-01-15T14:36:01.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1309e-05 2023-01-15T14:36:01.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.126 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:01.126 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.072166+0000", 2023-01-15T14:36:01.126 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.4918000000000004e-05 2023-01-15T14:36:01.126 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.126 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.127 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:01.127 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.098981+0000", 2023-01-15T14:36:01.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.026814726000000001 2023-01-15T14:36:01.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.127 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:01.127 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:54.208341+0000", 2023-01-15T14:36:01.127 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1093598659999999 2023-01-15T14:36:01.127 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.127 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.127 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:01.128 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:54.208516+0000", 2023-01-15T14:36:01.128 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000174833 2023-01-15T14:36:01.130 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.130 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.130 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:01.130 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:54.208747+0000", 2023-01-15T14:36:01.130 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023111900000000001 2023-01-15T14:36:01.131 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:01.131 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:01.131 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:01.131 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.131 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.132 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4444 3.11 3:8889e450:::smithi09025264-32:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2023-01-15T14:36:01.132 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:35:52.999991+0000", 2023-01-15T14:36:01.132 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 5.0498660019999999, 2023-01-15T14:36:01.132 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5352563290000001, 2023-01-15T14:36:01.132 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:01.132 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:01.133 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:01.133 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:01.133 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:01.133 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4444 2023-01-15T14:36:01.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.133 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:01.133 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.133 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:01.133 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:52.999991+0000", 2023-01-15T14:36:01.133 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:01.133 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:01.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:52.999991+0000", 2023-01-15T14:36:01.134 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:01.134 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.134 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.134 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:01.134 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:52.999991+0000", 2023-01-15T14:36:01.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:36:01.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:01.135 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:52.999994+0000", 2023-01-15T14:36:01.135 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.631e-06 2023-01-15T14:36:01.135 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.135 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.135 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:01.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.000031+0000", 2023-01-15T14:36:01.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7032000000000001e-05 2023-01-15T14:36:01.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:01.136 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.000038+0000", 2023-01-15T14:36:01.136 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6719999999999998e-06 2023-01-15T14:36:01.136 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.136 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.136 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:01.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.094990+0000", 2023-01-15T14:36:01.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.094951595999999999 2023-01-15T14:36:01.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:01.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.095371+0000", 2023-01-15T14:36:01.137 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000381765 2023-01-15T14:36:01.137 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.137 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.137 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:36:01.137 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.095979+0000", 2023-01-15T14:36:01.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060788699999999999 2023-01-15T14:36:01.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:01.138 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.096695+0000", 2023-01-15T14:36:01.138 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071590800000000002 2023-01-15T14:36:01.138 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.138 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.138 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:01.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.097084+0000", 2023-01-15T14:36:01.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038897200000000002 2023-01-15T14:36:01.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:01.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.097168+0000", 2023-01-15T14:36:01.139 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3552000000000002e-05 2023-01-15T14:36:01.139 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.139 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.139 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:01.139 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.098223+0000", 2023-01-15T14:36:01.140 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0010552300000000001 2023-01-15T14:36:01.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.140 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:01.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:53.200256+0000", 2023-01-15T14:36:01.140 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102032787 2023-01-15T14:36:01.140 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.140 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.140 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:01.140 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:54.535068+0000", 2023-01-15T14:36:01.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3348119899999999 2023-01-15T14:36:01.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:01.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:54.535175+0000", 2023-01-15T14:36:01.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000107738 2023-01-15T14:36:01.141 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:01.141 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:01.141 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:01.141 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:54.535248+0000", 2023-01-15T14:36:01.141 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2016999999999998e-05 2023-01-15T14:36:01.142 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:01.142 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:01.142 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:01.142 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:01.142 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:01.142 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:36:01.142 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2023-01-15T14:36:01.143 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2023-01-15T14:36:01.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:36:01.144 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 7, 1, 6, 5] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:36:01.144 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:36:01.144 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2023-01-15T14:36:01.144 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:36:01.291 INFO:tasks.rados.rados.0.smithi090.stdout:1744: done (8 left) 2023-01-15T14:36:01.291 INFO:tasks.rados.rados.0.smithi090.stdout:1745: rollback oid 26 current snap is 189 2023-01-15T14:36:01.291 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 26 to 184 2023-01-15T14:36:01.291 INFO:tasks.rados.rados.0.smithi090.stdout:1746: snap_remove snap 158 2023-01-15T14:36:01.291 INFO:tasks.rados.rados.0.smithi090.stdout:1746: done (9 left) 2023-01-15T14:36:01.291 INFO:tasks.rados.rados.0.smithi090.stdout:1747: write (excl) oid 12 current snap is 189 2023-01-15T14:36:01.292 INFO:tasks.rados.rados.0.smithi090.stdout:1747: seq_num 619 ranges {403533=403724,1539295=652201} 2023-01-15T14:36:01.292 INFO:tasks.rados.rados.0.smithi090.stdout:1747: writing smithi09025264-12 from 403533 to 807257 tid 1 2023-01-15T14:36:01.292 INFO:tasks.rados.rados.0.smithi090.stdout:1747: writing smithi09025264-12 from 1539295 to 2191496 tid 2 2023-01-15T14:36:01.292 INFO:tasks.rados.rados.0.smithi090.stdout:1748: snap_create 2023-01-15T14:36:01.292 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 10 2023-01-15T14:36:01.292 INFO:tasks.rados.rados.0.smithi090.stdout:1738: expect (ObjNum 406 snap 108 seq_num 406) 2023-01-15T14:36:01.292 INFO:tasks.rados.rados.0.smithi090.stdout:1737: finishing rollback tid 1 to smithi09025264-40 2023-01-15T14:36:01.292 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 40 v 314 (ObjNum 33 snap 0 seq_num 3182663392) dirty dne 2023-01-15T14:36:01.292 INFO:tasks.rados.rados.0.smithi090.stdout:1740: finishing copy_from to smithi09025264-45 2023-01-15T14:36:01.292 INFO:tasks.rados.rados.0.smithi090.stdout:1740: got expected ENOENT (src dne) 2023-01-15T14:36:01.293 INFO:tasks.rados.rados.0.smithi090.stdout:1740: finishing copy_from racing read to smithi09025264-45 2023-01-15T14:36:01.293 INFO:tasks.rados.rados.0.smithi090.stdout:1736: finishing rollback tid 0 to smithi09025264-14 2023-01-15T14:36:01.293 INFO:tasks.rados.rados.0.smithi090.stdout:1736: finishing rollback tid 1 to smithi09025264-14 2023-01-15T14:36:01.293 INFO:tasks.rados.rados.0.smithi090.stdout:1735: expect (ObjNum 513 snap 152 seq_num 513) 2023-01-15T14:36:01.293 INFO:tasks.rados.rados.0.smithi090.stdout:1736: finishing rollback tid 2 to smithi09025264-14 2023-01-15T14:36:01.293 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 461 (ObjNum 598 snap 182 seq_num 598) dirty exists 2023-01-15T14:36:01.293 INFO:tasks.rados.rados.0.smithi090.stdout:1742: expect (ObjNum 486 snap 142 seq_num 486) 2023-01-15T14:36:01.293 INFO:tasks.rados.rados.0.smithi090.stdout:1739: finishing rollback tid 0 to smithi09025264-46 2023-01-15T14:36:01.293 INFO:tasks.rados.rados.0.smithi090.stdout:1741: finishing write tid 1 to smithi09025264-43 2023-01-15T14:36:01.293 INFO:tasks.rados.rados.0.smithi090.stdout:1741: finishing write tid 2 to smithi09025264-43 2023-01-15T14:36:01.465 INFO:tasks.rados.rados.0.smithi090.stdout:1741: finishing write tid 3 to smithi09025264-43 2023-01-15T14:36:01.465 INFO:tasks.rados.rados.0.smithi090.stdout:1741: finishing write tid 4 to smithi09025264-43 2023-01-15T14:36:01.467 INFO:tasks.rados.rados.0.smithi090.stdout:1741: oid 43 updating version 0 to 133 2023-01-15T14:36:01.467 INFO:tasks.rados.rados.0.smithi090.stdout:1741: oid 43 version 133 is already newer than 131 2023-01-15T14:36:01.467 INFO:tasks.rados.rados.0.smithi090.stdout:1741: oid 43 version 133 is already newer than 132 2023-01-15T14:36:01.467 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 43 v 133 (ObjNum 618 snap 189 seq_num 618) dirty exists 2023-01-15T14:36:01.467 INFO:tasks.rados.rados.0.smithi090.stdout:1741: left oid 43 (ObjNum 618 snap 189 seq_num 618) 2023-01-15T14:36:01.468 INFO:tasks.rados.rados.0.smithi090.stdout:1745: finishing rollback tid 1 to smithi09025264-26 2023-01-15T14:36:01.468 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 340 (ObjNum 289 snap 0 seq_num 1) dirty dne 2023-01-15T14:36:01.468 INFO:tasks.rados.rados.0.smithi090.stdout:1735: done (9 left) 2023-01-15T14:36:01.468 INFO:tasks.rados.rados.0.smithi090.stdout:1736: done (8 left) 2023-01-15T14:36:02.039 INFO:tasks.rados.rados.0.smithi090.stdout:1737: done (7 left) 2023-01-15T14:36:02.040 INFO:tasks.rados.rados.0.smithi090.stdout:1738: done (6 left) 2023-01-15T14:36:02.040 INFO:tasks.rados.rados.0.smithi090.stdout:1740: done (5 left) 2023-01-15T14:36:02.041 INFO:tasks.rados.rados.0.smithi090.stdout:1741: done (4 left) 2023-01-15T14:36:02.041 INFO:tasks.rados.rados.0.smithi090.stdout:1742: done (3 left) 2023-01-15T14:36:02.041 INFO:tasks.rados.rados.0.smithi090.stdout:1745: done (2 left) 2023-01-15T14:36:02.041 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:36:02.042 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:36:01.137+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:36:02.043 INFO:tasks.rados.rados.0.smithi090.stdout:1747: finishing write tid 1 to smithi09025264-12 2023-01-15T14:36:02.043 INFO:tasks.rados.rados.0.smithi090.stdout:1747: finishing write tid 2 to smithi09025264-12 2023-01-15T14:36:02.043 INFO:tasks.rados.rados.0.smithi090.stdout:1747: finishing write tid 3 to smithi09025264-12 2023-01-15T14:36:02.043 INFO:tasks.rados.rados.0.smithi090.stdout:1747: finishing write tid 4 to smithi09025264-12 2023-01-15T14:36:02.043 INFO:tasks.rados.rados.0.smithi090.stdout:1747: oid 12 updating version 0 to 365 2023-01-15T14:36:02.043 INFO:tasks.rados.rados.0.smithi090.stdout:1747: oid 12 version 365 is already newer than 364 2023-01-15T14:36:02.043 INFO:tasks.rados.rados.0.smithi090.stdout:1747: oid 12 version 365 is already newer than 363 2023-01-15T14:36:02.044 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 365 (ObjNum 619 snap 189 seq_num 619) dirty exists 2023-01-15T14:36:02.044 INFO:tasks.rados.rados.0.smithi090.stdout:1747: left oid 12 (ObjNum 619 snap 189 seq_num 619) 2023-01-15T14:36:02.044 INFO:tasks.rados.rados.0.smithi090.stdout:1747: done (1 left) 2023-01-15T14:36:02.044 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:36:02.046 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:36:02.037+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:36:02.147 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:36:02.137+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:36:02.247 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:36:02.247 INFO:teuthology.orchestra.run.smithi090.stdout: "ops": [ 2023-01-15T14:36:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "description": "osd_repop(osd.7.0:1200 3.11 e471/459)", 2023-01-15T14:36:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "initiated_at": "2023-01-15T14:36:02.143386+0000", 2023-01-15T14:36:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "age": 0.042616899, 2023-01-15T14:36:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0.042638409000000002, 2023-01-15T14:36:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "type_data": { 2023-01-15T14:36:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "flag_point": "no flag points reached", 2023-01-15T14:36:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "events": [ 2023-01-15T14:36:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "initiated", 2023-01-15T14:36:02.248 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:36:02.143386+0000", 2023-01-15T14:36:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:36:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "throttled", 2023-01-15T14:36:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:36:02.143386+0000", 2023-01-15T14:36:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 0 2023-01-15T14:36:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "header_read", 2023-01-15T14:36:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:36:02.143385+0000", 2023-01-15T14:36:02.249 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 4294967295.9999986 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "all_read", 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:36:02.143394+0000", 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 9.2709999999999998e-06 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "event": "dispatched", 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "time": "2023-01-15T14:36:02.143408+0000", 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: "duration": 1.385e-05 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:02.250 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:36:02.251 INFO:teuthology.orchestra.run.smithi090.stdout: "num_ops": 1 2023-01-15T14:36:02.251 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:36:02.251 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2023-01-15T14:36:02.252 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:36:02.252 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:36:02.244+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:36:02.286 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:36:02.449 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:36:02.449 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:36:02.450 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:02.450 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:02.451 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2023-01-15T14:36:02.611 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:36:02.611 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:36:02.614 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:36:02.448+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:36:02.781 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:36:02.784 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:36:02.785 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:36:02.786 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":702,"stamp":"2023-01-15T14:36:01.981703+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":400515675,"num_objects":184,"num_object_clones":128,"num_object_copies":552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":156,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":184,"num_whiteouts":17,"num_read":1414,"num_read_kb":1451489,"num_write":1137,"num_write_kb":392457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1215,"num_bytes_recovered":824812604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28232,"ondisk_log_size":28232,"up":465,"acting":445,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":520,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":7190056,"kb_used_data":648716772,"kb_used_omap":103929,"kb_used_meta":0,"kb_avail":648705604,"statfs":{"total":671637155840,"available":664274538496,"internally_reserved":11436032,"allocated":664285974528,"data_stored":664285974528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":106423800,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":9,"commit_latency_ns":0,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":37348156,"num_objects":18,"num_object_clones":11,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":156,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":161,"num_read_kb":180756,"num_write":123,"num_write_kb":42120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":119741382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.982487"},"pg_stats":[{"pgid":"3.91","version":"458'396","reported_seq":2014,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.306149+0000","last_change":"2023-01-15T14:35:57.306206+0000","last_active":"2023-01-15T14:35:57.306149+0000","last_peered":"2023-01-15T14:35:57.306149+0000","last_clean":"2023-01-15T14:35:57.306149+0000","last_became_active":"2023-01-15T14:35:55.531898+0000","last_became_peered":"2023-01-15T14:35:55.531898+0000","last_unstale":"2023-01-15T14:35:57.306149+0000","last_undegraded":"2023-01-15T14:35:57.306149+0000","last_fullsized":"2023-01-15T14:35:57.306149+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":8,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":396,"log_dups_size":0,"ondisk_log_size":396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.20081180700000001,"stat_sum":{"num_bytes":10810582,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":25,"num_read_kb":24146,"num_write":21,"num_write_kb":14138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":22433542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"}]},{"pgid":"3.90","version":"454'443","reported_seq":1816,"reported_epoch":467,"state":"remapped+peering","last_fresh":"2023-01-15T14:35:56.993087+0000","last_change":"2023-01-15T14:35:56.994060+0000","last_active":"2023-01-15T14:35:56.015619+0000","last_peered":"2023-01-15T14:35:55.759600+0000","last_clean":"2023-01-15T14:35:50.519633+0000","last_became_active":"2023-01-15T14:35:53.205241+0000","last_became_peered":"2023-01-15T14:35:53.205241+0000","last_unstale":"2023-01-15T14:35:56.993087+0000","last_undegraded":"2023-01-15T14:35:56.993087+0000","last_fullsized":"2023-01-15T14:35:56.993087+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":8,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4003031,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":31691,"num_write":28,"num_write_kb":9855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17879824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4,5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.8f","version":"155'145","reported_seq":1255,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.309414+0000","last_change":"2023-01-15T14:35:57.309469+0000","last_active":"2023-01-15T14:35:57.309414+0000","last_peered":"2023-01-15T14:35:57.309414+0000","last_clean":"2023-01-15T14:35:57.309414+0000","last_became_active":"2023-01-15T14:35:55.308909+0000","last_became_peered":"2023-01-15T14:35:55.308909+0000","last_unstale":"2023-01-15T14:35:57.309414+0000","last_undegraded":"2023-01-15T14:35:57.309414+0000","last_fullsized":"2023-01-15T14:35:57.309414+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":8,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.21363404,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.8e","version":"139'38","reported_seq":30,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.308804+0000","last_change":"2023-01-15T14:35:57.308856+0000","last_active":"2023-01-15T14:35:57.308804+0000","last_peered":"2023-01-15T14:35:57.308804+0000","last_clean":"2023-01-15T14:35:57.308804+0000","last_became_active":"2023-01-15T14:35:52.189818+0000","last_became_peered":"2023-01-15T14:35:52.189818+0000","last_unstale":"2023-01-15T14:35:57.308804+0000","last_undegraded":"2023-01-15T14:35:57.308804+0000","last_fullsized":"2023-01-15T14:35:57.308804+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":8,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.30807628300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.8d","version":"296'232","reported_seq":1623,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.012172+0000","last_change":"2023-01-15T14:35:57.707401+0000","last_active":"2023-01-15T14:35:58.012172+0000","last_peered":"2023-01-15T14:35:58.012172+0000","last_clean":"2023-01-15T14:35:58.012172+0000","last_became_active":"2023-01-15T14:35:55.205822+0000","last_became_peered":"2023-01-15T14:35:55.205822+0000","last_unstale":"2023-01-15T14:35:58.012172+0000","last_undegraded":"2023-01-15T14:35:58.012172+0000","last_fullsized":"2023-01-15T14:35:58.012172+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":8,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.71273130399999995,"stat_sum":{"num_bytes":2306688,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7821,"num_write":6,"num_write_kb":1640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.8c","version":"464'160","reported_seq":1625,"reported_epoch":468,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:58.733373+0000","last_change":"2023-01-15T14:35:58.733415+0000","last_active":"2023-01-15T14:35:58.733373+0000","last_peered":"2023-01-15T14:35:58.733373+0000","last_clean":"2023-01-15T14:35:58.733373+0000","last_became_active":"2023-01-15T14:35:58.408649+0000","last_became_peered":"2023-01-15T14:35:58.408649+0000","last_unstale":"2023-01-15T14:35:58.733373+0000","last_undegraded":"2023-01-15T14:35:58.733373+0000","last_fullsized":"2023-01-15T14:35:58.733373+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":8,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4142098,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":33625,"num_write":13,"num_write_kb":2913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":23960242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.8b","version":"155'53","reported_seq":28,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.221949+0000","last_change":"2023-01-15T14:35:59.222041+0000","last_active":"2023-01-15T14:35:59.221949+0000","last_peered":"2023-01-15T14:35:59.221949+0000","last_clean":"2023-01-15T14:35:59.221949+0000","last_became_active":"2023-01-15T14:35:52.450506+0000","last_became_peered":"2023-01-15T14:35:52.450506+0000","last_unstale":"2023-01-15T14:35:59.221949+0000","last_undegraded":"2023-01-15T14:35:59.221949+0000","last_fullsized":"2023-01-15T14:35:59.221949+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":8,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.10585623299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.8a","version":"252'279","reported_seq":1494,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.729463+0000","last_change":"2023-01-15T14:35:58.729908+0000","last_active":"2023-01-15T14:35:58.729463+0000","last_peered":"2023-01-15T14:35:58.729463+0000","last_clean":"2023-01-15T14:35:58.729463+0000","last_became_active":"2023-01-15T14:35:55.203949+0000","last_became_peered":"2023-01-15T14:35:55.203949+0000","last_unstale":"2023-01-15T14:35:58.729463+0000","last_undegraded":"2023-01-15T14:35:58.729463+0000","last_fullsized":"2023-01-15T14:35:58.729463+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":8,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.52303755200000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.89","version":"110'149","reported_seq":483,"reported_epoch":469,"state":"peering","last_fresh":"2023-01-15T14:35:59.015327+0000","last_change":"2023-01-15T14:35:59.015674+0000","last_active":"2023-01-15T14:35:59.015273+0000","last_peered":"2023-01-15T14:35:58.318896+0000","last_clean":"2023-01-15T14:35:55.208377+0000","last_became_active":"2023-01-15T14:35:58.318093+0000","last_became_peered":"2023-01-15T14:35:58.318093+0000","last_unstale":"2023-01-15T14:35:59.015327+0000","last_undegraded":"2023-01-15T14:35:59.015327+0000","last_fullsized":"2023-01-15T14:35:59.015327+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":8,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":714293,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6218,"num_write":4,"num_write_kb":1543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4,5],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.88","version":"179'204","reported_seq":1618,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.018587+0000","last_change":"2023-01-15T14:35:59.022936+0000","last_active":"2023-01-15T14:35:59.018587+0000","last_peered":"2023-01-15T14:35:59.018587+0000","last_clean":"2023-01-15T14:35:59.018587+0000","last_became_active":"2023-01-15T14:35:55.305153+0000","last_became_peered":"2023-01-15T14:35:55.305153+0000","last_unstale":"2023-01-15T14:35:59.018587+0000","last_undegraded":"2023-01-15T14:35:59.018587+0000","last_fullsized":"2023-01-15T14:35:59.018587+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":8,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:34:48.269667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.269667+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.0048867820000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.87","version":"450'200","reported_seq":1465,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.957090+0000","last_change":"2023-01-15T14:36:00.957090+0000","last_active":"2023-01-15T14:36:00.957090+0000","last_peered":"2023-01-15T14:36:00.957090+0000","last_clean":"2023-01-15T14:36:00.957090+0000","last_became_active":"2023-01-15T14:36:00.435802+0000","last_became_peered":"2023-01-15T14:36:00.435802+0000","last_unstale":"2023-01-15T14:36:00.957090+0000","last_undegraded":"2023-01-15T14:36:00.957090+0000","last_fullsized":"2023-01-15T14:36:00.957090+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":8,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.417771381,"stat_sum":{"num_bytes":5689755,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":34583,"num_write":13,"num_write_kb":5140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1773172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.86","version":"102'52","reported_seq":359,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.000874+0000","last_change":"2023-01-15T14:35:57.000874+0000","last_active":"2023-01-15T14:35:57.000874+0000","last_peered":"2023-01-15T14:35:57.000874+0000","last_clean":"2023-01-15T14:35:57.000874+0000","last_became_active":"2023-01-15T14:35:46.362243+0000","last_became_peered":"2023-01-15T14:35:46.362243+0000","last_unstale":"2023-01-15T14:35:57.000874+0000","last_undegraded":"2023-01-15T14:35:57.000874+0000","last_fullsized":"2023-01-15T14:35:57.000874+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":8,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.92800982200000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.85","version":"467'216","reported_seq":1665,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.731844+0000","last_change":"2023-01-15T14:35:58.732195+0000","last_active":"2023-01-15T14:35:58.731844+0000","last_peered":"2023-01-15T14:35:58.731844+0000","last_clean":"2023-01-15T14:35:58.731844+0000","last_became_active":"2023-01-15T14:35:54.304302+0000","last_became_peered":"2023-01-15T14:35:54.304302+0000","last_unstale":"2023-01-15T14:35:58.731844+0000","last_undegraded":"2023-01-15T14:35:58.731844+0000","last_fullsized":"2023-01-15T14:35:58.731844+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":8,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.524455538,"stat_sum":{"num_bytes":7940183,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":13074,"num_write":17,"num_write_kb":4850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":26383411,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.84","version":"447'372","reported_seq":1934,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.618791+0000","last_change":"2023-01-15T14:35:58.618954+0000","last_active":"2023-01-15T14:35:58.618791+0000","last_peered":"2023-01-15T14:35:58.618791+0000","last_clean":"2023-01-15T14:35:58.618791+0000","last_became_active":"2023-01-15T14:35:46.409164+0000","last_became_peered":"2023-01-15T14:35:46.409164+0000","last_unstale":"2023-01-15T14:35:58.618791+0000","last_undegraded":"2023-01-15T14:35:58.618791+0000","last_fullsized":"2023-01-15T14:35:58.618791+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":8,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0.40473406200000001,"stat_sum":{"num_bytes":4170482,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":6,"num_read_kb":4895,"num_write":3,"num_write_kb":1888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2277448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.83","version":"444'351","reported_seq":1856,"reported_epoch":471,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:36:01.436373+0000","last_change":"2023-01-15T14:36:01.436440+0000","last_active":"2023-01-15T14:36:01.436373+0000","last_peered":"2023-01-15T14:36:01.436373+0000","last_clean":"2023-01-15T14:36:01.436373+0000","last_became_active":"2023-01-15T14:36:01.342026+0000","last_became_peered":"2023-01-15T14:36:01.342026+0000","last_unstale":"2023-01-15T14:36:01.436373+0000","last_undegraded":"2023-01-15T14:36:01.436373+0000","last_fullsized":"2023-01-15T14:36:01.436373+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":8,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.50929170800000001,"stat_sum":{"num_bytes":2605168,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":12727,"num_write":3,"num_write_kb":3665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":25328842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.82","version":"155'101","reported_seq":816,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.319807+0000","last_change":"2023-01-15T14:35:59.319924+0000","last_active":"2023-01-15T14:35:59.319807+0000","last_peered":"2023-01-15T14:35:59.319807+0000","last_clean":"2023-01-15T14:35:59.319807+0000","last_became_active":"2023-01-15T14:35:49.301959+0000","last_became_peered":"2023-01-15T14:35:49.301959+0000","last_unstale":"2023-01-15T14:35:59.319807+0000","last_undegraded":"2023-01-15T14:35:59.319807+0000","last_fullsized":"2023-01-15T14:35:59.319807+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":8,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.30445602199999999,"stat_sum":{"num_bytes":271017,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":817245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.81","version":"155'121","reported_seq":1013,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.003205+0000","last_change":"2023-01-15T14:35:57.548509+0000","last_active":"2023-01-15T14:35:58.003205+0000","last_peered":"2023-01-15T14:35:58.003205+0000","last_clean":"2023-01-15T14:35:58.003205+0000","last_became_active":"2023-01-15T14:35:49.400406+0000","last_became_peered":"2023-01-15T14:35:49.400406+0000","last_unstale":"2023-01-15T14:35:58.003205+0000","last_undegraded":"2023-01-15T14:35:58.003205+0000","last_fullsized":"2023-01-15T14:35:58.003205+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":8,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.55345092200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.80","version":"223'257","reported_seq":1239,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-01-15T14:35:57.713796+0000","last_change":"2023-01-15T14:35:57.714412+0000","last_active":"2023-01-15T14:35:57.713796+0000","last_peered":"2023-01-15T14:35:57.713796+0000","last_clean":"2023-01-15T14:35:57.713796+0000","last_became_active":"2023-01-15T14:35:49.398227+0000","last_became_peered":"2023-01-15T14:35:49.398227+0000","last_unstale":"2023-01-15T14:35:57.713796+0000","last_undegraded":"2023-01-15T14:35:57.713796+0000","last_fullsized":"2023-01-15T14:35:57.713796+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":8,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:35:01.091016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:01.091016+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55784813600000005,"objects_trimmed":0,"snaptrim_duration":0.71694233299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.7f","version":"460'275","reported_seq":1757,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.307768+0000","last_change":"2023-01-15T14:35:57.307826+0000","last_active":"2023-01-15T14:35:57.307768+0000","last_peered":"2023-01-15T14:35:57.307768+0000","last_clean":"2023-01-15T14:35:57.307768+0000","last_became_active":"2023-01-15T14:35:50.412810+0000","last_became_peered":"2023-01-15T14:35:50.412810+0000","last_unstale":"2023-01-15T14:35:57.307768+0000","last_undegraded":"2023-01-15T14:35:57.307768+0000","last_fullsized":"2023-01-15T14:35:57.307768+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":7,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.209750574,"stat_sum":{"num_bytes":11708859,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":34267,"num_write":18,"num_write_kb":7357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":23702706,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"1"}]},{"pgid":"3.7e","version":"467'90","reported_seq":324,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.524291+0000","last_change":"2023-01-15T14:35:58.524401+0000","last_active":"2023-01-15T14:35:58.524291+0000","last_peered":"2023-01-15T14:35:58.524291+0000","last_clean":"2023-01-15T14:35:58.524291+0000","last_became_active":"2023-01-15T14:35:46.257756+0000","last_became_peered":"2023-01-15T14:35:46.257756+0000","last_unstale":"2023-01-15T14:35:58.524291+0000","last_undegraded":"2023-01-15T14:35:58.524291+0000","last_fullsized":"2023-01-15T14:35:58.524291+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":7,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.30548204299999998,"stat_sum":{"num_bytes":3420307,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":3026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":15336751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.7d","version":"424'277","reported_seq":1722,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.730785+0000","last_change":"2023-01-15T14:35:58.731775+0000","last_active":"2023-01-15T14:35:58.730785+0000","last_peered":"2023-01-15T14:35:58.730785+0000","last_clean":"2023-01-15T14:35:58.730785+0000","last_became_active":"2023-01-15T14:35:24.190356+0000","last_became_peered":"2023-01-15T14:35:24.190356+0000","last_unstale":"2023-01-15T14:35:58.730785+0000","last_undegraded":"2023-01-15T14:35:58.730785+0000","last_fullsized":"2023-01-15T14:35:58.730785+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":7,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:57.389","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.52436749100000002,"stat_sum":{"num_bytes":2748011,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":18,"num_read_kb":20693,"num_write":10,"num_write_kb":4359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":7352992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.7c","version":"141'46","reported_seq":129,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.726665+0000","last_change":"2023-01-15T14:35:58.726739+0000","last_active":"2023-01-15T14:35:58.726665+0000","last_peered":"2023-01-15T14:35:58.726665+0000","last_clean":"2023-01-15T14:35:58.726665+0000","last_became_active":"2023-01-15T14:35:57.511542+0000","last_became_peered":"2023-01-15T14:35:57.511542+0000","last_unstale":"2023-01-15T14:35:58.726665+0000","last_undegraded":"2023-01-15T14:35:58.726665+0000","last_fullsized":"2023-01-15T14:35:58.726665+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.52141369900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.7b","version":"221'60","reported_seq":319,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.112464+0000","last_change":"2023-01-15T14:35:57.706627+0000","last_active":"2023-01-15T14:35:58.112464+0000","last_peered":"2023-01-15T14:35:58.112464+0000","last_clean":"2023-01-15T14:35:58.112464+0000","last_became_active":"2023-01-15T14:35:21.099347+0000","last_became_peered":"2023-01-15T14:35:21.099347+0000","last_unstale":"2023-01-15T14:35:58.112464+0000","last_undegraded":"2023-01-15T14:35:58.112464+0000","last_fullsized":"2023-01-15T14:35:58.112464+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.61179092199999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.7a","version":"223'238","reported_seq":1698,"reported_epoch":467,"state":"remapped+peering","last_fresh":"2023-01-15T14:35:56.997736+0000","last_change":"2023-01-15T14:35:56.998043+0000","last_active":"2023-01-15T14:35:56.015699+0000","last_peered":"2023-01-15T14:35:55.105078+0000","last_clean":"2023-01-15T14:35:55.105078+0000","last_became_active":"2023-01-15T14:35:24.294312+0000","last_became_peered":"2023-01-15T14:35:24.294312+0000","last_unstale":"2023-01-15T14:35:56.997736+0000","last_undegraded":"2023-01-15T14:35:56.997736+0000","last_fullsized":"2023-01-15T14:35:56.997736+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":7,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.41617046099999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.79","version":"467'469","reported_seq":1909,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.329879+0000","last_change":"2023-01-15T14:35:59.430813+0000","last_active":"2023-01-15T14:35:59.329879+0000","last_peered":"2023-01-15T14:35:59.329879+0000","last_clean":"2023-01-15T14:35:59.329879+0000","last_became_active":"2023-01-15T14:35:32.375313+0000","last_became_peered":"2023-01-15T14:35:32.375313+0000","last_unstale":"2023-01-15T14:35:59.329879+0000","last_undegraded":"2023-01-15T14:35:59.329879+0000","last_fullsized":"2023-01-15T14:35:59.329879+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.21031976599999999,"stat_sum":{"num_bytes":6815725,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":79994,"num_write":12,"num_write_kb":6932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":13431307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.78","version":"469'374","reported_seq":1999,"reported_epoch":469,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:59.428257+0000","last_change":"2023-01-15T14:35:59.428257+0000","last_active":"2023-01-15T14:35:59.428257+0000","last_peered":"2023-01-15T14:35:59.428257+0000","last_clean":"2023-01-15T14:35:59.428257+0000","last_became_active":"2023-01-15T14:35:30.253853+0000","last_became_peered":"2023-01-15T14:35:30.253853+0000","last_unstale":"2023-01-15T14:35:59.428257+0000","last_undegraded":"2023-01-15T14:35:59.428257+0000","last_fullsized":"2023-01-15T14:35:59.428257+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":374,"log_dups_size":0,"ondisk_log_size":374,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.41370238999999998,"stat_sum":{"num_bytes":5601676,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":27,"num_read_kb":21425,"num_write":29,"num_write_kb":8453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":18795610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.77","version":"200'78","reported_seq":642,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.728344+0000","last_change":"2023-01-15T14:35:58.729177+0000","last_active":"2023-01-15T14:35:58.728344+0000","last_peered":"2023-01-15T14:35:58.728344+0000","last_clean":"2023-01-15T14:35:58.728344+0000","last_became_active":"2023-01-15T14:35:35.101268+0000","last_became_peered":"2023-01-15T14:35:35.101268+0000","last_unstale":"2023-01-15T14:35:58.728344+0000","last_undegraded":"2023-01-15T14:35:58.728344+0000","last_fullsized":"2023-01-15T14:35:58.728344+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":7,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.62121211899999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.76","version":"194'81","reported_seq":115,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.306878+0000","last_change":"2023-01-15T14:35:57.306940+0000","last_active":"2023-01-15T14:35:57.306878+0000","last_peered":"2023-01-15T14:35:57.306878+0000","last_clean":"2023-01-15T14:35:57.306878+0000","last_became_active":"2023-01-15T14:35:21.302673+0000","last_became_peered":"2023-01-15T14:35:21.302673+0000","last_unstale":"2023-01-15T14:35:57.306878+0000","last_undegraded":"2023-01-15T14:35:57.306878+0000","last_fullsized":"2023-01-15T14:35:57.306878+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:34:51.296526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.296526+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.66934546500000003,"objects_trimmed":0,"snaptrim_duration":0.305816635,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.31","version":"204'150","reported_seq":1048,"reported_epoch":468,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:35:58.523974+0000","last_change":"2023-01-15T14:35:58.523804+0000","last_active":"2023-01-15T14:35:58.523974+0000","last_peered":"2023-01-15T14:35:58.523974+0000","last_clean":"2023-01-15T14:35:55.707831+0000","last_became_active":"2023-01-15T14:35:58.523804+0000","last_became_peered":"2023-01-15T14:35:58.523804+0000","last_unstale":"2023-01-15T14:35:58.523974+0000","last_undegraded":"2023-01-15T14:35:58.523974+0000","last_fullsized":"2023-01-15T14:35:58.217824+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:22.439","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.60698155499999995,"stat_sum":{"num_bytes":3558907,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10724,"num_write":3,"num_write_kb":3089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3605110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4],"avail_no_missing":["1","0","4","5"],"object_location_counts":[{"shards":"1,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.30","version":"206'252","reported_seq":1304,"reported_epoch":471,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:01.331366+0000","last_change":"2023-01-15T14:36:01.331924+0000","last_active":"2023-01-15T14:36:01.331366+0000","last_peered":"2023-01-15T14:36:01.331366+0000","last_clean":"2023-01-15T14:36:01.331366+0000","last_became_active":"2023-01-15T14:36:00.718985+0000","last_became_peered":"2023-01-15T14:36:00.718985+0000","last_unstale":"2023-01-15T14:36:01.331366+0000","last_undegraded":"2023-01-15T14:36:01.331366+0000","last_fullsized":"2023-01-15T14:36:01.331366+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:35:03.572774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:03.572774+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:14.331","scrub_duration":0.36120358800000002,"objects_trimmed":0,"snaptrim_duration":0.208089622,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":871,"reported_epoch":471,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:01.436328+0000","last_change":"2023-01-15T14:36:01.436414+0000","last_active":"2023-01-15T14:36:01.436328+0000","last_peered":"2023-01-15T14:36:01.436328+0000","last_clean":"2023-01-15T14:36:01.436328+0000","last_became_active":"2023-01-15T14:36:00.437003+0000","last_became_peered":"2023-01-15T14:36:00.437003+0000","last_unstale":"2023-01-15T14:36:01.436328+0000","last_undegraded":"2023-01-15T14:36:01.436328+0000","last_fullsized":"2023-01-15T14:36:01.436328+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50470070300000003,"objects_trimmed":0,"snaptrim_duration":0.39467190400000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"}]},{"pgid":"3.2e","version":"139'38","reported_seq":239,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.724502+0000","last_change":"2023-01-15T14:35:59.724673+0000","last_active":"2023-01-15T14:35:59.724502+0000","last_peered":"2023-01-15T14:35:59.724502+0000","last_clean":"2023-01-15T14:35:59.724502+0000","last_became_active":"2023-01-15T14:35:57.299974+0000","last_became_peered":"2023-01-15T14:35:57.299974+0000","last_unstale":"2023-01-15T14:35:59.724502+0000","last_undegraded":"2023-01-15T14:35:59.724502+0000","last_fullsized":"2023-01-15T14:35:59.724502+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:35:30.876307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:30.876307+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.60835654400000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1544,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.623004+0000","last_change":"2023-01-15T14:35:59.623172+0000","last_active":"2023-01-15T14:35:59.623004+0000","last_peered":"2023-01-15T14:35:59.623004+0000","last_clean":"2023-01-15T14:35:59.623004+0000","last_became_active":"2023-01-15T14:35:38.280808+0000","last_became_peered":"2023-01-15T14:35:38.280808+0000","last_unstale":"2023-01-15T14:35:59.623004+0000","last_undegraded":"2023-01-15T14:35:59.623004+0000","last_fullsized":"2023-01-15T14:35:59.623004+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:00.115","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.40665984900000002,"stat_sum":{"num_bytes":1516832,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2709,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":650,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.225485+0000","last_change":"2023-01-15T14:35:59.226292+0000","last_active":"2023-01-15T14:35:59.225485+0000","last_peered":"2023-01-15T14:35:59.225485+0000","last_clean":"2023-01-15T14:35:59.225485+0000","last_became_active":"2023-01-15T14:35:41.525322+0000","last_became_peered":"2023-01-15T14:35:41.525322+0000","last_unstale":"2023-01-15T14:35:59.225485+0000","last_undegraded":"2023-01-15T14:35:59.225485+0000","last_fullsized":"2023-01-15T14:35:59.225485+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:58.535965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:58.535965+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:00.841","scrub_duration":0.104214843,"objects_trimmed":0,"snaptrim_duration":0.20027524299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":353,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.059097+0000","last_change":"2023-01-15T14:35:57.059097+0000","last_active":"2023-01-15T14:35:57.059097+0000","last_peered":"2023-01-15T14:35:57.059097+0000","last_clean":"2023-01-15T14:35:57.059097+0000","last_became_active":"2023-01-15T14:34:56.436012+0000","last_became_peered":"2023-01-15T14:34:56.436012+0000","last_unstale":"2023-01-15T14:35:57.059097+0000","last_undegraded":"2023-01-15T14:35:57.059097+0000","last_fullsized":"2023-01-15T14:35:57.059097+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:32.542","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.82420926100000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.2a","version":"191'191","reported_seq":833,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-01-15T14:36:01.329270+0000","last_change":"2023-01-15T14:36:01.330084+0000","last_active":"2023-01-15T14:36:01.329270+0000","last_peered":"2023-01-15T14:36:01.329270+0000","last_clean":"2023-01-15T14:36:01.329270+0000","last_became_active":"2023-01-15T14:36:00.426291+0000","last_became_peered":"2023-01-15T14:36:00.426291+0000","last_unstale":"2023-01-15T14:36:01.329270+0000","last_undegraded":"2023-01-15T14:36:01.329270+0000","last_fullsized":"2023-01-15T14:36:01.329270+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.19251327600000001,"stat_sum":{"num_bytes":1518386,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":5930,"num_write":14,"num_write_kb":6501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9447726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.29","version":"110'149","reported_seq":816,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.122447+0000","last_change":"2023-01-15T14:35:59.122837+0000","last_active":"2023-01-15T14:35:59.122447+0000","last_peered":"2023-01-15T14:35:59.122447+0000","last_clean":"2023-01-15T14:35:59.122447+0000","last_became_active":"2023-01-15T14:35:11.345959+0000","last_became_peered":"2023-01-15T14:35:11.345959+0000","last_unstale":"2023-01-15T14:35:59.122447+0000","last_undegraded":"2023-01-15T14:35:59.122447+0000","last_fullsized":"2023-01-15T14:35:59.122447+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:34.348","scrub_duration":0.40398525099999999,"objects_trimmed":0,"snaptrim_duration":0.10366882299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.28","version":"333'248","reported_seq":1687,"reported_epoch":471,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:01.329045+0000","last_change":"2023-01-15T14:36:01.329184+0000","last_active":"2023-01-15T14:36:01.329045+0000","last_peered":"2023-01-15T14:36:01.329045+0000","last_clean":"2023-01-15T14:36:01.329045+0000","last_became_active":"2023-01-15T14:36:00.628513+0000","last_became_peered":"2023-01-15T14:36:00.628513+0000","last_unstale":"2023-01-15T14:36:01.329045+0000","last_undegraded":"2023-01-15T14:36:01.329045+0000","last_fullsized":"2023-01-15T14:36:01.329045+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'248","last_scrub_stamp":"2023-01-15T14:34:20.620068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.620068+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40448800800000001,"objects_trimmed":0,"snaptrim_duration":0.203623685,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":794,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.327763+0000","last_change":"2023-01-15T14:35:59.328467+0000","last_active":"2023-01-15T14:35:59.327763+0000","last_peered":"2023-01-15T14:35:59.327763+0000","last_clean":"2023-01-15T14:35:59.327763+0000","last_became_active":"2023-01-15T14:35:57.303183+0000","last_became_peered":"2023-01-15T14:35:57.303183+0000","last_unstale":"2023-01-15T14:35:59.327763+0000","last_undegraded":"2023-01-15T14:35:59.327763+0000","last_fullsized":"2023-01-15T14:35:59.327763+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.29949426899999998,"stat_sum":{"num_bytes":1242779,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":4228,"num_write":1,"num_write_kb":368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.26","version":"102'52","reported_seq":184,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.012852+0000","last_change":"2023-01-15T14:36:00.012852+0000","last_active":"2023-01-15T14:36:00.012852+0000","last_peered":"2023-01-15T14:36:00.012852+0000","last_clean":"2023-01-15T14:36:00.012852+0000","last_became_active":"2023-01-15T14:35:04.285753+0000","last_became_peered":"2023-01-15T14:35:04.285753+0000","last_unstale":"2023-01-15T14:36:00.012852+0000","last_undegraded":"2023-01-15T14:36:00.012852+0000","last_fullsized":"2023-01-15T14:36:00.012852+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:29.161","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.0014637649999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.25","version":"333'108","reported_seq":1312,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.732305+0000","last_change":"2023-01-15T14:35:58.732674+0000","last_active":"2023-01-15T14:35:58.732305+0000","last_peered":"2023-01-15T14:35:58.732305+0000","last_clean":"2023-01-15T14:35:58.732305+0000","last_became_active":"2023-01-15T14:35:38.285989+0000","last_became_peered":"2023-01-15T14:35:38.285989+0000","last_unstale":"2023-01-15T14:35:58.732305+0000","last_undegraded":"2023-01-15T14:35:58.732305+0000","last_fullsized":"2023-01-15T14:35:58.732305+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'108","last_scrub_stamp":"2023-01-15T14:35:02.975751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:02.975751+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:08.244","scrub_duration":0.30430094200000002,"objects_trimmed":0,"snaptrim_duration":0.52463558600000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.24","version":"467'308","reported_seq":1749,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.226432+0000","last_change":"2023-01-15T14:35:59.327177+0000","last_active":"2023-01-15T14:35:59.226432+0000","last_peered":"2023-01-15T14:35:59.226432+0000","last_clean":"2023-01-15T14:35:59.226432+0000","last_became_active":"2023-01-15T14:35:11.349162+0000","last_became_peered":"2023-01-15T14:35:11.349162+0000","last_unstale":"2023-01-15T14:35:59.226432+0000","last_undegraded":"2023-01-15T14:35:59.226432+0000","last_fullsized":"2023-01-15T14:35:59.226432+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:35.350","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.30097771400000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":20,"num_read_kb":22662,"num_write":11,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.23","version":"149'233","reported_seq":1027,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.058897+0000","last_change":"2023-01-15T14:35:57.058884+0000","last_active":"2023-01-15T14:35:57.058897+0000","last_peered":"2023-01-15T14:35:57.058897+0000","last_clean":"2023-01-15T14:35:57.058897+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:35:57.058897+0000","last_undegraded":"2023-01-15T14:35:57.058897+0000","last_fullsized":"2023-01-15T14:35:57.058897+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:34:55.674244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:55.674244+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:18.674","scrub_duration":0.50374311999999999,"objects_trimmed":0,"snaptrim_duration":0.62304353999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.22","version":"323'161","reported_seq":1466,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:01.058897+0000","last_change":"2023-01-15T14:36:01.058897+0000","last_active":"2023-01-15T14:36:01.058897+0000","last_peered":"2023-01-15T14:36:01.058897+0000","last_clean":"2023-01-15T14:36:01.058897+0000","last_became_active":"2023-01-15T14:36:00.521188+0000","last_became_peered":"2023-01-15T14:36:00.521188+0000","last_unstale":"2023-01-15T14:36:01.058897+0000","last_undegraded":"2023-01-15T14:36:01.058897+0000","last_fullsized":"2023-01-15T14:36:01.058897+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.43385756599999997,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.21","version":"321'138","reported_seq":1117,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.833897+0000","last_change":"2023-01-15T14:36:00.833897+0000","last_active":"2023-01-15T14:36:00.833897+0000","last_peered":"2023-01-15T14:36:00.833897+0000","last_clean":"2023-01-15T14:36:00.833897+0000","last_became_active":"2023-01-15T14:36:00.121542+0000","last_became_peered":"2023-01-15T14:36:00.121542+0000","last_unstale":"2023-01-15T14:36:00.833897+0000","last_undegraded":"2023-01-15T14:36:00.833897+0000","last_fullsized":"2023-01-15T14:36:00.833897+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"321'138","last_scrub_stamp":"2023-01-15T14:34:49.495975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.495975+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.71176880200000003,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6059,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.20","version":"324'273","reported_seq":1560,"reported_epoch":467,"state":"remapped+peering","last_fresh":"2023-01-15T14:35:56.998449+0000","last_change":"2023-01-15T14:35:56.998449+0000","last_active":"2023-01-15T14:35:56.237512+0000","last_peered":"2023-01-15T14:35:55.509873+0000","last_clean":"2023-01-15T14:35:55.509873+0000","last_became_active":"2023-01-15T14:35:41.521596+0000","last_became_peered":"2023-01-15T14:35:41.521596+0000","last_unstale":"2023-01-15T14:35:56.998449+0000","last_undegraded":"2023-01-15T14:35:56.998449+0000","last_fullsized":"2023-01-15T14:35:56.998449+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"324'273","last_scrub_stamp":"2023-01-15T14:34:49.100502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.100502+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204192292,"objects_trimmed":0,"snaptrim_duration":0.50428342900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.1f","version":"463'325","reported_seq":1821,"reported_epoch":467,"state":"remapped+peering","last_fresh":"2023-01-15T14:35:57.098556+0000","last_change":"2023-01-15T14:35:57.098896+0000","last_active":"2023-01-15T14:35:56.125368+0000","last_peered":"2023-01-15T14:35:55.715457+0000","last_clean":"2023-01-15T14:35:55.715457+0000","last_became_active":"2023-01-15T14:35:44.351971+0000","last_became_peered":"2023-01-15T14:35:44.351971+0000","last_unstale":"2023-01-15T14:35:57.098556+0000","last_undegraded":"2023-01-15T14:35:57.098556+0000","last_fullsized":"2023-01-15T14:35:57.098556+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":325,"log_dups_size":0,"ondisk_log_size":325,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:59.076","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.70970781699999996,"stat_sum":{"num_bytes":9370202,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":52,"num_read_kb":68771,"num_write":24,"num_write_kb":7801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":29785074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.1e","version":"215'39","reported_seq":1275,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.726142+0000","last_change":"2023-01-15T14:35:58.726210+0000","last_active":"2023-01-15T14:35:58.726142+0000","last_peered":"2023-01-15T14:35:58.726142+0000","last_clean":"2023-01-15T14:35:58.726142+0000","last_became_active":"2023-01-15T14:35:41.422705+0000","last_became_peered":"2023-01-15T14:35:41.422705+0000","last_unstale":"2023-01-15T14:35:58.726142+0000","last_undegraded":"2023-01-15T14:35:58.726142+0000","last_fullsized":"2023-01-15T14:35:58.726142+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.30571665999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1409,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.836890+0000","last_change":"2023-01-15T14:36:00.836890+0000","last_active":"2023-01-15T14:36:00.836890+0000","last_peered":"2023-01-15T14:36:00.836890+0000","last_clean":"2023-01-15T14:36:00.836890+0000","last_became_active":"2023-01-15T14:36:00.222343+0000","last_became_peered":"2023-01-15T14:36:00.222343+0000","last_unstale":"2023-01-15T14:36:00.836890+0000","last_undegraded":"2023-01-15T14:36:00.836890+0000","last_fullsized":"2023-01-15T14:36:00.836890+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":5,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:51.945865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.945865+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.61345776799999996,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":31,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.019682+0000","last_change":"2023-01-15T14:36:00.019682+0000","last_active":"2023-01-15T14:36:00.019682+0000","last_peered":"2023-01-15T14:36:00.019682+0000","last_clean":"2023-01-15T14:36:00.019682+0000","last_became_active":"2023-01-15T14:35:57.500868+0000","last_became_peered":"2023-01-15T14:35:57.500868+0000","last_unstale":"2023-01-15T14:36:00.019682+0000","last_undegraded":"2023-01-15T14:36:00.019682+0000","last_fullsized":"2023-01-15T14:36:00.019682+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":5,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:22.440826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:22.440826+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30343327599999997,"objects_trimmed":0,"snaptrim_duration":0.001455617,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.1b","version":"466'161","reported_seq":412,"reported_epoch":467,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:57.509384+0000","last_change":"2023-01-15T14:35:56.956593+0000","last_active":"2023-01-15T14:35:57.509384+0000","last_peered":"2023-01-15T14:35:57.509384+0000","last_clean":"2023-01-15T14:35:57.509384+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:35:57.509384+0000","last_undegraded":"2023-01-15T14:35:57.509384+0000","last_fullsized":"2023-01-15T14:35:57.509384+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"370'101","last_scrub_stamp":"2023-01-15T14:34:24.137270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:24.137270+0000","objects_scrubbed":3,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.106791863,"objects_trimmed":0,"snaptrim_duration":0.41102317399999999,"stat_sum":{"num_bytes":8660087,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":13301,"num_write":12,"num_write_kb":5069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.1a","version":"468'392","reported_seq":1873,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.330849+0000","last_change":"2023-01-15T14:35:59.331297+0000","last_active":"2023-01-15T14:35:59.330849+0000","last_peered":"2023-01-15T14:35:59.330849+0000","last_clean":"2023-01-15T14:35:59.330849+0000","last_became_active":"2023-01-15T14:35:44.351248+0000","last_became_peered":"2023-01-15T14:35:44.351248+0000","last_unstale":"2023-01-15T14:35:59.330849+0000","last_undegraded":"2023-01-15T14:35:59.330849+0000","last_fullsized":"2023-01-15T14:35:59.330849+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":5,"last_scrub":"375'356","last_scrub_stamp":"2023-01-15T14:34:29.460843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:29.460843+0000","objects_scrubbed":4,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30741503599999997,"objects_trimmed":0,"snaptrim_duration":0.31104477699999999,"stat_sum":{"num_bytes":6907842,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":16,"num_write_kb":5138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":9225015,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.19","version":"468'296","reported_seq":1649,"reported_epoch":468,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:58.620324+0000","last_change":"2023-01-15T14:35:58.620324+0000","last_active":"2023-01-15T14:35:58.620324+0000","last_peered":"2023-01-15T14:35:58.620324+0000","last_clean":"2023-01-15T14:35:58.620324+0000","last_became_active":"2023-01-15T14:34:43.263173+0000","last_became_peered":"2023-01-15T14:34:43.263173+0000","last_unstale":"2023-01-15T14:35:58.620324+0000","last_undegraded":"2023-01-15T14:35:58.620324+0000","last_fullsized":"2023-01-15T14:35:58.620324+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.0010643110000000001,"stat_sum":{"num_bytes":5393852,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":14,"num_read_kb":12321,"num_write":14,"num_write_kb":3780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4941423,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.18","version":"444'360","reported_seq":1895,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-01-15T14:35:57.612753+0000","last_change":"2023-01-15T14:35:57.713576+0000","last_active":"2023-01-15T14:35:57.612753+0000","last_peered":"2023-01-15T14:35:57.612753+0000","last_clean":"2023-01-15T14:35:57.612753+0000","last_became_active":"2023-01-15T14:35:11.544123+0000","last_became_peered":"2023-01-15T14:35:11.544123+0000","last_unstale":"2023-01-15T14:35:57.612753+0000","last_undegraded":"2023-01-15T14:35:57.612753+0000","last_fullsized":"2023-01-15T14:35:57.612753+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":5,"last_scrub":"384'341","last_scrub_stamp":"2023-01-15T14:34:35.477211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:35.477211+0000","objects_scrubbed":5,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.54215560600000001,"objects_trimmed":0,"snaptrim_duration":0.71649034099999998,"stat_sum":{"num_bytes":2974905,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":354,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.595437+0000","last_change":"2023-01-15T14:35:57.595437+0000","last_active":"2023-01-15T14:35:57.595437+0000","last_peered":"2023-01-15T14:35:57.595437+0000","last_clean":"2023-01-15T14:35:57.595437+0000","last_became_active":"2023-01-15T14:35:57.494559+0000","last_became_peered":"2023-01-15T14:35:57.494559+0000","last_unstale":"2023-01-15T14:35:57.595437+0000","last_undegraded":"2023-01-15T14:35:57.595437+0000","last_fullsized":"2023-01-15T14:35:57.595437+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:30.569","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.40686568200000001,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.16","version":"444'157","reported_seq":980,"reported_epoch":468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:58.523179+0000","last_change":"2023-01-15T14:35:58.523179+0000","last_active":"2023-01-15T14:35:58.523179+0000","last_peered":"2023-01-15T14:35:58.523179+0000","last_clean":"2023-01-15T14:35:34.287688+0000","last_became_active":"2023-01-15T14:35:58.422019+0000","last_became_peered":"2023-01-15T14:35:58.422019+0000","last_unstale":"2023-01-15T14:35:58.523179+0000","last_undegraded":"2023-01-15T14:35:58.118118+0000","last_fullsized":"2023-01-15T14:35:58.117497+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.70950577100000001,"stat_sum":{"num_bytes":5025425,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6898,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9731517,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[1,7],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":299,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.113296+0000","last_change":"2023-01-15T14:35:57.547246+0000","last_active":"2023-01-15T14:35:58.113296+0000","last_peered":"2023-01-15T14:35:58.113296+0000","last_clean":"2023-01-15T14:35:58.113296+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:35:58.113296+0000","last_undegraded":"2023-01-15T14:35:58.113296+0000","last_fullsized":"2023-01-15T14:35:58.113296+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.49245620600000001,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.14","version":"445'181","reported_seq":1544,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.327273+0000","last_change":"2023-01-15T14:35:59.327675+0000","last_active":"2023-01-15T14:35:59.327273+0000","last_peered":"2023-01-15T14:35:59.327273+0000","last_clean":"2023-01-15T14:35:59.327273+0000","last_became_active":"2023-01-15T14:35:14.050115+0000","last_became_peered":"2023-01-15T14:35:14.050115+0000","last_unstale":"2023-01-15T14:35:59.327273+0000","last_undegraded":"2023-01-15T14:35:59.327273+0000","last_fullsized":"2023-01-15T14:35:59.327273+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"351'159","last_scrub_stamp":"2023-01-15T14:34:18.607751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:18.607751+0000","objects_scrubbed":2,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40666956999999998,"objects_trimmed":0,"snaptrim_duration":0.30106295500000002,"stat_sum":{"num_bytes":2453855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2453855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":105,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.217443+0000","last_change":"2023-01-15T14:35:41.423046+0000","last_active":"2023-01-15T14:35:58.217443+0000","last_peered":"2023-01-15T14:35:58.217443+0000","last_clean":"2023-01-15T14:35:58.217443+0000","last_became_active":"2023-01-15T14:35:41.421509+0000","last_became_peered":"2023-01-15T14:35:41.421509+0000","last_unstale":"2023-01-15T14:35:58.217443+0000","last_undegraded":"2023-01-15T14:35:58.217443+0000","last_fullsized":"2023-01-15T14:35:58.217443+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":347,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.104274+0000","last_change":"2023-01-15T14:35:57.104247+0000","last_active":"2023-01-15T14:35:57.104274+0000","last_peered":"2023-01-15T14:35:57.104274+0000","last_clean":"2023-01-15T14:35:57.104274+0000","last_became_active":"2023-01-15T14:35:44.557253+0000","last_became_peered":"2023-01-15T14:35:44.557253+0000","last_unstale":"2023-01-15T14:35:57.104274+0000","last_undegraded":"2023-01-15T14:35:57.104274+0000","last_fullsized":"2023-01-15T14:35:57.104274+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.93086281500000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":96,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-01-15T14:35:57.305103+0000","last_change":"2023-01-15T14:35:04.384333+0000","last_active":"2023-01-15T14:35:57.305103+0000","last_peered":"2023-01-15T14:35:57.305103+0000","last_clean":"2023-01-15T14:35:57.305103+0000","last_became_active":"2023-01-15T14:35:04.383307+0000","last_became_peered":"2023-01-15T14:35:04.383307+0000","last_unstale":"2023-01-15T14:35:57.305103+0000","last_undegraded":"2023-01-15T14:35:57.305103+0000","last_fullsized":"2023-01-15T14:35:57.305103+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"467'212","reported_seq":1514,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.356939+0000","last_change":"2023-01-15T14:35:58.356939+0000","last_active":"2023-01-15T14:35:58.356939+0000","last_peered":"2023-01-15T14:35:58.356939+0000","last_clean":"2023-01-15T14:35:58.356939+0000","last_became_active":"2023-01-15T14:35:44.762050+0000","last_became_peered":"2023-01-15T14:35:44.762050+0000","last_unstale":"2023-01-15T14:35:58.356939+0000","last_undegraded":"2023-01-15T14:35:58.356939+0000","last_fullsized":"2023-01-15T14:35:58.356939+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.240279522,"stat_sum":{"num_bytes":6822268,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":34584,"num_write":17,"num_write_kb":6246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":1773171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":517,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.113465+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:35:58.113465+0000","last_peered":"2023-01-15T14:35:58.113465+0000","last_clean":"2023-01-15T14:35:58.113465+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:35:58.113465+0000","last_undegraded":"2023-01-15T14:35:58.113465+0000","last_fullsized":"2023-01-15T14:35:58.113465+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"468'388","reported_seq":1954,"reported_epoch":468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:58.411712+0000","last_change":"2023-01-15T14:35:58.411712+0000","last_active":"2023-01-15T14:35:58.411712+0000","last_peered":"2023-01-15T14:35:58.411712+0000","last_clean":"2023-01-15T14:35:55.249264+0000","last_became_active":"2023-01-15T14:35:58.408039+0000","last_became_peered":"2023-01-15T14:35:58.408039+0000","last_unstale":"2023-01-15T14:35:58.411712+0000","last_undegraded":"2023-01-15T14:35:58.213451+0000","last_fullsized":"2023-01-15T14:35:58.211795+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0.087195613000000005,"stat_sum":{"num_bytes":2119929,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":4894,"num_write":5,"num_write_kb":1887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":8887191,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1","objects":-3},{"shards":"0,1,6","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":86,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-01-15T14:35:57.208872+0000","last_change":"2023-01-15T14:34:43.371445+0000","last_active":"2023-01-15T14:35:57.208872+0000","last_peered":"2023-01-15T14:35:57.208872+0000","last_clean":"2023-01-15T14:35:57.208872+0000","last_became_active":"2023-01-15T14:34:43.253700+0000","last_became_peered":"2023-01-15T14:34:43.253700+0000","last_unstale":"2023-01-15T14:35:57.208872+0000","last_undegraded":"2023-01-15T14:35:57.208872+0000","last_fullsized":"2023-01-15T14:35:57.208872+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"443'207","reported_seq":1562,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.956125+0000","last_change":"2023-01-15T14:36:00.956125+0000","last_active":"2023-01-15T14:36:00.956125+0000","last_peered":"2023-01-15T14:36:00.956125+0000","last_clean":"2023-01-15T14:36:00.956125+0000","last_became_active":"2023-01-15T14:36:00.218840+0000","last_became_peered":"2023-01-15T14:36:00.218840+0000","last_unstale":"2023-01-15T14:36:00.956125+0000","last_undegraded":"2023-01-15T14:36:00.956125+0000","last_fullsized":"2023-01-15T14:36:00.956125+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.73554225699999998,"stat_sum":{"num_bytes":5687913,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":13074,"num_write":11,"num_write_kb":3965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":13282154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.3","version":"444'351","reported_seq":1840,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.328591+0000","last_change":"2023-01-15T14:35:59.329116+0000","last_active":"2023-01-15T14:35:59.328591+0000","last_peered":"2023-01-15T14:35:59.328591+0000","last_clean":"2023-01-15T14:35:59.328591+0000","last_became_active":"2023-01-15T14:35:44.252095+0000","last_became_peered":"2023-01-15T14:35:44.252095+0000","last_unstale":"2023-01-15T14:35:59.328591+0000","last_undegraded":"2023-01-15T14:35:59.328591+0000","last_fullsized":"2023-01-15T14:35:59.328591+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.208449256,"stat_sum":{"num_bytes":2605167,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12727,"num_write":3,"num_write_kb":3665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":14908172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.115575+0000","last_change":"2023-01-15T14:35:57.310164+0000","last_active":"2023-01-15T14:36:00.115575+0000","last_peered":"2023-01-15T14:36:00.115575+0000","last_clean":"2023-01-15T14:36:00.115575+0000","last_became_active":"2023-01-15T14:35:57.209780+0000","last_became_peered":"2023-01-15T14:35:57.209780+0000","last_unstale":"2023-01-15T14:36:00.115575+0000","last_undegraded":"2023-01-15T14:36:00.115575+0000","last_fullsized":"2023-01-15T14:36:00.115575+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1278,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.727771+0000","last_change":"2023-01-15T14:35:58.728184+0000","last_active":"2023-01-15T14:35:58.727771+0000","last_peered":"2023-01-15T14:35:58.727771+0000","last_clean":"2023-01-15T14:35:58.727771+0000","last_became_active":"2023-01-15T14:35:44.254173+0000","last_became_peered":"2023-01-15T14:35:44.254173+0000","last_unstale":"2023-01-15T14:35:58.727771+0000","last_undegraded":"2023-01-15T14:35:58.727771+0000","last_fullsized":"2023-01-15T14:35:58.727771+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:35:01.091016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:01.091016+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55784813600000005,"objects_trimmed":0,"snaptrim_duration":0.62124064400000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":74,"reported_epoch":466,"state":"active+clean","last_fresh":"2023-01-15T14:35:56.123236+0000","last_change":"2023-01-15T14:35:35.298341+0000","last_active":"2023-01-15T14:35:56.123236+0000","last_peered":"2023-01-15T14:35:56.123236+0000","last_clean":"2023-01-15T14:35:56.123236+0000","last_became_active":"2023-01-15T14:35:35.296662+0000","last_became_peered":"2023-01-15T14:35:35.296662+0000","last_unstale":"2023-01-15T14:35:56.123236+0000","last_undegraded":"2023-01-15T14:35:56.123236+0000","last_fullsized":"2023-01-15T14:35:56.123236+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":1003,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.003492+0000","last_change":"2023-01-15T14:35:57.003492+0000","last_active":"2023-01-15T14:35:57.003492+0000","last_peered":"2023-01-15T14:35:57.003492+0000","last_clean":"2023-01-15T14:35:57.003492+0000","last_became_active":"2023-01-15T14:35:44.457965+0000","last_became_peered":"2023-01-15T14:35:44.457965+0000","last_unstale":"2023-01-15T14:35:57.003492+0000","last_undegraded":"2023-01-15T14:35:57.003492+0000","last_fullsized":"2023-01-15T14:35:57.003492+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.63607082599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":535,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.007090+0000","last_change":"2023-01-15T14:34:43.269231+0000","last_active":"2023-01-15T14:35:58.007090+0000","last_peered":"2023-01-15T14:35:58.007090+0000","last_clean":"2023-01-15T14:35:58.007090+0000","last_became_active":"2023-01-15T14:34:43.268688+0000","last_became_peered":"2023-01-15T14:34:43.268688+0000","last_unstale":"2023-01-15T14:35:58.007090+0000","last_undegraded":"2023-01-15T14:35:58.007090+0000","last_fullsized":"2023-01-15T14:35:58.007090+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":808,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-01-15T14:36:01.160446+0000","last_change":"2023-01-15T14:36:01.161055+0000","last_active":"2023-01-15T14:36:01.160446+0000","last_peered":"2023-01-15T14:36:01.160446+0000","last_clean":"2023-01-15T14:36:01.160446+0000","last_became_active":"2023-01-15T14:36:00.324138+0000","last_became_peered":"2023-01-15T14:36:00.324138+0000","last_unstale":"2023-01-15T14:36:01.160446+0000","last_undegraded":"2023-01-15T14:36:01.160446+0000","last_fullsized":"2023-01-15T14:36:01.160446+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.101707193,"stat_sum":{"num_bytes":271017,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":817244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.015867+0000","last_change":"2023-01-15T14:35:57.399122+0000","last_active":"2023-01-15T14:36:00.015867+0000","last_peered":"2023-01-15T14:36:00.015867+0000","last_clean":"2023-01-15T14:36:00.015867+0000","last_became_active":"2023-01-15T14:35:57.398571+0000","last_became_peered":"2023-01-15T14:35:57.398571+0000","last_unstale":"2023-01-15T14:36:00.015867+0000","last_undegraded":"2023-01-15T14:36:00.015867+0000","last_fullsized":"2023-01-15T14:36:00.015867+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":668,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.002744+0000","last_change":"2023-01-15T14:35:26.786467+0000","last_active":"2023-01-15T14:35:58.002744+0000","last_peered":"2023-01-15T14:35:58.002744+0000","last_clean":"2023-01-15T14:35:58.002744+0000","last_became_active":"2023-01-15T14:35:26.685181+0000","last_became_peered":"2023-01-15T14:35:26.685181+0000","last_unstale":"2023-01-15T14:35:58.002744+0000","last_undegraded":"2023-01-15T14:35:58.002744+0000","last_fullsized":"2023-01-15T14:35:58.002744+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"179'204","reported_seq":1611,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.726765+0000","last_change":"2023-01-15T14:35:58.727503+0000","last_active":"2023-01-15T14:35:58.726765+0000","last_peered":"2023-01-15T14:35:58.726765+0000","last_clean":"2023-01-15T14:35:58.726765+0000","last_became_active":"2023-01-15T14:35:50.304954+0000","last_became_peered":"2023-01-15T14:35:50.304954+0000","last_unstale":"2023-01-15T14:35:58.726765+0000","last_undegraded":"2023-01-15T14:35:58.726765+0000","last_fullsized":"2023-01-15T14:35:58.726765+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:35:56.732493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:56.732493+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:58.010","scrub_duration":0.29859745199999999,"objects_trimmed":0,"snaptrim_duration":0.62093016199999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":1434,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.023028+0000","last_change":"2023-01-15T14:35:59.121730+0000","last_active":"2023-01-15T14:35:59.023028+0000","last_peered":"2023-01-15T14:35:59.023028+0000","last_clean":"2023-01-15T14:35:59.023028+0000","last_became_active":"2023-01-15T14:35:50.507792+0000","last_became_peered":"2023-01-15T14:35:50.507792+0000","last_unstale":"2023-01-15T14:35:59.023028+0000","last_undegraded":"2023-01-15T14:35:59.023028+0000","last_fullsized":"2023-01-15T14:35:59.023028+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.103268899,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.a","version":"252'279","reported_seq":1569,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.330073+0000","last_change":"2023-01-15T14:35:59.330770+0000","last_active":"2023-01-15T14:35:59.330073+0000","last_peered":"2023-01-15T14:35:59.330073+0000","last_clean":"2023-01-15T14:35:59.330073+0000","last_became_active":"2023-01-15T14:35:50.405112+0000","last_became_peered":"2023-01-15T14:35:50.405112+0000","last_unstale":"2023-01-15T14:35:59.330073+0000","last_undegraded":"2023-01-15T14:35:59.330073+0000","last_fullsized":"2023-01-15T14:35:59.330073+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.104749547,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":1123,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.058375+0000","last_change":"2023-01-15T14:35:57.058375+0000","last_active":"2023-01-15T14:35:57.058375+0000","last_peered":"2023-01-15T14:35:57.058375+0000","last_clean":"2023-01-15T14:35:57.058375+0000","last_became_active":"2023-01-15T14:35:50.506517+0000","last_became_peered":"2023-01-15T14:35:50.506517+0000","last_unstale":"2023-01-15T14:35:57.058375+0000","last_undegraded":"2023-01-15T14:35:57.058375+0000","last_fullsized":"2023-01-15T14:35:57.058375+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.62354816300000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.c","version":"451'156","reported_seq":1614,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.137816+0000","last_change":"2023-01-15T14:36:00.137816+0000","last_active":"2023-01-15T14:36:00.137816+0000","last_peered":"2023-01-15T14:36:00.137816+0000","last_clean":"2023-01-15T14:36:00.137816+0000","last_became_active":"2023-01-15T14:36:00.134582+0000","last_became_peered":"2023-01-15T14:36:00.134582+0000","last_unstale":"2023-01-15T14:36:00.137816+0000","last_undegraded":"2023-01-15T14:36:00.137816+0000","last_fullsized":"2023-01-15T14:36:00.137816+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.0023793410000000001,"stat_sum":{"num_bytes":3057636,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":33623,"num_write":8,"num_write_kb":1853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":14538648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.d","version":"296'232","reported_seq":1616,"reported_epoch":469,"state":"peering","last_fresh":"2023-01-15T14:35:59.114996+0000","last_change":"2023-01-15T14:35:59.115508+0000","last_active":"2023-01-15T14:35:59.114924+0000","last_peered":"2023-01-15T14:35:58.418181+0000","last_clean":"2023-01-15T14:35:55.008005+0000","last_became_active":"2023-01-15T14:35:58.416797+0000","last_became_peered":"2023-01-15T14:35:58.416797+0000","last_unstale":"2023-01-15T14:35:59.114996+0000","last_undegraded":"2023-01-15T14:35:59.114996+0000","last_fullsized":"2023-01-15T14:35:59.114996+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:45.258","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.81136931700000003,"stat_sum":{"num_bytes":2306688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":7820,"num_write":5,"num_write_kb":1640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5,6],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":338,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.425263+0000","last_change":"2023-01-15T14:35:59.425463+0000","last_active":"2023-01-15T14:35:59.425263+0000","last_peered":"2023-01-15T14:35:59.425263+0000","last_clean":"2023-01-15T14:35:59.425263+0000","last_became_active":"2023-01-15T14:35:50.309574+0000","last_became_peered":"2023-01-15T14:35:50.309574+0000","last_unstale":"2023-01-15T14:35:59.425263+0000","last_undegraded":"2023-01-15T14:35:59.425263+0000","last_fullsized":"2023-01-15T14:35:59.425263+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:57.164","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.40520778200000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1520,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.116915+0000","last_change":"2023-01-15T14:35:57.549186+0000","last_active":"2023-01-15T14:35:58.116915+0000","last_peered":"2023-01-15T14:35:58.116915+0000","last_clean":"2023-01-15T14:35:58.116915+0000","last_became_active":"2023-01-15T14:35:50.511018+0000","last_became_peered":"2023-01-15T14:35:50.511018+0000","last_unstale":"2023-01-15T14:35:58.116915+0000","last_undegraded":"2023-01-15T14:35:58.116915+0000","last_fullsized":"2023-01-15T14:35:58.116915+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.449450399,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.10","version":"454'443","reported_seq":1790,"reported_epoch":467,"state":"remapped+peering","last_fresh":"2023-01-15T14:35:56.996003+0000","last_change":"2023-01-15T14:35:56.996404+0000","last_active":"2023-01-15T14:35:56.035980+0000","last_peered":"2023-01-15T14:35:55.614381+0000","last_clean":"2023-01-15T14:35:55.614381+0000","last_became_active":"2023-01-15T14:35:50.309873+0000","last_became_peered":"2023-01-15T14:35:50.309873+0000","last_unstale":"2023-01-15T14:35:56.996003+0000","last_undegraded":"2023-01-15T14:35:56.996003+0000","last_fullsized":"2023-01-15T14:35:56.996003+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.50988759299999997,"stat_sum":{"num_bytes":4003030,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":31691,"num_write":27,"num_write_kb":9855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":10329425,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.11","version":"469'410","reported_seq":2034,"reported_epoch":469,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:59.726917+0000","last_change":"2023-01-15T14:35:59.726917+0000","last_active":"2023-01-15T14:35:59.726917+0000","last_peered":"2023-01-15T14:35:59.726917+0000","last_clean":"2023-01-15T14:35:59.726917+0000","last_became_active":"2023-01-15T14:35:50.408210+0000","last_became_peered":"2023-01-15T14:35:50.408210+0000","last_unstale":"2023-01-15T14:35:59.726917+0000","last_undegraded":"2023-01-15T14:35:59.726917+0000","last_fullsized":"2023-01-15T14:35:59.726917+0000","mapping_epoch":422,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":410,"log_dups_size":0,"ondisk_log_size":410,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:43.295","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.35434899399999997,"stat_sum":{"num_bytes":11458789,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":24145,"num_write":24,"num_write_kb":14137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":22433541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.12","version":"467'237","reported_seq":1233,"reported_epoch":467,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:35:57.714534+0000","last_change":"2023-01-15T14:35:57.714534+0000","last_active":"2023-01-15T14:35:57.714534+0000","last_peered":"2023-01-15T14:35:57.714534+0000","last_clean":"2023-01-15T14:35:57.714534+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:35:57.714534+0000","last_undegraded":"2023-01-15T14:35:57.714534+0000","last_fullsized":"2023-01-15T14:35:57.714534+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:35:58.076","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.40075952100000001,"stat_sum":{"num_bytes":9689420,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":6954,"num_write":25,"num_write_kb":12679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.13","version":"467'429","reported_seq":1794,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.712489+0000","last_change":"2023-01-15T14:35:58.712489+0000","last_active":"2023-01-15T14:35:58.712489+0000","last_peered":"2023-01-15T14:35:58.712489+0000","last_clean":"2023-01-15T14:35:58.712489+0000","last_became_active":"2023-01-15T14:35:41.221503+0000","last_became_peered":"2023-01-15T14:35:41.221503+0000","last_unstale":"2023-01-15T14:35:58.712489+0000","last_undegraded":"2023-01-15T14:35:58.712489+0000","last_fullsized":"2023-01-15T14:35:58.712489+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"374'376","last_scrub_stamp":"2023-01-15T14:34:32.489483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.489483+0000","objects_scrubbed":4,"log_size":429,"log_dups_size":0,"ondisk_log_size":429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.381937625,"objects_trimmed":0,"snaptrim_duration":0.59936913000000003,"stat_sum":{"num_bytes":13617083,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":29953,"num_write":8,"num_write_kb":4428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":13617083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.32","version":"464'212","reported_seq":1207,"reported_epoch":467,"state":"remapped+peering","last_fresh":"2023-01-15T14:35:56.998492+0000","last_change":"2023-01-15T14:35:56.998830+0000","last_active":"2023-01-15T14:35:56.025027+0000","last_peered":"2023-01-15T14:35:55.442927+0000","last_clean":"2023-01-15T14:35:55.442927+0000","last_became_active":"2023-01-15T14:34:56.393418+0000","last_became_peered":"2023-01-15T14:34:56.393418+0000","last_unstale":"2023-01-15T14:35:56.998492+0000","last_undegraded":"2023-01-15T14:35:56.998492+0000","last_fullsized":"2023-01-15T14:35:56.998492+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:29.581","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.43735380400000001,"stat_sum":{"num_bytes":7445043,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":24666,"num_write":11,"num_write_kb":3706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2663942,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,5],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.33","version":"204'275","reported_seq":1400,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.732758+0000","last_change":"2023-01-15T14:35:58.733306+0000","last_active":"2023-01-15T14:35:58.732758+0000","last_peered":"2023-01-15T14:35:58.732758+0000","last_clean":"2023-01-15T14:35:58.732758+0000","last_became_active":"2023-01-15T14:35:41.223122+0000","last_became_peered":"2023-01-15T14:35:41.223122+0000","last_unstale":"2023-01-15T14:35:58.732758+0000","last_undegraded":"2023-01-15T14:35:58.732758+0000","last_fullsized":"2023-01-15T14:35:58.732758+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.52455606200000005,"stat_sum":{"num_bytes":3485477,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11189,"num_write":7,"num_write_kb":3285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":1487,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.224493+0000","last_change":"2023-01-15T14:35:59.225262+0000","last_active":"2023-01-15T14:35:59.224493+0000","last_peered":"2023-01-15T14:35:59.224493+0000","last_clean":"2023-01-15T14:35:59.224493+0000","last_became_active":"2023-01-15T14:35:19.392381+0000","last_became_peered":"2023-01-15T14:35:19.392381+0000","last_unstale":"2023-01-15T14:35:59.224493+0000","last_undegraded":"2023-01-15T14:35:59.224493+0000","last_fullsized":"2023-01-15T14:35:59.224493+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:35:46.590173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:46.590173+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:52.600","scrub_duration":0.204901835,"objects_trimmed":0,"snaptrim_duration":0.19943985,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.35","version":"110'55","reported_seq":45,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.023266+0000","last_change":"2023-01-15T14:36:00.023266+0000","last_active":"2023-01-15T14:36:00.023266+0000","last_peered":"2023-01-15T14:36:00.023266+0000","last_clean":"2023-01-15T14:36:00.023266+0000","last_became_active":"2023-01-15T14:35:57.301471+0000","last_became_peered":"2023-01-15T14:35:57.301471+0000","last_unstale":"2023-01-15T14:36:00.023266+0000","last_undegraded":"2023-01-15T14:36:00.023266+0000","last_fullsized":"2023-01-15T14:36:00.023266+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:35:58.219589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:58.219589+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:37:26.346","scrub_duration":0.44852532699999997,"objects_trimmed":0,"snaptrim_duration":0.000944519,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.36","version":"194'81","reported_seq":31,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.021848+0000","last_change":"2023-01-15T14:36:00.021848+0000","last_active":"2023-01-15T14:36:00.021848+0000","last_peered":"2023-01-15T14:36:00.021848+0000","last_clean":"2023-01-15T14:36:00.021848+0000","last_became_active":"2023-01-15T14:35:57.305669+0000","last_became_peered":"2023-01-15T14:35:57.305669+0000","last_unstale":"2023-01-15T14:36:00.021848+0000","last_undegraded":"2023-01-15T14:36:00.021848+0000","last_fullsized":"2023-01-15T14:36:00.021848+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:34:51.296526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.296526+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.66934546500000003,"objects_trimmed":0,"snaptrim_duration":0.00087794100000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.37","version":"200'78","reported_seq":725,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.317467+0000","last_change":"2023-01-15T14:35:57.704658+0000","last_active":"2023-01-15T14:35:58.317467+0000","last_peered":"2023-01-15T14:35:58.317467+0000","last_clean":"2023-01-15T14:35:58.317467+0000","last_became_active":"2023-01-15T14:35:57.603486+0000","last_became_peered":"2023-01-15T14:35:57.603486+0000","last_unstale":"2023-01-15T14:35:58.317467+0000","last_undegraded":"2023-01-15T14:35:58.317467+0000","last_fullsized":"2023-01-15T14:35:58.317467+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.30959137399999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.38","version":"426'347","reported_seq":1789,"reported_epoch":471,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:01.533883+0000","last_change":"2023-01-15T14:36:01.534088+0000","last_active":"2023-01-15T14:36:01.533883+0000","last_peered":"2023-01-15T14:36:01.533883+0000","last_clean":"2023-01-15T14:36:01.533883+0000","last_became_active":"2023-01-15T14:36:00.626409+0000","last_became_peered":"2023-01-15T14:36:00.626409+0000","last_unstale":"2023-01-15T14:36:01.533883+0000","last_undegraded":"2023-01-15T14:36:01.533883+0000","last_fullsized":"2023-01-15T14:36:01.533883+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.40823593800000002,"stat_sum":{"num_bytes":4230716,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":21424,"num_write":20,"num_write_kb":6727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":16442530,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.39","version":"423'435","reported_seq":1881,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.518791+0000","last_change":"2023-01-15T14:35:59.519018+0000","last_active":"2023-01-15T14:35:59.518791+0000","last_peered":"2023-01-15T14:35:59.518791+0000","last_clean":"2023-01-15T14:35:59.518791+0000","last_became_active":"2023-01-15T14:35:38.280557+0000","last_became_peered":"2023-01-15T14:35:38.280557+0000","last_unstale":"2023-01-15T14:35:59.518791+0000","last_undegraded":"2023-01-15T14:35:59.518791+0000","last_fullsized":"2023-01-15T14:35:59.518791+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.50163052399999997,"stat_sum":{"num_bytes":17098873,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":34,"num_read_kb":57397,"num_write":12,"num_write_kb":6931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":13431306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1705,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-01-15T14:36:01.325386+0000","last_change":"2023-01-15T14:36:01.326176+0000","last_active":"2023-01-15T14:36:01.325386+0000","last_peered":"2023-01-15T14:36:01.325386+0000","last_clean":"2023-01-15T14:36:01.325386+0000","last_became_active":"2023-01-15T14:36:00.527253+0000","last_became_peered":"2023-01-15T14:36:00.527253+0000","last_unstale":"2023-01-15T14:36:01.325386+0000","last_undegraded":"2023-01-15T14:36:01.325386+0000","last_fullsized":"2023-01-15T14:36:01.325386+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.16648763,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.3b","version":"221'60","reported_seq":326,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.426354+0000","last_change":"2023-01-15T14:35:59.426528+0000","last_active":"2023-01-15T14:35:59.426354+0000","last_peered":"2023-01-15T14:35:59.426354+0000","last_clean":"2023-01-15T14:35:59.426354+0000","last_became_active":"2023-01-15T14:35:35.100811+0000","last_became_peered":"2023-01-15T14:35:35.100811+0000","last_unstale":"2023-01-15T14:35:59.426354+0000","last_undegraded":"2023-01-15T14:35:59.426354+0000","last_fullsized":"2023-01-15T14:35:59.426354+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.40528648699999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":39,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.020154+0000","last_change":"2023-01-15T14:36:00.020154+0000","last_active":"2023-01-15T14:36:00.020154+0000","last_peered":"2023-01-15T14:36:00.020154+0000","last_clean":"2023-01-15T14:36:00.020154+0000","last_became_active":"2023-01-15T14:35:57.196630+0000","last_became_peered":"2023-01-15T14:35:57.196630+0000","last_unstale":"2023-01-15T14:36:00.020154+0000","last_undegraded":"2023-01-15T14:36:00.020154+0000","last_fullsized":"2023-01-15T14:36:00.020154+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.0015976619999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.3d","version":"462'310","reported_seq":1809,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.099918+0000","last_change":"2023-01-15T14:35:57.096205+0000","last_active":"2023-01-15T14:35:57.099918+0000","last_peered":"2023-01-15T14:35:57.099918+0000","last_clean":"2023-01-15T14:35:57.099918+0000","last_became_active":"2023-01-15T14:35:44.656577+0000","last_became_peered":"2023-01-15T14:35:44.656577+0000","last_unstale":"2023-01-15T14:35:57.099918+0000","last_undegraded":"2023-01-15T14:35:57.099918+0000","last_fullsized":"2023-01-15T14:35:57.099918+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.92734123400000001,"stat_sum":{"num_bytes":7672659,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":28971,"num_write":15,"num_write_kb":5220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":14950019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.3e","version":"448'86","reported_seq":289,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.525166+0000","last_change":"2023-01-15T14:35:58.525206+0000","last_active":"2023-01-15T14:35:58.525166+0000","last_peered":"2023-01-15T14:35:58.525166+0000","last_clean":"2023-01-15T14:35:58.525166+0000","last_became_active":"2023-01-15T14:35:44.355920+0000","last_became_peered":"2023-01-15T14:35:44.355920+0000","last_unstale":"2023-01-15T14:35:58.525166+0000","last_undegraded":"2023-01-15T14:35:58.525166+0000","last_fullsized":"2023-01-15T14:35:58.525166+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.30598364700000003,"stat_sum":{"num_bytes":3420307,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":3025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":8496136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.3f","version":"468'279","reported_seq":1756,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.220859+0000","last_change":"2023-01-15T14:35:59.221019+0000","last_active":"2023-01-15T14:35:59.220859+0000","last_peered":"2023-01-15T14:35:59.220859+0000","last_clean":"2023-01-15T14:35:59.220859+0000","last_became_active":"2023-01-15T14:35:44.661557+0000","last_became_peered":"2023-01-15T14:35:44.661557+0000","last_unstale":"2023-01-15T14:35:59.220859+0000","last_undegraded":"2023-01-15T14:35:59.220859+0000","last_fullsized":"2023-01-15T14:35:59.220859+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.103138536,"stat_sum":{"num_bytes":11708858,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":14973,"num_write":18,"num_write_kb":7357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":13214874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"2"}]},{"pgid":"3.40","version":"465'361","reported_seq":1774,"reported_epoch":468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:58.618418+0000","last_change":"2023-01-15T14:35:58.618418+0000","last_active":"2023-01-15T14:35:58.618418+0000","last_peered":"2023-01-15T14:35:58.618418+0000","last_clean":"2023-01-15T14:35:56.059614+0000","last_became_active":"2023-01-15T14:35:58.418109+0000","last_became_peered":"2023-01-15T14:35:58.418109+0000","last_unstale":"2023-01-15T14:35:58.618418+0000","last_undegraded":"2023-01-15T14:35:58.006773+0000","last_fullsized":"2023-01-15T14:35:58.005697+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":361,"log_dups_size":0,"ondisk_log_size":361,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.70691918099999995,"stat_sum":{"num_bytes":12637268,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":35543,"num_write":40,"num_write_kb":14039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":43425481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6],"avail_no_missing":["7","0","6"],"object_location_counts":[{"shards":"0,6,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"}]},{"pgid":"3.41","version":"155'121","reported_seq":1116,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.748970+0000","last_change":"2023-01-15T14:36:00.748970+0000","last_active":"2023-01-15T14:36:00.748970+0000","last_peered":"2023-01-15T14:36:00.748970+0000","last_clean":"2023-01-15T14:36:00.748970+0000","last_became_active":"2023-01-15T14:36:00.224181+0000","last_became_peered":"2023-01-15T14:36:00.224181+0000","last_unstale":"2023-01-15T14:36:00.748970+0000","last_undegraded":"2023-01-15T14:36:00.748970+0000","last_fullsized":"2023-01-15T14:36:00.748970+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.423709108,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":1044,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:56.999391+0000","last_change":"2023-01-15T14:35:56.999391+0000","last_active":"2023-01-15T14:35:56.999391+0000","last_peered":"2023-01-15T14:35:56.999391+0000","last_clean":"2023-01-15T14:35:56.999391+0000","last_became_active":"2023-01-15T14:35:41.516587+0000","last_became_peered":"2023-01-15T14:35:41.516587+0000","last_unstale":"2023-01-15T14:35:56.999391+0000","last_undegraded":"2023-01-15T14:35:56.999391+0000","last_fullsized":"2023-01-15T14:35:56.999391+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:34:57.046809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:57.046809+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.50012570499999998,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.43","version":"459'354","reported_seq":1814,"reported_epoch":467,"state":"remapped+peering","last_fresh":"2023-01-15T14:35:57.103085+0000","last_change":"2023-01-15T14:35:57.103375+0000","last_active":"2023-01-15T14:35:56.023739+0000","last_peered":"2023-01-15T14:35:55.414086+0000","last_clean":"2023-01-15T14:35:55.414086+0000","last_became_active":"2023-01-15T14:35:41.425468+0000","last_became_peered":"2023-01-15T14:35:41.425468+0000","last_unstale":"2023-01-15T14:35:57.103085+0000","last_undegraded":"2023-01-15T14:35:57.103085+0000","last_fullsized":"2023-01-15T14:35:57.103085+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":0,"snaptrim_duration":0.20877104399999999,"stat_sum":{"num_bytes":6185141,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":13,"num_read_kb":16586,"num_write":12,"num_write_kb":5193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12504705,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[5,6,7],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.44","version":"245'268","reported_seq":1478,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.122918+0000","last_change":"2023-01-15T14:35:59.123430+0000","last_active":"2023-01-15T14:35:59.122918+0000","last_peered":"2023-01-15T14:35:59.122918+0000","last_clean":"2023-01-15T14:35:59.122918+0000","last_became_active":"2023-01-15T14:35:41.325427+0000","last_became_peered":"2023-01-15T14:35:41.325427+0000","last_unstale":"2023-01-15T14:35:59.122918+0000","last_undegraded":"2023-01-15T14:35:59.122918+0000","last_fullsized":"2023-01-15T14:35:59.122918+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.103500246,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.45","version":"253'125","reported_seq":1223,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.096754+0000","last_change":"2023-01-15T14:35:57.096754+0000","last_active":"2023-01-15T14:35:57.096754+0000","last_peered":"2023-01-15T14:35:57.096754+0000","last_clean":"2023-01-15T14:35:57.096754+0000","last_became_active":"2023-01-15T14:35:38.500236+0000","last_became_peered":"2023-01-15T14:35:38.500236+0000","last_unstale":"2023-01-15T14:35:57.096754+0000","last_undegraded":"2023-01-15T14:35:57.096754+0000","last_fullsized":"2023-01-15T14:35:57.096754+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:34:50.381073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:50.381073+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:05.295","scrub_duration":0.52191215300000005,"objects_trimmed":0,"snaptrim_duration":0.83296549799999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.46","version":"102'52","reported_seq":219,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.624855+0000","last_change":"2023-01-15T14:35:58.624900+0000","last_active":"2023-01-15T14:35:58.624855+0000","last_peered":"2023-01-15T14:35:58.624855+0000","last_clean":"2023-01-15T14:35:58.624855+0000","last_became_active":"2023-01-15T14:35:04.373495+0000","last_became_peered":"2023-01-15T14:35:04.373495+0000","last_unstale":"2023-01-15T14:35:58.624855+0000","last_undegraded":"2023-01-15T14:35:58.624855+0000","last_fullsized":"2023-01-15T14:35:58.624855+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:21.428","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.406021723,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2142,"num_write":3,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.47","version":"245'112","reported_seq":1420,"reported_epoch":469,"state":"peering","last_fresh":"2023-01-15T14:35:59.027555+0000","last_change":"2023-01-15T14:35:59.027812+0000","last_active":"2023-01-15T14:35:59.027514+0000","last_peered":"2023-01-15T14:35:58.327183+0000","last_clean":"2023-01-15T14:35:55.214754+0000","last_became_active":"2023-01-15T14:35:58.326335+0000","last_became_peered":"2023-01-15T14:35:58.326335+0000","last_unstale":"2023-01-15T14:35:59.027555+0000","last_undegraded":"2023-01-15T14:35:59.027555+0000","last_fullsized":"2023-01-15T14:35:59.027555+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:34:21.450320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:21.450320+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30497759099999999,"objects_trimmed":0,"snaptrim_duration":0.204948255,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5,7],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.48","version":"179'204","reported_seq":1480,"reported_epoch":471,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:01.327957+0000","last_change":"2023-01-15T14:36:01.328097+0000","last_active":"2023-01-15T14:36:01.327957+0000","last_peered":"2023-01-15T14:36:01.327957+0000","last_clean":"2023-01-15T14:36:01.327957+0000","last_became_active":"2023-01-15T14:36:00.627683+0000","last_became_peered":"2023-01-15T14:36:00.627683+0000","last_unstale":"2023-01-15T14:36:01.327957+0000","last_undegraded":"2023-01-15T14:36:01.327957+0000","last_fullsized":"2023-01-15T14:36:01.327957+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.20317748699999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.49","version":"110'149","reported_seq":1235,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.322058+0000","last_change":"2023-01-15T14:35:59.322218+0000","last_active":"2023-01-15T14:35:59.322058+0000","last_peered":"2023-01-15T14:35:59.322058+0000","last_clean":"2023-01-15T14:35:59.322058+0000","last_became_active":"2023-01-15T14:35:38.285855+0000","last_became_peered":"2023-01-15T14:35:38.285855+0000","last_unstale":"2023-01-15T14:35:59.322058+0000","last_undegraded":"2023-01-15T14:35:59.322058+0000","last_fullsized":"2023-01-15T14:35:59.322058+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.30585161799999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.4a","version":"468'438","reported_seq":1899,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.417495+0000","last_change":"2023-01-15T14:35:59.417673+0000","last_active":"2023-01-15T14:35:59.417495+0000","last_peered":"2023-01-15T14:35:59.417495+0000","last_clean":"2023-01-15T14:35:59.417495+0000","last_became_active":"2023-01-15T14:35:22.264683+0000","last_became_peered":"2023-01-15T14:35:22.264683+0000","last_unstale":"2023-01-15T14:35:59.417495+0000","last_undegraded":"2023-01-15T14:35:59.417495+0000","last_fullsized":"2023-01-15T14:35:59.417495+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:34.189","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.29885415100000001,"stat_sum":{"num_bytes":15111564,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":84,"num_read_kb":95190,"num_write":88,"num_write_kb":26766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":39901450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"}]},{"pgid":"3.4b","version":"155'53","reported_seq":131,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.625404+0000","last_change":"2023-01-15T14:35:58.625451+0000","last_active":"2023-01-15T14:35:58.625404+0000","last_peered":"2023-01-15T14:35:58.625404+0000","last_clean":"2023-01-15T14:35:58.625404+0000","last_became_active":"2023-01-15T14:35:57.397539+0000","last_became_peered":"2023-01-15T14:35:57.397539+0000","last_unstale":"2023-01-15T14:35:58.625404+0000","last_undegraded":"2023-01-15T14:35:58.625404+0000","last_fullsized":"2023-01-15T14:35:58.625404+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:34:44.116779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:44.116779+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50673716899999999,"objects_trimmed":0,"snaptrim_duration":0.41758954799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.4c","version":"309'71","reported_seq":399,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.423204+0000","last_change":"2023-01-15T14:35:59.423381+0000","last_active":"2023-01-15T14:35:59.423204+0000","last_peered":"2023-01-15T14:35:59.423204+0000","last_clean":"2023-01-15T14:35:59.423204+0000","last_became_active":"2023-01-15T14:35:04.379289+0000","last_became_peered":"2023-01-15T14:35:04.379289+0000","last_unstale":"2023-01-15T14:35:59.423204+0000","last_undegraded":"2023-01-15T14:35:59.423204+0000","last_fullsized":"2023-01-15T14:35:59.423204+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:47.266","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.406683091,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.4d","version":"463'343","reported_seq":1712,"reported_epoch":468,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:35:58.837054+0000","last_change":"2023-01-15T14:35:58.626587+0000","last_active":"2023-01-15T14:35:58.837054+0000","last_peered":"2023-01-15T14:35:58.837054+0000","last_clean":"2023-01-15T14:35:53.810817+0000","last_became_active":"2023-01-15T14:35:58.514991+0000","last_became_peered":"2023-01-15T14:35:58.514991+0000","last_unstale":"2023-01-15T14:35:58.837054+0000","last_undegraded":"2023-01-15T14:35:58.837054+0000","last_fullsized":"2023-01-15T14:35:58.013791+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.49040107999999999,"stat_sum":{"num_bytes":12395770,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":15962,"num_write":27,"num_write_kb":9640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":32418803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[3,7],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":6},{"shards":"3,7","objects":-2}],"blocked_by":[],"up_primary":5,"acting_primary":3,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.4e","version":"139'38","reported_seq":388,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.004178+0000","last_change":"2023-01-15T14:35:57.004178+0000","last_active":"2023-01-15T14:35:57.004178+0000","last_peered":"2023-01-15T14:35:57.004178+0000","last_clean":"2023-01-15T14:35:57.004178+0000","last_became_active":"2023-01-15T14:35:35.391600+0000","last_became_peered":"2023-01-15T14:35:35.391600+0000","last_unstale":"2023-01-15T14:35:57.004178+0000","last_undegraded":"2023-01-15T14:35:57.004178+0000","last_fullsized":"2023-01-15T14:35:57.004178+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.93167261499999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.4f","version":"155'145","reported_seq":1135,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.954324+0000","last_change":"2023-01-15T14:36:00.954324+0000","last_active":"2023-01-15T14:36:00.954324+0000","last_peered":"2023-01-15T14:36:00.954324+0000","last_clean":"2023-01-15T14:36:00.954324+0000","last_became_active":"2023-01-15T14:36:00.415211+0000","last_became_peered":"2023-01-15T14:36:00.415211+0000","last_unstale":"2023-01-15T14:36:00.954324+0000","last_undegraded":"2023-01-15T14:36:00.954324+0000","last_fullsized":"2023-01-15T14:36:00.954324+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51591872999999999,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.50","version":"453'498","reported_seq":1942,"reported_epoch":468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:58.621726+0000","last_change":"2023-01-15T14:35:58.621726+0000","last_active":"2023-01-15T14:35:58.621726+0000","last_peered":"2023-01-15T14:35:58.621726+0000","last_clean":"2023-01-15T14:35:55.005690+0000","last_became_active":"2023-01-15T14:35:58.619369+0000","last_became_peered":"2023-01-15T14:35:58.619369+0000","last_unstale":"2023-01-15T14:35:58.621726+0000","last_undegraded":"2023-01-15T14:35:58.004926+0000","last_fullsized":"2023-01-15T14:35:58.003805+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:51.263","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.81950787300000005,"stat_sum":{"num_bytes":10573847,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":92165,"num_write":64,"num_write_kb":23394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":40073843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":6},{"shards":"0,7","objects":-3}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.51","version":"455'353","reported_seq":1891,"reported_epoch":468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:58.424107+0000","last_change":"2023-01-15T14:35:58.424514+0000","last_active":"2023-01-15T14:35:58.424107+0000","last_peered":"2023-01-15T14:35:58.424107+0000","last_clean":"2023-01-15T14:35:55.101917+0000","last_became_active":"2023-01-15T14:35:58.423581+0000","last_became_peered":"2023-01-15T14:35:58.423581+0000","last_unstale":"2023-01-15T14:35:58.424107+0000","last_undegraded":"2023-01-15T14:35:58.320157+0000","last_fullsized":"2023-01-15T14:35:58.318929+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.51849322600000003,"stat_sum":{"num_bytes":6386825,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":2393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":8487799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,0],"avail_no_missing":["1","0","3"],"object_location_counts":[{"shards":"0,1,3","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":956,"reported_epoch":469,"state":"peering","last_fresh":"2023-01-15T14:35:59.017582+0000","last_change":"2023-01-15T14:35:59.017941+0000","last_active":"2023-01-15T14:35:59.017524+0000","last_peered":"2023-01-15T14:35:58.218856+0000","last_clean":"2023-01-15T14:35:55.009927+0000","last_became_active":"2023-01-15T14:35:58.107780+0000","last_became_peered":"2023-01-15T14:35:58.107780+0000","last_unstale":"2023-01-15T14:35:59.017582+0000","last_undegraded":"2023-01-15T14:35:59.017582+0000","last_fullsized":"2023-01-15T14:35:59.017582+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.98206584500000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5,6],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.53","version":"466'540","reported_seq":2005,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.009378+0000","last_change":"2023-01-15T14:35:57.705583+0000","last_active":"2023-01-15T14:35:58.009378+0000","last_peered":"2023-01-15T14:35:58.009378+0000","last_clean":"2023-01-15T14:35:58.009378+0000","last_became_active":"2023-01-15T14:35:41.426783+0000","last_became_peered":"2023-01-15T14:35:41.426783+0000","last_unstale":"2023-01-15T14:35:58.009378+0000","last_undegraded":"2023-01-15T14:35:58.009378+0000","last_fullsized":"2023-01-15T14:35:58.009378+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.61228132899999999,"stat_sum":{"num_bytes":23459187,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":76,"num_read_kb":75188,"num_write":61,"num_write_kb":30769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":72326676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.54","version":"245'148","reported_seq":1416,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.001225+0000","last_change":"2023-01-15T14:35:57.001225+0000","last_active":"2023-01-15T14:35:57.001225+0000","last_peered":"2023-01-15T14:35:57.001225+0000","last_clean":"2023-01-15T14:35:57.001225+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:35:57.001225+0000","last_undegraded":"2023-01-15T14:35:57.001225+0000","last_fullsized":"2023-01-15T14:35:57.001225+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:34:47.392699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.392699+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:14.435","scrub_duration":0.50581767899999996,"objects_trimmed":0,"snaptrim_duration":0.63713540800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.55","version":"110'55","reported_seq":1119,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.095635+0000","last_change":"2023-01-15T14:35:57.095635+0000","last_active":"2023-01-15T14:35:57.095635+0000","last_peered":"2023-01-15T14:35:57.095635+0000","last_clean":"2023-01-15T14:35:57.095635+0000","last_became_active":"2023-01-15T14:35:11.347080+0000","last_became_peered":"2023-01-15T14:35:11.347080+0000","last_unstale":"2023-01-15T14:35:57.095635+0000","last_undegraded":"2023-01-15T14:35:57.095635+0000","last_fullsized":"2023-01-15T14:35:57.095635+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:34:59.815570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:59.815570+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.93349607499999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.56","version":"320'127","reported_seq":908,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.427643+0000","last_change":"2023-01-15T14:35:59.427817+0000","last_active":"2023-01-15T14:35:59.427643+0000","last_peered":"2023-01-15T14:35:59.427643+0000","last_clean":"2023-01-15T14:35:59.427643+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:35:59.427643+0000","last_undegraded":"2023-01-15T14:35:59.427643+0000","last_fullsized":"2023-01-15T14:35:59.427643+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.40733391200000002,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.57","version":"462'130","reported_seq":506,"reported_epoch":467,"state":"remapped+peering","last_fresh":"2023-01-15T14:35:57.096873+0000","last_change":"2023-01-15T14:35:57.097106+0000","last_active":"2023-01-15T14:35:56.134039+0000","last_peered":"2023-01-15T14:35:55.412961+0000","last_clean":"2023-01-15T14:35:55.412961+0000","last_became_active":"2023-01-15T14:35:40.183681+0000","last_became_peered":"2023-01-15T14:35:40.183681+0000","last_unstale":"2023-01-15T14:35:57.096873+0000","last_undegraded":"2023-01-15T14:35:57.096873+0000","last_fullsized":"2023-01-15T14:35:57.096873+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'100","last_scrub_stamp":"2023-01-15T14:34:23.447852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.447852+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.52101476099999999,"objects_trimmed":0,"snaptrim_duration":0.30863261199999997,"stat_sum":{"num_bytes":8003958,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":21,"num_read_kb":25129,"num_write":12,"num_write_kb":2743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":17584212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5,6],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.58","version":"323'320","reported_seq":1860,"reported_epoch":469,"state":"peering","last_fresh":"2023-01-15T14:35:59.016002+0000","last_change":"2023-01-15T14:35:59.016309+0000","last_active":"2023-01-15T14:35:59.015950+0000","last_peered":"2023-01-15T14:35:58.617330+0000","last_clean":"2023-01-15T14:35:55.005197+0000","last_became_active":"2023-01-15T14:35:58.315742+0000","last_became_peered":"2023-01-15T14:35:58.315742+0000","last_unstale":"2023-01-15T14:35:59.016002+0000","last_undegraded":"2023-01-15T14:35:59.016002+0000","last_fullsized":"2023-01-15T14:35:59.016002+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'320","last_scrub_stamp":"2023-01-15T14:34:47.541655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.541655+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40398073200000001,"objects_trimmed":0,"snaptrim_duration":0.61028652000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,5,6],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.59","version":"294'256","reported_seq":1596,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.329258+0000","last_change":"2023-01-15T14:35:59.329770+0000","last_active":"2023-01-15T14:35:59.329258+0000","last_peered":"2023-01-15T14:35:59.329258+0000","last_clean":"2023-01-15T14:35:59.329258+0000","last_became_active":"2023-01-15T14:35:07.462526+0000","last_became_peered":"2023-01-15T14:35:07.462526+0000","last_unstale":"2023-01-15T14:35:59.329258+0000","last_undegraded":"2023-01-15T14:35:59.329258+0000","last_fullsized":"2023-01-15T14:35:59.329258+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:34:30.207952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:30.207952+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024191634,"objects_trimmed":0,"snaptrim_duration":0.20882869900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.5a","version":"466'379","reported_seq":1867,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.305505+0000","last_change":"2023-01-15T14:35:57.305587+0000","last_active":"2023-01-15T14:35:57.305505+0000","last_peered":"2023-01-15T14:35:57.305505+0000","last_clean":"2023-01-15T14:35:57.305505+0000","last_became_active":"2023-01-15T14:35:28.411845+0000","last_became_peered":"2023-01-15T14:35:28.411845+0000","last_unstale":"2023-01-15T14:35:57.305505+0000","last_undegraded":"2023-01-15T14:35:57.305505+0000","last_fullsized":"2023-01-15T14:35:57.305505+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":379,"log_dups_size":0,"ondisk_log_size":379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.20614443299999999,"stat_sum":{"num_bytes":3664025,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":37194,"num_write":8,"num_write_kb":1588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":15630873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.5b","version":"283'80","reported_seq":1198,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.726772+0000","last_change":"2023-01-15T14:35:59.726858+0000","last_active":"2023-01-15T14:35:59.726772+0000","last_peered":"2023-01-15T14:35:59.726772+0000","last_clean":"2023-01-15T14:35:59.726772+0000","last_became_active":"2023-01-15T14:35:57.399025+0000","last_became_peered":"2023-01-15T14:35:57.399025+0000","last_unstale":"2023-01-15T14:35:59.726772+0000","last_undegraded":"2023-01-15T14:35:59.726772+0000","last_fullsized":"2023-01-15T14:35:59.726772+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:53.267","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.60551423699999996,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.5c","version":"141'46","reported_seq":353,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.320949+0000","last_change":"2023-01-15T14:35:59.321031+0000","last_active":"2023-01-15T14:35:59.320949+0000","last_peered":"2023-01-15T14:35:59.320949+0000","last_clean":"2023-01-15T14:35:59.320949+0000","last_became_active":"2023-01-15T14:35:41.024937+0000","last_became_peered":"2023-01-15T14:35:41.024937+0000","last_unstale":"2023-01-15T14:35:59.320949+0000","last_undegraded":"2023-01-15T14:35:59.320949+0000","last_fullsized":"2023-01-15T14:35:59.320949+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.30529500300000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1607,"reported_epoch":469,"state":"peering","last_fresh":"2023-01-15T14:35:59.022732+0000","last_change":"2023-01-15T14:35:59.022930+0000","last_active":"2023-01-15T14:35:59.022682+0000","last_peered":"2023-01-15T14:35:58.417812+0000","last_clean":"2023-01-15T14:35:55.212907+0000","last_became_active":"2023-01-15T14:35:58.416809+0000","last_became_peered":"2023-01-15T14:35:58.416809+0000","last_unstale":"2023-01-15T14:35:59.022732+0000","last_undegraded":"2023-01-15T14:35:59.022732+0000","last_fullsized":"2023-01-15T14:35:59.022732+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:19.729021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:19.729021+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50499948100000003,"objects_trimmed":0,"snaptrim_duration":0.20345338199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5,7],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.5e","version":"215'39","reported_seq":35,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.121913+0000","last_change":"2023-01-15T14:36:00.121913+0000","last_active":"2023-01-15T14:36:00.121913+0000","last_peered":"2023-01-15T14:36:00.121913+0000","last_clean":"2023-01-15T14:36:00.121913+0000","last_became_active":"2023-01-15T14:35:57.307259+0000","last_became_peered":"2023-01-15T14:35:57.307259+0000","last_unstale":"2023-01-15T14:36:00.121913+0000","last_undegraded":"2023-01-15T14:36:00.121913+0000","last_fullsized":"2023-01-15T14:36:00.121913+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.00063915599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1674,"reported_epoch":467,"state":"remapped+peering","last_fresh":"2023-01-15T14:35:57.097025+0000","last_change":"2023-01-15T14:35:57.097568+0000","last_active":"2023-01-15T14:35:56.032751+0000","last_peered":"2023-01-15T14:35:55.410742+0000","last_clean":"2023-01-15T14:35:55.410742+0000","last_became_active":"2023-01-15T14:35:38.282477+0000","last_became_peered":"2023-01-15T14:35:38.282477+0000","last_unstale":"2023-01-15T14:35:57.097025+0000","last_undegraded":"2023-01-15T14:35:57.097025+0000","last_fullsized":"2023-01-15T14:35:57.097025+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.40687483499999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,5,7],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f2","length":"1"},{"start":"112","length":"1"},{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"119","length":"1"},{"start":"11f","length":"1"},{"start":"124","length":"1"},{"start":"126","length":"1"},{"start":"12d","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"136","length":"4"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.60","version":"463'316","reported_seq":1816,"reported_epoch":468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:58.427590+0000","last_change":"2023-01-15T14:35:58.419371+0000","last_active":"2023-01-15T14:35:58.427590+0000","last_peered":"2023-01-15T14:35:58.427590+0000","last_clean":"2023-01-15T14:35:55.200789+0000","last_became_active":"2023-01-15T14:35:58.318911+0000","last_became_peered":"2023-01-15T14:35:58.318911+0000","last_unstale":"2023-01-15T14:35:58.427590+0000","last_undegraded":"2023-01-15T14:35:58.014101+0000","last_fullsized":"2023-01-15T14:35:58.012773+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":7,"last_scrub":"402'291","last_scrub_stamp":"2023-01-15T14:34:55.535113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:55.535113+0000","objects_scrubbed":4,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.814267513,"objects_trimmed":0,"snaptrim_duration":0.242416507,"stat_sum":{"num_bytes":6004978,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":19,"num_read_kb":19071,"num_write":6,"num_write_kb":1008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":7126465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.61","version":"465'182","reported_seq":1344,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.003825+0000","last_change":"2023-01-15T14:35:57.003825+0000","last_active":"2023-01-15T14:35:57.003825+0000","last_peered":"2023-01-15T14:35:57.003825+0000","last_clean":"2023-01-15T14:35:57.003825+0000","last_became_active":"2023-01-15T14:34:44.473866+0000","last_became_peered":"2023-01-15T14:34:44.473866+0000","last_unstale":"2023-01-15T14:35:57.003825+0000","last_undegraded":"2023-01-15T14:35:57.003825+0000","last_fullsized":"2023-01-15T14:35:57.003825+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:43.765","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.044800974,"stat_sum":{"num_bytes":3823692,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":16,"num_read_kb":16964,"num_write":7,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.62","version":"465'219","reported_seq":1773,"reported_epoch":468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:58.320468+0000","last_change":"2023-01-15T14:35:58.320468+0000","last_active":"2023-01-15T14:35:58.320468+0000","last_peered":"2023-01-15T14:35:58.320468+0000","last_clean":"2023-01-15T14:35:56.031924+0000","last_became_active":"2023-01-15T14:35:58.318499+0000","last_became_peered":"2023-01-15T14:35:58.318499+0000","last_unstale":"2023-01-15T14:35:58.320468+0000","last_undegraded":"2023-01-15T14:35:58.116417+0000","last_fullsized":"2023-01-15T14:35:58.115610+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":7,"last_scrub":"403'189","last_scrub_stamp":"2023-01-15T14:34:56.851447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:56.851447+0000","objects_scrubbed":5,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30960584400000002,"objects_trimmed":0,"snaptrim_duration":1.031653277,"stat_sum":{"num_bytes":10205020,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":9593,"num_write":4,"num_write_kb":962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10723130,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,4],"avail_no_missing":["1","0","4"],"object_location_counts":[{"shards":"0,1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.63","version":"149'233","reported_seq":1099,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-01-15T14:36:01.236872+0000","last_change":"2023-01-15T14:36:01.238617+0000","last_active":"2023-01-15T14:36:01.236872+0000","last_peered":"2023-01-15T14:36:01.236872+0000","last_clean":"2023-01-15T14:36:01.236872+0000","last_became_active":"2023-01-15T14:36:00.347347+0000","last_became_peered":"2023-01-15T14:36:00.347347+0000","last_unstale":"2023-01-15T14:36:01.236872+0000","last_undegraded":"2023-01-15T14:36:01.236872+0000","last_fullsized":"2023-01-15T14:36:01.236872+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:34:56.637790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:56.637790+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.63257800399999997,"objects_trimmed":0,"snaptrim_duration":0.0017279159999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.64","version":"334'251","reported_seq":1717,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.001552+0000","last_change":"2023-01-15T14:35:57.001552+0000","last_active":"2023-01-15T14:35:57.001552+0000","last_peered":"2023-01-15T14:35:57.001552+0000","last_clean":"2023-01-15T14:35:57.001552+0000","last_became_active":"2023-01-15T14:35:07.057736+0000","last_became_peered":"2023-01-15T14:35:07.057736+0000","last_unstale":"2023-01-15T14:35:57.001552+0000","last_undegraded":"2023-01-15T14:35:57.001552+0000","last_fullsized":"2023-01-15T14:35:57.001552+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"334'251","last_scrub_stamp":"2023-01-15T14:34:48.644529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.644529+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70872698300000003,"objects_trimmed":0,"snaptrim_duration":0.096761838000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.65","version":"465'209","reported_seq":1660,"reported_epoch":468,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:35:58.421712+0000","last_change":"2023-01-15T14:35:58.421712+0000","last_active":"2023-01-15T14:35:58.421712+0000","last_peered":"2023-01-15T14:35:58.421712+0000","last_clean":"2023-01-15T14:35:55.911448+0000","last_became_active":"2023-01-15T14:35:58.419261+0000","last_became_peered":"2023-01-15T14:35:58.419261+0000","last_unstale":"2023-01-15T14:35:58.421712+0000","last_undegraded":"2023-01-15T14:35:58.010100+0000","last_fullsized":"2023-01-15T14:35:58.007251+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'126","last_scrub_stamp":"2023-01-15T14:34:23.903906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.903906+0000","objects_scrubbed":6,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30981859099999998,"objects_trimmed":1,"snaptrim_duration":0.50843005500000005,"stat_sum":{"num_bytes":10562578,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":27857,"num_write":26,"num_write_kb":10132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20310327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,6],"avail_no_missing":["1","6","7"],"object_location_counts":[{"shards":"1,6,7","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.66","version":"102'52","reported_seq":31,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.021529+0000","last_change":"2023-01-15T14:36:00.021529+0000","last_active":"2023-01-15T14:36:00.021529+0000","last_peered":"2023-01-15T14:36:00.021529+0000","last_clean":"2023-01-15T14:36:00.021529+0000","last_became_active":"2023-01-15T14:35:57.408886+0000","last_became_peered":"2023-01-15T14:35:57.408886+0000","last_unstale":"2023-01-15T14:36:00.021529+0000","last_undegraded":"2023-01-15T14:36:00.021529+0000","last_fullsized":"2023-01-15T14:36:00.021529+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:34:32.591909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.591909+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.39200112999999998,"objects_trimmed":0,"snaptrim_duration":0.00085080400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.67","version":"191'79","reported_seq":328,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.217272+0000","last_change":"2023-01-15T14:35:57.750439+0000","last_active":"2023-01-15T14:35:58.217272+0000","last_peered":"2023-01-15T14:35:58.217272+0000","last_clean":"2023-01-15T14:35:58.217272+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:35:58.217272+0000","last_undegraded":"2023-01-15T14:35:58.217272+0000","last_fullsized":"2023-01-15T14:35:58.217272+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:02.115","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.65169827700000005,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.68","version":"460'285","reported_seq":1790,"reported_epoch":467,"state":"active+clean","last_fresh":"2023-01-15T14:35:57.309633+0000","last_change":"2023-01-15T14:35:57.612665+0000","last_active":"2023-01-15T14:35:57.309633+0000","last_peered":"2023-01-15T14:35:57.309633+0000","last_clean":"2023-01-15T14:35:57.309633+0000","last_became_active":"2023-01-15T14:35:39.400045+0000","last_became_peered":"2023-01-15T14:35:39.400045+0000","last_unstale":"2023-01-15T14:35:57.309633+0000","last_undegraded":"2023-01-15T14:35:57.309633+0000","last_fullsized":"2023-01-15T14:35:57.309633+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:00.032","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.61760228699999997,"stat_sum":{"num_bytes":42018,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":23162510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.69","version":"110'149","reported_seq":1099,"reported_epoch":470,"state":"active+clean","last_fresh":"2023-01-15T14:36:00.955236+0000","last_change":"2023-01-15T14:36:00.955236+0000","last_active":"2023-01-15T14:36:00.955236+0000","last_peered":"2023-01-15T14:36:00.955236+0000","last_clean":"2023-01-15T14:36:00.955236+0000","last_became_active":"2023-01-15T14:36:00.324381+0000","last_became_peered":"2023-01-15T14:36:00.324381+0000","last_unstale":"2023-01-15T14:36:00.955236+0000","last_undegraded":"2023-01-15T14:36:00.955236+0000","last_fullsized":"2023-01-15T14:36:00.955236+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.62944658399999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.6a","version":"191'191","reported_seq":889,"reported_epoch":468,"state":"active+clean","last_fresh":"2023-01-15T14:35:58.730121+0000","last_change":"2023-01-15T14:35:58.730533+0000","last_active":"2023-01-15T14:35:58.730121+0000","last_peered":"2023-01-15T14:35:58.730121+0000","last_clean":"2023-01-15T14:35:58.730121+0000","last_became_active":"2023-01-15T14:35:41.324516+0000","last_became_peered":"2023-01-15T14:35:41.324516+0000","last_unstale":"2023-01-15T14:35:58.730121+0000","last_undegraded":"2023-01-15T14:35:58.730121+0000","last_fullsized":"2023-01-15T14:35:58.730121+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.523354553,"stat_sum":{"num_bytes":1518387,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":5931,"num_write":14,"num_write_kb":6502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9447727,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.6b","version":"155'53","reported_seq":208,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.725861+0000","last_change":"2023-01-15T14:35:59.725980+0000","last_active":"2023-01-15T14:35:59.725861+0000","last_peered":"2023-01-15T14:35:59.725861+0000","last_clean":"2023-01-15T14:35:59.725861+0000","last_became_active":"2023-01-15T14:35:57.296638+0000","last_became_peered":"2023-01-15T14:35:57.296638+0000","last_unstale":"2023-01-15T14:35:59.725861+0000","last_undegraded":"2023-01-15T14:35:59.725861+0000","last_fullsized":"2023-01-15T14:35:59.725861+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.705251925,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.6c","version":"141'46","reported_seq":189,"reported_epoch":469,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:59.424264+0000","last_change":"2023-01-15T14:35:59.424472+0000","last_active":"2023-01-15T14:35:59.424264+0000","last_peered":"2023-01-15T14:35:59.424264+0000","last_clean":"2023-01-15T14:35:59.424264+0000","last_became_active":"2023-01-15T14:35:04.085770+0000","last_became_peered":"2023-01-15T14:35:04.085770+0000","last_unstale":"2023-01-15T14:35:59.424264+0000","last_undegraded":"2023-01-15T14:35:59.424264+0000","last_fullsized":"2023-01-15T14:35:59.424264+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:57.577405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:57.577405+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.40832549200000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.6d","version":"168'156","reported_seq":872,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.123576+0000","last_change":"2023-01-15T14:35:59.224371+0000","last_active":"2023-01-15T14:35:59.123576+0000","last_peered":"2023-01-15T14:35:59.123576+0000","last_clean":"2023-01-15T14:35:59.123576+0000","last_became_active":"2023-01-15T14:35:41.216059+0000","last_became_peered":"2023-01-15T14:35:41.216059+0000","last_unstale":"2023-01-15T14:35:59.123576+0000","last_undegraded":"2023-01-15T14:35:59.123576+0000","last_fullsized":"2023-01-15T14:35:59.123576+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.20227679000000001,"stat_sum":{"num_bytes":1516832,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2710,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.6e","version":"139'38","reported_seq":190,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.329373+0000","last_change":"2023-01-15T14:35:59.329960+0000","last_active":"2023-01-15T14:35:59.329373+0000","last_peered":"2023-01-15T14:35:59.329373+0000","last_clean":"2023-01-15T14:35:59.329373+0000","last_became_active":"2023-01-15T14:34:59.097277+0000","last_became_peered":"2023-01-15T14:34:59.097277+0000","last_unstale":"2023-01-15T14:35:59.329373+0000","last_undegraded":"2023-01-15T14:35:59.329373+0000","last_fullsized":"2023-01-15T14:35:59.329373+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:32.337","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.20353211800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.6f","version":"155'145","reported_seq":1248,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.098595+0000","last_change":"2023-01-15T14:35:57.098595+0000","last_active":"2023-01-15T14:35:57.098595+0000","last_peered":"2023-01-15T14:35:57.098595+0000","last_clean":"2023-01-15T14:35:57.098595+0000","last_became_active":"2023-01-15T14:35:02.254791+0000","last_became_peered":"2023-01-15T14:35:02.254791+0000","last_unstale":"2023-01-15T14:35:57.098595+0000","last_undegraded":"2023-01-15T14:35:57.098595+0000","last_fullsized":"2023-01-15T14:35:57.098595+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:56.751","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.72409595500000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.70","version":"206'252","reported_seq":1352,"reported_epoch":471,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:01.432859+0000","last_change":"2023-01-15T14:36:01.433033+0000","last_active":"2023-01-15T14:36:01.432859+0000","last_peered":"2023-01-15T14:36:01.432859+0000","last_clean":"2023-01-15T14:36:01.432859+0000","last_became_active":"2023-01-15T14:36:00.616209+0000","last_became_peered":"2023-01-15T14:36:00.616209+0000","last_unstale":"2023-01-15T14:36:01.432859+0000","last_undegraded":"2023-01-15T14:36:01.432859+0000","last_fullsized":"2023-01-15T14:36:01.432859+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:34:36.078678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:36.078678+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.158842567,"objects_trimmed":0,"snaptrim_duration":0.30775480900000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.71","version":"204'150","reported_seq":1054,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-01-15T14:35:59.121860+0000","last_change":"2023-01-15T14:35:59.122349+0000","last_active":"2023-01-15T14:35:59.121860+0000","last_peered":"2023-01-15T14:35:59.121860+0000","last_clean":"2023-01-15T14:35:59.121860+0000","last_became_active":"2023-01-15T14:35:02.151543+0000","last_became_peered":"2023-01-15T14:35:02.151543+0000","last_unstale":"2023-01-15T14:35:59.121860+0000","last_undegraded":"2023-01-15T14:35:59.121860+0000","last_fullsized":"2023-01-15T14:35:59.121860+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:31.314","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.10352167299999999,"stat_sum":{"num_bytes":3558907,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10725,"num_write":4,"num_write_kb":3090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3605110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.72","version":"394'169","reported_seq":1183,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.308394+0000","last_change":"2023-01-15T14:35:57.308456+0000","last_active":"2023-01-15T14:35:57.308394+0000","last_peered":"2023-01-15T14:35:57.308394+0000","last_clean":"2023-01-15T14:35:57.308394+0000","last_became_active":"2023-01-15T14:35:02.148756+0000","last_became_peered":"2023-01-15T14:35:02.148756+0000","last_unstale":"2023-01-15T14:35:57.308394+0000","last_undegraded":"2023-01-15T14:35:57.308394+0000","last_fullsized":"2023-01-15T14:35:57.308394+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:52.712","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.312764086,"stat_sum":{"num_bytes":2663943,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3524,"num_write":1,"num_write_kb":1428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2663943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.73","version":"204'275","reported_seq":1535,"reported_epoch":468,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:35:58.320276+0000","last_change":"2023-01-15T14:35:58.320126+0000","last_active":"2023-01-15T14:35:58.320276+0000","last_peered":"2023-01-15T14:35:58.320276+0000","last_clean":"2023-01-15T14:35:55.346251+0000","last_became_active":"2023-01-15T14:35:58.320126+0000","last_became_peered":"2023-01-15T14:35:58.320126+0000","last_unstale":"2023-01-15T14:35:58.320276+0000","last_undegraded":"2023-01-15T14:35:58.320276+0000","last_fullsized":"2023-01-15T14:35:58.107090+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:34.588","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.24967597,"stat_sum":{"num_bytes":3485477,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":11189,"num_write":7,"num_write_kb":3286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4],"avail_no_missing":["1","0","4","5"],"object_location_counts":[{"shards":"1,4","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.74","version":"169'138","reported_seq":906,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:57.100502+0000","last_change":"2023-01-15T14:35:57.100502+0000","last_active":"2023-01-15T14:35:57.100502+0000","last_peered":"2023-01-15T14:35:57.100502+0000","last_clean":"2023-01-15T14:35:57.100502+0000","last_became_active":"2023-01-15T14:35:38.078356+0000","last_became_peered":"2023-01-15T14:35:38.078356+0000","last_unstale":"2023-01-15T14:35:57.100502+0000","last_undegraded":"2023-01-15T14:35:57.100502+0000","last_fullsized":"2023-01-15T14:35:57.100502+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.50407087299999997,"stat_sum":{"num_bytes":613464,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2809,"num_write":2,"num_write_kb":305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.75","version":"110'55","reported_seq":129,"reported_epoch":468,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:35:58.620122+0000","last_change":"2023-01-15T14:35:58.620274+0000","last_active":"2023-01-15T14:35:58.620122+0000","last_peered":"2023-01-15T14:35:58.620122+0000","last_clean":"2023-01-15T14:35:58.620122+0000","last_became_active":"2023-01-15T14:35:57.308332+0000","last_became_peered":"2023-01-15T14:35:57.308332+0000","last_unstale":"2023-01-15T14:35:58.620122+0000","last_undegraded":"2023-01-15T14:35:58.620122+0000","last_fullsized":"2023-01-15T14:35:58.620122+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51281216799999996,"stat_sum":{"num_bytes":796228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5170,"num_write":3,"num_write_kb":1557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":531514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12c","length":"2"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"150","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":146,"stat_sum":{"num_bytes":399925307,"num_objects":182,"num_object_clones":128,"num_object_copies":546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":156,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":17,"num_read":1298,"num_read_kb":1451389,"num_write":978,"num_write_kb":390041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1205,"num_bytes_recovered":821860764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28115,"ondisk_log_size":28115,"up":438,"acting":418,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574944,"num_pgs":77,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1133268,"kb_used_data":92566536,"kb_used_omap":15336,"kb_used_meta":0,"kb_avail":92566112,"statfs":{"total":95948165120,"available":94787698688,"internally_reserved":434176,"allocated":94788132864,"data_stored":94788132864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15704834,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":6.6619999999999999,"5min":4.6289999999999996,"15min":4.0369999999999999},"min":{"1min":0.53400000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":83.435000000000002,"5min":148.10900000000001,"15min":148.10900000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.77100000000000002,"5min":4.0030000000000001,"15min":7.4720000000000004},"min":{"1min":0.59599999999999997,"5min":0.314,"15min":0.314},"max":{"1min":1.1599999999999999,"5min":96.337000000000003,"15min":96.337000000000003},"last":0.67900000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":3.6589999999999998,"15min":5.5670000000000002},"min":{"1min":0.64300000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.03,"5min":204.74199999999999,"15min":204.74199999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.022,"5min":8.3290000000000006,"15min":6.899},"min":{"1min":0.67500000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":3.4089999999999998,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.74199999999999999}]},{"osd":2,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":9.9890000000000008,"5min":5.6239999999999997,"15min":5.6769999999999996},"min":{"1min":0.54400000000000004,"5min":0.377,"15min":0.377},"max":{"1min":76.272999999999996,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.75,"15min":4.1500000000000004},"min":{"1min":0.64900000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.105,"5min":162.96799999999999,"15min":162.96799999999999},"last":0.69199999999999995}]},{"osd":3,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":8.1660000000000004,"5min":6.375,"15min":3.6560000000000001},"min":{"1min":0.69499999999999995,"5min":0.434,"15min":0.434},"max":{"1min":76.114999999999995,"5min":209.05500000000001,"15min":209.05500000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.97299999999999998,"5min":2.2599999999999998,"15min":3.8149999999999999},"min":{"1min":0.70199999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7170000000000001,"5min":98.825000000000003,"15min":98.825000000000003},"last":0.80500000000000005}]},{"osd":4,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":3.6850000000000001,"5min":3.8860000000000001,"15min":7.3040000000000003},"min":{"1min":0.48099999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":40.563000000000002,"5min":96.185000000000002,"15min":96.185000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":24.846,"5min":9.4670000000000005,"15min":4.8280000000000003},"min":{"1min":0.74199999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":184.37200000000001,"5min":184.37200000000001,"15min":184.37200000000001},"last":0.54100000000000004}]},{"osd":5,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":3.4340000000000002,"15min":5.3040000000000003},"min":{"1min":0.58799999999999997,"5min":0.51200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1400000000000001,"5min":96.263999999999996,"15min":96.263999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":8.2710000000000008,"5min":3.2669999999999999,"15min":4.4139999999999997},"min":{"1min":0.871,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":76.159000000000006,"5min":76.159000000000006,"15min":95.430000000000007},"last":0.71599999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":3.4929999999999999,"15min":6.6550000000000002},"min":{"1min":0.64200000000000002,"5min":0.496,"15min":0.496},"max":{"1min":1.401,"5min":96.299999999999997,"15min":96.299999999999997},"last":0.77300000000000002},{"interface":"front","average":{"1min":8.3369999999999997,"5min":3.3340000000000001,"15min":2.1400000000000001},"min":{"1min":0.91500000000000004,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":76.209000000000003,"5min":76.209000000000003,"15min":76.209000000000003},"last":0.626}]}]},{"osd":6,"up_from":13,"seq":55834574946,"num_pgs":74,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1063156,"kb_used_data":92638396,"kb_used_omap":9019,"kb_used_meta":0,"kb_avail":92636224,"statfs":{"total":95948165120,"available":94859493376,"internally_reserved":2224128,"allocated":94861717504,"data_stored":94861717504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9236406,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":16.390000000000001,"5min":5.2050000000000001,"15min":2.2189999999999999},"min":{"1min":0.56000000000000005,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":166.19900000000001,"5min":166.19900000000001,"15min":166.19900000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.76000000000000001,"15min":0.85099999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.92900000000000005,"5min":1.474,"15min":19.413},"last":0.91200000000000003}]},{"osd":1,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.8109999999999999,"5min":2.194,"15min":2.3559999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":45.875999999999998,"5min":56.844999999999999,"15min":99.203000000000003},"last":0.93999999999999995},{"interface":"front","average":{"1min":5.5110000000000001,"5min":2.5609999999999999,"15min":2.218},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.439},"max":{"1min":69.733000000000004,"5min":69.733000000000004,"15min":98.882000000000005},"last":0.77500000000000002}]},{"osd":2,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.069,"15min":1.518},"min":{"1min":0.55600000000000005,"5min":0.45600000000000002,"15min":0.41399999999999998},"max":{"1min":2.081,"5min":23.135999999999999,"15min":128.173},"last":0.96099999999999997},{"interface":"front","average":{"1min":3.9500000000000002,"5min":1.7430000000000001,"15min":1.95},"min":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":45.773000000000003,"5min":45.773000000000003,"15min":114.724},"last":0.752}]},{"osd":3,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.8319999999999999,"5min":2.4910000000000001,"15min":4.8010000000000002},"min":{"1min":0.56200000000000006,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":45.813000000000002,"5min":93.710999999999999,"15min":93.710999999999999},"last":0.87},{"interface":"front","average":{"1min":11.069000000000001,"5min":5.4180000000000001,"15min":8.8190000000000008},"min":{"1min":0.625,"5min":0.625,"15min":0.60599999999999998},"max":{"1min":152.66499999999999,"5min":167.178,"15min":255.24299999999999},"last":0.79400000000000004}]},{"osd":4,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":12.531000000000001,"5min":21.152000000000001,"15min":7.7300000000000004},"min":{"1min":0.46300000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":107.03,"5min":1256.867,"15min":1256.867},"last":95.697000000000003},{"interface":"front","average":{"1min":0.90400000000000003,"5min":14.872,"15min":9.3019999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.4350000000000001,"5min":1255.77,"15min":1255.77},"last":0.622}]},{"osd":5,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.8370000000000002,"5min":1.532,"15min":1.415},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":45.673999999999999,"5min":45.673999999999999,"15min":67.469999999999999},"last":5.718},{"interface":"front","average":{"1min":5.5330000000000004,"5min":1.8799999999999999,"15min":1.2589999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.46100000000000002},"max":{"1min":69.554000000000002,"5min":69.554000000000002,"15min":69.554000000000002},"last":0.621}]},{"osd":7,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.0289999999999999,"15min":3.056},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51300000000000001},"max":{"1min":2.0219999999999998,"5min":2.0219999999999998,"15min":78.888000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":9.6920000000000002,"5min":3.6299999999999999,"15min":3.6520000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.499},"max":{"1min":85.358000000000004,"5min":85.358000000000004,"15min":121.297},"last":0.81200000000000006}]}]},{"osd":5,"up_from":13,"seq":55834574946,"num_pgs":63,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":815780,"kb_used_data":92891028,"kb_used_omap":1743,"kb_used_meta":0,"kb_avail":92883600,"statfs":{"total":95948165120,"available":95112806400,"internally_reserved":7606272,"allocated":95120412672,"data_stored":95120412672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1785176,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:10 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.81599999999999995,"15min":2.133},"min":{"1min":0.441,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.034,"5min":3.4009999999999998,"15min":40.482999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":2.4060000000000001,"5min":1.478,"15min":1.0649999999999999},"min":{"1min":0.59699999999999998,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":30.481000000000002,"5min":30.481000000000002,"15min":30.481000000000002},"last":0.80900000000000005}]},{"osd":1,"last update":"Sun Jan 15 14:35:10 2023","interfaces":[{"interface":"back","average":{"1min":5.6230000000000002,"5min":3.8919999999999999,"15min":1.9339999999999999},"min":{"1min":0.52700000000000002,"5min":0.432,"15min":0.432},"max":{"1min":86.620000000000005,"5min":86.620000000000005,"15min":86.620000000000005},"last":0.65900000000000003},{"interface":"front","average":{"1min":5.6120000000000001,"5min":3.9039999999999999,"15min":1.887},"min":{"1min":0.626,"5min":0.503,"15min":0.503},"max":{"1min":86.581000000000003,"5min":86.581000000000003,"15min":86.581000000000003},"last":0.72099999999999997}]},{"osd":2,"last update":"Sun Jan 15 14:35:10 2023","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":2.532,"15min":1.425},"min":{"1min":0.65000000000000002,"5min":0.54800000000000004,"15min":0.48999999999999999},"max":{"1min":1.0169999999999999,"5min":87.251000000000005,"15min":87.251000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.86899999999999999,"15min":1.038},"min":{"1min":0.52300000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2989999999999999,"5min":4.9349999999999996,"15min":34.939},"last":0.61299999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:35:10 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":2.02,"15min":1.4910000000000001},"min":{"1min":0.48499999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.288,"5min":98.257000000000005,"15min":98.257000000000005},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.88400000000000001,"5min":3.7679999999999998,"15min":3.5800000000000001},"min":{"1min":0.59599999999999997,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.151,"5min":81.894000000000005,"15min":81.894000000000005},"last":0.78300000000000003}]},{"osd":4,"last update":"Sun Jan 15 14:35:10 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":11.286,"15min":7.7080000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.3500000000000001,"5min":1129.999,"15min":1129.999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.88800000000000001,"5min":11.246,"15min":4.4390000000000001},"min":{"1min":0.52200000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1739999999999999,"5min":1130.568,"15min":1130.568},"last":0.96899999999999997}]},{"osd":6,"last update":"Sun Jan 15 14:35:10 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.96599999999999997,"15min":1.54},"min":{"1min":0.58199999999999996,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.161,"5min":2.6840000000000002,"15min":93.903999999999996},"last":0.874},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0229999999999999,"15min":1.218},"min":{"1min":0.68700000000000006,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.6699999999999999,"5min":2.988,"15min":57.375999999999998},"last":0.93799999999999994}]},{"osd":7,"last update":"Sun Jan 15 14:35:10 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":1.0820000000000001,"15min":7.0679999999999996},"min":{"1min":0.71499999999999997,"5min":0.497,"15min":0.497},"max":{"1min":1.371,"5min":4.6379999999999999,"15min":143.858},"last":1.004},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.1160000000000001,"15min":7.1059999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.4419999999999999,"5min":2.6139999999999999,"15min":143.922},"last":0.95499999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574944,"num_pgs":69,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":997872,"kb_used_data":92701864,"kb_used_omap":20793,"kb_used_meta":0,"kb_avail":92701508,"statfs":{"total":95948165120,"available":94926344192,"internally_reserved":364544,"allocated":94926708736,"data_stored":94926708736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21292162,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":21.449999999999999,"15min":10.609},"min":{"1min":0.505,"5min":0.44,"15min":0.44},"max":{"1min":1.1200000000000001,"5min":2165.8420000000001,"15min":2165.8420000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":14.188000000000001,"5min":5.1139999999999999,"15min":2.4140000000000001},"min":{"1min":0.53300000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":180.33600000000001,"5min":180.33600000000001,"15min":180.33600000000001},"last":1.026}]},{"osd":1,"last update":"Sun Jan 15 14:35:21 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.954,"15min":6.4569999999999999},"min":{"1min":0.5,"5min":0.49299999999999999,"15min":0.47399999999999998},"max":{"1min":1.9039999999999999,"5min":69.415000000000006,"15min":171.91499999999999},"last":0.82299999999999995},{"interface":"front","average":{"1min":4.3010000000000002,"5min":6.0970000000000004,"15min":2.8900000000000001},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":70.757000000000005,"5min":162.976,"15min":162.976},"last":1.0760000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":5.4720000000000004,"5min":23.584,"15min":11.519},"min":{"1min":0.58499999999999996,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":98.298000000000002,"5min":2165.3220000000001,"15min":2165.3220000000001},"last":1},{"interface":"front","average":{"1min":0.85599999999999998,"5min":1.276,"15min":1.0289999999999999},"min":{"1min":0.66200000000000003,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.089,"5min":32.081000000000003,"15min":32.081000000000003},"last":1.1399999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":15.048,"5min":5.5209999999999999,"15min":2.7400000000000002},"min":{"1min":0.67600000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":180.37200000000001,"5min":180.37200000000001,"15min":180.37200000000001},"last":2.1499999999999999},{"interface":"front","average":{"1min":6.3010000000000002,"5min":24.248000000000001,"15min":12.223000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":79.888000000000005,"5min":2165.4690000000001,"15min":2165.4690000000001},"last":0.89000000000000001}]},{"osd":5,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.88200000000000001,"15min":0.90500000000000003},"min":{"1min":0.60299999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.214,"5min":2.577,"15min":2.577},"last":0.97899999999999998},{"interface":"front","average":{"1min":12.973000000000001,"5min":4.3120000000000003,"15min":2.8780000000000001},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":180.255,"5min":180.255,"15min":180.255},"last":2.3199999999999998}]},{"osd":6,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":3.6509999999999998,"5min":2.0070000000000001,"15min":4.7270000000000003},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":58.231999999999999,"5min":58.231999999999999,"15min":99.780000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":1.024,"5min":21.626999999999999,"15min":10.375999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.3109999999999999,"5min":2165.7330000000002,"15min":2165.7330000000002},"last":1.2050000000000001}]},{"osd":7,"last update":"Sun Jan 15 14:35:17 2023","interfaces":[{"interface":"back","average":{"1min":24.492000000000001,"5min":8.2859999999999996,"15min":10.702},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":338.72699999999998,"5min":338.72699999999998,"15min":338.72699999999998},"last":2.2330000000000001},{"interface":"front","average":{"1min":1.028,"5min":22.184999999999999,"15min":14.502000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.4530000000000001,"5min":2165.5309999999999,"15min":2165.5309999999999},"last":1.0660000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574944,"num_pgs":78,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1009040,"kb_used_data":92690508,"kb_used_omap":21089,"kb_used_meta":0,"kb_avail":92690340,"statfs":{"total":95948165120,"available":94914908160,"internally_reserved":172032,"allocated":94915080192,"data_stored":94915080192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21595437,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":6.3230000000000004,"15min":5.6289999999999996},"min":{"1min":0.32400000000000001,"5min":0.255,"15min":0.255},"max":{"1min":0.98199999999999998,"5min":122.065,"15min":174.52199999999999},"last":57.884999999999998},{"interface":"front","average":{"1min":4.3959999999999999,"5min":3.7160000000000002,"15min":6.0439999999999996},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":68.239000000000004,"5min":74.280000000000001,"15min":93.674000000000007},"last":57.920999999999999}]},{"osd":1,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.0190000000000001,"5min":5.391,"15min":10.595000000000001},"min":{"1min":0.41499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":62.805999999999997,"5min":117.664,"15min":159.256},"last":0.71899999999999997},{"interface":"front","average":{"1min":2.5659999999999998,"5min":10.003,"15min":4.351},"min":{"1min":0.32900000000000001,"5min":0.317,"15min":0.317},"max":{"1min":35.75,"5min":155.44,"15min":174.578},"last":0.65400000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.3869999999999996,"5min":2.843,"15min":3.129},"min":{"1min":0.433,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":68.156000000000006,"5min":86.412000000000006,"15min":86.412000000000006},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":5.1600000000000001,"15min":3.633},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.004,"5min":122.13,"15min":174.45099999999999},"last":0.85099999999999998}]},{"osd":4,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.5679999999999996,"5min":29.02,"15min":13.804},"min":{"1min":0.58699999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":68.296000000000006,"5min":2059.6680000000001,"15min":2059.6680000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":4.242,"5min":29.707000000000001,"15min":19.187000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":62.914999999999999,"5min":2058.4229999999998,"15min":2058.4229999999998},"last":0.90700000000000003}]},{"osd":5,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":4.2949999999999999,"5min":4.6929999999999996,"15min":10.602},"min":{"1min":0.58399999999999996,"5min":0.504,"15min":0.504},"max":{"1min":63.055,"5min":117.744,"15min":159.43000000000001},"last":1.268},{"interface":"front","average":{"1min":0.93500000000000005,"5min":4.3940000000000001,"15min":2.6699999999999999},"min":{"1min":0.73399999999999999,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.306,"5min":122.208,"15min":174.61000000000001},"last":0.94199999999999995}]},{"osd":6,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":6.6639999999999997,"5min":5.5419999999999998,"15min":3.0579999999999998},"min":{"1min":0.67500000000000004,"5min":0.65900000000000003,"15min":0.63500000000000001},"max":{"1min":54.311999999999998,"5min":122.26600000000001,"15min":174.63999999999999},"last":1.03},{"interface":"front","average":{"1min":8.7650000000000006,"5min":2.5550000000000002,"15min":3.3069999999999999},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":75.340999999999994,"5min":75.340999999999994,"15min":75.340999999999994},"last":1.1379999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":10.919,"5min":3.0070000000000001,"15min":3.6200000000000001},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":96.635000000000005,"5min":96.635000000000005,"15min":96.635000000000005},"last":1.202},{"interface":"front","average":{"1min":4.351,"5min":10.726000000000001,"15min":15.128},"min":{"1min":0.69499999999999995,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":62.984000000000002,"5min":281.13400000000001,"15min":281.13400000000001},"last":1.1839999999999999}]}]},{"osd":2,"up_from":0,"seq":55834574944,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834574945,"num_pgs":70,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1031576,"kb_used_data":92668208,"kb_used_omap":19620,"kb_used_meta":0,"kb_avail":92667804,"statfs":{"total":95948165120,"available":94891831296,"internally_reserved":413696,"allocated":94892244992,"data_stored":94892244992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20090892,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":3.6840000000000002,"5min":2.4870000000000001,"15min":3.6699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":57.098999999999997,"5min":100.533,"15min":144.12799999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":6.8630000000000004,"5min":7.8300000000000001,"15min":5.2439999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":101.319,"5min":176.01599999999999,"15min":176.01599999999999},"last":1.1060000000000001}]},{"osd":2,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":3.4319999999999999,"15min":1.762},"min":{"1min":0.36299999999999999,"5min":0.25700000000000001,"15min":0.245},"max":{"1min":0.80900000000000005,"5min":95.879999999999995,"15min":95.879999999999995},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.68600000000000005,"15min":2.8769999999999998},"min":{"1min":0.372,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.98099999999999998,"5min":3.9780000000000002,"15min":48.119999999999997},"last":0.93999999999999995}]},{"osd":3,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":9.3140000000000001,"5min":7.5700000000000003,"15min":14.470000000000001},"min":{"1min":0.33700000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":125.736,"5min":245.637,"15min":282.01400000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":6.9790000000000001,"5min":9.157,"15min":7.5739999999999998},"min":{"1min":0.40899999999999997,"5min":0.253,"15min":0.253},"max":{"1min":101.401,"5min":176.13900000000001,"15min":176.13900000000001},"last":0.64300000000000002}]},{"osd":4,"last update":"Sun Jan 15 14:35:20 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":19.507000000000001,"15min":9.5679999999999996},"min":{"1min":0.44900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.155,"5min":1181.3389999999999,"15min":1181.3389999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.81499999999999995,"5min":15.012,"15min":7.5220000000000002},"min":{"1min":0.56499999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.28,"5min":1181.8720000000001,"15min":1181.8720000000001},"last":1.246}]},{"osd":5,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":2.367,"15min":1.47},"min":{"1min":0.61299999999999999,"5min":0.56899999999999995,"15min":0.433},"max":{"1min":1.3999999999999999,"5min":69.478999999999999,"15min":69.478999999999999},"last":1.444},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.95499999999999996,"15min":3.044},"min":{"1min":0.64300000000000002,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8049999999999999,"5min":4.0780000000000003,"15min":46.905999999999999},"last":1.2849999999999999}]},{"osd":6,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":7.2590000000000003,"5min":8.1709999999999994,"15min":5.7880000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":101.45999999999999,"5min":176.249,"15min":176.249},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.98099999999999998,"5min":2.4409999999999998,"15min":1.55},"min":{"1min":0.66600000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.6759999999999999,"5min":69.549999999999997,"15min":69.549999999999997},"last":1.3129999999999999}]},{"osd":7,"last update":"Sun Jan 15 14:35:15 2023","interfaces":[{"interface":"back","average":{"1min":7.4199999999999999,"5min":6.4710000000000001,"15min":6.7809999999999997},"min":{"1min":0.58699999999999997,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":97.539000000000001,"5min":157.03700000000001,"15min":157.03700000000001},"last":102.152},{"interface":"front","average":{"1min":17.943000000000001,"5min":10.349,"15min":6.367},"min":{"1min":0.55400000000000005,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":193.74600000000001,"5min":193.74600000000001,"15min":193.74600000000001},"last":0.83399999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574945,"num_pgs":89,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1139364,"kb_used_data":92560232,"kb_used_omap":16327,"kb_used_meta":0,"kb_avail":92560016,"statfs":{"total":95948165120,"available":94781456384,"internally_reserved":221184,"allocated":94781677568,"data_stored":94781677568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16718893,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":2,"commit_latency_ns":0,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":10.833,"5min":4.6390000000000002,"15min":2.2109999999999999},"min":{"1min":0.27100000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":127.86,"5min":127.86,"15min":127.86},"last":137.69999999999999},{"interface":"front","average":{"1min":7.3220000000000001,"5min":5.681,"15min":2.7040000000000002},"min":{"1min":0.29899999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":100.407,"5min":162.91200000000001,"15min":162.91200000000001},"last":0.312}]},{"osd":2,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":7.4370000000000003,"5min":3.052,"15min":2.0499999999999998},"min":{"1min":0.38900000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":100.58799999999999,"5min":100.58799999999999,"15min":100.58799999999999},"last":5.0060000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":1.0900000000000001,"15min":1.339},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.28499999999999998},"max":{"1min":0.80600000000000005,"5min":48.893000000000001,"15min":71.406999999999996},"last":5.0529999999999999}]},{"osd":3,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":11.68,"5min":9.2279999999999998,"15min":13.448},"min":{"1min":0.35999999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":127.803,"5min":199.08000000000001,"15min":284.62799999999999},"last":138.31999999999999},{"interface":"front","average":{"1min":7.452,"5min":6.2939999999999996,"15min":3.6000000000000001},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.33600000000000002},"max":{"1min":100.371,"5min":162.61600000000001,"15min":174.09399999999999},"last":0.38300000000000001}]},{"osd":4,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":15.289999999999999,"15min":6.1970000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.96899999999999997,"5min":1486.309,"15min":1486.309},"last":5.1100000000000003},{"interface":"front","average":{"1min":11.106,"5min":18.841999999999999,"15min":7.4020000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":127.61499999999999,"5min":1486.3789999999999,"15min":1486.3789999999999},"last":137.91499999999999}]},{"osd":5,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":7.6619999999999999,"5min":3.2709999999999999,"15min":7.3550000000000004},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":100.46299999999999,"5min":100.46299999999999,"15min":176.34399999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.92100000000000004,"15min":1.2490000000000001},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.391,"5min":2.2959999999999998,"15min":71.563999999999993},"last":5.1580000000000004}]},{"osd":6,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":11.205,"5min":5.0679999999999996,"15min":2.7120000000000002},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":127.68000000000001,"5min":127.68000000000001,"15min":127.68000000000001},"last":137.863},{"interface":"front","average":{"1min":7.7210000000000001,"5min":3.3650000000000002,"15min":2.1949999999999998},"min":{"1min":0.65200000000000002,"5min":0.64600000000000002,"15min":0.57499999999999996},"max":{"1min":100.544,"5min":100.544,"15min":100.544},"last":0.60399999999999998}]},{"osd":7,"last update":"Sun Jan 15 14:35:19 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":1.7789999999999999,"15min":10.553000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.4690000000000001,"5min":73.864999999999995,"15min":292.61099999999999},"last":5.1340000000000003},{"interface":"front","average":{"1min":17.018999999999998,"5min":6.2489999999999997,"15min":3.1059999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":127.46599999999999,"5min":127.46599999999999,"15min":127.46599999999999},"last":137.78}]}]}],"pool_statfs":[]}} 2023-01-15T14:36:03.549 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:36:02.612+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:36:03.557 INFO:tasks.ceph.ceph_manager.ceph:increase pool size by 10 2023-01-15T14:36:03.557 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_num 156 2023-01-15T14:36:03.558 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:36:03.558 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:36:03.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:36:03.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:36:03.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:36:03.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:36:03.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:36:03.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:36:03.559 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:36:03.560 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:36:02.817+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:36:03.561 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is unset 2023-01-15T14:36:03.605 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:36:03.597+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:36:03.754 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:03.754 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:03.754 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2023-01-15T14:36:03.755 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:36:03.752+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:36:03.855 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:36:03.852+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:36:03.917 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:03.917 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:03.918 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2023-01-15T14:36:03.956 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:36:03.952+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:36:04.078 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:04.080 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:04.081 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2023-01-15T14:36:04.237 INFO:teuthology.orchestra.run.smithi090.stderr:set pool 3 pg_num to 156 2023-01-15T14:36:04.404 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:36:04.069+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:36:04.406 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:36:04.404+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:36:04.507 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:36:04.497+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:36:04.559 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:04.559 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:04.559 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2023-01-15T14:36:04.607 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:36:04.604+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:36:04.714 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:04.714 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:04.715 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:36:04.712+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:36:04.719 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2023-01-15T14:36:04.816 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:36:04.812+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:36:04.976 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:04.976 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:04.977 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2023-01-15T14:36:04.977 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:36:04.973+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:36:05.077 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:36:05.069+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:36:05.137 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:05.138 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:05.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2023-01-15T14:36:05.315 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:36:05.312+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:36:05.368 INFO:tasks.rados.rados.0.smithi090.stdout:1739: finishing rollback tid 1 to smithi09025264-46 2023-01-15T14:36:05.516 INFO:tasks.rados.rados.0.smithi090.stdout:1739: finishing rollback tid 2 to smithi09025264-46 2023-01-15T14:36:05.516 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 46 v 431 (ObjNum 525 snap 158 seq_num 525) dirty exists 2023-01-15T14:36:05.517 INFO:tasks.rados.rados.0.smithi090.stdout:1739: done (0 left) 2023-01-15T14:36:05.517 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:05.518 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:05.519 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2023-01-15T14:36:05.519 INFO:tasks.ceph.osd.2.smithi090.stderr:2023-01-15T14:36:05.509+0000 7fe358b3c700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 23471) UID: 0 2023-01-15T14:36:05.560 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2023-01-15T14:36:05.723 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2023-01-15T14:36:05.726 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:05.726 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:05.727 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2023-01-15T14:36:05.727 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:36:05.724+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:36:05.826 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:36:05.824+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:36:05.927 INFO:teuthology.orchestra.run.smithi090.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:05.927 INFO:teuthology.orchestra.run.smithi090.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:05.927 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2023-01-15T14:36:05.928 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:36:05.928 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:36:05.929 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:36:05.917+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:36:05.935 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:36:05.952 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:36:06.133 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:36:06.133 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:36:06.134 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:36:06.134 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:36:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:36:06.134 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 0 2023-01-15T14:36:06.135 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:36:06.135 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2023-01-15T14:36:06.135 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:36:06.132+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:36:06.136 INFO:tasks.rados.rados.0.smithi090.stdout:1748: done (0 left) 2023-01-15T14:36:06.137 INFO:tasks.rados.rados.0.smithi090.stdout:1749: write oid 19 current snap is 190 2023-01-15T14:36:06.137 INFO:tasks.rados.rados.0.smithi090.stdout:1749: seq_num 620 ranges {420340=478973,1597777=512212} 2023-01-15T14:36:06.137 INFO:tasks.rados.rados.0.smithi090.stdout:1749: writing smithi09025264-19 from 420340 to 899313 tid 1 2023-01-15T14:36:06.137 INFO:tasks.rados.rados.0.smithi090.stdout:1749: writing smithi09025264-19 from 1597777 to 2109989 tid 2 2023-01-15T14:36:06.137 INFO:tasks.rados.rados.0.smithi090.stdout:1750: write oid 14 current snap is 190 2023-01-15T14:36:06.137 INFO:tasks.rados.rados.0.smithi090.stdout:1750: seq_num 621 ranges {437147=554222,1656259=655870} 2023-01-15T14:36:06.137 INFO:tasks.rados.rados.0.smithi090.stdout:1750: writing smithi09025264-14 from 437147 to 991369 tid 1 2023-01-15T14:36:06.138 INFO:tasks.rados.rados.0.smithi090.stdout:1750: writing smithi09025264-14 from 1656259 to 2312129 tid 2 2023-01-15T14:36:06.138 INFO:tasks.rados.rados.0.smithi090.stdout:1751: write oid 26 current snap is 190 2023-01-15T14:36:06.138 INFO:tasks.rados.rados.0.smithi090.stdout:1751: seq_num 622 ranges {453954=629471,1598388=515881} 2023-01-15T14:36:06.138 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:36:06.139 INFO:tasks.rados.rados.0.smithi090.stdout:1751: writing smithi09025264-26 from 453954 to 1083425 tid 1 2023-01-15T14:36:06.139 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'true' 2023-01-15T14:36:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'true' 2023-01-15T14:36:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'true' 2023-01-15T14:36:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'true' 2023-01-15T14:36:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'true' 2023-01-15T14:36:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'true' 2023-01-15T14:36:06.140 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'true' 2023-01-15T14:36:06.141 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'true' 2023-01-15T14:36:06.144 INFO:tasks.rados.rados.0.smithi090.stdout:1751: writing smithi09025264-26 from 1598388 to 2114269 tid 2 2023-01-15T14:36:06.144 INFO:tasks.rados.rados.0.smithi090.stdout:1752: copy_from oid 17 from oid 46 current snap is 190 2023-01-15T14:36:06.145 INFO:tasks.rados.rados.0.smithi090.stdout:1753: read oid 27 snap -1 2023-01-15T14:36:06.145 INFO:tasks.rados.rados.0.smithi090.stdout:1753: expect (ObjNum 596 snap 180 seq_num 596) 2023-01-15T14:36:06.145 INFO:tasks.rados.rados.0.smithi090.stdout:1754: read oid 25 snap -1 2023-01-15T14:36:06.145 INFO:tasks.rados.rados.0.smithi090.stdout:1754: expect (ObjNum 487 snap 142 seq_num 487) 2023-01-15T14:36:06.146 INFO:tasks.rados.rados.0.smithi090.stdout:1755: snap_create 2023-01-15T14:36:06.146 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 6 2023-01-15T14:36:06.201 INFO:tasks.rados.rados.0.smithi090.stdout:1749: finishing write tid 1 to smithi09025264-19 2023-01-15T14:36:06.292 INFO:tasks.rados.rados.0.smithi090.stdout:1749: finishing write tid 2 to smithi09025264-19 2023-01-15T14:36:06.294 INFO:tasks.rados.rados.0.smithi090.stdout:1750: finishing write tid 1 to smithi09025264-14 2023-01-15T14:36:06.294 INFO:tasks.rados.rados.0.smithi090.stdout:1754: expect (ObjNum 487 snap 142 seq_num 487) 2023-01-15T14:36:06.294 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:36:06.285+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:36:06.295 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:36:06.295 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:36:06.295 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:36:06.295 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:36:06.295 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:36:06.296 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2023-01-15T14:36:06.393 INFO:tasks.rados.rados.0.smithi090.stdout:1753: expect (ObjNum 596 snap 180 seq_num 596) 2023-01-15T14:36:06.459 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:36:06.459 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:36:06.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:36:06.459 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:36:06.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.459 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3 3.3 3:ca59e2b0:::smithi09025264-1:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:36:06.460 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:36:06.460 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 460.62069180600002, 2023-01-15T14:36:06.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62968594499999997, 2023-01-15T14:36:06.460 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:06.460 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:06.460 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:06.460 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:06.461 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:06.461 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3 2023-01-15T14:36:06.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.461 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:06.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:06.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:36:06.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:06.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775132+0000", 2023-01-15T14:36:06.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:06.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775131+0000", 2023-01-15T14:36:06.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:06.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:06.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775138+0000", 2023-01-15T14:36:06.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:36:06.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:06.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775148+0000", 2023-01-15T14:36:06.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7969999999999995e-06 2023-01-15T14:36:06.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:06.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775161+0000", 2023-01-15T14:36:06.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2469999999999999e-05 2023-01-15T14:36:06.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.857974+0000", 2023-01-15T14:36:06.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.082812887000000002 2023-01-15T14:36:06.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:06.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.858147+0000", 2023-01-15T14:36:06.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000173883 2023-01-15T14:36:06.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106098+0000", 2023-01-15T14:36:06.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.24795019600000001 2023-01-15T14:36:06.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:06.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106181+0000", 2023-01-15T14:36:06.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3248e-05 2023-01-15T14:36:06.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 0,6", 2023-01-15T14:36:06.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106330+0000", 2023-01-15T14:36:06.471 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014918200000000001 2023-01-15T14:36:06.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:06.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106510+0000", 2023-01-15T14:36:06.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017979399999999999 2023-01-15T14:36:06.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:06.472 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106527+0000", 2023-01-15T14:36:06.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6643999999999999e-05 2023-01-15T14:36:06.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:06.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106605+0000", 2023-01-15T14:36:06.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8650000000000001e-05 2023-01-15T14:36:06.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:06.474 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.106644+0000", 2023-01-15T14:36:06.474 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8686999999999997e-05 2023-01-15T14:36:06.474 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.474 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.474 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:06.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404687+0000", 2023-01-15T14:36:06.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.298042745 2023-01-15T14:36:06.475 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.475 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.475 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:06.475 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404735+0000", 2023-01-15T14:36:06.475 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.8306000000000001e-05 2023-01-15T14:36:06.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:06.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404772+0000", 2023-01-15T14:36:06.476 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6977000000000002e-05 2023-01-15T14:36:06.476 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.476 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.476 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:06.476 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404818+0000", 2023-01-15T14:36:06.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.6439e-05 2023-01-15T14:36:06.477 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.477 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:06.477 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.477 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.477 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.477 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:4 3.3 3:ca59e2b0:::smithi09025264-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:36:06.477 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:36:06.477 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 460.62065373899998, 2023-01-15T14:36:06.477 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.62995383500000002, 2023-01-15T14:36:06.477 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:06.478 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:06.478 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:06.478 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:06.478 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:06.478 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 4 2023-01-15T14:36:06.478 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.478 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:06.478 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.478 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:06.478 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:36:06.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:06.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:36:06.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.479 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.479 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.479 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:06.479 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775170+0000", 2023-01-15T14:36:06.479 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:36:06.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:06.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775174+0000", 2023-01-15T14:36:06.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3669999999999999e-06 2023-01-15T14:36:06.480 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.480 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.480 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:06.480 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775184+0000", 2023-01-15T14:36:06.480 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5869999999999999e-06 2023-01-15T14:36:06.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:06.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.775190+0000", 2023-01-15T14:36:06.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:36:06.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.481 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.481 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.481 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868440+0000", 2023-01-15T14:36:06.481 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.093250126000000003 2023-01-15T14:36:06.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:06.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.868555+0000", 2023-01-15T14:36:06.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115003 2023-01-15T14:36:06.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.302962+0000", 2023-01-15T14:36:06.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.43440729700000003 2023-01-15T14:36:06.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:06.483 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.303197+0000", 2023-01-15T14:36:06.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023508000000000001 2023-01-15T14:36:06.483 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.483 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.483 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404866+0000", 2023-01-15T14:36:06.484 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101668723 2023-01-15T14:36:06.484 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.484 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.484 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:06.484 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.404985+0000", 2023-01-15T14:36:06.484 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118853 2023-01-15T14:36:06.484 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.485 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:06.485 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.405124+0000", 2023-01-15T14:36:06.485 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013951999999999999 2023-01-15T14:36:06.485 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.485 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:06.485 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.485 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.485 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.486 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:144 3.9 3:9988ac9d:::smithi09025264-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:36:06.486 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:36:06.486 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 459.529315193, 2023-01-15T14:36:06.486 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.63423976199999998, 2023-01-15T14:36:06.486 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:06.486 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:06.486 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:06.486 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:06.487 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:06.487 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 144 2023-01-15T14:36:06.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.487 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:06.487 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.487 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:06.487 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:36:06.487 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.487 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.487 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.487 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:06.488 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866509+0000", 2023-01-15T14:36:06.488 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.488 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:06.488 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866508+0000", 2023-01-15T14:36:06.488 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:36:06.488 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.488 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:06.488 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866512+0000", 2023-01-15T14:36:06.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.5999999999999998e-06 2023-01-15T14:36:06.489 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.489 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.489 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:06.489 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866523+0000", 2023-01-15T14:36:06.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0740000000000001e-05 2023-01-15T14:36:06.489 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.489 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.489 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:06.489 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.866531+0000", 2023-01-15T14:36:06.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.3310000000000004e-06 2023-01-15T14:36:06.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.620 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.620 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.620 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867744+0000", 2023-01-15T14:36:06.620 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0012127399999999999 2023-01-15T14:36:06.620 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.867810+0000", 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6506000000000003e-05 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097225+0000", 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.22941514900000001 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:06.621 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.097295+0000", 2023-01-15T14:36:06.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:36:06.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500386+0000", 2023-01-15T14:36:06.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40309051699999998 2023-01-15T14:36:06.622 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.622 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.622 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:06.622 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500521+0000", 2023-01-15T14:36:06.622 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013561700000000001 2023-01-15T14:36:06.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.623 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:06.623 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.500749+0000", 2023-01-15T14:36:06.623 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022754900000000001 2023-01-15T14:36:06.623 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.623 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:06.623 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.623 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.623 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.623 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:509 3.9 3:9f86bdcb:::smithi09025264-22:head [write 2873212~588997] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:36:06.624 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:36:06.624 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 436.58089085, 2023-01-15T14:36:06.624 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69623497099999998, 2023-01-15T14:36:06.624 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:06.624 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:06.624 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:06.624 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:06.624 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:06.624 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 509 2023-01-15T14:36:06.624 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.625 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:06.625 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:06.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:36:06.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:06.626 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814933+0000", 2023-01-15T14:36:06.626 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.626 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.626 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.626 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:06.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.814931+0000", 2023-01-15T14:36:06.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:36:06.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:06.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815512+0000", 2023-01-15T14:36:06.627 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00058082200000000004 2023-01-15T14:36:06.627 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.627 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.627 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:06.627 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815553+0000", 2023-01-15T14:36:06.628 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1081999999999997e-05 2023-01-15T14:36:06.628 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.628 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.628 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:06.629 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815571+0000", 2023-01-15T14:36:06.629 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8028000000000001e-05 2023-01-15T14:36:06.629 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.630 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.630 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906452+0000", 2023-01-15T14:36:06.630 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090880586999999999 2023-01-15T14:36:06.630 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.630 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906564+0000", 2023-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000111976 2023-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208424+0000", 2023-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.301860199 2023-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.631 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:06.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.208718+0000", 2023-01-15T14:36:06.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029347299999999999 2023-01-15T14:36:06.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:36:06.632 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209175+0000", 2023-01-15T14:36:06.632 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00045692399999999999 2023-01-15T14:36:06.632 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.632 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.632 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:06.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209725+0000", 2023-01-15T14:36:06.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00055027600000000004 2023-01-15T14:36:06.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:06.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.209751+0000", 2023-01-15T14:36:06.633 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5812999999999998e-05 2023-01-15T14:36:06.633 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.633 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.633 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:06.633 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.210006+0000", 2023-01-15T14:36:06.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025564099999999997 2023-01-15T14:36:06.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:06.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311247+0000", 2023-01-15T14:36:06.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10124119199999999 2023-01-15T14:36:06.634 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.634 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.634 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:06.634 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311454+0000", 2023-01-15T14:36:06.634 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020695800000000001 2023-01-15T14:36:06.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.635 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:06.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511022+0000", 2023-01-15T14:36:06.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19956729200000001 2023-01-15T14:36:06.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.635 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.635 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:06.635 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511100+0000", 2023-01-15T14:36:06.635 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.8589000000000003e-05 2023-01-15T14:36:06.635 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.636 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.636 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:06.636 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511168+0000", 2023-01-15T14:36:06.636 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8039000000000004e-05 2023-01-15T14:36:06.636 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.637 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:06.637 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.637 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.637 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.637 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:510 3.9 3:9f86bdcb:::smithi09025264-22:head [setxattr _header (57) in=7b,truncate 3462209] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:36:06.637 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:36:06.638 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 436.58022738699998, 2023-01-15T14:36:06.638 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69609656399999997, 2023-01-15T14:36:06.638 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:06.638 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:06.638 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:06.638 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:06.638 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:06.638 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 510 2023-01-15T14:36:06.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.639 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:06.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:06.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:36:06.639 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.639 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.639 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.639 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:06.639 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815597+0000", 2023-01-15T14:36:06.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:06.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815596+0000", 2023-01-15T14:36:06.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:06.640 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.640 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.640 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:06.640 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815606+0000", 2023-01-15T14:36:06.640 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.7920000000000004e-06 2023-01-15T14:36:06.641 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.641 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.641 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:06.642 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815627+0000", 2023-01-15T14:36:06.642 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1239000000000001e-05 2023-01-15T14:36:06.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:06.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815637+0000", 2023-01-15T14:36:06.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0349e-05 2023-01-15T14:36:06.643 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.643 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.643 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.643 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906600+0000", 2023-01-15T14:36:06.643 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.090963098000000006 2023-01-15T14:36:06.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:06.644 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.906717+0000", 2023-01-15T14:36:06.644 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000116759 2023-01-15T14:36:06.644 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.644 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.644 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310000+0000", 2023-01-15T14:36:06.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40328322599999999 2023-01-15T14:36:06.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.645 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:06.645 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310249+0000", 2023-01-15T14:36:06.645 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024874400000000002 2023-01-15T14:36:06.645 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.645 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,7", 2023-01-15T14:36:06.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310559+0000", 2023-01-15T14:36:06.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000310382 2023-01-15T14:36:06.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:06.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310951+0000", 2023-01-15T14:36:06.646 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00039124500000000003 2023-01-15T14:36:06.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:06.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.310975+0000", 2023-01-15T14:36:06.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4496000000000001e-05 2023-01-15T14:36:06.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:06.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311041+0000", 2023-01-15T14:36:06.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6309000000000005e-05 2023-01-15T14:36:06.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:06.777 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311260+0000", 2023-01-15T14:36:06.777 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021863399999999999 2023-01-15T14:36:06.777 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.777 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.777 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311588+0000", 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032813599999999998 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511351+0000", 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19976275700000001 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511507+0000", 2023-01-15T14:36:06.778 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015585100000000001 2023-01-15T14:36:06.779 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.779 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.779 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:06.779 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511693+0000", 2023-01-15T14:36:06.779 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000186627 2023-01-15T14:36:06.779 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.779 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:06.779 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.780 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.780 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.780 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:511 3.9 3:9f86bdcb:::smithi09025264-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2023-01-15T14:36:06.780 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:36:06.780 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 436.58017001899998, 2023-01-15T14:36:06.780 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.69666647500000001, 2023-01-15T14:36:06.780 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:06.780 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:06.780 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:06.780 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:06.781 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:06.781 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 511 2023-01-15T14:36:06.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.781 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:06.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.781 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:06.781 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:36:06.781 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.781 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.781 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815654+0000", 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815653+0000", 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:06.782 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815660+0000", 2023-01-15T14:36:06.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0509999999999999e-06 2023-01-15T14:36:06.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:06.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815683+0000", 2023-01-15T14:36:06.783 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2742999999999999e-05 2023-01-15T14:36:06.783 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.783 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.783 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:06.783 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.815698+0000", 2023-01-15T14:36:06.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5133999999999999e-05 2023-01-15T14:36:06.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.784 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.784 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908029+0000", 2023-01-15T14:36:06.784 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.092331448999999996 2023-01-15T14:36:06.784 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.784 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:06.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:49.908193+0000", 2023-01-15T14:36:06.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016342799999999999 2023-01-15T14:36:06.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.785 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311115+0000", 2023-01-15T14:36:06.785 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40292222300000002 2023-01-15T14:36:06.785 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.785 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.785 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:06.786 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.311224+0000", 2023-01-15T14:36:06.786 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010888899999999999 2023-01-15T14:36:06.786 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.786 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.786 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.786 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.511800+0000", 2023-01-15T14:36:06.786 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20057596899999999 2023-01-15T14:36:06.786 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.786 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.787 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:06.787 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512036+0000", 2023-01-15T14:36:06.787 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236247 2023-01-15T14:36:06.787 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.787 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.787 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:06.787 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:50.512321+0000", 2023-01-15T14:36:06.787 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028457599999999998 2023-01-15T14:36:06.787 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.788 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:06.788 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.788 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.788 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.788 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:693 3.3 3:ca59e2b0:::smithi09025264-1:head [sparse-read 0~2016807] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2023-01-15T14:36:06.788 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:36:06.788 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 418.49743667199999, 2023-01-15T14:36:06.788 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70514199600000005, 2023-01-15T14:36:06.788 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:06.788 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:06.789 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:06.790 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:06.790 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:06.790 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 693 2023-01-15T14:36:06.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.790 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:06.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.790 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:06.790 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:36:06.790 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.790 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.790 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:06.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898388+0000", 2023-01-15T14:36:06.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:06.791 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898387+0000", 2023-01-15T14:36:06.791 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:36:06.791 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.791 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.791 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:06.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898390+0000", 2023-01-15T14:36:06.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.252e-06 2023-01-15T14:36:06.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.792 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:06.792 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.898401+0000", 2023-01-15T14:36:06.792 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0977e-05 2023-01-15T14:36:06.792 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.792 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:06.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:07.998532+0000", 2023-01-15T14:36:06.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100130443 2023-01-15T14:36:06.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.793 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602663+0000", 2023-01-15T14:36:06.793 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.60413100500000005 2023-01-15T14:36:06.793 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.793 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.793 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:06.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.602817+0000", 2023-01-15T14:36:06.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015440500000000001 2023-01-15T14:36:06.794 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.794 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.794 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:06.794 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:29:08.603530+0000", 2023-01-15T14:36:06.794 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071246299999999999 2023-01-15T14:36:06.795 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.795 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:06.795 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.795 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.795 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.795 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1421 3.a 3:5919d83e:::smithi09025264-13:head [write 1723287~549075] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:36:06.795 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:36:06.795 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 351.48352216299998, 2023-01-15T14:36:06.795 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61118175699999999, 2023-01-15T14:36:06.795 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:06.795 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1421 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.796 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.797 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:06.797 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912302+0000", 2023-01-15T14:36:06.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.797 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:06.797 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.912300+0000", 2023-01-15T14:36:06.797 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:36:06.797 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.797 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.797 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:06.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913127+0000", 2023-01-15T14:36:06.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00082704000000000002 2023-01-15T14:36:06.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:06.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913158+0000", 2023-01-15T14:36:06.798 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1146999999999998e-05 2023-01-15T14:36:06.798 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.798 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.798 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:06.798 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913177+0000", 2023-01-15T14:36:06.799 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9156000000000001e-05 2023-01-15T14:36:06.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.799 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.799 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913384+0000", 2023-01-15T14:36:06.799 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000206589 2023-01-15T14:36:06.799 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.799 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.799 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:36:06.799 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913408+0000", 2023-01-15T14:36:06.799 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3493e-05 2023-01-15T14:36:06.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.800 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.800 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.800 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318579+0000", 2023-01-15T14:36:06.800 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40517134399999999 2023-01-15T14:36:06.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.800 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.800 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:06.800 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.318806+0000", 2023-01-15T14:36:06.800 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022680899999999999 2023-01-15T14:36:06.800 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:36:06.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319060+0000", 2023-01-15T14:36:06.801 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000254069 2023-01-15T14:36:06.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.801 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:06.801 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319527+0000", 2023-01-15T14:36:06.801 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000467081 2023-01-15T14:36:06.801 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.801 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:06.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319544+0000", 2023-01-15T14:36:06.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6949999999999999e-05 2023-01-15T14:36:06.802 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.802 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.802 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:06.802 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319778+0000", 2023-01-15T14:36:06.802 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023391500000000001 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421126+0000", 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101348626 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522834+0000", 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10170761 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.803 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:06.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523142+0000", 2023-01-15T14:36:06.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854800000000001 2023-01-15T14:36:06.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:06.804 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523251+0000", 2023-01-15T14:36:06.804 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108411 2023-01-15T14:36:06.804 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.804 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.804 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523484+0000", 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023292500000000001 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1422 3.a 3:5919d83e:::smithi09025264-13:head [write 2781837~703343] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 351.48258336700002, 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.61062001700000001, 2023-01-15T14:36:06.805 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:06.806 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:06.806 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:06.806 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:06.806 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:06.806 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1422 2023-01-15T14:36:06.806 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.806 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:06.806 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.806 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:06.806 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:36:06.806 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913241+0000", 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.913239+0000", 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.807 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914075+0000", 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00083594099999999998 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914104+0000", 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8938999999999999e-05 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:06.808 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914121+0000", 2023-01-15T14:36:06.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7306999999999999e-05 2023-01-15T14:36:06.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914234+0000", 2023-01-15T14:36:06.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000112737 2023-01-15T14:36:06.809 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.809 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.809 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:36:06.809 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914257+0000", 2023-01-15T14:36:06.809 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3156e-05 2023-01-15T14:36:06.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:06.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319798+0000", 2023-01-15T14:36:06.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40554153100000001 2023-01-15T14:36:06.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.810 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.810 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:06.810 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.319991+0000", 2023-01-15T14:36:06.810 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019217999999999999 2023-01-15T14:36:06.810 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:36:06.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320237+0000", 2023-01-15T14:36:06.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024621099999999998 2023-01-15T14:36:06.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.811 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.811 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:06.811 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320759+0000", 2023-01-15T14:36:06.811 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00052174700000000003 2023-01-15T14:36:06.811 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.320776+0000", 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7173e-05 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.321080+0000", 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030424400000000001 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.812 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421162+0000", 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100082085 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522963+0000", 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10180109399999999 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523576+0000", 2023-01-15T14:36:06.813 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061277499999999997 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523664+0000", 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.8002000000000002e-05 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523861+0000", 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019685899999999999 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.814 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:06.815 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:06.815 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.815 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.815 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1423 3.a 3:5919d83e:::smithi09025264-13:head [setxattr _header (57) in=7b,truncate 3630312] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:36:06.815 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:36:06.815 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 351.48166083199999, 2023-01-15T14:36:06.815 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70862486899999999, 2023-01-15T14:36:06.815 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:06.815 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:06.815 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:06.815 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:06.816 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:06.816 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1423 2023-01-15T14:36:06.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.816 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:06.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:06.816 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:36:06.816 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.816 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.816 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.816 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:06.817 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914163+0000", 2023-01-15T14:36:06.817 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:06.817 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:06.817 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:06.817 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.030 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914162+0000", 2023-01-15T14:36:07.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:36:07.031 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.031 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.031 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.031 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914172+0000", 2023-01-15T14:36:07.031 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0267e-05 2023-01-15T14:36:07.032 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.032 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.032 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.032 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914188+0000", 2023-01-15T14:36:07.032 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6212999999999999e-05 2023-01-15T14:36:07.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.033 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.033 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.033 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914201+0000", 2023-01-15T14:36:07.033 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3073e-05 2023-01-15T14:36:07.033 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.034 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.034 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.034 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914315+0000", 2023-01-15T14:36:07.034 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011357399999999999 2023-01-15T14:36:07.034 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:36:07.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914334+0000", 2023-01-15T14:36:07.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9004e-05 2023-01-15T14:36:07.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.035 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.035 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.035 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421258+0000", 2023-01-15T14:36:07.035 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.506924606 2023-01-15T14:36:07.035 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.036 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:07.036 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421518+0000", 2023-01-15T14:36:07.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025950499999999999 2023-01-15T14:36:07.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.036 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:36:07.036 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.421843+0000", 2023-01-15T14:36:07.036 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00032498299999999999 2023-01-15T14:36:07.036 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.036 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.037 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:07.037 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422222+0000", 2023-01-15T14:36:07.037 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037867599999999999 2023-01-15T14:36:07.037 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.037 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:07.037 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422245+0000", 2023-01-15T14:36:07.037 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3096999999999999e-05 2023-01-15T14:36:07.037 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.037 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.037 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.422343+0000", 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8487999999999996e-05 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.522890+0000", 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10054658900000001 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523075+0000", 2023-01-15T14:36:07.038 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000185386 2023-01-15T14:36:07.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622151+0000", 2023-01-15T14:36:07.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099075724000000004 2023-01-15T14:36:07.039 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.039 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.039 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:07.039 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622244+0000", 2023-01-15T14:36:07.039 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.3387999999999994e-05 2023-01-15T14:36:07.040 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.040 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.040 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:07.040 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622788+0000", 2023-01-15T14:36:07.040 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000543973 2023-01-15T14:36:07.040 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.040 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.040 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.041 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.041 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.041 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1424 3.a 3:5919d83e:::smithi09025264-13:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:36:07.041 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:36:07.041 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 351.48160546000003, 2023-01-15T14:36:07.041 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70984700199999995, 2023-01-15T14:36:07.041 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:07.041 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:07.042 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:07.042 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:07.042 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:07.042 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1424 2023-01-15T14:36:07.042 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.042 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:07.042 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.042 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:07.042 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:36:07.042 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.042 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:07.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914219+0000", 2023-01-15T14:36:07.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.043 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.043 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914218+0000", 2023-01-15T14:36:07.043 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:07.043 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.043 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.044 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.044 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914224+0000", 2023-01-15T14:36:07.044 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6769999999999998e-06 2023-01-15T14:36:07.044 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.044 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.044 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.044 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:14.914242+0000", 2023-01-15T14:36:07.045 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7759000000000001e-05 2023-01-15T14:36:07.045 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.045 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.045 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014331+0000", 2023-01-15T14:36:07.045 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100088819 2023-01-15T14:36:07.045 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.045 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.045 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.045 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114642+0000", 2023-01-15T14:36:07.045 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100311027 2023-01-15T14:36:07.046 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.046 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.046 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:36:07.046 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114682+0000", 2023-01-15T14:36:07.046 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0028999999999999e-05 2023-01-15T14:36:07.046 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.046 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.046 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.046 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.523918+0000", 2023-01-15T14:36:07.046 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40923631500000002 2023-01-15T14:36:07.046 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524124+0000", 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020524900000000001 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.622921+0000", 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098797069000000001 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.047 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:07.048 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.623328+0000", 2023-01-15T14:36:07.048 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000407539 2023-01-15T14:36:07.048 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.048 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.048 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.624066+0000", 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00073762899999999995 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1430 3.a 3:52df5138:::smithi09025264-25:head [copy-from ver 59] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 351.38144569899998, 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81872653100000004, 2023-01-15T14:36:07.049 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:07.050 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:07.050 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:07.050 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:07.050 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:07.050 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1430 2023-01-15T14:36:07.050 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.050 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:07.050 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.050 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:07.050 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:36:07.050 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014379+0000", 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014376+0000", 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.051 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.052 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014391+0000", 2023-01-15T14:36:07.052 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4922999999999999e-05 2023-01-15T14:36:07.052 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.052 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.052 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.052 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014417+0000", 2023-01-15T14:36:07.052 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5698999999999999e-05 2023-01-15T14:36:07.052 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.052 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.052 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.052 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014435+0000", 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7827000000000001e-05 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114759+0000", 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100324859 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114771+0000", 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1919999999999999e-05 2023-01-15T14:36:07.053 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524222+0000", 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40945100800000001 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524545+0000", 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000322367 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.054 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:36:07.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.626981+0000", 2023-01-15T14:36:07.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102436308 2023-01-15T14:36:07.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.055 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.055 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:07.055 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627874+0000", 2023-01-15T14:36:07.055 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089274899999999997 2023-01-15T14:36:07.055 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:07.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.627930+0000", 2023-01-15T14:36:07.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6122000000000002e-05 2023-01-15T14:36:07.056 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.056 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.056 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.056 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628609+0000", 2023-01-15T14:36:07.056 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00067958700000000005 2023-01-15T14:36:07.057 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.057 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:07.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628798+0000", 2023-01-15T14:36:07.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018809 2023-01-15T14:36:07.057 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.057 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.057 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:07.057 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628870+0000", 2023-01-15T14:36:07.057 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2844999999999998e-05 2023-01-15T14:36:07.203 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.203 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.203 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.203 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832404+0000", 2023-01-15T14:36:07.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20353310299999999 2023-01-15T14:36:07.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:07.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.832606+0000", 2023-01-15T14:36:07.204 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020207999999999999 2023-01-15T14:36:07.204 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.204 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.204 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:07.204 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833105+0000", 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049945399999999998 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:1431 3.a 3:52df5138:::smithi09025264-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e128)", 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 351.38135441100002, 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.81967516600000001, 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:07.205 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:07.206 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:07.206 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:07.206 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:07.206 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 1431 2023-01-15T14:36:07.206 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.206 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:07.206 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.206 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:07.206 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:36:07.206 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.206 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014470+0000", 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014469+0000", 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.207 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014477+0000", 2023-01-15T14:36:07.208 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9300000000000003e-06 2023-01-15T14:36:07.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.208 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.208 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.208 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.014495+0000", 2023-01-15T14:36:07.208 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8363999999999999e-05 2023-01-15T14:36:07.208 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.208 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.114610+0000", 2023-01-15T14:36:07.209 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10011521700000001 2023-01-15T14:36:07.209 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.209 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.209 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.209 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214948+0000", 2023-01-15T14:36:07.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100337739 2023-01-15T14:36:07.210 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.210 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:36:07.210 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.214962+0000", 2023-01-15T14:36:07.210 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4516e-05 2023-01-15T14:36:07.210 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.210 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.210 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.210 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.524874+0000", 2023-01-15T14:36:07.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30991196599999998 2023-01-15T14:36:07.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.211 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.211 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:36:07.211 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.525013+0000", 2023-01-15T14:36:07.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013873999999999999 2023-01-15T14:36:07.211 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.211 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.211 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.211 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628222+0000", 2023-01-15T14:36:07.211 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103208721 2023-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.628504+0000", 2023-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028245700000000001 2023-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833214+0000", 2023-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.204709691 2023-01-15T14:36:07.212 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:07.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.833555+0000", 2023-01-15T14:36:07.213 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00034094000000000003 2023-01-15T14:36:07.213 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.213 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.213 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:07.213 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:30:15.834145+0000", 2023-01-15T14:36:07.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00059006900000000003 2023-01-15T14:36:07.214 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.214 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.214 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.214 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.214 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.214 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2439 3.3 3:c11c7d81:::smithi09025264-11:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:36:07.214 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:36:07.214 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 251.31244990600001, 2023-01-15T14:36:07.214 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70719632099999996, 2023-01-15T14:36:07.214 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2439 2023-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:07.215 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083374+0000", 2023-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.216 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083373+0000", 2023-01-15T14:36:07.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:07.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083380+0000", 2023-01-15T14:36:07.217 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.1899999999999998e-06 2023-01-15T14:36:07.217 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.217 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.217 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.217 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083391+0000", 2023-01-15T14:36:07.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0366e-05 2023-01-15T14:36:07.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083397+0000", 2023-01-15T14:36:07.218 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2740000000000004e-06 2023-01-15T14:36:07.218 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.218 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.218 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.218 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083648+0000", 2023-01-15T14:36:07.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025052599999999999 2023-01-15T14:36:07.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:36:07.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083666+0000", 2023-01-15T14:36:07.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8848999999999999e-05 2023-01-15T14:36:07.219 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.219 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.219 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.219 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237725+0000", 2023-01-15T14:36:07.219 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.154058361 2023-01-15T14:36:07.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:07.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.237945+0000", 2023-01-15T14:36:07.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021978300000000001 2023-01-15T14:36:07.220 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.220 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.220 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.220 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.386967+0000", 2023-01-15T14:36:07.220 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.149022562 2023-01-15T14:36:07.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.221 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:07.221 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387230+0000", 2023-01-15T14:36:07.221 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026256100000000002 2023-01-15T14:36:07.221 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.221 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,3", 2023-01-15T14:36:07.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.387695+0000", 2023-01-15T14:36:07.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046480099999999998 2023-01-15T14:36:07.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:07.222 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388112+0000", 2023-01-15T14:36:07.222 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00041713200000000002 2023-01-15T14:36:07.222 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.222 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.222 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:07.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388130+0000", 2023-01-15T14:36:07.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8315999999999999e-05 2023-01-15T14:36:07.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.223 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:07.223 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388189+0000", 2023-01-15T14:36:07.223 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8665999999999997e-05 2023-01-15T14:36:07.223 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.223 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.224 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:07.224 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.388829+0000", 2023-01-15T14:36:07.224 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00063992699999999999 2023-01-15T14:36:07.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.489989+0000", 2023-01-15T14:36:07.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10116059199999999 2023-01-15T14:36:07.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.225 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.225 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.225 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790300+0000", 2023-01-15T14:36:07.225 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30031108699999998 2023-01-15T14:36:07.225 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790396+0000", 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.5721000000000001e-05 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790571+0000", 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017464999999999999 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.226 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.227 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.227 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.227 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2440 3.3 3:c11c7d81:::smithi09025264-11:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e228)", 2023-01-15T14:36:07.227 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:36:07.227 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 251.31235817500001, 2023-01-15T14:36:07.227 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.70759506000000005, 2023-01-15T14:36:07.227 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:07.227 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:07.227 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:07.227 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:07.227 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:07.228 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2440 2023-01-15T14:36:07.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.228 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:07.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:07.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:36:07.228 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.228 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.228 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.228 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:07.228 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083466+0000", 2023-01-15T14:36:07.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.229 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083465+0000", 2023-01-15T14:36:07.229 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:07.229 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.229 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.229 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083470+0000", 2023-01-15T14:36:07.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2630000000000003e-06 2023-01-15T14:36:07.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083481+0000", 2023-01-15T14:36:07.230 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0699999999999999e-05 2023-01-15T14:36:07.230 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.230 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.230 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.230 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.083488+0000", 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.8870000000000003e-06 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180331+0000", 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.096843296999999995 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.180342+0000", 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.031e-05 2023-01-15T14:36:07.231 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238012+0000", 2023-01-15T14:36:07.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.057670829 2023-01-15T14:36:07.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.232 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.232 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:07.232 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.238257+0000", 2023-01-15T14:36:07.232 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000244959 2023-01-15T14:36:07.232 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590126+0000", 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35186851699999999 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.590389+0000", 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026276799999999998 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.233 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790692+0000", 2023-01-15T14:36:07.234 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20030350299999999 2023-01-15T14:36:07.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.234 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.234 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:07.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.790891+0000", 2023-01-15T14:36:07.234 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019893999999999999 2023-01-15T14:36:07.234 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.234 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.234 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:07.234 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:55.791061+0000", 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169913 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:2450 3.a 3:52c3e133:::smithi09025264-3:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e230)", 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 249.54850556299999, 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.68251353100000001, 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:07.235 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:07.236 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:07.236 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:07.236 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:07.236 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 2450 2023-01-15T14:36:07.236 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.236 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:07.236 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.236 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:07.236 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:36:07.236 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.236 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847319+0000", 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847318+0000", 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.237 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847324+0000", 2023-01-15T14:36:07.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9660000000000001e-06 2023-01-15T14:36:07.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847339+0000", 2023-01-15T14:36:07.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5582000000000001e-05 2023-01-15T14:36:07.238 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.238 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.238 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.238 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.847349+0000", 2023-01-15T14:36:07.238 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0139999999999999e-05 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947644+0000", 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10029455600000001 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:56.947657+0000", 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2796000000000001e-05 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.239 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125638+0000", 2023-01-15T14:36:07.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.17798127799999999 2023-01-15T14:36:07.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:07.240 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.125780+0000", 2023-01-15T14:36:07.240 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014173000000000001 2023-01-15T14:36:07.240 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.240 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.240 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.241 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529035+0000", 2023-01-15T14:36:07.241 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40325542199999997 2023-01-15T14:36:07.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.241 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.241 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:07.241 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529366+0000", 2023-01-15T14:36:07.241 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033111700000000002 2023-01-15T14:36:07.241 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.241 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.241 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:07.241 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:31:57.529832+0000", 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00046592399999999999 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3390 3.1b 3:d85f42d9:::smithi09025264-45:head [write 1944996~791497] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.335826854, 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.611364087, 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:07.242 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:07.243 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:07.243 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:07.243 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:07.243 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3390 2023-01-15T14:36:07.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.243 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:07.243 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.243 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:07.243 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:36:07.243 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.243 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.244 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:07.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059997+0000", 2023-01-15T14:36:07.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.244 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.244 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.059995+0000", 2023-01-15T14:36:07.244 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:36:07.244 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.244 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061164+0000", 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001168325 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061194+0000", 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.0485999999999999e-05 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.245 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061213+0000", 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.891e-05 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159748+0000", 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.098535175000000003 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159766+0000", 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7486000000000002e-05 2023-01-15T14:36:07.246 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263011+0000", 2023-01-15T14:36:07.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10324554699999999 2023-01-15T14:36:07.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.247 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:07.247 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263253+0000", 2023-01-15T14:36:07.247 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000241568 2023-01-15T14:36:07.247 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.247 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.248 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.248 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466464+0000", 2023-01-15T14:36:07.439 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.203211006 2023-01-15T14:36:07.439 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.439 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.439 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466638+0000", 2023-01-15T14:36:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00017412100000000001 2023-01-15T14:36:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:36:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.466911+0000", 2023-01-15T14:36:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000272971 2023-01-15T14:36:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:07.440 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467425+0000", 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00051373 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467457+0000", 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.2558999999999998e-05 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467906+0000", 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000448985 2023-01-15T14:36:07.441 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.442 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:07.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469895+0000", 2023-01-15T14:36:07.442 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0019888810000000001 2023-01-15T14:36:07.442 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.442 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.442 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470611+0000", 2023-01-15T14:36:07.442 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00071556699999999996 2023-01-15T14:36:07.442 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.442 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671178+0000", 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20056716699999999 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671277+0000", 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.9594999999999998e-05 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.671361+0000", 2023-01-15T14:36:07.443 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4048000000000006e-05 2023-01-15T14:36:07.444 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.444 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.444 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.444 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.444 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.444 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3391 3.1b 3:d85f42d9:::smithi09025264-45:head [write 3196560~554574] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:36:07.444 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:36:07.444 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.33458373100001, 2023-01-15T14:36:07.444 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71066412899999998, 2023-01-15T14:36:07.444 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:07.445 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:07.445 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:07.445 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:07.445 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:07.445 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3391 2023-01-15T14:36:07.445 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.445 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:07.445 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.445 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:07.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:36:07.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.446 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:07.446 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061240+0000", 2023-01-15T14:36:07.446 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.446 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.446 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.061238+0000", 2023-01-15T14:36:07.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:36:07.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.447 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.447 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062134+0000", 2023-01-15T14:36:07.447 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00089621299999999998 2023-01-15T14:36:07.447 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.447 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062160+0000", 2023-01-15T14:36:07.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5908999999999999e-05 2023-01-15T14:36:07.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.448 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.448 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062172+0000", 2023-01-15T14:36:07.448 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.189e-05 2023-01-15T14:36:07.448 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.448 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159827+0000", 2023-01-15T14:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097654866000000007 2023-01-15T14:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159846+0000", 2023-01-15T14:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8516e-05 2023-01-15T14:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.449 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263326+0000", 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103480766 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263535+0000", 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020861299999999999 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467576+0000", 2023-01-15T14:36:07.450 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20404118600000001 2023-01-15T14:36:07.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:07.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467736+0000", 2023-01-15T14:36:07.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016000699999999999 2023-01-15T14:36:07.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.451 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.451 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:36:07.451 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.467987+0000", 2023-01-15T14:36:07.451 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025042199999999998 2023-01-15T14:36:07.451 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:07.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468369+0000", 2023-01-15T14:36:07.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038238600000000002 2023-01-15T14:36:07.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.452 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:07.452 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468387+0000", 2023-01-15T14:36:07.452 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7730000000000001e-05 2023-01-15T14:36:07.452 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.452 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468719+0000", 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033272599999999999 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470317+0000", 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001597336 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.453 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570851+0000", 2023-01-15T14:36:07.454 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10053426 2023-01-15T14:36:07.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.454 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.454 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.454 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771618+0000", 2023-01-15T14:36:07.454 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200767152 2023-01-15T14:36:07.454 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.454 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.454 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:07.454 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771768+0000", 2023-01-15T14:36:07.454 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015015699999999999 2023-01-15T14:36:07.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.455 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.455 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:07.455 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.771905+0000", 2023-01-15T14:36:07.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013621099999999999 2023-01-15T14:36:07.455 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.455 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.455 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.455 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3392 3.1b 3:d85f42d9:::smithi09025264-45:head [setxattr _header (58) in=7b,truncate 3882483] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.333634496, 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71008243400000004, 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3392 2023-01-15T14:36:07.456 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.457 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:07.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:07.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:36:07.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.457 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.457 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:07.457 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062190+0000", 2023-01-15T14:36:07.457 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.457 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062188+0000", 2023-01-15T14:36:07.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:36:07.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062195+0000", 2023-01-15T14:36:07.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2529999999999998e-06 2023-01-15T14:36:07.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062208+0000", 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3003000000000001e-05 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062215+0000", 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.5200000000000003e-06 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159883+0000", 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.097667532000000001 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.159899+0000", 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6759999999999999e-05 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.263853+0000", 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10395361 2023-01-15T14:36:07.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264116+0000", 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026336099999999999 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468602+0000", 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20448550900000001 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.468867+0000", 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026495700000000002 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 1,7", 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469376+0000", 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00050893100000000001 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469789+0000", 2023-01-15T14:36:07.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000413158 2023-01-15T14:36:07.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:07.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469806+0000", 2023-01-15T14:36:07.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7513e-05 2023-01-15T14:36:07.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:07.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469884+0000", 2023-01-15T14:36:07.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7816000000000002e-05 2023-01-15T14:36:07.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.464 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:07.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470332+0000", 2023-01-15T14:36:07.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00044807700000000001 2023-01-15T14:36:07.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.464 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.464 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.570976+0000", 2023-01-15T14:36:07.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10064426899999999 2023-01-15T14:36:07.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772058+0000", 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20108148200000001 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772169+0000", 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011120099999999999 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:07.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772272+0000", 2023-01-15T14:36:07.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000102982 2023-01-15T14:36:07.466 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.466 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.466 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.466 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:3393 3.1b 3:d85f42d9:::smithi09025264-45:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e325)", 2023-01-15T14:36:07.466 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:36:07.466 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 151.33360042000001, 2023-01-15T14:36:07.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.71076842399999995, 2023-01-15T14:36:07.466 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 3393 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:07.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062224+0000", 2023-01-15T14:36:07.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062223+0000", 2023-01-15T14:36:07.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:07.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062227+0000", 2023-01-15T14:36:07.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.3900000000000003e-06 2023-01-15T14:36:07.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062237+0000", 2023-01-15T14:36:07.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4539999999999996e-06 2023-01-15T14:36:07.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.062243+0000", 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0329999999999997e-06 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161412+0000", 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099168708999999994 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for peered", 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.161424+0000", 2023-01-15T14:36:07.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2415999999999999e-05 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264185+0000", 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.1027609 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.264421+0000", 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023640900000000001 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.471 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.472 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.469985+0000", 2023-01-15T14:36:07.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.205564001 2023-01-15T14:36:07.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:07.472 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.470264+0000", 2023-01-15T14:36:07.472 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027918700000000002 2023-01-15T14:36:07.472 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.472 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.472 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772390+0000", 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30212558299999998 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772684+0000", 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029361799999999999 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:33:35.772992+0000", 2023-01-15T14:36:07.473 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030854100000000001 2023-01-15T14:36:07.474 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.474 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.474 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.474 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.474 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.474 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:36:07.474 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2023-01-15T14:36:07.476 INFO:tasks.ceph.osd.1.smithi090.stderr:2023-01-15T14:36:06.449+0000 7ffaee597700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 23469) UID: 0 2023-01-15T14:36:07.599 INFO:tasks.rados.rados.0.smithi090.stdout:1751: finishing write tid 1 to smithi09025264-26 2023-01-15T14:36:07.599 INFO:tasks.rados.rados.0.smithi090.stdout:1751: finishing write tid 2 to smithi09025264-26 2023-01-15T14:36:07.599 INFO:tasks.rados.rados.0.smithi090.stdout:1751: finishing write tid 3 to smithi09025264-26 2023-01-15T14:36:07.599 INFO:tasks.rados.rados.0.smithi090.stdout:1751: finishing write tid 4 to smithi09025264-26 2023-01-15T14:36:07.600 INFO:tasks.rados.rados.0.smithi090.stdout:1751: oid 26 updating version 0 to 358 2023-01-15T14:36:07.600 INFO:tasks.rados.rados.0.smithi090.stdout:1751: oid 26 updating version 358 to 360 2023-01-15T14:36:07.600 INFO:tasks.rados.rados.0.smithi090.stdout:1751: oid 26 version 360 is already newer than 359 2023-01-15T14:36:07.600 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 26 v 360 (ObjNum 622 snap 190 seq_num 622) dirty exists 2023-01-15T14:36:07.600 INFO:tasks.rados.rados.0.smithi090.stdout:1751: left oid 26 (ObjNum 622 snap 190 seq_num 622) 2023-01-15T14:36:07.600 INFO:tasks.rados.rados.0.smithi090.stdout:1749: finishing write tid 3 to smithi09025264-19 2023-01-15T14:36:07.600 INFO:tasks.rados.rados.0.smithi090.stdout:1749: finishing write tid 4 to smithi09025264-19 2023-01-15T14:36:07.600 INFO:tasks.rados.rados.0.smithi090.stdout:1749: oid 19 updating version 0 to 328 2023-01-15T14:36:07.601 INFO:tasks.rados.rados.0.smithi090.stdout:1749: oid 19 version 328 is already newer than 327 2023-01-15T14:36:07.601 INFO:tasks.rados.rados.0.smithi090.stdout:1749: oid 19 version 328 is already newer than 326 2023-01-15T14:36:07.601 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 19 v 328 (ObjNum 620 snap 190 seq_num 620) dirty exists 2023-01-15T14:36:07.601 INFO:tasks.rados.rados.0.smithi090.stdout:1749: left oid 19 (ObjNum 620 snap 190 seq_num 620) 2023-01-15T14:36:07.601 INFO:tasks.rados.rados.0.smithi090.stdout:1749: done (5 left) 2023-01-15T14:36:07.601 INFO:tasks.rados.rados.0.smithi090.stdout:1751: done (4 left) 2023-01-15T14:36:07.601 INFO:tasks.rados.rados.0.smithi090.stdout:1753: done (3 left) 2023-01-15T14:36:07.601 INFO:tasks.rados.rados.0.smithi090.stdout:1754: done (2 left) 2023-01-15T14:36:07.602 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:36:07.602 INFO:tasks.rados.rados.0.smithi090.stdout:1750: finishing write tid 2 to smithi09025264-14 2023-01-15T14:36:07.602 INFO:tasks.rados.rados.0.smithi090.stdout:1750: finishing write tid 3 to smithi09025264-14 2023-01-15T14:36:07.602 INFO:tasks.rados.rados.0.smithi090.stdout:1750: finishing write tid 4 to smithi09025264-14 2023-01-15T14:36:07.602 INFO:tasks.rados.rados.0.smithi090.stdout:1750: oid 14 updating version 0 to 464 2023-01-15T14:36:07.602 INFO:tasks.rados.rados.0.smithi090.stdout:1750: oid 14 version 464 is already newer than 462 2023-01-15T14:36:07.602 INFO:tasks.rados.rados.0.smithi090.stdout:1750: oid 14 version 464 is already newer than 463 2023-01-15T14:36:07.602 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 14 v 464 (ObjNum 621 snap 190 seq_num 621) dirty exists 2023-01-15T14:36:07.602 INFO:tasks.rados.rados.0.smithi090.stdout:1750: left oid 14 (ObjNum 621 snap 190 seq_num 621) 2023-01-15T14:36:07.602 INFO:tasks.rados.rados.0.smithi090.stdout:1750: done (1 left) 2023-01-15T14:36:07.603 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:36:07.603 INFO:teuthology.orchestra.run.smithi090.stderr:noscrub is set 2023-01-15T14:36:07.604 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:36:07.472+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:36:07.605 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:36:07.605 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:36:07.605 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.605 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "MOSDPGRecoveryDelete(3.50 e477,467 3:0b54190a:::smithi09025264-17:99,243'315)", 2023-01-15T14:36:07.605 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:36:07.518583+0000", 2023-01-15T14:36:07.606 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 0.076015632999999999, 2023-01-15T14:36:07.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.076053572, 2023-01-15T14:36:07.606 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:07.606 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "queued for pg", 2023-01-15T14:36:07.606 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:07.606 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.606 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:07.606 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:07.518583+0000", 2023-01-15T14:36:07.606 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.606 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:07.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:07.518583+0000", 2023-01-15T14:36:07.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:07.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.607 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.607 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:07.607 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:07.518582+0000", 2023-01-15T14:36:07.607 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:07.607 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:07.608 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:07.518592+0000", 2023-01-15T14:36:07.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0125e-05 2023-01-15T14:36:07.608 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.608 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.608 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:07.608 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:07.518617+0000", 2023-01-15T14:36:07.608 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.5194999999999998e-05 2023-01-15T14:36:07.609 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:07.609 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:07.609 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:07.609 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:07.518632+0000", 2023-01-15T14:36:07.609 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5150999999999999e-05 2023-01-15T14:36:07.609 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.609 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:07.609 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.609 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:07.862 INFO:teuthology.orchestra.run.smithi141.stdout: ], 2023-01-15T14:36:07.865 INFO:teuthology.orchestra.run.smithi141.stdout: "num_ops": 1 2023-01-15T14:36:07.865 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:36:07.866 INFO:tasks.ceph.osd.4.smithi141.stderr:2023-01-15T14:36:07.604+0000 7f515d9aa700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 23516) UID: 0 2023-01-15T14:36:07.867 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2023-01-15T14:36:07.869 INFO:tasks.rados.rados.0.smithi090.stdout:1752: finishing copy_from to smithi09025264-17 2023-01-15T14:36:07.869 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 431 (ObjNum 525 snap 158 seq_num 525) dirty exists 2023-01-15T14:36:07.869 INFO:tasks.rados.rados.0.smithi090.stdout:1752: finishing copy_from racing read to smithi09025264-17 2023-01-15T14:36:07.869 INFO:tasks.rados.rados.0.smithi090.stdout:1752: done (0 left) 2023-01-15T14:36:07.870 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:36:07.868+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:36:07.970 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:36:07.968+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:36:08.067 INFO:tasks.rados.rados.0.smithi090.stdout:1755: done (0 left) 2023-01-15T14:36:08.067 INFO:tasks.rados.rados.0.smithi090.stdout:1756: snap_remove snap 183 2023-01-15T14:36:08.067 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:36:08.068 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [], 2023-01-15T14:36:08.068 INFO:teuthology.orchestra.run.smithi141.stdout: "complaint_time": 30, 2023-01-15T14:36:08.068 INFO:teuthology.orchestra.run.smithi141.stdout: "num_blocked_ops": 0 2023-01-15T14:36:08.068 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:36:08.068 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2023-01-15T14:36:08.071 INFO:tasks.ceph.osd.7.smithi141.stderr:2023-01-15T14:36:08.068+0000 7f956dc5a700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 23521) UID: 0 2023-01-15T14:36:08.148 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2023-01-15T14:36:08.253 INFO:teuthology.orchestra.run.smithi141.stdout:{ 2023-01-15T14:36:08.253 INFO:teuthology.orchestra.run.smithi141.stdout: "size": 20, 2023-01-15T14:36:08.254 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 600, 2023-01-15T14:36:08.254 INFO:teuthology.orchestra.run.smithi141.stdout: "ops": [ 2023-01-15T14:36:08.254 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.254 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:5 3.5 3:a1c47f99:::smithi09025264-2:head [write 416807~475249,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:36:08.254 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:36:08.254 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 462.39875748499998, 2023-01-15T14:36:08.254 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.40548699700000002, 2023-01-15T14:36:08.254 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:08.254 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:08.254 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:08.255 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:08.255 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:08.255 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 5 2023-01-15T14:36:08.255 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.255 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:08.255 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.255 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:08.255 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:36:08.255 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.255 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.255 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:08.256 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781746+0000", 2023-01-15T14:36:08.256 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.256 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:08.256 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.781744+0000", 2023-01-15T14:36:08.256 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:36:08.256 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.256 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.256 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:08.257 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782566+0000", 2023-01-15T14:36:08.257 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000821578 2023-01-15T14:36:08.257 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.257 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.257 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:08.257 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782588+0000", 2023-01-15T14:36:08.257 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.226e-05 2023-01-15T14:36:08.257 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.257 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.257 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:08.257 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.782601+0000", 2023-01-15T14:36:08.258 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2998e-05 2023-01-15T14:36:08.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.258 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.258 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.258 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882850+0000", 2023-01-15T14:36:08.258 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100248908 2023-01-15T14:36:08.258 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.258 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op must wait for map", 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882871+0000", 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0873000000000002e-05 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083679+0000", 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.200808083 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:08.259 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.083948+0000", 2023-01-15T14:36:08.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000269342 2023-01-15T14:36:08.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:36:08.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084161+0000", 2023-01-15T14:36:08.260 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021326900000000001 2023-01-15T14:36:08.260 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.260 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.260 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:08.260 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084655+0000", 2023-01-15T14:36:08.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00049371300000000001 2023-01-15T14:36:08.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:08.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084674+0000", 2023-01-15T14:36:08.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9012999999999999e-05 2023-01-15T14:36:08.261 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.261 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.261 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:08.261 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084924+0000", 2023-01-15T14:36:08.261 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024989899999999999 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085221+0000", 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029719600000000001 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186953+0000", 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10173174 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.262 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:08.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187026+0000", 2023-01-15T14:36:08.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2957e-05 2023-01-15T14:36:08.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:08.263 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187124+0000", 2023-01-15T14:36:08.263 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.8146000000000005e-05 2023-01-15T14:36:08.263 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.263 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.263 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:08.264 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187233+0000", 2023-01-15T14:36:08.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010931599999999999 2023-01-15T14:36:08.264 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:08.264 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:08.264 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:08.264 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.264 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.264 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:6 3.5 3:a1c47f99:::smithi09025264-2:head [write 1542129~474678] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:36:08.264 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:36:08.264 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 462.393504613, 2023-01-15T14:36:08.264 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50419954199999995, 2023-01-15T14:36:08.265 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:08.265 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:08.265 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:08.265 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:08.265 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:08.265 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 6 2023-01-15T14:36:08.265 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.265 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:08.265 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.265 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:08.265 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:36:08.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:08.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786999+0000", 2023-01-15T14:36:08.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:08.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.786997+0000", 2023-01-15T14:36:08.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787486+0000", 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00048882999999999999 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787514+0000", 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7591999999999999e-05 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.268 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:08.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787526+0000", 2023-01-15T14:36:08.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.1894e-05 2023-01-15T14:36:08.268 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.268 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.268 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882966+0000", 2023-01-15T14:36:08.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095440450999999996 2023-01-15T14:36:08.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.269 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:36:08.269 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.882973+0000", 2023-01-15T14:36:08.269 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2570000000000001e-06 2023-01-15T14:36:08.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.269 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.269 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.269 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084809+0000", 2023-01-15T14:36:08.269 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.201835763 2023-01-15T14:36:08.269 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.270 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:08.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084928+0000", 2023-01-15T14:36:08.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000118946 2023-01-15T14:36:08.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.270 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.270 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187347+0000", 2023-01-15T14:36:08.270 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102419285 2023-01-15T14:36:08.270 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.270 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:08.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187450+0000", 2023-01-15T14:36:08.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010286299999999999 2023-01-15T14:36:08.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:36:08.271 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187735+0000", 2023-01-15T14:36:08.271 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028429900000000001 2023-01-15T14:36:08.271 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.271 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.271 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:08.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188113+0000", 2023-01-15T14:36:08.272 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037883200000000001 2023-01-15T14:36:08.272 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.272 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.272 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:08.272 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188138+0000", 2023-01-15T14:36:08.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4110000000000001e-05 2023-01-15T14:36:08.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:08.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188389+0000", 2023-01-15T14:36:08.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025173900000000001 2023-01-15T14:36:08.273 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.273 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.273 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:08.273 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189251+0000", 2023-01-15T14:36:08.273 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00086129800000000003 2023-01-15T14:36:08.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:08.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290989+0000", 2023-01-15T14:36:08.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101738524 2023-01-15T14:36:08.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.274 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.274 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:08.274 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291075+0000", 2023-01-15T14:36:08.274 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.5469000000000004e-05 2023-01-15T14:36:08.274 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:08.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291144+0000", 2023-01-15T14:36:08.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9666999999999996e-05 2023-01-15T14:36:08.275 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.275 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.275 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:08.275 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291199+0000", 2023-01-15T14:36:08.275 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4413000000000002e-05 2023-01-15T14:36:08.275 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:08.275 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:08.276 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:08.276 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.276 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.276 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:7 3.5 3:a1c47f99:::smithi09025264-2:head [setxattr _header (55) in=7b,truncate 2016807] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:36:08.276 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:36:08.276 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 462.39295815600002, 2023-01-15T14:36:08.455 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50407251399999997, 2023-01-15T14:36:08.457 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:08.457 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:08.457 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:08.457 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:08.457 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:08.457 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 7 2023-01-15T14:36:08.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.458 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:08.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:08.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:36:08.458 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.458 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.458 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.458 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:08.458 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787546+0000", 2023-01-15T14:36:08.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:08.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787545+0000", 2023-01-15T14:36:08.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:08.459 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.459 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.459 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:08.459 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787551+0000", 2023-01-15T14:36:08.459 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.8669999999999999e-06 2023-01-15T14:36:08.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:08.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787559+0000", 2023-01-15T14:36:08.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.9780000000000005e-06 2023-01-15T14:36:08.460 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.460 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.460 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:08.460 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787564+0000", 2023-01-15T14:36:08.460 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.5709999999999997e-06 2023-01-15T14:36:08.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883054+0000", 2023-01-15T14:36:08.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.095489722999999999 2023-01-15T14:36:08.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.461 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.461 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:36:08.461 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.883061+0000", 2023-01-15T14:36:08.461 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.2760000000000003e-06 2023-01-15T14:36:08.461 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.084987+0000", 2023-01-15T14:36:08.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20192545000000001 2023-01-15T14:36:08.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.462 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:08.462 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085190+0000", 2023-01-15T14:36:08.462 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020288899999999999 2023-01-15T14:36:08.462 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.462 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188374+0000", 2023-01-15T14:36:08.463 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103184449 2023-01-15T14:36:08.463 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.463 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.463 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:08.463 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188543+0000", 2023-01-15T14:36:08.464 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016925299999999999 2023-01-15T14:36:08.464 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.464 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:36:08.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188748+0000", 2023-01-15T14:36:08.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00020473600000000001 2023-01-15T14:36:08.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.465 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.465 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:08.465 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188986+0000", 2023-01-15T14:36:08.465 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023780900000000001 2023-01-15T14:36:08.465 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:08.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189006+0000", 2023-01-15T14:36:08.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0604000000000001e-05 2023-01-15T14:36:08.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.466 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:08.466 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189084+0000", 2023-01-15T14:36:08.466 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7526000000000003e-05 2023-01-15T14:36:08.466 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.466 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:08.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189272+0000", 2023-01-15T14:36:08.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00018762 2023-01-15T14:36:08.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:08.467 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291333+0000", 2023-01-15T14:36:08.467 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102061185 2023-01-15T14:36:08.467 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.467 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.467 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:08.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291466+0000", 2023-01-15T14:36:08.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013325699999999999 2023-01-15T14:36:08.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.468 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.468 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:08.468 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291535+0000", 2023-01-15T14:36:08.468 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.9309999999999999e-05 2023-01-15T14:36:08.468 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.469 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:08.469 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291618+0000", 2023-01-15T14:36:08.469 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2773000000000002e-05 2023-01-15T14:36:08.469 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:08.469 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:08.469 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:08.469 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.469 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.469 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:8 3.5 3:a1c47f99:::smithi09025264-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:36:08.469 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:36:08.470 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 462.39293217400001, 2023-01-15T14:36:08.470 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.50442076000000002, 2023-01-15T14:36:08.470 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:08.470 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:08.470 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:08.470 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:08.470 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:08.646 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 8 2023-01-15T14:36:08.646 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.646 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:08.646 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.646 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:08.646 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:36:08.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:08.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787572+0000", 2023-01-15T14:36:08.647 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.647 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.647 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.647 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:08.647 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787571+0000", 2023-01-15T14:36:08.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:36:08.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:08.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787575+0000", 2023-01-15T14:36:08.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7890000000000001e-06 2023-01-15T14:36:08.648 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.648 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.648 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:08.648 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787589+0000", 2023-01-15T14:36:08.648 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4212999999999999e-05 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.787594+0000", 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.1429999999999997e-06 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983143+0000", 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.195549003 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.649 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.650 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:36:08.650 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983153+0000", 2023-01-15T14:36:08.650 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.4369999999999996e-06 2023-01-15T14:36:08.650 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.650 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.650 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.650 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085286+0000", 2023-01-15T14:36:08.652 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102133064 2023-01-15T14:36:08.652 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.653 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.653 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:08.653 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.085401+0000", 2023-01-15T14:36:08.653 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000115176 2023-01-15T14:36:08.653 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.654 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.654 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.654 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189116+0000", 2023-01-15T14:36:08.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103715444 2023-01-15T14:36:08.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.842 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:08.842 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.189221+0000", 2023-01-15T14:36:08.842 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010438 2023-01-15T14:36:08.842 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.842 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291724+0000", 2023-01-15T14:36:08.843 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102502998 2023-01-15T14:36:08.843 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.843 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.843 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:08.843 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291824+0000", 2023-01-15T14:36:08.846 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00010010599999999999 2023-01-15T14:36:08.846 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.846 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291992+0000", 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00016856900000000001 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:21 3.5 3:a75958b1:::smithi09025264-6:head [write 484035~776245,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 462.34884325899998, 2023-01-15T14:36:08.847 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.45851924500000002, 2023-01-15T14:36:08.848 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:08.848 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:08.848 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:08.848 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:08.848 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:08.848 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 21 2023-01-15T14:36:08.848 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.848 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:08.848 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.848 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:08.848 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:36:08.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:08.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831660+0000", 2023-01-15T14:36:08.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.849 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.849 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.849 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:08.849 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.831657+0000", 2023-01-15T14:36:08.849 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999967 2023-01-15T14:36:08.850 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.850 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.851 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:08.851 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832814+0000", 2023-01-15T14:36:08.852 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0011564909999999999 2023-01-15T14:36:08.852 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.852 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:08.853 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832864+0000", 2023-01-15T14:36:08.853 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.0243e-05 2023-01-15T14:36:08.853 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.853 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.853 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:08.853 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.832903+0000", 2023-01-15T14:36:08.853 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8704000000000001e-05 2023-01-15T14:36:08.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983202+0000", 2023-01-15T14:36:08.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.150299404 2023-01-15T14:36:08.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.854 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.854 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting_for_map not empty", 2023-01-15T14:36:08.854 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.983208+0000", 2023-01-15T14:36:08.854 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.0039999999999996e-06 2023-01-15T14:36:08.854 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:08.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185539+0000", 2023-01-15T14:36:08.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202330867 2023-01-15T14:36:08.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.855 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:08.855 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.185803+0000", 2023-01-15T14:36:08.855 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026394299999999999 2023-01-15T14:36:08.855 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.855 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:36:08.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186037+0000", 2023-01-15T14:36:08.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023386300000000001 2023-01-15T14:36:08.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:08.856 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186637+0000", 2023-01-15T14:36:08.856 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00060001800000000001 2023-01-15T14:36:08.856 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.856 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.856 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:08.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186665+0000", 2023-01-15T14:36:08.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.8459999999999999e-05 2023-01-15T14:36:08.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.857 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:08.857 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.187045+0000", 2023-01-15T14:36:08.857 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00037992899999999999 2023-01-15T14:36:08.857 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.857 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:08.858 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.188201+0000", 2023-01-15T14:36:08.858 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.001155309 2023-01-15T14:36:08.858 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.858 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:08.858 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289903+0000", 2023-01-15T14:36:08.858 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101702601 2023-01-15T14:36:08.858 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.858 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.858 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289985+0000", 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.1617000000000001e-05 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290065+0000", 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0276000000000002e-05 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290180+0000", 2023-01-15T14:36:08.859 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00011464300000000001 2023-01-15T14:36:08.860 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:08.860 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:08.860 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:08.860 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.860 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.860 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:22 3.5 3:a75958b1:::smithi09025264-6:head [write 2059704~24331] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:36:08.860 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:36:08.860 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 462.344375571, 2023-01-15T14:36:08.860 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65790599599999999, 2023-01-15T14:36:08.860 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:08.860 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:08.861 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:08.861 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:08.861 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:08.861 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 22 2023-01-15T14:36:08.861 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.861 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:08.861 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.861 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:08.861 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:36:08.861 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.862 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:08.862 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836128+0000", 2023-01-15T14:36:08.862 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:08.862 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.862 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.862 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:08.862 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836126+0000", 2023-01-15T14:36:08.862 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:36:08.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:08.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836181+0000", 2023-01-15T14:36:08.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.4246999999999997e-05 2023-01-15T14:36:08.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.863 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.863 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:08.863 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836212+0000", 2023-01-15T14:36:08.863 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1843e-05 2023-01-15T14:36:08.863 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:08.864 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:08.864 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:08.864 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836226+0000", 2023-01-15T14:36:09.089 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3216e-05 2023-01-15T14:36:09.089 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.090 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.090 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186757+0000", 2023-01-15T14:36:09.090 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35053103899999999 2023-01-15T14:36:09.090 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.090 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.091 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:09.091 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.186871+0000", 2023-01-15T14:36:09.091 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000114403 2023-01-15T14:36:09.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.092 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.092 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.092 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290263+0000", 2023-01-15T14:36:09.092 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.103391976 2023-01-15T14:36:09.092 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:09.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290413+0000", 2023-01-15T14:36:09.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014991200000000001 2023-01-15T14:36:09.093 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.093 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.093 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:36:09.093 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290627+0000", 2023-01-15T14:36:09.093 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021379299999999999 2023-01-15T14:36:09.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:09.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290861+0000", 2023-01-15T14:36:09.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00023479600000000001 2023-01-15T14:36:09.094 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.094 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.094 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:09.094 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290888+0000", 2023-01-15T14:36:09.094 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6570000000000001e-05 2023-01-15T14:36:09.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.095 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.095 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:09.095 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.290990+0000", 2023-01-15T14:36:09.095 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000101723 2023-01-15T14:36:09.095 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.097 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.097 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:09.097 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.291245+0000", 2023-01-15T14:36:09.097 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025543899999999999 2023-01-15T14:36:09.098 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.098 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.098 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:09.098 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392822+0000", 2023-01-15T14:36:09.098 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10157725500000001 2023-01-15T14:36:09.099 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.099 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.099 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:09.099 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493881+0000", 2023-01-15T14:36:09.099 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101058262 2023-01-15T14:36:09.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.100 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.100 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:09.100 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493964+0000", 2023-01-15T14:36:09.100 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.2973000000000007e-05 2023-01-15T14:36:09.100 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.101 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.101 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:09.101 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494034+0000", 2023-01-15T14:36:09.101 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.0432999999999995e-05 2023-01-15T14:36:09.101 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.102 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:09.102 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.102 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.102 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.102 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:23 3.5 3:a75958b1:::smithi09025264-6:head [setxattr _header (55) in=7b,truncate 2084035] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:36:09.103 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:36:09.103 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 462.34426044200001, 2023-01-15T14:36:09.103 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.65796116900000001, 2023-01-15T14:36:09.103 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:09.103 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:09.103 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:09.104 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:09.104 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:09.104 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 23 2023-01-15T14:36:09.104 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.104 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:09.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.105 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:09.105 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:36:09.105 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.105 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.105 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.106 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:09.106 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836243+0000", 2023-01-15T14:36:09.106 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.106 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.106 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.107 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:09.107 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836242+0000", 2023-01-15T14:36:09.107 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.999999 2023-01-15T14:36:09.107 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.107 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.107 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:09.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836256+0000", 2023-01-15T14:36:09.108 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3677e-05 2023-01-15T14:36:09.108 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.108 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.108 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:09.108 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.836265+0000", 2023-01-15T14:36:09.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2769999999999994e-06 2023-01-15T14:36:09.109 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.109 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.109 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:09.109 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936398+0000", 2023-01-15T14:36:09.109 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100132559 2023-01-15T14:36:09.110 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.110 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.110 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.110 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289431+0000", 2023-01-15T14:36:09.110 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.35303322399999998 2023-01-15T14:36:09.111 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.111 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.111 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:09.111 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289587+0000", 2023-01-15T14:36:09.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000155873 2023-01-15T14:36:09.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292052+0000", 2023-01-15T14:36:09.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0024647919999999999 2023-01-15T14:36:09.289 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.289 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.289 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:09.289 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292180+0000", 2023-01-15T14:36:09.289 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012810999999999999 2023-01-15T14:36:09.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:36:09.290 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292373+0000", 2023-01-15T14:36:09.290 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00019258599999999999 2023-01-15T14:36:09.290 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.290 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.290 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:09.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292597+0000", 2023-01-15T14:36:09.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022465500000000001 2023-01-15T14:36:09.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:09.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292618+0000", 2023-01-15T14:36:09.291 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.0319999999999999e-05 2023-01-15T14:36:09.291 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.291 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.291 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:09.291 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.292692+0000", 2023-01-15T14:36:09.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.4643999999999999e-05 2023-01-15T14:36:09.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:09.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393103+0000", 2023-01-15T14:36:09.292 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100411156 2023-01-15T14:36:09.292 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.292 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.292 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:09.292 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.493827+0000", 2023-01-15T14:36:09.293 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100724143 2023-01-15T14:36:09.293 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.293 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.293 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:09.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494085+0000", 2023-01-15T14:36:09.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000257891 2023-01-15T14:36:09.297 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.297 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.297 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:09.297 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494141+0000", 2023-01-15T14:36:09.297 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.6132999999999999e-05 2023-01-15T14:36:09.298 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.298 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.298 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:09.298 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494204+0000", 2023-01-15T14:36:09.298 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.2966000000000007e-05 2023-01-15T14:36:09.299 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.299 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:09.300 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.300 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.300 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.301 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:24 3.5 3:a75958b1:::smithi09025264-6:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e22)", 2023-01-15T14:36:09.301 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:36:09.301 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 462.244079016, 2023-01-15T14:36:09.301 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.55808677600000001, 2023-01-15T14:36:09.302 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:09.302 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:09.302 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:09.302 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:09.302 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:09.302 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 24 2023-01-15T14:36:09.302 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.302 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:09.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:09.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:36:09.303 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.303 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.303 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.303 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:09.303 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936425+0000", 2023-01-15T14:36:09.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.304 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.304 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:09.304 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936423+0000", 2023-01-15T14:36:09.304 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:36:09.304 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.305 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:09.305 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936433+0000", 2023-01-15T14:36:09.305 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0443e-05 2023-01-15T14:36:09.305 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.305 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.305 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:09.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936451+0000", 2023-01-15T14:36:09.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7842999999999998e-05 2023-01-15T14:36:09.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:09.306 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:25.936461+0000", 2023-01-15T14:36:09.306 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0057e-05 2023-01-15T14:36:09.306 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.306 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.306 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289693+0000", 2023-01-15T14:36:09.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.353231617 2023-01-15T14:36:09.307 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.307 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.307 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:09.307 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.289814+0000", 2023-01-15T14:36:09.307 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120763 2023-01-15T14:36:09.308 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.308 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.308 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.308 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.392923+0000", 2023-01-15T14:36:09.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10310955300000001 2023-01-15T14:36:09.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:09.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.393072+0000", 2023-01-15T14:36:09.309 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000148469 2023-01-15T14:36:09.309 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.309 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.309 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.309 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494242+0000", 2023-01-15T14:36:09.310 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117034499999999 2023-01-15T14:36:09.310 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.310 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.310 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:09.310 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494363+0000", 2023-01-15T14:36:09.310 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000120703 2023-01-15T14:36:09.310 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.310 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.310 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:09.311 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.494511+0000", 2023-01-15T14:36:09.311 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00014873299999999999 2023-01-15T14:36:09.311 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.311 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:09.311 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.312 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.312 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.312 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:78 3.c 3:3137f0cb:::smithi09025264-20:head [write 1962099~357234] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:36:09.312 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:36:09.312 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 461.85411302, 2023-01-15T14:36:09.313 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20331801699999999, 2023-01-15T14:36:09.313 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:09.313 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:09.313 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:09.313 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:09.313 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:09.313 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 78 2023-01-15T14:36:09.314 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.314 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:09.314 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.314 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:09.314 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:36:09.314 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.315 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.315 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:09.315 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326391+0000", 2023-01-15T14:36:09.315 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.315 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:09.316 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326388+0000", 2023-01-15T14:36:09.316 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:36:09.316 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.316 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.316 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:09.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326672+0000", 2023-01-15T14:36:09.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028367900000000001 2023-01-15T14:36:09.317 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.317 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.317 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:09.317 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326713+0000", 2023-01-15T14:36:09.317 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1539999999999999e-05 2023-01-15T14:36:09.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:09.318 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326729+0000", 2023-01-15T14:36:09.318 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.519e-05 2023-01-15T14:36:09.318 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.318 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.318 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326879+0000", 2023-01-15T14:36:09.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015064899999999999 2023-01-15T14:36:09.319 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.319 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.319 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:09.319 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327096+0000", 2023-01-15T14:36:09.319 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00021682299999999999 2023-01-15T14:36:09.320 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.320 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.320 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:36:09.320 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327308+0000", 2023-01-15T14:36:09.320 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000211366 2023-01-15T14:36:09.321 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.321 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.321 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:09.321 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327757+0000", 2023-01-15T14:36:09.321 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000449009 2023-01-15T14:36:09.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.322 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.322 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:09.322 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.327776+0000", 2023-01-15T14:36:09.322 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9201000000000001e-05 2023-01-15T14:36:09.322 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:09.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.328012+0000", 2023-01-15T14:36:09.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000236054 2023-01-15T14:36:09.323 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.323 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.323 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:09.323 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429534+0000", 2023-01-15T14:36:09.323 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10152201499999999 2023-01-15T14:36:09.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:09.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529340+0000", 2023-01-15T14:36:09.324 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099806249 2023-01-15T14:36:09.324 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.324 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.324 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:09.324 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529437+0000", 2023-01-15T14:36:09.325 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.6949999999999998e-05 2023-01-15T14:36:09.481 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.482 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:09.482 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529587+0000", 2023-01-15T14:36:09.482 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00015008600000000001 2023-01-15T14:36:09.482 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.482 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.483 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:09.483 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529709+0000", 2023-01-15T14:36:09.483 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000121636 2023-01-15T14:36:09.483 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.483 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:09.488 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.488 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.488 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.488 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:79 3.c 3:3137f0cb:::smithi09025264-20:head [setxattr _header (56) in=7b,truncate 2319333] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:36:09.488 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:36:09.489 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 461.85374342400002, 2023-01-15T14:36:09.489 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20348171300000001, 2023-01-15T14:36:09.489 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:09.489 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:09.489 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:09.489 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:09.489 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:09.490 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 79 2023-01-15T14:36:09.490 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.490 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:09.490 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.490 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:09.490 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:36:09.490 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.490 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.490 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.490 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326760+0000", 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326759+0000", 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326770+0000", 2023-01-15T14:36:09.491 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.0696999999999999e-05 2023-01-15T14:36:09.492 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.492 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.492 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:09.492 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326791+0000", 2023-01-15T14:36:09.492 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1556000000000002e-05 2023-01-15T14:36:09.492 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.492 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.492 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:09.492 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326818+0000", 2023-01-15T14:36:09.492 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6979999999999999e-05 2023-01-15T14:36:09.492 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.427993+0000", 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10117517199999999 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428266+0000", 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00027233299999999998 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.493 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 3,6", 2023-01-15T14:36:09.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.428696+0000", 2023-01-15T14:36:09.494 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00042996499999999998 2023-01-15T14:36:09.494 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.494 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:09.494 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429057+0000", 2023-01-15T14:36:09.494 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000360952 2023-01-15T14:36:09.494 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.494 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.495 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:09.495 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429076+0000", 2023-01-15T14:36:09.495 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.9167000000000001e-05 2023-01-15T14:36:09.495 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.495 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.495 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:09.495 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429160+0000", 2023-01-15T14:36:09.496 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:36:09.496 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.496 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.496 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:09.496 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429549+0000", 2023-01-15T14:36:09.498 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00038884599999999998 2023-01-15T14:36:09.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.498 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:09.498 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529886+0000", 2023-01-15T14:36:09.498 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100336261 2023-01-15T14:36:09.498 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.498 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.498 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:09.499 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.529978+0000", 2023-01-15T14:36:09.499 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.2775999999999994e-05 2023-01-15T14:36:09.499 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.499 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.499 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:09.500 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530108+0000", 2023-01-15T14:36:09.500 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012958300000000001 2023-01-15T14:36:09.500 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.500 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.500 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:09.500 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530242+0000", 2023-01-15T14:36:09.500 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00013406 2023-01-15T14:36:09.501 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.501 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:09.502 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.502 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.502 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.502 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:80 3.c 3:3137f0cb:::smithi09025264-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:36:09.502 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:36:09.502 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 461.85360506500001, 2023-01-15T14:36:09.502 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20370874, 2023-01-15T14:36:09.502 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:09.502 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 80 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.503 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:09.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326899+0000", 2023-01-15T14:36:09.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:09.504 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326897+0000", 2023-01-15T14:36:09.504 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:36:09.504 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.504 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.504 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:09.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326906+0000", 2023-01-15T14:36:09.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.6929999999999996e-06 2023-01-15T14:36:09.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:09.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326931+0000", 2023-01-15T14:36:09.505 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.4757e-05 2023-01-15T14:36:09.505 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.505 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.505 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:09.505 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.326945+0000", 2023-01-15T14:36:09.506 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.402e-05 2023-01-15T14:36:09.506 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.678 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.678 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.678 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429239+0000", 2023-01-15T14:36:09.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.102294418 2023-01-15T14:36:09.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.679 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:09.679 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.429487+0000", 2023-01-15T14:36:09.679 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024789300000000001 2023-01-15T14:36:09.679 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.679 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530309+0000", 2023-01-15T14:36:09.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100821459 2023-01-15T14:36:09.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:09.680 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530438+0000", 2023-01-15T14:36:09.680 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012928700000000001 2023-01-15T14:36:09.680 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.680 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.680 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:09.681 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:26.530607+0000", 2023-01-15T14:36:09.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000169483 2023-01-15T14:36:09.681 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.681 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:09.681 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.681 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.681 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.681 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:201 3.5 3:a1c47f99:::smithi09025264-2:head [copy-from ver 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:36:09.681 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:36:09.681 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 460.77410694700001, 2023-01-15T14:36:09.681 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30931411399999997, 2023-01-15T14:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 201 2023-01-15T14:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:09.682 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:36:09.683 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.683 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.683 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.683 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:09.683 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406397+0000", 2023-01-15T14:36:09.684 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.684 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.684 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.684 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:09.684 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406394+0000", 2023-01-15T14:36:09.686 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999971 2023-01-15T14:36:09.686 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.687 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.687 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:09.687 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406408+0000", 2023-01-15T14:36:09.687 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4416999999999999e-05 2023-01-15T14:36:09.688 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.689 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.690 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:09.692 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406446+0000", 2023-01-15T14:36:09.692 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.7339999999999998e-05 2023-01-15T14:36:09.693 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.694 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.695 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:09.696 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406467+0000", 2023-01-15T14:36:09.697 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1137000000000001e-05 2023-01-15T14:36:09.698 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.699 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.699 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.700 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406677+0000", 2023-01-15T14:36:09.700 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000210275 2023-01-15T14:36:09.701 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.701 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.701 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:09.701 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406902+0000", 2023-01-15T14:36:09.702 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00022529000000000001 2023-01-15T14:36:09.702 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.702 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.703 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:36:09.703 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.513012+0000", 2023-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.106109527 2023-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.704 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514754+0000", 2023-01-15T14:36:09.705 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0017419779999999999 2023-01-15T14:36:09.706 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.706 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.706 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:09.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.514773+0000", 2023-01-15T14:36:09.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8700000000000001e-05 2023-01-15T14:36:09.707 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.707 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.707 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:09.707 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.515531+0000", 2023-01-15T14:36:09.707 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00075844300000000003 2023-01-15T14:36:09.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:09.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615334+0000", 2023-01-15T14:36:09.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.099803327999999997 2023-01-15T14:36:09.708 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.708 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.708 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:09.708 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615376+0000", 2023-01-15T14:36:09.708 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.1675999999999999e-05 2023-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715546+0000", 2023-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10016992500000001 2023-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715626+0000", 2023-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.0112999999999997e-05 2023-01-15T14:36:09.709 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.715711+0000", 2023-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 8.4798999999999999e-05 2023-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:202 3.5 3:a1c47f99:::smithi09025264-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e23)", 2023-01-15T14:36:09.710 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 460.77399440099998, 2023-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.31002637300000002, 2023-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 202 2023-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:09.711 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.712 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:09.712 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:36:09.712 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.877 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.877 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:09.877 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406509+0000", 2023-01-15T14:36:09.877 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.877 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.877 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406508+0000", 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406516+0000", 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.413e-06 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:09.878 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406534+0000", 2023-01-15T14:36:09.879 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.8936999999999998e-05 2023-01-15T14:36:09.879 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.879 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:09.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.406548+0000", 2023-01-15T14:36:09.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3196e-05 2023-01-15T14:36:09.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.880 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.880 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407271+0000", 2023-01-15T14:36:09.880 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00072301500000000003 2023-01-15T14:36:09.880 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.880 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for blocked object", 2023-01-15T14:36:09.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.407466+0000", 2023-01-15T14:36:09.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000195009 2023-01-15T14:36:09.881 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.881 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.881 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.881 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.614996+0000", 2023-01-15T14:36:09.881 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.207530192 2023-01-15T14:36:09.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for rw locks", 2023-01-15T14:36:09.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.615227+0000", 2023-01-15T14:36:09.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000230979 2023-01-15T14:36:09.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.882 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.882 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.882 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716282+0000", 2023-01-15T14:36:09.882 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.101055465 2023-01-15T14:36:09.882 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:09.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716409+0000", 2023-01-15T14:36:09.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012653999999999999 2023-01-15T14:36:09.883 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.883 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.883 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:09.883 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.716536+0000", 2023-01-15T14:36:09.883 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012683 2023-01-15T14:36:09.883 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.884 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:09.884 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.884 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.884 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.884 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:189 3.d e23/20 3:b3f111b5:::smithi09025264-48:head v 23'10, mlcod=23'6)", 2023-01-15T14:36:09.884 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:36:09.884 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 460.728789753, 2023-01-15T14:36:09.884 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20400475100000001, 2023-01-15T14:36:09.889 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:09.889 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:09.889 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:09.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.889 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:09.889 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:36:09.889 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.889 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.889 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.890 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:09.890 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451714+0000", 2023-01-15T14:36:09.890 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.890 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.891 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.891 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:09.891 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.451712+0000", 2023-01-15T14:36:09.891 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999976 2023-01-15T14:36:09.891 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.892 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.892 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:09.892 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452272+0000", 2023-01-15T14:36:09.892 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00056035899999999997 2023-01-15T14:36:09.892 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.893 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:09.893 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452310+0000", 2023-01-15T14:36:09.893 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.8145999999999997e-05 2023-01-15T14:36:09.893 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.893 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.894 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:09.894 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.452324+0000", 2023-01-15T14:36:09.894 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4443999999999999e-05 2023-01-15T14:36:09.894 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.894 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.895 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.895 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652703+0000", 2023-01-15T14:36:09.895 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20037818199999999 2023-01-15T14:36:09.895 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.895 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.896 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:09.896 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.652762+0000", 2023-01-15T14:36:09.896 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.9302999999999998e-05 2023-01-15T14:36:09.896 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.896 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.897 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:09.897 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653302+0000", 2023-01-15T14:36:09.897 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00054012199999999998 2023-01-15T14:36:09.897 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.897 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.898 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:09.898 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653333+0000", 2023-01-15T14:36:09.898 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.112e-05 2023-01-15T14:36:09.898 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.899 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.899 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:09.899 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653667+0000", 2023-01-15T14:36:09.899 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00033337900000000002 2023-01-15T14:36:09.899 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.900 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.900 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:09.900 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.653709+0000", 2023-01-15T14:36:09.900 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2469999999999998e-05 2023-01-15T14:36:09.900 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.901 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.901 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:09.901 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:27.655719+0000", 2023-01-15T14:36:09.901 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0020096699999999999 2023-01-15T14:36:09.901 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.902 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:09.902 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.903 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.904 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.904 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:269 3.2 e30/20 3:47dfe935:::smithi09025264-29:head v 30'11, mlcod=23'8)", 2023-01-15T14:36:09.904 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:36:09.905 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 453.30274431200002, 2023-01-15T14:36:09.905 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.23185540399999999, 2023-01-15T14:36:09.905 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:09.905 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:09.905 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.906 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.907 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.907 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:09.907 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:36:09.907 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:09.908 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.908 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.909 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:09.909 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.877759+0000", 2023-01-15T14:36:09.909 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:36:09.909 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.909 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.910 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:09.910 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878379+0000", 2023-01-15T14:36:09.910 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00061979600000000004 2023-01-15T14:36:09.910 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.910 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.911 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:09.911 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.878406+0000", 2023-01-15T14:36:09.911 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.7419999999999998e-05 2023-01-15T14:36:09.911 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.911 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.912 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:09.912 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978515+0000", 2023-01-15T14:36:09.912 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100108461 2023-01-15T14:36:09.912 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.912 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.913 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:09.913 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078759+0000", 2023-01-15T14:36:09.913 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.100244496 2023-01-15T14:36:09.913 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.914 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.914 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:09.914 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.078796+0000", 2023-01-15T14:36:09.915 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.6758999999999998e-05 2023-01-15T14:36:09.915 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.915 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.915 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:09.915 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079567+0000", 2023-01-15T14:36:09.916 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00077137199999999999 2023-01-15T14:36:09.917 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.917 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.917 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:09.917 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079593+0000", 2023-01-15T14:36:09.918 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.6069000000000001e-05 2023-01-15T14:36:09.918 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.918 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.918 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:09.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079844+0000", 2023-01-15T14:36:09.919 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00025075899999999998 2023-01-15T14:36:09.919 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.919 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.919 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:09.919 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.079910+0000", 2023-01-15T14:36:09.920 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 6.6406000000000001e-05 2023-01-15T14:36:09.920 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.920 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.920 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:09.920 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.109615+0000", 2023-01-15T14:36:09.921 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.029704323000000001 2023-01-15T14:36:09.921 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.921 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:09.921 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:09.921 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.921 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.922 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_op(client.4530.0:281 3.5 3:a75958b1:::smithi09025264-6:head [append 0~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30)", 2023-01-15T14:36:09.923 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:36:09.923 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 453.20182068299999, 2023-01-15T14:36:09.923 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.30574310900000001, 2023-01-15T14:36:09.923 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:09.923 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:09.924 INFO:teuthology.orchestra.run.smithi141.stdout: "client_info": { 2023-01-15T14:36:09.924 INFO:teuthology.orchestra.run.smithi141.stdout: "client": "client.4530", 2023-01-15T14:36:09.925 INFO:teuthology.orchestra.run.smithi141.stdout: "client_addr": "172.21.15.90:0/4254192437", 2023-01-15T14:36:09.925 INFO:teuthology.orchestra.run.smithi141.stdout: "tid": 281 2023-01-15T14:36:09.925 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:09.925 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:09.925 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:09.926 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:10.058 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:36:10.058 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:10.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.058 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.058 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:10.058 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:36:10.058 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:10.058 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.058 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.059 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:10.059 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978683+0000", 2023-01-15T14:36:10.059 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999995 2023-01-15T14:36:10.059 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.059 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.061 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:10.061 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978686+0000", 2023-01-15T14:36:10.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4429999999999999e-06 2023-01-15T14:36:10.061 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.061 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.061 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:10.061 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978690+0000", 2023-01-15T14:36:10.061 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0300000000000004e-06 2023-01-15T14:36:10.061 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.061 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:10.062 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.978693+0000", 2023-01-15T14:36:10.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.179e-06 2023-01-15T14:36:10.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.062 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:10.062 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981592+0000", 2023-01-15T14:36:10.062 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0028984140000000002 2023-01-15T14:36:10.062 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.062 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.062 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:10.063 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981718+0000", 2023-01-15T14:36:10.063 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00012616600000000001 2023-01-15T14:36:10.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.063 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "waiting for subops from 2,3", 2023-01-15T14:36:10.063 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.981982+0000", 2023-01-15T14:36:10.063 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00026445199999999998 2023-01-15T14:36:10.063 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.063 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.063 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:10.064 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982278+0000", 2023-01-15T14:36:10.064 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00029514100000000001 2023-01-15T14:36:10.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.064 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.064 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:10.064 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982290+0000", 2023-01-15T14:36:10.064 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.271e-05 2023-01-15T14:36:10.064 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.064 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.064 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:10.064 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982343+0000", 2023-01-15T14:36:10.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 5.2914000000000002e-05 2023-01-15T14:36:10.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "op_commit", 2023-01-15T14:36:10.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:34.982385+0000", 2023-01-15T14:36:10.065 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.2114999999999999e-05 2023-01-15T14:36:10.065 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.065 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.065 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:10.065 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.181937+0000", 2023-01-15T14:36:10.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.19955213099999999 2023-01-15T14:36:10.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "sub_op_commit_rec", 2023-01-15T14:36:10.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284183+0000", 2023-01-15T14:36:10.066 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10224538 2023-01-15T14:36:10.066 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.066 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.066 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:10.066 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284318+0000", 2023-01-15T14:36:10.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000134843 2023-01-15T14:36:10.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.067 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:10.067 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:28:35.284426+0000", 2023-01-15T14:36:10.067 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000108513 2023-01-15T14:36:10.067 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:10.067 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:10.067 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:10.067 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.067 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.068 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "MOSDPGRecoveryDelete(3.62 e469,467 3:47dfe935:::smithi09025264-29:81,232'123)", 2023-01-15T14:36:10.068 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:35:59.960095+0000", 2023-01-15T14:36:10.068 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 8.2204089590000002, 2023-01-15T14:36:10.068 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.66952493999999996, 2023-01-15T14:36:10.068 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:10.068 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:10.068 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:10.068 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.068 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:10.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:59.960095+0000", 2023-01-15T14:36:10.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:10.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.069 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.069 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:10.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:59.960095+0000", 2023-01-15T14:36:10.069 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:10.069 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.069 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.069 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:10.069 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:59.960093+0000", 2023-01-15T14:36:10.070 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:36:10.070 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.070 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:10.070 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:59.960106+0000", 2023-01-15T14:36:10.070 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.2687000000000001e-05 2023-01-15T14:36:10.070 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.070 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.070 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:10.070 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:35:59.960138+0000", 2023-01-15T14:36:10.070 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.1801999999999998e-05 2023-01-15T14:36:10.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.071 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:10.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.060291+0000", 2023-01-15T14:36:10.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.10015354 2023-01-15T14:36:10.071 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.071 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.071 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:10.071 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.629365+0000", 2023-01-15T14:36:10.071 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.56907354799999998 2023-01-15T14:36:10.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:10.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.629378+0000", 2023-01-15T14:36:10.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.3376e-05 2023-01-15T14:36:10.072 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.072 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.072 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:10.072 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.629620+0000", 2023-01-15T14:36:10.072 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00024153099999999999 2023-01-15T14:36:10.073 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:10.073 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:10.073 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:10.073 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.073 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.073 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "MOSDPGRecoveryDelete(3.62 e470,467 3:47dfe935:::smithi09025264-29:9f,246'128)", 2023-01-15T14:36:10.073 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:36:00.316730+0000", 2023-01-15T14:36:10.073 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 7.8637734659999996, 2023-01-15T14:36:10.073 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.41294129499999999, 2023-01-15T14:36:10.073 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:10.074 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:10.074 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:10.074 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.074 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:10.074 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.316730+0000", 2023-01-15T14:36:10.074 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:10.074 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.074 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.074 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:10.074 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.316730+0000", 2023-01-15T14:36:10.074 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:10.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.075 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:10.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.316729+0000", 2023-01-15T14:36:10.075 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999981 2023-01-15T14:36:10.075 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.075 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.075 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:10.075 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.316744+0000", 2023-01-15T14:36:10.075 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.5362999999999998e-05 2023-01-15T14:36:10.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:10.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.316784+0000", 2023-01-15T14:36:10.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4.0460000000000002e-05 2023-01-15T14:36:10.076 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.076 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.076 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:10.076 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.316807+0000", 2023-01-15T14:36:10.076 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.2804999999999999e-05 2023-01-15T14:36:10.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:10.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.729355+0000", 2023-01-15T14:36:10.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.41254739699999998 2023-01-15T14:36:10.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.077 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.077 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:10.077 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.729371+0000", 2023-01-15T14:36:10.077 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6847999999999999e-05 2023-01-15T14:36:10.077 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.078 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.078 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:10.078 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.729672+0000", 2023-01-15T14:36:10.078 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00030015599999999999 2023-01-15T14:36:10.078 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:10.078 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:10.078 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:10.078 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.078 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.078 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "osd_repop(client.4530.0:4501 3.40 e470/467 3:03b54904:::smithi09025264-12:head v 470'363, mlcod=0'0)", 2023-01-15T14:36:10.078 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:36:00.421608+0000", 2023-01-15T14:36:10.079 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 7.7588956209999997, 2023-01-15T14:36:10.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.80378465300000002, 2023-01-15T14:36:10.079 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:10.079 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "commit sent; apply or cleanup", 2023-01-15T14:36:10.079 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:10.079 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.079 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:10.079 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.421608+0000", 2023-01-15T14:36:10.079 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:10.079 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:10.080 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.421608+0000", 2023-01-15T14:36:10.080 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:10.080 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.080 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.080 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:10.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.421607+0000", 2023-01-15T14:36:10.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:36:10.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:10.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.421623+0000", 2023-01-15T14:36:10.081 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.6308999999999999e-05 2023-01-15T14:36:10.081 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.081 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.081 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:10.081 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.421658+0000", 2023-01-15T14:36:10.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.4659999999999997e-05 2023-01-15T14:36:10.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:10.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:00.421675+0000", 2023-01-15T14:36:10.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.7183000000000001e-05 2023-01-15T14:36:10.082 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.082 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.082 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:10.082 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:01.224564+0000", 2023-01-15T14:36:10.082 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.80288932000000002 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:01.224642+0000", 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 7.7804000000000002e-05 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_queued_for_journal_write", 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:01.224956+0000", 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000314531 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.083 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "write_thread_in_journal_buffer", 2023-01-15T14:36:10.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:01.224979+0000", 2023-01-15T14:36:10.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.3065e-05 2023-01-15T14:36:10.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.084 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "journaled_completion_queued", 2023-01-15T14:36:10.084 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:01.225071+0000", 2023-01-15T14:36:10.084 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 9.1692000000000002e-05 2023-01-15T14:36:10.084 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.084 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "commit_sent", 2023-01-15T14:36:10.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:01.225109+0000", 2023-01-15T14:36:10.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 3.79e-05 2023-01-15T14:36:10.085 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.085 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.085 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:10.085 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:01.225393+0000", 2023-01-15T14:36:10.085 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.00028366900000000001 2023-01-15T14:36:10.085 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:10.085 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:10.085 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:10.086 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.086 INFO:teuthology.orchestra.run.smithi141.stdout: "description": "MOSDPGPush(3.4d 472/467 [PushOp(3:b2a32959:::smithi09025264-27:head, version: 463'343, data_included: [1048576~968396], data_size: 968396, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(3:b2a32959:::smithi09025264-27:head@463'343, size: 2016972, copy_subset: [0~2016972], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:2016972, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:1048576, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2023-01-15T14:36:10.086 INFO:teuthology.orchestra.run.smithi141.stdout: "initiated_at": "2023-01-15T14:36:02.092569+0000", 2023-01-15T14:36:10.086 INFO:teuthology.orchestra.run.smithi141.stdout: "age": 6.0879347739999998, 2023-01-15T14:36:10.086 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.202648622, 2023-01-15T14:36:10.086 INFO:teuthology.orchestra.run.smithi141.stdout: "type_data": { 2023-01-15T14:36:10.086 INFO:teuthology.orchestra.run.smithi141.stdout: "flag_point": "started", 2023-01-15T14:36:10.086 INFO:teuthology.orchestra.run.smithi141.stdout: "events": [ 2023-01-15T14:36:10.086 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.087 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "initiated", 2023-01-15T14:36:10.087 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:02.092569+0000", 2023-01-15T14:36:10.087 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:10.087 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.087 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "throttled", 2023-01-15T14:36:10.087 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:02.092569+0000", 2023-01-15T14:36:10.087 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0 2023-01-15T14:36:10.087 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.087 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "header_read", 2023-01-15T14:36:10.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:02.092568+0000", 2023-01-15T14:36:10.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 4294967295.9999986 2023-01-15T14:36:10.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.088 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.088 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "all_read", 2023-01-15T14:36:10.088 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:02.094426+0000", 2023-01-15T14:36:10.088 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.0018580669999999999 2023-01-15T14:36:10.088 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.089 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.089 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "dispatched", 2023-01-15T14:36:10.089 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:02.094448+0000", 2023-01-15T14:36:10.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 2.1911000000000001e-05 2023-01-15T14:36:10.266 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.266 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.266 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "queued_for_pg", 2023-01-15T14:36:10.266 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:02.094463+0000", 2023-01-15T14:36:10.266 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.508e-05 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "reached_pg", 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:02.294479+0000", 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.20001574599999999 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "started", 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:02.294493+0000", 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 1.4902999999999999e-05 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: }, 2023-01-15T14:36:10.267 INFO:teuthology.orchestra.run.smithi141.stdout: { 2023-01-15T14:36:10.268 INFO:teuthology.orchestra.run.smithi141.stdout: "event": "done", 2023-01-15T14:36:10.268 INFO:teuthology.orchestra.run.smithi141.stdout: "time": "2023-01-15T14:36:02.295218+0000", 2023-01-15T14:36:10.268 INFO:teuthology.orchestra.run.smithi141.stdout: "duration": 0.000724115 2023-01-15T14:36:10.268 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:10.268 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:10.268 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:10.268 INFO:teuthology.orchestra.run.smithi141.stdout: } 2023-01-15T14:36:10.268 INFO:teuthology.orchestra.run.smithi141.stdout: ] 2023-01-15T14:36:10.268 INFO:teuthology.orchestra.run.smithi141.stdout:} 2023-01-15T14:36:10.269 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2023-01-15T14:36:10.269 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 7, 1, 6, 5] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2023-01-15T14:36:10.270 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2023-01-15T14:36:10.270 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2023-01-15T14:36:10.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2023-01-15T14:36:10.270 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2023-01-15T14:36:10.271 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:36:08.245+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:36:10.273 INFO:teuthology.orchestra.run.smithi090.stdout:osd.0: {} 2023-01-15T14:36:10.273 INFO:teuthology.orchestra.run.smithi090.stdout:osd.1: {} 2023-01-15T14:36:10.273 INFO:teuthology.orchestra.run.smithi090.stdout:osd.2: {} 2023-01-15T14:36:10.273 INFO:teuthology.orchestra.run.smithi090.stdout:osd.3: {} 2023-01-15T14:36:10.273 INFO:teuthology.orchestra.run.smithi090.stdout:osd.4: {} 2023-01-15T14:36:10.273 INFO:teuthology.orchestra.run.smithi090.stdout:osd.5: {} 2023-01-15T14:36:10.273 INFO:teuthology.orchestra.run.smithi090.stdout:osd.6: {} 2023-01-15T14:36:10.273 INFO:teuthology.orchestra.run.smithi090.stdout:osd.7: {} 2023-01-15T14:36:10.274 INFO:teuthology.orchestra.run.smithi090.stderr:osd.0: osd_enable_op_tracker = 'false' 2023-01-15T14:36:10.274 INFO:teuthology.orchestra.run.smithi090.stderr:osd.1: osd_enable_op_tracker = 'false' 2023-01-15T14:36:10.274 INFO:teuthology.orchestra.run.smithi090.stderr:osd.2: osd_enable_op_tracker = 'false' 2023-01-15T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:osd.3: osd_enable_op_tracker = 'false' 2023-01-15T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:osd.4: osd_enable_op_tracker = 'false' 2023-01-15T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:osd.5: osd_enable_op_tracker = 'false' 2023-01-15T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:osd.6: osd_enable_op_tracker = 'false' 2023-01-15T14:36:10.275 INFO:teuthology.orchestra.run.smithi090.stderr:osd.7: osd_enable_op_tracker = 'false' 2023-01-15T14:36:10.277 INFO:tasks.rados.rados.0.smithi090.stdout:1756: done (0 left) 2023-01-15T14:36:10.277 INFO:tasks.rados.rados.0.smithi090.stdout:1757: read oid 20 snap -1 2023-01-15T14:36:10.278 INFO:tasks.rados.rados.0.smithi090.stdout:1757: expect (ObjNum 617 snap 188 seq_num 617) 2023-01-15T14:36:10.278 INFO:tasks.rados.rados.0.smithi090.stdout:1758: write oid 17 current snap is 191 2023-01-15T14:36:10.278 INFO:tasks.rados.rados.0.smithi090.stdout:1758: seq_num 623 ranges {470761=704720,1656870=659539} 2023-01-15T14:36:10.278 INFO:tasks.rados.rados.0.smithi090.stdout:1758: writing smithi09025264-17 from 470761 to 1175481 tid 1 2023-01-15T14:36:10.278 INFO:tasks.rados.rados.0.smithi090.stdout:1758: writing smithi09025264-17 from 1656870 to 2316409 tid 2 2023-01-15T14:36:10.278 INFO:tasks.rados.rados.0.smithi090.stdout:1759: read oid 44 snap -1 2023-01-15T14:36:10.278 INFO:tasks.rados.rados.0.smithi090.stdout:1759: expect (ObjNum 575 snap 177 seq_num 575) 2023-01-15T14:36:10.278 INFO:tasks.rados.rados.0.smithi090.stdout:1760: snap_create 2023-01-15T14:36:10.278 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 3 2023-01-15T14:36:10.278 INFO:tasks.rados.rados.0.smithi090.stdout:1757: expect (ObjNum 617 snap 188 seq_num 617) 2023-01-15T14:36:10.279 INFO:tasks.rados.rados.0.smithi090.stdout:1757: done (2 left) 2023-01-15T14:36:10.279 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 2 2023-01-15T14:36:10.279 INFO:tasks.rados.rados.0.smithi090.stdout:1758: finishing write tid 1 to smithi09025264-17 2023-01-15T14:36:10.279 INFO:tasks.rados.rados.0.smithi090.stdout:1759: expect (ObjNum 575 snap 177 seq_num 575) 2023-01-15T14:36:10.279 INFO:tasks.rados.rados.0.smithi090.stdout:1758: finishing write tid 2 to smithi09025264-17 2023-01-15T14:36:10.279 INFO:tasks.rados.rados.0.smithi090.stdout:1759: done (1 left) 2023-01-15T14:36:10.279 INFO:tasks.rados.rados.0.smithi090.stdout: waiting on 1 2023-01-15T14:36:10.279 INFO:tasks.rados.rados.0.smithi090.stdout:1758: finishing write tid 3 to smithi09025264-17 2023-01-15T14:36:10.279 INFO:tasks.rados.rados.0.smithi090.stdout:1758: finishing write tid 4 to smithi09025264-17 2023-01-15T14:36:10.279 INFO:tasks.rados.rados.0.smithi090.stdout:1758: oid 17 updating version 0 to 504 2023-01-15T14:36:10.279 INFO:tasks.rados.rados.0.smithi090.stdout:1758: oid 17 version 504 is already newer than 503 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:1758: oid 17 version 504 is already newer than 502 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 17 v 504 (ObjNum 623 snap 191 seq_num 623) dirty exists 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:1758: left oid 17 (ObjNum 623 snap 191 seq_num 623) 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:1758: done (0 left) 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:1760: done (0 left) 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:1761: read oid 30 snap -1 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:1761: expect (ObjNum 605 snap 183 seq_num 605) 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:1762: rollback oid 12 current snap is 192 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:rollback oid 12 to 187 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:1763: read oid 45 snap -1 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:1763: expect (ObjNum 612 snap 188 seq_num 612) 2023-01-15T14:36:10.280 INFO:tasks.rados.rados.0.smithi090.stdout:1764: delete oid 41 current snap is 192 2023-01-15T14:36:10.281 INFO:tasks.rados.rados.0.smithi090.stdout:1761: expect (ObjNum 605 snap 183 seq_num 605) 2023-01-15T14:36:10.281 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:36:10.261+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:36:10.317 INFO:tasks.rados.rados.0.smithi090.stdout:1762: finishing rollback tid 0 to smithi09025264-12 2023-01-15T14:36:10.628 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:10.628 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:10.628 INFO:tasks.rados.rados.0.smithi090.stdout:1762: finishing rollback tid 1 to smithi09025264-12 2023-01-15T14:36:10.628 INFO:tasks.rados.rados.0.smithi090.stdout:1761: done (3 left) 2023-01-15T14:36:10.629 INFO:tasks.rados.rados.0.smithi090.stdout:1764: done (2 left) 2023-01-15T14:36:10.629 INFO:tasks.rados.rados.0.smithi090.stdout:1765: write (excl) oid 28 current snap is 192 2023-01-15T14:36:10.629 INFO:tasks.rados.rados.0.smithi090.stdout:1765: seq_num 625 ranges {504375=571571,1490187=663208} 2023-01-15T14:36:10.629 INFO:tasks.rados.rados.0.smithi090.stdout:1765: writing smithi09025264-28 from 504375 to 1075946 tid 1 2023-01-15T14:36:10.629 INFO:tasks.rados.rados.0.smithi090.stdout:1765: writing smithi09025264-28 from 1490187 to 2153395 tid 2 2023-01-15T14:36:10.629 INFO:tasks.rados.rados.0.smithi090.stdout:1766: snap_remove snap 180 2023-01-15T14:36:10.629 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2023-01-15T14:36:10.631 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:36:10.631 INFO:teuthology.orchestra.run.smithi090.stdout:{ 2023-01-15T14:36:10.631 INFO:teuthology.orchestra.run.smithi090.stdout: "epoch": 480, 2023-01-15T14:36:10.631 INFO:teuthology.orchestra.run.smithi090.stdout: "fsid": "1c77fb2b-d1a7-4884-a24e-3935e16d0b6b", 2023-01-15T14:36:10.631 INFO:teuthology.orchestra.run.smithi090.stdout: "created": "2023-01-15T14:27:57.735717+0000", 2023-01-15T14:36:10.631 INFO:teuthology.orchestra.run.smithi090.stdout: "modified": "2023-01-15T14:36:10.004269+0000", 2023-01-15T14:36:10.631 INFO:teuthology.orchestra.run.smithi090.stdout: "last_up_change": "2023-01-15T14:28:04.837522+0000", 2023-01-15T14:36:10.631 INFO:teuthology.orchestra.run.smithi090.stdout: "last_in_change": "2023-01-15T14:35:55.993237+0000", 2023-01-15T14:36:10.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2023-01-15T14:36:10.631 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_num": 5801984, 2023-01-15T14:36:10.632 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_set": [ 2023-01-15T14:36:10.632 INFO:teuthology.orchestra.run.smithi090.stdout: "noscrub", 2023-01-15T14:36:10.632 INFO:teuthology.orchestra.run.smithi090.stdout: "pglog_hardlimit", 2023-01-15T14:36:10.632 INFO:teuthology.orchestra.run.smithi090.stdout: "purged_snapdirs", 2023-01-15T14:36:10.632 INFO:teuthology.orchestra.run.smithi090.stdout: "recovery_deletes", 2023-01-15T14:36:10.632 INFO:teuthology.orchestra.run.smithi090.stdout: "sortbitwise" 2023-01-15T14:36:10.632 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:36:10.632 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_version": 4, 2023-01-15T14:36:10.632 INFO:teuthology.orchestra.run.smithi090.stdout: "full_ratio": 0.94999998807907104, 2023-01-15T14:36:10.632 INFO:teuthology.orchestra.run.smithi090.stdout: "backfillfull_ratio": 0.89999997615814209, 2023-01-15T14:36:10.632 INFO:teuthology.orchestra.run.smithi090.stdout: "nearfull_ratio": 0.85000002384185791, 2023-01-15T14:36:10.633 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_snapshot": "", 2023-01-15T14:36:10.633 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_max": 3, 2023-01-15T14:36:10.633 INFO:teuthology.orchestra.run.smithi090.stdout: "max_osd": 8, 2023-01-15T14:36:10.633 INFO:teuthology.orchestra.run.smithi090.stdout: "require_min_compat_client": "luminous", 2023-01-15T14:36:10.633 INFO:teuthology.orchestra.run.smithi090.stdout: "min_compat_client": "luminous", 2023-01-15T14:36:10.633 INFO:teuthology.orchestra.run.smithi090.stdout: "require_osd_release": "reef", 2023-01-15T14:36:10.633 INFO:teuthology.orchestra.run.smithi090.stdout: "pools": [ 2023-01-15T14:36:10.633 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.633 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 1, 2023-01-15T14:36:10.633 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": ".mgr", 2023-01-15T14:36:10.633 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:04.899282+0000", 2023-01-15T14:36:10.634 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:36:10.634 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:36:10.634 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:36:10.634 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:36:10.634 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:36:10.634 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:36:10.634 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:36:10.634 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:36:10.634 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:36:10.634 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 1, 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 1, 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 1, 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 1, 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 1, 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:36:10.635 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:36:10.636 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:36:10.636 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:36:10.636 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.636 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "16", 2023-01-15T14:36:10.636 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:36:10.636 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:36:10.637 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:36:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:36:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:36:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:36:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:36:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:36:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:36:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:36:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:36:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:36:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:36:10.638 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:36:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:36:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:36:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:36:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:36:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:36:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:36:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:36:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:36:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:36:10.639 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:36:10.702 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:36:10.702 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:36:10.702 INFO:teuthology.orchestra.run.smithi090.stdout: "options": { 2023-01-15T14:36:10.702 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_max": 32, 2023-01-15T14:36:10.702 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_min": 1 2023-01-15T14:36:10.702 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.703 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:36:10.703 INFO:teuthology.orchestra.run.smithi090.stdout: "mgr": {} 2023-01-15T14:36:10.703 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:10.703 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.703 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.703 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 2, 2023-01-15T14:36:10.703 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "rbd", 2023-01-15T14:36:10.703 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:11.115002+0000", 2023-01-15T14:36:10.703 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 1, 2023-01-15T14:36:10.796 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool", 2023-01-15T14:36:10.796 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:36:10.796 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:36:10.796 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:36:10.796 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:36:10.796 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:36:10.796 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:36:10.797 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:36:10.797 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:36:10.797 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:36:10.797 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:36:10.798 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 8, 2023-01-15T14:36:10.798 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 8, 2023-01-15T14:36:10.798 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 8, 2023-01-15T14:36:10.798 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 8, 2023-01-15T14:36:10.798 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 8, 2023-01-15T14:36:10.798 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:36:10.798 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:36:10.798 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:36:10.798 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:36:10.798 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:36:10.798 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:36:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:36:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "19", 2023-01-15T14:36:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:36:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:36:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "0", 2023-01-15T14:36:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:36:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "selfmanaged", 2023-01-15T14:36:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 0, 2023-01-15T14:36:10.799 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 0, 2023-01-15T14:36:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [], 2023-01-15T14:36:10.800 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:36:10.865 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:36:10.865 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:36:10.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:36:10.865 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:36:10.866 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:36:10.866 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:36:10.866 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:36:10.866 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:36:10.866 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:36:10.866 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:36:10.866 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:36:10.866 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:36:10.866 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:36:10.866 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:36:10.866 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:36:10.867 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:36:10.867 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:36:10.867 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.867 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:36:10.867 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:36:10.867 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:36:10.867 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:36:10.867 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:36:10.867 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:36:10.867 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:36:10.867 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:36:10.868 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:36:10.868 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:36:10.868 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:36:10.868 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:36:10.868 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:36:10.868 INFO:teuthology.orchestra.run.smithi090.stdout: "rbd": {} 2023-01-15T14:36:10.868 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:10.868 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.868 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.868 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:36:10.868 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_name": "unique_pool_0", 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "create_time": "2023-01-15T14:28:22.434903+0000", 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "flags": 16385, 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "flags_names": "hashpspool,pool_snaps", 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "type": 1, 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "size": 3, 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "min_size": 2, 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_rule": 0, 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_count": 0, 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_target": 0, 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_barrier": 0, 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2023-01-15T14:36:10.869 INFO:teuthology.orchestra.run.smithi090.stdout: "object_hash": 2, 2023-01-15T14:36:10.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_autoscale_mode": "off", 2023-01-15T14:36:10.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num": 156, 2023-01-15T14:36:10.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num": 146, 2023-01-15T14:36:10.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_placement_num_target": 156, 2023-01-15T14:36:10.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_target": 156, 2023-01-15T14:36:10.870 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_num_pending": 156, 2023-01-15T14:36:10.870 INFO:teuthology.orchestra.run.smithi090.stdout: "last_pg_merge_meta": { 2023-01-15T14:36:10.870 INFO:teuthology.orchestra.run.smithi090.stdout: "source_pgid": "0.0", 2023-01-15T14:36:10.870 INFO:teuthology.orchestra.run.smithi090.stdout: "ready_epoch": 0, 2023-01-15T14:36:10.870 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_started": 0, 2023-01-15T14:36:10.870 INFO:teuthology.orchestra.run.smithi090.stdout: "last_epoch_clean": 0, 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: "source_version": "0'0", 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: "target_version": "0'0" 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: "last_change": "480", 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend": "0", 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_prenautilus": "0", 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: "last_force_op_resend_preluminous": "480", 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: "auid": 0, 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_mode": "pool", 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_seq": 365, 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: "snap_epoch": 480, 2023-01-15T14:36:10.871 INFO:teuthology.orchestra.run.smithi090.stdout: "pool_snaps": [ 2023-01-15T14:36:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 246, 2023-01-15T14:36:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:33:19.645695+0000", 2023-01-15T14:36:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap127" 2023-01-15T14:36:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 247, 2023-01-15T14:36:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:33:20.830206+0000", 2023-01-15T14:36:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap128" 2023-01-15T14:36:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.872 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 297, 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:34:36.938666+0000", 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap158" 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 303, 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:34:44.970657+0000", 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap163" 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 306, 2023-01-15T14:36:10.873 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:34:48.256065+0000", 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap165" 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 316, 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:34:57.966853+0000", 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap171" 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 317, 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:34:59.830917+0000", 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap172" 2023-01-15T14:36:10.874 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.875 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 319, 2023-01-15T14:36:10.875 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:35:02.012776+0000", 2023-01-15T14:36:10.875 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap173" 2023-01-15T14:36:10.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.875 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 343, 2023-01-15T14:36:10.875 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:35:34.125209+0000", 2023-01-15T14:36:10.875 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap181" 2023-01-15T14:36:10.875 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.875 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 344, 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:35:36.713027+0000", 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap182" 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 348, 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:35:45.770578+0000", 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap183" 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 350, 2023-01-15T14:36:10.876 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:35:47.471517+0000", 2023-01-15T14:36:10.877 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap185" 2023-01-15T14:36:10.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.877 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 351, 2023-01-15T14:36:10.877 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:35:48.814605+0000", 2023-01-15T14:36:10.877 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap186" 2023-01-15T14:36:10.877 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.877 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.877 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 352, 2023-01-15T14:36:10.877 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:35:50.020648+0000", 2023-01-15T14:36:10.877 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap187" 2023-01-15T14:36:10.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.878 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 355, 2023-01-15T14:36:10.878 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:35:52.497549+0000", 2023-01-15T14:36:10.878 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap188" 2023-01-15T14:36:10.878 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.878 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.878 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 357, 2023-01-15T14:36:10.878 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:35:55.005618+0000", 2023-01-15T14:36:10.878 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap189" 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 362, 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:36:05.367011+0000", 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap190" 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 363, 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:36:07.819768+0000", 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap191" 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.879 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: "snapid": 365, 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: "stamp": "2023-01-15T14:36:09.641805+0000", 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: "name": "smithi09025264-snap192" 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps": "[]", 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_bytes": 0, 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: "quota_max_objects": 0, 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: "tiers": [], 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: "tier_of": -1, 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: "read_tier": -1, 2023-01-15T14:36:10.880 INFO:teuthology.orchestra.run.smithi090.stdout: "write_tier": -1, 2023-01-15T14:36:10.881 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_mode": "none", 2023-01-15T14:36:10.881 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_bytes": 0, 2023-01-15T14:36:10.881 INFO:teuthology.orchestra.run.smithi090.stdout: "target_max_objects": 0, 2023-01-15T14:36:10.881 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_ratio_micro": 400000, 2023-01-15T14:36:10.881 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2023-01-15T14:36:10.881 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_target_full_ratio_micro": 800000, 2023-01-15T14:36:10.881 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_flush_age": 0, 2023-01-15T14:36:10.881 INFO:teuthology.orchestra.run.smithi090.stdout: "cache_min_evict_age": 0, 2023-01-15T14:36:10.881 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profile": "", 2023-01-15T14:36:10.881 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_params": { 2023-01-15T14:36:10.881 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "none" 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_period": 0, 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_count": 0, 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: "use_gmt_hitset": true, 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: "min_read_recency_for_promote": 0, 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: "min_write_recency_for_promote": 0, 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_grade_decay_rate": 0, 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: "hit_set_search_last_n": 0, 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: "grade_table": [], 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: "stripe_width": 0, 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: "expected_num_objects": 0, 2023-01-15T14:36:10.882 INFO:teuthology.orchestra.run.smithi090.stdout: "fast_read": false, 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: "options": {}, 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: "application_metadata": { 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: "rados": {} 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "e55c5ac4-3cff-4ee5-8719-373648f83d8d", 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:36:10.883 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:36:10.884 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:36:10.884 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:36:10.884 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:36:10.884 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:36:10.884 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:36:10.884 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 479, 2023-01-15T14:36:10.884 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:36:10.884 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:36:10.884 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:36:10.884 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:10.884 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.885 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:10.885 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6808", 2023-01-15T14:36:10.885 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:36:10.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.885 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.885 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:10.885 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6811", 2023-01-15T14:36:10.885 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:36:10.885 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:10.885 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:10.885 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.886 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:36:10.886 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:10.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.886 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:10.886 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6813", 2023-01-15T14:36:10.886 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:36:10.886 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.886 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.886 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:10.886 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6815", 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6822", 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.887 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:10.888 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6823", 2023-01-15T14:36:10.888 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:36:10.888 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:10.888 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:10.888 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.888 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:36:10.888 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:10.888 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.888 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:10.888 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6818", 2023-01-15T14:36:10.889 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:36:10.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.889 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.889 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:10.889 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6820", 2023-01-15T14:36:10.889 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23473 2023-01-15T14:36:10.889 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:10.889 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:10.889 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.889 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6811/23473", 2023-01-15T14:36:10.889 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6815/23473", 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6823/23473", 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6820/23473", 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "2c1b33c9-0a1e-4734-936d-d51d64fe997a", 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:36:10.890 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 479, 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:10.891 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6800", 2023-01-15T14:36:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:36:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:10.892 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.000 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6801", 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6802", 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:36:11.001 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6803", 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.002 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6806", 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6807", 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.003 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6804", 2023-01-15T14:36:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:36:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6805", 2023-01-15T14:36:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23474 2023-01-15T14:36:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.004 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6801/23474", 2023-01-15T14:36:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6803/23474", 2023-01-15T14:36:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6807/23474", 2023-01-15T14:36:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6805/23474", 2023-01-15T14:36:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:36:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:36:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:36:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.005 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "b314174a-eca9-4baa-a60c-2589f1fc9a8e", 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 0, 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 0, 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0, 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 438, 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:36:11.006 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:36:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6809", 2023-01-15T14:36:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:36:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.007 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6810", 2023-01-15T14:36:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:36:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:36:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.008 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6812", 2023-01-15T14:36:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:36:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6814", 2023-01-15T14:36:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:36:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:36:11.009 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.036 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.036 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.037 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6819", 2023-01-15T14:36:11.037 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:36:11.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.037 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.037 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.037 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6821", 2023-01-15T14:36:11.037 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:36:11.037 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.037 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.037 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.038 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:36:11.038 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.038 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.038 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.039 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6816", 2023-01-15T14:36:11.039 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:36:11.039 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.039 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.039 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.039 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6817", 2023-01-15T14:36:11.039 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23475 2023-01-15T14:36:11.039 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.039 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.040 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.040 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6810/23475", 2023-01-15T14:36:11.040 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6814/23475", 2023-01-15T14:36:11.040 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6821/23475", 2023-01-15T14:36:11.040 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6817/23475", 2023-01-15T14:36:11.040 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:36:11.040 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:36:11.041 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:36:11.041 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.041 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.041 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.041 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:36:11.041 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "29eb0b17-ed5d-4665-a347-2fb8406d9c62", 2023-01-15T14:36:11.041 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:36:11.041 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:36:11.041 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:36:11.041 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0, 2023-01-15T14:36:11.041 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:36:11.042 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:36:11.042 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:36:11.042 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 472, 2023-01-15T14:36:11.042 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:36:11.042 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:36:11.042 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:36:11.042 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.042 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.042 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.042 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6824", 2023-01-15T14:36:11.043 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:36:11.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.043 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.043 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.043 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6825", 2023-01-15T14:36:11.043 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:36:11.043 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.043 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.043 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.043 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:36:11.043 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.044 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6826", 2023-01-15T14:36:11.044 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:36:11.044 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.044 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.044 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.044 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6827", 2023-01-15T14:36:11.044 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:36:11.044 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.044 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6830", 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6831", 2023-01-15T14:36:11.045 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6828", 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.046 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.047 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.90:6829", 2023-01-15T14:36:11.047 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23778 2023-01-15T14:36:11.047 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.047 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.047 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.047 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.90:6825/23778", 2023-01-15T14:36:11.047 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.90:6827/23778", 2023-01-15T14:36:11.047 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.90:6831/23778", 2023-01-15T14:36:11.047 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.90:6829/23778", 2023-01-15T14:36:11.047 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:36:11.047 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:36:11.048 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:36:11.048 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.048 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.048 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.048 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:36:11.048 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "59362feb-caa9-44e3-b6d0-b79397dacbfc", 2023-01-15T14:36:11.048 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:36:11.049 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:36:11.049 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:36:11.049 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:36:11.049 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:36:11.049 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:36:11.049 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:36:11.049 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 469, 2023-01-15T14:36:11.049 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:36:11.050 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:36:11.050 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:36:11.050 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.050 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.050 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6800", 2023-01-15T14:36:11.050 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:36:11.050 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.050 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.050 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.050 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6801", 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6802", 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.051 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.052 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.052 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6803", 2023-01-15T14:36:11.052 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:36:11.052 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.052 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.052 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.052 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:36:11.052 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.052 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.052 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.053 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6806", 2023-01-15T14:36:11.053 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:36:11.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.053 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.053 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.053 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6807", 2023-01-15T14:36:11.053 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:36:11.053 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.053 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.053 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.053 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:36:11.054 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.054 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.054 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6804", 2023-01-15T14:36:11.054 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:36:11.054 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.054 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.054 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.054 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6805", 2023-01-15T14:36:11.054 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23523 2023-01-15T14:36:11.054 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.055 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.055 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6801/23523", 2023-01-15T14:36:11.055 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6803/23523", 2023-01-15T14:36:11.055 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6807/23523", 2023-01-15T14:36:11.055 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6805/23523", 2023-01-15T14:36:11.055 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:36:11.055 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:36:11.055 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:36:11.055 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.055 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.056 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.056 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:36:11.056 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "fbc8821c-8d78-40f8-a571-b18a70d42000", 2023-01-15T14:36:11.056 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:36:11.056 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:36:11.056 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:36:11.056 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:36:11.056 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:36:11.057 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:36:11.057 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:36:11.057 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 473, 2023-01-15T14:36:11.057 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:36:11.057 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:36:11.057 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:36:11.057 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.057 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.057 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6824", 2023-01-15T14:36:11.122 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:36:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6825", 2023-01-15T14:36:11.123 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6826", 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.124 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6827", 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6830", 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.125 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.126 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.126 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6831", 2023-01-15T14:36:11.126 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:36:11.126 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.126 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.126 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.127 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:36:11.127 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.127 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.127 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.127 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6828", 2023-01-15T14:36:11.127 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:36:11.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.128 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.128 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.128 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6829", 2023-01-15T14:36:11.128 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23525 2023-01-15T14:36:11.128 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.128 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.128 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.128 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6825/23525", 2023-01-15T14:36:11.128 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6827/23525", 2023-01-15T14:36:11.128 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6831/23525", 2023-01-15T14:36:11.129 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6829/23525", 2023-01-15T14:36:11.129 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:36:11.129 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:36:11.129 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:36:11.129 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.129 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.129 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.129 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:36:11.129 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "34550470-3561-492f-b605-b5f0e8052c1e", 2023-01-15T14:36:11.129 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:36:11.130 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:36:11.130 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:36:11.130 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 0.02410888671875, 2023-01-15T14:36:11.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:36:11.130 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:36:11.130 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:36:11.130 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 449, 2023-01-15T14:36:11.130 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:36:11.130 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:36:11.130 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:36:11.130 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.131 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.131 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6808", 2023-01-15T14:36:11.131 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:36:11.131 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.131 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.131 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.131 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6809", 2023-01-15T14:36:11.131 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:36:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:36:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6810", 2023-01-15T14:36:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:36:11.132 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6811", 2023-01-15T14:36:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:36:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:36:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.133 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.134 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6814", 2023-01-15T14:36:11.134 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:36:11.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.134 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.134 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.134 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6815", 2023-01-15T14:36:11.134 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:36:11.134 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.134 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.134 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.134 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:36:11.135 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.135 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.135 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6812", 2023-01-15T14:36:11.135 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:36:11.135 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.135 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.135 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.135 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6813", 2023-01-15T14:36:11.135 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23524 2023-01-15T14:36:11.135 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.136 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.136 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.136 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6809/23524", 2023-01-15T14:36:11.136 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6811/23524", 2023-01-15T14:36:11.136 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6815/23524", 2023-01-15T14:36:11.136 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6813/23524", 2023-01-15T14:36:11.136 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:36:11.200 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:36:11.200 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:36:11.200 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.200 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.200 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.201 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:36:11.201 INFO:teuthology.orchestra.run.smithi090.stdout: "uuid": "1f123d0d-d617-4771-b516-21aa5a29b734", 2023-01-15T14:36:11.201 INFO:teuthology.orchestra.run.smithi090.stdout: "up": 1, 2023-01-15T14:36:11.201 INFO:teuthology.orchestra.run.smithi090.stdout: "in": 1, 2023-01-15T14:36:11.201 INFO:teuthology.orchestra.run.smithi090.stdout: "weight": 1, 2023-01-15T14:36:11.201 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_affinity": 1, 2023-01-15T14:36:11.201 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_begin": 0, 2023-01-15T14:36:11.201 INFO:teuthology.orchestra.run.smithi090.stdout: "last_clean_end": 0, 2023-01-15T14:36:11.201 INFO:teuthology.orchestra.run.smithi090.stdout: "up_from": 13, 2023-01-15T14:36:11.201 INFO:teuthology.orchestra.run.smithi090.stdout: "up_thru": 477, 2023-01-15T14:36:11.201 INFO:teuthology.orchestra.run.smithi090.stdout: "down_at": 0, 2023-01-15T14:36:11.202 INFO:teuthology.orchestra.run.smithi090.stdout: "lost_at": 0, 2023-01-15T14:36:11.202 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addrs": { 2023-01-15T14:36:11.202 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.202 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.202 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6816", 2023-01-15T14:36:11.202 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:36:11.202 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.202 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6817", 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addrs": { 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6818", 2023-01-15T14:36:11.203 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:36:11.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.204 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.204 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.204 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6819", 2023-01-15T14:36:11.204 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:36:11.204 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.204 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.204 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.204 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addrs": { 2023-01-15T14:36:11.204 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.205 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.205 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6822", 2023-01-15T14:36:11.205 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:36:11.205 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.205 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.205 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.205 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6823", 2023-01-15T14:36:11.205 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:36:11.205 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.205 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.206 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addrs": { 2023-01-15T14:36:11.206 INFO:teuthology.orchestra.run.smithi090.stdout: "addrvec": [ 2023-01-15T14:36:11.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.206 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v2", 2023-01-15T14:36:11.206 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6820", 2023-01-15T14:36:11.206 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:36:11.206 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.206 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.206 INFO:teuthology.orchestra.run.smithi090.stdout: "type": "v1", 2023-01-15T14:36:11.206 INFO:teuthology.orchestra.run.smithi090.stdout: "addr": "172.21.15.141:6821", 2023-01-15T14:36:11.207 INFO:teuthology.orchestra.run.smithi090.stdout: "nonce": 23526 2023-01-15T14:36:11.207 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.207 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.207 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.207 INFO:teuthology.orchestra.run.smithi090.stdout: "public_addr": "172.21.15.141:6817/23526", 2023-01-15T14:36:11.207 INFO:teuthology.orchestra.run.smithi090.stdout: "cluster_addr": "172.21.15.141:6819/23526", 2023-01-15T14:36:11.207 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_back_addr": "172.21.15.141:6823/23526", 2023-01-15T14:36:11.207 INFO:teuthology.orchestra.run.smithi090.stdout: "heartbeat_front_addr": "172.21.15.141:6821/23526", 2023-01-15T14:36:11.207 INFO:teuthology.orchestra.run.smithi090.stdout: "state": [ 2023-01-15T14:36:11.207 INFO:teuthology.orchestra.run.smithi090.stdout: "exists", 2023-01-15T14:36:11.207 INFO:teuthology.orchestra.run.smithi090.stdout: "up" 2023-01-15T14:36:11.208 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.208 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.208 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:36:11.208 INFO:teuthology.orchestra.run.smithi090.stdout: "osd_xinfo": [ 2023-01-15T14:36:11.208 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.208 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 0, 2023-01-15T14:36:11.208 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:36:11.208 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:36:11.208 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:36:11.208 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:36:11.208 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:35:29.600037+0000", 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 1, 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:35:53.225011+0000", 2023-01-15T14:36:11.209 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 2, 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 65536, 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:35:51.682355+0000", 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.210 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 3, 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:35:11.413612+0000", 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 4, 2023-01-15T14:36:11.211 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:36:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:36:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:36:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:36:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:36:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:35:49.679259+0000", 2023-01-15T14:36:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:36:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 5, 2023-01-15T14:36:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:36:11.212 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:36:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:36:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:36:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:36:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:35:12.273170+0000", 2023-01-15T14:36:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:36:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 6, 2023-01-15T14:36:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:36:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:36:11.213 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:36:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:36:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:36:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:35:31.178874+0000", 2023-01-15T14:36:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:36:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "osd": 7, 2023-01-15T14:36:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "down_stamp": "0.000000", 2023-01-15T14:36:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_probability": 0, 2023-01-15T14:36:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "laggy_interval": 0, 2023-01-15T14:36:11.214 INFO:teuthology.orchestra.run.smithi090.stdout: "features": 4540138322906710015, 2023-01-15T14:36:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: "old_weight": 0, 2023-01-15T14:36:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: "last_purged_snaps_scrub": "2023-01-15T14:34:58.084828+0000", 2023-01-15T14:36:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: "dead_epoch": 0 2023-01-15T14:36:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:36:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap": [], 2023-01-15T14:36:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_items": [ 2023-01-15T14:36:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.3", 2023-01-15T14:36:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:36:11.215 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.216 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 4, 2023-01-15T14:36:11.216 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 0 2023-01-15T14:36:11.216 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.216 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.216 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.216 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.11", 2023-01-15T14:36:11.216 INFO:teuthology.orchestra.run.smithi090.stdout: "mappings": [ 2023-01-15T14:36:11.216 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.216 INFO:teuthology.orchestra.run.smithi090.stdout: "from": 4, 2023-01-15T14:36:11.216 INFO:teuthology.orchestra.run.smithi090.stdout: "to": 0 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_upmap_primaries": [], 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: "pg_temp": [ 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.16", 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: 1, 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: 7 2023-01-15T14:36:11.217 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.1f", 2023-01-15T14:36:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:36:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: 0, 2023-01-15T14:36:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: 6 2023-01-15T14:36:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.50", 2023-01-15T14:36:11.218 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: 7, 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: 0 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.51", 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: 1, 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: 0 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.219 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.57", 2023-01-15T14:36:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:36:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: 0, 2023-01-15T14:36:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: 6 2023-01-15T14:36:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.65", 2023-01-15T14:36:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:36:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: 1, 2023-01-15T14:36:11.220 INFO:teuthology.orchestra.run.smithi090.stdout: 6 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: "pgid": "3.90", 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: "osds": [ 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: 0, 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: 6 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: "primary_temp": [], 2023-01-15T14:36:11.221 INFO:teuthology.orchestra.run.smithi090.stdout: "blocklist": {}, 2023-01-15T14:36:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "range_blocklist": {}, 2023-01-15T14:36:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "erasure_code_profiles": { 2023-01-15T14:36:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "default": { 2023-01-15T14:36:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "crush-failure-domain": "osd", 2023-01-15T14:36:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "k": "2", 2023-01-15T14:36:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "m": "1", 2023-01-15T14:36:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "plugin": "jerasure", 2023-01-15T14:36:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "technique": "reed_sol_van" 2023-01-15T14:36:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.222 INFO:teuthology.orchestra.run.smithi090.stdout: "removed_snaps_queue": [ 2023-01-15T14:36:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: "pool": 3, 2023-01-15T14:36:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: "snaps": [ 2023-01-15T14:36:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 299, 2023-01-15T14:36:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2023-01-15T14:36:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 307, 2023-01-15T14:36:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:36:11.223 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 320, 2023-01-15T14:36:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 2 2023-01-15T14:36:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 323, 2023-01-15T14:36:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:36:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 328, 2023-01-15T14:36:11.224 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:36:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 330, 2023-01-15T14:36:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:36:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 334, 2023-01-15T14:36:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:36:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.225 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 336, 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: }, 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: { 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: "begin": 349, 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: "length": 1 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: ] 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: ], 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: "new_removed_snaps": [], 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: "new_purged_snaps": [], 2023-01-15T14:36:11.226 INFO:teuthology.orchestra.run.smithi090.stdout: "crush_node_flags": {}, 2023-01-15T14:36:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: "device_class_flags": {}, 2023-01-15T14:36:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode": { 2023-01-15T14:36:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_enabled": false, 2023-01-15T14:36:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_bucket_count": 0, 2023-01-15T14:36:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: "degraded_stretch_mode": 0, 2023-01-15T14:36:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: "recovering_stretch_mode": 0, 2023-01-15T14:36:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: "stretch_mode_bucket": 0 2023-01-15T14:36:11.227 INFO:teuthology.orchestra.run.smithi090.stdout: } 2023-01-15T14:36:11.227 INFO:teuthology.orchestra.run.smithi090.stdout:} 2023-01-15T14:36:11.229 INFO:tasks.thrashosds.thrasher:j is {'epoch': 480, 'fsid': '1c77fb2b-d1a7-4884-a24e-3935e16d0b6b', 'created': '2023-01-15T14:27:57.735717+0000', 'modified': '2023-01-15T14:36:10.004269+0000', 'last_up_change': '2023-01-15T14:28:04.837522+0000', 'last_in_change': '2023-01-15T14:35:55.993237+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-01-15T14:28:04.899282+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-01-15T14:28:11.115002+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'unique_pool_0', 'create_time': '2023-01-15T14:28:22.434903+0000', 'flags': 16385, 'flags_names': 'hashpspool,pool_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 156, 'pg_placement_num': 146, 'pg_placement_num_target': 156, 'pg_num_target': 156, 'pg_num_pending': 156, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '480', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '480', 'auid': 0, 'snap_mode': 'pool', 'snap_seq': 365, 'snap_epoch': 480, 'pool_snaps': [{'snapid': 246, 'stamp': '2023-01-15T14:33:19.645695+0000', 'name': 'smithi09025264-snap127'}, {'snapid': 247, 'stamp': '2023-01-15T14:33:20.830206+0000', 'name': 'smithi09025264-snap128'}, {'snapid': 297, 'stamp': '2023-01-15T14:34:36.938666+0000', 'name': 'smithi09025264-snap158'}, {'snapid': 303, 'stamp': '2023-01-15T14:34:44.970657+0000', 'name': 'smithi09025264-snap163'}, {'snapid': 306, 'stamp': '2023-01-15T14:34:48.256065+0000', 'name': 'smithi09025264-snap165'}, {'snapid': 316, 'stamp': '2023-01-15T14:34:57.966853+0000', 'name': 'smithi09025264-snap171'}, {'snapid': 317, 'stamp': '2023-01-15T14:34:59.830917+0000', 'name': 'smithi09025264-snap172'}, {'snapid': 319, 'stamp': '2023-01-15T14:35:02.012776+0000', 'name': 'smithi09025264-snap173'}, {'snapid': 343, 'stamp': '2023-01-15T14:35:34.125209+0000', 'name': 'smithi09025264-snap181'}, {'snapid': 344, 'stamp': '2023-01-15T14:35:36.713027+0000', 'name': 'smithi09025264-snap182'}, {'snapid': 348, 'stamp': '2023-01-15T14:35:45.770578+0000', 'name': 'smithi09025264-snap183'}, {'snapid': 350, 'stamp': '2023-01-15T14:35:47.471517+0000', 'name': 'smithi09025264-snap185'}, {'snapid': 351, 'stamp': '2023-01-15T14:35:48.814605+0000', 'name': 'smithi09025264-snap186'}, {'snapid': 352, 'stamp': '2023-01-15T14:35:50.020648+0000', 'name': 'smithi09025264-snap187'}, {'snapid': 355, 'stamp': '2023-01-15T14:35:52.497549+0000', 'name': 'smithi09025264-snap188'}, {'snapid': 357, 'stamp': '2023-01-15T14:35:55.005618+0000', 'name': 'smithi09025264-snap189'}, {'snapid': 362, 'stamp': '2023-01-15T14:36:05.367011+0000', 'name': 'smithi09025264-snap190'}, {'snapid': 363, 'stamp': '2023-01-15T14:36:07.819768+0000', 'name': 'smithi09025264-snap191'}, {'snapid': 365, 'stamp': '2023-01-15T14:36:09.641805+0000', 'name': 'smithi09025264-snap192'}], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'e55c5ac4-3cff-4ee5-8719-373648f83d8d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 479, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6808', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6811', 'nonce': 23473}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6813', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6815', 'nonce': 23473}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6822', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6823', 'nonce': 23473}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6818', 'nonce': 23473}, {'type': 'v1', 'addr': '172.21.15.90:6820', 'nonce': 23473}]}, 'public_addr': '172.21.15.90:6811/23473', 'cluster_addr': '172.21.15.90:6815/23473', 'heartbeat_back_addr': '172.21.15.90:6823/23473', 'heartbeat_front_addr': '172.21.15.90:6820/23473', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '2c1b33c9-0a1e-4734-936d-d51d64fe997a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 479, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6800', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6801', 'nonce': 23474}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6802', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6803', 'nonce': 23474}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6806', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6807', 'nonce': 23474}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6804', 'nonce': 23474}, {'type': 'v1', 'addr': '172.21.15.90:6805', 'nonce': 23474}]}, 'public_addr': '172.21.15.90:6801/23474', 'cluster_addr': '172.21.15.90:6803/23474', 'heartbeat_back_addr': '172.21.15.90:6807/23474', 'heartbeat_front_addr': '172.21.15.90:6805/23474', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b314174a-eca9-4baa-a60c-2589f1fc9a8e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 438, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6809', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6810', 'nonce': 23475}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6812', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6814', 'nonce': 23475}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6819', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6821', 'nonce': 23475}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6816', 'nonce': 23475}, {'type': 'v1', 'addr': '172.21.15.90:6817', 'nonce': 23475}]}, 'public_addr': '172.21.15.90:6810/23475', 'cluster_addr': '172.21.15.90:6814/23475', 'heartbeat_back_addr': '172.21.15.90:6821/23475', 'heartbeat_front_addr': '172.21.15.90:6817/23475', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '29eb0b17-ed5d-4665-a347-2fb8406d9c62', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 472, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6824', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6825', 'nonce': 23778}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6826', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6827', 'nonce': 23778}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6830', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6831', 'nonce': 23778}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.90:6828', 'nonce': 23778}, {'type': 'v1', 'addr': '172.21.15.90:6829', 'nonce': 23778}]}, 'public_addr': '172.21.15.90:6825/23778', 'cluster_addr': '172.21.15.90:6827/23778', 'heartbeat_back_addr': '172.21.15.90:6831/23778', 'heartbeat_front_addr': '172.21.15.90:6829/23778', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '59362feb-caa9-44e3-b6d0-b79397dacbfc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 469, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6800', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6801', 'nonce': 23523}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6802', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6803', 'nonce': 23523}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6806', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6807', 'nonce': 23523}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6804', 'nonce': 23523}, {'type': 'v1', 'addr': '172.21.15.141:6805', 'nonce': 23523}]}, 'public_addr': '172.21.15.141:6801/23523', 'cluster_addr': '172.21.15.141:6803/23523', 'heartbeat_back_addr': '172.21.15.141:6807/23523', 'heartbeat_front_addr': '172.21.15.141:6805/23523', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'fbc8821c-8d78-40f8-a571-b18a70d42000', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 473, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6824', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6825', 'nonce': 23525}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6826', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6827', 'nonce': 23525}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6830', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6831', 'nonce': 23525}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6828', 'nonce': 23525}, {'type': 'v1', 'addr': '172.21.15.141:6829', 'nonce': 23525}]}, 'public_addr': '172.21.15.141:6825/23525', 'cluster_addr': '172.21.15.141:6827/23525', 'heartbeat_back_addr': '172.21.15.141:6831/23525', 'heartbeat_front_addr': '172.21.15.141:6829/23525', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '34550470-3561-492f-b605-b5f0e8052c1e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.02410888671875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 449, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6808', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6809', 'nonce': 23524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6810', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6811', 'nonce': 23524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6814', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6815', 'nonce': 23524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6812', 'nonce': 23524}, {'type': 'v1', 'addr': '172.21.15.141:6813', 'nonce': 23524}]}, 'public_addr': '172.21.15.141:6809/23524', 'cluster_addr': '172.21.15.141:6811/23524', 'heartbeat_back_addr': '172.21.15.141:6815/23524', 'heartbeat_front_addr': '172.21.15.141:6813/23524', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1f123d0d-d617-4771-b516-21aa5a29b734', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 477, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6816', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6817', 'nonce': 23526}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6818', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6819', 'nonce': 23526}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6822', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6823', 'nonce': 23526}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.141:6820', 'nonce': 23526}, {'type': 'v1', 'addr': '172.21.15.141:6821', 'nonce': 23526}]}, 'public_addr': '172.21.15.141:6817/23526', 'cluster_addr': '172.21.15.141:6819/23526', 'heartbeat_back_addr': '172.21.15.141:6823/23526', 'heartbeat_front_addr': '172.21.15.141:6821/23526', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:35:29.600037+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:35:53.225011+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 65536, 'last_purged_snaps_scrub': '2023-01-15T14:35:51.682355+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:35:11.413612+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:35:49.679259+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:35:12.273170+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:35:31.178874+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2023-01-15T14:34:58.084828+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '3.11', 'mappings': [{'from': 4, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.16', 'osds': [1, 7]}, {'pgid': '3.1f', 'osds': [0, 6]}, {'pgid': '3.50', 'osds': [7, 0]}, {'pgid': '3.51', 'osds': [1, 0]}, {'pgid': '3.57', 'osds': [0, 6]}, {'pgid': '3.65', 'osds': [1, 6]}, {'pgid': '3.90', 'osds': [0, 6]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 299, 'length': 2}, {'begin': 307, 'length': 1}, {'begin': 320, 'length': 2}, {'begin': 323, 'length': 1}, {'begin': 328, 'length': 1}, {'begin': 330, 'length': 1}, {'begin': 334, 'length': 1}, {'begin': 336, 'length': 1}, {'begin': 349, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2023-01-15T14:36:11.283 DEBUG:teuthology.orchestra.run.smithi090:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-01-15T14:36:11.284 INFO:tasks.ceph.osd.3.smithi090.stderr:2023-01-15T14:36:10.617+0000 7f91064d7700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 23772) UID: 0 2023-01-15T14:36:11.285 INFO:tasks.rados.rados.0.smithi090.stdout:1766: done (3 left) 2023-01-15T14:36:11.286 INFO:tasks.rados.rados.0.smithi090.stdout:1767: read oid 33 snap 164 2023-01-15T14:36:11.286 INFO:tasks.rados.rados.0.smithi090.stdout:1767: expect (ObjNum 378 snap 102 seq_num 378) 2023-01-15T14:36:11.286 INFO:tasks.rados.rados.0.smithi090.stdout:1762: finishing rollback tid 2 to smithi09025264-12 2023-01-15T14:36:11.286 INFO:tasks.rados.rados.0.smithi090.stdout:update_object_version oid 12 v 371 (ObjNum 439 snap 121 seq_num 439) dirty exists 2023-01-15T14:36:11.286 INFO:tasks.rados.rados.0.smithi090.stdout:1768: read oid 24 snap -1 2023-01-15T14:36:11.286 INFO:tasks.rados.rados.0.smithi090.stdout:1768: expect deleted 2023-01-15T14:36:11.286 INFO:tasks.rados.rados.0.smithi090.stdout:1762: done (4 left) 2023-01-15T14:36:11.286 INFO:tasks.rados.rados.0.smithi090.stdout:1769: snap_remove snap 186 2023-01-15T14:36:11.287 INFO:teuthology.orchestra.run.smithi090.stderr:nodeep-scrub is set 2023-01-15T14:36:11.288 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:36:11.277+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:36:11.386 INFO:tasks.ceph.osd.6.smithi141.stderr:2023-01-15T14:36:11.384+0000 7f16b287e700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 23515) UID: 0 2023-01-15T14:36:11.402 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:11.402 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:11.412 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2023-01-15T14:36:11.488 INFO:tasks.ceph.osd.5.smithi141.stderr:2023-01-15T14:36:11.484+0000 7fea4af52700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 23519) UID: 0 2023-01-15T14:36:11.562 INFO:teuthology.orchestra.run.smithi141.stdout:ERROR: (22) Invalid argument 2023-01-15T14:36:11.562 INFO:teuthology.orchestra.run.smithi141.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2023-01-15T14:36:11.572 DEBUG:teuthology.orchestra.run.smithi141:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2023-01-15T14:36:11.589 INFO:tasks.ceph.osd.0.smithi090.stderr:2023-01-15T14:36:11.581+0000 7f4f39afe700 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 23467) UID: 0 2023-01-15T14:36:11.613 INFO:teuthology.orchestra.run.smithi090.stdout: 2023-01-15T14:36:11.617 INFO:teuthology.orchestra.run.smithi090.stderr:dumped all 2023-01-15T14:36:11.631 INFO:teuthology.orchestra.run.smithi090.stdout:{"pg_ready":true,"pg_map":{"version":716,"stamp":"2023-01-15T14:36:11.023387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":392055222,"num_objects":179,"num_object_clones":123,"num_object_copies":537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":95,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":179,"num_whiteouts":14,"num_read":1440,"num_read_kb":1469725,"num_write":1163,"num_write_kb":401668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1355,"num_bytes_recovered":901015760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28372,"ondisk_log_size":28372,"up":465,"acting":456,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":490,"num_osds":7,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":655895660,"kb_used":7170752,"kb_used_data":648753820,"kb_used_omap":107494,"kb_used_meta":0,"kb_avail":648724908,"statfs":{"total":671637155840,"available":664294305792,"internally_reserved":29605888,"allocated":664323911680,"data_stored":664323911680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":110073997,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,0,1,0,0,1,0,2],"upper_bound":16384},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":7,"commit_latency_ns":0,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-9886165,"num_objects":-5,"num_object_clones":-5,"num_object_copies":-15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-52,"num_objects_misplaced":-5,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":-2,"num_read":12,"num_read_kb":6410,"num_write":18,"num_write_kb":7363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":48988990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.020572"},"pg_stats":[{"pgid":"3.9b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:36:10.004269+0000","last_change":"2023-01-15T14:36:10.004269+0000","last_active":"2023-01-15T14:36:10.004269+0000","last_peered":"2023-01-15T14:36:10.004269+0000","last_clean":"2023-01-15T14:36:10.004269+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:36:10.004269+0000","last_undegraded":"2023-01-15T14:36:10.004269+0000","last_fullsized":"2023-01-15T14:36:10.004269+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_clean_scrub_stamp":"2023-01-15T14:36:10.004269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:36:10.004269+0000","last_change":"2023-01-15T14:36:10.004269+0000","last_active":"2023-01-15T14:36:10.004269+0000","last_peered":"2023-01-15T14:36:10.004269+0000","last_clean":"2023-01-15T14:36:10.004269+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:36:10.004269+0000","last_undegraded":"2023-01-15T14:36:10.004269+0000","last_fullsized":"2023-01-15T14:36:10.004269+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_clean_scrub_stamp":"2023-01-15T14:36:10.004269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.99","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:36:10.004269+0000","last_change":"2023-01-15T14:36:10.004269+0000","last_active":"2023-01-15T14:36:10.004269+0000","last_peered":"2023-01-15T14:36:10.004269+0000","last_clean":"2023-01-15T14:36:10.004269+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:36:10.004269+0000","last_undegraded":"2023-01-15T14:36:10.004269+0000","last_fullsized":"2023-01-15T14:36:10.004269+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_clean_scrub_stamp":"2023-01-15T14:36:10.004269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.98","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:36:10.004269+0000","last_change":"2023-01-15T14:36:10.004269+0000","last_active":"2023-01-15T14:36:10.004269+0000","last_peered":"2023-01-15T14:36:10.004269+0000","last_clean":"2023-01-15T14:36:10.004269+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:36:10.004269+0000","last_undegraded":"2023-01-15T14:36:10.004269+0000","last_fullsized":"2023-01-15T14:36:10.004269+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_clean_scrub_stamp":"2023-01-15T14:36:10.004269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.97","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:36:10.004269+0000","last_change":"2023-01-15T14:36:10.004269+0000","last_active":"2023-01-15T14:36:10.004269+0000","last_peered":"2023-01-15T14:36:10.004269+0000","last_clean":"2023-01-15T14:36:10.004269+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:36:10.004269+0000","last_undegraded":"2023-01-15T14:36:10.004269+0000","last_fullsized":"2023-01-15T14:36:10.004269+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_clean_scrub_stamp":"2023-01-15T14:36:10.004269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.96","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:36:10.004269+0000","last_change":"2023-01-15T14:36:10.004269+0000","last_active":"2023-01-15T14:36:10.004269+0000","last_peered":"2023-01-15T14:36:10.004269+0000","last_clean":"2023-01-15T14:36:10.004269+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:36:10.004269+0000","last_undegraded":"2023-01-15T14:36:10.004269+0000","last_fullsized":"2023-01-15T14:36:10.004269+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_clean_scrub_stamp":"2023-01-15T14:36:10.004269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.95","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:36:10.004269+0000","last_change":"2023-01-15T14:36:10.004269+0000","last_active":"2023-01-15T14:36:10.004269+0000","last_peered":"2023-01-15T14:36:10.004269+0000","last_clean":"2023-01-15T14:36:10.004269+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:36:10.004269+0000","last_undegraded":"2023-01-15T14:36:10.004269+0000","last_fullsized":"2023-01-15T14:36:10.004269+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_clean_scrub_stamp":"2023-01-15T14:36:10.004269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.94","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:36:10.004269+0000","last_change":"2023-01-15T14:36:10.004269+0000","last_active":"2023-01-15T14:36:10.004269+0000","last_peered":"2023-01-15T14:36:10.004269+0000","last_clean":"2023-01-15T14:36:10.004269+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:36:10.004269+0000","last_undegraded":"2023-01-15T14:36:10.004269+0000","last_fullsized":"2023-01-15T14:36:10.004269+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_clean_scrub_stamp":"2023-01-15T14:36:10.004269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.93","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:36:10.004269+0000","last_change":"2023-01-15T14:36:10.004269+0000","last_active":"2023-01-15T14:36:10.004269+0000","last_peered":"2023-01-15T14:36:10.004269+0000","last_clean":"2023-01-15T14:36:10.004269+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:36:10.004269+0000","last_undegraded":"2023-01-15T14:36:10.004269+0000","last_fullsized":"2023-01-15T14:36:10.004269+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_clean_scrub_stamp":"2023-01-15T14:36:10.004269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.92","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-01-15T14:36:10.004269+0000","last_change":"2023-01-15T14:36:10.004269+0000","last_active":"2023-01-15T14:36:10.004269+0000","last_peered":"2023-01-15T14:36:10.004269+0000","last_clean":"2023-01-15T14:36:10.004269+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-01-15T14:36:10.004269+0000","last_undegraded":"2023-01-15T14:36:10.004269+0000","last_fullsized":"2023-01-15T14:36:10.004269+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:36:10.004269+0000","last_clean_scrub_stamp":"2023-01-15T14:36:10.004269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.91","version":"458'396","reported_seq":2052,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.105637+0000","last_change":"2023-01-15T14:36:04.787181+0000","last_active":"2023-01-15T14:36:07.105637+0000","last_peered":"2023-01-15T14:36:07.105637+0000","last_clean":"2023-01-15T14:36:07.105637+0000","last_became_active":"2023-01-15T14:35:55.531898+0000","last_became_peered":"2023-01-15T14:35:55.531898+0000","last_unstale":"2023-01-15T14:36:07.105637+0000","last_undegraded":"2023-01-15T14:36:07.105637+0000","last_fullsized":"2023-01-15T14:36:07.105637+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":8,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":396,"log_dups_size":0,"ondisk_log_size":396,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.62781841199999999,"stat_sum":{"num_bytes":10810582,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":25,"num_read_kb":24146,"num_write":21,"num_write_kb":14138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":22433542,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.90","version":"454'443","reported_seq":1833,"reported_epoch":477,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:36:07.104652+0000","last_change":"2023-01-15T14:35:58.624461+0000","last_active":"2023-01-15T14:36:07.104652+0000","last_peered":"2023-01-15T14:36:07.104652+0000","last_clean":"2023-01-15T14:35:50.519633+0000","last_became_active":"2023-01-15T14:35:58.619270+0000","last_became_peered":"2023-01-15T14:35:58.619270+0000","last_unstale":"2023-01-15T14:36:07.104652+0000","last_undegraded":"2023-01-15T14:35:58.011672+0000","last_fullsized":"2023-01-15T14:35:58.010289+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":8,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4003031,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":31691,"num_write":28,"num_write_kb":9855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":17879824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6],"avail_no_missing":["0","3","4","6"],"object_location_counts":[{"shards":"0,3,4,6","objects":3},{"shards":"0,6","objects":-1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.8f","version":"155'145","reported_seq":1285,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.005484+0000","last_change":"2023-01-15T14:36:04.480187+0000","last_active":"2023-01-15T14:36:07.005484+0000","last_peered":"2023-01-15T14:36:07.005484+0000","last_clean":"2023-01-15T14:36:07.005484+0000","last_became_active":"2023-01-15T14:35:55.308909+0000","last_became_peered":"2023-01-15T14:35:55.308909+0000","last_unstale":"2023-01-15T14:36:07.005484+0000","last_undegraded":"2023-01-15T14:36:07.005484+0000","last_fullsized":"2023-01-15T14:36:07.005484+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":8,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.31882782399999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.8e","version":"139'38","reported_seq":56,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.005469+0000","last_change":"2023-01-15T14:36:03.470255+0000","last_active":"2023-01-15T14:36:07.005469+0000","last_peered":"2023-01-15T14:36:07.005469+0000","last_clean":"2023-01-15T14:36:07.005469+0000","last_became_active":"2023-01-15T14:35:52.189818+0000","last_became_peered":"2023-01-15T14:35:52.189818+0000","last_unstale":"2023-01-15T14:36:07.005469+0000","last_undegraded":"2023-01-15T14:36:07.005469+0000","last_fullsized":"2023-01-15T14:36:07.005469+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":8,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.41855158999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.8d","version":"296'232","reported_seq":1661,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.329971+0000","last_change":"2023-01-15T14:36:09.330276+0000","last_active":"2023-01-15T14:36:09.329971+0000","last_peered":"2023-01-15T14:36:09.329971+0000","last_clean":"2023-01-15T14:36:09.329971+0000","last_became_active":"2023-01-15T14:35:55.205822+0000","last_became_peered":"2023-01-15T14:35:55.205822+0000","last_unstale":"2023-01-15T14:36:09.329971+0000","last_undegraded":"2023-01-15T14:36:09.329971+0000","last_fullsized":"2023-01-15T14:36:09.329971+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":8,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.31742833100000001,"stat_sum":{"num_bytes":2306688,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7821,"num_write":6,"num_write_kb":1640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.8c","version":"472'166","reported_seq":1667,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.005546+0000","last_change":"2023-01-15T14:36:02.957673+0000","last_active":"2023-01-15T14:36:07.005546+0000","last_peered":"2023-01-15T14:36:07.005546+0000","last_clean":"2023-01-15T14:36:07.005546+0000","last_became_active":"2023-01-15T14:35:58.408649+0000","last_became_peered":"2023-01-15T14:35:58.408649+0000","last_unstale":"2023-01-15T14:36:07.005546+0000","last_undegraded":"2023-01-15T14:36:07.005546+0000","last_fullsized":"2023-01-15T14:36:07.005546+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":468,"parent":"0.0","parent_split_bits":8,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.60423175799999995,"stat_sum":{"num_bytes":4142098,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":33625,"num_write":13,"num_write_kb":2913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":23960242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.8b","version":"155'53","reported_seq":62,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.328123+0000","last_change":"2023-01-15T14:36:09.328990+0000","last_active":"2023-01-15T14:36:09.328123+0000","last_peered":"2023-01-15T14:36:09.328123+0000","last_clean":"2023-01-15T14:36:09.328123+0000","last_became_active":"2023-01-15T14:35:52.450506+0000","last_became_peered":"2023-01-15T14:35:52.450506+0000","last_unstale":"2023-01-15T14:36:09.328123+0000","last_undegraded":"2023-01-15T14:36:09.328123+0000","last_fullsized":"2023-01-15T14:36:09.328123+0000","mapping_epoch":461,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":462,"parent":"0.0","parent_split_bits":8,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.31647914100000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.8a","version":"252'279","reported_seq":1513,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.001732+0000","last_change":"2023-01-15T14:36:01.051889+0000","last_active":"2023-01-15T14:36:07.001732+0000","last_peered":"2023-01-15T14:36:07.001732+0000","last_clean":"2023-01-15T14:36:07.001732+0000","last_became_active":"2023-01-15T14:35:55.203949+0000","last_became_peered":"2023-01-15T14:35:55.203949+0000","last_unstale":"2023-01-15T14:36:07.001732+0000","last_undegraded":"2023-01-15T14:36:07.001732+0000","last_fullsized":"2023-01-15T14:36:07.001732+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":8,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.83811411899999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.89","version":"110'149","reported_seq":531,"reported_epoch":479,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:09.323256+0000","last_change":"2023-01-15T14:36:09.323443+0000","last_active":"2023-01-15T14:36:09.323256+0000","last_peered":"2023-01-15T14:36:09.323256+0000","last_clean":"2023-01-15T14:36:09.323256+0000","last_became_active":"2023-01-15T14:36:00.220638+0000","last_became_peered":"2023-01-15T14:36:00.220638+0000","last_unstale":"2023-01-15T14:36:09.323256+0000","last_undegraded":"2023-01-15T14:36:09.323256+0000","last_fullsized":"2023-01-15T14:36:09.323256+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":8,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.30979219299999999,"stat_sum":{"num_bytes":714293,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":6218,"num_write":4,"num_write_kb":1543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.88","version":"179'204","reported_seq":1637,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.019319+0000","last_change":"2023-01-15T14:36:09.020167+0000","last_active":"2023-01-15T14:36:09.019319+0000","last_peered":"2023-01-15T14:36:09.019319+0000","last_clean":"2023-01-15T14:36:09.019319+0000","last_became_active":"2023-01-15T14:35:55.305153+0000","last_became_peered":"2023-01-15T14:35:55.305153+0000","last_unstale":"2023-01-15T14:36:09.019319+0000","last_undegraded":"2023-01-15T14:36:09.019319+0000","last_fullsized":"2023-01-15T14:36:09.019319+0000","mapping_epoch":464,"log_start":"0'0","ondisk_log_start":"0'0","created":459,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":8,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:34:48.269667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.269667+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30409415099999998,"objects_trimmed":0,"snaptrim_duration":0.0012993220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.87","version":"450'200","reported_seq":1470,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.273802+0000","last_change":"2023-01-15T14:36:00.957090+0000","last_active":"2023-01-15T14:36:05.273802+0000","last_peered":"2023-01-15T14:36:05.273802+0000","last_clean":"2023-01-15T14:36:05.273802+0000","last_became_active":"2023-01-15T14:36:00.435802+0000","last_became_peered":"2023-01-15T14:36:00.435802+0000","last_unstale":"2023-01-15T14:36:05.273802+0000","last_undegraded":"2023-01-15T14:36:05.273802+0000","last_fullsized":"2023-01-15T14:36:05.273802+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":8,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.417771381,"stat_sum":{"num_bytes":5689755,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":34583,"num_write":13,"num_write_kb":5140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1773172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.86","version":"102'52","reported_seq":385,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.005724+0000","last_change":"2023-01-15T14:36:02.456636+0000","last_active":"2023-01-15T14:36:07.005724+0000","last_peered":"2023-01-15T14:36:07.005724+0000","last_clean":"2023-01-15T14:36:07.005724+0000","last_became_active":"2023-01-15T14:35:46.362243+0000","last_became_peered":"2023-01-15T14:35:46.362243+0000","last_unstale":"2023-01-15T14:36:07.005724+0000","last_undegraded":"2023-01-15T14:36:07.005724+0000","last_fullsized":"2023-01-15T14:36:07.005724+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":8,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.10701709299999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.85","version":"470'220","reported_seq":1688,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.004050+0000","last_change":"2023-01-15T14:36:01.757724+0000","last_active":"2023-01-15T14:36:07.004050+0000","last_peered":"2023-01-15T14:36:07.004050+0000","last_clean":"2023-01-15T14:36:07.004050+0000","last_became_active":"2023-01-15T14:35:54.304302+0000","last_became_peered":"2023-01-15T14:35:54.304302+0000","last_unstale":"2023-01-15T14:36:07.004050+0000","last_undegraded":"2023-01-15T14:36:07.004050+0000","last_fullsized":"2023-01-15T14:36:07.004050+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":8,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.60137614699999997,"stat_sum":{"num_bytes":7940183,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":14,"num_read_kb":13074,"num_write":17,"num_write_kb":4850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":26383411,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.84","version":"447'372","reported_seq":1956,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.006509+0000","last_change":"2023-01-15T14:36:01.859660+0000","last_active":"2023-01-15T14:36:07.006509+0000","last_peered":"2023-01-15T14:36:07.006509+0000","last_clean":"2023-01-15T14:36:07.006509+0000","last_became_active":"2023-01-15T14:35:46.409164+0000","last_became_peered":"2023-01-15T14:35:46.409164+0000","last_unstale":"2023-01-15T14:36:07.006509+0000","last_undegraded":"2023-01-15T14:36:07.006509+0000","last_fullsized":"2023-01-15T14:36:07.006509+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":8,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0.60299173399999995,"stat_sum":{"num_bytes":4170482,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":6,"num_read_kb":4895,"num_write":3,"num_write_kb":1888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":2277448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.83","version":"444'351","reported_seq":1878,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.271720+0000","last_change":"2023-01-15T14:36:01.643032+0000","last_active":"2023-01-15T14:36:05.271720+0000","last_peered":"2023-01-15T14:36:05.271720+0000","last_clean":"2023-01-15T14:36:05.271720+0000","last_became_active":"2023-01-15T14:36:01.342026+0000","last_became_peered":"2023-01-15T14:36:01.342026+0000","last_unstale":"2023-01-15T14:36:05.271720+0000","last_undegraded":"2023-01-15T14:36:05.271720+0000","last_fullsized":"2023-01-15T14:36:05.271720+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":8,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.29977211300000001,"stat_sum":{"num_bytes":2605168,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":12727,"num_write":3,"num_write_kb":3665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":25328842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.82","version":"155'101","reported_seq":846,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.329560+0000","last_change":"2023-01-15T14:36:09.329834+0000","last_active":"2023-01-15T14:36:09.329560+0000","last_peered":"2023-01-15T14:36:09.329560+0000","last_clean":"2023-01-15T14:36:09.329560+0000","last_became_active":"2023-01-15T14:35:49.301959+0000","last_became_peered":"2023-01-15T14:35:49.301959+0000","last_unstale":"2023-01-15T14:36:09.329560+0000","last_undegraded":"2023-01-15T14:36:09.329560+0000","last_fullsized":"2023-01-15T14:36:09.329560+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":8,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.31709314100000002,"stat_sum":{"num_bytes":271017,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":817245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.81","version":"155'121","reported_seq":1044,"reported_epoch":479,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:09.212149+0000","last_change":"2023-01-15T14:36:09.212149+0000","last_active":"2023-01-15T14:36:09.212149+0000","last_peered":"2023-01-15T14:36:09.212149+0000","last_clean":"2023-01-15T14:36:09.212149+0000","last_became_active":"2023-01-15T14:35:49.400406+0000","last_became_peered":"2023-01-15T14:35:49.400406+0000","last_unstale":"2023-01-15T14:36:09.212149+0000","last_undegraded":"2023-01-15T14:36:09.212149+0000","last_fullsized":"2023-01-15T14:36:09.212149+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":8,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.21889883399999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.80","version":"223'257","reported_seq":1262,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.205914+0000","last_change":"2023-01-15T14:36:02.752664+0000","last_active":"2023-01-15T14:36:07.205914+0000","last_peered":"2023-01-15T14:36:07.205914+0000","last_clean":"2023-01-15T14:36:07.205914+0000","last_became_active":"2023-01-15T14:35:49.398227+0000","last_became_peered":"2023-01-15T14:35:49.398227+0000","last_unstale":"2023-01-15T14:36:07.205914+0000","last_undegraded":"2023-01-15T14:36:07.205914+0000","last_fullsized":"2023-01-15T14:36:07.205914+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":8,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:35:01.091016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:01.091016+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55784813600000005,"objects_trimmed":0,"snaptrim_duration":0.30271682900000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,3],"acting":[0,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.7f","version":"474'279","reported_seq":1811,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.008179+0000","last_change":"2023-01-15T14:36:05.574366+0000","last_active":"2023-01-15T14:36:07.008179+0000","last_peered":"2023-01-15T14:36:07.008179+0000","last_clean":"2023-01-15T14:36:07.008179+0000","last_became_active":"2023-01-15T14:35:50.412810+0000","last_became_peered":"2023-01-15T14:35:50.412810+0000","last_unstale":"2023-01-15T14:36:07.008179+0000","last_undegraded":"2023-01-15T14:36:07.008179+0000","last_fullsized":"2023-01-15T14:36:07.008179+0000","mapping_epoch":459,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":7,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.40683741299999998,"stat_sum":{"num_bytes":8196130,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":34267,"num_write":18,"num_write_kb":7357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":23702706,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.7e","version":"470'92","reported_seq":348,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.007978+0000","last_change":"2023-01-15T14:36:01.966068+0000","last_active":"2023-01-15T14:36:07.007978+0000","last_peered":"2023-01-15T14:36:07.007978+0000","last_clean":"2023-01-15T14:36:07.007978+0000","last_became_active":"2023-01-15T14:35:46.257756+0000","last_became_peered":"2023-01-15T14:35:46.257756+0000","last_unstale":"2023-01-15T14:36:07.007978+0000","last_undegraded":"2023-01-15T14:36:07.007978+0000","last_fullsized":"2023-01-15T14:36:07.007978+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":453,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":7,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.90919021899999997,"stat_sum":{"num_bytes":3420307,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":3026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":15336751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,3],"acting":[1,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.7d","version":"424'277","reported_seq":1741,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.104134+0000","last_change":"2023-01-15T14:36:00.947821+0000","last_active":"2023-01-15T14:36:07.104134+0000","last_peered":"2023-01-15T14:36:07.104134+0000","last_clean":"2023-01-15T14:36:07.104134+0000","last_became_active":"2023-01-15T14:35:24.190356+0000","last_became_peered":"2023-01-15T14:35:24.190356+0000","last_unstale":"2023-01-15T14:36:07.104134+0000","last_undegraded":"2023-01-15T14:36:07.104134+0000","last_fullsized":"2023-01-15T14:36:07.104134+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":7,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:57.389","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.83289080500000001,"stat_sum":{"num_bytes":2748011,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":18,"num_read_kb":20693,"num_write":10,"num_write_kb":4359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":7352992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.7c","version":"141'46","reported_seq":151,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.005871+0000","last_change":"2023-01-15T14:36:01.964953+0000","last_active":"2023-01-15T14:36:07.005871+0000","last_peered":"2023-01-15T14:36:07.005871+0000","last_clean":"2023-01-15T14:36:07.005871+0000","last_became_active":"2023-01-15T14:35:57.511542+0000","last_became_peered":"2023-01-15T14:35:57.511542+0000","last_unstale":"2023-01-15T14:36:07.005871+0000","last_undegraded":"2023-01-15T14:36:07.005871+0000","last_fullsized":"2023-01-15T14:36:07.005871+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.70867527399999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.7b","version":"221'60","reported_seq":353,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.439666+0000","last_change":"2023-01-15T14:36:09.540508+0000","last_active":"2023-01-15T14:36:09.439666+0000","last_peered":"2023-01-15T14:36:09.439666+0000","last_clean":"2023-01-15T14:36:09.439666+0000","last_became_active":"2023-01-15T14:35:21.099347+0000","last_became_peered":"2023-01-15T14:35:21.099347+0000","last_unstale":"2023-01-15T14:36:09.439666+0000","last_undegraded":"2023-01-15T14:36:09.439666+0000","last_fullsized":"2023-01-15T14:36:09.439666+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.52547443299999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.7a","version":"223'238","reported_seq":1745,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.105604+0000","last_change":"2023-01-15T14:36:04.673758+0000","last_active":"2023-01-15T14:36:07.105604+0000","last_peered":"2023-01-15T14:36:07.105604+0000","last_clean":"2023-01-15T14:36:07.105604+0000","last_became_active":"2023-01-15T14:36:00.323653+0000","last_became_peered":"2023-01-15T14:36:00.323653+0000","last_unstale":"2023-01-15T14:36:07.105604+0000","last_undegraded":"2023-01-15T14:36:07.105604+0000","last_fullsized":"2023-01-15T14:36:07.105604+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.61496046900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.79","version":"470'473","reported_seq":1931,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.227214+0000","last_change":"2023-01-15T14:36:09.327813+0000","last_active":"2023-01-15T14:36:09.227214+0000","last_peered":"2023-01-15T14:36:09.227214+0000","last_clean":"2023-01-15T14:36:09.227214+0000","last_became_active":"2023-01-15T14:35:32.375313+0000","last_became_peered":"2023-01-15T14:35:32.375313+0000","last_unstale":"2023-01-15T14:36:09.227214+0000","last_undegraded":"2023-01-15T14:36:09.227214+0000","last_fullsized":"2023-01-15T14:36:09.227214+0000","mapping_epoch":441,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":442,"parent":"0.0","parent_split_bits":7,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.107657711,"stat_sum":{"num_bytes":6815725,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":79994,"num_write":12,"num_write_kb":6932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":13431307,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.78","version":"474'378","reported_seq":2036,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.325100+0000","last_change":"2023-01-15T14:36:09.326765+0000","last_active":"2023-01-15T14:36:09.325100+0000","last_peered":"2023-01-15T14:36:09.325100+0000","last_clean":"2023-01-15T14:36:09.325100+0000","last_became_active":"2023-01-15T14:35:30.253853+0000","last_became_peered":"2023-01-15T14:35:30.253853+0000","last_unstale":"2023-01-15T14:36:09.325100+0000","last_undegraded":"2023-01-15T14:36:09.325100+0000","last_fullsized":"2023-01-15T14:36:09.325100+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":7,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.31481225899999998,"stat_sum":{"num_bytes":1870423,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":27,"num_read_kb":21425,"num_write":29,"num_write_kb":8453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":18795610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.77","version":"200'78","reported_seq":661,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.107757+0000","last_change":"2023-01-15T14:36:00.948915+0000","last_active":"2023-01-15T14:36:07.107757+0000","last_peered":"2023-01-15T14:36:07.107757+0000","last_clean":"2023-01-15T14:36:07.107757+0000","last_became_active":"2023-01-15T14:35:35.101268+0000","last_became_peered":"2023-01-15T14:35:35.101268+0000","last_unstale":"2023-01-15T14:36:07.107757+0000","last_undegraded":"2023-01-15T14:36:07.107757+0000","last_fullsized":"2023-01-15T14:36:07.107757+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":7,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.83281208100000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.76","version":"194'81","reported_seq":141,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.106992+0000","last_change":"2023-01-15T14:36:02.865140+0000","last_active":"2023-01-15T14:36:07.106992+0000","last_peered":"2023-01-15T14:36:07.106992+0000","last_clean":"2023-01-15T14:36:07.106992+0000","last_became_active":"2023-01-15T14:35:21.302673+0000","last_became_peered":"2023-01-15T14:35:21.302673+0000","last_unstale":"2023-01-15T14:36:07.106992+0000","last_undegraded":"2023-01-15T14:36:07.106992+0000","last_fullsized":"2023-01-15T14:36:07.106992+0000","mapping_epoch":430,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:34:51.296526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.296526+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.66934546500000003,"objects_trimmed":0,"snaptrim_duration":0.51536483,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.31","version":"204'150","reported_seq":1085,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.001006+0000","last_change":"2023-01-15T14:36:02.254131+0000","last_active":"2023-01-15T14:36:07.001006+0000","last_peered":"2023-01-15T14:36:07.001006+0000","last_clean":"2023-01-15T14:36:07.001006+0000","last_became_active":"2023-01-15T14:36:00.218279+0000","last_became_peered":"2023-01-15T14:36:00.218279+0000","last_unstale":"2023-01-15T14:36:07.001006+0000","last_undegraded":"2023-01-15T14:36:07.001006+0000","last_fullsized":"2023-01-15T14:36:07.001006+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:22.439","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":1.1985165090000001,"stat_sum":{"num_bytes":3558907,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10724,"num_write":3,"num_write_kb":3089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3605110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.30","version":"206'252","reported_seq":1320,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.171461+0000","last_change":"2023-01-15T14:36:01.639975+0000","last_active":"2023-01-15T14:36:05.171461+0000","last_peered":"2023-01-15T14:36:05.171461+0000","last_clean":"2023-01-15T14:36:05.171461+0000","last_became_active":"2023-01-15T14:36:00.718985+0000","last_became_peered":"2023-01-15T14:36:00.718985+0000","last_unstale":"2023-01-15T14:36:05.171461+0000","last_undegraded":"2023-01-15T14:36:05.171461+0000","last_fullsized":"2023-01-15T14:36:05.171461+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:35:03.572774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:03.572774+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:14.331","scrub_duration":0.36120358800000002,"objects_trimmed":0,"snaptrim_duration":0.51672567000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.2f","version":"155'145","reported_seq":879,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.273437+0000","last_change":"2023-01-15T14:36:01.640156+0000","last_active":"2023-01-15T14:36:05.273437+0000","last_peered":"2023-01-15T14:36:05.273437+0000","last_clean":"2023-01-15T14:36:05.273437+0000","last_became_active":"2023-01-15T14:36:00.437003+0000","last_became_peered":"2023-01-15T14:36:00.437003+0000","last_unstale":"2023-01-15T14:36:05.273437+0000","last_undegraded":"2023-01-15T14:36:05.273437+0000","last_fullsized":"2023-01-15T14:36:05.273437+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50470070300000003,"objects_trimmed":0,"snaptrim_duration":0.59848938699999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.2e","version":"139'38","reported_seq":273,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.850380+0000","last_change":"2023-01-15T14:36:09.851227+0000","last_active":"2023-01-15T14:36:09.850380+0000","last_peered":"2023-01-15T14:36:09.850380+0000","last_clean":"2023-01-15T14:36:09.850380+0000","last_became_active":"2023-01-15T14:35:57.299974+0000","last_became_peered":"2023-01-15T14:35:57.299974+0000","last_unstale":"2023-01-15T14:36:09.850380+0000","last_undegraded":"2023-01-15T14:36:09.850380+0000","last_fullsized":"2023-01-15T14:36:09.850380+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:35:30.876307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:30.876307+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.73744850699999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.2d","version":"168'156","reported_seq":1574,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.434023+0000","last_change":"2023-01-15T14:36:09.434981+0000","last_active":"2023-01-15T14:36:09.434023+0000","last_peered":"2023-01-15T14:36:09.434023+0000","last_clean":"2023-01-15T14:36:09.434023+0000","last_became_active":"2023-01-15T14:35:38.280808+0000","last_became_peered":"2023-01-15T14:35:38.280808+0000","last_unstale":"2023-01-15T14:36:09.434023+0000","last_undegraded":"2023-01-15T14:36:09.434023+0000","last_fullsized":"2023-01-15T14:36:09.434023+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.42128199199999999,"stat_sum":{"num_bytes":1516832,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":2709,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.2c","version":"141'46","reported_seq":669,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.120538+0000","last_change":"2023-01-15T14:36:09.121653+0000","last_active":"2023-01-15T14:36:09.120538+0000","last_peered":"2023-01-15T14:36:09.120538+0000","last_clean":"2023-01-15T14:36:09.120538+0000","last_became_active":"2023-01-15T14:35:41.525322+0000","last_became_peered":"2023-01-15T14:35:41.525322+0000","last_unstale":"2023-01-15T14:36:09.120538+0000","last_undegraded":"2023-01-15T14:36:09.120538+0000","last_fullsized":"2023-01-15T14:36:09.120538+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:58.535965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:58.535965+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104214843,"objects_trimmed":0,"snaptrim_duration":0.101264824,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.2b","version":"155'53","reported_seq":380,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.004948+0000","last_change":"2023-01-15T14:36:02.755767+0000","last_active":"2023-01-15T14:36:07.004948+0000","last_peered":"2023-01-15T14:36:07.004948+0000","last_clean":"2023-01-15T14:36:07.004948+0000","last_became_active":"2023-01-15T14:34:56.436012+0000","last_became_peered":"2023-01-15T14:34:56.436012+0000","last_unstale":"2023-01-15T14:36:07.004948+0000","last_undegraded":"2023-01-15T14:36:07.004948+0000","last_fullsized":"2023-01-15T14:36:07.004948+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:32.542","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.40545990900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.2a","version":"191'191","reported_seq":838,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072662+0000","last_change":"2023-01-15T14:36:01.330084+0000","last_active":"2023-01-15T14:36:05.072662+0000","last_peered":"2023-01-15T14:36:05.072662+0000","last_clean":"2023-01-15T14:36:05.072662+0000","last_became_active":"2023-01-15T14:36:00.426291+0000","last_became_peered":"2023-01-15T14:36:00.426291+0000","last_unstale":"2023-01-15T14:36:05.072662+0000","last_undegraded":"2023-01-15T14:36:05.072662+0000","last_fullsized":"2023-01-15T14:36:05.072662+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.19251327600000001,"stat_sum":{"num_bytes":1518386,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":5930,"num_write":14,"num_write_kb":6501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9447726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.29","version":"110'149","reported_seq":835,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.221542+0000","last_change":"2023-01-15T14:36:09.223061+0000","last_active":"2023-01-15T14:36:09.221542+0000","last_peered":"2023-01-15T14:36:09.221542+0000","last_clean":"2023-01-15T14:36:09.221542+0000","last_became_active":"2023-01-15T14:35:11.345959+0000","last_became_peered":"2023-01-15T14:35:11.345959+0000","last_unstale":"2023-01-15T14:36:09.221542+0000","last_undegraded":"2023-01-15T14:36:09.221542+0000","last_fullsized":"2023-01-15T14:36:09.221542+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:34.348","scrub_duration":0.40398525099999999,"objects_trimmed":0,"snaptrim_duration":0.103686579,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.28","version":"333'248","reported_seq":1703,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.472326+0000","last_change":"2023-01-15T14:36:01.639266+0000","last_active":"2023-01-15T14:36:05.472326+0000","last_peered":"2023-01-15T14:36:05.472326+0000","last_clean":"2023-01-15T14:36:05.472326+0000","last_became_active":"2023-01-15T14:36:00.628513+0000","last_became_peered":"2023-01-15T14:36:00.628513+0000","last_unstale":"2023-01-15T14:36:05.472326+0000","last_undegraded":"2023-01-15T14:36:05.472326+0000","last_fullsized":"2023-01-15T14:36:05.472326+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'248","last_scrub_stamp":"2023-01-15T14:34:20.620068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.620068+0000","objects_scrubbed":0,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40448800800000001,"objects_trimmed":0,"snaptrim_duration":0.513833608,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.27","version":"191'79","reported_seq":813,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.226114+0000","last_change":"2023-01-15T14:36:09.226668+0000","last_active":"2023-01-15T14:36:09.226114+0000","last_peered":"2023-01-15T14:36:09.226114+0000","last_clean":"2023-01-15T14:36:09.226114+0000","last_became_active":"2023-01-15T14:35:57.303183+0000","last_became_peered":"2023-01-15T14:35:57.303183+0000","last_unstale":"2023-01-15T14:36:09.226114+0000","last_undegraded":"2023-01-15T14:36:09.226114+0000","last_fullsized":"2023-01-15T14:36:09.226114+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.10552292100000001,"stat_sum":{"num_bytes":1242779,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":4228,"num_write":1,"num_write_kb":368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":865560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.26","version":"102'52","reported_seq":189,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.266492+0000","last_change":"2023-01-15T14:36:00.012852+0000","last_active":"2023-01-15T14:36:05.266492+0000","last_peered":"2023-01-15T14:36:05.266492+0000","last_clean":"2023-01-15T14:36:05.266492+0000","last_became_active":"2023-01-15T14:35:04.285753+0000","last_became_peered":"2023-01-15T14:35:04.285753+0000","last_unstale":"2023-01-15T14:36:05.266492+0000","last_undegraded":"2023-01-15T14:36:05.266492+0000","last_fullsized":"2023-01-15T14:36:05.266492+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":6,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:40.368924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:40.368924+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:29.161","scrub_duration":0.20740639699999999,"objects_trimmed":0,"snaptrim_duration":0.0014637649999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[6,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.25","version":"333'108","reported_seq":1331,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.004557+0000","last_change":"2023-01-15T14:36:01.052973+0000","last_active":"2023-01-15T14:36:07.004557+0000","last_peered":"2023-01-15T14:36:07.004557+0000","last_clean":"2023-01-15T14:36:07.004557+0000","last_became_active":"2023-01-15T14:35:38.285989+0000","last_became_peered":"2023-01-15T14:35:38.285989+0000","last_unstale":"2023-01-15T14:36:07.004557+0000","last_undegraded":"2023-01-15T14:36:07.004557+0000","last_fullsized":"2023-01-15T14:36:07.004557+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":6,"last_scrub":"333'108","last_scrub_stamp":"2023-01-15T14:35:02.975751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:02.975751+0000","objects_scrubbed":0,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:08.244","scrub_duration":0.30430094200000002,"objects_trimmed":0,"snaptrim_duration":0.83682968400000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.24","version":"467'308","reported_seq":1768,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.121277+0000","last_change":"2023-01-15T14:36:09.122044+0000","last_active":"2023-01-15T14:36:09.121277+0000","last_peered":"2023-01-15T14:36:09.121277+0000","last_clean":"2023-01-15T14:36:09.121277+0000","last_became_active":"2023-01-15T14:35:11.349162+0000","last_became_peered":"2023-01-15T14:35:11.349162+0000","last_unstale":"2023-01-15T14:36:09.121277+0000","last_undegraded":"2023-01-15T14:36:09.121277+0000","last_fullsized":"2023-01-15T14:36:09.121277+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":193,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":6,"last_scrub":"245'223","last_scrub_stamp":"2023-01-15T14:32:16.729866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.729866+0000","objects_scrubbed":1,"log_size":308,"log_dups_size":0,"ondisk_log_size":308,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:35.350","scrub_duration":0.30480155599999997,"objects_trimmed":0,"snaptrim_duration":0.101351991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":20,"num_read_kb":22662,"num_write":11,"num_write_kb":3763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.23","version":"149'233","reported_seq":1053,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.004680+0000","last_change":"2023-01-15T14:36:02.755043+0000","last_active":"2023-01-15T14:36:07.004680+0000","last_peered":"2023-01-15T14:36:07.004680+0000","last_clean":"2023-01-15T14:36:07.004680+0000","last_became_active":"2023-01-15T14:33:55.379557+0000","last_became_peered":"2023-01-15T14:33:55.379557+0000","last_unstale":"2023-01-15T14:36:07.004680+0000","last_undegraded":"2023-01-15T14:36:07.004680+0000","last_fullsized":"2023-01-15T14:36:07.004680+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":6,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:34:55.674244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:55.674244+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:18.674","scrub_duration":0.50374311999999999,"objects_trimmed":0,"snaptrim_duration":0.40490607299999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.22","version":"323'161","reported_seq":1471,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072529+0000","last_change":"2023-01-15T14:36:01.058897+0000","last_active":"2023-01-15T14:36:05.072529+0000","last_peered":"2023-01-15T14:36:05.072529+0000","last_clean":"2023-01-15T14:36:05.072529+0000","last_became_active":"2023-01-15T14:36:00.521188+0000","last_became_peered":"2023-01-15T14:36:00.521188+0000","last_unstale":"2023-01-15T14:36:05.072529+0000","last_undegraded":"2023-01-15T14:36:05.072529+0000","last_fullsized":"2023-01-15T14:36:05.072529+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"181'107","last_scrub_stamp":"2023-01-15T14:31:30.784357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:30.784357+0000","objects_scrubbed":2,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.29661209799999999,"objects_trimmed":0,"snaptrim_duration":0.43385756599999997,"stat_sum":{"num_bytes":2605134,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":6733,"num_write":8,"num_write_kb":5014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.21","version":"321'138","reported_seq":1122,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072441+0000","last_change":"2023-01-15T14:36:00.833897+0000","last_active":"2023-01-15T14:36:05.072441+0000","last_peered":"2023-01-15T14:36:05.072441+0000","last_clean":"2023-01-15T14:36:05.072441+0000","last_became_active":"2023-01-15T14:36:00.121542+0000","last_became_peered":"2023-01-15T14:36:00.121542+0000","last_unstale":"2023-01-15T14:36:05.072441+0000","last_undegraded":"2023-01-15T14:36:05.072441+0000","last_fullsized":"2023-01-15T14:36:05.072441+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"321'138","last_scrub_stamp":"2023-01-15T14:34:49.495975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.495975+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.71176880200000003,"stat_sum":{"num_bytes":1033663,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":6059,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.20","version":"324'273","reported_seq":1607,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.206566+0000","last_change":"2023-01-15T14:36:04.685566+0000","last_active":"2023-01-15T14:36:07.206566+0000","last_peered":"2023-01-15T14:36:07.206566+0000","last_clean":"2023-01-15T14:36:07.206566+0000","last_became_active":"2023-01-15T14:36:00.727135+0000","last_became_peered":"2023-01-15T14:36:00.727135+0000","last_unstale":"2023-01-15T14:36:07.206566+0000","last_undegraded":"2023-01-15T14:36:07.206566+0000","last_fullsized":"2023-01-15T14:36:07.206566+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"324'273","last_scrub_stamp":"2023-01-15T14:34:49.100502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:49.100502+0000","objects_scrubbed":0,"log_size":273,"log_dups_size":0,"ondisk_log_size":273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204192292,"objects_trimmed":0,"snaptrim_duration":0.62912420599999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.1f","version":"476'328","reported_seq":1842,"reported_epoch":477,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:36:07.108673+0000","last_change":"2023-01-15T14:35:58.624792+0000","last_active":"2023-01-15T14:36:07.108673+0000","last_peered":"2023-01-15T14:36:07.108673+0000","last_clean":"2023-01-15T14:35:55.715457+0000","last_became_active":"2023-01-15T14:35:58.619489+0000","last_became_peered":"2023-01-15T14:35:58.619489+0000","last_unstale":"2023-01-15T14:36:07.108673+0000","last_undegraded":"2023-01-15T14:35:58.020484+0000","last_fullsized":"2023-01-15T14:35:58.019453+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":5,"last_scrub":"268'238","last_scrub_stamp":"2023-01-15T14:32:37.665761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:37.665761+0000","objects_scrubbed":2,"log_size":328,"log_dups_size":0,"ondisk_log_size":328,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.437764553,"objects_trimmed":0,"snaptrim_duration":0.70970781699999996,"stat_sum":{"num_bytes":11790542,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":53,"num_read_kb":68772,"num_write":28,"num_write_kb":8770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":29785074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,6],"avail_no_missing":["0","4","6"],"object_location_counts":[{"shards":"0,4,6","objects":6},{"shards":"0,6","objects":-2}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"133","length":"1"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.1e","version":"215'39","reported_seq":1297,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.209725+0000","last_change":"2023-01-15T14:36:01.863785+0000","last_active":"2023-01-15T14:36:07.209725+0000","last_peered":"2023-01-15T14:36:07.209725+0000","last_clean":"2023-01-15T14:36:07.209725+0000","last_became_active":"2023-01-15T14:35:41.422705+0000","last_became_peered":"2023-01-15T14:35:41.422705+0000","last_unstale":"2023-01-15T14:36:07.209725+0000","last_undegraded":"2023-01-15T14:36:07.209725+0000","last_fullsized":"2023-01-15T14:36:07.209725+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.63053790200000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.1d","version":"221'200","reported_seq":1414,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.371568+0000","last_change":"2023-01-15T14:36:00.836890+0000","last_active":"2023-01-15T14:36:05.371568+0000","last_peered":"2023-01-15T14:36:05.371568+0000","last_clean":"2023-01-15T14:36:05.371568+0000","last_became_active":"2023-01-15T14:36:00.222343+0000","last_became_peered":"2023-01-15T14:36:00.222343+0000","last_unstale":"2023-01-15T14:36:05.371568+0000","last_undegraded":"2023-01-15T14:36:05.371568+0000","last_fullsized":"2023-01-15T14:36:05.371568+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":5,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:51.945865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.945865+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.61345776799999996,"stat_sum":{"num_bytes":2869927,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":23811,"num_write":12,"num_write_kb":5315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3464322,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.1c","version":"141'46","reported_seq":36,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.172283+0000","last_change":"2023-01-15T14:36:00.019682+0000","last_active":"2023-01-15T14:36:05.172283+0000","last_peered":"2023-01-15T14:36:05.172283+0000","last_clean":"2023-01-15T14:36:05.172283+0000","last_became_active":"2023-01-15T14:35:57.500868+0000","last_became_peered":"2023-01-15T14:35:57.500868+0000","last_unstale":"2023-01-15T14:36:05.172283+0000","last_undegraded":"2023-01-15T14:36:05.172283+0000","last_fullsized":"2023-01-15T14:36:05.172283+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":5,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:22.440826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:22.440826+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30343327599999997,"objects_trimmed":0,"snaptrim_duration":0.001455617,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.1b","version":"473'167","reported_seq":447,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.106766+0000","last_change":"2023-01-15T14:36:04.477719+0000","last_active":"2023-01-15T14:36:07.106766+0000","last_peered":"2023-01-15T14:36:07.106766+0000","last_clean":"2023-01-15T14:36:07.106766+0000","last_became_active":"2023-01-15T14:33:52.108619+0000","last_became_peered":"2023-01-15T14:33:52.108619+0000","last_unstale":"2023-01-15T14:36:07.106766+0000","last_undegraded":"2023-01-15T14:36:07.106766+0000","last_fullsized":"2023-01-15T14:36:07.106766+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":5,"last_scrub":"370'101","last_scrub_stamp":"2023-01-15T14:34:24.137270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:24.137270+0000","objects_scrubbed":3,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.106791863,"objects_trimmed":0,"snaptrim_duration":0.31640131900000001,"stat_sum":{"num_bytes":8660087,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":13301,"num_write":12,"num_write_kb":5069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.1a","version":"468'392","reported_seq":1892,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.020431+0000","last_change":"2023-01-15T14:36:09.021014+0000","last_active":"2023-01-15T14:36:09.020431+0000","last_peered":"2023-01-15T14:36:09.020431+0000","last_clean":"2023-01-15T14:36:09.020431+0000","last_became_active":"2023-01-15T14:35:44.351248+0000","last_became_peered":"2023-01-15T14:35:44.351248+0000","last_unstale":"2023-01-15T14:36:09.020431+0000","last_undegraded":"2023-01-15T14:36:09.020431+0000","last_fullsized":"2023-01-15T14:36:09.020431+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":5,"last_scrub":"375'356","last_scrub_stamp":"2023-01-15T14:34:29.460843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:29.460843+0000","objects_scrubbed":4,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30741503599999997,"objects_trimmed":0,"snaptrim_duration":0.0015082400000000001,"stat_sum":{"num_bytes":6907842,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":16,"num_write_kb":5138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":9225015,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.19","version":"471'302","reported_seq":1680,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.001270+0000","last_change":"2023-01-15T14:36:02.989439+0000","last_active":"2023-01-15T14:36:07.001270+0000","last_peered":"2023-01-15T14:36:07.001270+0000","last_clean":"2023-01-15T14:36:07.001270+0000","last_became_active":"2023-01-15T14:34:43.263173+0000","last_became_peered":"2023-01-15T14:34:43.263173+0000","last_unstale":"2023-01-15T14:36:07.001270+0000","last_undegraded":"2023-01-15T14:36:07.001270+0000","last_fullsized":"2023-01-15T14:36:07.001270+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:33:43.952506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.952506+0000","objects_scrubbed":2,"log_size":302,"log_dups_size":0,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30730086699999998,"objects_trimmed":0,"snaptrim_duration":0.71763895700000002,"stat_sum":{"num_bytes":5393852,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":14,"num_read_kb":12321,"num_write":14,"num_write_kb":3780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4941423,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.18","version":"444'360","reported_seq":1918,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.304902+0000","last_change":"2023-01-15T14:36:02.751852+0000","last_active":"2023-01-15T14:36:07.304902+0000","last_peered":"2023-01-15T14:36:07.304902+0000","last_clean":"2023-01-15T14:36:07.304902+0000","last_became_active":"2023-01-15T14:35:11.544123+0000","last_became_peered":"2023-01-15T14:35:11.544123+0000","last_unstale":"2023-01-15T14:36:07.304902+0000","last_undegraded":"2023-01-15T14:36:07.304902+0000","last_fullsized":"2023-01-15T14:36:07.304902+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":5,"last_scrub":"384'341","last_scrub_stamp":"2023-01-15T14:34:35.477211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:35.477211+0000","objects_scrubbed":5,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.54215560600000001,"objects_trimmed":0,"snaptrim_duration":0.30216801700000001,"stat_sum":{"num_bytes":2974905,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.17","version":"278'93","reported_seq":384,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.007788+0000","last_change":"2023-01-15T14:36:04.478880+0000","last_active":"2023-01-15T14:36:07.007788+0000","last_peered":"2023-01-15T14:36:07.007788+0000","last_clean":"2023-01-15T14:36:07.007788+0000","last_became_active":"2023-01-15T14:35:57.494559+0000","last_became_peered":"2023-01-15T14:35:57.494559+0000","last_unstale":"2023-01-15T14:36:07.007788+0000","last_undegraded":"2023-01-15T14:36:07.007788+0000","last_fullsized":"2023-01-15T14:36:07.007788+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":5,"last_scrub":"254'86","last_scrub_stamp":"2023-01-15T14:32:24.763408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.763408+0000","objects_scrubbed":2,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:30.569","scrub_duration":0.20627088599999999,"objects_trimmed":0,"snaptrim_duration":0.31702119600000001,"stat_sum":{"num_bytes":1512663,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.16","version":"444'157","reported_seq":991,"reported_epoch":477,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:36:07.007027+0000","last_change":"2023-01-15T14:35:58.523179+0000","last_active":"2023-01-15T14:36:07.007027+0000","last_peered":"2023-01-15T14:36:07.007027+0000","last_clean":"2023-01-15T14:35:34.287688+0000","last_became_active":"2023-01-15T14:35:58.422019+0000","last_became_peered":"2023-01-15T14:35:58.422019+0000","last_unstale":"2023-01-15T14:36:07.007027+0000","last_undegraded":"2023-01-15T14:35:58.118118+0000","last_fullsized":"2023-01-15T14:35:58.117497+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"320'127","last_scrub_stamp":"2023-01-15T14:33:44.924959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:44.924959+0000","objects_scrubbed":3,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30679945600000003,"objects_trimmed":0,"snaptrim_duration":0.70950577100000001,"stat_sum":{"num_bytes":5025425,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":6898,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":9731517,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[1,7],"avail_no_missing":["1","3","7"],"object_location_counts":[{"shards":"1,3,7","objects":2}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[{"start":"133","length":"1"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.15","version":"110'55","reported_seq":333,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.746580+0000","last_change":"2023-01-15T14:36:09.847621+0000","last_active":"2023-01-15T14:36:09.746580+0000","last_peered":"2023-01-15T14:36:09.746580+0000","last_clean":"2023-01-15T14:36:09.746580+0000","last_became_active":"2023-01-15T14:33:57.577637+0000","last_became_peered":"2023-01-15T14:33:57.577637+0000","last_unstale":"2023-01-15T14:36:09.746580+0000","last_undegraded":"2023-01-15T14:36:09.746580+0000","last_fullsized":"2023-01-15T14:36:09.746580+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":5,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:24.052709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:24.052709+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.81694163399999997,"stat_sum":{"num_bytes":796227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":5169,"num_write":3,"num_write_kb":1556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":531513,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.14","version":"445'181","reported_seq":1563,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.223578+0000","last_change":"2023-01-15T14:36:09.324755+0000","last_active":"2023-01-15T14:36:09.223578+0000","last_peered":"2023-01-15T14:36:09.223578+0000","last_clean":"2023-01-15T14:36:09.223578+0000","last_became_active":"2023-01-15T14:35:14.050115+0000","last_became_peered":"2023-01-15T14:35:14.050115+0000","last_unstale":"2023-01-15T14:36:09.223578+0000","last_undegraded":"2023-01-15T14:36:09.223578+0000","last_fullsized":"2023-01-15T14:36:09.223578+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":5,"last_scrub":"351'159","last_scrub_stamp":"2023-01-15T14:34:18.607751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:18.607751+0000","objects_scrubbed":2,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40666956999999998,"objects_trimmed":0,"snaptrim_duration":0.20481128200000001,"stat_sum":{"num_bytes":2453855,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2453855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":116,"reported_epoch":478,"state":"active+clean","last_fresh":"2023-01-15T14:36:08.012751+0000","last_change":"2023-01-15T14:35:41.423046+0000","last_active":"2023-01-15T14:36:08.012751+0000","last_peered":"2023-01-15T14:36:08.012751+0000","last_clean":"2023-01-15T14:36:08.012751+0000","last_became_active":"2023-01-15T14:35:41.421509+0000","last_became_peered":"2023-01-15T14:35:41.421509+0000","last_unstale":"2023-01-15T14:36:08.012751+0000","last_undegraded":"2023-01-15T14:36:08.012751+0000","last_fullsized":"2023-01-15T14:36:08.012751+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.198056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.198056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20367893200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"102'52","reported_seq":373,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.005933+0000","last_change":"2023-01-15T14:36:02.760413+0000","last_active":"2023-01-15T14:36:07.005933+0000","last_peered":"2023-01-15T14:36:07.005933+0000","last_clean":"2023-01-15T14:36:07.005933+0000","last_became_active":"2023-01-15T14:35:44.557253+0000","last_became_peered":"2023-01-15T14:35:44.557253+0000","last_unstale":"2023-01-15T14:36:07.005933+0000","last_undegraded":"2023-01-15T14:36:07.005933+0000","last_fullsized":"2023-01-15T14:36:07.005933+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:32:39.564141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.564141+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.458500148,"objects_trimmed":0,"snaptrim_duration":0.41053635700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":106,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.008093+0000","last_change":"2023-01-15T14:35:04.384333+0000","last_active":"2023-01-15T14:36:07.008093+0000","last_peered":"2023-01-15T14:36:07.008093+0000","last_clean":"2023-01-15T14:36:07.008093+0000","last_became_active":"2023-01-15T14:35:04.383307+0000","last_became_peered":"2023-01-15T14:35:04.383307+0000","last_unstale":"2023-01-15T14:36:07.008093+0000","last_undegraded":"2023-01-15T14:36:07.008093+0000","last_fullsized":"2023-01-15T14:36:07.008093+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:51.551094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.551094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.23765246300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"478'220","reported_seq":1556,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.439231+0000","last_change":"2023-01-15T14:36:09.439569+0000","last_active":"2023-01-15T14:36:09.439231+0000","last_peered":"2023-01-15T14:36:09.439231+0000","last_clean":"2023-01-15T14:36:09.439231+0000","last_became_active":"2023-01-15T14:35:44.762050+0000","last_became_peered":"2023-01-15T14:35:44.762050+0000","last_unstale":"2023-01-15T14:36:09.439231+0000","last_undegraded":"2023-01-15T14:36:09.439231+0000","last_fullsized":"2023-01-15T14:36:09.439231+0000","mapping_epoch":366,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:32:38.598504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:38.598504+0000","objects_scrubbed":1,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50449108200000004,"objects_trimmed":0,"snaptrim_duration":0.424882547,"stat_sum":{"num_bytes":4531639,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":34584,"num_write":17,"num_write_kb":6246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":1773171,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":478,"state":"active+clean","last_fresh":"2023-01-15T14:36:08.013039+0000","last_change":"2023-01-15T14:33:52.126613+0000","last_active":"2023-01-15T14:36:08.013039+0000","last_peered":"2023-01-15T14:36:08.013039+0000","last_clean":"2023-01-15T14:36:08.013039+0000","last_became_active":"2023-01-15T14:33:52.126161+0000","last_became_peered":"2023-01-15T14:33:52.126161+0000","last_unstale":"2023-01-15T14:36:08.013039+0000","last_undegraded":"2023-01-15T14:36:08.013039+0000","last_fullsized":"2023-01-15T14:36:08.013039+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:34.578670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:34.578670+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42425565700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"477'394","reported_seq":2065,"reported_epoch":478,"state":"active+clean","last_fresh":"2023-01-15T14:36:08.816826+0000","last_change":"2023-01-15T14:36:08.817512+0000","last_active":"2023-01-15T14:36:08.816826+0000","last_peered":"2023-01-15T14:36:08.816826+0000","last_clean":"2023-01-15T14:36:08.816826+0000","last_became_active":"2023-01-15T14:36:07.308882+0000","last_became_peered":"2023-01-15T14:36:07.308882+0000","last_unstale":"2023-01-15T14:36:08.816826+0000","last_undegraded":"2023-01-15T14:36:08.816826+0000","last_fullsized":"2023-01-15T14:36:08.816826+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"367'311","last_scrub_stamp":"2023-01-15T14:34:20.074587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:20.074587+0000","objects_scrubbed":4,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50790537000000002,"objects_trimmed":0,"snaptrim_duration":0.50150171499999996,"stat_sum":{"num_bytes":2119929,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":4894,"num_write":6,"num_write_kb":1887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":15177602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.006614+0000","last_change":"2023-01-15T14:34:43.371445+0000","last_active":"2023-01-15T14:36:07.006614+0000","last_peered":"2023-01-15T14:36:07.006614+0000","last_clean":"2023-01-15T14:36:07.006614+0000","last_became_active":"2023-01-15T14:34:43.253700+0000","last_became_peered":"2023-01-15T14:34:43.253700+0000","last_unstale":"2023-01-15T14:36:07.006614+0000","last_undegraded":"2023-01-15T14:36:07.006614+0000","last_fullsized":"2023-01-15T14:36:07.006614+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:21.422201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:21.422201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"443'207","reported_seq":1567,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.171023+0000","last_change":"2023-01-15T14:36:00.956125+0000","last_active":"2023-01-15T14:36:05.171023+0000","last_peered":"2023-01-15T14:36:05.171023+0000","last_clean":"2023-01-15T14:36:05.171023+0000","last_became_active":"2023-01-15T14:36:00.218840+0000","last_became_peered":"2023-01-15T14:36:00.218840+0000","last_unstale":"2023-01-15T14:36:05.171023+0000","last_undegraded":"2023-01-15T14:36:05.171023+0000","last_fullsized":"2023-01-15T14:36:05.171023+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"259'127","last_scrub_stamp":"2023-01-15T14:32:35.357895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:35.357895+0000","objects_scrubbed":1,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.75048249099999997,"objects_trimmed":0,"snaptrim_duration":0.73554225699999998,"stat_sum":{"num_bytes":5687913,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":13074,"num_write":11,"num_write_kb":3965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":13282154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.3","version":"444'351","reported_seq":1860,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.020327+0000","last_change":"2023-01-15T14:36:09.020327+0000","last_active":"2023-01-15T14:36:09.020327+0000","last_peered":"2023-01-15T14:36:09.020327+0000","last_clean":"2023-01-15T14:36:09.020327+0000","last_became_active":"2023-01-15T14:35:44.252095+0000","last_became_peered":"2023-01-15T14:35:44.252095+0000","last_unstale":"2023-01-15T14:36:09.020327+0000","last_undegraded":"2023-01-15T14:36:09.020327+0000","last_fullsized":"2023-01-15T14:36:09.020327+0000","mapping_epoch":425,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"274'309","last_scrub_stamp":"2023-01-15T14:33:18.059469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.059469+0000","objects_scrubbed":1,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30457408600000002,"objects_trimmed":0,"snaptrim_duration":0.0012264649999999999,"stat_sum":{"num_bytes":2605167,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":12727,"num_write":3,"num_write_kb":3665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":14908172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072396+0000","last_change":"2023-01-15T14:35:57.310164+0000","last_active":"2023-01-15T14:36:05.072396+0000","last_peered":"2023-01-15T14:36:05.072396+0000","last_clean":"2023-01-15T14:36:05.072396+0000","last_became_active":"2023-01-15T14:35:57.209780+0000","last_became_peered":"2023-01-15T14:35:57.209780+0000","last_unstale":"2023-01-15T14:36:05.072396+0000","last_undegraded":"2023-01-15T14:36:05.072396+0000","last_fullsized":"2023-01-15T14:36:05.072396+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:32:57.480601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:32:57.480601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"223'257","reported_seq":1297,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.101438+0000","last_change":"2023-01-15T14:36:00.946920+0000","last_active":"2023-01-15T14:36:07.101438+0000","last_peered":"2023-01-15T14:36:07.101438+0000","last_clean":"2023-01-15T14:36:07.101438+0000","last_became_active":"2023-01-15T14:35:44.254173+0000","last_became_peered":"2023-01-15T14:35:44.254173+0000","last_unstale":"2023-01-15T14:36:07.101438+0000","last_undegraded":"2023-01-15T14:36:07.101438+0000","last_fullsized":"2023-01-15T14:36:07.101438+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"223'257","last_scrub_stamp":"2023-01-15T14:35:01.091016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:01.091016+0000","objects_scrubbed":0,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.55784813600000005,"objects_trimmed":0,"snaptrim_duration":0.83251707500000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":84,"reported_epoch":476,"state":"active+clean","last_fresh":"2023-01-15T14:36:06.085136+0000","last_change":"2023-01-15T14:35:35.298341+0000","last_active":"2023-01-15T14:36:06.085136+0000","last_peered":"2023-01-15T14:36:06.085136+0000","last_clean":"2023-01-15T14:36:06.085136+0000","last_became_active":"2023-01-15T14:35:35.296662+0000","last_became_peered":"2023-01-15T14:35:35.296662+0000","last_unstale":"2023-01-15T14:36:06.085136+0000","last_undegraded":"2023-01-15T14:36:06.085136+0000","last_fullsized":"2023-01-15T14:36:06.085136+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:28.990852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:28.990852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"155'121","reported_seq":1029,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.004696+0000","last_change":"2023-01-15T14:36:02.661191+0000","last_active":"2023-01-15T14:36:07.004696+0000","last_peered":"2023-01-15T14:36:07.004696+0000","last_clean":"2023-01-15T14:36:07.004696+0000","last_became_active":"2023-01-15T14:35:44.457965+0000","last_became_peered":"2023-01-15T14:35:44.457965+0000","last_unstale":"2023-01-15T14:36:07.004696+0000","last_undegraded":"2023-01-15T14:36:07.004696+0000","last_fullsized":"2023-01-15T14:36:07.004696+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:18.226500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:18.226500+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30398668899999998,"objects_trimmed":0,"snaptrim_duration":0.27596795499999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":545,"reported_epoch":478,"state":"active+clean","last_fresh":"2023-01-15T14:36:08.113432+0000","last_change":"2023-01-15T14:34:43.269231+0000","last_active":"2023-01-15T14:36:08.113432+0000","last_peered":"2023-01-15T14:36:08.113432+0000","last_clean":"2023-01-15T14:36:08.113432+0000","last_became_active":"2023-01-15T14:34:43.268688+0000","last_became_peered":"2023-01-15T14:34:43.268688+0000","last_unstale":"2023-01-15T14:36:08.113432+0000","last_undegraded":"2023-01-15T14:36:08.113432+0000","last_fullsized":"2023-01-15T14:36:08.113432+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:33:06.912420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:33:06.912420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50447083299999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"155'101","reported_seq":813,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072106+0000","last_change":"2023-01-15T14:36:01.161055+0000","last_active":"2023-01-15T14:36:05.072106+0000","last_peered":"2023-01-15T14:36:05.072106+0000","last_clean":"2023-01-15T14:36:05.072106+0000","last_became_active":"2023-01-15T14:36:00.324138+0000","last_became_peered":"2023-01-15T14:36:00.324138+0000","last_unstale":"2023-01-15T14:36:05.072106+0000","last_undegraded":"2023-01-15T14:36:05.072106+0000","last_fullsized":"2023-01-15T14:36:05.072106+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:32:31.593524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:31.593524+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.101707193,"stat_sum":{"num_bytes":271017,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":817244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.172253+0000","last_change":"2023-01-15T14:35:57.399122+0000","last_active":"2023-01-15T14:36:05.172253+0000","last_peered":"2023-01-15T14:36:05.172253+0000","last_clean":"2023-01-15T14:36:05.172253+0000","last_became_active":"2023-01-15T14:35:57.398571+0000","last_became_peered":"2023-01-15T14:35:57.398571+0000","last_unstale":"2023-01-15T14:36:05.172253+0000","last_undegraded":"2023-01-15T14:36:05.172253+0000","last_fullsized":"2023-01-15T14:36:05.172253+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:31:27.768470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:11.770822+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.768470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40338101300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":678,"reported_epoch":478,"state":"active+clean","last_fresh":"2023-01-15T14:36:08.009076+0000","last_change":"2023-01-15T14:35:26.786467+0000","last_active":"2023-01-15T14:36:08.009076+0000","last_peered":"2023-01-15T14:36:08.009076+0000","last_clean":"2023-01-15T14:36:08.009076+0000","last_became_active":"2023-01-15T14:35:26.685181+0000","last_became_peered":"2023-01-15T14:35:26.685181+0000","last_unstale":"2023-01-15T14:36:08.009076+0000","last_undegraded":"2023-01-15T14:36:08.009076+0000","last_fullsized":"2023-01-15T14:36:08.009076+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'117","last_scrub_stamp":"2023-01-15T14:32:54.588021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:05.841513+0000","last_clean_scrub_stamp":"2023-01-15T14:32:54.588021+0000","objects_scrubbed":2,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.204778408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"179'204","reported_seq":1630,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.000625+0000","last_change":"2023-01-15T14:36:01.052400+0000","last_active":"2023-01-15T14:36:07.000625+0000","last_peered":"2023-01-15T14:36:07.000625+0000","last_clean":"2023-01-15T14:36:07.000625+0000","last_became_active":"2023-01-15T14:35:50.304954+0000","last_became_peered":"2023-01-15T14:35:50.304954+0000","last_unstale":"2023-01-15T14:36:07.000625+0000","last_undegraded":"2023-01-15T14:36:07.000625+0000","last_fullsized":"2023-01-15T14:36:07.000625+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:35:56.732493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:56.732493+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:58.010","scrub_duration":0.29859745199999999,"objects_trimmed":0,"snaptrim_duration":0.83837941900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.9","version":"110'149","reported_seq":1453,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.121782+0000","last_change":"2023-01-15T14:36:09.221360+0000","last_active":"2023-01-15T14:36:09.121782+0000","last_peered":"2023-01-15T14:36:09.121782+0000","last_clean":"2023-01-15T14:36:09.121782+0000","last_became_active":"2023-01-15T14:35:50.507792+0000","last_became_peered":"2023-01-15T14:35:50.507792+0000","last_unstale":"2023-01-15T14:36:09.121782+0000","last_undegraded":"2023-01-15T14:36:09.121782+0000","last_fullsized":"2023-01-15T14:36:09.121782+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:12.335477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:12.335477+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.103530501,"objects_trimmed":0,"snaptrim_duration":0.102287767,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.a","version":"252'279","reported_seq":1588,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.021560+0000","last_change":"2023-01-15T14:36:09.022162+0000","last_active":"2023-01-15T14:36:09.021560+0000","last_peered":"2023-01-15T14:36:09.021560+0000","last_clean":"2023-01-15T14:36:09.021560+0000","last_became_active":"2023-01-15T14:35:50.405112+0000","last_became_peered":"2023-01-15T14:35:50.405112+0000","last_unstale":"2023-01-15T14:36:09.021560+0000","last_undegraded":"2023-01-15T14:36:09.021560+0000","last_fullsized":"2023-01-15T14:36:09.021560+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"252'279","last_scrub_stamp":"2023-01-15T14:33:34.672786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:34.672786+0000","objects_scrubbed":0,"log_size":279,"log_dups_size":0,"ondisk_log_size":279,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42281940099999998,"objects_trimmed":0,"snaptrim_duration":0.0022387819999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.b","version":"155'53","reported_seq":1149,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.004608+0000","last_change":"2023-01-15T14:36:02.754349+0000","last_active":"2023-01-15T14:36:07.004608+0000","last_peered":"2023-01-15T14:36:07.004608+0000","last_clean":"2023-01-15T14:36:07.004608+0000","last_became_active":"2023-01-15T14:35:50.506517+0000","last_became_peered":"2023-01-15T14:35:50.506517+0000","last_unstale":"2023-01-15T14:36:07.004608+0000","last_undegraded":"2023-01-15T14:36:07.004608+0000","last_fullsized":"2023-01-15T14:36:07.004608+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:39.868223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:39.868223+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20657772799999999,"objects_trimmed":0,"snaptrim_duration":0.404547408,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.c","version":"451'156","reported_seq":1619,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072034+0000","last_change":"2023-01-15T14:36:00.137816+0000","last_active":"2023-01-15T14:36:05.072034+0000","last_peered":"2023-01-15T14:36:05.072034+0000","last_clean":"2023-01-15T14:36:05.072034+0000","last_became_active":"2023-01-15T14:36:00.134582+0000","last_became_peered":"2023-01-15T14:36:00.134582+0000","last_unstale":"2023-01-15T14:36:05.072034+0000","last_undegraded":"2023-01-15T14:36:05.072034+0000","last_fullsized":"2023-01-15T14:36:05.072034+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.0023793410000000001,"stat_sum":{"num_bytes":3057636,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":33623,"num_write":8,"num_write_kb":1853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":14538648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.d","version":"296'232","reported_seq":1663,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.645612+0000","last_change":"2023-01-15T14:36:09.746343+0000","last_active":"2023-01-15T14:36:09.645612+0000","last_peered":"2023-01-15T14:36:09.645612+0000","last_clean":"2023-01-15T14:36:09.645612+0000","last_became_active":"2023-01-15T14:36:00.325646+0000","last_became_peered":"2023-01-15T14:36:00.325646+0000","last_unstale":"2023-01-15T14:36:09.645612+0000","last_undegraded":"2023-01-15T14:36:09.645612+0000","last_fullsized":"2023-01-15T14:36:09.645612+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":232,"log_dups_size":0,"ondisk_log_size":232,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:45.258","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.71602631100000003,"stat_sum":{"num_bytes":2306688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":7820,"num_write":5,"num_write_kb":1640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.e","version":"139'38","reported_seq":365,"reported_epoch":479,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:09.214390+0000","last_change":"2023-01-15T14:36:09.214390+0000","last_active":"2023-01-15T14:36:09.214390+0000","last_peered":"2023-01-15T14:36:09.214390+0000","last_clean":"2023-01-15T14:36:09.214390+0000","last_became_active":"2023-01-15T14:35:50.309574+0000","last_became_peered":"2023-01-15T14:35:50.309574+0000","last_unstale":"2023-01-15T14:36:09.214390+0000","last_undegraded":"2023-01-15T14:36:09.214390+0000","last_fullsized":"2023-01-15T14:36:09.214390+0000","mapping_epoch":392,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:57.164","scrub_duration":0.50427325000000001,"objects_trimmed":0,"snaptrim_duration":0.50173284100000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.f","version":"155'145","reported_seq":1554,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.540780+0000","last_change":"2023-01-15T14:36:09.541485+0000","last_active":"2023-01-15T14:36:09.540780+0000","last_peered":"2023-01-15T14:36:09.540780+0000","last_clean":"2023-01-15T14:36:09.540780+0000","last_became_active":"2023-01-15T14:35:50.511018+0000","last_became_peered":"2023-01-15T14:35:50.511018+0000","last_unstale":"2023-01-15T14:36:09.540780+0000","last_undegraded":"2023-01-15T14:36:09.540780+0000","last_fullsized":"2023-01-15T14:36:09.540780+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.303791015,"objects_trimmed":0,"snaptrim_duration":0.52626688899999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.10","version":"454'443","reported_seq":1838,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.104182+0000","last_change":"2023-01-15T14:36:04.786421+0000","last_active":"2023-01-15T14:36:07.104182+0000","last_peered":"2023-01-15T14:36:07.104182+0000","last_clean":"2023-01-15T14:36:07.104182+0000","last_became_active":"2023-01-15T14:36:00.816582+0000","last_became_peered":"2023-01-15T14:36:00.816582+0000","last_unstale":"2023-01-15T14:36:07.104182+0000","last_undegraded":"2023-01-15T14:36:07.104182+0000","last_fullsized":"2023-01-15T14:36:07.104182+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":5,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.73049102899999996,"stat_sum":{"num_bytes":4003030,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":31691,"num_write":27,"num_write_kb":9855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":10329425,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.11","version":"479'420","reported_seq":2075,"reported_epoch":479,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:36:09.851470+0000","last_change":"2023-01-15T14:36:09.851470+0000","last_active":"2023-01-15T14:36:09.851470+0000","last_peered":"2023-01-15T14:36:09.851470+0000","last_clean":"2023-01-15T14:36:09.851470+0000","last_became_active":"2023-01-15T14:35:50.408210+0000","last_became_peered":"2023-01-15T14:35:50.408210+0000","last_unstale":"2023-01-15T14:36:09.851470+0000","last_undegraded":"2023-01-15T14:36:09.851470+0000","last_fullsized":"2023-01-15T14:36:09.851470+0000","mapping_epoch":422,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":460,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'156","last_scrub_stamp":"2023-01-15T14:31:40.900356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:40.900356+0000","objects_scrubbed":5,"log_size":420,"log_dups_size":0,"ondisk_log_size":420,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:43.295","scrub_duration":0.30779894200000002,"objects_trimmed":0,"snaptrim_duration":0.42091096500000003,"stat_sum":{"num_bytes":10776960,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":24146,"num_write":28,"num_write_kb":15320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":22433541,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.12","version":"472'243","reported_seq":1264,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.105910+0000","last_change":"2023-01-15T14:36:03.469454+0000","last_active":"2023-01-15T14:36:07.105910+0000","last_peered":"2023-01-15T14:36:07.105910+0000","last_clean":"2023-01-15T14:36:07.105910+0000","last_became_active":"2023-01-15T14:34:02.134565+0000","last_became_peered":"2023-01-15T14:34:02.134565+0000","last_unstale":"2023-01-15T14:36:07.105910+0000","last_undegraded":"2023-01-15T14:36:07.105910+0000","last_fullsized":"2023-01-15T14:36:07.105910+0000","mapping_epoch":352,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":5,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":243,"log_dups_size":0,"ondisk_log_size":243,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.41686969200000001,"stat_sum":{"num_bytes":9689420,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":6954,"num_write":25,"num_write_kb":12679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":11160046,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.13","version":"472'433","reported_seq":1832,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.541778+0000","last_change":"2023-01-15T14:36:09.542774+0000","last_active":"2023-01-15T14:36:09.541778+0000","last_peered":"2023-01-15T14:36:09.541778+0000","last_clean":"2023-01-15T14:36:09.541778+0000","last_became_active":"2023-01-15T14:35:41.221503+0000","last_became_peered":"2023-01-15T14:35:41.221503+0000","last_unstale":"2023-01-15T14:36:09.541778+0000","last_undegraded":"2023-01-15T14:36:09.541778+0000","last_fullsized":"2023-01-15T14:36:09.541778+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":111,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":5,"last_scrub":"374'376","last_scrub_stamp":"2023-01-15T14:34:32.489483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.489483+0000","objects_scrubbed":4,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.381937625,"objects_trimmed":0,"snaptrim_duration":0.52725945500000004,"stat_sum":{"num_bytes":13617083,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":18,"num_read_kb":29953,"num_write":8,"num_write_kb":4428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":13617083,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.32","version":"464'212","reported_seq":1246,"reported_epoch":477,"state":"active+undersized+remapped","last_fresh":"2023-01-15T14:36:08.014166+0000","last_change":"2023-01-15T14:36:08.014485+0000","last_active":"2023-01-15T14:36:08.014166+0000","last_peered":"2023-01-15T14:36:08.014166+0000","last_clean":"2023-01-15T14:35:55.442927+0000","last_became_active":"2023-01-15T14:35:58.513685+0000","last_became_peered":"2023-01-15T14:35:58.513685+0000","last_unstale":"2023-01-15T14:36:08.014166+0000","last_undegraded":"2023-01-15T14:36:08.014166+0000","last_fullsized":"2023-01-15T14:35:58.209741+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":6,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:29.581","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.43735380400000001,"stat_sum":{"num_bytes":7445043,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":24666,"num_write":11,"num_write_kb":3706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":12772928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1],"avail_no_missing":["0","1","3","5"],"object_location_counts":[{"shards":"0,1","objects":4}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"133","length":"1"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.33","version":"204'275","reported_seq":1419,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.101718+0000","last_change":"2023-01-15T14:36:01.050784+0000","last_active":"2023-01-15T14:36:07.101718+0000","last_peered":"2023-01-15T14:36:07.101718+0000","last_clean":"2023-01-15T14:36:07.101718+0000","last_became_active":"2023-01-15T14:35:41.223122+0000","last_became_peered":"2023-01-15T14:35:41.223122+0000","last_unstale":"2023-01-15T14:36:07.101718+0000","last_undegraded":"2023-01-15T14:36:07.101718+0000","last_fullsized":"2023-01-15T14:36:07.101718+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":6,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":0.89083579700000004,"stat_sum":{"num_bytes":3485477,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":11189,"num_write":7,"num_write_kb":3285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.34","version":"169'138","reported_seq":1506,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.022257+0000","last_change":"2023-01-15T14:36:09.121097+0000","last_active":"2023-01-15T14:36:09.022257+0000","last_peered":"2023-01-15T14:36:09.022257+0000","last_clean":"2023-01-15T14:36:09.022257+0000","last_became_active":"2023-01-15T14:35:19.392381+0000","last_became_peered":"2023-01-15T14:35:19.392381+0000","last_unstale":"2023-01-15T14:36:09.022257+0000","last_undegraded":"2023-01-15T14:36:09.022257+0000","last_fullsized":"2023-01-15T14:36:09.022257+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":429,"parent":"0.0","parent_split_bits":6,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:35:46.590173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:46.590173+0000","objects_scrubbed":0,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:52.600","scrub_duration":0.204901835,"objects_trimmed":0,"snaptrim_duration":0.100992969,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.35","version":"110'55","reported_seq":50,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.371784+0000","last_change":"2023-01-15T14:36:00.023266+0000","last_active":"2023-01-15T14:36:05.371784+0000","last_peered":"2023-01-15T14:36:05.371784+0000","last_clean":"2023-01-15T14:36:05.371784+0000","last_became_active":"2023-01-15T14:35:57.301471+0000","last_became_peered":"2023-01-15T14:35:57.301471+0000","last_unstale":"2023-01-15T14:36:05.371784+0000","last_undegraded":"2023-01-15T14:36:05.371784+0000","last_fullsized":"2023-01-15T14:36:05.371784+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:35:58.219589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:35:58.219589+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:37:26.346","scrub_duration":0.44852532699999997,"objects_trimmed":0,"snaptrim_duration":0.000944519,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.36","version":"194'81","reported_seq":36,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072502+0000","last_change":"2023-01-15T14:36:00.021848+0000","last_active":"2023-01-15T14:36:05.072502+0000","last_peered":"2023-01-15T14:36:05.072502+0000","last_clean":"2023-01-15T14:36:05.072502+0000","last_became_active":"2023-01-15T14:35:57.305669+0000","last_became_peered":"2023-01-15T14:35:57.305669+0000","last_unstale":"2023-01-15T14:36:05.072502+0000","last_undegraded":"2023-01-15T14:36:05.072502+0000","last_fullsized":"2023-01-15T14:36:05.072502+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:34:51.296526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:51.296526+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.66934546500000003,"objects_trimmed":0,"snaptrim_duration":0.00087794100000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.37","version":"200'78","reported_seq":763,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.544536+0000","last_change":"2023-01-15T14:36:09.645393+0000","last_active":"2023-01-15T14:36:09.544536+0000","last_peered":"2023-01-15T14:36:09.544536+0000","last_clean":"2023-01-15T14:36:09.544536+0000","last_became_active":"2023-01-15T14:35:57.603486+0000","last_became_peered":"2023-01-15T14:35:57.603486+0000","last_unstale":"2023-01-15T14:36:09.544536+0000","last_undegraded":"2023-01-15T14:36:09.544536+0000","last_fullsized":"2023-01-15T14:36:09.544536+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"200'78","last_scrub_stamp":"2023-01-15T14:31:43.835701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:43.835701+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20364526299999999,"objects_trimmed":0,"snaptrim_duration":0.62874452700000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.38","version":"426'347","reported_seq":1797,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.272105+0000","last_change":"2023-01-15T14:36:01.636641+0000","last_active":"2023-01-15T14:36:05.272105+0000","last_peered":"2023-01-15T14:36:05.272105+0000","last_clean":"2023-01-15T14:36:05.272105+0000","last_became_active":"2023-01-15T14:36:00.626409+0000","last_became_peered":"2023-01-15T14:36:00.626409+0000","last_unstale":"2023-01-15T14:36:05.272105+0000","last_undegraded":"2023-01-15T14:36:05.272105+0000","last_fullsized":"2023-01-15T14:36:05.272105+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"125'195","last_scrub_stamp":"2023-01-15T14:30:13.697250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:13.697250+0000","objects_scrubbed":2,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.104041683,"objects_trimmed":0,"snaptrim_duration":0.51100263199999996,"stat_sum":{"num_bytes":4230716,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":21424,"num_write":20,"num_write_kb":6727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":16442530,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.39","version":"423'435","reported_seq":1911,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.330331+0000","last_change":"2023-01-15T14:36:09.330725+0000","last_active":"2023-01-15T14:36:09.330331+0000","last_peered":"2023-01-15T14:36:09.330331+0000","last_clean":"2023-01-15T14:36:09.330331+0000","last_became_active":"2023-01-15T14:35:38.280557+0000","last_became_peered":"2023-01-15T14:35:38.280557+0000","last_unstale":"2023-01-15T14:36:09.330331+0000","last_undegraded":"2023-01-15T14:36:09.330331+0000","last_fullsized":"2023-01-15T14:36:09.330331+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":6,"last_scrub":"305'324","last_scrub_stamp":"2023-01-15T14:33:18.934086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:18.934086+0000","objects_scrubbed":6,"log_size":435,"log_dups_size":0,"ondisk_log_size":435,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21145576099999999,"objects_trimmed":0,"snaptrim_duration":0.31785237500000002,"stat_sum":{"num_bytes":17098873,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":34,"num_read_kb":57397,"num_write":12,"num_write_kb":6931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":13431306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.3a","version":"223'238","reported_seq":1710,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072280+0000","last_change":"2023-01-15T14:36:01.326176+0000","last_active":"2023-01-15T14:36:05.072280+0000","last_peered":"2023-01-15T14:36:05.072280+0000","last_clean":"2023-01-15T14:36:05.072280+0000","last_became_active":"2023-01-15T14:36:00.527253+0000","last_became_peered":"2023-01-15T14:36:00.527253+0000","last_unstale":"2023-01-15T14:36:05.072280+0000","last_undegraded":"2023-01-15T14:36:05.072280+0000","last_fullsized":"2023-01-15T14:36:05.072280+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":6,"last_scrub":"223'238","last_scrub_stamp":"2023-01-15T14:32:08.329672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:08.329672+0000","objects_scrubbed":0,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50870150400000003,"objects_trimmed":0,"snaptrim_duration":0.16648763,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.3b","version":"221'60","reported_seq":356,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.435346+0000","last_change":"2023-01-15T14:36:09.436315+0000","last_active":"2023-01-15T14:36:09.435346+0000","last_peered":"2023-01-15T14:36:09.435346+0000","last_clean":"2023-01-15T14:36:09.435346+0000","last_became_active":"2023-01-15T14:35:35.100811+0000","last_became_peered":"2023-01-15T14:35:35.100811+0000","last_unstale":"2023-01-15T14:36:09.435346+0000","last_undegraded":"2023-01-15T14:36:09.435346+0000","last_fullsized":"2023-01-15T14:36:09.435346+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":6,"last_scrub":"221'60","last_scrub_stamp":"2023-01-15T14:32:44.563332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:44.563332+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40476896600000001,"objects_trimmed":0,"snaptrim_duration":0.42256941100000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.3c","version":"141'46","reported_seq":44,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072266+0000","last_change":"2023-01-15T14:36:00.020154+0000","last_active":"2023-01-15T14:36:05.072266+0000","last_peered":"2023-01-15T14:36:05.072266+0000","last_clean":"2023-01-15T14:36:05.072266+0000","last_became_active":"2023-01-15T14:35:57.196630+0000","last_became_peered":"2023-01-15T14:35:57.196630+0000","last_unstale":"2023-01-15T14:36:05.072266+0000","last_undegraded":"2023-01-15T14:36:05.072266+0000","last_fullsized":"2023-01-15T14:36:05.072266+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":6,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:33:22.287615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:22.287615+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.36730637599999999,"objects_trimmed":0,"snaptrim_duration":0.0015976619999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.3d","version":"470'314","reported_seq":1839,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.107162+0000","last_change":"2023-01-15T14:36:02.959684+0000","last_active":"2023-01-15T14:36:07.107162+0000","last_peered":"2023-01-15T14:36:07.107162+0000","last_clean":"2023-01-15T14:36:07.107162+0000","last_became_active":"2023-01-15T14:35:44.656577+0000","last_became_peered":"2023-01-15T14:35:44.656577+0000","last_unstale":"2023-01-15T14:36:07.107162+0000","last_undegraded":"2023-01-15T14:36:07.107162+0000","last_fullsized":"2023-01-15T14:36:07.107162+0000","mapping_epoch":453,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":6,"last_scrub":"232'203","last_scrub_stamp":"2023-01-15T14:32:10.144171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:10.144171+0000","objects_scrubbed":4,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.270895677,"objects_trimmed":0,"snaptrim_duration":0.60875921600000005,"stat_sum":{"num_bytes":7672659,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":28971,"num_write":15,"num_write_kb":5220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":14950019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.3e","version":"448'86","reported_seq":311,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.108874+0000","last_change":"2023-01-15T14:36:01.758834+0000","last_active":"2023-01-15T14:36:07.108874+0000","last_peered":"2023-01-15T14:36:07.108874+0000","last_clean":"2023-01-15T14:36:07.108874+0000","last_became_active":"2023-01-15T14:35:44.355920+0000","last_became_peered":"2023-01-15T14:35:44.355920+0000","last_unstale":"2023-01-15T14:36:07.108874+0000","last_undegraded":"2023-01-15T14:36:07.108874+0000","last_fullsized":"2023-01-15T14:36:07.108874+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":6,"last_scrub":"242'47","last_scrub_stamp":"2023-01-15T14:32:16.150108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:16.150108+0000","objects_scrubbed":1,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.205054665,"objects_trimmed":0,"snaptrim_duration":0.62644149400000004,"stat_sum":{"num_bytes":3420307,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":3025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":8496136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.3f","version":"477'283","reported_seq":1799,"reported_epoch":479,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:09.322377+0000","last_change":"2023-01-15T14:36:09.322527+0000","last_active":"2023-01-15T14:36:09.322377+0000","last_peered":"2023-01-15T14:36:09.322377+0000","last_clean":"2023-01-15T14:36:09.322377+0000","last_became_active":"2023-01-15T14:35:44.661557+0000","last_became_peered":"2023-01-15T14:35:44.661557+0000","last_unstale":"2023-01-15T14:36:09.322377+0000","last_undegraded":"2023-01-15T14:36:09.322377+0000","last_fullsized":"2023-01-15T14:36:09.322377+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":454,"parent":"0.0","parent_split_bits":6,"last_scrub":"297'220","last_scrub_stamp":"2023-01-15T14:33:13.802825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.802825+0000","objects_scrubbed":1,"log_size":283,"log_dups_size":0,"ondisk_log_size":283,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40496514300000003,"objects_trimmed":0,"snaptrim_duration":0.30620882500000002,"stat_sum":{"num_bytes":10674709,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":14973,"num_write":18,"num_write_kb":7357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":13214874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.40","version":"478'367","reported_seq":1850,"reported_epoch":479,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:09.432480+0000","last_change":"2023-01-15T14:36:09.432651+0000","last_active":"2023-01-15T14:36:09.432480+0000","last_peered":"2023-01-15T14:36:09.432480+0000","last_clean":"2023-01-15T14:36:09.432480+0000","last_became_active":"2023-01-15T14:36:08.315016+0000","last_became_peered":"2023-01-15T14:36:08.315016+0000","last_unstale":"2023-01-15T14:36:09.432480+0000","last_undegraded":"2023-01-15T14:36:09.432480+0000","last_fullsized":"2023-01-15T14:36:09.432480+0000","mapping_epoch":477,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":478,"parent":"0.0","parent_split_bits":7,"last_scrub":"180'236","last_scrub_stamp":"2023-01-15T14:31:12.647418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:12.647418+0000","objects_scrubbed":2,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40692495299999998,"objects_trimmed":0,"snaptrim_duration":0.41901162600000003,"stat_sum":{"num_bytes":13693193,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":35544,"num_write":44,"num_write_kb":15071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":57913440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"1"},{"start":"141","length":"1"},{"start":"143","length":"1"}]},{"pgid":"3.41","version":"155'121","reported_seq":1121,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.071801+0000","last_change":"2023-01-15T14:36:00.748970+0000","last_active":"2023-01-15T14:36:05.071801+0000","last_peered":"2023-01-15T14:36:05.071801+0000","last_clean":"2023-01-15T14:36:05.071801+0000","last_became_active":"2023-01-15T14:36:00.224181+0000","last_became_peered":"2023-01-15T14:36:00.224181+0000","last_unstale":"2023-01-15T14:36:05.071801+0000","last_undegraded":"2023-01-15T14:36:05.071801+0000","last_fullsized":"2023-01-15T14:36:05.071801+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":224,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'121","last_scrub_stamp":"2023-01-15T14:32:41.704155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:41.704155+0000","objects_scrubbed":0,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51457867999999996,"objects_trimmed":0,"snaptrim_duration":0.423709108,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.42","version":"155'101","reported_seq":1070,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.206575+0000","last_change":"2023-01-15T14:36:02.560152+0000","last_active":"2023-01-15T14:36:07.206575+0000","last_peered":"2023-01-15T14:36:07.206575+0000","last_clean":"2023-01-15T14:36:07.206575+0000","last_became_active":"2023-01-15T14:35:41.516587+0000","last_became_peered":"2023-01-15T14:35:41.516587+0000","last_unstale":"2023-01-15T14:36:07.206575+0000","last_undegraded":"2023-01-15T14:36:07.206575+0000","last_fullsized":"2023-01-15T14:36:07.206575+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'101","last_scrub_stamp":"2023-01-15T14:34:57.046809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:57.046809+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40498292400000002,"objects_trimmed":0,"snaptrim_duration":0.209411134,"stat_sum":{"num_bytes":542034,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1634489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.43","version":"476'360","reported_seq":1898,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.207362+0000","last_change":"2023-01-15T14:36:06.003042+0000","last_active":"2023-01-15T14:36:07.207362+0000","last_peered":"2023-01-15T14:36:07.207362+0000","last_clean":"2023-01-15T14:36:07.207362+0000","last_became_active":"2023-01-15T14:36:05.365399+0000","last_became_peered":"2023-01-15T14:36:05.365399+0000","last_unstale":"2023-01-15T14:36:07.207362+0000","last_undegraded":"2023-01-15T14:36:07.207362+0000","last_fullsized":"2023-01-15T14:36:07.207362+0000","mapping_epoch":474,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":475,"parent":"0.0","parent_split_bits":7,"last_scrub":"265'308","last_scrub_stamp":"2023-01-15T14:32:33.436220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:33.436220+0000","objects_scrubbed":2,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50665982700000001,"objects_trimmed":1,"snaptrim_duration":0.63590074100000005,"stat_sum":{"num_bytes":5849034,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":16587,"num_write":16,"num_write_kb":6312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":18689846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.44","version":"245'268","reported_seq":1497,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.020321+0000","last_change":"2023-01-15T14:36:09.120434+0000","last_active":"2023-01-15T14:36:09.020321+0000","last_peered":"2023-01-15T14:36:09.020321+0000","last_clean":"2023-01-15T14:36:09.020321+0000","last_became_active":"2023-01-15T14:35:41.325427+0000","last_became_peered":"2023-01-15T14:35:41.325427+0000","last_unstale":"2023-01-15T14:36:09.020321+0000","last_undegraded":"2023-01-15T14:36:09.020321+0000","last_fullsized":"2023-01-15T14:36:09.020321+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'268","last_scrub_stamp":"2023-01-15T14:33:13.045057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:13.045057+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30435891300000001,"objects_trimmed":0,"snaptrim_duration":0.101029732,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.45","version":"253'125","reported_seq":1250,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.207806+0000","last_change":"2023-01-15T14:36:02.858664+0000","last_active":"2023-01-15T14:36:07.207806+0000","last_peered":"2023-01-15T14:36:07.207806+0000","last_clean":"2023-01-15T14:36:07.207806+0000","last_became_active":"2023-01-15T14:35:38.500236+0000","last_became_peered":"2023-01-15T14:35:38.500236+0000","last_unstale":"2023-01-15T14:36:07.207806+0000","last_undegraded":"2023-01-15T14:36:07.207806+0000","last_fullsized":"2023-01-15T14:36:07.207806+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"253'125","last_scrub_stamp":"2023-01-15T14:34:50.381073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:50.381073+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.52191215300000005,"objects_trimmed":0,"snaptrim_duration":0.50752270200000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.46","version":"102'52","reported_seq":241,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.108475+0000","last_change":"2023-01-15T14:36:01.861877+0000","last_active":"2023-01-15T14:36:07.108475+0000","last_peered":"2023-01-15T14:36:07.108475+0000","last_clean":"2023-01-15T14:36:07.108475+0000","last_became_active":"2023-01-15T14:35:04.373495+0000","last_became_peered":"2023-01-15T14:35:04.373495+0000","last_unstale":"2023-01-15T14:36:07.108475+0000","last_undegraded":"2023-01-15T14:36:07.108475+0000","last_fullsized":"2023-01-15T14:36:07.108475+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:33:32.865819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:32.865819+0000","objects_scrubbed":2,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:21.428","scrub_duration":0.32316732300000001,"objects_trimmed":0,"snaptrim_duration":0.804679484,"stat_sum":{"num_bytes":342443,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2142,"num_write":3,"num_write_kb":1187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.47","version":"245'112","reported_seq":1459,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.325092+0000","last_change":"2023-01-15T14:36:09.326050+0000","last_active":"2023-01-15T14:36:09.325092+0000","last_peered":"2023-01-15T14:36:09.325092+0000","last_clean":"2023-01-15T14:36:09.325092+0000","last_became_active":"2023-01-15T14:36:00.220945+0000","last_became_peered":"2023-01-15T14:36:00.220945+0000","last_unstale":"2023-01-15T14:36:09.325092+0000","last_undegraded":"2023-01-15T14:36:09.325092+0000","last_fullsized":"2023-01-15T14:36:09.325092+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'112","last_scrub_stamp":"2023-01-15T14:34:21.450320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:21.450320+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30497759099999999,"objects_trimmed":0,"snaptrim_duration":0.104932382,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.48","version":"179'204","reported_seq":1496,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.271826+0000","last_change":"2023-01-15T14:36:01.641018+0000","last_active":"2023-01-15T14:36:05.271826+0000","last_peered":"2023-01-15T14:36:05.271826+0000","last_clean":"2023-01-15T14:36:05.271826+0000","last_became_active":"2023-01-15T14:36:00.627683+0000","last_became_peered":"2023-01-15T14:36:00.627683+0000","last_unstale":"2023-01-15T14:36:05.271826+0000","last_undegraded":"2023-01-15T14:36:05.271826+0000","last_fullsized":"2023-01-15T14:36:05.271826+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"179'204","last_scrub_stamp":"2023-01-15T14:33:43.443273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:43.443273+0000","objects_scrubbed":0,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61494061600000005,"objects_trimmed":0,"snaptrim_duration":0.516159602,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.49","version":"110'149","reported_seq":1265,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.326954+0000","last_change":"2023-01-15T14:36:09.327958+0000","last_active":"2023-01-15T14:36:09.326954+0000","last_peered":"2023-01-15T14:36:09.326954+0000","last_clean":"2023-01-15T14:36:09.326954+0000","last_became_active":"2023-01-15T14:35:38.285855+0000","last_became_peered":"2023-01-15T14:35:38.285855+0000","last_unstale":"2023-01-15T14:36:09.326954+0000","last_undegraded":"2023-01-15T14:36:09.326954+0000","last_fullsized":"2023-01-15T14:36:09.326954+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:16.680437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.680437+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.404092059,"objects_trimmed":0,"snaptrim_duration":0.31567940700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.4a","version":"477'444","reported_seq":1947,"reported_epoch":479,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:09.324852+0000","last_change":"2023-01-15T14:36:09.325030+0000","last_active":"2023-01-15T14:36:09.324852+0000","last_peered":"2023-01-15T14:36:09.324852+0000","last_clean":"2023-01-15T14:36:09.324852+0000","last_became_active":"2023-01-15T14:35:22.264683+0000","last_became_peered":"2023-01-15T14:35:22.264683+0000","last_unstale":"2023-01-15T14:36:09.324852+0000","last_undegraded":"2023-01-15T14:36:09.324852+0000","last_fullsized":"2023-01-15T14:36:09.324852+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":7,"last_scrub":"0'0","last_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:28:22.711572+0000","objects_scrubbed":0,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:34.189","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.31056876,"stat_sum":{"num_bytes":11615609,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":91,"num_read_kb":101596,"num_write":88,"num_write_kb":26766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":39901450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.4b","version":"155'53","reported_seq":153,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.102525+0000","last_change":"2023-01-15T14:36:01.862747+0000","last_active":"2023-01-15T14:36:07.102525+0000","last_peered":"2023-01-15T14:36:07.102525+0000","last_clean":"2023-01-15T14:36:07.102525+0000","last_became_active":"2023-01-15T14:35:57.397539+0000","last_became_peered":"2023-01-15T14:35:57.397539+0000","last_unstale":"2023-01-15T14:36:07.102525+0000","last_undegraded":"2023-01-15T14:36:07.102525+0000","last_fullsized":"2023-01-15T14:36:07.102525+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":254,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:34:44.116779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:44.116779+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50673716899999999,"objects_trimmed":0,"snaptrim_duration":0.80626661600000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.4c","version":"309'71","reported_seq":429,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.436912+0000","last_change":"2023-01-15T14:36:09.437998+0000","last_active":"2023-01-15T14:36:09.436912+0000","last_peered":"2023-01-15T14:36:09.436912+0000","last_clean":"2023-01-15T14:36:09.436912+0000","last_became_active":"2023-01-15T14:35:04.379289+0000","last_became_peered":"2023-01-15T14:35:04.379289+0000","last_unstale":"2023-01-15T14:36:09.436912+0000","last_undegraded":"2023-01-15T14:36:09.436912+0000","last_fullsized":"2023-01-15T14:36:09.436912+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"260'66","last_scrub_stamp":"2023-01-15T14:32:43.603893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:43.603893+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:47.266","scrub_duration":0.81085226099999996,"objects_trimmed":0,"snaptrim_duration":0.42392472199999998,"stat_sum":{"num_bytes":2221859,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":10494,"num_write":3,"num_write_kb":687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.4d","version":"475'347","reported_seq":1742,"reported_epoch":476,"state":"active+clean","last_fresh":"2023-01-15T14:36:06.231791+0000","last_change":"2023-01-15T14:36:06.091441+0000","last_active":"2023-01-15T14:36:06.231791+0000","last_peered":"2023-01-15T14:36:06.231791+0000","last_clean":"2023-01-15T14:36:06.231791+0000","last_became_active":"2023-01-15T14:36:04.361610+0000","last_became_peered":"2023-01-15T14:36:04.361610+0000","last_unstale":"2023-01-15T14:36:06.231791+0000","last_undegraded":"2023-01-15T14:36:06.231791+0000","last_fullsized":"2023-01-15T14:36:06.231791+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":7,"last_scrub":"198'164","last_scrub_stamp":"2023-01-15T14:31:27.831109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:27.831109+0000","objects_scrubbed":4,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30956831899999998,"objects_trimmed":0,"snaptrim_duration":0.0019865550000000001,"stat_sum":{"num_bytes":12395770,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":27786,"num_write":27,"num_write_kb":9640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":49427949,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.4e","version":"139'38","reported_seq":414,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.006180+0000","last_change":"2023-01-15T14:36:02.753662+0000","last_active":"2023-01-15T14:36:07.006180+0000","last_peered":"2023-01-15T14:36:07.006180+0000","last_clean":"2023-01-15T14:36:07.006180+0000","last_became_active":"2023-01-15T14:35:35.391600+0000","last_became_peered":"2023-01-15T14:35:35.391600+0000","last_unstale":"2023-01-15T14:36:07.006180+0000","last_undegraded":"2023-01-15T14:36:07.006180+0000","last_fullsized":"2023-01-15T14:36:07.006180+0000","mapping_epoch":444,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":445,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:53.915692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:53.915692+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.42873577200000001,"objects_trimmed":0,"snaptrim_duration":0.403737662,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":3,"num_read_kb":2882,"num_write":1,"num_write_kb":1012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":371855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.4f","version":"155'145","reported_seq":1140,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.273478+0000","last_change":"2023-01-15T14:36:00.954324+0000","last_active":"2023-01-15T14:36:05.273478+0000","last_peered":"2023-01-15T14:36:05.273478+0000","last_clean":"2023-01-15T14:36:05.273478+0000","last_became_active":"2023-01-15T14:36:00.415211+0000","last_became_peered":"2023-01-15T14:36:00.415211+0000","last_unstale":"2023-01-15T14:36:05.273478+0000","last_undegraded":"2023-01-15T14:36:05.273478+0000","last_fullsized":"2023-01-15T14:36:05.273478+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:33:15.699378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.699378+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.51591872999999999,"stat_sum":{"num_bytes":1556753,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8397,"num_write":10,"num_write_kb":3208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3724851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.50","version":"479'504","reported_seq":2010,"reported_epoch":479,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2023-01-15T14:36:09.854976+0000","last_change":"2023-01-15T14:36:07.515272+0000","last_active":"2023-01-15T14:36:09.854976+0000","last_peered":"2023-01-15T14:36:09.854976+0000","last_clean":"2023-01-15T14:35:55.005690+0000","last_became_active":"2023-01-15T14:35:58.619369+0000","last_became_peered":"2023-01-15T14:35:58.619369+0000","last_unstale":"2023-01-15T14:36:09.854976+0000","last_undegraded":"2023-01-15T14:36:07.515043+0000","last_fullsized":"2023-01-15T14:35:58.003805+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":353,"parent":"0.0","parent_split_bits":7,"last_scrub":"130'234","last_scrub_stamp":"2023-01-15T14:30:32.051893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:32.051893+0000","objects_scrubbed":7,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:51.263","scrub_duration":0.83880702699999998,"objects_trimmed":0,"snaptrim_duration":0.81950787300000005,"stat_sum":{"num_bytes":14761781,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":87,"num_read_kb":92166,"num_write":69,"num_write_kb":27486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":85,"num_bytes_recovered":51990336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,0],"avail_no_missing":["7","0","3"],"object_location_counts":[{"shards":"0,3,7","objects":1},{"shards":"0,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"133","length":"1"},{"start":"141","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.51","version":"455'353","reported_seq":1902,"reported_epoch":477,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:36:07.001594+0000","last_change":"2023-01-15T14:35:58.424514+0000","last_active":"2023-01-15T14:36:07.001594+0000","last_peered":"2023-01-15T14:36:07.001594+0000","last_clean":"2023-01-15T14:35:55.101917+0000","last_became_active":"2023-01-15T14:35:58.423581+0000","last_became_peered":"2023-01-15T14:35:58.423581+0000","last_unstale":"2023-01-15T14:36:07.001594+0000","last_undegraded":"2023-01-15T14:35:58.320157+0000","last_fullsized":"2023-01-15T14:35:58.318929+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":455,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'298","last_scrub_stamp":"2023-01-15T14:33:33.733586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:33.733586+0000","objects_scrubbed":2,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.21404013099999999,"objects_trimmed":0,"snaptrim_duration":0.51849322600000003,"stat_sum":{"num_bytes":6386825,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":12,"num_write_kb":2393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":8487799,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,0],"avail_no_missing":["1","0","3"],"object_location_counts":[{"shards":"0,1,3","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"133","length":"1"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.52","version":"259'164","reported_seq":1003,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.847843+0000","last_change":"2023-01-15T14:36:09.848906+0000","last_active":"2023-01-15T14:36:09.847843+0000","last_peered":"2023-01-15T14:36:09.847843+0000","last_clean":"2023-01-15T14:36:09.847843+0000","last_became_active":"2023-01-15T14:36:00.523514+0000","last_became_peered":"2023-01-15T14:36:00.523514+0000","last_unstale":"2023-01-15T14:36:09.847843+0000","last_undegraded":"2023-01-15T14:36:09.847843+0000","last_fullsized":"2023-01-15T14:36:09.847843+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"252'162","last_scrub_stamp":"2023-01-15T14:32:23.169174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:23.169174+0000","objects_scrubbed":2,"log_size":164,"log_dups_size":0,"ondisk_log_size":164,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60757779499999998,"objects_trimmed":0,"snaptrim_duration":0.83501119000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.53","version":"479'558","reported_seq":2051,"reported_epoch":479,"state":"active+clean+snaptrim","last_fresh":"2023-01-15T14:36:09.330794+0000","last_change":"2023-01-15T14:36:09.330794+0000","last_active":"2023-01-15T14:36:09.330794+0000","last_peered":"2023-01-15T14:36:09.330794+0000","last_clean":"2023-01-15T14:36:09.330794+0000","last_became_active":"2023-01-15T14:35:41.426783+0000","last_became_peered":"2023-01-15T14:35:41.426783+0000","last_unstale":"2023-01-15T14:36:09.330794+0000","last_undegraded":"2023-01-15T14:36:09.330794+0000","last_fullsized":"2023-01-15T14:36:09.330794+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"324'368","last_scrub_stamp":"2023-01-15T14:33:37.396669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:37.396669+0000","objects_scrubbed":5,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.80317512999999996,"objects_trimmed":0,"snaptrim_duration":0.52994414199999995,"stat_sum":{"num_bytes":20030460,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":76,"num_read_kb":75188,"num_write":61,"num_write_kb":30769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":72326676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.54","version":"245'148","reported_seq":1442,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.106068+0000","last_change":"2023-01-15T14:36:02.557818+0000","last_active":"2023-01-15T14:36:07.106068+0000","last_peered":"2023-01-15T14:36:07.106068+0000","last_clean":"2023-01-15T14:36:07.106068+0000","last_became_active":"2023-01-15T14:33:55.775462+0000","last_became_peered":"2023-01-15T14:33:55.775462+0000","last_unstale":"2023-01-15T14:36:07.106068+0000","last_undegraded":"2023-01-15T14:36:07.106068+0000","last_fullsized":"2023-01-15T14:36:07.106068+0000","mapping_epoch":345,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":346,"parent":"0.0","parent_split_bits":7,"last_scrub":"245'148","last_scrub_stamp":"2023-01-15T14:34:47.392699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.392699+0000","objects_scrubbed":0,"log_size":148,"log_dups_size":0,"ondisk_log_size":148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled to 2023-01-15T14:36:14.435","scrub_duration":0.50581767899999996,"objects_trimmed":0,"snaptrim_duration":0.20861616999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.55","version":"110'55","reported_seq":1145,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.106383+0000","last_change":"2023-01-15T14:36:02.756622+0000","last_active":"2023-01-15T14:36:07.106383+0000","last_peered":"2023-01-15T14:36:07.106383+0000","last_clean":"2023-01-15T14:36:07.106383+0000","last_became_active":"2023-01-15T14:35:11.347080+0000","last_became_peered":"2023-01-15T14:35:11.347080+0000","last_unstale":"2023-01-15T14:36:07.106383+0000","last_undegraded":"2023-01-15T14:36:07.106383+0000","last_fullsized":"2023-01-15T14:36:07.106383+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":310,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:34:59.815570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:59.815570+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.497624124,"objects_trimmed":0,"snaptrim_duration":0.30619014700000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.56","version":"320'127","reported_seq":935,"reported_epoch":479,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:09.116085+0000","last_change":"2023-01-15T14:36:09.116085+0000","last_active":"2023-01-15T14:36:09.116085+0000","last_peered":"2023-01-15T14:36:09.116085+0000","last_clean":"2023-01-15T14:36:09.116085+0000","last_became_active":"2023-01-15T14:34:00.238318+0000","last_became_peered":"2023-01-15T14:34:00.238318+0000","last_unstale":"2023-01-15T14:36:09.116085+0000","last_undegraded":"2023-01-15T14:36:09.116085+0000","last_fullsized":"2023-01-15T14:36:09.116085+0000","mapping_epoch":350,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":351,"parent":"0.0","parent_split_bits":7,"last_scrub":"194'81","last_scrub_stamp":"2023-01-15T14:31:39.635020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:39.635020+0000","objects_scrubbed":2,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.61474270600000003,"objects_trimmed":0,"snaptrim_duration":0.60383242299999995,"stat_sum":{"num_bytes":3689236,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10280,"num_write":7,"num_write_kb":2576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.57","version":"468'133","reported_seq":528,"reported_epoch":477,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:36:07.007265+0000","last_change":"2023-01-15T14:35:58.519777+0000","last_active":"2023-01-15T14:36:07.007265+0000","last_peered":"2023-01-15T14:36:07.007265+0000","last_clean":"2023-01-15T14:35:55.412961+0000","last_became_active":"2023-01-15T14:35:58.516572+0000","last_became_peered":"2023-01-15T14:35:58.516572+0000","last_unstale":"2023-01-15T14:36:07.007265+0000","last_undegraded":"2023-01-15T14:35:58.010930+0000","last_fullsized":"2023-01-15T14:35:58.009368+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'100","last_scrub_stamp":"2023-01-15T14:34:23.447852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.447852+0000","objects_scrubbed":2,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.52101476099999999,"objects_trimmed":0,"snaptrim_duration":0.30863261199999997,"stat_sum":{"num_bytes":10390684,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":25130,"num_write":16,"num_write_kb":3559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":17584212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,6],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":3},{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"133","length":"1"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.58","version":"323'320","reported_seq":1904,"reported_epoch":479,"state":"active+clean+snaptrim_wait","last_fresh":"2023-01-15T14:36:09.321042+0000","last_change":"2023-01-15T14:36:09.321207+0000","last_active":"2023-01-15T14:36:09.321042+0000","last_peered":"2023-01-15T14:36:09.321042+0000","last_clean":"2023-01-15T14:36:09.321042+0000","last_became_active":"2023-01-15T14:36:00.718508+0000","last_became_peered":"2023-01-15T14:36:00.718508+0000","last_unstale":"2023-01-15T14:36:09.321042+0000","last_undegraded":"2023-01-15T14:36:09.321042+0000","last_fullsized":"2023-01-15T14:36:09.321042+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"323'320","last_scrub_stamp":"2023-01-15T14:34:47.541655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:47.541655+0000","objects_scrubbed":0,"log_size":320,"log_dups_size":0,"ondisk_log_size":320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40398073200000001,"objects_trimmed":0,"snaptrim_duration":0.30826853799999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.59","version":"294'256","reported_seq":1616,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.223542+0000","last_change":"2023-01-15T14:36:09.223542+0000","last_active":"2023-01-15T14:36:09.223542+0000","last_peered":"2023-01-15T14:36:09.223542+0000","last_clean":"2023-01-15T14:36:09.223542+0000","last_became_active":"2023-01-15T14:35:07.462526+0000","last_became_peered":"2023-01-15T14:35:07.462526+0000","last_unstale":"2023-01-15T14:36:09.223542+0000","last_undegraded":"2023-01-15T14:36:09.223542+0000","last_fullsized":"2023-01-15T14:36:09.223542+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"294'256","last_scrub_stamp":"2023-01-15T14:34:30.207952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:30.207952+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024191634,"objects_trimmed":0,"snaptrim_duration":0.10376478,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.5a","version":"472'381","reported_seq":1895,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.106207+0000","last_change":"2023-01-15T14:36:03.672905+0000","last_active":"2023-01-15T14:36:07.106207+0000","last_peered":"2023-01-15T14:36:07.106207+0000","last_clean":"2023-01-15T14:36:07.106207+0000","last_became_active":"2023-01-15T14:35:28.411845+0000","last_became_peered":"2023-01-15T14:35:28.411845+0000","last_unstale":"2023-01-15T14:36:07.106207+0000","last_undegraded":"2023-01-15T14:36:07.106207+0000","last_fullsized":"2023-01-15T14:36:07.106207+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":7,"last_scrub":"329'348","last_scrub_stamp":"2023-01-15T14:33:49.426430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:49.426430+0000","objects_scrubbed":2,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.85071893499999995,"objects_trimmed":0,"snaptrim_duration":0.61987404800000001,"stat_sum":{"num_bytes":3664025,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":37194,"num_write":8,"num_write_kb":1588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":15630873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.5b","version":"283'80","reported_seq":1232,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.438376+0000","last_change":"2023-01-15T14:36:09.439142+0000","last_active":"2023-01-15T14:36:09.438376+0000","last_peered":"2023-01-15T14:36:09.438376+0000","last_clean":"2023-01-15T14:36:09.438376+0000","last_became_active":"2023-01-15T14:35:57.399025+0000","last_became_peered":"2023-01-15T14:35:57.399025+0000","last_unstale":"2023-01-15T14:36:09.438376+0000","last_undegraded":"2023-01-15T14:36:09.438376+0000","last_fullsized":"2023-01-15T14:36:09.438376+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:31:37.483611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:37.483611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:53.267","scrub_duration":0.40660880900000002,"objects_trimmed":0,"snaptrim_duration":0.424862762,"stat_sum":{"num_bytes":31660,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":10510,"num_write":3,"num_write_kb":1011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.5c","version":"141'46","reported_seq":383,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.849153+0000","last_change":"2023-01-15T14:36:09.850161+0000","last_active":"2023-01-15T14:36:09.849153+0000","last_peered":"2023-01-15T14:36:09.849153+0000","last_clean":"2023-01-15T14:36:09.849153+0000","last_became_active":"2023-01-15T14:35:41.024937+0000","last_became_peered":"2023-01-15T14:35:41.024937+0000","last_unstale":"2023-01-15T14:36:09.849153+0000","last_undegraded":"2023-01-15T14:36:09.849153+0000","last_fullsized":"2023-01-15T14:36:09.849153+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"94'45","last_scrub_stamp":"2023-01-15T14:29:43.948230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:29:43.948230+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.73741684100000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.5d","version":"221'200","reported_seq":1646,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.021069+0000","last_change":"2023-01-15T14:36:09.021491+0000","last_active":"2023-01-15T14:36:09.021069+0000","last_peered":"2023-01-15T14:36:09.021069+0000","last_clean":"2023-01-15T14:36:09.021069+0000","last_became_active":"2023-01-15T14:36:00.421311+0000","last_became_peered":"2023-01-15T14:36:00.421311+0000","last_unstale":"2023-01-15T14:36:09.021069+0000","last_undegraded":"2023-01-15T14:36:09.021069+0000","last_fullsized":"2023-01-15T14:36:09.021069+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"221'200","last_scrub_stamp":"2023-01-15T14:34:19.729021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:19.729021+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.50499948100000003,"objects_trimmed":0,"snaptrim_duration":0.001827941,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.5e","version":"215'39","reported_seq":40,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072403+0000","last_change":"2023-01-15T14:36:00.121913+0000","last_active":"2023-01-15T14:36:05.072403+0000","last_peered":"2023-01-15T14:36:05.072403+0000","last_clean":"2023-01-15T14:36:05.072403+0000","last_became_active":"2023-01-15T14:35:57.307259+0000","last_became_peered":"2023-01-15T14:35:57.307259+0000","last_unstale":"2023-01-15T14:36:05.072403+0000","last_undegraded":"2023-01-15T14:36:05.072403+0000","last_fullsized":"2023-01-15T14:36:05.072403+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"215'39","last_scrub_stamp":"2023-01-15T14:33:16.134791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:16.134791+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.20235508099999999,"objects_trimmed":0,"snaptrim_duration":0.00063915599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.5f","version":"318'256","reported_seq":1721,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.106980+0000","last_change":"2023-01-15T14:36:04.572821+0000","last_active":"2023-01-15T14:36:07.106980+0000","last_peered":"2023-01-15T14:36:07.106980+0000","last_clean":"2023-01-15T14:36:07.106980+0000","last_became_active":"2023-01-15T14:36:00.333265+0000","last_became_peered":"2023-01-15T14:36:00.333265+0000","last_unstale":"2023-01-15T14:36:07.106980+0000","last_undegraded":"2023-01-15T14:36:07.106980+0000","last_fullsized":"2023-01-15T14:36:07.106980+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":325,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"318'256","last_scrub_stamp":"2023-01-15T14:33:42.042362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:42.042362+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.169368144,"objects_trimmed":0,"snaptrim_duration":0.411474485,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.60","version":"463'316","reported_seq":1834,"reported_epoch":478,"state":"active+recovering+undersized+remapped","last_fresh":"2023-01-15T14:36:08.831003+0000","last_change":"2023-01-15T14:36:08.510203+0000","last_active":"2023-01-15T14:36:08.831003+0000","last_peered":"2023-01-15T14:36:08.831003+0000","last_clean":"2023-01-15T14:35:55.200789+0000","last_became_active":"2023-01-15T14:35:58.318911+0000","last_became_peered":"2023-01-15T14:35:58.318911+0000","last_unstale":"2023-01-15T14:36:08.831003+0000","last_undegraded":"2023-01-15T14:36:08.831003+0000","last_fullsized":"2023-01-15T14:35:58.012773+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":7,"last_scrub":"402'291","last_scrub_stamp":"2023-01-15T14:34:55.535113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:55.535113+0000","objects_scrubbed":4,"log_size":316,"log_dups_size":0,"ondisk_log_size":316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.814267513,"objects_trimmed":0,"snaptrim_duration":0.242416507,"stat_sum":{"num_bytes":6004978,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":19,"num_read_kb":19071,"num_write":6,"num_write_kb":1008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":7126465,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4],"avail_no_missing":["1","4","7"],"object_location_counts":[{"shards":"1,4,7","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"133","length":"1"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.61","version":"470'186","reported_seq":1375,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.005071+0000","last_change":"2023-01-15T14:36:02.763507+0000","last_active":"2023-01-15T14:36:07.005071+0000","last_peered":"2023-01-15T14:36:07.005071+0000","last_clean":"2023-01-15T14:36:07.005071+0000","last_became_active":"2023-01-15T14:34:44.473866+0000","last_became_peered":"2023-01-15T14:34:44.473866+0000","last_unstale":"2023-01-15T14:36:07.005071+0000","last_undegraded":"2023-01-15T14:36:07.005071+0000","last_fullsized":"2023-01-15T14:36:07.005071+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":7,"last_scrub":"244'130","last_scrub_stamp":"2023-01-15T14:32:51.105424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:51.105424+0000","objects_scrubbed":1,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:43.765","scrub_duration":0.70550966500000001,"objects_trimmed":0,"snaptrim_duration":0.378204602,"stat_sum":{"num_bytes":3823692,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":16,"num_read_kb":16964,"num_write":7,"num_write_kb":3591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.62","version":"474'223","reported_seq":1852,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.002904+0000","last_change":"2023-01-15T14:36:05.972168+0000","last_active":"2023-01-15T14:36:07.002904+0000","last_peered":"2023-01-15T14:36:07.002904+0000","last_clean":"2023-01-15T14:36:07.002904+0000","last_became_active":"2023-01-15T14:36:04.162093+0000","last_became_peered":"2023-01-15T14:36:04.162093+0000","last_unstale":"2023-01-15T14:36:07.002904+0000","last_undegraded":"2023-01-15T14:36:07.002904+0000","last_fullsized":"2023-01-15T14:36:07.002904+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":7,"last_scrub":"403'189","last_scrub_stamp":"2023-01-15T14:34:56.851447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:56.851447+0000","objects_scrubbed":5,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30960584400000002,"objects_trimmed":0,"snaptrim_duration":0.70281312600000001,"stat_sum":{"num_bytes":10205020,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":10,"num_read_kb":9593,"num_write":4,"num_write_kb":962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":20928150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.63","version":"149'233","reported_seq":1104,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.171336+0000","last_change":"2023-01-15T14:36:01.238617+0000","last_active":"2023-01-15T14:36:05.171336+0000","last_peered":"2023-01-15T14:36:05.171336+0000","last_clean":"2023-01-15T14:36:05.171336+0000","last_became_active":"2023-01-15T14:36:00.347347+0000","last_became_peered":"2023-01-15T14:36:00.347347+0000","last_unstale":"2023-01-15T14:36:05.171336+0000","last_undegraded":"2023-01-15T14:36:05.171336+0000","last_fullsized":"2023-01-15T14:36:05.171336+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"149'233","last_scrub_stamp":"2023-01-15T14:34:56.637790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:56.637790+0000","objects_scrubbed":0,"log_size":233,"log_dups_size":0,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.63257800399999997,"objects_trimmed":0,"snaptrim_duration":0.0017279159999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.64","version":"334'251","reported_seq":1744,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.105062+0000","last_change":"2023-01-15T14:36:02.559054+0000","last_active":"2023-01-15T14:36:07.105062+0000","last_peered":"2023-01-15T14:36:07.105062+0000","last_clean":"2023-01-15T14:36:07.105062+0000","last_became_active":"2023-01-15T14:35:07.057736+0000","last_became_peered":"2023-01-15T14:35:07.057736+0000","last_unstale":"2023-01-15T14:36:07.105062+0000","last_undegraded":"2023-01-15T14:36:07.105062+0000","last_fullsized":"2023-01-15T14:36:07.105062+0000","mapping_epoch":416,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":417,"parent":"0.0","parent_split_bits":7,"last_scrub":"334'251","last_scrub_stamp":"2023-01-15T14:34:48.644529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:48.644529+0000","objects_scrubbed":0,"log_size":251,"log_dups_size":0,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70872698300000003,"objects_trimmed":0,"snaptrim_duration":0.20970626000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,6],"acting":[0,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.65","version":"465'209","reported_seq":1671,"reported_epoch":477,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2023-01-15T14:36:07.005020+0000","last_change":"2023-01-15T14:35:58.421712+0000","last_active":"2023-01-15T14:36:07.005020+0000","last_peered":"2023-01-15T14:36:07.005020+0000","last_clean":"2023-01-15T14:35:55.911448+0000","last_became_active":"2023-01-15T14:35:58.419261+0000","last_became_peered":"2023-01-15T14:35:58.419261+0000","last_unstale":"2023-01-15T14:36:07.005020+0000","last_undegraded":"2023-01-15T14:35:58.010100+0000","last_fullsized":"2023-01-15T14:35:58.007251+0000","mapping_epoch":467,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":7,"last_scrub":"371'126","last_scrub_stamp":"2023-01-15T14:34:23.903906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:23.903906+0000","objects_scrubbed":6,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.30981859099999998,"objects_trimmed":1,"snaptrim_duration":0.50843005500000005,"stat_sum":{"num_bytes":10562578,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":28,"num_read_kb":27857,"num_write":26,"num_write_kb":10132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20310327,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,6],"avail_no_missing":["1","6","7"],"object_location_counts":[{"shards":"1,6,7","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"133","length":"1"},{"start":"141","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"}]},{"pgid":"3.66","version":"102'52","reported_seq":36,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072281+0000","last_change":"2023-01-15T14:36:00.021529+0000","last_active":"2023-01-15T14:36:05.072281+0000","last_peered":"2023-01-15T14:36:05.072281+0000","last_clean":"2023-01-15T14:36:05.072281+0000","last_became_active":"2023-01-15T14:35:57.408886+0000","last_became_peered":"2023-01-15T14:35:57.408886+0000","last_unstale":"2023-01-15T14:36:05.072281+0000","last_undegraded":"2023-01-15T14:36:05.072281+0000","last_fullsized":"2023-01-15T14:36:05.072281+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"102'52","last_scrub_stamp":"2023-01-15T14:34:32.591909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:32.591909+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.39200112999999998,"objects_trimmed":0,"snaptrim_duration":0.00085080400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.67","version":"191'79","reported_seq":363,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.543100+0000","last_change":"2023-01-15T14:36:09.544242+0000","last_active":"2023-01-15T14:36:09.543100+0000","last_peered":"2023-01-15T14:36:09.543100+0000","last_clean":"2023-01-15T14:36:09.543100+0000","last_became_active":"2023-01-15T14:33:52.412250+0000","last_became_peered":"2023-01-15T14:33:52.412250+0000","last_unstale":"2023-01-15T14:36:09.543100+0000","last_undegraded":"2023-01-15T14:36:09.543100+0000","last_fullsized":"2023-01-15T14:36:09.543100+0000","mapping_epoch":342,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":343,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'79","last_scrub_stamp":"2023-01-15T14:32:11.667860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:11.667860+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.45094481199999997,"objects_trimmed":0,"snaptrim_duration":0.528493659,"stat_sum":{"num_bytes":1242779,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4229,"num_write":1,"num_write_kb":369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":865561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.68","version":"460'285","reported_seq":1813,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.103278+0000","last_change":"2023-01-15T14:36:02.750865+0000","last_active":"2023-01-15T14:36:07.103278+0000","last_peered":"2023-01-15T14:36:07.103278+0000","last_clean":"2023-01-15T14:36:07.103278+0000","last_became_active":"2023-01-15T14:35:39.400045+0000","last_became_peered":"2023-01-15T14:35:39.400045+0000","last_unstale":"2023-01-15T14:36:07.103278+0000","last_undegraded":"2023-01-15T14:36:07.103278+0000","last_fullsized":"2023-01-15T14:36:07.103278+0000","mapping_epoch":448,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":449,"parent":"0.0","parent_split_bits":7,"last_scrub":"132'188","last_scrub_stamp":"2023-01-15T14:30:23.362477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:23.362477+0000","objects_scrubbed":1,"log_size":285,"log_dups_size":0,"ondisk_log_size":285,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.70563046500000004,"objects_trimmed":0,"snaptrim_duration":0.302118522,"stat_sum":{"num_bytes":42018,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":21,"num_read_kb":21385,"num_write":11,"num_write_kb":8288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":23162510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.69","version":"110'149","reported_seq":1104,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.072820+0000","last_change":"2023-01-15T14:36:00.955236+0000","last_active":"2023-01-15T14:36:05.072820+0000","last_peered":"2023-01-15T14:36:05.072820+0000","last_clean":"2023-01-15T14:36:05.072820+0000","last_became_active":"2023-01-15T14:36:00.324381+0000","last_became_peered":"2023-01-15T14:36:00.324381+0000","last_unstale":"2023-01-15T14:36:05.072820+0000","last_undegraded":"2023-01-15T14:36:05.072820+0000","last_fullsized":"2023-01-15T14:36:05.072820+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":335,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'149","last_scrub_stamp":"2023-01-15T14:33:15.196041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:33:15.196041+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.27905627500000002,"objects_trimmed":0,"snaptrim_duration":0.62944658399999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.6a","version":"191'191","reported_seq":908,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.002407+0000","last_change":"2023-01-15T14:36:00.949991+0000","last_active":"2023-01-15T14:36:07.002407+0000","last_peered":"2023-01-15T14:36:07.002407+0000","last_clean":"2023-01-15T14:36:07.002407+0000","last_became_active":"2023-01-15T14:35:41.324516+0000","last_became_peered":"2023-01-15T14:35:41.324516+0000","last_unstale":"2023-01-15T14:36:07.002407+0000","last_undegraded":"2023-01-15T14:36:07.002407+0000","last_fullsized":"2023-01-15T14:36:07.002407+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"191'191","last_scrub_stamp":"2023-01-15T14:31:31.658924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:31.658924+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.83384637100000003,"stat_sum":{"num_bytes":1518387,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":5931,"num_write":14,"num_write_kb":6502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":9447727,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.6b","version":"155'53","reported_seq":242,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.329083+0000","last_change":"2023-01-15T14:36:09.329500+0000","last_active":"2023-01-15T14:36:09.329083+0000","last_peered":"2023-01-15T14:36:09.329083+0000","last_clean":"2023-01-15T14:36:09.329083+0000","last_became_active":"2023-01-15T14:35:57.296638+0000","last_became_peered":"2023-01-15T14:35:57.296638+0000","last_unstale":"2023-01-15T14:36:09.329083+0000","last_undegraded":"2023-01-15T14:36:09.329083+0000","last_fullsized":"2023-01-15T14:36:09.329083+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'53","last_scrub_stamp":"2023-01-15T14:32:55.664149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:55.664149+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.40639952299999998,"objects_trimmed":0,"snaptrim_duration":0.31680413800000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.6c","version":"141'46","reported_seq":219,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.432712+0000","last_change":"2023-01-15T14:36:09.433702+0000","last_active":"2023-01-15T14:36:09.432712+0000","last_peered":"2023-01-15T14:36:09.432712+0000","last_clean":"2023-01-15T14:36:09.432712+0000","last_became_active":"2023-01-15T14:35:04.085770+0000","last_became_peered":"2023-01-15T14:35:04.085770+0000","last_unstale":"2023-01-15T14:36:09.432712+0000","last_undegraded":"2023-01-15T14:36:09.432712+0000","last_fullsized":"2023-01-15T14:36:09.432712+0000","mapping_epoch":413,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":414,"parent":"0.0","parent_split_bits":7,"last_scrub":"141'46","last_scrub_stamp":"2023-01-15T14:34:57.577405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:57.577405+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.60435901599999997,"objects_trimmed":0,"snaptrim_duration":0.420149196,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":289917,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.6d","version":"168'156","reported_seq":891,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.223723+0000","last_change":"2023-01-15T14:36:09.224683+0000","last_active":"2023-01-15T14:36:09.223723+0000","last_peered":"2023-01-15T14:36:09.223723+0000","last_clean":"2023-01-15T14:36:09.223723+0000","last_became_active":"2023-01-15T14:35:41.216059+0000","last_became_peered":"2023-01-15T14:35:41.216059+0000","last_unstale":"2023-01-15T14:36:09.223723+0000","last_undegraded":"2023-01-15T14:36:09.223723+0000","last_fullsized":"2023-01-15T14:36:09.223723+0000","mapping_epoch":450,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":451,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'154","last_scrub_stamp":"2023-01-15T14:30:47.381645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:47.381645+0000","objects_scrubbed":3,"log_size":156,"log_dups_size":0,"ondisk_log_size":156,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.858963539,"objects_trimmed":0,"snaptrim_duration":0.104768205,"stat_sum":{"num_bytes":1516832,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2710,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.6e","version":"139'38","reported_seq":209,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.225264+0000","last_change":"2023-01-15T14:36:09.225976+0000","last_active":"2023-01-15T14:36:09.225264+0000","last_peered":"2023-01-15T14:36:09.225264+0000","last_clean":"2023-01-15T14:36:09.225264+0000","last_became_active":"2023-01-15T14:34:59.097277+0000","last_became_peered":"2023-01-15T14:34:59.097277+0000","last_unstale":"2023-01-15T14:36:09.225264+0000","last_undegraded":"2023-01-15T14:36:09.225264+0000","last_fullsized":"2023-01-15T14:36:09.225264+0000","mapping_epoch":408,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":7,"last_scrub":"139'38","last_scrub_stamp":"2023-01-15T14:31:47.583915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:47.583915+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:32.337","scrub_duration":0.58480571100000001,"objects_trimmed":0,"snaptrim_duration":0.10516239500000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.6f","version":"155'145","reported_seq":1274,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.109370+0000","last_change":"2023-01-15T14:36:02.759211+0000","last_active":"2023-01-15T14:36:07.109370+0000","last_peered":"2023-01-15T14:36:07.109370+0000","last_clean":"2023-01-15T14:36:07.109370+0000","last_became_active":"2023-01-15T14:35:02.254791+0000","last_became_peered":"2023-01-15T14:35:02.254791+0000","last_unstale":"2023-01-15T14:36:07.109370+0000","last_undegraded":"2023-01-15T14:36:07.109370+0000","last_fullsized":"2023-01-15T14:36:07.109370+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"155'145","last_scrub_stamp":"2023-01-15T14:32:05.799608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.799608+0000","objects_scrubbed":0,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:56.751","scrub_duration":0.40426945399999997,"objects_trimmed":0,"snaptrim_duration":0.20228891900000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.70","version":"206'252","reported_seq":1364,"reported_epoch":475,"state":"active+clean","last_fresh":"2023-01-15T14:36:05.171687+0000","last_change":"2023-01-15T14:36:01.637490+0000","last_active":"2023-01-15T14:36:05.171687+0000","last_peered":"2023-01-15T14:36:05.171687+0000","last_clean":"2023-01-15T14:36:05.171687+0000","last_became_active":"2023-01-15T14:36:00.616209+0000","last_became_peered":"2023-01-15T14:36:00.616209+0000","last_unstale":"2023-01-15T14:36:05.171687+0000","last_undegraded":"2023-01-15T14:36:05.171687+0000","last_fullsized":"2023-01-15T14:36:05.171687+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"206'252","last_scrub_stamp":"2023-01-15T14:34:36.078678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:34:36.078678+0000","objects_scrubbed":0,"log_size":252,"log_dups_size":0,"ondisk_log_size":252,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.158842567,"objects_trimmed":0,"snaptrim_duration":0.51240869499999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"115","length":"1"},{"start":"117","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"1"},{"start":"137","length":"3"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.71","version":"204'150","reported_seq":1073,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-01-15T14:36:09.122255+0000","last_change":"2023-01-15T14:36:09.222369+0000","last_active":"2023-01-15T14:36:09.122255+0000","last_peered":"2023-01-15T14:36:09.122255+0000","last_clean":"2023-01-15T14:36:09.122255+0000","last_became_active":"2023-01-15T14:35:02.151543+0000","last_became_peered":"2023-01-15T14:35:02.151543+0000","last_unstale":"2023-01-15T14:36:09.122255+0000","last_undegraded":"2023-01-15T14:36:09.122255+0000","last_fullsized":"2023-01-15T14:36:09.122255+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"204'150","last_scrub_stamp":"2023-01-15T14:32:05.834732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:05.834732+0000","objects_scrubbed":4,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:31.314","scrub_duration":0.106793473,"objects_trimmed":0,"snaptrim_duration":0.103200008,"stat_sum":{"num_bytes":3558907,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10725,"num_write":4,"num_write_kb":3090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3605110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"},{"start":"15d","length":"1"}]},{"pgid":"3.72","version":"394'169","reported_seq":1209,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.104767+0000","last_change":"2023-01-15T14:36:02.968610+0000","last_active":"2023-01-15T14:36:07.104767+0000","last_peered":"2023-01-15T14:36:07.104767+0000","last_clean":"2023-01-15T14:36:07.104767+0000","last_became_active":"2023-01-15T14:35:02.148756+0000","last_became_peered":"2023-01-15T14:35:02.148756+0000","last_unstale":"2023-01-15T14:36:07.104767+0000","last_undegraded":"2023-01-15T14:36:07.104767+0000","last_fullsized":"2023-01-15T14:36:07.104767+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":7,"last_scrub":"212'151","last_scrub_stamp":"2023-01-15T14:31:44.594582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:31:44.594582+0000","objects_scrubbed":2,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:52.712","scrub_duration":0.70660182000000005,"objects_trimmed":0,"snaptrim_duration":0.61956889699999995,"stat_sum":{"num_bytes":2663943,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":3524,"num_write":1,"num_write_kb":1428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2663943,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.73","version":"204'275","reported_seq":1572,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.102313+0000","last_change":"2023-01-15T14:36:02.170853+0000","last_active":"2023-01-15T14:36:07.102313+0000","last_peered":"2023-01-15T14:36:07.102313+0000","last_clean":"2023-01-15T14:36:07.102313+0000","last_became_active":"2023-01-15T14:36:00.160714+0000","last_became_peered":"2023-01-15T14:36:00.160714+0000","last_unstale":"2023-01-15T14:36:07.102313+0000","last_undegraded":"2023-01-15T14:36:07.102313+0000","last_fullsized":"2023-01-15T14:36:07.102313+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":405,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":7,"last_scrub":"122'213","last_scrub_stamp":"2023-01-15T14:30:11.733854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:30:11.733854+0000","objects_scrubbed":4,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"priority scrub will be retried not before 2023-01-15T14:36:34.588","scrub_duration":0.50670270299999998,"objects_trimmed":0,"snaptrim_duration":1.114955728,"stat_sum":{"num_bytes":3485477,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":12,"num_read_kb":11189,"num_write":7,"num_write_kb":3286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.74","version":"169'138","reported_seq":932,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.005519+0000","last_change":"2023-01-15T14:36:02.759956+0000","last_active":"2023-01-15T14:36:07.005519+0000","last_peered":"2023-01-15T14:36:07.005519+0000","last_clean":"2023-01-15T14:36:07.005519+0000","last_became_active":"2023-01-15T14:35:38.078356+0000","last_became_peered":"2023-01-15T14:35:38.078356+0000","last_unstale":"2023-01-15T14:36:07.005519+0000","last_undegraded":"2023-01-15T14:36:07.005519+0000","last_fullsized":"2023-01-15T14:36:07.005519+0000","mapping_epoch":447,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":448,"parent":"0.0","parent_split_bits":7,"last_scrub":"169'138","last_scrub_stamp":"2023-01-15T14:32:56.861830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:56.861830+0000","objects_scrubbed":1,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.51303928099999996,"objects_trimmed":0,"snaptrim_duration":0.45484852100000001,"stat_sum":{"num_bytes":613464,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":2809,"num_write":2,"num_write_kb":305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]},{"pgid":"3.75","version":"110'55","reported_seq":151,"reported_epoch":477,"state":"active+clean","last_fresh":"2023-01-15T14:36:07.003617+0000","last_change":"2023-01-15T14:36:01.860878+0000","last_active":"2023-01-15T14:36:07.003617+0000","last_peered":"2023-01-15T14:36:07.003617+0000","last_clean":"2023-01-15T14:36:07.003617+0000","last_became_active":"2023-01-15T14:35:57.308332+0000","last_became_peered":"2023-01-15T14:35:57.308332+0000","last_unstale":"2023-01-15T14:36:07.003617+0000","last_undegraded":"2023-01-15T14:36:07.003617+0000","last_fullsized":"2023-01-15T14:36:07.003617+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":428,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":7,"last_scrub":"110'55","last_scrub_stamp":"2023-01-15T14:32:45.943719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-01-15T14:28:22.711572+0000","last_clean_scrub_stamp":"2023-01-15T14:32:45.943719+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.80482230600000004,"stat_sum":{"num_bytes":796228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":5170,"num_write":3,"num_write_kb":1557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":531514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"133","length":"1"},{"start":"140","length":"2"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"14a","length":"1"},{"start":"14e","length":"1"},{"start":"150","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":156,"stat_sum":{"num_bytes":391464854,"num_objects":177,"num_object_clones":123,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":95,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":14,"num_read":1324,"num_read_kb":1469625,"num_write":1004,"num_write_kb":399252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1345,"num_bytes_recovered":898063920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28255,"ondisk_log_size":28255,"up":438,"acting":429,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574946,"num_pgs":73,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":1109128,"kb_used_data":92590432,"kb_used_omap":15964,"kb_used_meta":0,"kb_avail":92590252,"statfs":{"total":95948165120,"available":94812418048,"internally_reserved":184320,"allocated":94812602368,"data_stored":94812602368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16347500,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,0,1,0,0,1],"upper_bound":4096},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":1,"commit_latency_ns":0,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":6.6619999999999999,"5min":4.6289999999999996,"15min":4.0369999999999999},"min":{"1min":0.53400000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":83.435000000000002,"5min":148.10900000000001,"15min":148.10900000000001},"last":1.1659999999999999},{"interface":"front","average":{"1min":0.77100000000000002,"5min":4.0030000000000001,"15min":7.4720000000000004},"min":{"1min":0.59599999999999997,"5min":0.314,"15min":0.314},"max":{"1min":1.1599999999999999,"5min":96.337000000000003,"15min":96.337000000000003},"last":1.004}]},{"osd":1,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":3.6589999999999998,"15min":5.5670000000000002},"min":{"1min":0.64300000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.03,"5min":204.74199999999999,"15min":204.74199999999999},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.022,"5min":8.3290000000000006,"15min":6.899},"min":{"1min":0.67500000000000004,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":3.4089999999999998,"5min":97.704999999999998,"15min":97.704999999999998},"last":0.90600000000000003}]},{"osd":2,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":9.9890000000000008,"5min":5.6239999999999997,"15min":5.6769999999999996},"min":{"1min":0.54400000000000004,"5min":0.377,"15min":0.377},"max":{"1min":76.272999999999996,"5min":151.31800000000001,"15min":151.31800000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.82999999999999996,"5min":4.75,"15min":4.1500000000000004},"min":{"1min":0.64900000000000002,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.105,"5min":162.96799999999999,"15min":162.96799999999999},"last":1.0920000000000001}]},{"osd":3,"last update":"Sun Jan 15 14:35:22 2023","interfaces":[{"interface":"back","average":{"1min":8.1660000000000004,"5min":6.375,"15min":3.6560000000000001},"min"